Kubelet density

Kubelet density determines the resilience of the kubelet by creating pods on a specific node.

• In distributed systems like Kube resilience, application replicas might not be sufficient to manage the traffic (indicated by SLIs) during system (or application) failures.
• A common application failure occurs when the pressure on other replicas increases, and the horizontal pod autoscaler (HPA) scales based on the observed resource utilization, and the amount of time it takes the persistent volume to mount on rescheduling.
• In case of failures, the application needs to meet the SLOs (service level objectives) by making a minimum number of replicas available.

Permissions required

Below is a sample Kubernetes role that defines the permissions required to execute the fault.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: hce
  name: kubelet-density
spec:
  definition:
    scope: Namespaced # Supports Cluster mode too
permissions:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["create", "delete", "get", "list", "patch", "deletecollection", "update"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["create", "get", "list", "patch", "update"]
  - apiGroups: [""]
    resources: ["configMaps"]
    verbs: ["get", "list"]
  - apiGroups: [""]
    resources: ["chaosEngines", "chaosExperiments", "chaosResults"]
    verbs: ["create", "delete", "get", "list", "patch", "update"]
  - apiGroups: [""]
    resources: ["pods/log"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get", "list"]
  - apiGroups: ["batch"]
    resources: ["jobs"]
    verbs: ["create", "delete", "get", "list", "deletecollection"]
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get", "list"]

Use cases

Kubelet density:

• Determines the resilience of an application to unplanned scaling of Kubernetes pods.
• It simulates pod-storm (due to autoscale) on high traffic conditions.
• It verifies functioning of the application services from latency and availability standpoint.
• It ensures that the topology constraints are adhered to on pod scale (node selectors, tolerations, zone distribution, affinity or anti-affinity policies).
• It also verifies pod creation and scheduling SLIs on the cluster nodes.
• It also helps determine the performance of the kubelet for a specific node.

Prerequisites

• Kubernetes > 1.16
• The target nodes should be in the healthy state before and after injecting chaos.

Mandatory tunables

Tunable	Description	Notes
TARGET_NODE	Name of the target node.	If this environment variable isn't set, a random target node is selected. For more information, go to target node.

Optional tunables

Tunable	Description	Notes
TOTAL_CHAOS_DURATION	Duration that you specify, through which chaos is injected into the target resource (in seconds).	Defaults to 90s. For more information, go to duration of the chaos.
POD_COUNT	Total number of pods that are created during chaos.	Default: 50. For more information, go to pod count.
TARGET_NAMESPACE	Namespace where the pods will be created.	Defaults to the namespace specified in CHAOS_NAMESPACE. For more information, go to target namespace.
POD_TEMPLATE_CM	Name of the config map that contains the pod template.	For example: stress-app-manifest. For more information, go to pod template provided as a configmap.
POD_TEMPLATE_PATH	Path to the pod template configMap mount.	Defaults to /templates/pod.yml. For more information, go to pod template provided as a configmap
POD_SELECTOR	Labels of destination pods.	Defaults to {name: kubelet-density-app}. For more information, go to pod selector.
POD_IMAGE	Pod image used to create multiple pods.	Defaults to gcr.io/google_containers/pause-amd64:3.0. For more information, go to pod image.
RAMP_TIME	Period to wait before and after injecting chaos (in seconds).	For example, 30s. For more information, go to ramp time.

Pod template provided as a configmap

A chaos experiment creates pods on the target node during execution. The template for the pod is provided by mounting the pod template using the configmap and passing the name and mountPath of the pod using the POD_TEMPLATE_CM and POD_TEMPLATE_PATH environment variables, respectively.

The following YAML snippet illustrates the use of this environment variable:

# defines pod template cm and its mount path
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  annotationCheck: "false"
  chaosServiceAccount: litmus-admin
  experiments:
    - name: kubelet-density
      spec:
        components:
          env:
            # name of the pod template cm
            - name: POD_TEMPLATE_CM
              value: "pod-template-cm"
              # mount path of the cm
            - name: POD_TEMPLATE_PATH
              VALUE: "/templates/pod.yml"
            - name: TARGET_NODE
              value: "node1"

Pod count

It specifies the number of the pods that will be created on the target node. Tune it by using the POD_COUNT environment variable.

The following YAML snippet illustrates the use of this environment variable:

# defines count of the pod
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  annotationCheck: "false"
  chaosServiceAccount: litmus-admin
  experiments:
    - name: kubelet-density
      spec:
        components:
          env:
            # number of pods, which needs to be created
            - name: POD_COUNT
              value: "50"
            - name: TARGET_NODE
              value: "node1"

Target namespace

It specifies the namespace where the pods are created. Tune it by using the TARGET_NAMESPACE environment variable.

The following YAML snippet illustrates the use of this environment variable:

# defines pod namespace
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  annotationCheck: "false"
  chaosServiceAccount: litmus-admin
  experiments:
    - name: kubelet-density
      spec:
        components:
          env:
            # namespace where pods need to be created
            - name: TARGET_NAMESPACE
              value: "litmus"
            - name: TARGET_NODE
              value: "node1"

Pod image and pod selectors

Tune the pod image and label selectors by using the POD_IMAGE and POD_SELECTOR environment variables, respectively.

The following YAML snippet illustrates the use of this environment variable:

# defines pod image and label selectors
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  annotationCheck: "false"
  chaosServiceAccount: litmus-admin
  experiments:
    - name: kubelet-density
      spec:
        components:
          env:
            # image of the pod
            - name: POD_IMAGE
              value: "nginx"
            # pod label selectors
            - name: POD_SELECTOR
              value: "{name: kubelet-density-app}"
            - name: TARGET_NODE
              value: "node1"