Kubelet density

Kubelet density determines the resilience of the kubelet by creating pods on a specific node.

• It helps determine how resilient an application is to the unplanned scaling of K8s pods.
• It also helps determine the performance of the kubelet for a specific node.

Usage

View the uses of the fault

This fault helps determine how resilient an application is to the unplanned scaling of K8s pods. In distributed systems like Kube resilience, application replicas may not be sufficient to manage the traffic (indicated by SLIs) during any kind of failures (system or application failure). In such cases, the application needs to meet the SLOs (service level objectives). For this purpose, it is important to ensure that the application has a minimum number of replicas available. A common application failure is when the pressure on other replicas increases, how the horizontal pod autoscaler (HPA) scales based on the observed resource utilization, and the amount of time the persistent volume takes to mount on rescheduling. It simulates pod-storm (due to autoscale) on high traffic conditions and verifies successful functioning of the application services (from latency and availability standpoint). It ensures that the topology constraints are adhered to on pod scale (node selectors, tolerations, zone distribution, affinity (or anti-affinity policies)). It also verifies pod creation and scheduling SLIs on the cluster nodes.

Prerequisites

• Kubernetes > 1.16.

Default validations

The target nodes should be in a healthy state before and after injecting chaos.

Fault Tunables

Fault tunables

Mandatory Fields

Variables	Description	Notes
TARGET_NODE	Name of the target node.	If this environment variable isn't set, a random target node is selected.

Optional Fields

Variables	Description	Notes
TOTAL_CHAOS_DURATION	Duration that you specify, through which chaos is injected into the target resource (in seconds).	Defaults to 90s.
POD_COUNT	Total number of pods that are created during chaos.	Defaults to 50.
TARGET_NAMESPACE	Namespace where the pods will be created.	Defaults to the namespace specified in CHAOS_NAMESPACE.
POD_TEMPLATE_CM	Name of the config map that contains the pod template.	For example: stress-app-manifest.
POD_TEMPLATE_PATH	Path to the pod template configMap mount.	Defaults to /templates/pod.yml
POD_SELECTOR	Labels of destination pods.	Defaults to {name: kubelet-density-app}.
POD_IMAGE	Pod image used to create multiple pods.	Defaults to gcr.io/google_containers/pause-amd64:3.0.
RAMP_TIME	Period to wait before and after injecting chaos (in seconds).	For example, 30s.

Fault examples

Common fault tunables

Refer to the common attributes to tune the common tunables for all the faults.

Pod template provided as configmap

A chaos experiment creates pods on the target node during execution. The template for the pod is provided by mounting the pod template using the configmap and passing the name and mountPath of the pod using the POD_TEMPLATE_CM and POD_TEMPLATE_PATH environment variables, respectively.

Use the following example to tune it:

# defines pod template cm and its mount path
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  annotationCheck: "false"
  chaosServiceAccount: litmus-admin
  experiments:
    - name: kubelet-density
      spec:
        components:
          env:
            # name of the pod template cm
            - name: POD_TEMPLATE_CM
              value: "pod-template-cm"
              # mount path of the cm
            - name: POD_TEMPLATE_PATH
              VALUE: "/templates/pod.yml"
            - name: TARGET_NODE
              value: "node1"

Pod count

It specifies the number of the pods that will be created on the target node. You can tune it using the POD_COUNT environment variable.

Use the following example to tune it:

# defines count of the pod
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  annotationCheck: "false"
  chaosServiceAccount: litmus-admin
  experiments:
    - name: kubelet-density
      spec:
        components:
          env:
            # number of pods, which needs to be created
            - name: POD_COUNT
              value: "50"
            - name: TARGET_NODE
              value: "node1"

Target namespace

It specifies the namespace where the pods are created. You can tune it using the TARGET_NAMESPACE environment variable.

Use the following example to tune it:

# defines pod namespace
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  annotationCheck: "false"
  chaosServiceAccount: litmus-admin
  experiments:
    - name: kubelet-density
      spec:
        components:
          env:
            # namespace where pods need to be created
            - name: TARGET_NAMESPACE
              value: "litmus"
            - name: TARGET_NODE
              value: "node1"

Pod image and pod selectors

You can tune pod image and label selectors using the POD_IMAGE and POD_SELECTOR environment variables, respectively.

Use the following example to tune it:

# defines pod image and label selectors
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  annotationCheck: "false"
  chaosServiceAccount: litmus-admin
  experiments:
    - name: kubelet-density
      spec:
        components:
          env:
            # image of the pod
            - name: POD_IMAGE
              value: "nginx"
            # pod label selectors
            - name: POD_SELECTOR
              value: "{name: kubelet-density-app}"
            - name: TARGET_NODE
              value: "node1"