Skip to main content

Pod memory hog

Pod memory hog is a Kubernetes pod-level chaos fault that consumes memory resources in excess, resulting in a significant spike in the memory usage of a pod.

  • Simulates a condition where the memory usage of an application spikes up unexpectedly.

Pod Memory Hog

Usage

View fault usage
Memory usage within containers is subject to various constraints in Kubernetes. If the limits are specified in their spec, exceeding them results in termination of the container (due to OOMKill of the primary process, often pid 1). This restarts container dependng on policy specified. For containers with no limits on memory, node can be killed based on their oom_score. This results in a bigger blast radius. It simulates the situation of memory leaks in the deployment of microservices, application slowness due to memory starvation, and noisy neighbour problems due to hogging. It verifies pod priority and QoS setting for eviction purposes. It also verifies application restarts on OOM kills. This fault causes stress within the target container, which may result in the primary process in the container to be constrained or eat up the available system memory on the node.

Prerequisites

  • Kubernetes> 1.16.

Default validations

The application pods should be in running state before and after chaos injection.

Fault tunables

Fault tunables

Optional fields

Variables Description s
MEMORY_CONSUMPTION The amount of memory used of hogging a Kubernetes pod (megabytes) Defaults to 500MB
NUMBER_OF_WORKERS The number of workers used to run the stress process Defaults to 1
TOTAL_CHAOS_DURATION The time duration for chaos insertion (seconds) Defaults to 60s
LIB_IMAGE Image used to run the helper pod. Defaults to litmuschaos/go-runner:1.13.8
STRESS_IMAGE Container run on the node at runtime by the pumba lib to inject stressors. Only used in LIB pumba Default to alexeiled/stress-ng:latest-ubuntu
TARGET_PODS Comma separated list of application pod name subjected to pod memory hog chaos If not provided, it will select target pods randomly based on provided appLabels
TARGET_CONTAINER Name of the target container under chaos. If not provided, it will select the first container of the target pod
CONTAINER_RUNTIME container runtime interface for the cluster Defaults to docker, supported values: docker, containerd and crio for litmus and only docker for pumba LIB
SOCKET_PATH Path of the containerd/crio/docker socket file Defaults to /var/run/docker.sock
PODS_AFFECTED_PERC The Percentage of total pods to target Defaults to 0 (corresponds to 1 replica), provide numeric value only
RAMP_TIME Period to wait before injection of chaos in sec For example, 30
SEQUENCE It defines sequence of chaos execution for multiple target pods Default value: parallel. Supported: serial, parallel

Fault examples

Common and pod-specific tunables

Refer to the common attributes and pod-specific tunables to tune the common tunables for all fault and pod specific tunables.

Memory consumption

It specifies the amount of memory consumed by the target pod for a duration specified by TOTAL_CHAOS_DURATION environment variable. You can tune it using the MEMORY_CONSUMPTION environment variable.

Use the following example to tune it:

# define the memory consumption in MB
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
name: engine-nginx
spec:
engineState: "active"
annotationCheck: "false"
appinfo:
appns: "default"
applabel: "app=nginx"
appkind: "deployment"
chaosServiceAccount: litmus-admin
experiments:
- name: pod-memory-hog
spec:
components:
env:
# memory consumption value
- name: MEMORY_CONSUMPTION
value: "500" #in MB
- name: TOTAL_CHAOS_DURATION
value: "60"

Workers for stress

It specifies the number of workers used to stress the resources. You can tune it using the NUMBER_OF_WORKERS environment variable.

Use the following example to tune it:

# number of workers used for the stress
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
name: engine-nginx
spec:
engineState: "active"
annotationCheck: "false"
appinfo:
appns: "default"
applabel: "app=nginx"
appkind: "deployment"
chaosServiceAccount: litmus-admin
experiments:
- name: pod-memory-hog
spec:
components:
env:
# number of workers for stress
- name: NUMBER_OF_WORKERS
value: "1"
- name: TOTAL_CHAOS_DURATION
value: "60"

Container runtime and socket path

It defines the CONTAINER_RUNTIME and SOCKET_PATH envrionment variables to set the container runtime and socket file path, respectively.

  • CONTAINER_RUNTIME: It supports docker, containerd, and crio runtimes. The default value is docker.
  • SOCKET_PATH: It contains path of docker socket file by default(/var/run/docker.sock). For containerd, specify path as /var/containerd/containerd.sock. For crio, speecify path as /var/run/crio/crio.sock.