Skip to main content

Disk Fill

Introduction

  • It causes Disk Stress by filling up the ephemeral storage of the pod on any given node.
  • It causes the application pod to get evicted if the capacity filled exceeds the pod's ephemeral storage limit.
  • It tests the Ephemeral Storage Limits, to ensure those parameters are sufficient.
  • It tests the application's resiliency to disk stress/replica evictions.
Fault execution flow chart

Disk Fill

Uses

View the uses of the fault
Coming soon.

Prerequisites

info
  • Ensure that Kubernetes Version > 1.16.
  • Appropriate Ephemeral Storage Requests and Limits should be set for the application before running the fault. An example specification is shown below:
    apiVersion: v1
    kind: Pod
    metadata:
    name: frontend
    spec:
    containers:
    - name: db
    image: mysql
    env:
    - name: MYSQL_ROOT_PASSWORD
    value: "password"
    resources:
    requests:
    ephemeral-storage: "2Gi"
    limits:
    ephemeral-storage: "4Gi"
    - name: wp
    image: wordpress
    resources:
    requests:
    ephemeral-storage: "2Gi"
    limits:
    ephemeral-storage: "4Gi"

Default Validations

note

The application pods should be in running state before and after chaos injection.

Fault Tunables

Check the Fault Tunables

Optional Fields

Variables Description Notes
FILL_PERCENTAGE Percentage to fill the Ephemeral storage limit Can be set to more than 100 also, to force evict the pod. The ephemeral-storage limits must be set in targeted pod to use this ENV.
EPHEMERAL_STORAGE_MEBIBYTES Ephemeral storage which need to fill (unit: MiBi)It is mutually exclusive with the FILL_PERCENTAGE ENV. If both are provided then it will use the FILL_PERCENTAGE

Optional Fields

Variables Description Notes
TARGET_CONTAINER Name of container which is subjected to disk-fill If not provided, the first container in the targeted pod will be subject to chaos
CONTAINER_PATH Storage Location of containers Defaults to '/var/lib/docker/containers'
TOTAL_CHAOS_DURATION The time duration for chaos insertion (sec) Defaults to 60s
TARGET_PODS Comma separated list of application pod name subjected to disk fill chaos If not provided, it will select target pods randomly based on provided appLabels
DATA_BLOCK_SIZE It contains data block size used to fill the disk(in KB) Defaults to 256, it supports unit as KB only
PODS_AFFECTED_PERC The Percentage of total pods to target Defaults to 0 (corresponds to 1 replica), provide numeric value only
LIB The chaos lib used to inject the chaos Defaults to `litmus` supported litmus only
LIB_IMAGE The image used to fill the disk Defaults to litmuschaos/go-runner:latest
RAMP_TIME Period to wait before injection of chaos in sec Eg. 30
SEQUENCE It defines sequence of chaos execution for multiple target pods Default value: parallel. Supported: serial, parallel

Fault Examples

Common and Pod specific tunables

Refer the common attributes and Pod specific tunable to tune the common tunables for all fault and pod specific tunables.

Disk Fill Percentage

It fills the FILL_PERCENTAGE percentage of the ephemeral-storage limit specified at resource.limits.ephemeral-storage inside the target application.

Use the following example to tune this:

## percentage of ephemeral storage limit specified at `resource.limits.ephemeral-storage` inside target application 
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
name: engine-nginx
spec:
engineState: "active"
annotationCheck: "false"
appinfo:
appns: "default"
applabel: "app=nginx"
appkind: "deployment"
chaosServiceAccount: litmus-admin
experiments:
- name: disk-fill
spec:
components:
env:
## percentage of ephemeral storage limit, which needs to be filled
- name: FILL_PERCENTAGE
value: '80' # in percentage
- name: TOTAL_CHAOS_DURATION
VALUE: '60'

Disk Fill Mebibytes

It fills the EPHEMERAL_STORAGE_MEBIBYTES MiBi of ephemeral storage of the targeted pod. It is mutually exclusive with the FILL_PERCENTAGE ENV. If FILL_PERCENTAGE ENV is set then it will use the percentage for the fill otherwise, it will fill the ephemeral storage based on EPHEMERAL_STORAGE_MEBIBYTES ENV.

Use the following example to tune this:

# ephemeral storage which needs to fill in will application
# if ephemeral-storage limits is not specified inside target application
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
name: engine-nginx
spec:
engineState: "active"
annotationCheck: "false"
appinfo:
appns: "default"
applabel: "app=nginx"
appkind: "deployment"
chaosServiceAccount: litmus-admin
experiments:
- name: disk-fill
spec:
components:
env:
## ephemeral storage size, which needs to be filled
- name: EPHEMERAL_STORAGE_MEBIBYTES
value: '256' #in MiBi
- name: TOTAL_CHAOS_DURATION
VALUE: '60'

Data Block Size

It defines the size of the data block used to fill the ephemeral storage of the targeted pod. It can be tuned via DATA_BLOCK_SIZE ENV. Its unit is KB. The default value of DATA_BLOCK_SIZE is 256.

Use the following example to tune this:

# size of the data block used to fill the disk
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
name: engine-nginx
spec:
engineState: "active"
annotationCheck: "false"
appinfo:
appns: "default"
applabel: "app=nginx"
appkind: "deployment"
chaosServiceAccount: litmus-admin
experiments:
- name: disk-fill
spec:
components:
env:
## size of data block used to fill the disk
- name: DATA_BLOCK_SIZE
value: '256' #in KB
- name: TOTAL_CHAOS_DURATION
VALUE: '60'

Container Path

It defines the storage location of the containers inside the host(node/VM). It can be tuned via CONTAINER_PATH ENV.

Use the following example to tune this:

# path inside node/vm where containers are present 
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
name: engine-nginx
spec:
engineState: "active"
annotationCheck: "false"
appinfo:
appns: "default"
applabel: "app=nginx"
appkind: "deployment"
chaosServiceAccount: litmus-admin
experiments:
- name: disk-fill
spec:
components:
env:
# storage location of the containers
- name: CONTAINER_PATH
value: '/var/lib/docker/containers'
- name: TOTAL_CHAOS_DURATION
VALUE: '60'