Disk Fill

Introduction

• It causes Disk Stress by filling up the ephemeral storage of the pod on any given node.
• It causes the application pod to get evicted if the capacity filled exceeds the pod's ephemeral storage limit.
• It tests the Ephemeral Storage Limits, to ensure those parameters are sufficient.
• It tests the application's resiliency to disk stress/replica evictions.

Fault execution flow chart

Uses

View the uses of the fault

Coming soon.

Prerequisites

info

• Ensure that Kubernetes Version > 1.16.
• Appropriate Ephemeral Storage Requests and Limits should be set for the application before running the fault. An example specification is shown below:

  apiVersion: v1
  kind: Pod
  metadata:
    name: frontend
  spec:
    containers:
    - name: db
      image: mysql
      env:
      - name: MYSQL_ROOT_PASSWORD
        value: "password"
      resources:
        requests:
          ephemeral-storage: "2Gi"
        limits:
          ephemeral-storage: "4Gi"
    - name: wp
      image: wordpress
      resources:
        requests:
          ephemeral-storage: "2Gi"
        limits:
          ephemeral-storage: "4Gi"

Default Validations

note

The application pods should be in running state before and after chaos injection.

Fault Tunables

Check the Fault Tunables

Optional Fields

Variables	Description	Notes
FILL_PERCENTAGE	Percentage to fill the Ephemeral storage limit	Can be set to more than 100 also, to force evict the pod. The ephemeral-storage limits must be set in targeted pod to use this ENV.
EPHEMERAL_STORAGE_MEBIBYTES	Ephemeral storage which need to fill (unit: MiBi)	It is mutually exclusive with the FILL_PERCENTAGE ENV. If both are provided then it will use the FILL_PERCENTAGE

Optional Fields

Variables	Description	Notes
TARGET_CONTAINER	Name of container which is subjected to disk-fill	If not provided, the first container in the targeted pod will be subject to chaos
CONTAINER_PATH	Storage Location of containers	Defaults to '/var/lib/docker/containers'
TOTAL_CHAOS_DURATION	The time duration for chaos insertion (sec)	Defaults to 60s
TARGET_PODS	Comma separated list of application pod name subjected to disk fill chaos	If not provided, it will select target pods randomly based on provided appLabels
DATA_BLOCK_SIZE	It contains data block size used to fill the disk(in KB)	Defaults to 256, it supports unit as KB only
PODS_AFFECTED_PERC	The Percentage of total pods to target	Defaults to 0 (corresponds to 1 replica), provide numeric value only
LIB	The chaos lib used to inject the chaos	Defaults to `litmus` supported litmus only
LIB_IMAGE	The image used to fill the disk	Defaults to litmuschaos/go-runner:latest
RAMP_TIME	Period to wait before injection of chaos in sec	Eg. 30
SEQUENCE	It defines sequence of chaos execution for multiple target pods	Default value: parallel. Supported: serial, parallel

Fault Examples

Common and Pod specific tunables

Refer the common attributes and Pod specific tunable to tune the common tunables for all fault and pod specific tunables.

Disk Fill Percentage

It fills the FILL_PERCENTAGE percentage of the ephemeral-storage limit specified at resource.limits.ephemeral-storage inside the target application.

Use the following example to tune this:

## percentage of ephemeral storage limit specified at `resource.limits.ephemeral-storage` inside target application 
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  annotationCheck: "false"
  appinfo:
    appns: "default"
    applabel: "app=nginx"
    appkind: "deployment"
  chaosServiceAccount: litmus-admin
  experiments:
  - name: disk-fill
    spec:
      components:
        env:
        ## percentage of ephemeral storage limit, which needs to be filled
        - name: FILL_PERCENTAGE
          value: '80' # in percentage
        - name: TOTAL_CHAOS_DURATION
          VALUE: '60'

Disk Fill Mebibytes

It fills the EPHEMERAL_STORAGE_MEBIBYTES MiBi of ephemeral storage of the targeted pod. It is mutually exclusive with the FILL_PERCENTAGE ENV. If FILL_PERCENTAGE ENV is set then it will use the percentage for the fill otherwise, it will fill the ephemeral storage based on EPHEMERAL_STORAGE_MEBIBYTES ENV.

Use the following example to tune this:

# ephemeral storage which needs to fill in will application
# if ephemeral-storage limits is not specified inside target application
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  annotationCheck: "false"
  appinfo:
    appns: "default"
    applabel: "app=nginx"
    appkind: "deployment"
  chaosServiceAccount: litmus-admin
  experiments:
  - name: disk-fill
    spec:
      components:
        env:
        ## ephemeral storage size, which needs to be filled
        - name: EPHEMERAL_STORAGE_MEBIBYTES
          value: '256' #in MiBi
        - name: TOTAL_CHAOS_DURATION
          VALUE: '60'

Data Block Size

It defines the size of the data block used to fill the ephemeral storage of the targeted pod. It can be tuned via DATA_BLOCK_SIZE ENV. Its unit is KB. The default value of DATA_BLOCK_SIZE is 256.

Use the following example to tune this:

# size of the data block used to fill the disk
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  annotationCheck: "false"
  appinfo:
    appns: "default"
    applabel: "app=nginx"
    appkind: "deployment"
  chaosServiceAccount: litmus-admin
  experiments:
  - name: disk-fill
    spec:
      components:
        env:
        ## size of data block used to fill the disk
        - name: DATA_BLOCK_SIZE
          value: '256' #in KB
        - name: TOTAL_CHAOS_DURATION
          VALUE: '60'

Container Path

It defines the storage location of the containers inside the host(node/VM). It can be tuned via CONTAINER_PATH ENV.

Use the following example to tune this:

# path inside node/vm where containers are present 
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  annotationCheck: "false"
  appinfo:
    appns: "default"
    applabel: "app=nginx"
    appkind: "deployment"
  chaosServiceAccount: litmus-admin
  experiments:
  - name: disk-fill
    spec:
      components:
        env:
        # storage location of the containers
        - name: CONTAINER_PATH
          value: '/var/lib/docker/containers'
        - name: TOTAL_CHAOS_DURATION
          VALUE: '60'