Skip to main content

fs fill

FS fill is a Kubernetes pod-level chaos fault that applies FS stress by filling the pod's ephemeral storage. This fault evicts the application pod if its capacity exceeds the pod's ephemeral storage limit.

FS Fill

Use cases

FS fill:

  • Tests the ephemeral storage limits and ensures that the parameters are sufficient.
  • Determines the resilience of the application to unexpected storage exhaustion.
  • Evaluates the application's resilience to FS stress or replica evictions.
  • Verifies file system performance, and thin-provisioning support.
  • Verifies space reclamation (UNMAP) capabilities on storage.

Permissions required

Below is a sample Kubernetes role that defines the permissions required to execute the fault.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
namespace: hce
name: fs-fill
spec:
definition:
scope: Cluster # Supports "Namespaced" mode too
permissions:
- apiGroups: [""]
resources: ["pods"]
verbs: ["create", "delete", "get", "list", "patch", "deletecollection", "update"]
- apiGroups: [""]
resources: ["events"]
verbs: ["create", "get", "list", "patch", "update"]
- apiGroups: [""]
resources: ["pods/log"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["deployments, statefulsets"]
verbs: ["get", "list"]
- apiGroups: [""]
resources: ["replicasets, daemonsets"]
verbs: ["get", "list"]
- apiGroups: [""]
resources: ["chaosEngines", "chaosExperiments", "chaosResults"]
verbs: ["create", "delete", "get", "list", "patch", "update"]
- apiGroups: ["batch"]
resources: ["jobs"]
verbs: ["create", "delete", "get", "list", "deletecollection"]

Prerequisites

  • Kubernetes > 1.16
  • The application pods should be in the running before and after injecting chaos.

Mandatory tunables

Tunable Description Notes
FILL_SIZE File system space to be filled in bytes. Specify in bytes (without unit), kilobytes (k/K/KB), megabytes (m/M/MB), gigabytes (g/G/GB). Example: 100M, 1G, etc For more information, go to fill size
FILE_PATH Container path to be filled For more information, go to file path

Optional tunables

Tunable Description Notes
TARGET_CONTAINER Name of the container subject to FS fill If it is not provided, the first container in the target pod will be subject to chaos. For more information, go to kill specific container
LIB_IMAGE Image used to run the chaos command Default: harness/chaos-go-runner:main-latest. For more information, go to image used by the helper pod.
TOTAL_CHAOS_DURATION Duration for which to insert chaos (in seconds). Default: 60 s. For more information, go to duration of the chaos
TARGET_PODS Comma-separated list of application pod names subject to FS fill chaos. If not provided, the fault selects the target pods randomly based on provided appLabels. For more information, go to target specific pods
PODS_AFFECTED_PERC Percentage of total pods to target. Provide numeric values. Default: 0 (corresponds to 1 replica). For more information, go to pod affected percentage
CONTAINER_RUNTIME Container runtime interface for the cluster. Default: containerd. Supports docker, containerd and crio. For more information, go to container runtime
SOCKET_PATH Path to the containerd/crio/docker socket file. Default: /run/containerd/containerd.sock. For more information, go to socket path
RAMP_TIME Period to wait before injecting chaos (in seconds). For example, 30 s. For more information, go to ramp time
SEQUENCE Sequence of chaos execution for multiple target pods. Default: parallel. Supports serial and parallel. For more information, go to sequence of chaos execution

FS fill

  • FILL_SIZE: Size of ephemeral storage to be filled inside the target pod. Specify in bytes (without unit), kilobytes (k/K/KB), megabytes (m/M/MB), gigabytes (g/G/GB). Example: 100M, 1G, etc.
  • FILE_PATH: Container path to be filled. It should be a valid directory path inside the target container.

The following YAML snippet illustrates the use of this environment variable:

apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
name: engine-nginx
spec:
engineState: "active"
annotationCheck: "false"
appinfo:
appns: "default"
applabel: "app=nginx"
appkind: "deployment"
chaosServiceAccount: litmus-admin
experiments:
- name: fs-fill
spec:
components:
env:
# space to be filled in bytes
# or specify the size in K, KB, M, MB, G, GB
- name: FILL_SIZE
value: "500M"
# container path to be filled
- name: FILE_PATH
value: "/tmp"
- name: TOTAL_CHAOS_DURATION
VALUE: "60"

Container runtime and socket path

The CONTAINER_RUNTIME and SOCKET_PATH environment variables set the container runtime and socket file path, respectively.

  • CONTAINER_RUNTIME: It supports docker, containerd, and crio runtimes. The default value is containerd.
  • SOCKET_PATH: It contains path of containerd socket file by default(/run/containerd/containerd.sock). For docker, specify path as /var/run/docker.sock. For crio, specify path as /var/run/crio/crio.sock.

The following YAML snippet illustrates the use of these environment variables:

## provide the container runtime and socket file path
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
name: engine-nginx
spec:
engineState: "active"
annotationCheck: "false"
appinfo:
appns: "default"
applabel: "app=nginx"
appkind: "deployment"
chaosServiceAccount: litmus-admin
experiments:
- name: fs-fill
spec:
components:
env:
# runtime for the container
# supports docker, containerd, crio
- name: CONTAINER_RUNTIME
value: "containerd"
# path of the socket file
- name: SOCKET_PATH
value: "/run/containerd/containerd.sock"
- name: FILL_SIZE
value: "500M"
- name: FILE_PATH
value: "/tmp"