Skip to main content

Pod IO error

The pod IO error simulates an error that can occur during system calls of the files located within the mounted volume of the pod. When triggered, it causes the call to fail and return an error, potentially disrupting critical processes that rely on accurate file operations.

Pod IO Error

Use cases

Pod IO error:

  • Simulates the error to test whether the application is able to recover gracefully from a failure in that service.
  • Simulates errors into file system calls during a load test to see how the application responds, and whether it is able to handle the increased traffic,
  • Simulates errors to test how your application responds to unexpected failures so you can plan accordingly.
  • Simulates errors into file system call to test whether the application is vulnerable to attacks that exploit file system vulnerabilities.
warning
  • Due to the large blast radius of this fault, we recommend you do not execute it in the production environment.
  • Through the fault execution, the application pod can potentially fail to perform successful IO writes if the write system call is being targeted. This can cause any data produced in this duration to be lost.
  • Any data produced before the execution of the fault is not harmed as a result of its execution.

Permissions required

Below is a sample Kubernetes role that defines the permissions required to execute the fault.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
namespace: hce
name: pod-io-error
spec:
definition:
scope: Cluster # Supports "Namespaced" mode too
permissions:
- apiGroups: [""]
resources: ["pods"]
verbs: ["create", "delete", "get", "list", "patch", "deletecollection", "update"]
- apiGroups: [""]
resources: ["events"]
verbs: ["create", "get", "list", "patch", "update"]
- apiGroups: [""]
resources: ["pods/log"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["deployments, statefulsets"]
verbs: ["get", "list"]
- apiGroups: [""]
resources: ["replicasets, daemonsets"]
verbs: ["get", "list"]
- apiGroups: [""]
resources: ["chaosEngines", "chaosExperiments", "chaosResults"]
verbs: ["create", "delete", "get", "list", "patch", "update"]
- apiGroups: ["batch"]
resources: ["jobs"]
verbs: ["create", "delete", "get", "list", "deletecollection"]

Prerequisites

  • Kubernetes > 1.16
  • The application pods should be in the running state before and after injecting chaos.

Mandatory tunables

Tunable Description Notes
ERROR_CODE Specify the error code to be injected in file system calls For more information, go to error code.
MOUNT_PATH The absolute mount path of the volume mounted to the target pod For more information, go to mount path.

Optional tunables

Tunable Description Notes
TARGET_CONTAINER Name of the container subject to IO error If the value is not provided, the fault injects chaos on the first container of the pod. For more information, go to target specific container.
FILE_PATH The path for injecting faults can be specified as either a single file or a wildcard. If not provided, it will target all the files present inside the mount path For more information, go to file path.
PERCENTAGE The likelihood of failure per operation, expressed as a percentage For more information, go to percentage.
METHOD_TYPES This contains the file system call or methods. For more information, go to method types.
CONTAINER_RUNTIME Container runtime interface for the cluster Default: containerd. Supports docker, containerd and crio. For more information, go to container runtime .
SOCKET_PATH Path of the containerd or crio or docker socket file Defaults to /run/containerd/containerd.sock. For more information, go to socket path.
TOTAL_CHAOS_DURATION Duration to inject insert chaos (in seconds) Default: 60s.
NODE_LABEL Node label used to filter the target node if TARGET_NODE environment variable is not set. It is mutually exclusive with the TARGET_NODE environment variable. If both are provided, the fault uses TARGET_NODE. For more information, go to node label.
TARGET_PODS Comma-separated list of application pod names subject to pod IO error If not provided, the fault selects target pods randomly based on provided appLabels. For more information go to target specific pods.
LIB_IMAGE Image used to inject chaos. Default: harness/chaos-go-runner:main-latest. For more information, go to image used by the helper pod.
PODS_AFFECTED_PERC Percentage of total pods to target. Provide numeric values. Default: 0 (corresponds to 1 replica). For more information, go to pod affected percentage.
RAMP_TIME Period to wait before and after injecting chaos (in seconds) For example, 30s. For more information, go to ramp time.
SEQUENCE Sequence of chaos execution for multiple target pods Default: parallel. Supports serial and parallel. For more information, go to sequence of chaos execution.

Error code

Error code to be injected in system calls of the files located within the mounted volume of the pod. Tune it by using the ERROR_CODE environment variable. For all supported error codes, refer to the Linux Error Codes

The following YAML snippet illustrates the use of this environment variable:

# it injects io error in the file system calls
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
name: engine-nginx
spec:
engineState: "active"
annotationCheck: "false"
appinfo:
appns: "default"
applabel: "app=nginx"
appkind: "deployment"
chaosServiceAccount: litmus-admin
experiments:
- name: pod-io-error
spec:
components:
env:
# error code
- name: ERROR_CODE
value: '5'
- name: MOUNT_PATH
value: '/etc/config'
- name: TOTAL_CHAOS_DURATION
value: '60'

Mount path

Mount path of the volume mounted to the target application. Tune it by using the MOUNT_PATH environment variable.

The following YAML snippet illustrates the use of this environment variable:

# it injects io error in the file system calls
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
name: engine-nginx
spec:
engineState: "active"
annotationCheck: "false"
appinfo:
appns: "default"
applabel: "app=nginx"
appkind: "deployment"
chaosServiceAccount: litmus-admin
experiments:
- name: pod-io-error
spec:
components:
env:
# mount path of the volume
- name: MOUNT_PATH
value: '/etc/config'
- name: ERROR_CODE
value: '5'
- name: TOTAL_CHAOS_DURATION
value: '60'

Advanced fault tunables

  • FILE_PATH: The path for injecting faults can be specified as either a single file or a wildcard. By default it targets all the files present inside the mount path.
  • PERCENTAGE: The likelihood of failure per operation, expressed as a percentage. Default is 100%.
  • METHOD_TYPES: This contains the file system call or methods. By default it targets all the methods. For all supported methods, refer to the Methods

The following YAML snippet illustrates the use of this environment variable:

# it injects io error in the file system calls
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
name: engine-nginx
spec:
engineState: "active"
annotationCheck: "false"
appinfo:
appns: "default"
applabel: "app=nginx"
appkind: "deployment"
chaosServiceAccount: litmus-admin
experiments:
- name: pod-io-error
spec:
components:
env:
# filter the target files
- name: FILE_PATH
value: '/etc/config/file.txt'
# percentage of i/o calls
- name: PERCENTAGE
value: '50'
# names of the io methods
- name: METHOD_TYPES
value: '["read","write"]'
- name: ERROR_CODE
value: '5'
- name: MOUNT_PATH
value: '/etc/config'
- name: TOTAL_CHAOS_DURATION
value: '60'

Container runtime and socket path

The CONTAINER_RUNTIME and SOCKET_PATH are environment variables to set the container runtime and socket file path, respectively.

  • CONTAINER_RUNTIME: This supports docker, containerd, and crio runtimes. The default value is containerd.
  • SOCKET_PATH: This contains the path of containerd socket file by default (/run/containerd/containerd.sock). For docker, specify the path as /var/run/docker.sock. For crio, specify the path as /var/run/crio/crio.sock.

The following YAML snippet illustrates the use of these environment variables:

## provide the container runtime and socket file path
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
name: engine-nginx
spec:
engineState: "active"
annotationCheck: "false"
appinfo:
appns: "default"
applabel: "app=nginx"
appkind: "deployment"
chaosServiceAccount: litmus-admin
experiments:
- name: pod-io-error
spec:
components:
env:
# runtime for the container
# supports docker, containerd, crio
- name: CONTAINER_RUNTIME
value: 'containerd'
# path of the socket file
- name: SOCKET_PATH
value: '/run/containerd/containerd.sock'
- name: ERROR_CODE
value: '5'
- name: MOUNT_PATH
value: '/etc/config'
- name: TOTAL_CHAOS_DURATION
VALUE: '60'