Pod IO mistake

Pod IO mistake simulates a scenario where the file system in the mounted volume of the pod reads or writes incorrect values. This fault determines how the application handles data corruption or errors during file operations, ensuring robustness and stability under adverse conditions.

Use cases

Pod IO mistake:

• Verify the application's ability to detect and recover from data corruption caused by incorrect file read/write operations
• Ensure the application can handle and log errors appropriately when encountering unexpected data values.
• Test how the application manages unauthorized or corrupted data access attempts to prevent security breaches.
• Assess the application's performance and stability when operating with corrupted data to identify potential bottlenecks or failure points.

warning

• Due to the large blast radius of this fault, we recommend you do not execute it in the production environment.
• Through the fault execution, the application pod can potentially fail to perform successful IO writes if the write system call is being targeted. This can cause any data produced in this duration to be lost.
• Any data produced before the execution of the fault is not harmed as a result of its execution.

warning

It is recommended to use "mistake" for READ file system calls only. Applying "mistake" to other file system calls can result in unforeseen issues, such as file system damage, corrupted writes and program crashes.

Permissions required

Below is a sample Kubernetes role that defines the permissions required to execute the fault.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: hce
  name: pod-io-mistake
spec:
  definition:
    scope: Cluster # Supports "Namespaced" mode too
permissions:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["create", "delete", "get", "list", "patch", "deletecollection", "update"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["create", "get", "list", "patch", "update"]
  - apiGroups: [""]
    resources: ["pods/log"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["deployments, statefulsets"]
    verbs: ["get", "list"]
  - apiGroups: [""]
    resources: ["replicasets, daemonsets"]
    verbs: ["get", "list"]
  - apiGroups: [""]
    resources: ["chaosEngines", "chaosExperiments", "chaosResults"]
    verbs: ["create", "delete", "get", "list", "patch", "update"]
  - apiGroups: ["batch"]
    resources: ["jobs"]
    verbs: ["create", "delete", "get", "list", "deletecollection"]

Prerequisites

• Kubernetes > 1.16
• The application pods should be in the running state before and after injecting chaos.

Mandatory tunables

Tunable	Description	Notes
SEED	Specify the wrong data to be seed	It supports zero and random values. Defaults to zero. For more information, go to seed.
MAX_TIMES	Maximum number of places to seed the wrong data	Defaults to 1. For more information, go to max times.
MAX_SIZE	Maximum size of the data to be seeded in bytes	Defaults to 8. For more information, go to max size.
MOUNT_PATH	The absolute mount path of the volume mounted to the target pod	For more information, go to mount path.

Optional tunables

Tunable	Description	Notes
TARGET_CONTAINER	Name of the container subject to IO error	If the value is not provided, the fault injects chaos on the first container of the pod. For more information, go to target specific container.
FILE_PATH	The path for injecting faults can be specified as either a single file or a wildcard. If not provided, it will target all the files present inside the mount path	For more information, go to file path.
PERCENTAGE	The likelihood of failure per operation, expressed as a percentage	For more information, go to percentage.
METHOD_TYPES	This contains the file system call or methods.	For more information, go to method types.
CONTAINER_RUNTIME	Container runtime interface for the cluster	Default: containerd. Supports docker, containerd and crio. For more information, go to container runtime .
SOCKET_PATH	Path of the containerd or crio or docker socket file	Defaults to /run/containerd/containerd.sock. For more information, go to socket path.
TOTAL_CHAOS_DURATION	Duration to inject insert chaos (in seconds)	Default: 60s.
NODE_LABEL	Node label used to filter the target node if TARGET_NODE environment variable is not set.	It is mutually exclusive with the TARGET_NODE environment variable. If both are provided, the fault uses TARGET_NODE. For more information, go to node label.
TARGET_PODS	Comma-separated list of application pod names subject to pod IO error	If not provided, the fault selects target pods randomly based on provided appLabels. For more information go to target specific pods.
LIB_IMAGE	Image used to inject chaos.	Default: harness/chaos-go-runner:main-latest. For more information, go to image used by the helper pod.
PODS_AFFECTED_PERC	Percentage of total pods to target. Provide numeric values.	Default: 0 (corresponds to 1 replica). For more information, go to pod affected percentage.
RAMP_TIME	Period to wait before and after injecting chaos (in seconds)	For example, 30s. For more information, go to ramp time.
SEQUENCE	Sequence of chaos execution for multiple target pods	Default: parallel. Supports serial and parallel. For more information, go to sequence of chaos execution.

Mistake

• SEED: Wrong data to be seeded in system calls of the files located within the mounted volume of the pod. It supports zero and random values.
• MAX_TIMES: Maximum number of locations where incorrect data can be seeded at once.
• MAX_SIZE: Maximum size of the data to be seeded, in bytes.

The following YAML snippet illustrates the use of this environment variable:

# it injects io mistake in the file system calls
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  annotationCheck: "false"
  appinfo:
    appns: "default"
    applabel: "app=nginx"
    appkind: "deployment"
  chaosServiceAccount: litmus-admin
  experiments:
  - name: pod-io-mistake
    spec:
      components:
        env:
        # seed the wrong data
        - name: SEED
          value: 'zero'
        # maximum number of places to seed the wrong data
        - name: MAX_TIMES
          value: '1'
        # maximum size of the data to be seeded in bytes
        - name: MAX_SIZE
          value: '8'
        - name: MOUNT_PATH
          value: '/etc/config'
        - name: TOTAL_CHAOS_DURATION
          value: '60'

Mount path

Mount path of the volume mounted to the target application. Tune it by using the MOUNT_PATH environment variable.

The following YAML snippet illustrates the use of this environment variable:

# it injects io mistake in the file system calls
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  annotationCheck: "false"
  appinfo:
    appns: "default"
    applabel: "app=nginx"
    appkind: "deployment"
  chaosServiceAccount: litmus-admin
  experiments:
  - name: pod-io-mistake
    spec:
      components:
        env:
        # mount path of the volume
        - name: MOUNT_PATH
          value: '/etc/config'
        - name: SEED
          value: 'zero'
        - name: TOTAL_CHAOS_DURATION
          value: '60'

Advanced fault tunables

• FILE_PATH: The path for injecting faults can be specified as either a single file or a wildcard. By default, it targets all the files present inside the mount path.
• PERCENTAGE: The likelihood of failure per operation, expressed as a percentage. Default is 100%.
• METHOD_TYPES: This contains the file system call or methods. By default, it targets read and write methods. For all supported methods, refer to the Methods

The following YAML snippet illustrates the use of this environment variable:

# it injects io mistake in the file system calls
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  annotationCheck: "false"
  appinfo:
    appns: "default"
    applabel: "app=nginx"
    appkind: "deployment"
  chaosServiceAccount: litmus-admin
  experiments:
  - name: pod-io-mistake
    spec:
      components:
        env:
        # filter the target files
        - name: FILE_PATH
          value: '/etc/config/file.txt'
        # percentage of i/o calls
        - name: PERCENTAGE
          value: '50'
        # names of the io methods
        - name: METHOD_TYPES
          value: '["read","write"]'
        - name: SEED
          value: 'zero'
        - name: MOUNT_PATH
          value: '/etc/config'
        - name: TOTAL_CHAOS_DURATION
          value: '60'

Container runtime and socket path

The CONTAINER_RUNTIME and SOCKET_PATH are environment variables to set the container runtime and socket file path, respectively.

• CONTAINER_RUNTIME: This supports docker, containerd, and crio runtimes. The default value is containerd.
• SOCKET_PATH: This contains the path of containerd socket file by default (/run/containerd/containerd.sock). For docker, specify the path as /var/run/docker.sock. For crio, specify the path as /var/run/crio/crio.sock.

The following YAML snippet illustrates the use of these environment variables:

## provide the container runtime and socket file path
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  annotationCheck: "false"
  appinfo:
    appns: "default"
    applabel: "app=nginx"
    appkind: "deployment"
  chaosServiceAccount: litmus-admin
  experiments:
  - name: pod-io-mistake
    spec:
      components:
        env:
        # runtime for the container
        # supports docker, containerd, crio
        - name: CONTAINER_RUNTIME
          value: 'containerd'
        # path of the socket file
        - name: SOCKET_PATH
          value: '/run/containerd/containerd.sock'
        - name: SEED
          value: 'zero'
        - name: MOUNT_PATH
          value: '/etc/config'
        - name: TOTAL_CHAOS_DURATION
          VALUE: '60'