Pod IO attribute override

Pod IO attribute override modifies the properties of files located within the mounted volume of the pod. This fault should be used as a sanity test for validating your application's failover capability against unexpected changes in file attributes, such as permissions, ownership, or timestamps.

Use cases

Pod IO attribute override:

• Tests whether your application is able to detect and respond to unexpected changes in file attributes, which can help you ensure that your system remains compliant and secure.
• Tests whether your disaster recovery system is able to restore files and maintain file attributes correctly.
• Tests whether your backup system is able to restore files and maintain file attributes correctly.
• Tests whether your change management process is able to detect and respond to unexpected changes in file attributes, which can help you ensure that your system remains stable and secure.

warning

• Due to the large blast radius of this fault, we recommend you do not execute it in the production environment.
• Through the fault execution, the application pod can potentially fail to perform successful IO writes if the write system call is being targeted. This can cause any data produced in this duration to be lost.
• Any data produced before the execution of the fault is not harmed as a result of its execution.

Permissions required

Below is a sample Kubernetes role that defines the permissions required to execute the fault.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: hce
  name: pod-io-attribute-override
spec:
  definition:
    scope: Cluster # Supports "Namespaced" mode too
permissions:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["create", "delete", "get", "list", "patch", "deletecollection", "update"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["create", "get", "list", "patch", "update"]
  - apiGroups: [""]
    resources: ["pods/log"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["deployments, statefulsets"]
    verbs: ["get", "list"]
  - apiGroups: [""]
    resources: ["replicasets, daemonsets"]
    verbs: ["get", "list"]
  - apiGroups: [""]
    resources: ["chaosEngines", "chaosExperiments", "chaosResults"]
    verbs: ["create", "delete", "get", "list", "patch", "update"]
  - apiGroups: ["batch"]
    resources: ["jobs"]
    verbs: ["create", "delete", "get", "list", "deletecollection"]

Prerequisites

• Kubernetes > 1.16
• The application pods should be in the running state before and after injecting chaos.

Mandatory tunables

Tunable	Description	Notes
ATTRIBUTES	Specify the file attributes and their values that need to be overridden	For more information, go to attributes.
MOUNT_PATH	The absolute mount path of the volume mounted to the target pod	For more information, go to mount path.

Optional tunables

Tunable	Description	Notes
TARGET_CONTAINER	Name of the container subject to IO attribute override	If the value is not provided, the fault injects chaos on the first container of the pod. For more information, go to target specific container.
NODE_LABEL	Node label used to filter the target node if TARGET_NODE environment variable is not set.	It is mutually exclusive with the TARGET_NODE environment variable. If both are provided, the fault uses TARGET_NODE. For more information, go to node label.
FILE_PATH	The path for injecting faults can be specified as either a single file or a wildcard. If not provided, it will target all the files present inside the mount path	For more information, go to file path.
PERCENTAGE	The likelihood of failure per operation, expressed as a percentage	For more information, go to percentage.
METHOD_TYPES	This contains the file system call or methods	For more information, go to method types.
CONTAINER_RUNTIME	Container runtime interface for the cluster	Default: containerd. Supports docker, containerd and crio. For more information, go to container runtime .
SOCKET_PATH	Path of the containerd or crio or docker socket file	Defaults to /run/containerd/containerd.sock. For more information, go to socket path.
TOTAL_CHAOS_DURATION	Duration to inject insert chaos (in seconds)	Default: 60s. For more information, go to duration of the chaos.
LIB_IMAGE	Image used to inject chaos.	Default: harness/chaos-go-runner:main-latest. For more information, go to image used by the helper pod.
TARGET_PODS	Comma-separated list of application pod names subject to pod io attribute override	If not provided, the fault selects target pods randomly based on provided appLabels. For more information, go to target specific pods.
PODS_AFFECTED_PERC	Percentage of total pods to target. Provide numeric values	Default: 0 (corresponds to 1 replica). For more information, go to pod affected percentage.
RAMP_TIME	Period to wait before and after injecting chaos (in seconds)	For example, 30s. For more information, go to ramp time.
SEQUENCE	Sequence of chaos execution for multiple target pods	Default: parallel. Supports serial and parallel. For more information, go to sequence of chaos execution.

Attributes

Modify the properties of files located within the mounted volume of the pod. Tune it by using the ATTRIBUTES environment variable.

The attributes to override is defined below:

Parameter	Type	Description	Example
atime	TimeSpec	Last access time
blocks	int	Number of blocks that the file uses
ctime	TimeSpec	Last status change time
gid	int	Group ID of the owner
ino	int	ino number
kind	string	File type, see fuser::FileType
mtime	TimeSpec	Last modified time
nlink	int	Number of hard links
perm	int	File permissions in decimal	72 (110 in octal) - default value
rdev	int	Device ID
size	int	File size
uid	int	User ID of the owner

The following YAML snippet illustrates the use of this environment variable:

# it overrides the file attributes
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  annotationCheck: "false"
  appinfo:
    appns: "default"
    applabel: "app=nginx"
    appkind: "deployment"
  chaosServiceAccount: litmus-admin
  experiments:
  - name: pod-io-attribute-override
    spec:
      components:
        env:
        # file attributes
        - name: ATTRIBUTES
          value: '{"perm": 72}'
        - name: MOUNT_PATH
          value: '/etc/config'
        - name: TOTAL_CHAOS_DURATION
          value: '60'

Mount path

Mount path of the volume mounted to the target application. Tune it by using the MOUNT_PATH environment variable.

The following YAML snippet illustrates the use of this environment variable:

# it overrides the file attributes
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  annotationCheck: "false"
  appinfo:
    appns: "default"
    applabel: "app=nginx"
    appkind: "deployment"
  chaosServiceAccount: litmus-admin
  experiments:
  - name: pod-io-attribute-override
    spec:
      components:
        env:
        # mount path of the volume
        - name: MOUNT_PATH
          value: '/etc/config'
        - name: ATTRIBUTES
          value: '{"perm": 72}'
        - name: TOTAL_CHAOS_DURATION
          value: '60'

Advanced fault tunables

• FILE_PATH: The path for injecting faults can be specified as either a single file or a wildcard. By default it targets all the files present inside the mount path.
• PERCENTAGE: The likelihood of failure per operation, expressed as a percentage. Default is 100%.
• METHOD_TYPES: This contains the file system call or methods. By default it targets all the methods. For all supported methods, refer to the Methods

The following YAML snippet illustrates the use of this environment variable:

# it overrides the file attributes
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  annotationCheck: "false"
  appinfo:
    appns: "default"
    applabel: "app=nginx"
    appkind: "deployment"
  chaosServiceAccount: litmus-admin
  experiments:
  - name: pod-io-attribute-override
    spec:
      components:
        env:
        # filter the target files
        - name: FILE_PATH
          value: '/etc/config/file.txt'
        # percentage of i/o calls
        - name: PERCENTAGE
          value: '50'
        # names of the io methods
        - name: METHOD_TYPES
          value: '["read","write"]'
        - name: ATTRIBUTES
          value: '{"perm": 72}'
        - name: MOUNT_PATH
          value: '/etc/config'
        - name: TOTAL_CHAOS_DURATION
          value: '60'

Container runtime and socket path

The CONTAINER_RUNTIME and SOCKET_PATH environment variables to set the container runtime and socket file path, respectively.

• CONTAINER_RUNTIME: This supports docker, containerd, and crio runtimes. The default value is containerd.
• SOCKET_PATH: This contains the path of containerd socket file by default(/run/containerd/containerd.sock). For docker, specify the path as /var/run/docker.sock. For crio, specify the path as /var/run/crio/crio.sock.

The following YAML snippet illustrates the use of these environment variables:

## provide the container runtime and socket file path
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  annotationCheck: "false"
  appinfo:
    appns: "default"
    applabel: "app=nginx"
    appkind: "deployment"
  chaosServiceAccount: litmus-admin
  experiments:
  - name: pod-io-attribute-override
    spec:
      components:
        env:
        # runtime for the container
        # supports docker, containerd, crio
        - name: CONTAINER_RUNTIME
          value: 'containerd'
        # path of the socket file
        - name: SOCKET_PATH
          value: '/run/containerd/containerd.sock'
        - name: ATTRIBUTES
          value: '{"perm": 72}'
        - name: MOUNT_PATH
          value: '/etc/config'
        - name: TOTAL_CHAOS_DURATION
          VALUE: '60'