> For the complete documentation index, see [llms.txt](https://developer.harness.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://developer.harness.io/resilience-testing/chaos-engineering/faults/chaos-fault-categories/kubernetes/pod/pod-io-latency.md).

# Pod IO latency

Pod IO latency is a Kubernetes pod-level chaos fault that adds a configurable delay to filesystem syscalls on a target container's mounted volume for a configurable duration. The scope is the container's mount namespace, so other pods on the same node and unrelated paths see normal IO performance. When the fault ends, syscalls return to their baseline latency immediately.

Use this fault to test how a service behaves when reads or writes against a backing volume suddenly become slow: a degraded CSI driver, an EBS volume hitting burst-credit exhaustion, a network-attached volume on a saturated link, or a noisy-neighbor disk.

{% hint style="warning" %}

* Due to the large blast radius of this fault, we recommend you do not execute it in the production environment.
* Through the fault execution, the application pod can potentially fail to perform successful IO writes if the `write` system call is being targeted. This can cause any data produced in this duration to be lost.
* Any data produced before the execution of the fault is not harmed as a result of its execution.
  {% endhint %}

{% hint style="info" %}
**RUN YOUR FIRST EXPERIMENT**

If you have not configured the chaos infrastructure yet, go to [Quickstart](/resilience-testing/chaos-engineering/new-to-chaos-engineering/quickstart.md) to install the chaos infrastructure and run an experiment end to end.
{% endhint %}

***

### Use cases <a href="#use-cases" id="use-cases"></a>

Run this fault when you want to answer concrete questions like:

* **Slow disk simulation:** When a database `fsync` takes 5 seconds instead of 5 ms, do transactions block, time out, or retry correctly?
* **Log-write back-pressure:** If log writes get slow, does the application back off or does the in-memory buffer grow without bound until OOM?
* **Liveness probe robustness:** Does a probe that reads a status file fail when the read is slow, oscillating the pod in and out of `Ready`?
* **Tail-latency budgets:** A p99 of 5 seconds on disk reads, does the service still meet its SLO or do upstream timeouts cascade?
* **Volume failover validation:** For workloads with read replicas backed by separate volumes, does traffic shift away from the slow replica?

***

### Prerequisites <a href="#prerequisites" id="prerequisites"></a>

* **Kubernetes version:** 1.21 or later. Go to [What's supported](/resilience-testing/chaos-engineering/new-to-chaos-engineering/whats-supported.md) to confirm distribution support.
* **Target pod is Running:** The pod you intend to target is in the `Running` state with a mounted volume.
* **Privileged pods allowed:** The cluster lets you schedule privileged pods in the chaos namespace. GKE Autopilot supports this fault but requires the one-time setup in [Chaos on GKE Autopilot](/resilience-testing/chaos-engineering/use-chaos-engineering/gke-autopilot.md); other locked-down distributions may need similar exemptions.
* **Container runtime access:** The chaos pod can reach the container runtime socket on the target node (`/run/containerd/containerd.sock`, `/var/run/docker.sock`, or `/var/run/crio/crio.sock`).
* **Mounted volume:** The target container has at least one writable mounted volume reachable at `MOUNT_PATH`.
* **Workload selector defined:** The chaos experiment knows the target workload by kind, namespace, and either names or labels.

***

### Supported environments <a href="#supported-environments" id="supported-environments"></a>

| Platform                                 | Support status                                                                                                 |
| ---------------------------------------- | -------------------------------------------------------------------------------------------------------------- |
| Amazon EKS                               | Supported                                                                                                      |
| Azure AKS                                | Supported                                                                                                      |
| Google GKE                               | Supported                                                                                                      |
| Red Hat OpenShift                        | Supported                                                                                                      |
| Rancher                                  | Supported                                                                                                      |
| VMware Tanzu                             | Supported                                                                                                      |
| Self-managed Kubernetes (CNCF-certified) | Supported                                                                                                      |
| GKE Autopilot                            | Supported with [Autopilot setup](/resilience-testing/chaos-engineering/use-chaos-engineering/gke-autopilot.md) |
| EKS Fargate, ACI virtual nodes           | Not supported (no access to container runtime sockets)                                                         |

***

### Permissions required <a href="#permissions-required" id="permissions-required"></a>

The fault runs under the chaos infrastructure's service account.

| Resource (`apiGroup`)                                               | Verbs                                                                    | Why it is needed                                            |
| ------------------------------------------------------------------- | ------------------------------------------------------------------------ | ----------------------------------------------------------- |
| `pods` (`""`)                                                       | `get`, `list`, `create`, `delete`, `deletecollection`, `patch`, `update` | Discover target pods and run the chaos pod on the same node |
| `pods/log` (`""`)                                                   | `get`, `list`, `watch`                                                   | Stream chaos pod logs for status and debugging              |
| `deployments`, `statefulsets`, `replicasets`, `daemonsets` (`apps`) | `get`, `list`                                                            | Resolve the target workload to the pods it owns             |
| `events` (`""`)                                                     | `get`, `list`, `create`, `patch`, `update`                               | Record fault progress as Kubernetes events                  |
| `jobs` (`batch`)                                                    | `get`, `list`, `create`, `delete`, `deletecollection`                    | Run the chaos job that drives the fault                     |

The default Harness chaos infrastructure service account already includes these permissions.

***

### Fault tunables <a href="#fault-tunables" id="fault-tunables"></a>

Configure the following fault parameters when you add Pod IO latency to an experiment in Chaos Studio. Defaults are shown for reference.

**Chaos parameters**

| Tunable                | Description                                                                                     | Default |
| ---------------------- | ----------------------------------------------------------------------------------------------- | ------- |
| `LATENCY`              | Delay to add to each matched syscall. Accepts Go duration strings such as `200ms`, `5s`, `1m`.  | `"5s"`  |
| `PERCENTAGE`           | Percentage of matching IO requests to delay, between `0` and `100`. `100` delays every request. | `100`   |
| `TOTAL_CHAOS_DURATION` | Duration of the fault in seconds.                                                               | `60`    |

**Filters**

| Tunable        | Description                                                                                               | Default |
| -------------- | --------------------------------------------------------------------------------------------------------- | ------- |
| `MOUNT_PATH`   | Volume mount path inside the container to scope the fault. Empty applies to all mounts.                   | `""`    |
| `FILE_PATH`    | File path or glob beneath `MOUNT_PATH` to scope the fault. Empty applies to all files under `MOUNT_PATH`. | `""`    |
| `METHOD_TYPES` | Comma-separated syscall types to delay: `read`, `write`, `open`. Empty applies to all three.              | `""`    |

**Targeting**

| Tunable                   | Description                                                                                                          | Default    |
| ------------------------- | -------------------------------------------------------------------------------------------------------------------- | ---------- |
| `TARGET_PODS`             | Comma-separated list of pod names to target. Empty selects from the workload's pods using `POD_AFFECTED_PERCENTAGE`. | `""`       |
| `TARGET_CONTAINER`        | Container in the pod whose filesystem to slow. Empty targets the first container in the pod spec.                    | `""`       |
| `NODE_LABEL`              | Label selector to filter target pods by the node they run on. Empty disables node-based filtering.                   | `""`       |
| `POD_AFFECTED_PERCENTAGE` | Percentage of the workload's pods to target. `0` means one pod.                                                      | `0`        |
| `SEQUENCE`                | When multiple pods are targeted, inject `parallel` (all at once) or `serial` (one after another).                    | `parallel` |

**Runtime and helper**

| Tunable             | Description                                                                                                                                                                                                      | Default                           |
| ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------- |
| `CONTAINER_RUNTIME` | Container runtime on the target nodes. One of `containerd`, `docker`, `crio`.                                                                                                                                    | `containerd`                      |
| `SOCKET_PATH`       | Path to the container runtime socket on the target node. Set to match `CONTAINER_RUNTIME`.                                                                                                                       | `/run/containerd/containerd.sock` |
| `RAMP_TIME`         | Wait period in seconds before and after the fault. Go to [ramp time](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/common-tunables-for-all-faults.md#ramp-time) to read how it is applied. | `0`                               |

Common pod selection tunables (`TARGET_WORKLOAD_KIND`, `TARGET_WORKLOAD_NAMESPACE`, `TARGET_WORKLOAD_NAMES`, `TARGET_WORKLOAD_LABELS`) are documented in [common pod fault tunables](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/kubernetes/pod/common-tunables-for-pod-faults.md). Tunables that apply to every fault are documented in [common tunables for all faults](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/common-tunables-for-all-faults.md).

{% hint style="info" %}
**SCOPE THE DELAY NARROWLY**

Combine `MOUNT_PATH`, `FILE_PATH`, and `METHOD_TYPES` to slow only the path under test. Slowing every read in the container often delays library loads and probes in ways that are difficult to interpret.
{% endhint %}

#### Configure for your container runtime <a href="#configure-for-your-container-runtime" id="configure-for-your-container-runtime"></a>

Set `CONTAINER_RUNTIME` and `SOCKET_PATH` to match the runtime on the target node:

| `CONTAINER_RUNTIME`    | `SOCKET_PATH`                     |
| ---------------------- | --------------------------------- |
| `containerd` (default) | `/run/containerd/containerd.sock` |
| `docker`               | `/var/run/docker.sock`            |
| `crio`                 | `/var/run/crio/crio.sock`         |

***

### Fault execution in brief <a href="#fault-execution-in-brief" id="fault-execution-in-brief"></a>

Intercepts filesystem syscalls (`read`, `write`, `open`) inside the target container's mount namespace and delays each matched call by `LATENCY`, optionally limited to a configurable percentage so other IO is unaffected.

***

### Expected behavior during fault execution <a href="#expected-behavior-during-fault-execution" id="expected-behavior-during-fault-execution"></a>

* Matched syscalls block for `LATENCY` before returning. The kernel reports normal success once the delay elapses; no errors are injected.
* Wall-clock metrics on the application (request latency, transaction time, queue depth) rise to reflect the added delay.
* TCP-backed services that hold a connection across the slow syscall may see clients time out before the syscall completes.
* Concurrency-bound applications (thread pools, semaphore-guarded code paths) saturate quickly as more requests wait on slow IO.
* Health probes that touch the filesystem may exceed their `timeoutSeconds` and mark the pod `NotReady`.

{% hint style="info" %}
**WHEN THE FAULT ENDS**

Syscalls return to baseline latency immediately. Threads already waiting on a delayed syscall complete on schedule and return to the pool.
{% endhint %}

#### Signals to watch <a href="#signals-to-watch" id="signals-to-watch"></a>

Attach [resilience probes](/resilience-testing/chaos-engineering/use-chaos-engineering/probes.md) to assert each layer:

* **Application latency:** Use a [Prometheus probe](/resilience-testing/chaos-engineering/use-chaos-engineering/probes/apm-probes.md) on p99 request latency to confirm the rise matches the injected delay.
* **Filesystem-bound endpoints:** Use an [HTTP probe](/resilience-testing/chaos-engineering/use-chaos-engineering/probes/http-probe.md) on an endpoint backed by the slow volume to detect client timeouts.
* **Pod readiness:** Use a [Kubernetes probe](/resilience-testing/chaos-testing/probes/k8s-probe.md) to fail when the target pod oscillates `NotReady`.

***

### Verify the fault execution effect <a href="#verify-the-fault-execution-effect" id="verify-the-fault-execution-effect"></a>

While the experiment is running, confirm the syscalls are slower:

1. **Time a matched syscall from inside the container.**

   ```bash
   kubectl exec -n <namespace> <target-pod> -c <target-container> -- \
     sh -c "time cat <MOUNT_PATH>/<FILE_PATH>"
   ```

   Wall-clock time should rise by approximately `LATENCY`.
2. **Confirm application-level impact.**

   Watch the application's p99 latency metric or an HTTP probe against an endpoint that exercises the slow path. The increase should track `LATENCY`.

***

### Recovery and cleanup <a href="#recovery-and-cleanup" id="recovery-and-cleanup"></a>

* **End of duration:** Syscalls return to baseline latency automatically.
* **Abort the experiment:** Stopping the experiment from Chaos Studio triggers the same cleanup path.
* **Stuck threads:** If the application is wedged because of saturated thread pools, restart the target pod to clear in-flight blocked work.

***

### Limitations <a href="#limitations" id="limitations"></a>

* **Serverless Kubernetes (EKS Fargate, ACI virtual nodes):** These platforms do not expose container runtime sockets and reject the privileged access the fault needs. GKE Autopilot is supported once the one-time setup in [Chaos on GKE Autopilot](/resilience-testing/chaos-engineering/use-chaos-engineering/gke-autopilot.md) is in place.
* **Windows containers:** This fault is supported on Linux pods only.
* **Read-only volumes:** `ConfigMap`, `Secret`, and downward API volumes may not respond to the delay.
* **Async IO and `mmap`:** Delay applies on syscall boundaries; data already in the page cache or mapped before the fault began may bypass the delay.

***

### Troubleshooting <a href="#troubleshooting" id="troubleshooting"></a>

<details>

<summary>Pod IO latency experiment stays Pending or never starts in Harness Chaos Engineering</summary>

Inspect the chaos pods in the experiment namespace with kubectl describe pod -n . The most common causes are taints on the target node that the chaos pods do not tolerate, insufficient resources, or a PodSecurity admission policy blocking privileged pods. Add the required tolerations to the experiment or run in a namespace with privileged Pod Security level.

</details>

<details>

<summary>No delay observed during pod-io-latency</summary>

The most common causes are: MOUNT\_PATH does not match a real mount inside the container (verify with kubectl exec -- mount); FILE\_PATH or METHOD\_TYPES filters are too narrow; or the application reads through the page cache and never hits the slow syscall. Re-run with PERCENTAGE=100 and empty filters, time a fresh cat or dd against the mount, and confirm the wall-clock increase.

</details>

<details>

<summary>Connection to container runtime fails for pod-io-latency in Harness Chaos Engineering</summary>

The default SOCKET\_PATH is /run/containerd/containerd.sock. For Docker, set CONTAINER\_RUNTIME=docker and SOCKET\_PATH=/var/run/docker.sock. For CRI-O, set CONTAINER\_RUNTIME=crio and SOCKET\_PATH=/var/run/crio/crio.sock.

</details>

<details>

<summary>Threads or connections wedged after pod-io-latency ends</summary>

Some applications cannot drain saturated thread or connection pools after slow IO and need a restart. Restart the target pod to recover, and if the issue recurs, lower LATENCY or PERCENTAGE for subsequent runs.

</details>

***

### Related faults <a href="#related-faults" id="related-faults"></a>

* [Pod IO error](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/kubernetes/pod/pod-io-error.md): Fail filesystem syscalls with a configurable error code instead of slowing them.
* [Pod IO mistake](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/kubernetes/pod/pod-io-mistake.md): Corrupt filesystem data with wrong values.
* [Pod IO stress](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/kubernetes/pod/pod-io-stress.md): Generate sustained read and write load to create contention.
* [Disk fill](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/kubernetes/pod/disk-fill.md): Fill the container's ephemeral storage to trigger eviction.
* [Common pod fault tunables](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/kubernetes/pod/common-tunables-for-pod-faults.md): Shared environment variables for selecting target pods and workloads.