> For the complete documentation index, see [llms.txt](https://developer.harness.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://developer.harness.io/resilience-testing/chaos-engineering/faults/chaos-fault-categories/kubernetes/pod/pod-http-latency.md).

# Pod HTTP latency

Pod HTTP latency is a Kubernetes pod-level chaos fault that adds a configurable delay to HTTP responses served by the target pod for a configurable duration. The delay is applied at the application protocol layer on the chosen service port; non-HTTP traffic and traffic on other ports passes through unaffected. When the fault ends, the delay is removed and HTTP responses return to normal immediately.

Use this fault to test how a service behaves when a downstream HTTP API or internal handler becomes slow: a sluggish dependency, a backed-up worker pool, or a GC pause that adds tens of milliseconds to every response.

{% hint style="info" %}
**RUN YOUR FIRST EXPERIMENT**

If you have not configured the chaos infrastructure yet, go to [Quickstart](/resilience-testing/chaos-engineering/new-to-chaos-engineering/quickstart.md) to install the chaos infrastructure and run an experiment end to end.
{% endhint %}

***

### Use cases <a href="#use-cases" id="use-cases"></a>

Run this fault when you want to answer concrete questions like:

* **Client timeout budgets:** When the dependency takes 2 seconds instead of 50 ms, do callers honor their timeout configuration, or do they hang waiting for a response?
* **Retry storms:** Does the client's retry policy amplify the slowdown into a thundering herd, or does it apply backoff and jitter?
* **Tail-latency SLOs:** Adding 500 ms to a small fraction of requests reproduces realistic tail-latency degradation. Does your p99 alert fire, and does the runbook hold up?
* **Circuit breaker tripping:** Does the upstream client trip its circuit breaker after `n` slow responses, or does it keep sending traffic to the degraded replica?
* **HTTP/2 and gRPC stream behavior:** A long-lived stream multiplexes many calls. Do per-call deadlines fire, or do all multiplexed calls share the same delay?

***

### Prerequisites <a href="#prerequisites" id="prerequisites"></a>

* **Kubernetes version:** 1.21 or later. Go to [What's supported](/resilience-testing/chaos-engineering/new-to-chaos-engineering/whats-supported.md) to confirm distribution support.
* **Target pods are Running:** The application pods you intend to target are in the `Running` state before the fault is launched.
* **Privileged pods allowed:** The cluster lets you schedule privileged pods in the chaos namespace. GKE Autopilot supports this fault but requires the one-time setup in [Chaos on GKE Autopilot](/resilience-testing/chaos-engineering/use-chaos-engineering/gke-autopilot.md); other locked-down distributions may need similar exemptions.
* **Container runtime access:** The chaos pod can reach the container runtime socket on the target node (`/run/containerd/containerd.sock`, `/var/run/docker.sock`, or `/var/run/crio/crio.sock`).
* **HTTP service on a known port:** The target container serves HTTP, HTTPS, or gRPC traffic on a port you can specify with `TARGET_SERVICE_PORT`.
* **Workload selector defined:** The chaos experiment knows the target workload by kind, namespace, and either names or labels.

***

### Supported environments <a href="#supported-environments" id="supported-environments"></a>

| Platform                                 | Support status                                                                                                 |
| ---------------------------------------- | -------------------------------------------------------------------------------------------------------------- |
| Amazon EKS                               | Supported                                                                                                      |
| Azure AKS                                | Supported                                                                                                      |
| Google GKE                               | Supported                                                                                                      |
| Red Hat OpenShift                        | Supported                                                                                                      |
| Rancher                                  | Supported                                                                                                      |
| VMware Tanzu                             | Supported                                                                                                      |
| Self-managed Kubernetes (CNCF-certified) | Supported                                                                                                      |
| GKE Autopilot                            | Supported with [Autopilot setup](/resilience-testing/chaos-engineering/use-chaos-engineering/gke-autopilot.md) |
| EKS Fargate, ACI virtual nodes           | Not supported (no access to container runtime sockets)                                                         |

***

### Permissions required <a href="#permissions-required" id="permissions-required"></a>

The fault runs under the chaos infrastructure's service account.

| Resource (`apiGroup`)                                               | Verbs                                                                    | Why it is needed                                            |
| ------------------------------------------------------------------- | ------------------------------------------------------------------------ | ----------------------------------------------------------- |
| `pods` (`""`)                                                       | `get`, `list`, `create`, `delete`, `deletecollection`, `patch`, `update` | Discover target pods and run the chaos pod on the same node |
| `pods/log` (`""`)                                                   | `get`, `list`, `watch`                                                   | Stream chaos pod logs for status and debugging              |
| `deployments`, `statefulsets`, `replicasets`, `daemonsets` (`apps`) | `get`, `list`                                                            | Resolve the target workload to the pods it owns             |
| `events` (`""`)                                                     | `get`, `list`, `create`, `patch`, `update`                               | Record fault progress as Kubernetes events                  |
| `jobs` (`batch`)                                                    | `get`, `list`, `create`, `delete`, `deletecollection`                    | Run the chaos job that drives the fault                     |

The default Harness chaos infrastructure service account already includes these permissions.

***

### Fault tunables <a href="#fault-tunables" id="fault-tunables"></a>

Configure the following fault parameters when you add Pod HTTP latency to an experiment in Chaos Studio. Defaults are shown for reference.

**Chaos parameters**

| Tunable                | Description                                                                                                      | Default |
| ---------------------- | ---------------------------------------------------------------------------------------------------------------- | ------- |
| `LATENCY`              | Delay to add to each HTTP response, in milliseconds.                                                             | `2000`  |
| `TARGET_SERVICE_PORT`  | Port the target container listens on for HTTP traffic.                                                           | `80`    |
| `TOXICITY`             | Percentage of intercepted requests to delay, between `0` and `100`. `100` delays every request; `0` delays none. | `100`   |
| `TOTAL_CHAOS_DURATION` | Duration of the fault in seconds.                                                                                | `60`    |

**Proxy and interface**

| Tunable             | Description                                                                                                                                       | Default |
| ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | ------- |
| `PROXY_PORT`        | Port the chaos proxy listens on inside the container's network namespace. Must not conflict with any port already in use on the target container. | `20000` |
| `NETWORK_INTERFACE` | Network interface inside the target container's namespace. Almost always `eth0` for standard CNI plugins.                                         | `eth0`  |

**Targeting**

| Tunable                   | Description                                                                                                          | Default    |
| ------------------------- | -------------------------------------------------------------------------------------------------------------------- | ---------- |
| `TARGET_PODS`             | Comma-separated list of pod names to target. Empty selects from the workload's pods using `POD_AFFECTED_PERCENTAGE`. | `""`       |
| `TARGET_CONTAINER`        | Container in the pod whose network namespace to enter. Empty targets the first container in the pod spec.            | `""`       |
| `NODE_LABEL`              | Label selector to filter target pods by the node they run on. Empty disables node-based filtering.                   | `""`       |
| `POD_AFFECTED_PERCENTAGE` | Percentage of the workload's pods to target. `0` means one pod.                                                      | `0`        |
| `SEQUENCE`                | When multiple pods are targeted, inject `parallel` (all at once) or `serial` (one after another).                    | `parallel` |

**Runtime and helper**

| Tunable             | Description                                                                                                                                                                                                      | Default                           |
| ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------- |
| `CONTAINER_RUNTIME` | Container runtime on the target nodes. One of `containerd`, `docker`, `crio`.                                                                                                                                    | `containerd`                      |
| `SOCKET_PATH`       | Path to the container runtime socket on the target node. Set to match `CONTAINER_RUNTIME`.                                                                                                                       | `/run/containerd/containerd.sock` |
| `RAMP_TIME`         | Wait period in seconds before and after the fault. Go to [ramp time](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/common-tunables-for-all-faults.md#ramp-time) to read how it is applied. | `0`                               |

Tunables that apply to every chaos fault are documented in [common tunables for all faults](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/common-tunables-for-all-faults.md).

{% hint style="info" %}
**USE `TOXICITY` TO MODEL PARTIAL FAILURES**

Setting `TOXICITY=10` slows roughly 10% of requests instead of all of them. This is closer to real-world partial degradation and exercises clients' tail-latency handling, not just hard timeouts.
{% endhint %}

#### Configure for your container runtime <a href="#configure-for-your-container-runtime" id="configure-for-your-container-runtime"></a>

Set `CONTAINER_RUNTIME` and `SOCKET_PATH` to match the runtime on the target node:

| `CONTAINER_RUNTIME`    | `SOCKET_PATH`                     |
| ---------------------- | --------------------------------- |
| `containerd` (default) | `/run/containerd/containerd.sock` |
| `docker`               | `/var/run/docker.sock`            |
| `crio`                 | `/var/run/crio/crio.sock`         |

***

### Fault execution in brief <a href="#fault-execution-in-brief" id="fault-execution-in-brief"></a>

Intercepts HTTP traffic on `TARGET_SERVICE_PORT` inside the container's network namespace and adds `LATENCY` milliseconds to each response, optionally limited to a configurable percentage of requests so other traffic is unaffected.

***

### Expected behavior during fault execution <a href="#expected-behavior-during-fault-execution" id="expected-behavior-during-fault-execution"></a>

* HTTP and HTTPS responses on `TARGET_SERVICE_PORT` are delayed by `LATENCY` milliseconds before being returned to the client. gRPC unary calls (built on HTTP/2) are delayed the same way.
* Only requests selected by `TOXICITY` are delayed; the rest pass through with normal latency.
* Traffic on other ports of the same container is not affected. Non-HTTP TCP traffic on the same port is passed through but no chaos is applied.
* Long-lived HTTP/2 or gRPC streams that were established before the fault began continue to serve frames; new requests through those streams are delayed.
* Clients with timeouts shorter than `LATENCY` see request timeouts; clients with longer timeouts succeed but with elevated p99.
* Service-mesh outlier detection that triggers on slow responses may eject the pod from upstream load-balancing pools.

{% hint style="info" %}
**WHEN THE FAULT ENDS**

After `TOTAL_CHAOS_DURATION`, the proxy is torn down and HTTP responses return to baseline latency within a couple of seconds. In-flight requests already buffered for delay are released as soon as cleanup runs.
{% endhint %}

#### Signals to watch <a href="#signals-to-watch" id="signals-to-watch"></a>

Attach [resilience probes](/resilience-testing/chaos-engineering/use-chaos-engineering/probes.md) to assert each layer:

* **Tail latency:** Use a [Prometheus probe](/resilience-testing/chaos-engineering/use-chaos-engineering/probes/apm-probes.md) on the service's p99 HTTP latency metric to confirm the increase matches `LATENCY` scaled by `TOXICITY`.
* **Client error rate:** Use an [HTTP probe](/resilience-testing/chaos-engineering/use-chaos-engineering/probes/http-probe.md) against the calling service to detect timeout-induced 5xx responses.
* **Pod readiness:** Use a [Kubernetes probe](/resilience-testing/chaos-testing/probes/k8s-probe.md) to fail when the target pod oscillates `NotReady` because its own readiness probe times out at the new latency.

***

### Verify the fault execution effect <a href="#verify-the-fault-execution-effect" id="verify-the-fault-execution-effect"></a>

While the experiment is running, measure HTTP response time and confirm the increase:

1. **Time a request to the target service from another pod.**

   ```bash
   kubectl run -n <namespace> tester --image=curlimages/curl --rm -it -- \
     curl -w "time=%{time_total}\n" -o /dev/null -s \
       http://<target-pod-ip>:<TARGET_SERVICE_PORT>/<known-path>
   ```

   `time_total` should reflect the added delay on at least `TOXICITY` percent of repeated calls.
2. **Confirm the proxy is intercepting the right port.**

   ```bash
   kubectl exec -n <namespace> <target-pod> -- ss -tlnp
   ```

   The proxy listens on `PROXY_PORT` and reroutes the configured `TARGET_SERVICE_PORT` through it for the fault's duration.

***

### Recovery and cleanup <a href="#recovery-and-cleanup" id="recovery-and-cleanup"></a>

* **End of duration:** The proxy is removed automatically and HTTP latency returns to baseline.
* **Abort the experiment:** Stopping the experiment from Chaos Studio triggers the same cleanup path.
* **Failed cleanup:** If automated cleanup did not complete, restart the target pod to reset its network state.

***

### Limitations <a href="#limitations" id="limitations"></a>

* **Serverless Kubernetes (EKS Fargate, ACI virtual nodes):** These platforms do not expose container runtime sockets and reject the privileged access the fault needs. GKE Autopilot is supported once the one-time setup in [Chaos on GKE Autopilot](/resilience-testing/chaos-engineering/use-chaos-engineering/gke-autopilot.md) is in place.
* **Windows containers:** This fault is supported on Linux pods only.
* **TLS-terminated traffic on the same container:** If the target container serves HTTPS and the chaos proxy does not have the matching certificate, the proxy cannot inspect or delay encrypted bodies. For HTTPS scenarios with a custom CA, use [Pod API latency](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/kubernetes/pod/pod-api-latency.md), which supports the TLS certificate inputs needed to terminate HTTPS at the proxy.
* **Long-lived HTTP/2 connections:** Frames on streams already in flight when the fault starts are not delayed; only new requests routed through the proxy are.
* **Port already bound:** If `PROXY_PORT` collides with a port the target container is already using, the fault fails to start. Pick a port number outside the application's range.

***

### Troubleshooting <a href="#troubleshooting" id="troubleshooting"></a>

<details>

<summary>Pod HTTP latency experiment stays Pending or never starts in Harness Chaos Engineering</summary>

Inspect the chaos pods in the experiment namespace with kubectl describe pod -n . The most common causes are taints on the target node that the chaos pods do not tolerate, insufficient resources, or a PodSecurity admission policy blocking privileged pods. Add the required tolerations to the experiment or run in a namespace with privileged Pod Security level.

</details>

<details>

<summary>No HTTP latency observed during pod-http-latency</summary>

The most common causes are: TARGET\_SERVICE\_PORT does not match the port the application actually listens on (verify with kubectl exec -- ss -tlnp); the traffic you are measuring is HTTPS but the chaos proxy is not terminating TLS (use Pod API latency instead); or TOXICITY is set lower than expected so only a fraction of requests are delayed. Re-run the experiment with TOXICITY=100 and confirm the application is reachable on the configured port.

</details>

<details>

<summary>Connection to container runtime fails for pod-http-latency in Harness Chaos Engineering</summary>

The default SOCKET\_PATH is /run/containerd/containerd.sock. For Docker, set CONTAINER\_RUNTIME=docker and SOCKET\_PATH=/var/run/docker.sock. For CRI-O, set CONTAINER\_RUNTIME=crio and SOCKET\_PATH=/var/run/crio/crio.sock.

</details>

<details>

<summary>Latency persists after pod-http-latency ends</summary>

Automated cleanup did not complete. Restart the target pod to reset its network state. If the issue recurs, capture the chaos pod logs from the experiment namespace before the next run and share them with Harness support.

</details>

***

### Related faults <a href="#related-faults" id="related-faults"></a>

* [Pod HTTP modify body](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/kubernetes/pod/pod-http-modify-body.md): Overwrite the HTTP response body instead of delaying it.
* [Pod HTTP modify header](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/kubernetes/pod/pod-http-modify-header.md): Modify or remove HTTP headers on request or response.
* [Pod HTTP reset peer](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/kubernetes/pod/pod-http-reset-peer.md): Reset HTTP TCP connections instead of slowing them.
* [Pod HTTP status code](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/kubernetes/pod/pod-http-status-code.md): Change the HTTP response status code returned to the client.
* [Pod API latency](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/kubernetes/pod/pod-api-latency.md): Add HTTP-protocol latency with rich path, method, header, source, and destination filters (and HTTPS support via supplied TLS certificates).
* [Pod network latency](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/kubernetes/pod/pod-network-latency.md): Add packet-level delay at the network layer instead of the HTTP layer.
* [Common pod fault tunables](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/kubernetes/pod/common-tunables-for-pod-faults.md): Shared environment variables for selecting target pods and workloads.