> For the complete documentation index, see [llms.txt](https://developer.harness.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://developer.harness.io/resilience-testing/chaos-engineering/faults/chaos-fault-categories/linux/linux-api-latency.md).

# Linux API latency

Linux API latency is a chaos fault that starts a local proxy on the target Linux machine, redirects traffic to/from `TARGET_SERVICE_PORT` through the proxy on `PROXY_PORT`, and adds `LATENCY` of delay to matching API requests for `DURATION`. `DATA_DIRECTION` chooses whether the delay is applied to the `request`, `response`, or `both`. Filters and HTTPS interception controls mirror [Linux API block](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/linux/linux-api-block.md). The fault runs through the Linux Chaos Infrastructure (LCI) systemd service installed on the target VM.

Use this fault to test how callers behave when an API gets slow: whether they honor their own timeouts, whether retries amplify the delay, whether thread pools exhaust, and whether monitoring detects the elevated tail latency within the alerting SLA.

{% hint style="info" %}
**RUN YOUR FIRST EXPERIMENT**

If you have not installed the Linux Chaos Infrastructure yet, go to [Linux Chaos Infrastructure](/resilience-testing/chaos-engineering/use-chaos-engineering/infrastructure/types/legacy-infra/linux.md) to install the agent and connect the VM to the control plane.
{% endhint %}

***

### Use cases <a href="#use-cases" id="use-cases"></a>

Run this fault when you want to answer concrete questions like:

* **Caller timeouts:** When the API takes `LATENCY` longer than usual, do callers honor their own timeouts cleanly?
* **Thread-pool starvation:** Do upstream worker pools back up when responses get slow?
* **Retry storms:** Do retries amplify the slow traffic, or do exponential backoff and jitter contain it?
* **Monitoring fidelity:** Do alerts on end-to-end p99, queue depth, and timeout counters fire within the alerting SLA?

***

### Prerequisites <a href="#prerequisites" id="prerequisites"></a>

* **Linux Chaos Infrastructure installed:** The `linux-chaos-infrastructure` systemd service is `active` on the target VM and the infrastructure is in `CONNECTED` state. Go to [Linux Chaos Infrastructure](/resilience-testing/chaos-engineering/use-chaos-engineering/infrastructure/types/legacy-infra/linux.md) to install it.
* **Target service port reachable:** `TARGET_SERVICE_PORT` is the port the application listens on (ingress) or the upstream service port (egress).
* **Proxy port free:** `PROXY_PORT` (default `20000`) is not bound by another process on the target VM.
* **HTTPS material if `HTTPS_ENABLED=true`:** Provide a CA bundle in `HTTPS_ROOT_CERT_FILE` and additional certificates in `CUSTOM_CERTIFICATES`.
* **`/tmp` is exec-mountable:** Verify with `findmnt -l | grep noexec | grep /tmp`. If `/tmp` is mounted `noexec`, remount with `sudo mount /tmp -o remount,exec`.

***

### Supported environments <a href="#supported-environments" id="supported-environments"></a>

The fault has been tested on the following Linux distributions. Go to [Linux fault requirements](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/linux/permissions.md) to see the full compatibility matrix.

| Platform                                        | Support status |
| ----------------------------------------------- | -------------- |
| Ubuntu 16+, Debian 10+                          | Supported      |
| CentOS 7+, RHEL 7+, Fedora 30+                  | Supported      |
| openSUSE LEAP 15.4+ / SUSE Linux Enterprise 15+ | Supported      |

***

### Permissions required <a href="#permissions-required" id="permissions-required"></a>

This fault is classified as an **Advanced** Linux fault. It requires the Linux Chaos Infrastructure systemd service to run with the root user and root user group on the target VM so it can install iptables redirects and bind the proxy port. No cloud credentials are needed.

***

### Fault tunables <a href="#fault-tunables" id="fault-tunables"></a>

Configure the following fault parameters when you add Linux API latency to an experiment in Chaos Studio. Defaults are shown for reference.

**Chaos parameters**

| Tunable               | Description                                                                                                                                                                                                      | Default |
| --------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- |
| `DURATION`            | Total duration of the fault. Accepts `[hours]h[minutes]m[seconds]s` format.                                                                                                                                      | `30s`   |
| `LATENCY`             | Latency to inject per matching request (for example, `2s`, `500ms`).                                                                                                                                             | `2s`    |
| `TARGET_SERVICE_PORT` | Port of the application (ingress) or upstream service (egress) to intercept.                                                                                                                                     | `80`    |
| `PROXY_PORT`          | Port on which the local proxy listens.                                                                                                                                                                           | `20000` |
| `NETWORK_INTERFACE`   | Network interface used for ingress redirection.                                                                                                                                                                  | `eth0`  |
| `DATA_DIRECTION`      | Direction of data to delay. Accepts `request`, `response`, or `both`.                                                                                                                                            | `both`  |
| `RAMP_TIME`           | Wait period in seconds before and after the fault. Go to [ramp time](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/common-tunables-for-all-faults.md#ramp-time) to read how it is applied. | `0`     |

**Request filters (combine as needed)**

| Tunable           | Description                                                 | Default |
| ----------------- | ----------------------------------------------------------- | ------- |
| `PATH_FILTER`     | API path glob to match. Leave empty to match all paths.     | `""`    |
| `HEADERS_FILTERS` | HTTP headers filter (comma-separated `Header=value` pairs). | `""`    |
| `METHODS`         | Comma-separated HTTP methods to match.                      | `""`    |

**Direction and source/destination filters**

| Tunable             | Description                                                                                | Default   |
| ------------------- | ------------------------------------------------------------------------------------------ | --------- |
| `SERVICE_DIRECTION` | `ingress` (delay requests arriving at the VM) or `egress` (delay requests leaving the VM). | `ingress` |
| `SOURCE_HOSTS`      | Comma-separated source host names to match (ingress only).                                 | `""`      |
| `SOURCE_IPS`        | Comma-separated source IPs to match (ingress only).                                        | `""`      |
| `DESTINATION_HOSTS` | Comma-separated destination host names to match (egress only).                             | `""`      |
| `DESTINATION_IPS`   | Comma-separated destination IPs to match (egress only).                                    | `""`      |
| `DESTINATION_PORTS` | Comma-separated destination ports for egress traffic.                                      | `""`      |

**HTTPS interception (egress)**

| Tunable                | Description                                                                                           | Default |
| ---------------------- | ----------------------------------------------------------------------------------------------------- | ------- |
| `HTTPS_ENABLED`        | Enable HTTPS interception. Required for matching HTTPS traffic by `PATH_FILTER`, headers, or methods. | `false` |
| `HTTPS_ROOT_CERT_FILE` | Path to the root CA bundle used by the proxy to terminate egress HTTPS.                               | `""`    |
| `CUSTOM_CERTIFICATES`  | Base64-encoded custom certificates added to the proxy trust store.                                    | `""`    |

Tunables that apply to every fault are documented in [common tunables for all faults](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/common-tunables-for-all-faults.md).

***

### Fault execution in brief <a href="#fault-execution-in-brief" id="fault-execution-in-brief"></a>

Adds an iptables redirect on `NETWORK_INTERFACE` that routes `TARGET_SERVICE_PORT` traffic through the local proxy on `PROXY_PORT` for `DURATION`. Matching requests are held by `LATENCY` (in `DATA_DIRECTION`) before being forwarded; non-matching requests pass through unchanged. On exit, the iptables redirect and proxy are removed.

***

### Expected behavior during fault execution <a href="#expected-behavior-during-fault-execution" id="expected-behavior-during-fault-execution"></a>

* Matching API calls take `LATENCY` longer than usual for the duration of the fault.
* Callers see slower responses; tail latency (`p99`) shifts upward by approximately `LATENCY`.
* Callers with shorter timeouts than `LATENCY` see clean timeout errors.
* After the duration ends, the iptables redirect is removed and the proxy stops; traffic resumes its original path with no added latency.

{% hint style="info" %}
**WHEN THE FAULT ENDS**

The chaos pod removes the iptables redirect and stops the local proxy. The next request flows directly with no added latency.
{% endhint %}

#### Signals to watch <a href="#signals-to-watch" id="signals-to-watch"></a>

Attach [resilience probes](/resilience-testing/chaos-engineering/use-chaos-engineering/probes.md) to assert each layer:

* **End-to-end latency:** Use a [Prometheus probe](/resilience-testing/chaos-engineering/use-chaos-engineering/probes/apm-probes.md) on application p95/p99 metrics.
* **Caller timeout counters:** Use a Prometheus probe on caller timeout/connection-error counters.
* **HTTP probe:** Use an [HTTP probe](/resilience-testing/chaos-engineering/use-chaos-engineering/probes/http-probe.md) on a user-visible endpoint that exercises the slowed API.

***

### Verify the fault execution effect <a href="#verify-the-fault-execution-effect" id="verify-the-fault-execution-effect"></a>

1. **Send a matching request from the VM.**

   ```bash
   curl -w '%{time_total}\n' -o /dev/null -s http://localhost:<TARGET_SERVICE_PORT><PATH_FILTER-example>
   ```

   Latency should rise by approximately `LATENCY` during the chaos window.
2. **Inspect iptables redirects.**

   ```bash
   sudo iptables -t nat -L -n -v | grep <PROXY_PORT>
   ```
3. **Inspect Linux Chaos Infrastructure logs.**

   ```bash
   sudo journalctl -u linux-chaos-infrastructure -n 100 --no-pager
   ```

***

### Recovery and cleanup <a href="#recovery-and-cleanup" id="recovery-and-cleanup"></a>

* **End of duration:** The chaos pod removes the iptables redirect and stops the proxy when `DURATION` elapses.
* **Abort the experiment:** Stopping the experiment from Chaos Studio also removes the redirect and stops the proxy.
* **Manual recovery:** If the redirect survives an abort, inspect with `sudo iptables -t nat -L -n -v` and delete the rule added by the fault; kill the proxy with `sudo lsof -i :<PROXY_PORT>` followed by `sudo kill <pid>`.

***

### Limitations <a href="#limitations" id="limitations"></a>

* **HTTP/3 (QUIC) not intercepted.**
* **HTTPS requires certificates:** Without `HTTPS_ROOT_CERT_FILE`, HTTPS traffic cannot be matched on path/header/method.
* **Single VM scope:** Each fault run targets one VM.
* **Per-request delay:** `LATENCY` is applied per matching request; high-request-rate workloads accumulate large delays.
* **`/tmp` exec required.**

***

### Troubleshooting <a href="#troubleshooting" id="troubleshooting"></a>

<details>

<summary>Linux API latency fault did not slow any request in Harness Chaos Engineering</summary>

Verify PATH\_FILTER, METHODS, and HEADERS\_FILTERS match your test traffic. For HTTPS endpoints, set HTTPS\_ENABLED=true and provide HTTPS\_ROOT\_CERT\_FILE. Confirm the iptables redirect is in place with sudo iptables -t nat -L -n -v.

</details>

<details>

<summary>Proxy port already in use</summary>

PROXY\_PORT (default 20000) conflicted with another listener. Change PROXY\_PORT to a free port and re-run.

</details>

<details>

<summary>Latency persists after the experiment ends</summary>

If the iptables redirect was not removed, inspect with sudo iptables -t nat -L -n -v and delete the rule added by the fault. Kill any orphan proxy on PROXY\_PORT.

</details>

***

### Related faults <a href="#related-faults" id="related-faults"></a>

* [Linux API block](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/linux/linux-api-block.md): Block requests entirely instead of delaying them.
* [Linux API status code](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/linux/linux-api-status-code.md): Change the status code without adding latency.
* [Linux network latency](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/linux/linux-network-latency.md): Delay at the network layer instead of the application layer.