> For the complete documentation index, see [llms.txt](https://developer.harness.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://developer.harness.io/resilience-testing/chaos-engineering/faults/chaos-fault-categories/kubernetes/pod/pod-api-modify-response-custom.md).

# Pod API modify response custom

Pod API modify response custom is a Kubernetes pod-level chaos fault that combines status code, header, and body modifications on selected API calls in a single experiment. Think of it as [Pod API modify body](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/kubernetes/pod/pod-api-modify-body.md), [Pod API modify header](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/kubernetes/pod/pod-api-modify-header.md), and [Pod API status code](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/kubernetes/pod/pod-api-status-code.md) rolled into one fault. It accepts the same rich filter set (path, method, headers, query, source, destination) and supports HTTPS through user-supplied TLS certificates. When the fault ends, responses return to whatever the application produces.

Use this fault when one realistic failure scenario requires changing multiple aspects of a response at once: returning `429` with a `Retry-After` header and a JSON `{"error":"rate_limited"}` body, or simulating an upstream that returns `200` with corrupted JSON, or returning `503` with a custom `X-Maintenance` header to model planned downtime.

{% hint style="info" %}
**RUN YOUR FIRST EXPERIMENT**

If you have not configured the chaos infrastructure yet, go to [Quickstart](/resilience-testing/chaos-engineering/new-to-chaos-engineering/quickstart.md) to install the chaos infrastructure and run an experiment end to end.
{% endhint %}

***

### Use cases <a href="#use-cases" id="use-cases"></a>

Run this fault when you want to answer concrete questions like:

* **Realistic rate-limit responses:** Set `STATUS_CODE=429`, `HEADERS_MAP={"Retry-After":"30"}`, and `RESPONSE_BODY={"error":"rate_limited"}` to simulate a real-world rate-limited upstream. Does the client honor `Retry-After`?
* **Maintenance-window simulation:** Return `503` with a custom `X-Maintenance: planned` header and an HTML maintenance page body. Does the client surface a useful message to end users?
* **Auth refresh path:** Return `401` with `WWW-Authenticate: Bearer error="invalid_token"` and a matching JSON body. Does the client trigger the token-refresh flow rather than failing?
* **Backwards-compatibility breakage:** Return `200` with a body missing a field the client expects. Does the client tolerate the new shape, or does it crash on the missing field?
* **Tenant-specific custom errors:** Use `HEADERS_FILTERS=X-Tenant: prod-eu` to apply a different status, body, and headers only to one tenant.

***

### Prerequisites <a href="#prerequisites" id="prerequisites"></a>

* **Kubernetes version:** 1.21 or later. Go to [What's supported](/resilience-testing/chaos-engineering/new-to-chaos-engineering/whats-supported.md) to confirm distribution support.
* **Target pods are Running:** The application pods you intend to target are in the `Running` state before the fault is launched.
* **Privileged pods allowed:** The cluster lets you schedule privileged pods in the chaos namespace. GKE Autopilot supports this fault but requires the one-time setup in [Chaos on GKE Autopilot](/resilience-testing/chaos-engineering/use-chaos-engineering/gke-autopilot.md); other locked-down distributions may need similar exemptions.
* **Container runtime access:** The chaos pod can reach the container runtime socket on the target node (`/run/containerd/containerd.sock`, `/var/run/docker.sock`, or `/var/run/crio/crio.sock`).
* **API service on a known port:** The target container serves HTTP, HTTPS, or gRPC traffic on a port you can specify with `TARGET_SERVICE_PORT`.
* **TLS material for HTTPS targets:** When `HTTPS_ENABLED=true`, you provide CA, server, and (optionally) client certificate secrets so the proxy can terminate TLS and apply filters.
* **Workload selector defined:** The chaos experiment knows the target workload by kind, namespace, and either names or labels.

***

### Supported environments <a href="#supported-environments" id="supported-environments"></a>

| Platform                                 | Support status                                                                                                 |
| ---------------------------------------- | -------------------------------------------------------------------------------------------------------------- |
| Amazon EKS                               | Supported                                                                                                      |
| Azure AKS                                | Supported                                                                                                      |
| Google GKE                               | Supported                                                                                                      |
| Red Hat OpenShift                        | Supported                                                                                                      |
| Rancher                                  | Supported                                                                                                      |
| VMware Tanzu                             | Supported                                                                                                      |
| Self-managed Kubernetes (CNCF-certified) | Supported                                                                                                      |
| GKE Autopilot                            | Supported with [Autopilot setup](/resilience-testing/chaos-engineering/use-chaos-engineering/gke-autopilot.md) |
| EKS Fargate, ACI virtual nodes           | Not supported (no access to container runtime sockets)                                                         |

***

### Permissions required <a href="#permissions-required" id="permissions-required"></a>

The fault runs under the chaos infrastructure's service account.

| Resource (`apiGroup`)                                               | Verbs                                                                    | Why it is needed                                            |
| ------------------------------------------------------------------- | ------------------------------------------------------------------------ | ----------------------------------------------------------- |
| `pods` (`""`)                                                       | `get`, `list`, `create`, `delete`, `deletecollection`, `patch`, `update` | Discover target pods and run the chaos pod on the same node |
| `pods/log` (`""`)                                                   | `get`, `list`, `watch`                                                   | Stream chaos pod logs for status and debugging              |
| `deployments`, `statefulsets`, `replicasets`, `daemonsets` (`apps`) | `get`, `list`                                                            | Resolve the target workload to the pods it owns             |
| `events` (`""`)                                                     | `get`, `list`, `create`, `patch`, `update`                               | Record fault progress as Kubernetes events                  |
| `jobs` (`batch`)                                                    | `get`, `list`, `create`, `delete`, `deletecollection`                    | Run the chaos job that drives the fault                     |
| `secrets` (`""`)                                                    | `get`, `list`                                                            | Read TLS certificate secrets when `HTTPS_ENABLED=true`      |

The default Harness chaos infrastructure service account already includes these permissions.

***

### Fault tunables <a href="#fault-tunables" id="fault-tunables"></a>

Configure the following fault parameters when you add Pod API modify response custom to an experiment in Chaos Studio. Defaults are shown for reference. Any modification field (`STATUS_CODE`, `HEADERS_MAP`, `RESPONSE_BODY`) left empty is not applied; you can use any subset of the three.

**Chaos parameters**

| Tunable                  | Description                                                                                                                                | Default |
| ------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------ | ------- |
| `STATUS_CODE`            | HTTP status code to return. `0` keeps the application's original status code.                                                              | `0`     |
| `HEADERS_MAP`            | JSON object of header key-value pairs to inject or replace. Set a value to `""` to remove that header. Empty leaves all headers unchanged. | `""`    |
| `RESPONSE_BODY`          | Body string to overwrite the response (or request, when `DATA_DIRECTION` includes `request`). Empty leaves the body unchanged.             | `""`    |
| `DATA_DIRECTION`         | Which direction to modify: `request`, `response`, or `both`.                                                                               | `both`  |
| `TRANSACTION_PERCENTAGE` | Percentage of matched API calls to modify, between `0` and `100`.                                                                          | `0`     |
| `TARGET_SERVICE_PORT`    | Port the target container listens on for API traffic.                                                                                      | `80`    |
| `TOTAL_CHAOS_DURATION`   | Duration of the fault in seconds.                                                                                                          | `60`    |

**Filters (API matching)**

All filters are optional. Leave empty to match everything in that dimension. Combining filters narrows the match (AND across dimensions).

| Tunable           | Description                                                                                                                   | Default |
| ----------------- | ----------------------------------------------------------------------------------------------------------------------------- | ------- |
| `PATH_FILTER`     | URL path the fault matches. Empty matches all paths.                                                                          | `""`    |
| `METHODS`         | Comma-separated HTTP methods to match. Empty matches all methods.                                                             | `""`    |
| `QUERY_PARAMS`    | Query-parameter filter. Empty matches all query strings.                                                                      | `""`    |
| `HEADERS_FILTERS` | Header filter used to match incoming calls (distinct from `HEADERS_MAP`, which is used to inject). Empty matches all headers. | `""`    |

**Filters (traffic source and destination)**

| Tunable             | Description                                                                                        | Default   |
| ------------------- | -------------------------------------------------------------------------------------------------- | --------- |
| `SERVICE_DIRECTION` | Whether to filter `ingress` traffic (received by the pod) or `egress` traffic (sent from the pod). | `ingress` |
| `SOURCE_HOSTS`      | Hostnames of the calling client (ingress only). Empty matches any source.                          | `""`      |
| `SOURCE_IPS`        | Source IPs of the calling client (ingress only). Empty matches any source.                         | `""`      |
| `DESTINATION_HOSTS` | Destination hostnames for the call (egress only). Empty matches any destination.                   | `""`      |
| `DESTINATION_IPS`   | Destination IPs for the call (egress only). Empty matches any destination.                         | `""`      |
| `DESTINATION_PORTS` | Comma-separated destination ports (egress only). Empty matches any port.                           | `""`      |

**TLS (for HTTPS targets)**

| Tunable               | Description                                                                                           | Default |
| --------------------- | ----------------------------------------------------------------------------------------------------- | ------- |
| `HTTPS_ENABLED`       | Set to `true` when the target serves HTTPS so the proxy terminates TLS to apply filters.              | `false` |
| `CA_CERTIFICATES`     | Kubernetes secret holding the Base64-encoded CA certificate (`ca.key`, `ca.crt`).                     | `""`    |
| `SERVER_CERTIFICATES` | Secret holding the Base64-encoded server certificate (`server.key`, `server.crt`).                    | `""`    |
| `CLIENT_CERTIFICATES` | Secret holding the Base64-encoded client certificate (`client.key`, `client.crt`) for mTLS upstreams. | `""`    |

**Proxy and interface**

| Tunable             | Description                                                                                               | Default |
| ------------------- | --------------------------------------------------------------------------------------------------------- | ------- |
| `PROXY_PORT`        | Port the chaos proxy listens on inside the container's network namespace.                                 | `20000` |
| `NETWORK_INTERFACE` | Network interface inside the target container's namespace. Almost always `eth0` for standard CNI plugins. | `eth0`  |

**Targeting**

| Tunable                   | Description                                                                                                          | Default    |
| ------------------------- | -------------------------------------------------------------------------------------------------------------------- | ---------- |
| `TARGET_PODS`             | Comma-separated list of pod names to target. Empty selects from the workload's pods using `POD_AFFECTED_PERCENTAGE`. | `""`       |
| `TARGET_CONTAINER`        | Container in the pod whose network namespace to enter. Empty targets the first container in the pod spec.            | `""`       |
| `NODE_LABEL`              | Label selector to filter target pods by the node they run on. Empty disables node-based filtering.                   | `""`       |
| `POD_AFFECTED_PERCENTAGE` | Percentage of the workload's pods to target. `0` means one pod.                                                      | `0`        |
| `SEQUENCE`                | When multiple pods are targeted, inject `parallel` (all at once) or `serial` (one after another).                    | `parallel` |

**Runtime and helper**

| Tunable             | Description                                                                                                                                                                                                      | Default                           |
| ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------- |
| `CONTAINER_RUNTIME` | Container runtime on the target nodes. One of `containerd`, `docker`, `crio`.                                                                                                                                    | `containerd`                      |
| `SOCKET_PATH`       | Path to the container runtime socket on the target node. Set to match `CONTAINER_RUNTIME`.                                                                                                                       | `/run/containerd/containerd.sock` |
| `RAMP_TIME`         | Wait period in seconds before and after the fault. Go to [ramp time](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/common-tunables-for-all-faults.md#ramp-time) to read how it is applied. | `0`                               |

Tunables that apply to every chaos fault are documented in [common tunables for all faults](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/common-tunables-for-all-faults.md).

{% hint style="info" %}
**USE JUST WHAT YOU NEED**

If you only want to change the body, use [Pod API modify body](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/kubernetes/pod/pod-api-modify-body.md) instead. Reach for this fault when a realistic failure requires more than one of status, body, and headers to change together.
{% endhint %}

#### Configure for your container runtime <a href="#configure-for-your-container-runtime" id="configure-for-your-container-runtime"></a>

Set `CONTAINER_RUNTIME` and `SOCKET_PATH` to match the runtime on the target node:

| `CONTAINER_RUNTIME`    | `SOCKET_PATH`                     |
| ---------------------- | --------------------------------- |
| `containerd` (default) | `/run/containerd/containerd.sock` |
| `docker`               | `/var/run/docker.sock`            |
| `crio`                 | `/var/run/crio/crio.sock`         |

***

### Fault execution in brief <a href="#fault-execution-in-brief" id="fault-execution-in-brief"></a>

Intercepts API traffic on `TARGET_SERVICE_PORT` inside the container's network namespace and, for the configured percentage of calls that match the path, method, header, query, and source or destination filters, simultaneously overrides the status code, headers, and body with the supplied values, optionally terminating TLS to apply the same logic to HTTPS calls.

***

### Expected behavior during fault execution <a href="#expected-behavior-during-fault-execution" id="expected-behavior-during-fault-execution"></a>

* API calls that match every configured filter have their response (and optionally request, depending on `DATA_DIRECTION`) rewritten with whatever combination of `STATUS_CODE`, `HEADERS_MAP`, and `RESPONSE_BODY` you set. Any of the three left empty is preserved as the application produced it.
* Calls outside the filter set pass through unchanged. Only the percentage selected by `TRANSACTION_PERCENTAGE` is modified.
* For HTTPS targets, the proxy terminates TLS using the supplied certificates so it can intercept and rewrite the call.
* Clients see a single coherent response (for example, status + headers + body that all match the simulated scenario), exercising the same code paths a real upstream failure would.

{% hint style="info" %}
**WHEN THE FAULT ENDS**

After `TOTAL_CHAOS_DURATION`, the proxy is torn down and API responses return to whatever the application produces.
{% endhint %}

#### Signals to watch <a href="#signals-to-watch" id="signals-to-watch"></a>

Attach [resilience probes](/resilience-testing/chaos-engineering/use-chaos-engineering/probes.md) to assert each layer:

* **Client behavior matches the scenario:** Use an [HTTP probe](/resilience-testing/chaos-engineering/use-chaos-engineering/probes/http-probe.md) to verify the client honors the headers you injected (such as `Retry-After`).
* **End-user error rate:** Use a Prometheus probe on the top-level error counter to detect leakage of the simulated failure to end users.
* **Application logs:** Use a [command probe](/resilience-testing/chaos-testing/probes/command-probe.md) to look for stack traces from clients that did not handle the combined response correctly.

***

### Verify the fault execution effect <a href="#verify-the-fault-execution-effect" id="verify-the-fault-execution-effect"></a>

While the experiment is running, confirm all three modifications land together:

1. **Make a matched call and inspect status, headers, and body.**

   ```bash
   kubectl run -n <namespace> tester --image=curlimages/curl --rm -it -- \
     curl -i -X <METHOD> "http://<target-pod-ip>:<TARGET_SERVICE_PORT><PATH_FILTER>"
   ```

   The status line should equal `HTTP/1.1 <STATUS_CODE> ...`, headers from `HEADERS_MAP` should appear (and removed keys should be absent), and the body should equal `RESPONSE_BODY` (or remain unchanged if it was empty).
2. **Confirm an unmatched call is untouched.**

   A call to a different path or method should return the application's normal status, headers, and body.

***

### Recovery and cleanup <a href="#recovery-and-cleanup" id="recovery-and-cleanup"></a>

* **End of duration:** The proxy is removed automatically and API traffic returns to baseline.
* **Abort the experiment:** Stopping the experiment from Chaos Studio triggers the same cleanup path.
* **Failed cleanup:** If automated cleanup did not complete, restart the target pod to reset its network state.

***

### Limitations <a href="#limitations" id="limitations"></a>

* **Serverless Kubernetes (EKS Fargate, ACI virtual nodes):** These platforms do not expose container runtime sockets and reject the privileged access the fault needs. GKE Autopilot is supported once the one-time setup in [Chaos on GKE Autopilot](/resilience-testing/chaos-engineering/use-chaos-engineering/gke-autopilot.md) is in place.
* **Windows containers:** This fault is supported on Linux pods only.
* **HTTPS without certificates:** When `HTTPS_ENABLED=true`, the proxy must terminate TLS. If the supplied certificates do not chain to one the client trusts, the client refuses the connection before any filter is applied.
* **gRPC status:** gRPC carries status in HTTP/2 trailers (`grpc-status`). This fault overrides HTTP status; gRPC clients also check `grpc-status` and may behave differently.
* **Pseudo-headers (HTTP/2):** `:method`, `:path`, `:status`, `:authority` cannot be modified through `HEADERS_MAP`.
* **Port already bound:** If `PROXY_PORT` collides with a port the target container is already using, the fault fails to start.

***

### Troubleshooting <a href="#troubleshooting" id="troubleshooting"></a>

<details>

<summary>Pod API modify response custom experiment stays Pending or never starts in Harness Chaos Engineering</summary>

Inspect the chaos pods in the experiment namespace with kubectl describe pod -n . The most common causes are taints on the target node that the chaos pods do not tolerate, insufficient resources, or a PodSecurity admission policy blocking privileged pods. Add the required tolerations to the experiment or run in a namespace with privileged Pod Security level.

</details>

<details>

<summary>Responses are unchanged during pod-api-modify-response-custom</summary>

The most common causes are: TRANSACTION\_PERCENTAGE is 0 (default) so no calls are modified; STATUS\_CODE, HEADERS\_MAP, and RESPONSE\_BODY are all empty (nothing to apply); filters are over-specified and match no real traffic; HTTPS\_ENABLED is false but the target serves HTTPS; or the supplied TLS certificates do not chain correctly. Re-run with TRANSACTION\_PERCENTAGE=100 and a known STATUS\_CODE like 503 to confirm the path.

</details>

<details>

<summary>TLS handshake fails for pod-api-modify-response-custom in Harness Chaos Engineering</summary>

Verify the secrets referenced by CA\_CERTIFICATES, SERVER\_CERTIFICATES, and CLIENT\_CERTIFICATES exist in the chaos namespace and contain Base64-encoded key/crt pairs (ca.key/ca.crt, server.key/server.crt, client.key/client.crt). The server certificate must include the target pod's service name in its SAN list.

</details>

<details>

<summary>Custom responses persist after pod-api-modify-response-custom ends</summary>

Automated cleanup did not complete. Restart the target pod to reset its network state. If the issue recurs, capture the chaos pod logs from the experiment namespace before the next run and share them with Harness support.

</details>

***

### Related faults <a href="#related-faults" id="related-faults"></a>

* [Pod API block](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/kubernetes/pod/pod-api-block.md): Drop API calls instead of returning a custom response.
* [Pod API latency](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/kubernetes/pod/pod-api-latency.md): Delay API calls.
* [Pod API modify body](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/kubernetes/pod/pod-api-modify-body.md): Only overwrite the body.
* [Pod API modify header](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/kubernetes/pod/pod-api-modify-header.md): Only modify headers.
* [Pod API status code](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/kubernetes/pod/pod-api-status-code.md): Only change the status code.
* [Common pod fault tunables](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/kubernetes/pod/common-tunables-for-pod-faults.md): Shared environment variables for selecting target pods and workloads.