> For the complete documentation index, see [llms.txt](https://developer.harness.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://developer.harness.io/resilience-testing/chaos-engineering/faults/chaos-fault-categories/kubernetes/pod/disk-fill.md).

# Disk fill

Disk fill is a Kubernetes pod-level chaos fault that consumes a configurable percentage of a target container's ephemeral storage. The fault writes data into the container's writable layer until the configured percentage of the container's `ephemeral-storage` limit is reached, then holds the consumption for the rest of the duration.

Use this fault to test how a workload behaves when its scratch space, log directory, or temp files exceed the budget you have set: whether the kubelet evicts the pod under `DiskPressure`, whether the application gracefully rotates and discards data, and whether downstream systems detect the disrupted writes.

{% hint style="info" %}
**RUN YOUR FIRST EXPERIMENT**

If you have not configured the chaos infrastructure yet, go to [Quickstart](/resilience-testing/chaos-engineering/new-to-chaos-engineering/quickstart.md) to install the chaos infrastructure and run an experiment end to end.
{% endhint %}

***

### Use cases <a href="#use-cases" id="use-cases"></a>

Run this fault when you want to answer concrete questions like:

* **Ephemeral-storage limits:** When the container fills its `resources.limits.ephemeral-storage`, does the kubelet evict the pod with a `DiskPressure` reason inside your expected window?
* **Log rotation under pressure:** Does the application's log rotation policy keep the writable layer below the limit when log volume spikes, or do uncapped logs themselves cause the eviction?
* **Temp-file cleanup:** Does the workload clean up temp files on `SIGTERM`, or does an eviction leak temp data into the next replacement pod?
* **Disk-bound back-pressure:** When the disk fills, does the application detect write failures and shed load, surface 5xx, or hang on the next `write()`?
* **Replica recovery after eviction:** Does the workload's controller (Deployment, StatefulSet) reschedule the evicted pod onto a node with capacity inside your SLO?

***

### Prerequisites <a href="#prerequisites" id="prerequisites"></a>

* **Kubernetes version:** 1.21 or later. Go to [What's supported](/resilience-testing/chaos-engineering/new-to-chaos-engineering/whats-supported.md) to confirm distribution support.
* **Target pods are Running:** The pods you intend to fill are in the `Running` state before the fault is launched. The fault reports a precheck failure otherwise.
* **Privileged pods allowed:** The cluster lets you schedule privileged pods in the chaos namespace. The fault writes into the target container's writable layer through the container runtime.
* **Container runtime access:** The chaos infrastructure can reach the container runtime on the target nodes. The default `containerd` socket path is mounted automatically.
* **Ephemeral-storage limit configured (recommended):** The target container has `resources.limits.ephemeral-storage` set. Without a limit, `FILL_PERCENTAGE` is interpreted against the node's filesystem capacity and the experiment can spill onto neighboring pods.

***

### Supported environments <a href="#supported-environments" id="supported-environments"></a>

| Platform                                 | Support status                                                                                                 |
| ---------------------------------------- | -------------------------------------------------------------------------------------------------------------- |
| Amazon EKS                               | Supported                                                                                                      |
| Azure AKS                                | Supported                                                                                                      |
| Google GKE                               | Supported                                                                                                      |
| Red Hat OpenShift                        | Supported                                                                                                      |
| Rancher                                  | Supported                                                                                                      |
| VMware Tanzu                             | Supported                                                                                                      |
| Self-managed Kubernetes (CNCF-certified) | Supported                                                                                                      |
| GKE Autopilot                            | Supported with [Autopilot setup](/resilience-testing/chaos-engineering/use-chaos-engineering/gke-autopilot.md) |
| EKS Fargate, ACI virtual nodes           | Not supported (no access to container runtime sockets)                                                         |

***

### Permissions required <a href="#permissions-required" id="permissions-required"></a>

The fault runs under the chaos infrastructure's service account.

| Resource (`apiGroup`)                                                 | Verbs                                                                    | Why it is needed                                            |
| --------------------------------------------------------------------- | ------------------------------------------------------------------------ | ----------------------------------------------------------- |
| `pods` (`""`)                                                         | `get`, `list`, `create`, `delete`, `deletecollection`, `patch`, `update` | Discover target pods and run the chaos pod on the same node |
| `pods/log` (`""`)                                                     | `get`, `list`, `watch`                                                   | Stream chaos pod logs for status and debugging              |
| `events` (`""`)                                                       | `get`, `list`, `create`, `patch`, `update`                               | Record fault progress as Kubernetes events                  |
| `configmaps` (`""`)                                                   | `get`, `list`                                                            | Mount configuration into the chaos pod when specified       |
| `deployments`, `statefulsets`, `replicasets`, `daemonsets` (`apps`)   | `get`, `list`                                                            | Resolve the parent controller for each target pod           |
| `replicationcontrollers` (`""`)                                       | `get`, `list`                                                            | Resolve the parent controller for legacy workloads          |
| `deploymentconfigs` (`apps.openshift.io`)                             | `get`, `list`                                                            | Resolve the parent controller on OpenShift                  |
| `rollouts` (`argoproj.io`)                                            | `get`, `list`                                                            | Resolve the parent controller for Argo Rollouts             |
| `jobs` (`batch`)                                                      | `get`, `list`, `create`, `delete`, `deletecollection`                    | Run the chaos job that drives the fault                     |
| `chaosengines`, `chaosexperiments`, `chaosresults` (`litmuschaos.io`) | `get`, `list`, `create`, `patch`, `update`, `delete`                     | Manage the chaos engine, experiment, and result CRDs        |

The default Harness chaos infrastructure service account already includes these permissions. You only need to extend it if you are running with a restricted scope.

***

### Fault tunables <a href="#fault-tunables" id="fault-tunables"></a>

Configure the following fault parameters when you add Disk fill to an experiment in Chaos Studio. Defaults are shown for reference.

**Chaos parameters**

| Tunable                       | Description                                                                                                              | Default |
| ----------------------------- | ------------------------------------------------------------------------------------------------------------------------ | ------- |
| `FILL_PERCENTAGE`             | Percentage of the target container's `ephemeral-storage` limit to consume.                                               | `80`    |
| `DATA_BLOCK_SIZE`             | Write block size in KB. Larger blocks fill the disk faster; smaller blocks generate more I/O syscalls.                   | `256`   |
| `EPHEMERAL_STORAGE_MEBIBYTES` | Override for the detected ephemeral-storage limit, in MiB. Set to `0` to auto-detect from the container's resource spec. | `0`     |
| `TOTAL_CHAOS_DURATION`        | Duration of the fault in seconds. The fill is held for the duration once the target percentage is reached.               | `60`    |

**Targeting**

| Tunable                   | Description                                                                                                          | Default    |
| ------------------------- | -------------------------------------------------------------------------------------------------------------------- | ---------- |
| `TARGET_PODS`             | Comma-separated list of pod names to target. Empty selects from the workload's pods using `POD_AFFECTED_PERCENTAGE`. | `""`       |
| `TARGET_CONTAINER`        | Container in the pod whose ephemeral storage is filled. Empty targets the first container in the pod spec.           | `""`       |
| `NODE_LABEL`              | Label selector to filter target pods by the node they run on. Empty disables node-based filtering.                   | `""`       |
| `POD_AFFECTED_PERCENTAGE` | Percentage of the workload's pods to target. `0` means one pod.                                                      | `0`        |
| `SEQUENCE`                | When multiple pods are targeted, fill `parallel` (all at once) or `serial` (one after another).                      | `parallel` |

**Runtime and helper**

| Tunable             | Description                                                                                                                                                                                                      | Default                           |
| ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------- |
| `CONTAINER_RUNTIME` | Container runtime on the target nodes. One of `containerd`, `docker`, `crio`.                                                                                                                                    | `containerd`                      |
| `SOCKET_PATH`       | Path to the container runtime socket on the target node. Set to match `CONTAINER_RUNTIME`.                                                                                                                       | `/run/containerd/containerd.sock` |
| `RAMP_TIME`         | Wait period in seconds before and after the fault. Go to [ramp time](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/common-tunables-for-all-faults.md#ramp-time) to read how it is applied. | `0`                               |

Tunables that apply to every chaos fault are documented in [common tunables for all faults](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/common-tunables-for-all-faults.md).

{% hint style="warning" %}
**WITHOUT AN EPHEMERAL-STORAGE LIMIT, THE FAULT HAS NO BOUNDARY**

If the target container does not have `resources.limits.ephemeral-storage` set, the fault falls back to the node's filesystem size for the percentage calculation. A `FILL_PERCENTAGE` of 80 then targets 80 percent of the entire node disk and can starve neighboring pods. Set a sensible container limit or use the `EPHEMERAL_STORAGE_MEBIBYTES` override.
{% endhint %}

#### Configure for your container runtime <a href="#configure-for-your-container-runtime" id="configure-for-your-container-runtime"></a>

Set `CONTAINER_RUNTIME` and `SOCKET_PATH` to match the runtime on the target node:

| `CONTAINER_RUNTIME`    | `SOCKET_PATH`                     |
| ---------------------- | --------------------------------- |
| `containerd` (default) | `/run/containerd/containerd.sock` |
| `docker`               | `/var/run/docker.sock`            |
| `crio`                 | `/var/run/crio/crio.sock`         |

***

### Fault execution in brief <a href="#fault-execution-in-brief" id="fault-execution-in-brief"></a>

Writes data into the target container's writable layer until the configured percentage of its ephemeral-storage limit is consumed, holds the fill for the duration, and frees the data when the fault ends.

***

### Expected behavior during fault execution <a href="#expected-behavior-during-fault-execution" id="expected-behavior-during-fault-execution"></a>

* The container's ephemeral-storage usage rises toward the configured `FILL_PERCENTAGE` of its limit and stays there for the duration.
* If the fill exceeds `resources.limits.ephemeral-storage`, the kubelet evicts the pod with `Reason: Evicted` and `Message: container ... has used ... ephemeral-storage which exceeds limit`. The pod's controller reschedules a replacement on another Ready node.
* Application writes to the writable layer (logs, temp files, scratch space) may fail with `ENOSPC` (no space left on device) before the eviction completes. How the application handles that error determines the user-visible behavior.
* Reads are not affected. Existing files remain readable; only new writes can fail.
* Volume-backed paths (`emptyDir.medium: Memory`, PVCs, `hostPath` mounts) are not affected by ephemeral-storage limits and continue to function normally.
* The kubelet may also apply node-level `DiskPressure` if the node's filesystem fills past the eviction threshold; this is a node-wide condition, not specific to the target pod.

{% hint style="info" %}
**WHEN THE FAULT ENDS**

After `TOTAL_CHAOS_DURATION`, the fault removes the data it wrote and the container's ephemeral-storage usage returns to baseline within seconds. If the pod was evicted before the fault completed, the eviction stands; the replacement pod is a fresh container.
{% endhint %}

#### Signals to watch <a href="#signals-to-watch" id="signals-to-watch"></a>

A useful experiment captures signals from three layers. Attach [resilience probes](/resilience-testing/chaos-engineering/use-chaos-engineering/probes.md) to assert each layer automatically:

* **Application error rate:** Watch for spikes in 5xx and disk-related error logs (`ENOSPC`, `no space left`) during the fault. Use an [HTTP probe](/resilience-testing/chaos-engineering/use-chaos-engineering/probes/http-probe.md) to assert direct endpoint health.
* **Eviction events:** Track `kube_pod_status_reason{reason="Evicted"}` for the target workload's namespace. Use a [Prometheus probe](/resilience-testing/chaos-engineering/use-chaos-engineering/probes/apm-probes.md) to confirm whether the eviction did or did not happen, depending on what you are testing.
* **Replica recovery:** Look for `SuccessfulCreate` events on the workload after the eviction. Use a [Kubernetes probe](/resilience-testing/chaos-testing/probes/k8s-probe.md) to fail when the workload does not converge back to its declared replicas inside your SLO.

***

### Verify the fault execution effect <a href="#verify-the-fault-execution-effect" id="verify-the-fault-execution-effect"></a>

While the experiment is running, confirm that the target container's storage is actually being consumed:

1. **Check ephemeral-storage usage on the pod.**

   ```bash
   kubectl describe pod -n <namespace> <pod-name> | grep -A 3 'ephemeral-storage'
   ```

   The `Usage` line should be near `FILL_PERCENTAGE` of the configured limit.
2. **List disk usage inside the target container.**

   ```bash
   kubectl exec -n <namespace> <pod-name> -c <container> -- df -h /
   ```

   The root filesystem usage should be elevated. If the container has its own emptyDir or hostPath mounts, check those separately; ephemeral-storage is the writable layer of the container itself.
3. **Look for eviction events.**

   ```bash
   kubectl get events -n <namespace> --field-selector reason=Evicted --sort-by='.lastTimestamp'
   ```

   At `FILL_PERCENTAGE` above 100, expect an `Evicted` event referencing `ephemeral-storage`. Below 100, no eviction should occur.

***

### Recovery and cleanup <a href="#recovery-and-cleanup" id="recovery-and-cleanup"></a>

* **End of duration:** When `TOTAL_CHAOS_DURATION` elapses, the written data is removed and the container's ephemeral-storage usage returns to baseline within seconds.
* **Evicted pods reschedule:** If the fault exceeded the container's limit and the pod was evicted, the replacement is scheduled by the workload's controller on any Ready node with capacity.
* **Replacement on the same node may re-evict immediately:** If the replacement lands on a node that is itself under `DiskPressure`, it may be evicted before becoming Ready. Free node space or wait for the kubelet to garbage-collect unused images.
* **Application-level retention:** Applications that keep state in the writable layer (caches, scratch space, intermediate files) lose that state when evicted. Validate that the workload reconstructs the state cleanly on the replacement pod.
* **Abort the experiment early:** Stopping the experiment from Chaos Studio triggers cleanup of the written data. Any eviction that has already happened cannot be undone.

***

### Limitations <a href="#limitations" id="limitations"></a>

This fault is not appropriate in the following scenarios:

* **Serverless Kubernetes (EKS Fargate, ACI virtual nodes):** These platforms do not expose container runtime sockets and reject the privileged access the fault needs. GKE Autopilot is supported once the one-time setup in [Chaos on GKE Autopilot](/resilience-testing/chaos-engineering/use-chaos-engineering/gke-autopilot.md) is in place.
* **Containers without an ephemeral-storage limit:** Without a limit, the percentage calculation falls back to the node's filesystem and the blast radius extends beyond the target. Set a container limit before running this fault in production.
* **Volume-backed writable paths:** This fault fills the container's writable layer. It does not fill `emptyDir` volumes, PVCs, or `hostPath` mounts. Use a workload-specific fault if you need to fill those.
* **Read-only root filesystems:** Containers configured with `readOnlyRootFilesystem: true` reject writes from the fault. Use [FS fill](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/kubernetes/pod/fs-fill.md) against a writable mount path instead.
* **Tiny ephemeral-storage limits:** Limits below a few hundred MiB fill in well under a second and may not give the kubelet time to detect the breach. Raise the limit or use a larger duration with a lower `FILL_PERCENTAGE` to produce sustained pressure.

***

### Troubleshooting <a href="#troubleshooting" id="troubleshooting"></a>

<details>

<summary>Disk fill experiment stays Pending or never starts in Harness Chaos Engineering</summary>

Inspect the chaos pods in the experiment namespace with kubectl describe pod -n . The most common causes are taints on the target node, missing RBAC for the chaos service account, or a PodSecurity admission policy blocking privileged pods. Confirm the service account has the permissions listed above and the chaos namespace has the required Pod Security level.

</details>

<details>

<summary>Disk fill runs but the target container's ephemeral-storage usage does not increase</summary>

The most common causes are: TARGET\_CONTAINER does not match any container in the pod spec, the container has readOnlyRootFilesystem set to true, or the writable layer is too small to register a measurable change. Verify the container name with kubectl get pod -o jsonpath='{.spec.containers\[\*].name}', check the container's securityContext, and consider using FS fill with a writable mount path instead.

</details>

<details>

<summary>Pod was evicted but the workload's replica count did not recover</summary>

The replacement pod is failing to schedule. Run kubectl describe pod -n for the message. Common causes: every node is under DiskPressure (the fault filled too aggressively), insufficient CPU or memory available cluster-wide, or PV topology constraints. Lower FILL\_PERCENTAGE so the fault stays under the limit, or add cluster capacity.

</details>

<details>

<summary>Connection to container runtime fails for disk-fill in Harness Chaos Engineering</summary>

The default SOCKET\_PATH is /run/containerd/containerd.sock. If your nodes use Docker, set CONTAINER\_RUNTIME=docker and SOCKET\_PATH=/var/run/docker.sock. For CRI-O, set CONTAINER\_RUNTIME=crio and SOCKET\_PATH=/var/run/crio/crio.sock. Confirm the path by SSHing to the node and running ls -l on each candidate socket file.

</details>

***

### Related faults <a href="#related-faults" id="related-faults"></a>

* [FS fill](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/kubernetes/pod/fs-fill.md): Write a specific size of data to a specific path inside the container, useful for filling mounted volumes or non-root paths.
* [Pod CPU hog](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/kubernetes/pod/pod-cpu-hog.md) and [Pod memory hog](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/kubernetes/pod/pod-memory-hog.md): Apply CPU or memory pressure on the target container.
* [Node I/O stress](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/kubernetes/node/node-io-stress.md): Drive disk I/O at the node level rather than against a single container.
* [Common pod fault tunables](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/kubernetes/pod/common-tunables-for-pod-faults.md): Shared environment variables for selecting target pods and workloads.