Skip to main content

Linux disk fill

Last updated on

Linux disk fill is a chaos fault that writes a file under FILL_PATH until it occupies FILL_STORAGE of space (or FILL_STORAGE percent of the volume) on the target Linux machine for DURATION, then removes the file and frees the space. The fault runs through the Linux Chaos Infrastructure (LCI) systemd service installed on the target VM.

Use this fault to test how a workload behaves when its writable storage fills up: whether the application surfaces a clean ENOSPC error or wedges, whether log rotation kicks in, whether monitoring detects low disk space within the alerting SLA, and whether the system recovers automatically once space is reclaimed.

Run your first experiment

If you have not installed the Linux Chaos Infrastructure yet, go to Linux Chaos Infrastructure to install the agent and connect the VM to the control plane.

Use cases​

Run this fault when you want to answer concrete questions like:

  • Write paths under ENOSPC: When FILL_PATH runs out of space, do application writes fail cleanly or wedge?
  • Log rotation: Does logrotate (or equivalent) kick in and free space before the volume fills?
  • Monitoring fidelity: Do disk-space alerts (node_filesystem_avail_bytes, df -h) fire within the alerting SLA?
  • Recovery: After the fill file is removed, do the application and monitoring return to baseline without a restart?

Prerequisites​

  • Linux Chaos Infrastructure installed: The linux-chaos-infrastructure systemd service is active on the target VM and the infrastructure is in CONNECTED state. Go to Linux Chaos Infrastructure to install it.
  • Writable target path: FILL_PATH exists, is writable by the LCI service, and resides on a filesystem with enough headroom for FILL_STORAGE.
  • dd available: The fault uses dd from coreutils to write the fill file.

Supported environments​

The fault has been tested on the following Linux distributions. Go to Linux fault requirements to see the full compatibility matrix.

PlatformSupport status
Ubuntu 16+, Debian 10+Supported
CentOS 7+, RHEL 7+, Fedora 30+Supported
openSUSE LEAP 15.4+ / SUSE Linux Enterprise 15+Supported

Permissions required​

This fault is classified as a Basic Linux fault. It runs with the privileges of the Linux Chaos Infrastructure systemd service (root user and root user group) on the target VM. No cloud credentials are needed.

Fault tunables​

Configure the following fault parameters when you add Linux disk fill to an experiment in Chaos Studio. Defaults are shown for reference.

Required parameters

TunableDescriptionDefault
FILL_PATHAbsolute path where the fill file is created. Must exist and be writable by the LCI service.(required)

Chaos parameters

TunableDescriptionDefault
DURATIONTotal duration of the fault. Accepts [hours]h[minutes]m[seconds]s format (for example, 30s, 1m25s, 1h3m2s).30s
FILL_STORAGEAmount of storage to fill. Accepts absolute values (for example, 256m, 1g) or a percentage of the volume (for example, 50%).50%
DATA_BLOCK_SIZEBlock size used by dd when writing the fill file. Larger blocks fill faster; smaller blocks complete on volumes with small free space.256k
RAMP_TIMEWait period in seconds before and after the fault. Go to ramp time to read how it is applied.0

Tunables that apply to every fault are documented in common tunables for all faults.

Fault execution in brief​

Writes a fill file under FILL_PATH using dd until it occupies FILL_STORAGE, holds for DURATION, then deletes the file and frees the space.

Expected behavior during fault execution​

  • Free space on the volume hosting FILL_PATH drops by approximately FILL_STORAGE for the duration of the fault.
  • New writes by other processes on the same volume may fail with ENOSPC ("No space left on device") if the volume reaches 100%.
  • Disk-space monitoring (node_filesystem_avail_bytes, df -h) reports the reduced free space.
  • After the duration ends, the fill file is removed and free space returns to baseline.
When the fault ends

The chaos pod deletes the fill file. Free space returns immediately; the kernel may keep page-cache entries around until reclaim.

Signals to watch​

Attach resilience probes to assert each layer:

  • Disk space: Use a Prometheus probe on node_filesystem_avail_bytes{mountpoint="<mount>"} and assert it dropped during the chaos window.
  • Write errors: Use a command probe to scan application logs for ENOSPC or "No space left on device" entries.
  • End-to-end availability: Use an HTTP probe on a user-visible endpoint that exercises a write path.

Verify the fault execution effect​

While the experiment is running, confirm the volume filled and was reclaimed:

  1. Observe free space on the target volume.

    df -h <mount-of-FILL_PATH>
    ls -lh <FILL_PATH>

    Free space should drop by approximately FILL_STORAGE during the chaos window and return to baseline afterwards.

  2. Inspect Linux Chaos Infrastructure logs.

    sudo journalctl -u linux-chaos-infrastructure -n 100 --no-pager

    Look for the fault start, the fill file path, the dd progress, and the fault end markers.

Recovery and cleanup​

  • End of duration: The chaos pod deletes the fill file when DURATION elapses; free space returns to baseline.
  • Abort the experiment: Stopping the experiment from Chaos Studio also triggers the cleanup.
  • Manual recovery: If the fill file survives an abort, remove it with sudo rm <FILL_PATH>/<fill-file> (the LCI logs record the exact filename).
  • Workload recovery: Applications that experienced ENOSPC may need to be restarted if they cached the failure state.

Limitations​

  • Filesystem-level only: The fault fills space at the filesystem level. It does not exhaust inodes; for inode exhaustion use a custom fault.
  • One path per run: A single experiment fills exactly one path on the target VM. Use multiple experiments to fill multiple volumes.
  • Disk-cache effects: Filling a volume close to 100% can degrade other I/O on the same disk due to fragmentation and cache pressure.
  • No mid-flight resize: FILL_STORAGE cannot be adjusted during the chaos window; abort and re-run the experiment to change it.

Troubleshooting​

Linux disk fill fault fails with no such file or directory in Harness Chaos Engineering

FILL_PATH must exist on the target VM and be writable by the linux-chaos-infrastructure service (which runs as root). Verify with sudo test -w FILL_PATH and create the directory if missing with sudo mkdir -p FILL_PATH.

Volume did not fill to the configured FILL_STORAGE

If the volume has less free space than FILL_STORAGE, the fault fills to whatever is available. Reduce FILL_STORAGE or run df -h to confirm headroom before the experiment.

Disk space stays low after the experiment ends

If the fill file was not removed (for example, the experiment aborted ungracefully), delete it manually with sudo rm FILL_PATH/<fill-file>. The exact filename is recorded in sudo journalctl -u linux-chaos-infrastructure logs.