> For the complete documentation index, see [llms.txt](https://developer.harness.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://developer.harness.io/resilience-testing/chaos-engineering/faults/chaos-fault-categories/gcp/gcp-vm-disk-loss-by-label.md).

# GCP VM disk loss by label

GCP VM disk loss by label is a GCP chaos fault that resolves the set of non-boot persistent disks matching `DISK_VOLUME_LABEL` in the zones listed in `ZONES` (project `GCP_PROJECT_ID`), selects `DISK_AFFECTED_PERCENTAGE` of them, detaches them from their attached VMs for `TOTAL_CHAOS_DURATION` seconds, then reattaches them.

Use this fault to test how a workload behaves when a labeled subset of storage volumes disappears: whether the application surfaces clean `EIO` errors, whether stateful workloads recover, whether DR procedures kick in, and whether monitoring detects the volume loss within the alerting SLA.

{% hint style="info" %}
**RUN YOUR FIRST EXPERIMENT**

If you have not configured the chaos infrastructure yet, go to [Quickstart](/resilience-testing/chaos-engineering/new-to-chaos-engineering/quickstart.md) to install the chaos infrastructure and run an experiment end to end.
{% endhint %}

***

### Use cases <a href="#use-cases" id="use-cases"></a>

Run this fault when you want to answer concrete questions like:

* **Tagged subset disk loss:** When `DISK_AFFECTED_PERCENTAGE` of disks labeled `DISK_VOLUME_LABEL` disappear, do dependents handle the failure cleanly?
* **Stateful workload resilience:** Do replicated stateful workloads (Cassandra, Postgres replicas) survive losing a labeled subset of storage?
* **DR rehearsal:** Validate the recovery procedure for losing a tagged subset of disks across zones.
* **Monitoring fidelity:** Do alerts on disk I/O and application errors fire within the alerting SLA?

***

### Prerequisites <a href="#prerequisites" id="prerequisites"></a>

* **Kubernetes version:** 1.21 or later for the chaos infrastructure cluster.
* **Label exists on at least one disk:** `DISK_VOLUME_LABEL` (formatted `key:value`) matches at least one non-boot disk in `ZONES`/`GCP_PROJECT_ID`.
* **Disks are non-boot:** Boot disks are rejected by design.
* **GCP credentials available:** Either a Google service account JSON key uploaded as a **File Secret in Harness Secret Manager** (referenced via `GCP_AUTHENTICATION_SECRET`) or Workload Identity bound to the chaos infrastructure service account.
* **IAM permissions granted:** The service account includes the permissions listed below.

***

### Supported environments <a href="#supported-environments" id="supported-environments"></a>

| Platform                                          | Support status                        |
| ------------------------------------------------- | ------------------------------------- |
| Compute Engine VMs with non-boot persistent disks | Supported                             |
| Regional persistent disks                         | Supported                             |
| Local SSD                                         | Not supported (cannot be detached)    |
| Boot disks                                        | Not supported                         |
| Multi-zone targeting in a single run              | Supported via comma-separated `ZONES` |

***

### Permissions required <a href="#permissions-required" id="permissions-required"></a>

The Google service account used by the chaos pod needs the following IAM permissions on the target project.

```json
{
  "permissions": [
    "compute.disks.get",
    "compute.disks.list",
    "compute.instances.get",
    "compute.instances.attachDisk",
    "compute.instances.detachDisk"
  ]
}
```

Granting `roles/compute.instanceAdmin.v1` is the simplest setup.

***

### Authentication <a href="#authentication" id="authentication"></a>

The fault supports two credential delivery models.

| Method                             | When to use it                                                                 | How to configure                                                                                                                                    |
| ---------------------------------- | ------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------- |
| Harness Secret Manager File Secret | Chaos infrastructure runs outside GKE, or you want explicit static credentials | Upload the GCP service account JSON key as a **File Secret** in Harness Secret Manager and reference its identifier via `GCP_AUTHENTICATION_SECRET` |
| Workload Identity                  | Chaos infrastructure runs on GKE with Workload Identity enabled                | Bind a Google service account to the chaos infra Kubernetes service account; no tunable changes required                                            |

Go to [Creating secrets for GCP experiments](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/gcp/security-configurations/prepare-secret-for-gcp.md) to read the secret format.

***

### Fault tunables <a href="#fault-tunables" id="fault-tunables"></a>

Configure the following fault parameters when you add GCP VM disk loss by label to an experiment in Chaos Studio. Defaults are shown for reference.

**Required parameters**

| Tunable             | Description                                                                        | Default    |
| ------------------- | ---------------------------------------------------------------------------------- | ---------- |
| `GCP_PROJECT_ID`    | ID of the GCP project that contains the disks.                                     | (required) |
| `ZONES`             | Comma-separated list of zones to scan for the label.                               | (required) |
| `DISK_VOLUME_LABEL` | Label that selects the target disks (format `key:value`, for example `tier:data`). | (required) |

**Chaos parameters**

| Tunable                    | Description                                                                                                                                                                                                      | Default    |
| -------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- |
| `TOTAL_CHAOS_DURATION`     | Total duration of the fault in seconds. The disks stay detached for this period.                                                                                                                                 | `30`       |
| `CHAOS_INTERVAL`           | Delay in seconds between successive iterations when running for more than one cycle.                                                                                                                             | `30`       |
| `DISK_AFFECTED_PERCENTAGE` | Percentage of label-matching disks to detach (0-100). Empty defaults to all matches.                                                                                                                             | `""`       |
| `SEQUENCE`                 | Order in which selected disks are detached: `parallel` or `serial`.                                                                                                                                              | `parallel` |
| `RAMP_TIME`                | Wait period in seconds before and after the fault. Go to [ramp time](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/common-tunables-for-all-faults.md#ramp-time) to read how it is applied. | `0`        |

**Authentication**

| Tunable                     | Description                                                                                                                                            | Default |
| --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | ------- |
| `GCP_AUTHENTICATION_SECRET` | Identifier of the **File Secret in Harness Secret Manager** that contains the GCP service account JSON key. Not required when using Workload Identity. | `""`    |

Tunables that apply to every fault are documented in [common tunables for all faults](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/common-tunables-for-all-faults.md).

***

### Fault execution in brief <a href="#fault-execution-in-brief" id="fault-execution-in-brief"></a>

Lists non-boot Compute Engine disks across `ZONES` (in `GCP_PROJECT_ID`) that match `DISK_VOLUME_LABEL`, picks `DISK_AFFECTED_PERCENTAGE` of them, calls `instances.detachDisk` for each on its attached VM, waits for `TOTAL_CHAOS_DURATION`, then calls `instances.attachDisk` to reattach with the same device path.

***

### Expected behavior during fault execution <a href="#expected-behavior-during-fault-execution" id="expected-behavior-during-fault-execution"></a>

* Each affected VM loses the matching disk for the duration; the device entry disappears under `/dev/disk/by-id/`.
* Filesystems mounted from the detached disk go into an I/O error state; processes accessing them block or return `EIO`.
* After the duration ends, the disks reattach with the same device name.
* Cloud Monitoring metrics (`disk/read_ops_count`, `disk/write_ops_count`) drop to zero on the affected devices for the duration.

{% hint style="info" %}
**WHEN THE FAULT ENDS**

The chaos pod reattaches each disk to its original VM on the same device path. Whether the filesystem remounts automatically depends on fstab options and application I/O retry behavior.
{% endhint %}

#### Signals to watch <a href="#signals-to-watch" id="signals-to-watch"></a>

Attach [resilience probes](/resilience-testing/chaos-engineering/use-chaos-engineering/probes.md) to assert each layer:

* **Disk count by label:** Use a [command probe](/resilience-testing/chaos-testing/probes/command-probe.md) running `gcloud compute disks list --filter='labels.<key>=<value>' --format='value(users)'` and assert some had an empty `users` field during the chaos window.
* **Application I/O errors:** Use a [Prometheus probe](/resilience-testing/chaos-engineering/use-chaos-engineering/probes/apm-probes.md) on application error counters.
* **End-to-end availability:** Use an [HTTP probe](/resilience-testing/chaos-engineering/use-chaos-engineering/probes/http-probe.md) on a user-visible endpoint that exercises a write path.

***

### Verify the fault execution effect <a href="#verify-the-fault-execution-effect" id="verify-the-fault-execution-effect"></a>

1. **List affected disks.**

   ```bash
   gcloud compute disks list \
     --filter="labels.<key>=<value>" \
     --format="table(name,zone,users)"
   ```

   You should see empty `users` rows during the chaos window and populated `users` rows afterwards.
2. **Inspect Cloud Monitoring.**

   Use the Cloud Console to confirm `disk/read_ops_count` dropped on the affected devices.
3. **Inspect audit logs.**

   ```bash
   gcloud logging read 'protoPayload.methodName=v1.compute.instances.detachDisk' --limit=20
   ```

***

### Recovery and cleanup <a href="#recovery-and-cleanup" id="recovery-and-cleanup"></a>

* **End of duration:** The chaos pod calls `instances.attachDisk` to reattach every detached disk.
* **Abort the experiment:** Stopping the experiment from Chaos Studio also reattaches the disks.
* **Manual recovery:** Run `gcloud compute instances attach-disk <vm-name> --disk=<disk-name> --device-name=<device-name> --zone=<zone>` for any disk that stayed detached.
* **Workload recovery:** Filesystems may need `mount -a` or an application restart depending on retry behavior.

***

### Limitations <a href="#limitations" id="limitations"></a>

* **Non-boot only:** Boot disks are not supported.
* **Same-project targeting:** A single experiment targets one `GCP_PROJECT_ID`.
* **Label scoped to listed zones:** Disks in zones not listed in `ZONES` are not considered.
* **Percentage rounding:** `DISK_AFFECTED_PERCENTAGE` rounds down; empty selects all matches.
* **No mid-flight retarget:** Label or percentage cannot be changed during the chaos window.

***

### Troubleshooting <a href="#troubleshooting" id="troubleshooting"></a>

<details>

<summary>GCP VM disk loss by label fails with no matching disks in Harness Chaos Engineering</summary>

Confirm DISK\_VOLUME\_LABEL is formatted key:value and the zones in ZONES contain non-boot disks with that label. List them with gcloud compute disks list --filter='labels.=' --format='table(name,zone,users)'.

</details>

<details>

<summary>GCP VM disk loss by label fails with PermissionDenied</summary>

The service account used by the chaos pod is missing compute.disks.list, compute.instances.attachDisk, or compute.instances.detachDisk. Grant roles/compute.instanceAdmin.v1 on the target project.

</details>

<details>

<summary>Disks reattached but filesystems did not remount</summary>

The filesystem may not auto-remount depending on fstab options. SSH into each affected VM and run sudo mount -a (or remount the specific device). For application-level recovery, restart the dependent service.

</details>

***

### Related faults <a href="#related-faults" id="related-faults"></a>

* [GCP VM disk loss](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/gcp/gcp-vm-disk-loss.md): Detach named disks instead of label-selected ones.
* [GCP VM instance stop by label](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/gcp/gcp-vm-instance-stop-by-label.md): Stop the VMs instead of detaching disks.