> For the complete documentation index, see [llms.txt](https://developer.harness.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://developer.harness.io/resilience-testing/chaos-engineering/faults/chaos-fault-categories/azure/azure-aks-node-down.md).

# Azure AKS node down

Azure AKS node down is an Azure chaos fault that resolves the VMSS instances backing the AKS cluster `AKS_CLUSTER_NAME` in `AKS_RESOURCE_GROUP` (limited to `TARGET_NODE_POOL_NAMES` and `TARGET_ZONES`), selects `NODE_AFFECTED_PERCENTAGE` of them, deallocates them for `TOTAL_CHAOS_DURATION` seconds, then starts them again.

Use this fault to test how a workload behaves when AKS nodes disappear: whether the scheduler reschedules pods inside the alerting SLA, whether load balancers fail traffic over, whether ephemeral storage is preserved correctly, and whether monitoring detects the node failures within the alerting SLA.

{% hint style="info" %}
**RUN YOUR FIRST EXPERIMENT**

If you have not configured the chaos infrastructure yet, go to [Quickstart](/resilience-testing/chaos-engineering/new-to-chaos-engineering/quickstart.md) to install the chaos infrastructure and run an experiment end to end.
{% endhint %}

***

### Use cases <a href="#use-cases" id="use-cases"></a>

Run this fault when you want to answer concrete questions like:

* **Pod rescheduling:** When a percentage of nodes disappear, does the scheduler find homes for affected pods inside the SLA?
* **PodDisruptionBudgets:** Do PDBs honor the budget across the node-down window without serving fewer replicas than required?
* **Cluster autoscaler:** Does the cluster autoscaler add capacity in time when pods become unschedulable?
* **Zone-level resilience:** Targeting a single zone tests how the workload survives losing an entire AZ's worth of nodes.

***

### Prerequisites <a href="#prerequisites" id="prerequisites"></a>

* **Kubernetes version:** 1.21 or later for the chaos infrastructure cluster.
* **Target AKS cluster:** `AKS_CLUSTER_NAME` exists in `AKS_RESOURCE_GROUP`. `TARGET_NODE_POOL_NAMES` and `TARGET_ZONES` (empty means all) refer to existing node pools and zones.
* **Azure credentials available:** Service principal File Secret, workload identity, or managed identity on the AKS node pool that hosts the chaos infrastructure.
* **RBAC granted:** The principal includes the role listed below.

***

### Supported environments <a href="#supported-environments" id="supported-environments"></a>

| Platform                                | Support status                                  |
| --------------------------------------- | ----------------------------------------------- |
| AKS with VMSS node pools                | Supported                                       |
| AKS Virtual Machines (Standalone)       | Not supported                                   |
| AKS Automatic / Karpenter-managed nodes | Supported (cluster autoscaler decides recovery) |
| Multi-zone AKS clusters                 | Supported via `TARGET_ZONES`                    |

***

### Permissions required <a href="#permissions-required" id="permissions-required"></a>

The Azure principal used by the chaos pod needs the following role on the AKS resource group (or the auto-generated `MC_<rg>_<cluster>_<region>` resource group that owns the VMSS).

**Recommended built-in role:** `Virtual Machine Contributor` on the node resource group (`MC_<rg>_<cluster>_<region>`).

**Custom role (minimum actions):**

```json
{
  "Name": "Harness Chaos AKS Node Down",
  "Actions": [
    "Microsoft.ContainerService/managedClusters/read",
    "Microsoft.ContainerService/managedClusters/agentPools/read",
    "Microsoft.Compute/virtualMachineScaleSets/read",
    "Microsoft.Compute/virtualMachineScaleSets/virtualMachines/read",
    "Microsoft.Compute/virtualMachineScaleSets/virtualMachines/start/action",
    "Microsoft.Compute/virtualMachineScaleSets/virtualMachines/deallocate/action"
  ],
  "AssignableScopes": ["/subscriptions/<SUBSCRIPTION_ID>"]
}
```

Go to [Azure fault permissions](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/azure/security-configurations/fault-permissions.md) to read the full permission catalog.

***

### Authentication <a href="#authentication" id="authentication"></a>

Go to [Azure authentication methods](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/azure/security-configurations/azure-authentication-methods.md) to set up Service principal, Workload identity, or Managed identity.

***

### Fault tunables <a href="#fault-tunables" id="fault-tunables"></a>

Configure the following fault parameters when you add Azure AKS node down to an experiment in Chaos Studio. Defaults are shown for reference.

**Required parameters**

| Tunable              | Description                                   | Default    |
| -------------------- | --------------------------------------------- | ---------- |
| `AKS_CLUSTER_NAME`   | Name of the AKS cluster.                      | (required) |
| `AKS_RESOURCE_GROUP` | Resource group that contains the AKS cluster. | (required) |

**Selection parameters**

| Tunable                    | Description                                                                                      | Default |
| -------------------------- | ------------------------------------------------------------------------------------------------ | ------- |
| `TARGET_NODE_POOL_NAMES`   | Comma-separated list of node pool names to target. Empty means all node pools in the cluster.    | `""`    |
| `TARGET_ZONES`             | Comma-separated list of availability zones to target (for example `1,2`). Empty means all zones. | `""`    |
| `NODE_AFFECTED_PERCENTAGE` | Percentage of matching nodes to deallocate (0-100). `0` defaults to all matches.                 | `0`     |

**Chaos parameters**

| Tunable                | Description                                                                                                                                                                                                      | Default    |
| ---------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- |
| `TOTAL_CHAOS_DURATION` | Total duration of the fault in seconds. Nodes stay deallocated for this period.                                                                                                                                  | `30`       |
| `CHAOS_INTERVAL`       | Delay in seconds between successive iterations when running for more than one cycle.                                                                                                                             | `30`       |
| `SEQUENCE`             | Order in which multiple nodes are deallocated: `parallel` or `serial`.                                                                                                                                           | `parallel` |
| `RAMP_TIME`            | Wait period in seconds before and after the fault. Go to [ramp time](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/common-tunables-for-all-faults.md#ramp-time) to read how it is applied. | `0`        |

**Authentication**

| Tunable                       | Description                                                                                           | Default |
| ----------------------------- | ----------------------------------------------------------------------------------------------------- | ------- |
| `AZURE_AUTHENTICATION_SECRET` | Identifier of the **File Secret in Harness Secret Manager** that contains the service principal JSON. | `""`    |

Tunables that apply to every fault are documented in [common tunables for all faults](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/common-tunables-for-all-faults.md).

***

### Fault execution in brief <a href="#fault-execution-in-brief" id="fault-execution-in-brief"></a>

Lists VMSS instances backing the AKS cluster in `AKS_RESOURCE_GROUP`, filters by `TARGET_NODE_POOL_NAMES` and `TARGET_ZONES`, picks `NODE_AFFECTED_PERCENTAGE` of the matches, deallocates them via the VMSS API for `TOTAL_CHAOS_DURATION`, then starts them again.

***

### Expected behavior during fault execution <a href="#expected-behavior-during-fault-execution" id="expected-behavior-during-fault-execution"></a>

* The selected AKS nodes transition `running` → `deallocated` and stay there for the duration.
* Pods on the affected nodes go to `NotReady`/`Unknown`; the scheduler reschedules them onto healthy nodes (within PDB constraints).
* LoadBalancer Services drop the affected backends and shift traffic.
* After the duration ends, the nodes start back, the kubelet re-registers, and pods may rebalance.

{% hint style="info" %}
**WHEN THE FAULT ENDS**

The chaos pod calls `start` on every deallocated VMSS instance. AKS marks the node `Ready` once the kubelet checks in.
{% endhint %}

#### Signals to watch <a href="#signals-to-watch" id="signals-to-watch"></a>

* **Node readiness:** Use a [command probe](/resilience-testing/chaos-testing/probes/command-probe.md) running `kubectl get nodes` and assert ready-count is consistent with PDBs.
* **Application availability:** Use an [HTTP probe](/resilience-testing/chaos-engineering/use-chaos-engineering/probes/http-probe.md) on a user-visible endpoint.

***

### Verify the fault execution effect <a href="#verify-the-fault-execution-effect" id="verify-the-fault-execution-effect"></a>

1. **List VMSS instances and their state.**

   ```bash
   az vmss list-instances --resource-group MC_<rg>_<cluster>_<region> --name <vmss> \
     --query "[].{name:name,powerState:instanceView.statuses[?starts_with(code, 'PowerState/')].code}"
   ```
2. **Inspect AKS nodes from kubectl.**

   ```bash
   kubectl get nodes -o wide
   ```

   Affected nodes should appear `NotReady` during the chaos window and `Ready` afterwards.

***

### Recovery and cleanup <a href="#recovery-and-cleanup" id="recovery-and-cleanup"></a>

* **End of duration:** The chaos pod starts every deallocated VMSS instance.
* **Abort the experiment:** Stopping the experiment from Chaos Studio also starts them.
* **Manual recovery:** Run `az vmss start --resource-group <node-rg> --name <vmss> --instance-ids <id>` for any instance that stayed deallocated.

***

### Limitations <a href="#limitations" id="limitations"></a>

* **VMSS node pools only:** AKS Virtual Machines (standalone) node pools are not supported.
* **Single cluster per run:** Each fault run targets one AKS cluster.
* **Cluster autoscaler interaction:** The autoscaler may interpret the deallocation as capacity loss; this is the intended behavior under test.
* **Zone selection:** `TARGET_ZONES` values must match the zone labels reported by `kubectl get nodes --label-columns=topology.kubernetes.io/zone`.

***

### Troubleshooting <a href="#troubleshooting" id="troubleshooting"></a>

<details>

<summary>Azure AKS node down fails with AuthorizationFailed in Harness Chaos Engineering</summary>

The principal is missing VMSS deallocate/start actions on the node resource group (MC\_\_\_). Assign Virtual Machine Contributor on that resource group, not the AKS resource group.

</details>

<details>

<summary>Azure AKS node down selected zero nodes</summary>

Verify TARGET\_NODE\_POOL\_NAMES matches the actual node pool names with az aks nodepool list -g --cluster-name --query '\[].name'. Verify TARGET\_ZONES values match kubectl get nodes --label-columns=topology.kubernetes.io/zone.

</details>

<details>

<summary>Nodes did not become Ready after the experiment ended</summary>

Allow up to 5 minutes for kubelet to re-register after the start call. If a node stays NotReady, inspect Azure activity log for failed start actions and run az vmss start manually.

</details>

***

### Related faults <a href="#related-faults" id="related-faults"></a>

* [Azure instance stop](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/azure/azure-instance-stop.md): Stop a named VM instead of a percentage of AKS nodes.
* [Node CPU hog](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/kubernetes/node/node-cpu-hog.md): Stress CPU on a node instead of deallocating it.
* [Node drain](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/kubernetes/node/node-drain.md): Cordon and drain a node from inside Kubernetes.