> For the complete documentation index, see [llms.txt](https://developer.harness.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://developer.harness.io/resilience-testing/chaos-engineering/faults/chaos-fault-categories/kubernetes/node/node-network-loss.md).

# Node network loss

Node network loss is a Kubernetes node-level chaos fault that drops a configurable percentage of packets on the network path serving a target node for a configurable duration. The fault affects every workload that shares the node's network stack: application pods, DaemonSets, kube-proxy, CNI agents, and the kubelet itself.

Use this fault to simulate a node becoming a lossy network neighbor: a flapping NIC, an overloaded top-of-rack switch, a misconfigured CNI overlay, or an unhealthy underlay route.

{% hint style="info" %}
**RUN YOUR FIRST EXPERIMENT**

If you have not configured the chaos infrastructure yet, go to [Quickstart](/resilience-testing/chaos-engineering/new-to-chaos-engineering/quickstart.md) to install the chaos infrastructure and run an experiment end to end.
{% endhint %}

***

### Use cases <a href="#use-cases" id="use-cases"></a>

Run this fault when you want to answer concrete questions like:

* **Control-plane resilience:** Does the cluster detect, taint, and reschedule workloads from a partially or fully unreachable node within your service-level objective?
* **Stateful workload behavior:** When a node hosting a database or stateful replica (for example PostgreSQL, Kafka, Cassandra, etcd) becomes lossy, do leader election, quorum, and failover happen as expected, and do clients converge before timeouts cascade?
* **Load balancer and traffic routing:** Do your service-mesh or external load balancer health checks evict the degraded node's endpoints quickly enough, and how long does residual traffic continue to hit it?
* **Client retry and timeout configuration:** Are application HTTP and gRPC clients tuned with sensible deadlines, retries, and hedging, so that calls into a degraded node do not exhaust caller resources?
* **DaemonSet-delivered platform services:** When a node hosting CoreDNS, log shippers, or a CNI agent loses packets, does it cause regressions visible from other nodes (for example, cluster-wide DNS resolution delays)?
* **Observability and alerting:** Do your dashboards and alerts surface partial network failures, or does the failure hide as elevated error rates without a clear root cause?

***

### Prerequisites <a href="#prerequisites" id="prerequisites"></a>

* **Kubernetes version:** 1.21 or later. Go to [What's supported](/resilience-testing/chaos-engineering/new-to-chaos-engineering/whats-supported.md) to confirm distribution support.
* **Standard Linux networking modules:** Target nodes run a Linux kernel that includes the standard networking modules (present by default on most distributions). On minimal or hardened images, your platform team may need to install `kernel-modules-extra` (RHEL-family) or the equivalent package.
* **Privileged pods allowed:** The cluster lets you schedule privileged pods with `NET_ADMIN` and `SYS_ADMIN` capabilities in the chaos namespace. GKE Autopilot supports this fault but requires the one-time setup in [Chaos on GKE Autopilot](/resilience-testing/chaos-engineering/use-chaos-engineering/gke-autopilot.md); other locked-down distributions may need similar exemptions.
* **Node readiness:** Target nodes are in `Ready` state before the fault is launched. The fault reports a precheck failure otherwise.
* **Chaos infrastructure isolation:** The target nodes are not single points of failure for the chaos infrastructure itself. If chaos control-plane pods are scheduled on the node you are about to partition, the experiment loses the ability to observe and roll back.

***

### Supported environments <a href="#supported-environments" id="supported-environments"></a>

| Platform                                 | Support status                                                                                                 |
| ---------------------------------------- | -------------------------------------------------------------------------------------------------------------- |
| Amazon EKS                               | Supported                                                                                                      |
| Azure AKS                                | Supported                                                                                                      |
| Google GKE                               | Supported                                                                                                      |
| Red Hat OpenShift                        | Supported                                                                                                      |
| Rancher                                  | Supported                                                                                                      |
| VMware Tanzu                             | Supported                                                                                                      |
| Self-managed Kubernetes (CNCF-certified) | Supported                                                                                                      |
| GKE Autopilot                            | Supported with [Autopilot setup](/resilience-testing/chaos-engineering/use-chaos-engineering/gke-autopilot.md) |

***

### Permissions required <a href="#permissions-required" id="permissions-required"></a>

The fault runs under the chaos infrastructure's service account. The account must be able to perform the following operations against the target cluster.

| Resource (`apiGroup`) | Verbs                                                                    | Why it is needed                                            |
| --------------------- | ------------------------------------------------------------------------ | ----------------------------------------------------------- |
| `pods` (`""`)         | `get`, `list`, `create`, `delete`, `deletecollection`, `patch`, `update` | Run the chaos pod that injects the fault on the target node |
| `pods/log` (`""`)     | `get`, `list`, `watch`                                                   | Stream chaos pod logs for status and debugging              |
| `events` (`""`)       | `get`, `list`, `create`, `patch`, `update`                               | Record fault progress as Kubernetes events                  |
| `nodes` (`""`)        | `get`, `list`                                                            | Discover target nodes and validate selectors                |
| `jobs` (`batch`)      | `get`, `list`, `create`, `delete`, `deletecollection`                    | Run the chaos job that drives the fault                     |

The default Harness chaos infrastructure service account already includes these permissions. You only need to extend it if you are running with a restricted scope, for example a namespace-scoped install where you have removed cluster-wide node access.

***

### Fault tunables <a href="#fault-tunables" id="fault-tunables"></a>

Configure the following fault parameters when you add Node network loss to an experiment in Chaos Studio. Defaults are shown for reference.

**Chaos parameters**

| Tunable                          | Description                                                                             | Default |
| -------------------------------- | --------------------------------------------------------------------------------------- | ------- |
| `NETWORK_PACKET_LOSS_PERCENTAGE` | Percentage of packets dropped on the shaped path. Accepts an integer from `0` to `100`. | `100`   |
| `TOTAL_CHAOS_DURATION`           | Duration of the fault in seconds.                                                       | `60`    |

**Scoping**

| Tunable             | Description                                                                                       | Default |
| ------------------- | ------------------------------------------------------------------------------------------------- | ------- |
| `DESTINATION_IPS`   | Comma-separated list of IP addresses or CIDRs to scope the loss to. Empty means all destinations. | `""`    |
| `DESTINATION_HOSTS` | Comma-separated list of DNS names or FQDNs to scope the loss to. Empty means all destinations.    | `""`    |
| `NETWORK_INTERFACE` | Interface on which `tc` rules are applied.                                                        | `eth0`  |

**Targeting**

| Tunable                     | Description                                                                                                                                                                                                                               | Default    |
| --------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------- |
| `TARGET_NODES`              | Comma-separated list of node names to target. Go to [target multiple nodes](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/kubernetes/node/common-tunables-for-node-faults.md#target-multiple-nodes) to read more.   | `""`       |
| `NODE_LABEL`                | Label selector for choosing target nodes. Go to [target nodes with labels](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/kubernetes/node/common-tunables-for-node-faults.md#target-nodes-with-labels) to read more. | `""`       |
| `NODES_AFFECTED_PERCENTAGE` | Percentage of nodes (matching the selector) to target. `0` means one node.                                                                                                                                                                | `0`        |
| `SEQUENCE`                  | When multiple nodes are targeted, inject `parallel` (all at once) or `serial` (one after another).                                                                                                                                        | `parallel` |

**Runtime and helper**

| Tunable     | Description                                                                                                                                                                                                      | Default |
| ----------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- |
| `RAMP_TIME` | Wait period in seconds before and after the fault. Go to [ramp time](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/common-tunables-for-all-faults.md#ramp-time) to read how it is applied. | `0`     |

Tunables that apply to every chaos fault are documented in [common tunables for all faults](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/common-tunables-for-all-faults.md).

{% hint style="warning" %}
**DEFAULT BLAST RADIUS IS THE ENTIRE NODE**

When `DESTINATION_IPS` and `DESTINATION_HOSTS` are both empty, every flow leaving the node is affected, including kubelet to apiserver. That is the right setting for simulating a true node partition, but it is a large blast radius. For experiments that test a specific service-to-service path, set `DESTINATION_HOSTS` to the FQDN of the target service to keep the rest of the node healthy.
{% endhint %}

***

### Fault execution in brief <a href="#fault-execution-in-brief" id="fault-execution-in-brief"></a>

Configures the target node's network interface to drop a specified percentage of packets for the configured duration, optionally scoped to certain destination IPs or hosts so other traffic passes through unaffected.

Because loss is applied at the node's network path, every connection passing through it is affected unless you scope it. The traffic paths most commonly impacted are:

| Path                      | What is affected                                                                                                       |
| ------------------------- | ---------------------------------------------------------------------------------------------------------------------- |
| Pod to kube-apiserver     | Kubelet heartbeats and node lease renewals, in-cluster Kubernetes API calls made from any pod on this node             |
| Pod to pod across nodes   | East-west service traffic for any pod on this node, including DaemonSets such as CoreDNS, log shippers, and CNI agents |
| Pod to external endpoints | Egress to image registries, third-party APIs, managed databases, and SaaS providers                                    |
| Node to node              | Overlay and underlay traffic between nodes; etcd peer traffic if the target is a control-plane node                    |

Pods on the same node that talk to each other over `localhost` or the node's CNI bridge are typically unaffected, since that traffic does not traverse the shaped path. This depends on your CNI: bridge-mode CNIs (for example `kindnet`, `flannel` with VXLAN, default Calico) keep intra-node traffic local, while some overlay configurations hairpin through the host NIC and are affected.

***

### Expected behavior during fault execution <a href="#expected-behavior-during-fault-execution" id="expected-behavior-during-fault-execution"></a>

At 100% loss (the default), the node behaves as if it has been network-partitioned from the rest of the cluster. The controller-manager and the taint manager drive the following sequence while the fault is active:

* After the `node-monitor-grace-period` elapses without a successful node lease renewal (default 40 seconds, raised to 50 seconds in Kubernetes 1.32), the controller-manager flips the node's Ready condition to `Unknown` and applies the `node.kubernetes.io/unreachable:NoExecute` taint.
* Pods on the node tolerate that taint for `tolerationSeconds: 300` by default (set by the `DefaultTolerationSeconds` admission controller). After that 5-minute window, the taint manager evicts them.
* `Deployment` pods are recreated on other Ready nodes. `StatefulSet` pods remain `Terminating` because the StatefulSet controller will not recreate a pod that still exists with the same identity; you must force-delete it to reclaim the slot.
* Pods bound to local persistent volumes or `hostPath` storage cannot reschedule and remain `Pending` with a volume node-affinity conflict until the node recovers.
* On the node itself, kubelet keeps the existing containers running. Liveness probes execute locally and may still restart containers.

At lower loss percentages (for example, 30 to 60%), most TCP connections survive but throughput collapses and tail latency spikes. This is the more interesting regime for testing application-level retries, circuit breakers, and timeout tuning.

{% hint style="info" %}
**EXACT TIMINGS VARY BY CLUSTER**

Both `node-monitor-grace-period` and the default toleration seconds are kube-controller-manager flags. Managed Kubernetes providers and hardened clusters often tune them. Go to [Kubernetes node status](https://kubernetes.io/docs/reference/node/node-status/) to read how the controller-manager observes nodes, and use the numbers above as defaults rather than guarantees.
{% endhint %}

#### Signals to watch <a href="#signals-to-watch" id="signals-to-watch"></a>

A useful experiment captures signals from three layers. Attach [resilience probes](/resilience-testing/chaos-engineering/use-chaos-engineering/probes.md) to assert each layer automatically:

* **Cluster state:** Run `kubectl get node <name> -w` to confirm the `NotReady` transition, and `kubectl get pods -o wide --field-selector spec.nodeName=<name> -w` to track eviction and reschedule. Use a [Kubernetes probe](/resilience-testing/chaos-testing/probes/k8s-probe.md) or the [node status check](/resilience-testing/chaos-engineering/use-chaos-engineering/probes/probe-templates/kubernetes/node-status-check.md) template to validate these conditions.
* **Application service-level indicators:** Watch error rate, p99 latency, and saturation for the workloads that route to or through the affected node. The signal that matters is whether callers degraded gracefully or the failure amplified. Use an [HTTP probe](/resilience-testing/chaos-engineering/use-chaos-engineering/probes/http-probe.md) for direct endpoint health.
* **Platform signals:** Service-mesh outlier statistics, load balancer health-check states, and alert firings. Use a [Prometheus probe](/resilience-testing/chaos-engineering/use-chaos-engineering/probes/apm-probes.md) or an [APM probe](/resilience-testing/chaos-engineering/use-chaos-engineering/probes/apm-probes.md) to fail the experiment when your platform misses the incident.

***

### Verify the fault execution effect <a href="#verify-the-fault-execution-effect" id="verify-the-fault-execution-effect"></a>

While the experiment is running, confirm loss is reaching the node:

1. **Confirm loss from outside the node.** From another pod (on a different node):

   ```bash
   kubectl run -it --rm netshoot --image=nicolaka/netshoot --restart=Never -- \
     ping -c 50 -W 1 <target-pod-ip-or-svc>
   ```

   Loss should be close to the configured `NETWORK_PACKET_LOSS_PERCENTAGE`. Tail latency from successful packets stays close to baseline; that is the signature that distinguishes loss from latency injection.
2. **Watch node and pod state.**

   ```bash
   kubectl get node <target-node> -w
   kubectl get pods --field-selector spec.nodeName=<target-node> -o wide -w
   ```

   At 100% loss, expect the node to transition to `NotReady` within roughly one minute.

***

### Recovery and cleanup <a href="#recovery-and-cleanup" id="recovery-and-cleanup"></a>

* **End of duration:** When `TOTAL_CHAOS_DURATION` elapses, the loss configuration is removed automatically and packet flow returns to baseline within a few seconds. There is no node reboot.
* **Node Ready state:** Once the kubelet successfully renews its lease, the controller-manager removes the `node.kubernetes.io/unreachable:NoExecute` taint and the node transitions back to `Ready`. New pods can be scheduled on it again.
* **Evicted Deployment pods:** Pods that were evicted during the fault are not automatically rescheduled back to the recovered node. They remain on the replacement nodes the scheduler placed them on.
* **Terminating StatefulSet pods:** Force-delete to release the StatefulSet identity slot:

  ```bash
  kubectl delete pod <pod-name> -n <namespace> --force --grace-period=0
  ```
* **If automated cleanup did not complete:** Reboot the affected node (or have your admin reset its network configuration). The fault does not persist across a node reboot.
* **Abort the experiment early:** Stop the experiment from the Harness Chaos Studio. Cleanup runs before the chaos pod exits.

***

### Limitations <a href="#limitations" id="limitations"></a>

This fault is not appropriate in the following scenarios:

* **Serverless Kubernetes (EKS Fargate, ACI virtual nodes):** These platforms do not expose real nodes or allow the privileged access this fault needs. GKE Autopilot is supported once the one-time setup in [Chaos on GKE Autopilot](/resilience-testing/chaos-engineering/use-chaos-engineering/gke-autopilot.md) is in place.
* **Windows nodes:** This fault is supported on Linux nodes only. Use the equivalent [Windows network loss](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/windows/windows-network-loss.md) on Windows-only workloads.
* **Single-node clusters or co-located chaos infrastructure:** If the chaos infrastructure pods live on the node you are about to partition, the experiment loses observability and may fail to roll back. Schedule chaos infrastructure on a node outside the blast radius.
* **Pods using `hostNetwork: true`:** These bypass per-pod network namespaces. Loss is still applied to the node, but observed behavior depends on how the pod uses the host stack.
* **Hardened or stripped kernels:** Some custom distribution kernels omit the networking modules this fault depends on, and the fault fails immediately on those nodes. On RHEL-family hosts, install `kernel-modules-extra` and reboot to restore the missing modules.

***

### Troubleshooting <a href="#troubleshooting" id="troubleshooting"></a>

<details>

<summary>Node network loss experiment stays Pending or never starts in Harness Chaos Engineering</summary>

Inspect the chaos pods in the experiment namespace with kubectl describe pod -n . The most common causes are taints on the target node, insufficient CPU or memory on the node, or a PodSecurity admission policy blocking privileged pods. Add the required tolerations to the experiment, free resources on the node, or run the experiment in a namespace with privileged Pod Security level.

</details>

<details>

<summary>Node network loss fails immediately on hardened nodes or with an unknown interface</summary>

Either the required Linux networking kernel modules are missing on the target node, or the NETWORK\_INTERFACE value does not match a real interface on the node. SSH to the node and run ip -br link to list interfaces. On RHEL-family hosts install kernel-modules-extra and reboot to restore the missing modules; otherwise set NETWORK\_INTERFACE to the actual interface name (for example ens5 on EKS, eth0 on most others).

</details>

<details>

<summary>Node network loss fault runs but no packet loss is observed on the target node</summary>

Three common causes: (1) DESTINATION\_HOSTS resolves to IPs not on the shaped path so loss never matches, (2) the workload you are probing routes over a different interface than NETWORK\_INTERFACE, or (3) traffic stays intra-node and never crosses the shaped interface. Probe loss with ping from a pod on a different node to confirm whether matched flows are degrading.

</details>

<details>

<summary>Node remains NotReady after node-network-loss completes in Harness Chaos Engineering</summary>

Automated cleanup likely did not complete. Reboot the affected node to reset its network state. The node returns to Ready once the kubelet successfully renews its lease. If this recurs, capture the chaos pod logs from the experiment namespace and share them with Harness support.

</details>

***

### Related faults <a href="#related-faults" id="related-faults"></a>

* [Node network latency](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/kubernetes/node/node-network-latency.md): Same delivery mechanism, but injects delay instead of drops. Use it to test timeouts without provoking the unreachable-node path.
* [Pod network loss](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/kubernetes/pod/pod-network-loss.md): Scopes the impact to a single pod's network path rather than the whole node.
* [Node restart](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/kubernetes/node/node-restart.md) and [Node drain](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/kubernetes/node/node-drain.md): Test planned and unplanned node loss, which is the failure mode network loss eventually escalates into.
* [Common node fault tunables](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/kubernetes/node/common-tunables-for-node-faults.md): Shared environment variables for selecting target nodes across node faults.