Chaos Faults for Kubernetes
Introduction
Kubernetes faults disrupt the resources running on a Kubernetes cluster. They can be categorized into pod-level faults and node-level faults.
On Amazon EKS Fargate, only the following pod-level faults are supported:
All other Kubernetes faults require standard EC2-based worker nodes.
Kubelet service kill
Kubelet service kill makes the application unreachable on the account of the node turning unschedulable (NotReady).
Node CPU hog
Node CPU hog exhausts the CPU resources on a Kubernetes node.
Node drain
Node drain drains the node of all its resources running on it.
Node IO stress
Node IO stress causes I/O stress on the Kubernetes node.
Node memory hog
Node memory hog causes memory resource exhaustion on the Kubernetes node.
Node network latency
Node network latency introduces network latency to the Kubernetes node.
Kubelet service kill
Kubelet service kill makes the application unreachable on the account of the node turning unschedulable (NotReady).
- Kubelet service is stopped (or killed) on a node to make it unschedulable for a specific duration defined by the
TOTAL_CHAOS_DURATIONenvironment variable. - The application node goes back to normal state and services are resumed after the chaos duration.
Details
Use cases
This fault determines the resilience of an application when a node becomes unschedulable, i.e. NotReady state.Node CPU hog
Node CPU hog exhausts the CPU resources on a Kubernetes node for the period defined by the TOTAL_CHAOS_DURATION environment variable.Details
Use cases
The fault aims to verify the resiliency of applications whose replicas may be evicted on account of nodes turning unschedulable (Not Ready) or new replicas not being able to schedule due to a lack of CPU resources.
The fault causes CPU stress on the target node(s). It simulates the situation of lack of CPU for processes running on the application, which degrades their performance. It also helps verify metrics-based horizontal pod autoscaling as well as vertical autoscale, i.e. demand based CPU addition. It helps scalability of nodes based on growth beyond budgeted pods. It verifies the autopilot functionality of (cloud) managed clusters.
It benefits include verifying multi-tenant load issues (when the load increases on one container, it does not cause downtime in other containers).
Node drain
Node drain drains the node of all its resources running on it. Due to this, services running on the target node should be rescheduled to run on other nodes.Details
Use cases
Node drain fault drains all the resources running on a node. This fault determines the resilience of the application when the application replicas scheduled on a node are removed. It validates the application failover capabilities when a node suddenly becomes unavailable.
It simulates node maintenance activity (hardware refresh, OS patching, Kubernetes upgrade). It verifies resource budgeting on cluster nodes (whether request (or limit) settings honored on available nodes), and whether topology constraints are adhered to (node selectors, tolerations, zone distribution, affinity(or anti-affinity) policies) or not.
Node IO stress
Node IO stress causes I/O stress on the Kubernetes node. The amount of I/O stress is specifed as the size in percentage of the total free space available on the file system using FILESYSTEM_UTILIZATION_PERCENTAGE environment variable or in gigabytes(GB) using FILESYSTEM_UTILIZATION_BYTES environment variable. When both the values are provided, FILESYSTEM_UTILIZATION_PERCENTAGE takes precendence. It tests application resiliency on replica evictions that occur due I/O stress on the available disk space.Details
Use cases
The fault aims to verify the resilience of applications that share the disk resource for ephemeral or persistent storage purposes during high disk I/O usage.
It simulates slower disk operations by the application and nosiy neighbour problems by hogging the disk bandwidth. It also verifies the disk performance on increasing I/O threads and varying I/O block sizes. It checks if the application functions under high disk latency conditions, when I/O traffic is very high and includes large I/O blocks, and when other services monopolize the I/O disks.
Node memory hog
Node memory hog causes memory resource exhaustion on the Kubernetes node for the duration specified by the TOTAL_CHAOS_DURATION environment variable.Details
Use cases
Node memory hog causes memory resource exhaustion on the Kubernetes node. The fault aims to verify resilience of applications whose replicas may be evicted on account on nodes becoming unschedulable (Not Ready) due to lack of memory resources.
It simulates the situation of memory leaks in the deployment of microservices, application slowness due to memory starvation, and noisy neighbour problems due to hogging. It verifies pod priority and QoS setting for eviction purposes. It also verifies application restarts on OOM kills.
Node network latency
Node network latency causes network latency on the Kubernetes node. The chaos affects the application running on the target node for a duration specified by the It simulates the scenarios of high-latency network connections, such as cross-continental data transfers, or situations where a service is communicating with a slow or overburdened external data source. The fault tests the application's ability to maintain service quality and responsiveness in sub-optimal network conditions. It verifies how well the application handles increased response times, timeouts, and the potential for increased queue lengths or backlogs due to network delays. It can also be used to confirm the correct functioning of timeout settings and retry mechanisms in applications.TOTAL_CHAOS_DURATION environment variable.Details
Use cases
Node network latency introduces a delay in the network communication of a Kubernetes node. The fault aims to verify the resilience of applications when faced with increased network response times. It is designed to test the behavior of applications under delayed network conditions, especially in systems where timely data transfer and communication are crucial.
Node network loss
Node network loss causes network loss on the Kubernetes node. The chaos affects the application running on the target node for a duration specified by the It mimics situations where the network becomes unreliable, leading to potential data transmission failures, retries, and extended communication delays. The fault challenges applications by hindering their ability to communicate with other services, data stores, or external APIs effectively. It verifies the robustness of applications in handling network interruptions, ensuring data integrity in the face of packet loss, and the effectiveness of error-handling mechanisms under network failures. Additionally, it can be used to test failover strategies, data synchronization policies, and the efficiency of retry logic in applications.TOTAL_CHAOS_DURATION environment variable.Details
Use cases
Node network loss simulates packet loss in the network communication of a Kubernetes node. The fault aims to verify the resilience of applications when faced with disrupted network communication, reflecting real-world scenarios such as unstable connections, network partitions, or infrastructure outages.
Node restart
Node restart disrupts the state of the node by restarting it. It tests deployment sanity (replica availability and uninterrupted service) and recovery workflows of the application pod.Details
Use cases
This fault determines the deployment sanity (replica availability and uninterrupted service) and recovery workflows of the application pod in the event of an unexpected node restart. It simulates loss of critical services (or node-crash). It verifies resource budgeting on cluster nodes (whether request(or limit) settings honored on available nodes), and whether topology constraints are adhered to (node selectors, tolerations, zone distribution, affinity(or anti-affinity) policies) or not.
Node taint
Node taint taints (contaminates) the node by applying the desired effect. The resources that contain the corresponding tolerations only can bypass the taints.Details
Use cases
The fault aims to verify the resiliency of applications when a certain taint is added to a node. It simulates loss of critical services (or node-crash). It verifies resource budgeting on cluster nodes (whether request(or limit) settings honored on available nodes), and whether topology constraints are adhered to (node selectors, tolerations, zone distribution, affinity(or anti-affinity) policies) or not.
Container kill
Container kill is a Kubernetes pod-level chaos fault that terminates a single container inside a target pod, leaving the pod scheduled so the kubelet restarts the container in place.
- It tests an application's deployment sanity (replica availability and uninterrupted service) and recovery workflow.
- It tests the recovery of pods that possess sidecar containers.
Details
Use cases
It tests an application's deployment sanity (replica availability and uninterrupted service) and recovery workflow when certain replicas are not available.Disk fill
Disk fill is a Kubernetes pod-level chaos fault that consumes a configurable percentage of a target container's ephemeral storage to test eviction and write-failure handling.
- It evicts the application pod if its capacity exceeds the pod's ephemeral storage limit.
- It tests the ephemeral storage limits and ensures that the parameters are sufficient.
- It evaluates the application's resilience to disk stress (or replica) evictions.
Details
Use cases
This fault tests the ephemeral storage limits and determines the resilience of the application to unexpected storage exhaustions.FS fill
FS fill is a Kubernetes pod-level chaos fault that writes a configurable amount of data into a specific path inside a target container to test mounted-volume capacity and write-failure handling.Use cases
Pod API block
Pod API block is a Kubernetes pod-level chaos fault that blocks selected API requests or responses on a target pod using path, method, header, query parameter, and source or destination filters (with HTTPS support via supplied TLS certificates).Use cases
POST/PUT/DELETE.
Pod API latency
Pod API latency is a Kubernetes pod-level chaos fault that adds a configurable delay to selected API calls on a target pod using path, method, header, query, and source or destination filters (with HTTPS support via supplied TLS certificates).Use cases
Pod API modify body
Pod API modify body is a Kubernetes pod-level chaos fault that overwrites request or response bodies on selected API calls of a target pod using path, method, header, query, and source or destination filters (with HTTPS support via supplied TLS certificates).Use cases
Pod API modify header
Pod API modify header is a Kubernetes pod-level chaos fault that overrides request or response headers on selected API calls of a target pod using path, method, query, and source or destination filters (with HTTPS support via supplied TLS certificates).Use cases
Authorization on one path to validate clean 401 handling.Cache-Control directives on a specific endpoint's responses.
Pod API modify response custom
Pod API modify response custom is a Kubernetes pod-level chaos fault that combines status code, header, and body modifications on selected API calls of a target pod in a single experiment, with path, method, query, and source or destination filters (with HTTPS support via supplied TLS certificates).Use cases
429 + Retry-After + JSON body).401, WWW-Authenticate, and an error body.200 plus a body missing an expected field.
Pod API status code
Pod API status code is a Kubernetes pod-level chaos fault that overrides the HTTP status code returned by selected API calls of a target pod using path, method, header, query, and source or destination filters (with HTTPS support via supplied TLS certificates).Use cases
503 on /v2/checkout only).429, 503, 400).401 on the user-info endpoint.
Pod autoscaler
Pod autoscaler is a Kubernetes pod-level chaos fault that scales a target Deployment or StatefulSet to a configured replica count for a fixed duration to test cluster capacity and node autoscaling.
- It examines the node auto-scaling feature by determining whether the pods were successfully rescheduled within a specified time frame if the existing nodes are running at the specified limits.
Details
Use cases
This fault determines how an application accomodates multiple replicas of a given application pod at unexpected point in time.Pod CPU hog
Pod CPU hog is a Kubernetes pod-level chaos fault that excessively consumes CPU resources, resulting in a significant increase in the CPU resource usage of a pod.
- Simulates a situation where an application's CPU resource usage unexpectedly spikes.
Use cases
- The fault causes CPU stress on the target pod(s). It simulates the situation of lack of CPU for processes running on the application, which degrades their performance.
- It also helps verify metrics-based horizontal pod autoscaling as well as vertical autoscale, i.e. demand based CPU addition.
- It helps scalability of nodes based on growth beyond budgeted pods.
- It verifies the autopilot functionality of (cloud) managed clusters.
- Injecting a rogue process into a target container starves the main microservice (typically pid 1) of the resources allocated to it (where limits are defined). This slows down the application traffic or exhausts the resources leading to eviction of all pods. These faults helps build immunity to such stress cases.
- Its benefits include verifying multi-tenant load issues (when the load increases on one container, it does not cause downtime in other containers).
Pod delete
Pod delete is a Kubernetes pod-level chaos fault that removes one or more pods of a target workload through the Kubernetes API to test replica availability, controller recovery, and graceful termination.
- It tests an application's deployment sanity (replica availability and uninterrupted service) and recovery workflow.
Details
Use cases
In distributed systems like Kubernetes, your application replicas may not be sufficient to manage the traffic (indicated by SLIs) when some of the replicas are unavailable due to failures. It is important to ensure that the applications have minimum number of available replicas. One of the common application failures is when the pressure on other replicas increases, and how the horizontal pod autoscaler scales based on the observed resource utilization. It is also important to understand how much time it takes for persistent volume to after rescheduling. This fault helps reproduce such a situation with forced (or graceful) pod failure on specific (or random) replicas of an application resource. It checks the deployment sanity (replica availability and uninterrupted service) and recovery workflow of the application.Pod DNS error
Pod DNS error is a Kubernetes pod-level chaos fault that fails DNS lookups from inside the target pod for a list of hostnames (or all hostnames) to test how the application handles upstream lookup failures and cluster DNS outages.Use cases
NXDOMAIN-style failures.
Pod DNS spoof
Pod DNS spoof is a Kubernetes pod-level chaos fault that redirects DNS lookups for selected hostnames inside the target pod to a different address so the application opens connections to the wrong destination.Use cases
Pod HTTP latency
Pod HTTP latency is a Kubernetes pod-level chaos fault that adds a configurable delay to HTTP responses served by a target pod on a chosen service port to test client timeouts, retries, and tail-latency budgets.Use cases
TOXICITY partial-affect probability.
Pod HTTP modify body
Pod HTTP modify body is a Kubernetes pod-level chaos fault that overwrites the HTTP response body served by a target pod (with the value of RESPONSE_BODY) to test client behavior under corrupted, empty, or unexpected payloads.Use cases
CONTENT_TYPE alongside the body.
Pod HTTP modify header
Pod HTTP modify header is a Kubernetes pod-level chaos fault that overrides HTTP request or response headers on a target pod (via HEADERS_MAP) to test resilience to missing, altered, or unexpected header values.Use cases
Authorization to validate clean auth-error handling.Cache-Control directives to expose cache-poisoning risks.X-Request-ID, traceparent) to reveal observability gaps.Content-Type on the wire.
Pod HTTP reset peer
Pod HTTP reset peer is a Kubernetes pod-level chaos fault that forcibly resets the TCP connection carrying an HTTP request to a target pod after a configurable delay to test client retry, connection-pool, and circuit-breaker behavior on abrupt disconnects.Use cases
5xx error.RST packets.
Pod HTTP status code
Pod HTTP status code is a Kubernetes pod-level chaos fault that overrides the HTTP response status code returned by a target pod (and optionally overwrites the body) to test client error handling, retry classification, and circuit-breaker behavior on specific HTTP statuses.Use cases
503, 502, and 429 responses.500s.404 on previously-cached resources.401/403 responses.
Pod IO attribute override
Pod IO attribute override rewrites file attributes returned by stat syscalls on a target container's mounted volume to test how the application reacts to changed permissions, ownership, size, or timestamps.
- It can test the application's resilience for the different values of file properties.
Details
View fault usage
It can test the application's resilience for the different values of file properties.Pod IO error
Pod IO error makes filesystem syscalls on a target container's mounted volume return a configurable error code so you can validate how the application handles failed reads, writes, and opens.
- It can test the application's resilience for the errors in i/o operations.
Details
View fault usage
It can test the application's resilience for the errors in i/o operations.Pod IO latency
Pod IO latency adds a configurable delay to filesystem syscalls against a target container's mounted volume so you can test how the application behaves under slow storage.
- It can test the application's resilience for the latency in i/o operations.
Details
View fault usage
It can test the application's resilience for the latency in i/o operations.Pod IO mistake
Pod IO mistake seeds wrong bytes into reads or writes against a target container's mounted volume so you can validate how the application detects and recovers from silent data corruption.
- It can test the application's resilience to mistakenly writing or reading invalid data from files.
Details
View fault usage
It can test the application's resilience to mistakenly writing or reading invalid data from files..Pod IO stress
Pod IO stress is a Kubernetes pod-level chaos fault that generates sustained filesystem read and write load inside a target container's mounted volume to test how the application handles disk pressure, slow IO, and ephemeral-storage exhaustion.Use cases
ENOSPC handling.
Pod JVM CPU stress
Pod JVM CPU stress drives a configurable number of CPU cores inside a target JVM to test how the application behaves when its Java process is starved of CPU.Use cases
Pod JVM method exception
Pod JVM method exception causes a specific Java method in a target JVM to throw a configurable exception on every invocation so you can test how callers handle the failure.Use cases
Pod JVM method latency
Pod JVM method latency adds a configurable delay to every invocation of a specific Java method in a target JVM so you can test how callers and dependents behave under slow methods.Use cases
Pod JVM modify return
Pod JVM modify return overrides the return value of a specific Java method in a target JVM so you can test how callers behave when a method silently returns wrong data.Use cases
Pod JVM Solace Latency
Pod JVM Solace Latency adds a configurable delay to Solace publisher or subscriber calls from a target JVM, scoped by topic or queue, to test timeout and back-pressure behavior under slow Solace messaging.Use cases
Pod JVM Solace Exception
Pod JVM Solace Exception causes Solace publisher or subscriber calls from a target JVM to throw a configurable exception on a chosen topic or queue so you can test caller error handling.Use cases
Pod JVM trigger gc
Pod JVM trigger gc forces an immediate full garbage collection in a target JVM so you can measure GC pause impact on latency, throughput, and downstream timeouts.Use cases
Pod JVM SQL Exception
Pod JVM SQL Exception causes JDBC calls from a target JVM to throw a configurable exception, scoped by table and SQL operation, so you can test how the application handles database failures.Use cases
Pod JVM Mongo Latency
Pod JVM Mongo Latency adds a configurable delay to MongoDB calls from a target JVM, scoped by database and collection, so you can test how the application behaves under slow MongoDB operations.Use cases
Pod JVM Mongo Exception
Pod JVM Mongo Exception causes MongoDB calls from a target JVM to throw a configurable exception, scoped by database and collection, so you can test how the application handles MongoDB failures.Use cases
Pod JVM Kafka Exception
Pod JVM Kafka Exception causes Kafka producer or consumer calls from a target JVM to throw a configurable exception on a chosen topic so you can test how the application handles Kafka failures.Use cases
Pod JVM Kafka Latency
Pod JVM Kafka Latency adds a configurable delay to Kafka producer or consumer calls from a target JVM on a chosen topic so you can test how the application behaves under slow Kafka messaging.Use cases
Pod JVM SQL Latency
Pod JVM SQL Latency adds a configurable delay to JDBC calls from a target JVM, scoped by table and SQL operation, so you can test connection-pool exhaustion and timeout handling under slow databases.Use cases
Pod memory hog
Pod memory hog is a Kubernetes pod-level chaos fault that consumes memory resources in excess, resulting in a significant spike in the memory usage of a pod.
- Simulates a condition where the memory usage of an application spikes up unexpectedly.
Details
Use cases
Memory usage within containers is subject to various constraints in Kubernetes. If the limits are specified in their spec, exceeding them results in termination of the container (due to OOMKill of the primary process, often pid 1). This restarts container dependng on policy specified. For containers with no limits on memory, node can be killed based on their oom_score. This results in a bigger blast radius.This fault causes stress within the target container, which may result in the primary process in the container to be constrained or eat up the available system memory on the node.
Pod network corruption
Pod network corruption is a Kubernetes pod-level chaos fault that flips random bits in a configurable percentage of packets on a target container's network path, simulating a degraded link that mangles bytes on the wire.
- Tests the application's resilience to lossy (or flaky) network.
Details
Use cases
This fault tests the application's resilience to lossy (or flaky) network.Pod network duplication
Pod network duplication is a Kubernetes pod-level chaos fault that duplicates a configurable percentage of packets on a target container's network path, exercising TCP duplicate-segment handling and application-level dedup logic.
- It determines the application's resilience to duplicate network packets.
Details
Use cases
It determines the application's resilience to duplicate network.Pod network latency
Pod network latency is a Kubernetes pod-level chaos fault that adds a configurable delay to packets on a target container's network path, simulating slow upstream dependencies, congested links, or cross-region failover.
- It tests the application's resilience to lossy (or flaky) networks.
Details
View fault usage
The fault degrades the network without the pod being marked as unhealthy (or unworthy) of traffic by kube-proxy (unless there is a liveness probe that measures the latency and restarts or crashes the container). This fault simulates issues within the pod network (or microservice communication) across services in different availability zones or regions.This can be resolved by using middleware that switches traffic based on certain SLOs or performance parameters. Another way is to set up alerts and notifications to highlight a degradation so that it can be addressed and fixed. Another way is to understand the impact of the failure and determine the last point in the application stack before degradation.
The applications may stall or get corrupted while waiting endlessly for a packet. This fault limits the impact (blast radius) to only the traffic that you wish to test by specifying the IP addresses. This fault helps to improve the resilience of your services over time.
Pod network loss
Pod network loss is a Kubernetes pod-level chaos fault that drops a configurable percentage of packets on a target container's network path, simulating a flaky NIC, degraded overlay link, or CNI hiccup.
- It tests the application's resilience to lossy (or flaky) network.
Details
Use cases
It tests the application's resilience to lossy (or flaky) network.Pod network partition
Pod network partition is a Kubernetes pod-level fault that blocks 100% ingress and egress traffic of the target application by creating network policy.
- It can test the application's resilience to lossy (or flaky) network.
Details
Use cases
It can test the application's resilience to lossy (or flaky) network.Pod network rate limit
Pod network rate limit is a Kubernetes pod-level chaos fault that generates Traffic Control (tc) rules with Token Bucket Filter (TBF) to assess Kubernetes pod resilience under limited network bandwidth condition.
- It tests the application's resilience to limited or slow network bandwidth.
Details
Use cases
It tests the application's resilience to limited or slow network bandwidth.Redis cache expire
Redis cache expire expires a configurable set of keys (or all keys) on a target Redis instance to test cold-cache resilience, refill behavior, and downstream database back-pressure.Details
Use cases
Redis cache expire determines the resilience of Redis-dependant applications against frequent cache expiry.
Redis cache limit
Redis cache limit caps the maximum memory of a target Redis instance to force evictions and out-of-memory write errors, then restores the original limit when the fault ends. Redis cache limit determines the resilience of Redis-dependant applications on frequent cache misses that occur due to a low cache size.Use cases
Redis cache penetration
Redis cache penetration issues a configurable burst of reads for keys that do not exist against a target Redis instance to test downstream database load and null-cache protection.Use cases
Time chaos
Time chaos is a Kubernetes pod-level chaos fault that shifts the wall-clock or monotonic time observed by selected processes inside a target container by a configurable offset to test application behavior under clock skew, token expiry, and time-based scheduling errors.Use cases
Pod Application Function Error
Pod Application Function Error makes a specific instrumented application function return a configurable error on a chosen percentage of invocations so you can test caller error handling and recovery.Use cases
Pod Application Function Latency
Pod Application Function Latency adds a configurable delay to invocations of a specific instrumented application function so you can test how callers and dependents behave under slow functions.Use cases
Pod Application Function Exception
Pod Application Function Exception throws a configurable exception from a specific instrumented application function so you can test how callers and dependents handle thrown failures, including retry filters and circuit breakers.Use cases