> For the complete documentation index, see [llms.txt](https://developer.harness.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://developer.harness.io/resilience-testing/chaos-engineering/faults/chaos-fault-categories/kubernetes/pod/probe-validations.md).

# Best Practices for Probe Validation - Pod Level Faults

This topic describes the best practices to use with resilience probes in Kubernetes pod-level chaos faults.

#### [Container Kill](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/kubernetes/pod/container-kill.md) <a href="#container-killcontainer-killmd" id="container-killcontainer-killmd"></a>

* [HTTP Probe (Error Handling and Application Availability)](/resilience-testing/chaos-engineering/use-chaos-engineering/probes/http-probe.md): To validate if the ALB (Application Load Balancer) can redirect the traffic to other healthy replicas and if proper error messages and error codes are provided to the user.
* [CMD Probe/APM Queries (Alerts)](/resilience-testing/chaos-engineering/use-chaos-engineering/probes/command-probes/index.md): To check if any alerts are fired when container restarts or when stuck in CrashLoopBackOff.
* CMD Source Probe / APM Queries (Liveness probe): To check whether the liveness probe is able to detect the container restart. In case of liveness probe failure, you will see "unsuccessful" event.
* CMD Source Probe (Container startup time): To check container startup or readiness time to benchmark the application performance and determine SLA violations. For example, if a container takes more time to start up, is it because:
  * The Image Pull Policy is set to "Always" ?
  * Secrets getting expired in your private registry ?
  * It takes more time due to the exponential delay since the container started multiple times in a short span of time.

#### [Pod CPU Hog](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/kubernetes/pod/pod-cpu-hog.md) <a href="#pod-cpu-hogpod-cpu-hogmd" id="pod-cpu-hogpod-cpu-hogmd"></a>

* [HTTP Probe (Latency)](/resilience-testing/chaos-engineering/use-chaos-engineering/probes/http-probe.md): To check the application end point and determine if the response time of the application is in the acceptable range.
* CMD Probe (Kubectl Command): To check if the replica's count is increasing based on your HPA.
* CMD Source Probe (Horizontal Pod Autoscaler or Vertical Pod Scaling): To check how long it takes for the system to scale.
* CMD Source Probe (Throttling): To check if the CPU resources are being throttled in the acceptable range.
* [Prometheus](/resilience-testing/chaos-engineering/use-chaos-engineering/probes/prometheus-probes/index.md)/[DataDog](/resilience-testing/chaos-engineering/use-chaos-engineering/probes/apm-probes/index.md)/[Dynatrace (Alerts)](/resilience-testing/chaos-engineering/use-chaos-engineering/probes/apm-probes/index.md): To check if alerts are being fired when CPU usage is greater than the threshold value.
* Noisy Neighbour: Validate the CPU utilisation of node where application is scheduled when CPU limits is not set.

#### [Pod Memory Hog](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/kubernetes/pod/pod-memory-hog.md) <a href="#pod-memory-hogpod-memory-hogmd" id="pod-memory-hogpod-memory-hogmd"></a>

* [HTTP Probe (Latency)](/resilience-testing/chaos-engineering/use-chaos-engineering/probes/http-probe.md): To check the application end point and determine if the response time of the application is in the acceptable range.
* CMD Probe (Kubectl Command): To check if the replica's count is increasing based on your HPA.
* CMD Source Probe (Horizontal Pod Autoscaler or Vertical Pod Scaling): To check how long it takes for the system to scale.
* CMD Source Probe (Throttling): To check if you get error messages like OOM when memory spike is greater than the memory resource limit and whether target container restarted.
* [Prometheus](/resilience-testing/chaos-engineering/use-chaos-engineering/probes/prometheus-probes/index.md)/[DataDog](/resilience-testing/chaos-engineering/use-chaos-engineering/probes/apm-probes/index.md)/[Dynatrace (Alerts)](/resilience-testing/chaos-engineering/use-chaos-engineering/probes/apm-probes/index.md): To check if alerts are being fired when memory usage is greater than the threshold value.
* Noisy Neighbour: Validates memory utilisation of node where application is scheduled when memory limits are not set

#### [Pod Delete](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/kubernetes/pod/pod-delete.md) <a href="#pod-deletepod-deletemd" id="pod-deletepod-deletemd"></a>

* [HTTP Probe](/resilience-testing/chaos-engineering/use-chaos-engineering/probes/http-probe.md): To check the application's health and determine whether the application is healthy or not.
* [CMD Source Probe (Pod startup time)](/resilience-testing/chaos-engineering/use-chaos-engineering/probes/command-probes/index.md#configure-command-probe-with-source-parameter): To check the pod start up time to benchmark application performance and find the SLA violations. For example, if a pod takes more time to start up, is it because the Image Pull Policy is set to **Always** or Secrets getting expired from your private registry or the node does not have enough resources to schedule a pod?
* CMD Probe (replica count): To check if number of replicas is maintained as per your design.
* CMD Source Probe (Availability zones, node selector and tolerations): To check if the new pods are being created in new availability zones, on new nodes and following the node selector and tolerations rules.

#### [Pod DNS Error](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/kubernetes/pod/pod-dns-error.md) <a href="#pod-dns-errorpod-dns-errormd" id="pod-dns-errorpod-dns-errormd"></a>

* [CMD Probe/ APM Queries (Metrics and Alerts)](/resilience-testing/chaos-engineering/use-chaos-engineering/probes/command-probes/index.md#configure-command-probe-with-source-parameter): To check the application response time, throughput and DNS query failure rates. To check if any alerts are fired.
* CMD Source Probe/APM Queries (Error Handling and Retry Mechanism): To check if the application provides relevant error codes, messages and retries for the DNS queries.
* CMD Source Probe/HTTP Probe (Fallback mechanism): To check if the application uses any fallback mechanism like cached DNS entries etc.
* APM Queries (Application Recovery): To check if the DNS impact is restored after chaos experiment and the application functions properly.
* CMD Source Probe (Application Design- Error Handling): To check if the target application or dependent services are handling errors (circuit breaker).

#### [Pod Network Latency](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/kubernetes/pod/pod-network-latency.md) <a href="#pod-network-latencypod-network-latencymd" id="pod-network-latencypod-network-latencymd"></a>

* [HTTP Probe (Health Check)](/resilience-testing/chaos-engineering/use-chaos-engineering/probes/http-probe.md): To check the application end point and determine if the application is still responsive or hangs.
* [CMD Probe/ APM Queries (Latency)](/resilience-testing/chaos-engineering/use-chaos-engineering/probes/command-probes/index.md): To check how long the end user waits to get a response to understand the retry, exponential back off, failover mechanism, or client-timeout (if any).
* CMD Source Probe/APM Queries (Script): To check if the alerts are fired when your system is un-responsive.
* CMD Source Probe (Application Design - Error handling): To check if the application or dependant services can handle the incoming traffic or not (circuit breaker).
* CMD/HTTP Probe Health checks: If the liveness/readiness probes are set up for the target application and if the probes fail, the container should restart and recover the application after chaos is reverted.

#### [Pod Network Loss](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/kubernetes/pod/pod-network-loss.md) <a href="#pod-network-losspod-network-lossmd" id="pod-network-losspod-network-lossmd"></a>

* [HTTP Probe (Service Availability and Failover Mechanism)](/resilience-testing/chaos-engineering/use-chaos-engineering/probes/http-probe.md): To check the application end point to understand if the application is still responsive or hangs.
* [CMD Probe/APM Queries (Latency)](/resilience-testing/chaos-engineering/use-chaos-engineering/probes/command-probes/index.md): To check how long the end user waited to get response to understand your retry, exponential back off, failover mechanism or client-timeout (if any).
* CMD Source Probe/APM Queries (Alerts): To check if the alerts are fired when your system was un-responsive.
* CMD Source Probe (Application Design - Error handling): To check if the application can handle the incoming traffic or not (circuit breaker).
* CMD/HTTP Probe Health checks: If the liveness/readiness probes are set up for the target application and if the probes fail, the container should restart and recover the application after chaos is reverted.

#### [Time Chaos](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/kubernetes/pod/time-chaos.md) <a href="#time-chaostime-chaosmd" id="time-chaostime-chaosmd"></a>

* [HTTP Probe (Health Check)](/resilience-testing/chaos-engineering/use-chaos-engineering/probes/http-probe.md): To validate the system responsiveness and determine if error codes are handled properly.
* [CMD Probe/APM Queries (Alerts)](/resilience-testing/chaos-engineering/use-chaos-engineering/probes/command-probes/index.md): To check if any alerts are fired when your certificates expire in the application. This may occur when the system time is greater than the certificate expiry time.
* CMD Source Probe (MTTD): To check the mean time to detect a failure if the target application's certificates expire.