Datadog APM Probe Templates
Pre-built Datadog APM Probe templates for validating service health using Datadog metrics during chaos experiments. These templates query Datadog container metrics or APM metrics and compare the result against a threshold you configure at runtime.
Here are Datadog probe templates that you can use in your chaos experiments.
Datadog CPU Check
Validates the CPU utilisation of a service using Datadog container metrics.
Datadog Memory Check
Validates the memory utilisation of a service using Datadog container metrics.
Datadog P95 Latency Check
Validates the p95 latency of a service using Datadog APM metrics.
Datadog P99 Latency Check
Validates the p99 latency of a service using Datadog APM metrics.
Datadog Avg Latency Check
Validates the average latency of a service using Datadog APM metrics.
Datadog Error Rate Check
Validates the error rate of a service using Datadog APM metrics.
Datadog CPU Check
Validates the CPU utilisation of a service using Datadog container metrics. The template queries container.cpu.usage filtered by Kubernetes deployment and namespace.
Required probe variables:
SERVICE_NAME: Kubernetes deployment name of the target serviceNAMESPACE: Kubernetes namespace of the target service
Required inputs:
CONNECTOR_ID: Datadog connector identifier
Use cases
- Verify CPU stays within limits during pod CPU hog faults
- Monitor container CPU after resource stress experiments
- Validate autoscaling behavior under load chaos
- Detect CPU saturation before it affects latency
Datadog Memory Check
Validates the memory utilisation of a service using Datadog container metrics. The template queries container.memory.usage filtered by Kubernetes deployment and namespace.
Required probe variables:
SERVICE_NAME: Kubernetes deployment name of the target serviceNAMESPACE: Kubernetes namespace of the target service
Required inputs:
CONNECTOR_ID: Datadog connector identifier
Use cases
- Verify memory stays within limits during memory hog faults
- Monitor container memory after OOM-related chaos
- Validate memory limits during stress experiments
- Detect memory leaks under sustained load
Datadog P95 Latency Check
Validates the 95th percentile latency of a service using Datadog APM metrics (latency_p95).
Required probe variables:
SERVICE_NAME: Datadog APM service nameENV: Datadog APM environment tag (optional)
Required inputs:
CONNECTOR_ID: Datadog connector identifier
Use cases
- Verify tail latency stays within SLO during network chaos
- Monitor p95 latency after dependency failures
- Validate latency recovery after fault injection
- Detect latency regressions during rollout chaos
Datadog P99 Latency Check
Validates the 99th percentile latency of a service using Datadog APM metrics (latency_p99).
Required probe variables:
SERVICE_NAME: Datadog APM service nameENV: Datadog APM environment tag (optional)
Required inputs:
CONNECTOR_ID: Datadog connector identifier
Use cases
- Verify worst-case latency during high-percentile SLO checks
- Monitor p99 latency after infrastructure faults
- Validate tail latency recovery after chaos injection
- Detect outlier latency spikes during experiments
Datadog Avg Latency Check
Validates the average latency of a service using Datadog APM metrics (latency_avg).
Required probe variables:
SERVICE_NAME: Datadog APM service nameENV: Datadog APM environment tag (optional)
Required inputs:
CONNECTOR_ID: Datadog connector identifier
Use cases
- Verify mean latency stays within budget during chaos
- Monitor average response time after fault injection
- Validate latency stability across experiment phases
- Compare baseline and post-chaos average latency
Datadog Error Rate Check
Validates the error rate of a service using Datadog APM metrics (error_rate).
Required probe variables:
SERVICE_NAME: Datadog APM service nameENV: Datadog APM environment tag (optional)
Required inputs:
CONNECTOR_ID: Datadog connector identifier
Use cases
- Verify error rate stays below threshold during chaos
- Monitor error budget consumption during fault injection
- Validate service recovery after dependency failures
- Detect error spikes during network or pod chaos
Next steps
- Go to Datadog probe to configure a custom Datadog probe.
- Go to APM probes to understand APM probe types and connectors.
- Go to Built-in Probe Templates to browse all probe template categories.