Datadog APM Probe Templates

Pre-built Datadog APM Probe templates for CPU, memory, latency, and error rate validation

Pre-built Datadog APM Probe templates for validating service health using Datadog metrics during chaos experiments. These templates query Datadog container metrics or APM metrics and compare the result against a threshold you configure at runtime.

Here are Datadog probe templates that you can use in your chaos experiments.

Datadog CPU Check

Validates the CPU utilisation of a service using Datadog container metrics.

datadogcpumetricskubernetes

Datadog Memory Check

Validates the memory utilisation of a service using Datadog container metrics.

datadogmemorymetricskubernetes

Datadog P95 Latency Check

Validates the p95 latency of a service using Datadog APM metrics.

datadogapmlatencyp95

Datadog P99 Latency Check

Validates the p99 latency of a service using Datadog APM metrics.

datadogapmlatencyp99

Datadog Avg Latency Check

Validates the average latency of a service using Datadog APM metrics.

datadogapmlatencyavg

Datadog Error Rate Check

Validates the error rate of a service using Datadog APM metrics.

datadogapmerror-rate

Datadog CPU Check

Validates the CPU utilisation of a service using Datadog container metrics. The template queries container.cpu.usage filtered by Kubernetes deployment and namespace.

Required probe variables:

• SERVICE_NAME: Kubernetes deployment name of the target service

• NAMESPACE: Kubernetes namespace of the target service

Required inputs:

• CONNECTOR_ID: Datadog connector identifier

Use cases

• Verify CPU stays within limits during pod CPU hog faults

• Monitor container CPU after resource stress experiments

• Validate autoscaling behavior under load chaos

• Detect CPU saturation before it affects latency

View details

Datadog Memory Check

Validates the memory utilisation of a service using Datadog container metrics. The template queries container.memory.usage filtered by Kubernetes deployment and namespace.

Required probe variables:

• SERVICE_NAME: Kubernetes deployment name of the target service

• NAMESPACE: Kubernetes namespace of the target service

Required inputs:

• CONNECTOR_ID: Datadog connector identifier

Use cases

• Verify memory stays within limits during memory hog faults

• Monitor container memory after OOM-related chaos

• Validate memory limits during stress experiments

• Detect memory leaks under sustained load

View details

Datadog P95 Latency Check

Validates the 95th percentile latency of a service using Datadog APM metrics (latency_p95).

Required probe variables:

• SERVICE_NAME: Datadog APM service name

• ENV: Datadog APM environment tag (optional)

Required inputs:

• CONNECTOR_ID: Datadog connector identifier

Use cases

• Verify tail latency stays within SLO during network chaos

• Monitor p95 latency after dependency failures

• Validate latency recovery after fault injection

• Detect latency regressions during rollout chaos

View details

Datadog P99 Latency Check

Validates the 99th percentile latency of a service using Datadog APM metrics (latency_p99).

Required probe variables:

• SERVICE_NAME: Datadog APM service name

• ENV: Datadog APM environment tag (optional)

Required inputs:

• CONNECTOR_ID: Datadog connector identifier

Use cases

• Verify worst-case latency during high-percentile SLO checks

• Monitor p99 latency after infrastructure faults

• Validate tail latency recovery after chaos injection

• Detect outlier latency spikes during experiments

View details

Datadog Avg Latency Check

Validates the average latency of a service using Datadog APM metrics (latency_avg).

Required probe variables:

• SERVICE_NAME: Datadog APM service name

• ENV: Datadog APM environment tag (optional)

Required inputs:

• CONNECTOR_ID: Datadog connector identifier

Use cases

• Verify mean latency stays within budget during chaos

• Monitor average response time after fault injection

• Validate latency stability across experiment phases

• Compare baseline and post-chaos average latency

View details

Datadog Error Rate Check

Validates the error rate of a service using Datadog APM metrics (error_rate).

Required probe variables:

• SERVICE_NAME: Datadog APM service name

• ENV: Datadog APM environment tag (optional)

Required inputs:

• CONNECTOR_ID: Datadog connector identifier

Use cases

• Verify error rate stays below threshold during chaos

• Monitor error budget consumption during fault injection

• Validate service recovery after dependency failures

• Detect error spikes during network or pod chaos

View details

---

Next steps

• Go to Datadog probe to configure a custom Datadog probe.

• Go to APM probes to understand APM probe types and connectors.

• Go to Built-in Probe Templates to browse all probe template categories.