> For the complete documentation index, see [llms.txt](https://developer.harness.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://developer.harness.io/resilience-testing/3.0/chaos-testing/probes/probe-template-library/datadog/index.md).

# Datadog APM Probe Templates

Pre-built Datadog APM Probe templates for validating service health using Datadog metrics during chaos experiments. These templates query Datadog container metrics or APM metrics and compare the result against a threshold you configure at runtime.

Here are Datadog probe templates that you can use in your chaos experiments.

<table data-view="cards"><thead><tr><th></th><th></th><th><select multiple><option value="8IlYsumU" label="datadog" color="blue"></option><option value="GFFzwX3j" label="cpu" color="blue"></option><option value="G8iGcllX" label="metrics" color="blue"></option><option value="quM66HnF" label="kubernetes" color="blue"></option><option value="2WSXwzHf" label="memory" color="blue"></option><option value="R-egqMwr" label="apm" color="blue"></option><option value="jmLcVIbo" label="latency" color="blue"></option><option value="MvtjE8NG" label="p95" color="blue"></option><option value="nR2otje6" label="p99" color="blue"></option><option value="DRUv09Ev" label="avg" color="blue"></option><option value="El7yN8eO" label="error-rate" color="blue"></option></select></th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td><strong>Datadog CPU Check</strong></td><td>Validates the CPU utilisation of a service using Datadog container metrics.</td><td><span data-option="8IlYsumU">datadog, </span><span data-option="GFFzwX3j">cpu, </span><span data-option="G8iGcllX">metrics, </span><span data-option="quM66HnF">kubernetes</span></td><td><a href="#datadog-cpu-check">#datadog-cpu-check</a></td></tr><tr><td><strong>Datadog Memory Check</strong></td><td>Validates the memory utilisation of a service using Datadog container metrics.</td><td><span data-option="8IlYsumU">datadog, </span><span data-option="2WSXwzHf">memory, </span><span data-option="G8iGcllX">metrics, </span><span data-option="quM66HnF">kubernetes</span></td><td><a href="#datadog-memory-check">#datadog-memory-check</a></td></tr><tr><td><strong>Datadog P95 Latency Check</strong></td><td>Validates the p95 latency of a service using Datadog APM metrics.</td><td><span data-option="8IlYsumU">datadog, </span><span data-option="R-egqMwr">apm, </span><span data-option="jmLcVIbo">latency, </span><span data-option="MvtjE8NG">p95</span></td><td><a href="#datadog-p95-latency-check">#datadog-p95-latency-check</a></td></tr><tr><td><strong>Datadog P99 Latency Check</strong></td><td>Validates the p99 latency of a service using Datadog APM metrics.</td><td><span data-option="8IlYsumU">datadog, </span><span data-option="R-egqMwr">apm, </span><span data-option="jmLcVIbo">latency, </span><span data-option="nR2otje6">p99</span></td><td><a href="#datadog-p99-latency-check">#datadog-p99-latency-check</a></td></tr><tr><td><strong>Datadog Avg Latency Check</strong></td><td>Validates the average latency of a service using Datadog APM metrics.</td><td><span data-option="8IlYsumU">datadog, </span><span data-option="R-egqMwr">apm, </span><span data-option="jmLcVIbo">latency, </span><span data-option="DRUv09Ev">avg</span></td><td><a href="#datadog-avg-latency-check">#datadog-avg-latency-check</a></td></tr><tr><td><strong>Datadog Error Rate Check</strong></td><td>Validates the error rate of a service using Datadog APM metrics.</td><td><span data-option="8IlYsumU">datadog, </span><span data-option="R-egqMwr">apm, </span><span data-option="El7yN8eO">error-rate</span></td><td><a href="#datadog-error-rate-check">#datadog-error-rate-check</a></td></tr></tbody></table>

#### Datadog CPU Check <a href="#datadog-cpu-check" id="datadog-cpu-check"></a>

Validates the CPU utilisation of a service using Datadog container metrics. The template queries `container.cpu.usage` filtered by Kubernetes deployment and namespace.

**Required probe variables:**

* `SERVICE_NAME`: Kubernetes deployment name of the target service
* `NAMESPACE`: Kubernetes namespace of the target service

**Required inputs:**

* `CONNECTOR_ID`: Datadog connector identifier

<details>

<summary>Use cases</summary>

* Verify CPU stays within limits during pod CPU hog faults
* Monitor container CPU after resource stress experiments
* Validate autoscaling behavior under load chaos
* Detect CPU saturation before it affects latency

</details>

#### Datadog Memory Check <a href="#datadog-memory-check" id="datadog-memory-check"></a>

Validates the memory utilisation of a service using Datadog container metrics. The template queries `container.memory.usage` filtered by Kubernetes deployment and namespace.

**Required probe variables:**

* `SERVICE_NAME`: Kubernetes deployment name of the target service
* `NAMESPACE`: Kubernetes namespace of the target service

**Required inputs:**

* `CONNECTOR_ID`: Datadog connector identifier

<details>

<summary>Use cases</summary>

* Verify memory stays within limits during memory hog faults
* Monitor container memory after OOM-related chaos
* Validate memory limits during stress experiments
* Detect memory leaks under sustained load

</details>

#### Datadog P95 Latency Check <a href="#datadog-p95-latency-check" id="datadog-p95-latency-check"></a>

Validates the 95th percentile latency of a service using Datadog APM metrics (`latency_p95`).

**Required probe variables:**

* `SERVICE_NAME`: Datadog APM service name
* `ENV`: Datadog APM environment tag (optional)

**Required inputs:**

* `CONNECTOR_ID`: Datadog connector identifier

<details>

<summary>Use cases</summary>

* Verify tail latency stays within SLO during network chaos
* Monitor p95 latency after dependency failures
* Validate latency recovery after fault injection
* Detect latency regressions during rollout chaos

</details>

#### Datadog P99 Latency Check <a href="#datadog-p99-latency-check" id="datadog-p99-latency-check"></a>

Validates the 99th percentile latency of a service using Datadog APM metrics (`latency_p99`).

**Required probe variables:**

* `SERVICE_NAME`: Datadog APM service name
* `ENV`: Datadog APM environment tag (optional)

**Required inputs:**

* `CONNECTOR_ID`: Datadog connector identifier

<details>

<summary>Use cases</summary>

* Verify worst-case latency during high-percentile SLO checks
* Monitor p99 latency after infrastructure faults
* Validate tail latency recovery after chaos injection
* Detect outlier latency spikes during experiments

</details>

#### Datadog Avg Latency Check <a href="#datadog-avg-latency-check" id="datadog-avg-latency-check"></a>

Validates the average latency of a service using Datadog APM metrics (`latency_avg`).

**Required probe variables:**

* `SERVICE_NAME`: Datadog APM service name
* `ENV`: Datadog APM environment tag (optional)

**Required inputs:**

* `CONNECTOR_ID`: Datadog connector identifier

<details>

<summary>Use cases</summary>

* Verify mean latency stays within budget during chaos
* Monitor average response time after fault injection
* Validate latency stability across experiment phases
* Compare baseline and post-chaos average latency

</details>

#### Datadog Error Rate Check <a href="#datadog-error-rate-check" id="datadog-error-rate-check"></a>

Validates the error rate of a service using Datadog APM metrics (`error_rate`).

**Required probe variables:**

* `SERVICE_NAME`: Datadog APM service name
* `ENV`: Datadog APM environment tag (optional)

**Required inputs:**

* `CONNECTOR_ID`: Datadog connector identifier

<details>

<summary>Use cases</summary>

* Verify error rate stays below threshold during chaos
* Monitor error budget consumption during fault injection
* Validate service recovery after dependency failures
* Detect error spikes during network or pod chaos

</details>

***

### Next steps <a href="#next-steps" id="next-steps"></a>

* Go to [Datadog probe](https://app.gitbook.com/s/RmV5EkLS9r45VPaoUnJZ/chaos-testing/probes/datadog-probe) to configure a custom Datadog probe.
* Go to [APM probes](/resilience-testing/chaos-testing/probes/apm-probes.md) to understand APM probe types and connectors.
* Go to [Built-in Probe Templates](/resilience-testing/chaos-testing/probes/probe-templates.md) to browse all probe template categories.