> For the complete documentation index, see [llms.txt](https://developer.harness.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://developer.harness.io/resilience-testing/chaos-testing/probes/apm-probes/datadog-apm-probes.md).

# Datadog

{% @harness-package-selector/package-selector platforms="%5B%7B%22label%22%3A%22Prometheus%22%2C%22slug%22%3A%22prometheus%22%2C%22path%22%3A%22resilience-testing%2Fchaos-testing%2Fprobes%2Fapm-probes%2Fprometheus-apm-probes%22%7D%2C%7B%22label%22%3A%22AppDynamics%22%2C%22slug%22%3A%22appdynamics%22%2C%22path%22%3A%22resilience-testing%2Fchaos-testing%2Fprobes%2Fapm-probes%2Fappdynamics-apm-probes%22%7D%2C%7B%22label%22%3A%22Splunk%20Observability%22%2C%22slug%22%3A%22splunk-observability%22%2C%22path%22%3A%22resilience-testing%2Fchaos-testing%2Fprobes%2Fapm-probes%2Fsplunk-observability-apm-probes%22%7D%2C%7B%22label%22%3A%22Splunk%20Enterprise%22%2C%22slug%22%3A%22splunk-enterprise%22%2C%22path%22%3A%22resilience-testing%2Fchaos-testing%2Fprobes%2Fapm-probes%2Fsplunk-enterprise-apm-probes%22%7D%2C%7B%22label%22%3A%22Dynatrace%22%2C%22slug%22%3A%22dynatrace%22%2C%22path%22%3A%22resilience-testing%2Fchaos-testing%2Fprobes%2Fapm-probes%2Fdynatrace-apm-probes%22%7D%2C%7B%22label%22%3A%22New%20Relic%22%2C%22slug%22%3A%22new-relic%22%2C%22path%22%3A%22resilience-testing%2Fchaos-testing%2Fprobes%2Fapm-probes%2Fnew-relic-apm-probes%22%7D%2C%7B%22label%22%3A%22GCP%20Cloud%20Monitoring%22%2C%22slug%22%3A%22gcp-cloud-monitoring%22%2C%22path%22%3A%22resilience-testing%2Fchaos-testing%2Fprobes%2Fapm-probes%2Fgcp-cloud-monitoring-apm-probes%22%7D%2C%7B%22label%22%3A%22Datadog%22%2C%22slug%22%3A%22datadog%22%2C%22path%22%3A%22resilience-testing%2Fchaos-testing%2Fprobes%2Fapm-probes%2Fdatadog-apm-probes%22%7D%5D" selectedPlatform="datadog" %}

Datadog APM probe allows you to query Datadog metrics or run Synthetic tests and compare the results against specified criteria. It supports a legacy single-query metrics path and a **Datadog v2 timeseries** path for structured, multi-query validation.

For pre-built Datadog APM probe templates (CPU, memory, latency, and error rate), go to [Datadog APM Probe Templates](/resilience-testing/chaos-testing/probes/probe-template-library/datadog/index.md).

### When to use <a href="#when-to-use" id="when-to-use"></a>

* Monitor Datadog metrics (e.g., `system.cpu.user`, `trace.servlet.request.duration`) as steady-state indicators during chaos
* Query Datadog APM metrics (e.g., P95 latency, error rate) for a specific service using the v2 timeseries API
* Use Datadog Synthetic tests to validate end-to-end user flows under failure conditions
* Validate that Datadog-monitored SLOs remain within acceptable thresholds during fault injection
* Reuse a probe template across services by supplying service and environment values at runtime through probe variables

### Prerequisites <a href="#prerequisites" id="prerequisites"></a>

To use the Datadog APM probe, you need:

* An active Datadog account
* Access to the Datadog API from the Kubernetes execution plane
* A Datadog API key and Application key. Go to [Datadog API Keys](https://docs.datadoghq.com/account_management/api-app-keys/#api-keys) to create an API key and go to [Application Keys](https://docs.datadoghq.com/account_management/api-app-keys/#application-keys) to create an application key for the Harness Datadog connector.

#### Datadog application key scopes <a href="#datadog-application-key-scopes" id="datadog-application-key-scopes"></a>

When you create the application key in Datadog, grant the scopes that match the query mode you use. Grant the minimum scopes required for your probe configuration.

| Scope              | Required for                                      | Purpose                                                                                          |
| ------------------ | ------------------------------------------------- | ------------------------------------------------------------------------------------------------ |
| `timeseries_query` | **Metrics** and **Metrics (v2)**                  | Query metric timeseries data via the Datadog Metrics API (legacy v1 and v2 timeseries endpoints) |
| `apm_read`         | **Metrics (v2)** with **APM Metrics** data source | Query APM service metrics such as latency percentiles and error rate                             |
| `synthetics_read`  | **Synthetic Test**                                | Read Synthetic test configuration and results                                                    |

{% hint style="info" %}
**MINIMUM SCOPES BY MODE**

* **Legacy Metrics** or **Metrics (v2)** with the **Metrics** data source: `timeseries_query`
* **Metrics (v2)** with the **APM Metrics** data source: `timeseries_query` and `apm_read`
* **Synthetic Test**: `synthetics_read`
  {% endhint %}

Go to [Datadog API and Application Keys](https://docs.datadoghq.com/account_management/api-app-keys/) to create scoped application keys.

### Steps to configure <a href="#steps-to-configure" id="steps-to-configure"></a>

1. Navigate to **Project Settings** > **Chaos Probes** and click **+ New Probe**
2. Select **APM Probe**, provide a name, and select **Datadog** under APM Type
3. Under **Variables**, define any reusable values you want to reference in probe properties or run properties. For each variable, specify the type (`String` or `Number`), name, value (fixed or runtime input), and whether it's required at runtime. Use expressions such as `<+probe.variables.SERVICE_NAME>` in query parameters to make a probe reusable across services.
4. Under **Datadog Connector**, select an existing connector or click **+ New Connector** to create one. Provide the Datadog instance URL, Application key, and API key, configure the delegate, verify the connection, and click **Finish**.
5. Under **Probe Properties**, select the query mode at the top. Harness sets the query type automatically from your selection. You do **not** need to enter a **Query Type** field in the UI.

   **Legacy metrics mode (Metrics):**

   | Field                            | Description                                                                                                                                                                                                          |
   | -------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
   | **Datadog Query**                | <p>Single Datadog metrics query string.<br>Example: <code>avg:system.cpu.user{host:my-host}</code>. Go to <a href="https://docs.datadoghq.com/metrics/">Datadog Metrics documentation</a> to learn query syntax.</p> |
   | **Lookback Window (in minutes)** | Time range from the specified number of minutes ago to now                                                                                                                                                           |

   **v2 timeseries mode (Metrics (v2)):**

   Use this mode when you need structured queries, APM metrics, or formulas across multiple named queries. Select **Metrics (v2)** at the top of **Probe Properties** and define one or more named queries.

   | Field                            | Description                                                                                                                                                          |
   | -------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
   | **Queries**                      | List of named queries evaluated by Datadog. Each query has a **Name**, **Data Source**, and params                                                                   |
   | **Formula**                      | Expression evaluated against the named queries (for example, `p95*1000` or `errors / hits`). If only one query is defined, the formula defaults to that query's name |
   | **Aggregation**                  | How datapoints in the lookback window are collapsed before comparison. Supported values: `mean` (default), `max`, `min`, `last`                                      |
   | **Lookback Window (in minutes)** | Time range from the specified number of minutes ago to now (`durationInMin`)                                                                                         |

   **Supported query data sources:**

   | Data source   | UI label    | Description                 | Params format                                                                                                                                          |
   | ------------- | ----------- | --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ |
   | `metrics`     | Metrics     | Datadog metrics query       | `query` string (for example, `sum:trace.servlet.request.hits{service:account-service}.as_count()`)                                                     |
   | `apm_metrics` | APM Metrics | Datadog APM service metrics | JSON object in **Params (JSON)**. `stat` is required; `service` is typically required. `env`, `span_kind`, `group_by`, and `query_filter` are optional |

   **Synthetic Test mode:**

   | Field              | Description                                                                                                                                                                                               |
   | ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
   | **Synthetic Test** | Provide the Synthetic test details (API test or Browser test) to evaluate the probe outcome. Go to [Datadog Synthetics documentation](https://docs.datadoghq.com/synthetics/) to create and manage tests. |

   Under **Datadog Data Comparison**, provide:

   | Field                   | Description                                                               |
   | ----------------------- | ------------------------------------------------------------------------- |
   | **Type**                | Data type for comparison: `Float` or `Int`                                |
   | **Comparison Criteria** | Comparison operator: `>=`, `<=`, `==`, `!=`, `>`, `<`, `oneOf`, `between` |
   | **Value**               | The expected value to compare against the metric result                   |
6. Provide the **Run Properties**:

   | Field                          | Description                                     |
   | ------------------------------ | ----------------------------------------------- |
   | **Timeout**                    | Maximum time for probe execution (e.g., `10s`)  |
   | **Interval**                   | Time between successive executions (e.g., `2s`) |
   | **Attempt**                    | Number of retry attempts (e.g., `1`)            |
   | **Polling Interval**           | Time between retries (e.g., `30s`)              |
   | **Initial Delay**              | Delay before first execution (e.g., `5s`)       |
   | **Verbosity**                  | Log detail level                                |
   | **Stop On Failure** (optional) | Stop the experiment if the probe fails          |
7. Click **Create Probe**

### APM Metrics JSON format <a href="#apm-metrics-json-format" id="apm-metrics-json-format"></a>

When you select **Metrics (v2)** and set **Data Source** to **APM Metrics**, enter JSON in the **Params (JSON)** field. Do **not** include `data_source` or `name`. Harness adds those from the **Data Source** dropdown and **Name** field.

**Minimal Params (JSON) for the UI:**

```json
{
  "service": "account-service",
  "stat": "latency_p95"
}
```

**With optional filters:**

```json
{
  "service": "<+probe.variables.SERVICE_NAME>",
  "stat": "error_rate",
  "env": "prod",
  "span_kind": "server",
  "query_filter": "version:v2"
}
```

**Complete `apm_metrics` query (Datadog v2 timeseries API shape):**

The JSON below shows how a single named query appears inside the v2 timeseries request after Harness merges your Params JSON with the data source and query name. You do not paste this full object in the UI. Enter only the inner `apm_metrics_query` fields in **Params (JSON)**.

```json
{
  "data_source": "apm_metrics",
  "name": "p95",
  "apm_metrics_query": {
    "service": "account-service",
    "stat": "latency_p95",
    "env": "prod",
    "span_kind": "server",
    "query_filter": "version:v2"
  }
}
```

| Field          | Required    | Description                                                                                        |
| -------------- | ----------- | -------------------------------------------------------------------------------------------------- |
| `stat`         | Yes         | APM statistic to query. Common values: `latency_p95`, `latency_p99`, `error_rate`, `hits`, `apdex` |
| `service`      | Recommended | Datadog APM service name. Required in practice for meaningful results.                             |
| `env`          | No          | APM environment (for example, `prod`, `staging`)                                                   |
| `span_kind`    | No          | Span kind filter: `server`, `client`, `consumer`, `producer`, or `internal`                        |
| `query_filter` | No          | Additional filters in Datadog query syntax (for example, `env:prod,version:2.1.0`)                 |

### Datadog v2 APM metrics example <a href="#datadog-v2-apm-metrics-example" id="datadog-v2-apm-metrics-example"></a>

The following probe template checks the P95 latency of a service using Datadog APM metrics. The service name is supplied at runtime; the environment is optional.

```yaml
identity: datadog-p95-latency-check
name: Datadog P95 Latency Check
type: apmProbe
infrastructureType: Kubernetes
probeProperties:
  apmProbe:
    comparator:
      type: float
      value: <+input>
      criteria: <=
    type: Datadog
    datadogApmProbeInputs:
      connectorID: <+input>
      durationInMin: <+input>
      queryType: v2
      queries:
        - name: p95
          dataSource: apm_metrics
          params:
            env: <+probe.variables.ENV>
            service: <+probe.variables.SERVICE_NAME>
            stat: latency_p95
      formula: p95*1000
      aggregation: mean
runProperties:
  timeout: 30s
  interval: 5s
  attempt: 1
  pollingInterval: 30s
  initialDelay: 1s
  verbosity: debug
variables:
  - name: SERVICE_NAME
    value: <+input>
    type: String
    description: The name of the service to monitor.
    required: true
  - name: ENV
    value: <+input>
    type: String
    description: The APM environment of the service to monitor.
    required: false
```

In this example:

* Select **Metrics (v2)** in the UI. Harness sets `queryType: v2` in YAML templates automatically.
* `dataSource: apm_metrics` queries Datadog APM metrics for the target service
* `stat: latency_p95` requests the P95 latency statistic
* `formula: p95*1000` converts seconds to milliseconds before comparison
* `aggregation: mean` averages datapoints across the lookback window
* `SERVICE_NAME` is required at runtime; `ENV` is optional and can be omitted when not needed

**Equivalent UI settings:**

* Mode: **Metrics (v2)**
* Data Source: **APM Metrics**
* Name: `p95`
* Params (JSON):

```json
{
  "service": "<+probe.variables.SERVICE_NAME>",
  "stat": "latency_p95",
  "env": "<+probe.variables.ENV>"
}
```

* Formula: `p95*1000`
* Aggregation: `mean`
* Comparator: `<=` / `500` (milliseconds)

### Multi-query metrics example <a href="#multi-query-metrics-example" id="multi-query-metrics-example"></a>

Use multiple named queries and a formula when validating a ratio or derived metric:

```yaml
datadogApmProbeInputs:
  connectorID: <+input>
  durationInMin: 5
  queries:
    - name: errors
      dataSource: metrics
      params:
        query: sum:trace.servlet.request.errors{service:account-service}.as_count()
    - name: hits
      dataSource: metrics
      params:
        query: sum:trace.servlet.request.hits{service:account-service}.as_count()
  formula: errors / hits
  aggregation: mean
```

### Probe YAML reference (v2) <a href="#probe-yaml-reference-v2" id="probe-yaml-reference-v2"></a>

```yaml
apmProbe/inputs:
  type: Datadog
  comparator:
    type: float
    value: "500"
    criteria: <=
  datadogApmProbeInputs:
    connectorID: my-datadog-connector
    durationInMin: 5
    queryType: v2
    queries:
      - name: p95
        dataSource: apm_metrics
        params:
          service: cartservice
          stat: latency_p95
          env: production
    formula: p95*1000
    aggregation: mean
runProperties:
  timeout: 30s
  interval: 5s
  attempt: 1
```

{% hint style="info" %}
**NOTE**

* In the UI, **Metrics** and **Metrics (v2)** are separate choices. Harness maps them to the correct Datadog API path. In YAML templates, `queryType: v2` (or a populated `queries` list) selects the v2 path.
* For `apm_metrics` queries, `stat` is required. Optional parameters such as `env` can be left unset when not needed.
* Synthetic test probes are evaluated in **EOT** mode only.
  {% endhint %}