> For the complete documentation index, see [llms.txt](https://developer.harness.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://developer.harness.io/resilience-testing/chaos-engineering/chaos-experiments/probe/datadog-probe-probe.md).

# Datadog Probe

{% @harness-package-selector/package-selector platforms="%5B%7B%22label%22%3A%22HTTP%20Probe%22%2C%22slug%22%3A%22http-probe%22%2C%22path%22%3A%22chaos-engineering%2Fguides%2Fprobe%2Fhttp-probe-probe%22%7D%2C%7B%22label%22%3A%22Command%20Probe%22%2C%22slug%22%3A%22command-probe%22%2C%22path%22%3A%22chaos-engineering%2Fguides%2Fprobe%2Fcommand-probe-probe%22%7D%2C%7B%22label%22%3A%22Prometheus%20Probe%22%2C%22slug%22%3A%22prometheus-probe%22%2C%22path%22%3A%22chaos-engineering%2Fguides%2Fprobe%2Fprometheus-probe-probe%22%7D%2C%7B%22label%22%3A%22K8S%20Probe%22%2C%22slug%22%3A%22k8s-probe%22%2C%22path%22%3A%22chaos-engineering%2Fguides%2Fprobe%2Fk8s-probe-probe%22%7D%2C%7B%22label%22%3A%22Datadog%20Probe%22%2C%22slug%22%3A%22datadog-probe%22%2C%22path%22%3A%22chaos-engineering%2Fguides%2Fprobe%2Fdatadog-probe-probe%22%7D%2C%7B%22label%22%3A%22Dynatrace%20Probe%22%2C%22slug%22%3A%22dynatrace-probe%22%2C%22path%22%3A%22chaos-engineering%2Fguides%2Fprobe%2Fdynatrace-probe-probe%22%7D%2C%7B%22label%22%3A%22SLO%20Probe%22%2C%22slug%22%3A%22slo-probe%22%2C%22path%22%3A%22chaos-engineering%2Fguides%2Fprobe%2Fslo-probe-probe%22%7D%2C%7B%22label%22%3A%22APM%20Probe%22%2C%22slug%22%3A%22apm-probe%22%2C%22path%22%3A%22chaos-engineering%2Fguides%2Fprobe%2Fapm-probe-probe%22%7D%2C%7B%22label%22%3A%22Container%20Probe%22%2C%22slug%22%3A%22container-probe%22%2C%22path%22%3A%22chaos-engineering%2Fguides%2Fprobe%2Fcontainer-probe-probe%22%7D%5D" selectedPlatform="datadog-probe" %}

Datadog probe allows you to query raw metrics or a [Synthetic test](https://docs.datadoghq.com/synthetics/) and use its results to evaluate the probe outcome.

* Synthetics probe query is supported for both [API tests](https://docs.datadoghq.com/synthetics/api_tests/) and [Browser tests](https://docs.datadoghq.com/synthetics/browser_tests).
* Synthetics probe query may only be executed in the **EOT mode**, as the probe evaluation is based on the result of all the test iterations executed through the fault chaos duration. Metrics querying is supported for all the probe modes.

{% hint style="info" %}

* If there are no iterations of the synthetics test through the chaos duration of the fault, the probe is marked as failed.
* Raw metrics are not yet available for Linux chaos infrastructure.
  {% endhint %}

### Providing secrets <a href="#providing-secrets" id="providing-secrets"></a>

Datadog secret keys need to be provided prior to using the probe, which are used to authenticate with the Datadog APIs. This includes an [API key](https://docs.datadoghq.com/account_management/api-app-keys/#api-keys) and an [Application key](https://docs.datadoghq.com/account_management/api-app-keys/#application-keys).

{% tabs %}
{% tab title="Kubernetes" %}
For a Kubernetes chaos infrastructure, the secrets shall be provided using a Kubernetes secret of the following format:

```yaml
apiVersion: v1
kind: Secret
metadata:
  name: datadog-secret
  type: Opaque
stringData:
  DD_API_KEY: "xxxxxxxxxxxxxxxxxxxx"
  DD_APP_KEY: "xxxxxxxxxxxxxxxxxxxx"
```

The secret name, that is, datadog-secret has to be provided while configuring the probe using the `datadogCredentialsSecretName` field.
{% endtab %}

{% tab title="Linux" %}
For a Linux chaos infrastructure, the secrets are provided using an environment file at the following path, which is located on the machine where the infrastructure executes:

```bash
DD_API_KEY="xxxxxxxxxxxxxxxxxxxx"
DD_APP_KEY="xxxxxxxxxxxxxxxxxxxx"
```

{% endtab %}
{% endtabs %}

### Probe definition <a href="#probe-definition" id="probe-definition"></a>

{% tabs %}
{% tab title="Kubernetes" %}
For a Kubernetes chaos infrastructure, the probe is defined at `.spec.experiments[].spec.probe` path in the chaos engine manifest:

```yaml
kind: Workflow
apiVersion: argoproj.io/v1alpha1
spec:
  templates:
    - inputs:
        artifacts:
          - raw:
              data: |
                apiVersion: litmuschaos.io/v1alpha1
                kind: ChaosEngine
                spec:
                  experiments:
                    - spec:
                        probe:
                          ####################################
                          Probes are defined here
                          ####################################
```

{% endtab %}

{% tab title="Linux" %}
For a Linux chaos infrastructure, the probe is defined at `.spec.tasks[].definition.chaos.probes` path in the Linux chaos experiment manifest:

```yaml
apiVersion: litmuschaos.io/v1alpha1
kind: LinuxChaosExperiment
spec:
  tasks:
    - name: task-1
      definition:
        chaos:
          probes:
            ####################################
            Probes are defined here
            ####################################
```

{% endtab %}
{% endtabs %}

### Schema <a href="#schema" id="schema"></a>

Listed below is the Datadog Probe schema with common properties shared across all probes and properties unique to Datadog probe.

| **Field**                    | **Description**                                     | **Type**  | **Range**                                                                                                | **Notes**                                                                                                                                         |
| ---------------------------- | --------------------------------------------------- | --------- | -------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- |
| name                         | Flag to hold the name of the probe                  | Mandatory | N/A `type: string`                                                                                       | The `name` holds the name of the probe. It can be set based on the use-case                                                                       |
| type                         | Flag to hold the type of the probe                  | Mandatory | `httpProbe, k8sProbe, cmdProbe, promProbe, and datadogProbe`                                             | The `type` supports four types of probes. It can one of the httpProbe, k8sProbe, cmdProbe, promProbe, and datadogProbe                            |
| mode                         | Flag to hold the mode of the probe                  | Mandatory | `SOT, EOT, Edge, Continuous, OnChaos`                                                                    | The `mode` supports five modes of probes. It can one of the SOT, EOT, Edge, Continuous, and OnChaos. For Datadog probe only EOT mode is supported |
| datadogSite                  | Site for datadog probe                              | Mandatory | `datadoghq.com, us3.datadoghq.com, us5.datadoghq.com, datadoghq.eu, ddog-gov.com, and ap1.datadoghq.com` | The `datadogSite` supports six values. Refer [here](https://docs.datadoghq.com/getting_started/site/#access-the-datadog-site) for details         |
| datadogCredentialsSecretName | Name of the secret having datadog probe secret keys | Optional  | N/A `type: string`                                                                                       | Name of the Kubernetes secret containing the Datadog secret keys. Only required for Kubernetes chaos infrastructure                               |
| syntheticsTest               | Synthetic test details for the probe                | Optional  | `type: syntheticsTest`                                                                                   | Provide the Synthetic test details. It could be an API or a Browser test                                                                          |
| metrics                      | Metrics details for the probe                       | Optional  | `type: metrics`                                                                                          | Provide the Datadog metrics details                                                                                                               |

**Synthetics test**

| **Field** | **Description**                 | **Type**  | **Range**          | **Notes**                                                                          |
| --------- | ------------------------------- | --------- | ------------------ | ---------------------------------------------------------------------------------- |
| publicId  | Public ID of the synthetic test | Mandatory | N/A `type: string` | The `publicId` holds the ID of the synthetic test.                                 |
| testType  | Type of the synthetic test      | Mandatory | `api, browser`     | The `testType` holds the type of the synthetic test. It can one of api and browser |

**Metrics**

| **Field**  | **Description**                                                                                                                                          | **Type**  | **Range**          | **Notes**                                                                                                                                                                                    |
| ---------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- | --------- | ------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| query      | Datadog metrics query                                                                                                                                    | Mandatory | N/A `type: string` |                                                                                                                                                                                              |
| timeFrame  | The time frame through which the metrics should be queried. It is relative to the present time and hence it must be expressed as `now-'timeFrameValue'`. | Mandatory | `type: string`     | Average or min or max of the timeframe specified. For example, `now-5m` provides average, `minvaluefrom(now-5m)` provides the minimum and `maxvaluefrom(now-5m)` provides the maximum value. |
| comparator | Checks for the correctness of the probe output                                                                                                           | Mandatory | `type: comparator` | Various fields to compare the desired and obtained data, includes type, criteria and value.                                                                                                  |

**Comparator**

|           |                                                   |           |                                                                                                                                                             |                                                                                                                       |
| --------- | ------------------------------------------------- | --------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------- |
| **Field** | **Description**                                   | **Type**  | **Range**                                                                                                                                                   | **Notes**                                                                                                             |
| type      | Flag to hold type of the data used for comparison | Mandatory | `float`                                                                                                                                                     | The `type` contains type of data, which should be compared as part of comparison operation.                           |
| criteria  | Flag to hold criteria for the comparison          | Mandatory | It supports `{>=, <=, ==, >, <, !=, oneOf, between}` for int and float type. And `{equal, notEqual, contains, matches, notMatches, oneOf}` for string type. | The `criteria` contains criteria of the comparison, which should be fulfill as part of comparison operation.          |
| value     | Flag to hold value for the comparison             | Mandatory | N/A `type: string`                                                                                                                                          | The `value` contains value of the comparison, which should follow the given criteria as part of comparison operation. |

**Run properties**

|                      |                                                                             |           |                     |                                                                                                                                              |
| -------------------- | --------------------------------------------------------------------------- | --------- | ------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- |
| **Field**            | **Description**                                                             | **Type**  | **Range**           | **Notes**                                                                                                                                    |
| probeTimeout         | Flag to hold the timeout of the probe                                       | Mandatory | N/A `type: string`  | The `probeTimeout` represents the time limit for the probe to execute the specified check and return the expected data                       |
| attempt              | Flag to hold the attempt of the probe                                       | Mandatory | N/A `type: integer` | The `attempt` contains the number of times a check is run upon failure in the previous attempts before declaring the probe status as failed. |
| interval             | Flag to hold the interval of the probe                                      | Mandatory | N/A `type: string`  | The `interval` contains the interval for which probes waits between subsequent retries                                                       |
| probePollingInterval | Flag to hold the polling interval for the probes (applicable for all modes) | Optional  | N/A `type: string`  | The `probePollingInterval` contains the time interval for which continuous and onchaos probe should be sleep after each iteration            |
| initialDelaySeconds  | Flag to hold the initial delay interval for the probes                      | Optional  | N/A `type: integer` | The `initialDelaySeconds` represents the initial waiting time interval for the probes.                                                       |
| stopOnFailure        | Flags to hold the stop or continue the experiment on probe failure          | Optional  | N/A `type: boolean` | The `stopOnFailure` can be set to true/false to stop or continue the experiment execution after probe fails                                  |

### Definition <a href="#definition" id="definition"></a>

{% tabs %}
{% tab title="Kubernetes" %}

```yaml
probe:
  - name: datadog-probe
    type: "DatadogProbe"
    mode: "EOT"
    datadogProbe/inputs:
      datadogSite: us5.datadoghq.com
      syntheticsTest:
        publicId: zgs-mq8-pgy
        testType: api
      datadogCredentialsSecretName: dd-secret
    runProperties:
      probeTimeout: 2s
      attempt: 1
      interval: 3s
      stopOnFailure: false
```

{% endtab %}

{% tab title="Linux" %}

```yaml
probes:
  - name: datadog-probe
    type: "DatadogProbe"
    mode: "EOT"
    datadogProbe/inputs:
      datadogSite: us5.datadoghq.com
      syntheticsTest:
        publicId: zgs-mq8-pgy
        testType: api
      metrics:
        query: avg:system.load.1{*}
        timeFrame: now-5m
        comparator:
          type: "float"
          criteria: "<="
          value: "100"
    runProperties:
      probeTimeout: 2s
      attempt: 1
      interval: 3s
      stopOnFailure: false
```

{% endtab %}
{% endtabs %}

**Metrics**

To trigger a probe that queries Datadog metrics, specify the `metrics` properties.

Use the following example to tune this:

```yaml
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  appinfo:
    appns: "default"
    applabel: "app=nginx"
    appkind: "deployment"
  chaosServiceAccount: litmus-admin
  experiments:
  - name: pod-delete
    spec:
      probe:
      - name: "datadog-probe"
        type: "datadogProbe"
        datadogProbe/inputs:
          datadogSite: us5.datadoghq.com
          metrics:
            query: avg:system.load.1{*}
            timeFrame: now-5m
            comparator:
              type: "float"
              criteria: "<="
              value: "100"
          datadogCredentialsSecretName: dd-secret
        mode: "EOT"
        runProperties:
          probeTimeout: 5s
          interval: 2s
          attempt: 1
          stopOnFailure: false
```

**API test**

To trigger an API test, specify the `syntheticsTest.testType` as `api`.

Use the following example to tune this:

{% tabs %}
{% tab title="Kubernetes" %}

```yaml
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  appinfo:
    appns: "default"
    applabel: "app=nginx"
    appkind: "deployment"
  chaosServiceAccount: litmus-admin
  experiments:
  - name: pod-delete
    spec:
      probe:
      - name: "datadog-probe"
        type: "datadogProbe"
        datadogProbe/inputs:
          datadogSite: us5.datadoghq.com
          syntheticsTest:
            publicId: zgs-mq8-pgy
            testType: api
          datadogCredentialsSecretName: dd-secret
        mode: "EOT"
        runProperties:
          probeTimeout: 5s
          interval: 2s
          attempt: 1
          stopOnFailure: false
```

{% endtab %}

{% tab title="Linux" %}

```yaml
apiVersion: litmuschaos.io/v1alpha1
kind: LinuxChaosExperiment
metadata:
  name: process-kill
  labels:
    context: process-kill
    name: process-kill
spec:
  steps:
    - - name: process-kill-task
  tasks:
    - name: process-kill-task
      taskType: "chaos"
      weight: 10
      definition:
        chaos:
          probes:
            - name: datadog-probe
              type: "DatadogProbe"
              mode: "EOT"
              datadogProbe/inputs:
                datadogSite: us5.datadoghq.com
                syntheticsTest:
                  publicId: zgs-mq8-pgy
                  testType: api
              runProperties:
                probeTimeout: 2s
                attempt: 1
                interval: 3s
                stopOnFailure: false
          experiment: linux-process-kill
          processKillChaos/inputs:
            duration: 30
            processNames: "nginx"
            forceKill: false
```

{% endtab %}
{% endtabs %}

**Browser test**

To trigger a browser test, specify the `syntheticsTest.testType` as `browser`.

Use the following example to tune this:

{% tabs %}
{% tab title="Kubernetes" %}

```yaml
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  appinfo:
    appns: "default"
    applabel: "app=nginx"
    appkind: "deployment"
  chaosServiceAccount: litmus-admin
  experiments:
  - name: pod-delete
    spec:
      probe:
      - name: "datadog-probe"
        type: "datadogProbe"
        datadogProbe/inputs:
          datadogSite: us5.datadoghq.com
          syntheticsTest:
            publicId: zgs-mq8-pgy
            testType: browser
          datadogCredentialsSecretName: dd-secret
        mode: "EOT"
        runProperties:
          probeTimeout: 5s
          interval: 2s
          attempt: 1
          stopOnFailure: false
```

{% endtab %}

{% tab title="Linux" %}

```yaml
apiVersion: litmuschaos.io/v1alpha1
kind: LinuxChaosExperiment
metadata:
  name: process-kill
  labels:
    context: process-kill
    name: process-kill
spec:
  steps:
    - - name: process-kill-task
  tasks:
    - name: process-kill-task
      taskType: "chaos"
      weight: 10
      definition:
        chaos:
          probes:
            - name: datadog-probe
              type: "DatadogProbe"
              mode: "EOT"
              datadogProbe/inputs:
                datadogSite: us5.datadoghq.com
                syntheticsTest:
                  publicId: zgs-mq8-pgy
                  testType: browser
              runProperties:
                probeTimeout: 2s
                attempt: 1
                interval: 3s
                stopOnFailure: false
          experiment: linux-process-kill
          processKillChaos/inputs:
            duration: 30
            processNames: "nginx"
            forceKill: false
```

{% endtab %}
{% endtabs %}