Skip to main content

Datadog probe

Datadog probe allows you to query raw metrics or a Synthetic test and use its results to evaluate the probe outcome.

  • Synthetics probe query is supported for both API tests and Browser tests.
  • Synthetics probe query may only be executed in the EOT mode, as the probe evaluation is based on the result of all the test iterations executed through the fault chaos duration. Metrics querying is supported for all the probe modes.
note
  • If there are no iterations of the synthetics test through the chaos duration of the fault, the probe is marked as failed.
  • Raw metrics are not yet available for Linux chaos infrastructure.

Providing secrets​

Datadog secret keys need to be provided prior to using the probe, which are used to authenticate with the Datadog APIs. This includes an API key and an Application key.

For a Kubernetes chaos infrastructure, the secrets shall be provided using a Kubernetes secret of the following format:

apiVersion: v1
kind: Secret
metadata:
  name: datadog-secret
  type: Opaque
stringData:
  DD_API_KEY: "xxxxxxxxxxxxxxxxxxxx"
  DD_APP_KEY: "xxxxxxxxxxxxxxxxxxxx"

The secret name, that is, datadog-secret has to be provided while configuring the probe using the datadogCredentialsSecretName field.

Probe definition​

For a Kubernetes chaos infrastructure, the probe is defined at .spec.experiments[].spec.probe path in the chaos engine manifest:

kind: Workflow
apiVersion: argoproj.io/v1alpha1
spec:
  templates:
    - inputs:
        artifacts:
          - raw:
              data: |
                apiVersion: litmuschaos.io/v1alpha1
                kind: ChaosEngine
                spec:
                  experiments:
                    - spec:
                        probe:
                          ####################################
                          Probes are defined here
                          ####################################

Schema​

Listed below is the Datadog Probe schema with common properties shared across all probes and properties unique to Datadog probe.

FieldDescriptionTypeRangeNotes
nameFlag to hold the name of the probeMandatoryN/A type: stringThe name holds the name of the probe. It can be set based on the use-case
typeFlag to hold the type of the probeMandatoryhttpProbe, k8sProbe, cmdProbe, promProbe, and datadogProbeThe type supports four types of probes. It can one of the httpProbe, k8sProbe, cmdProbe, promProbe, and datadogProbe
modeFlag to hold the mode of the probeMandatorySOT, EOT, Edge, Continuous, OnChaosThe mode supports five modes of probes. It can one of the SOT, EOT, Edge, Continuous, and OnChaos. For Datadog probe only EOT mode is supported
datadogSiteSite for datadog probeMandatorydatadoghq.com, us3.datadoghq.com, us5.datadoghq.com, datadoghq.eu, ddog-gov.com, and ap1.datadoghq.comThe datadogSite supports six values. Refer here for details
datadogCredentialsSecretNameName of the secret having datadog probe secret keysOptionalN/A type: stringName of the Kubernetes secret containing the Datadog secret keys. Only required for Kubernetes chaos infrastructure
syntheticsTestSynthetic test details for the probeOptionaltype: syntheticsTestProvide the Synthetic test details. It could be an API or a Browser test
metricsMetrics details for the probeOptionaltype: metricsProvide the Datadog metrics details

Synthetics test​

FieldDescriptionTypeRangeNotes
publicIdPublic ID of the synthetic testMandatoryN/A type: stringThe publicId holds the ID of the synthetic test.
testTypeType of the synthetic testMandatoryapi, browserThe testType holds the type of the synthetic test. It can one of api and browser

Metrics​

FieldDescriptionTypeRangeNotes
queryDatadog metrics queryMandatoryN/A type: string
timeFrameThe time frame through which the metrics should be queried. It is relative to the present time and hence it must be expressed as now-'timeFrameValue'.Mandatory type: string Average or min or max of the timeframe specified. For example, now-5m provides average, minvaluefrom(now-5m) provides the minimum and maxvaluefrom(now-5m) provides the maximum value.
comparatorChecks for the correctness of the probe outputMandatory type: comparator Various fields to compare the desired and obtained data, includes type, criteria and value.

Comparator​

Field Description Type Range Notes
type Flag to hold type of the data used for comparison Mandatory float The type contains type of data, which should be compared as part of comparison operation.
criteria Flag to hold criteria for the comparison Mandatory It supports >=, <=, ==, >, <, !=, oneOf, between for int and float type. And equal, notEqual, contains, matches, notMatches, oneOf for string type. The criteria contains criteria of the comparison, which should be fulfill as part of comparison operation.
value Flag to hold value for the comparison Mandatory N/A type: string The value contains value of the comparison, which should follow the given criteria as part of comparison operation.

Run properties​

Field Description Type Range Notes
probeTimeout Flag to hold the timeout of the probe Mandatory N/A type: string The probeTimeout represents the time limit for the probe to execute the specified check and return the expected data
attempt Flag to hold the attempt of the probe Mandatory N/A type: integer The attempt contains the number of times a check is run upon failure in the previous attempts before declaring the probe status as failed.
interval Flag to hold the interval of the probe Mandatory N/A type: string The interval contains the interval for which probes waits between subsequent retries
probePollingInterval Flag to hold the polling interval for the probes (applicable for all modes) Optional N/A type: string The probePollingInterval contains the time interval for which continuous and onchaos probe should be sleep after each iteration
initialDelaySeconds Flag to hold the initial delay interval for the probes Optional N/A type: integer The initialDelaySeconds represents the initial waiting time interval for the probes.
stopOnFailure Flags to hold the stop or continue the experiment on probe failure OptionalN/A type: booleanThe stopOnFailure can be set to true/false to stop or continue the experiment execution after probe fails

Definition​

In the case of Dedicated Chaos Infrastructure, the following apply:

  • The mode and type are mandatory fields in the probe schema when you define the entire configuration of the probe in the manifest (for Kubernetes (Legacy), Linux, and Windows infrastructure).
  • The name, mode, type and other input properties (depending on the probe) is required to rightly configure the resilience probe. If all the necessary details are not provided, the probe will not execute.

In the case of Harness Delegate, the following apply:

  • For Kubernetes (Harness Infrastructure) (also known as DDCR), the mandatory fields are mode and probeID, and the type field is derived. These fields are generated and patched in the backend to the same manifest. However, in the UI, you will only see the mode and probeID fields when configuring your experiment. This is because the manifest is minified in the UI.
  • If you define the entire probe in task.definition.chaos.probes, the entire configuration is required. If you use the task.probeRef, you only need to specify probeID and mode fields.
probe:
  - name: datadog-probe
    type: "DatadogProbe"
    mode: "EOT"
    datadogProbe/inputs:
      datadogSite: us5.datadoghq.com
      syntheticsTest:
        publicId: zgs-mq8-pgy
        testType: api
      datadogCredentialsSecretName: dd-secret
    runProperties:
      probeTimeout: 2s
      attempt: 1
      interval: 3s
      stopOnFailure: false

Metrics​

To trigger a probe that queries Datadog metrics, specify the metrics properties.

Use the following example to tune this:

apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  appinfo:
    appns: "default"
    applabel: "app=nginx"
    appkind: "deployment"
  chaosServiceAccount: litmus-admin
  experiments:
  - name: pod-delete
    spec:
      probe:
      - name: "datadog-probe"
        type: "datadogProbe"
        datadogProbe/inputs:
          datadogSite: us5.datadoghq.com
          metrics:
            query: avg:system.load.1{*}
            timeFrame: now-5m
            comparator:
              type: "float"
              criteria: "<="
              value: "100"
          datadogCredentialsSecretName: dd-secret
        mode: "EOT"
        runProperties:
          probeTimeout: 5s
          interval: 2s
          attempt: 1
          stopOnFailure: false

API test​

To trigger an API test, specify the syntheticsTest.testType as api.

Use the following example to tune this:

apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  appinfo:
    appns: "default"
    applabel: "app=nginx"
    appkind: "deployment"
  chaosServiceAccount: litmus-admin
  experiments:
  - name: pod-delete
    spec:
      probe:
      - name: "datadog-probe"
        type: "datadogProbe"
        datadogProbe/inputs:
          datadogSite: us5.datadoghq.com
          syntheticsTest:
            publicId: zgs-mq8-pgy
            testType: api
          datadogCredentialsSecretName: dd-secret
        mode: "EOT"
        runProperties:
          probeTimeout: 5s
          interval: 2s
          attempt: 1
          stopOnFailure: false

Browser test​

To trigger a browser test, specify the syntheticsTest.testType as browser.

Use the following example to tune this:

apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  appinfo:
    appns: "default"
    applabel: "app=nginx"
    appkind: "deployment"
  chaosServiceAccount: litmus-admin
  experiments:
  - name: pod-delete
    spec:
      probe:
      - name: "datadog-probe"
        type: "datadogProbe"
        datadogProbe/inputs:
          datadogSite: us5.datadoghq.com
          syntheticsTest:
            publicId: zgs-mq8-pgy
            testType: browser
          datadogCredentialsSecretName: dd-secret
        mode: "EOT"
        runProperties:
          probeTimeout: 5s
          interval: 2s
          attempt: 1
          stopOnFailure: false