Skip to main content

Datadog Integration Guide

Last updated on

Configure Datadog monitors to send webhook notifications to Harness AI SRE when alerts trigger.

Before you begin​

  • Harness webhook endpoint: Create a Datadog webhook in Harness AI SRE using the Datadog webhook template.
  • Datadog permissions: Access to create or modify monitors and integrations in Datadog.
  • Webhook URL: Copy the webhook URL from your Harness webhook configuration.
  • Datadog webhook documentation: Go to Datadog Webhooks Integration to understand Datadog's webhook capabilities.
  • Monitor notification syntax: Go to Datadog Monitor Notifications to learn about notification configuration and template variables.

Create webhook integration in Datadog​

Open the Webhooks integration to add a new webhook:

  1. In Datadog, navigate to Integrations, then select Integrations
  2. Search for and select Webhooks
  3. Go to the Configuration tab
  4. Click New

Configure the webhook​

Configure these fields:

  • Name: harness-ai-sre (or any descriptive name)
  • URL: Your Harness webhook URL 
    https://<your-harness-instance>/gateway/ai-sre/api/webhooks/<webhook-id>
  • Encode as form: Leave unchecked (use JSON)
  • Custom Headers: Add if your Harness webhook requires authentication 
    X-Webhook-Secret: your-secret-key

Configure the payload​

Use the Payload field to map Datadog variables to JSON:

{
  "alert_name": "$EVENT_TITLE",
  "alert_message": "$TEXT_ONLY_MSG",
  "alert_status": "$ALERT_STATUS",
  "priority": "$PRIORITY",
  "metric": "$ALERT_METRIC",
  "service": "$SERVICE",
  "env": "$ENV",
  "alert_url": "$LINK"
}

Save the webhook​

Click Save to create the webhook integration.

Configure field mapping in Harness​

In your Harness webhook configuration, map the Datadog payload fields to alert properties.

Basic field mapping example​

Use Mustache templates for simple field mapping:

title: "{{webhook.alert_name}}"
message: "{{webhook.alert_message}}"
severity: "{{webhook.priority}}"
source: "datadog"
link: "{{webhook.alert_url}}"
tags:
  - "service:{{webhook.service}}"
  - "env:{{webhook.env}}"
  - "metric:{{webhook.metric}}"

Advanced field mapping with CEL​

Use CEL expressions for conditional logic and transformations:

// Map Datadog priority to Harness severity
title: webhook.alert_name
message: webhook.alert_message
severity: webhook.priority == "P1" ? "critical" :
          webhook.priority == "P2" ? "high" :
          webhook.priority == "P3" ? "medium" : "low"
source: "datadog"
link: webhook.alert_url
tags: [
  "service:" + webhook.service,
  "env:" + webhook.env,
  "status:" + webhook.alert_status.toLowerCase()
]

// Filter: only process alerts (not warnings or recoveries)
filter: webhook.alert_status == "ALERT"

Add webhook to Datadog monitors​

Option 1: Add to existing monitor​

Add the webhook notification to a monitor you already have:

  1. Open the Datadog monitor you want to integrate
  2. Scroll to the Notify your team section
  3. Add the webhook using the @webhook- syntax:
{{#is_alert}}
@webhook-harness-ai-sre
{{/is_alert}}

{{#is_recovery}}
@webhook-harness-ai-sre
{{/is_recovery}}
  1. Click Save

Option 2: Add to monitor template​

For monitors created from templates:

{{#is_alert}}
Alert on {{service.name}} in {{env}}
@webhook-harness-ai-sre
{{/is_alert}}

{{#is_warning}}
Warning on {{service.name}}
@webhook-harness-ai-sre
{{/is_warning}}

Option 3: Use notification policies​

Configure Datadog notification policies to automatically add the webhook to matching monitors.

Test the integration​

Test from Datadog​

Send a test notification from a configured monitor:

  1. Open a monitor configured with the webhook
  2. Click Test Notifications
  3. Select a test scenario (Alert, Warning, Recovery)
  4. Click Run Test

Datadog sends a test webhook to Harness.

Verify in Harness​

Confirm the test alert arrived and mapped correctly:

  1. Navigate to Alerts in Harness AI SRE
  2. Check that the test alert appears
  3. Verify field mapping:
    • Alert title matches expected format
    • Severity is correct
    • Tags are populated
    • Link works

Available Datadog variables​

Datadog provides these template variables for webhook payloads:

VariableDescriptionExample
$EVENT_TITLEMonitor name and status[Alert] High CPU on prod-server-01
$TEXT_ONLY_MSGPlain text messageCPU usage above 90% for 5 minutes
$ALERT_STATUSCurrent statusALERT, WARNING, RECOVERED, NO DATA
$PRIORITYAlert priorityP1, P2, P3, P4, P5
$ALERT_METRICPrimary metric namesystem.cpu.user
$ALERT_QUERYMonitor queryavg(last_5m):avg:system.cpu.user{*} > 0.9
$SERVICEService tag valueapi-gateway
$ENVEnvironment tag valueproduction
$HOSTNAMEAffected hostprod-server-01
$ALERT_IDUnique alert identifier123456789
$ALERT_CYCLE_KEYAlert cycle identifierabc123def456
$LINKLink to alert in Datadoghttps://app.datadoghq.com/...
$SNAPSHOTGraph snapshot URLhttps://p.datadoghq.com/...
$LAST_UPDATEDTimestamp of last update2025-07-01 10:15:30 UTC
$TAGSAll monitor tagsservice:api,env:prod
$ORG_IDDatadog organization ID12345
$ORG_NAMEOrganization nameMy Company

The current metric value and alert threshold are not exposed as webhook payload variables. Access them with the {{value}} and {{threshold}} Mustache variables in the monitor message body instead, then include the rendered message in the webhook payload through $TEXT_ONLY_MSG.

Advanced configuration​

Filter by alert status​

Send different payloads for alerts vs recoveries:

{{#is_alert}}
@webhook-harness-ai-sre
{{/is_alert}}

Route by priority​

Create separate Harness webhooks for different priorities:

{{#is_alert}}
  {{#is_priority 'P1'}}
  @webhook-harness-critical
  {{/is_priority}}

  {{#is_priority 'P2'}}
  @webhook-harness-standard
  {{/is_priority}}
{{/is_alert}}

Include metric values and thresholds​

The webhook payload does not carry the metric value or threshold directly. Datadog exposes those only as {{value}} and {{threshold}} template variables in the monitor message body, so include them there and pass the rendered message through to the webhook:

{
  "alert_name": "$EVENT_TITLE",
  "alert_message": "$TEXT_ONLY_MSG",
  "priority": "$PRIORITY",
  "service": "$SERVICE",
  "env": "$ENV",
  "alert_url": "$LINK",
  "context": {
    "query": "$ALERT_QUERY",
    "snapshot_url": "$SNAPSHOT"
  }
}

Monitor message body:

{{#is_alert}}
Current value: {{value}}
Threshold: {{threshold}}
{{/is_alert}}

Harness CEL mapping:

title: webhook.alert_name
message: webhook.alert_message
severity: webhook.priority == "P1" ? "critical" : "high"
link: webhook.context.snapshot_url

Troubleshooting​

Datadog webhook is not triggering for Harness AI SRE

Verify the webhook appears under Integrations, then Webhooks, confirm the monitor notification includes @webhook-harness-ai-sre, test the notification manually from the monitor page, and review the Datadog Event Explorer for webhook calls.

Datadog webhook returns HTTP 4xx errors when calling the Harness AI SRE endpoint

Verify the webhook URL is correct with no typos in the webhook ID, check that custom headers match the Harness webhook configuration, ensure the Harness webhook is enabled, and test the URL with curl.

Datadog webhook variables are not mapping correctly in Harness AI SRE

Review the actual webhook payload sent by Datadog, ensure field names match exactly (case-sensitive), and use the CEL has() function to check whether fields exist.

High-frequency Datadog alerts are rate limiting the Harness AI SRE webhook

Configure the Datadog monitor to re-notify less frequently, use Harness alert routing rules to deduplicate similar alerts, and add debounce logic in the Harness webhook CEL.

Example: complete integration​

This example shows a production-ready Datadog-to-Harness integration for a microservices platform.

Datadog webhook payload​

{
  "alert_name": "$EVENT_TITLE",
  "alert_message": "$TEXT_ONLY_MSG",
  "alert_status": "$ALERT_STATUS",
  "priority": "$PRIORITY",
  "metric": "$ALERT_METRIC",
  "service": "$SERVICE",
  "env": "$ENV",
  "alert_url": "$LINK",
  "snapshot": "$SNAPSHOT",
  "timestamp": "$LAST_UPDATED",
  "tags": "$TAGS"
}

Harness webhook field mapping example​

title: "{{webhook.alert_name}}"
message: |
  {{webhook.alert_message}}
  
  Metric: {{webhook.metric}}
severity: |
  webhook.priority == "P1" ? "critical" :
  webhook.priority == "P2" ? "high" :
  webhook.priority == "P3" ? "medium" : "low"
source: "datadog"
link: "{{webhook.alert_url}}"
tags:
  - "source:datadog"
  - "service:{{webhook.service}}"
  - "env:{{webhook.env}}"
  - "metric:{{webhook.metric}}"
filter: |
  webhook.alert_status == "ALERT" && 
  webhook.env in ["production", "staging"]

Datadog monitor notification​

{{#is_alert}}
Production alert on {{service.name}}

Current: {{value}}
Threshold: {{threshold}}

@webhook-harness-ai-sre
@slack-platform-alerts
{{/is_alert}}

{{#is_recovery}}
✅ Recovered: {{service.name}}
@webhook-harness-ai-sre
{{/is_recovery}}

Next steps​

Datadog official documentation​