Skip to main content

Configure Datadog to Send Webhooks

Configure Datadog monitors to send webhook notifications to Harness AI SRE when alerts trigger.

Before you begin​

  • Harness webhook endpoint: Create a Datadog webhook in Harness AI SRE using the Datadog webhook template.
  • Datadog permissions: Access to create or modify monitors and integrations in Datadog.
  • Webhook URL: Copy the webhook URL from your Harness webhook configuration.
  • Datadog webhook documentation: Go to Datadog Webhooks Integration to understand Datadog's webhook capabilities.
  • Monitor notification syntax: Go to Datadog Monitor Notifications to learn about notification configuration and template variables.

Create webhook integration in Datadog​

  1. In Datadog, navigate to Integrations → Integrations
  2. Search for and select Webhooks
  3. Go to the Configuration tab
  4. Click New

Configure the webhook​

Configure these fields:

  • Name: harness-ai-sre (or any descriptive name)
  • URL: Your Harness webhook URL 
    https://<your-harness-instance>/gateway/ai-sre/api/webhooks/<webhook-id>
  • Encode as form: Leave unchecked (use JSON)
  • Custom Headers: Add if your Harness webhook requires authentication 
    X-Webhook-Secret: your-secret-key

Configure the payload​

Use the Payload field to map Datadog variables to JSON:

{
  "alert_name": "$EVENT_TITLE",
  "alert_message": "$TEXT_ONLY_MSG",
  "alert_status": "$ALERT_STATUS",
  "priority": "$PRIORITY",
  "metric": "$ALERT_METRIC",
  "service": "$SERVICE",
  "env": "$ENV",
  "alert_url": "$LINK"
}

Save the webhook​

Click Save to create the webhook integration.

Configure field mapping in Harness​

In your Harness webhook configuration, map the Datadog payload fields to alert properties.

Basic field mapping​

Use Mustache templates for simple field mapping:

title: "{{webhook.alert_name}}"
message: "{{webhook.alert_message}}"
severity: "{{webhook.priority}}"
source: "datadog"
link: "{{webhook.alert_url}}"
tags:
  - "service:{{webhook.service}}"
  - "env:{{webhook.env}}"
  - "metric:{{webhook.metric}}"

Advanced field mapping with CEL​

Use CEL expressions for conditional logic and transformations:

// Map Datadog priority to Harness severity
title: webhook.alert_name
message: webhook.alert_message
severity: webhook.priority == "P1" ? "critical" :
          webhook.priority == "P2" ? "high" :
          webhook.priority == "P3" ? "medium" : "low"
source: "datadog"
link: webhook.alert_url
tags: [
  "service:" + webhook.service,
  "env:" + webhook.env,
  "status:" + webhook.alert_status.toLowerCase()
]

// Filter: only process alerts (not warnings or recoveries)
filter: webhook.alert_status == "ALERT"

Add webhook to Datadog monitors​

Option 1: Add to existing monitor​

  1. Open the Datadog monitor you want to integrate
  2. Scroll to the Notify your team section
  3. Add the webhook using the @webhook- syntax:
{{#is_alert}}
@webhook-harness-ai-sre
{{/is_alert}}

{{#is_recovery}}
@webhook-harness-ai-sre
{{/is_recovery}}
  1. Click Save

Option 2: Add to monitor template​

For monitors created from templates:

{{#is_alert}}
Alert on {{service.name}} in {{env}}
@webhook-harness-ai-sre
{{/is_alert}}

{{#is_warning}}
Warning on {{service.name}}
@webhook-harness-ai-sre
{{/is_warning}}

Option 3: Use notification policies​

Configure Datadog notification policies to automatically add the webhook to matching monitors.

Test the integration​

Test from Datadog​

  1. Open a monitor configured with the webhook
  2. Click Test Notifications
  3. Select a test scenario (Alert, Warning, Recovery)
  4. Click Run Test

Datadog sends a test webhook to Harness.

Verify in Harness​

  1. Navigate to Alerts in Harness AI SRE
  2. Check that the test alert appears
  3. Verify field mapping:
    • Alert title matches expected format
    • Severity is correct
    • Tags are populated
    • Link works

Available Datadog variables​

Datadog provides these template variables for webhook payloads:

VariableDescriptionExample
$EVENT_TITLEMonitor name and status[Alert] High CPU on prod-server-01
$TEXT_ONLY_MSGPlain text messageCPU usage above 90% for 5 minutes
$ALERT_STATUSCurrent statusALERT, WARNING, RECOVERED, NO DATA
$PRIORITYAlert priorityP1, P2, P3, P4, P5
$ALERT_METRICPrimary metric namesystem.cpu.user
$ALERT_QUERYMonitor queryavg(last_5m):avg:system.cpu.user{*} > 0.9
$SERVICEService tag valueapi-gateway
$ENVEnvironment tag valueproduction
$HOSTNAMEAffected hostprod-server-01
$ALERT_IDUnique alert identifier123456789
$ALERT_CYCLE_KEYAlert cycle identifierabc123def456
$LINKLink to alert in Datadoghttps://app.datadoghq.com/...
$SNAPSHOTGraph snapshot URLhttps://p.datadoghq.com/...
$LAST_UPDATEDTimestamp of last update2025-07-01 10:15:30 UTC
$TAGSAll monitor tagsservice:api,env:prod
$ORG_IDDatadog organization ID12345
$ORG_NAMEOrganization nameMy Company

The current metric value and alert threshold are not exposed as webhook payload variables. Access them with the {{value}} and {{threshold}} Mustache variables in the monitor message body instead, then include the rendered message in the webhook payload through $TEXT_ONLY_MSG.

Advanced configuration​

Filter by alert status​

Send different payloads for alerts vs recoveries:

{{#is_alert}}
@webhook-harness-ai-sre
{{/is_alert}}

Route by priority​

Create separate Harness webhooks for different priorities:

{{#is_alert}}
  {{#is_priority 'P1'}}
  @webhook-harness-critical
  {{/is_priority}}

  {{#is_priority 'P2'}}
  @webhook-harness-standard
  {{/is_priority}}
{{/is_alert}}

Include metric values and thresholds​

The webhook payload does not carry the metric value or threshold directly. Datadog exposes those only as {{value}} and {{threshold}} template variables in the monitor message body, so include them there and pass the rendered message through to the webhook:

{
  "alert_name": "$EVENT_TITLE",
  "alert_message": "$TEXT_ONLY_MSG",
  "priority": "$PRIORITY",
  "service": "$SERVICE",
  "env": "$ENV",
  "alert_url": "$LINK",
  "context": {
    "query": "$ALERT_QUERY",
    "snapshot_url": "$SNAPSHOT"
  }
}

Monitor message body:

{{#is_alert}}
Current value: {{value}}
Threshold: {{threshold}}
{{/is_alert}}

Harness CEL mapping:

title: webhook.alert_name
message: webhook.alert_message
severity: webhook.priority == "P1" ? "critical" : "high"
link: webhook.context.snapshot_url

Troubleshooting​

Webhook not triggering​

Cause: Datadog webhook not configured or monitor not firing.

Solution:

  • Verify webhook appears in Integrations → Webhooks
  • Check monitor notification includes @webhook-harness-ai-sre
  • Test notification manually from monitor page
  • Review Datadog Event Explorer for webhook calls

HTTP 4xx errors​

Cause: Invalid webhook URL or authentication failure.

Solution:

  • Verify webhook URL is correct (no typos in webhook ID)
  • Check custom headers match Harness webhook configuration
  • Ensure Harness webhook is enabled
  • Test webhook URL with curl:
curl -X POST \
  -H "Content-Type: application/json" \
  -d '{"alert_name": "Test", "alert_message": "Test message"}' \
  https://your-harness-instance/gateway/ai-sre/api/webhooks/<webhook-id>

Variables not mapping correctly​

Cause: Field names in Harness configuration do not match Datadog payload.

Solution:

  • Review the actual webhook payload sent by Datadog (check Datadog webhook logs)
  • Ensure field names match exactly (case-sensitive)
  • Use CEL has() to check if fields exist:
filter: has(webhook.service) && webhook.service != ""

Rate limiting issues​

Cause: High-frequency Datadog alerts overwhelming Harness.

Solution:

  • Configure Datadog monitor to re-notify less frequently
  • Use Harness alert routing rules to deduplicate similar alerts
  • Add debounce logic in Harness webhook CEL:
// Only process if not seen in last 5 minutes
filter: !has(webhook.alert_cycle_key) || 
        (now - timestamp(webhook.last_updated)) > duration("5m")

Example: Complete integration​

This example shows a production-ready Datadog-to-Harness integration for a microservices platform.

Datadog webhook payload​

{
  "alert_name": "$EVENT_TITLE",
  "alert_message": "$TEXT_ONLY_MSG",
  "alert_status": "$ALERT_STATUS",
  "priority": "$PRIORITY",
  "metric": "$ALERT_METRIC",
  "service": "$SERVICE",
  "env": "$ENV",
  "alert_url": "$LINK",
  "snapshot": "$SNAPSHOT",
  "timestamp": "$LAST_UPDATED",
  "tags": "$TAGS"
}

Harness webhook field mapping​

title: "{{webhook.alert_name}}"
message: |
  {{webhook.alert_message}}
  
  Metric: {{webhook.metric}}
severity: |
  webhook.priority == "P1" ? "critical" :
  webhook.priority == "P2" ? "high" :
  webhook.priority == "P3" ? "medium" : "low"
source: "datadog"
link: "{{webhook.alert_url}}"
tags:
  - "source:datadog"
  - "service:{{webhook.service}}"
  - "env:{{webhook.env}}"
  - "metric:{{webhook.metric}}"
filter: |
  webhook.alert_status == "ALERT" && 
  webhook.env in ["production", "staging"]

Datadog monitor notification​

{{#is_alert}}
Production alert on {{service.name}}

Current: {{value}}
Threshold: {{threshold}}

@webhook-harness-ai-sre
@slack-platform-alerts
{{/is_alert}}

{{#is_recovery}}
✅ Recovered: {{service.name}}
@webhook-harness-ai-sre
{{/is_recovery}}

Next steps​

Further reading​

Datadog Official Documentation​