Skip to main content

Grafana Integration Guide

Last updated on

Configure Grafana Unified Alerting to send webhook notifications to Harness AI SRE when alerts fire.

Before you begin​

  • Harness webhook endpoint: Create a Grafana webhook in Harness AI SRE using the Grafana webhook template.
  • Grafana access: Admin permissions to configure contact points and notification policies.
  • Webhook URL: Copy the webhook URL from your Harness webhook configuration.
  • Grafana version: This guide covers Grafana 8.0+ with Unified Alerting. For legacy alerting, go to the Grafana documentation.
  • Unified Alerting documentation: Go to Grafana Unified Alerting to understand contact points and notification policies.
  • Webhook configuration reference: Go to Manage Contact Points for webhook setup details.

Create webhook contact point​

Open the contact points configuration in Grafana:

  1. In Grafana, go to Alerting, then select Contact points
  2. Click New contact point

Configure webhook contact point​

  • Name: Harness AI SRE
  • Integration: Select Webhook
  • URL: Your Harness webhook URL 
    https://<your-harness-instance>/gateway/ai-sre/api/webhooks/<webhook-id>
  • HTTP Method: POST

Test contact point​

Click Test to send a test notification and verify the webhook is reachable.

Save contact point​

Click Save contact point to create the webhook integration.

Configure notification policy​

Edit default notification policy​

Route alerts to the Harness contact point:

  1. Go to Alerting, then select Notification policies
  2. Edit the Default policy or create a new nested policy
  3. Select Harness AI SRE as the contact point

Create specific routing policy​

For selective alert routing:

  1. Click New nested policy
  2. Configure matching labels:
    • Label: severity
    • Operator: =
    • Value: critical
  3. Select Harness AI SRE contact point
  4. Click Save policy

Configure field mapping in Harness​

In your Harness webhook configuration, map Grafana payload fields to alert properties.

Grafana webhook payload structure​

{
  "receiver": "Harness AI SRE",
  "status": "firing",
  "alerts": [
    {
      "status": "firing",
      "labels": {
        "alertname": "High CPU Usage",
        "grafana_folder": "Production",
        "severity": "critical",
        "instance": "prod-server-01",
        "job": "node-exporter"
      },
      "annotations": {
        "summary": "CPU usage above 90%",
        "description": "Server prod-server-01 has CPU usage at 95%",
        "runbook_url": "https://wiki.company.com/runbooks/high-cpu"
      },
      "startsAt": "2025-07-01T10:30:00.000Z",
      "endsAt": "0001-01-01T00:00:00Z",
      "generatorURL": "https://grafana.example.com/alerting/grafana/abc123/view",
      "fingerprint": "a1b2c3d4e5f6",
      "silenceURL": "https://grafana.example.com/alerting/silence/new?...",
      "dashboardURL": "https://grafana.example.com/d/dashboard-id",
      "panelURL": "https://grafana.example.com/d/dashboard-id?viewPanel=1",
      "values": {
        "B": 95.2
      }
    }
  ],
  "groupLabels": {
    "alertname": "High CPU Usage"
  },
  "commonLabels": {
    "alertname": "High CPU Usage",
    "severity": "critical"
  },
  "commonAnnotations": {
    "summary": "CPU usage above 90%"
  },
  "externalURL": "https://grafana.example.com",
  "version": "1",
  "groupKey": "{}:{alertname=\"High CPU Usage\"}",
  "truncatedAlerts": 0,
  "orgId": 1,
  "title": "[FIRING:1] High CPU Usage (prod-server-01)",
  "state": "alerting",
  "message": "**Firing**\n\nValue: [ var='A' labels={instance=prod-server-01} value=95.2 ]\nLabels:\n - alertname = High CPU Usage\n - severity = critical\nAnnotations:\n - summary = CPU usage above 90%\n"
}

Map basic fields​

Use Mustache templates:

title: "{{webhook.title}}"
message: "{{webhook.message}}"
severity: "{{webhook.commonLabels.severity}}"
source: "grafana"
link: "{{webhook.externalURL}}"
tags:
  - "alertname:{{webhook.alerts[0].labels.alertname}}"
  - "grafana_folder:{{webhook.alerts[0].labels.grafana_folder}}"
  - "instance:{{webhook.alerts[0].labels.instance}}"

Advanced field mapping with CEL​

// Use Grafana's pre-formatted title
title: webhook.title

// Use Grafana's pre-formatted message or build custom
message: size(webhook.alerts) > 0 
  ? webhook.alerts[0].annotations.description + "\n\n" +
    "Dashboard: " + webhook.alerts[0].dashboardURL + "\n" +
    "Panel: " + webhook.alerts[0].panelURL
  : webhook.message

// Map Grafana severity to Harness severity
severity: has(webhook.commonLabels.severity)
  ? (webhook.commonLabels.severity == "critical" ? "critical" :
     webhook.commonLabels.severity == "warning" ? "high" : "medium")
  : "medium"

source: "grafana"
link: size(webhook.alerts) > 0 ? webhook.alerts[0].generatorURL : webhook.externalURL

// Extract all labels as tags
tags: size(webhook.alerts) > 0
  ? webhook.alerts[0].labels.keys().map(k, k + ":" + string(webhook.alerts[0].labels[k]))
  : []

// Filter: only firing alerts
filter: webhook.status == "firing" && size(webhook.alerts) > 0

Test the integration​

Create test alert rule​

Create a rule that fires immediately to confirm delivery:

  1. Go to Alerting, then select Alert rules
  2. Click New alert rule
  3. Configure a simple test rule:
    • Alert rule name: Test Harness Integration
    • Query: Use a test query that always fires
    • Condition: Set threshold that will trigger
    • Evaluate every: 1m
    • For: 0m
  4. Add labels:
    • severity: warning
    • test: true
  5. Add annotations:
    • summary: Test alert for Harness integration
  6. Select notification policy or contact point
  7. Click Save rule and exit

Verify in Grafana​

Confirm the rule fired and the notification was sent:

  1. Go to Alerting, then select Alert rules
  2. Wait for the rule to evaluate and fire
  3. Check Contact points, then Harness AI SRE, for recent notifications

Verify in Harness​

Confirm the alert arrived and mapped correctly:

  1. Navigate to Alerts in Harness AI SRE
  2. Check that the test alert appears
  3. Verify field mapping is correct

Available Grafana fields​

FieldDescriptionExample
receiverContact point nameHarness AI SRE
statusAlert statusfiring, resolved
alertsArray of firing alertsSee payload structure
alerts[].labelsAlert labels{"alertname": "...", "severity": "critical"}
alerts[].annotationsAlert annotations{"summary": "...", "description": "..."}
alerts[].startsAtAlert start time2025-07-01T10:30:00.000Z
alerts[].endsAtAlert end time0001-01-01T00:00:00Z (if not resolved)
alerts[].generatorURLLink to alert rulehttps://grafana.../alerting/.../view
alerts[].dashboardURLLink to dashboardhttps://grafana.../d/dashboard-id
alerts[].panelURLLink to specific panelhttps://grafana.../d/dashboard-id?viewPanel=1
alerts[].fingerprintUnique alert IDa1b2c3d4e5f6
alerts[].valuesQuery result values, keyed by refID{"B": 95.2}
titlePre-formatted alert title[FIRING:1] High CPU Usage
messagePre-formatted alert messageMulti-line formatted message
groupLabelsLabels used for grouping{"alertname": "..."}
commonLabelsLabels common to all alerts{"severity": "critical"}
commonAnnotationsAnnotations common to all{"summary": "..."}
externalURLGrafana instance URLhttps://grafana.example.com

Advanced configuration​

Route by folder​

Send alerts from specific Grafana folders to different Harness webhooks:

Grafana notification policy:

  1. Create nested policy with label matcher:
    • Label: grafana_folder
    • Operator: =
    • Value: Production
  2. Select appropriate contact point

Route by data source​

Label alerts by data source and route accordingly:

Alert rule labels:

labels:
  severity: critical
  datasource: prometheus
  team: platform

Notification policy matcher:

  • Label: datasource
  • Operator: =
  • Value: prometheus

Silence alerts​

Use Grafana's silence URL in Harness:

// Include silence link in message
message: webhook.alerts[0].annotations.description + "\n\n" +
         "Silence this alert: " + webhook.alerts[0].silenceURL

Multi-dimensional alerts​

Grafana supports multi-dimensional alerting. Access all dimensions:

// Extract all label dimensions
tags: size(webhook.alerts) > 0
  ? webhook.alerts[0].labels.keys().map(k, k + ":" + string(webhook.alerts[0].labels[k]))
  : []

Troubleshooting​

Grafana webhook is not sending notifications to Harness AI SRE

Test the contact point from the Grafana UI, check the Grafana logs, verify the webhook URL is accessible from the Grafana server, and check for a firewall or proxy blocking outbound requests.

Grafana alerts are not appearing in Harness AI SRE

Check the Harness webhook logs for errors, inspect the raw payload in the Grafana contact point test, verify the CEL filter logic by temporarily removing the filter, and ensure the required fields exist in the payload.

Resolved Grafana alerts are not clearing in Harness AI SRE

Ensure 'Disable resolved message' is unchecked in the contact point, and handle the resolved status in the Harness CEL mapping.

Grafana is creating duplicate alerts in Harness AI SRE

Review the notification policy routes to ensure no overlapping matchers, and adjust the group wait, group interval, and repeat interval in the notification policy.

Example: complete integration​

Grafana contact point configuration​

Configure the contact point with these settings:

  • Name: Harness AI SRE
  • Integration: Webhook
  • URL: https://app.harness.io/gateway/ai-sre/api/webhooks/wh_abc123
  • HTTP Method: POST
  • Disable resolved message: Unchecked

Grafana notification policy​

Root policy
├─ Default contact point: Harness AI SRE
├─ Group by: [alertname, grafana_folder]
├─ Group wait: 10s
├─ Group interval: 5m
├─ Repeat interval: 4h
└─ Nested policies:
   ├─ severity = critical → Harness AI SRE (immediate)
   └─ grafana_folder = Production → Harness AI SRE

Map fields in the Harness webhook​

title: "{{webhook.title}}"

message: |
  **Alert**: {{webhook.alerts[0].labels.alertname}}
  **Status**: {{webhook.status}}
  **Severity**: {{webhook.commonLabels.severity}}
  **Folder**: {{webhook.alerts[0].labels.grafana_folder}}
  
  **Summary**: {{webhook.alerts[0].annotations.summary}}
  {{webhook.alerts[0].annotations.description}}
  
  **Value**: {{webhook.alerts[0].values.B}}
  **Started At**: {{webhook.alerts[0].startsAt}}
  
  **Dashboard**: {{webhook.alerts[0].dashboardURL}}
  **Panel**: {{webhook.alerts[0].panelURL}}
  **Alert Rule**: {{webhook.alerts[0].generatorURL}}

severity: |
  webhook.status == "resolved" ? "info" :
  webhook.commonLabels.severity == "critical" ? "critical" :
  webhook.commonLabels.severity == "warning" ? "high" : "medium"

source: "grafana"
link: "{{webhook.alerts[0].dashboardURL}}"

tags:
  - "source:grafana"
  - "alertname:{{webhook.alerts[0].labels.alertname}}"
  - "severity:{{webhook.commonLabels.severity}}"
  - "folder:{{webhook.alerts[0].labels.grafana_folder}}"
  - "status:{{webhook.status}}"
  - "fingerprint:{{webhook.alerts[0].fingerprint}}"

filter: |
  size(webhook.alerts) > 0 &&
  webhook.status == "firing"

custom_fields:
  fingerprint: "{{webhook.alerts[0].fingerprint}}"
  dashboard_url: "{{webhook.alerts[0].dashboardURL}}"
  panel_url: "{{webhook.alerts[0].panelURL}}"
  generator_url: "{{webhook.alerts[0].generatorURL}}"
  grafana_instance: "{{webhook.externalURL}}"

Next steps​

Grafana official documentation​