Prometheus AlertManager Integration Guide

Last updated on Aug 7, 2026

Configure Prometheus AlertManager to send webhook notifications to Harness AI SRE when alerts fire.

Before you begin

Harness webhook endpoint: Create a Prometheus webhook in Harness AI SRE using the Prometheus webhook template.
AlertManager access: Permissions to modify AlertManager configuration.
Webhook URL: Copy the webhook URL from your Harness webhook configuration.
AlertManager configuration documentation: Go to Prometheus AlertManager Configuration to understand AlertManager setup and routing rules.
Webhook receiver reference: Go to Webhook Config for webhook-specific configuration options.

Configure AlertManager webhook receiver

Edit AlertManager configuration

AlertManager configuration is typically in alertmanager.yml. Add a webhook receiver configuration.

Add webhook receiver

Basic configuration
With authentication
Advanced routing

route:
  receiver: 'harness-ai-sre'
  routes:
    - match:
        severity: critical
      receiver: 'harness-ai-sre'

receivers:
  - name: 'harness-ai-sre'
    webhook_configs:
      - url: 'https://<your-harness-instance>/gateway/ai-sre/api/webhooks/<webhook-id>'
        send_resolved: true

receivers:
  - name: 'harness-ai-sre'
    webhook_configs:
      - url: 'https://<your-harness-instance>/gateway/ai-sre/api/webhooks/<webhook-id>'
        send_resolved: true
        http_config:
          authorization:
            credentials: 'your-webhook-secret'

route:
  receiver: 'default'
  group_by: ['alertname', 'cluster', 'service']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 12h
  routes:
    # Critical alerts to Harness
    - match:
        severity: critical
      receiver: 'harness-ai-sre'
      continue: true
    
    # Production alerts to Harness
    - match:
        environment: production
      receiver: 'harness-ai-sre'
      continue: true

receivers:
  - name: 'harness-ai-sre'
    webhook_configs:
      - url: 'https://<your-harness-instance>/gateway/ai-sre/api/webhooks/<webhook-id>'
        send_resolved: true
        max_alerts: 0  # Send all alerts

Reload AlertManager configuration

After updating the configuration:

# Send SIGHUP to reload config
kill -HUP $(pidof alertmanager)

# Or use the API
curl -X POST http://localhost:9093/-/reload

Configure field mapping in Harness

In your Harness webhook configuration, map AlertManager payload fields to alert properties.

AlertManager webhook payload structure

{
  "version": "4",
  "receiver": "harness-ai-sre",
  "status": "firing",
  "truncatedAlerts": 0,
  "alerts": [
    {
      "status": "firing",
      "labels": {
        "alertname": "HighMemoryUsage",
        "severity": "critical",
        "instance": "localhost:9090",
        "job": "prometheus",
        "service": "api-gateway",
        "environment": "production"
      },
      "annotations": {
        "summary": "Memory usage above 90%",
        "description": "Instance localhost:9090 has memory usage of 95%"
      },
      "startsAt": "2025-07-01T10:30:00.000Z",
      "endsAt": "0001-01-01T00:00:00Z",
      "generatorURL": "http://prometheus:9090/graph?g0.expr=...",
      "fingerprint": "a1b2c3d4e5f6"
    }
  ],
  "groupLabels": {
    "alertname": "HighMemoryUsage"
  },
  "commonLabels": {
    "alertname": "HighMemoryUsage",
    "severity": "critical"
  },
  "commonAnnotations": {
    "summary": "Memory usage above 90%"
  },
  "externalURL": "http://alertmanager:9093"
}

Map basic fields

Use Mustache templates:

title: "{{webhook.alerts[0].labels.alertname}}"
message: "{{webhook.alerts[0].annotations.description}}"
severity: "{{webhook.alerts[0].labels.severity}}"
source: "prometheus"
link: "{{webhook.alerts[0].generatorURL}}"
tags:
  - "alertname:{{webhook.alerts[0].labels.alertname}}"
  - "instance:{{webhook.alerts[0].labels.instance}}"
  - "service:{{webhook.alerts[0].labels.service}}"
  - "environment:{{webhook.alerts[0].labels.environment}}"

Advanced field mapping with CEL

// Extract first alert from the batch
title: size(webhook.alerts) > 0 ? webhook.alerts[0].labels.alertname : "Alert"
message: size(webhook.alerts) > 0 ? webhook.alerts[0].annotations.description : ""

// Map Prometheus severity to Harness severity
severity: size(webhook.alerts) > 0 && has(webhook.alerts[0].labels.severity)
  ? (webhook.alerts[0].labels.severity == "critical" ? "critical" :
     webhook.alerts[0].labels.severity == "warning" ? "high" : "medium")
  : "medium"

source: "prometheus"
link: size(webhook.alerts) > 0 ? webhook.alerts[0].generatorURL : ""

// Extract all labels as tags
tags: size(webhook.alerts) > 0 
  ? webhook.alerts[0].labels.keys().map(k, k + ":" + string(webhook.alerts[0].labels[k]))
  : []

// Filter: only process firing alerts
filter: webhook.status == "firing" && size(webhook.alerts) > 0

Handle multiple alerts in batch

AlertManager sends alerts in batches. Handle multiple alerts:

Option 1: Process first alert only

Use the CEL above with webhook.alerts[0].

Option 2: Process all alerts separately

Configure AlertManager to send one alert per webhook:

receivers:
  - name: 'harness-ai-sre'
    webhook_configs:
      - url: 'https://<your-harness-instance>/gateway/ai-sre/api/webhooks/<webhook-id>'
        send_resolved: true
        max_alerts: 1  # Send one alert at a time

Option 3: Create summary alert

Combine multiple alerts into one:

title: "Prometheus Alert Batch: " + string(size(webhook.alerts)) + " alerts"
message: webhook.alerts.map(a, 
  a.labels.alertname + ": " + a.annotations.summary
).join("\n")
severity: webhook.commonLabels.severity

Test the integration

Trigger a test alert

Create a test alert in Prometheus:

# Add to prometheus.yml rules
groups:
  - name: test
    interval: 10s
    rules:
      - alert: TestAlert
        expr: vector(1)
        for: 0m
        labels:
          severity: warning
          service: test
        annotations:
          summary: Test alert for Harness integration
          description: This is a test alert

Reload Prometheus rules:

curl -X POST http://localhost:9090/-/reload

Verify in AlertManager

Check AlertManager UI at http://localhost:9093 to see the alert.

Verify in Harness

Confirm the alert arrived and mapped correctly:

Navigate to Alerts in Harness AI SRE
Check that the test alert appears
Verify field mapping is correct

Available AlertManager fields

Field	Description	Example
`version`	Webhook payload format version	`4`
`status`	Alert batch status	`firing`, `resolved`
`truncatedAlerts`	Number of alerts truncated due to `max_alerts`	`0`
`alerts`	Array of alerts	See payload structure
`alerts[].status`	Individual alert status	`firing`, `resolved`
`alerts[].labels`	Alert labels (key-value)	`{"alertname": "HighCPU", "severity": "critical"}`
`alerts[].annotations`	Alert annotations	`{"summary": "...", "description": "..."}`
`alerts[].startsAt`	Alert start time	`2025-07-01T10:30:00.000Z`
`alerts[].endsAt`	Alert end time (for resolved)	`2025-07-01T10:35:00.000Z`
`alerts[].generatorURL`	Link to Prometheus query	`http://prometheus:9090/graph?...`
`alerts[].fingerprint`	Unique alert identifier	`a1b2c3d4e5f6`
`groupLabels`	Labels used for grouping	`{"alertname": "HighCPU"}`
`commonLabels`	Labels common to all alerts	`{"severity": "critical"}`
`commonAnnotations`	Annotations common to all	`{"summary": "..."}`
`externalURL`	AlertManager URL	`http://alertmanager:9093`

Advanced configuration

Route by severity

Send only critical alerts to Harness:

route:
  routes:
    - match:
        severity: critical
      receiver: 'harness-critical'
    - match:
        severity: warning
      receiver: 'harness-warning'

receivers:
  - name: 'harness-critical'
    webhook_configs:
      - url: 'https://harness/webhooks/critical-webhook-id'
  
  - name: 'harness-warning'
    webhook_configs:
      - url: 'https://harness/webhooks/warning-webhook-id'

Filter by label

Send alerts matching specific labels:

route:
  routes:
    - match:
        team: platform
        environment: production
      receiver: 'harness-ai-sre'

Inhibition rules

Prevent lower-severity alerts when critical alerts are firing:

inhibit_rules:
  - source_match:
      severity: critical
    target_match:
      severity: warning
    equal: ['alertname', 'instance']

Troubleshooting

Prometheus AlertManager webhook is not sending alerts to Harness AI SRE

Check the AlertManager logs, verify the configuration syntax with amtool check-config, and test the webhook URL manually with curl.

Prometheus AlertManager alerts are not appearing in Harness AI SRE

Check the Harness webhook logs for errors, verify the CEL filter logic by removing the filter temporarily, and inspect the raw payload in the AlertManager webhook logs.

Prometheus AlertManager is creating multiple duplicate alerts in Harness AI SRE

Adjust the AlertManager group_by and group_interval settings, and use Harness alert routing rules to deduplicate by fingerprint.

Resolved Prometheus AlertManager alerts are not clearing in Harness AI SRE

Set send_resolved: true in the AlertManager webhook config, and handle the resolved status in the Harness CEL severity mapping.

Example: complete integration

AlertManager configuration

global:
  resolve_timeout: 5m

route:
  receiver: 'default'
  group_by: ['alertname', 'cluster', 'service']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 12h
  routes:
    - match_re:
        severity: ^(critical|warning)$
      receiver: 'harness-ai-sre'
      continue: true

receivers:
  - name: 'default'
    webhook_configs:
      - url: 'http://localhost:5001/default'
  
  - name: 'harness-ai-sre'
    webhook_configs:
      - url: 'https://app.harness.io/gateway/ai-sre/api/webhooks/wh_abc123'
        send_resolved: true
        max_alerts: 1
        http_config:
          follow_redirects: true

inhibit_rules:
  - source_match:
      severity: critical
    target_match:
      severity: warning
    equal: ['alertname', 'cluster', 'service']

Map fields in the Harness webhook

title: |
  webhook.alerts[0].labels.alertname + 
  " (" + webhook.alerts[0].labels.service + ")"

message: |
  **Alert**: {{webhook.alerts[0].labels.alertname}}
  **Status**: {{webhook.status}}
  **Severity**: {{webhook.alerts[0].labels.severity}}
  **Instance**: {{webhook.alerts[0].labels.instance}}
  **Service**: {{webhook.alerts[0].labels.service}}
  **Environment**: {{webhook.alerts[0].labels.environment}}
  
  **Summary**: {{webhook.alerts[0].annotations.summary}}
  **Description**: {{webhook.alerts[0].annotations.description}}
  
  **Started At**: {{webhook.alerts[0].startsAt}}
  **Generator URL**: {{webhook.alerts[0].generatorURL}}

severity: |
  webhook.status == "resolved" ? "info" :
  webhook.alerts[0].labels.severity == "critical" ? "critical" :
  webhook.alerts[0].labels.severity == "warning" ? "high" : "medium"

source: "prometheus"
link: "{{webhook.alerts[0].generatorURL}}"

tags:
  - "source:prometheus"
  - "alertname:{{webhook.alerts[0].labels.alertname}}"
  - "severity:{{webhook.alerts[0].labels.severity}}"
  - "service:{{webhook.alerts[0].labels.service}}"
  - "environment:{{webhook.alerts[0].labels.environment}}"
  - "instance:{{webhook.alerts[0].labels.instance}}"
  - "status:{{webhook.status}}"
  - "fingerprint:{{webhook.alerts[0].fingerprint}}"

filter: |
  webhook.status == "firing" && 
  size(webhook.alerts) > 0 &&
  has(webhook.alerts[0].labels.severity)

custom_fields:
  fingerprint: "{{webhook.alerts[0].fingerprint}}"
  generator_url: "{{webhook.alerts[0].generatorURL}}"
  alertmanager_url: "{{webhook.externalURL}}"

Next steps

Route alerts: Route and deduplicate Prometheus alerts.
Use CEL in webhooks: Add advanced filtering and batching logic.
AI agent: Enable automated alert investigation.
Prometheus template: Use the pre-configured template.

Prometheus official documentation

AlertManager configuration: Complete guide to AlertManager configuration, webhook receivers, and routing rules.
Webhook config: Webhook receiver configuration options (url, send_resolved, max_alerts, http_config).
Notifications: Webhook payload format, alerts array structure, labels, and annotations.
Notification examples: Example webhook payloads and field names.

Before you begin​

Configure AlertManager webhook receiver​

Edit AlertManager configuration​

Add webhook receiver​

Reload AlertManager configuration​

Configure field mapping in Harness​

AlertManager webhook payload structure​

Map basic fields​

Advanced field mapping with CEL​

Handle multiple alerts in batch​

Option 1: Process first alert only​

Option 2: Process all alerts separately​

Option 3: Create summary alert​

Test the integration​

Trigger a test alert​

Verify in AlertManager​

Verify in Harness​

Available AlertManager fields​

Advanced configuration​

Route by severity​

Filter by label​

Inhibition rules​

Troubleshooting​

Example: complete integration​

AlertManager configuration​

Map fields in the Harness webhook​

Next steps​

Related documentation​

Prometheus official documentation​