> For the complete documentation index, see [llms.txt](https://developer.harness.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://developer.harness.io/ai-sre/ai-sre-for-administrators/set-up-alert-management/webhooks/integration-guides/monitoring/prometheus.md).

# Prometheus AlertManager Integration Guide

Configure Prometheus AlertManager to send webhook notifications to Harness AI SRE when alerts fire.

### Before you begin <a href="#before-you-begin" id="before-you-begin"></a>

* **Harness webhook endpoint**: Create a Prometheus webhook in Harness AI SRE using the [Prometheus webhook template](/ai-sre/ai-sre-for-administrators/set-up-alert-management/webhooks/templates/monitoring/prometheus.md).
* **AlertManager access**: Permissions to modify AlertManager configuration.
* **Webhook URL**: Copy the webhook URL from your Harness webhook configuration.
* **AlertManager configuration documentation**: Go to [Prometheus AlertManager Configuration](https://prometheus.io/docs/alerting/latest/configuration/) to understand AlertManager setup and routing rules.
* **Webhook receiver reference**: Go to [Webhook Config](https://prometheus.io/docs/alerting/latest/configuration/#webhook_config) for webhook-specific configuration options.

***

### Configure AlertManager webhook receiver <a href="#configure-alertmanager-webhook-receiver" id="configure-alertmanager-webhook-receiver"></a>

#### Edit AlertManager configuration <a href="#edit-alertmanager-configuration" id="edit-alertmanager-configuration"></a>

AlertManager configuration is typically in `alertmanager.yml`. Add a webhook receiver configuration.

#### Add webhook receiver <a href="#add-webhook-receiver" id="add-webhook-receiver"></a>

{% tabs %}
{% tab title="Basic configuration" %}

```yaml
route:
  receiver: 'harness-ai-sre'
  routes:
    - match:
        severity: critical
      receiver: 'harness-ai-sre'

receivers:
  - name: 'harness-ai-sre'
    webhook_configs:
      - url: 'https://<your-harness-instance>/gateway/ai-sre/api/webhooks/<webhook-id>'
        send_resolved: true
```

{% endtab %}

{% tab title="With authentication" %}

```yaml
receivers:
  - name: 'harness-ai-sre'
    webhook_configs:
      - url: 'https://<your-harness-instance>/gateway/ai-sre/api/webhooks/<webhook-id>'
        send_resolved: true
        http_config:
          authorization:
            credentials: 'your-webhook-secret'
```

{% endtab %}

{% tab title="Advanced routing" %}

```yaml
route:
  receiver: 'default'
  group_by: ['alertname', 'cluster', 'service']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 12h
  routes:
    # Critical alerts to Harness
    - match:
        severity: critical
      receiver: 'harness-ai-sre'
      continue: true
    
    # Production alerts to Harness
    - match:
        environment: production
      receiver: 'harness-ai-sre'
      continue: true

receivers:
  - name: 'harness-ai-sre'
    webhook_configs:
      - url: 'https://<your-harness-instance>/gateway/ai-sre/api/webhooks/<webhook-id>'
        send_resolved: true
        max_alerts: 0  # Send all alerts
```

{% endtab %}
{% endtabs %}

#### Reload AlertManager configuration <a href="#reload-alertmanager-configuration" id="reload-alertmanager-configuration"></a>

After updating the configuration:

```bash
# Send SIGHUP to reload config <a href="#send-sighup-to-reload-config" id="send-sighup-to-reload-config"></a>
kill -HUP $(pidof alertmanager)

# Or use the API <a href="#or-use-the-api" id="or-use-the-api"></a>
curl -X POST http://localhost:9093/-/reload
```

***

### Configure field mapping in Harness <a href="#configure-field-mapping-in-harness" id="configure-field-mapping-in-harness"></a>

In your Harness webhook configuration, map AlertManager payload fields to alert properties.

#### AlertManager webhook payload structure <a href="#alertmanager-webhook-payload-structure" id="alertmanager-webhook-payload-structure"></a>

```json
{
  "version": "4",
  "receiver": "harness-ai-sre",
  "status": "firing",
  "truncatedAlerts": 0,
  "alerts": [
    {
      "status": "firing",
      "labels": {
        "alertname": "HighMemoryUsage",
        "severity": "critical",
        "instance": "localhost:9090",
        "job": "prometheus",
        "service": "api-gateway",
        "environment": "production"
      },
      "annotations": {
        "summary": "Memory usage above 90%",
        "description": "Instance localhost:9090 has memory usage of 95%"
      },
      "startsAt": "2025-07-01T10:30:00.000Z",
      "endsAt": "0001-01-01T00:00:00Z",
      "generatorURL": "http://prometheus:9090/graph?g0.expr=...",
      "fingerprint": "a1b2c3d4e5f6"
    }
  ],
  "groupLabels": {
    "alertname": "HighMemoryUsage"
  },
  "commonLabels": {
    "alertname": "HighMemoryUsage",
    "severity": "critical"
  },
  "commonAnnotations": {
    "summary": "Memory usage above 90%"
  },
  "externalURL": "http://alertmanager:9093"
}
```

#### Map basic fields <a href="#map-basic-fields" id="map-basic-fields"></a>

Use Mustache templates:

```yaml
title: "{{webhook.alerts[0].labels.alertname}}"
message: "{{webhook.alerts[0].annotations.description}}"
severity: "{{webhook.alerts[0].labels.severity}}"
source: "prometheus"
link: "{{webhook.alerts[0].generatorURL}}"
tags:
  - "alertname:{{webhook.alerts[0].labels.alertname}}"
  - "instance:{{webhook.alerts[0].labels.instance}}"
  - "service:{{webhook.alerts[0].labels.service}}"
  - "environment:{{webhook.alerts[0].labels.environment}}"
```

#### Advanced field mapping with CEL <a href="#advanced-field-mapping-with-cel" id="advanced-field-mapping-with-cel"></a>

```cel
// Extract first alert from the batch
title: size(webhook.alerts) > 0 ? webhook.alerts[0].labels.alertname : "Alert"
message: size(webhook.alerts) > 0 ? webhook.alerts[0].annotations.description : ""

// Map Prometheus severity to Harness severity
severity: size(webhook.alerts) > 0 && has(webhook.alerts[0].labels.severity)
  ? (webhook.alerts[0].labels.severity == "critical" ? "critical" :
     webhook.alerts[0].labels.severity == "warning" ? "high" : "medium")
  : "medium"

source: "prometheus"
link: size(webhook.alerts) > 0 ? webhook.alerts[0].generatorURL : ""

// Extract all labels as tags
tags: size(webhook.alerts) > 0 
  ? webhook.alerts[0].labels.keys().map(k, k + ":" + string(webhook.alerts[0].labels[k]))
  : []

// Filter: only process firing alerts
filter: webhook.status == "firing" && size(webhook.alerts) > 0
```

***

### Handle multiple alerts in batch <a href="#handle-multiple-alerts-in-batch" id="handle-multiple-alerts-in-batch"></a>

AlertManager sends alerts in batches. Handle multiple alerts:

#### Option 1: Process first alert only <a href="#option-1-process-first-alert-only" id="option-1-process-first-alert-only"></a>

Use the CEL above with `webhook.alerts[0]`.

#### Option 2: Process all alerts separately <a href="#option-2-process-all-alerts-separately" id="option-2-process-all-alerts-separately"></a>

Configure AlertManager to send one alert per webhook:

```yaml
receivers:
  - name: 'harness-ai-sre'
    webhook_configs:
      - url: 'https://<your-harness-instance>/gateway/ai-sre/api/webhooks/<webhook-id>'
        send_resolved: true
        max_alerts: 1  # Send one alert at a time
```

#### Option 3: Create summary alert <a href="#option-3-create-summary-alert" id="option-3-create-summary-alert"></a>

Combine multiple alerts into one:

```cel
title: "Prometheus Alert Batch: " + string(size(webhook.alerts)) + " alerts"
message: webhook.alerts.map(a, 
  a.labels.alertname + ": " + a.annotations.summary
).join("\n")
severity: webhook.commonLabels.severity
```

***

### Test the integration <a href="#test-the-integration" id="test-the-integration"></a>

#### Trigger a test alert <a href="#trigger-a-test-alert" id="trigger-a-test-alert"></a>

Create a test alert in Prometheus:

```yaml
# Add to prometheus.yml rules <a href="#add-to-prometheusyml-rules" id="add-to-prometheusyml-rules"></a>
groups:
  - name: test
    interval: 10s
    rules:
      - alert: TestAlert
        expr: vector(1)
        for: 0m
        labels:
          severity: warning
          service: test
        annotations:
          summary: Test alert for Harness integration
          description: This is a test alert
```

Reload Prometheus rules:

```bash
curl -X POST http://localhost:9090/-/reload
```

#### Verify in AlertManager <a href="#verify-in-alertmanager" id="verify-in-alertmanager"></a>

Check AlertManager UI at `http://localhost:9093` to see the alert.

#### Verify in Harness <a href="#verify-in-harness" id="verify-in-harness"></a>

Confirm the alert arrived and mapped correctly:

1. Navigate to **Alerts** in Harness AI SRE
2. Check that the test alert appears
3. Verify field mapping is correct

***

### Available AlertManager fields <a href="#available-alertmanager-fields" id="available-alertmanager-fields"></a>

| Field                   | Description                                    | Example                                            |
| ----------------------- | ---------------------------------------------- | -------------------------------------------------- |
| `version`               | Webhook payload format version                 | `4`                                                |
| `status`                | Alert batch status                             | `firing`, `resolved`                               |
| `truncatedAlerts`       | Number of alerts truncated due to `max_alerts` | `0`                                                |
| `alerts`                | Array of alerts                                | See payload structure                              |
| `alerts[].status`       | Individual alert status                        | `firing`, `resolved`                               |
| `alerts[].labels`       | Alert labels (key-value)                       | `{"alertname": "HighCPU", "severity": "critical"}` |
| `alerts[].annotations`  | Alert annotations                              | `{"summary": "...", "description": "..."}`         |
| `alerts[].startsAt`     | Alert start time                               | `2025-07-01T10:30:00.000Z`                         |
| `alerts[].endsAt`       | Alert end time (for resolved)                  | `2025-07-01T10:35:00.000Z`                         |
| `alerts[].generatorURL` | Link to Prometheus query                       | `http://prometheus:9090/graph?...`                 |
| `alerts[].fingerprint`  | Unique alert identifier                        | `a1b2c3d4e5f6`                                     |
| `groupLabels`           | Labels used for grouping                       | `{"alertname": "HighCPU"}`                         |
| `commonLabels`          | Labels common to all alerts                    | `{"severity": "critical"}`                         |
| `commonAnnotations`     | Annotations common to all                      | `{"summary": "..."}`                               |
| `externalURL`           | AlertManager URL                               | `http://alertmanager:9093`                         |

***

### Advanced configuration <a href="#advanced-configuration" id="advanced-configuration"></a>

#### Route by severity <a href="#route-by-severity" id="route-by-severity"></a>

Send only critical alerts to Harness:

```yaml
route:
  routes:
    - match:
        severity: critical
      receiver: 'harness-critical'
    - match:
        severity: warning
      receiver: 'harness-warning'

receivers:
  - name: 'harness-critical'
    webhook_configs:
      - url: 'https://harness/webhooks/critical-webhook-id'
  
  - name: 'harness-warning'
    webhook_configs:
      - url: 'https://harness/webhooks/warning-webhook-id'
```

#### Filter by label <a href="#filter-by-label" id="filter-by-label"></a>

Send alerts matching specific labels:

```yaml
route:
  routes:
    - match:
        team: platform
        environment: production
      receiver: 'harness-ai-sre'
```

#### Inhibition rules <a href="#inhibition-rules" id="inhibition-rules"></a>

Prevent lower-severity alerts when critical alerts are firing:

```yaml
inhibit_rules:
  - source_match:
      severity: critical
    target_match:
      severity: warning
    equal: ['alertname', 'instance']
```

***

### Troubleshooting <a href="#troubleshooting" id="troubleshooting"></a>

<details>

<summary>Prometheus AlertManager webhook is not sending alerts to Harness AI SRE</summary>

Check the AlertManager logs, verify the configuration syntax with amtool check-config, and test the webhook URL manually with curl.

</details>

<details>

<summary>Prometheus AlertManager alerts are not appearing in Harness AI SRE</summary>

Check the Harness webhook logs for errors, verify the CEL filter logic by removing the filter temporarily, and inspect the raw payload in the AlertManager webhook logs.

</details>

<details>

<summary>Prometheus AlertManager is creating multiple duplicate alerts in Harness AI SRE</summary>

Adjust the AlertManager group\_by and group\_interval settings, and use Harness alert routing rules to deduplicate by fingerprint.

</details>

<details>

<summary>Resolved Prometheus AlertManager alerts are not clearing in Harness AI SRE</summary>

Set send\_resolved: true in the AlertManager webhook config, and handle the resolved status in the Harness CEL severity mapping.

</details>

***

### Example: complete integration <a href="#example-complete-integration" id="example-complete-integration"></a>

#### AlertManager configuration <a href="#alertmanager-configuration" id="alertmanager-configuration"></a>

```yaml
global:
  resolve_timeout: 5m

route:
  receiver: 'default'
  group_by: ['alertname', 'cluster', 'service']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 12h
  routes:
    - match_re:
        severity: ^(critical|warning)$
      receiver: 'harness-ai-sre'
      continue: true

receivers:
  - name: 'default'
    webhook_configs:
      - url: 'http://localhost:5001/default'
  
  - name: 'harness-ai-sre'
    webhook_configs:
      - url: 'https://app.harness.io/gateway/ai-sre/api/webhooks/wh_abc123'
        send_resolved: true
        max_alerts: 1
        http_config:
          follow_redirects: true

inhibit_rules:
  - source_match:
      severity: critical
    target_match:
      severity: warning
    equal: ['alertname', 'cluster', 'service']
```

#### Map fields in the Harness webhook <a href="#map-fields-in-the-harness-webhook" id="map-fields-in-the-harness-webhook"></a>

```yaml
title: |
  webhook.alerts[0].labels.alertname + 
  " (" + webhook.alerts[0].labels.service + ")"

message: |
  **Alert**: {{webhook.alerts[0].labels.alertname}}
  **Status**: {{webhook.status}}
  **Severity**: {{webhook.alerts[0].labels.severity}}
  **Instance**: {{webhook.alerts[0].labels.instance}}
  **Service**: {{webhook.alerts[0].labels.service}}
  **Environment**: {{webhook.alerts[0].labels.environment}}
  
  **Summary**: {{webhook.alerts[0].annotations.summary}}
  **Description**: {{webhook.alerts[0].annotations.description}}
  
  **Started At**: {{webhook.alerts[0].startsAt}}
  **Generator URL**: {{webhook.alerts[0].generatorURL}}

severity: |
  webhook.status == "resolved" ? "info" :
  webhook.alerts[0].labels.severity == "critical" ? "critical" :
  webhook.alerts[0].labels.severity == "warning" ? "high" : "medium"

source: "prometheus"
link: "{{webhook.alerts[0].generatorURL}}"

tags:
  - "source:prometheus"
  - "alertname:{{webhook.alerts[0].labels.alertname}}"
  - "severity:{{webhook.alerts[0].labels.severity}}"
  - "service:{{webhook.alerts[0].labels.service}}"
  - "environment:{{webhook.alerts[0].labels.environment}}"
  - "instance:{{webhook.alerts[0].labels.instance}}"
  - "status:{{webhook.status}}"
  - "fingerprint:{{webhook.alerts[0].fingerprint}}"

filter: |
  webhook.status == "firing" && 
  size(webhook.alerts) > 0 &&
  has(webhook.alerts[0].labels.severity)

custom_fields:
  fingerprint: "{{webhook.alerts[0].fingerprint}}"
  generator_url: "{{webhook.alerts[0].generatorURL}}"
  alertmanager_url: "{{webhook.externalURL}}"
```

***

### Next steps <a href="#next-steps" id="next-steps"></a>

* [Route alerts](/ai-sre/ai-sre-for-administrators/set-up-alert-management/alert-rules/overview.md): Route and deduplicate Prometheus alerts.
* [Use CEL in webhooks](/ai-sre/ai-sre-for-administrators/set-up-alert-management/webhooks/use-cel-webhooks.md): Add advanced filtering and batching logic.
* [AI agent](/ai-sre/ai-sre-for-incident-responders/use-ai-agents/ai-agent.md): Enable automated alert investigation.
* [Prometheus template](/ai-sre/ai-sre-for-administrators/set-up-alert-management/webhooks/templates/monitoring/prometheus.md): Use the pre-configured template.

***

### Related documentation <a href="#related-documentation" id="related-documentation"></a>

#### Prometheus official documentation <a href="#prometheus-official-documentation" id="prometheus-official-documentation"></a>

* [AlertManager configuration](https://prometheus.io/docs/alerting/latest/configuration/): Complete guide to AlertManager configuration, webhook receivers, and routing rules.
* [Webhook config](https://prometheus.io/docs/alerting/latest/configuration/#webhook_config): Webhook receiver configuration options (`url`, `send_resolved`, `max_alerts`, `http_config`).
* [Notifications](https://prometheus.io/docs/alerting/latest/notifications/): Webhook payload format, alerts array structure, labels, and annotations.
* [Notification examples](https://prometheus.io/docs/alerting/latest/notification_examples/): Example webhook payloads and field names.