> For the complete documentation index, see [llms.txt](https://developer.harness.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://developer.harness.io/ai-sre/ai-sre-for-administrators/set-up-alert-management/overview.md).

# Alert Management

Receive, route, and respond to alerts from monitoring systems.

Harness AI SRE provides flexible alert management to receive, route, and respond to alerts from any monitoring system or custom application.

### Overview <a href="#overview" id="overview"></a>

Alert management in Harness AI SRE enables you to:

* **Receive alerts from any source:** Webhook integrations support 25+ monitoring tools with pre-configured templates, plus custom webhooks for any system that can send HTTP requests.
* **Route alerts intelligently:** Route alerts based on service, environment, team, severity, or custom fields.
* **Enrich alerts with context:** Automatically add service metadata, team information, historical data, and related incidents.
* **Automate responses:** Trigger runbooks, create incidents, send notifications, and execute remediation actions.

***

### How alerts work <a href="#how-alerts-work" id="how-alerts-work"></a>

#### Alert ingestion <a href="#alert-ingestion" id="alert-ingestion"></a>

Alerts enter Harness AI SRE through **webhook integrations**:

1. **External monitoring system** (Datadog, PagerDuty, Prometheus, and others) detects an issue.
2. **Webhook POST** sends the alert payload to your unique webhook URL.
3. **Field extraction** parses the JSON payload using JSONPath expressions.
4. **Field mapping** populates alert properties using Mustache templates.
5. **Alert created** in Harness AI SRE with enriched context.

Go to [Ingest Alerts](/ai-sre/ai-sre-for-administrators/set-up-alert-management/webhooks/overview.md) to set up webhook integrations.

#### Alert routing rules <a href="#alert-routing-rules" id="alert-routing-rules"></a>

Once alerts are received, **alert routing rules** determine how they are processed:

* **Route to services:** Automatically link alerts to the affected service.
* **Assign to teams:** Route alerts to the responsible on-call team.
* **Create incidents:** Automatically create incidents for critical alerts.
* **Trigger runbooks:** Execute automated remediation workflows.
* **Suppress duplicates:** Deduplicate alerts based on custom rules.

Go to [Route Alerts](/ai-sre/ai-sre-for-administrators/set-up-alert-management/alert-rules/overview.md) to set up routing and automation.

***

### Alert sources <a href="#alert-sources" id="alert-sources"></a>

#### Webhook integrations <a href="#webhook-integrations" id="webhook-integrations"></a>

Harness AI SRE uses **webhook integrations** to receive alerts from monitoring systems. Webhooks provide:

* **25+ pre-configured templates:** Ready-to-use integrations for popular monitoring tools.
* **Custom webhooks:** Support for any system that can send HTTP POST requests.
* **Flexible payload mapping:** JSONPath extraction and Mustache templates for field mapping.
* **Dual trigger methods:** HTTP POST endpoint or email address for legacy systems.

**Supported monitoring tools (webhook templates)**

**Application performance monitoring:**

* [Datadog](/ai-sre/ai-sre-for-administrators/set-up-alert-management/webhooks/templates/monitoring/datadog.md)
* [New Relic](/ai-sre/ai-sre-for-administrators/set-up-alert-management/webhooks/templates/monitoring/new-relic.md)
* [Dynatrace](/ai-sre/ai-sre-for-administrators/set-up-alert-management/webhooks/templates/monitoring/dynatrace.md)
* [Sentry](/ai-sre/ai-sre-for-administrators/set-up-alert-management/webhooks/templates/monitoring/sentry.md)

**Infrastructure monitoring:**

* [Prometheus (AlertManager)](/ai-sre/ai-sre-for-administrators/set-up-alert-management/webhooks/templates/monitoring/prometheus.md)
* [Grafana](/ai-sre/ai-sre-for-administrators/set-up-alert-management/webhooks/templates/monitoring/grafana.md)
* [AWS CloudWatch](/ai-sre/ai-sre-for-administrators/set-up-alert-management/webhooks/templates/cloud/aws-cloudwatch.md)

**Incident management:**

* [PagerDuty](/ai-sre/ai-sre-for-administrators/set-up-alert-management/webhooks/templates/monitoring/pagerduty.md)
* [Opsgenie](/ai-sre/ai-sre-for-administrators/set-up-alert-management/webhooks/templates/monitoring/opsgenie.md)
* [Grafana Incident](/ai-sre/ai-sre-for-administrators/set-up-alert-management/webhooks/templates/monitoring/grafana-incident.md)

**Security and compliance:**

* [Lacework](/ai-sre/ai-sre-for-administrators/set-up-alert-management/webhooks/templates/monitoring/lacework.md)

**Website monitoring:**

* [AlertSite](/ai-sre/ai-sre-for-administrators/set-up-alert-management/webhooks/templates/monitoring/alertsite.md)

**Alert correlation:**

* [BigPanda](/ai-sre/ai-sre-for-administrators/set-up-alert-management/webhooks/templates/monitoring/bigpanda.md)

**SLO monitoring:**

* [Harness SLO](/ai-sre/ai-sre-for-administrators/set-up-alert-management/webhooks/templates/monitoring/harness-slo.md)

Go to [Webhook Templates](/ai-sre/ai-sre-for-administrators/set-up-alert-management/webhooks/templates/overview.md) to browse all available templates.

**Custom monitoring tools**

For custom monitoring solutions, internal applications, or legacy systems:

* **Generic webhook:** Accepts any JSON payload with custom field mapping.
* **Email triggers:** Send alerts via email for systems without webhook support.
* **Custom field mapping:** Use JSONPath, Mustache templates, and CEL expressions.

Go to [Create a Webhook](/ai-sre/ai-sre-for-administrators/set-up-alert-management/webhooks/create-webhook.md) for custom webhook setup.

#### Service paging webhooks <a href="#service-paging-webhooks" id="service-paging-webhooks"></a>

For dedicated on-call paging from external systems:

* **Service-specific webhook URLs:** Each service gets a unique paging webhook.
* **Automatic on-call routing:** Pages the current on-call engineer.
* **Email-based triggering:** Legacy systems can page via email.
* **Bypass alert routing:** Direct service paging without alert rule processing.

Go to [Service Paging Webhook](/ai-sre/ai-sre-for-administrators/set-up-on-call-management/service-paging-webhook.md) for dedicated service paging.

***

### Alert configuration <a href="#alert-configuration" id="alert-configuration"></a>

#### Alert routing configuration <a href="#alert-routing-configuration" id="alert-routing-configuration"></a>

Configure alert routing based on:

* **Service:** Route alerts to specific services based on payload fields.
* **Environment:** Separate production, staging, and development alerts.
* **Team:** Direct alerts to the responsible team or on-call schedule.
* **Severity:** Escalate critical alerts, suppress informational alerts.
* **Custom fields:** Route based on any field in the alert payload.

**Example routing rules:**

* Critical alerts: create an incident and page on-call.
* Warning alerts: create an alert and notify a Slack channel.
* Info alerts: log only, no notifications.

Go to [Route Alerts](/ai-sre/ai-sre-for-administrators/set-up-alert-management/alert-rules/overview.md) for routing configuration.

#### Alert enrichment <a href="#alert-enrichment" id="alert-enrichment"></a>

Enhance alerts with additional context automatically:

* **Service metadata:** Service owner, runbooks, documentation links.
* **Team information:** On-call schedule, escalation policy, Slack channel.
* **Environment details:** Region, cluster, deployment version.
* **Historical data:** Similar past incidents, resolution patterns.
* **Related incidents:** Link to active incidents for the same service.

Enrichment data is added at alert creation time and visible in the alert timeline.

#### Alert actions <a href="#alert-actions" id="alert-actions"></a>

Define automated actions when alerts are received:

**Incident creation:**

* Create incidents automatically for critical alerts.
* Link related alerts to existing incidents.
* Inherit incident metadata from service configuration.

**Runbook execution:**

* Trigger diagnostic runbooks automatically.
* Execute remediation workflows.
* Gather context before human intervention.

**Notifications:**

* Send Slack messages to team channels.
* Page on-call engineers via PagerDuty.
* Post to Microsoft Teams or Google Chat.

**Ticketing:**

* Create Jira issues for alerts requiring follow-up.
* Open ServiceNow incidents for escalation.
* Track alert resolution in external systems.

Go to [Create a Runbook](/ai-sre/ai-sre-for-administrators/set-up-runbook-management/create-runbook.md) to automate alert responses.

***

### Alert lifecycle <a href="#alert-lifecycle" id="alert-lifecycle"></a>

#### Alert states <a href="#alert-states" id="alert-states"></a>

Alerts progress through these states:

1. **New:** Alert received from monitoring system.
2. **Acknowledged:** On-call engineer acknowledges the alert.
3. **Resolved:** Underlying issue resolved (automatic or manual).
4. **Closed:** Alert closed after resolution confirmation.

#### Alert resolution <a href="#alert-resolution" id="alert-resolution"></a>

Alerts can be resolved in multiple ways:

**Automatic resolution:**

* Monitoring system sends a resolution webhook (for example, Datadog recovery).
* Alert rule detects the resolution condition.
* Linked incident is resolved.

**Manual resolution:**

* On-call engineer marks the alert as resolved.
* Runbook completes successfully.
* External system (Jira, ServiceNow) updates status.

#### Alert history <a href="#alert-history" id="alert-history"></a>

All alert events are tracked in the alert timeline:

* Alert creation from monitoring system
* Status changes (acknowledged, resolved)
* Associated incidents and runbooks
* Comments and annotations
* Related alerts and context

***

### Best practices <a href="#best-practices" id="best-practices"></a>

#### Alert design <a href="#alert-design" id="alert-design"></a>

**Use clear, actionable alert names:**

* ❌ `Alert triggered`
* ✅ `High CPU usage on api-service in us-east-1`

**Include relevant context in alert descriptions:**

* Service name and environment
* Current metric value and threshold
* Link to dashboard or logs
* Suggested remediation steps

**Set appropriate thresholds:**

* Avoid alert fatigue from noisy thresholds
* Balance sensitivity with false positive rate
* Use dynamic thresholds for variable workloads

**Configure proper severity levels:**

* **Critical:** Service down, immediate action required.
* **High:** Degraded performance, page on-call.
* **Medium:** Warning condition, notify team channel.
* **Low:** Informational, log only.

#### Alert management <a href="#alert-management" id="alert-management"></a>

**Document alert patterns:**

* Maintain a runbook for each alert type
* Document common causes and resolutions
* Link alerts to service documentation

**Review alert frequency:**

* Monitor alert volume over time
* Identify and tune noisy alerts
* Remove alerts that no longer provide value

**Update alert routing rules regularly:**

* Review routing rules quarterly
* Adjust thresholds based on service behavior
* Update team assignments as ownership changes

**Clean up stale alerts:**

* Archive resolved alerts after retention period
* Remove alerts for decommissioned services
* Consolidate duplicate alert definitions

#### Alert response <a href="#alert-response" id="alert-response"></a>

**Define clear ownership:**

* Assign each alert to a service
* Map services to on-call teams
* Document escalation contacts

**Set up escalation paths:**

* Primary on-call engineer (5 minutes)
* Secondary on-call engineer (15 minutes)
* Team lead or manager (30 minutes)

**Document response procedures:**

* Create runbooks for common alerts
* Link runbooks from alert descriptions
* Include diagnostic steps and remediation commands

**Configure automated responses:**

* Gather diagnostic data automatically
* Execute safe remediation steps
* Notify stakeholders of progress

***

### Related concepts <a href="#related-concepts" id="related-concepts"></a>

#### Alerts versus incidents <a href="#alerts-versus-incidents" id="alerts-versus-incidents"></a>

Alerts and incidents represent different stages of a problem:

* **Alerts:** Notifications from monitoring systems about potential issues.
* **Incidents:** Confirmed service disruptions requiring investigation and resolution.

Alerts can automatically create incidents based on severity and routing rules. Multiple related alerts can be grouped into a single incident.

#### Alerts versus events <a href="#alerts-versus-events" id="alerts-versus-events"></a>

Alerts and events differ in whether they require action:

* **Alerts:** Actionable notifications requiring attention or response.
* **Events:** Informational updates about system state (deploys, config changes).

Events provide context for alerts and incidents but do not require immediate action.

***

### Next steps <a href="#next-steps" id="next-steps"></a>

#### Set up alert ingestion <a href="#set-up-alert-ingestion" id="set-up-alert-ingestion"></a>

* [Webhook Templates](/ai-sre/ai-sre-for-administrators/set-up-alert-management/webhooks/templates/overview.md): Use pre-configured integrations for Datadog, PagerDuty, Prometheus, and other monitoring tools.
* [Create a Webhook](/ai-sre/ai-sre-for-administrators/set-up-alert-management/webhooks/create-webhook.md): Configure custom webhook integrations.
* [External System Setup](/ai-sre/ai-sre-for-administrators/set-up-alert-management/webhooks/integration-guides/overview.md): Follow provider-specific configuration guides.

#### Alert routing setup <a href="#alert-routing-setup" id="alert-routing-setup"></a>

* [Route Alerts](/ai-sre/ai-sre-for-administrators/set-up-alert-management/alert-rules/overview.md): Route and filter alerts.
* [Integrate Service Directory](/ai-sre/ai-sre-for-administrators/set-up-on-call-management/integrate-service-directory.md): Define services and map alerts to services.
* [Set Up On-Call Schedules](/ai-sre/ai-sre-for-administrators/set-up-on-call-management/create-oncall-schedules.md): Route alerts to on-call engineers.

#### Automate alert response <a href="#automate-alert-response" id="automate-alert-response"></a>

* [Create a Runbook](/ai-sre/ai-sre-for-administrators/set-up-runbook-management/create-runbook.md): Automate diagnostic and remediation steps.
* [Integrate External Tools](/ai-sre/ai-sre-for-administrators/set-up-runbook-management/integrations.md): Connect Slack, Jira, ServiceNow, and other tools.
* [AI SRE Best Practices](/ai-sre/troubleshooting-and-resources/ai-sre-best-practices.md): Optimize alert management.

{% @harness-feedback/feedback %}