> For the complete documentation index, see [llms.txt](https://developer.harness.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://developer.harness.io/ai-sre/3.0/ai-sre-for-administrators/set-up-incident-management/incident-workflows.md).
# Automate Incident Response with Runbooks
Learn how to automate incident response workflows in Harness AI SRE using runbooks, triggers, and integrations.
### Overview
AI SRE automates incident workflows through:
* **Runbooks** - Sequences of automated actions (notifications, API calls, scripts)
* **Triggers** - Conditions that automatically execute runbooks (route alerts, incident events, status changes)
* **Route Alerts** - Route alerts to on-call teams and auto-create incidents
* **Integrations** - Connect to Slack, Jira, ServiceNow, Zoom, PagerDuty, and more
Workflow automation in AI SRE uses **form-based UI configuration with Mustache templates**, not YAML files. Actions are configured through forms where you can:
* Select integration type (Slack, Jira, HTTP, etc.)
* Fill in action parameters using form fields
* Reference incident data using Mustache syntax like `{{incident.title}}` or `{{alert.severity}}`
* Test actions before saving
***
### Automation patterns
#### Pattern 1: Alert detection to incident creation
**Use case:** Automatically create incidents from high-severity alerts
**How to configure:**
1. Navigate to **Alerts** → **Route Alerts**
2. Click **Create Alert Rule**
3. Configure the rule:
* **Name**: "P1/P2 Alerts Auto-Create Incidents"
* **Conditions**: `alert.priority` in `[p1_critical, p2_error]`
* **Action**: Create Incident
* **Incident Type**: Select "Service Incident"
* **Map Fields**:
* Title: `{{alert.title}}`
* Service: `{{alert.service}}`
* Severity: `{{alert.priority}}`
4. Click **Save**
**Result:** When P1 or P2 alerts are received, incidents are automatically created with pre-populated fields.
#### Pattern 2: Incident creation to automated response
**Use case:** When a P1 incident is created, automatically notify on-call, create Zoom bridge, and trigger diagnostic runbook
**How to configure:**
1. Create a runbook for P1 response:
* Navigate to **Runbooks** → **Create Runbook**
* **Name**: "P1 Incident Response"
* Add actions:
1. **Zoom: Create Meeting**
* Name: `{{incident.title}} - Incident Bridge`
* Participants: `{{incident.responders}}`
2. **Slack: Post Message**
* Channel: `#incidents`
* Message:
```
🔴 **P1 Incident Created**
**Title**: {{incident.title}}
**Service**: {{incident.service}}
**Zoom Bridge**: {{runbook.outputs.zoom_create_meeting.join_url}}
**Incident Link**: {{incident.url}}
```
3. **On-Call: Page Service**
* Service: `{{incident.service}}`
* Message: `P1 incident - join bridge at {{runbook.outputs.zoom_create_meeting.join_url}}`
2. Configure automatic trigger:
* In the runbook editor, go to **Triggers** tab
* Click **Add Trigger**
* **Trigger Type**: Incident Created
* **Conditions**:
* `incident.severity` equals `SEV0` OR `SEV1`
* Click **Save**
**Result:** P1 incidents automatically trigger the runbook, creating a Zoom bridge and notifying responders.
#### Pattern 3: Status change to stakeholder notification
**Use case:** When incident status changes to "Resolved", notify stakeholders and create post-incident review task
**How to configure:**
1. Create a runbook:
* **Name**: "Incident Resolution Workflow"
* Add actions:
1. **Slack: Post Message**
* Channel: `#incidents`
* Message:
```
✅ **Incident Resolved**
**Title**: {{incident.title}}
**Duration**: {{incident.duration}}
**Resolved By**: {{incident.resolver}}
Post-incident review scheduled.
```
2. **Jira: Create Issue**
* Project: `SRE`
* Issue Type: Task
* Summary: `Post-Incident Review: {{incident.title}}`
* Description:
```
Incident: {{incident.url}}
Duration: {{incident.duration}}
Severity: {{incident.severity}}
Complete post-incident review within 3 business days.
```
* Assignee: `{{incident.owner}}`
2. Configure trigger:
* **Trigger Type**: Incident Field Updated
* **Field**: Status
* **New Value**: Resolved
**Result:** When incidents are marked resolved, stakeholders are notified and a review task is created in Jira.
#### Pattern 4: Time-based escalation
**Use case:** If a P1 incident is not acknowledged within 5 minutes, escalate to VP Engineering
**How to configure:**
1. Create escalation runbook:
* **Name**: "P1 Escalation"
* Add actions:
1. **Slack: Post Message**
* Channel: `#exec-alerts`
* Message:
```
🚨 **ESCALATION: P1 Incident Not Acknowledged**
**Title**: {{incident.title}}
**Created**: {{incident.created_at}}
**Service**: {{incident.service}}
**Link**: {{incident.url}}
@vp-engineering - immediate attention required
```
2. **PagerDuty: Trigger Escalation**
* Escalation Policy: "Executive Escalation"
2. Configure trigger:
* **Trigger Type**: Incident Created
* **Conditions**:
* `incident.severity` equals `SEV0` OR `SEV1`
* **Delay**: 5 minutes
* **Only run if**: `incident.status` not equals `Acknowledged`
**Result:** If P1 incidents remain unacknowledged for 5 minutes, executives are paged.
#### Pattern 5: Deployment change to proactive investigation
**Use case:** When a deployment occurs, automatically check for related alerts and create incident if errors spike
**How to configure:**
1. Send deployment webhooks to AI SRE:
* Configure your CI/CD pipeline to POST deployment events to:
```
POST https://app.harness.io/gateway/ai-sre/api/v1/orgs/{org}/projects/{project}/webhooks/deploy
```
* Webhook payload:
```json
{
"service": "payment-service",
"version": "v2.3.1",
"environment": "production",
"deployed_by": "jane@company.com",
"git_commit": "abc123"
}
```
2. AI SRE automatically:
* Correlates the deployment with any alerts occurring within 30 minutes
* Surfaces the deployment as a root cause theory in investigations
* Links related pull requests for code review
**No additional configuration needed** - deployment correlation is automatic when webhooks are sent.
***
### Runbook components
#### Actions
Runbooks execute sequences of actions. Available action types:
**Communication actions**
* **Slack: Post Message** - Send formatted messages to Slack channels
* **Slack: Update Message** - Update existing Slack messages
* **Microsoft Teams: Post Message** - Send messages to Teams channels
* **Google Chat: Post Message** - Send messages to Google Chat spaces
* **Zoom: Create Meeting** - Generate instant Zoom bridges
**Ticketing actions**
* **Jira: Create Issue** - Create Jira tickets with incident context
* **Jira: Update Issue** - Update existing Jira issues
* **ServiceNow: Create Incident** - Create ServiceNow incidents
* **ServiceNow: Update Incident** - Update ServiceNow incidents
**Deployment actions**
* **Harness: Run Pipeline** - Trigger Harness CD pipelines (rollback, scale, deploy)
**On-call actions**
* **Page Service** - Page the on-call responders for a service
* **Page User** - Directly page a specific user
* **Page Team** - Page all members of a team
**Custom actions**
* **HTTP Request** - Call any REST API
* **Script** - Run custom JavaScript/Python logic
#### Action inputs
Actions use form-based configuration with Mustache template support:
**Text fields** accept Mustache variables:
```mustache
Title: {{incident.title}}
Service: {{incident.service}}
Owner: {{incident.owner}}
```
**Available variables:**
* `{{incident.*}}` - Current incident fields (title, severity, service, owner, status, etc.)
* `{{alert.*}}` - Alert fields if runbook triggered by alert rule
* `{{runbook.outputs.*}}` - Outputs from previous actions in the runbook
* `{{user.*}}` - User who triggered the runbook (name, email)
**Example: reference previous action output**
```
Action 1: Zoom: Create Meeting
→ Outputs: join_url, meeting_id
Action 2: Slack: Post Message
Message: Join the incident bridge at {{runbook.outputs.zoom_create_meeting.join_url}}
```
#### Triggers
Runbooks can be triggered:
**Manual execution**
* Run from incident timeline
* Run from pinned runbooks list
* Run via `/harness run ` Slack command
**Automatic triggers**
**Incident Created**
* Condition: `incident.severity in [SEV0, SEV1]`
* Runs when new incidents match conditions
**Incident Field Updated**
* Condition: `incident.status changed_to Resolved`
* Runs when specific fields change
**Alert Rule Match**
* Configured in route alerts
* Runs when alerts meet routing criteria
**Scheduled**
* Cron syntax: `0 9 * * 1` (every Monday at 9am)
* Use for: daily health checks, weekly reports
#### Conditions
Trigger conditions use field comparisons:
**Operators:**
* `equals`, `not_equals`
* `in`, `not_in` (for arrays)
* `changed_to`, `changed_from` (for field updates)
* `contains`, `not_contains` (for strings)
* `greater_than`, `less_than` (for numbers)
**Examples:**
```
incident.severity in [SEV0, SEV1]
incident.service equals payment-service
incident.status changed_to Resolved
alert.priority equals p1_critical
```
***
### Integration examples
#### Slack incident channel creation
**Goal:** Create a dedicated Slack channel for each P1 incident
1. Enable Slack integration:
* Navigate to **Project Settings** → **Integrations**
* Connect Slack workspace
2. Create runbook:
* Action: **Slack: Create Channel**
* Channel Name: `inc-{{incident.short_id}}-{{incident.service}}`
* Topic: `{{incident.title}} - {{incident.severity}}`
* Invite Users: `{{incident.responders}}`
* Action: **Slack: Post Message**
* Channel: `{{runbook.outputs.slack_create_channel.channel_id}}`
* Message:
```
🚨 **Incident Summary**
**Service**: {{incident.service}}
**Severity**: {{incident.severity}}
**Owner**: {{incident.owner}}
View details: {{incident.url}}
```
3. Configure trigger:
* **Trigger Type**: Incident Created
* **Conditions**: `incident.severity in [SEV0, SEV1]`
#### Jira bidirectional sync
**Goal:** Create Jira tickets for incidents and sync status updates
**Outbound (AI SRE to Jira):**
1. Create runbook:
* Action: **Jira: Create Issue**
* Project: `INCIDENT`
* Summary: `{{incident.title}}`
* Description:
```
Incident: {{incident.url}}
Service: {{incident.service}}
Severity: {{incident.severity}}
```
* Action: **AI SRE: Add Timeline Event**
* Description: `Jira ticket created: {{runbook.outputs.jira_create_issue.issue_key}}`
2. Trigger: Incident Created with severity P1/P2
**Inbound (Jira to AI SRE):**
Requires custom configuration using Jira Automation Rules:
1. In Jira, create automation rule:
* **Trigger**: Issue Updated
* **Condition**: Status changed
* **Action**: Send web request
* URL: `https://app.harness.io/gateway/ai-sre/api/v1/incidents/{{issue.customfield_incident_id}}/status`
* Method: PUT
* Body: `{"status": "{{issue.status}}"}`
Go to the [Jira integration guide](/ai-sre/3.0/ai-sre-for-administrators/set-up-runbook-management/integrations/ticketing/jira.md) to complete setup.
#### ServiceNow change correlation
**Goal:** Automatically surface ServiceNow change requests in incident investigations
1. Configure ServiceNow connector:
* Navigate to **Project Settings** → **Connectors**
* Add ServiceNow connector with credentials
2. Enable RCA Change Agent:
* The agent automatically polls ServiceNow every 5 minutes
* Change requests are correlated with incidents by:
* Time window (changes within 1 hour of incident)
* Service/CI matching
* Configuration item relationships
3. View correlated changes:
* Open an incident
* Go to **Investigation** tab
* ServiceNow changes appear as root cause theories
Go to the [RCA Change Agent](/ai-sre/3.0/ai-sre-for-incident-responders/use-ai-agents/rca-change-agent.md) to review more details.
***
### Best practices
#### Start simple
* Begin with 1-2 critical workflows (e.g., P1 notifications)
* Add automation incrementally as you validate effectiveness
* Avoid over-automating before processes are stable
#### Test thoroughly
* Use runbook test mode to validate actions without creating real incidents
* Test with non-production services first
* Verify Mustache variables render correctly with sample data
#### Handle failures gracefully
* Add error notifications if critical actions fail
* Use HTTP action retries for flaky APIs
* Monitor runbook execution logs for patterns
#### Monitor effectiveness
* Track runbook execution success rates
* Measure time-to-response improvements
* Gather feedback from incident responders
* Iterate based on what works
#### Document runbook purpose
* Add clear descriptions to each runbook
* Document what triggers it and what it does
* Note any prerequisites (credentials, permissions)
* Keep runbooks focused on single workflows
***
### Next steps
* [Create a runbook](/ai-sre/3.0/ai-sre-for-administrators/set-up-runbook-management/create-runbook.md): Detailed runbook creation guide.
* [Runbook triggers](/ai-sre/3.0/ai-sre-for-administrators/set-up-runbook-management/triggers/create-trigger.md): Configure automatic execution.
* [Route alerts](/ai-sre/3.0/ai-sre-for-administrators/set-up-alert-management/alert-rules/overview.md): Route alerts and auto-create incidents.
* [Slack integration](/ai-sre/3.0/ai-sre-for-administrators/set-up-runbook-management/integrations/collaboration/slack.md): Slack action reference.
* [Jira integration](/ai-sre/3.0/ai-sre-for-administrators/set-up-runbook-management/integrations/ticketing/jira.md): Jira action reference.