> For the complete documentation index, see [llms.txt](https://developer.harness.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://developer.harness.io/ai-sre/ai-sre-for-administrators/set-up-runbook-management/runbooks.md).

# Runbook Management

Harness AI SRE allows teams to **automate incident resolution** by leveraging **Runbook automation**. Runbooks provide predefined workflows that execute **automated actions** based on specific triggers.

### Key features <a href="#key-features" id="key-features"></a>

#### Automated response actions <a href="#automated-response-actions" id="automated-response-actions"></a>

Runbooks automate the following response actions:

* Instant incident communication
* Automated remediation steps
* Multi-channel notifications
* Integrated team collaboration

#### Integration ecosystem <a href="#integration-ecosystem" id="integration-ecosystem"></a>

Runbooks connect to a broad set of communication and ticketing tools:

* **Communication tools**
  * [Slack](/ai-sre/ai-sre-for-administrators/set-up-runbook-management/integrations/collaboration/slack.md): Channel management and notifications
  * [Microsoft Teams](/ai-sre/ai-sre-for-administrators/set-up-runbook-management/integrations/collaboration/teams.md): Team collaboration
  * [Zoom](/ai-sre/ai-sre-for-administrators/set-up-runbook-management/integrations/collaboration/zoom.md): Incident bridges
* **Ticketing systems**
  * [Jira](/ai-sre/ai-sre-for-administrators/set-up-runbook-management/integrations/ticketing/jira.md): Issue tracking and updates
  * [ServiceNow](/ai-sre/ai-sre-for-administrators/set-up-runbook-management/integrations/ticketing/servicenow.md): Incident management

***

### Getting started <a href="#getting-started" id="getting-started"></a>

Follow this sequence to build and automate your first runbook:

1. [Create your first runbook](/ai-sre/ai-sre-for-administrators/set-up-runbook-management/create-runbook.md)
   * Design workflows
   * Configure triggers
   * Test and deploy
2. [Configure authentication](/ai-sre/ai-sre-for-administrators/set-up-runbook-management/integrations/overview.md)
   * Set up integration access
   * Manage permissions
   * Secure your runbooks
3. [Configure incident fields](/ai-sre/ai-sre-for-administrators/set-up-runbook-management/workflows/overview.md)
   * Define custom fields
   * Set up field mapping
   * Configure templates

***

### Example runbook templates <a href="#example-runbook-templates" id="example-runbook-templates"></a>

#### 1. High CPU usage response <a href="#id-1-high-cpu-usage-response" id="id-1-high-cpu-usage-response"></a>

**Purpose:** Automated response to CPU spikes

**Trigger configuration:**

* Alert Type: Datadog
* Metric: CPU Usage
* Threshold: > 90%
* Duration: 5 minutes

**Action steps:**

1. **Initial Alert**
   * Action Type: Slack
   * Channel: #sre-alerts
   * Message: "🚨 High CPU Alert: \[service] CPU usage > 90% for 5 minutes"
2. **Create Incident Bridge**
   * Action Type: Zoom
   * Operation: Create Meeting
   * Name: "CPU Incident - \[service]"
   * Participants: SRE Team
3. **Automated Remediation**
   * Action Type: Harness Pipeline
   * Pipeline: Scale Service Pipeline
   * Input Variables:
     * Service: \[service]
     * Replicas: +2
4. **On-Call Notification**
   * Action Type: PagerDuty
   * Priority: High
   * Assignee: SRE On-Call
   * Details: "High CPU incident - Scaling pipeline initiated"

#### 2. Database connection alert <a href="#id-2-database-connection-alert" id="id-2-database-connection-alert"></a>

**Purpose:** Multi-channel incident response coordination

**Trigger configuration:**

* Alert Type: Grafana Incident
* Service: Database
* Condition: Connection Timeout
* Priority: High

**Action steps:**

1. **Create Teams Channel**
   * Action Type: Microsoft Teams
   * Operation: Create Channel
   * Name: "db-incident-\[timestamp]"
   * Add Team: Database Support
2. **Execute Recovery**
   * Action Type: Jenkins
   * Job: DB-Recovery-Job
   * Parameters:
     * Service: \[database\_service]
     * Action: restart\_connections
3. **Status Update**
   * Action Type: OpsGenie
   * Operation: Create Alert
   * Team: Database
   * Message: "🔴 DB Connection Issues - Recovery Job Status: \[jenkins.status]"
4. **Incident Management**
   * Action Type: Grafana Incident
   * Operation: Update
   * Status: Investigating
   * Note: "Recovery procedures initiated via Jenkins"

#### 3. API error rate response <a href="#id-3-api-error-rate-response" id="id-3-api-error-rate-response"></a>

**Purpose:** Feature management and incident coordination

**Trigger configuration:**

* Alert Type: Datadog
* Metric: Error Rate
* Threshold: > 5%
* Time Window: 5 minutes

**Action steps:**

1. **Feature Control**
   * Action Type: Split
   * Operation: Disable Feature
   * Feature Name: new\_api\_version
   * Environment: production
2. **Pipeline Execution**
   * Action Type: GitHub Actions
   * Workflow: api-recovery
   * Inputs:
     * service: \[service]
     * action: rollback
3. **Team Communication**
   * Action Type: Slack
   * Channel: #api-incidents
   * Message: "⚠️ API Error Rate Incident\n- Feature flag disabled\n- Recovery workflow status: \[github.status]"
4. **Stakeholder Bridge**
   * Action Type: Zoom
   * Operation: Create Meeting
   * Name: "API Incident Bridge"
   * Participants: \["@api-team", "@product"]

***

### Next steps <a href="#next-steps" id="next-steps"></a>

#### Documentation <a href="#documentation" id="documentation"></a>

* [Create a runbook](/ai-sre/ai-sre-for-administrators/set-up-runbook-management/create-runbook.md)
* [Configure authentication](/ai-sre/ai-sre-for-administrators/set-up-runbook-management/integrations/overview.md)
* [Configure incident fields](/ai-sre/ai-sre-for-administrators/set-up-runbook-management/workflows/overview.md)

#### Integration guides <a href="#integration-guides" id="integration-guides"></a>

* Communication tools
  * [Slack integration](/ai-sre/ai-sre-for-administrators/set-up-runbook-management/integrations/collaboration/slack.md)
  * [Microsoft Teams integration](/ai-sre/ai-sre-for-administrators/set-up-runbook-management/integrations/collaboration/teams.md)
  * [Zoom integration](/ai-sre/ai-sre-for-administrators/set-up-runbook-management/integrations/collaboration/zoom.md)
* Ticketing systems
  * [Jira integration](/ai-sre/ai-sre-for-administrators/set-up-runbook-management/integrations/ticketing/jira.md)
  * [ServiceNow integration](/ai-sre/ai-sre-for-administrators/set-up-runbook-management/integrations/ticketing/servicenow.md)