Use RCA Change Agent
Learn about the RCA Change Agent in Harness AI SRE that analyzes incident timelines and telemetry to identify likely root cause candidates with confidence scores.
The RCA Change Agent operates continuously throughout an incident's lifecycle, automatically re-analyzing when new key events are added by the AI Scribe Agent.
As new information becomes available, it updates theory confidence scores and can add new theories or rule out unlikely candidates, providing engineers with an evolving understanding of potential root causes.
How it works
The RCA Change Agent runs automatically as a background job whenever key events are added to an active incident. This means:
Automatic triggering: No manual configuration required, the agent activates when the AI Scribe Agent captures key events
Continuous updates: Each time new key events are added, the agent re-evaluates its theories
Progressive refinement: Confidence scores adjust as more data becomes available
Theory evolution: New theories can be added, and unlikely theories can be ruled out as the incident progresses
ServiceNow change integration
When a Harness ServiceNow connector exists in your organization, the RCA Change Agent automatically ingests ServiceNow change records and correlates them with active incidents. This zero-configuration integration requires no separate setup beyond your existing ServiceNow connector.
Prerequisites
ServiceNow connector: An existing Harness ServiceNow connector configured in your organization (typically set up for pipeline approvals or other ServiceNow workflows)
Read access: The connector must have read access to the
change_requesttable in ServiceNow
How ServiceNow changes appear
ServiceNow change records appear in the RCA Change Agent theories panel alongside deployments and pull requests as candidate root causes. The system automatically:
Polls every 5 minutes: The RCA Change Agent queries the ServiceNow Table API for change records every 5 minutes
Historical lookback: On first activation, the system retrieves change records from the past 90 days
Correlates to incidents: Change records are matched to incident timelines based on timing and affected services
Generates theories: ServiceNow changes appear as theories with confidence scores, just like deployment and code changes
Zero-configuration activation
If your organization already uses a Harness ServiceNow connector (common for pipeline approval gates or ITSM workflows), ServiceNow change data automatically flows into RCA. No additional configuration, webhook setup, or ServiceNow admin involvement is required.
When you save or update a ServiceNow connector in Harness, the system automatically creates an ingest job that polls the change_request table and makes those changes available to the RCA Change Agent.
Investigation tools
The RCA Change Agent uses specialized tools to investigate incidents:
Deployment analysis
Search deployments: Finds deployment information for specific services, environments, and time ranges
Timeline correlation: Matches deployments to incident timeline events to identify suspect changes
Code change analysis
Find related PRs: Identifies pull requests between stable and broken deployments
Code correlation: Links code changes to symptoms observed during the incident
Change event analysis
Search change events: Finds feature flags, infrastructure changes, and configuration updates
Change timeline: Correlates change events with incident start and symptom progression
Code investigation (optional)
When configured, the agent can access investigator runbooks that execute code-level analysis to examine specific services or components identified as potential causes.
Root cause theories
The RCA Change Agent generates theories about potential root causes. Each theory includes:
Theory components
Theory message
Clear description of the potential root cause
Confidence score
Numerical value from 0 to 100 indicating likelihood
Status
Current investigation status (see below)
Related activities
Links to specific deployments, PRs, alerts, or change events that support the theory
Evidence
Specific timeline events or telemetry signals that led to this theory
Theory status
Theories progress through these statuses as investigation continues:
INVESTIGATING (default): Theory is being evaluated; more data is needed
CONFIRMED: High confidence this is the root cause based on strong evidence
RULED_OUT: Evidence indicates this is not the root cause
Integration with AI Scribe Agent
The RCA Change Agent depends on the AI Scribe Agent for structured incident data:
AI Scribe Agent captures communications and creates key events in the incident timeline
RCA Change Agent analyzes the timeline and runs investigation tools when key events are added
Confidence scores update as new key events provide additional context
Post-incident review action (when configured in a runbook) generates human-readable reports using the RCA theories
The quality of root cause theories depends directly on the completeness of the AI Scribe Agent's timeline. Teams that follow communication best practices, particularly explicitly stating observed symptoms and root cause hypotheses, will see more accurate analysis.
View RCA analysis
Root cause theories appear in the incident view as they are generated:
Theory list: View all active theories with their confidence scores and status
Related items: Click through to deployments, PRs, or change events linked to each theory
Timeline correlation: See which timeline events contributed to each theory
Confidence progression: Watch how confidence scores change as new data arrives
Example analysis
The following example shows how the RCA Change Agent might analyze an incident:
Initial analysis (after first key events):
After additional key events:
Configuration
Enable RCA analysis
The RCA Change Agent requires:
AI Scribe Agent active on the incident to generate key events
Valid services and environments configured in your incident type
Alert integrations connected to provide telemetry data
No additional runbook configuration is required, the agent runs automatically when these prerequisites are met.
Optional: change event analysis
When the IR_RCA_QUERY_CHANGES feature flag is enabled for your organization, the RCA Change Agent also searches change events (feature flags, infrastructure changes, configuration updates) as part of its analysis.
Optional: code investigation
Investigator runbooks can be configured to enable deeper code-level analysis. When configured:
The RCA Change Agent can invoke runbooks as tools during investigation
Runbooks can examine specific services, query logs, or analyze metrics
Results from runbook executions are incorporated into theory generation
Contact your Harness representative for information on configuring investigator runbooks.
Data sources
The RCA Change Agent automatically analyzes:
Incident timeline
Key events and decisions from AI Scribe Agent
Deployment history
Recent deployments to affected services
Pull request data
Code changes between stable and broken deployments
ServiceNow change records
Change requests from ServiceNow when connector exists
Change events
Feature flags, infrastructure changes, configuration updates
Alert data
Signals from connected observability platforms
System telemetry
Metrics and traces from affected services
Best practices
For accurate root cause analysis
Follow these practices to improve theory accuracy:
Enable AI Scribe Agent early: Add it at incident creation to capture the complete timeline
Communicate clearly: State symptoms and hypotheses explicitly in incident channels
Use consistent naming: Reference services and environments by their configured names
Tag relevant changes: Mark deployments, PRs, and change events with service names
Review theories regularly: Engineers should validate theories and provide feedback
For investigation efficiency
Follow these practices to investigate faster:
Check theories first: Before deep investigation, review RCA theories to focus efforts
Follow evidence links: Click through to related deployments and PRs for context
Update based on findings: As you confirm or rule out causes, theories will adjust
Document decisions: Add key events when you confirm or rule out a root cause
Get started
Quick setup
AI Scribe Agent: Capture the incident timeline that the RCA Change Agent analyzes.
Alert integration: Connect observability signals to provide telemetry data.
Runbook automation: Automate response and investigation actions.
Related resources
Incident management overview: Understand how incidents are structured in AI SRE.
Post-incident review: Generate retrospective reports from RCA theories.
Runbook automation: Automate response and investigation actions.
Next steps
Go to the AI Scribe Agent to capture the incident timeline that the RCA Change Agent analyzes.
Go to Investigator Agent Pipelines to extend investigations with custom data sources.
Go to runbook automation to automate response and investigation actions.
Summary
The RCA Change Agent enhances incident response by automatically investigating potential root causes whenever the AI Scribe Agent adds key events to an active incident.
By analyzing deployments, code changes, and telemetry data, it generates theories with confidence scores that help engineers focus their investigation efforts on the most likely causes, accelerating time to resolution.
Last updated
Was this helpful?