Key Concepts
Essential Harness Chaos Engineering concepts and terminology
This guide covers the essential terminology and concepts for Harness Resilience Testing.
Chaos Engineering
Chaos Engineering is the discipline of experimenting on a system to build confidence in its capability to withstand turbulent and unexpected conditions in production.
Core principles
Steady state hypothesis
The steady state represents your system's normal operating condition. Before running chaos experiments, you define:
Measurable system outputs that indicate normal behavior
Baseline metrics using Service Level Objectives (SLOs)
Acceptable thresholds for system performance
Example: "Our API should maintain 99.9% availability with response times under 200ms during normal operations."
Blast radius
The scope of impact a chaos experiment can have on your system. Best practices:
Start small with non-critical systems or components
Gradually increase experiment scope as confidence grows
Use infrastructure controls to limit impact
Implement automatic rollback mechanisms
Hypothesis-driven testing
Each chaos experiment follows a scientific approach:
Identify the steady state and specify SLOs
Hypothesize what will happen when a fault is injected
Inject the failure in a controlled manner with minimal blast radius
Validate whether the system maintains steady state and meets SLOs
Key components
Chaos experiments
A chaos experiment is a set of operations coupled together to inject faults into a target resource and validate the system's resilience. Each experiment:
Targets specific infrastructure or application components
Injects one or more chaos faults in a defined sequence
Uses probes to validate system behavior
Can include custom actions for notifications or integrations
Generates a resilience score based on results
Chaos faults
Chaos faults are pre-built failure scenarios that simulate real-world issues:
Fault categories
Kubernetes Faults: Pod deletions, container kills, resource stress
Cloud Platform Faults: AWS, GCP, Azure service disruptions
Infrastructure Faults: CPU stress, memory exhaustion, network latency, disk pressure
Application Faults: Service failures, error injection, timeout simulation
Harness provides 200+ ready-to-use chaos faults in the Enterprise ChaosHub.
Resilience probes
Resilience probes are health validation mechanisms that run during chaos experiments to verify your system maintains its steady state:
Probe modes
Continuous Mode: Monitor throughout experiment duration
Edge Mode: Check at specific experiment phases (before, during, after)
OnChaos Mode: Validate only during fault injection
Probe types
HTTP Probe: Validate API endpoints and services
Command Probe: Execute custom commands and validate output
Prometheus Probe: Query Prometheus metrics
Datadog Probe: Query Datadog metrics
Dynatrace Probe: Query Dynatrace metrics
Actions
Actions are custom tasks that execute within experiments:
Send notifications to Slack, PagerDuty, or email
Trigger webhooks for external integrations
Execute custom scripts or commands
Add delays between experiment steps
Integrate with monitoring and observability tools
ChaosHub
GIT-BASED CHAOSHUBS DEPRECATED
Git-based ChaosHubs have been removed. Use Templates and Resilience Probes to manage reusable chaos artifacts instead.
A ChaosHub was a centralized repository for reusable chaos engineering resources. Harness provides a default Enterprise ChaosHub with 200+ chaos faults that remains available. Custom Git-based ChaosHubs are no longer supported.
Chaos infrastructure
Chaos infrastructure represents the target environment where chaos experiments execute:
Deployment models
Agent-Based: Deploy chaos agents on Linux or Windows hosts
Agentless: Use Harness Delegate for Kubernetes and cloud resources
Supported targets
Kubernetes clusters (EKS, GKE, AKS, OpenShift)
Linux and Windows hosts
AWS, GCP, Azure cloud resources
VMware infrastructure
Load Testing
Load Testing validates that your system can handle expected and peak traffic while maintaining performance and reliability.
Virtual users
A virtual user (VU) simulates a real user executing your defined scenario: sending HTTP requests, waiting for responses, and looping continuously for the duration of the test. The Number of Users setting controls peak concurrency.
Load profile
The load profile defines how virtual users are introduced over time:
Ramp-Up Phase: Users increase linearly from 0 to the target count over the configured Ramp-Up Duration. This models realistic traffic growth and avoids an artificial cold-start spike.
Steady-State Phase: After ramp-up, the full user count runs for the remaining test duration (
Test Duration - Ramp-Up Duration).
Scenario
A scenario is the sequence of HTTP requests each virtual user executes. Depending on the Load Test Engine, you define it as a Python (Locust) script, a JavaScript (k6) script, or a Java (JMeter) plan.
Assertions
Assertions define per-request success criteria. Two types are supported:
Text: Validates that the response body contains a specific string
Response Time: Validates that the response arrives within a specified latency threshold
Requests that fail assertions are counted as errors in test results.
Load test infrastructure
The infrastructure where load tests execute. Two target types are supported:
Linux VM: A Linux host with the Harness chaos agent and load testing enabled. The agent runs the Python (Locust) process locally and streams metrics back to Harness.
Kubernetes: A Kubernetes cluster with the Harness chaos agent (v1.85.3 or later). Load testing is enabled by default. The agent orchestrates a master pod and optional worker pods for scalable, distributed load generation.
Go to Infrastructure to configure load test infrastructure.
Disaster Recovery Testing
Disaster Recovery Testing validates that backup systems, failover mechanisms, and recovery procedures work during catastrophic scenarios. Each DR test is a Harness pipeline stage, giving you full orchestration of failover, validation, and notification steps.
RTO and RPO
Recovery Time Objective (RTO): The maximum acceptable time for a system to be restored to full operation after a failure. DR tests validate that your recovery procedures complete within this window.
Recovery Point Objective (RPO): The maximum acceptable amount of data loss measured in time. DR tests validate backup recency and data consistency after a simulated recovery.
Pipeline-based DR tests
DR tests are built using Harness Pipeline Studio. Each DR test is a pipeline with a Disaster Recovery stage (DRTest) that has four tabs:
Overview: Stage name, objective, timeout, and stage variables
Environment: Target Harness environment and stage-level failure strategy
Execution: Step canvas for the forward workflow, plus a Rollback path for compensating steps
Advanced: Delegate selector, conditional execution, looping strategy, and additional failure strategy actions
Chaos Fault, Chaos Probe, and Chaos Action steps each select their own chaos infrastructure. The Environment tab does not set infrastructure for the whole stage.
Failure strategy
Defines what happens when a step or stage encounters an error. You can handle specific failure types (Authentication Errors, Connectivity Errors, Timeout Errors, etc.) with actions like Rollback Pipeline, Retry Step, Abort, Manual Intervention, or Mark As Failure. Pair Rollback Stage with the Rollback path on the Execution tab when you want compensating steps to run.
Conditional execution
Controls whether a stage runs based on pipeline state: on success (default), on failure, always, or via a custom JEXL expression. Useful for running rollback stages only when a failover stage fails.
Go to DR Testing Concepts for a full breakdown of all concepts.
Harness Resilience Testing concepts
These concepts apply across all resilience testing activities in the Harness platform.
Services
A service is an onboarded target, and it is the unit Harness Resilience Testing tests, scores, and reports on. Continuous discovery invents workloads in a Kubernetes cluster. Resilience Testing onboarding turns selected workloads into services, runs a risk scan as part of that flow, and (for bulk onboarding) attaches default health probes. Use the Custom Service Agent for Linux VMs, Windows VMs, AWS resources, and other targets you define by hand. Those custom services require an explicit infrastructure assignment.
The service is what ties the module together, because every other concept attaches to it:
Chaos experiments, load tests, and DR tests all target a service, and its details page reports all three side by side
Probes attach to a service and take their inputs from the target behind it
Risks are detected against a service, and risk scores are calculated per service
Go to Services to understand what onboarding creates. Go to Automated service onboarding for the three-stage wizard.
Environments
Logical groupings of your infrastructure where tests are executed:
Organize resources by purpose (dev, staging, production)
Control access and permissions per environment
Isolate test execution to specific infrastructure scopes
Governance
Controls and policies to ensure safe, controlled testing:
RBAC (Role-Based Access Control)
Fine-grained permissions for who can:
Create and modify tests
Execute tests on specific infrastructure
View results and analytics
Manage governance policies
ChaosGuard
ChaosGuard provides advanced governance specifically for chaos experiments:
Define when experiments can run (time windows)
Specify where experiments can execute (infrastructure scope)
Control what faults can be injected (fault restrictions)
Set approval requirements for high-risk tests
Risks
Automated identification and tracking of system weaknesses. A scan reads your application manifests, matches them against the Harness resilience rules, and records a risk for every condition that is likely to fail under stress. Each risk falls into one of four categories:
Availability: Redundancy and failover gaps, such as a single-replica workload
Performance: Capacity and resource issues, such as unset CPU or memory limits
Resilience: Recovery after a failure has already occurred, such as missing liveness or readiness probes
Config: Declared settings that weaken the workload, such as a container running as root
Risks are detected automatically and start out as passive, which means they are unproven. Associate a risk rule with a probe and run an experiment to confirm it. Go to Risks to understand risk scoring and the passive to confirmed lifecycle.
Next steps
Now that you understand the core concepts:
Architecture: Learn about the control plane and execution plane architecture
Get Started with Chaos Testing: Run your first chaos experiment
Explore Chaos Faults: Browse 200+ ready-to-use fault scenarios
Set Up Governance: Configure RBAC and ChaosGuard for safe testing
Last updated
Was this helpful?