Verification Overview
Get an overview of Harness Continuous Verification with AI Verify.
Continuous Verification is a critical step in the deployment pipeline that validates deployments. It integrates with APMs and logging tools to verify that the deployment is running safely and efficiently. Harness applies machine learning algorithms to every deployment for identifying normal behavior, which allows it to identify and flag anomalies in future deployments. During the AI Verify (v1) step, Continuous Verification automatically triggers a rollback if anomalies are found.
This topic provides an overview of the prerequisites and steps involved in setting up Continuous Verification with AI Verify (v1).
Before you begin
Deployment strategies
You can set up Continuous Verification by adding an AI Verify (v1) step to a pipeline. The following deployment strategies are available when configuring verification.
Continuous verification type
Auto: Harness automatically selects the best continuous verification type based on the deployment strategy.
Rolling Update: Rolling deployment is a deployment technique that gradually replaces old versions of a service with a new version by replacing the infrastructure on which the service runs. Rolling updates are useful in situations where a sudden changeover might cause downtime or errors.
Canary: Canary deployment involves a two-phased deployment. In phase one, new pods and instances with the new service version are added to a single environment. In phase two, a rolling update is performed in the same environment. Canary deployment helps to detect issues with the new deployment before fully deploying it.
Blue Green: Blue-green deployment is a technique used to deploy services to a production environment by gradually shifting user traffic from an old version to a new one. The previous version is referred to as the blue environment, while the new version is known as the green environment. Upon completion of the transfer, the blue environment remains on standby in case of a need for rollback, or can be removed from production and updated to serve as the template for future updates.
Load Test: Load testing is a strategy used in lower-level environments, such as quality assurance, where a consistent load is absent, and deployment validation is typically accomplished through the execution of load-generating scripts. This is useful to ensure that the application can handle the expected load and validate that the deployment is working as expected before releasing it to the production environment.
Sensitivity
You can set the sensitivity option as High, Medium, or Low.
Sensitivity controls how strictly the anomaly detection evaluates deviations in your deployment by adjusting the statistical threshold (measured in standard deviations) that determines what counts as an anomaly.
How it works
Harness uses machine learning to establish a baseline of normal behavior from your historical data. During deployment verification, it compares new data points against this baseline. The sensitivity setting determines how many standard deviations away from the baseline a data point must be before it is flagged as an anomaly:
High sensitivity (1σ threshold): Flags data points that deviate by 1 or more standard deviations from the baseline. This is the strictest threshold, catching smaller deviations that may indicate issues. Best for stable services with predictable behavior where even minor deviations may signal problems.
Medium sensitivity (2σ threshold): Flags data points that deviate by 2 or more standard deviations from the baseline. This provides a balanced threshold that filters out normal variance while catching significant anomalies. This is the default and works well for most production services.
Low sensitivity (3σ threshold): Flags only data points that deviate by 3 or more standard deviations from the baseline. This is the most lenient threshold, filtering out most normal variance and reducing false positives. Best for services with high variability or frequent changes where you want to catch only significant anomalies.
Choose the right sensitivity
Based on the statistical thresholds above, consider the following guidance when selecting sensitivity for your services:
Stable microservices with consistent traffic patterns
High
The 1σ threshold is strict and catches small deviations that may indicate issues in predictable services
Production services with moderate variance
Medium (default)
The 2σ threshold balances anomaly detection with tolerance for normal variance
Services with high variability (batch jobs, scheduled tasks)
Low
The 3σ threshold is lenient and filters out expected variance while catching extreme outliers
Services undergoing frequent changes or experiments
Low
Higher tolerance (3σ) reduces false positives from ongoing changes
Critical payment or security services
High
Lower tolerance (1σ) maximizes early detection of any deviation from normal behavior
Verification results by sensitivity
The following table shows how sensitivity affects the verification results based on the health status determined by anomaly detection.
High
Healthy
Pass
All metrics and logs are within acceptable thresholds (within 1σ). For logs: only Known event clusters detected. For metrics: all data points fall within the normal range established by baseline.
High
Medium Healthy
Fail
Anomalies detected that exceed 1σ threshold. For logs: Unexpected Frequency clusters found (known errors occurring more frequently). For metrics: data points deviate by 1-2σ from baseline. High sensitivity treats these deviations as deployment issues.
High
Medium Healthy (With feedback)
Pass
Same anomalies detected, but user has explicitly marked them as acceptable via event preferences. This overrides the automatic risk assessment.
High
Unhealthy
Fail
Severe anomalies detected. For logs: Unknown event clusters found (new errors not seen in baseline). For metrics: data points exceed 2σ deviation. These indicate genuine deployment problems.
Medium
Healthy
Pass
All metrics and logs within acceptable thresholds (within 2σ). Only Known event clusters for logs.
Medium
Medium Healthy
Pass
Minor anomalies detected but within medium tolerance (1-2σ). For logs: Unexpected Frequency clusters are tolerated. For metrics: deviations up to 2σ are considered normal variance. Medium sensitivity filters these as acceptable.
Medium
Medium Healthy (With feedback)
Pass
User feedback reinforces the default behavior of medium sensitivity.
Medium
Unhealthy
Fail
Severe anomalies exceeding 2σ threshold. For logs: Unknown event clusters (new errors). For metrics: data points deviate by more than 2σ from baseline.
Low
Healthy
Pass
All metrics and logs within acceptable thresholds (within 3σ). Only Known event clusters for logs.
Low
Medium Healthy
Pass
Moderate anomalies detected but within low tolerance (1-3σ). For logs: Unexpected Frequency clusters are tolerated. For metrics: deviations up to 3σ are filtered as normal variance. Low sensitivity is very permissive.
Low
Medium Healthy (With feedback)
Pass
User feedback aligns with the high tolerance of low sensitivity.
Low
Unhealthy
Fail
Extreme anomalies exceeding 3σ threshold. For logs: Unknown event clusters that are statistically significant outliers. For metrics: only data points with extreme deviations (>3σ) cause failure.
Per-metric sensitivity override
When your Verify step monitors a mix of metric types, a single step-level sensitivity setting often forces a difficult trade-off. Setting sensitivity to High catches genuine regressions on error-rate metrics but produces false positives on inherently noisy metrics like latency. Setting it to Low eliminates those false positives but risks missing real issues on critical metrics.
The per-metric sensitivity override solves this by letting you set an optional sensitivity field on individual metric definitions in a Monitored Service health source through the health source configuration UI. When set, the ML analysis for that specific metric uses the override instead of the step-level sensitivity. Metrics without an override continue to use the step-level setting, so existing configurations are unaffected.
Configure a per-metric sensitivity override
In the health source configuration UI, an optional Sensitivity dropdown is available per metric definition. Select High, Medium, or Low to override the step-level sensitivity for that metric. When left empty, the metric inherits the step-level sensitivity.
How the override is resolved
The resolution follows a simple rule: when the feature flag is enabled and a metric has a sensitivity value set, that value is passed to the Learning Engine for that metric's analysis. When the metric has no override, the step-level sensitivity is used as the fallback. If an invalid value is encountered, it falls back to the step-level setting and logs a warning.
The override is snapshotted at deployment start as part of the Monitored Service configuration, so edits to the Monitored Service during an active deployment do not affect in-flight analysis. The next deployment picks up the updated values.
The following table summarizes the resolution behavior:
HIGH
Not set
HIGH
HIGH
LOW
LOW
MEDIUM
HIGH
HIGH
Any
Invalid value
Step-level (with warning logged)
Scope and limitations
Per-metric sensitivity override applies to metric-based health sources only. Log analysis, SLI computation, and Service Guard continuous monitoring do not use this field. Metrics defined in canned metric packs (not user-defined metric definitions) also do not support the override in the current release.
Duration
Harness uses the data points within this duration for analysis. For instance, if you select 10 minutes, Harness analyzes the first 10 minutes of your log or APM data. Harness recommends choosing 10 minutes for logging providers and 15 minutes for APM and infrastructure providers. This helps you thoroughly analyze and detect issues before releasing the deployment to production.
Artifact tag
Use the Harness expression <+serviceConfig.artifacts.primary.tag> to reference this primary artifact. To learn about artifact expression, go to Service artifacts expressions.
Fail on no analysis
You can configure the pipeline to fail if there is no data from the health source. This ensures that the deployment fails when there is no data for Harness to analyze.
The AI Verify (v1) step also includes a metric-level option to fail the AI Verify (v1) step when the analysis of a given custom metric is not possible because there is no data for the custom metric on either the test nodes or the control nodes.
To enable the metric-level fail-on-no-analysis option, in the configuration pane of your AI Verify (v1) step, select Step Parameters, expand Optional, and select Fail if any custom metrics has no analysis.
Fail-fast thresholds
Fail-fast thresholds allow you to terminate verification immediately when specific conditions are met, enabling faster feedback on critical issues without waiting for the full analysis duration to complete.
Availability by data type
Metrics
Supported
Real-time threshold evaluation against Absolute Value, Percentage Deviation, or No Data criteria. Verification terminates immediately when a threshold is breached.
Logs
Supported (unknown cluster detection)
A per-query fail-fast toggle is available on log health sources. When enabled, verification terminates early if an unknown log cluster is detected during analysis.
Configure fail-fast for metrics
Metric health sources support three criteria types for fail-fast thresholds: Absolute Value, Percentage Deviation, and No Data. To configure a fail-fast threshold:
In your monitored service, navigate to the metric health source configuration and go to the Advanced (Optional) section.
Select the Fail-Fast Thresholds tab and select + Add Threshold.
From the Metric dropdown, select the metric.
In the Criteria field, select the criteria type:
Absolute Value: Enter a Greater than or Less than value.
Percentage Deviation: Enter a Less than percentage.
No Data: Enter the Fail if no data until value in minutes (integer, minimum 1).
In the Action field, select Fail Immediately. For the No Data criteria type, only Fail Immediately is available.
When a threshold is breached, verification terminates immediately and the deployment is marked as failed.
No Data fail-fast for metrics
The No Data criteria type terminates verification early when a metric produces no data within a configured window after deployment start. Use this when a missing metric indicates a broken query, wrong label, or instrumentation failure, and you do not want to wait for the full verification duration to surface the problem.
When you set a Fail if no data until value in minutes, Harness monitors whether any host records any data point for that metric within the configured window. If no data is collected by the time the window expires, verification terminates immediately. The metric is marked as Unhealthy with the reason "Custom fail-fast threshold."
Note the following behaviors:
A metric value of 0 is not a No Data failure; data was collected. The threshold triggers only when there are zero records.
If some hosts have data and others do not, verification continues. Any host with data in the window is sufficient to pass.
Delegate errors and query errors are not treated as No Data conditions.
The No Data criteria type is available only on the Fail-Fast Thresholds tab, not on the Ignore Thresholds tab.
Only Fail Immediately is supported for No Data. Fail after multiple or consecutive occurrences is not available for this criteria type.
Configure fail-fast for logs
Log health sources support a per-query fail-fast toggle that terminates verification early when an unknown log cluster is detected, rather than waiting for the full analysis window to complete.
To enable fail-fast on a log health source:
In your monitored service, go to the log health source configuration.
In the query definition, enable the Fail Fast toggle.
Save the monitored service.
When fail-fast is enabled and an unknown cluster is detected during analysis, the verify step terminates immediately and the deployment is marked as failed. The execution view shows a fail-fast banner to indicate that verification ended early due to an unknown log cluster.
Health source
Continuous Verification monitors health trend deviations using logs and metrics obtained from the health source, such as APM and logging tools, via a monitored service. A health source is an APM or logging tool that monitors and aggregates data in your deployment environment. You can add multiple health sources.
Service Instance Identifier (SII)
The Service Instance Identifier (SII) is a feature used in Continuous Verification to identify new or changed nodes during a deployment. It acts as a filter to pinpoint metrics related to what has been deployed. Continuous Verification uses the SII to calculate the deployed components based on observed metrics. This is particularly useful in scenarios like canary deployments, where the SII helps determine which nodes are stable and which nodes represent the canary in the current phase of analysis.
Use SII in Continuous Verification
Here's an example of how to set up Continuous Verification with a 5-minute analysis window using a Prometheus Health Source for a rolling deployment to a Kubernetes endpoint. This example also demonstrates using SII to filter and identify specific deployments.
PromQL:
SII: pod
Query execution process
Let's use the Prometheus query as an example to understand this process:
Queries the SII (in this case, pod) to list all possible pods within the specified time range.
/api/v1/label/**pod**/values?start=1685548800&end=1685549100&match[]={app="harness-cv-prom-example"}Retrieves the PromQL metric values by iterating over the returned SII results for each pod.
api/v1/label/**app**/values?start=1685548800&end=1685549100&match[]={app="harness-cv-prom-example"}
By leveraging the SII, Continuous Verification determines the pods that existed before and after the deployment by querying the monitoring solution. If a pod was present before and after, it is considered stable. If a pod is present after but not before, it is identified as a canary pod. The node determination is then made.
SII configuration tips
The purpose of the SII is to determine what has been deployed from a monitoring system. Different monitoring systems handle this differently, depending on how the system is configured and how labeling is applied to the deployed resources. In the query execution process, iterating over the SII is necessary. However, with other monitoring solutions, the process may vary, but the end result is the same: gathering information about what was available before and after deployment for node determination.
Potential SIIs
SII
Usage
pod/podname
Kubernetes
containername
ECS
version
When resources for deployment are tagged with versions, utilizing the version as SII can be an effective way to filter and identify specific deployments.
Next steps
To start using Continuous Verification, go to Configure verification.
To understand more about Continuous Verification, go to the knowledge base.
FAQs
For frequently asked questions about Continuous Verification, go to Continuous Verification FAQs.
Last updated
Was this helpful?