Ctrlk
Sign UpGet a DemoContact Us
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Argo Rollouts with Continuous Verification

Learn how to use Continuous Verification (CV) to validate canary deployments with Argo Rollouts.

Continuous Verification (CV) validates deployments by analyzing metrics and logs from your monitoring tools. When combined with Argo Rollouts canary deployments, CV ensures each canary stage is healthy before promoting to the next stage, providing automated safety checks throughout the progressive rollout.

Overview

Argo Rollouts with Continuous Verification enables you to:

  • Validate each canary stage: CV analyzes metrics and logs at each canary percentage (10%, 25%, 50%, 100%) before promotion

  • Automated rollback: If CV detects anomalies, it can automatically trigger rollback before the issue affects more users

  • Machine learning-based analysis: Harness uses ML algorithms to identify normal behavior and detect anomalies

  • Multi-stage validation: Each canary stage is independently validated before traffic increases

How CV Works with Canary Deployments

In a canary deployment with CV:

  1. Traffic Split: Argo Rollouts shifts a percentage of traffic to the new version (e.g., 10%)

  2. CV Analysis: The Verify step analyzes metrics and logs from the canary pods for the configured duration

  3. Health Decision: CV determines if the canary is healthy based on sensitivity settings

  4. Promotion or Rollback: If healthy, the rollout promotes to the next stage. If unhealthy, CV can trigger rollback

  5. Repeat: This process repeats for each canary stage (10% → 25% → 50% → 100%)

Prerequisites

Before setting up Argo Rollouts with Continuous Verification, ensure you have:

  • Argo Rollouts installed in your cluster (see Argo Rollouts Overview)

  • Harness GitOps Agent configured and connected to your cluster

  • GitOps Application with a Rollout resource configured for canary deployment

  • Monitored Service configured in Harness with health sources (APM or logging tools)

  • Health sources connected: Prometheus, Datadog, New Relic, or other supported monitoring tools

  • Pipeline created with deployment stage configured (see Managing Rollouts in Harness Pipeline)

For more information about setting up Continuous Verification, see Verify Overview.

Step-by-Step Guide

This guide walks you through the complete setup from creating a GitOps Application to running a pipeline with Continuous Verification.

Step 1: Prepare Your Git Repository

First, prepare your Git repository with the Rollout manifest and related Kubernetes resources.

1.1 Create the Rollout Manifest

Create a Rollout manifest file (e.g., rollout.yaml) in your Git repository. Your Argo Rollout manifest should define a canary strategy with pause steps. Here's a production-ready example:

Rollout manifest example

Key configuration elements:

  • replicas: 2: Number of pod replicas for the stable version. During canary, additional pods are created for the new version.

  • strategy.canary.steps: Defines the canary progression stages:

    • setWeight: 10 → Shifts 10% of traffic to canary (initial validation stage)

    • setWeight: 25 → Increases to 25% traffic

    • setWeight: 50 → Increases to 50% traffic

    • setWeight: 100 → Full promotion to 100% traffic

    • pause: {}: Pauses the rollout at each stage, waiting for CV validation or manual promotion

  • maxSurge: 25%: Allows up to 25% additional pods during rollout (beyond the replica count). This ensures capacity for both stable and canary versions.

  • maxUnavailable: 0: Ensures zero downtime by preventing any pods from being unavailable during the rollout.

  • minReadySeconds: 30: Waits 30 seconds after a pod becomes ready before considering it available. This helps ensure pods are truly ready before receiving traffic.

  • revisionHistoryLimit: 3: Keeps the last 3 rollout revisions for rollback purposes.

  • Prometheus annotations: The prometheus.io/scrape, prometheus.io/port, and prometheus.io/path annotations enable Prometheus to scrape metrics from the pods. This is essential for CV to collect metrics during canary validation.

  • Health probes:

    • livenessProbe: Checks if the container is running (restarts if unhealthy)

    • readinessProbe: Checks if the container is ready to receive traffic

  • Resource limits: Defines CPU and memory requests and limits to ensure proper resource allocation and prevent resource contention.

CV Integration Points:

  • Each pause: {} step is where your Verify step will validate the canary

  • Prometheus annotations enable CV to collect metrics for analysis

  • Health probes ensure pods are healthy before CV analysis begins

1.2 Commit and Push to Git

  1. Save the Rollout manifest to your Git repository

  2. Commit the changes

  3. Push to your repository (e.g., main branch)

Your repository structure should look similar to this:

Step 2: Create a Harness GitOps Application

You must create a GitOps Application before you can use it in a pipeline. The GitOps Application connects your Git repository to your Kubernetes cluster.

2.1 Navigate to GitOps Applications

  1. In Harness, navigate to Deployments > GitOps > Applications

  2. Click + New Application

2.2 Configure Application Basic Settings

Configure the application basic information:

  • Name: Enter a name for your application (e.g., loans-api-gitops)

  • GitOps Agent: Select the GitOps Agent connected to your cluster

  • Service (optional): Create or select a service to track this deployment (e.g., loans-api)

  • Environment (optional): Create or select an environment (e.g., prod-na)

Click Continue.

2.3 Configure Application Source

On the Source page, configure where your Rollout manifests are located:

  • Repository Type: Git

  • Repository URL: Your Git repository URL (e.g., https://github.com/org/my-gitops-repo)

  • Target Revision: Branch or tag (e.g., main, master)

  • Path: Path to your Rollout manifests (e.g., apps/loans-api/)

If using Kustomize:

  • Leave Kustomize options at their defaults, or configure as needed

Click Continue.

2.4 Configure Application Destination

Configure where the Rollout will be deployed:

  • Cluster: Select your target cluster

  • Namespace: Enter the namespace where the Rollout will be deployed (e.g., default)

Click Create.

2.5 Sync the Application

After creating the application, sync it to deploy the Rollout to your cluster:

  1. In the application details page, click Sync

  2. Configure sync options if needed:

    • Select Target Revision if needed (default is usually correct)

    • Leave other options at defaults

  3. Click Synchronize

Wait for the sync to complete and the application to reach Healthy state.

2.6 Verify Rollout is Deployed

  1. Navigate to the application's Resources tab

  2. Verify the following resources appear:

    • Rollout resource (e.g., loans-api)

    • Service resources (if configured)

    • ReplicaSet created by the Rollout

    • Pods managed by the Rollout

All resources should show as Healthy.

Rollout App Status

Your GitOps Application is now ready! The Rollout is deployed and synced to your cluster.

Step 3: Create a Monitored Service

Before adding Verify steps, create a Monitored Service that connects to your health sources:

  1. Navigate to Service Reliability Management > Monitored Services

  2. Click + New Monitored Service

  3. Configure:

    • Service: Select the service associated with your GitOps Application

    • Environment: Select the environment where the rollout is deployed

    • Name: Enter a descriptive name (e.g., loansapi_Prod_NA_GitOps_Product_Demo)

  4. Add Health Sources:

    • Click + Add New Health Source

    • Select your monitoring tool (e.g., Prometheus, Datadog)

    • Configure metrics and queries

    • Save the health source

    Prom Health Source

  5. Click Save

For detailed instructions, see Create a Monitored Service.

Step 4: Create Your Pipeline

Now that your GitOps Application and Monitored Service are configured, create a pipeline to orchestrate the canary deployment with CV validation.

4.1 Create a New Pipeline

  1. Navigate to Deployments > Pipelines

  2. Click + New Pipeline

  3. Enter a pipeline name (e.g., deploy-loans-api-canary)

  4. Select your project and organization

4.2 Add a Deployment Stage

  1. Click + Add Stage

  2. Select Deploy stage type

  3. Configure the stage:

Stage Configuration:

  • Name: Give your stage a name (e.g., Deploy Canary)

  • Service: Select the service that matches your GitOps Application (e.g., loans-api)

  • Environment: Select the environment that matches your GitOps Application (e.g., prod-na)

  • Infrastructure: Select the cluster/infrastructure

IMPORTANT

Critical: The service, environment, and cluster must match what you configured in your GitOps Application. This is required for the GitOps Rollout step to find and manage your rollout.

4.3 Configure Stage Execution

  1. Go to the Execution tab in your stage

  2. You'll see default steps (Service, GitopsClusters) - these are fine

  3. You can optionally add a GitOps Sync step if you want to sync changes before managing the rollout

Note: If your Rollout is already synced and healthy, you can skip the Sync step and go directly to the Rollout step.

4.4 Add GitOps Sync Step (Optional)

If you want to sync new changes from Git before managing the rollout:

  1. Click Add Step

  2. Search for and select GitOps Sync

  3. Configure the step:

    • Agent: Select your GitOps Agent

    • Application: Select your GitOps Application

    • Wait until healthy: Check this to wait until the application is healthy

    • Fail If Step Times Out: Check if you want the step to fail on timeout

    • Timeout: Set appropriate timeout (e.g., 10m)

Step 5: Add GitOps Rollout and Verify Steps

Add GitOps Rollout steps for each canary stage, followed by Verify steps for CV validation.

Pipeline Canary with Rolout

5.1 Add First Canary Stage (10%)

  1. Click Add Step

  2. Search for and select GitOps Rollout

  3. Configure the step:

Step Parameters:

  • Name: ArgoRollout Canary 10

  • Timeout: 10m

  • Agent: Select your GitOps Agent

  • Application: Select your GitOps Application (e.g., loans-api-gitops)

  • Rollout Name: Select your Rollout resource (e.g., loans-api)

  • Namespace: Enter the namespace (e.g., default)

Rollout Configuration:

  • Wait until Health Status: Select Suspended

  • Health Message: Enter CanaryPauseStep (or use regex pattern like Canary.*)

  • Auto Action for Rollouts: Leave empty (CV will validate first)

  • Fail On Timeout: Check this box

This step will wait until the rollout is paused at the 10% canary stage.

  1. In your pipeline, go to the Execution tab

  2. Click Add Step

  3. Search for and select GitOps Rollout

  4. Configure the step:

Step Parameters:

  • Name: ArgoRollout Canary 10

  • Timeout: 10m

  • Agent: Select your GitOps Agent

  • Application: Select your GitOps Application

  • Rollout Name: Select your Rollout resource (e.g., loans-api)

  • Namespace: Enter the namespace

Rollout Configuration:

  • Wait until Health Status: Select Suspended

  • Health Message: Enter CanaryPauseStep (or use regex pattern like Canary.*)

  • Auto Action for Rollouts: Leave empty (CV will validate first)

  • Fail On Timeout: Check this box

This step will wait until the rollout is paused at the 10% canary stage.

5.2 Add Verify Step After First Canary Stage

  1. Click Add Step after the first GitOps Rollout step

  2. Search for and select Verify

  3. Configure the step:

Step Parameters:

  • Name: Verify_1

  • Timeout: 2h (sufficient time for CV analysis)

  • Continuous Verification Type: Select Canary

  • Sensitivity: Select High (recommended for production)

  • Duration: Select 5 min (or adjust based on your needs)

  • Fail On No Analysis: Check this to fail if no data is available

Monitored Service:

  • Monitored Service Type: Select Default

  • Monitored Service Name: Select the Monitored Service you created in Step 2

Health Sources:

  • Verify your health sources are configured

  • Add additional health sources if needed

This Verify step will analyze metrics and logs from the 10% canary stage.

5.3 Add Wait Step (Optional)

You can add a Wait step after the Verify step to pause the pipeline before promoting to the next stage. This gives you time to review CV results:

  1. Click Add Step

  2. Search for and select Wait

  3. Configure:

    • Name: Wait

    • Duration: Set as needed (e.g., 5m)

5.4 Repeat for Remaining Canary Stages

Add similar steps for the remaining canary stages:

  • ArgoRollout Canary 25 → Verify_2 → Wait

  • ArgoRollout Canary 50 → Verify_3 → Wait

  • ArgoRollout Canary 100 → Final verification (optional)

For each GitOps Rollout step:

  • Update the Name to reflect the canary percentage (Canary 25, Canary 50, Canary 100)

  • Keep Wait until Health Status: Suspended

  • Keep Health Message: CanaryPauseStep

For each Verify step:

  • Update the Name (Verify_1, Verify_2, Verify_3)

  • Keep the same CV configuration (Canary type, Sensitivity, Duration)

  • Use the same Monitored Service

Step 6: Configure Verify Step for Canary Validation

The Verify step configuration is critical for proper canary validation:

Continuous Verification Type

Select Canary to match your deployment strategy. This tells CV to:

  • Compare canary pods (new version) against stable pods (old version)

  • Analyze metrics from both versions simultaneously

  • Detect anomalies in the canary version relative to baseline

Sensitivity

Choose the sensitivity level based on your risk tolerance:

  • High: Even minor anomalies cause verification failure (recommended for production)

  • Medium: Moderate anomalies are acceptable, major issues cause failure

  • Low: Only severe anomalies cause verification failure

For canary deployments, High sensitivity is recommended to catch issues early.

Duration

Set the analysis duration (e.g., 5 min). CV will:

  • Collect metrics and logs for this duration

  • Compare canary metrics against baseline

  • Make a health decision based on the analysis

Recommendation: Use 5-10 minutes for canary stages to get sufficient data while keeping deployment time reasonable.

Monitored Service

Select the Monitored Service that includes:

  • Health sources connected to your monitoring tools

  • Metrics and queries configured for your application

  • Proper service and environment mapping

CV Config

Step 7: Run the Pipeline

Now you're ready to run the pipeline and see the canary deployment with CV validation in action.

Pipeline YAML

7.1 Save and Run

  1. Click Save to save your pipeline

  2. Click Run to execute the pipeline

  3. The pipeline will start executing

7.2 Monitor Pipeline Execution

Watch the pipeline execution:

  1. Service Step: Should complete quickly (green checkmark)

  2. GitopsClusters Step: Should complete quickly (green checkmark)

  3. GitOps Sync Step (if added): Will sync the application if there are changes

  4. GitOps Rollout Step (Canary 10):

    • Will wait until the rollout is paused at 10%

    • Will detect the Suspended status with CanaryPauseStep message

  5. Verify Step:

    • Will analyze metrics and logs from the 10% canary

    • Will compare canary metrics against stable baseline

    • Will determine health status

  6. Repeat: Process continues for 25% → 50% → 100%

7.3 View Rollout Progression

During execution, you can:

  1. View Step Logs: Click on the GitOps Rollout step to see detailed logs

    • You'll see messages like: "Rollout 'loans-api' is SUSPENDED. Current weightage: 10 of 100"

    • The step will progress through stages: 10% → 25% → 50% → 100%

  2. View in GitOps UI: Navigate to Deployments > GitOps > Applications to see the rollout progress in real-time

  3. View Rollout Status: Check the rollout status in the GitOps Application Resources tab

Rollout Status

7.4 Successful Completion

When the pipeline completes successfully:

  • All steps will show green checkmarks

  • The GitOps Rollout steps will show progression through all canary stages

  • The Verify steps will show successful validation at each stage

  • The rollout will be at 100% weightage and HEALTHY status

Congratulations! Your canary deployment with CV validation is working! 🎉

Step 8: Understanding the Complete Workflow

Here's how the complete workflow operates:

  1. Pipeline Starts: GitOps Sync step (if present) syncs the Rollout manifest

  2. Rollout Begins: Argo Rollouts starts the canary deployment

  3. 10% Traffic Shift: Rollout shifts 10% of traffic to canary version (initial validation stage)

  4. Rollout Pauses: Argo Rollouts pauses at the first canary step

  5. GitOps Rollout Step: Detects the Suspended status with CanaryPauseStep message

  6. Verify Step Starts: CV begins analyzing metrics and logs from the 10% canary

  7. CV Analysis:

    • Collects data for the configured duration (e.g., 5 minutes)

    • Compares canary metrics against stable baseline

    • Applies ML algorithms to detect anomalies

    • Determines health status (Healthy, Medium Healthy, or Unhealthy)

  8. Health Decision:

    • If Healthy: Verify step passes, pipeline continues

    • If Unhealthy: Verify step fails, pipeline can trigger rollback

    • If Medium Healthy: Depends on sensitivity setting

  9. Promotion: If CV passes, GitOps Rollout step promotes to next stage (25%)

  10. Repeat: Process repeats for 25% → 50% → 100%

Step 9: Configure Auto-Promotion (Optional)

To fully automate the canary progression, you can configure the GitOps Rollout step to automatically promote after CV validation:

  1. In each GitOps Rollout step, set Auto Action for Rollouts: Promote Full

  2. This will automatically promote to the next stage after the Verify step passes

  3. The pipeline will progress through all stages without manual intervention

Note: Ensure your Verify steps are configured correctly, as auto-promotion will proceed even if CV detects issues (depending on sensitivity and fail settings).

Step 10: Configure Rollback on CV Failure

To automatically rollback when CV detects issues:

  1. In your pipeline, add a Rollback step after each Verify step

  2. Configure conditional execution:

    • Conditional Execution: Enable

    • Execute on: Select On Failure of Previous Step

  3. Configure rollback action:

    • Use GitOps Rollout step with Auto Action: Abort

    • Or use GitOps Sync step to sync previous version from Git

Alternatively, configure the Verify step to fail the pipeline, which can trigger automatic rollback if configured in your failure strategy.

How CV Validates Canary Deployments

Metrics Comparison

CV compares metrics from canary pods against stable pods:

  • Response Time: Canary response times vs. stable baseline

  • Error Rate: Error rates in canary vs. stable

  • Throughput: Request rates and patterns

  • Resource Usage: CPU, memory, and other resource metrics

  • Custom Metrics: Any custom metrics configured in your health sources

Anomaly Detection

Harness uses machine learning to:

  1. Establish Baseline: Learn normal behavior from stable pods

  2. Monitor Canary: Collect real-time metrics from canary pods

  3. Detect Deviations: Identify anomalies that deviate from baseline

  4. Classify Severity: Determine if anomalies are critical, moderate, or minor

  5. Make Decision: Based on sensitivity, decide if canary is healthy

Health Status Determination

CV determines health status based on:

  • Sensitivity Setting: High, Medium, or Low

  • Anomaly Severity: Critical, moderate, or minor deviations

  • Metric Trends: Whether metrics are improving or degrading

  • Duration: Analysis over the configured time window

Health Status Values:

  • Healthy: No significant anomalies detected

  • Medium Healthy: Minor anomalies detected (may pass with Medium/Low sensitivity)

  • Unhealthy: Critical anomalies detected (always fails)

Next Steps

You've learned how to use Continuous Verification with Argo Rollouts for canary deployments. You can now:

PreviousGitOps Sync with Multiple Rollout Steps
NextApplicationSets

Last updated

Was this helpful?