Test Quarantine

Skip unstable tests to unblock deployments while you investigate

PREREQUISITES

Test Quarantine requires the feature flag TI_POLICY_EVALUATION_ENABLED to be enabled on your account. To use hcli commands for managing quarantined tests, you must also set the environment variable CI_ENABLE_HCLI_FOR_TESTS=true in your pipeline.

Contact Harness Support to enable the feature flag. Go to Environment variables to configure environment variables.

Test Quarantine

When an unstable test blocks every deployment, quarantine lets you bypass it while you investigate. Quarantined tests still run, so you see if they're fixed, but their failures don't block the pipeline.

QUARANTINE IS TEMPORARY

Create a tracking issue with a deadline for every quarantined test. Quarantined tests represent untested code paths.

Prerequisites

Before using test quarantine commands, ensure you have:

• Harness account access: Account ID for your Harness account.

• Personal Access Token (PAT): Required for API authentication. Go to Manage API keys to create a PAT.

• hcli installed: Harness CLI tool must be available in your environment. Go to Install and configure Harness CLI to download hcli for your operating system and architecture.

---

Enable Quarantine in Your Pipeline

Many test steps are configured to fail the build when any test fails. When quarantine is enabled, Harness evaluates the test results and ignores failures from quarantined tests when determining the step status.

Quarantined tests still run and their results are recorded, but they won’t cause the step to fail.

To enable this behavior, add the following environment variable to your test step:

env:
  CI_ENABLE_QUARANTINED_TEST_SKIP: "true"

Full pipeline example:

Python

- run:
    script: |-
      pytest --junitxml=test-results.xml -v
      hcli test-reports upload test-results.xml
    env:
      CI_ENABLE_QUARANTINED_TEST_SKIP: "true"

Java/Gradle

- step:
    type: Run
    name: Run Tests
    spec:
      shell: Sh
      command: |
        ./gradlew clean test
        hcli test-reports upload "build/test-results/test/*.xml"
      envVariables:
        CI_ENABLE_QUARANTINED_TEST_SKIP: "true"

When to Quarantine

Situation	Quarantine?
Flaky test blocks every deployment	Yes
External service is temporarily down	Yes
Urgent release needed, will fix tomorrow	Yes
Test is slow but always passes	No, optimize it
Test fails consistently on all runs	No, fix the bug

How Quarantine Works

Original Result	With Quarantine	Pipeline Impact
FAIL	PASS (ignored)	Does not block
PASS	PASS	No change
SKIP	SKIP	No change

The test still executes, you'll see if it passes or fails—but failures are ignored for pipeline status.

Quarantine a Test

Use the set command with the --quarantine=true flag to quarantine a test:

hcli test-management set \
  --account-id="$HARNESS_ACCOUNT_ID" \
  --repo="<URL_for_your_repo>" \
  --endpoint="https://<your-instance>/gateway/ti-service/" \
  --api-key="$HARNESS_PAT" \
  --class-name="com.example.PaymentTest" \
  --test-name="testRefundTimeout" \
  --quarantine=true

REPOSITORY URL FORMAT

The .git suffix is optional. If omitted, it will be appended automatically.

WHEN TO USE SUITE NAME

Use the --suite-name parameter when your JUnit XML reports include a test suite structure, and you need to uniquely identify a test within a specific suite. This is helpful when the same test name appears in multiple suites.

ENDPOINT VARIES BY INSTANCE

The --endpoint value is based on your Harness instance domain: https://<your-instance>/gateway/ti-service/

For example, if you access Harness at https://your-instance.harness.io, use https://your-instance.harness.io/gateway/ti-service/.

Required Parameters

• --account-id: Your Harness account ID

• --repo: Repository URL (with or without .git suffix)

• --endpoint: TI service endpoint URL

• --api-key: Your Personal Access Token (PAT) starting with pat.

• --class-name: Test class name (must match the class name in your JUnit XML report)

• --test-name: Test method name (must match the test name in your JUnit XML report)

• --quarantine: Set to true to quarantine, false to remove from quarantine, or unmark to remove the marking entirely

Optional Parameters

• --suite-name: Test suite name (if the test belongs to a specific suite in your JUnit XML report)

• --org-id: Organization ID (if scoped to an organization)

• --project-id: Project ID (if scoped to a project)

View Quarantined Tests

hcli test-management get \
  --account-id="$HARNESS_ACCOUNT_ID" \
  --repo="<URL_for_your_repo>" \
  --endpoint="https://<your-instance>/gateway/ti-service/" \
  --api-key="$HARNESS_PAT"

Required Parameters

• --account-id: Your Harness account ID

• --repo: Repository URL (with or without .git suffix)

• --endpoint: TI service endpoint URL

• --api-key: Your Personal Access Token (PAT) starting with pat.

Optional Parameters

• --org-id: Organization ID (filters results to a specific organization)

• --project-id: Project ID (filters results to a specific project)

Example output:

Found 6 quarantined test(s):
  - com.taskmanager.FlakyTest::flakyTest1
  - com.taskmanager.FlakyTest::Flaky Test 2: Timing sensitive
  - com.taskmanager.FlakyTest::Flaky Test 3: Random number comparison
  - com.taskmanager.service.WorkItemServiceTest::Flaky test that randomly fails
Page 1 of 1 (total: 6 tests)

Remove from Quarantine

After fixing the test, use the set command with --quarantine=false:

hcli test-management set \
  --account-id="$HARNESS_ACCOUNT_ID" \
  --repo="<URL_for_your_repo>" \
  --endpoint="https://<your-instance>/gateway/ti-service/" \
  --api-key="$HARNESS_PAT" \
  --class-name="com.example.PaymentTest" \
  --test-name="testRefundTimeout" \
  --quarantine=false

---

Advanced: Set Multiple Test Statuses

The set command can update both flaky and quarantine status in a single command:

hcli test-management set \
  --account-id="$HARNESS_ACCOUNT_ID" \
  --repo="<URL_for_your_repo>" \
  --endpoint="https://<your-instance>/gateway/ti-service/" \
  --api-key="$HARNESS_PAT" \
  --class-name="com.example.PaymentTest" \
  --test-name="testRefundTimeout" \
  --flaky=true \
  --quarantine=true

Available Values

--flaky and --quarantine accept true, false, and unmark:

• true: Mark the test with the status

• false: Remove the status from the test

• unmark: Remove any manual override and let auto-detection decide

Examples

Mark a test as flaky only:

hcli test-management set \
  --class-name="tests.TestSuite" \
  --test-name="test_method" \
  --flaky=true \
  --account-id="$HARNESS_ACCOUNT_ID" \
  --repo="<URL_for_your_repo>" \
  --endpoint="https://<your-instance>/gateway/ti-service/" \
  --api-key="$HARNESS_PAT"

Quarantine a test with suite name:

hcli test-management set \
  --class-name="com.example.PaymentTest" \
  --test-name="testRefundTimeout" \
  --suite-name="com.example.PaymentTestSuite" \
  --quarantine=true \
  --account-id="$HARNESS_ACCOUNT_ID" \
  --repo="<URL_for_your_repo>" \
  --endpoint="https://<your-instance>/gateway/ti-service/" \
  --api-key="$HARNESS_PAT"

Remove quarantine marking entirely:

hcli test-management set \
  --class-name="tests.TestSuite" \
  --test-name="test_method" \
  --quarantine=unmark \
  --account-id="$HARNESS_ACCOUNT_ID" \
  --repo="<URL_for_your_repo>" \
  --endpoint="https://<your-instance>/gateway/ti-service/" \
  --api-key="$HARNESS_PAT"

---

Automate with Policies

Instead of manually quarantining tests, use policies to automate:

[
  {
    "when": ["test failed"],
    "action": ["mark quarantine"]
  }
]

This automatically quarantines any test that fails in the current pipeline execution.

Best Practices

Practice	Why
Create a tracking issue	Every quarantined test needs an owner
Set a deadline	Quarantine should be temporary (7 days max)
Review weekly	Prevent accumulation of quarantined tests
Fix the root cause	Don't just quarantine—investigate and fix

Next Steps

• Set up policies to automate quarantine decisions

• View flaky tests to find quarantine candidates