> For the complete documentation index, see [llms.txt](https://developer.harness.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://developer.harness.io/continuous-integration/use-harness-ci/use-harness-ci/run-tests/test-management/ci-flaky-tests.md).

# Flaky Test Detection

Automatically identify tests that pass and fail inconsistently

## What you will learn from this topic

* How flaky tests are detected automatically in Harness CI
* How to view flaky tests in the Harness UI and via CLI
* How to manually mark a test as flaky or stable
* How to automate flaky test management with policies

## Before you begin

* **Feature flag**: Flaky Test Detection requires the feature flag `TI_POLICY_EVALUATION_ENABLED`. Contact [Harness Support](mailto:support@harness.io) to enable it.

{% hint style="info" %}
**PREREQUISITES**

Flaky Test Detection requires the feature flag `TI_POLICY_EVALUATION_ENABLED` to be enabled on your account.

Contact [Harness Support](mailto:support@harness.io) to enable the feature flag.
{% endhint %}

## Flaky test detection <a href="#flaky-test-detection" id="flaky-test-detection"></a>

A flaky test is one that passes and fails inconsistently - without any code changes. Harness automatically detects these tests and marks them with a **FLAKY** badge so your team knows which tests are unreliable.

### How detection works <a href="#how-detection-works" id="how-detection-works"></a>

Harness automatically detects flaky tests by analyzing test results across pipeline executions. A test is identified by its **suite name + class name + repository + test name**.

### Detection criteria <a href="#detection-criteria" id="detection-criteria"></a>

A test is marked flaky when **both conditions** are met:

1. **Same-commit inconsistency**: The test has both **passed AND failed** on the same commit
2. **Within the observation window**: The inconsistency occurred within the last **14 days**

**Example:** If a build for commit `abc123` ran twice - once where test passed and once where it failed - the test is marked flaky.

{% hint style="info" %}
**WHY 14 DAYS?**

The 14-day window defines how far back Harness looks when detecting flaky tests. Test behavior older than 14 days is ignored. This ensures flaky detection reflects recent, relevant behavior, not issues that were fixed weeks ago.
{% endhint %}

### Auto-Recovery <a href="#auto-recovery" id="auto-recovery"></a>

Flaky status is automatically cleared in two ways:

1. **5 consecutive passes**: After 5 successful runs with no flaky occurrences, the status clears immediately
2. **Time expiry**: If no same-commit inconsistency occurs for 14 days, the flaky status naturally expires

### Detection parameters <a href="#detection-parameters" id="detection-parameters"></a>

| Parameter          | Value                      |
| ------------------ | -------------------------- |
| Observation window | 14 days                    |
| Flaky trigger      | Pass + fail on same commit |
| Auto-recovery      | 5 consecutive passes       |
| Scope              | Per repository             |

### View flaky tests <a href="#view-flaky-tests" id="view-flaky-tests"></a>

### In the Harness UI <a href="#in-the-harness-ui" id="in-the-harness-ui"></a>

Complete the following steps to view flaky tests in the UI:

1. Open your pipeline execution
2. Click the **Tests** tab
3. Look for the **FLAKY** badge next to test names
4. Use **Filter → Flaky** to show only flaky tests

### Via CLI <a href="#via-cli" id="via-cli"></a>

List all flaky tests for a repository:

```bash
hcli test-management flaky get \
  --account-id="$HARNESS_ACCOUNT_ID" \
  --repo="https://github.com/your-org/your-repo.git" \
  --api-key="$HARNESS_API_KEY" \
  --endpoint="https://app.harness.io/gateway/ti-service"
```

Example output:

```bash
Found 6 flaky test(s):
  - com.example.PaymentTest::testRefundTimeout
  - com.example.ApiTest::testWebhookRetry
  - tests.integration.test_api::test_concurrent_requests
```

### Manually mark a test <a href="#manually-mark-a-test" id="manually-mark-a-test"></a>

Sometimes you know a test is flaky before automatic detection catches it. Mark it manually:

```bash
hcli test-management flaky set \
  --account-id="$HARNESS_ACCOUNT_ID" \
  --repo="https://github.com/your-org/your-repo.git" \
  --api-key="$HARNESS_API_KEY" \
  --endpoint="https://app.harness.io/gateway/ti-service" \
  --class-name="com.example.PaymentTest" \
  --test-name="testRefundTimeout" \
  --marking=true
```

### Options for manual marking <a href="#marking-options" id="marking-options"></a>

| `--marking` value | Effect                                           |
| ----------------- | ------------------------------------------------ |
| `true`            | Force mark as flaky (overrides auto-detection)   |
| `false`           | Force mark as stable (overrides auto-detection)  |
| `unset`           | Remove manual marking, let auto-detection decide |

### Flaky vs quarantine <a href="#flaky-vs-quarantine" id="flaky-vs-quarantine"></a>

|                              | Flaky                    | Quarantine                               |
| ---------------------------- | ------------------------ | ---------------------------------------- |
| **Test runs?**               | Yes                      | Yes                                      |
| **Failure blocks pipeline?** | Yes (unless quarantined) | No                                       |
| **Purpose**                  | Track unreliable tests   | Unblock deployments                      |
| **Recovery**                 | Automatic (5 passes)     | Manual removal or automatic via policies |

A test can be both flaky AND quarantined. When quarantined, the flaky test runs but does not block the pipeline.

### Common causes of flaky tests <a href="#common-causes-of-flaky-tests" id="common-causes-of-flaky-tests"></a>

| Cause                   | Example                                | Fix                           |
| ----------------------- | -------------------------------------- | ----------------------------- |
| **Race conditions**     | Test depends on thread timing          | Add synchronization or waits  |
| **External services**   | Test calls real APIs                   | Mock external dependencies    |
| **Shared state**        | Tests do not clean up after themselves | Isolate test data             |
| **Time sensitivity**    | Test checks "now" vs. expected time    | Use fixed test clocks         |
| **Resource contention** | Tests compete for ports/files          | Use unique resources per test |

### Automate with policies <a href="#automate-with-policies" id="automate-with-policies"></a>

Instead of manually managing flaky tests, use [policies](/continuous-integration/use-harness-ci/use-harness-ci/run-tests/test-management/ci-test-policies.md) to automate quarantine:

```json
[
  {
    "when": ["test is flaky", "test failed"],
    "action": ["mark quarantine"]
  }
]
```

This policy automatically quarantines any flaky test that fails, preventing it from blocking your pipeline while still tracking its status.

## Troubleshooting

<details>

<summary>A step fails unexpectedly</summary>

Contact [Harness Support](mailto:support@harness.io).

</details>

## Next steps

* Go to [Test Quarantine](/continuous-integration/use-harness-ci/use-harness-ci/run-tests/test-management/ci-test-quarantine.md) to unblock deployments by quarantining flaky tests.
* Go to [Test Policies](/continuous-integration/use-harness-ci/use-harness-ci/run-tests/test-management/ci-test-policies.md) to automate test management decisions.

{% @harness-feedback/feedback %}
