> For the complete documentation index, see [llms.txt](https://developer.harness.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://developer.harness.io/harness-ai/use-harness-platform/pipelines/failure-handling/define-a-failure-strategy-on-stages-and-steps.md). # Define failure strategies for stages and steps A failure strategy defines how your stages and steps handle different failure conditions. A failure strategy defines how steps, stages, and step groups respond to different failure conditions. A failure strategy consists of error conditions that trigger the strategy and actions to take when those conditions occur. Failure strategies are a critical component of pipeline design. They determine which failures cause a step or stage to fail and how the pipeline should respond when a failure occurs. {% embed url="" %} Configure failure strategies {% endembed %} *** ## What you will learn from this topic * How to [configure failure strategies](#configure-failure-strategies) for stages, steps, and step groups. * How to [define stage failure strategies](#add-a-stage-failure-strategy) and [step failure strategies](#add-a-step-failure-strategy) in the Visual editor. * How to understand [error types](#error-types), [failure strategy actions](#failure-strategy-actions), and [prioritization rules](#prioritization-and-handling). *** ## Before you begin * **Pipeline access:** You need Create or Edit permissions on pipelines to configure failure strategies. For more information, refer to [RBAC in Harness](/harness-ai/use-harness-platform/platform-access-control.md). * **Pipeline basics:** You should understand stages and steps in pipelines. For more information, refer to [Add a stage](/harness-ai/use-harness-platform/pipelines/add-a-stage.md). *** ## Configure failure strategies You can apply failure strategies to: * **Stages:** A stage failure strategy applies to all steps and step groups in the stage that do not have step-level failure strategies. * **Steps:** This failure strategy overrides or enhances the stage failure strategy. * **Step groups:** You can set up a failure strategy for all steps in the group. Individual steps in the group do not have a failure strategy. * **Pipelines:** [Pipeline rollback](/harness-ai/use-harness-platform/pipelines/failure-handling/define-a-failure-strategy-for-pipelines.md) is a failure strategy for all stages in a pipeline. The pipeline rolls back if any of the stages in the pipeline fails. ### Add a stage failure strategy The stage failure strategy applies to all steps in the stage that do not have their own failure strategy configured. 1. In your pipeline, select the stage where you want to add the failure strategy. For more information, refer to [Add a stage](/harness-ai/use-harness-platform/pipelines/add-a-stage.md). 2. Select the **Advanced** tab. 3. Under **Failure Strategy**, the default stage failure strategy is shown: ``` On all errors other than those specified in failure strategies defined here, perform action. ``` You cannot remove the default strategy, but you can edit it to choose a different **Action**, **Timeout**, and **Post timeout action**. 4. To add an additional stage failure strategy, select **Add**, and then configure the [failure strategy settings](#failure-strategy-settings): * **On failure of type:** Select one or more of the error types to trigger the failure strategy. * **Perform Action:** Select the action that should occur when the specified failure event happens. * **Timeout** and **Post timeout action:** These are available if you selected **Manual Intervention** for the **Action**. The manual intervention action allows a user to intervene and choose an **Action** when the specified failure event occurs. You can enter a **Timeout** for the user to select an action, and a **Post Timeout Action** to fallback on if the user does not manually select an action in a certain amount of time. * **Retry Count**, **Retry Intervals**, and **Post retry failure action:** These are available if you selected **Retry** for the **Action**. Enter the number of times to retry the stage, the retry interval between attempts, and the **Post retry failure action** to specify what action to take when all retry attempts have been exhausted and failed. For example, you can configure the stage to rollback or abort after all retries fail. ### Stage failure strategy YAML examples The following examples demonstrate how to configure stage-level failure strategies in YAML.
Stage rollback on all errors This example shows a deployment stage configured to automatically roll back when any error occurs during stage execution. ```yaml - stage: name: k8s identifier: k8s description: "" type: Deployment spec: ... tags: {} failureStrategies: - onFailure: errors: - AllErrors action: type: StageRollback ```
Retry stage with rollback on retry failure This example demonstrates a deployment stage that retries up to 2 times with a 1-hour interval between retries. If all retry attempts fail, the stage performs a rollback. ```yaml stages: - stage: name: k8s identifier: k8s description: "" type: Deployment spec: ... tags: {} failureStrategies: - onFailure: errors: - AllErrors action: type: Retry spec: onRetryFailure: action: type: StageRollback retryCount: 2 retryIntervals: - 1h ```
### Add a step failure strategy Steps do not have a default failure strategy. Instead, steps inherit the stage failure strategy if there is no step-level failure strategy. When you add a step failure strategy, you override the stage failure strategy for that step. To add a step failure strategy: 1. Edit the step where you want to add the failure strategy. 2. Select the **Advanced** tab. 3. Select **Failure Strategy**, select **Add**, and then configure the [failure strategy settings](#failure-strategy-settings): * **On failure of type:** Select one or more of the error types to trigger the failure strategy. * **Perform Action:** Select the action that should occur when the specified failure event happens. * **Timeout** and **Post timeout action:** These are available if you selected **Manual Intervention** for the **Action**. The manual intervention action allows a user to intervene and choose an **Action** when the specified failure event occurs. You can enter a **Timeout** for the user to select an action, and a **Post Timeout Action** to fallback on if the user does not manually select an action in a certain amount of time. * **Retry Count**, **Retry Intervals**, and **Post retry failure action:** These are available if you selected **Retry** for the **Action**. Enter the number of times to retry the step, the retry interval between attempts, and the **Post retry failure action** to specify what action to take when all retry attempts have been exhausted and failed. For example, you can configure the step to perform a stage rollback or abort after all retries fail. ### Step failure strategy YAML examples The following examples demonstrate how to configure step-level failure strategies in YAML.
Manual intervention on failure This example shows a Kubernetes rolling deployment step with a failure strategy that triggers manual intervention when any error occurs. If no manual action is taken within 1 hour, the pipeline automatically aborts. ```yaml - step: name: Rollout Deployment identifier: rolloutDeployment type: K8sRollingDeploy timeout: 10m spec: skipDryRun: false pruningEnabled: false failureStrategies: - onFailure: errors: - AllErrors action: type: ManualIntervention spec: onTimeout: action: type: Abort timeout: 1h ```
Ignore failure This example shows a deployment step configured to ignore all errors and continue pipeline execution. In the UI, the step will be marked as **success (failure ignored)**, and the pipeline will proceed to the next step. ```yaml - step: name: Rollout Deployment identifier: rolloutDeployment type: K8sRollingDeploy timeout: 10m spec: skipDryRun: false pruningEnabled: false failureStrategies: - onFailure: errors: - AllErrors action: type: Ignore ```
Retry with stage rollback on retry failure This example demonstrates a deployment step that retries up to 2 times with a 1-hour interval between retries. If all retry attempts fail, the stage performs a rollback. ```yaml - step: name: Rollout Deployment identifier: rolloutDeployment type: K8sRollingDeploy timeout: 10m spec: skipDryRun: false pruningEnabled: false failureStrategies: - onFailure: errors: - AllErrors action: type: Retry spec: retryCount: 2 onRetryFailure: action: type: StageRollback retryIntervals: - 1h ```
### Failure strategies as runtime input You can also define stage, step, and step group failure strategies at runtime by configuring them as [runtime inputs](/harness-ai/use-harness-platform/variables-and-expressions/runtime-inputs.md). To do this, go to the **Failure Strategy** settings where you want to configure a failure strategy to be specified at runtime, select the **Thumbtack** icon, and change the input type to **Runtime Input**. When you run the pipeline, you will be prompted to define the failure strategy settings for that run. Due to the potential complexity of failure strategies, [input sets](/harness-ai/use-harness-platform/pipelines/input-sets.md) are useful for failure strategies as runtime input. Input sets contain pre-defined runtime inputs that you select at runtime. This eliminates the need to manually define the entire failure strategy each time. ### Failure strategies for CD steps and stages For guidance on configuring failure strategies for Continuous Delivery (CD) stages and steps, refer to [Define a failure strategy on Harness CD stages and steps](https://developer.harness.io/docs/continuous-delivery/x-platform-cd-features/executions/step-failure-strategy-settings/). *** ## Failure strategy settings ### Error types The following error types can be selected in a failure strategy. | **Error Type** | **Description** | | ---------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Authentication Errors** | Credentials provided in a connector are not valid. Typically, the Harness secret used for one of the credentials is incorrect. If Harness cannot determine if the error is for authentication or authorization, it is treated as an authentication error. | | **Authorization Errors** | The credentials are valid but the user permissions needed to access the resource are not sufficient. If Harness cannot determine if the error is for authentication or authorization, it is treated as an authentication error. | | **Connectivity Errors** | A Harness Delegate cannot connect to a specific resource, such as a repository, VM, or secrets manager. | | **Delegate Provisioning Errors** | No available delegate can accomplish the task, or the task is invalid. For example, if an HTTP step attempts to connect to a URL but there is no available delegate to perform the task. | | **Timeout Errors** | A Harness Delegate fails to complete a task within the stage/step timeout limit. For example, if the Kubernetes workload you are deploying fails to reach a steady state within the step timeout limit. | | **Unknown Errors** | Errors that do not fall into any other category. This includes Harness application errors. | | **Verification Failures** | A Harness Continuous Verification step fails. | | **Policy Evaluation Failures** | An Open Policy Evaluation (OPA) applied on a step fails. | | **Execution-time Inputs Timeout Errors** | A step times out when running a pipeline due to the unavailability of a runtime input. | | **Approval Rejection** | An approval step is rejected. You can select specific failure strategies for approval rejection across steps and stages. | | **Delegate Restart** | An error triggered when the delegate is unreachable when running a pipeline. | | **User Marked Failure** | A failure manually marked by the user during pipeline execution. | | **All Errors** | Any error, whether defined by the other error types or not. | ### Error scope The scope of a failure strategy is confined to where it is set. For example, a failure strategy set on a step does not impact the failure strategy set on a stage. Likewise, the failure strategy set at the stage does not override any failure strategies on its steps. ### Rollback stage Both step and stage failure strategies include the **Rollback Stage** action option. There is no rollback step option. *** ## Failure strategy actions The following table lists the failure strategy actions and how they work at the step, step group, and stage levels. These actions can be applied to the failure strategy as primary action and timeout action. | Action | Step | Step Group | Stage | | ------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------- | | **Manual Intervention** | A Harness user can perform a manual intervention when the error type occurs. There are several options to select from: **Mark as Success** **Ignore Failure** **Retry** **Abort** **Rollback Stage**Harness pauses the pipeline execution when waiting for manual intervention. The pipeline execution state appears as **Paused**. | Same as step. | Same as step, but applies to all steps. | | **Mark as Success** | The step is marked as **Successful** and the stage execution continues. | Same as step. | The failed step is marked as **Successful** and the pipeline execution continues. | | **Ignore Failure** | The stage execution continues. In the UI, the step is marked as **success (failure ignored)**, and rollback is not triggered. | Same as step. | Same as step. | | **Proceed with Default Values** | The step proceeds with default values when the failure occurs. This is typically used when optional inputs or runtime values are not provided. | Same as step. | Same as step. | | **Retry Step** | Harness retries the execution of the failed step automatically. You can set **Retry Count** and **Retry Intervals**. You can also configure a **Post retry failure action** to specify what happens when all retry attempts fail (for example, abort or rollback stage). Additionally, you can **define a JEXL condition to retry the step only when a specific condition is met**. | Same as step. | Same as step. | | **Retry Step Group** | N/A | Harness will retry the execution of the complete step group automatically, from the beginning. You can set **Retry Count** and **Retry Intervals**. You can also configure a **Post retry failure action** to specify what happens when all retry attempts fail. | N/A | | **Abort** | Pipeline execution is aborted. If you select this option, no timeout is needed. | Same as step. | Same as step. | | **Rollback Stage** | The stage rolls back to the state prior to stage execution. How the stage rolls back depends on the type of build or deployment it was performing. | Same as step. | Same as step. | | **Rollback Step Group** | N/A | The step group rolls back to the state prior to step group execution. How the step group rolls back depends on the type of build or deployment it was performing. | N/A | | **Mark As Failure** | Harness marks the step as **Failed**. | Harness marks the step group as **Failed**. | Harness marks the stage as **Failed** and executes the next stage. | {% hint style="info" %} **Note** **Mark As Failure** as a Failure Strategy marks the stage, step group, or step as failed and moves the execution to next step or stage according to when conditions applied on the next step or stage. {% endhint %} ### Manual interventions Here is what a Manual Intervention action looks like when a failure occurs: ![](https://173309742-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F3F2TpHXhur2QtQnORSM9%2Fuploads%2Fgit-blob-38df552e380978aaf9b4b8ef3c4ebdf9b2a1aa85%2Fdefine-a-failure-strategy-on-stages-and-steps-11.png?alt=media) You can select an **Action**. If the Manual Intervention exceeds the **Timeout** setting, Harness automatically selects the **Post Timeout Action**. ### Customize available actions for manual intervention {% hint style="info" %} Currently, this feature is behind the feature flag `CDS_MANUAL_INTERVENTION_CUSTOM_ACTIONS`. Contact [Harness Support](mailto:support@harness.io) to enable the feature. {% endhint %} You can now **restrict the set of available actions shown to the pipeline executor** during a manual intervention. This applies when a **step** or **stage** fails and the configured **Failure Strategy** is **Manual Intervention**. By customizing the available actions, you can: * Prevent unsafe actions like **Retry** or **Pipeline Rollback** * Ensure consistency and control over failure handling * Tailor manual intervention choices based on stage or step specific requirements At pipeline design time, you can whitelist the allowed manual intervention actions using a new **Allowed Actions** dropdown. **Available Actions** * Retry Step * Mark as Success * Mark as Failure * Ignore Failure * Retry from Stage * Rollback Pipeline The list of available actions may vary slightly based on the **stage type** (e.g., Deploy, Approval, Custom stage, etc.). You can also use the checkbox **All Actions** to allow all available actions. After selecting the allowed actions, you **must** also configure the following: * **Timeout**: Duration to wait for manual intervention. * **Post-Timeout Action**: Action to take automatically if no manual decision is made within the timeout. ![](https://173309742-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F3F2TpHXhur2QtQnORSM9%2Fuploads%2Fgit-blob-c2e4d7e270bcecd9a44e84a82837bb951c6ef420%2Fallowed-actions-manual-intervention.png?alt=media) *** ## Prioritization and handling Failure strategies are evaluated and resolved based on several factors, including the type of failure, the scope at which the strategy is defined, and the conditions under which the failure occurs. The following sections describe how failure strategies are prioritized and handled in different scenarios. ### Failure strategies take precedence over conditional executions Harness pipeline stages and steps can include both [conditional executions](/harness-ai/use-harness-platform/pipelines/step-skip-condition-settings.md) and failure strategies. ![](https://173309742-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F3F2TpHXhur2QtQnORSM9%2Fuploads%2Fgit-blob-5a1a496ada1ff96a1cdb7c4cd18f9f27fe89e57b%2Fstep-skip-condition-settings-09.png?alt=media) When you use these settings together across multiple stages, it is important to understand how they interact. For example, consider a pipeline with two stages, `stage1` and `stage2`. Suppose `stage2` is configured to **Execute this stage only if prior pipeline or stage failed**, and `stage1` is configured with a **Failure Strategy** of **Rollback Stage** for **All Errors**. If `stage1` encounters an error, it rolls back instead of being marked as failed. As a result, `stage2` does not run because its conditional execution is not triggered. To make `stage2` run when `stage1` fails, configure `stage1` to **Ignore Failure** instead. This allows the pipeline to continue while still marking `stage1` as failed, which triggers the conditional execution of `stage2`. ### Run steps when a stage fails If you want specific steps to run when a stage fails, add those steps to the stage's **Rollback** failure strategy. By default, a rollback stops if an error occurs. If you want a specific step to run even when the rollback encounters an error, configure that step's **Conditional Execution** to **Always** and set the preceding step's failure strategy to **Mark as failure** for **All Errors**. This ensures that the required step runs even if the preceding step fails. ### Stage, step, and step group failure strategy priority When multiple failure strategies are configured, the more specific strategy takes precedence: **Step → Step Group → Stage** * A **step** failure strategy takes precedence over a **step group** failure strategy. * A **step group** failure strategy takes precedence over a **stage** failure strategy. * The **stage** failure strategy applies to steps that do not have their own failure strategy. * Failure strategies are evaluated in the order the steps execute. * If a step does not have its own failure strategy, the failure strategy configured at the step group or stage level is used, depending on where the step belongs. ### Multiple failure strategies in a stage A stage can have multiple failure strategies. ![](https://173309742-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F3F2TpHXhur2QtQnORSM9%2Fuploads%2Fgit-blob-5f66820c031707daf67b7e9333e67b8ff70f4aaf%2Fstep-failure-strategy-settings-08.png?alt=media) When using multiple failure strategies in a stage, consider the following: * For failure strategies that do not overlap (different types of failures selected), they behave as expected. * Two failures cannot occur at the same time. Whichever error occurs first, that failure strategy is used. When you use failure strategies at different levels or configure multiple strategies at the same level, it is important to understand which strategy takes precedence. ### Failure strategy conflicts Failure strategy conflicts can occur when multiple strategies apply to the same failure, either at the same level or at different levels. #### Same level If multiple failure strategies are configured at the same level, the **first applicable strategy** is used. Any remaining strategies are ignored. For example, consider these two strategies: 1. **Abort** on verification failure or authentication failure. 2. **Ignore** on verification failure or connectivity error. The following behavior occurs: * For a **verification failure**, the first strategy applies, so the stage is aborted. * For an **authentication failure**, the first strategy applies, so the stage is aborted. * For a **connectivity error**, only the second strategy applies, so the error is ignored. > **Tip:** When multiple strategies at the same level can handle the same error, the strategy listed first takes precedence. #### Different levels Failure strategies can also conflict when they are configured at different levels, such as the **step** and **stage** levels. When both a step-level and stage-level strategy apply to the same failure, the **step-level strategy takes precedence** and the stage-level strategy is ignored. In general, failure strategies are evaluated from the most specific level to the least specific level: **Step → Step Group → Stage** *** ## Next steps * [Define failure strategy for pipelines](/harness-ai/use-harness-platform/pipelines/failure-handling/define-a-failure-strategy-for-pipelines.md): Configure pipeline-level rollback for all stages. * [Conditional executions for stages and steps](/harness-ai/use-harness-platform/pipelines/step-skip-condition-settings.md): Control when stages and steps run based on conditions. * [Resume pipeline deployments](/harness-ai/use-harness-platform/pipelines/failure-handling/resume-pipeline-deployments.md): Retry failed executions from any stage. {% @harness-feedback/feedback module="harness-ai" pagePath="harness-ai/use-harness-platform/pipelines/failure-handling/define-a-failure-strategy-on-stages-and-steps" %}