> For the complete documentation index, see [llms.txt](https://developer.harness.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://developer.harness.io/continuous-delivery/use-continuous-delivery/cd-building-blocks/failure-strategy-service-env.md).
# Failure strategy support for Service and Environment
Apply failure strategies to Service and Environment steps in Harness CD pipelines.
[Failure strategies](https://developer.harness.io/docs/platform/pipelines/failure-handling/define-a-failure-strategy-for-pipelines) apply to [Service](/continuous-delivery/use-continuous-delivery/cd-building-blocks/services/services-overview.md) and [Environment](/continuous-delivery/use-continuous-delivery/cd-building-blocks/environments/environment-overview.md) steps, letting you define failure actions such as retry, abort, or mark as failure when a step fails. This approach helps maintain deployment stability by letting the system handle unexpected issues gracefully. By default, the failure strategy for Service and Environment steps is inherited from the stage configuration.
## What you will learn from this topic
* Which [failure strategies are supported](#supported-failure-strategies) for Service and Environment steps.
* How to apply [failure strategies for multiservice and matrix deployments](#failure-strategy-support-for-multiservice-multi-infrastructure-and-matrix-deployment).
* How to [override stage-level failure strategies](#override-failure-strategy-at-serviceenvironment-level) at the service or environment level.
***
## Supported failure strategies
The following failure strategies are supported for Service and Environment steps.
### Services
The following failure strategies are supported for Service steps:
1. Rollback Pipeline
2. Retry
3. Abort
4. Mark as Failure
### Environment and infrastructure
The following failure strategies are supported for Environment and Infrastructure steps:
1. Rollback Pipeline
2. Retry
3. Abort
4. Mark as Failure
5. Rollback Stage
***
## Examples of failure strategy for service and infrastructure
{% hint style="info" %}
**Failure strategy inheritance**
The Service and Infrastructure step does not support its own failure strategy configuration and instead inherits one. This means any failure action configured for the stage also applies to the Service step.
{% endhint %}
### Retry service step
YAML configuration demonstrating the Retry strategy for a service step
```yaml
pipeline:
projectIdentifier: testServiceInfraRetry
orgIdentifier: default
tags: {}
stages:
- stage:
name: t1
identifier: t1
description: ""
type: Deployment
spec:
deploymentType: Kubernetes
service:
serviceRef: ser1
serviceInputs:
serviceDefinition:
type: Kubernetes
spec:
artifacts:
primary:
primaryArtifactRef: <+invalidRef>
sources: <+input>
environment:
environmentRef: env1
deployToAll: false
infrastructureDefinitions:
- identifier: infra1
inputs:
identifier: infra1
type: KubernetesDirect
spec:
connectorRef: <+input>
execution:
steps:
- step:
type: ShellScript
name: ShellScript_1
identifier: ShellScript_1
spec:
shell: Bash
executionTarget: {}
source:
type: Inline
spec:
script: echo "forward"
environmentVariables: []
outputVariables: []
timeout: 10m
rollbackSteps:
- step:
type: ShellScript
name: ShellScript_1
identifier: ShellScript_1
spec:
shell: Bash
executionTarget: {}
source:
type: Inline
spec:
script: echo "rollback"
environmentVariables: []
outputVariables: []
timeout: 10m
tags: {}
failureStrategies:
- onFailure:
errors:
- AllErrors
action:
type: Retry
spec:
retryCount: 2
retryIntervals:
- 10s
onRetryFailure:
action:
type: MarkAsFailure
identifier: serviceRetryWithoutPostAction
name: serviceRetryWithMarkASFailure
```
This pipeline YAML demonstrates a service step that references an invalid artifact (`primaryArtifactRef`). As a result, the step fails. The failure strategy is configured to retry the step twice, waiting 10 seconds between retries. If the issue persists, the pipeline is marked as failed.
### Pipeline rollback on infrastructure failure
YAML configuration where an incorrect infrastructure configuration triggers a pipeline rollback
```yaml
pipeline:
identifier: infraPipelineRollback
projectIdentifier: testServiceInfraRetry
orgIdentifier: default
tags: {}
stages:
- stage:
identifier: t1
type: Deployment
name: t1
description: ""
spec:
deploymentType: Kubernetes
service:
serviceRef: ser1
serviceInputs:
serviceDefinition:
type: Kubernetes
spec:
artifacts:
primary:
primaryArtifactRef: testArtifact
sources: <+input>
environment:
environmentRef: env1
deployToAll: false
infrastructureDefinitions:
- identifier: infra1
inputs:
identifier: infra1
type: KubernetesDirect
spec:
connectorRef: account.KubernetesAccLvl
execution:
steps:
- step:
identifier: ShellScript_1
type: ShellScript
name: ShellScript_1
spec:
shell: Bash
executionTarget: {}
source:
type: Inline
spec:
script: echo "forward"
environmentVariables: []
outputVariables: []
timeout: 10m
rollbackSteps:
- step:
identifier: ShellScript_1
type: ShellScript
name: ShellScript_1
spec:
shell: Bash
executionTarget: {}
source:
type: Inline
spec:
script: echo "rollback"
environmentVariables: []
outputVariables: []
timeout: 10m
tags: {}
failureStrategies:
- onFailure:
errors:
- AllErrors
action:
type: Abort
- stage:
identifier: t2
type: Deployment
name: t2
description: ""
spec:
deploymentType: Kubernetes
service:
serviceRef: ser1
serviceInputs:
serviceDefinition:
type: Kubernetes
spec:
artifacts:
primary:
primaryArtifactRef: testArtifact
sources: <+input>
environment:
environmentRef: env1
deployToAll: false
provisioner:
steps:
- step:
identifier: dc
type: ShellScriptProvision
name: dc
spec:
source:
type: Inline
spec:
script: echo "DC"
environmentVariables: []
timeout: 10m
rollbackSteps:
- step:
identifier: ShellScriptProvision_1
type: ShellScriptProvision
name: ShellScriptProvision_1
spec:
source:
type: Inline
spec:
script: echo "pRol"
environmentVariables: []
timeout: 10m
infrastructureDefinitions:
- identifier: infra1
inputs:
identifier: infra1
type: KubernetesDirect
spec:
connectorRef: <+cds>
execution:
steps:
- step:
identifier: ShellScript_1
type: ShellScript
name: ShellScript_1
spec:
shell: Bash
executionTarget: {}
source:
type: Inline
spec:
script: exit 1
environmentVariables: []
outputVariables: []
timeout: 10m
rollbackSteps:
- step:
identifier: ShellScript_1
type: ShellScript
name: ShellScript_1
spec:
shell: Bash
executionTarget: {}
source:
type: Inline
spec:
script: echo "rollback"
environmentVariables: []
outputVariables: []
timeout: 10m
tags: {}
failureStrategies:
- onFailure:
errors:
- AllErrors
action:
type: PipelineRollback
name: infraPipelineRollback
```
This pipeline defines an infrastructure step with an incorrect `connectorRef`. The failure strategy is configured to trigger a `PipelineRollback` on failure. This means that if the infrastructure setup fails, the entire pipeline reverts to its previous state.
The infrastructure step fails because of the incorrect connector reference, causing the pipeline to roll back both the `t2` and `t1` stages.
***
## Failure strategy support for multiservice, multi-infrastructure, and matrix deployment
Multiservice deployments can be achieved using either the native multiservice configuration or the matrix configuration.
### Apply failure strategy during multi-deployment
In the example below, two services (`ser1`, `ser2`) are deployed across two infrastructures (`infra2`, `infra1`). The infrastructures are configured to deploy in parallel, and a failure strategy is defined to ensure resilient handling of any errors during deployment.
Failure strategy during multi-deployment
```yaml
pipeline:
identifier: testMultiService
name: testMultiService
projectIdentifier: testServiceInfraRetry
orgIdentifier: default
tags: {}
stages:
- stage:
identifier: ds
type: Deployment
name: ds
description: ""
spec:
deploymentType: Kubernetes
multiDeploymentConfig:
onFailure: RunAll
services:
values:
- serviceRef: ser1
serviceInputs:
serviceDefinition:
type: Kubernetes
spec:
artifacts:
primary:
primaryArtifactRef: <+invalid>
- serviceRef: ser2
serviceInputs:
serviceDefinition:
type: Kubernetes
spec:
artifacts:
primary:
primaryArtifactRef: testArtifact
metadata:
parallel: false
environments:
metadata:
parallel: true
values:
- environmentRef: env1
deployToAll: false
infrastructureDefinitions:
- identifier: infra2
inputs:
identifier: infra2
type: KubernetesDirect
spec:
connectorRef: <+WCD>
- identifier: infra1
inputs:
identifier: infra1
type: KubernetesDirect
spec:
connectorRef: account.KubernetesAccLvlConnectorForAutomationTest
execution:
steps:
- step:
identifier: ShellScript_1
type: ShellScript
name: ShellScript_1
spec:
shell: Bash
executionTarget: {}
source:
type: Inline
spec:
script: echo "CD"
environmentVariables: []
outputVariables: []
timeout: 10m
rollbackSteps: []
tags: {}
failureStrategies:
- onFailure:
errors:
- AllErrors
action:
type: Retry
spec:
retryCount: 1
retryIntervals:
- 10s
onRetryFailure:
action:
type: MarkAsFailure
```
In this deployment, the following order applies:
(`ser1` and `infra2`) and (`ser2` and `infra2`) deploy in parallel, since they are different services within the same infrastructure. (`ser1` and `infra1`) and (`ser2` and `infra1`) then deploy in parallel.
Add the following field in the YAML configuration to handle failure scenarios in multi-deployment setups:
```yaml
multiDeploymentConfig:
onFailure:
```
The `onFailure` field supports two values:
1. `RunAll`: this value ensures that if one deployment fails, the system continues executing all remaining deployments.
```yaml
multiDeploymentConfig:
onFailure: RunAll
```
In the example above, the multi-deployment failure strategy is set to `RunAll`. Suppose `ser1` has an issue and both stages that use it in series fail, and `infra2` also fails because of an invalid artifact and connector configuration. In this case, the stages for (`ser1`, `infra2`), (`ser2`, `infra2`), and (`ser1`, `infra1`) fail. The `RunAll` failure strategy ensures that even though these stages fail, (`ser2`, `infra1`) still proceeds with the deployment.
2. `SkipQueued`: when you use this value, if any deployment fails, all queued deployments that have not yet started are skipped.
```yaml
multiDeploymentConfig:
onFailure: SkipQueued
```
If a failure occurs during the deployment of `ser1` on `infra2`, all subsequent deployments, such as `ser2` on `infra1`, are skipped, halting the pipeline for queued tasks.
{% hint style="info" %}
**Deployments already in progress cannot be stopped**
In a parallel deployment, any deployment that has already started cannot be stopped, even if a failure occurs. Only queued deployments that have not begun execution are skipped.
{% endhint %}
### Apply failure strategy for multi-service with matrix looping
Adding a failure strategy using a matrix for multi-deployment
```yaml
pipeline:
identifier: testMatrixP0
name: testMatrixPO
projectIdentifier: defaultProject
orgIdentifier: default
tags: {}
stages:
- stage:
identifier: service
type: Deployment
name: service
description: ""
spec:
deploymentType: Kubernetes
service:
serviceRef: <+matrix.service>
serviceInputs:
serviceDefinition:
type: Kubernetes
spec:
artifacts:
primary:
primaryArtifactRef: artifactIden
sources:
- identifier: artifactIden
type: Gcr
spec:
tag: "49"
environment:
environmentRef: env
deployToAll: false
infrastructureDefinitions:
- identifier: <+matrix.infra>
inputs:
identifier: <+matrix.infra>
type: Kubernetes
spec:
connectorRef: <+invalid>
execution:
steps:
- step:
identifier: ShellScript_1
type: ShellScript
name: ShellScript_1
spec:
shell: Bash
executionTarget: {}
source:
type: Inline
spec:
script: echo "deploy"
environmentVariables: []
outputVariables: []
timeout: 10m
rollbackSteps:
- step:
identifier: ShellScript_1
type: ShellScript
name: ShellScript_1
spec:
shell: Bash
executionTarget: {}
source:
type: Inline
spec:
script: echo "rollback"
environmentVariables: []
outputVariables: []
timeout: 10m
failureStrategies:
- onFailure:
errors:
- AllErrors
action:
type: Retry
spec:
retryCount: 1
retryIntervals:
- 10s
onRetryFailure:
action:
type: StageRollback
strategy:
onFailure: SkipQueued
matrix:
service:
- rollingDeploy2
- rollingDeploy1
infra:
- infra1
- infra2
maxConcurrency: 2
```
The matrix strategy defines two variables, `service` and `infra`, with each service (`rollingDeploy2` and `rollingDeploy1`) deployed across both `infra1` and `infra2`.
Add the `onFailure` field inside the matrix configuration to define how the system handles failures.
The `onFailure` field supports two values:
```yaml
strategy:
onFailure:
```
1. `RunAll`: this ensures that even if one or more infrastructure-service combinations fail, the remaining ones continue to execute. For instance, if `infra1` encounters issues, the system continues deploying to `infra2`.
```yaml
strategy:
onFailure: RunAll
```
If `infra1` encounters connector issues and fails during deployment, the remaining combination still proceeds to deployment. This allows the other deployments to continue without interruption.
2. `SkipQueued`: when you apply this value, any failure causes the system to skip the remaining queued deployments that have not started yet.
```yaml
strategy:
onFailure: SkipQueued
```
In this case, if `infra1` fails during deployment because of an issue, such as an invalid connector reference, the remaining deployment is skipped, stopping execution for all pending or queued combinations.
***
## Override failure strategy at service/environment level
You can override stage-level failure strategies and define service- or environment-specific strategies to control pipeline behavior during failures.
### Supported failure strategies
The following failure strategies are supported at each level:
* **Service level**:
* **Rollback Pipeline**
* **Retry Step**
* **Abort**
* **Mark as Failure**
* **Environment and infrastructure level**:
* **Abort**
* **Rollback Pipeline**
* **Retry Step**
* **Mark as Failure**
* **Rollback Stage**
Configure these strategies in the **Advanced Configuration** section of the service/environment tabs in the pipeline. Harness reflects them in the YAML under the `failureStrategies` field. Failure strategies are directly associated with the service/environment configuration, keeping them flexible and aligned with the existing failure strategy schema.
### Behavior of failure strategies
Failure strategies at the service and environment level behave as follows:
* **Fallback to stage-level strategy**: If no failure strategy is set at the service or environment level, the pipeline falls back to the failure strategy defined at the stage level. This fallback only works if the stage-level strategy is within the subset of allowed failure strategies for the service or environment level. Failure strategies at the service/environment level are completely independent of step-level failure strategies.
* **Flexibility over propagation**: When you propagate services and environments across stages, failure strategies are not propagated. Instead, you can define a specific failure strategy for each stage, so each stage keeps a tailored configuration.
### Limitations
Keep the following limitations in mind when you configure service and environment failure strategies:
* **Excluded strategies for environment failures**: Strategies such as **Manual Intervention**, **Ignore Failure**, **Mark as Success**, and **Proceed with Default Values** are not supported, because they do not apply to environment failures.
* **Excluded strategies for service failures**: Strategies such as **Manual Intervention**, **Ignore Failure**, **Mark as Success**, and **Proceed with Default Values** are not supported, because they do not apply to service failures.
* **Stage-level failure strategy versus service/environment-level failures**: If you define a failure strategy at the stage level, it only applies if the failure occurs at a level that supports the chosen strategy. For example, suppose a failure occurs at the service or environment level, but the stage-level failure strategy is set to Mark as Success. Since the service level only supports Rollback Pipeline, Retry Step, Abort, and Mark as Failure, the Mark as Success strategy does not apply in this scenario. Failure strategies must be aligned with the supported failure-handling mechanisms at each level.
***
## Next steps
* Go to [Step failure strategy settings](/continuous-delivery/use-continuous-delivery/cd-building-blocks/executions/step-failure-strategy-settings.md) to configure failure strategies at the step level.
* Go to [Define a failure strategy for pipelines](https://developer.harness.io/docs/platform/pipelines/failure-handling/define-a-failure-strategy-for-pipelines) to configure pipeline-level failure strategies.