Skip to main content

Kubernetes rollback

Last updated on

This topic describes how Harness rolls back to the last successful version of your workload in response to failures.

Harness follows standard Kubernetes behavior during rollback. See kubectl rollout undo.

Workload rollback only​

Rollback rolls back workloads only. If there are other objects or operations executed in your stage, Harness does not roll those back.

For details on Harness Kubernetes workloads, see What Can I Deploy in Kubernetes?.

Declarative rollback​

You can use an alternative rollback behavior for Kubernetes deployments called declarative rollback.

Declarative rollback uses kubectl apply -f <prevision version of manifest> instead of kubectl rollout undo.

This method reduces the risk of getting in the wrong state in the subsequent deployment because the merge state is impacted by the rollout undo command.

To enable declarative rollback, configure the following Harness service options. These options are defined in the service because they are tied to the service's manifests.

Set the following options in the Harness service YAML:

  • skipResourceVersioning: false
  • enableDeclarativeRollback: true

Here's an example of a service with these settings configured.

service:
  name: nginx
  identifier: K8s
  tags:
    harness: "app"
    sample: "demo"
  serviceDefinition:
    spec:
      artifacts:
        primary: {}
      manifests:
        - manifest:
            identifier: dev
            type: K8sManifest
            spec:
              store:
                type: Harness
                spec:
                  files:
                    - /Sample Manifest Onboarding
              valuesPaths:
                - /values.yaml
              skipResourceVersioning: false
              enableDeclarativeRollback: true
      variables:
        - name: replica
          type: String
          description: ""
          value: "1"
        - name: cpu
          type: String
          description: "nginx cpu"
          value: "0.25"
        - name: memory
          type: String
          description: "nginx memory"
          value: 1Gb
    type: Kubernetes

Important notes:​

  • Harness stores the manifest in the release history and fetches the last successful manifest for rollback.
  • There is no more incremental versioning of the ConfigMap and Secret objects as there is with kubectl rollout undo.
  • During rollback, Harness reapplies the previous manifest. This is the declarative method, and it includes the ConfigMap and Secrets of the last known good state.
  • With standard rollback, Harness stores release history in one ConfigMap. With declarative rollback, Harness stores release history in Secrets. Each release is stored in a different Secret, with one Secret per release.
  • Harness uses a fixed limit of 2 in its release history cleanup logic, but release history cleanup is performed before the currently executing release goes through. Consequently, on average, you will have 3 secrets:
    • Current release Secret.
    • Previous successful release Secret.
    • One more previous release Secret.

Imperative rollback​

Imperative rollback uses Kubernetes' internal rollout history mechanism to revert workloads to a previous state. When a rollback is triggered, Harness issues specific commands to Kubernetes, instructing it to roll back the Deployment to a previous ReplicaSet based on its rollout history.

Harness executes the following command:

kubectl rollout undo workloadType/workloadName --to-revision=<REVISION_NUMBER>

How imperative rollback works​

  • Kubernetes undo mechanism: Imperative rollback uses Kubernetes' native undo mechanism with internal revision tracking. The rollback rewinds the Deployment to a previous ReplicaSet without redefining the entire desired state.
  • Helm 2-style rollback logic: Harness uses Helm 2-style rollback logic for imperative rollback.
  • Release history storage: Release history is stored in a single Kubernetes ConfigMap if there are no custom resources in the manifest. If custom resources are present, release history is stored in a Kubernetes Secret instead. The release history contains metadata about each deployed revision, including manifests, resource sets, pruning metadata, and rollout information.

Difference from declarative rollback​

The key difference between imperative and declarative rollback is in their approach:

  • Imperative rollback issues step-by-step commands to revert to a previous state using Kubernetes' rollout history.
  • Declarative rollback reapplies the entire manifest set from a previous successful release, causing Kubernetes to reconcile the cluster to match the declared state. Each release is stored in separate Kubernetes Secrets.

Because these underlying mechanisms differ, behaviors around pruning, resource recreation, and release history management also differ between the two approaches.

Do not switch between imperative and declarative rollback

Harness maintains separate release histories for imperative and declarative rollback types.

Harness supports switching from imperative to declarative rollback only once. This is a one-time migration. Switching from declarative to imperative rollback is NOT supported.

If you switch between rollback types multiple times, rollback behavior will be unpredictable. When you switch between rollback types, Harness cannot reliably determine which release history to use as the source of truth, leading to ambiguity in selecting the target rollback release, which can cause rollback failures or unexpected behavior.

Recommendation: Once you enable declarative rollback (enableDeclarativeRollback: true), do not switch back to imperative rollback for that service.

Rollback Support for ConfigMap and Secret Changes​

By default, Kubernetes does not restart pods when only ConfigMap or Secret objects are updated. As a result, declarative rollbacks that involve changes to these objects may not take effect as expected.

Harness now automatically handles this scenario to ensure ConfigMap and Secret changes are correctly rolled back. With this capability, Harness:

  • Generate a hash for ConfigMap and Secret objects during deployment.
  • Add the hash to the kubernetes workload/manifest.
  • Apply the previous manifest when a rollback is triggered. Now, Kubernetes will enforce the diff in the manifest caused by the hash and restart the pod when it needs, ensuring that the rolled-back configuration is applied.

This feature enhances rollback and roll-forward fidelity by ensuring pod restarts reflect all manifest changes, including non-restart-triggering objects like ConfigMaps and Secrets.

note

This feature is only applied to blue/green and rolling deployments.

Canary and blue green deployments​

For canary and blue green deployments, Harness appends ConfigMaps and Secrets present in your manifests with suffixes. This is to differentiate them from the ConfigMaps and Secrets in already running, production manifests.

During canary deployments:

  1. All ConfigMaps and Secrets are appended with the suffix -canary.
  2. The rolling deployment segment of the canary deployment continues as usual.

During blue green deployments, when the stage color is GREEN, ConfigMaps and Secrets are appended with the suffix -green. Harness already appends the managed workload with the color name.

Suffixes and character limits​

If the name + suffix reaches the 253 character limit for ConfigMaps and Secrets, Harness tries to reduce the suffix length to fit the limit.

For example, if name is 250 characters long, the suffix would be -ca for canary deployments.

Scaling and rollback​

Each time Harness deploys, the deployment gets new replicas that get scaled up and the old replicas are scaled down.

By default, Kubernetes keeps the last 10 revisions as ReplicaSets.

When Harness performs rollback it identifies which revision number to use (rollback-to). Harness identifies that last successful ReplicaSet, and selects it to be scaled up.

Rollback and artifacts​

The artifact(s) used for the replicas that are scaled up as part of rollback are simply the artifact(s) from the time that version was deployed.

Blue Green rollbacks​

In the case of Blue Green, the resources are not versioned because a Blue Green deployment uses rapid rollback. Network traffic is simply routed back to the original instances.

You do not need to redeploy previous versions of the service/artifact and the instances that comprised their environment.

Rolling Rollback step​

You can add a Rolling Rollback step to roll back the workloads deployed by the Rolling Deployment step.

Simply add this step where you want to initiate a rollback. Note that this step applies to the deployments of the Rolling Deployment command, and not the Apply Step.

Troubleshooting Common errors​

Rollback Doesn't Occur `Unable to identify any canary deployments for release **********​

Rollbacks for Canaries are dependant on the ability to save resource versions. Harness has multiple ways to detect the canary resources that were created:

  1. Based on task result, whenever task response contains the required information. In the case of some failures, this data will be missing.
  2. If task response doesn’t have any details, as a fallback, Harness tries to look in the release history and find the canary release.

There are events that result in an ability to save additional counts, which is how Harness deems a "new version" and a "rollback". For example, delegate logs may indicate:

error: Invalid request: Failed to create reservation-service/Secret/harness.test.96. Code: 403, message: Forbidden Response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"secrets \"harness.test.96\" is forbidden: exceeded quota: resource-quota, requested: count/secrets=1, used: count/secrets=25, limited: count/secrets=25","reason":"Forbidden","details":{"name":"harness.test.96","kind":"secrets"},"code":403}

In this case, Harness was unable to save the release details in cluster. Since Harness uses the new revision, and we could not save it, then the new version essentially "does not exist" and so Harness cannot roll back the resources, as it never proceeded to the new version.

See also​