> For the complete documentation index, see [llms.txt](https://developer.harness.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://developer.harness.io/continuous-delivery/use-continuous-delivery/cd-building-blocks/cd-steps/containerized-steps/ephemeral-delegate-mode.md).

# Ephemeral delegate mode

Run containerized step group steps in a short-lived delegate container, in the same Kubernetes cluster as the delegate or in a different one

{% hint style="info" %}
This feature is in Beta[^1] and requires enablement. [View all early access features.](https://developer.harness.io/release-notes/features.md)

{% @harness-feature-flag-access/request-feature-flag-access module="continuous-delivery" pagePath="continuous-delivery/use-continuous-delivery/cd-building-blocks/cd-steps/containerized-steps/ephemeral-delegate-mode" pageTitle="Ephemeral%20delegate%20mode%20for%20containerized%20step%20groups" featureFlags="%5B%7B%22name%22%3A%22Ephemeral%20delegate%20mode%22%2C%22key%22%3A%22PIPE\_ENABLE\_EPHEMERAL\_DELEGATE\_MODE%22%2C%22status%22%3A%22BETA%22%2C%22description%22%3A%22Enables%20ephemeral%20delegate%20mode%2C%20which%20runs%20containerized%20step%20group%20steps%20in%20a%20short-lived%20delegate%20container%20added%20to%20the%20step%20group%20pod.%20The%20pod%20can%20run%20in%20the%20same%20Kubernetes%20cluster%20as%20the%20delegate%20or%20in%20a%20different%20one.%22%7D%2C%7B%22name%22%3A%22Task%20runner%22%2C%22key%22%3A%22PL\_USE\_TASK\_RUNNER%22%2C%22status%22%3A%22BETA%22%2C%22description%22%3A%22Enables%20the%20task%20runner%20that%20executes%20containerized%20step%20group%20steps.%22%7D%2C%7B%22name%22%3A%22Asynchronous%20Initialize%20step%20for%20containerized%20step%20groups%22%2C%22key%22%3A%22CDS\_INIT\_CONTAINER\_V2\_ASYNC\_STEP%22%2C%22status%22%3A%22BETA%22%2C%22description%22%3A%22Uses%20AsyncExecutable%20for%20the%20InitContainerV2%20step%20so%20it%20can%20submit%20a%20delegate%20task%20and%20return%20a%20callback%20ID%20without%20blocking%20pipeline%20progress.%22%7D%2C%7B%22name%22%3A%22Runner-based%20execution%20for%20containerized%20step%20group%20steps%22%2C%22key%22%3A%22CDS\_CONTAINER\_STEP\_GROUP\_STEPS\_USE\_RUNNER%22%2C%22status%22%3A%22BETA%22%2C%22description%22%3A%22Switches%20Continuous%20Delivery%20container%20step%20group%20steps%20to%20runner-based%20execution.%22%7D%5D" %}
{% endhint %}

Ephemeral delegate mode allows a Harness Delegate to run containerized step group steps in a short-lived delegate container that Harness adds to the step group pod. The pod can run in the same Kubernetes cluster as the delegate or in a different cluster, without installing a delegate there. Use it to run steps such as Git Clone and Run in a separate cluster, while keeping a single delegate to orchestrate them.

A standard delegate runs its work in the cluster where you install it, so running steps elsewhere means installing and maintaining another delegate there. In ephemeral delegate mode, the delegate that you select still orchestrates the step group. Harness adds a short-lived delegate container to the step group pod, which runs the steps. The ephemeral delegate exists only for the step group pod, so when the pod runs in a different cluster, there is nothing to install, upgrade, or monitor in that cluster.

***

## What you will learn from this topic

* Understand the [step group pod execution flow](#execution-flow) in ephemeral delegate mode.
* How to [enable ephemeral delegate mode](#enable-ephemeral-delegate-mode) and set the delegate image.
* How to [configure a step group](#configure-a-step-group) to use ephemeral delegate mode in the same cluster as the delegate or in another cluster.
* How to [read the Initialize logs](#verify-the-execution) and [troubleshoot](#troubleshooting) common issues.

***

## Before you begin

* **Containerized step groups**: Go to [Containerize step groups](https://developer.harness.io/continuous-delivery/use-continuous-delivery/cd-building-blocks/cd-steps/containerized-steps/containerized-step-groups.md) to learn how to configure a step group that uses Kubernetes infrastructure.
* **Delegate**: Delegate 1.x version 26.10.90103 or later, or Runner (task runner) version 3.8.0 or later. For more information, see [Delegate overview](https://developer.harness.io/docs/platform/delegates/delegate/delegate-concepts/delegate-overview).
* **Target cluster**: The Kubernetes cluster where the step group pod runs, which can be the cluster where the delegate runs or a different one. You need a namespace in that cluster and a Harness Kubernetes Cluster connector that can reach it.
* **Network access**: The target cluster must be able to reach the Harness control plane and pull the delegate image from `us-docker.pkg.dev`.
* **Feature flags**: Contact [Harness Support](mailto:support@harness.io) to enable `PIPE_ENABLE_EPHEMERAL_DELEGATE_MODE`, `PL_USE_TASK_RUNNER`, `CDS_INIT_CONTAINER_V2_ASYNC_STEP`, and `CDS_CONTAINER_STEP_GROUP_STEPS_USE_RUNNER`. Enabling the feature flag alone is not enough. You must also enable the account, organization, or project setting.
* **Pipeline permissions**: You need **View**, **Create/Edit**, and **Execute** on pipelines. For more information, see [RBAC in Harness](https://developer.harness.io/docs/platform/role-based-access-control/rbac-in-harness).

***

## Execution flow

When ephemeral delegate mode is enabled, Harness changes how the step group pod is built. The flow has the following stages:

1. The delegate you select in the step group creates the step group pod in the namespace and cluster defined by the step group's Kubernetes infrastructure.
2. Harness adds an ephemeral delegate container to that pod, along with an init container, volumes, and a secret that holds the ephemeral delegate configuration.
3. The task runner in the ephemeral delegate container runs the steps in the step group, such as Git Clone and Run.
4. When the steps finish, the task runner reports completion.

Running the step group in a different cluster is optional. The delegate that you select can run in the target cluster or in a different cluster. Steps run in ephemeral delegate mode only when the setting and the four feature flags are enabled. When they are enabled, the steps that you can run in a containerized step group or a CI stage are expected to run in this mode.

The ephemeral delegate container adds its own CPU and memory overhead to the pod. The Initialize log lists the total resources for the pod, including this overhead.

{% hint style="warning" %}
**Limitations**

Ephemeral delegate mode does not support private registries for the ephemeral delegate image. The target cluster must be able to pull the image from `us-docker.pkg.dev`.
{% endhint %}

***

## Enable ephemeral delegate mode

To enable ephemeral delegate mode, enable the setting and then set the delegate image.

### Enable the setting

You can enable the setting at the account, organization, or project scope. The setting is False by default. A project inherits the value from its organization or account unless you override it.

Perform the following steps to enable the setting:

1. Navigate to **Settings** -> **Default Settings** -> **Pipeline** at the account, organization, or project scope.
2. Find **Enable Ephemeral Delegate Mode**.
3. Select **True**.
4. Click **Save**.

![The Enable Ephemeral Delegate Mode setting in the Pipeline settings, set to True and inherited from the account](https://3694223630-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fy1JhZ4oKIppwY7d5AhPj%2Fuploads%2Fgit-blob-5215922a5bc980588e101c776045504db8643444%2Fenable-ephemeral-delegate-mode-setting.png?alt=media)

### Set the ephemeral delegate image

Delegate 1.x requires the `HARNESS_EPHEMERAL_DELEGATE_IMAGE` environment variable. There is no default value, so set it as an environment variable in the delegate Helm values or manifest YAML to the image that Harness adds to the step group pod. If you use the Runner, the variable is optional because the Runner discovers the image automatically.

The following example sets the variable:

{% code overflow="wrap" %}

```bash
HARNESS_EPHEMERAL_DELEGATE_IMAGE=us-docker.pkg.dev/gar-prod-setup/harness-public/harness/delegate:3.9.0
```

{% endcode %}

The following table describes the variable and the parts of its value:

| Parameter                                                          | Description                                                                                                                                                                                                                                                            |
| ------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `HARNESS_EPHEMERAL_DELEGATE_IMAGE`                                 | Environment variable that you set on the Harness Delegate (Delegate 1.x) that orchestrates the step group. Its value is the image that Harness uses for the ephemeral delegate container in the step group pod. Required for Delegate 1.x and optional for the Runner. |
| `us-docker.pkg.dev/gar-prod-setup/harness-public/harness/delegate` | Repository path of the Harness delegate image in the public Harness Google Artifact Registry.                                                                                                                                                                          |
| `3.9.0`                                                            | Image tag. The example uses `3.9.0`. Harness does not set a default tag.                                                                                                                                                                                               |

The cluster that runs the step group pod must be able to pull this image. In the Initialize log, the pulled image appears in the container events for the step group pod.

***

## Configure a step group

Ephemeral delegate mode uses the same step group configuration as other containerized step groups. To run the step group in the same cluster as the delegate, select the connector and namespace for that cluster. To run it in a different cluster, select the connector and namespace for the other cluster. In the step group, enable container-based execution and configure the following settings:

* **Infrastructure**: Select **Kubernetes**.
* **Kubernetes cluster**: Select the Kubernetes Cluster connector for the target cluster.
* **Namespace**: Enter the namespace in the target cluster where Harness creates the pod.
* **Delegate selector**: Select the delegate that orchestrates the step group.

The following pipeline runs a Git Clone step and a Run step in the target cluster:

```yaml
pipeline:
  name: ephemeral-delegate-demo
  identifier: ephemeraldelegatedemo
  projectIdentifier: <PROJECT_ID>
  orgIdentifier: default
  tags: {}
  stages:
    - stage:
        name: cd1
        identifier: cd1
        description: ""
        type: Deployment
        spec:
          deploymentType: Kubernetes
          service:
            serviceRef: <SERVICE_ID>
          environment:
            environmentRef: <ENVIRONMENT_ID>
            deployToAll: false
            infrastructureDefinitions:
              - identifier: <INFRASTRUCTURE_DEFINITION_ID>
          execution:
            steps:
              - stepGroup:
                  name: sg1
                  identifier: sg1
                  steps:
                    - step:
                        type: GitClone
                        name: GitClone_1
                        identifier: GitClone_1
                        spec:
                          connectorRef: <GIT_CONNECTOR_ID>
                          build:
                            type: branch
                            spec:
                              branch: <BRANCH_NAME>
                    - step:
                        type: Run
                        name: Run_1
                        identifier: Run_1
                        spec:
                          connectorRef: account.harnessImage
                          image: alpine
                          shell: Sh
                          command: echo cross cluster
                  stepGroupInfra:
                    type: KubernetesDirect
                    spec:
                      connectorRef: <TARGET_CLUSTER_CONNECTOR_ID>
                      namespace: <TARGET_NAMESPACE>
                  delegateSelectors:
                    - <DELEGATE_SELECTOR>
            rollbackSteps: []
        tags: {}
        failureStrategies:
          - onFailure:
              errors:
                - AllErrors
              action:
                type: StageRollback
```

In this example:

* `stepGroupInfra.spec.connectorRef` and `namespace` identify the target cluster and namespace where the step group pod runs.
* `delegateSelectors` selects the delegate that orchestrates the step group.
* The stage environment and infrastructure definition are independent of where the step group runs.

***

## Verify the execution

When the pipeline runs, the **Initialize** step of the step group shows how Harness builds the pod. A successful run includes log entries similar to the following:

```
Starting job to create pod <POD_NAME> on <TARGET_NAMESPACE> namespace
created ephemeral delegate secret [<POD_NAME>-ephemeral-delegate-config] in namespace [<TARGET_NAMESPACE>]
injected ephemeral delegate topology: 1 init container(s), 1 delegate container(s), 2 volume(s), 1 secret(s)
ephemeral delegate container resources (stage total + delegate overhead): requests(cpu="600m", memory="756Mi"), limits(cpu="600m", memory="756Mi")
pod [<POD_NAME>] in namespace [<TARGET_NAMESPACE>] is ready!
Task runner completed successfully
```

The `injected ephemeral delegate topology` line confirms that ephemeral delegate mode is active for the step group.

***

## Troubleshooting

The following issues can appear when you run a step group in ephemeral delegate mode.

<details>

<summary>The Initialize log shows "Namespaces do not match" and uses the default service account</summary>

The log message is `cannot use current delegate's service account name. Namespaces do not match`. It appears when the step group pod's namespace differs from the delegate's namespace. Harness uses the `default` service account in the target namespace for the pod. No specific service account is required, and the default service account works.

If the steps need additional permissions, create a service account with those permissions in the target namespace and set it in the step group Kubernetes infrastructure. For more information, see [Service account name](https://developer.harness.io/continuous-delivery/use-continuous-delivery/cd-building-blocks/cd-steps/containerized-steps/containerized-step-groups.md#service-account-name).

</details>

<details>

<summary>The Initialize log shows readiness probe failures with "connection refused"</summary>

Lines such as `Readiness probe failed ... connection refused` can appear while the containers in the pod start. If the log then shows that the pod is ready, no action is needed.

If the pod never becomes ready, check that the cluster can pull the delegate image set in `HARNESS_EPHEMERAL_DELEGATE_IMAGE` and has outbound access to `us-docker.pkg.dev`.

</details>

<details>

<summary>The step group does not use ephemeral delegate mode</summary>

Check the following:

* The four feature flags listed under [Before you begin](#before-you-begin) are enabled for your account.
* **Enable Ephemeral Delegate Mode** is set to **True** at the scope that applies to the project. The setting must be on in addition to the feature flag.
* The delegate is Delegate 1.x version 26.10.90103 or later, or Runner 3.8.0 or later.
* On Delegate 1.x, the `HARNESS_EPHEMERAL_DELEGATE_IMAGE` environment variable is set on the delegate.
* The Initialize log contains the `injected ephemeral delegate topology` line.

</details>

***

## Next steps

* [Containerize step groups](https://developer.harness.io/continuous-delivery/use-continuous-delivery/cd-building-blocks/cd-steps/containerized-steps/containerized-step-groups.md): Configure Kubernetes, VM, and Amazon ECS infrastructure for step groups.
* [Git Clone step](https://developer.harness.io/continuous-delivery/use-continuous-delivery/cd-building-blocks/cd-steps/containerized-steps/git-clone-step.md): Clone a repository in a containerized step group.
* [Run step](https://developer.harness.io/continuous-delivery/use-continuous-delivery/cd-building-blocks/cd-steps/containerized-steps/run-step.md): Run scripts in a containerized step group.

{% @harness-feedback/feedback %}

[^1]: Off by default. If you want to try it and share feedback, request access and Harness will turn it on for your account. Beta features can change and may not be ready for production use.
