> For the complete documentation index, see [llms.txt](https://developer.harness.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://developer.harness.io/harness-platform/use-harness-platform/delegates/delegate/delegate-concepts/graceful-delegate-shutdown-process.md).

# Graceful delegate shutdown

Harness Delegate is designed to shut down gracefully.

### Shutdown without upgrade <a href="#shutdown-without-upgrade" id="shutdown-without-upgrade"></a>

The process of graceful delegate shutdown without upgrade is as follows:

* The delegate receives an instruction to quit.
* A grace period begins during which the delegate:
  * Stops accepting new tasks.
  * Works to complete running tasks.
* The grace period ends.
* Delegates that have not quit are force-terminated.
* Incomplete tasks are discarded.

### Shutdown with upgrade <a href="#shutdown-with-upgrade" id="shutdown-with-upgrade"></a>

The process of graceful delegate shutdown with `upgrader` is as follows:

* When `upgrader` updates the delegate image, it starts a new delegate and waits for a heartbeat and healthy state.
* When the delegate is connected, `upgrader` terminates the old pod. However, the old pod will not be terminated immediately. It will first
  * Stop accepting new tasks.
  * Wait for currently executing tasks to finish before terminating. The maximum time `upgrader` waits for tasks to finish before force-termination is 10 minutes.

{% hint style="info" %}
The wait time before force-termination is configured using `terminationGracePeriodSeconds` in the Kubernetes delegate YAML. When you download the YAML from Harness, it's set to 10 minutes by default.
{% endhint %}

### Grace period <a href="#grace-period" id="grace-period"></a>

{% hint style="warning" %}
**DELEGATE-LEGACY: END OF SUPPORT (EOS)**

***

<details>

<summary>Upgrade Delegate-Legacy to Delegate image</summary>

```
    This is an End of Support (EOS) notice for the Delegate-Legacy image type. This image type reached End of Support (EOS) as of **January 31, 2024**.

    End of Support means the following:

    - Harness Support will no longer accept support requests for the Delegate-Legacy image type in both Harness FirstGen and Harness NextGen (including Harness Self-Managed Enterprise Edition (SMP)).
    - Security fixes will still be addressed.
    - Product defects will not be addressed.

    Follow the below steps to upgrade Delegate-Legacy to Delegate image
    * Download new yaml from Harness by keeping the same name as the previous delegate
    * Check if the existing delegate has any tags/selector, if yes then add them in DELEGATE_TAGS
    * Compare the permissions given to the legacy delegate in their yaml and give the same permissions to new delegates
    * Check if custom image is used, if yes then build a new image with immutable delegate as base image and override the account setting to point to that image
    * Ensure that auto upgrade is enabled for Kubernetes delegates
    * Our delegate yaml ships with default HPA of min and max replicas to be 1, adjust the desired number of replicas in HPA
    * Deploy the new yaml and see new replicas coming under the same delegate
    * Scale down the old stateful set and verify that everything is correct
```

</details>
{% endhint %}

The length of the grace period is configurable.

| **Delegate type** | **Grace period** |     **Default interval**     |
| ----------------- | :--------------: | :--------------------------: |
| Immutable image   |        Yes       | Configurable (details below) |
| Legacy image      |        No        |          30 seconds          |

{% hint style="info" %}
**NOTE**

The grace period is not currently configurable for Helm deployments.
{% endhint %}

#### Configure the default interval for a Kubernetes deployment <a href="#configure-the-default-interval-for-a-kubernetes-deployment" id="configure-the-default-interval-for-a-kubernetes-deployment"></a>

Open the delegate manifest file and locate the container `spec` (`spec.containers`). Change the `terminationGracePeriodSeconds` as shown in the following YAML. In the example below, `terminationGracePeriodSeconds` is set to 10 minutes.

```yaml
 spec:
     terminationGracePeriodSeconds: 600
     restartPolicy: Always
     containers:
     - image: example/org:custom-delegate
       imagePullPolicy: Always
       name: delegate
       securityContext:
         allowPrivilegeEscalation: false
         runAsUser: 0
```

#### Configure the default interval for an Amazon ECS deployment <a href="#configure-the-default-interval-for-an-amazon-ecs-deployment" id="configure-the-default-interval-for-an-amazon-ecs-deployment"></a>

Open the delegate manifest file and locate the container `containerDefinitions`. Change the `stopTimeout` as shown in the following JSON. In the example below, `stopTimeout` is set to 10 minutes.

{% hint style="info" %}
**NOTE**

For more information on `stopTimeout`, go to [Container timeouts](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_definition_parameters.html#container_definition_timeout) in the Amazon ECS documentation.
{% endhint %}

```json
  {
    "containerDefinitions": [
      {
        "portMappings": [
          {
            "hostPort": 8080,
            "protocol": "tcp",
            "containerPort": 8080
          }
        ],
        "cpu": 1,
        "environment": [
          {
            "name": "ACCOUNT_ID",
            "value": "<ACCOUNT_ID>"
          },
          {
            "name": "DELEGATE_TOKEN",
            "value": "<DELEGATE_TOKEN>"
          },
          {
            "name": "DELEGATE_TYPE",
            "value": "DOCKER"
          },
          {
            "name": "INIT_SCRIPT",
            "value": ""
          },
          {
            "name": "MANAGER_HOST_AND_PORT",
            "value": "<MANAGER_HOST_AND_PORT>"
          },
          {
            "name": "DELEGATE_NAME",
            "value": "<DELEGATE_NAME>"
          },
         {
            "name": "DELEGATE_TAGS",
            "value": ""
          },

          {
            "name": "NEXT_GEN",
            "value": "true"
          }
         ],
        "memory": 2048,
        "image": "harness/delegate:22.12.77802",
        "essential": true,
        "hostname": "<DELEGATE_HOST>",
        "name": "<DELEGATE_NAME>",
        "stopTimeout": 120
      }
    ],
      "memory": "2048",
      "requiresCompatibilities": [
      "EC2"
    ],

    "cpu": "1024",
    "family": "harness-delegate-task-spec"
  }
```

#### Configure the default interval for a Docker deployment <a href="#configure-the-default-interval-for-a-docker-deployment" id="configure-the-default-interval-for-a-docker-deployment"></a>

For Docker deployments, you use the `docker stop` command to set the default interval. In the example below, the interval is set to 10 minutes.

```
docker container stop -t=600 <delegatename>
```

{% hint style="info" %}
**NOTE**

In the syntax above, you can choose to use `--time` or `-t`.
{% endhint %}

### Graceful shutdown events <a href="#graceful-shutdown-events" id="graceful-shutdown-events"></a>

The event that initiates the graceful shutdown depends on delegate type.

| **Delegate environment** |                      **Trigger**                     |
| ------------------------ | :--------------------------------------------------: |
| Kubernetes               | Pod termination, eviction, or user-initiated scaling |
| Docker                   |                 `docker stop` command                |
| Shell                    |                `./stop.sh` instruction               |