Graceful delegate shutdown

Harness Delegate is designed to shut down gracefully.

Shutdown without upgrade

The process of graceful delegate shutdown without upgrade is as follows:

• The delegate receives an instruction to quit.
• A grace period begins during which the delegate:
  • Stops accepting new tasks.
  • Works to complete running tasks.
• The grace period ends.
• Delegates that have not quit are force-terminated.
• Incomplete tasks are discarded.

Shutdown with upgrade

The process of graceful delegate shutdown with upgrader is as follows:

• When upgrader updates the delegate image, it starts a new delegate and waits for a heartbeat and healthy state.
• When the delegate is connected, upgrader terminates the old pod. However, the old pod will not be terminated immediately. It will first
  • Stop accepting new tasks.
  • Wait for currently executing tasks to finish before terminating. The maximum time upgrader waits for tasks to finish before force-termination is 10 minutes.

info

The wait time before force-termination is configured using terminationGracePeriodSeconds in the Kubernetes delegate YAML. When you download the YAML from Harness, it's set to 10 minutes by default.

Grace period

Delegate-Legacy End of Support (EOS) notice

This is an End of Support (EOS) notice for the Delegate-Legacy image type. This image type reached End of Support (EOS) as of January 31, 2024.

End of Support means the following:

• Harness Support will no longer accept support requests for the Delegate-Legacy image type in both Harness FirstGen and Harness NextGen (including Harness Self-Managed Enterprise Edition (SMP)).
• Security fixes will still be addressed.
• Product defects will not be addressed.

Follow the below steps to upgrade Delegate-Legacy to Delegate image

• Download new yaml from Harness by keeping the same name as the previous delegate
• Check if the existing delegate has any tags/selector, if yes then add them in DELEGATE_TAGS
• Compare the permissions given to the legacy delegate in their yaml and give the same permissions to new delegates
• Check if custom image is used, if yes then build a new image with immutable delegate as base image and override the account setting to point to that image
• Ensure that auto upgrade is enabled for Kubernetes delegates
• Our delegate yaml ships with default HPA of min and max replicas to be 1, adjust the desired number of replicas in HPA
• Deploy the new yaml and see new replicas coming under the same delegate
• Scale down the old stateful set and verify that everything is correct

The length of the grace period is configurable.

Delegate type	Grace period	Default interval
Immutable image	Yes	Configurable (details below)
Legacy image	No	30 seconds

note

The grace period is not currently configurable for Helm deployments.

Configure the default interval for a Kubernetes deployment

Open the delegate manifest file and locate the container spec (spec.containers). Change the terminationGracePeriodSeconds as shown in the following YAML. In the example below, terminationGracePeriodSeconds is set to 10 minutes.

 spec:
     terminationGracePeriodSeconds: 600
     restartPolicy: Always
     containers:
     - image: example/org:custom-delegate
       imagePullPolicy: Always
       name: delegate
       securityContext:
         allowPrivilegeEscalation: false
         runAsUser: 0

Configure the default interval for an Amazon ECS deployment

Open the delegate manifest file and locate the container containerDefinitions. Change the stopTimeout as shown in the following JSON. In the example below, stopTimeout is set to 10 minutes.

note

For more information on stopTimeout, go to Container timeouts in the Amazon ECS documentation.

  {
    "containerDefinitions": [
      {
        "portMappings": [
          {
            "hostPort": 8080,
            "protocol": "tcp",
            "containerPort": 8080
          }
        ],
        "cpu": 1,
        "environment": [
          {
            "name": "ACCOUNT_ID",
            "value": "<ACCOUNT_ID>"
          },
          {
            "name": "DELEGATE_TOKEN",
            "value": "<DELEGATE_TOKEN>"
          },
          {
            "name": "DELEGATE_TYPE",
            "value": "DOCKER"
          },
          {
            "name": "INIT_SCRIPT",
            "value": ""
          },
          {
            "name": "DEPLOY_MODE",
            "value": "KUBERNETES"
          },
          {
            "name": "MANAGER_HOST_AND_PORT",
            "value": "<MANAGER_HOST_AND_PORT>"
          },
          {
            "name": "DELEGATE_NAME",
            "value": "<DELEGATE_NAME>"
          },
          {
            "name": "LOG_STREAMING_SERVICE_URL",
            "value": "<LOG_STREAMING_SERVICE_URL>"
          },
         {
            "name": "DELEGATE_TAGS",
            "value": ""
          },

          {
            "name": "NEXT_GEN",
            "value": "true"
          }
         ],
        "memory": 2048,
        "image": "harness/delegate:22.12.77802",
        "essential": true,
        "hostname": "<DELEGATE_HOST>",
        "name": "<DELEGATE_NAME>",
        "stopTimeout": 120
      }
    ],
      "memory": "2048",
      "requiresCompatibilities": [
      "EC2"
    ],

    "cpu": "1024",
    "family": "harness-delegate-task-spec"
  }

Configure the default interval for a Docker deployment

For Docker deployments, you use the docker stop command to set the default interval. In the example below, the interval is set to 10 minutes.

docker container stop -t=600 <delegatename>

note

In the syntax above, you can choose to use --time or -t.

Graceful shutdown events

The event that initiates the graceful shutdown depends on delegate type.

Delegate environment	Trigger
Kubernetes	Pod termination, eviction, or user-initiated scaling
Docker	docker stop command
Shell	./stop.sh instruction