Graceful delegate shutdown
Harness Delegate is designed to shut down gracefully.
Shutdown without upgrade
The process of graceful delegate shutdown without upgrade is as follows:
- The delegate receives an instruction to quit.
- A grace period begins during which the delegate:
- Stops accepting new tasks.
- Works to complete running tasks.
- The grace period ends.
- Delegates that have not quit are force-terminated.
- Incomplete tasks are discarded.
Shutdown with upgrade
The process of graceful delegate shutdown with upgrader
is as follows:
- When
upgrader
updates the delegate image, it starts a new delegate and waits for a heartbeat and healthy state. - When the delegate is connected,
upgrader
terminates the old pod. However, the old pod will not be terminated immediately. It will first- Stop accepting new tasks.
- Wait for currently executing tasks to finish before terminating. The maximum time
upgrader
waits for tasks to finish before force-termination is 10 minutes.
The wait time before force-termination is configured using terminationGracePeriodSeconds
in the Kubernetes delegate YAML. When you download the YAML from Harness, it's set to 10 minutes by default.
Grace period
This is an End of Support (EOS) notice for the Delegate-Legacy image type. This image type reached End of Support (EOS) as of January 31, 2024.
End of Support means the following:
- Harness Support will no longer accept support requests for the Delegate-Legacy image type in both Harness FirstGen and Harness NextGen (including Harness Self-Managed Enterprise Edition (SMP)).
- Security fixes will still be addressed.
- Product defects will not be addressed.
Follow the below steps to upgrade Delegate-Legacy to Delegate image
- Download new yaml from Harness by keeping the same name as the previous delegate
- Check if the existing delegate has any tags/selector, if yes then add them in DELEGATE_TAGS
- Compare the permissions given to the legacy delegate in their yaml and give the same permissions to new delegates
- Check if custom image is used, if yes then build a new image with immutable delegate as base image and override the account setting to point to that image
- Ensure that auto upgrade is enabled for Kubernetes delegates
- Our delegate yaml ships with default HPA of min and max replicas to be 1, adjust the desired number of replicas in HPA
- Deploy the new yaml and see new replicas coming under the same delegate
- Scale down the old stateful set and verify that everything is correct
The length of the grace period is configurable.
Delegate type | Grace period | Default interval |
---|---|---|
Immutable image | Yes | Configurable (details below) |
Legacy image | No | 30 seconds |
The grace period is not currently configurable for Helm deployments.
Configure the default interval for a Kubernetes deployment
Open the delegate manifest file and locate the container spec
(spec.containers
). Change the terminationGracePeriodSeconds
as shown in the following YAML. In the example below, terminationGracePeriodSeconds
is set to 10 minutes.
spec:
terminationGracePeriodSeconds: 600
restartPolicy: Always
containers:
- image: example/org:custom-delegate
imagePullPolicy: Always
name: delegate
securityContext:
allowPrivilegeEscalation: false
runAsUser: 0
Configure the default interval for an Amazon ECS deployment
Open the delegate manifest file and locate the container containerDefinitions
. Change the stopTimeout
as shown in the following JSON. In the example below, stopTimeout
is set to 10 minutes.
For more information on stopTimeout
, go to Container timeouts in the Amazon ECS documentation.
{
"containerDefinitions": [
{
"portMappings": [
{
"hostPort": 8080,
"protocol": "tcp",
"containerPort": 8080
}
],
"cpu": 1,
"environment": [
{
"name": "ACCOUNT_ID",
"value": "<ACCOUNT_ID>"
},
{
"name": "DELEGATE_TOKEN",
"value": "<DELEGATE_TOKEN>"
},
{
"name": "DELEGATE_TYPE",
"value": "DOCKER"
},
{
"name": "INIT_SCRIPT",
"value": ""
},
{
"name": "DEPLOY_MODE",
"value": "KUBERNETES"
},
{
"name": "MANAGER_HOST_AND_PORT",
"value": "<MANAGER_HOST_AND_PORT>"
},
{
"name": "DELEGATE_NAME",
"value": "<DELEGATE_NAME>"
},
{
"name": "LOG_STREAMING_SERVICE_URL",
"value": "<LOG_STREAMING_SERVICE_URL>"
},
{
"name": "DELEGATE_TAGS",
"value": ""
},
{
"name": "NEXT_GEN",
"value": "true"
}
],
"memory": 2048,
"image": "harness/delegate:22.12.77802",
"essential": true,
"hostname": "<DELEGATE_HOST>",
"name": "<DELEGATE_NAME>",
"stopTimeout": 120
}
],
"memory": "2048",
"requiresCompatibilities": [
"EC2"
],
"cpu": "1024",
"family": "harness-delegate-task-spec"
}
Configure the default interval for a Docker deployment
For Docker deployments, you use the docker stop
command to set the default interval. In the example below, the interval is set to 10 minutes.
docker container stop -t=600 <delegatename>
In the syntax above, you can choose to use --time
or -t
.
Graceful shutdown events
The event that initiates the graceful shutdown depends on delegate type.
Delegate environment | Trigger |
---|---|
Kubernetes | Pod termination, eviction, or user-initiated scaling |
Docker | docker stop command |
Shell | ./stop.sh instruction |