ECS task stop
ECS task stop is an AWS fault that injects chaos to stop the ECS tasks based on the services or task replica ID and checks the task availability.
- This fault results in the unavailability of the application running on the tasks.
- This experiment is applicable to both serverless ECS tasks and those backed by EC2 instances. These experiments generally involve task-level chaos or access restrictions without causing direct in-container or in-VM disruptions.
Use cases
This fault determines the resilience of an application when ECS tasks unexpectedly stop due to task being unavailable.
Prerequisites
- Kubernetes >= 1.17
- Sufficient AWS access to stop the ECS tasks.
- The target ECS tasks should be in a healthy state.
- Kubernetes secret that has the AWS access configuration (key) in the
CHAOS_NAMESPACE
. Below is a sample secret file:
apiVersion: v1
kind: Secret
metadata:
name: cloud-secret
type: Opaque
stringData:
cloud_config.yml: |-
# Add the cloud AWS credentials respectively
[default]
aws_access_key_id = XXXXXXXXXXXXXXXXXXX
aws_secret_access_key = XXXXXXXXXXXXXXX
HCE recommends that you use the same secret name, that is, cloud-secret
. Otherwise, you will need to update the AWS_SHARED_CREDENTIALS_FILE
environment variable in the fault template with the new secret name and you won't be able to use the default health check probes.
Below is an example AWS policy to help execute the fault.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"ecs:ListServices",
"ecs:ListTasks",
"ecs:StopTask",
"ecs:DescribeServices",
"ecs:DescribeTasks"
],
"Resource": "*"
}
]
}
- Refer to AWS Named Profile For Chaos to know how to use a different profile for AWS faults.
- Refer to the superset permission (or policy) to execute all AWS faults.
- Refer to the common attributes and AWS-specific tunables to tune the common tunables for all faults and aws specific tunables.
Mandatory tunables
Tunable | Description | Notes |
---|---|---|
CLUSTER_NAME | Name of the target ECS cluster. | For example, cluster-1 . |
REGION | Region name of the target ECS cluster. | For example, us-east-1 . |
SERVICE_NAME | Target ECS service name. | For example, app-svc . For more information, go to ECS service name. |
TASK_REPLICA_ID | Comma-separated target task replica IDs. | SERVICE_NAME and TASK_REPLICA_ID are mutually exclusive. If both the values are provided, SERVICE_NAME takes precedence. For more information, go to ECS task replica ID. |
Optional tunables
Tunable | Description | Notes |
---|---|---|
TOTAL_CHAOS_DURATION | Duration to insert chaos (in seconds). | Defaults: 30s. For more information, go to duration of the chaos. |
CHAOS_INTERVAL | Time interval between two successive instance terminations (in seconds). | Default: 30s. For more information, go to chaos interval. |
TASK_REPLICA_AFFECTED_PERC | Percentage of total tasks that are targeted. | Default: 100. For more information, go to ECS task replica affected percentage. |
SEQUENCE | Sequence of chaos execution for multiple instances. | Default: parallel. Supports serial and parallel. For more information, go to sequence of chaos execution. |
RAMP_TIME | Period to wait before and after injecting chaos (in seconds). | For example, 30 s. For more information, go to ramp time. |
AWS_SHARED_CREDENTIALS_FILE | Path to the AWS secret credentials. | Defaults to /tmp/cloud_config.yml . |
ECS service name
Service name whose tasks are stopped. Tune it by using the SERVICE_NAME
environment variable.
The following YAML snippet illustrates the use of this environment variable:
# stop the tasks of an ECS cluster
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
name: engine-nginx
spec:
engineState: "active"
annotationCheck: "false"
chaosServiceAccount: litmus-admin
experiments:
- name: ecs-task-stop
spec:
components:
env:
# provide the name of ECS cluster
- name: CLUSTER_NAME
value: 'demo'
- name: SERVICE_NAME
vale: 'test-svc'
- name: REGION
value: 'us-east-1'
- name: TOTAL_CHAOS_DURATION
VALUE: '60'
ECS task replica IDs
Task replicas that have a specific ID which are to be stopped. Tune it by using the TASK_REPLICA_ID
environment variable.
The following YAML snippet illustrates the use of this environment variable:
# stop the tasks of an ECS cluster
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
name: engine-nginx
spec:
engineState: "active"
annotationCheck: "false"
chaosServiceAccount: litmus-admin
experiments:
- name: ecs-task-stop
spec:
components:
env:
# provide the name of ECS cluster
- name: CLUSTER_NAME
value: 'demo'
- name: TASK_REPLICA_ID
vale: '1b751cf956e34e54b9d83b6a5c067f60,20d5041c044941dfb2126f1722d10558'
- name: REGION
value: 'us-east-1'
- name: TOTAL_CHAOS_DURATION
VALUE: '60'
ECS task replica affected percentage
Number of tasks to target (in percentage). Tune it by using the TASK_REPLICA_AFFECTED_PERC
environment variable.
The following YAML snippet illustrates the use of this environment variable:
# stop the tasks of an ECS cluster
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
name: engine-nginx
spec:
engineState: "active"
annotationCheck: "false"
chaosServiceAccount: litmus-admin
experiments:
- name: ecs-task-stop
spec:
components:
env:
# provide the name of ECS cluster
- name: CLUSTER_NAME
value: 'demo'
- name: SERVICE_NAME
vale: 'test-svc'
- name: TASK_REPLICA_AFFECTED_PERC
vale: '100'
- name: REGION
value: 'us-east-1'
- name: TOTAL_CHAOS_DURATION
VALUE: '60'