Skip to main content

ECS task stop

ECS task stop is an AWS fault that injects chaos to stop the ECS tasks based on the services or task replica ID and checks the task availability.

  • This fault results in the unavailability of the application running on the tasks.
  • This experiment is applicable to both serverless ECS tasks and those backed by EC2 instances. These experiments generally involve task-level chaos or access restrictions without causing direct in-container or in-VM disruptions.

ECS Task Stop

Use cases

This fault determines the resilience of an application when ECS tasks unexpectedly stop due to task being unavailable.

Prerequisites

  • Kubernetes >= 1.17
  • Sufficient AWS access to stop the ECS tasks.
  • The target ECS tasks should be in a healthy state.
  • Kubernetes secret that has the AWS access configuration (key) in the CHAOS_NAMESPACE. Below is a sample secret file:
apiVersion: v1
kind: Secret
metadata:
name: cloud-secret
type: Opaque
stringData:
cloud_config.yml: |-
# Add the cloud AWS credentials respectively
[default]
aws_access_key_id = XXXXXXXXXXXXXXXXXXX
aws_secret_access_key = XXXXXXXXXXXXXXX
tip

HCE recommends that you use the same secret name, that is, cloud-secret. Otherwise, you will need to update the AWS_SHARED_CREDENTIALS_FILE environment variable in the fault template with the new secret name and you won't be able to use the default health check probes.

Below is an example AWS policy to help execute the fault.

{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"ecs:ListServices",
"ecs:ListTasks",
"ecs:StopTask",
"ecs:DescribeServices",
"ecs:DescribeTasks"
],
"Resource": "*"
}
]
}
note

Mandatory tunables

Tunable Description Notes
CLUSTER_NAME Name of the target ECS cluster. For example, cluster-1.
REGION Region name of the target ECS cluster. For example, us-east-1.
SERVICE_NAME Target ECS service name. For example, app-svc. For more information, go to ECS service name.
TASK_REPLICA_ID Comma-separated target task replica IDs. SERVICE_NAME and TASK_REPLICA_ID are mutually exclusive. If both the values are provided, SERVICE_NAME takes precedence. For more information, go to ECS task replica ID.

Optional tunables

Tunable Description Notes
TOTAL_CHAOS_DURATION Duration to insert chaos (in seconds). Defaults: 30s. For more information, go to duration of the chaos.
CHAOS_INTERVAL Time interval between two successive instance terminations (in seconds). Default: 30s. For more information, go to chaos interval.
TASK_REPLICA_AFFECTED_PERC Percentage of total tasks that are targeted. Default: 100. For more information, go to ECS task replica affected percentage.
SEQUENCE Sequence of chaos execution for multiple instances. Default: parallel. Supports serial and parallel. For more information, go to sequence of chaos execution.
RAMP_TIME Period to wait before and after injecting chaos (in seconds). For example, 30 s. For more information, go to ramp time.
AWS_SHARED_CREDENTIALS_FILE Path to the AWS secret credentials. Defaults to /tmp/cloud_config.yml.

ECS service name

Service name whose tasks are stopped. Tune it by using the SERVICE_NAME environment variable.

The following YAML snippet illustrates the use of this environment variable:

# stop the tasks of an ECS cluster
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
name: engine-nginx
spec:
engineState: "active"
annotationCheck: "false"
chaosServiceAccount: litmus-admin
experiments:
- name: ecs-task-stop
spec:
components:
env:
# provide the name of ECS cluster
- name: CLUSTER_NAME
value: 'demo'
- name: SERVICE_NAME
vale: 'test-svc'
- name: REGION
value: 'us-east-1'
- name: TOTAL_CHAOS_DURATION
VALUE: '60'

ECS task replica IDs

Task replicas that have a specific ID which are to be stoppee. Tune it by using the TASK_REPLICA_ID environment variable.

The following YAML snippet illustrates the use of this environment variable:

# stop the tasks of an ECS cluster
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
name: engine-nginx
spec:
engineState: "active"
annotationCheck: "false"
chaosServiceAccount: litmus-admin
experiments:
- name: ecs-task-stop
spec:
components:
env:
# provide the name of ECS cluster
- name: CLUSTER_NAME
value: 'demo'
- name: TASK_REPLICA_ID
vale: '1b751cf956e34e54b9d83b6a5c067f60,20d5041c044941dfb2126f1722d10558'
- name: REGION
value: 'us-east-1'
- name: TOTAL_CHAOS_DURATION
VALUE: '60'

ECS task replica affected percentage

Number of tasks to target (in percentage). Tune it by using the TASK_REPLICA_AFFECTED_PERC environment variable.

The following YAML snippet illustrates the use of this environment variable:

# stop the tasks of an ECS cluster
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
name: engine-nginx
spec:
engineState: "active"
annotationCheck: "false"
chaosServiceAccount: litmus-admin
experiments:
- name: ecs-task-stop
spec:
components:
env:
# provide the name of ECS cluster
- name: CLUSTER_NAME
value: 'demo'
- name: SERVICE_NAME
vale: 'test-svc'
- name: TASK_REPLICA_AFFECTED_PERC
vale: '100'
- name: REGION
value: 'us-east-1'
- name: TOTAL_CHAOS_DURATION
VALUE: '60'