Skip to main content

EC2 stop by tag

EC2 stop by tag stops an EC2 instance using the provided tag.

  • It brings back the instance after a specific duration.
  • It checks the performance of the application (or process) running on the EC2 instance.
  • When the MANAGED_NODEGROUP environment variable is enabled, the fault will not try to start the instance after chaos. Instead, it checks for the addition of a new node instance to the cluster.

EC2 Stop By Tag

Usage​

View fault usage
This fault determines the resilience of an application to unexpected halts in the EC2 instance by validating its failover capabilities.

Prerequisites​

  • Kubernetes > 1.16.
  • Ensure that you have sufficient AWS access to stop and start an EC2 instance.
  • Ensure to create a Kubernetes secret having the AWS access configuration(key) in the CHAOS_NAMESPACE. Below is the sample secret file.
apiVersion: v1
kind: Secret
metadata:
  name: cloud-secret
type: Opaque
stringData:
  cloud_config.yml: |-
    # Add the cloud AWS credentials respectively
    [default]
    aws_access_key_id = XXXXXXXXXXXXXXXXXXX
    aws_secret_access_key = XXXXXXXXXXXXXXX
  • If you change the secret key name (from cloud_config.yml), ensure that you update the AWS_SHARED_CREDENTIALS_FILE environment variable in the ChaosExperiment CR with the new name.

Warning​

If the target EC2 instance is a part of a self-managed nodegroup, ensure that you drain the target node if any application is running on it. Cordon the target node before running the fault so that the fault pods do not schedule on it.

Permissions required​

Here is an example AWS policy to execute the fault.

View policy for the fault
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ec2:StartInstances",
                "ec2:StopInstances",
                "ec2:DescribeInstanceStatus",
                "ec2:DescribeInstances"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "autoscaling:DescribeAutoScalingInstances"
            ],
            "Resource": "*"
        }
    ]
}

Refer to the superset permission/policy to execute all AWS faults.

Default validations​

The EC2 instances should be in a healthy state.

Fault tunables​

Fault tunables

Mandatory fields

Variables Description Notes
INSTANCE_TAG Instance Tag to filter the target EC2 instance. The INSTANCE_TAG should be provided as key:value ex: team:devops.
REGION The region name of the target instance.

Optional fields

Variables Description Notes
INSTANCE_AFFECTED_PERC The Percentage of total EC2 instance to target. Defaults to 0 (corresponds to 1 instance), provide numeric value only.
TOTAL_CHAOS_DURATION Duration that you specify, through which chaos is injected into the target resource (in seconds). Defaults to 30s.
CHAOS_INTERVAL The interval (in sec) between successive instance termination. Defaults to 30s.
MANAGED_NODEGROUP Set to enable if the target instance is the part of self-managed nodegroups. Defaults to disable.
SEQUENCE It defines sequence of chaos execution for multiple instance. Defaults to parallel. Supports serial sequence as well.
RAMP_TIME Period to wait before and after injecting chaos (in seconds). For example, 30s.

Fault examples​

Common and AWS-specific tunables​

Refer to the common attributes and AWS-specific tunables to tune the common tunables for all faults and aws specific tunables.

Target single instance​

It will stop a random single EC2 instance with the given INSTANCE_TAG tag and the REGION region.

You can tune it using the following example.

# target the EC2 instances with matching tag
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  chaosServiceAccount: litmus-admin
  experiments:
  - name: ec2-terminate-by-tag
    spec:
      components:
        env:
        # tag of the EC2 instance
        - name: INSTANCE_TAG
          value: 'key:value'
        # region for the EC2 instance
        - name: REGION
          value: 'us-east-1'

Target Percent of instances​

It will stop the INSTANCE_AFFECTED_PERC percentage of EC2 instances with the given INSTANCE_TAG tag and REGION region.

You can tune it using the following example.

# percentage of EC2 instances, needs to terminate with provided tags
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  chaosServiceAccount: litmus-admin
  experiments:
  - name: ec2-terminate-by-tag
    spec:
      components:
        env:
        # percentage of EC2 instance filtered by tags
        - name: INSTANCE_AFFECTED_PERC
          value: '100'
        # tag of the EC2 instance
        - name: INSTANCE_TAG
          value: 'key:value'
        # region for the EC2 instance
        - name: REGION
          value: 'us-east-1'