Skip to main content

ALB AZ down

ALB (Application Load Balancer) AZ (Availability Zones) down takes down the AZ on a target application load balancer for a specific duration thereby impacting the delivery. This fault restricts access to certain availability zones for a specific duration.

ALB AZ Down

Use cases​

ALB AZ down:

  • Detaches the AZ from the application load balancer thereby disrupting the application's performance.
  • Tests the application sanity, availability, and recovery workflows of the application pod attached to the load balancer.

Prerequisites​

  • Kubernetes >= 1.17
  • Appropriate AWS access to attach or detach subnet from the target ALB.
  • There needs to be a minimum of two AZs attached to the target ALB after injecting chaos; otherwise, the fault will fail to detach the given AZ.
  • The ALB should be attached to the given availability zones.
    • The Kubernetes secret should have AWS access configuration (key) in the CHAOS_NAMESPACE. Below is a sample secret file.
    apiVersion: v1
    kind: Secret
    metadata:
      name: cloud-secret
    type: Opaque
    stringData:
      cloud_config.yml: |-
        # Add the cloud AWS credentials respectively
        [default]
        aws_access_key_id = XXXXXXXXXXXXXXXXXXX
        aws_secret_access_key = XXXXXXXXXXXXXXX
tip

HCE recommends that you use the same secret name, that is, cloud-secret. Otherwise, you will need to update the AWS_SHARED_CREDENTIALS_FILE environment variable in the fault template with the new secret name and you won't be able to use the default health check probes.

Below is an example AWS policy to execute the fault.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeInstanceStatus",
                "ec2:DescribeInstances",
                "ec2:DescribeSubnets",
                "elasticloadbalancing:DetachLoadBalancerFromSubnets",
                "elasticloadbalancing:AttachLoadBalancerToSubnets",
                "elasticloadbalancing:DescribeLoadBalancers"
            ],
            "Resource": "*"
        }
    ]
}
note

Mandatory tunables​

Tunable Description Notes
LOAD_BALANCER_ARN Target load balancer ARN whose AZ should be detached. For example, arn:aws:elasticloadbalancing:us-east-2:100054111296:loadbalancer/app/test-alb/09121290906ffab7.
ZONES Target zones that should be detached from the ALB. For example, us-east-1a. For more information, go to zones.
REGION Region name for the target volumes. For example, us-east-1.

Optional tunables​

Tunable Description Notes
TOTAL_CHAOS_DURATION Duration to insert chaos (in seconds). Default: 30 s. For more information, go to duration of the chaos.
AWS_SHARED_CREDENTIALS_FILE Path to the AWS secret credentials. Default: /tmp/cloud_config.yml.
CHAOS_INTERVAL Duration between the attachment and detachment of the volumes (in seconds). Default: 30 s. For more information, go to chaos interval.
SEQUENCE Sequence of chaos execution for multiple volumes. Default: parallel. Supports serial and parallel. For more information, go to sequence of chaos execution.
RAMP_TIME Duration to wait before and after injecting chaos (in seconds). For example, 30 s. For more information, go to ramp time.

Target zones​

Comma-separated list of target zones. Tune it by using the ZONES environment variable.

The following YAML snippet illustrates the use of this environment variable:

# contains alb az down for given zones
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  chaosServiceAccount: litmus-admin
  experiments:
  - name: alb-az-down
    spec:
      components:
        env:
        # load balancer arn for chaos
        - name: LOAD_BALANCER_ARN
          value: 'arn:aws:elasticloadbalancing:us-east-2:100054111296:loadbalancer/app/test-alb/09121290906ffab7'
        # target zones for the chaos
        - name: ZONES
          value: 'us-east-1a,us-east-1b'
        # region for chaos
        - name: REGION
          value: 'us-east-1'