Skip to main content

EC2 DNS chaos

EC2 DNS chaos causes DNS errors such as unavailability or malfunctioning of DNS servers on the specified EC2 instance for a specific duration.

EC2 DNS Chaos

Use cases​

EC2 DNS chaos:

  • Determines the performance of the application (or process) running on the EC2 instance(s).
  • Simulates the unavailability (or distorted) network connectivity from the VM to the target hosts.
  • Determines the impact of DNS chaos on the infrastructure and standalone tasks.
  • Simulates unavailability of the DNS server (loss of access to any external domain from a given microservice, access to cloud provider dependencies, and access to specific third party services).

Prerequisites​

  • Kubernetes >= 1.17
  • SSM agent is installed and running on the target EC2 instance.
  • The EC2 instance should be in a healthy state.
  • You can pass the VM credentials as secrets or as a ChaosEngine environment variable.
  • The Kubernetes secret should have the AWS Access Key ID and Secret Access Key credentials in the CHAOS_NAMESPACE. Below is the sample secret file: 
    apiVersion: v1
    kind: Secret
    metadata:
      name: cloud-secret
    type: Opaque
    stringData:
      cloud_config.yml: |-
        # Add the cloud AWS credentials respectively
        [default]
        aws_access_key_id = XXXXXXXXXXXXXXXXXXX
        aws_secret_access_key = XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
tip

HCE recommends that you use the same secret name, that is, cloud-secret. Otherwise, you will need to update the AWS_SHARED_CREDENTIALS_FILE environment variable in the fault template with the new secret name and you won't be able to use the default health check probes.

Below is an example AWS policy to execute the fault.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ssm:GetDocument",
                "ssm:DescribeDocument",
                "ssm:GetParameter",
                "ssm:GetParameters",
                "ssm:SendCommand",
                "ssm:CancelCommand",
                "ssm:CreateDocument",
                "ssm:DeleteDocument",
                "ssm:GetCommandInvocation",          
                "ssm:UpdateInstanceInformation",
                "ssm:DescribeInstanceInformation"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "ec2messages:AcknowledgeMessage",
                "ec2messages:DeleteMessage",
                "ec2messages:FailMessage",
                "ec2messages:GetEndpoint",
                "ec2messages:GetMessages",
                "ec2messages:SendReply"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeInstanceStatus",
                "ec2:DescribeInstances"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}
note

Fault tunables​

Mandatory tunables

Tunable Description Notes
EC2_INSTANCE_ID ID of the target EC2 instance. For example, i-044d3cb4b03b8af1f. For more information, go to EC2 instance ID.
REGION AWS region ID where the EC2 instance has been created. For example: us-east-1.
PORT DNS port where chaos is injected. Default: port 54. For more information, go to port.

Optional tunables

Tunable Description Notes
MACHINE Whether chaos is applied on a Windows VM or a Linux VM.
TOTAL_CHAOS_DURATION Duration that you specify, through which chaos is injected into the target resource (in seconds). Default: 30 s. For more information, go to duration of the chaos.
CHAOS_INTERVAL Time interval between two successive instance terminations (in seconds). Default: 30 s. For more information, go to chaos interval.
SEQUENCE Sequence of chaos execution for multiple instance Default: parallel. Supports serial and parallel. For more information, go to sequence of chaos execution.
RAMP_TIME Period to wait before and after injection of chaos in sec For example, 30 s. For more information, go to ramp time.
INSTALL_DEPENDENCY Select to install dependencies used to run the network chaos. It can be either True or False If the dependency already exists, you can turn it off. Defaults to True.
TARGET_HOSTNAMES List of the target host names. If this is not provided, all the host names (or domains) will be targeted. For example, '["litmuschaos","chaosnative.com"]'. For more information, go to target host names.
MATCH_SCHEME Determines whether the DNS query should exactly match the targets or can be a substring. Default: exact. For more information, go to match scheme.
UPSTREAM_SERVER Custom upstream server to which the intercepted DNS requests are forwarded. Default: Server mentioned in the resolv.conf file. For more information, go to upstream server.

Run DNS chaos with port​

DNS port to inject DNS chaos. Tune it by using the PORT environment variable.

The following YAML snippet illustrates the use of this environment variable:

# induces dns chaos on the EC2 Instances
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  chaosServiceAccount: litmus-admin
  experiments:
  - name: ec2-dns-chaos
    spec:
      components:
        env:
        # target port
        - name: PORT
          value: '54'
        - name: EC2_INSTANCE_ID
          value: 'instance-1'
        - name: REGION
          value: 'us-west-2'

Run DNS chaos with target host names​

List of the target host names to inject DNS chaos. Tune it by using the TARGET_HOSTNAMES environment variable.

The following YAML snippet illustrates the use of this environment variable:

# induces dns chaos on the EC2 Instances
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  chaosServiceAccount: litmus-admin
  experiments:
  - name: ec2-dns-chaos
    spec:
      components:
        env:
        # list of target host names
        - name: TARGET_HOSTNAMES
          value: '["litmuschaos","chaosnative.com"]'
        - name: EC2_INSTANCE_ID
          value: 'instance-1'
        - name: REGION
          value: 'us-west-2'

Run DNS chaos with match scheme​

Determine whether the DNS query exactly matches the target or is a substring. Tune it by using the MATCH_SCHEME environment variable.

The following YAML snippet illustrates the use of this environment variable:

# induces dns chaos on the EC2 Instances
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  chaosServiceAccount: litmus-admin
  experiments:
  - name: ec2-dns-chaos
    spec:
      components:
        env:
        # match scheme type
        - name: MATCH_SCHEME
          value: 'exact'
        - name: EC2_INSTANCE_ID
          value: 'instance-1'
        - name: REGION
          value: 'us-west-2'

Run DNS chaos with upstream server​

Custom upstream server where the intercepted DNS requests are forwarded. It defaults to the server mentioned in the resolv.conf file. Tune it by using the UPSTREAM_SERVER environment variable.

The following YAML snippet illustrates the use of this environment variable:

# induces dns chaos on the EC2 Instances
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  chaosServiceAccount: litmus-admin
  experiments:
  - name: ec2-dns-chaos
    spec:
      components:
        env:
        # name of the upstream server
        - name: UPSTREAM_SERVER
          value: '8.8.8.8'
        - name: EC2_INSTANCE_ID
          value: 'instance-1'
        - name: REGION
          value: 'us-west-2'