Skip to main content

EC2 DNS chaos

EC2 DNS chaos causes DNS errors such as unavailability or malfunctioning of DNS servers on the specified EC2 instance for a specific duration.

  • It determines the performance of the application (or process) running on the EC2 instance(s).

EC2 DNS Chaos

Usage

View fault usage
This fault results in DNS errors on the target EC2 instances. This results in unavailability (or distorted) network connectivity from the VM to the target hosts. This fault determines the impact of DNS chaos on the infrastructure and standalone tasks. It simulates unavailability of DNS server (loss of access to any external domain from a given microservice) and malfunctioning of DNS server (loss of access to specific domains from a given microservice, access to cloud provider dependencies, and access to specific third party services).

Prerequisites

  • Kubernetes > 1.16
  • SSM agent is installed and running on the target EC2 instance.
  • Create a Kubernetes secret that has the AWS Access Key ID and Secret Access Key credentials in the CHAOS_NAMESPACE. Below is the sample secret file:
apiVersion: v1
kind: Secret
metadata:
name: cloud-secret
type: Opaque
stringData:
cloud_config.yml: |-
# Add the cloud AWS credentials respectively
[default]
aws_access_key_id = XXXXXXXXXXXXXXXXXXX
aws_secret_access_key = XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
  • If you change the secret name, ensure that you update the experiment.yml environment variables to derive the data from the secret. Also account for the path at which this secret is mounted as a file in the manifest ENV AWS_SHARED_CREDENTIALS_FILE.

Note

You can pass the VM credentials as secrets or as a ChaosEngine environment variable.

Permissions required

Here is an example AWS policy to execute the fault.

View policy for the fault
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ssm:GetDocument",
"ssm:DescribeDocument",
"ssm:GetParameter",
"ssm:GetParameters",
"ssm:SendCommand",
"ssm:CancelCommand",
"ssm:CreateDocument",
"ssm:DeleteDocument",
"ssm:GetCommandInvocation",
"ssm:UpdateInstanceInformation",
"ssm:DescribeInstanceInformation"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"ec2messages:AcknowledgeMessage",
"ec2messages:DeleteMessage",
"ec2messages:FailMessage",
"ec2messages:GetEndpoint",
"ec2messages:GetMessages",
"ec2messages:SendReply"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"ec2:DescribeInstanceStatus",
"ec2:DescribeInstances"
],
"Resource": [
"*"
]
}
]
}

Refer to the superset permission/policy to execute all AWS faults.

Default validations

The EC2 instance should be in a healthy state.

Fault tunables

Fault tunables

Mandatory fields

Variables Description Notes
EC2_INSTANCE_ID ID of the target EC2 instance. For example, i-044d3cb4b03b8af1f.
REGION AWS region ID where the EC2 instance has been created. For example: us-east-1.
PORT DNS port where chaos is injected. Defaults to port 54.

Optional fields

Variables Description Notes
TOTAL_CHAOS_DURATION Duration that you specify, through which chaos is injected into the target resource (in seconds). Defaults to 30s.
CHAOS_INTERVAL Time interval between two successive instance terminations (in seconds). Defaults to 30s.
SEQUENCE It defines sequence of chaos execution for multiple instance Defaults to parallel. Supports serial sequence as well.
RAMP_TIME Period to wait before and after injection of chaos in sec For example, 30s
INSTALL_DEPENDENCY Select to install dependencies used to run the network chaos. It can be either True or False If the dependency already exists, you can turn it off. Defaults to True.
TARGET_HOSTNAMES List of the target host names. If this is not provided, all the host names (or domains) will be targeted. For example, '["litmuschaos","chaosnative.com"]'.
MATCH_SCHEME Determines whether the DNS query should exactly match the targets or can be a substring. Defaults to exact.
UPSTREAM_SERVER Custom upstream server to which the intercepted DNS requests are forwarded. defaults to the server mentioned in the resolv.conf file.

Fault examples

Fault tunables

Refer to the common attributes to tune the common tunables for all the faults.

Run DNS chaos with port

It contains the DNS port to inject DNS chaos. You can tune it using the PORT environment variable.

Use the following example to tune it:

# induces dns chaos on the EC2 Instances
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
name: engine-nginx
spec:
engineState: "active"
chaosServiceAccount: litmus-admin
experiments:
- name: ec2-dns-chaos
spec:
components:
env:
# target port
- name: PORT
value: '54'
- name: EC2_INSTANCE_ID
value: 'instance-1'
- name: REGION
value: 'us-west-2'

Run DNS chaos with target host names

It contains the list of the target host names to inject DNS chaos. You can tune it using the TARGET_HOSTNAMES environment variable.

Use the following example to tune it:

# induces dns chaos on the EC2 Instances
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
name: engine-nginx
spec:
engineState: "active"
chaosServiceAccount: litmus-admin
experiments:
- name: ec2-dns-chaos
spec:
components:
env:
# list of target host names
- name: TARGET_HOSTNAMES
value: '["litmuschaos","chaosnative.com"]'
- name: EC2_INSTANCE_ID
value: 'instance-1'
- name: REGION
value: 'us-west-2'

Run DNS chaos with match scheme

It determines whether the DNS query exactly matches the target or is a substring. You can tune it using the MATCH_SCHEME environment variable.

Use the following example to tune it:

# induces dns chaos on the EC2 Instances
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
name: engine-nginx
spec:
engineState: "active"
chaosServiceAccount: litmus-admin
experiments:
- name: ec2-dns-chaos
spec:
components:
env:
# match scheme type
- name: MATCH_SCHEME
value: 'exact'
- name: EC2_INSTANCE_ID
value: 'instance-1'
- name: REGION
value: 'us-west-2'

Run DNS chaos with upstream server

It contains the custom upstream server to which intercepted DNS requests are forwarded. It defaults to the server mentioned in the resolv.conf file. You can tune it using the UPSTREAM_SERVER environment variable.

Use the following example to tune it:

# induces dns chaos on the EC2 Instances
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
name: engine-nginx
spec:
engineState: "active"
chaosServiceAccount: litmus-admin
experiments:
- name: ec2-dns-chaos
spec:
components:
env:
# name of the upstream server
- name: UPSTREAM_SERVER
value: '8.8.8.8'
- name: EC2_INSTANCE_ID
value: 'instance-1'
- name: REGION
value: 'us-west-2'