EC2 HTTP latency

EC2 HTTP latency disrupts the state of infrastructure resources. This fault induces HTTP chaos on an AWS EC2 instance using the Amazon SSM Run command, carried out using SSM Docs that is in-built in the fault.

• It injects HTTP response latency to the service whose port is specified using TARGET_SERVICE_PORT environment variable by starting the proxy server and redirecting the traffic through the proxy server.
• It introduces HTTP latency chaos on the EC2 instance using an SSM doc for a certain chaos duration.

Usage

View fault usage

This fault results in delays on the target EC2 instances. This results in delayed network connectivity from the VM to the target hosts. It simulates latency to specific API services for (or from) a given microservice. It also simulates a slow response on specific third party (or dependent) components (or services).

Prerequisites

• Kubernetes >= 1.17
• SSM agent is installed and running on the target EC2 instance.
• Kubernetes secret with AWS Access Key ID and Secret Access Key credentials in the CHAOS_NAMESPACE. Below is the sample secret file:

apiVersion: v1
kind: Secret
metadata:
  name: cloud-secret
type: Opaque
stringData:
  cloud_config.yml: |-
    # Add the cloud AWS credentials respectively
    [default]
    aws_access_key_id = XXXXXXXXXXXXXXXXXXX
    aws_secret_access_key = XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

• If you change the secret key name (from cloud_config.yml), ensure that you update the AWS_SHARED_CREDENTIALS_FILE environment variable in the ChaosExperiment CR with the new name.

Note

You can pass the VM credentials as secrets or as a ChaosEngine environment variable.

Permissions required

Here is an example AWS policy to execute the fault.

View policy for the fault

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ssm:GetDocument",
                "ssm:DescribeDocument",
                "ssm:GetParameter",
                "ssm:GetParameters",
                "ssm:SendCommand",
                "ssm:CancelCommand",
                "ssm:CreateDocument",
                "ssm:DeleteDocument",
                "ssm:GetCommandInvocation",          
                "ssm:UpdateInstanceInformation",
                "ssm:DescribeInstanceInformation"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "ec2messages:AcknowledgeMessage",
                "ec2messages:DeleteMessage",
                "ec2messages:FailMessage",
                "ec2messages:GetEndpoint",
                "ec2messages:GetMessages",
                "ec2messages:SendReply"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeInstanceStatus",
                "ec2:DescribeInstances"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

Refer to the superset permission/policy to execute all AWS faults.

Default validations

The EC2 instance should be in a healthy state.

Fault tunables

Fault tunables

Mandatory fields

Variables	Description	Notes
EC2_INSTANCE_ID	ID of the target EC2 instance	For example, i-044d3cb4b03b8af1f.
REGION	The AWS region ID where the EC2 instance has been created.	For example, us-east-1.
LATENCY	Provide latency to be added to request in milliseconds.	For example, 1000
TARGET_SERVICE_PORT	Port of the service to target	Defaults to port 80.

Optional fields

Variables	Description	Notes
TOTAL_CHAOS_DURATION	Duration that you specify, through which chaos is injected into the target resource (in seconds).	Defaults to 30s.
CHAOS_INTERVAL	Time interval between two successive instance terminations (in seconds).	Defaults to 30s.
AWS_SHARED_CREDENTIALS_FILE	Provide the path for AWS secret credentials.	Defaults to /tmp/cloud_config.yml.
SEQUENCE	It defines a sequence of chaos execution for multiple instances.	Defaults to parallel. Supports serial sequence as well.
RAMP_TIME	Period to wait before and after injection of chaos (in seconds).	For example, 30s.
INSTALL_DEPENDENCY	Specify the dependencies to be installed to run the network chaos. If the dependency exists, it can be turned off.	If the dependency already exists, you can turn it off. Defaults to True.
PROXY_PORT	Port where the proxy listens to requests.	Defaults to 20000.
TOXICITY	Percentage of HTTP requests affected.	Defaults to 100
NETWORK_INTERFACE	Network interface used for the proxy.	Defaults to `eth0`.

Fault examples

Fault tunables

Refer to the common attributes to tune the common tunables for all the faults.

Target service port

It is the targeted service's port being targeted. You can tune it using the TARGET_SERVICE_PORT environment variable.

You can tune it using the following example:

## provide the port of the targeted service
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  chaosServiceAccount: litmus-admin
  experiments:
  - name: ec2-http-latency
    spec:
      components:
        env:
        # provide the port of the targeted service
        - name: TARGET_SERVICE_PORT
          value: "80"

Proxy port

It is the port where the proxy server listens to the requests. You can tune it using the PROXY_PORT environment variable.

You can use the following example to tune it:

# provide the port for proxy server
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  chaosServiceAccount: litmus-admin
  experiments:
  - name: ec2-http-latency
    spec:
      components:
        env:
        # provide the port for proxy server
        - name: PROXY_PORT
          value: '8080'
        # provide the port of the targeted service
        - name: TARGET_SERVICE_PORT
          value: "80"

Latency

It is the latency value that is added to the http request. You can tune it using the LATENCY environment variable.

You can use the following example to tune it:

## provide the latency value
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  chaosServiceAccount: litmus-admin
  experiments:
  - name: ec2-http-latency
    spec:
      components:
        env:
        # provide the latency value
        - name: LATENCY
          value: '2000'
        # provide the port of the targeted service
        - name: TARGET_SERVICE_PORT
          value: "80"

Toxicity

It defines the toxicity value to be added to the http request. You can tune it using the TOXICITY environment variable. Toxicity value defines the percentage of the total number of http requests that are affected.

You can use the following example to tune it:

## provide the toxicity
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  chaosServiceAccount: litmus-admin
  experiments:
  - name: ec2-http-latency
    spec:
      components:
        env:
        # toxicity is the probability of the request to be affected
        # provide the percentage value in the range of 0-100
        # 0 means no request will be affected and 100 means all request will be affected
        - name: TOXICITY
          value: "100"
        # provide the port of the targeted service
        - name: TARGET_SERVICE_PORT
          value: "80"

Network interface

It defines the network interface used for the proxy. You can tune it using the NETWORK_INTERFACE environment variable.

You can use the following example to tune it:

## provide the network interface for proxy
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  chaosServiceAccount: litmus-admin
  experiments:
  - name: ec2-http-latency
    spec:
      components:
        env:
        # provide the network interface for proxy
        - name: NETWORK_INTERFACE
          value: "eth0"
        # provide the port of the targeted service
        - name: TARGET_SERVICE_PORT
          value: '80'