EC2 HTTP Reset Peer

Introduction

• It injects HTTP reset on the service whose port is provided as TARGET_SERVICE_PORT which stops outgoing HTTP requests by resetting the TCP connection for the requests.
• It can assess the application's resilience to a lossy/flaky http connection.

Fault execution flow chart

Prerequisites

info

• Ensure that Kubernetes Version > 1.17
• SSM agent is installed and running in the target EC2 instance.
• Kubernetes secret with AWS Access Key ID and Secret Access Key credentials in the CHAOS_NAMESPACE. A secret file looks like:

apiVersion: v1
kind: Secret
metadata:
  name: cloud-secret
type: Opaque
stringData:
  cloud_config.yml: |-
    # Add the cloud AWS credentials respectively
    [default]
    aws_access_key_id = XXXXXXXXXXXXXXXXXXX
    aws_secret_access_key = XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

• If you change the secret key name (from cloud_config.yml) please also update the AWS_SHARED_CREDENTIALS_FILE ENV value in the ChaosExperiment CR with the same name.

NOTE

You can pass the VM credentials as secrets or as an chaosengine environment variable.

Permission Requirement

• Here is an example AWS policy to execute ec2-http-reset-peer fault.

View policy for this fault

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ssm:GetDocument",
                "ssm:DescribeDocument",
                "ssm:GetParameter",
                "ssm:GetParameters",
                "ssm:SendCommand",
                "ssm:CancelCommand",
                "ssm:CreateDocument",
                "ssm:DeleteDocument",
                "ssm:GetCommandInvocation",          
                "ssm:UpdateInstanceInformation",
                "ssm:DescribeInstanceInformation"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "ec2messages:AcknowledgeMessage",
                "ec2messages:DeleteMessage",
                "ec2messages:FailMessage",
                "ec2messages:GetEndpoint",
                "ec2messages:GetMessages",
                "ec2messages:SendReply"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeInstanceStatus",
                "ec2:DescribeInstances"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

• Refer a superset permission/policy to execute all AWS faults.

Default Validations

info

• The EC2 instance should be in a healthy state.

Fault Tunables

Check the Fault Tunables

Mandatory Fields

Variables	Description	Notes
EC2_INSTANCE_ID	ID of the target EC2 instance	For example: i-044d3cb4b03b8af1f
REGION	The AWS region ID where the EC2 instance has been created	For example: us-east-1
RESET_TIMEOUT	Reset Timeout specifies after how much duration to reset the connection	Defaults to 0
TARGET_SERVICE_PORT	Port of the service to target	Defaults to port 80

Optional Fields

Variables	Description	Notes
TOTAL_CHAOS_DURATION	The total time duration for chaos insertion (in sec)	Defaults to 30s
CHAOS_INTERVAL	The interval (in sec) between successive instance termination	Defaults to 30s
AWS_SHARED_CREDENTIALS_FILE	Provide the path for aws secret credentials	Defaults to /tmp/cloud_config.yml
SEQUENCE	It defines sequence of chaos execution for multiple instance	Default value: parallel. Supported: serial, parallel
RAMP_TIME	Period to wait before and after injection of chaos (in sec)	For example: 30
INSTALL_DEPENDENCY	Whether to install the dependency to run the fault	If the dependency already exists, you can turn it off (defaults to True)
PROXY_PORT	Port where the proxy will be listening for requests	Defaults to 20000
TOXICITY	Percentage of HTTP requests to be affected	Defaults to 100
NETWORK_INTERFACE	Network interface to be used for the proxy	Defaults to `eth0`

Fault Examples

Common Fault Tunables

Refer the common attributes to tune the common tunables for all the faults.

Target Service Port

It is the targeted service's port being targeted. You can tune it using the TARGET_SERVICE_PORT environment variable.

You can use the following example to tune it:

## provide the port of the targeted service
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  chaosServiceAccount: litmus-admin
  experiments:
  - name: ec2-http-reset-peer
    spec:
      components:
        env:
        # provide the port of the targeted service
        - name: TARGET_SERVICE_PORT
          value: "80"

Proxy Port

It is the port where the proxy server listens for requests. You can tune it using the PROXY_PORT environment variable.

You can use the following example to tune it:

# provide the port for proxy server
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  chaosServiceAccount: litmus-admin
  experiments:
  - name: ec2-http-reset-peer
    spec:
      components:
        env:
        # provide the port for proxy server
        - name: PROXY_PORT
          value: '8080'
        # provide the port of the targeted service
        - name: TARGET_SERVICE_PORT
          value: "80"

Reset Timeout

It defines the reset timeout value that is added to the http request. You can tune it using RESET_TIMEOUT environment variable.

You can use the following example to tune it:

## provide the reset timeout value
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  chaosServiceAccount: litmus-admin
  experiments:
  - name: ec2-http-reset-peer
    spec:
      components:
        env:
        # reset timeout specifies after how much duration to reset the connection
        - name: RESET_TIMEOUT #in ms
          value: '2000'
        # provide the port of the targeted service
        - name: TARGET_SERVICE_PORT
          value: "80"

Toxicity

It defines the toxicity value to be added to the http request. You can tune it using the TOXICITY environment variable. Toxicity value defines the percentage of the total number of http requests that are affected.

You can use the following example to tune it:

## provide the toxicity
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  chaosServiceAccount: litmus-admin
  experiments:
  - name: ec2-http-reset-peer
    spec:
      components:
        env:
        # toxicity is the probability of the request that is affected
        # provide the percentage value in the range of 0-100
        # 0 means no request will be affected and 100 means all requests will be affected
        - name: TOXICITY
          value: "100"
        # provide the port of the targeted service
        - name: TARGET_SERVICE_PORT
          value: "80"

Network Interface

It defines the network interface used for the proxy. You can tune it using the NETWORK_INTERFACE environment variable.

You can use the following example to tune it:

## provide the network interface for proxy
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  chaosServiceAccount: litmus-admin
  experiments:
  - name: ec2-http-reset-peer
    spec:
      components:
        env:
        # provide the network interface for proxy
        - name: NETWORK_INTERFACE
          value: "eth0"
        # provide the port of the targeted service
        - name: TARGET_SERVICE_PORT
          value: '80'