Skip to main content

EC2 HTTP Reset Peer

Introduction

  • It injects HTTP reset on the service whose port is provided as TARGET_SERVICE_PORT which stops outgoing HTTP requests by resetting the TCP connection for the requests.
  • It can assess the application's resilience to a lossy/flaky http connection.
Fault execution flow chart

EC2 HTTP Reset Peer

Prerequisites

info
  • Ensure that Kubernetes Version > 1.17
  • SSM agent is installed and running in the target EC2 instance.
  • Kubernetes secret with AWS Access Key ID and Secret Access Key credentials in the CHAOS_NAMESPACE. A secret file looks like:
apiVersion: v1
kind: Secret
metadata:
name: cloud-secret
type: Opaque
stringData:
cloud_config.yml: |-
# Add the cloud AWS credentials respectively
[default]
aws_access_key_id = XXXXXXXXXXXXXXXXXXX
aws_secret_access_key = XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
  • If you change the secret key name (from cloud_config.yml) please also update the AWS_SHARED_CREDENTIALS_FILE ENV value in the ChaosExperiment CR with the same name.

NOTE

You can pass the VM credentials as secrets or as an chaosengine environment variable.

Permission Requirement

  • Here is an example AWS policy to execute ec2-http-reset-peer fault.
View policy for this fault
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ssm:GetDocument",
"ssm:DescribeDocument",
"ssm:GetParameter",
"ssm:GetParameters",
"ssm:SendCommand",
"ssm:CancelCommand",
"ssm:CreateDocument",
"ssm:DeleteDocument",
"ssm:GetCommandInvocation",
"ssm:UpdateInstanceInformation",
"ssm:DescribeInstanceInformation"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"ec2messages:AcknowledgeMessage",
"ec2messages:DeleteMessage",
"ec2messages:FailMessage",
"ec2messages:GetEndpoint",
"ec2messages:GetMessages",
"ec2messages:SendReply"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"ec2:DescribeInstanceStatus",
"ec2:DescribeInstances"
],
"Resource": [
"*"
]
}
]
}

Default Validations

info
  • The EC2 instance should be in a healthy state.

Fault Tunables

Check the Fault Tunables

Mandatory Fields

Variables Description Notes
EC2_INSTANCE_ID ID of the target EC2 instance For example: i-044d3cb4b03b8af1f
REGION The AWS region ID where the EC2 instance has been created For example: us-east-1
RESET_TIMEOUT Reset Timeout specifies after how much duration to reset the connection Defaults to 0
TARGET_SERVICE_PORT Port of the service to target Defaults to port 80

Optional Fields

Variables Description Notes
TOTAL_CHAOS_DURATION The total time duration for chaos insertion (in sec) Defaults to 30s
CHAOS_INTERVAL The interval (in sec) between successive instance termination Defaults to 30s
AWS_SHARED_CREDENTIALS_FILE Provide the path for aws secret credentials Defaults to /tmp/cloud_config.yml
SEQUENCE It defines sequence of chaos execution for multiple instance Default value: parallel. Supported: serial, parallel
RAMP_TIME Period to wait before and after injection of chaos (in sec) For example: 30
INSTALL_DEPENDENCY Whether to install the dependency to run the fault If the dependency already exists, you can turn it off (defaults to True)
PROXY_PORT Port where the proxy will be listening for requests Defaults to 20000
TOXICITY Percentage of HTTP requests to be affected Defaults to 100
NETWORK_INTERFACE Network interface to be used for the proxy Defaults to `eth0`

Fault Examples

Common Fault Tunables

Refer the common attributes to tune the common tunables for all the faults.

Target Service Port

It is the targeted service's port being targeted. You can tune it using the TARGET_SERVICE_PORT environment variable.

You can use the following example to tune it:

## provide the port of the targeted service
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
name: engine-nginx
spec:
engineState: "active"
chaosServiceAccount: litmus-admin
experiments:
- name: ec2-http-reset-peer
spec:
components:
env:
# provide the port of the targeted service
- name: TARGET_SERVICE_PORT
value: "80"

Proxy Port

It is the port where the proxy server listens for requests. You can tune it using the PROXY_PORT environment variable.

You can use the following example to tune it:

# provide the port for proxy server
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
name: engine-nginx
spec:
engineState: "active"
chaosServiceAccount: litmus-admin
experiments:
- name: ec2-http-reset-peer
spec:
components:
env:
# provide the port for proxy server
- name: PROXY_PORT
value: '8080'
# provide the port of the targeted service
- name: TARGET_SERVICE_PORT
value: "80"

Reset Timeout

It defines the reset timeout value that is added to the http request. You can tune it using RESET_TIMEOUT environment variable.

You can use the following example to tune it:

## provide the reset timeout value
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
name: engine-nginx
spec:
engineState: "active"
chaosServiceAccount: litmus-admin
experiments:
- name: ec2-http-reset-peer
spec:
components:
env:
# reset timeout specifies after how much duration to reset the connection
- name: RESET_TIMEOUT #in ms
value: '2000'
# provide the port of the targeted service
- name: TARGET_SERVICE_PORT
value: "80"

Toxicity

It defines the toxicity value to be added to the http request. You can tune it using the TOXICITY environment variable. Toxicity value defines the percentage of the total number of http requests that are affected.

You can use the following example to tune it:

## provide the toxicity
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
name: engine-nginx
spec:
engineState: "active"
chaosServiceAccount: litmus-admin
experiments:
- name: ec2-http-reset-peer
spec:
components:
env:
# toxicity is the probability of the request that is affected
# provide the percentage value in the range of 0-100
# 0 means no request will be affected and 100 means all requests will be affected
- name: TOXICITY
value: "100"
# provide the port of the targeted service
- name: TARGET_SERVICE_PORT
value: "80"

Network Interface

It defines the network interface used for the proxy. You can tune it using the NETWORK_INTERFACE environment variable.

You can use the following example to tune it:

## provide the network interface for proxy
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
name: engine-nginx
spec:
engineState: "active"
chaosServiceAccount: litmus-admin
experiments:
- name: ec2-http-reset-peer
spec:
components:
env:
# provide the network interface for proxy
- name: NETWORK_INTERFACE
value: "eth0"
# provide the port of the targeted service
- name: TARGET_SERVICE_PORT
value: '80'