RDS instance reboot

RDS instance reboot can induce an RDS instance reboot chaos on AWS RDS cluster. It derives the instance under chaos from RDS cluster.

Usage

View fault usage

This fault determines the resilience of an application to RDS instance reboot.

Prerequisites

• Kubernetes >= 1.17
• AWS access to reboot RDS instances.
• Kubernetes secret that has the AWS access configuration(key) in the CHAOS_NAMESPACE. A sample secret file looks like:

apiVersion: v1
kind: Secret
metadata:
  name: cloud-secret
type: Opaque
stringData:
  cloud_config.yml: |-
    # Add the cloud AWS credentials respectively
    [default]
    aws_access_key_id = XXXXXXXXXXXXXXXXXXX
    aws_secret_access_key = XXXXXXXXXXXXXXX

• If you change the secret key name (from cloud_config.yml), update the AWS_SHARED_CREDENTIALS_FILE environment variable value in the ChaosExperiment CR with the same name.

Permissions required

Here is an example AWS policy to execute the fault.

View policy for the fault

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeInstanceStatus",
                "ec2:DescribeInstances",
                "rds:DescribeDBClusters",
                "rds:DescribeDBInstances",
                "rds:RebootDBInstance"
            ],
            "Resource": "*"
        }
    ]
}

Refer to the superset permission/policy to execute all AWS faults.

Default validations

• The RDS instance should be in a healthy state.

Fault tunables

Fault tunables

Mandatory fields

Variables	Description	Notes
CLUSTER_NAME	Name of the target RDS cluster	For example, rds-cluster-1
RDS_INSTANCE_IDENTIFIER	Name of the target RDS Instances	For example, rds-cluster-1-instance
REGION	The region name of the target ECS cluster	For example, us-east-1

Optional fields

Variables	Description	Notes
TOTAL_CHAOS_DURATION	The total time duration for chaos insertion (sec)	Defaults to 30s
INSTANCE_AFFECTED_PERC	The Percentage of total RDS instance that are part of RDS cluster to target	Defaults to 0 (corresponds to 1 instance), provide numeric value only
SEQUENCE	It defines sequence of chaos execution for multiple instance	Default value: parallel. Supported: serial, parallel
AWS_SHARED_CREDENTIALS_FILE	Provide the path for aws secret credentials	Defaults to /tmp/cloud_config.yml
RAMP_TIME	Period to wait before and after injection of chaos in sec	For example, 30

Fault examples

Common and AWS-specific tunables

Refer to the common attributes and AWS-specific tunables to tune the common tunables for all faults and aws specific tunables.

RDS cluster name

It defines the cluster name of the target RDS cluster. You can provide the RDS_CLUSTER_NAME using CLUSTER_NAME environment variable as well. If not provided, the fault selects the Instance Idenfier provided.

Use the following example to tune it:

# reboot the RDS instances
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  annotationCheck: "false"
  chaosServiceAccount: litmus-admin
  experiments:
  - name: rds-instance-reboot
    spec:
      components:
        env:
        # provide the name of RDS cluster
        - name: CLUSTER_NAME
          value: 'rds-demo-cluster'
        - name: REGION
          value: 'us-east-2'
        - name: TOTAL_CHAOS_DURATION
          value: '60'

RDS instance identifier

It defines the RDS instance name. You can provide the RDS_INSTANCE_IDENTIFIER using RDS_INSTANCE_IDENTIFIER environment variable.

Use the following example to tune it:

# reboot the RDS instances
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  annotationCheck: "false"
  chaosServiceAccount: litmus-admin
  experiments:
  - name: rds-instance-reboot
    spec:
      components:
        env:
        # provide the RDS instance identifier
        - name: RDS_INSTANCE_IDENTIFIER
          value: 'rds-demo-instance-1,rds-demo-instance-2'
        - name: INSTANCE_AFFECTED_PERC
          value: '100'
        - name: REGION
          value: 'us-east-2'
        - name: TOTAL_CHAOS_DURATION
          value: '60'