Lambda inject latency

Lambda inject latency is an AWS fault that simulates runtime delays in Lambda function execution to emulate network lag and resource contention. This allows you to proactively evaluate and enhance youe system's responsiveness under real-world latency conditions.

Use cases

Lambda inject latency:

• Checks integrated services handle delayed responses, ensuring that timeouts and fallback mechanisms are appropriately configured.
• Injects latency when interacting with external APIs or databases to determine if your system can maintain functionality under slower-than-expected response times.
• Evaluates the impact of delays typically experienced during cold starts or resource contention, and refine scaling strategies accordingly.

Prerequisites

• Kubernetes >= 1.17
• Access to operate AWS Lambda service.
• Lambda function must be up and running.
• Kubernetes secret must have the AWS access configuration(key) in the CHAOS_NAMESPACE. A secret file looks like this:

  apiVersion: v1
  kind: Secret
  metadata:
    name: cloud-secret
  type: Opaque
  stringData:
    cloud_config.yml: |-
      # Add the cloud AWS credentials respectively
      [default]
      aws_access_key_id = XXXXXXXXXXXXXXXXXXX
      aws_secret_access_key = XXXXXXXXXXXXXXX

tip

Harness CE recommends that you use the same secret name, that is, cloud-secret. Otherwise, you will need to update the AWS_SHARED_CREDENTIALS_FILE environment variable in the fault template with the new secret name and you won't be able to use the default health check probes.

Below is an example AWS policy to execute the fault.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "lambda:GetFunction",
        "lambda:GetFunctionConfiguration",
        "lambda:UpdateFunctionConfiguration",
        "lambda:ListVersionsByFunction",
        "lambda:ListLayers",
        "lambda:GetLayerVersion",
        "lambda:PublishLayerVersion",
        "lambda:DeleteLayerVersion"
      ],
      "Resource": "*"
    }
  ]
}

note

• Go to superset permission/policy to execute all AWS faults.
• Go to common attributes and AWS-specific tunables to tune the common tunables for all faults and AWS-specific tunables.
• Go to AWS named profile for chaos to use a different profile for AWS faults.
• Currently, this chaos fault supports Lambda functions implemented in Python; support for additional languages will be added in future releases.

Mandatory Tunables

Tunable	Description	Notes
FUNCTION_NAME	Function name of the target Lambda function. It supports single function name.	For example, test-function
LAMBDA_LATENCY	Provide the value of latency in seconds.	For example, "5" to cause a latency of 5 seconds. For more information, go to lambda-latency.
REGION	The region name of the target lambda function	For example, us-east-2

Optional Tunables

Tunable	Description	Notes
TOTAL_CHAOS_DURATION	Duration that you specify, through which chaos is injected into the target resource (in seconds).	Default: 30 s. For more information, go to duration of the chaos.
AWS_SHARED_CREDENTIALS_FILE	Path to the AWS secret credentials.	Default: /tmp/cloud_config.yml.
CHAOS_INTERVAL	The interval (in seconds) between successive instance termination.	Default: 30 s. For more information, go to chaos interval.
RAMP_TIME	Period to wait before and after injection of chaos in seconds	For example, 30 s. For more information, go to ramp time.

Lambda Latency

Updates the Lambda function to cause runtime delays in Lambda function execution. Tune it by using LAMBDA_LATENCY environment variable.

The following YAML snippet illustrates the use of this environment variable:

# contains the lambda latency value for the lambda function
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  chaosServiceAccount: litmus-admin
  experiments:
    - name: lambda-inject-latency
      spec:
        components:
          env:
            # provide the latency value for the lambda function execution
            - name: LAMBDA_LATENCY
              value: '5'
            - name: FUNCTION_NAME
              value: 'chaos-function'