Pod API latency

Pod API latency is a Kubernetes pod-level chaos fault that injects api request and response latency by starting proxy server and redirecting the traffic through it.

A transient proxy (also known as helper pod) is brought up for a chaos duration to intercept the requests made from the target application to another service and elsewhere, and inject chaos accordingly. Target application refers to the application pods calling/sending egress traffic to an endpoint.

Pod API Latency

Use cases

Pod API latency:

Simulate high traffic scenarios and testing the resilience and performance of an application or API, where the API may experience delays due to heavy load.
Simulate situations where an API request takes longer than expected to respond. By introducing latency, you can test how well your application handles timeouts and implements appropriate error handling mechanisms.
It can be used to test, how well the application handles network delays and failures, and if it recovers gracefully when network connectivity is restored.

Permissions required

Below is a sample Kubernetes role that defines the permissions required to execute the fault.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: hce
  name: pod-api-latency
spec:
  definition:
    scope: Cluster # Supports "Namespaced" mode too
permissions:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["create", "delete", "get", "list", "patch", "deletecollection", "update"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["create", "get", "list", "patch", "update"]
  - apiGroups: [""]
    resources: ["pods/log"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["deployments, statefulsets"]
    verbs: ["get", "list"]
  - apiGroups: [""]
    resources: ["replicasets, daemonsets"]
    verbs: ["get", "list"]
  - apiGroups: [""]
    resources: ["chaosEngines", "chaosExperiments", "chaosResults"]
    verbs: ["create", "delete", "get", "list", "patch", "update"]
  - apiGroups: ["batch"]
    resources: ["jobs"]
    verbs: ["create", "delete", "get", "list", "deletecollection"]

Prerequisites

Kubernetes > 1.16
The application pods should be in the running state before and after injecting chaos.

Mandatory tunables

Tunable	Description	Notes
TARGET_CONTAINER	Name of the container subject to API latency.	None. For more information, go to target specific container
NODE_LABEL	Node label used to filter the target node if `TARGET_NODE` environment variable is not set.	It is mutually exclusive with the `TARGET_NODE` environment variable. If both are provided, the fault uses `TARGET_NODE`. For more information, go to node label.
TARGET_SERVICE_PORT	Port of the target service.	Default: port 80. For more information, go to target service port
LATENCY	Delay added to the API requests and responses.	It supports ms, s, m, h units, Default: 2s. For more information, go to latency

Optional tunables

Tunable	Description	Notes
PATH_FILTER	API path or route used for the filtering.	Targets all paths if not provided. For more information, go to path filter .
HEADERS_FILTERS	Filters for HTTP request headers accept multiple comma-separated headers in the format `key1:value1,key2:value2`.	For more information, go to header filters.
METHODS	The HTTP request method type accepts comma-separated HTTP methods in upper cases, such as "GET,POST".	For more information, go to methods.
QUERY_PARAMS	HTTP request query parameter filters accept multiple comma-separated query parameters in the format of `param1:value1,param2:value2`.	For more information, go to query params.
SOURCE_HOSTS	Includes comma-separated source host names as filters, indicating the origin of the HTTP request. This is specifically relevant to the "ingress" type.	For more information, go to source hosts.
SOURCE_IPS	This includes comma-separated source IPs as filters, indicating the origin of the HTTP request. This is specifically relevant to the "ingress" type.	For more information, go to source ips.
DESTINATION_HOSTS	Comma-separated destination host names are used as filters, indicating the hosts on which you call the API. This specification applies exclusively to the "egress" type.	For more information, go to destination hosts.
DESTINATION_IPS	Comma-separated destination IPs are used as filters, indicating the hosts on which you call the API. This specification applies exclusively to the "egress" type.	For more information, go to destination hosts.
PROXY_PORT	Port where the proxy listens for requests.	Default: 20000. For more information, go to proxy port
LIB_IMAGE	Image used to inject chaos.	Default: `harness/chaos-go-runner:main-latest`. For more information, go to image used by the helper pod.
SERVICE_DIRECTION	Direction of the flow of control, ingress or egress	Default: `ingress`. For more information, go to service direction
DATA_DIRECTION	API payload type, request or response	Default: `both`. For more information, go to data direction
DESTINATION_PORTS	comma-separated list of the destination service or host ports for which `egress` traffic should be affected	Default: 80,443. For more information, go to destination ports
HTTPS_ENABLED	facilitate HTTPS support for both incoming and outgoing traffic	Default: false. For more information, go to https
CA_CERTIFICATES	These CA certificates are used by the proxy server to generate the server certificates for the TLS handshake between the target application and the proxy server.	These CA certificates must also be added to the target application's root certificate. For more information, go to CA certificates.
SERVER_CERTIFICATES	These server certificates are used by the proxy server for the TLS handshake between the target application and the proxy server.	The corresponding CA certificates should be loaded as root certificates inside the target application. For more information, go to server certificates.
HTTPS_ROOT_CERT_PATH	Provide the root CA certificate directory path	This setting must be configured if the root CA certificate directory differs from /etc/ssl/certs. Please refer to https://go.dev/src/crypto/x509/root_linux.go to understand the default certificate directory based on various Linux distributions. For more information, go to https
HTTPS_ROOT_CERT_FILE_NAME	Provide the root CA certificate file name	This setting must be configured if the root CA certificate file name differs from ca-certificates.crt. Please refer to https://go.dev/src/crypto/x509/root_linux.go to understand the default certificate file names based on various Linux distributions. For more information, go to https
NETWORK_INTERFACE	Network interface used for the proxy.	Default: `eth0`. For more information, go to network interface
CONTAINER_RUNTIME	Container runtime interface for the cluster.	Default: containerd. Supports docker, containerd and crio. For more information, go to container runtime
SOCKET_PATH	Path to the containerd/crio/docker socket file.	Default: `/run/containerd/containerd.sock`. For more information, go to socket path
TOTAL_CHAOS_DURATION	Duration to inject chaos (in seconds).	Default: 60s. For more information, go to duration of the chaos
TARGET_PODS	Comma-separated list of application pod names subject to pod HTTP latency.	If not provided, the fault selects target pods randomly based on provided appLabels. For more information, go to target specific pods
PODS_AFFECTED_PERC	Percentage of total pods to target. Provide numeric values.	Default: 0 (corresponds to 1 replica). For more information, go to pod affected percentage
RAMP_TIME	Period to wait before and after injecting chaos (in seconds).	For example, 30 s. For more information, go to ramp time
SEQUENCE	Sequence of chaos execution for multiple target pods.	Default: parallel. Supports serial and parallel. For more information, go to sequence of chaos execution

Target service port

Port of the target service. Tune it by using the TARGET_SERVICE_PORT environment variable.

The following YAML snippet illustrates the use of this environment variable:

## provide the port of the targeted service
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  annotationCheck: "false"
  appinfo:
    appns: "default"
    applabel: "app=nginx"
    appkind: "deployment"
  chaosServiceAccount: litmus-admin
  experiments:
    - name: pod-api-latency
      spec:
        components:
          env:
            # provide the port of the targeted service
            - name: TARGET_SERVICE_PORT
              value: "80"
            - name: PATH_FILTER
              value: '/status'

Latency

Delay added to the API request and response. Tune it by using the LATENCY environment variable.

The following YAML snippet illustrates the use of this environment variable:

## provide the latency value
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  annotationCheck: "false"
  appinfo:
    appns: "default"
    applabel: "app=nginx"
    appkind: "deployment"
  chaosServiceAccount: litmus-admin
  experiments:
    - name: pod-api-latency
      spec:
        components:
          env:
            # provide the latency value
            - name: LATENCY
              value: "2s"
            # provide the port of the targeted service
            - name: TARGET_SERVICE_PORT
              value: "80"
            - name: PATH_FILTER
              value: '/status'

Path filter

API sub path (or route) to filter the API calls. Tune it by using the PATH_FILTER environment variable.

The following YAML snippet illustrates the use of this environment variable:

## provide api path filter
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  annotationCheck: "false"
  appinfo:
    appns: "default"
    applabel: "app=nginx"
    appkind: "deployment"
  chaosServiceAccount: litmus-admin
  experiments:
    - name: pod-api-latency
      spec:
        components:
          env:
            # provide the api path filter
            - name: PATH_FILTER
              value: '/status'
            # provide the port of the targeted service
            - name: TARGET_SERVICE_PORT
              value: "80"

Destination ports

A comma-separated list of the destination service or host ports for which egress traffic should be affected as a result of chaos testing on the target application. Tune it by using the DESTINATION_PORTS environment variable.

note

It is applicable only for the egress SERVICE_DIRECTION.

The following YAML snippet illustrates the use of this environment variable:

## provide destination ports
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  annotationCheck: "false"
  appinfo:
    appns: "default"
    applabel: "app=nginx"
    appkind: "deployment"
  chaosServiceAccount: litmus-admin
  experiments:
    - name: pod-api-latency
      spec:
        components:
          env:
            # provide destination ports
            - name: DESTINATION_PORTS
              value: '80,443'
            # provide the api path filter
            - name: PATH_FILTER
              value: '/status'
            # provide the port of the targeted service
            - name: TARGET_SERVICE_PORT
              value: "80"

HTTPS

To enable HTTPS support for both incoming and outgoing traffic between the target and the proxy, set the HTTPS_ENABLED field to true. The next step is to configure TLS, for which you can follow one of the ways: Using self-signed certificate or using trusted certificate.

Using Self-Signed Certificates

To establish TLS communication between the target application and the proxy server, you can use one of the following.

CA certificates

The certificates provided by CA (or self-signed) to sign or validate the server certificates.
- ca.key: The private key used to generate CA certificates.
- ca.crt: The actual CA certificate.
- Prerequisite: Load the ca.crt as the root CA certificate in the target application before experiment creation or execution.
- Fault Configuration: Provide base64-encoded ca.key and ca.crt in the CA_CERTIFICATES environment variable.
```
cat ca.key ca.crt > ca.pem
cat ca.pem | base64  # Add this to the CA_CERTIFICATES environment variable
```
In this setup, the proxy dynamically generates the server certificates with appropriate domain names during TLS handshake and sign them by the self-signed CA (provided in CA_CERTIFICATES environemtn variable) before sending them to the target application server.
Server certificates

The certificates used by the transient proxy server for TLS handshake with the target application.
- server.key: The private key used to generate the server certificates.
- server.crt: The actual Server certificate (signed by the CA).
- Internal Domain: Domain in which the local (organization-managed) services are deployed. Typically, it contains the local upstream or soft microservice dependencies.
- External Domain: Domain in which the third-party services are deployed (for example, AWS services like DynamoDB or S3). Typically, it comprises the hard dependencies.
- SAN (Subject Alternative Names): List of domain names provided to the cert-generation command or within the config file (*.cnf) passed as args to the cert-generators while creating the CSR (Certificate Signing Request).
- Prerequisites: Load the ca.crt as the root CA certificate in the target application before experiment creation or execution. Create a server.crt with the necessary domain names and sign it using CA certificates.
- Fault Configuration: Provide base64-encoded server.key and server.crt in the SERVER_CERTIFICATES environment variable.
```
cat server.key server.crt > server.pem
cat server.pem | base64  # Add this to the SERVER_CERTIFICATES environment variable
```
In this setup, the proxy sends the server certificates (provided in SERVER_CERTIFICATES environment variable) to the target application server as part of the TLS handshake.

note

Intermediate Certificates are the chain of certificates that are used to generate the eventual server certificate for security purposes.

Using Trusted Certificates

You can use trusted server certificates for internal or external domains, both of which are described below.

For Internal Domains
- Prerequisites: Generate the server certificates with the internal domain names and sign them using a trusted CA. Since the CA is trusted, there is no need to load the CA certificate in the target application.
- Fault Configuration: Provide the base64-encoded server.key, server.crt, and any intermediate certificates in the SERVER_CERTIFICATES environment variable.
```
cat server.key server.crt <intermediate-certificates> > server.pem
cat server.pem | base64  # Add this to the SERVER_CERTIFICATES env
```
For External Domains
- Prerequisites: Generate the server certificates that include both internal and external domain names, and sign them with the trusted CA. Since the CA is trusted, there is no need to load the CA certificates in the target application.
- Fault Configuration: Provide the base64-encoded server.key, server.crt, and any intermediate certificates in the SERVER_CERTIFICATES environment variable.
```
cat server.key server.crt <intermediate-certificates> > server.pem
cat server.pem | base64  # Add this to the SERVER_CERTIFICATES env
```

Advanced fault tunables

PROXY_PORT: Port where the proxy listens for requests and responses.
SERVICE_DIRECTION: Direction of the flow of control, either ingress or egress.
DATA_DIRECTION: API payload type, request, or response. It supports request, response, and both values.
NETWORK_INTERFACE: Network interface used for the proxy.

The following YAML snippet illustrates the use of this environment variable:

# it injects the api latency fault
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  annotationCheck: "false"
  appinfo:
    appns: "default"
    applabel: "app=nginx"
    appkind: "deployment"
  chaosServiceAccount: litmus-admin
  experiments:
    - name: pod-api-latency
      spec:
        components:
          env:
            # provide the proxy port
            - name: PROXY_PORT
              value: '20000'
            # provide the connection type
            - name: SERVICE_DIRECTION
              value: 'ingress'
            # provide the payload type
            - name: DATA_DIRECTION
              value: 'both'
            # provide the network interface
            - name: NETWORK_INTERFACE
              value: 'eth0'
            # provide the api path filter
            - name: PATH_FILTER
              value: '/status'
            # provide the port of the targeted service
            - name: TARGET_SERVICE_PORT
              value: "80"

Advanced filters

HEADERS_FILTERS: The HTTP request headers filters, that accept multiple comma-separated headers in the format of key1:value1,key2:value2.
METHODS: The HTTP request method type filters, that accept comma-separated HTTP methods in upper case, that is, GET,POST.
QUERY_PARAMS: The HTTP request query parameters filter, accepts multiple comma-separated query parameters in the format of param1:value1,param2:value2.
SOURCE_HOSTS: Comma-separated source host names filters, indicating the origin of the HTTP request. This is relevant to the ingress type, specified by SERVICE_DIRECTION environment variable.
SOURCE_IPS: Comma-separated source IPs filters, indicating the origin of the HTTP request. This is specifically relevant to the ingress type, specified by SERVICE_DIRECTION environment variable.
DESTINATION_HOSTS: Comma-separated destination host names filters, indicating the hosts on which you call the API. This specification applies exclusively to the egress type, specified by SERVICE_DIRECTION environment variable.
DESTINATION_IPS: Comma-separated destination IPs filters, indicating the hosts on which you call the API. This specification applies exclusively to the egress type, specified by SERVICE_DIRECTION environment variable.

The following YAML snippet illustrates the use of this environment variable:

# it injects the api latency fault
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  annotationCheck: "false"
  appinfo:
    appns: "default"
    applabel: "app=nginx"
    appkind: "deployment"
  chaosServiceAccount: litmus-admin
  experiments:
    - name: pod-api-latency
      spec:
        components:
          env:
            # provide the headers filters
            - name: HEADERS_FILTERS
              value: 'key1:value1,key2:value2'
            # provide the methods filters
            - name: METHODS
              value: 'GET,POST'
            # provide the query params filters
            - name: QUERY_PARAMS
              value: 'param1:value1,param2:value2'
            # provide the source hosts filters
            - name: SOURCE_HOSTS
              value: 'host1,host2'
            # provide the source ips filters
            - name: SOURCE_IPS
              value: 'ip1,ip2'
            # provide the connection type
            - name: SERVICE_DIRECTION
              value: 'ingress'
            # provide the port of the targeted service
            - name: TARGET_SERVICE_PORT
              value: "80"

Container runtime and socket path

The CONTAINER_RUNTIME and SOCKET_PATH environment variables to set the container runtime and socket file path, respectively.

CONTAINER_RUNTIME: It supports docker, containerd, and crio runtimes. The default value is containerd.
SOCKET_PATH: It contains path of containerd socket file by default(/run/containerd/containerd.sock). For docker, specify path as /var/run/docker.sock. For crio, specify path as /var/run/crio/crio.sock.

The following YAML snippet illustrates the use of these environment variables:

## provide the container runtime and socket file path
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  annotationCheck: "false"
  appinfo:
    appns: "default"
    applabel: "app=nginx"
    appkind: "deployment"
  chaosServiceAccount: litmus-admin
  experiments:
    - name: pod-api-latency
      spec:
        components:
          env:
            # runtime for the container
            # supports docker, containerd, crio
            - name: CONTAINER_RUNTIME
              value: "containerd"
            # path of the socket file
            - name: SOCKET_PATH
              value: "/run/containerd/containerd.sock"
            # provide the port of the targeted service
            - name: TARGET_SERVICE_PORT
              value: "80"
            - name: PATH_FILTER
              value: '/status'

Use cases​

Permissions required​

Prerequisites​

Mandatory tunables​

Optional tunables​

Target service port​

Latency​

Path filter​

Destination ports​

HTTPS​

Using Self-Signed Certificates​

Using Trusted Certificates​

Advanced fault tunables​

Advanced filters​

Container runtime and socket path​

Use cases

Permissions required

Prerequisites

Mandatory tunables

Optional tunables

Target service port

Latency

Path filter

Destination ports

HTTPS

Using Self-Signed Certificates

Using Trusted Certificates

Advanced fault tunables

Advanced filters

Container runtime and socket path