Linux API latency

Linux API latency injects API request and response latency into a Linux machine by starting proxy server and redirecting the traffic through it. It induces API call latency that adds a time delay before sending a response.

Linux API Latency

Use cases

Linux API latency:

Simulate high-traffic scenarios and test the resilience and performance of an application or API, where the API may experience delays due to heavy load.
Simulate situations where an API request takes longer than expected to respond. By introducing latency, you can test how well your application handles timeouts and implements appropriate error-handling mechanisms.
Helps test how well the application handles network delays and failures, and if it recovers gracefully when network connectivity is restored.

note

This fault can be executed on Ubuntu 16 or higher, Debian 10 or higher, CentOS 7 or higher, RHEL 7 or higher, Fedora 30 or higher, and openSUSE LEAP 15.4 or higher.
The linux-chaos-infrastructure systemd service should be in an active state, and the infrastructure should be in CONNECTED state.

Fault permissions

The fault uses the root Linux user and root user group.

External packages

This fault uses stress-ng, which is installed as part of the infrastructure installation.

Optional tunables

Tunable	Description	Notes
networkInterface	Network interface used for the proxy when the `SERVICE_DIRECTION` is ingress.	Default: `eth0`. For more information, go to network interface .
duration	Duration through which chaos is injected into the target resource. Should be provided in `[numeric-hours]h[numeric-minutes]m[numeric-seconds]s` format.	Default: `30 s`. Examples: `1m25s`, `1h3m2s`, `1h3s`. For more information, go to duration of the chaos .
rampTime	Period to wait before and after injecting chaos. Should be provided in `[numeric-hours]h[numeric-minutes]m[numeric-seconds]s` format.	Default: `0s`. Examples: `1m25s`, `1h3m2s`, `1h3s`. For more information, go to ramp time.
targetServicePort	Port of the target service used when the `serviceDirection` is ingress.	Default: 80. For more information, go to target service port.
proxyPort	Port where the proxy listens for requests.	Default: 20000. For more information, go to proxy port.
pathFilter	API path or route used for the filtering.	Targets all paths if not provided. For more information, go to path filter .
serviceDirection	Direction of the flow of control, `ingress` or `egress`.	Default: `ingress`. For more information, go to service direction .
httpsEnabled	Facilitate HTTPS support for both incoming and outgoing traffic.	Default: false. For more information, go to HTTPS.
destinationPorts	Comma-separated list of the destination service or host ports for which `egress` traffic should be affected.	Default: 80,8443. For more information, go to destination ports
httpsRootCertFile	Provide the root CA certificate file name.	This setting must be configured if the root CA certificate file name differs from ca-certificates.crt. Go to [root Linux] (https://go.dev/src/crypto/x509/root_linux.go) for the default certificate file names based on various Linux distributions. For more information, go to HTTPS.
customCertificates	Provide the custom certificates for the proxy server to serve as intermediate certificates for HTTPS communication.	HTTPS communication necessitates its use as intermediate certificates by the proxy server. These certificates should be loaded into the target application. For more information, go to HTTPS.
headersFilters	Filters for HTTP request headers accept multiple comma-separated headers in the format `key1:value1,key2:value2`.	For more information, go to headers filters.
methods	The HTTP request method type accepts comma-separated HTTP methods in upper cases, such as "GET,POST".	For more information, go to methods.
queryParams	HTTP request query parameter filters accept multiple comma-separated query parameters in the format of `param1:value1,param2:value2`.	For more information, go to query params.
sourceHosts	Includes comma-separated source host names as filters, indicating the origin of the HTTP request. This is specifically relevant to the "ingress" type.	For more information, go to source hosts.
sourceIPs	This includes comma-separated source IPs as filters, indicating the origin of the HTTP request. This is specifically relevant to the "ingress" type.	For more information, go to source IPs.
destinationHosts	Comma-separated destination host names are used as filters, indicating the hosts on which you call the API. This specification applies exclusively to the "egress" type.	For more information, go to destination hosts.
destinationIPs	Comma-separated destination IPs are used as filters, indicating the hosts on which you call the API. This specification applies exclusively to the "egress" type.	For more information, go to destination hosts.
latency	Delay introduced in the application.	Default: 2s.
dataDirection	The direction in which the latency is to be injected.	Supports values `request`, `response` and `both`.

Target service port

Port of the target service. Tune it by using the targetServicePort input variable.

The following YAML snippet illustrates the use of this input variable:

## provide the port of the target service
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  annotationCheck: "false"
  appinfo:
    appns: "default"
    applabel: "app=nginx"
    appkind: "deployment"
  chaosServiceAccount: litmus-admin
  experiments:
    - name: linux-api-latency
      spec:
        components:
          env:
            # provide the port of the target service
            - name: targetServicePort
              value: 80
            - name: pathFilter
              value: '/status'

Path filter

API sub-path (or route) to filter the API calls. Tune it by using the pathFilter input variable.

The following YAML snippet illustrates the use of this input variable:

## provide api path filter
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  annotationCheck: "false"
  appinfo:
    appns: "default"
    applabel: "app=nginx"
    appkind: "deployment"
  chaosServiceAccount: litmus-admin
  experiments:
    - name: linux-api-latency
      spec:
        components:
          env:
            # provide the api path filter
            - name: pathFilter
              value: '/status'
            # provide the port of the target service
            - name: targetServicePort
              value: 80

Destination ports

Comma-separated list of the destination service or host ports for which egress traffic takes affect as a result of applying chaos on the target application. Tune it by using the destinationPorts input variable.

tip

It is applicable only when serviceDirection input variables has the value egress.

The following YAML snippet illustrates the use of this input variable:

## provide destination ports
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  annotationCheck: "false"
  appinfo:
    appns: "default"
    applabel: "app=nginx"
    appkind: "deployment"
  chaosServiceAccount: litmus-admin
  experiments:
    - name: linux-api-latency
      spec:
        components:
          env:
            # provide destination ports
            - name: destinationPorts
              value: '80,443'
            # provide the api path filter
            - name: pathFilter
              value: '/status'
            # provide the port of the target service
            - name: targetServicePort
              value: 80

HTTPS

Enable the HTTPS support for both incoming and outgoing traffic by setting the httpsEnabled field to true. Its usage varies depending on whether it is applied to ingress or egress scenario.

Ingress

Set this parameter if the HTTPS URL of the target application includes a port whose format is https://<hostname>:port. However, if the HTTPS URL is in the format https://<hostname> without a port, this setting is not required.

Egress

For outbound traffic, set httpsEnabled to true to enable HTTPS support for external services. This enables using TLS certificates for the proxy within the target application.

If the HTTP client in the target application is configured to reload certificates with each API call, set httpsEnabled to true. You won't need to provide customCertificates input variable.
However, if the root CA certificate file path is not /etc/ssl/certs/ca-certificates.crt, provide it using httpsRootCertFile input variable.
If the HTTP client in the target application isn't configured to reload certificates with each API call, provide the customCertificates input variable to the chaos experiment. There is no need to set httpsRootCertFile input variable. The same custom certificates should be loaded into the target application.

You can generate custom certificates using the following commands:

openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.crt -days 365 -nodes -subj '/CN=*'
cat key.pem cert.crt > ca-cert.pem
cat ca-cert.pem | base64 # provide it inside the customCertificates input variable

Load the cert.crt into the target application and provide the base64 encoded value of ca-cert.pem to the customCertificates input variable.

The following YAML snippet illustrates the use of this input variable:

## enable https support
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  annotationCheck: "false"
  appinfo:
    appns: "default"
    applabel: "app=nginx"
    appkind: "deployment"
  chaosServiceAccount: litmus-admin
  experiments:
    - name: linux-api-latency
      spec:
        components:
          env:
            # enable https support
            - name: httpsEnabled
              value: 'true'
            - name: customCertificates
              value: 'Y3VzdG9tIGNlcnRpZmljYXRlcwo='
            # provide the api path filter
            - name: pathFilter
              value: '/status'
            # provide the port of the targeted service
            - name: targetServicePort
              value: 80

Advanced fault tunables

proxyPort: Port where the proxy listens for requests and responses.
serviceDirection: Direction of the flow of control, either ingress or egress.
networkInterface: Network interface used for the proxy.
latency: Delay introduced in the application.
dataDirection: The direction in which the latency is to be injected. Supports values request, response and both.

The following YAML snippet illustrates the use of these input variables:

# it injects the api modify body fault
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  annotationCheck: "false"
  appinfo:
    appns: "default"
    applabel: "app=nginx"
    appkind: "deployment"
  chaosServiceAccount: litmus-admin
  experiments:
    - name: linux-api-latency
      spec:
        components:
          env:
            # provide the proxy port
            - name: proxyPort
              value: '20000'
            # provide the connection type
            - name: serviceDirection
              value: 'ingress'
            # provide the network interface
            - name: networkInterface
              value: 'eth0'
            # provide the api path filter
            - name: pathFilter
              value: '/status'
            # provide the port of the target service
            - name: targetServicePort
              value: 80
            - name: latency
              value: 2
            - name: dataDirection
              value: 'both'

Advanced filters

headersFilters: The HTTP request headers filters, that accept multiple comma-separated headers in the format of key1:value1,key2:value2.
methods: The HTTP request method type filters, that accept comma-separated HTTP methods in upper case, that is, GET,POST.
queryParams: The HTTP request query parameters filter, accepts multiple comma-separated query parameters in the format of param1:value1,param2:value2.
sourceHosts: Comma-separated source host names filters, indicating the origin of the HTTP request. This is relevant to the ingress type, specified by SERVICE_DIRECTION input variable.
sourceIPs: Comma-separated source IPs filters, indicating the origin of the HTTP request. This is specifically relevant to the ingress type, specified by serviceDirection input variable.
destinationHosts: Comma-separated destination host names filters, indicating the hosts on which you call the API. This specification applies exclusively to the egress type, specified by serviceDirection input variable.
destinationIPs: Comma-separated destination IPs filters, indicating the hosts on which you call the API. This specification applies exclusively to the egress type, specified by serviceDirection input variable.

The following YAML snippet illustrates the use of this input variable:

# it injects the api latency fault
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  annotationCheck: "false"
  appinfo:
    appns: "default"
    applabel: "app=nginx"
    appkind: "deployment"
  chaosServiceAccount: litmus-admin
  experiments:
    - name: linux-api-latency
      spec:
        components:
          env:
            # provide the headers filters
            - name: headersFilters
              value: 'key1:value1,key2:value2'
            # provide the methods filters
            - name: methods
              value: 'GET,POST'
            # provide the query params filters
            - name: queryParams
              value: 'param1:value1,param2:value2'
            # provide the source hosts filters
            - name: sourceHosts
              value: 'host1,host2'
            # provide the source ips filters
            - name: sourceIPs
              value: 'ip1,ip2'
            # provide the connection type
            - name: serviceDirection
              value: 'ingress'
            # provide the port of the target service
            - name: targetServicePort
              value: 80

Use cases​

Fault permissions​

External packages​

Optional tunables​

Target service port​

Path filter​

Destination ports​

HTTPS​

Ingress​

Egress​

Advanced fault tunables​

Advanced filters​

Use cases

Fault permissions

External packages

Optional tunables

Target service port

Path filter

Destination ports

HTTPS

Ingress

Egress

Advanced fault tunables

Advanced filters