Pod API block

Pod API block is a Kubernetes pod-level chaos fault that blocks the API requests through path filtering. This is achieved by starting the proxy server and redirecting the traffic through the proxy server.

Pod API Block

Use cases

Pod API block:

Validates how well your system can handle disruptions in API services for a specific pod.
Ensures that your load balancer is effectively distributing traffic to healthy pods in the cluster.
Checks if your system's failover mechanisms work as expected when one of the pods becomes unresponsive.
Evaluates if your system can gracefully degrade performance when a specific component (in this case, a pod) is experiencing issues.

Permissions required

Below is a sample Kubernetes role that defines the permissions required to execute the fault.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: hce
  name: pod-api-block
spec:
  definition:
    scope: Cluster # Supports "Namespaced" mode too
permissions:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["create", "delete", "get", "list", "patch", "deletecollection", "update"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["create", "get", "list", "patch", "update"]
  - apiGroups: [""]
    resources: ["pods/log"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["deployments, statefulsets"]
    verbs: ["get", "list"]
  - apiGroups: [""]
    resources: ["replicasets, daemonsets"]
    verbs: ["get", "list"]
  - apiGroups: [""]
    resources: ["chaosEngines", "chaosExperiments", "chaosResults"]
    verbs: ["create", "delete", "get", "list", "patch", "update"]
  - apiGroups: ["batch"]
    resources: ["jobs"]
    verbs: ["create", "delete", "get", "list", "deletecollection"]

Prerequisites

Kubernetes > 1.17
The application pods should be running before and after injecting chaos.

Mandatory tunables

Tunable	Description	Notes
TARGET_CONTAINER	Name of the container subject to API block.	None. For more information, go to target specific container
TARGET_SERVICE_PORT	Port of the target service.	Default: port 80. For more information, go to target service port.

Optional tunables

Tunable	Description	Notes
PATH_FILTER	API path or route used for the filtering.	Targets all paths if not provided. For more information, go to path filter .
HEADERS_FILTERS	Filters for HTTP request headers accept multiple comma-separated headers in the format `key1:value1,key2:value2`.	For more information, go to header filters.
METHODS	The HTTP request method type accepts comma-separated HTTP methods in upper cases, such as "GET,POST".	For more information, go to methods.
QUERY_PARAMS	HTTP request query parameter filters accept multiple comma-separated query parameters in the format of `param1:value1,param2:value2`.	For more information, go to query params.
SOURCE_HOSTS	Includes comma-separated source host names as filters, indicating the origin of the HTTP request. This is specifically relevant to the "ingress" type.	For more information, go to source hosts.
SOURCE_IPS	This includes comma-separated source IPs as filters, indicating the origin of the HTTP request. This is specifically relevant to the "ingress" type.	For more information, go to source IPs.
DESTINATION_HOSTS	Comma-separated destination host names are used as filters, indicating the hosts on which you call the API. This specification applies exclusively to the "egress" type.	For more information, go to destination hosts.
DESTINATION_IPS	Comma-separated destination IPs are used as filters, indicating the hosts on which you call the API. This specification applies exclusively to the "egress" type.	For more information, go to destination hosts.
PROXY_PORT	Port where the proxy listens for requests.	Default: 20000. For more information, go to proxy port.
LIB_IMAGE	Image used to inject chaos.	Default: `harness/chaos-go-runner:main-latest`. For more information, go to image used by the helper pod.
SERVICE_DIRECTION	Direction of the flow of control, `ingress` or `egress`.	Default: `ingress`. For more information, go to service direction .
DESTINATION_PORTS	Comma-separated list of the destination service or host ports for which `egress` traffic should be affected.	Default: 80,443. For more information, go to destination ports.
HTTPS_ENABLED	Facilitate HTTPS support for both incoming and outgoing traffic.	Default: false. For more information, go to HTTPS.
CA_CERTIFICATES	Provide the CA certificates for the proxy server to serve as intermediate certificates for HTTPS communication.	HTTPS communication necessitates its use as intermediate certificates by the proxy server. These certificates should be loaded into the target application. For more information, go to HTTPS
SERVER_CERTIFICATES	Provide the Server certificates for the proxy server to serve as intermediate certificates for HTTPS communication.	HTTPS communication necessitates its use as intermediate certificates by the proxy server. The corresponding CA certificates should be loaded as root certificates inside the target application. For more information, go to HTTPS
HTTPS_ROOT_CERT_PATH	Provide the root CA certificate directory path.	This setting must be configured if the root CA certificate directory differs from /etc/ssl/certs. Go to root Linux for the default certificate directory based on various Linux distributions. For more information, go to HTTPS.
HTTPS_ROOT_CERT_FILE_NAME	Provide the root CA certificate file name.	This setting must be configured if the root CA certificate file name differs from ca-certificates.crt. Go to [root Linux] (https://go.dev/src/crypto/x509/root_linux.go) for the default certificate file names based on various Linux distributions. For more information, go to HTTPS.
NETWORK_INTERFACE	Network interface used for the proxy.	Default: `eth0`. For more information, go to network interface .
CONTAINER_RUNTIME	Container runtime interface for the cluster.	Default: containerd. Support values: docker, containerd, and crio. For more information, go to container runtime .
SOCKET_PATH	Path of the containerd or crio or docker socket file.	Default: `/run/containerd/containerd.sock`. For more information, go to socket path .
NODE_LABEL	Node label used to filter the target node if `TARGET_NODE` environment variable is not set.	It is mutually exclusive with the `TARGET_NODE` environment variable. If both are provided, the fault uses `TARGET_NODE`. For more information, go to node label.
TOTAL_CHAOS_DURATION	Duration of chaos injection (in seconds).	Default: 60 s. For more information, go to duration of the chaos .
TARGET_PODS	Comma-separated list of application pod names subject to pod HTTP modify body.	If not provided, the fault selects target pods randomly based on provided appLabels. For more information, go to target specific pods.
PODS_AFFECTED_PERC	Percentage of total pods to target. Provide numeric values.	Default: 0 (corresponds to 1 replica). For more information, go to pod affected percentage .
RAMP_TIME	Period to wait before and after injecting chaos (in seconds).	For example, 30 s. For more information, go to ramp time.
SEQUENCE	Sequence of chaos execution for multiple target pods.	Default: parallel. Supports serial and parallel. For more information, go to sequence of chaos execution.

Target service port

Port of the target service. Tune it by using the TARGET_SERVICE_PORT environment variable.

The following YAML snippet illustrates the use of this environment variable:

## provide the port of the targeted service
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  annotationCheck: "false"
  appinfo:
    appns: "default"
    applabel: "app=nginx"
    appkind: "deployment"
  chaosServiceAccount: litmus-admin
  experiments:
    - name: pod-api-block
      spec:
        components:
          env:
            # provide the port of the targeted service
            - name: TARGET_SERVICE_PORT
              value: "80"
            - name: PATH_FILTER
              value: '/status'

Path filter

API sub-path (or route) to filter the API calls. Tune it by using the PATH_FILTER environment variable.

The following YAML snippet illustrates the use of this environment variable:

## provide api path filter
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  annotationCheck: "false"
  appinfo:
    appns: "default"
    applabel: "app=nginx"
    appkind: "deployment"
  chaosServiceAccount: litmus-admin
  experiments:
    - name: pod-api-block
      spec:
        components:
          env:
            # provide the api path filter
            - name: PATH_FILTER
              value: '/status'
            # provide the port of the targeted service
            - name: TARGET_SERVICE_PORT
              value: "80"

Destination ports

Comma-separated list of the destination service or host ports for which egress traffic should be affected as a result of chaos testing on the target application. Tune it by using the DESTINATION_PORTS environment variable.

tip

It is applicable only when the SERVICE_DIRECTION environment variable is set to egress .

The following YAML snippet illustrates the use of this environment variable:

## provide destination ports
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  annotationCheck: "false"
  appinfo:
    appns: "default"
    applabel: "app=nginx"
    appkind: "deployment"
  chaosServiceAccount: litmus-admin
  experiments:
    - name: pod-api-block
      spec:
        components:
          env:
            # provide destination ports
            - name: DESTINATION_PORTS
              value: '80,443'
            # provide the api path filter
            - name: PATH_FILTER
              value: '/status'
            # provide the port of the targeted service
            - name: TARGET_SERVICE_PORT
              value: "80"

HTTPS

To enable HTTPS support for both incoming and outgoing traffic, set the HTTPS_ENABLED field to true. The configuration details vary depending on whether it's applied to ingress or egress traffic.

Ingress

Set the HTTPS_ENABLED parameter if the target application's HTTPS URL includes a port (e.g., https://<hostname>:port). If the URL is in the format https://<hostname> without a port, this setting is not required.

Egress

For outgoing traffic, enable HTTPS support by setting HTTPS_ENABLED to true. This ensures TLS certificates are used for secure communication through the proxy within the target application.

If the HTTP client in the target application is configured to reload certificates with each API call, set HTTPS_ENABLED to true, and there's no need to provide CA_CERTIFICATES or SERVER_CERTIFICATES. However, if the root certificate directory or file name differs from /etc/ssl/certs and ca-certificates.crt, specify them using the HTTPS_ROOT_CERT_PATH and HTTPS_ROOT_CERT_FILE_NAME environment variables.
If the HTTP client doesn't reload certificates with every API call, provide either the CA_CERTIFICATES or SERVER_CERTIFICATES environment variables to the chaos experiment. In this case, you don’t need to set HTTPS_ROOT_CERT_PATH and HTTPS_ROOT_CERT_FILE_NAME. The relevant CA certificates must be provided to the target application.

Generate Certificates

You can set either CA_CERTIFICATES or SERVER_CERTIFICATES depending on the use case.
1. Self-Signed Certificates: To generate self-signed certificates, use the following commands:
```
openssl req -x509 -newkey rsa:4096 -keyout ca.key -out ca.crt -days 365 -nodes -subj '/CN=*'
cat ca.key ca.crt > ca-cert.pem
cat ca-cert.pem | base64 # provide this value in the CA_CERTIFICATES environment variable
```
  Load the ca.crt CA certificate as the root CA into the target application, and set the base64-encoded value of ca-cert.pem in the CA_CERTIFICATES environment variable inside the experiment manifest.
2. CA-Signed Certificates: If you're using a trusted Certificate Authority (CA) to sign the server certificates, loading CA certificates into the target application may not be necessary, as it might already have the required CA certificates. Instead, you can simply set the SERVER_CERTIFICATES environment variable with the server certificates, as shown below:
```
cat server.key server.crt > server-cert.pem
cat server-cert.pem | base64 # provide this value in the SERVER_CERTIFICATES environment variable
```

The following YAML snippet illustrates the use of this environment variable:

## enable https support
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  annotationCheck: "false"
  appinfo:
    appns: "default"
    applabel: "app=nginx"
    appkind: "deployment"
  chaosServiceAccount: litmus-admin
  experiments:
    - name: pod-api-block
      spec:
        components:
          env:
            # enable https support
            - name: HTTPS_ENABLED
              value: 'true'
            - name: CA_CERTIFICATES
              value: 'Y3VzdG9tIGNlcnRpZmljYXRlcwo='
            # provide the api path filter
            - name: PATH_FILTER
              value: '/status'
            # provide the port of the targeted service
            - name: TARGET_SERVICE_PORT
              value: "80"

Advanced fault tunables

PROXY_PORT: Port where the proxy listens for requests and responses.
SERVICE_DIRECTION: Direction of the flow of control, either ingress or egress. It supports ingress, and egress values.
NETWORK_INTERFACE: Network interface used for the proxy.

The following YAML snippet illustrates the use of this environment variable:

# it injects the api modify body fault
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  annotationCheck: "false"
  appinfo:
    appns: "default"
    applabel: "app=nginx"
    appkind: "deployment"
  chaosServiceAccount: litmus-admin
  experiments:
    - name: pod-api-block
      spec:
        components:
          env:
            # provide the proxy port
            - name: PROXY_PORT
              value: '20000'
            # provide the connection type
            - name: SERVICE_DIRECTION
              value: 'ingress'
            # provide the network interface
            - name: NETWORK_INTERFACE
              value: 'eth0'
            # provide the api path filter
            - name: PATH_FILTER
              value: '/status'
            # provide the port of the targeted service
            - name: TARGET_SERVICE_PORT
              value: "80"

Advanced filters

HEADERS_FILTERS: The HTTP request headers filters, that accept multiple comma-separated headers in the format of key1:value1,key2:value2.
METHODS: The HTTP request method type filters, that accept comma-separated HTTP methods in upper case, that is, GET,POST.
QUERY_PARAMS: The HTTP request query parameters filter, accepts multiple comma-separated query parameters in the format of param1:value1,param2:value2.
SOURCE_HOSTS: Comma-separated source host names filters, indicating the origin of the HTTP request. This is relevant to the ingress type, specified by SERVICE_DIRECTION environment variable.
SOURCE_IPS: Comma-separated source IPs filters, indicating the origin of the HTTP request. This is specifically relevant to the ingress type, specified by SERVICE_DIRECTION environment variable.
DESTINATION_HOSTS: Comma-separated destination host names filters, indicating the hosts on which you call the API. This specification applies exclusively to the egress type, specified by SERVICE_DIRECTION environment variable.
DESTINATION_IPS: Comma-separated destination IPs filters, indicating the hosts on which you call the API. This specification applies exclusively to the egress type, specified by SERVICE_DIRECTION environment variable.

The following YAML snippet illustrates the use of this environment variable:

# it injects the api block fault
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  annotationCheck: "false"
  appinfo:
    appns: "default"
    applabel: "app=nginx"
    appkind: "deployment"
  chaosServiceAccount: litmus-admin
  experiments:
    - name: pod-api-block
      spec:
        components:
          env:
            # provide the headers filters
            - name: HEADERS_FILTERS
              value: 'key1:value1,key2:value2'
            # provide the methods filters
            - name: METHODS
              value: 'GET,POST'
            # provide the query params filters
            - name: QUERY_PARAMS
              value: 'param1:value1,param2:value2'
            # provide the source hosts filters
            - name: SOURCE_HOSTS
              value: 'host1,host2'
            # provide the source ips filters
            - name: SOURCE_IPS
              value: 'ip1,ip2'
            # provide the connection type
            - name: SERVICE_DIRECTION
              value: 'ingress'
            # provide the port of the targeted service
            - name: TARGET_SERVICE_PORT
              value: "80"

Container runtime and socket path

The CONTAINER_RUNTIME and SOCKET_PATH environment variables set the container runtime and socket file path, respectively.

CONTAINER_RUNTIME: It supports docker, containerd, and crio runtimes. The default value is containerd.
SOCKET_PATH: It contains path of containerd socket file by default(/run/containerd/containerd.sock). For docker, specify the path as /var/run/docker.sock. For crio, specify the path as /var/run/crio/crio.sock.

The following YAML snippet illustrates the use of these environment variables:

## provide the container runtime and socket file path
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  annotationCheck: "false"
  appinfo:
    appns: "default"
    applabel: "app=nginx"
    appkind: "deployment"
  chaosServiceAccount: litmus-admin
  experiments:
    - name: pod-api-block
      spec:
        components:
          env:
            # runtime for the container
            # supports docker, containerd, crio
            - name: CONTAINER_RUNTIME
              value: "containerd"
            # path of the socket file
            - name: SOCKET_PATH
              value: "/run/containerd/containerd.sock"
            # provide the port of the targeted service
            - name: TARGET_SERVICE_PORT
              value: "80"
            # provide the api path filter
            - name: PATH_FILTER
              value: '/status'

Use cases​

Permissions required​

Prerequisites​

Mandatory tunables​

Optional tunables​

Target service port​

Path filter​

Destination ports​

HTTPS​

Ingress​

Egress​

Generate Certificates​

Advanced fault tunables​

Advanced filters​

Container runtime and socket path​

Use cases

Permissions required

Prerequisites

Mandatory tunables

Optional tunables

Target service port

Path filter

Destination ports

HTTPS

Ingress

Egress

Generate Certificates

Advanced fault tunables

Advanced filters

Container runtime and socket path