Skip to main content

Pod API modify header

Pod API modify header is a Kubernetes pod-level chaos fault that overrides the header values of API requests and responses with the user-provided values for the given keys. This is achieved by starting the proxy server and redirecting the traffic through the proxy server.

Pod API Modify Header

Use cases​

Pod API modify header:

  • Simulate different authentication states or test the behavior of your application when using invalid or expired credentials.
  • This fault can be utilized to validate the caching behavior of your API or client applications. By overriding cache-related headers, such as the "Cache-Control" or "ETag" headers, you can simulate cache validation scenarios.
  • It can be used to test content negotiation capabilities. By modifying the "Accept" header in the API request, you can simulate different content types or formats that the client application can accept.

Permissions required​

Below is a sample Kubernetes role that defines the permissions required to execute the fault.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: hce
  name: pod-api-modify-header
spec:
  definition:
    scope: Cluster # Supports "Namespaced" mode too
permissions:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["create", "delete", "get", "list", "patch", "deletecollection", "update"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["create", "get", "list", "patch", "update"]
  - apiGroups: [""]
    resources: ["pods/log"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["deployments, statefulsets"]
    verbs: ["get", "list"]
  - apiGroups: [""]
    resources: ["replicasets, daemonsets"]
    verbs: ["get", "list"]
  - apiGroups: [""]
    resources: ["chaosEngines", "chaosExperiments", "chaosResults"]
    verbs: ["create", "delete", "get", "list", "patch", "update"]
  - apiGroups: ["batch"]
    resources: ["jobs"]
    verbs: ["create", "delete", "get", "list", "deletecollection"]

Prerequisites​

  • Kubernetes> 1.17
  • The application pods should be in the running state before and after injecting chaos.

Mandatory tunables​

Tunable Description Notes
TARGET_CONTAINER Name of the container subject to API header modification. None. For more information, go to target specific container
NODE_LABEL Node label used to filter the target node if TARGET_NODE environment variable is not set. It is mutually exclusive with the TARGET_NODE environment variable. If both are provided, the fault uses TARGET_NODE. For more information, go to node label.
TARGET_SERVICE_PORT Port of the target service. Defaults to port 80. For more information, go to target service port.
HEADERS_MAP Map of the headers to modify or add For example, {"X-Litmus-Test-Header": "X-Litmus-Test-Value"}. To remove a header, just set the value to ""; For example, {"X-Litmus-Test-Header": ""}. For more information, go to headers map.

Optional tunables​

Tunable Description Notes
PATH_FILTER API path or route used for the filtering. Targets all paths if not provided. For more information, go to path filter .
HEADERS_FILTERS Filters for HTTP request headers accept multiple comma-separated headers in the format key1:value1,key2:value2. For more information, go to header filters.
METHODS The HTTP request method type accepts comma-separated HTTP methods in upper cases, such as "GET,POST". For more information, go to methods.
QUERY_PARAMS HTTP request query parameter filters accept multiple comma-separated query parameters in the format of param1:value1,param2:value2. For more information, go to query params.
SOURCE_HOSTS Includes comma-separated source host names as filters, indicating the origin of the HTTP request. This is specifically relevant to the "ingress" type. For more information, go to source hosts.
SOURCE_IPS This includes comma-separated source IPs as filters, indicating the origin of the HTTP request. This is specifically relevant to the "ingress" type. For more information, go to source ips.
DESTINATION_HOSTS Comma-separated destination host names are used as filters, indicating the hosts on which you call the API. This specification applies exclusively to the "egress" type. For more information, go to destination hosts.
DESTINATION_IPS Comma-separated destination IPs are used as filters, indicating the hosts on which you call the API. This specification applies exclusively to the "egress" type. For more information, go to destination hosts.
PROXY_PORT Port where the proxy listens for requests. Default: 20000. For more information, go to proxy port.
LIB_IMAGE Image used to inject chaos. Default: harness/chaos-go-runner:main-latest. For more information, go to image used by the helper pod.
SERVICE_DIRECTION Direction of the flow of control, ingress or egress Default: ingress. For more information, go to service direction .
DATA_DIRECTION API payload type, request or response Default: both. For more information, go to data direction .
DESTINATION_PORTS comma-separated list of the destination service or host ports for which egress traffic should be affected Default: 80,8443. For more information, go to destination ports
HTTPS_ENABLED facilitate HTTPS support for both incoming and outgoing traffic Default: false. For more information, go to https
CUSTOM_CERTIFICATES Provide the custom certificates for the proxy server to serve as intermediate certificates for HTTPS communication The HTTPS communications necessitate its use as intermediate certificates by the proxy server. These same certificates must also be loaded into the target application. For more information, go to HTTPS
HTTPS_ROOT_CERT_PATH Provide the root CA certificate directory path This setting must be configured if the root CA certificate directory differs from /etc/ssl/certs. Please refer to https://go.dev/src/crypto/x509/root_linux.go to understand the default certificate directory based on various Linux distributions. For more information, go to HTTPS
HTTPS_ROOT_CERT_FILE_NAME Provide the root CA certificate file name This setting must be configured if the root CA certificate file name differs from ca-certificates.crt. Please refer to https://go.dev/src/crypto/x509/root_linux.go to understand the default certificate file names based on various Linux distributions. For more information, go to HTTPS
NETWORK_INTERFACE Network interface used for the proxy. Default: eth0. For more information, go to network interface .
CONTAINER_RUNTIME Container runtime interface for the cluster Default: containerd. Support values: docker, containerd and crio. For more information, go to container runtime .
SOCKET_PATH Path of the containerd or crio or docker socket file. Default: /run/containerd/containerd.sock. For more information, go to socket path.
TOTAL_CHAOS_DURATION Duration of chaos injection (in seconds). Default: 60 s. For more information, go to duration of the chaos .
TARGET_PODS Comma-separated list of application pod names subject to pod HTTP modify body. If not provided, the fault selects target pods randomly based on provided appLabels. For more information, go to target specific pods.
PODS_AFFECTED_PERC Percentage of total pods to target. Provide numeric values. Default: 0 (corresponds to 1 replica). For more information, go to pod affected percentage
RAMP_TIME Period to wait before and after injecting chaos (in seconds). For example, 30 s. For more information, go to ramp time.
SEQUENCE Sequence of chaos execution for multiple target pods. Default: parallel. Supports serial and parallel. For more information, go to sequence of chaos execution.

Target service port​

Port of the targeted service. Tune it by using the TARGET_SERVICE_PORT environment variable.

The following YAML snippet illustrates the use of this environment variable:

## provide the port of the targeted service
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  annotationCheck: "false"
  appinfo:
    appns: "default"
    applabel: "app=nginx"
    appkind: "deployment"
  chaosServiceAccount: litmus-admin
  experiments:
    - name: pod-api-modify-header
      spec:
        components:
          env:
            # provide the port of the targeted service
            - name: TARGET_SERVICE_PORT
              value: "80"
            - name: HEADERS_MAP
              value: '{"X-Litmus-Test-Header": "X-Litmus-Test-Value"}'
            - name: PATH_FILTER
              value: '/status'

Headers map​

Map of the headers that need to be modified or added to the HTTP request or response. Tune it by using the HEADERS_MAP environment variable.

The following YAML snippet illustrates the use of this environment variable:

## provide the headers as a map
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  annotationCheck: "false"
  appinfo:
    appns: "default"
    applabel: "app=nginx"
    appkind: "deployment"
  chaosServiceAccount: litmus-admin
  experiments:
    - name: pod-api-modify-header
      spec:
        components:
          env:
            # map of headers to modify/add; Eg: {"X-Litmus-Test-Header": "X-Litmus-Test-Value"}
            # to remove a header, just set the value to ""; Eg: {"X-Litmus-Test-Header": ""}
            - name: HEADERS_MAP
              value: '{"X-Litmus-Test-Header": "X-Litmus-Test-Value"}'
            # provide the port of the targeted service
            - name: TARGET_SERVICE_PORT
              value: "80"
            - name: PATH_FILTER
              value: '/status'

Path filter​

API sub path (or route) to filter the API calls. Tune it by using the PATH_FILTER environment variable.

The following YAML snippet illustrates the use of this environment variable:

## provide api path filter
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  annotationCheck: "false"
  appinfo:
    appns: "default"
    applabel: "app=nginx"
    appkind: "deployment"
  chaosServiceAccount: litmus-admin
  experiments:
    - name: pod-api-modify-header
      spec:
        components:
          env:
            # provide the api path filter
            - name: PATH_FILTER
              value: '/status'
            - name: HEADERS_MAP
              value: '{"X-Litmus-Test-Header": "X-Litmus-Test-Value"}'
            # provide the port of the targeted service
            - name: TARGET_SERVICE_PORT
              value: "80"

Destination ports​

A comma-separated list of the destination service or host ports for which egress traffic should be affected as a result of chaos testing on the target application. Tune it by using the DESTINATION_PORTS environment variable.

note

It is applicable only for the egress SERVICE_DIRECTION.

The following YAML snippet illustrates the use of this environment variable:

## provide destination ports
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  annotationCheck: "false"
  appinfo:
    appns: "default"
    applabel: "app=nginx"
    appkind: "deployment"
  chaosServiceAccount: litmus-admin
  experiments:
    - name: pod-api-modify-header
      spec:
        components:
          env:
            # provide destination ports
            - name: DESTINATION_PORTS
              value: '80,443'
            # provide the api path filter
            - name: PATH_FILTER
              value: '/status'
            - name: HEADERS_MAP
              value: '{"X-Litmus-Test-Header": "X-Litmus-Test-Value"}'
            # provide the port of the targeted service
            - name: TARGET_SERVICE_PORT
              value: "80"

HTTPS​

Enable the HTTPS support for both incoming and outgoing traffic by setting the HTTPS_ENABLED field to true. Its usage varies depending on whether it's applied to ingress or egress scenarios.

Ingress​

Set this parameter if the HTTPS URL of the target application includes a port, formatted as https://<hostname>:port. However, if the HTTPS URL is in the format https://<hostname> without a port, this setting is not required.

Egress​

For outbound traffic, setting HTTPS_ENABLED to true is required to enable HTTPS support for external services. This enables the establishment of TLS certificates for the proxy within the target application.

  • If the HTTP client in the target application is configured to reload certificates with each API call, set HTTPS_ENABLED to true, and there's no need to provide CUSTOM_CERTIFICATES. However, if the root certificate directory and file name differ from /etc/ssl/certs and ca-certificates.crt respectively, set the root certificate directory path using the HTTPS_ROOT_CERT_PATH ENV variable and the file name using the HTTPS_ROOT_CERT_FILE_NAME ENV variable.
  • If the HTTP client in the target application isn't configured to reload certificates with each API call, you must provide the CUSTOM_CERTIFICATES ENV variable to the chaos experiment and no need to set HTTPS_ROOT_CERT_PATH and HTTPS_ROOT_CERT_FILE_NAME ENV. The same custom certificates should be loaded into the target application. You can generate custom certificates using the following commands: 
    openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.crt -days 365 -nodes -subj '/CN=*'
    cat key.pem cert.crt > ca-cert.pem
    cat ca-cert.pem | base64 # provide it inside the CUSTOM_CERTIFICATES ENV
    Load the cert.crt into the target application and provide the base64 encoded value of ca-cert.pem to the CUSTOM_CERTIFICATES ENV variable.

The following YAML snippet illustrates the use of this environment variable:

## enable https support
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  annotationCheck: "false"
  appinfo:
    appns: "default"
    applabel: "app=nginx"
    appkind: "deployment"
  chaosServiceAccount: litmus-admin
  experiments:
    - name: pod-api-modify-header
      spec:
        components:
          env:
            # enable https support
            - name: HTTPS_ENABLED
              value: 'true'
            - name: CUSTOM_CERTIFICATES
              value: 'Y3VzdG9tIGNlcnRpZmljYXRlcwo='
            # provide the api path filter
            - name: PATH_FILTER
              value: '/status'
            # provide the port of the targeted service
            - name: TARGET_SERVICE_PORT
              value: "80"

Advanced fault tunables​

  • PROXY_PORT: Port where the proxy listens for requests and responses.
  • SERVICE_DIRECTION: Direction of the flow of control, either ingress or egress. It supports ingress, egress values.
  • DATA_DIRECTION: API payload type, request, or response. It supports request, response, and both values.
  • NETWORK_INTERFACE: Network interface used for the proxy.

The following YAML snippet illustrates the use of this environment variable:

# it injects the api modify header fault
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  annotationCheck: "false"
  appinfo:
    appns: "default"
    applabel: "app=nginx"
    appkind: "deployment"
  chaosServiceAccount: litmus-admin
  experiments:
    - name: pod-api-modify-header
      spec:
        components:
          env:
            # provide the proxy port
            - name: PROXY_PORT
              value: '20000'
            # provide the connection type
            - name: SERVICE_DIRECTION
              value: 'ingress'
            # provide the payload type
            - name: DATA_DIRECTION
              value: 'both'
            # provide the network interface
            - name: NETWORK_INTERFACE
              value: 'eth0'
            # provide the api path filter
            - name: PATH_FILTER
              value: '/status'
            # provide the port of the targeted service
            - name: TARGET_SERVICE_PORT
              value: "80"
            - name: HEADERS_MAP
              value: '{"X-Litmus-Test-Header": "X-Litmus-Test-Value"}'

Advanced filters​

  • HEADERS_FILTERS: The HTTP request headers filters, that accept multiple comma-separated headers in the format of key1:value1,key2:value2.
  • METHODS: The HTTP request method type filters, that accept comma-separated HTTP methods in upper case, that is, GET,POST.
  • QUERY_PARAMS: The HTTP request query parameters filter, accepts multiple comma-separated query parameters in the format of param1:value1,param2:value2.
  • SOURCE_HOSTS: Comma-separated source host names filters, indicating the origin of the HTTP request. This is relevant to the ingress type, specified by SERVICE_DIRECTION environment variable.
  • SOURCE_IPS: Comma-separated source IPs filters, indicating the origin of the HTTP request. This is specifically relevant to the ingress type, specified by SERVICE_DIRECTION environment variable.
  • DESTINATION_HOSTS: Comma-separated destination host names filters, indicating the hosts on which you call the API. This specification applies exclusively to the egress type, specified by SERVICE_DIRECTION environment variable.
  • DESTINATION_IPS: Comma-separated destination IPs filters, indicating the hosts on which you call the API. This specification applies exclusively to the egress type, specified by SERVICE_DIRECTION environment variable.

The following YAML snippet illustrates the use of this environment variable:

# it injects the api modify header fault
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  annotationCheck: "false"
  appinfo:
    appns: "default"
    applabel: "app=nginx"
    appkind: "deployment"
  chaosServiceAccount: litmus-admin
  experiments:
    - name: pod-api-modify-header
      spec:
        components:
          env:
            # provide the headers filters
            - name: HEADERS_FILTERS
              value: 'key1:value1,key2:value2'
            # provide the methods filters
            - name: METHODS
              value: 'GET,POST'
            # provide the query params filters
            - name: QUERY_PARAMS
              value: 'param1:value1,param2:value2'
            # provide the source hosts filters
            - name: SOURCE_HOSTS
              value: 'host1,host2'
            # provide the source ips filters
            - name: SOURCE_IPS
              value: 'ip1,ip2'
            # provide the connection type
            - name: SERVICE_DIRECTION
              value: 'ingress'
            # provide the port of the targeted service
            - name: TARGET_SERVICE_PORT
              value: "80"
            - name: HEADERS_MAP
              value: '{"X-Litmus-Test-Header": "X-Litmus-Test-Value"}'

Container runtime and socket path​

The CONTAINER_RUNTIME and SOCKET_PATH environment variable to set the container runtime and socket file path, respectively.

  • CONTAINER_RUNTIME: It supports docker, containerd, and crio runtimes. The default value is containerd.
  • SOCKET_PATH: It contains path of containerd socket file by default(/run/containerd/containerd.sock). For docker, specify path as /var/run/docker.sock. For crio, specify path as /var/run/crio/crio.sock.

The following YAML snippet illustrates the use of these environment variables:

## provide the container runtime and socket file path
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  annotationCheck: "false"
  appinfo:
    appns: "default"
    applabel: "app=nginx"
    appkind: "deployment"
  chaosServiceAccount: litmus-admin
  experiments:
    - name: pod-api-modify-header
      spec:
        components:
          env:
            # runtime for the container
            # supports docker, containerd, crio
            - name: CONTAINER_RUNTIME
              value: "containerd"
            # path of the socket file
            - name: SOCKET_PATH
              value: "/run/containerd/containerd.sock"
            # provide the port of the targeted service
            - name: TARGET_SERVICE_PORT
              value: "80"
            # provide the api path filter
            - name: PATH_FILTER
              value: '/status'
            - name: HEADERS_MAP
              value: '{"X-Litmus-Test-Header": "X-Litmus-Test-Value"}'