Skip to main content

Node taint

Node taint taints the node by applying the desired effect. Only the resources that contain the corresponding toleration can bypass the taints.

Node Taint

Use cases

  • Node taint fault verifies the resilience of applications when a certain taint is added to a node.
  • It simulates loss of critical services (or node-crash).
  • It verifies resource budgeting on cluster nodes (whether request(or limit) settings are honored on the available nodes).
  • It verifies whether topology constraints are adhered to (node selectors, tolerations, zone distribution, affinity(or anti-affinity) policies) or not.

Permissions required

Below is a sample Kubernetes role that defines the permissions required to execute the fault.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
namespace: hce
name: node-taint
spec:
definition:
scope: Cluster
permissions:
- apiGroups: [""]
resources: ["pods"]
verbs: ["create", "delete", "get", "list", "patch", "deletecollection", "update"]
- apiGroups: [""]
resources: ["events"]
verbs: ["create", "get", "list", "patch", "update"]
- apiGroups: [""]
resources: ["chaosEngines", "chaosExperiments", "chaosResults"]
verbs: ["create", "delete", "get", "list", "patch", "update"]
- apiGroups: [""]
resources: ["pods/log"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["pods/exec"]
verbs: ["get", "list", "create"]
- apiGroups: ["batch"]
resources: ["jobs"]
verbs: ["create", "delete", "get", "list", "deletecollection"]
- apiGroups: [""]
resources: ["nodes"]
verbs: ["get", "list", "patch", "update"]
- apiGroups: [""]
resources: ["pod eviction"]
verbs: ["get", "list", "create"]

Prerequisites

  • Kubernetes > 1.16
  • Node specified in the TARGET_NODE environment variable should be cordoned before executing the chaos fault. This ensures that the fault resources are not scheduled on it (or subject to eviction). This is achieved by the following steps:
    • Get node names against the applications pods using command kubectl get pods -o wide.
    • Cordon the node using command kubectl cordon NODENAME.
  • The target nodes should be in the ready state before and after injecting chaos.

Mandatory tunables

Tunable Description Notes
TARGET_NODE Name of the node to be tainted. For more information, go to target node.
NODE_LABEL It contains node label, which will be used to filter the target nodes if TARGET_NODES ENV is not set It is mutually exclusive with the TARGET_NODES environment variable. If both are provided, the fault uses the TARGET_NODES. For more information, go to node label.
TAINT_LABEL Label and the effect to be tainted on the application node. For more information, go to taint label.

Optional tunables

Tunable Description Notes
TOTAL_CHAOS_DURATION Duration that you specify, through which chaos is injected into the target resource (in seconds). Default: 60 s. For more information, go to duration of the chaos.
RAMP_TIME Period to wait before and after injecting chaos (in seconds). For example, 30 s. For more information, go to ramp time.

Taint label

Label and the effect to taint on the application node. Tune it by using the TAINT_LABEL environment variable.

The following YAML snippet illustrates the use of this environment variable:

# node tainted with provided key and effect
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
name: engine-nginx
spec:
engineState: "active"
annotationCheck: "false"
chaosServiceAccount: litmus-admin
experiments:
- name: node-taint
spec:
components:
env:
# label and effect to be tainted on the targeted node
- name: TAINT_LABEL
value: 'key=value:effect'
- name: TOTAL_CHAOS_DURATION
VALUE: '60'