Skip to main content

Command probe

This topic describes the definition and schema of the command probe. It also describes how to tune the probe in two modes: inline mode and source mode.

The command probe allows you to run Bash commands and match the output as part of the entry or exit criteria. The intent behind this probe is to implement a non-standard and imperative way to express the hypothesis. For example, you can check for specific data within a database, parse the value out of a JSON blob that is dumped into a certain path, or check for the existence of a particular string in the service logs.

YAML only feature

By default, this probe can be defined in inline mode from the user interface, where the command is run from within the experiment image. It can also be run in source mode, where the command execution is carried out from within a new pod whose image is specified. Inline mode is preferred for simple shell commands, and source mode is preferred when application-specific binaries are required.

Probe definition

You can define the probe at .spec.experiments[].spec.probe path inside the chaos engine.

kind: Workflow
apiVersion: argoproj.io/v1alpha1
spec:
templates:
- inputs:
artifacts:
- raw:
data: |
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
spec:
experiments:
- spec:
probe:
####################################
Probes are defined here
####################################
tip

The command probe expects you to provide a shell command to implement checks that are highly specific to the application use case.

Schema

Listed below is the probe schema for the command probe with properties shared across all the probes and properties unique to the command probe.

Field Description Type Range Notes
name Flag that holds the name of the probe Mandatory N/A type: string The name holds the name of the probe. It can be set based on the use-case
type Flag to hold the type of the probe Mandatory httpProbe, k8sProbe, cmdProbe, promProbe, and datadogProbe The type supports five types of probes: httpProbe, k8sProbe, cmdProbe, promProbe, and datadogProbe.
mode Flag to hold the mode of the probe Mandatory SOT, EOT, Edge, Continuous, OnChaos The mode supports five modes of probes: SOT, EOT, Edge, Continuous, and OnChaos. Datadog probe supports EOT mode only.
command Flag to hold the command for the cmdProbeMandatoryN/A type: stringThe command contains the shell command, which should be run as part of cmdProbe

Source

Field DescriptionType Range Notes
image Flag to hold the image for the cmdProbe Mandatory any source docker image The image provides the source image which can be used to launch a external pod where the command execution is carried out.
hostNetworkFlag to allow or deny the image access to the node network namespace Optional true, false The hostNetwork provides a possibility to allow the pod executing the cmdProbe access to the network of the node he is running on. For more details, go to the official Kubernetes documentation.
inheritInputs Flag for inheriting experiment pod attributes such as ENV, volumes, and volumeMounts into the probe pod Optional true, false Default is false i.e. the experiment pod attributes won't be inherited by the probe pod by default
args Arguments to be used for the source probe Optional type: string List of arguments to be used along with the probe command
env Environment variables to be used for the source probe pod Optional type: string List of environment variables to be used probe pod container
labels Labels for the source probe pod Optional type: string Labels to be added to the source probe pod
annotations Annotations for the source probe pod Optional type: string Annotations to be added to the source probe pod
commandCommand to be executed using the source probe image Optional type: string Command to be executed using the source probe image, which can be provided as a list
imagePullPolicy Image Pull Policy to be used for the source pod image Optional type: string Image Pull Policy to be used for the source pod image, supports Always, IfNotPresent, and Never
privileged Privileged execution permissions for the probe image Optional true, false Privileged execution permissions for the probe image, supports a boolean value
nodeSelector Label(s) of the node(s) to be used for deploying the source probe pod Optional type: string Key-Value label(s) of the node(s) to be used for deploying the source probe pod
volumes Volumes to be mounted to the source probe pod Optional N/A Depending on the type of the volume, additional properties need to be provided
volumesMount Volume mount paths for the corresponding source pod volumes Optional type: string List of volume mount paths for the corresponding source pod volumes
imagePullSecrets Image pull secrets for the source pod image Optional type: string List of image pull secrets for the source pod image

Comparator

Field Description Type Range Notes
type Flag to hold type of the data used for comparison Mandatory string, int, float The type contains type of data, which should be compared as part of comparison operation.
criteria Flag to hold criteria for the comparison Mandatory It supports >=, <=, ==, >, <, !=, oneOf, between for int and float type. And equal, notEqual, contains, matches, notMatches, oneOf for string type. The criteria contains criteria of the comparison, which should be fulfill as part of comparison operation.
value Flag to hold value for the comparison Mandatory N/A type: string The value contains value of the comparison, which should follow the given criteria as part of comparison operation.

Run properties

Field Description Type Range Notes
probeTimeout Flag to hold the timeout of the probe Mandatory N/A type: string The probeTimeout represents the time limit for the probe to execute the specified check and return the expected data
attempt Flag to hold the attempt of the probe Mandatory N/A type: integer The attempt contains the number of times a check is run upon failure in the previous attempts before declaring the probe status as failed.
interval Flag to hold the interval of the probe Mandatory N/A type: string The interval contains the interval for which probes waits between subsequent retries
probePollingInterval Flag to hold the polling interval for the probes (applicable for all modes) Optional N/A type: string The probePollingInterval contains the time interval for which continuous and onchaos probe should be sleep after each iteration
initialDelaySeconds Flag to hold the initial delay interval for the probes Optional N/A type: integer The initialDelaySeconds represents the initial waiting time interval for the probes.
stopOnFailure Flags to hold the stop or continue the experiment on probe failure Optional N/A type: boolean The stopOnFailure can be set to true/false to stop or continue the experiment execution after probe fails

Definition

A sample definition of the probe is given below.

In the case of Dedicated Chaos Infrastructure, the following apply:

  • The mode and type are mandatory fields in the probe schema when you define the entire configuration of the probe in the manifest (for Kubernetes (Legacy), Linux, and Windows infrastructure).
  • The name, mode, type and other input properties (depending on the probe) is required to rightly configure the resilience probe. If all the necessary details are not provided, the probe will not execute.

In the case of Harness Delegate, the following apply:

  • For Kubernetes (Harness Infrastructure) (also known as DDCR), the mandatory fields are mode and probeID, and the type field is derived. These fields are generated and patched in the backend to the same manifest. However, in the UI, you will only see the mode and probeID fields when configuring your experiment. This is because the manifest is minified in the UI.
  • If you define the entire probe in task.definition.chaos.probes, the entire configuration is required. If you use the task.probeRef, you only need to specify probeID and mode fields.
probe:
- name: "check-database-integrity"
type: "cmdProbe"
cmdProbe/inputs:
command: "<command>"
comparator:
type: "string" # supports: string, int, float
criteria: "contains" #supports >=,<=,>,<,==,!= for int and float, and contains, equal, notEqual, matches, notMatches, oneOf for string values
value: "<value-for-criteria-match>"
source: # omit this tag to "inline" the probe
image: "<repo>/<tag>"
hostNetwork: false
mode: "Edge"
runProperties:
probeTimeout: 5
interval: 5
retry: 1
initialDelaySeconds: 5

Inline mode

In the inline mode, the command probe is executed directly within the experiment pod. This mode is recommended for executing straightforward shell commands. It is the default mode and can be tuned by leaving out the 'source' field.

Use the following example to tune this:

# execute the command inside the experiment pod itself
# cases where command doesn't need any extra binaries, which is not available in the experiment image
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
name: engine-nginx
spec:
engineState: "active"
appinfo:
appns: "default"
applabel: "app=nginx"
appkind: "deployment"
chaosServiceAccount: litmus-admin
experiments:
- name: pod-delete
spec:
probe:
- name: "check-database-integrity"
type: "cmdProbe"
cmdProbe/inputs:
# command which needs to run in cmdProbe
command: "echo 'hello world'"
comparator:
# output type for the above command
# supports: string, int, float
type: "string"
# criteria which should be followed by the actual output and the expected output
#supports [>=, <=, >, <, ==, !=] for int and float
# supports [contains, equal, notEqual, matches, notMatches, oneOf] for string values
criteria: "contains"
# expected value, which should follow the specified criteria
value: "hello"
mode: "Edge"
runProperties:
probeTimeout: 5s
interval: 2s
attempt: 1

Source Mode

In source mode, the command is executed within a newly created pod, allowing for the specification of its image. This mode is useful when application-specific binaries are needed.

Use the following example to tune this:

# it launches the external pod with the source image and run the command inside the same pod
# cases where command needs an extra binaries which is not available in the experiment image
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
name: engine-nginx
spec:
engineState: "active"
appinfo:
appns: "default"
applabel: "app=nginx"
appkind: "deployment"
chaosServiceAccount: litmus-admin
experiments:
- name: pod-delete
spec:
probe:
- name: "check-database-integrity"
type: "cmdProbe"
cmdProbe/inputs:
# command which needs to run in cmdProbe
command: "<command>"
comparator:
# output type for the above command
# supports: string, int, float
type: "string"
# criteria which should be followed by the actual output and the expected output
#supports [>=, <=, >, <, ==, !=, oneOf, between] for int and float
# supports [contains, equal, notEqual, matches, notMatches, oneOf] for string values
criteria: "contains"
# expected value, which should follow the specified criteria
value: "<value-for-criteria-match>"
# source for the cmdProbe
source:
image: "<source-image>"
imagePullPolicy: Always
privileged: true
hostNetwork: false
mode: "Edge"
runProperties:
probeTimeout: 5s
interval: 2s
attempt: 1