Prometheus probe

The Prometheus probe allows users to run Prometheus queries and match the resulting output against specific conditions. The intent behind this probe is to allow users to define metrics-based SLOs in a declarative way and determine the experiment verdict based on their success. The probe runs the query on a Prometheus server defined by the endpoint and checks whether the output satisfies the specified criteria. The outcome of a PromQL query (that is provided) is used for probe validation.

YAML only feature

In case of complex queries that span multiple lines, the queryPath attribute can be used to provide the link to a file consisting of the query. This file can be made available in the experiment pod via a ConfigMap resource, with the ConfigMap being passed in the ChaosEngine or the ChaosExperiment CR. Also, query and queryPath attributes are mutually exclusive. Refer to the probe schema here.

Probe definition

You can define the probes at .spec.experiments[].spec.probe path inside the chaos engine.

kind: Workflow
apiVersion: argoproj.io/v1alpha1
spec:
  templates:
    - inputs:
        artifacts:
          - raw:
              data: |
                apiVersion: litmuschaos.io/v1alpha1
                kind: ChaosEngine
                spec:
                  experiments:
                    - spec:
                        probe:
                          ####################################
                          Probes are defined here
                          ####################################

tip

The Prometheus probe expects you to provide a PromQL query along with Prometheus service endpoints to check for specific criteria.

Schema

Listed below is the probe schema for the Prometheus probe, with properties shared across all the probes and properties unique to the Prometheus probe.

Field	Description	Type	Range	Notes
name	Flag to hold the name of the probe	Mandatory	N/A `type: string`	The `name` holds the name of the probe. It can be set based on the use case
type	Flag to hold the type of the probe	Mandatory	`httpProbe, k8sProbe, cmdProbe, promProbe, and datadogProbe`	The `type` supports five types of probes: httpProbe, k8sProbe, cmdProbe, promProbe, and datadogProbe.
mode	Flag to hold the mode of the probe	Mandatory	`SOT, EOT, Edge, Continuous, OnChaos`	The `mode` supports five modes of probes: SOT, EOT, Edge, Continuous, and OnChaos. Datadog probe supports EOT mode only.
endpoint	Flag to hold the prometheus endpoints for the promProbe	Mandatory	N/A `type: string`	The `endpoint` contains the prometheus endpoints
query	Flag to hold the promql query for the promProbe	Mandatory	N/A `type: string`	The `query` contains the promql query to extract out the desired prometheus metrics via running it on the given prometheus endpoint
queryPath	Flag to hold the path of the promql query for the promProbe	Optional	N/A `type: string`	The `queryPath` field is used in case of complex queries that spans multiple lines, the queryPath attribute can be used to provide the path to a file consisting of the same. This file can be made available to the experiment pod via a ConfigMap resource, with the ConfigMap name being defined in the ChaosEngine OR the ChaosExperiment CR.

Comparator

Field	Description	Type	Range	Notes
type	Flag to hold type of the data used for comparison	Mandatory	`float`	The `type` contains type of data, which should be compared as part of comparison operation. Prometheus probe only compares with float data.
criteria	Flag to hold criteria for the comparison	Mandatory	It supports `<, >, <=, >=, !=, ==, oneOf, between` for int and float type. And `equal, notEqual, contains, matches, notMatches, oneOf` for string type.	The `criteria` contains criteria of the comparison, as a part of comparison operation.
value	Flag to hold value for the comparison	Mandatory	N/A `type: string`	The `value` contains value of the comparison, which should follow the given criteria as part of comparison operation.

Authentication

This establishes a fundamental authentication mechanism for the Prometheus server. The "username:password", encoded in base64, should be placed either within the credentials field or as a file path in the credentialsFile field.

tip

The credentials and credentialsFile are two options that can't be used simultaneously.

Field	Description	Type	Range	Notes
type	Flag to hold the authentication type	Optional	`string`	The `type` encompasses the authentication method, which includes support for both basic and bearer authentication types
credentials	Flag to hold the basic auth credentials in `base64` format or `bearer` token	Optional	`string`	The `credentials` consists of the basic authentication credentials, either as username:password encoded in `base64` format or as a `bearer` token, depending on the authentication type
credentialsFile	Flag to hold the basic auth credentials or bearer token file path	Optional	`string`	The `credentials` consists of file path for basic authentication credentials or a bearer token, which are then attached to the experiment pod as volume secrets. These secret resources contain either the username:password encoded in `base64` format or a `bearer` token, depending on the authentication type

TLS

It offers a mechanism to validate TLS certifications for the Prometheus server. You can supply the cacert or the client certificate and client key to perform the validation. Alternatively, you have the option to enable the insecureSkipVerify check to bypass certificate validation.

Field	Description	Type	Range	Notes
caFile	Flag to hold the ca file path	Optional	`string`	The `caFile` holds the file path of the CA certificates utilized for server TLS verification
certFile	Flag to hold the client cert file path	Optional	`string`	The `certFile` holds the file path of the client certificates utilized for TLS verification
keyFile	Flag to hold the client key file path	Optional	`string`	The `keyFile` holds the file path of the client key utilized for TLS verification
insecureSkipVerify	Flag to skip the tls certificates checks	Optional	`boolean`	The `insecureSkipVerify` skip the tls certificates checks
serverName	Flag to hold the server name	Optional	`string`	The `serverName` name of the server

Run properties

Field	Description	Type	Range	Notes
probeTimeout	Flag to hold the timeout of the probe	Mandatory	N/A `type: string`	The `probeTimeout` represents the time limit for the probe to execute the specified check and return the expected data
attempt	Flag to hold the attempt of the probe	Mandatory	N/A `type: integer`	The `attempt` contains the number of times a check is run upon failure in the previous attempts before declaring the probe status as failed.
interval	Flag to hold the interval of the probe	Mandatory	N/A `type: string`	The `interval` contains the interval for which probes waits between subsequent retries
probePollingInterval	Flag to hold the polling interval for the probes (applicable for all modes)	Optional	N/A `type: string`	The `probePollingInterval` contains the time interval for which continuous and onchaos probe should be sleep after each iteration
initialDelaySeconds	Flag to hold the initial delay interval for the probes	Optional	N/A `type: integer`	The `initialDelaySeconds` represents the initial waiting time interval for the probes.
stopOnFailure	Flags to hold the stop or continue the experiment on probe failure	Optional	N/A `type: boolean`	The `stopOnFailure` can be set to true/false to stop or continue the experiment execution after probe fails

Definition

In the case of Dedicated Chaos Infrastructure, the following apply:

The mode and type are mandatory fields in the probe schema when you define the entire configuration of the probe in the manifest (for Kubernetes (Legacy), Linux, and Windows infrastructure).
The name, mode, type and other input properties (depending on the probe) is required to rightly configure the resilience probe. If all the necessary details are not provided, the probe will not execute.

In the case of Harness Delegate, the following apply:

For Kubernetes (Harness Infrastructure) (also known as DDCR), the mandatory fields are mode and probeID, and the type field is derived. These fields are generated and patched in the backend to the same manifest. However, in the UI, you will only see the mode and probeID fields when configuring your experiment. This is because the manifest is minified in the UI.
If you define the entire probe in task.definition.chaos.probes, the entire configuration is required. If you use the task.probeRef, you only need to specify probeID and mode fields.

probe:
  - name: "check-probe-success"
    type: "promProbe"
    promProbe/inputs:
      endpoint: "prometheus-server.prometheus.svc.cluster.local:9090"
      query: "sum(rate(http_requests_total{code=~\"2..\"}[1m])) by (job)"
      comparator:
        criteria: ">" #supports >=,<=,>,<,==,!= comparison
        value: "0"
      auth:
        credentials: "base64(<username:password>)"
      tlsConfig:
        insecureSkipVerify: true
    mode: "Edge"
    runProperties:
      probeTimeout: 5s
      interval: 2s
      attempt: 1

Prometheus query (simple query)

This section holds the PromQL query used to extract the desired Prometheus metrics by executing it on the specified Prometheus endpoint. You can input the Prometheus query in the 'query' field, and this can be initiated by configuring the .promProbe/inputs.query field.