> For the complete documentation index, see [llms.txt](https://developer.harness.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://developer.harness.io/resilience-testing/chaos-engineering/chaos-experiments/probe/prometheus-probe-probe.md).
# Prometheus Probe
{% @harness-package-selector/package-selector platforms="%5B%7B%22label%22%3A%22HTTP%20Probe%22%2C%22slug%22%3A%22http-probe%22%2C%22path%22%3A%22chaos-engineering%2Fguides%2Fprobe%2Fhttp-probe-probe%22%7D%2C%7B%22label%22%3A%22Command%20Probe%22%2C%22slug%22%3A%22command-probe%22%2C%22path%22%3A%22chaos-engineering%2Fguides%2Fprobe%2Fcommand-probe-probe%22%7D%2C%7B%22label%22%3A%22Prometheus%20Probe%22%2C%22slug%22%3A%22prometheus-probe%22%2C%22path%22%3A%22chaos-engineering%2Fguides%2Fprobe%2Fprometheus-probe-probe%22%7D%2C%7B%22label%22%3A%22K8S%20Probe%22%2C%22slug%22%3A%22k8s-probe%22%2C%22path%22%3A%22chaos-engineering%2Fguides%2Fprobe%2Fk8s-probe-probe%22%7D%2C%7B%22label%22%3A%22Datadog%20Probe%22%2C%22slug%22%3A%22datadog-probe%22%2C%22path%22%3A%22chaos-engineering%2Fguides%2Fprobe%2Fdatadog-probe-probe%22%7D%2C%7B%22label%22%3A%22Dynatrace%20Probe%22%2C%22slug%22%3A%22dynatrace-probe%22%2C%22path%22%3A%22chaos-engineering%2Fguides%2Fprobe%2Fdynatrace-probe-probe%22%7D%2C%7B%22label%22%3A%22SLO%20Probe%22%2C%22slug%22%3A%22slo-probe%22%2C%22path%22%3A%22chaos-engineering%2Fguides%2Fprobe%2Fslo-probe-probe%22%7D%2C%7B%22label%22%3A%22APM%20Probe%22%2C%22slug%22%3A%22apm-probe%22%2C%22path%22%3A%22chaos-engineering%2Fguides%2Fprobe%2Fapm-probe-probe%22%7D%2C%7B%22label%22%3A%22Container%20Probe%22%2C%22slug%22%3A%22container-probe%22%2C%22path%22%3A%22chaos-engineering%2Fguides%2Fprobe%2Fcontainer-probe-probe%22%7D%5D" selectedPlatform="prometheus-probe" %}
Prometheus probe allows you to run Prometheus queries and match the resulting output against specific conditions. You can define metrics-based SLOs in a declarative way and determine the experiment verdict based on their success. The probe runs the query on a Prometheus server defined by the endpoint and checks whether the output satisfies the specified criteria. The outcome of a PromQL query (that is provided) is used for probe validation.
{% hint style="info" %}
**YAML ONLY FEATURE**
In case of complex queries that span multiple lines, the `queryPath` attribute can be used to provide the link to a file consisting of the query. This file can be made available in the experiment pod via a ConfigMap resource, with the ConfigMap being passed in the [ChaosEngine](https://litmuschaos.github.io/litmus/experiments/concepts/chaos-resources/chaos-engine/contents/) or the [ChaosExperiment](https://litmuschaos.github.io/litmus/experiments/concepts/chaos-resources/chaos-experiment/contents/) CR. Refer to the probe schema [here](https://docs.litmuschaos.io/docs/concepts/probes#promprobe).
{% endhint %}
**Input**
Prometheus probe takes a PromQL query along with Prometheus service endpoints as input to check for specific criteria.
### Schema
Listed below is the probe schema for the Prometheus probe, with properties shared across all the probes and properties unique to the Prometheus probe.
| | | | | |
| --------- | ----------------------------------------------------------- | --------- | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Field** | **Description** | **Type** | **Range** | **Notes** |
| name | Flag to hold the name of the probe | Mandatory | N/A `type: string` | The `name` holds the name of the probe. It can be set based on the use case |
| endpoint | Flag to hold the prometheus endpoints for the promProbe | Mandatory | N/A `type: string` | The `endpoint` contains the prometheus endpoints |
| query | Flag to hold the promql query for the promProbe | Mandatory | N/A `type: string` | The `query` contains the promql query to extract out the desired prometheus metrics via running it on the given prometheus endpoint |
| queryPath | Flag to hold the path of the promql query for the promProbe | Optional | N/A `type: string` | The `queryPath` field is used in case of complex queries that spans multiple lines, the queryPath attribute can be used to provide the path to a file consisting of the same. This file can be made available to the experiment pod via a ConfigMap resource, with the ConfigMap name being defined in the ChaosEngine OR the ChaosExperiment CR. |
**Comparator**
| | | | | |
| --------- | ------------------------------------------------- | --------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- |
| **Field** | **Description** | **Type** | **Range** | **Notes** |
| type | Flag to hold type of the data used for comparison | Mandatory | `float` | The `type` contains type of data, which should be compared as part of comparison operation. Prometheus probe only compares with float data. |
| criteria | Flag to hold criteria for the comparison | Mandatory | It supports `{<, >, <=, >=, !=, ==, oneOf, between}` for int and float type. And `{equal, notEqual, contains, matches, notMatches, oneOf}` for string type. | The `criteria` contains criteria of the comparison, as a part of comparison operation. |
| value | Flag to hold value for the comparison | Mandatory | N/A `type: string` | The `value` contains value of the comparison, which should follow the given criteria as part of comparison operation. |
**Authentication**
This establishes a fundamental authentication mechanism for the Prometheus server. The "username:password", encoded in base64, should be placed either within the `credentials` field or as a file path in the `credentialsFile` field.
{% hint style="info" %}
The `credentials` and `credentialsFile` are two options that can't be used simultaneously.
{% endhint %}
| | | | | |
| --------------- | ----------------------------------------------------------------------------- | -------- | --------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Field** | **Description** | **Type** | **Range** | **Notes** |
| type | Flag to hold the authentication type | Optional | `string` | The `type` encompasses the authentication method, which includes support for both `basic` and `bearer` authentication types. |
| credentials | Flag to hold the basic auth credentials in `base64` format or `bearer`. token | Optional | `string` | The `credentials` consists of the basic authentication credentials, either as username:password encoded in `base64` format or as a `bearer` token, depending on the authentication type |
| credentialsFile | Flag to hold the basic auth credentials or bearer token file path | Optional | `string` | The `credentials` consists of file path for basic authentication credentials or a bearer token, which are then attached to the experiment pod as volume secrets. These secret resources contain either the username:password encoded in `base64` format or a `bearer` token, depending on the authentication type |
**TLS**
It offers a mechanism to validate TLS certifications for the Prometheus server. You can supply the `cacert` or the client certificate and client key to perform the validation. Alternatively, you have the option to enable the `insecureSkipVerify` check to bypass certificate validation.
| | | | | |
| ------------------ | ---------------------------------------- | -------- | --------- | -------------------------------------------------------------------------------------------- |
| **Field** | **Description** | **Type** | **Range** | **Notes** |
| caFile | Flag to hold the ca file path | Optional | `string` | The `caFile` holds the file path of the CA certificates utilized for server TLS verification |
| certFile | Flag to hold the client cert file path | Optional | `string` | The `certFile` holds the file path of the client certificates utilized for TLS verification |
| keyFile | Flag to hold the client key file path | Optional | `string` | The `keyFile` holds the file path of the client key utilized for TLS verification |
| insecureSkipVerify | Flag to skip the tls certificates checks | Optional | `boolean` | The `insecureSkipVerify` skip the tls certificates checks |
| serverName | Flag to hold the server name | Optional | `string` | The `serverName` name of the server |
**Run properties**
| | | | | |
| -------------------- | --------------------------------------------------------------------------- | --------- | ------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- |
| **Field** | **Description** | **Type** | **Range** | **Notes** |
| probeTimeout | Flag to hold the timeout of the probe | Mandatory | N/A `type: string` | The `probeTimeout` represents the time limit for the probe to execute the specified check and return the expected data |
| attempt | Flag to hold the attempt of the probe | Mandatory | N/A `type: integer` | The `attempt` contains the number of times a check is run upon failure in the previous attempts before declaring the probe status as failed. |
| interval | Flag to hold the interval of the probe | Mandatory | N/A `type: string` | The `interval` contains the interval for which probes waits between subsequent retries |
| probePollingInterval | Flag to hold the polling interval for the probes (applicable for all modes) | Optional | N/A `type: string` | The `probePollingInterval` contains the time interval for which continuous and onchaos probe should be sleep after each iteration |
| initialDelaySeconds | Flag to hold the initial delay interval for the probes | Optional | N/A `type: integer` | The `initialDelaySeconds` represents the initial waiting time interval for the probes. |
| stopOnFailure | Flags to hold the stop or continue the experiment on probe failure | Optional | N/A `type: boolean` | The `stopOnFailure` can be set to true/false to stop or continue the experiment execution after probe fails |
### Definition
```yaml
probe:
- name: "check-probe-success"
type: "promProbe"
promProbe/inputs:
endpoint: "prometheus-server.prometheus.svc.cluster.local:9090"
query: "sum(rate(http_requests_total{code=~\"2..\"}[1m])) by (job)"
comparator:
criteria: ">" #supports >=,<=,>,<,==,!= comparison
value: "0"
auth:
credentials: "base64()"
tlsConfig:
insecureSkipVerify: true
mode: "Edge"
runProperties:
probeTimeout: 5s
interval: 2s
attempt: 1
```
**Prometheus query (simple query)**
This section holds the PromQL query used to extract the desired Prometheus metrics by executing it on the specified Prometheus endpoint. You can input the Prometheus query in the 'query' field, and this can be initiated by configuring the `.promProbe/inputs.query` field.
Use the following example to tune this:
```yaml
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
name: engine-nginx
spec:
engineState: "active"
appinfo:
appns: "default"
applabel: "app=nginx"
appkind: "deployment"
chaosServiceAccount: litmus-admin
experiments:
- name: pod-delete
spec:
probe:
- name: "check-probe-success"
type: "promProbe"
promProbe/inputs:
# endpoint for the promethus service
endpoint: "prometheus-server.prometheus.svc.cluster.local:9090"
# promql query, which should be executed
query: "sum(rate(http_requests_total{code=~\"2..\"}[1m])) by (job)"
comparator:
# criteria which should be followed by the actual output and the expected output
#supports >=,<=,>,<,==,!= comparison
criteria: ">"
# expected value, which should follow the specified criteria
value: "0"
mode: "Edge"
runProperties:
probeTimeout: 5s
interval: 2s
attempt: 1
```
**Prometheus query (complex query)**
For intricate queries that extend across multiple lines, you can use the 'queryPath' attribute to specify the path to a file containing the query. This file can be accessed by the experiment pod through a ConfigMap resource, with the ConfigMap name defined in either the ChaosEngine or the ChaosExperiment CR. To set this up, configure the `promProbe/inputs.queryPath` field.
{% hint style="info" %}
The fields `queryPath` and `query` are mutually exclusive. If `query` is specified, it is used for the query; otherwise, `queryPath` is used.
{% endhint %}
Use the following example to tune this:
```yaml
# contains the prom probe which execute the query and match for the expected criteria
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
name: engine-nginx
spec:
engineState: "active"
appinfo:
appns: "default"
applabel: "app=nginx"
appkind: "deployment"
chaosServiceAccount: litmus-admin
experiments:
- name: pod-delete
spec:
probe:
- name: "check-probe-success"
type: "promProbe"
promProbe/inputs:
# endpoint for the promethus service
endpoint: "prometheus-server.prometheus.svc.cluster.local:9090"
# the configMap should be mounted to the experiment which contains promql query
# use the mounted path here
queryPath: "/etc/config/prometheus-query"
comparator:
# criteria which should be followed by the actual output and the expected output
#supports >=,<=,>,<,==,!= comparison
criteria: ">"
# expected value, which should follow the specified criteria
value: "0"
mode: "Edge"
runProperties:
probeTimeout: 5s
interval: 2s
attempt: 1
```
**Authentication**
This establishes a fundamental authentication mechanism for the Prometheus server. The "username:password" encoded in `base64` or `bearer` token, should be placed either within the `credentials` field or as a file path in the `credentialsFile` field.
{% hint style="info" %}
The `credentials` and `credentialsFile` are mutually exclusive, that is, these fields can't be used simultaneously.
{% endhint %}
Use the following example to tune this:
```yaml
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
name: engine-nginx
spec:
engineState: "active"
appinfo:
appns: "default"
applabel: "app=nginx"
appkind: "deployment"
chaosServiceAccount: litmus-admin
experiments:
- name: pod-delete
spec:
probe:
- name: "check-probe-success"
type: "promProbe"
promProbe/inputs:
# endpoint for the promethus service
endpoint: "prometheus-server.prometheus.svc.cluster.local:9090"
# promql query, which should be executed
query: "sum(rate(http_requests_total{code=~\"2..\"}[1m])) by (job)"
comparator:
# criteria which should be followed by the actual output and the expected output
#supports >=,<=,>,<,==,!= comparison
criteria: ">"
# expected value, which should follow the specified criteria
value: "0"
auth:
type: Basic
credentials: "base64()"
mode: "Edge"
runProperties:
probeTimeout: 5s
interval: 2s
attempt: 1
```
**TLS with custom certificates**
It offers a mechanism to validate TLS certifications for the Prometheus server. You can supply the `cacert` or the client certificate and client key to perform the validation.
{% hint style="info" %}
The CA certificate file must be incorporated into the experiment pod either as a configMap or a secret. The volume name (configMap or secret) and mountPath should be specified within the chaosengine at the `spec.components.secrets` path.
{% endhint %}
Use the following example to tune this:
```yaml
# contains the prom probe which execute the query and match for the expected criteria
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
name: engine-nginx
spec:
engineState: "active"
appinfo:
appns: "default"
applabel: "app=nginx"
appkind: "deployment"
chaosServiceAccount: litmus-admin
experiments:
- name: pod-delete
spec:
components:
secrets:
- name: ca-cert
mountPath: /etc/config
probe:
- name: "check-probe-success"
type: "promProbe"
promProbe/inputs:
# endpoint for the promethus service
endpoint: "https://prometheus-server.harness.io"
# promql query, which should be executed
query: "sum(rate(http_requests_total{code=~\"2..\"}[1m])) by (job)"
comparator:
# criteria which should be followed by the actual output and the expected output
#supports >=,<=,>,<,==,!= comparison
criteria: ">"
# expected value, which should follow the specified criteria
value: "0"
tlsConfig:
caFile: "/etc/config/ca.crt"
mode: "Edge"
runProperties:
probeTimeout: 5s
interval: 2s
attempt: 1
```
**TLS skip certificate verification**
You can bypass the TLS certificate checks by enabling the `insecureSkipVerify` option.
Use the following example to tune this:
```yaml
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
name: engine-nginx
spec:
engineState: "active"
appinfo:
appns: "default"
applabel: "app=nginx"
appkind: "deployment"
chaosServiceAccount: litmus-admin
experiments:
- name: pod-delete
spec:
probe:
- name: "check-probe-success"
type: "promProbe"
promProbe/inputs:
# endpoint for the promethus service
endpoint: "https://prometheus-server.harness.io"
# promql query, which should be executed
query: "sum(rate(http_requests_total{code=~\"2..\"}[1m])) by (job)"
comparator:
# criteria which should be followed by the actual output and the expected output
#supports >=,<=,>,<,==,!= comparison
criteria: ">"
# expected value, which should follow the specified criteria
value: "0"
tlsConfig:
insecureSkipVerify: true
mode: "Edge"
runProperties:
probeTimeout: 5s
interval: 2s
attempt: 1
```
***
### Configuration
1. Go to **Chaos Engineering** module and select **Resilience Probes**. Select **New Probe**.
2. Select infrastructure type as **Kubernetes** and chaos probe as **Prometheus**.
3. Provide the name, and click **Configure Details**.
4. Based on your application's requirements, provide values for the following parameters.
* **Prometheus Endpoint**: It is the target HTTP/HTTPS endpoint that the probe will send requests to.
5. **Authorization** section has the following fields:
* **Type**: Type of HTTP request to be performed. Supports `GET` and `POST`.
* **Credentials**: Authentication credentials (username and password) required to access the target URL/endpoint. This field is mutually exclusive with **Credentials file** field.
* **Credentials file**: Path of the file that contains authentication credentials to access the HTTP endpoint. This field is mutually exclusive with **Credentials** field.
Go to [Authorization](/resilience-testing/chaos-engineering/use-chaos-engineering/probes/prometheus-probes/index.md#authentication) for more information.
```
**TLS Config** has the following fields:
- **CA file**: Path of the file to validate the custom certificates for TLS of the target URL.
- **Cert file**: Path of the file to the client certificate required for mTLS.
- **Key file**: Path of the file to the client key required for mTLS.
- **Insecure Skip Verify**: If enabled, the probe bypasses the SSL/TLS certificate verification, allowing requests to proceed even if the certificate is invalid or self-signed.
```
Go to [TLS](/resilience-testing/chaos-engineering/use-chaos-engineering/probes/prometheus-probes/index.md#tls) for more information.
 
6. Provide the **Prometheus Query** (**Query** or **Query Path** depending on your usage).
* **Query**: The PromQL query used with the probe to fetch the desired Prometheus metrics. Ensure that the strings inside the query are enclosed within backslash ("/"). This field is mutually exclusive with **Query Path** field.
* **Query Path**: Path of the file where PromQL query is present. This field is mutually exclusive with **Query** field.
Go to [Schema](/resilience-testing/chaos-engineering/use-chaos-engineering/probes/prometheus-probes/index.md#schema) for more information.
7. Specify the data comparison fields, and click **Configure Properties**.
The data returned using the PromQL **Query** or **Query Path** is compared to the following fields:
* **Type**: Type of data compared with result of Prometheus query. Accepts only `float` data type.
* **Comparison Criteria**: The criteria (`>=`, `<=`, `==`, `<`, `>`, `!=`, and so on) based on which the **value** and the result of Prometheus query are compared.
* **Value**: The value with which the result of Prometheus query is compared
Go to [Comparator](/resilience-testing/chaos-engineering/use-chaos-engineering/probes/prometheus-probes/index.md#comparator) for more information.
8. Specify general probe properties such as timeout, interval, and so on. Click **Create Probe**.
* **Timeout**: Time limit for the probe to execute the check and return the expected output.
* **Interval**: Duration for which the probe waits between subsequent attempts.
* **Attempt**: Number of times a check is executed upon failure in the previous attempts before declaring the probe status as `FAILED`.
* **Polling Interval**: Time interval for which `continuous` and `onchaos` probe modes should wait after each iteration.
* **Initial Delay**: Duration to wait before the probe begins execution.
* **Verbosity**: Level of detail to include in the logs generated during the execution of the probe. Choose between `info` (essential logs, probe status are printed) and `debug` (in-depth logs, timestamps, and execution logs are printed) mode.
* **Stop on Failure (Optional)**: Enable it to continue or disable it to stop the experiment execution after the probe fails. Disabled by default.
Go to [Run Properties](/resilience-testing/chaos-engineering/use-chaos-engineering/probes/prometheus-probes/index.md#run-properties) for more information.
