> For the complete documentation index, see [llms.txt](https://developer.harness.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://developer.harness.io/resilience-testing/chaos-engineering/use-chaos-engineering/probes/apm-probes/prometheus-probe.md).

# Prometheus probe

Features and specification of the Prometheus probe

Prometheus probe allows you to run Prometheus queries and match the resulting output against specific conditions. You can define metrics-based SLOs in a declarative way and determine the experiment verdict based on their success. The probe runs the query on a Prometheus server defined by the endpoint and checks whether the output satisfies the specified criteria. The outcome of a PromQL query (that is provided) is used for probe validation.

#### Prerequisites <a href="#prerequisites" id="prerequisites"></a>

To use the Prometheus APM probe, you need:

* A running Prometheus server
* Access to the Prometheus API endpoint from the kubernetes execution plane
* Proper configuration of your application to expose metrics to Prometheus

#### Interactive Setup Guide <a href="#interactive-setup-guide" id="interactive-setup-guide"></a>

Follow along with this interactive guide to learn how to configure Prometheus probe:

{% embed url="<https://app.tango.us/app/embed/87f20060-9449-4ac1-84ef-e69eefa35e87?skipCover=false&defaultListView=false&skipBranding=false&makeViewOnly=false&hideAuthorAndDetails=true>" %}
Create Prometheus APM Probe
{% endembed %}

#### Steps to configure <a href="#steps-to-configure" id="steps-to-configure"></a>

1. In Resilience probe section under chaos module, click on **New Probe** button

   ![Create Prometheus Probe](https://2284173296-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FlSkpbpeYJ3rfGUkIrcyQ%2Fuploads%2Fgit-blob-466d102092ac9ed522d10ccaf4721e57421af017%2Fcreate-splunk-probe.png?alt=media)
2. Select the **APM Probe**
3. Provide the name of the probe and select **Prometheus** under APM Type

   ![Select Prometheus Probe](https://2284173296-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FlSkpbpeYJ3rfGUkIrcyQ%2Fuploads%2Fgit-blob-ba67417ef38f3bf301829c2974b9e9bb28e66dcc%2Fselect-prometheus-probe.png?alt=media)
4. Under Prometheus connector select connector
5. In Connector settings, you can either choose an existing connector or click **New Connector**

   ![Create Prometheus Connector](https://2284173296-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FlSkpbpeYJ3rfGUkIrcyQ%2Fuploads%2Fgit-blob-265e104e543e9733a089463c6912de1bafa483cd%2Fprometheus-connector.png?alt=media)
6. Provide the credentials of the Prometheus

   ![Prometheus Credentials](https://2284173296-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FlSkpbpeYJ3rfGUkIrcyQ%2Fuploads%2Fgit-blob-b400156f6768f62ea56db4f5131de92f18322413%2Fprometheus-credentials.png?alt=media)
7. Select the delegate and verify the connection and click on **Finish**

   ![Delegate](https://2284173296-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FlSkpbpeYJ3rfGUkIrcyQ%2Fuploads%2Fgit-blob-f67f4608e3b8ecdf3fd6d78f93f5c6bf99369628%2Fdelegate.png?alt=media)
8. Now connector is created and selected, click on **Configure Details**

   ![Configure Details](https://2284173296-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FlSkpbpeYJ3rfGUkIrcyQ%2Fuploads%2Fgit-blob-83b6ac0ca4bd93e2a89813781750c082dccf716d%2Fconfigure-details.png?alt=media)
9. Under Probe Properties pass the value of required parameters

   * **TLS Config**:
     * It offers a mechanism to validate TLS certifications for the Prometheus server. You can supply the `cacert` or the client certificate and client key to perform the validation. Alternatively, you have the option to enable the `insecureSkipVerify` check to bypass certificate validation.
     * For more details, refer to [Prometheus TLS configuration documentation](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#tls_config).
     * **CA File**: The `caFile` holds the file path of the CA certificates utilized for server TLS verification
     * **Cert File**: The `certFile` holds the file path of the client certificates utilized for TLS verification
     * **Key File**: The `keyFile` holds the file path of the client key utilized for TLS verification
     * **Insecure Skip Verify**: The `insecureSkipVerify` setting skips the TLS certificates checks
   * **Query**:
     * The query contains the PromQL query to extract out the desired Prometheus metrics via running it on the given Prometheus endpoint
     * Please Note that all the double quotes need to be parsed in the provided query.
     * **Example**: `avg_over_time(probe_duration_seconds{job=\"prometheus-blackbox-exporter\",instance=\"frontend.boutique.svc.cluster.local:80\"}[60s:1s])*1000`
     * For more details, refer to [Prometheus PromQL documentation](https://prometheus.io/docs/prometheus/latest/querying/basics/)

   ![Prometheus Query](https://2284173296-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FlSkpbpeYJ3rfGUkIrcyQ%2Fuploads%2Fgit-blob-fb49737bcc21fc7be7c09e3d327ff30ff3759351%2Fprometheus-query.png?alt=media)
10. Provide the comparison criteria under Prometheus Data Comparison

    ![Prometheus Data Comparison](https://2284173296-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FlSkpbpeYJ3rfGUkIrcyQ%2Fuploads%2Fgit-blob-6b0e0e7b9e7607e2d392a30f619f42939e9dfe3e%2Fprometheus-data-comparison.png?alt=media)
11. Provide the Run Properties

    ![Run Properties](https://2284173296-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FlSkpbpeYJ3rfGUkIrcyQ%2Fuploads%2Fgit-blob-c606a8e1e6959c248708794284e6e310750a2989%2Fcreate-probe.png?alt=media)
12. Then click on **Create Probe**

{% @harness-feedback/feedback %}