> For the complete documentation index, see [llms.txt](https://developer.harness.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://developer.harness.io/resilience-testing/chaos-testing/probes/apm-probes/prometheus-apm-probes.md).
# Prometheus
{% @harness-package-selector/package-selector platforms="%5B%7B%22label%22%3A%22Prometheus%22%2C%22slug%22%3A%22prometheus%22%2C%22path%22%3A%22resilience-testing%2Fchaos-testing%2Fprobes%2Fapm-probes%2Fprometheus-apm-probes%22%7D%2C%7B%22label%22%3A%22AppDynamics%22%2C%22slug%22%3A%22appdynamics%22%2C%22path%22%3A%22resilience-testing%2Fchaos-testing%2Fprobes%2Fapm-probes%2Fappdynamics-apm-probes%22%7D%2C%7B%22label%22%3A%22Splunk%20Observability%22%2C%22slug%22%3A%22splunk-observability%22%2C%22path%22%3A%22resilience-testing%2Fchaos-testing%2Fprobes%2Fapm-probes%2Fsplunk-observability-apm-probes%22%7D%2C%7B%22label%22%3A%22Splunk%20Enterprise%22%2C%22slug%22%3A%22splunk-enterprise%22%2C%22path%22%3A%22resilience-testing%2Fchaos-testing%2Fprobes%2Fapm-probes%2Fsplunk-enterprise-apm-probes%22%7D%2C%7B%22label%22%3A%22Dynatrace%22%2C%22slug%22%3A%22dynatrace%22%2C%22path%22%3A%22resilience-testing%2Fchaos-testing%2Fprobes%2Fapm-probes%2Fdynatrace-apm-probes%22%7D%2C%7B%22label%22%3A%22New%20Relic%22%2C%22slug%22%3A%22new-relic%22%2C%22path%22%3A%22resilience-testing%2Fchaos-testing%2Fprobes%2Fapm-probes%2Fnew-relic-apm-probes%22%7D%2C%7B%22label%22%3A%22GCP%20Cloud%20Monitoring%22%2C%22slug%22%3A%22gcp-cloud-monitoring%22%2C%22path%22%3A%22resilience-testing%2Fchaos-testing%2Fprobes%2Fapm-probes%2Fgcp-cloud-monitoring-apm-probes%22%7D%2C%7B%22label%22%3A%22Datadog%22%2C%22slug%22%3A%22datadog%22%2C%22path%22%3A%22resilience-testing%2Fchaos-testing%2Fprobes%2Fapm-probes%2Fdatadog-apm-probes%22%7D%5D" selectedPlatform="prometheus" %}
Prometheus probe allows you to run Prometheus queries and match the resulting output against specific conditions. The probe runs the query on a Prometheus server and checks whether the output satisfies the specified criteria.
### When to use
* Track response latency or error rates via PromQL while injecting faults (e.g., verify p99 latency stays below a threshold during pod-delete)
* Validate custom application metrics exposed to Prometheus as steady-state indicators
* Define metrics-based SLOs declaratively and use them as experiment pass/fail criteria
### Prerequisites
* A running Prometheus server
* Access to the Prometheus API endpoint from the Kubernetes execution plane
* Proper configuration of your application to expose metrics to Prometheus
### Steps to configure
1. Navigate to **Project Settings** > **Chaos Probes** and click **+ New Probe**
2. Select **APM Probe**, provide a name, and select **Prometheus** under APM Type
3. Under **Variables**, define any reusable values you want to reference in probe properties or run properties. For each variable, specify the type (`String` or `Number`), name, value (fixed or runtime input), and whether it's required at runtime.
4. Under **Prometheus Connector**, select an existing connector or click **+ New Connector** to create one. Provide the Prometheus server URL, configure the delegate, verify the connection, and click **Finish**.
5. Under **Probe Properties**, configure:
| Field | Description |
| ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Query** | PromQL query to retrieve the desired metrics. All double quotes must be escaped.
Example: avg\_over\_time(probe\_duration\_seconds{job="prometheus-blackbox-exporter"}\[60s:1s])\*1000. See PromQL documentation
|
| **TLS Config** (optional) | TLS certificate validation for the Prometheus server. See [Prometheus TLS configuration](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#tls_config) |
| **CA File** (optional) | File path of the CA certificates for server TLS verification |
| **Cert File** (optional) | File path of the client certificates for TLS verification |
| **Key File** (optional) | File path of the client key for TLS verification |
| **Insecure Skip Verify** (optional) | Toggle to skip TLS certificate checks |
Under **Prometheus Data Comparison**, provide:
| Field | Description |
| ----------------------- | ------------------------------------------------------------------------- |
| **Type** | Data type for comparison: `Float` or `Int` |
| **Comparison Criteria** | Comparison operator: `>=`, `<=`, `==`, `!=`, `>`, `<`, `oneOf`, `between` |
| **Value** | The expected value to compare against the query result |
6. Provide the **Run Properties**:
| Field | Description |
| ------------------------------ | ----------------------------------------------- |
| **Timeout** | Maximum time for probe execution (e.g., `10s`) |
| **Interval** | Time between successive executions (e.g., `2s`) |
| **Attempt** | Number of retry attempts (e.g., `1`) |
| **Polling Interval** | Time between retries (e.g., `30s`) |
| **Initial Delay** | Delay before first execution (e.g., `5s`) |
| **Verbosity** | Log detail level |
| **Stop On Failure** (optional) | Stop the experiment if the probe fails |
7. Click **Create Probe**