> For the complete documentation index, see [llms.txt](https://developer.harness.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://developer.harness.io/resilience-testing/chaos-testing/probes/apm-probes/splunk-observability-apm-probes.md).

# Splunk Observability

{% @harness-package-selector/package-selector platforms="%5B%7B%22label%22%3A%22Prometheus%22%2C%22slug%22%3A%22prometheus%22%2C%22path%22%3A%22resilience-testing%2Fchaos-testing%2Fprobes%2Fapm-probes%2Fprometheus-apm-probes%22%7D%2C%7B%22label%22%3A%22AppDynamics%22%2C%22slug%22%3A%22appdynamics%22%2C%22path%22%3A%22resilience-testing%2Fchaos-testing%2Fprobes%2Fapm-probes%2Fappdynamics-apm-probes%22%7D%2C%7B%22label%22%3A%22Splunk%20Observability%22%2C%22slug%22%3A%22splunk-observability%22%2C%22path%22%3A%22resilience-testing%2Fchaos-testing%2Fprobes%2Fapm-probes%2Fsplunk-observability-apm-probes%22%7D%2C%7B%22label%22%3A%22Splunk%20Enterprise%22%2C%22slug%22%3A%22splunk-enterprise%22%2C%22path%22%3A%22resilience-testing%2Fchaos-testing%2Fprobes%2Fapm-probes%2Fsplunk-enterprise-apm-probes%22%7D%2C%7B%22label%22%3A%22Dynatrace%22%2C%22slug%22%3A%22dynatrace%22%2C%22path%22%3A%22resilience-testing%2Fchaos-testing%2Fprobes%2Fapm-probes%2Fdynatrace-apm-probes%22%7D%2C%7B%22label%22%3A%22New%20Relic%22%2C%22slug%22%3A%22new-relic%22%2C%22path%22%3A%22resilience-testing%2Fchaos-testing%2Fprobes%2Fapm-probes%2Fnew-relic-apm-probes%22%7D%2C%7B%22label%22%3A%22GCP%20Cloud%20Monitoring%22%2C%22slug%22%3A%22gcp-cloud-monitoring%22%2C%22path%22%3A%22resilience-testing%2Fchaos-testing%2Fprobes%2Fapm-probes%2Fgcp-cloud-monitoring-apm-probes%22%7D%2C%7B%22label%22%3A%22Datadog%22%2C%22slug%22%3A%22datadog%22%2C%22path%22%3A%22resilience-testing%2Fchaos-testing%2Fprobes%2Fapm-probes%2Fdatadog-apm-probes%22%7D%5D" selectedPlatform="splunk-observability" %}

The Splunk Observability probe allows you to query Splunk Observability (SignalFx) metrics and compare the results against specified criteria.

### When to use <a href="#when-to-use" id="when-to-use"></a>

* Validate infrastructure or application metrics tracked in Splunk Observability during fault injection
* Use SignalFx metric time series as steady-state indicators for chaos experiments
* Monitor host-level metrics (CPU, memory, network) from Splunk while running chaos on the underlying infrastructure

### Prerequisites <a href="#prerequisites" id="prerequisites"></a>

* An active Splunk Observability (SignalFx) account
* Access to the Splunk Observability API from the Kubernetes execution plane
* An API token for authentication

### Steps to configure <a href="#steps-to-configure" id="steps-to-configure"></a>

1. Navigate to **Project Settings** > **Chaos Probes** and click **+ New Probe**
2. Select **APM Probe**, provide a name, and select **Splunk Observability** under APM Type
3. Under **Variables**, define any reusable values you want to reference in probe properties or run properties. For each variable, specify the type (`String` or `Number`), name, value (fixed or runtime input), and whether it's required at runtime.
4. Under **Splunk Observability Connector**, select an existing connector or click **+ New Connector** to create one. Provide the Splunk Observability credentials and API token, configure the delegate, verify the connection, and click **Finish**.
5. Under **Probe Properties**, configure:

   | Field                            | Description                                                                                                                                                                                                                                                                                                                                                                             |
   | -------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
   | **Splunk Observability Query**   | <p>Search criteria for the metric time series (MTS) you want to retrieve. Supports metrics, dimensions, properties, and tags.<br>Example: <code>sf\_metric:cpu.utilization AND host.name:gke-default-pool-667be17c-t588.c.test.internal</code>. See <a href="https://dev.splunk.com/observability/reference/api/retrieve_timeserieswindow/latest">Splunk Observability API docs</a></p> |
   | **Lookback Window (in minutes)** | Time range from the specified number of minutes ago to now, over which data is aggregated                                                                                                                                                                                                                                                                                               |

   Under **Splunk Observability Data Comparison**, provide:

   | Field                   | Description                                                               |
   | ----------------------- | ------------------------------------------------------------------------- |
   | **Type**                | Data type for comparison: `Float` or `Int`                                |
   | **Comparison Criteria** | Comparison operator: `>=`, `<=`, `==`, `!=`, `>`, `<`, `oneOf`, `between` |
   | **Value**               | The expected value to compare against the metric result                   |
6. Provide the **Run Properties**:

   | Field                          | Description                                     |
   | ------------------------------ | ----------------------------------------------- |
   | **Timeout**                    | Maximum time for probe execution (e.g., `10s`)  |
   | **Interval**                   | Time between successive executions (e.g., `2s`) |
   | **Attempt**                    | Number of retry attempts (e.g., `1`)            |
   | **Polling Interval**           | Time between retries (e.g., `30s`)              |
   | **Initial Delay**              | Delay before first execution (e.g., `5s`)       |
   | **Verbosity**                  | Log detail level                                |
   | **Stop On Failure** (optional) | Stop the experiment if the probe fails          |
7. Click **Create Probe**