Skip to main content

splunk-probe

Last updated on

The Splunk Observability probe allows you to query Splunk Observability (SignalFx) metrics and compare the results against specified criteria.

When to use​

  • Validate infrastructure or application metrics tracked in Splunk Observability during fault injection
  • Use SignalFx metric time series as steady-state indicators for chaos experiments
  • Monitor host-level metrics (CPU, memory, network) from Splunk while running chaos on the underlying infrastructure

Prerequisites​

  • An active Splunk Observability (SignalFx) account
  • Access to the Splunk Observability API from the kubernetes execution plane
  • An API token for authentication

Interactive Setup Guide​

Follow along with this interactive guide to learn how to configure Splunk Observability probe:

Steps to configure​

  1. Navigate to Project Settings > Chaos Probes and click New Probe

    Create Splunk Probe

  2. Select the APM Probe

  3. Provide the name of the probe and select Splunk Observability under APM Type

    Select Splunk Observability Probe

  4. Under Variables, define any reusable values you want to reference in probe properties or run properties. For each variable, specify the type (String or Number), name, value (fixed or runtime input), and whether it's required at runtime.

  5. Under Splunk Observability connector select connector

  6. In Connector settings, you can either choose an existing connector or click New Connector

    Create Splunk Connector

  7. Provide the credentials of the Splunk Observability

    Splunk Credentials

  8. Select the delegate and verify the connection and click on Finish

    Delegate

  9. Now connector is created and selected, click on Configure Details

    Configure Details

  10. Under Probe Properties, Pass the value of Splunk Observability Query and Lookback Window

  • Splunk Observability Query:
    • The Splunk Observability Query input is a string that specifies the search criteria for the metric time series (MTS) you want to retrieve. It follows a specific syntax that allows you to search for metrics, dimensions, properties, and tags
    • Example query: sf_metric:cpu.utilization AND host.name:gke-default-pool-667be17c-t588.c.test.internal
    • For more details refer to query section in Splunk Observability doc
  • LookBack Window (In Minutes):
    • The lookback window refers to the time range from a specified number of minutes ago up to the current moment, over which data is aggregated

Lookback Window

  1. Provide the comparison criteria under Splunk Observability Data Comparison

    Splunk Data Comparison

  2. Provide the Run Properties

    Run Properties

  3. Then click on Create Probe