> For the complete documentation index, see [llms.txt](https://developer.harness.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://developer.harness.io/continuous-delivery/use-continuous-delivery/verify-deployments/configure-cv/configure-verify-step-with-ai/health-sources/custom-log-health-source.md). # Configure a custom health source for AI Verify Connect any APM tool to AI Verify (v2) using a Custom Health connector to collect logs or metrics for automated deployment analysis. AI Verify (v2) supports a Custom Health health source type that lets you connect any APM or observability tool to your deployment pipeline. You configure a Custom Health connector pointing to your tool's API endpoint, choose whether the health source collects logs or metrics, define one or more queries, and map the response fields so the AI analysis engine can interpret the data. This is useful when your organization uses an APM tool that is not one of the natively supported integrations, such as a self-hosted OpenSearch cluster, a proprietary metrics backend, or any HTTP-accessible observability API. *** ## Before you begin * **Custom Health connector:** You need a Custom Health connector configured with your APM tool's base URL and any required headers or query parameters. Go to [Add a custom health source connector](https://developer.harness.io) to create one. * **AI Verify step:** The health source must be referenced in an AI Verify (v2) step inside a Container Step Group. Go to [Configure the AI Verify (v2) step](/continuous-delivery/use-continuous-delivery/verify-deployments/configure-cv/configure-verify-step-with-ai/ai-verify.md) to set it up. *** ## Create a custom health source Navigate to your project, organization, or account **Settings**, then select **Health Sources** from the left navigation. Click **+ New Health Source** to open the configuration wizard. {% stepper %} {% step %} **Configure the overview** On the **Overview** tab, provide the basic health source details. 1. Enter a **Health Source Name**, for example, `opensearch-custom-logs`. 2. In **Select health source type**, select the **Custom Health** tile. This tile covers both the logs and metrics variants of Custom Health; the feature you select next determines the rest of the configuration. 3. In **Select Connector**, choose the Custom Health connector that points to your APM tool. The base URL stored in the connector is displayed as a read-only field in the query form; you cannot edit it here. 4. In **Select Feature**, select **Metrics** to collect metric data, or **Custom Health Log** to collect log data. The query specification fields are the same for both; field mapping differs and is covered separately in the steps below. 5. Under **Setup Approach**, the only option is **Manual Setup**. Custom Health health sources do not support AI-assisted setup. 6. Click **Continue**. {% endstep %} {% step %} **Define query specifications** The **Query Specifications** tab is where you configure the HTTP request that fetches data from your APM tool. Each health source can have one or more named queries. This step is the same regardless of which feature you selected on the Overview tab. The left panel lists your queries. Click **+ Add Query** to add additional queries. Select a query in the list to edit its settings in the right panel. For each query, configure the following fields: **Query Name and identifier** Enter a human-readable **Query Name**, for example, `Error Logs` or `Checkout Latency`. Harness derives the query identifier automatically. The identifier must be unique within the health source. **Request method and path** Select **GET** or **POST** as the **Request Method**. Enter the relative **Path** for the request, starting with `/`. Do not include the scheme or host; those come from the connector's base URL. For a **POST** request, enter the request **Body** as a JSON string. The body must not contain Harness secret expressions such as `<+secrets.getValue("...")>`. Use connector headers or query parameters for authentication instead. The request body field is required for POST requests and must not be set for GET requests. **Time range placeholders** Enter a **Start time placeholder** and an **End time placeholder** as literal strings that appear in your path or body. These must differ from each other, and each must appear at least once in the path or body, so Harness can inject the time range at runtime. For each placeholder, select either **Seconds** or **Milliseconds** as the unit. Click **Run Query** to test the configuration. Harness sends the request to your APM tool using the current time window. For a Logs health source, the response appears as a list of JSON objects in the **Records** section. For a Metrics health source, Harness displays a single sample record from the response in the **Records** section. {% endstep %} {% step %} **Map fields** What you map next depends on the feature you selected on the Overview tab. **If you selected Custom Health Log** After the query returns records, map the log fields required by the AI analysis engine. Each field takes an absolute path from the root of the response, written in bracket notation: object keys in single quotes and array positions as numbers, with segments joined by dots. For example, `['hits'].['hits'].[0].['_source'].['log']`. You do not need to tell Harness where the list of log records is. Harness works it out from the paths you map. An array index in the middle of a path means "every element of this array", so `[0]` in `['hits'].['hits'].[0].['_source'].['log']` reads the log field from every record in `hits.hits`, not just the first one. An index at the very end of a path stays fixed to that position, which is useful when each record is itself an array, such as Loki's `[timestamp, line]` pairs. | Field | Required | Description | | ------------------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------- | | **Message Identifier** | Yes | Path to the log message text, for example `['hits'].['hits'].[0].['_source'].['log']` | | **Timestamp Identifier** | Yes | Path to the log timestamp, for example `['hits'].['hits'].[0].['_source'].['@timestamp']` | | **Service Instance Identifier** | Yes | Path to the identifier that distinguishes individual instances, for example `['hits'].['hits'].[0].['_source'].['kubernetes'].['pod_name']` | | **Severity** | No | Path to the log severity or level, for example `['hits'].['hits'].[0].['_source'].['loglevel']` | To select a path interactively, click the **+** icon next to a field. A path picker opens, showing the parsed response tree. Click any value in the tree to fill in its path. You can also type the path directly, but it must use the bracket notation shown above. Dotted GJSON paths such as `hits.hits.0._source.log` and JSONPath expressions such as `$.hits[0].log` are rejected when you save. {% hint style="info" %} * All fields must share the same array up to the point where they diverge; every field's path must agree on where the record array sits. * A path can cross at most two nested arrays. * The record array must sit under an object key. Responses that are a root-level array are not supported. * Fields with no array index in their path are read once from the response and applied to every record. * When two fields' paths both cross two arrays, both must use the same inner array. {% endhint %} **If you selected Metrics** After **Run Query** returns a sample record, open the **Metric Field Mapping** section and map the fields the AI analysis engine needs. Each field takes a path from the sample record, written in the same bracket notation used for logs, for example `['data'].[0].['value']`. | Field | Required | Description | | ------------------------------- | -------- | ---------------------------------------------------------------------------------------- | | **Metric Value JSON Path** | Yes | Path to the numeric metric value | | **Timestamp Identifier** | Yes | Path to the metric's timestamp | | **Service Instance Identifier** | Yes | Path to the identifier that distinguishes individual instances | | **Response Timestamp Format** | Yes | **Seconds** or **Milliseconds**, depending on how the backend encodes the timestamp | | **Records Path (optional)** | No | Path to the array root when the response nests records under a parent key | | **Metric List Path (optional)** | No | Path to the list of metrics within a record, when a record contains more than one metric | Click **Build Chart** to preview the mapping. Harness converts the mapped paths to a JSONPath expression, re-parses the sample record, and renders the resulting values as a chart so you can confirm the mapping extracts the correct metric before saving. {% endstep %} {% step %} **Save the health source** Click **Save** to create the health source. The health source appears on the **Health Sources** listing page with a **Log** signal-type badge and **Custom Health Log** as the type if you selected Custom Health Log, or a **Metric** signal-type badge and **Custom Metrics** as the type if you selected Metrics. The health source is now ready to reference in an AI Verify (v2) step. {% endstep %} {% endstepper %} *** ## Add multiple queries Custom Health health sources support multiple queries within a single health source. You might add multiple queries to cover different log streams or metrics from the same APM endpoint, for example, one query for error logs and another for application logs, or one query for latency and another for error rate. In the **Query Specifications** tab, click **+ Add Query** to add a new query entry in the left panel. Each query has its own name, request configuration, and field mapping. All queries in the health source run as part of the same AI Verify step execution, and each produces a separate query summary in the analysis results. *** ## Use the health source in an AI Verify pipeline Reference the custom health source by its identifier in the `AIVerifyNG` step. The step accepts multiple health source references, and you can mix Logs and Metrics health sources in the same list. During execution, the step fetches baseline data, then post-deployment data, and submits both sets to the AI analysis engine for comparison.
Kubernetes rolling deployment with Custom Health health source: full YAML ```yaml pipeline: name: Custom Health E2E identifier: Custom_Health_E2E projectIdentifier: your_project orgIdentifier: default stages: - stage: name: test-stage identifier: teststage type: Deployment spec: deploymentType: Kubernetes service: serviceRef: your_service environment: environmentRef: your_environment deployToAll: false infrastructureDefinitions: - identifier: your_k8s_infrastructure execution: steps: - step: name: Rollout Deployment identifier: rolloutDeployment type: K8sRollingDeploy timeout: 10m spec: skipDryRun: false pruningEnabled: false - stepGroup: name: AI Verify identifier: aiVerifyGroup steps: - step: type: AIVerifyNG name: AIVerifyV2_1 identifier: AIVerifyV2_1 spec: dataCollectionWindow: 2m healthSources: - healthSourceRef: opensearchcustomlogHS - healthSourceRef: prodcustommetricsHS resources: limits: cpu: 250m memory: 256Mi timeout: 10m stepGroupInfra: type: KubernetesDirect spec: connectorRef: your_k8s_connector namespace: default rollbackSteps: - step: name: Rollback Rollout Deployment identifier: rollbackRolloutDeployment type: K8sRollingRollback timeout: 10m spec: pruningEnabled: false failureStrategies: - onFailure: errors: - AllErrors action: type: StageRollback ```
You can include multiple custom health sources in a single AI Verify step by adding multiple entries under `healthSources`, in any combination of Logs and Metrics. The step runs one plugin container per health source inside a single pod. *** ## View analysis results After the pipeline runs, click the AI Verify step in the execution view. The **Analysis** tab shows a result for each health source. For a Logs health source, click the health source entry to see its **Query Summaries** section, which lists each query with a **healthy** or **unhealthy** badge. When a query is marked unhealthy, the analysis displays a summary of the detected anomalous log patterns, a root cause hypothesis, and the affected queries. The **Logs** tab in the execution view shows the raw plugin output, including the baseline log fetch, post-deployment log fetch, and the AI analysis submission. For a Metrics health source, click the health source entry to see the metric anomaly timeline the AI analysis engine generates by comparing the baseline window with the post-deployment window. In both cases, this output is generated by the AI analysis engine, which compares the baseline window against the post-deployment window. Go to [Configure the AI Verify (v2) step](/continuous-delivery/use-continuous-delivery/verify-deployments/configure-cv/configure-verify-step-with-ai/ai-verify.md#view-execution-results) for more on reading the Analysis tab. *** ## Next steps * [Configure the AI Verify (v2) step](/continuous-delivery/use-continuous-delivery/verify-deployments/configure-cv/configure-verify-step-with-ai/ai-verify.md): Add it to your pipeline and reference this health source. * [AI Verify (v2) overview](/continuous-delivery/use-continuous-delivery/verify-deployments/configure-cv/configure-verify-step-with-ai/overview.md): Understand how the AI analysis engine compares baseline and post-deployment data to produce a verdict. * [Interpret log results](/continuous-delivery/use-continuous-delivery/verify-deployments/cv-results/interpret-log-results.md): For a Logs health source, read and act on the log analysis results from verification runs. {% @harness-feedback/feedback %}