> For the complete documentation index, see [llms.txt](https://developer.harness.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://developer.harness.io/continuous-delivery/use-continuous-delivery/verify-deployments/configure-cv/ai-verify-v1/health-sources/dynatrace/dynatrace-health-source.md). # Dynatrace Learn how to set up Dynatrace as a health source for Continuous Verification in Harness. Harness Continuous Verification can use Dynatrace as a health source to analyze metrics and logs during deployments. This topic explains how to add Dynatrace as a health source and configure Full Stack Observability APM, Grail Logs, and Grail Metrics. ## What you will learn from this topic * How to [add Dynatrace as a health source](#add-dynatrace-as-a-health-source) for Continuous Verification. * How to configure [Full Stack Observability APM](#configuration) with metric packs and custom metrics. * How to configure Dynatrace Grail Logs for log-based verification. * How to configure Dynatrace Grail Metrics using Dynatrace Query Language (DQL). * How to write [sample Dynatrace queries](#sample-dynatrace-queries) for latency, traffic, errors, and saturation. *** ## Before you begin Go to [Configure CV](/continuous-delivery/use-continuous-delivery/verify-deployments/configure-cv/ai-verify-v1/verify-deployments.md) to complete the Continuous Verification setup steps that come before this page. ## Add Dynatrace as a health source This option is available only if you have configured the service and environment as fixed values. A Health Source is a mapping of a Harness Monitored Service to the Service in a deployment environment monitored by an APM or logging tool. ### Define health source In **Health Sources**, click **Add**. The **Add New Health Source** settings appear. ![](https://3694223630-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fy1JhZ4oKIppwY7d5AhPj%2Fuploads%2Fgit-blob-a94471c5f3e955ccfa1af00d216bacab7c59d6c7%2Fverify-deployments-with-dynatrace-14.png?alt=media) 1. In **Select health source type**, select **Dynatrace**. 2. In **Health Source Name**, enter a name for the Health Source. For example, Quickstart. 3. Under **Connect Health Source**, click **Select Connector**. 4. In **Connector** settings, you can choose an existing connector or click **New Connector** to create a new **Connector.** ![](https://3694223630-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fy1JhZ4oKIppwY7d5AhPj%2Fuploads%2Fgit-blob-45c375e69c9a62dc43bcc03b918cf69f5dea7350%2Fverify-deployments-with-dynatrace-15.png?alt=media)

Dynatrace connector types

When creating a new Dynatrace connector:

For more information on configuring Dynatrace connectors, go to Add Dynatrace connector.

5. After selecting the connector, click **Apply Selected**. The Connector is added to the Health Source. 6. In **Select Feature**, choose one of the following options. Your choice determines which configuration options appear next.

Feature requirements

### Configuration Depending on your feature choice, do the following configuration steps. {% tabs %} {% tab title="Full Stack Observability: APM" %} 7\. Click **Next**. The **Configuration** settings appear. ![](https://3694223630-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fy1JhZ4oKIppwY7d5AhPj%2Fuploads%2Fgit-blob-608e500a8d504d24bed5185154c2b2628dd640e5%2Fverify-deployments-with-dynatrace-16.png?alt=media) 8. Next, you will have the choice between using **Metric Packs** or **Custom Metrics**. ``` Choose **Metric Packs** if you want to use one of the predefined **Infrastructure** or **Performance** packs. Choose **Custom Metrics** to make your own metrics against your data. You may choose both options. ``` {% tabs %} {% tab title="Metric Packs" %} 8\. In **Find a Dynatrace service**, enter the name of the desired Dynatrace service. This Dynatrace service must be marked as a [key request](https://docs.dynatrace.com/docs/observe/applications-and-microservices/services/analysis/monitor-key-requests) in order to appear in this dropdown menu. 9\. In **Select Metric Packs to be monitored**, you can select **Infrastructure**, **Performance**, or both. {% endtab %} {% tab title="Custom Metrics" %} 8\. Click **Add Metric** if you want to add any specific metric to be monitored (optional) or click **Submit.** 9\. If you click Add Metric, click **Map Metric(s) to Harness Services**. 10\. In **Metric Name**, enter the name of the metric. 11\. In **Group Name**, enter the group name of the metric. 12\. Click **Query Specifications and mapping**. To build your query, do the following: ``` 1. In **Metric**, choose the desired metric from the list. 2. In **Select Metric Filter**, choose the desired entity from the list. This will filter your metrics using [entitySelectors](https://docs.dynatrace.com/docs/discover-dynatrace/references/dynatrace-api/environment-api/entity-v2/entity-selector).

Selecting metric filters is behind the feature flag CDS_CV_DYNATRACE_CANARY_ENABLED. Contact Harness Support to enable this feature.

3. Click **Fetch Records** to retrieve data for the provided query. ``` 13\. In **Assign**, choose the services for which you want to apply the metric. ``` If you select **Continuous Verification** or **Service Health**, you will need to configure a risk profile. Expand the following block for details. ```
Risk Profile settings For Dynatrace, the only possible values of the SII are your entity selectors. {% hint style="info" %} The ability to set a SII is behind the feature flag `CDS_CV_DYNATRACE_CANARY_ENABLED`. Contact [Harness Support](mailto:support@harness.io) to enable this feature. {% endhint %}
{% endtab %} {% endtabs %} Finally, click **Submit**. The Health Source is displayed in the AI Verify (v1) step. You can add one or more Health Sources for each APM or logging provider. {% endtab %} {% tab title="Dynatrace Grail Logs" %} {% hint style="info" %} This feature is behind the feature flag `CDS_CV_DYNATRACE_GRAIL_LOGS_ENABLED`. This feature requires a minimum delegate version of `869xx`. Contact [Harness Support](mailto:support@harness.io) to enable the feature flag. {% endhint %} 7. Click **Next**. The **Configuration** settings will appear. You should see one button, **+ Add Query**. 8. Click **+ Add Query**. 9. Choose a **Query name** and click **Submit**. 10. Under **Define Query**, enter your [query](https://docs.dynatrace.com/docs/discover-dynatrace/references/dynatrace-query-language). This query can also be a runtime input or expression. 11. After writing your fixed input query, click **Validate** to ensure your query is valid. 12. Then click **Run Query**. The query must be validated first from the previous step. ![](https://3694223630-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fy1JhZ4oKIppwY7d5AhPj%2Fuploads%2Fgit-blob-2e69775a54d8d1c2351d9a539157fd6185853db5%2Fverify-deployment-dynatrace-grail-1.png?alt=media) 13. Next, complete the field mapping for the **Timestamp Identifier**, **Service Instance Identifier**, and **Message Identifier**. To do so, click the `+` button and select the relevant field from the log that appears. ![](https://3694223630-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fy1JhZ4oKIppwY7d5AhPj%2Fuploads%2Fgit-blob-d07ec2466363f88ff134e5bd04f6869ba0022a65%2Ffield-mapping.png?alt=media) 14. Click **Submit**. The health source is displayed in the AI Verify (v1) step. {% endtab %} {% tab title="Dynatrace Grail Metrics" %} {% hint style="info" %} This feature is behind the feature flag `CDS_CV_DYNATRACE_GRAIL_METRICS_ENABLED` and requires a minimum delegate version of `88900`. Contact [Harness Support](mailto:support@harness.io) to enable the feature flag. {% endhint %} Dynatrace Grail Metrics uses [Dynatrace Query Language (DQL)](https://docs.dynatrace.com/docs/discover-dynatrace/references/dynatrace-query-language) to query timeseries metric data from the Dynatrace Grail data lakehouse. You write a `timeseries` DQL query, validate it, fetch sample records, and then map the metric value column and service instance dimension to Harness for continuous verification and live monitoring. ![Define Health Source, Dynatrace Grail Metrics selected](https://3694223630-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fy1JhZ4oKIppwY7d5AhPj%2Fuploads%2Fgit-blob-7b7f32249d4d3b91a9beaf659f24d8889701f40d%2Fdynatrace-grail-metrics-define-health-source.png?alt=media) **Map metric to Harness services** 7. Click **Next**. The **Configuration** tab opens. Click **+ Add Metric**. 8. In **Metric Name**, enter a name for the metric. 9. In **Group Name**, select an existing group or enter a new name to create one. ![Map Metric(s) to Harness Services](https://3694223630-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fy1JhZ4oKIppwY7d5AhPj%2Fuploads%2Fgit-blob-229f0ed640848b82bb0ad8f07f88b65552510132%2Fdynatrace-grail-metrics-map-metric.png?alt=media) **Write and validate the query** 10. Expand **Query Specifications and mapping** and enter your DQL `timeseries` query in the **DQL Query** field. The query must start with the `timeseries` command. ```dql timeseries avg(custom.app.response.time), filter: {deployment == "canary" and health=="unhealthy"}, by: {host} ```

The by: clause splits results per dimension (for example, by: {host}). Include it when you need per-instance data for continuous verification. You can omit it for aggregate SLI queries, but note that a Service Instance Identifier is required when Continuous Verification is enabled.

11. Click **Validate**. When the query is syntactically correct, a **Validation Passed** confirmation appears below the query field. 12. Click **Fetch Records**. Harness runs the query against your Dynatrace environment and shows the raw JSON response inline. ![DQL query, validation, and fetch records](https://3694223630-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fy1JhZ4oKIppwY7d5AhPj%2Fuploads%2Fgit-blob-3a1f6579c876ea9d38d43d534a02568f9d2e9192%2Fdynatrace-grail-metrics-dql-query.png?alt=media) **Configure metric values and charts** 13. Expand **Metric values and charts**. Harness auto-populates the fields based on the sample response. Review and confirm the following: * **Metric Value JSON Path**: the JSONPath to the metric values array in the response (for example, `['avg(custom.app.response.time)'].[0]`). * **Service Instance Identifier**: the JSONPath to the dimension that identifies each service instance (for example, `['host']`). Required when **Continuous Verification** is enabled. 14. Click **Build Chart** to preview the timeseries chart from your live data. Verify the chart renders correctly before proceeding. ![Metric values and charts with Build Chart](https://3694223630-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fy1JhZ4oKIppwY7d5AhPj%2Fuploads%2Fgit-blob-1e917f39333a40aad8874e4d5a9a11b92c047c83%2Fdynatrace-grail-metrics-values-chart.png?alt=media)

DQL timeseries responses return metric values as arrays where each element corresponds to one time slot. Harness reconstructs per-point timestamps using timeframe.start and interval from the response. Because interval is in nanoseconds, Harness converts it before applying the formula:

timestamp[i] = timeframe.start + (i × (interval / 1,000,000))

**Assign and configure risk profile** 15. Expand **Assign** and select which services this metric applies to: * **SLI**: uses the metric to track SLO compliance. * **Service Health**: monitors the metric continuously outside of deployments. * **Continuous Verification**: gates deployments based on this metric. If you select **Continuous Verification** or **Service Health**, configure the **Risk Category** and **Deviation Compared to Baseline**: * **Risk Category**: select one of **Errors**, **Infrastructure**, **Performance/Throughput**, **Performance/Other**, or **Performance/Response Time**. * **Deviation Compared to Baseline**: select **Higher value is higher risk**, **Lower value is higher risk**, or both, depending on what the metric measures. ![Assign section with risk category and deviation settings](https://3694223630-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2Fy1JhZ4oKIppwY7d5AhPj%2Fuploads%2Fgit-blob-5d63d4f79dd269d72559c384d649a38b559aa215%2Fdynatrace-grail-metrics-assign.png?alt=media) **Configure advanced thresholds (optional)** 16. Expand **Advanced (Optional)** to configure metric thresholds. There are two threshold types: * **Ignore Thresholds**: metrics matching these rules are excluded from anomaly flagging during continuous verification. * **Fail-Fast Thresholds**: metrics matching these rules immediately fail the verification step. Click **+ Add Threshold** to define rules by metric type, metric name, and criteria. 17. Click **Submit**. The health source is displayed in the AI Verify (v1) step.
Sample DQL timeseries queries The following queries illustrate common patterns for use with the Dynatrace Grail Metrics health source. **CPU usage grouped by host:** ```dql timeseries usage=avg(dt.host.cpu.usage), by:{dt.entity.host} ``` **Service response time at the 90th percentile:** ```dql timeseries p90=percentile(dt.service.request.response_time, 90), filter:{startsWith(endpoint.name, "/api/accounts")} ``` **Failure rate per second:** ```dql timeseries sum(dt.service.request.failure_count, rate:1s), filter:{startsWith(endpoint.name, "/api/accounts")} ``` **Multiple disk metrics in one query:** ```dql timeseries { bytes_read=sum(dt.host.disk.bytes_read), bytes_written=sum(dt.host.disk.bytes_written) }, by:{dt.entity.host} ``` **Week-over-week availability comparison:** ```dql timeseries avail=avg(dt.host.disk.avail), by:{dt.entity.host}, from:-24h | append [ timeseries avail_7d=avg(dt.host.disk.avail), by:{dt.entity.host}, shift:-7d ] ``` For the full DQL `timeseries` reference, see the [Dynatrace DQL timeseries documentation](https://docs.dynatrace.com/docs/discover-dynatrace/references/dynatrace-query-language/commands/data-retrieval/timeseries).
{% endtab %} {% endtabs %} *** ### Sample Dynatrace queries The following sample queries are grouped by the four golden signals: latency, traffic, errors, and saturation. **Latency** * Latency trend over time: `timeseries(avg(response.time))` * Latency distribution: `histogram(response.time)` * Latency by application version: `avg(response.time) by application.version` * Latency by geographical region: `avg(response.time) by geoip.country_name` * Latency spike detection: `spike(response.time)` * Latency comparison between environments: `avg(response.time) by environment` * Latency by HTTP method: `avg(response.time) by http.method` * Latency by service: `avg(response.time) by service.name` * Latency anomaly detection: `anomaly(response.time)` * Latency percentiles: `percentile(response.time, 50)`, `percentile(response.time, 90)`, `percentile(response.time, 99)` **Traffic** * Requests per minute trend: `timeseries(count(request) / 60)` * Requests by HTTP status code: `count(request) by http.status_code` * Requests by user agent: `count(request) by useragent.name` * Requests by endpoint and HTTP method: `count(request) by endpoint, http.method` * Requests by response time range: `histogram(response.time)` * Requests by geo-location: `count(request) by geoip.country_name` * Slow endpoint detection: `top(avg(response.time), 10, endpoint)` * Requests by hostname: `count(request) by hostname` * Requests by service: `count(request) by service.name` * Requests by HTTP version: `count(request) by http.version` **Errors** * Error rate trend over time: `timeseries(count(error) / count(request) * 100)` * Top error types: `count(error) by errorType` * Error rate by geographical region: `count(error) by geoip.country_name` * Error rate by application version: `count(error) by application.version` * Error rate by HTTP status code: `count(error) by http.status_code` * Error rate by service: `count(error) by service.name` * Error rate by user agent: `count(error) by useragent.name` * Error spike detection: `spike(count(error))` * Error anomaly detection: `anomaly(count(error))` **Saturation** * CPU utilization across hosts: `avg(cpu.usage) by host` * Memory utilization across hosts: `avg(memory.usage) by host` * Disk utilization across hosts: `avg(disk.usage) by host` * Network utilization across hosts: `avg(network.usage) by host` * CPU utilization by geographical region: `avg(cpu.usage) by geoip.country_name` * Memory utilization by geographical region: `avg(memory.usage) by geoip.country_name` * Disk utilization by geographical region: `avg(disk.usage) by geoip.country_name` * Network utilization by geographical region: `avg(network.usage) by geoip.country_name` *** ## Next steps * Go to [Configure CV](/continuous-delivery/use-continuous-delivery/verify-deployments/configure-cv/ai-verify-v1/verify-deployments.md) to learn how to add a verification step to your pipeline. * Go to [Add Dynatrace connector](https://developer.harness.io/harness-platform/use-harness-platform/connectors/monitoring-and-logging-systems/add-a-dynatrace-connector) to set up the connector required for this health source. * Go to [Validate releases with Dynatrace Site Reliability Guardian](/continuous-delivery/use-continuous-delivery/verify-deployments/configure-cv/ai-verify-v1/health-sources/dynatrace/dynatrace-srg.md) to use Dynatrace SRG with Harness. {% @harness-feedback/feedback %}