.apps.dynatrace.com`) and a platform token. A connector is configured for one mode or the other, not both.
{% endhint %}
### Permissions
Classic and Grail use different authentication models. Grant the minimum permissions that match the query mode you use.
#### Classic API token scope
| Scope | Required for | Purpose |
| -------------- | ------------------------------------------ | ------------------------------------------- |
| `metrics.read` | **Classic** (metrics and entity selectors) | Read metric data through the Metrics v2 API |
Go to [Dynatrace API authentication](https://www.dynatrace.com/support/help/dynatrace-api/basics/dynatrace-api-authentication) to create an API token with this scope.
#### Grail platform token permissions
Grail permissions are granted through Dynatrace IAM policies rather than token scopes. The Grail Query API requires **both** a bucket permission and the table permissions for every table your DQL touches. A bucket permission alone does not grant access to data.
| Permission | Required | Affected DQL commands and functions |
| ------------------------- | ------------------------------- | ------------------------------------------------------------ |
| `storage:buckets:read` | Always | Required in addition to any table permission |
| `storage:metrics:read` | Always | `timeseries` |
| `storage:entities:read` | When filtering by entity | `entityName`, `entityAttr`, `classicEntitySelector`, `fetch` |
| `storage:smartscape:read` | When using Smartscape functions | `getNodeName`, `getNodeField`, `smartscapeNodes` |
A minimal IAM policy for the probe queries in this page:
```
ALLOW storage:buckets:read;
ALLOW storage:metrics:read, storage:entities:read;
```
{% hint style="info" %}
**ENTITY PERMISSIONS**
`storage:metrics:read` alone is enough only when your DQL filters on plain metric dimensions such as `k8s.namespace.name`. The example queries in this page filter services with `entityName(dt.entity.service)`, which reads the `entities` table and therefore also requires `storage:entities:read`. Without it, the query fails even though the metric permission is present.
{% endhint %}
You can narrow permissions with a `WHERE` clause. For example, limit metric access to a specific metric key:
```
ALLOW storage:metrics:read WHERE storage:metric.key = "dt.service.request.response_time";
```
Go to [Permissions in Grail](https://docs.dynatrace.com/docs/platform/grail/organize-data/assign-permissions-in-grail) and the [IAM policy reference](https://docs.dynatrace.com/docs/manage/identity-access-management/permission-management/manage-user-permissions-policies/advanced/iam-policystatements) to define these policies.
#### Harness permissions
To create and use the probe in Harness, your role needs:
* **Resilience Probe**: **Create / Edit** to add the probe, and **View** to reference it in an experiment
* **Connector**: **Create / Edit** to add the Dynatrace connector, and **View** and **Access** to select it in the probe
* **Secret**: **Create / Edit** to store the API token or platform token, and **View** and **Access** to reference it from the connector
Go to [RBAC](/resilience-testing/shared-capabilities/rbac.md) for the full list of resilience resources and permissions.
### Steps to configure
1. Navigate to **Project Settings** > **Chaos Probes** and click **+ New Probe**
2. Select **APM Probe**, provide a name, and select **Dynatrace** under APM Type
3. Under **Variables**, define any reusable values you want to reference in probe properties or run properties. For each variable, specify the type (`String` or `Number`), name, value (fixed or runtime input), and whether it's required at runtime. Use expressions such as `<+probe.variables.SERVICE_NAME>` inside DQL or selectors to make a probe reusable across services.
4. Under **Dynatrace Connector**, select an existing connector or click **+ New Connector** to create one, configure the delegate, verify the connection, and click **Finish**. See [Dynatrace API tokens documentation](https://www.dynatrace.com/support/help/dynatrace-api/basics/dynatrace-api-authentication) for Classic token details.
5. Under **Probe Properties**, choose **Classic** or **Grail** at the top. Harness sets the query path from your selection.
**Classic mode:**
| Field | Description |
| -------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Metrics Selector** | Dynatrace metrics selector query.
Example: builtin:service.response.time:avg:filter(eq("dt.entity.service","SERVICE-1234567890")). See Dynatrace Metrics API docs
|
| **Entity Selector** | Filter metrics by specific entities.
Example: type("SERVICE"),tag("environment:production"). See Entity Selector docs
|
| **Lookback Window (in minutes)** | Time range from the specified number of minutes ago to now, over which metrics are queried |
**Grail mode:**
| Field | Description |
| -------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **DQL Query** | A single [Dynatrace Query Language (DQL)](https://docs.dynatrace.com/docs/discover-dynatrace/references/dynatrace-query-language) `timeseries` query. The probe executes this query against the Grail Query API and compares the aggregated result to your threshold. |
| **Aggregation** | How datapoints in the lookback window are collapsed before comparison. Supported values: `mean` (default), `max`, `min`, `last` |
| **Lookback Window (in minutes)** | Time range from the specified number of minutes ago to now (`durationInMin`). The probe sends this as the Grail query timeframe; do not embed the timeframe in the DQL string. |
Under **Dynatrace Data Comparison**, provide:
| Field | Description |
| ----------------------- | ------------------------------------------------------------------------- |
| **Type** | Data type for comparison: `Float` or `Int` |
| **Comparison Criteria** | Comparison operator: `>=`, `<=`, `==`, `!=`, `>`, `<`, `oneOf`, `between` |
| **Value** | The expected value to compare against the metric result |
6. Provide the **Run Properties**:
| Field | Description |
| ------------------------------ | ----------------------------------------------- |
| **Timeout** | Maximum time for probe execution (e.g., `10s`) |
| **Interval** | Time between successive executions (e.g., `2s`) |
| **Attempt** | Number of retry attempts (e.g., `1`) |
| **Polling Interval** | Time between retries (e.g., `30s`) |
| **Initial Delay** | Delay before first execution (e.g., `5s`) |
| **Verbosity** | Log detail level |
| **Stop On Failure** (optional) | Stop the experiment if the probe fails |
7. Click **Create Probe**
### Grail DQL guidelines
When you write DQL for a Dynatrace Grail APM probe, follow these rules so the probe can parse and compare the result correctly.
**Return one numeric series**
The probe reads every numeric array in the query result and applies the configured **Aggregation**. Keep a single metric field (for example, `result`) using `fieldsKeep`:
```dql
timeseries p99 = percentile(dt.service.request.response_time, 99),
filter: {
entityName(dt.entity.service) == "<+probe.variables.SERVICE_NAME>"
and k8s.namespace.name == "<+probe.variables.NAMESPACE>"
and k8s.cluster.name == "<+probe.variables.CLUSTER_NAME>"
},
nonempty: true
| fieldsAdd result = p99[] / 1000
| fieldsKeep timeframe, interval, result
```
Do not use `append[...]` or multiple `timeseries` blocks in one query. Each probe run accepts exactly one DQL statement.
**Convert units in `fieldsAdd`, not inside `timeseries`**
Grail requires a bare metric aggregation in the `timeseries` assignment. Perform unit conversion afterward with array arithmetic (`[]`):
* `dt.service.request.response_time` is reported in **microseconds**. Divide by `1000` in `fieldsAdd` to compare in milliseconds.
* `dt.kubernetes.container.cpu_usage` is reported in **millicores**. Divide by `1000` in `fieldsAdd` to compare in CPU cores.
* `dt.kubernetes.container.memory_working_set` is reported in **bytes**. Divide by `1048576` in `fieldsAdd` to compare in MiB.
**Filter by service display name, not entity ID**
Filter service metrics with `entityName(dt.entity.service)` rather than hard-coding `SERVICE-…` entity IDs:
```dql
filter: { entityName(dt.entity.service) == "frontend" }
```
`entityName` reads the Grail `entities` table, so the platform token also needs `storage:entities:read`. Go to [Permissions](#permissions) for the full list.
**Disambiguate duplicate service names with Kubernetes dimensions**
If the same service name exists in multiple namespaces or clusters, add `k8s.namespace.name` and `k8s.cluster.name` to the filter. Omit `by:` in the probe query; splitting by dimension produces multiple numeric series that the probe averages together.
| Variable | Purpose |
| -------------- | ----------------------------------------------------------------------- |
| `SERVICE_NAME` | Dynatrace service display name (`entityName(dt.entity.service)`) |
| `NAMESPACE` | Kubernetes namespace (`k8s.namespace.name`) |
| `CLUSTER_NAME` | Kubernetes cluster name as registered in Dynatrace (`k8s.cluster.name`) |
If `CLUSTER_NAME` is omitted from the filter and the same service name exists in more than one cluster, the probe may merge metrics from all matching clusters into one comparison value.
### Dynatrace Grail APM example
The following probe template checks P99 latency for a service using Grail DQL. Service, namespace, and cluster are supplied at runtime through probe variables.
```yaml
identity: dynatrace-p99-latency-check
name: Dynatrace P99 Latency Check
type: apmProbe
infrastructureType: Kubernetes
probeProperties:
apmProbe:
comparator:
type: float
value: <+input>
criteria: <=
type: Dynatrace
dynatraceApmProbeInputs:
connectorID: <+input>
durationInMin: <+input>
queryType: grail
dqlQuery: |
timeseries p99 = percentile(dt.service.request.response_time, 99),
filter: {
entityName(dt.entity.service) == "<+probe.variables.SERVICE_NAME>"
and k8s.namespace.name == "<+probe.variables.NAMESPACE>"
and k8s.cluster.name == "<+probe.variables.CLUSTER_NAME>"
},
nonempty: true
| fieldsAdd result = p99[] / 1000
| fieldsKeep timeframe, interval, result
aggregation: mean
runProperties:
timeout: 30s
interval: 5s
attempt: 1
pollingInterval: 30s
initialDelay: 1s
verbosity: debug
variables:
- name: SERVICE_NAME
value: <+input>
type: String
description: The Dynatrace service display name to monitor.
required: true
- name: NAMESPACE
value: <+input>
type: String
description: The Kubernetes namespace of the target service.
required: true
- name: CLUSTER_NAME
value: <+input>
type: String
description: The Kubernetes cluster name as registered in Dynatrace.
required: true
```
In this example:
* Select **Grail** in the UI. Harness sets `queryType: grail` in YAML templates automatically.
* The DQL query returns one numeric field (`result`) in milliseconds
* `aggregation: mean` averages datapoints across the lookback window before comparison
* `durationInMin` controls the Grail query timeframe; it is not part of the DQL string
**Equivalent UI settings:**
* Mode: **Grail**
* DQL Query: the `dqlQuery` block above
* Aggregation: `mean`
* Lookback Window (in minutes): `10`
* Comparator: `<=` / `800` (milliseconds)
### Probe YAML reference
**Classic:**
```yaml
apmProbe/inputs:
type: Dynatrace
comparator:
type: float
value: "300"
criteria: <=
dynatraceApmProbeInputs:
connectorID: my-dynatrace-classic-connector
durationInMin: 5
metricsSelector: 'builtin:service.response.time:avg:filter(eq("dt.entity.service","SERVICE-1234567890"))'
entitySelector: 'type("SERVICE")'
runProperties:
timeout: 30s
interval: 5s
attempt: 1
```
**Grail:**
```yaml
apmProbe/inputs:
type: Dynatrace
comparator:
type: float
value: "800"
criteria: <=
dynatraceApmProbeInputs:
connectorID: my-dynatrace-grail-connector
durationInMin: 10
queryType: grail
dqlQuery: |
timeseries p99 = percentile(dt.service.request.response_time, 99),
filter: { entityName(dt.entity.service) == "frontend" },
nonempty: true
| fieldsAdd result = p99[] / 1000
| fieldsKeep timeframe, interval, result
aggregation: mean
runProperties:
timeout: 30s
interval: 5s
attempt: 1
```
{% hint style="info" %}
**NOTE**
* In the UI, **Classic** and **Grail** are separate choices. Harness maps them to the correct Dynatrace API path. In YAML templates, `queryType: grail` (or a populated `dqlQuery`) selects the Grail path.
* **Grail** requires a Dynatrace Grail connector (Platform URL + Platform Token), the feature flag `CDS_CV_DYNATRACE_GRAIL_METRICS_ENABLED`, and a minimum delegate version of `88900`. Contact [Harness Support](mailto:support@harness.io) to enable it.
* Classic probes continue to use the Dynatrace Metrics API and are unchanged when Grail is enabled for your account.
{% endhint %}
{% @harness-feedback/feedback %}