> For the complete documentation index, see [llms.txt](https://developer.harness.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://developer.harness.io/harness-platform/3.0/harness-platform-resources/delegates/delegate-closed-beta/configure-opentelemetry-observability.md).

# OpenTelemetry Observability

Export Delegate 3.x logs and metrics to your own observability stack using OpenTelemetry

Delegate 3.x can export logs and metrics using OpenTelemetry (OTEL), allowing you to route delegate telemetry directly into your own observability stack, such as Splunk, Datadog, or Amazon CloudWatch.

### What you will learn from this topic <a href="#what-you-will-learn-from-this-topic" id="what-you-will-learn-from-this-topic"></a>

* How to observe [delegate activity before OTEL](#delegate-observability-before-otel) using GCP Cloud Logging and the `/metrics` endpoint.
* How to configure and use [OTEL export](#otel-export-configuration-and-behavior), including log shipping, metric shipping, transport, and backward compatibility.
* How to [enable OTEL export](#enable-otel-export) with environment variables in `config.env`.
* How to [build an observability dashboard](#build-an-observability-dashboard) from the exported logs and metrics.

### Before you begin <a href="#before-you-begin" id="before-you-begin"></a>

Before you enable OTEL export, make sure you have:

* A running OTEL Collector that is reachable from the delegate.
* The OTLP endpoint for the OTEL Collector.
* The required configuration for your OTEL Collector and destination backend.

### Delegate observability before OTEL <a href="#delegate-observability-before-otel" id="delegate-observability-before-otel"></a>

Without OTEL support, Delegate 3.x offers two ways to observe delegate activity, and both have gaps for self-hosted deployments:

* GCP Cloud Logging: built-in, but only works for delegates running on Harness-hosted infrastructure.
* The `/metrics` endpoint: works everywhere, but requires you to run and maintain your own Prometheus scraper.

For on-premises and local binary deployments in particular, these options do not provide a native path for exporting delegate logs and metrics to your existing observability stack.

### OTEL export configuration and behavior <a href="#otel-export-configuration-and-behavior" id="otel-export-configuration-and-behavior"></a>

The delegate ships logs and metrics to an OTEL Collector over OTLP (gRPC or HTTP). The collector then routes the data to your backend of choice.

* **Log shipping**: The delegate captures all log levels and forwards them as OTEL logs using OTLP.
* **Metric shipping**: Existing Prometheus metrics are bridged to the OTEL SDK and exported over OTLP, so you get the same metrics the `/metrics` endpoint exposes, delivered through OTEL instead.
* **Backend-agnostic transport**: The delegate speaks OTLP only, over gRPC or HTTP (`HARNESS_OTEL_PROTOCOL`). There is no delegate-side setting for the destination backend. Backend selection, such as Splunk, Datadog, or Amazon CloudWatch, happens at the OTEL Collector the delegate points to.
* **Configuration**: You enable and configure OTEL export entirely through environment variables, with no code changes required.
* **Backward compatibility**: GCP Cloud Logging is controlled by its own `ENABLE_REMOTE_LOGGING` flag, independent of OTEL export. The `/metrics` endpoint always stays on as a fallback, regardless of your OTEL settings.

The delegate does not send telemetry directly to every observability backend. It exports telemetry to the configured OTEL endpoint, and the OTEL Collector handles processing and routing to the destination backend.

![Delegate OTEL export architecture](https://204875495-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FiHCl4N3pQkjhOYjQoOs7%2Fuploads%2Fgit-blob-4e275de4142e35a94b575c35fe44fb47e32bd342%2Fdelegate-otel-architecture.png?alt=media)

A sidecar-based exporter can still complement this integration for deployments that already standardize on a sidecar pattern. The native integration described in this topic works across all deployment models, including local binaries, without requiring a sidecar.

### Enable OTEL export <a href="#enable-otel-export" id="enable-otel-export"></a>

Enable OTEL export by adding the following variables to the delegate's `config.env`:

```bash
HARNESS_OTEL_ENABLED=true
HARNESS_OTEL_ENDPOINT=collector:4317
HARNESS_OTEL_PROTOCOL=grpc
HARNESS_OTEL_INSECURE=true
HARNESS_OTEL_EXPORT_LOGS=true
HARNESS_OTEL_EXPORT_METRICS=true
```

* **`HARNESS_OTEL_ENABLED`**: Enables OTEL export when set to `true`. Default: `false`.
* **`HARNESS_OTEL_ENDPOINT`**: The OTEL Collector endpoint that receives exported telemetry. The endpoint must be reachable from the delegate. For example, `collector:4317` uses the standard OTLP/gRPC port. No default; required when OTEL export is enabled.
* **`HARNESS_OTEL_PROTOCOL`**: The OTLP transport protocol. Supported values are `grpc` and `http`. Default: `grpc`.
* **`HARNESS_OTEL_INSECURE`**: Skips TLS certificate verification when set to `true`. Default: `true`. Set to `false` and configure the collector with a valid certificate for a secure connection.
* **`HARNESS_OTEL_EXPORT_LOGS`**: Enables log export when set to `true`. Default: `true`.
* **`HARNESS_OTEL_EXPORT_METRICS`**: Enables metric export when set to `true`. Default: `true`.

Optionally, set the following variables to control the resource attributes attached to exported telemetry:

* **`HARNESS_OTEL_SERVICE_NAME`**: The service name attached to exported telemetry. Default: `harness-delegate`.
* **`HARNESS_OTEL_SERVICE_VERSION`**: The service version attached to exported telemetry. No default.
* **`HARNESS_OTEL_ENVIRONMENT`**: The deployment environment label attached to exported telemetry, such as `production`. No default.

<figure><img src="https://204875495-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FiHCl4N3pQkjhOYjQoOs7%2Fuploads%2Fgit-blob-adb33da813bae5b4edc314b76abb00405548236b%2Fdelegate-otel-env-vars.png?alt=media" alt="OTEL environment variables" width="300"><figcaption></figcaption></figure>

Set `HARNESS_OTEL_EXPORT_LOGS` and `HARNESS_OTEL_EXPORT_METRICS` independently of each other, depending on whether you want to export logs, metrics, or both.

### Build an observability dashboard <a href="#build-an-observability-dashboard" id="build-an-observability-dashboard"></a>

Once logs and metrics are flowing into your observability stack, you can build a dashboard to monitor delegate health, such as log volume by severity, error and warning counts, and metric throughput.

![Example delegate observability dashboard](https://204875495-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FiHCl4N3pQkjhOYjQoOs7%2Fuploads%2Fgit-blob-2684c05080de043b8542fd5fb22d7eec5fbe1409%2Fdelegate-otel-dashboard.png?alt=media)

### Troubleshoot OTEL export <a href="#troubleshoot-otel-export" id="troubleshoot-otel-export"></a>

<details>

<summary>Telemetry does not appear in the observability backend</summary>

If telemetry does not appear in your observability backend, check the following:

* `HARNESS_OTEL_ENABLED` is set to `true`.
* `HARNESS_OTEL_PROTOCOL` is set to a supported value (`grpc` or `http`).
* `HARNESS_OTEL_EXPORT_LOGS` and `HARNESS_OTEL_EXPORT_METRICS` are set to `true` for the telemetry you want to export.
* The endpoint in `HARNESS_OTEL_ENDPOINT` is reachable from the delegate.
* If the collector requires TLS, `HARNESS_OTEL_INSECURE` is set to `false` and the delegate trusts the collector's certificate.
* The OTEL Collector is running and accepting OTLP traffic.
* The OTEL Collector is configured to route telemetry to the expected backend.
* OTEL SDK errors are surfaced through the delegate's error handler into its logs. Review delegate logs for OTEL export errors.

</details>

### Next steps <a href="#next-steps" id="next-steps"></a>

* [Delegate Configuration Reference](https://developer.harness.io/harness-platform/3.0/harness-platform-resources/delegates/delegate-closed-beta/install-a-delegate/configure-delegate.md): Go to the Metrics section to compare the OTEL export with the built-in `/metrics` endpoint.
* [Overview](https://developer.harness.io/harness-platform/3.0/harness-platform-resources/delegates/delegate-closed-beta/delegate-overview.md): Go to the Logging and observability section for background on structured logging in Delegate 3.x.
