For the complete documentation index, see llms.txt. Markdown versions of documentation pages are available by appending .md to page URLs.
Skip to content

OpenTelemetry Observability

Export Delegate 3.x logs and metrics to your own observability stack using OpenTelemetry

Delegate 3.x can export logs and metrics using OpenTelemetry (OTEL), allowing you to route delegate telemetry directly into your own observability stack, such as Splunk, Datadog, or Amazon CloudWatch.

What you will learn from this topic

Before you begin

Before you enable OTEL export, make sure you have:

  • A running OTEL Collector that is reachable from the delegate.

  • The OTLP endpoint for the OTEL Collector.

  • The required configuration for your OTEL Collector and destination backend.

Delegate observability before OTEL

Without OTEL support, Delegate 3.x offers two ways to observe delegate activity, and both have gaps for self-hosted deployments:

  • GCP Cloud Logging: built-in, but only works for delegates running on Harness-hosted infrastructure.

  • The /metrics endpoint: works everywhere, but requires you to run and maintain your own Prometheus scraper.

For on-premises and local binary deployments in particular, these options do not provide a native path for exporting delegate logs and metrics to your existing observability stack.

OTEL export configuration and behavior

The delegate ships logs and metrics to an OTEL Collector over OTLP (gRPC or HTTP). The collector then routes the data to your backend of choice.

  • Log shipping: The delegate captures all log levels and forwards them as OTEL logs using OTLP.

  • Metric shipping: Existing Prometheus metrics are bridged to the OTEL SDK and exported over OTLP, so you get the same metrics the /metrics endpoint exposes, delivered through OTEL instead.

  • Backend-agnostic transport: The delegate speaks OTLP only, over gRPC or HTTP (HARNESS_OTEL_PROTOCOL). There is no delegate-side setting for the destination backend. Backend selection, such as Splunk, Datadog, or Amazon CloudWatch, happens at the OTEL Collector the delegate points to.

  • Configuration: You enable and configure OTEL export entirely through environment variables, with no code changes required.

  • Backward compatibility: GCP Cloud Logging is controlled by its own ENABLE_REMOTE_LOGGING flag, independent of OTEL export. The /metrics endpoint always stays on as a fallback, regardless of your OTEL settings.

The delegate does not send telemetry directly to every observability backend. It exports telemetry to the configured OTEL endpoint, and the OTEL Collector handles processing and routing to the destination backend.

Delegate OTEL export architecture

A sidecar-based exporter can still complement this integration for deployments that already standardize on a sidecar pattern. The native integration described in this topic works across all deployment models, including local binaries, without requiring a sidecar.

Enable OTEL export

Enable OTEL export by adding the following variables to the delegate's config.env:

  • HARNESS_OTEL_ENABLED: Enables OTEL export when set to true. Default: false.

  • HARNESS_OTEL_ENDPOINT: The OTEL Collector endpoint that receives exported telemetry. The endpoint must be reachable from the delegate. For example, collector:4317 uses the standard OTLP/gRPC port. No default; required when OTEL export is enabled.

  • HARNESS_OTEL_PROTOCOL: The OTLP transport protocol. Supported values are grpc and http. Default: grpc.

  • HARNESS_OTEL_INSECURE: Skips TLS certificate verification when set to true. Default: true. Set to false and configure the collector with a valid certificate for a secure connection.

  • HARNESS_OTEL_EXPORT_LOGS: Enables log export when set to true. Default: true.

  • HARNESS_OTEL_EXPORT_METRICS: Enables metric export when set to true. Default: true.

Optionally, set the following variables to control the resource attributes attached to exported telemetry:

  • HARNESS_OTEL_SERVICE_NAME: The service name attached to exported telemetry. Default: harness-delegate.

  • HARNESS_OTEL_SERVICE_VERSION: The service version attached to exported telemetry. No default.

  • HARNESS_OTEL_ENVIRONMENT: The deployment environment label attached to exported telemetry, such as production. No default.

OTEL environment variables

Set HARNESS_OTEL_EXPORT_LOGS and HARNESS_OTEL_EXPORT_METRICS independently of each other, depending on whether you want to export logs, metrics, or both.

Build an observability dashboard

Once logs and metrics are flowing into your observability stack, you can build a dashboard to monitor delegate health, such as log volume by severity, error and warning counts, and metric throughput.

Example delegate observability dashboard

Troubleshoot OTEL export

Telemetry does not appear in the observability backend

If telemetry does not appear in your observability backend, check the following:

  • HARNESS_OTEL_ENABLED is set to true.

  • HARNESS_OTEL_PROTOCOL is set to a supported value (grpc or http).

  • HARNESS_OTEL_EXPORT_LOGS and HARNESS_OTEL_EXPORT_METRICS are set to true for the telemetry you want to export.

  • The endpoint in HARNESS_OTEL_ENDPOINT is reachable from the delegate.

  • If the collector requires TLS, HARNESS_OTEL_INSECURE is set to false and the delegate trusts the collector's certificate.

  • The OTEL Collector is running and accepting OTLP traffic.

  • The OTEL Collector is configured to route telemetry to the expected backend.

  • OTEL SDK errors are surfaced through the delegate's error handler into its logs. Review delegate logs for OTEL export errors.

Next steps

  • Delegate Configuration Reference: Go to the Metrics section to compare the OTEL export with the built-in /metrics endpoint.

  • Overview: Go to the Logging and observability section for background on structured logging in Delegate 3.x.

Last updated

Was this helpful?