> For the complete documentation index, see [llms.txt](https://developer.harness.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://developer.harness.io/self-managed-enterprise-edition/troubleshooting-and-resources/monitoring/enabling-tracing.md).

# Enabling Tracing for Harness Leveraging OpenTelemetry

### Enable Tracing in Harness Services <a href="#enable-tracing-in-harness-services" id="enable-tracing-in-harness-services"></a>

The following Harness services(Platform, CD, CI, IACM, SSCA) support OpenTelemetry tracing:

* access-control, harness-manager, ng-manager, platform-service, pipeline-service, ci-manager, sto-manager, cv-nextgen, iacm-manager, audit-event-streaming, debezium-service, template-service, idp-service, ssca-manager.

#### Common Global Configuration <a href="#common-global-configuration" id="common-global-configuration"></a>

```yaml
global:
  monitoring:
    otel:
      enabled: true
      collectorEndpoint: http://opentelemetry-collector.otel.svc.cluster.local:4317/
```

### Overview of Tracing Architecture <a href="#overview-of-tracing-architecture" id="overview-of-tracing-architecture"></a>

The tracing setup for Harness leverages OpenTelemetry to capture distributed traces across services, enabling deeper observability and troubleshooting.

<figure><img src="/files/JggrCd5miK1ExoqLhn4H" alt="Tracing Arch Placeholder"><figcaption><p>Overview of Tracing Architecture for Harness Self-Managed Enterprise Edition.</p></figcaption></figure>

The architecture follows this flow:

1. Harness Instance with OpenTelemetry Agents: Each service in the Harness cluster is instrumented with OpenTelemetry agents to generate trace data.
2. OpenTelemetry Collector: Trace data is sent to the collector, which processes, batches, and exports it to supported backends.
3. Grafana Tempo: The collector forwards the traces to Grafana Tempo, a high-scale distributed tracing backend.
4. MinIO: Tempo stores the trace data in an object store, such as MinIO, for durability and querying.

### Install Grafana Tempo (Optional) <a href="#install-grafana-tempo-optional" id="install-grafana-tempo-optional"></a>

Grafana Tempo is used to store and query trace data collected via OTel.

#### Step 1: Install MinIO for Trace Storage (Optional) <a href="#step-1-install-minio-for-trace-storage-optional" id="step-1-install-minio-for-trace-storage-optional"></a>

* Create a file named `minio.yaml`:

  ```yaml
  fullnameOverride: "minio"
  mode: standalone
  provisioning:
    enabled: true
    buckets:
    - name: tempo
      region: us-east-1
      lifecycle:
        - id: 7dRetention
          expiry:
            days: 7
            nonconcurrentDays: 3
      tags:
        owner: tempo
  persistence:
    size: 20Gi
  auth:
    rootUser: admin
    rootPassword: "admin123"
  ```
* Install MinIO:

  ```bash
  helm repo add bitnami https://charts.bitnami.com/bitnami
  helm install minio bitnami/minio -f minio.yaml -n tempo --create-namespace
  ```

#### Step 2: Install Tempo Distributed <a href="#step-2-install-tempo-distributed" id="step-2-install-tempo-distributed"></a>

* Create a file `tempo.yaml` with appropriate overrides.
* Install Tempo:

  ```bash
  helm repo add grafana https://grafana.github.io/helm-charts
  helm install tempo-distributed grafana/tempo-distributed -f tempo.yaml -n tempo
  ```

### Configure Grafana with Tempo as Trace Datasource <a href="#configure-grafana-with-tempo-as-trace-datasource" id="configure-grafana-with-tempo-as-trace-datasource"></a>

<figure><img src="/files/XdbBFUTAX5pD9CsQVN1q" alt="Configure-Tempo-In-Grafana"><figcaption><p>Configure Tempo in Grafana</p></figcaption></figure>

In Grafana:

* Go to **Settings > Data Sources**
* Add a new **Tempo** data source
* If everything is deployed in the same namespace, set the URL as:

  ```http
  http://tempo-distributed-gateway.tempo.svc.cluster.local:80/
  ```

### Install OpenTelemetry Collector (Optional) <a href="#install-opentelemetry-collector-optional" id="install-opentelemetry-collector-optional"></a>

This step allows fine-grained control over trace ingestion, processing, and exporting.

#### Step 1: Create a Collector Override <a href="#step-1-create-a-collector-override" id="step-1-create-a-collector-override"></a>

Create a file `otel-collector.yaml` and define the receivers, exporters, and pipelines.

#### Step 2: Install Collector <a href="#step-2-install-collector" id="step-2-install-collector"></a>

```bash
  helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
  helm install opentelemetry-collector open-telemetry/opentelemetry-collector -f otel-collector.yaml -n otel --create-namespace
```

### Visualize Traces in Grafana <a href="#visualize-traces-in-grafana" id="visualize-traces-in-grafana"></a>

Once traces flow from Harness services through the OTel Collector and are stored in Tempo, you can use the **Traces** tab in Grafana to:

* Query traces by service name
* Explore duration, calls, and trace sizes
* Debug performance bottlenecks and error spikes

<figure><img src="/files/q4SVKiKRenzP83N8BTHQ" alt="Example 1a"><figcaption><p>Example of browsing traces</p></figcaption></figure>

<figure><img src="/files/SiqIwnwYWJGGMeQFoIoA" alt="Example 1b"><figcaption><p>Example of browsing traces</p></figcaption></figure>

### Summary <a href="#summary" id="summary"></a>

This setup provides end-to-end distributed tracing visibility into Harness using:

* OpenTelemetry for instrumentation
* Tempo for trace storage and query
* Grafana for visualization

With minimal effort, you now gain deep insight into request flows across services, aiding performance debugging and reliability engineering.