Skip to main content

LiteLLM Proxy

Last updated on

"LiteLLM" refers to two different products, and how you send its cost data to Harness depends on which one you run:

  • LiteLLM proxy: A standalone server that routes LLM traffic for your applications. It has OpenTelemetry export built in, so you instrument the proxy once and every call routed through it emits a trace, with no application code changes. This page covers the proxy.
  • LiteLLM SDK: The Python library called directly in application code (import litellm). There is no proxy to configure, so you instrument the library itself. Go to the Harness SDK integration to set it up. This page does not cover the SDK.

The proxy is configured the same way whether or not it was already exporting traces:

Your setupWhat you do
The proxy already exports OpenTelemetry traces to another backend.Enable the otel callback if it is not on already, then change the endpoint and headers to the Harness values in the setup below.
The proxy does not export traces yet.Enable the otel callback and set the Harness endpoint variables, as shown in the setup below.
You call the LiteLLM SDK in Python (import litellm), not a proxy.Go to the Harness SDK integration instead.

Before you start, generate an ingestion token and connect a provider. Go to the AI Cost Management Quickstart for those steps. Replace <YOUR_TOKEN> below with that token and adjust the endpoint for your Harness cluster.

GenAI semantic conventions required

Cost is calculated from OpenTelemetry traces with GenAI semantic conventions. Go to the GenAI Span Attribute Reference to review the attributes CACM reads.

Instrument the LiteLLM proxy​

LiteLLM proxy emits OTel traces for every LLM call via its otel callback.

Enable the callback in config.yaml:

litellm_settings:
  callbacks: ["otel"]

Set these environment variables on the proxy, then restart it:

export OTEL_EXPORTER="otlp_http"
export OTEL_ENDPOINT="https://app.harness.io/udp-ingest/otel/v1/traces"
export OTEL_HEADERS="Authorization=Bearer <YOUR_TOKEN>"
LiteLLM proxy uses its own variable names

OTEL_EXPORTER, OTEL_ENDPOINT, and OTEL_HEADERS are LiteLLM-proxy-specific variable names, not the standard OpenTelemetry SDK variables (OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_HEADERS) used for other integrations. Use these exact names only for the LiteLLM proxy.

Alternative configuration with standard OTel variables

Some LiteLLM proxy versions emit traces with the otel callback and the standard OpenTelemetry exporter variables instead of the proxy-specific names above. If traces do not appear with the variables above, enable the callback and use the standard variables instead:

export OTEL_EXPORTER_OTLP_ENDPOINT=https://app.harness.io/udp-ingest/otel
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer <YOUR_TOKEN>"
export OTEL_TRACES_EXPORTER=otlp
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf

Once the proxy restarts, every request routed through it emits traces to Harness.

What this produces:

  • One span per LLM call routed through the proxy.
  • Span attributes: gen_ai.system, gen_ai.request.model, token usage.
  • Cost calculated from token counts and model pricing.
Proxy instrumentation covers only proxied traffic

This instruments the proxy, not the application code. If the application makes direct LLM calls that bypass the proxy, those calls do not appear in telemetry. Instrument those calls with the Harness SDK.

Verify Traces in Cost Explorer​

  1. Send a request through the proxy so traces flow. They usually appear within a few minutes; allow up to about 20 minutes.
  2. Go to Cloud & AI Cost Management > Cost Explorer.
  3. Select the AI Traces view or group by Service Name, and find your service.
  4. Select a service row to open the Service Traces drawer and inspect the span waterfall.

Next Steps​