> For the complete documentation index, see [llms.txt](https://developer.harness.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://developer.harness.io/continuous-delivery/use-gitops/gitops-entities/agents/gitops-agent-metrics.md).

# GitOps Agent metrics

Enable and scrape GitOps Agent metrics for monitoring agent performance and health.

GitOps Agent metrics expose performance and health data for the agent on an internal endpoint, so you can scrape and monitor them with tools like Prometheus.

{% hint style="info" %}
**Agent version requirement**

All new metrics are available only from GitOps Agent version 0.91.0 and later. If you cannot find or use GitOps Agent metrics, confirm your agent runs version 0.91.0 or later.
{% endhint %}

## What you will learn from this topic <a href="#what-you-will-learn-from-this-topic" id="what-you-will-learn-from-this-topic"></a>

* How to [expose and scrape](#exposing-metrics-for-scraping) GitOps Agent metrics through the agent's metrics endpoint.
* Understand the [full list of available metrics](#available-metrics) that the GitOps Agent exposes, including advanced task-processing metrics.
* How to interpret [histogram vectors](#understanding-histogram-metrics) and [task-processing time metrics](#useful-information-about-task-metrics) for debugging scaling issues.
* How to use example [PromQL queries](#example-prometheus-queries-promql) for monitoring HTTP request duration, payload size, and task execution time.

## Before you begin <a href="#before-you-begin" id="before-you-begin"></a>

Before you begin, make sure you have the following:

* **GitOps Agent version**: Metrics in this topic require GitOps Agent version 0.91.0 or later.
* **Access to agent installation configuration**: You need access to the Helm overrides file or the Kubernetes manifests used to install the GitOps Agent, depending on how you install it.

## Overview <a href="#overview" id="overview"></a>

The GitOps Agent exposes internal metrics on port `2112` at the `/metrics` endpoint. These metrics are useful for monitoring the performance and health of the agent. Go to [Prometheus](https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/user-guides/running-exporters.md) to review how to scrape metrics using an exporter tool.

Metrics are enabled by default in the GitOps Agent and exposed through a Kubernetes Service:

* Metrics are served from the agent's container on port `2112` at `/metrics`.
* If these metrics are not exposed by default, you must explicitly [expose](#exposing-metrics-for-scraping) this port.
* All metrics reset when the GitOps Agent pod restarts.

## Expose metrics for scraping <a href="#exposing-metrics-for-scraping" id="exposing-metrics-for-scraping"></a>

The default agent installation already exposes metrics. The following configuration controls how metrics are exposed for the agent.

### Values overrides for Helm-based agent installation

Perform the following steps to enable metrics through Helm overrides:

1. **Enable metrics service** by setting the following value in your Helm overrides file:

   ```yaml
   agent:
     metrics:
       enabled: true
   ```
2. **Enable Prometheus ServiceMonitor** (if you are using Prometheus Operator) by setting the following value:

   ```yaml
   agent:
     metrics:
       serviceMonitor:
         enabled: true
   ```

{% hint style="info" %}
**Avoid a `LoadBalancer` type service with multiple agent pods**

Avoid using a `LoadBalancer` type service if you are running multiple agent pods. In such cases, using a `ServiceMonitor` ensures metrics from all replicas are correctly scraped. Go to the [Prometheus Operator](https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/user-guides/running-exporters.md) guide to review running exporters.
{% endhint %}

Alternatively, you can pass the values directly during the Helm installation:

```bash
helm install harness-agent <chart-path> \
  --set agent.metrics.enabled=true \
  --set agent.metrics.serviceMonitor.enabled=true
```

### For an agent installed through Kubernetes manifests directly

To expose the metrics port (`2112`) manually, apply the following Kubernetes Service manifest in the same namespace as your GitOps Agent:

```yaml
apiVersion: v1
kind: Service
metadata:
  name: gitops-agent-metrics
  namespace: <AGENT_NAMESPACE>
  labels:
    app.kubernetes.io/name: harness-gitops-agent-metrics
    app.kubernetes.io/component: gitops-agent
    app.kubernetes.io/part-of: harness-gitops
spec:
  type: ClusterIP
  ports:
    - name: http-metrics
      protocol: TCP
      port: 2112
      targetPort: metrics
  selector:
    app.kubernetes.io/name: harness-gitops-agent
```

Once the service is created, you can test locally by port-forwarding:

```bash
kubectl port-forward svc/gitops-agent-metrics 2112:2112 -n <AGENT_NAMESPACE>
```

Visit `http://localhost:2112/metrics` to confirm the metrics endpoint is accessible.

## Available metrics <a href="#available-metrics" id="available-metrics"></a>

The following table lists each metric that the GitOps Agent exposes, with its type, description, and minimum agent version.

| Metric Name                                            | Type      | Description                                                                            | Min. Req. Agent Version |
| ------------------------------------------------------ | --------- | -------------------------------------------------------------------------------------- | ----------------------- |
| `entities_count`                                       | Gauge     | Number of entities being managed by the GitOps Agent, grouped by label "entity\_type". | v0.91.0                 |
| `task_complete_processing_time`                        | Histogram | Time a task spends in the agent from fetch to response.                                | v0.91.0                 |
| `http_request_duration`                                | Histogram | Time taken to complete HTTP requests to Harness.                                       | v0.91.0                 |
| `http_request_payload_size`                            | Histogram | Size (in bytes) of payloads sent in HTTP requests.                                     | v0.91.0                 |
| `reconciler_duration` (ADVANCED)                       | Histogram | Time taken for one reconciliation cycle (in milliseconds).                             | v0.91.0                 |
| `task_receiver_channel_blocked_time` (ADVANCED)        | Histogram | Wait time before task is picked up by a processor.                                     | v0.91.0                 |
| `task_execution_time` (ADVANCED)                       | Histogram | Time taken to execute a task (includes Argo CD work).                                  | v0.91.0                 |
| `task_response_sender_channel_blocked_time` (ADVANCED) | Histogram | Time a completed task waits before sending response.                                   | v0.91.0                 |

{% hint style="info" %}
**When to use advanced metrics**

The "ADVANCED" metrics become useful when you run into non-obvious scaling issues. They help you identify which component is the bottleneck so you can scale workloads accordingly. You can resolve most basic scaling issues with a combination of high availability (HA) mode, HPA, and spec increases.
{% endhint %}

### Task metrics <a href="#useful-information-about-task-metrics" id="useful-information-about-task-metrics"></a>

The GitOps Agent has three concurrent worker pools controlled by the environment variables `GITOPS_AGENT_NUM_FETCHERS`, `GITOPS_AGENT_NUM_PROCESSORS`, and `GITOPS_AGENT_NUM_RESPONDERS`. The **FETCHER** fetches a task from Harness, hands it to a **PROCESSOR**, and then the result is picked up by a **RESPONDER** and sent back to Harness.

`task_receiver_channel_blocked_time` observes the time tasks spend within the agent waiting for a **PROCESSOR** worker to pick them up.

`task_response_sender_channel_blocked_time` observes the time a completed task spends within the agent waiting for a **RESPONDER** worker to send its response back to Harness.

`task_execution_time` is the time a **PROCESSOR** worker actually spent on the task. It is the time between the above two channels, once a **PROCESSOR** picks up a task.

`task_complete_processing_time` is the overall time spent by the task within the agent. This is the sum of the above three metrics:

`task_receiver_channel_blocked_time + task_execution_time + task_response_sender_channel_blocked_time`

This sum usually equals `task_complete_processing_time` for each task.

You can use this information to debug instances where the reason the GitOps Agent is taking too long is not obvious.

For example, if there are sporadic issues with tasks not completing properly and there is a spike in `task_response_sender_channel_blocked_time`, you can increase `GITOPS_AGENT_NUM_RESPONDERS` so that there are more **RESPONDER** workers and task results are not blocked from being sent to Harness.

### Histogram metrics <a href="#understanding-histogram-metrics" id="understanding-histogram-metrics"></a>

Most metrics are histograms and include these vectors corresponding to each metric:

`_count`: Number of samples.

`_sum`: Sum of all samples.

`_bucket`: Buckets representing value ranges.

To calculate averages, for example:

```promql
rate(task_complete_processing_time_sum[5m]) / rate(task_complete_processing_time_count[5m])
```

## Example Prometheus queries (PromQL) <a href="#example-prometheus-queries-promql" id="example-prometheus-queries-promql"></a>

Based on the metrics described above, you can configure alerts using the following PromQL queries.

### Average HTTP request duration

Use the following query to calculate the average HTTP request duration:

```promql
rate(http_request_duration_sum[5m]) / rate(http_request_duration_count[5m])
```

### 95th percentile HTTP payload size

Use the following query to calculate the 95th percentile HTTP payload size:

```promql
histogram_quantile(0.95, rate(http_request_payload_size_bucket[5m]))
```

### 95th percentile task execution time

Use the following query to calculate the 95th percentile task execution time:

```promql
histogram_quantile(0.95, rate(task_execution_time_bucket[5m]))
```

### 95th percentile receiver channel block time (Advanced)

Use the following query to calculate the 95th percentile receiver channel block time:

```promql
histogram_quantile(0.95, rate(task_receiver_channel_blocked_time_bucket[5m]))
```

**Tip**: If this increases under load, scale `GITOPS_AGENT_NUM_PROCESSORS`. Recommended max: 16 for agents with 1 vCPU.

### 95th percentile responder channel block (Advanced)

Use the following query to calculate the 95th percentile responder channel block time:

```promql
histogram_quantile(0.95, rate(task_response_sender_channel_blocked_time_bucket[5m]))
```

**Tip**: If this increases under load, scale `GITOPS_AGENT_NUM_RESPONDERS`.

## Next steps <a href="#next-steps" id="next-steps"></a>

* [GitOps Agent](/continuous-delivery/use-gitops/gitops-entities/agents.md): Review other agent configuration options.
* [Sync GitOps applications](/continuous-delivery/use-gitops/application/sync-gitops-applications.md): Understand how syncs generate the task load that these metrics measure.

{% @harness-feedback/feedback %}