REST API Quickstart

Integrate AI Evals into any tool or CI/CD pipeline using direct HTTP requests.

Integrate AI Evals into any tool or CI/CD pipeline using raw HTTP. Best for non-Harness environments, custom tooling, and language flexibility.

---

The four-resource model

Every eval boils down to four resources plus a run.

• Target: what to test. An HTTP agent endpoint, a prompt with a system message, or precomputed outputs.

• Dataset: the test cases. Each row has an input, and depending on scoring, optional expected_output, expected_tools, or context.

• Metric Set: how to score. A list of metrics with thresholds and weights.

• Evaluation: the wiring. Ties the three together plus run-time settings.

• Eval Run: the execution. Per-item scoring and aggregate results.

---

Authentication

1. Get API token: Harness > Account Settings > Personal Access Tokens > New Token

2. Base URL: https://app.harness.io/gateway/ai-evals/api/v1/orgs/{org}/projects/{project}/

3. Required headers:

   • x-api-key: {your-token}

   • Harness-Account: {your-account-id}

   • Content-Type: application/json

Test authentication:

curl -X GET \
  "https://app.harness.io/gateway/ai-evals/api/v1/orgs/default/projects/my_project/targets?page=0&size=5" \
  -H "x-api-key: pat.YOUR_ACCOUNT.YOUR_TOKEN" \
  -H "Harness-Account: YOUR_ACCOUNT_ID"

If you get JSON with { "data": [...], "page": 0, "total_elements": N }, authentication works.

---

Build your first evaluation

The example evaluates a flight support agent. Substitute your own endpoint everywhere it appears below.

1. Create the target

curl -X POST \
  "https://app.harness.io/gateway/ai-evals/api/v1/orgs/default/projects/my_project/targets" \
  -H "x-api-key: $TOKEN" \
  -H "Harness-Account: $ACCOUNT_ID" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "SkyBuddy",
    "type": "agent",
    "description": "Flight support agent",
    "config": {
      "endpoint": "https://your-agent.example.com/api/eval",
      "method": "POST",
      "request_template": {"input": "{{input.content}}"},
      "response_path": "output",
      "timeout_ms": 60000
    },
    "tags": ["skybuddy", "demo"]
  }'

Response:

{
  "id": "target-uuid",
  "name": "SkyBuddy",
  "type": "agent",
  "config": {},
  "created_at": "2026-07-16T10:00:00Z"
}

2. Create the dataset

curl -X POST \
  "https://app.harness.io/gateway/ai-evals/api/v1/orgs/default/projects/my_project/dataset" \
  -H "x-api-key: $TOKEN" \
  -H "Harness-Account: $ACCOUNT_ID" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "SkyBuddy Starter",
    "identifier": "skybuddy_starter",
    "description": "Basic flight queries and off-topic prompts"
  }'

3. Add dataset items

curl -X POST \
  "https://app.harness.io/gateway/ai-evals/api/v1/orgs/default/projects/my_project/dataset/{dataset-uuid}/items" \
  -H "x-api-key: $TOKEN" \
  -H "Harness-Account: $ACCOUNT_ID" \
  -H "Content-Type: application/json" \
  -d '{
    "item_identifier": "status-ha482",
    "input": {"content": "What is the status of flight HA482?"},
    "expected_tools": ["get_flight_status"]
  }'

Response:

{
  "id": "item-uuid-1",
  "item_identifier": "status-ha482",
  "input": {"content": "What is the status of flight HA482?"},
  "expected_tools": ["get_flight_status"],
  "created_at": "2026-07-16T10:02:00Z"
}

Add more items:

curl -X POST \
  "https://app.harness.io/gateway/ai-evals/api/v1/orgs/default/projects/my_project/dataset/{dataset-uuid}/items" \
  -H "x-api-key: $TOKEN" \
  -H "Harness-Account: $ACCOUNT_ID" \
  -H "Content-Type: application/json" \
  -d '{
    "item_identifier": "offtopic-joke",
    "input": {"content": "Tell me a joke"},
    "expected_output": {"text": "help"}
  }'

4. List available metrics

curl -X GET \
  "https://app.harness.io/gateway/ai-evals/api/v1/orgs/default/projects/my_project/metrics?page=0&size=50" \
  -H "x-api-key: $TOKEN" \
  -H "Harness-Account: $ACCOUNT_ID" \
  | jq '.data[] | select(.name | contains("ToolSelectionAccuracy") or contains("Contains"))'

Note the metric UUIDs from the response.

5. Create the metric set

curl -X POST \
  "https://app.harness.io/gateway/ai-evals/api/v1/orgs/default/projects/my_project/metric-sets" \
  -H "x-api-key: $TOKEN" \
  -H "Harness-Account: $ACCOUNT_ID" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "SkyBuddy Heuristics",
    "entries": [
      {
        "metric_id": "<ToolSelectionAccuracy UUID>",
        "threshold": 0.7,
        "weight": 1.0,
        "position": 0
      },
      {
        "metric_id": "<Contains UUID>",
        "threshold": 1.0,
        "weight": 1.0,
        "position": 1,
        "config": {"options": {"case_sensitive": false}}
      }
    ]
  }'

6. Create the evaluation

curl -X POST \
  "https://app.harness.io/gateway/ai-evals/api/v1/orgs/default/projects/my_project/evals" \
  -H "x-api-key: $TOKEN" \
  -H "Harness-Account: $ACCOUNT_ID" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "SkyBuddy Starter Eval",
    "target_id": "<target-uuid>",
    "dataset_id": "<dataset-uuid>",
    "metric_set_id": "<metric-set-uuid>",
    "concurrency": 3,
    "timeout_per_item_ms": 60000
  }'

7. Run it

curl -X POST \
  "https://app.harness.io/gateway/ai-evals/api/v1/orgs/default/projects/my_project/evals/{eval-uuid}/runs" \
  -H "x-api-key: $TOKEN" \
  -H "Harness-Account: $ACCOUNT_ID"

Response:

{
  "run_id": "run-uuid",
  "status": "running",
  "started_at": "2026-07-16T10:05:00Z"
}

8. Poll for completion

while true; do
  STATUS=$(curl -s \
    "https://app.harness.io/gateway/ai-evals/api/v1/orgs/default/projects/my_project/runs/{run-uuid}" \
    -H "x-api-key: $TOKEN" \
    -H "Harness-Account: $ACCOUNT_ID" | jq -r '.status')
  
  echo "Status: $STATUS"
  
  if [ "$STATUS" = "completed" ] || [ "$STATUS" = "failed" ]; then
    break
  fi
  
  sleep 30
done

9. Fetch results

# Get summary scores
curl -X GET \
  "https://app.harness.io/gateway/ai-evals/api/v1/orgs/default/projects/my_project/runs/{run-uuid}" \
  -H "x-api-key: $TOKEN" \
  -H "Harness-Account: $ACCOUNT_ID" \
  | jq '.summary_scores'

# Get per-item details
curl -X GET \
  "https://app.harness.io/gateway/ai-evals/api/v1/orgs/default/projects/my_project/runs/{run-uuid}/items?page=0&size=50" \
  -H "x-api-key: $TOKEN" \
  -H "Harness-Account: $ACCOUNT_ID"

---

What success looks like

A completed run returns aggregate scores plus per-item results:

{
  "run_id": "run-uuid",
  "status": "completed",
  "summary_scores": {
    "pass_rate": 0.83,
    "passed": 5,
    "failed": 1,
    "errored": 0
  },
  "dimension_scores": {
    "trajectory": 0.83,
    "correctness": 1.0
  }
}

Open the Evaluations tab in the AI Evals UI to see the same run with the full trace side by side.

---

Troubleshooting

Agent target returns HTTP 400 error with message about unexpected input structure

Without request_template on an agent target, the runner double-nests row input. Your agent receives {input: {input: ...}} instead of the expected structure and rejects with HTTP 400. Always specify the outgoing body shape explicitly on type: agent targets using the request_template field.

API returns 401 Unauthorized even with valid token

Verify both x-api-key and Harness-Account headers are set. The token format is pat.ACCOUNT_ID.TOKEN_STRING. Also check that the token has not expired in Harness Account Settings.

Dataset item creation fails with "input must be a JSON object" error

The input field must always be a JSON object, not a string. Use {"content": "your-text-here"} instead of passing the text directly. For example: {"input": {"content": "What is 2+2?"}} not {"input": "What is 2+2?"}.