What's Supported

Platforms, targets, metrics, and integrations supported by Harness AI Evals

Harness AI Evals integrates with a wide range of target types, metrics, LLM providers, and CI/CD platforms. This page outlines all supported integrations, providers, and features available in Harness AI Evals.

For information about what is supported for other Harness modules and the Harness Platform overall, go to Supported platforms and technologies.

---

1. Target types

AI Evals evaluates three types of AI systems:

Prompt targets

Single LLM call with a prompt template

Harness calls the LLM with your system message and user input.

Supported providers:

• OpenAI (GPT-4, GPT-3.5 Turbo)

• Anthropic (Claude 3 Opus, Sonnet, Haiku)

• Azure OpenAI

• Google Vertex AI (Gemini)

Use case: Testing prompt templates before deployment, rapid iteration on system messages

App targets

Any HTTP endpoint (black box evaluation)

Harness sends HTTP requests to your agent or RAG pipeline.

Supported configurations:

• Any publicly accessible HTTPS endpoint

• Custom request templates (map dataset input to your API format)

• Private endpoints behind VPC/firewall (contact support)

Use case: Evaluating deployed agents, RAG pipelines, multi-step systems

Static targets

Precomputed outputs from file

Upload outputs that were generated externally.

Supported format:

• JSONL (one JSON object per line)

• Uploaded via UI or API

Use case: Comparing model versions offline, scoring outputs without re-running inference

---

2. Metrics (47+ built-in)

By category

Deterministic (free, no LLM judge)

• ExactMatch: Output exactly matches expected

• Contains: Output contains expected substring

• Regex: Output matches regular expression

• NumericDiff: Numeric accuracy within threshold

• JsonDiff: JSON structure and values match

• SchemaValidation: JSON validates against schema

Similarity (embedding-based)

• BLEU: Translation quality metric

• Levenshtein: Edit distance

• EmbeddingSimilarity: Cosine similarity between embeddings

LLM-as-a-judge (requires Judge connector)

• GEval: General evaluation with custom rubric

• RubricJudge: Multi-criteria rubric scoring

• Pairwise: Compare two outputs side by side

• PromptAlignment: Output matches prompt intent

• TaskCompletion: Task was completed successfully

• PolitenessMetric: Professional and courteous tone

• ToneMetric: Tone matches target (formal, casual, etc.)

• ClarityMetric: Clear and easy to understand

RAG (Retrieval-Augmented Generation)

• Faithfulness: Answer is grounded in retrieved context

• AnswerRelevancy: Answer addresses the question

• ContextPrecision: Retrieved context is relevant

• ContextRecall: Retrieved context contains the answer

Safety

• PII: Detects personally identifiable information leakage

• Toxicity: Offensive or harmful content

• PromptInjection: Attempts to manipulate system behavior

• Hallucination: Confident but factually wrong statements

• Bias: Unfair treatment of groups

Agent / tool use

• ToolCorrectness: Called tools with correct arguments

• ToolSelectionAccuracy: Selected the right tools

• PlanAdherence: Followed the expected plan

Conversation (multi-turn)

• GoalAccuracy: Achieved conversation goal

• RoleAdherence: Stayed in character

• KnowledgeRetention: Remembered context from earlier turns

• TopicAdherence: Stayed on topic

Operational

• LatencyMetric: Response time within threshold

• TokenCostMetric: Token usage within budget

• CostEfficiencyMetric: Cost per quality point

Go to Metrics for the complete catalog with examples.

---

3. LLM providers (judge models)

For LLM-as-a-judge metrics, configure a Judge LLM Connector in your Harness project:

Provider	Models	Setup Guide
OpenAI	GPT-4, GPT-4 Turbo, GPT-3.5 Turbo	OpenAI connector
Anthropic	Claude 3 Opus, Sonnet, Haiku	Anthropic connector
Azure OpenAI	GPT-4, GPT-3.5 Turbo (Azure-hosted)	Azure OpenAI connector
Google Vertex AI	Gemini Pro, Gemini Ultra	Contact support

---

4. Dataset formats

Upload methods

Method	Format	Use Case
JSONL upload	One JSON object per line	Bulk import from existing test suites
Inline creation	Add rows via UI form	Quick testing with 5-10 cases
Synthetic generation	LLM-powered strategies	Generate edge cases and variations
API creation	REST API or MCP	Programmatic dataset management

Synthetic generation strategies

• Adversarial: Inputs designed to trick or confuse the agent

• Rephrase: Rephrase existing rows in different ways

• Complexity ladder: Progressively harder versions of a base case

• Use case expansion: Generate cases covering a range of scenarios

Required fields

{
  "input": {"content": "What is 2+2?"},
  "expected_output": {"text": "4"},
  "expected_tools": ["calculator"],
  "context": ["Math textbook page 42..."]
}

---

5. CI/CD integrations

Harness Pipelines (native)

• AI Eval step: Native step type in CI/CD pipelines

• Suite-based gating: Block deployments on eval suite results

• Pass strategies: all_must_pass (blocks on any failure), weighted_threshold (advisory signal)

Python SDK (any CI system)

The harness-evals SDK runs in any environment that supports Python 3.10+:

pytest

from harness_evals import Evaluation

def test_my_agent():
    eval = Evaluation.from_registry("my-eval")
    results = eval.run()
    assert results.pass_rate >= 0.8

GitHub Actions

- name: Run AI Evals
  run: |
    pip install harness-evals
    harness-evals run --suite my-suite

GitLab CI

test:
  script:
    - pip install harness-evals
    - harness-evals run --suite my-suite

Works in: GitHub Actions, GitLab CI, Jenkins, CircleCI, Travis CI, Bitbucket Pipelines, any system that runs Python

---

6. Observability integrations

Harness AgentTrace

Native integration for production trace ingestion:

• Trace ingestion: OpenTelemetry spans via OTLP protocol

• Trace viewer: Inspect LLM calls, tool invocations, costs, latency

• Promote to dataset: Convert production traces into test cases

• Online evaluation: Score sampled production traffic

Third-party tracing

AI Evals ingests traces from:

• Langfuse (via OTLP export)

• Custom OTLP exporters (any OpenTelemetry-compatible system)

---

7. API and tooling access

REST API

Full CRUD operations for all resources:

• Targets, Datasets, Metrics, Metric Sets: Create and manage evaluation components

• Evaluations, Runs, Suites: Execute and monitor evaluation runs

• Authentication: Via Harness PAT (Personal Access Token)

Go to REST API Quickstart for examples.

MCP (Model Context Protocol)

Drive evals from AI assistants:

• Supported clients: Claude Code, Cursor, any MCP client

• Requirements: harness-mcp-v2 v2.9.7+ with ai-evals toolset enabled

Go to MCP Quickstart for setup.

---

8. Beta features and roadmap

Beta features

The following features are in beta. Contact Harness Support to enable them for your account:

• Private endpoint targets: Evaluate agents behind VPC or firewall

• Custom metric SDK: Write custom metrics in Python

• Multi-region datasets: Store datasets in EU or APAC regions

Recently released

Go to the AI Evals release notes for the latest updates.

Roadmap

Go to the AI Evals roadmap for planned features.

---

Next steps

• Go to Get Started to build your first evaluation.

• Go to Core concepts to understand the data model.

• Contact Harness Support for platform questions or to request new integrations.