Get Started with AI Evals

Set up prerequisites and build your first eval in Harness AI Evals, either from the UI or with the harness-evals SDK.

Set up your project, pick your path, and build your first eval. Choose the UI for a no-code walkthrough, the REST API for direct integration, or MCP for AI assistant automation.

---

Before you begin

• Harness account: Access to a Harness project with AI Evals enabled. Go to RBAC in Harness to confirm you have project-level access. Go to Manage projects if you need to create a new project.

• LLM provider API key: An API key from OpenAI, Anthropic, or another LLM provider. Required for LLM-as-a-judge metrics. Go to Add and reference text secrets to learn how to store API keys securely.

• Harness pipeline: A pipeline with a CI or CD stage if you plan to gate deployments on eval suites. Optional for local SDK runs. Go to Get started with Harness CI or CD & GitOps tutorials to create your first pipeline.

• Python 3.10 or later: Required for the harness-evals SDK and CLI path. Skip if you only use the UI.

---

Connect to the AI Evals control plane

Complete these steps once per project before you build your first eval.

1. Add your LLM provider key as a Harness Secret. Go to Project Settings > Secrets > + New Secret > Text. Name the secret to match how your evals reference it, for example OPENAI_API_KEY, and paste the API key into the Value field. Go to Add and reference text secrets for detailed steps.

2. Create an LLM Connector. Go to Project Settings > Connectors > + New Connector and choose your provider (OpenAI, Anthropic, Azure OpenAI, or Google Vertex AI). Reference the secret you created in step 1 as the API key source, test the connection, and save. Go to OpenAI model connector or Anthropic model connector for detailed setup steps.

3. Verify project-level AI Evals access. Open the AI Evals module in the left navigation and confirm you can see the Targets, Datasets, Scorers, Evaluations, Suites, Observe, Analytics, and Registry tabs. If you don't see the module, contact your account administrator to enable AI Evals for your project.

---

Choose your path

Pick the path that matches how you work. All three paths cover the same four-resource model (Target, Dataset, Metric Set, Evaluation, Run).

• UI Quickstart

• REST API Quickstart

• MCP Quickstart

Next steps

Learn the fundamentals

• Go to Core concepts to understand targets, datasets, metric sets, evaluations, and suites in depth.

• Go to Targets to learn about Prompt, App, and Static target types.

• Go to Datasets to learn dataset design best practices and synthetic generation.

• Go to Metrics to explore the 47+ built-in metrics across 9 categories.

Integrate with CI/CD

• Go to Pipeline integration to gate a Harness pipeline on an eval suite.

• Go to Create your first pipeline if you need to set up CI/CD first.

• Go to Policy as Code to learn about governance and quality gates.

Advanced evaluation

• Go to Multi-turn and agentic evals to evaluate conversations and tool-use trajectories.

• Go to Observability and analytics to inspect production traces and track pass-rate trends.

Platform essentials

• Go to Harness Platform key concepts to understand how AI Evals fits into the Harness ecosystem.

• Go to Manage roles to configure permissions for your team.

• Go to Harness secrets management to learn about secure credential storage.