UI Quickstart

Follow the New Evaluation wizard to run your first AI evaluation in Harness AI Evals. No code required.

The fastest way to your first passing evaluation is the Create Evaluation wizard. It walks you through five short screens, then offers to auto-create a pipeline and run your eval on the spot.

The wizard covers each of the five steps in order.

The example evaluates a customer support agent that answers ecommerce questions. Substitute your own target everywhere it appears.

---

Open the wizard

Complete these three actions to reach the wizard.

1. In your Harness project, select AI Evals in the left nav.

2. Select Evaluations > Create Evaluation.

3. The wizard opens on the Basic info step.

---

Step 1: Basic info

Give the evaluation a name and a short description.

Field	Value
Name	Support Agent Quickstart
Description	(optional) e.g. First eval for the customer support agent

Select Continue.

---

Step 2: Target

The target is the AI system you are evaluating. You can pick an existing one or create a new one inline.

Select Select Existing and choose your registered target from the dropdown. If this is your first target, select Create New and fill in the target form:

Field	Value
Name	Support Agent
Type	Agent — called App (agent) in the API. Other options: Prompt (prompt), Precomputed — called Static (static) in the API.
Endpoint	https://your-agent.example.com/api/eval
Method	POST
Request template	{ "input": "{{input.content}}" }
Response path	output

Use the Test Target panel to send a sample input and confirm the agent returns a real answer. Select Continue.

request_template is not optional. Without it, the runner double-wraps row.input. Your agent sees {input: {content: "..."}} instead of {input: "..."}, and every row fails with HTTP 400.

---

Step 3: Dataset

The dataset is your ground truth. Each row is a scenario the agent will be tested against.

Select Select Existing to pick a dataset, or Create New to add one inline. You can add rows three ways:

• Paste JSONL in the inline editor.

• Upload a .jsonl file for bulk imports.

• Generate synthetic rows with an LLM. Pick a strategy (adversarial, rephrase, complexity_ladder, or use_case), describe the use case, and review the generated rows before saving.

For this quickstart, paste this in the JSONL editor:

{"id": "iphone-stock",   "input": {"content": "Do you have iPhone 15 Pro in stock?"}, "expected_output": {"text": "yes"}}
{"id": "macbook-price",  "input": {"content": "How much does the MacBook Pro 14-inch cost?"}, "expected_output": {"text": "$1,999"}}
{"id": "shipping",       "input": {"content": "Do you offer free shipping?"}, "expected_output": {"text": "shipping"}}

Select Continue.

Field rules. input must be a JSON object, not a bare string. expected_output for ContainsMetric must be {"text": "..."}. A bare string silently fails.

---

Step 4: Metric set

The metric set defines what "good" means for your agent. Pick a few metrics from the built-in catalog. Filter by type (heuristic, LLM judge, embedding, AI judge) to narrow the list.

Select Create New and name the metric set (for example, Support Quality). Add three metrics that cover writing quality, tone, and outcome.

Metric	Threshold	What it checks
NoTyposMetric	0.8	The output is grammatically clean.
PolitenessMetric	0.8	The response is professional and courteous.
TaskCompletionMetric	0.7	The agent actually answered the question, not something adjacent.

These are LLM-as-a-judge metrics, so select a Judge LLM Connector at the top of the form. Any OpenAI or Anthropic connector configured in your project works.

Select Continue.

Prefer no judge model? Start with heuristic metrics like ContainsMetric and ExactMatchMetric. They score for free with no connector needed. Layer in LLM-judge metrics once your first run works.

---

Step 5: Configuration

The last screen controls how the run behaves. The defaults work for most first runs.

Setting	Default	What it does
Sampling strategy	All Items	Run every row in the dataset. Use first N or random N for smoke tests on large datasets.
Concurrency	5	How many rows run against your agent in parallel.
Timeout per item	30000 ms	How long a single row can take before failing.
Cost limit	none	Optional hard stop if judge tokens exceed a dollar amount.

Select Save & Continue.

---

Create the pipeline and run

Once the eval is saved, a modal appears with two options.

• Create Pipeline & Run. Harness bundles the evaluation into a Suite, generates a default CI/CD pipeline, attaches your eval as an AI Eval step, and starts the first run right away.

• Skip for now. The eval is saved for later. You can trigger runs manually from the eval detail page.

Select Create Pipeline & Run. The run kicks off in a minute or two and forwards results to the Results page as it completes.

---

You just shipped your first AI quality gate. Every time you push code, this eval runs automatically and blocks deployment if quality drops. No more guessing whether your agent still works after a prompt tweak or a model swap.

Prefer to build resources one at a time? You can also register a target, dataset, and metric set from their own dedicated tabs (Targets, Datasets, Scorers) and then wire them together in Evaluations. Useful when you want to reuse existing resources across many evaluations.