> For the complete documentation index, see [llms.txt](https://developer.harness.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://developer.harness.io/ai-evals/get-started/get-started.md). # Get Started with AI Evals Set up prerequisites and build your first eval in Harness AI Evals, either from the UI or with the harness-evals SDK. Set up your project, pick your path, and build your first eval. Choose the UI for a no-code walkthrough, the REST API for direct integration, or MCP for AI assistant automation. *** ## Before you begin * **Harness account:** Access to a Harness project with AI Evals enabled. Go to [RBAC in Harness](https://developer.harness.io/docs/platform/role-based-access-control/rbac-in-harness) to confirm you have project-level access. Go to [Manage projects](https://developer.harness.io/docs/platform/organizations-and-projects/projects-and-organizations) if you need to create a new project. * **LLM provider API key:** An API key from OpenAI, Anthropic, or another LLM provider. Required for LLM-as-a-judge metrics. Go to [Add and reference text secrets](https://developer.harness.io/docs/platform/secrets/add-use-text-secrets) to learn how to store API keys securely. * **Harness pipeline:** A pipeline with a CI or CD stage if you plan to gate deployments on eval suites. Optional for local SDK runs. Go to [Get started with Harness CI](https://developer.harness.io/docs/continuous-integration/get-started/onboarding-guide) or [CD & GitOps tutorials](https://developer.harness.io/docs/continuous-delivery/get-started/tutorials/cd-gitops-tutorials) to create your first pipeline. * **Python 3.10 or later:** Required for the `harness-evals` SDK and CLI path. Skip if you only use the UI. *** ## Connect to the AI Evals control plane Complete these steps once per project before you build your first eval. 1. **Add your LLM provider key as a Harness Secret.** Go to **Project Settings** > **Secrets** > **+ New Secret** > **Text**. Name the secret to match how your evals reference it, for example `OPENAI_API_KEY`, and paste the API key into the **Value** field. Go to [Add and reference text secrets](https://developer.harness.io/docs/platform/secrets/add-use-text-secrets) for detailed steps. 2. **Create an LLM Connector.** Go to **Project Settings** > **Connectors** > **+ New Connector** and choose your provider (OpenAI, Anthropic, Azure OpenAI, or Google Vertex AI). Reference the secret you created in step 1 as the API key source, test the connection, and save. Go to [OpenAI model connector](https://developer.harness.io/docs/platform/harness-ai/openai-model-connector) or [Anthropic model connector](https://developer.harness.io/docs/platform/harness-ai/anthropic-model-connector) for detailed setup steps. 3. **Verify project-level AI Evals access.** Open the **AI Evals** module in the left navigation and confirm you can see the **Targets**, **Datasets**, **Scorers**, **Evaluations**, **Suites**, **Observe**, **Analytics**, and **Registry** tabs. If you don't see the module, contact your account administrator to enable AI Evals for your project. *** ## Choose your path Pick the path that matches how you work. All three paths cover the same four-resource model (Target, Dataset, Metric Set, Evaluation, Run). * [UI Quickstart](/ai-evals/get-started/quickstart-ui.md) * [REST API Quickstart](/ai-evals/get-started/quickstart-api.md) * [MCP Quickstart](/ai-evals/get-started/quickstart-mcp.md) ## Next steps ### Learn the fundamentals * Go to [Core concepts](/ai-evals/core-concepts/targets.md) to understand targets, datasets, metric sets, evaluations, and suites in depth. * Go to [Targets](/ai-evals/core-concepts/targets.md) to learn about Prompt, App, and Static target types. * Go to [Datasets](/ai-evals/core-concepts/datasets.md) to learn dataset design best practices and synthetic generation. * Go to [Metrics](/ai-evals/core-concepts/metrics.md) to explore the 47+ built-in metrics across 9 categories. ### Integrate with CI/CD * Go to [Pipeline integration](/ai-evals/guides/overview-2.md) to gate a Harness pipeline on an eval suite. * Go to [Create your first pipeline](https://developer.harness.io/docs/continuous-integration/get-started/onboarding-guide) if you need to set up CI/CD first. * Go to [Policy as Code](https://developer.harness.io/docs/platform/governance/policy-as-code/harness-governance-overview) to learn about governance and quality gates. ### Advanced evaluation * Go to [Multi-turn and agentic evals](/ai-evals/guides/overview.md) to evaluate conversations and tool-use trajectories. * Go to [Observability and analytics](/ai-evals/guides/overview-1.md) to inspect production traces and track pass-rate trends. ### Platform essentials * Go to [Harness Platform key concepts](https://developer.harness.io/docs/platform/get-started/key-concepts) to understand how AI Evals fits into the Harness ecosystem. * Go to [Manage roles](https://developer.harness.io/docs/platform/role-based-access-control/add-manage-roles) to configure permissions for your team. * Go to [Harness secrets management](https://developer.harness.io/docs/platform/secrets/secrets-management/harness-secret-manager-overview) to learn about secure credential storage.