Skip to main content

Playwright in AI Test Automation

Last updated on

Harness AI Test Automation runs your existing Playwright suite, your playwright.config, your spec files, your package.json scripts, without rewrites, wrappers, or migration. Point Harness at the repository that contains your project, and your suite executes on managed cloud workers using the config you already have.

Playwright test suites live in the same Test Suites list as AI-authored tests, so on-demand runs, environments, variables, tunnels, run history, and per-spec pass/fail reporting all work the same way across both. You keep authoring tests in Playwright as you do today; Harness handles execution and brings the results into one place, including AI-assisted failure triage on every failed test so you can see why a test broke, not just that it broke.

Your playwright.config in the repo controls how tests run by default. For a single run, you can override specific settings from the Run dialog without changing the repo, for example bumping retries, raising a timeout, or pointing the run through a tunnel to reach a firewall-protected app. Any field you leave blank falls back to your config.

What you can do from here​

  • Create a Playwright test suite in the UI. From Test Suites, select Create new suite and choose Playwright. The wizard covers source, branch, and environment in three steps.
  • Trigger the same test suite from a pipeline. Add the AI Test Automation step in a CI, CD, or Custom stage and set Framework to Playwright. Pipeline variables support Harness expressions (for example <+pipeline.variables.env_url>), so you can parameterize the suite, environment, and tunnel at trigger time. Go to Harness Pipeline for the pipeline-side setup.
  • Author new tests in AI Test Automation alongside your Playwright suite when you want intent-based tests for flows that are painful to script. Go to AI Test Automation for the AI test flow.

Prerequisites​

  • Project access: An AI Test Automation project where you can open Test Suites in the left navigation.
  • Connected source: Access to the repository or provider where your Playwright project lives (Harness Code Repository or a third-party Git provider you select in the UI), with connectors configured as needed.
  • Playwright layout: A valid Playwright project in that source (for example playwright.config and spec files) so Harness can fetch the revision you choose and execute tests using your configuration.
  • Environments: At least one application environment if you plan to run against a specific target (for example Default). To define or review environments, see Add application environments.
Source repositories and connectors

Choose Harness Code Repository when the Playwright project is in Harness Code Repository.

Choose Third-party Git Provider when the project is in GitHub, GitLab, Bitbucket, Azure Repos, or another supported system. Those paths use Harness code repository connectors (credentials and access) that you create in the Platform module, then select when you configure the GIT step. Go to Connect to a code repository for setup and provider-specific guides.

Open Playwright in Suites​

  1. In your Harness account, open the AI Test Automation project.
  2. In the left sidebar, select Test Suites.
  3. Select Create new suite, then choose Playwright from the dropdown.

You see the Suites list with all suites, recent runs for each, Run and row actions. Use the Create new suite dropdown to add a new configuration.

Test Suites page with the Create new suite dropdown expanded showing AI Test and Playwright options

Create a configuration​

The New Playwright Test wizard has three steps: About Test Suite, GIT, and Environment. You move through them with Next and finish with Save.

New Playwright Test wizard step About the Test Suite with name field and the three-step rail (About Test Suite, GIT, Environment)

  1. From Test Suites, select Create new suite and choose Playwright.

  2. In Step 1: About Test Suite:

    • Name: Enter a clear suite name (required).
    • Description and Tags (optional): Expand and add if you use them for organization.

  3. Click Next, then in Step 2: GIT (GIT Source):

    • Choose Harness Code Repository or Third-party Git Provider.
    • Select Repository: Pick the repo that contains your Playwright project.
    • GIT fetch type: For example Latest from Branch.
    • Git Branch: Select or enter the branch to run from.
    • Project Root (optional): Restrict to a subdirectory if your Playwright project is not at the repository root.

  4. Click Next, then in Step 3: Environment, add the first application environment for this suite. At minimum, enter an Environment Name, an Environment URL, and a Test execution mode (use Execution Alias with the alias left blank to run all tests). The fields here are the same as the Environments tab covered in the next section, where you can also add more environments or edit existing ones.

  5. Click Save to finish creating the suite. You can add more environments or edit this one later from the Environments tab.

Manage your Playwright setup​

After you save the wizard, select the suite name from the list to open its details page. The UI matches the product tabs below: Configuration, Variables, Environments, and Run History. Use Run in the header from any tab when you are ready to execute (see Run Playwright tests).

On the Configuration tab:

  1. Review or edit General information: name, description, tags.
  2. Under GIT Source, use the same options as in the wizard (provider, repository, fetch type, branch, and optional Project Root path).
  3. To remove the suite, use Delete Playwright suite in the danger zone at the bottom. That action permanently deletes the suite configuration.
Playwright suite Configuration tab with General information, GIT Source, and delete action

Run Playwright tests​

  1. From the suite list or the suite details page, click Run.

  2. In Run Configuration, choose Environment (for example Default).

  3. Under Override, adjust any fields you need. All override fields are optional. If you leave them unset, Harness uses the defaults from your Playwright config in the repository.

    Typical overrides include Number of Retries, Delay, Worker, Navigation timeout, Overall timeout, Action timeout, Report pattern, and Tunnel (for example Harness Cloud). Use the info icons next to labels for field-level help.

  4. Click Run to start execution.

For private or firewall-reached apps, align Tunnel and networking with your environment setup. See Test firewall-protected applications.

Run Configuration modal with environment, optional Playwright overrides, and Run button

Monitor run progress​

While the suite is starting, Harness shows a progress bar with ordered phases, for example:

  1. Initializing infrastructure
  2. Setting up the codebase
  3. Triggering test execution

Before execution begins, Harness runs a preflight discovery step that detects your test files, resolves dependencies, and validates the Playwright configuration. If discovery fails (for example a missing dependency or invalid config), the run stops early with a clear error instead of timing out mid-execution.

Harness automatically parallelizes your suite across 4 shards by default. Tests are distributed evenly across shards so total wall-clock duration is reduced without any configuration on your part.

A full run often takes on the order of a few minutes, depending on suite size and infrastructure.

Running progress with phases for infrastructure, codebase, and test execution

Abort a run​

To cancel a run that is in progress, open the running execution from Run History and click Abort. The run stops, infrastructure is released, and the run status updates to Aborted. You can also abort from the pipeline execution view when the suite was triggered through a pipeline step.

Review Playwright run results​

When the run finishes, open it from Run History on the suite or follow the UI to the run results page.

The Overview tab summarizes the execution:

  • Total tests and pass rate (for example a breakdown of passed vs failed).
  • Counts by outcome: Failed, Passed, Flaky, Skipped, Timed out, Interrupted (where applicable).
  • Timing: Avg test time, Total duration, Started at.
  • Re-run runs the suite again from the report.
Run Overview with pass rate, status counts, duration, and Re-run

Set up notifications for Playwright runs​

Configure centralised notifications to get alerted when Playwright runs complete, fail, or are aborted. Notifications use the Harness Platform notification system and can be delivered through Slack, Email, Microsoft Teams, PagerDuty, or Webhooks.

  1. Go to Project Settings → General → Notifications Management.
  2. Click + New Notification.
  3. Select AI Test Automation as the resource type.
  4. Add a condition and select the events you want to be notified about: Playwright Run Completed, Playwright Run Failed, or Playwright Run Aborted.
  5. Choose a notification channel and click Submit.

For the full notification setup, go to Centralised notifications.

Next steps​