Run tests in CI pipelines

You can run all types of tests in CI pipeline, including integration tests, functional tests, mutation tests, unit tests, and more.

To run tests in CI pipelines, you can use Run steps or Test Intelligence steps (also known as Test steps).

Run step

You can run any type of test for any codebase in a Run step.

For example, this step runs pytest and produces a test report in JUnit XML format.

              - step:
                  type: Run
                  name: Pytest
                  identifier: Pytest
                  spec:
                    shell: Sh
                    command: |-
                      pytest test_main.py --junit-xml=output-test.xml
                    reports:
                      type: JUnit
                      spec:
                        paths:
                          - output-test.xml

For more information about configuring Run steps, go to Use Run steps.

tip

Harness supports code coverage in Run steps.

Test step

The Test step is required to enable Test Intelligence. You can use Test steps with or without Test Intelligence; however, the Test step doesn't support all languages yet.

This example runs tests with Maven and Test Intelligence, and it produces a test report in JUnit XML format.

              - step:
                  type: Test
                  name: RunTestsWithIntelligence
                  identifier: RunTestsWithIntelligence
                  spec:
                    command: |-
                      mvn test -Dmaven.test.failure.ignore=true -DfailIfNoTests=false
                      mvn package -DskipTests
                    shell: Sh
                    connectorRef: account.harnessImage
                    image: maven:3.8-jdk-11
                    intelligenceMode: true
                    reports:
                      - "target/surefire-reports/*.xml"

For more information about configuring Test steps and Test Intelligence, go to Test Intelligence overview.

Improve test times

These Harness CI features can improve test times:

• Test Intelligence: Test Intelligence speeds up your test cycles by running only the unit tests required to confirm the quality of the code changes that triggered a build. You must use the Test step to enable Test Intelligence.
• Test splitting (parallelism): You can use parallelism with either the Run or Test steps to speed up test times. For more information, go to Split tests in Run steps or Split tests in Test steps.
• Step groups: You can use step groups to organize and condense pipelines that run a lot of tests.

You might also try these test optimization practices:

• Use mock services in your unit tests, rather than setting up and connecting to third-party services. Mockups can often test your code as well as fully-running services but with fewer resources.
• Avoid integration tests in your build pipelines when possible. You might want to move these to a separate pipeline.
• Look for obsolete tests that you can delete.
• Look for unnecessary sleep statements in your unit test code.
• Order your tests so that the tests most likely to fail run first.

Test results

You can view test results in Harness.

Troubleshoot tests in Harness CI

Go to the CI Knowledge Base for questions and issues related to running tests in Harness CI, including:

• Test suites incorrectly parsed
• Test reports missing
• Does Harness support test splitting (parallelism)?
• How do I use Test Intelligence?
• Test Intelligence call graph is empty