Split tests in Run steps

When you run tests in Harness CI, you use Run or Test steps.

In a Run step, you can split tests for any language or test tool by using test splitting and the parallelism looping strategy to improve test times.

To split tests in a Run step, you need to:

1. Define a parallelism strategy.

2. Define a test splitting strategy.

3. Produce test reports in JUnit XML format.

4. Run the pipeline and inspect the test results.

( 75 seconds x 50 commits ) / 60 seconds = 62.5 minutes saved

Here are examples of test splitting and parallelism applied to Run steps. For more YAML examples of test splitting, go to YAML Examples: Test splitting.

    - stage:
        name: tests
        identifier: tests
        description: ""
        type: CI
        strategy: ## This is the parallelism strategy for the stage. Stage-level parallelism is recommended.
          parallelism: 4 ## Tests are split into a maximum of four workloads. Stage-level parallelism generates a stage for each workload.
          maxConcurrency: 2 ## Optional. This setting limits the number of workloads that can run at once.
        spec:
          cloneCodebase: true
          platform:
            os: Linux
            arch: Amd64
          runtime:
            type: Cloud
            spec: {}
          execution:
            steps:
              - step:
                  type: Run ## Test splitting can be applied to any Run steps where you run tests.
                  name: tests
                  identifier: tests
                  spec:
                    envVariables: ## These environment variables are used in the 'command' to get the index value for each parallel instance and the total number of parallel instances.
                      HARNESS_NODE_INDEX: <+strategy.iteration>
                      HARNESS_NODE_TOTAL: <+strategy.iterations>
                    shell: Sh
                    command: |- ## Split tests commands are included alongside the regular test commands.
                      # Install dependencies.
                      pip install -r requirements.txt

                      # Call split_tests, define splitting strategy, and generate the list of test files.
                      FILES=`./split_tests --glob "**/test_*.py" --split-by file_timing \
                         --split-index ${HARNESS_NODE_INDEX} \
                         --split-total ${HARNESS_NODE_TOTAL}`
                      echo $FILES

                      # Use the test files list as input for pytest and produce results in JUnit XML format.
                      pytest -v --junitxml="result_<+strategy.iteration>.xml" $FILES

                    reports:
                      type: JUnit
                      spec:
                        paths:
                          - "**/result_<+strategy.iteration>.xml" ## Using the expression '<+strategy.iteration>' in the file name ensures that the results of parallel runs don't overwrite each other.

Define a parallelism strategy

In the context of test splitting, the parallelism strategy defines the number of workloads into which tests can be divided. Each parallel instance (or workload) is a duplicate of the stage where you've defined a parallelism strategy, but each instance runs different tests according to the test splitting strategy.

Optimize parallelism

In general, a higher parallelism value means a faster pipeline run time, because the tests can be divided into more parallel instances. However, this depends on your test suite and resource limitations in your build infrastructure. For example, if you try to run 10 groups of tests, but your build infrastructure can't handle 10 parallel instances, the pipeline can fail or take longer than expected.

To optimize your parallelism strategy:

• Try different parallelism values to determine your infrastructure's limits. Parallelism impacts resource allocation for the pipeline. A pipeline with five sequential stages can require fewer resources than a pipeline running five parallel instances of a stage, because the second pipeline has to run all five instances at once.

• Use maxConcurrency to control the flow of parallel instances and avoid overtaxing infrastructure resources. Concurrency limits the number of parallel instances that can run at once and queues additional instances.

  • For example, if you set parallelism: 12, Harness attempts to run 12 instances of the stage at once. If you set parallelism: 12 and maxConcurrency: 3, Harness generates 12 instances of the stage, but only runs three instances at a time. The remaining nine instances are queued, and the queued instances start running as space clears in the concurrency limit (when prior instances finish).

  • Concurrency allows you to divide tests into more workloads without overloading your system resources.

  • There are resource requirements to generate parallel instances (even if they are not all running at the same time) and handle queues. Try different combinations of parallelism and maxConcurrency values to determine your ideal configuration.

• Review the Best practices for looping strategies, including how to calculate ideal concurrency.

Define a test splitting strategy

The test splitting strategy determines how you want to divide the tests, such as by number of tests or test timing.

To do this, use the split_tests binary in the Run step that runs your tests, for example:

The split_tests tool outputs a list of test files based on your chosen splitting strategy. You then use this list as input for your test tool's commands so that each parallel instance only runs an assigned subset of tests.

For example, these commands use split_tests with pytest:

Binary path

Call the split_tests binary. The path depends on your build infrastructure.

• For Harness Cloud, use ./split_tests.

• For kubernetes and VM build infrastructures, use /addon/bin/split_tests.

• For steps running locally or in containerized executions on a VM, rather than on the host, use /usr/bin/split_tests. You must also declare this path in shared paths.

Glob or file-path

Specify the set of all tests that you want to run across all parallel instances.

Whether you use --glob or --file-path is determined by --split-by.

• For --split-by file_timing or --split-by file_size, you can use a glob expression to specify the set of files to split, such as --glob "**/test_*.py".

• For --split-by class_timing, --split-by testcase_timing, and --split-by testsuite_timing, you must provide a text file of the elements to split. For example, if you want to split by Java class timing, you could specify the set of classes to split and test in a new-line-delineated string and then reference the text file in with --file-path FILE_NAME.txt:

Split-by

Specify a test splitting strategy, such as test timing or file size. If unspecified, the default is --split-by file_timing.

The split_tests binary supports these test splitting strategies:

• --split-by class_timing: Split tests into groups based on the timing data for individual classes. This strategy requires timing data from the previous run. If timing data isn't available, split_tests falls back to --split-by file_size.

• --split-by file_size: Split tests into groups based on the size of individual files.

• --split-by file_timing: (Default) Split tests into groups based on the test times of individual files. split_tests uses the most recent timing data to ensure that all parallel test runs finish at approximately the same time. This strategy requires timing data from the previous run. If timing data isn't available, split_tests falls back to --split-by file_size.

• --split-by testcase_timing: Split tests into groups based on the timing data for individual test cases. This strategy requires timing data from the previous run. If timing data isn't available, split_tests falls back to --split-by file_size.

• --split-by testsuite_timing: Split tests into groups based on the timing data for individual test suites. This strategy requires timing data from the previous run. If timing data isn't available, split_tests falls back to --split-by file_size.

Split-index and split-total

Harness automatically runs the --split-index and --split-total commands to create groups of tests files based on the total number of parallel instances (--split-total) and assign specific test files for each individual instance (--split-index).

Harness automatically populates the HARNESS_NODE_INDEX and HARNESS_NODE_TOTAL environment variables supplied to these commands from the environment variables you declared when you defined a parallelism strategy.

If you need these commands to use different values, you can include them with your split tests commands:

Split tests output

split_tests stores list of test files assigned to the current instance in a variable. For example, the following command stores the list of tests in $FILES:

You then call the variable (such as $FILES) in your test tool's commands, for example:

You can include echo $FILES to print the list of assigned tests in each step's logs, for example:

Produce test reports

Logs and results

When you run the pipeline, you can observe the parallel instances running on the Build details page.

When the build finishes, go to the Tests tab to view the results. Use the Test Executions stage dropdown menu to view results for each parallel instance.

YAML examples: Test splitting

With Harness CI, you can split tests for any language or tool. Here are some examples of test splitting for different languages and tools.