Skip to main content

Run Tests step settings

This topic describes settings for the Harness CI Run Tests step. Use the Run Tests step to run tests and enable Test Intelligence in CI pipelines.

Hidden settings
  • Some settings are located under Additional Configuration in the Pipeline Studio's visual editor.
  • Some settings are only applicable to certain languages or build tools.
  • Settings specific to containers, such as Set Container Resources, are not applicable when using the step in a stage with VM or Harness Cloud build infrastructure.

Name​

Enter a name summarizing the step's purpose. Harness automatically assigns an Id (Entity Identifier Reference) based on the Name. You can edit the Id.

Description​

Optional text string.

Container Registry and Image​

The Container Registry is a Harness container registry connector for the image that you want Harness to run build commands on, such as Docker Hub.

The Image is the FQN (fully-qualified name) or artifact name of the Docker image to use when this step runs commands, for example us.gcr.io/playground-123/quickstart-image. The image name should include the tag. If you don't include a tag, Harness uses the latest tag.

You can use any Docker image from any Docker registry, including Docker images from private registries. Different container registries require different name formats:

  • Docker Registry: Enter the name of the artifact you want to deploy, such as library/tomcat. Wildcards aren't supported. FQN is required for images in private container registries.
  • ECR: Enter the FQN (fully-qualified name) of the artifact you want to deploy. Images in repos must reference a path, for example: 40000005317.dkr.ecr.us-east-1.amazonaws.com/todolist:0.2.
  • GCR: Enter the FQN (fully-qualified name) of the artifact you want to deploy. Images in repos must reference a path starting with the project ID that the artifact is in, for example: us.gcr.io/playground-243019/quickstart-image:latest.
info

The stage's build infrastructure determines whether these fields are required or optional:

Language​

Select the source code language to build: C#, Java, Kotlin, or Scala.

Additional settings appear if you select C# or Java.

Build Environment​

Select the build environment to test.

Framework Version​

Select the framework version to test.

Namespaces​

This setting is only available if you select DOTNET as the Build Tool.

Supply a comma-separated list of namespace prefixes that you want to test.

Build Tool​

Select the build automation tool. Supported tools vary by Language.

Build Arguments​

Enter the arguments for the build tool. These are used as input for the chosen build tool.

The following languages and build tools have specific build argument requirements:

  • Java: Provide runtime arguments for the tests, for example: Test -Dmaven.test.failure.ignore=true -DfailIfNoTests=false.
  • C#: Provide runtime arguments for the tests, for example: /path/to/test.dll /path/to/testProject.dll. Do not inject another instrumenting agent, such as a code-coverage agent, in the argument string.
  • NUnit C#: Provide runtime executables and arguments for the tests, for example: . "path/to/nunit3-console.exe" path/to/TestProject.dll --result="UnitTestResults.xml" /path/to/testProject.dll. You must include the executable in the string. Do not inject another instrumenting agent, such as a code-coverage agent, in the string.

Test Report Paths​

Specify one or more paths to files that store test results in JUnit XML format. You can add multiple paths. If you specify multiple paths, make sure the files contain unique tests to avoid duplicates. Glob is supported.

This field is required for the Run Tests step to publish test results.

Pre-Command​

Enter the commands for setting up the environment before running the tests. For example, printenv prints all or part of the environment.

If a script is supplied here, select the corresponding Shell option.

Post-Command​

Enter the commands used for cleaning up the environment after running the tests. For example, sleep 600 suspends the process for 600 seconds.

If a script is supplied here, select the corresponding Shell option.

Run Only Selected Tests​

This option must be selected to enable Test Intelligence.

If this option is unchecked, Test Intelligence is disabled and all tests run on every build.

Packages​

Leave blank or provide a comma-separated list of source code package prefixes, such as com.company., io.company.migrations. If you do not provide a list, Harness auto-detects the packages.

Environment Variables​

Variables passed to the container as environment variables and used in the step's commands.

Output Variables​

Output variables expose values for use by other steps or stages in the pipeline.

To create an output variable, do the following in the step where the output variable originates:

  1. In the Command field, export the output variable. For example, the following command exports a variable called myVar with a value of varValue:

    export myVar=varValue

  2. In the step's Output Variables, declare the variable name, such as myVar.

To call a previously-exported output variable in a later step or stage in the same pipeline, use a variable expression that includes the originating step's ID and the variable name.

To reference an output variable in another step in the same stage, use either of the following expressions:

<+steps.[stepID].output.outputVariables.[varName]>
<+execution.steps.[stepID].output.outputVariables.[varName]>

To reference an output variable in a different stage than the one where it originated, use the following expression:

<+stages.[stageID].spec.execution.steps.[stepID].output.outputVariables.[varName]>
<+pipeline.stages.[stageID].spec.execution.steps.[stepID].output.outputVariables.[varName]>
YAML example: Output variable

In the following YAML example, step a exports an output variable called myVar, and then step b references that output variable.

              - step:
                  type: Run
                  name: a
                  identifier: a
                  spec:
                    shell: Sh
                    command: export myVar=varValue
                    outputVariables:
                      - name: myVar
              - step:
                  type: Run
                  name: b
                  identifier: b
                  spec:
                    shell: Sh
                    command: |-
                      echo <+steps.a.output.outputVariables.myVar>
                      echo <+execution.steps.a.output.outputVariables.myVar>

Image Pull Policy​

If you specified a Container Registry and Image, you can specify an image pull policy:

  • Always: The kubelet queries the container image registry to resolve the name to an image digest every time the kubelet launches a container. If the kubelet encounters an exact digest cached locally, it uses its cached image; otherwise, the kubelet downloads (pulls) the image with the resolved digest, and uses that image to launch the container.
  • If Not Present: The image is pulled only if it isn't already present locally.
  • Never: The image is not pulled.

Run as User​

If you specified a Container Registry and Image, you can specify the user ID to use for running processes in containerized steps.

For a Kubernetes cluster build infrastructure, the step uses this user ID to run all processes in the pod. For more information, go to Set the security context for a pod.

Set Container Resources​

These settings specify the maximum resources used by the container at runtime:

  • Limit Memory: The maximum memory that the container can use. You can express memory as a plain integer or as a fixed-point number using the suffixes G or M. You can also use the power-of-two equivalents Gi and Mi. The default is 500Mi.
  • Limit CPU: The maximum number of cores that the container can use. CPU limits are measured in CPU units. Fractional requests are allowed. For example, you can specify one hundred millicpu as 0.1 or 100m. The default is 400m. For more information go to Resource units in Kubernetes.

Timeout​

The timeout limit for the step. Once the timeout is reached, the step fails and pipeline execution continues.

To change what happens when steps fail, go to Step Failure Strategy settings.

To configure when pipelines should skip certain steps, go to Step Skip Condition settings.