Skip to main content

Code coverage

You can add code coverage to a Harness CI pipeline by configuring code coverage tools in your codebase and adding code coverage commands to steps that run tests.

info

This documentation is not exhaustive. For information about languages or code coverage tools not described here, refer to the documentation for that tool or language.

For more information about running tests in Harness CI, go to Run tests in CI pipelines.

Code coverage by language​

The following examples show how to include code coverage in a Harness CI pipeline for different languages.

Go​

Go has built-in code coverage functionality.

  1. Add the following commands to the Run step where you run your tests:

    go test -cover -coverprofile=c.out
    go tool cover -html=c.out -o coverage.html

    For example:

                  - step:
                      type: Run
                      identifier: test
                      name: Test
                      spec:
                        shell: Sh
                        command: |-
                          go test -cover -coverprofile=c.out
                          go tool cover -html=c.out -o coverage.html

  2. Add a step to upload your code coverage report to cloud storage.

  3. Add a step to view your code coverage report on the Artifacts tab.

Java​

  1. Set up a Java code coverage tool, such as JaCoCo. By including JaCoCo in pom.xml, the mvn test command automatically writes a code coverage report to an exec file.

  2. Run your tests in a Run step, for example:

                    - step:
                      type: Run
                      name: run test
                      identifier: run_test
                      spec:
                        shell: Sh
                        command: |-
                          mvn test
                        reports:
                          type: JUnit
                          spec:
                            paths:
                              - target/surefire-reports/*.xml

  3. Store and publish your code coverage report:

JavaScript​

  1. If necessary, set up a JavaScript code coverage tool, such as Istanbul. Your test tool may already include code coverage; for example, Istanbul is included with Jest.

  2. Add code coverage arguments or commands to the relevant Run step. For example, with Jest, add --collectCoverage=true to your jest command.

                  - step:
                      type: Run
                      name: Run Jest Tests
                      identifier: run_jest_tests
                      spec:
                        shell: Sh
                        command: |-
                          yarn add --dev jest-junit
                          jest --ci --runInBand --reporters=default --reporters=jest-junit --collectCoverage=true
                        envVariables:
                          JEST_JUNIT_OUTPUT_DIR: "/harness/reports"
                        reports:
                          type: JUnit
                          spec:
                            paths:
                              - "/harness/reports/*.xml"

  3. Add a step to upload your code coverage report to cloud storage.

  4. Add a step to view your code coverage report on the Artifacts tab.

PHP​

The built-in phpdbg tool can generate code coverage reports.

  1. Add the following command to the Run step where your run your tests:

    phpdbg -qrr vendor/bin/phpunit --coverage-html build/coverage-report

    For example:

                  - step:
                      type: Run
                      identifier: test
                      name: Test
                      spec:
                        shell: Sh
                        command: |-
                          mkdir -p /harness/phpunit
                          phpunit --log-junit /harness/phpunit/junit.xml tests
                          phpdbg -qrr vendor/bin/phpunit --coverage-html build/coverage-report
                        reports:
                          type: JUnit
                          spec:
                            paths:
                              - /harness/phpunit/junit.xml

  2. Add a step to upload your code coverage report to cloud storage.

  3. Add a step to view your code coverage report on the Artifacts tab.

Python​

Use these steps to install code coverage tools when you run Python tests in Run steps.

  1. Install a Python code coverage tool, such as Coverage.py. Depending on your build infrastructure, you can install this directly on the host machine or use a Run step to set up the test environment at runtime.

                 - step:
                  type: Run
                  identifier: installdependencies
                  name: Install dependencies
                  spec:
                    command: |
                      python3 -m pip install --upgrade pip
                      pip install -r requirements.txt
                      pip install pytest
                      python3 -m pip install coverage

  2. Add code coverage commands to the Run step where your run your tests.

              - step:
                  type: Run
                  identifier: run pytest
                  name: Run pytest
                  spec:
                    command: |
                      coverage run -m pytest --junit-xml=report.xml
                      coverage report
                      coverage html
    Coverage.py usage

    With Coverage.py, replace the initial python or pytest in your usual test commands with coverage run.

    For more information, refer to the Coverage.py quick start guide.

  3. Add a step to upload your code coverage report to cloud storage.

  4. Add a step to view your code coverage report on the Artifacts tab.

Ruby​

Use these steps to install code coverage tools when you run Ruby tests in Run steps.

  1. Set up a Ruby code coverage tool, such as SimpleCov.

  2. Run your tests in a Run step.

    SimpleCov doesn't require additional commands in the Run step since it is loaded in test/test_helper.rb.

  3. Add a step to upload your code coverage report to cloud storage.

  4. Add a step to view your code coverage report on the Artifacts tab.

Code coverage services​

You can use code coverage services with Harness.

Codacy​

You can use the Codacy Drone plugin in a Plugin step to upload Golang coverage reports to Codacy.

For other languages, go to the Codacy documentation on uploading coverage data to Codacy.

CodeCov​

You can use a Run step to include CodeCov code coverage in a Harness CI pipeline. The Run step contains a script that runs tests with code coverage, then downloads and runs the CodeCov Uploader tool.

  1. Make sure you have a CodeCov account with code coverage enabled on a code repo and a CodeCov Upload Token. For instructions, go to CodeCov Quick Start.

  2. Create a Harness text secret containing your CodeCov Upload Token. Make note of the secret's ID.

  3. In your Harness CI pipeline, add a Run step.

  4. Enter a Name for the step.

  5. Depending on your build infrastructure, you might need to specify a Container Registry and Image containing the binaries that the step needs to run your commands.

    For example, you need to run pytest, and pytest isn't available on the build machine, you must specify a Docker connector and image, such as pytest:latest.

    For information about when these fields are required and how to specify images, go to Use Run steps.

  6. Make sure the Command field includes all commands necessary to prepare the test environment, run tests with code coverage, and download and run the CodeCov Uploader tool, for example:

    echo "Run tests with code coverage"
    uname -a
    pip install pytest
    pip install pytest-cov
    pip install -r requirements.txt

    pytest -v --cov --junitxml="result.xml" test_api.py test_api_2.py test_api_3.py

    # Download Codecov uploader and run it.
    curl -Os https://uploader.codecov.io/latest/linux/codecov
    chmod +x codecov
    ./codecov -t ${CODECOV_TOKEN}

  7. In Environment Variables, add a CODECOV_TOKEN environment variable and set the value to an expression referencing your CodeCov Upload token secret, such as <+secrets.getValue("my_codecov_upload_token")>.

  8. Add one or more Report Paths for your code coverage reports.

  9. Select Apply Changes to save the step, and then select Save to save the pipeline.

Here are examples of Run steps configured for CodeCov code coverage in the Visual and YAML editors.

- step:
    type: Run
    name: Pytest code coverage
    identifier: pytest_code_coverage
    spec:
      connectorRef: account.harnessImage ## Provide a Docker connector ID, if required for your build infrastructure.
      image: python:latest ## Specify an image, if required for your build infrastructure.
      shell: Sh
      command: |-
        # Prepare test environment.
        echo "set up to run tests"
        uname -a
        pip install pytest
        pip install pytest-cov
        pip install -r requirements.txt

        # Run tests with code coverage.
        pytest -v --cov --junitxml="result.xml" test_api.py test_api_2.py test_api_3.py

        # Download Codecov uploader and run it. Uses CODECOV_TOKEN envVariable.
        curl -Os https://uploader.codecov.io/latest/linux/codecov
        chmod +x codecov
        ./codecov -t ${CODECOV_TOKEN}
      reports: ## Specify one or more report paths.
        type: JUnit
        spec:
          paths:
            - "**/*.xml"
      envVariables:
        CODECOV_TOKEN: <+secrets.getValue("my_codecov_upload_token")> ## Reference your CodeCov Upload Token secret in an environment variable.

When you run your pipeline, you can review CodeCov information in the Run step's logs on the Build details page. If the results were successfully uploaded to CodeCov, the logs include a resultURL that you can follow to view the code coverage output in your CodeCov account, such as:

...
145  info  [date] [time]    [timestamp] [`info`] Uploading...
146  info  [date] [time]    [timestamp] [`info`] {"status":"success","resultURL":"https://codecov.io..."}
...

Alternately, you can run your tests with code coverage in a Run step and then upload the results to CodeCov by running the CodeCov Drone plugin in a Plugin step.

Coveralls​

To integrate Coveralls in your Harness CI pipelines, follow the Coveralls documentation to Integrate Coveralls with your codebase. Note the following:

  • For Step 2: Choose an integration, use the Universal Coverage Reporter.
  • For Step 3: Configure your project to send coverage to Coveralls:
    • Create a Harness text secret for your COVERALLS_REPO_TOKEN.
    • Add the COVERALLS_REPO_TOKEN environment variable to steps in your CI pipelines that run tests with code coverage.
    • For the environment variable value, use a Harness expression to reference the encrypted text secret, such as <+secrets.getValue("YOUR_COVERALLS_SECRET_ID")>.
Add an environment variable to a step

Add envVariables to the step.spec for the relevant Run or Test step.

              - step:
                  type: Run
                  name: npm test
                  identifier: npm_test
                  spec:
                    shell: Sh
                    command: |-
                      npm install
                      npm run build --if-present
                      npm test
                    reports:
                      type: JUnit
                      spec:
                        paths:
                          - report.xml
                    envVariables:
                      COVERALLS_REPO_TOKEN: <+secrets.getValue("YOUR_COVERALLS_SECRET_ID")>

You can also upload coverage reports to Coveralls by using the Coveralls Drone plugin in a Plugin step.

View code coverage reports on the Artifacts tab​

You can use Drone plugins to view code coverage reports on the Artifacts tab on the Build details page.

The Artifact Metadata Publisher plugin pulls content from cloud storage and publishes it to the Artifacts tab.

  1. Add steps to your pipeline that run tests with code coverage and produce code coverage reports.

  2. Add a step to upload the report artifact to cloud storage.

  3. Add a Plugin step that uses the artifact-metadata-publisher plugin, for example:

                   - step:
                      type: Plugin
                      name: publish artifact metadata
                      identifier: publish_artifact_metadata
                      spec:
                        connectorRef: account.harnessImage
                        image: plugins/artifact-metadata-publisher
                        settings:
                          file_urls: ## Provide the URL to the code coverage artifact that was uploaded in the Upload Artifacts step. If you uploaded multiple artifacts, you can provide a list of URLs.
                          artifact_file: artifact.txt ## Provide any '.txt' file name. Harness uses this to store the artifact URL and display it on the Artifacts tab. This value is not the name of your uploaded artifact, and it has no relationship to the artifact object itself.
tip

Code coverage reports are not the only artifacts you can publish to the Artifacts tab. You can publish any URL to the Artifacts tab.