Ctrlk
Sign UpGet a DemoContact Us
For the complete documentation index, see llms.txt. This page is also available as Markdown.

(STO license) Create a build-scan-push pipeline

Launch pipeline builds and scans automatically based on GitLab merge requests.

This topic describes how to create an end-to-end pipeline that builds an image and pushes it to Docker Hub only if the codebase and image contain no critical vulnerabilities. This pipeline uses two free tools:

Once you complete this workflow, you'll have a complete end-to-end pipeline that you can easily adapt to a wide variety of use cases. You can also copy/paste the YAML pipeline example below into Harness and update it with your own infrastructure, connectors, and access tokens.

The following steps describe the workflow:

  1. A Run step scans the codebase using Semgrep and saves the results to a SARIF file.

  2. A Semgrep step ingests the scan results (ingestion-only workflow).

  3. If the code has no critical vulnerabilities, another Run steps builds the image.

  4. An Aqua Trivy step scans the image and ingests the results (orchestration workflow).

  5. If the image has no critical vulnerabilities, another Run step pushes the image to Docker Hub.

scan-build-scan-push pipeline

PREREQUISITES

  • This workflow has the following prerequisites:

    • A Harness account and STO module license.

    • You must have a Security Testing Developer or AppSec role assigned.

    • A basic understanding of key STO concepts and good practices is recommended. This workflow builds on the SAST code scans using Semgrep and Container image scans with Aqua Trivy workflows.

    • A Semgrep account login and access token. For specific instructions, go to Getting started from the CLI in the README on GitHub.

    • GitHub requirements — This workflow assumes you have the following:

      • A GitHub account and access token.

      • A GitHub connector that specifies your account (http://github.com/my-account) but not a specific repository (http://github.com/my-account/my-repository).

      • Your GitHub account should include a repository with code in a language supported by Semgrep such as Python or NodeJS. The repo should also include a Dockerfile for creating an image.

        This workflow uses the dvpwa repository as an example. The simplest setup is to fork this repository into your GitHub account.

    • Docker requirements — The last step in this pipeline pushes the built image to your image registry. To do this step, you must have the following:

      • A Docker Hub account and access token.

      • A Docker connector is required to push the image.

    • Your Semgrep, GitHub, and Docker Hub access tokens must be stored as Harness secrets.

Set up your pipeline

Do the following:

  1. Select Security Testing Orchestration (left menu, top) > Pipelines > Create a Pipeline. Enter a name and click Start.

  2. In the new pipeline, select Add stage > Security.

  3. Set up your stage as follows:

    1. Enter a Stage Name.

    2. Disable Clone Codebase. You will add a Run step to clone the codebase later.

  4. In the Pipeline Editor, go to Overview and add the following shared path:

    • /shared/customer-artifacts You'll use this shared folder to store the code repo so that all steps can access it.

  5. Go to Overview and add the following Shared Path: /shared/scan_results

  6. Expand Overview > Advanced and add the following stage variables.

    You'll be specifying runtime inputs for some of these variables. This enables you to specify the code repo, branch, image label, and image tag, and other variables at runtime.

    • GITHUB_USERNAME — Select Secret as the type and enter your GitHub login name.

    • GITHUB_PAT — Select Secret as the type and then select the Harness secret for your GitHub access token.

    • GITHUB_REPO — Select String for the type and Runtime Input for the value (click the "tack button" to the right of the value field).

    • GITHUB_BRANCH — Select String and Runtime Input.

    • DOCKERHUB_USERNAME — Select String as the type and enter your DockerHub login name.

    • DOCKERHUB_PAT — Select Secret as the type and then select the Harness secret for your Docker Hub access token.

    • DOCKER_IMAGE_LABEL — Select String and Runtime Input.

    • DOCKER_IMAGE_TAG — Select String and Runtime Input.

  7. In the Pipeline Editor, go to Infrastructure and select Cloud, Linux, and AMD64 for the infrastructure, OS, and architecture.

    You can also use a Kubernetes or Docker build infrastructure, but these require additional work to set up. For more information, go to Supported Infrastructures documentation.

The following step is required for Kubernetes or Docker infrastructures only. If you're using Harness Cloud, go to Add the codebase scan step.

Add a Docker-in-Docker background step

The following use cases require a Docker-in-Docker background step in your pipeline:

  • Container image scans on Kubernetes and Docker build infrastructures

  • Custom Scan steps on Kubernetes and Docker build infrastructures

    • Required for all target types and Orchestration/DataLoad modes

The following use cases do not require a Docker-in-Docker background step:

  • Harness Cloud AMD64 build infrastructures

  • SAST/DAST/configuration scans that use a scanner-specific step and not a Custom Scan step.

  • Ingestion scans where the data file has already been generated

Set up a Docker-in-Docker background step

  1. Go to the stage where you want to run the scan.

  2. In Overview, add the shared path /var/run.

  3. In Execution, do the following:

    1. Click Add Step and then choose Background.

    2. Configure the Background step as follows:

      1. Dependency Name = dind

      2. Container Registry = The Docker connector to download the DinD image. If you don't have one defined, go to Docker connector settings reference.

      3. Image = docker:dind

      4. Under Entry Point, add the following: dockerd

        In most cases, using dockerd is a faster and more secure way to set up the background step. For more information, go to the TLS section in the Docker quick reference.

        If the DinD service doesn't start with dockerd, clear the Entry Point field and then run the pipeline again. This starts the service with the default entry point.

      5. Under Additional Configuration, select the Privileged checkbox.

Add a Background step to your pipeline and set it up as follows:- step: type: Background name: background-dind-service identifier: Background_1 spec: connectorRef: CONTAINER_IMAGE_REGISTRY_CONNECTOR image: docker:dind shell: Sh entrypoint: - dockerd privileged: true

Add the codebase scan step

Now you will add a step that runs a scan using the local Semgrep container image maintained by Harness.

  1. Go to Execution and add a Run step.

  2. Configure the step as follows:

    1. Name = run_semgrep_scan

    2. Command =

    3. Open Optional Configuration and set the following options:

      1. Container Registry — When prompted, select Account and then Harness Docker Connector. The step uses this connector to download the scanner image.

      2. Image = returntocorp/semgrep

      3. Add the following environment variable:

        • Key : SEMGREP_APP_TOKEN

        • Value : Click the type selector (right), set the value type to Expression, and enter the value <+secrets.getValue("YOUR_SEMGREP_TOKEN_SECRET")>.

          set the value type

Add a Run step to your SecurityTests stage and configure it as follows:

  • type: Run

  • name: A name for the step.

  • identifier: A unique step ID.

  • spec :

    • connectorRef : account.HarnessImage

      This is a connector to the Harness image registry. The step uses this connector to download the scanner image.

    • image : returntocorp/semgrep

    • shell : Sh

    • command : |-

    • envVariables:

      • SEMGREP_APP_TOKEN: <+secrets.getValue("YOUR_SEMGREP_TOKEN_SECRET")>

Here's an example:

Add the Semgrep ingest step

Now that you've added a step to run the scan, it's a simple matter to ingest it into your pipeline. Harness provides a set of customized steps for popular scanners such as Semgrep.

It's generally good practice to set the fail_on_severity for every scan step. Leave this setting at None for now so you can run and test the entire-end-to-end workflow.

  1. In Execution, add a Semgrep step after your Run step.

  2. Configure the step as follows:

    1. Name = ingest_semgrep_data

    2. Type = Repository

    3. Under Target:

      1. Name = Select Runtime Input as the value type.

      2. Variant = Select Runtime Input as the value type.

    4. Ingestion File = /shared/scan_results/semgrep.sarif

Add a step after the Run step and configure it as follows:

  • type: Semgrep

    • name: A name for the step.

    • identifier: A unique step ID.

    • spec :

      • mode : ingestion

      • config: default

        • target :

          • name : <+stage.variables.GITHUB_REPO>

          • type : repository

          • variant : <+stage.variables.GITHUB_BRANCH> When scanning a repository, you will generally use the repository name and branch for the target name and variant.

        • advanced :

        • ingestion :

          • file : /shared/scan_results/semgrep.sarif

Here's a YAML example:

Run the pipeline and verify your results

This is a good time to run your pipeline and verify that it can scan the repo and ingest the results correctly.

  1. Click Run and set the GITHUB_REPO and GITHUB_BRANCH variables. (You don't need to set the image variables.)

    If you forked the dvpwa repository repo into your GitHub account and want to use that, set the fields like this:

    • GITHUB_REPO = dvpwa

    • GITHUB_BRANCH= master

  2. Click Run Pipeline and wait for the execution to finish. You can then view your scan results in Vulnerabilities tab.

Add the image build step

Assuming that the Semgrep scanner detected no critical vulnerabilities, the next step is to build a local image using the Dockerfile in your codebase.

  1. Add a Run step after the Semgrep ingest step.

  2. Configure the step as follows:

    1. Name = build_local_image

    2. Command =

    3. Open Optional Configuration and set the following options:

      1. Container Registry — When prompted, select Account and then your Docker Hub connector.

      2. Image = docker

Add a Run step and configure it as follows:

  • type: Run

  • name: A name for the step.

  • identifier: A unique step ID.

  • spec :

    • connectorRef : YOUR_DOCKERHUB_CONNECTOR

    • image : returntocorp/semgrep

    • shell : Sh

    • command : |-

    • envVariables:

      • SEMGREP_APP_TOKEN: <+secrets.getValue("YOUR_SEMGREP_TOKEN_SECRET")>

Here's an example:

Add the Aqua-Trivy scan/ingest step

Add an Aqua Trivy step to your pipeline after the build step and configure it as follows:

  1. Scan Mode = Orchestration In orchestrated mode, the step runs the scan and ingests the results in one step.

  2. Target name — Click the "tack" button on the right side of the input field and select Expression. Then enter the following expression: <+stage.variables.DOCKERHUB_USERNAME>/<+stage.variables.DOCKER_IMAGE_LABEL>

  3. Target variant — Select Expression for the value type, then enter the following expression: <+stage.variables.DOCKER_IMAGE_TAG>

  4. Container image Type = Local Image

  5. Container image name — Select Expression for the value type, then enter the following expression: <+stage.variables.DOCKERHUB_USERNAME>/<+stage.variables.DOCKER_IMAGE_LABEL>

  6. Container image tag — Select Expression for the value type, then enter the following expression: <+stage.variables.DOCKER_IMAGE_TAG>

  7. Fail on Severity = None

Add an Aqua Trivy step to your pipeline after the build step and configure it as follows:

  • type: AquaTrivy

  • name: A name for the step.

  • identifier: A unique step ID.

  • spec :

    • mode : orchestration In orchestrated mode, the step runs the scan and ingests the results in one step.

    • config: default

    • target :

      • name : <+stage.variables.DOCKERHUB_USERNAME>/<+stage.variables.DOCKER_IMAGE_LABEL>

      • type : container

      • variant : <+stage.variables.DOCKER_IMAGE_TAG> When scanning an image, you generally use the image label and tag for the target name and variant .

      • advanced :

      • privileged: true

      • image:

        • type : local_image

        • name: <+stage.variables.DOCKERHUB_USERNAME>/<+stage.variables.DOCKER_IMAGE_LABEL>

        • tag: <+stage.variables.DOCKER_IMAGE_TAG>

Here's an example:

Run the pipeline and verify your results

This is a good time to run your pipeline and verify that it can scan the repo and ingest the results correctly.

  1. Click Run and set the GitHub and Docker variables. (You don't need to set the image variables.)

    If you forked the dvpwa repository repo into your GitHub account and want to use that, set the fields like this:

    • GITHUB_REPO = dvpwa

    • GITHUB_BRANCH= master

    • DOCKER_IMAGE_LABEL = dvpwa

    • DOCKER_IMAGE_TAG= master-scantest-DONOTUSE

    Input sets enable you to reuse a single pipeline for multiple scenarios. You can define each scenario in an input set and then select the relevant input setat runtime. To save these inputs, click Save as New Input Set.

  2. Click Run Pipeline and wait for the execution to finish. You can then view your scan results, for both the repo and the image, in Vulnerabilities tab.

Add the image push step

Assuming that the Aqua Trivy scanner detected no critical vulnerabilities, you can now push your new image to Docker Hub.

  1. Add a Run step after the Aqua Trivy scan/ingest step.

  2. Configure the step as follows:

    1. Name = push_image

    2. Command =

    3. Open Optional Configuration and set the following options:

      1. Container Registry — Select your Docker Hub connector.

      2. Image = docker

Add a Run step after the Bandit scan step and configure it as follows:

  • type: Run

  • name: A name for the step.

  • identifier: A unique step ID.

  • spec :

    • connectorRef : MY_DOCKERHUB_CONNECTOR

    • image : docker

    • shell : Sh

    • command : |-

Here's an example:

YAML pipeline example

Here's an example of the pipeline you created in this workflow. If you copy this example, replace the placeholder values with appropriate values for your project, organization, and connectors.

PreviousBuild/scan/push pipelines for container images
Next(STO/CI licenses) Create a build-scan-push pipeline

Last updated

Was this helpful?