Skip to main content

Tutorial: Create a build-scan-push pipeline (STO and CI)

In this tutorial, you'll create an end-to-end pipeline that uses STO and CI steps to build an image and pushes it to Docker Hub only if the codebase and image contain no critical vulnerabilties. This pipeline uses two popular open-source tools:

You can copy/paste the YAML pipeline example below into Harness and update it with your own infrastructure, connectors, and access tokens.

The following steps describe the workflow:

  1. An STO Bandit step scans the codebase and ingests the scan results.

  2. If the code has no critical vulnerabilities, a CI Build and Push step builds a test image and pushes it to Docker Hub.

  3. An Aqua Trivy step scans the image and ingests the results.

  4. If the image has no critical vulnerabilities, another Build and Push step pushes a prod image to Docker Hub.

scan-build-scan-push tutorial pipeline

Prerequisites

  • This tutorial has the following prerequisites:

    • Harness STO and CI module licenses.

    • You must have a Security Testing Developer or SecOps role assigned.

    • A basic understanding of key STO concepts and good practices is recommended. Your first STO pipeline is a good introduction.

    • GitHub requirements:

      • A GitHub account and access token.

      • A GitHub connector that specifies your account (http://github.com/my-account) but not a specific repository (http://github.com/my-account/my-repository).

      • Your GitHub account should include a repository with Python code. The repo should also include a Dockerfile for creating an image.

        This tutorial uses the dvpwa repository as an example. The simplest setup is to fork this repository into your GitHub account.

    • Docker requirements:

      • A Docker Hub account and access token.
      • A Docker connector is required to push the image.

    • Your GitHub and Docker Hub access tokens must be stored as Harness secrets.

Set up your codebase​

This tutorial uses Bandit to scan the target repository https://github.com/williamwissemann/dvpwa (specified in the Codebase for this pipeline).

  1. Fork the following example repository into your GitHub account. This is a Python repo with known vulnerabilities: https://github.com/williamwissemann/dvpwa.

  2. If you don't have a GitHub connector, do the following:

    1. In your Harness project, select Project Setup > Connectors.
    2. Select New Connector, then select Code Repositories > GitHub.
    3. Set the GitHub connector settings as appropriate.
      • Use Account for the URL type.
      • This tutorial uses Harness Cloud, so select Connect through Harness Platform when prompted for the connectivity mode.

Set up your pipeline​

Do the following:

  1. Select Security Testing Orchestration (left menu, top) > Pipelines > Create a Pipeline. Enter a name and click Start.

  2. In the new pipeline, select Add stage > Build.

  3. Set up your stage as follows:

    1. Enter a Stage Name.

    2. Select Third-party Git provider and then select your GitHub connector.

    3. For Repository Name, click the type selector ("tack" button on the right) and then select Runtime Expression.

  4. Expand Overview > Advanced and add the following stage variables.

    You'll be specifying runtime inputs for some of these variables. This enables you to specify the code repo, branch, image label, and image tag, and other variables at runtime.

    • GITHUB_USERNAME — Select Secret as the type and enter your GitHub login name.

    • GITHUB_REPO — Select String for the type and Runtime Input for the value (click the "tack button" to the right of the value field).

    • GITHUB_BRANCH — Select String and Runtime Input.

    • DOCKERHUB_USERNAME — Select String as the type and enter your DockerHub login name.

    • DOCKER_IMAGE_LABEL — Select String and Runtime Input.

    • DOCKER_IMAGE_TAG — Select String and Runtime Input.

  5. In the Pipeline Editor, go to Infrastructure and select Cloud, Linux, and AMD64 for the infrastructure, OS, and architecture.

    You can also use a Kubernetes or Docker build infrastructure, but these require additional work to set up. For more information, go to Set up a build infrastructure for STO.

note

The following step is required for Kubernetes or Docker infrastructures only. If you're using Harness Cloud, go to Scan the code.

Add a Docker-in-Docker background step​

The following use cases require a Docker-in-Docker background step in your pipeline:

  • Container image scans on Kubernetes and Docker build infrastructures
  • Security steps (not step palettes) on Kubernetes and Docker build infrastructures
    • Required for all target types and Orchestration/DataLoad modes

The following use cases do not require Docker-in-Docker:

Set up a Docker-in-Docker background step

  1. Go to the stage where you want to run the scan.

  2. In Overview, add the shared path /var/run.

  3. In Execution, do the following:

    1. Click Add Step and then choose Background.

    2. Configure the Background step as follows:

      1. Dependency Name = dind

      2. Container Registry = The Docker connector to download the DinD image. If you don't have one defined, go to Docker connector settings reference.

      3. Image = docker:dind

      4. Under Entry Point, add the following: dockerd

        In most cases, using dockerd is a faster and more secure way to set up the background step. For more information, go to the TLS section in the Docker quick reference.

      If the DinD service doesn't start with dockerd, clear the Entry Point field and then run the pipeline again. This starts the service with the default entry point.

      1. Under Optional Configuration, select the Privileged checkbox.
Configure the background step

Scan the code​

  1. In the Pipeline Studio, go to Execution and add a Bandit step to your pipeline.

  2. Configure the step as follows:

    1. Scan Mode = Orchestration

    2. Target name — Click the value-type selector (tack button to the right of the input field) and select Expression. Then enter the following expression: <+stage.variables.GITHUB_REPO>

    3. Target variant — Select Expression as the value type and enter the following: <+stage.variables.GITHUB_BRANCH>

      When scanning a code repo, you generally want to specify the repo name and branch for the target.

info

In most cases, you want to set the Fail on Severity when you add a scan step to your pipeline. In this tutorial, you're setting the first of two scan steps. In this case, you can keep Fail on Severity set to None and set it after you finish setting up and testing your pipeline.

Analyze the results​

At this point, you might want to run a scan and view the detected issues.

  1. Select Save, and then select Run.

  2. In Run Pipeline, enter the repository and target settings you want to use. If you're using the dvpwa repository, enter the following:

    1. Under Codebase:
      • Repository name : dvpwa
      • Branch name : master
    2. Under Stage:
      • Target name : dvpwa (= the repo name)
      • Target variant : master (= the branch name)

  3. Run the pipeline. When the execution finishes, select Security Tests to view the scan results.

Build and push a test image​

Harness CI includes a set of Build and Push steps that take a code repo with a Dockerfile, build a container image, and push it to an artifact repository.

You'll now add one of these steps to build and push to your Docker Hub account.

  1. Add a Build and Push to Docker Registry step after the Bandit step.

  2. Configure the step as follows:

    1. Name = build_push_test_image

    2. Docker Connector — Select your Docker Hub connector.

    3. Docker Repository — Select Expression for the value type, then enter the following:

      <+stage.variables.DOCKERHUB_USERNAME>/<+stage.variables.DOCKER_IMAGE_LABEL>

    4. Tags— Select Expression for the value type, then enter the following:

      <+stage.variables.DOCKER_IMAGE_TAG>-scantest-DONOTUSE

Scan the test image​

Add an Aqua Trivy step to your pipeline after the build step and configure it as follows:

  1. Scan Mode = Orchestration In orchestrated mode, the step runs the scan and ingests the results in one step.

  2. Target name — Click the "tack" button on the right side of the input field and select Expression. Then enter the following expression: <+stage.variables.DOCKERHUB_USERNAME>/<+stage.variables.DOCKER_IMAGE_LABEL>

  3. Target variant — Select Expression for the value type, then enter the following expression: <+stage.variables.DOCKER_IMAGE_TAG>-scantest-DONOTUSE

  4. Container image Type = V2

  5. Domain = docker.io

  6. Container image name — Select Expression for the value type, then enter the following expression: <+stage.variables.DOCKERHUB_USERNAME>/<+stage.variables.DOCKER_IMAGE_LABEL>

  7. Container image tag — Select Expression for the value type, then enter the following expression: <+stage.variables.DOCKER_IMAGE_TAG>

  8. Fail on Severity = Critical

Run the pipeline and verify your results​

This is a good time to run your pipeline and verify that it can scan the image.

  1. Click Run and set the GitHub and Docker variables.

    If you forked the dvpwa repository repo into your GitHub account and want to use that, set the fields like this:

    • GITHUB_REPO = dvpwa
    • GITHUB_BRANCH= master
    • DOCKER_IMAGE_LABEL = dvpwa
    • DOCKER_IMAGE_TAG = 1.

    With this setup, you'll build and push an image with a tag that looks like this: 1.x-scantest-DONOTUSE

    tip

    Input sets enable you to reuse a single pipeline for multiple scenarios. You can define each scenario in an input set and then select the relevant input set at runtime. To save these inputs, click Save as New Input Set.

  2. Click Run Pipeline and view the results in Security Tests.

Build and push the prod image​

Assuming that the Trivy scan detected no critical vulnerabilities, you can now build and push a prod version of your image to Docker Hub. This step is identical to the previous test image step, except for the image tag: the test image tag is 1.x-scantest-DONOTUSE and the prod image tag is 1.x.

  1. Add a Build and Push to Docker Registry step after the Semgrep ingest step.

  2. Configure the step as follows:

    1. Name = build_push_test_image

    2. Docker Connector — Select your Docker Hub connector.

    3. Container image name — Select Expression for the value type, then enter the following expression: <+stage.variables.DOCKERHUB_USERNAME>/<+stage.variables.DOCKER_IMAGE_LABEL>

    4. Container image tag — Select Expression for the value type, then enter the following expression: <+stage.variables.DOCKER_IMAGE_TAG><+pipeline.sequenceID>

YAML pipeline example​

Here's an example of the pipeline you created in this tutorial. If you copy this example, replace the placeholder values with appropriate values for your project, organization, connectors, and access token.

pipeline:
  name: sto-ci-tutorial-test
  identifier: stocitutorialtest
  projectIdentifier: YOUR_HARNESS_PROJECT_ID
  orgIdentifier: YOUR_HARNESS_ORGANIZATION_ID
  tags: {}
  properties:
    ci:
      codebase:
        connectorRef: YOUR_CODEBASE_CONNECTOR_ID
        repoName: <+input>
        build: <+input>
  stages:
    - stage:
        name: build_scan_push_with_ci
        identifier: build_scan_push_with_ci
        description: ""
        type: CI
        spec:
          cloneCodebase: true
          platform:
            os: Linux
            arch: Amd64
          runtime:
            type: Cloud
            spec: {}
          execution:
            steps:
              - step:
                  type: Bandit
                  name: scan_python_code
                  identifier: scan_python_code
                  spec:
                    mode: orchestration
                    config: default
                    target:
                      name: <+stage.variables.GITHUB_REPO>
                      type: repository
                      variant: <+stage.variables.GITHUB_BRANCH>
                    advanced:
                      log:
                        level: info
              - step:
                  type: BuildAndPushDockerRegistry
                  name: build_push_test_image
                  identifier: build_push_test_image
                  spec:
                    connectorRef: YOUR_IMAGE_REGISTRY_CONNECTOR
                    repo: <+stage.variables.DOCKERHUB_USERNAME>/<+stage.variables.DOCKER_IMAGE_LABEL>
                    tags:
                      - <+stage.variables.DOCKER_IMAGE_TAG><+pipeline.sequenceId>-scantest-DONOTUSE
              - step:
                  type: AquaTrivy
                  name: scan_test_image
                  identifier: scan_test_image
                  spec:
                    mode: orchestration
                    config: default
                    target:
                      name: <+stage.variables.DOCKERHUB_USERNAME>/<+stage.variables.DOCKER_IMAGE_LABEL>
                      type: container
                      variant: <+stage.variables.DOCKER_IMAGE_TAG><+pipeline.sequenceId>
                    advanced:
                      log:
                        level: info
                    privileged: true
                    image:
                      type: docker_v2
                      name: <+stage.variables.DOCKERHUB_USERNAME>/<+stage.variables.DOCKER_IMAGE_LABEL>
                      domain: docker.io
                      access_token: <+secrets.getValue("YOUR_IMAGE_REGISTRY_ACCESS_TOKEN")
                      tag: <+stage.variables.DOCKER_IMAGE_TAG><+pipeline.sequenceId>-scantest-DONOTUSE
                    sbom:
                      format: spdx-json
              - step:
                  type: BuildAndPushDockerRegistry
                  name: build_push_prod_image
                  identifier: build_push_prod_image
                  spec:
                    connectorRef: YOUR_IMAGE_REGISTRY_CONNECTOR
                    repo: <+stage.variables.DOCKERHUB_USERNAME>/<+stage.variables.DOCKER_IMAGE_LABEL>
                    tags:
                      - <+stage.variables.DOCKER_IMAGE_TAG><+pipeline.sequenceId>
        timeout: 30m
        variables:
          - name: GITHUB_REPO
            type: String
            description: ""
            required: true
            value: <+input>
          - name: GITHUB_BRANCH
            type: String
            description: ""
            required: false
            value: <+input>
          - name: DOCKERHUB_USERNAME
            type: String
            description: ""
            required: false
            value: myusername
          - name: _IMAGE_LABEL
            type: String
            description: ""
            required: false
            value: <+input>
          - name: DOCKER_IMAGE_TAG
            type: String
            description: ""
            required: false
            value: <+input>