CI Pipeline Tutorial
This tutorial shows you how to build a simple, two-stage CI Pipeline in Harness. Setting up and running the Pipeline will take about 30 minutes.
The Pipeline will build and run a unit test on a codebase, upload the artifact to Docker Hub, and then run integration tests.
You will use publicly-available code, images, and your Github and Docker Hub accounts.
Objectives
You'll learn how to create a CI Pipeline that does the following:
- Clone a code repo for an app.
- Use a Kubernetes cluster build farm.
- Build the code and run unit tests in the build farm.
- Package the app as a Docker image and upload it to Docker Hub.
- Pull the uploaded image to the build farm as a service dependency.
- Run an integration test against the app.
Before You Begin
Make sure you have the following set up before you begin this tutorial:
- Github account: This tutorial clones a codebase from a Github repo. You will need a Github account so Harness can connect to Github.
- Docker Hub account and repo: You will need to push and pull the image you build to Docker Hub. You can use any repo you want, or create a new one for this tutorial.
- Kubernetes cluster for Harness Delegate and build farm:
- You'll need a Kubernetes cluster for Harness to use for the Harness Delegate and as a build farm. Ensure you have a cluster that meets the following requirements:
- Number of pods: 3 (two pods for the Harness Delegate, the remaining pod for the build farm).
- Machine type: 4vCPU
- Memory: 16GB RAM. The build farm and Delegate requirements are low but the remaining memory is for Kubernetes, the Docker container, and other default services.
- Networking: Outbound HTTPS for the Harness connection, and to connect to Docker Hub. Allow TCP port 22 for SSH.
- Namespace: When you install the Harness Delegate, it will create the
harness-delegate
namespace. You'll use the same namespace for the build farm.
- You'll need a Kubernetes cluster for Harness to use for the Harness Delegate and as a build farm. Ensure you have a cluster that meets the following requirements:
Google Kubernetes Engine (GKE) Autopilot is not supported at this time.
A Kubernetes service account with permission to create entities in the target namespace is required. The set of permissions should include list
, get
, create
, and delete
permissions. In general, the cluster-admin permission or namespace admin permission is enough.
For more information, see User-Facing Roles from Kubernetes.
Visual Summary
Here's an architecture diagram of the very simple setup we use for Harness CI:
You will install the Harness Delegate in the same cluster you use for the build farm. The Delegate will create the namespace
harness-delegate
. You'll use that namespace for both the Delegate and build farm. You can change it if you like.
Video Overview
Here's a quick video that provides an overview of Harness CI:
Option: Use Your Own Code Repo
For this tutorial, we use a codebase located at:
https://github.com/keen-software/goHelloWorldServer
All steps in this tutorial work with any git repo, so you can use your own code repo instead.
Step 1: Start a New Pipeline
Pipelines are a collection of one or more stages. They manage and automate builds, testing, deployments, and other important build and release stages.
- In Harness, click Projects and create a new Project with the following settings:
- Name: CI Quickstart
- Organization: default
- Invite People to Collaborate: You don't need to add yourself.
- In the Modules page, click Continuous Integration.
- Click Pipelines and then Create a Pipeline.
- Enter the name CI Pipeline and click Start.
As you enter a name for the Pipeline, the ID for the Pipeline is created. A Pipeline name can change, but an ID is permanent. The ID is how you can reference subordinate elements of a Pipeline, such as the names of variables within the Pipeline.
Step 2: Set Up the Build Stage
The "work horse" of most CI Pipelines is the Build Stage. This is where you specify the end-to-end workflow for your build: the codebase to build, the infrastructure to build it, where to post the finished artifact, and any additional tasks (such as automated tests or validations) you want the build to run.
To run a build, a Build Stage needs to connect to the codebase, the build infrastructure, and the artifact repository. A Connector is a configurable object that connects to an external resource automatically.
In this tutorial you'll create a Connector to a GitHub repo. You'll also create a Delegate service that handles communications between Harness and your build infrastructure. Later in this tutorial, you'll create a Connector to a Docker Hub repo so the Build Stage can post the resulting artifact.
Create the Build Stage
- In Pipeline Studio, click Add Stage and select Build. The About your Stage screen appears.
- In Stage Name, enter Build Test and Push.
- Under Configure Codebase, cick Select Connector (under Configure Codebase).
- In the Create or Select an Existing Connector window, select New Connector.
- For the Connector type, select GitHub Connector.
Create a Connector to your Codebase
You'll now create a new Connector to the GitHub codebase. Set up the Connector as follows.
Connector Overview
- Name: CI QuickStart
Connector Details
You can create Connectors for GitHub accounts or specific repos. If you use an account Connector, you specify the repo when you run the build. In this case, we'll connect to a specific repo.
- URL Type: Repository
- Connection Type: HTTP
- GitHub Repository URL:
https://github.com/keen-software/goHelloWorldServer.git
Connector Credentials
Now you need to specify the username and Personal Access Token for the Connector to use. Harness secrets are safe. They're stored in the Harness Secret Manager. You can also use your own Secret Manager with Harness.
- Username: The username for your GitHub account.
- Personal Access Token: Create a Harness Secret for the Personal Access Token you use for your GitHub repo.
- Select Enable API Access.
Connector Connectivity Mode
You can choose to establish connections directly from your Harness platform or through a Delegate service running within your corporate firewall.
The Harness Delegate is a local service that connects your infrastructure, collaboration, verification, and other providers with your Harness platform. For this tutorial, you'll install a Delegate in your Kubernetes cluster.
- Connect to the provider: Connect through a Harness Delegate
Delegate Setup
You should now be in the Delegates Setup screen of the GitHub Connector wizard. Click Install new Delegate.
Delegate Location
You can install the Delegate in different locations. Usually it makes sense to install and run the Delegate on a pod in your Kubernetes build infrastructure. You'll do this in the next steps.
- Select Kubernetes.
Delegate Details
Now you specify the Delegate name, size and permissions. You can use a small delegate for this tutorial and leave the other options at their defaults.
- Specify the following:
- Delegate name: quickstart
- Delegate size: Small
- Delegate Permissions: Install Delegate with cluster-wide read/write access
Delegate Install
Harness now generates and displays a workspace-definition YAML file that you can install in your build infrastructure.
- Click Download Script. This downloads the YAML file for the Kubernetes Delegate.
- Open a terminal and navigate to where the Delegate file is located. You will connect to your cluster using the terminal so you can simply run the YAML file on the cluster.
- In the same terminal, log into your Kubernetes cluster. In most platforms, you select the cluster, click Connect, and copy the access command.
- Install the Harness Delegate using the harness-delegate.yaml file you just downloaded. Click Next in the Harness UI, then run the command shown. For example:
kubectl apply -f harness-delegate.yaml
You should see output similar to this:
% kubectl apply -f harness-delegate.yaml
namespace/harness-delegate-ng created
clusterrolebinding.rbac.authorization.k8s.io/harness-delegate-ng-cluster-admin created
secret/ci-quickstart created
statefulset.apps/ci-quickstart created
service/delegate-service created
Connect to the Delegate
- Return to the Harness UI. It might take a few minutes to verify the Delegate. Once it is verified, close the wizard.
- Back in Delegates Setup, you can select the new Delegate:
- In the list of Delegates, you can see your new Delegate and its tags.
- Select the Connect using Delegates with the following Tags option.
- Enter the tag of the new Delegate and click Save and Continue.
- Wait for the connection test to complete and then click Finish.
- Back in About Your Stage, the Connector and repo are displayed.
- Click Set Up Stage. The new stage is added to the Pipeline.
Step 3: Define the Build Farm Infrastructure
Here you'll define the build farm infrastructure and the stage steps.
- Under Select a Kubernetes Cluster, click Select.
- Click New Connector and set up the new Connector as follows:
- Name: ci-delegate
- Details: Select Use the credentials of a specific Harness Delegate. You'll add this Delegate next.
- Delegates Setup: Select the Kubernetes Delegate you added earlier using its Tags.
- Connection Test: Wait for the test to finish and then click Finish. The new Connector is added to the Kubernetes Cluster field.
- In Namespace, enter the namespace
harness-delegate-ng
. Click Next to proceed to the Execution tab.
Now that the build farm infrastructure is set up, you can run unit tests against your code.
Step 4: Build and Run Unit Tests
Next, we'll add a Run step to the stage that will build the code and run a unit test.
- You should now be in the Execution tab. Click Add step and then select Run (under Build).
The Run step executes one or more commands on a container image. Configure the step as follows.
- Name = Run Unit Tests.
Configure Run Step
- Name: Run Unit Tests
- Container Registry: Click Select and then +New Connector.
- Select your Connector Type: Docker Registry
You will now create a new Connector to your DockerHub account as follows.
Docker Connector Overview
- Name: Docker Quickstart
Docker Connector Details
- Docker Registry URL:
https://index.docker.io/v1/
- Provider Type: Docker Hub
- Username: The username for your Docker Hub account
- Password: Create a Harness Secret for the Personal Access Token you use with your Docker Hub account. You can also use your own Secret Manager.
Delegates Setup
- Delegates Setup: Select the new Delegate you installed previously using its Tags.
- Wait for the Connector test to complete and then click Finish.
- You should now be in the Configure Run Step pane, with the new Connector in the Container Registry setting. Configure the step as follows:
- Name: Run Unit Tests
- Container Registry: This should show the Docker Hub Connector you just created.
- Image:
golang:1.15
- Command:
The last line contains thego get gotest.tools/gotestsum
gotestsum --format=standard-verbose --junitfile unit-tests.xml || true
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -a -tags netgogo build
command to compile the package, along with its dependencies.- Report Paths (Optional Configuration):
*.xml
- Click Apply Changes.
- In the Pipeline Studio, click Save.
This Run step will intentionally fail the test. This will be useful to see how a test failure is recorded in Harness.
Step 5: Build and Push Image to Docker Hub
Next, you'll add a step to build your container and push it to your Docker Hub repo. You'll need a repo in your Docker Hub account to receive the artifact.
- You should be in the Execution tab of the Built Test and Push Stage. Click Add step, then click Build and Push an Image to Docker Registry.
- Configure the step as follows:
- Name: Build and push an image to Docker Registry
- Docker Connector: Select the Docker Hub Connector you set up previously.
- Docker Repository:
- Create a
ciquickstart
repository in your Docker Hub account. - Then enter
<your_repo>/ciquickstart
where<your_repo>
is your Docker Hub username.
- Create a
- Tags:
<+pipeline.sequenceId>
This tag is a built-in Harness variable that represents the Build ID number, for exampleBuild ID: 9
. The pipeline uses the Build ID to tag the image that it pushes in stage 1 and pulls in stage 2. You will see the Build ID when you run the pipeline. You will also use this variable to identify the image location when you set up the Configure Service Dependency step in the next stage. - Click Apply Changes to return to the Pipeline Studio.
- Click Save to save the Pipeline. You can run the Pipeline now, or you can continue and add the Integration Test stage.
Step 6: Create the Integration Test Stage
Now you have a Stage to clone, build, containerize, and then push your image to Docker Hub. In this step you'll add a Stage to pull that image, run it in a container, and run integration tests on it.
- Click Add Stage, and select Build.
- Enter the name Run Integration Test, disable Clone Codebase, and then click Set Up Stage.
Define Stage Infrastructure
Here you configure the stage to use the same infrastructure as the previous stage.
- Click Infrastructure.
- Select Propagate from an existing stage.
- Select the previous stage. This will use the same Kubernetes cluster and namespace as the first stage.
- Click Next.
Add Built Image from Stage 1 as a Service Dependency
In the Build Test and Push stage, you built your code and pushed your built image to Docker Hub. Now, in Run Integration Test, you will identify the image as a Service Dependency for your test stage. Harness will pull the image onto the container in your infrastructure. Next, it will start the Hello World Server in the image.
- In the Run Integration Test stage > Execution tab, click Add Service Dependency.
- Configure the dependency:
- Dependency Name: Run Hello World Server
- Container Registry: The same Docker Hub Connector you used in the Build Test and Push stage.
- Image:
<your_repo>/ciquickstart:<+pipeline.sequenceId>
Replace<your_repo>
with the name of the Docker Hub repo you are using for this tutorial.
- Click Apply Changes.
Notice that the Image field uses the same variable <+pipeline.sequenceId>
for the Image that you used in the previous stage. This tells Harness to pull the image with the same tag as the image pushed previously. Here's an example:
Build and Test and Push (stage 1) Build and Push to Docker Hub step | Run Integration Test (stage 2) Configure Service Dependency step |
![]() | ![]() |
Add Integration Test Step
Next, we can run an integration test. We'll simply test the connection to the server.
- Click Add Step, and then click Run.
- Configure the step as follows:
- Name: test connection to server
- Container Registry: The Docker Hub Connector you've been using.
- Image:
curlimages/curl:7.73.0
- Command:
sleep 10
curl localhost:8080
curl localhost:8080?Hello!_I_am_a_nice_demo!- Click Apply Changes. You are now back the Pipeline Studio.
Step 7: Run the Pipeline
The Build Pipeline is complete. You can now run it.
- Click ↑Save.
- Click Run. The Pipeline Inputs settings appear.
- Under CI Codebase, select Git branch.
- In Git Branch, enter the name of the branch where the codebase is, such as main.
- Click Run Pipeline.
Step 8: View the Results
Click each stage's steps to see the logs in real time.
Click Console View to see more details.
You can see the build number in the Build and push image to Docker Hub step used as an image tag. For example, 11:
--destination=myrepo/ciquickstart:11
In the Initialize step of the Run Integration Test stage, you can see the image with the same tag pulled from your Docker Hub repo:
8 Pulling image "myrepo/ciquickstart:11"
9 Successfully pulled image "myrepo/ciquickstart:11" in 1.878887425s
Now look in your Docker Hub repo. You can see the image and tag:
Click Tests. You can see the failed test from stage 1.
Click Builds. Your build appears.
Congratulations! You have a CI Pipeline that builds and tests your code.
Using YAML
The entire Pipeline is available as YAML, also.
- In Builds, click more options (︙) and select Edit Pipeline.
- Click YAML.
You can see the entire Pipeline as YAML. You can edit anything in the Pipeline and run it again.
pipeline:
name: CI Pipeline
identifier: CI_Pipeline
allowStageExecutions: false
projectIdentifier: CI_QuickStart_20220401
orgIdentifier: default
tags: {}
properties:
ci:
codebase:
connectorRef: CI_QuickStart
build: <+input>
stages:
- stage:
name: Build Test and Push
identifier: Build_Test_and_Push
type: CI
spec:
cloneCodebase: true
infrastructure:
type: KubernetesDirect
spec:
connectorRef: cidelegate
namespace: harness-delegate-ng
execution:
steps:
- step:
type: Run
name: Run Unit Tests
identifier: Run_Unit_Tests
spec:
connectorRef: Docker_Quickstart
image: golang:1.15
command: |-
go get gotest.tools/gotestsum
gotestsum --format=standard-verbose --junitfile unit-tests.xml || true
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -a -tags netgo
privileged: false
reports:
type: JUnit
spec:
paths:
- "*.xml"
- step:
type: BuildAndPushDockerRegistry
name: Build and push image to Docker Hub
identifier: Build_and_push_image_to_Docker_Hub
spec:
connectorRef: Docker_Quickstart
repo: dockerhubuser/ciquickstart
tags:
- <+pipeline.sequenceId>
- stage:
name: Run Integration Test
identifier: Run_Integration_Test
type: CI
spec:
cloneCodebase: false
infrastructure:
useFromStage: Build_Test_and_Push
serviceDependencies:
- identifier: Run_Hello_World_Server
name: Run Hello World Server
type: Service
spec:
connectorRef: Docker_Quickstart
image: dockerhubuser/ciquickstart:<+pipeline.sequenceId>
execution:
steps:
- step:
type: Run
name: test connection to server
identifier: test_connection_to_server
spec:
connectorRef: Docker_Quickstart
image: curlimages/curl:7.73.0
command: |-
sleep 10
curl localhost:8080
curl localhost:8080?Hello!_I_am_a_nice_demo!