CI Pipeline Creation Overview

An overview of CI pipeline components and Build stage settings

This topic provides an overview of CI pipeline creation and configuration, including common components, such as stages, steps, and codebases, as well as advanced settings.

For information about Harness CI's key features, Harness Platform components in Harness CI, and definitions of terms like build infrastructure, delegate, connector, and so on, go to Harness CI key concepts.

Pipelines

A CI pipeline is an end-to-end integration workflow that, in its simplest form, pulls a codebase, builds an artifact, and then uploads the artifact to storage or a registry such as Docker Hub, Google Cloud Registry, JFrog Artifactory, and many others. Pipelines are comprised of stages and steps.

You can run a pipeline manually or set up triggers to automatically run it on a schedule or when an event occurs, such as a Git merge in your codebase.

Create a Harness CI pipeline

1. Log in to your Harness account. If you don't have an account yet, go to Try Harness CI.

2. Go to the Continuous Integration module, select Pipelines, and then select Create a Pipeline.

3. Enter a Name for the pipeline. Description and Tags are optional.

4. Select Inline to store your pipeline configuration in Harness, or select Remote to store your pipeline as code in a Git repository.

5. If you want to use a pipeline template, select Start with Template.

6. Select Start.

You can now add stages and steps to the pipeline, as well as configure pipeline settings.

You can also import pipelines from Git.

Pipeline settings

In addition to a default codebase, you can configure the following settings at the pipeline level:

• Input sets and overlays

• Triggers

• Variables

• Notifications

• Flow Control: Synchronization barriers

• Policy Sets

• Advanced Options: Pipeline Timeout Settings, Stage Execution Settings, and Delegate Selectors

Harness Input sets are collections of runtime inputs for a pipeline run. Overlays are groups of input sets. You can use overlays to provide multiple input sets when you run a pipeline.

With input sets and overlays, you can use the same pipeline for multiple scenarios. You can define each scenario in an input set or overlay, and then select the appropriate scenario at runtime.

Visual and YAML editors

Harness CI provides two interchangeable modes for creating pipelines: The Visual editor and the YAML editor.

• The Visual editor provides a GUI experience where you can easily configure settings, add and remove steps and stages, and drag-and-drop steps and stages to rearrange them, organize them in parallel, or add or remove them from step groups.

• The YAML editor provides a text editor experience for creating pipelines.

You can freely switch between the two editors. When editing a pipeline in Harness, use the selector at the top of the Pipeline Studio to switch between the Visual and YAML editors.

For more information about Harness YAML, go to Write pipelines in YAML.

Stages

A CI stage is a subset of a pipeline that contains one major segment of the CI workflow. All stages have stage settings and one or more steps.

To add a stage to a pipeline, select Add Stage in the Pipeline Studio. The most essential stage for CI pipelines is the Build stage, which includes steps that test code, build and push images, and upload artifacts, among other steps.

CI Build stage settings include codebase configuration, build infrastructure, shared paths, and other advanced settings.

To make pipelines more versatile, you can create templates, use stage variables, and create custom stages, among other optimization strategies.

Codebases

CI pipelines build and test code that is pulled from a Git code repository. When you add a Build stage to a CI pipeline, you specify the Git account and repository where your code is stored. The codebase declared in the first stage in a pipeline becomes the pipeline's default codebase. You can use this codebase for later stages in the pipeline, or you can use multiple codebases in the same pipeline. For more information, go to Configure codebases.

Harness uses code repo connectors to connect to Git repositories.

By enabling the feature flag CI_DYNAMIC_BRANCH_SELECTOR, Git branches in the codebase branch selector will be dynamically populated when running a pipeline. Contact Harness Support to enable this feature.

Build infrastructure

All stages have an infrastructure definition, which represents the build infrastructure used by a CI pipeline: the target clusters, hosts, and so on. Build infrastructure components and specifications depend on the build infrastructure you choose. For more information, go to Which build infrastructure is right for me.

TUTORIALS

• Build on Harness Cloud build infrastructure

• Build on a Kubernetes cluster build infrastructure

Shared Paths

You can use Shared Paths in a stage to share data across steps or customize cache paths for Cache Intelligence.

When a pipeline runs, it creates a temporary volume for each stage called a workspace. During initialization, the stage clones your codebase to the root of the workspace. Then, the steps in the stage run inside the root. The workspace is the current working directory for each step in the stage. The workspace persists for the lifetime of the stage and enables steps in that stage to communicate and share state information. The default shared working directory for a stage is /harness. The workspace is destroyed when the stage ends.

Individual steps can communicate and share state using the workspace filesystem. The workspace is a volume, so filesystem changes persist throughout the stage lifetime. If you need to share additional volumes, you can add Shared Paths. Paths must begin with a forward slash, such as /vol.

For example, the maven m2 repo is stored in /root/.m2 by default. If your Build stage uses Maven, you can specify /root/.m2 as a Shared Path so that all steps in that stage can access that directory.

If you need to shared data across stages, use caching.

Steps

A stage contains one or more steps. Each step is a series of commands that perform a task. For example, A Build and Push step builds an image and pushes it to a cloud repo, a Run step runs a series of shell commands, and so on. Go to the following documentation for details about how you can use various CI steps in your pipelines:

• Build and push images or upload and download artifacts

• Run tests

• Manage dependencies

• Share and cache CI data

• Run scripts

• Use plugins

• Security step (Harness STO)

Use step groups to organize complex stages that have many steps.

Use templates to share and utilize pre-built pipelines, stages, and steps.

Advanced settings

Stages and steps have advanced settings you can use to control the flow of operations.

Conditional Executions

Use conditional execution settings to specify when a stage or step should run. For example, you can specify that a particular stage should run only if the prior pipeline or stage failed.

You can specify conditional execution settings for an entire stage and for individual steps. A stage's conditional execution settings apply to all steps in that stage that don't have their own step-level conditional execution settings. A step's conditional execution settings overrides the stage's conditional execution settings.

Looping Strategies

Looping strategies enable you to run a stage or step multiple times with different inputs. This eliminates the need to copy the same stage or step for each variation you need. It also makes the pipeline more organized, clean, and easy to maintain. Looping strategies enable use cases such as:

• You want to test a UI feature in multiple browsers and platforms. You can define a matrix that specifies the browsers and platforms to test.

• You want to build artifacts for multiple JDK versions in the same Build Stage.

• You have a Build Pipeline with 20 unit tests. To speed up execution, you want to run the tests in parallel across 4 jobs that run 5 tests each.

Failure Strategies

Failure strategies define how your stages and steps handle different failure conditions.

Each failure strategy is comprised of the following:

• Error conditions that trigger the failure strategy.

• Actions to take when the specified error conditions occur.

Failure strategies are a critical pipeline design component that determine what causes a stage or step to fail and what to do when a failure occurs.

Optimization strategies

You can optimize your CI pipelines to make them faster, more efficient, and more versatile. For example:

Dependencies

• Harness CI Intelligence

• Caching and data sharing

• Dependency management strategies

Variables

Expressions and runtime inputs make your pipelines more dynamic.

• Fixed values, runtime inputs, and expressions

• Variables and expressions

• CI environment variables

• Input sets and overlays

Automation

Strategies to automate and manage build sequences include:

• Pipeline chaining

• Failure strategies

• Conditional executions

• Looping strategies

• Triggers

Integration

There are many ways you can incorporate third party tools and services in your CI pipelines, including:

• Plugins

• Secrets Managers

• Notifications

• Branch protection and status checks

Resource consumption

• Understand resource allocation.

• Enforce pipeline concurrency limits.

• Use Queue Intelligence.