Skip to main content

Continuous Integration (CI) FAQs

Can I use Harness CI for mobile app development?​

Yes. Harness CI offers many options for mobile app development.

I have a MacOS build, do I have to use homebrew as the installer?​

No. Your build infrastructure can be configured to use whichever tools you like. For example, Harness Cloud build infrastructure includes pre-installed versions of xcode and other tools, and you can install other tools or versions of tools that you prefer to use. For more information, go to the CI macOS and iOS development guide.

Build infrastructure​

What is build infrastructure and why do I need it for Harness CI?​

A build stage's infrastructure definition, the build infrastructure, defines "where" your stage runs. It can be a Kubernetes cluster, a VM, or even your own local machine. While individual steps can run in their own containers, your stage itself requires a build infrastructure to define a common workspace for the entire stage. For more information about build infrastructure and CI pipeline components go to:

What kind of build infrastructure can I use? Which operating systems are supported?​

For support operating systems, architectures, and cloud providers, go to Which build infrastructure is right for me.

Can I use multiple build infrastructures in one pipeline?​

Yes, each stage can have a different build infrastructure. Additionally, depending on your stage's build infrastructure, you can also run individual steps on containers rather than the host. This flexibility allows you to choose the most suitable infrastructure for each part of your CI pipeline.

Local runner build infrastructure​

Can I run builds locally? Can I run builds directly on my computer?​

Yes. For instructions, go to Set up a local runner build infrastructure.

How do I check the runner status for a local runner build infrastructure?​

To confirm that the runner is running, send a cURL request like curl http://localhost:3000/healthz.

If the running is running, you should get a valid response, such as:

{
 "version": "0.1.2",
 "docker_installed": true,
 "git_installed": true,
 "lite_engine_log": "no log file",
 "ok": true
}

How do I check the delegate status for a local runner build infrastructure?​

The delegate should connect to your instance after you finish the installation workflow above. If the delegate does not connect after a few minutes, run the following commands to check the status:

docker ps
docker logs --follow <docker-delegate-container-id>

The container ID should be the container with image name harness/delegate:latest.

Successful setup is indicated by a message such as Finished downloading delegate jar version 1.0.77221-000 in 168 seconds.

Runner can't find an available, non-overlapping IPv4 address pool.​

The following runner error can occur during stage setup (the Initialize step in build logs):

Could not find an available, non-overlapping IPv4 address pool among the defaults to assign to the network.

This error means the number of Docker networks has exceeded the limit. To resolve this, you need to clean up unused Docker networks. To get a list of existing networks, run docker network ls, and then remove unused networks with docker network rm or docker network prune.

Docker daemon fails with invalid working directory path on Windows local runner build infrastructure​

The following error can occur in Windows local runner build infrastructures:

Error response from daemon: the working directory 'C:\harness-DIRECTORY_ID' is invalid, it needs to be an absolute path

This error indicates there may be a problem with the Docker installation on the host machine.

  1. Run the following command (or a similar command) to check if the same error occurs:

    docker run -w C:\blah -it -d mcr.microsoft.com/windows/servercore:ltsc2022

  2. If you get the working directory is invalid error again, uninstall Docker and follow the instructions in the Windows documentation to Prepare Windows OS containers for Windows Server.

  3. Restart the host machine.

How do I check if the Docker daemon is running in a local runner build infrastructure?​

To check if the Docker daemon is running, use the docker info command. An error response indicates the daemon is not running. For more information, go to the Docker documentation on Troubleshooting the Docker daemon

Runner process quits after terminating SSH connection for local runner build infrastructure​

If you launch the Harness Docker Runner binary within an SSH session, the runner process can quit when you terminate the SSH session.

To avoid this with macOS runners, use this command when you start the runner binary:

./harness-docker-runner-darwin-amd64 server >log.txt 2>&1 &
disown

For Linux runners, you can use a tool such as nohup when you start the runner, for example:

nohup ./harness-docker-runner-darwin-amd64 server >log.txt 2>&1 &

Where does the harness-docker-runner create the hostpath volume directories on macOS?​

The harness-docker-runner creates the host volumes under /tmp/harness-* on macOS platforms.

Why do I get a "failed to create directory" error when trying to run a build on local build infra?​

failed to create directory for host volume path: /addon: mkdir /addon: read-only file system

This error could occur when there's a mismatch between the OS type of the local build infrastructure and the OS type selected in the pipeline's infrastructure settings. For example, if your local runner is on a macOS platform, but the pipeline's infrastructure is set to Linux, this error can occur.

Self-managed VM build infrastructure​

Can I use the same build VM for multiple CI stages?​

No. The build VM terminates at the end of the stage and a new VM is used for the next stage.

Why are build VMs running when there are no active builds?​

With self-managed VM build infrastructure, the pool value in your pool.yml specifies the number of "warm" VMs. These VMs are kept in a ready state so they can pick up build requests immediately.

If there are no warm VMs available, the runner can launch additional VMs up to the limit in your pool.yml.

If you don't want any VMs to sit in a ready state, set your pool to 0. Note that having no ready VMs can increase build time.

For AWS VMs, you can set hibernate in your pool.yml to hibernate warm VMs when there are no active builds. For more information, go to Configure the Drone pool on the AWS VM.

Do I need to install Docker on the VM that runs the Harness Delegate and Runner?​

Yes. Docker is required for self-managed VM build infrastructure.

AWS build VM creation fails with no default VPC​

When you run the pipeline, if VM creation in the runner fails with the error no default VPC, then you need to set subnet_id in pool.yml.

AWS VM builds stuck at the initialize step on health check​

If your CI build gets stuck at the initialize step on the health check for connectivity with lite engine, either lite engine is not running on your build VMs or there is a connectivity issue between the runner and lite engine.

  1. Verify that lite-engine is running on your build VMs.
    1. SSH/RDP into a VM from your VM pool that is in a running state.
    2. Check whether the lite-engine process is running on the VM.
    3. Check the cloud init output logs to debug issues related to startup of the lite engine process. The lite engine process starts at VM startup through a cloud init script.
  2. If lite-engine is running, verify that the runner can communicate with lite-engine from the delegate VM.
    1. Run nc -vz <build-vm-ip> 9079 from the runner.
    2. If the status is not successful, make sure the security group settings in runner/pool.yml are correct, and make sure your security group setup in AWS allows the runner to communicate with the build VMs.
    3. Make sure there are no firewall or anti-malware restrictions on your AMI that are interfering with the cloud init script's ability to download necessary dependencies. For details about these dependencies, go to Set up an AWS VM Build Infrastructure - Start the runner.

AWS VM delegate connected but builds fail​

If the delegate is connected but your AWS VM builds are failing, check the following:

  1. Make sure your the AMIs, specified in pool.yml, are still available.
    • Amazon reprovisions their AMIs every two months.
    • For a Windows pool, search for an AMI called Microsoft Windows Server 2022 Base with Containers and update ami in pool.yml.
  2. Confirm your security group setup and security group settings in runner/pool.yml.

Use internal or custom AMIs with self-managed AWS VM build infrastructure​

If you are using an internal or custom AMI, make sure it has Docker installed.

Additionally, make sure there are no firewall or anti-malware restrictions interfering with initialization, as described in CI builds stuck at the initialize step on health check.

Where can I find logs for self-managed AWS VM lite engine and cloud init output?​

  • Linux
    • Lite engine logs: /var/log/lite-engine.log
    • Cloud init output logs: /var/log/cloud-init-output.log
  • Windows
    • Lite engine logs: C:\Program Files\lite-engine\log.out
    • Cloud init output logs: C:\ProgramData\Amazon\EC2-Windows\Launch\Log\UserdataExecution.log

What does it mean if delegate.task throws a "ConnectException failed to connect" error?​

Before submitting a task to a delegate, Harness runs a capability check to confirm that the delegate is connected to the runner. If the delegate can't connect, then the capability check fails and that delegate is ignored for the task. This can cause failed to connect errors on delegate task assignment, such as:

INFO  io.harness.delegate.task.citasks.vm.helper.HttpHelper - [Retrying failed to check pool owner; attempt: 18 [taskId=1234-DEL] \
java.net.ConnectException: Failed to connect to /127.0.0.1:3000\

To debug this issue, investigate delegate connectivity in your VM build infrastructure configuration:

Harness Cloud​

What is Harness Cloud?​

Harness Cloud lets you run builds on Harness-managed runners that are preconfigured with tools, packages, and settings commonly used in CI pipelines. It is one of several build infrastructure options offered by Harness. For more information, go to Which build infrastructure is right for me.

How do I use Harness Cloud build infrastructure?​

Configuring your pipeline to use Harness Cloud takes just a few minutes. Make sure you meet the requirements for connectors and secrets, then follow the quick steps to use Harness Cloud.

Account verification error with Harness Cloud on Free plan​

Recently Harness has been the victim of several Crypto attacks that use our Harness-managed build infrastructure (Harness Cloud) to mine cryptocurrencies. Harness Cloud is available to accounts on the Free tier of Harness CI. Unfortunately, to protect our infrastructure, Harness now limits the use of the Harness Cloud build infrastructure to business domains and block general-use domains, like Gmail, Hotmail, Yahoo, and other unverified domains.

To address these issues, you can do one of the following:

  • Use the local runner build infrastructure option, or upgrade to a paid plan to use the self-managed VM or Kubernetes cluster build infrastructure options. There are no limitations on builds using your own infrastructure.
  • Create a Harness account with your work email and not a generic email address, like a Gmail address.

What is the Harness Cloud build credit limit for the Free plan?​

The Free plan allows 2,000 build credits per month. For more information, go to Harness Cloud billing and build credits.

Can I use xcode for a MacOS build with Harness Cloud?​

Yes. Harness Cloud macOS runners include several versions of xcode as well as homebrew. For details, go to the Harness Cloud image specifications. You can also install additional tools at runtime.

Can I use my own secrets manager with Harness Cloud build infrastructure?​

No. To use Harness Cloud build infrastructure, you must use the built-in Harness secrets manager.

Connector errors with Harness Cloud build infrastructure​

To use Harness Cloud build infrastructure, all connectors used in the stage must connect through the Harness Platform. This means that:

  • GCP connectors can't inherit credentials from the delegate. They must be configured to connect through the Harness Platform.
  • Azure connectors can't inherit credentials from the delegate. They must be configured to connect through the Harness Platform.
  • AWS connectors can't use IRSA, AssumeRole, or delegate connectivity mode. They must connect through the Harness Platform with access key authentication.

For more information, go to Use Harness Cloud build infrastructure - Requirements for connectors and secrets.

To change the connector's connectivity mode:

  1. Go to the Connectors page at the account, organization, or project scope. For example, to edit account-level connectors, go to Account Settings, select Account Resources, and then select Connectors.
  2. Select the connector that you want to edit.
  3. Select Edit Details.
  4. Select Continue until you reach Select Connectivity Mode.
  5. Select Change and select Connect through Harness Platform.
  6. Select Save and Continue and select Finish.

Built-in Harness Docker Connector doesn't work with Harness Cloud build infrastructure​

Depending on when your account was created, the built-in Harness Docker Connector (account.harnessImage) might be configured to connect through a Harness Delegate instead of the Harness Platform. In this case, attempting to use this connector with Harness Cloud build infrastructure generates the following error:

While using hosted infrastructure, all connectors should be configured to go via the Harness platform instead of via the delegate. \
Please update the connectors: [harnessImage] to connect via the Harness platform instead. \
This can be done by editing the connector and updating the connectivity to go via the Harness platform.

To resolve this error, you can either modify the Harness Docker Connector or use another Docker connector that you have already configured to connect through the Harness Platform.

To change the connector's connectivity settings:

  1. Go to Account Settings and select Account Resources.
  2. Select Connectors and select the Harness Docker Connector (ID: harnessImage).
  3. Select Edit Details.
  4. Select Continue until you reach Select Connectivity Mode.
  5. Select Change and select Connect through Harness Platform.
  6. Select Save and Continue and select Finish.

Can I change the CPU/memory allocation for steps running on Harness cloud?​

Unlike with other build infrastructures, you can't change the CPU/memory allocation for steps running on Harness Cloud. Step containers running on Harness Cloud build VMs automatically use as much as CPU/memory as required up to the available resource limit in the build VM.

Does gsutil work with Harness Cloud?​

No, gsutil is deprecated. You should use gcloud-equivalent commands instead, such as gcloud storage cp instead of gsutil cp.

However, neither gsutil nor gcloud are recommended with Harness Cloud build infrastructure. Harness Cloud sources build VMs from a variety of cloud providers, and it is impossible to predict which specific cloud provider hosts the Harness Cloud VM that your build uses for any single execution. Therefore, avoid using tools (such as gsutil or gcloud) that require a specific cloud provider's environment.

Can't use STO steps with Harness Cloud macOS runners​

Currently, STO scan steps aren't compatible with Harness Cloud macOS runners, because Apple's M1 CPU doesn't support nested virtualization. You can use STO scan steps with Harness Cloud Linux and Windows runners.

How do I configure OIDC with GCP WIF for Harness Cloud builds?​

Go to Configure OIDC with GCP WIF for Harness Cloud builds.

When I run a build on Harness cloud, which delegate is used? Do I need to install a delegate to use Harness Cloud?​

Harness Cloud builds use a delegate hosted in the Harness Cloud runner. You don't need to install a delegate in your local infrastructure to use Harness Cloud.

Can I use Harness Cloud run CD steps/stages?​

No. Currently, you can't use Harness Cloud build infrastructure to run CD steps or stages. Currently, Harness Cloud is specific to Harness CI.

Can I connect to services running in a private corporate network when using Harness Cloud?​

Yes. You can use Secure Connect for Harness Cloud.

Kubernetes clusters​

What is the difference between a Kubernetes cluster build infrastructure and other build infrastructures?​

For a comparison of build infrastructures go to Which build infrastructure is right for me.

For requirements, recommendations, and settings for using a Kubernetes cluster build infrastructure, go to:

Can I run Docker commands on a Kubernetes cluster build infrastructure?​

If you want to run Docker commands when using a Kubernetes cluster build infrastructure, Docker-in-Docker (DinD) with privileged mode is required. For instructions, go to Run DinD in a Build stage.

If your cluster doesn't support privileged mode, you must use a different build infrastructure option, such as Harness Cloud, where you can run Docker commands directly on the host without the need for Privileged mode. For more information, go to Set up a Kubernetes cluster build infrastructure - Privileged mode is required for Docker-in-Docker.

Can I use Istio MTLS STRICT mode with Harness CI?​

Yes, but you must create a headless service for Istio MTLS STRICT mode.

How can you execute Docker commands in a CI pipeline that runs on a Kubernetes cluster that lacks a Docker runtime?​

You can run Docker-in-Docker (DinD) as a service with the sharedPaths set to /var/run. Following that, the steps can be executed as Docker commands. This works regardless of the Kubernetes container runtime.

The DinD service does not connect to the Kubernetes node daemon. It launches a new Docker daemon on the pod, and then other containers use that Docker daemon to run their commands.

For details, go to Run Docker-in-Docker in a Build stage.

Resource allocation for Kubernetes cluster build infrastructure​

You can adjust CPU and memory allocation for individual steps running on a Kubernetes cluster build infrastructure or container. For information about how resource allocation is calculated, go to Resource allocation.

What is the default CPU and memory limit for a step container?​

For default resource request and limit values, go to Build pod resource allocation.

Why do steps request less memory and CPU than the maximum limit? Why do step containers request fewer resources than the limit I set in the step settings?​

By default, resource requests are always set to the minimum, and additional resources (up to the specified maximum limit) are requested only as needed during build execution. For more information, go to Build pod resource allocation.

How do I configure the build pod to communicate with the Kubernetes API server?​

By default, the namespace's default service account is auto-mounted on the build pod, through which it can communicate with API server. To use a non-default service account, specify the Service Account Name in the Kubernetes cluster build infrastructure settings.

Do I have to mount a service account on the build pod?​

No. Mounting a service account isn't required if the pod doesn't need to communicate with the Kubernetes API server during pipeline execution. To disable service account mounting, deselect Automount Service Account Token in the Kubernetes cluster build infrastructure settings.

What types of volumes can be mounted on a CI build pod?​

You can mount many types of volumes, such as empty directories, host paths, and persistent volumes, onto the build pod. Use the Volumes in the Kubernetes cluster build infrastructure settings to do this.

How can I run the build pod on a specific node?​

Use the Node Selector setting to do this.

It is possible to make a toleration configuration at the project, org, or account level?​

Tolerations in a Kubernetes cluster build infrastructure can only be set at the stage level.

I want to use an EKS build infrastructure with an AWS connector that uses IRSA​

You need to set the Service Account Name in the Kubernetes cluster build infrastructure settings.

If you get error checking push permissions or similar, go to the Build and Push to ECR error article.

Why are build pods being evicted?​

Harness CI pods shouldn't be evicted due to autoscaling of Kubernetes nodes because Kubernetes doesn't evict pods that aren't backed by a controller object. However, build pods can be evicted due to CPU or memory issues in the pod or using spot instances as worker nodes.

If you notice either sporadic pod evictions or failures in the Initialize step in your Build logs, add the following Annotation to your Kubernetes cluster build infrastructure settings:

"cluster-autoscaler.kubernetes.io/safe-to-evict": "false"

AKS builds timeout​

Azure Kubernetes Service (AKS) security group restrictions can cause builds running on an AKS build infrastructure to timeout.

If you have a custom network security group, it must allow inbound traffic on port 8080, which the delegate service uses.

For more information, refer to the following Microsoft Azure troubleshooting documentation: A custom network security group blocks traffic.

How do I set the priority class level? Can I prioritize my build pod if there are resource shortages on the host node?​

Use the Priority Class setting to ensure that the build pod is prioritized in cases of resource shortages on the host node.

What's the default priority class level?​

If you leave the Priority Class field blank, the PriorityClass is set to the globalDefault, if your infrastructure has one defined, or 0, which is the lowest priority.

Can I transfer files into my build pod?​

To do this, use a script in a Run step.

Can I mount an existing Kubernetes secret into the build pod?​

Currently, Harness doesn't offer built-in support for mounting existing Kubernetes secrets into the build pod.

How are step containers named within the build pod?​

Step containers are named sequentially starting with step-1.

When I run a build, Harness creates a new pod and doesn't run the build on the delegate​

This is the expected behavior. When you run a Build (CI) stage, each step runs on a new build farm pod that isn't connected to the delegate.

What permissions are required to run CI builds in an OpenShift cluster?​

For information about building on OpenShift clusters, go to Permissions Required and OpenShift Support in the Kubernetes Cluster Connector Settings Reference.

What are the minimum permissions required for the service account role for a Kubernetes Cluster connector?​

For information about permissions required to build on Kubernetes clusters, go to Permissions Required in the Kubernetes Cluster Connector Settings Reference.

How does the build pod communicate with the delegate? What port does the lite-engine listen on?​

The delegate communicates to the temp pod created by the container step through the build pod IP. Build pods have a lite engine running on port 20001.

Experiencing OOM on java heap for the delegate​

Check CPU utilization and try increasing the CPU request and limit amounts.

Your Java options must use UseContainerSupport instead of UseCGroupMemoryLimitForHeap, which was removed in JDK 11.

I have multiple delegates in multiple instances. How can I ensure the same instance is used for each step?​

Use single replica delegates for tasks that require the same instance, and use a delegate selector by delegate name. The tradeoff is that you might have to compromise on your delegates' high availability.

Delegate is unable to connect to the created build farm​

If you get this error when using a Kubernetes cluster build infrastructure, and you have confirmed that the delegate is installed in the same cluster where the build is running, you may need to allow port 20001 in your network policy to allow pod-to-pod communication.

If the delegate is unable to connect to the created build farm with Istio MTLS STRICT mode, and you are seeing that the pod is removed after a few seconds, you might need to add Istio ProxyConfig with "holdApplicationUntilProxyStarts": true. This setting delays application start until the pod is ready to accept traffic so that the delegate doesn't attempt to connect before the pod is ready.

For more delegate and Kubernetes troubleshooting guidance, go to Troubleshooting Harness.

If my pipeline consists of multiple CI stages, are all the steps across different stages executed within the same build pod?​

No. Each CI stage execution triggers the creation of a new build pod. The steps within a stage are then carried out within the stage's dedicated pod. If your pipeline has multiple CI stages, distinct build pods are generated for each individual stage.

When does the cleanup of build pods occur? Does it happen after the entire pipeline execution is finished?​

Build pod cleanup takes place immediately after the completion of a stage's execution. This is true even if there are multiple CI stages in the same pipeline; as each build stage ends, the pod for that stage is cleaned up.

Is the build pod cleaned up in the event of a failed stage execution?​

Yes, the build pod is cleaned up after stage execution, regardless of whether the stage succeeds or fails.

How do I know if the pod cleanup task fails?​

To help identify pods that aren't cleaned up after a build, pod deletion logs include details such as the cluster endpoint targeted for deletion. If a pod can't be located for cleanup, then the logs include the pod identifier, namespace, and API endpoint response from the pod deletion API. You can find logs in the Build details.

Can I use an ECS cluster for my Kubernetes cluster build infrastructure?​

Currently, Harness CI doesn't support running CI builds on ECS clusters.

Can I use a Docker delegate with a Kubernetes cluster build infrastructure?​

Yes, if the Kubernetes connector is configured correctly. For more information, go to Use delegate selectors with Kubernetes cluster build infrastructure.

Self-signed certificates​

Can I mount internal CA certs on the CI build pod?​

Yes. To do this with a Kubernetes cluster build infrastructure, go to Configure a Kubernetes build farm to use self-signed certificates.

Can I use self-signed certs with local runner build infrastructure?​

With a local runner build infrastructure, you can use CI_MOUNT_VOLUMES to use self-signed certificates. For more information, go to Set up a local runner build infrastructure.

How do I make internal CA certs available to the delegate pod?​

There are multiple ways you can do this:

  • Build the delegate image with the certs baked into it, if you are custom building the delegate image.
  • Create a secret/configmap with the certs data, and then mount it on the delegate pod.
  • Run commands in the INIT_SCRIPT to download the certs while the delegate launches and make them available to the delegate pod file system.

Where should I mount internal CA certs on the build pod?​

The usage of the mounted CA certificates depends on the specific container image used for the step. The default certificate location depends on the base image you use. The location where the certs need to be mounted depends on the container image being used for the steps that you intend to run on the build pod.

Git connector SCM connection errors when using self-signed certificates​

If you have configured your build infrastructure to use self-signed certificates, your builds may fail when the code repo connector attempts to connect to the SCM service. Build logs may contain the following error messages:

Connectivity Error while communicating with the scm service
Unable to connect to Git Provider, error while connecting to scm service

To resolve this issue, add SCM_SKIP_SSL=true to the environment section of the delegate YAML. For example, here is the environment section of a docker-compose.yml file with the SCM_SKIP_SSL variable:

environment:
      - ACCOUNT_ID=XXXX
      - DELEGATE_TOKEN=XXXX
      - MANAGER_HOST_AND_PORT=https://app.harness.io
      - LOG_STREAMING_SERVICE_URL=https://app.harness.io/log-service/
      - DEPLOY_MODE=KUBERNETES
      - DELEGATE_NAME=test
      - NEXT_GEN=true
      - DELEGATE_TYPE=DOCKER
      - SCM_SKIP_SSL=true

For more information about self-signed certificates, delegates, and delegate environment variables, go to:

Certificate volumes aren't mounted to the build pod​

If the volumes are not getting mounted to the build containers, or you see other certificate errors in your pipeline, try the following:

  1. Add a Run step that prints the contents of the destination path. For example, you can include a command such as:

    cat /kaniko/ssl/certs/additional-ca-cert-bundle.crt

  2. Double-check that the base image used in the step reads certificates from the same path given in the destination path on the Delegate.

pnpm enters infinite loop without logs​

If your pipeline runs pnpm or npm commands that cause it to enter an infinite loop or wait indefinitely without producing logs, try adding the following command to your script to see if this allows the build to proceed:

npm config set strict-ssl false

If your pnpm commands are waiting for user input. Try using the append-only flag.

Windows builds​

Error when running Docker commands on Windows build servers​

Make sure that the build server has the Windows Subsystem for Linux installed. This error can occur if the container can't start on the build system.

Docker commands aren't supported for Windows builds on Kubernetes cluster build infrastructures.

Is rootless configuration supported for builds on Windows-based build infrastructures?​

No, currently this is not supported for Windows builds.

What is the default user set on the Windows Lite-Engine and Addon image? Can I change it?​

The default user for these images is ContainerAdministrator. For more information, go to Run Windows builds in a Kubernetes build infrastructure - Default user for Windows builds.

Can I use custom cache paths on a Windows platform with Cache Intelligence?​

Yes, you can use custom cache paths with Cache Intelligence on Windows platforms.

How do I specify the disk size for a Windows instance in pool.yml?​

With self-managed VM build infrastructure, the disk configuration in your pool.yml specifies the disk size (in GB) and type.

For example, here is a Windows pool configuration for an AWS VM build infrastructure:

version: "1"
instances:
  - name: windows-ci-pool
    default: true
    type: amazon
    pool: 1
    limit: 4
    platform:
      os: windows
    spec:
      account:
        region: us-east-2
        availability_zone: us-east-2c
        access_key_id: 
        access_key_secret: 
        key_pair_name: XXXXX
      ami: ami-088d5094c0da312c0
      size: t3.large ## VM machine size.
      hibernate: true
      network:
        security_groups:
          - sg-XXXXXXXXXXXXXX
      disk:
        size: 100 ## Disk size in GB.
        type: "pd-balanced"

Step continues running for a long time after the command is complete​

In Windows builds, if the primary command in a Powershell script starts a long-running subprocess, the step continues to run as long as the subprocess exits (or until it reaches the step timeout limit). To resolve this:

  1. Check if your command launches a subprocess.
  2. If it does, check whether the process is exiting, and how long it runs before exiting.
  3. If the run time is unacceptable, you might need to add commands to sleep or force exit the subprocess.
Example: Subprocess with two-minute life

Here's a sample pipeline that includes a Powershell script that starts a subprocess. The subprocess runs for no more than two minutes.

pipeline:
  identifier: subprocess_demo
  name: subprocess_demo
  projectIdentifier: default
  orgIdentifier: default
  tags: {}
  stages:
    - stage:
        identifier: BUild
        type: CI
        name: Build
        spec:
          cloneCodebase: true
          execution:
            steps:
              - step:
                  identifier: Run_1
                  type: Run
                  name: Run_1
                  spec:
                    connectorRef: YOUR_DOCKER_CONNECTOR_ID
                    image: jtapsgroup/javafx-njs:latest
                    shell: Powershell
                    command: |-
                      cd folder
                      gradle --version
                      Start-Process -NoNewWindow -FilePath "powershell" -ArgumentList "Start-Sleep -Seconds 120"
                      Write-Host "Done!"
                    resources:
                      limits:
                        memory: 3Gi
                        cpu: "1"
          infrastructure:
            type: KubernetesDirect
            spec:
              connectorRef: YOUR_KUBERNETES_CLUSTER_CONNECTOR_ID
              namespace: YOUR_KUBERNETES_NAMESPACE
              initTimeout: 900s
              automountServiceAccountToken: true
              nodeSelector:
                kubernetes.io/os: windows
              os: Windows
          caching:
            enabled: false
            paths: []
  properties:
    ci:
      codebase:
        connectorRef: YOUR_CODEBASE_CONNECTOR_ID
        build:
          type: branch
          spec:
            branch: main

Concatenated variable values in PowerShell scripts print to multiple lines​

If your PowerShell script (in a Run step) echoes a stage variable that has a concatenated value that includes a ToString representation of a PowerShell object (such as the result of Get-Date), this output might unexpectedly print to multiple lines in the build logs.

To resolve this, exclude the ToString portion from the stage variable's concatenated value, and then, in your PowerShell script, call ToString separately and "manually concatenate" the values. Expand the sections below to learn more about the cause and solution for this issue.

What causes unexpected multiline output from PowerShell scripts?

For example, the following two stage variables include one variable that has a ToString value and another variable that concatenates three expressions into a single expression, including the ToString value.

        variables:
          - name: DATE_FORMATTED ## This variable's value is 'ToString' output.
            type: String
            description: ""
            required: false
            value: (Get-Date).ToString("yyyy.MMdd")
          - name: BUILD_VAR ## This variable's value concatenates the execution ID, the sequence ID, and the value of DATE_FORMATTED.
            type: String
            description: ""
            required: false
            value: <+<+pipeline.executionId>+"-"+<+pipeline.sequenceId>+"-"+<+stage.variables.DATE_FORMATTED>>

When a PowerShell script calls the concatenated variable, such as echo <+pipeline.stages.test.variables.BUILD_VAR>, the ToString portion of the output prints on a separate line from the rest of the value, despite being part of one concatenated expression.

How do I fix unexpected multiline output from PowerShell scripts?

To resolve this, exclude the ToString portion from the stage variable's concatenated value, and then, in your PowerShell script, call ToString separately and "manually concatenate" the values.

For example, here are the two stage variables from the previous example without the ToString value in the concatenated expression.

        variables:
          - name: DATE_FORMATTED ## This variable is unchanged.
            type: String
            description: ""
            required: false
            value: (Get-Date).ToString("yyyy.MMdd")
          - name: BUILD_VAR ## This variable's value concatenates only the execution ID and sequence ID. It no longer includes DATE_FORMATTED.
            type: String
            description: ""
            required: false
            value: <+<+pipeline.executionId>+"-"+<+pipeline.sequenceId>>

In the Run step's PowerShell script, call the ToString value separately and then "manually concatenate" it onto the concatenated expression. For example:

              - step:
                  identifier: echo
                  type: Run
                  name: echo
                  spec:
                    shell: Powershell
                    command: |- ## DATE_FORMATTED is resolved separately and then appended to BUILD_VAR.
                      $val = <+stage.variables.DATE_FORMATTED>
                      echo <+pipeline.stages.test.variables.BUILD_VAR>-$val

User data isn't running on AWS Windows Server 2022 VM Pool​

Windows only runs User Data during initialization. To fix this, go to C:\ProgramData\Amazon\EC2Launch\state and delete the .run-once file. This file is generated after the Windows VM initializes. On startup Windows will check for this file and decide whether or not to run the User Data script. If this file is not present, Windows will run the User Data script.

How do I install Docker on Windows?​

To install Docker on Windows, run:

Invoke-WebRequest -UseBasicParsing "https://raw.githubusercontent.com/microsoft/Windows-Containers/Main/helpful_tools/Install-DockerCE/install-docker-ce.ps1" -o install-docker-ce.ps1
.\install-docker-ce.ps1

More information on this can be found in the Microsoft Documentation.

Do I need to enable Hyper V for AWS Windows VM Pool?​

Hyper V is not required to run Harness Builds in a Windows VM Pool. Hyper V is a requirement for Docker Desktop and Docker Desktop is not required for self-managed Windows VM build infrastructure.

Do I need to install WSL for AWS Windows VM Pool?​

WSL is not required to run Harness Builds in a Windows VM Pool. WSL is a requirement for Docker Desktop and Docker Desktop is not required for Windows Self-Managed Build Infrastructure.

How do I check the logs for Windows Server 2022 when using EC2Launchv2​

Logs are generated in the C:\ProgramData\Amazon\EC2Launch\log directory for EC2Launchv2. To view startup logs, check the C:\ProgramData\Amazon\EC2Launch\log\agent file. For any errors check the C:\ProgramData\Amazon\EC2Launch\log\err file.

Default user, root access, and run as non-root​

Which user does Harness use to run steps like Git Clone, Run, and so on? What is the default user ID for step containers?​

Harness uses user 1000 by default. You can use a step's Run as User setting to use a different user for a specific step.

Can I enable root access for a single step?​

If your build runs as non-root (meaning you have set runAsNonRoot: true in your build infrastructure settings), you can run a specific step as root by setting Run as User to 0 in the step's settings. This setting uses the root user for this specific step while preserving the non-root user configuration for the rest of the build. This setting is not available for all build infrastructures, as it is not applicable to all build infrastructures.

When I try to run as non-root, the build fails with "container has runAsNonRoot and image has non-numeric user (harness), cannot verify user is non-root"​

This error occurs if you enable Run as Non-Root without configuring the default user ID in Run as User. For more information, go to CI Build stage settings - Run as non-root or a specific user.

Codebases​

What is a codebase in a Harness pipeline?​

The codebase is the Git repository where your code is stored. Pipelines usually have one primary or default codebase. If you need files from multiple repos, you can clone additional repos.

How do I connect my code repo to my Harness pipeline?​

For instructions on configuring your pipeline's codebase, go to Configure codebase.

What permissions are required for GitHub Personal Access Tokens in Harness GitHub connectors?​

For information about configuring GitHub connectors, including required permissions for personal access tokens, go to the GitHub connector settings reference.

Can I skip the built-in clone codebase step in my CI pipeline?​

Yes, you can disable the built-in clone codebase step for any Build stage. For instructions, go to Disable Clone Codebase for specific stages.

Can I configure a failure strategy for a built-in clone codebase step?​

No, you can't configure a failure strategy for the built-in clone codebase step. If you have concerns about clone failures, you can disable Clone Codebase, and then add a Git Clone step with a step failure strategy at the beginning of each stage where you need to clone your codebase.

Can I recursively clone a repo?​

Currently, the built-in clone codebase step doesn't support recursive cloning. However, you can disable Clone Codebase, and then add a Run step with git clone --recursive. This is similar to the process you would follow to clone a subdirectory instead of the entire repo.

If you want to recursively clone a repo in addition to your default codebase, you can pull the Git credentials from your code repo connector to use in your Run step.

Can I clone a specific subdirectory rather than an entire repo?​

Yes. For instructions, go to Clone a subdirectory.

Does the built-in clone codebase step fetch all branches? How can I fetch all branches?​

The built-in clone codebase step fetches only one or two branches from the repo, depending on the build type (tag, PR, or branch).

If you need to clone all branches in a repo, you can execute the necessary git commands in a Run step.

Can I clone the default codebase to a different folder than the root?​

The built-in clone codebase step always clones your repo to the root of the workspace, /harness. If you need to clone elsewhere, you can disable Clone Codebase and use a Git Clone or Run step to clone your codebase to a specific subdirectory.

What is the default clone depth setting for CI builds?​

For information about the default clone depth setting, go to Configure codebase - Depth.

Can I change the depth of the built-in clone codebase step?​

Yes. Use the Depth setting to do this.

How can I reduce clone codebase time?​

There are several strategies you can use to improve codebase clone time:

  • Depending on your build infrastructure, you can set Limit Memory to 1Gi in your codebase configuration.
  • For builds triggered by PRs, set the Pull Request Clone Strategy to Source Branch and set Depth to 1.
  • If you don't need the entire repo contents for your build, you can disable the built-in clone codebase step and use a Run step to execute specific git clone arguments, such as to clone a subdirectory.

What codebase environment variables are available to use in triggers, commands, output variables, or otherwise?​

For a list of <+codebase.*> and similar expressions you can use in your build triggers and otherwise, go to the CI codebase variables reference.

What expression can I use to get the repository name and the project/organization name for a trigger?​

You can use the expressions <+eventPayload.repository.name> or <+trigger.payload.repository.name> to reference the repository name from the incoming trigger payload.

If you want both the repo and project name, and your Git provider's webhook payload doesn't include a single payload value with both names, you can concatenate two expressions together, such as <+trigger.payload.repository.project.key>/<+trigger.payload.repository.name>.

The expression eventPayload.repository.name causes the clone step to fail when used with a Bitbucket account connector.​

Try using the expression <+trigger.payload.repository.name> instead.

Codebase expressions aren't resolved or resolve to null.​

Empty or null values primarily occur due to the build type (tag, branch, or PR) and start conditions (manual or automated trigger). For example, <+codebase.branch> is always null for tag builds, and <+trigger.*> expressions are always null for manual builds.

Other possible causes for null values are that the connector doesn't have API access enabled in the connector's settings or that your pipeline doesn't use the built-in clone codebase step.

For more information about when codebase expressions are resolved, go to CI codebase variables reference.

How can I share the codebase configuration between stages in a CI pipeline?​

The pipeline's default codebase is automatically available to each subsequent Build stage in the pipeline. When you add additional Build stages to a pipeline, Clone Codebase is enabled by default, which means the stage clones the default codebase declared in the first Build stage.

If you don't want a stage to clone the default codebase, you can disable Clone Codebase for specific stages.

The same Git commit is not used in all stages​

If your pipeline has multiple stages, each stage that has Clone Codebase enabled clones the codebase during stage initialization. If your pipeline uses the generic Git connector and a commit is made to the codebase after a pipeline run has started, it is possible for later stages to clone the newer commit, rather than the same commit that the pipeline started with.

If you want to force all stages to use the same commit ID, even if there are changes in the repository while the pipeline is running, you must use a code repo connector for a specific SCM provider, rather than the generic Git connector.

Git fetch fails with invalid index-pack output when cloning large repos​

  • Error: During the Initialize step, when cloning the default codebase, git fetch throws fetch-pack: invalid index-pack output.
  • Cause: This can occur with large code repos and indicates that the build machine might have insufficient resources to clone the repo.
  • Soltuion: To resolve this, edit the pipeline's YAML and allocate memory and cpu resources in the codebase configuration. For example:
properties:
  ci:
    codebase:
      connectorRef: YOUR_CODEBASE_CONNECTOR_ID
      repoName: YOUR_CODE_REPO_NAME
      build:
        type: branch
        spec:
          branch: <+input>
      sslVerify: false
      resources:
        limits:
          memory: 4G ## Set the maximum memory to use. You can express memory as a plain integer or as a fixed-point number using the suffixes `G` or `M`. You can also use the power-of-two equivalents `Gi` and `Mi`. The default is `500Mi`.
          cpu: "2" ## Set the maximum number of cores to use. CPU limits are measured in CPU units. Fractional requests are allowed; for example, you can specify one hundred millicpu as `0.1` or `100m`.

Clone codebase fails due to missing plugin​

  • Error: Git clone fails during stage initialization, and the runner's logs contain Error response from daemon: plugin \"<plugin>\" not found
  • Platform: This error can occur in build infrastructures that use a Harness Docker Runner, such as the local runner build infrastructure or the VM build infrastructures.
  • Cause: A required plugin is missing from your build infrastructure container's Docker installation. The plugin is required to configure Docker networks.
  • Solution:
    1. On the machine where the runner is running, stop the runner.
    2. Set the NETWORK_DRIVER environment variable to your preferred network driver plugin (such as export NETWORK_DRIVER="nat" or export NETWORK_DRIVER="bridge"). For Windows, use PowerShell variable syntax, such as $Env:NETWORK_DRIVER="nat" or $Env:NETWORK_DRIVER="bridge".
    3. Restart the runner.

How do I configure the Git Clone step? What is the Clone Directory setting?​

For details about Git Clone step settings, go to:

Does Harness CI support Git Large File Storage (git-lfs)?​

Yes. For more information, go to the Harness CI documentation on Git Large File Storage.

Can I run git commands in a CI Run step?​

Yes. You can run any commands in a Run step. With respect to Git, for example, you can use a Run step to clone multiple code repos in one pipeline, clone a subdirectory, or use Git LFS.

How do I handle authentication for git commands in a Run step?​

You can store authentication credentials as secrets and use expressions, such as <+secrets.getValue("YOUR_TOKEN_SECRET")>, to call them in your git commands.

You could also pull credentials from a git connector used elsewhere in the pipeline.

Can I use codebase variables when cloning a codebase in a Run step?​

No. Codebase variables are resolved only for the built-in Clone Codebase functionality. These variables are not resolved for git commands in Run steps or Git Clone steps.

Git connector fails to connect to the SCM service. SCM request fails with UNKNOWN​

This error may occur if your code repo connector uses SSH authentication. To resolve this error, make sure HTTPS is enabled on port 443. This is the protocol and port used by the Harness connection test for Git connectors.

Also, SCM service connection failures can occur when using self-signed certificates.

How can I see which files I have cloned in the codebase?​

You can add a Run step to the beginning of your Build stage that runs ls -ltr. This returns all content cloned by the Clone Codebase step.

Why is the codebase connector config not saved?​

Changes to a pipeline's codebase configuration won't save if all CI stages in the pipeline have Clone Codebase disabled in the Build stage's settings.

SCM status updates and PR checks​

Does Harness supports Pull Request status updates?​

Yes. Your PRs can use the build status as a PR status check. For more information, go to SCM status checks.

How do I configure my pipelines to send PR build validations?​

For instructions, go to SCM status checks - Pipeline links in PRs.

What connector does Harness uses to send build status updates to PRs?​

Harness uses the pipeline's codebase connector, specified in the pipeline's default codebase configuration to send status updates to PRs in your Git provider.

Can I use the Git Clone step, instead of the built-in clone codebase step, to get build statues on my PRs?​

No. You must use the built-in clone codebase step (meaning, you must configure a default codebase) to get pipeline links in PRs.

Pipeline status updates aren't sent to PRs​

Harness uses the pipeline's codebase connector to send status updates to PRs in your Git provider. If status updates aren't being sent, make sure that you have configured a default codebase and that it is using the correct code repo connector. Also make sure the build that ran was a PR build and not a branch or tag build.

Build statuses don't show on my PRs, even though the code base connector's token has all repo permissions.​

If the user account used to generate the token doesn't have repository write permissions, the resulting token won't have sufficient permissions to post the build status update to the PR. Specific permissions vary by connector. For example, GitHub connector credentials require that personal access tokens have all repo, user, and admin:repo_hook scopes, and the user account used to generate the token must have admin permissions on the repo.

For repos under organizations or projects, check the role/permissions assigned to the user in the target repository. For example, a user in a GitHub organization can have some permissions at the organization level, but they might not have those permissions at the individual repository level.

Can I export a failed step's output to a pull request comment?​

To do this, you could:

  1. Modify the failed step's command to save output to a file, such as your_command 2>&1 | tee output_file.log.
  2. After the failed step, add a Run step that reads the file's content and uses your Git provider's API to export the file's contents to a pull request comment.
  3. Configure the subsequent step's conditional execution settings to Always execute this step.

Does my pipeline have to have a Build stage to get the build status on the PR?​

Yes, the build status is updated on a PR only if a Build (CI) stage runs.

My pipeline has multiple Build stages. Is the build status updated for each stage or for the entire pipeline?​

The build status on the PR is updated for each individual Build stage.

My pipeline has multiple Build stages, and I disabled Clone Codebase for some of them. Why is the PR status being updated for the stages that don't clone my codebase?​

Currently, Harness CI updates the build status on a PR even if you disabled Clone Codebase for a specific build stage. We are investigating enhancements that could change this behavior.

Is there any character limit for the PR build status message?​

Yes. For GitHub, the limit is 140 characters. If the message is too long, the request fails with description is too long (maximum is 140 characters).

What identifiers are included in the PR build status message?​

The pipeline identifier and stage identifier are included in the build status message.

What is the format of the content in the PR build status message?​

The PR build status message format is PIPELINE_IDENTIFIER-STAGE_IDENTIFIER — Execution status of Pipeline - PIPELINE_IDENTIFIER (EXECUTION_ID) Stage - STAGE_IDENTIFIER was STATUS

I don't want to send build statuses to my PRs.​

Because the build status updates operate through the default codebase connector, the easiest way to prevent sending PR status updates would be to disable Clone Codebase for all Build stages in your pipeline, and then use a Git Clone or Run step to clone your codebase.

You could try modifying the permissions of the code repo connector's token so that it can't write to the repo, but this could interfere with other connector functionality.

Removing API access from the connector is not recommended because API access is required for other connector functions, such as cloning codebases from PRs, auto-populating branch names when you manually run builds, and so on.

Why was the PR build status not updated for an Approval stage?​

Build status updates occur for Build stages only.

Failed pipelines don't block PR merges​

Although Harness can send pipeline statuses to your PRs, you must configure branch protection rules and other checks in your SCM provider.

Troubleshoot Git event (webhook) triggers​

For troubleshooting information for Git event (webhook) triggers, go to Troubleshoot Git event triggers.

Pipeline initialization and Harness CI images​

Initialize step fails with a "Null value" error or timeout​

This can occur if an expression or variable is called before it's value is resolved, if a variable/expression references a secret that doesn't exist, or if an expression incorrectly references a secret (such as the incorrect scope path or secret ID).

In Build (CI) stages, steps run in separate containers/build pods, and the pipeline can only successfully resolve expressions if the target value is available.

For example, assume you have a step (named, for example, my-cool-step) that uses an expression to reference the output variable of a step in a repeat looping strategy. If my-cool-step runs before the repeat loop completes, then the expression's value isn't resolved and therefore it isn't available when my-cool-step calls that value.

Depending on how your expression/variable's value is generated, you need to either rearrange the flow of steps in your stage/pipeline or determine how you can provide the value earlier (such as by declaring it in a pipeline variable or stage variable).

With a Kubernetes cluster build infrastructure, all step-level variables must be resolved upfront during pod creation. Therefore, steps referencing output variable from prior steps in the same stage resolve as null, regardless of how the steps are arranged in your stage. To avoid this, generate the output variables in a prior stage and then use an expression referencing the value from the prior stage, for example:

<+pipeline.stages.PRIOR_STAGE.spec.execution.steps.PRIOR_STAGE_STEP.output.outputVariables.SOME_VAR>

Similarly, when using step groups or step group templates with a Kubernetes cluster build infrastructure, Harness can resolve only stage variables and pipeline variables during initialization. Step/group variables resolve as null. This is because stage and pipeline variables are available to be resolved when creating the Kubernetes pod, and step/step group variables are not. In this case, if you encounter the null value error and you are using step-level variables, try configuring these as stage or pipeline variables instead.

Make sure to update the expressions referencing the variables if you change them from step variables to stage/pipeline variables.

Initialize step occasionally times out at 8 minutes​

Eight minutes is the default time out limit for the Initialize step. If your build is hitting the timeout limit due to resource constraints, such as pulling large images, you can increase the Init Timeout in the stage Infrastructure settings.

Problems cloning code repo during initialization.​

For codebase issues, go to Codebases.

When a pipeline pulls artifacts or images, are they stored on the delegate?​

Artifacts and images are pulled into the stage workspace, which is a temporary volume that exists during stage execution. Images are not stored on the delegate during pipeline execution. In a Kubernetes cluster build infrastructure, build stages run on build pods that are cleaned automatically after the execution.

Can I get a list of internal Harness-specific images that CI uses?​

For information about the backend/Harness-specific images that Harness CI uses to execute builds, including how to get a list of images and tags that your builds use, go to Harness CI images.

How often are Harness CI images updated?​

Harness publishes updates for all CI images on the second and fourth Monday of each month. For more information, go to Harness CI images - Harness CI image updates.

How do I get a list of tags available for an image in the Harness image registry?​

To list all available tags for an image in app.harness.io/regstry, call the following endpoint and replace IMAGE_NAME with the name of the image you want to query.

https://app.harness.io/registry/harness/IMAGE_NAME/tags/list

What access does Harness use to pull the Harness internal images from the public image repo?​

By default, Harness uses anonymous access to to pull Harness images.

If you have security concerns about using anonymous access or pulling Harness-specific images from a public repo, you can change how your builds connect to the Harness container image registry.

Can I use my own private registry to store Harness CI images?​

Yes, you can pull Harness CI images from a private registry.

Build failed with "failed to pull image" or "ErrImagePull"​

  • Error messages: ErrImagePull or some variation of the following, which may have a different image name, tag, or registry: Failed to pull image "artifactory.domain.com/harness/ci-addon:1.16.22": rpc error: code = Unknown desc = Error response from daemon: unknown: Not Found.
  • Causes:
    • Harness couldn't pull an image that is needed to run the pipeline.
    • ErrImagePull can be caused by networking issues or if the specified image doesn't exist in the specified repository.
    • Failed to pull image - Not Found means that a Harness-specific image or tag, in this case ci-addon:1.16.22, isn't present in the specified artifact repository, and you are using the account.harnessImage connector to pull Harness images. You can use this connector to pull from your own registry or to pull images from any Docker registry, but it is also used to pull Harness-required CI images. Modifying this connector can cause it to fail to pull the necessary Harness CI images.
  • Solutions:
    • If you modified the built-in Harness Docker connector, check the connector's configuration to make sure it uses one of the compatible methods for pull Harness-required images, as described in Connect to the Harness container image registry.
    • If you are trying to pull images from your own registry, check your configuration for pulling Harness images from a private registry. You might need to use a different connecter than the built-in Harness Docker connector.
    • If you modified tags for some images, check that your configuration uses valid tags that are present in the repository from which Harness is attempting to pull the tags.
    • If you believe the issue is due to networking issues, try again later if you think the issue is transient, or check your connector or network configuration to make sure Harness is able to connect to the given registry.

What pipeline environment variables are there for CI pipelines?​

Go to CI environment variables reference.

Docker Hub rate limiting​

By default, Harness uses anonymous Docker access to pull Harness-required images. If you experience rate limiting issues when pulling images, try the solutions described in Harness CI images - Docker Hub rate limiting.

Build and push images​

Where does a pipeline get code for a build?​

The codebase declared in the first stage of a pipeline becomes the pipeline's default codebase. If your build requires files from multiple repos, you can clone additional repos.

How do I use a Harness CI pipeline to build and push artifacts and images?​

You can use Build and Push steps or Run steps. For information about this go to Build and push artifacts and images.

I need to get the Maven project version from pom.xml and pass it as a Docker build argument​

To do this, you can:

  1. Use a Run step to get the version and assign it to a variable. For example, you could use a command like:

    version=$(cat pom.xml | grep -oP '(?<=<version>)[^<]+')

  2. Specify this variable as an output variable from the Run step.

  3. Use an expression to reference the output variable in your build arguments, such as in the Build and Push to Docker step's Build Arguments or docker build commands executed in a Run step.

Where do I store Maven project settings.xml in Harness CI?​

For information about this, go to Maven settings.xml.

How do I publish maven artifacts to AWS CodeArtifact?​

Typically, this is configured within your Maven settings.xml file to publish artifacts upon build, as explained in the AWS documentation on Use CodeArtifact with mvn.

However, if you're not publishing directly via Maven, you can push directly using the AWS CLI or cURL, as explained in the AWS documentation on Publishing with curl.

Gradle build, daemon, OOM, and other Gradle issues​

For Gradle build or daemon issues, go to the Knowledge Base article on Gradle build and daemon issues.

Can I push without building?​

Harness CI provides several options to upload artifacts. The Upload Artifact steps don't include a "build" component.

Can I build without pushing?​

You can build without pushing.

What drives the Build and Push steps? What is kaniko?​

With Kubernetes cluster build infrastructures, Build and Push steps use kaniko. Other build infrastructures use drone-docker. kaniko requires root access to build the Docker image.

For more information, go to:

Can I set kaniko and drone-docker runtime flags, such as skip-tls-verify or custom-dns?​

Yes, you can set plugin runtime flags on any Build and Push step.

Can I run Build and Push steps as non-root? Does kaniko support non-root users?​

With a Kubernetes cluster build infrastructure, Build and Push steps use the kaniko plugin. kaniko requires root access to build Docker images, and it does not support non-root users. However, you can use the buildah plugin to build and push with non-root users.

Can I run Build and Push steps as root if my build infrastructure runs as non-root?​

If your build infrastructure is configured to run as a non-root user (meaning you have set runAsNonRoot: true), you can run a specific step as root by setting Run as User to 0 in the step's settings. This setting uses the root user for this specific step while preserving the non-root user configuration for the rest of the build. This setting is not available for all build infrastructures, as it is not applicable to all build infrastructures.

What if my security policy doesn't allow running as root?​

If your security policy strictly forbids running any step as root, you can use the buildah plugin to build and push with non-root users.

The buildah plugin requires that you use a Kubernetes cluster build infrastructure that is configured to run as non-root with anyuid SCC (Security Context Constraints) enabled. For information about the buildah plugin, go to Build and push with non-root users.

Can I enable BuildKit support with Build and Push steps? How do I run Docker build commands with buildkit enabled?​

The Build and Push steps use kaniko or drone-docker to build images. If you need to use BuildKit, you can't use the built-in Build and Push steps.

Instead, you need to:

  1. Run Docker-in-Docker in a Background step.

  2. Run docker build and docker push commands in a Run step. To run docker build with builtkit enabled, use something like:

    DOCKER_BUILDKIT=1 docker build -t IMAGE_NAME:TAG .

Is there a way to use a newer or older version of kaniko?​

Yes, you can update the tag for the kaniko image that Harness uses, as explained in Harness CI images - Specify the Harness CI images used in your pipelines.

Does a kaniko build use images cached locally on the node? Can I enable caching for kaniko?​

By default, kaniko does not use the node cache. It performs a full container image build from scratch, so it always pulls the base image. If you want kaniko to cache images and use previously-built layers that haven't changed, specify the Remote Cache Repository setting in the Build and Push step. If not specified, caching isn't used. Layer caching can significantly speed up the image building process.

How can I improve build time when a Build and Push step isn't able to apply remote caching or doesn't cache effectively?​

Make sure your Docker file is configured in least- to most-often changed. Make sure it installs dependencies before moving other files. Docker Layer Caching depends on the order that layers are loaded in your Dockerfile. As soon as it detects a changed layer, it reloads all subsequent layers. Therefore, may sure your Dockerfile is structured for optimum caching efficiency.

Where does the Build and Push to ECR step pull the base images specified in the Dockerfile?​

By default, the Build and Push to ECR step downloads base images from the public container registry. You can use the Base Image Connector setting to specify an authenticated connector to use. This can prevent rate limiting issues.

How can I configure the Build and Push to ECR step to pull base images from a different container registry or my internal container registry?​

Create a Docker connector for your desired container registry and use it in the Base Image Connector setting.

Build and Push to ECR step fails with error building image, failed to execute command, exec format error​

  • Error: Error building image: error building stage: failed to execute command: starting command: ...: exec format error
  • Cause: This error can occur if you're running an ARM node pool instead of AMD.
  • Solution: Change your node pool to AMD and retry the build.

Build and Push to ECR step fails with error checking push permissions​

Go to the Build and Push to ECR error article.

Where does the Build and Push step expect the Dockerfile to be?​

The Dockerfile is assumed to be in the root folder of the codebase. You can use the Dockerfile setting in a Build and Push step to specify a different path to your Dockerfile.

Can I use Harness expressions in my Dockerfile?​

No. Harness expressions aren't supported in Dockerfiles.

Why isn't the Build and Push step parsing the syntax in my Dockerfile?​

In a Kubernetes cluster build infrastructure, Build and Push steps use kaniko, which doesn't support # syntax in the Dockerfile.

If you need to parse syntax in your builds, you can't use the Build and Push steps. Instead, you can use the buildah plugin or run Docker-in-Docker in a Background step with buildkit enabled.

Why doesn't the Build and Push step include the content of VOLUMES from my Dockerfile in the final image?​

In a Kubernetes cluster build infrastructure, the Build and Push steps use kaniko. Kaniko optimizes the build process by creating directories for volumes, but it skips copying the contents into image layers. This behavior is expected and enhances the efficiency of the build.

If you want to include the volume content in the image layers, consider using COPY instructions in your Dockerfile to directly copy the data into the image during the build.

Can I use images from multiple Azure Container Registries (ACRs)?​

Yes. Go to Use images from multiple ACRs.

Is remote caching supported in Build and Push steps?​

Harness supports multiple Docker layer caching methods depending on what infrastructure is used. Go to Docker layer caching to learn more.

Build and Push to Docker fails with kaniko container runtime error​

Go to the Kaniko container runtime error article.

What is the default build context when using Build and Push steps?​

The default build context is the stage workspace directory, which is /harness.

Why is the Build and Push step trying to push to a public Docker Hub repository even though the connector used in the step points to an internal container registry?​

This can occur if the Build and Push step doesn't have the repo's Fully Qualified Name (FQN), even if your Docker connector points to an internal private container registry.

Make sure to use the FQN for the repo when pushing to an internal private container registry.

Why do I get an out of memory error when pushing images to Docker Hub?​

The Build and Push to Docker step can return out of memory errors, such as:

exit status 255 Found possible error on line 70. Log: signal: killed . Possible error: Out of memory. Possible resolution: Increase memory resources for the step

To address this error, enable the Optimize setting in the Build and Push step.

You can also try adjusting container resources, if these settings are applicable to your build infrastructure.

Upload artifacts​

Can I send emails from CI pipelines?​

You can use the Drone Email plugin to send emails and attachments from CI pipelines.

How do I show content on the Artifacts tab?​

You can use the Artifact Metadata Publisher plugin to store artifact URLs and display them on the Artifacts tab.

Is it possible to publish custom data, such as outputs from variables or custom messages, to the Artifacts tab?​

Currently, the Artifacts tab contains only links. Therefore, any content you want to make available on the Artifacts tab must be uploaded to cloud storage and then queried. You can use the Artifacts Metadata Publisher plugin for this.

You can provide one or more URLs to artifacts. For example, to reference artifacts stored in S3 buckets, you can provide the URL to the target artifact, such as https://BUCKET.s3.REGION.amazonaws.com/TARGET/ARTIFACT_NAME_WITH_EXTENSION. If you uploaded multiple artifacts, you can provide a list of URLs. If your S3 bucket is private, use the console view URL, such as https://s3.console.aws.amazon.com/s3/object/BUCKET?region=REGION&prefix=TARGET/ARTIFACT_NAME_WITH_EXTENSION.

In addition to the console view URL, you can reference privately-stored artifact by generating pre-signed URLs or temporary URLs, such as Google Cloud Storage signed URLs or AWS S3 pre-signed URLs.

Does the Upload Artifacts to S3 step compress files before uploading them?​

No. If you want to upload a compressed file, you must use a Run step to compress the artifact before uploading it.

Can I trim the parent folder name before uploading to S3?​

Yes. Go to Trim parent folder name when uploading to S3.

Connector errors with Upload Artifacts to S3 step.​

There are a variety of potential causes for AWS connector errors due to specific requirements for the AWS connector in the Upload Artifacts to S3 step.

Can I use non-default ACLs, IAM roles, or ARNs with the Upload Artifacts to S3 step?​

Yes, but there are specific requirements for the AWS connector in the Upload Artifacts to S3 step.

Does the Upload Artifacts to GCS step support GCP connectors that inherit delegate credentials?​

No. Currently, the Upload Artifacts to GCS step doesn't support GCP connectors that inherit delegate credentials.

Upload Artifacts to JFrog step throws certificate signed by unknown authority​

If you get a certificate signed by unknown authority error with the Upload Artifacts to JFrog step, make sure the correct server certificates are uploaded to the correct container path. For example, the container path for Windows is C:/Users/ContainerAdministrator/.jfrog/security/certs.

Can I run the Upload Artifacts to JFrog Artifactory step with a non-root user?​

No. The jfrog commands in the Upload Artifacts to JFrog Artifactory step create a .jfrog folder at the root level of the stage workspace, which fails if you use a non-root user.

mkdir permission denied when running Upload Artifacts to JFrog as non-root​

With a Kubernetes cluster build infrastructure, the Upload Artifacts to JFrog step must run as root. If you set Run as User to anything other than 1000, the step fails with mkdir /.jfrog: permission denied.

What is PLUGIN_USERNAME and PLUGIN_PASSWORD used in the Upload Artifacts to JFrog Artifactory step?​

These are derived from your Artifactory connector.

Test reports​

Test reports missing or test suites incorrectly parsed​

The parsed test report in the Tests tab comes strictly from the provided test reports (declared in the step's Report Paths). Test reports must be in JUnit XML format to appear on the Tests tab, because Harness parses test reports that are in JUnit XML format only. It is important to adhere to the standard JUnit format to improve test suite parsing. For more information, go to Format test reports.

What if my test tool's default report format isn't JUnit?​

There are converters available for many test tools that don't produce results in JUnit format by default.

For example, the default report format for Jest is JSON, and you can use the Jest JUnit Reporter to convert the JSON results to JUnit XML format.

For more information, go to Format test reports.

Can I specify multiple paths for test reports in a Run step?​

Yes, you can specify multiple paths for test reports. Ensure that the reports do not contain duplicate tests when specifying multiple paths.

Why is the test report truncated in Tests tab?​

The Tests tab truncates content if a field in your test report XML file surpasses 8,000 characters.

Run step in a containerized step group can't publish test reports, and it throws "Unable to collect test reports" though the report path is correctly​

Currently, publishing test reports from a Run step in a CD containerized step group is not supported. Try running your tests in a Build (CI) stage.

Is the Tests tab only for Test Intelligence?​

No. Test reports from tests run in Run steps also appear there if they are correctly formatted.

Test splitting​

Does Harness support test splitting (parallelism)?​

Yes. How you do this depends on whether your tests run in a Run step or a Run Tests step. For instructions, go to:

Test Intelligence​

How do I use Test Intelligence?​

For instructions, go to Test Intelligence overview.

Can Test Intelligence speed up my build times? What are the benefits of Test Intelligence?​

Test Intelligence improves test time by running only the unit tests required to confirm the quality of the code changes that triggered a build. It can identify negative trends and help you gain insight into unit test quality. For more information, go to Test Intelligence overview.

What criteria does Test Intelligence use to select tests?​

For information about how Test Selection selects tests, go to Test Intelligence overview.

Does Test Intelligence split tests? Why would I use test splitting with Test Intelligence?​

Test Intelligence doesn't split tests. Instead, Test Intelligence selects specific tests to run based on the changes made to your code. It can reduce the overall number of tests that run each time you make changes to your code.

For additional time savings, you can apply test splitting in addition to Test Intelligence. This can further reduce your test time by splitting the selected tests into parallel workloads.

If the Run Tests step fails, does the Post-Command script run?​

No. The Post-Command script runs only if the Run Tests step succeeds.

Can I limit memory and CPU for Run Tests steps running on Harness Cloud?​

No. Resource limits are not customizable when using Harness Cloud or self-managed VM build infrastructures. In these cases, the step can consume the entire memory allocation of the VM.

How can I understand the relationship between code changes and the selected tests?​

On the Tests tab, the visualization call graph provides insights into why each test was selected. It visually represents the relationship between the selected tests and the specific code changes in the PR. For more information, go to View tests - Results from Run Tests steps.

On the Tests tab, the Test Intelligence call graph is empty and says "No call graph is created when all tests are run"​

No call graph is generated if Test Intelligence selects to run all tests because the call graph would be huge and not useful (no test selection logic to demonstrate).

Additionally, the first run with TI doesn't include test selection, because Harness must establish a baseline for comparison in future runs. On subsequent runs, Harness can use the baseline to select relevant tests based on the content of the code changes.

For information about how and when TI selects tests, go to How does Test Intelligence work?

Ruby Test Intelligence can't find rspec helper file​

The following log line indicates that Test Intelligence can't locate an rspec helper file in your code repo:

Unable to write rspec helper file automatically cannot find rspec helper file. Please make change manually to enable TI.

This usually occurs if the helper file has a name other than spec_helper.rb.

To resolve this, add the following line to your rspec helper file:

set -e; echo "require_relative '/tmp/engine/ruby/harness/ruby-agent/test_intelligence.rb'" >> lib/vagrant/shared_helpers.rb

Can I use Test Intelligence for Ruby on Rails?​

You can, however, Harness doesn't recommend using Test Intelligence with Rails apps using Spring.

Test Intelligence fails due to Bazel not installed, but the container image has Bazel​

If your build tool is Bazel, and you use a container image to provide the Bazel binary to the Run Tests step, your build will fail if Bazel isn't already installed in your build infrastructure. This is because the Run Tests step calls bazel query before pulling the container image.

Bazel is already installed on Harness Cloud runners. For other build infrastructures, you must manually confirm that Bazel is already installed. If Bazel isn't already installed on your build infrastructure, you need to install Bazel in a Run step prior to the Run Tests step.

Gradle version not compatible with Test Intelligence.​

For information about Gradle compatibility with TI and how to modify build.gradle for TI, go to Enable TI for Java, Kotlin, or Scala - Build Tool - Java Gradle compatibility.

Test Intelligence errors with Maven​

If you encounter issues with Test Intelligence when using Maven as your build tool, check the following configurations:

  • If your pom.xml contains <argLine>, you might need to modify your argLine setup as explained in Enable TI for Java, Kotlin, Scala - Build tool - Java Maven argLine setup.

  • If you attach Jacoco or any agent while running unit tests, then you must modify your argLine setup as explained in Enable TI for Java, Kotlin, Scala - Build tool - Java Maven argLine setup.

  • If you use Jacoco, Surefire, or Failsafe, make sure that forkCount is not set to 0. For example, the following configuration in pom.xml removes forkCount and applies useSystemClassLoader as a workaround:

    <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-plugin</artifactId>
        <version>2.22.1</version>
        <configuration>
            <!--  <forkCount>0</forkCount> -->
            <useSystemClassLoader>false</useSystemClassLoader>
        </configuration>
    </plugin>

Python Test Intelligence errors​

If you encounter errors with Python TI, make sure that:

  • Your project is written in Python 3, and your repo is a pure Python 3 repo.
  • You don't use resource file relationships. TI for Python doesn't support resource file relationships.
  • You don't use dynamic loading and metaclasses. TI for Python might miss tests or changes in repos that use dynamic loading or metaclasses.
  • Your build tool is pytest or unittest.
  • The Python 3 binary is present. This means it is preinstalled on the build machine, available in the step's Container Registry and Image, or installed at runtime in the step's Pre-Command.
  • If you use another command to invoke Python 3, such as python, you have added an alias, such as python3 = "python".
  • If you get code coverage errors, your Build Arguments don't need coverage flags (--cov or coverage), and you don't need to install coverage tools in Pre-Command.

Does Test Intelligence support dynamic code?​

Harness doesn't recommend using TI with Ruby projects using dynamically generated code or Python projects using dynamic loading or metaclasses.

Script execution​

Does Harness CI support script execution?​

Yes. Go to Run scripts.

Running a Python shell in a Run step, the expression <+codebase.gitUser> resolves to "None"​

This means the codebase variable wasn't resolved or resolved to None. Codebase expression values depend on the build trigger type, among other factors.

Can I use an image that doesn't have a shell in a Run step?​

The Run step requires the Command and Shell fields. Some shell must be available on the specified image to be able to run commands.

Is a Docker image required to use the Run step on local runner build infrastructure?​

Yes. Container Registry and Image are always required for Run steps on a local runner build infrastructure. For more information about when Container Registry and Image are required, go to Use Run steps - Container Registry and Image.

Is it required for a Run step's image to have Docker and Docker CLI installed?​

If your step needs to execute Docker commands, then the image needs to have Docker and the Docker CLI.

When attempting to export an output variable from a Run step using a Python shell, the step fails with "no such file or directory"​

This can happen if you manually exit the Python script by calling exit(0). When you declare an output variable in a Run step, Harness runs some additional code at the end of your script to export the variable. If you exit the script manually in your Run step's command, then Harness can't run those additional lines of code.

Secrets with line breaks and shell-interpreted special characters​

For information about handling secrets with new line characters or other shell-interpreted special characters, go to Add and reference text secrets - Line breaks and shell-interpreted characters and Use GCP secrets in scripts.

Output variable length limit​

If an output variable's length is greater than 64KB, steps can fail or truncate the output. If you need to export large amounts of data, consider uploading artifacts or exporting artifacts by email.

Multi-line output variables truncated​

Output variables don't support multi-line output. Content after the first line is truncated. If you need to export multi-line data, consider uploading artifacts or exporting artifacts by email.

How do I get a file from file store in a Run step?​

You can use a file store reference expression to get a file from file store, such as <+fileStore.getAsBase64("someFile")>.

Here's an example in a script:

raw_file=<+fileStore.getAsBase64("someFile")>
config_file="$(echo "$raw_file" | base64 --decode)"
warning

File store expressions don't work for all secrets, and some secrets require additional handling.

For more information, go to:

Can I start containers during pipeline execution? For example, I need to start some containers while executing tests.​

You could do this by running DinD in a Background step so that those services are available when you need to reference them during pipeline execution.

Entry point​

What does the "Failed to get image entrypoint" error indicate in a Kubernetes cluster build?​

This error suggests that there is an issue accessing the entrypoint of the Docker image. It can occur in a Kubernetes cluster build infrastructure when running PostgreSQL services in Background steps.

To resolve this error, you might need to mount volumes for the PostgreSQL data in the build infrastructure's Volumes setting, and then reference those volumes in the Background step running your PostgreSQL instance. For instructions, go to Troubleshooting: Failed to get image entry point.

Does the Harness Run step overwrite the base image container entry point?​

Yes, this is the expected behavior. The entry point in the base image is overwritten so Harness can run the commands specified in the Run step.

Why is the default entry point not running for the container image used in the Run step?​

The default entry point is overwritten by the commands you specified in the Run step's commands.

Since the default entry point isn't executed for the container image used in the Run step, how do I start a service started in a container that would usually be started by the default entry point?​

You can run the service in a Background step, which can execute the default entry point.

How do I run the default entry point of the image used in the Run step?​

The commands specified in the Run step's commands override the default entry point. If you want to run those commands in the Run step, you need to include them in the Run step's commands.

Docker in Docker​

Does CI support running Docker-in-Docker images?​

Yes. For details, go to Run Docker-in-Docker in a Build stage.

Can I run docker-compose from Docker-in-Docker in a Background step?​

Yes.

Is privileged mode necessary for running DinD in Harness CI?​

Yes, Docker-in-Docker (DinD) must run in privileged mode to function correctly.

Generally, you can use DinD on platforms that don't support privileged mode, such as platforms that run containers on Windows or fargate nodes that don't support privileged mode.

Why is my DinD Background step failing with "Pod not supported on Fargate: invalid SecurityContext fields: Privileged"?​

This error occurs because AWS Fargate doesn't support the use of privileged containers. Privileged mode is required for DinD.

Can't connect to Docker daemon​

Go to GitHub Action step can't connect to Docker daemon.

DinD Background step fails when Docker daemon disconnects or hit quota limit after some time​

This typically indicates that the DinD Background step doesn't have sufficient resources. Try modifying the container resources for the Background step.

You can also add a parallel Run step to monitor and help debug the Background step. For example, in this case it would help to use the Run step to monitor resource consumption by the Background step.

My DinD build fails with an Out Of Memory error, but I increased the memory and CPU limit on the Run step that runs my docker build command​

If you run Docker-in-Docker in a Background step, and your docker build commands fail due to OOM errors, you need to increase the memory and CPU limit for the Background step. While the docker build command can be in a Run step or a Build and Push step, the build executes on the DinD container, which is the Background step running DinD. Therefore, you need to increase the container resources for the Background step.

My pipeline runs DinD in a Background step, and I need to start another container in a subsequent run step. How can I connect to the application running in the Background step from the Run step?​

When you start the container in the Run step, attach the container to host network by passing the flag --net host. Then, you can hit the endpoint localhost:PORT from the run step to connect to the application running inside the container.

For more information go to Background step settings - Name and ID.

Plugins and integrations​

Which Drone plugins are supported in Harness CI?​

You can build your own plugins or use one of the many preexisting plugins from the Drone Plugins Marketplace. For more information, go to Explore plugins.

How do I convert Drone plugin settings to Harness CI?​

For information about using Drone plugins in Harness CI, including converting Drone YAML to Harness YAML, go to Use Drone plugins.

How do I add a custom plugin to my Harness CI pipeline?​

For instructions on writing and using custom plugins, go to Write custom plugins.

Can I test my custom plugin before using it in a pipeline?​

Yes, you can test plugins locally.

Why is the PATH variable overwritten in parallel GitHub Actions steps?​

When steps run in parallel and modify the same variables, the resulting value of that common variable depends on the step that modified it last. This can be different with each build, depending on how fast each parallel step executes. This is true for any parallel steps.

When running multiple instances of the same GitHub Action, you must set XDG_CACHE_HOME, as explained in Duplicate Actions.

If you need a variable's value to remain distinct, either run the steps sequentially (rather than in parallel), or find a way to differentiate the variable that each step is modifying, such as by exporting each value as an output variable where you use looping strategy expressions to assign unique identifiers to each variable name.

GitHub Action step can't connect to Docker daemon​

Error messages like cannot connect to the Docker daemon indicate that you might have multiple steps attempting to run Docker at the same time. This can occur when running GitHub Actions in stages that have Docker-in-Docker (DinD) Background steps.

Actions that launch DinD: You can't use GitHub Actions that launch DinD in the same stage where DinD is already running in a Background step. If possible, run the GitHub Action in a separate stage or try to find a GitHub Action that doesn't use DinD.

Actions that launch the Docker daemon: If your Action attempts to launch the Docker daemon, and you have a DinD Background step in the same stage, you must add PLUGIN_DAEMON_OFF: true as a stage variable. For example:

        variables:
          - name: PLUGIN_DAEMON_OFF
            type: String
            description: ""
            required: false
            value: "true"

Harness Cloud: You don't need DinD Background steps with Harness Cloud build infrastructure, and you can run GitHub Actions in Action steps instead of Plugin steps.

GitHub Action step fails with "not a git repository or any of the parent directories"​

This error occurs if the GitHub Action you're using requires a codebase to be present, such as the GraphQL Inspector or DevCycle Feature Flag Code Usages Actions. The Action step isn't compatible with such Actions at this time.

If the Action allows you to override the working-directory, such as with the CodeCov Action, you can use this setting to specify the correct working directory. If no such setting is available, then the Action is not compatible with Harness CI at this time.

Can I integrate my CI builds with the Datadog Pipeline Visibility feature?​

Harness doesn't have OOTB support for Datadog Pipeline Visibility, but you can use the Datadog Drone plugin in a Plugin step.

Why is the Plugin step trying to fetch the entrypoint from the public Docker Hub endpoint even though the connector used in the step points to an internal container registry?​

This can occur if the Build and Push step doesn't have the image's Fully Qualified Name (FQN), even if your Docker connector points to an internal private container registry.

Make sure to use the FQN for the image when pulling from an internal private container registry.

Workspaces, shared volumes, and shared paths​

What is a workspace in a CI pipeline?​

Workspace is a temporary volume that is created when a pipeline runs. It serves as the current working directory for all the steps within a stage. You can use Shared Paths to share additional volumes to the workspace. For more information about the workspace and shared paths go to CI pipeline creation overview - Shared Paths.

Does the workspace persist after a stage ends?​

No, the workspace is destroyed when the stage ends. If you need to shared data across stages, use caching.

How do I share data and volumes between steps in a CI stage?​

The workspace is the current working directory for all steps in a stage. Any data stored to the workspace is available to other steps in the stage. If you need to share additional volumes, decare them as Shared Paths. For more information, go to CI pipeline creation overview - Shared Paths and Share data across steps and stages.

How do I share files between CD/Custom stages and CI stages?​

You can use caching to share data across stages; save the cache at the end of one stage, and restore the cache at the beginning of the next stage. You can also use a Git clone step to clone a repo where you have stored the files you want to share. Git clones steps are available in CI and Custom stages.

What volume is created when I add a shared path?​

When you add a shared path, Harness creates an empty directory type volume and mounts it on each step container.

Does a shared path determine where a file is downloaded?​

No, declaring a shared path doesn't dictate where a download happens. Your stage must include steps or commands that load files to volumes declared in your shared paths, otherwise the volumes remain empty. For example, depending on your pipeline configuration, cache steps might automatically interact with a shared path volume, or you might have a Git Clone step that clones a repo to a shared path volume.

Why are changes made to a container image filesystem in a CI step is not available in the subsequent step that uses the same container image?​

Changes to a container image are isolated to the current step. While a subsequent step might use the same base container image, it is not the literal same container as the previous step. To permanently modify workspace data, you need to interact with the /harness directory (which is the base workspace directory for all steps in the stage), use shared paths, or use caching to share data between steps and stages.

Caching​

What does caching do in Harness CI?​

Caching improves build times and lets you share data across stages.

What caching options does Harness CI offer?​

Harness CI offers a variety of caching options, including S3, GCS, and Harness-managed Cache Intelligence.

How can I download files from an S3 bucket in Harness CI?​

There are two options to download files from an S3 bucket in Harness:

Can I use GCS for caching with Harness CI?​

Yes. Go to Save and Restore Cache from GCS.

Does Harness CI support multilayer caching?​

Yes. Go to Multilayer caching.

How can I use an artifact in a different stage from where it was created?​

Use caching to share data across stages.

What does the Fail if Key Doesn't Exist setting do?​

The Fail if Key Doesn't Exist setting causes the Restore Cache from GCS or Restore Cache from S3 step to fail if the defined cache key isn't found.

Does Harness override the cache when using the Save Cache to S3 step?​

By default, the Save Cache to S3 step doesn't override the cache. You can use the Override Cache setting if you want to override the cache if a cache with a matching key already exists.

How can I check if the cache was restored?​

You can use conditional executions and failure strategies to check if a cache was downloaded and, if it wasn't, install the dependencies that would be provided by the cache.

How do I handle a corrupted file when using the Restore Cache from S3 step?​

If a file becomes corrupted in the bucket during the restoration process, the best practice is to remove the corrupted file from the bucket.

To ensure robustness in your pipeline, consider adding a Failure Strategy to the restore step to mitigate pipeline failures. For example, you can check if a cache was downloaded and, if it wasn't, install the dependencies that would be provided by the cache.

Cache Intelligence​

Cache Intelligence on Harness Cloud Infrastructure​

Cache Intelligence is available on Linux and Windows platforms on Harness Cloud build infrastructure only.

Why can't I enable Cache Intelligence in my CI pipeline?​

Currently, Cache Intelligence is only available for Linux and Windows platforms on Harness Cloud build infrastructure. For more information, go to Cache Intelligence - Supported build infrastructures.

What is the Cache Intelligence cache storage limit?​

Harness Cloud provides up to 2GB of cache storage per account.

What is the cache retention window for Cache Intelligence? Can the cache expire?​

Cache storage is retained for 15 days. This limit resets whenever the cache is updated.

Can different pipelines within the same account access and use the same cache storage for Cache Intelligence?​

All pipelines in the account use the same cache storage, and each build tool has a unique cache key that is used to restore the appropriate cache data at runtime.

What is the cache storage location for Cache Intelligence?​

By default, Cache Intelligence stores data to be cached in the /harness directory. You can specify custom cache paths.

How does Harness generate cache keys for caching build artifacts?​

Harness generates a cache key from a hash of the build lock file (such as pom.xml, build.gradle, or package.json) that Harness detects. If Harness detects multiple tools or multiple lock files, Harness combines the hashes to create the cache key. You can also set custom cache keys.

Is there any API available for Cache Intelligence?​

Yes. Go to Cache Intelligence API.

Background steps and service dependencies​

What is the purpose of Background steps in a CI stage?​

Background steps are used to manage dependent services that need to run for the entire lifetime of a Build stage.

How do I configure the Background step settings?​

For instructions on configuring Background steps, go to Background step settings.

Can Background steps run multiple services simultaneously?​

Yes, you can use multiple background steps to run multiple background services, creating a local, multi-service application.

Do Background steps have limitations?​

Yes. Background steps have these limitations:

  • Background steps don't support failure strategies or output variables.
  • Other steps running in containers can't communicate with Background steps running on Harness Cloud build infrastructure because they don't have a common host.
  • If your build stage uses Harness Cloud build infrastructure and you are running a Docker image in a Background step, you must specify Port Bindings if you want to reference the Background step in a later step in the pipeline (such as in a cURL command in a Run step).

How can a step call a service started by a Background step?​

For information about calling services started by Background steps, go to Name and ID and Port Bindings in Background step settings.

I can't connect to the hostname using the step ID from my Background step, and I get an "Unknown server host" error​

Not all build infrastructures use the step ID when referencing services running in Background steps. For more information, go to Background step settings - Name and ID and Background step settings - Port Bindings.

How is the Background step's entrypoint retrieved during a build?​

Harness connects to the container registry endpoint, based on the container registry specified in the step settings, to retrieve the entrypoint.

Why is Background step always marked as successful even if there are failures executing the entry point?​

This is the expected behavior. Once a Background step initializes, Harness proceeds to the next step in the stage and marks the Background step successful. If your services in Background steps aren't starting, or your subsequent steps are running too soon, add a Run step after the Background step as a health check.

How can I make sure my background service is healthy before running the rest of my pipeline? How can I test that my background service is running?​

Add a Run step after the Background step as a health check.

What are the prerequisites for running Background steps?​

The build infrastructure or environment must have the necessary binaries to run your service. Depending on the build infrastructure, Background steps can use existing binaries in the environment (such as those that are preinstalled on Harness Cloud runners) or pull an image, such as a Docker image, containing the required binaries. For more information, go to Background step settings - Container Registry and Image.

Can Background steps use an external image for PostgreSQL services?​

Yes. Depending on the build infrastructure, Background steps can either use existing binaries in the build environment or pull an image containing the required PostgreSQL binaries. For more information, go to Background step settings - Container Registry and Image.

How do I add volumes for PostgreSQL data in the build workspace?​

With a Kubernetes cluster build infrastructure, use the Volumes setting to add one empty directory volume for each PostgreSQL service you plan to run. For moe information, go to Troubleshooting: Failed to get image entry point.

Can I run a LocalStack service in a Background step?​

Yes. Go to Tutorial: Run LocalStack as a Background step.

Can I configure service dependencies in Gradle builds?​

Yes. For details, go to the Knowledge Base article on Gradle build and daemon issues.

What happens if I don't provide the Fully Qualified Name (FQN) for an image in a private repo?​

FQN is required for images in private repos.

Conditional executions, looping, parallelism, and failure strategies​

Run a step only run if a certain file, like a .toml configuration file, changes in my repo​

To run a step only when a certain file changes, you can define conditional executions based on a JEXL condition that evaluates to true for the specific file. For example, you might use a payload expression to get details from a Git event payload, such as a PR event that triggers a build.

Alternately, you could isolate the step in a stage by itself, configure a Git webhook trigger with a Changed File trigger condition that listens for changes to the target file, and then configure the trigger to run selective stage execution and run all stages that you want to run when that file changes, including the stage with your isolated step.

Can I assert an environment variable within a JEXL conditions?​

While we support output variables that can point to an environment variable, we do not support the direct referencing of environment variables in JEXL conditions, even when using the feature flag CI_OUTPUT_VARIABLES_AS_ENV (which automatically makes environment variables available for other steps in the same build stage).

The conditions in JEXL only allow the use of variable expressions that can be resolved before the stage is executed. Since environment variables are resolved during runtime, it is not possible to utilize variable expressions that cannot be resolved until the stage is run.

What does a failure strategy consist of?​

Failure strategies include error conditions that trigger the failure and actions to take when the specified failure occurs.

Can I make a step, stage, or pipeline fail based on the percentage of test cases that fail or succeed?​

Currently, Harness can't fail a step/stage/pipeline based on a percentage of test results. To achieve this, you would need to manually parse the test results (which are created after the test step execution) and export some variables containing the percentages you want to track. You could then have a step throw an error code based on the variable values to trigger a failure strategy, or you could manually review the outputs and manually mark the stage as failed.

Due to potential subjectivity of test results, it would probably be better to handle this case with an Approval stage or step where the approver reviews the test results.

Can I abort a pipeline if the referenced branch is deleted?​

This is not natively supported; however you could create a Git webhook trigger that listens for a specific delete event with auto-abort previous execution. This trigger would only go off on the specified delete event and, therefore, it would only cancel the ongoing executions if the delete event occurred.

Is there a way to abort a running pipeline from a step in that pipeline?​

You can use the putHandleInterrupt API to abort a running pipeline from a step in the pipeline.

Can I add notifications, such as failure notifications, to stage templates?​

While notifications are a pipeline-level setting that is not explicitly available at the stage level, you can use Plugin steps to add notifications in your stage templates. Configure the Plugin step to use a use a Drone plugin or a custom plugin to send an email notification, Slack notification, or otherwise.

Logs and execution history​

How do I access build logs?​

For information about viewing build and build logs, go to:

Does Harness limit log line length?​

Yes, there is a single-line limit of 25KB. If an individual line exceeds this limit, it is truncated and ends with (log line truncated). If you need to extract long log lines, you can export full logs.

Truncated execution logs​

Each CI step supports a maximum log size of 5MB. Harness truncates logs larger than 5MB. If necessary, you can export full logs.

Furthermore, there is a single-line limit of 25KB. If an individual line exceeds this limit, it is truncated and ends with (log line truncated).

Note that the CI log limit is different from the Harness CD log limit.

Export full logs​

If your log files are larger than 5MB, you can export execution logs to an external cache and examine the full logs there.

  1. Add a step to your pipeline that records each step's complete logs into one or more files.
  2. If you have a lot of log files or your logs are large, add a step to compress the log files into an archive.
  3. Use an Upload Artifact step to upload the log files to cloud storage.
  4. Repeat the above process for each stage in your pipeline for which you want to export the full logs.
  5. Examine the log files in your cloud storage. If you used the S3 Upload and Publish or Artifact Metadata Publisher plugins, you can find direct links to your uploaded files on the Artifacts tab on the Build detail page.
Log forwarding

You can also use a service, such as env0, to forward logs to platforms suited for ingesting large logs.

Step logs disappear​

If step logs disappear from pipelines that are using a Kubernetes cluster build infrastructure, you must either allow outbound communication with storage.googleapis.com or contact Harness Support to enable the CI_INDIRECT_LOG_UPLOAD feature flag.

For more information about configuring connectivity, go to:

Step logs don't load in real time​

Go to CI step logs don't load in real time.

Step succeeds even when explicitly executing exit 1 in a Bash script that runs in script's background​

Harness determines the execution status for a step based on the exit status received from the primary script execution.

When you call a function in the background of a script, it doesn't directly impact the exit status of the main script. Therefore, if you manually call exit 1 within a background function, it won't cause the step to fail if the primary script succeeds.

This behavior is consistent with how scripts operate both inside and outside of Harness.

Build step fails due to ResourceExhausted​

Build and Push steps can return a ResourceExhausted error, such as:

exit status 1 rpc error: code = ResourceExhausted desc = grpc: received message larger than max (4950319 vs. 4194304)

This can be related to log streaming during the Build and Push step or a Run step executing a build script. It indicates that the logs are too large for the log streaming service to handle.

If your build uses tee commands to print logs to the console, consider removing these commands or output these logs to a file that you can then upload as an artifact or send by email.

If your builds time out with this error during stage initialization, and you're using a Kubernetes cluster build infrastructure, you can contact Harness Support to enable a beta feature that shortens long environment variables, such as commit messages and PR titles.

Can I get logs for a service running on Harness Cloud when a specific Run step is executing?​

Yes. To do this, you can add a step that runs in parallel to the Run step, and have that parallel step get the service's logs while the build runs. For an example, go to Use a parallel step to monitor failures.

How to get the build ID of a pipeline execution?​

You can use the expression <+execution.steps.stepId.build.buildNumber> in a Run step to echo the build ID for that execution.

Builds older than 30 days aren't on the Project Overview page.​

The default timescale setting for the overview page is 30 days. You can change this setting.

A previous execution is missing from my Builds dashboard.​

First, check the timescale setting on the dashboard. The default is 30 days, which hides builds older than 30 days.

Then, make sure you are in the correct project and that you have permission to view that particular pipeline.

Finally, if your build is older than six months, it is outside the data retention window. For more information, go to data retention.

Can I compare pipeline changes between builds?​

Yes. Go to view and compare pipeline executions.

How do I create a dashboard to identify builds that end with a timeout in a specific task?​

You can create a custom dimension for this, such as:

contains(${pipeline_execution_summary_ci.error_message}, "Timeout")

Debug mode​

Why does the debug mode SSH session close after some time?​

Sessions automatically terminate after one hour or at the step timeout limit, whichever occurs first.

Why can't I launch a remote debug session? Can I debug a pipeline that doesn't have an obvious failure?​

There are several debug mode requirements, namely that the pipeline must have a failed step in order to generate the debug session details. If your build doesn't have any failed steps, you won't be able to access a remote debug session. However, you can force a build to fail if you need to troubleshoot pipelines that appear to build successfully but still need remote troubleshooting. To do this, add a Run step with the command exit 1. This forces the build to fail so you can re-run it in debug mode.

Re-run in debug mode isn't available for a new pipeline​

Debug mode is not available for a pipeline's first build. Run the pipeline again and, if it meets the debug mode requirements, you should be able to trigger re-run in debug mode.

AIDA for CI​

For information about using AIDA to troubleshoot your Harness CI builds, go to Troubleshoot builds with AIDA.

CI with CD​

Why did the CI stage still go through despite setting a freeze window?​

Freeze windows only apply to CD stages.

Can I use the expression <+codebase.commitSha> in a CD stage to get the commit ID?​

Yes, you can use <+codebase.commitSha> to get the commit ID if the CD stage is after the Build (CI) stage in your pipeline.

This expression doesn't work if there is no CI stage in your pipeline, or if the CD stage runs before the CI stage.

Additionally, including a clone step in your CD stage won't populate the <+codebase.commitSha> expression. This expression is dependent on the CI stage's built-in clone codebase step.

Can I reference a secret type output variable exported from a CD or custom stage in CI stage?​

No. Currently CI stages don't support secret type output variables from CD or custom stages.

Can I trigger a Build stage with an artifact trigger?​

While it is possible to trigger deployments with artifact triggers, there are currently no CI-specific triggers for artifacts.

Performance and build time​

What are the best practices to improve build time?​

For information about making your pipelines faster and more efficient, go to Optimize and enhance CI pipelines.

How do I reduce the time spent downloading dependencies for CI builds?​

You can create pre-built Docker images that have all required dependencies, and then periodically update these images with the latest dependencies. This approach minimizes dependency download time during the build process by packaging your dependencies into one image. Harness offers pre-build public images that contain common and useful tools for CI pipelines.

What are the benefits of excluding unnecessary files and packages from Docker images?​

Excluding unnecessary files and packages reduces build times and creates in smaller, more efficient, and more portable Docker images.

How can Harness input sets help automate a CI pipeline?​

Input sets are collections of runtime inputs for a pipeline executions. With input sets, you can use the same pipeline for multiple scenarios. You can define each scenario in an input set or overlay, and then select the appropriate scenario when you execute the pipeline.

Harness Platform rate limits​

For stability, Harness applies limits to prevent excessive API usage. Harness reserves the right to change these limits at any time. For more information, go to Platform rate limits.

Running concurrent builds shows "queued license limit reached"​

Queued license limit reached means that your account has reached the maximum build concurrency limit. The concurrency limit is the number of builds that can run at the same time. Any builds triggered after hitting the concurrency limit either fail or are queued.

If you frequently run many concurrent builds, consider enabling Queue Intelligence for Harness CI, which queues additional builds rather than failing them.

Can I use Harness Queue Intelligence with Kubernetes cluster build infra?​

Queue intelligence is currently supported for Harness Cloud only.

What is the timeout limit for a CI pipeline?​

By default, a stage can run for a maximum of 24 hours on a Kubernetes cluster build infrastructure and a maximum of 30 minutes on Harness Cloud build infrastructure.

For pipelines, the default timeout limit is, generally, the product of the stage limit multiplied by the number of stages. For example, a pipeline with three stages that use a Kubernetes cluster build infrastructure could run for a maximum of 72 hours. However, you can also set an overall pipeline timeout limit in each pipeline's Advanced Options.

For steps, you can set a custom timeout limit in each step's Optional Configuration settings. In stages that use a Kubernetes cluster build infrastructure, the default timeout for steps is 10 hours. However, this is constrained by the stage timeout limit of 24 hours. For example, if a stage has three steps, the total run time for the three steps can't exceed 24 hours or the stage fails due to the stage timeout limit.

Can I add an Approval step to a CI stage?​

Currently, Approval steps aren't compatible with CI stages.

General issues with connectors, secrets, delegates, and other Platform components​

For troubleshooting and FAQs for Platform components that aren't specific to CI, such as RBAC, secrets, secrets managers, connectors, delegates, and otherwise, go to the Harness Platform Knowledge Base or Troubleshooting Harness.