Skip to main content

Build Intelligence Overview

Build Intelligence is part of Harness CI Intelligence, a suite of features in Harness CI designed to improve build times. It saves time by reusing outputs from previous builds. BI works by storing these outputs locally or remotely and retrieving them when inputs haven't changed. This process avoids the need to regenerate outputs, significantly speeding up the build process and enhancing efficiency.

note
  • Build Intelligence is currently only offered to Harness Cloud, with support for self hosted coming soon. This feature is behind the feature flag CI_CACHE_ENABLED.
  • 'Build intelligence' CI stage property, which enables automatic setup of Build Intelligence on Harness Cloud when using supported build tools in Run and Test steps is behind the feature flag CI_ENABLE_BUILD_CACHE_HOSTED_VM.

Contact Harness Support to enable the feature.

Build Intelligence in Harness CI is currently available for Gradle and Bazel with Maven support coming soon. Regardless of the programming language used in your projects, as long as you're building with a supported build tool, you can take advantage of Build Intelligence to optimize your builds.

Auto-setup of Build Intelligence​

The Build Intelligence stage property simplifies the setup of Build Intelligence in Harness Cloud. When enabled, it automatically configures Build Intelligence for supported build tools (currently Gradle and Bazel) in Run and Test steps without requiring any additional configuration. This automation is particularly beneficial in CI pipelines, as it eliminates the need for developers to modify project settings in their git repository (such as Gradle’s settings.gradle) to configure the cache.

Build Intelligence setup is fully automated when the CI_ENABLE_BUILD_CACHE_HOSTED_VM feature flag is enabled. However, for local development (e.g., on a developer's laptop), manual configuration is necessary to take advantage of caching.

Enabling Auto-setup​

  • Via Visual editor: To enable Build Intelligence, go to the CI Stage Overview tab and toggle Build Intelligence to true.
  • In yaml, set build intelligence with buildIntelligence property, as you can see below
    - stage:
        identifier: build
        name: build
        type: CI
        spec:
          cloneCodebase: true
          buildIntelligence: 
            enabled: true # Build intelligence enabled
          execution:
            steps:
              - step:
                  type: Action
                  name: Set up Gradle
                  identifier: Set_up_Gradle
                  spec:
                    uses: gradle/gradle-build-action@v2
                    with:
                      gradle-version: "8.5"
              - step:
                  type: Run
                  name: build
                  identifier: build
                  spec:
                    shell: Sh
                    command: ./gradlew build 
          platform:
            os: Linux
            arch: Amd64
          runtime:
            type: Cloud
            spec: {}

Build Intelligence configuration​

For local development (e.g., running build commands on a developer's laptop), manual configuration is necessary to take advantage of the remote cache. Manual configuration may also be needed in case you run your build in Harness CI but would not like to use auto-setup.

Please follow the instructions below, for either Bazel or Gradle, in case manual configuration is needed.

Build Intelligence Support for Gradle​

Gradle is the open source build system of choice for Java, Android, and Kotlin developers. Harness CI offers Build Intelligence support for Gradle to optimize build times by reusing outputs from previous builds.

How it works?​

  1. Plugin Integration: The Build Intelligence plugin for Gradle is imported into your project. This plugin interacts with Gradle to handle cache pull and push operations.
  2. Cache Operations: At the start of the build, the plugin registers with Gradle to check for cached build outputs. If available, it retrieves and provides them to Gradle, avoiding the need to regenerate them.

The above operation is transparent to you as a user and happens in the background.

Configuration for Gradle​

  1. Import the build cache plugin in settings.gradle file: Customers using prod1 or prod2 clusters don't need to configure the endpoint parameter in the settings below and it'll be populated by the plugin. The default value for this endpoint for prod1 or prod2 is https://app.harness.io/gateway/cache-service. For customers not using prod1 or prod2 clusters, they'll need to configure the endpoint parameter.
// import the plugin
buildscript {
    repositories {
        mavenCentral()
    }
    dependencies {
        classpath 'io.harness:gradle-cache:0.0.4'
    }
}
...

// apply the plugin
apply plugin: 'io.harness.gradle-cache'

// build cache config

buildCache {
    local {
        // Local build cache is dangerous as it might produce inconsistent results
        // in case developer modifies files while the build is running
        enabled = false
    }
    remote(io.harness.Cache) {
        accountId = System.getenv('HARNESS_ACCOUNT_ID') 
        token = System.getenv('HARNESS_PAT') 
        push = "true"
        endpoint = System.getenv('HARNESS_CACHE_SERVICE_ENDPOINT') 
    }
}

For Build Intelligence, you'll need to turn on CI_CACHE_ENABLED FF.

  1. Enable build cache in gradle.properties file:
org.gradle.caching = true

Sample pipeline for build intelligence for Gradle​

pipeline:
  projectIdentifier: SAMPLE_PROJECT
  orgIdentifier: SAMPLE_ORG
  tags: {}
  properties:
    ci:
      codebase:
        connectorRef: SAMPLE_CONNECTOR_ID
        build: <+input>
  stages:
    - stage:
        name: Build and Test
        identifier: Build
        description: ""
        type: CI
        spec:
          cloneCodebase: true
          caching:
            enabled: false
            paths: []
          execution:
            steps:
              - step:
                  identifier: SAMPLE_Test
                  type: Test
                  name: SAMPLE_Test
                  spec:
                    connectorRef: <+input>
                    image: <+input>
                    shell: Sh
                    command: gradle --build-cache unitTest -PmaxParallelForks=16 -PignoreFailures=true --profile
          platform:
            os: <+input>
            arch: Amd64
          runtime:
            type: Cloud
            spec: {}
  identifier: SAMPLE_REPO
  name: SAMPLE_REPO
  variables:
    - name: var1
      type: String
      description: ""
      required: false
      value: <+input>

Build Intelligence Support for Bazel​

Bazel is an open-source build and test tool designed for high performance, scalability, and handling large codebases across multiple languages and platforms. Harness CI offers Build Intelligence support for Bazel to optimize build times by reusing outputs from previous builds.

How it works?​

  1. Proxy binary: Download the proxy binary from https://app.harness.io/storage/harness-download/harness-ti/cache-proxy/OS/ARCH/cache-proxy. Replace OS/ARCH in the URL with one of the following options:
  • linux/amd64
  • linux/arm
  • mac/amd64
  • mac/arm
  • windows/amd64

The cache proxy facilitates secure uploading and downloading of cache blobs by interacting with the cache service to obtain necessary access URLs.

  1. Start the proxy server before the bazel command. You need to set the following environment variables:
HARNESS_CACHE_PROXY_ENABLED=true
HARNESS_CACHE_SERVER_URL=
HARNESS_CACHE_SERVER_API_TOKEN=

Use https://app.harness.io/gateway/cache-service for HARNESS_CACHE_SERVER_URL for prod1 or prod2 clusters.

For HARNESS_CACHE_SERVER_API_TOKEN, create a Harness Personal Access Token or Service Account Token with core_account_edit permission.

Here is a sample script:

if [[ ! -z "${ENABLE_CI_CACHE// }" ]] && [[ "$ENABLE_CI_CACHE" == "true" ]]
then
    # Start the proxy server
    mkdir -p /tmp/cachebin
    curl -L -o /tmp/cachebin/harness-cache https://app.harness.io/storage/harness-download/harness-ti/cache-proxy/OS/ARCH/cache-proxy # Replace **OS/ARCH** in the URL
    chmod +x /tmp/cachebin/harness-cache
    echo "Starting cache proxy with account $HARNESS_ACCOUNT_ID"
    /tmp/cachebin/harness-cache server > /tmp/cachebin/server.log 2>&1 &
fi
  1. Add the build cache endpoint to the relevant bazelrc file:
build --remote_cache=http://localhost:8082/cache/bazel

You can also use a build command without rc files:

bazel build --remote_cache=http://localhost:8082/cache/bazel //...