> For the complete documentation index, see [llms.txt](https://developer.harness.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://developer.harness.io/continuous-integration/use-harness-ci/use-harness-ci/caching-ci-data/docker-layer-caching.md).

# Docker layer caching

Caching Docker layers between pipeline executions can reduce build times.

When a layer changes in your Docker image, that layer, and all subsequent layers, are rebuilt. Docker layer caching can cache unchanged layers and reuse them across pipeline runs, which can dramatically reduce build times.

{% hint style="info" %}
To maximize savings, consider modifying your Dockerfile to [use the cache efficiently](https://docs.docker.com/build/cache/#how-can-i-use-the-cache-efficiently).
{% endhint %}

### Docker Layer Caching, an Intelligence Feature <a href="#docker-layer-caching-an-intelligence-feature" id="docker-layer-caching-an-intelligence-feature"></a>

With **Docker Layer Caching (DLC)** , a [Harness CI Intelligence](/continuous-integration/use-harness-ci/use-harness-ci/harness-ci-intelligence.md) feature, Harness seamlessly caches Docker layers between builds to accelerate the time it takes to build Docker images.

You can use DLC with any [build infrastructure](/continuous-integration/use-harness-ci/use-harness-ci/set-up-build-infrastructure/which-build-infrastructure-is-right-for-me.md). When you use DLC with Harness CI Cloud, the cache is stored in the Harness-managed environment.

{% hint style="info" %}

* Docker Layer Caching is now Generally Available (GA).

If this feature is not yet enabled in your account, reach out to [Harness Support](mailto:support@harness.io) for assistance.
{% endhint %}

#### Cache storage <a href="#cache-storage" id="cache-storage"></a>

When you use Docker Layer Caching with Harness CI Cloud, the cache is stored in the Harness-managed environment. When running builds on self-managed infrastructure, you will need to provide your own storage.

{% tabs %}
{% tab title="Harness Cloud" %}
When you use Docker Layer Caching with [Harness CI Cloud](/continuous-integration/use-harness-ci/use-harness-ci/set-up-build-infrastructure/use-harness-cloud-build-infrastructure.md), you don't need to bring your own storage, because the cache is stored in Harness-managed Harness Cloud storage.

All pipelines in the account use the same cache storage, and each build tool has a unique cache key that is used to restore the appropriate cache data at runtime.

The cache storage limit depends on your subscription plan type. Go to [Subscriptions and licenses](/continuous-integration/troubleshooting-and-resources/ci-subscription-mgmt.md#usage-limits) to review usage limits.

Harness doesn't limit the number of caches you can store, but, once you reach your storage limit, Harness continues to save new caches by automatically evicting old caches.

The cache retention window is 15 days, which resets whenever a cache is updated.
{% endtab %}

{% tab title="Self-managed build infrastructures" %}
When running builds in self-managed infrastructures, [configure default object storage (Azure Blob Storage, GCP Cloud Storage, AWS S3, or any S3-compatible storage)](/harness-ai/use-harness-platform/settings/default-settings.md#continuous-integration) that Harness can use to seamlessly store and manage the cache.

If your storage is not S3-compatible or you do not want to use access key and secret key for authentication, consider using [remote cache image](#remote-cache-image) instead.

Set a bucket-level retention policy for efficient cache management.

{% hint style="info" %}

* Enabling DLC when running on Kubernetes requires *privileged mode* on the cluster where the builds run.
* To use path-style S3 addressing in self-hosted Build and Push steps with DLC, set `PLUGIN_PATH_STYLE: "true"` in envVariables. This allows compatibility with S3 providers that do not support virtual-hosted style URLs.
* When using Azure Blob Storage, authenticate with the Azure connector using **Service Principal** or **OIDC**. Managed Identity authentication is not supported for Docker Layer Caching.
  {% endhint %}
  {% endtab %}
  {% endtabs %}

#### Build engine <a href="#build-engine" id="build-engine"></a>

When you enable Docker Layer Caching on a **Build and Push** step, the step uses [BuildKit / Buildx](https://docs.docker.com/build/buildkit/) to build the image and store the cache, instead of the default plugin (Kaniko on Kubernetes cluster build infrastructure, or drone-docker on other infrastructures).

DLC stores cached layers in Harness-managed storage on Harness CI Cloud or in object storage on self-managed build infrastructure. Go to [Object Store for Self-Managed Build Infrastructure](/harness-ai/use-harness-platform/settings/default-settings.md#object-store-for-self-managed-build-infrastructure) to configure storage for self-managed infrastructures. This switch happens automatically whenever `caching: true` is set on the step. For self-managed Kubernetes, Buildx requires *privileged mode* on the cluster.

#### Enable Docker Layer Caching <a href="#enable-docker-layer-caching" id="enable-docker-layer-caching"></a>

1. If you are *not* using Harness Cloud build infrastructure, you must [configure default object storage (Azure Blob Storage, GCP Cloud Storage, AWS S3, or any S3-compatible storage)](/harness-ai/use-harness-platform/settings/default-settings.md#continuous-integration) that Harness can use to store and manage caches.

   This is not required for Harness Cloud build infrastructure. For more information, go to [Cache storage](#cache-storage).
2. To enable Docker layer caching with Harness CI cloud, select **Enable Docker layer caching** in your [Build and Push steps](/continuous-integration/use-harness-ci/use-harness-ci/build-and-upload-artifacts/build-and-push.md).

Here is a YAML example of a **Build and Push an image to Docker Registry** step that uses DLC.

```yaml
   - step:
      type: BuildAndPushDockerRegistry
      name: Build and push to Docker
      identifier: Build_and_push_to_Docker
      spec:
         connectorRef: YOUR_DOCKER_CONNECTOR_ID
         repo: DOCKER_USERNAME/DOCKER_REPO_NAME
         caching: true 
         tags:
            - <+pipeline.sequenceId>
```

### Remote cache image <a href="#remote-cache-image" id="remote-cache-image"></a>

Remote cache image is an alternative to Harness CI Intelligence Docker layer caching.

{% hint style="info" %}
Remote cache image support is currently available only for Kubernetes infrastructure.
{% endhint %}

Remote caching leverages your existing Docker registry to pull previously built layers. Each Docker layer is uploaded as an image to a Docker repo you identify. If the same layer is used in subsequent builds, Harness downloads the layer from the Docker repo. You can also specify the same Docker repo for multiple Build and Push steps, enabling them to share the same remote cache.

{% hint style="info" %}
If you enable both Harness Intelligence Docker layer caching *and* set remote cache image on the same [Build and Push step](/continuous-integration/use-harness-ci/use-harness-ci/build-and-upload-artifacts/build-and-push.md), Harness will use the remote cache image for caching.
{% endhint %}

To enable remote cache image in your [Build and Push steps](/continuous-integration/use-harness-ci/use-harness-ci/build-and-upload-artifacts/build-and-push.md):

1. Select the **Remote Cache Image** option.
2. Enter the name of the remote cache repository/registry and image, depending on the step you are using. For example:
   * Build and Push to Docker: `NAMESPACE/IMAGE` or `REGISTRY/IMAGE`
   * Build and Push to GCR: `gcr.io/PROJECT_ID/IMAGE`
   * Build and Push to GAR: `LOCATION-docker.pkg.dev/PROJECT_ID/REPO/IMAGE`
   * Build and Push to ECR: `APP/IMAGE`
   * Build and Push to ACR: `CONTAINER_REGISTRY.azurecr.io/IMAGE`
3. For the best performance, make sure:
   * The remote cache repo/registry exists in the same host/account and project/organization as the build image.
   * The specified repo/registry and image already exist. For some providers, Harness can automatically create the repository if it doesn't already exist, but this is not guaranteed. For any provider, Harness needs an existing image to overwrite.

### Optimize Docker images and Dockerfiles <a href="#optimize-docker-images-and-dockerfiles" id="optimize-docker-images-and-dockerfiles"></a>

In addition to enabling Docker layer caching, the following practices can reduce Docker image build times.

<details>

<summary>Pre-build images that include all required dependencies</summary>

If most of the build time is spent downloading dependencies, you should pre-build an image with all required dependencies in a separate pipeline. Then, set up a periodic pipeline that builds the image with all the latest dependencies and pushes it to your Docker registry. Use this image in all of your build pipelines.

Pre-building images with all required dependencies is more efficient than downloading them to a baseline image as part of the Build setup. This is especially true if you update your images often to ensure that they include all the latest updates.

</details>

<details>

<summary>Exclude unnecessary files and packages from your images</summary>

In addition to reducing build times, excluding unnecessary files and packages makes the resulting images smaller, simpler, and more portable. You can use [dockerignore](https://docs.docker.com/engine/reference/builder/#dockerignore-file) files to exclude unnecessary files and folders from your images.

</details>

<details>

<summary>Optimize Dockerfiles</summary>

* Sort multi-line arguments in your Dockerfile alphabetically. This makes it easier to update and avoid duplicate packages.
* Review [Docker's best practices for writing Dockerfiles](https://docs.docker.com/develop/develop-images/dockerfile_best-practices/).
* Write your Dockerfiles to [use Docker layer caching efficiently](https://docs.docker.com/build/cache/#how-can-i-use-the-cache-efficiently).

</details>

### Troubleshooting DLC <a href="#troubleshooting-dlc" id="troubleshooting-dlc"></a>

#### I use DLC but I do not observe time savings <a href="#i-use-dlc-but-i-do-not-observe-time-savings" id="i-use-dlc-but-i-do-not-observe-time-savings"></a>

Docker Layer Caching (DLC) provides caching benefits, but certain operations may not see significant improvements due to how caching works. For example, `FROM` statements never use cache, as base image layers are always pulled to ensure freshness. Additionally, external dependencies (like copying files from non-cached sources) may not be fully cached. DLC primarily caches self-contained operations, and checksum-based steps (like `COPY` or `ADD`) only reuse cache when source files remain unchanged. To maximize caching benefits, optimize Dockerfile instructions to reduce dependency on external sources.

See [Optimize Docker images and Dockerfiles](#optimize-docker-images-and-dockerfiles) to learn more.

#### Base Image Connector <a href="#base-image-connector" id="base-image-connector"></a>

When using Docker as a Base Image Connector, select the Docker connector to use for base image pulls. This capability is generally available. Authenticated base image pulls help avoid Docker rate limiting.

{% @harness-feedback/feedback %}