Ctrlk
Sign UpGet a DemoContact Us
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Build and push with Docker Buildx Bake

Use Docker Buildx Bake for advanced multi-platform and multi-registry builds

Docker Buildx Bake is an advanced build orchestration feature that allows you to define and manage complex build configurations using HCL, JSON, or Docker Compose files. Unlike traditional single-target builds, Bake enables you to build multiple targets simultaneously, push to multiple registries, and manage sophisticated build matrices with declarative configuration.

In Harness CI, you can use Docker Buildx Bake through the Plugin step with the drone-buildx plugin to orchestrate complex container builds, such as:

  • Multi-architecture builds (AMD64, ARM64, etc.)

  • Multi-registry pushes in a single step

  • Build matrices (multiple versions, variants)

  • Advanced caching strategies

  • OIDC-based authentication

ADVANCED FEATURE

Docker Buildx Bake is designed for advanced use cases where you need to push to multiple registries or orchestrate complex multi-target builds. For simpler scenarios with a single registry, use the standard Build and Push steps.

When to use Buildx Bake

Use Docker Buildx Bake when you need:

  • Multi-registry pushes: Push the same image to multiple registries (Docker Hub, ECR, GCR, GitHub Container Registry) in a single build step

  • Multi-architecture builds: Build and push images for multiple platforms (linux/amd64, linux/arm64) in parallel

  • Build matrices: Build multiple versions or variants of an image (e.g., different Python versions, different base images)

  • Complex build orchestration: Coordinate multiple related builds with dependencies

  • Declarative configuration: Define all build targets, platforms, tags, and outputs in a single bake file

PLUGIN STEP ONLY

Buildx Bake is only available through the Plugin step, not the out-of-the-box (OOTB) Build and Push steps. OOTB steps support only a single connector, while Bake mode requires passing authentication credentials via the plugin's configuration.

Prerequisites

You need:

  • A Harness CI pipeline with a Build stage

  • A Dockerfile and codebase to build from

  • A Buildx Bake definition file (HCL, JSON, or Docker Compose format)

  • Docker registry credentials (as Harness secrets or via OIDC)

  • For OIDC: Configured OIDC trust relationship with your cloud provider

Bake file basics

A Buildx Bake file defines your build targets, platforms, tags, cache configuration, and outputs. Here's a basic example:

Use Buildx Bake in a Plugin step

To use Buildx Bake in Harness CI, add a Plugin step in your Build stage and configure it to use the drone-buildx plugin.

Basic example

Here's a minimal example that builds and pushes to Docker Hub using Bake:

  1. In your Build stage, add a Plugin step

  2. Configure the following settings:

    • Name: Build and Push with Bake

    • Container Registry: Your Docker connector

    • Image: plugins/buildx

    • Settings:

SINGLE REGISTRY LIMITATION

The username and password settings only work for a single registry. For multi-registry pushes, you must use the config setting with a Docker config JSON containing credentials for all registries. See the complete example below.

Complete example: Multi-registry with OIDC and matrix builds

This comprehensive example demonstrates building multi-architecture images for multiple Python versions and pushing to both AWS ECR and GitHub Container Registry using OIDC authentication.

Pipeline overview

The pipeline consists of three stages:

  1. Setup Docker Config: Authenticate with AWS using OIDC and create a Docker config JSON with credentials for multiple registries

  2. Parallel Build Stages: Build for AMD64 and ARM64 architectures in parallel, with a matrix of Python versions (3.9-3.12)

  3. Manifest Stage: Create multi-architecture manifests combining AMD64 and ARM64 images

Bake files

Main bake file:

Version-specific override files:

Complete pipeline YAML

PLATFORM SELECTION

The build architecture is determined by the stage's platform configuration. Set platform.arch to Amd64 or Arm64 to control which architecture the build runs on. The plugins/buildx image is multi-arch and will automatically run the correct variant for your selected platform.

Key components explained

1. Docker Config JSON format

The Docker config uses base64-encoded authentication:

The tr -d '\n' command removes newlines to create a single-line JSON string, which is required when passing the config as a pipeline variable.

2. Bake options syntax

The bake_options setting uses semicolon-delimited arguments:

This expands to:

  • -f flag followed by the version-specific file (e.g., 3.9.hcl)

  • image as the target name to build

Multiple bake files are merged, with later files overriding earlier ones.

3. Environment variables in bake files

Pass variables to bake files via environment variables:

These are automatically available in your bake file as variable blocks.

4. Multi-arch manifest creation

The manifest stage uses docker buildx imagetools to combine platform-specific images:

This creates a single manifest that points to both architecture variants, allowing Docker to automatically pull the correct image for the runtime platform.

Plugin settings

Required settings

Setting
Description
Example

bake_file

Path to your Buildx Bake definition file (HCL/JSON/Compose). When set, the plugin runs docker buildx bake instead of docker buildx build.

docker-bake.hcl

Authentication settings

You must provide authentication for all registries referenced in your bake file. Choose one of these methods:

Setting
Description
Example

username and password

Registry username and password. Only works for single registry.

username: <+secrets.getValue("dockerhub_username")>

password: <+secrets.getValue("dockerhub_password")>

config

Docker config JSON with auth for multiple registries. Required for multi-registry pushes.

config: <+pipeline.stages.Setup.spec.execution.steps.Create_Docker_Config.output.outputVariables.DOCKER_CONFIG_JSON>

Optional settings

Setting
Description
Default
Example

bake_options

Semicolon-delimited extra bake CLI args and/or target names. Do NOT include --push or --load (added automatically).

None

--progress=plain;web;api

-f;override.hcl;image

builder_driver

Buildx builder driver. Use docker-container for registry cache exports.

docker

docker-container

builder_name

Custom builder name

Auto-generated

my-builder

metadata_file

Path to write build metadata

None

/tmp/metadata.json

IMPORTANT NOTES

  • Do NOT include --push or --load in bake_options. The plugin automatically adds --push for normal builds and --load for dry runs.

  • Driver selection: If your bake file uses cache-to with registry exports, set builder_driver: docker-container explicitly. The plugin does not auto-switch the driver in Bake mode.

  • Ignored settings: In Bake mode, the plugin ignores classic cache environment variables (cache_from, cache_to, no_cache). Define cache configuration in your bake file instead.

  • Tar export: Classic tar export (tar_path) is not applied in Bake mode. Define outputs in your bake file if needed.

Bake file examples

HCL format (recommended)

JSON format

Docker Compose format

Advanced use cases

Multiple bake files with overrides

Use multiple bake files to compose configurations:

Base configuration:

Environment-specific overrides:

Pipeline step:

Dynamic tagging with Git info

Use Harness expressions for dynamic tags:

Bake file:

Pipeline step:

Build-time secrets

Pass secrets to your build without storing them in the final image:

Dockerfile:

Bake file:

Pipeline step:

Troubleshooting

Build fails with "cache export not supported"

Problem: Error message indicates that cache export to registry is not supported.

Solution: Set builder_driver: docker-container in your plugin settings:

Authentication fails for one of multiple registries

Problem: Build succeeds but push fails for one registry in a multi-registry setup.

Solution: Verify all registries are in your Docker config JSON with valid credentials:

Docker config JSON formatting error

Problem: Error about invalid config JSON or authentication failures.

Solution: Ensure the config JSON is a single line without newlines:

Also verify base64 encoding includes the -w 0 flag to prevent line wrapping:

Platform-specific build fails

Problem: Build fails for specific platforms (e.g., ARM64).

Solution: Verify the stage's platform configuration matches your build requirements:

For ARM builds on AMD64 hosts, QEMU emulation may be needed (slower but functional).

Bake file not found

Problem: Error indicates the bake file cannot be found.

Solution: Ensure the bake_file path is relative to your repository root:

Matrix strategy not loading override files

Problem: Version-specific override files are not being loaded.

Solution: Ensure you use the -f flag in bake_options:

This tells bake to load both the main file and the version-specific file.

Environment variables not recognized in bake file

Problem: Variables defined in the pipeline are not available in the bake file.

Solution: Ensure variables are defined in the bake file and passed via envVariables:

Bake file:

Pipeline:

Manifest creation fails with "manifest not found"

Problem: docker buildx imagetools create fails with manifest not found.

Solution: Verify that the source platform-specific images exist before creating the manifest:

Ensure the Build stages completed successfully before the Manifest stage runs.

See also

PreviousBuild and Push to GAR
NextBuild and Push to GHCR

Last updated

Was this helpful?