Configure a Kubernetes build farm to use self-signed certificates

CI build pods can interact with servers using self-signed certificates.

CI build infrastructure pods can interact with servers using self-signed certificates. This option is useful for organizations that prefer to use internal certificates instead of certificates generated by a public Certificate Authority (CA).

• This topic assumes that you are familiar with how to implement SSL in Kubernetes. General information about implementing SSL is outside the scope of this topic.

• With a Kubernetes cluster build infrastructure, all Build and Push steps use Chainguard's maintained Kaniko fork. Kaniko reads certificates from /kaniko/ssl/certs/additional-ca-cert-bundle.crt.

• Harness uses a UBI image for the Git Clone step. UBI reads certificates from /etc/ssl/certs/ca-bundle.crt.

• Different base images use different paths as their default certificate location. For example, Alpine images use the path /etc/ssl/certs/ca-certificates.crt. For other images, make sure you verify the default certificate path.

Custom certificates or self-signed certificates are only supported for linux nodes in a Kubernetes cluster. For Windows, directly mounting the certificate to the destination path will not work.

Enable self-signed certificates

1. Create a Kubernetes secret or config map with the required certificates in the same namespace used by the Harness Delegate. For example:

   apiVersion: v1
   kind: Secret
   metadata:
     name: addcerts
     namespace: harness-delegate-ng
   type: Opaque
   stringData:
     ca.bundle: |
       -----BEGIN CERTIFICATE-----
       XXXXXXXXXXXXXXXXXXXXXXXXXXX
       -----END CERTIFICATE-------
       -----BEGIN CERTIFICATE-----
       XXXXXXXXXXXXXXXXXXXXXXXXXXX
       -----END CERTIFICATE-------

2. Mount the secret as a volume on the delegate pod and add a volumeMount to your certificate files at /opt/harness-delegate/ca-bundle.

   In the delegate pod, do the following:

   1. Provide a comma-separated list of paths in the build pod where you want the certs to be mounted.

   2. Mount your certificate files to /opt/harness-delegate/ca-bundle.

   To mount the secret, go to the delegate Deployment YAML and update volumeMounts and volumes. For an example, expand the section below.

   Secrets are retrieved by the delegate. For Kubernetes build infrastructure, secrets are set as Kubernetes secrets and passed to the pod.

   Both CI build pods and the SCM client on the delegate support this method.

   • Custom certificates or self-signed certificates are only supported for Linux nodes in a Kubernetes cluster. For Windows, directly mounting the certificate to the destination path will not work.

   • Make sure the destination path is not same as the default CA certificate path of the corresponding container image.

   • If you override the default certificate file, make sure the Kubernetes secret or config map (from step one) includes all certificates required by the pipelines that will use this build infrastructure.

3. Restart the delegate. Once it is up and running, exec into the container and ensure that the volume exists at the mounted path and contains your certificates.

Enable self-signed certificates with a self-hosted image registry (advanced)

IMPORTANT

This workflow is applicable only if you're using a self-hosted registry to store your container images. For all other workflows, go to Enable self-signed certificates.

1. Create a Kubernetes secret or config map with the required certificates in the same namespace used by the Harness delegate. For example:

   apiVersion: v1
   kind: Secret
   metadata:
     name: addcerts
     namespace: harness-delegate-ng
   type: Opaque
   stringData:
     ca.bundle: |
       -----BEGIN CERTIFICATE-----
       XXXXXXXXXXXXXXXXXXXXXXXXXXX
       -----END CERTIFICATE-------
       -----BEGIN CERTIFICATE-----
       XXXXXXXXXXXXXXXXXXXXXXXXXXX
       -----END CERTIFICATE-------

2. Mount the secret as a volume on the delegate pod.

   In the delegate pod, do the following:

   1. Add DESTINATION_CA_PATH to the environment.

   2. Provide a comma-separated list of paths in the build pod where you want the certs to be mounted.

   3. Mount your certificate files to /opt/harness-delegate/ca-bundle.

   To add DESTINATION_CA_PATH and mount the secret, go to the delegate Deployment YAML and update env, volumeMounts, and volumes. For an example, expand the section below.

   Both CI build pods and the SCM client on the delegate support this method.

   • Make sure the destination path is not same as the default CA certificate path of the corresponding container image.

   • If you override the default certificate file, make sure the Kubernetes secret or config map (from step one) includes all certificates required by the pipelines that will use this build infrastructure.

3. If you're storing your certificates in a local registry and need to run Docker-in-Docker, specify the local certificate path on the delegate.

   For example, if your self-signed certs are stored at https://my-registry.local.org:799 and you log in with docker login my-registry.local.org:799, then you add the path to your DESTINATION_CA_PATH environment like this:

    spec:
      containers:
      - image: harness/delegate:yy.mm.verno
        env:
        - name: DESTINATION_CA_PATH
          value: "/etc/docker/certs.d/my-registry.local.org:799/ca.crt,/etc/ssl/certs/ca-bundle.crt,/kaniko/ssl/certs/additional-ca-cert-bundle.crt"
        volumeMounts:
          - name: certvol
            mountPath: /opt/harness-delegate/ca-bundle/ca.bundle
            subPath:  ca.bundle
      volumes:
      - name: certvol
        secret:
            secretName: addcerts
             items:
             - key: ca.bundle
               path: ca.bundle

4. Restart the delegate. Once it's up and running, exec into the container and ensure that the volume exists at the mounted path and contains your certificates.

Secret Name Creation for Certificate Bundles in Build Pods

When a customer mounts a certificate bundle in their build pods, the bundle is created as a secret and mounted in the pod.

The secret names follow a specific format: Podname-filename-hash.

For Example:- Pod Name: harnessci-buildpod-9rwcbkl6 Secret Name: harnessci-buildpod-9rwcbkl6-custom-cr-pem-0-pem-1667255857 Here, custom-cr-pem is the pem file name and 1667255857 is the random hash that is generated during secret creation.

The name is sanitized to comply with Kubernetes naming conventions. If the total length exceeds 63 characters, it is trimmed from the end to ensure compliance.

NOTE

1. Previously, if the characters exceeded the limit, users had to manually shorten the file or stage names to avoid errors. Now, this change for automatic trimming of secret names beyond 63 characters is available starting from delegate release 821xx.

2. This behavior applies only to Kubernetes architecture.

3. To avoid potential failures in Kubernetes environments, ensure that all secret names are lowercase. Starting from delegate release 825xx, any uppercase characters in the secret name are automatically converted to lowercase to ensure compatibility with Kubernetes naming conventions.

Using Dockerfile builds with self-signed certificates (BuildKit-specific)

When using the Build and Push to Docker step (or any step that performs Dockerfile builds using BuildKit), mounting CA certificates to the Delegate is not enough. BuildKit must be explicitly configured to trust your private registry's certificate.

This workaround is only required for Build and Push to Docker steps using Docker BuildKit. Steps using Kaniko, like most default Kubernetes builds in Harness, do not require this additional configuration if the certificate is already mounted using DESTINATION_CA_PATH.

To make BuildKit trust your self-signed registry certs, you need to:

1. Dynamically prepare the BuildKit config and cert path in a Run step.

2. Share the cert and config path using sharedPaths and environment variables.

3. Concatenate multiple PEM files (if applicable) into a single CA bundle used by both Docker and BuildKit.

Step-by-step example

sharedPaths:
  - /etc/buildkit/certs/your.registry.com

envVariables:
  PLUGIN_BUILDER_CONFIG: /harness/buildkit.toml

Run step to prepare certs and BuildKit config

Here is an example Run step that uses CA certs already mounted into the Delegate and build pod (e.g., via volume mounts to /harness-shared-certs-path), prepares the BuildKit config file, and builds a Dockerfile

- step:
    type: Run
    name: Prepare BuildKit Config
    identifier: buildkit_config
    spec:
      connectorRef: account.harnessImage
      image: alpine
      shell: Sh
      command: |-
        # Registry address (update this to match your use case)
        REG="your.registry.com"
        CERT_PATH_BUILDX="/etc/buildkit/certs/${REG}"
        BUILDKIT_TOML="/harness/buildkit.toml"

        # Create cert paths
        mkdir -p "$CERT_PATH_BUILDX"

        # Concatenate all mounted PEMs into a single CA bundle
        for cert in /harness-shared-certs-path/*.pem; do
          cat "$cert" >> "$CERT_PATH_BUILDX/ca.crt"
          echo >> "$CERT_PATH_BUILDX/ca.crt"
        done

        # Write buildkit.toml
        echo "Writing buildkit.toml..."
        cat <<EOF > "$BUILDKIT_TOML"
        [registry."$REG"]
          ca = ["$CERT_PATH_BUILDX/ca.crt"]
        EOF

        ls -lr "$CERT_PATH_BUILDX"
        cat "$BUILDKIT_TOML"

Build step using the config

- step:
    type: BuildAndPushDockerRegistry
    name: Docker Build
    identifier: docker_build
    spec:
      ...
      envVariables:
        PLUGIN_BUILDER_CONFIG: /harness/buildkit.toml

• Certs should be mounted ahead of time (e.g., through Harness secrets or volume mounts).

• This approach supports multiple PEM files and ensures both Docker and BuildKit trust the self-signed certs.

• You must ensure that the sharedPaths include both cert directories so that they persist across steps.

This ensures that BuildKit inside your pipeline trusts the self-signed certificate used by your private registry.

Troubleshoot Kubernetes cluster build infrastructures

Go to the CI Knowledge Base for questions and issues related to Kubernetes cluster build infrastructures, including use of self-signed certificates, such as:

• SCM service connection issues

• How do I make internal CA certs available to the delegate pod?

• Where should I mount internal CA certs on the build pod?

• Certificate volumes aren't mounted to the build pod