Configure a Kubernetes build farm to use self-signed certificates

CI build infrastructure pods can interact with servers using self-signed certificates. This option is useful for organizations that prefer to use internal certificates instead of certificates generated by a public Certificate Authority (CA).

info

This topic assumes that you are familiar with how to implement SSL in Kubernetes. General information about implementing SSL is outside the scope of this topic.
With a Kubernetes cluster build infrastructure, all Build and Push steps use kaniko. Kaniko reads certificates from /kaniko/ssl/certs/additional-ca-cert-bundle.crt.
Harness uses a UBI image for the Git Clone step. UBI reads certificates from /etc/ssl/certs/ca-bundle.crt.
Different base images use different paths as their default certificate location. For example, Alpine images use the path /etc/ssl/certs/ca-certificates.crt. For other images, make sure you verify the default certificate path.

Enable self-signed certificates

Create a Kubernetes secret or config map with the required certificates in the same namespace used by the Harness Delegate. For example:

apiVersion: v1
kind: Secret
metadata:
  name: addcerts
  namespace: harness-delegate-ng
type: Opaque
stringData:
  ca.bundle: |
    -----BEGIN CERTIFICATE-----
    XXXXXXXXXXXXXXXXXXXXXXXXXXX
    -----END CERTIFICATE-------
    -----BEGIN CERTIFICATE-----
    XXXXXXXXXXXXXXXXXXXXXXXXXXX
    -----END CERTIFICATE-------

Mount the secret as a volume on the delegate pod and add a volumeMount to your certificate files at /opt/harness-delegate/ca-bundle.

In the delegate pod, do the following:

Provide a comma-separated list of paths in the build pod where you want the certs to be mounted.
Mount your certificate files to /opt/harness-delegate/ca-bundle.

To mount the secret, go to the delegate Deployment YAML and update volumeMounts and volumes. For an example, expand the section below.

info

Secrets are retrieved by the delegate. For Kubernetes build infrastructure, secrets are set as Kubernetes secrets and passed to the pod.

YAML example: Mount the SSL certificate

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     labels:
       harness.io/name: abc-ca-kubernetes-delegate
     name: abc-ca-kubernetes-delegate
     namespace: abc-ca-kubernetes-delegate
   spec:
     replicas: 1
     minReadySeconds: 120
     selector:
       matchLabels:
         harness.io/name: abc-ca-kubernetes-delegate
     template:
       metadata:
         labels:
           harness.io/name: abc-ca-kubernetes-delegate
         annotations:
           # ...
       spec:
         terminationGracePeriodSeconds: 600
         restartPolicy: Always
         containers:
         - image: harness/delegate:yy.mm.verno
           imagePullPolicy: Always
           name: delegate
           securityContext:
             # ...
           ports:
             # ...
           resources:
             # ...
           livenessProbe:
             # ...
           startupProbe:
             # ...
           envFrom:
             # ...
           env:
             # ...
           volumeMounts:
           ### ADD THE VOLUME MOUNT HERE
           - name: certvol
             mountPath: /opt/harness-delegate/ca-bundle/ca.bundle
             subPath: ca.bundle
         volumes:
         ### ADD THE VOLUME HERE
         - name: certvol
           secret:
             secretName: addcerts
             items:
               - key: ca.bundle
                 path: ca.bundle

Both CI build pods and the SCM client on the delegate support this method.

warning

Make sure the destination path is not same as the default CA certificate path of the corresponding container image.
If you override the default certificate file, make sure the Kubernetes secret or config map (from step one) includes all certificates required by the pipelines that will use this build infrastructure.

Restart the delegate. Once it is up and running, exec into the container and ensure that the volume exists at the mounted path and contains your certificates.

Enable self-signed certificates with a self-hosted image registry (advanced)

important

This workflow is applicable only if you're using a self-hosted registry to store your container images. For all other workflows, go to Enable self-signed certificates.

Create a Kubernetes secret or config map with the required certificates in the same namespace used by the Harness delegate. For example:

apiVersion: v1
kind: Secret
metadata:
  name: addcerts
  namespace: harness-delegate-ng
type: Opaque
stringData:
  ca.bundle: |
    -----BEGIN CERTIFICATE-----
    XXXXXXXXXXXXXXXXXXXXXXXXXXX
    -----END CERTIFICATE-------
    -----BEGIN CERTIFICATE-----
    XXXXXXXXXXXXXXXXXXXXXXXXXXX
    -----END CERTIFICATE-------

Mount the secret as a volume on the delegate pod.

In the delegate pod, do the following:

Add DESTINATION_CA_PATH to the environment.
Provide a comma-separated list of paths in the build pod where you want the certs to be mounted.
Mount your certificate files to /opt/harness-delegate/ca-bundle.

To add DESTINATION_CA_PATH and mount the secret, go to the delegate Deployment YAML and update env, volumeMounts, and volumes. For an example, expand the section below.

YAML example: Add DESTINATION_CA_PATH and mount the SSL certificate

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     labels:
       harness.io/name: abc-ca-kubernetes-delegate
     name: abc-ca-kubernetes-delegate
     namespace: abc-ca-kubernetes-delegate
   spec:
     replicas: 1
     minReadySeconds: 120
     selector:
       matchLabels:
         harness.io/name: abc-ca-kubernetes-delegate
     template:
       metadata:
         labels:
           harness.io/name: abc-ca-kubernetes-delegate
         annotations:
           # ...
       spec:
         terminationGracePeriodSeconds: 600
         restartPolicy: Always
         containers:
         - image: harness/delegate:yy.mm.verno
           imagePullPolicy: Always
           name: delegate
           securityContext:
             # ...
           ports:
             # ...
           resources:
             # ...
           livenessProbe:
             # ...
           startupProbe:
             # ...
           envFrom:
             # ...
           env:
           ### ADD DESTINATION_CA_PATH HERE
           - name: DESTINATION_CA_PATH
             value: "/etc/ssl/certs/ca-bundle.crt,/kaniko/ssl/certs/additional-ca-cert-bundle.crt"
           volumeMounts:
           ### ADD THE VOLUME MOUNT HERE
           - name: certvol
             mountPath: /opt/harness-delegate/ca-bundle/ca.bundle
             subPath: ca.bundle
         volumes:
         ### ADD THE VOLUME HERE
         - name: certvol
           secret:
             secretName: addcerts
             items:
               - key: ca.bundle
                 path: ca.bundle

Both CI build pods and the SCM client on the delegate support this method.

warning

Make sure the destination path is not same as the default CA certificate path of the corresponding container image.
If you override the default certificate file, make sure the Kubernetes secret or config map (from step one) includes all certificates required by the pipelines that will use this build infrastructure.

Legacy: CI_MOUNT_VOLUMES

Prior to the introduction of DESTINATION_CA_PATH, you used ADDITIONAL_CERTS_PATH and CI_MOUNT_VOLUMES to mount certs.

The legacy method is still supported, but Harness recommends DESTINATION_CA_PATH. If you include both, DESTINATION_CA_PATH takes precedence. If Harness can't resolve DESTINATION_CA_PATH, it falls back to CI_MOUNT_VOLUMES and ADDITIONAL_CERTS_PATH.

You must specify both ADDITIONAL_CERTS_PATH and CI_MOUNT_VOLUMES.

For ADDITIONAL_CERTS_PATH, provide the path to the certificates in the delegate, such as /tmp/ca.bundle.

For CI_MOUNT_VOLUMES, provide a comma-separated list of source:destination mappings where source is the certificate path on the delegate, and destination is the path where you want to expose the certificates on the build containers. For example:

/tmp/ca.bundle:/etc/ssl/certs/ca-bundle.crt,/tmp/ca.bundle:/kaniko/ssl/certs/additional-ca-cert-bundle.crt

The CI_MOUNT_VOLUMES list must include all certificates that your build containers need to interact with external services.

      spec:
        containers:
          - image: harness/delegate:yy.mm.verno
            # ...
            env:
            # ADD ADDITIONAL_CERTS_PATH AND CI_MOUNT_VOLUMES HERE
              - name: ADDITIONAL_CERTS_PATH
                value: /tmp/ca.bundle
              - name: CI_MOUNT_VOLUMES
                value: "/tmp/ca.bundle:/etc/ssl/certs/ca-bundle.crt,/tmp/ca.bundle:/kaniko/ssl/certs/additional-ca-cert-bundle.crt"
            volumeMounts:
            # ADD VOLUME MOUNT HERE
              - name: certvol
                mountPath: /tmp/ca.bundle
                subPath:  ca.bundle
        volumes:
          # ADD VOLUME HERE
          - name: certvol
            secret:
              secretName: addcerts
              items:
              - key: ca.bundle
                path: ca.bundle

If you're storing your certificates in a local registry and need to run Docker-in-Docker, specify the local certificate path on the delegate.

For example, if your self-signed certs are stored at https://my-registry.local.org:799 and you log in with docker login my-registry.local.org:799, then you add the path to your DESTINATION_CA_PATH environment like this:

 spec:
   containers:
   - image: harness/delegate:yy.mm.verno
     env:
     - name: DESTINATION_CA_PATH
       value: "/etc/docker/certs.d/my-registry.local.org:799/ca.crt,/etc/ssl/certs/ca-bundle.crt,/kaniko/ssl/certs/additional-ca-cert-bundle.crt"
     volumeMounts:
       - name: certvol
         mountPath: /opt/harness-delegate/ca-bundle/ca.bundle
         subPath:  ca.bundle
   volumes:
   - name: certvol
     secret:
         secretName: addcerts
          items:
          - key: ca.bundle
            path: ca.bundle

Restart the delegate. Once it's up and running, exec into the container and ensure that the volume exists at the mounted path and contains your certificates.

Secret Name Creation for Certificate Bundles in Build Pods

When a customer mounts a certificate bundle in their build pods, the bundle is created as a secret and mounted in the pod.

The secret names follow a specific format: Podname-filename-hash.

For Exampple:- Pod Name: harnessci-buildpod-9rwcbkl6 Secret Name: harnessci-buildpod-9rwcbkl6-custom-cr-pem-0-pem-1667255857 Here, custom-cr-pem is the pem file name and 1667255857 is the random hash that is generated during secret creation.

The name is sanitized to comply with Kubernetes naming conventions. If the total length exceeds 63 characters, it is trimmed from the end to ensure compliance.

note

Previously, if the characters exceeded the limit, users had to manually shorten the file or stage names to avoid errors. Now, this change for automatic trimming of secret names beyond 63 characters is available starting from delegate release 821xx.
This behavior applies only to Kubernetes architecture.
To avoid potential failures in Kubernetes environments, ensure that all secret names are lowercase. Starting from delegate release 825xx, any uppercase characters in the secret name are automatically converted to lowercase to ensure compatibility with Kubernetes naming conventions.

Troubleshoot Kubernetes cluster build infrastructures

Go to the CI Knowledge Base for questions and issues related to Kubernetes cluster build infrastructures, including use of self-signed certificates, such as:

Enable self-signed certificates​

Enable self-signed certificates with a self-hosted image registry (advanced)​

Secret Name Creation for Certificate Bundles in Build Pods​

Troubleshoot Kubernetes cluster build infrastructures​

Enable self-signed certificates

Enable self-signed certificates with a self-hosted image registry (advanced)

Secret Name Creation for Certificate Bundles in Build Pods

Troubleshoot Kubernetes cluster build infrastructures