Run multiple PostgreSQL instances in Background steps

Use Background steps to run multiple PostgreSQL instances.

This topic explains how to use Background steps to run multiple PostgreSQL instances in a Harness CI pipeline.

You need:

• A CI pipeline with a Build stage.

• An understanding of PostgreSQL and the psql command.

Add PostgreSQL Background steps

Add one Background step for each PostgreSQL service you need to run.

For the Background steps to run the PostgreSQL services, the build environment must have the necessary binaries. Depending on the stage's build infrastructure, Background steps can use binaries that exist in the build environment or pull an image, such as a public or private Docker image, that contains the required binaries. For more information about when and how to specify images, go to the Background step Container Registry and Image settings.

Visual

1. Go to the Build stage in the pipeline where you want to add the Background steps.

2. On the Execution tab, select Add Step, and select the Background step from the Step Library.

3. Configure the Background step settings. Required settings depend on your build infrastructure.

Harness Cloud

• Enter a Name and note the Id. You need the Id to call the service later in your pipeline.

• Add the following Environment Variables:

  • POSTGRES_USER: Postgres database user name.

  • POSTGRES_DB: Name for the database.

  • POSTGRES_PASSWORD: Password for the specified user/database. You can use an expression to reference a Harness text secret.

• Add a Port Binding, and then set the Host Post to 5432 and the Container Port to the desired port for this database instance, such as 5433.

• Configure other settings, such as Command, Entry Point, Container Registry, and Image, if needed.

Self-managed Kubernetes cluster

• Enter a Name and note the Id. You need the Id to call the service in later steps in this stage.

• For Container Registry and Image, select a Docker connector and specify a PostgreSQL Docker image.

• Add the following Entry Point arguments, and replace PORT_NUMBER with the desired port binding for this database instance, such as 5433.

  • docker-entrypoint.sh

  • -p PORT_NUMBER

• Add the following Environment Variables:

  • POSTGRES_USER: Postgres database user name.

  • POSTGRES_DB: Name for the database.

  • POSTGRES_PASSWORD: Password for the specified user/database. You can use an expression to reference a Harness text secret.

• Configure other settings, if needed.

1. Select Apply Changes to save the step.

2. Add additional Background steps for each PostgreSQL instance you want to run. Make sure each instance has a different port binding. For example, if the first Background step was 5433, the second one could be 5434.

3. Arrange the Background steps to run in parallel.

YAML

In Harness, go to the pipeline where you want to run the PostgreSQL services. In the CI stage, add one Background step for each PostgreSQL database, and group the steps to run in parallel. Required Background step settings depend on your build infrastructure.

Harness Cloud

            steps:
              - parallel: ## Group the Background steps to run in parallel so all PostgreSQL services run simultaneously.
                  - step:
                      identifier: Background_1 ## You use the identifier to call background services in later steps in this stage.
                      type: Background
                      name: Background_1
                      spec:
                        shell: Sh
                        envVariables: ## Add environment variables for the PostgreSQL user name, database name, and password.
                          POSTGRES_USER: postgres
                          POSTGRES_DB: test
                          POSTGRES_PASSWORD: password ## For security, use an expression to reference a Harness text secret.
                        portBindings:
                          "5432": "5433" ## Specify the host port (5432) and the desired container port (such as 5433). Each Background step must have a different container port.
                  - step:
                      identifier: Background_2
                      type: Background
                      name: Background_2
                      spec:
                        shell: Sh
                        envVariables:
                          POSTGRES_USER: postgres
                          POSTGRES_DB: test1
                          POSTGRES_PASSWORD: password
                        portBindings: ## Each Background step must have a different container port.
                          "5432": "5434"

Self-managed Kubernetes cluster

            steps:
              - parallel: ## Group the Background steps to run in parallel so all PostgreSQL services run simultaneously.
                  - step:
                      identifier: Background_1
                      type: Background
                      name: Background_1
                      spec:
                        connectorRef: YOUR_IMAGE_REGISTRY_CONNECTOR 
                        image: postgres ## Specify a PostgreSQL Docker image.
                        shell: Sh
                        entrypoint: ## Include both the image ENTRYPOINT and the additional port binding argument.
                          - docker-entrypoint.sh
                          - "-p 5433" ## Specify the desired container port. Each Background step must have a different container port.
                        envVariables: ## Add environment variables for the PostgreSQL user name, database name, and password.
                          POSTGRES_USER: postgres
                          POSTGRES_DB: test
                          POSTGRES_PASSWORD: password ## For security, use an expression to reference a Harness text secret.
                  - step:
                      identifier: Background_2
                      type: Background
                      name: Background_2
                      spec:
                        connectorRef: YOUR_IMAGE_REGISTRY_CONNECTOR
                        image: postgres
                        shell: Sh
                        entrypoint:
                          - docker-entrypoint.sh
                          - "-p 5434" ## Each Background step must have a different container port.
                        envVariables:
                          POSTGRES_USER: postgres
                          POSTGRES_DB: test1
                          POSTGRES_PASSWORD: password

For the POSTGRES_PASSWORD, you can use an expression to reference a Harness text secret.

Test the PostgreSQL services

You can add a Run step to confirm your PostgreSQL services are running before running other steps that need to interact with those services.

For the Run step to run psql commands, the build environment must have the necessary binaries. Depending on the stage's build infrastructure, Run steps can use binaries that exist in the build environment or pull an image, such as a public or private Docker image, that contains the required binaries. For more information about when and how to specify images, go to the Run step Container Registry and Image settings.

Visual

1. In the same Build stage as your Background steps, add a Run step after (not in parallel with) your Background steps.

2. Configure the Run step settings. Required settings depend on your build infrastructure.

Harness Cloud

• Enter a Name.

• For Shell, select Sh.

• Enter the following Commands and replace the placeholders with corresponding values from your Background steps.

  sleep 15
  psql -U POSTGRES_USER -d FIRST_DATABASE_NAME -h FIRST_BACKGROUND_STEP_ID -p FIRST_BACKGROUND_STEP_CONTAINER_PORT
  psql -U POSTGRES_USER -d SECOND_DATABASE_NAME -h SECOND_BACKGROUND_STEP_ID -p SECOND_BACKGROUND_STEP_CONTAINER_PORT

  For example:

  sleep 15
  psql -U postgres -d test -h Background_1 -p 5433
  psql -U postgres -d test1 -h Background_2 -p 5434

• Configure other settings, if needed.

Self-managed Kubernetes cluster

• Enter a Name.

• For Container Registry and Image, select a Docker connector and specify a PostgreSQL Docker image.

• For Shell, select Sh.

• Enter the following Commands and replace the placeholders with corresponding values from your Background steps.

  sleep 15
  psql -U POSTGRES_USER -d FIRST_DATABASE_NAME -h localhost -p FIRST_BACKGROUND_STEP_CONTAINER_PORT
  psql -U POSTGRES_USER -d SECOND_DATABASE_NAME -h localhost -p SECOND_BACKGROUND_STEP_CONTAINER_PORT

  For example:

  sleep 15
  psql -U postgres -d test -h localhost -p 5433
  psql -U postgres -d test1 -h localhost -p 5434

• Configure other settings, if needed.

1. Select Apply Changes to save the step.

2. Run your pipeline to test your PostgreSQL services. You can monitor and review build logs on the Build details page. Once you've confirmed that the services run as expected, you can configure the rest of your pipeline. Note that Background steps don't persist across stages.

YAML

In the same CI stage where you added the Background steps, add a Run step after the Background steps. Make sure the Run step isn't in the -parallel group. Required Run step settings depend on your build infrastructure.

Harness Cloud

              - step:
                  identifier: Run_1
                  type: Run
                  name: Run_1
                  spec:
                    shell: Sh
                    command: |- ## Provide the user name, database name, Background step identifier, and container port from each of your Background steps.
                      sleep 15
                      psql -U postgres -d test -h Background_1 -p 5433
                      psql -U postgres -d test1 -h Background_2 -p 5434

In command, you must provide values from your Background steps:

sleep 15
psql -U POSTGRES_USER -d FIRST_DATABASE_NAME -h FIRST_BACKGROUND_STEP_ID -p FIRST_BACKGROUND_STEP_CONTAINER_PORT
psql -U POSTGRES_USER -d SECOND_DATABASE_NAME -h SECOND_BACKGROUND_STEP_ID -p SECOND_BACKGROUND_STEP_CONTAINER_PORT

Self-managed Kubernetes cluster

              - step:
                  identifier: Run_1
                  type: Run
                  name: Run_1
                  spec:
                    connectorRef: YOUR_IMAGE_REGISTRY_CONNECTOR 
                    image: postgres:9-alpine ## Specify a PostgreSQL Docker image.
                    shell: Sh
                    command: |- ## Provide the user name, database name, and container port from each of your Background steps.
                      sleep 15
                      psql -U postgres -d test -h localhost -p 5433
                      psql -U postgres -d test1 -h localhost -p 5434

In command, you must provide values from your Background steps:

sleep 15
psql -U POSTGRES_USER -d FIRST_DATABASE_NAME -h localhost -p FIRST_BACKGROUND_STEP_CONTAINER_PORT
psql -U POSTGRES_USER -d SECOND_DATABASE_NAME -h localhost -p SECOND_BACKGROUND_STEP_CONTAINER_PORT

For the POSTGRES_PASSWORD, you can use an expression to reference a Harness text secret.

After adding the Runstep, run your pipeline to test your PostgreSQL services. You can monitor and review build logs on the Build details page. Once you've confirmed that the services run as expected, you can configure the rest of your pipeline. Note that Background steps don't persist across stages.

Pipeline YAML examples

Harness Cloud

pipeline:
  name: default
  identifier: default
  projectIdentifier: default
  orgIdentifier: default
  tags: {}
  stages:
    - stage:
        identifier: stage
        type: CI
        name: stage
        description: ""
        spec:
          cloneCodebase: false
          execution:
            steps:
              - parallel: ## Group the Background steps to run in parallel so all PostgreSQL services run simultaneously.
                  - step:
                      identifier: Background_1 ## You use the identifier to call background services in later steps in this stage.
                      type: Background
                      name: Background_1
                      spec:
                        shell: Sh
                        envVariables: ## Add environment variables for the PostgreSQL user name, database name, and password.
                          POSTGRES_USER: postgres
                          POSTGRES_DB: test
                          POSTGRES_PASSWORD: password ## For security, use an expression to reference a Harness text secret.
                        portBindings:
                          "5432": "5433" ## Specify the host port (5432) and the desired container port (such as 5433). Each Background step must have a different container port.
                  - step:
                      identifier: Background_2
                      type: Background
                      name: Background_2
                      spec:
                        shell: Sh
                        envVariables:
                          POSTGRES_USER: postgres
                          POSTGRES_DB: test1
                          POSTGRES_PASSWORD: password
                        portBindings: ## Each Background step must have a different container port.
                          "5432": "5434"
              - step:
                  identifier: Run_1
                  type: Run
                  name: Run_1
                  spec:
                    shell: Sh
                    command: |- ## Provide the user name, database name, Background step identifier, and container port from each of your Background steps.
                      sleep 15
                      psql -U postgres -d test -h Background_1 -p 5433
                      psql -U postgres -d test1 -h Background_2 -p 5434
          platform:
            os: Linux
            arch: Amd64
          runtime:
            type: Cloud
            spec: {}

self-managed Kubernetes cluster

pipeline:
  name: default
  identifier: default
  projectIdentifier: default
  orgIdentifier: default
  tags: {}
  stages:
    - stage:
        identifier: run_postgres
        type: CI
        name: run postgres
        description: ""
        spec:
          cloneCodebase: false
          infrastructure:
            type: KubernetesDirect
            spec:
              connectorRef: YOUR_KUBERNETES_CLUSTER_CONNECTOR_ID
              namespace: YOUR_KUBERNETES_NAMESPACE
              automountServiceAccountToken: true
              nodeSelector: {}
              os: Linux
          execution:
            steps:
              - parallel: ## Group the Background steps to run in parallel so all PostgreSQL services run simultaneously.
                  - step:
                      identifier: Background_1
                      type: Background
                      name: Background_1
                      spec:
                        connectorRef: YOUR_IMAGE_REGISTRY_CONNECTOR 
                        image: postgres ## Specify a PostgreSQL Docker image.
                        shell: Sh
                        entrypoint: ## Include both the image ENTRYPOINT and the additional port binding argument.
                          - docker-entrypoint.sh
                          - "-p 5433" ## Specify the desired container port. Each Background step must have a different container port.
                        envVariables: ## Add environment variables for the PostgreSQL user name, database name, and password.
                          POSTGRES_USER: postgres
                          POSTGRES_DB: test
                          POSTGRES_PASSWORD: password ## For security, use an expression to reference a Harness text secret.
                  - step:
                      identifier: Background_2
                      type: Background
                      name: Background_2
                      spec:
                        connectorRef: YOUR_IMAGE_REGISTRY_CONNECTOR
                        image: postgres
                        shell: Sh
                        entrypoint:
                          - docker-entrypoint.sh
                          - "-p 5434" ## Each Background step must have a different container port.
                        envVariables:
                          POSTGRES_USER: postgres
                          POSTGRES_DB: test1
                          POSTGRES_PASSWORD: password
              - step:
                  identifier: Run_1
                  type: Run
                  name: Run_1
                  spec:
                    connectorRef: YOUR_IMAGE_REGISTRY_CONNECTOR 
                    image: postgres:9-alpine ## Specify a PostgreSQL Docker image.
                    shell: Sh
                    command: |- ## Provide the user name, database name, and container port from each of your Background steps.
                      sleep 15
                      psql -U postgres -d test -h localhost -p 5433
                      psql -U postgres -d test1 -h localhost -p 5434

Troubleshooting: Failed to get image entrypoint

If you get a failed to get image entrypoint error when using a Kubernetes cluster build infrastructure, you might need to mount volumes for the PostgreSQL data and then reference those volumes in the Background steps.

1. In the build infrastructure settings, mount one empty directory volume for each PostgreSQL service.

       - stage:
           identifier: run_postgres
           type: CI
           name: run postgres
           description: ""
           spec:
             cloneCodebase: false
             infrastructure:
               type: KubernetesDirect
               spec:
                 connectorRef: YOUR_KUBERNETES_CLUSTER_CONNECTOR_ID
                 namespace: YOUR_KUBERNETES_NAMESPACE
                 volumes:
                   - mountPath: /tmp/pgdata1 ## Empty volume for first PostgreSQL instance.
                     type: EmptyDir
                     spec:
                       medium: ""
                   - mountPath: /tmp/pgdata2 ## Empty volume for second PostgreSQL instance.
                     type: EmptyDir
                     spec:
                       medium: ""
                 automountServiceAccountToken: true
                 nodeSelector: {}
                 os: Linux

2. In each PostgreSQL Background step, add a PGDATA environment variable, and set the value to the corresponding empty directory path.

                     - step:
                         identifier: Background_1
                         type: Background
                         name: Background_1
                         spec:
                           connectorRef: YOUR_IMAGE_REGISTRY_CONNECTOR
                           image: postgres
                           shell: Sh
                           entrypoint:
                             - docker-entrypoint.sh
                             - "-p 5433"
                           envVariables:
                             POSTGRES_USER: postgres
                             POSTGRES_DB: test
                             POSTGRES_PASSWORD: password
                             PGDATA: /tmp/pgdata1 ## Path for first mounted volume.
                     - step:
                         identifier: Background_2
                         type: Background
                         name: Background_2
                         spec:
                           connectorRef: YOUR_IMAGE_REGISTRY_CONNECTOR
                           image: postgres
                           shell: Sh
                           entrypoint:
                             - docker-entrypoint.sh
                             - "-p 5434"
                           envVariables:
                             POSTGRES_USER: postgres
                             POSTGRES_DB: test1
                             POSTGRES_PASSWORD: password
                             PGDATA: /tmp/pgdata2 ## Path for second mounted volume.