> For the complete documentation index, see [llms.txt](https://developer.harness.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://developer.harness.io/continuous-integration/use-harness-ci/use-harness-ci/manage-dependencies/multiple-postgres.md). # Run multiple PostgreSQL instances in Background steps Use Background steps to run multiple PostgreSQL instances. This topic explains how to use **Background** steps to run multiple PostgreSQL instances in a Harness CI pipeline. You need: * A [CI pipeline](/continuous-integration/use-harness-ci/use-harness-ci/prep-ci-pipeline-components.md) with a [Build stage](/continuous-integration/use-harness-ci/use-harness-ci/set-up-build-infrastructure/ci-stage-settings.md). * An understanding of PostgreSQL and the [psql command](https://www.postgresql.org/docs/current/app-psql.html). ### Add PostgreSQL Background steps Add one **Background** step for each PostgreSQL service you need to run. For the **Background** steps to run the PostgreSQL services, the build environment must have the necessary binaries. Depending on the stage's build infrastructure, **Background** steps can use binaries that exist in the build environment or pull an image, such as a public or private Docker image, that contains the required binaries. For more information about when and how to specify images, go to the [Background step Container Registry and Image settings](/continuous-integration/use-harness-ci/use-harness-ci/manage-dependencies/background-step-settings.md#container-registry-and-image). {% tabs %} {% tab title="Visual" %} 1. Go to the **Build** stage in the pipeline where you want to add the **Background** steps. 2. On the **Execution** tab, select **Add Step**, and select the **Background** step from the Step Library. 3. Configure the [Background step settings](/continuous-integration/use-harness-ci/use-harness-ci/manage-dependencies/background-step-settings.md). Required settings depend on your build infrastructure. {% tabs %} {% tab title="Harness Cloud" %} * Enter a **Name** and note the **Id**. You need the **Id** to call the service later in your pipeline. * Add the following **Environment Variables**: * `POSTGRES_USER`: Postgres database user name. * `POSTGRES_DB`: Name for the database. * `POSTGRES_PASSWORD`: Password for the specified user/database. You can use an expression to [reference a Harness text secret](/harness-platform/use-harness-platform/secrets/add-use-text-secrets.md). * Add a **Port Binding**, and then set the **Host Post** to `5432` and the **Container Port** to the desired port for this database instance, such as `5433`. * Configure [other settings](/continuous-integration/use-harness-ci/use-harness-ci/manage-dependencies/background-step-settings.md), such as **Command**, **Entry Point**, **Container Registry**, and **Image**, if needed. {% endtab %} {% tab title="Self-managed Kubernetes cluster" %} * Enter a **Name** and note the **Id**. You need the **Id** to call the service in later steps in this stage. * For **Container Registry** and **Image**, select a Docker connector and specify a [PostgreSQL Docker image](https://hub.docker.com/_/postgres). * Add the following **Entry Point** arguments, and replace `PORT_NUMBER` with the desired port binding for this database instance, such as `5433`. * `docker-entrypoint.sh` * `-p PORT_NUMBER` * Add the following **Environment Variables**: * `POSTGRES_USER`: Postgres database user name. * `POSTGRES_DB`: Name for the database. * `POSTGRES_PASSWORD`: Password for the specified user/database. You can use an expression to [reference a Harness text secret](/harness-platform/use-harness-platform/secrets/add-use-text-secrets.md). * Configure [other settings](/continuous-integration/use-harness-ci/use-harness-ci/manage-dependencies/background-step-settings.md), if needed. {% endtab %} {% endtabs %} 4. Select **Apply Changes** to save the step. 5. Add additional **Background** steps for each PostgreSQL instance you want to run. Make sure each instance has a different port binding. For example, if the first **Background** step was `5433`, the second one could be `5434`. 6. Arrange the **Background** steps to run in parallel.
{% endtab %} {% tab title="YAML" %} In Harness, go to the pipeline where you want to run the PostgreSQL services. In the `CI` stage, add one `Background` step for each PostgreSQL database, and group the steps to run in parallel. Required [Background step settings](/continuous-integration/use-harness-ci/use-harness-ci/manage-dependencies/background-step-settings.md) depend on your build infrastructure. {% tabs %} {% tab title="Harness Cloud" %} ```yaml steps: - parallel: ## Group the Background steps to run in parallel so all PostgreSQL services run simultaneously. - step: identifier: Background_1 ## You use the identifier to call background services in later steps in this stage. type: Background name: Background_1 spec: shell: Sh envVariables: ## Add environment variables for the PostgreSQL user name, database name, and password. POSTGRES_USER: postgres POSTGRES_DB: test POSTGRES_PASSWORD: password ## For security, use an expression to reference a Harness text secret. portBindings: "5432": "5433" ## Specify the host port (5432) and the desired container port (such as 5433). Each Background step must have a different container port. - step: identifier: Background_2 type: Background name: Background_2 spec: shell: Sh envVariables: POSTGRES_USER: postgres POSTGRES_DB: test1 POSTGRES_PASSWORD: password portBindings: ## Each Background step must have a different container port. "5432": "5434" ``` {% endtab %} {% tab title="Self-managed Kubernetes cluster" %} ```yaml steps: - parallel: ## Group the Background steps to run in parallel so all PostgreSQL services run simultaneously. - step: identifier: Background_1 type: Background name: Background_1 spec: connectorRef: YOUR_IMAGE_REGISTRY_CONNECTOR image: postgres ## Specify a PostgreSQL Docker image. shell: Sh entrypoint: ## Include both the image ENTRYPOINT and the additional port binding argument. - docker-entrypoint.sh - "-p 5433" ## Specify the desired container port. Each Background step must have a different container port. envVariables: ## Add environment variables for the PostgreSQL user name, database name, and password. POSTGRES_USER: postgres POSTGRES_DB: test POSTGRES_PASSWORD: password ## For security, use an expression to reference a Harness text secret. - step: identifier: Background_2 type: Background name: Background_2 spec: connectorRef: YOUR_IMAGE_REGISTRY_CONNECTOR image: postgres shell: Sh entrypoint: - docker-entrypoint.sh - "-p 5434" ## Each Background step must have a different container port. envVariables: POSTGRES_USER: postgres POSTGRES_DB: test1 POSTGRES_PASSWORD: password ``` {% endtab %} {% endtabs %} {% hint style="info" %} For the `POSTGRES_PASSWORD`, you can use an expression to [reference a Harness text secret](/harness-platform/use-harness-platform/secrets/add-use-text-secrets.md). {% endhint %} {% endtab %} {% endtabs %} ### Test the PostgreSQL services You can add a [Run step](/continuous-integration/use-harness-ci/use-harness-ci/run-step-settings.md) to confirm your PostgreSQL services are running before running other steps that need to interact with those services. For the **Run** step to run `psql` commands, the build environment must have the necessary binaries. Depending on the stage's build infrastructure, **Run** steps can use binaries that exist in the build environment or pull an image, such as a public or private Docker image, that contains the required binaries. For more information about when and how to specify images, go to the [Run step Container Registry and Image settings](/continuous-integration/use-harness-ci/use-harness-ci/run-step-settings.md#container-registry-and-image). {% tabs %} {% tab title="Visual" %} 1. In the same **Build** stage as your **Background** steps, add a **Run** step after (*not* in parallel with) your **Background** steps. 2. Configure the [Run step settings](/continuous-integration/use-harness-ci/use-harness-ci/run-step-settings.md#run-step-settings). Required settings depend on your build infrastructure. {% tabs %} {% tab title="Harness Cloud" %} * Enter a **Name**. * For **Shell**, select **Sh**. * Enter the following **Commands** and replace the placeholders with corresponding values from your **Background** steps. ``` sleep 15 psql -U POSTGRES_USER -d FIRST_DATABASE_NAME -h FIRST_BACKGROUND_STEP_ID -p FIRST_BACKGROUND_STEP_CONTAINER_PORT psql -U POSTGRES_USER -d SECOND_DATABASE_NAME -h SECOND_BACKGROUND_STEP_ID -p SECOND_BACKGROUND_STEP_CONTAINER_PORT ``` For example: ``` sleep 15 psql -U postgres -d test -h Background_1 -p 5433 psql -U postgres -d test1 -h Background_2 -p 5434 ``` * Configure [other settings](/continuous-integration/use-harness-ci/use-harness-ci/run-step-settings.md#run-step-settings), if needed. {% endtab %} {% tab title="Self-managed Kubernetes cluster" %} * Enter a **Name**. * For **Container Registry** and **Image**, select a Docker connector and specify a [PostgreSQL Docker image](https://hub.docker.com/_/postgres). * For **Shell**, select **Sh**. * Enter the following **Commands** and replace the placeholders with corresponding values from your **Background** steps. ``` sleep 15 psql -U POSTGRES_USER -d FIRST_DATABASE_NAME -h localhost -p FIRST_BACKGROUND_STEP_CONTAINER_PORT psql -U POSTGRES_USER -d SECOND_DATABASE_NAME -h localhost -p SECOND_BACKGROUND_STEP_CONTAINER_PORT ``` For example: ``` sleep 15 psql -U postgres -d test -h localhost -p 5433 psql -U postgres -d test1 -h localhost -p 5434 ``` * Configure [other settings](/continuous-integration/use-harness-ci/use-harness-ci/run-step-settings.md#run-step-settings), if needed. {% endtab %} {% endtabs %} 3. Select **Apply Changes** to save the step. 4. Run your pipeline to test your PostgreSQL services. You can monitor and review build logs on the [Build details page](/continuous-integration/use-harness-ci/use-harness-ci/viewing-builds.md). Once you've confirmed that the services run as expected, you can configure the rest of your pipeline. Note that **Background** steps don't persist across stages. {% endtab %} {% tab title="YAML" %} In the same `CI` stage where you added the `Background` steps, add a `Run` step after the `Background` steps. Make sure the `Run` step *isn't* in the `-parallel` group. Required [Run step settings](/continuous-integration/use-harness-ci/use-harness-ci/run-step-settings.md) depend on your build infrastructure. {% tabs %} {% tab title="Harness Cloud" %} ```yaml - step: identifier: Run_1 type: Run name: Run_1 spec: shell: Sh command: |- ## Provide the user name, database name, Background step identifier, and container port from each of your Background steps. sleep 15 psql -U postgres -d test -h Background_1 -p 5433 psql -U postgres -d test1 -h Background_2 -p 5434 ``` In `command`, you must provide values from your `Background` steps: ``` sleep 15 psql -U POSTGRES_USER -d FIRST_DATABASE_NAME -h FIRST_BACKGROUND_STEP_ID -p FIRST_BACKGROUND_STEP_CONTAINER_PORT psql -U POSTGRES_USER -d SECOND_DATABASE_NAME -h SECOND_BACKGROUND_STEP_ID -p SECOND_BACKGROUND_STEP_CONTAINER_PORT ``` {% endtab %} {% tab title="Self-managed Kubernetes cluster" %} ```yaml - step: identifier: Run_1 type: Run name: Run_1 spec: connectorRef: YOUR_IMAGE_REGISTRY_CONNECTOR image: postgres:9-alpine ## Specify a PostgreSQL Docker image. shell: Sh command: |- ## Provide the user name, database name, and container port from each of your Background steps. sleep 15 psql -U postgres -d test -h localhost -p 5433 psql -U postgres -d test1 -h localhost -p 5434 ``` In `command`, you must provide values from your `Background` steps: ``` sleep 15 psql -U POSTGRES_USER -d FIRST_DATABASE_NAME -h localhost -p FIRST_BACKGROUND_STEP_CONTAINER_PORT psql -U POSTGRES_USER -d SECOND_DATABASE_NAME -h localhost -p SECOND_BACKGROUND_STEP_CONTAINER_PORT ``` {% endtab %} {% endtabs %} {% hint style="info" %} For the `POSTGRES_PASSWORD`, you can use an expression to [reference a Harness text secret](/harness-platform/use-harness-platform/secrets/add-use-text-secrets.md). {% endhint %} After adding the `Run`step, run your pipeline to test your PostgreSQL services. You can monitor and review build logs on the [Build details page](/continuous-integration/use-harness-ci/use-harness-ci/viewing-builds.md). Once you've confirmed that the services run as expected, you can configure the rest of your pipeline. Note that `Background` steps don't persist across stages. {% endtab %} {% endtabs %} ### Pipeline YAML examples {% tabs %} {% tab title="Harness Cloud" %} ```yaml pipeline: name: default identifier: default projectIdentifier: default orgIdentifier: default tags: {} stages: - stage: identifier: stage type: CI name: stage description: "" spec: cloneCodebase: false execution: steps: - parallel: ## Group the Background steps to run in parallel so all PostgreSQL services run simultaneously. - step: identifier: Background_1 ## You use the identifier to call background services in later steps in this stage. type: Background name: Background_1 spec: shell: Sh envVariables: ## Add environment variables for the PostgreSQL user name, database name, and password. POSTGRES_USER: postgres POSTGRES_DB: test POSTGRES_PASSWORD: password ## For security, use an expression to reference a Harness text secret. portBindings: "5432": "5433" ## Specify the host port (5432) and the desired container port (such as 5433). Each Background step must have a different container port. - step: identifier: Background_2 type: Background name: Background_2 spec: shell: Sh envVariables: POSTGRES_USER: postgres POSTGRES_DB: test1 POSTGRES_PASSWORD: password portBindings: ## Each Background step must have a different container port. "5432": "5434" - step: identifier: Run_1 type: Run name: Run_1 spec: shell: Sh command: |- ## Provide the user name, database name, Background step identifier, and container port from each of your Background steps. sleep 15 psql -U postgres -d test -h Background_1 -p 5433 psql -U postgres -d test1 -h Background_2 -p 5434 platform: os: Linux arch: Amd64 runtime: type: Cloud spec: {} ``` {% endtab %} {% tab title="self-managed Kubernetes cluster" %} ```yaml pipeline: name: default identifier: default projectIdentifier: default orgIdentifier: default tags: {} stages: - stage: identifier: run_postgres type: CI name: run postgres description: "" spec: cloneCodebase: false infrastructure: type: KubernetesDirect spec: connectorRef: YOUR_KUBERNETES_CLUSTER_CONNECTOR_ID namespace: YOUR_KUBERNETES_NAMESPACE automountServiceAccountToken: true nodeSelector: {} os: Linux execution: steps: - parallel: ## Group the Background steps to run in parallel so all PostgreSQL services run simultaneously. - step: identifier: Background_1 type: Background name: Background_1 spec: connectorRef: YOUR_IMAGE_REGISTRY_CONNECTOR image: postgres ## Specify a PostgreSQL Docker image. shell: Sh entrypoint: ## Include both the image ENTRYPOINT and the additional port binding argument. - docker-entrypoint.sh - "-p 5433" ## Specify the desired container port. Each Background step must have a different container port. envVariables: ## Add environment variables for the PostgreSQL user name, database name, and password. POSTGRES_USER: postgres POSTGRES_DB: test POSTGRES_PASSWORD: password ## For security, use an expression to reference a Harness text secret. - step: identifier: Background_2 type: Background name: Background_2 spec: connectorRef: YOUR_IMAGE_REGISTRY_CONNECTOR image: postgres shell: Sh entrypoint: - docker-entrypoint.sh - "-p 5434" ## Each Background step must have a different container port. envVariables: POSTGRES_USER: postgres POSTGRES_DB: test1 POSTGRES_PASSWORD: password - step: identifier: Run_1 type: Run name: Run_1 spec: connectorRef: YOUR_IMAGE_REGISTRY_CONNECTOR image: postgres:9-alpine ## Specify a PostgreSQL Docker image. shell: Sh command: |- ## Provide the user name, database name, and container port from each of your Background steps. sleep 15 psql -U postgres -d test -h localhost -p 5433 psql -U postgres -d test1 -h localhost -p 5434 ``` {% endtab %} {% endtabs %} ### Troubleshooting: Failed to get image entrypoint If you get a `failed to get image entrypoint` error when using a Kubernetes cluster build infrastructure, you might need to [mount volumes](/continuous-integration/use-harness-ci/use-harness-ci/set-up-build-infrastructure/k8s-build-infrastructure/set-up-a-kubernetes-cluster-build-infrastructure.md#volumes) for the PostgreSQL data and then reference those volumes in the **Background** steps. 1. In the build infrastructure settings, mount one empty directory volume for each PostgreSQL service. ```yaml - stage: identifier: run_postgres type: CI name: run postgres description: "" spec: cloneCodebase: false infrastructure: type: KubernetesDirect spec: connectorRef: YOUR_KUBERNETES_CLUSTER_CONNECTOR_ID namespace: YOUR_KUBERNETES_NAMESPACE volumes: - mountPath: /tmp/pgdata1 ## Empty volume for first PostgreSQL instance. type: EmptyDir spec: medium: "" - mountPath: /tmp/pgdata2 ## Empty volume for second PostgreSQL instance. type: EmptyDir spec: medium: "" automountServiceAccountToken: true nodeSelector: {} os: Linux ``` 2. In each PostgreSQL **Background** step, add a `PGDATA` environment variable, and set the value to the corresponding empty directory path. ```yaml - step: identifier: Background_1 type: Background name: Background_1 spec: connectorRef: YOUR_IMAGE_REGISTRY_CONNECTOR image: postgres shell: Sh entrypoint: - docker-entrypoint.sh - "-p 5433" envVariables: POSTGRES_USER: postgres POSTGRES_DB: test POSTGRES_PASSWORD: password PGDATA: /tmp/pgdata1 ## Path for first mounted volume. - step: identifier: Background_2 type: Background name: Background_2 spec: connectorRef: YOUR_IMAGE_REGISTRY_CONNECTOR image: postgres shell: Sh entrypoint: - docker-entrypoint.sh - "-p 5434" envVariables: POSTGRES_USER: postgres POSTGRES_DB: test1 POSTGRES_PASSWORD: password PGDATA: /tmp/pgdata2 ## Path for second mounted volume. ``` {% @harness-feedback/feedback %}