Provision Database DevOps

Learn how to provision Database DevOps using Terraform and the Harness API. This guide covers schema management, database instance provisioning, and schema change automation.

This guide walks you through provisioning Database DevOps using Terraform and the Harness API. The configuration enables seamless schema management, database instance provisioning, and automation of schema changes as part of your CI/CD workflows.

Provisioning via Terraform

This guide walks you through provisioning Database DevOps using Terraform and the Harness Terraform Provider. The configuration enables seamless schema management, database instance provisioning, and automation of schema changes as part of your CI/CD workflows.

Before you begin

Before proceeding, ensure you have:

• Terraform v1.3+ installed.

• Access to a Harness account with Database DevOps Licence. Contact Harness Database DevOps Support if you need assistance.

• Properly configured connectors for your databases (e.g., CockroachDB, PostgreSQL).

• A valid Harness Platform API Key.

Step 1: Define a database schema

The harness_platform_db_schema resource allows you to define and manage a database schema.

resource "harness_platform_db_schema" "my_schema" {
  identifier = "my_db_schema"        
  org_id     = "default"    
  project_id = "default_project"  
  name       = "My Database Schema" 
  tags       = ["env:prod", "team:backend"]
  schema_source {
    connector = "cockroachDB"    # database connector
    location  = "changelog.yaml" # schema changelog file
  }
}

In the above configuration:

• schema_source.connector: Connector type (e.g., CockroachDB, PostgreSQL).

• schema_source.location: Location of the migration file (e.g., Liquibase/YAML changelog).

• tags: Useful for environment/team attribution.

This ensures schema migrations are version-controlled and can be applied consistently across environments.

Step 2: Provision a database instance

Next, use the harness_platform_db_instance resource to provision a database instance.

resource "harness_platform_db_instance" "my_instance" {
  identifier = "my_db_instance"
  org_id     = "default"
  project_id = "default_project"
  name       = "My Database Instance"
  tags       = ["env:prod"]
  schema    = "my_db_schema"
  branch    = "main"                 
  connector = "pg"     
  context   = "production"           
}

In this configuration, you specify:

• schema: Binds the instance to the defined schema.

• branch: Ties schema changes to a Git branch (commonly main).

• connector: Database connector (PostgreSQL in this example).

• context: Logical environment (e.g., production).

• depends_on: Ensures schema creation before instance provisioning.

This setup allows you to manage database instances that automatically track schema changes.

Example Terraform configuration

The following example combines both resources into a complete, runnable configuration:

terraform {  
  required_providers {  
    harness = {  
      source  = "harness/harness"  
      version = "<version>"  
    }  
  }  
}

provider "harness" {  
  endpoint         = "https://app.harness.io/gateway"  
  account_id       = "your_account_id"  
  platform_api_key = "pat_XXXXXXXXXXXXXXXX"  
}

resource "harness_platform_db_schema" "my_schema" {
  identifier = "my_db_schema"        
  org_id     = "default"    
  project_id = "default_project"  
  name       = "My Database Schema" 
  tags       = ["env:prod", "team:backend"]
  
  schema_source {
    connector = "cockroachDB"           # which type of database change tracking / connector type
    location  = "changelog.yaml"         # file / path where schema changes are defined
  }
}

resource "harness_platform_db_instance" "my_instance" {
  identifier = "my_db_instance"
  org_id     = "default"
  project_id = "default_project"
  name       = "My Database Instance"
  tags       = ["env:prod"]
  
  schema    = "my_db_schema"             # references the above schema
  branch    = "main"                     # branch in version control where migrations are tracked
  connector = "pg"                       # the database connector for the instance (PostgreSQL in this case)
  context   = "production"               # logical context / environment label

  depends_on = [harness_platform_db_schema.my_schema]
}

Step 3: Apply the configuration

To apply the configuration:

terraform init
terraform plan
terraform apply

This will provision:

• A database schema my_db_schema with versioned migrations.

• A database instance my_db_instance bound to the schema.

If you also want to set up a JDBC connection via Terraform, you can use the harness_platform_connector_jdbc resource.

Provisioning via API

This guide shows you how to create a database schema and then provision a database instance that tracks changes to the schema. This approach ensures your database objects are versioned and deployed consistently across environments.

Before you begin

Before you start, ensure you have:

• A valid Harness account and API key. Go to Manage API keys to create an API key.

• Organization (orgIdentifier) and Project (projectIdentifier) IDs.

• A pre-configured DB Connector. You can create one via the Harness API as well or Harness UI. This connector allows Harness to connect to your database (e.g., PostgreSQL, MySQL, CockroachDB).

• An HTTP client such as curl or Postman.

If you are using Harness API to create a database connector, connector.type should be set to JDBC. Go to Set up connectors to configure a JDBC connector.

In the cURL examples below, replace:

• <org> with your organization identifier. You can find this in the URL when logged into Harness: https://app.harness.io/ng/org/<org>/...

• <project> with your project identifier. You can find this in the URL when logged into Harness: https://app.harness.io/ng/org/<org>/projects/<project>/...

• <account_id> with your Harness account ID. You can find this in the URL when logged into Harness: https://app.harness.io/ng/account/<account_id>/...

• <your_api_key> with your Harness API key. You can find this under Profile Overview.

Step 1: Create a database schema

Use the Harness Database DevOps API to define a schema that your instances will use.

Request

Endpoint:

POST /v1/orgs/{org}/projects/{project}/dbschema

Body Example

{
  "changeLogScript": {
    "command": "curl -X -o changeLog.yaml https://www.filestore.com/changeLog.yaml",
    "image": "plugins/image:latest",
    "location": "changelog.yaml",
    "shell": "Sh, Bash, <+pipeline.variables.shell>"
  },
  "changelog": {
    "connector": "cockroachDB",
    "location": "changelog.yaml",
    "repo": "changelog.yaml"
  },
  "identifier": "string",
  "migrationType": "Liquibase",
  "name": "string",
  "type": "Repository"
}

In the above request, we define a schema with the following parameters:

• identifier : Unique identifier for the schema. Must be alphanumeric and can include _ or $.

• name : Friendly display name of the schema.

• branch : Git branch where schema changes are tracked (e.g., main).

• description : (Optional) Description for the schema.

• connector : The pre-configured database connector to use (e.g., postgres-connector).

• tags : Optional key-value pairs to categorize schemas (e.g., {"env":"prod"}).

Creating a schema first ensures that all instances are properly linked and can track versioned changes via Git.

Example curl request

curl -i -X POST \
  'https://app.harness.io/v1/orgs/<org>/projects/<project>/dbschema' \
  -H 'Content-Type: application/json' \
  -H 'Harness-Account: <account_id>' \
  -H 'x-api-key: <your_api_key>' \
  -d '{
    "changeLogScript": {
      "command": "curl -X -o changeLog.yaml https://www.filestore.com/changeLog.yaml",
      "image": "plugins/image:latest",
      "location": "changelog.yaml",
      "shell": "Sh, Bash, <+pipeline.variables.shell>"
    },
    "changelog": {
      "connector": "cockroachDB",
      "location": "changelog.yaml",
      "repo": "changelog.yaml"
    },
    "identifier": "my_db_schema",
    "migrationType": "Liquibase",
    "name": "My Database Schema",
    "type": "Repository"
  }'

Above command creates a new database schema, and you can now use it to provision instances. Below is the response:

{
  "changelog": {
    "connector": "cockroachDB",
    "location": "changelog.yaml",
    "repo": "changelog.yaml"
  },
  "created": 1761120868,
  "identifier": "my_db_schema",
  "instanceCount": 0,
  "migrationType": "Liquibase",
  "name": "My Database Schema",
  "parentId": "UGHnTeYhRPOv_ttpbEQFKg",
  "schemaSourceType": "Git",
  "type": "Repository",
  "updated": 1761120868
}

The changelog must be provided for Repository type schemas and instanceCount starts at 0.

Step 2: Create a database instance

After the schema is ready, let us provision an instance for it.

Request

Endpoint:

POST /v1/orgs/{org}/projects/{project}/dbschema/{dbschema}/instance

Body Example

{
  "identifier": "customer_db_instance",
  "name": "Customer DB Instance",
  "connector": "pg",
  "branch": "main",
  "context": "dev",
  "substituteProperties": { "DB_NAME": "string" },
  "tags": { "env": "dev" }
}

In this request, we specify:

• identifier: Unique identifier for the database instance.

• name: Friendly name for the instance.

• connector: The database connector to use (e.g., pg for PostgreSQL).

• branch: Git branch to track schema changes (e.g., main).

• context: Logical environment label (e.g., dev, prod).

• substituteProperties: Key-value pairs for any schema property substitutions (e.g., database name).

• tags: Optional key-value pairs to categorize instances (e.g., {"env":"dev"}).

Example curl request

curl -i -X POST \
  'https://app.harness.io/v1/orgs/<org>/projects/<project>/dbschema/<db_schema_identifier>/instance' \
  -H 'Content-Type: application/json' \
  -H 'Harness-Account: <account_id>' \
  -H 'x-api-key: <your_api_key>' \
  -d '{
    "identifier": "customer_db_instance",
    "name": "Customer DB Instance",
    "connector": "pg",
    "branch": "main",
    "context": "dev",
    "substituteProperties": { "DB_NAME": "my_db_schema" },
    "tags": { "env": "dev" }
  }'

When you run the above command, you will get the following response:

{
  "branch": "main",
  "connector": "pg",
  "context": "dev",
  "created": 1761122195,
  "identifier": "customer_db_instance",
  "lastDeployedChangeSetTag": "",
  "name": "Customer DB Instance",
  "schemaId": "47e6efdf-ff43-4c79-b319-d761b1c0b91d",
  "schemaIdentifier": "my_db_schema1",
  "substituteProperties": {
    "DB_NAME": "my_db_schema1"
  },
  "tags": {
    "env": "dev"
  },
  "toOnboard": true,
  "updated": 1761122195
}

Things to note

• Schema first: The changelog is mandatory for Repository-type schemas.

• Instance second: Instances reference an existing schema.

• Branch: Both schema and instance must point to the correct Git branch.

• Connector: Must match the DB type (Postgres, MySQL, etc.).

Bulk Onboarding from CSV

When you need to onboard hundreds or thousands of database instances, manual resource creation through the UI or individual Terraform configurations becomes impractical. The Database DevOps Bulk Onboarding Terraform module streamlines this process: it provisions JDBC connectors, database instances, and schemas directly from CSV files.

Before you begin

Before you begin, ensure you have:

• Terraform v1.3 or later installed.

• Harness provider version 0.30.0 or later. Go to Harness Terraform provider documentation to configure the provider.

• A valid Harness Platform API Key.

• Active Database DevOps license. Go to Subscription Overview and Management to view your subscriptions.

• Prepared CSV files containing your database details (see format below).

Step 1: Prepare your CSV files

The module requires two CSV files: one for JDBC connectors and database instances, and another for database schemas.

JDBC connector and instance CSV

This CSV file creates both a JDBC connector and a database instance per row, sharing the same identifier and name.

Create a file named jdbc_connector_and_instance.csv with these columns:

Column	Required	Description
identifier	yes	Unique identifier for both the connector and instance.
name	yes	Display name for both resources.
org_id	yes	Harness organization identifier.
project_id	yes	Harness project identifier.
url	yes	JDBC connection URL (for example, jdbc:postgresql://host:5432/db).
description	no	Description for both connector and instance.
delegate_selectors	no	Delegate tags, separated by pipes (tag1|tag2).
tags	no	Tags as comma-separated key:value pairs.
credentials_json	yes	JSON for the credentials block. Wrap in double quotes and escape internal quotes as "".
schema_identifier	yes	Must match the identifier of a row in the schema CSV.
branch	no	Git branch for the changelog.
context	no	Liquibase context (optional filter).
substitute_properties_json	no	JSON object for Liquibase property substitution or Flyway placeholder replacement.

Example row:

identifier,name,org_id,project_id,url,description,delegate_selectors,tags,credentials_json,schema_identifier,branch,context,substitute_properties_json
prod_postgres_001,Production PostgreSQL 001,default,myproject,jdbc:postgresql://prod-db-01.example.com:5432/appdb,Production database instance,prod-delegate,env:prod,"{""type"":""UsernamePassword"",""username"":""dbuser"",""password_ref"":""account.dbpassword""}",app_schema_v1,main,production,{}

Database schema CSV

This CSV file creates database schema definitions that instances reference.

Create a file named db_schemas.csv with these columns:

Column	Required	Description
identifier	yes	Unique identifier for the schema (referenced by schema_identifier in the connector CSV).
name	yes	Display name.
org_id	yes	Harness organization identifier.
project_id	yes	Harness project identifier.
description	no	Optional description.
service	no	Associated service identifier.
type	no	Schema type: Repository or Script (title case).
migration_type	no	Migration tool: Liquibase or Flyway.
use_percona	no	Set to true or false for Percona toolkit usage.
tags	no	Comma-separated key:value tags.
schema_source_json	conditional	JSON for repository-based schemas. Includes connector (Git connector ID), location, optional repo, archive_path, toml.
changelog_script_json	conditional	JSON for script-based schemas.

Example row for a repository-based schema:

identifier,name,org_id,project_id,description,service,type,migration_type,use_percona,tags,schema_source_json,changelog_script_json
app_schema_v1,Application Schema v1,default,myproject,Main application database schema,,Repository,Liquibase,false,team:backend,"{""connector"":""github_connector"",""location"":""db/changelog.yaml"",""repo"":""my-repo""}",

Step 2: Configure the Terraform module

Create a new directory for your Terraform configuration:

mkdir dbdevops-onboarding
cd dbdevops-onboarding

Create a main.tf file that references the module:

terraform {
  required_providers {
    harness = {
      source  = "harness/harness"
      version = ">= 0.30.0"
    }
  }
}

provider "harness" {
  endpoint         = "https://app.harness.io/gateway"
  account_id       = var.harness_account_id
  platform_api_key = var.harness_api_key
}

module "dbdevops_onboarding" {
  source = "git::https://github.com/harness-community/terraform-harness-database-devops-onboarding.git//modules/harness-dbdevops-onboarding"

  jdbc_connector_and_instance_csv_file_path = "${path.module}/jdbc_connector_and_instance.csv"
  db_schema_csv_file_path                   = "${path.module}/db_schemas.csv"
}

output "jdbc_instance_count" {
  description = "Number of JDBC connectors and instances created"
  value       = module.dbdevops_onboarding.jdbc_instance_csv_record_count
}

output "schema_count" {
  description = "Number of database schemas created"
  value       = module.dbdevops_onboarding.db_schema_csv_record_count
}

Create a variables.tf file:

variable "harness_account_id" {
  description = "Harness account identifier"
  type        = string
}

variable "harness_api_key" {
  description = "Harness platform API key"
  type        = string
  sensitive   = true
}

Create a terraform.tfvars file with your credentials:

harness_account_id = "your_account_id"
harness_api_key    = "pat.xxxxx.xxxxx.xxxxx"

Place your jdbc_connector_and_instance.csv and db_schemas.csv files in the same directory.

Step 3: Apply the Terraform configuration

Initialize Terraform to download the required providers and module:

terraform init

Review the planned changes:

terraform plan

The output displays how many connectors, instances, and schemas will be created based on your CSV row counts.

Apply the configuration to create all resources:

terraform apply

Type yes when prompted to confirm the creation.

Step 4: Verify the resources

After Terraform completes, verify the resources were created:

1. In Harness, select Database DevOps from the module picker.

2. Navigate to DB Schemas to see your newly created schemas.

3. Navigate to DB Instances to see your database instances.

4. Navigate to Connectors under Project Setup to see your JDBC connectors.

Troubleshooting

CSV parsing errors during terraform plan with csvdecode errors in Harness Database DevOps bulk onboarding

Ensure every data row has the same number of fields as the header. Do not add trailing commas. Wrap comma-containing fields in double quotes and escape internal double quotes as double double-quotes.

Database instances fail to create with schema identifier mismatch error during Harness Database DevOps bulk onboarding via Terraform

Verify that every schema_identifier value in the JDBC CSV exactly matches an identifier in the schema CSV. Identifiers are case-sensitive.

Credentials JSON format error in Harness Database DevOps bulk onboarding CSV

Wrap the entire JSON in double quotes and escape internal double quotes as double double-quotes. Example: {""type"":""UsernamePassword"",""username"":""dbuser""}

Schema type validation error in Harness Database DevOps Terraform provider when using bulk onboarding CSV

The type field in the schema CSV must use title case: Repository or Script. Values such as REPOSITORY or repository are not accepted.

Additional resources

The following resources provide further context for the bulk onboarding Terraform module.

• Go to terraform-harness-database-devops-onboarding repository to view the module source code and examples.

• Go to Set up connectors to understand JDBC connector configuration.

• Go to Onboarding guide to learn about the Database DevOps workflow after onboarding.

Next steps

• Go to Create a pipeline in Database DevOps to build the pipeline that runs your migrations.

• Go to Secure database connectivity with SSL to enable encrypted connections for your database.

• Go to Set up connectors to configure JDBC connectors for your database instances.