Provision Database DevOps
- Provisioning via Terraform
- Provisioning via API
- Bulk Onboarding from CSV
This guide walks you through provisioning Database DevOps using Terraform and the Harness Terraform Provider. The configuration enables seamless schema management, database instance provisioning, and automation of schema changes as part of your CI/CD workflows.
Prerequisites
Before proceeding, ensure you have:
- Terraform v1.3+ installed.
- Access to a Harness account with Database DevOps Licence. Contact Harness Database DevOps Support if you need assistance.
- Properly configured connectors for your databases (e.g., CockroachDB, PostgreSQL).
- A valid Harness Platform API Key.
Step 1: Define a Database Schema
The harness_platform_db_schema resource allows you to define and manage a database schema.
resource "harness_platform_db_schema" "my_schema" {
identifier = "my_db_schema"
org_id = "default"
project_id = "default_project"
name = "My Database Schema"
tags = ["env:prod", "team:backend"]
schema_source {
connector = "cockroachDB" # database connector
location = "changelog.yaml" # schema changelog file
}
}
In the above configuration:
schema_source.connector: Connector type (e.g., CockroachDB, PostgreSQL).schema_source.location: Location of the migration file (e.g., Liquibase/YAML changelog).tags: Useful for environment/team attribution.
This ensures schema migrations are version-controlled and can be applied consistently across environments.
Step 2: Provision a Database Instance
Next, use the harness_platform_db_instance resource to provision a database instance.
resource "harness_platform_db_instance" "my_instance" {
identifier = "my_db_instance"
org_id = "default"
project_id = "default_project"
name = "My Database Instance"
tags = ["env:prod"]
schema = "my_db_schema"
branch = "main"
connector = "pg"
context = "production"
}
In this configuration, you specify:
schema: Binds the instance to the defined schema.branch: Ties schema changes to a Git branch (commonly main).connector: Database connector (PostgreSQL in this example).context: Logical environment (e.g., production).depends_on: Ensures schema creation before instance provisioning.
This setup allows you to manage database instances that automatically track schema changes.
Example Terraform Configuration
Here’s a complete example combining both resources:
terraform {
required_providers {
harness = {
source = "harness/harness"
version = "<version>"
}
}
}
provider "harness" {
endpoint = "https://app.harness.io/gateway"
account_id = "your_account_id"
platform_api_key = "pat_XXXXXXXXXXXXXXXX"
}
resource "harness_platform_db_schema" "my_schema" {
identifier = "my_db_schema"
org_id = "default"
project_id = "default_project"
name = "My Database Schema"
tags = ["env:prod", "team:backend"]
schema_source {
connector = "cockroachDB" # which type of database change tracking / connector type
location = "changelog.yaml" # file / path where schema changes are defined
}
}
resource "harness_platform_db_instance" "my_instance" {
identifier = "my_db_instance"
org_id = "default"
project_id = "default_project"
name = "My Database Instance"
tags = ["env:prod"]
schema = "my_db_schema" # references the above schema
branch = "main" # branch in version control where migrations are tracked
connector = "pg" # the database connector for the instance (PostgreSQL in this case)
context = "production" # logical context / environment label
depends_on = [harness_platform_db_schema.my_schema]
}
Step 3: Apply the Configuration
To apply the configuration:
terraform init
terraform plan
terraform apply
This will provision:
- A database schema
my_db_schemawith versioned migrations. - A database instance
my_db_instancebound to the schema.
If you also want to set up a JDBC connection via Terraform, you can use the harness_platform_connector_jdbc resource.
This guide shows you how to create a database schema and then provision a database instance that tracks changes to the schema. This approach ensures your database objects are versioned and deployed consistently across environments.
Prerequisites
Before you start, ensure you have:
- A valid Harness account and API key. Learn more about managing API keys here.
- Organization (
orgIdentifier) and Project (projectIdentifier) IDs. - A pre-configured DB Connector. You can create one via the Harness API as well or Harness UI. This connector allows Harness to connect to your database (e.g., PostgreSQL, MySQL, CockroachDB).
- An HTTP client such as
curlor Postman.
If you are using Harness API to Create a Database Connector, connector.type should be set to JDBC. Learn more about creating a JDBC connector here.
In the cURL examples below, replace:
<org>with your organization identifier. You can find this in the URL when logged into Harness:https://app.harness.io/ng/org/<org>/...<project>with your project identifier. You can find this in the URL when logged into Harness:https://app.harness.io/ng/org/<org>/projects/<project>/...<account_id>with your Harness account ID. You can find this in the URL when logged into Harness:https://app.harness.io/ng/account/<account_id>/...<your_api_key>with your Harness API key. You can find this under Profile Overview.
Step 1: Create a Database Schema
Use the Harness Database DevOps API to define a schema that your instances will use.
Request
Endpoint:
POST /v1/orgs/{org}/projects/{project}/dbschema
Body Example
{
"changeLogScript": {
"command": "curl -X -o changeLog.yaml https://www.filestore.com/changeLog.yaml",
"image": "plugins/image:latest",
"location": "changelog.yaml",
"shell": "Sh, Bash, <+pipeline.variables.shell>"
},
"changelog": {
"connector": "cockroachDB",
"location": "changelog.yaml",
"repo": "changelog.yaml"
},
"identifier": "string",
"migrationType": "Liquibase",
"name": "string",
"type": "Repository"
}
In the above request, we define a schema with the following parameters:
identifier: Unique identifier for the schema. Must be alphanumeric and can include_or$.name: Friendly display name of the schema.branch: Git branch where schema changes are tracked (e.g.,main).description: (Optional) Description for the schema.connector: The pre-configured database connector to use (e.g.,postgres-connector).tags: Optional key-value pairs to categorize schemas (e.g.,{"env":"prod"}).
Creating a schema first ensures that all instances are properly linked and can track versioned changes via Git.
Example Curl Request
curl -i -X POST \
'https://app.harness.io/v1/orgs/<org>/projects/<project>/dbschema' \
-H 'Content-Type: application/json' \
-H 'Harness-Account: <account_id>' \
-H 'x-api-key: <your_api_key>' \
-d '{
"changeLogScript": {
"command": "curl -X -o changeLog.yaml https://www.filestore.com/changeLog.yaml",
"image": "plugins/image:latest",
"location": "changelog.yaml",
"shell": "Sh, Bash, <+pipeline.variables.shell>"
},
"changelog": {
"connector": "cockroachDB",
"location": "changelog.yaml",
"repo": "changelog.yaml"
},
"identifier": "my_db_schema",
"migrationType": "Liquibase",
"name": "My Database Schema",
"type": "Repository"
}'
Above command creates a new database schema, and you can now use it to provision instances. Below is the response:
{
"changelog": {
"connector": "cockroachDB",
"location": "changelog.yaml",
"repo": "changelog.yaml"
},
"created": 1761120868,
"identifier": "my_db_schema",
"instanceCount": 0,
"migrationType": "Liquibase",
"name": "My Database Schema",
"parentId": "UGHnTeYhRPOv_ttpbEQFKg",
"schemaSourceType": "Git",
"type": "Repository",
"updated": 1761120868
}
The changelog must be provided for Repository type schemas and instanceCount starts at 0.
Step 2: Create a Database Instance
After the schema is ready, let's provision an instance for it.
Request
Endpoint:
POST /v1/orgs/{org}/projects/{project}/dbschema/{dbschema}/instance
Body Example
{
"identifier": "customer_db_instance",
"name": "Customer DB Instance",
"connector": "pg",
"branch": "main",
"context": "dev",
"substituteProperties": { "DB_NAME": "string" },
"tags": { "env": "dev" }
}
In this request, we specify:
identifier: Unique identifier for the database instance.name: Friendly name for the instance.connector: The database connector to use (e.g.,pgfor PostgreSQL).branch: Git branch to track schema changes (e.g.,main).context: Logical environment label (e.g.,dev,prod).substituteProperties: Key-value pairs for any schema property substitutions (e.g., database name).tags: Optional key-value pairs to categorize instances (e.g.,{"env":"dev"}).
Example Curl Request
curl -i -X POST \
'https://app.harness.io/v1/orgs/<org>/projects/<project>/dbschema/<db_schema_identifier>/instance' \
-H 'Content-Type: application/json' \
-H 'Harness-Account: <account_id>' \
-H 'x-api-key: <your_api_key>' \
-d '{
"identifier": "customer_db_instance",
"name": "Customer DB Instance",
"connector": "pg",
"branch": "main",
"context": "dev",
"substituteProperties": { "DB_NAME": "my_db_schema" },
"tags": { "env": "dev" }
}'
When you run the above command, you'll get the following response:
{
"branch": "main",
"connector": "pg",
"context": "dev",
"created": 1761122195,
"identifier": "customer_db_instance",
"lastDeployedChangeSetTag": "",
"name": "Customer DB Instance",
"schemaId": "47e6efdf-ff43-4c79-b319-d761b1c0b91d",
"schemaIdentifier": "my_db_schema1",
"substituteProperties": {
"DB_NAME": "my_db_schema1"
},
"tags": {
"env": "dev"
},
"toOnboard": true,
"updated": 1761122195
}
Things to Note
- Schema first: The changelog is mandatory for Repository-type schemas.
- Instance second: Instances reference an existing schema.
- Branch: Both schema and instance must point to the correct Git branch.
- Connector: Must match the DB type (Postgres, MySQL, etc.).
When you need to onboard hundreds or thousands of database instances, manual resource creation through the UI or individual Terraform configurations becomes impractical. The Database DevOps Bulk Onboarding Terraform module streamlines this process: it provisions JDBC connectors, database instances, and schemas directly from CSV files.
Prerequisites
Before you begin, ensure you have:
- Terraform v1.3 or later installed.
- Harness provider version 0.30.0 or later. Go to Harness Terraform provider documentation to configure the provider.
- A valid Harness Platform API Key.
- Active Database DevOps license. Go to Subscription Overview and Management to view your subscriptions.
- Prepared CSV files containing your database details (see format below).
Step 1: Prepare your CSV files
The module requires two CSV files: one for JDBC connectors and database instances, and another for database schemas.
JDBC connector and instance CSV
This CSV file creates both a JDBC connector and a database instance per row, sharing the same identifier and name.
Create a file named jdbc_connector_and_instance.csv with these columns:
| Column | Required | Description |
|---|---|---|
identifier | yes | Unique identifier for both the connector and instance. |
name | yes | Display name for both resources. |
org_id | yes | Harness organization identifier. |
project_id | yes | Harness project identifier. |
url | yes | JDBC connection URL (for example, jdbc:postgresql://host:5432/db). |
description | no | Description for both connector and instance. |
delegate_selectors | no | Delegate tags, separated by pipes (tag1|tag2). |
tags | no | Tags as comma-separated key:value pairs. |
credentials_json | yes | JSON for the credentials block. Wrap in double quotes and escape internal quotes as "". |
schema_identifier | yes | Must match the identifier of a row in the schema CSV. |
branch | no | Git branch for the changelog. |
context | no | Liquibase context (optional filter). |
substitute_properties_json | no | JSON object for Liquibase property substitution. |
Example row:
identifier,name,org_id,project_id,url,description,delegate_selectors,tags,credentials_json,schema_identifier,branch,context,substitute_properties_json
prod_postgres_001,Production PostgreSQL 001,default,myproject,jdbc:postgresql://prod-db-01.example.com:5432/appdb,Production database instance,prod-delegate,env:prod,"{""type"":""UsernamePassword"",""username"":""dbuser"",""password_ref"":""account.dbpassword""}",app_schema_v1,main,production,{}
Database schema CSV
This CSV file creates database schema definitions that instances reference.
Create a file named db_schemas.csv with these columns:
| Column | Required | Description |
|---|---|---|
identifier | yes | Unique identifier for the schema (referenced by schema_identifier in the connector CSV). |
name | yes | Display name. |
org_id | yes | Harness organization identifier. |
project_id | yes | Harness project identifier. |
description | no | Optional description. |
service | no | Associated service identifier. |
type | no | Schema type: Repository or Script (title case). |
migration_type | no | Migration tool: Liquibase or Flyway. |
use_percona | no | Set to true or false for Percona toolkit usage. |
tags | no | Comma-separated key:value tags. |
schema_source_json | conditional | JSON for repository-based schemas. Includes connector (Git connector ID), location, optional repo, archive_path, toml. |
changelog_script_json | conditional | JSON for script-based schemas. |
Example row for a repository-based schema:
identifier,name,org_id,project_id,description,service,type,migration_type,use_percona,tags,schema_source_json,changelog_script_json
app_schema_v1,Application Schema v1,default,myproject,Main application database schema,,Repository,Liquibase,false,team:backend,"{""connector"":""github_connector"",""location"":""db/changelog.yaml"",""repo"":""my-repo""}",
Step 2: Configure the Terraform module
Create a new directory for your Terraform configuration:
mkdir dbdevops-onboarding
cd dbdevops-onboarding
Create a main.tf file that references the module:
terraform {
required_providers {
harness = {
source = "harness/harness"
version = ">= 0.30.0"
}
}
}
provider "harness" {
endpoint = "https://app.harness.io/gateway"
account_id = var.harness_account_id
platform_api_key = var.harness_api_key
}
module "dbdevops_onboarding" {
source = "git::https://github.com/harness-community/terraform-harness-database-devops-onboarding.git//modules/harness-dbdevops-onboarding"
jdbc_connector_and_instance_csv_file_path = "${path.module}/jdbc_connector_and_instance.csv"
db_schema_csv_file_path = "${path.module}/db_schemas.csv"
}
output "jdbc_instance_count" {
description = "Number of JDBC connectors and instances created"
value = module.dbdevops_onboarding.jdbc_instance_csv_record_count
}
output "schema_count" {
description = "Number of database schemas created"
value = module.dbdevops_onboarding.db_schema_csv_record_count
}
Create a variables.tf file:
variable "harness_account_id" {
description = "Harness account identifier"
type = string
}
variable "harness_api_key" {
description = "Harness platform API key"
type = string
sensitive = true
}
Create a terraform.tfvars file with your credentials:
harness_account_id = "your_account_id"
harness_api_key = "pat.xxxxx.xxxxx.xxxxx"
Place your jdbc_connector_and_instance.csv and db_schemas.csv files in the same directory.
Step 3: Apply the Terraform configuration
Initialize Terraform to download the required providers and module:
terraform init
Review the planned changes:
terraform plan
The output displays how many connectors, instances, and schemas will be created based on your CSV row counts.
Apply the configuration to create all resources:
terraform apply
Type yes when prompted to confirm the creation.
Step 4: Verify the resources
After Terraform completes, verify the resources were created:
- In Harness, select Database DevOps from the module picker.
- Navigate to DB Schemas to see your newly created schemas.
- Navigate to DB Instances to see your database instances.
- Navigate to Connectors under Project Setup to see your JDBC connectors.
Troubleshooting
CSV parsing errors
If you see csvdecode errors during terraform plan:
- Ensure every data row has exactly the same number of fields as the header row.
- Do not add trailing commas at the end of rows.
- Wrap fields containing commas in double quotes.
- Escape double quotes inside quoted fields as
"".
Schema identifier mismatch
If instances fail to create with a message about missing schemas:
- Verify that every
schema_identifiervalue in the JDBC CSV matches anidentifierin the schema CSV. - Check for typos or case mismatches (identifiers are case-sensitive).
Credentials JSON format
The credentials_json column must contain valid JSON. For CSV compatibility:
- Wrap the entire JSON in double quotes.
- Escape internal double quotes as
"".
Example:
"{""type"":""UsernamePassword"",""username"":""dbuser"",""password_ref"":""account.dbpassword""}"
Schema type validation
The type field in the schema CSV must use title case: Repository or Script, not REPOSITORY or repository. The Harness provider validates these values strictly.
Additional resources
- Go to terraform-harness-database-devops-onboarding repository to view the module source code and examples.
- Go to Set up connectors to understand JDBC connector configuration.
- Go to Onboarding guide to learn about the Database DevOps workflow after onboarding.