Get Started with Google Cloud BigTable
Configure Harness Database DevOps to manage Google Cloud BigTable schema changes using Liquibase-based change types.
This guide explains how to connect Harness Database DevOps to Google Cloud BigTable and manage schema changes using declarative YAML changelogs. You will create tables, column families, and garbage collection rules through a Liquibase-based extension purpose-built for BigTable's native gRPC API.
Before you begin
Active Harness account with Database DevOps module enabled.
A Google Cloud project with BigTable API enabled and at least one BigTable instance provisioned.
Permissions to manage BigTable tables and column families (
bigtable.tables.create,bigtable.tables.update,bigtable.tables.delete).Permissions to create connectors and Database DevOps pipelines in Harness. Go to RBAC in Harness to configure roles.
BigTable schema concepts
BigTable is a wide-column NoSQL database. Its schema model differs from relational databases:
Table: The top-level container, identified by a table ID within a BigTable instance.
Column family: A logical grouping of columns within a table. All columns within a family share the same garbage collection policy. You define column families at schema design time.
Garbage collection (GC) rules: Policies that automatically delete stale cell versions from a column family. BigTable stores multiple timestamped versions of each cell value. GC rules control how many versions or how long versions are retained.
Unlike SQL databases, BigTable has no CREATE TABLE DDL. Harness Database DevOps handles this through custom Liquibase change types that call the BigTable Admin API directly via gRPC.
OIDC / Workload Identity
When the Harness runner operates on a GCP-managed environment (GKE, Cloud Run, Compute Engine), it can use Application Default Credentials via Workload Identity without a key file. No additional credential configuration is required when Workload Identity is properly configured for the runner's service account.
Connect to BigTable
The BigTable connection URL uses the following format:
Default app profile
bigtable:my-gcp-project/my-instance
Named app profile
bigtable:my-gcp-project/my-instance?app-profile=analytics
To create a BigTable connector in Harness:
In your Harness project, go to Connectors under Project Setup.
Select New Connector, then select Google Cloud BigTable under Cloud Providers.
Enter a Name for the connector (for example,
gcp-bigtable-prod).Enter the Connection URL in the format above.
Under Credentials, select OIDC / Workload Identity. The runner uses Application Default Credentials from the GCP-managed environment.
Select Save and Continue to test the connection.
Supported change types
Harness Database DevOps supports thirteen change types for BigTable. Each maps directly to a BigTable Admin API operation.
createBigtableTable
Creates a new BigTable table. You can define one or more column families and their GC rules in the same changeset.
Properties:
tableName
Yes
The ID of the table to create.
columnFamilies
Yes
List of column family definitions. BigTable requires at least one column family per table.
columnFamilies[].name
Yes
Column family name.
columnFamilies[].gcRule
No
GC rule object. Supports maxNumVersions, maxAge, union, and intersection. Omit to store all versions indefinitely.
Example - Create a table with two column families and a rollback:
Example - Create a table with composite GC rules:
deleteBigtableTable
Deletes a BigTable table and all its data. This operation is irreversible. It is most commonly used as the rollback target for a createBigtableTable changeset.
Properties:
tableName
Yes
The ID of the table to delete.
Example - Delete a table:
createColumnFamily
Adds a new column family to an existing table. Use this change type when you need to extend a table's schema after initial creation.
Properties:
tableName
Yes
The ID of the existing table.
familyName
Yes
The column family name to create.
gcRule
No
GC rule object. Supports maxNumVersions, maxAge, union, and intersection. Omit to retain all versions indefinitely.
Example - Add a column family with a max-age rule and a rollback:
Example - Add a column family with a version-based rule:
Example - Add a column family with a union rule:
Example - Add a column family with an intersection rule:
Example - Add a column family with no GC rule:
deleteColumnFamily
Removes a column family and all its data from a table. This operation is irreversible. Harness Database DevOps tracks this changeset in the DATABASECHANGELOG table so it does not re-execute on subsequent runs.
Properties:
tableName
Yes
The ID of the table containing the column family.
familyName
Yes
The name of the column family to delete.
Example - Remove a deprecated column family:
Example - Remove multiple column families in sequence:
modifyColumnFamilyGCRule
Updates the garbage collection rule on an existing column family. Use this change type to tighten or relax data retention policies without recreating the column family.
Properties:
tableName
Yes
The ID of the table containing the column family.
familyName
Yes
The name of the column family to modify.
gcRule
Yes
The new GC rule object to apply. Supports maxNumVersions, maxAge, union, and intersection.
Example - Reduce retention to one version per cell:
Example - Extend retention to 1 year:
Example - Apply an intersection rule to reduce storage costs:
dropBigTableRows
Deletes rows from a BigTable table that match a row key prefix or range. Use this change type to purge data as part of a schema migration, for example when retiring a row key scheme before switching to a new one.
Properties:
tableName
Yes
The ID of the table to delete rows from.
rowKeyPrefix
No
Delete all rows whose row key starts with this string.
startRowKey
No
Delete rows with row keys greater than or equal to this value. Used together with endRowKey to define a range.
endRowKey
No
Delete rows with row keys less than this value (exclusive). Used together with startRowKey.
Example - Delete all rows with a given prefix:
Example - Delete rows in a key range:
modifyChangeStreamConfig
Enables or updates the change stream on a BigTable table. A change stream captures row mutations in real time and makes them available to consumers such as Dataflow pipelines. Use this change type to set the retention period for captured changes.
Properties:
tableName
Yes
The ID of the table to configure the change stream on.
changeStreamConfig.retentionPeriod
Yes
How long change stream data is retained. Accepts a duration string with a unit suffix: d (days), h (hours), m (minutes), s (seconds). Maximum is 7d.
Example - Enable a change stream with 3-day retention:
Example - Extend change stream retention:
createBigtableMaterializedView
Creates a materialized view on a BigTable table. A materialized view pre-computes and stores the results of a SQL query, enabling fast reads against aggregated or projected data without scanning the full table on every request.
Properties:
name
Yes
A unique name for the materialized view.
query
Yes
A SQL query string that defines the view's contents.
deletionProtection
No
When true, prevents the view from being deleted. Defaults to false.
Example - Create a materialized view with a rollback:
Example - Create a materialized view with deletion protection:
deleteBigtableMaterializedView
Removes a materialized view. This does not affect the underlying table or its data.
Properties:
name
Yes
The name of the materialized view to delete.
Example - Delete a materialized view:
modifyBigtableMaterializedView
Updates an existing materialized view. Use this change type to replace the SQL query or toggle deletion protection.
Properties:
name
Yes
The name of the materialized view to modify.
query
Yes
The SQL query string that defines the view.
deletionProtection
No
Set to true to enable or false to disable deletion protection.
Example - Update the query on an existing materialized view:
Example - Disable deletion protection before removing a materialized view:
createBigtableLogicalView
Creates a logical view over a BigTable table. A logical view defines a SQL query that selects specific columns visible to applications, enabling column-level access control and query isolation without duplicating data.
Properties:
name
Yes
A unique name for the logical view.
query
Yes
A SQL query string that defines the view.
deletionProtection
No
When true, prevents the view from being deleted. Defaults to false.
Example - Create a view over selected columns and a rollback:
Example - Create a view with deletion protection enabled:
deleteBigtableLogicalView
Removes a logical view. This does not affect the underlying table or its data.
Properties:
name
Yes
The name of the logical view to delete.
Example - Delete a logical view:
modifyBigtableLogicalView
Updates an existing logical view. Use this change type to replace the SQL query or toggle deletion protection.
Properties:
name
Yes
The name of the logical view to modify.
query
Yes
The SQL query string that defines the view.
deletionProtection
No
Set to true to enable or false to disable deletion protection.
Example - Update the query on an existing view:
Example - Disable deletion protection before removing a view:
Create a changelog file
Create a YAML changelog file in your repository. A single changelog can contain multiple changesets targeting the same or different tables.
Example - Full schema setup for a new instance:
Last updated
Was this helpful?