Diff Changelog

Learn how Harness Database DevOps can create a diff changelog to synchronize database schemas.

Traditional database change workflows rely on manually written scripts. This creates risk:

• Missing dependencies

• Incomplete rollbacks

• Inconsistent naming and standards

Diff-changelog eliminates these risks by deriving changes directly from database state. You can define the desired schema by modifying a real database. The system generates the migration plan automatically.

What is a Diff Changelog?

The diff-changelog is a command that generates a changelog based on the current state of a database. This command becomes exponentially more powerful when it is embedded inside a governed Database DevOps pipeline. Instead of being a one-time developer utility, it becomes an automated change-authoring engine that converts real database state into deployable, policy-compliant migrations.

Prerequisites

Before you begin, ensure you have the following prerequisites in place:

• A Harness account with access to the Database DevOps module.

• Database instances set up and connected to Harness via database connectors.

• A database schema and database instance defined in the Database DevOps module.

How to use diff-changelog in Database DevOps ?

1. Synchronize Development with Git

Before any changes are captured, the development database is brought in sync with Git by running a standard Liquibase apply. This ensures the environment is a trusted baseline.

2. Capture the Authoring Schema

A snapshot of the authoring environment is taken after engineers have completed their schema changes. This snapshot represents the desired database state.

Example:

- step:
    type: LiquibaseCommand
    name: Authoring Schema Snapshot
    identifier: snapshot_authoring
    spec:
      connectorRef: account.harnessImage
      command: snapshot
      resources:
        limits:
          memory: 2Gi
          cpu: "1"
      settings:
        output-file: mySnapshot.json
        snapshot-format: json
        dbSchema: pipeline_authored
        dbInstance: authoring
        excludeChangeLogFile: true
      timeout: 10m
      contextType: Pipeline

3. Generate the Diff Changelog

The pipeline runs diff-changelog using the snapshot as the reference and development as the target. This produces a changelog file containing only the delta between the two states.

Example:

- step:
    type: LiquibaseCommand
    name: Diff as Changelog
    identifier: diff_dev_as_changelog
    spec:
      connectorRef: account.harnessImage
      command: diff-changelog
      resources:
        limits:
          memory: 2Gi
          cpu: "1"
      settings:
        reference-url: offline:mssql?snapshot=mySnapshot.json
        author: <+pipeline.variables.email>
        label-filter: <+pipeline.variables.ticket_id>
        generate-changeset-created-values: "true"
        generated-changeset-ids-contains-description: "true"
        changelog-file: diff.yaml
        dbSchema: pipeline_authored
        dbInstance: development
        excludeChangeLogFile: true
      timeout: 10m
      when:
        stageStatus: Success

The resulting file diff.yaml will contain:

• New columns

• New indexes

• Modified objects

• All required rollback metadata

4. Merge the Diff Changelog into Primary Changelog

The diff file is merged into the primary changelog, which is then deployed to the target database. For which we will use the RunStep. In the Stage, select "Custom" and then create a "Step Group". Then a new add step, Run Step with the following configuration:

• Container Registry: used to pull images from private or public registries.

• Image: "mikefarah/yq:4.45.4"

• Shell: "sh"

• Command: Add the following script under the command palette:

      # Optionally annotate changesets
      yq '.databaseChangeLog.[].changeSet.comment = "<+pipeline.variables.comment>" | .databaseChangeLog.[] |= .changeSet.id = "<+pipeline.variables.ticket_id>-"+(path | .[-1])' diff.yaml > diff-comments.yaml
      
      # Merge new changesets into the main changelog

      yq -i 'load("diff-comments.yaml") as $d2 | .databaseChangeLog += $d2.databaseChangeLog' dbops/ensure_dev_matches_git/changelogs/pipeline-authored/changelog.yml

      # Output the merged changelog (for transparency/logging)
      cat dbops/ensure_dev_matches_git/changelogs/pipeline-authored/changelog.yml

You can commit the generated changelog file to your git repository using the Run Command step in the pipeline. This allows you to version control your changelog file and keep track of changes over time. Otherwise, once the pipeline is executed, pods will be deleted and the changelog file will be lost.

1. In the Pipeline, under the Step Group section, add a new step Run Command as the step type.

• Name: The name of the step.

• Registry Type: The type of registry to use. We can use Third Party Registry or Harness Artifact Registry.

• Container Registry: The container registry to use. This is the location where the image is stored. In this case, we will use Docker Hub as the registry.

• Image: The name of the image to use. In this case, we will use alpine/git.

• Shell: The shell to use. We can use bash or sh, depending on the image used.

• Command: The command to be executed. In this case, we will use following command to commit the changelog file to the git repository:

  git init
  
  # Configure Git user
  git config --global user.email <User Email>
  git config --global user.name <User Name>
  git config --global user.password <PAT Token> ## PAT saved in Harness Secrets Manager
  
  git add generated.yml ## Our changelog file name which we generated in the previous step
  git commit -m "generated changelog from running instance" -s
  
  # Get current branch name
  CURRENT_BRANCH=$(git rev-parse --abbrev-ref HEAD)
  
  # Add remote repository
  git remote add origin <User Email>:<PAT>@<Git Repo URL>.git ## Artifact Registry URL with https:// after @
  
  # Push to remote using the current branch name
  git push -u origin $CURRENT_BRANCH -f

1. Click on Apply Changes. Save the Pipeline and click on the Run button to run the pipeline.