Skip to main content

Harness GitOps ApplicationSet and PR pipeline tutorial

note

Currently, this feature is behind the feature flags ENV_GROUP, NG_SVC_ENV_REDESIGN, and OPTIMIZED_GIT_FETCH_FILES. Contact Harness Support to enable the feature.

This tutorial shows you how to create a GitOps ApplicationSet and PR Pipeline in Harness GitOps.

In this tutorial, we'll demonstrate two major use cases:

  1. We'll create an ApplicationSet defines one application and syncs it to multiple target environments.
  2. We'll create a Harness PR Pipeline to change the application in just one of the target environments.
note

New to ApplicationSets and PR Pipelines? See the ApplicationSet and PR Pipeline Summary below.

Before you begin

This tutorial extends Harness GitOps features covered the following topics:

Review these topics before proceeding.

Requirements

To perform this tutorial you'll need the following:

  • 3 Kubernetes clusters with a minimum of 2 vCPUs, 8GB Memory, 100 GB disk size (for example the e2-standard-2 in GCP):
    • 2 target clusters: the ApplicationSet will deploy an app to these 2 clusters.
      • For this tutorial, we will name the clusters engineering-dev and engineering-prod.
        We will use the Kubernetes default namespace for the applications, but you can use any namespace.
    • 1 cluster for the Harness GitOps Agent. You can install the Agent in a cluster with or without an existing Argo CD project.
  • GitHub account. You will be cloning the Argo Project's ApplicationSet repo and using one of its examples.

Install the GitOps Agent

For steps on how to install a Harness GitOps Agent, go to Install a Harness GitOps Agent.

  1. Install the Agent in the Kubernetes cluster you have set up for your GitOps Agent, not the target dev or prod target clusters.

Ensure the Agent has access to Harness and to the other 2 target clusters. Once it's installed you'll see it register with Harness:

note

Mapped Harness Project: if you installed the Agent in a cluster without an existing Argo CD project, there will not be a mapping initially. Once you create a Harness GitOps entity using the Agent, such as a Cluster or Repo, Harness will automatically create the Argo CD project and map it to the Agent.

Add the GitOps Clusters

Now we'll add 3 GitOps Clusters in Harness:

  • 1 GitOps Cluster for the cluster running your GitOps Agent.
  • 1 GitOps Cluster for the cluster that will host the dev application.
  • 1 GitOps Cluster for the cluster that will host the prod application.

GitOps Agent Cluster

  1. In GitOps, click Settings, and then Clusters.

  2. Click New Cluster.

  3. In Name, enter the name appset-example.

  4. In GitOps Agent, select the Agent you installed earlier.

  5. Click Continue.

  6. In Details, click Use the credentials of a specific Harness GitOps Agent.

  7. Click Save and Continue. The connections is verified.

  8. Click Finish.

Dev and Prod Application Clusters

We'll add Harness GitOps Clusters for the two target clusters where we want to add our application.

  1. Create a new Harness GitOps Cluster for your dev cluster.

  2. Name the cluster engineering-dev.

  3. In GitOps Agent, select the GitOps Agent you added earlier.

  4. In Details, select Specify Kubernetes Cluster URL and credentials.

  5. In Master URL, enter the Endpoint URL for the target cluster (you can use kubectl cluster-info or your cloud console).Ensure that you use the https:// scheme.Here's an example:

  6. In Authentication, use the authentication method you prefer. In this tutorial, we use the default namespace service-account token.

  7. Click Save and Continue. The GitOps Cluster is verified.

  8. Click Finish.

  9. Repeat the process for the Prod cluster.

    1. Name the cluster engineering-prod.
    2. Use the same Agent.
    3. For Master URL, ensure that you use the https:// scheme.
    4. Use whatever authentication method you want.

When you're done, you will have three Harness GitOps Clusters: 1 for the GitOps Agent and two for the target clusters.

You might see a Warning status. This status simply indicates that nothing has been deployed to the cluster yet.

Clone and update the ApplicationSet repo

The repo and example we will use is located in the Argo Project's public ApplicationSet repo:

https://github.com/argoproj/applicationset/tree/master/examples/git-generator-files-discovery

For a summary of this example, go to Argo CD docs.You will need to clone this repo into your GitHub account and then update 3 files.

  1. Clone the repo https://github.com/argoproj/applicationset.
  2. Navigate to applicationset/examples/git-generator-files-discovery/git-generator-files.yaml.
  3. Update git-generator-files.yaml with the following YAML keys/values:
apiVersion: argoproj.io/v1alpha1  
kind: ApplicationSet
metadata:
name: guestbook
spec:
generators:
- git:
repoURL: https://github.com/<your account name>/applicationset.git
revision: HEAD
files:
- path: "examples/git-generator-files-discovery/cluster-config/**/config.json"
template:
metadata:
name: '{{cluster.name}}-guestbook'
spec:
project: <Harness GitOps Agent Project Id>
source:
repoURL: https://github.com/<your account name>/applicationset.git
targetRevision: HEAD
path: "examples/git-generator-files-discovery/apps/guestbook"
destination:
server: '{{cluster.address}}'
namespace: default
syncPolicy:
automated: {}

Make sure you update the following:

  1. Update repoURL: https://github.com/argoproj/applicationset.git with repoURL: https://github.com/<your account name>/applicationset.git.

  2. spec.template.spec.project: replace default with the Harness GitOps Agent Project Id.

  3. spec.template.spec.destination.server: replace server: https://kubernetes.default.svc with server: '{{cluster.address}}'.

  4. Add syncPolicy.automated: {} to spec.template.spec.

  5. Save your changes.

Next, we'll update the config.json files for the two target clusters.

  1. Navigate to applicationset/examples/git-generator-files-discovery/cluster-config/engineering/dev/config.json.
  2. Replace "address": "https://1.2.3.4" with the Endpoint IP address for the dev cluster. Ensure that you use the https:// scheme.
  3. Navigate to applicationset/examples/git-generator-files-discovery/cluster-config/engineering/prod/config.json.
  4. Replace "address": "https://1.2.3.4" with the Endpoint IP address for the prod cluster. Ensure that you use the https:// scheme.

Here's an example. Your IP addresses will be different.

Add the GitOps Repository

Let's add a Harness GitOps Repository for your ApplicationSet repo. Later, you'll select this GitOps Repository when you set up the Harness GitOps Application.

  1. In your Harness Project, click GitOps, and then click Settings.

  2. Click Repositories.

  3. Click New Repository.

  4. In Specify Repository Type, click Git.

  5. Enter the following and click Continue.

    • Repository Name: applicationset_examples.
    • GitOps Agent: select the Agent you added earlier.
    • Git Repository URL: the URL to your repo. You can simply copy and paste the HTTPS URL from GitHub.

    1. In Credentials, either user your GitHub credentials or, if your repo is public, select Anonymous in Authentication.
    2. Click Save & Continue.
  6. Once the connection is verified, click Finish.

Now you can create the Harness GitOps Application using the Harness GitOps Agent, Clusters, and Repositories you have set up.

Create the Harness GitOps Application

Now that we have the Agent, Clusters, and Repo, we can create the GitOps Application.

  1. In GitOps, click Applications, and then click New Application.

  2. Enter the following settings and then click Continue.

    1. Application Name: enter git-generator-files-discovery.
    2. GitOps Agent: select the Agent you added earlier.
    3. Service: create a new Service named git-generator-files-discovery.
    4. Environment: create a new Environment named git-generator-files-discovery.
    5. Select Pre-Production.

  3. Leave the default Sync Policy settings, and click Continue.

  4. In Source, enter the following settings and then click Continue.

    1. Repository URL: select the GitOps Repository you added earlier.
    2. Target Revision: select master.
    3. Path: enter examples/git-generator-files-discovery and click + to add it.

  5. In Destination, enter the following settings and then click Finish.

    1. Cluster: select the Agent cluster appset-example.
    2. Namespace: enter default.

The GitOps Application is added. Now you can sync it.

Sync the ApplicationSet to create the applications

A Sync brings the live state to its desired state by applying the declarative description. The guestbook application in the ApplicationSet will be created in the two target clusters. After the sync, the resources will look like this in Harness:

When you click Sync, Harness will use the ApplicationSet to create the new Harness Applications for the dev and prod clusters.

  1. In the GitOps Application, click SYNC.

  2. In the Sync settings, click Synchronize. Synchronization will take a minute.

In the git-generator-files-discovery Application Resource View, you can see the ApplicationSet and new Applications:

Congratulations! Now you have a working ApplicationSet in Harness deploying an application to two target clusters.

Next, we'll create a PR Pipeline to change the application in just one of the target clusters.

Create a PR Pipeline

As noted earlier, when you deploy a Harness PR Pipeline, you simply indicate the target environment application and the config.json keys/values you want changed. Harness creates the pull request in your Git repo and merges it for you. Once complete, the target environment application has the new keys/values.

For the PR Pipeline, we'll create two Harness Environments, dev and prod. These names are the same as the folder names in the repo:

We use the same names so that when we select a Harness Environment we can pass along the same name as the target folder.

Next, we'll create a Harness Service that points to the config.json files in these directories.

The path to the config.json files in the Service will use the expression <+env.name>: examples/git-generator-files-discovery/cluster-config/engineering/<+env.name>/config.json.

At runtime, this expression resolves to the Harness Environment you selected.

When you run the Pipeline, you'll select which Environment to use, dev or prod, and Harness will use the corresponding repo folder and update that application only.

Create Harness Environments for each target environment

First, let's create the dev Environment.

  1. In your Harness Project, click Environments.
  2. Click New Environment.
  3. Enter the following and click Save:
    1. Name: dev.
    2. Environment Type: Pre-Production.

The new Environment is created.

Create Variable for JSON key:value

Next, we'll add a Variable for the JSON key:value we will be updating.

  1. In Advanced, in Variables, click New Variable Override.

  2. In Variable Name, enter asset_id and click Save.

    The asset_id name is a key:value in the config.json files for both dev and prod:

  3. For the Variable Value, select Runtime Input:

    Later, when you run the Pipeline, you'll provide a new value for this variable, and that value will be used to update the config.json file.

Select GitOps Clusters to add to Environment

Next, we'll link the GitOps Clusters for dev with the dev Environment. Once you link GitOps Clusters to an Environment, you can then select from an Environment's linked GitOps Clusters when you select the Environment in a Pipeline.

  1. Click GitOps Clusters.
  2. Click Select Cluster(s).
  3. Select engineering-dev.
  4. Click Add.

The GitOps Cluster is now linked to the Environment.

Repeat the process for the prod Environment

  1. Create a new Environment named prod.
  2. Add the same asset_id Variable to the prod Environment.
  3. Link the engineering-prod GitOps Cluster to the Environment.

Create the PR Pipeline

To create the Pipeline, we'll simply create a new Service that includes the manifest you want deployed and select the dev Environment you created earlier.

  1. In your Harness Project, click Pipelines.

  2. Click Create a Pipeline.

  3. In Create new Pipeline, enter the name PR Pipeline, and then click Start.

  4. Click Add Stage, and select Deploy.

  5. Enter the following and click Set Up Stage:

    1. Stage Name: enter PR Example.
    2. Deployment Type: select Kubernetes.
    3. Enable the GitOps option.

    The stage is created and the Service settings appear.

Create the Service

The Harness Service represents what you're deploying. In this case, we're deploying a config.json file.

  1. In Select Service, click New Service.
  2. In Name, enter PR Example.
  3. In Manifests, click Add Release Repo Manifest.
  4. In Release Repo Store, select Github.

Now we'll add a Github Connector to tell Harness where to pull the config.json from.

Add Github Connector

  1. In Github Connector, click New Github Connector.

  2. Enter the following Github Connector settings:

    1. Name: enter gitops-github.
    2. URL Type: select Repository.
    3. Connection Type: select HTTP.
    4. GitHub Repository URL: enter the HTTP URL for repo you used for your ApplicationSet, such as https://github.com/johnsmith/applicationset.git.
    5. Authentication: select Username and Token. For the Token, you'll need to use a Personal Access Token (PAT) from Github. If you are logged into Github, just go to https://github.com/settings/tokens.
    6. Ensure the PAT has the repo scope selected.

    You will store the PAT in a Harness Text Secret. For details on Secrets Management, go to Harness Secrets Management Overview. 7. Select Enable API access and use the same Harness Secret.

  3. Click Continue.

  4. In Connect to the provider, select Connect through Harness Platform., and then click Save and Continue.

  5. When the Connection Test in complete, click Continue.

Specify manifest details

Now we'll define the manifest to use for the PR Pipeline. We'll use the path to the config.json files. We'll use the expression <+env.name> in the path so that we can dynamically select the path based on the Harness Environment we select: dev or prod.

  1. In Manifest Details, enter the following settings and then click Submit.

    1. Manifest Name: enter config.json.
    2. Git Fetch Type: select Latest from Branch.
    3. Branch: enter the name of the main branch (master, main, etc).
    4. File Path: enter examples/git-generator-files-discovery/cluster-config/engineering/<+env.name>/config.json.

    Note the use of <+env.name>.

  2. Back in New Service, click Save.

    The Service is added to the Pipeline.

  3. Click Continue to add the Environment.

Add Environment Runtime Input

For the stage Environment, we'll use a Runtime Input. When you run the Pipeline, Harness will prompt you for a value for the Environment. You can select the Environment you want to use for the PR.

  1. Set Specify environment or environment group as a Runtime Input.

  2. Click Continue.

Review Execution steps

In Execution, Harness automatically adds the following steps:

  • Update Release Repo: this step will fetch your JSON files, update them with your changes, Commit and Push, and then create the PR. You can also enter variables in this step to update key:value pairs in the config file you are deploying.

    If there is a matching variable name in the variables of the Service or Environment used in this Pipeline, the variable entered in this step will override them.

  • Merge PR: merges the new PR.

You don't have to edit anything in these steps.

Run and verify the PR Pipeline

Now your PR Pipeline is ready.

  1. Click Save, and then click Run.

  2. In Run Pipeline, in Specify Environment, select the dev Environment.

  3. In Environment Variables, for asset_id, enter the value 12345678.

  4. In Specify GitOps Clusters, select the engineeringdev cluster.

  5. Click Run Pipeline.

    You can review the deployment steps in real-time.

    Here's an example of each step:

  • Service:

    Starting service step...  
    Processing service variables...
    Applying environment variables and service overrides
    Processed service variables
    Processed artifacts and manifests
    Completed service step
  • GitOps Clusters:

    Environment(s): {dev}   

    Processing clusters at scope PROJECT
    Following 1 cluster(s) are present in Harness Gitops
    Identifiers: {engineeringdev}

    Following 1 cluster(s) are selected after filtering
    Identifiers: {engineeringdev}

    Completed
  • Update Release Repo:

  • Merge PR:

    PR Link: https://github.com/michaelcretzman/applicationset/pull/5  
    Pull Request successfully merged
    Commit Sha is 36f99ff737b98986045365e1b2be1326e97d4836
    Done.
  1. Check the repo to see that the config.json file for the dev environment has been updated with the new asset_id value:

Congratulations! You PR Pipeline was successful.

In this tutorial, you learned how to:

  1. Create an ApplicationSet that defines one application and syncs it to multiple target environments.
  2. Create a Harness PR Pipeline to change the application in just one of the target environments.

ApplicationSet and PR Pipeline Summary

A typical GitOps Application syncs a source manifest to a destination cluster. If you have multiple target clusters, you could create separate GitOps Applications for each one, but that makes management more challenging. Also, what if you want to sync an application with 100s of target clusters? Managing 100s of GitOps Applications is not acceptable.

To solve this use case, Harness supports ApplicationSets.

ApplicationSets

An ApplicationSet uses an ApplicationSet controller to automatically and dynamically generate applications in multiple target environments. A GitOps ApplicationSet is similar to a GitOps Application but uses a template to achieve application automation using multiple target environments.

ApplicationSet is supported in your cluster using a Kubernetes controller for the ApplicationSet CustomResourceDefinition (CRD).You add an ApplicationSet manifest to a Harness GitOps Application just like you would add a typical Deployment manifest. At runtime, Harness uses the ApplicationSet to deploy the application to all the target environments' clusters.

Template parameters

ApplicationSets use generators to generate parameters that are substituted into the template: section of the ApplicationSet resource during template rendering.

There are many types of generators. For the list, go to Generators from Argo CD docs.Generators support parameters in the format {{parameter name}}.

For example, here's the template section of a guestbook List generator that uses {{cluster.name}} and {{cluster.address}}:

  template:  
metadata:
name: '{{cluster.name}}-guestbook'
spec:
project: 191b68fc
source:
repoURL: https://github.com/johndoe/applicationset.git
targetRevision: HEAD
path: "examples/git-generator-files-discovery/apps/guestbook"
destination:
server: '{{cluster.address}}'
namespace: default
syncPolicy:
automated: {}

The values for these parameters will be taken from the cluster list config.json cluster.name and cluster.address:

{  
"releaseTag" : "k8s-v0.4",
"cluster" : {
"owner" : "[email protected]",
"address" : "https://34.133.127.118",
"name" : "dev"
},
"asset_id" : "12345678"
}

After substitution, this guestbook ApplicationSet resource is applied to the Kubernetes cluster:

apiVersion: argoproj.io/v1alpha1  
kind: Application
metadata:
name: dev-guestbook
spec:
source:
repoURL: https://github.com/johndoe/applicationset.git
path: examples/git-generator-files-discovery/apps/guestbook
targetRevision: HEAD
destination:
server: https://34.133.127.118
namespace: default
project: 191b68fc
syncPolicy:
automated: {}

PR Pipelines

Often, even though your ApplicationSet syncs one microservice/application to multiple target environments, you might want to change a microservice in just one of the target environments, such as a dev environment. A Harness PR Pipeline enables you to do this.

When you deploy a Harness PR Pipeline, you simply indicate what target environment application you want to update and the config.json keys/values you want changed, such as release tags. Harness creates the pull request in your Git repo and merges it for you. Now, the target environment application has the new keys/values.

Notes

ApplicationSet Support

Syncing

You can sync using the Harness ApplicationSet Application or sync the individual Harness Applications independently.

If you configure automatic syncing in the ApplicationSet template, then the Applications will be synced automatically. See syncPolicy.automated here:

  template:  
metadata:
name: '{{cluster.name}}-guestbook'
spec:
project: 191b68fc
source:
repoURL: https://github.com/johndoe/applicationset.git
targetRevision: HEAD
path: "examples/git-generator-files-discovery/apps/guestbook"
destination:
server: '{{cluster.address}}'
namespace: default
syncPolicy:
automated: {}

The syncPolicy.automated is applied to all Applications created by the ApplicationSet because it is part of the template.

If you make a change to one of the target Applications and then perform an ApplicationSet sync, the change to target Application will be preserved.