Service Onboarding using Scripts on Catalog
Scripts to create new services, register new services
- GitHub
- Bitbucket
-
The GitHub Catalog Discovery plugin registers one location per repository. This might not be a good idea when there are many (3000+ in this case) as any error in fetching one
catalog-yaml
would mark the whole location as failed and create trouble with the entity sync. -
To solve this we would recommend you to use the following scripts which would register separate locations for all the matching catalog-info.yaml files and hence would be synchronised separately.
Pre-Requisites to Use the Script
- Harness API Key - Docs
- GitHub Token - Docs
- A Repository to store all the IDP Config YAMLs
- Python 3 installed on your machine where you're trying to execute the code along with requests library.
Download the Script
-
A pregenerated/created repo (let's call it chosen_repo) is to be cloned. After opening it on your code editor and from chosen_repo, below commands can be run to first download and then generate and register
catalog-info.yaml
files of all the repos in your org. Thecatalog-info.yaml
files will be in -chosen_repo/services/{repos}/catalog-info.yaml
where repos will be all the repo in your org. -
Command to Download the Script
curl -o idp-catalog-wizard-github.py https://raw.githubusercontent.com/harness-community/idp-samples/main/catalog-scripts/idp-catalog-wizard-github.py
python3 idp-catalog-wizard-github.py [OPTIONS]
-
Case 1: Run command using
--create-yamls
args, then you'll have to manually push thefiles - "services/" .....
after which you can run command using--register-yamls
args to register all the yamls. -
Case 2: Run command using
--run-all
args, all actions will be performed - create, push and register in one go.
Registered Locations
Register YAML files using X-API-Key and account name (all given args in command below are required)
python3 idp-catalog-wizard-github.py --register-yamls --org org_name --x_api_key your_x_api_key --account your_account
org
: Github Org namex_api_key
: Refer https://developer.harness.io/docs/platform/automation/api/api-quickstart/#create-a-harness-api-key-and-token to generate one- -
account
: This is your Harness Account ID. You can get it from the URL e.g. -https://app.harness.io/ng/account/{Your account ID}/module/idp/overview
Perform all actions: create YAML files, push changes, and register YAML files (all given args in command below are required)
python3 idp-catalog-wizard-github.py --run-all --org example-org --token your_token --x_api_key your_x_api_key --account your_account
Registered Locations - For Monorepos
-
Discover
catalog-info.yaml
matching the regex filter and register under the catalog provided inapiurl
. This would separate locations for all the matching catalog-info.yaml files and hence would be synchronized separately. -
To use the script you need to add the appropriate flags and run it.
Create Services
- Generates a monorepo with the following file structure, assigning random english names.
repo
- antronasal-service
- catalog-info.yaml
- cespititous-service
- catalog-info.yaml
- ....
- geomaly-service
- catalog-info.yaml
Delete Services
- Will clean up the services already created.
-
Similar to GitHub, Bitbucket Catalog Discovery plugin registers one location per repository. This might not be a good idea when there are many (3000+ in this case) as any error in fetching one
catalog-yaml
would mark the whole location as failed and create trouble with the entity sync. -
To solve this we would recommend you to use the following scripts which would register separate locations for all the matching
catalog-info.yaml
files and hence would be synchronised separately.
Pre-Requisites to Use the Script
- Harness API Key - Docs
- Bitbucket App Password - Docs
- A Repository to store all the IDP Config YAMLs
- Python 3 installed on your machine where you're trying to execute the code along with requests library.
Download the Script
-
A pregenerated/created repo (let's call it chosen_repo) is to be cloned. After opening it on your code editor and from chosen_repo, below commands can be run to first download and then generate and register
catalog-info.yaml
files of all the repos in your org. Thecatalog-info.yaml
files will be in -chosen_repo/services/{repos}/catalog-info.yaml
where repos will be all the repo in your org. -
Command to Download the Script
curl -o idp-catalog-wizard-bitbucket.py https://raw.githubusercontent.com/harness-community/idp-samples/main/catalog-scripts/idp-catalog-wizard-bitbucket.py
python3 idp-catalog-wizard-bitbucket.py [OPTIONS]
-
Case 1.a: Run command using
--create-yamls
args, then you'll have to manually push the files- "services/" .....
after which you can run command using--register-yamls
args to register all the yamls. Scope of this command is your whole workspace. -
Case 1.b: Add all args as given for case 1.a example below with
--project_key
which will keep your scope to project instead of workspace. -
Case 2.a: Run command using
--run-all
args, all actions will be performed - create, push and register in one go. -
Case 2.b: Add all args as given for case 2.a example below with
--project_key
which will keep your scope to project instead of workspace.
Example:
Create YAML files for repositories in the organization "example-org" with the provided token (all given args in command below are required)
Bitbucket username
isn't same as your email address, rather follow this docs, to find your username.
python3 idp-catalog-wizard-bitbucket.py --create-yamls --workspace example_workspace --username bitbucket_username --password bitbucket --project_key bitbucket_project_key
workspace
: Bitbucket workspace nameusername
: Bitbucket Usernamepassword
: Bitbucket app passwordproject_key
: (OPTIONAL) Bitbucket project keyrepo-pattern
: (OPTIONAL) Your repo pattern
Registered Locations - For Monorepos
Registered Locations
Register YAML files using X-API-Key and account name (all given args in command below are required)
python3 idp-catalog-wizard-bitbucket.py --register-yamls --workspace bitbucket_workspace --x_api_key your_x_api_key --account harness_account
workspace
: Bitbucket workspace namex_api_key
: Refer docs to generate one.account
: This is your Harness Account ID. You can get it from the URL e.g. -https://app.harness.io/ng/account/{Your account ID}/module/idp/overview
Perform all actions: create YAML files, push changes, and register YAML files (all given args in command below are required)
python3 idp-catalog-wizard-bitbucket.py --run-all --workspace example-workspace --password app_password --x_api_key your_x_api_key --account your_account --project_key (optional) project-key
Refer Create YAML and Register YAML for the arg details
- Discover
catalog-info.yaml
matching the regex filter and register under the catalog provided inapiurl
. This would separate locations for all the matching catalog-info.yaml files and hence would be synchronized separately.