> For the complete documentation index, see [llms.txt](https://developer.harness.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://developer.harness.io/internal-developer-portal/use-idp/software-catalog/tutorials/migrate-catalog-scripts/content/populate-catalog/github-idp-2.md).

# Github Idp 2

In large engineering organizations, service sprawl across hundreds or even thousands of GitHub repositories is common. Manually onboarding each service into the Harness Software Catalog, either by creating catalog-info YAMLs individually or configuring one catalog location per repository, quickly becomes unmanageable. Moreover, using the default discovery plugins to register each repository as a location can lead to fragility; a single failure during sync can prevent the entire catalog from updating correctly.

This script provides a scalable and GitOps-friendly solution by automating the end-to-end onboarding process using the Harness IDP 2.0 **Entities API** and **Git Experience (GitX)**. It programmatically fetches all repositories from your GitHub organization, dynamically generates a valid `idp.yaml` for each service, and pushes that file into a centralized GitHub repository. Each YAML is committed to a unique path using the Git connector you have configured in Harness. This ensures that service metadata is not only standardized but also version-controlled in Git, promoting visibility and auditability.

## Script source <a href="#script-source" id="script-source"></a>

[Source Code](https://github.com/harness-community/idp-samples/blob/main/IDP-2.0-Samples/catalog-scripts/idp-catalog-population-multirepo-github.py)

```bash
curl -o idp-catalog-population-multirepo-github.py https://raw.githubusercontent.com/harness-community/idp-samples/main/IDP-2.0-Samples/catalog-scripts/idp-catalog-population-multirepo-github.py
```

## Before you begin <a href="#before-you-begin" id="before-you-begin"></a>

* [Python 3](https://www.python.org/downloads/) with [`requests`](https://pypi.org/project/requests/) and [`python-dotenv`](https://pypi.org/project/python-dotenv/) libraries installed.
* A `.env` file configured with the following:

```ini
GITHUB_TOKEN = '<github-token>'
HARNESS_API_KEY = '<harness-api-key>'
HARNESS_ACCOUNT_ID = '<harness-account-id>'
CONNECTOR_REF = '<harness-git-connector-ref>'
ORG_IDENTIFIER = '<harness-org-id>'
PROJECT_IDENTIFIER = '<harness-project-id>'
CENTRAL_REPO = '<name-of-central-repo-to-store-yamls>'
GITHUB_ORG = '<github-org-name>'
```

> `GITHUB_TOKEN` must have `repo` and `read:org` permissions. `HARNESS_API_KEY` must be a [User/API Key](/harness-ai/use-harness-platform/automation/api/api-quickstart.md#create-a-harness-api-key-and-token) with write access to IDP entities.

## Execution <a href="#execution" id="execution"></a>

After creating your `.env` file, run the script:

```bash
python3 idp-catalog-population-multirepo-github.py
```

This will:

1. Fetch all repositories from your GitHub org.
2. For each repo:
   * Sanitize the identifier.
   * Generate a valid `idp.yaml`.
   * Push it to the specified folder in `CENTRAL_REPO`.
   * Register the entity in Harness using the Entities API.

## Output structure <a href="#output-structure" id="output-structure"></a>

The catalog YAML files will be stored in the following pattern inside your central GitHub repository:

```
central-repo/
  ├── service-one/
  │   └── idp.yaml
  ├── service-two/
  │   └── idp.yaml
  └── ...
```

Each YAML will look like:

```yaml
apiVersion: harness.io/v1
kind: component
orgIdentifier: <your-org>
projectIdentifier: <your-project>
type: Service
identifier: sanitized_unique_id
name: repo-name
owner: group:account/IDP_Test
spec:
  lifecycle: production
metadata:
  description: "repo description from GitHub"
  annotations:
    backstage.io/source-location: url:https://github.com/<your-org>/<repo-name>
    backstage.io/techdocs-ref: dir:.
  tags:
    - auto-onboarded
```

## Logs & troubleshooting <a href="#logs-and-troubleshooting" id="logs-and-troubleshooting"></a>

* Script output includes a status message for each repo (success/failure).
* Failures are logged with full error messages from the Harness API.
* For personal GitHub accounts, change the GitHub API URL from:

```python
https://api.github.com/orgs/{GITHUB_ORG}/repos
```

to:

```python
https://api.github.com/users/{GITHUB_ORG}/repos
```

{% hint style="info" %}
If you are interested to try out different request to work with entities, you can use the new [Entities APIs](https://apidocs.harness.io/tag/Entities).
{% endhint %}