Skip to main content

Catalog Ingestion API

Introduction

Software Catalog in Harness IDP supports modifying metadata associated with the services, libraries, websites and any other entities registered in the Catalog, without having to manually edit the catalog-info.yaml. This allows integrations with many systems such as cost trackers, service health checkers, security scans or even simple spreadsheets tracking personnel details like who is on-call this week.

The ingested data can be used to display on the UI, can be consumed in Custom Plugins and measured in Scorecard Checks.

Use Cases - API Quick Overview

1. Update a Single Property of a Single Catalog Entity

POST /catalog/custom-properties/entity
Payload
{
"entity_ref": "boutique-service",
"property": "metadata.teamLead",
"value": "David John"
}

2. Update Multiple Properties of a Single Catalog Entity

POST /catalog/custom-properties/entity
Payload
{
"entity_ref": "boutique-service",
"properties": [
{
"property": "metadata.teamLead",
"value": "David John"
},
{
"property": "metadata.teamOwner",
"value": "Grace"
}
]
}

3. Update a Single Property of Multiple Catalog Entities

POST /catalog/custom-properties/property
Payload
{
"property": "metadata.releaseVersion",
"entity_refs": [
{
"entity_ref": "component:default/order-service",
"value": 1.2.0
},
{
"entity_ref": "component:default/idp-service",
},
{
"entity_ref": "component:default/pipeline-service",
}
],
"value": 1.5.0,
}

4. Update a Single Property of Multiple Unspecified Catalog Entities using Filters

POST /catalog/custom-properties
Payload
{
"property": "metadata.teamLead",
"filter": {
"kind": "Component",
"type": "service"
},
"value": "Jane Doe"
}

Common API Request Details

API Base URL

https://app.harness.io/gateway/v1/

Headers

X-API-KEY: Harness API Key
Harness-Account: YOUR ACCOUNT ID
Content-Type: application/json

You can generate your Harness API Keys by following the Harness API Key docs.

You can find your account ID in any of your Harness URLs, for example: https://app.harness.io/ng/account/<ACCOUNT_ID>/idp/overview.

Detailed API Usage and Examples

Update a Single Property of a Single Catalog Entity

cURL Example

curl \
--location 'https://app.harness.io/gateway/v1/catalog/custom-properties/entity' \
--header 'Harness-Account: ADD_YOUR_ACCOUNT_ID' \
--header 'Content-Type: application/json' \
--header 'x-api-key: ADD_YOUR_API_KEY' \
--data '{
"entity_ref": "boutique-service",
"property": "metadata.codeCoverageScore",
"value": "83"
}'

Endpoint

HTTP Method

POST

URL

https://app.harness.io/gateway/v1/catalog/custom-properties/entity

Request Body

{
"entity_ref": "boutique-service",
"property": "metadata.codeCoverageScore",
"value": "83"
}

In the above example, we update/add only the metadata.codeCoverageScore property for the mentioned entity boutique-service.

What is Entity Ref?

Entity ref is a stringified way of referencing a Catalog entity with the format [<kind>:][<namespace>/]<name>. namespaces are optional and are usually default. kind is often optional too and is assumed to be either Component, API or Group depending upon the context. The simplest way to represent a component is to use the name field metadata.name. For example

  • boutique-service

This internally translates into component:default/boutique-frontend which means the entity is of kind: Component in the default namespace.

Here are some other examples of valid entity_refs -

  • component:order-service
  • api:petstore
  • group:my-team
  • component:default/my-service

Entity Refs are case-insensitive.

How to check the final entity?

You can use the Entity Inspector to view the final Raw YAML.

Example Processed Entity YAML as displayed in Harness IDP

apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
name: my-new-service
description: Description of my new service
annotations:
harness.io/project-url: https://app.harness.io/ng/account/vpCkHK/module/idp-admin/orgs/default/projects/Backstage/pipelines/Releasenpm/
codeCoverageScore: 83
...

Update Multiple Properties of a Single Catalog Entity

cURL Example

curl \
--location 'https://app.harness.io/gateway/v1/catalog/custom-properties/entity' \
--header 'Harness-Account: ADD_YOUR_ACCOUNT_ID' \
--header 'Content-Type: application/json' \
--header 'x-api-key: ADD_YOUR_API_KEY' \
--data '{
"entity_ref": "boutique-service"
"properties": [
{
"property": "metadata.codeCoverageScore",
"value": "83"
},
{
"property": "metadata.cloudCost",
"value": "$2400",
}
]
}'

Endpoint

HTTP Method

POST

URL

https://app.harness.io/gateway/v1/catalog/custom-properties/entity

Request Body

{
"entity_ref": "boutique-service",
"properties": [
{
"property": "metadata.codeCoverageScore",
"value": "83"
},
{
"property": "metadata.cloudCost",
"value": "$2400"
}
]
}

In the above example, we add the metadata.codeCoverageScore & metadata.cloudCost properties for the mentioned boutique-service.

What is Entity Ref?

Entity ref is a stringified way of referencing a Catalog entity with the format [<kind>:][<namespace>/]<name>. namespaces are optional and are usually default. kind is often optional too and is assumed to be either Component, API or Group depending upon the context. The simplest way to represent a component is to use the name field metadata.name. For example

  • boutique-service

This internally translates into component:default/boutique-frontend which means the entity is of kind: Component in the default namespace.

Here are some other examples of valid entity_refs -

  • component:order-service
  • api:petstore
  • group:my-team
  • component:default/my-service

Entity Refs are case-insensitive.

How to check the final entity?

You can use the Entity Inspector to view the final Raw YAML.

Example Processed Entity YAML as displayed in Harness IDP

apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
name: my-new-service
description: Description of my new service
annotations:
harness.io/project-url: https://app.harness.io/ng/account/vpCkHK/module/idp-admin/orgs/default/projects/Backstage/pipelines/Releasenpm/
codeCoverageScore: 83
...

Update a Single Property of Multiple Catalog Entities

cURL Example

curl --location 'https://app.harness.io/gateway/v1/catalog/custom-properties/entity' \
--header 'Harness-Account: ADD_YOUR_ACCOUNT_ID' \
--header 'Content-Type: application/json' \
--header 'x-api-key: ADD_YOUR_API_KEY' \
--data '{
"property": "metadata.releaseVersion",
"entity_refs": [
{
"entity_ref": "component:default/order-service",
"value": 1.2.0
},
{
"entity_ref": "component:default/idp-service",
},
{
"entity_ref": "component:default/pipeline-service",
}
],
"value": 1.5.0,
}'
info

You can skip the corresponding values of some entities in the entity_refs array and provide the default value under root value field of the payload. Value field is optional if all entities have a corresponding value specified.

How to check the final entity?

You can use the Entity Inspector to view the final Raw YAML.

Example Processed Entity YAML as displayed in Harness IDP

apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
name: my-new-service
description: Description of my new service
annotations:
harness.io/project-url: https://app.harness.io/ng/account/vpCkHK/module/idp-admin/orgs/default/projects/Backstage/pipelines/Releasenpm/
codeCoverageScore: 83
...

Endpoint

HTTP Method

POST

URL

https://app.harness.io/gateway/v1/catalog/custom-properties/entity

Request Body

{
"property": "metadata.releaseVersion",
"entity_refs": [
{
"entity_ref": "component:default/order-service",
"value": 1.2.0
},
{
"entity_ref": "component:default/idp-service",
},
{
"entity_ref": "component:default/pipeline-service",
}
],
"value": 1.5.0,
}

In the above example, we add the metadata.releaseVersion property for the mentioned entities.

What is Entity Ref?

Entity ref is a stringified way of referencing a Catalog entity with the format [<kind>:][<namespace>/]<name>. namespaces are optional and are usually default. kind is often optional too and is assumed to be either Component, API or Group depending upon the context. The simplest way to represent a component is to use the name field metadata.name. For example

  • boutique-service

This internally translates into component:default/boutique-frontend which means the entity is of kind: Component in the default namespace.

Here are some other examples of valid entity_refs -

  • component:order-service
  • api:petstore
  • group:my-team
  • component:default/my-service

Entity Refs are case-insensitive.

How to check the final entity?

You can use the Entity Inspector to view the final Raw YAML.

Example Processed Entity YAML as displayed in Harness IDP

apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
name: my-new-service
description: Description of my new service
annotations:
harness.io/project-url: https://app.harness.io/ng/account/vpCkHK/module/idp-admin/orgs/default/projects/Backstage/pipelines/Releasenpm/
codeCoverageScore: 83
...

Update a Single Property of Multiple Unspecified Catalog Entities using Filters

HTTP Method

POST

URL

https://app.harness.io/gateway/v1/catalog/custom-properties

Request Body

{
"property": "metadata.teamLead",
"filter": {
"kind": "Component",
"type": "service"
},
"skip_entity_refs": ["idp-service"],
"value": "Jane Doe"
}
  • skip_entity_refs: (Optional) Entities mentioned in this array are not modified with this request.
What is Entity Ref?

Entity ref is a stringified way of referencing a Catalog entity with the format [<kind>:][<namespace>/]<name>. namespaces are optional and are usually default. kind is often optional too and is assumed to be either Component, API or Group depending upon the context. The simplest way to represent a component is to use the name field metadata.name. For example

  • boutique-service

This internally translates into component:default/boutique-frontend which means the entity is of kind: Component in the default namespace.

Here are some other examples of valid entity_refs -

  • component:order-service
  • api:petstore
  • group:my-team
  • component:default/my-service

Entity Refs are case-insensitive.

  • field: It contains the information on the metadata name to be added, here in the above example it would ingest the teamLead under metadata. This won't append your catalog-info.yaml stored in your git, rather you could view the changes on IDP.
info

We need to add escape character for any field has an additional DOT in the path like metadata.annotation.harness.io/idp-test , that part needs to be escaped with \" like this metadata.annotations.\"harness.io/idp-test\"

  • filter: This is used to identify the software components where you want to ingest the new entity, you can filter through kind, type, owners, lifecycle and tags. Where only kind is the mandatory fields.
info

Error Handling: We validate the body of the API and certain fields like kind, metadata, metadata.name, metadata.namespace, are uneditable and if you try to change these, the endpoint returns an Error Code 400. Also make sure your metadata updates adhere to the backstage schema

How to check the final entity?

You can use the Entity Inspector to view the final Raw YAML.

Example Processed Entity YAML as displayed in Harness IDP

apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
name: my-new-service
description: Description of my new service
annotations:
harness.io/project-url: https://app.harness.io/ng/account/vpCkHK/module/idp-admin/orgs/default/projects/Backstage/pipelines/Releasenpm/
codeCoverageScore: 83
...

Other Examples

Dry Run

Using the dry_run field, you can check all the metadata and components getting affected by the Catalog Metadata Ingestion API without actually modifying them. dry_run won't apply any change rather will provide a preview of all the changes as shown in the example below.

[
{
"field": "metadata.offShoreTeamLead",
"entities_with_additions": {
"count": 0,
"entity_refs": []
},
"entities_with_updates": {
"count": 2,
"entity_refs": [
"component:default/order-service",
"component:default/foodservice"
]
}
}
]

As you could see in the example above we display the affected software components under entity_refs. To use dry_run you need to add ?dry_run=true field in the URL

https://app.harness.io/gateway/v1/catalog/custom-properties?dry_run=true

Advanced example of using Catalog Filters

POST https://app.harness.io/gateway/v1/catalog/custom-properties
Payload
{
"property": "metadata.releaseVersions",
"filter": {
"kind": "Component",
"type": "service",
"owners": ["harness_account_all_users"],
"lifecycle": ["experimental", "production"],
"tags": ["food-ordering", "java", "tag1"]
},
"value": [
{
"prod1": "1.5"
},
{
"prod2": "1.4"
},
{
"prod3": "1.3"
}
]
}

Delete a single custom property for a given entity

curl --location --request DELETE 'https://app.harness.io/gateway/v1/catalog/custom-properties/entity' \
--header 'Harness-Account: ADD_YOUR_ACCOUNT_ID' \
--header 'Content-Type: application/json' \
--header 'x-api-key: ADD_YOUR_API_KEY' \
--data '{
"entity_ref": "boutique-service",
"property": "metadata.teamLead"
}'

Note that this only works for the custom properties added using the Ingestion APIs. This API will not remove any property added using the Catalog Info YAML.

Delete multiple properties for a given entity

curl --location --request DELETE 'https://app.harness.io/gateway/v1/catalog/custom-properties/entity' \
--header 'Harness-Account: ADD_YOUR_ACCOUNT_ID' \
--header 'Content-Type: application/json' \
--header 'x-api-key: ADD_YOUR_API_KEY' \
--data '{
"entity_ref": "boutique-service",
"properties": ["metadata.teamLead", "metadata.teamOwner"]
}'

Delete a single property for multiple entities

curl --location --request DELETE 'https://app.harness.io/gateway/v1/catalog/custom-properties/property' \
--header 'Harness-Account: ADD_YOUR_ACCOUNT_ID' \
--header 'Content-Type: application/json' \
--header 'x-api-key: ADD_YOUR_API_KEY' \
--data '{
"property": "metadata.releaseVersion",
"entity_refs": ["idp-service", "order-service"]
}'

Delete one property on Unspecified Catalog Entities using Filters

curl --location --request DELETE 'https://app.harness.io/gateway/v1/catalog/custom-properties' \
--header 'Harness-Account: ADD_YOUR_ACCOUNT_ID' \
--header 'Content-Type: application/json' \
--header 'x-api-key: ADD_YOUR_API_KEY' \
--data '{
"property": "metadata.teamLead",
"filter": {
"kind": "Component",
"type": "service"
},
"skip_entity_refs": ["order-service"],
}'

In the above example it will delete the property metadata.teamLead, across all the entities except order-service mentioned under skip_entity_refs.

Get Catalog Custom Properties for a given Entity

GET /catalog/custom-properties/entity?entity_ref=boutique-service

Get Entities associated with a Custom Property

GET /catalog/custom-properties/entity?property=metadata.releaseVersion

Other Catalog Endpoints

This page describes only the ingestion related Catalog endpoints. Please look at other Catalog API endpoints which contains basic endpoints for fetching full entity, registering and unregistering entities.

API takes priority over Catalog YAML file (in case of a conflict)

Any property updated using Ingestion APIs will take priority over what is specified in the Catalog Info YAML files. For example, if a catalog-info.yaml has the following

# ...
metadata:
name: boutique-frontend
customProperty: valueA

And you use the Ingestion API to update the metadata.customProperty to valueB. Then the final value of the property will be valueB. However, note that this will not update the actual catalog-info.yaml inside your Git repository.