Use Catalog Ingestion API

Steps to use Catalog Ingestion API to ingest metadata and use the information on catalog overview and workflows

In this tutorial we will be using the catalog metadata ingestion APIs to add additional metadata for the software components in the catalog, display them in the component overview page using additional info card and track the values ingested using Scorecard.

Before you begin

1. You must have components registered in your software catalog. If you do not have any components registered, follow this tutorial to register one. We recommend you to register this software component for this tutorial.

Add a new metadata

Use cURL command

• Use the following command to add a new metadata codeCoverageScore to the warehouse component.

curl --location 'https://app.harness.io/gateway/v1/catalog/custom-properties/entity/' \
--header 'Content-Type: application/json' \
--header 'x-api-key: <HARNESS_TOKEN>' \
--data '{
    "entity_ref": "warehouse",
    "property": "metadata.codeCoverageScore",
    "value": 33
}'

OR

Use the python script

• You can use the python script, to auto-update the <+metadata.codeCoverageScore>

OR

Use Harness pipeline

• We recommend the use of following Harness Pipeline to add a new metadata codeCoverageScore to the warehouse component. While using the following pipeline YAML make sure to add the x-api-key as pipeline variable of type secret and name apikey.

Harness Pipeline

pipeline:
  name: catalog-ingestion-api-tutorial
  identifier: catalogingestionapitutorial
  projectIdentifier: <PROJECT_ID>
  orgIdentifier: <ORG_ID>
  tags: {}
  stages:
    - stage:
        name: demo-stage
        identifier: demostage
        description: ""
        type: IDP
        spec:
          platform:
            os: Linux
            arch: Amd64
          runtime:
            type: Cloud
            spec: {}
          execution:
            steps:
              - step:
                  type: Run
                  name: Run_1
                  identifier: Run_1
                  spec:
                    shell: Python
                    command: |-
                      import requests
                      import random

                      # Define the API endpoint and headers
                      url = 'https://app.harness.io/gateway/v1/catalog/custom-properties/entity'
                      headers = {
                          'Content-Type': 'application/json',
                          'x-api-key': '<+pipeline.variables.apikey>'  # Replace with your actual API key
                      }

                      # Fetch the product data from the API
                      product_url = 'https://dummyjson.com/products'
                      response = requests.get(product_url)

                      # Check if the response is successful
                      if response.status_code == 200:
                          data = response.json()

                          # Extract stock data
                          products = data.get("products", [])
                          total_stock = sum(product.get("stock", 0) for product in products)

                          # Introduce variability
                          random_max_possible_stock = random.randint(500, 2000)  # Randomize max stock
                          random_factor = random.uniform(0.5, 1.5)  # Random multiplier for variation

                          # Calculate base score and apply randomness
                          base_score = (total_stock / random_max_possible_stock) * 100
                          code_coverage_score = min(base_score * random_factor, 100)  # Ensure < 100

                          print(f"Code Coverage Score: {code_coverage_score:.2f}")

                          # Prepare the data to update the code coverage score
                          data_payload = {
                              "entity_ref": "warehouse",
                              "property": "metadata.codeCoverageScore",
                              "value": round(code_coverage_score, 2)  # Send as a number, not a string
                          }

                          # Make the POST request to update the value
                          update_response = requests.post(url, headers=headers, json=data_payload)

                          # Check the response from the update request
                          if update_response.status_code == 200:
                              print("Code coverage score updated successfully!")
                              print("Response:", update_response.json())
                          else:
                              print(f"Failed to update the code coverage score. HTTP Status Code: {update_response.status_code}")
                              print("Response:", update_response.text)
                      else:
                          print(f"Failed to fetch product data. HTTP Status Code: {response.status_code}")
        tags: {}
  variables:
    - name: apikey
      type: Secret
      description: ""
      required: false
      value: x-api-key

Schedule a cron to update the catalog metadata

• If you use the cURL command or the python script to auto-update the metadata on regular intervals you need to configure a cron job.

• In case you are using Harness Pipeline use a trigger to update the data ingested on regular intervals. Create a pipeline trigger of type Cron to schedule an hourly trigger.

Code coverage scorecard {#code-coverage-scorecard}

In case you want to track the information you have ingested using Scorecards, follow the steps below.

1. Go to the Scorecards and under Configure and select Checks from the top nav.

2. Now select Create Custom Check and add a New Custom Check to check the code coverage score, select the data source as Catalog Info YAML and data point as Evaluate Expression(JEXL).

3. Add the jexl to be evaluated as <+metadata.codeCoverageScore> and operator Greater than or equal to from the dropdown and add the value as 50

4. Now you can create a scorecard using the Custom Check you have created above.

5. Now go to the Warehouse Software Component in the Catalog, and you will find the Code Coverage Scorecard under the Scorecards tab and on the overview page. Go to Scorecards custom data sources to read more.

Additional info card {#additional-info-card}

In case you want to display the same information you have ingested on your Overview page as an additional card, follow the steps below.

1. Go to the Layout Page and under Configure and select Catalog Entities.

2. Now Select Edit Layout for component Service and add the following under Overview card.

        - component: EntityAdditionalInfoCard
          specs:
            props:
              title: Additional Info Card
              items:
                - label: Code Coverage Score
                  value: <+metadata.additionalInfo.codeCoverageScore>
                  type: string
                  style:
                    bold: true
            gridProps:
              md: 6

1. Now go to the Warehouse Software Component in the Catalog, and you will find an additional info card populated with information we ingested using the API above. You can read more about additional info card

Note: For detailed instructions on adding an Additional Info Card to display custom information on your entity's Overview page, see the Additional Info Card section in the Catalog Layout documentation.

Conclusion

Using the Catalog Metadata Ingestion API, you can source information into your catalog from internal systems such as cost trackers, service health checkers, security scans, or even from simple spreadsheets tracking personnel details and use them to just display the information to users as wells as use them as an input for workflows.

Check out other examples of the Catalog Ingestion API.