Ctrlk
Sign UpGet a DemoContact Us
For the complete documentation index, see llms.txt. This page is also available as Markdown.

How to use Catalog Ingestion API to push data into Harness IDP

A tutorial on using rest APIs to fetch information in IDP

In this tutorial we will be creating Jira tickets using Workflows and add the information (i.e. ticket numbers) to the corresponding Software Component in the Catalog and display the same in the UI using Additional Info Card. The aim of this tutorial is to help you understand the usage of REST APIs to push information in the Catalog and then use them in different parts of IDP according to your use-cases.

Before you begin

  1. We assume you have JIRA set-up as your ticket management system, and you have access to the same to create tickets in your corresponding project as well as administrator access to create projects.

Create the workflow

Create Jira ticket

Use RUN step

  1. Go to Configure in your IDP

  2. Now select the project where you want to create the pipeline for the Workflows.

  3. Begin by selecting the Create a Pipeline button followed by adding a name for the pipeline and set up your pipeline as inline.

  4. Now select the Developer Portal Stage and give it a name.

  5. Add a RUN step, name it as create jira ticket and select the Shell as Bash

  6. Now add the following under the Command.

EMAIL_ID="<+pipeline.variables.email-id>"
JIRA_TOKEN="<+pipeline.variables.jiratoken>"
PROJECT_KEY="<+pipeline.variables.projectkey>"
COMPONENT_NAME="<+pipeline.variables.componentname>"
ISSUE_TYPE="<+pipeline.variables.issuetype>"
ISSUE_SUMMARY="<+pipeline.variables.issuesummary>"
ISSUE_CONTENT="<+pipeline.variables.issuecontent>"
LABELS="<+pipeline.variables.labels>"

# Perform the POST request with curl and capture the response <a href="#perform-the-post-request-with-curl-and-capture-the-response" id="perform-the-post-request-with-curl-and-capture-the-response"></a>
response=$(curl --silent --request POST \
  --url 'https://harness.atlassian.net/rest/api/3/issue' \
  --user "$EMAIL_ID:$JIRA_TOKEN" \
  --header 'Accept: application/json' \
  --header 'Content-Type: application/json' \
  --data '{
    "fields": {
        "project": {
            "key": "$PROJECT_KEY"
        },
        "components": {
            "name": "$COMPONENT_NAME"
        },
        "issuetype": {
            "name": "$ISSUE_TYPE"
        },
        "summary": "$ISSUE_SUMMARY",
        "description": {
            "version": 1,
            "type": "doc",
            "content": [
                {
                    "type": "paragraph",
                    "content": [
                        {
                            "type": "text",
                            "text": "$ISSUE_CONTENT"
                        }
                    ]
                }
            ]
        },
        "labels": [
            "$LABELS"
        ]
    }
}')

# Extract the key from the JSON response using jq <a href="#extract-the-key-from-the-json-response-using-jq" id="extract-the-key-from-the-json-response-using-jq"></a>
issue_key=$(echo "$response" | jq -r '.key')

# Export the issue key as an environment variable <a href="#export-the-issue-key-as-an-environment-variable" id="export-the-issue-key-as-an-environment-variable"></a>
export ISSUE_KEY="$issue_key"

# Print the issue key (optional) <a href="#print-the-issue-key-optional" id="print-the-issue-key-optional"></a>
echo "The created issue key is: $ISSUE_KEY"

The above given request body can create a Jira ticket based on project and component and add a label to the same.

We have used few pipeline variables in the body, which will be used to take input from the IDP Workflows and for the users to choose project, add the summary, description for the tickets.

  1. Now under Optional Configuration add the Output Variables as ISSUE_KEY.

  2. Apply the Changes.

  3. Go to Variables on the right side of the page and add the following variables and have the input type as Runtime Input.

    • emailid

    • jiratoken

    • projectkey

    • componentname

    • issuetype

    • issuesummary

    • issuecontent

    • labels

    • usergroup

  4. Apply the changes.

Use catalog metadata ingestion API

  1. Start by adding another RUN step.

  2. Name it as Ingestion API and select the Shell as Bash

  3. Now add the following under the Command.

  1. Under header x-api-key: <+secrets.getValue('account.TOKEN_ID')>, add the token ID for your API key. Get your token ID from your Profile

In the above body the openTicket which got created in JIRA will be added, to kind component and type service owned by the usergroup selected in the Workflows. Under entity_ref add the component link to which you want to add the ticket ID, the unique entity reference could be found using inspect entity for the component in Catalog.

  1. Now Apply the changes.

Use HTTP step

HTTP Step is available under custom stage and it is part of CD License

  1. Go to Configure in your IDP

  2. Now select the project where you want to create the pipeline for the Workflows.

  3. Begin by selecting the Create a Pipeline button followed by adding a name for the pipeline and set up your pipeline as inline.

  4. Now select the Custom Stage and give it a name.

  5. Add a HTTP step and name it as create jira ticket

  6. Under the URL add this endpoint https://YOUR_COMPANY.atlassian.net/rest/api/3/issue and replace the YOUR_COMPANY with your domain name.

  7. Select the Method as POST.

  8. Under the body add the following:

The above given request body can create a Jira ticket based on project and component and add a label to the same.

We have used few pipeline variables in the body, which will be used to take input from the IDP Workflows and for the users to choose project, add the summary, description for the tickets.

  1. Under Optional Configuration add the Assertion as <+httpResponseCode>==201.

  2. Under Headers add the following key value pairs:

    • Accept: application/json

    • Authorization: Basic <Base64 encoded username and password>, the username is the email-id and the password would be the JIRA API key

    • Accept-Encoding: gzip, deflate, br

  3. Under output add a variable issue_key and assign a value as expression <+json.object(httpResponseBody).key>

  4. Now Apply Changes.

  5. Go to Variables on the right side of the page and add the following variables and have the input type as Runtime Input.

    • projectkey

    • componentname

    • issuetype

    • issuesummary

    • issuecontent

    • labels

    • usergroup

  6. Apply the changes.

Use catalog metadata ingestion API

  1. Start by adding another HTTP step.

  2. Add the Timeout as 30s.

  3. Add this endpoint as URL https://app.harness.io/gateway/v1/catalog/custom-properties, read more about Catalog Metadata Ingestion API

  4. Select the Method as POST.

  5. And add the following json as Request Body

In the above body the openTicket which got created in JIRA will be added, to kind component and type service owned by the usergroup selected in the Workflows. Under entity_ref add the component link to which you want to add the ticket ID, the component link could be found using inspect entity for the component in catalog.

  1. Under Optional Configuration add the Assertion as <+httpResponseCode>==200.

  2. Under Headers add the following key value pairs:

    • Content-Type: application/json

    • Accept: */*

    • x-api-key: <+secrets.getValue('account.TOKEN_ID')>, add the token ID for your API key. Get your token ID from your Profile

  3. Now Apply Changes and SAVE the pipeline.

Create workflow

Now we have to create a workflow, which takes the input from the user and triggers the pipeline. Here is the workflow YAML

In the above YAML just replace the url with the pipeline URL we created above, also make sure the key values under inputset exactly matches with the pipeline variable names.

Also for Jira token input the input should be the personal access token from JIRA.

  1. Now go to your git provider and add this workflow yaml and save it, make sure it is public in case it is in private repo make sure you have the git integration setup.

  2. Once the file is created in your git repo, copy the full URL to the file. For example, https://github.com/harness-community/idp-samples/blob/main/tutorial-jira-ticket-catalog-ingestion.yaml.

  1. In the left navigation, select Create, and then select Register Software Component.

  1. Enter the URL to your new workflow.yaml.

  1. Click Import.

  1. Now go to the workflow and select the workflow you just added.

  1. Add the input values in the field that would be used by the workflow

Create additional info card

In case you want to display the same information you have ingested on your Overview page as an additional card, follow the steps below.

  1. Go to the Layout Page and under Configure and add the following for Services and Save it.

navigation-layout

  1. Now go to the Software Component in the Catalog and you will find an additional info card populated with information we ingested using the API above. You can read more about additional info card

Create Jira Project

Use RUN step

  1. Go to Configure in your IDP

  2. Now select the project where you want to create the pipeline for the Workflows.

  3. Begin by selecting the Create a Pipeline button followed by adding a name for the pipeline and set up your pipeline as inline.

  4. Now select the Developer Portal Stage and give it a name.

  5. Add a RUN step, name it as create jira project and select the Shell as Bash

  6. Now add the following under the Command.

The above given request body can create a Jira project

We have used few pipeline variables in the body, which will be used to take input from the IDP Workflows and for the users to choose project, add the summary, description for the tickets.

  1. Now under Optional Configuration add the Output Variables as PROJECT_KEY.

  2. Apply the Changes.

  3. Go to Variables on the right side of the page and add the following variables and have the input type as Runtime Input.

    • emailid

    • jiratoken

    • newprojectname

    • newprojectkey

    • projectleadaccountid

    • projecttemplatekey

    • usergroup

  4. Apply the changes.

Use catalog metadata ingestion API

  1. Start by adding another RUN step.

  2. Name it as Ingestion API and select the Shell as Bash

  3. Now add the following under the Command.

  1. Under header x-api-key: <+secrets.getValue('account.TOKEN_ID')>, add the token ID for your API key. Get your token ID from your Profile

In the above body the openTicket which got created in JIRA will be added, to kind component and type service owned by the usergroup selected in the Workflows. Under entity_ref add the component link to which you want to add the ticket ID, the unique entity reference could be found using inspect entity for the component in Catalog.

  1. Now Apply the changes.

  1. Go to Configure in your IDP

  2. Now select the project where you want to create the pipeline for the Workflows.

  3. Begin by selecting the Create a Pipeline button followed by adding a name for the pipeline and set up your pipeline as inline.

  4. Now select the Custom Stage and give it a name.

  5. Add a HTTP step and name it as create jira project

  6. Under the URL add this endpoint https://YOUR_COMPANY.atlassian.net/rest/simplified/latest/project and replace the YOUR_COMPANY with your domain name.

  7. Select the Method as POST.

  8. Under the body add the following:

The above given request body can create a Jira Project based on templatekey

We have used few pipeline variables in the body, which will be used to take input from the IDP Workflows and for the users to provide the name, key and template.

  1. Under Optional Configuration add the Assertion as <+httpResponseCode>==200.

  2. Under Headers add the following key value pairs:

    • Accept: application/json

    • Authorization: Basic <Base64 encoded username and password>, the username is the email-id and the password would be the JIRA API key

    • Accept-Encoding: gzip, deflate, br

  3. Under output add few variable likeproject_key, assign a value as expression <+json.object(httpResponseBody).projectKey> and project_name, assign the values as expression <+json.object(httpResponseBody).projectName>

  4. Now Apply Changes.

  5. Go to Variables on the right side of the page and add the following variables and have the input type as Runtime Input.

    • projectkey

    • templatekey

    • projectname

    • usergroup

  6. Apply the changes.

Use catalog metadata ingestion API

  1. Start by adding another HTTP step.

  2. Add the Timeout as 30s.

  3. Add this endpoint as URL https://app.harness.io/gateway/v1/catalog/custom-properties, read more about Catalog Metadata Ingestion API

  4. Select the Method as POST.

  5. And add the following json as Request Body

In the above body the openTicket which got created in JIRA will be added, to kind component and type service owned by the usergroup selected in the Workflows. Under entity_ref add the component link to which you want to add the ticket ID, the component link could be found using inspect entity for the component in Catalog.

  1. Under Optional Configuration add the Assertion as <+httpResponseCode>==200.

  2. Under Headers add the following key value pairs:

    • Content-Type: application/json

    • Accept: */*

    • x-api-key: <+secrets.getValue('account.TOKEN_ID')>, add the token ID for your API key. Get your token ID from your Profile

  3. Now Apply Changes and SAVE the pipeline.

Create workflow

Now we have to create a workflow, which takes the input from the user and triggers the pipeline. Here is the workflow YAML

In the above YAML just replace the url with the pipeline URL we created above, also make sure the key values under inputset exactly matches with the pipeline variable names.

  1. Now go to your git provider and add this workflow yaml and save it, make sure it is public in case it is in private repo make sure you have the git integration setup.

  2. Once the file is created in your git repo, copy the full URL to the file. For example, https://github.com/harness-community/idp-samples/blob/main/tutorial-jira-ticket-catalog-ingestion.yaml.

  1. In the left navigation, select Create, and then select Register Software Component.

  1. Enter the URL to your new workflow.yaml.

  1. Click Import.

  1. Now go to the workflow and select the workflow you just added.

  2. Add the values, under template key, make sure you add one of the mentioned template key, for eg., here we used com.pyxis.greenhopper.jira:gh-simplified-agility-kanban.

Create additional info card

In case you want to display the same information you have ingested on your Overview page as an additional card, follow the steps below.

  1. Go to the Layout Page and under Configure and add the following for Services and Save it.

navigation-layout

  1. Now go to the Software Component in the Catalog and you will find an additional info card populated with information we ingested using the API above. You can read more about additional info card

Create UI picker to dynamically values in workflows

In case you need to use the data present in your Catalog as an input for the Workflows, you can do so by using the EntityFieldPicker.

Here is an example workflow template that uses the jira projectName for the corresponding new service being created.

The ui:displayField fetches all the projectName names from the Catalog.

PreviousUsing a short-lived secret to trigger a service onboarding pipeline
NextRegister a Software Component in Catalog

Last updated

Was this helpful?