Dynamic Workflow Picker based on API Response

It is quite common to create a UI field in IDP Workflow forms which shows a static list options to the user. For example -

properties:
  repository_type:
    type: string
    title: Repository Type
    enum:
      - public
      - private

will generate the following

However, often times a static list does not provide much value to the user. Here are some of the use-cases -

• Choose your Repository
• Choose your Bitbucket Project
• Choose your Jenkins Pipeline
• Choose your Infrastructure Component
• Choose a Jira Project

As a creator of the workflow, you want to provide real-time options for these input fields for a better user experience. This would also allow you to expect the results in a consistent format and leave no room for user formatting errors.

This is where our Dynamic Workflow Picker comes into play. Let's see how it works.

The Workflow UI makes a request to our Backend Proxy which you can configure as a way to securely make requests to third party APIs and get a response. The UI Picker allows you to customize the response a bit and present it as a list in the dropdown.

How to create a Dynamic Workflow Picker

There are two steps to adding a dynamic workflow picker in Harness IDP.

1. Define a Backend Proxy so that requests from the UI can be populated with authorization headers and forwarded to the third party APIs.
2. Create the dynamic field in the workflow form, consuming the backend proxy and the API response.

Step 1: Create a Backend Proxy

The first step is to declare a new Backend Proxy so that the Workflow forms UI can make authenticated requests to our third party provider. Let's assume you are trying to make requests to GitHub.

Go to IDP Admin -> Plugins. Find the plugin called "Configure Backend Proxies".

Inside the plugin, you get three options (like any other IDP plugin configuration).

1. Declare a Backend Proxy endpoint and headers
2. Configure necessary secrets
3. Configure Delegate Proxy (in case the API is not publicly accessible, or the secret is on your infrastructure)

In order to set up a proxy to connect with GitHub APIs, you can add the following in the configuration YAML

proxy:
  endpoints:
    /github-api:
      target: https://api.github.com
      pathRewrite:
        /api/proxy/github-api/?: /
      headers:
        Authorization: token ${PROXY_GITHUB_TOKEN}

Here the github-api is the unique name of the endpoint of this Backend proxy. We will need it next.

The target should point to the API base URL of your 3rd party provider e.g. api.github.com, https://api.bitbucket.org/2.0 etc.

The pathRewrite is field used by the system to ensure the API requests are correctly rerouted. It needs to be of the format /api/proxy/<endpoint_name>/?: / as shown above.

In the headers you can add an Authorization header. Ensure you use a unique token name here as variables are GITHUB_TOKEN or BITBUCKET_TOKEN are system defined. The token name does not matter, as long as a secret is set up for the corresponding variable.

Hit "Save Configuration" and now our backend proxy is ready to use!

You can verify this endpoint by making requests to the proxy endpoint at https://idp.harness.io/{ACCOUNT_IDENTIFIER}/idp/proxy. For example in order to test the GitHub example above, you can make a request to

https://idp.harness.io/{ACCOUNT_IDENTIFIER}/idp/proxy/github-api/user

Here https://idp.harness.io/{ACCOUNT_IDENTIFIER}/idp/proxy/github-api/ can be seen exactly as https://api.github.com/. So all the endpoint paths on the GitHub API can be used after the proxy endpoint URL. You can learn more about how to consume Harness IDP APIs on our API Docs.

Step 2: Create the dropdown picker in Workflows form

Now that our Backend proxy is ready, it is time to create that dropdown picker. Here is a small example to start with

parameters:
  # ...
  properties:
    github_repo:
      type: string
      ui:field: SelectFieldFromApi
      ui:options:
        title: GitHub Repository
        description: Pick one of the GitHub Repositories
        placeholder: "Choose a Repository"
        path: proxy/github-api/users/OrkoHunter/repos
        valueSelector: full_name

Let us understand these properties in detail -

• ui:field - This has to be set to SelectFieldFromApi. This is the name of the UI Field Component responsible for rendering the API response based picker.
• title, description and placeholder are text fields responsible for telling the user what this field is about. See screenshot.
• path - This is the most important field and has to be of the format proxy/<endpoint-name>/<API-path>. Here the endpoint-name is github-api as we have defined above. And users/OrkoHunter/repos is the API endpoint path on api.github.com.
• valueSelector - This is an optional field. If the response of the API is an array of strings, then you do not need this field. If the response is an array of objects/map, then valueSelector can be used to point to the key in the object/map that needs to be used for the dropdown.

And that's it! We now have a Workflow dropdown where results are coming from an external API response.

Reference Docs

SelectFieldFromApi

Here is an elaborate example of what all properties are possible with the SelectFieldFromApi field.

properties:
  api-picker:
    type: string
    ui:field: SelectFieldFromApi

    ui:options:
      title: Title
      description: Description

      # The Path on the Harness IDP backend API and the parameters to fetch the data for the dropdown
      path: "proxy/proxy-endpoint/api-path"
      params:
        urlparamkey: "urlparamvalue"

      # (Optional) Renders the provided text as a placeholder value into the select box.
      placeholder: "Select from options"

      # (Optional) This field is used to point to the Array element inside the API response. It can be nested as shown here. If the response itself is an array, you can skip this.
      arraySelector: "object1.key1"

      # (Optional) If the response array is not an array of strings but rather an array of objects, valueSelector can be used to set the value of each selected item.
      valueSelector: "id"

      # (Optional) In case you want to show the user something else rather than the value itself, you can use the labelSelector for it.
      labelSelector: "name"

You can find the detailed docs on the project's README.

Parsing API Response using filters

Let's look at some different types of API responses and how to create a picker based on that using the arraySelector, valueSelector and labelSelector filters.

Case 1: The response is an array

["item1", "item2"]

This is the most straightforward case, and we do not need any of the additional filters here.

properties:
  api-picker:
    type: string
    ui:field: SelectFieldFromApi
    ui:options:
      title: Title
      description: Description
      path: "proxy/proxy-endpoint/api-path"

Case 2: The response is an array of objects

[
  {
    "id": "abc123",
    "name": "service A"
  },
  {
    "id": "abc124",
    "name": "service B"
  }
]

properties:
  api-picker:
    type: string
    ui:field: SelectFieldFromApi
    ui:options:
      title: Title
      description: Description
      path: "proxy/proxy-endpoint/api-path"
      # We need to show `name` to the users
      valueSelector: "name"

If you want to show the user the names of the service here, but want to store the ID of the selected service to be used in the Pipeline execution later on, you can do this using

properties:
  api-picker:
    type: string
    ui:field: SelectFieldFromApi
    ui:options:
      title: Title
      description: Description
      path: "proxy/proxy-endpoint/api-path"
      # We show the `name` to the users but store the `id` as the selected value
      valueSelector: "id"
      labelSelector: "name

Case 3: The response is a nested object of arrays

"data": {
  "property": "cities",
  "array": [
    {
      "id": "city123",
      "name": "City A"
    },
    {
      "id": "city124",
      "name": "City B"
    }
  ]
}

properties:
  api-picker:
    type: string
    ui:field: SelectFieldFromApi
    ui:options:
      title: Title
      description: Description
      path: "proxy/proxy-endpoint/api-path"
      # We need to point to the array inside the response
      arraySelector: "data.array"
      # We show the `name` to the users but store the `id` as the selected value
      valueSelector: "id"
      labelSelector: "name

Notes

Advanced processing the API response

If the filters here are not sufficient for your use case, and you require additional data processing of the response, then we recommend you setting up a Lambda function in your cloud provider or a lightweight backend to do this job. You can use your Backend Proxy and Delegate Proxy to communicate to your custom Lambda/Backend.