Ctrlk
Sign UpGet a DemoContact Us
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Getting started with service onboarding pipeline (using Custom Stage)

Create a basic service onboarding pipeline in Harness IDP

This tutorial is designed to help a platform engineer to get started with Harness IDP. We will create a basic service onboarding pipeline that uses a software Workflow and provisions a Next.js application for a developer. After you create the software Workflow, developers can choose the Workflow on the Create page and enter details such as a name for the application and the path to their Git repository. The service onboarding pipeline creates a hello world repository for storing code.

Your users (developers) must perform a sequence of tasks to create the application. First, they interact with a software Workflow. A software Workflow is a form that collects a user's requirements. After a user submits the form, IDP executes a Harness pipeline that onboard the new service. Usually the pipeline fetches a hello-world skeleton code, creates a new repository, and interacts with third-party providers such as cloud providers, Jira, and Slack.

Before you begin

Before you begin this tutorial, make sure that you fulfil the following requirements:

  • Enable Harness IDP for your account.

  • Obtain a CI or CD license if you do not have one. This is a temporary requirement.

Create a pipeline

Begin by creating a pipeline for onboarding the service.

Create a build or custom stage

To create a Build or Custom stage, perform the following steps:

  1. In the sidebar of the Harness application, select Projects, and then select a project.

You can also create a new project for the service onboarding pipelines. Eventually, all the users in your account should have permissions to execute the pipelines in this project. For information about creating a project, go to Create organizations and projects.

  1. Select Pipelines, and then select Create a Pipeline.

  2. In Pipeline Studio, select Add Stage.

  3. In Select Stage Type, select Custom Stage. (If you have a Harness CI license, you could also use the Build stage type. However, for this tutorial, we recommend that you use the Custom Stage type.)

  4. In Stage Name, enter a name for the stage, and then click Set Up Stage.

  5. Select Add step, and then, in the menu that appears, select Add Step. A sidebar with available steps is displayed.

  6. Select Container Step to run a Python CLI called cookiecutter. We need a publicly available Python image for this purpose. You can use Container Step for any such project generators (for example, yeoman).

    NOTE

    In the CI or Build stage type, container step is named Run, and it has the same functionality.

  7. Configure the step as follows:

    1. Enter a name for the step. For example, name it Create React app.

    2. You can enter 10m (10 minutes) in the Timeout field.

    3. In Container Registry, create or choose an anonymous Docker connector that connects to DockerHub (https://registry.hub.docker.com/v2/).

    4. In Image, enter python.

    Before we write the command, we must make an infrastructure choice, which means that we specify where the pipeline executes. You can execute the pipeline on your own infrastructure or on the Harness platform. If you have an existing delegate set up for deployments, you can use the associated connector and specify its Kubernetes namespace. If you want to use the Harness platform, you have to use the CI or Build stage type instead of the Custom stage type and choose the Harness platform as your infrastructure.

    NOTE

    Depending upon our operation, we might have to adjust the memory limit of the container. If required, you can change Limit Memory from 500Mi to 4000Mi.

Cookiecutter scripts based on your SCM

  1. Paste the following cookiecutter-based script into Command.

    The script performs the following tasks:

    1. Generates a basic Next.js app.

    2. Creates a repository with the contents. The sample code used in the command is available here, which essentially is a cookiecutter project. You can choose from available cookiecutter projects or create your own project from scratch.

In the above script you need to add the Personal Access Token to make the API call to register your catalog,

  1. Click Apply Changes.

Manage variables in the pipeline

The script uses several pipeline variables. The variables are as follows:

  • <+pipeline.variables.project_name>

  • <+pipeline.variables.github_username>

  • <+pipeline.variables.github_token>

  • <+pipeline.variables.github_org>

  • <+pipeline.variables.github_repo>

Except for the secrets all the variables should have a runtime input type and the variable name should match with the parameter name used in the Workflow as the values would be pre-populated from the values entered as input in the below IDP Workflow.

For eg: <+pipeline.variables.project_name> variable is pre-populated by project_name: ${{ parameters.project_name }} under input set: in the below given Workflow.

  1. Paste the following cookiecutter-based script into Command.

    The script performs the following tasks:

    1. Generates a basic Next.js app.

    2. Creates a repository with the contents. The sample code used in the command is available here, which essentially is a cookiecutter project. You can choose from available cookiecutter projects or create your own project from scratch.

  2. Click Apply Changes.

    Manage variables in the pipeline

    The script uses several pipeline variables. The variables are as follows:

  • <+pipeline.variables.project_name>

  • <+pipeline.variables.gitlab_username>

  • <+pipeline.variables.gitlab_token>

  • <+pipeline.variables.gitlab_org>

  • <+pipeline.variables.gitlab_repo>

Except for the secrets all the variables should have a runtime input type and the variable name should match with the parameter name used in the Workflow as the values would be pre-populated from the values entered as input in the below IDP Workflow.

For e.g., : <+pipeline.variables.project_name> variable is pre-populated by project_name: ${{ parameters.project_name }} under input set: in the below given Workflow.

You can use the Variables button on the floating sidebar on the right-hand side to open the Variables page for the pipeline.

You can create any number of pipeline variables and decide their value type. Some variables, such as a GitHub token, a username, and organization, can have a fixed value. The token used in the code above is a Harness secret whose value is decoded during pipeline execution.

Variables such as project name and GitHub repository are runtime inputs. They are needed at the time of pipeline execution. When creating a new variable, you can specify its type in the UI. For more information about reference variables, go to the reference documentation on pipeline variables.

Create a software workflow definition in IDP

Now that our pipeline is ready to execute when a project name and a GitHub repository name are provided, let us create the UI counterpart of it in IDP. This is powered by the Backstage Software Workflow. Create a Workflow.yaml file anywhere in your Git repository. Usually, that would be the same place as your skeleton hello world code. We use the react-jsonschema-form playground to build the Workflow. Nunjucks is templating engine for the IDP Workflows.

Source

Source

This YAML code is governed by Backstage. You can change the name and description of the software Workflow. The Workflow has the following parts:

  1. Input from the user

  2. Execution of pipeline

Let us take a look at the inputs that the Workflow expects from a developer. The inputs are written in the spec.parameters field. It has two parts, but you can combine them. The keys in properties are the unique IDs of fields (for example, github_repo and project_name). If you recall, they are the pipeline variables that we set as runtime inputs earlier. This is what we want the developer to enter when creating their new application.

The YAML definition includes fields such as cloud provider and database choice. They are for demonstration purposes only and are not used in this tutorial.

Add the owner

By default the owner is of type Group which is same as the User Group in Harness. In case the owner is a user you have to mention it as user:default/debabrata.panigrahi, and it should only contain the username not the complete email ID.

Authenticate the request

Once you have written all the inputs that the Workflow requires, you must add the following YAML snippet under spec.parameters.properties.

The token property we use to fetch Harness Auth Token is hidden on the Review Step using ui:widget: password, but for this to work the token property needs to be mentioned under the first page in case you have multiple pages.

Also the token input is used as a parameter under steps as apikey

This is a custom component we created to authenticate the call to execute the pipeline on the basis of the logged-in user's credentials.

Action to trigger the pipeline

The Workflow actions currently supports only custom stage and codebase disabled CI stage with Run step, also all input, except for pipeline input as variables, must be of fixed value.

The spec.steps field contains only one action, and that is to trigger a Harness pipeline. Update the url and replace it with the URL of your service onboarding pipeline. Also, ensure that the inputset is correct, and it contains all the runtime input variables that the pipeline needs.

Register the workflow

Use the URL to the workflow.yaml created above and register it by using the same process for registering a new software component.

Now navigate to the Create page in IDP. You will see the newly created Workflow appear. Try it out!

Unregister/Delete workflow

  1. Navigate to the Catalog page, and select Template under Kind.

  2. Select the Workflow Name you want to Unregister.

  3. Now on the Workflow overview page, click on the 3 dots on top right corner and select Unregister Entity.

  4. Now on the Dialog box select Unregister Location.

  5. This will delete the Workflow.

Extended reading

  1. How to add conditional Inputs in Workflow?

  2. How to upload a file in a Workflow?

  3. How to ingest data dynamically into Workflows?

PreviousUse Dynamic Pickers for a Pull Request Creator Workflow
NextProvision Infrastructure using IDP and IaCM

Last updated

Was this helpful?