Dynamic execution of Pipelines
Dynamic execution of Pipelines allows you to execute pipelines by providing pipeline YAML configuration during runtime without requiring pre-saved configurations in Harness. Essentially, Harness is being used as a pure execution engine, while an external system within the ecosystem handles YAML creation and editing. This is particularly useful when pipelines are dynamically generated based on user actions. At the same time, it enables seamless integration with the existing ecosystem while maintaining Harness's robust security and governance controls.
Pre-Requisites
Account and Pipeline Level Setting
Currently, this setting is behind the Feature Flag PIPE_DYNAMIC_PIPELINES_EXECUTION. Contact Harness Support to enable this Feature Flag.
Before you can execute pipelines dynamically, you need to enable two settings at both the Account and Pipeline levels.
- Account Level Setting
Navigate to Account Setting -> Default Settings -> Pipeline -> Enable Allow Dynamic Execution for Pipelines
If you need to disable dynamic execution for the account, you can do so at any time by simply turning off this setting.
- Pipeline Level Setting
To execute a pipeline dynamically, the pipeline must already exist in the UI. The pipeline-level setting for dynamic execution will only be visible once the Account-Level Setting is enabled.
After enabling the account setting, you must enable the pipeline-level setting in your pipeline:
Navigate to Advanced Options -> Dynamic Execution Settings (optional) -> Enable Allow Dynamic Execution for Pipeline
Permissions
Users need both Edit and Execute permissions on the Pipelines.
YAML Configuration
Dynamic execution of pipelines is only supported via the API and not through the UI.
To execute a pipeline dynamically, you must provide a valid YAML configuration through the Dynamic Execution API.
Dynamic Execution API
The Dynamic Execution API in Harness allows you to execute a pipeline dynamically by passing YAML configurations directly in the request body.
curl --location 'https://app.harness.io/gateway/pipeline/api/v1/orgs/default/projects/PROJECT_ID/pipelines/PIPELINE_ID/execute/dynamic' \
--header 'accept: */*' \
--header 'content-type: application/json' \
--header 'origin: https://app.harness.io' \
--header 'Harness-Account: ACCOUNT_ID' \
--header 'x-api-key: HARNESS_API_KEY' \
--data '{
"yaml": ""
}'
YAML Format Requirement
To execute a pipeline dynamically, the YAML should follow the required format as shown in the example below.
Example
curl --location 'https://app.harness.io/gateway/pipeline/api/v1/orgs/default/projects/Krishika_CD_Samples/pipelines/Deploy_Web_Application/execute/dynamic' \
--header 'accept: */*' \
--header 'content-type: application/json' \
--header 'origin: https://app.harness.io' \
--header 'Harness-Account: ACCOUNT_ID' \
--header 'x-api-key: HARNESS_API_KEY' \
--data '{
"yaml": "pipeline:\n name: Deploy_Web_Application\n identifier: Deploy_Web_Application\n projectIdentifier: Krishika_CD_Samples\n orgIdentifier: default\n tags: {}\n stages:\n - stage:\n name: Build_Web_App\n identifier: Build_Web_App\n description: \"\"\n type: CI\n spec:\n cloneCodebase: false\n caching:\n enabled: true\n buildIntelligence:\n enabled: true\n platform:\n os: Linux\n arch: Amd64\n runtime:\n type: Cloud\n spec: {}\n execution:\n steps:\n - step:\n type: Run\n name: Run_1\n identifier: Run_1\n spec:\n shell: Sh\n command: echo \"Building Web App\"\n - stage:\n name: Deploy_Web_App\n identifier: Deploy_Web_App\n description: \"\"\n type: Deployment\n spec:\n deploymentType: Kubernetes\n service:\n serviceRef: k8s_service\n serviceInputs:\n serviceDefinition:\n type: Kubernetes\n spec:\n artifacts:\n primary:\n primaryArtifactRef: <+input>\n sources: <+input>\n environment:\n environmentRef: k8sdemoenv\n deployToAll: false\n infrastructureDefinitions:\n - identifier: k8sdemoinfra\n execution:\n steps:\n - step:\n name: Rollout Deployment\n identifier: rolloutDeployment\n type: K8sRollingDeploy\n timeout: 10m\n spec:\n skipDryRun: false\n pruningEnabled: false\n - step:\n type: ShellScript\n name: ShellScript_1\n identifier: ShellScript_1\n spec:\n shell: Bash\n executionTarget: {}\n source:\n type: Inline\n spec:\n script: echo \"Web App Deployed\"\n environmentVariables: []\n outputVariables: []\n timeout: 10m\n rollbackSteps:\n - step:\n name: Rollback Rollout Deployment\n identifier: rollbackRolloutDeployment\n type: K8sRollingRollback\n timeout: 10m\n spec:\n pruningEnabled: false\n tags: {}\n failureStrategies:\n - onFailure:\n errors:\n - AllErrors\n action:\n type: StageRollback"
}'
Upon successful execution, the API returns the following response:
{
"execution_details": {
"execution_id": "xD908VCSQVaP3Zo14tEI8g",
"status": "RUNNING"
}
}
Once the API is triggered, you can monitor the pipeline execution in the Harness UI.
In the trigger summary you will see a message indicating This was executed dynamically. This confirms that the pipeline was triggered using the Dynamic Execution API.
What's Supported
Dynamically executing pipelines support usage of following features:-
- Template resolution at runtime
- Expression resolution
- Services and environments
- Secrets and connectors
- OPA policy enforcement
Dynamic Stage Generation/Execution
If you wish to generate pipeline stages/steps on the fly (when the pipeline is running) and append them to the executing pipeline, you can use a Dynamic Stage like the pipeline shown below.
You will notice that this stage has a "Source Location" option as shown above. This source location should point to the dynamically generated YAML that is being generated as the pipeline runs. In the example below, the stage before the dynamic stage is generating pipeline YAML and then outputting it to the variable that the dynamic stage is consuming to run the new pipeline.
Example Pipeline with Dynamic Stage
This example demonstrates how to transform and execute dynamically generated YAML within a running pipeline:
pipeline:
name: Dynamic Stage Example
identifier: Dynamic_Stage_Example
projectIdentifier: <PROJECT_ID>
orgIdentifier: <ORG_ID>
stages:
- stage:
name: generate_yaml
identifier: generate_yaml
description: ""
type: CI
spec:
cloneCodebase: true
caching:
enabled: true
override: true
paths: []
buildIntelligence:
enabled: true
infrastructure:
type: KubernetesDirect
spec:
connectorRef: <K8S_CONNECTOR>
namespace: default
automountServiceAccountToken: true
nodeSelector: {}
os: Linux
execution:
steps:
- step:
type: Plugin
name: Generate_Pipeline_YAML
identifier: Generate_Pipeline_YAML
spec:
connectorRef: <DOCKER_CONNECTOR>
image: <PLUGIN_IMAGE>:<IMAGE_TAG>
settings:
source_yaml: <SOURCE_YAML_PATH>
options: "--repo-connector <REPO_CONNECTOR> --docker-connector <DOCKER_CONNECTOR> --build-branch <BUILD_BRANCH>"
- stage:
name: execute_generated_yaml
identifier: execute_generated_yaml
description: ""
type: Dynamic
spec:
source: <+pipeline.stages.generate_yaml.spec.execution.steps.Generate_Pipeline_YAML.output.outputVariables.PLUGIN_HARNESS_YAML>
tags: {}
properties:
ci:
codebase:
connectorRef: <REPO_CONNECTOR>
build: <+input>
sparseCheckout: []
resources:
limits:
cpu: 1600m
memory: 2000Mi
tags: {}
In this example:
- The
generate_yamlstage uses a plugin to generate Harness pipeline YAML dynamically - The plugin outputs the generated YAML to the variable
PLUGIN_HARNESS_YAML - The
execute_generated_yamldynamic stage consumes this output variable via thesourcefield - When the pipeline executes, the dynamic stage appends and runs the newly generated stages/steps
Limitations
Dynamically executing pipelines doesn't support following features:-
- Input sets
- Selective stage Execution
- Retry/Re-Run capability
- Automatic Triggers
- Post Production Rollback
- Runtime Inputs