Set up an AWS VM build infrastructure

Set up a CI build infrastructure using AWS VMs.

This feature is planned to be deprecated at the end of January 2027 as we transition to our Unified Runner (Delegate 3.0), which merges the runner and delegate into a single component and introduces additional capabilities and improvements.

The current implementation will continue to be fully supported while the Unified Runner is introduced, and both will be available side-by-side during the transition period. We will share detailed guidance well ahead of the change, and customers will have sufficient time, tooling, and support to plan and complete their migration before the deprecation takes effect.

If you have any questions, please contact your account representative or Harness Support.

This topic describes how to use AWS VMs as Harness CI build infrastructure. To do this, you will create an Ubuntu VM and install a Harness Delegate and Drone VM Runner on it. The runner creates VMs dynamically in response to CI build requests. You can also configure the runner to hibernate AWS Linux and Windows VMs when they aren't needed.

This is one of several CI build infrastructure options. For example, you can also set up a Kubernetes cluster build infrastructure.

The following diagram illustrates a CI build farm using AWS VMs. The Harness Delegate communicates directly with your Harness instance. The VM runner maintains a pool of VMs for running builds. When the delegate receives a build request, it forwards the request to the runner, which runs the build on an available VM.

This is an advanced configuration. Before beginning, you should be familiar with:

• Using the AWS EC2 console and interacting with AWS VMs.

• Harness key concepts

• CI pipeline creation

• Harness Delegates

• Drone VM Runners and pools:

  • Drone documentation - VM runner overview

  • Drone documentation - Drone Pool

  • Drone documentation - Amazon drivers

  • GitHub repository - Drone runner AWS

Prepare the AWS EC2 instance

These are the requirements to configure the AWS EC2 instance. This instance is the primary VM where you will host your Harness Delegate and runner.

IMPORTANT

AWS Spot instances, of any kind, are not supported to use as self-managed build infrastructure.

Configure authentication for the EC2 instance

The recommended authentication method is an IAM role on the VM instance, but using IAM user and access key and secret (AWS secret) is also supported. It is best practice to use an IAM role over an access key and secret for security reasons.

1. Create or select an IAM role for the primary VM instance. This IAM role must have CRUD permissions on EC2. This role provides the runner with temporary security credentials to create VMs and manage the build pool. For details, go to the Amazon documentation on AmazonEC2FullAccess Managed policy.

2. If you plan to run Windows builds, You must add the AdministratorAccess policy to the IAM role associated with the access key and access secret.

3. If you haven't done so already, create an access key and secret for the IAM role.

Launch the EC2 instance

1. In the AWS EC2 Console, launch a VM instance that will host your Harness Delegate and runner. This instance must use an Ubuntu AMI that is t2.large or greater.

   The primary VM must be Ubuntu. The build VMs (in your VM pool) can be Ubuntu, AWS Linux, or Windows Server 2019 or higher. All machine images must have Docker installed.

2. Attach a key pair to your EC2 instance. Create a key pair if you don't already have one.

3. You don't need to enable Allow HTTP/HTTPS traffic.

Configure ports and security group settings

1. Create a Security Group in the EC2 console. You need the Security Group ID to configure the runner. For information on creating Security Groups, go to the AWS documentation on authorizing inbound traffic for your Linux instances.

2. In the Security Group's Inbound Rules, allow ingress on port 9079. This is required for security groups within the VPC.

3. In the EC2 console, go to your EC2 VM instance's Inbound Rules, and allow ingress on port 22.

4. If you want to run Windows builds and be able to RDP into your build VMs, you must also allow ingress on port 3389.

5. Allow ingress rules for port 3000 as well.

6. Outbound access to githubusercontent.com over 443, which is allowed by default in a typical security group.

7. Outbound access to googleapis.com over 443 to ship the task logs. Can be avoided by using the account setting "Account Settings"->"Default Settings"->"Continuous Integration->"Upload Logs via Harness".

8. Set up VPC firewall rules for the build instances on EC2.

Install Docker and attach IAM role

1. SSH into your EC2 instance.

2. Install Docker.

3. Install Docker Compose.

4. Attach the IAM role to the EC2 VM. For instructions, go to the AWS documentation on attaching an IAM role to an instance.

Use a custom Windows AMI

If you plan to use a custom Windows AMI in your AWS VM build farm, you must delete state.run-once from your custom AMI.

In Windows, sysprep checks if state.run-once exists at C:\ProgramData\Amazon\EC2Launch\state.run-once. If the file exists, sysprep doesn't run post-boot scripts (such as cloudinit, which is required for Harness VM build infrastructure). Therefore, you must delete this file from your AMI so it doesn't block the VM init script.

If C:\ProgramData\Amazon\EC2Launch\state.run-once is not found, run the following command instead:

C:\ProgramData\Amazon\EC2-Windows\Launch\Scripts\InitializeInstance.ps1 -Schedule

If you get an error about an unrecognized refreshenv command, you might need to install Chocolatey and add it to $profile to enable the refreshenv command.

Environment Variables

Optionally set the following environment variables:

Variable Name	Description	Default
HEALTH_CHECK_TIMEOUT	Integer. Set a time out (in minutes) for the health check. Works only for Mac and Linux. For example, HEALTH_CHECK_TIMEOUT=6 would set a 6 minute timeout.	3
HEALTH_CHECK_WINDOWS_TIMEOUT	Integer. Set a time out (in minutes) for the health check. Works only for Windows. For example, HEALTH_CHECK_WINDOWS_TIMEOUT=6 would set a 6 minute timeout.	5

Configure the Drone pool on the AWS VM

The pool.yml file defines the VM spec and pool size for the VM instances used to run the pipeline. A pool is a group of instantiated VMs that are immediately available to run CI pipelines. You can configure multiple pools in pool.yml, such as a Windows VM pool and a Linux VM pool. To avoid unnecessary costs, you can configure pool.yml to hibernate VMs when not in use.

1. Create a /runner folder on your delegate VM and cd into it:

   mkdir /runner
   cd /runner

2. In the /runner folder, create a pool.yml file.

3. Modify pool.yml as described in the following example and the Pool settings reference.

Example pool.yml

The following pool.yml example defines both an Ubuntu pool and a Windows pool.

version: "1"
instances:
  - name: ubuntu-ci-pool ## The settings nested below this define the Ubuntu pool.
    default: true
    type: amazon
    pool: 1
    limit: 4
    platform:
      os: linux
      arch: amd64
    spec:
      account:
        region: us-east-2 ## To minimize latency, use the same region as the delegate VM.
        availability_zone: us-east-2c ## To minimize latency, use the same availability zone as the delegate VM.
        access_key_id: XXXXXXXXXXXXXXXXX # Optional if using an IAM role
        access_key_secret: XXXXXXXXXXXXXXXXXXX # Optional if using an IAM role
        key_pair_name: XXXXX
      ami: ami-xxx ## Ubuntu Amd64 AMI must be passed
      size: t2.nano
      iam_profile_arn: arn:aws:iam::XXXX:instance-profile/XXXXX
      network:
        security_groups:
          - sg-XXXXXXXXXXX
  - name: windows-ci-pool ## The settings nested below this define the Windows pool.
    default: true
    type: amazon
    pool: 1
    limit: 4
    platform:
      os: windows
    spec:
      account:
        region: us-east-2 ## To minimize latency, use the same region as the delegate VM.
        availability_zone: us-east-2c ## To minimize latency, use the same availability zone as the delegate VM.
        access_key_id: XXXXXXXXXXXXXXXXXXXXXX
        access_key_secret: XXXXXXXXXXXXXXXXXXXXXX
        key_pair_name: XXXXX
      ami: ami-xxx ## Windows AMI with Docker Installed should be passed
      size: m5.large
      hibernate: true
      disk:
        size: 60 ## Min Size Generally Required for Windows based AMI with Docker Installed , The size mentioned is in GBs
      network:
        security_groups:
          - sg-XXXXXXXXXXXXXX

Pool settings reference

You can configure the following settings in your pool.yml file. You can also learn more in the Drone documentation for the Pool File and Amazon drivers.

Setting	Type	Example	Description
name	String	name: windows_pool	Unique identifier of the pool. You will need to specify this pool name in Harness when you set up the CI stage build infrastructure.
pool	Integer	pool: 1	Warm pool size number. Denotes the number of VMs in ready state to be used by the runner.
limit	Integer	limit: 3	Maximum number of VMs the runner can create at any time. pool indicates the number of warm VMs, and the runner can create more VMs on demand up to the limit.
For example, assume pool: 3 and limit: 10. If the runner gets a request for 5 VMs, it immediately provisions the 3 warm VMs (from pool) and provisions 2 more, which are not warm and take time to initialize.
platform	Key-value pairs, strings	Go to platform example.	Specify VM platform operating system (os: linux or os: windows). arch and variant are optional. os_name: amazon-linux is required for AL2 AMIs. The default configuration is os: linux and arch: amd64.
spec	Key-value pairs, various	Go to Example pool.yml and the examples in the following rows.	Configure settings for the build VMs and AWS instance. Contains a series of individual and mapped settings, including account, tags, ami, size, hibernate, iam_profile_arn, network, user_data, user_data_path, and disk. Details about these settings are provided below.
account	Key-value pairs, strings	Go to account example.	AWS account configuration, including region and access key authentication. region (required): AWS region. To minimize latency, use the same region as the delegate VM. availability_zone (optional): AWS region availability zone. To minimize latency, use the same availability zone as the delegate VM. access_key_id: The AWS access key for authentication. If using an IAM role, this is the access key associated with the IAM role. access_key_secret: The secret associated with the specified access_key_id. key_pair_name: The key pair name specified when you set up the EC2 instance. Don't include .pem.
tags	Key-value pairs, strings	Go to tags example.	Optional tags to apply to the instance.
ami	String	ami: ami-092f63f22143765a3	The AMI ID. You can use the same AMI as your EC2 instance or search for AMIs in your Availability Zone for supported models (Ubuntu, AWS Linux, Windows 2019+). AMI IDs differ by Availability Zone.
size	String	size: t3.large	The AMI size, such as t2.nano, t2.micro, m4.large, and so on. Make sure the size is large enough to handle your builds.
hibernate	Boolean	hibernate: true	When set to true (which is the default), VMs hibernate after startup. When false, VMs are always in a running state. This option is supported for AWS Linux and Windows VMs. Hibernation for Ubuntu VMs is not currently supported. For more information, go to the AWS documentation on hibernating on-demand Linux instances.
iam_profile_arn	String	iam_profile_arn: arn:aws:iam::XXXX:instance-profile/XXX	If using IAM roles, this is the instance profile ARN of the IAM role to apply to the build instances.
network	Key-value pairs, various	Go to network example.	AWS network information, including security groups. For more information on these attributes, go to the AWS documentation on creating security groups. security_groups: List of security group IDs as strings. vpc: If using VPC, this is the VPC ID as an integer. vpc_security_groups: If using VPC, this is a list of VPC security group IDs as strings. private_ip: Boolean. subnet_id: The subnet ID as a string.
user_data or user_data_path	Key-value pairs, strings	Go to user data example.	Define custom user data to apply to the instance. Provide cloud-init data in either user_data_path or user_data if you need custom configuration.
disk	Key-value pairs, various	Go to disk example.	Optional AWS block information. size: Integer, size in GB. type: gp2, io1, or standard. iops: If type: io1, then iops: iops. kms_key_id: Your AWS KMS Key ID

platform example

instance:
  platform:
    os: linux
    arch: amd64
    version:
    os_name: amazon-linux

account example

account:
  region: us-east-2
  availability_zone: us-east-2c
  access_key_id: XXXXX
  access_key_secret: XXXXX
  key_pair_name: XXXXX

tags example

tags:
  owner: USER
  ttl: "-1"

network example

network:
  private_ip: true
  subnet_id: subnet-XXXXXXXXXX
  security_groups:
    - sg-XXXXXXXXXXXXXX

user data example

Provide cloud-init data in either user_data_path or user_data if you need custom configuration. Refer to the user data examples for supported runtime environments.

Below is a sample pool.yml for GCP with user_data configuration:

version: "1"
instances:
  - name: linux-amd64
    type: google
    pool: 1
    limit: 10
    platform:
      os: linux
      arch: amd64
    spec:
      account:
        project_id: YOUR_PROJECT_ID
        json_path: PATH_TO_SERVICE_ACCOUNT_JSON
      image: IMAGE_NAME_OR_PATH
      machine_type: e2-medium
      zones:
        - YOUR_GCP_ZONE # e.g., us-central1-a
      disk:
        size: 100
      user_data: |
        #cloud-config
        {{ if and (.IsHosted) (eq .Platform.Arch "amd64") }}
        packages: []
        {{ else }}
        apt:
          sources:
            docker.list:
              source: deb [arch={{ .Platform.Arch }}] https://download.docker.com/linux/ubuntu $RELEASE stable
              keyid: 9DC858229FC7DD38854AE2D88D81803C0EBFCD88
        packages: []
        {{ end }}
        write_files:
          - path: {{ .CaCertPath }}
            path: {{ .CertPath }}
            permissions: '0600'
            encoding: b64
            content: {{ .TLSCert | base64 }}
          - path: {{ .KeyPath }}
        runcmd:
          - 'set -x'
          - |
            if .ShouldUseGoogleDNS; then
              echo "DNS=8.8.8.8 8.8.4.4\nFallbackDNS=1.1.1.1 1.0.0.1\nDomains=~." | sudo tee -a /etc/systemd/resolved.conf
              systemctl restart systemd-resolved
            fi
          - ufw allow 9079

disk example

disk:
  size: 16
  type: io1
  iops: iops
  kms_key_id: arn:aws:kms:us-west-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab
  tags:
    volumekey: volumeValue
    key1: value1

The tags property exemplified above follows a key/value pair format. This will add tags to the disk/volume directly.

MINIMUM VERSION

In order to tag the disk, please ensure you are using drone runner version of 1.0.0-rc.190 or newer.

Start the runner

SSH into your EC2 instance and run the following command to start the runner:

docker run -v /runner:/runner -p 3000:3000 drone/drone-runner-aws:latest  delegate --pool /runner/pool.yml

This command mounts the volume to the Docker runner container and provides access to pool.yml, which is used to authenticate with AWS and pass the spec for the pool VMs to the container. It also exposes port 3000.

You might need to modify the command to use sudo and specify the runner directory path, for example:

sudo docker run --network host -v ./runner:/runner -p 3000:3000 drone/drone-runner-aws:latest  delegate --pool /runner/pool.yml

WHAT DOES THE RUNNER DO?

When a build starts, the delegate receives a request for VMs on which to run the build. The delegate forwards the request to the runner, which then allocates VMs from the warm pool (specified by pool in pool.yml) and, if necessary, spins up additional VMs (up to the limit specified in pool.yml).

The runner includes lite engine, and the lite engine process triggers VM startup through a cloud init script. This script downloads and installs Scoop package manager, Git, the Drone plugin, and lite engine on the build VMs. The plugin and lite engine are downloaded from GitHub releases. Scoop is downloaded from get.scoop.sh which redirects to raw.githubusercontent.com.

Firewall restrictions can prevent the script from downloading these dependencies. Make sure your images don't have firewall or anti-malware restrictions that are interfering with downloading the dependencies. For more information, go to Troubleshooting.

Install the delegate

Install a Harness Docker Delegate on your AWS EC2 instance.

1. In Harness, go to Account Settings, select Account Resources, and then select Delegates.

   You can also create delegates at the project scope. In your Harness project, select Project Settings, and then select Delegates.

2. Select New Delegate or Install Delegate.

3. Select Docker.

4. Enter a Delegate Name.

5. Copy the delegate install command and paste it in a text editor.

6. To the first line, add --network host, and, if required, sudo. For example:

   sudo docker run --cpus=1 --memory=2g --network host

7. SSH into your EC2 instance and run the delegate install command.

The delegate install command uses the default authentication token for your Harness account. If you want to use a different token, you can create a token and then specify it in the delegate install command:

1. In Harness, go to Account Settings, then Account Resources, and then select Delegates.

2. Select Tokens in the header, and then select New Token.

3. Enter a token name and select Apply to generate a token.

4. Copy the token and paste it in the value for DELEGATE_TOKEN.

For more information about delegates and delegate installation, go to Delegate installation overview.

Verify connectivity

1. Verify that the delegate and runner containers are running correctly. You might need to wait a few minutes for both processes to start. You can run the following commands to check the process status:

   docker ps
   docker logs DELEGATE_CONTAINER_ID
   docker logs RUNNER_CONTAINER_ID

2. In the Harness UI, verify that the delegate appears in the delegates list. It might take two or three minutes for the Delegates list to update. Make sure the Connectivity Status is Connected. If the Connectivity Status is Not Connected, make sure the Docker host can connect to https://app.harness.io.

The delegate and runner are now installed, registered, and connected.

Specify build infrastructure

Configure your pipeline's Build (CI) stage to use your AWS VMs as build infrastructure.

Visual

1. In Harness, go to the CI pipeline that you want to use the AWS VM build infrastructure.

2. Select the Build stage, and then select the Infrastructure tab.

3. Select VMs.

4. Enter the Pool Name from your pool.yml.

5. Save the pipeline.

YAML

    - stage:
        name: build
        identifier: build
        description: ""
        type: CI
        spec:
          cloneCodebase: true
          infrastructure:
            type: VM
            spec:
              type: Pool
              spec:
                poolName: POOL_NAME_FROM_POOL_YML
                os: Linux
          execution:
            steps:
            ...

Delegate selectors with self-managed VM build infrastructures

Currently, delegate selectors for self-managed VM build infrastructures is behind the feature flag CI_ENABLE_VM_DELEGATE_SELECTOR. Contact Harness Support to enable the feature.

Although you must install a delegate to use a self-managed VM build infrastructure, you can choose to use a different delegate for executions and cleanups in individual pipelines or stages. To do this, use pipeline-level delegate selectors or stage-level delegate selectors.

Delegate selections take precedence in the following order:

1. Stage

2. Pipeline

3. Platform (build machine delegate)

This means that if delegate selectors are present at the pipeline and stage levels, then these selections override the platform delegate, which is the delegate that you installed on your primary VM with the runner. If a stage has a stage-level delegate selector, then it uses that delegate. Stages that don't have stage-level delegate selectors use the pipeline-level selector, if present, or the platform delegate.

For example, assume you have a pipeline with three stages called alpha, beta, and gamma. If you specify a stage-level delegate selector on alpha and you don't specify a pipeline-level delegate selector, then alpha uses the stage-level delegate, and the other stages (beta and gamma) use the platform delegate.

Early access feature: Use delegate selectors for codebase tasks

Currently, delegate selectors for CI codebase tasks is behind the feature flag CI_CODEBASE_SELECTOR. Contact Harness Support to enable the feature.

By default, delegate selectors aren't applied to delegate-related CI codebase tasks.

With this feature flag enabled, Harness uses your delegate selectors for delegate-related codebase tasks. Delegate selection for these tasks takes precedence in order of pipeline selectors over connector selectors.

AWS Fargate Limitations

If you are running builds on AWS Fargate, please be aware of the following limitations.

Docker delegate on AWS ECS Fargate backed instance

When operating ECS delegates on AWS Fargate, it's critical to note that AWS Fargate will terminate the delegate if the tasks running on the delegate exceed the infrastructure's specified limits. This is a limitation inherent in using infrastructure not owned by the customer. Harness Delegate cannot circumvent this restriction. However, ECS delegates operating on an EC2 instance do not have this issue. To avoid this limitation, consider using Kubernetes delegates where the infrastructure and associated YAML definitions address these issues.

Docker in Docker does not work with AWS Fargate

AWS Fargate doesn't support the use of privileged containers. Privileged mode is required for DinD, thus, you cannot use DinD with AWS Fargate.

AWS Fargate does not support IAM roles

Amazon requires the Amazon EKS Pod execution role to run pods on the AWS Fargate infrastructure. For more information, go to Amazon EKS Pod execution IAM role in the AWS documentation.

If you deploy pods to Fargate nodes in an EKS cluster, and your nodes needs IAM credentials, you must configure IRSA in your AWS EKS configuration (and then select the Use IRSA option for your connector credentials in Harness). This is due to Fargate limitations.

Troubleshoot AWS VM build infrastructure

• Optimize Windows VM runner

• Build VM creation fails with no default VPC

• AWS VM builds stuck at the initialize step on health check

• Delegate connected but builds fail

• Use internal or custom AMIs

• Where can I find self-managed VM lite engine and cloud init output logs?

• Can I use the same build VM for multiple CI stages?

• Why are build VMs running when there are no active builds?

• How do I specify the disk size for a Windows instance in pool.yml?

• Clone codebase fails due to missing plugin

• Can I limit memory and CPU for Run Tests steps running on self-managed VM build infrastructure?

Go to the CI Knowledge Base for a broader list of frequently asked questions and answers.