Skip to content
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Deployment using Terraform

The Terraform module in this topic deploys the AWS CloudWatch agent into your AWS account. You choose whether the agent runs on an EC2 instance or as ECS Fargate tasks, and you choose which collectors it runs: API Gateway, Bedrock, AgentCore, or any mix of the three.

What will you learn in this topic?

By the end of this topic, you will understand:

  • The steps to prepare your AWS services and credentials for the agent.

  • The steps to configure and apply the Terraform module.

  • The purpose of each Terraform variable that you are likely to set.

  • The way to verify, change, and remove the deployment.


Before you begin

  • Make sure that Terraform is installed on the machine where you will run the module, because you apply the module's configuration files from that machine.

  • Make sure that the AWS CLI is configured with credentials that can deploy the resources in the target account and region. Step 2 covers this.

  • Make sure that you know which collectors you want. You must turn on at least one, or terraform apply stops with an error before it creates anything.

  • Make sure that you have a TPA set up. The agent sends all its data to the TPA, so follow the TPA installation guide first, then note the TPA's URL. The URL must be reachable from the place where the agent runs. You enter it as traceable_agent_endpoint in Step 6.

  • Make sure that you have a refresh token for your Harness account. In ec2 mode, the module passes it as traceable_refresh_token so that the instance can connect to Harness.

  • Make sure that API Gateway logging, Bedrock model-invocation logging, or AgentCore observability is turned on for the services you want to collect. Step 1 covers all three.


Understand the deployment modes

The deployment_mode variable decides where the agent runs. Pick it first, because the two modes use different variables.

The following table describes the two modes, what Terraform creates in each, and when to choose one over the other.

Mode
What Terraform Creates
Use It When

ec2 (default)

An EC2 instance in the subnet you specify, with the agent running on it

You want the agent to run on one instance and you do not want to manage a container cluster

ecs

Resources that start the agent as an ECS task on a schedule, through Amazon EventBridge

You already run an ECS Fargate cluster and want the agent to run as a task

In ecs mode, you must supply an existing ECS Fargate cluster through cluster_arn. The module does not create the cluster.


Step 1: Prepare your AWS services

The agent does not create logs. It reads logs that your AWS services already write to CloudWatch Logs, so each service you want to collect from has to be logging first. The three collectors read different logs, so the preparation differs for each. API Gateway needs access logs in a specific JSON format. Bedrock needs model invocation logging enabled, plus the name of its log group, in your agent configuration. AgentCore needs observability turned on and nothing more.

Do only the parts for the collectors you plan to turn on. If you turn on more than one, do each part.

API Gateway (REST APIs)

Use this section if the API Gateway collector will read REST APIs. The collector reads each stage's access logs, so every stage you want to collect from must write them in a format the agent can parse.

Complete the following for every stage you want to collect from:

  1. Open the API Gateway console and select your API, then select the stage.

  2. Go to Logs/tracing.

  3. In the CloudWatch Logs drop-down, select Error and info logs.

  4. Turn on Custom Access Logging.

  5. In Access log destination ARN, enter the ARN of a CloudWatch log group. This is where API Gateway writes the access logs that the agent reads.

  6. In Log Format, paste the following JSON:

Each field maps to an API Gateway $context variable. API Gateway fills the variables in for every request, so each log line carries the request ID, the caller's IP address, the HTTP method, the path, the status code, the response size, and the domain name. The agent expects these fields in every log line, so leave none out.

API Gateway (HTTP APIs)

Use this section if the API Gateway collector will read HTTP APIs. The agent reads them the same way it reads REST APIs, so the log format is the same. Only the place where you turn logging on differs.

Turn on access logging for the stage and use the same JSON format as in the REST APIs section. That way, every log line carries the fields the agent needs.

Bedrock

Use this section if you turn on the Bedrock collector. The collector reads Bedrock's model invocation logs from CloudWatch Logs, so you need to complete two tasks: turn on logging in Amazon Bedrock, and tell the agent which log group to read.

Note: The Bedrock collector reads model invocation logs, which AWS provides for Amazon Bedrock Runtime. It does not collect data from Bedrock Mantle, because AWS does not support model invocation logging for it.

  1. Turn on model invocation logging in Amazon Bedrock, and send the logs to a CloudWatch Logs log group. AWS describes the steps in its model invocation logging documentation.

  2. Add the name of that log group to the log_groups field of the matching entry in accounts, which Step 6 explains. The Bedrock collector does not discover this log group on its own, so it reads only the log groups that you list.

If the API Gateway collector is also on, list the API Gateway log groups in the same log_groups field as well.

AgentCore

Use this section if you turn on the AgentCore collector. It needs the least setup on your side, because the agent finds its own log groups. All it needs is for AgentCore to produce observability data.

Turn on AgentCore observability by following the AWS documentation. The agent discovers all AgentCore log groups on its own, so you do not list any.


Step 2: Configure AWS credentials

Terraform uses your AWS credentials to create resources, and it deploys into the region those credentials point . Configure them with the AWS CLI:

This prompts for your AWS access key ID, secret access key, default region name, and default output format (use json). The CLI stores the answers so that Terraform and other AWS tools can find them.

If you use named profiles, tell the CLI which one to use:

This sets the profile for your current shell session. The profile must hold the credentials you use to deploy the Harness resources.

Then check the region:

The command prints the default region. It should match the region where you want to install the agent, because a mismatch deploys the resources somewhere you did not intend.

If you prefer environment variables to aws configure, set AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_REGION (or AWS_DEFAULT_REGION), and AWS_DEFAULT_OUTPUT instead.


Step 3: Create a cross-account IAM role (multiple accounts only)

Skip this step if every API gateway you want to monitor lives in the same account as the agent.

To collect from another account, the agent needs permission to read that account's logs. It gets that permission by assuming an IAM role that you create in the monitored account. You create one role per monitored account.

Use the following Terraform template to create the role. Run it with credentials for the monitored account, because the role must exist in that account.

Here is what each piece does:

  • The policy document grants read-only access: it can list log groups, read log events, and read API Gateway details. It applies to all resources (*) in the account.

  • The role carries the trust policy. The Principal line decides who may assume the role, and ${var.account_id} is the account you trust.

  • The attachment links the policy to the role.

  • account_id is the one value you supply. It is the ID of the AWS account that assumes the role, which is the host account, the account where the agent runs.

Note: The trust policy names the whole account (:root), so any identity in that account with sts:AssumeRole permission can assume this role. If your security team wants a narrower trust, restrict the principal or add a condition.

After you apply the template, copy the ARN of the new role. You enter it as cross_account_role_arn in Step 6.


Step 4: Download and extract the module

Download the Terraform module:

The --remote-name option (short form -O) saves the file under its original name, aws-cloudwatch-tf.tar.gz, in your current directory. The latest segment of the path selects the most recent release of the module.

Extract the archive and move into the directory:

tar xvzf unpacks the gzip archive and lists each file as it goes. The && cd part moves you into the new directory only if the extraction succeeded. Run every remaining command from this directory.


Step 5: Initialize Terraform

Before Terraform can create anything, it needs to download the providers the module uses. Run:

This downloads those providers and prepares the directory for the next command. You run it once per directory, and again if you change providers. Make a note of this directory. Terraform stores its state here, so you must run terraform destroy from the same directory when you uninstall the module.


Step 6: Create the terraform.tfvars file

Terraform reads your settings from a file named terraform.tfvars in the module directory. The example below is for ec2 mode with the API Gateway and Bedrock collectors turned on. Replace each placeholder with your own value.

The comment in the example marks the setting that most often causes errors. You must set at least one collector setting to true, because the module refuses to apply with all three off. The accounts list has one entry per account and region, so a second region needs a second entry.

The variables are grouped by when you need them. The sections below cover the variables that apply to both modes, the ones specific to ec2 or ecs, the ones that send credentials to a hosted TPA, and the accounts variable in detail.

Variables that apply to both modes

These variables apply whichever mode you choose, so start here. The following table lists each one with its default and the reason you might change it.

Variable
Required
Default
What It Does and Why You Might Change It

accounts

Yes

None

The accounts and regions to collect from. See Configure the accounts variable.

subnet_id

Yes

""

The subnet where the agent runs. Choose one that can reach your TPA endpoint and the AWS APIs.

deployment_mode

No

ec2

Where the agent runs: ec2 or ecs. Set it to ecs to run the agent as ECS tasks.

enable_api_gateway_collector

See note

false

Turns on the API Gateway collector.

enable_bedrock_collector

See note

false

Turns on the Bedrock collector.

enable_agentcore_collector

See note

false

Turns on the AgentCore collector.

traceable_agent_endpoint

No

http://localhost:5442

Where the agent sends data. Change it when your TPA runs somewhere else.

traceable_agent_max_batch_size

No

Set by the module

The maximum number of spans in one request to the TPA. Lower it if requests to your TPA time out.

traceable_service_name

No

aws-cloudwatch

The service name Harness shows. It applies to the API Gateway and Bedrock collectors only.

cron_interval_mins

No

5

How often the agent runs, in minutes. A shorter interval brings data in sooner, and a longer one means fewer runs.

max_concurrency

No

8

The maximum number of agent executions that can run at the same time.

log_group_arns

No

*

The log groups the agent may read. Narrow it to limit what the agent's IAM policy allows.

api_gateway_stages_arns

No

*

The API Gateway stages the agent may read, in the format restapis/<api_id>/stages/<stage_name>.

tags

No

None

Tags to add to every resource the module creates.

Note: You must set at least one of the three collector variables to true.

Variables for ec2 mode

Use these variables only when deployment_mode is ec2. They control the instance the module creates and where the agent installs on it. The following table lists each one.

Variable
Required
Default
What It Does and Why You Might Change It

key_name

No

""

The key pair for the instance. Set it if you need SSH access to troubleshoot.

instance_type

No

m4.xlarge

The EC2 instance type. Pick a larger type if you collect heavy traffic.

traceable_refresh_token

Yes

None

The refresh token for your Harness account. The instance uses it to authenticate with the Harness platform.

traceable_api_endpoint

No

api.traceable.ai

The Harness platform API endpoint. Change it if your account lives in another region.

tpa_version

No

<TPA_VERSION>

The TPA version to install.

service_version

No

latest

The agent version to install.

package_url

No

""

A custom location for the install package.

root_directory

No

/var/traceable/aws-cloudwatch

Where the agent is installed on the instance.

log_directory

No

/var/traceable/log/aws-cloudwatch

Where the agent writes its logs. You look here when you troubleshoot.

ami_name_ssm

No

/aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-x86_64

The SSM parameter that holds the AMI ID. The default resolves to the latest Amazon Linux 2023 image.

Variables for ecs mode

Use these variables only when deployment_mode is ecs. They describe the cluster and the task the agent runs as. The following table lists each one.

Variable
Required
Default
What It Does and Why You Might Change It

cluster_arn

Yes

None

The ARN of the ECS Fargate cluster that runs the tasks.

agent_image

Yes

traceableai/awscloudwatch:latest

The container image that ECS runs for each task. Change it only if you mirror the image into your own registry.

ecs_task_resources

No

{ cpu = 1024, memory = 2048 }

CPU (in millicores) and memory (in MB) for each task.

enable_logging

No

false

Saves the task logs to CloudWatch. Turn it on while you troubleshoot.

allow_images_from_ecr

No

true

Lets the task's IAM role pull images from Amazon ECR. Leave it on if you mirror the image into your own registry.

Variables that send credentials to a hosted TPA

Set these only if your agent sends data to a Harness-hosted (SaaS) TPA. The TPA uses them to identify your environment and authenticate the agent. The following table lists the three variables and the request header each one becomes.

Variable
Default
What It Does

traceable_environment

""

Sent to the TPA as the traceableai-environment-name header.

traceable_token

""

Sent to the TPA as the traceableai-agent-token header.

traceable_guid

""

Sent to the TPA as the x-traceable-guid header.

The agent leaves out any header whose value is empty. You need these three only when traceable_agent_endpoint points to a Harness-hosted (SaaS) TPA. For a self-hosted TPA, leave them empty.

Configure the accounts variable

The accounts variable is the one that tells the agent where to look, so it deserves a closer read. Each item in the list describes one account and region pair. The following table describes each key you can set inside an item.

Key
Required
What It Does

region

Yes

The AWS region to collect from.

api_list

Yes

A list of API IDs. To target one stage, write <api_id>/<stage_name>.

exclude

Yes

Decides how the agent treats the api_list entries. It applies to the API Gateway collector only. Set it to false to collect only the listed APIs. Set it to true to collect everything except the listed APIs.

cross_account_role_arn

Yes

The ARN of the role from Step 3. Use an empty string when the APIs are in the same account as the agent.

log_groups

No

A list of log group names to read directly. For the Bedrock collector, list the model-invocation log group here. If the API Gateway collector is also on and you list log groups for Bedrock, list the API Gateway log groups here too.

access_log_format

No

A map that describes the access log format your API Gateway instances use. It is required when you set log_groups.

Note: The AgentCore collector uses only region and cross_account_role_arn from each entry, and it discovers its own log groups.


Step 7: Apply the module

Applying the module is the step that creates the resources. Run:

Terraform first shows you the plan, which is the list of resources it will create, and waits for you to confirm. When you do, it creates them. Review the plan the first time you apply, because it is the quickest way to catch a wrong region, subnet, or mode before any resource exists.


Step 8: Verify the deployment

Give the agent one run interval (5 minutes by default), then check Harness for your traffic:

  • Go to Insights → Explore Data to view the traces and spans directly.

  • Go to Discovery → Inventory to view the services and endpoints the agent discovered.

API Gateway and Bedrock traffic appear under the service name you set in traceable_service_name, which is aws-cloudwatch by default. AgentCore traffic appears under the name of the AgentCore resource.

If nothing arrives, go through the checks in Troubleshooting the AWS CloudWatch agent.


Change or remove the deployment

To change a setting, edit terraform.tfvars and run terraform apply again. Terraform compares the new settings with what exists and changes only the difference.

To remove everything the module created, run the following from the directory where you ran terraform init:

Terraform reads its saved state from that directory, so running the command anywhere else leaves the resources in place.

For more information, see AWS CloudWatch agent overview and Configure the AgentCore collector.

Was this helpful?