Deployment using Terraform
The Terraform module in this topic deploys the AWS CloudWatch agent into your AWS account. You choose whether the agent runs on an EC2 instance or as ECS Fargate tasks, and you choose which collectors it runs: API Gateway, Bedrock, AgentCore, or any mix of the three.
What will you learn in this topic?
By the end of this topic, you will understand:
The steps to prepare your AWS services and credentials for the agent.
The steps to configure and apply the Terraform module.
The purpose of each Terraform variable that you are likely to set.
The way to verify, change, and remove the deployment.
Before you begin
Make sure that Terraform is installed on the machine where you will run the module, because you apply the module's configuration files from that machine.
Make sure that the AWS CLI is configured with credentials that can deploy the resources in the target account and region. Step 2 covers this.
Make sure that you know which collectors you want. You must turn on at least one, or
terraform applystops with an error before it creates anything.Make sure that you have a TPA set up. The agent sends all its data to the TPA, so follow the TPA installation guide first, then note the TPA's URL. The URL must be reachable from the place where the agent runs. You enter it as
traceable_agent_endpointin Step 6.Make sure that you have a refresh token for your Harness account. In
ec2mode, the module passes it astraceable_refresh_tokenso that the instance can connect to Harness.Make sure that API Gateway logging, Bedrock model-invocation logging, or AgentCore observability is turned on for the services you want to collect. Step 1 covers all three.
Understand the deployment modes
The deployment_mode variable decides where the agent runs. Pick it first, because the two modes use different variables.
The following table describes the two modes, what Terraform creates in each, and when to choose one over the other.
ec2 (default)
An EC2 instance in the subnet you specify, with the agent running on it
You want the agent to run on one instance and you do not want to manage a container cluster
ecs
Resources that start the agent as an ECS task on a schedule, through Amazon EventBridge
You already run an ECS Fargate cluster and want the agent to run as a task
In ecs mode, you must supply an existing ECS Fargate cluster through cluster_arn. The module does not create the cluster.
Step 1: Prepare your AWS services
The agent does not create logs. It reads logs that your AWS services already write to CloudWatch Logs, so each service you want to collect from has to be logging first. The three collectors read different logs, so the preparation differs for each. API Gateway needs access logs in a specific JSON format. Bedrock needs model invocation logging enabled, plus the name of its log group, in your agent configuration. AgentCore needs observability turned on and nothing more.
Do only the parts for the collectors you plan to turn on. If you turn on more than one, do each part.
API Gateway (REST APIs)
Use this section if the API Gateway collector will read REST APIs. The collector reads each stage's access logs, so every stage you want to collect from must write them in a format the agent can parse.
Complete the following for every stage you want to collect from:
Open the API Gateway console and select your API, then select the stage.
Go to Logs/tracing.
In the CloudWatch Logs drop-down, select Error and info logs.
Turn on Custom Access Logging.
In Access log destination ARN, enter the ARN of a CloudWatch log group. This is where API Gateway writes the access logs that the agent reads.
In Log Format, paste the following JSON:
Each field maps to an API Gateway $context variable. API Gateway fills the variables in for every request, so each log line carries the request ID, the caller's IP address, the HTTP method, the path, the status code, the response size, and the domain name. The agent expects these fields in every log line, so leave none out.
API Gateway (HTTP APIs)
Use this section if the API Gateway collector will read HTTP APIs. The agent reads them the same way it reads REST APIs, so the log format is the same. Only the place where you turn logging on differs.
Turn on access logging for the stage and use the same JSON format as in the REST APIs section. That way, every log line carries the fields the agent needs.
Bedrock
Use this section if you turn on the Bedrock collector. The collector reads Bedrock's model invocation logs from CloudWatch Logs, so you need to complete two tasks: turn on logging in Amazon Bedrock, and tell the agent which log group to read.
Turn on model invocation logging in Amazon Bedrock, and send the logs to a CloudWatch Logs log group. AWS describes the steps in its model invocation logging documentation.
Add the name of that log group to the
log_groupsfield of the matching entry inaccounts, which Step 6 explains. The Bedrock collector does not discover this log group on its own, so it reads only the log groups that you list.
If the API Gateway collector is also on, list the API Gateway log groups in the same log_groups field as well.
AgentCore
Use this section if you turn on the AgentCore collector. It needs the least setup on your side, because the agent finds its own log groups. All it needs is for AgentCore to produce observability data.
Turn on AgentCore observability by following the AWS documentation. The agent discovers all AgentCore log groups on its own, so you do not list any.
Step 2: Configure AWS credentials
Terraform uses your AWS credentials to create resources, and it deploys into the region those credentials point . Configure them with the AWS CLI:
This prompts for your AWS access key ID, secret access key, default region name, and default output format (use json). The CLI stores the answers so that Terraform and other AWS tools can find them.
If you use named profiles, tell the CLI which one to use:
This sets the profile for your current shell session. The profile must hold the credentials you use to deploy the Harness resources.
Then check the region:
The command prints the default region. It should match the region where you want to install the agent, because a mismatch deploys the resources somewhere you did not intend.
If you prefer environment variables to aws configure, set AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_REGION (or AWS_DEFAULT_REGION), and AWS_DEFAULT_OUTPUT instead.
Step 3: Create a cross-account IAM role (multiple accounts only)
Skip this step if every API gateway you want to monitor lives in the same account as the agent.
To collect from another account, the agent needs permission to read that account's logs. It gets that permission by assuming an IAM role that you create in the monitored account. You create one role per monitored account.
Use the following Terraform template to create the role. Run it with credentials for the monitored account, because the role must exist in that account.
Here is what each piece does:
The policy document grants read-only access: it can list log groups, read log events, and read API Gateway details. It applies to all resources (
*) in the account.The role carries the trust policy. The
Principalline decides who may assume the role, and${var.account_id}is the account you trust.The attachment links the policy to the role.
account_idis the one value you supply. It is the ID of the AWS account that assumes the role, which is the host account, the account where the agent runs.
After you apply the template, copy the ARN of the new role. You enter it as cross_account_role_arn in Step 6.
Step 4: Download and extract the module
Download the Terraform module:
The --remote-name option (short form -O) saves the file under its original name, aws-cloudwatch-tf.tar.gz, in your current directory. The latest segment of the path selects the most recent release of the module.
Extract the archive and move into the directory:
tar xvzf unpacks the gzip archive and lists each file as it goes. The && cd part moves you into the new directory only if the extraction succeeded. Run every remaining command from this directory.
Step 5: Initialize Terraform
Before Terraform can create anything, it needs to download the providers the module uses. Run:
This downloads those providers and prepares the directory for the next command. You run it once per directory, and again if you change providers. Make a note of this directory. Terraform stores its state here, so you must run terraform destroy from the same directory when you uninstall the module.
Step 6: Create the terraform.tfvars file
Terraform reads your settings from a file named terraform.tfvars in the module directory. The example below is for ec2 mode with the API Gateway and Bedrock collectors turned on. Replace each placeholder with your own value.
The comment in the example marks the setting that most often causes errors. You must set at least one collector setting to true, because the module refuses to apply with all three off. The accounts list has one entry per account and region, so a second region needs a second entry.
The variables are grouped by when you need them. The sections below cover the variables that apply to both modes, the ones specific to ec2 or ecs, the ones that send credentials to a hosted TPA, and the accounts variable in detail.
Variables that apply to both modes
These variables apply whichever mode you choose, so start here. The following table lists each one with its default and the reason you might change it.
subnet_id
Yes
""
The subnet where the agent runs. Choose one that can reach your TPA endpoint and the AWS APIs.
deployment_mode
No
ec2
Where the agent runs: ec2 or ecs. Set it to ecs to run the agent as ECS tasks.
enable_api_gateway_collector
See note
false
Turns on the API Gateway collector.
enable_bedrock_collector
See note
false
Turns on the Bedrock collector.
enable_agentcore_collector
See note
false
Turns on the AgentCore collector.
traceable_agent_endpoint
No
http://localhost:5442
Where the agent sends data. Change it when your TPA runs somewhere else.
traceable_agent_max_batch_size
No
Set by the module
The maximum number of spans in one request to the TPA. Lower it if requests to your TPA time out.
traceable_service_name
No
aws-cloudwatch
The service name Harness shows. It applies to the API Gateway and Bedrock collectors only.
cron_interval_mins
No
5
How often the agent runs, in minutes. A shorter interval brings data in sooner, and a longer one means fewer runs.
max_concurrency
No
8
The maximum number of agent executions that can run at the same time.
log_group_arns
No
*
The log groups the agent may read. Narrow it to limit what the agent's IAM policy allows.
api_gateway_stages_arns
No
*
The API Gateway stages the agent may read, in the format restapis/<api_id>/stages/<stage_name>.
tags
No
None
Tags to add to every resource the module creates.
Variables for ec2 mode
Use these variables only when deployment_mode is ec2. They control the instance the module creates and where the agent installs on it. The following table lists each one.
key_name
No
""
The key pair for the instance. Set it if you need SSH access to troubleshoot.
instance_type
No
m4.xlarge
The EC2 instance type. Pick a larger type if you collect heavy traffic.
traceable_refresh_token
Yes
None
The refresh token for your Harness account. The instance uses it to authenticate with the Harness platform.
traceable_api_endpoint
No
api.traceable.ai
The Harness platform API endpoint. Change it if your account lives in another region.
tpa_version
No
<TPA_VERSION>
The TPA version to install.
service_version
No
latest
The agent version to install.
package_url
No
""
A custom location for the install package.
root_directory
No
/var/traceable/aws-cloudwatch
Where the agent is installed on the instance.
log_directory
No
/var/traceable/log/aws-cloudwatch
Where the agent writes its logs. You look here when you troubleshoot.
ami_name_ssm
No
/aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-x86_64
The SSM parameter that holds the AMI ID. The default resolves to the latest Amazon Linux 2023 image.
Variables for ecs mode
Use these variables only when deployment_mode is ecs. They describe the cluster and the task the agent runs as. The following table lists each one.
cluster_arn
Yes
None
The ARN of the ECS Fargate cluster that runs the tasks.
agent_image
Yes
traceableai/awscloudwatch:latest
The container image that ECS runs for each task. Change it only if you mirror the image into your own registry.
ecs_task_resources
No
{ cpu = 1024, memory = 2048 }
CPU (in millicores) and memory (in MB) for each task.
enable_logging
No
false
Saves the task logs to CloudWatch. Turn it on while you troubleshoot.
allow_images_from_ecr
No
true
Lets the task's IAM role pull images from Amazon ECR. Leave it on if you mirror the image into your own registry.
Variables that send credentials to a hosted TPA
Set these only if your agent sends data to a Harness-hosted (SaaS) TPA. The TPA uses them to identify your environment and authenticate the agent. The following table lists the three variables and the request header each one becomes.
traceable_environment
""
Sent to the TPA as the traceableai-environment-name header.
traceable_token
""
Sent to the TPA as the traceableai-agent-token header.
traceable_guid
""
Sent to the TPA as the x-traceable-guid header.
The agent leaves out any header whose value is empty. You need these three only when traceable_agent_endpoint points to a Harness-hosted (SaaS) TPA. For a self-hosted TPA, leave them empty.
Configure the accounts variable
The accounts variable is the one that tells the agent where to look, so it deserves a closer read. Each item in the list describes one account and region pair. The following table describes each key you can set inside an item.
region
Yes
The AWS region to collect from.
api_list
Yes
A list of API IDs. To target one stage, write <api_id>/<stage_name>.
exclude
Yes
Decides how the agent treats the api_list entries. It applies to the API Gateway collector only. Set it to false to collect only the listed APIs. Set it to true to collect everything except the listed APIs.
cross_account_role_arn
Yes
The ARN of the role from Step 3. Use an empty string when the APIs are in the same account as the agent.
log_groups
No
A list of log group names to read directly. For the Bedrock collector, list the model-invocation log group here. If the API Gateway collector is also on and you list log groups for Bedrock, list the API Gateway log groups here too.
access_log_format
No
A map that describes the access log format your API Gateway instances use. It is required when you set log_groups.
Step 7: Apply the module
Applying the module is the step that creates the resources. Run:
Terraform first shows you the plan, which is the list of resources it will create, and waits for you to confirm. When you do, it creates them. Review the plan the first time you apply, because it is the quickest way to catch a wrong region, subnet, or mode before any resource exists.
Step 8: Verify the deployment
Give the agent one run interval (5 minutes by default), then check Harness for your traffic:
Go to Insights → Explore Data to view the traces and spans directly.
Go to Discovery → Inventory to view the services and endpoints the agent discovered.
API Gateway and Bedrock traffic appear under the service name you set in traceable_service_name, which is aws-cloudwatch by default. AgentCore traffic appears under the name of the AgentCore resource.
If nothing arrives, go through the checks in Troubleshooting the AWS CloudWatch agent.
Change or remove the deployment
To change a setting, edit terraform.tfvars and run terraform apply again. Terraform compares the new settings with what exists and changes only the difference.
To remove everything the module created, run the following from the directory where you ran terraform init:
Terraform reads its saved state from that directory, so running the command anywhere else leaves the resources in place.
For more information, see AWS CloudWatch agent overview and Configure the AgentCore collector.
Was this helpful?