AWS CloudWatch agent overview
The AWS CloudWatch agent collects API traffic data from AWS services that write logs to Amazon CloudWatch Logs and sends it to Harness. Today it covers three services: API Gateway, Amazon Bedrock, and Amazon Bedrock AgentCore. The agent reads the CloudWatch logs that these services already write. You do not need to change the applications behind them.
If you already run the AWS API Gateway agent, this agent replaces it. For the migration steps, see Migrate from the AWS API Gateway agent.
What will you learn in this topic?
By the end of this topic, you will understand:
The purpose of the AWS CloudWatch agent and the AWS services it collects from.
The way the agent moves data from CloudWatch Logs to Harness.
The purpose of each collector and the setting that turns it on.
The steps to choose the installation method that fits your environment.
Before you begin
Before you install the agent, whichever method you use, complete the following:
Turn on logging for the AWS services you want to monitor. The agent does not generate data itself. It reads the logs that API Gateway, Amazon Bedrock, and Amazon Bedrock AgentCore write to Amazon CloudWatch Logs, so a service that does not write logs there gives the agent nothing to collect.
Decide which collectors you need: API Gateway, Bedrock, AgentCore, or any mix. All three are off by default, and you must turn on at least one.
Set up a TPA, make sure the host or task is running, and ensure the host or task that runs the agent can reach its endpoint. The agent sends everything it collects to the TPA, which passes it on to Harness.
Confirm that you have permission to deploy resources in the AWS account that hosts the agent. Terraform and CloudFormation create the instance or tasks, and the IAM roles for you, and the manual install needs access to the host.
Each installation topic lists the extra prerequisites for that method.
How the agent works
The agent does not sit in the path of your API traffic. It reads the logs your AWS services have already written, so it sees each request only after it has finished.
The flow has four stages:
Your AWS services write logs to CloudWatch log groups.
On a schedule, the agent reads the log groups for every collector you enabled. The default interval is 5 minutes.
The agent sends the data to the TPA endpoint that you configure. The default endpoint is
http://localhost:5442.The TPA passes the data to the Harness platform. You see it under Insights → Explore Data and Discovery → Inventory.
Understand the collectors
The agent has three collectors. Each one reads a different set of log groups, and each one is independent of the others. You turn on only the ones you need.
The following table summarizes what each collector reads and the TPA path it sends the data to.
API Gateway
API Gateway access and execution log groups
$TPA_ENDPOINT/ext_cap/v1/req_res_list_cap
Bedrock
The Bedrock model-invocation log group, which you name in the log_groups field
$TPA_ENDPOINT/ext_cap/v1/req_res_list_cap
AgentCore
The aws/spans log group, plus the AgentCore runtime log groups for enrichment. The agent discovers these on its own.
$TPA_ENDPOINT/v1/traces (OTLP)
All three collectors are off by default, and you must turn on at least one. If you turn on none, the deployment or installation fails with the error "at least one collector must be enabled".
Each installation method has its own setting for each collector. The following table lists the setting names for every method and the type of value each one takes.
Terraform
enable_api_gateway_collector, enable_bedrock_collector, enable_agentcore_collector
Boolean (true or false)
CloudFormation
EnableApiGatewayCollector, EnableBedrockCollector, EnableAgentCoreCollector
String ("true" or "false")
Manual install on EC2
--enable-api-gateway, --enable-bedrock, --enable-agentcore
Explicit true or false after each flag
For more information about the AgentCore collector, see Configure the AgentCore collector.
Choose an installation method
You can install the agent in three ways. They all run the same agent, so the choice comes down to how your team already manages infrastructure.
The following table compares the three installation methods by who they suit best and how each one runs the agent.
Terraform
Teams that manage AWS with infrastructure as code
Creates the resources for you, as an EC2 instance (ec2 mode) or as ECS Fargate tasks (ecs mode)
CloudFormation
AWS-native deployments that you can repeat from a template
Creates the resources for you, in the same two modes
Manual install on EC2
An EC2 or other Linux host you already manage, or an airgapped host that cannot pull container images
Registers the agent as a cron job on that host
In ecs mode, an Amazon EventBridge schedule starts an ECS task on each run, and you provide the ECS Fargate cluster. In ec2 mode, the stack creates an EC2 instance that runs the agent. The manual install skips both and runs the agent on a host you choose.
For the steps, see:
Collect from multiple accounts and regions
One host account can collect from API gateways in other AWS accounts. The agent assumes a cross-account IAM role in each account it monitors, and that role gives it read access to the logs and API Gateway details it needs.
Each region needs its own entry, even within a single account, so add one entry per account and region to the accounts setting.
Understand the AWS permissions the agent uses
The agent needs read access to CloudWatch Logs and API Gateway. The Terraform and CloudFormation methods create the required IAM roles and policies for you. For a manual install, you attach the permissions to the host’s IAM role or instance profile yourself, as described in the Manual install on EC2 topic. To collect from another AWS account, the cross-account role in that account needs the same read permissions.
The following table lists each permission the agent uses and the reason it needs it.
logs:DescribeLogGroups
Lists log groups, so the agent can find the ones to read. The AgentCore collector also uses it to discover runtime log groups.
logs:FilterLogEvents
Reads log events from the log groups.
apigateway:GET
Reads API Gateway details for the APIs and stages you listed.
sts:AssumeRole
Lets the agent assume the cross-account role. You need it only when you monitor another account.
For more information, see Deployment using Terraform, Deployment using CloudFormation, and Manual install on EC2.
Last updated
Was this helpful?