For the complete documentation index, see llms.txt. This page is also available as Markdown.

AWS CloudWatch agent overview

The AWS CloudWatch agent collects API traffic data from AWS services that write logs to Amazon CloudWatch Logs and sends it to Harness. Today it covers three services: API Gateway, Amazon Bedrock, and Amazon Bedrock AgentCore. The agent reads the CloudWatch logs that these services already write. You do not need to change the applications behind them.

If you already run the AWS API Gateway agent, this agent replaces it. For the migration steps, see Migrate from the AWS API Gateway agent.

What will you learn in this topic?

By the end of this topic, you will understand:

  • The purpose of the AWS CloudWatch agent and the AWS services it collects from.

  • The way the agent moves data from CloudWatch Logs to Harness.

  • The purpose of each collector and the setting that turns it on.

  • The steps to choose the installation method that fits your environment.


Before you begin

Before you install the agent, whichever method you use, complete the following:

  • Turn on logging for the AWS services you want to monitor. The agent does not generate data itself. It reads the logs that API Gateway, Amazon Bedrock, and Amazon Bedrock AgentCore write to Amazon CloudWatch Logs, so a service that does not write logs there gives the agent nothing to collect.

  • Decide which collectors you need: API Gateway, Bedrock, AgentCore, or any mix. All three are off by default, and you must turn on at least one.

  • Set up a TPA, make sure the host or task is running, and ensure the host or task that runs the agent can reach its endpoint. The agent sends everything it collects to the TPA, which passes it on to Harness.

  • Confirm that you have permission to deploy resources in the AWS account that hosts the agent. Terraform and CloudFormation create the instance or tasks, and the IAM roles for you, and the manual install needs access to the host.

Each installation topic lists the extra prerequisites for that method.


How the agent works

The agent does not sit in the path of your API traffic. It reads the logs your AWS services have already written, so it sees each request only after it has finished.

The flow has four stages:

  1. Your AWS services write logs to CloudWatch log groups.

  2. On a schedule, the agent reads the log groups for every collector you enabled. The default interval is 5 minutes.

  3. The agent sends the data to the TPA endpoint that you configure. The default endpoint is http://localhost:5442.

  4. The TPA passes the data to the Harness platform. You see it under Insights → Explore Data and Discovery → Inventory.


Understand the collectors

The agent has three collectors. Each one reads a different set of log groups, and each one is independent of the others. You turn on only the ones you need.

The following table summarizes what each collector reads and the TPA path it sends the data to.

Collector
What It Reads
Where It Sends Data

API Gateway

API Gateway access and execution log groups

$TPA_ENDPOINT/ext_cap/v1/req_res_list_cap

Bedrock

The Bedrock model-invocation log group, which you name in the log_groups field

$TPA_ENDPOINT/ext_cap/v1/req_res_list_cap

AgentCore

The aws/spans log group, plus the AgentCore runtime log groups for enrichment. The agent discovers these on its own.

$TPA_ENDPOINT/v1/traces (OTLP)

All three collectors are off by default, and you must turn on at least one. If you turn on none, the deployment or installation fails with the error "at least one collector must be enabled".

Each installation method has its own setting for each collector. The following table lists the setting names for every method and the type of value each one takes.

Installation Method
Setting Names
Value Type

Terraform

enable_api_gateway_collector, enable_bedrock_collector, enable_agentcore_collector

Boolean (true or false)

CloudFormation

EnableApiGatewayCollector, EnableBedrockCollector, EnableAgentCoreCollector

String ("true" or "false")

Manual install on EC2

--enable-api-gateway, --enable-bedrock, --enable-agentcore

Explicit true or false after each flag

Note: The service name that Harness shows for your traffic depends on the collector. The service name setting (traceable_service_name, TraceableServiceName, or --service-name) applies to the API Gateway and Bedrock collectors only. The AgentCore collector always takes its service name from the AgentCore resource (the AgentCore agent or runtime name), so the setting has no effect on it.

Note: The Bedrock collector reads model invocation logs, which AWS provides for Amazon Bedrock Runtime. It does not collect data from Bedrock Mantle, because AWS does not support model invocation logging for it.

For more information about the AgentCore collector, see Configure the AgentCore collector.


Choose an installation method

You can install the agent in three ways. They all run the same agent, so the choice comes down to how your team already manages infrastructure.

The following table compares the three installation methods by who they suit best and how each one runs the agent.

Installation Method
Best For
How It Runs

Terraform

Teams that manage AWS with infrastructure as code

Creates the resources for you, as an EC2 instance (ec2 mode) or as ECS Fargate tasks (ecs mode)

CloudFormation

AWS-native deployments that you can repeat from a template

Creates the resources for you, in the same two modes

Manual install on EC2

An EC2 or other Linux host you already manage, or an airgapped host that cannot pull container images

Registers the agent as a cron job on that host

In ecs mode, an Amazon EventBridge schedule starts an ECS task on each run, and you provide the ECS Fargate cluster. In ec2 mode, the stack creates an EC2 instance that runs the agent. The manual install skips both and runs the agent on a host you choose.

For the steps, see:


Collect from multiple accounts and regions

One host account can collect from API gateways in other AWS accounts. The agent assumes a cross-account IAM role in each account it monitors, and that role gives it read access to the logs and API Gateway details it needs.

Each region needs its own entry, even within a single account, so add one entry per account and region to the accounts setting.


Understand the AWS permissions the agent uses

The agent needs read access to CloudWatch Logs and API Gateway. The Terraform and CloudFormation methods create the required IAM roles and policies for you. For a manual install, you attach the permissions to the host’s IAM role or instance profile yourself, as described in the Manual install on EC2 topic. To collect from another AWS account, the cross-account role in that account needs the same read permissions.

The following table lists each permission the agent uses and the reason it needs it.

Permission
Why the Agent Needs It

logs:DescribeLogGroups

Lists log groups, so the agent can find the ones to read. The AgentCore collector also uses it to discover runtime log groups.

logs:FilterLogEvents

Reads log events from the log groups.

apigateway:GET

Reads API Gateway details for the APIs and stages you listed.

sts:AssumeRole

Lets the agent assume the cross-account role. You need it only when you monitor another account.

For more information, see Deployment using Terraform, Deployment using CloudFormation, and Manual install on EC2.

Last updated

Was this helpful?