Skip to content
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Troubleshooting the AWS CloudWatch agent

Two things tend to go wrong with this agent. Either the install fails outright, or it finishes without complaint and your spans never show up in Harness. This topic walks you through both.

The sections are organized by what you see, not by what caused it, so find the symptom that matches yours and start there. Some fixes depend on how you deployed the agent, and those sections are labeled. ec2 covers the manual install and ec2 mode in Terraform and CloudFormation. ecs covers ecs mode.


The installation or deployment fails

Use this part when a failure occurs during agent installation or deployment, before the agent has collected any data. The error appears in your terminal or in the AWS console, so these sections start from what you see there.

No collector is enabled

Cause: The deployment or install fails immediately with an error that contains "at least one collector must be enabled". All three collectors are off by default, and the agent will not start with any of them on.

Resolution: Turn on at least one collector. The following table lists the settings that turn on each collector for each installation method.

Installation Method
Settings

Terraform

enable_api_gateway_collector, enable_bedrock_collector, enable_agentcore_collector

CloudFormation

EnableApiGatewayCollector, EnableBedrockCollector, EnableAgentCoreCollector

Manual install on EC2

--enable-api-gateway, --enable-bedrock, --enable-agentcore

Set at least one to true, then run the deployment or installer again.


The CloudFormation stack rolls back

Cause: One resource in the stack failed to create, and CloudFormation rolled the whole stack back to leave nothing half-built.

Resolution:

  1. Open the CloudFormation console and go to Stacks.

  2. Select your stack and open its events.

  3. Find the resource whose creation failed, and read the reason next to it.

The reason usually points straight at the cause, such as a missing parameter, a subnet in the wrong VPC, or a permission your credentials lack.


The ECSCustomResource fails (ecs mode, CloudFormation only)

Cause: ECSCustomResource is a helper AWS Lambda function in the CloudFormation template. It creates the EventBridge schedules that start the agent. When it fails, the stack fails with it. The Terraform module does this work itself and has no such function.

Resolution:

  1. In CloudWatch, go to Log groups and search for your stack name.

  2. Open the log group for the Lambda function.

  3. Open the log event whose timestamp matches the time you created the stack. It shows what caused the failure.


Spans do not reach Harness

Start here when the install succeeded but your traffic does not appear. Work through the checks that match your deployment, in order. Most problems come from the first one.

API Gateway logging is not configured correctly

Cause: The API Gateway collector reads access logs in a specific format. If the stage does not write them, or writes them in another format, there is nothing for the agent to read.

Resolution: Go back to the "Prepare your AWS services" step in your installation topic and check each stage: Custom Access Logging is on, the log group ARN is correct, and the log format matches the JSON in that step.


Bedrock data does not appear

Cause: The Bedrock collector does not discover its log group. It reads only the log groups you name. If model invocation logging is off, or the log group is missing from your configuration, there is nothing for the agent to read.

Resolution:

  1. Confirm that model invocation logging is turned on in Amazon Bedrock, and that it writes to a CloudWatch Logs log group. AWS describes the steps in its model invocation logging documentation.

  2. Confirm that the log group name is in the log_groups field of the matching account entry.

  3. If the API Gateway collector is also on and you list log groups for Bedrock, confirm that you list the API Gateway log groups in the same field too.


The cron job is missing or not running (ec2)

Cause: The agent runs from a cron job on the host. If the job does not exist, or the system does not run it, no new data leaves the host.

Resolution:

  1. Check the agent logs in /var/traceable/log/aws-cloudwatch, or in the directory you set with --log-dir. If a healthy agent is running, new log entries appear on every run. If nothing new appears, the cron job is not running.

  2. Check whether the cron file exists:

This prints the schedule the installer registered. If the command reports that the file does not exist, run the installer again.

  1. If the file exists but the job still does not run, another system process may be stopping it from running as a system cron job. Register it as a cron job for the traceable user instead:

This loads the schedule from the cron file into the traceable user's own crontab, which bypasses the system-level problem.


The IAM permissions are wrong, or the TPA is unreachable (ec2)

Cause: The cron job runs, but the agent logs show errors, and your API Gateway and log group settings are correct. In that case, the usual cause is a missing IAM permission or a TPA that the agent cannot reach.

Resolution: Check the following in order.

  • The IAM role is missing a permission. Make sure that the role attached to the instance has apigateway:GET, logs:DescribeLogGroups, and logs:FilterLogEvents. The agent cannot find or read the logs without them. If you collect from another account, the role also needs sts:AssumeRole.

  • The agent cannot reach the TPA. Make sure that the TPA is running and reporting, and that the host running the agent can reach the TPA endpoint. The agent sends everything it collects to the TPA, so nothing reaches Harness while the endpoint is unreachable.


The EventBridge schedules are missing (ecs)

Cause: In ecs mode, EventBridge schedules start the agent as tasks. If the schedules do not exist, nothing ever starts. The usual reason is an incorrect IAM policy on the role that creates them.

Resolution:

  1. Open Amazon EventBridge → Schedules and confirm that schedules exist for your deployment.

  2. If none exist, open AWS CloudTrail and find the CreateSchedule event. A successful event has no error code. A failed one shows the reason.


The schedules exist, but the tasks do not run (ecs)

Cause: A schedule can exist and still fail to start a task, for example when the role it uses cannot run the task.

Resolution: In AWS CloudTrail, find the RunTask events and check each one for an error code. A successful event has none. The error code identifies why ECS could not start the task, such as a permission that the role lacks.


AgentCore spans do not appear

Cause: The AgentCore collector needs spans in aws/spans and, for the full trace, readable runtime logs.

Resolution:

  1. Confirm that AgentCore observability is turned on and that spans exist in the aws/spans log group.

  2. Confirm that the enrichment log group, /aws/bedrock-agentcore/runtimes/*/otel-rt-logs, is reachable by the agent's role.

  3. Search the agent logs, or the task logs in ecs mode, for Dropped N span(s). The collector drops a leaf span when it cannot enrich it, and logs the number it dropped each time. It never drops the parent of a dropped span.


For more information, see Configure the AgentCore collector and AWS CloudWatch agent overview.

Last updated

Was this helpful?