Deployment using Terraform

The Terraform template creates AWS resources to enable the monitoring of AWS API gateway logs. The template creates an EC2 instance where Traceable services are running. Traceable fetches the cloudwatch logs every 5 minutes, parses the data, and sends it to the Traceable platform. The processed data is displayed in the Traceable UI. The following flow diagram displays how the traffic flows through:

AWS API Gateway currently limits log events to 1024 bytes. It truncates log events larger than 1024 bytes, such as request and response headers/bodies, before submission to CloudWatch Logs. For information on important notes about the AWS API gateway, see the AWS docs.

---

Before you begin

Make a note of the following points before proceeding with configurations:

• Keep Traceable's access token handy. It will be used when configuring the variables in the tfvars (*.tfvars) file. You can copy the access token by logging into your Traceable account and then navigating to Settings ()→Account → Agent Token.

• Make sure that Terraform is already installed. For more information on installing Terraform, see Download Terraform.

REST API gateway

To monitor the REST API gateway, complete the following:

1. Navigate to the console. aws.amazon.com/apigateway/.

2. Select the API and then select the stage.

3. Navigate to Logs/tracing.

4. Select Full Request and Response Logs from Cloudwatch settings → Cloudwatch Logs drop-down list.

5. Mark Enable Access Logging as true under Custom Access Logging and append the following JSON in Log Format:

   {
     "requestId":"$context.requestId","ip":"$context.identity.sourceIp","httpMethod":"$context.httpMethod","path":"$context.path","status":"$context.status","responseLength":"$context.responseLength","domainName":"$context.domainName"
   }

HTTP API gateway

To monitor the HTTP API gateway, enable access logging and append the JSON mentioned above in the Log Format.

Configure AWS

Configure AWS in your shell and verify that the region is set correctly. Enter the following command to set up your AWS CLI installation:

aws configure

The following example shows sample values. Replace them with your values to configure the credentials correctly.

$ aws configure
AWS Access Key ID [None]: AKIAIOSFODNN7EXAMPLE
AWS Secret Access Key [None]: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
Default region name [None]: us-west-2
Default output format [None]: json

For more information on the credentials file, see Configuration and credential file settings.

If you have configured named AWS profiles, export the environment variable AWS_PROFILE=myprofile where the profile named myprofile has the credentials that you wish to use to deploy the Traceable mirroring resources.

Finally, run the following command and verify that the AWS region is set to the region where you wish to install Traceable:

aws configure get region

Alternatively, you can configure the following environment variables:

• AWS_ACCESS_KEY_ID

• AWS_SECRET_ACCESS_KEY

• AWS_REGION or AWS_DEFAULT_REGION

• AWS_DEFAULT_OUTPUT

For more configuration information, see AWS documentation.

Multiple accounts

A single AWS API Gateway account can support multiple accounts across different regions. However, each region within an account must be configured separately. This setup requires establishing cross-account IAM roles in each account, allowing the host account to authenticate and access the necessary resources.

Cross account IAM role

An IAM role is required to authenticate the host account to monitor the API gateways from another account. The following are the required permissions:

  "logs:describeLogGroups"
  "logs:FilterLogEvents"
  "apigateway:GET"

Alternatively, you can use the following Terraform template to create cross account role. Override the account_id variable with the host account’s ID.

data "aws_iam_policy_document" "traceable" {
  statement {
    sid = "readAccess"
    actions = [
      "logs:describeLogGroups",
      "logs:FilterLogEvents",
      "apigateway:GET"
    ]
    effect    = "Allow"
    resources = ["*"]
  }
}

resource "aws_iam_policy" "traceable" {
  name        = "traceable-cross-account-policy"
  description = "IAM policy for traceable instance"
  path        = "/"
  policy      = data.aws_iam_policy_document.traceable.json
}

resource "aws_iam_role" "traceable" {
  name                  = "traceable-cross-account-role"
  description           = "IAM role for traceable instance"
  path                  = "/"
  force_detach_policies = true
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Principal = {
          AWS = "arn:aws:iam::${var.account_id}:root"
        }
        Condition = {}
      }
    ]
  })
}

resource "aws_iam_role_policy_attachment" "traceable" {
  depends_on = [aws_iam_policy.traceable, aws_iam_role.traceable]
  role       = aws_iam_role.traceable.name
  policy_arn = aws_iam_policy.traceable.arn
}

variable "account_id" {
  type = string
}

---

Download

Traceable provides AWS API gateway traffic mirroring tarball. Complete the following steps to download and untar the tarball:

1. Enter the following command to download the tarball:

   curl -O https://downloads.traceable.ai/install/aws-api-gateway/terraform/latest/aws-api-gateway-tf.tar.gz

2. Untar the tarball. Enter the following command:

   tar xvzf aws-api-gateway-tf.tar.gz

3. Change directory. Enter the following command:

   cd aws-api-gateway-tf/

---

Create tfvars file

Create a terraform.tfvars file with terraform variables as shown below:

subnet_id                = "subnet-0fbc2fd166a8e94eb"
key_name                 = "traceable-local"
traceable_refresh_token  = "***"
traceable_agent_endpoint = "https://1.2.3.4:5443"
traceable_environment    = "awsapigw"
traceable_service_name   = "awsapigw"
traceable_api_endpoint   = "api.apac2.traceable.ai"
accounts = [
  {
    region                 = "us-east-1"
    api_list               = ["625vetvy0f/test"]
    cross_account_role_arn = ""
    exclude                = false
  },
  {
    region                 = "us-east-1"
    api_list               = []
    cross_account_role_arn = "arn:aws:iam::1234567890:role/aws-apigw-cross-account-role"
    exclude                = true
  }
]

tags = {
  owner             = "Traceable"
  reason            = "aws apigw"
  expected-end-date = "07/05/2024"
}

service_version = "1.45.0"

---

Configuration variables

The following tables describe the various terraform variables.

Name	Optional/Mandatory	Default value	Description
accounts	Mandatory	-	Refer to the next accounts section for the description.
subnet_id	Mandatory	""	The subnet ID where the Traceable instance is created.
instance_type	Optional	m4.xlarge	The type of Traceable instance.
traceable_agent_endpoint	Optional	http://localhost:5442	The Traceable Platform agent reporting endpoint for the agent. Platform agent deployment will also be done on the same VM if not overridden. For more information, see the Traceable Platform agent.
traceable_agent_max_batch_size	Optional	500	The maximum number of spans that you wish to bundle in a single request to the Traceable Platform agent.
root_directory	Optional	/var/traceable/aws-api-gateway	Root directory path for traceable files.
log_directory	Optional	/var/traceable/log/aws-api-gateway	Log directory path for traceable log files.
api_gateway_stages_arns	Optional	*	Enter a comma-separated list of ARNs of stages of API Gateways for creating the IAM policy. For Example, arn:aws:apigateway:us-east-1::/restapis/api_id/stages/stage_name
log_group_arns	Optional	*	The ARNs of log groups used by the API gateways in this account. This is used to create the IAM policy used by the agent instance.
tags	Optional		Tags that should be attached to resources created.
cluster_arn	Mandatory if using ECS	-	ECS Fargate Cluster ARN for running the tasks
deployment_mode		ec2	This configuration controls whether to make an EC2-based deployment or an ECS-based one. Allowed values are ecs and ec2
enable_logging	Optional	false	On ECS, the logs can be saved to cloudwatch. Allowed values are true and false.
agent_image	Mandatory	traceableai/awsapigw:<TEMPLATE_VERSION>	AWS API Gateway Agent image is to be deployed.
allow_images_from_ecr	Optional	true	Configure IAM role to allow pulling images from ECR.
cron_interval_mins	Optional	5 minutes	Time interval (in minutes) at which the cron job is triggered.

IAM roles for ECS deployment

If you choose ECS deployment, the following IAM roles are created:

1. IAM Role for the ECS Task to Call AWS SDK APIs

Key Permissions:

• API Gateway Access (apigateway:GET): This allows the ECS task to perform GET operations on API Gateway. This could be for retrieving API configurations or monitoring API statuses.

  {
      "Action": "apigateway:GET",
      "Effect": "Allow",
      "Resource": "*",
      "Sid": "getApiGateways"
  }

• CloudWatch Logs Access (logs:describeLogGroups, logs:FilterLogEvents): These permissions allow the task to describe CloudWatch log groups and filter log events. This is useful for the ECS task to monitor logs if needed.

  {
      "Action": [
          "logs:describeLogGroups",
          "logs:FilterLogEvents"
      ],
      "Effect": "Allow",
      "Resource": "*",
      "Sid": "filterLogEvents"
  }

• Cross-Account Role Assumption (sts:AssumeRole): This permission allows the task to assume a cross-account role. This is important when the ECS task needs to access resources from a different AWS account.

  {
      "Action": "sts:AssumeRole",
      "Effect": "Allow",
      "Resource": "arn:aws:iam::909318972178:role/jacob-aws-apigw-cross-account-role"
  }

2. IAM Role for Task Execution

Key Permissions:

• Pulling Docker Images from ECR (ecr:GetDownloadUrlForLayer, ecr:GetAuthorizationToken, ecr:BatchGetImage, ecr:BatchCheckLayerAvailability): These permissions allow the ECS service to pull container images from AWS ECR, which is required to run the task.

  {
      "Action": [
          "ecr:GetDownloadUrlForLayer",
          "ecr:GetAuthorizationToken",
          "ecr:BatchGetImage",
          "ecr:BatchCheckLayerAvailability"
      ],
      "Effect": "Allow",
      "Resource": "*"
  }

• CloudWatch Logs Access (logs:PutLogEvents, logs:CreateLogStream): These permissions allow the ECS task to write logs to CloudWatch by creating log streams and putting log events into them.

  {
      "Action": [
          "logs:PutLogEvents",
          "logs:CreateLogStream"
      ],
      "Effect": "Allow",
      "Resource": "arn:aws:logs:us-east-1:141020838:log-group:traceable-awsapigw-77722ddbacd:*"
  }

3. IAM Role for EventBridge Scheduler

Key Permissions:

• ECS Task Execution (ecs:RunTask): This permission allows EventBridge to trigger the execution of ECS tasks. The resources specified here are the specific ECS task definitions that can be run.

  {
      "Action": [
          "ecs:RunTask"
      ],
      "Effect": "Allow",
      "Resource": [
          "arn:aws:ecs:us-east-1:141020914838:task-definition/traceable-9093963563178-us-east-1-77722ddba6:1",
          "arn:aws:ecs:us-east-1:141020914838:task-definition/traceable-base-us-east-1-77722ddba6:1"
      ]
  }

• IAM Role Passing (iam:PassRole): This permission allows EventBridge to pass the necessary IAM roles (the task execution role and instance role) to the ECS tasks when they are triggered. Without this permission, EventBridge would not be able to assign the correct roles to the tasks it runs.

  {
      "Action": [
          "iam:PassRole"
      ],
      "Effect": "Allow",
      "Resource": [
          "arn:aws:iam::141020914838:role/traceable-task-exec-role-77722ddba6",
          "arn:aws:iam::141020914838:role/traceable-instance-role-77722ddba6"
      ]
  }

accounts

The accounts object has the following values:

Name	Optional/Mandatory	Description
region	Mandatory	The AWS region from where you wish to capture the API traffic.
exlude	Mandatory	When set to true, the traffic is captured from all the API gateways except the ones provided in the api_list. When set to false, the traffic is captured only from the API listed in the api_list.
api_list	Mandatory	List of API IDs to consider for the above action. API_ID — If exclude = false, then monitor the API gateway that has ID = API_ID. However, if exclude = true, then do not monitor this API gateway. API_ID/STAGE_NAME — If exclude = false, then monitor the API gateway stage with name = STAGE_NAME and ID = API_ID. If exclude = true, then do not monitor this stage.
cross_account_role_arn	Mandatory	The ARN of the IAM role that authenticates this account for accessing the required resources from another account. Provide an empty string if the listed API gateways are in the same account.

Example

The following example explains the configuration. The configuration is for 3 regions across 2 accounts.

accounts = [
  {
    region                 = "us-east-1"
    api_list               = ["625vetvy0f"]
    cross_account_role_arn = ""
    exclude                = false
  },
  {
    region                 = "us-east-2"
    api_list               = []
    cross_account_role_arn = ""
    exclude                = true
  },
  {
    region                 = "us-east-1"
    api_list               = []
    cross_account_role_arn = "arn:aws:iam::account_2_id:role/traceable-cross-account-role"
    exclude                = true
  },
  {
    region                 = "us-west-1"
    api_list               = ["example-api/example-stage"]
    cross_account_role_arn = "arn:aws:iam::account_2_id:role/traceable-cross-account-role"
    exclude                = true
  }
]

1. The first object is for the region us-east-1 in Account 1. It will capture traffic only from the API gateway instance with the ID 625vetvy0f in this account. It is assumed that Account 1 is the host account (where the agent instance is deployed).

2. The second object is for the region us-east-2 in Account 1. It will capture traffic from all the API gateway instances in this region in this account.

3. The third object is for the region us-east-1 in Account 2. It will capture traffic from all the API gateway instances in this region in this account. The cross_account_role_arn in this case, it is the ARN of the IAM Role authenticating Account 1 to access required objects in Account 2.

4. The fourth object is for region us-west-1 in Account 2. It will capture traffic only from the stage example-stage of the API gateway instance with the ID example-api in this account.

   The IAM role ARN is the same as that of case number 3.

Traceable Platform Agent

Configure these parameters only when you wish to deploy the Traceable Platform agent alongside the tracing agent instance. The Traceable Platform Agent will be deployed alongside the agent instance only if the parameter traceable_agent_endpoint of above is configured as http://localhost:5442.

Parameter	Description	Default value
traceable_refresh_token	Traceable Platform token. This is the token you have generated as part of a step in the Before you begin section.	-
traceable_environment	Defines the Traceable environment, for example, dev, QE, staging, etc.	aws-api-gateway
traceable_api_endpoint	Traceable Platform API endpoint	api.traceable.ai
service_version	Traceable Platform Agent version to be installed.	-

---

Apply terraform

As a last step to configure, run the following command to apply the terraform changes:

terraform init
terraform apply

---

Verification

Log into your Traceable Platform account and navigate to API Catalog → Services to view the service name (traceable_service_name) that you configured earlier.

---

Uninstall

To uninstall, run the following command from the same directory to destroy all the resources created in the installation step:

terraform destroy

---

Troubleshooting

Loss of traffic

If your setup is experiencing a loss of traffic or observing less amount of traffic, you may try doing the following:

1. Login to the Traceable EC2 instance you created using the Terraform template earlier. The instance's name should be with the prefix traceable-instance-.

   Make sure to update the security group attached to this instance to allow SSH connection from your machine.

2. After you log in, check the service status of the running service. Enter the following command:

   sudo systemctl status traceable logstash

You may also view the logs generated by Traceable by navigating to the logs directory:

cd /var/traceable/log/aws-api-gateway/
less gateway.err