> For the complete documentation index, see [llms.txt](https://developer.harness.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://developer.harness.io/web-application-and-api-protection-waap/data-collection/gateways-load-balancers/aws/deployment-using-terraform.md).
# Deployment using Terraform
The Terraform template creates AWS resources to enable the monitoring of AWS API gateway logs. The template creates an EC2 instance where Traceable services are running. Traceable fetches the cloudwatch logs every 5 minutes, parses the data, and sends it to the Traceable platform. The processed data is displayed in the Traceable UI. The following flow diagram displays how the traffic flows through:
{% hint style="info" %}
AWS API Gateway currently limits log events to 1024 bytes. It truncates log events larger than 1024 bytes, such as request and response headers/bodies, before submission to CloudWatch Logs. For information on important notes about the AWS API gateway, see the [AWS docs](https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-known-issues.html#api-gateway-known-issues-all-apis).
{% endhint %}
***
## Before you begin
Make a note of the following points before proceeding with configurations:
* Keep Traceable's access token handy. It will be used when configuring the variables in the tfvars (`*.tfvars)` file. You can copy the access token by logging into your Traceable account and then navigating to **Settings** ()**→Account** → **Agent Token**.
* Make sure that Terraform is already installed. For more information on installing Terraform, see [Download Terraform](https://www.terraform.io/downloads.html).
#### **REST API gateway**
To monitor the REST API gateway, complete the following:
1. Navigate to the console. [aws.amazon.com/apigateway/](https://aws.amazon.com/api-gateway/).
2. Select the API and then select the stage.
3. Navigate to Logs/tracing.
4. Select **Full Request and Response Logs** from **Cloudwatch settings → Cloudwatch Logs** drop-down list.
5. Mark **Enable Access Logging** as `true` under **Custom Access Logging** and append the following JSON in **Log Format**:
```language-json
{
"requestId":"$context.requestId","ip":"$context.identity.sourceIp","httpMethod":"$context.httpMethod","path":"$context.path","status":"$context.status","responseLength":"$context.responseLength","domainName":"$context.domainName"
}
```
#### **HTTP API gateway**
To monitor the HTTP API gateway, enable access logging and append the JSON mentioned above in the **Log Format**.
#### **Configure AWS**
Configure AWS in your shell and verify that the region is set correctly. Enter the following command to set up your AWS CLI installation:
```language-actionscript
aws configure
```
The following example shows sample values. Replace them with your values to configure the credentials correctly.
```language-actionscript
$ aws configure
AWS Access Key ID [None]: AKIAIOSFODNN7EXAMPLE
AWS Secret Access Key [None]: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
Default region name [None]: us-west-2
Default output format [None]: json
```
For more information on the `credentials` file, see [Configuration and credential file settings](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html).
If you have configured named AWS profiles, export the environment variable `AWS_PROFILE=myprofile` where the profile named `myprofile` has the credentials that you wish to use to deploy the Traceable mirroring resources.
Finally, run the following command and verify that the AWS region is set to the region where you wish to install Traceable:
```language-actionscript
aws configure get region
```
Alternatively, you can configure the following environment variables:
* AWS\_ACCESS\_KEY\_ID
* AWS\_SECRET\_ACCESS\_KEY
* AWS\_REGION or AWS\_DEFAULT\_REGION
* AWS\_DEFAULT\_OUTPUT
For more configuration information, see [AWS documentation](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-quickstart.html#cli-configure-quickstart-config).
### **Multiple accounts**
A single AWS API Gateway account can support multiple accounts across different regions. However, each region within an account must be configured separately. This setup requires establishing cross-account IAM roles in each account, allowing the host account to authenticate and access the necessary resources.
#### Cross account IAM role
An IAM role is required to authenticate the host account to monitor the API gateways from another account. The following are the required permissions:
```language-plaintext
"logs:describeLogGroups"
"logs:FilterLogEvents"
"apigateway:GET"
```
Alternatively, you can use the following Terraform template to create cross account role. Override the `account_id` variable with the host account’s ID.
```language-plaintext
data "aws_iam_policy_document" "traceable" {
statement {
sid = "readAccess"
actions = [
"logs:describeLogGroups",
"logs:FilterLogEvents",
"apigateway:GET"
]
effect = "Allow"
resources = ["*"]
}
}
resource "aws_iam_policy" "traceable" {
name = "traceable-cross-account-policy"
description = "IAM policy for traceable instance"
path = "/"
policy = data.aws_iam_policy_document.traceable.json
}
resource "aws_iam_role" "traceable" {
name = "traceable-cross-account-role"
description = "IAM role for traceable instance"
path = "/"
force_detach_policies = true
assume_role_policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Action = "sts:AssumeRole"
Effect = "Allow"
Principal = {
AWS = "arn:aws:iam::${var.account_id}:root"
}
Condition = {}
}
]
})
}
resource "aws_iam_role_policy_attachment" "traceable" {
depends_on = [aws_iam_policy.traceable, aws_iam_role.traceable]
role = aws_iam_role.traceable.name
policy_arn = aws_iam_policy.traceable.arn
}
variable "account_id" {
type = string
}
```
***
## Download
Traceable provides AWS API gateway traffic mirroring tarball. Complete the following steps to download and untar the tarball:
1. Enter the following command to download the tarball:
```language-actionscript
curl -O https://downloads.traceable.ai/install/aws-api-gateway/terraform/latest/aws-api-gateway-tf.tar.gz
```
2. Untar the tarball. Enter the following command:
```language-actionscript
tar xvzf aws-api-gateway-tf.tar.gz
```
3. Change directory. Enter the following command:
```language-actionscript
cd aws-api-gateway-tf/
```
***
## Create tfvars file
Create a `terraform.tfvars` file with terraform variables as shown below:
```language-terraform.tfvars
subnet_id = "subnet-0fbc2fd166a8e94eb"
key_name = "traceable-local"
traceable_refresh_token = "***"
traceable_agent_endpoint = "https://1.2.3.4:5443"
traceable_environment = "awsapigw"
traceable_service_name = "awsapigw"
traceable_api_endpoint = "api.apac2.traceable.ai"
accounts = [
{
region = "us-east-1"
api_list = ["625vetvy0f/test"]
cross_account_role_arn = ""
exclude = false
},
{
region = "us-east-1"
api_list = []
cross_account_role_arn = "arn:aws:iam::1234567890:role/aws-apigw-cross-account-role"
exclude = true
}
]
tags = {
owner = "Traceable"
reason = "aws apigw"
expected-end-date = "07/05/2024"
}
service_version = "1.45.0"
```
***
## Configuration variables
The following tables describe the various terraform variables.
| Name | Optional/Mandatory | Default value | Description |
| -------------------------------- | ---------------------- | ----------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `accounts` | Mandatory | `-` | Refer to the next `accounts` section for the description. |
| `subnet_id` | Mandatory | `""` | The subnet ID where the Traceable instance is created. |
| `instance_type` | Optional | `m4.xlarge` | The type of Traceable instance. |
| `traceable_agent_endpoint` | Optional | `http://localhost:5442` | The Traceable Platform agent reporting endpoint for the agent. Platform agent deployment will also be done on the same VM if not overridden. For more information, see the [Traceable Platform agent](https://docs.traceable.ai/docs/aws-api-gateway#traceable-platform-agent). |
| `traceable_agent_max_batch_size` | Optional | 500 | The maximum number of spans that you wish to bundle in a single request to the Traceable Platform agent. |
| `root_directory` | Optional | `/var/traceable/aws-api-gateway` | Root directory path for traceable files. |
| `log_directory` | Optional | `/var/traceable/log/aws-api-gateway` | Log directory path for traceable log files. |
| `api_gateway_stages_arns` | Optional | \* | Enter a comma-separated list of ARNs of stages of API Gateways for creating the IAM policy.
For Example, arn:aws:apigateway:us-east-1::/restapis/api\_id/stages/stage\_name
|
| `log_group_arns` | Optional | \* | The ARNs of log groups used by the API gateways in this account. This is used to create the IAM policy used by the agent instance. |
| `tags` | Optional | | Tags that should be attached to resources created. |
| `cluster_arn` | Mandatory if using ECS | - | ECS Fargate Cluster ARN for running the tasks |
| `deployment_mode` | | `ec2` | This configuration controls whether to make an EC2-based deployment or an ECS-based one. Allowed values are `ecs` and `ec2` |
| `enable_logging` | Optional | `false` | On ECS, the logs can be saved to cloudwatch. Allowed values are `true` and `false`. |
| `agent_image` | Mandatory | `traceableai/awsapigw:` | AWS API Gateway Agent image is to be deployed. |
| `allow_images_from_ecr` | Optional | `true` | Configure IAM role to allow pulling images from ECR. |
| `cron_interval_mins` | Optional | 5 minutes | Time interval (in minutes) at which the cron job is triggered. |
### IAM roles for ECS deployment
If you choose ECS deployment, the following IAM roles are created:
#### 1. **IAM Role for the ECS Task to Call AWS SDK APIs**
**Key Permissions**:
* **API Gateway Access** (`apigateway:GET`): This allows the ECS task to perform **GET** operations on API Gateway. This could be for retrieving API configurations or monitoring API statuses.
```language-plaintext
{
"Action": "apigateway:GET",
"Effect": "Allow",
"Resource": "*",
"Sid": "getApiGateways"
}
```
* **CloudWatch Logs Access** (`logs:describeLogGroups`, `logs:FilterLogEvents`): These permissions allow the task to describe CloudWatch log groups and filter log events. This is useful for the ECS task to monitor logs if needed.
```language-plaintext
{
"Action": [
"logs:describeLogGroups",
"logs:FilterLogEvents"
],
"Effect": "Allow",
"Resource": "*",
"Sid": "filterLogEvents"
}
```
* **Cross-Account Role Assumption** (`sts:AssumeRole`): This permission allows the task to assume a cross-account role. This is important when the ECS task needs to access resources from a different AWS account.
```language-plaintext
{
"Action": "sts:AssumeRole",
"Effect": "Allow",
"Resource": "arn:aws:iam::909318972178:role/jacob-aws-apigw-cross-account-role"
}
```
#### 2. **IAM Role for Task Execution**
**Key Permissions**:
* **Pulling Docker Images from ECR** (`ecr:GetDownloadUrlForLayer`, `ecr:GetAuthorizationToken`, `ecr:BatchGetImage`, `ecr:BatchCheckLayerAvailability`): These permissions allow the ECS service to pull container images from AWS ECR, which is required to run the task.
```language-plaintext
{
"Action": [
"ecr:GetDownloadUrlForLayer",
"ecr:GetAuthorizationToken",
"ecr:BatchGetImage",
"ecr:BatchCheckLayerAvailability"
],
"Effect": "Allow",
"Resource": "*"
}
```
* **CloudWatch Logs Access** (`logs:PutLogEvents`, `logs:CreateLogStream`): These permissions allow the ECS task to write logs to CloudWatch by creating log streams and putting log events into them.
```language-plaintext
{
"Action": [
"logs:PutLogEvents",
"logs:CreateLogStream"
],
"Effect": "Allow",
"Resource": "arn:aws:logs:us-east-1:141020838:log-group:traceable-awsapigw-77722ddbacd:*"
}
```
#### 3. **IAM Role for EventBridge Scheduler**
**Key Permissions**:
* **ECS Task Execution** (`ecs:RunTask`): This permission allows EventBridge to trigger the execution of ECS tasks. The resources specified here are the specific ECS task definitions that can be run.
```language-plaintext
{
"Action": [
"ecs:RunTask"
],
"Effect": "Allow",
"Resource": [
"arn:aws:ecs:us-east-1:141020914838:task-definition/traceable-9093963563178-us-east-1-77722ddba6:1",
"arn:aws:ecs:us-east-1:141020914838:task-definition/traceable-base-us-east-1-77722ddba6:1"
]
}
```
* **IAM Role Passing** (`iam:PassRole`): This permission allows EventBridge to pass the necessary IAM roles (the task execution role and instance role) to the ECS tasks when they are triggered. Without this permission, EventBridge would not be able to assign the correct roles to the tasks it runs.
```language-plaintext
{
"Action": [
"iam:PassRole"
],
"Effect": "Allow",
"Resource": [
"arn:aws:iam::141020914838:role/traceable-task-exec-role-77722ddba6",
"arn:aws:iam::141020914838:role/traceable-instance-role-77722ddba6"
]
}
```
### accounts
The accounts object has the following values:
| Name | Optional/Mandatory | Description |
| ------------------------ | ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `region` | Mandatory | The AWS region from where you wish to capture the API traffic. |
| `exlude` | Mandatory | When set to `true`, the traffic is captured from all the API gateways **except the ones** provided in the `api_list`. When set to `false`, the traffic is captured **only from** the API listed in the `api_list`. |
| `api_list` | Mandatory | List of API IDs to consider for the above action.
API\_ID — If exclude = false, then monitor the API gateway that has ID = API\_ID. However, if exclude = true, then do not monitor this API gateway.API\_ID/STAGE\_NAME — If exclude = false, then monitor the API gateway stage with name = STAGE\_NAME and ID = API\_ID. If exclude = true, then do not monitor this stage.
|
| `cross_account_role_arn` | Mandatory | The ARN of the IAM role that authenticates this account for accessing the required resources from another account. Provide an empty string if the listed API gateways are in the same account. |
#### Example
The following example explains the configuration. The configuration is for 3 regions across 2 accounts.
```language-actionscript
accounts = [
{
region = "us-east-1"
api_list = ["625vetvy0f"]
cross_account_role_arn = ""
exclude = false
},
{
region = "us-east-2"
api_list = []
cross_account_role_arn = ""
exclude = true
},
{
region = "us-east-1"
api_list = []
cross_account_role_arn = "arn:aws:iam::account_2_id:role/traceable-cross-account-role"
exclude = true
},
{
region = "us-west-1"
api_list = ["example-api/example-stage"]
cross_account_role_arn = "arn:aws:iam::account_2_id:role/traceable-cross-account-role"
exclude = true
}
]
```
1. The first object is for the region `us-east-1` in Account 1. It will capture traffic only from the API gateway instance with the ID `625vetvy0f` in this account. It is assumed that Account 1 is the host account (where the agent instance is deployed).
2. The second object is for the region `us-east-2` in Account 1. It will capture traffic from all the API gateway instances in this region in this account.
3. The third object is for the region `us-east-1` in Account 2. It will capture traffic from all the API gateway instances in this region in this account. The `cross_account_role_arn` in this case, it is the ARN of the IAM Role authenticating Account 1 to access required objects in Account 2.
4. The fourth object is for region `us-west-1` in Account 2. It will capture traffic only from the stage `example-stage` of the API gateway instance with the ID `example-api` in this account.
The IAM role ARN is the same as that of case number 3.
Make sure to update the security group attached to this instance to allow SSH connection from your machine.