> For the complete documentation index, see [llms.txt](https://developer.harness.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://developer.harness.io/resilience-testing/chaos-engineering/faults/chaos-fault-categories/aws/ecs-fargate-memory-hog.md).

# ECS Fargate memory hog

ECS Fargate memory hog is an AWS chaos fault that consumes `MEMORY_IN_MEGABYTE` MB of memory inside a percentage of running ECS Fargate tasks for a configurable duration. Because Fargate does not expose the underlying host, the fault adds a chaos sidecar container (`SIDECAR_CONTAINER`) to a new task definition revision and rolls the service onto that revision; the sidecar allocates and holds memory inside the same task boundary as the main container. When the fault ends, the original revision is restored and the sidecar is removed.

Use this fault to test how a Fargate service behaves under sustained memory pressure inside the same task as the workload: whether the main container is OOM-killed first, whether the application detects low-memory conditions and sheds load, and whether memory-based autoscaling reacts correctly.

{% hint style="info" %}
**RUN YOUR FIRST EXPERIMENT**

If you have not configured the chaos infrastructure yet, go to [Quickstart](/resilience-testing/chaos-engineering/new-to-chaos-engineering/quickstart.md) to install the chaos infrastructure and run an experiment end to end.
{% endhint %}

***

### Use cases <a href="#use-cases" id="use-cases"></a>

Run this fault when you want to answer concrete questions like:

* **OOM behaviour:** When the sidecar consumes most of the task's memory budget, does the main container OOM-kill first, and does the task restart cleanly?
* **Autoscaling response:** Does the service's memory-based scaling policy fire and add capacity within the expected window?
* **Cache eviction:** For caches that hold a large working set in process memory, do they evict gracefully under pressure, or do they crash?
* **Memory-safe languages vs unsafe:** Does the application's GC behaviour stabilize, or does it enter a death spiral?
* **Recovery time:** When the chaos sidecar is removed, how quickly does the workload return to baseline memory utilization?

***

### Prerequisites <a href="#prerequisites" id="prerequisites"></a>

* **Kubernetes version:** 1.21 or later for the chaos infrastructure cluster. Go to [What's supported](/resilience-testing/chaos-engineering/new-to-chaos-engineering/whats-supported.md) to confirm distribution support.
* **Target ECS Fargate service:** `SERVICE_NAME` exists in `CLUSTER_NAME` within `REGION` and runs on the Fargate launch type with at least one running task.
* **Sufficient task memory budget:** The task definition has enough memory allocated that adding the sidecar's reservation is meaningful but does not violate Fargate's valid CPU/memory combinations.
* **AWS credentials available:** Either an AWS credentials file uploaded as a **File Secret in Harness Secret Manager** (see Authentication below) or an IAM role for service accounts (IRSA) bound to the chaos infrastructure service account.
* **IAM permissions granted:** The credentials or role include the permissions listed below, including `iam:PassRole`.

***

### Supported environments <a href="#supported-environments" id="supported-environments"></a>

| Platform                                                         | Support status                                                                                                                                              |
| ---------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Amazon ECS on Fargate launch type                                | Supported                                                                                                                                                   |
| Amazon ECS on EC2 launch type                                    | Not supported (use [ECS container memory hog](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/aws/ecs-container-memory-hog.md) instead) |
| Services with task-level memory defined                          | Supported                                                                                                                                                   |
| Tasks that already contain a container named `SIDECAR_CONTAINER` | Not supported; pick a unique sidecar name                                                                                                                   |
| AWS regions                                                      | Supported in every commercial region where Fargate is available; pass the region in `REGION`                                                                |

***

### Permissions required <a href="#permissions-required" id="permissions-required"></a>

The IAM principal that the chaos pod uses (the credentials mounted from the Harness Secret Manager file secret, the IRSA role on the chaos service account, or the role assumed via `ASSUME_ROLE_ARN`) needs the following AWS actions.

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ecs:DescribeServices",
        "ecs:DescribeTaskDefinition",
        "ecs:RegisterTaskDefinition",
        "ecs:DeregisterTaskDefinition",
        "ecs:UpdateService",
        "ecs:ListTasks",
        "ecs:DescribeTasks"
      ],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "iam:PassRole"
      ],
      "Resource": "*"
    }
  ]
}
```

Go to [common policy for all AWS faults](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/aws/security-configurations/policy-for-all-aws-faults.md) to use a single superset IAM policy across every AWS fault.

***

### Authentication <a href="#authentication" id="authentication"></a>

The fault supports three credential delivery models. Pick one based on how your chaos infrastructure is deployed.

| Method                                | When to use it                                                                 | How to configure                                                                                                                                                                                                                    |
| ------------------------------------- | ------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Harness Secret Manager file secret    | Chaos infrastructure runs outside EKS, or you want explicit static credentials | Upload the AWS credentials file as a **File Secret** in Harness Secret Manager and reference its identifier via `AWS_AUTHENTICATION_SECRET`                                                                                         |
| IAM Roles for Service Accounts (IRSA) | Chaos infrastructure runs in EKS and uses an OIDC-bound service account        | No tunable changes; the chaos pod inherits the role automatically. Go to [AWS IAM integration](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/aws/security-configurations/aws-iam-integration.md) to set it up |
| Assume role                           | The fault needs to act in a different account or with elevated permissions     | Set `ASSUME_ROLE_ARN` to the role ARN; the chaos pod assumes the role on top of its base credentials                                                                                                                                |

When using the Harness Secret Manager method, the contents of the File Secret should be the AWS credentials file in the standard `~/.aws/credentials` format:

```ini
[default]
aws_access_key_id = REPLACE_WITH_ACCESS_KEY_ID
aws_secret_access_key = REPLACE_WITH_SECRET_ACCESS_KEY
```

Upload this file as a **File Secret** in Harness Secret Manager (Project Setup → Secrets → New File Secret), and pass the secret identifier in `AWS_AUTHENTICATION_SECRET` when configuring the fault.

Go to [AWS named profile for chaos](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/aws/security-configurations/aws-switch-profile.md) to switch between profiles inside a single credentials file.

***

### Fault tunables <a href="#fault-tunables" id="fault-tunables"></a>

Configure the following fault parameters when you add ECS Fargate memory hog to an experiment in Chaos Studio. Defaults are shown for reference.

**Required parameters**

| Tunable        | Description                                                      | Default    |
| -------------- | ---------------------------------------------------------------- | ---------- |
| `CLUSTER_NAME` | Name of the target ECS cluster.                                  | (required) |
| `SERVICE_NAME` | Name of the target ECS service.                                  | (required) |
| `REGION`       | AWS region that hosts the ECS cluster (for example `us-east-1`). | (required) |

**Chaos parameters**

| Tunable                      | Description                                                                                                                                                                                                      | Default   |
| ---------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------- |
| `MEMORY_IN_MEGABYTE`         | Amount of memory (in MB) the chaos sidecar allocates inside the task.                                                                                                                                            | `500`     |
| `SIDECAR_CONTAINER`          | Name to use for the chaos sidecar container in the task definition. Must not clash with any existing container name in the task.                                                                                 | `default` |
| `TASK_REPLICA_AFFECTED_PERC` | Percentage of task replicas to affect through the deployment.                                                                                                                                                    | `100`     |
| `TOTAL_CHAOS_DURATION`       | Duration of the fault in seconds. The service runs on the chaos revision (with the stress sidecar) for this period.                                                                                              | `30`      |
| `CHAOS_INTERVAL`             | Delay in seconds between successive iterations when running for more than one cycle.                                                                                                                             | `30`      |
| `RAMP_TIME`                  | Wait period in seconds before and after the fault. Go to [ramp time](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/common-tunables-for-all-faults.md#ramp-time) to read how it is applied. | `0`       |

**Authentication**

| Tunable                     | Description                                                                                                                       | Default |
| --------------------------- | --------------------------------------------------------------------------------------------------------------------------------- | ------- |
| `ASSUME_ROLE_ARN`           | ARN of an IAM role to assume on top of the base credentials. Leave empty to use the base credentials directly.                    | `""`    |
| `AWS_AUTHENTICATION_SECRET` | Identifier of the **File Secret in Harness Secret Manager** that contains the AWS credentials file. Not required when using IRSA. | `""`    |

Tunables that apply to every fault are documented in [common tunables for all faults](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/common-tunables-for-all-faults.md). AWS-specific shared tunables are documented in [common AWS fault tunables](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/aws/aws-fault-tunables.md).

***

### Fault execution in brief <a href="#fault-execution-in-brief" id="fault-execution-in-brief"></a>

Reads the current task definition for `SERVICE_NAME`, registers a new revision with an additional chaos sidecar (`SIDECAR_CONTAINER`) configured to allocate `MEMORY_IN_MEGABYTE` MB, updates the service to that revision, waits for `TOTAL_CHAOS_DURATION` seconds, then restores the original revision and deregisters the chaos revision.

***

### Expected behavior during fault execution <a href="#expected-behavior-during-fault-execution" id="expected-behavior-during-fault-execution"></a>

* A new task definition revision is registered with the chaos sidecar added.
* The service deploys new tasks on the chaos revision; old tasks drain.
* Inside the new tasks, the sidecar allocates and holds the requested memory. Because Fargate enforces task-level memory, the main container's available memory is reduced.
* If the combined working set exceeds the task's memory limit, ECS OOM-kills the container that exceeded its limit (usually whichever container hit its container-level limit first). Frequently, this is the chaos sidecar itself.
* Application latency may rise; the memory-based autoscaling alarm may fire.

{% hint style="info" %}
**WHEN THE FAULT ENDS**

The chaos pod restores the original task definition revision and waits for the rollout to begin. The chaos revision (and its sidecar) is removed from the service; the chaos task definition revision is deregistered.
{% endhint %}

#### Signals to watch <a href="#signals-to-watch" id="signals-to-watch"></a>

Attach [resilience probes](/resilience-testing/chaos-engineering/use-chaos-engineering/probes.md) to assert each layer:

* **Memory utilization:** Use a [Prometheus probe](/resilience-testing/chaos-engineering/use-chaos-engineering/probes/apm-probes.md) on `aws_ecs_memory_utilization` for the service to confirm utilization rises.
* **Task OOM count:** Use a Prometheus probe on a counter incremented when ECS reports OOM-killed containers (visible via `aws ecs describe-tasks --query "tasks[].containers[].exitCode"`).
* **Application latency:** Use an [HTTP probe](/resilience-testing/chaos-engineering/use-chaos-engineering/probes/http-probe.md) and assert percentile latency stays within SLO.
* **Autoscaling alarm:** Confirm the memory-based scaling policy alarm fires when expected.

***

### Verify the fault execution effect <a href="#verify-the-fault-execution-effect" id="verify-the-fault-execution-effect"></a>

While the experiment is running, confirm the chaos revision is active and memory utilization rose:

1. **Inspect the active task definition.**

   ```bash
   aws ecs describe-services \
     --cluster <cluster> \
     --services <service> \
     --region <region> \
     --query "services[].taskDefinition"
   aws ecs describe-task-definition \
     --task-definition <active-arn> \
     --region <region> \
     --query "taskDefinition.containerDefinitions[].name"
   ```

   During the fault the container list should include `SIDECAR_CONTAINER`; afterwards it should not.
2. **Check memory utilization.**

   In the AWS console (or via CloudWatch Insights), the service's `MemoryUtilization` should rise during the chaos window and fall after recovery.
3. **Check for OOM kills.**

   ```bash
   aws ecs list-tasks --cluster <cluster> --service-name <service> --desired-status STOPPED --region <region>
   aws ecs describe-tasks --cluster <cluster> --tasks <arn> --region <region> \
     --query "tasks[].containers[].[name,exitCode,reason]"
   ```

   Look for `OutOfMemoryError` in the reason field for containers that were OOM-killed.

***

### Recovery and cleanup <a href="#recovery-and-cleanup" id="recovery-and-cleanup"></a>

* **End of duration:** The chaos pod restores the original revision and deregisters the chaos revision.
* **Abort the experiment:** Stopping the experiment from Chaos Studio also triggers the restore-and-deregister cleanup.
* **Manual recovery:** If the fault exits before cleanup, roll the service back manually with `aws ecs update-service --task-definition <original-arn>` and deregister the chaos revision.
* **Workload recovery:** Restarting tasks pick up the original memory layout; in-memory state held by the OOM-killed processes is lost.

***

### Limitations <a href="#limitations" id="limitations"></a>

* **Fargate-only:** This fault is meaningful only on Fargate; for EC2 launch type, use [ECS container memory hog](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/aws/ecs-container-memory-hog.md).
* **Valid CPU/memory pairings:** Adding the sidecar must not push the task definition outside Fargate's valid CPU/memory combinations; otherwise registration fails.
* **No in-place stress:** Existing tasks are not stressed; only new tasks on the chaos revision feel the load.
* **Sidecar name must be unique:** If a container named `SIDECAR_CONTAINER` already exists in the task, registration fails.
* **Cross-region targeting:** A single experiment targets one region (the value of `REGION`).

***

### Troubleshooting <a href="#troubleshooting" id="troubleshooting"></a>

<details>

<summary>ECS Fargate memory hog fails with InvalidParameterException task cpu/memory invalid</summary>

Adding the chaos sidecar pushed the task definition outside Fargate's valid CPU/memory combinations. Either lower MEMORY\_IN\_MEGABYTE, increase the task-level memory allocation in the base task definition, or pick a different (CPU, memory) pairing that accommodates the sidecar. Valid combinations are documented in the AWS Fargate task definition guide.

</details>

<details>

<summary>Chaos sidecar is OOM-killed but main container is not</summary>

ECS OOM kills the container that first exceeds its container-level memory limit. If the sidecar has a smaller container-level memory limit than its allocation, it is killed first and the main container is not stressed. Either raise the sidecar's memory limit (it's part of the chaos revision) by increasing MEMORY\_IN\_MEGABYTE relative to the task budget, or pick a smaller task to make the effect visible.

</details>

<details>

<summary>Memory utilization does not rise during the chaos window</summary>

The most common causes are: the deployment never rolled over because maximumPercent prevents new tasks from starting; the sidecar started but exited immediately (check the container's logs); or the task-level memory allocation is so large that MEMORY\_IN\_MEGABYTE does not move the needle. Increase MEMORY\_IN\_MEGABYTE, set maximumPercent to 200, or pick a smaller task.

</details>

<details>

<summary>Service is left on the chaos task definition after the experiment</summary>

If the experiment was killed mid-cleanup, the service may still be on the chaos revision. Run 'aws ecs update-service --cluster --service ~~--task-definition ' to roll back, then 'aws ecs deregister-task-definition --task-definition ' to clean up. The original ARN is recorded in the chaos pod logs.~~

</details>

~~Related faults~~[~~ECS Fargate CPU hog~~](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/aws/ecs-fargate-cpu-hog.md)~~: Stress CPU inside Fargate tasks instead of memory.~~[~~ECS container memory hog~~](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/aws/ecs-container-memory-hog.md)~~: Stress memory inside containers on the EC2 launch type.~~[~~ECS update container resource limit~~](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/aws/ecs-update-container-resource-limit.md)~~: Reduce memory limits to simulate starvation instead of injection.~~[~~Common AWS fault tunables~~](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/aws/aws-fault-tunables.md)~~: Shared environment variables for AWS faults.~~