> For the complete documentation index, see [llms.txt](https://developer.harness.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://developer.harness.io/resilience-testing/chaos-engineering/faults/chaos-fault-categories/cloud-foundry/cf-app-jvm-method-latency.md).

# CF app JVM method latency

CF app JVM method latency is a Cloud Foundry chaos fault that adds `latency` milliseconds of delay to every invocation of a specific JVM method (`class`.`method`) inside a running Java app. The fault lasts for `duration` seconds, after which the method runs at its normal speed again.

Use this fault to simulate a slow downstream dependency (database, third-party API, internal service) at the method boundary, and validate timeouts, retries, fallbacks, and circuit-breaker thresholds in callers.

{% hint style="info" %}
**RUN YOUR FIRST EXPERIMENT**

If you have not configured the chaos infrastructure yet, go to [Quickstart](/resilience-testing/chaos-engineering/new-to-chaos-engineering/quickstart.md) to install the Linux chaos infrastructure and run an experiment end to end.
{% endhint %}

***

### Use cases <a href="#use-cases" id="use-cases"></a>

* **Timeout tuning:** Confirm caller timeouts trip before user-visible SLAs are violated.
* **Retry budget:** Validate the caller does not pile on retries that amplify the slowdown.
* **Circuit-breaker thresholds:** Confirm the breaker opens after the configured threshold.
* **End-to-end SLO:** Measure how much a single slow method contributes to the overall request budget.

***

### Before you begin <a href="#before-you-begin" id="before-you-begin"></a>

* **Chaos infrastructure:** A Linux chaos infrastructure (LCI) installed in one of the supported deployment models.
* **CF and BOSH credentials:** The LCI host has `CF_*`, `UAA_SERVER_ENDPOINT`, and `BOSH_*` credentials configured.
* **Target identifiers:** You know the `organization`, `space`, `app`, and the `boshDeployment`.
* **Java app:** The target app exposes a JVM agent on `port` (default `9091`).
* **Method signature:** You know the fully qualified class and method to target.

***

### Supported environments <a href="#supported-environments" id="supported-environments"></a>

| Platform                            | Support status |
| ----------------------------------- | -------------- |
| Java apps deployed to Cloud Foundry | Supported      |
| Non-Java workloads                  | Not supported  |

***

### Permissions required <a href="#permissions-required" id="permissions-required"></a>

| Action                                       | Requirement                                                                                                                 |
| -------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------- |
| List apps the CF user can access             | `SpaceDeveloper`, `SpaceAuditor`, `OrgManager`, or `OrgAuditor`; scopes `cloud_controller.read` or `cloud_controller.admin` |
| List BOSH deployments                        | BOSH user with `bosh.read` scope                                                                                            |
| SSH to a Diego cell via BOSH                 | BOSH UAA token with `bosh.ssh` or `bosh.admin` scope                                                                        |
| Attach the JVM agent to the target container | Operator with `sudo` or root on the cell host                                                                               |

***

### Authentication <a href="#authentication" id="authentication"></a>

| Layer                                                | Where to provide                                         | Tunables                                                                                                                                        |
| ---------------------------------------------------- | -------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- |
| Cloud Foundry API + BOSH director                    | `/etc/linux-chaos-infrastructure/cf.env` on the LCI host | `CF_API_ENDPOINT`, `CF_USERNAME`, `CF_PASSWORD`, `UAA_SERVER_ENDPOINT`, `BOSH_CLIENT`, `BOSH_CLIENT_SECRET`, `BOSH_CA_CERT`, `BOSH_ENVIRONMENT` |
| vSphere (only when `faultInjectorLocation: vSphere`) | `/etc/linux-chaos-infrastructure/vsphere.env`            | `GOVC_URL`, `GOVC_USERNAME`, `GOVC_PASSWORD`, `GOVC_INSECURE`, `VM_NAME`, `VM_USERNAME`, `VM_PASSWORD`                                          |

***

### Fault tunables <a href="#fault-tunables" id="fault-tunables"></a>

**Required parameters**

| Tunable           | Description                                         | Default    |
| ----------------- | --------------------------------------------------- | ---------- |
| `deploymentModel` | LCI placement model. One of `model-1` or `model-2`. | (required) |
| `organization`    | CF organization that owns the app.                  | (required) |
| `space`           | CF space within the organization.                   | (required) |
| `app`             | Java app whose method is targeted.                  | (required) |
| `class`           | Fully qualified class name.                         | (required) |
| `method`          | Method on `class` to instrument.                    | (required) |

**Chaos parameters**

| Tunable                      | Description                                                    | Default |
| ---------------------------- | -------------------------------------------------------------- | ------- |
| `latency`                    | Latency added per method call, in milliseconds.                | `2000`  |
| `port`                       | JVM agent port inside the container.                           | `9091`  |
| `javaHome`                   | Value of `JAVA_HOME`.                                          | `""`    |
| `instanceAffectedPercentage` | Percentage of instances to target. `0` targets exactly one.    | `0`     |
| `boshDeployment`             | BOSH deployment name. Required for `deploymentModel: model-2`. | `""`    |
| `faultInjectorLocation`      | `local` or `vSphere`. Required for `deploymentModel: model-2`. | `local` |
| `faultInjectorPort`          | Local port used by the fault-injector.                         | `50320` |
| `duration`                   | Total chaos duration.                                          | `30s`   |
| `skipSSLValidation`          | Skip SSL validation when calling CF APIs.                      | `false` |
| `rampTime`                   | Wait period in seconds before and after the fault.             | `0`     |

Tunables that apply to every fault are documented in [common tunables for all faults](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/common-tunables-for-all-faults.md).

***

### Fault execution in brief <a href="#fault-execution-in-brief" id="fault-execution-in-brief"></a>

Authenticates to Cloud Foundry and BOSH, locates the target app instance(s), attaches an agent to the JVM via the debug port, and installs a rule that sleeps for `latency` milliseconds before each invocation of `class.method` completes. The rule is removed when `duration` elapses.

***

### Expected behavior during fault execution <a href="#expected-behavior-during-fault-execution" id="expected-behavior-during-fault-execution"></a>

* Every invocation of the targeted method takes at least `latency` ms longer than usual.
* Caller-side response times rise; timeouts may trip if `latency` exceeds the configured budget.
* Thread pools that serve the method may saturate, causing back-pressure on the rest of the app.
* After the fault ends, response times return to baseline.

#### Signals to watch <a href="#signals-to-watch" id="signals-to-watch"></a>

* **End-to-end latency:** Use an [HTTP probe](/resilience-testing/chaos-engineering/use-chaos-engineering/probes/http-probe.md) and assert P95 latency.
* **Thread pool depth:** Inspect app-level metrics (active threads, queue length) for back-pressure signals.

***

### Recovery and cleanup <a href="#recovery-and-cleanup" id="recovery-and-cleanup"></a>

* The instrumentation is removed at the end of `duration`.
* Active in-flight calls completed during the fault may still observe the added latency.

***

### Limitations <a href="#limitations" id="limitations"></a>

* Matches methods by name only; overloaded methods on the same class all receive the added latency.
* Latency is added synchronously inside the method body and counts against the caller's thread.

***

### Troubleshooting <a href="#troubleshooting" id="troubleshooting"></a>

<details>

<summary>CF app JVM method latency: target method runs at normal speed</summary>

The method may not have been invoked during the chaos window, or it was JIT-inlined before the agent attached. Generate traffic that exercises the endpoint, or restart the app instance before re-running.

</details>

<details>

<summary>Latency value seems off</summary>

latency is in milliseconds, not seconds. 2000 means 2 seconds. Re-check the value if the perceived delay is far larger than expected.

</details>

***

### Common configurations <a href="#common-configurations" id="common-configurations"></a>

#### Add 500 ms to a method <a href="#add-500-ms-to-a-method" id="add-500-ms-to-a-method"></a>

```yaml
apiVersion: litmuchaos.io/v1alpha1
kind: LinuxFault
metadata:
  name: cf-app-jvm-method-latency
  labels:
    name: app-jvm-method-latency
spec:
  cfAppJVMChaos/inputs:
    duration: 60s
    deploymentModel: model-2
    faultInjectorLocation: vSphere
    app: cf-app
    organization: dev-org
    space: dev-space
    boshDeployment: cf
    class: com.example.OrderService
    method: lookupCustomer
    latency: 500
```

***

The following Cloud Foundry secrets reside on the same machine where the chaos infrastructure is executed. These secrets are provided in the `/etc/linux-chaos-infrastructure/cf.env` file in the following format:

```env
CF_API_ENDPOINT=XXXXXXXXXXXXXXXXXXX
CF_USERNAME=XXXXXXXXXXXXXXXXXXXXXXX
CF_PASSWORD=XXXXXXXXXXXXXXXXXXXXXXX
UAA_SERVER_ENDPOINT=XXXXXXXXXXXXXXX
BOSH_CLIENT=XXXXXXXXXXXXXXXXXXXXXXX
BOSH_CLIENT_SECRET=XXXXXXXXXXXXXXXX
BOSH_CA_CERT=XXXXXXXXXXXXXXXXXXXXXX
BOSH_ENVIRONMENT=XXXXXXXXXXXXXXXXXX
```

{% hint style="info" %}
If the secrets file is not provided, the secrets are attempted to be derived from environment variables and the config file by the fault-injector.
{% endhint %}

| ENV name              | Description                                                   | Example                           |
| --------------------- | ------------------------------------------------------------- | --------------------------------- |
| CF\_API\_ENDPOINT     | API endpoint for the CF setup                                 | `https://api.system.cf-setup.com` |
| CF\_USERNAME          | Username for the CF user                                      | `username`                        |
| CF\_PASSWORD          | Password for the CF user                                      | `password`                        |
| UAA\_SERVER\_ENDPOINT | API endpoint for the UAA server for the CF setup              | `https://uaa.system.cf-setup.com` |
| BOSH\_CLIENT          | Used by the `bosh` CLI, the BOSH client                       | `admin`                           |
| BOSH\_CLIENT\_SECRET  | Used by the `bosh` CLI, the BOSH client secret                | `UBu9Fu3oW35sO6fw12auPH76gsRTy7`  |
| BOSH\_CA\_CERT        | Used by the `bosh` CLI, the file path for BOSH CA certificate | `/root/root_ca_certificate`       |
| BOSH\_ENVIRONMENT     | Used by the `bosh` CLI, the BOSH environment                  | `bosh.corp.local`                 |

## Fault injector ENVs and config file <a href="#fault-injector-envs-and-config-file" id="fault-injector-envs-and-config-file"></a>

If `/etc/linux-chaos-infrastructure/cf.env` file is not provided, fault-injector attempts to derive the secrets from environment variables or a configuration file. Any secret that is re-declared will be overridden in the following order of decreasing precedence:

1. `/etc/linux-chaos-infrastructure/cf.env` file
2. Environment variables
3. Configuration file

The configuration file should be provided at `/etc/linux-chaos-infrastructure/cf-fault-injector.yaml`:

```yaml
cf-api-endpoint: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
username: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
password: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
uaa-server-endpoint: XXXXXXXXXXXXXXXXXXXXXXXXXX
bosh-client: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
bosh-client-secret: XXXXXXXXXXXXXXXXXXXXXXXXXXX
bosh-ca-cert: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
bosh-environment: XXXXXXXXXXXXXXXXXXXXXXXXXXXXX
```

A mapping between all the three formats for providing the secrets is as follows:

| cf.env                | ENV                   | cf-fault-injector.yaml |
| --------------------- | --------------------- | ---------------------- |
| CF\_API\_ENDPOINT     | CF\_API\_ENDPOINT     | cf-api-endpoint        |
| CF\_USERNAME          | USERNAME              | username               |
| CF\_PASSWORD          | PASSWORD              | password               |
| UAA\_SERVER\_ENDPOINT | UAA\_SERVER\_ENDPOINT | uaa-server-endpoint    |
| BOSH\_CLIENT          | BOSH\_CLIENT          | bosh-client            |
| BOSH\_CLIENT\_SECRET  | BOSH\_CLIENT\_SECRET  | bosh-client-secret     |
| BOSH\_CA\_CERT        | BOSH\_CA\_CERT        | bosh-ca-cert           |
| BOSH\_ENVIRONMENT     | BOSH\_ENVIRONMENT     | bosh-environment       |

These secrets are provided only if vSphere is used as the deployment platform for CF.

The following vSphere secrets reside on the same machine where the chaos infrastructure is executed. These secrets are provided in the `/etc/linux-chaos-infrastructure/vsphere.env` file in the following format:

```env
GOVC_URL=XXXXXXXXXXXXXXXXXXXXXX
GOVC_USERNAME=XXXXXXXXXXXXXXXXX
GOVC_PASSWORD=XXXXXXXXXXXXXXXXX
GOVC_INSECURE=XXXXXXXXXXXXXXXXX
VM_NAME=XXXXXXXXXXXXXXXXXXXXXXX
VM_USERNAME=XXXXXXXXXXXXXXXXXXX
VM_PASSWORD=XXXXXXXXXXXXXXXXXXX
```

| ENV Name       | Description                                                          | Notes                          |
| -------------- | -------------------------------------------------------------------- | ------------------------------ |
| GOVC\_URL      | Endpoint for vSphere                                                 | For example, `192.168.214.244` |
| GOVC\_USERNAME | Username for the vSphere user                                        | For example, `username`        |
| GOVC\_PASSWORD | Password for the vSphere user                                        | For example, `password`        |
| GOVC\_INSECURE | Skip SSL validation for govc commands                                | For example, `true`            |
| VM\_NAME       | Name of the vSphere VM where the fault-injector utility is installed | For example, `cf-vm`           |
| VM\_USERNAME   | Username for the VM guest user                                       | For example, `root`            |
| VM\_PASSWORD   | Password for the VM guest user                                       | For example, `password`        |

***

### Related faults <a href="#related-faults" id="related-faults"></a>

* [CF app JVM method exception](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/cloud-foundry/cf-app-jvm-method-exception.md): Make the same method fail instead of slow.
* [CF app network latency](/resilience-testing/chaos-engineering/faults/chaos-fault-categories/cloud-foundry/cf-app-network-latency.md): Add latency at the network layer instead of inside the JVM.