CF app JVM method latency

Add artificial latency to a specific JVM method inside a Cloud Foundry app instance so you can test how slow downstream calls cascade through the system.

CF app JVM method latency is a Cloud Foundry chaos fault that adds latency milliseconds of delay to every invocation of a specific JVM method (class.method) inside a running Java app. The fault lasts for duration seconds, after which the method runs at its normal speed again.

Use this fault to simulate a slow downstream dependency (database, third-party API, internal service) at the method boundary, and validate timeouts, retries, fallbacks, and circuit-breaker thresholds in callers.

RUN YOUR FIRST EXPERIMENT

If you have not configured the chaos infrastructure yet, go to Quickstart to install the Linux chaos infrastructure and run an experiment end to end.

Use cases

Timeout tuning: Confirm caller timeouts trip before user-visible SLAs are violated.
Retry budget: Validate the caller does not pile on retries that amplify the slowdown.
Circuit-breaker thresholds: Confirm the breaker opens after the configured threshold.
End-to-end SLO: Measure how much a single slow method contributes to the overall request budget.

Before you begin

Chaos infrastructure: A Linux chaos infrastructure (LCI) installed in one of the supported deployment models.
CF and BOSH credentials: The LCI host has CF_*, UAA_SERVER_ENDPOINT, and BOSH_* credentials configured.
Target identifiers: You know the organization, space, app, and the boshDeployment.
Java app: The target app exposes a JVM agent on port (default 9091).
Method signature: You know the fully qualified class and method to target.

Supported environments

Platform

Support status

Java apps deployed to Cloud Foundry

Supported

Non-Java workloads

Not supported

Permissions required

Action

Requirement

List apps the CF user can access

SpaceDeveloper, SpaceAuditor, OrgManager, or OrgAuditor; scopes cloud_controller.read or cloud_controller.admin

List BOSH deployments

BOSH user with bosh.read scope

SSH to a Diego cell via BOSH

BOSH UAA token with bosh.ssh or bosh.admin scope

Attach the JVM agent to the target container

Operator with sudo or root on the cell host

Authentication

Layer

Where to provide

Tunables

Cloud Foundry API + BOSH director

/etc/linux-chaos-infrastructure/cf.env on the LCI host

CF_API_ENDPOINT, CF_USERNAME, CF_PASSWORD, UAA_SERVER_ENDPOINT, BOSH_CLIENT, BOSH_CLIENT_SECRET, BOSH_CA_CERT, BOSH_ENVIRONMENT

vSphere (only when faultInjectorLocation: vSphere)

/etc/linux-chaos-infrastructure/vsphere.env

GOVC_URL, GOVC_USERNAME, GOVC_PASSWORD, GOVC_INSECURE, VM_NAME, VM_USERNAME, VM_PASSWORD

Fault tunables

Required parameters

Tunable

Description

Default

deploymentModel

LCI placement model. One of model-1 or model-2.

(required)

organization

CF organization that owns the app.

(required)

space

CF space within the organization.

(required)

app

Java app whose method is targeted.

(required)

class

Fully qualified class name.

(required)

method

Method on class to instrument.

(required)

Chaos parameters

Tunable

Description

Default

latency

Latency added per method call, in milliseconds.

2000

port

JVM agent port inside the container.

9091

javaHome

Value of JAVA_HOME.

""

instanceAffectedPercentage

Percentage of instances to target. 0 targets exactly one.

0

boshDeployment

BOSH deployment name. Required for deploymentModel: model-2.

""

faultInjectorLocation

local or vSphere. Required for deploymentModel: model-2.

local

faultInjectorPort

Local port used by the fault-injector.

50320

duration

Total chaos duration.

30s

skipSSLValidation

Skip SSL validation when calling CF APIs.

false

rampTime

Wait period in seconds before and after the fault.

0

Tunables that apply to every fault are documented in common tunables for all faults.

Fault execution in brief

Authenticates to Cloud Foundry and BOSH, locates the target app instance(s), attaches an agent to the JVM via the debug port, and installs a rule that sleeps for latency milliseconds before each invocation of class.method completes. The rule is removed when duration elapses.

Expected behavior during fault execution

Every invocation of the targeted method takes at least latency ms longer than usual.
Caller-side response times rise; timeouts may trip if latency exceeds the configured budget.
Thread pools that serve the method may saturate, causing back-pressure on the rest of the app.
After the fault ends, response times return to baseline.

Signals to watch

End-to-end latency: Use an HTTP probe and assert P95 latency.
Thread pool depth: Inspect app-level metrics (active threads, queue length) for back-pressure signals.

Recovery and cleanup

The instrumentation is removed at the end of duration.
Active in-flight calls completed during the fault may still observe the added latency.

Limitations

Matches methods by name only; overloaded methods on the same class all receive the added latency.
Latency is added synchronously inside the method body and counts against the caller's thread.

Troubleshooting

CF app JVM method latency: target method runs at normal speed

The method may not have been invoked during the chaos window, or it was JIT-inlined before the agent attached. Generate traffic that exercises the endpoint, or restart the app instance before re-running.

Latency value seems off

latency is in milliseconds, not seconds. 2000 means 2 seconds. Re-check the value if the perceived delay is far larger than expected.

Common configurations

Add 500 ms to a method

apiVersion: litmuchaos.io/v1alpha1
kind: LinuxFault
metadata:
  name: cf-app-jvm-method-latency
  labels:
    name: app-jvm-method-latency
spec:
  cfAppJVMChaos/inputs:
    duration: 60s
    deploymentModel: model-2
    faultInjectorLocation: vSphere
    app: cf-app
    organization: dev-org
    space: dev-space
    boshDeployment: cf
    class: com.example.OrderService
    method: lookupCustomer
    latency: 500

The following Cloud Foundry secrets reside on the same machine where the chaos infrastructure is executed. These secrets are provided in the /etc/linux-chaos-infrastructure/cf.env file in the following format:

CF_API_ENDPOINT=XXXXXXXXXXXXXXXXXXX
CF_USERNAME=XXXXXXXXXXXXXXXXXXXXXXX
CF_PASSWORD=XXXXXXXXXXXXXXXXXXXXXXX
UAA_SERVER_ENDPOINT=XXXXXXXXXXXXXXX
BOSH_CLIENT=XXXXXXXXXXXXXXXXXXXXXXX
BOSH_CLIENT_SECRET=XXXXXXXXXXXXXXXX
BOSH_CA_CERT=XXXXXXXXXXXXXXXXXXXXXX
BOSH_ENVIRONMENT=XXXXXXXXXXXXXXXXXX

If the secrets file is not provided, the secrets are attempted to be derived from environment variables and the config file by the fault-injector.

ENV name

Description

Example

CF_API_ENDPOINT

API endpoint for the CF setup

https://api.system.cf-setup.com

CF_USERNAME

Username for the CF user

username

CF_PASSWORD

Password for the CF user

password

UAA_SERVER_ENDPOINT

API endpoint for the UAA server for the CF setup

https://uaa.system.cf-setup.com

BOSH_CLIENT

Used by the bosh CLI, the BOSH client

admin

BOSH_CLIENT_SECRET

Used by the bosh CLI, the BOSH client secret

UBu9Fu3oW35sO6fw12auPH76gsRTy7

BOSH_CA_CERT

Used by the bosh CLI, the file path for BOSH CA certificate

/root/root_ca_certificate

BOSH_ENVIRONMENT

Used by the bosh CLI, the BOSH environment

bosh.corp.local

Fault injector ENVs and config file

If /etc/linux-chaos-infrastructure/cf.env file is not provided, fault-injector attempts to derive the secrets from environment variables or a configuration file. Any secret that is re-declared will be overridden in the following order of decreasing precedence:

/etc/linux-chaos-infrastructure/cf.env file
Environment variables
Configuration file

The configuration file should be provided at /etc/linux-chaos-infrastructure/cf-fault-injector.yaml:

cf-api-endpoint: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
username: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
password: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
uaa-server-endpoint: XXXXXXXXXXXXXXXXXXXXXXXXXX
bosh-client: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
bosh-client-secret: XXXXXXXXXXXXXXXXXXXXXXXXXXX
bosh-ca-cert: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
bosh-environment: XXXXXXXXXXXXXXXXXXXXXXXXXXXXX

A mapping between all the three formats for providing the secrets is as follows:

cf.env

ENV

cf-fault-injector.yaml

CF_API_ENDPOINT

cf-api-endpoint

CF_USERNAME

USERNAME

username

CF_PASSWORD

PASSWORD

password

UAA_SERVER_ENDPOINT

uaa-server-endpoint

BOSH_CLIENT

bosh-client

BOSH_CLIENT_SECRET

bosh-client-secret

BOSH_CA_CERT

bosh-ca-cert

BOSH_ENVIRONMENT

bosh-environment

These secrets are provided only if vSphere is used as the deployment platform for CF.

The following vSphere secrets reside on the same machine where the chaos infrastructure is executed. These secrets are provided in the /etc/linux-chaos-infrastructure/vsphere.env file in the following format:

GOVC_URL=XXXXXXXXXXXXXXXXXXXXXX
GOVC_USERNAME=XXXXXXXXXXXXXXXXX
GOVC_PASSWORD=XXXXXXXXXXXXXXXXX
GOVC_INSECURE=XXXXXXXXXXXXXXXXX
VM_NAME=XXXXXXXXXXXXXXXXXXXXXXX
VM_USERNAME=XXXXXXXXXXXXXXXXXXX
VM_PASSWORD=XXXXXXXXXXXXXXXXXXX

ENV Name

Description

Notes

GOVC_URL

Endpoint for vSphere

For example, 192.168.214.244

GOVC_USERNAME

Username for the vSphere user

For example, username

GOVC_PASSWORD

Password for the vSphere user

For example, password

GOVC_INSECURE

Skip SSL validation for govc commands

For example, true

VM_NAME

Name of the vSphere VM where the fault-injector utility is installed

For example, cf-vm

VM_USERNAME

Username for the VM guest user

For example, root

VM_PASSWORD

Password for the VM guest user

For example, password

CF app JVM method exception: Make the same method fail instead of slow.
CF app network latency: Add latency at the network layer instead of inside the JVM.

Last updated 3 days ago

Was this helpful?

Use cases

Before you begin

Supported environments

Permissions required

Authentication

Fault tunables

Fault execution in brief

Expected behavior during fault execution

Signals to watch

Recovery and cleanup

Limitations

Troubleshooting

Common configurations

Add 500 ms to a method

Fault injector ENVs and config file

Related faults