Ctrlk
Sign UpGet a DemoContact Us
For the complete documentation index, see llms.txt. This page is also available as Markdown.

CF app network latency

Inject network latency on the egress of a Cloud Foundry app instance so you can test how the app and its callers behave when downstream calls become slow.

CF app network latency is a Cloud Foundry chaos fault that adds networkLatency milliseconds of delay (with optional jitter) on the egress traffic of one or more app instances of app in organization/space. You can restrict the delay to specific destination hosts, IP ranges, source ports, or destination ports. The fault lasts for duration seconds.

Use this fault to simulate a slow downstream dependency at the network layer, validate timeouts and retries inside the app, and quantify the end-to-end latency cost contributed by a single slow link.

RUN YOUR FIRST EXPERIMENT

If you have not configured the chaos infrastructure yet, go to Quickstart to install the Linux chaos infrastructure and run an experiment end to end.

Use cases

  • Slow downstream rehearsal: Simulate a slow database or third-party API and validate the app's timeout configuration.

  • Tail-latency budget: Measure how an extra 100/500/1000 ms downstream impacts the user-visible P99.

  • Geo-latency simulation: Approximate cross-region call latency to test caching and locality decisions.

  • Targeted partition: Slow traffic only to a specific dependency without affecting the rest of the app.

Before you begin

  • Chaos infrastructure: A Linux chaos infrastructure (LCI) installed in one of the supported deployment models.

  • CF and BOSH credentials: The LCI host has CF_*, UAA_SERVER_ENDPOINT, and BOSH_* credentials configured.

  • Target identifiers: You know the organization, space, app, and the boshDeployment.

  • Network interface: You know the interface name inside the container (typically eth0).

  • Destinations: You know the host(s), IP range(s), or port(s) you want to slow down. Leave blank to affect all egress.

Supported environments

Platform
Support status

Cloud Foundry (TAS, PCF, open-source) running on BOSH-managed Diego cells

Supported

Permissions required

Action
Requirement

List apps the CF user can access

SpaceDeveloper, SpaceAuditor, OrgManager, or OrgAuditor; scopes cloud_controller.read or cloud_controller.admin

List BOSH deployments

BOSH user with bosh.read scope

SSH to a Diego cell via BOSH

BOSH UAA token with bosh.ssh or bosh.admin scope

Apply traffic-shaping rules inside the target container's namespace

Operator with sudo or root on the cell host

Authentication

Layer
Where to provide
Tunables

Cloud Foundry API + BOSH director

/etc/linux-chaos-infrastructure/cf.env on the LCI host

CF_API_ENDPOINT, CF_USERNAME, CF_PASSWORD, UAA_SERVER_ENDPOINT, BOSH_CLIENT, BOSH_CLIENT_SECRET, BOSH_CA_CERT, BOSH_ENVIRONMENT

vSphere (only when faultInjectorLocation: vSphere)

/etc/linux-chaos-infrastructure/vsphere.env

GOVC_URL, GOVC_USERNAME, GOVC_PASSWORD, GOVC_INSECURE, VM_NAME, VM_USERNAME, VM_PASSWORD

Fault tunables

Required parameters

Tunable
Description
Default

deploymentModel

LCI placement model. One of model-1 or model-2.

(required)

organization

CF organization that owns the app.

(required)

space

CF space within the organization.

(required)

app

App whose egress traffic is delayed.

(required)

Chaos parameters

Tunable
Description
Default

networkLatency

Added latency in milliseconds.

2000

jitter

Random variation around networkLatency, in milliseconds.

0

networkInterface

Network interface inside the container.

eth0

destinationHosts

Comma-separated list of hostnames to affect. Leave empty to affect all egress.

""

destinationIPs

Comma-separated list of destination IPs or CIDRs to affect.

""

sourcePorts

Comma-separated list of source ports to affect (egress from these ports).

""

destinationPorts

Comma-separated list of destination ports to affect.

""

instanceAffectedPercentage

Percentage of instances to target. 0 targets exactly one.

0

boshDeployment

BOSH deployment name. Required for deploymentModel: model-2.

""

faultInjectorLocation

local or vSphere. Required for deploymentModel: model-2.

local

faultInjectorPort

Local port used by the fault-injector.

50320

duration

Total chaos duration.

30s

skipSSLValidation

Skip SSL validation when calling CF APIs.

false

rampTime

Wait period in seconds before and after the fault.

0

Tunables that apply to every fault are documented in common tunables for all faults.

Fault execution in brief

Authenticates to Cloud Foundry and BOSH, locates the target app instance(s), and installs a traffic-shaping rule on networkInterface inside the container's network namespace that delays packets matching the configured destinations/ports by networkLatency ms (±jitter). The rule is removed when duration elapses.

Expected behavior during fault execution

  • TCP round-trip times to the affected destinations rise by approximately networkLatency ms.

  • Caller-side response times reflect the added delay; retries may stack if timeouts are too short.

  • Throughput typically falls in proportion to the added round-trip time.

  • After the fault ends, latency returns to baseline.

Signals to watch

  • App latency: Use an HTTP probe on a public route.

  • Downstream metrics: Inspect the app's outbound call metrics (response time histograms) for the affected dependency.

Recovery and cleanup

  • The traffic-shaping rule is removed at the end of duration.

  • If the experiment is aborted, the fault still attempts to remove the rule before exiting.

Limitations

  • Operates on the egress side of the affected instance only. Ingress traffic is not delayed.

  • Without a destination filter (destinationHosts, destinationIPs, or port filters), ALL egress is affected, which may also slow control-plane health checks.

Troubleshooting

CF app network latency: no observable delay in Harness Chaos Engineering

Confirm networkInterface matches the interface inside the container (cf ssh -c 'ip a'). Confirm destinationHosts or destinationIPs actually match the traffic you want to slow.

App health checks fail during the experiment

Without destination filters, control-plane traffic is also delayed. Add destinationHosts or destinationIPs to scope the delay to your dependency only.

Common configurations

Slow only one downstream

The following Cloud Foundry secrets reside on the same machine where the chaos infrastructure is executed. These secrets are provided in the /etc/linux-chaos-infrastructure/cf.env file in the following format:

If the secrets file is not provided, the secrets are attempted to be derived from environment variables and the config file by the fault-injector.

ENV name
Description
Example

CF_API_ENDPOINT

API endpoint for the CF setup

https://api.system.cf-setup.com

CF_USERNAME

Username for the CF user

username

CF_PASSWORD

Password for the CF user

password

UAA_SERVER_ENDPOINT

API endpoint for the UAA server for the CF setup

https://uaa.system.cf-setup.com

BOSH_CLIENT

Used by the bosh CLI, the BOSH client

admin

BOSH_CLIENT_SECRET

Used by the bosh CLI, the BOSH client secret

UBu9Fu3oW35sO6fw12auPH76gsRTy7

BOSH_CA_CERT

Used by the bosh CLI, the file path for BOSH CA certificate

/root/root_ca_certificate

BOSH_ENVIRONMENT

Used by the bosh CLI, the BOSH environment

bosh.corp.local

Fault injector ENVs and config file

If /etc/linux-chaos-infrastructure/cf.env file is not provided, fault-injector attempts to derive the secrets from environment variables or a configuration file. Any secret that is re-declared will be overridden in the following order of decreasing precedence:

  1. /etc/linux-chaos-infrastructure/cf.env file

  2. Environment variables

  3. Configuration file

The configuration file should be provided at /etc/linux-chaos-infrastructure/cf-fault-injector.yaml:

A mapping between all the three formats for providing the secrets is as follows:

cf.env
ENV
cf-fault-injector.yaml

CF_API_ENDPOINT

CF_API_ENDPOINT

cf-api-endpoint

CF_USERNAME

USERNAME

username

CF_PASSWORD

PASSWORD

password

UAA_SERVER_ENDPOINT

UAA_SERVER_ENDPOINT

uaa-server-endpoint

BOSH_CLIENT

BOSH_CLIENT

bosh-client

BOSH_CLIENT_SECRET

BOSH_CLIENT_SECRET

bosh-client-secret

BOSH_CA_CERT

BOSH_CA_CERT

bosh-ca-cert

BOSH_ENVIRONMENT

BOSH_ENVIRONMENT

bosh-environment

These secrets are provided only if vSphere is used as the deployment platform for CF.

The following vSphere secrets reside on the same machine where the chaos infrastructure is executed. These secrets are provided in the /etc/linux-chaos-infrastructure/vsphere.env file in the following format:

ENV Name
Description
Notes

GOVC_URL

Endpoint for vSphere

For example, 192.168.214.244

GOVC_USERNAME

Username for the vSphere user

For example, username

GOVC_PASSWORD

Password for the vSphere user

For example, password

GOVC_INSECURE

Skip SSL validation for govc commands

For example, true

VM_NAME

Name of the vSphere VM where the fault-injector utility is installed

For example, cf-vm

VM_USERNAME

Username for the VM guest user

For example, root

VM_PASSWORD

Password for the VM guest user

For example, password

Related faults

Last updated

Was this helpful?