VMware network latency
VMware network latency injects network packet latency from the VMware VM(s) into the application (or service).
- It results in flaky access to the application.
- It causes network degradation without the VM being marked as unhealthy (or unworthy) of traffic.
- It checks the performance of the application (or process) running on the VMware VM(s).
HCE doesn't support injecting VMWare Windows faults on Bare metal server.
Use cases
VMware network latency:
- Simulates issues within the VM network (or microservice) communication across services in different hosts.
- Determines the impact of degradation while accessing a microservice.
- Limits the impact (blast radius) to the traffic that you wish to test by specifying the IP addresses, if the VM stalls or gets corrupted while waiting endlessly for a packet.
- Simulates a consistently slow network connection between microservices, for example, cross-region connectivity between active-active peers of a given service or across services or poor cni-performance in the inter-pod-communication network.
- Simulates jittery connection with transient latency spikes between microservices.
- Simulates slow response on specific third party (or dependent) components (or services).
- Simulates degraded data-plane of service-mesh infrastructure.
Prerequisites
- Kubernetes > 1.16 is required to execute this fault.
- Appropriate vCenter permissions should be provided to start and stop the VMs.
- The VM should be in a healthy state before and after injecting chaos.
- Kubernetes secret has to be created that has the Vcenter credentials in the
CHAOS_NAMESPACE
. VM credentials can be passed as secrets or as aChaosEngine
environment variable. Below is a sample secret file:
apiVersion: v1
kind: Secret
metadata:
name: vcenter-secret
namespace: litmus
type: Opaque
stringData:
VCENTERSERVER: XXXXXXXXXXX
VCENTERUSER: XXXXXXXXXXXXX
VCENTERPASS: XXXXXXXXXXXXX
Mandatory tunables
Tunable | Description | Notes |
---|---|---|
VM_NAMES | Names of the target VMs as comma-separated values. | For example, vm-1,vm-2 . |
VM_USER_NAME | Username of the target VM(s). | Multiple usernames can be provided as comma-separated values which corresponds to more than one VM under chaos. It is used to run the govc command. |
VM_PASSWORD | Password for the target VM(s). | It is used to run the govc command. |
Optional tunables
Tunable | Description | Notes |
---|---|---|
TOTAL_CHAOS_DURATION | Duration that you specify, through which chaos is injected into the target resource (in seconds). | Defaults to 30s. For more information, go to duration of the chaos. |
CHAOS_INTERVAL | Time interval between two successive instance terminations (in seconds). | Defaults to 30s. For more information, go to chaos interval. |
NETWORK_LATENCY | Delay (in milliseconds). Provide numeric value only. | Defaults to 2000 ms. For more information, go to network packet latency. |
JITTER | Network jitter (in milliseconds). Provide numeric value only. | Defaults to 0. For more information, go to run with jitter. |
DESTINATION_IPS | IP addresses of the services or pods whose accessibility you want to affect. You can also specify a CIDR block. | Comma-separated IPs (or CIDRs) can be provided. If it has not been provided, network chaos is induced on all IPs (or destinations). For more information, go to run with destination IPs. |
DESTINATION_HOSTS | DNS names (or FQDN names) of the services whose accessibility is affected. | If it has not been provided, network chaos is induced on all IPs (or destinations). For more information, go to run with destination hosts. |
SOURCE_PORTS | Comma-separated ports of the target application whose accessibility is impacted. If not provided, network chaos is induced on all ports. For example, 5000,8080 . | Alternatively, the source ports to be exempted from the chaos can be provided by prepending a ! to the list of ports. For example, !5000,8080 . For more information, go to source ports. |
DESTINATION_PORTS | Ports of the destination services or pods or the CIDR blocks(range of IPs) whose accessibility is impacted. If not provided, network chaos is induced on all ports. For example, 5000,8080 . | Alternatively, the destination ports to be exempted from the chaos can be provided by prepending a ! to the list of ports. For example, !5000,8080 . For more information, go to destination ports. |
SEQUENCE | Sequence of chaos execution for multiple instances. | Defaults to parallel. Supports serial sequence as well. For more information, go to sequence of chaos execution. |
RAMP_TIME | Period to wait before and after injecting chaos (in seconds). | For example, 30s. For more information, go to ramp time. |
DEFAULT_HEALTH_CHECK | Determines if you wish to run the default health check which is present inside the fault. | Default: 'true'. For more information, go to default health check. |
Secret tunables
Tunable | Description | Notes |
---|---|---|
GOVC_URL | vCenter server URL used to perform API calls using the govc command. | It is derived from a secret. |
GOVC_USERNAME | Username of the vCenter server used for authentication purposes. | It can be set up using a secret. |
GOVC_PASSWORD | Password of the vCenter server used for authentication purposes. | It can be set up using a secret. |
GOVC_INSECURE | Runs the govc command in insecure mode. It is set to true . | It can be set up using a secret. |
If the environment variables DESTINATION_HOSTS
or DESTINATION_IPS
are left empty, the default behaviour is to target all hosts. To limit the impact on all the hosts, you can specify the IP addresses of the service (use commas to separate multiple values) or the DNS or the FQDN names of the services in DESTINATION_HOSTS
.
Network packet latency
Network packet latency injected into the VM. Tune it by using the NETWORK_LATENCY
environment variable.
The following YAML snippet illustrates the use of this environment variable:
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
name: VMware-engine
spec:
engineState: "active"
chaosServiceAccount: litmus-admin
experiments:
- name: VMware-network-latency
spec:
components:
env:
# network packet latency
- name: NETWORK_LATENCY
value: '2000'
- name: VM_NAME
value: 'vm-1,vm-2'
- name: VM_USER_NAME
value: 'ubuntu,debian'
- name: VM_PASSWORD
value: '123,123'
Run with jitter
Parameter that introduces network delay variation (in milliseconds). Tune it by using the JITTER
environment variable. Its default value is 0.
The following YAML snippet illustrates the use of this environment variable:
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
name: VMware-engine
spec:
engineState: "active"
chaosServiceAccount: litmus-admin
experiments:
- name: VMware-network-latency
spec:
components:
env:
# value of the network latency jitter (in ms)
- name: JITTER
value: '200'
- name: NETWORK_LATENCY
value: '2000'
- name: VM_NAME
value: 'vm-1,vm-2'
- name: VM_USER_NAME
value: 'ubuntu,debian'
- name: VM_PASSWORD
value: '123,123'
Run with destination IPs and destination hosts
IPs/hosts that interrupt traffic by default. Tune it by using the DESTINATION_IPS
and DESTINATION_HOSTS
environment variables, respectively.
DESTINATION_IPS
: IP addresses of the services or the CIDR blocks (range of IPs) whose accessibility is impacted.
DESTINATION_HOSTS
: DNS names of the services whose accessibility is impacted.
The following YAML snippet illustrates the use of this environment variable:
## it injects the chaos for the egress traffic for specific ips/hosts
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
name: VMware-engine
spec:
engineState: "active"
chaosServiceAccount: litmus-admin
experiments:
- name: VMware-network-latency
spec:
components:
env:
# supports comma separated destination ips
- name: DESTINATION_IPS
value: '8.8.8.8,192.168.5.6'
# supports comma separated destination hosts
- name: DESTINATION_HOSTS
value: 'google.com'
- name: VM_NAME
value: 'vm-1,vm-2'
- name: VM_USER_NAME
value: 'ubuntu,debian'
- name: VM_PASSWORD
value: '123,123'
Source and destination ports
By default, the network experiments disrupt traffic for all the source and destination ports. Tune the interruption of specific port(s) using the SOURCE_PORTS
and DESTINATION_PORTS
environment variables, respectively.
SOURCE_PORTS
: Ports of the target application whose accessibility is impacted.DESTINATION_PORTS
: Ports of the destination services or pods or the CIDR blocks(range of IPs) whose accessibility is impacted.
The following YAML snippet illustrates the use of this environment variable:
# it inject the chaos for the egress traffic for specific ports
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
name: VMware-engine
spec:
engineState: "active"
annotationCheck: "false"
chaosServiceAccount: litmus-admin
experiments:
- name: vmware-network-latency
spec:
components:
env:
# supports comma separated source ports
- name: SOURCE_PORTS
value: '80'
# supports comma separated destination ports
- name: DESTINATION_PORTS
value: '8080,9000'
- name: TOTAL_CHAOS_DURATION
value: '60'
Ignore source and destination ports
By default, the network experiments disrupt traffic for all the source and destination ports. Ignore specific port(s) using the SOURCE_PORTS
and DESTINATION_PORTS
environment variables, respectively.
-
SOURCE_PORTS
: Source ports that are not subject to chaos as comma-separated values preceded by!
. -
DESTINATION_PORTS
: Destination ports that are not subject to chaos as comma-separated values preceded by!
.
The following YAML snippet illustrates the use of this environment variable:
# ignore the source and destination ports
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
name: engine-nginx
spec:
engineState: "active"
annotationCheck: "false"
chaosServiceAccount: litmus-admin
experiments:
- name: vmware-network-latency
spec:
components:
env:
# it will ignore 80 and 8080 source ports
- name: SOURCE_PORTS
value: '!80,8080'
# it will ignore 8080 and 9000 destination ports
- name: DESTINATION_PORTS
value: '!8080,9000'
- name: TOTAL_CHAOS_DURATION
value: '60'
Network interface
Name of the ethernet interface that shapes the traffic. Tune it by using the NETWORK_INTERFACE
environment variable. Its default value is eth0
.
The following YAML snippet illustrates the use of this environment variable:
## it injects the chaos for the egress traffic for specific ips/hosts
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
name: VMware-engine
spec:
engineState: "active"
chaosServiceAccount: litmus-admin
experiments:
- name: VMware-network-latency
spec:
components:
env:
# name of the network interface
- name: NETWORK_INTERFACE
value: 'eth0'
- name: VM_NAME
value: 'vm-1,vm-2'
- name: VM_USER_NAME
value: 'ubuntu,debian'
- name: VM_PASSWORD
value: '123,123'