VMware Network Latency
Introduction
- It injects network packet latency from the VMware VM(s) into the application (or service) and results in flaky access.
- It checks the performance of the application (or process) running on the VMWare VM(s).
Uses
View the uses of the fault
Mitigation (in this case, keeping the timeout i.e, the network latency low) can be achieved using a middleware that can switch traffic based on certain SLOs/performance parameters. If such an arrangement is not available, the next best solution would be to verify if a degradation is highlighted by notifying about it using alerts so that the admin (or SRE) has the opportunity to investigate and fix these issues.
Another utility of the test is to see the extent of impact caused to the end-user or the last point in the application stack on account of degradation in accessing a downstream/dependent microservice; whether it accepts or breaks the system to an unacceptable degree. The fault provides DESTINATION_IPS
or DESTINATION_HOSTS
so that you can control the chaos against specific services within or outside the VM.
The VM may stall or get corrupted while it waits endlessly for a packet. The fault limits the impact (blast radius) to only the traffic you wish to test by specifying the IP addresses or application information. This fault helps improve the resilience of your services over time.
Prerequisites
- Kubernetes > 1.16
- Vcenter access to stop and start the VM.
- Kubernetes secret that has Vcenter credentials in the
CHAOS_NAMESPACE
. A sample secret file looks like:
apiVersion: v1
kind: Secret
metadata:
name: vcenter-secret
namespace: litmus
type: Opaque
stringData:
VCENTERSERVER: XXXXXXXXXXX
VCENTERUSER: XXXXXXXXXXXXX
VCENTERPASS: XXXXXXXXXXXXX
NOTE
You can pass the VM credentials as a secret or as a chaosengine environment variable.
Default Validations
- The VM should be in a healthy state before and after chaos.
Fault Tunables
Check the Fault Tunables
Mandatory Fields
Variables | Description | Notes |
---|---|---|
VM_NAMES | Provide the target VM names. | You can provide multiple VM names as comma separated values, for example: vm-1,vm-2. |
VM_USER_NAME | Provide the username of the target VM(s). | Multiple usernames can be provided as comma separated values (for more than one VM under chaos). It is used to run the 'govc' command. |
VM_PASSWORD | Provide the password for the target VM(s). | It is used to run the govc command. |
Optional Fields
Variables | Description | Notes |
---|---|---|
TOTAL_CHAOS_DURATION | The total duration to insert chaos (in seconds). | Its default value is 30s. |
CHAOS_INTERVAL | The interval between successive instance terminations (in seconds). | Its default value is 30s. |
NETWORK_LATENCY | The latency (or delay) in milliseconds. | Its default value is 2000, and it accepts numeric values only. |
JITTER | The network jitter value in milliseconds. | Its default value is 0, and it accepts numeric values only. |
DESTINATION_IPS | The IP addresses of the services or the CIDR blocks(range of IPs), whose accessibility is impacted. | Comma separated IP(S) or CIDR(S) can be provided. If it is not provided, it induces network chaos for all IPs/destinations. |
DESTINATION_HOSTS | DNS Names of the services whose accessibility is impacted. | If it is not provided, it induces network chaos for all IPs/destinations or `DESTINATION_IPS` if already defined. |
SEQUENCE | It defines the sequence of chaos execution for multiple instances. | Its default value is 'parallel', and it supports 'serial' value too. |
RAMP_TIME | Period to wait before and after injection of chaos (in seconds). | For example, 30s. |
Secret Fields
Variables | Description | Notes |
---|---|---|
GOVC_URL | Provide the VMCenter Server URL. | It is used to perform the VMware API calls using the 'govc' command and is derived from a secret. |
GOVC_USERNAME | Provide the username of VMCenter Server. | This environment variable is used for authentiation purposes and is setup using a secret. |
GOVC_PASSWORD | Provide the password of VMCenter Server. | This environment variable is used for authentiation purposes and is setup using a secret. |
GOVC_INSECURE | Provide the value as true . | This environment variable is used to run the 'govc' command in insecure mode and is setup using a secret. |
Fault Examples
Common Fault Tunables
Refer to the common attributes to tune the common tunables for all the faults.
Network Packet Latency
It defines the network packet latency that is injected to the VM. You can tune it using the NETWORK_LATENCY
environment variable.
Use the following example to tune it:
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
name: vmware-engine
spec:
engineState: "active"
chaosServiceAccount: litmus-admin
experiments:
- name: vmware-network-latency
spec:
components:
env:
# network packet latency
- name: NETWORK_LATENCY
value: '2000'
- name: VM_NAME
value: 'vm-1,vm-2'
- name: VM_USER_NAME
value: 'ubuntu,debian'
- name: VM_PASSWORD
value: '123,123'
Run With Jitter
It defines jitter (in ms), a parameter that introduces a network delay variation. You can tune it using the JITTER
environment variable. Its default value is 0.
Use the following example to tune it:
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
name: vmware-engine
spec:
engineState: "active"
chaosServiceAccount: litmus-admin
experiments:
- name: vmware-network-latency
spec:
components:
env:
# value of the network latency jitter (in ms)
- name: JITTER
value: '200'
- name: NETWORK_LATENCY
value: '2000'
- name: VM_NAME
value: 'vm-1,vm-2'
- name: VM_USER_NAME
value: 'ubuntu,debian'
- name: VM_PASSWORD
value: '123,123'
Run With Destination IPs And Destination Hosts
The network faults interrupt traffic for all the IPs/hosts by default. You can tune this using the DESTINATION_IPS
and DESTINATION_HOSTS
environment variables.
DESTINATION_IPS
: It contains the IP addresses of the services or the CIDR blocks(range of IPs) that impacts its accessibility.
DESTINATION_HOSTS
: It contains the DNS Names of the services that impacts its accessibility.
Use the following example to tune it:
## it injects the chaos for the egress traffic for specific ips/hosts
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
name: vmware-engine
spec:
engineState: "active"
chaosServiceAccount: litmus-admin
experiments:
- name: vmware-network-latency
spec:
components:
env:
# supports comma separated destination ips
- name: DESTINATION_IPS
value: '8.8.8.8,192.168.5.6'
# supports comma separated destination hosts
- name: DESTINATION_HOSTS
value: 'google.com'
- name: VM_NAME
value: 'vm-1,vm-2'
- name: VM_USER_NAME
value: 'ubuntu,debian'
- name: VM_PASSWORD
value: '123,123'
Network Interface
The name of the ethernet interface that shapes the traffic. You can tune it using the NETWORK_INTERFACE
environment variable. Its default value is eth0
.
Use the following example to tune it:
## it injects the chaos for the egress traffic for specific ips/hosts
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
name: vmware-engine
spec:
engineState: "active"
chaosServiceAccount: litmus-admin
experiments:
- name: vmware-network-latency
spec:
components:
env:
# name of the network interface
- name: NETWORK_INTERFACE
value: 'eth0'
- name: VM_NAME
value: 'vm-1,vm-2'
- name: VM_USER_NAME
value: 'ubuntu,debian'
- name: VM_PASSWORD
value: '123,123'