VMware network rate limit
VMware network rate limit fault injects network rate limit from the VMware VM(s) into the application (or service). This results in flaky access to the application. It checks the performance of the application (or process) running on the VMware VM(s).
HCE doesn't support injecting VMWare Windows faults on Bare metal server.
Use cases
VMware network rate limit:
- Simulates issues within the VM network (or microservice) communication across services in different hosts.
- Determines the impact of degradation while accessing a microservice.
- Simulates network congestion by artificially limiting the available bandwidth to understand how the system behaves under reduced network capacity.
- Helps assess the impact of network rate limits on the quality of service (QoS) and compliance with service level agreements (SLAs) by observing the system's response time and performance.
- Validates whether rate-limiting mechanisms in your application or network infrastructure are functioning correctly and effectively.
- Helps assess how the system performs when network resources are constrained, mimicking real-world scenarios where network connectivity may be compromised during a disaster or outage.
- Determines how the system prioritizes and handles different types of traffic when network rate limits are in place, ensuring that critical services receive preferential treatment.
Prerequisites
- Kubernetes > 1.16 is required to execute this fault.
- Appropriate vCenter permissions should be provided to start and stop the VMs.
- The VM should be in a healthy state before and after injecting chaos.
- Kubernetes secret has to be created that has the Vcenter credentials in the
CHAOS_NAMESPACE
. VM credentials can be passed as secrets or as aChaosEngine
environment variable. Below is a sample secret file:
apiVersion: v1
kind: Secret
metadata:
name: vcenter-secret
namespace: litmus
type: Opaque
stringData:
VCENTERSERVER: XXXXXXXXXXX
VCENTERUSER: XXXXXXXXXXXXX
VCENTERPASS: XXXXXXXXXXXXX
Mandatory tunables
Tunable | Description | Notes |
---|---|---|
VM_NAME | Comma-separated names of the target VMs. | For example, vm-1,vm-2 . |
VM_USER_NAME | Username of the target VM(s). | Multiple usernames can be provided as comma-separated values which corresponds to more than one VM under chaos. It is used to run the govc command. |
VM_PASSWORD | Password for the target VM(s). | It is used to run the govc command. |
NETWORK_INTERFACE | Name of the ethernet interface considered for shaping traffic. | For example, ens160 . For more information, go to network interface. |
Optional tunables
Tunable | Description | Notes |
---|---|---|
TOTAL_CHAOS_DURATION | Duration that you specify, through which chaos is injected into the target resource (in seconds). | Defaults to 30s. For more information, go to duration of the chaos. |
CHAOS_INTERVAL | Time interval between two successive instance terminations (in seconds). | Defaults to 30s. For more information, go to chaos interval. |
NETWORK_BANDWIDTH | Specify the network bandwidth rate limit. | Defaults to 1mbit . For more information, go to network bandwidth. |
BURST | Burst for the size of bucket, that is, the maximum amount of bytes that tokens can be available for instantaneously. | Defaults to 2kb . For more information, go to burst. |
LIMIT | Limit on the number of bytes that can be queued while waiting for tokens to become available. | Defaults to 2kb . For more information, go to limit. |
MIN_BURST | Size of the peakrate bucket. | For example, 1kb . |
PEAK_RATE | Maximum depletion rate of the bucket. | For example, 1mbit . |
DESTINATION_IPS | IP addresses of the services or pods whose accessibility should be affected. You can also specify a CIDR block. | Comma-separated IPs (or CIDRs) can be provided. If it has not been provided, network chaos is induced on all IPs (or destinations). For more information, go to run with destination IPs. |
DESTINATION_HOSTS | DNS names (or FQDN names) of the services whose accessibility is affected. | If it has not been provided, network chaos is induced on all IPs (or destinations). For more information, go to run with destination hosts. |
SEQUENCE | Sequence of chaos execution for multiple instances. | Defaults to parallel. Supports serial sequence as well. For more information, go to sequence of chaos execution. |
RAMP_TIME | Period to wait before and after injecting chaos (in seconds). | For example, 30s. For more information, go to ramp time. |
DEFAULT_HEALTH_CHECK | Determines if you wish to run the default health check which is present inside the fault. | Default: 'true'. For more information, go to default health check. |
If the environment variables DESTINATION_HOSTS
or DESTINATION_IPS
are left empty, the default behaviour is to target all hosts. To limit the impact on all the hosts, you can specify the IP addresses of the service (use commas to separate multiple values) or the DNS or the FQDN names of the services in DESTINATION_HOSTS
.
Secret tunables
Tunable | Description | Notes |
---|---|---|
GOVC_URL | vCenter server URL used to perform API calls using the govc command. | It is derived from a secret. |
GOVC_USERNAME | Username of the vCenter server used for authentication purposes. | It can be set up using a secret. |
GOVC_PASSWORD | Password of the vCenter server used for authentication purposes. | It can be set up using a secret. |
GOVC_INSECURE | Runs the govc command in insecure mode. It is set to true . | It can be set up using a secret. |
Network bandwidth
Network bandwidth injected to the VM. Tune it by using the NETWORK_BANDWIDTH
environment variable.
Following YAML snippet illustrates the use of this input variable.
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
name: VMware-engine
spec:
engineState: "active"
chaosServiceAccount: litmus-admin
experiments:
- name: VMware-network-rate-limit
spec:
components:
env:
- name: NETWORK_BANDWIDTH
value: '2mbit'
- name: VM_NAME
value: 'vm-1,vm-2'
- name: VM_USER_NAME
value: 'ubuntu,debian'
- name: VM_PASSWORD
value: '123,123'
Burst
Size of bucket, in bytes. It is the maximum number of bytes for which tokens can be instantaneously available. Tune it by using the BURST
environment variable.
Following YAML snippet illustrates the use of this input variable.
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
name: VMware-engine
spec:
engineState: "active"
chaosServiceAccount: litmus-admin
experiments:
- name: VMware-network-rate-limit
spec:
components:
env:
- name: NETWORK_BANDWIDTH
value: '2mbit'
- name: BURST
value: '2kb'
- name: VM_NAME
value: 'vm-1,vm-2'
- name: VM_USER_NAME
value: 'ubuntu,debian'
- name: VM_PASSWORD
value: '123,123'
Limit
Limit on the number of bytes that can be queued while waiting for tokens to become available. Tune it by using the LIMIT
environment variable.
Following YAML snippet illustrates the use of this input variable.
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
name: VMware-engine
spec:
engineState: "active"
chaosServiceAccount: litmus-admin
experiments:
- name: VMware-network-rate-limit
spec:
components:
env:
- name: NETWORK_BANDWIDTH
value: '2mbit'
- name: BURST
value: '2kb'
- name: LIMIT
value: '2kb'
- name: VM_NAME
value: 'vm-1,vm-2'
- name: VM_USER_NAME
value: 'ubuntu,debian'
- name: VM_PASSWORD
value: '123,123'
Run with destination IPs and destination hosts
The IPs/hosts that interrupt traffic by default. Tune it by using the DESTINATION_IPS
and DESTINATION_HOSTS
environment variables, respectively.
DESTINATION_IPS
: IP addresses of the services or the CIDR blocks (range of IPs) whose accessibility is impacted.
DESTINATION_HOSTS
: DNS names of the services whose accessibility is impacted.
Following YAML snippet illustrates the use of this input variable.
## it injects the chaos for the egress traffic for specific ips/hosts
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
name: VMware-engine
spec:
engineState: "active"
chaosServiceAccount: litmus-admin
experiments:
- name: VMware-network-rate-limit
spec:
components:
env:
# supports comma separated destination ips
- name: DESTINATION_IPS
value: '8.8.8.8,192.168.5.6'
# supports comma separated destination hosts
- name: DESTINATION_HOSTS
value: 'google.com'
- name: VM_NAME
value: 'vm-1,vm-2'
- name: VM_USER_NAME
value: 'ubuntu,debian'
- name: VM_PASSWORD
value: '123,123'
Network interface
Name of the ethernet interface that shapes the traffic. Tune it by using the NETWORK_INTERFACE
environment variable. Its default value is eth0
.
Following YAML snippet illustrates the use of this input variable.
## it injects the chaos for the egress traffic for specific ips/hosts
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
name: VMware-engine
spec:
engineState: "active"
chaosServiceAccount: litmus-admin
experiments:
- name: VMware-network-rate-limit
spec:
components:
env:
# name of the network interface
- name: NETWORK_INTERFACE
value: 'eth0'
- name: VM_NAME
value: 'vm-1,vm-2'
- name: VM_USER_NAME
value: 'ubuntu,debian'
- name: VM_PASSWORD
value: '123,123'