Pod JVM Solace exception
Pod JVM Solace exception is a Kubernetes pod-level chaos fault that causes Solace messaging calls from a JVM running in a target container to throw a configurable exception on a chosen destination and client mode for a configurable duration. Only matched calls fail; other Solace traffic and other code paths run normally. When the fault ends, Solace calls behave normally again immediately.
Use this fault to test how a Java service handles Solace messaging failures: a broker rejecting publishes, a subscriber that fails deserialization, or a JMS session error.
If you have not configured the chaos infrastructure yet, go to Quickstart to install the chaos infrastructure and run an experiment end to end.
Use cases
Run this fault when you want to answer concrete questions like:
- Publisher error handling: When publish throws on a critical destination, does the application retry, buffer, or drop the message?
- Subscriber poison messages: Simulate processing errors on the subscriber side and verify dead-letter or skip semantics.
- JMS vs native API: Verify behavior of
jmsand other approaches on the same destination. - Destination scope: Confirm that failing one destination does not unintentionally fail other destinations on the same client.
- Observability coverage: Do publisher-error metrics and dashboards surface the failure clearly?
Prerequisites
- Kubernetes version: 1.21 or later. Go to What's supported to confirm distribution support.
- Target pod is Running: The Java application pod is in the
Runningstate. - Java agent attach available: The Java process allows agent attach. Utilities such as
ps,pgrep, andbashare present in the container, and the JVM is not built with a restricted runtime that strips attach modules. - Solace client in classpath: The target JVM uses a supported Solace Java messaging client (for example the Solace JMS client) and exercises the configured
STREAM,SOLACE_DESTINATION_TYPE,SOLACE_MODE, andSOLACE_APPROACH. - Privileged pods allowed: The cluster lets you schedule privileged pods in the chaos namespace. GKE Autopilot supports this fault but requires the one-time setup in Chaos on GKE Autopilot; other locked-down distributions may need similar exemptions.
- Container runtime access: The chaos pod can reach the container runtime socket on the target node (
/run/containerd/containerd.sock,/var/run/docker.sock, or/var/run/crio/crio.sock). - Workload selector defined: The chaos experiment knows the target workload by kind, namespace, and either names or labels.
This fault attaches a Byteman agent to the target JVM over BYTEMAN_PORT. The port must be reachable from the chaos pod and must not be in use by the application.
Supported environments
| Platform | Support status |
|---|---|
| Amazon EKS | Supported |
| Azure AKS | Supported |
| Google GKE | Supported |
| Red Hat OpenShift | Supported |
| Rancher | Supported |
| VMware Tanzu | Supported |
| Self-managed Kubernetes (CNCF-certified) | Supported |
| GKE Autopilot | Supported with Autopilot setup |
| EKS Fargate, ACI virtual nodes | Not supported (no access to container runtime sockets) |
Permissions required
The fault runs under the chaos infrastructure's service account.
Resource (apiGroup) | Verbs | Why it is needed |
|---|---|---|
pods ("") | get, list, create, delete, deletecollection, patch, update | Discover target pods and run the chaos pod on the same node |
pods/log ("") | get, list, watch | Stream chaos pod logs for status and debugging |
deployments, statefulsets, replicasets, daemonsets (apps) | get, list | Resolve the target workload to the pods it owns |
events ("") | get, list, create, patch, update | Record fault progress as Kubernetes events |
jobs (batch) | get, list, create, delete, deletecollection | Run the chaos job that drives the fault |
The default Harness chaos infrastructure service account already includes these permissions.
Fault tunables
Configure the following fault parameters when you add Pod JVM Solace exception to an experiment in Chaos Studio. Defaults are shown for reference.
Solace filters
| Tunable | Description | Default |
|---|---|---|
STREAM | Name of the Solace topic or queue to target. | "" |
SOLACE_MODE | Client mode to target. One of publisher (publish-side) or subscriber (subscribe-side). | "publisher" |
SOLACE_DESTINATION_TYPE | Destination type. One of topic or queue. | "topic" |
SOLACE_APPROACH | Client API approach. For example jms. | "jms" |
TRANSACTION_PERCENTAGE | Percentage of matched Solace operations to fail, between 0 and 100. 0 fails none; 100 fails every match. | 0 |
Exception
| Tunable | Description | Default |
|---|---|---|
EXCEPTION_CLASS | Exception class to throw. Defaults to a common runtime exception. | "IllegalArgumentException" |
EXCEPTION_MESSAGE | Message attached to the thrown exception. | "CHAOS BOOM!" |
Chaos parameters
| Tunable | Description | Default |
|---|---|---|
TOTAL_CHAOS_DURATION | Duration of the fault in seconds. | 60 |
JVM
| Tunable | Description | Default |
|---|---|---|
BYTEMAN_PORT | Port on which the Byteman agent listens inside the container. Must not conflict with any port already in use. | 9091 |
JAVA_HOME | Absolute path to the Java installation inside the container. Empty auto-detects from PATH. | "" |
Targeting
| Tunable | Description | Default |
|---|---|---|
TARGET_PODS | Comma-separated list of pod names to target. Empty selects from the workload's pods using POD_AFFECTED_PERCENTAGE. | "" |
TARGET_CONTAINER | Container in the pod running the JVM. Empty targets the first container in the pod spec. | "" |
NODE_LABEL | Label selector to filter target pods by the node they run on. Empty disables node-based filtering. | "" |
POD_AFFECTED_PERCENTAGE | Percentage of the workload's pods to target. 0 means one pod. | 0 |
SEQUENCE | When multiple pods are targeted, inject parallel (all at once) or serial (one after another). | parallel |
Runtime and helper
| Tunable | Description | Default |
|---|---|---|
CONTAINER_RUNTIME | Container runtime on the target nodes. One of containerd, docker, crio. | containerd |
SOCKET_PATH | Path to the container runtime socket on the target node. Set to match CONTAINER_RUNTIME. | /run/containerd/containerd.sock |
RAMP_TIME | Wait period in seconds before and after the fault. Go to ramp time to read how it is applied. | 0 |
Common pod selection tunables (TARGET_WORKLOAD_KIND, TARGET_WORKLOAD_NAMESPACE, TARGET_WORKLOAD_NAMES, TARGET_WORKLOAD_LABELS) are documented in common pod fault tunables. Tunables that apply to every fault are documented in common tunables for all faults.
Choose an exception the caller can plausibly receive (for example javax.jms.JMSException for JMS clients). Picking an unrelated exception type often surfaces uncaught-exception bugs that would not happen in real failures.
Configure for your container runtime
Set CONTAINER_RUNTIME and SOCKET_PATH to match the runtime on the target node:
CONTAINER_RUNTIME | SOCKET_PATH |
|---|---|
containerd (default) | /run/containerd/containerd.sock |
docker | /var/run/docker.sock |
crio | /var/run/crio/crio.sock |
Fault execution in brief
Attaches a Java agent to the target JVM and intercepts Solace client operations matching STREAM, SOLACE_MODE, SOLACE_DESTINATION_TYPE, and SOLACE_APPROACH to throw an instance of EXCEPTION_CLASS with EXCEPTION_MESSAGE on the configured percentage of calls, for TOTAL_CHAOS_DURATION seconds.
Expected behavior during fault execution
- Matched Solace operations throw the configured exception. Other destinations and the unmatched mode run normally.
- Application logs show stack traces from the configured exception class.
- The Solace broker is not stressed; from the broker's perspective, no real message was published or consumed for the failed calls.
- Tracing systems show the Solace span ending in error.
Solace calls behave normally again immediately. Cached state in callers (open circuits, exhausted retry budgets, dead-letter queues that filled during the fault) may take additional time to drain.
Signals to watch
Attach resilience probes to assert each layer:
- Publisher error rate: Use a Prometheus probe on your application's Solace error counter.
- Subscriber error rate and queue depth: Use a Prometheus probe on subscriber processing counters to detect failed handlers.
- Application logs: Use a command probe to grep for the configured
EXCEPTION_MESSAGE.
Verify the fault execution effect
While the experiment is running, confirm operations are failing:
-
Exercise the matched code path from a client.
kubectl run -n <namespace> tester --image=nicolaka/netshoot --rm -it -- \curl -s http://<service>:<port>/<endpoint-that-uses-the-stream> -
Confirm the exception in logs.
kubectl logs -n <namespace> <target-pod> --tail=200 | grep "<EXCEPTION_MESSAGE>"
Recovery and cleanup
- End of duration: Solace calls behave normally again automatically.
- Abort the experiment: Stopping the experiment from Chaos Studio triggers the same cleanup path.
- Backlog drainage: Allow time for normal processing to drain any backlog, or scale subscribers temporarily if needed.
Limitations
- Serverless Kubernetes (EKS Fargate, ACI virtual nodes): These platforms do not expose container runtime sockets and reject the privileged access the fault needs. GKE Autopilot is supported once the one-time setup in Chaos on GKE Autopilot is in place.
- Windows containers: This fault is supported on Linux pods only.
- Non-JVM and non-Solace workloads: This fault targets Solace Java messaging clients inside a JVM.
Troubleshooting
Pod JVM Solace exception experiment stays Pending or never starts in Harness Chaos Engineering
Inspect the chaos pods in the experiment namespace with kubectl describe pod -n <chaos-namespace>. The most common causes are taints on the target node that the chaos pods do not tolerate, insufficient resources, or a PodSecurity admission policy blocking privileged pods. Add the required tolerations or run in a namespace with privileged Pod Security level.
No exception observed during pod-jvm-solace-exception
The most common causes are: STREAM does not match the topic or queue name; SOLACE_MODE does not match what the application uses; SOLACE_DESTINATION_TYPE or SOLACE_APPROACH do not match the client library; or TRANSACTION_PERCENTAGE is 0 (default). Re-run with TRANSACTION_PERCENTAGE=100 to confirm the path is working.
Connection to container runtime fails for pod-jvm-solace-exception in Harness Chaos Engineering
The default SOCKET_PATH is /run/containerd/containerd.sock. For Docker, set CONTAINER_RUNTIME=docker and SOCKET_PATH=/var/run/docker.sock. For CRI-O, set CONTAINER_RUNTIME=crio and SOCKET_PATH=/var/run/crio/crio.sock.
Related faults
- Pod JVM Solace latency: Add latency to Solace calls instead of failing them.
- Pod JVM Kafka exception: Kafka equivalent for the Apache Kafka Java client.
- Pod JVM method exception: Generic Java method-level exception injection.
- Common pod fault tunables: Shared environment variables for selecting target pods and workloads.