Ctrlk
Sign UpGet a DemoContact Us
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Pod JVM SQL exception

Cause JDBC calls from a JVM running in a target Kubernetes pod to throw a configurable exception on a chosen table and SQL operation so you can test caller error handling.

Pod JVM SQL exception is a Kubernetes pod-level chaos fault that causes JDBC calls from a JVM running in a target container to throw a configurable exception on a chosen table and SQL operation for a configurable duration. Only matched calls fail; unrelated queries and other code paths run normally. When the fault ends, JDBC calls behave normally again immediately.

Use this fault to test how a Java service handles database failures: a primary that goes read-only, a specific table that returns errors, or an UPDATE that rejects writes.

RUN YOUR FIRST EXPERIMENT

If you have not configured the chaos infrastructure yet, go to Quickstart to install the chaos infrastructure and run an experiment end to end.

Use cases

Run this fault when you want to answer concrete questions like:

  • Driver-level error handling: When a SELECT on a specific table throws, does the caller retry, fall back, or fail closed?

  • Write-path resilience: If INSERT or UPDATE raises an exception on a critical table, does the application surface a clear error or silently lose the write?

  • Transaction rollback: For multi-statement transactions, does an exception trigger a clean rollback?

  • Retry semantics: Does the application honor JDBC driver retry semantics, or do user-code retries amplify load?

  • Mixed-operation scope: Confirm that failing SELECT on one table does not unintentionally fail other tables or other operations on the same table.

Prerequisites

  • Kubernetes version: 1.21 or later. Go to What's supported to confirm distribution support.

  • Target pod is Running: The Java application pod is in the Running state.

  • Java agent attach available: The Java process allows agent attach. Utilities such as ps, pgrep, and bash are present in the container, and the JVM is not built with a restricted runtime that strips attach modules.

  • JDBC driver in classpath: The target JVM uses a supported JDBC driver matching SQL_DATA_ACCESS_FRAMEWORK.

  • Privileged pods allowed: The cluster lets you schedule privileged pods in the chaos namespace. GKE Autopilot supports this fault but requires the one-time setup in Chaos on GKE Autopilot; other locked-down distributions may need similar exemptions.

  • Container runtime access: The chaos pod can reach the container runtime socket on the target node (/run/containerd/containerd.sock, /var/run/docker.sock, or /var/run/crio/crio.sock).

  • Workload selector defined: The chaos experiment knows the target workload by kind, namespace, and either names or labels.

JVM CHAOS USES THE BYTEMAN AGENT

This fault attaches a Byteman agent to the target JVM over BYTEMAN_PORT. The port must be reachable from the chaos pod and must not be in use by the application.

Supported environments

Platform
Support status

Amazon EKS

Supported

Azure AKS

Supported

Google GKE

Supported

Red Hat OpenShift

Supported

Rancher

Supported

VMware Tanzu

Supported

Self-managed Kubernetes (CNCF-certified)

Supported

GKE Autopilot

Supported with Autopilot setup

EKS Fargate, ACI virtual nodes

Not supported (no access to container runtime sockets)

Permissions required

The fault runs under the chaos infrastructure's service account.

Resource (apiGroup)

Verbs

Why it is needed

pods ("")

get, list, create, delete, deletecollection, patch, update

Discover target pods and run the chaos pod on the same node

pods/log ("")

get, list, watch

Stream chaos pod logs for status and debugging

deployments, statefulsets, replicasets, daemonsets (apps)

get, list

Resolve the target workload to the pods it owns

events ("")

get, list, create, patch, update

Record fault progress as Kubernetes events

jobs (batch)

get, list, create, delete, deletecollection

Run the chaos job that drives the fault

The default Harness chaos infrastructure service account already includes these permissions.

Fault tunables

Configure the following fault parameters when you add Pod JVM SQL exception to an experiment in Chaos Studio. Defaults are shown for reference.

SQL filters

Tunable
Description
Default

TABLE

Target database table name. Empty matches all tables.

""

SQL_OPERATION

SQL operation to target. Common values: SELECT, INSERT, UPDATE, DELETE. Empty matches all operations.

""

SQL_DATA_ACCESS_FRAMEWORK

JDBC driver framework identifier. For example MYSQL8.

"MYSQL8"

TRANSACTION_PERCENTAGE

Percentage of matched SQL statements to fail, between 0 and 100. 0 fails none; 100 fails every match.

0

Exception

Tunable
Description
Default

EXCEPTION_CLASS

Exception class to throw. Defaults to a common runtime exception.

"IllegalArgumentException"

EXCEPTION_MESSAGE

Message attached to the thrown exception.

"CHAOS BOOM!"

Chaos parameters

Tunable
Description
Default

TOTAL_CHAOS_DURATION

Duration of the fault in seconds.

60

JVM

Tunable
Description
Default

BYTEMAN_PORT

Port on which the Byteman agent listens inside the container. Must not conflict with any port already in use.

9091

JAVA_HOME

Absolute path to the Java installation inside the container. Empty auto-detects from PATH.

""

Targeting

Tunable
Description
Default

TARGET_PODS

Comma-separated list of pod names to target. Empty selects from the workload's pods using POD_AFFECTED_PERCENTAGE.

""

TARGET_CONTAINER

Container in the pod running the JVM. Empty targets the first container in the pod spec.

""

NODE_LABEL

Label selector to filter target pods by the node they run on. Empty disables node-based filtering.

""

POD_AFFECTED_PERCENTAGE

Percentage of the workload's pods to target. 0 means one pod.

0

SEQUENCE

When multiple pods are targeted, inject parallel (all at once) or serial (one after another).

parallel

Runtime and helper

Tunable
Description
Default

CONTAINER_RUNTIME

Container runtime on the target nodes. One of containerd, docker, crio.

containerd

SOCKET_PATH

Path to the container runtime socket on the target node. Set to match CONTAINER_RUNTIME.

/run/containerd/containerd.sock

RAMP_TIME

Wait period in seconds before and after the fault. Go to ramp time to read how it is applied.

0

Common pod selection tunables (TARGET_WORKLOAD_KIND, TARGET_WORKLOAD_NAMESPACE, TARGET_WORKLOAD_NAMES, TARGET_WORKLOAD_LABELS) are documented in common pod fault tunables. Tunables that apply to every fault are documented in common tunables for all faults.

USE A REALISTIC EXCEPTION CLASS

Choose an exception the caller code can plausibly receive (for example java.sql.SQLException or a driver-specific subclass). Picking an unrelated exception type often surfaces uncaught-exception bugs that would not happen in real failures.

Configure for your container runtime

Set CONTAINER_RUNTIME and SOCKET_PATH to match the runtime on the target node:

CONTAINER_RUNTIME

SOCKET_PATH

containerd (default)

/run/containerd/containerd.sock

docker

/var/run/docker.sock

crio

/var/run/crio/crio.sock

Fault execution in brief

Attaches a Java agent to the target JVM and intercepts JDBC statements matching TABLE and SQL_OPERATION for the configured SQL_DATA_ACCESS_FRAMEWORK to throw an instance of EXCEPTION_CLASS with EXCEPTION_MESSAGE on the configured percentage of calls, for TOTAL_CHAOS_DURATION seconds.

Expected behavior during fault execution

  • Matched SQL statements throw the configured exception. Other statements and unrelated tables run normally.

  • Application logs show stack traces from the configured exception class. Caller code paths surface the failure as application-level errors, retries, or fallbacks.

  • The database itself is not stressed; from the database's perspective, no statement was executed for the failed calls.

  • Tracing systems show the JDBC span ending in error.

WHEN THE FAULT ENDS

JDBC calls behave normally again immediately. Open transactions that hit the exception remain in their post-exception state; the framework or the application's retry logic is responsible for cleanup.

Signals to watch

Attach resilience probes to assert each layer:

  • Application error rate: Use an HTTP probe against endpoints that read or write to the targeted table.

  • JDBC error metrics: Use a Prometheus probe on jdbc.connections.errors or your APM's database error counter.

  • Application logs: Use a command probe to grep container logs for the configured EXCEPTION_MESSAGE.

Verify the fault execution effect

While the experiment is running, confirm operations are failing:

  1. Exercise the matched operation from a client.

    The response should reflect the failure.

  2. Confirm the exception in logs.

Recovery and cleanup

  • End of duration: JDBC calls behave normally again automatically.

  • Abort the experiment: Stopping the experiment from Chaos Studio triggers the same cleanup path.

  • Stuck state: If a downstream circuit breaker stays open or a connection pool is exhausted, follow the application's recovery procedure or restart the pod.

Limitations

  • Serverless Kubernetes (EKS Fargate, ACI virtual nodes): These platforms do not expose container runtime sockets and reject the privileged access the fault needs. GKE Autopilot is supported once the one-time setup in Chaos on GKE Autopilot is in place.

  • Windows containers: This fault is supported on Linux pods only.

  • Non-JVM and non-JDBC workloads: This fault targets JDBC drivers inside a JVM.

  • ORM caching: ORMs that cache results may continue to serve queries from cache without going through the driver, bypassing the fault on those calls.

Troubleshooting

Pod JVM SQL exception experiment stays Pending or never starts in Harness Chaos Engineering

Inspect the chaos pods in the experiment namespace with kubectl describe pod -n . The most common causes are taints on the target node that the chaos pods do not tolerate, insufficient resources, or a PodSecurity admission policy blocking privileged pods. Add the required tolerations or run in a namespace with privileged Pod Security level.

No exception observed during pod-jvm-sql-exception

The most common causes are: TABLE does not match the table name used by the application; SQL_OPERATION does not match the executed verb; SQL_DATA_ACCESS_FRAMEWORK does not match the driver in classpath; or TRANSACTION_PERCENTAGE is 0 (default). Re-run with TRANSACTION_PERCENTAGE=100 and empty TABLE/SQL_OPERATION filters to confirm the path is working.

Connection to container runtime fails for pod-jvm-sql-exception in Harness Chaos Engineering

The default SOCKET_PATH is /run/containerd/containerd.sock. For Docker, set CONTAINER_RUNTIME=docker and SOCKET_PATH=/var/run/docker.sock. For CRI-O, set CONTAINER_RUNTIME=crio and SOCKET_PATH=/var/run/crio/crio.sock.

Related faults

Last updated

Was this helpful?