Ctrlk
Sign UpGet a DemoContact Us
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Istio

Istio is an open-source service mesh that integrates with existing distributed applications. It also serves as a platform with APIs that enable integration with logging and telemetry systems. Traceable provides an agent that integrates with Istio to monitor your system and detect anomalies and attacks. Traceable can also block attacks on your system based on the configurations that you make on the platform. For more information on blocking, see Policies. The following is a high-level deployment diagram for Traceable's Istio solution. Note that Traceable attaches to only the Istio ingress gateway.

The deployment steps assume that a standard Istio deployment is already available.

Traceable module extension (TME)

A Traceable Module Extension (TME) is a type of sidecar container that a Traceable Platform agent can inject into an application pod. It can be considered a miniature Traceable Platform agent.

TME has ext_cap as a component. This synchronously receives requests from applications, such as a gateway. The traceable module extension also houses libtraceable, which has the request-blocking logic. TME is used when native instrumentation or a native plugin is unavailable. In such cases, TME captures data inline and sends it to the Traceable Platform agent asynchronously. TME also blocks the requests inline.

Before you begin

Before you begin deploying Traceable's tracing agent, note that the deployment assumes a stock deployment of Istio is already available.

Support matrix

Component
Version

Istio

  • 1.13.0 and above for Traceable Istio agent 2.x

  • 1.10.0 and above for Traceable Istio agent 1.x

Traceable Platform agent

  • 1.52.0 and above for Traceable Istio agent 2.x

  • 1.36.2 and above for Traceable Istio agent 1.x

Verify your Istio deployment. It is a good practice to have your Istio's deployment diagram with you for reference. If you are installing for a demo environment, the control and ingress are in the Istio-system namespace; however, the namespace may vary. Verify where the control plane and ingress are installed by entering the following command:

Following is a sample output of the above command:

Explanation:

  • istio-ingressgateway: This is the external gateway for traffic entering the cluster. It is usually of LoadBalancer type in cloud environments.

  • istiod: This is the Istio control plane, typically exposed internally via ClusterIP.

  • The NAMESPACE is istio-system, which is the default in many setups, but could vary in customized deployments.

Other notable points

  • Modify the Istio networking configuration to use Envoy Filter for authorization

  • Make sure that Traceable's platform agent is already installed. For more information on installing, see Platform agent.

  • Use the sidecar injector to deploy the Traceable module extension (tme). For more information on the Traceable module extension, see Annotations and Labels. The tme receives data from the Istio module and reports it to traceable-agent.

  • Point the Envoy filter to the tme.

  • Ensure that port 5442 is available for Traceable module extension (tme).

Tracing agent at Istio ingress gateway

Configuring the Traceable agent for the Istio ingress gateway consists of the following four steps:

  1. Adding a label to the namespace

  2. Adding an annotation to the spec in your deployment

  3. Enabling Envoy filter

  4. Restart Ingress controller

Step 1 - Add Label

Add traceableai-inject-tme=enabled label to the istio-system namespace.

Step 2 - Add Annotation

Add "tme.traceable.ai/inject:" "true" annotation to the Istio ingress gateway deployment. Enter the following command:

Verification of TME Injection:

  • After injecting the TME, check the istio-ingressgateway pod status. If the TME has been successfully injected, the status should change from 1/1 to 2/2.

  • You can further confirm this by running kubectl describe pod <istio-ingressgateway-POD_ID> and verifying that the second container listed is indeed the TME.

To get the POD_ID for a specific pod like istio-ingressgateway, you can use the following kubectl command:

For example,

Look for a pod with a name similar to istio-ingressgateway-<unique_ID>. The <unique_ID> at the end is the POD_ID you are looking for.

Example output:

In this case, the POD_ID is istio-ingressgateway-7f6bb877-6qhzz.

You can now use this POD_ID to run commands like kubectl describe pod istio-ingressgateway-7f6bb877-6qhzz -n istio-system.

Step 3 - Enable Envoy Filter

Add traceableai-istio=enabled label to the Istio ingress gateway deployment.

If everything has default names, the following command will add the above label to the Istio ingress in istio-system namespace:

Step 4 - Restart the Ingress controller

Restart the Ingress controller for the tme to attach. Enter the following command:

Verify the deployment

Enter the following command to verify the deployment.

Example output

Install Traceable Istio helm chart

Enter the following command to install the Traceable Istio helm chart:

The above command creates an Envoy Filter object in the Ingress controller.

Helm values

The following tab lists the Helm values for Traceable’s Istio agents 1.x and 2.x. Use the values as per your deployment.

ExtProc filter configurations

Helm value
Default
Description

extProc.allowFailureMode

true

Allows Istio to accept client requests even if communication with TME fails or ext_cap returns HTTP 5xx.

extProc.allowModeOverride

false

Enables override of the filter processing mode via TME response.

extProc.messageTimeout

0.2s

Timeout for individual messages on the stream in synchronous mode.

extProc.observabilityMode

false

Enables "Send and Go" mode, sending each HTTP request/response part without pausing.

extProc.requestBodyProcessingMode

BUFFERED

Controls request body processing mode (Allowed values: STREAMED, BUFFERED, BUFFERED_PARTIAL, NONE).

extProc.requestHeaderProcessingMode

SEND

Controls request header handling (Allowed values: SEND, SKIP).

extProc.requestTrailerProcessingMode

SKIP

Controls request trailer handling (Allowed values: SEND, SKIP).

extProc.responseBodyProcessingMode

BUFFERED

Controls response body processing mode (Allowed values: STREAMED, BUFFERED, BUFFERED_PARTIAL, NONE).

extProc.responseHeaderProcessingMode

SEND

Controls response header handling (Allowed values: SEND, SKIP).

extProc.responseTrailerProcessingMode

SKIP

Controls response trailer handling (Allowed values: SEND, SKIP).

ext.Proc.grpcType

envoy_grpc

Specifies the gRPC service type for ext_proc filter. Must be one of: envoy_grpc (uses dynamic cluster_name) or google_grpc (uses target_uri).

Module extension configurations

Helm value
Default
Description

moduleExtension.connectTimeout

50ms

Timeout for connecting to the TME.

moduleExtension.grpcPort

5441

gRPC port for the TME sidecar.

moduleExtension.ipAddress

127.0.0.1

IP address of the TME sidecar (defaults to the same pod as the Istio gateway).

Webhook configurations

Helm value
Default
Description

webhook.caBundle

(None)

Base64 PEM encoded CA bundle for validating webhook server certificates.

webhook.enable

false

Enables webhook if the Traceable agent runs outside the Istio cluster.

webhook.failurePolicy

Ignore

Defines the webhook failure handling policy.

webhook.url

(None)

External Traceable agent URL (must be HTTPS).

webhook.useNamespaceSelectors.tme

true

Uses namespace selector match labels to inject the TME sidecar.

webhook.namespaceSelectors

traceableai-inject-tme: enabled

Namespace selector labels for TME injection. Only namespaces with these labels will have the webhook applied.

General configurations

Helm value
Default
Description

requestTimeout

0.2s

Timeout for the ext_proc filter.

workloadSelectors

“traceableai-istio:enabled“

Labels to add to pods in the mesh to enable the Traceable Istio module.

serviceName

istiov2

The service name for the Traceable agent.

environment

The default configured TPA environment

The name of the environment in which the Traceable agent is running.

agentToken

-

Agent token for authentication

filterName

traceableai-istio

Name of the EnvoyFilter resource that will be created.

filterNamespace

-

Namespace where the EnvoyFilter will be created. If not specified, defaults to the release namespace.

The following table details the various Helm values.

Value
Default value
Description

bodyCaptureSize

131072

Defines the maximum payload size in bytes to capture.

allowFailureMode

true

When set to true, Istio accepts client requests even if the communication with the Traceable Module Extension (tme) fails or the ext_cap service has returned an HTTP 5xx error.

moduleExtensionGRPCPort

5441

Traceable Module Extension (tme) sidecar gRPC port.

moduleExtensionHTTPPort

5442

Traceable Module Extension (tme) sidecar HTTP port.

moduleExtensionIPAddress

127.0.0.1

Traceable Module Extension (tme) sidecar IP address. By default, it is 127.0.0.1 since it is in the same pod as the Istio gateway.

workloadSelectors

- "traceableai-istio: enabled"

Labels to add to pods in the mesh to enable Traceable Istio module.

allowedContentTypes

{'json', 'x-www-form-urlencoded', 'grpc'}

Capture response bodies whose content-type header contains one of the configured substrings.

serviceName

istio

The service name for spans that are captured by the Traceable Istio module.

connectTimeout

50ms

Timeout to connect to Traceable Module Extension (tme).

requestHandleTimeout

0.2s

Timeout to the ext.authz request filter.

responseHandleTimeout

50

Timeout to the envoy response handler in milliseconds.

webhook.enable

false

Webhook must only be enabled if traceable-agent is running outside the Istio cluster. This Webhook will be used to call into the external traceable-agent to inject the tme sidecar.

webhook.url

-

External traceable-agent URL. It must be HTTPS.

webhook.caBundle

-

A base64 PEM encoded CA bundle that will be used to validate the external traceable-agent’s (which is Webhook's server) certificate.

webhook.failurePolicy

ignore

Defines how to handle webhook failures.

webhook.useNamespaceSelectors.tme

true

Use namespace selector match labels to inject the tme sidecar.

webhook.useNamespaceSelectors.java

true

Use namespace selector match labels to inject the Java sidecar.

The maximum body size that will be captured in spans is controlled by the maxBodySize variable in the Traceable Platform Agent (TPA) Helm chart or the max_body_size variable in the TPA Terraform configuration. This setting is defined in the TPA configuration, not in the Istio agent configuration.

If you need to capture larger payloads, you must adjust this setting in your TPA configuration. The default value is typically 131072 bytes (128KB).

Restart Ingress gateway

Restart the ingress gateway so the changes take effect. Enter the following command:

Verify the deployment

Enter the following command to verify the deployment:

Example output:

Upgrade

Follow these steps to upgrade the Traceable components within your Istio deployment. This process includes updating the Traceable Platform Agent (TPA), upgrading the Istio filter, and verifying the deployment to ensure a successful upgrade.

Complete the following steps:

  1. Update the Helm Repository.

  2. Upgrade the Traceable Platform Agent (TPA) to the latest version to maintain compatibility with the latest Istio filter and features.

  3. Upgrade the Traceable Istio Filter

    1. Search for the Latest Filter Version

    2. Upgrade the Istio Filter. This command upgrades the Traceable Istio filter in the istio-system namespace to the latest version.

    3. Restart the Istio Ingress Gateway. Restarting the ingress gateway ensures that the updated filter is applied to incoming traffic.

  4. Verify the deployed chart versions.

    1. Check the Traceable Chart Version in the Traceable Namespace.

    2. Check the Traceable Istio Chart Version in the Istio System Namespace.

  5. Verify the TME Image Version in the Istio Ingress Gateway. Describe any istio-ingressgateway pod to confirm the TME (Traceable Module Extension) image version:

  6. Verify Versions in the Traceable UI

    • Check Platform Agent Version: Confirm that the Platform Agent version reflects the latest release.

    • Check Tracing Agent Version: Ensure the Tracing Agent version matches the latest deployment.

    • Check TME Version in Span Attributes: Inspect the traceableai.module.version attribute in incoming spans to verify that they reflect the latest module version post-upgrade.

(Optional) Custom error message

You can optionally configure a custom error code and message for the blocking function by editing the Traceable Platform agent configuration, as shown below. Configure this inside the injector section.

If you are using Helm in values.yaml, then you would need to set:

Make sure that the status code is set between 400 and 499. If you add values outside the 4xx range, then 403 (the default status code) is displayed in the Traceable Platform.

PreviousIstio
NextTraceable MCP Server

Last updated

Was this helpful?