Enable Redis Stream Monitors
Overview
To set up end-to-end monitoring for Redis, you need to gather metrics from two sources: the redis-exporter and redis-stream-monitor. The redis-exporter provides basic functional metrics, while the stream-monitor publishes more descriptive metrics as a separate service.
This guide focuses on deploying the redis-stream-monitor to capture detailed event framework metrics that are useful for monitoring entire events activity under Harness.
Part 1: Deploy Redis Stream Monitor Chart
The redis stream monitor chart provides descriptive and meaningful metrics about Redis streams used by the Harness event framework.
Pull the Chart
helm pull oci://us-west1-docker.pkg.dev/gcr-prod/harness-helm-artifacts/redis-stream-monitor --version 1.6.0 --untar
Configure Override Values
Create a file named override.yaml
with the following content:
global:
monitoring:
enabled: true
port: 8889
path: /metrics
managedPlatform: "oss"
resources:
limits:
memory: 4096Mi
requests:
cpu: 1
memory: 4096Mi
config:
EVENTS_FRAMEWORK_ENV_NAMESPACE: ""
STACK_DRIVER_METRICS_PUSH_ENABLED: "true"
ENV: "SMP"
Install the Chart
helm install redis-stream-monitor redis-stream-monitor -n <namespace> -f override.yaml
Available Metrics
The following metrics are captured and made available through this service:
redis_streams_length
- Number of messages in the streamredis_streams_memory_usage
- Memory used by the streamredis_streams_average_message_size
- Average size of messages in the streamredis_streams_events_framework_deadletter_queue_size
- Size of the dead letter queueredis_streams_consumer_group_pending_count
- Number of pending messages in consumer groupredis_streams_consumer_group_behind_by_count
- How far behind the consumer group is
PromQL Dashboard Queries
Single Stream Dashboards
Use these queries to monitor a specific use case by setting the $usecase
variable:
avg by(usecaseName) (redis_streams_memory_usage{usecaseName="$usecase"})
avg by(usecaseName) (redis_streams_length{usecaseName="$usecase"})
avg by(usecaseName) (redis_streams_average_message_size{usecaseName="$usecase"})
avg by(usecaseName, consumergroupName) (redis_streams_consumer_group_behind_by_count{usecaseName="$usecase"})
avg by(usecaseName, consumergroupName) (redis_streams_consumer_group_pending_count{usecaseName="$usecase"})
avg by(exported_namespace, usecaseName) (redis_streams_events_framework_deadletter_queue_size{usecaseName="$usecase"})
Multiple Stream Dashboards
Use these queries to monitor all streams across all use cases:
avg by(usecaseName) (redis_streams_memory_usage{})
avg by(usecaseName) (redis_streams_length{})
avg by(usecaseName) (redis_streams_average_message_size{})
avg by(usecaseName, consumergroupName) (redis_streams_consumer_group_behind_by_count{})
avg by(usecaseName, consumergroupName) (redis_streams_consumer_group_pending_count{})
avg by(exported_namespace, usecaseName) (redis_streams_events_framework_deadletter_queue_size{})
Available Use Cases
The usecaseName
label can be one of the following values:
Expand to view all use cases
DEBEZIUM_gitOpsMongo.harness-gitops.applications
DEBEZIUM_gitOpsMongo.harness-gitops.utilization_snapshot
DEBEZIUM_harnessMongo.harness.applications
DEBEZIUM_ng-harness.ng-harness.moduleLicenses
DEBEZIUM_ngMongo.ng-harness.moduleLicenses
DEBEZIUM_pms-harness.pms-harness.planExecutionsSummary
DEBEZIUM_pmsMongo.pms-harness.planExecutionsSummary
DEBEZIUM_sscaMongo.ng-harness.instanceNG
LICENSES_USAGE_REDIS_EVENT_CONSUMER
async_filter_creation
cache_refresh
cd_deployment_event
cf_archive_ff_activation_audit
cf_archive_ff_audit
cf_create_env
cf_create_ff_activation_audit
cf_create_ff_audit
cf_create_seg_audit
cf_delete_env
cf_delete_ff_activation_audit
cf_delete_ff_audit
cf_delete_seg_audit
cf_dismiss_anomaly
cf_feature_metrics_data
cf_git_sync
cf_git_sync_now_events
cf_patch_ff_activation_audit
cf_patch_ff_audit
cf_patch_seg_audit
cf_proxy_key
cf_restore_ff_activation_audit
cf_restore_ff_audit
cf_svc_updates
cf_target_metrics
cg_general_event
cg_notify_event
chaos_change_events
ci_orchestration_notify_event
entity_activity
entity_crud
full_sync_stream
git_branch_hook_event_stream
git_config_stream
git_pr_event_stream
git_push_event_stream
iacm_orchestration_notify_event
instance_stats
ldap_group_sync
modulelicense
observer_event_channel
orchestration_log
pipeline_initiate_node
pipeline_interrupt
pipeline_interrupt_cd
pipeline_interrupt_ci
pipeline_interrupt_cv
pipeline_interrupt_iacm
pipeline_interrupt_pms
pipeline_interrupt_sto
pipeline_node_advise
pipeline_node_advise_cd
pipeline_node_advise_ci
pipeline_node_advise_cv
pipeline_node_advise_iacm
pipeline_node_advise_pms
pipeline_node_advise_sto
pipeline_node_facilitation
pipeline_node_facilitation_cd
pipeline_node_facilitation_ci
pipeline_node_facilitation_cv
pipeline_node_facilitation_iacm
pipeline_node_facilitation_pms
pipeline_node_facilitation_sto
pipeline_node_progress
pipeline_node_progress_cd
pipeline_node_progress_ci
pipeline_node_progress_cv
pipeline_node_progress_iacm
pipeline_node_progress_pms
pipeline_node_progress_sto
pipeline_node_resume
pipeline_node_resume_cd
pipeline_node_resume_ci
pipeline_node_resume_cv
pipeline_node_resume_iacm
pipeline_node_resume_pms
pipeline_node_resume_sto
pipeline_node_start
pipeline_node_start_cd
pipeline_node_start_ci
pipeline_node_start_cv
pipeline_node_start_iacm
pipeline_node_start_pms
pipeline_node_start_sto
pipeline_orchestration
pipeline_partial_plan_response
pipeline_sdk_response
pipeline_sdk_spawn
pipeline_sdk_step_response
pipeline_start_plan
plan_notify_event
pms_orchestration_notify_event
polling_events_stream
saml_authorization_assertion
setup_usage
srm_custom_change
srm_statemachine_event
sto_orchestration_notify_event
trigger_execution_events_stream
usermembership
webhook_events_stream
webhook_request_payload_data
Part 2: Configure Metric Scraping
Option 1: Using Prometheus Operator (monitoring.coreos.com/v1)
If you are using the Prometheus Operator with monitoring.coreos.com/v1
CRDs, create a PodMonitor
resource to scrape metrics from the redis-stream-monitor pods.
If you encounter an error that the CRD is not present, ensure that you install the necessary CRDs when deploying your Prometheus instance. This resource is critical for Prometheus to scrape the metrics.
Create a file named podmonitor.yaml
:
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
name: redis-stream-monitor
namespace: <namespace>
spec:
selector:
matchLabels:
app.kubernetes.io/name: redis-stream-monitor
podMetricsEndpoints:
- port: "8889"
interval: 120s
path: "/metrics"
Apply the PodMonitor:
helm install redis-stream-monitor redis-stream-monitor -n <namespace> -f override.yaml
The PodMonitor resource will be automatically created by the Helm chart when managedPlatform: "oss"
is configured.
Option 2: Using Other Observability Tools
For other observability platforms, configure your scraper to:
- Target port:
8889
- Metrics path:
/metrics
- Pod selector:
app.kubernetes.io/name: redis-stream-monitor
Summary
You have now deployed the redis-stream-monitor to capture detailed Redis stream metrics. These metrics provide valuable insights into the Harness event framework, including stream length, memory usage, consumer lag, and dead letter queue sizes, enabling proactive monitoring and troubleshooting of event processing in your Harness environment.