Skip to main content

Datadog Memory Check

Last updated on

Datadog memory check validates the memory utilisation of a service using Datadog container metrics. The target service name and Kubernetes namespace are supplied at runtime via probe variables, allowing this probe to be reused across services.

Infrastructure type

  • Kubernetes

Use cases

Datadog Memory Check probe helps you:

  • Verify memory stays within limits during memory hog faults
  • Monitor container memory after OOM-related chaos
  • Validate memory limits during stress experiments
  • Detect memory leaks under sustained load

Overview

This probe queries Datadog for sum:container.memory.usage filtered by kube_deployment and kube_namespace, converts bytes to MiB using the formula memory/1048576, and compares the mean aggregated value against your threshold.

Probe type

APM Probe (Datadog)

Prerequisites

  • Kubernetes chaos infrastructure with Harness Delegate
  • A configured Datadog connector
  • Datadog agent reporting container.memory.usage for the target deployment
  • Probe variables SERVICE_NAME and NAMESPACE set at experiment runtime

Probe properties

Comparator

TypeCriteriaDefault value
float<=1024

The probe passes when the mean memory usage (in MiB) is less than or equal to the comparator value.

Datadog APM probe inputs

FieldDescriptionDefault
connectorIDIdentifier of the Datadog connectorRequired at runtime
durationInMinLook-back window in minutes5
queryTypeDatadog query API versionv2
queries[0].nameQuery alias used in the formulamemory
queries[0].dataSourceMetric source typemetrics
queries[0].params.queryDatadog metric querysum:container.memory.usage{kube_deployment:<+probe.variables.SERVICE_NAME>,kube_namespace:<+probe.variables.NAMESPACE>}
formulaConverts query result to MiBmemory/1048576
aggregationAggregation over the time windowmean

Probe variables

VariableDescriptionRequired
SERVICE_NAMEKubernetes deployment name of the target serviceYes
NAMESPACEKubernetes namespace of the target serviceYes

Configurable inputs

InputDescriptionRequiredDefault
CONNECTOR_IDDatadog connector identifierYes-
DURATION_IN_MINLook-back window in minutesNo5
COMPARATOR_VALUEMemory threshold in MiBYes1024

Run properties

PropertyDescriptionTypeDefault
timeoutMaximum time to wait for the probe to completeString30s
intervalTime between probe executionsString5s
attemptNumber of retry attempts before marking as failedInteger1
pollingIntervalTime between retry attemptsString30s
initialDelayInitial delay before starting the probeString1s
stopOnFailureStop the experiment if the probe failsBooleanfalse
verbosityLog verbosity levelStringdebug

Probe definition

You can define this probe in your chaos experiment as follows:

probe:
- name: datadog-memory-check
type: apmProbe
mode: Continuous
apmProbe/inputs:
type: Datadog
comparator:
type: float
value: "1024"
criteria: <=
datadogApmProbeInputs:
connectorID: <+connector.identifier>
durationInMin: 5
queryType: v2
queries:
- name: memory
dataSource: metrics
params:
query: sum:container.memory.usage{kube_deployment:<+probe.variables.SERVICE_NAME>,kube_namespace:<+probe.variables.NAMESPACE>}
formula: memory/1048576
aggregation: mean
runProperties:
timeout: 30s
interval: 5s
attempt: 1
pollingInterval: 30s
initialDelay: 1s
stopOnFailure: false
verbosity: debug

Next steps