Alerts in Harness FME
Harness FME provides a flexible alerting system to help teams monitor the impact of feature flags and experiments in real time. By configuring automated alerts on keyA key metric is a specific metric that is critical for evaluating the success of a feature flag or experiment. It is used to determine whether a feature rollout or experiment is achieving its intended goals and can influence decisions about further rollouts or adjustments. or guardrail metricsA guardrail metric is a specific type of metric used to monitor for negative impacts during a feature rollout or experiment. It serves as a safety check to ensure that the new feature does not cause significant issues or degrade user experience. If a guardrail metric indicates a problem, it can trigger alerts or rollbacks to mitigate potential harm., teams can quickly detect regressions, respond to issues, and protect customer experience during rollouts.
With alerts, you can:
- Detect when a metricA metric measures events that are sent to Harness FME and can count the occurrence of events, measure event values, or measure event properties. Metrics are used to evaluate the impact of feature flags and experiments on user behavior and system performance. crosses a critical threshold
- Automatically trigger notifications based on experiment results
- Respond to issues before they escalate
Harness FME alerts are designed to work seamlessly with your existing workflows, ensuring you stay informed and in control during every stage of a release or experiment.
Determine an alert mechanism
Choose the alert type that matches how you want to detect and respond to metric changes:
| Alert type | How it's triggered | Fires on | Configured for | Environment support | How to enable |
|---|---|---|---|---|---|
| Significance alert (automatic) – Key metric | Statistically significant impact detected (threshold = 0). | Good or bad impact | Key metric linked to a flag | Production only | Mark a metric as a key metric for a feature flag on the Metrics impact tab. |
| Significance alert (automatic) – Guardrail metric | Statistically significant impact detected (threshold = 0). | Good or bad impact | Guardrail metric | Production only | Set a metric category to Guardrail in the metric definition. |
| Manual metric alert policy (any metric) | Manually configured threshold is crossed. | Degradations only | Any metric with a policy | Any environment | Create a metric alert policy from the metric definition on the Alert Policy tab. |
For key metrics, alerts are triggered only when the When a key metric reaches significance option is enabled in a feature flag's alert settings. Guardrail metric alerts are automatically evaluated.
Significance alertsA significance alert is a notification triggered when the results of an experiment reach a predefined level of statistical significance. This indicates that the observed effects are unlikely to be due to chance, helping teams make informed decisions about feature rollouts or adjustments based on the experiment's outcomes. are automatic and require no threshold configuration. Use them to detect statistically meaningful changes during feature flag rollouts. These alerts are only evaluated in production environments.
Metric alert policies are manually configured and allow you to define thresholds and recipients. Use them when you:
- Need alerts in non-production environments (for example, staging or preview)
- Want explicit control over thresholds
- Need to customize notification recipients
Manual metric alertsA metric alert is a notification triggered when a specific metric reaches a predefined threshold. This can indicate potential issues or significant changes in user behavior, allowing teams to respond quickly to mitigate negative impacts or capitalize on positive trends. fire when any metric crosses a defined threshold, regardless of feature flag or experiment.
Configure alerts
Choose the type of alert you want to configure based on your use case:
- To set up automatic alerts based on statistical significance, see Automated alerts and notifications.
- To configure threshold-based alerts with custom conditions or non-production environments, see Alert policies.
Alert policies
Control how and when alerts trigger by creating an alert policy. Define thresholds, notification rules, and alert behaviors that match your team's processes.
Monitoring window
Set the time window over which metric performance is evaluated for alert policies. Monitoring windows help you tune sensitivity and reduce alert noise by limiting the period during which alerts are automatically triggered based on observed metric degradations.
Harness FME continues to monitor and alert your team of a metric degradation for up to 28 days after a version change. The default monitor window is 24 hours. Administrators can change this in the Monitor window and statistics settings.
The monitoring window only applies to alert policies. If the monitoring window is set to 24 hours, Harness FME stops evaluating metrics for alerts triggering after 24 hours from the version change. However, significance-based alerts can still trigger later if metrics are recalculated (for example, during deeper analysis or manual recalculation).
Alert baseline treatment
Compare metrics against a baseline treatment to improve alert accuracy and minimize false positives. This treatment serves as the control group in impact comparisons, allowing Harness FME to evaluate whether changes in a metric are statistically significant when users receive a different treatment.
Manage alerts
When an alert fires, you can access the alert details and take action from the Feature flags page.
| Action | Description |
|---|---|
| Kill feature flag | If you decide to kill a feature flag due to an alert, the default treatment overrides the existing targeting rules and is returned for all users. |
Troubleshooting alerts
Fix common configuration or delivery issues, verify metric inputs, and fine tune thresholds for better alert performance. For more information, see Troubleshooting alerts.