Key concepts

This document introduces key concepts to help you manage your team's experiments efficiently and effectively.

Key Experimentation concepts

Experimentation is a process of testing software variants among randomized user groups to measure outcomes with statistical rigor. The following terms are foundational to understanding and running effective experiments.

Term	Definition
Statistical significance	A measure of whether observed differences in outcomes are likely due to chance or reflect a real effect.
P-value	The probability of observing results as extreme as the current data, assuming the null hypothesis is true.
Frequentist vs Bayesian	Two statistical approaches: Frequentist relies on long-run frequencies, while Bayesian incorporates prior beliefs.
Fixed horizon	A testing approach where results are only analyzed after collecting a pre-specified amount of data.
Sequential testing	A method allowing results to be checked continuously without inflating false-positive rates.
Multiple comparison correction (MCC)	Techniques used to reduce error rates when multiple hypotheses are tested simultaneously.
Sample size calculator	A tool for estimating how many users are needed in a test to detect a meaningful effect with desired confidence.