Experimentation Best Practices
Experimentation is a powerful way to drive growth and insight, but only when done with intention. This guide covers the principles that separate high-quality experiments from ones that produce misleading or meaningless results.
Planning and purpose
Before running any test, define what you are trying to learn, what success looks like, and how the experiment fits your broader business goals. Without this, even well-executed tests produce noise.
Anchor every experiment in a clear hypothesis, a reasoned prediction based on observed behavior. For example: "Reducing the number of form fields from five to three will increase conversion rates due to reduced user friction." A concrete hypothesis gives the experiment focus and a measurable outcome.
To prioritize which experiments to run, use the PIE framework: Potential, Importance, Ease. Focus first on areas with high business impact that are technically straightforward to implement.
Design and setup
Start with the problem, not the solution
Resist the urge to jump straight to a solution. Start with a user problem (what is causing friction or drop-off?), then design variations to address it. Solutionizing leads to narrow thinking and superficial wins.
One change at a time
Design experiments around a single change. If you adjust multiple variables simultaneously, you will not be able to attribute the result to any one of them. You can test a single change across multiple variants (e.g., A/B/C/D), but keep the variable itself isolated.
Choose one primary metric
Pick a single metric that determines success, tightly linked to your hypothesis (e.g., conversion rate for a sign-up experiment). Secondary metrics can provide context but shouldn't drive the decision. Go to Create a metric for more details.
Avoid overlapping KPIs across experiments
If multiple tests influence the same metric (like activation rate), their effects become impossible to disentangle. Coordinate across teams to prevent this.
You can leverage dependant flags to exclude experiments from one another.
Bucket users at the right point
Assign users to variants on the page where the change occurs. If your experiment is on the pricing page, only bucket users when they reach that page. Bucketing earlier dilutes and skews your audience.
You can set an entry level exposure filter for experiments.
Limit the number of treatments
Stick to three treatments per test (control + two variants) as a default. This balances learning with statistical power. You can use up to six treatments, but beyond that, sample size requirements grow and complexity adds diminishing returns.
Statistical rigor
Size your sample before launching
Too few users makes results unreliable. Use a sample size calculator based on your baseline performance, expected lift, and chosen testing method. Go to Monitor and experiment settings to define the required sample size for the Fixed Horizon method.
Set a fixed run time
Decide how long the experiment will run before you launch it, typically one to two weeks, depending on traffic and time-to-action. Do not stop early because the numbers look good or bad. See testing methods to review Fixed Horizon guidance.
Do not make midway changes
Once a test is live, do not adjust copy, design, or measurement. Changes invalidate results and introduce bias.
If you have made a change or a fix, you can re-allocate flag traffic and reset the experiment dates.
Use equal traffic splits
Unless you are using a formal ramp strategy, split traffic equally across variants. Equal exposure ensures fair comparisons and maximizes statistical power. You can use limit exposure to manage ramp strategy.
Enable measurement in the platform
Before running an experiment, make sure events have been sent and measurement is enabled by toggling to percentage-based targeting in the feature definition. See Define feature flag treatments and targeting.
Validate with an A/A test first
Run an A/A test before experimenting to confirm your metrics are behaving as expected. Go to A/A testing to validate your setup.
Interpret results
Be open to being wrong
The best learnings often come from unexpected results. Treat experiments as learning tools, not validation tools. If a test disproves an assumption, that's a successful experiment.
Inconclusive results are still results
An inconclusive outcome means your change had no measurable impact on users for the metrics in question. That's useful signal, not a failure.
Add metrics retroactively
Once an experiment is running, you may realize you've forgotten to track a metric or discover an interesting behavior you'd like to measure.
In these cases, you can create new metrics and recalculate metric data for your experiment at any time. The key requirement is that the relevant event data must already have been sent to Harness before you add the metric.
Because of this, it's a good practice to send any events that might be relevant to the user journey from the outset, giving you the flexibility to define and analyze additional metrics later without needing to rerun the experiment.
Document everything
Record every test: win, loss, or inconclusive. Keep summaries short, jargon-free, and paired with visual aids where possible. Centralize this knowledge so the rest of the organization can benefit.
Do not overgeneralize
A result that holds in one region, channel, or customer segment may not transfer elsewhere. Revalidate before scaling.
You can set up Dimensional Analysis, which allows you to interpret your experiment data using pre-configured segments.
Organize experiments into themes
Group experiments into categories, such as onboarding, pricing, retention, UX, conversion, and so on. Over time, this surfaces patterns and builds thematic insight across teams and customer verticals. Categories like "do no harm" or "UX polish" also help set stakeholder expectations: a small UX change is unlikely to move revenue metrics directly.
Strategic mindset
Balance exploration and exploitation
Exploration means testing bold, risky ideas to find breakthroughs. Exploitation means fine-tuning known performers. Both have a place. A healthy experimentation roadmap blends the two.
Iterate with purpose
Iteration lets you build on your hypothesis and converge on the best solution. That said, know when to move on; if there's nothing further to learn from a line of experiments, redirect effort toward higher-priority areas.
By following these principles, you can run experiments that generate trust, insight, and meaningful growth.