> For the complete documentation index, see [llms.txt](https://developer.harness.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://developer.harness.io/feature-management-experimentation/use-fme/experimentation/best-practices.md). # Experimentation Best Practices Experimentation is a powerful way to drive growth and insight, but only when done with intention. This guide covers the principles that separate high-quality experiments from ones that produce misleading or meaningless results. ### Planning and purpose Before running any test, define [what you are trying to learn, what success looks like, and how the experiment fits your broader business goals](/feature-management-experimentation/use-fme/experimentation/foundations.md). Without this, even well-executed tests produce noise. Anchor every experiment in a clear hypothesis, a reasoned prediction based on observed behavior. For example: *"Reducing the number of form fields from five to three will increase conversion rates due to reduced user friction."* A concrete hypothesis gives the experiment focus and a measurable outcome. To prioritize which experiments to run, use the **PIE framework**: Potential, Importance, Ease. Focus first on areas with high business impact that are technically straightforward to implement. ### Design and setup
Start with the problem, not the solution Resist the urge to jump straight to a solution. Start with a user problem (what is causing friction or drop-off?), then design variations to address it. Solutionizing leads to narrow thinking and superficial wins.
One change at a time Design experiments around a single change. If you adjust multiple variables simultaneously, you will not be able to attribute the result to any one of them. You can test a single change across multiple variants (e.g., A/B/C/D), but keep the variable itself isolated.
Choose one primary metric Pick a single metric that determines success, tightly linked to your hypothesis (e.g., conversion rate for a sign-up experiment). Secondary metrics can provide context but shouldn't drive the decision. Go to [Create a metric](/feature-management-experimentation/new-to-fme/get-started/create-a-metric.md) for more details.
Avoid overlapping KPIs across experiments If multiple tests influence the same metric (like activation rate), their effects become impossible to disentangle. Coordinate across teams to prevent this. You can leverage dependant flags to [exclude experiments from one another](/feature-management-experimentation/use-fme/experimentation/setup/mutually-exclusive-experiments.md).
Bucket users at the right point Assign users to variants on the page where the change occurs. If your experiment is on the pricing page, only bucket users when they reach that page. Bucketing earlier dilutes and skews your audience. You can set an [entry level exposure filter](/feature-management-experimentation/use-fme/experimentation/metrics/setup/filtering.md#applying-a-filter) for experiments.
Limit the number of treatments Stick to **three treatments per test** (control + two variants) as a default. This balances learning with statistical power. You can use up to six treatments, but beyond that, sample size requirements grow and complexity adds diminishing returns.
### Statistical rigor
Size your sample before launching Too few users makes results unreliable. Use a sample size calculator based on your baseline performance, expected lift, and chosen testing method. Go to [Monitor and experiment settings](/feature-management-experimentation/use-fme/experimentation/setup/experiment-settings.md#testing-methods) to define the required sample size for the Fixed Horizon method.
Set a fixed run time Decide how long the experiment will run before you launch it, typically one to two weeks, depending on traffic and time-to-action. Do not stop early because the numbers look good or bad. See [testing methods](/feature-management-experimentation/use-fme/experimentation/setup/experiment-settings.md#testing-methods) to review Fixed Horizon guidance.
Do not make midway changes Once a test is live, do not adjust copy, design, or measurement. Changes invalidate results and introduce bias. If you have made a change or a fix, you can [re-allocate flag traffic](/feature-management-experimentation/use-fme/feature-management/manage-feature-flags/reallocate-traffic.md) and reset the experiment dates.
Use equal traffic splits Unless you are using a formal ramp strategy, split traffic equally across variants. Equal exposure ensures fair comparisons and maximizes statistical power. You can use [limit exposure](https://help.split.io/hc/en-us/articles/360020525572-Limiting-exposure) to manage ramp strategy.
Enable measurement in the platform Before running an experiment, make sure events have been sent and measurement is enabled by toggling to percentage-based targeting in the feature definition. See [Define feature flag treatments and targeting](https://help.split.io/hc/en-us/articles/360020791591-Define-feature-flag-treatments-and-targeting#targeting-rules).
Validate with an A/A test first Run an A/A test before experimenting to confirm your metrics are behaving as expected. Go to [A/A testing](https://help.split.io/hc/en-us/articles/360031279312-Running-an-A-A-test) to validate your setup.
### Interpret results
Be open to being wrong The best learnings often come from unexpected results. Treat experiments as learning tools, not validation tools. If a test disproves an assumption, that's a successful experiment.
Inconclusive results are still results An inconclusive outcome means your change had no measurable impact on users for the metrics in question. That's useful signal, not a failure.
Add metrics retroactively Once an experiment is running, you may realize you've forgotten to track a metric or discover an interesting behavior you'd like to measure. In these cases, you can create new metrics and recalculate metric data for your experiment at any time. The key requirement is that the relevant event data must already have been sent to Harness before you add the metric. Because of this, it's a good practice to send any events that might be relevant to the user journey from the outset, giving you the flexibility to define and analyze additional metrics later without needing to rerun the experiment.
Document everything Record every test: win, loss, or inconclusive. Keep summaries short, jargon-free, and paired with visual aids where possible. Centralize this knowledge so the rest of the organization can benefit.
Do not overgeneralize A result that holds in one region, channel, or customer segment may not transfer elsewhere. Revalidate before scaling. You can set up [Dimensional Analysis](/feature-management-experimentation/use-fme/experimentation/experiment-results/analyzing-experiment-results/dimensional-analysis.md), which allows you to interpret your experiment data using pre-configured segments.
Organize experiments into themes Group experiments into categories, such as onboarding, pricing, retention, UX, conversion, and so on. Over time, this surfaces patterns and builds thematic insight across teams and customer verticals. Categories like "do no harm" or "UX polish" also help set stakeholder expectations: a small UX change is unlikely to move revenue metrics directly.
### Strategic mindset
Balance exploration and exploitation **Exploration** means testing bold, risky ideas to find breakthroughs. **Exploitation** means fine-tuning known performers. Both have a place. A healthy experimentation roadmap blends the two.
Iterate with purpose Iteration lets you build on your hypothesis and converge on the best solution. That said, know when to move on; if there's nothing further to learn from a line of experiments, redirect effort toward higher-priority areas.
By following these principles, you can run experiments that generate trust, insight, and meaningful growth.