Read results
What the verdict, lift, probabilities, split check and run-time estimate mean, how segments work, and how to conclude.
Open an experiment to see its results. The top shows the verdict, the recommendation and the exposures. Below that are the primary metric per variant, exposures per day, segments, secondary metrics and Summary · what agents see. Results always cover every exposure since launch.
The examples below come from a test of an annual-plan discount banner: control vs test, 50/50, with "upgraded within 14 days" as the primary metric.
The numbers
| Number | Meaning | Example |
|---|---|---|
| Exposures | People counted in each variant | 1,475 vs 1,525 |
| Rate | Conversion rate, or average per person for mean metrics | 30.8% vs 39.1% |
| Lift | How much higher than control, relatively | +26.9% |
| Interval | The range the true lift falls in with 95% probability | +15.0% to +39.8% |
| P(beats control) | Chance this variant truly beats control | 100% |
| P(best) | Chance this variant is the best of all variants | 100% |
Read it as: test converts 39.1% of exposed trials vs 30.8%. The true lift is very likely between +15% and +40%, and there's essentially no chance control wins.
A wide interval means you know the direction but not yet the size. With three or more variants, a variant can beat control and still not be the best, so the verdict uses P(best).
Verdicts
| Verdict | When |
|---|---|
| Needs more data | A variant has fewer than 100 exposures. |
| Winner | A variant has at least a 95% chance of being best. |
| Keep control | Control is best with 95% probability, or every variant has at most a 5% chance of beating it. The change doesn't help. |
| Inconclusive | None of the above, yet. |
"Keep control" is a clear result, not a failure. Variants with fewer than 100 exposures are marked small sample.
The verdict and recommendation use only the primary metric. Secondary metrics get their own verdicts. With several secondary metrics, one may look like a winner by chance, so treat a surprise there as an idea for the next test.
Split check
If you set 50/50 and got 70/30, something is broken: maybe the flag is read on only one code path, a redirect loses events in one variant, or bots are getting in. Datahop checks the split. When it's off, you see a warning banner and the recommendation tells you to check your implementation before trusting the results. Below 50 exposures the check is skipped.
How long to run
For conversion metrics, the recommendation estimates how many more days you need to detect your minimum lift at today's traffic:
Keep running — roughly 6 more day(s) to reach 3,609 exposures per variant.
Treat it as a guide, not a stopping rule. Before you decide, give every exposed person their full metric window, and run at least a full week if behaviour differs by weekday.
Segments
An average can hide very different stories. Segments show the primary metric again for each cohort you pick, with its own lift and verdict. Add up to 10 with Add cohort segments when you create the experiment, or Edit segments at any time, even after it ended.
| Segment | Control | Test | Lift | Verdict |
|---|---|---|---|---|
| Everyone | 30.8% | 39.1% | +26.9% [+15.0%, +39.8%] | Winner |
| New users | 17.9% | 28.9% | +61.2% [+27.5%, +102.8%] | Winner |
| Power users | 58.7% | 62.2% | +5.9% [−6.7%, +20.6%] | Inconclusive |
The banner works, and almost all of the effect comes from new users. Power users already upgrade at 59%, so a discount for them may cost revenue without changing much. A sensible decision: ship it, then target the banner flag at people not in Power users.
The overall verdict tells you whether it works. Segments tell you for whom.
Pick segments the test can't change
Segments use cohort membership as it is when you look, not when people entered.
- Time windows move. "Signed up in the last 7 days" means as of today, so after two weeks the people who were new at launch have left the segment.
- Outcomes can leak in. If a cohort is defined by the thing you're measuring (say, upgraded users), the test itself moves people into it.
The cleanest segments are fixed before the test starts: plan, country, platform, signup month.
Conclude
When you've decided, click Conclude. Pick the winner (the dialog suggests the one the results point to), write the reason, and leave Roll out winner to everyone on or turn it off. Then click Ship.
With roll-out on, everyone gets the winning variant right away through the same flag, so your getFeatureFlag(…) === 'test' code keeps working until you clean it up. The decision, the reason and who made it are saved on the experiment.
A good reason mentions the segments, for example: "+26.9% upgrades, driven by new users (+61%). Power users flat. Follow-up: exclude Power users from the banner."