Experiments
A/B tests you describe in a sentence, with a clear verdict, a recommendation and results per cohort.
An experiment tests whether a change moves a metric. Describe what you want to test, review the plan Datahop drafts, and put the variant in your code. Once it's live, results update continuously and tell you:
- the chance each variant is best;
- the lift over control, with a range it very likely falls in;
- a verdict: winner, keep control, inconclusive or needs more data;
- whether the split looks healthy, how much longer to run, and a one-line recommendation;
- the same numbers per cohort, such as new users vs power users.
You can check results as often as you like. Looking early doesn't invalidate them.
Experiment "Annual plan discount banner" (running). Exposures: control 1475, test 1525.
Primary: Upgraded — control: 30.8% (n=1475); test: 39.1% (n=1525) lift +26.9% [15.0%, 39.8%], P(beats control) 100.0%. Verdict: winner (test).
Segment "New users": Upgraded — … lift +61.2% [27.5%, 102.8%], P(beats control) 100.0%. Verdict: winner (test).
Recommendation: Ship "test": it is the best variant with 100.0% probability.Lifecycle
| Status | What happens |
|---|---|
| Draft | The experiment and its flag exist, but the flag is off, so nothing changes for users. |
| Running | Launch turns the flag on. Exposures and metrics count from here. |
| Stopped | No new people enter. People already in keep counting until their metric window closes. |
| Concluded | The decision is recorded: the winner, the reason, and who decided. Optionally the winner is rolled out to everyone. |
Every step shows up in the activity log with the person or agent who took it. Experiments need the Feature flags product.