Create an experiment
Describe an A/B test in plain language, review the plan, choose good metrics, then put the variant in your code.
Want an agent to run the whole test? This prompt walks it through schema, cohorts, experiment, implementation, launch, results and decision, and it checks with you before anything goes live.
Describe it
Experiments → New experiment asks one question: What do you want to test? Write it the way you'd say it to a colleague:
Does an annual-plan discount banner on the pricing page get more trials to upgrade?
The chips under the box are your most frequent events. Click one to start a sentence with it as the goal. Then click Draft experiment (or ⌘ ↵).
Datahop reads your project's events, cohorts and existing flags and drafts a complete plan:
- a name, a hypothesis and a flag key;
- variants:
controlandtest, 50/50, unless you ask for something else; - a primary metric on one of your real events, plus up to three secondary metrics, including a guardrail where it can;
- cohorts to compare, and the smallest lift worth detecting;
- a short rationale, and warnings, for example when an event you'd need isn't tracked yet.
It never invents an event you don't track. Nothing is created yet.
Prefer to skip the description? Click Set it up yourself: name the test, click a goal event and click Create draft. You get the same plan with sensible defaults. If drafting isn't available, New experiment opens this short form.
Review the plan
The draft appears as a sentence you can edit:
Split everyone 50/50 between control and test on flag annual-discount-banner. Success = % of people who do plan_upgraded within 14 days. Compare New users vs Power users. Needs ~5,157 people per variant to detect a 10% lift — about 33 days at today's traffic.
Click any highlighted part to change it: who's included, the variants and split, the flag key, the metric, the secondary metrics, the cohorts and the minimum lift. The sample size is a rough estimate based on your recent traffic.
Advanced opens every setting: notes, variants, metric filters, targeting and the code snippet. Then choose:
- Create draft: creates the experiment and its flag (off). Launch it once your code reads the flag.
- Create & launch: creates it and turns the flag on right away.
- Hand this to your agent: copies the plan as instructions for Claude, ChatGPT or a coding agent.
Choose good metrics
The primary metric decides the verdict. There are two kinds:
- Conversion: the share of exposed people who did an event at least once. Use it for yes/no outcomes: upgraded, ordered, invited a teammate.
- Mean: the average per exposed person of a number (such as revenue: the sum of
amount) or of an event count (such as projects created). People who never did it count as 0, so "revenue per visitor" includes visitors who bought nothing.
Each metric has a window: how many days after their first exposure a person's actions count. Everyone gets the same window, however late they joined. Match it to the behaviour: hours to days for checkout, about 14 days for a trial upgrade.
Secondary metrics get full results but don't decide the verdict. Use them as:
- guardrails: things that mustn't get worse, like refunds, cancellations or support tickets;
- value: revenue per user next to a conversion primary.
Money metrics are noisy
A few very large orders can swing a mean metric. Make revenue a secondary metric, and use a conversion like "ordered at all" as the primary.
Minimum lift (minimum detectable effect) is the smallest relative improvement worth acting on. It only affects the sample size estimate. Smaller lifts need far more people: halving it needs about four times the sample. Rare goals, like a 2% checkout rate, need many more people than common ones.
Decide on the primary metric before launch. You can add a guardrail later, but swapping the primary halfway through is a way to cherry-pick a winner.
Put the variant in your code
The experiment only gets data when your app reads its flag:
if (datahop.getFeatureFlag('annual-discount-banner') === 'test') {
showDiscountBanner()
}- Read the flag where the experience starts, such as on the pricing page and not on app load. Everyone who reads the flag counts as exposed.
- Show control for anything else, including
undefined. - Send the goal event, such as
plan_upgraded. Send it from your server for anything involving money.
See Flags in your code for details.
Launch, stop, delete
- Launch turns the flag on and starts counting.
- Stop lets no new people in. The flag keeps serving variants until you conclude or switch it off.
- Conclude records the decision. See Read results.
- Delete removes the experiment, and its flag too if the flag is off.