1. Turn an idea into a decision
“Try a shorter onboarding” is an idea. A testable hypothesis explains which audience will see which change and why that should improve a specific outcome. For example: moving an optional preference question after first use may help new installations reach a first useful action within 24 hours.
Record a primary metric, baseline estimate, minimum effect worth acting on, allocation, eligibility, observation window and stopping rule before launching. Estimate sample requirements using the chosen statistical method and realistic traffic. There is no universal number of users or days that makes every test reliable.
A completion metric may be useful, but a downstream activation event can better represent the intended value. Include guardrails such as crashes, request failures and unintended purchase behavior. Do not change the definition after seeing which metric looks best.
2. Keep assignment stable and exposure meaningful
Assign an eligible installation to Control A or Variant B consistently. If a restart moves an installation between variants, the outcomes no longer correspond cleanly to the assigned experience. Keep the experiment version and assignment with relevant events.
Assignment, configuration delivery and exposure are separate events. Record exposure when the assigned experience is actually shown. Define whether the primary analysis includes all eligible randomized installations or only an exposure population. Filtering after assignment can introduce bias if the variant changes the likelihood of being exposed.
Before collecting a decision-making sample, test both branches, persistence across restarts, duplicate event handling and scope boundaries. Inspect unexpected allocation imbalance; a 50/50 plan does not guarantee exactly equal counts, but a substantial discrepancy deserves investigation.
3. Report the effect in percentage points
Imagine 500 of 1,000 installations complete Control A and 540 of 1,000 complete Variant B. The observed rates are 50% and 54%. The absolute difference is +4 percentage points; the relative lift is +8%. Reporting “8% better” without the base rates is ambiguous.
The chart below shows these invented outcomes. Under a simple independent-binomial, fixed-horizon normal approximation, the approximate 95% interval for the difference is −0.4 to +8.4 percentage points. The observed improvement is compatible with a small negative effect as well as a useful positive one. It is not a demonstrated winner.
That illustrative interval uses 1.96 × the standard error of the difference. Normal approximations can perform poorly with small samples or rates near zero or one. Use an interval method appropriate to the experiment design; repeated observation, clustered identities and multiple comparisons require additional care.
An observed lift is not a certain lift
Illustrative completion outcomes with 1,000 installations in each variant.
4. Follow the stopping rule
Ordinary fixed-horizon confidence intervals do not support repeatedly checking the result and stopping the moment it looks favorable. Choose a fixed analysis horizon or a valid sequential method in advance, and apply that method consistently. Allow the outcome window to mature for the final entrants.
Keep a record of exclusions, outages and releases during the experiment. Separate planned segments from exploratory findings; trying many countries, devices or metrics increases the chance of a persuasive-looking result by accident. Treat an interesting unplanned segment as a hypothesis for another test.
If the test is inconclusive, say so. Low traffic can leave meaningful effects unresolved. Consider whether improving instrumentation or testing a larger, coherent change would be more useful than extending an unplanned analysis indefinitely.
5. Make a product decision, then monitor
A rollout decision combines evidence, practical effect size, guardrails and implementation cost. A positive completion estimate with worse activation or reliability may not justify shipping. Preserve the exact tested versions and record why the decision was made.
Aplyna supports documented experiment configuration and scoped event collection; it does not automatically declare or deploy a winning variant. Existing native branches must already be shipped in the app. Remote configuration cannot deliver arbitrary new native code. Read the current experiment guide and availability limits before planning a release.
Sources & further reading
Technical references may be in English. Examples are educational; product behavior follows the current source SDKs and availability reference. No customer performance claim is implied.
Published by Aplyna. Editorial illustrations were generated with AI; they are not product screenshots. Charts use labeled teaching examples or conceptual diagrams.