An A/B test needs a decision rule before it needs a winner. If the team changes the stopping date after seeing early results, the test may reward noise or a calendar effect.
Write the protocol before launch
Document the hypothesis, one primary outcome, guardrails, audience, variants, allocation method, minimum run conditions, and stop rule. Choose an outcome close to the business decision—qualified inquiries or completed bookings, for example—when volume and measurement allow. If the event is too rare, select a valid leading indicator and state that the business outcome remains unproven.
Check the evidence in order
- Is assignment stable and are users seeing the intended variant?
- Is tracking firing once and using the same definition for both groups?
- Did stock, price, campaign mix, device mix, or sales follow-up change during the test?
- Did the observation window include the full weekly pattern and relevant local dates?
- Is there enough data for the planned comparison, or should the result be labeled inconclusive?
In MENA, do not assume every audience shares one weekend, payday, or seasonal pattern. Note the country and date range; avoid comparing a Ramadan promotion directly with an ordinary month without a reasoned design.
A small difference after three days is not a universal winner rule. Use a suitable statistical method or a transparent practical threshold agreed before the test; if uncertain, get a measurement specialist to review it. Stop for safety, broken tracking, or business constraints—not because one chart briefly looks favorable.
Optional AI-assisted work brief — Evidence sufficiency review
Assess whether this experiment has enough valid exposure, independent users, stable implementation, and a predeclared stopping rule. Explain the difference between a directional lift and reliable evidence. Check repeated peeking, multiple metrics, segment selection, and calendar effects. If evidence is insufficient, recommend what can be learned safely and what data is still needed; do not force a winner.
Worked scenario — When a test cannot declare a winner
If only a few dozen people saw each variant, a 3% difference may be noise. Keep the result “inconclusive,” inspect session recordings or run five moderated task walkthroughs, and fix obvious usability issues before collecting more traffic. If the change is low risk, ship it as a usability improvement only when the qualitative evidence supports it—not as a proven conversion lift.
AI disclosure: AI tools assisted with drafting. No sample size or significance result is claimed here.