Read profit alongside conversion
Conversion tells you how often exposed shoppers order. Revenue tells you what they spend. Profit also accounts for the configured product margin and shipping cost. A higher conversion rate can still leave less profit when you subsidize more deliveries.
Compare the reported profit measure for each side, not raw revenue totals alone. Different exposure counts make raw totals especially misleading. Added Profit has a specific billing meaning: Added Profit is the eligible experiment's profit above its control baseline. A Success Fee applies once to an eligible completed experiment with positive Added Profit. There is no per-experiment cap.
A completed demonstration result
Synthetic data in the real app, not a merchant outcome. Profit per shopper improves by about 15% while conversion stays at 5%. The interface rounds the illustrative 99.9% confidence to 100%; that is not certainty.Check the strength of the evidence
At least seven full days, 1,000 exposures per side, and 200 confidently matched orders in total. Confidence and traffic-quality checks must also pass. A displayed leader before those checks pass is directional. It is not a confirmed winner, and meeting the minimum counts does not remove uncertainty.
Check confidence, minimum-data progress, and traffic quality together. A broken measurement path or uneven assignment needs investigation before you act on the numbers.
Choose a next step when there is no winner
An inconclusive result means the available evidence cannot reliably separate the offers. It does not prove that they perform equally.
If the experiment can continue and trading conditions remain comparable, give it more time within the app's allowed duration. Otherwise keep your baseline, record what you learned, and consider a more meaningful threshold difference next time.
Smaller stores can still check whether messages and checkout agree, spot operational problems, and learn about basket behavior. Treat early performance patterns as questions for the next test, not proof of a lift. Plan a test around your traffic.
Read the confirmation before applying
Review the outcome and the exact live change before confirming. Applying the alternative updates the offer to that threshold. Keeping the control preserves its threshold. Applying store rates pauses the tested free-shipping offer.
At the end of a 28-day test, the app can show a final performance comparison awaiting your decision. A performance leader is not necessarily a statistically confirmed winner. Follow the available selection or manual-close action, and keep your baseline when the evidence does not justify a change.
After applying, check the storefront and checkout again. A result does not promise future profit.
Your next decision
You can explain the outcome, its uncertainty, and the exact live change you are confirming. If evidence is inconclusive, keeping the baseline is a valid decision.