No liftis an answer

Exposures to a decision

Crossed-out dots are users dropped for mismatched exposure or duplication. The same clean data then goes through three independent tests. Click a stage to read what happens there.

Is the experiment even valid?

3,893 rows (1.3%) had users seeing the wrong page for their group. I dropped them rather than guess, then removed one duplicate user. A sample ratio mismatch test then asks: did randomisation split traffic the way it was meant to?

294,478 raw exposures → 290,585 clean → 290,584 unique users

The interval crosses zero

Control converted at 12.04%, the new page at 11.88%. The difference is −0.16 points with a 95% interval from −0.39 to +0.08. A z-test, a bootstrap and a Bayesian model all agree: the probability the new page is better is about 9.5%.

Decision: don't ship. The redesign costs effort and shows no gain. The country splits (US −0.22, UK +0.11, CA −0.69) are point estimates, and none survives a Holm correction.

How many users would you need?

Drag the lift you want to detect. At a 12% baseline, 5% significance and 80% power, the sample per group grows with the inverse square of the effect. This test had about 145,000 per group, enough to see a lift of roughly 0.34 points.

Moving the gate cost players

The game moved its first progress gate from level 30 to level 40. One day later, the difference is within noise. After seven days, players with the later gate were measurably less likely to still be playing: −0.82 points, a 4.3% relative drop, interval −1.33 to −0.31.

Decision: keep the gate at 30. The probability that gate 40 is better for 7-day retention is about 0.1%.

Same answer, three ways