Claim
A/B testing has a sample-size cost most teams ignore. If the change won't accrue enough sample in 30 days, don't test it — ship it and use a pre/post readout (24h, 7d, 28d, 1yr). Reserve scientific A/B for high-traffic real estate or strategic pivots. Treating every change as an experiment is a velocity-killer dressed as rigor.
Mechanism
Underpowered tests yield indeterminate results — "no significant difference" — which leadership reads as "the change didn't work." Teams then revert good changes because the test couldn't prove them, or freeze in indecision. Pre/post readouts give honest directional signal at low cost, fast enough to keep velocity. Statistical rigor matters when the stakes justify it; it actively harms velocity when it doesn't.
Conditions
Holds when:
- The team can read pre/post data honestly without confusing seasonal effects.
- Leadership accepts directional reads on lower-stakes work.
Fails when:
- The change has high stakes (pricing, core flow). Skipping rigor there compounds errors.
- Pre/post can't separate the change from other variables (multiple shipped at once, big external event).
Evidence
"If we cannot collect the sample size in a month, we shouldn't test it. Period."
— Elena Verna on Lenny's Podcast, 2026-04-28
The default is pre-vs-post readouts at 24h, 7d, 28d, 1yr. A/B testing is reserved for high-traffic surfaces and strategic decisions.
Signals
- The team's testing cadence accelerates while quality of decisions does not degrade.
- Indeterminate-test "we don't know if it worked" results disappear from review.
- Leadership reads pre/post outcomes as legitimate signal, not lazy work.
Counter-evidence
A/B test purists argue any pre/post comparison is confounded by external variables. They are technically right and operationally wrong: in low-traffic environments, pre/post is the only honest tool. Use the A/B test religion only where data volume genuinely supports it.
Cross-references
- Add a new growth model every 18 months and protect it from KPIs for 12 — the seeding period explicitly allows pre/post over A/B