Convergence
Three operators converge on a measurement reset: stop optimising stage conversion rates, stop pretending you have long feedback loops, and stop running experiments that won't reach sample size. Optimise absolute counts at each stage. Find correlated short signals for the long outcome you care about. Use pre/post when sample size won't cooperate.
Operators
- Archie Abrams (Shopify) — Optimise for absolute count of users reaching each stage, not stage conversion rates and 30–40% of growth experiments with short-term lift show no incremental value at one year. Optimise absolute counts; 30-40% of short-term-lift wins evaporate at 1 year.
- Annie Duke — There is no such thing as a long feedback loop — find a correlated short signal. There is no such thing as a long feedback loop — find a correlated short signal.
- Elena Verna — Don't test what won't reach sample size in a month — pre/post is fine. Don't test what won't reach sample size in a month — pre/post is fine.
- Chris Orlob — Coach reps on skill-friction, not call counts. Coach reps on skill-friction, not call counts (the same principle in sales-coaching shape).
Variation
- Abrams provides the metric reframe + the validation evidence (long-term holdouts).
- Duke provides the cognitive move (find the correlate).
- Verna provides the operational ruthlessness (pre/post is fine).
- Orlob applies the same logic to coaching analytics.
- Convergence: the loop is shorter than you think and the metric you're using is wrong.
Implication
For every experiment you run, ask: (1) will it reach sample size in 30 days? if not, run pre/post. (2) am I optimising rate or count? optimise count. (3) is the outcome I care about months out? find a correlated 7-day proxy and validate it. Audit one quarter of "wins" against a 1-year holdout and expect 30-40% evaporation.
Sources
- ins_absolute-counts-over-conversion-rates — Archie Abrams
- ins_long-term-holdouts-30-40-evaporate — Archie Abrams
- ins_no-such-thing-as-long-feedback-loop — Annie Duke
- ins_experimentation-paralysis — Elena Verna
- ins_competency-analytics-over-activity-dashboards — Chris Orlob