product-skill/ PRO/ Growth PRO depth

PRO: Growth · loop math & experiment rigor

Instrument the loop; report the funnel as its cross-section.

The worked depth behind §Growth and §Metrics: the gain math for viral, engagement and monetization loops, hybrid-PLG sequencing, and the experiment discipline — OEC, multiplicity correction, marketplace interference, novelty — with every number carried through. All figures illustrative / point-in-time; re-derive on your own data.

Funnel

One-shot, top-to-bottom. Asks "where do users drop?" — no engine; every cohort starts from paid input.

stalls the day you stop buying input

Loop

The output of one turn re-feeds the next. Asks "what does a converted user produce that pulls in the next?" — compounding, harder to copy.

gain K decides amplify vs leak

If you can't name what the output re-feeds, you have a funnel pretending to be a loop.

Above the fold

Three moves that carry the module.

01

Pick ONE loop per stage.

Acquisition early, engagement mid, monetization at scale — they trade off. Don't run three half-instrumented loops. The gain math finds the real lever; it's almost never the obvious knob.

02

Set the OEC before launch.

One short-term proxy chosen because it predicts the long-term goal, validated on history, with must-not-regress guardrails. Never pick metrics after peeking — that's how you ship a confusing UI that "raises sessions."

03

Name the validity threat in the doc.

Many comparisons, marketplace spillover, novelty — each silently manufactures a fake win. "We randomized by user" in a marketplace is a bug, not a detail. Correct it before you read the result.

§1–2 · model the right thing, then the re-feed math

The loop's gain decides amplify or leak.

A generic loop: new_input(t+1) = output(t) × conversion. The gain K tells you whether it compounds or just bleeds. Three loop types, opposite signs — keep them separate.

Reforge · acquisition / viral loop

Viral loop — the leverage is on conversion, not invite volume

K = invites_per_user × invite→signup conversion · steady-state amplification = 1 / (1 − K) for K<1 · K≥1 = runaway (rare — audit for fraud)

Worked (illustrative)

Base = 1,000 organic signups/wk; each new user sends 2 invites; 8% convert → K = 0.16. Amplifier = 1/(1−0.16) = 1.19 → effective ~1,190 signups/wk. Double invite-conversion to 16% → K = 0.32 → amplifier 1.47. Invites are cheap noise; the accept step is the constraint.

Reforge · engagement loop

Engagement — model a weekly return rate r, and keep the two regimes apart

Output = the content / data / habit a user creates that pulls them (or others) back. r = the share of active users who come back the next period. The two regimes have opposite signs:

With constant acquisition A: N → A / (1 − r) → A = 2k/wk, r = 0.3 → N ≈ 2,000 / 0.7 ≈ 2,857 sustained actives

Without new acquisition the base decays geometrically by r each week toward zero — 10k → ~3k → ~900 → … There is no positive floor; only acquisition holds the base up.

The lever is r itself — measure it by killing the loop output (e.g. notifications off) in a holdout and watching the return rate drop.

Reforge · monetization loop

Monetization — gated by payback period, not the LTV:CAC ratio

Revenue re-feeds paid acquisition. Self-funding iff LTV > CAC AND payback < cash runway. Reinvestable surplus per customer = LTV − CAC; recycle it into CAC at the same unit economics → compounding spend.

Worked

LTV $300, CAC $100, payback 4 mo → each customer funds ~2 more at the same CAC after payback. A 3× ratio with an 18-mo payback starves the loop — the ratio looks great while cash never returns in time to recycle.

Decision: pick the one loop that is your primary engine this stage — acquisition early, engagement mid, monetization at scale. They trade off; don't run three half-instrumented loops.

§3–4 · motion sequencing & the pre-build signal

Layer the motions; score accounts, not users.

Elena Verna · hybrid-PLG

Sequence the motions — pure PLG is rare at scale

1

PLG self-serve

Low TTV, freemium/trial, in-product upgrade triggers. Owns acquisition + activation + low-end expansion.

2

Product-led sales

Reps engage only accounts past a usage threshold — layered on top, not instead.

3

Marketing + sales-led

Demand-gen + outbound for segments PLG can't reach — high-ACV, compliance-gated, low-frequency.

PQA > PQL. B2B buys at the account level — score Product-Qualified Accounts (usage aggregated per logo), not lone-user PQLs, or sales chases noise. PQA = fit (ICP, employee count, industry) × engagement (breadth: active seats · depth: core-action frequency · velocity: WoW growth).

Worked routing table (illustrative)

Account	Fit (0–5)	Active seats	Core-actions/wk	WoW velocity	PQA	Route
Acme	5	14	90	+22%	high	PLS rep now
Bolt	2	30	200	+5%	mid (low fit)	nurture, don't staff
Cog	5	3	12	+40%	rising	watch; trigger at seat-threshold

Ring-fence PLS · the metrics · when PLG is only a feeder

Ring-fence PLS (critical). Reps touch an account only past the usage threshold AND where self-conversion is unlikely (multi-team, security review pending). Do not let reps work accounts that would self-convert — that cannibalizes margin and inflates "sales-sourced" revenue. Measure PLS by incremental conversion over a self-serve holdout, not gross.

Metrics: PQL/PQA, TTV, activation rate, free→paid (~2–5% directional), NRR — the PLG North Star (>100% healthy, ~120%+ best-in-class; directional / point-in-time).

Not for high-touch, low-frequency, or compliance-gated buying — there, lead sales-led; PLG becomes a lead-gen feeder, not the engine.

PQA routing + NRR North Star ↔ §Growth · pro §metrics §5

Painted-door / fake-door test

Ship the entry point, count intent — then kill it fast

1

Ship the entry point

Button, menu item, pricing tier — the door, not the room behind it.

2

Count intent

Intent rate = clicks ÷ exposed users, against a pre-set bar (e.g. build only if >5% click — illustrative). Segment the clickers: are they your ICP?

3

Show "coming soon"

Then kill or build. One exposure window only.

Never leave it live — repeated dead doors cost trust. Pair with a follow-up survey / concierge to confirm the click meant the job, not curiosity (novelty inflates first-touch).

Cheapest test that can produce a NO for a Desirability bet ↔ pro §discovery §4

§5–6 · the metric you optimize, and the doc you pre-commit

A short-term proxy that predicts the thing you can't measure yet.

Kohavi / Tang / Xu · OEC

OEC design — worked (food-delivery)

The Overall Evaluation Criterion is the one metric an experiment optimizes — a short-term, in-window proxy chosen because it predicts the long-term goal you can't measure in two weeks. Set it and its guardrails before launch; never pick metrics after peeking.

1

Name the goal

The long-term value you actually care about (6-mo retained revenue, LTV).

2

List proxies

Short-term metrics measurable inside the window.

3

Validate

On history: does moving the proxy retain/monetize better? Check sensitivity + directional alignment (no proxy that rises while value falls — Goodhart).

4

Compose

Into a single OEC — one primary metric or a normalized weighted combo.

5

Attach guardrails

Must-not-regress metrics that catch the proxy being gamed.

Worked — food-delivery app (illustrative)

Goal: 26-week retained gross profit per signup. Window: 14 days. Candidates: sessions/user, items browsed, completed orders/user (14d), push opt-in.

Validation: users with ≥3 completed orders in 14d retained at 58% to week 26 vs 19% for <3 → strong, sensitive, aligned. "Sessions" rose with a confusing UI (users lost, clicking around) → rejected (rises while value falls).

OEC = completed orders per user in first 14 days.

Guardrails (ship-blocking if they regress): median delivery-time SLA, refund/cancellation rate, support-contacts per order, app-crash rate, unsubscribe rate, and (regulated) any consent/disclosure metric. A treatment that lifts the OEC but blows a guardrail does not ship.

Effect-size rule: a stat-sig tiny OEC lift may still not be worth shipping once you price the opportunity cost (eng, complexity, surface risk). Judge effect size vs the next-best use of the slot, not just p<0.05.

Raviv · state the causal why

Experiment-design template — pre-commit before launch

Fill this in and freeze it before unblinding. The hypothesis states the mechanism (why), not just the prediction (what) — the why is the learning.

Hypothesis (causal WHY): We believe [change] will [mechanism] → moving [OEC] because [reason]. OEC: ____ Guardrails (must-not-regress): ____, ____, ____ Unit of randomization: user / session / account / cluster / time-slice MDE: __% Power: 80% α: 0.05 Required N / duration: ____ (don't peek; fix the horizon) Validity gates: SRM check (split = design?) · pre-period / A-A balance · no overlapping confound (outage/holiday/price/PR/campaign) · invariant-metric check (a metric the change should NOT move) Pre-committed decision rule: win→__ / flat→__ / loss→__ (write before unblinding) Segments to read: STATIC pre-treatment only (country, device, signup cohort) Multiplicity plan: # comparisons = __ ; correction = Bonferroni / BH Interference risk: none / networked / marketplace → design = __ Analysis: novelty / primacy date-segmentation plan

Model-Capability assumption in an AI experiment ↔ pro §discovery §4 · pro §ai-evals

§7–9 · the three validity threats that manufacture fake wins

Correct multiplicity, respect interference, wait out novelty.

At α=0.05, 20 independent comparisons → ~64% chance of ≥1 false "win" (1−0.95²⁰). Testing many metrics, variants, or segments inflates false positives — correct it.

Bonferroni vs Benjamini-Hochberg

The multiplicity ladder — worked BH, m = 5, α = 0.05

Bonferroni controls the family-wise error: threshold α/m (m=10, α=0.05 → require p<0.005 each). Simple, conservative — right when any single false positive is costly. Benjamini-Hochberg controls the false-discovery rate: sort p ascending, find the largest i with p₍ᵢ₎ ≤ (i/m)·α, declare 1…i significant. More powerful for screening.

1

p = 0.001 ≤ 0.010 pass

2

p = 0.008 ≤ 0.020 pass

3

p = 0.02 ≤ 0.030 pass

4

p = 0.04 ≤ 0.040 cutoff ✓

5

p = 0.21 > 0.050 reject

p-value (bar) threshold (i/m)·α Largest i with p₍ᵢ₎ ≤ threshold: i = 4 → declare first 4 significant

Bonferroni at 0.01 would pass only the first 2. Choose BH for exploration, Bonferroni for a go/no-go safety call.

Interference / SUTVA

When treatment spills between units, stop randomizing by user

Standard A/B assumes SUTVA: one unit's treatment doesn't affect another's outcome. False in two-sided marketplaces, social graphs, shared-supply, shared-budget systems — a treated rider depletes the same driver pool a control rider draws from, biasing the measured effect. Two fixes:

Cluster / geo randomization

spatial / organizational interference

Randomize whole self-contained units — city, market, region, company/team — so spillover stays within an arm. Cost: far fewer effective units → less power; needs many balanced clusters and often CUPED/variance reduction.

Use for: delivery markets, ride-hail cities, B2B per-account features.

Switchback randomization

market too coupled to split by geography

Randomize time windows (e.g. 30-min slices) ON/OFF across the entire market — each market is its own control over time. Watch carry-over between slices (add washout buffers) and time-of-day confounds (balance the schedule).

Use for: pricing, matching, surge.

Always state the interference risk in the design doc — "we randomized by user" in a marketplace is a silent validity bug, not a detail.

Novelty / primacy

Week-1 lift is contaminated — trust the steady state

Don't read the launch spike as the durable effect. Plot the daily/weekly treatment effect over the window and read the shape:

Novelty — fades to baseline (don't ship the spike) Primacy — rises then plateaus (the steady state is the real, larger effect)

Segment new vs existing users: novelty hits existing users hardest, so a new-user-only readout often shows the true long-run effect sooner. Extend the window past the decay/learning curve; if you must decide early, decide on the trend's asymptote, not the integral.

Pitfalls · fast scan

The validity traps to clear first.

Keep reading