product-skill/ PRO/ Metrics PRO depth

PRO: Metrics · the worked decompositions

Factor the value moment into movable inputs — each with the guardrail that catches it being gamed.

The worked depth behind §Metrics: a North-Star input tree, growth accounting, HEART, NRR vs OMTM, agent outcome metrics, and the two "value trees" — L.E.K.'s driver tornado and the MIT R&D-portfolio ratios. All ~/band numbers are directional / point-in-time; verify against your own data.

North Star Metric

Time Spent Listening

= WAU × sessions/WAU/week × minutes/session

Breadth

220M

WAU

owner Growth · guard: new-user D7 retention not down

Frequency

11

sessions / user / wk

owner Core/Home · guard: skip-rate/session not up

Depth

24

minutes / session

owner Discovery/ML · guard: thumbs-down/session not up

NSM ≈ 220M × 11 × 24 = ~58.1B listener-minutes/wk (illustrative)

Amplitude North Star Playbook — depth needs the satisfaction guardrail in the same row (Goodhart).

Above the fold

Three moves that keep a metric honest.

01

Read the retention floor, not the slope.

The curve must flatten to a non-zero asymptote — that floor is your retained core. A rising floor over cohorts is the single best leading sign of product-market fit, better than any acquisition number.

02

Pair every NSM with a Goodhart sentence.

Write one sentence for how the metric could rise while the customer is worse off. If a paired guardrail on the dashboard can't refuse it, the metric isn't safe to optimize. Count value received, not value captured.

03

Rank drivers by leverage, not manageability.

A value-driver tree needs both tests: significant value impact AND controllable. The highest-impact driver can be uncontrollable — then it's a hedge, not a KPI. Most ops lists fail by treating every factor as equal.

§1–2 · the decomposition and its guardrails

One metric of delivered value, broken into 3–5 movable inputs.

Amplitude North Star Playbook

The method — factor value into breadth × frequency × depth

ONE metric that captures delivered customer value + revenue, broken into 3–5 movable input metrics that ladder to it. Inputs must be (a) leading, (b) ownable by a team, (c) collectively ~explain the NSM. If an input can't be moved by a squad next quarter, it's a context metric, not an input.

Do it on a whiteboard: state the value moment in one sentence ("a user listens to music they like") → write the NSM as a quantity of that value over time, not a count of events → algebraically factor it into breadth × frequency × depth (× quality) → assign each input an owner + a guardrail → sanity-check: if every input moves +10%, does the NSM move ~+33%? If not, the tree leaks.

Worked — Spotify (illustrative)

Time Spent Listening (NSM) = WAU × sessions/WAU/week × minutes/session · WAU 220M · sessions/user/wk 11 · minutes/session 24 → NSM ≈ 220M × 11 × 24 = ~58.1B listener-minutes/wk. A recommender that autoplays filler raises depth while quality (thumbs-down) degrades — Goodhart. That's why depth needs the satisfaction guardrail in the same row.

Templates & the re-validation rule

B2C habit: Value = Active users × core-action frequency × success-rate-per-action

Content/marketplace consumption: = consumers × sessions × items-consumed/session × completion-rate

Two-sided marketplace → TWO trees (supply liquidity + demand liquidity); the binding side (min(supply,demand)) is the real NSM.

Re-validation: every quarter, regress NSM against 90-day retention + revenue. If correlation has decayed, the NSM became a target and got gamed — re-derive. NSM is a proxy, never the goal.

§2 · guardrail taxonomy

Pair at least one guardrail per input — the classes that move against the user

Class	Example guardrail	Catches
Quality / health	error rate, p95 latency, crash-free %	"engagement" that's actually rage-clicking
Customer sentiment	CSAT, thumbs-down rate, support-ticket rate	depth bought by annoyance
Trust & safety	report rate, policy-violation exposure	growth via toxic/unsafe content
Unit economics	gross margin, cost per active user, COGS/session	usage that loses money
Retention / churn	logo + revenue churn, dormancy rate	a spike that doesn't stick

Counting-method drift applies here too: hold N-day vs rolling vs bracketed definitions fixed across the guardrail and the NSM.

§3 · AARRR growth accounting — the worked math

The point is the flows, not the level.

Jonathan Hsu / Social Capital

MAU decomposition + Quick Ratio

Every active user this month is one of five states vs last month: New (first-ever active) · Retained (active last month AND this month) · Resurrected (dormant last month, active now) · Churned (active last month, NOT this — a negative flow) · Dormant (churned and still gone — the reservoir resurrection draws from).

MAU(t) = Retained + New + Resurrected · ΔMAU = (New + Resurrected) − Churned · Quick Ratio = (New + Resurrected) ÷ Churned

Worked (illustrative)

Last month MAU 100k. This month: New +18k, Resurrected +6k, Churned −20k → MAU = 80k retained + 18k + 6k = 104k. Quick Ratio = (18+6)/20 = 1.2. Below 1 = leaking faster than filling; ~1.0 treading water; growth SaaS often targets ~1.5–2+ (directional). A QR of 1.2 off heavy acquisition is fragile — you're one acquisition dip from shrinking. Shrinking churn usually beats buying new users, because churn compounds against every future cohort.

L-day engagement histogram · reading the retention floor · vanity vs value

L-day engagement histogram (Ln, Amplitude): of the last 28 (or 30) days, on how many was each user active? A bimodal "smile" (mass at L1-3 AND L25-28) = a power-user core + a tourist tail → segment them; the average L-value hides both. Watch the L28/L7 split — daily-habit products want mass at the high end.

Retention curve — read the FLOOR, not the slope: plot % of a signup cohort still active by day (D1, D7, D14, D30, D60, D90). The curve must flatten to a non-zero asymptote — that floor is your retained core; if it decays toward 0, you have no PMF, only churn-and-replace. A rising floor over cohorts is the single best leading sign of PMF. Trace a low D1–D7 floor upstream: it's almost always an Activation / time-to-value miss, not Acquisition — fix the aha, not the ad spend.

Vanity vs value gate: Acquisition (signups, downloads, MAU level) is vanity; Activation/Retention/Revenue flows are value. Always pair a retention fix with a North-Star input + guardrail so you don't fix retention while degrading activation or monetization.

Loop gain & monetization payback → pro §growth §2 · retention floor pairs with the NSM tree above

§4–5 · the feature lens and the B2B North Star

HEART for one feature; NRR paired with an OMTM by stage.

Google · Rodden, Hutchinson, Fu

HEART — worked for "Saved Replies" (illustrative)

Feature/UX level, NOT company level. Pick the 1–2 rows the feature actually moves; don't fill all five for show. Each row: Goal → Signal (observable behavior) → Metric (the ratio you ship).

Dim	Goal	Signal	Metric (target, illustrative)
Happiness	agents trust the suggestion	post-use thumbs-up	≥70% positive
Engagement	becomes part of the flow	replies sent via feature	≥3 / agent / day
Adoption	new agents pick it up	% of active agents who used it in first week	≥40%
Retention	sticks past novelty	% of week-1 users still using it week-4	≥60%
Task success	speeds resolution w/o errors	median handle-time + edited-before-send rate	−20% time, edit-rate <30%

Task-success carries the guardrail: edit-rate up = the suggestions are wrong; speed bought by sending bad replies. Alt lens when the question is habit loss rather than UX regression → trigger/hook-decay analysis, not HEART.

Croll & Yoskovitz · Lean Analytics

NRR vs OMTM-by-stage

Default B2B NSM = Net Revenue Retention (expansion − contraction − churn on the existing base): >100% healthy, ~120%+ best-in-class (directional). NRR proves land-and-expand works and survives a CFO. But NRR is a lagging outcome — pair it with a stage-specific OMTM (One Metric That Matters), the single metric for your current constraint:

Stage / constraint	OMTM to obsess this quarter
Pre-PMF	% of accounts hitting the activation/aha event
Early expansion	seats/account growth or feature-attach rate
Scaling	NRR + gross logo retention
Efficiency	CAC payback months, magic number

Run NRR as the durable scoreboard AND one OMTM as the focusing function — reporting only NRR lets a team coast on last year's expansions while the leading flow rots.

PLG variant: score Product-Qualified Accounts, not PQLs → pro §growth §3

§6–7 · outcome metrics for agents, and the metrics fraud to catch

Count jobs, not events — and never confuse capture with value.

§6 · agent / AI outcome metrics

The moment the product acts for the user, event volume is vanity

A looping agent inflates "messages sent" / "API calls." Measure the outcome:

• Resolved-without-handoff rate (autonomous success) — the agent NSM.

• Task success / goal-completion vs a labeled set.

• Time-to-resolution and turns-to-resolution (fewer is better — the inverse of a chat product).

• Cost per successful task = (tokens + infra) ÷ resolved jobs — not cost per call; a 90%-success agent at 3× the cost-per-success of an 80% one may be worse.

Escalation-to-human rate + silent-failure rate (acted wrong, no flag) — the dangerous one; instrument an observable signal or it's invisible.

Tie to the AI three-tier stack: Quality (eval pass-rate) → Experience (task success, no-retry) → Unit economics (cost per successful task). A quality win that doesn't move experience is vanity; an experience win that blows the cost envelope isn't a win.

Eval pass-rate + silent-failure instrumentation → pro §ai-evals

§7 · the most common metrics fraud

Business-capture disguised as customer value

A metric that rises because the business extracted more, dressed up as the user receiving more value. The two move together in healthy products and diverge exactly when you're harvesting.

"Value" metric going up	What it may actually be	Pair it with
Ad impressions / screen	more interruption, not more value	sessions/user, D30 retention
Time-in-app	confusion / dark-pattern friction	task-success, time-to-complete
Revenue per user	price-gouging a captive base	NRR, voluntary churn, CSAT
"Engagement" notifications	re-engagement spam	unsubscribe + uninstall rate
GMV (marketplace)	subsidy-driven, unprofitable	take-rate, contribution margin, repeat rate

Test: for any NSM, write one sentence describing how it could rise while the customer is worse off (Goodhart). If you can't refuse that sentence with a paired guardrail already on the dashboard, the metric isn't safe to optimize. Count value the customer received and would pay again for, never value the business captured this quarter.

§8 · benchmark bands

Directional priors — verify current before you lean on them.

Directional / point-in-time

The bands, and the Sean Ellis PMF probe

Metric	Weak	Decent	Strong
Retention D1	<25%	~30–40%	40%+
Retention D7	<10%	~10–20%	20%+
Retention D30	<5%	~5–10%	10%+
DAU/MAU stickiness	<10%	~20%	50%+ (daily habit)
Quick Ratio (growth co.)	<1 (shrinking)	~1–1.5	2+
NRR (B2B SaaS)	<90%	100–110%	120%+
Freemium free→paid	<2%	~2–5%	5%+

PMF probe (Sean Ellis test): ≥40% of activated users would be "very disappointed" without the product → signal of product-market fit. Survey activated users only; an unactivated cohort answers about a product they never experienced.

§9–10 · the enterprise-value twin and the R&D-portfolio twin

Two more trees — rank by the swing, and read the ratio the budget hides.

The §1 North-Star tree factors customer value. These factor enterprise value (L.E.K.) and R&D-portfolio health (MIT) — same discipline, different numerator. HIGH

L.E.K. · Identifying & Managing Key Value Drivers (2017)

The value-driver tornado — rank by ΔNPV, then filter by controllability

Factor enterprise value (NPV / operating profit) into the operating drivers a team can move — price · volume · mix · cost · capital · retention — then rank by leverage, not manageability. Three steps: Map (disaggregate "until you reach the level where daily operating decisions reside") → Sensitivity test (perturb each ±10%, measure ΔNPV, rank by swing — the tornado) → Controllability filter (keep only what management can move).

Worked — L.E.K. petroleum case (all figures illustrative, ΔNPV swing)

Consumer/retail margin
uncontrollable — NOT a KPI

$485–681M · hedge

Retail investment

$385M · KPI

Retail volume (discounting)

$321M · KPI

Rail cost

$281M · KPI

Trucking cost

$26–33M

Industrial volume

+10% ≈ $14M

controllable, high-impact → KPI + tie to incentives high-impact but uncontrollable → hedge / strategy problem

The ranking upends priorities: the business had obsessed over industrial volume and trucking cost — both near-zero — while consumer/retail margin, retail investment, retail volume and rail cost dwarfed them. The sharpest lesson: the single highest-impact driver, margin, was uncontrollable (regulated pricing) — so it is NOT a KPI; the actual KPIs became the controllable high-impact drivers: discounting, retail volume, investment, rail cost.

The value-driver matrix · both tests required

	Low value impact	High value impact
High control	Monitor (cheap to watch)	Manage actively → KPIs + incentives
Low control	Low priority — ignore	Hedge the downside / reconfigure strategy

Two tests, both required: significant value impact AND controllable. Assign each surviving driver an owner by type — growth → marketing/sales, efficiency → ops, financial → finance — and substitute value drivers into the incentive system, tailored by function. Re-run when the cost/revenue structure shifts.

Meyer, Tertzakian & Utterback · MIT WP #124-95 (1995)

R&D-portfolio meta-metrics — Platform Efficiency & Leverage

Two ratios that read the family, not the project, and need no accurate up-front estimate:

E = R&D cost of a follow-on product ÷ R&D cost of the platform · L = product sales ÷ R&D cost

Platform Efficiency E — what fraction of the platform's build cost each derivative costs. ≤ ~0.10 ("ten cents on the platform dollar") = a strong, reusable architecture; rising toward 1.0 = a derivative costs what the platform did → not leveraging (poor design). If learning is real, E should fall across successive derivatives; a rising E is the warning. Platform Leverage L — commercial return per R&D dollar (use sales, not profit). In a healthy family E falls as L rises.

Leverage L — sales per R&D $ (peak = start renewal) Efficiency E — a rising E, not a high one, is the saturation signal

The renewal-timing read. Rising E + falling L = the architecture is saturating (the S-curve flattening) — invisible in annual budgets or product sales; only the ratios surface it. Start renewal at the leverage peak — the first sign E begins to climb — while the platform still looks healthy (note "efficiency" is a ratio where low is good, so a rising E, not a high one, is the saturation signal; don't wait for E to top out). A late start forces a crash effort — the paper's imaging family swung engineering allocation from 10% → 70% in a single year.

Worked — the paper's measurement-systems firm, Family B

Five derivatives off one platform ran ~0.11 efficiency until the fourth spiked to 0.42 (architecture at its limit); the third product returned $259 in sales per R&D dollar before leverage fell — management should have started renewal at that peak, not after a crash.

A sudden efficiency spike can also flag org dysfunction — a charter moved between labs, key engineers gone, "charter wars" — not just tech limits; diagnose the cause before acting. Read E and L with product-change, quality and market context (a fast-growing market flatters L; a component breakthrough flatters E).

Pair the renewal call with share-vs-vary platform decisions → pro §strategy §6

Keep reading