Golden set
canonSmall, curated, labeled inputs + pass/fail. Promote every new production failure into a permanent regression case.
Evals-as-QA · MVQ · AIPDL · the mode fork
When building is cheap, proving it works becomes the constraint — and the eval suite is the spec.
The deepest shift: knowing what to build and proving it works is the scarce input. Read this for any LLM-backed feature. Evals convert "the model is good enough" from a bet into measured evidence.
Step 1
Analyze
Ship thin, collect real traces, read them by hand — error analysis.
Step 2
Measure
Build evaluators ONLY for observed failure modes — not before.
Step 3
Improve
Fix the failure mode; promote it to a permanent regression case.
Re-run every change. NOT test-first — the failure surface is ~infinite, so the spec is discovered from data, not authored upfront.
Analyze → measure → improve · the eval loop the PM owns
Above the fold
The three moves that carry AI-native judgment.
01
Analyze → measure → improve, not test-first.
The failure surface is ~infinite; don't write evals before seeing failures. Ship thin, read real traces, then build evaluators only for observed failure modes.
02
Error analysis IS the job.
~60–80% of dev time is reading traces by hand (directional). The PM owns the failure taxonomy and "what good means" — be the single benevolent-dictator labeler.
03
Name the mode before you match the tools.
Generative (artifact a human reviews) vs agentic (multi-step actions). Misclassifying agentic as generative is the root of "the demo worked but it loops in prod."
Evals-as-QA · the AI-era QA discipline the PM owns
The eval suite IS the spec — vibe-checks don't survive production.
"The model is good enough for this job" is a bet. Evals convert it to measured evidence and catch the day a prompt tweak or model swap silently breaks it. Exception: hard known constraints (never names a competitor, valid JSON) you assert up front.
Error analysis IS the job · ~60–80% of dev time reading traces by hand (directional)
01
~100 traces
Gather ~100 representative traces from real usage.
02
Open coding
A domain expert free-text notes the FIRST thing that went wrong.
03
Axial coding
Cluster notes into a named failure taxonomy + counts (illustrative buckets — hallucination / refusal / format-break / edge-case / demographic-skew, not prescriptive). Stop at saturation (~20 traces, no new category).
04
Targeted evals
Build a metric only for categories that actually hurt — these become targeted evals. Then 10–20 fresh traces/week.
LLM-as-judge — validate the judge
canonBinary pass/fail + written critique, NOT 1–5 Likert (adjacent points are noise; annotators camp the middle). A judge is a prompt with an error rate — align to human labels on a held-out set; report TPR and TNR (no single fixed % is canonical). Watch position / verbosity / self-preference bias; prefer a panel.
Offline vs online
EDDOpsOffline (CI): small golden set + cheap deterministic assertions every change. Online (prod): sample live traces, reference-free judges, track CIs, feed new failures back.
Three Gulfs
Shankar & HusainComprehension (you can't read all your data), Specification (intent → prompt is lossy), Generalization (works on seen inputs, fails on unseen) — evals attack all three.
PM owns the taxonomy — and the warnings
The PM owns the failure taxonomy and "what good means" (helpful differs for an investor vs a first-time homebuyer) — be the single benevolent-dictator labeler; hand harness/CI to eng. 100% pass = system not challenged; generic off-the-shelf "helpfulness" judges = vanity. "Evals are the new PRD" is useful folklore (Foody / Braintrust) — the spec is discovered from data, not authored upfront.
Sources: Hamel Husain & Shreya Shankar (hamel.dev evals-faq, Field Guide); Aman Khan (Arize, Lenny's "Beyond vibe checks"); Shankar et al. arXiv:2404.12272 (criteria drift).
AI product sense + MVQ guardrails · Marily Nika
Set a Minimum Viable Quality bar per feature.
Do-not-ship
Fails a hard constraint or erodes trust.
Acceptable
Ships with fallback / human-in-loop.
Delight
The bar you aim to raise toward across versions.
Rule of thumb
directionalAn answer that's right ~8–9/10 with no retry feels magical; ~1-in-5 visible misses erodes trust. Directional, not a standard.
DirectionalCost + cadence
NikaDefine a cost-per-inference envelope up front and run a weekly failure-mode review on real outputs.
Attribution
noteMVQ is Nika's framework, not Meta's. That Meta added a "Product Sense with AI" interview in 2026 is a separate signal that AI product sense is now a hiring bar.
AIPDL · Nika & Granados (Wiley 2025)
A ~6-stage lifecycle where deployment ≠ completion.
Models drift, inputs shift, failure modes surface only in production — so monitoring / iteration is a stage, not an afterthought. Three AI-PM archetypes span platform / applied / research-adjacent PMs.
01
Scope
02
Data
03
Prototype
04
Evaluate
05
Deploy
06
Monitor & iterate
Model-Capability FIRST
D/V/FAI-adapted Desirability / Viability / Feasibility (base = IDEO / Tim Brown). Test Model-Capability first (offline golden-set eval) — it's the AI-specific risk, prove it before D/V/F. Desirability: do users want AI in this job (not novelty)? Viability: does margin survive token cost at scale? Feasibility: integration + Model-Capability against the customer's Job to Be Done, on a golden set before committing design.
Quality ceiling across versions
lifecycleV1 launches with a quality ceiling needing human-in-the-loop fallbacks; later versions may run autonomously. Early adopters tolerate imperfection, mainstream users expect near-perfection — so raise confidence thresholds across versions. Set expectations on the V1 ceiling up front: an optics deliverable, not just an eng note.
Shadow mode + metrics
rolloutBefore flag → internal → live-cohort, run AI silently alongside humans to gather at-scale eval data at zero customer risk. Non-generative ML uses classification metrics — precision, recall, F1, ROC-AUC. For clinical/regulated features, add a demographic bias / fairness evaluation across segments.
PM-as-bottleneck · discovery matters more as build cheapens
Learning-velocity beats shipping-velocity.
As code gets cheap, the scarce input becomes which problem — shipping fast is vanity if you're shipping the wrong thing. The leverage moves to discovery and taste.
The inverted ratio
Provocation
Some AI-native teams floated "1 PM : 0.5 eng." Read literally, it's a headcount target.
Read it as
A provocation, not a norm — it dramatizes that discovery is the constraint. (Ng via Rachitsky; Cagan/SVPG; Torres; Oji Udezue)
AI won't hide a bad product
Tell
Faster build feels like progress, so teams accelerate output.
Fix
Faster build only ships the wrong thing sooner (paraphrase, not a quote). Move leverage to discovery.
Generative vs agentic · name the mode, then match everything
The classification that sets eval, metric, price and guardrail.
A feature is generative when it produces an artifact a human reviews; agentic when it takes multi-step actions toward an outcome with less review. The mode forks everything downstream.
Generative
Agentic
What it is
Produces an artifact a human reviews — draft, summary, answer.
Multi-step actions toward an outcome with less review — plans, calls tools, acts for the user.
Eval
Output quality + human-in-loop. Score the artifact — golden set, LLM-as-judge on outputs (§Evals-as-QA as written).
Scoring the final string isn't enough — add trajectory + tool-call evals: did it finish the job, via which steps, wrong turns, and tool calls?
Metric
Output-quality pass-rate.
Jobs completed, not events fired (outcome-not-event).
Price
Seats can still work.
Seats break → outcome / consumption pricing.
Guardrail
Human reviews the artifact before it lands.
More autonomy = bigger blast radius → tighter guardrails + human-fallback on high-stakes steps.
The root cause of "worked in the demo"
Many agent failures are step-repetitions or reasoning↔action mismatches a final-output check never catches. Misclassifying an agentic feature as generative is the root of "the demo worked but it loops / forgets in prod."