product-skill/ PRO/ AI Products & Evals PRO module

PRO · AI Products & Evals

A judge is a prompt with its own error rate — validate it before you trust it, or you're manufacturing false confidence.

The base file gives the doctrine; this module gives the executable depth — the fill-in eval plan, the worked error-analysis pass, the confusion matrix, the rollout gates. The spec is discovered from data, not authored upfront. All ~numbers are directional / point-in-time.

Validate the judge · "fail / bug found" = positive illustrative

human = FAIL

human = PASS

judge
= FAIL

TP40

FP7

judge
= PASS

FN5

TN48

TPR = TP / (TP+FN) = 40/45

0.89

catches real failures

TNR = TN / (TN+FP) = 48/55

0.87

doesn't cry wolf on good outputs

Report BOTH — high TPR / low TNR floods false alarms; high TNR / low TPR ships bugs.

Husain & Shankar · the loop

Analyze → measure → improve. Not test-first.

The failure surface is ~infinite — don't write evals before you've seen failures. Ship thin, collect traces, error-analyze, then build an evaluator only for an observed failure mode. (Exception: hard constraints — assert those up front.)

Analyze

error-analyze traces

Measure

build the evaluator

Improve

fix, then re-run

re-run every change — error analysis is ~60–80% of dev time, by hand (directional)

Discover, don't author

The spec comes from data.

"Evals are the new PRD" is useful folklore — but the spec is discovered from failures, not written upfront. A generic off-the-shelf "helpfulness" judge is vanity.

By hand

Error analysis is the work.

~60–80% of dev time is a human reading traces and coding the first thing that went wrong. 100% pass = the system isn't being challenged.

Validate the judge

An unvalidated judge lies.

It's a prompt with its own error rate. Without a confusion matrix against human labels, it manufactures false confidence — report TPR and TNR both.

Section 1 · the fill-in eval plan

If a slot is blank, you have a vibe — not a spec.

Copy this and fill every slot. It's the operational spec your eng team will ask for.

FEATURE: ____ JOB-TO-BE-DONE (Christensen): When I __, I want to __, so I can __. "GOOD" DEFINED FOR (audience): ____ (helpful differs per persona) HARD CONSTRAINTS (assert up front, no judge): valid JSON | never names competitor | no PII | ____ MVQ BAR: do-not-ship = ___ | acceptable = ___ (+fallback) | delight = ___ COST ENVELOPE: $___ /successful task ceiling · p95 latency ___ ms DATASET: ~100 representative traces from ____ (real ⟶ preferred; else synth via dims ____) FAILURE TAXONOMY: discovered from data — NOT authored upfront JUDGE: binary pass/fail + critique; validated vs human labels; TPR/TNR reported OFFLINE (CI): golden set N=___, deterministic asserts + judge, runs every change ONLINE (prod): sample ___% live traces, reference-free judge, CIs, weekly review ROLLOUT: shadow → internal → cohort %; kill-switch = ___ OWNER: PM owns taxonomy + "what good means" + golden labels; eng owns harness/CI.

The PM owns the taxonomy, "what good means," and the golden labels; eng owns the harness and CI. That ownership line is the one people skip.

Section 3 · the worked pass

Open-code 100 traces, then build evals only for the rows that hurt.

Feature: an AI mortgage-doc summarizer for first-time homebuyers. A domain expert reads each trace, free-text note on the first thing that went wrong (upstream errors cause downstream symptoms — code the head). illustrative

Open-code the first thing that went wrong on each trace, then axial-code until saturation ≈ ~20 traces with no new bucket (an LLM can propose groupings; you refine).

Read the raw open-coding notes

"invented an APR figure not in the source doc" · "refused — said 'consult a financial advisor' instead of summarizing" · "summary correct but 1100 words, user wanted the gist" · "missed the prepayment-penalty clause entirely" · "used 'amortization' with no plain-English gloss" · "JSON broke — trailing comma"

Failure mode	Count /100	Hurts?	→ Targeted eval
Hallucinated number (APR, fee)	14	critical (trust/legal)	judge: every figure traceable to source
Missed material clause	11	critical	checklist eval vs clause list
Over-length / not "the gist"	23	medium	deterministic: word-count ≤ 200
Jargon unglossed for persona	9	medium	judge: persona-appropriate language
Unhelpful refusal	6	high	judge: refused a benign request?
Format break (JSON)	4	low (assert)	deterministic schema check
No issue	33	—	—

Build evals for the rows that hurt, not all of them. The 23 over-length cases are cheap to fix (a deterministic assert, no judge). The 14 hallucinated numbers are the real spec — that's the MVQ do-not-ship line. Then 10–20 fresh traces/week catch drift and new modes. 100% pass = system not challenged.

Section 4 · LLM-as-judge validation

Validate the judge, or it manufactures false confidence.

Shankar et al. arXiv:2404.12272 · Husain & Shankar

The protocol

Binary pass/fail + written critique. NOT a 1–5 Likert — adjacent points are noise; annotators camp the middle and criteria drift.
The PM hand-labels a held-out set (~50–100 traces) — the benevolent-dictator labeler. Multiple annotators = noise.
Run the judge on the same set. Build the confusion matrix (treat "fail / bug found" as positive) → the hero above: TP 40, FN 5, TN 48, FP 7 → TPR 0.89, TNR 0.87.
Report both — no single fixed % is canonical; iterate the judge prompt until both are high enough for the stakes.

TPR = TP/(TP+FN) (catches real failures) · TNR = TN/(TN+FP) (doesn't cry wolf)

Bias controls

Position — randomize A/B order in pairwise judging. Verbosity — the judge prefers longer answers; control length or instruct against it. Self-preference — models favor their own output; don't let the generator judge itself.

Prefer a panel of judges (e.g. 3 different models, majority / average) over one. Re-validate the judge whenever you change its prompt or swap the model behind it.

Sections 5, 6, 9 · run it in production

Two lanes, a growing golden set, three gulfs to bridge.

Nika & Granados (AIPDL) · EDDOps

5 · Offline (CI) vs Online (prod)

	Offline / CI	Online / prod
Dataset	small curated golden set	sampled live traces
Evaluators	deterministic asserts first, then validated judge	reference-free judges (no ground truth)
Cadence	every commit / prompt / model swap	continuous sample + weekly failure-mode review
Watch	pass-rate per failure mode, regression count	CIs, drift, new clusters → promote to golden set

EDDOps (eval-driven dev): the golden set is the regression suite; prod failures flow back into it. This is the AIPDL "deployment ≠ completion" stage made operational — monitoring is a stage, not an afterthought, because models drift and inputs shift.

6 · Golden set + regression promotion

Small, curated, labeled inputs + pass/fail — high-signal, not big. Seed one canonical example per failure mode + clear passes, balance across the synth dimensions, label personally.

prod failure → minimal reproducing input → permanent golden case (dated, tagged with the mode it guards)

The set only grows; you never delete a regression case.

Shankar & Husain · the Three Gulfs

9 · What each eval artifact attacks

Comprehension

You can't read all your data.

→ error analysis on a 100-trace sample; online sampling

Specification

Intent → prompt is lossy.

→ golden set + validated judge = the real spec; criteria pinned

Generalization

Works on seen, fails on unseen.

→ fresh weekly traces; shadow-mode at scale; regression promotion

Sections 7 & 8 · set the bar, count the cost

What "good enough to ship" means — and what it costs.

Nika · MVQ tiers + cost envelope

7 · MVQ tier worksheet

Set the bar per feature before building. (Numbers are a contextual rule of thumb, not a standard.)

Delight≥9.5/10, no material-clause misses — autonomous, near-perfectraise threshold / version

Acceptable≥8/10 right, no-retry; misses caught by human reviewship w/ fallback

Do-not-shipfails a hard constraint or erodes trust — any hallucinated numberblock

Rule of thumb (directional): ~8–9/10 right with no retry feels magical; ~1-in-5 visible misses erodes trust. The quality ceiling rises across versions — V1 launches with human-in-loop fallback and you set leadership / customer expectations on that ceiling up front (an optics deliverable, not just an eng note).

cost-per-inference envelope: (tokens_in·$ + tokens_out·$ + infra) ÷ resolved jobs ≤ $___

Delight that loses money per successful task doesn't ship. Hold this envelope when pricing on outcomes — model cost is variable COGS.

8 · Non-generative ML — classification metrics

For ranking / scoring / extraction / risk models, evals are classification metrics, not judges. Pick by which error hurts more — a business call, not a default.

Metric	Use when	Trap
Precision = TP/(TP+FP)	false positives costly (spam flag, fraud block)	ignores misses
Recall = TP/(TP+FN)	false negatives costly (disease, churn, AML)	ignores false alarms
F1 = harmonic mean(P,R)	need one number, classes imbalanced	hides which error dominates — report P and R too
ROC-AUC	threshold-independent ranking quality	optimistic on heavy imbalance → prefer PR-AUC there

For clinical / credit / regulated features add a demographic bias / fairness evaluation across segments — fairness is a guardrail, not an afterthought.

Section 10 · shadow-mode rollout

Run it silently first, then widen behind a flag.

Run the AI alongside the human / incumbent to gather at-scale eval data at zero customer risk, THEN widen.

Stage 0 · SHADOW

AI runs, output logged, NOT shown

Compare vs human on live volume. Gate: offline pass-rate holds on real traffic; no new critical mode.

Stage 1 · INTERNAL

Dogfood

PM + experts review daily.

Stage 2 · COHORT

Feature-flag X%

Start ~1–5% (directional). Monitor MVQ tiers + cost.

Stage 3 · WIDEN

Raise % only while it holds

Raise only while online judge pass-rate + cost envelope hold.

Kill-switch — define BEFORE Stage 2

Auto-rollback trigger = do-not-ship rate > ___% OR cost/task > envelope OR latency p95 > ___ OR a critical-mode spike. Flag flip = instant revert to human / previous version, no redeploy. Name the owner who can pull it and the alert that pages them — a kill-switch nobody can see fire is a silent failure (actor-observability). Post-rollback: capture the failing input as a regression case before re-attempting.

Closing pitfalls

A generic off-the-shelf "helpfulness" judge = vanity. And "evals are the new PRD" is useful folklore (Foody / Braintrust) — the spec is discovered from data, not authored upfront.

Keep reading