Note
#56 - Grading AI with Evals / Multi-Bot Rubrics / LLM as Judge
Link to Folder with Everything
Recording
No video this session — Meet's recording didn't run. The session notes below were compiled from Granola, and the full notes doc is in the Drive folder.
Run the Eval Bench Yourself
Want the build, not just the story? The companion note has the one prompt every bot got, the wiring between Todoist and Notion, the rubric, the dry-run scores, and the screens.
#56 - Build The Eval Bench: Prompts, Runs & Screens
The live rubric, transcripts, and Evals database are linked from there.
Summary
Office hours 56 took on the question every AI builder eventually hits: how do you know your AI is any good — and how do you know it's still good next month? The answer is evals: grade AI output against a rubric, with a different model as the judge. Rahul and Jabari Grubb each built a working eval pipeline live, from opposite ends of the cost spectrum.
Why evals, and the one rule that matters
An eval is three things: a task, a rubric, and a golden set of expected behavior. You run it to catch model drift — prompts change, models upgrade, tools get added, and quality moves without anyone noticing — and to grade the intermediate steps of agentic pipelines, not just the final answer. The one rule: never let a model grade its own work. Claude always gives itself an A. Cross-model grading — ChatGPT judging Claude, Grok judging ChatGPT — is what makes the scores mean something.
Rahul's build: three bots, one rubric, Todoist as dispatcher
Claude, ChatGPT, and Grok each summarized the same transcript into an action plan. A four-criteria rubric — completeness, fidelity, format, audience fit, scored 0–2 — was itself generated by asking Claude what matters for the task type. Todoist dispatched the work and collected each bot's answer as comments on the task; results landed in Notion, where the rubric stays live and editable so the whole test reruns after any change. The scores had personality: ChatGPT graded Grok harshly and itself generously, Grok was the most discerning judge, and Claude won both rounds.
Jabari's build: the zero-dollar pipeline
The same discipline, no API bill: Gemma 3 generating answers, Mistral judging them, Python glue, and Notion holding the golden set (12 cases), the results, and a run dashboard. Notion AI triages the failures by error type. The punchline came from a prompt A/B test: pass rates moved dramatically, not marginally — and the best run still only hit ~58%, which the dashboard rightly flagged as barely better than a coin flip. Without the eval, nobody would have known.
Choosing your layer
Three tiers, cheapest first: deterministic checks (schema, regex, required fields — no AI needed), LLM-as-judge for the fuzzy qualities like faithfulness and tone, and humans in the loop for high-stakes cases. Judge bias never goes away — you pick the bias you can live with. Evals can double your API cost, so a follow-up session is planned on decision models as the cheaper alternative to LLM judges. And the discussion landed on why any of this matters commercially: when a client says "this is not right," months of eval history is the difference between an argument and an answer.
Next steps
- [Rahul Singh] Publish the rubric, Notion instructions, and handoff doc on intelcraft.ai.
- [Jabari Grubb] Share his Notion workspace and Notion AI playbook with attendees.
- [IntelCraft] Follow-up office hours on decision models vs. LLM judges — the time and cost case for in-code decision evals.
- Privacy note for anyone logging eval data: anonymize question/answer pairs before grading, and use enterprise or zero-retention APIs when customer data is involved.
Details
- Session: officehours 56, October 28, 2026. Rahul Singh hosting, Jabari Grubb co-presenting.
- The rubric mechanics: criteria scored 0–2 with decimals allowed; rubric generation itself delegated to a model, then human-edited — the rubric lives in Notion as data, not in a prompt.
- Fine-tuning angle: high-scoring golden answers accumulate into a fine-tuning dataset (~$100 to run) — a practical by-product of running evals, versus full training at costs no one in the room is paying.
- Lightweight alternative: thumbs up/down user feedback covers a surprising amount of ground and sits inside standard ToS — the floor, not the ceiling.
- No recording this session: Meet didn't capture video; this write-up was composed by Claude from the Granola notes, per the series practice when a scribe is the only artifact.