Senior SDET, a decade building test platforms.
01. GenAI
View All
Series
Evals Are Checks, Not Tests
GENAI_TESTING
Simulating the User: Test Between the Turns
A golden case tests one turn; the failures that matter live across many. Simulate persona conversations before production, scored by a calibrated evaluator.
14 MIN READ
GENAI_TESTING Agent Trajectory Testing: Grade the Path, Not the Answer
An agent can say the right thing and do the wrong thing. Grade the trajectory it took, the tools, the order, the confirm, not just the final answer.
GENAI_TESTINGRAG Testing: Grade the Layers, Not the Box
A fluent answer hides a broken retriever, so you do not test a RAG pipeline as one box. Test each layer: retrieval, reranking, grounded versus true, graph traversal, and the poisoned document. Then test the parts that are not deterministic with classical tools.
GENAI_TESTINGHonest Benchmarks: Eval Numbers That Hold Up
An 84% that beats 81% can be pure noise: p=0.55 on a 3-point gap. Put a bar on every number, run the paired test, gate on the lower bound, never the point.