AI Engineering

Notes on making probabilistic systems shippable: evaluation harnesses, LLM judges, context engineering, agents, and the GCP estate underneath them. Everything here comes from production systems — method extracted from work, written to be reused.

2026-08-05 · evals · harness · agents

Calibrating an LLM Judge Until Its Verdict Can Block a Release

Everyone builds an LLM judge. Almost nobody calibrates it. Here is what calibration actually looked like on a production agent system — and the three failures that taught me what a judge should never be asked to score.

Read →

2026-08-05 · evals · testing · agents

Hermetic Golden Cases: Your Agent Suite Is Lying to You

A golden suite whose cases share state reports whatever the accident of execution order produces. How per-case hermetic users fixed a lying pass rate — and made parallelism free.

Read →

Related: the case study on evaluation gates for a 700,000-student agent system, the GCP dossier, and the working papers on agent harnesses and structural waste.