← Capabilities
LLM evaluation
not yet earnedI can say with a number whether an LLM system is any good, defend the method that produced the number, and show where the measurement itself breaks down.
What this claim costs
- 6 assessed items at tier 2 or above0 of 6 — 6 more to do, from 45 available
- A public artifact built for this claim — eval-harness
- A documented negative result — where this stops workingNothing recorded. A capability with no known failure boundary has not been explored.
- Defended aloud, against questions you did not write2 available, none done
- Recalled cold, a week later, without notes8 scheduled, none done
The proof
A replay-first harness with per-tag reporting and a baseline that fails CI on regression, run against a hand-built gold set including negatives.
What it has to contain
- A gold set of 30+ cases with 6 or more genuine negatives
- A harness run producing per-tag results, not a single number
- A baseline the system had to beat, and the comparison with an interval
- A regression suite wired to fail on a real regression
Where it stops working. Not established yet. Anyone can ship a demo; the boundary is the part that is worth reading, so this claim is not earned until it is written down.
Skills underneath
- Eval design
- Building eval datasets
- LLM-as-judge and its pathologies
- Statistical significance in evals
- Calibrating automated scores to human judgement
- Regression testing and CI for models
- Tracing and production observability
45 assessed items across the specified courses evidence these, 0 done.