← Skill graph
Calibrating automated scores to human judgement
Part of LLM evaluation
Helps first
Nothing recommended beyond what is required.
Unlocks
Nothing depends on this one.
Where it is assessed
6 assessed items, 0 done.
- AIE-102 M4.3 Label fifty cases yourself, blind · tier 3
- AIE-102 M4.4 Calibrate to threshold, and show the trajectory · tier 1
- AIE-102 M4.5 The disagreement that mattered · tier 2
- AIE-102 M4.6 Retention: calibration and agreement statistics · tier 1
- AIE-102 M4.7 Re-grade your own advanced standing · tier 2
- AIE-102 M7.1b Evaluation report: results, intervals, and the unflattering finding · tier 2
Verified sources
- Your AI Product Needs Evals
- Creating a LLM-as-a-Judge That Drives Business Results
- Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference
- Training language models to follow instructions with human feedback
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
- Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences
- sklearn.metrics.cohen_kappa_score
- Why AI evals are the hottest new skill for product builders · Lenny's Podcast