← Skill graph
Statistical significance in evals
Part of LLM evaluation
Needs first
Helps first
Nothing recommended beyond what is required.
Where it is assessed
8 assessed items, 0 done.
- AIE-102 M3.8 Measure your own judge's biases · tier 1
- AIE-102 M4.4 Calibrate to threshold, and show the trajectory · tier 1
- AIE-102 M5.3 An interval, not a number · tier 1
- AIE-102 M5.4 How many cases do you actually need? · tier 1
- AIE-102 M5.5 Retention: intervals, power, and variance · tier 1
- AIE-102 M5.6 Rerun M3.7 with intervals · tier 1
- AIE-102 M7.1b Evaluation report: results, intervals, and the unflattering finding · tier 2
- AIE-102 M7.2 Oral defense · tier 4
Verified sources
- Task-Specific LLM Evals that Do & Don't Work
- Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference
- A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification
- Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators
- Am I More Pointwise or Pairwise? Revealing Position Bias in Rubric-Based LLM-as-a-Judge
- Self-Consistency Improves Chain of Thought Reasoning in Language Models
- scipy.stats.bootstrap
- sklearn.metrics.cohen_kappa_score