← Skill graph
LLM-as-judge and its pathologies
Part of LLM evaluation
Needs first
Helps first
Nothing recommended beyond what is required.
Where it is assessed
6 assessed items, 0 done.
- AIE-102 M3.7 Build a judge, version one · tier 1
- AIE-102 M3.8 Measure your own judge's biases · tier 1
- AIE-102 M3.9 Retention: named judge pathologies and their controls · tier 1
- AIE-102 M5.6 Rerun M3.7 with intervals · tier 1
- AIE-102 M7.1b Evaluation report: results, intervals, and the unflattering finding · tier 2
- AIE-102 M7.2 Oral defense · tier 4
Verified sources
- LLM Evals: Everything You Need to Know
- Creating a LLM-as-a-Judge That Drives Business Results
- G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment
- Judging the Judges: A Systematic Evaluation of Bias Mitigation Strategies in LLM-as-a-Judge Pipelines
- Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
- Am I More Pointwise or Pairwise? Revealing Position Bias in Rubric-Based LLM-as-a-Judge
- Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences
- confident-ai/deepeval
- Why AI evals are the hottest new skill for product builders · Lenny's Podcast