Building eval datasets

Where it is assessed

3 assessed items, 0 done.

• AIE-102 M2.5 A gold eval set for evidence extraction · tier 1
• AIE-102 M2.6 Retention: eval vocabulary and error-analysis method · tier 1
• AIE-102 M7.1a Evaluation report: methodology and eval sets · tier 2

Verified sources

• LLM Evals: Everything You Need to Know
• BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models
• Common Voice: A Massively-Multilingual Speech Corpus
• CS-Dialogue: A 104-Hour Dataset of Spontaneous Mandarin-English Code-Switching Dialogues for Speech Recognition
• DocVQA: A Dataset for VQA on Document Images
• Language Models are Few-Shot Learners
• Holistic Evaluation of Language Models
• SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
• stanford-crfm/helm
• EleutherAI/lm-evaluation-harness
• openai/evals
• Why AI evals are the hottest new skill for product builders · Lenny's Podcast