← Skill graph
Eval design
Part of LLM evaluation
Needs first
Helps first
Nothing recommended beyond what is required.
Where it is assessed
29 assessed items, 0 done.
- AIE-102 M1.5 The question you cannot answer · tier 2
- AIE-102 M2.4 Error analysis on 1 Percent More Fluent · tier 2
- AIE-102 M2.6 Retention: eval vocabulary and error-analysis method · tier 1
- AIE-102 M2.7 What your error analysis missed · tier 2
- AIE-102 M4.7 Re-grade your own advanced standing · tier 2
- AIE-102 M7.1a Evaluation report: methodology and eval sets · tier 2
- AIE-102 M7.1c Evaluation report: publish · tier 3
- AIE-102 M7.2 Oral defense · tier 4
- AIE-300 M1.3 The evaluation plan, registered before any code · tier 1
- AIE-300 M1.6 Publish the scope document · tier 3
- AIE-300 M2.9 Observability, so failures are visible · tier 1
- AIE-300 M3.1 Build the eval set the plan specified · tier 1
- AIE-300 M3.2 Run it, and record the baseline · tier 1
- AIE-300 M3.3 Error analysis on real traces · tier 2
- AIE-300 M3.4 Beat the baseline you named · tier 2
- AIE-300 M3.5 How much of this is noise · tier 2
- AIE-300 M3.6 A regression suite that runs itself · tier 2
- AIE-300 M3.7 Act on the error analysis, and measure · tier 2
- AIE-300 M3.8 Where this system stops working · tier 2
- AIE-300 M3.9 Retention: the system's measured behaviour and limits · tier 1
- AIE-300 M5.1 The ten-minute assessment · tier 3
- AIE-300 M5.2 The technical write-up · tier 3
- AIE-300 M5.4 The limitations section, written honestly · tier 2
- AIE-300 M5.6 Update the capability claims this earns · tier 2
- AIE-300 M5.7 Retention: the write-up's claims and their evidence · tier 1
- AIE-300 M6.1 The questions you are afraid of · tier 2
- AIE-300 M6.2 The capstone defense · tier 4
- AIE-300 M6.3 Correct what the defense exposed · tier 2
- AIE-300 M6.5 Retention: the capstone, cold · tier 1
Verified sources
- AI Engineering
- Building Machine Learning Powered Applications
- Task-Specific LLM Evals that Do & Don't Work
- Patterns for Building LLM-based Systems & Products
- LLMOps (LLM Bootcamp) · The Full Stack
- Your AI Product Needs Evals
- LLM Evals: Everything You Need to Know
- Extrinsic Hallucinations in LLMs
- Holistic Evaluation of Language Models
- RAGAS: Automated Evaluation of Retrieval Augmented Generation
- stanford-crfm/helm
- UKGovernmentBEIS/inspect_ai
- openai/evals
- promptfoo/promptfoo
- Why AI evals are the hottest new skill for product builders · Lenny's Podcast
- How to Build and Evaluate AI systems in the Age of LLMs · DataTalksClub ⬛
- Escaping Proof-of-Concept Purgatory: Building Robust LLM Powered Applications · SciPy