← Skill graph
Preference optimisation
Part of Fine-tuning and post-training
Needs first
Helps first
Nothing recommended beyond what is required.
Unlocks
Where it is assessed
7 assessed items, 0 done.
- AIE-205 M4.3 The DPO loss, from the equation · tier 1
- AIE-205 M4.4 Run DPO, and find what else moved · tier 2
- AIE-205 M4.6 Retention: preference optimisation · tier 1
- AIE-205 M5.1 Regression sweep on what you did not intend to change · tier 2
- AIE-205 M5.3 The alignment tax, measured on your own model · tier 2
- AIE-205 M5.4 Oral defense · tier 4
- AIE-205 M5.6 Retention: the post-training pipeline and its costs · tier 1
Verified sources
- Deep Dive into LLMs like ChatGPT · Andrej Karpathy
- Constitutional AI: Harmlessness from AI Feedback
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- Training language models to follow instructions with human feedback
- Llama 2: Open Foundation and Fine-Tuned Chat Models
- huggingface/trl