Reinforcement Learning
AIE-2063 credits · 65h required · after AIE-201, AIE-203
The lowest immediate employability of the deep track, and placed late on purpose rather than apologised for. Its value is that RLHF and reasoning-model training are built on it, and AIE-205 taught preference optimisation as a method without ever explaining the problem it was avoiding. This course explains the problem. By the end, DPO's whole appeal — that it removes the reinforcement learning — is something you can argue about rather than repeat.
The lab. Classic control environments on a laptop, and then your own AIE-205 checkpoints. The first half needs nothing but CPU — cart-pole and grid worlds are small enough to train in seconds and are the right size for watching an algorithm fail. The second half returns to the model you fine-tuned, which is where the abstractions stop being toys.
By the end you can
- Formalise a problem as an MDP
- Implement a value-based and a policy-gradient agent
- Explain the credit assignment problem and how RLHF inherits it
- Connect policy optimisation to preference optimisation from AIE-205
What this builds
- Reinforcement learning and RLHF0/16 demonstrated
Next assessment. Formalise something of your own as an MDP — 180 min, tier 1. Machine-checkable, so you can attempt it before reading anything and find out whether you need to.
0/16 assessed items demonstrated · 0/14 read or watched. The second number is how you close a gap; only the first is evidence that it closed.
0 of 16 demonstrated · 0/14 read
0m of 65h
M1 · Markov decision processes
0/6 · 13hreading · 3h · tier 0 self-marked
reading · 2h 30m · tier 0 self-marked
assignment · 3h · tier 1 machine-verified
assignment · 2h 30m · tier 2 panel-assessed
reading · 1h 45m · tier 0 self-marked
retention · 15m · tier 1 machine-verified
M2 · Value-based methods
0/6 · 13hreading · 2h 45m · tier 0 self-marked
lecture · 2h 30m · tier 0 self-marked
assignment · 3h · tier 1 machine-verified
assignment · 2h 45m · tier 2 panel-assessed
reading · 1h 45m · tier 0 self-marked
retention · 15m · tier 1 machine-verified
M3 · Policy gradients
0/6 · 14hreading · 3h · tier 0 self-marked
reading · 2h 30m · tier 0 self-marked
assignment · 3h · tier 1 machine-verified
assignment · 2h 45m · tier 2 panel-assessed
reading · 2h 30m · tier 0 self-marked
retention · 15m · tier 1 machine-verified
M4 · From policy optimisation to preference optimisation
0/6 · 13hreading · 2h 30m · tier 0 self-marked
Training language models to follow instructions with human feedback
reading · 2h · tier 0 self-marked
assignment · 3h · tier 2 panel-assessed
assignment · 2h 45m · tier 2 panel-assessed
reading · 2h 30m · tier 0 self-marked
retention · 15m · tier 1 machine-verified
M5 · Publish, and defend
0/6 · 12hproject · 3h · tier 3 artifact
assignment · 2h 45m · tier 2 panel-assessed
lecture · 2h 30m · tier 0 self-marked
defense · 1h · tier 4 defended
reading · 2h 30m · tier 0 self-marked
retention · 15m · tier 1 machine-verified