Indie Degree
GitHubSign in
← Programme

Transformers and LLMs from Scratch

AIE-204

4 credits · 80h required · after AIE-203

The flagship of Block II, and the largest course in the programme. It ends with a working GPT you can explain line by line — from the byte-pair tokeniser you wrote through to the sampler — which is the single most legible artifact this degree produces. The thing being bought is not the ability to build one; nanoGPT already exists. It is that after this, every architectural claim in a paper or a vendor's release notes is checkable against something you have implemented rather than taken on trust.

The lab. 1 Percent More Fluent, for the half of this course nobody expects. It carries text in several languages, and the same sentence costs wildly different numbers of tokens depending on which — a real, measurable cost and latency problem sitting inside a product you already run. The tokeniser module is not an academic exercise there; it is the explanation for a bill.

By the end you can

  • Implement multi-head self-attention from scratch, without reference
  • Train a small GPT end to end, from tokeniser to sampling
  • Explain every architectural choice in a modern decoder and what it is buying
  • Reproduce a published result and account for the gap where you fail to

What this builds

Next assessment. Byte-pair encoding from scratch — 180 min, tier 1. Machine-checkable, so you can attempt it before reading anything and find out whether you need to.

0/20 assessed items demonstrated · 0/17 read or watched. The second number is how you close a gap; only the first is evidence that it closed.

0 of 20 demonstrated · 0/17 read

0m of 80h

M1 · Tokenisation, which is where the bill comes from

0/5 · 12h

  • lecture · 3h · tier 0 self-marked

    Neural Networks: Zero to Hero — Andrej Karpathy

  • reading · 2h 45m · tier 0 self-marked

    Introduction - Hugging Face NLP Course

  • assignment · 3h · tier 1 machine-verified

  • assignment · 3h · tier 2 panel-assessed

  • retention · 15m · tier 1 machine-verified

M2 · Attention, implemented

0/7 · 14h

M3 · A GPT you can explain line by line

0/6 · 15h

  • lecture · 3h · tier 0 self-marked

    Let's build GPT: from scratch, in code, spelled out — Andrej Karpathy

  • reading · 2h 45m · tier 0 self-marked

    The Annotated Transformer

  • assignment · 3h · tier 1 machine-verified

  • assignment · 3h · tier 2 panel-assessed

  • reading · 3h · tier 0 self-marked

    Build a Large Language Model (From Scratch)

  • retention · 15m · tier 1 machine-verified

M4 · What changed since 2017

0/7 · 13h

M5 · Reproduce a published result

0/6 · 14h

  • lecture · 3h · tier 0 self-marked

    Let's reproduce GPT-2 (124M) — Andrej Karpathy

  • reading · 2h 45m · tier 0 self-marked

    karpathy/nanoGPT

  • project · 3h · tier 2 panel-assessed

  • project · 2h 30m · tier 2 panel-assessed

  • reading · 2h 30m · tier 0 self-marked

    karpathy/minGPT

  • retention · 15m · tier 1 machine-verified

M6 · Publish, and defend

0/6 · 12h

  • project · 3h · tier 3 artifact

  • project · 3h · tier 3 artifact

  • lecture · 2h 45m · tier 0 self-marked

    CMU Advanced NLP Fall 2024 — Graham Neubig

  • assignment · 2h · tier 2 panel-assessed

  • defense · 1h · tier 4 defended

  • retention · 15m · tier 1 machine-verified