← Skill graph
Implementing a transformer from scratch
Part of Transformer architecture
Needs first
Helps first
Nothing recommended beyond what is required.
Unlocks
Nothing depends on this one.
Where it is assessed
11 assessed items, 0 done.
- AIE-204 M2.4 Multi-head self-attention, from scratch · tier 1
- AIE-204 M3.3 Build the model · tier 1
- AIE-204 M3.4 Sampling is a design decision · tier 2
- AIE-204 M3.6 Retention: the decoder stack, component by component · tier 1
- AIE-204 M4.3 Three modern components, measured one at a time · tier 2
- AIE-204 M4.6 A KV cache, and what it buys · tier 1
- AIE-204 M5.3 The reproduction: run it · tier 2
- AIE-204 M5.4 The reproduction: account for the gap · tier 2
- AIE-204 M5.6 Retention: reproduction and its failure modes · tier 1
- AIE-204 M6.1 The annotated implementation, published · tier 3
- AIE-204 M6.5 Oral defense · tier 4
Verified sources
- The Annotated Transformer
- Build a Large Language Model (From Scratch)
- Let's build GPT: from scratch, in code, spelled out · Andrej Karpathy
- Let's reproduce GPT-2 (124M) · Andrej Karpathy
- FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
- karpathy/llm.c
- karpathy/minGPT
- karpathy/nanoGPT
- Stanford CS336 Language Modeling from Scratch I 2025 · Stanford Online