← Skill graph
Latency budgets and streaming
Part of LLM inference cost and latency optimisation
Needs first
Helps first
Nothing recommended beyond what is required.
Unlocks
Where it is assessed
8 assessed items, 0 done.
- AIE-105 M2.3 Measure the split · tier 1
- AIE-105 M2.4 The latency budget · tier 2
- AIE-105 M2.5 Retention: prefill, decode and latency structure · tier 1
- AIE-105 M6.3 Time to first token, on the text path · tier 1
- AIE-105 M6.4 Perceived versus measured · tier 2
- AIE-105 M6.5 Retention: streaming, speculation and perceived speed · tier 1
- AIE-105 M8.1 Ship a measured improvement · tier 3
- AIE-105 M8.4 Oral defense · tier 4
Verified sources
- Streaming messages
- Making Deep Learning Go Brrrr From First Principles
- Transformer Inference Arithmetic
- MIT 6.5940 EfficientML.ai · MIT HAN Lab
- Efficiently Scaling Transformer Inference
- Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads
- Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve
- Fast Inference from Transformers via Speculative Decoding