Free course
Inference Optimization
Training is a one time capital cost. Inference is the bill that arrives every month. This course is about the engineering that moves it.
- Free on YouTube
- Five to ten minute chapters
What this is
Latency is a product decision.
Training is a one time capital cost. Inference is the recurring one, and it is where the engineering leverage now sits.
The spine of the whole course is one number: arithmetic intensity. Every technique either raises it, lowers the bytes it divides by, or hides latency behind it.
Episodes run five to ten minutes. Every module closes with a CLINIC episode, where the symptom comes first and the theory is used to find the cause.
Who it is for
- Beginners starting out
- Curious engineers
- People interviewing for inference and ML systems roles
- Professionals reviewing or deep diving
Why it matters
One slide justifies the whole course.
One slide justifies the entire course. DeepSeek published 24 hours of its own production economics:
The shape of it
Every technique is one of four levers.
Whatever the trick is called, it does one of these four things. Once you can name which, you can reason about whether it will help you.
Move fewer bytes
Cut the denominator of arithmetic intensity.
Do fewer FLOPs
Cut real work.
Share work across requests
Amortise a fixed cost.
Overlap
Hide one resource behind another.
The chapters
Every chapter so far.
Chapter 00: What is Inference Optimization
Chapter 01: Where the time actually goes?
Chapter 02: Full request path from HTTP to token
Chapter 03: Prefill and Decode: two stages of an inference request
Chapter 04: Understanding inference metrics: TTFT, TPOT, ITL, E2EL, throughput and goodput
Chapter 05: Throughput, goodput, and the latency you refused to exceed
Chapter 06: Draw your curve and pick your operating point
Finished a chapter? Go deeper.
The newsletter goes further on the same material, the agents course builds on it, and the podcast argues the business side.
Keep exploring