Inference.
Memory, latency and the cost of useful work.
Inference · 4 min read ·
Why a fast decoder can still leave you waiting, and the four measurements that make a comparison useful.
↗Inference · 3 min read ·
Idle hours, output tokens and rejected answers belong in the same budget. Start with your own assumptions.
↗