Why Inference Matters 12x More Than Training | YC Paper Club

Why Inference Matters More Than Training [00:01:05]

• The ratio of inference flops to training flops is critical for understanding AI's future computational needs.
• DeepSeq data suggests inference requires 6x more flops than training, and potentially 12x more over a model's useful life.
• As AI adoption grows, the importance of efficient inference will only increase.

  "And so it's probably much more than that. But even those numbers is about 6x more flops on inference than train for that wait file per year."

The Cost of Inference: COGS and Margins [00:01:55]

• For model servers, the cost of goods sold (COGS) is a critical factor.
• Companies with highly optimized inference can achieve impressive profit margins.
• DeepSeq's reported 80% margins highlight the financial benefits of inference innovation.

  "Well, Francois, why is it so complicated? It's just like model dot forward. It's not that hard."

Understanding the Inference Process: Pre-fill and Decode [00:02:45]

• Inference involves processing a prompt (pre-fill) and generating output tokens (decode).
• Pre-fill involves processing the initial prompt, with opportunities for caching to avoid redundant computation.
• Decode is an iterative loop where the model generates tokens one by one.

  "You get served some prompt, some set of tokens, token IDs, and then you have to call pre-fill."

Key Performance Indicators (KPIs) in Inference [00:04:00]

• Time to first token measures the duration of the pre-fill stage.
• Time per output token (or decode step) measures the speed of generating subsequent tokens.
• Throughput and goodput quantify the number of tokens produced per unit of time and service level.

  "And the terms of art, the things that the KPIs that Dan Fu and others are constantly trying to optimize is time to first token."

Throughput Discrepancies: Pre-fill vs. Decode [00:05:00]

• Pre-fill throughput can be extremely high, especially with cache hits (e.g., 1.5 million tokens/sec).
• Decode throughput is significantly lower (e.g., 14,000 tokens/sec), highlighting it as a major bottleneck.
• This disparity drives innovation in speeding up the decode stage.

  "And you can see the throughput for decode versus pre-fill, 1.5 million to 14,000."

The KV Cache: A Critical Component [00:06:00]

• The KV cache stores intermediate computations to speed up subsequent token generation.
• Retaining KV cache data for extended periods (e.g., 72 hours) significantly improves cache hit rates.
• Effective KV cache management is crucial for high-performance inference.

  "And then this is actually the most interesting one that people clued in on, which is the KV cache retention policy."

Arithmetic Intensity: A Measure of Compute Efficiency [00:07:00]

• Arithmetic intensity measures the ratio of computation (flops) to memory transfers.
• High arithmetic intensity is desirable, meaning more computation is performed per byte of data moved.
• LLMs often have poor arithmetic intensity compared to other domains like RNNs or diffusion models.

  "As I go from DRAM to SRAM, if I transfer, you know, four bytes, how much flops or how much compute am I doing on those four bytes?"

Innovations in Model Architecture for Inference [00:09:00]

• Model innovations aim to reduce computational load during inference.
• Techniques include Mixture of Experts (MoE), State Space Models (SSMs), and multi-headed latent attention.
• Quantization and distillation also reduce model size and computational requirements.

  "Some people might, you know, bicker about which one, which, where buckets should live. But the way I think about it is there's model innovations where you do this thing called MOE where you if you have K experts, then you can amortize, reduce the amount of flops at test time over all of the experts by one over K essentially."

Kernel and Runtime Innovations [00:10:30]

• Kernel and runtime optimizations focus on improving the efficiency of the underlying code executed on hardware.
• Examples include FlashAttention, PagedAttention, and RadixAttention.
• These often draw inspiration from operating system and database principles like caching and prefix trees.

  "They're kind of a lot of these are just taking from classic OS and database theory."

Harness Innovations for Inference Efficiency [00:11:30]

• Harness innovations involve system-level strategies to manage and route inference requests.
• Techniques include intelligent routing to avoid unnecessary computation on large models.
• Recursive language models and compaction strategies further optimize resource utilization.

  "So of decoding, we have a draft model that is validated. And then my favourite one is the recursive language model where never put the full context into the KV cache."

The Rise of Open Source AI and Together AI [00:15:00]

• Open source AI has seen explosive growth, with platforms like Together AI serving massive amounts of tokens.
• Together AI is building an AI-native cloud focused on pre-training, fine-tuning, and inference.
• Open models are becoming increasingly competitive in quality and significantly cheaper than closed-source alternatives.

  "By earlier this year, more than half the tokens through OpenRouter were through OpenModels."

Request Flow Through the Inference System [00:19:00]

• An inference request follows a path from API to data center, involving routing, tokenization, batching, pre-fill, decode, and token translation.
• Routing must consider both load balancing and KV cache residency for optimal performance.
• Tokenization involves converting human language and tool calls into a format the model can process.

  "So imagine you send in a request to some REST API. It goes through the normal parts of the internet through Cloudflare and DNS and all that stuff."

Pre-fill vs. Decode: Different Workloads, Different Optimizations [00:22:00]

• Pre-fill (prompt processing) is compute-bound with high model flop utilization (MFU).
• Decode (token generation) is memory-bound, requiring loading model weights for each token.
• Separating and optimizing these two stages differently is key to efficient inference.

  "That's a very compute bound operation. So it's going to be very heavy on your GPU and it's going to have some of these words that Francois mentioned, MFU."

Optimizing Inference: Kernels, Parallelism, and Quantization [00:24:30]

• Kernels are GPU programs that can be highly optimized for specific operations.
• Parallelism strategies (tensor, pipeline, data) are needed to handle models too large for a single GPU.
• Quantization reduces the precision of model weights, decreasing memory footprint and increasing speed.

  "So in no particular order, you have kernels. Kernel is just a fancy word for a GPU program."

The Role of Compilers in Inference Optimization [00:31:00]

• Compilers play a crucial role in translating models into efficient code for diverse hardware.
• Luminal uses a quality saturation approach, searching for optimal program transformations rather than relying on manual heuristics.
• This allows for efficient compilation across various hardware, including GPUs, TPUs, and ASICs.

  "We really see compilers as being a very critical part of the stack and kind of defining your conversion function between GPU flops or GPU hours and tokens."

Challenges in Serving Open Models: Quality and Consistency [00:35:00]

• The gap in performance between the best and worst providers for open models can be larger than the gap between open and closed models.
• Inconsistent quality can arise from differences in model precision, tool calling implementation, and sampling techniques.
• Agentic and coding workloads exacerbate quality issues due to compounding errors.

  "And the gap between the worst provider for a model and the best provider is actually bigger than the gap between closed and open models."

Measuring and Ensuring Open Model Quality [00:37:00]

• Evaluating open model quality involves benchmarks, tool verifiers, distribution tests, and log probability analysis.
• Monitoring metrics like tool call success rates and output length shapes can reveal underlying issues.
• The industry is moving towards more standardized inference stacks and quantized models released by labs to ensure consistency.

  "Well, the first thing you can obviously do is run evals, which is kind of like the brute force way and, um, probably the most expensive way to do it."

The Future of Local and Cloud Inference [00:43:00]

• The rise of powerful local models is leading to a hybrid approach combining local and cloud inference.
• This introduces new routing challenges, similar to those faced in cloud-based inference.
• Ensuring consistent quality across diverse inference engines and hardware remains a key challenge.

  "I think the future of Ollama, I think, um, is probably people running a mix of local and cloud models."

SGLang: Efficient Inference for Agentic Workloads [00:45:00]

• SGLang focuses on efficient inference for agentic workloads, particularly managing KV cache and memory.
• It employs a radix tree for memory management, allowing for shared prefixes and efficient eviction policies.
• Overlapping scheduling and unified memory pools are used to maximize GPU utilization.

  "So the story begins when the agent workload. Actually, the story begins in the late 2023."

Advancements in Memory Management for LLMs [00:47:00]

• Unified memory pools and compaction techniques allow for flexible allocation and re-organization of GPU memory.
• This addresses the challenge of managing different KV cache types (full token, sliding window, recurrent states) efficiently.
• The goal is to maximize cache hit rates and minimize memory waste across diverse workloads.

  "So, so previously we used different, Redix tree, like, which was implemented, standalone, uh, uh, without consideration of each different, low level, memory pool."

The Evolution of Transformer Architectures and Inference [00:49:00]

• Traditional transformers face limitations with long contexts, driving innovation in architectures like sliding windows and recurrent states.
• Efficiently sharing memory and computation across these different architectural components is a significant challenge.
• Unified memory management systems are being developed to handle these complexities.

  "And also, there is a current, recurrent states like a Mamba state, KDA states. They were, they were just putting, the information into the, fixed the shape of the state, like, token by token."

The Role of Compilers in Hardware-Software Co-design [00:51:00]

• Compilers are essential for bridging the gap between complex AI models and diverse hardware.
• Luminal's approach uses quality saturation and a simplified intermediate representation (IR) to target various hardware efficiently.
• This reduces the need for massive compiler teams and allows for rapid adaptation to new hardware.

  "And so we take a matmul, for instance. We don't even have a matmul operation. We take a matmul and we decompose that into a broadcasted multiply and then a sum reduction."

The Bitter Lesson and Search-Based Optimization [00:53:00]

• The "bitter lesson" suggests that search-based approaches, rather than human-designed heuristics, are more effective for optimization.
• Luminal uses extensive compile-time search to find optimal program transformations, minimizing the need for manual rule-writing.
• This allows for the discovery of performance improvements that might not be obvious to human engineers.

  "And so what you often see from these companies is they hire compiler teams and their compiler team headcount, uh, scales exponentially with how many different applications they want to target."

Debugging and Scaling Compiler Optimizations [00:55:00]

• Debugging compiler optimizations involves tracking metadata through the compilation process and using proof-generating rewrite rules.
• Scaling search involves using algorithms like genetic algorithms and pruning the search space to find near-optimal solutions efficiently.
• The focus is on achieving high performance at runtime, even if compile times are longer.

  "And what a quality saturation does, what Aglog specifically does for us quite nicely, is it lets us say, okay, we got an output program out and this program is wrong."

The Synergy of Compilers and Inference Engines [00:57:00]

• Compilers like Luminal focus on optimizing the core forward pass of models.
• They integrate with specialized inference engines (like vLLM and SGLang) for features like speculative decoding and advanced scheduling.
• This division of labor allows each component to excel in its area, leading to overall improved inference performance.

  "So, generally, no. We try to basically just handle the forward pass. So, you know, you specify like a forward pass graph and we compile that."