Round robin isn’t just “not ideal” for LLM inference. According to recent scheduling research, it’s actively harmful, because it treats inference like stateless, interchangeable API traffic when it’s anything but. In this episode of #F5's Pop Goes the Stack, Lori MacVittie is joined by F5's Josh Mendoza, Principal Solutions Engineer, to break down why classic load-balancing assumptions fail under #LLM workloads, and what to think about instead. Josh walks through the evolution from early “spray and pray” distribution to smarter approaches that account for server load, workload type, and state. That history matters because #AI introduces the same challenge at a new intensity: inference is a heavy compute-and-memory math pipeline, and conversations accumulate state. Once context and KV cache are involved, moving a request to a different server isn’t a clean failover, it’s a forced cache miss and a recomputation penalty that shows up as slower time-to-first-token and higher cost. They connect this back to patterns teams already understand: VM migration, session persistence, and why “just move it” has always been expensive when the working set is large. LLMs raise the bar because users won’t tolerate latency, and the payload you’d need to move grows as the interaction continues. You also can’t ignore the request itself, since “summarize this” and “write a full analysis” have very different compute profiles, even if they hit the same endpoint. The main takeaway is simple: don’t panic, but stop treating inference like generic API traffic. Effective LLM scheduling is closer to dispatching the right resources to the right job, with awareness of state, model placement, cache locality, and the true cost of moving work. The tools exist, but the mental model has to change first. Chapters: 00:00 Welcome to Pop Goes the Stack 00:23 Round robin is actively harmful for LLM inference 02:38 Why classic load balancing evolved beyond “spray and pray” 04:04 Not all requests are equal: Workload-aware distribution 05:04 Microservices forced smarter traffic steering 07:00 Load balancing → traffic management → dispatching 07:37 Dispatcher analogy: Send the right resources, not “any” 10:27 LLM routing must consider the request (not just servers) 12:35 AI isn’t a “box”: It’s a massive math + memory workload 13:54 VMotion déjà vu: Migrating state is slow, but latency demands aren’t 16:16 KV cache = SSL persistence: Keep sessions where the work is 17:38 Key takeaway: Don’t panic—learn the workload, then route smart Learn how you can stay ahead of the curve and keep your stack whole with additional insights on app security, multicloud, AI, and emerging tech: https://go.f5.net/mci69jjc More about F5: https://go.f5.net/pcwsg159 Read our blog: https://go.f5.net/h5r7b7yh Follow us on LinkedIn: https://go.f5.net/kxo8s9n9

 50      2