Large language models like DeepSeek-R1 need a large amount of parameters to perform complex tasks, creating the need for a distributed hardware system. Such a system requires distributed inference to optimize performance. Enter llm-d, an open source framework for distributed LLM inference. Join Robert Shaw, Red Hat’s Director of Engineering for AI, as he dives into llm-d’s well-lit paths approach—a straightforward and efficient way to manage LLM inference distribution and meet the demands of large-scale AI workloads. 00:00 Introduction 00:43 The Enterprise Generative AI Inference Platform Stack 04:36 The llm-d Architecture Overview 08:39 Introducing Well-Lit Paths 09:54 Intelligent Inference Scheduling: Prefix-Aware & Load-Aware Routing 14:14 P/D Disaggregation: Splitting Prefill and Decode for Efficiency 17:45 Efficient KV Cache Transfer in VLLM with NIXL and RDMA 18:36 Flexible, Configurable Deployments with Heterogeneous Tensor Parallelism 19:32 KV Cache Management 22:58 Mixture of Experts Overview and Model Deployment 24:26 Wide Expert Parallelism (WideEP) Optimizations for MoE Scaling 27:45 Performance Summary and Closing 🔗 Read more about distributed inference: https://www.redhat.com/en/topics/ai/what-is-distributed-inference #AI #RedHat #Kubernetes #llmd

 391      15