Training and inference get lumped together in casual AI conversations, but they behave differently enough that the distinction matters for cost, architecture, and security. In this episode of #F5's Pop Goes the Stack, Lori MacVittie, Joel Moses, Ken Arora, and Kevin Baughman, who leads F5’s AI Center of Excellence, unpack what’s truly different, what’s the same, and where people get misled. Joel makes the “math is the same” case: both phases run similar computations, but training must retain intermediate activations for backpropagation, while inference can discard them. Ken and Kevin pull the conversation back to practical differences: training is about baking knowledge into the model, while inference is about using a frozen model and shaping behavior with context, retrieval, and few-shot examples. The weights don’t change during inference; the input does, which is why it can feel like “learning” without actually being permanent. That distinction becomes a security and governance lever. If you don’t want sensitive or proprietary data baked into a model, you avoid training on it and instead keep it in a controlled knowledge base (RAG or similar) that can be updated, removed, or scoped per tenant. Meanwhile, training pipelines emphasize massive data ingestion and throughput, and inference emphasizes responsiveness, session context, and efficient serving at scale. The practical takeaway is to stop treating “#AI workloads” as one thing. Training and inference require different pipeline designs, different tradeoffs in memory and bandwidth, and different approaches to data control. Pick your phase, understand the constraints, and build for it intentionally. Chapters: 00:00 Welcome to Pop Goes the Stack 00:44 Inference vs training: Same math, very different realities 02:12 Why it matters: Cost, infra, and “what the model remembers” 02:30 Joel’s take: Training = saving activations for backprop 03:52 Ken's pushback: “Differences matter” (Hardware, workflows, and input) 05:56 Kevin’s view: Inference/training are blurring via context 06:49 “Frozen weights” vs ephemeral learning (and why it feels like learning) 07:58 Where context lives: Memory (and why it’s not permanent) 09:59 Fine-tuning vs context: When/why people stopped tuning 10:49 Do weights change during inference? (No—context influences) 13:02 Training pipeline: Tons of data + curated inputs = security control 14:47 RAG/KB approach: Tenant separation + removable knowledge 16:19 Why ChatGPT “remembers”: Systems around the model, not weights 17:19 Hardware split: Training bandwidth/HBM vs inference efficiency 19:30 Key takeaway: Pick the right components for cost and efficiency Learn how you can stay ahead of the curve and keep your stack whole with additional insights on app security, multicloud, AI, and emerging tech: https://go.f5.net/ge6n8x6w More about F5: https://go.f5.net/eeo4ssnw Read our blog: https://go.f5.net/oxsejyio Follow us on LinkedIn: https://go.f5.net/5pcqk118

 56      1