We're back with another episode of Pop Goes the Stack and the newest bright idea in #AI: don’t pay humans to evaluate model outputs, let another model do it. This is the “LLM-as-a-judge” craze. Models not just spitting answers but grading them too, like a student slipping themselves the answer key. It sounds efficient, until you realize you’ve built the academic equivalent of letting someone’s cousin sit on their jury. The problem is called preference leakage. Li et al. nailed it in their paper “Preference Leakage: A Contamination Problem in LLM-as-a-Judge.” They found that when a model judges an output that looks like itself—same architecture, same training lineage, or same family—it tends to give a higher score. Not because the output is objectively better, but because it “feels familiar.” That’s not evaluation, that’s model nepotism. Watch as #F5's Lori MacVittie, Joel Moses, and Ken Arora explore the concept of preference leakage in AI judgement systems. Tune in to understand the risks, the impact on the enterprise, and actionable strategies to improve model fairness, security, and reliability. Chapters: 00:00 Welcome to Pop Goes the Stack 00:34 LLM-as-a-judge and preference leakage 01:26 Why are judgement systems necessary? 03:05 Bias in judgment systems and model families 06:56 Is AI bias a problem or a feature? 08:13 Preference leakage vs data leakage 10:00 Impact of synthetic data: Generation and model training 12:07 Evaluating models: Red teaming and diversity 14:44 What can we do about preference leakage? 17:21 SLMs, preference leakage, and protecting data transactions 19:21 Correctness' implication on the enterprise: GenAI security and reliability 20:45 Key takeaways: Measure carefully and diversify Learn how you can stay ahead of the curve and keep your stack whole with additional insights on app security, multicloud, AI, and emerging tech: https://go.f5.net/i4g89z0k Read the paper, Preference Leakage: A contamination Problem in LLM-as-a-judge: https://go.f5.net/eacp2top

 117      3