When an outage impacted two LLM teams, one team spent 3 hours piecing together what happened. The other team confirmed "no action needed" within minutes. Same company. Same incident. The difference was Production Readiness and Datadog was how we proved it. As the number of teams adopting LLMs grew 5x in six months, Timee created a Production Readiness Checklist for LLM Applications and built an AI Gateway platform that includes Datadog and delivers Production Readiness from day 1. This platform has supported their rapid growth by making new team onboarding quick and instrumentation automatic. In this session, Tomoyuki Saito (MLOps Engineer) shares Timee’s Production Readiness Checklist and the LLM-specific dimensions they measure. He analyzes the two different outcomes from the outage, highlighting what LLM Traces and APM revealed about why one feature failed and the other didn't. He explains the Gateway architecture that makes Datadog observability automatic for every new team—enabling a self-service model for onboarding. This is not a success story. It’s a story of what broke, what was learned, and what the Timee team—and you—can do to avoid failure.

 136      1