World Models | John Langford (Microsoft AI Labs)
World Models | John Langford (Microsoft AI Labs)  
Podcast: The Information Bottleneck
Published On: Sat Sep 05 2026
Description: John Langford, one of the heads of Microsoft's AI Labs, the creator of Vowpal Wabbit, and a co-inventor of CAPTCHA, joins us to talk about world models. Transformers need orders of magnitude more data than humans to learn the same thing, and John argues a compact, implicit world model is how you close that gap. He explains why he's skeptical of JEPA-style objectives, why a transformer's KV cache is the Ptolemaic epicycle model of belief states, and what his Next Latent work does differently.We also get into whether research still matters in the age of scale; open versus closed models; agent-driven research after running 2,000 pre-training experiments in 90 days; the origin story of CAPTCHA; and why Muon and orthonormal optimizers actually work.Topics:Implicit vs. explicit world models, and the case against JEPA-style objectivesCompact belief states: why compression beats a growing KV cacheDoes research still matter in the age of scale? The Kimi K3 argumentAgent-driven research: 2,000 pre-training experiments in 90 daysThe invention of CAPTCHAOptimizers from SGD and Vowpal Wabbit to MuonChapters00:00 Why world models: the sample-complexity gap09:48 The case against JEPA; a transformer-style implicit world model15:52 Compact belief states: epicycles vs. heliocentrism23:41 Does research still matter? The Kimi K3 argument27:35 Open vs. closed models35:57 Recursive self-improvement and agent-driven research42:30 2,000 pre-training experiments in 90 days; weak baselines and reproducibility54:54 The invention of CAPTCHA1:00:53 Optimizers: from Vowpal Wabbit to MuonMusic"Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0