Continual Learning Is the Next Bottleneck | Rohan Anil (Core Automation )
Continual Learning Is the Next Bottleneck | Rohan Anil (Core Automation )  
Podcast: The Information Bottleneck
Published On: Thu Sep 10 2026
Description: Rohan Anil spent eleven and a half years at Google, where he went from writing memory allocators to large-scale linear solvers, then optimization at Google Brain, where he co-developed distributed Shampoo and led optimization for PaLM and Gemini pre-training, including the work that produced Gemini Flash. He then joined Anthropic's pre-training team, and left before the IPO to co-found Core Automation with Jerry Tworek (ex-VP of Research at OpenAI). We talk with him about how Brain worked at its peak, why he left two of the world's best labs, and what he thinks is missing from today's models.Rohan's view is that pre-training and RL were split by organizational convenience rather than by science. Pre-training builds a prior, and RL sharpens it to the tasks we care about, and neither gives a model a way to absorb new data or learn from its own experience once it is deployed. Post-training more every day plateaus, on-policy distillation plateaus, and in-context learning only goes as far as the context does. He argues the next architecture needs better ways to fold in new knowledge at inference time, and that this is a fundamental optimization question rather than a harness-engineering one.We also get into why coding agents still fail on low-level systems work, his take on Muon, why second-order methods matter once you leave the noise-dominated regime, and why nobody can yet use a few million GPUs for a single training run.Timeline00:00 Intro01:09 From computer vision to Google systems engineering02:37 Large-scale linear solvers and sparse features06:01 Getting into optimization: SDCA and Yonghui Wu's team07:29 Joining the Shampoo crew09:35 The Google Brain ethos, and why 2017 to 2019 was special14:23 Is open research going to keep winning?15:53 Frontier models are only as good as the prior you give them17:30 Missing the language model wave, then Common Crawl and online distillation18:31 Paternity leave, DALL-E Mini, and the 14 days that became two years20:32 PaLM, Gemini pre-training, and Gemini Flash23:59 The Shampoo origin story: Tomer Koren's two-week proof26:54 Why leave Google for Anthropic30:20 Why leave Anthropic for a startup31:33 Meeting Jerry Tworek at Dolores Park34:00 What Core Automation is building36:17 Continual learning and the pre-training vs RL split39:11 Why coding agents fail at kernels and low-level pipelines42:00 The QR factorization kernel competition and reward hacking44:57 Numerics, verification, and hardware that keeps changing46:07 Are LLMs creative, or just good at search?49:50 Getting models to extrapolate instead of interpolate52:03 Why did we ever call it pre-training?55:02 What RL is really learning56:27 Competing with the big labs with fewer people58:31 Will kernel generation keep old GPUs alive? Amdahl's law1:01:58 Open source plans1:02:55 Audience question: agentic optimizers1:04:53 Audience question: Muon, Shampoo, and the future of second-order methods1:08:58 Hiring at Core Automationkey topicsJourney from Google Brain to startupEvolution of AI research and optimizationPre-training and reinforcement learningKernel optimization and system efficiencyOpen source AI and collaborative researchChallenges in AI creativity and explorationFuture directions in continual learning and model scalingMusic"Kid Kodi" - Blue Dot Sessions - via Free Music Archive - CC BY-NC 4.0