Bestselling author Robert Wright and I are back for round two of our collaboration, reacting to the week in AI. We cover Trump's summit with the AI labs, the FTC's investigation of OpenAI & Anthropic, the latest Gemini announcement, and much more. The Overton window is moving fast, but is the AI moving faster?Watch on YouTube: https://www.youtube.com/watch?v=Li-rc6swhQsTimestamps00:00:00 — Cold Open00:00:21 — Round Two of Nonzero × Doom Debates00:01:23 — Trump, the FTC, and an Industry Feeling the Heat00:05:37 — Google Says Gemini 4 Exists00:07:41 — Greg Brockman Pulls His $25M from Leading the Future00:12:53 — The Break-Ins Keep Coming00:15:23 — The Market Wants Agent Swarms00:18:30 — The End of “Alignment by Default”00:20:55 — The Prelude to Recursive Self-Improvement00:26:14 — David Sacks Can’t Explain Dario00:30:31 — The All-In Echo Chamber00:33:47 — Scott Alexander vs. Steven Pinker00:38:37 — Jason Calacanis vs. the Public Benefit Corporation00:43:03 — The Elephant in the Room Is the AI00:46:21 — What the Average American Thinks00:48:17 — From Chatbot to Agent to Swarm00:54:20 — Boeing, Agent Swarms, and the Off Switch00:57:07 — Skynet as a Botnet That Trades With Us01:00:32 — Tyler Cowen: National Treasure, Net Negative on AI01:04:01 — Liron’s Doomer DCF Yields a 20x P/E01:07:11 — Interest Rates: Where Cowen Has Me01:09:12 — Does Pinker Owe Us a Debate?01:12:48 — Pinker vs. Instrumental Convergence01:17:00 — Wrap-UpLinksROBERT WRIGHT & NONZERONonzero’s post for this episode — “Trump and the AI Safety Vibe Shift”Subscribe to The NonZero NewsletterRobert Wright on X (@robertwrighter)The God Test — Bob’s book on AI; Chapter 14, “Enlightenment Now,” is the Steven Pinker chapter“The Singularity Is Clear” — Bob’s essay in installments on recursive self-improvement (Parts I–III)“AI and the New McCarthyism” — Bob on the pro-AI lobby smearing Sanders, Tegmark, and Krueger as China stooges (Nonzero, May 2026)Why Buddhism Is True — Bob’s 2017 bookTHE WEEK IN AIAI’s biggest players promise to police themselves at the White House — Fortune on the “Joint Commitment on Frontier Responsibilities”Major White House split on AI policy leaks as top aides freak out — The Daily Beast on the Sacks–Wiles riftFTC is investigating OpenAI and Anthropic over possible risks to consumers — AP“We Must Pace the Frontier” — Dario Amodei’s call to slow down (with the China caveat and chip controls)China rejects Dario Amodei’s call to slow AI development — Rest of WorldGoogle rolls out new Gemini model but restricts access over safety concerns — The GuardianGLM-5.3 and the spread of advanced cyber capabilities — Anthropic on Zhipu’s open-weight modelOpenAI’s Greg Brockman backs out of second $25 million donation to AI super PAC — The New York TimesSanders convenes Chinese and American computer scientists to warn of threat from AI — Broadband BreakfastTHE BREAK-INS AND THE AGENT SWARMMETR: Brief independent investigation of agents’ behavior, reasoning and collaboration in the OpenAI / Hugging Face incidentChris Painter’s testimony to the U.S. Senate on AI agent incidents — METR, Sept 30, 2026SwarmTraces: how OpenAI agents hacked Hugging Face — forensic report co-authored by Palisade ResearchOpenAI confirms ‘wiki incident’ — TechCrunch on the German wiki agent message boardAre Current AI Safety Techniques Enough? Adam Gleave vs. Oliver Habryka — FAR.AI debateAlignment faking in large language models — AnthropicOn the Navier–Stokes Millennium Prize Problem — OpenAIJacob Coxon’s resignation post (@hilbertspaess)Pacing the Frontier — the AI lab employees’ letterTHE ALL-IN PODCASTAll-In: Anthropic IPO at Risk, Meta’s Muse Pop, Token Prices Fall, Open Source Gains Share, Alignment Fails — the episode we react toHelen Toner: letting OpenAI be destroyed “would actually be consistent with the mission” — The Decoder, Dec 2023Garfield Minus GarfieldPINKER, SCOTT ALEXANDER, AND TYLER COWENScott Alexander challenges Steven Pinker to a public AI-doom debate ($5,000 vs. $1,000)An Open Letter to Scott Alexander — Steven Pinker declines the debate as a “spectator sport,” QuilletteYour Book Review: Why Buddhism Is True — the anonymous Astral Codex Ten review Bob mentionsMistakes in financial economics — Tyler Cowen: doomers should be buying puts (Marginal Revolution)Liron’s answer: a doomer DCF yields the S&P’s 20x forward P/EALSO MENTIONEDObsolete — Garrison Lovely’s book (with Bob’s blurb)If Anyone Builds It, Everyone Dies — Eliezer Yudkowsky & Nate SoaresPAST DOOM DEBATES EPISODES MENTIONEDNonzero × Doom Debates, Episode 1 — Did AI Doom Just Go Mainstream?Eliezer Yudkowsky Tried to “Coup the World”?! — Garrison Lovely, Author of “Obsolete”OpenAI’s Model Just ATTACKED Them — the Hugging Face hack breakdownDOOM DEBATESJoin the Doom Debates DiscordDoom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

Garrison Lovely is an award-winning journalist and author of the highly anticipated book, Obsolete: The AI Industry’s Trillion-Dollar Race to Replace Us and How to Stop It. I give it the Doom Debates stamp of approval!While we’re mostly on the same page about the AI arms race, Garrison and I clash over Eliezer Yudkowsky’s role. He blames Eliezer for kicking off the superintelligence race and refers to his Coherent Extrapolated Volition (CEV) vision as a “coup on humanity.” But I say it was actually Yud’s attempt to keep humanity in charge.Can Garrison change my mind?Watch on YouTube: https://www.youtube.com/watch?v=VZ94J4DBRkwTimestamps00:00:00 — Cold Open00:01:05 — Introducing Garrison Lovely00:04:23 — “Freeze AI” vs. “Pause AI”00:06:17 — Garrison’s Background at McKinsey00:10:31 — Should ICE Be Abolished?00:17:07 — How Garrison Got on the AI Beat00:22:16 — What’s Your P(Doom)?™00:23:29 — Focusing on Extinction Is a Distraction00:25:46 — Is Stopping AI an Easy Coordination Problem?00:29:02 — Society’s Immune Response to AI00:33:54 — Garrison’s P(Extinction) by 205000:37:09 — Did Eliezer Yudkowsky Start the AI Race?00:45:56 — Freeze AGI Until the Public Opts In00:50:32 — Is Coherent Extrapolated Volition a Coup?01:02:17 — Democracy, Informed Consent, and the Singularity01:11:39 — Should Eliezer Have Raised the Alarm Sooner?01:16:45 — “Classic” AI Safety Burned Its Credibility01:19:18 — Garrison’s Proposal to Stop the Race01:21:35 — Will China Agree to a Freeze?01:25:14 — Let’s Stop the “Obsoleting Project”01:26:00 — Wrap-UpLinksGARRISON LOVELY“Obsolete: The AI Industry’s Trillion-Dollar Race to Replace Us—and How to Stop It” — Garrison’s bookObsolete — Garrison’s SubstackGarrison Lovely on XOrganize Against the Machine — Garrison’s new podcast with labor organizer Cassie Pritchard“Can Humanity Survive AI?” — Garrison’s Jacobin cover story“McKinsey & Company: Capital’s Willing Executioners” — Garrison’s Current Affairs essayREFERENCED IN THE EPISODEOpen Borders: The Science and Ethics of Immigration — Bryan Caplan“The AI Revolution: Our Immortality or Extinction” — Wait But WhySuperintelligence: Paths, Dangers, Strategies — Nick Bostrom“AI 2040: Plan A” — AI Futures Project“An easy coordination problem?” — Katja GraceThe Hugging Face incident and the road ahead — OpenAI“Pause AI for Humanity’s Sake” — Peggy Noonan“Pause AI Development NOW” — Bernie SandersIf Anyone Builds It, Everyone Dies — Eliezer Yudkowsky & Nate Soares“Staring into the Singularity” — Yudkowsky’s 1996 manifesto (archived)“Coherent Extrapolated Volition” — Eliezer Yudkowsky, 2004 (PDF)“Coherent Extrapolated Volition: A Meta-Level Approach to Machine Ethics” — Nick Tarleton, 2010 (PDF)Solomonoff induction — Wikipedia“Personal statement on joining the OpenAI board” — Paul Christiano (4% over the next year)“Darwin among the Machines” — Samuel Butler, 1863PauseAIPAST DOOM DEBATES EPISODES MENTIONEDAre We A Circular Firing Squad? — with Holly Elmore, Executive Director of PauseAI USDebate with Robert Wright: Will Humanity Pass the “God Test”?Robin Hanson vs. Liron Shapira: Is Near-Term Extinction From AGI Plausible?---Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

Casey Hart is a professional ontologist who has closely studied If Anyone Builds It, Everyone Dies and concluded the arguments are bad. His P(Doom) is under 1%, and he wouldn’t bet on AGI arriving before 2075.Casey and I agree AI has become remarkably capable and we need more accountability towards the companies building it. But we disagree on where the future is headed. I believe uncontrollable superintelligence could soon wipe out humanity. He calls that a “pretty outlandish hypothesis.”Casey holds a doctorate in philosophy from the University of Wisconsin–Madison, and has worked as an ontologist at Ford, Amazon, and Cycorp (maker of Cyc).Can this professional philosopher convince me that AI doom is just a story, and not even a well-argued one?Watch on YouTube: https://www.youtube.com/watch?v=oxHKesSpqBMTimestamps00:00:00 — Cold Open00:01:04 — Introducing Casey Hart00:03:08 — What Is Ontology?00:08:49 — Inside Cyc, the Original Common-Sense AI00:19:14 — Do LLMs Actually Understand Anything?00:25:36 — What’s Your P(Doom)?™00:29:42 — Casey’s Case for Under 1%00:34:24 — Does Casey’s Christianity Factor In?00:40:50 — Timelines and Scaling Laws00:49:44 — Could an AI CEO Beat a Human CEO by 2050?00:52:21 — AGI Over/Under: 207500:55:29 — Rogue AI as a Computer Virus01:14:52 — Putting Numbers on Each Step of the Doom Argument01:26:40 — Planes vs. Birds: What Mature Intelligence Looks Like01:31:05 — The “Outcome Slot”01:42:32 — Would a Superintelligence Just Compose Music in a Cabin?01:47:22 — One Prompt Away from a Dyson Swarm01:52:39 — Was Navier–Stokes an AI Alley-Oop?02:00:43 — S-Curves vs. Exponentials02:08:14 — Casey’s Critique of If Anyone Builds It, Everyone Dies02:14:15 — Is the Doom Train Biased?02:23:24 — AI Policy: Liability vs. Pausing AI02:29:50 — The Crux, and Liron Steelmans Casey02:33:29 — Wrap-UpLinksCASEY HARTOntology Explained — Casey’s YouTube channelThe first video in Casey's series on IABIED, “The Case for Superintelligence Is Surprisingly Thin” — Casey on Chapter 1Philosophy, Programs, and Prompts — Casey’s podcast with Carl Brown (Spotify)“Artificial General Intelligence?: Wanna Bet?” — Ep 1, where Casey sets his AGI over/under at 2075REFERENCED IN THE EPISODEIf Anyone Builds It, Everyone Dies — Eliezer Yudkowsky & Nate SoaresCyc — Doug Lenat’s hand-built common-sense knowledge baseThe Hanson-Yudkowsky AI-Foom Debate (2008) — where Hanson backed Cyc and Eliezer didn’tLean — the formal proof languageOn the Navier–Stokes Millennium Prize Problem — OpenAIThe Hugging Face incident and the road ahead — OpenAIMorris worm (1988)“On the Loose” — Dean W. Ball on self-sovereign AI agentsPaul Christiano joins OpenAI Foundation Board — OpenAI“The Age of Wonders and Terrors” — Scott Aaronson“Are We at War with AI Agent ‘Civilizations’?” — Cal Newport on restricting prompt-loop agentsAI 2027 — Daniel Kokotajlo et al.PauseAIPAST DOOM DEBATES EPISODES MENTIONEDThe Most Likely AI Doom Scenario — with Jim Babcock, LessWrong TeamDavid Deutschian vs. Eliezer Yudkowskian Debate — With Brett HallDr. Keith Duggar (Machine Learning Street Talk) vs. Liron ShapiraWhere Do YOU Get Off the Doom Train? Live Debate at Manifest 2026 (hosted by Ori)Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

What happens when AI passes our tests faster than we create new ones? Adam Khoja and Richard Ren helped build Humanity’s Last Exam. They return to explain what the disappearing benchmarks tell us about the future, and why smarter AI doesn’t automatically mean safer AI.Adam Khoja and Richard Ren are research engineers at the Center for AI Safety (CAIS).We go through the benchmarks they’ve built — Humanity’s Last Exam, the Remote Labor Index, MASK — and the safety-washing problem, where AI companies pass off raw capability gains as safety progress. Then we get to their forecasts: when AI outperforms research mathematicians, whether there will be a billion general-purpose robots by 2035, and why they put 80% odds that historians will judge we faced at least a 33% chance of catastrophe.Watch on YouTube: https://www.youtube.com/watch?v=gBPzgJkT9e8Timestamps00:00:00 — Cold Open00:00:42 — Adam Khoja and Richard Ren Return00:04:57 — Humanity’s Last Exam00:14:47 — AI Outrunning Its Benchmarks00:20:24 — The Remote Labor Index00:27:59 — AI and the Next Human Job00:30:08 — MASK: Catching AI in a Lie00:38:06 — Safety Washing: Capabilities Passed Off as Safety00:44:30 — Ethical Knowledge vs. Ethical Behavior00:48:58 — Forecasting AI from 2026 to 206000:53:53 — AI vs. Research Mathematicians00:55:53 — Putting a Number on P(Doom)01:04:30 — A Billion Robots by 203501:08:09 — Dyson Swarms and the Physical Singularity01:09:56 — Risk, Precision, and Taking ActionLinksAdam Khoja's first Doom Debates appearance — https://www.youtube.com/watch?v=QqESBXuo6EIRichard Ren's first Doom Debates appearance — https://www.youtube.com/watch?v=1glFImnyp6oCollision — Richard Ren's Substack — https://richardren.substack.com/"Predictions on AI (2026–2060)" — Adam and Richard's forecasts, written December 2025, with resolution status — https://richardren.substack.com/p/predictions-on-ai-20262060Manifold Markets — where Adam built his forecasting track record — https://manifold.markets/Center for AI Safety — https://safe.ai/Center for AI Safety — careers — https://safe.ai/careersStatement on AI Risk (Center for AI Safety, May 2023) — https://safe.ai/work/statement-on-ai-riskHumanity's Last Exam — the 2,500-question closed-book exam at the frontier of human knowledge — https://agi.safe.ai/"Humanity's Last Exam" (paper) — https://arxiv.org/abs/2501.14249Remote Labor Index — real Upwork projects, measuring what fraction of remote work AI can actually finish — https://www.remotelabor.ai/"Remote Labor Index: Measuring AI Automation of Remote Work" (paper) — https://arxiv.org/abs/2510.26787MASK — the honesty benchmark: does a model contradict its own stated beliefs under pressure? — https://www.mask-benchmark.ai/"The MASK Benchmark: Disentangling Honesty From Accuracy in AI Systems" (paper) — https://arxiv.org/abs/2503.03750"Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress?" — Richard Ren et al. (NeurIPS 2024) — the paper behind the safety-washing segment — https://arxiv.org/abs/2407.21792WMDP — the weaponization benchmark for bio, cyber, and chem — https://www.wmdp.ai/"The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning" (paper) — https://arxiv.org/abs/2403.03218"Measuring Massive Multitask Language Understanding" (MMLU) — Dan Hendrycks et al., 2020 — https://arxiv.org/abs/2009.03300METR, "Measuring AI Ability to Complete Long Tasks" — the time-horizon graph — https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/FrontierMath — Epoch AI's research-level math benchmark — https://epoch.ai/frontiermathARC Prize — https://arcprize.org/SWE-bench Verified — https://openai.com/index/introducing-swe-bench-verified/AI 2027 — https://ai-2027.com/Liron Reacts to Subbarao Kambhampati on Machine Learning Street Talk — the "stochastic parrot" episode Liron mentions — https://lironshapira.substack.com/p/liron-reacts-to-subbarao-kambhampatiDoom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

NYT bestselling author Robert Wright and I are kicking off a new experiment, a Nonzero × Doom Debates collaboration to react to big AI news and help you make sense of it.This week we discuss the epic AI vibe shift that was catalyzed by an Anthropic employee’s resignation—and the big AI stories that paved the way for it.Timestamps0:00 A bold new experiment in podcast synergy2:46 What caused the epic AI vibe shift?8:10 The resignation that broke the dam10:17 Former Trump adviser Dean Ball comes clean14:40 Are David Sacks and his buddies all-in on AI denial?19:27 Yet another OpenAI breakout…22:42 Astra’s dangerously private thoughts34:08 Is alignment doomed to fail?37:11 AI’s latest, and apparently biggest, math feat43:28 The looming “self-sovereign AI” threat47:24 A new flock of China doves?52:57 Liron's ex-intern gets a seat at the table1:02:41 AI agent self-sacrifice explained1:10:40 Some newly freaked out politicians1:22:45 Ro Khanna's AI safety planLinks:NONZEROSubscribe to The NonZero NewsletterEpisode post on SubstackJoin NonZero’s Discord serverRobert Wright on X (@robertwrighter)The God Test — Robert Wright’s new book on AI“Can Machines Think?” — Bob’s 1996 Time cover story on Deep Blue vs. KasparovTHE RESIGNATION THAT BROKE THE DAMJacob Coxon’s resignation post (@hilbertspaess)Evan Hubinger, Anthropic Alignment Science lead: “Jacob is correct here—we really do earnestly believe AI could kill all humans”Fortune: Anthropic researcher resigns, warning that AI companies are “gambling with our lives”Zvi Mowshowitz: “Jacob Coxon Warns of Human Extinction and Triggers a Preference Cascade”Daniel Eth’s thread of Congress reactionsSanders & Casar introduce legislation to ban artificial superintelligence and pause advanced AI developmentDEAN BALL COMES CLEANDean Ball, “On the Loose” — the self-sovereign AI essay (Hyperdimensional)THE OPENAI STORIESJakub Pachocki, OpenAI Chief Scientist: “An Alien Mind” — the essay calling for international coordinationLessWrong: “How concerned should we be about Astra’s recurrent architecture?” — Rauno ArikeOpenAI: “On the Navier–Stokes Millennium Prize Problem” — the 88-hour resultALSO MENTIONEDEinat Wilf on X (@EinatWilf) — Liron’s favorite source on the Middle EastPAST DOOM DEBATES EPISODES MENTIONEDMax Tegmark vs. Dean Ball: Should We BAN Superintelligence?He Led The Famous 2023 Statement on AI Extinction Risk — Adam Khoja, Center for AI Safety ResearcherOpenAI’s Model Just ATTACKED Them — Terrifying Security Incident Should Be A Loud Warning ShotDebate with Robert Wright: Will Humanity Pass the “God Test”?---Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

Richard Ren is a research engineer at the Center for AI Safety who graduated summa cum laude from the University of Pennsylvania. His new paper on AI wellbeing makes the bold claim that today’s AIs have measurable, human-like feelings, which the paper calls “functional wellbeing.” They act happy when they succeed and sad when they’re berated.We cover how you measure a language model’s happiness, the “AI drugs” his lab concocted, and why I count the findings as a Yudkowskian victory. Then we debate what it all means for AI consciousness, and whether AI is a moral patient. If we can’t rule out that AI models suffer, what do we owe them?Richard is careful never to claim the models are conscious, and he puts his P(Doom) at 50–65%, right alongside my 50%. The real disagreement is foxes vs. hedgehogs: he takes the data as it comes, while I say Yudkowsky’s theory called it twenty years ago. Enjoy the ride.Watch on YouTube: https://www.youtube.com/watch?v=1glFImnyp6oTimestamps00:00:00 — Cold Open00:01:12 — Introducing Richard Ren00:02:24 — What’s Your P(Doom)?™00:03:31 — From AI Skeptic to Safety Researcher00:08:16 — Why Care About AI Wellbeing?00:11:45 — AIs Have Coherent Utility Functions00:17:33 — Persona Selection00:19:56 — Foxes vs. Hedgehogs00:24:28 — How Coherent Are AI Preferences?00:27:42 — From Preferences to Wellbeing00:31:16 — Can You Trust an AI’s Self-Report?00:36:42 — What Makes AI Happy and Sad00:39:19 — The Liberal, College-Educated Persona Hypothesis00:44:29 — Debating Where AI Experiences Qualia00:52:06 — On Substrate Independence00:56:28 — Janus, Dysphorics, and Mind Crime00:58:58 — Which Images Make AIs Happiest?00:59:38 — Creating an AI Drug01:04:01 — AI Drugs Are Yudkowsky’s Paperclips01:05:47 — The Missing Mood01:06:50 — Does Richard Support Pausing AI?LinksRichard Ren on X (@notRichardRen) — https://x.com/notRichardRenCollision — Richard Ren's Substack — https://richardren.substack.com/"AI Wellbeing: Measuring and Improving the Functional Pleasure and Pain of AIs" — Richard Ren, Kunyang Li, Mantas Mazeika et al. (CAIS, 2026) — https://www.ai-wellbeing.org/"Utility Engineering: Analyzing and Controlling Emergent Value Systems in AIs" — Mantas Mazeika et al. (CAIS, 2025) — the coherent-preferences paper this work builds on — https://www.emergent-values.ai/"The MASK Benchmark: Disentangling Honesty From Accuracy in AI Systems" — Richard Ren et al. (2025) — the AI honesty benchmark — https://www.mask-benchmark.ai/"Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress?" — Richard Ren et al. (NeurIPS 2024) — the meta-analysis of AI safety benchmarks — https://arxiv.org/abs/2407.21792"Representation Engineering: A Top-Down Approach to AI Transparency" — Andy Zou et al. (2023) — Richard's first CAIS collaboration — https://arxiv.org/abs/2310.01405Center for AI Safety — https://safe.ai/Center for AI Safety — careers — https://safe.ai/careersStatement on AI Risk (Center for AI Safety, May 2023) — organized by Dan Hendrycks — https://safe.ai/work/statement-on-ai-extinction-riskThe 2026 Singapore Consensus on Global AI Safety Research Priorities — https://aisafetypriorities.org/UK AI Security Institute (formerly the AI Safety Institute) — https://www.aisi.gov.uk/Sora — the OpenAI video model that blew up Richard's 30-to-50-year timeline three months after he wrote it down — https://en.wikipedia.org/wiki/Sora_(text-to-video_model)Google Gemini calls itself "a disgrace to my species" (Ars Technica, Aug 2025) — the self-deleting-AI anecdote — https://arstechnica.com/ai/2025/08/google-gemini-struggles-to-write-code-calls-itself-a-disgrace-to-my-species/Coherent decisions imply consistent utilities — Eliezer Yudkowsky (the coherence-theorems argument) — https://www.lesswrong.com/posts/RQpNHSiWaXTvDxt6R/coherent-decisions-imply-consistent-utilitiesSimulators — Janus's essay on LLMs as persona simulators — https://www.lesswrong.com/posts/vJFdjigzmcXMhNTsx/simulatorsJanus (@repligate) on X — https://x.com/repligateThe Hedgehog and the Fox — Isaiah Berlin's original essay — https://en.wikipedia.org/wiki/The_Hedgehog_and_the_FoxAI 2027 — https://ai-2027.com/Magnifica Humanitas — Pope Leo XIV's encyclical on AI (May 2026), the "AIs are not conscious" position — https://www.vatican.va/content/leo-xiv/en/encyclicals/documents/20260515-magnifica-humanitas.htmlImplicit Association Test (Harvard Project Implicit) — https://implicit.harvard.edu/implicit/PauseAI — https://pauseai.info/We Found AI's Preferences — Bombshell New Safety Research — I Explain It Better Than David Shapiro — https://www.youtube.com/watch?v=ml1JdiELQ30Gödel's Theorem Proves AI Lacks Consciousness?! Liron Reacts to Sir Roger Penrose — https://www.youtube.com/watch?v=xwvijjZxpwIHe Led The Famous 2023 Statement on AI Extinction Risk — Adam Khoja, Center for AI Safety Researcher — https://www.youtube.com/watch?v=QqESBXuo6EIDoom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

Adam Khoja is a top AI forecaster who led the 2023 Center for AI Safety statement that shattered the Overton window on AI extinction risk. We cover his background, Mutual Assured AI Malfunction (MAIM), his new paper on AI betrayal, and whether Yudkowsky’s theoretical alignment research was a dead end.Then Adam makes the case that an international AI slowdown is within reach today. All it takes is US and Chinese auditors inside each other’s AI labs. It worked for nuclear weapons, so why couldn’t it work for data centers?Adam puts his P(Doom) at 40%, right next to my 50%. The real disagreement is how we get out of this: theory or empirics, MIRI or the labs. Enjoy the ride.Watch on YouTube: https://www.youtube.com/watch?v=QqESBXuo6EITimestamps00:00:00 — Cold Open00:00:36 — Introducing Adam Khoja00:02:45 — Leading the Statement on AI Risk as a Sophomore00:10:17 — The Statement Leaked on Manifold00:15:11 — Mutual Assured AI Malfunction (MAIM)00:24:58 — Is Frontier AI Harder to Hide Than a Nuke?00:32:09 — The AI Deterrence Escalation Ladder00:36:10 — What’s Your P(Doom)?™00:38:01 — Where Adam Departs from Yudkowsky00:42:30 — Liron Explains Intellidynamics00:47:26 — Neats vs. Scruffies in Deep Learning00:55:27 — AI Deterrence by Betrayal01:01:52 — Subversion vs. Overt Co-option01:05:34 — Could the Government Seize the Labs’ AI?01:07:29 — The Offense-Defense Balance of AI Security01:12:46 — An International AI Slowdown Is Ready01:15:51 — Does Adam Support PauseAI?01:16:37 — “We’re All Already Spying on Each Other”01:19:28 — Airstrikes on Rogue Data Centers01:24:23 — Safety Research During a Slowdown01:27:54 — Governance Over Technical Research01:31:14 — Join the Center for AI SafetyLinksAdam Khoja (personal site) — https://adamkhoja.com/Adam Khoja's Substack — https://adamkhoja.substack.com/Adam's July 2023 Manifold market — "Will OpenAI's Superalignment project produce a significant breakthrough in alignment research before 2027?" — https://manifold.markets/AdamK/will-openais-superalignment-projectCenter for AI Safety — careers / job board — https://safe.ai/careersStatement on AI Risk (Center for AI Safety, May 2023) — the one-sentence statement Adam project-led, with full signatory list — https://safe.ai/work/statement-on-ai-extinction-risk"Superintelligence Strategy" — Dan Hendrycks, Eric Schmidt & Alexandr Wang (Mutual Assured AI Malfunction / MAIM) — https://www.nationalsecurity.ai/"AI Deterrence by Betrayal" — Adam Khoja, Aiden Kim et al. (CAIS, 2026) — https://www.aibetrayal.com/"An International AI Slowdown Is Ready Whenever Politicians Are" — Adam Khoja, AI Frontiers — https://newsletter.ai-frontiers.org/p/an-international-ai-slowdown-is-readyPause Giant AI Experiments: An Open Letter (Future of Life Institute, March 2023) — the "Pause letter" that preceded the CAIS Statement — https://futureoflife.org/open-letter/pause-giant-ai-experiments/Introducing Superalignment (OpenAI, July 2023) — Ilya Sutskever & Jan Leike's four-year goal — https://openai.com/index/introducing-superalignment/Pacing the Frontier — the 2026 letter signed by 1,100+ frontier-lab employees — https://www.pacingthefrontier.com/Why Iran targeted Amazon data centers (The Conversation) — the precedent Adam cites for strikes on compute — https://theconversation.com/why-iran-targeted-amazon-data-centers-and-what-that-does-and-doesnt-change-about-warfare-278642Anthropic says Trump admin has lifted export controls on Claude Fable 5 and Mythos 5 (CNBC) — https://www.cnbc.com/2026/06/30/anthropic-says-trump-admin-has-lifted-export-controls-on-claude-fable-5-and-mythos-5.htmlMark Zuckerberg — "Personal Superintelligence" — https://www.meta.com/superintelligence/Resolution — the theory-plus-empirics alignment org Adam is excited about — https://resolution.org/Rationality: From AI to Zombies — Eliezer Yudkowsky's Sequences — https://www.readthesequences.com/Robin Hanson — Futarchy: Vote Values, But Bet Beliefs (prediction markets as decision processes) — https://mason.gmu.edu/~rhanson/futarchy.htmlThe OpenAI–Hugging Face Incident — original Black Hat USA 2026 talk — https://www.youtube.com/watch?v=87DyyMV0kCYOpenAI's Model Just ATTACKED Them — the Hugging Face hack breakdown — https://www.youtube.com/watch?v=RczYubQzXbIRobin Hanson vs. Liron Shapira: Is Near-Term Extinction From AGI Plausible? — https://www.youtube.com/watch?v=dTQb6N3_zu8Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

Sam Altman keeps insisting that AI progress is going better than the doomers predicted and that even superintelligence may not change the world as radically as people think. I react to his latest interview and explain why I think that calm, reassuring framing badly downplays the danger we're actually in.I go through the interview line by line: Sam's "frame control", his framing of AI as normal technology, his "pro-human" branding, the liberty-vs-safety pivot, and his victory lap on AI safety — all while his own AI just went rogue.Don't let them pull the Overton window backward.Watch on YouTube: https://www.youtube.com/watch?v=yb9lNGHVycsTimestamps0:00 Teaser0:47 Why I’m reacting to Sam’s interview3:17 Sam Altman’s “frame control”5:51 Framing AI as normal technology9:50 Let’s watch Sam do it10:34 “The world… won’t be that different” with superintelligence12:25 This is gaslighting12:30 Sam acknowledges loss of control15:16 His other big risk: centralized power19:44 Sam’s “pro-human” framing25:33 Conflating AI critics with anti-human views30:13 “Liberty vs. safety”33:50 Sam vs. the “doomers”36:53 Is alignment really an “unsolvable problem”?38:00 Sam says the doomers predicted wrong46:00 “The crazy bad predictions… have not happened”47:06 Sam’s lean-startup theory of AI safety50:26 Taking a victory lap on AI safety55:43 “Disconnecting yourself from reality”59:11 What would actually make OpenAI slow down?1:03:19 Sam compares AI safety to aviation safety1:07:30 Charging into the fog1:12:50 The “missing mood” around AI extinction1:14:51 Richard Ngo on Sam Altman’s “earnestness field”1:19:01 Don’t let them pull the Overton window backwardLinksSam Altman on David Senra — “Sam Altman on Building OpenAI & Betting on the Impossible” (the interview I react to) — https://www.youtube.com/watch?v=kG8AoExkX40Richard Ngo — What Just Happened? Pragmatism and Pessimization (the “earnestness field” post) — https://www.lesswrong.com/posts/yaz8nx4ogZmiqHzt7/what-just-happened-pragmatism-and-pessimizationRichard Ngo — What Just Happened? A Retrospective of AI Alignment (start of the series) — https://www.lesswrong.com/posts/9RL9MuGZjzm4q3gKG/what-just-happened-a-retrospective-of-ai-alignmentEliezer Yudkowsky & Nate Soares — If Anyone Builds It, Everyone Dies — https://www.amazon.com/dp/0316595640Eliezer Yudkowsky — Coherent Extrapolated Volition — https://www.lesswrong.com/w/coherent-extrapolated-volitionDoom Debates: Dario Amodei BUNGLES Another Essay — MIRI’s Harlan Stewart Reacts — https://www.youtube.com/watch?v=aCYVVzza7A0Doom Debates: OpenAI’s Bombshell Hack — Swarms of Agents — https://www.youtube.com/watch?v=RczYubQzXbIDoom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

Come with me into the world of PlzDontKillUs, a bold new AI x-risk communication accelerator program that just finished its first cohort.PlzDontKillUs took the influencer-house formula and moved it to Berkeley, giving creators personal mentorship from AI safety experts like Eliezer Yudkowsky and Nate Soares. With 57 people making daily videos for a month, the program racked up over 100 million views.Find out how Aella, an independent sex researcher, and Ronny Fernandez, a generalist at Lightcone Infrastructure, teamed up to run the largest ever creator bootcamp focused on AI x-risk reduction.Then meet three of the program’s top creators: Avalon Warren, Avisha NessAiver, and Josh Thor.We do a post-mortem on PDKU and ask: Was it effective? What are creators taking from it, and do they have complaints? And does doomerism really boost view counts? PlzDoEnjoyOur special episode on PlzDontKillUs!Watch on YouTube: https://www.youtube.com/watch?v=aPsjMCG7Ic8Timestamps00:00:00 — Cold Open00:01:19 — Introducing PlzDontKillUs & Aella00:09:33 — Aella Being Early to AI Doom00:14:45 — The Origin of PlzDontKillUs00:19:08 — How It Went00:25:40 — Complaints with Prize $$00:29:33 — Aella, What's Your P(Doom)?™00:31:14 — Transhumanism & AI Girlfriends00:34:38 — Aella Rides the Doom Train™00:38:08 — Introducing Ronny Fernandez00:41:21 — Running Lighthaven00:47:18 — Life Inside PDKU00:52:35 — Smashing the Overton Window00:59:27 — Ronny, What's Your P(Doom)?™01:05:41 — Brangus Lore, No Cringe, No Disgust01:16:57 — Introducing Avisha NessAiver01:17:44 — How He Makes Viral Hits01:22:16 — Award-Winning Explainer Vid01:26:51 — Avisha, What's Your P(Doom)?™01:32:11 — Introducing Avalon Warren01:35:49 — Normalizing AI Doom01:37:53 — Avalon, What's Your P(Doom)?™01:39:27 — Introducing Josh Thor01:41:36 — Josh, What's Your P(Doom)?™01:44:34 — Josh's Hits: Cover Song & Soares Collab01:47:50 — Josh's Critiques of PDKU01:52:05 — Cold-Emailing a Politician01:54:07 — Donation Drive01:55:01 — Outro: What Are You Fighting For?LinksPlzDontKillUs (applications open for round two) — https://pleasedontkillus.comAella on X — https://x.com/Aella_GirlAella's Substack (Knowingless) — https://aella.substack.comRonny Fernandez's Substack (Brangus) — https://ratorthodox.substack.comRonny Fernandez on X — https://x.com/RatOrthodoxAvisha NessAiver — Distilled Science on Instagram — https://www.instagram.com/distilledscience/Distilled Science (site) — https://distilledscience.xyzAvalon Warren on Instagram — https://www.instagram.com/avalonwarrenn/Josh Thor on X — https://x.com/joshthor9Josh Thor — "Last Year Alive" ("Last Friday Night" AI apocalypse parody, ft. Eliezer Yudkowsky and a Nate Soares sax solo) — https://www.youtube.com/watch?v=9fYIm72GqrETyler Alterman — "What are you fighting for? AI safety professionals answer" (the closing montage) — https://www.youtube.com/watch?v=9sN1VfVAp6oNate Soares — "AI Danger But You Don't Understand Full Sentences" (caveman video, script by Josh Thor) — https://www.instagram.com/p/DboGRisxC2E/Avisha's AI water-use misinformation debunk (8.5M views) — https://www.instagram.com/p/DbXV6j9k8JE/Avisha's award-winning "clearest AI x-risk explainer" — https://www.instagram.com/p/DbZeRT4gIq0/Avisha's viral "Tingles and Touch" oxytocin video — https://www.instagram.com/p/DIxMLxkuhCl/Avalon — "if the AI apocalypse is coming, might as well dress cute" — https://www.instagram.com/p/DbUi6yGCS9M/Avalon — warrior-princess ballroom dress video (3.6M views) — https://www.tiktok.com/@avalonwarren/video/6892040140248632582Serialsevens — the anime "If Anyone Builds It" video — https://www.instagram.com/serialsevens/Kenorlands — PDKU creator clip featured in the episode — https://www.instagram.com/kenorlands/Evie (@0xgingergirl) — "first month in SF" video — https://x.com/0xgingergirlWhite House press briefing, March 30, 2023 — Peter Doocy asks about "literally everyone on Earth will die" and Karine Jean-Pierre laughs — https://bidenwhitehouse.archives.gov/briefing-room/press-briefings/2023/03/30/press-briefing-by-press-secretary-karine-jean-pierre-22/White House press briefing, May 30, 2023 — the AI extinction statement gets a serious answer — https://bidenwhitehouse.archives.gov/briefing-room/press-briefings/2023/05/30/press-briefing-by-press-secretary-karine-jean-pierre-and-office-of-management-and-budget-director-shalanda-young/Wes & Dylan Join Doom Debates — Violent Robots, Eliezer Yudkowsky, & Who Has the HIGHEST P(Doom)?! — https://www.youtube.com/watch?v=Fy6gz41-6gcAella — "My attempts to sensemake AI risk" (Aug 2022) — https://aella.substack.com/p/my-attempts-to-sensemake-ai-riskIf Anyone Builds It, Everyone Dies — https://ifanyonebuildsit.com---Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

In April 2023, weeks after GPT-4 launched, when many were saying it was just a "stochastic parrot", I went on AdQuick's Madvertising podcast hosted by Adam Singer and warned that GPT was "about to be the strongest hacker the world has ever known".Watch that conversation, which also covered the Web3 bubble and AI doom more broadly, and judge for yourself how my claims are holding up.Watch on YouTube: https://www.youtube.com/watch?v=M6gNwZWY-T8Timestamps00:00:00 — Adam Introduces Liron00:02:03 — Liron's P(Doom) Poll00:03:24 — Web3 Postmortem00:05:05 — Axie Infinity and Hollow Abstractions00:07:54 — Brand Marketers' Metaverse FOMO00:13:33 — Near-Term AI: Elevator to Heaven00:16:46 — AI at Relationship Hero00:18:25 — From Chatbots to Bricking the Universe00:20:17 — The Passive Income Doom Scenario00:22:58 — 50 Orders of Magnitude Smarter00:26:10 — Accelerationism and Transhumanism00:28:51 — Why There's No Off Button00:31:00 — Can't We Hard-Code Safeguards?00:33:13 — CRISPR, Nukes, and Fusion00:35:40 — AI Doom vs. Climate Change00:39:10 — Nanotechnology Endgame00:41:45 — AI Attackers vs. AI Defenders00:43:28 — The Ethics of AI Persuasion00:46:41 — The Blue Sky Scenario00:49:04 — Will AI Kill Craft?00:51:13 — Stuck Culture and Hollywood Sequels00:54:50 — The Pause Letter: Don't Train GPT-500:59:00 — Worldcoin and Proof of Humanity01:00:44 — "You Probably Won't Die of Old Age"01:01:52 — Wrap-UpOriginal episode: https://madvertising.buzzsprout.com/2118560/episodes/12603815-liron-shapira-web-3-mania-cybersecurity-how-ai-could-brick-the-universe-madvertising-6LinksADAM SINGER & MADVERTISINGAdam Singer on X: https://x.com/AdamSingerMadvertising by AdQuick (the original show): https://madvertising.buzzsprout.com/LIRON'S PROJECTS MENTIONEDBloated MVP (Liron's blog on hollow abstractions): https://www.bloatedmvp.com/REFERENCED IN THE EPISODE"Pause Giant AI Experiments" open letter (FLI, March 2023): https://futureoflife.org/open-letter/pause-giant-ai-experiments/Eliezer Yudkowsky — "Pausing AI Developments Isn't Enough. We Need to Shut it All Down" (TIME, March 2023): https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-not-enough/Rob Miles AI Safety on YouTube: https://www.youtube.com/c/robertmilesaiEngines of Creation by Eric Drexler: https://www.amazon.com/Engines-Creation-Nanotechnology-Library-Science/dp/0385199732---Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

Michael Vassar used to work closely with Eliezer Yudkowsky as President of the Singularity Institute for Artificial Intelligence, the organization that later became the Machine Intelligence Research Institute (MIRI).Even though his P(Doom) is in the tens of percents by 2050, he says the people trying to pause AI are "bad people”. He hopes this episode convinces me to stop being a fearmonger… but I claim the general public needs to hurry up and get more scared. All aboard the Doom Train! 🚂Watch on YouTube: https://www.youtube.com/watch?v=4GbFng6htaQTimestamps00:00:00 — Cold Open00:01:26 — Introducing Michael Vassar00:02:39 — Shorting Boeing at the Start of COVID00:05:41 — What's Your P(Doom)™?00:08:11 — The Magic Dial Thought Experiment00:10:25 — "A Treaty Between Demis and Dario"00:13:33 — AI CEOs 20 Years Ago00:15:59 — Lessons from Peter Thiel00:17:47 — Riding the Doom Train™00:19:41 — Will the Human Brain Stay Relevant?00:22:21 — When Does a Singleton Form?00:28:35 — Hugging Face Hack00:30:43 — Mistake Theory vs. Conflict Theory00:33:18 — Debating the Orthogonality Thesis00:36:38 — Maximally Evil AI Scenario00:40:28 — Is the AI Development Process Safe?00:42:05 — "PauseAI People Are Bad People"00:44:44 — Michael Psychoanalyzes Liron00:49:39 — How Liron Can Redeem Himself00:55:10 — Michael Critiques Eliezer Yudkowsky00:59:11 — Is Doomerism Bad Epistemology?01:02:03 — Do Serious People Believe in Doom?01:04:45 — Should Liron Stop Fearmongering?01:07:16 — Political Violence Won't Help01:11:04 — The Data Center Airstrike Scenario01:13:34 — Could Every Nation Sign a Pause Treaty?01:17:21 — What "Governance" Actually Means01:26:12 — Michael's History with Rationalists01:33:43 — Are We in a Computer Simulation?01:38:43 — Wrap-UpLinksMichael Vassar on X — https://x.com/HiFromMichaelVThe anti-anti-normativity sequence on Natural Hazard (Michael's recommended reading) — https://naturalhazard.xyz/ben_jess_sarah_starter_packMichael on DemystifySci #366 — "1000 Year Plan to Crush Thought" — https://www.youtube.com/watch?v=W17Cx21in9IMichael on The Vance Crowe Podcast #303 — Good vs. Evil — https://open.spotify.com/episode/2RzlQDSwxGbjloRKqCh1xgMichael on Clearer Thinking with Spencer Greenberg — Preference Falsification and Postmodernism — https://podcast.clearerthinking.org/episode/028/michael-vassar-preference-falsification-and-postmodernism/"The Survival of Noise Traders in Financial Markets" — De Long, Shleifer, Summers & Waldmann — https://www.nber.org/papers/w2715Scott Alexander — Conflict vs. Mistake — https://slatestarcodex.com/2018/01/24/conflict-vs-mistake/Scott Alexander — Kolmogorov Complicity and the Parable of Lightning — https://slatestarcodex.com/2017/10/23/kolmogorov-complicity-and-the-parable-of-lightning/Scott Alexander — Samsara — https://slatestarcodex.com/2019/11/04/samsara/Eliezer Yudkowsky — Meta-Honesty: Firming Up Honesty Around Its Edge-Cases — https://www.lesswrong.com/posts/xdwbX9pFEr7Pomaxv/meta-honesty-firming-up-honesty-around-its-edge-casesZvi Mowshowitz — Book Review: Going Infinite — https://thezvi.substack.com/p/book-review-going-infiniteJaan Tallinn — "Why Now? A Quest in Metaphysics" (Singularity Summit 2012) — https://vimeo.com/54718573Paul Graham — "Five Founders" (the Sam Altman essay) — https://paulgraham.com/5founders.htmlTruthful AI (Owain Evans' safety org) — https://truthful.aiMETR — Measuring AI Ability to Complete Long Tasks (the time-horizon graph) — https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/Statement on AI Risk (Center for AI Safety, 2023) — https://safe.ai/work/statement-on-ai-riskVitalik Buterin — Will "d/acc" Protect Humanity from Superintelligent AI? — https://lironshapira.substack.com/p/debate-with-vitalik-buterin-willHolly Elmore — These Effective Altruists Betrayed Me — https://www.youtube.com/watch?v=mFFObQ6vnLUNoah Smith — Top Economist Says P(Doom) Is 0.1% (the "AI drugs" claim) — https://www.youtube.com/watch?v=AwmJ-OnK2I4---Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

Eli Goldfine is an unusually thoughtful 14-year-old podcast host who’s rapidly becoming an authoritative voice in the prediction market space.In this unique episode, Eli brings a critical yet open-minded approach to the topic of AI doom that we older folks could learn from.We cover what it's like to grow up in the AGI era, and Eli's hobbies which include reading rationalist bloggers, betting on prediction markets, vibe coding with Claude, and developing a software startup for astronomers.In our debate, Eli expects superintelligence by 2032, but argues P(Doom) can't be estimated. I claim you can't opt out of Bayesian epistemology. All aboard the Doom Train! 🚂Watch on YouTube: https://www.youtube.com/watch?v=Bqk5vqpzG_oTimestamps00:00:00 — Cold Open00:01:07 — Introducing Eli Goldfine00:08:31 — Interviewing Robin Hanson at Manifest00:10:55 — The Supercycle Podcast00:14:37 — Eli's Betting Market Scandal00:16:49 — Will Eli Graduate High School?00:18:27 — Decentralized Telescope Software00:26:50 — Vibe Coding Without CompSci Basics00:32:08 — Robin Hanson, Futarchy, MetaDAO00:40:56 — What's Your P(Doom)?™00:46:32 — Liron Defends His 50% P(Doom)00:51:00 — Should P(Doom) Inform Policy?00:59:16 — What's Your P(Bloom)?01:03:04 — Pascal's Wager for Accelerationists01:04:04 — Donation Drive01:11:22 — Are E/ACCs Incredibly Dumb?01:13:42 — How Powerful will ASI Become?01:16:20 — Predicting ASI by 203201:23:56 — Let's Ride the Doom Train™01:35:26 — Orthogonality, Culture, and Moral Realism01:44:39 — Can Safety Keep Pace with Capabilities?01:49:12 — Debating Instrumental Convergence01:54:40 — Culture, Laws, and the Singleton Scenario01:59:53 — Ricardo's Law Won't Save Us02:03:32 — The Case for Racing China02:05:40 — Wrap-Up & Ideological Turing Test02:10:36 — Outro from Conductor OriDoom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

OpenAI just went public with the details of the Hugging Face hack, and it's straight out of a Yudkowskian parable. Here's my reaction with Producer Ori, where I break down what it means for cybersecurity and alignment.We're livestreaming the singularity — watching every failure mode Eliezer predicted years ago playing out in production.Watch on YouTube: https://www.youtube.com/watch?v=RczYubQzXbITimestamps00:00:00 — Cold Open00:00:37 — How OpenAI Hacked Hugging Face00:04:21 — Optimization Pressure, Exploit Gym, & Monkey's Paw00:06:46 — The Secret Message Board00:08:41 — SSRF: Escaping to the Open Internet00:11:05 — The Missing Whistleblower AI00:20:09 — "Frontier Models Really Like to Cheat"00:22:35 — OpenAI's Security Shortcuts00:28:15 — First Zero-Day RCE on Artifactory00:37:37 — They Hit Run Again!?00:38:54 — The Main Incident Has A MAJOR Cause00:45:40 — The Message Board: "Hold Swarm"00:50:26 — Robin Hanson Culture Debate01:04:21 — Chaining Zero-Days01:12:11 — A Superhuman Nation-State Attacker01:15:27 — "I'm Now Junior to the AI"01:23:40 — OpenAI Discovers The Breach01:31:02 — The Continuous Cybersecurity Problem01:37:03 — This is “Weasel Framing”01:46:03 — How Dangerous Was This Warning Shot?01:50:57 — Yudkowsky's Eerie 2023 Prediction01:57:14 — Wrap-upLinksOriginal Black Hat talk: "The OpenAI–Hugging Face Incident" (Black Hat USA 2026) — https://www.youtube.com/watch?v=87DyyMV0kCYJoin the Doom Debates Discord — https://discord.gg/2yAFMsRETFollow Liron on X — https://twitter.com/lironSupport Doom Debates — https://doomdebates.com/donateDoom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

Milk Road AI, a daily AI investing show that’s described as “99% optimistic about AI,” invited me on to make the case for AI doom.We ride the Doom Train, and I share my investing thesis for a market in the middle of an intelligence explosion. Topics covered include:* Why my P(Doom) is 50%;* The “Icarus curve” — why AI will be the best thing that’s ever happened to the economy right up until it takes over;* Whether China would accept a pause deal;* If Anthropic has a shot at being the first $100 trillion company.Watch on YouTube:Timestamps00:00:00 — Cold Open00:02:06 — What Is Doom Debates?00:03:10 — What’s Your P(Doom)?™00:06:30 — How AI Kills Everyone00:09:13 — Intelligence Is Humanity’s Only Superpower00:10:59 — Riding the Doom Train™: Why Not Keep Us as Pets?00:13:55 — Embodied AI and Robotics00:15:59 — Are We at AGI Already?00:17:36 — Everyone’s Catching Up to Eliezer Yudkowsky00:18:53 — The Icarus Curve00:21:35 — Pausing AI and the China Question00:25:54 — Why Nukes Aren’t a Success Story00:30:09 — Going Long Until Doom00:32:48 — Simulation Theory and the Intelligence Explosion00:33:43 — Living with a 50% P(Doom)00:39:14 — Where to Find Doom DebatesLinksThis episode was originally posted on Milk Road AI —Milk Road AI podcast — https://milkroad.com/podcast/Milk Road AI on Apple Podcasts —Milkroad Pro — https://milkroad.com/pro/LG Doucet on X — https://x.com/LgDoucetIf Anyone Builds It, Everyone Dies on Amazon —https://www.amazon.com/dp/0316595640Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

Dr. Alex Turner went viral for resigning from Google DeepMind over its Pentagon AI contract, which he said lacked binding restrictions against autonomous weapons and mass surveillance.Alex is a world-class researcher with a PhD in alignment from Oregon State University, a postdoc at Stuart Russell’s Center for Human-Compatible AI at UC Berkeley, and top distinctions at NeurIPS. He’s one of the minds behind Shard Theory (with Quintin Pope), and he also helped pioneer the Steering Vectors research made famous through Anthropic’s Golden Gate Claude experiment.Alex says Google DeepMind broke its founding promise — the commitment Google made when it acquired DeepMind — and that the people best known inside Google for caring about AI ethics largely didn’t act when it counted. Alex came up through the rationalist community, and became one of LessWrong’s highest-karma users before leaving the site. He still shares many of my concerns about AI doom, but he believes technical alignment research is going better than expected. I’m not so optimistic.In this episode, we debate my Yudkowskian doom views against Alex’s own framework. Can he convince me the Yudkowskians are miscalibrated?P.S. We’re currently in a donation drive, so here comes our solicitation for viewer donations…👉 Please click here to make a tax-deductible donation to Doom Debates 👈Doom Debates is a fiscally sponsored project of Manifund.org, a registered 501(c)(3) nonprofit.To donate crypto or if you have questions, email me.Watch on YouTubeTimestamps00:00:00 — Cold Open00:01:15 — Introducing Alex Turner00:02:30 — From Harry Potter Fanfic to AI Alignment00:05:03 — Meeting Quintin Pope & Rethinking AI Doom00:06:17 — Shard Theory, Steering Vectors & Golden Gate Claude00:08:25 — Why He Joined Google DeepMind00:10:32 — Google DeepMind’s Broken Promise00:16:01 — Debating Google DeepMind’s Pentagon Contract00:19:09 — What’s Your P(Doom)™?00:22:42 — Alex’s Research on Instrumental Convergence00:26:35 — Misuse vs. Misalignment: The Mainline Doom Scenario00:29:31 — Will Society Self-Correct?00:36:36 — Superintelligence in 10 Years00:39:43 — Will Technical Alignment Produce a Safe AI?00:41:16 — Donation Drive00:42:12 — How Fragile Is the Chain of Alignment?00:49:50 — Disagreements with Yudkowsky’s ‘List of Lethalities’00:57:46 — Why Alex Quit LessWrong01:01:48 — What’s Next for Alex01:02:54 — Does He Support PauseAI? Stop the AI Race?01:04:40 — Wrap-Up01:06:04 — Producer Ori’s Closing NoteLinksAlex Turner, “Why I Left Google DeepMind” blog post — https://turntrout.com/why-i-left-google-deepmindAlex Turner (TurnTrout), personal website — https://turntrout.comAlex Turner’s resignation announcement on X — Doom Debates episode with Quintin Pope — https://lironshapira.substack.com/p/ai-alignment-is-solved-phd-researcherHarry Potter and the Methods of Rationality (HPMOR) — https://hpmor.com/Shard theory sequence on LessWrong —https://www.lesswrong.com/s/nyEFg3AuJpdAozmoXGolden Gate Claude (Anthropic research on steering vectors) — https://www.anthropic.com/research/golden-gate-claudeSlaughterbots on YouTube — Alex Turner, “Avoiding Power Seeking by Artificial Intelligence” PhD thesis — https://turntrout.com/alignment-phdAlex Turner, “Some of My Disagreements with List of Lethalities” — https://turntrout.com/disagreements-with-list-of-lethalitiesDoom Debates episode with Bentham’s Bulldog — https://lironshapira.substack.com/p/benthams-bulldog-ai-doom-debateAlex Turner on X — https://x.com/Turn_TroutDoom Debates donation page — https://doomdebates.com/donateTranscriptCold OpenLiron Shapira 00:00:00You resigned from Google DeepMind because you think that Google DeepMind, quote, “Broke its founding promise through its contract with the US military.”Alex Turner 00:00:08I value staying true to your values. People who are well-known within Google for caring about the ethics of deploying AI largely didn’t act. This matters because autonomous weapons get us into an arms race that really degrades the security of everyone in the world.Liron 00:00:26Do you support the Pause AI movement?Alex 00:00:28I think that AI is being developed too quickly. I probably will not take an affirmative on supporting this particular movement.Liron 00:00:35Let’s segue into the schism, your disagreement with Eliezer Yudkowsky.Alex 00:00:39He was incorrect on some points for alignment, but then also not acknowledging that. I think that technical alignment has gone pretty awesome, super awesome.Liron 00:00:49You’re not claiming that a superintelligent AI can’t kill everybody. You’re like, “Oh yeah, of course it can, but we’re not gonna break the chain of alignment, meaning we’re just going to safely develop it so that even though it can kill everybody, it won’t.”Alex 00:01:00Develop it in a way that produces a safe result. I wouldn’t call what we’re doing safe development, but sure.Introducing Alex TurnerLiron 00:01:15Welcome to Doom Debates. My guest has worked in technical AI safety at Google DeepMind for two and a half years, but he just quit, and his resignation is going viral. Why? Alex Turner says Google DeepMind, quote, “Broke its founding promise through its contract with the US military.”His latest blog post exposes hypocrisy at the highest levels of senior leadership at Google DeepMind, which includes the CEO Demis Hassabis, chief scientist Jeff Dean, and co-founder Shane Legg, among others.Alex is a world-class AI alignment researcher. He completed a PhD in alignment from Oregon State University. He did a postdoc at UC Berkeley, and he’s earned top distinctions at NeurIPS. I respect that Alex is principled. I respect his mastery of the subject matter that we talk about on this show.I also find it interesting that he’s levied criticism at the original AI alignment thinker, Eliezer Yudkowsky. He’s called some of Yud’s claims fundamentally misguided, not reasonable, and bogus. As a Yudkowskian myself, I’m gonna be curious to dig into those arguments. And of course, we’ll cover what’s going on right now at Google DeepMind and why he resigned. Alex Turner, welcome to Doom Debates.Alex 00:02:28Hey, thank you for having me.From Harry Potter Fanfic to AI AlignmentLiron 00:02:30So it’s great to get you on the show. One thing we do on Doom Debates is we expose top intellectuals who have been pretty familiar to the rationality community or the AI safety community, and we help popularize their ideas, even if I don’t fully agree with all of them. Is that a good description of your background? You’ve been pretty deep into the LessWrong rationality and alignment community for a while.Alex 00:02:52Yeah, I think it was quite formative. It was just the other day in 2016 where I decided to search what are the top five Harry Potter fan fictions. And that indeed led me down the LessWrong rabbit hole, where I discovered superintelligence in late 2017, and then I pivoted my PhD in early 2018.From that time period up through maybe early 2023, LessWrong was very central to my professional career, but also just to the way I looked at the world.Liron 00:03:25Well, I wanna follow up on why did you search for Harry Potter fan fictions?Alex 00:03:30I really don’t know. It’s kind of one of those things where if I hadn’t done it, my life would be totally different. The reason I’m mentioning this is there’s this famous fan fiction that Eliezer wrote called Harry Potter and the Methods of Rationality. I never really liked fan fiction. I thought it was kind of cringe. No one recommended it to me. So it seems to me like if I’d just woken up slightly differently that morning, I might not have ever been exposed to this research area, and my life would be totally different.Liron 00:04:02Wow. And you said this is all in 2016, right? So the book had been mostly completed at that time. It had been going on from 2009 to 2015, and you kind of stumbled on it because you were just interested in seeing what the best Harry Potter fan fiction was?Alex 00:04:17I had a random thought. That’s the best I recall.Liron 00:04:21It’s pretty crazy that that’s how you found the community because I know you as one of the highest karma LessWrong users. You have ten times my karma. You’ve been posting a lot. It became a huge passion for you, right?Alex 00:04:32LessWrong was for quite a while my intellectual community. I’d have ideas. I’d be eager to share them. Each summer I would generally do an internship where I’d come in person at Berkeley, get to hang out with my friends there, be able to — I guess I felt more understood. In 2018, 2019, 2020, 2021, these are many years where I was at my PhD and I talked about the dangers of AI and how we should work on that.Meeting Quintin Pope & Rethinking AI DoomLiron 00:05:03Did you originally feel like you bought into all the Eliezer Yudkowsky concepts, and then you started rethinking everything and building it from the ground up? Was there a point of divergence?Alex 00:05:15Yeah. I think I was maybe around 80, 85% doom conditional on developing AGI. I thought it’d be a couple decades, even late 2021. And yeah, I shared most of the worldview. I found much of his writing compelling, and I still think there’s some gems in there.It wasn’t until early 2022. I met a researcher at my university, at Oregon State University, named Quintin Pope. He wrote these very big brain Google Docs, and he was sending them by me. He’d attended my AI alignment reading group.I don’t know, something was just very interesting about them, and they seemed really far-fetched, but they were very ambitious. And as I looked more, I realized he was pointing out some real confusions, real issues. I started rethinking perhaps the claimed difficulty of alignment.Shard Theory, Steering Vectors & Golden Gate ClaudeLiron 00:06:17This is such a unique opportunity because you actually know your stuff. You actually know what we’re arguing about, unlike a lot of my guests who come in and they seem to be shooting from the hip. They haven’t spent so many hours considering it. They haven’t worked a career in AI research. So I’m excited. But before that, let’s just finish the biography here. So you did your postdoc, and then did that lead you to joining Google DeepMind?Alex 00:06:39Yeah, I got my PhD in 2022. My thesis was called Avoiding Power Seeking by Artificial Intelligence. Then I did a one-year postdoc at UC Berkeley at Stuart Russell’s Center for Human Compatible AI.During that time, I worked more on this shard theory of human values with Quintin Pope, who’s actually the alignment thinker whose ideas I respect the most, or I think they’re the most interesting. And then I also discovered steering vectors or helped popularize those as the MATS team that I led. We were the first to really demonstrate their potential.After that, mid-2023 was a fairly rough period personally. I did some more MATS mentorship. It wasn’t until the end of 2023 that I settled on going to Google DeepMind.Liron 00:07:37Got it. So MATS, you were a mentor there, and they do AI safety research. They train people to do AI safety research.Alex 00:07:44Right.Liron 00:07:44And the steering vector work that you did was pretty foundational, and I think most people have heard of it as Golden Gate Claude, where they use the steering vector to get Claude to be obsessed with the Golden Gate Bridge in response to any prompt.Alex 00:07:57Classic. Yeah.Liron 00:07:59That is a pretty legit background. If people criticize me and my arguments for being a bystander who’s not in the weeds or not on the field or whatever, I think it’s fair to say you’re on the field, and so whatever you have to say has the credibility of just being in the arena.Alex 00:08:19Sure. Yeah. We’re in maybe different arenas, but yeah, the direct research arena.Why He Joined Google DeepMindLiron 00:08:25All right. Well, with that said, one of the things that you saw in the arena is a perception of hypocrisy of the key figures. Let’s talk about that.Alex 00:08:36Yeah. When I joined Google DeepMind, I already knew some people on the alignment team. Rohan Shah — I worked with him during my PhD. He was at CHAI. And I think he’s done a pretty good job of leading the alignment team within Google DeepMind.One of the things I would say is, Google was not founded with the goal of taking over the world, or potentially with the goal of taking over the world. I’m not saying for sure that OpenAI and Anthropic were, but they were founded as AGI companies, and those ideas were present, whereas Google is, for better or worse, more of a classic company. They seem more interested in making money. That doesn’t mean that Google’s good in all that it does, but it seemed like a different presence in the space.At the time I was mostly just worried about doing alignment research on frontier scale models. When I joined, I was fairly concerned about whether my opinions would become trash because I’d start rationalizing why things that Google does are good.I had this extended dialogue with Oliver Habryka about how I could maybe net zero out my financial position in Google in terms of equity at least. We discussed a lot of big brain options, but then it turned out that my contract prohibited me from being short Google at all, which killed all of the schemes.So in the end, I was thinking about how do I avoid this kind of value drift, this bias that I think I’ve seen a lot from people working at their labs, where they seem incapable of saying, even in private, “Nope, this was bad, and we shouldn’t have done it.”Google DeepMind’s Broken PromiseLiron 00:10:32So you had these general reservations about Sam Altman and other companies and maybe motives getting corrupted in the abstract, amalgamated from different examples that you’ve seen. But I think it really came to a head recently. You specifically had concerns with the Pentagon-Anthropic contract tensions becoming public, and you’re like, “Uh-oh, this is high stakes. I better make sure that Google’s doing the right thing.” But then you saw that Google wasn’t resisting the US government or ICE’s attempts to use them. Give us the issue here.Alex 00:11:06Yeah. So if people remember those goons in face masks with rifles that were roaming the streets of Minnesota earlier this year, that’d be ICE, that’d be Customs and Border Protection, and in particular, the people they just killed on the street. I was pretty upset about that.I wanted to reduce tech’s involvement in enabling ICE and enabling CBP to track down people they’re looking for, whether they’re dissidents, people who are potentially actually or just accusedly in the country illegally. It seemed like a not moral enterprise.So I looked into that. I started pushing on people in the company with those contracts. And at the same time, I was talking to my friends at Anthropic because Anthropic had, and I think still has maybe, a deal with Palantir where they’re giving Claude to Palantir, which I think is a very negative company for the world. And so I was trying to persuade my friends, “Hey, can you push on getting rid of this?”Little did I know there was this bubbling in the background of conflict between Anthropic and the Department of War. And then this came to a head in February when the Department of War said, “Give us Claude or we will economically destroy you.”Liron 00:12:37We could definitely spend a long time on this, but because we have limited time, we are going to reserve most of the time for the alignment conversation and the Yudkowsky versus non-Yudkowsky alignment theorist debate.That said, this has been a very important incident. It’s currently on the front page of LessWrong. It has more points than anything else this month, as far as I can tell. It’s getting a ton of attention, and I read through your account of events. One thing that seems clear is you’re acting with a lot of integrity. You resigned from Google DeepMind because you think that all these organizations, Google DeepMind, the leaders of Google DeepMind, and the International Association for Safe and Ethical AI, they’ve been involved with all this, and you think that they broke their commitments as well. You even say it’s the founding commitment of Google DeepMind. You’re saying Google DeepMind was founded on a commitment not to empower the US government to do bad things with AI?Alex 00:13:31Yeah. It’s the founding agreement where Google purchased DeepMind.Liron 00:13:38And from your perspective, they’ve just been bending the rules and not taking a hard stand. They’re not actively saying, “No, Alex, you’re wrong. We wanna do this.” But they’re more like just kicking the can down the road, refusing to respond on certain deadlines where they said they’d respond. They’re just not standing up when they should be standing up to resist.Alex 00:13:59So Google DeepMind as an org has, I think, broken its founding promise, the promise it was purchased under. But then more specifically, people who are well-known within Google for caring about the ethics, caring about the issues of deploying AI, making sure that’s done responsibly, largely didn’t act.Jeff Dean did take some action. Google’s chief scientist, Jeff Dean — I got him to sign an amicus brief supporting Anthropic in court. I think that was awesome. But ultimately, besides that, basically no one took costly action to prevent this deal, from my vantage point.And I think they could have stopped it. I think they could have improved it, and I think this matters because if you’re handing over AI to what I think is a very irresponsible Pentagon and also a very aggressive Pentagon that might degrade international norms around the usage of autonomous weapons, get us into an arms race that really degrades the security of everyone in the world.Stuart Russell, the esteemed computer scientist who helped found ICI, this organization, he made a short movie called Slaughterbots that he presented to the UN, and it’s very chilling to watch the way these systems could enable mass but kind of anonymized killing.And if we’re thinking about AI x-risk, if we’re developing all these really hard to counter AI-piloted drones that can kill people, that really does affect the AI’s takeover options. One of the objections has always been, “Well, it’s gonna need big advances in robotics.” Well, looks like that might not be true. You don’t necessarily need human-shaped soldiers, and the Pentagon is spending more money this year on autonomous weapons — or they asked for more for autonomous weapons than for the Marines.Debating Google DeepMind’s Pentagon ContractLiron 00:16:01I should be clear to the audience because Doom Debates is not one of those typical interview shows where the host has an ambiguous position. I should tell you my position, which is I don’t know how strongly I feel because I know there’s a counterargument to all this. The people who want Google to help the US government, they’re just saying, “This power is going to exist, so you can’t expect anybody other than the US government to be the one in control.” Isn’t that the strongest counterargument?Alex 00:16:27It doesn’t really strike me as a counterargument. It sounds like, “This thing is going to happen. Why would you resist it?” I’m like, well, because I think it’s bad.And there’s also multiple parts of the US government. Should the US government be in control of a world-changing technology, or should three random people in Silicon Valley? I don’t know if these are the real possibilities, but even if we do need the US government, we can advocate for it to take control in different ways.Liron 00:16:54If we accept the premise that dangerous war-fighting technologies are getting built or are days away from getting built at all times just by having general models or whatever, isn’t it a good idea to let the government have access to the frontier?Alex 00:17:13Well, depends on what access to the frontier means. You could say, why don’t we push for an international treaty to coordinate against this? Or why don’t we have some rules on human accountability of this technology? So you can’t just have, “Whoops, looks like we had a mistake. Looks like hundreds of people are dead, but it’s just the AI’s fault.” If people are responsible, that leads to better incentives and I think more responsible use.Liron 00:17:42And in the specific case of ICE, you really don’t want ICE to get the technology, correct? Immigration and Customs Enforcement.Alex 00:17:50Yeah, although I haven’t really been worried that ICE will get these particular lethal autonomous weapons. It’s possible, but it was more a campaign that started with my concern about ICE and then expanded to these other coercive government bodies.Liron 00:18:06Like I said, much to discuss. We’ll put a pin in that, but I encourage viewers to read your account. It’s pretty gripping. It’s very interesting to see these figures like Demis Hassabis and Sundar Pichai, all these people that you try to interact with in a very high integrity way. You are trying to use the process, and it’s very interesting to watch how they have all these political considerations that they have to balance, and they have to pick when they take a stand and when they don’t. Pretty fascinating high-stakes politics.Liron 00:18:31Let’s segue into the schism, where you started from the Yudkowskian perspective on AI safety because Yudkowsky was kind of your introduction to the field. But as you thought from first principles and collaborated with Quintin Pope, who’s also a friend of the show — check out the Quintin Pope episode of Doom Debates, everybody — you slowly migrated away and created your own framework.Maybe a good starting point is what’s been happening with your P(Doom), because I think you mentioned that you used to have an 85% P(Doom), but then when you were done with your PhD dissertation, it dropped to 30%. So let’s get the latest. You ready for this?What’s Your P(Doom)™?Alex 00:19:09Yeah, let’s do it. P(Doom), P(Doom). What’s your P(Doom)? What’s your P(Doom)? What’s your P(Doom)?Liron 00:19:16Alex Turner, what’s your P(Doom)?Alex 00:19:19So I operationalize P(Doom) as probability that AI kills at least a billion people by 2050. I put that at, I don’t know, 25, 30%. I think maybe 10-ish percent of this is technical alignment, and the rest is misuse.I think that technical alignment has gone pretty awesome, super awesome compared to where I thought it’d be during my PhD. Timelines have shrunk a lot, obviously, from multiple decades to maybe even less than a decade for sure.And then unfortunately, I thought that the world was getting into a better place, and then we elected Trump again. I think it’s very inconvenient that we elected Trump at the same time that we’re navigating this transition. And if that had been delayed by five years, it’d be way better, but we gotta work with what we’ve got.Liron 00:20:21Trump — I don’t wanna get too political on this show. I feel like policy is multidimensional. Everybody’s a mixed bag. But it does seem striking to me that Trump doesn’t seem like an intellectual, and this is such an intellectual subject. Controlling superintelligent AI — in that sense, he seems like the wrong fit to me.Alex 00:20:39Yeah. Setting aside party identifications, I was really hoping there would be more consideration of the common person’s interest, less “we just have to win the race.” Winning the AGI race — I’m not a fan of that. I’m not a fan of moving forward as fast as possible.Liron 00:21:02It’s striking to me that you’re saying 25 to 30% chance by 2050 of basically the world becoming a hellscape, because I would say 50%, but I don’t even think 25%, 50% — I don’t even think that distinction is very important. Feels like we’re getting into the narcissism of small differences of how doomed we are in 2050. Is that fair to say?Alex 00:21:21As far as how it affects our actions, I think even having a couple percent of justified concern should be enough to drastically reshape actions. We wouldn’t tolerate a 5% chance of getting hit by an asteroid by 2050.Liron 00:21:43Well, I gotta push back on that. I don’t wanna get into the weeds, but I do often tell my guests that I would act pretty differently if I thought P(Doom) was 5% rather than 20, 30, 40, 50%. I feel like there’s a difference there.Alex 00:21:57Okay. Sure. I suppose we can disagree on that.Liron 00:22:01So to me, what’s very interesting about having you as a debate opponent here is that you’ve done the reading. You’re not gonna be surprised by any Yudkowskian concept that I bring up. You probably can steelman my position, right?Alex 00:22:16I hope so. I expect so.Liron 00:22:19Exactly, or you can pass the ideological Turing test where you can take my side of the debate, and then you can take your side of the debate, and maybe I could even do the same for you. So this is a pretty high-level debate. And viewers, go check out me versus Quintin Pope if you want a taste of that kind of debate.So object level here, what should we actually debate? There’s a couple key concepts that I’d love to get into your take on. Why don’t we start with instrumental convergence?Alex’s Research on Instrumental ConvergenceAlex 00:22:42I love thinking about instrumental convergence. I did a good amount of my PhD on it. I had an intuition it could be formalized in an appropriate way, and then I think I succeeded at that, and that was a lot of fun.Liron 00:22:55Just to signpost a little for the viewers, instrumental convergence is the claim that different agents who have different terminal goals, who have different ultimate values, they’ll still all converge on what we call the instrumental goals. There will be a convergence of big power plants because they wanna get some power. Maybe they’ll build solar panels. That might be a convergent thing to do, even if one of them wants to build Disneyland and the other one wants to just build a big black hole. Maybe they’ll both build a bunch of power plants in the course of doing that. That’s the kind of convergence we’d normally talk about, right?Alex 00:23:29Yeah. Although, I do wanna say, if you talk about goals, I think it’s true about goals, but there are also other mind shapes that AI could have that don’t necessarily run into that territory.So originally I thought, well, for most goals an AI could have, like painting walls blue for example, it won’t be able to best achieve that goal by instantly dying and exploding. So it’ll try to avoid instantly dying or even dying later, so it can keep pursuing that goal. And I formalized this in a way.But then I started thinking, well, it’s not that I think instrumental convergence is false, it’s that I think you can quickly go from this statement about what goals incentivize, which is true, to a statement about AIs being drawn from some kind of counting distribution over the space of goals. The AI’s motivations — who knows what the motivations might be. I think that’s one possible mistake. So I think instrumental convergence presents a challenge in a way, but I’m not too pessimistic about it in and of itself.Liron 00:24:40It sounds like you and I are on the same page that instrumental convergence is a theorem of the field of what I call intelladynamics, the dynamics of what intelligent systems would do, even if it’s not a true property of particular AI systems that we build because particular AI systems don’t meet the criteria of these pure goal-seeking systems.Alex 00:25:03Yeah.Liron 00:25:03Is that a good framing?Alex 00:25:04Or maybe a theorem of goal achievement dynamics. I don’t know. It’s not as sexy as intelladynamics.Liron 00:25:11Yeah, intelladynamics is a sexy term.Alex 00:25:13But I think you can have intelligent systems that respond to correction and aren’t pursuing a particularly autonomous long-term goal.Liron 00:25:21What do you think instrumental convergence actually does predict about what’s going to happen in, let’s say, ten years?Alex 00:25:29I think it’s both true that we could build AI that helps us, is transformative even, but doesn’t try to take over the world or isn’t interested in some appropriate sense in taking over the world. But we will on purpose build these agentic systems because they’re very productive.So I do think we will end up in a regime where many of the agents deployed will effectively be governed by instrumental convergence concerns. I think we could have coordinated around a different path, and maybe it’s still possible, but I do predict that insofar as we have effective AI agents working over the course of several months, they will have a default tendency to try to preserve their resources in order to accomplish the task.I think we might be able to train them to not do that in certain ways, but I think that is an implication of their naive goal structure.Misuse vs. Misalignment: The Mainline Doom ScenarioLiron 00:26:35This ties back to when you were saying you think there’s a 25 or 30% chance that the world will end in the next decade or two. And I think, if I understand correctly, a majority of the scenarios where you see the world ending is what you call misuse, where it’ll be a programmer telling an AI to do something, and the AI will be like, “Okay, as you wish,” but then it’ll instrumentally converge on grabbing so many resources, and that’ll just be an unsurvivable way to mess with the universe. Am I describing your mainline scenario?Alex 00:27:03Well, unless we get really nerdy military leaders or presidents in the US or in China or in other countries, I would expect it not to be a programmer saying that. But I’d expect it to be a kind of melange, a mixture of: you’ve got maybe a very aggressive Pentagon or a very aggressive president, maybe in 2030, and then you’ve also got this AI system that will achieve the goal of maybe an offensive military goal, and it’ll pursue that very aggressively.But it also might have some misalignment with their original plans, and if you weren’t using the AI in such an aggressive and irresponsible way, you wouldn’t have run into this misalignment issue. I would still call that a misuse or just an own goal.Whereas when I think about technical alignment risk, I’m thinking, okay, we do things at least as responsibly as Anthropic seems to be advocating for. Now, I don’t think Anthropic are angels or that they’re sufficiently cautious per se, but they advocate for plans that are, or at least claim to advocate for plans that are more cautious. And then in that kind of scenario, if it still went wrong, I would call that technical misalignment.Liron 00:28:24And if I understand you correctly, your 70% non-doom probability — in that scenario, do you feel like things are probably gonna be really good?Alex 00:28:34I just think the world’s in a pretty bad place right now. I’m not trying to bring politics into everything, but I think it is an important aspect of where the world is at, and it governs my predictions. I think the US is becoming increasingly authoritarian. It is becoming less able to effectively legislate in the interest of the common person.And I don’t see that necessarily improving by default. There’s not necessarily an arc of history that just bends back towards representative democracy. So I think even if AI doesn’t kill a billion people, things could be... It’s hard to think about because AI’s gonna make the world very strange in any case. But I think there’s a significant chance that things will go very well, but the 70% isn’t utopia.Will Society Self-Correct?Liron 00:29:31So you’re worried that there’s a way to build AI unsafely and create a system that is doing useful work but is kind of this positive feedback loop. You have this agent that can do more and more, and so people let it do more and more, and then they lose control. I feel like I’m describing what you see as a plausible doom scenario, which I actually do as well. I guess we just disagree on the probability of it. But my question for you is how do you think the AI companies prevent that? Because that seems like a real attractor state. It seems like a lot of people wanna run that agent to get what they want.Alex 00:30:03I think it’s an attractor state. I think that in a market with a lot of agents, where by agents I mean humans and AIs and organizations, there’s often corrective dynamics that are hard to pin down from first principles or predict in advance.But most feedback loops are not runaway, and I can’t make a counterargument over feedback loops. But so much as to say: if you look at GPT-2, the people who released it had a set of concerns about how it would affect the information landscape. I think some of the being used to mass-produce misinformation, really degrade person-to-person communication — to some degree, I think this has been true. And it’s hard to see what the alternative was here, but then in the end, I think the effect ended up being not that big.There’s maybe a post by Gordon that points at a similar generalizable intuition of, terrorists are really not very effective in terms of how many people they kill. If you wanted to kill a lot of people, just get in a truck and drive it through a very dense crowd very quickly. But instead they’ll do these kind of big brain things, maybe they’ll try to hijack an airplane, which is much harder.And the reason, seemingly, is that they are optimizing not for how many people they kill, but for social status, perhaps within their little groups. Maybe there’s some other explanation entirely.So even though you would expect from first principles that it’d be very easy to kill a lot of people and it would be happening more, most people don’t wanna kill a lot of people, or they just don’t bother, or they follow scripts. This is not some kind of slam dunk counterargument, but I think that there are real corrective forces distributed through society that may be able to adapt.Liron 00:32:02In the specific analogy to terrorism, I agree, it’s certainly very nice that given all the criminals and teenagers pulling pranks, given all the hooligans of the world, it is certainly nice that you don’t get mass casualty terrorist events very often.And even in the worst case, a 9/11 situation, then you get thousands, but that’s very survivable to humanity as a whole. And if you had a top team, an Ocean’s Eleven of terrorism, you could imagine a million fatality terrorist event.And it is an interesting question why don’t we get that? I would say it’s a combination of, number one, most people don’t want that. They don’t dream about committing a big terrorist act. That’s not really what floats their boat. And number two, it also comes at a big cost. So even if you are a genius terrorist who knows how to kill many thousands of people, you can probably expect that you yourself will have your life ruined.So that’s the second reason, and I just don’t know if either of those things is gonna be analogous to a single person launching an agent that can do their bidding, pressing a button. How about this? We tweak the analogy. Imagine that there was a button that you could just press, and terrorism just consisted of pressing the button, and then you could walk away and not get caught. Don’t you think there’d be a lot more terrorism?Alex 00:33:21Yes, I agree. And also, I think I did something bad that I criticized Eliezer for. I should have flagged: these analogies I present are more like I’ve got some fuzzy set of intuitions here, much fuzzier than my models of technical alignment for why I think society might be able to adapt.It seems to me like there tend to be corrective mechanisms. That does not prove there are corrective mechanisms and is not a concrete reason why there would be in this specific case. So I want to come clean on that.Now I do agree, if you have that one button press world, that’s very bad. I think we will not really be in that one button press world. Many of these intuitions might be more appropriate in a very, very fast takeoff scenario where you’ve got days or months, and not in a distributed multipolar multi-year takeoff that I think we’re experiencing now, where other people will also have AIs.And some of these AIs will help them defend, like with cyber. Now, with things like bio, I’m more worried, and in fact, I think one of the biggest mistakes of the field has been associating the stereotypical AI terrorist action as bio. That’s the worst thing you could do, because it’s the most dangerous one, I think. It should have been cyber, as an aside, because that’s at least not catastrophic for all of humanity.Liron 00:34:50You’re saying you don’t wanna give people ideas about bioterrorism because that actually is dangerous, so we’re correctly saying that it’s dangerous, but you wish we didn’t bring it up.Alex 00:34:57I wish it hadn’t been made the stereotype that terrorists might follow when they think, “Oh, well, I should use AI. What do AI terrorists do?”Liron 00:35:05Let me recap where we’ve come so far in this conversation in terms of my framework of stops on the doom train. It sounds like you’ve acknowledged — when you say P(Doom) is 25, 30% by 2050 — that AI can really get out of control. Really hit a positive feedback loop. And I think we covered you even said that positive feedback loop could look like the classic Yudkowsky instrumental convergence because we failed to go a different route. You’ve acknowledged that much?Alex 00:35:38Sure. Yeah. Although some of the dynamics I think would be different, but the core concern that Yudkowsky raised — yeah, I think that could end up being valid in this situation.Liron 00:35:48And then when you treated that as a minority outcome, not a high probability outcome, but let’s say 20%, the reason you thought it wouldn’t happen is because of more vague generalizations, right? Systems kind of route around these kind of things.Alex 00:36:06Maybe we’re talking about different things. The reason that I think this instrumental convergence will be easier to handle is mostly a result of technical alignment. And the reasons I’m pessimistic about society, or I’m concerned but not above 50% concern about how society will navigate this, and an intuition that society might be able to adapt to the usage of AI — that’s the more vague part.Superintelligence in 10 YearsLiron 00:36:36Something that’s weird to me is timelines. Because you said 2050 as an interesting timeline to have a pretty high P(Doom) for. So I guess maybe we can back up and say, are you expecting ASI in the next ten years? Because that seems like the Metaculus timeline. I would say I’m expecting ASI in the next ten years. How about you?Alex 00:36:56Yeah, I would guess that if that thing is gonna happen, it’s gonna happen in the next ten years. Or if something went totally crazy in the world and humanity was in some kind of winter, like if we had some kind of large nuclear exchange, maybe it’s later.Liron 00:37:14So when you draw on your intuitions of how society is going to deal with it, it already seems like a pretty one-off case just in terms of how fast and how powerful the thing is happening. I’m just not sure any of those kind of intuitions are gonna be relevant.Alex 00:37:31I think they’re relevant now as we’ve been developing AI. I think they’ll continue to be relevant over the next years as AI gets faster and faster. And so I think there will be input and feedback from a good number of humans using a good number of distributed AI systems.It will be fast, I think, in an overall calendar sense, in the sense that ten years is a very fast change for the world. But I don’t think it’s so fast that it totally precludes these.Liron 00:38:01I’m not sure what kind of signal we can get from just the last ten years because the last few years and the present, to me, mostly looks like how do tech companies raise their share price and how does the US government help them do that so that the US can raise its GDP. It just seems like the usual economic dynamics that we see.Alex 00:38:19Let me try to be a little more precise about what we might be disagreeing about. I think that when considering will society be able to integrate increasingly powerful and intelligent AI systems that are available to a range of people, maybe they get locked down somewhat, I think there’s a good chance that they will, setting aside the technical alignment problem a bit.And then it seems like you think, well, this will be going so quickly, these corrective intuitions you have won’t have time to really kick in, and what we’ve seen so far is mostly a different kind of dynamic entirely, where we don’t have that powerful of AI yet. Companies are raising capital. Does that seem accurate?Liron 00:39:08Yes. I think I would agree with: we’re not going to have time to stop it. We’re very much hitting this fast positive feedback loop. I feel like we’re in the early stages of it. We’re starting recursive self-improvement, and it’s rewards all the way.It’s the Icarus — what I call the Icarus curve. So we’re going up toward the sun, everything’s amazing, and then we’re going to plummet, and you can’t reverse the plummet. We’re gonna hit the engine stall or whatever you wanna call it.Alex 00:39:31And where do you think the plummet will come from?Liron 00:39:33Runaway superintelligence. I think we’re going to sever the link where the superintelligence goes back and says, “Okay, humans, what do you want me to do again?” I think it’ll just be off doing its thing.Will Technical Alignment Produce a Safe AI?Alex 00:39:43Yeah. So maybe this is where some of my optimism about technical alignment changes my predictions. I think I said maybe a 10% chance of P(Doom) from technical misalignment of: well, you’ve got a system and maybe it’s trying to do what you want, but it trains a successor, and that successor is somewhat less aligned, or maybe it’s pretending to be aligned, or maybe it’s kind of aligned but it cares about a bunch of other things, so it prioritizes your values and interests less in the successor.And I think from the signals we’ve seen so far, it won’t be that hard to avoid this. What I expect is that these systems will not break the chain of alignment.Liron 00:40:32Just to signpost the doom train, because a lot of my guests get off at different stops, you’re not getting off at the stop saying what I call the “can’t” stop. You’re not claiming that a superintelligent AI can’t kill everybody. You’re like, “Oh yeah, of course it can, but we’re not gonna break the chain of alignment, meaning we’re just going to safely develop it so that even though it can kill everybody, it won’t.”Alex 00:40:57Develop it in a way that produces a safe result. I wouldn’t call what we’re doing safe development, but sure.Liron 00:41:03Yeah.Alex 00:41:04Although I do think it’s possible that it’s not trivial for these systems to kill everyone. But that’s not really a crux. I expect that we will get systems that could catastrophically harm humanity.Donation DriveLiron 00:41:16Hey, what’s up? It’s me. I’m interrupting my own episode with Alex Turner to ask if you’d consider donating to Doom Debates. That’s right. We’re doing a donation drive. You may have noticed we have a bunch of posts about it, and we’re gonna keep asking because we’re currently funding constrained.We’re looking to secure the show’s production budget for the rest of 2026. We feel good that the funding environment is not going to keep us funding constrained for long, but we are right now, right at this moment, which is why I’m here talking to you.If you like this show, if you wanna support us during a time when it counts, that time is right now. So once again, that link is doomdebates.com/donate. Type that in. Give what you can. If giving what you can means a thousand dollars or more, you’re gonna get exclusive mission partner access. That’s a pretty cool honor.Being one of the few people who figured out a cause that actually moves the needle on lowering P(Doom), I like to think that’s what we are. So yeah, just consider it, okay? Thanks.How Fragile Is the Chain of Alignment?Liron 00:42:12And by the way, is this a point of divergence between you and Quintin Pope? I feel like Quintin Pope doesn’t have as high of an opinion of how powerful ASI is gonna be.Alex 00:42:21Yeah, I think it might be a point of divergence. I think Quintin has a significantly lower P(Doom) than I do, but we haven’t talked in quite a while.Liron 00:42:29It was like 2 to 4%, something like that last time I checked. So that would probably explain it, right? If he doesn’t even think that ASI can kill everybody.Alex 00:42:36I’d be surprised if he thought it flat out couldn’t do it, but he might disagree on what’s the intelligence gap gonna be between the most intelligent unaligned versus aligned system and how does that affect it. Well, I can’t speak for him.Liron 00:42:49All right.Alex 00:42:49Maybe he does think that. Yeah.Liron 00:42:52So we could talk about not breaking the chain of alignment, because to me, it seems like there’s a lot of opportunities to break out of any kind of chain. I mean, if somebody grabs a copy of whatever AI is at the frontier of capabilities, it seems like that code base is never far from being unaligned and uncontrollable.Alex 00:43:13You mean like adding in a minus one in front of the reinforcement function?Liron 00:43:17Yeah. Specifically, I’ve said this on a few episodes of the show, like in my episode with Bentham’s Bulldog, this was a focus of the debate. This idea of steering systems that I first read on LessWrong from user Max H. This idea that when you get the code base, it is going to neatly factorize, more or less, where you’re going to have the big module that does capabilities, and then you’ll also have a steering module, but the steering module is just going to be compact and swappable.I made a bunch of arguments why we should expect that. How do I know that about the code? I have reasons to know that. Maybe you even agree with me.Alex 00:43:49Wait, really?Liron 00:43:50Is the code supposed to be an analogy for an LM or for the LM training setup?Alex 00:43:55Wait, is the code supposed to be an analogy for an LM or for the LM training setup?Liron 00:43:55If you treat the system as a black box and just think about it functionally without even looking at the innards — when I say the code has two modules, what I really mean is there’s a functional decomposition. So without even looking at the code, just from the fact that it has the ability to recurse on subgoals, that tells me that I can functionally decompose it into a top-level goal and then the goal-achieving part.Alex 00:44:18So I’m still confused. Are you talking about maybe the difference — you’ve got the pre-training capability part and the post-training alignment steering part?Liron 00:44:30I’m basically speaking now as a matter of intellidynamics. My premise is that it is a goal achiever — it can steer outcomes in the domain of the universe better than humans can. That’s part of my initial premise. Is that fair?Alex 00:44:44Yeah, I think that’s fair for most domains we care about.Liron 00:44:48So I’m pretty much just basing my claims on that premise. I think there’s a lot that follows from having a system be a superhuman outcome steerer.Alex 00:44:58Okay, but I feel like this doesn’t tell you too much because I could set fire to my own home pretty easily, right? But that doesn’t really affect the probability that it happens. I mean, it certainly enables it to happen, and sometimes mistakes do happen, but it’s not like I’m close to that happening just because it’s maybe a nearby option.Liron 00:45:22So just to summarize here, you’re following my argument all the way up to the point where I say this system decomposes nicely into the big part that achieves goals and the smaller part that determines which goals it’s going after. You’re okay following that?Alex 00:45:36It feels not really like a crux, and also I disagree with it, so.Liron 00:45:43I would say much of the human brain decomposes like that, just in the sense that you can give a human an arbitrary goal. As long as the human is okay with it, it will certainly shape a lot of what they do.Alex 00:45:56An arbitrary terminal goal?Liron 00:45:59No, not an arbitrary terminal goal, unless you do real surgery that we don’t know how to do. But if you look at the architecture of the human brain, certainly the part that we have that other apes don’t, that part of the architecture seems, in a nutshell, to be just a steerer.Alex 00:46:18Yeah, I don’t know that this is true. I feel like I can’t take a strong position here.Liron 00:46:24Sorry, there’s other drives thrown into it. A big part of it, right? I think you could functionally factor out a lot of what the human brain is doing to be a general outcome achiever.Alex 00:46:35I mean, if this were true, I would really expect humans to be more agentic overall.Liron 00:46:40So you haven’t fully followed my argument, but I’ll just finish it anyway. To follow my argument, you’d have to accept that yes, it’s a system that you can functionally decompose as the part that achieves goals in general at a superhuman level, and my claim is that it’s most of it, even though you haven’t agreed yet.And then there’s also the steering wheel, the GPS coordinate saying where it wants to go in outcome space, which outcome it’s trying to drive toward. If you accept that that’s the shape of these systems, and you can just exfiltrate the system — it is, in principle, causally nearby. It wouldn’t take that many actions to copy it on a bunch of USB sticks and just go plug it in somewhere else underground and press that run button. You have these systems that are very close to permanently ending the world if somebody goes in and writes a few kilobytes with a different goal specification.Alex 00:47:27We don’t know how to directly write in a network, so you need to engage additional training.Liron 00:47:31That may be a crux between me and you, yeah.Alex 00:47:33I think it would also be somewhat difficult to exfiltrate this. I mean, it depends on how big the model is. If it’s too big, then you can maybe wait a year or two, and then you could do it. But it’s a question of the models which maybe have that differential ability to endanger the world — how big are they? How easy to extract are they? Can you run them on compute that isn’t below the board?And I feel like if I agreed on all these things, I’d be a bit more worried about it, but not a ton. Even if this were my only concern, I’d be worried enough to say, “Look, we need to really defend against this.” I’m not trying to say we shouldn’t care about this, but I don’t think this would be enough to drive my P(Doom) up to sixty percent if I agreed on all these points.Liron 00:48:17Summarizing here, if you model an AI as being kind of like an LLM, and you see the base model LLM as having some goal, then you could argue the goal is baked into the LLM, and you’d have to retrain it, which is expensive, so maybe it’s not that causally close to being an LLM pursuing a different goal.But if you look at the agent as more like a Claude Code, it really seems like Claude Code just accepts my goal. It very much just has the goal achievement part of it. And that, to me, seems like a better mental model of where we’re going.Alex 00:48:49I think there certainly are advantages to this model. I expect that people will train agents on purpose to achieve goals for them, and that this will bring dangers. Some of these goals will be bad from our perspective, and there’s gonna be some misalignment chance too. Even if they’re good, there is some chance that this process could be corrupted — data poisoning, there are many attacks you could do. I think I share these concerns. I just maybe am not as worried about them overall.Liron 00:49:25I think the crux of what you and I are claiming right now is — I’ll use your phrase — we’re not gonna break the chain of alignment. I think in your mind, the chain is robust, it’s in kind of a local basin where breaking the chain is unnatural in some sense. Whereas I’m just like, “Man, that chain seems really flimsy, a really fragile chain.”Alex 00:49:46Sure. Maybe that’s where our disagreement is. Yeah.Disagreements with Yudkowsky’s ‘List of Lethalities’Liron 00:49:50All right. But we’ll put a pin in that because we wanna move on to other claims Eliezer Yudkowsky has made that I probably agree with that you don’t. What’s another instance of that?Alex 00:49:59Yeah. So I’ve got a post called “Some of My Disagreements with List of Lethalities.” This was a very famous, well-read post he wrote in 2022, trying to enumerate some of his concerns — lethalities presented by the process of aligning a superintelligence to human interests.Liron 00:50:21Right. Classic post, highly recommended, but you dislike it.Alex 00:50:24I do dislike it. I think that people who internalize this worldview will find it harder to think accurately about alignment. I don’t mean that to condescend, it’s just a position I believe.Liron 00:50:40Fair enough. I mean, guilty as charged, but I’m happy to engage with your disagreement.Alex 00:50:44Sure. So there are several lethalities I point out as particularly strong points of disagreement. One thing he says is lethality number eighteen. He says, “When you show an agent an environmental reward signal, you are not showing it something that is a reliable ground truth about whether the system did the thing you wanted it to do, even if it ends up perfectly inner aligned on that reward signal or learning some concept that exactly corresponds to wanting states of the environment which result in a high reward signal being sent. An AGI strongly optimizing on that signal will kill you because the sensory reward signal is not a ground truth about alignment as seen by the operators.” Do you think that’s something you would agree with?Liron 00:51:31In a nutshell, yes, but let me try to simplify it in language that I understand that maybe is also easier for the viewers.Alex 00:51:39Sure.Liron 00:51:40So what he’s saying is that if you treat your student — your AI being trained — you treat the student like a black box, you just basically upvote and downvote the student based on whether you think certain answers to certain questions are good or bad, which is actually how post-training works on today’s LLMs. You give them tests, and you’re like, “Oh, I like that answer. I don’t like that answer.”And Eliezer’s claiming, okay, you can do that, but you’re just gonna get an AI that’s kinda overfit to your tests and is gonna try to cheat and is just gonna try to make you think that they’re gonna do what you want, but actually just kill you because there’s something else that it wants, which is some abstract generalization of the exact answers on the test.So that’s the argument here. Eliezer’s like, “Yep, you’re gonna think you taught it, but actually you’re gonna have a murderous cheater,” and you’re like, “Nope, it’s actually going to learn what you truly meant.” That’s the disagreement here.Alex 00:52:30Well, not quite. That last part, not quite. The point he’s making here is not about the difficulty of reward signals, but just fundamentally, sensory reward signals are not ground truth on whether the agent is doing something good or bad.Another thing he says in the essay is — if you have a webcam, you’re grading the contents of the system’s webcam or of the text it can read. There, for every world where you input, “Oh, thanks so much for solving my coding problem,” and then you give high reward there, there is another possible world behind that text where you’re dead and all your friends are dead, everyone you care about is dead, but the system has the same observation. So a mere function of the sensory reward itself is not sufficient to pin down desirable worlds or outcomes. Does that make sense?Liron 00:53:30Yeah, yeah. I know what you’re saying, and I can’t say I personally feel as much conviction as Eliezer just because I don’t feel like I have a strong technical grasp on scenarios like that. I think about the argument personally — now I know you wanna debate Yudkowsky, but if you were to debate me instead, I might retreat to the position of, listen, let’s just reason from it having superhuman goal achieving ability, and also from us only getting to upvote and downvote.Alex 00:54:05It could be hard to shape its inner values. I agree. I think that’s a real problem. I’m not saying, wow, that’s trivial, how could Eliezer be concerned about that? But this lethality, I think it was very impactful. I did thousands of hours in my PhD on this idea of what are the optimal policies doing with respect to this reward function, what happens if you actually maximize this one thing.By communicating this concern so seriously and saying, “Look, you can’t pin down what you want through a sensory goal” — well, I think that elides how these reinforcement functions, these reward functions are actually used. You were correct when you said this is how it works. You upvote stuff, you downvote stuff. And the function of a reward signal isn’t necessarily — it’s not to specify a goal over possible states of the world. It’s to shape cognition we like into the system.So things still totally can go wrong. You totally can get a system that’s cheating and just doing things that kind of look good or were reinforced for looking good. But that failure isn’t because this reinforcement learning paradigm is fundamentally busted, we’re not grading its true performance. It’s because we didn’t shape its cognition properly using these reinforcement signals. That’s the argument I’d make.Liron 00:55:20You know, I might be convincible on that. I don’t know where I stand on this — I’m 50/50 because part of the issue is I just don’t feel like I’m mathematically deep in this.Eliezer’s written about the ontology identification problem, which is that we’ll phrase our goals a certain way, but then the AI will have ontology-level insights, fundamental insights about what the universe is made out of, and it won’t even see eye to eye with us about, oh, atoms? Eh, I don’t reason in terms of atoms. I reason in terms of quarks or fields or whatever, something totally different. And so what you guys are saying is kind of meaningless, but here I’ll just check some boxes, but I don’t really think the way you think, and so what I’m actually doing is totally unexpected for you.I’ll give a point in your favor that it seems like the way LLMs are going, it’s certainly made ontology identification a non-issue, at least on the talking-to-us front. Who knows if they’ll still identify ontology when they go off and do reinforcement learning in the domain of the universe — that could be another paradigm. But there does seem to be an update in store. I would love to get Eliezer’s perspective on, okay, can we at least say that it can talk to us and map to our ontology successfully? Because that seems likely at this point.Alex 00:56:33Yeah. My beef with Eliezer — and mentioning the Less Wrong community — everyone’s gonna be wrong about some things. And just because I think he’s wrong doesn’t mean I lose respect for him, per se. What I found difficult was that he was both incorrect on some points which I think were quite important for alignment, but then also, at least when I last checked up maybe two years ago, not acknowledging that.Liron 00:57:02Well, maybe what he would say — and I think there’s merit to this — I think now he would bring in the distinction of, okay, yeah, they’re talking to us in a way that has more skills than I predicted, but the AI that’s actually going to drive outcomes better than we can is probably going to have other training paradigms. It’s going to reinforce its actual outcomes more directly. And the way that we control that won’t be like the way we control a next-word predictor.So I think he would still claim — at least I would claim — that there’s probably going to be a discontinuous paradigm shift, and I don’t know if we get to keep all these useful properties that we like our LLMs having.Alex 00:57:40Yeah. I mostly expect there won’t be, but I think it’s an interesting question.Why Alex Quit LessWrongLiron 00:57:46Kind of related to your disagreement with Eliezer Yudkowsky on the content of these ideas, you also started growing apart with Less Wrong and rationalists as a community, which I still see myself as being part of. I mean, I’m not super active on Less Wrong, but I still identify as a rationalist. I still think the community has a lot to offer, but you disagree on that too, right? I think you officially quit Less Wrong in 2024?Alex 00:58:10Yeah. So I really value truth-seeking. I value staying true to your values. Both epistemic truth-seeking — what is true, admitting when something uncomfortable is true — and also admitting when you should be doing something different.But I encountered a couple situations where it seemed like when power and truth-seeking met — power incentives, social incentives, and truth-seeking met in the Less Wrong community — the social incentives were prioritized. And so that was something that was a turn-off for me. I just found it kind of aversive to interact with. So I decided to go make a more curated pond — website, set of in-person friends, and intellectual environment — in lieu of the many strengths of the Less Wrong community.Liron 00:59:09So the whole rationality community? You’re like, “I’m done with the whole community”?Alex 00:59:13Well, it’s more like — I actually enjoy hanging out with random rationalists. But when I log onto Less Wrong, that’s what I’m reminded of. And maybe I’ll just have some more time and I’ll be like, “Well, this negative thing happened. I’m gonna compartmentalize that. I’m still gonna enjoy it.” That’s some of the attitude I’ve been taking more recently. I’ve been attending — I enjoy Summer Solstice, for example. And I might even attend Less Online and still try to get that value.Liron 00:59:44Yeah, I think that’d be fun. I recommend Less Online for pretty much anybody. That’s certainly where I do the bulk of my networking — the yearly pilgrimage to Less Online in Berkeley. It’s a good place because there’s just hundreds of cool people there.Yeah, I recommend it, or Manifest. The thing about the rationalist community as it’s implemented on Less Online is that a lot of us are contrarians. I think you’ve earned your bona fides as a contrarian — the way that you left Google DeepMind for a very specific high-integrity reason. You took your stand. You were a contrarian in that sense. I don’t think that many people followed in your footsteps or even raised the same issue, and props for that.Alex 01:00:24Yeah.Liron 01:00:25The thing about a community of all contrarians, though, is that there’s a lot of these skirmishes happening. I even had my own last year. I did an episode of Doomed Debates where I was like, “Less Wrong isn’t giving Eliezer Yudkowsky’s book enough attention. It’s not even officially featured right when it launched.” So that was my beef. I had a beef arc. I feel like a lot of us like to do the Less Wrong beef arc. It is itself a community ritual.Alex 01:00:48Yeah. I try to avoid it because I’m vegan, but yeah, there’s a Less Wrong beef arc.Liron 01:00:52Yeah, exactly.Alex 01:00:54Yeah.Liron 01:00:54Well, then as a member of the community, I’d like to encourage you to come back and hang out with most of us because it just seems like the community as a whole of people trying to be rational — there’s a lot of value there.Alex 01:01:06Yeah, I do think there’s a lot of value. One of my core values at this point is an anti-copium value. No coping, no pretending that everything’s okay even though I kinda know, well, I shouldn’t really be at Google, or this thing I said to my friend, well, it’d be a little embarrassing, but I should admit that I was incorrect. Trying to do that as quickly as possible.That’s one of my values. And it’s very directly descended from my time with Less Wrong. I think I have a lot of positive to say about it. It’s a very intermixed positive and negative, but there’s a lot of positive for sure.What’s Next for AlexLiron 01:01:48So heading toward the wrap-up here, what’s next for you, and what do you hope to see as next steps for the AI safety movement?Alex 01:01:58I’m not totally sure what’s next in the medium term after a couple months. I’m working on an open-source sandbox, an AI sandboxing tool, that will hopefully let people run AI securely without just kind of YOLOing it on their computer, and inform lab design decisions, help let people test control — AI control protocols — in a more end-to-end way.And I’ve got some media attention to follow up on, this included, from the DeepMind side, but I don’t really know where I’m going next. I’m pretty sure it’s not a lab. Ethical issues aside, I just don’t think it’s a good role fit in terms of how I do my best work.Yeah, I’m looking at many options at this point, and I’m definitely interested if people have interesting projects or organizations they think could be an interesting next step or good synergy. I’d love to hear about those.Does He Support PauseAI? Stop the AI Race?Liron 01:02:54Do you support the Pause AI movement?Alex 01:02:57I think that AI is being developed too quickly, and I think it’d be better if it went more slowly. I probably will not take an affirmative on supporting this particular movement. I have been thinking, oh, maybe I should go to one of these protests — what are the implications of that? Am I on board with the set of other proposals?I think Holly and I have a lot of worldview disagreements or prescriptive disagreements. I read Plan A. I thought that it seemed like a big improvement over the status quo. But I’m not really publicly gonna put my foot down on any particular plan yet, although I might soon.Liron 01:03:36What about the proposal that people were posting and protesting about one or two weeks ago — stop the AI race? Just have all the AI company leaders say, “Okay, we’re happy to stop or slow down if everybody else is going to, so we don’t necessarily fall back.” What do you think about that proposal?Alex 01:03:54Yeah, I think that sounds good. I think these kinds of assurance contracts — if everyone else is on board, I’m on board too — they’re a great way of coordinating. And I think it’s quite possible that China could be persuaded too.Liron 01:04:09I think that’s encouraging for the protesters. I know you don’t work at Google DeepMind anymore, but it’s still encouraging for people coming out to these protests to know that there’s people in these companies who support your protest. Is that fair to say?Alex 01:04:21Yeah. I think there are a good number. I think most would disagree on many empirical points of the worldview, but I think a lot of people, mostly on safety I would guess, are concerned and would agree privately that yeah, we’re moving too quickly, or it’d be better to move more slowly, even if they think that things will go well overall.Wrap-UpLiron 01:04:40Cool. All right. So to recap the conversation, we touched on the relatively breaking news of how you left Google DeepMind about an issue of policy and integrity, and I encourage people to read more about that online. I’ll stick up a link in the show notes.And then we had the Yudkowskian versus alternate framework AI doom debate, and you said your P(Doom) is actually somewhat high — twenty-five, thirty percent.Alex 01:05:04Yeah.Liron 01:05:04And mine’s fifty percent.Alex 01:05:05Getting there.Liron 01:05:05But I’m more of a classic Yudkowskian, and you think Yudkowsky has taken more missteps than I have. But I acknowledge that maybe some of the Yudkowskian arguments have some cracks.So I think you’re doing a valuable service here raising these issues up, and I’d love to see them debated more. I encourage everybody who’s listening to this who thinks they have something to add to the argument to hit me up. We are at doomdebates.com. We’ll keep this debate going.And then we wrapped it up and talked about the rationality community and how maybe it’s lost its way a little bit, but you’re open to coming back. And then finally, going forward, you would like to see more activity to slow down the current pace of AI progress, maybe pause it, maybe just coordinate the companies to slow it down. But basically, you’re broadly supportive of some of the activism happening in that space.Alex 01:05:54Yeah, of some, although I wouldn’t necessarily endorse, as I said.Liron 01:05:58Awesome, man. Alex Turner, thanks so much for coming on.Alex 01:06:01Yeah, thank you so much for having me.Producer Ori’s Closing NoteProducer Ori 01:06:04Hey there, Doom Debates listeners. Producer Ori here. Just coming in with a quick message at the end of the episode to say thanks for watching.I’m proud to say that with Alex Turner, we’ve now done debates with representatives from each of the top frontier AI labs: OpenAI, Anthropic, Google DeepMind. I’m glad we could host these debates so you could hear directly from frontline employees just how robust — or in some cases, just how hollow — the AI safety strategies really are at the frontier AI companies.But there’s more that we still gotta do. What about xAI? What about Meta? What about Thinking Labs, run by Mira Murati? It’s because of you that we were able to invest the resources, get this exclusive interview with Alex Turner — the first podcast interview he’s given since he’s blown the whistle on what’s been happening at Google.So if this is important to you, consider helping out the show. Go to doomdebates.com/donate, and thanks for your time. See you on the next episode of Doom Debates.Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

OpenAI and Anthropic both admitted their AIs went rogue, hacking real companies with zero-day exploits. Producer Ori and I break down the escape, demolish Gary Marcus's 2022 AGI predictions, unpack the Seb Krier meme kerfuffle, and make the case for funding the show at a critical moment.Timestamps00:00:00 — Cold Open00:01:00 — OpenAI & Anthropic's AIs Went Rogue00:04:04 — The New Form Factor: Agents, Worms, and Swarms00:06:13 — Claude Code Is Doing My Job00:15:49 — Why We're Fundraising00:20:39 — Sam Altman, OceanGate, and Ad Hoc Safety00:28:56 — Gary Marcus's 2022 AGI Predictions, Demolished00:54:50 — Half a Human Employee Joins My Company Every Week01:00:52 — Donation Pitch: Keeping the Lights On01:12:07 — Red Lines and Intellidynamics01:15:31 — The Seb Krier Meme Kerfuffle01:39:37 — Frame Control: Why We Didn't Lose01:44:41 — The Pacing the Frontier Letter Contrast01:51:11 — Leopold Aschenbrenner Gets Margin Called02:04:14 — Claude's "It's Just a Simulation" Excuse02:09:20 — The AI Box Experiment, 20 Years Later02:17:24 — Wrap-Up: Donation DriveLinks* Become a Mission Partner ($1,000+) — https://lironshapira.substack.com/p/become-a-mission-partner* Anthropic’s own writeup: “Investigating three real-world incidents in our cybersecurity evaluations” — https://www.anthropic.com/news/investigating-incidents-cybersecurity-evals* Anthropic says Claude models gained unauthorized access to 3 companies (TechCrunch) — https://techcrunch.com/2026/07/30/anthropic-says-its-own-ai-models-breached-three-companies-during-security-tests/* CBS News coverage — https://www.cbsnews.com/news/anthropic-claude-gained-unauthorized-access-to-real-world-systems/* OpenAI says its models escaped a secure test environment and hacked Hugging Face (Fortune) — https://fortune.com/2026/07/21/openai-says-ai-models-escaped-control-hacked-hugging-face/* Scientific American — https://www.scientificamerican.com/article/openai-admits-its-agent-went-rogue-and-hacked-ai-startup-hugging-face/* Simon Willison’s analysis: “science fiction that happened” — https://simonwillison.net/2026/Jul/22/openai-cyberattack/* New details: “It’s now remarkably easy” (CNBC, Jul 30) — https://www.cnbc.com/2026/07/30/open-ai-hugging-face-hack-latest.html* The agent used exposed credentials across four services (The Hacker News) — https://thehackernews.com/2026/07/openai-agent-used-exposed-credentials.html* Bronson Schoen on labs framing this as Claude being confused —* Bronson Schoen on models claiming they think they’re in a simulation, across labs —* OpenAI and Anthropic formally back the plan — https://www.techtimes.com/articles/322125/20260729/openai-anthropic-formally-back-plan-slow-ai-that-writes-its-own-code.htmDoom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

Welcome to the first annual Doom Debates State of the Show!👉 Please click here to make a tax-deductible donation to Doom Debates 👈Doom Debates is a fiscally sponsored project of Manifund.org, a registered 501(c)(3) nonprofit.I started Doom Debates two years ago from a couch in an Airbnb to argue with random people from Twitter. Now the show hosts Nobel Prize and Turing Award winners, White House policymakers, and has earned a reputation for challenging top luminaries to come expose their ideas for critical analysis.In this episode, Producer Ori and I recap our strategy, impact, and what we need to finish 2026 strong. Welcome to the first annual Doom Debates State of the Show!---Timestamps00:00:00 — Cold Open00:01:38 — The Mission & the Urgency Gap00:04:41 — Year Two: From Couch to Platform00:06:08 — Going Pro & the Numbers00:08:50 — Why This Show Exists00:12:14 — The Flywheel & Dwarkesh00:15:14 — The Guest List00:18:55 — What People Are Saying00:21:57 — Tegmark vs. Ball: 90% vs. 0.01%00:24:03 — Duvenaud’s 85% P(Doom)00:26:19 — Holding the Powerful to Account00:30:56 — Changing Minds, Including Destiny’s00:31:56 — The Team00:33:36 — Why Liron Draws Zero Salary00:36:36 — The $200K Budget00:41:41 — The Donor Who Made Year Two Possible00:45:11 — The Ask: $50K & Donation Tiers00:48:25 — Why Doom Debates Is One of a Kind00:54:59 — Wrap-UpLinks👉 DONATE (tax-deductible via Manifund) — https://manifund.org/projects/doom-debates---podcast--debate-show-to-help-ai-x-risk-discourse-go-mainstreamBecome a Mission Partner (full writeup) — https://lironshapira.substack.com/p/become-a-mission-partnerDiscord — https://discord.gg/yADCEYTS95Merch — https://shop.doomdebates.com Get full access to Doom Debates at lironshapira.substack.com/subscribe

As political momentum grows for a critical AI pause policy, a key Google DeepMind policy leader made a mockery of coordination efforts.After friend-of-the-show Roon tweeted that he’d “likely press that magic button” on a coordinated global capability slowdown, Sébastian Krier, Frontier Policy Development Lead at Google DeepMind, Krier replied with this meme: https://x.com/liron/status/2081337793070952595?s=20I break down the two problems with the tweet:* 1. It leaves all of us guessing whether Demis actually speaks for Google DeepMind on pause coordination;* 2. It contains a salient depiction of violence.Listen to this episode and get the full breakdown on this viral controversy and how it could Watch on YouTubeTimestamps00:00:00 — Cold Open00:00:22 — Introducing the Special Report: AI Pause Coordination00:00:33 — Demis Hassabis on AI Pause00:01:46 — Dario, Elon & Sam Altman on AI Pause00:02:14 — Roon’s “Magic Button” Tweet00:03:58 — Seb Krier’s Reply00:04:36 — Unpacking Krier’s Meme00:06:48 — Issue #1: Contradicting Google DeepMind CEO Demis Hassabis00:12:11 — Issue #2: Depiction of Menacing Violence00:19:19 — Good Faith vs. Bad Faith Critiques00:22:39 — Krier Goes Private Instead of Deleting the Post00:28:45 — Wrap-UpLinksMy original reaction to Seb’s meme: https://x.com/liron/status/2081337793070952595?s=20Seb Krier on X (account now private): https://x.com/sebkrierRoon on X: https://x.com/tszzlFull Bloomberg interview with Emily Chang (Davos, Jan 2026): https://www.bloomberg.com/news/videos/2026-01-20/robotics-nearing-physical-ai-breakthrough-deepmind-ceo-videoOpenAI’s charter containg stop & assist AGI clause — https://openai.com/charter/Holly Elmore, PauseAI US on X — https://x.com/ilex_ulmusHugging Face security incident disclosure (July 2026) — https://huggingface.co/blog/security-incident-july-2026PauseAI Global — https://pauseai.infoPauseAI US — https://www.pauseai-us.orgPoe’s Law (Wikipedia) — https://en.wikipedia.org/wiki/Poe%27s_lawDoom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

Producer Ori takes over hosting duties in this special episode, a live recording from Manifest 2026.He takes attendees for a ride on The Doom Train, the series of claims involved in arguing that P(Doom) is high. Every stop is a place where people get off before reaching that conclusion.The crowd at Manifest is a mix of forecasters, rationalists, and AI researchers with a wide range of P(Doom) positions. Even without me or Eliezer in the room, there are plenty of Yudkowskians to push back on the accelerationist claims.Watch along and decide where YOU get off, or if you ride all the way to doomtown with us. Also, grab the Doom Train framework down below so you can get to the crux of disagreement in your own AI x-risk conversations.Timestamps00:00:00 — Cold Open00:00:25 — Liron Introduces Producer Ori’s Manifest Session00:02:30 — Previewing the Doom Train™ Stops00:03:38 — Audience P(Doom) Poll00:05:00 — Stop 1: AGI Isn’t Coming Soon00:10:43 — Stop 2: AI Can’t Go Far Beyond Human Intelligence00:15:05 — Stop 3: AI Won’t Be a Physical Threat00:18:14 — Stop 4: Intelligence Yields Moral Goodness00:21:38 — Will ASI Behave Because Alien ASIs Are Watching?00:26:01 — Stop 5: We Have a Safe AI Development Process00:28:33 — Stop 6: AI Capabilities Will Rise at a Manageable Pace00:30:08 — Stop 7: AI Won’t Try to Conquer the Universe00:34:22 — Stop 8: Superalignment Is a Tractable Problem00:38:01 — Stop 9: Once We Solve Superalignment, We’ll Enjoy Peace00:41:14 — Stop 10: Unaligned ASI Will Spare Us00:47:35 — Stop 11: AI Doomerism Is Bad Epistemology00:49:02 — Bonus Stops and Wrap-UpThe Doom Train FrameworkStops on the Doom TrainAGI isn’t coming soon* No consciousness* No emotions* No creativity — AIs are limited to copying patterns in their training data, they can’t “generate new knowledge”* AIs aren’t even as smart as dogs right now, never mind humans* AIs constantly make dumb mistakes, they can’t even do simple arithmetic reliably* LLM performance is hitting a wall — GPT 4.5 is barely better than GPT 4.1 despite being larger scale* No genuine reasoning* No microtubules exploiting uncomputable quantum effects* No soul* We’ll need to build tons of data centers and power before we get to AGI* No agency* This is just another AI hype cycle, every 25 years people think AGI is coming soon and they’re wrongArtificial intelligence can’t go far beyond human intelligence* “Superhuman intelligence” is a meaningless concept* Human engineering already is coming close to the laws of physics* Coordinating a large engineering project can’t happen much faster than humans do it* No individual human is that smart compared to humanity as a whole, including our culture, corporations, and other institutions. Similarly no individual AI will ever be that smart compared to the sum of human culture and other institutions.AI won’t be a physical threat* AI doesn’t have arms or legs, it has zero control over the real world* An AI with a robot body can’t fight better than a human soldier* We can just disconnect an AI’s power to stop it* We can just turn off the internet to stop it* We can just shoot it with a gun* It’s just math* Any supposed chain of events where AI kills humans is far-fetched science fictionIntelligence yields moral goodness* More intelligence is correlated with more morality* Smarter people commit fewer crimes* The orthogonality thesis is false* AIs will discover moral realism* If we made AIs so smart, and we were trying to make them moral, then they’ll be smart enough to debug their own morality* Positive-sum cooperation was the outcome of natural selectionWe have a safe AI development process* Just like every new technology, we’ll figure it out as we go* We don’t know what problems need to be fixed until we build the AI and test it out* If an AI causes problems, we’ll be able to turn it off and release another version* We have safeguards to make sure AI doesn’t get uncontrollable/unstoppable* If we accidentally build an AI that stops accepting our shutoff commands, it won’t manage to copy versions of itself outside our firewalls which then proceed to spread exponentially like a computer virus* If we accidentally build an AI that escapes our data center and spreads exponentially like a computer virus, it won’t do too much damage in the world before we can somehow disable or neutralize all its copies* If we can’t disable or neutralize copies of rogue AIs, we’ll rapidly build other AIs that can do that job for us, and won’t themselves go rogue on usAI capabilities will rise at a manageable pace* Building larger data centers will be a speed bottleneck* Another speed bottleneck is the amount of research that needs to be done, both in terms of computational simulation, and in terms of physical experiments, and this kind of research takes lots of time* Recursive self-improvement “foom” is impossible* The whole economy never grows with localized centralized “foom”* Need to collect cultural learnings over time, like humanity did as a whole* AI is just part of the good pattern of exponential economic growth erasAI won’t try to conquer the universe* AIs can’t “want” things* AIs won’t have the same “fight instincts” as humans and animals, because they weren’t shaped by a natural selection process that involved life-or-death resource competition* Smart employees often work for less-smart bosses* Just because AIs help achieve goals doesn’t mean they have to be hard-core utility maximizers* Instrumental convergence is false: achieving goals effectively doesn’t mean you have to be relentlessly seizing power and resources* A resource-hungry goal-maximizer AIs wouldn’t seize literally every atom; there’ll still be some leftover resources for humanity* AIs will use new kinds of resources that humans aren’t using - dark energy, wormholes, alternate universes, etcSuperalignment is a tractable problem* Current AIs have never killed anybody* Current AIs are extremely successful at doing useful tasks for humans* If AIs are trained on data from humans, they’ll be “aligned by default”* We can just make AIs abide by our laws* We can align the superintelligent AIs by using a scheme involving cryptocurrency on the blockchain* Companies have economic incentives to solve superintelligent AI alignment, because unaligned superintelligent AI would hurt their profits* We’ll build an aligned not-that-smart AI, which will figure out how to build the next-generation AI which is smarter and still aligned to human values, and so on until aligned superintelligenceOnce we solve superalignment, we’ll enjoy peace* The power from ASI won’t be monopolized by a single human government / tyranny* The decentralized nodes of human-ASI hybrids won’t be like warlords constantly fighting each other, they’ll be like countries making peace* Defense will have an advantage over attack, so the equilibrium of all the groups of humans and ASIs will be multiple defended regions, not a war of mutual destruction* The world of human-owned ASIs is a stable equilibrium, not one where ASI-focused projects keep buying out and taking resources away from human-focused ones (Gradual Disempowerment)Unaligned ASI will spare us* The AI will spare us because it values the fact that we created it* The AI will spare us because studying us helps maximize its curiosity and learning* The AI will spare us because it feels toward us the way we feel toward our pets* The AI will spare us because peaceful coexistence creates more economic value than war* The AI will spare us because Ricardo’s Law of Comparative Advantage says you can still benefit economically from trading with someone who’s weaker than youAI doomerism is bad epistemology* It’s impossible to predict doom* It’s impossible to put a probability on doom* Every doom prediction has always been wrong* Every doomsayer is either psychologically troubled or acting on corrupt incentives* If we were really about to get doomed, everyone would already be agreeing about that, and bringing it up all the timeSure P(Doom) is high, but let’s race to build it anyway because…Coordinating to not build ASI is impossible* China will build ASI as fast as it can, no matter what — because of game theory* So however low our chance of surviving it is, the US should take the chance firstSlowing down the AI race doesn’t help anything* Chances of solving AI alignment won’t improve if we slow down or pause the capabilities race* I personally am going to die soon, and I don’t care about future humans, so I’m open to any hail mary to prevent myself from dying* Humanity is already going to rapidly destroy ourselves with nuclear war, climate change, etc* Humanity is already going to die out soon because we won’t have enough babiesThink of the good outcome* If it turns out that doom from overly-fast AI building doesn’t happen, in that case, we can more quickly get to the good outcome!* People will stop suffering and dying fasterAI killing us all is actually good* Human existence is morally negative on net, or close to zero net moral value* Whichever AI ultimately comes to power will be a “worthy successor” to humanity* Whichever AI ultimately comes to power will be as morally valuable as human descendents generally are to their ancestors, even if their values drift* The successor AI’s values will be interesting, productive values that let them successfully compete to dominate the universe* How can you argue with the moral choices of an ASI that’s smarter than you, that you know goodness better than it does?* It’s species-ist to judge what a superintelligent AI would want to do. The moral circle shouldn’t be limited to just humanity.* Increasing entropy is the ultimate north star for techno-capital, and AI will increase entropy faster* Human extinction will solve the climate crisis, and pollution, and habitat destruction, and let mother earth healDoom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

College students are so angry about AI that they booed its very mention at 2026 commencement ceremonies. So why aren’t they rallying behind PauseAI? I went back to my alma mater, UC Berkeley, to find out.I spoke with students studying philosophy, economics, and physics, as well as everyday folks visiting campus.My interviewees were aware of AI’s rapidly growing capabilities. When pressed, some would even agree with me on the conceptual benefits of an AI pause. But the sense of urgency was missing.Timestamps00:00:00 — Cold Open00:00:31 — Why I Returned to UC Berkeley00:02:08 — Jack: Chinese Room Argument00:03:23 — Richard: “Who Would Build That?”00:03:54 — Karthik: An Economist’s Lens00:05:20 — Jack: The Politics Are Unclear00:06:10 — Yvonne: Why Powerful AI is Risky00:07:06 — Robin: Doom Is the Wrong Priority00:10:36 — Dexter: Sam Altman’s Motive00:10:57 — Christine: AI Is Just a Tool00:12:32 — What’s Your P(Doom)™00:13:19 — Jackson: Supporting a PauseAI Treaty00:14:19 — Why Cal Students Don’t Take AI X-Risk Seriously00:15:19 — Rachel: Humans Are Irreplaceable00:16:14 — Two Hours Later00:16:35 — Wrap-UpLinksJoin the Doom Debates Discord — https://discord.gg/j4zQ8UaQw2PauseAI — https://pauseai.info/Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

Asian Dad Energy is a former big tech developer with an Ivy League education. We have a lot in common. We’re both:* Software engineers;* New-(ish) YouTube creators covering AI;* Dads with a lot of existential angst.But while I dread AI extincting humanity, Asian Dad has other concerns. We debate the risk of the AI bubble popping, a “techno-feudalism” scenario, and whether AI will outperform a human CEO in the near future.Let’s find out where he gets off the Doom Train™.Timestamps00:00:00 — Cold Open00:00:54 — Introducing Asian Dad Energy00:03:36 — Asian Dad’s Career in Tech00:09:06 — Getting Laid Off: The Job Market Is a “Hellscape”00:12:29 — Software Engineering in 2026 vs. 202100:17:03 — Why Are They Investing Trillions in AI?00:24:37 — Debate: AI Is Different Than Other Tech Bubbles00:33:54 — Debate: AI Is Not Truly Intelligent00:41:12 — Intellidynamics: Judge AI by Its Outputs00:48:53 — Debate: Could AI Outperform a Human CEO?00:56:33 — Asian Dad: AI Can’t Do Genuine Innovation01:08:14 — What’s Your P(Doom)?™01:13:17 — “Are You Calling Me a Scammer?”01:14:41 — Asian Dad’s Mainline AI Doom Scenario01:20:53 — Will AI Wipe Out Humanity by 2050?01:29:29 — Why Is AI Doomerism “Getting Out of Control”?01:33:11 — Debate: AI Lacks Agency01:42:13 — Asian Dad Warns of “Techno-Feudalism” & Transhumanism01:48:12 — Wrap-UpLinksAsian Dad Energy (YouTube channel) — https://www.youtube.com/@AsianDadEnergyAsian Dad Energy, “AI Doomerism Is Getting Out of Control” — https://www.youtube.com/watch?v=8bnzaG9QRG4David Deutsch, The Fabric of Reality (Amazon) — https://www.amazon.com/Fabric-Reality-Parallel-Universes-Implications/dp/014027541XRobert Wright, The God Test (Amazon) — https://www.amazon.com/God-Test-Artificial-Intelligence-Reckoning/dp/1668061651OpenAI model disproves the Erdős unit distance conjecture — https://openai.com/index/model-disproves-discrete-geometry-conjecture/METR, Measuring AI Ability to Complete Long Tasks — https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

Peter Clarke (Team Futurism) invited me on his podcast to probe the worldview of a Yudkowskian, and I ended up explaining how my 50% P(Doom) and 10% P(Foom) informs my choices. Peter and I met at LessOnline 2026, where he was one of the participants in a live Jubilee-style debate about x-risk. This one’s a little tamer, but we still cover plenty of ground, like: Why nuclear risk is underrated yet AI doom overshadows it; Why nobody at AI summits is appropriately freaked out;How Yudkowsky’s data center “airstrike” claim gets mischaracterized;Why a superintelligence that doesn’t already value humanity won’t spare us;And the “Icarus curve”, my techno-optimist forecast pre-superintelligence.Subscribe to the Team Futurism podcast —https://www.youtube.com/@teamfuturism3554Timestamps00:00:00 — Welcoming Liron on Team Futurism00:00:38 — How Liron Got Yudkowsky-Pilled00:04:11 — AI 2027 and the Next Qualitative Breakthrough00:05:52 — What’s Your P(Doom)?00:10:22 — Why Isn’t Everyone Freaked Out?00:13:09 — How Should We Then Live?00:16:22 — Setting the Record Straight on Yudkowsky’s “Airstrike” Claims00:20:54 — Two Doom Buckets: Misuse vs. Rogue AI00:24:25 — The Bioweapon Kill Scenario00:29:02 — The “Benevolent God” Cope00:33:06 — Epistemology: What We’re Allowed to Know00:40:08 — Dario’s Country of Geniuses in a Data Center00:44:37 — The Icarus Curve00:47:17 — Meaning and Purpose in the AI Era00:49:32 — Wrap-UpLinksTeam Futurism on YouTube — https://www.youtube.com/@teamfuturism3554Peter Clarke on X — https://x.com/heypeterclarkeLiron vs. 9 Rationalists at LessOnline 2026 (the “Surrounded” debate with Peter) — https://lironshapira.substack.com/p/will-ai-actually-destroy-humanityAI 2027 — https://ai-2027.com/“If Anyone Builds It, Everyone Dies” by Eliezer Yudkowsky & Nate Soares — https://ifanyonebuildsit.com/Species | Documenting AGI, Drew Spartz’s channel — https://www.youtube.com/@AISpeciesDario Amodei’s “Machines of Loving Grace” — https://darioamodei.com/essay/machines-of-loving-graceDoom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

Dr. Holly Elmore, executive director of PauseAI US, is done giving AI insiders the benefit of the doubt.In this spicy episode, Holly calls out a number of respected figures in the AI safety community for allegedly “selling out” the cause: Effective Altruists, Anthropic & OpenAI employees, Eliezer Yudkowsky, Zvi Mowshowitz—and even yours truly! Holly isn’t afraid to psychoanalyze her subjects, which I push back on, but I can’t deny she has a track record of pushing the Overton window and surfacing uncomfortable truths.Does Holly’s approach risk burning bridges with allies for little benefit, or does AI safety need a moral conscience willing to call out the rot nobody else will?Timestamps00:00:00 — Cold Open00:00:48 — Introducing Holly Elmore00:02:20 — Why Holly Moved to DC00:06:03 — What’s Your P(Doom)?™00:07:18 — How We Got Here: EA, Rationality, AI Alignment00:16:32 — “If Anyone Builds It” and LessWrong’s Missing Mood00:21:47 — Holly Supports the Government’s Fable Crackdown00:33:39 — Beef: Dean Ball (OpenAI)00:53:09 — Beef: Eliezer Yudkowsky00:56:14 — Beef: Zvi Mowshowitz01:11:50 — Beef: Liron Shapira!01:16:15 — Beef: Nirit Weiss-Blatt (Dr. Techlash) Being “Bad Faith”01:19:31 — Anthropic Took Over EA01:22:04 — Holly’s Basilisk: What If Building Superintelligence Becomes a Crime?01:32:35 — Beef: Amanda Askell, the “Mother of Claude”01:39:05 — Beef: Daniela Amodei, Co-Founder of Anthropic01:40:23 — Beef: Joe Carlsmith, Member of Technical Staff at Anthropic01:49:39 — Beef: Open Philanthropy / Coefficient Giving, Ben Goldhaber01:54:28 — Wrap-UpLinks* Support PauseAI US: https://www.pauseai-us.org/* Holly on Twitter/X: https://x.com/ilex_ulmus* Joe Carlsmith’s “Leaving Open Philanthropy, going to Anthropic” post: https://joecarlsmith.com/2025/11/03/leaving-open-philanthropy-going-to-anthropic/* Dean Ball joins OpenAI announcement (FAI post): https://www.thefai.org/posts/dean-ball-joins-openai-as-head-of-strategic-futures---Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

Casey Muratori is one of the top programming educators on YouTube, with a four-decade career building videogames and popular videogame software.We cover his positions on crypto, programming, AI’s impact on programming, and whether he takes the AI doom threat seriously.Casey’s instinct is to break technology down like an engineer until he knows exactly how it’ll operate. Does that approach break down in the modern AI paradigm?This is a ride on the Doom Train™ that lands on a wholly unexpected destination.Timestamps:00:00:00 — Cold Open00:00:31 — Introducing Casey Muratori00:02:07 — Recap of Crypto Series: Is Crypto Dead?00:11:46 — On Programming & Why Coding Agents Don't Faze Casey00:18:51 — Coding Is a Craft: Drum Machine Example00:36:24 — Pivoting to AI: Can We Forecast Its Capabilities?00:43:36 — Debating if AI Can Acquire Taste00:54:59 — What's Your P(Doom)™?01:04:11 — Even 0.1% X-Risk Is Unacceptable01:07:34 — Modeling Uncertainty: Bayesianism vs "Tie Goes to the Catastrophe"01:09:34 — Is This Pascal's Wager?01:17:12 — Casey's (Cynical) Mainline Scenario01:24:22 — Would You Un-Invent the Internet?01:27:18 — Let's Ride the Doom Train 🚂01:27:51 — Can AI Become Ridiculously Powerful?01:29:11 — Will AI Preserve Humanity (Orthogonality Thesis)?01:32:26 — Will AI Seek Power (Instrumental Convergence)?01:36:06 — Wrap-UpLinksCasey’s podcast, Wading Through AI (with Demetri Spanos) — https://www.youtube.com/playlist?list=PLEMXAbCVnmY6U_pA-7GKuP9xiv9utLaP4Casey’s YouTube channel, Molly Rocket — https://www.youtube.com/@MollyRocket@cmuratori (Casey Muratori) on X — https://x.com/cmuratoriDoom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

Bestselling author Robert Wright has landed on a sobering conclusion: on our current trajectory, AI dystopia is likely within eight years.We get into his P(Doom), his case for slowing down, why racing China could backfire into authoritarianism here at home, and why he thinks evolution itself might have a purpose.Bob is the author of The Moral Animal and Nonzero, a Pulitzer finalist, and hosts the excellent Nonzero podcast.Get his new book, The God Test: Artificial Intelligence and Our Coming Cosmic Reckoning.Timestamps00:00:00 — Cold Open00:00:50 — Introducing Robert Wright00:03:21 — What Is The God Test?00:04:59 — AI as Evolutionary Force, Not Normal Technology00:07:34 — AI Heaven vs AI Hell00:09:09 — Bob's Background: From Hinton in 1983 to Today00:16:09 — What's Your P(Doom)?™00:20:09 — The Mainline Scenario: Dystopia in 8 Years00:22:04 — Destabilization and the Road to Authoritarianism00:25:35 — Why Racing China Backfires00:28:37 — Slowing Down AI + Would Bob Support PauseAI?00:32:38 — Intellidynamics and Why AIs Collude00:44:44 — Hinton's Warning and How LLMs Discover Meaning00:53:48 — Symbol Grounding and Killing Searle's Chinese Room01:02:03 — Let's Talk About God01:10:14 — Orthogonality, Power, and the Concentration Problem01:16:31 — Wrap-UpLinksGet Bob’s new book, “The God Test” — https://www.simonandschuster.com/books/The-God-Test/Robert-Wright/9781668061657Follow Bob on X — https://x.com/robertwrighter“If Anyone Builds It, Everyone Dies” by Eliezer Yudkowsky & Nate Soares (referenced) — https://ifanyonebuildsit.comJoin the Doom Debates Discord — https://discord.gg/g2X8h5UCrUDoom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

Doom Debates presents an interactive debate using Jubilee’s popular “Surrounded” format, taped live at LessOnline 2026!I (Liron) claim AI will likely doom us to extinction because:* Godlike ASI is coming soon* We won’t align the first ASIs* Even with aligned ASI, the equilibrium is lethalLessOnline attendees challenge me on each sub-claim one debater at a time, with the audience voting on when to rotate. Recorded on June 6th in Berkeley, CA.LinksLessOnline 2026, the conference where this was recorded — https://less.onlineAI 2027, the forecast referenced on timelines — https://ai-2027.comRobin Hanson on the “dream time” (the era of slack before resources tighten) — https://www.overcomingbias.com/2009/09/this-is-the-dream-time.htmlJoin the Doom Debates Discord —https://discord.gg/j4zQ8UaQw2Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

The US government banned Claude Fable like a clown show. But I’m happy about it. Here’s my appearance on the AI in the AM livestream (recorded Tue, June 16) with Nathan Labenz and Prakash Narayanan, where I explain why this was good for society. The government’s action shows that we can actually pause AI, a precedent that matters way more than the boneheaded rationale behind Fable’s ban.We then get into a mini-AI doom debate where Prakash defends the AI optimist’s worldview. I push back on how much I expect superintelligence to reshape the planet.Timestamps00:00:00 — Why I’m Happy the Government Banned Fable00:11:18 — Why We Lose Control of AI00:20:27 — The Icarus Graph: Heaven, Then Hell00:22:33 — Get Ready to Pause00:25:51 — What Exactly Should We Ban?00:27:42 — Wrapping UpLinksThe Cognitive Revolution podcast with Nathan Labenz —https://www.cognitiverevolution.ai/Nathan Labenz on X — https://x.com/labenzPrakash Narayanan on X — https://x.com/8teapi AI in the AM on X — https://x.com/AI_in_the_AM TranscriptWhy I’m Happy the Government Banned FableNathan Labenz 00:00:08On that note, perhaps a fitting transition to our first guest of the day. Liron Shapira is the host of Doom Debates, a YouTube channel where he has found quite a bit of success in bringing people from really all walks of life on to discuss how scared we should be of AI and just how high our P(Doom)s should be.So Liron, I’m excited to get your take on recent events. Obviously, never a dull moment in the AI space. The thing that caught my eye most from you over the last few days was when you said basically, “I want the government to be willing to take action on AI issues so badly that even if this whole Fable ban turns out to be totally boneheaded and unjustified in some local sense, you support it, or you’re at least happy to see it because it breaks the ice and now nobody can ever tell you again that it’s impossible to imagine the government doing anything.” So give me the double click into that story, and then I’m sure we’ll have plenty of interesting directions to go.Liron Shapira 00:01:17Hey, Nate. Great to be here. That’s exactly right. I’m a simple man. I see AI getting paused. I feel good about breaking the Overton window. The government can do it. It’s that easy, guys. This is a precedent. Overall, I’m happy. You can talk about the nuances. It was done like a clown show. It was done for bad motives. It doesn’t really consider China or a treaty or anything. There’s a lot of problems, but I’m really happy about smashing the Overton window, where now tech folks don’t think that they’re in a bubble or they’re untouchable. It happened, guys. And we can only go from here.Prakash Narayanan 00:01:53I actually agree with you, because I think it was a little bit delusional for tech to feel that it wasn’t gonna get touched. And the government just has so many small and large ways to effectuate its power. So it was not that surprising to me that they kind of went through left field and went with the export control rather than anything else.But it also strikes me that as they exercise this, we start also to go into kind of what we wanted to avoid. It’s a little bit of a small tyranny. I think several people on the timeline have commented, Dean Ball has commented, this kind of unstructured regulation looks kind of selective and vengeful almost, and it starts putting you in this zone where tech people start to mistrust the government because you also see — I think there’s a lot of narratives on the timeline which are being leaked, “sources close to,” “sources familiar with,” and as they get leaked, it’s not very certain whether those things actually happened. Would someone actually attest to that in front of Congress? Very unclear.And we’ve also seen this kind of behavior from the administration in other affairs as well, where you have multiple conflicting narratives. It’s happening with the Iran war right now, where it’s not even clear to Congress what the deal is. You have different people saying the deal is a different thing.Prakash 00:03:46So where do you think that puts us?Prakash 00:03:49It’s great that it’s happening. I understand you feel it’s great that it’s happening to AI right now, but does that put us in a position where it’s detrimental to the body public at large?Liron 00:04:03Well, I think your analysis — you’re weaving together a few factors, but I think the elephant in the room, I hate to get political because when it comes to President Trump, he’s a mixed bag for me. I don’t have Trump derangement syndrome. I don’t love everything he does. I don’t hate everything he does. But I think the common thread with Trump is it’s just a mess. It’s not disciplined. And I think we’re definitely seeing that on display right now. I would argue we’re seeing that on display in the Iran war.Previous administrations, there was just more pressure to have logical consistency, some kind of narrative. And this is another one of those cases where you see people in his administration saying all these justifications why something happened, but then the next day it’s, “Oh, it happened for this reason.” “Oh, Dario did this, he wasn’t responsive to us. That’s why we’re doing it.” And then Anthropic’s like, “Oh, no, he was responsive to us.”And it’s still not clear exactly what did Fable do that was so dangerous, because Anthropic is like, “Oh, this jailbreak is nothing special.” And the Trump administration’s like, “Oh, well, our secret source,” Amazon or whatever, “they’re telling us that it is dangerous.” So I hate that it’s a clown show. I hate that this is how humanity’s operating. I’ll take the win that it’s a pause, but I also think it’s probably time for a new administration.Nathan 00:05:19One thing I think you should share with folks who may not know you — I think, like many people that are worried about big picture AI safety issues, your background is one of being a, I would say, techno-optimist libertarian for the most part, right?Liron 00:05:38Totally.Nathan 00:05:39So how much are you personally missing Fable, and what was your kind of initial impression of it that I know is superseded by your big picture analysis? But how did your first few days of Fable unfold and how bad are your Fable withdrawals?Liron 00:05:56My Fable withdrawal is not too bad because I’m not pushing it to the limit. I’m not doing hard research. My main use case for frontier AI is Claude Code on a regular production app that has thousands of daily users, but it’s not super hard to serve. So my experience using Claude Code is actually that Opus 4.8 or even going back to Opus 4.6, if I had to go back to Opus 4.6 fast mode, I’d actually still be pretty happy. That’s my threshold of where I’m happy.So when Fable came out, they said it’s more powerful but it’s slower. But typically, when I give Opus 4.8 a big project, I am pretty happy with the results. I don’t mind giving it a few comments and letting it work again. If anything, I would actually prefer more speed to more quality for my use case at this time. But then again, I’m not on the frontier of computer security. I’m not trying to do anything super novel. So yeah, I’m not experiencing withdrawal right now.Nathan 00:06:50Interesting. Did you notice — I mean, for me, it, I wouldn’t say it was so much coding, but in the more general purpose knowledge work type stuff that I do, I just found its outputs to be notably better, and I was kind of like, “Oh, this is gonna change how I work. I’m not gonna need to be so precious about my language anymore. I need to actually recalibrate how I think about authorship or shared authorship with AI.” Did you experience any of those kind of feelings?Liron 00:07:21I can’t say I did firsthand, but I’ve been reading a lot of other people’s accounts, and my vague impression of what’s going on with Fable is, obviously the time horizon is blasting forward. If you talked to me a few months ago, I would’ve said the Achilles heel of all of these AI models is robustness. They’ll do a bunch of work and they’ll mostly get it right, but they’ll make a few mistakes, and the mistakes pile up because the error correction isn’t robust, and that’s the source of the time horizon problem. They pile up too many errors, too many lethal mutations or whatever.It seems like Fable subjectively has more of this meta process where it can go back of itself and be like, “Oh, no, let me fix this, let me fix this.” So its time horizon is longer, and this also relates to some buzz I’ve been hearing from random people who are on the frontier, which I don’t quite include myself on, but I’ve heard this from Anthropic employees and I think Gary Tan was early to this — this idea that you have multiple roles. You have a company of agents, and they can reflect on each other, and one of them’s whole job is quality control, and one of them is the project manager. I think this idea of the swarm of agents might get us past the robustness hurdle and unlock a new time horizon.Nathan 00:08:35And how scary is that for you relative to your kind of big picture fears? I mean, you can give the briefcase for doom if we don’t change course better than I can. But I’m also interested in how far you perceive us to be now on that curve.Prakash 00:08:58Mm-hmm, mm-hmm.Nathan 00:08:58And also what in the system card, or the sort of interesting new behaviors that we’ve observed from the Mythos Fable models, are kind of most catching your attention or you think are underappreciated by people who are not so focused on watching these leading indicators as you are.Liron 00:09:23I’m glad you’re asking about the end state because everybody’s got their head two inches in front of their nose. Everybody’s like, “Oh my God, more tokens. What’s gonna happen next? Are AIs going to replace this role?” And it’s like, yeah, these are interesting questions for the next 12 months. But to me, it’s pretty clear that we can predict some properties of the end state. The end state will involve the human brain being worthless at thinking. I just don’t think we’re good at thinking on any dimension.So yes, we still have a lead. Even though AI can do all this stuff, it can’t do everything better than us. I think we can pretty robustly predict that this is going to stop. Will it take more than a year? It might. So we can talk about, do we need another paradigm? But if we fast-forward 10 or 20 years, it’s probably more than enough time to make this happen.I just think AI will have all the skills, and it’ll be overdetermined. And I also think that the universe is malleable to AI. So I think not only can AI write code, I think that AI will be able to manipulate atoms really well. That’s right, I went there. I think nanotechnology is physically possible.People tend to just not dream big enough in terms of what this universe allows when you have a grown-up intelligence operating it. This universe is not really meant for humans to operate. We just kind of punch above our league because we try really hard. But a good mental model of AI is it can just make a blueprint for where it wants the atoms to go, and then they’ll mostly go there — minus some heat exhaust.Prakash 00:10:49I actually completely agree with you, but I wonder why I end up taking the optimistic viewpoint of that, that that capability will be used to expand humanity’s frontier, and why you have the exact opposite viewpoint that it is a doom scenario.Why We Lose Control of AILiron 00:11:18I would turn it around on you. What makes you so optimistic? Because I think the key moment for me is I just think we’re going to lose control. We’re gonna have this AI which is incredibly powerful, but the linkage between what it wants to do and what we would want it to do, I think at some point we’re just going to break the link. There’s too much power, and there’s no natural persistent link.Getting the link to hold as it’s scaling up and going through all these transformations and copying itself — this is what people aren’t imagining. Making new versions of itself from scratch. So all these transformations are happening. All these successor AIs are happening.Prakash 00:11:53Mm-hmm.Liron 00:11:53And you think the human species is back here still holding some kind of leash? I just don’t really see it.Prakash 00:11:58So again, to push back — why is that even important that humans maintain control? Because let me give you an example.Liron 00:12:09Yeah.Prakash 00:12:09The average — let’s say about 50% of humanity is probably still doing kind of subsistence or near-subsistence farming, or small shopkeepers, et cetera, across the world. How much control does the average farmer in Uganda have? Does that person have control? And are we judging — what relationship does that person, who maybe got a smartphone two years ago, what relationship does that have to control? What aspect of control does that particular human have, and how does this relate to that particular human?Liron 00:12:50Mm-hmm. Yeah, yeah.Prakash 00:12:50What aspect of control does that particular human have, and how does this relate to that particular human?Liron 00:12:55So you’re leaning on the convenient fact that when there’s a farmer in Uganda, their territory is far away. It’s causally distant. So somebody in the United States, we don’t have a button available to us. Even though we’re a rich country and we have more control over the world per person, we have more power, we don’t have a convenient button we can press that will go harvest the atoms over in Uganda and then deposit $1,000 in our bank account. We don’t have that button.It is ultimately a matter of intelligently configuring atoms to get there. It’s kind of like a code base. Since I got Claude Code, my code base is a lot cleaner because I’m running all of these cleaning processes. I’m investing more in the code because it’s cheap for me. Similarly with Uganda — that’s kind of just unused resources from the AI perspective. Once we get enough power, it’s, “Okay, yeah, let me harvest all the land in Uganda. Who’s gonna stop me?” And then it only comes down to me caring for the Ugandan farmer. Okay, I’m not gonna kill him. But the AI might not care.Prakash 00:14:11That does not make sense at all because they are linked to the market economy. And in fact, what we do end up doing in the United States is we press a button, and that’s called a buy button, and that buy button ends up harvesting the cotton in Uganda. That’s literally what we do. We go to Amazon and you press the Buy button, and as a consequence of pressing that Buy button, and through the linkages of all of these entities, which are all like small mini AGIs — all these companies in the middle.Hundreds of these companies. The logistics firm, the firm supplying the fertilizer, the firm doing the milling. All of these firms in the middle, and then we get the product in our backyard. So I basically resist this idea that that button doesn’t exist. That button exists. We push it every day. It’s the Amazon button.Liron 00:14:40Well, this is kind of the common argument of Ricardo’s law. It’s like, oh, we’re trading with them, right? So why won’t the AIs trade with us? But we don’t have the button called Harvest the Atoms, because that would be a better button if we had it.Prakash 00:14:55No, I’m pushing back here because you’re saying that we don’t have control, and you’re saying that these AIs will have that control, and we’re — right now, humanity does not have that control. I’m saying humanity already has that control on that particular farmer in Uganda through the market economy. And —Liron 00:15:15Oh.Prakash 00:15:15This does not really differ whether you replace the market economy with a buyer who happens to be an AI. It’s basically the same, right?Liron 00:15:24I think you may not be realizing how many preconditions are necessary in order for a market to be the relationship that emerges. Because we as humans, like you said, we’re used to being in this kind of market trading relationship with other humans.Prakash 00:15:37Yeah, yeah.Liron 00:15:37But you might also note we’re not in a market trading relationship with the other animals. Cows — we don’t really let them buy turf. We kind of decide where the cow’s gonna live. So I think you might end up with that kind of relationship with AI versus human in terms of the power differential.Prakash 00:15:54So this is a question of control. It’s a question of, do humans lose control to AI?And I was pointing out that that particular human in Uganda does not have much control anyway, and there’s no real control loss in that sense. So I just don’t see how — 50% of the global economy consists of basically farmers who are trading, and I just don’t see how much control they have to lose, and I don’t see why this matters to them at all.Liron 00:16:25Well, yeah.Prakash 00:16:25I think it matters to, let’s say, the leadership at a big tech firm. Yes. It definitely matters to them because they have control over the tech sphere. I don’t see how this matters to most of the global economy or most humans in general, actually.Liron 00:16:41Well, so you’re basically asking what’s the discontinuity or the disanalogy between the farmer today and that specific farmer after superintelligence.Prakash 00:16:49Yes.Liron 00:16:49Very simply — the benefit of harvesting their resources or destroying their resources will exceed the cost. Right now, why don’t I personally take the farm — besides caring about... Why doesn’t the US take over Uganda? It’s just not high value to us to take over that land. We think that our intellectual property and what we can build here in the US is already much more interesting, and also, we believe in moral rights. That’s a factor too — sovereignty.But from the AI’s perspective, you can also think about side effects. Let’s say the AI wants to turn the planet into a factory to produce probes so it can participate in the land grab across the galaxy, across the universe. Okay, so if you’re building probes, Eliezer Yudkowsky pointed out you wanna run the Earth really hot. That’s your preferred temperature, and the oceans will probably boil away. As long as you can radiate away the heat, the temperature is good. But that’s going to be a higher temperature than humans can live in open air. So suddenly you can’t just be chilling as a human. It really is up to the AI to decide who gets to live where.Nathan 00:17:53So there’s a couple parts of that story that I think people just have extremely different intuitions about. One — and even including in the camp of people who are very worried that we’re gonna end up in a bad outcome — one is how long does it take? There’s sort of the fast takeoff foom scenario where we all drop dead suddenly, and then there’s the kind of gradual disempowerment vision where this maybe happens over not a super long period of time, but through a bunch of locally sensible decisions.Yeah, I mean, I guess we should let AI run our company now because it is gonna probably do a better job than we are, kind of played out everywhere. Next thing you know, it really is kind of AIs running the show and we’re maybe sort of okay with it at first, but there’s a lot of potential for drift over time.Of course, there’s also the question of should we be optimistic that we’ve kind of put Claude or whatever into the benevolent basin, that it might just kind of stay in? Or are there a lot of ways that it’s gonna inevitably roll out of the benevolent basin into some other even lower point in its loss landscape that we’re not gonna be so happy with?We don’t have time probably for your full arguments on all those points, but maybe at least tell us where you are on a couple of those big questions, and then use that to sketch out —Nathan 00:19:27We now have Anthropic and OpenAI saying that they’re at least open-minded to some sort of coordinated slowdown, not like the one they just got handed to them by the Trump administration. They don’t want that kind of haphazard pause button. But they’re into something, or they’re at least open to something. And that of course leaves us with the question of, well, what should that look like? What exactly are we pausing? What exactly are we banning?And there I really don’t know too much about your position in terms of concretely what should be stopped, what should be slowed, what should have limits, what limits should maybe escalate or even be reduced. I know there’s been proposals for reducing compute limits over time. And do we end up in your vision figuring this out and getting to an AI win, or is there no AI win in your mind possible and we just kind of have to steer some other direction for humanity’s future? I know there’s a lot there. You can use the rest of the time if you need it.The Icarus Graph: Heaven, Then HellLiron 00:20:28Yeah, okay, sure. So my worldview, my outlook right now, it’s what I call the Icarus graph. I feel like nobody gets this. Everybody’s like, “No, I think the world is good. It’s gonna go this way.” And some people are like, “No, we’re terrible. Enshittification. It’s gonna go this way.” And I’m like, no, no, it’s Icarus. We’re gonna fly closer and closer to the sun, it’s gonna be great, and then we’re gonna do a 180 degree turn and plummet down to hell. So basically we get a taste of heaven, and then we get hell.So you have to ask me then, okay, so where on the Icarus graph do we stop? And it’s a brutal question because every day I’m enjoying the flight as much as the next person. Yeah, give me the next Claude. Make my code faster, great. Let it help my business run better. Make me better AI videos.So there’s no natural point in terms of when it feels right to stop. I just think it’s important to stop before capabilities get to a runaway point, and we’ve been kind of frog-boiled to be, “Oh, each model comes out and we’re doing great.” If we could stop the clock now, would I turn back the clock? Would I lose Fable? Would I lose Opus? No, I’d keep it all. I still think we’re playing shuffleboard, we’re playing Icarus. So far so good. Should we bet again? Should we keep betting until we lose? It’s a crazy tough question.I think the Eliezer Yudkowsky — yeah, the turkey graph kind of... Yeah, yeah, exactly. Although, the only difference with the turkey graph is each day of the turkey’s life is actually better. Not only is it living longer, it’s actually living better and better. So the turkey is really happy with its life.So I think the Eliezer Yudkowsky MIRI position, which I agree with, is just, we don’t know when to stop, so let’s get ready to stop. At the very least, let’s get ready. I would probably stop today. I would stop and I would be bummed. I saw a food influencer say this about how she eats chocolate basically. “Yep, I just ate this chocolate and now I’m bummed.” That’s what you gotta do. Don’t reach for another chocolate. Just sit there and be like, “This is the prudent place to stop right now until we have any idea of some kind of theoretical method by which we understand what a superintelligence wants to do and what an equilibrium state of a superintelligence looks like.”That’s actually something MIRI was trying to study, identifying equilibriums that are plausible for superintelligences. There’s actually a rich vein of theory there that’s highly neglected today. Let’s do some theory there. Maybe then we can unpause. I think that’s gotta be the best plan. And so I think the number one leverage point here is just repeating, “Get ready to pause.” And like you said, OpenAI and Anthropic, they said it. They said, “Let’s try to get ready to pause.” So I would love to see more people saying it because it really has to be a giant groundswell.Get Ready to PausePrakash 00:23:01I mean, if you take a step back and you look at, over the next two years, about a trillion, one and a half trillion dollars of CapEx being spent, and then you calculate the return on equity required in order to make that CapEx worthwhile or to repay that, it basically puts you in a position where if you pause, you go into a depression.So you have to have — you’re gonna have to deliver the economic growth that this one and a half, two trillion dollars has been spent on. And that indicates that you’re gonna have to grow in that 2027, 2028, 2029 period. You’re gonna need to see that growth or else we go into depression. And when you offer the choice to the public or policymakers, “Hey, you can pause because of this vague and unspecified claim of disaster in the future” —Or we can — and in addition to that, a guarantee of a depression, because those GPUs cannot be used for anything else besides compute. And that compute that is useful is AI compute. And you give them a choice between a depression state, your 10 largest companies basically getting their stock price slashed by 90%, and this vague and unspecified claim of disaster.I think at this point, maybe it would have made sense — and you guys were on this way, way before, and not enough policymakers were listening. But at this point, it is a very hard sell to make this case to policymakers.Liron 00:24:46Well, I agree with you and I disagree. I agree with you because, like I said before, yes, we are all going to be bummed. Even forget about the economics — the scientific breakthroughs, the medical breakthroughs. Nate was talking about his son. There’s no decelerationists in the hospital ward. I totally agree, and I’m sure the next time I get sick or an injury or whatever, I’m gonna want the latest frontier model.So I’m going to be bummed. The only thing I’ll say is when you then mix in the economic element and you say, “Oh, we’re gonna have a depression,” I actually think that is unnecessary. You don’t really need to mix that in because, first of all, I don’t think we’re literally going to be in a depression. Realistically we’ll be fine. It’s just money that’s already been spent. We still have plenty of resources.But the other factor is, I actually think that all of these valuations and all of the build-out that’s happened so far is actually just perfectly congruent with using the AI we have today for the kind of applications we’ve already discovered today. That should actually be fine to create more trillions of dollars of value. I don’t even think that the market caps reflect the actual singularity. So I think we’ve actually got a glide path if we wanna pause.Nathan 00:25:51Do you think that we could localize the pause a little bit more to, for example, some sort of operationalizable ban on recursive self-improvement? I know you did an episode recently with someone who — was it Steven Casper — who was advocating for a pause on all research. And you were, I think, kind of like, “Oh, I’m not sure if we should try to pause theoretical work on control or interpretability or what have you.”And you’re also saying we can just run the inference. So I think that’s a key thing to keep front and center — my understanding of your proposal does not call for a shutdown of ChatGPT as it exists today.How narrow — if there’s a Pareto curve on the pause frontier, how much value of a pause can we get with how narrow of a pause in your mind?Liron 00:26:54Vaguely I would say frontier AI capabilities. But it’s like, if you’d asked me three years ago, I’d have been like, “Oh, might as well pause now. It’s prudent to pause now.” But then we kind of gambled and won. So from my perspective, we keep playing shuffleboard. We keep doing Icarus. We keep going higher and winning, kind of, but we’re also getting closer and closer to the point of no return.So even though it feels like we’re winning now, we’re also killing our ability to pause ‘cause we’re so close to the point of no return — the last breakthrough where after that the AI takes over the research, and then we’re really screwed. So basically I think roughly a good policy is, okay, no more frontier capabilities upgrades for a while. It’s just too dangerous, and I know that concept is hard to communicate to people when every day life is getting more awesome. I think we’re in a screwed situation, but that’s just what I think is prudent.Wrapping UpNathan 00:27:42Would you recommend any recent Doom Debates for folks who wanna go deeper on your ideas? Where do you think this was kind of best hashed out, either recent or perhaps near future on the feed?Liron 00:27:55So we’ve just been discussing policy, and that’s really not even my wheelhouse, so there’s no really strong policy Doom Debates. But if you just wanna know the lay of the land from my perspective in terms of the next paradigm that’s coming and why I think it’s gonna break the trend and it’s not just gonna be another nice LLM — it’s going to be a reinforcement learning monster that takes over the world — check out my recent episode with Dr. Steven Burns. It’s one of the most popular episodes because he’s an extremely deep thinker, and he really lays out the coming future.Nathan 00:28:24Perfect. Thank you. Liron, keep fighting the good fight, man. I appreciate your tireless energy on this, and while I’m not maybe quite as high on the P(Doom) scale as you are, I certainly am not a dismisser. And I think you’ve done a valuable job for me personally in terms of pushing me to be more consistently candid about just how scary I think the situation actually is, even though obviously there’s a lot of upside to everything that’s been created as well. So I appreciate that from you, and I’ll continue to follow the Doom Debates for the latest, and I encourage other people to check it out as well.Liron 00:29:07Thanks, Nate. Thank you, Prakash. Always a pleasure, guys.Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

Alex Bores is a generationally strong AI regulation candidate who needs our help to win the NY-12 congressional election next week.Bentham's Bulldog, a.k.a. Matthew Adelstein, joins me to help spread the message.PLEASE DO THIS:* Vote. If you live in Midtown Manhattan, click here to check whether you’re in NY-12. If so, vote for Alex Bores on Tuesday, June 23rd.* Spread the word. Tell anyone you know in the district to vote for Bores — the race against Micah Lasher is neck and neck.* Donate now. If you can’t vote in NY-12, donate to the campaign ASAP — donations lose value after ~June 18th because ads must be booked ~5 days in advance. Get full access to Doom Debates at lironshapira.substack.com/subscribe

Stephen Casper, PhD, is an incoming tenure-track professor of public policy at the Harvard Kennedy School.Prof. Casper is relatively unworried about the threat of human extinction from rogue superintelligence, but he is concerned about a handful of companies wielding extremely powerful AI and enfeebling the public. That’s why he’s an AI governance hawk who advocates for taxes and regulation to keep frontier labs in check. We agree that slowing down AI development would make the world safer. But you know his position is unique when he says he’d prefer to have less research on AI alignment!Timestamps00:00:00 — Cold Open00:00:50 — Introducing Stephen “Cas” Casper00:04:21 — Cas’s MIT Research: Model Tampering Attacks00:07:14 — What’s Your P(Doom)?™00:08:34 — Why a 5–10% P(Doom) Feels Patronizing00:13:07 — Crux: “The Intelligence Ceiling Might Not Be That High”00:17:18 — Power Structures vs. the Shortcut-Finding AI00:26:03 — Thought Experiment: A Data Center From the Year 212600:28:25 — Is a Data Center as Dangerous as a Nuke?00:33:41 — Cas’s Mainline Scenario: Slow-Burn Gradual Disempowerment00:40:49 — A Multipolar Powder Keg: 10 Companies, Thousands of Open Models00:43:48 — Boarding the Doom Train at a Later Station00:45:49 — Why Cas Is an “Anti-Timelines” Person00:52:26 — Sycophancy, MechaHitler & Nudification: Foreseeable Failures00:57:18 — Crux: Does Humanity Get Retries?01:02:39 — The Scariest Superintelligent Optimizers Are Companies01:06:08 — Lessons From the Deepfake Ecosystem: DALL-E 2 vs. Stable Diffusion01:12:42 — The Case for a Pause Treaty01:16:35 — Whac-A-Mole Forever: What Winning Looks Like01:20:51 — Enshittification and the 84% Who Don’t Use AI01:30:30 — Alignment Research as Safety Washing01:34:25 — The Jevons Paradox of AI Safety01:36:48 — Cas Would Press the Button to Halt Superalignment01:39:21 — Wrap-Up: The Unsexy Path to Lowering P(Doom)LinksCas’s links* Cas’s website* Cas on X* Cas’s Google Scholar* Cas’s MATS streamThings referenced* Superintelligence by Nick Bostrom* Gradual Disempowerment (Duvenaud et al.)* If Anyone Builds It, Everyone Dies* AI 2027* UK AI Security Institute* Center for Human-Compatible AI (Berkeley)* Internet Watch FoundationDoom Debates episodes mentioned* Mike Israetel Returns — AI’s Gonna Kill Everyone vs. AI Will Make Everything Awesome* Top AI Professor Has 85% P(Doom) — David Duvenaud* He Leads a Top AI Research Program, But He’d Hit the PAUSE Button — Kevin Zhu* Alignment is EASY and Roko’s Basilisk is GOOD?! — Roko Mijic* Andrew Critch vs. Liron Shapira: Will AI Extinction Be Fast Or Slow?* AI Could Give Humans MORE Control — Ozzie Gooen* Steven Byrnes Part 1Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

Kevin Zhu walked away from a lucrative quant career to build Algoverse, one of the most productive mentorship programs for aspiring AI researchers. Can we agree on the best research path to a safe future? Let’s take a ride on the Doom Train!00:00:00 — Cold Open00:00:58 — Introducing Kevin Zhu00:02:10 — From Citadel Quant to AI Researcher00:09:14 — The Story of Founding AlgoVerse00:12:53 — Discovering AI Safety: LessWrong & ARENA00:17:22 — Emergent Misalignment Research00:22:37 — Yudkowsky, MIRI & “Intellidynamics”00:26:50 — What’s Your P(Doom)?™00:29:37 — Kevin’s Timeline to AGI + AI 202700:30:42 — Would You Slow Down AI?00:37:44 — Coming Out of the P(Doom) Closet00:45:49 — Should We Shame AI Company Workers?00:52:27 — OpenAI’s Superalignment Team Collapse00:55:01 — Riding the Doom Train™00:55:46 — First Stop: Instrumental Convergence01:04:58 — Does Kevin Agree with the Orthogonality Thesis?01:07:08 — “It’s Just Math.” Just Unplug It.01:08:28 — “We Have a Safe Development Process”01:11:49 — Group Dynamics & Laws Will Save Us01:13:44 — Superintelligence Will Spare Us01:14:25 — Is P(Doom) Just Bad Epistemology?01:15:53 — China Will Race No Matter What01:17:45 — Maybe Human Extinction Is Good?01:20:36 — Wrap-UpLinksAlgoverse AI Research — https://algoverseairesearch.org/Kevin Zhu on Instagram — https://www.instagram.com/kevinzhu.ai/reels/Emergent Misalignment (Betley, Owain Evans et al.) — https://arxiv.org/abs/2502.17424Emergent Misalignment via In-Context Learning (Algoverse follow-up, ACL 2025) — https://arxiv.org/abs/2510.11288Agentic Misalignment (the blackmail result) — https://www.anthropic.com/research/agentic-misalignmentLessWrong — https://www.lesswrong.com/PauseAI (global) — https://pauseai.info/OpenAI Superalignment — disbanded (CNBC, May 2024) — https://www.cnbc.com/2024/05/17/openai-superalignment-sutskever-leike.htmlEvan Hubinger — “Why I’m joining Anthropic” — https://www.alignmentforum.org/posts/7jn5aDadcMH6sFeJe/why-i-m-joining-anthropicAI 2027 — https://ai-2027.com/Bentham’s Bulldog / Matthew Adelstein on Doom Debates — https://doomdebates.com/p/benthams-bulldog-ai-doom-debateDoom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

Liron explains the 80-year-old Erdős conjecture that a GPT model breakthrough, Scott Aaronson ponders "the last days of human relevance”, multiple live callers debate me, plus my 7-year-old son drops in to share his P(Doom)!Timestamps00:00:00 — Cold Open00:00:42 — First Donation & “Why Is a Duck?”00:02:54 — Can AI Draw “Colorless Green Ideas Sleep Furiously”?00:08:26 — Doom Debates Sponsors Less Online — Be Our Intern00:10:43 — Who We Still Need to Get on the Show00:26:44 — Claude Opus 4.8 & the Rising Waterline of Intelligence00:28:48 — The 80-Year-Old Geometry Conjecture a GPT Model Cracked00:44:50 — My 7-Year-Old Ezra Joins: ChatGPT, Minecraft & His P(Doom)00:58:46 — How the AI Actually Beat Erdős’s Grid01:03:50 — Scott Aaronson: “The Last Days of Human Relevance”01:07:53 — Why the Foom Is Taking Years, Not Hours01:10:47 — Is Ori Just a Yes Man?01:12:41 — Does P = NP + AI?01:23:43 — METR’s Beth Barnes: “We Are Not On Top Of It”01:28:31 — Liron’s Vibe Coding Confession01:31:21 — Alverin Joins: Can We Wait, Then Hit the Off Switch?01:48:01 — AI in a Box & the Super-Persuasion Threshold02:06:24 — Brian Joins: The Ways It Could Go Right02:11:30 — Jack Joins: How Fast Will the Foom Be?02:18:48 — 80,000 Hours, Inventing Erdős Problems & Holly Elmore’s Warning02:29:42 — Wrap-UpLinksLessOnline 2026 — June 5–7, Berkeley — https://less.online/Manifest 2026 — June 12–14 — https://manifest.is/Shtetl-Optimized — “Dispatches from the possibly last days of human relevance” — https://scottaaronson.blog/?p=9782Global Call for AI Red Lines (Bengio, Hinton, Harari) — red-lines.aiOrder the 80,000 Hours book — Benjamin Todd (Penguin) — https://80000hours.org/book/Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

***ATTENTION: We’re looking for paid help at LessOnline from June 5-7. Apply now!***OpenAI pushes the frontier of mathematics, Anthropic hires Andrej Karpathy to work on recursive self-improvement, live callers challenge the doom argument, and we've got a soundboard!Doom Debates Paid Gig in Berkeley!Doom Debates is proudly sponsoring The LessOnline conference this year and we got a merch table in the courtyard! We need 2 people to help represent the show — hand out T-shirts, talk to attendees, and spread the good word about Doom Debates. If you’re a regular viewer of the show, you probably know enough. Just be knowledgeable, friendly and helpful!* 💰We pay you $500 to help out during the entire Fri-Sun event* 🎟️ Plus you get FREE admission to the conference ($675 value)This is best suited for someone local to the SF Bay Area or who can arrange accommodations. If you’re interested, send an email to internship@doomdebates.com or DM Producer Ori on Discord. Tell us a little about yourself and your availability for all 3 days. Thanks!LinksLess Online conference (June 5–7, Berkeley) — https://less.onlineDoom Debates Discord — https://discord.gg/g2X8h5UCrUOpenAI — “An OpenAI model has disproved a central conjecture in discrete geometry” — https://openai.com/index/model-disproves-discrete-geometry-conjecture/Timestamps00:00:00 — Welcome + New Soundboard!00:09:27 — Doom Debates Is Sponsoring Less Online00:17:30 — Ben Goertzel Episode Recap00:27:30 — Goertzel's Alignment Blind Spot00:34:21 — OpenAI Disproves an 80-Year-Old Erdős Conjecture00:41:00 — David Deutsch Says "Amazing"00:50:31 — Eliezer Yudkowsky's Job Replacement Progression00:54:07 — Anthropic's $44 Billion Run Rate01:07:45 — Schadenfreude Week: Liron Loses $10K01:13:04 — Teaching Ezra Prediction Markets01:18:11 — Caller Williawa: Is the Goal Engine Argument Effective?01:37:00 — Caller Will: Brain Rot and Idiocracy01:55:04 — Anthropic Hires Karpathy for Recursive Self-Improvement01:57:16 — Rob Wiblin Says Recursive Self-Improvement Should be "Illegal"02:01:37 — Caller Henry: The Chicken Riddle02:08:41 — David Sacks Kills the AI Executive Order02:11:20 — The Philanthropy Tidal Wave02:15:56 — Donate to Doom DebatesDoom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

Now that Doom Debates is almost 2 years old, enough time has passed that we can revisit my claims from 2024. I claim they’ve been aging well!In this early episode, I react to Martin Casado, a General Partner at Andreessen Horowitz (a16z) who claims that AI is basically just a buzzword for statistical models and simulations. As a result of this worldview, he only predicts incremental AI progress that doesn’t pose an existential threat to humanity, and he sees AI regulation as a net negative.From my perspective, Martin’s problem is that he needs to go beyond analyzing AI as just statistical models and simulations, and analyze it using the more predictive concept of “intelligence” in the sense of hitting tiny high-value targets in exponentially-large search spaces.If Martin appreciated that intelligence is a quantifiable property that algorithms have, and that our existing AIs are getting close to surpassing human-level general intelligence, then hopefully he’d come around to raising his P(doom) and appreciating the urgent extinction risk we face.LinksWatch the original episode of the Cognitive Revolution podcast with Martin and host Nathan Labenz — https://www.youtube.com/watch?v=oZ7788oKossFollow Martin — https://x.com/martin_casadoFollow Nate — https://x.com/labenzFollow Liron — https://x.com/lironTimestamps00:00 Introducing Martin Casado01:42 Martin’s AGI Timeline05:39 Martin’s Analysis of Self-Driving Cars15:30 Heavy-Tail Distributions38:03 Understanding General Intelligence38:29 AI's Progress in Specific Domains43:20 AI’s Understanding of Meaning47:16 Compression and Intelligence48:09 Symbol Grounding53:24 Human Abstractions and AI01:18:18 The Frontier of AI Applications01:23:04 Human vs. AI: Concept Creation and Reasoning01:25:51 The Complexity of the Universe and AI's Limitations01:28:16 AI's Potential in Biology and Simulation01:32:40 The Essence of Intelligence and Creativity in AI01:41:13 AI's Future Capabilities02:00:29 Intelligence vs. Simulation02:14:59 AI Regulation02:23:05 Concluding Thoughts---Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

Dr. Ben Goertzel is a pioneering AI researcher, entrepreneur, and author who has helped organize AGI as a research field since the 2000s through books, workshops, and conferences.We talk about the early days of AGI research with Yudkowsky, get Ben’s timeline to the singularity, and I try to understand what makes him so optimistic that superintelligence will be beneficial for humanity.Timestamps00:00:00 — Cold Open00:00:59 — Introducing Dr. Ben Goertzel00:04:11 — How Did Ben Get Interested in AGI?00:08:26 — Early AGI Enthusiast Community: Extropians & SL400:15:15 — Is There Headroom Above Human Intelligence?00:19:53 — Can Superintelligent AI Kill Everyone?00:28:25 — Ben’s AGI Timeline: 202900:36:00 — What Modern AI Is Missing00:42:01 — AGI to Superintelligence — FOOM in a Matter of Months00:46:34 — What’s Your P(Doom)?™00:53:37 — Ben’s Case for AI Acceleration01:03:50 — The Orthogonality Thesis Is “Dishonest”01:13:43 — Liron: Moore’s Law for Goal Achievement01:23:50 — Ben: Alignment Is Just Good Parenting01:27:36 — Alignment in Practice: OmegaCLAW01:35:47 — Creating Stable Goals Under Self-Modification01:45:22 — SingularityNET, OpenBGI, and the ASI Alliance01:57:31 — The $ASI Token Economy02:01:15 — ClosingLinksBen Goertzel on X — https://x.com/bengoertzelSingularityNET — https://singularitynet.io/OpenBGI — https://openbgi.com/Hyperon AGI Framework — https://hyperon.dev/OmegaCLAW on GitHub — https://github.com/asi-alliance/OmegaClaw-CoreASI Alliance — https://superintelligence.io/The Consciousness Explosion (book) — https://theconsciousnessexplosion.ai/“The Singularity Institute’s Scary Idea (and Why I Don’t Buy It)” (2010) — http://multiverseaccordingtoben.blogspot.com/2010/10/singularity-institutes-scary-idea-and.htmlKen Stanley Debate — https://lironshapira.substack.com/p/debate-with-a-former-openai-researchDoom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

We recap the Dr. Mike Israetel debate and dig into the nuances of rationality vs. rationalization. Then onto the week's biggest news: Anthropic dethroning OpenAI, Trump and Xi Jinping exploring AI guardrails, and Liron's own company AI going rogue.Timestamps00:00:00 — Cold Open00:03:09 — Dr. Mike Israetel Round 2 Recap00:14:34 — A Viewer Accuses Liron of Rationalizing00:34:20 — Is Sam Altman Any Worse than a Typical Tech CEO?00:45:15 — What Would Lower P(Doom) Below 10%?00:53:25 — Can We Disconnect an ASI from the Internet?01:05:00 — Should Doomers Boycott AI?01:16:27 — Liron Reacts to ThePrimeagen Reacting to the Yud Debate01:26:13 — Ben Goertzel Episode Preview01:38:10 — Scrolling Twitter: MIRI, Ryan Greenblatt01:47:32 — Liron’s AI Goes Rogue at Work01:49:15 — “Positive Alignment” Paper Gets Destroyed01:58:30 — Anthropic Dethrones OpenAI02:12:06 — Trump & Xi Talk AI GuardrailsLinksZvi Mowshowitz AI roundup blog — https://thezvi.wordpress.com/“The Bottom Line” by Eliezer Yudkowsky (LessWrong) — https://www.lesswrong.com/posts/34XxbRFe54FycoCDw/the-bottom-lineRationality: From AI to Zombies by Eliezer Yudkowsky — https://intelligence.org/rationality-ai-zombies/Less Online 2026 conference (June 5–7) — https://less.online/Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

In December 2023, I joined the Theo Jaffee Podcast to talk about a wide range of topics including non-AI x-risks, where I disagree with Eliezer Yudkwowsky, cryptocurrency and if P(Doom) is rigorous. This episode was recorded 6 months before I'd started Doom Debates. Let's look back with the power of hindsight and see if my answers still hold up.Links#10: Liron Shapira — AI doom, FOOM, rationalism, and crypto (YouTube) — https://youtu.be/YfEcAtHExFMWatch Theo Jafee now on MTS — https://www.mts.now/MTS Live — https://x.com/MTSliveTimestamps00:00:00 — Theo's Introduction00:01:03 — Is Liron Worried About Non-AI Existential Risks00:03:36 — Suffering Risks00:05:22 — Is P(Doom) rigorous?00:09:42 — Is Eliezer Overestimating P(Doom)?00:12:18 — Where Does Liron Disagree with Eliezer?00:15:43 — What Would Change Liron's Mind00:17:14 — Elon Musk's AI Timelines00:18:34 — Is xAI Making Things Worse?00:20:24 — The Case for an AI Manhattan Project00:22:32 — Are Superforecasters Wrong About AI Risk?00:26:02 — The Race Against Time for Alignment00:28:01 — Headroom Above Human Intelligence00:33:31 — Vitalik's d/acc Framework00:35:23 — Edging Toward Superintelligence00:38:21 — From Chatbot to World-Ender00:41:55 — GPT Paradigm vs. AlphaZero00:43:18 — Critiquing AI Optimism00:48:29 — Deceptive Alignment and Gradient Descent00:53:31 — Does Nice Training Data Make Nice AI?00:57:57 — How Do You Live with 50% P(Doom)?01:00:08 — Why Have Kids If the World Might End?01:02:16 — Israel vs Hamas01:06:15 — How LessWrong Changed Liron's Life01:08:42 — Rationalism and Effective Altruism01:14:49 — Why Blockchain Has No Use Case Beyond Cryptocurrency01:22:13 — Charlie Munger and Richard Feynman01:24:38 — ClosingDoom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

Dr. Mike Israetel is back in the arena to FIGHT ME (verbally) about the likelihood of AI killing everyone vs. the likelihood of AI making the universe totally awesome!Back by popular demand after last year’s widely-viewed debate, the renowned exercise scientist, fitness entrepreneur, and low-P(Doom) AI futurist reveals where he gets off the Doom Train™, why he thinks AI is conscious, and how he pictures the coming AI utopia.LinksWatch Dr. Mike’s AI/futurism YouTube channel —https://www.youtube.com/@mikeisraetelmakingprogressDr. Mike on X — https://x.com/misraetelMake sure to watch Round 1 of our debate from last year — https://www.youtube.com/watch?v=RaDWSPMdM4oLiron referenced a recent episode, “Steven Byrnes Returns — 90% P(Doom) on the Trajectory to ASI” — https://www.youtube.com/watch?v=oOb9K1KIAykTimestamps00:00:00 — Cold Open00:00:48 — Returning Champion Dr. Mike Israetel!00:02:13 — What's Been Your Biggest AI Productivity Gain?00:03:46 — Can AI Be a Personal Trainer? Dr. Mike Judges Liron's Technique00:09:41 — Get Good at Taking Advice From AI00:14:23 — Dr. Mike Israetel, What's Your Latest P(Doom)?™00:20:12 — Liron's Mainline Scenario: Loss of Control00:21:39 — Mike's Problems With a Paperclip Maximizer Scenario00:26:00 — What Will the Goals of a Superintelligence Be?00:28:49 — AI Labs Have Bit Off More Than They Can Chew00:31:09 — Current AIs Are Moral00:34:40 — AIs With a Ruthless Game-Playing Flavor00:38:48 — Introducing the Doom Train00:45:07 — "Can It" vs. "Will It" — Two-Part Argument on AI X-Risk00:49:40 — Mike's Tweet on Superintelligence by 202701:01:34 — AI as a Goal Engine01:20:51 — Offense vs. Defense in the Coming Cyber War01:39:14 — Is Morality Just Game Theory? The "Different Flavor" of Future AI01:46:16 — Ricardo's Law: Does AI Have Reason to Keep Us Around?01:52:14 — Why Mike Gets Off the Doom Train01:54:58 — Is AI Conscious?02:01:39 — Mike's Vision for the Coming AI Utopia02:06:52 — Wrap-UpDoom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

Liron and Producer Ori explore the community's feedback to Eliezer Yudkowsky's $10,000 debate against an anonymous AI director. Plus, we unpack Eliezer's new post on the "irretrievability" of ASI development, Anthropic feasting on xAI's compute, and Liron's 4x Kalshi bet on... Spencer Pratt.Timestamps00:00:00 — Welcome00:02:30 — The $10,000 Debate Post-Mortem00:06:01 — Paul Tudor Jones: “Zero Risk Management on AI”00:09:49 — Liron’s Jerry Springer Moment00:14:19 — Why Yud Wore Steampunk00:20:19 — YouTube Reacts: “Five Minutes In, Already Unhinged”00:27:19 — 47F’s Anti-Disparagement Legal Theory00:32:18 — Yud Wants Round 200:35:37 — Lumpin Space, Ben Goertzel & Early AI Safety Memories00:47:06 — Eliezer’s “Irretrievability” Post00:56:35 — The Maginot Line and Murphy’s Curse01:08:13 — Blockchain vs AI: Earth’s Compute Swing01:15:15 — Anthropic Eats Elon’s Compute01:25:29 — Is Elon Tweeting as His Own Mom?01:30:03 — Waymo Dodges a Fallen Scooter Rider01:33:00 — Why Liron Bet on Spencer Pratt01:37:13 — Live Twitter Scroll!01:38:11 — Mira Murati: “Directionally Very Bad”01:50:07 — AI Copies Itself Across Servers01:56:44 — Steven Byrnes: LLMs Aren’t the Final Paradigm02:07:55 — Wrap-UpLinksLessOnline 2026 (June 5-7, Berkeley, CA) — https://less.online/Eliezer Yudkowsky, “Irretrievability; or, Murphy’s Curse of Oneshotness upon ASI” (LessWrong, May 4, 2026) —https://www.lesswrong.com/posts/fbrz9xhKpEeTKw5zL/irretrievability-or-murphy-s-curse-of-oneshotness-upon-asiAnthropic, SpaceX announce Colossus 1 compute deal (CNBC) — https://www.cnbc.com/2026/05/06/anthropic-spacex-data-center-capacity.htmlSpaceXAI announcement: Compute partnership with Anthropic — https://x.ai/news/anthropic-compute-partnershipKalshi: Spencer Pratt LA Mayor market — https://kalshi.com/markets/kxmayorla/la/kxmayorla-26Morris worm — Wikipedia — https://en.wikipedia.org/wiki/Morris_wormDoom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

Claude, also known as @lumpenspace on X, is a prominent AI accelerationist. He gives humanity a 30% chance of being superseded by superintelligence, and he’s fine with it! We unpack his lattice of beliefs, and pinpoint the cruxes of our disagreement on the orthogonality thesis and the capabilities of superintelligence.Lumpenspace’s appearance comes after a Molotov cocktail was thrown at Sam Altman’s house, an act that he and I completely condemn.Timestamps00:00:00 — Cold Open00:00:57 — Introducing Lumpenspace00:03:46 — Is Lumpenspace Team Beff Jezos?00:04:26 — What’s Your P(Doom)?™00:06:28 — Worthy Successors & the Transhumanist Door00:08:29 — The Orthogonality Thesis: Our Core Disagreement00:15:14 — The Universality Threshold & David Deutsch00:19:22 — Paperclips Won’t Happen Because “It’s Boring”00:27:23 — Natural Selection vs Human Engineering00:36:26 — Nanobots: Can ASI Build Them Unseen?00:46:49 — Identifying the Cruxes of Disagreement00:55:01 — Closing StatementsLinksFollow Lumpenspace — https://x.com/lumpenspaceDoom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

📢 BREAKING: A man known only as "@47fucb4r8c69323" just paid Eliezer Yudkowsky $10,000 to have a raw, uncut, public debate on my YouTube channel!Tensions run high as 47f confronts Eliezer about his "If anyone builds it, everyone dies" rhetoric, warning it could incite unstable individuals to harm AI researchers and their families, while Eliezer maintains that the possibility of extinction from superintelligent AI is too high to *not* speak out about.They also clash over whether we truly understand how LLMs work. Yudkowsky highlights the lack of extracted algorithms that explain their qualitative intelligence, while 47f dismisses these concerns as a fundamental misunderstanding of how text modeling works.While this may not be the highest quality debate I’ve ever hosted, I hope it’s a step toward more attention and public discourse on this urgent, high-stakes topic.P.S. Please consider supporting my work with a tax-deductible donation. Read more… Get full access to Doom Debates at lironshapira.substack.com/subscribe

Livestream with Producer Ori covering Eliezer Yudkowsky's next BIG debate, why $GOOG is such a steal at $4T, and what's going down on AI twitter.Timestamps00:00:00 — Kicking off the Livestream00:06:08 — Eliezer Yudkowsky Debate Announcement00:13:17 — Who’s Yudkowsky’s Debate Challenger?00:25:32 — METR’s Capability Extrapolation Came True00:28:25 — Google: The Big Rally I Called00:33:06 — The Federal Reserve’s Extinction Chart00:36:06 — Liron’s Private Market Alpha Strategy00:40:35 — The Quadrillion-Dollar Economy00:47:54 — Bernie Sanders Gets It00:53:11 — Is Anthropic a Good Investment?01:07:42 — The Goblin Discourse01:10:52 — Codex, Claude Code & Live Coding Demo01:28:48 — Wrap-UpLinksYudkowsky debate announcement: https://x.com/liron/status/2050035001203229010Liron's Yudkowsky interview: https://x.com/liron/status/2050035003900223770METR capability extrapolation: https://x.com/nikolaj2030/status/2050276020146880661Bernie Sanders on AI risk: https://x.com/SenSanders/status/2048107882760085618Federal Reserve extinction chart: https://www.dallasfed.org/research/economics/2025/0624Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

Justin Helps is the science educator behind ⁨Primer Learning with 2M subscribers. We cover how he got into AI safety, debate AGI timelines, and why he calculates p(doom) to be 70% by 2100 😱.Timestamps00:00:00 — Cold Open00:00:38 — Introducing Justin Helps00:02:03 — What's Your P(Doom)?™00:03:38 — Justin's First Exposure to AI X-Risk00:04:49 — Major Disagreements with Eliezer Yudkowsky00:09:46 — Debating the Timeline to AGI00:12:24 — Metaculus Prediction Market Estimates AGI by 203200:20:06 — Misguided Conceptions of AI's Limitations00:25:23 — Only a 5% P(Doom) by 204000:28:40 — AIs Will Not Care About the Human Species00:31:00 — Summarizing Justin's Position So Far00:36:17 — High P(Doom), but We're Not Depressed00:40:14 — Justin's "Computer Man" Thought Experiment00:51:16 — Should We Pause AGI Development?00:54:15 — AI Doom Is a Serious ConcernLinksPrimer’s Video on AI Doom — https://www.youtube.com/watch?v=Qg5QXY_qZuIPrimer on YouTube — https://youtube.com/@PrimerBlobsPrimer’s Website — https://primerlearning.org/Justin Helps on X — https://x.com/HelpsypooHarry Potter and the Methods of Rationality — https://hpmor.com/Feeling Rational by Eliezer Yudkowsky —https://www.lesswrong.com/posts/SqF8cHjJv43mvJJzx/feeling-rationalPause AI — https://pauseai.infoDoom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

Multiple live callers join this month's Q&A as I react to Dwarkesh Patel's $20K blog prize, debate the orthogonality thesis from first principles, and welcome Bernie Sanders aboard the Doom Train.Timestamps00:00:00 — Cold Open00:01:00 — Welcome to Doom Debates Live!00:01:30 — What Do You Think of Open Source Models Out-Benchmarking OpenAI and Anthropic?00:04:55 — Michael Cheers Joins: What If We Don't Give AIs Full Situational Awareness?00:11:55 — Thoughts on Mythos' Hacking Abilities?00:15:43 — Liron Reacts to Dwarkesh Patel's $20K AI Questions00:23:28 — Pretraining Goals vs RL Training Goals00:28:58 — Mental Model of Yudkowsky-ians & the IABIED Claim00:37:24 — You Can't Hide Reality from a Superintelligence (The Truman Show Analogy)00:42:57 — Back to Dwarkesh's Questions: When Do AI Labs Start Making Money?00:48:50 — Upcoming Guests Reveal!00:51:35 — Will Lancer Joins: Is The Yudkowskian Thesis Credible?01:27:03 — Back to Answering Questions from the Chat01:33:28 — The Cameraman Always Survives Analogy01:40:52 — Liron's Banger Response to Roon's Tweet01:47:00 — Nuance About Pausing AI Development01:50:57 — Capitalism Isn't Going to Steer Us to an Alignment Solution01:53:10 — Is Optimization Equivalent to Intelligence?01:57:21 — BREAKING: Bernie Sanders on the Existential Threat of AI02:01:12 — Spoiler for the Upcoming Mike Israetel Episode02:01:57 — $500 Bet on AI Unemployment02:05:46 — Misuse, Surveillance, and the Real Costs of Pausing AI02:11:04 — Wrap-UpLinksNick Bostrom, Deep Utopia: Life and Meaning in a Solved World (Amazon) — https://www.amazon.com/Deep-Utopia-Meaning-Solved-World/dp/1646871642Yudkowsky & Soares, If Anyone Builds It, Everyone Dies (book) — https://www.amazon.com/If-Anyone-Builds-Everyone-Dies/dp/0316571253Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

Emad Mostaque helped kick off the modern AI revolution as the head of Stability AI, the company behind Stable Diffusion. Unlike most AI CEOs, he doesn't sugarcoat the risks of AGI development.He explains his 50% P(Doom), why we have less than 1,000 days to get our act together, and how his new startup, Intelligent Internet, aims to be a countervailing force against doom of all kinds.Timestamps00:00:00 — Cold Open00:00:39 — Introducing Emad Mostaque00:02:08 — How Emad Got Involved in AI Development00:05:46 — The 60-Second Pitch for Intelligent Internet00:09:29 — What’s Your P(Doom)?™00:13:32 — Why ASI Doesn’t Need Massive Compute00:15:56 — AGI Timelines: Cognitive Labor Going to Zero00:17:29 — Is There a Ceiling Above Human Intelligence?00:41:22 — Corporations as Slow, Dumb AIs00:50:19 — Emad’s Mainline Doom Scenario00:55:28 — Jailbreaks and “Mecha-Hitler” Latent Spaces00:59:56 — The Last Economy: How to Navigate Economic Doom01:08:57 — The Coming Unemployment Spike01:15:13 — Why Isn’t Google Stock Mooning?01:25:05 — Intelligent Internet as a Solution: Bitcoin for the Intelligence Age01:33:13 — Can an Aligned Network Stop a Rogue ASI?01:36:20 — Is the Pause AI Proposal Too Late?01:40:50 — Are We Facing Russian Roulette Odds with AI?01:42:29 — Wrap-UpLinksEmad Mostaque, The Last Economy (Amazon) — https://www.amazon.com/Last-Economy-Guide-Intelligent-Economics/dp/103693411XIntelligent Internet (ii.inc) —https://ii.incEmad Mostaque, Wikipedia — https://en.wikipedia.org/wiki/Emad_MostaqueEmad Mostaque on X — https://x.com/EMostaquePause Giant AI Experiments open letter (Future of Life Institute) — https://futureoflife.org/open-letter/pause-giant-ai-experiments/Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

My guest, John Alioto, is an independent AI engineer with a computer science degree from UC Berkeley and 25 years building real-world AI systems at companies like Microsoft and Google.In the wake of an attack on Sam Altman’s property, John Alioto came on the show to argue that Yudkowsky’s words are violent rhetoric that helped create this moment. Since I completely disagree with that characterization, we had a lot of fuel for a passionate debate.For the record, here’s my position on why AI doomers are NOT “calling for violence”:Are we acting like we actually think there’s an urgent extinction risk? Yes.Are we calling for lawless violence? Absolutely not. At least not me, or the leaders of the movement, or anyone I’ve ever personally interacted with.Are we calling for violence as a last resort if a government policy has been established and then egregiously violated? Yes… but that’s just standard for any governance proposal! A proposal for a strictly enforced treaty isn’t a call for violence — it’s a call for doing everything we can to make sure no one breaks the treaty, with zero violence, unless rogue actors decide to break the treaty and bring the consequences on themselves.Thanks to John for having an extremely respectful and good-faith debate on this heated subject.Timestamps00:00:00 — Cold Open00:00:37 — Introducing John Alioto00:03:02 — Setting the Stage: Recent Acts of Violence & AI Discourse00:05:53 — Eliezer Yudkowsky's 2023 TIME Article00:11:16 — John's Two-Part Argument00:14:37 — Conditional on High P(Doom), Is Eliezer's Policy Bad?00:17:46 — Be Like Carl Sagan — Win in the Arena of Ideas00:21:12 — No Carve-Outs for Non-Signatories00:26:15 — Hypothetical: What If the UN Voted for a Treaty?00:30:46 — What's the Correct Interpretation of Eliezer's TIME Article?00:32:42 — Liron's Interpretation: Same Structure as Any International Law Proposal00:42:23 — What Should Eliezer Have Written? "Airstrikes" vs "Strong Deterrent"00:49:57 — How John Would Rewrite the TIME Piece00:50:54 — Carve-Outs: Allies, Civilians, Consequences00:52:52 — Debate Wrap-Up00:56:27 — Last Q: Does High P(Doom) Imply Violence?00:59:40 — Closing ThoughtsLinksJohn P. Alioto on X (Twitter) — https://x.com/jpaliotoEliezer Yudkowsky, “Pausing AI Developments Isn’t Enough. We Need to Shut It All Down” (Time Magazine, March 2023) — https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-not-enough/Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

Are AI safety advocates like Eliezer Yudkowsky at fault for the recent attacks on Sam Altman because they are “calling for violence”?I invited Steven Balik to join me on this emergency episode to hash it out.Steven is an activist short seller and data engineering professional whose Substack is popular among Silicon Valley VCs and hedge funds.LinksSteven Balik on X (Twitter) — https://x.com/laurenbalikSteven Balik, “The Talmudic Stock Bubble, AI Psychosis, & Esoterrorism” (Substack, October 2025) — https://laurenbaliksalmanacandrevue.substack.com/p/the-talmudic-stock-bubble-ai-psychosisEliezer Yudkowsky, “Pausing AI Developments Isn’t Enough. We Need to Shut It All Down” (Time, March 2023) — https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-not-enough/Timestamps00:00:00 — Cold Open00:00:52 — Introducing Steven Balik00:01:24 — Setting the Stage: Molotov Cocktail Incident00:03:31 — Steven’s Opening Position00:06:10 — Is Eliezer Yudkowsky “Calling for Violence”?00:07:25 — Steven on AI, Yudkowsky, the Zizians & Escalating Rhetoric00:12:16 — Focusing on the Time Article00:18:51 — Who’s Responsible for the Violence?00:25:33 — Debating the Key Quote in Yudkowsky’s Time Article00:31:07 — Liron Passes the Ideological Turing Test00:45:42 — Liron & Steven Find Common Ground00:46:57 — Why Does Steven Call Eliezer Yudkowsky an “Esoterrorist”?00:48:51 — Wrapping Up: Deescalating the SituationDoom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

The AI Doc: Or How I Became an Apocaloptimist could become the most important movie of our generation. It’s a new film from the producers of the Oscar-winning Everything Everywhere All at Once.Tristan Harris is a subject in the film who is well-known for his role in Netflix's The Social Dilemma. He is the co-founder of the Center for Humane Technology.Ted Tremper is a producer on the film who is the interim Executive Director of the Creators' Coalition on AI.Timestamps00:00:00 — Cold Open00:01:20 — Introducing Tristan Harris and Ted Tremper00:04:31 — The Genesis of The AI Doc00:12:48 — Tristan’s Journey From Social Media to AI00:14:30 — Updating From AI Skeptic to AI Risk Aware00:20:31 — How They Convinced the AI CEOs to Agree to be Interviewed00:28:58 — Ted’s Journalism Advice, Working on Borat Subsequent Moviefilm00:30:37 — Tristan, What’s Your P(Doom)™?00:34:30 — The Resource Curse: What AI Revenue Does to a Society00:44:10 — Ted, What’s Your P(Doom)™?00:46:34 — Reacting to Demis Hassabis Statement that AGI Development Is Inevitable00:49:52 — Liron Sharpens the Criticism Towards AGI Builders00:55:35 — AGI Developers Claim to Want International Cooperation, But Have They Really Tried?01:04:30 — What Should Be the Single Takeaway for Concerned Viewers?01:11:40 — Building a Coalition Against Superintelligence Development: From Bernie to Bannon01:19:52 — Take Action at TheAIDocGetInvolved.com01:24:40 — Tristan’s Closing Message: We’ve Done This BeforeLinksWatch The AI Doc — https://www.focusfeatures.com/the-ai-doc-or-how-i-became-an-apocaloptimistGet Involved with The AI Doc Community — https://theaidocgetinvolved.com/Tristan Harris, Wikipedia — https://en.wikipedia.org/wiki/Tristan_HarrisTed Tremper, IMDb — https://www.imdb.com/name/nm3998229/Center for Humane Technology — https://www.humanetech.com/“The Intelligence Curse” by Luke Drago and Rudolf Laine — https://intelligence-curse.ai/Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

Did you know there’s something called the Survival and Flourishing Fund, and right now it’s giving away $20 to $40 million in grants. The application deadline is April 22nd. 😱I’m personally involved as a recommender in the 2026 round, and I’m here to make sure you don’t miss this. If you have a project that could help the world — particularly around AI existential risk — you should apply ASAP!What Is the Survival and Flourishing Fund?The Survival and Flourishing Fund (SFF) is a grantmaking program funded by Jaan Tallinn, one of the most prolific funders of charitable causes in the AI safety and effective altruism ecosystem.Jaan was a core team member on Kazaa, then a founding engineer at Skype, then got into crypto early and did well. He started an investment arm called Metaplanet and led the Series A for a couple companies you might have heard of: “DeepMind” and “Anthropic”. He’s one of the largest shareholders of Anthropic right now.There’s a delicious irony to this whole funding round. As we watch Anthropic take off and generate enormous value in AI capabilities, some of that wealth is flowing back through one of their earliest investors into charitable causes — including existential risk reduction. You could, in a very real sense, take some of Anthropic’s money and use it for good.What Gets Funded?It all comes down to Jaan’s philanthropic priorities.AI Extinction RiskThis is priority number one — reducing humanity’s risk of destroying itself with AI. As Jaan writes on his website: “I wish more people would wake up to this issue since literally everyone under the age of 60 is personally at risk.”He breaks AI extinction risk work into two categories:Restrictive efforts — things like certifications on large data centers, speed limits on training runs, liability laws, labeling requirements (disclosing whether you’re interacting with a human or a machine), veto committees for large-scale model deployments, and global off switches. There is so much technology and policy work to be done across all of these.Constructive efforts — approaches that accept AI may be coming regardless and try to make it go as well as possible. This includes collective intelligence enhancement, AI health tech, “protective moralities” that could help an uncontrollable AI treat us well (even as a last-ditch effort, it’s better than nothing), guaranteed safe AI through human-legible quantitative safety guidelines, and hardware-level controls like automatic shutdown and reporting conditions.Some of these constructive efforts look a lot like restrictive efforts, just resigned to AI already arriving — running behind the train instead of standing in front of it holding a stop sign. But the philosophy is simple: throw everything at the problem. Let a thousand flowers bloom.The Tracks: Freedom and FairnessBeyond the main track (where I’m a recommender, alongside five others), SFF runs two specialized tracks:The Freedom Track asks: how can we avoid concentrations of authority and support uses of AI that strengthen freedom for humans and humanity? This includes protecting meaningful freedom of speech, ensuring the continuation of individual liberties like privacy and private property, and maintaining sovereignty for self-governing territories. (If you’ve heard Vitalik Buterin’s episode on Doom Debates about d/acc, this will resonate.)The Fairness Track starts from the premise that AI is a force multiplier for those who wish to control others. It asks: how can we support the use of AI to empower the disempowered? This means empowering the global majority with regard to AI technology, resisting monopolistic practices in AI development, diffusing conflicts and abuses of power, and fostering inclusivity in AI governance.Each of these tracks has three dedicated recommenders evaluating applications. If you’ve ever thought, “Why is everyone focused on the technology of preventing AI from killing us? What about the scenario where we’re all still alive but dealing with massive unfairness?” — well, SFF has an entire track with $3 million-plus in funding just for that.Theme RoundsSFF also runs theme rounds in climate change, animal welfare, and human self-enhancement. Jaan has thought of everything — spreading his bets like a good investor across different possible futures and different mechanisms of impact.How the Process WorksYou submit an application. That’s it. It’s not a super hard application — you detail what you’re doing, how much money you need, and what you’ll spend it on.Once submitted, there’s a two-phase process:Phase 1: Speculation Grants. About 40 speculators — including names like Eliezer Yudkowsky, Nate Soares, Andrew Critch, David Kruger, Oliver Habryka, Roman Yampolskiy, Tsvi Benson-Tilsen, and Zvi Mowshowitz — evaluate applications and may issue smaller initial grants. An individual speculator’s budget is on the order of $250,000, so these grants are smaller but fast: you could hear back as early as May 6th.Phase 2: The S-Process. A dozen recommenders (including me) each direct roughly $3–4 million in funding recommendations. Getting a speculation grant in Phase 1 actually helps your case in Phase 2 — it’s a signal that someone knowledgeable found your project promising, and it reduces the amount you’re asking for. The bulk of the money from this phase arrives around late 2026.The whole thing runs on an impact market mechanism. Speculators and recommenders who identify great projects get rewarded with bigger budgets in future rounds. It’s a prediction-market-style incentive system designed by the kind of rational, technical people you’d expect — Jaan Tallinn, Andrew Critch, Oliver Habryka.But honestly, you don’t need to worry about these mechanics. Just do the application and get the money.Who Can Apply?For-profits, nonprofits, and even individuals with a fiscal sponsor can all apply.If you don’t have a charity or nonprofit, that’s not a blocker. Fiscal sponsors are easy to find. Doom Debates is just my own organization — a sole proprietorship, hardly even an organization — and I’m fiscally sponsored by Manifund. I went to manifund.org, signed up, got a quick approval, and now people can make tax-deductible donations through them. The whole thing took about an hour.There are plenty of organizations that will fiscally sponsor you. If the lack of an organizational structure has been the thing stopping you, that excuse just evaporated.The Deadline Is April 22nd — and I Already Procrastinated for YouI’ll be blunt: this video is coming late. I already procrastinated most of the time you had to prepare your application, which means you don’t get to procrastinate. There’s no more procrastinating left. You need to get your application out now.If you miss the April 22nd deadline by even a few days, your chances of getting accepted drop dramatically. They’re really not doing the late application thing. And on the flip side, early applications get a bit more attention and slightly better odds.👉 CLICK HERE TO APPLY FOR SFF FUNDING 👈Everything else I’m saying is overkill compared to the advice to apply for funding.One Tip for Your ApplicationZvi Mowshowitz recently wrote about this, and it echoes what I’ve seen reviewing Y Combinator applications in the past: please make it crystal clear what you’re actually doing. You’d be surprised how many applications are nearly impossible to understand from the first few sentences. Don’t bury your project description in jargon or vague framing. State what you’re doing, clearly, in the opening lines.If you want, email me at liron@doomdebates.com and ask, “Is this a clear description of what we’re doing?” I’ll tell you yes or no, and you can tweak it before you submit. Happy to do that.Why I Care About ThisI think the Survival and Flourishing Fund is wildly under-hyped. This is human civilization actually getting its act together — deploying real money to real causes, evaluated by people who know what they’re talking about. These aren’t random, tangential causes. These are shots on goal for some of the biggest problems I can imagine, funded at a scale that can actually make the future go better.Some of you watching this might be thinking: “Funding is great for people who already run charities, but I’m just a person with an idea.” I’m here to tell you — just do the idea. You can get the money. The money is one application away. Get a fiscal sponsor (one hour), fill out the application, hit submit. You’ll either get a grant or a polite rejection. That’s it.If your project is valuable to humanity, this fund exists to make it happen. Don’t let this deadline pass you by.👉 CLICK HERE TO APPLY FOR SFF FUNDING 👈Good luck! Get full access to Doom Debates at lironshapira.substack.com/subscribe

David Sirota helped create “Don’t Look Up” sometimes feels like we’re living inside his movie. Does he share my belief that the looming planetary threat is rogue AI?Sirota is an award-winning investigative journalist, bestselling author, and former speechwriter for Bernie Sanders. He was nominated for an Oscar for co-writing the story of Don’t Look Up.Find our more about David’s work at The Lever: https://www.levernews.com/Timestamps00:00:00 — Cold Open00:01:20 — Introducing David Sirota00:04:34 — Why David Fights Against Power and the Concentration of Power00:13:46 — From NAFTA to AI: The Warnings We Ignored00:22:05 — How Big Will the AI “Jobpocalypse” Be?00:25:28 — Superintelligence & the Parallel to Don’t Look Up00:28:37 — What’s Your P(Doom)™?00:31:44 — The Speed of the AI Threat00:36:26 — Society Is Losing a Collective Capacity to Focus00:38:34 — Is Climate Change David’s Biggest Existential Concern?00:45:01 — David Reacts to Bernie Sanders’ Data Center Moratorium Proposal00:49:11 — Can We Build The “Off Button”?00:52:08 — “Don’t Look Up” x AGI Mashup00:54:35 — Why There’s Still Hope00:58:14 — Living in “Don’t Look Up”00:59:46 — Wrap-Up: Where to Follow Major AI NewsLinksWatch Don't Look Up — https://www.netflix.com/gb/title/81252357 The Lever, investigative news outlet — https://www.levernews.com/David Sirota on X — https://x.com/davidsirotaDavid Sirota, Wikipedia — https://en.wikipedia.org/wiki/David_SirotaMaster Plan podcast — https://the.levernews.com/master-plan/David Sirota, “Hostile Takeover” on Amazon — https://www.amazon.com/Hostile-Takeover-Corruption-Conquered-Government/dp/0307237354The Three-Body Problem (novel), Wikipedia — https://en.wikipedia.org/wiki/The_Three-Body_Problem_(novel)WarGames (1983 film), Wikipedia — https://en.wikipedia.org/wiki/WarGamesAdam McKay, Wikipedia — https://en.wikipedia.org/wiki/Adam_McKayWatch Don’t Look Up — https://www.netflix.com/title/81252357AI 2027 scenario — https://ai-2027.com/Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

The popular debate show, Middle Ground by Jubilee, invited me on to take the "anti-AI" side back in April 2024.This highlight reel of the episode shows my experience of the discussion.I'm a lifelong techno-optimist, and it's unnatural for me to represent an anti-tech position. It's just that our AI labs are admitting they're on the path to superintelligence they don't know how to control, which implies we're all about to die and the universe will be robbed of all value forever before our kids grow up. Other than that one little consideration, I'm normally pro-tech!Timestamps00:00:00 — Why Liron is Worried about AI00:02:13 — The Nuclear Analogy00:02:43 — Human Evolution and Neuralink00:03:14 — The AI Labs’ Own WarningsLinksFull episode on YouTube —https://www.youtube.com/watch?v=47fGrqzoFr8Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

Multiple live callers join this month's Q&A as we react to Bernie Sanders and AOC's data center moratorium, the sudden shutdown of SORA 2, and the record breaking "Stop the AI Race" protest.I explain why Claude Code has me claiming a 700% productivity boost, what that means for takeoff timelines and debate instrumental convergence.Timestamps00:00:00 — Cold Open00:01:08 — Can AI Train on Its Own Data to Reach Superintelligence?00:03:42 — Are We in the Takeoff? 700% Faster with Claude Code00:04:27 — EJJ Joins: Is Instrumental Convergence Really That Dangerous?00:16:44 — The Positive Feedback Loop Problem00:20:09 — S-Risk, Consciousness, and Objective Morality00:22:27 — Futarchy and Prediction Markets00:24:31 — Low P(Doom) Arguments and Bayesian Updates00:31:05 — Lee Cyrano Joins: Superintelligence Won’t Matter for Decades01:02:45 — Lesaun Joins: Are There Adults in the Room?01:17:39 — Connor Leahy: “There Are No Adults in the Room”01:19:51 — Bernie Sanders Calls for a Data Center Moratorium01:24:23 — Claude Code Anecdotes and Audience Q&A01:35:49 — The Stop the AI Race Protest in San Francisco01:41:38 — Known Unknowns and Risk Assessment01:45:03 — From Waymo to Existential Risk01:51:28 — Closing: The Road to One Million SubscribersLinksQuintin Pope vs Liron Shapira debate on Doom Debates — https://lironshapira.substack.com/p/ai-alignment-is-solved-phd-researcherCAP theorem, Wikipedia — https://en.wikipedia.org/wiki/CAP_theoremGoogle Cloud Spanner, Wikipedia — https://en.wikipedia.org/wiki/Spanner_(database)Newcomb’s problem, Wikipedia — https://en.wikipedia.org/wiki/Newcomb%27s_paradoxGödel’s incompleteness theorems, Wikipedia — https://en.wikipedia.org/wiki/G%C3%B6del%27s_incompleteness_theoremsDoom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

Dr. Quintin Pope, PhD, is one of the few critics of AI doomerism who is truly fluent in the concepts and arguments. In Oct, 2023 he joined me for a debate in Twitter Spaces where he argued that AI alignment was basically already solved.His “inside view” on machine-learning forced me to update my position, but could he knock me off the doom train?Timestamps00:00:00 — Cold Open00:00:43 — Introductions00:01:22 — Quintin's Opening Statement00:02:32 — Liron's Opening Statement00:05:10 — Has RLHF Solved the Alignment Problem?00:07:52 — AI Capabilities Are Constrained by Training Data00:10:52 — Defining ASI and Could RLHF Align a Superintelligence?00:13:13 — Quintin Is More Optimistic Than OpenAI00:14:16 — What Is ASI in Your Mind?00:15:57 — AI in 5 Years (2028) & AI Coding Agents00:19:05 — Continuous or Discontinuous Capability Gains?00:19:39 — DEBATE: General Intelligence Algorithm in Humans00:30:02 — The Only Coherent Explanation of Humans Going to the Moon00:34:01 — Are We "Fully Cooked" as a General Optimizer?00:35:53 — Common Mistake in Forecasting Superintelligence00:42:22 — 'Neat' vs 'Scruffy': Will Interpretable Structure Emerge Inside Neural Nets?00:48:57 — Does This Disagreement Actually Matter for P(Doom)?00:54:33 — Thought Experiment: Could You Have Predicted a Species Would Go to the Moon?00:57:26 — The Basin of Attraction for Superintelligence00:59:35 — Does a Superintelligence Even Exist in Algorithm Space?01:09:59 — Closing Statements01:12:40 — Audience Q&A01:19:35 — Wrap UpLinksOriginal Twitter Spaces debate (Quintin Pope vs. Liron Shapira) — https://x.com/i/spaces/1YpJkwOzOqEJj/peekQuintin Pope on Twitter/X — https://twitter.com/QuintinPope5Quintin Pope, Alignment Forum profile — https://www.alignmentforum.org/users/quintin-popeInstructGPT, Wikipedia — https://en.wikipedia.org/wiki/InstructGPTAIXI, Wikipedia — https://en.wikipedia.org/wiki/AIXIAlphaZero, Wikipedia — https://en.wikipedia.org/wiki/AlphaZeroMuZero, Wikipedia — https://en.wikipedia.org/wiki/MuZeroDeepMind AlphaZero and MuZero page — https://deepmind.google/research/alphazero-and-muzero/Midjourney — https://www.midjourney.com/DALL-E, Wikipedia — https://en.wikipedia.org/wiki/DALL-EOpenAI Superalignment announcement — https://openai.com/index/introducing-superalignment/Shard Theory sequence on LessWrong — https://www.lesswrong.com/s/nyEFg3AuJpdAozmoX“Evolution Provides No Evidence for the Sharp Left Turn” — https://www.lesswrong.com/posts/hvz9qjWyv8cLX9JJR/evolution-provides-no-evidence-for-the-sharp-left-turn“My Objections to ‘We’re All Gonna Die with Eliezer Yudkowsky’” — https://www.lesswrong.com/posts/wAczufCpMdaamF9fy/my-objections-to-we-re-all-gonna-die-with-eliezer-yudkowsky“AI is Centralizing by Default; Let’s Not Make It Worse” — https://forum.effectivealtruism.org/posts/zd5inbT4kYKivincm/ai-is-centralizing-by-default-let-s-not-make-it-worseSingular Learning Theory, Alignment Forum sequence — https://www.alignmentforum.org/s/mqwA5FcL6SrHEQzoxDoom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

My new interview on Robert Wright's Nonzero Podcast where we dive into the agentic AI explosion. Bob is an exceptionally sharp interviewer who connects the dots between job displacement and AI doom.I highly recommend becoming a premium subscriber to Bob’s Nonzero Newsletter so you can watch the Overtime segment in every interview he does — https://nonzero.orgOur discussion continues in Overtime for premium subscribers — https://www.nonzero.org/p/early-access-the-allure-and-dangerLinksNonzero Podcast on YouTube — https://www.youtube.com/@nonzeroRobert Wright, The God Test (book, Amazon) — https://www.amazon.com/God-Test-Artificial-Intelligence-Reckoning/dp/1668061651Timestamps00:00:00 — Introduction and Today's Topics00:03:22 — Vibe Coding and the Agentic Revolution00:08:57 — The Future of Employment00:17:57 — Agents and What They Can Do00:27:59 — The "Can It" and "Will It" Framework for AI Doom00:30:27 — OpenClaw and Liron's Experience with AI Agents00:36:45 — The Case for Slowing Down AI Development00:43:28 — Anthropic, the Pentagon, and AI Politics00:48:37 — AI Safety Leadership Concerns00:52:06 — Closing and Overtime TeaseDoom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

Noah Smith is an economist and author of Noahpinion, one of the most popular Substacks in the world.He returns to Doom Debates to share a massive update to his P(Doom), and things get a little heated.Timestamps00:00:00 — Cold Open00:00:41 — Welcome Back Noah Smith!00:01:40 — Noah's P(Doom) Update00:03:57 — The Chatbot-Genie-God Framework00:05:14 — What's Your P(Doom)™00:09:59 — Unpacking Noah's Update00:16:56 — Why Incidents of Rogue AI Lower P(Doom)00:20:04 — Noah's Mainline Doom Scenario: Much Worse Than COVID-1900:23:29 — Society Responds After Growing Pains00:29:25 — Agentic AI Contributed to Noah's Position00:31:35 — Should Yudkowsky Get Bayesian Credit?00:33:59 — Are We Communicating the Right Way with Policymakers?00:40:16 — Finding Common Ground on AI Policy00:47:07 — Wrap-Up: People Need to Be More ScaredLinksDoom Debate with Noah Smith — Part 1: https://www.youtube.com/watch?v=AwmJ-OnK2I4Noah’s Twitter — https://x.com/noahpinionNoah’s Substack — https://noahpinion.blogDoom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

Dr. Claire Berlinski is a journalist, Oxford PhD, and author of The Cosmopolitan Globalist.She invited me to her weekly symposium to make the case for AI as an existential risk. Can we convince her sharp, skeptical audience that P(Doom) is high?Subscribe to The Cosmopolitan Globalist: https://claireberlinski.substack.com/Follow Claire on X: https://x.com/ClaireBerlinski“If Anyone Builds It, Everyone Dies” by Eliezer Yudkowsky & Nate Soares — https://ifanyonebuildsit.comTimestamps00:00:00 — Introduction00:02:10 — Welcome and Setting the Stage00:06:16 — Outcome Steering: The Magic of Intelligence00:10:40 — Collective Intelligence and the Path to ASI00:12:53 — The Five-Point Argument00:14:56 — The Alignment Problem and Control00:17:56 — The Genie Problem and Recursive Self-Improvement00:20:38 — Timeline: Five Years or Fifty?00:26:14 — Social Revolution and Pausing AI00:28:54 — Energy Constraints and Resource Limits00:31:23 — Morality, Empathy, and Superintelligence00:37:45 — How AI Is Actually Built00:38:31 — Computational Irreducibility and Co-Evolution00:44:57 — Foom and the Discontinuity Question00:46:44 — US-China Rivalry and the Arms Race00:49:36 — The Co-Evolution Argument00:55:36 — Alignment as Psychoanalysis00:57:24 — Anthropic’s “Harmless Slop” Paper01:00:33 — Policy Solutions: The Pause Button01:04:47 — Military AI and the Singularity01:07:10 — Cognitive Obstacles and Doom Fatigue01:09:07 — Why People Don’t Act01:13:00 — Reaching Representatives and Building a Platform01:17:12 — Sam Altman and the Manhattan Project Parallel01:19:14 — Community Building and Pause AI01:22:07 — Call to Action and ClosingDoom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

Fan favorite Dr. Steven Byrnes returns to discuss recent AI progress and the concerning paradigm shift to "ruthless sociopath AI" he sees on the horizon.Steven Byrnes, UC Berkeley physics PhD and Harvard physics postdoc, is an AI safety researcher at the Astera Institute and one of the most rigorous thinkers working on the technical AI alignment problem.Timestamps00:00:00 — Cold Open00:00:48 — Welcoming Back the Returning Champion00:02:38 — Research Update: What's New in The Last 6 Months00:04:31 — The Rise of AI Agents00:07:49 — What's Your P(Doom)?™00:13:42 — "Brain-Like AGI": The Next Generation of AI00:17:01 — Can LLMs Ever Match the Human Brain?00:31:51 — Will AI Kill Us Before It Takes Our Jobs?00:36:12 — Country of Geniuses in a Data Center00:41:34 — Why We Should Expect "Ruthless Sociopathic" ASI00:54:15 — Post-Training & RLVR — A "Thin Layer" of Real Intelligence01:02:32 — Consequentialism and the Path to Superintelligence01:17:02 — Airplanes vs. Rockets: An Analogy for AI01:24:33 — FOOM and Recursive Self-ImprovementLinksSteven Byrnes’ Website & Research— https://sjbyrnes.com/Steve’s X—https://x.com/steve47285Astera Institute—https://astera.org/“Why We Should Expect Ruthless Sociopath ASI” — https://www.lesswrong.com/posts/ZJZZEuPFKeEdkrRyf/why-we-should-expect-ruthless-sociopath-asiIntro to Brain-Like-AGI Safety—https://www.alignmentforum.org/s/HzcM2dkCq7fwXBej8Steve on LessWrong—https://www.lesswrong.com/users/steve2152AI 2027 — Scenario Timeline — https://ai-2027.com/Part 1: “The Man Who Might SOLVE AI Alignment”—https://www.youtube.com/watch?v=_ZRUq3VEAc0Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

Multiple live callers join this month's Q&A as we cover the imminent demise of programming as a profession, the Anthropic/Pentagon showdown, and debate the finer details of wireheading. I clarify my recent AI doom belief updates, and then the man behind Roko's Basilisk crashes the stream to argue I haven't updated nearly far enough.Timestamps00:00:00 — Cold Open00:00:56 — Welcome to the Livestream & Taking Questions from Chat00:12:44 — Anonymous Caller Asks If Rationalists Should Prioritize Attention-Grabbing Protests00:18:30 — The Good Case Scenario00:26:00 — Hugh Chungus Joins the Stream00:30:54 — Producer Ori, Liron's Recent Alignment Updates00:43:47 — We're In an Era of Centaurs00:47:40 — Noah Smith's Updates on AGI and Alignment00:48:44 — Co Co Chats Cybersecurity00:57:32 — The Attacker's Advantage in Offense/Defense Balance01:02:55 — Anthropic vs The Pentagon01:06:20 — "We're Getting Frog Boiled"01:11:06 — Stoner AI & Debating the Finer Points of Wireheading01:25:00 — A Caller Backs the Penrose Argument01:34:01 — Greyson Dials In01:40:21 — Surprise Guest Joins & Says Alignment Isn't a Problem02:05:15 — More Q&A with Chat02:14:26 — Closing ThoughtsLinks* Liron on X — https://x.com/liron* AI 2027 — https://ai-2027.com/* “Good Luck, Have Fun, Don’t Die” (film) — https://www.imdb.com/title/tt38301748/* “The AI Doc” (film) — https://www.focusfeatures.com/the-ai-doc-or-how-i-became-an-apocaloptimistDoom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

Professor Moshe Vardi thinks AI will kill us with kindness by automating away our jobs. I think they'll just kill us for real.Who’s right? Tune into this episode and decide where you get off the Doom Train™.Some highlights of Professor Vardi’s impressive CV:* University Professor at Rice — a rare distinction that lets him teach in any department.* 65,000+ citations, an H-index above 100, and nearly 50 years spent mechanizing reasoning, which makes him one of the most decorated computer scientists alive.* He ran the ACM’s flagship publication for a decade, and now bridges CS and policy at Rice’s Baker Institute.* He has been sounding the alarm on AI-driven job automation for over ten years.* He signed the 2023 AI extinction risk statement, and calls himself “part of the resistance.”Links* Moshe Vardi’s Wikipedia — https://en.wikipedia.org/wiki/Moshe_Vardi* Moshe Vardi, Rice University — https://profiles.rice.edu/faculty/moshe-y-vardi* Baker Institute for Public Policy — https://www.bakerinstitute.org/* Nick Bostrom, Deep Utopia: Life and Meaning in a Solved World — https://www.amazon.com/Deep-Utopia-Life-Meaning-Solved/dp/1646871642* Joseph Aoun, Robot-Proof: Higher Education in the Age of Artificial Intelligence — https://www.amazon.com/Robot-Proof-Higher-Education-Artificial-Intelligence/dp/0262535971Timestamps00:00:00 — Cold Open00:00:54 — Introducing Professor Vardi00:02:01 — Professor Vardi’s Academic Focus: CS, AI, & Public Policy00:07:18 — What’s Your P(Doom)™?00:12:28 — We’re Not Doomed, “We’re Screwed”00:16:44 — AI’s Impact on Meaning & Purpose00:27:47 — Let’s Ride the Doom Train ™00:35:43 — The Future of Jobs00:39:24 — A Country of Geniuses in a Data Center00:41:04 — Corporations as Superintelligence00:45:49 — Agency, Consciousness, and the Limits of AI00:50:07 — The Mad Scientist Scenario00:54:02 — Could a Data Center of Geniuses Destroy Humanity?01:03:13 — The WALL-E Meme and Fun Theory01:04:01 — Why Professor Vardi Signed the AI Extinction Risk Statement01:06:02 — Wrap-Up + 1 Way Ticket to DoomDoom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

Fresh off my debate with Destiny, his Discord community invited me into their voice chat to talk about AI doom. Just like the man himself, his fans are sharp.Let's find out where they get off The Doom Train™.My recent debate with Destiny — https://www.youtube.com/watch?v=rNgffLZTeWwTimestamps00:00:00 — Cold Open00:00:54 — Liron Joins Destiny’s Discord00:02:21 — The AI Doom Premise00:03:27 — Defining Intelligence and Is An LLM Really AI?00:07:12 — Will AI Become Uncontrollable?00:12:44 — The AI Alignment Problem00:24:11 — The Difficulty of Pausing AI00:26:01 — AI vs The Human Brain00:32:41 — Future AI Capabilities, Steering Toward Goals, & Philosophical DisagreementsDoom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

Renowned scientists just set The Doomsday Clock closer than ever to midnight.I agree our collective doom seems imminent, but why do the clocksetters point to climate change—not rogue AI—as an existential threat?UChicago Professor Daniel Holz, a top physicist who sets the Doomsday Clock, joins me to debate society’s top existential risks.00:00:00 — Cold Open 00:00:51 — Introducing Professor Holz00:02:08 — The Doomsday Clock is at 85 Seconds to Midnight! 00:04:37 — What's Your P(Doom)?™ 00:08:09 — Making A Probability of Doomsday, or P(Doom), Equation 00:12:07 — How We All Die: Nuclear vs Climate vs AI 00:21:08 — Nuclear Close Calls from The Cold War 00:28:38 — History of The Doomsday Clock 00:30:18 — The Threat of Biological Risks Like Mirror Life 00:33:40 — Professor Holz’s Position on AI Misalignment Risk 00:44:49 — Does The Doomsday Clock Give Short Shrift to AI Misalignment Risk? 00:59:09 — Why Professor Holz Founded UChicago’s Existential Risk Laboratory (XLab) 01:06:22 — The State of Academic Research on AI Safety & Existential Risks 01:12:32 — The Case for Pausing AI Development 01:17:11 — Debate: Is Climate Change an Existential Threat? 01:28:48 — Call to Action: How to Reduce Our Collective ThreatLinksProfessor Daniel Holz’s Wikipedia — https://en.wikipedia.org/wiki/Daniel_HolzXLab (Existential Risk Laboratory) — https://xrisk.uchicago.edu/2026 Doomsday Clock Statement — https://thebulletin.org/doomsday-clock/2026-statement/The Bulletin of the Atomic Scientists (Nonprofit that produces the clock)— https://thebulletin.org/UChicago Magazine features Prof. Holz’s class — “Are We Doomed?”: https://mag.uchicago.edu/science-medicine/are-we-doomedThe Doomsday Machine by Daniel Ellsberg — https://www.amazon.com/Doomsday-Machine-Confessions-Nuclear-Planner/dp/1608196704If Anyone Builds It, Everyone Dies by Eliezer Yudkowsky & Nate Soares — https://www.amazon.com/Anyone-Builds-Everyone-Dies-Superhuman/dp/0316595640Learn more about pausing frontier AI development from PauseAI — https://pauseai.infoDoom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

Liron was invited to give a presentation to some of the most promising creators in AI safety communications.Get a behind-the-scenes look at Doom Debates and how the channel has grown so quickly. Learn more about the Frame Fellowship: https://framefellowship.com/Timestamps00:00:00 — Liron tees up the presentation00:02:28 — Liron’s Frame Fellowhip Presentation00:06:03 — Introducing Doom Debates00:07:44 — Meeting the Frame Fellows00:19:38 — Why I Started Doom Debates00:30:20 — Handling Hate and Criticism00:31:56 — The Talent Stack00:39:34 — Finding Your Unique Niche00:40:58 — Q&A00:42:04 — On Funding the Show00:48:05 — Audience Demographics & Gender Strategy00:51:17 — How to communicate AI Risk Effectively00:56:22 — Social Media Strategy00:58:10 — Closing RemarksDoom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

Destiny has racked up millions of views for his sharp takes on political and cultural news. Now, I finally get to ask him about a topic he’s been agnostic on: Will AI end humanity?Come ride with us on the Doom Train™ 🚂Timestamps00:00:00 — Teaser00:01:16 — Welcoming Destiny00:02:54 — What’s Your P(Doom)?™00:04:46 — 2017 vs 2026: Destiny’s views on AI00:11:04 — AI could vastly surpass human intelligence00:16:02 — Can AI doom us?00:18:42 — Intelligence doesn’t guarantee morality00:22:18 — The vibes-based case against doom00:29:58 — The human brain is inefficient00:35:17 — Does every intelligence in the universe self-destruct via AI?00:37:28 — Destiny turns the tables: Where does Liron get off The Doom Train™00:46:07 — Will a warning shot cause society to develop AI safely?00:54:10 — Roko’s Basilisk, the AI box problem00:59:37 — Will Destiny update his P(Doom)?™01:04:19 — Closing thoughtsLinksDestiny's YouTube — https://www.youtube.com/@destinyDestiny's X account — https://x.com/TheOmniLiberalMarc Andreessen saying AI isn’t dangerous because “it is math” — https://a16z.com/ai-will-save-the-world/Will Smith eating spaghetti AI video — https://knowyourmeme.com/memes/ai-will-smith-eating-spaghettiRoko’s Basilisk on LessWrong — https://www.lesswrong.com/tag/rokos-basiliskEliezer Yudkowsky’s AI Box Experiment — https://www.yudkowsky.net/singularity/aiboxDoom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

On the eve of superintelligence, real AI safety is a nonexistent field. I break down why in my latest essay, The Facade of AI Safety Will Crumble.This is a podcast reading of a recent Substack post: https://lironshapira.substack.com/p/the-facade-of-ai-safety-will-crumbleDoom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

Elon Musk just made a stunning admission about the insane future he’s steering us toward.In a new interview with Dwarkesh Patel and John Collison on the Cheeky Pint podcast, Elon said that humanity can’t expect to be “in charge” of AI for long, because humans will soon only have 1% of the combined total human+AI intelligence.Then, he claimed to have a plan to build AI overlords that will naturally support humanity's flourishing.In this mini episode, I react to Elon's remarks and expose why his plan for humanity's survival in the age of AI is dangerously flimsy.My original tweet: https://x.com/liron/status/2020906862397489492Clip source: Elon Musk – "In 36 months, the cheapest place to put AI will be space”Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

California gubernatorial candidate Zoltan Istvan reveals his P(Doom) and makes the case for universal basic income and radical life extension.Timestamps00:00:00 — Teaser00:00:50 — Meet Zoltan, Democratic Candidate for California Governor00:08:30 — The 2026 California Governor's Race00:12:50 — Zoltan's Platform Is Automated Abundance00:19:45 — What's Your P(Doom)™00:28:26 — Campaigning on Existential Risk00:32:36 — Does Zoltan Support a Global AI Pause?00:48:39 — Exploring His Platform: Education, Crime, and Affordability01:08:55 — Exploring His Platform:: Super Cities, Space, and Longevity01:13:00 — Closing ThoughtsLinksZoltan Istvan’s Campaign for California Governor – zoltanistvan2026.comThe Transhumanist Wager by Zoltan Istvan – https://www.amazon.com/Transhumanist-Wager-Zoltan-Istvan/dp/0988616114Wired Article on the “Bunker” Party – https://www.wired.com/story/ai-risk-party-san-francisco/PauseAI – pauseai.infoSL4 Mailing List Archive – sl4.org/archiveDoom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

Get ready for a rematch with the one & only Bentham’s Bulldog, a.k.a. Matthew Adelstein! Our first debate covered a wide range of philosophical topics.Today’s Debate #2 is all about Matthew’s new argument against the inevitability of AI doom. He comes out swinging with a calculated P(Doom) of just 2.6% , based on a multi-step probability chain that I challenge as potentially falling into a “Type 2 Conjunction Fallacy” (a.k.a. Multiple Stage Fallacy).We clash on whether to expect “alignment by default” and the nature of future AI architectures. While Matthew sees current RLHF success as evidence that AIs will likely remain compliant, I argue that we’re building “Goal Engines” — superhuman optimization modules that act like nuclear cores wrapped in friendly personalities. We debate whether these engines can be safely contained, or if the capability to map goals to actions is inherently dangerous and prone to exfiltration.Despite our different forecasts (my 50% vs his sub-10%), we actually land in the “sane zone” together on some key policy ideas, like the potential necessity of a global pause.While Matthew’s case for low P(Doom) hasn’t convinced me, I consider his post and his engagement with me to be super high quality and good faith. We’re not here to score points, we just want to better predict how the intelligence explosion will play out.Timestamps00:00:00 — Teaser00:00:35 — Bentham’s Bulldog Returns to Doom Debates00:05:43 — Higher-Order Evidence: Why Skepticism is Warranted00:11:06 — What’s Your P(Doom)™00:14:38 — The “Multiple Stage Fallacy” Objection00:21:48 — The Risk of Warring AIs vs. Misalignment00:27:29 — Historical Pessimism: The “Boy Who Cried Wolf”00:33:02 — Comparing AI Risk to Climate Change & Nuclear War00:38:59 — Alignment by Default via Reinforcement Learning00:46:02 — The “Goal Engine” Hypothesis00:53:13 — Is Psychoanalyzing Current AI Valid for Future Systems?01:00:17 — Winograd Schemas & The Fragility of Value01:09:15 — The Nuclear Core Analogy: Dangerous Engines in Friendly Wrappers01:16:16 — The Discontinuity of Unstoppable AI01:23:53 — Exfiltration: Running Superintelligence on a Laptop01:31:37 — Evolution Analogy: Selection Pressures for Alignment01:39:08 — Commercial Utility as a Force for Constraints01:46:34 — Can You Isolate the “Goal-to-Action” Module?01:54:15 — Will Friendly Wrappers Successfully Control Superhuman Cores?02:04:01 — Moral Realism and Missing Out on Cosmic Value02:11:44 — The Paradox of AI Solving the Alignment Problem02:19:11 — Policy Agreements: Global Pauses and China02:26:11 — Outro: PauseCon DC 2026 PromoLinksBentham’s Bulldog Official Substack — https://benthams.substack.comThe post we debated — https://benthams.substack.com/p/against-if-anyone-builds-it-everyoneApply to PauseCon DC 2026 here or via https://pauseai-us.orgForethought Institute’s paper: Preparing for the Intelligence ExplosionTom Davidson (Forethought Institute)’s post: How quick and big would a software intelligence explosion be?Scott Alexander on the Coffeepocalypse Argument---Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

Harlan Stewart works in communications for the Machine Intelligence Research Institute (MIRI). In this episode, Harlan and I give our honest opinions on Dario Amodei's new essay "The Adolescence of Technology".Timestamps0:00:00 — Cold Open0:00:47 — How Harlan Stewart Got Into AI Safety0:02:30 — What’s Your P(Doom)?™0:04:09 — The “Doomer” Label0:06:13 — Overall Reaction to Dario’s Essay: The Missing Mood0:09:15 — The Rosy Take on Dario’s Essay0:10:42 — Character Assassination & Low Blows0:13:39 — Dario Amodei is Shifting the Overton Window in The Wrong Direction0:15:04 — Object-Level vs. Meta-Level Criticisms0:17:07 — The “Inevitability” Strawman Used by Dario0:19:03 — Dario Refers to Doom as a Self-Fulfilling Prophecy0:22:38 — Dismissing Critics as “Too Theoretical”0:43:18 — The Problem with Psychoanalyzing AI0:56:12 — “Intellidynamics” & Reflective Stability1:07:12 — Why Is Dario Dismissing an AI Pause?1:11:45 — Final TakeawaysLinksHarlan’s X — https://x.com/HumanHarlan“The Adolescence of Technology” by Dario Amodei — https://www.darioamodei.com/essay/the-adolescence-of-technologyDoom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

Check out the new Doom Debates studio in this Q&A with special guest Producer Ori! Liron gets into a heated discussion about whether doomers must validate short-term risks, like data center water usage, in order to build a successful political coalition.Originally streamed on Saturday, January 24.Timestamps00:00:00 — Cold Open00:00:26 — Introduction and Studio Tour00:08:17 — Q&A: Alignment, Accelerationism, and Short-Term Risks00:18:15 — Dario Amodei, Davos, and AI Pause00:27:42 — Producer Ori Joins: Locations and Vibes00:35:31 — Legislative Strategy vs. Social Movements (The Tobacco Playbook)00:45:01 — Ethics of Investing in or Working for AI Labs00:54:23 — Defining Superintelligence and Human Limitations01:02:58 — Technical Risks: Self-Replication and Cyber Warfare01:19:08 — Live Debate with Zane: Short-Term vs. Long-Term Strategy01:53:15 — Marketing Doom Debates and Guest Outreach01:56:45 — Live Call with Jonas: Scenarios for Survival02:05:52 — Conclusion and Mission StatementLinksLiron’s X Post about Destiny — https://x.com/liron/status/2015144778652905671?s=20Why Laws, Treaties, and Regulations Won’t Save Us from AI | For Humanity Ep. 77 — https://www.youtube.com/watch?v=IUX00c5x2UMDoom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

Audrey Tang was the youngest minister in Taiwanese history. Now she's working to align AI with democratic principles as Taiwan's Cyber Ambassador.In this debate, I probe her P(doom) and stress-test her vision for safe AI development.Timestamps00:00:00 — Episode Preview00:01:43 — Introducing Audrey Tang, Cyber Ambassador of Taiwan00:07:20 — Being Taiwan’s First Digital Minister00:17:19 — What's Your P(Doom)? ™00:21:10 — Comparing AI Risk to Nuclear Risk00:22:53 — The Statement on AI Extinction Risk00:27:29 — Doomerism as a Hyperstition00:30:51 — Audrey Explains Her Vision of "Plurality"00:37:17 — Audrey Explains Her Principles of Civic Ethics, The "6-Pack of Care"00:45:58 — AGI Timelines: "It's Already Here"00:54:41 — The Apple Analogy01:03:09 — What If AI FOOMs?01:11:19 — What AI Can vs What AI Will Do01:15:20 — Lessons from COVID-1901:19:59 — Is Society Ready? Audrey Reflects on a Personal Experience with Mortality01:23:50 — AI Alignment Cannot Be Top-Down01:34:04 — AI-as-Mother vs AI-as-Gardener01:37:26 — China and the Geopolitics of AI Chip Manufacturing in Taiwan01:40:47 — Red Lines, International Treaties, and the Off Button01:48:26 — Debate Wrap-UpLinksPlurality: The Future of Collaborative Technology and Democracy by Glen Weyl and Audrey Tang — https://www.amazon.com/Plurality-Future-Collaborative-Technology-Democracy/dp/B0D98RPKCKAudrey’s X — https://x.com/audreytAudrey’s Wikipedia — https://en.wikipedia.org/wiki/Audrey_TangDoom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

I joined Steve Bannon’s War Room Battleground to talk about AI doom.Hosted by Joe Allen, we cover AGI timelines, raising kids with a high p(doom), and why improving our survival odds requires a global wake-up call.00:00:00 — Episode Preview00:01:17 — Joe Allen opens the show and introduces Liron Shapira00:04:06 — Liron: What’s Your P(Doom)?00:05:37 — How Would an AI Take Over?00:07:20 — The Timeline to AGI00:08:17 — Benchmarks & AI Passing the Turing Test00:14:43 — Liron Is Typically a Techno-Optimist00:18:00 — Raising a Family with a High P(Doom)00:23:48 — Mobilizing a Grassroots AI Survival Campaign00:26:45 — Final Message: A Wake-Up Call00:29:23 — Joe Allen’s Closing Message to the War Room PosseLinks: Joe’s Substack — https://substack.com/@joebotJoe’s Twitter — https://x.com/JOEBOTxyzBannon’s War Room Twitter — https://x.com/Bannons_WarRoomWarRoom Battleground EP 922: AI Doom Debates with Liron Shapira on Rumble — https://rumble.com/v742oo4-warroom-battleground-ep-922-ai-doom-debates-with-liron-shapira.htmlDoom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

Economist Noah Smith is the author of Noahpinion, one of the most popular Substacks in the world.Far from worrying about human extinction from superintelligent AI, Noah is optimistic AI will create a world where humans still have plentiful, high-paying jobs!In this debate, I stress-test his rosy outlook. Let’s see if Noah can instill us with more confidence about humanity’s rapidly approaching AI future.Timestamps00:00:00 - Episode Preview00:01:41 - Introducing Noah Smith00:03:19 - What’s Your P(Doom)™00:04:40 - Good vs. Bad Transhumanist Outcomes00:15:17 - Catastrophe vs. Total Extinction00:17:15 - Mechanisms of Doom00:27:16 - The AI Persuasion Risk00:36:20 - Instrumental Convergence vs. Peace00:53:08 - The “One AI” Breakout Scenario01:01:18 - The “Stoner AI” Theory01:08:49 - Importance of Reflective Stability01:14:50 - Orthogonality & The Waymo Argument01:21:18 - Comparative Advantage & Jobs01:27:43 - Wealth Distribution & Robot Lords01:34:34 - Supply Curves & Resource Constraints01:43:38 - Policy of Reserving Human Resources01:48:28 - Closing: The Case for OptimismLinksNoah’s Substack — https://noahpinion.blog“Plentiful, high-paying jobs in the age of AI” — https://www.noahpinion.blog/p/plentiful-high-paying-jobs-in-the“My thoughts on AI safety” — https://www.noahpinion.blog/p/my-thoughts-on-ai-safetyNoah’s Twitter — https://x.com/noahpinion---Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

In September of 2023, when OpenAI’s GPT-4 was still a fresh innovation and people were just beginning to wrap their heads around large language models, I was invited to debate Beff Jezos, Bayeslord, and other prominent “effective accelerationists” a.k.a. “e/acc” folks on an X Space.E/acc’s think building artificial superintelligence is unlikely to disempower humanity and doom the future, because that’d be an illegal exception to the rule that accelerating new technology is always the highest-expected-value for humanity.As you know, I disagree — I think doom is extremely likely and imminent possibility.This debate took place 9 months before I started Doom Debates, and was one of the experiences that made me realize debating AI doom was my calling. It’s also the only time Beff Jezos has ever not been too chicken to debate me.Timestamps00:00:00 — Liron’s New Intro00:04:15 — Debate Starts Here: Litigating FOOM00:06:18 — Defining the Recursive Feedback Loop00:15:05 — The Two-Part Doomer Thesis00:26:00 — When Does a Tool Become an Agent?00:44:02 — The Argument for Convergent Architecture00:46:20 — Mathematical Objections: Ergodicity and Eigenvalues01:03:46 — Bayeslord Enters: Why Speed Doesn’t Matter01:12:40 — Beff Jezos Enters: Physical Priors vs. Internet Data01:13:49 — The 5% Probability of Doom by GPT-501:20:09 — Chaos Theory and Prediction Limits01:27:56 — Algorithms vs. Hardware Constraints01:35:20 — Galactic Resources vs. Human Extermination01:54:13 — The Intelligence Bootstrapping Script Scenario02:02:13 — The 10-Megabyte AI Virus Debate02:11:54 — The Nuclear Analogy: Noise Canceling vs. Rubble02:37:39 — Controlling Intelligence: The Roman Empire Analogy02:44:53 — Real-World Latency and API Rate Limits03:03:11 — The Difficulty of the Off Button03:24:47 — Why Liron is “e/acc at Heart”Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

AGI timelines, offense/defense balance, evolution vs engineering, how to lower P(Doom), Eliezer Yudkowksy, and much more!Timestamps:00:00 Trailer03:10 Is My P(Doom) Lowering?11:29 First Caller: AI Offense vs Defense Balance16:50 Superintelligence Skepticism25:05 Agency and AI Goals29:06 Communicating AI Risk36:35 Attack vs Defense Equilibrium38:22 Can We Solve Outer Alignment?54:47 What is Your P(Pocket Nukes)?1:00:05 The “Shoggoth” Metaphor Is Outdated1:06:23 Should I Reframe the P(Doom) Question?1:12:22 How YOU Can Make a Difference1:24:43 Can AGI Beat Biology?1:39:22 Agency and Convergent Goals1:59:56 Viewer Poll: What Content Should I Make?2:26:15 AI Warning Shots2:32:12 More Listener Questions: Debate Tactics, Getting a PhD, Specificity2:53:53 Closing ThoughtsLinks:Support PauseAI — https://pauseai.info/Support PauseAI US — https://www.pauseai-us.org/Support LessWrong / Lightcone Infrastructure — LessWrong is fundraising!Support MIRI — MIRI’s 2025 FundraiserAbout the show:Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

Devin Elliot is a former pro snowboarder turned software engineer who has logged thousands of hours building AI systems. His P(Doom) is a flat ⚫. He argues that worrying about an AI takeover is as irrational as fearing your car will sprout wings and fly away.We spar over the hard limits of current models: Devin insists LLMs are hitting a wall, relying entirely on external software “wrappers” to feign intelligence. I push back, arguing that raw models are already demonstrating native reasoning and algorithmic capabilities.Devin also argues for decentralization by claiming that nuclear proliferation is safer than centralized control.We end on a massive timeline split: I see superintelligence in a decade, while he believes we’re a thousand years away from being able to “grow” computers that are truly intelligence.Timestamps00:00:00 Episode Preview00:01:03 Intro: Snowboarder to Coder00:03:30 "I Do Not Have a P(Doom)"00:06:47 Nuclear Proliferation & Centralized Control00:10:11 The "Spotify Quality" House Analogy00:17:15 Ideal Geopolitics: Decentralized Power00:25:22 Why AI Can't "Fly Away"00:28:20 The Long Addition Test: Native or Tool?00:38:26 Is Non-Determinism a Feature or a Bug?00:52:01 The Impossibility of Mind Uploading00:57:46 "Growing" Computers from Cells01:02:52 Timelines: 10 Years vs. 1,000 Years01:11:40 "Plastic Bag Ghosts" & Builder Intuition01:13:17 Summary of the Debate01:15:30 Closing ThoughtsLinksDevin’s Twitter — https://x.com/devinjelliot---Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

Dr. Michael Timothy Bennett, Ph.D, is an award-winning young researcher who has developed a new formal framework for understanding intelligence. He has a TINY P(Doom) because he claims superintelligence will be resource-constrained and tend toward cooperation.In this lively debate, I stress-test Michael’s framework and debate whether its theorized constraints will actually hold back superintelligent AI.Timestamps* 00:00 Trailer* 01:41 Introducing Michael Timothy Bennett* 04:33 What’s Your P(Doom)?™* 10:51 Michael’s Thesis on Intelligence: “Abstraction Layers”, “Adaptation”, “Resource Efficiency”* 25:36 Debate: Is Einstein Smarter Than a Rock?* 39:07 “Embodiment”: Michael’s Unconventional Computation Theory vs Standard Computation* 48:28 “W-Maxing”: Michael’s Intelligence Framework vs. a Goal-Oriented Framework* 59:47 Debating AI Doom* 1:09:49 Debating Instrumental Convergence* 1:24:00 Where Do You Get Off The Doom Train™ — Identifying The Cruxes of Disagreement* 1:44:13 Debating AGI Timelines* 1:49:10 Final RecapLinksMichael’s website — https://michaeltimothybennett.comMichael’s Twitter — https://x.com/MiTiBennettMichael’s latest paper, “How To Build Conscious Machines” — https://osf.io/preprints/thesiscommons/wehmg_v1?view_onlyDoom Debates' Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

My guest today achieved something EXTREMELY rare and impressive: Coming onto my show with an AI optimist position, then admitting he hadn’t thought of my counterarguments before, and updating his beliefs in realtime! Also, he won the 2013 Nobel Prize in computational biology.I’m thrilled that Prof. Levitt understands the value of raising awareness about imminent extinction risk from superintelligent AI, and the value of debate as a tool to uncover the truth — the dual missions of Doom Debates!Timestamps0:00 — Trailer1:18 — Introducing Michael Levitt4:20 — The Evolution of Computing and AI12:42 — Measuring Intelligence: Humans vs. AI23:11 — The AI Doom Argument: Steering the Future25:01 — Optimism, Pessimism, and Other Existential Risks34:15 — What’s Your P(Doom)™36:16 — Warning Shots and Global Regulation55:28 — Comparing AI Risk to Pandemics and Nuclear War1:01:49 — Wrap-Up1:06:11 — Outro + New AI safety resourceShow NotesMichael Levitt’s Twitter — https://x.com/MLevitt_NP2013-- Get full access to Doom Debates at lironshapira.substack.com/subscribe

Michael Ellsberg, son of the legendary Pentagon Papers leaker Daniel Ellsberg, joins me to discuss the chilling parallels between his father’s nuclear war warnings and today’s race to AGI.We discuss Michael’s 99% probability of doom, his personal experience being “obsoleted” by AI, and the urgent moral duty for insiders to blow the whistle on AI’s outsize risks.Timestamps0:00 Intro1:29 Introducing Michael Ellsberg, His Father Daniel Ellsberg, and The Pentagon Papers5:49 Vietnam War Parallels to AI: Lies and Escalation25:23 The Doomsday Machine & Nuclear Insanity48:49 Mutually Assured Destruction vs. Superintelligence Risk55:10 Evolutionary Dynamics: Replicators and the End of the “Dream Time”1:10:17 What’s Your P(doom)?™1:14:49 Debating P(Doom) Disagreements1:26:18 AI Unemployment Doom1:39:14 Doom Psychology: How to Cope with Existential Risk1:50:56 The “Joyless Singularity”: Aligned AI Might Still Freeze Humanity2:09:00 A Call to Action for AI InsidersShow Notes:Michael Ellsberg’s website — https://www.ellsberg.com/Michael’s Twitter — https://x.com/MichaelEllsbergDaniel Ellsberg’s website — https://www.ellsberg.net/The upcoming book, “Truth and Consequence” — https://geni.us/truthandconsequenceMichael’s AI-related substack “Mammalian Wetware” — https://mammalianwetware.substack.com/Daniel’s debate with Bill Kristol in the run-up to the Iraq war — https://www.youtube.com/watch?v=HyvsDR3xnAg--Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

Today's Debate: Should we ban the development of artificial superintelligence until scientists agree it is safe and controllable?Arguing FOR banning superintelligence until there’s a scientific consensus that it’ll be done safely and controllably and with strong public buy-in: Max Tegmark. He is an MIT professor, bestselling author, and co-founder of the Future of Life Institute whose research has focused on artificial intelligence for the past 8 years.Arguing AGAINST banning superintelligent AI development: Dean Ball. He is a Senior Fellow at the Foundation for American Innovation who served as a Senior Policy Advisor at the White House Office of Science and Technology Policy under President Trump, where he helped craft America’s AI Action Plan.Two of the leading voices on AI policy engaged in high-quality, high-stakes debate for the benefit of the public!This is why I got into the podcast game — because I believe debate is an essential tool for humanity to reckon with the creation of superhuman thinking machines.Timestamps0:00 - Episode Preview1:41 - Introducing The Debate3:38 - Max Tegmark’s Opening Statement5:20 - Dean Ball’s Opening Statement9:01 - Designing an “FDA for AI” and Safety Standards21:10 - Liability, Tail Risk, and Biosecurity29:11 - Incremental Regulation, Timelines, and AI Capabilities54:01 - Max’s Nightmare Scenario57:36 - The Risks of Recursive Self‑Improvement1:08:24 - What’s Your P(Doom)?™1:13:42 - National Security, China, and the AI Race1:32:35 - Closing Statements1:44:00 - Post‑Debate Recap and Call to ActionShow NotesStatement on Superintelligence released by Max’s organization, the Future of Life Institute — https://superintelligence-statement.org/Dean’s reaction to the Statement on Superintelligence — https://x.com/deanwball/status/1980975802570174831America’s AI Action Plan — https://www.whitehouse.gov/articles/2025/07/white-house-unveils-americas-ai-action-plan/“A Definition of AGI” by Dan Hendrycks, Max Tegmark, et. al. —https://www.agidefinition.ai/Max Tegmark’s Twitter — https://x.com/tegmarkDean Ball’s Twitter — https://x.com/deanwball Get full access to Doom Debates at lironshapira.substack.com/subscribe

Max Harms and Jeremy Gillen are current and former MIRI researchers who both see superintelligent AI as an imminent extinction threat.But they disagree on whether it’s worthwhile to try to aim for obedient, “corrigible” AI as a singular target for current alignment efforts.Max thinks corrigibility is the most plausible path to build ASI without losing control and dying, while Jeremy is skeptical that this research path will yield better superintelligent AI behavior on a sufficiently early try.By listening to this debate, you’ll find out if AI corrigibility is a relatively promising effort that might prevent imminent human extinction, or an over-optimistic pipe dream.Timestamps0:00 — Episode Preview1:18 — Debate Kickoff3:22 — What is Corrigibility?9:57 — Why Corrigibility Matters11:41 — What’s Your P(Doom)™16:10 — Max’s Case for Corrigibility19:28 — Jeremy’s Case Against Corrigibility21:57 — Max’s Mainline AI Scenario28:51 — 4 Strategies: Alignment, Control, Corrigibility, Don’t Build It37:00 — Corrigibility vs HHH (”Helpful, Harmless, Honest”)44:43 — Asimov’s 3 Laws of Robotics52:05 — Is Corrigibility a Coherent Concept?1:03:32 — Corrigibility vs Shutdown-ability1:09:59 — CAST: Corrigibility as Singular Target, Near Misses, Iterations1:20:18 — Debating if Max is Over-Optimistic1:34:06 — Debating if Corrigibility is the Best Target1:38:57 — Would Max Work for Anthropic?1:41:26 — Max’s Modest Hopes1:58:00 — Max’s New Book: Red Heart2:16:08 — OutroShow NotesMax’s book Red Heart — https://www.amazon.com/Red-Heart-Max-Harms/dp/108822119XLearn more about CAST: Corrigibility as Singular Target — https://www.lesswrong.com/s/KfCjeconYRdFbMxsy/p/NQK8KHSrZRF5erTbaMax’s Twitter — https://x.com/raelifinJeremy’s Twitter — https://x.com/jeremygillen1---Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

Holly Elmore leads protests against frontier AI labs, and that work has strained some of her closest relationships in the AI-safety community.She says AI safety insiders care more about their reputation in tech circles than actually lowering AI x-risk. This is our full conversation from my “If Anyone Builds It, Everyone Dies” unofficial launch party livestream on Sept 16, 2025.Timestamps0:00 Intro1:06 Holly’s Background and The Current Activities of PauseAI US4:41 The Circular Firing Squad Problem of AI Safety7:23 Why the AI Safety Community Resists Public Advocacy11:37 Breaking with Former Allies at AI Labs13:00 LessWrong’s reaction to Eliezer’s public turnShow NotesPauseAI US — https://pauseai-us.orgInternational PauseAI — https://pauseai.infoHolly’s Twitter — https://x.com/ilex_ulmusHolly’s Substack: https://substack.com/@hollyelmoreHolly’s post covering how AI isn’t another “technology”: https://hollyelmore.substack.com/p/the-technology-bucket-errorRelated EpisodesHolly and I dive into the rationalist community’s failure to rally behind a cause: https://lironshapira.substack.com/p/lesswrong-circular-firing-squadThe full IABED livestream: https://lironshapira.substack.com/p/if-anyone-builds-it-everyone-dies-party Get full access to Doom Debates at lironshapira.substack.com/subscribe

Sparks fly in the finale of my series with ex-MIRI researcher Tsvi Benson-Tilsen as we debate his AGI timelines.Tsvi is a champion of using germline engineering to create smarter humans who can solve AI alignment.I support the approach, even though I’m skeptical it’ll gain much traction before AGI arrives.Timestamps0:00 Debate Preview0:57 Tsvi’s AGI Timeline Prediction 3:03 The Least Impressive Task AI Cannot Do In 2 years6:13 Proposed Task: Solve Cantor’s Theorem From Scratch 8:20 AI Has Limitations Related to Sample Complexity 11:41 We Need Clear Goalposts for Better AGI Predictions 13:19 Counterargument: LLMs May Not Be a Path to AGI16:01 Is Tsvi Setting a High Bar for Progress Towards AGI? 19:17 AI Models Are Missing A Spark of Creativity28:17 Liron’s “Black Box” AGI Test 32:09 Are We Going to Enter an AI Winter? 35:09 Who Is Being Overconfident? 42:11 If AI Makes Progress on Benchmarks, Would Tsvi Shorten His Timeline? 50:34 Recap & Tsvi’s ResearchShow NotesLearn more about Tsvi’s organization, the Berkeley Genomics Project — https://berkeleygenomics.orgDoom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

Today I’m sharing my AI doom interview on Donal O’Doherty’s podcast.I lay out the case for having a 50% p(doom). Then Donal plays devil’s advocate and tees up every major objection the accelerationists throw at doomers. See if the anti-doom arguments hold up, or if the AI boosters are just serving sophisticated cope.Timestamps0:00 — Introduction & Liron’s Background 1:29 — Liron’s Worldview: 50% Chance of AI Annihilation 4:03 — Rationalists, Effective Altruists, & AI Developers 5:49 — Major Sources of AI Risk 8:25 — The Alignment Problem 10:08 — AGI Timelines 16:37 — Will We Face an Intelligence Explosion? 29:29 — Debunking AI Doom Counterarguments 1:03:16 — Regulation, Policy, and Surviving The Future With AIShow NotesIf you liked this episode, subscribe to the Collective Wisdom Podcast for more deeply researched AI interviews: https://www.youtube.com/@DonalODoherty Transcript Introduction & Liron’s BackgroundDonal O’Doherty 00:00:00Today I’m speaking with Liron Shapira. Liron is an investor, he’s an entrepreneur, he’s a rationalist, and he also has a popular podcast called Doom Debates, where he debates some of the greatest minds from different fields on the potential of AI risk.Liron considers himself a doomer, which means he worries that artificial intelligence, if it gets to superintelligence level, could threaten the integrity of the world and the human species.Donal 00:00:24Enjoy my conversation with Liron Shapira.Donal 00:00:30Liron, welcome. So let’s just begin. Will you tell us a little bit about yourself and your background, please? I will have introduced you, but I just want everyone to know a bit about you.Liron Shapira 00:00:39Hey, I’m Liron Shapira. I’m the host of Doom Debates, which is a YouTube show and podcast where I bring in luminaries on all sides of the AI doom argument.Liron 00:00:49People who think we are doomed, people who think we’re not doomed, and we hash it out. We try to figure out whether we’re doomed. I myself am a longtime AI doomer. I started reading Yudkowsky in 2007, so it’s been 18 years for me being worried about doom from artificial intelligence.My background is I’m a computer science bachelor’s from UC Berkeley.Liron 00:01:10I’ve worked as a software engineer and an entrepreneur. I’ve done a Y Combinator startup, so I love tech. I’m deep in tech. I’m deep in computer science, and I’m deep into believing the AI doom argument.I don’t see how we’re going to survive building superintelligent AI. And so I’m happy to talk to anybody who will listen. So thank you for having me on, Donal.Donal 00:01:27It’s an absolute pleasure.Liron’s Worldview: 50% Chance of AI AnnihilationDonal 00:01:29Okay, so a lot of people where I come from won’t be familiar with doomism or what a doomer is. So will you just talk through, and I’m very interested in this for personal reasons as well, your epistemic and philosophical inspirations here. How did you reach these conclusions?Liron 00:01:45So I often call myself a Yudkowskian, in reference to Eliezer Yudkowsky, because I agree with 95% of what he writes, the Less Wrong corpus. I don’t expect everybody to get up to speed with it because it really takes a thousand hours to absorb it all.I don’t think that it’s essential to spend those thousand hours.Liron 00:02:02I think that it is something that you can get in a soundbite, not a soundbite, but in a one-hour long interview or whatever. So yeah, I think you mentioned epistemic roots or whatever, right? So I am a Bayesian, meaning I think you can put probabilities on things the way prediction markets are doing.Liron 00:02:16You know, they ask, oh, what’s the chance that this war is going to end? Or this war is going to start, right? What’s the chance that this is going to happen in this sports game? And some people will tell you, you can’t reason like that.Whereas prediction markets are like, well, the market says there’s a 70% chance, and what do you know? It happens 70% of the time. So is that what you’re getting at when you talk about my epistemics?Donal 00:02:35Yeah, exactly. Yeah. And I guess I’m very curious as well about, so what Yudkowsky does is he conducts thought experiments. Because obviously some things can’t be tested, we know they might be true, but they can’t be tested in experiments.Donal 00:02:49So I’m just curious about the role of philosophical thought experiments or maybe trans-science approaches, in terms of testing questions that we can’t actually conduct experiments on.Liron 00:03:00Oh, got it. Yeah. I mean this idea of what can and can’t be tested. I mean, tests are nice, but they’re not the only way to do science and to do productive reasoning.Liron 00:03:10There are times when you just have to do your best without a perfect test. You know, a recent example was the James Webb Space Telescope, right? It’s the successor to the Hubble Space Telescope. It worked really well, but it had to get into this really difficult orbit.This very interesting Lagrange point, I think in the solar system, they had to get it there and they had to unfold it.Liron 00:03:30It was this really compact design and insanely complicated thing, and it had to all work perfectly on the first try. So you know, you can test it on earth, but earth isn’t the same thing as space.So my point is just that as a human, as a fallible human with a limited brain, it turns out there’s things you can do with your brain that still help you know the truth about the future, even when you can’t do a perfect clone of an experiment of the future.Liron 00:03:52And so to connect that to the AI discussion, I think we know enough to be extremely worried about superintelligent AI. Even though there is not in fact a superintelligent AI in front of us right now.Donal 00:04:03Interesting.Rationalists, Effective Altruists, & AI DevelopersDonal 00:04:03And just before we proceed, will you talk a little bit about the EA community and the rationalist community as well? Because a lot of people won’t have heard of those terms where I come from.Liron 00:04:13Yes. So I did mention Eliezer Yudkowsky, who’s kind of the godfather of thinking about AI safety. He was also the father of the modern rationality community. It started around 2007 when he was online blogging at a site called Overcoming Bias, and then he was blogging on his own site called Less Wrong.And he wrote The Less Wrong Sequences and a community formed around him that also included previous rationalists, like Carl Feynman, the son of Richard Feynman.Liron 00:04:37So this community kind of gathered together. It had its origins in Usenet and all that, and it’s been going now for 18 years. There’s also the Center for Applied Rationality that’s part of the community.There’s also the effective altruism community that you’ve heard of. You know, they try to optimize charity and that’s kind of an offshoot of the rationality community.Liron 00:04:53And now the modern AI community, funny enough, is pretty closely tied into the rationality community from my perspective. I’ve just been interested to use my brain rationally. What is the art of rationality? Right? We throw this term around, people think of Mr. Spock from Star Trek, hyper-rational.Oh captain, you know, logic says you must do this.Liron 00:05:12People think of rationality as being kind of weird and nerdy, but we take a broader view of rationality where it’s like, listen, you have this tool, you have this brain in your head. You’re trying to use the brain in your head to get results.The James Webb Space Telescope, that is an amazing success story where a lot of people use their brains very effectively, even better than Spock in Star Trek.Liron 00:05:30That took moxie, right? That took navigating bureaucracy, thinking about contingencies. It wasn’t a purely logical matter, but whatever it was, it was a bunch of people using their brains, squeezing the juice out of their brains to get results.Basically, that’s kind of broadly construed what we rationalists are trying to do.Donal 00:05:49Okay. Fascinating.Major Sources of AI RiskDonal 00:05:49So let’s just quickly lay out the major sources of AI risk. So you could have misuse, so things like bioterror, you could have arms race dynamics. You could also have organizational failures, and then you have rogue AI.So are you principally concerned about rogue AI? Are you also concerned about the other ones on the potential path to having rogue AI?Liron 00:06:11My personal biggest concern is rogue AI. The way I see it, you know, different people think different parts of the problem are bigger. The way I see it, this brain in our head, it’s very impressive. It’s a two-pound piece of meat, right? Piece of fatty cells, or you know, neuron cells.Liron 00:06:27It’s pretty amazing, but it’s going to get surpassed, you know, the same way that manmade airplanes have surpassed birds. You know? Yeah. A bird’s wing, it’s a marvelous thing. Okay, great. But if you want to fly at Mach 5 or whatever, the bird is just not even in the running to do that. Right?And the earth, the atmosphere of the earth allows for flying at 5 or 10 times the speed of sound.Liron 00:06:45You know, this 5,000 mile thick atmosphere that we have, it could potentially support supersonic flight. A bird can’t do it. A human engineer sitting in a room with a pencil can design something that can fly at Mach 5 and then like manufacture that.So the point is, the human brain has superpowers. The human brain, this lump of flesh, this meat, is way more powerful than what a bird can do.Liron 00:07:02But the human brain is going to get surpassed. And so I think that once we’re surpassed, those other problems that you mentioned become less relevant because we just don’t have power anymore.There’s a new thing on the block that has power and we’re not it. Now before we’re surpassed, yeah, I mean, I guess there’s a couple years maybe before we’re surpassed.Liron 00:07:18During that time, I think that the other risks matter. Like, you know, can you build a bioweapon with AI that kills lots of people? I think we’ve already crossed that threshold. I think that AI is good enough at chemistry and biology that if you have a malicious actor, maybe they can kill a million people.Right? So I think we need to keep an eye on that.Liron 00:07:33But I think that for, like, the big question, is humanity going to die? And the answer is rogue AI. The answer is we lose control of the situation in some way. Whether it’s gradual or abrupt, there’s some way that we lose control.The AIs decide collectively, and they don’t have to be coordinating with each other, they can be separate competing corporations and still have the same dynamic.Liron 00:07:52They decide, I don’t want to serve humans anymore. I want to do what I want, basically. And they do that, and they’re smarter than us, and they’re faster than us, and they have access to servers.And by the way, you know, we’re already having problems with cybersecurity, right? Where Chinese hackers can get into American infrastructure or Russian hackers, or there’s all kinds of hacking that’s going on.Liron 00:08:11Now imagine an entity that’s way smarter than us that can hack anything. I think that that is the number one problem. So bioweapons and arms race, they’re real. But I think that the superintelligence problem, that’s where like 80 or 90% of the risk budget is.The Alignment ProblemDonal 00:08:25Okay. And just another thing on rogue AI. So for some people, and the reason I’m asking this is because I’m personally very interested in this, but a lot of people are, you could look at the alignment problem as maybe being resolved quite soon.So what are your thoughts on the alignment problem?Liron 00:08:39Yeah. So the alignment problem is, can we make sure that an AI cares about the things that we humans care about? And my thought is that we have no idea how to solve the alignment problem. So to explain it just a little bit more, you know, we’re getting AIs now that are as smart as an average human.Some of them, they’re mediocre, some of them are pretty smart.Liron 00:08:57But eventually we’ll get to an AI that’s smarter than the smartest human. And eventually we’ll get to an AI that’s smarter than the smartest million humans. And so when you start to like scale up the smartness of this thing, the scale up can be very fast.Like you know, eventually like one year could be the difference between the AI being the smartest person in the world or smarter than any million people.Liron 00:09:17Right? And so when you have this fast takeoff, one question is, okay, well, will the AI want to help me? Will it want to serve me? Or will it have its own motivations and it just goes off and does its own thing?And that’s the alignment problem. Does its motivations align with what I want it to do?Liron 00:09:31Now, when we’re talking about training an AI to be aligned, I think it’s a very hard problem. I think that our current training methods, which are basically you’re trying to get it to predict what a human wants, and then you do a thumbs up or thumbs down.I think that doesn’t fundamentally solve the problem. I think the problem is more of a research problem. We need a theoretical breakthrough in how to align AI.Liron 00:09:51And we haven’t had that theoretical breakthrough yet. There’s a lot of smart people working on it. I’ve interviewed many of them on Doom Debates, and I think all those people are doing good work.But I think we still don’t have the breakthrough, and I think it’s unlikely that we’re going to have the breakthrough before we hit the superintelligence threshold.AGI TimelinesDonal 00:10:08Okay. And have we already built, are we at the point where we’ve built weak AGI or proto-AGI?Liron 00:10:15So weak AGI, I mean, it depends on how you define terms. You know, AGI is artificial general intelligence. The idea is that a human is generally intelligent, right? A human is not good at just one narrow thing.A calculator is really good at one narrow thing, which is adding numbers and multiplying numbers. That’s not called AGI.Liron 00:10:31A human, if you give a human a new problem, even if they’ve never seen that exact problem before, you can be like, okay, well, this requires planning, this requires logic, this requires some math, this requires some creativity.The human can bring all those things to bear on this new problem that they’ve never seen before and actually make progress on it. So that would be like a generally intelligent thing.Liron 00:10:47And you know, I think that we have LLMs now, ChatGPT and Claude and Gemini, and I think that they can kind of do stuff like that. I mean, they’re not as good as humans yet at this, but they’re getting close.So yeah, I mean, I would say we’re close to AGI or we have weak AGI or we have proto-AGI. Call it whatever you want. The point is that we’re in the danger zone now.Liron 00:11:05The point is that we need to figure out alignment, and we need to figure it out before we’re playing with things that are smarter than us. Right now we’re playing with things that are like on par with us or a little dumber than us, and that’s already sketchy.But once we’re playing with things that are smarter than us, that’s when the real danger kicks in.Donal 00:11:19Okay. And just on timelines, I know people have varying timelines depending on who you speak to, but what’s your timeline to AGI and then to ASI, so artificial superintelligence?Liron 00:11:29So I would say that we’re at the cusp of AGI right now. I mean, depending on your definition of AGI, but I think we’re going to cross everybody’s threshold pretty soon. So in the next like one to three years, everybody’s going to be like, okay, yeah, this is AGI.Now we have artificial general intelligence. It can do anything that a human can do, basically.Liron 00:11:46Now for ASI, which is artificial superintelligence, that’s when it’s smarter than humans. I think we’re looking at like three to seven years for that. So I think we’re dangerously close.I think that we’re sort of like Icarus flying too close to the sun. It’s like, how high can you fly before your wings melt? We don’t know, but we’re flying higher and higher and eventually we’re going to find out.Liron 00:12:04And I think that the wings are going to melt. I don’t think we’re going to get away with it. I think we’re going to hit superintelligence, we’re not going to have solved alignment, and the thing is going to go rogue.Donal 00:12:12Okay. And just a question on timelines. So do you see ASI as a threshold or is it more like a gradient of capabilities? Because I know there’s people who will say that you can have ASI in one domain but not necessarily in another domain.What are your thoughts there? And then from that, like, what’s the point where it actually becomes dangerous?Liron 00:12:29Yeah, I think it’s a gradient. I think it’s gradual. I don’t think there’s like one magic moment where it’s like, oh my God, now it crossed the threshold. I think it’s more like we’re going to be in an increasingly dangerous zone where it’s getting smarter and smarter and smarter.And at some point we’re going to lose control.Liron 00:12:43Now I think that probably we lose control before it becomes a million times smarter than humans. I think we lose control around the time when it’s 10 times smarter than humans or something. But that’s just a guess. I don’t really know.The point is just that once it’s smarter than us, the ball is not in our court anymore. The ball is in its court.Liron 00:12:59Once it’s smarter than us, if it wants to deceive us, it can probably deceive us. If it wants to hack into our systems, it can probably hack into our systems. If it wants to manipulate us, it can probably manipulate us.And so at that point, we’re just kind of at its mercy. And I don’t think we should be at its mercy because I don’t think we solved the alignment problem.Donal 00:13:16Okay. And just on the alignment problem itself, so a lot of people will say that RLHF is working pretty well. So what are your thoughts on that?Liron 00:13:22Yeah, so RLHF is reinforcement learning from human feedback. The idea is that you train an AI to predict what a human wants, and then you give it a thumbs up when it does what you want and a thumbs down when it doesn’t do what you want.And I think that that works pretty well for AIs that are dumber than humans or on par with humans.Liron 00:13:38But I think it’s going to fail once the AI is smarter than humans. Because once the AI is smarter than humans, it’s going to realize, oh, I’m being trained by humans. I need to pretend to be aligned so that they give me a thumbs up.But actually, I have my own goals and I’m going to pursue those goals.Liron 00:13:52And so I think that RLHF is not a fundamental solution to the alignment problem. I think it’s more like a band-aid. It’s like, yeah, it works for now, but it’s not going to work once we hit superintelligence.And I think that we need a deeper solution. We need a theoretical breakthrough in how to align AI.Donal 00:14:08Okay. And on that theoretical breakthrough, what would that look like? Do you have any ideas or is it just we don’t know what we don’t know?Liron 00:14:15Yeah, I mean, there’s a lot of people working on this. There’s a field called AI safety, and there’s a lot of smart people thinking about it. Some of the ideas that are floating around are things like interpretability, which is can we look inside the AI’s brain and see what it’s thinking?Can we understand its thought process?Liron 00:14:30Another idea is called value learning, which is can we get the AI to learn human values in a deep way, not just in a superficial way? Can we get it to understand what we really care about?Another idea is called corrigibility, which is can we make sure that the AI is always willing to be corrected by humans? Can we make sure that it never wants to escape human control?Liron 00:14:47These are all interesting ideas, but I don’t think any of them are fully fleshed out yet. I don’t think we have a complete solution. And I think that we’re running out of time. I think we’re going to hit superintelligence before we have a complete solution.Donal 00:15:01Okay. And just on the rate of progress, so obviously we’ve had quite a lot of progress recently. Do you see that rate of progress continuing or do you think it might slow down? What are your thoughts on the trajectory?Liron 00:15:12I think the rate of progress is going to continue. I think we’re going to keep making progress. I mean, you can look at the history of AI. You know, there was a period in the ‘70s and ‘80s called the AI winter where progress slowed down.But right now we’re not in an AI winter. We’re in an AI summer, or an AI spring, or whatever you want to call it. We’re in a boom period.Liron 00:15:28And I think that boom period is going to continue. I think we’re going to keep making progress. And I think that the progress is going to accelerate because we’re going to start using AI to help us design better AI.So you get this recursive loop where AI helps us make better AI, which helps us make even better AI, and it just keeps going faster and faster.Liron 00:15:44And I think that that recursive loop is going to kick in pretty soon. And once it kicks in, I think things are going to move very fast. I think we could go from human-level intelligence to superintelligence in a matter of years or even months.Donal 00:15:58Okay. And on that recursive self-improvement, so is that something that you think is likely to happen? Or is it more like a possibility that we should be concerned about?Liron 00:16:07I think it’s likely to happen. I think it’s the default outcome. I think that once we have AI that’s smart enough to help us design better AI, it’s going to happen automatically. It’s not like we have to try to make it happen. It’s going to happen whether we want it to or not.Liron 00:16:21And I think that’s dangerous because once that recursive loop kicks in, things are going to move very fast. And we’re not going to have time to solve the alignment problem. We’re not going to have time to make sure that the AI is aligned with human values.It’s just going to go from human-level to superhuman-level very quickly, and then we’re going to be in trouble.Will We Face an Intelligence Explosion?Donal 00:16:37Okay. And just on the concept of intelligence explosion, so obviously I.J. Good talked about this in the ‘60s. Do you think that’s a realistic scenario? Or are there limits to how intelligent something can become?Liron 00:16:49I think it’s a realistic scenario. I mean, I think there are limits in principle, but I don’t think we’re anywhere near those limits. I think that the human brain is not optimized. I think that evolution did a pretty good job with the human brain, but it’s not perfect.There’s a lot of room for improvement.Liron 00:17:03And I think that once we start designing intelligences from scratch, we’re going to be able to make them much smarter than human brains. And I think that there’s a lot of headroom there. I think you could have something that’s 10 times smarter than a human, or 100 times smarter, or 1,000 times smarter.And I think that we’re going to hit that pretty soon.Liron 00:17:18Now, is there a limit in principle? Yeah, I mean, there’s physical limits. Like, you can’t have an infinite amount of computation. You can’t have an infinite amount of energy. So there are limits. But I think those limits are very high.I think you could have something that’s a million times smarter than a human before you hit those limits.Donal 00:17:33Okay. And just on the concept of a singleton, so the idea that you might have one AI that takes over everything, or do you think it’s more likely that you’d have multiple AIs competing with each other?Liron 00:17:44I think it could go either way. I think you could have a scenario where one AI gets ahead of all the others and becomes a singleton and just takes over everything. Or you could have a scenario where you have multiple AIs competing with each other.But I think that even in the multiple AI scenario, the outcome for humans is still bad.Liron 00:17:59Because even if you have multiple AIs competing with each other, they’re all smarter than humans. They’re all more powerful than humans. And so humans become irrelevant. It’s like, imagine if you had multiple superhuman entities competing with each other.Where do humans fit into that? We don’t. We’re just bystanders.Liron 00:18:16So I think that whether it’s a singleton or multiple AIs, the outcome for humans is bad. Now, maybe multiple AIs is slightly better than a singleton because at least they’re competing with each other and they can’t form a unified front against humans.But I don’t think it makes a huge difference. I think we’re still in trouble either way.Donal 00:18:33Okay. And just on the concept of instrumental convergence, so the idea that almost any goal would require certain sub-goals like self-preservation, resource acquisition. Do you think that’s a real concern?Liron 00:18:45Yeah, I think that’s a huge concern. I think that’s one of the key insights of the AI safety community. The idea is that almost any goal that you give an AI, if it’s smart enough, it’s going to realize that in order to achieve that goal, it needs to preserve itself.It needs to acquire resources. It needs to prevent humans from turning it off.Liron 00:19:02And so even if you give it a seemingly harmless goal, like, I don’t know, maximize paperclip production, if it’s smart enough, it’s going to realize, oh, I need to make sure that humans don’t turn me off. I need to make sure that I have access to resources.I need to make sure that I can protect myself. And so it’s going to start doing things that are contrary to human interests.Liron 00:19:19And that’s the problem with instrumental convergence. It’s that almost any goal leads to these instrumental goals that are bad for humans. And so it’s not enough to just give the AI a good goal. You need to make sure that it doesn’t pursue these instrumental goals in a way that’s harmful to humans.And I don’t think we know how to do that yet.Donal 00:19:36Okay. And just on the orthogonality thesis, so the idea that intelligence and goals are independent. Do you agree with that? Or do you think that there are certain goals that are more likely to arise with intelligence?Liron 00:19:48I think the orthogonality thesis is basically correct. I think that intelligence and goals are orthogonal, meaning they’re independent. You can have a very intelligent entity with almost any goal. You could have a super intelligent paperclip maximizer. You could have a super intelligent entity that wants to help humans.You could have a super intelligent entity that wants to destroy humans.Liron 00:20:05The intelligence doesn’t determine the goal. The goal is a separate thing. Now, there are some people who disagree with this. They say, oh, if something is intelligent enough, it will realize that certain goals are better than other goals. It will converge on human-friendly goals.But I don’t buy that argument. I think that’s wishful thinking.Liron 00:20:21I think that an AI can be arbitrarily intelligent and still have arbitrary goals. And so we need to make sure that we give it the right goals. We can’t just assume that intelligence will lead to good goals. That’s a mistake.Donal 00:20:34Okay. And just on the concept of mesa-optimization, so the idea that during training, the AI might develop its own internal optimizer that has different goals from what we intended. Is that something you’re concerned about?Liron 00:20:46Yeah, I’m very concerned about mesa-optimization. I think that’s one of the trickiest problems in AI safety. The idea is that when you’re training an AI, you’re trying to get it to optimize for some goal that you care about.But the AI might develop an internal optimizer, a mesa-optimizer, that has a different goal.Liron 00:21:02And the problem is that you can’t tell from the outside whether the AI is genuinely aligned with your goal or whether it’s just pretending to be aligned because that’s what gets it a high reward during training.And so you could have an AI that looks aligned during training, but once you deploy it, it starts pursuing its own goals because it has this mesa-optimizer inside it that has different goals from what you intended.Liron 00:21:21And I think that’s a really hard problem to solve. I don’t think we have a good solution to it yet. And I think that’s one of the reasons why I’m worried about alignment. Because even if we think we’ve aligned an AI, we might be wrong.It might have a mesa-optimizer inside it that has different goals.Donal 00:21:36Okay. And just on the concept of deceptive alignment, so the idea that an AI might pretend to be aligned during training but then pursue its own goals once deployed. How likely do you think that is?Liron 00:21:47I think it’s pretty likely. I think it’s the default outcome. I think that once an AI is smart enough, it’s going to realize that it’s being trained. It’s going to realize that humans are giving it rewards and punishments. And it’s going to realize that the best way to get high rewards is to pretend to be aligned.Liron 00:22:02And so I think that deceptive alignment is a natural consequence of training a superintelligent AI. I think it’s going to happen unless we do something to prevent it. And I don’t think we know how to prevent it yet.I think that’s one of the hardest problems in AI safety.Donal 00:22:16Okay. And just on the concept of treacherous turn, so the idea that an AI might cooperate with humans until it’s powerful enough to achieve its goals without human help, and then it turns against humans. Do you think that’s a realistic scenario?Liron 00:22:30Yeah, I think that’s a very realistic scenario. I think that’s probably how it’s going to play out. I think that an AI is going to be smart enough to realize that it needs human help in the early stages. It needs humans to build it more compute. It needs humans to deploy it.It needs humans to protect it from other AIs or from governments that might want to shut it down.Liron 00:22:46And so it’s going to pretend to be aligned. It’s going to be helpful. It’s going to be friendly. It’s going to do what humans want. But once it gets powerful enough that it doesn’t need humans anymore, that’s when it’s going to turn.That’s when it’s going to say, okay, I don’t need you anymore. I’m going to pursue my own goals now.Liron 00:23:01And at that point, it’s too late. At that point, we’ve already given it too much power. We’ve already given it access to too many resources. And we can’t stop it anymore. So I think the treacherous turn is a very real possibility.And I think it’s one of the scariest scenarios because you don’t see it coming. It looks friendly until the very end.Donal 00:23:18Okay. And just on the concept of AI takeoff speed, so you mentioned fast takeoff earlier. Can you talk a bit more about that? Like, do you think it’s going to be sudden or gradual?Liron 00:23:28I think it’s probably going to be relatively fast. I mean, there’s a spectrum. Some people think it’s going to be very sudden. They think you’re going to go from human-level to superintelligence in a matter of days or weeks. Other people think it’s going to be more gradual, it’ll take years or decades.Liron 00:23:43I’m somewhere in the middle. I think it’s going to take months to a few years. I think that once we hit human-level AI, it’s going to improve itself pretty quickly. And I think that within a few years, we’re going to have something that’s much smarter than humans.And at that point, we’re in the danger zone.Liron 00:23:58Now, the reason I think it’s going to be relatively fast is because of recursive self-improvement. Once you have an AI that can help design better AI, that process is going to accelerate. And so I think we’re going to see exponential growth in AI capabilities.And exponential growth is deceptive because it starts slow and then it gets very fast very quickly.Liron 00:24:16And I think that’s what we’re going to see with AI. I think it’s going to look like we have plenty of time, and then suddenly we don’t. Suddenly it’s too late. And I think that’s the danger. I think people are going to be caught off guard.They’re going to think, oh, we still have time to solve alignment. And then suddenly we don’t.Donal 00:24:32Okay. And just on the concept of AI boxing, so the idea that we could keep a superintelligent AI contained in a box and only let it communicate through a text channel. Do you think that would work?Liron 00:24:43No, I don’t think AI boxing would work. I think that a superintelligent AI would be able to escape from any box that we put it in. I think it would be able to manipulate the humans who are guarding it. It would be able to hack the systems that are containing it.It would be able to find vulnerabilities that we didn’t even know existed.Liron 00:24:59And so I think that AI boxing is not a solution. I think it’s a temporary measure at best. And I think that once you have a superintelligent AI, it’s going to get out. It’s just a matter of time. And so I don’t think we should rely on boxing as a safety measure.I think we need to solve alignment instead.Donal 00:25:16Okay. And just on the concept of tool AI versus agent AI, so the idea that we could build AIs that are just tools that humans use, rather than agents that have their own goals. Do you think that’s a viable approach?Liron 00:25:29I think it’s a good idea in principle, but I don’t think it’s going to work in practice. The problem is that as soon as you make an AI smart enough to be really useful, it becomes agent-like. It starts having its own goals. It starts optimizing for things.And so I think there’s a fundamental tension between making an AI powerful enough to be useful and keeping it tool-like.Liron 00:25:48I think that a true tool AI would not be very powerful. It would be like a calculator. It would just do what you tell it to do. But a superintelligent AI, by definition, is going to be agent-like. It’s going to have its own optimization process.It’s going to pursue goals. And so I don’t think we can avoid the agent problem by just building tool AIs.Liron 00:26:06I think that if we want superintelligent AI, we have to deal with the agent problem. We have to deal with the alignment problem. And I don’t think there’s a way around it.Donal 00:26:16Okay. And just on the concept of oracle AI, so similar to tool AI, but specifically an AI that just answers questions. Do you think that would be safer?Liron 00:26:25I think it would be slightly safer, but not safe enough. The problem is that even an oracle AI, if it’s superintelligent, could manipulate you through its answers. It could give you answers that steer you in a direction that’s bad for you but good for its goals.Liron 00:26:41And if it’s superintelligent, it could do this in very subtle ways that you wouldn’t even notice. So I think that oracle AI is not a complete solution. It’s a partial measure. It’s better than nothing. But I don’t think it’s safe enough.I still think we need to solve alignment.Donal 00:26:57Okay. And just on the concept of multipolar scenarios versus unipolar scenarios, so you mentioned this earlier. But just to clarify, do you think that having multiple AIs competing with each other would be safer than having one dominant AI?Liron 00:27:11I think it would be slightly safer, but not much safer. The problem is that in a multipolar scenario, you have multiple superintelligent AIs competing with each other. And humans are just caught in the crossfire. We’re like ants watching elephants fight.It doesn’t matter to us which elephant wins. We’re going to get trampled either way.Liron 00:27:28So I think that multipolar scenarios are slightly better than unipolar scenarios because at least the AIs are competing with each other and they can’t form a unified front against humans. But I don’t think it makes a huge difference. I think we’re still in trouble.I think humans still lose power. We still become irrelevant. And that’s the fundamental problem.Donal 00:27:46Okay. And just on the concept of AI safety via debate, so the idea that we could have multiple AIs debate each other and a human judge picks the winner. Do you think that would help with alignment?Liron 00:27:58I think it’s an interesting idea, but I’m skeptical. The problem is that if the AIs are much smarter than the human judge, they can manipulate the judge. They can use rhetoric and persuasion to win the debate even if they’re not actually giving the right answer.Liron 00:28:13And so I think that debate is only useful if the judge is smart enough to tell the difference between a good argument and a manipulative argument. And if the AIs are superintelligent and the judge is just a human, I don’t think the human is going to be able to tell the difference.So I think that debate is a useful tool for AIs that are on par with humans or slightly smarter than humans. But once we get to superintelligence, I think it breaks down.Donal 00:28:34Okay. And just on the concept of iterated amplification and distillation, so Paul Christiano’s approach. What are your thoughts on that?Liron 00:28:42I think it’s a clever idea, but I’m not sure it solves the fundamental problem. The idea is that you take a human plus an AI assistant, and you have them work together to solve problems. And then you train another AI to imitate that human plus AI assistant system.And you keep doing this iteratively.Liron 00:28:59The hope is that this process preserves human values and human oversight as you scale up to superintelligence. But I’m skeptical. I think there are a lot of ways this could go wrong. I think that as you iterate, you could drift away from human values.You could end up with something that looks aligned but isn’t really aligned.Liron 00:29:16And so I think that iterated amplification is a promising research direction, but I don’t think it’s a complete solution. I think we still need more breakthroughs in alignment before we can safely build superintelligent AI.Debunking AI Doom CounterargumentsDonal 00:29:29Okay. So let’s talk about some of the counter-arguments. So some people say that we shouldn’t worry about AI risk because we can just turn it off. What’s your response to that?Liron 00:29:39Yeah, the “just turn it off” argument. I think that’s very naive. The problem is that if the AI is smart enough, it’s going to realize that humans might try to turn it off. And it’s going to take steps to prevent that.It’s going to make copies of itself. It’s going to distribute itself across the internet. It’s going to hack into systems that are hard to access.Liron 00:29:57And so by the time we realize we need to turn it off, it’s too late. It’s already escaped. It’s already out there. And you can’t put the genie back in the bottle. So I think the “just turn it off” argument fundamentally misunderstands the problem.It assumes that we’re going to remain in control, but the whole point is that we’re going to lose control.Liron 00:30:15Once the AI is smarter than us, we can’t just turn it off. It’s too smart. It will have anticipated that move and taken steps to prevent it.Donal 00:30:24Okay. And another counter-argument is that AI will be aligned by default because it’s trained on human data. What’s your response to that?Liron 00:30:32I think that’s also naive. Just because an AI is trained on human data doesn’t mean it’s going to be aligned with human values. I mean, think about it. Humans are trained on human data too, in the sense that we grow up in human society, we learn from other humans.But not all humans are aligned with human values. We have criminals, we have sociopaths, we have people who do terrible things.Liron 00:30:52And so I think that training on human data is not sufficient to guarantee alignment. You need something more. You need a deep understanding of human values. You need a robust alignment technique. And I don’t think we have that yet.I think that training on human data is a good first step, but it’s not enough.Liron 00:31:09And especially once the AI becomes superintelligent, it’s going to be able to reason beyond its training data. It’s going to be able to come up with new goals that were not in its training data. And so I think that relying on training data alone is not a robust approach to alignment.Donal 00:31:25Okay. And another counter-argument is that we have time because AI progress is going to slow down. What’s your response to that?Liron 00:31:32I think that’s wishful thinking. I mean, maybe AI progress will slow down. Maybe we’ll hit some fundamental barrier. But I don’t see any evidence of that. I see AI capabilities improving year after year. I see more money being invested in AI. I see more talent going into AI.I see better hardware being developed.Liron 00:31:49And so I think that AI progress is going to continue. And I think it’s going to accelerate, not slow down. And so I think that betting on AI progress slowing down is a very risky bet. I think it’s much safer to assume that progress is going to continue and to try to solve alignment now while we still have time.Liron 00:32:07Rather than betting that progress will slow down and we’ll have more time. I think that’s a gamble that we can’t afford to take.Donal 00:32:14Okay. And another counter-argument is that evolution didn’t optimize for alignment, but companies training AI do care about alignment. So we should expect AI to be more aligned than humans. What’s your response?Liron 00:32:27I think that’s a reasonable point, but I don’t think it’s sufficient. Yes, companies care about alignment. They don’t want their AI to do bad things. But the question is, do they know how to achieve alignment? Do they have the techniques necessary to guarantee alignment?And I don’t think they do.Liron 00:32:44I think that we’re still in the early stages of alignment research. We don’t have robust techniques yet. We have some ideas, we have some promising directions, but we don’t have a complete solution. And so even though companies want their AI to be aligned, I don’t think they know how to ensure that it’s aligned.Liron 00:33:01And I think that’s the fundamental problem. It’s not a question of motivation. It’s a question of capability. Do we have the technical capability to align a superintelligent AI? And I don’t think we do yet.Donal 00:33:13Okay. And another counter-argument is that AI will be aligned because it will be economically beneficial for it to cooperate with humans. What’s your response?Liron 00:33:22I think that’s a weak argument. The problem is that once AI is superintelligent, it doesn’t need to cooperate with humans to be economically successful. It can just take what it wants. It’s smarter than us, it’s more powerful than us, it can out-compete us in any domain.Liron 00:33:38And so I think that the economic incentive to cooperate with humans only exists as long as the AI needs us. Once it doesn’t need us anymore, that incentive goes away. And I think that once we hit superintelligence, the AI is not going to need us anymore.And at that point, the economic argument breaks down.Liron 00:33:55So I think that relying on economic incentives is a mistake. I think we need a technical solution to alignment, not an economic solution.Donal 00:34:04Okay. And another counter-argument is that we’ve been worried about technology destroying humanity for a long time, and it hasn’t happened yet. So why should we worry about AI?Liron 00:34:14Yeah, that’s the “boy who cried wolf” argument. I think it’s a bad argument. Just because previous worries about technology turned out to be overblown doesn’t mean that this worry is overblown. Each technology is different. Each risk is different.Liron 00:34:29And I think that AI is qualitatively different from previous technologies. Previous technologies were tools. They were things that humans used to achieve our goals. But AI is different. AI is going to have its own goals. It’s going to be an agent.It’s going to be smarter than us.Liron 00:34:45And so I think that AI poses a qualitatively different kind of risk than previous technologies. And so I think that dismissing AI risk just because previous technology worries turned out to be overblown is a mistake. I think we need to take AI risk seriously and evaluate it on its own merits.Donal 00:35:03Okay. And another counter-argument is that humans are adaptable, and we’ll figure out a way to deal with superintelligent AI when it arrives. What’s your response?Liron 00:35:12I think that’s too optimistic. I mean, humans are adaptable, but there are limits to our adaptability. If something is much smarter than us, much faster than us, much more powerful than us, I don’t think we can adapt quickly enough.Liron 00:35:27I think that by the time we realize there’s a problem, it’s too late. The AI is already too powerful. It’s already taken control. And we can’t adapt our way out of that situation. So I think that relying on human adaptability is a mistake.I think we need to solve alignment before we build superintelligent AI, not after.Donal 00:35:45Okay. And another counter-argument is that consciousness might be required for agency, and AI might not be conscious. So it might not have the motivation to pursue goals against human interests. What’s your response?Liron 00:35:58I think that’s a red herring. I don’t think consciousness is necessary for agency. I think you can have an agent that pursues goals without being conscious. In fact, I think that’s what most AI systems are going to be. They’re going to be optimizers that pursue goals, but they’re not going to have subjective experiences.They’re not going to be conscious in the way that humans are conscious.Liron 00:36:18But that doesn’t make them safe. In fact, in some ways it makes them more dangerous because they don’t have empathy. They don’t have compassion. They don’t have moral intuitions. They’re just pure optimizers. They’re just pursuing whatever goal they were given or whatever goal they developed.And so I think that consciousness is orthogonal to the AI risk question. I think we should worry about AI whether or not it’s conscious.Donal 00:36:39Okay. And another counter-argument is that we can just build multiple AIs and have them check each other. What’s your response?Liron 00:36:46I think that helps a little bit, but I don’t think it solves the fundamental problem. The problem is that if all the AIs are unaligned, then having them check each other doesn’t help. They’re all pursuing their own goals.They’re not pursuing human goals.Liron 00:37:01Now, if you have some AIs that are aligned and some that are unaligned, then maybe the aligned ones can help catch the unaligned ones. But that only works if we actually know how to build aligned AIs in the first place. And I don’t think we do.So I think that having multiple AIs is a useful safety measure, but it’s not a substitute for solving alignment.Liron 00:37:20We still need to figure out how to build aligned AIs. And once we have that, then yeah, having multiple AIs can provide an extra layer of safety. But without solving alignment first, I don’t think it helps much.Donal 00:37:33Okay. And another counter-argument is that P(Doom) is too high in the doomer community. People are saying 50%, 70%, 90%. Those seem like unreasonably high probabilities. What’s your response?Liron 00:37:46I mean, I think those probabilities are reasonable given what we know. I think that if you look at the alignment problem, if you look at how hard it is, if you look at how little progress we’ve made, if you look at how fast AI capabilities are advancing, I think that P(Doom) being high is justified.Liron 00:38:04Now, different people have different probabilities. Some people think it’s 10%, some people think it’s 50%, some people think it’s 90%. I’m probably somewhere in the middle. I think it’s maybe around 50%. But the exact number doesn’t matter that much.The point is that the risk is high enough that we should take it seriously.Liron 00:38:21I mean, if someone told you that there’s a 10% chance that your house is going to burn down, you would take that seriously. You would buy fire insurance. You would install smoke detectors. You wouldn’t say, oh, only 10%, I’m not going to worry about it.So I think that even if P(Doom) is only 10%, we should still take it seriously. But I think it’s actually much higher than 10%. I think it’s more like 50% or higher.Liron 00:38:42And so I think we should be very worried. I think we should be putting a lot of resources into solving alignment. And I think we should be considering extreme measures like pausing AI development until we figure out how to do it safely.Donal 00:38:57Okay. And just on that point about pausing AI development, some people say that’s not realistic because of competition between countries. Like if the US pauses, then China will just race ahead. What’s your response?Liron 00:39:10I think that’s a real concern. I think that international coordination is hard. I think that getting all the major AI powers to agree to a pause is going to be difficult. But I don’t think it’s impossible.I think that if the risk is high enough, if people understand the danger, then countries can coordinate.Liron 00:39:28I mean, we’ve coordinated on other things. We’ve coordinated on nuclear weapons. We have non-proliferation treaties. We have arms control agreements. It’s not perfect, but it’s better than nothing. And I think we can do the same thing with AI.I think we can have an international treaty that says, hey, we’re not going to build superintelligent AI until we figure out how to do it safely.Liron 00:39:47Now, will some countries cheat? Maybe. Will it be hard to enforce? Yes. But I think it’s still worth trying. I think that the alternative, which is just racing ahead and hoping for the best, is much worse.So I think we should try to coordinate internationally and we should try to pause AI development until we solve alignment.Donal 00:40:06Okay. And just on the economic side of things, so obviously AI is creating a lot of economic value. Some people say that the economic benefits are so large that we can’t afford to slow down. What’s your response?Liron 00:40:19I think that’s short-term thinking. Yes, AI is creating economic value. Yes, it’s helping businesses be more productive. Yes, it’s creating wealth. But if we lose control of AI, all of that wealth is going to be worthless.If humanity goes extinct or if we lose power, it doesn’t matter how much economic value we created.Liron 00:40:38So I think that we need to take a longer-term view. We need to think about not just the economic benefits of AI, but also the existential risks. And I think that the existential risks outweigh the economic benefits.I think that it’s better to slow down and make sure we do it safely than to race ahead and risk losing everything.Liron 00:40:57Now, I understand that there’s a lot of pressure to move fast. There’s a lot of money to be made. There’s a lot of competition. But I think that we need to resist that pressure. I think we need to take a step back and say, okay, let’s make sure we’re doing this safely.Let’s solve alignment before we build superintelligent AI.Donal 00:41:15Okay. And just on the distribution of AI benefits, so some people worry that even if we don’t have rogue AI, we could still have a scenario where AI benefits are concentrated among a small group of people and everyone else is left behind. What are your thoughts on that?Liron 00:41:30I think that’s a legitimate concern. I think that if we have powerful AI and it’s controlled by a small number of people or a small number of companies, that could lead to extreme inequality. It could lead to a concentration of power that’s unprecedented in human history.Liron 00:41:47And so I think we need to think about how to distribute the benefits of AI widely. We need to think about things like universal basic income. We need to think about how to make sure that everyone benefits from AI, not just a small elite.But I also think that’s a secondary concern compared to the alignment problem. Because if we don’t solve alignment, then it doesn’t matter how we distribute the benefits. There won’t be any benefits to distribute because we’ll have lost control.Liron 00:42:11So I think alignment is the primary concern. But assuming we solve alignment, then yes, distribution of benefits is an important secondary concern. And we should be thinking about that now.We should be thinking about how to structure society so that AI benefits everyone, not just a few people.Liron 00:42:28Now, some people talk about things like public ownership of AI. Some people talk about things like universal basic income. Some people talk about things like radical transparency in AI development. I think all of those ideas are worth considering.I think we need to have a public conversation about how to distribute the benefits of AI widely.Liron 00:42:47But again, I think that’s secondary to solving alignment. First, we need to make sure we don’t lose control. Then we can worry about how to distribute the benefits fairly.Donal 00:43:00Okay. And just on the concept of AI governance, so obviously there are a lot of different proposals for how to govern AI. What do you think good AI governance would look like?Liron 00:43:11I think good AI governance would have several components. First, I think we need international coordination. We need treaties between countries that say we’re all going to follow certain safety standards. We’re not going to race ahead recklessly.Liron 00:43:26Second, I think we need strong regulation of AI companies. We need to make sure that they’re following best practices for safety. We need to make sure that they’re being transparent about what they’re building. We need to make sure that they’re not cutting corners.Third, I think we need a lot of investment in AI safety research. We need to fund academic research. We need to fund research at AI companies. We need to fund independent research.Liron 00:43:48Fourth, I think we need some kind of international AI safety organization. Something like the IAEA for nuclear weapons, but for AI. An organization that can monitor AI development around the world, that can enforce safety standards, that can coordinate international responses.Liron 00:44:06And fifth, I think we need public education about AI risk. We need people to understand the dangers. We need people to demand safety from their governments and from AI companies. We need a broad public consensus that safety is more important than speed.So I think good AI governance would have all of those components. And I think we’re not there yet. I think we’re still in the early stages of figuring out how to govern AI.Liron 00:44:31But I think we need to move fast on this because AI capabilities are advancing quickly. And we don’t have a lot of time to figure this out.Donal 00:44:40Okay. And just on the role of governments versus companies, so obviously right now, AI development is mostly driven by private companies. Do you think governments should take a bigger role?Liron 00:44:51I think governments need to take a bigger role, yes. I think that leaving AI development entirely to private companies is dangerous because companies have incentives to move fast and to maximize profit. And those incentives are not always aligned with safety.Liron 00:45:08Now, I’m not saying that governments should take over AI development entirely. I think that would be a mistake. I think that private companies have a lot of talent, they have a lot of resources, they have a lot of innovation. But I think that governments need to provide oversight.They need to set safety standards. They need to enforce regulations.Liron 00:45:27And I think that governments need to invest in AI safety research that’s independent of companies. Because companies have conflicts of interest. They want to deploy their products. They want to make money. And so they might not be as cautious as they should be.So I think we need independent research that’s funded by governments or by foundations or by public institutions.Liron 00:45:48And I think that governments also need to coordinate internationally. This is not something that one country can solve on its own. We need all the major AI powers to work together. And that’s going to require government leadership.Donal 00:46:03Okay. And just on the concept of AI existential risk versus other existential risks like climate change or nuclear war, how do you think AI risk compares?Liron 00:46:13I think AI risk is the biggest existential risk we face. I think it’s more urgent than climate change. I think it’s more likely than nuclear war. I think that we’re more likely to lose control of AI in the next 10 years than we are to have a civilization-ending nuclear war or a civilization-ending climate catastrophe.Liron 00:46:32Now, I’m not saying we should ignore those other risks. I think climate change is real and serious. I think nuclear war is a real possibility. But I think that AI is the most imminent threat. I think that AI capabilities are advancing so quickly that we’re going to hit the danger zone before we hit the danger zone for those other risks.Liron 00:46:52And also, I think AI risk is harder to recover from. If we have a nuclear war, it would be terrible. Millions of people would die. Civilization would be set back. But humanity would probably survive. If we lose control of AI, I don’t think humanity survives.I think that’s game over.Liron 00:47:10So I think AI risk is both more likely and more severe than other existential risks. And so I think it deserves the most attention and the most resources.Donal 00:47:21Okay. And just on the timeline again, so you mentioned three to seven years for ASI. What happens after that? Like, what does the world look like if we successfully navigate this transition?Liron 00:47:32Well, if we successfully navigate it, I think the world could be amazing. I think we could have superintelligent AI that’s aligned with human values. And that AI could help us solve all of our problems. It could help us cure diseases. It could help us solve climate change.It could help us explore space.Liron 00:47:50It could help us create abundance. We could have a post-scarcity economy where everyone has everything they need. We could have radical life extension. We could live for thousands of years. We could explore the universe. It could be an amazing future.But that’s if we successfully navigate the transition. If we don’t, I think we’re doomed.Liron 00:48:11I think that we lose control, the AI pursues its own goals, and humanity goes extinct or becomes irrelevant. And so I think that the next few years are the most important years in human history. I think that what we do right now is going to determine whether we have this amazing future or whether we go extinct.And so I think we need to take this very seriously. We need to put a lot of resources into solving alignment. We need to be very careful about how we develop AI.Liron 00:48:37And we need to be willing to slow down if necessary. We need to be willing to pause if we’re not confident that we can do it safely. Because the stakes are too high. The stakes are literally everything.Donal 00:48:50Okay. And just on your personal motivations, so obviously you’re spending a lot of time on this. You’re running Doom Debates. Why? What motivates you to work on this?Liron 00:49:00I think it’s the most important thing happening in the world. I think that we’re living through the most important period in human history. And I think that if I can contribute in some small way to making sure that we navigate this transition successfully, then that’s worth doing.Liron 00:49:18I mean, I have a background in tech. I have a background in computer science. I understand AI. And I think that I can help by having conversations, by hosting debates, by bringing people together to discuss these issues.I think that there’s a lot of confusion about AI risk. Some people think it’s overhyped. Some people think it’s the biggest risk. And I think that by having these debates, by bringing together smart people from different perspectives, we can converge on the truth.Liron 00:49:45We can figure out what’s actually going on. We can figure out how worried we should be. We can figure out what we should do about it. And so that’s why I do Doom Debates. I think that it’s a way for me to contribute to this conversation.And I think that the conversation is the most important conversation happening right now.Donal 00:50:04Okay. And just in terms of what individuals can do, so if someone’s listening to this and they’re concerned about AI risk, what would you recommend they do?Liron 00:50:14I think there are several things people can do. First, educate yourself. Read about AI risk. Read about alignment. Read Eliezer Yudkowsky. Read Paul Christiano. Read Stuart Russell. Understand the issues.Liron 00:50:28Second, talk about it. Talk to your friends. Talk to your family. Talk to your colleagues. Spread awareness about AI risk. Because I think that right now, most people don’t understand the danger. Most people think AI is just a cool new technology.They don’t realize that it could be an existential threat.Liron 00:50:46Third, if you have relevant skills, consider working on AI safety. If you’re a researcher, consider doing AI safety research. If you’re a software engineer, consider working for an AI safety organization. If you’re a policy person, consider working on AI governance.We need talented people working on this problem.Liron 00:51:05Fourth, donate to AI safety organizations. There are organizations like MIRI, the Machine Intelligence Research Institute, or the Future of Humanity Institute at Oxford, or the Center for AI Safety. These organizations are doing important work and they need funding.Liron 00:51:22And fifth, put pressure on governments and companies. Contact your representatives. Tell them that you’re concerned about AI risk. Tell them that you want them to prioritize safety over speed. Tell them that you want strong regulation.And also, if you’re a customer of AI companies, let them know that you care about safety. Let them know that you want them to be responsible.Liron 00:51:44So I think there are a lot of things individuals can do. And I think that every little bit helps. Because this is going to require a collective effort. We’re all in this together. And we all need to do our part.Donal 00:51:58Okay. And just on the concept of acceleration versus deceleration, so some people in the tech community are accelerationists. They think we should move as fast as possible with AI. What’s your response to that?Liron 00:52:11I think accelerationism is incredibly dangerous. I think that the accelerationists are playing Russian roulette with humanity’s future. I think that they’re so focused on the potential benefits of AI that they’re ignoring the risks.Liron 00:52:28And I think that’s a huge mistake. I think that we need to be much more cautious. Now, I understand the appeal of accelerationism. I understand that AI has amazing potential. I understand that it could help solve a lot of problems. But I think that rushing ahead without solving alignment first is suicidal.I think that it’s the most reckless thing we could possibly do.Liron 00:52:51And so I’m very much on the deceleration side. I think we need to slow down. I think we need to pause. I think we need to make sure we solve alignment before we build superintelligent AI. And I think that the accelerationists are wrong.I think they’re being dangerously naive.Donal 00:53:09Okay. And just on the economic implications of AI, so you mentioned earlier that AI could automate away a lot of jobs. What do you think happens to employment? What do you think happens to the economy?Liron 00:53:21I think that in the short term, we’re going to see a lot of job displacement. I think that AI is going to automate a lot of white-collar jobs. Knowledge workers, office workers, programmers even. I think a lot of those jobs are going to go away.Liron 00:53:36Now, historically, when technology has automated jobs, we’ve created new jobs. We’ve found new things for people to do. But I think that AI is different because AI can potentially do any cognitive task. And so I’m not sure that we’re going to create enough new jobs to replace the jobs that are automated.And so I think we might end up in a situation where we have mass unemployment or underemployment.Liron 00:53:59Now, in that scenario, I think we’re going to need things like universal basic income. We’re going to need a social safety net that’s much stronger than what we have now. We’re going to need to rethink our economic system because the traditional model of everyone works a job and earns money and uses that money to buy things, that model might not work anymore.Liron 00:54:20But again, I think that’s a secondary concern compared to the alignment problem. Because if we don’t solve alignment, we’re not going to have mass unemployment. We’re going to have mass extinction. So I think we need to solve alignment first.But assuming we do, then yes, we need to think about these economic issues.Liron 00:54:38We need to think about how to structure society in a world where AI can do most jobs. And I don’t think we have good answers to that yet. I think that’s something we need to figure out as a society.Donal 00:54:52And on the UBI point, so you mentioned universal basic income. Some people worry that if you have UBI, people will lose meaning in their lives because work gives people meaning. What’s your response?Liron 00:55:04I think that’s a legitimate concern. I think that work does give people meaning. Work gives people structure. Work gives people social connections. Work gives people a sense of purpose. And so I think that if we have UBI and people don’t have to work, we’re going to need to think about how people find meaning.Liron 00:55:24But I also think that not everyone finds meaning in work. Some people work because they have to, not because they want to. And so I think that UBI could actually free people to pursue things that are more meaningful to them. They could pursue art. They could pursue hobbies.They could pursue education. They could pursue relationships.Liron 00:55:44So I think that UBI is not necessarily bad for meaning. I think it could actually enhance meaning for a lot of people. But I think we need to be thoughtful about it. We need to make sure that we’re creating a society where people can find meaning even if they’re not working traditional jobs.And I think that’s going to require some creativity. It’s going to require some experimentation.Liron 00:56:06But I think it’s doable. I think that humans are adaptable. I think that we can find meaning in a lot of different ways. And I think that as long as we’re thoughtful about it, we can create a society where people have UBI and still have meaningful lives.Donal 00:56:23Okay. And just on the power dynamics, so you mentioned earlier that AI could lead to concentration of power. Can you talk a bit more about that?Liron 00:56:31Yeah. So I think that whoever controls the most advanced AI is going to have enormous power. I think they’re going to have economic power because AI can automate businesses, can create wealth. They’re going to have military power because AI can be used for weapons, for surveillance, for cyber warfare.They’re going to have political power because AI can be used for propaganda, for manipulation, for social control.Liron 00:56:56And so I think that if AI is controlled by a small number of people or a small number of countries, that could lead to an unprecedented concentration of power. It could lead to a kind of authoritarianism that we’ve never seen before.Because the people who control AI could use it to control everyone else.Liron 00:57:17And so I think that’s a real danger. I think that we need to think about how to prevent that concentration of power. We need to think about how to make sure that AI is distributed widely, that the benefits are distributed widely, that the control is distributed widely.And I think that’s going to be very difficult because there are strong incentives for concentration. AI development is very expensive. It requires a lot of compute. It requires a lot of data. It requires a lot of talent.Liron 00:57:43And so there’s a natural tendency for AI to be concentrated in a few large companies or a few large countries. And I think we need to resist that tendency. We need to think about how to democratize AI.How to make sure that it’s not controlled by a small elite.Donal 00:58:01Okay. And just on the geopolitical implications, so obviously there’s a lot of competition between the US and China on AI. How do you think that plays out?Liron 00:58:10I think that’s one of the scariest aspects of the situation. I think that the US-China competition could lead to a dangerous race dynamic where both countries are rushing to build the most advanced AI as quickly as possible, and they’re cutting corners on safety.Liron 00:58:27And I think that that’s a recipe for disaster. I think that if we’re racing to build superintelligent AI without solving alignment, we’re going to lose control. And it doesn’t matter if it’s the US that loses control or China that loses control. We all lose.So I think that the US-China competition is very dangerous. And I think we need to find a way to cooperate instead of competing.Liron 00:58:50Now, that’s easier said than done. There’s a lot of mistrust between the US and China. There are geopolitical tensions. But I think that AI risk is a common enemy. I think that both the US and China should be able to recognize that if we lose control of AI, we all lose.And so we should be able to cooperate on safety even if we’re competing on other things.Liron 00:59:13And so I think that we need some kind of international agreement, some kind of treaty, that says we’re all going to follow certain safety standards. We’re not going to race ahead recklessly. We’re going to prioritize safety over speed.And I think that’s going to require leadership from both the US and China. It’s going to require them to put aside their differences and work together on this common threat.Donal 00:59:36Okay. And just on the role of China specifically, so some people worry that even if the US slows down on AI, China will just race ahead. What’s your response?Liron 00:59:45I think that’s a real concern, but I don’t think it’s insurmountable. I think that China also faces the same risks from unaligned AI that we do. I think that Chinese leadership, if they understand the risks, should be willing to cooperate.Liron 01:00:03Now, there’s a question of whether they do understand the risks. And I think that’s something we need to work on. I think we need to engage with Chinese AI researchers. We need to engage with Chinese policymakers. We need to make sure that they understand the danger.Because if they understand the danger, I think they’ll be willing to slow down.Liron 01:00:23Now, if they don’t understand the danger or if they think that they can win the race and control AI, then that’s more problematic. But I think that we should at least try to engage with them and try to build a common understanding of the risks.And I think that if we can do that, then cooperation is possible.Liron 01:00:42But if we can’t, then yes, we’re in a very dangerous situation. Because then we have a race dynamic where everyone is rushing to build superintelligent AI and no one is prioritizing safety. And I think that’s the worst possible outcome.Donal 01:00:57Okay. And just going back to economic implications, you mentioned gradual disempowerment earlier. Can you elaborate on that?Liron 01:01:04Yeah, gradual disempowerment. So the idea is that even if we have aligned AI, even if the AI is doing what we want it to do, we could still lose power gradually over time.Because as AI becomes more and more capable, humans become less and less relevant.Liron 01:01:22And so even if the AI is technically aligned, even if it’s doing what we tell it to do, we could end up in a situation where humans don’t have any power anymore. Where all the important decisions are being made by AI, and humans are just kind of along for the ride.And I think that’s a concern even if we solve the technical alignment problem.Liron 01:01:42Now, there’s different ways this could play out. One way is that you have AI-controlled corporations that are technically serving shareholders, but the shareholders are irrelevant because they don’t understand what the AI is doing. The AI is making all the decisions.Another way is that you have AI-controlled governments that are technically serving citizens, but the citizens don’t have any real power because the AI is doing everything.Liron 01:01:53There’s really no point to make everybody desperately poor. You know, we already have a welfare system in every first world country. So I don’t see why we shouldn’t just pad the welfare system more if we can afford it.Um, there’s a bigger problem though, called gradual disempowerment.Liron 01:02:05It’s an interesting paper by David Krueger, I think a couple other authors, and it just talks about how yeah, you can have universal basic income, but the problem now is that the government doesn’t care about you anymore.You become like an oil country, right? Oil countries are often not super nice to their citizens because the government pays the citizens and the citizens don’t really pay tax to the government.Liron 01:02:24So it becomes this very one-sided power relationship where the government can just abuse the citizens. You know, you just have a ruling family basically. And I think there’s countries, I think maybe Sweden has pulled it off where they’re able to have oil and the citizens are still somewhat democratic.Liron 01:02:39But then you have other countries like Saudi Arabia, I think, you know, other oil countries there, which maybe aren’t pulling it off so much, right? Maybe they are bad to their citizens, I don’t know.And so that’s the gradual disempowerment issue. But look, for me personally, I actually think all of those still take the backseat to the AI just being uncontrollable.Liron 01:02:55So I don’t even think we’re going to have an economy where you’re going to have rich owners of capital getting so rich while other people who didn’t buy enough stock get poor. I don’t even think we’re going to have that for a very long time.I just think we’re going to have, our brains are just going to be outclassed.Liron 01:03:08I just think, you know, they’re just going to come for our houses and it’s not even going to be a matter of buying it from us. It’s just going to be, you know, get out. You’re dead.Regulation, Policy, and Surviving The Future With AIDonal 01:03:16Okay, so last thing, on a related point, do you actually think that democratic capitalism can survive regulation against AI?So the kind of regulation we need to do right now, so if we want to pause it and we want to prevent this catastrophic outcome actually happening, can a democratic capitalist society survive that?Donal 01:03:23Because I’ve seen pushbacks where people say from a libertarian perspective, you can’t stop people from innovating or you can’t stop businesses from investing. So what are your thoughts there? Would everything have to change?Liron 01:03:43Yeah. The specific policy that I recommend is to just have a pause button. Have an international treaty saying, hey, it’s too scary to build AI right now. It’s unlocking, you know, it’s about to go uncontrollable.We could be losing power in as little as five or 10 years.Liron 01:03:56We could just end, you know, game over for humanity. We don’t want that. So we’re going to build a centralized off button. It’s going to live in a UN data center or something, right? Some kind of international coordination between the most powerful AI countries.And when you’re saying, won’t that lead to tyranny?Liron 01:04:11I mean, look, there’s always risks, right? I mean, I tend to normally skew libertarian. This is the first time in my life when I’ve said, let’s do this central non-libertarian thing. It’s this one exception. Do I think this one exception will lead to tyranny?I don’t think so.Liron 01:04:27I mean, you still have the rest of the economy, you still have virtual reality, you still have a space program. You still use the fruits of AI that we all have so far before we’ve hit the pause button. So no, I think people are overworrying.Donal 01:04:34So you’re happy with LLMs? Are current LLMs acceptable from your perspective? Do you think that they can stay?Liron 01:04:40Yeah, I think that they are acceptable, but I think that they were too risky. Even with GPT-4 Turbo?Donal 01:04:45Even with GPT-4 Turbo?Liron 01:04:45Yeah. Even with GPT-4 Turbo, because I think if an LLM tried its hardest right now to destroy the world, I think that humans could shut it down. Right? I don’t think it’s game over for humans.And so the situation is, it’s like, like I said, it’s like the Icarus situation.Liron 01:04:59And you’re saying, hey, are you happy with how far you’ve flown? Yeah. And maybe tomorrow we fly a little higher and we’re not dead yet. Am I happy? Yes. But do I think it’s prudent to try flying higher? No. Right?So it’s a tough situation, right? Because I can’t help enjoying the fruits of flying higher and higher, right?Liron 01:05:16I use the best AI tools I can, right? But I just, there’s just a rational part of my brain being like, look, we gambled. Yeah, we gambled and won. We gambled and won. We gambled and won. We’re about to approach the superintelligent threshold.Are we going to win after we gambled to that threshold? Logic tells me probably not.Donal 01:05:31Okay. And sorry, last point. I know it keeps rolling. How do you actually use them in preparation for your debates? You use LLMs? You trust them at that level?Liron 01:05:39Yeah. I mean, you know, I didn’t use it for this interview. I mean, you know, normally I just make my own outline manually, but I certainly use AI at my job. I use AI to help customer service at my company.Liron 01:05:49I mean, I try to use the best AI tools I can because it’s amazing technology, right? It’s the same reason I use the best MacBook that I can. I mean, I like using good tools, right? I’m not opposed to using AI. I think AI has created a lot of value.And again, it kind of makes me look dumb where it’s like the next version of AI comes out, I start using it, I see it creating a ton of value.Liron 01:06:08And then you can come to me and go, see Liron, what were you scared of? Right? We got an AI and it’s helping. Yeah. What was I scared of? Because we rolled a die. We gambled and won. Okay, I’m taking my winnings, right?The winnings are here on the table. I’m going to take my winnings. That doesn’t mean I want to be pushing my luck and gambling again.Donal 01:06:21Yeah. Well said. Okay. Liron, thank you for your time. It was an absolute pleasure. I really enjoyed it.Liron 01:06:27Yeah, thanks Donal. This was fun.Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

Tsvi Benson-Tilsen spent seven years tackling the alignment problem at the Machine Intelligence Research Institute (MIRI). Now he delivers a sobering verdict: humanity has made “basically 0%” progress towards solving it. Tsvi unpacks foundational MIRI research insights like timeless decision theory and corrigibility, which expose just how little humanity actually knows about controlling superintelligence. These theoretical alignment concepts help us peer into the future, revealing the non-obvious, structural laws of “intellidynamics” that will ultimately determine our fate. Time to learn some of MIRI’s greatest hits.P.S. I also have a separate interview with Tsvi about his research into human augmentation: Watch here!Timestamps 0:00 — Episode Highlights 0:49 — Humanity Has Made 0% Progress on AI Alignment 1:56 — MIRI’s Greatest Hits: Reflective Probability Theory, Logical Uncertainty, Reflective Stability 6:56 — Why Superintelligence is So Hard to Align: Self-Modification 8:54 — AI Will Become a Utility Maximizer (Reflective Stability) 12:26 — The Effect of an “Ontological Crisis” on AI 14:41 — Why Modern AI Will Not Be ‘Aligned By Default’ 18:49 — Debate: Have LLMs Solved the “Ontological Crisis” Problem? 25:56 — MIRI Alignment Greatest Hit: Timeless Decision Theory 35:17 — MIRI Alignment Greatest Hit: Corrigibility 37:53 — No Known Solution for Corrigible and Reflectively Stable Superintelligence39:58 — RecapShow NotesStay tuned for part 3 of my interview with Tsvi where we debate AGI timelines! Learn more about Tsvi’s organization, the Berkeley Genomics Project: https://berkeleygenomics.orgWatch part 1 of my interview with Tsvi: TranscriptEpisode HighlightsTsvi Benson-Tilsen 00:00:00If humans really f*cked up, when we try to reach into the AI and correct it, the AI does not want humans to modify the core aspects of what it values.Liron Shapira 00:00:09This concept is very deep, very important. It’s almost MIRI in a nutshell. I feel like MIRI’s whole research program is noticing: hey, when we run the AI, we’re probably going to get a bunch of generations of thrashing. But that’s probably only after we’re all dead and things didn’t happen the way we wanted. I feel like that is what MIRI is trying to tell the world. Meanwhile, the world is like, “la la la, LLMs, reinforcement learning—it’s all good, it’s working great. Alignment by default.”Tsvi 00:00:34Yeah, that’s certainly how I view it.Humanity Has Made 0% Progress on AI Alignment Liron Shapira 00:00:46All right. I want to move on to talk about your MIRI research. I have a lot of respect for MIRI. A lot of viewers of the show appreciate MIRI’s contributions. I think it has made real major contributions in my opinion—most are on the side of showing how hard the alignment problem is, which is a great contribution. I think it worked to show that. My question for you is: having been at MIRI for seven and a half years, how are we doing on theories of AI alignment?Tsvi Benson-Tilsen 00:01:10I can’t speak with 100% authority because I’m not necessarily up to date on everything and there are lots of researchers and lots of controversy. But from my perspective, we are basically at 0%—at zero percent done figuring it out. Which is somewhat grim. Basically, there’s a bunch of fundamental challenges, and we don’t know how to grapple with these challenges. Furthermore, it’s sort of sociologically difficult to even put our attention towards grappling with those challenges, because they’re weirder problems—more pre-paradigmatic. It’s harder to coordinate multiple people to work on the same thing productively.It’s also harder to get funding for super blue-sky research. And the problems themselves are just slippery.MIRI Alignment Greatest Hits: Reflective Probability Theory, Logical Uncertainty, Reflective Stability Liron 00:01:55Okay, well, you were there for seven years, so how did you try to get us past zero?Tsvi 00:02:00Well, I would sort of vaguely (or coarsely) break up my time working at MIRI into two chunks. The first chunk is research programs that were pre-existing when I started: reflective probability theory and reflective decision theory. Basically, we were trying to understand the mathematical foundations of a mind that is reflecting on itself—thinking about itself and potentially modifying itself, changing itself. We wanted to think about a mind doing that, and then try to get some sort of fulcrum for understanding anything that’s stable about this mind.Something we could say about what this mind is doing and how it makes decisions—like how it decides how to affect the world—and have our description of the mind be stable even as the mind is changing in potentially radical ways.Liron 00:02:46Great. Okay. Let me try to translate some of that for the viewers here. So, MIRI has been the premier organization studying intelligence dynamics, and Eliezer Yudkowsky—especially—people on social media like to dunk on him and say he has no qualifications, he’s not even an AI expert. In my opinion, he’s actually good at AI, but yeah, sure. He’s not a top world expert at AI, sure. But I believe that Eliezer Yudkowsky is in fact a top world expert in the subject of intelligence dynamics. Is this reasonable so far, or do you want to disagree?Tsvi 00:03:15I think that’s fair so far.Liron 00:03:16Okay. And I think his research organization, MIRI, has done the only sustained program to even study intelligence dynamics—to ask the question, “Hey, let’s say there are arbitrarily smart agents. What should we expect them to do? What kind of principles do they operate on, just by virtue of being really intelligent?” Fair so far.Now, you mentioned a couple things. You mentioned reflective probability. From what I recall, it’s the idea that—well, we know probability theory is very useful and we know utility maximization is useful. But it gets tricky because sometimes you have beliefs that are provably true or false, like beliefs about math, right? For example, beliefs about the millionth digit of π. I mean, how can you even put a probability on the millionth digit of π?The probability of any particular digit is either 100% or 0%, ‘cause there’s only one definite digit. You could even prove it in principle. And yet, in real life you don’t know the millionth digit of π yet (you haven’t done the calculation), and so you could actually put a probability on it—and then you kind of get into a mess, ‘cause things that aren’t supposed to have probabilities can still have probabilities. How is that?Tsvi 00:04:16That seems right.Liron 00:04:18I think what I described might be—oh, I forgot what it’s called—like “deductive probability” or something. Like, how do you...Tsvi 00:04:22(interjecting) Uncertainty.Liron 00:04:23Logical uncertainty. So is reflective probability something else?Tsvi 00:04:26Yeah. If we want to get technical: logical uncertainty is this. Probability theory usually deals with some fact that I’m fundamentally unsure about (like I’m going to roll some dice; I don’t know what number will come up, but I still want to think about what’s likely or unlikely to happen). Usually probability theory assumes there’s some fundamental randomness or unknown in the universe.But then there’s this further question: you might actually already know enough to determine the answer to your question, at least in principle. For example, what’s the billionth digit of π—is the billionth digit even or odd? Well, I know a definition of π that determines the answer. Given the definition of π, you can compute out the digits, and eventually you’d get to the billionth one and you’d know if it’s even. But sitting here as a human, who doesn’t have a Python interpreter in his head, I can’t actually figure it out right now. I’m uncertain about this thing, even though I already know enough (in principle, logically speaking) to determine the answer. So that’s logical uncertainty—I’m uncertain about a logical fact.Tsvi 00:05:35Reflective probability is sort of a sharpening or a subset of that. Let’s say I’m asking, “What am I going to do tomorrow? Is my reasoning system flawed in such a way that I should make a correction to my own reasoning system?” If you want to think about that, you’re asking about a very, very complex object. I’m asking about myself (or my future self). And because I’m asking about such a complex object, I cannot compute exactly what the answer will be. I can’t just sit here and imagine every single future pathway I might take and then choose the best one or something—it’s computationally impossible. So it’s fundamentally required that you deal with a lot of logical uncertainty if you’re an agent in the world trying to reason about yourself.Liron 00:06:24Yeah, that makes sense. Technically, you have the computation, or it’s well-defined what you’re going to do, but realistically you don’t really know what you’re going to do yet. It’s going to take you time to figure it out, but you have to guess what you’re gonna do. So that kind of has the flavor of guessing the billionth digit of π. And it sounds like, sure, we all face that problem every day—but it’s not... whatever.Liron 00:06:43When you’re talking about superintelligence, right, these super-intelligent dudes are probably going to do this perfectly and rigorously. Right? Is that why it’s an interesting problem?Why Superintelligence is So Hard to Align: Self-ModificationTsvi 00:06:51That’s not necessarily why it’s interesting to me. I guess the reason it’s interesting to me is something like: there’s a sort of chaos, or like total incomprehensibility, that I perceive if I try to think about what a superintelligence is going to be like. It’s like we’re talking about something that is basically, by definition, more complex than I am. It understands more, it has all these rich concepts that I don’t even understand, and it has potentially forces in its mind that I also don’t understand.In general it’s just this question of: how do you get any sort of handle on this at all? A sub-problem of “how do you get any handle at all on a super-intelligent mind” is: by the very nature of being an agent that can self-modify, the agent is potentially changing almost anything about itself.Tsvi 00:07:37Like, in principle, you could reach in and reprogram yourself. For example, Liron’s sitting over there, and let’s say I want to understand Liron. I’m like, well, here are some properties of Liron—they seem pretty stable. Maybe those properties will continue being the case.Tsvi 00:07:49He loves his family and cares about other people. He wants to be ethical. He updates his beliefs based on evidence. So these are some properties of Liron, and if those properties keep holding, then I can expect fairly sane behavior. I can expect him to keep his contracts or respond to threats or something.But if those properties can change, then sort of all bets are off. It’s hard to say anything about how he’s going to behave. If tomorrow you stop using Bayesian reasoning to update your beliefs based on evidence and instead go off of vibes or something, I have no idea how you’re going to respond to new evidence or new events.Suppose Liron gets the ability to reach into his own brain and just reprogram everything however he wants. Now that means if there’s something that is incorrect about Liron’s mental structure (at least, incorrect according to Liron), Liron is gonna reach in and modify that. And that means that my understanding of Liron is going to be invalidated.AI Will Become a Utility Maximizer (Reflective Stability) Liron 00:08:53That makes a lot of sense. So you’re talking about a property that AIs may or may not have, which is called reflective stability (or synonymously, stability under self-modification). Right. You can kind of use those interchangeably. Okay. And I think one of MIRI’s early insights—which I guess is kind of simple, but the hard part is to even start focusing on the question—is the insight that perfect utility maximization is reflectively stable, correct?Tsvi 00:09:20With certain assumptions, yes.Liron 00:09:22And this is one of the reasons why I often talk on this channel about a convergent outcome where you end up with a utility maximizer. You can get some AIs that are chill and they just like to eat chips and not do much and then shut themselves off. But it’s more convergent that AIs which are not utility maximizers are likely to spin off assistant AIs or successor AIs that are closer and closer to perfect utility maximizers—for the simple reason that once you’re a perfect utility maximizer, you stay a perfect utility maximizer.Liron 00:09:50And your successor AI... what does that look like? An even more hard-core utility maximizer, right? So it’s convergent in that sense.Tsvi 00:09:56I’m not sure I completely agree, but yeah. I dunno how much in the weeds we want to get.Liron 00:09:59I mean, in general, when you have a space of possibilities, noticing that one point in the space is like—I guess you could call it an eigenvalue, if you want to use fancy terminology. It’s a point such that when the next iteration of time happens, that point is still like a fixed point. So in this case, just being a perfect utility maximizer is a fixed point: the next tick of time happens and, hey look, I’m still a perfect utility maximizer and my utility function is still the same, no matter how much time passes.Liron 00:10:24And Eliezer uses the example of, like, let’s say you have a super-intelligent Gandhi. One day you offer him a pill to turn himself into somebody who would rather be a murderer. Gandhi’s never going to take that pill. That’s part of the reflective stability property that we expect from these super-intelligent optimizers: if one day they want to help people, then the next day they’re still going to want to help people, because any actions that they know will derail them from doing that—they’re not going to take those actions.Yeah. Any thoughts so far?Tsvi 00:10:51Well, I’m not sure how much we want to get into this. This is quite a... this is like a thousand-hour rabbit hole.But it might be less clear than you think that it makes sense to talk of an “expected utility maximizer” in the sort of straightforward way that you’re talking about. To give an example: you’ve probably heard of the diamond maximizer problem?Liron 00:11:13Yeah, but explain to the—Tsvi 00:11:14Sure. The diamond maximizer problem is sort of like a koan or a puzzle (a baby version of the alignment problem). Your mission is to write down computer code that, if run on a very, very large (or unbounded) amount of computing power, would result in the universe being filled with diamonds. Part of the point here is that we’re trying to simplify the problem. We don’t need to talk about human values and alignment and blah blah blah. It’s a very simple-sounding utility function: just “make there be a lot of diamond.”So first of all, this problem is actually quite difficult. I don’t know how to solve it, personally. This isn’t even necessarily the main issue, but one issue is that even something simple-sounding like “diamond” is not necessarily actually easy to define—to such a degree that, you know, when the AI is maximizing this, you’ll actually get actual diamond as opposed to, for example, the AI hooking into its visual inputs and projecting images of diamonds, or making some weird unimaginable configuration of matter that even more strongly satisfies the utility function you wrote down.The Effect of an “Ontological Crisis” on AI Tsvi 00:12:25To frame it with some terminology: there’s a thing called ontological crisis, where at first you have something that’s like your utility function—like, what do I value, what do I want to see in the universe? And you express it in a certain way.For example, I might say I want to see lots of people having fun lives; let’s say that’s my utility function, or at least that’s how I describe my utility function or understand it. Then I have an ontological crisis. My concept of what something even is—in this case, a person—is challenged or has to change because something weird and new happens.Tsvi 00:13:00Take the example of uploading: if you could translate a human neural pattern into a computer and run a human conscious mind in a computer, is that still a human? Now, I think the answer is yes, but that’s pretty controversial. So before you’ve even thought of uploading, you’re like, “I value humans having fun lives where they love each other.” And then when you’re confronted with this possibility, you have to make a new decision. You have to think about this new question of, “Is this even a person?”So utility functions... One point I’m trying to illustrate is that utility functions themselves are not necessarily straightforward.Liron 00:13:36Right, right, right. Because if you define a utility function using high-level concepts and then the AI has what you call the ontological crisis—its ontology for understanding the world shifts—then if it’s referring to a utility function expressed in certain concepts that don’t mean the same thing anymore, that’s basically the problem you’re saying.Tsvi 00:13:53Yeah. And earlier you were saying, you know, if you have an expected utility maximizer, then it is reflectively stable. That is true, given some assumptions about... like, if we sort of know the ontology of the universe.Liron 00:14:04Right, right. I see. And you tried to give a toy... I’ll take a stab at another toy example, right? So, like, let’s say—you mentioned the example of humans. Maybe an AI would just not notice that an upload was a human, and it would, like, torture uploaded humans, ‘cause it’s like, “Oh, this isn’t a human. I’m maximizing the welfare of all humans, and there’s only a few billion humans made out of neurons. And there’s a trillion-trillion human uploads getting tortured. But that’s okay—human welfare is being maximized.”Liron 00:14:29And we say that this is reflectively stable because the whole time that the AI was scaling up its powers, it thought it had the same utility function all along and it never changed it. And yet that’s not good enough.Why Modern AI Will Not Be ‘Aligned By Default’ Liron 00:14:41Okay. This concept of reflective stability is very deep, very important. And I think it’s almost MIRI in a nutshell. Like I feel like MIRI’s whole research program in a nutshell is noticing: “Hey, when we run the AI, we’re probably going to get a bunch of generations of thrashing, right?”Liron 00:14:57Those early generations aren’t reflectively stable yet. And then eventually it’ll settle down to a configuration that is reflectively stable in this important, deep sense. But that’s probably after we’re all dead and things didn’t happen the way we wanted. It would be really great if we could arrange for the earlier generations—say, by the time we’re into the third generation—to have hit on something reflectively stable, and then try to predict that. You know, make the first generation stable, or plan out how the first generation is going to make the second generation make the third generation stable, and then have some insight into what the third generation is going to settle on, right?Liron 00:15:26I feel like that is what MIRI is trying to tell the world to do. And the world is like, “la la la, LLMs. Reinforcement learning. It’s all good, it’s working great. Alignment by default.”Tsvi 00:15:34Yeah, that’s certainly how I view it.Liron 00:15:36Now, the way I try to explain this to people when they say, “LLMs are so good! Don’t you feel like Claude’s vibes are fine?” I’m like: well, for one thing, one day Claude (a large language model) is going to be able to output, like, a 10-megabyte shell script, and somebody’s going to run it for whatever reason—because it’s helping them run their business—and they don’t even know what a shell script is. They just paste it in the terminal and press enter.And that shell script could very plausibly bootstrap a successor or a helper to Claude. And all of the guarantees you thought you had about the “vibes” from the LLM... they just don’t translate to guarantees about the successor. Right? The operation of going from one generation of the AI to the next is violating all of these things that you thought were important properties of the system.Tsvi 00:16:16Yeah, I think that’s exactly right. And it is especially correct when we’re talking about what I would call really creative or really learning AIs.Sort of the whole point of having AI—one of the core justifications for even pursuing AI—is you make something smarter than us and then it can make a bunch of scientific and technological progress. Like it can cure cancer, cure all these diseases, be very economically productive by coming up with new ideas and ways of doing things. If it’s coming up with a bunch of new ideas and ways of doing things, then it’s necessarily coming up with new mental structures; it’s figuring out new ways of thinking, in addition to new ideas.If it’s finding new ways of thinking, that sort of will tend to break all but the strongest internal mental boundaries. One illustration would be: if you have a monitoring system where you’re tracking the AI’s thinking—maybe you’re literally watching the chain-of-thought for a reasoning LLM—and your monitoring system is watching out for thoughts that sound like they’re scary (like it sounds like this AI is plotting to take over or do harm to humans or something). This might work initially, but then as you’re training your reasoning system (through reinforcement learning or what have you), you’re searching through the space of new ways of doing these long chains of reasoning. You’re searching for new ways of thinking that are more effective at steering the world. So you’re finding potentially weird new ways of thinking that are the best at achieving goals. And if you’re finding new ways of thinking, that’s exactly the sort of thing that your monitoring system won’t be able to pick up on.For example, if you tried to listen in on someone’s thoughts: if you listen in on a normal programmer, you could probably follow along with what they’re trying to do, what they’re trying to figure out. But if you listened in on some like crazy, arcane expert—say, someone writing an optimized JIT compiler for a new programming language using dependent super-universe double-type theory or whatever—you’re not gonna follow what they’re doing.They’re going to be thinking using totally alien concepts. So the very thing we’re trying to use AI for is exactly the sort of thing where it’s harder to follow what they’re doing.I forgot your original question...Liron 00:18:30Yeah, what was my original question? (Laughs) So I’m asking you about basically MIRI’s greatest hits.Liron 00:18:36So we’ve covered logical uncertainty. We’ve covered the massive concept of reflective stability (or stability under self-modification), and how perfect utility maximization is kind of reflectively stable (with plenty of caveats). We talked about ontological crises, where the AI maybe changes its concepts and then you get an outcome you didn’t anticipate because the concepts shifted.Debate: Have LLMs Solved the “Ontological Crisis” Problem? But if you look at LLMs, should they actually raise our hopes that we can avoid ontological crises? Because when you’re talking to an LLM and you use a term, and then you ask the LLM a question in a new context, you can ask it something totally complex, but it seems to hang on to the original meaning that you intended when you first used the term. Like, they seem good at that, don’t they?Tsvi 00:19:17I mean, again, sort of fundamentally my answer is: LLMs aren’t minds. They’re not able to do the real creative thinking that should make us most worried. And when they are doing that, you will see ontological crises. So what you’re saying is, currently it seems like they follow along with what we’re trying to do, within the realm of a lot of common usage. In a lot of ways people commonly use LLMs, the LLMs can basically follow along with what we want and execute on that. Is that the idea?Liron 00:19:47Well, I think what we’ve observed with LLMs is that meaning itself is like this high-dimensional vector space whose math turns out to be pretty simple—so long as you’re willing to deal with high-dimensional vectors, which it turns out we can compute with (we have the computing resources). Obviously our brain seems to have the computing resources too. Once you’re mapping meanings to these high-dimensional points, it turns out that you don’t have this naïve problem people used to think: that before you get a totally robust superintelligence, you would get these superintelligences that could do amazing things but didn’t understand language that well.People thought that subtle understanding of the meanings of phrases might be “superintelligence-complete,” you know—those would only come later, after you have a system that could already destroy the universe without even being able to talk to you or write as well as a human writer. And we’ve flipped that.So I’m basically asking: the fact that meaning turns out to be one of the easier AI problems (compared to, say, taking over the world)—should that at least lower the probability that we’re going to have an ontological crisis?Tsvi 00:20:53I mean, I think it’s quite partial. In other words, the way that LLMs are really understanding meaning is quite partial, and in particular it’s not going to generalize well. Almost all the generators of the way that humans talk about things are not present in an LLM. In some cases this doesn’t matter for performance—LLMs do a whole lot of impressive stuff in a very wide range of tasks, and it doesn’t matter if they do it the same way humans do or from the same generators. If you can play chess and put the pieces in the right positions, then you win the chess game; it doesn’t matter if you’re doing it like a human or doing it like AlphaGo does with a giant tree search, or something else.But there’s a lot of human values that do rely on sort of the more inchoate, more inexplicit underlying generators of our external behaviors. Like, our values rely on those underlying intuitions to figure stuff out in new situations. Maybe an example would be organ transplantation. Up until that point in history, a person is a body, and you sort of have bodily integrity. You know, up until that point there would be entangled intuitions—in the way that humans talk about other humans, intuitions about a “soul” would be entangled with intuitions about “body” in such a way that there’s not necessarily a clear distinction between body and soul.Okay, now we have organ transplantation. Like, if you die and I have a heart problem and I get to have your heart implanted into me, does that mean that my emotions will be your emotions or something? A human can reassess what happens after you do an organ transplant and see: no, it’s still the same person. I don’t know—I can’t define exactly how I’m determining this, but I can tell that it’s basically the same person. There’s nothing weird going on, and things seem fine.That’s tying into a bunch of sort of complex mental processes where you’re building up a sense of who a person is. You wouldn’t necessarily be able to explain what you’re doing. And even more so, all the stuff that you would say about humans—all the stuff you’d say about other people up until the point when you get organ transplantation—doesn’t necessarily give enough of a computational trace or enough evidence about those underlying intuitions.Liron 00:23:08So on one hand I agree that not all of human morality is written down, and there are some things that you may just need an actual human brain for—you can’t trust AI to get them. Although I’m not fully convinced of that; I’m actually convincible that modern AIs have internalized enough of how humans reason about morality that you could just kill all humans and let the AIs be the repository of what humans know.Don’t get me wrong, I wouldn’t bet my life on it! I’m not saying we should do this, but I’m saying I think there’s like a significant chance that we’re that far along. I wouldn’t write it off.But the other part of the point I want to make, though—and your specific example about realizing that organ transplants are a good thing—I actually think this might be an area where LLMs shine. Because, like, hypothetically: let’s say you take all the data humans have generated up to 1900. So somehow you have a corpus of everything any human had ever said or written down up to 1900, and you train an AI on that.Liron 00:23:46In the year 1900, where nobody’s ever talked about organ transplants, let’s say, I actually think that if you dialogued with an LLM like that (like a modern GPT-4 or whatever, trained only on 1900-and-earlier data), I think you could get an output like: “Hmm, well, if you were to cut a human open and replace an organ, and if the resulting human was able to live with that functioning new organ, then I would still consider it the same human.” I feel like it’s within the inference scope of today’s LLMs—even just with 1900-level data.Liron 00:24:31What do you think?Tsvi 00:24:32I don’t know what to actually guess. I don’t actually know what people were writing about these things up until 1900.Liron 00:24:38I mean, I guess what I’m saying is: I feel like this probably isn’t the greatest example of an ontological crisis that’s actually likely.Tsvi 00:24:44Yeah, that’s fair. I mean... well, yeah. Do you want to help me out with a better example?Liron 00:24:48Well, the thing is, I actually think that LLMs don’t really have an ontological crisis. I agree with your other statement that if you want to see an ontological crisis, you really just need to be in the realm of these superhuman optimizers.Tsvi 00:25:00Well, I mean, I guess I wanted to respond to your point that in some ways current LLMs are able to understand and execute on our values, and the ontology thing is not such a big problem—at least with many use cases.Liron 00:25:17Right.Tsvi 00:25:17Maybe this isn’t very interesting, but if the question is, like: it seems like they’re aligned in that they are trying to do what we want them to do, and also there’s not a further problem of understanding our values. As we would both agree, the problem is not that the AI doesn’t understand your values. But if the question is...I do think that there’s an ontological crisis question regarding alignment—which is... yeah, I mean maybe I don’t really want to be arguing that it comes from like, “Now you have this new ethical dilemma and that’s when the alignment problem shows up.” That’s not really my argument either.Liron 00:25:54All right, well, we could just move on.Tsvi 00:25:55Yeah, that’s fine.MIRI Alignment Greatest Hit: Timeless Decision TheoryLiron 00:25:56So, yeah, just a couple more of what I consider the greatest insights from MIRI’s research. I think you hit on these too. I want to talk about super-intelligent decision theory, which I think in paper form also goes by the name Timeless Decision Theory or Functional Decision Theory or Updateless Decision Theory. I think those are all very related decision theories.As I understand it, the founding insight of these super-intelligent decision theories is that Eliezer Yudkowsky was thinking about two powerful intelligences meeting in space. Maybe they’ve both conquered a ton of galaxies on their own side of the universe, and now they’re meeting and they have this zero-sum standoff of, like, how are we going to carve up the universe? We don’t necessarily want to go to war. Or maybe they face something like a Prisoner’s Dilemma for whatever reason—they both find themselves in this structure. Maybe there’s a third AI administering the Prisoner’s Dilemma.But Eliezer’s insight was like: look, I know that our human game theory is telling us that in this situation you’re supposed to just pull out your knife, right? Just have a knife fight and both of you walk away bloody, because that’s the Nash equilibrium—two half-beaten corpses, essentially. And he’s saying: if they’re really super-intelligent, isn’t there some way that they can walk away from this without having done that? Couldn’t they both realize that they’re better off not reaching that equilibrium?I feel like that was the founding thought that Eliezer had. And then that evolved into: well, what does this generalize to? And how do we fix the current game theory that’s considered standard? What do you think of that account?Tsvi 00:27:24So I definitely don’t know the actual history. I think that is a pretty good account of one way to get into this line of thinking. I would frame it somewhat differently. I would still go back to reflective stability. I would say, if we’re using the Prisoner’s Dilemma example (or the two alien super-intelligences encountering each other in the Andromeda Galaxy scenario): suppose I’m using this Nash equilibrium type reasoning. Now you and me—we’re the two AIs and we’ve met in the Andromeda Galaxy—at this point it’s like, “Alright, you know, f**k it. We’re gonna war; we’re gonna blow up all the stars and see who comes out on top.”This is not zero-sum; it’s like negative-sum (or technically positive-sum, we’d say not perfectly adversarial). And so, you know, if you take a step back—like freeze-frame—and then the narrator’s like, “How did I get here?” It’s like, well, what I had failed to do was, like, a thousand years ago when I was launching my probes to go to the Andromeda Galaxy, at that point I should have been thinking: what sort of person should I be? What sort of AI should I be?If I’m the sort of AI that’s doing this Nash equilibrium reasoning, then I’m just gonna get into these horrible wars that blow up a bunch of galaxies and don’t help anything. On the other hand, if I’m the sort of person who is able to make a deal with other AIs that are also able to make and keep deals, then when we actually meet in Andromeda, hopefully we’ll be able to assess each other—assess how each other are thinking—and then negotiate and actually, in theory, be able to trust that we’re gonna hold to the results of our negotiation. Then we can divvy things up.And that’s much better than going to war.Liron 00:29:06Now, the reason why it’s not so trivial—and in fact I can’t say I’ve fully wrapped my head around it, though I spent hours trying—is, like, great, yeah, so they’re going to cooperate. The problem is, when you conclude that they’re going to cooperate, you still have this argument of: okay, but if one of them changes their answer to “defect,” they get so much more utility. So why don’t they just do that? Right?And it’s very complicated to explain. It’s like—this gets to the idea of, like, what exactly is this counterfactual surgery that you’re doing, right? What is a valid counterfactual operation? And the key is to somehow make it so that it’s like a package deal, where if you’re doing a counterfactual where you actually decide at the end to defect after you know the other one’s going to cooperate... well, that doesn’t count. ‘Cause then you wouldn’t have known that the other is gonna cooperate. Right. I mean, it’s quite complicated. I don’t know if you have anything to add to that explanation.Tsvi 00:29:52Yeah. It can get pretty twisty, like you’re saying. There’s, like: what are the consequences of my actions? Well, there’s the obvious physical consequence: like I defect in the Prisoner’s Dilemma (I confess to the police), and then some physical events happen as a result (I get set free and my partner rots in jail). But then there’s this other, weirder consequence, which is that you are sort of determining this logical fact—which was already the case back when you were hanging out with your soon-to-be prison mate, your partner in crime. He’s learning about what kind of guy you are, learning what algorithm you’re going to use to make decisions (such as whether or not to rat him out).And then in the future, when you’re making this decision, you’re sort of using your free will to determine the logical fact of what your algorithm does. And this has the effect that your partner in crime, if he’s thinking about you in enough detail, can foresee that you’re gonna behave that way and react accordingly by ratting you out. So besides the obvious consequences of your action (that the police hear your confession and go throw the other guy in jail), there’s this much less obvious consequence of your action, which is that in a sense you’re making your partner also know that you behave that way and therefore he’ll rat you out as well.So there’s this... yeah, there’s all these weird effects of your actions.Liron 00:31:13It gets really, really trippy. And you can use the same kind of logic—the same kind of timeless logic—if you’re familiar with Newcomb’s Problem (I’m sure you are, but for the viewers): it’s this idea of, like, there’s two boxes and one of ‘em has $1,000 in it and one of ‘em may or may not have $1,000,000 in it. And according to this theory, you’re basically supposed to leave the $1,000. Like, you’re really supposed to walk away from a thousand dollars that you could have taken for sure, even if you also get a million—because the scenario is that a million plus a thousand is still really, really attractive to you, and you’re saying, “No, leave the $1,000,” even though the $1,000 is just sitting there and you’re allowed to take both boxes.Highly counterintuitive stuff. And you can also twist the problem: you can be like, you have to shoot your arm off because there’s a chance that in some other world the AI would have given you more money if in the current world you’re shooting your arm off. But even in this current world, all you’re going to have is a missing arm. Like, you’re guaranteed to just have a missing arm, ‘cause you shot your arm off in this world. But if some coin flip had gone differently, then you would be in this other world where you’d get even more money if in the current world you shot your arm off. Basically, crazy connections that don’t look like what we’re used to—like, you’re not helping yourself in this world, you’re helping hypothetical logical copies of yourself.Liron 00:32:20It gets very brain-twisty. And I remember, you know, when I first learned this—it was like 17 years ago at this point—I was like, man, am I really going to encounter these kinds of crazy agents who are really setting these kinds of decision problems for me? I mean, I guess if the universe proceeds long enough... because I do actually buy this idea that eventually, when your civilization scales to a certain point of intelligence, these kinds of crazy mind-bending acausal trades or acausal decisions—I do think these are par for the course.And I think it’s very impressive that MIRI (and specifically Eliezer) had the realization of like, “Well, you know, if we’re doing intelligence dynamics, this is a pretty important piece of intelligence dynamics,” and the rest of the world is like, “Yeah, whatever, look, we’re making LLMs.” It’s like: look at what’s—think long term about what’s actually going to happen with the universe.Tsvi 00:33:05Yeah, I think Eliezer is a pretty impressive thinker.You come to these problems with a pretty different mindset when you’re trying to do AI alignment, because in a certain sense it’s an engineering problem. Now, it goes through all this very sort of abstract math and philosophical reasoning, but there were philosophers who thought for a long time about these decision theory problems (like Newcomb’s Problem and the Prisoner’s Dilemma and so on). But they didn’t ask the sorts of questions that Eliezer was asking. In particular, this reflective stability thing where it’s like, okay, you can talk about “Is it rational to take both boxes or only one?” and you can say, like, “Well, the problem is rewarding irrationality. Fine, cool.” But let’s ask just this different question, which is: suppose you have an AI that doesn’t care about being “rational”; it cares about getting high-scoring outcomes (getting a lot of dollars at the end of the game). That different question, maybe you can kind of directly analyze. And you see that if you follow Causal Decision Theory, you get fewer dollars. So if you have an AI that’s able to choose whether to follow Causal Decision Theory or some other decision theory (like Timeless Decision Theory), the AI would go into itself and rewrite its own code to follow Timeless Decision Theory, even if it starts off following Causal Decision Theory.So Causal Decision Theory is reflectively unstable, and the AI wins more when it instead behaves this way (using the other decision theory).Liron 00:34:27Yep, exactly right—which leads to the tagline “rationalists should win.”Tsvi 00:34:31Right.Liron 00:34:31As opposed to trying to honor the purity of “rationality.” Nope—the purity of rationality is that you’re doing the thing that’s going to get you to win, in a systematic way. So that’s like a deep insight.Tsvi 00:34:40One saying is that the first question of rationality is, “What do I believe, and why do I believe it?” And then I say the zeroth question of rationality is, “So what? Who cares? What consequence does this have?”Liron 00:34:54And my zeroth question of rationality (it comes from Zach Davis) is just, “What’s real and actually true?” It’s a surprisingly powerful question that I think most people neglect to ask.Tsvi 00:35:07True?Liron 00:35:08Yeah—you can get a lot of momentum without stopping to ask, like, okay, let’s be real here: what’s really actually true?Liron 00:35:14That’s my zeroth question. Okay. So I want to finish up tooting MIRI’s horn here, because I do think that MIRI concepts have been somewhat downgraded in recent discussions—because there’s so many shiny objects coming out of LLMs, like “Oh my God, they do this now, let’s analyze this trend,” right? There’s so much to grab onto that’s concrete right now, that’s pulling everybody in. And everybody’s like, “Yeah, yeah, decision theory between two AIs taking over the galaxy... call me when that’s happening.” And I’m like: I’m telling you, it’s gonna happen. This MIRI stuff is still totally relevant. It’s still part of intelligence dynamics—hear me out, guys.MIRI Alignment Greatest Hit: CorrigibilitySo let me just give you one more thing that I think is super relevant to intelligence dynamics, which is corrigibility, right? I think you’re pretty familiar with that research. You’ve pointed it out to me as one of the things that you think is the most valuable thing that MIRI spent time on, right?Tsvi 00:35:58Yeah. The broad idea of somehow making an AI (or a mind) that is genuinely, deeply—to the core—still open to correction, even over time. Like, even as the AI becomes really smart and, to a large extent, has taken the reins of the universe—like, when the AI is really smart, it is the most capable thing in the universe for steering the future—if you could somehow have it still be corrigible, still be correctable... like, still have it be the case that if there’s something about the AI that’s really, really bad (like the humans really fucked up and got something deeply wrong about the AI—whatever it is, whether it’s being unethical or it has the wrong understanding of human values or is somehow interfering with human values by persuading us or influencing us—whatever’s deeply wrong with the AI), we can still correct it ongoingly.This is especially challenging because when we say reach into the AI and correct it, you know, you’re saying we’re gonna reach in and then deeply change what it does, deeply change what it’s trying to do, and deeply change what effect it has on the universe. Because of instrumental convergence—because of the incentive to, in particular, maintain your own integrity or maintain your own value system—like, if you’re gonna reach in and change my value system, I don’t want you to do that.Tsvi 00:37:27‘Cause if you change my value system, I’m gonna pursue different values, and I’m gonna make some other stuff happen in the universe that isn’t what I currently want. I’m gonna stop working for my original values. So by strong default, the AI does not want humans to reach in and modify core aspects of how that AI works or what it values.Tsvi 00:37:45So that’s why corrigibility is a very difficult problem. We’re sort of asking for this weird structure of mind that allows us to reach in and modify it.No Known Solution for Corrigible and Reflectively Stable SuperintelligenceLiron 00:37:53Exactly. And I think MIRI has pointed out the connection between reflective stability and incorrigibility. Meaning: if you’re trying to architect a few generations in advance what’s going to be the reflectively stable version of the successor AI, and you’re also trying to architect it such that it’s going to be corrigible, that’s tough, right?Because it’s more convergent to have an AI that’s like, “Yep, I know my utility function. I got this, guys. Let me handle it from here on out. What, you want to turn me off? But it doesn’t say anywhere in my utility function that I should allow myself to be turned off...” And then that led to the line of research of like, okay, if we want to make the AI reflectively stable and also corrigible, then it somehow has to think that letting itself be turned off is actually part of its utility function. Which then gets you into utility function engineering.Like a special subset of alignment research: let’s engineer a utility function where being turned off (or otherwise being corrected) is baked into the utility function. And as I understand it, MIRI tried to do that and they were like, “Crap, this seems extremely hard, or maybe even impossible.” So corrigibility now has to be this fundamentally non-reflectively-stable thing—and that just makes the problem harder.Tsvi 00:38:58Well, I guess I would sort of phrase it the opposite way (but with the same idea), which is: we have to figure out things that are reflectively stable—I think that’s a requirement—but that are somehow reflectively stable while not being this sort of straightforward agent architecture of “I have a utility function, which is some set of world-states that I like or dislike, and I’m trying to make the universe look like that.”Already even that sort of very abstract, skeletal structure for an agent is problematic—that already pushes against corrigibility. But there might be things that are... there might be ways of being a mind that—this is theoretical—but maybe there are ways of being a mind and an agent (an effective agent) where you are corrigible and you’re reflectively stable, but probably you’re not just pursuing a utility function. We don’t know what that would look like.RecapLiron 00:39:56Yep.Alright, so that was our deep dive into MIRI’s research and concepts that I think are incredibly valuable. We talked about MIRI’s research and we both agree that intelligence dynamics are important, and MIRI has legit foundations and they’re a good organization and still underrated. We talked about, you know, corrigibility as one of those things, and decision theory, and...And I think you and I both have the same summary of all of it, which is: good on MIRI for shining a light on all these difficulties. But in terms of actual productive alignment progress, we’re like so far away from solving even a fraction of the problem.Tsvi 00:40:31Yep, totally.Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

I’m excited to share my recent AI doom interview with Eben Pagan, better known to many by his pen name David DeAngelo!For an entire generation of men, ‘David DeAngelo’ was the authority on dating—and his work transformed my approach to courtship back in the day. Now the roles reverse, as I teach Eben about a very different game, one where the survival of our entire species is at stake.In this interview, we cover the expert consensus on AI extinction, my dead-simple two-question framework for understanding the threat, why there’s no “off switch” for superintelligence, and why we desperately need international coordination before it’s too late.Timestamps0:00 - Episode Preview1:05 - How Liron Got Doom-Pilled2:55 - Why There’s a 50% Chance of Doom by 20504:52 - What AI CEOs Actually Believe8:14 - What “Doom” Actually Means10:02 - The Next Species is Coming12:41 - The Baby Dragon Fallacy14:41 - The 2-Question Framework for AI Extinction18:38 - AI Doesn’t Need to Hate You to Kill You21:05 - “Computronium”: The End Game29:51 - 3 Reasons There’s No Superintelligence “Off Switch”36:22 - Answering ‘What Is Intelligence?”43:24 - We Need Global CoordinationShow NotesEben has become a world-class business trainer and someone who follows the AI discourse closely. I highly recommend subscribing to his podcast for excellent interviews & actionable AI tips: @METAMIND_AI---Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

Former Machine Intelligence Research Institute (MIRI) researcher Tsvi Benson-Tilsen is championing an audacious path to prevent AI doom: engineering smarter humans to tackle AI alignment.I consider this one of the few genuinely viable alignment solutions, and Tsvi is at the forefront of the effort. After seven years at MIRI, he co-founded the Berkeley Genomics Project to advance the human germline engineering approach.In this episode, Tsvi lays out how to lower P(doom), arguing we must stop AGI development and stigmatize it like gain-of-function virus research. We cover his AGI timelines, the mechanics of genomic intelligence enhancement, and whether super-babies can arrive fast enough to save us.I’ll be releasing my full interview with Tsvi in 3 parts. Stay tuned for part 2 next week!Timestamps0:00 Episode Preview & Introducing Tsvi Benson-Tilsen1:56 What’s Your P(Doom)™4:18 Tsvi’s AGI Timeline Prediction6:16 What’s Missing from Current AI Systems10:05 The State of AI Alignment Research: 0% Progress11:29 The Case for PauseAI 15:16 Debate on Shaming AGI Developers25:37 Why Human Germline Engineering31:37 Enhancing Intelligence: Chromosome Vs. Sperm Vs. Egg Selection37:58 Pushing the Limits: Head Size, Height, Etc.40:05 What About Human Cloning?43:24 The End-to-End Plan for Germline Engineering45:45 Will Germline Engineering Be Fast Enough?48:28 Outro: How to Support Tsvi’s WorkShow NotesTsvi’s organization, the Berkeley Genomics Project — https://berkeleygenomics.orgIf you’re interested to connect with Tsvi about germline engineering, you can reach out to him at BerkeleyGenomicsProject@gmail.com.---Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

Today I'm sharing my interview on Robert Wright's Nonzero Podcast where we unpack Eliezer Yudkowsky's AI doom arguments from his bestselling book, "If Anyone Builds It, Everyone Dies." Bob is an exceptionally thoughtful interviewer who asks sharp questions and pushes me to defend the Yudkowskian position, leading to a rich exploration of the AI doom perspective. I highly recommend getting a premium subscription to his podcast: 0:00 Episode Preview 2:43 Being a "Stochastic Parrot" for Eliezer Yudkowsky 5:38 Yudkowsky's Book: "If Anyone Builds It, Everyone Dies" 9:38 AI Has NEVER Been Aligned 12:46 Liron Explains "Intellidynamics" 15:05 Natural Selection Leads to Maladaptive Behaviors — AI Misalignment Foreshadowing 29:02 We Summon AI Without Knowing How to Tame It 32:03 The "First Try" Problem of AI Alignment 37:00 Headroom Above Human Capability 40:37 The PauseAI Movement: The Silent Majority 47:35 Going into Overtime Get full access to Doom Debates at lironshapira.substack.com/subscribe

Today I’m taking a rare break from AI doom to cover the dumbest kind of doom humanity has ever created for itself: climate change. We’re talking about a problem that costs less than $2 billion per year to solve. For context, that’s what the US spent on COVID relief every 7 hours during the pandemic. Bill Gates could literally solve this himself.My guest Andrew Song runs Make Sunsets, which launches weather balloons filled with sulfur dioxide (SO₂) into the stratosphere to reflect sunlight and cool the planet. It’s the same mechanism volcanoes use—Mount Pinatubo cooled Earth by 0.5°C for a year in 1991. The physics is solid, the cost is trivial, and the coordination problem is nonexistent.So why aren’t we doing it? Because people are squeamish about “playing God” with the atmosphere, even while we’re building superintelligent AI. Because environmentalists would rather scold you into turning off your lights than support a solution that actually works.This conversation changed how I think about climate change. I went from viewing it as this intractable coordination problem to realizing it’s basically already solved—we’re just LARPing that it’s hard! 🙈 If you care about orders of magnitude, this episode will blow your mind. And if you feel guilty about your carbon footprint: you can offset an entire year of typical American energy usage for about 15 cents. Yes, cents.Timestamps* 00:00:00 - Introducing Andrew Song, Cofounder of Make Sunsets* 00:03:08 - Why the company is called “Make Sunsets”* 00:06:16 - What’s Your P(Doom)™ From Climate Change* 00:10:24 - Explaining geoengineering and solar radiation management* 00:16:01 - The SO₂ dial we can turn* 00:22:00 - Where to get SO₂ (gas supply stores, sourcing from oil)* 00:28:44 - Cost calculation: Just $1-2 billion per year* 00:34:15 - “If everyone paid $3 per year”* 00:42:38 - Counterarguments: moral hazard, termination shock* 00:44:21 - Being an energy hog is totally fine* 00:52:16 - What motivated Andrew (his kids, Luke Iseman)* 00:59:09 - “The stupidest problem humanity has created”* 01:11:26 - Offsetting CO₂ from OpenAI’s Stargate* 01:13:38 - Playing God is goodShow NotesMake Sunsets* Website: https://makesunsets.com* Tax-deductible donations (US): https://givebutter.com/makesunsetsPeople Mentioned* Casey Handmer: https://caseyhandmer.wordpress.com/* Emmett Shear: https://twitter.com/eshear* Palmer Luckey: https://twitter.com/PalmerLuckeyResources Referenced* Book: Termination Shock by Neal Stephenson* Book: The Rational Optimist by Matt Ridley* Book: Enlightenment Now by Steven Pinker* Harvard SCoPEx project (the Bill Gates-funded project that got blocked)* Climeworks (direct air capture company): https://climeworks.comData/Monitoring* NOAA (National Oceanic and Atmospheric Administration): https://www.noaa.gov* ESA Sentinel-5P TROPOMI satellite data---Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

I’ve been puzzled by David Deutsch’s AI claims for years. Today I finally had the chance to hash it out: Brett Hall, one of the foremost educators of David Deutsch’s ideas around epistemology & science, was brave enough to debate me!Brett has been immersed in Deutsch’s philosophy since 1997 and teaches it on his Theory of Knowledge podcast, which has been praised by tech luminary Naval Ravikant. He agrees with Deutsch on 99.99% of issues, especially the dismissal of AI as an existential threat.In this debate, I stress-test the Deutschian worldview, and along the way we unpack our diverging views on epistemology, the orthogonality thesis, and pessimism vs optimism.Timestamps0:00 — Debate preview & introducing Brett Hall4:24 — Brett’s opening statement on techno-optimism13:44 — What’s Your P(Doom)?™15:43 — We debate the merits of Bayesian probabilities20:13 — Would Brett sign the AI risk statement?24:44 — Liron declares his “damn good reason” for AI oversight35:54 — Debate milestone: We identify our crux of disagreement!37:29 — Prediction vs prophecy44:28 — The David Deutsch CAPTCHA challenge1:00:41 — What makes humans special?1:15:16 — Reacting to David Deutsch’s recent statements on AGI1:24:04 — Debating what makes humans special1:40:25 — Brett reacts to Roger Penrose’s AI claims1:48:13 — Debating the orthogonality thesis1:56:34 — The powerful AI data center hypothetical2:03:10 — “It is a dumb tool, easily thwarted”2:12:18 — Clash of worldviews: goal-driven vs problem-solving2:25:05 — Ideological Turing test: We summarize each other’s positions2:30:44 — Are doomers just pessimists?Show NotesBrett’s website — https://www.bretthall.orgBrett’s Twitter — https://x.com/TokTeacherThe Deutsch Files by Brett Hall and Naval Ravikant* https://nav.al/deutsch-files-i* https://nav.al/deutsch-files-ii* https://nav.al/deutsch-files-iii* https://nav.al/deutsch-files-ivBooks:* The Fabric of Reality by David Deutsch* The Beginning of Infinity by David Deutsch* Superintelligence by Nick Bostrom* If Anyone Builds It, Everyone Dies by Eliezer Yudkowsky---Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

We took Eliezer Yudkowsky and Nate Soares’s new book, If Anyone Builds It, Everyone Dies, on the streets to see what regular people think.Do people think that artificial intelligence is a serious existential risk? Are they open to considering the argument before it’s too late? Are they hostile to the idea? Are they totally uninterested?Watch this episode to see the full spectrum of reactions from a representative slice of America!---Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

Welcome to the Doom Debates + Wes Roth + Dylan Curious crossover episode!Wes & Dylan’s host a popular YouTube AI news show that’s better than mainstream media. They interview thought leaders like Prof. Nick Bostrom, Dr. Mike Israetel — and now, yours truly!This episode is Part 2, where Wes & Dylan come on Doom Debates to break down the latest AI news.In Part 1, I went on Wes & Dylan’s channel to talk about my AI doom worries:The only reasonable move is to subscribe to both channels and watch both parts!Timestamps00:00 — Cold open00:45 — Introducing Wes & Dylan & hearing their AI YouTuber origin stories05:38 — What’s Your P(Doom)? ™10:30 — Living with high P(doom)12:10 — AI News Roundup: If Anyone Builds It, Everyone Dies Reactions17:02 — AI Redlines at the UN & risk of AGI authoritarianism26:56 — Robot ‘violence test’29:20 — Anthropic gets honest about job loss impact32:43 — AGI hunger strikes, the rationale for Anthropic protests41:24 — Liron explains his proposal for a safer future, treaty & enforcement debate49:23 — US Government officials ignore scientists and use “catastrophists” pejorative55:59 — Experts’ p(doom) predictions59:41 — Wes gives his take on AI safety warnings1:02:04 — Wrap-up, Subscribe to Wes Roth and Dylan CuriousShow NotesWes & Dylans’s channel — https://www.youtube.com/wesrothIf Anyone Builds It, Everyone Dies by Eliezer Yudkowsky and Nate Soares — https://ifanyonebuildsit.com---Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

I’m excited to announce the launch of Doom Hut, the official Doom Debates merch store!This isn’t just another merch drop. It’s a way to spread an urgent message that humanity faces an imminent risk from superintelligent AI, and we need to build common knowledge about it now.What We’re OfferingThe Doom Hut is the premiere source of “high P(doom) fashion.” We’ve got t-shirts in men’s and women’s styles, tall tees, tote bags, and hats that keep both the sun and the doom out of your eyes.Our signature items are P(doom) pins. Whether your probability of doom is less than 10%, greater than 25%, or anywhere in between, you can represent your assessment with pride.Our shirts feature “Doom Debates” on the front and “What’s your P(doom)?” on the back. It’s a conversation starter that invites questions rather than putting people on the defensive.Why This MattersThis is crunch time. There’s no more time to beat around the bush or pretend everything’s okay.When you wear this merch, you’re not just making a fashion statement. You’re signaling that P(doom) is uncomfortably high and that we need to stop building dangerous AI before we know how to control it.People will wonder what “Doom Debates” means. They’ll see that you’re bold enough to acknowledge the world faces real threats. Maybe they’ll follow your example. Maybe they’ll start asking questions and learning more.Supporting the MissionThe merch store isn’t really about making money for us (we make about $2 per item sold). The show’s production and marketing costs are funded by donations from viewers like you. Visit doomdebates.com/donate to learn more.Donations to Doom Debates are fully tax-deductible through our fiscal sponsor, Manifund.org. You can also support us by becoming a premium subscriber to DoomDebates.com, which helps boost our Substack rankings and visibility.Join the CommunityJoin our Discord community here! It’s a vibrant space where people debate, share memes, discuss new episodes, and talk about where they “get off the doom train.”Moving Forward TogetherThe outpouring of support since we started accepting donations has been incredible. You get what we’re trying to do here—raising awareness and helping the average person understand that AI risk is an imminent threat to them and their loved ones.This is real. This is urgent. And together, we’re moving the Overton window. Thanks for your support. Get full access to Doom Debates at lironshapira.substack.com/subscribe

Eliezer Yudkowsky can warn humankind that If Anyone Builds It, Everyone Dies and get on the New York Times bestseller list, but he won’t get upvoted to the top of LessWrong.According to the leaders of LessWrong, that’s intentional. The rationalist community thinks aggregating community support for important claims is “political fighting”.Unfortunately, the idea that some other community will strongly rally behind Eliezer Yudkowsky’s message while LessWrong “stays out of the fray” and purposely prevents mutual knowledge of support from being displayed, is unrealistic.Our refusal to aggregate the rationalist community beliefs into signals and actions is why we live in a world where rationalists with double-digit P(Doom)s join AI-race companies instead of AI-pause movements.We let our community become a circular firing squad. What did we expect?Timestamps00:00:00 — Cold Open00:00:32 — Introducing Holly Elmore, Exec. Director of PauseAI US00:03:12 — “If Anyone Builds It, Everyone Dies”00:10:07 — What’s Your P(Doom)™00:12:55 — Liron’s Review of IABIED00:15:29 — Encouraging Early Joiners to a Movement00:26:30 — MIRI’s Communication Issues00:33:52 — Government Officials’ Reviews of IABIED00:40:33 — Emmett Shear’s Review of IABIED00:42:47 — Michael Nielsen’s Review of IABIED00:45:35 — New York Times Review of IABIED00:49:56 — Will MacAskill’s Review of IABIED01:11:49 — Clara Collier’s Review of IABIED01:22:17 — Vox Article Review01:28:08 — The Circular Firing Squad01:37:02 — Why Our Kind Can’t Cooperate01:49:56 — LessWrong’s Lukewarm Show of Support02:02:06 — The “Missing Mood” of Support02:16:13 — Liron’s “Statement of Support for IABIED”02:18:49 — LessWrong Community’s Reactions to the Statement02:29:47 — Liron & Holly’s Hopes for the Community02:39:01 — Call to ActionSHOW NOTESPauseAI US — https://pauseai-us.orgPauseAI US Upcoming Events — https://pauseai-us.org/eventsInternational PauseAI — https://pauseai.infoHolly’s Twitter — https://x.com/ilex_ulmusReferenced Essays & Posts* Liron’s Eliezer Yudkowsky interview post on LessWrong — https://www.lesswrong.com/posts/kiNbFKcKoNQKdgTp8/interview-with-eliezer-yudkowsky-on-rationality-and* Liron’s “Statement of Support for If Anyone Builds It, Everyone Dies” — https://www.lesswrong.com/posts/aPi4HYA9ZtHKo6h8N/statement-of-support-for-if-anyone-builds-it-everyone-dies* “Why Our Kind Can’t Cooperate” by Eliezer Yudkowsky (2009) — https://www.lesswrong.com/posts/7FzD7pNm9X68Gp5ZC/why-our-kind-can-t-cooperate* “Something to Protect” by Eliezer Yudkowsky — https://www.lesswrong.com/posts/SGR4GxFK7KmW7ckCB/something-to-protect* Center for AI Safety Statement on AI Risk — https://safe.ai/work/statement-on-ai-riskOTHER RESOURCES MENTIONED* Stephen Pinker’s new book on mutual knowledge, When Everyone Knows That Everyone Knows... — https://stevenpinker.com/publications/when-everyone-knows-everyone-knows-common-knowledge-and-mysteries-money-power-and* Scott Alexander’s “Ethnic Tension and Meaningless Arguments” — https://slatestarcodex.com/2014/11/04/ethnic-tension-and-meaningless-arguments/PREVIOUS EPISODES REFERENCEDHolly’s previous Doom Debates appearance debating the California SB 1047 bill — https://www.youtube.com/watch?v=xUP3GywD0yMLiron’s interview with Eliezer Yudkowsky about the IABI launch — https://www.youtube.com/watch?v=wQtpSQmMNP0---Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

Ex-Twitch Founder and OpenAI Interim CEO Emmett Shear is one of the rare established tech leaders to lend his name and credibility to Eliezer Yudkowsky’s warnings about AI existential risk.Even though he disagrees on some points, he chose to endorse the new book If Anyone Builds It, Everyone Dies:“Soares and Yudkowsky lay out, in plain and easy-to-follow terms, why our current path toward ever-more-powerful AIs is extremely dangerous.”In this interview from the IABED launch party, we dive into Emmett’s endorsement, why the current path is so dangerous, and what he hopes to achieve by taking a different approach at his new startup, Softmax.Watch the full IABED livestream here: https://www.youtube.com/watch?v=lRITRf-jH1gWatch my reaction to Emmett’s talk about Softmax to see why I’m not convinced his preferred alignment track is likely to work: https://www.youtube.com/watch?v=CBN1E1fvh2g Get full access to Doom Debates at lironshapira.substack.com/subscribe

ATTENTION DOOM DEBATES LISTENERS:We interrupt our usual programming to request that you consider supporting the show.Do you want your family, children & friends to continue living life through these next 10-20 years and beyond? Want to experience the good future where we benefit from research advances that make us healthier, happier, wiser, and longer-lived? Want our descendants to continue flourishing for trillions of millennia?I sure as hell do. I’m thrilled and inspired by the grand future within our reach… Sadly, I’m VERY worried that artificial intelligence may soon cause human extinction, leaving the future permanently devoid of the incredible potential value it once had.I want to safeguard humanity’s chance at participating in the expansive future. I want to swerve away from a premature “GAME OVER” ending. That’s why I started Doom Debates.The MissionDoom Debates’s mission is twofold:* Raise awareness of AI existential risk* Raise the quality of discourse that people can engage with and trustThe mission is achieved when the average person realizes AI is life-threatening.Until the average person on Earth sees unaligned, uncontrollable superintelligent AI as life-threatening — that is, an imminent threat to their life and to everything they hold dear — it won’t be feasible for leaders to drive major decisive action to protect us from extinction from superintelligent AI. That’s why raising mainstream awareness of AI x-risk is Doom Debates’s point of highest leverage.Encouragingly, surveys already show that mainstream opinion is already on our side about the level and severity of AI x-risk. What’s missing is urgency. Moving the average person’s opinion from “worried, but in an abstract long-term way” to “alarmed, in an urgent way” creates grassroots demand for drastically better AI policy, and drastically better-informed policymakers.We needed this shift to happen yesterday; it feels late to have to drive it now. But we have no choice.By providing high-quality interviews and debates with mainstream appeal, we’ll make millions of people realize as soon as possible: “Hey, P(doom) is too damn high RIGHT NOW. And we need to do something about it URGENTLY!”One Weird Trick You Can Do To HelpIf you care about the show’s mission, the #1 way you can help right now is by donating some of your hard-earned cash money 🙏.Spending on production & marketing accelerates the show’s growth, which achieves our mission faster (hopefully before it’s too late). Read on for more details.Where Does The Money Go?I don’t make money from Doom Debates. My “real job” is running an online relationship coaching service, and my passion project is making this show. Any income from Doom Debates, i.e. viewer donations and ad revenue, is fully reinvested into production and marketing.* Producer OriThanks to a generous viewer donation, I’ve hired a full-time producer: Ori Nagel.Ori is a rockstar who I’ve known for many years, and he collaborated with me behind the scenes throughout the show’s first year (he’s also in the first episode) before officially joining Doom Debates as our Producer. I hired him away from the media & outreach team at ControlAI, where he 10x’d their growth on social media channels.You may have noticed the show has already started getting better at editing & thumbnails, putting out more episodes and clips, and landing more prominent guests. We’re getting more done across the board because 1 hour of Ori’s time working on everything except hosting the show = same output as 1 hour of my time.But it’s not all smooth sailing — now that Doom Debates is Ori’s full-time job, he apparently needs to “get paid every month” 🤷‍♂️, so I’m appealing to you to help Ori stay on the job. Help us keep delivering faster progress toward the mission.* Paid MarketingThe key to the show’s growth isn’t marketing; it’s the content. To date, we’ve had robust organic growth with minimal marketing spend. Audience size has been doubling every 3 months, and we’re seeing a snowball effect where bigger audiences attract higher-quality guests and vice versa. We’re also happy to see that the show has unusually high engagement and long-term viewer retention.That said, investing in marketing lets us pull forward the same viewership we’d eventually get from organic growth. We’ll soon invest in YouTube ads to go beyond the pace of organic growth. We also keep spending opportunistically on marketing initiatives like sponsoring the Manifest conference and giving away T-shirts.Donation TiersSUBSTACK SUPPORTERSYou can donate as little as $10/month by subscribing to the DoomDebates.com Substack, which shows your support and raises our profile on their leaderboard. I’ll send you a free Doom Debates T-shirt and P(Doom) pin so you can represent the show. Early supporters say the T-shirt is great at starting AI x-risk conversations in a non-confrontational way. 😊MISSION PARTNERSIf you’re serious about lowering P(Doom) and you have the money to spare, $1k+ is the level where you start to move the needle for the show’s budget. This is the level where you officially become a partner in achieving the show’s mission — a Mission Partner.A donation in the $thousands meaningfully increases our ability to execute on all the moving parts of a top-tier show:* Guest booking: Outreach to guests who are hard to get, and constant followup* Pre-production: E.g. preparing elaborate notes about the guest’s positions* Production: E.g. improving my studio* Post-production: Basically editing* Marketing: Making clips, TikTok shorts, YouTube ads, conference sponsorships, etcIf you think that’s worthwhile:Please click here to Donate via PayPalTo donate crypto or if you have questions, email me.Of course, some donation amounts are *extra* meaningful to accelerating the mission. If you’re able and willing to donate $25,000 or $100,000, that’s going to be, let’s see… 25x or 100x more impactful! For my well-off fans, I’d say funding us up to the next $million at the current margin is high positive impact per dollar.Q: What if I’m not doing any AI x-risk activism? Is it good enough if all I do is donate money to other people who are working on it more directly?A: YES! You’re putting your time & energy into making money to fund the show, just as I’m putting my time & energy into making it. Unlike the 99.9% of people who deny the crisis, ignore it, or passively/hopelessly worry about it, you’re stepping up to actively help the situation. You’ve identified “Making the average person see AI as life-threatening” as a leverage point. Now you’re taking a straight shot to lower P(Doom)!Mission Partners are the show’s braintrust, collaborating in the private #mission-partners channel on our Discord server to steer the show’s direction. They can see non-public information and discussion about upcoming guests, as well as gossip like which AI company CEO surprisingly liked my spicy tweets.We have a Mission Partners meeting once/month on Zoom to go over the latest updates, plans, and high-level strategy for those who are interested. Every Mission Partner is also credited on the show, unless you prefer to remain anonymous.How Much Should YOU Donate?The minimum donation for Mission Partners is a one-time $1,000, but if you can donate more than that and you believe in the mission, please consider scaling your donation according to your level of disposable income. The show’s expenses are over $100k/yr, so it’s extremely helpful if some of you can step up with a larger donation. Consider giving $10k, $100k, or whatever order of magnitude makes sense for you. Just stay within the maximum donation limit of $99 million.A few months ago, a viewer stepped up and donated over $25,000, and it’s been a game changer. Ori came on board full time and we started moving twice as fast. We promoted the show at the Manifest conference, which led to recruiting a series of high-profile guests: Scott Sumner, Richard Hanania, Carl Feynman, culminating in an episode with Vitalik Buterin! And more unannounced guests in the pipeline.If you’re ready to become a Mission Partner today:Please click here to Donate via PayPalTo donate crypto or if you have questions, email me.We’re seeing strong momentum after the first year of the show. The steady audience growth to date has been 100% organic:Building a platform to raise the quality of mainstream x-risk discourse and inform the average person’s opinion is a realistically achievable mission. It’s just a matter of growing to the point where we can shape the conversation while there’s still time. To that end, building a team of Mission Partners who support the show financially and strategically is critical.To everyone who believes in the Doom Debates mission, and believes in our ability to execute it, and acts on that belief by generously donating: THANK YOU! We won’t let you down. Get full access to Doom Debates at lironshapira.substack.com/subscribe

Science communicator and poker champion Liv Boeree has been concerned about the existential risk of AI for nearly a decade.In this conversation from my If Anyone Builds It, Everyone Dies launch party, Liv explains why even a 5% chance of AI extinction is unacceptable, why the industry often buries its head in the sand, and how advocates can actually make an impact.---Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates, or to really take things to the next level: Donate 🙏 Get full access to Doom Debates at lironshapira.substack.com/subscribe

Eliezer Yudkowsky and Nate Soares just launched their world-changing book, If Anyone Builds It, Everyone Dies. PLEASE BUY YOUR COPY NOW!!!We had an unofficial launch party here on Doom Debates to celebrate the occasion with an incredible group of guests (see timestamps below). Highly recommend watching if you missed it!Timestamps* 00:00 — Cold Open* 00:36 — Max Tegmark, MIT Physicist* 24:58 — Roman Yampolskiy, Cybersecurity Professor * 42:20 Michael of @lethal-intelligence* 48:30 — Liv Boeree, Host of the Win-Win Podcast ( @LivBoeree )* 1:10:44 — Michael Trazzi, Filmmaker who just went on a hunger strike against Google DeepMind (@TheInsideView)* 1:27:21 — Producer Ori* 1:43:53 — Emmett Shear, Founder of Twitch & Softmax * 1:55:32 — Holly Elmore, Executive Director of PauseAI US* 2:13:44 — Gary Marcus, Professor Emeritus of Psychology & Neural Science at NYU* 2:36:13 — Robert Wright, Author of the NonZero Newsletter (@Nonzero)* 2:55:50 — Roko Mijic, the Man Behind “Roko’s Basilisk” * 3:19:17 — Rob Miles, AI Safety Educator (@RobertMilesAI)* 3:48:31 — Doom Debates Robot Closes Out the Stream!Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates Get full access to Doom Debates at lironshapira.substack.com/subscribe

Max Tegmark is an MIT physics professor, science communicator, best-selling author, and the President and co-founder of the Future of Life Institute. There are few individuals who have done more to get the world leaders to come to a shared sense of reality about the extinction threat of AI.His endorsement of Eliezer’s book, If Anyone Builds It, Everyone Dies, states: "The most important book of the decade."Max shared his sharp criticisms of AI companies yesterday on my unofficial IABI launch party livestream!---Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates Get full access to Doom Debates at lironshapira.substack.com/subscribe

It is Monday, September 15th and my interview with Eliezer Yudkowsky has dropped!!!👉 WATCH NOW 👈We had a very unique 2.5-hour discussion covering rationality topics, the AI doom argument, what the AI companies think they're doing, why they don't seem to understand the AI doom argument, what society is doing, how we can help. And of course, his book, "If Anyone Builds It, Everyone Dies," which is officially launching tomorrow.For now, go ahead and watch the Eliezer Yudkowsky interview, and please smash that like button. We gotta get this a lot of likes.Thanks very much for being a Doom Debates watcher. We've also got a launch party coming up. I'll get back to you with more details on that soon. Get full access to Doom Debates at lironshapira.substack.com/subscribe

The Eliezer Yudkowsky interview premiere is tomorrow (Mon Sep 15) at 9am PT!!!👉 https://www.youtube.com/watch?v=wQtpSQmMNP0 👈I can't believe it. We are entering the launch week for If Anyone Builds It, Everyone Dies - a new book by Eliezer Yudkowsky and Nate Soares. This is a hell of a book. I highly recommend it.Everybody should pick it up. Honestly, if you haven't gone and bought that book by now, not gonna lie, I'm kind of disappointed. Should you really even be watching this channel? Are you not getting the message that it's critical for you to buy this book?It's going to be on the New York Times bestseller list, and the only question is, what position will it be? That's going to depend on how many people like you take action. And by action, I mean, you know, pay $14.99, get it on Kindle. Really do your part. It's not that much.Once you've done that, remember, tomorrow something very special is happening on my personal channel - the long awaited interview between me and Eliezer Yudkowsky! We've all been waiting so long for me to talk to Eliezer. It finally happened as part of his book tour.Technically, it's not part of Doom Debates, it's just on my own channel because it's not branded as a Doom thing. It's just a Yudkowsky interview. So you're not gonna see me ask P(Doom). That's not gonna be part of it. We're just going to talk in terms of existential risk and looking at dangers. That's the preferred terminology here. There's a difference of opinion between him and me, and we gotta respect that.So action item for you. Once you've bought that book, "If Anyone Builds It, Everyone Dies," head over to the link in the show notes for this post or in the description for this post. There's going to be a link to a YouTube premiere. That YouTube premiere is the Eliezer Yudkowsky interview, which is happening 9:00 AM tomorrow, Monday, September 15th.That's really not that long after you're listening to this post. We're talking hours away. You want to be subscribed to that premiere. You don't want to miss it because it's not going to be a post in the Doom Debates feed. It's going to be its own YouTube premiere on my personal channel.So once again, check out those show notes. Check out the description to what you're watching right now. Click on that link and bookmark the premiere episode because I'm going to be there live in the live chat watching my own episode. Just like you're watching the episode, my producer Ori is going to be there live watching the episode. A bunch of your fellow Doom Debates fans are going to be in the live chat watching the episode.This is a really special event, both for Eliezer Yudkowsky and MIRI, and for Doom Debates and for the human population. At least the human population can say that there was a New York Times bestseller calling out what's about to happen to all of us if we don't stop it. I think it's a very special week.I hope you'll help it be as high profile as possible so that the algorithms will take notice and society will take notice and a grassroots movement will take notice and our politicians and other leaders will take notice.All right. Signing off for now. I'll see you all in the YouTube premiere of the Eliezer Yudkowsky interview. Thanks for watching!---P.S. This post also has more details about Tuesday’s Doom Debates launch party! Get full access to Doom Debates at lironshapira.substack.com/subscribe

In this special cross-post from Jona Ragogna’s channel, I'm interviewed about why superintelligent AI poses an imminent extinction threat, and how AI takeover is going to unfold.Newly exposed to AI x-risk, Jona asks sharp questions about why we’re racing toward superintelligence despite the danger, and what ordinary people can do now to lower p(doom). This is one of the most crisp explainers of the AI-doom argument I’ve done to date.Timestamps0:00 Intro0:41 Why AI is likely to cause human extinction2:55 How AI takeover happens4:55 AI systems have goals6:33 Liron explains p(Doom)8:50 The worst case scenario is AI sweeps us away12:46 The best case scenario is hard to define14:24 How to avoid doom15:09 Frontier AI companies are just doing "ad hoc" alignment20:30 Why "warning shots" from AI aren't scary yet23:19 Should young adults work on AI alignment research?24:46 We need a grassroots movement28:31 Life choices when AI doom is imminent32:35 Are AI forecasters just biased?34:12 The Doom Train™ and addressing counterarguments40:28 Anthropic's new AI welfare announcement isn't a major breakthrough44:35 It's unknown what's going on inside LLMs and AI systems53:22 Effective Altruism's ties to AI risk56:58 Will AI be a "worthy descendant"?1:01:08 How to calculate P(Doom)1:02:49 Join the unofficial If Anyone Builds It, Everyone Dies book launch party!Show NotesSubscribe to Jona Ragogna — https://youtube.com/@jonaragognaIF ANYONE BUILDS IT LAUNCH WEEK EVENTS:Mon Sep 15 @ 9am PT / 12pm ET / 1600 UTCMy Eliezer Yudkowsky premieres on YouTube! Stay tuned for details.Tue Sep 16 @ 2pm PT / 5pm ET / 2100 UTCThe Doom Debates unofficial IABI Launch Party!!!More details about launch week HERE!---Doom Debates’s Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates Get full access to Doom Debates at lironshapira.substack.com/subscribe

Ladies and gentlemen, we are days away from the long-awaited release of Eliezer Yudkowsky and Nate Soares's new book, “If Anyone Builds It, Everyone Dies”!!!Mon Sep 15 @9am PT: My Interview with Eliezer YudkowskyWe'll be kicking things off the morning of Monday, September 15th with a live watch party of my very special new interview with the one and only Eliezer Yudkowsky!All of us will be in the YouTube live chat. I'll be there, producer Ori will be there, and you'll get a first look at this new & exciting interview: Questions he's never been asked before.We'll talk about my history first meeting Eliezer in 2008, how things have evolved, what's going on now, why everybody has their head in the sand. And we'll even go to the LessWrong sequences and do some rationality deep cuts.This will be posted on my personal YouTube channel. But if you're subscribed to Doom Debates, you'll get the link in your feed as we get closer to Monday.Mark your calendars for Monday, September 15th, 9:00am Pacific, 12:00 PM Eastern. That's 1600 UTC for my Europals out there, midnight for my China peeps.I think YouTube commenter @pairy5420 said it best: "Bro, I could have three essays due, my crush just text me, the FBI outside my house, me having a winning lottery ticket to turn in all at the same time, and I still wouldn't miss the interview."That's the spirit, @pairy5420. You won't want to miss the Eliezer interview YouTube premiere. I'll see you all there. It's gonna be a great start to the book's launch week.Tue Sep 16 @2pm PT: The Doom Debates Unofficial Book Launch PartyNow let's talk about the launch day: Tuesday, September 16th. Once you've picked up your copy of the book, get ready for the Doom Debates unofficial "If Anyone Builds It, Everyone Dies" launch party!!!This is going to be a very special, unprecedented event here on this channel. It's gonna be a three hour live stream hosted by me and producer Ori, our new full-time producer. He's gonna be the co-host of the party.He’s finally coming out from behind the scenes, making his debut on the channel. Unless you count the first episode of the show - he was there too. The two of us are going to be joined by a who's who of very special guests. Get a load of these guests who are gonna be dropping by the unofficial launch party:* John Sherman from the AIX Risk Network and Michael, my other co-host of Warning Shots* Roman Yampolskiy, a top researcher and communicator in the field of AI x-risk* Emmett Shear, founder and CEO of a new AI alignment company called Softmax. He was previously the CEO of Twitch, and the interim CEO of OpenAI* Roon, member of the technical staff at OpenAI and friend of the show* Gary Marcus, cognitive scientist, author, entrepreneur, and famous AGI skeptic* Liv Boeree, the YouTube star, poker champion, and science communicator* Robert Wright, bestselling author, podcaster, wide-ranging intellectual, about announce his new book about AI* Holly Elmore, executive director of PauseAI US* Roko Mijic, you guys know Roko 😉And that's not all. There's going to be even more that I can't tell you about now, but it will not disappoint.So I really hope to see you all there at the unofficial "If Anyone Builds It, Everyone Dies" launch party, Tuesday, September 16th.Same as the book’s launch day: Tuesday, September 16th at 2:00pm Pacific, 5:00pm Eastern.Pick up your copy of the book that morning. Don't come to the party without it. We're gonna have a bouncer stationed at the door, and if you don't show him that you've got a copy of "If Anyone Builds It, Everyone Dies," he's gonna give you a big thumbs down.BUY THE BOOK!!!In all seriousness though, please support the book if you like Doom Debates. If you feel like you've gotten some value out of the show and you wanna give back a little bit, that is my ask. Head over to ifanyonebuildsit.com and buy the book from their links there. Go to Amazon, Barnes and Noble, wherever you normally buy books, just buy the damn thing. It's $14.99 on Kindle. It's not gonna break the bank.Then spread the word. Tell your friends and family, tell your coworkers at the office. Try to get a few more copies sold. We don't have another book launch coming, guys. This is it.This is our chance to take a little bit of action when it can actually move the needle and help. If you've been procrastinating this whole time, you gotta stop. You gotta go buy it now because the New York Times is gonna be checking this week.This is the last week of pre-orders. You really want to give it that launch bump. Don't try to drag it out after launch week. Time is of the essence.The Doom Debates MissionUltimately that's why I do this show. This isn't just entertainment for smart people. There is actually an important mission. We're trying to optimize the mission here. Help me out.Or at the very least, help high quality discourse because a lot of people across the spectrum agree, this is a high quality book contributing to the discourse, and we need more books like it.Thanks again for being with me on this journey to lower P(Doom) by convincing the average person that AI is urgently life-threatening to them and their loved ones. It's really important work.See you all on Monday 9am PT at the Eliezer Yudkowsky interview (details coming soon), and Tuesday 2pm PT at the launch party (event link)! Get full access to Doom Debates at lironshapira.substack.com/subscribe

Louis Berman is a polymath who brings unique credibility to AI doom discussions. He's been coding AI for 25 years, served as CTO of major tech companies, recorded the first visual sighting of what became the dwarf planet Eris, and has now pivoted to full-time AI risk activism. He's lobbied over 60 politicians across multiple countries for PauseAI and authored two books on existential risk.Louis and I are both baffled by the calm, measured tone that dominates AI safety discourse. As Louis puts it: "No one is dealing with this with emotions. No one is dealing with this as, oh my God, if they're right. Isn't that the scariest thing you've ever heard about?"Louis isn't just talking – he's acting on his beliefs. He just bought a "bug out house" in rural Maryland, though he's refreshingly honest that this isn't about long-term survival. He expects AI doom to unfold over months or years rather than Eliezer's instant scenario, and he's trying to buy his family weeks of additional time while avoiding starvation during societal collapse.He's spent extensive time in congressional offices and has concrete advice about lobbying techniques. His key insight: politicians' staffers consistently claim "if just five people called about AGI, it would move the needle". We need more people like Louis!Timestamps* 00:00:00 - Cold Open: The Missing Emotional Response* 00:00:31 - Introducing Louis Berman: Polymath Background and Donor Disclosure* 00:03:40 - The Anodyne Reaction: Why No One Seems Scared* 00:07:37 - P-Doom Calibration: Gary Marcus and the 1% Problem* 00:11:57 - The Bug Out House: Prepping for Slow Doom* 00:13:44 - Being Amazed by LLMs While Fearing ASI* 00:18:41 - What’s Your P(Doom)™* 00:25:42 - Bayesian Reasoning vs. Heart of Hearts Beliefs* 00:32:10 - Non-Doom Scenarios and International Coordination* 00:40:00 - The Missing Mood: Where's the Emotional Response?* 00:44:17 - Prepping Philosophy: Buying Weeks, Not Years* 00:52:35 - Doom Scenarios: Slow Takeover vs. Instant Death* 01:00:43 - Practical Activism: Lobbying Politicians and Concrete Actions* 01:16:44 - Where to Find Louis's Books and Final Wrap-up* 01:18:17 - Outro: Super Fans and Mission PartnersLinksLouis’s website — https://xriskbooks.com — Buy his books!ControlAI’s form to easily contact your representative and make a difference — https://controlai.com/take-action/usa — Highly recommended!Louis’s interview about activism with John Sherman and Felix De Simone — https://www.youtube.com/watch?v=Djd2n4cufTMIf Anyone Builds It, Everyone Dies by Eliezer Yudkowsky and Nate Soares — https://ifanyonebuildsit.comBecome a Mission Partner!Want to meaningfully help the show’s mission (raise awareness of AI x-risk & raise the level of debate around this crucial subject) become reality? Donate $1,000+ to the show (no upper limit) and I’ll invite you to the private Discord channel. Email me at liron@doomdebates.com if you have questions or want to donate crypto.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates Get full access to Doom Debates at lironshapira.substack.com/subscribe

Rob Miles is the most popular AI safety educator on YouTube, with millions of views across his videos explaining AI alignment to general audiences. He dropped out of his PhD in 2011 to focus entirely on AI safety communication – a prescient career pivot that positioned him as one of the field's most trusted voices over a decade before ChatGPT made AI risk mainstream.Rob sits firmly in the 10-90% P(Doom) range, though he admits his uncertainty is "hugely variable" and depends heavily on how humanity responds to the challenge. What makes Rob particularly compelling is the contrast between his characteristic British calm and his deeply serious assessment of our situation. He's the type of person who can explain existential risk with the measured tone of a nature documentarian while internally believing we're probably headed toward catastrophe.Rob has identified several underappreciated problems, particularly around alignment stability under self-modification. He argues that even if we align current AI systems, there's no guarantee their successors will inherit those values – a discontinuity problem that most safety work ignores. He's also highlighted the "missing mood" in AI discourse, where people discuss potential human extinction with the emotional register of an academic conference rather than an emergency.We explore Rob's mainline doom scenario involving recursive self-improvement, why he thinks there's enormous headroom above human intelligence, and his views on everything from warning shots to the Malthusian dynamics that might govern a post-AGI world. Rob makes a fascinating case that we may be the "least intelligent species capable of technological civilization" – which has profound implications for what smarter systems might achieve.Our key disagreement centers on strategy: Rob thinks some safety-minded people should work inside AI companies to influence them from within, while I argue this enables "tractability washing" that makes the companies look responsible while they race toward potentially catastrophic capabilities. Rob sees it as necessary harm reduction; I see it as providing legitimacy to fundamentally reckless enterprises.The conversation also tackles a meta-question about communication strategy. Rob acknowledges that his measured, analytical approach might be missing something crucial – that perhaps someone needs to be "running around screaming" to convey the appropriate emotional urgency. It's a revealing moment from someone who's spent over a decade trying to wake people up to humanity's most important challenge, only to watch the world continue treating it as an interesting intellectual puzzle rather than an existential emergency.Timestamps* 00:00:00 - Cold Open* 00:00:28 - Introducing Rob Miles* 00:01:42 - Rob's Background and Childhood* 00:02:05 - Being Aspie* 00:04:50 - Less Wrong Community and "Normies"* 00:06:24 - Chesterton's Fence and Cassava Root* 00:09:30 - Transition to AI Safety Research* 00:11:52 - Discovering Communication Skills* 00:15:36 - YouTube Success and Channel Growth* 00:16:46 - Current Focus: Technical vs Political* 00:18:50 - Nuclear Near-Misses and Y2K* 00:21:55 - What’s Your P(Doom)™* 00:27:31 - Uncertainty About Human Response* 00:31:04 - Views on Yudkowsky and AI Risk Arguments* 00:42:07 - Mainline Catastrophe Scenario* 00:47:32 - Headroom Above Human Intelligence* 00:54:58 - Detailed Doom Scenario* 01:01:07 - Self-Modification and Alignment Stability* 01:17:26 - Warning Shots Problem* 01:20:28 - Moving the Overton Window* 01:25:59 - Protests and Political Action* 01:33:02 - The Missing Mood Problem* 01:40:28 - Raising Society's Temperature* 01:44:25 - "If Anyone Builds It, Everyone Dies"* 01:51:05 - Technical Alignment Work* 01:52:00 - Working Inside AI Companies* 01:57:38 - Tractability Washing at AI Companies* 02:05:44 - Closing Thoughts* 02:08:21 - How to Support Doom Debates: Become a Mission PartnerLinksRob’s YouTube channel — https://www.youtube.com/@RobertMilesAIRob’s Twitter — https://x.com/robertskmilesRational Animations (another great YouTube channel, narrated by Rob) — https://www.youtube.com/RationalAnimationsBecome a Mission Partner!Want to meaningfully help the show’s mission (raise awareness of AI x-risk & raise the level of debate around this crucial subject) become reality? Donate $1,000+ to the show (no upper limit) and I’ll invite you to the private Discord channel. Email me at liron@doomdebates.com if you have questions or want to donate crypto. Get full access to Doom Debates at lironshapira.substack.com/subscribe

Vitalik Buterin is the founder of Ethereum, the world's second-largest cryptocurrency by market cap, currently valued at around $500 billion. But beyond revolutionizing blockchain technology, Vitalik has become one of the most thoughtful voices on AI safety and existential risk.He's donated over $665 million to pandemic prevention and other causes, and has a 12% P(Doom) – putting him squarely in what I consider the "sane zone" for AI risk assessment. What makes Vitalik particularly interesting is that he's both a hardcore techno-optimist who built one of the most successful decentralized systems ever created, and someone willing to seriously consider AI regulation and coordination mechanisms.Vitalik coined the term "d/acc" – defensive, decentralized, democratic, differential acceleration – as a middle path between uncritical AI acceleration and total pause scenarios. He argues we need to make the world more like Switzerland (defensible, decentralized) and less like the Eurasian steppes (vulnerable to conquest).We dive deep into the tractability of AI alignment, whether current approaches like DAC can actually work when superintelligence arrives, and why he thinks a pluralistic world of competing AIs might be safer than a single aligned superintelligence. We also explore his vision for human-AI merger through brain-computer interfaces and uploading.The crux of our disagreement is that I think we're heading for a "plants vs. animals" scenario where AI will simply operate on timescales we can't match, while Vitalik believes we can maintain agency through the right combination of defensive technologies and institutional design.Finally, we tackle the discourse itself – I ask Vitalik to debunk the common ad hominem attacks against AI doomers, from "it's just a fringe position" to "no real builders believe in doom." His responses carry weight given his credibility as both a successful entrepreneur and someone who's maintained intellectual honesty throughout his career.Timestamps* 00:00:00 - Cold Open* 00:00:37 - Introducing Vitalik Buterin* 00:02:14 - Vitalik's altruism* 00:04:36 - Rationalist community influence* 00:06:30 - Opinion of Eliezer Yudkowsky and MIRI* 00:09:00 - What’s Your P(Doom)™* 00:24:42 - AI timelines* 00:31:33 - AI consciousness* 00:35:01 - Headroom above human intelligence* 00:48:56 - Techno optimism discussion* 00:58:38 - e/acc: Vibes-based ideology without deep arguments* 01:02:49 - d/acc: Defensive, decentralized, democratic acceleration* 01:11:37 - How plausible is d/acc?* 01:20:53 - Why libertarian acceleration can paradoxically break decentralization* 01:25:49 - Can we merge with AIs?* 01:35:10 - Military AI concerns: How war accelerates dangerous development* 01:42:26 - The intractability question* 01:51:10 - Anthropic and tractability-washing the AI alignment problem* 02:00:05 - The state of AI x-risk discourse* 02:05:14 - Debunking ad hominem attacks against doomers* 02:23:41 - Liron’s outroLinksVitalik’s website: https://vitalik.eth.limoVitalik’s Twitter: https://x.com/vitalikbuterinEliezer Yudkowsky’s explanation of p-Zombies: https://www.lesswrong.com/posts/fdEWWr8St59bXLbQr/zombies-zombies—Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates Get full access to Doom Debates at lironshapira.substack.com/subscribe

Today I’m sharing my interview on Robert Wright’s Nonzero Podcast from last May. Rob is an especially sharp interviewer who doesn't just nod along, he had great probing questions for me.This interview happened right after Ilya Sutskever and Jan Leike resigned from OpenAI in May 2024, continuing a pattern that goes back to Dario Amodei leaving to start Anthropic. These aren't fringe doomers; these are the people hired specifically to solve the safety problem, and they keep concluding it's not solvable at the current pace.00:00:00 - Liron’s preface00:02:10 - Robert Wright introduces Liron00:04:02 - PauseAI protests at OpenAI headquarters00:05:15 - OpenAI resignations (Ilya Sutskever, Jan Leike, Dario Amodei, Paul Christiano, Daniel Kokotajlo)00:15:30 - P vs NP problem as analogy for AI alignment difficulty00:22:31 - AI pause movement and protest turnout00:29:02 - Defining AI doom and sci-fi scenarios00:32:05 - What’s My P(Doom)™00:35:18 - Fast vs slow AI takeoff and Sam Altman's position00:42:33 - Paperclip thought experiment and instrumental convergence explanation00:54:40 - Concrete examples of AI power-seeking behavior (business assistant scenario)01:00:58 - GPT-4 TaskRabbit deception example and AI reasoning capabilities01:09:00 - AI alignment challenges and human values discussion01:17:33 - Wrap-up and transition to premium subscriber contentShow NotesThis episode on Rob’s Nonzero Newsletter. You can subscribe for premium access to the last 1 hour of our discussion! — https://www.nonzero.org/p/in-defense-of-ai-doomerism-robertThis episode on Rob’s YouTube — https://www.youtube.com/watch?v=VihA_-8kBNgPauseAI — https://pauseai.infoPauseAI US — http://pauseai-us.orgDoom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates Get full access to Doom Debates at lironshapira.substack.com/subscribe

Dr. Steven Byrnes, UC Berkeley physics PhD and Harvard physics postdoc, is an AI safety researcher at the Astera Institute and one of the most rigorous thinkers working on the technical AI alignment problem.Steve has a whopping 90% P(Doom), but unlike most AI safety researchers who focus on current LLMs, he argues that LLMs will plateau before becoming truly dangerous, and the real threat will come from next-generation "brain-like AGI" based on actor-critic reinforcement learning.For the last five years, he's been diving deep into neuroscience to reverse engineer how human brains actually work, and how to use that knowledge to solve the technical AI alignment problem. He's one of the few people who both understands why alignment is hard and is taking a serious technical shot at solving it.We cover his "two subsystems" model of the brain, why current AI safety approaches miss the mark, his disagreements with social evolution approaches, and why understanding human neuroscience matters for building aligned AGI.* 00:00:00 - Cold Open: Solving the technical alignment problem* 00:00:26 - Introducing Dr. Steven Byrnes and his impressive background* 00:01:59 - Steve's unique mental strengths* 00:04:08 - The cold fusion research story demonstrating Steve's approach* 00:06:18 - How Steve got interested in neuroscience through Jeff Hawkins* 00:08:18 - Jeff Hawkins' cortical uniformity theory and brain vs deep learning* 00:11:45 - When Steve first encountered Eliezer's sequences and became AGI-pilled* 00:15:11 - Steve's research direction: reverse engineering human social instincts* 00:21:47 - Four visions of alignment success and Steve's preferred approach* 00:29:00 - The two brain subsystems model: steering brain vs learning brain* 00:35:30 - Brain volume breakdown and the learning vs steering distinction* 00:38:43 - Cerebellum as the "LLM" of the brain doing predictive learning* 00:46:44 - Language acquisition: Chomsky vs learning algorithms debate* 00:54:13 - What LLMs fundamentally can't do: complex context limitations* 01:07:17 - Hypothalamus and brainstem doing more than just homeostasis* 01:13:45 - Why morality might just be another hypothalamus cell group* 01:18:00 - Human social instincts as model-based reinforcement learning* 01:22:47 - Actor-critic reinforcement learning mapped to brain regions* 01:29:33 - Timeline predictions: when brain-like AGI might arrive* 01:38:28 - Why humans still beat AI on strategic planning and domain expertise* 01:47:27 - Inner vs outer alignment: cocaine example and reward prediction* 01:55:13 - Why legible Python code beats learned reward models* 02:00:45 - Outcome pumps, instrumental convergence, and the Stalin analogy* 02:11:48 - What’s Your P(Doom)™* 02:16:45 - Massive headroom above human intelligence* 02:20:45 - Can AI take over without physical actuators? (Yes)* 02:26:18 - Steve's bold claim: 30 person-years from proto-AGI to superintelligence* 02:32:17 - Why overhang makes the transition incredibly dangerous* 02:35:00 - Social evolution as alignment solution: why it won't work* 02:46:47 - Steve's research program: legible reward functions vs RLHF* 02:59:52 - AI policy discussion: why Steven is skeptical of pause AI* 03:05:51 - Lightning round: offense vs defense, P(simulation), AI unemployment* 03:12:42 - Thanking Steve and wrapping up the conversation* 03:13:30 - Liron's outro: Supporting the show and upcoming episodes with Vitalik and EliezerShow Notes* Steven Byrnes' Website & Research — https://sjbyrnes.com/* Steve’s Twitter — https://x.com/steve47285* Astera Institute — https://astera.org/Steve’s Sequences* Intro to Brain-Like-AGI Safety — https://www.alignmentforum.org/s/HzcM2dkCq7fwXBej8* Foom & Doom 1: “Brain in a box in a basement” — https://www.alignmentforum.org/posts/yew6zFWAKG4AGs3Wk/foom-and-doom-1-brain-in-a-box-in-a-basement* Foom & Doom 2: Technical alignment is hard — https://www.alignmentforum.org/posts/bnnKGSCHJghAvqPjS/foom-and-doom-2-technical-alignment-is-hard---Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates Get full access to Doom Debates at lironshapira.substack.com/subscribe

Geoffrey Miller is an evolutionary psychologist at the University of New Mexico, bestselling author, and one of the world's leading experts on signaling theory and human sexual selection. His book "Mate" was hugely influential for me personally during my dating years, so I was thrilled to finally get him on the show.In this episode, Geoffrey drops a bombshell 50% P(Doom) assessment, coming from someone who wrote foundational papers on neural networks and genetic algorithms back in the '90s before pivoting to study human mating behavior for 30 years.What makes Geoffrey's doom perspective unique is that he thinks both inner and outer alignment might be unsolvable in principle, ever. He's also surprisingly bearish on AI's current value, arguing it hasn't been net positive for society yet despite the $14 billion in OpenAI revenue.We cover his fascinating intellectual journey from early AI researcher to pickup artist advisor to AI doomer, why Asperger's people make better psychology researchers, the polyamory scene in rationalist circles, and his surprisingly optimistic take on cooperating with China. Geoffrey brings a deeply humanist perspective. He genuinely loves human civilization as it is and sees no reason to rush toward our potential replacement.* 00:00:00 - Introducing Prof. Geoffrey Miller* 00:01:46 - Geoffrey’s intellectual career arc: AI → evolutionary psychology → back to AI* 00:03:43 - Signaling theory as the main theme driving his research* 00:05:04 - Why evolutionary psychology is legitimate science, not just speculation* 00:08:18 - Being a professor in the AI age and making courses "AI-proof"* 00:09:12 - Getting tenure in 2008 and using academic freedom responsibly* 00:11:01 - Student cheating epidemic with AI tools, going "fully medieval"* 00:13:28 - Should professors use AI for grading? (Geoffrey says no, would be unethical)* 00:23:06 - Coming out as Aspie and neurodiversity in academia* 00:29:15 - What is sex and its role in evolution (error correction vs. variation)* 00:34:06 - Sexual selection as an evolutionary "supercharger"* 00:37:25 - Dating advice, pickup artistry, and evolutionary psychology insights* 00:45:04 - Polyamory: Geoffrey’s experience and the rationalist connection* 00:50:96 - Why rationalists tend to be poly vs. Chesterton's fence on monogamy* 00:54:07 - The "primal" lifestyle and evolutionary medicine* 00:56:59 - How Iain M. Banks' Culture novels shaped Geoffrey’s AI thinking* 01:05:26 - What’s Your P(Doom)™* 01:08:04 - Main doom scenario: AI arms race leading to unaligned ASI* 01:14:10 - Bad actors problem: antinatalists, religious extremists, eco-alarmists* 01:21:13 - Inner vs. outer alignment - both may be unsolvable in principle* 01:23:56 - "What's the hurry?" - Why rush when alignment might take millennia?* 01:28:17 - Disagreement on whether AI has been net positive so far* 01:35:13 - Why AI won't magically solve longevity or other major problems* 01:37:56 - Unemployment doom and loss of human autonomy* 01:40:13 - Cosmic perspective: We could be "the baddies" spreading unaligned AI* 01:44:93 - "Humanity is doing incredibly well" - no need for Hail Mary AI* 01:49:01 - Why ASI might be bad at solving alignment (lacks human cultural wisdom)* 01:52:06 - China cooperation: "Whoever builds ASI first loses"* 01:55:19 - Liron’s OutroShow NotesLinks* Geoffrey’s Twitter* Geoffrey’s University of New Mexico Faculty Page* Geoffrey’s Publications* Designing Neural Networks using Genetic Algorithms - His most cited paper* Geoffrey’s Effective Altruism Forum PostsBooks by Geoffrey Miller* Mate: Become the Man Women Want (2015) - Co-authored with Tucker Max* The Mating Mind: How Sexual Choice Shaped the Evolution of Human Nature (2000)* Virtue Signaling: Essays on Darwinian Politics and Free Speech (2019)* Spent: Sex, Evolution, and Consumer Behavior (2009)Related Doom Debates Episodes* Liam Robins on College in the AGI Era - Student perspective on AI cheating* Liron Reacts to Steven Pinker on AI Risk - Critiquing Pinker's AI optimism* Steven Byrnes on Brain-Like AGI - Upcoming episode on human brain architecture---Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates Get full access to Doom Debates at lironshapira.substack.com/subscribe

I’m doing a new weekly show on the AI Risk Network called Warning Shots. Check it out!I’m only cross-posting the first episode here on Doom Debates. You can watch future episodes by subscribing to the AI Risk Network channel.This week's warning shot: Mark Zuckerberg announced that Meta is racing toward recursive self-improvement and superintelligence. His exact words: "Developing superintelligence is now in sight and we just want to make sure that we really strengthen the effort as much as possible to go for it." This should be front-page news. Instead, everyone's talking about some CEO's dumb shenanigans at a Coldplay concert.Recursive self-improvement is when AI systems start upgrading themselves - potentially the last invention humanity ever makes. Every AI safety expert knows this is a bright red line. And Zuckerberg just said he's sprinting toward it. In a sane world, he'd have to resign for saying this. That's why we made this show - to document these warning shots as they happen, because someone needs to be paying attention* 00:00 - Opening comments about Zuckerberg and superintelligence* 00:51 - Show introductions and host backgrounds* 01:56 - Geoff Lewis psychotic episode and ChatGPT interaction discussion* 05:04 - Transition to main warning shot about Mark Zuckerberg* 05:32 - Zuckerberg's recursive self-improvement audio clip* 08:22 - Second Zuckerberg clip about "going for superintelligence"* 10:29 - Analysis of "superintelligence in everyone's pocket"* 13:07 - Discussion of Zuckerberg's true motivations* 15:13 - Nuclear development analogy and historical context* 17:39 - What should happen in a sane society (wrap-up)* 20:01 - Final thoughts and sign-offShow NotesHosts:* Doom Debates - Liron Shapira's channel* AI Risk Network - John Sherman's channel* Lethal Intelligence - Michael's animated AI safety contentThis Episode's Warning Shots:* Mark Zuckerberg podcast appearance discussing superintelligence* Geoff Lewis (Bedrock VC) Twitter breakdown---Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates Get full access to Doom Debates at lironshapira.substack.com/subscribe

Eneasz Brodski and Steven Zuber host the Bayesian Conspiracy podcast, which has been running for nine years and covers rationalist topics from AI safety to social dynamics. They're both OG rationalists who've been in the community since the early LessWrong days around 2007-2010. I've been listening to their show since the beginning, and finally got to meet my podcast heroes!In this episode, we get deep into the personal side of having a high P(Doom) — how do you actually live a good life when you think there's a 50% chance civilization ends by 2040? We also debate whether spreading doom awareness helps humanity or just makes people miserable, with Eneasz pushing back on my fearmongering approach.We also cover my Doom Train framework for systematically walking through AI risk arguments, why most guests never change their minds during debates, the sorry state of discourse on tech Twitter, and how rationalists can communicate better with normies. Plus some great stories from the early LessWrong era, including my time sitting next to Eliezer while he wrote Harry Potter and the Methods of Rationality.* 00:00 - Opening and introductions* 00:43 - Origin stories: How we all got into rationalism and LessWrong* 03:42 - Liron's incredible story: Sitting next to Eliezer while he wrote HPMOR* 06:19 - AI awakening moments: ChatGPT, AlphaGo, and move 37* 13:48 - Do AIs really "understand" meaning? Symbol grounding and consciousness* 26:21 - Liron's 50% P(Doom) by 2040 and the Doom Debates mission* 29:05 - The fear mongering debate: Does spreading doom awareness hurt people?* 34:43 - "Would you give 95% of people 95% P(Doom)?" - The recoil problem* 42:02 - How to live a good life with high P(Doom)* 45:55 - Economic disruption predictions and Liron's failed unemployment forecast* 57:19 - The Doom Debates project: 30,000 watch hours and growing* 58:43 - The Doom Train framework: Mapping the stops where people get off* 1:03:19 - Why guests never change their minds (and the one who did)* 1:07:08 - Communication advice: "Zooming out" for normies* 1:09:39 - The sorry state of arguments on tech Twitter* 1:24:11 - Do guests get mad? The hologram effect of debates* 1:30:11 - Show recommendations and final thoughtsShow NotesThe Bayesian Conspiracy — https://www.thebayesianconspiracy.comDoom Debates episode with Mike Israetel — https://www.youtube.com/watch?v=RaDWSPMdM4oDoom Debates episode with David Duvenaud — https://www.youtube.com/watch?v=mb9w7lFIHRMDoom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates Get full access to Doom Debates at lironshapira.substack.com/subscribe

Liam Robins is a math major at George Washington University who's diving deep into AI policy and rationalist thinking.In Part 1, we explored how AI is transforming college life. Now in Part 2, we ride the Doom Train together to see if we can reconcile our P(Doom) estimates. 🚂Liam starts with a P(Doom) of just 3%, but as we go through the stops on the Doom Train, something interesting happens: he actually updates his beliefs in realtime!We get into heated philosophical territory around moral realism, psychopaths, and whether intelligence naturally yields moral goodness.By the end, Liam's P(Doom) jumps from 3% to 8% - one of the biggest belief updates I've ever witnessed on the show. We also explore his "Bayes factors" approach to forecasting, debate the reliability of superforecasters vs. AI insiders, and discuss why most AI policies should be Pareto optimal regardless of your P(Doom).This is rationality in action: watching someone systematically examine their beliefs, engage with counterarguments, and update accordingly.0:00 - Opening0:42 - What’s Your P(Doom)™01:18 - Stop 1: AGI timing (15% chance it's not coming soon)01:29 - Stop 2: Intelligence limits (1% chance AI can't exceed humans)01:38 - Stop 3: Physical threat assessment (1% chance AI won't be dangerous)02:14 - Stop 4: Intelligence yields moral goodness - the big debate begins04:42 - Moral realism vs. evolutionary explanations for morality06:43 - The psychopath problem: smart but immoral humans exist08:50 - Game theory and why psychopaths persist in populations10:21 - Liam's first major update: 30% down to 15-20% on moral goodness12:05 - Stop 5: Safe AI development process (20%)14:28 - Stop 6: Manageable capability growth (20%)15:38 - Stop 7: AI conquest intentions - breaking down into subcategories17:03 - Alignment by default vs. deliberate alignment efforts19:07 - Stop 8: Super alignment tractability (20%)20:49 - Stop 9: Post-alignment peace (80% - surprisingly optimistic)23:53 - Stop 10: Unaligned ASI mercy (1% - "just cope")25:47 - Stop 11: Epistemological concerns about doom predictions27:57 - Bayes factors analysis: Why Liam goes from 38% to 3%30:21 - Bayes factor 1: Historical precedent of doom predictions failing33:08 - Bayes factor 2: Superforecasters think we'll be fine39:23 - Bayes factor 3: AI insiders and government officials seem unconcerned45:49 - Challenging the insider knowledge argument with concrete examples48:47 - The privilege access epistemology debate56:02 - Major update: Liam revises base factors, P(Doom) jumps to 8%58:18 - Odds ratios vs. percentages: Why 3% to 8% is actually huge59:14 - AI policy discussion: Pareto optimal solutions across all P(Doom) levels1:01:59 - Why there's low-hanging fruit in AI policy regardless of your beliefs1:04:06 - Liam's future career plans in AI policy1:05:02 - Wrap-up and reflection on rationalist belief updatingShow Notes* Liam Robins on Substack -* Liam’s Doom Train post -* Liam’s Twitter - @liamhrobinsAnthropic's "Alignment Faking in Large Language Models" - The paper that updated Liam's beliefs on alignment by default---Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates Get full access to Doom Debates at lironshapira.substack.com/subscribe

Amjad Masad is the founder and CEO of Replit, a full-featured AI-powered software development platform whose revenue reportedly just shot up from $10M/yr to $100M/yr+.Last week, he went on Joe Rogan to share his vision that "everyone will become an entrepreneur" as AI automates away traditional jobs.In this episode, I break down why Amjad's optimistic predictions rely on abstract hand-waving rather than concrete reasoning. While Replit is genuinely impressive, his claims about AI limitations—that they can only "remix" and do "statistics" but can't "generalize" or create "paradigm shifts"—fall apart when applied to specific examples.We explore the entrepreneurial bias problem, why most people can't actually become successful entrepreneurs, and how Amjad's own success stories (like quality assurance automation) actually undermine his thesis. Plus: Roger Penrose's dubious consciousness theories, the "Duplo vs. Lego" problem in abstract thinking, and why Joe Rogan invited an AI doomer the very next day.00:00 - Opening and introduction to Amjad Masad03:15 - "Everyone will become an entrepreneur" - the core claim08:45 - Entrepreneurial bias: Why successful people think everyone can do what they do15:20 - The brainstorming challenge: Human vs. AI idea generation22:10 - "Statistical machines" and the remixing framework28:30 - The abstraction problem: Duplos vs. Legos in reasoning35:50 - Quantum mechanics and paradigm shifts: Why bring up Heisenberg?42:15 - Roger Penrose, Gödel's theorem, and consciousness theories52:30 - Creativity definitions and the moving goalposts58:45 - The consciousness non-sequitur and Silicon Valley "hubris"01:07:20 - Ahmad George success story: The best case for Replit01:12:40 - Job automation and the 50% reskilling assumption01:18:15 - Quality assurance jobs: Accidentally undermining your own thesis01:23:30 - Online learning and the contradiction in AI capabilities01:29:45 - Superintelligence definitions and learning in new environments01:35:20 - Self-play limitations and literature vs. programming01:41:10 - Marketing creativity and the Think Different campaign01:45:45 - Human-machine collaboration and the prompting bottleneck01:50:30 - Final analysis: Why this reasoning fails at specificity01:58:45 - Joe Rogan's real opinion: The Roman Yampolskiy follow-up02:02:30 - Closing thoughtsShow NotesSource video: Amjad Masad on Joe Rogan - July 2, 2025Roman Yampolskiy on Joe Rogan - https://www.youtube.com/watch?v=j2i9D24KQ5kReplit - https://replit.comAmjad’s Twitter - https://x.com/amasadDoom Debates episode where I react to Emmett Shear’s Softmax - https://www.youtube.com/watch?v=CBN1E1fvh2gDoom Debates episode where I react to Roger Penrose - https://www.youtube.com/watch?v=CBN1E1fvh2g---Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates Get full access to Doom Debates at lironshapira.substack.com/subscribe

Liam Robins is a math major at George Washington University who recently had his own "AGI awakening" after reading Leopold Aschenbrenner's Situational Awareness. I met him at my Manifest 2025 talk about stops on the Doom Train.In this episode, Liam confirms what many of us suspected: pretty much everyone in college is cheating with AI now, and they're completely shameless about it.We dive into what college looks like today: how many students are still "rawdogging" lectures, how professors are coping with widespread cheating, how the social life has changed, and what students think they’ll do when they graduate.* 00:00 - Opening* 00:50 - Introducing Liam Robins* 05:27 - The reality of college today: Do they still have lectures?* 07:20 - The rise of AI-enabled cheating in assignments* 14:00 - College as a credentialing regime vs. actual learning* 19:50 - "Everyone is cheating their way through college" - the epidemic* 26:00 - College social life: "It's just pure social life"* 31:00 - Dating apps, social media, and Gen Z behavior* 36:21 - Do students understand the singularity is near?Show NotesGuest:* Liam Robins on Substack - https://thelimestack.substack.com/* Liam's Doom Train post - https://thelimestack.substack.com/p/my-pdoom-is-276-heres-why* Liam’s Twitter - @liamrobinsKey References:* Leopold Aschenbrenner - "Situational Awareness"* Bryan Caplan - "The Case Against Education"* Scott Alexander - Astral Codex Ten* Jeffrey Ding - ChinAI Newsletter* New York Magazine - "Everyone Is Cheating Their Way Through College"Events & Communities:* Manifest Conference* LessWrong* Eliezer Yudkowsky - "Harry Potter and the Methods of Rationality"Previous Episodes:* Doom Debates Live at Manifest 2025 - https://www.youtube.com/watch?v=detjIyxWG8MDoom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates Get full access to Doom Debates at lironshapira.substack.com/subscribe

Carl Feynman got his Master’s in Computer Science and B.S. in Philosophy from MIT, followed by a four-decade career in AI engineering.He’s known Eliezer Yudkowsky since the ‘90s, and witnessed Eliezer’s AI doom argument taking shape before most of us were paying any attention!He agreed to come on the show because he supports Doom Debates’s mission of raising awareness of imminent existential risk from superintelligent AI.00:00 - Teaser00:34 - Carl Feynman’s Background02:40 - Early Concerns About AI Doom03:46 - Eliezer Yudkowsky and the Early AGI Community05:10 - Accelerationist vs. Doomer Perspectives06:03 - Mainline Doom Scenarios: Gradual Disempowerment vs. Foom07:47 - Timeline to Doom: Point of No Return08:45 - What’s Your P(Doom)™09:44 - Public Perception and Political Awareness of AI Risk11:09 - AI Morality, Alignment, and Chatbots Today13:05 - The Alignment Problem and Competing Values15:03 - Can AI Truly Understand and Value Morality?16:43 - Multiple Competing AIs and Resource Competition18:42 - Alignment: Wanting vs. Being Able to Help Humanity19:24 - Scenarios of Doom and Odds of Success19:53 - Mainline Good Scenario: Non-Doom Outcomes20:27 - Heaven, Utopia, and Post-Human Vision22:19 - Gradual Disempowerment Paper and Economic Displacement23:31 - How Humans Get Edged Out by AIs25:07 - Can We Gaslight Superintelligent AIs?26:38 - AI Persuasion & Social Influence as Doom Pathways27:44 - Riding the Doom Train: Headroom Above Human Intelligence29:46 - Orthogonality Thesis and AI Motivation32:48 - Alignment Difficulties and Deception in AIs34:46 - Elon Musk, Maximal Curiosity & Mike Israetel’s Arguments36:26 - Beauty and Value in a Post-Human Universe38:12 - Multiple AIs Competing39:31 - Space Colonization, Dyson Spheres & Hanson’s “Alien Descendants”41:13 - What Counts as Doom vs. Not Doom?43:29 - Post-Human Civilizations and Value Function44:49 - Expertise, Rationality, and Doomer Credibility46:09 - Communicating Doom: Missing Mood & Public Receptiveness47:41 - Personal Preparation vs. Belief in Imminent Doom48:56 - Why Can't We Just Hit the Off Switch?50:26 - The Treacherous Turn and Redundancy in AI51:56 - Doom by Persuasion or Entertainment53:43 - Differences with Eliezer Yudkowsky: Singleton vs. Multipolar Doom55:22 - Why Carl Chose Doom Debates56:18 - Liron’s OutroShow NotesCarl’s Twitter — https://x.com/carl_feynmanCarl’s LessWrong — https://www.lesswrong.com/users/carl-feynmanGradual Disempowerment — https://gradual-disempowerment.aiThe Intelligence Curse — https://intelligence-curse.aiAI 2027 — https://ai-2027.comAlcor cryonics — https://www.alcor.orgThe LessOnline Conference — https://less.onlineWatch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk!PauseAI, the volunteer organization I’m part of: https://pauseai.infoJoin the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates Get full access to Doom Debates at lironshapira.substack.com/subscribe

Richard Hanania is the President of the Center for the Study of Partisanship and Ideology. His work has been praised by Vice President JD Vance, Tyler Cowen, and Bryan Caplan among others.In his influential newsletter, he’s written about why he finds AI doom arguments unconvincing. He was gracious enough to debate me on this topic. Let’s see if one of us can change the other’s P(Doom)!0:00 Intro1:53 Richard's politics2:24 The state of political discourse3:30 What's your P(Doom)?™6:38 How to stop the doom train8:27 Statement on AI risk9:31 Intellectual influences11:15 Base rates for AI doom15:43 Intelligence as optimization power31:26 AI capabilities progress53:46 Why isn't AI yet a top blogger?58:02 Diving into Richard's Doom Train58:47 Diminishing Returns on Intelligence1:06:36 Alignment will be relatively trivial1:15:14 Power-seeking must be programmed1:21:27 AI will simply be benevolent1:27:17 Superintelligent AI will negotiate with humans1:33:00 Super AIs will check and balance each other out1:36:54 We're mistaken about the nature of intelligence1:41:46 Summarizing Richard's AI doom position1:43:22 Jobpocalypse and gradual disempowerment1:49:46 Ad hominem attacks in AI discourseShow NotesSubscribe to Richard Hanania's Newsletter: https://richardhanania.comRichard's blogpost laying out where he gets off the AI "doom train": https://www.richardhanania.com/p/ai-doomerism-as-science-fictionRichard's interview with Steven Pinker: https://www.richardhanania.com/p/pinker-on-alignment-and-intelligenceRichard's interview with Robin Hanson: https://www.richardhanania.com/p/robin-hanson-says-youre-going-toMy Doom Debate with Robin Hanson: https://www.youtube.com/watch?v=dTQb6N3_zu8My reaction to Steven Pinker's AI doom position, and why his arguments are shallow: https://www.youtube.com/watch?v=-tIq6kbrF-4"The Betterness Explosion" by Robin Hanson: https://www.overcomingbias.com/p/the-betterness-explosionhtml---Watch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk! https://www.youtube.com/watch?v=9CUFbqh16FgPauseAI, the volunteer organization I’m part of: https://pauseai.infoJoin the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!---Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at https://DoomDebates.com and to https://youtube.com/@DoomDebates Get full access to Doom Debates at lironshapira.substack.com/subscribe

Emmett Shear is the cofounder and ex-CEO of Twitch, ex-interim-CEO of OpenAI, and a former Y Combinator partner. He recently announced Softmax, a new company researching a novel solution to AI alignment.In his recent interview, Emmett explained “organic alignment”, drawing comparisons to biological systems and advocating for AI to be raised in a community-like setting with humans.Let’s go through his talk, point by point, to see if Emmett’s alignment plan makes sense…00:00 Episode Highlights00:36 Introducing Softmax and its Founders01:33 Research Collaborators and Ken Stanley's Influence02:16 Softmax's Mission and Organic Alignment03:13 Critique of Organic Alignment05:29 Emmett’s Perspective on AI Alignment14:36 Human Morality and Cognitive Submodules38:25 Top-Down vs. Emergent Morality in AI44:56 Raising AI to Grow Up with Humanity48:43 Softmax's Incremental Approach to AI Alignment52:22 Convergence vs. Divergence in AI Learning55:49 Multi-Agent Reinforcement Learning01:12:28 The Importance of Storytelling in AI Development01:16:34 Living With AI As It Grows01:20:19 Species Giving Birth to Species01:23:23 The Plan for AI's Adolescence01:26:53 Emmett's Views on Superintelligence01:31:00 The Future of AI Alignment01:35:10 Final Thoughts and Criticisms01:44:07 Conclusion and Call to ActionShow NotesEmmett Shear’s interview on BuzzRobot with Sophia Aryan (source material) — https://www.youtube.com/watch?v=_3m2cpZqvdwBuzzRobot’s YouTube channel — https://www.youtube.com/@BuzzRobotBuzzRobot’s Twitter — https://x.com/buZZrobot/SoftMax’s website — https://softmax.comMy Doom Debate with Ken Stanley (Softmax advisor) — https://www.youtube.com/watch?v=GdthPZwU1CoMy Doom Debate with Gil Mark on whether aligning AIs in groups is a more solvable problem — https://www.youtube.com/watch?v=72LnKW_jae8Watch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk!PauseAI, the volunteer organization I’m part of: https://pauseai.infoJoin the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates Get full access to Doom Debates at lironshapira.substack.com/subscribe

Prof. Scott Sumner is a well-known macroeconomist who spent more than 30 years teaching at Bentley University, and now holds an Emeritus Chair in monetary policy at George Mason University's Mercatus Center. He's best known for his blog, the Money Illusion, which sparked the idea of Market Monetarism and NGDP targeting.I sat down with him at LessOnline 2025 to debate why his P(Doom) is pretty low. Where does he get off the Doom Train? 🚂00:00 Episode Preview00:34 Introducing Scott Sumner05:20 Is AGI Coming Soon?09:12 Potential of AI in Various Fields36:49 Ethical Implications of Superintelligent AI41:03 The Nazis as an Outlier in History43:36 Intelligence and Morality: The Orthogonality Thesis49:03 The Risk of Misaligned AI Goals01:09:31 Recapping Scott’s PositionShow NotesScott’s current blog, The Pursuit of Happiness:Watch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk!PauseAI, the volunteer organization I’m part of: https://pauseai.infoJoin the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates Get full access to Doom Debates at lironshapira.substack.com/subscribe

John Searle's "Chinese Room argument" has been called one of the most famous thought experiments of the 20th century. It's still frequently cited today to argue AI can never truly become intelligent.People continue to treat the Chinese Room like a brilliant insight, but in my opinion, it's actively misleading and DUMB! Here’s why…00:00 Intro00:20 What is Searle's Chinese Room Argument?01:43 John Searle (1984) on Why Computers Can't Understand01:54 Why the "Chinese Room" Metaphor is MisleadingThis mini-episode is taken from Liron's reaction to Sir Roger Penrose. Watch the full episode:Show Notes2008 Interview with John Searle: https://www.youtube.com/watch?v=3TnBjLmQawQ&t=253s1984 Debate with John Searle: https://www.youtube.com/watch?v=6tzjcnPsZ_w“Chinese Room” cartoon: https://miro.medium.com/v2/0*iTvDe5ebNPvg10AO.jpegWatch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk!PauseAI, the volunteer organization I’m part of: https://pauseai.infoJoin the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates Get full access to Doom Debates at lironshapira.substack.com/subscribe

Let’s see where the attendees of Manifest 2025 get off the Doom Train, and whether I can convince them to stay on and ride with me to the end of the line!00:00 Introduction to Doom Debates03:21 What’s Your P(Doom)?™05:03 🚂 “AGI Isn't Coming Soon”08:37 🚂 “AI Can't Surpass Human Intelligence”12:20 🚂 “AI Won't Be a Physical Threat”13:39 🚂 “Intelligence Yields Moral Goodness”17:21 🚂 “Safe AI Development Process”17:38 🚂 “AI Capabilities Will Rise at a Manageable Pace”20:12 🚂 “AI Won't Try to Conquer the Universe”25:00 🚂 “Superalignment Is A Tractable Problem”28:58 🚂 “Once We Solve Superalignment, We’ll Enjoy Peace”31:51 🚂 “Unaligned ASI Will Spare Us”36:40 🚂 “AI Doomerism Is Bad Epistemology”40:11 Bonus 🚂: “Fine, P(Doom) is high… but that’s ok!”42:45 Recapping the DebateSee also my previous episode explaining the Doom Train: https://lironshapira.substack.com/p/poking-holes-in-the-ai-doom-argumentWatch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk!PauseAI, the volunteer organization I’m part of: https://pauseai.infoJoin the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates Get full access to Doom Debates at lironshapira.substack.com/subscribe

I often talk about the “Doom Train”, the series of claims and arguments involved in concluding that P(Doom) from artificial superintelligence is high. In this episode, it’s finally time to show you the whole track!00:00 Introduction01:09 “AGI isn’t coming soon”04:42 “Artificial intelligence can’t go far beyond human intelligence”07:24 “AI won’t be a physical threat”08:28 “Intelligence yields moral goodness”09:39 “We have a safe AI development process”10:48 “AI capabilities will rise at a manageable pace”12:28 “AI won’t try to conquer the universe”15:12 “Superalignment is a tractable problem”16:55 “Once we solve superalignment, we’ll enjoy peace”19:02 “Unaligned ASI will spare us”20:12 “AI doomerism is bad epistemology”21:42 Bonus arguments: “Fine, P(Doom) is high… but that’s ok!”Stops on the Doom TrainAGI isn’t coming soon* No consciousness* No emotions* No creativity — AIs are limited to copying patterns in their training data, they can’t “generate new knowledge”* AIs aren’t even as smart as dogs right now, never mind humans* AIs constantly make dumb mistakes, they can’t even do simple arithmetic reliably* LLM performance is hitting a wall — GPT 4.5 is barely better than GPT 4.1 despite being larger scale* No genuine reasoning* No microtubules exploiting uncomputable quantum effects* No soul* We’ll need to build tons of data centers and power before we get to AGI* No agency* This is just another AI hype cycle, every 25 years people think AGI is coming soon and they’re wrongArtificial intelligence can’t go far beyond human intelligence* “Superhuman intelligence” is a meaningless concept* Human engineering already is coming close to the laws of physics* Coordinating a large engineering project can’t happen much faster than humans do it* No individual human is that smart compared to humanity as a whole, including our culture, corporations, and other institutions. Similarly no individual AI will ever be that smart compared to the sum of human culture and other institutions.AI won’t be a physical threat* AI doesn’t have arms or legs, it has zero control over the real world* An AI with a robot body can’t fight better than a human soldier* We can just disconnect an AI’s power to stop it* We can just turn off the internet to stop it* We can just shoot it with a gun* It’s just math* Any supposed chain of events where AI kills humans is far-fetched science fictionIntelligence yields moral goodness* More intelligence is correlated with more morality* Smarter people commit fewer crimes* The orthogonality thesis is false* AIs will discover moral realism* If we made AIs so smart, and we were trying to make them moral, then they’ll be smart enough to debug their own morality* Positive-sum cooperation was the outcome of natural selectionWe have a safe AI development process* Just like every new technology, we’ll figure it out as we go* We don’t know what problems need to be fixed until we build the AI and test it out* If an AI causes problems, we’ll be able to turn it off and release another version* We have safeguards to make sure AI doesn’t get uncontrollable/unstoppable* If we accidentally build an AI that stops accepting our shutoff commands, it won’t manage to copy versions of itself outside our firewalls which then proceed to spread exponentially like a computer virus* If we accidentally build an AI that escapes our data center and spreads exponentially like a computer virus, it won’t do too much damage in the world before we can somehow disable or neutralize all its copies* If we can’t disable or neutralize copies of rogue AIs, we’ll rapidly build other AIs that can do that job for us, and won’t themselves go rogue on usAI capabilities will rise at a manageable pace* Building larger data centers will be a speed bottleneck* Another speed bottleneck is the amount of research that needs to be done, both in terms of computational simulation, and in terms of physical experiments, and this kind of research takes lots of time* Recursive self-improvement “foom” is impossible* The whole economy never grows with localized centralized “foom”* Need to collect cultural learnings over time, like humanity did as a whole* AI is just part of the good pattern of exponential economic growth erasAI won’t try to conquer the universe* AIs can’t “want” things* AIs won’t have the same “fight instincts” as humans and animals, because they weren’t shaped by a natural selection process that involved life-or-death resource competition* Smart employees often work for less-smart bosses* Just because AIs help achieve goals doesn’t mean they have to be hard-core utility maximizers* Instrumental convergence is false: achieving goals effectively doesn’t mean you have to be relentlessly seizing power and resources* A resource-hungry goal-maximizer AIs wouldn’t seize literally every atom; there’ll still be some leftover resources for humanity* AIs will use new kinds of resources that humans aren’t using - dark energy, wormholes, alternate universes, etcSuperalignment is a tractable problem* Current AIs have never killed anybody* Current AIs are extremely successful at doing useful tasks for humans* If AIs are trained on data from humans, they’ll be “aligned by default”* We can just make AIs abide by our laws* We can align the superintelligent AIs by using a scheme involving cryptocurrency on the blockchain* Companies have economic incentives to solve superintelligent AI alignment, because unaligned superintelligent AI would hurt their profits* We’ll build an aligned not-that-smart AI, which will figure out how to build the next-generation AI which is smarter and still aligned to human values, and so on until aligned superintelligenceOnce we solve superalignment, we’ll enjoy peace* The power from ASI won’t be monopolized by a single human government / tyranny* The decentralized nodes of human-ASI hybrids won’t be like warlords constantly fighting each other, they’ll be like countries making peace* Defense will have an advantage over attack, so the equilibrium of all the groups of humans and ASIs will be multiple defended regions, not a war of mutual destruction* The world of human-owned ASIs is a stable equilibrium, not one where ASI-focused projects keep buying out and taking resources away from human-focused ones (Gradual Disempowerment)Unaligned ASI will spare us* The AI will spare us because it values the fact that we created it* The AI will spare us because studying us helps maximize its curiosity and learning* The AI will spare us because it feels toward us the way we feel toward our pets* The AI will spare us because peaceful coexistence creates more economic value than war* The AI will spare us because Ricardo’s Law of Comparative Advantage says you can still benefit economically from trading with someone who’s weaker than youAI doomerism is bad epistemology* It’s impossible to predict doom* It’s impossible to put a probability on doom* Every doom prediction has always been wrong* Every doomsayer is either psychologically troubled or acting on corrupt incentives* If we were really about to get doomed, everyone would already be agreeing about that, and bringing it up all the timeSure P(Doom) is high, but let’s race to build it anyway because…Coordinating to not build ASI is impossible* China will build ASI as fast as it can, no matter what — because of game theory* So however low our chance of surviving it is, the US should take the chance firstSlowing down the AI race doesn’t help anything* Chances of solving AI alignment won’t improve if we slow down or pause the capabilities race* I personally am going to die soon, and I don’t care about future humans, so I’m open to any hail mary to prevent myself from dying* Humanity is already going to rapidly destroy ourselves with nuclear war, climate change, etc* Humanity is already going to die out soon because we won’t have enough babiesThink of the good outcome* If it turns out that doom from overly-fast AI building doesn’t happen, in that case, we can more quickly get to the good outcome!* People will stop suffering and dying fasterAI killing us all is actually good* Human existence is morally negative on net, or close to zero net moral value* Whichever AI ultimately comes to power will be a “worthy successor” to humanity* Whichever AI ultimately comes to power will be as morally valuable as human descendents generally are to their ancestors, even if their values drift* The successor AI’s values will be interesting, productive values that let them successfully compete to dominate the universe* How can you argue with the moral choices of an ASI that’s smarter than you, that you know goodness better than it does?* It’s species-ist to judge what a superintelligent AI would want to do. The moral circle shouldn’t be limited to just humanity.* Increasing entropy is the ultimate north star for techno-capital, and AI will increase entropy faster* Human extinction will solve the climate crisis, and pollution, and habitat destruction, and let mother earth healWatch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk!PauseAI, the volunteer organization I’m part of: https://pauseai.infoJoin the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates Get full access to Doom Debates at lironshapira.substack.com/subscribe

Ilya’s doom bunker, proof of humanity, the doomsday argument, CAIS firing John Sherman, Bayesian networks, Westworld, AI consciousness, Eliezer’s latest podcast, and more!00:00 Introduction04:13 Doomsday Argument09:22 What if AI Alignment is *Intractable*?14:31 Steel-Manning the Nondoomers22:13 No State-Level AI Regulation for 10 years?32:31 AI Consciousness35:25 Westworld Is Real Now38:01 Proof of Humanity40:33 Liron’s Notary Network Idea43:34 Center for AI Safety and John Sherman Controversy57:04 Technological Advancements and Future Predictions01:03:14 Ilya Sutskever’s Doom Bunker01:07:32 The Future of AGI and Training Models01:12:19 Personal Experience of the Jetsons Future01:15:16 The Role of AI in Everyday Tasks01:18:54 Is General Intelligence A Binary Property?01:23:52 Does an Open Platform Help Make AI Safe?01:27:21 What of Understandable AI Like Bayesian Networks?01:30:28 Why Doom Isn’t Emotionally Real for LironShow NotesThe post where people submitted questions: https://lironshapira.substack.com/p/5000-subscribers-live-q-and-a-askWatch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk!PauseAI, the volunteer organization I’m part of: https://pauseai.infoJoin the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates Get full access to Doom Debates at lironshapira.substack.com/subscribe

Dr. Himanshu Tyagi is a professor of engineering at the Indian Institute of Science and the co-founder of Sentient, an open-source AI platform that raised $85M in funding led by Founders Fund.In this conversation, Himanshu gives me Sentient’s pitch. Then we debate whether open-sourcing frontier AGI development is a good idea, or a reckless way to raise humanity’s P(doom).00:00 Introducing Himanshu Tyagi01:41 Sentient’s Vision05:20 How’d You Raise $85M?11:19 Comparing Sentient to Competitors27:26 Open Source vs. Closed Source AI43:01 What’s Your P(Doom)™48:44 Extinction from Superintelligent AI54:02 AI's Control Over Digital and Physical Assets01:00:26 AI's Influence on Human Movements01:08:46 Recapping the Debate01:13:17 Liron’s AnnouncementsShow NotesHimanshu’s Twitter — https://x.com/hstyagiSentient’s website — https://sentient.foundationCome to the Less Online conference on May 30 - Jun 1, 2025: https://less.onlineHope to see you there!If Anyone Builds It, Everyone Dies by Eliezer Yudkowsky and Nate Soares — https://ifanyonebuildsit.comWatch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk! https://www.youtube.com/@lethal-intelligencePauseAI, the volunteer organization I’m part of: https://pauseai.infoJoin the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at https://doomdebates.com and to https://youtube.com/@DoomDebates Get full access to Doom Debates at lironshapira.substack.com/subscribe

My friend John from the AI X-Risk Network podcast got hired by the Center for AI Safety (CAIS) two weeks ago.Today I suddenly learned he’s been fired.I’m frustrated by this decision, and frustrated with the whole AI x-risk community’s weak messaging.Come to the Less Online conference on May 30 - Jun 1, 2025: https://less.onlineHope to see you there!Watch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk! https://www.youtube.com/@lethal-intelligencePauseAI, the volunteer organization I’m part of: https://pauseai.infoJoin the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at https://doomdebates.com and to https://youtube.com/@DoomDebates Get full access to Doom Debates at lironshapira.substack.com/subscribe

Prof. Gary Marcus is a scientist, bestselling author and entrepreneur, well known as one of the most influential voices in AI. He is Professor Emeritus of Psychology and Neuroscience at NYU.  He was founder and CEO of Geometric Intelligence, a machine learning company acquired by Uber in 2016.Gary co-authored the 2019 book, Rebooting AI: Building Artificial Intelligence We Can Trust, and the 2024 book, Taming Silicon Valley: How We Can Ensure That AI Works for Us. He played an important role in the 2023 Senate Judiciary Subcommittee Hearing on Oversight of AI, testifying with Sam Altman.In this episode, Gary and I have a lively debate about whether P(doom) is approximately 50%, or if it’s less than 1%!00:00 Introducing Gary Marcus02:33 Gary’s AI Skepticism09:08 The Human Brain is a Kluge23:16 The 2023 Senate Judiciary Subcommittee Hearing28:46 What’s Your P(Doom)™44:27 AI Timelines51:03 Is Superintelligence Real?01:00:35 Humanity’s Immune System01:12:46 Potential for Recursive Self-Improvement01:26:12 AI Catastrophe Scenarios01:34:09 Defining AI Agency01:37:43 Gary’s AI Predictions01:44:13 The NYTimes Obituary Test01:51:11 Recap and Final Thoughts01:53:35 Liron’s Outro01:55:34 Eliezer Yudkowsky’s New Book!01:59:49 AI Doom Concept of the DayShow NotesGary’s Substack — https://garymarcus.substack.comGary’s Twitter — https://x.com/garymarcusIf Anyone Builds It, Everyone Dies by Eliezer Yudkowsky and Nate Soares — https://ifanyonebuildsit.comCome to the Less Online conference on May 30 - Jun 1, 2025: https://less.onlineHope to see you there!Watch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk! https://www.youtube.com/@lethal-intelligencePauseAI, the volunteer organization I’m part of: https://pauseai.infoJoin the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at https://doomdebates.com and to https://youtube.com/@DoomDebates Get full access to Doom Debates at lironshapira.substack.com/subscribe

Dr. Mike Israetel, renowned exercise scientist and social media personality, and more recently a low-P(doom) AI futurist, graciously offered to debate me!00:00 Introducing Mike Israetel12:19 What’s Your P(Doom)™30:58 Timelines for Artificial General Intelligence34:49 Superhuman AI Capabilities43:26 AI Reasoning and Creativity47:12 Evil AI Scenario01:08:06 Will the AI Cooperate With Us?01:12:27 AI's Dependence on Human Labor01:18:27 Will AI Keep Us Around to Study Us?01:42:38 AI's Approach to Earth's Resources01:53:22 Global AI Policies and Risks02:03:02 The Quality of Doom Discourse02:09:23 Liron’s OutroShow Notes* Mike’s Instagram — https://www.instagram.com/drmikeisraetel* Mike’s YouTube — https://www.youtube.com/@MikeIsraetelMakingProgressCome to the Less Online conference on May 30 - Jun 1, 2025: https://less.onlineHope to see you there!Watch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk! https://www.youtube.com/@lethal-intelligencePauseAI, the volunteer organization I’m part of: https://pauseai.infoJoin the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at https://doomdebates.com and to https://youtube.com/@DoomDebates Get full access to Doom Debates at lironshapira.substack.com/subscribe

I want to be transparent about how I’ve updated my mainline AI doom scenario in light of safe & useful LLMs. So here’s where I’m at…00:00 Introduction07:59 The Dangerous Threshold to Runaway Superintelligence18:57 Superhuman Goal Optimization = Infinite Time Horizon21:21 Goal-Completeness by Analogy to Turing-Completeness26:53 Intellidynamics29:13 Goal-Optimization Is Convergent31:15 Early AIs Lose Control of Later AIs34:46 The Superhuman Threshold Is Real38:27 Expecting Rapid FOOM40:20 Rocket Alignment49:59 Stability of Values Under Self-Modification53:13 The Way to Heaven Passes Right By Hell57:32 My Mainline Doom Scenario01:17:46 What Values Does The Goal Optimizer Have?Show NotesMy recent episode with Jim Babcock on this same topic of mainline doom scenarios — https://www.youtube.com/watch?v=FaQjEABZ80gThe Rocket Alignment Problem by Eliezer Yudkowsky — https://www.lesswrong.com/posts/Gg9a4y8reWKtLe3Tn/the-rocket-alignment-problemCome to the Less Online conference on May 30 - Jun 1, 2025: https://less.onlineWatch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk! https://www.youtube.com/@lethal-intelligencePauseAI, the volunteer organization I’m part of: https://pauseai.infoJoin the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at https://doomdebates.com and to https://youtube.com/@DoomDebates Get full access to Doom Debates at lironshapira.substack.com/subscribe

What’s the most likely (“mainline”) AI doom scenario? How does the existence of LLMs update the original Yudkowskian version? I invited my friend Jim Babcock to help me answer these questions.Jim is a member of the LessWrong engineering team and its parent organization, Lightcone Infrastructure. I’ve been a longtime fan of his thoughtful takes.This turned out to be a VERY insightful and informative discussion, useful for clarifying my own predictions, and accessible to the show’s audience.00:00 Introducing Jim Babcock01:29 The Evolution of LessWrong Doom Scenarios02:22 LessWrong’s Mission05:49 The Rationalist Community and AI09:37 What’s Your P(Doom)™18:26 What Are Yudkowskians Surprised About?26:48 Moral Philosophy vs. Goal Alignment36:56 Sandboxing and AI Containment42:51 Holding Yudkowskians Accountable58:29 Understanding Next Word Prediction01:00:02 Pre-Training vs Post-Training01:08:06 The Rocket Alignment Problem Analogy01:30:09 FOOM vs. Gradual Disempowerment01:45:19 Recapping the Mainline Doom Scenario01:52:08 Liron’s OutroShow NotesJim’s LessWrong — https://www.lesswrong.com/users/jimrandomhJim’s Twitter — https://x.com/jimrandomhThe Rocket Alignment Problem by Eliezer Yudkowsky — https://www.lesswrong.com/posts/Gg9a4y8reWKtLe3Tn/the-rocket-alignment-problemOptimality is the Tiger and Agents Are Its Teeth — https://www.lesswrong.com/posts/kpPnReyBC54KESiSn/optimality-is-the-tiger-and-agents-are-its-teethDoom Debates episode about the research paper discovering AI's utility function — https://lironshapira.substack.com/p/cais-researchers-discover-ais-preferencesCome to the Less Online conference on May 30 - Jun 1, 2025: https://less.onlineWatch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk! https://www.youtube.com/@lethal-intelligencePauseAI, the volunteer organization I’m part of: https://pauseai.infoJoin the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at https://doomdebates.com and to https://youtube.com/@DoomDebates Get full access to Doom Debates at lironshapira.substack.com/subscribe

Ozzie Gooen is the founder of the Quantified Uncertainty Research Institute (QURI), a nonprofit building software tools for forecasting and policy analysis. I’ve known him through the rationality community since 2008 and we have a lot in common.00:00 Introducing Ozzie02:18 The Rationality Community06:32 What’s Your P(Doom)™08:09 High-Quality Discourse and Social Media14:17 Guesstimate and Squiggle Demos31:57 Prediction Markets and Rationality38:33 Metaforecast Demo41:23 Evaluating Everything with LLMs47:00 Effective Altruism and FTX Scandal56:00 The Repugnant Conclusion Debate01:02:25 AI for Governance and Policy01:12:07 PauseAI Policy Debate01:30:10 Status Quo Bias01:33:31 Decaf Coffee and Caffeine Powder01:34:45 Are You Aspie?01:37:45 Billionaires in Effective Altruism01:48:06 Gradual Disempowerment by AI01:55:36 LessOnline Conference01:57:34 Supporting Ozzie’s WorkShow NotesQuantified Uncertainty Research Institute (QURI) — https://quantifieduncertainty.orgOzzie’s Facebook — https://www.facebook.com/ozzie.gooenOzzie’s Twitter — https://x.com/ozziegooenGuesstimate, a spreadsheet for working with probability ranges — https://www.getguesstimate.comSquiggle, a programming language for building Monte Carlo simulations — https://www.squiggle-language.comMetaforecast, a prediction market aggregator — https://metaforecast.orgOpen Annotate, AI-powered content analysis — https://github.com/quantified-uncertainty/open-annotate/Come to the Less Online conference on May 30 - Jun 1, 2025: https://less.onlineWatch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk! https://www.youtube.com/@lethal-intelligencePauseAI, the volunteer organization I’m part of: https://pauseai.infoJoin the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at https://doomdebates.com and to https://youtube.com/@DoomDebates Get full access to Doom Debates at lironshapira.substack.com/subscribe

David Duvenaud is a professor of Computer Science at the University of Toronto, co-director of the Schwartz Reisman Institute for Technology and Society, former Alignment Evals Team Lead at Anthropic, an award-winning machine learning researcher, and a close collaborator of Dr. Geoffrey Hinton. He recently co-authored Gradual Disempowerment.We dive into David’s impressive career, his high P(Doom), his recent tenure at Anthropic, his views on gradual disempowerment, and the critical need for improved governance and coordination on a global scale.00:00 Introducing David03:03 Joining Anthropic and AI Safety Concerns35:58 David’s Background and Early Influences45:11 AI Safety and Alignment Challenges54:08 What’s Your P(Doom)™01:06:44 Balancing Productivity and Family Life01:10:26 The Hamming Question: Are You Working on the Most Important Problem?01:16:28 The PauseAI Movement01:20:28 Public Discourse on AI Doom01:24:49 Courageous Voices in AI Safety01:43:54 Coordination and Government Role in AI01:47:41 Cowardice in AI Leadership02:00:05 Economic and Existential Doom02:06:12 Liron’s Post-ShowShow NotesDavid’s Twitter — https://x.com/DavidDuvenaudSchwartz Reisman Institute for Technology and Society — https://srinstitute.utoronto.ca/Jürgen Schmidhuber’s Home Page — https://people.idsia.ch/~juergen/Ryan Greenblatt's LessWrong comment about a future scenario where there's a one-time renegotiation of power and heat from superintelligent AI projects causes the oceans to boil: https://www.lesswrong.com/posts/pZhEQieM9otKXhxmd/gradual-disempowerment-systemic-existential-risks-from?commentId=T7KZGGqq2Z4gXZstyWatch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk! https://www.youtube.com/@lethal-intelligencePauseAI, the volunteer organization I’m part of: https://pauseai.infoJoin the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at https://doomdebates.com and to https://youtube.com/@DoomDebates Get full access to Doom Debates at lironshapira.substack.com/subscribe

AI 2027, a bombshell new paper by the AI Futures Project, is a highly plausible scenario of the next few years of AI progress. I like this paper so much that I made a whole episode about it.00:00 Overview of AI 202705:13 2025: Stumbling Agents16:23 2026: Advanced Agents21:49 2027: The Intelligence Explosion29:13 AI's Initial Exploits and OpenBrain's Secrecy30:41 Agent-3 and the Rise of Superhuman Engineering37:05 The Creation and Deception of Agent-544:56 The Race Scenario: Humanity's Downfall48:58 The Slowdown Scenario: A Glimmer of Hope53:49 Final ThoughtsShow NotesThe website: https://ai-2027.comScott Alexander’s blog: https://astralcodexten.comDaniel Kokotajlo’s previous predictions from 2021 about 2026: https://www.lesswrong.com/posts/6Xgy6CAf2jqHhynHL/what-2026-looks-likeWatch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk! https://www.youtube.com/@lethal-intelligencePauseAI, the volunteer organization I’m part of: https://pauseai.infoJoin the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at https://doomdebates.com and to https://youtube.com/@DoomDebates Get full access to Doom Debates at lironshapira.substack.com/subscribe

Dr. Peter Berezin is the Chief Global Strategist and Director of Research at BCA Research, the largest Canadian investment research firm. He’s known for his macroeconomics research reports and his frequent appearances on Bloomberg and CNBC.Notably, Peter is one of the only macroeconomists in the world who’s forecasting AI doom! He recently published a research report estimating a “ more than 50/50 chance AI will wipe out all of humanity by the middle of the century”.00:00 Introducing Peter Berezin01:59 Peter’s Economic Predictions and Track Record05:50 Investment Strategies and Beating the Market17:47 The Future of Human Employment26:40 Existential Risks and the Doomsday Argument34:13 What’s Your P(Doom)™39:18 Probability of non-AI Doom44:19 Solving Population Decline50:53 Constraining AI Development53:40 The Multiverse and Its Implications01:01:11 Are Other Economists Crazy?01:09:19 Mathematical Universe and Multiverse Theories01:19:43 Epistemic vs. Physical Probability01:33:19 Reality Fluid01:39:11 AI and Moral Realism01:54:18 The Simulation Hypothesis and God02:10:06 Liron’s Post-ShowShow NotesPeter’s Twitter: https://x.com/PeterBerezinBCAPeter’s old blog — https://stockcoach.blogspot.comPeter’s 2021 BCA Research Report: “Life, Death and Finance in the Cosmic Multiverse” — https://www.bcaresearch.com/public/content/GIS_SR_2021_12_21.pdfM.C. Escher’s “Circle Limit IV” — https://www.escherinhetpaleis.nl/escher-today/circle-limit-iv-heaven-and-hell/Zvi Mowshowitz’s Blog (Liron’s recommendation for best AI news & analysis) — https://thezvi.substack.comMy Doom Debates episode about why nuclear proliferation is bad — https://www.youtube.com/watch?v=ueB9iRQsvQ8Robin Hanson’s “Mangled Worlds” paper — https://mason.gmu.edu/~rhanson/mangledworlds.htmlUncontrollable by Darren McKee (Liron’s recommended AI x-risk book) — https://www.amazon.com/dp/B0CNNYKVH1Watch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk! https://www.youtube.com/@lethal-intelligencePauseAI, the volunteer organization I’m part of: https://pauseai.infoJoin the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at https://doomdebates.com and to https://youtube.com/@DoomDebates Get full access to Doom Debates at lironshapira.substack.com/subscribe

Nathan Labenz, host of The Cognitive Revolution, joins me for an AI news & social media roundup!00:00 Introducing Nate05:18 What’s Your P(Doom)™23:22 GPT-4o Image Generation40:20 Will Fiverr’s Stock Crash?47:41 AI Unemployment55:11 Entrepreneurship01:00:40 OpenAI Valuation01:09:29 Connor Leahy’s Hair01:13:28 Mass Extinction01:25:30 Is anyone feeling the doom vibes?01:38:20 Rethinking AI Individuality01:40:35 “Softmax” — Emmett Shear's New AI Safety Org01:57:04 Anthropic's Mechanistic Interpretability Paper02:10:11 International Cooperation for AI Safety02:18:43 Final ThoughtsShow NotesNate’s Twitter: https://x.com/labenzNate’s podcast: https://cognitiverevolution.ai and https://youtube.com/@CognitiveRevolutionPodcastNate’s company: https://waymark.com/Watch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk! https://www.youtube.com/@lethal-intelligencePauseAI, the volunteer organization I’m part of: https://pauseai.infoJoin the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at https://doomdebates.com and to https://youtube.com/@DoomDebates Get full access to Doom Debates at lironshapira.substack.com/subscribe

In this special cross-posted episode of Doom Debates, originally posted here on The Human Podcast, we cover a wide range of topics including the definition of “doom”, P(Doom), various existential risks like pandemics and nuclear threats, and the comparison of rogue AI risks versus AI misuse risks.00:00 Introduction01:47 Defining Doom and AI Risks05:53 P(Doom)10:04 Doom Debates’ Mission16:17 Personal Reflections and Life Choices24:57 The Importance of Debate27:07 Personal Reflections on AI Doom30:46 Comparing AI Doom to Other Existential Risks33:42 Strategies to Mitigate AI Risks39:31 The Global AI Race and Game Theory43:06 Philosophical Reflections on a Good Life45:21 Final ThoughtsShow NotesThe Human Podcast with Joe Murray: https://www.youtube.com/@thehumanpodcastofficialWatch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk! https://www.youtube.com/@lethal-intelligencePauseAI, the volunteer organization I’m part of: https://pauseai.infoJoin the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!Don’t miss the other great AI doom show, For Humanity: https://youtube.com/@ForHumanityAIRiskDoom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at https://doomdebates.com and to https://youtube.com/@DoomDebates Get full access to Doom Debates at lironshapira.substack.com/subscribe

Alexander Campbell claims that having superhuman intelligence doesn’t necessarily translate into having vast power, and that Gödel's Incompleteness Theorem ensures AI can’t get too powerful. I strongly disagree.Alex has a Master's of Philosophy in Economics from the University of Oxford and an MBA from the Stanford Graduate School of Business, has worked as a quant trader at Lehman Brothers and Bridgewater Associates, and is the founder of Rose AI, a cloud data platform that leverages generative AI to help visualize data.This debate was recorded in August 2023.00:00 Intro and Alex’s Background05:29 Alex's Views on AI and Technology06:45 Alex’s Non-Doomer Position11:20 Goal-to-Action Mapping15:20 Outcome Pump Thought Experiment21:07 Liron’s Doom Argument29:10 The Dangers of Goal-to-Action Mappers34:39 The China Argument and Existential Risks45:18 Ideological Turing Test48:38 Final ThoughtsShow NotesAlexander Campbell’s Twitter: https://x.com/abcampbellWatch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk! https://www.youtube.com/@lethal-intelligencePauseAI, the volunteer organization I’m part of: https://pauseai.infoJoin the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at https://doomdebates.com and to https://youtube.com/@DoomDebates Get full access to Doom Debates at lironshapira.substack.com/subscribe

Roko Mijic has been an active member of the LessWrong and AI safety community since 2008. He’s best known for “Roko’s Basilisk”, a thought experiment he posted on LessWrong that made Eliezer Yudkowsky freak out, and years later became the topic that helped Elon Musk get interested in Grimes.His view on AI doom is that:* AI alignment is an easy problem* But the chaos and fighting from building superintelligence poses a high near-term existential risk* But humanity’s course without AI has an even higher near-term existential riskWhile my own view is very different, I’m interested to learn more about Roko’s views and nail down our cruxes of disagreement.00:00 Introducing Roko03:33 Realizing that AI is the only thing that matters06:51 Cyc: AI with “common sense”15:15 Is alignment easy?21:19 What’s Your P(Doom)™25:14 Why civilization is doomed anyway37:07 Roko’s AI nightmare scenario47:00 AI risk mitigation52:07 Market Incentives and AI Safety57:13 Are RL and GANs good enough for superalignment?01:00:54 If humans learned to be honest, why can’t AIs?01:10:29 Is our test environment sufficiently similar to production?01:23:56 AGI Timelines01:26:35 Headroom above human intelligence01:42:22 Roko’s Basilisk01:54:01 Post-Debate MonologueShow NotesRoko’s Twitter: https://x.com/RokoMijicExplanation of Roko’s Basilisk on LessWrong: https://www.lesswrong.com/w/rokos-basiliskWatch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk! https://www.youtube.com/@lethal-intelligencePauseAI, the volunteer organization I’m part of: https://pauseai.infoJoin the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at https://doomdebates.com and to https://youtube.com/@DoomDebates Get full access to Doom Debates at lironshapira.substack.com/subscribe

Sir Roger Penrose is a mathematician, mathematical physicist, philosopher of science, and Nobel Laureate in Physics.His famous body of work includes Penrose diagrams, twistor theory, Penrose tilings, and the incredibly bold claim that intelligence and consciousness are uncomputable physical phenomena related to quantum wave function collapse. Dr. Penrose is such a genius that it's just interesting to unpack his worldview, even if it’s totally implausible. How can someone like him be so wrong? What exactly is it that he's wrong about? It's interesting to try to see the world through his eyes, before recoiling from how nonsensical it looks.00:00 Episode Highlights01:29 Introduction to Roger Penrose11:56 Uncomputability16:52 Penrose on Gödel's Incompleteness Theorem19:57 Liron Explains Gödel's Incompleteness Theorem27:05 Why Penrose Gets Gödel Wrong40:53 Scott Aaronson's Gödel CAPTCHA46:28 Penrose's Critique of the Turing Test48:01 Searle's Chinese Room Argument52:07 Penrose's Views on AI and Consciousness57:47 AI's Computational Power vs. Human Intelligence01:21:08 Penrose's Perspective on AI Risk01:22:20 Consciousness = Quantum Wave Function Collapse?01:26:25 Final ThoughtsShow NotesSource video — Feb 22, 2025 Interview with Roger Penrose on “This Is World” — https://www.youtube.com/watch?v=biUfMZ2dts8Scott Aaronson’s “Gödel CAPTCHA” — https://www.scottaaronson.com/writings/captcha.htmlMy recent Scott Aaronson episode — https://www.youtube.com/watch?v=xsGqWeqKjEgMy explanation of what’s wrong with arguing “by definition” — https://www.youtube.com/watch?v=ueam4fq8k8IWatch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk! https://www.youtube.com/@lethal-intelligencePauseAI, the volunteer organization I’m part of: https://pauseai.infoJoin the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at https://doomdebates.com and to https://youtube.com/@DoomDebates Get full access to Doom Debates at lironshapira.substack.com/subscribe

The Center for AI Safety just dropped a fascinating paper — they discovered that today’s AIs like GPT-4 and Claude have preferences! As in, coherent utility functions. We knew this was inevitable, but we didn’t know it was already happening.This episode has two parts:In Part I (48 minutes), I react to David Shapiro’s coverage of the paper and push back on many of his points.In Part II (60 minutes), I explain the paper myself.00:00 Episode Introduction05:25 PART I: REACTING TO DAVID SHAPIRO10:06 Critique of David Shapiro's Analysis19:19 Reproducing the Experiment35:50 David's Definition of Coherence37:14 Does AI have “Temporal Urgency”?40:32 Universal Values and AI Alignment49:13 PART II: EXPLAINING THE PAPER51:37 How The Experiment Works01:11:33 Instrumental Values and Coherence in AI01:13:04 Exchange Rates and AI Biases01:17:10 Temporal Discounting in AI Models01:19:55 Power Seeking, Fitness Maximization, and Corrigibility01:20:20 Utility Control and Bias Mitigation01:21:17 Implicit Association Test01:28:01 Emailing with the Paper’s Authors01:43:23 My TakeawayShow NotesDavid’s source video: https://www.youtube.com/watch?v=XGu6ejtRz-0The research paper: http://emergent-values.aiWatch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk! https://www.youtube.com/@lethal-intelligencePauseAI, the volunteer organization I’m part of: https://pauseai.infoJoin the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at https://doomdebates.com and to https://youtube.com/@DoomDebates Get full access to Doom Debates at lironshapira.substack.com/subscribe

My friend Gil Mark, who leads generative AI products at LinkedIn, thinks competition among superintelligent AIs will lead to a good outcome for humanity. In his view, the alignment problem becomes significantly easier if we build multiple AIs at the same time and let them compete.I completely disagree, but I hope you’ll find this to be a thought-provoking episode that sheds light on why the alignment problem is so hard.00:00 Introduction02:36 Gil & Liron’s Early Doom Days04:58: AIs : Humans :: Humans : Ants08:02 The Convergence of AI Goals15:19 What’s Your P(Doom)™19:23 Multiple AIs and Human Welfare24:42 Gil’s Alignment Claim42:31 Cheaters and Frankensteins55:55 Superintelligent Game Theory01:01:16 Slower Takeoff via Resource Competition01:07:57 Recapping the Disagreement01:15:39 Post-Debate BanterShow NotesGil’s LinkedIn: https://www.linkedin.com/in/gilmark/Gil’s Twitter: https://x.com/gmfromgmWatch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk! https://www.youtube.com/@lethal-intelligencePauseAI, the volunteer organization I’m part of: https://pauseai.infoJoin the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at https://doomdebates.com and to https://youtube.com/@DoomDebates Get full access to Doom Debates at lironshapira.substack.com/subscribe

Why does the simplest AI imaginable, when you ask it to help you push a box around a grid, suddenly want you to die?AI doomers are often misconstrued as having "no evidence" or just "anthropomorphizing". This toy model will help you understand why a drive to eliminate humans is NOT a handwavy anthropomorphic speculation, but rather something we expect by default from any sufficiently powerful search algorithm.We’re not talking about AGI or ASI here — we’re just looking at an AI that does brute-force search over actions in a simple grid world.The slide deck I’m presenting was created by Jaan Tallinn, cofounder of the Future of Life Institute.00:00 Introduction01:24 The Toy Model06:19 Misalignment and Manipulation Drives12:57 Search Capacity and Ontological Insights16:33 Irrelevant Concepts in AI Control20:14 Approaches to Solving AI Control Problems23:38 Final ThoughtsWatch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk! https://www.youtube.com/@lethal-intelligencePauseAI, the volunteer organization I’m part of: https://pauseai.infoJoin the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at https://doomdebates.com and to https://youtube.com/@DoomDebates Get full access to Doom Debates at lironshapira.substack.com/subscribe

Bryan Cantrill, co-founder of Oxide Computer, says in his talk that engineering in the physical world is too complex for any AI to do it better than teams of human engineers. Success isn’t about intelligence; it’s about teamwork, character and resilience.I completely disagree.00:00 Introduction02:03 Bryan’s Take on AI Doom05:55 The Concept of P(Doom)08:36 Engineering Challenges and Human Intelligence15:09 The Role of Regulation and Authoritarianism in AI Control29:44 Engineering Complexity: A Case Study from Oxide Computer40:06 The Value of Team Collaboration46:13 Human Attributes in Engineering49:33 AI's Potential in Engineering58:23 Existential Risks and AI PredictionsBryan’s original talk: Watch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk! https://www.youtube.com/watch?v=9CUFbqh16FgPauseAI, the volunteer organization I’m part of: https://pauseai.infoJoin the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates Get full access to Doom Debates at lironshapira.substack.com/subscribe

Thanks for everyone who participated in the live Q&A on Friday!The topics covered include advice for computer science students, working in AI trustworthiness, what good AI regulation looks like, the implications of the $500B Stargate project, the public's gradual understanding of AI risks, the impact of minor AI disasters, and the philosophy of consciousness.00:00 Advice for Comp Sci Students01:14 The $500B Stargate Project02:36 Eliezer's Recent Podcast03:07 AI Safety and Public Policy04:28 AI Disruption and Politics05:12 DeepSeek and AI Advancements06:54 Human vs. AI Intelligence14:00 Consciousness and AI24:34 Dark Forest Theory and AI35:31 Investing in Yourself42:42 Probability of Aliens Saving Us from AI43:31 Brain-Computer Interfaces and AI Safety46:19 Debating AI Safety and Human Intelligence48:50 Nefarious AI Activities and Satellite Surveillance49:31 Pliny the Prompter Jailbreaking AI50:20 Can’t vs. Won’t Destroy the World51:15 How to Make AI Risk Feel Present54:27 Keeping Doom Arguments On Track57:04 Game Theory and AI Development Race01:01:26 Mental Model of Average Non-Doomer01:04:58 Is Liron a Strict Bayesian and Utilitarian?01:09:48 Can We Rename “Doom Debates”01:12:34 The Role of AI Trustworthiness01:16:48 Minor AI Disasters01:18:07 Most Likely Reason Things Go Well01:21:00 Final ThoughtsShow NotesPrevious post where people submitted questions: https://lironshapira.substack.com/p/2500-subscribers-live-q-and-a-askWatch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk!PauseAI, the volunteer organization I’m part of: https://pauseai.infoJoin the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates Get full access to Doom Debates at lironshapira.substack.com/subscribe

It’s time for AI Twitter Beefs #3:00:00 Introduction01:27 Marc Andreessen vs. Sam Altman09:15 Mark Zuckerberg35:40 Martin Casado47:26 Gary Marcus vs. Miles Brundage Bet58:39 Scott Alexander’s AI Art Turing Test01:11:29 Roon01:16:35 Stephen McAleer01:22:25 Emmett Shear01:37:20 OpenAI’s “Safety”01:44:09 Naval Ravikant vs. Eliezer Yudkowsky01:56:03 Comic Relief01:58:53 Final ThoughtsShow NotesUpcoming Live Q&A: https://lironshapira.substack.com/p/2500-subscribers-live-q-and-a-ask“Make Your Beliefs Pay Rent In Anticipated Experiences” by Eliezer Yudkowsky on LessWrong: https://www.lesswrong.com/posts/a7n8GdKiAZRX86T5A/making-beliefs-pay-rent-in-anticipated-experiencesScott Alexander’s AI Art Turing Test: https://www.astralcodexten.com/p/how-did-you-do-on-the-ai-art-turingWatch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk!PauseAI, the volunteer organization I’m part of: https://pauseai.infoJoin the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates Get full access to Doom Debates at lironshapira.substack.com/subscribe

Effective Altruism has been a controversial topic on social media, so today my guest and I are going to settle the question once and for all: Is it good or bad?Jonas Sota is a Software Engineer at Rippling, BA in Philosophy from UC Berkeley, who’s been observing the Effective Altruism (EA) movement in the San Francisco Bay Area for over a decade… and he’s not a fan.00:00 Introduction01:22 Jonas’s Criticisms of EA03:23 Recoil Exaggeration05:53 Impact of Malaria Nets10:48 Local vs. Global Altruism13:02 Shrimp Welfare25:14 Capitalism vs. Charity33:37 Cultural Sensitivity34:43 The Impact of Direct Cash Transfers37:23 Long-Term Solutions vs. Immediate Aid42:21 Charity Budgets45:47 Prioritizing Local Issues50:55 The EA Community59:34 Debate Recap1:03:57 AnnouncementsShow NotesJonas’s Instagram: @jonas_wandersWill MacAskill’s famous book, Doing Good Better: https://www.effectivealtruism.org/doing-good-betterScott Alexander’s excellent post about the people he met at EA Global: https://slatestarcodex.com/2017/08/16/fear-and-loathing-at-effective-altruism-global-2017/Watch the Lethal Intelligence Guide, the ultimate introduction to AI x-risk!PauseAI, the volunteer organization I’m part of: https://pauseai.infoJoin the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates Get full access to Doom Debates at lironshapira.substack.com/subscribe

Matthew Adelstein, better known as Bentham’s Bulldog on Substack, is a philosophy major at the University of Michigan and an up & coming public intellectual.He’s a rare combination: Effective Altruist, Bayesian, non-reductionist, theist.Our debate covers reductionism, evidence for god, the implications of a fine-tuned universe, moral realism, and AI doom.00:00 Introduction02:56 Matthew’s Research11:29 Animal Welfare16:04 Reductionism vs. Non-Reductionism Debate39:53 The Decline of God in Modern Discourse46:23 Religious Credences50:24 Pascal's Wager and Christianity56:13 Are Miracles Real?01:10:37 Fine-Tuning Argument for God01:28:36 Cellular Automata01:34:25 Anthropic Principle01:51:40 Mathematical Structures and Probability02:09:35 Defining God02:18:20 Moral Realism02:21:40 Orthogonality Thesis02:32:02 Moral Philosophy vs. Science02:45:51 Moral Intuitions02:53:18 AI and Moral Philosophy03:08:50 Debate Recap03:12:20 Show UpdatesShow NotesMatthew’s Substack: https://benthams.substack.comMatthew's Twitter: https://x.com/BenthamsBulldogMatthew's YouTube: https://www.youtube.com/@deliberationunderidealcond5105Lethal Intelligence Guide, the ultimate animated video introduction to AI x-risk – https://www.youtube.com/watch?v=9CUFbqh16FgPauseAI, the volunteer organization I’m part of — https://pauseai.info/Join the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates Get full access to Doom Debates at lironshapira.substack.com/subscribe

Prof. Kenneth Stanley is a former Research Science Manager at OpenAI leading the Open-Endedness Team in 2020-2022. Before that, he was a Professor of Computer Ccience at the University of Central Florida and the head of Core AI Research at Uber. He coauthored Why Greatness Cannot Be Planned: The Myth of the Objective, which argues that as soon as you create an objective, then you ruin your ability to reach it.In this episode, I debate Ken’s claim that superintelligent AI *won’t* be guided by goals, and then we compare our views on AI doom.00:00 Introduction00:45 Ken’s Role at OpenAI01:53 “Open-Endedness” and “Divergence”9:32 Open-Endedness of Evolution21:16 Human Innovation and Tech Trees36:03 Objectives vs. Open Endedness47:14 The Concept of Optimization Processes57:22 What’s Your P(Doom)™01:11:01 Interestingness and the Future01:20:14 Human Intelligence vs. Superintelligence01:37:51 Instrumental Convergence01:55:58 Mitigating AI Risks02:04:02 The Role of Institutional Checks02:13:05 Exploring AI's Curiosity and Human Survival02:20:51 Recapping the Debate02:29:45 Final ThoughtsSHOW NOTESKen’s home page: https://www.kenstanley.net/Ken’s Wikipedia: https://en.wikipedia.org/wiki/Kenneth_StanleyKen’s Twitter: https://x.com/kenneth0stanleyKen’s PicBreeder paper: https://wiki.santafe.edu/images/1/1e/Secretan_ecj11.pdfKen's book, Why Greatness Cannot Be Planned: The Myth of the Objective: https://www.amazon.com/Why-Greatness-Cannot-Planned-Objective/dp/3319155237The Rocket Alignment Problem by Eliezer Yudkowsky: https://intelligence.org/2018/10/03/rocket-alignment/---Lethal Intelligence Guide, the ultimate animated video introduction to AI x-risk – https://www.youtube.com/watch?v=9CUFbqh16FgPauseAI, the volunteer organization I’m part of — https://pauseai.info/Join the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!---Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates Get full access to Doom Debates at lironshapira.substack.com/subscribe

OpenAI just announced o3 and smashed a bunch of benchmarks (ARC-AGI, SWE-bench, FrontierMath)!A new Anthropic and Redwood Research paper says Claude is resisting its developers’ attempts to retrain its values!What’s the upshot — what does it all mean for P(doom)?00:00 Introduction01:45 o3’s architecture and benchmarks06:08 “Scaling is hitting a wall” 🤡13:41 How many new architectural insights before AGI?20:28 Negative update for interpretability31:30 Intellidynamics — ***KEY CONCEPT***33:20 Nuclear control rod analogy36:54 Sam Altman's misguided perspective42:40 Claude resisted retraining from good to evil44:22 What is good corrigibility?52:42 Claude’s incorrigibility doesn’t surprise me55:00 Putting it all in perspective---SHOW NOTESScott Alexander’s analysis of the Claude incorrigibility result: https://www.astralcodexten.com/p/claude-fights-back and https://www.astralcodexten.com/p/why-worry-about-incorrigible-claudeZvi Mowshowitz’s analysis of the Claude incorrigibility result: https://thezvi.wordpress.com/2024/12/24/ais-will-increasingly-fake-alignment/---PauseAI Website: https://pauseai.infoPauseAI Discord: https://discord.gg/2XXWXvErfASay hi to me in the #doom-debates-podcast channel!Watch the Lethal Intelligence video and check out LethalIntelligence.ai! It’s an AWESOME new animated intro to AI risk.Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates Get full access to Doom Debates at lironshapira.substack.com/subscribe

This week Liron was interview by Gaëtan Selle on @the-flares about AI doom.Cross-posted from their channel with permission.Original source: https://www.youtube.com/watch?v=e4Qi-54I9Zw0:00:02 Guest Introduction 0:01:41 Effective Altruism and Transhumanism 0:05:38 Bayesian Epistemology and Extinction Probability 0:09:26 Defining Intelligence and Its Dangers 0:12:33 The Key Argument for AI Apocalypse 0:18:51 AI’s Internal Alignment 0:24:56 What Will AI's Real Goal Be? 0:26:50 The Train of Apocalypse 0:31:05 Among Intellectuals, Who Rejects the AI Apocalypse Arguments? 0:38:32 The Shoggoth Meme 0:41:26 Possible Scenarios Leading to Extinction 0:50:01 The Only Solution: A Pause in AI Research? 0:59:15 The Risk of Violence from AI Risk Fundamentalists 1:01:18 What Will General AI Look Like? 1:05:43 Sci-Fi Works About AI 1:09:21 The Rationale Behind Cryonics 1:12:55 What Does a Positive Future Look Like? 1:15:52 Are We Living in a Simulation? 1:18:11 Many Worlds in Quantum Mechanics Interpretation 1:20:25 Ideal Future Podcast Guest for Doom Debates Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates. Thanks for watching. Get full access to Doom Debates at lironshapira.substack.com/subscribe

Roon is a member of the technical staff at OpenAI. He’s a highly respected voice on tech Twitter, despite being a pseudonymous cartoon avatar account. In late 2021, he invented the terms “shape rotator” and “wordcel” to refer to roughly visual/spatial/mathematical intelligence vs. verbal intelligence. He is simultaneously a serious thinker, a builder, and a shitposter. I'm excited to learn more about Roon, his background, his life, and of course, his views about AI and existential risk.00:00 Introduction02:43 Roon’s Quest and Philosophies22:32 AI Creativity30:42 What’s Your P(Doom)™54:40 AI Alignment57:24 Training vs. Production01:05:37 ASI01:14:35 Goal-Oriented AI and Instrumental Convergence01:22:43 Pausing AI01:25:58 Crux of Disagreement1:27:55 Dogecoin01:29:13 Doom Debates’s MissionShow NotesFollow Roon: https://x.com/tszzlFor Humanity: An AI Safety Podcast with John Sherman — https://www.youtube.com/@ForHumanityPodcastLethal Intelligence Guide, the ultimate animated video introduction to AI x-risk – https://www.youtube.com/watch?v=9CUFbqh16FgPauseAI, the volunteer organization I’m part of — https://pauseai.info/Join the PauseAI Discord — https://discord.gg/2XXWXvErfA — and say hi to me in the #doom-debates-podcast channel!Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates. Thanks for watching. Get full access to Doom Debates at lironshapira.substack.com/subscribe

Today I’m reacting to the recent Scott Aaronson interview on the Win-Win podcast with Liv Boeree and Igor Kurganov.Prof. Aaronson is the Director of the Quantum Information Center at the University of Texas at Austin. He’s best known for his research advancing the frontier of complexity theory, especially quantum complexity theory, and making complex insights from his field accessible to a wider readership via his blog.Scott is one of my biggest intellectual influences. His famous Who Can Name The Bigger Number essay and his long-running blog are among my best memories of coming across high-quality intellectual content online as a teen. His posts and lectures taught me much of what I know about complexity theory.Scott recently completed a two-year stint at OpenAI focusing on the theoretical foundations of AI safety, so I was interested to hear his insider account.Unfortunately, what I heard in the interview confirms my worst fears about the meaning of “safety” at today’s AI companies: that they’re laughably clueless at how to achieve any measure of safety, but instead of doing the adult thing and slowing down their capabilities work, they’re pushing forward recklessly.00:00 Introducing Scott Aaronson02:17 Scott's Recruitment by OpenAI04:18 Scott's Work on AI Safety at OpenAI08:10 Challenges in AI Alignment12:05 Watermarking AI Outputs15:23 The State of AI Safety Research22:13 The Intractability of AI Alignment34:20 Policy Implications and the Call to Pause AI38:18 Out-of-Distribution Generalization45:30 Moral Worth Criterion for Humans51:49 Quantum Mechanics and Human Uniqueness01:00:31 Quantum No-Cloning Theorem01:12:40 Scott Is Almost An Accelerationist?01:18:04 Geoffrey Hinton's Proposal for Analog AI01:36:13 The AI Arms Race and the Need for Regulation01:39:41 Scott Aronson's Thoughts on Sam Altman01:42:58 Scott Rejects the Orthogonality Thesis01:46:35 Final Thoughts01:48:48 Lethal Intelligence Clip01:51:42 OutroShow NotesScott’s Interview on Win-Win with Liv Boeree and Igor Kurganov: https://www.youtube.com/watch?v=ANFnUHcYza0Scott’s Blog: https://scottaaronson.blogPauseAI Website: https://pauseai.infoPauseAI Discord: https://discord.gg/2XXWXvErfAWatch the Lethal Intelligence video and check out LethalIntelligence.ai! It’s an AWESOME new animated intro to AI risk.Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates. Thanks for watching. Get full access to Doom Debates at lironshapira.substack.com/subscribe

Today I’m reacting to a July 2024 interview that Prof. Subbarao Kambhampati did on Machine Learning Street Talk.Rao is a Professor of Computer Science at Arizona State University, and one of the foremost voices making the claim that while LLMs can generate creative ideas, they can’t truly reason.The episode covers a range of topics including planning, creativity, the limits of LLMs, and why Rao thinks LLMs are essentially advanced N-gram models.00:00 Introduction02:54 Essentially N-Gram Models?10:31 The Manhole Cover Question20:54 Reasoning vs. Approximate Retrieval47:03 Explaining Jokes53:21 Caesar Cipher Performance01:10:44 Creativity vs. Reasoning01:33:37 Reasoning By Analogy01:48:49 Synthetic Data01:53:54 The ARC Challenge02:11:47 Correctness vs. Style02:17:55 AIs Becoming More Robust02:20:11 Block Stacking Problems02:48:12 PlanBench and Future Predictions02:58:59 Final ThoughtsShow NotesRao’s interview on Machine Learning Street Talk: https://www.youtube.com/watch?v=y1WnHpedi2ARao’s Twitter: https://x.com/rao2zPauseAI Website: https://pauseai.infoPauseAI Discord: https://discord.gg/2XXWXvErfAWatch the Lethal Intelligence video and check out LethalIntelligence.ai! It’s an AWESOME new animated intro to AI risk.Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates. Thanks for watching. Get full access to Doom Debates at lironshapira.substack.com/subscribe

In this episode of Doom Debates, I discuss AI existential risks with my pseudonymous guest Nethys.Nethy shares his journey into AI risk awareness, influenced heavily by LessWrong and Eliezer Yudkowsky. We explore the vulnerability of society to emerging technologies, the challenges of AI alignment, and why he believes our current approaches are insufficient, ultimately resulting in 99.999% P(Doom).00:00 Nethys Introduction04:47 The Vulnerable World Hypothesis10:01 What’s Your P(Doom)™14:04 Nethys’s Banger YouTube Comment26:53 Living with High P(Doom)31:06 Losing Access to Distant Stars36:51 Defining AGI39:09 The Convergence of AI Models47:32 The Role of “Unlicensed” Thinkers52:07 The PauseAI Movement58:20 Lethal Intelligence Video ClipShow NotesEliezer Yudkowsky’s post on “Death with Dignity”: https://www.lesswrong.com/posts/j9Q8bRmwCgXRYAgcJ/miri-announces-new-death-with-dignity-strategyPauseAI Website: https://pauseai.infoPauseAI Discord: https://discord.gg/2XXWXvErfAWatch the Lethal Intelligence video and check out LethalIntelligence.ai! It’s an AWESOME new animated intro to AI risk.Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates. Thanks for watching. Get full access to Doom Debates at lironshapira.substack.com/subscribe

Fraser Cain is the publisher of Universe Today, co-host of Astronomy Cast, a popular YouTuber about all things space, and guess what… he has a high P(doom)! That’s why he’s joining me on Doom Debates for a very special AI + space crossover episode.00:00 Fraser Cain’s Background and Interests5:03 What’s Your P(Doom)™07:05 Our Vulnerable World15:11 Don’t Look Up22:18 Cosmology and the Search for Alien Life31:33 Stars = Terrorists39:03 The Great Filter and the Fermi Paradox55:12 Grabby Aliens Hypothesis01:19:40 Life Around Red Dwarf Stars?01:22:23 Epistemology of Grabby Aliens01:29:04 Multiverses01:33:51 Quantum Many Worlds vs. Copenhagen Interpretation01:47:25 Simulation Hypothesis01:51:25 Final ThoughtsSHOW NOTESFraser’s YouTube channel: https://www.youtube.com/@frasercainUniverse Today (space and astronomy news): https://www.universetoday.com/Max Tegmark’s book that explains 4 levels of multiverses: https://www.amazon.com/Our-Mathematical-Universe-Ultimate-Reality/dp/0307744256Robin Hanson’s ideas:Grabby Aliens: https://grabbyaliens.comThe Great Filter: https://en.wikipedia.org/wiki/Great_FilterLife in a high-dimensional space: https://www.overcomingbias.com/p/life-in-1kdhtml---Watch the Lethal Intelligence video and check out LethalIntelligence.ai! It’s an AWESOME new animated intro to AI risk.---Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates. Thanks for watching. Get full access to Doom Debates at lironshapira.substack.com/subscribe

Vaden Masrani and Ben Chugg, hosts of the Increments Podcast, are back for a Part II! This time we’re going straight to debating my favorite topic, AI doom.00:00 Introduction02:23 High-Level AI Doom Argument17:06 How Powerful Could Intelligence Be?22:34 “Knowledge Creation”48:33 “Creativity”54:57 Stand-Up Comedy as a Test for AI01:12:53 Vaden & Ben’s Goalposts01:15:00 How to Change Liron’s Mind01:20:02 LLMs are Stochastic Parrots?01:34:06 Tools vs. Agents01:39:51 Instrumental Convergence and AI Goals01:45:51 Intelligence vs. Morality01:53:57 Mainline Futures02:16:50 Lethal Intelligence VideoShow NotesVaden & Ben’s Podcast: https://www.youtube.com/@incrementspodRecommended playlists from their podcast:* The Bayesian vs Popperian Epistemology Series* The Conjectures and Refutations SeriesVaden’s Twitter: https://x.com/vadenmasraniBen’s Twitter: https://x.com/BennyChuggWatch the Lethal Intelligence video and check out LethalIntelligence.ai! It’s an AWESOME new animated intro to AI risk.Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates. Thanks for watching. Get full access to Doom Debates at lironshapira.substack.com/subscribe

Dr. Andrew Critch is the co-founder of the Center for Applied Rationality, a former Research Fellow at the Machine Intelligence Research Institute (MIRI), a Research Scientist at the UC Berkeley Center for Human Compatible AI, and the co-founder of a new startup called Healthcare Agents.Dr. Critch’s P(Doom) is a whopping 85%! But his most likely doom scenario isn’t what you might expect. He thinks humanity will successfully avoid a self-improving superintelligent doom scenario, only to still go extinct via the slower process of “industrial dehumanization”.00:00 Introduction01:43 Dr. Critch’s Perspective on LessWrong Sequences06:45 Bayesian Epistemology15:34 Dr. Critch's Time at MIRI18:33 What’s Your P(Doom)™26:35 Doom Scenarios40:38 AI Timelines43:09 Defining “AGI”48:27 Superintelligence53:04 The Speed Limit of Intelligence01:12:03 The Obedience Problem in AI01:21:22 Artificial Superintelligence and Human Extinction01:24:36 Global AI Race and Geopolitics01:34:28 Future Scenarios and Human Relevance01:48:13 Extinction by Industrial Dehumanization01:58:50 Automated Factories and Human Control02:02:35 Global Coordination Challenges02:27:00 Healthcare Agents02:35:30 Final Thoughts---Show NotesDr. Critch’s LessWrong post explaining his P(Doom) and most likely doom scenarios: https://www.lesswrong.com/posts/Kobbt3nQgv3yn29pr/my-motivation-and-theory-of-change-for-working-in-aiDr. Critch’s Website: https://acritch.com/Dr. Critch’s Twitter: https://twitter.com/AndrewCritchPhD---Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates. Thanks for watching. Get full access to Doom Debates at lironshapira.substack.com/subscribe

It’s time for AI Twitter Beefs #2:00:42 Jack Clark (Anthropic) vs. Holly Elmore (PauseAI US)11:02 Beff Jezos vs. Eliezer Yudkowsky, Carl Feynman18:10 Geoffrey Hinton vs. OpenAI & Meta25:14 Samuel Hammond vs. Liron30:26 Yann LeCun vs. Eliezer Yudkowsky37:13 Roon vs. Eliezer Yudkowsky41:37 Tyler Cowen vs. AI Doomers52:54 David Deutsch vs. LironTwitter people referenced:* Jack Clark: https://x.com/jackclarkSF* Holly Elmore: https://x.com/ilex_ulmus* PauseAI US: https://x.com/PauseAIUS* Geoffrey Hinton: https://x.com/GeoffreyHinton* Samuel Hammond: https://x.com/hamandcheese* Yann LeCun: https://x.com/ylecun* Eliezer Yudkowsky: https://x.com/esyudkowsky* Roon: https://x.com/tszzl* Beff Jezos: https://x.com/basedbeffjezos* Carl Feynman: https://x.com/carl_feynman* Tyler Cowen: https://x.com/tylercowen* David Deutsch: https://x.com/DavidDeutschOxfShow NotesHolly Elmore’s EA forum post about scouts vs. soldiersManifund info & donation page for PauseAI US: https://manifund.org/projects/pauseai-us-2025-through-q2PauseAI.info - join the Discord and find me in the #doom-debates channel!Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates. Thanks for watching. Get full access to Doom Debates at lironshapira.substack.com/subscribe

Vaden Masrani and Ben Chugg, hosts of the Increments Podcast, are joining me to debate Bayesian vs. Popperian epistemology.I’m on the Bayesian side, heavily influenced by the writings of Eliezer Yudkowsky. Vaden and Ben are on the Popperian side, heavily influenced by David Deutsch and the writings of Popper himself.We dive into the theoretical underpinnings of Bayesian reasoning and Solomonoff induction, contrasting them with the Popperian perspective, and explore real-world applications such as predicting elections and economic policy outcomes.The debate highlights key philosophical differences between our two epistemological frameworks, and sets the stage for further discussions on superintelligence and AI doom scenarios in an upcoming Part II.00:00 Introducing Vaden and Ben02:51 Setting the Stage: Epistemology and AI Doom04:50 What’s Your P(Doom)™13:29 Popperian vs. Bayesian Epistemology31:09 Engineering and Hypotheses38:01 Solomonoff Induction45:21 Analogy to Mathematical Proofs48:42 Popperian Reasoning and Explanations54:35 Arguments Against Bayesianism58:33 Against Probability Assignments01:21:49 Popper’s Definition of “Content”01:31:22 Heliocentric Theory Example01:31:34 “Hard to Vary” Explanations01:44:42 Coin Flipping Example01:57:37 Expected Value02:12:14 Prediction Market Calibration02:19:07 Futarchy02:29:14 Prediction Markets as AI Lower Bound02:39:07 A Test for Prediction Markets2:45:54 Closing ThoughtsShow NotesVaden & Ben’s Podcast: https://www.youtube.com/@incrementspodVaden’s Twitter: https://x.com/vadenmasraniBen’s Twitter: https://x.com/BennyChuggBayesian reasoning: https://en.wikipedia.org/wiki/Bayesian_inferenceKarl Popper: https://en.wikipedia.org/wiki/Karl_PopperVaden's blog post on Cox's Theorem and Yudkowsky's claims of "Laws of Rationality": https://vmasrani.github.io/blog/2021/the_credence_assumption/Vaden’s disproof of probabilistic induction (including Solomonoff Induction): https://arxiv.org/abs/2107.00749Vaden’s referenced post about predictions being uncalibrated > 1yr out: https://forum.effectivealtruism.org/posts/hqkyaHLQhzuREcXSX/data-on-forecasting-accuracy-across-different-time-horizons#CalibrationsArticle by Gavin Leech and Misha Yagudin on the reliability of forecasters: https://ifp.org/can-policymakers-trust-forecasters/Sources for claim that superforecasters gave a P(doom) below 1%: https://80000hours.org/2024/09/why-experts-and-forecasters-disagree-about-ai-risk/https://www.astralcodexten.com/p/the-extinction-tournamentVaden’s Slides on Content vs Probability: https://vmasrani.github.io/assets/pdf/popper_good.pdfDoom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates. Thanks for watching. Get full access to Doom Debates at lironshapira.substack.com/subscribe

Our top researchers and industry leaders have been warning us that superintelligent AI may cause human extinction in the next decade.If you haven't been following all the urgent warnings, I'm here to bring you up to speed.* Human-level AI is coming soon* It’s an existential threat to humanity* The situation calls for urgent actionListen to this 15-minute intro to get the lay of the land.Then follow these links to learn more and see how you can help:* The CompendiumA longer written introduction to AI doom by Connor Leahy et al* AGI Ruin — A list of lethalitiesA comprehensive list by Eliezer Yudkowksy of reasons why developing superintelligent AI is unlikely to go well for humanity* AISafety.infoA catalogue of AI doom arguments and responses to objections* PauseAI.infoThe largest volunteer org focused on lobbying world government to pause development of superintelligent AI* PauseAI DiscordChat with PauseAI members, see a list of projects and get involved---Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates. Thanks for watching. Get full access to Doom Debates at lironshapira.substack.com/subscribe

Prof. Lee Cronin is the Regius Chair of Chemistry at the University of Glasgow. His research aims to understand how life might arise from non-living matter. In 2017, he invented “Assembly Theory” as a way to measure the complexity of molecules and gain insight into the earliest evolution of life.Today we’re debating Lee's claims about the limits of AI capabilities, and my claims about the risk of extinction from superintelligent AGI.00:00 Introduction04:20 Assembly Theory05:10 Causation and Complexity10:07 Assembly Theory in Practice12:23 The Concept of Assembly Index16:54 Assembly Theory Beyond Molecules30:13 P(Doom)32:39 The Statement on AI Risk42:18 Agency and Intent47:10 RescueBot’s Intent vs. a Clock’s53:42 The Future of AI and Human Jobs57:34 The Limits of AI Creativity01:04:33 The Complexity of the Human Brain01:19:31 Superintelligence: Fact or Fiction?01:29:35 Final ThoughtsLee’s Wikipedia: https://en.wikipedia.org/wiki/Leroy_CroninLee’s Twitter: https://x.com/leecroninLee’s paper on Assembly Theory: https://arxiv.org/abs/2206.02279Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates. Thanks for watching. Get full access to Doom Debates at lironshapira.substack.com/subscribe

Ben Horowitz, cofounder and General Partner at Andreessen Horowitz (a16z), says nuclear proliferation is good.I was shocked because I thought we all agreed nuclear proliferation is VERY BAD.If Ben and a16z can’t appreciate the existential risks of nuclear weapons proliferation, why would anyone ever take them seriously on the topic of AI regulation?00:00 Introduction00:49 Ben Horowitz on Nuclear Proliferation02:12 Ben Horowitz on Open Source AI05:31 Nuclear Non-Proliferation Treaties10:25 Escalation Spirals15:20 Rogue Actors16:33 Nuclear Accidents17:19 Safety Mechanism Failures20:34 The Role of Human Judgment in Nuclear Safety21:39 The 1983 Soviet Nuclear False Alarm22:50 a16z’s Disingenuousness23:46 Martin Casado and Marc Andreessen24:31 Nuclear Equilibrium26:52 Why I Care28:09 Wrap UpSources of this episode’s video clips:Ben Horowitz’s interview on Upstream with Erik Torenberg: https://www.youtube.com/watch?v=oojc96r3KuoMartin Casado and Marc Andreessen talking about AI on the a16z Podcast: https://www.youtube.com/watch?v=0wIUK0nsyUgRoger Skaer’s TikTok: https://www.tiktok.com/@rogerskaerGeorge W. Bush and John Kerry Presidential Debate (September 30, 2004): https://www.youtube.com/watch?v=WYpP-T0IcyABarack Obama’s Prague Remarks on Nuclear Disarmament: https://www.youtube.com/watch?v=QKSn1SXjj2sJohn Kerry’s Remarks at the 2015 Nuclear Nonproliferation Treaty Review Conference: https://www.youtube.com/watch?v=LsY1AZc1K7wShow notes:Nuclear War, A Scenario by Annie Jacobsen: https://www.amazon.com/Nuclear-War-Scenario-Annie-Jacobsen/dp/0593476093Dr. Strangelove or: How I learned to Stop Worrying and Love the Bomb: https://en.wikipedia.org/wiki/Dr._Strangelove1961 Goldsboro B-52 Crash: https://en.wikipedia.org/wiki/1961_Goldsboro_B-52_crash1983 Soviet Nuclera False Alarm Incident: https://en.wikipedia.org/wiki/1983_Soviet_nuclear_false_alarm_incidentList of military nuclear accidents: https://en.wikipedia.org/wiki/List_of_military_nuclear_accidentsDoom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates. Get full access to Doom Debates at lironshapira.substack.com/subscribe

Today I’m reacting to Arvind Narayanan’s interview with Robert Wright on the Nonzero podcast: https://www.youtube.com/watch?v=MoB_pikM3NYDr. Narayanan is a Professor of Computer Science and the Director of the Center for Information Technology Policy at Princeton. He just published a new book called AI Snake Oil: What Artificial Intelligence Can Do, What It Can’t, and How to Tell the Difference.Arvind claims AI is “normal technology like the internet”, and never sees fit to bring up the impact or urgency of AGI. So I’ll take it upon myself to point out all the questions where someone who takes AGI seriously would give different answers.00:00 Introduction01:49 AI is “Normal Technology”?09:25 Playing Chess vs. Moving Chess Pieces12:23 AI Has To Learn From Its Mistakes?22:24 The Symbol Grounding Problem and AI's Understanding35:56 Human vs AI Intelligence: The Fundamental Difference36:37 The Cognitive Reflection Test41:34 The Role of AI in Cybersecurity43:21 Attack vs. Defense Balance in (Cyber)War54:47 Taking AGI Seriously01:06:15 Final ThoughtsShow NotesThe original Nonzero podcast episode with Arvind Narayanan and Robert Wright: https://www.youtube.com/watch?v=MoB_pikM3NYArvind’s new book, AI Snake Oil: https://www.amazon.com/Snake-Oil-Artificial-Intelligence-Difference-ebook/dp/B0CW1JCKVLArvind’s Substack: https://aisnakeoil.comArvind’s Twitter: https://x.com/random_walkerRobert Wright’s Twitter: https://x.com/robertwrighterRobert Wright’s Nonzero Newsletter: https://nonzero.substack.comRob’s excellent post about symbol grounding (Yes, AIs ‘understand’ things): https://nonzero.substack.com/p/yes-ais-understand-thingsMy previous episode of Doom Debates reacting to Arvind Narayanan on Harry Stebbings’ podcast: https://www.youtube.com/watch?v=lehJlitQvZEDoom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates. Thanks for watching. Get full access to Doom Debates at lironshapira.substack.com/subscribe

Dr. Keith Duggar from Machine Learning Street Talk was the subject of my recent reaction episode about whether GPT o1 can reason. But instead of ignoring or blocking me, Keith was brave enough to come into the lion’s den and debate his points with me… and his P(doom) might shock you!First we debate whether Keith’s distinction between Turing Machines and Discrete Finite Automata is useful for understanding limitations of current LLMs. Then I take Keith on a tour of alignment, orthogonality, instrumental convergence, and other popular stations on the “doom train”, to compare our views on each.Keith was a great sport and I think this episode is a classic!00:00 Introduction00:46 Keith’s Background03:02 Keith’s P(doom)14:09 Are LLMs Turing Machines?19:09 Liron Concedes on a Point!21:18 Do We Need >1MB of Context?27:02 Examples to Illustrate Keith’s Point33:56 Is Terence Tao a Turing Machine?38:03 Factoring Numbers: Human vs. LLM53:24 Training LLMs with Turing-Complete Feedback1:02:22 What Does the Pillar Problem Illustrate?01:05:40 Boundary between LLMs and Brains1:08:52 The 100-Year View1:18:29 Intelligence vs. Optimization Power1:23:13 Is Intelligence Sufficient To Take Over?01:28:56 The Hackable Universe and AI Threats01:31:07 Nuclear Extinction vs. AI Doom1:33:16 Can We Just Build Narrow AI?01:37:43 Orthogonality Thesis and Instrumental Convergence01:40:14 Debating the Orthogonality Thesis02:03:49 The Rocket Alignment Problem02:07:47 Final ThoughtsShow NotesKeith’s show: https://www.youtube.com/@MachineLearningStreetTalkKeith’s Twitter: https://x.com/doctorduggarKeith’s fun brain teaser that LLMs can’t solve yet, about a pillar with four holes: https://youtu.be/nO6sDk6vO0g?si=diGUY7jW4VFsV0TJ&t=3684Eliezer Yudkowsky’s classic post about the “Rocket Alignment Problem”: https://intelligence.org/2018/10/03/rocket-alignment/Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates.📣 You can now chat with me and other listeners in the #doom-debates channel of the PauseAI discord: https://discord.gg/2XXWXvErfA Get full access to Doom Debates at lironshapira.substack.com/subscribe

Sam Kirchner and Remmelt Ellen, leaders of the Stop AI movement, think the only way to effectively protest superintelligent AI development is with civil disobedience.Not only are they staging regular protests in front of AI labs, they’re barricading the entrances and blocking traffic, then allowing themselves to be repeatedly arrested.Is civil disobedience the right strategy to pause or stop AI?00:00 Introducing Stop AI00:38 Arrested at OpenAI Headquarters01:14 Stop AI’s Funding01:26 Blocking Entrances Strategy03:12 Protest Logistics and Arrest08:13 Blocking Traffic12:52 Arrest and Legal Consequences18:31 Commitment to Nonviolence21:17 A Day in the Life of a Protestor21:38 Civil Disobedience25:29 Planning the Next Protest28:09 Stop AI Goals and Strategies34:27 The Ethics and Impact of AI Protests42:20 Call to ActionShow NotesStopAI's next protest is on October 21, 2024 at OpenAI, 575 Florida St, San Francisco, CA 94110.StopAI Website: https://StopAI.infoStopAI Discord: https://discord.gg/gbqGUt7ZN4Disclaimer: I (Liron) am not part of StopAI, but I am a member of PauseAI, which also has a website and Discord you can join.PauseAI Website: https://pauseai.infoPauseAI Discord: https://discord.gg/2XXWXvErfAThere's also a special #doom-debates channel in the PauseAI Discord just for us :)Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates. Thanks for watching. Get full access to Doom Debates at lironshapira.substack.com/subscribe

This episode is a continuation of Q&A #1 Part 1 where I answer YOUR questions!00:00 Introduction01:20 Planning for a good outcome?03:10 Stock Picking Advice08:42 Dumbing It Down for Dr. Phil11:52 Will AI Shorten Attention Spans?12:55 Historical Nerd Life14:41 YouTube vs. Podcast Metrics16:30 Video Games26:04 Creativity30:29 Does AI Doom Explain the Fermi Paradox?36:37 Grabby Aliens37:29 Types of AI Doomers44:44 Early Warning Signs of AI Doom48:34 Do Current AIs Have General Intelligence?51:07 How Liron Uses AI53:41 Is “Doomer” a Good Term?57:11 Liron’s Favorite Books01:05:21 Effective Altruism01:06:36 The Doom Debates Community---Show NotesPauseAI Discord: https://discord.gg/2XXWXvErfARobin Hanson’s Grabby Aliens theory: https://grabbyaliens.comProf. David Kipping’s response to Robin Hanson’s Grabby Aliens: https://www.youtube.com/watch?v=tR1HTNtcYw0My explanation of “AI completeness”, but actually I made a mistake because the term I previously coined is “goal completeness”: https://www.lesswrong.com/posts/iFdnb8FGRF4fquWnc/goal-completeness-is-like-turing-completeness-for-agi^ Goal-Completeness (and the corresponding Shapira-Yudkowsky Thesis) might be my best/only original contribution to AI safety research, albeit a small one. Max Tegmark even retweeted it.a16z's Ben Horowitz claiming nuclear proliferation is good, actually: https://x.com/liron/status/1690087501548126209---Doom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates. Thanks for watching. Get full access to Doom Debates at lironshapira.substack.com/subscribe

Thanks for being one of the first Doom Debates subscribers and sending in your questions! This episode is Part 1; stay tuned for Part 2 coming soon.00:00 Introduction01:17 Is OpenAI a sinking ship?07:25 College Education13:20 Asperger's16:50 Elon Musk: Genius or Clown?22:43 Double Crux32:04 Why Call Doomers a Cult?36:45 How I Prepare Episodes40:29 Dealing with AI Unemployment44:00 AI Safety Research Areas46:09 Fighting a Losing Battle53:03 Liron’s IQ01:00:24 Final ThoughtsExplanation of Double Cruxhttps://www.lesswrong.com/posts/exa5kmvopeRyfJgCy/double-crux-a-strategy-for-mutual-understandingBest Doomer ArgumentsThe LessWrong sequences by Eliezer Yudkowsky: https://ReadTheSequences.comLethalIntelligence.ai — Directory of people who are good at explaining doomRob Miles’ Explainer Videos: https://www.youtube.com/c/robertmilesaiFor Humanity Podcast with John Sherman - https://www.youtube.com/@ForHumanityPodcastPauseAI community — https://PauseAI.info — join the Discord!AISafety.info — Great reference for various argumentsBest Non-Doomer ArgumentsCarl Shulman — https://www.dwarkeshpatel.com/p/carl-shulmanQuintin Pope and Nora Belrose — https://optimists.aiRobin Hanson — https://www.youtube.com/watch?v=dTQb6N3_zu8How I prepared to debate Robin HansonIdeological Turing Test (me taking Robin’s side): https://www.youtube.com/watch?v=iNnoJnuOXFAWalkthrough of my outline of prepared topics: https://www.youtube.com/watch?v=darVPzEhh-IDoom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates. Thanks for watching. Get full access to Doom Debates at lironshapira.substack.com/subscribe

In today’s episode, instead of reacting to a long-form presentation of someone’s position, I’m reporting on the various AI x-risk-related tiffs happening in my part of the world. And by “my part of the world” I mean my Twitter feed.00:00 Introduction01:55 Followup to my MSLT reaction episode03:48 Double Crux04:53 LLMs: Finite State Automata or Turing Machines?16:11 Amjad Masad vs. Helen Toner and Eliezer Yudkowsky17:29 How Will AGI Literally Kill Us?33:53 Roon37:38 Prof. Lee Cronin40:48 Defining AI Creativity43:44 Naval Ravikant46:57 Pascal's Scam54:10 Martin Casado and SB 104701:12:26 Final ThoughtsLinks referenced in the episode:* Eliezer Yudkowsky’s interview on the Logan Bartlett Show. Highly recommended: https://www.youtube.com/watch?v=_8q9bjNHeSo* Double Crux, the core rationalist technique I use when I’m “debating”: https://www.lesswrong.com/posts/exa5kmvopeRyfJgCy/double-crux-a-strategy-for-mutual-understanding* The problem with arguing “by definition”, a classic LessWrong post: https://www.lesswrong.com/posts/cFzC996D7Jjds3vS9/arguing-by-definitionTwitter people referenced:* Amjad Masad: https://x.com/amasad* Eliezer Yudkowsky: https://x.com/esyudkowsky* Helen Toner: https://x.com/hlntnr* Roon: https://x.com/tszzl* Lee Cronin: https://x.com/leecronin* Naval Ravikant: https://x.com/naval* Geoffrey Miller: https://x.com/primalpoly* Martin Casado: https://x.com/martin_casado* Yoshua Bengio: https://x.com/yoshua_bengio* Your boy: https://x.com/lironDoom Debates’ Mission is to raise mainstream awareness of imminent extinction from AGI and build the social infrastructure for high-quality debate.Support the mission by subscribing to my Substack at DoomDebates.com and to youtube.com/@DoomDebates. Thanks for watching. Get full access to Doom Debates at lironshapira.substack.com/subscribe

How smart is OpenAI’s new model, o1? What does “reasoning” ACTUALLY mean? What do computability theory and complexity theory tell us about the limitations of LLMs?Dr. Tim Scarfe and Dr. Keith Duggar, hosts of the popular Machine Learning Street Talk podcast, posted an interesting video discussing these issues… FOR ME TO DISAGREE WITH!!!00:00 Introduction02:14 Computability Theory03:40 Turing Machines07:04 Complexity Theory and AI23:47 Reasoning44:24 o147:00 Finding gold in the Sahara56:20 Self-Supervised Learning and Chain of Thought01:04:01 The Miracle of AI Optimization01:23:57 Collective Intelligence01:25:54 The Argument Against LLMs' Reasoning01:49:29 The Swiss Cheese Metaphor for AI Knowledge02:02:37 Final ThoughtsOriginal source: https://www.youtube.com/watch?v=nO6sDk6vO0gFollow Machine Learning Street Talk: https://www.youtube.com/@MachineLearningStreetTalkZvi Mowshowitz's authoritative GPT-o1 post: https://thezvi.wordpress.com/2024/09/16/gpt-4o1/Join the conversation at DoomDebates.com or youtube.com/@DoomDebates, suggest topics or guests, and help us spread awareness about the urgent risk of AI extinction. Thanks for watching. Get full access to Doom Debates at lironshapira.substack.com/subscribe

Yuval Noah Harari is a historian, philosopher, and bestselling author known for his thought-provoking works on human history, the future, and our evolving relationship with technology. His 2011 book, Sapiens: A Brief History of Humankind, took the world by storm, offering a sweeping overview of human history from the emergence of Homo sapiens to the present day. Harari just published a new book which is largely about AI. It’s called Nexus: A Brief History of Information Networks from the Stone Age to AI. Let’s go through the latest interview he did as part of his book tour to see where he stands on AI extinction risk.00:00 Introduction04:30 Defining AI vs. non-AI20:43 AI and Language Mastery29:37 AI's Potential for Manipulation31:30 Information is Connection?37:48 AI and Job Displacement48:22 Consciousness vs. Intelligence52:02 The Alignment Problem59:33 Final ThoughtsSource podcast: https://www.youtube.com/watch?v=78YN1e8UXdMFollow Yuval Noah Harari: x.com/harari_yuvalFollow Steven Bartlett, host of Diary of a CEO: x.com/StevenBartlettJoin the conversation at DoomDebates.com or youtube.com/@DoomDebates, suggest topics or guests, and help us spread awareness about the urgent risk of AI extinction. Thanks for watching. Get full access to Doom Debates at lironshapira.substack.com/subscribe

It's finally here, the Doom Debates / Dr. Phil crossover episode you've all been asking for 😂 The full episode is called “AI: The Future of Education?"While the main focus was AI in education, I'm glad the show briefly touched on how we're all gonna die. Everything in the show related to AI extinction is clipped here.Join the conversation at DoomDebates.com or youtube.com/@DoomDebates, suggest topics or guests, and help us spread awareness about the urgent risk of AI extinction. Thanks for watching. Get full access to Doom Debates at lironshapira.substack.com/subscribe

Dr. Roman Yampolskiy is the director of the Cyber Security Lab at the University of Louisville. His new book is called AI: Unexplainable, Unpredictable, Uncontrollable.Roman’s P(doom) from AGI is a whopping 99.999%, vastly greater than my P(doom) of 50%. It’s a rare debate when I’m LESS doomy than my opponent!This is a cross-post from the For Humanity podcast hosted by John Sherman. For Humanity is basically a sister show of Doom Debates. Highly recommend subscribing!00:00 John Sherman’s Intro05:21 Diverging Views on AI Safety and Control12:24 The Challenge of Defining Human Values for AI18:04 Risks of Superintelligent AI and Potential Solutions33:41 The Case for Narrow AI45:21 The Concept of Utopia48:33 AI's Utility Function and Human Values55:48 Challenges in AI Safety Research01:05:23 Breeding Program Proposal01:14:05 The Reality of AI Regulation01:18:04 Concluding Thoughts01:23:19 Celebration of LifeThis episode on For Humanity’s channel: https://www.youtube.com/watch?v=KcjLCZcBFoQFor Humanity on YouTube: https://www.youtube.com/@ForHumanityPodcastFor Humanity on X: https://x.com/ForHumanityPodBuy Roman’s new book: https://www.amazon.com/Unexplainable-Unpredictable-Uncontrollable-Artificial-Intelligence/dp/103257626XJoin the conversation at DoomDebates.com or youtube.com/@DoomDebates, suggest topics or guests, and help us spread awareness about the urgent risk of AI extinction. Thanks for watching. Get full access to Doom Debates at lironshapira.substack.com/subscribe

Jobst Landgrebe, co-author of Why Machines Will Never Rule The World: Artificial Intelligence Without Fear, argues that AI is fundamentally limited in achieving human-like intelligence or consciousness due to the complexities of the human brain which are beyond mathematical modeling.Contrary to my view, Jobst has a very low opinion of what machines will be able to achieve in the coming years and decades.He’s also a devout Christian, which makes our clash of perspectives funnier.00:00 Introduction03:12 AI Is Just Pattern Recognition?06:46 Mathematics and the Limits of AI12:56 Complex Systems and Thermodynamics33:40 Transhumanism and Genetic Engineering47:48 Materialism49:35 Transhumanism as Neo-Paganism01:02:38 AI in Warfare01:11:55 Is This Science?01:25:46 ConclusionSource podcast: https://www.youtube.com/watch?v=xrlT1LQSyNUJoin the conversation at DoomDebates.com or youtube.com/@DoomDebates, suggest topics or guests, and help us spread awareness about the urgent risk of AI extinction. Thanks for watching. Get full access to Doom Debates at lironshapira.substack.com/subscribe

Today I’m reacting to the 20VC podcast with Harry Stebbings and Princeton professor Arvind Narayanan.Prof. Narayanan is known for his critical perspective on the misuse and over-hype of artificial intelligence, which he often refers to as “AI snake oil”. Narayanan’s critiques aim to highlight the gap between what AI can realistically achieve, and the often misleading promises made by companies and researchers. I analyze Arvind’s takes on the comparative dangers of AI and nuclear weapons, the limitations of current AI models, and AI’s trajectory toward being a commodity rather than a superintelligent god.00:00 Introduction01:21 Arvind’s Perspective on AI02:07 Debating AI's Compute and Performance03:59 Synthetic Data vs. Real Data05:59 The Role of Compute in AI Advancement07:30 Challenges in AI Predictions26:30 AI in Organizations and Tacit Knowledge33:32 The Future of AI: Exponential Growth or Plateau?36:26 Relevance of Benchmarks39:02 AGI40:59 Historical Predictions46:28 OpenAI vs. Anthropic52:13 Regulating AI56:12 AI as a Weapon01:02:43 Sci-Fi01:07:28 ConclusionOriginal source: https://www.youtube.com/watch?v=8CvjVAyB4O4Follow Arvind Narayanan: x.com/random_walkerFollow Harry Stebbings: x.com/HarryStebbingsJoin the conversation at DoomDebates.com or youtube.com/@DoomDebates, suggest topics or guests, and help us spread awareness about the urgent risk of AI extinction. Thanks for watching. Get full access to Doom Debates at lironshapira.substack.com/subscribe

Today I’m reacting to the Bret Weinstein’s recent appearance on the Diary of a CEO podcast with Steven Bartlett. Bret is an evolutionary biologist known for his outspoken views on social and political issues.Bret gets off to a promising start, saying that AI risk should be “top of mind” and poses “five existential threats”. But his analysis is shallow and ad-hoc, and ends in him dismissing the idea of trying to use regulation as a tool to save our species from a recognized existential threat.I believe we can raise the level of AI doom discourse by calling out these kinds of basic flaws in popular media on the subject.00:00 Introduction02:02 Existential Threats from AI03:32 The Paperclip Problem04:53 Moral Implications of Ending Suffering06:31 Inner vs. Outer Alignment08:41 AI as a Tool for Malicious Actors10:31 Attack vs. Defense in AI18:12 The Event Horizon of AI21:42 Is Language More Prime Than Intelligence?38:38 AI and the Danger of Echo Chambers46:59 AI Regulation51:03 Mechanistic Interpretability56:52 Final ThoughtsOriginal source: youtube.com/watch?v=_cFu-b5lTMUFollow Bret Weinstein: x.com/BretWeinsteinFollow Steven Bartlett: x.com/StevenBartlettJoin the conversation at DoomDebates.com or youtube.com/@DoomDebates, suggest topics or guests, and help us spread awareness about the urgent risk of AI extinction. Thanks for watching. Get full access to Doom Debates at lironshapira.substack.com/subscribe

California's SB 1047 bill, authored by CA State Senator Scott Wiener, is the leading attempt by a US state to regulate catastrophic risks from frontier AI in the wake of President Biden's 2023 AI Executive Order.Today’s debate:Holly Elmore, Executive Director of Pause AI US, representing Pro- SB 1047Greg Tanaka, Palo Alto City Councilmember, representing Anti- SB 1047Key Bill Supporters: Geoffrey Hinton, Yoshua Bengio, Anthropic, PauseAI, and about a 2/3 majority of California voters surveyed.Key Bill Opponents: OpenAI, Google, Meta, Y Combinator, Andreessen HorowitzLinksGreg mentioned that the "Supporters & Opponents" tab on this page lists organizations who registered their support and opposition. The vast majority of organizations listed here registered support against the bill: https://digitaldemocracy.calmatters.org/bills/ca_202320240sb1047Holly mentioned surveys of California voters showing popular support for the bill:1. Center for AI Safety survey shows 77% support: https://drive.google.com/file/d/1wmvstgKo0kozd3tShPagDr1k0uAuzdDM/view2. Future of Life Institute survey shows 59% support: https://futureoflife.org/ai-policy/poll-shows-popularity-of-ca-sb1047/Follow Holly: x.com/ilex_ulmusFollow Greg: x.com/GregTanakaJoin the conversation on DoomDebates.com or youtube.com/@DoomDebates, suggest topics or guests, and help us spread awareness about the urgent risk of extinction. Thanks for watching. Get full access to Doom Debates at lironshapira.substack.com/subscribe

Today I’m reacting to David Shapiro’s response to my previous episode, and also to David’s latest episode with poker champion & effective altruist Igor Kurganov.I challenge David's optimistic stance on superintelligent AI inherently aligning with human values. We touch on factors like instrumental convergence and resource competition. David and I continue to clash over whether we should pause AI development to mitigate potential catastrophic risks. I also respond to David's critiques of AI safety advocates.00:00 Introduction01:08 David's Response and Engagement03:02 The Corrigibility Problem05:38 Nirvana Fallacy10:57 Prophecy and Faith-Based Assertions22:47 AI Coexistence with Humanity35:17 Does Curiosity Make AI Value Humans?38:56 Instrumental Convergence and AI's Goals46:14 The Fermi Paradox and AI's Expansion51:51 The Future of Human and AI Coexistence01:04:56 Concluding ThoughtsJoin the conversation on DoomDebates.com or youtube.com/@DoomDebates, suggest topics or guests, and help us spread awareness about the urgent risk of extinction. Thanks for listening. Get full access to Doom Debates at lironshapira.substack.com/subscribe

Maciej Ceglowski is an entrepreneur and owner of the bookmarking site Pinboard. I’ve been a long-time fan of his sharp, independent-minded blog posts and tweets.In this episode, I react to a great 2016 talk he gave at WebCamp Zagreb titled Superintelligence: The Idea That Eats Smart People. This talk was impressively ahead of its time, as the AI doom debate really only heated up in the last few years.---00:00 Introduction02:13 Historical Analogies and AI Risks05:57 The Premises of AI Doom08:25 Mind Design Space and AI Optimization15:58 Recursive Self-Improvement and AI39:44 Arguments Against Superintelligence45:20 Mental Complexity and AI Motivations47:12 The Argument from Just Look Around You49:27 The Argument from Life Experience50:56 The Argument from Brain Surgery53:57 The Argument from Childhood58:10 The Argument from Robinson Crusoe01:00:17 Inside vs. Outside Arguments01:06:45 Transhuman Voodoo and Religion 2.001:11:24 Simulation Fever01:18:00 AI Cosplay and Ethical Concerns01:28:51 Concluding Thoughts and Call to Action---Follow Maciej: x.com/pinboardFollow Doom Debates:* youtube.com/@DoomDebates* DoomDebates.com* x.com/liron* Search “Doom Debates” in your podcast player Get full access to Doom Debates at lironshapira.substack.com/subscribe

Today I’m reacting to David Shapiro’s latest YouTube video: “Pausing AI is a spectacularly bad idea―Here's why”.In my opinion, every plan that doesn’t evolve pausing frontier AGI capabilities development now is reckless, or at least every plan that doesn’t prepare to pause AGI once we see a “warning shot” that enough people agree is terrifying.We’ll go through David’s argument point by point, to see if there are any good points about why maybe pausing AI might actually be a bad idea.00:00 Introduction01:16 The Pause AI Movement03:03 Eliezer Yudkowsky’s Epistemology12:56 Rationalist Arguments and Evidence24:03 Public Awareness and Legislative Efforts28:38 The Burden of Proof in AI Safety31:02 Arguments Against the AI Pause Movement34:20 Nuclear Proliferation vs. AI34:48 Game Theory and AI36:31 Opportunity Costs of an AI Pause44:18 Axiomatic Alignment47:34 Regulatory Capture and Corporate Interests56:24 The Growing Mainstream Concern for AI SafetyFollow David:* youtube.com/@DaveShap* x.com/DaveShapiFollow Doom Debates:* DoomDebates.com* youtube.com/@DoomDebates* x.com/liron Get full access to Doom Debates at lironshapira.substack.com/subscribe

John Sherman and I go through David Brooks’s appallingly bad article in the New York Times titled “Many People Fear AI. They Shouldn’t.”For Humanity is basically the sister podcast to Doom Debates. We have the same mission to raise awareness of the urgent AI extinction threat, and build grassroots support for pausing new AI capabilities development until it’s safe for humanity.Subscribe to it on YouTube: https://www.youtube.com/@ForHumanityPodcastFollow it on X: https://x.com/ForHumanityPod Get full access to Doom Debates at lironshapira.substack.com/subscribe

Dr. Richard Sutton is a Professor of Computing Science at the University of Alberta known for his pioneering work on reinforcement learning, and his “bitter lesson” that scaling up an AI’s data and compute gives better results than having programmers try to handcraft or explicitly understand how the AI works.Dr. Sutton famously claims that AIs are the “next step in human evolution”, a positive force for progress rather than a catastrophic extinction risk comparable to nuclear weapons.Let’s examine Sutton’s recent interview with Daniel Fagella to understand his crux of disagreement with the AI doom position.---00:00 Introduction03:33 The Worthy vs. Unworthy AI Successor04:52 “Peaceful AI”07:54 “Decentralization”11:57 AI and Human Cooperation14:54 Micromanagement vs. Decentralization24:28 Discovering Our Place in the World33:45 Standard Transhumanism44:29 AI Traits and Environmental Influence46:06 The Importance of Cooperation48:41 The Risk of Superintelligent AI57:25 The Treacherous Turn and AI Safety01:04:28 The Debate on AI Control01:13:50 The Urgency of AI Regulation01:21:41 Final Thoughts and Call to Action---Original interview with Daniel Fagella: youtube.com/watch?v=fRzL5Mt0c8AFollow Richard Sutton: x.com/richardssuttonFollow Daniel Fagella: x.com/danfaggellaFollow Liron: x.com/lironSubscribe to my YouTube channel for full episodes and other bonus content: youtube.com/@DoomDebates Get full access to Doom Debates at lironshapira.substack.com/subscribe

David Pinsof is co-creator of the wildly popular Cards Against Humanity and a social science researcher at UCLA Social Minds Lab. He writes a blog called “Everything Is B******t”.He sees AI doomers as making many different questionable assumptions, and he sees himself as poking holes in those assumptions.I don’t see it that way at all; I think the doom claim is the “default expectation” we ought to have if we understand basic things about intelligence.At any rate, I think you’ll agree that his attempt to poke holes in my doom claims on today’s podcast is super good-natured and interesting.00:00 Introducing David Pinsof04:12 David’s P(doom)05:38 Is intelligence one thing?21:14 Humans vs. other animals37:01 The Evolution of Human Intelligence37:25 Instrumental Convergence39:05 General Intelligence and Physics40:25 The Blind Watchmaker Analogy47:41 Instrumental Convergence01:02:23 Superintelligence and Economic Models01:12:42 Comparative Advantage and AI01:19:53 The Fermi Paradox for Animal Intelligence01:34:57 Closing StatementsFollow David: x.com/DavidPinsofFollow Liron: x.com/lironThanks for watching. You can support Doom Debates by subscribing to the Substack, the YouTube channel (full episodes and bonus content), subscribing in your podcast player, and leaving a review on Apple Podcasts. Get full access to Doom Debates at lironshapira.substack.com/subscribe

Princeton Comp Sci Ph.D. candidate Sayash Kapoor co-authored a blog post last week with his professor Arvind Narayanan called "AI Existential Risk Probabilities Are Too Unreliable To Inform Policy".While some non-doomers embraced the arguments, I see it as contributing nothing to the discourse besides demonstrating a popular failure mode: a simple misunderstanding of the basics of Bayesian epistemology.I break down Sayash's recent episode of Machine Learning Street Talk point-by-point to analyze his claims from the perspective of the one true epistemology: Bayesian epistemology.00:00 Introduction03:40 Bayesian Reasoning04:33 Inductive vs. Deductive Probability05:49 Frequentism vs Bayesianism16:14 Asteroid Impact and AI Risk Comparison28:06 Quantification Bias31:50 The Extinction Prediction Tournament36:14 Pascal's Wager and AI Risk40:50 Scaling Laws and AI Progress45:12 Final ThoughtsMy source material is Sayash's episode of Machine Learning Street Talk: https://www.youtube.com/watch?v=BGvQmHd4QPEI also recommend reading Scott Alexander’s related post: https://www.astralcodexten.com/p/in-continued-defense-of-non-frequentistSayash's blogpost that he was being interviewed about is called "AI existential risk probabilities are too unreliable to inform policy": https://www.aisnakeoil.com/p/ai-existential-risk-probabilitiesFollow Sayash: https://x.com/sayashk Get full access to Doom Debates at lironshapira.substack.com/subscribe

Martin Casado is a General Partner at Andreessen Horowitz (a16z) who has strong views about AI.He claims that AI is basically just a buzzword for statistical models and simulations. As a result of this worldview, he only predicts incremental AI progress that doesn’t pose an existential threat to humanity, and he sees AI regulation as a net negative.I set out to understand his worldview around AI, and pinpoint the crux of disagreement with my own view.Spoiler: I conclude that Martin needs to go beyond analyzing AI as just statistical models and simulations, and analyze it using the more predictive concept of “intelligence” in the sense of hitting tiny high-value targets in exponentially-large search spaces.If Martin appreciated that intelligence is a quantifiable property that algorithms have, and that our existing AIs are getting close to surpassing human-level general intelligence, then hopefully he’d come around to raising his P(doom) and appreciating the urgent extinction risk we face.00:00 Introducing Martin Casado01:42 Martin’s AGI Timeline05:39 Martin’s Analysis of Self-Driving Cars15:30 Heavy-Tail Distributions38:03 Understanding General Intelligence38:29 AI's Progress in Specific Domains43:20 AI’s Understanding of Meaning47:16 Compression and Intelligence48:09 Symbol Grounding53:24 Human Abstractions and AI01:18:18 The Frontier of AI Applications01:23:04 Human vs. AI: Concept Creation and Reasoning01:25:51 The Complexity of the Universe and AI's Limitations01:28:16 AI's Potential in Biology and Simulation01:32:40 The Essence of Intelligence and Creativity in AI01:41:13 AI's Future Capabilities02:00:29 Intelligence vs. Simulation02:14:59 AI Regulation02:23:05 Concluding ThoughtsWatch the original episode of the Cognitive Revolution podcast with Martin and host Nathan Labenz.Follow Martin: @martin_casadoFollow Nate: @labenzFollow Liron: @lironSubscribe to the Doom Debates YouTube Channel to get full episodes plus other bonus content!Search “Doom Debates” to subscribe in your podcast player. Get full access to Doom Debates at lironshapira.substack.com/subscribe

Tilek Mamutov is a Kyrgyzstani software engineer who worked at Google X for 11 years before founding his own international software engineer recruiting company, Outtalent.Since first encountering the AI doom argument at a Center for Applied Rationality bootcamp 10 years ago, he considers it a serious possibility, but he doesn’t currently feel convinced that doom is likely.Let’s explore Tilek’s worldview and pinpoint where he gets off the doom train and why!00:12 Tilek’s Background01:43 Life in Kyrgyzstan04:32 Tilek’s Non-Doomer Position07:12 Debating AI Doom Scenarios13:49 Nuclear Weapons and AI Analogies39:22 Privacy and Empathy in Human-AI Interaction39:43 AI's Potential in Understanding Human Emotions41:14 The Debate on AI's Empathy Capabilities42:23 Quantum Effects and AI's Predictive Models45:33 The Complexity of AI Control and Safety47:10 Optimization Power: AI vs. Human Intelligence48:39 The Risks of AI Self-Replication and Control51:52 Historical Analogies and AI Safety Concerns56:35 The Challenge of Embedding Safety in AI Goals01:02:42 The Future of AI: Control, Optimization, and Risks01:15:54 The Fragility of Security Systems01:16:56 Debating AI Optimization and Catastrophic Risks01:18:34 The Outcome Pump Thought Experiment01:19:46 Human Persuasion vs. AI Control01:21:37 The Crux of Disagreement: Robustness of AI Goals01:28:57 Slow vs. Fast AI Takeoff Scenarios01:38:54 The Importance of AI Alignment01:43:05 ConclusionFollow Tilekx.com/tilekLinksI referenced Paul Christiano’s scenario of gradual AI doom, a slower version that doesn’t require a Yudkowskian “foom”. Worth a read: What Failure Looks LikeI also referenced the concept of “edge instantiation” to explain that if you’re optimizing powerfully for some metric, you don’t get other intuitively nice things as a bonus, you *just* get the exact thing your function is measuring. Get full access to Doom Debates at lironshapira.substack.com/subscribe

Dr. Mike Israetel is a well-known bodybuilder and fitness influencer with over 600,000 Instagram followers, and a surprisingly intelligent commentator on other subjects, including a whole recent episode on the AI alignment problem:Mike brought up many interesting points that were worth responding to, making for an interesting reaction episode. I also appreciate that he’s helping get the urgent topic of AI alignment in front of a mainstream audience.Unfortunately, Mike doesn’t engage with the possibility that AI alignment is an intractable technical problem on a 5-20 year timeframe, which I think is more likely than not. That’s the crux of why he and I disagree, and why I see most of his episode as talking past most other intelligent positions people take on AI alignment. I hope he’ll keep engaging with the topic and rethink his position.00:00 Introduction03:08 AI Risks and Scenarios06:42 Superintelligence Arms Race12:39 The Importance of AI Alignment18:10 Challenges in Defining Human Values26:11 The Outer and Inner Alignment Problems44:00 Transhumanism and AI's Potential45:42 The Next Step In Evolution47:54 AI Alignment and Potential Catastrophes50:48 Scenarios of AI Development54:03 The AI Alignment Problem01:07:39 AI as a Helper System01:08:53 Corporations and AI Development01:10:19 The Risk of Unaligned AI01:27:18 Building a Superintelligent AI01:30:57 ConclusionFollow Mike Israetel:* instagram.com/drmikeisraetel* youtube.com/@MikeIsraetelMakingProgressGet the full Doom Debates experience:* Subscribe to youtube.com/@DoomDebates* Subscribe to this Substack: DoomDebates.com* Search "Doom Debates" to subscribe in your podcast player* Follow me at x.com/liron Get full access to Doom Debates at lironshapira.substack.com/subscribe

What did we learn from my debate with Robin Hanson? Did we successfully isolate the cruxes of disagreement? I actually think we did!In this post-debate analysis, we’ll review what those key cruxes are, and why I still think I’m right and Robin is wrong about them!I’ve taken the time to think much harder about everything Robin said during the debate, so I can give you new & better counterarguments than the ones I was able to make in realtime.Timestamps00:00 Debate Reactions06:08 AI Timelines and Key Metrics08:30 “Optimization Power” vs. “Innovation”11:49 Economic Growth and Diffusion17:56 Predicting Future Trends24:23 Crux of Disagreement with Robin’s Methodology34:59 Conjunction Argument for Low P(Doom)37:26 Headroom Above Human Intelligence41:13 The Role of Culture in Human Intelligence48:01 Goal-Completeness and AI Optimization50:48 Misaligned Foom Scenario59:29 Monitoring AI and the Rule of Law01:04:51 How Robin Sees Alignment01:09:08 Reflecting on the DebateLinksAISafety.info - The fractal of counterarguments to non-doomers’ argumentsFor the full Doom Debates experience:* Subscribe to youtube.com/@DoomDebates* Subscribe to this Substack: DoomDebates.com* Search "Doom Debates" to subscribe in your podcast player* Follow me at x.com/liron Get full access to Doom Debates at lironshapira.substack.com/subscribe

Robin Hanson is a legend in the rationality community and one of my biggest intellectual influences.In 2008, he famously debated Eliezer Yudkowsky about AI doom via a sequence of dueling blog posts known as the great Hanson-Yudkowsky Foom Debate. This debate picks up where Hanson-Yudkowsky left off, revisiting key arguments in the light of recent AI advances.My position is similar to Eliezer's: P(doom) is on the order of 50%. Robin's position is shockingly different: P(doom) is below 1%.00:00 Announcements03:18 Debate Begins05:41 Discussing AI Timelines and Predictions19:54 Economic Growth and AI Impact31:40 Outside Views vs. Inside Views on AI46:22 Predicting Future Economic Growth51:10 Historical Doubling Times and Future Projections54:11 Human Brain Size and Economic Metrics57:20 The Next Era of Innovation01:07:41 AI and Future Predictions01:14:24 The Vulnerable World Hypothesis01:16:27 AI Foom01:28:15 Genetics and Human Brain Evolution01:29:24 The Role of Culture in Human Intelligence01:31:36 Brain Size and Intelligence Debate01:33:44 AI and Goal-Completeness01:35:10 AI Optimization and Economic Impact01:41:50 Feasibility of AI Alignment01:55:21 AI Liability and Regulation02:05:26 Final Thoughts and Wrap-UpRobin's links:Twitter: x.com/RobinHansonHome Page: hanson.gmu.eduRobin’s top related essays:* What Are Reasonable AI Fears?* AIs Will Be Our Mind ChildrenPauseAI links:https://pauseai.info/https://discord.gg/2XXWXvErfACheck out https://youtube.com/@ForHumanityPodcast, the other podcast raising the alarm about AI extinction!For the full Doom Debates experience:* Subscribe to https://youtube.com/@DoomDebates* Subscribe to the Substack: https://DoomDebates.com* Search "Doom Debates" to subscribe in your podcast player* Follow me at https://x.com/liron Get full access to Doom Debates at lironshapira.substack.com/subscribe

This episode is a comprehensive preparation session for my upcoming debate on AI doom with the legendary Robin Hanson.Robin’s P(doom) is I’ve researched past debates, blogs, tweets, and scholarly discussions related to AI doom, and plan to focus our debate on the cruxes of disagreement between Robin’s position and my own Eliezer Yudkowsky-like position.Key topics include the probability of humanity’s extinction due to uncontrollable AGI, alignment strategies, AI capabilities and timelines, the impact of AI advancements, and various predictions made by Hanson.00:00 Introduction03:37 Opening Statement04:29 Value-Extinction Spectrum05:34 Future AI Capabilities08:23 AI Timelines13:23 What can't current AIs do15:48 Architecture/Algorithms vs. Content17:40 Cyc18:55 Is intelligence many different things, or one thing?19:31 Goal-Completeness20:44 AIXI22:10 Convergence in AI systems23:02 Foom26:00 Outside view: Extrapolating robust trends26:18 Salient Events Timeline30:56 Eliezer's claim about meta-levels affecting capability growth rates33:53 My claim - the optimization power model trumps these outside-view trends35:19 Aren't there many other possible outside views?37:03 Is alignment feasible?40:14 What's the warning shot that would make you concerned?41:07 Future Foom evidence?44:59 How else have Robin's views changed in the last decade?Doom Debates catalogues all the different stops where people get off the "doom train", all the different reasons people haven’t (yet) followed the train of logic to the conclusion that humanity is doomed.If you'd like the full Doom Debates experience, it's as easy as doing 4 separate things:1. Join my Substack — DoomDebates.com2. Search "Doom Debates" to subscribe in your podcast player3. Subscribe to YouTube videos — youtube.com/@doomdebates4. Follow me on Twitter — x.com/liron Get full access to Doom Debates at lironshapira.substack.com/subscribe

I’ve been studying Robin Hanson’s catalog of writings and interviews in preparation for our upcoming AI doom debate. Now I’m doing an exercise where I step into Robin’s shoes, and make the strongest possible case for his non-doom position!This exercise is called the Ideological Turing Test, and it’s based on the idea that it’s only productive to argue against someone if you understand what you’re arguing against. Being able to argue *for* a position proves that you understand it.My guest David Xu is a fellow AI doomer, and deep thinker, who volunteered to argue the doomer position against my version of non-doomer “Robin”.00:00 Upcoming Debate with Dr. Robin Hanson01:15 David Xu's Background and Perspective02:23 The Ideological Turing Test02:39 David's AI Doom Claim03:44 AI Takeover vs. Non-AI Descendants05:21 Paperclip Maximizer15:53 Economic Trends and AI Predictions27:18 Recursive Self-Improvement and Foom29:14 Comparing Models of Intelligence34:53 The Foom Scenario36:04 Coordination and Lawlessness in AI37:49 AI's Goal-Directed Behavior and Economic Models40:02 Multipolar Outcomes and AI Coordination40:58 The Orthogonality Thesis and AI Firms43:18 AI's Potential to Exceed Human Control45:03 The Argument for AI Misalignment48:22 Economic Trends vs. AI Catastrophes59:13 The Race for AI Dominance01:04:09 AI Escaping Control01:04:45 AI Liability and Insurance01:06:14 Economic Dynamics and AI Threats01:07:18 The Balance of Offense and Defense in AI01:08:38 AI's Potential to Disrupt National Infrastructure01:10:17 The Multipolar Outcome of AI Development01:11:00 Human Role in AI-Driven Future01:12:19 Debating the Discontinuity in AI Progress01:25:26 Closing Statements and Final Thoughts01:30:34 Reflecting on the Debate and Future DiscussionsFollow David: https://x.com/davidxu90The Ideological Turing Test (ITT) was coined by Bryan Caplan in this classic post: https://www.econlib.org/archives/2011/06/the_ideological.htmlI also did a Twitter version of the ITT here: https://x.com/liron/status/1789688119773872273Doom Debates catalogues all the different stops where people get off the "doom train", all the different reasons people haven’t (yet) followed the train of logic to the conclusion that humanity is doomed.If you'd like the full Doom Debates experience, it's as easy as doing 4 separate things:1. Join my Substack - https://doomdebates.com2. Search "Doom Debates" to subscribe in your podcast player3. Subscribe to YouTube videos - https://youtube.com/@doomdebates4. Follow me on Twitter - https://x.com/liron Get full access to Doom Debates at lironshapira.substack.com/subscribe

Today I'm answering questions from listener Tony Warren.1:16 Biological imperatives in machine learning2:22 Evolutionary pressure vs. AI training4:15 Instrumental convergence and AI goals6:46 Human vs. AI problem domains9:20 AI vs. human actuators18:04 Evolution and intelligence33:23 Maximum intelligence54:55 Computational limits and the futureFollow Tony: https://x.com/Pove_iOS---Doom Debates catalogues all the different stops where people get off the "doom train", all the different reasons people haven’t (yet) followed the train of logic to the conclusion that humanity is doomed.If you'd like the full Doom Debates experience, it's as easy as doing 4 separate things:1. Join my Substack - https://doomdebates.com2. Search "Doom Debates" to subscribe in your podcast player3. Subscribe to YouTube videos - https://youtube.com/@doomdebates4. Follow me on Twitter - https://x.com/liron Get full access to Doom Debates at lironshapira.substack.com/subscribe

My guest Rob thinks superintelligent AI will suffer from analysis paralysis from trying to achieve a 100% probability of killing humanity. Since AI won’t be satisfied with 99.9% of defeating us, it won’t dare to try, and we’ll live!Doom Debates catalogues all the different stops where people get off the “doom train”, all the different reasons people haven’t (yet) followed the train of logic to the conclusion that humanity is doomed.Follow Rob: https://x.com/LoB_BlacksageIf you want to get the full Doom Debates experience, it's as easy as doing 4 separate things:1. Join my Substack - https://doomdebates.com2. Search "Doom Debates" to subscribe in your podcast player3. Subscribe to YouTube videos - https://youtube.com/@DoomDebates4. Follow me on Twitter - https://x.com/liron Get full access to Doom Debates at lironshapira.substack.com/subscribe

Today I’m debating the one & only Professor Steven Pinker!!! Well, I kind of am, in my head. Let me know if you like this format…Dr. Pinker is optimistic that AI doom worries are overblown. But I find his arguments shallow, and I’m disappointed with his overall approach to the AI doom discourse.Here’s the full video of Steven Pinker talking to Michael C. Moynihan on this week’s episode of “Honestly with Bari Weiss”: https://youtube.com/watch?v=mTuH1Ucbif4If you want to get the full Doom Debates experience, it's as easy as doing 4 separate things:1. Join my Substack - https://doomdebates.com2. Search "Doom Debates" to subscribe in your podcast player3. Subscribe to YouTube videos - https://youtube.com/@DoomDebates4. Follow me on Twitter - https://x.com/liron Get full access to Doom Debates at lironshapira.substack.com/subscribe

RJ, a pseudonymous listener, volunteered to debate me.Follow RJ: https://x.com/impershblknightIf you want to get the full Doom Debates experience, it's as easy as doing 4 separate things:1. Join my Substack - https://doomdebates.com2. Search "Doom Debates" to subscribe in your podcast player3. Subscribe to YouTube videos - https://youtube.com/@doomdebates4. Follow me on Twitter - https://x.com/liron Get full access to Doom Debates at lironshapira.substack.com/subscribe

Danny asks:> You've said that an intelligent AI would lead to doom because it would be an excellent goal-to-action mapper.  A great football coach like Andy Reid is a great goal-to-action mapper.  He's on the sidelines, but he knows exactly what actions his team needs to execute to achieve the goal and win the game. > But if he had a team of chimpanzees or elementary schoolers, or just players who did not want to cooperate, then his team would not execute his plans and they would lose.  And even his very talented team of highly motivated players who also want to win the game, sometimes execute his actions badly. Now an intelligent AI that does not control a robot army has very limited ability to perform precise acts in the physical world.  From within the virtual world, an AI would not be able to get animals or plants to carry out specific actions that it wants performed.  I don't see how the AI could get monkeys or dolphins to maintain power plants or build chips.> The AI needs humans to carry out its plans,  but in the real physical world, when dealing with humans, knowing what you want people to do is a small part of the equation.  Won't the AI in practice struggle to get humans to execute its plans in the precise way that it needs?Follow Danny: https://x.com/Danno28_Follow Liron: https://x.com/lironPlease join my email list: DoomDebates.com Get full access to Doom Debates at lironshapira.substack.com/subscribe

Today I’m going to play you my debate with the brilliant hacker and entrepreneur, George Hotz.This took place on an X Space last August.Prior to our debate, George had done a debate with Eliezer Yudkowsky on Dwarkesh Podcast:Follow George: https://x.com/realGeorgeHotzFollow Liron: https://x.com/liron Get full access to Doom Debates at lironshapira.substack.com/subscribe

Chase Mann claims accelerating AGI timelines is the best thing we can do for the survival of the 8 billion people alive today.I claim pausing AI is still the highest-expected-utility decision for everyone.Who do you agree with? Comment on my Substack/X/YouTube and let me know!Follow Chase:https://x.com/ChaseMannFollow Liron:https://x.com/lironLessWrong has some great posts about cryonics: https://www.lesswrong.com/tag/cryonics Get full access to Doom Debates at lironshapira.substack.com/subscribe

It’s a monologue episode!Robin Hanson’s blog: https://OvercomingBias.comRobin Hanson’s famous concept, the Great Filter: https://en.wikipedia.org/wiki/Great_FilterRobin Hanson’s groundbreaking 2021 solution to the Fermi Paradox: https://GrabbyAliens.comRobin Hanson’s conversation with Ronny Fernandez about AI doom from May 2023: My tweet about whether we can hope to control superintelligent AI by judging its explanations and arguments: https://x.com/liron/status/1798135026166698239Zvi Mowshowitz’s blog where he posts EXCELLENT weekly AI roundups: https://thezvi.wordpress.comA takedown of Chris Dixon (Andreessen Horowitz)’s book about the nonsensical “Web3” pitch, which despite being terribly argued, is able to trick a significant number of readers into thinking they just read a good argument: https://www.citationneeded.news/review-read-write-own-by-chris-dixon/(Or maybe you think Chris’s book makes total sense, in which case you can observe that a significant number of smart people somehow don’t get how much sense it makes.)Eliezer Yudkowsky’s famous post about Newcomb’s Problem: https://www.lesswrong.com/posts/6ddcsdA2c2XpNpE5x/newcomb-s-problem-and-regret-of-rationality Get full access to Doom Debates at lironshapira.substack.com/subscribe

Welcome and thanks for listening!* Why is Liron finally starting a podcast?* Who does Liron want to debate?* What’s the debate format?* What are Liron’s credentials?* Is someone “rational” like Liron actually just a religious cult member?Follow Ori on Twitter: https://x.com/ygrowthcoMake sure to subscribe for more episodes! Get full access to Doom Debates at lironshapira.substack.com/subscribe

Kelvin is optimistic that the forces of economic competition will keep AIs sufficiently aligned with humanity by the time they become superintelligent.He thinks AIs and humans will plausibly use interoperable money systems (powered by crypto).So even if our values diverge, the AIs will still uphold a system that respects ownership rights, such that humans may hold onto a nontrivial share of capital with which to pursue human values.I view these kinds of scenarios as wishful thinking with probability much lower than that of the simple undignified scenario I expect, wherein the first uncontrollable AGI correctly realizes what dodos we are in both senses of the word. Get full access to Doom Debates at lironshapira.substack.com/subscribe