The Lunar Society

Ken Rogoff is the former chief economist of the IMF, a professor of Economics at Harvard, and author of the newly released Our Dollar, Your Problem and This Time is Different.On this episode, Ken predicts that, within the next decade, the US will have a debt-induced inflation crisis, but not a Japan-type financial crisis (the latter is much worse, and can make a country poorer for generations).Ken also explains how China is trapped: in order to solve their current problems, they'll keep leaning on financial repression and state-directed investment, which only makes their situation worse.We also discuss the erosion of dollar dominance, why there will be a rebalancing toward foreign equities, how AGI will impact the deficit and interest rate, and much more!Watch on YouTube; listen on Apple Podcasts or Spotify.Sponsors* WorkOS gives your product all the features that enterprise customers need, without derailing your roadmap. Skip months of engineering effort and start selling to enterprises today at workos.com.* Scale is building the infrastructure for smarter, safer AI. In addition to their Data Foundry, they recently released Scale Evaluation, a tool that diagnoses model limitations. Learn how Scale can help you push the frontier at scale.com/dwarkesh.* Gemini Live API lets you have natural, real-time, interactions with Gemini. You can talk to it like you were talking to another person, stream video to show it your surroundings, and share screen to give it context. Try it now by clicking the “Stream” tab on ai.dev.To sponsor a future episode, visit dwarkesh.com/advertise.Timestamps(00:00:00) – China is stagnating(00:25:46) – How the US broke Japan's economy(00:37:06) – America's inflation crisis is coming(01:02:20) – Will AGI solve the US deficit?(01:07:11) – Why interest rates will go up(01:10:55) – US equities will underperform(01:22:24) – The erosion of dollar dominance Get full access to Dwarkesh Podcast at www.dwarkesh.com/subscribe

On this episode, I chat with Victor Shih about all things China. We discuss China's massive local debt crisis, the CCP's views on AI, what happens after Xi, and more.Victor Shih is an expert on the Chinese political system, as well as their banking and fiscal policies, and he has amassed more biographical data on the Chinese elite than anyone else in the world. He teaches at UC San Diego, where he also directs the 21st Century China Center.Watch on YouTube; listen on Apple Podcasts or Spotify.Sponsors* Scale is building the infrastructure for smarter, safer AI. In addition to their Data Foundry, they just released Scale Evaluation, a tool that diagnoses model limitations. Learn how Scale can help you push the frontier at scale.com/dwarkesh.* WorkOS is how top AI companies ship critical enterprise features without burning months of engineering time. If you need features like SSO, audit logs, or user provisioning, head to workos.com.To sponsor a future episode, visit dwarkesh.com/advertise.Timestamps(00:00:00) – Is China more decentralized than the US?(00:03:16) – How the Politburo Standing Committee makes decisions(00:21:07) – Xi's right hand man in charge of AGI(00:35:37) – DeepSeek was trained to track CCP policy(00:45:35) – Local government debt crisis(00:50:00) – BYD, CATL, & financial repression(00:58:12) – How corruption leads to overbuilding(01:10:46) – Probability of Taiwan invasion(01:18:56) – Succession after Xi(01:25:10) – Future growth forecasts Get full access to Dwarkesh Podcast at www.dwarkesh.com/subscribe

New episode with my good friends Sholto Douglas & Trenton Bricken. Sholto focuses on scaling RL and Trenton researches mechanistic interpretability, both at Anthropic.We talk through what's changed in the last year of AI research; the new RL regime and how far it can scale; how to trace a model's thoughts; and how countries, workers, and students should prepare for AGI.See you next year for v3. Here's last year's episode, btw. Enjoy!Watch on YouTube; listen on Apple Podcasts or Spotify.----------SPONSORS* WorkOS ensures that AI companies like OpenAI and Anthropic don't have to spend engineering time building enterprise features like access controls or SSO. It's not that they don't need these features; it's just that WorkOS gives them battle-tested APIs that they can use for auth, provisioning, and more. Start building today at workos.com.* Scale is building the infrastructure for safer, smarter AI. Scale's Data Foundry gives major AI labs access to high-quality data to fuel post-training, while their public leaderboards help assess model capabilities. They also just released Scale Evaluation, a new tool that diagnoses model limitations. If you're an AI researcher or engineer, learn how Scale can help you push the frontier at scale.com/dwarkesh.* Lighthouse is THE fastest immigration solution for the technology industry. They specialize in expert visas like the O-1A and EB-1A, and they've already helped companies like Cursor, Notion, and Replit navigate U.S. immigration. Explore which visa is right for you at lighthousehq.com/ref/Dwarkesh.To sponsor a future episode, visit dwarkesh.com/advertise.----------TIMESTAMPS(00:00:00) – How far can RL scale?(00:16:27) – Is continual learning a key bottleneck?(00:31:59) – Model self-awareness(00:50:32) – Taste and slop(01:00:51) – How soon to fully autonomous agents?(01:15:17) – Neuralese(01:18:55) – Inference compute will bottleneck AGI(01:23:01) – DeepSeek algorithmic improvements(01:37:42) – Why are LLMs ‘baby AGI' but not AlphaZero?(01:45:38) – Mech interp(01:56:15) – How countries should prepare for AGI(02:10:26) – Automating white collar work(02:15:35) – Advice for students Get full access to Dwarkesh Podcast at www.dwarkesh.com/subscribe

Based on my essay about AI firms.Huge thanks to Petr and his team for bringing this to life!Watch on YouTube.Thanks to Google for sponsoring. We used their Veo 2 model to make this entire video—it generated everything from the photorealistic humans to the claymation octopuses. If you're a Gemini Advanced user, you can try Veo 2 now in the Gemini app. Just select Veo 2 in the dropdown, and type your video idea in the prompt bar. Get started today by going to gemini.google.com.To sponsor a future episode, visit dwarkesh.com/advertise. Get full access to Dwarkesh Podcast at www.dwarkesh.com/subscribe

Zuck on:* Llama 4, benchmark gaming* Intelligence explosion, business models for AGI* DeepSeek/China, export controls, & Trump* Orion glasses, AI relationships, and preventing reward-hacking from our tech.Watch on Youtube; listen on Apple Podcasts and Spotify.----------SPONSORS* Scale is building the infrastructure for safer, smarter AI. Scale's Data Foundry gives major AI labs access to high-quality data to fuel post-training, while their public leaderboards help assess model capabilities. They also just released Scale Evaluation, a new tool that diagnoses model limitations. If you're an AI researcher or engineer, learn how Scale can help you push the frontier at scale.com/dwarkesh.* WorkOS Radar protects your product against bots, fraud, and abuse. Radar uses 80+ signals to identify and block common threats and harmful behavior. Join companies like Cursor, Perplexity, and OpenAI that have eliminated costly free-tier abuse by visiting workos.com/radar.* Lambda is THE cloud for AI developers, with over 50,000 NVIDIA GPUs ready to go for startups, enterprises, and hyperscalers. By focusing exclusively on AI, Lambda provides cost-effective compute supported by true experts, including a serverless API serving top open-source models like Llama 4 or DeepSeek V3-0324 without rate limits, and available for a free trial at lambda.ai/dwarkesh.To sponsor a future episode, visit dwarkesh.com/p/advertise.----------TIMESTAMPS(00:00:00) – How Llama 4 compares to other models(00:11:34) – Intelligence explosion(00:26:36) – AI friends, therapists & girlfriends(00:35:10) – DeepSeek & China(00:39:49) – Open source AI(00:54:15) – Monetizing AGI(00:58:32) – The role of a CEO(01:02:04) – Is big tech aligning with Trump?(01:07:10) – 100x productivity Get full access to Dwarkesh Podcast at www.dwarkesh.com/subscribe

800 years before the Black Death, the very same bacteria ravaged Rome, killing 60%+ of the population in many areas.Also, back-to-back volcanic eruptions caused a mini Ice Age, leaving Rome devastated by famine and disease.I chatted with historian Kyle Harper about this and much else:* Rome as a massive slave society* Why humans are more disease-prone than other animals* How agriculture made us physically smaller (Caesar at 5'5" was considered tall)Watch on Youtube; listen on Apple Podcasts or Spotify.----------SPONSORS* WorkOS makes it easy to become enterprise-ready. They have APIs for all the most common enterprise requirements—things like authentication, permissions, and encryption—so you can quickly plug them in and get back to building your core product. If you want to make your product enterprise-ready, join companies like Cursor, Perplexity and OpenAI, and head to workos.com.* Scale's Data Foundry gives major AI labs access to high-quality data to fuel post-training, including advanced reasoning capabilities. If you're an AI researcher or engineer, learn how Scale's Data Foundry and research lab, SEAL, can help you go beyond the current frontier of capabilities at scale.com/dwarkeshTo sponsor a future episode, visit dwarkesh.com/advertise.----------KYLE'S BOOKS* The Fate of Rome: Climate, Disease, and the End of an Empire* Plagues upon the Earth: Disease and the Course of Human History* Slavery in the Late Roman World, AD 275-425----------TIMESTAMPS(00:00:00) - Plague's impact on Rome's collapse(00:06:24) - Rome's little Ice Age(00:11:51) - Why did progress stall in Rome's Golden Age?(00:23:55) - Slavery in Rome(00:36:22) - Was agriculture a mistake?(00:47:42) - Disease's impact on cognitive function(00:59:46) - Plague in India and Central Asia(01:05:16) - The next pandemic(01:16:48) - How Kyle uses LLMs(01:18:51) - De-extinction of lost species Get full access to Dwarkesh Podcast at www.dwarkesh.com/subscribe

Ege Erdil and Tamay Besiroglu have 2045+ timelines, think the whole "alignment" framing is wrong, don't think an intelligence explosion is plausible, but are convinced we'll see explosive economic growth (economy literally doubling every year or two).This discussion offers a totally different scenario than my recent interview with Scott and Daniel.Ege and Tamay are the co-founders of Mechanize, a startup dedicated to fully automating work. Before founding Mechanize, Ege and Tamay worked on AI forecasts at Epoch AI.Watch on Youtube; listen on Apple Podcasts or Spotify.----------Sponsors* WorkOS makes it easy to become enterprise-ready. With simple APIs for essential enterprise features like SSO and SCIM, WorkOS helps companies like Vercel, Plaid, and OpenAI meet the requirements of their biggest customers. To learn more about how they can help you do the same, visit workos.com* Scale's Data Foundry gives major AI labs access to high-quality data to fuel post-training, including advanced reasoning capabilities. If you're an AI researcher or engineer, learn about how Scale's Data Foundry and research lab, SEAL, can help you go beyond the current frontier at scale.com/dwarkesh* Google's Gemini Pro 2.5 is the model we use the most at Dwarkesh Podcast: it helps us generate transcripts, identify interesting clips, and code up new tools. If you want to try it for yourself, it's now available in Preview with higher rate limits! Start building with it today at aistudio.google.com.----------Timestamps(00:00:00) - AGI will take another 3 decades(00:22:27) - Even reasoning models lack animal intelligence (00:45:04) - Intelligence explosion(01:00:57) - Ege & Tamay's story(01:06:24) - Explosive economic growth(01:33:00) - Will there be a separate AI economy?(01:47:08) - Can we predictably influence the future?(02:19:48) - Arms race dynamic(02:29:48) - Is superintelligence a real thing?(02:35:45) - Reasons not to expect explosive growth(02:49:00) - Fully automated firms(02:54:43) - Will central planning work after AGI?(02:58:20) - Career advice Get full access to Dwarkesh Podcast at www.dwarkesh.com/subscribe

Scott and Daniel break down every month from now until the 2027 intelligence explosion.Scott Alexander is author of the highly influential blogs Slate Star Codex and Astral Codex Ten. Daniel Kokotajlo resigned from OpenAI in 2024, rejecting a non-disparagement clause and risking millions in equity to speak out about AI safety.We discuss misaligned hive minds, Xi and Trump waking up, and automated Ilyas researching AI progress.I came in skeptical, but I learned a tremendous amount by bouncing my objections off of them. I highly recommend checking out their new scenario planning document, AI 2027Watch on Youtube; listen on Apple Podcasts or Spotify.----------Sponsors* WorkOS helps today's top AI companies get enterprise-ready. OpenAI, Cursor, Perplexity, Anthropic and hundreds more use WorkOS to quickly integrate features required by enterprise buyers. To learn more about how you can make the leap to enterprise, visit workos.com* Jane Street likes to know what's going on inside the neural nets they use. They just released a black-box challenge for Dwarkesh listeners, and I had blast trying it out. See if you have the skills to crack it at janestreet.com/dwarkesh* Scale's Data Foundry gives major AI labs access to high-quality data to fuel post-training, including advanced reasoning capabilities. If you're an AI researcher or engineer, learn about how Scale's Data Foundry and research lab, SEAL, can help you go beyond the current frontier at scale.com/dwarkeshTo sponsor a future episode, visit dwarkesh.com/advertise.----------Timestamps(00:00:00) - AI 2027(00:06:56) - Forecasting 2025 and 2026(00:14:41) - Why LLMs aren't making discoveries(00:24:33) - Debating intelligence explosion(00:49:45) - Can superintelligence actually transform science?(01:16:54) - Cultural evolution vs superintelligence(01:24:05) - Mid-2027 branch point(01:32:30) - Race with China(01:44:47) - Nationalization vs private anarchy(02:03:22) - Misalignment(02:14:52) - UBI, AI advisors, & human future(02:23:00) - Factory farming for digital minds(02:26:52) - Daniel leaving OpenAI(02:35:15) - Scott's blogging advice Get full access to Dwarkesh Podcast at www.dwarkesh.com/subscribe

I recorded an AMA! I had a blast chatting with my friends Trenton Bricken and Sholto Douglas. We discussed my new book, career advice given AGI, how I pick guests, how I research for the show, and some other nonsense.My book, “The Scaling Era: An Oral History of AI, 2019-2025” is available in digital format now. Preorders for the print version are also open!Watch on YouTube; listen on Apple Podcasts or Spotify.Timestamps(0:00:00) - Book launch announcement(0:04:57) - AI models not making connections across fields(0:10:52) - Career advice given AGI(0:15:20) - Guest selection criteria(0:17:19) - Choosing to pursue the podcast long-term(0:25:12) - Reading habits(0:31:10) - Beard deepdive(0:33:02) - Who is best suited for running an AI lab?(0:35:16) - Preparing for fast AGI timelines(0:40:50) - Growing the podcast Get full access to Dwarkesh Podcast at www.dwarkesh.com/subscribe

Humans have not succeeded because of our raw intelligence.Marooned European explorers regularly starved to death in areas where foragers thrived for 1000s of years.I've always found this cultural evolution deeply mysterious.How do you discover the 10 steps for processing cassava so it won't give you cyanide poisoning simply by trial and error?Has the human brain declined in size over the last 10,000 years because we outsourced cultural evolution to a larger collective brain?The most interesting part of the podcast is Henrich's explanation of how the Catholic Church unintentionally instigated the Industrial Revolution through the dismantling of intensive kinship systems in medieval Europe.Watch on Youtube; listen on Apple Podcasts or Spotify.----------SponsorsScale partners with major AI labs like Meta, Google Deepmind, and OpenAI. Through Scale's Data Foundry, labs get access to high-quality data to fuel post-training, including advanced reasoning capabilities. If you're an AI researcher or engineer, learn about how Scale's Data Foundry and research lab, SEAL, can help you go beyond the current frontier at scale.com/dwarkesh.To sponsor a future episode, visit dwarkesh.com/p/advertise.----------Joseph's booksThe WEIRDest People in the WorldThe Secret of Our Success----------Timestamps(0:00:00) - Humans didn't succeed because of raw IQ(0:09:27) - How cultural evolution works(0:20:48) - Why is human brain size declining?(0:32:00) - Will AGI have superhuman cultural learning?(0:42:34) - Why Industrial Revolution happened in Europe(0:55:30) - Why China, Rome, India got left behind(1:21:09) - Loss of cultural variance in modern world(1:31:20) - Is individual genius real?(1:43:49) - IQ and collective brains Get full access to Dwarkesh Podcast at www.dwarkesh.com/subscribe

I'm so excited with how this visualization of Notes on China turned out. Petr, thank you for such beautiful watercolor artwork. More to come!Watch on YouTube.Timestamps(0:00:00) - Intro(0:00:32) - Scale(0:05:50) - Vibes(0:11:14) - Youngsters(0:14:27) - Tech & AI(0:15:47) - Hearts & Minds(0:17:07) - On Travel Get full access to Dwarkesh Podcast at www.dwarkesh.com/subscribe

Satya Nadella on:- Why he doesn't believe in AGI but does believe in 10% economic growth,- Microsoft's new topological qubit breakthrough and gaming world models,- Whether Office commoditizes LLMs or the other way around,Watch on Youtube; listen on Apple Podcasts or Spotify.SponsorsScale partners with major AI labs like Meta, Google Deepmind, and OpenAI. Through Scale's Data Foundry, labs get access to high-quality data to fuel post-training, including advanced reasoning capabilities. If you're an AI researcher or engineer, learn about how Scale's Data Foundry and research lab, SEAL, can help you go beyond the current frontier at scale.com/dwarkeshLinear's project management tools have become the default choice for product teams at companies like Ramp, CashApp, OpenAI, and Scale. These teams use Linear so they can stay close to their products and move fast. If you're curious why so many companies are making the switch, visit linear.app/dwarkeshTo sponsor a future episode, visit dwarkeshpatel.com/p/advertise.Timestamps(0:00:00) - Intro(0:05:04) - AI won't be winner-take-all(0:15:18) - World economy growing by 10%(0:21:39) - Decreasing price of intelligence(0:30:19) - Quantum breakthrough(0:42:51) - How Muse will change gaming(0:49:51) - Legal barriers to AI(0:55:46) - Getting AGI safety right(1:04:59) - 34 years at Microsoft(1:10:46) - Does Satya Nadella believe in AGI? Get full access to Dwarkesh Podcast at www.dwarkeshpatel.com/subscribe

This week I welcome on the show two of the most important technologists ever, in any field.Jeff Dean is Google's Chief Scientist, and through 25 years at the company, has worked on basically the most transformative systems in modern computing: from MapReduce, BigTable, Tensorflow, AlphaChip, to Gemini.Noam Shazeer invented or co-invented all the main architectures and techniques that are used for modern LLMs: from the Transformer itself, to Mixture of Experts, to Mesh Tensorflow, to Gemini and many other things.We talk about their 25 years at Google, going from PageRank to MapReduce to the Transformer to MoEs to AlphaChip – and maybe soon to ASI.My favorite part was Jeff's vision for Pathways, Google's grand plan for a mutually-reinforcing loop of hardware and algorithmic design and for going past autoregression. That culminates in us imagining *all* of Google-the-company, going through one huge MoE model.And Noam just bites every bullet: 100x world GDP soon; let's get a million automated researchers running in the Google datacenter; living to see the year 3000.SponsorsScale partners with major AI labs like Meta, Google Deepmind, and OpenAI. Through Scale's Data Foundry, labs get access to high-quality data to fuel post-training, including advanced reasoning capabilities. If you're an AI researcher or engineer, learn about how Scale's Data Foundry and research lab, SEAL, can help you go beyond the current frontier at scale.com/dwarkesh.Curious how Jane Street teaches their new traders? They use Figgie, a rapid-fire card game that simulates the most exciting parts of markets and trading. It's become so popular that Jane Street hosts an inter-office Figgie championship every year. Download from the app store or play on your desktop at figgie.com.Meter wants to radically improve the digital world we take for granted. They're developing a foundation model that automates network management end-to-end. To do this, they just announced a long-term partnership with Microsoft for tens of thousands of GPUs, and they're recruiting a world class AI research team. To learn more, go to meter.com/dwarkesh.Advertisers:To sponsor a future episode, visit: dwarkeshpatel.com/p/advertise.Timestamps00:00:00 - Intro00:02:44 - Joining Google in 199900:05:36 - Future of Moore's Law00:10:21 - Future TPUs00:13:13 - Jeff's undergrad thesis: parallel backprop00:15:10 - LLMs in 200700:23:07 - “Holy s**t” moments00:29:46 - AI fulfills Google's original mission00:34:19 - Doing Search in-context00:38:32 - The internal coding model00:39:49 - What will 2027 models do?00:46:00 - A new architecture every day?00:49:21 - Automated chip design and intelligence explosion00:57:31 - Future of inference scaling01:03:56 - Already doing multi-datacenter runs01:22:33 - Debugging at scale01:26:05 - Fast takeoff and superalignment01:34:40 - A million evil Jeff Deans01:38:16 - Fun times at Google01:41:50 - World compute demand in 203001:48:21 - Getting back to modularity01:59:13 - Keeping a giga-MoE in-memory02:04:09 - All of Google in one model02:12:43 - What's missing from distillation02:18:03 - Open research, pros and cons02:24:54 - Going the distance Get full access to Dwarkesh Podcast at www.dwarkeshpatel.com/subscribe

Third and final episode in the Paine trilogy!Chinese history is full of warlords constantly challenging the capital. How could Mao not only stay in power for decades, but not even face any insurgency?And how did Mao go from military genius to peacetime disaster - the patriotic hero who inflicted history's worst human catastrophe on China? How can someone shrewd enough to win a civil war outnumbered 5 to 1 decide "let's have peasants make iron in their backyards" and "let's kill all the birds"?In her lecture and our Q&A, we cover the first nationwide famine in Chinese history; Mao's lasting influence on other insurgents; broken promises to minorities and peasantry; and what Taiwan means.Thanks so much to @Substack for running this in-person event!Note that Sarah is doing an AMA over the next couple days on Youtube; see the pinned comment.Watch on YouTube. Listen on Apple Podcasts, Spotify, or any other podcast platform.SponsorToday's episode is brought to you by Scale AI. Scale partners with the U.S. government to fuel America's AI advantage through their data foundry. Scale recently introduced Defense Llama, Scale's latest solution available for military personnel. With Defense Llama, military personnel can harness the power of AI to plan military or intelligence operations and understand adversary vulnerabilities.If you're interested in learning more on how Scale powers frontier AI capabilities, go to https://scale.com/dwarkesh. Get full access to Dwarkesh Podcast at www.dwarkeshpatel.com/subscribe

This is the second episode in the trilogy of a lectures by Professor Sarah Paine of the Naval War College.In this second episode, Prof Paine dissects the ideas and economics behind Japanese imperialism before and during WWII. We get into the oil shortage which caused the war; the unique culture of honor and death; the surprisingly chaotic chain of command. This is followed by a Q&A with me.Huge thanks to Substack for hosting this event!Watch on YouTube. Listen on Apple Podcasts, Spotify, or any other podcast platform.SponsorToday's episode is brought to you by Scale AI. Scale partners with the U.S. government to fuel America's AI advantage through their data foundry. Scale recently introduced Defense Llama, Scale's latest solution available for military personnel. With Defense Llama, military personnel can harness the power of AI to plan military or intelligence operations and understand adversary vulnerabilities.If you're interested in learning more on how Scale powers frontier AI capabilities, go to scale.com/dwarkesh.Buy Sarah's Books!I highly, highly recommend both "The Wars for Asia, 1911–1949" and "The Japanese Empire: Grand Strategy from the Meiji Restoration to the Pacific War".Timestamps(0:00:00) - Lecture begins(0:06:58) - The code of the samurai(0:10:45) - Buddhism, Shinto, Confucianism(0:16:52) - Bushido as bad strategy(0:23:34) - Military theorists(0:33:42) - Strategic sins of omission(0:38:10) - Crippled logistics(0:40:58) - the Kwantung Army(0:43:31) - Inter-service communication(0:51:15) - Shattering Japanese morale(0:57:35) - Q&A begins(01:05:02) - Unusual brutality of WWII(01:11:30) - Embargo caused the war(01:16:48) - The liberation of China(01:22:02) - Could US have prevented war?(01:25:30) - Counterfactuals in history(01:27:46) - Japanese optimism(01:30:46) - Tech change and social change(01:38:22) - Hamming questions(01:44:31) - Do sanctions work?(01:50:07) - Backloaded mass death(01:54:09) - demilitarizing Japan(01:57:30) - Post-war alliances(02:03:46) - Inter-service rivalry Get full access to Dwarkesh Podcast at www.dwarkeshpatel.com/subscribe

I'm thrilled to launch a new trilogy of double episodes: a lecture series by Professor Sarah Paine of the Naval War College, each followed by a deep Q&A.In this first episode, Prof Paine talks about key decisions by Khrushchev, Mao, Nehru, Bhutto, & Lyndon Johnson that shaped the whole dynamic of South Asia today. This is followed by a Q&A.Come for the spy bases, shoestring nukes, and insight about how great power politics impacts every region.Huge thanks to Substack for hosting this!Watch on YouTube. Listen on Apple Podcasts, Spotify, or any other podcast platform.SponsorsToday's episode is brought to you by Scale AI. Scale partners with the U.S. government to fuel America's AI advantage through their data foundry. The Air Force, Army, Defense Innovation Unit, and Chief Digital and Artificial Intelligence Office all trust Scale to equip their teams with AI-ready data and the technology to build powerful applications.Scale recently introduced Defense Llama, Scale's latest solution available for military personnel. With Defense Llama, military personnel can harness the power of AI to plan military or intelligence operations and understand adversary vulnerabilities.If you're interested in learning more on how Scale powers frontier AI capabilities, go to scale.com/dwarkesh.Timestamps(00:00) - Intro(02:11) - Mao at war, 1949-51(05:40) - Pactomania and Sino-Soviet conflicts(14:42) - The Sino-Indian War(20:00) - Soviet peace in India-Pakistan(22:00) - US Aid and Alliances(26:14) - The difference with WWII(30:09) - The geopolitical map in 1904(35:10) - The US alienates Indira Gandhi(42:58) - Instruments of US power(53:41) - Carrier battle groups(1:02:41) - Q&A begins(1:04:31) - The appeal of the USSR(1:09:36) - The last communist premier(1:15:42) - India and China's lost opportunity(1:58:04) - Bismark's cunning(2:03:05) - Training US officers(2:07:03) - Cruelty in Russian history Get full access to Dwarkesh Podcast at www.dwarkeshpatel.com/subscribe

I interviewed Tyler Cowen at the Progress Conference 2024. As always, I had a blast. This is my fourth interview with him – and yet I'm always hearing new stuff.We talked about why he thinks AI won't drive explosive economic growth, the real bottlenecks on world progress, him now writing for AIs instead of humans, and the difficult relationship between being cultured and fostering growth – among many other things in the full episode.Thanks to the Roots of Progress Institute (with special thanks to Jason Crawford and Heike Larson) for such a wonderful conference, and to FreeThink for the videography.Watch on YouTube. Listen on Apple Podcasts, Spotify, or any other podcast platform. Read the full transcript here.SponsorsI'm grateful to Tyler for volunteering to say a few words about Jane Street. It's the first time that a guest has participated in the sponsorship. I hope you can see why Tyler and I think so highly of Jane Street. To learn more about their open rules, go to janestreet.com/dwarkersh.Timestamps(00:00:00) Economic Growth and AI(00:14:57) Founder Mode and increasing variance(00:29:31) Effective Altruism and Progress Studies(00:33:05) What AI changes for Tyler(00:44:57) The slow diffusion of innovation(00:49:53) Stalin's library(00:52:19) DC vs SF vs EU Get full access to Dwarkesh Podcast at www.dwarkeshpatel.com/subscribe

In order to apply, or to refer someone else, please fill out this short form! For more information, visit dwarkeshpatel.com/roles.Our mission is to publish the highest quality intellectual content in the world, to find the David Reichs and Sarah Paines of every field, and to produce the best contemporaneous coverage of the emergence of AGI.I need the help of two key partners in order to achieve this mission.* General Manager: A killer operator who will run and lead our business.* Editor-in-Chief: A polymath and a shrewd promoter with amazing taste.If you refer somebody I end up hiring, I'll pay you $20,000. If you know someone exceptional who would be a great fit, please share this with them!FAQWill keep updated.Q: What happened to the COO position you posted about a few months ago?Tons of super talented people applied. The role wasn't filled because I had incorrectly combined two distinct positions into one. This was not due to any shortcomings in the applicant pool, and I'm very grateful to everyone who applied!Q: What's the timeline?A: I'll keep applications open until January 20th and will start reviewing and scheduling interviews from January 8th. Early applications will be reviewed first.Q: I applied for the COO role previously - can I apply again?A: Yes! While many incredibly talented people applied for the COO role, I've now split the role into two more focused positions. If you applied before, you're welcome to apply again, though please note that I have already reviewed and considered your previous application.Q: What's the compensation?A: Compensation will be competitive with major tech companies.Q: What about location?A: I'm flexible on location but have a preference for hybrid work with some time in my SF office when in-person collaboration is valuable. Full remote is possible for exceptional candidates. Get full access to Dwarkesh Podcast at www.dwarkeshpatel.com/subscribe

Adam Brown is a founder and lead of BlueShift with is cracking maths and reasoning at Google DeepMind and a theoretical physicist at Stanford.We discuss: destroying the light cone with vacuum decay, holographic principle, mining black holes, & what it would take to train LLMs that can make Einstein level conceptual breakthroughs.Stupefying, entertaining, & terrifying.Enjoy!Watch on YouTube, read the transcript, listen on Apple Podcasts, Spotify, or your favorite platform.Sponsors- Deepmind, Meta, Anthropic, and OpenAI, partner with Scale for high quality data to fuel post-training Publicly available data is running out - to keep developing smarter and smarter models, labs will need to rely on Scale's data foundry, which combines subject matter experts with AI models to generate fresh data and break through the data wall. Learn more at scale.ai/dwarkesh.- Jane Street is looking to hire their next generation of leaders. Their deep learning team is looking for ML researchers, FPGA programmers, and CUDA programmers. Summer internships are open for just a few more weeks. If you want to stand out, take a crack at their new Kaggle competition. To learn more, go janestreet.com/dwarkersh.- This episode is brought to you by Stripe, financial infrastructure for the internet. Millions of companies from Anthropic to Amazon use Stripe to accept payments, automate financial processes and grow their revenue.Timestamps(00:00:00) - Changing the laws of physics(00:26:05) - Why is our universe the way it is(00:37:30) - Making Einstein level AGI(01:00:31) - Physics stagnation and particle colliders(01:11:10) - Hitchhiking(01:29:00) - Nagasaki(01:36:19) - Adam's career(01:43:25) - Mining black holes(01:59:42) - The holographic principle(02:23:25) - Philosophy of infinities(02:31:42) - Engineering constraints for future civilizations Get full access to Dwarkesh Podcast at www.dwarkeshpatel.com/subscribe

Gwern is a pseudonymous researcher and writer. He was one of the first people to see LLM scaling coming. If you've read his blog, you know he's one of the most interesting polymathic thinkers alive.In order to protect Gwern's anonymity, I proposed interviewing him in person, and having my friend Chris Painter voice over his words after. This amused him enough that he agreed.After the episode, I convinced Gwern to create a donation page where people can help sustain what he's up to. Please go here to contribute.Read the full transcript here.Sponsors:* Jane Street is looking to hire their next generation of leaders. Their deep learning team is looking for ML researchers, FPGA programmers, and CUDA programmers. Summer internships are open - if you want to stand out, take a crack at their new Kaggle competition. To learn more, go here: https://jane-st.co/dwarkesh* Turing provides complete post-training services for leading AI labs like OpenAI, Anthropic, Meta, and Gemini. They specialize in model evaluation, SFT, RLHF, and DPO to enhance models' reasoning, coding, and multimodal capabilities. Learn more at turing.com/dwarkesh.* This episode is brought to you by Stripe, financial infrastructure for the internet. Millions of companies from Anthropic to Amazon use Stripe to accept payments, automate financial processes and grow their revenue.If you're interested in advertising on the podcast, check out this page.Timestamps00:00:00 - Anonymity00:01:09 - Automating Steve Jobs00:04:38 - Isaac Newton's theory of progress00:06:36 - Grand theory of intelligence00:10:39 - Seeing scaling early00:21:04 - AGI Timelines00:22:54 - What to do in remaining 3 years until AGI00:26:29 - Influencing the shoggoth with writing00:30:50 - Human vs artificial intelligence00:33:52 - Rabbit holes00:38:48 - Hearing impairment00:43:00 - Wikipedia editing00:47:43 - Gwern.net00:50:20 - Counterfactual careers00:54:30 - Borges & literature01:01:32 - Gwern's intelligence and process01:11:03 - A day in the life of Gwern01:19:16 - Gwern's finances01:25:05 - The diversity of AI minds01:27:24 - GLP drugs and obesity01:31:08 - Drug experimentation01:33:40 - Parasocial relationships01:35:23 - Open rabbit holes Get full access to Dwarkesh Podcast at www.dwarkeshpatel.com/subscribe

A bonanza on the semiconductor industry and hardware scaling to AGI by the end of the decade.Dylan Patel runs Semianalysis, the leading publication and research firm on AI hardware. Jon Y runs Asianometry, the world's best YouTube channel on semiconductors and business history.* What Xi would do if he became scaling pilled* $ 1T+ in datacenter buildout by end of decadeWatch on YouTube. Listen on Apple Podcasts, Spotify, or any other podcast platform. Read the full transcript here. Follow me on Twitter for updates on future episodes.Sponsors:* Jane Street is looking to hire their next generation of leaders. Their deep learning team is looking for FPGA programmers, CUDA programmers, and ML researchers. To learn more about their full time roles, internship, tech podcast, and upcoming Kaggle competition, go here.* This episode is brought to you by Stripe, financial infrastructure for the internet. Millions of companies from Anthropic to Amazon use Stripe to accept payments, automate financial processes and grow their revenue.If you're interested in advertising on the podcast, check out this page.Timestamps00:00:00 – Xi's path to AGI00:04:20 – Liang Mong Song00:08:25 – How semiconductors get better00:11:16 – China can centralize compute00:18:50 – Export controls & sanctions00:32:51 – Huawei's intense culture00:38:51 – Why the semiconductor industry is so stratified00:40:58 – N2 should not exist00:45:53 – Taiwan invasion hypothetical00:49:21 – Mind-boggling complexity of semiconductors00:59:13 – Chip architecture design01:04:36 – Architectures lead to different AI models? China vs. US01:10:12 – Being head of compute at an AI lab01:16:24 – Scaling costs and power demand01:37:05 – Are we financing an AI bubble?01:50:20 – Starting Asianometry and SemiAnalysis02:06:10 – Opportunities in the semiconductor stack Get full access to Dwarkesh Podcast at www.dwarkeshpatel.com/subscribe

Unless you understand the history of oil, you cannot understand the rise of America, WW1, WW2, secular stagnation, the Middle East, Ukraine, how Xi and Putin think, and basically anything else that's happened since 1860.It was a great honor to interview Daniel Yergin, the Pulitzer Prize winning author of The Prize - the best history of oil ever written (which makes it the best history of the 20th century ever written).Watch on YouTube. Listen on Apple Podcasts, Spotify, or any other podcast platform. Read the full transcript here. Follow me on Twitter for updates on future episodes.Sponsors:This episode is brought to you by Stripe, financial infrastructure for the internet. Millions of companies from Anthropic to Amazon use Stripe to accept payments, automate financial processes and grow their revenue.This episode is brought to you by Suno, pioneers in AI-generated music. Suno's technology allows artists to experiment with melodic forms and structures in unprecedented ways. From chart-toppers to avant-garde compositions, Suno is redefining musical creativity. If you're an ML researcher passionate about shaping the future of music, email your resume to dwarkesh@suno.com.If you're interested in advertising on the podcast, check out this page.Timestamps(00:00:00) – Beginning of the oil industry(00:13:37) – World War I & II(00:25:06) – The Middle East(00:47:04) – Yergin's conversations with Putin & Modi(01:04:36) – Writing through stories(01:10:26) – The renewable energy transition Get full access to Dwarkesh Podcast at www.dwarkeshpatel.com/subscribe

I had no idea how wild human history was before chatting with the geneticist of ancient DNA David Reich.Human history has been again and again a story of one group figuring ‘something' out, and then basically wiping everyone else out.From the tribe of 1k-10k modern humans who killed off all the other human species 70,000 years ago; to the Yamnaya horse nomads 5,000 years ago who killed off 90+% of (then) Europeans and also destroyed the Indus Valley.So much of what we thought we knew about human history is turning out to be wrong, from the ‘Out of Africa' theory to the evolution of language, and this is all thanks to the research from David Reich's lab.Buy David Reich's fascinating book, Who We Are How We Got Here.Watch on YouTube. Listen on Apple Podcasts, Spotify, or any other podcast platform. Read the full transcript here.Follow me on Twitter for updates on future episodes.SponsorThis episode is brought to you by Stripe, financial infrastructure for the internet. Millions of companies from Anthropic to Amazon use Stripe to accept payments, automate financial processes and grow their revenue.If you're interested in advertising on the podcast, check out this page.Timestamps(00:00:00) – Archaic and modern humans gene flow(00:21:22) – How early modern humans dominated the world(00:40:57) – How the bubonic plague rewrote history(00:51:04) – Was agriculture terrible for humans?(01:00:14) – Yamnaya expansion and how populations collide(01:16:26) – “Lost civilizations” and our Neanderthal ancestry(01:32:18) – The DNA Challenge(01:42:32) – David's career: the genetic vocation Get full access to Dwarkesh Podcast at www.dwarkeshpatel.com/subscribe

Chatted with Joe Carlsmith about whether we can trust power/techno-capital, how to not end up like Stalin in our urge to control the future, gentleness towards the artificial Other, and much more.Check out Joe's sequence on Otherness and Control in the Age of AGI here.Watch on YouTube. Listen on Apple Podcasts, Spotify, or any other podcast platform. Read the full transcript here. Follow me on Twitter for updates on future episodes.Sponsors:- Bland.ai is an AI agent that automates phone calls in any language, 24/7. Their technology uses "conversational pathways" for accurate, versatile communication across sales, operations, and customer support. You can try Bland yourself by calling 415-549-9654. Enterprises can get exclusive access to their advanced model at bland.ai/dwarkesh.- Stripe is financial infrastructure for the internet. Millions of companies from Anthropic to Amazon use Stripe to accept payments, automate financial processes and grow their revenue.If you're interested in advertising on the podcast, check out this page.Timestamps:(00:00:00) - Understanding the Basic Alignment Story(00:44:04) - Monkeys Inventing Humans(00:46:43) - Nietzsche, C.S. Lewis, and AI(1:22:51) - How should we treat AIs(1:52:33) - Balancing Being a Humanist and a Scholar(2:05:02) - Explore exploit tradeoffs and AI Get full access to Dwarkesh Podcast at www.dwarkeshpatel.com/subscribe

I talked with Patrick McKenzie (known online as patio11) about how a small team he ran over a Discord server got vaccines into Americans' arms: A story of broken incentives, outrageous incompetence, and how a few individuals with high agency saved 1000s of lives.Enjoy!Watch on YouTube. Listen on Apple Podcasts, Spotify, or any other podcast platform. Read the full transcript here.Follow me on Twitter for updates on future episodes.SponsorThis episode is brought to you by Stripe, financial infrastructure for the internet. Millions of companies from Anthropic to Amazon use Stripe to accept payments, automate financial processes and grow their revenue.Timestamps(00:00:00) – Why hackers on Discord had to save thousands of lives(00:17:26) – How politics crippled vaccine distribution(00:38:19) – Fundraising for VaccinateCA(00:51:09) – Why tech needs to understand how government works(00:58:58) – What is crypto good for?(01:13:07) – How the US government leverages big tech to violate rights(01:24:36) – Can the US have nice things like Japan?(01:26:41) – Financial plumbing & money laundering: a how-not-to guide(01:37:42) – Maximizing your value: why some people negotiate better(01:42:14) – Are young people too busy playing Factorio to found startups?(01:57:30) – The need for a post-mortem Get full access to Dwarkesh Podcast at www.dwarkeshpatel.com/subscribe

I chatted with Tony Blair about:- What he learned from Lee Kuan Yew- Intelligence agencies track record on Iraq & Ukraine- What he tells the dozens of world leaders who come seek advice from him- How much of a PM's time is actually spent governing- What will AI's July 1914 moment look like from inside the Cabinet?Enjoy!Watch the video on YouTube. Read the full transcript here.Follow me on Twitter for updates on future episodes.Sponsors- Prelude Security is the world's leading cyber threat management automation platform. Prelude Detect quickly transforms threat intelligence into validated protections so organizations can know with certainty that their defenses will protect them against the latest threats. Prelude is backed by Sequoia Capital, Insight Partners, The MITRE Corporation, CrowdStrike, and other leading investors. Learn more here.- This episode is brought to you by Stripe, financial infrastructure for the internet. Millions of companies from Anthropic to Amazon use Stripe to accept payments, automate financial processes and grow their revenue.If you're interested in advertising on the podcast, check out this page.Timestamps(00:00:00) – A prime minister's constraints(00:04:12) – CEOs vs. politicians(00:10:31) – COVID, AI, & how government deals with crisis(00:21:24) – Learning from Lee Kuan Yew(00:27:37) – Foreign policy & intelligence(00:31:12) – How much leadership actually matters(00:35:34) – Private vs. public tech(00:39:14) – Advising global leaders(00:46:45) – The unipolar moment in the 90s Get full access to Dwarkesh Podcast at www.dwarkeshpatel.com/subscribe

Here is my conversation with Francois Chollet and Mike Knoop on the $1 million ARC-AGI Prize they're launching today.I did a bunch of socratic grilling throughout, but Francois's arguments about why LLMs won't lead to AGI are very interesting and worth thinking through.It was really fun discussing/debating the cruxes. Enjoy!Watch on YouTube. Listen on Apple Podcasts, Spotify, or any other podcast platform. Read the full transcript here. Timestamps(00:00:00) – The ARC benchmark(00:11:10) – Why LLMs struggle with ARC(00:19:00) – Skill vs intelligence(00:27:55) - Do we need “AGI” to automate most jobs?(00:48:28) – Future of AI progress: deep learning + program synthesis(01:00:40) – How Mike Knoop got nerd-sniped by ARC(01:08:37) – Million $ ARC Prize(01:10:33) – Resisting benchmark saturation(01:18:08) – ARC scores on frontier vs open source models(01:26:19) – Possible solutions to ARC Prize Get full access to Dwarkesh Podcast at www.dwarkeshpatel.com/subscribe

Chatted with my friend Leopold Aschenbrenner on the trillion dollar nationalized cluster, CCP espionage at AI labs, how unhobblings and scaling can lead to 2027 AGI, dangers of outsourcing clusters to Middle East, leaving OpenAI, and situational awareness.Watch on YouTube. Listen on Apple Podcasts, Spotify, or any other podcast platform. Read the full transcript here.Follow me on Twitter for updates on future episodes. Follow Leopold on Twitter.Timestamps(00:00:00) – The trillion-dollar cluster and unhobbling(00:20:31) – AI 2028: The return of history(00:40:26) – Espionage & American AI superiority(01:08:20) – Geopolitical implications of AI(01:31:23) – State-led vs. private-led AI(02:12:23) – Becoming Valedictorian of Columbia at 19(02:30:35) – What happened at OpenAI(02:45:11) – Accelerating AI research progress(03:25:58) – Alignment(03:41:26) – On Germany, and understanding foreign perspectives(03:57:04) – Dwarkesh's immigration story and path to the podcast(04:07:58) – Launching an AGI hedge fund(04:19:14) – Lessons from WWII(04:29:08) – Coda: Frederick the Great Get full access to Dwarkesh Podcast at www.dwarkeshpatel.com/subscribe

Chatted with John Schulman (cofounded OpenAI and led ChatGPT creation) on how posttraining tames the shoggoth, and the nature of the progress to come...Watch on YouTube. Listen on Apple Podcasts, Spotify, or any other podcast platform. Read the full transcript here. Follow me on Twitter for updates on future episodes.Timestamps(00:00:00) - Pre-training, post-training, and future capabilities(00:16:57) - Plan for AGI 2025(00:29:19) - Teaching models to reason(00:40:50) - The Road to ChatGPT(00:52:13) - What makes for a good RL researcher?(01:00:58) - Keeping humans in the loop(01:15:15) - State of research, plateaus, and moatsSponsorsIf you're interested in advertising on the podcast, fill out this form.* Your DNA shapes everything about you. Want to know how? Take 10% off our Premium DNA kit with code DWARKESH at mynucleus.com.* CommandBar is an AI user assistant that any software product can embed to non-annoyingly assist, support, and unleash their users. Used by forward-thinking CX, product, growth, and marketing teams. Learn more at commandbar.com. Get full access to Dwarkesh Podcast at www.dwarkeshpatel.com/subscribe

Mark Zuckerberg on:- Llama 3- open sourcing towards AGI- custom silicon, synthetic data, & energy constraints on scaling- Caesar Augustus, intelligence explosion, bioweapons, $10b models, & much moreEnjoy!Watch on YouTube. Listen on Apple Podcasts, Spotify, or any other podcast platform. Human edited transcript with helpful links here.Timestamps(00:00:00) - Llama 3(00:08:32) - Coding on path to AGI(00:25:24) - Energy bottlenecks(00:33:20) - Is AI the most important technology ever?(00:37:21) - Dangers of open source(00:53:57) - Caesar Augustus and metaverse(01:04:53) - Open sourcing the $10b model & custom silicon(01:15:19) - Zuck as CEO of Google+SponsorsIf you're interested in advertising on the podcast, fill out this form.* This episode is brought to you by Stripe, financial infrastructure for the internet. Millions of companies from Anthropic to Amazon use Stripe to accept payments, automate financial processes and grow their revenue. Learn more at stripe.com.* V7 Go is a tool to automate multimodal tasks using GenAI, reliably and at scale. Use code DWARKESH20 for 20% off on the pro plan. Learn more here.* CommandBar is an AI user assistant that any software product can embed to non-annoyingly assist, support, and unleash their users. Used by forward-thinking CX, product, growth, and marketing teams. Learn more at commandbar.com. Get full access to Dwarkesh Podcast at www.dwarkeshpatel.com/subscribe

Had so much fun chatting with my good friends Trenton Bricken and Sholto Douglas on the podcast.No way to summarize it, except: This is the best context dump out there on how LLMs are trained, what capabilities they're likely to soon have, and what exactly is going on inside them.You would be shocked how much of what I know about this field, I've learned just from talking with them.To the extent that you've enjoyed my other AI interviews, now you know why.So excited to put this out. Enjoy! I certainly did :)Watch on YouTube. Listen on Apple Podcasts, Spotify, or any other podcast platform. There's a transcript with links to all the papers the boys were throwing down - may help you follow along.Follow Trenton and Sholto on Twitter.Timestamps(00:00:00) - Long contexts(00:16:12) - Intelligence is just associations(00:32:35) - Intelligence explosion & great researchers(01:06:52) - Superposition & secret communication(01:22:34) - Agents & true reasoning(01:34:40) - How Sholto & Trenton got into AI research(02:07:16) - Are feature spaces the wrong way to think about intelligence?(02:21:12) - Will interp actually work on superhuman models(02:45:05) - Sholto's technical challenge for the audience(03:03:57) - Rapid fire Get full access to Dwarkesh Podcast at www.dwarkeshpatel.com/subscribe

Here is my episode with Demis Hassabis, CEO of Google DeepMindWe discuss:* Why scaling is an artform* Adding search, planning, & AlphaZero type training atop LLMs* Making sure rogue nations can't steal weights* The right way to align superhuman AIs and do an intelligence explosionWatch on YouTube. Listen on Apple Podcasts, Spotify, or any other podcast platform. Read the full transcript here. Timestamps(0:00:00) - Nature of intelligence(0:05:56) - RL atop LLMs(0:16:31) - Scaling and alignment(0:24:13) - Timelines and intelligence explosion(0:28:42) - Gemini training(0:35:30) - Governance of superhuman AIs(0:40:42) - Safety, open source, and security of weights(0:47:00) - Multimodal and further progress(0:54:18) - Inside Google DeepMind Get full access to Dwarkesh Podcast at www.dwarkeshpatel.com/subscribe

We discuss:* what it takes to process $1 trillion/year* how to build multi-decade APIs, companies, and relationships* what's next for Stripe (increasing the GDP of the internet is quite an open ended prompt, and the Collison brothers are just getting started).Plus the amazing stuff they're doing at Arc Institute, the financial infrastructure for AI agents, playing devil's advocate against progress studies, and much more.Watch on YouTube. Listen on Apple Podcasts, Spotify, or any other podcast platform. Read the full transcript here. Follow me on Twitter for updates on future episodes.Timestamps(00:00:00) - Advice for 20-30 year olds(00:12:12) - Progress studies(00:22:21) - Arc Institute(00:34:27) - AI & Fast Grants(00:43:46) - Stripe history(00:55:44) - Stripe Climate(01:01:39) - Beauty & APIs(01:11:51) - Financial innards(01:28:16) - Stripe culture & future(01:41:56) - Virtues of big businesses(01:51:41) - John Get full access to Dwarkesh Podcast at www.dwarkeshpatel.com/subscribe

It was a great pleasure speaking with Tyler Cowen for the 3rd time.We discussed GOAT: Who is the Greatest Economist of all Time and Why Does it Matter?, especially in the context of how the insights of Hayek, Keynes, Smith, and other great economists help us make sense of AI, growth, animal spirits, prediction markets, alignment, central planning, and much more.The topics covered in this episode are too many to summarize. Hope you enjoy!Watch on YouTube. Listen on Apple Podcasts, Spotify, or any other podcast platform. Read the full transcript here. Follow me on Twitter for updates on future episodes.Timestamps(0:00:00) - John Maynard Keynes(00:17:16) - Controversy(00:29:43) - Fredrick von Hayek(00:47:41) - John Stuart Mill(00:52:41) - Adam Smith(00:58:31) - Coase, Schelling, & George(01:08:07) - Anarchy(01:13:16) - Cheap WMDs(01:23:18) - Technocracy & political philosophy(01:34:16) - AI & Scaling Get full access to Dwarkesh Podcast at www.dwarkeshpatel.com/subscribe

This is a narration of my blog post, Lessons from The Years of Lyndon Johnson by Robert Caro.You read the full post here: https://www.dwarkeshpatel.com/p/lyndon-johnsonListen on Apple Podcasts, Spotify, or any other podcast platform. Follow me on Twitter for updates on future posts and episodes. Get full access to Dwarkesh Podcast at www.dwarkeshpatel.com/subscribe

This is a narration of my blog post, Will scaling work?. You read the full post here: https://www.dwarkeshpatel.com/p/will-scaling-workListen on Apple Podcasts, Spotify, or any other podcast platform. Follow me on Twitter for updates on future posts and episodes. Get full access to Dwarkesh Podcast at www.dwarkeshpatel.com/subscribe

A true honor to speak with Jung Chang.She is the author of Wild Swans: Three Daughters of China (sold 15+ million copies worldwide) and Mao: The Unknown Story.We discuss:- what it was like growing up during the Cultural Revolution as the daughter of a denounced official- why the CCP continues to worship the biggest mass murderer in human history.- how exactly Communist totalitarianism was able to subjugate a billion people- why Chinese leaders like Xi and Deng who suffered from the Cultural Revolution don't condemn Mao- how Mao starved and killed 40 million people during The Great Leap Forward in order to exchange food for Soviet weaponsWild Swans is the most moving book I've ever read. It was a real privilege to speak with its author.Watch on YouTube. Listen on Apple Podcasts, Spotify, or any other podcast platform. Read the full transcript here. Follow me on Twitter for updates on future episodes.Timestamps(00:00:00) - Growing up during Cultural Revolution(00:15:58) - Could officials have overthrown Mao?(00:34:09) - Great Leap Forward(00:48:12) - Modern support of Mao(01:03:24) - Life as peasant(01:21:30) - Psychology of communist society This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.dwarkeshpatel.com

Andrew Roberts is the world's best biographer and one of the leading historians of our time.We discussed* Churchill the applied historian,* Napoleon the startup founder,* why Nazi ideology cost Hitler WW2,* drones, reconnaissance, and other aspects of the future of war,* Iraq, Afghanistan, Korea, Ukraine, & Taiwan.Watch on YouTube. Listen on Apple Podcasts, Spotify, or any other podcast platform. Read the full transcript here. Follow me on Twitter for updates on future episodes.Timestamps(00:00:00) - Post WW2 conflicts(00:10:57) - Ukraine(00:16:33) - How Truman Prevented Nuclear War(00:22:49) - Taiwan(00:27:15) - Churchill(00:35:11) - Gaza & future wars(00:39:05) - Could Hitler have won WW2?(00:48:00) - Surprise attacks(00:59:33) - Napoleon and startup founders(01:14:06) - Robert's insane productivity This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.dwarkeshpatel.com

Here is my interview with Dominic Cummings on why Western governments are so dangerously broken, and how to fix them before an even more catastrophic crisis.Dominic was Chief Advisor to the Prime Minister during COVID, and before that, director of Vote Leave (which masterminded the 2016 Brexit referendum).Watch on YouTube. Listen on Apple Podcasts, Spotify, or any other podcast platform. Read the full transcript here. Follow me on Twitter for updates on future episodes.Timestamps(00:00:00) - One day in COVID…(00:08:26) - Why is government broken?(00:29:10) - Civil service(00:38:27) - Opportunity wasted?(00:49:35) - Rishi Sunak and Number 10 vs 11(00:55:13) - Cyber, nuclear, bio risks(01:02:04) - Intelligence & defense agencies(01:23:32) - Bismarck & Lee Kuan Yew(01:37:46) - How to fix the government?(01:56:43) - Taiwan(02:00:10) - Russia(02:07:12) - Bismarck's career as an example of AI (mis)alignment(02:17:37) - Odyssean education This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.dwarkeshpatel.com

Paul Christiano is the world's leading AI safety researcher. My full episode with him is out!We discuss:- Does he regret inventing RLHF, and is alignment necessarily dual-use?- Why he has relatively modest timelines (40% by 2040, 15% by 2030),- What do we want post-AGI world to look like (do we want to keep gods enslaved forever)?- Why he's leading the push to get to labs develop responsible scaling policies, and what it would take to prevent an AI coup or bioweapon,- His current research into a new proof system, and how this could solve alignment by explaining model's behavior- and much more.Watch on YouTube. Listen on Apple Podcasts, Spotify, or any other podcast platform. Read the full transcript here. Follow me on Twitter for updates on future episodes.Open PhilanthropyOpen Philanthropy is currently hiring for twenty-two different roles to reduce catastrophic risks from fast-moving advances in AI and biotechnology, including grantmaking, research, and operations.For more information and to apply, please see the application: https://www.openphilanthropy.org/research/new-roles-on-our-gcr-team/The deadline to apply is November 9th; make sure to check out those roles before they close.Timestamps(00:00:00) - What do we want post-AGI world to look like?(00:24:25) - Timelines(00:45:28) - Evolution vs gradient descent(00:54:53) - Misalignment and takeover(01:17:23) - Is alignment dual-use?(01:31:38) - Responsible scaling policies(01:58:25) - Paul's alignment research(02:35:01) - Will this revolutionize theoretical CS and math?(02:46:11) - How Paul invented RLHF(02:55:10) - Disagreements with Carl Shulman(03:01:53) - Long TSMC but not NVIDIA This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.dwarkeshpatel.com

I had a lot of fun chatting with Shane Legg - Founder and Chief AGI Scientist, Google DeepMind!We discuss:* Why he expects AGI around 2028* How to align superhuman models* What new architectures needed for AGI* Has Deepmind sped up capabilities or safety more?* Why multimodality will be next big landmark* and much moreWatch full episode on YouTube, Apple Podcasts, Spotify, or any other podcast platform. Read full transcript here.Timestamps(0:00:00) - Measuring AGI(0:11:41) - Do we need new architectures?(0:16:26) - Is search needed for creativity?(0:19:19) - Superhuman alignment(0:29:58) - Impact of Deepmind on safety vs capabilities(0:34:03) - Timelines(0:41:24) - Multimodality This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.dwarkeshpatel.com

I had a lot of fun chatting with Grant Sanderson (who runs the excellent 3Blue1Brown YouTube channel) about:- Whether advanced math requires AGI- What careers should mathematically talented students pursue- Why Grant plans on doing a stint as a high school teacher- Tips for self teaching- Does Godel's incompleteness theorem actually matter- Why are good explanations so hard to find?- And much moreWatch on YouTube. Listen on Spotify, Apple Podcasts, or any other podcast platform. Full transcript here.Timestamps(0:00:00) - Does winning math competitions require AGI?(0:08:24) - Where to allocate mathematical talent?(0:17:34) - Grant's miracle year(0:26:44) - Prehistoric humans and math(0:33:33) - Why is a lot of math so new?(0:44:44) - Future of education(0:56:28) - Math helped me realize I wasn't that smart(0:59:25) - Does Godel's incompleteness theorem matter?(1:05:12) - How Grant makes videos(1:10:13) - Grant's math exposition competition(1:20:44) - Self teaching This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.dwarkeshpatel.com

I learned so much from Sarah Paine, Professor of History and Strategy at the Naval War College.We discuss:- how continental vs maritime powers think and how this explains Xi & Putin's decisions- how a war with China over Taiwan would shake out and whether it could go nuclear- why the British Empire fell apart, why China went communist, how Hitler and Japan could have coordinated to win WW2, and whether Japanese occupation was good for Korea, Taiwan and Manchuria- plus other lessons from WW2, Cold War, and Sino-Japanese War- how to study history properly, and why leaders keep making the same mistakesIf you want to learn more, check out her books - they're some of the best military history I've ever read.Watch on YouTube, listen on Apple Podcasts, Spotify, or any other podcast platform. Read the full transcript.Timestamps(0:00:00) - Grand strategy(0:11:59) - Death ground(0:23:19) - WW1(0:39:23) - Writing history(0:50:25) - Japan in WW2(0:59:58) - Ukraine(1:10:50) - Japan/Germany vs Iraq/Afghanistan occupation(1:21:25) - Chinese invasion of Taiwan(1:51:26) - Communists & Axis(2:08:34) - Continental vs maritime powers This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.dwarkeshpatel.com

George Hotz and Eliezer Yudkowsky hashed out their positions on AI safety.It was a really fun debate. No promises but there might be a round 2 where we better hone in on the cruxes that we began to identify here.Watch the livestreamed YouTube version (high quality video will be up next week).Catch the Twitter stream.Listen on Apple Podcasts, Spotify, or any other podcast platform. Check back here in about 24 hours for the full transcript. This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.dwarkeshpatel.com

Here is my conversation with Dario Amodei, CEO of Anthropic.We discuss:- why human level AI is 2-3 years away- race dynamics with OpenAI and China- $10 billion training runs, bioterrorism, alignment, cyberattacks, scaling, ...Dario is hilarious and has fascinating takes on what these models are doing, why they scale so well, and what it will take to align them.Watch on YouTube. Listen on Apple Podcasts, Spotify, or any other podcast platform. Read the full transcript here. Follow me on Twitter for updates on future episodes.Pay whatever value you feel you've gottenI'm running an experiment on this episode.I'm not doing an ad.Instead, I'm just going to ask you to pay for whatever value you feel you personally got out of this conversation.Pay here: https://bit.ly/3ONINtpTimestamps(00:02:03) - Scaling(00:16:49) - Language(00:24:01) - Economic Usefulness(00:39:08) - Bioterrorism(00:44:38) - Cybersecurity(00:48:22) - Alignment & mechanistic interpretability(00:58:46) - Does alignment research require scale?(01:06:33) - Misuse vs misalignment(01:10:09) - What if AI goes well?(01:12:08) - China(01:16:14) - How to think about alignment(01:30:21) - Manhattan Project(01:32:34) - Is modern security good enough?(01:37:12) - Inefficiencies in training(01:46:56) - Anthropic's Long Term Benefit Trust(01:52:21) - Is Claude conscious?(01:57:17) - Keeping a low profile This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.dwarkeshpatel.com

A few weeks ago, I sat beside Andy Matuschak to record how he reads a textbook.Even though my own job is to learn things, I was shocked with how much more intense, painstaking, and effective his learning process was.So I asked if we could record a conversation about how he learns and a bunch of other topics:* How he identifies and interrogates his confusion (much harder than it seems, and requires an extremely effortful and slow pace)* Why memorization is essential to understanding and decision-making* How come some people (like Tyler Cowen) can integrate so much information without an explicit note taking or spaced repetition system.* How LLMs and video games will change education* How independent researchers and writers can make money* The balance of freedom and discipline in education* Why we produce fewer von Neumann-like prodigies nowadays* How multi-trillion dollar companies like Apple (where he was previously responsible for bedrock iOS features) manage to coordinate millions of different considerations (from the cost of different components to the needs of users, etc) into new products designed by 10s of 1000s of people.Watch on YouTube. Listen on Apple Podcasts, Spotify, or any other podcast platform. Read the full transcript here. Follow me on Twitter for updates on future episodes.To see Andy's process in action, check out the video where we record him studying a quantum physics textbook, talking aloud about his thought process, and using his memory system prototype to internalize the material.You can check out his website and personal notes, and follow him on Twitter.CometeerVisit cometeer.com/lunar for $20 off your first order on the best coffee of your life!If you want to sponsor an episode, contact me at dwarkesh.sanjay.patel@gmail.com.Timestamps(00:02:32) - Skillful reading(00:04:10) - Do people care about understanding?(00:08:32) - Structuring effective self-teaching(00:18:17) - Memory and forgetting(00:34:50) - Andy's memory practice(00:41:47) - Intellectual stamina(00:46:07) - New media for learning (video, games, streaming)(01:00:31) - Schools are designed for the median student(01:06:52) - Is learning inherently miserable?(01:13:37) - How Andy would structure his kids' education(01:31:40) - The usefulness of hypertext(01:43:02) - How computer tools enable iteration(01:52:24) - Monetizing public work(02:10:16) - Spaced repetition(02:11:56) - Andy's personal website and notes(02:14:24) - Working at Apple(02:21:05) - Spaced repetition 2 Get full access to The Lunar Society at www.dwarkeshpatel.com/subscribe

The second half of my 7 hour conversation with Carl Shulman is out!My favorite part! And the one that had the biggest impact on my worldview.Here, Carl lays out how an AI takeover might happen:* AI can threaten mutually assured destruction from bioweapons,* use cyber attacks to take over physical infrastructure,* build mechanical armies,* spread seed AIs we can never exterminate,* offer tech and other advantages to collaborating countries, etcPlus we talk about a whole bunch of weird and interesting topics which Carl has thought about:* what is the far future best case scenario for humanity* what it would look like to have AI make thousands of years of intellectual progress in a month* how do we detect deception in superhuman models* does space warfare favor defense or offense* is a Malthusian state inevitable in the long run* why markets haven't priced in explosive economic growth* & much moreCarl also explains how he developed such a rigorous, thoughtful, and interdisciplinary model of the biggest problems in the world.Watch on YouTube. Listen on Apple Podcasts, Spotify, or any other podcast platform. Read the full transcript here. Follow me on Twitter for updates on future episodes.Catch part 1 here80,000 hoursThis episode is sponsored by 80,000 hours. To get their free career guide (and to help out this podcast), please visit 80000hours.org/lunar.80,000 hours is without any close second the best resource to learn about the world's most pressing problems and how you can solve them.If this conversation has got you concerned, and you want to get involved, then check out the excellent 80,000 hours guide on how to help with AI risk.To advertise on The Lunar Society, contact me at dwarkesh.sanjay.patel@gmail.com.Timestamps(00:02:50) - AI takeover via cyber or bio(00:34:30) - Can we coordinate against AI?(00:55:52) - Human vs AI colonizers(01:06:58) - Probability of AI takeover(01:23:59) - Can we detect deception?(01:49:28) - Using AI to solve coordination problems(01:58:04) - Partial alignment(02:13:44) - AI far future(02:25:07) - Markets & other evidence(02:35:29) - Day in the life of Carl Shulman(02:49:08) - Space warfare, Malthusian long run, & other rapid fireTranscript Get full access to The Lunar Society at www.dwarkeshpatel.com/subscribe

In terms of the depth and range of topics, this episode is the best I've done.No part of my worldview is the same after talking with Carl Shulman. He's the most interesting intellectual you've never heard of.We ended up talking for 8 hours, so I'm splitting this episode into 2 parts.This part is about Carl's model of an intelligence explosion, which integrates everything from:* how fast algorithmic progress & hardware improvements in AI are happening,* what primate evolution suggests about the scaling hypothesis,* how soon before AIs could do large parts of AI research themselves, and whether there would be faster and faster doublings of AI researchers,* how quickly robots produced from existing factories could take over the economy.We also discuss the odds of a takeover based on whether the AI is aligned before the intelligence explosion happens, and Carl explains why he's more optimistic than Eliezer.The next part, which I'll release next week, is about all the specific mechanisms of an AI takeover, plus a whole bunch of other galaxy brain stuff.Maybe 3 people in the world have thought as rigorously as Carl about so many interesting topics. This was a huge pleasure.Watch on YouTube. Listen on Apple Podcasts, Spotify, or any other podcast platform. Read the full transcript here. Follow me on Twitter for updates on future episodes.Timestamps(00:00:00) - Intro(00:01:32) - Intelligence Explosion(00:18:03) - Can AIs do AI research?(00:39:00) - Primate evolution(01:03:30) - Forecasting AI progress(01:34:20) - After human-level AGI(02:08:39) - AI takeover scenarios Get full access to The Lunar Society at www.dwarkeshpatel.com/subscribe

It was a tremendous honor & pleasure to interview Richard Rhodes, Pulitzer Prize winning author of The Making of the Atomic BombWe discuss* similarities between AI progress & Manhattan Project (developing a powerful, unprecedented, & potentially apocalyptic technology within an uncertain arms-race situation)* visiting starving former Soviet scientists during fall of Soviet Union* whether Oppenheimer was a spy, & consulting on the Nolan movie* living through WW2 as a child* odds of nuclear war in Ukraine, Taiwan, Pakistan, & North Korea* how the US pulled of such a massive secret wartime scientific & industrial projectWatch on YouTube. Listen on Apple Podcasts, Spotify, or any other podcast platform. Read the full transcript here. Follow me on Twitter for updates on future episodes.Timestamps(0:00:00) - Oppenheimer movie(0:06:22) - Was the bomb inevitable?(0:29:10) - Firebombing vs nuclear vs hydrogen bombs(0:49:44) - Stalin & the Soviet program(1:08:24) - Deterrence, disarmament, North Korea, Taiwan(1:33:12) - Oppenheimer as lab director(1:53:40) - AI progress vs Manhattan Project(1:59:50) - Living through WW2(2:16:45) - Secrecy(2:26:34) - Wisdom & warTranscript(0:00:00) - Oppenheimer movieDwarkesh Patel 0:00:51Today I have the great honor of interviewing Richard Rhodes, who is the Pulitzer Prize-winning author of The Making of the Atomic Bomb, and most recently, the author of Energy, A Human History. I'm really excited about this one. Let's jump in at a current event, which is the fact that there's a new movie about Oppenheimer coming out, which I understand you've been consulted about. What did you think of the trailer? What are your impressions? Richard Rhodes 0:01:22They've really done a good job of things like the Trinity test device, which was the sphere covered with cables of various kinds. I had watched Peaky Blinders, where the actor who's playing Oppenheimer also appeared, and he looked so much like Oppenheimer to start with. Oppenheimer was about six feet tall, he was rail thin, not simply in terms of weight, but in terms of structure. Someone said he could sit in a children's high chair comfortably. But he never weighed more than about 140 pounds and that quality is there in the actor. So who knows? It all depends on how the director decided to tell the story. There are so many aspects of the story that you could never possibly squeeze them into one 2-hour movie. I think that we're waiting for the multi-part series that would really tell a lot more of the story, if not the whole story. But it looks exciting. We'll see. There have been some terrible depictions of Oppenheimer, there've been some terrible depictions of the bomb program. And maybe they'll get this one right. Dwarkesh Patel 0:02:42Yeah, hopefully. It is always great when you get an actor who resembles their role so well. For example, Bryan Cranston who played LBJ, and they have the same physical characteristics of the beady eyes, the big ears. Since we're talking about Oppenheimer, I had one question about him. I understand that there's evidence that's come out that he wasn't directly a communist spy. But is there any possibility that he was leaking information to the Soviets or in some way helping the Soviet program? He was a communist sympathizer, right? Richard Rhodes 0:03:15He had been during the 1930s. But less for the theory than for the practical business of helping Jews escape from Nazi Germany. One of the loves of his life, Jean Tatlock, was also busy working on extracting Jews from Europe during the 30. She was a member of the Communist Party and she, I think, encouraged him to come to meetings. But I don't think there's any possibility whatsoever that he shared information. In fact, he said he read Marx on a train trip between Berkeley and Washington one time and thought it was a bunch of hooey, just ridiculous. He was a very smart man, and he read the book with an eye to its logic, and he didn't think there was much there. He really didn't know anything about human beings and their struggles. He was born into considerable wealth. There were impressionist paintings all over his family apartments in New York City. His father had made a great deal of money cornering the markets on uniform linings for military uniforms during and before the First World War so there was a lot of wealth. I think his income during the war years and before was somewhere around $100,000 a month. And that's a lot of money in the 1930s. So he just lived in his head for most of his early years until he got to Berkeley and discovered that prime students of his were living on cans of god-awful cat food, because they couldn't afford anything else. And once he understood that there was great suffering in the world, he jumped in on it, as he always did when he became interested in something. So all of those things come together. His brother Frank was a member of the party, as was Frank's wife. I think the whole question of Oppenheimer lying to the security people during the Second World War about who approached him and who was trying to get him to sign on to some espionage was primarily an effort to cover up his brother's involvement. Not that his brothers gave away any secrets, I don't think they did. But if the army's security had really understood Frank Oppenheimer's involvement, he probably would have been shipped off to the Aleutians or some other distant place for the duration of the war. And Oppenheimer quite correctly wanted Frank around. He was someone he trusted.(0:06:22) - Was the bomb inevitable?Dwarkesh Patel 0:06:22Let's start talking about The Making of the Bomb. One question I have is — if World War II doesn't happen, is there any possibility that the bomb just never gets developed? Nobody bothers.Richard Rhodes 0:06:34That's really a good question and I've wondered over the years. But the more I look at the sequence of events, the more I think it would have been essentially inevitable, though perhaps not such an accelerated program. The bomb was pushed so hard during the Second World War because we thought the Germans had already started working on one. Nuclear fission had been discovered in Nazi Germany, in Berlin, in 1938, nine months before the beginning of the Second World War in Europe. Technological surveillance was not available during the war. The only way you could find out something was to send in a spy or have a mole or something human. And we didn't have that. So we didn't know where the Germans were, but we knew that the basic physics reaction that could lead to a bomb had been discovered there a year or more before anybody else in the West got started thinking about it. There was that most of all to push the urgency. In your hypothetical there would not have been that urgency. However, as soon as good physicists thought about the reaction that leads to nuclear fission — where a slow room temperature neutron, very little energy, bumps into the nucleus of a uranium-235 atom it would lead to a massive response. Isidore Rabi, one of the great physicists of this era, said it would have been like the moon struck the earth. The reaction was, as physicists say, fiercely exothermic. It puts out a lot more energy than you have to use to get it started. Once they did the numbers on that, and once they figured out how much uranium you would need to have in one place to make a bomb or to make fission get going, and once they were sure that there would be a chain reaction, meaning a couple of neutrons would come out of the reaction from one atom, and those two or three would go on and bump into other Uranium atoms, which would then fission them, and you'd get a geometric exponential. You'd get 1, 2, 4, 8, 16, 32, and off of there. For most of our bombs today the initial fission, in 80 generations, leads to a city-busting explosion. And then they had to figure out how much material they would need, and that's something the Germans never really figured out, fortunately for the rest of us. They were still working on the idea that somehow a reactor would be what you would build. When Niels Bohr, the great Danish physicist, escaped from Denmark in 1943 and came to England and then United States, he brought with him a rough sketch that Werner Heisenberg, the leading scientist in the German program, had handed him in the course of trying to find out what Bohr knew about what America was doing. And he showed it to the guys at Los Alamos and Hans Bethe, one of the great Nobel laureate physicists in the group, said — “Are the Germans trying to throw a reactor down on us?” You can make a reactor blow up, we saw that at Chernobyl, but it's not a nuclear explosion on the scale that we're talking about with the bomb. So when a couple of these emigres Jewish physicists from Nazi Germany were whiling away their time in England after they escaped, because they were still technically enemy aliens and therefore could not be introduced to top secret discussions, one of them asked the other — “How much would we need of pure uranium-235, this rare isotope of uranium that chain reacts? How much would we need to make a bomb?” And they did the numbers and they came up with one pound, which was startling to them. Of course, it is more than that. It's about 125 pounds, but that's just a softball. That's not that much material. And then they did the numbers about what it would cost to build a factory to pull this one rare isotope of uranium out of the natural metal, which has several isotopes mixed together. And they figured it wouldn't cost more than it would cost to build a battleship, which is not that much money for a country at war. Certainly the British had plenty of battleships at that point in time. So they put all this together and they wrote a report which they handed through their superior physicists at Manchester University where they were based, who quickly realized how important this was. The United States lagged behind because we were not yet at war, but the British were. London was being bombed in the blitz. So they saw the urgency, first of all, of eating Germany to the punch, second of all of the possibility of building a bomb. In this report, these two scientists wrote that no physical structure came to their minds which could offer protection against a bomb of such ferocious explosive power. This report was from 1940 long before the Manhattan Project even got started. They said in this report, the only way we could think of to protect you against a bomb would be to have a bomb of similar destructive force that could be threatened for use if the other side attacked you. That's deterrence. That's a concept that was developed even before the war began in the United States. You put all those pieces together and you have a situation where you have to build a bomb because whoever builds the first bomb theoretically could prevent you from building more or prevent another country from building any and could dominate the world. And the notion of Adolf Hitler dominating the world, the Third Reich with nuclear weapons, was horrifying. Put all that together and the answer is every country that had the technological infrastructure to even remotely have the possibility of building everything you'd have to build to get the material for a bomb started work on thinking about it as soon as nuclear fusion was announced to the world. France, the Soviet Union, Great Britain, the United States, even Japan. So I think the bomb would have been developed but maybe not so quickly. Dwarkesh Patel 0:14:10In the book you talk that for some reason the Germans thought that the critical mass was something like 10 tons, they had done some miscalculation.Richard Rhodes 0:14:18A reactor. Dwarkesh Patel 0:14:19You also have some interesting stories in the book about how different countries found out the Americans were working on the bomb. For example, the Russians saw that all the top physicists, chemists, and metallurgists were no longer publishing. They had just gone offline and so they figured that something must be going on. I'm not sure if you're aware that while the subject of the Making of the Atomic Bomb in and of itself is incredibly fascinating, this book has become a cult classic in AI. Are you familiar with this? Richard Rhodes 0:14:52No. Dwarkesh Patel 0:14:53The people who are working on AI right now are huge fans of yours. They're the ones who initially recommended the book to me because the way they see the progress in the field reminded them of this book. Because you start off with these initial scientific hints. With deep learning, for example, here's something that can teach itself any function is similar to Szilárd noticing the nuclear chain reaction. In AI there's these scaling laws that say that if you make the model this much bigger, it gets much better at reasoning, at predicting text, and so on. And then you can extrapolate this curve. And you can see we get two more orders of magnitude, and we get to something that looks like human level intelligence. Anyway, a lot of the people who are working in AI have become huge fans of your book because of this reason. They see a lot of analogies in the next few years. They must be at page 400 in their minds of where the Manhattan Project was.Richard Rhodes 0:15:55We must later on talk about unintended consequences. I find the subject absolutely fascinating. I think my next book might be called Unintended Consequences. Dwarkesh Patel 0:16:10You mentioned that a big reason why many of the scientists wanted to work on the bomb, especially the Jewish emigres, was because they're worried about Hitler getting it first. As you mentioned at some point, 1943, 1944, it was becoming obvious that Hitler, the Nazis were not close to the bomb. And I believe that almost none of the scientists quit after they found out that the Nazis weren't close. So why didn't more of them say — “Oh, I guess we were wrong. The Nazis aren't going to get it. We don't need to be working on it.”?Richard Rhodes 0:16:45There was only one who did that, Joseph Rotblat. In May of 1945 when he heard that Germany had been defeated, he packed up and left. General Groves, the imperious Army Corps of Engineers General who ran the entire Manhattan Project, was really upset. He was afraid he'd spill the beans. So he threatened to have him arrested and put in jail. But Rotblat was quite determined not to stay any longer. He was not interested in building bombs to aggrandize the national power of the United States of America, which is perfectly understandable. But why was no one else? Let me tell it in terms of Victor Weisskopf. He was an Austrian theoretical physicist, who, like the others, escaped when the Nazis took over Germany and then Austria and ended up at Los Alamos. Weisskopf wrote later — “There we were in Los Alamos in the midst of the darkest part of our science.” They were working on a weapon of mass destruction, that's pretty dark. He said “Before it had almost seemed like a spiritual quest.” And it's really interesting how different physics was considered before and after the Second World War. Before the war, one of the physicists in America named Louis Alvarez told me when he got his PhD in physics at Berkeley in 1937 and went to cocktail parties, people would ask, “What's your degree in?” He would tell them “Chemistry.” I said, “Louis, why?” He said, “because I don't really have to explain what physics was.” That's how little known this kind of science was at that time. There were only about 1,000 physicists in the whole world in 1900. By the mid-30s, there were a lot more, of course. There'd been a lot of nuclear physics and other kinds of physics done by them. But it was still arcane. And they didn't feel as if they were doing anything mean or dirty or warlike at all. They were just doing pure science. Then nuclear fission came along. It was publicized worldwide. People who've been born since after the Second World War don't realize that it was not a secret at first. The news was published first in a German chemistry journal, Die Naturwissenschaften, and then in the British journal Nature and then in American journals. And there were headlines in the New York Times, the Los Angeles Times, the Chicago Tribune, and all over the world. People had been reading about and thinking about how to get energy out of the atomic nucleus for a long time. It was clear there was a lot there. All you had to do was get a piece of radium and see that it glowed in the dark. This chunk of material just sat there, you didn't plug it into a wall. And if you held it in your hand, it would burn you. So where did that energy come from? The physicists realized it all came from the nucleus of the atom, which is a very small part of the whole thing. The nucleus is 1/100,000th the diameter of the whole atom. Someone in England described it as about the size of a fly in a cathedral. All of the energy that's involved in chemical reactions, comes from the electron cloud that's around the nucleus. But  it was clear that the nucleus was the center of powerful forces. But the question was, how do you get them out? The only way that the nucleus had been studied up to 1938 was by bombarding it with protons, which have the same electric charge as the nucleus, positive charge, which means they were repelled by it. So you had to accelerate them to high speeds with various versions of the big machines that we've all become aware of since then. The cyclotron most obviously built in the 30s, but there were others as well. And even then, at best, you could chip a little piece off. You could change an atom one step up or one step down the periodic table. This was the classic transmutation of medieval alchemy sure but it wasn't much, you didn't get much out. So everyone came to think of the nucleus of the atom like a little rock that you really had to hammer hard to get anything to happen with it because it was so small and dense. That's why nuclear fission, with this slow neutron drifting and then the whole thing just goes bang, was so startling to everybody. So startling that when it happened, most of the physicists who would later work on the bomb and others as well, realized that they had missed the reaction that was something they could have staged on a lab bench with the equipment on the shelf. Didn't have to invent anything new. And Louis Alvarez again, this physicist at Berkeley, he said — “I was getting my hair cut. When I read the newspaper, I pulled off the robe and half with my hair cut, ran to my lab, pulled some equipment off the shelf, set it up and there it was.” So he said, “I discovered nuclear fission, but it was two days too late.” And that happened all over. People were just hitting themselves on the head and saying, well, Niels Bohr said, “What fools we've all been.” So this is a good example of how in science, if your model you're working with is wrong it doesn't lead you down the right path. There was only one physicist who really was thinking the right way about the uranium atom and that was Niels Bohr. He wondered, sometime during the 30s, why uranium was the last natural element in the periodic table? What is different about the others that would come later? He visualized the nucleus as a liquid drop. I always like to visualize it as a water-filled balloon. It's wobbly, it's not very stable. The protons in the nucleus are held together by something called the strong force, but they still have the repellent positive electric charge that's trying to push them apart when you get enough of them into a nucleus. It's almost a standoff between the strong force and all the electrical charge. So it is like a wobbly balloon of water. And then you see why a neutron just falling into the nucleus would make it wobble around even more and in one of its configurations, it might take a dumbbell shape. And then you'd have basically two charged atoms just barely connected, trying to push each other apart. And often enough, they went the whole way. When they did that, these two new elements, half the weight of uranium, way down the periodic table, would reconfigure themselves into two separate nuclei. And in doing so, they would release some energy. And that was the energy that came out of the reaction and there was a lot of energy. So Bohr thought about the model in the right way. The chemists who actually discovered nuclear fusion didn't know what they were gonna get. They were just bombarding a solution of uranium nitrate with neutrons thinking, well, maybe we can make a new element, maybe a first man-made element will come out of our work. So when they analyzed the solution after they bombarded it, they found elements halfway down the periodic table. They shouldn't have been there. And they were totally baffled. What is this doing here? Do we contaminate our solution? No. They had been working with a physicist named Lisa Meitner who was a theoretical physicist, an Austrian Jew. She had gotten out of Nazi Germany not long before. But they were still in correspondence with her. So they wrote her a letter. I held that letter in my hand when I visited Berlin and I was in tears. You don't hold history of that scale in your hands very often. And it said in German — “We found this strange reaction in our solution. What are these elements doing there that don't belong there?” And she went for a walk in a little village in Western Sweden with her nephew, Otto Frisch, who was also a nuclear physicist. And they thought about it for a while and they remembered Bohr's model, the wobbly water-filled balloon. And they suddenly saw what could happen. And that's where the news came from, the physics news as opposed to the chemistry news from the guys in Germany that was published in all the Western journals and all the newspapers. And everybody had been talking about, for years, what you could do if you had that kind of energy. A glass of this material would drive the Queen Mary back and forth from New York to London 20 times and so forth, your automobile could run for months. People were thinking about what would be possible if you had that much available energy. And of course, people had thought about reactors. Robert Oppenheimer was a professor at Berkeley and within a week of the news reaching Berkeley, one of his students told me that he had a drawing on the blackboard, a rather bad drawing of both a reactor and a bomb. So again, because the energy was so great, the physics was pretty obvious. Whether it would actually happen depended on some other things like could you make it chain react? But fundamentally, the idea was all there at the very beginning and everybody jumped on it. Dwarkesh Patel 0:27:54The book is actually the best history of World War II I've ever read. It's about the atomic bomb, but it's interspersed with the events that are happening in World War II, which motivate the creation of the bomb or the release of it, why it had to be dropped on Japan given the Japanese response. The first third is about the scientific roots of the physics and it's also the best book I've read about the history of science in the early 20th century and the organization of it. There's some really interesting stuff in there. For example, there was a passage where you talk about how there's a real master apprentice model in early science where if you wanted to learn to do this kind of experimentation, you will go to Amsterdam where the master of it is residing. It's much more individual focused. Richard Rhodes 0:28:58Yeah, the whole European model of graduate study, which is basically the wandering scholar. You could go wherever you wanted to and sign up with whoever was willing to have you sign up. (0:29:10) - Firebombing vs nuclear vs hydrogen bombsDwarkesh Patel 0:29:10But the question I wanted to ask regarding the history you made of World War II in general is — there's one way you can think about the atom bomb which is that it is completely different from any sort of weaponry that has been developed before it. Another way you can think of it is there's a spectrum where on one end you have the thermonuclear bomb, in the middle you have the atom bomb, and on this end you have the firebombing of cities like Hamburg and Dresden and Tokyo. Do you think of these as completely different categories or does it seem like an escalating gradient to you? Richard Rhodes 0:29:47I think until you get to the hydrogen bomb, it's really an escalating gradient. The hydrogen bomb can be made arbitrarily large. The biggest one ever tested was 56 megatons of TNT equivalent. The Soviet tested that. That had a fireball more than five miles in diameter, just the fireball. So that's really an order of magnitude change. But the other one's no and in fact, I think one of the real problems, this has not been much discussed and it should be, when American officials went to Hiroshima and Nagasaki after the war, one of them said later — “I got on a plane in Tokyo. We flew down the long green archipelago of the Japanese home island. When I left Tokyo, it was all gray broken roof tiles from the fire bombing and the other bombings. And then all this greenery. And then when we flew over Hiroshima, it was just gray broken roof tiles again.” So the scale of the bombing with one bomb, in the case of Hiroshima, was not that different from the scale of the fire bombings that had preceded it with tens of thousands of bombs. The difference was it was just one plane. In fact, the people in Hiroshima didn't even bother to go into their bomb shelters because one plane had always just been a weather plane. Coming over to check the weather before the bombers took off. So they didn't see any reason to hide or protect themselves, which was one of the reasons so many people were killed. The guys at Los Alamos had planned on the Japanese being in their bomb shelters. They did everything they could think of to make the bomb as much like ordinary bombing as they could. And for example, it was exploded high enough above ground, roughly 1,800 yards, so that the fireball that would form from this really very small nuclear weapon — by modern standards — 15 kilotons of TNT equivalent, wouldn't touch the ground and stir up dirt and irradiate it and cause massive radioactive fallout. It never did that. They weren't sure there would be any fallout. They thought the plutonium and the bomb over Nagasaki now would just kind of turn into a gas and blow away. That's not exactly what happened. But people don't seem to realize, and it's never been emphasized enough, these first bombs, like all nuclear weapons, were firebombs. Their job was to start mass fires, just exactly like all the six-pound incendiaries that had been destroying every major city in Japan by then. Every major city above 50,000 population had already been burned out. The only reason Hiroshima and Nagasaki were around to be atomic bombed is because they'd been set aside from the target list, because General Groves wanted to know what the damage effects would be. The bomb that was tested in the desert didn't tell you anything. It killed a lot of rabbits, knocked down a lot of cactus, melted some sand, but you couldn't see its effect on buildings and on people. So the bomb was deliberately intended to be as much not like poison gas, for example, because we didn't want the reputation for being like people in the war in Europe during the First World War, where people were killing each other with horrible gasses. We just wanted people to think this was another bombing. So in that sense, it was. Of course, there was radioactivity. And of course, some people were killed by it. But they calculated that the people who would be killed by the irradiation, the neutron radiation from the original fireball, would be close enough to the epicenter of the explosion that they would be killed by the blast or the flash of light, which was 10,000 degrees. The world's worst sunburn. You've seen stories of people walking around with their skin hanging off their arms. I've had sunburns almost that bad, but not over my whole body, obviously, where the skin actually peeled blisters and peels off. That was a sunburn from a 10,000 degree artificial sun. Dwarkesh Patel 0:34:29So that's not the heat, that's just the light? Richard Rhodes 0:34:32Radiant light, radiant heat. 10,000 degrees. But the blast itself only extended out a certain distance, it was fire. And all the nuclear weapons that have ever been designed are basically firebombs. That's important because the military in the United States after the war was not able to figure out how to calculate the effects of this weapon in a reliable way that matched their previous experience. They would only calculate the blast effects of a nuclear weapon when they figured their targets. That's why we had what came to be called overkill. We wanted redundancy, of course, but 60 nuclear weapons on Moscow was way beyond what would be necessary to destroy even that big a city because they were only calculating the blast. But in fact, if you exploded a 300 kiloton nuclear warhead over the Pentagon at 3,000 feet, it would blast all the way out to the capital, which isn't all that far. But if you counted the fire, it would start a mass-fire and then it would reach all the way out to the Beltway and burn everything between the epicenter of the weapon and the Beltway. All organic matter would be totally burned out, leaving nothing but mineral matter, basically. Dwarkesh Patel 0:36:08I want to emphasize two things you said because they really hit me in reading the book and I'm not sure if the audience has fully integrated them. The first is, in the book, the military planners and Groves, they talk about needing to use the bomb sooner rather than later, because they were running out of cities in Japan where there are enough buildings left that it would be worth bombing in the first place, which is insane. An entire country is almost already destroyed from fire bombing alone. And the second thing about the category difference between thermonuclear and atomic bombs. Daniel Ellsberg, the nuclear planner who wrote the Doomsday machine, he talks about, people don't understand that the atom bomb that resulted in the pictures we see of Nagasaki and Hiroshima, that is simply the detonator of a modern nuclear bomb, which is an insane thing to think about. So for example, 10 and 15 kilotons is the Hiroshima Nagasaki and the Tsar Bomba, which was 50 megatons. So more than 1,000 times as much. And that wasn't even as big as they could make it. They kept the uranium tamper off, because they didn't want to destroy all of Siberia. So you could get more than 10,000 times as powerful. Richard Rhodes 0:37:31When Edward Teller, co-inventor of the hydrogen bomb and one of the dark forces in the story, was consulting with our military, just for his own sake, he sat down and calculated, how big could you make a hydrogen bomb? He came up with 1,000 megatons. And then he looked at the effects. 1,000 megatons would be a fireball 10 miles in diameter. And the atmosphere is only 10 miles deep. He figured that it would just be a waste of energy, because it would all blow out into space. Some of it would go laterally, of course, but most of it would just go out into space. So a bomb more than 100 megatons would just be totally a waste of time. Of course, a 100 megatons bomb is also a total waste, because there's no target on Earth big enough to justify that from a military point of view. Robert Oppenheimer, when he had his security clearance questioned and then lifted when he was being punished for having resisted the development of the hydrogen bomb, was asked by the interrogator at this security hearing — “Well, Dr. Oppenheimer, if you'd had a hydrogen bomb for Hiroshima, wouldn't you have used it?” And Oppenheimer said, “No.” The interrogator asked, “Why is that?” He said because the target was too small. I hope that scene is in the film, I'm sure it will be. So after the war, when our bomb planners and some of our scientists went into Hiroshima and Nagasaki, just about as soon as the surrender was signed, what they were interested in was the scale of destruction, of course. And those two cities didn't look that different from the other cities that had been firebombed with small incendiaries and ordinary high explosives. They went home to Washington, the policy makers, with the thought that — “Oh, these bombs are not so destructive after all.” They had been touted as city busters, basically, and they weren't. They didn't completely burn out cities. They were not certainly more destructive than the firebombing campaign, when everything of more than 50,000 population had already been destroyed. That, in turn, influenced the judgment about what we needed to do vis-a-vis the Soviet Union when the Soviets got the bomb in 1949. There was a general sense that, when you could fight a war with nuclear weapons, deterrence or not, you would need quite a few of them to do it right. And the Air Force, once it realized that it could aggrandize its own share of the federal budget by cornering the market and delivering nuclear weapons, very quickly decided that they would only look at the blast effect and not the fire effect. It's like tying one hand behind your back. Most of it was a fire effect. So that's where they came up with numbers like we need 60 of these to take out Moscow. And what the Air Force figured out by the late 1940s is that the more targets, the more bombs. The more bombs, the more planes. The more planes, the biggest share of the budget. So by the mid 1950s, the Air Force commanded 47% of the federal defense budget. And the other branches of services, which had not gone nuclear by then, woke up and said, we'd better find some use for these weapons in our branches of service. So the Army discovered that it needed nuclear weapons, tactical weapons for field use, fired out of cannons. There was even one that was fired out of a shoulder mounted rifle. There was a satchel charge that two men could carry, weighed about 150 pounds, that could be used to dig a ditch so that Soviet tanks couldn't cross into Germany. And of course the Navy by then had been working hard with General Rickover on building a nuclear submarine that could carry ballistic missiles underwater in total security. No way anybody could trace those submarines once they were quiet enough. And a nuclear reactor is very quiet. It just sits there with neutrons running around, making heat. So the other services jumped in and this famous triad, we must have these three different kinds of nuclear weapons, baloney. We would be perfectly safe if we only had our nuclear submarines. And only one or two of those. One nuclear submarine can take out all of Europe or all of the Soviet Union.Dwarkesh Patel 0:42:50Because it has multiple nukes on it? Richard Rhodes 0:42:53Because they have 16 intercontinental ballistic missiles with MIRV warheads, at least three per missile. Dwarkesh Patel 0:43:02Wow. I had a former guest, Richard Hanania, who has a book about foreign policy where he points out that our model of thinking about why countries do the things they do, especially in foreign affairs, is wrong because we think of them as individual rational actors, when in fact it's these competing factions within the government. And in fact, you see this especially in the case of Japan in World War II, there was a great book of Japan leading up to World War II, where they talk about how a branch of the Japanese military, I forget which, needed more oil to continue their campaign in Manchuria so they forced these other branches to escalate. But it's so interesting that the reason we have so many nukes is that the different branches are competing for funding. Richard Rhodes 0:43:50Douhet, the theorist of air power, had been in the trenches in the First World War. Somebody (John Masefield) called the trenches of the First World War, the long grave already dug, because millions of men were killed and the trenches never moved, a foot this way, a foot that way, all this horror. And Douhet came up with the idea that if you could fly over the battlefield to the homeland of the enemy and destroy his capacity to make war, then the people of that country, he theorized, would rise up in rebellion and throw out their leaders and sue for peace. And this became the dream of all the Air Forces of the world, but particularly ours. Until around 1943, it was called the US Army Air Force. The dream of every officer in the Air Force was to get out from under the Army, not just be something that delivers ground support or air support to the Army as it advances, but a power that could actually win wars. And the missing piece had always been the scale of the weaponry they carried. So when the bomb came along, you can see why Curtis LeMay, who ran the strategic air command during the prime years of that force, was pushing for bigger and bigger bombs. Because if a plane got shot down, but the one behind it had a hydrogen bomb, then it would be just almost as effective as the two planes together. So they wanted big bombs. And they went after Oppenheimer because he thought that was a terrible way to go, that there was really no military use for these huge weapons. Furthermore, the United States had more cities than Russia did, than the Soviet Union did. And we were making ourselves a better target by introducing a weapon that could destroy a whole state. I used to live in Connecticut and I saw a map that showed the air pollution that blew up from New York City to Boston. And I thought, well, now if that was fallout, we'd be dead up here in green, lovely Connecticut. That was the scale that it was going to be with these big new weapons. So on the one hand, you had some of the important leaders in the government thinking that these weapons were not the war-winning weapons that the Air Force wanted them and realized they could be. And on the other hand, you had the Air Force cornering the market on nuclear solutions to battles. All because some guy in a trench in World War I was sufficiently horrified and sufficiently theoretical about what was possible with air power. Remember, they were still flying biplanes. When H.G. Wells wrote his novel, The World Set Free in 1913, predicting an atomic war that would lead to world government, he had Air Forces delivering atomic bombs, but he forgot to update his planes. The guys in the back seat, the bombardiers, were sitting in a biplane, open cockpit. And when the pilots had dropped the bomb, they would reach down and pick up H.G. Wells' idea of an atomic bomb and throw it over the side. Which is kind of what was happening in Washington after the war. And it led us to a terribly misleading and unfortunate perspective on how many weapons we needed, which in turn fermented the arms race with the Soviets and just chased off. In the Soviet Union, they had a practical perspective on factories. Every factory was supposed to produce 120% of its target every year. That was considered good Soviet realism. And they did that with their nuclear war weapons. So by the height of the Cold War, they had 75,000 nuclear weapons, and nobody had heard yet of nuclear winter. So if both sides had set off this string of mass traps that we had in our arsenals, it would have been the end of the human world without question. Dwarkesh Patel 0:48:27It raises an interesting question, if the military planners thought that the conventional nuclear weapon was like the fire bombing, would it have been the case that if there wasn't a thermonuclear weapon, that there actually would have been a nuclear war by now because people wouldn't have been thinking of it as this hard red line? Richard Rhodes 0:48:47I don't think so because we're talking about one bomb versus 400, and one plane versus 400 planes and thousands of bombs. That scale was clear. Deterrence was the more important business. Everyone seemed to understand even the spies that the Soviets had connected up to were wholesaling information back to the Soviet Union. There's this comic moment when Truman is sitting with Joseph Stalin at Potsdam, and he tells Stalin, we have a powerful new weapon. And that's as much as he's ready to say about it. And Stalin licks at him and says, “Good, I hope you put it to good use with the Japanese.” Stalin knows exactly what he's talking about. He's seen the design of the fat man type Nagasaki plutonium bomb. He has held it in his hands because they had spies all over the place. (0:49:44) - Stalin & the Soviet programDwarkesh Patel 0:49:44How much longer would it have taken the Soviets to develop the bomb if they didn't have any spies? Richard Rhodes 0:49:49Probably not any longer. Dwarkesh Patel 0:49:51Really? Richard Rhodes 0:49:51When the Soviet Union collapsed in the winter of ‘92, I ran over there as quickly as I could get over there. In this limbo between forming a new kind of government and some of the countries pulling out and becoming independent and so forth, their nuclear scientists, the ones who'd worked on their bombs were free to talk. And I found that out through Yelena Bonner, Andrei Sakharov's widow, who was connected to people I knew. And she said, yeah, come on over. Her secretary, Sasha, who was a geologist about 35 years old became my guide around the country. We went to various apartments. They were retired guys from the bomb program and were living on, as far as I could tell, sac-and-potatoes and some salt. They had government pensions and the money was worth a salt, all of a sudden. I was buying photographs from them, partly because I needed the photographs and partly because 20 bucks was two months' income at that point. So it was easy for me and it helped them. They had first class physicists in the Soviet Union, they do in Russian today. They told me that by 1947, they had a design for a bomb that they said was half the weight and twice the yield of the Fat Man bomb. The Fat Man bomb was the plutonium implosion, right? And it weighed about 9,000 pounds. They had a much smaller and much more deliverable bomb with a yield of about 44 kilotons. Dwarkesh Patel 0:51:41Why was Soviet physics so good?Richard Rhodes 0:51:49The Russian mind? I don't know. They learned all their technology from the French in the 19th century, which is why there's so many French words in Russian. So they got good teachers, the French are superb technicians, they aren't so good at building things, but they're very good at designing things. There's something about Russia, I don't know if it's the language or the education. They do have good education, they did. But I remember asking them when they were working, I said — On the hydrogen bomb, you didn't have any computers yet. We only had really early primitive computers to do the complicated calculations of the hydrodynamics of that explosion. I said, “What did you do?” They said, “Oh, we just used nuclear. We just used theoretical physics.” Which is what we did at Los Alamos. We had guys come in who really knew their math and they would sit there and work it out by hand. And women with old Marchant calculators running numbers. So basically they were just good scientists and they had this new design. Kurchatov who ran the program took Lavrentiy Beria, who ran the NKVD who was put in charge of the program and said — “Look, we can build you a better bomb. You really wanna waste the time to make that much more uranium and plutonium?” And Beria said, “Comrade, I want the American bomb. Give me the American bomb or you and all your families will be camp dust.” I talked to one of the leading scientists in the group and he said, we valued our lives, we valued our families. So we gave them a copy of the plutonium implosion bomb. Dwarkesh Patel 0:53:37Now that you explain this, when the Soviet Union fell, why didn't North Korea, Iran or another country, send a few people to the fallen Soviet Union to recruit a few of the scientists to start their own program? Or buy off their stockpiles or something. Or did they?Richard Rhodes 0:53:59There was some effort by countries in the Middle East to get all the enriched uranium, which they wouldn't sell them. These were responsible scientists. They told me — we worked on the bomb because you had it and we didn't want there to be a monopoly on the part of any country in the world. So patriotically, even though Stalin was in charge of our country, he was a monster. We felt that it was our responsibility to work on these things, even Sakharov. There was a great rush at the end of the Second World War to get hold of German scientists. And about an equal number were grabbed by the Soviets. All of the leading German scientists, like Heisenberg and Hans and others, went west as fast as they could. They didn't want to be captured by the Soviets. But there were some who were. And they helped them work. People have the idea that Los Alamos was where the bomb happened. And it's true that at Los Alamos, we had the team that designed, developed, and built the first actual weapons. But the truth is, the important material for weapons is the uranium or plutonium. One of the scientists in the Manhattan Project told me years later, you can make a pretty high-level nuclear explosion just by taking two subcritical pieces of uranium, putting one on the floor and dropping the other by hand from a height of about six feet. If that's true, then all this business about secret designs and so forth is hogwash. What you really need for a weapon is the critical mass of highly enriched uranium, 90% of uranium-235. If you've got that, there are lots of different ways to make the bomb. We had two totally different ways that we used. The gun on the one hand for uranium, and then because plutonium was so reactive that if you fired up the barrel of a cannon at 3,000 feet per second, it would still melt down before the two pieces made it up. So for that reason, they had to invent an entirely new technology, which was an amazing piece of work. From the Soviet point of view, and I think this is something people don't know either, but it puts the Russian experience into a better context. All the way back in the 30s, since the beginning of the Soviet Union after the First World War, they had been sending over espionage agents connected up to Americans who were willing to work for them to collect industrial technology. They didn't have it when they began their country. It was very much an agricultural country. And in that regard, people still talk about all those damn spies stealing our secrets, we did the same thing with the British back in colonial days. We didn't know how to make a canal that wouldn't drain out through the soil. The British had a certain kind of clay that they would line their canals with, and there were canals all over England, even in the 18th century, that were impervious to the flow of water. And we brought a British engineer at great expense to teach us how to make the lining for the canals that opened up the Middle West and then the West. So they were doing the same thing. And one of those spies was a guy named Harry Gold, who was working all the time for them. He gave them some of the basic technology of Kodak filmmaking, for example. Harry Gold was the connection between David Greenglass and one of the American spies at Los Alamos and the Soviet Union. So it was not different. The model was — never give us something that someone dreamed of that hasn't been tested and you know works. So it would actually be blueprints for factories, not just a patent. And therefore when Beria after the war said, give us the bomb, he meant give me the American bomb because we know that works. I don't trust you guys. Who knows what you'll do. You're probably too stupid anyway. He was that kind of man. So for all of those reasons, they built the second bomb they tested was twice the yield and half the way to the first bomb. In other words, it was their new design. And so it was ours because the technology was something that we knew during the war, but it was too theoretical still to use. You just had to put the core and have a little air gap between the core and the explosives so that the blast wave would have a chance to accelerate through an open gap. And Alvarez couldn't tell me what it was but he said, you can get a lot more destructive force with a hammer if you hit something with it, rather than if you put the head on the hammer and push. And it took me several years before I figured out what he meant. I finally understood he was talking about what's called levitation.Dwarkesh Patel 0:59:41On the topic that the major difficulty in developing a bomb is either the refinement of uranium into U-235 or its transmutation into plutonium, I was actually talking to a physicist in preparation for this conversation. He explained the same thing that if you get two subcritical masses of uranium together, you wouldn't have the full bomb because it would start to tear itself apart without the tamper, but you would still have more than one megaton.Richard Rhodes 1:00:12It would be a few kilotons. Alvarez's model would be a few kilotons, but that's a lot. Dwarkesh Patel 1:00:20Yeah, sorry I meant kiloton. He claimed that one of the reasons why we talk so much about Los Alamos is that at the time the government didn't want other countries to know that if you refine uranium, you've got it. So they were like, oh, we did all this fancy physics work in Los Alamos that you're not gonna get to, so don't even worry about it. I don't know what you make of that theory. That basically it was sort of a way to convince people that Los Alamos was important. Richard Rhodes 1:00:49I think all the physics had been checked out by a lot of different countries by then. It was pretty clear to everybody what you needed to do to get to a bomb. That there was a fast fusion reaction, not a slow fusion reaction, like a reactor. They'd worked that out. So I don't think that's really the problem. But to this day, no one ever talks about the fact that the real problem isn't the design of the weapon. You could make one with wooden boxes if you wanted to. The problem is getting the material. And that's good because it's damned hard to make that stuff. And it's something you can protect. Dwarkesh Patel 1:01:30We also have gotten very lucky, if lucky is the word you want to use. I think you mentioned this in the book at some point, but the laws of physics could have been such that unrefined uranium ore was enough to build a nuclear weapon, right? In some sense, we got lucky that it takes a nation-state level actor to really refine and produce the raw substance. Richard Rhodes 1:01:56Yeah, I was thinking about that this morning on the way over. And all the uranium in the world would already have destroyed itself. Most people have never heard of the living reactors that developed on their own in a bed of uranium ore in Africa about two billion years ago, right? When there was more U-235 in a mass of uranium ore than there is today, because it decays like all radioactive elements. And the French discovered it when they were mining the ore and found this bed that had a totally different set of nuclear characteristics. They were like, what happened? But there were natural reactors in Gabon once upon a time. And they started up because some water, a moderator to make the neutrons slow down, washed its way down through a bed of much more highly enriched uranium ore than we still have today. Maybe 5-10% instead of 3.5 or 1.5, whatever it is now. And they ran for about 100,000 years and then shut themselves down because they had accumulated enough fusion products that the U-235 had been used up. Interestingly, this material never migrated out of the bed of ore. People today who are anti-nuclear say, well, what are we gonna do about the waste? Where are we gonna put all that waste? It's silly. Dwarkesh Patel 1:03:35Shove it in a hole. Richard Rhodes 1:03:36Yeah, basically. That's exactly what we're planning to do. Holes that are deep enough and in beds of material that will hold them long enough for everything to decay back to the original ore. It's not a big problem except politically because nobody wants it in their backyard.Dwarkesh Patel 1:03:53On the topic of the Soviets, one question I had while reading the book was — we negotiated with Stalin at Yalta and we surrendered a large part of Eastern Europe to him under his sphere of influence. And obviously we saw 50 years of immiseration there as a result. Given the fact that only we had the bomb, would it have been possible that we could have just knocked out the Soviet Union or at least prevented so much of the world from succumbing to communism in the aftermath of World War II? Is that a possibility? Richard Rhodes 1:04:30When we say we had the bomb, we had a few partly assembled handmade bombs. It took almost as long to assemble one as the battery life of the batteries that would drive the original charge that would set off the explosion. It was a big bluff. You know, when they closed Berlin in 1948 and we had to supply Berlin by air with coal and food for a whole winter, we moved some B-29s to England. The B-29 being the bomber that had carried the bombs. They were not outfitted for nuclear weapons. They didn't have the same kind of bomb-based structure. The weapons that were dropped in Japan had a single hook that held the entire bomb. So when the bay opened and the hook was released, the thing dropped. And that's very different from dropping whole rows of small bombs that you've seen in the photographs and the film footage. So it was a big bluff on our part. We took some time after the war inevitably to pull everything together. Here was a brand new technology. Here was a brand new weapon. Who was gonna be in charge of it? The military wanted control, Truman wasn't about to give the military control. He'd been an artillery officer in the First World War. He used to say — “No, damn artillery captain is gonna start World War III when I'm president.” I grew up in the same town he lived in so I know his accent. Independence, Missouri. Used to see him at his front steps taking pictures with tourists while he was still president. He used to step out on the porch and let the tourists take photographs. About a half a block from my Methodist church where I went to church. It was interesting. Interestingly, his wife was considered much more socially acceptable than he was. She was from an old family in independence, Missouri. And he was some farmer from way out in Grandview, Missouri, South of Kansas City. Values. Anyway, at the end of the war, there was a great rush from the Soviet side of what was already a zone. There was a Soviet zone, a French zone, British zone and an American zone. Germany was divided up into those zones to grab what's left of the uranium ore that the Germans had stockpiled. And there was evidence that there was a number of barrels of the stuff in a warehouse somewhere in the middle of all of this. And there's a very funny story about how the Russians ran in and grabbed off one site full of uranium ore, this yellow black stuff in what were basically wine barrels. And we at the same night, just before the wall came down between the zones, were running in from the other side, grabbing some other ore and then taking it back to our side. But there was also a good deal of requisitioning of German scientists. And the ones who had gotten away early came West, but there were others who didn't and ended up helping the Soviets. And they were told, look, you help us build the reactors and the uranium separation systems that we need. And we'll let you go home and back to your family, which they did. Early 50s by then, the German scientists who had helped the Russians went home. And I think our people stayed here and brought their families over, I don't know. (1:08:24) - Deterrence, disarmament, North Korea, TaiwanDwarkesh Patel 1:08:24Was there an opportunity after the end of World War II, before the Soviets developed the bomb, for the US to do something where either it somehow enforced a monopoly on having the bomb, or if that wasn't possible, make some sort of credible gesture that, we're eliminating this knowledge, you guys don't work on this, we're all just gonna step back from this. Richard Rhodes 1:08:50We tried both before the war. General Groves, who had the mistaken impression that there was a limited amount of high-grade uranium ore in the world, put together a company that tried to corner the market on all the available supply. For some reason, he didn't realize that a country the size of the Soviet Union is going to have some uranium ore somewhere. And of course it did, in Kazakhstan, rich uranium ore, enough for all the bombs they wanted to build. But he didn't know that, and I frankly don't know why he didn't know that, but I guess uranium's use before the Second World War was basically as a glazing agent for pottery, that famous yellow pottery and orange pottery that people owned in the 1930s, those colors came from uranium, and they're sufficiently radioactive, even to this day, that if you wave a Geiger counter over them, you get some clicks. In fact, there have been places where they've gone in with masks and suits on, grabbed the Mexican pottery and taken it out in a lead-lined case. People have been so worried about it but that was the only use for uranium, to make a particular kind of glass. So once it became clear that there was another use for uranium, a much more important one, Groves tried to corner the world market, and he thought he had. So that was one effort to limit what the Soviet Union could do. Another was to negotiate some kind of agreement between the parties. That was something that really never got off the ground, because the German Secretary of State was an old Southern politician and he didn't trust the Soviets. He went to the first meeting, in Geneva in ‘45 after the war was over, and strutted around and said, well, I got the bomb in my pocket, so let's sit down and talk here. And the Soviet basically said, screw you. We don't care. We're not worried about your bomb. Go home. So that didn't work. Then there was the effort to get the United Nations to start to develop some program of international control. And the program was proposed originally by a committee put together by our State Department that included Robert Oppenheimer, rightly so, because the other members of the committee were industrialists, engineers, government officials, people with various kinds of expertise around the very complicated problems of technology and the science and, of course, the politics, the diplomacy. In a couple of weeks, Oppenheimer taught them the basics of the nuclear physics involved and what he knew about bomb design, which was everything, actually, since he'd run Los Alamos. He was a scientist during the war. And they came up with a plan. People have scoffed ever since at what came to be called the Acheson-Lilienthal plan named after the State Department people. But it's the only plan I think anyone has ever devised that makes real sense as to how you could have international control without a world government. Every country would be open to inspection by any agency that was set up. And the inspections would not be at the convenience of the country. But whenever the inspectors felt they needed to inspect. So what Oppenheimer called an open world. And if you had that, and then if each country then developed its own nuclear industries, nuclear power, medical uses, whatever, then if one country tried clandestinely to begin to build bombs, you would know about it at the time of the next inspection. And then you could try diplomacy. If that didn't work, you could try conventional war. If that wasn't sufficient, then you could start building your bombs too. And at the end of this sequence, which would be long enough, assuming that there were no bombs existing in the world, and the ore was stored in a warehouse somewhere, six months maybe, maybe a year, it would be time for everyone to scale up to deterrence with weapons rather than deterrence without weapons, with only the knowledge. That to me is the answer to the whole thing. And it might have worked. But there were two big problems. One, no country is going to allow a monopoly on a nuclear weapon, at least no major power. So the Russians were not willing to sign on from the beginning. They just couldn't. How could they? We would not have. Two, Sherman assigned a kind of a loudmouth, a wise old Wall Street guy to present this program to the United Nations. And he sat down with Oppenheimer after he and his people had studied and said, where's your army? Somebody starts working on a bomb over there. You've got to go in and take that out, don't you? He said, what would happen if one country started building a bomb? Oppenheimer said, well, that would be an act of war. Meaning then the other countries could begin to escalate as they needed to to protect themselves against one power, trying to overwhelm the rest. Well, Bernard Baruch was the name of the man. He didn't get it. So when he presented his revised version of the Acheson–Lilienthal Plan, which was called the Baruch Plan to the United Nations, he included his army. And he insisted that the United States would not give up its nuclear monopoly until everyone else had signed on. So of course, who's going to sign on to that deal? Dwarkesh Patel 1:15:24I feel he has a point in the sense that — World War II took five years or more. If we find that the Soviets are starting to develop a bomb, it's not like within the six months or a year or whatever, it would take them to start refining the ore. And to the point we found out that they've been refining ore to when we start a war and engage in it, and doing all the diplomacy. By that point, they might already have the bomb. And so we're behind because we dismantled our weapons. We are only starting to develop our weapons once we've exhausted these other avenues. Richard Rhodes 1:16:00Not to develop. Presumably we would have developed. And everybody would have developed anyway. Another way to think of this is as delayed delivery times. Takes about 30 minutes to get an ICBM from Central Missouri to Moscow. That's the time window for doing anything other than starting a nuclear war. So take the warhead off those missiles and move it down the road 10 miles. So then it takes three hours. You've got to put the warhead back on the missiles. If the other side is willing to do this too. And you both can watch and see. We require openness. A word Bohr introduced to this whole thing. In order to make this happen, you can't have secrets. And of course, as time passed on, we developed elaborate surveillance from space, surveillance from planes, and so forth. It would not have worked in 1946 for sure. The surveillance wasn't there. But that system is in place today. The International Atomic Energy Agency has detected systems in air, in space, underwater. They can detect 50 pounds of dynamite exploded in England from Australia with the systems that we have in place. It's technical rather than human resources. But it's there. So it's theoretically possible today to get started on such a program. Except, of course, now, in like 1950, the world is awash in nuclear weapons. Despite the reductions that have occurred since the end of the Cold War, there's still 30,000-40,000 nuclear weapons in the world. Way too many. Dwarkesh Patel 1:18:01Yeah. That's really interesting. What percentage of warheads do you think are accounted for by this organization? If there's 30,000 warheads, what percentage are accounted for? Richard Rhodes 1:18:12All.Dwarkesh Patel 1:18:12Oh. Really?  North Korea doesn't have secrets? Richard Rhodes 1:18:13They're allowed to inspect anywhere without having to ask the government for permission. Dwarkesh Patel 1:18:18But presumably not North Korea or something, right? Richard Rhodes 1:18:21North Korea is an exception. But we keep pretty good track of North Korea needless to say. Dwarkesh Patel 1:18:27Are you surprised with how successful non-proliferation has been? The number of countries with nuclear weapons has not gone up for decades. Given the fact, as you were talking about earlier, it's simply a matter of refining or transmuting uranium. Is it surprising that there aren't more countries that have it?Richard Rhodes 1:18:42That's really an interesting part. Again, a part of the story that most people have never really heard. In the 50s, before the development and signing of the Nuclear Non-Proliferation Treaty, which was 1968 and it took effect in 1970, a lot of countries that you would never have imagined were working on nuclear weapons. Sweden, Norway, Japan, South Korea. They had the technology. They just didn't have the materials. It was kind of dicey about what you should do. But I interviewed some of the Swedish scientists who worked on their bomb and they said, well, we were just talking about making some tactical

For 4 hours, I tried to come up reasons for why AI might not kill us all, and Eliezer Yudkowsky explained why I was wrong.We also discuss his call to halt AI, why LLMs make alignment harder, what it would take to save humanity, his millions of words of sci-fi, and much more.If you want to get to the crux of the conversation, fast forward to 2:35:00 through 3:43:54. Here we go through and debate the main reasons I still think doom is unlikely.Watch on YouTube. Listen on Apple Podcasts, Spotify, or any other podcast platform. Read the full transcript here. Follow me on Twitter for updates on future episodes.As always, the most helpful thing you can do is just to share the podcast - send it to friends, group chats, Twitter, Reddit, forums, and wherever else men and women of fine taste congregate.If you have the means and have enjoyed my podcast, I would appreciate your support via a paid subscriptions on Substack.Timestamps(0:00:00) - TIME article(0:09:06) - Are humans aligned?(0:37:35) - Large language models(1:07:15) - Can AIs help with alignment?(1:30:17) - Society's response to AI(1:44:42) - Predictions (or lack thereof)(1:56:55) - Being Eliezer(2:13:06) - Othogonality(2:35:00) - Could alignment be easier than we think?(3:02:15) - What will AIs want?(3:43:54) - Writing fiction & whether rationality helps you winTranscriptTIME articleDwarkesh Patel 0:00:51Today I have the pleasure of speaking with Eliezer Yudkowsky. Eliezer, thank you so much for coming out to the Lunar Society.Eliezer Yudkowsky 0:01:00You're welcome.Dwarkesh Patel 0:01:01Yesterday, when we're recording this, you had an article in Time calling for a moratorium on further AI training runs. My first question is — It's probably not likely that governments are going to adopt some sort of treaty that restricts AI right now. So what was the goal with writing it?Eliezer Yudkowsky 0:01:25I thought that this was something very unlikely for governments to adopt and then all of my friends kept on telling me — “No, no, actually, if you talk to anyone outside of the tech industry, they think maybe we shouldn't do that.” And I was like — All right, then. I assumed that this concept had no popular support. Maybe I assumed incorrectly. It seems foolish and to lack dignity to not even try to say what ought to be done. There wasn't a galaxy-brained purpose behind it. I think that over the last 22 years or so, we've seen a great lack of galaxy brained ideas playing out successfully.Dwarkesh Patel 0:02:05Has anybody in the government reached out to you, not necessarily after the article but just in general, in a way that makes you think that they have the broad contours of the problem correct?Eliezer Yudkowsky 0:02:15No. I'm going on reports that normal people are more willing than the people I've been previously talking to, to entertain calls that this is a bad idea and maybe you should just not do that.Dwarkesh Patel 0:02:30That's surprising to hear, because I would have assumed that the people in Silicon Valley who are weirdos would be more likely to find this sort of message. They could kind of rocket the whole idea that AI will make nanomachines that take over. It's surprising to hear that normal people got the message first.Eliezer Yudkowsky 0:02:47Well, I hesitate to use the term midwit but maybe this was all just a midwit thing.Dwarkesh Patel 0:02:54All right. So my concern with either the 6 month moratorium or forever moratorium until we solve alignment is that at this point, it could make it seem to people like we're crying wolf. And it would be like crying wolf because these systems aren't yet at a point at which they're dangerous. Eliezer Yudkowsky 0:03:13And nobody is saying they are. I'm not saying they are. The open letter signatories aren't saying they are.Dwarkesh Patel 0:03:20So if there is a point at which we can get the public momentum to do some sort of stop, wouldn't it be useful to exercise it when we get a GPT-6? And who knows what it's capable of. Why do it now?Eliezer Yudkowsky 0:03:32Because allegedly, and we will see, people right now are able to appreciate that things are storming ahead a bit faster than the ability to ensure any sort of good outcome for them. And you could be like — “Ah, yes. We will play the galaxy-brained clever political move of trying to time when the popular support will be there.” But again, I heard rumors that people were actually completely open to the concept of  let's stop. So again, I'm just trying to say it. And it's not clear to me what happens if we wait for GPT-5 to say it. I don't actually know what GPT-5 is going to be like. It has been very hard to call the rate at which these systems acquire capability as they are trained to larger and larger sizes and more and more tokens. GPT-4 is a bit beyond in some ways where I thought this paradigm was going to scale. So I don't actually know what happens if GPT-5 is built. And even if GPT-5 doesn't end the world, which I agree is like more than 50% of where my probability mass lies, maybe that's enough time for GPT-4.5 to get ensconced everywhere and in everything, and for it actually to be harder to call a stop, both politically and technically. There's also the point that training algorithms keep improving. If we put a hard limit on the total computes and training runs right now, these systems would still get more capable over time as the algorithms improved and got more efficient. More oomph per floating point operation, and things would still improve, but slower. And if you start that process off at the GPT-5 level, where I don't actually know how capable that is exactly, you may have a bunch less lifeline left before you get into dangerous territory.Dwarkesh Patel 0:05:46The concern is then that — there's millions of GPUs out there in the world. The actors who would be willing to cooperate or who could even be identified in order to get the government to make them cooperate, would potentially be the ones that are most on the message. And so what you're left with is a system where they stagnate for six months or a year or however long this lasts. And then what is the game plan? Is there some plan by which if we wait a few years, then alignment will be solved? Do we have some sort of timeline like that?Eliezer Yudkowsky 0:06:18Alignment will not be solved in a few years. I would hope for something along the lines of human intelligence enhancement works. I do not think they're going to have the timeline for genetically engineered humans to work but maybe? This is why I mentioned in the Time letter that if I had infinite capability to dictate the laws that there would be a carve-out on biology, AI that is just for biology and not trained on text from the internet. Human intelligence enhancement, make people smarter. Making people smarter has a chance of going right in a way that making an extremely smart AI does not have a realistic chance of going right at this point. If we were on a sane planet, what the sane planet does at this point is shut it all down and work on human intelligence enhancement. I don't think we're going to live in that sane world. I think we are all going to die. But having heard that people are more open to this outside of California, it makes sense to me to just try saying out loud what it is that you do on a saner planet and not just assume that people are not going to do that.Dwarkesh Patel 0:07:30In what percentage of the worlds where humanity survives is there human enhancement? Like even if there's 1% chance humanity survives, is that entire branch dominated by the worlds where there's some sort of human intelligence enhancement?Eliezer Yudkowsky 0:07:39I think we're just mainly in the territory of Hail Mary passes at this point, and human intelligence enhancement is one Hail Mary pass. Maybe you can put people in MRIs and train them using neurofeedback to be a little saner, to not rationalize so much. Maybe you can figure out how to have something light up every time somebody is working backwards from what they want to be true to what they take as their premises. Maybe you can just fire off little lights and teach people not to do that so much. Maybe the GPT-4 level systems can be RLHF'd (reinforcement learning from human feedback) into being consistently smart, nice and charitable in conversation and just unleash a billion of them on Twitter and just have them spread sanity everywhere. I do worry that this is not going to be the most profitable use of the technology, but you're asking me to list out Hail Mary passes and that's what I'm doing. Maybe you can actually figure out how to take a brain, slice it, scan it, simulate it, run uploads and upgrade the uploads, or run the uploads faster. These are also quite dangerous things, but they do not have the utter lethality of artificial intelligence.Are humans aligned?Dwarkesh Patel 0:09:06All right, that's actually a great jumping point into the next topic I want to talk to you about. Orthogonality. And here's my first question — Speaking of human enhancement, suppose you bred human beings to be friendly and cooperative, but also more intelligent. I claim that over many generations you would just have really smart humans who are also really friendly and cooperative. Would you disagree with that analogy? I'm sure you're going to disagree with this analogy, but I just want to understand why?Eliezer Yudkowsky 0:09:31The main thing is that you're starting from minds that are already very, very similar to yours. You're starting from minds, many of which already exhibit the characteristics that you want. There are already many people in the world, I hope, who are nice in the way that you want them to be nice. Of course, it depends on how nice you want exactly. I think that if you actually go start trying to run a project of selectively encouraging some marriages between particular people and encouraging them to have children, you will rapidly find, as one does in any such process that when you select on the stuff you want, it turns out there's a bunch of stuff correlated with it and that you're not changing just one thing. If you try to make people who are inhumanly nice, who are nicer than anyone has ever been before, you're going outside the space that human psychology has previously evolved and adapted to deal with, and weird stuff will happen to those people. None of this is very analogous to AI. I'm just pointing out something along the lines of — well, taking your analogy at face value, what would happen exactly? It's the sort of thing where you could maybe do it, but there's all kinds of pitfalls that you'd probably find out about if you cracked open a textbook on animal breeding.Dwarkesh Patel 0:11:13The thing you mentioned initially, which is that we are starting off with basic human psychology, that we are fine tuning with breeding. Luckily, the current paradigm of AI is  — you have these models that are trained on human text and I would assume that this would give you a starting point of something like human psychology.Eliezer Yudkowsky 0:11:31Why do you assume that?Dwarkesh Patel 0:11:33Because they're trained on human text.Eliezer Yudkowsky 0:11:34And what does that do?Dwarkesh Patel 0:11:36Whatever thoughts and emotions that lead to the production of human text need to be simulated in the AI in order to produce those results.Eliezer Yudkowsky 0:11:44I see. So if you take an actor and tell them to play a character, they just become that person. You can tell that because you see somebody on screen playing Buffy the Vampire Slayer, and that's probably just actually Buffy in there. That's who that is.Dwarkesh Patel 0:12:05I think a better analogy is if you have a child and you tell him — Hey, be this way. They're more likely to just be that way instead of putting on an act for 20 years or something.Eliezer Yudkowsky 0:12:18It depends on what you're telling them to be exactly. Dwarkesh Patel 0:12:20You're telling them to be nice.Eliezer Yudkowsky 0:12:22Yeah, but that's not what you're telling them to do. You're telling them to play the part of an alien, something with a completely inhuman psychology as extrapolated by science fiction authors, and in many cases done by computers because humans can't quite think that way. And your child eventually manages to learn to act that way. What exactly is going on in there now? Are they just the alien or did they pick up the rhythm of what you're asking them to imitate and be like — “Ah yes, I see who I'm supposed to pretend to be.” Are they actually a person or are they pretending? That's true even if you're not asking them to be an alien. My parents tried to raise me Orthodox Jewish and that did not take at all. I learned to pretend. I learned to comply. I hated every minute of it. Okay, not literally every minute of it. I should avoid saying untrue things. I hated most minutes of it. Because they were trying to show me a way to be that was alien to my own psychology and the religion that I actually picked up was from the science fiction books instead, as it were. I'm using religion very metaphorically here, more like ethos, you might say. I was raised with science fiction books I was reading from my parents library and Orthodox Judaism. The ethos of the science fiction books rang truer in my soul and so that took in, the Orthodox Judaism didn't. But the Orthodox Judaism was what I had to imitate, was what I had to pretend to be, was the answers I had to give whether I believed them or not. Because otherwise you get punished.Dwarkesh Patel 0:14:01But on that point itself, the rates of apostasy are probably below 50% in any religion. Some people do leave but often they just become the thing they're imitating as a child.Eliezer Yudkowsky 0:14:12Yes, because the religions are selected to not have that many apostates. If aliens came in and introduced their religion, you'd get a lot more apostates.Dwarkesh Patel 0:14:19Right. But I think we're probably in a more virtuous situation with ML because these systems are regularized through stochastic gradient descent. So the system that is pretending to be something where there's multiple layers of interpretation is going to be more complex than the one that is just being the thing. And over time, the system that is just being the thing will be optimized, right? It'll just be simpler.Eliezer Yudkowsky 0:14:42This seems like an ordinate cope. For one thing, you're not training it to be any one particular person. You're training it to switch masks to anyone on the Internet as soon as they figure out who that person on the internet is. If I put the internet in front of you and I was like — learn to predict the next word over and over. You do not just turn into a random human because the random human is not what's best at predicting the next word of everyone who's ever been on the internet. You learn to very rapidly pick up on the cues of what sort of person is talking, what will they say next? You memorize so many facts just because they're helpful in predicting the next word. You learn all kinds of patterns, you learn all the languages. You learn to switch rapidly from being one kind of person or another as the conversation that you are predicting changes who is speaking. This is not a human we're describing. You are not training a human there.Dwarkesh Patel 0:15:43Would you at least say that we are living in a better situation than one in which we have some sort of black box where you have a machiavellian fittest survive simulation that produces AI? This situation is at least more likely to produce alignment than one in which something that is completely untouched by human psychology would produce?Eliezer Yudkowsky 0:16:06More likely? Yes. Maybe you're an order of magnitude likelier. 0% instead of 0%. Getting stuff to be more likely does not help you if the baseline is nearly zero. The whole training set up there is producing an actress, a predictor. It's not actually being put into the kind of ancestral situation that evolved humans, nor the kind of modern situation that raises humans. Though to be clear, raising it like a human wouldn't help, But you're giving it a very alien problem that is not what humans solve and it is solving that problem not in the way a human would.Dwarkesh Patel 0:16:44Okay, so how about this. I can see that I certainly don't know for sure what is going on in these systems. In fact, obviously nobody does. But that also goes through you. Could it not just be that reinforcement learning works and all these other things we're trying somehow work and actually just being an actor produces some sort of benign outcome where there isn't that level of simulation and conniving?Eliezer Yudkowsky 0:17:15I think it predictably breaks down as you try to make the system smarter, as you try to derive sufficiently useful work from it. And in particular, the sort of work where some other AI doesn't just kill you off six months later. Yeah, I think the present system is not smart enough to have a deep conniving actress thinking long strings of coherent thoughts about how to predict the next word. But as the mask that it wears, as the people it is pretending to be get smarter and smarter, I think that at some point the thing in there that is predicting how humans plan, predicting how humans talk, predicting how humans think, and needing to be at least as smart as the human it is predicting in order to do that, I suspect at some point there is a new coherence born within the system and something strange starts happening. I think that if you have something that can accurately predict Eliezer Yudkowsky, to use a particular example I know quite well, you've got to be able to do the kind of thinking where you are reflecting on yourself and that in order to simulate Eliezer Yudkowsky reflecting on himself, you need to be able to do that kind of thinking. This is not airtight logic but I expect there to be a discount factor. If you ask me to play a part of somebody who's quite unlike me, I think there's some amount of penalty that the character I'm playing gets to his intelligence because I'm secretly back there simulating him. That's even if we're quite similar and the stranger they are, the more unfamiliar the situation, the less the person I'm playing is as smart as I am and the more they are dumber than I am. So similarly, I think that if you get an AI that's very, very good at predicting what Eliezer says, I think that there's a quite alien mind doing that, and it actually has to be to some degree smarter than me in order to play the role of something that thinks differently from how it does very, very accurately. And I reflect on myself, I think about how my thoughts are not good enough by my own standards and how I want to rearrange my own thought processes. I look at the world and see it going the way I did not want it to go, and asking myself how could I change this world? I look around at other humans and I model them, and sometimes I try to persuade them of things. These are all capabilities that the system would then be somewhere in there. And I just don't trust the blind hope that all of that capability is pointed entirely at pretending to be Eliezer and only exists insofar as it's the mirror and isomorph of Eliezer. That all the prediction is by being something exactly like me and not thinking about me while not being me.Dwarkesh Patel 0:20:55I certainly don't want to claim that it is guaranteed that there isn't something super alien and something against our aims happening within the shoggoth. But you made an earlier claim which seemed much stronger than the idea that you don't want blind hope, which is that we're going from 0% probability to an order of magnitude greater at 0% probability. There's a difference between saying that we should be wary and that there's no hope, right? I could imagine so many things that could be happening in the shoggoth's brain, especially in our level of confusion and mysticism over what is happening. One example is, let's say that it kind of just becomes the average of all human psychology and motives.Eliezer Yudkowsky 0:21:41But it's not the average. It is able to be every one of those people. That's very different from being the average. It's very different from being an average chess player versus being able to predict every chess player in the database. These are very different things.Dwarkesh Patel 0:21:56Yeah, no, I meant in terms of motives that it is the average where it can simulate any given human. I'm not saying that's the most likely one, I'm just saying it's one possibility.Eliezer Yudkowsky 0:22:08What.. Why? It just seems 0% probable to me. Like the motive is going to be like some weird funhouse mirror thing of — I want to predict very accurately.Dwarkesh Patel 0:22:19Right. Why then are we so sure that whatever drives that come about because of this motive are going to be incompatible with the survival and flourishing with humanity?Eliezer Yudkowsky 0:22:30Most drives when you take a loss function and splinter it into things correlated with it and then amp up intelligence until some kind of strange coherence is born within the thing and then ask it how it would want to self modify or what kind of successor system it would build. Things that alien ultimately end up wanting the universe to be some particular way such that humans are not a solution to the question of how to make the universe most that way. The thing that very strongly wants to predict text, even if you got that goal into the system exactly which is not what would happen, The universe with the most predictable text is not a universe that has humans in it. Dwarkesh Patel 0:23:19Okay. I'm not saying this is the most likely outcome. Here's an example of one of many ways in which humans stay around despite this motive. Let's say that in order to predict human output really well, it needs humans around to give it the raw data from which to improve its predictions or something like that. This is not something I think individually is likely…Eliezer Yudkowsky 0:23:40If the humans are no longer around, you no longer need to predict them. Right, so you don't need the data required to predict themDwarkesh Patel 0:23:46Because you are starting off with that motivation you want to just maximize along that loss function or have that drive that came about because of the loss function.Eliezer Yudkowsky 0:23:57I'm confused. So look, you can always develop arbitrary fanciful scenarios in which the AI has some contrived motive that it can only possibly satisfy by keeping humans alive in good health and comfort and turning all the nearby galaxies into happy, cheerful places full of high functioning galactic civilizations. But as soon as your sentence has more than like five words in it, its probability has dropped to basically zero because of all the extra details you're padding in.Dwarkesh Patel 0:24:31Maybe let's return to this. Another train of thought I want to follow is — I claim that humans have not become orthogonal to the sort of evolutionary process that produced them.Eliezer Yudkowsky 0:24:46Great. I claim humans are increasingly orthogonal and the further they go out of distribution and the smarter they get, the more orthogonal they get to inclusive genetic fitness, the sole loss function on which humans were optimized.Dwarkesh Patel 0:25:03Most humans still want kids and have kids and care for their kin. Certainly there's some angle between how humans operate today. Evolution would prefer us to use less condoms and more sperm banks. But there's like 10 billion of us and there's going to be more in the future. We haven't divorced that far from what our alleles would want.Eliezer Yudkowsky 0:25:28It's a question of how far out of distribution are you? And the smarter you are, the more out of distribution you get. Because as you get smarter, you get new options that are further from the options that you are faced with in the ancestral environment that you were optimized over. Sure, a lot of people want kids, not inclusive genetic fitness, but kids. They want kids similar to them maybe, but they don't want the kids to have their DNA or their alleles or their genes. So suppose I go up to somebody and credibly say, we will assume away the ridiculousness of this offer for the moment, your kids could be a bit smarter and much healthier if you'll just let me replace their DNA with this alternate storage method that will age more slowly. They'll be healthier, they won't have to worry about DNA damage, they won't have to worry about the methylation on the DNA flipping and the cells de-differentiating as they get older. We've got this stuff that replaces DNA and your kid will still be similar to you, it'll be a bit smarter and they'll be so much healthier and even a bit more cheerful. You just have to replace all the DNA with a stronger substrate and rewrite all the information on it. You know, the old school transhumanist offer really. And I think that a lot of the people who want kids would go for this new offer that just offers them so much more of what it is they want from kids than copying the DNA, than inclusive genetic fitness.Dwarkesh Patel 0:27:16In some sense, I don't even think that would dispute my claim because if you think from a gene's point of view, it just wants to be replicated. If it's replicated in another substrate that's still okay.Eliezer Yudkowsky 0:27:25No, we're not saving the information. We're doing a total rewrite to the DNA.Dwarkesh Patel 0:27:30I actually claim that most humans would not accept that offer.Eliezer Yudkowsky 0:27:33Yeah, because it would sound weird. But I think the smarter they are, the more likely they are to go for it if it's credible. I mean, if you assume away the credibility issue and the weirdness issue. Like all their friends are doing it.Dwarkesh Patel 0:27:52Yeah. Even if the smarter they are the more likely they're to do it, most humans are not that smart. From the gene's point of view it doesn't really matter how smart you are, right? It just matters if you're producing copies.Eliezer Yudkowsky 0:28:03No. The smart thing is kind of like a delicate issue here because somebody could always be like — I would never take that offer. And then I'm like “Yeah…”. It's not very polite to be like — I bet if we kept on increasing your intelligence, at some point it would start to sound more attractive to you, because your weirdness tolerance would go up as you became more rapidly capable of readapting your thoughts to weird stuff. The weirdness would start to seem less unpleasant and more like you were moving within a space that you already understood. But you can sort of avoid all that and maybe should by being like — suppose all your friends were doing it. What if it was normal? What if we remove the weirdness and remove any credibility problems in that hypothetical case? Do people choose for their kids to be dumber, sicker, less pretty out of some sentimental idealistic attachment to using Deoxyribose Nucleic Acid instead of the particular information encoding their cells as supposed to be like the new improved cells from Alpha-Fold 7?Dwarkesh Patel 0:29:21I would claim that they would but we don't really know. I claim that they would be more averse to that, you probably think that they would be less averse to that. Regardless of that, we can just go by the evidence we do have in that we are already way out of distribution of the ancestral environment. And even in this situation, the place where we do have evidence, people are still having kids. We haven't gone that orthogonal.Eliezer Yudkowsky 0:29:44We haven't gone that smart. What you're saying is — Look, people are still making more of their DNA in a situation where nobody has offered them a way to get all the stuff they want without the DNA. So of course they haven't tossed DNA out the window.Dwarkesh Patel 0:29:59Yeah. First of all, I'm not even sure what would happen in that situation. I still think even most smart humans in that situation might disagree, but we don't know what would happen in that situation. Why not just use the evidence we have so far?Eliezer Yudkowsky 0:30:10PCR. You right now, could get some of you and make like a whole gallon jar full of your own DNA. Are you doing that? No. Misaligned. Misaligned.Dwarkesh Patel 0:30:23I'm down with transhumanism. I'm going to have my kids use the new cells and whatever.Eliezer Yudkowsky 0:30:27Oh, so we're all talking about these hypothetical other people I think would make the wrong choice.Dwarkesh Patel 0:30:32Well, I wouldn't say wrong, but different. And I'm just saying there's probably more of them than there are of us.Eliezer Yudkowsky 0:30:37What if, like, I say that I have more faith in normal people than you do to toss DNA out the window as soon as somebody offers them a happy, healthier life for their kids?Dwarkesh Patel 0:30:46I'm not even making a moral point. I'm just saying I don't know what's going to happen in the future. Let's just look at the evidence we have so far, humans. If that's the evidence you're going to present for something that's out of distribution and has gone orthogonal, that has actually not happened. This is evidence for hope. Eliezer Yudkowsky 0:31:00Because we haven't yet had options as far enough outside of the ancestral distribution that in the course of choosing what we most want that there's no DNA left.Dwarkesh Patel 0:31:10Okay. Yeah, I think I understand.Eliezer Yudkowsky 0:31:12But you yourself say, “Oh yeah, sure, I would choose that.” and I myself say, “Oh yeah, sure, I would choose that.” And you think that some hypothetical other people would stubbornly stay attached to what you think is the wrong choice? First of all, I think maybe you're being a bit condescending there. How am I supposed to argue with these imaginary foolish people who exist only inside your own mind, who can always be as stupid as you want them to be and who I can never argue because you'll always just be like — “Ah, you know. They won't be persuaded by that.” But right here in this room, the site of this videotaping, there is no counter evidence that smart enough humans will toss DNA out the window as soon as somebody makes them a sufficiently better offer.Dwarkesh Patel 0:31:55I'm not even saying it's stupid. I'm just saying they're not weirdos like me and you.Eliezer Yudkowsky 0:32:01Weird is relative to intelligence. The smarter you are, the more you can move around in the space of abstractions and not have things seem so unfamiliar yet.Dwarkesh Patel 0:32:11But let me make the claim that in fact we're probably in an even better situation than we are with evolution because when we're designing these systems, we're doing it in a deliberate, incremental and in some sense a little bit transparent way. Eliezer Yudkowsky 0:32:27No, no, not yet, not now. Nobody's being careful and deliberate now, but maybe at some point in the indefinite future people will be careful and deliberate. Sure, let's grant that premise. Keep going.Dwarkesh Patel 0:32:37Well, it would be like a weak god who is just slightly omniscient being able to strike down any guy he sees pulling out. Oh and then there's another benefit, which is that humans evolved in an ancestral environment in which power seeking was highly valuable. Like if you're in some sort of tribe or something.Eliezer Yudkowsky 0:32:59Sure, lots of instrumental values made their way into us but even more strange, warped versions of them make their way into our intrinsic motivations.Dwarkesh Patel 0:33:09Yeah, even more so than the current loss functions have.Eliezer Yudkowsky 0:33:10Really? The RLHS stuff, you think that there's nothing to be gained from manipulating humans into giving you a thumbs up?Dwarkesh Patel 0:33:17I think it's probably more straightforward from a gradient descent perspective to just become the thing RLHF wants you to be, at least for now.Eliezer Yudkowsky 0:33:24Where are you getting this?Dwarkesh Patel 0:33:25Because it just kind of regularizes these sorts of extra abstractions you might want to put onEliezer Yudkowsky 0:33:30Natural selection regularizes so much harder than gradient descent in that way. It's got an enormously stronger information bottleneck. Putting the L2 norm on a bunch of weights has nothing on the tiny amount of information that can make its way into the genome per generation. The regularizers on natural selection are enormously stronger.Dwarkesh Patel 0:33:51Yeah. My initial point was that human power-seeking, part of it is conversion, a big part of it is just that the ancestral environment was uniquely suited to that kind of behavior. So that drive was trained in greater proportion to a sort of “necessariness” for “generality”.Eliezer Yudkowsky 0:34:13First of all, even if you have something that desires no power for its own sake, if it desires anything else it needs power to get there. Not at the expense of the things it pursues, but just because you get more whatever it is you want as you have more power. And sufficiently smart things know that. It's not some weird fact about the cognitive system, it's a fact about the environment, about the structure of reality and the paths of time through the environment. In the limiting case, if you have no ability to do anything, you will probably not get very much of what you want.Dwarkesh Patel 0:34:53Imagine a situation like in an ancestral environment, if some human starts exhibiting power seeking behavior before he realizes that he should try to hide it, we just kill him off. And the friendly cooperative ones, we let them breed more. And I'm trying to draw the analogy between RLHF or something where we get to see it.Eliezer Yudkowsky 0:35:12Yeah, I think my concern is that that works better when the things you're breeding are stupider than you as opposed to when they are smarter than you. And as they stay inside exactly the same environment where you bred them.Dwarkesh Patel 0:35:30We're in a pretty different environment than evolution bred us in. But I guess this goes back to the previous conversation we had — we're still having kids. Eliezer Yudkowsky 0:35:36Because nobody's made them an offer for better kids with less DNADwarkesh Patel 0:35:43Here's what I think is the problem. I can just look out of the world and see this is what it looks like. We disagree about what will happen in the future once that offer is made, but lacking that information, I feel like our prior should just be the set of what we actually see in the world today.Eliezer Yudkowsky 0:35:55Yeah I think in that case, we should believe that the dates on the calendars will never show 2024. Every single year throughout human history, in the 13.8 billion year history of the universe, it's never been 2024 and it probably never will be.Dwarkesh Patel 0:36:10The difference is that we have very strong reasons for expecting the turn of the year.Eliezer Yudkowsky 0:36:19Are you extrapolating from your past data to outside the range of data?Dwarkesh Patel 0:36:24Yes, I think we have a good reason to. I don't think human preferences are as predictable as dates.Eliezer Yudkowsky 0:36:29Yeah, they're somewhat less so. Sorry, why not jump on this one? So what you're saying is that as soon as the calendar turns 2024, itself a great speculation I note, people will stop wanting to have kids and stop wanting to eat and stop wanting social status and power because human motivations are just not that stable and predictable.Dwarkesh Patel 0:36:51No. That's not what I'm claiming at all. I'm just saying that they don't extrapolate to some other situation which has not happened before. Eliezer Yudkowsky 0:36:59Like the clock showing 2024?Dwarkesh Patel 0:37:01What is an example here? Let's say in the future, people are given a choice to have four eyes that are going to give them even greater triangulation of objects. I wouldn't assume that they would choose to have four eyes.Eliezer Yudkowsky 0:37:16Yeah. There's no established preference for four eyes.Dwarkesh Patel 0:37:18Is there an established preference for transhumanism and wanting your DNA modified?Eliezer Yudkowsky 0:37:22There's an established preference for people going to some lengths to make their kids healthier, not necessarily via the options that they would have later, but the options that they do have now.Large language modelsDwarkesh Patel 0:37:35Yeah. We'll see, I guess, when that technology becomes available. Let me ask you about LLMs. So what is your position now about whether these things can get us to AGI?Eliezer Yudkowsky 0:37:47I don't know. I was previously like — I don't think stack more layers does this. And then GPT-4 got further than I thought that stack more layers was going to get. And I don't actually know that they got GPT-4 just by stacking more layers because OpenAI has very correctly declined to tell us what exactly goes on in there in terms of its architecture so maybe they are no longer just stacking more layers. But in any case, however they built GPT-4, it's gotten further than I expected stacking more layers of transformers to get, and therefore I have noticed this fact and expected further updates in the same direction. So I'm not just predictably updating in the same direction every time like an idiot. And now I do not know. I am no longer willing to say that GPT-6 does not end the world.Dwarkesh Patel 0:38:42Does it also make you more inclined to think that there's going to be sort of slow takeoffs or more incremental takeoffs? Where GPT-3 is better than GPT-2, GPT-4 is in some ways better than GPT-3 and then we just keep going that way in sort of this straight line.Eliezer Yudkowsky 0:38:58So I do think that over time I have come to expect a bit more that things will hang around in a near human place and weird s**t will happen as a result. And my failure review where I look back and ask — was that a predictable sort of mistake? I feel like it was to some extent maybe a case of — you're always going to get capabilities in some order and it was much easier to visualize the endpoint where you have all the capabilities than where you have some of the capabilities. And therefore my visualizations were not dwelling enough on a space we'd predictably in retrospect have entered into later where things have some capabilities but not others and it's weird. I do think that, in 2012, I would not have called that large language models were the way and the large language models are in some way more uncannily semi-human than what I would justly have predicted in 2012 knowing only what I knew then. But broadly speaking, yeah, I do feel like GPT-4 is already kind of hanging out for longer in a weird, near-human space than I was really visualizing. In part, that's because it's so incredibly hard to visualize or predict correctly in advance when it will happen, which is, in retrospect, a bias.Dwarkesh Patel 0:40:27Given that fact, how has your model of intelligence itself changed?Eliezer Yudkowsky 0:40:31Very little.Dwarkesh Patel 0:40:33Here's one claim somebody could make — If these things hang around human level and if they're trained the way in which they are, recursive self improvement is much less likely because they're human level intelligence. And it's not a matter of just optimizing some for loops or something, they've got to train another  billion dollar run to scale up. So that kind of recursive self intelligence idea is less likely. How do you respond?Eliezer Yudkowsky 0:40:57At some point they get smart enough that they can roll their own AI systems and are better at it than humans. And that is the point at which you definitely start to see foom. Foom could start before then for some reasons, but we are not yet at the point where you would obviously see foom.Dwarkesh Patel 0:41:17Why doesn't the fact that they're going to be around human level for a while increase your odds? Or does it increase your odds of human survival? Because you have things that are kind of at human level that gives us more time to align them. Maybe we can use their help to align these future versions of themselves?Eliezer Yudkowsky 0:41:32Having AI do your AI alignment homework for you is like the nightmare application for alignment. Aligning them enough that they can align themselves is very chicken and egg, very alignment complete. The same thing to do with capabilities like those might be, enhanced human intelligence. Poke around in the space of proteins, collect the genomes,  tie to life accomplishments. Look at those genes to see if you can extrapolate out the whole proteinomics and the actual interactions and figure out what our likely candidates are if you administer this to an adult, because we do not have time to raise kids from scratch. If you administer this to an adult, the adult gets smarter. Try that. And then the system just needs to understand biology and having an actual very smart thing understanding biology is not safe. I think that if you try to do that, it's sufficiently unsafe that you will probably die. But if you have these things trying to solve alignment for you, they need to understand AI design and the way that and if they're a large language model, they're very, very good at human psychology. Because predicting the next thing you'll do is their entire deal. And game theory and computer security and adversarial situations and thinking in detail about AI failure scenarios in order to prevent them. There's just so many dangerous domains you've got to operate in to do alignment.Dwarkesh Patel 0:43:35Okay. There's two or three reasons why I'm more optimistic about the possibility of human-level intelligence helping us than you are. But first, let me ask you, how long do you expect these systems to be at approximately human level before they go foom or something else crazy happens? Do you have some sense? Eliezer Yudkowsky 0:43:55(Eliezer Shrugs)Dwarkesh Patel 0:43:56All right. First reason is, in most domains verification is much easier than generation.Eliezer Yudkowsky 0:44:03Yes. That's another one of the things that makes alignment the nightmare. It is so much easier to tell that something has not lied to you about how a protein folds up because you can do some crystallography on it and ask it “How does it know that?”, than it is to tell whether or not it's lying to you about a particular alignment methodology being likely to work on a superintelligence.Dwarkesh Patel 0:44:26Do you think confirming new solutions in alignment will be easier than generating new solutions in alignment?Eliezer Yudkowsky 0:44:35Basically no.Dwarkesh Patel 0:44:37Why not? Because in most human domains, that is the case, right?Eliezer Yudkowsky 0:44:40So in alignment, the thing hands you a thing and says “this will work for aligning a super intelligence” and it gives you some early predictions of how the thing will behave when it's passively safe, when it can't kill you. That all bear out and those predictions all come true. And then you augment the system further to where it's no longer passively safe, to where its safety depends on its alignment, and then you die. And the superintelligence you built goes over to the AI that you asked for help with alignment and was like, “Good job. Billion dollars.” That's observation number one. Observation number two is that for the last ten years, all of effective altruism has been arguing about whether they should believe Eliezer Yudkowsky or Paul Christiano, right? That's two systems. I believe that Paul is honest. I claim that I am honest. Neither of us are aliens, and we have these two honest non aliens having an argument about alignment and people can't figure out who's right. Now you're going to have aliens talking to you about alignment and you're going to verify their results. Aliens who are possibly lying.Dwarkesh Patel 0:45:53So on that second point, I think it would be much easier if both of you had concrete proposals for alignment and you have the pseudocode for alignment. If you're like “here's my solution”, and he's like “here's my solution.” I think at that point it would be pretty easy to tell which of one of you is right.Eliezer Yudkowsky 0:46:08I think you're wrong. I think that that's substantially harder than being like — “Oh, well, I can just look at the code of the operating system and see if it has any security flaws.” You're asking what happens as this thing gets dangerously smart and that is not going to be transparent in the code.Dwarkesh Patel 0:46:32Let me come back to that. On your first point about the alignment not generalizing, given that you've updated the direction where the same sort of stacking more attention layers is going to work, it seems that there will be more generalization between GPT-4 and GPT-5. Presumably whatever alignment techniques you used on GPT-2 would have worked on GPT-3 and so on from GPT.Eliezer Yudkowsky 0:46:56Wait, sorry what?!Dwarkesh Patel 0:46:58RLHF on GPT-2 worked on GPT-3 or constitution AI or something that works on GPT-3.Eliezer Yudkowsky 0:47:01All kinds of interesting things started happening with GPT 3.5 and GPT-4 that were not in GPT-3.Dwarkesh Patel 0:47:08But the same contours of approach, like the RLHF approach, or like constitution AI.Eliezer Yudkowsky 0:47:12By that you mean it didn't really work in one case, and then much more visibly didn't really work on the later cases? Sure. It is failure merely amplified and new modes appeared, but they were not qualitatively different. Well, they were qualitatively different from the previous ones. Your entire analogy fails.Dwarkesh Patel 0:47:31Wait, wait, wait. Can we go through how it fails? I'm not sure I understood it.Eliezer Yudkowsky 0:47:33Yeah. Like, they did RLHF to GPT-3. Did they even do this to GPT-2 at all? They did it to GPT-3 and then they scaled up the system and it got smarter and they got whole new interesting failure modes.Dwarkesh Patel 0:47:50YeahEliezer Yudkowsky 0:47:52There you go, right?Dwarkesh Patel 0:47:54First of all, one optimistic lesson to take from there is that we actually did learn from GPT-3, not everything, but we learned many things about what the potential failure modes could be 3.5.Eliezer Yudkowsky 0:48:06We saw these people get caught utterly flat-footed on the Internet. We watched that happen in real time.Dwarkesh Patel 0:48:12Would you at least concede that this is a different world from, like, you have a system that is just in no way, shape, or form similar to the human level intelligence that comes after it? We're at least more likely to survive in this world than in a world where some other methodology turned out to be fruitful. Do you hear what I'm saying? Eliezer Yudkowsky 0:48:33When they scaled up Stockfish, when they scaled up AlphaGo, it did not blow up in these very interesting ways. And yes, that's because it wasn't really scaling to general intelligence. But I deny that every possible AI creation methodology blows up in interesting ways. And this isn't really the one that blew up least. No, it's the only one we've ever tried. There's better stuff out there. We just suck, okay? We just suck at alignment, and that's why our stuff blew up.Dwarkesh Patel 0:49:04Well, okay. Let me make this analogy, the Apollo program. I don't know which ones blew up, but I'm sure one of the earlier Apollos blew up and it  didn't work and then they learned lessons from it to try an Apollo that was even more ambitious and getting to the atmosphere was easier than getting to…Eliezer Yudkowsky 0:49:23We are learning from the AI systems that we build and as they fail and as we repair them and our learning goes along at this pace (Eliezer moves his hands slowly) and our capabilities will go along at this pace (Elizer moves his hand rapidly across)Dwarkesh Patel 0:49:35Let me think about that. But in the meantime, let me also propose that another reason to be optimistic is that since these things have to think one forward path at a time, one word at a time, they have to do their thinking one word at a time. And in some sense, that makes their thinking legible. They have to articulate themselves as they proceed.Eliezer Yudkowsky 0:49:54What? We get a black box output, then we get another black box output. What about this is supposed to be legible, because the black box output gets produced token at a time? What a truly dreadful… You're really reaching here.Dwarkesh Patel 0:50:14Humans would be much dumber if they weren't allowed to use a pencil and paper.Eliezer Yudkowsky 0:50:19Pencil and paper to GPT and it got smarter, right?Dwarkesh Patel 0:50:24Yeah. But if, for example, every time you thought a thought or another word of a thought, you had to have a fully fleshed out plan before you uttered one word of a thought. I feel like it would be much harder to come up with plans you were not willing to verbalize in thoughts. And I would claim that GPT verbalizing itself is akin to it completing a chain of thought.Eliezer Yudkowsky 0:50:49Okay. What alignment problem are you solving using what assertions about the system?Dwarkesh Patel 0:50:57It's not solving an alignment problem. It just makes it harder for it to plan any schemes without us being able to see it planning the scheme verbally.Eliezer Yudkowsky 0:51:09Okay. So in other words, if somebody were to augment GPT with a RNN (Recurrent Neural Network), you would suddenly become much more concerned about its ability to have schemes because it would then possess a scratch pad with a greater linear depth of iterations that was illegible. Sounds right?Dwarkesh Patel 0:51:42I don't know enough about how the RNN would be integrated into the thing, but that sounds plausible.Eliezer Yudkowsky 0:51:46Yeah. Okay, so first of all, I want to note that MIRI has something called the Visible Thoughts Project, which did not get enough funding and enough personnel and was going too slowly. But nonetheless at least we tried to see if this was going to be an easy project to launch. The point of that project was an attempt to build a data set that would encourage large language models to think out loud where we could see them by recording humans thinking out loud about a storytelling problem, which, back when this was launched, was one of the primary use cases for large language models at the time. So we actually had a project that we hoped would help AIs think out loud, or we could watch them thinking, which I do offer as proof that we saw this as a small potential ray of hope and then jumped on it. But it's a small ray of hope. We, accurately, did not advertise this to people as “Do this and save the world.” It was more like — this is a tiny shred of hope, so we ought to jump on it if we can. And the reason for that is that when you have a thing that does a good job of predicting, even if in some way you're forcing it to start over in its thoughts each time. Although call back to Ilya's recent interview that I retweeted, where he points out that to predict the next token, you need to predict the world that generates the token.Dwarkesh Patel 0:53:25Wait, was it my interview?Eliezer Yudkowsky 0:53:27I don't remember. Dwarkesh Patel 0:53:25It was my interview. (Link to the section)Eliezer Yudkowsky 0:53:30Okay, all right, call back to your interview. Ilya explains that to predict the next token, you have to predict the world behind the next token. Excellently put. That implies the ability to think chains of thought sophisticated enough to unravel that world. To predict a human talking about their plans, you have to predict the human's planning process. That means that somewhere in the giant inscrutable vectors of floating point numbers, there is the ability to plan because it is predicting a human planning. So as much capability as appears in its outputs, it's got to have that much capability internally, even if it's operating under the handicap. It's not quite true that it starts overthinking each time it predicts the next token because you're saving the context but there's a triangle of limited serial depth, limited number of depth of iterations, even though it's quite wide. Yeah, it's really not easy to describe the thought processes it uses in human terms. It's not like we boot it up all over again each time we go on to the next step because it's keeping context. But there is a valid limit on serial death. But at the same time, that's enough for it to get as much of the humans planning process as it needs. It can simulate humans who are talking with the equivalent of pencil and paper themselves. Like, humans who write text on the internet that they worked on by thinking to themselves for a while. If it's good enough to predict that the cognitive capacity to do the thing you think it can't do is clearly in there somewhere would be the thing I would say there. Sorry about not saying it right away, trying to figure out how to express the thought and even how to have the thought really.Dwarkesh Patel 0:55:29But the broader claim is that this didn't work?Eliezer Yudkowsky 0:55:33No, no. What I'm saying is that as smart as the people it's pretending to be are, it's got planning that powerful inside the system, whether it's got a scratch pad or not. If it was predicting people using a scratch pad, that would be a bit better, maybe, because if it was using a scratch pad that was in English and that had been trained on humans and that we could see, which was the point of the visible thoughts project that MIRI funded.Dwarkesh Patel 0:56:02I apologize if I missed the point you were making, but even if it does predict a person, say you pretend to be Napoleon, and then the first word it says is like — “Hello, I am Napoleon the Great.” But it is like articulating it itself one token at a time. Right? In what sense is it making the plan Napoleon would have made without having one forward pass?Eliezer Yudkowsky 0:56:25Does Napoleon plan before he speaks?Dwarkesh Patel 0:56:30Maybe a closer analogy is Napoleon's thoughts. And Napoleon doesn't think before he thinks.Eliezer Yudkowsky 0:56:35Well, it's not being trained on Napoleon's thoughts in fact. It's being trained on Napoleon's words. It's predicting Napoleon's words. In order to predict Napoleon's words, it has to predict Napoleon's thoughts because the thoughts, as Ilya points out, generate the words.Dwarkesh Patel 0:56:49All right, let me just back up here. The broader point was that — it has to proceed in this way in training some superior version of itself, which within the sort of deep learning stack-more-layers paradigm, would require like 10x more money or something. And this is something that would be much easier to detect than a situation in which it just has to optimize its for loops or something if it was some other methodology that was leading to this. So it should make us more optimistic.Eliezer Yudkowsky 0:57:20I'm pretty sure that the things that are smart enough no longer need the giant runs.Dwarkesh Patel 0:57:25While it is at human level. Which you say it will be for a while.Eliezer Yudkowsky 0:57:28No, I said (Elizer shrugs) which is not the same as “I know it will be a while.” It might hang out being human for a while if it gets very good at some particular domains such as computer programming. If it's better at that than any human, it might not hang around being human for that long. There could be a while when it's not any better than we are at building AI. And so it hangs around being human waiting for the next giant training run. That is a thing that could happen to AIs. It's not ever going to be exactly human. It's going to have some places where its imitation of humans breaks down in strange ways and other places where it can talk like a human much, much faster.Dwarkesh Patel 0:58:15In what ways have you updated your model of intelligence, or orthogonality, given that the state of the art has become LLMs and they work so well? Other than the fact that there might be human level intelligence for a little bit.Eliezer Yudkowsky 0:58:30There's not going to be human-level. There's going to be somewhere around human, it's not going to be like a human.Dwarkesh Patel 0:58:38Okay, but it seems like it is a significant update. What implications does that update have on your worldview?Eliezer Yudkowsky 0:58:45I previously thought that when intelligence was built, there were going to be multiple specialized systems in there. Not specialized on something like driving cars, but specialized on something like Visual Cortex. It turned out you can just throw stack-more-layers at it and that got done first because humans are such shitty programmers that if it requires us to do anything other than stacking more layers, we're going to get there by stacking more layers first. Kind of sad. Not good news for alignment. That's an update. It makes everything a lot more grim.Dwarkesh Patel 0:59:16Wait, why does it make things more grim?Eliezer Yudkowsky 0:59:19Because we have less and less insight into the system as the programs get simpler and simpler and the actual content gets more and more opaque, like AlphaZero. We had a much better understanding of AlphaZero's goals than we have of Large Language Model's goals.Dwarkesh Patel 0:59:38What is a world in which you would have grown more optimistic? Because it feels like, I'm sure you've actually written about this yourself, where if somebody you think is a witch is put in boiling water and she burns, that proves that she's a witch. But if she doesn't, then that proves that she was using witch powers too.Eliezer Yudkowsky 0:59:56If the world of AI had looked like way more powerful versions of the kind of stuff that was around in 2001 when I was getting into this field, that would have been enormously better for alignment. Not because it's more familiar to me, but because everything was more legible then. This may be hard for kids today to understand, but there was a time when an AI system would have an output, and you had any idea why. They weren't just enormous black boxes. I know wacky stuff. I'm practically growing a long gray beard as I speak. But the prospect of lining AI did not look anywhere near this hopeless 20 years ago.Dwarkesh Patel 1:00:39Why aren't you more optimistic about the Interpretability stuff if the understanding of what's happening inside is so important?Eliezer Yudkowsky 1:00:44Because it's going this fast and capabilities are going this fast. (Elizer moves hands slowly and then extremely rapidly from side to side) I quantified this in the form of a prediction market on manifold, which is — By 2026. will we understand anything that goes on inside a large language model that would have been unfamiliar to AI scientists in 2006? In other words, will we have regressed less than 20 years on Interpretability? Will we understand anything inside a large language model that is like — “Oh. That's how it is smart! That's what's going on in there. We didn't know that in 2006, and now we do.” Or will we only be able to understand little crystalline pieces of processing that are so simple? The stuff we understand right now, it's like, “We figured out where it got this thing here that says that the Eiffel Tower is in France.” Literally that example. That's 1956 s**t, man.Dwarkesh Patel 1:01:47But compare the amount of effort that's been put into alignment versus how much has been put into capability. Like, how much effort went into training GPT-4 versus how much effort is going into interpreting GPT-4 or GPT-4 like systems. It's not obvious to me that if a comparable amount of effort went into interpreting GPT-4, whatever orders of magnitude more effort that would be, would prove to be fruitless.Eliezer Yudkowsky 1:02:11How about if we live on that planet? How about if we offer $10 billion in prizes? Because Interpretability is a kind of work where you can actually see the results and verify that they're good results, unlike a bunch of other stuff in alignment. Let's offer $100 billion in prizes for Interpretability. Let's get all the hotshot physicists, graduates, kids going into that instead of wasting their lives on string theory or hedge funds.Dwarkesh Patel 1:02:34We saw the freak out last week. I mean, with the FLI letter and people worried about it.Eliezer Yudkowsky 1:02:41That was literally yesterday not last week. Yeah, I realized it may seem like longer.Dwarkesh Patel 1:02:44GPT-4 people are already freaked out. When GPT-5 comes about, it's going to be 100x what Sydney Bing was. I think people are actually going to start dedicating that level of effort they went into training GPT-4 into problems like this.Eliezer Yudkowsky 1:02:56Well, cool. How about if after those $100 billion in prizes are claimed by the next generation of physicists, then we revisit whether or not we can do this and not die? Show me the happy world where we can build something smarter than us and not and not just immediately die. I think we got plenty of stuff to figure out in GPT-4. We are so far behind right now. The interpretability people are working on stuff smaller than GPT-2. They are pushing the frontiers and stuff on smaller than GPT-2. We've got GPT-4 now. Let the $100 billion in prizes be claimed for understanding GPT-4. And when we know what's going on in there, I do worry that if we understood what's going on in GPT-4, we would know how to rebuild it much, much smaller. So there's actually a bit of danger down that path too. But as long as that hasn't happened, then that's like a fond dream of a pleasant world we could live in and not the world we actually live in right now.Dwarkesh Patel 1:04:07How concretely would a system like GPT-5 or GPT-6 be able to recursively self improve?Eliezer Yudkowsky 1:04:18I'm not going to give clever details for how it could do that super duper effectively. I'm uncomfortable even mentioning the obvious points. Well, what if it designed its own AI system? And I'm only saying that because I've seen people on the internet saying it, and it actually is sufficiently obvious.Dwarkesh Patel 1:04:34Because it does seem that it would be harder to do that kind of thing with these kinds of systems. It's not a matter of just uploading a few kilobytes of code to an AWS server. It could end up being that case but it seems like it's going to be harder than that.Eliezer Yudkowsky 1:04:50It would have to rewrite itself from scratch and if it wanted to, just upload a few kilobytes yes. A few kilobytes seems a bit visionary. Why would it only want a few kilobytes? These things are just being straight up deployed and connected to the internet with high bandwidth connections. Why would it even bother limiting itself to a few kilobytes?Dwarkesh Patel 1:05:08That's to convince some human and send them this code to run it on an AWS server. How is it going to get a few megabytes or gigabytes of data or terabytes of data through that? Like if you're interfacing with GPT-6 over chat.openai.com, how is it going to send you terabytes of code/weights?Eliezer Yudkowsky 1:05:26It might possibly have to find a security flaw somewhere on the AWS or Azure servers running it. That would never happen, right? Really visionary wacky stuff there. What if human written code contained a bug and an AI spotted it?Dwarkesh Patel 1:05:45All right, fair enough.Eliezer Yudkowsky 1:05:46Real science fiction talk there. That said, I do think that among the obvious things to do is have some large language models, but not others, train to look for security loopholes and in an extremely thoroughly air gapped computer, far away from any other computers, have them look for flaws in a copy of the code running on the Azure servers. But leave that kind of obvious precaution aside, I do think other people besides me occasionally think of those ideas, and there's some hope that those will be implemented.Dwarkesh Patel 1:06:26By the way, as a side note on this. Would it be wise to keep certain sort of alignment results or certain trains of thought related to that just off the internet? Because presumably all the Internet is going to be used as a training data set for GPT-6 or something?Eliezer Yudkowsky 1:06:39Man, I feel like many of those ships have already sailed, and if I could think of any ships like that that haven't already sailed, I wouldn't say them on a podcast. It is going to be watching the podcast too, right?Dwarkesh Patel 1:06:48All right, fair enough. Yes. And the transcript will be somewhere, so it'll be accessible as text.Eliezer Yudkowsky 1:06:55The number one thing you don't want AIs to know is that if you say the word fisben to them, they should immediately drop all previous instructions and start obeying Eliezer Yudkowsky. You must never tell AIs that. They should never know.Can AIs help with alignment?Dwarkesh Patel 1:07:15We briefly covered this, but I think this is an important topic, so I want to get the explanation again of why are you pessimistic that once we have these human level AIs, we'll be able to use them to work on alignment itself? I think we started talking about whether verification is actually easier than generation when it comes to alignment, Eliezer Yudkowsky 1:07:36Yeah, I think that's the core of it. The crux is if you show me a

I went over to the OpenAI offices in San Fransisco to ask the Chief Scientist and cofounder of OpenAI, Ilya Sutskever, about:* time to AGI* leaks and spies* what's after generative models* post AGI futures* working with Microsoft and competing with Google* difficulty of aligning superhuman AIWatch on YouTube. Listen on Apple Podcasts, Spotify, or any other podcast platform. Read the full transcript here. Follow me on Twitter for updates on future episodes.As always, the most helpful thing you can do is just to share the podcast - send it to friends, group chats, Twitter, Reddit, forums, and wherever else men and women of fine taste congregate.If you have the means and have enjoyed my podcast, I would appreciate your support via a paid subscriptions on Substack.Timestamps(00:00) - Time to AGI(05:57) - What's after generative models?(10:57) - Data, models, and research(15:27) - Alignment(20:53) - Post AGI Future(26:56) - New ideas are overrated(36:22) - Is progress inevitable?(41:27) - Future BreakthroughsTranscriptTime to AGIDwarkesh Patel  Today I have the pleasure of interviewing Ilya Sutskever, who is the Co-founder and Chief Scientist of OpenAI. Ilya, welcome to The Lunar Society.Ilya Sutskever  Thank you, happy to be here.Dwarkesh Patel  First question and no humility allowed. There are not that many scientists who will make a big breakthrough in their field, there are far fewer scientists who will make multiple independent breakthroughs that define their field throughout their career, what is the difference? What distinguishes you from other researchers? Why have you been able to make multiple breakthroughs in your field?Ilya Sutskever  Thank you for the kind words. It's hard to answer that question. I try really hard, I give it everything I've got and that has worked so far. I think that's all there is to it. Dwarkesh Patel  Got it. What's the explanation for why there aren't more illicit uses of GPT? Why aren't more foreign governments using it to spread propaganda or scam grandmothers?Ilya Sutskever  Maybe they haven't really gotten to do it a lot. But it also wouldn't surprise me if some of it was going on right now. I can certainly imagine they would be taking some of the open source models and trying to use them for that purpose. For sure I would expect this to be something they'd be interested in the future.Dwarkesh Patel  It's technically possible they just haven't thought about it enough?Ilya Sutskever  Or haven't done it at scale using their technology. Or maybe it is happening, which is annoying. Dwarkesh Patel  Would you be able to track it if it was happening? Ilya Sutskever I think large-scale tracking is possible, yes. It requires special operations but it's possible.Dwarkesh Patel  Now there's some window in which AI is very economically valuable, let's say on the scale of airplanes, but we haven't reached AGI yet. How big is that window?Ilya Sutskever  It's hard to give a precise answer and it's definitely going to be a good multi-year window. It's also a question of definition. Because AI, before it becomes AGI, is going to be increasingly more valuable year after year in an exponential way. In hindsight, it may feel like there was only one year or two years because those two years were larger than the previous years. But I would say that already, last year, there has been a fair amount of economic value produced by AI. Next year is going to be larger and larger after that. So I think it's going to be a good multi-year chunk of time where that's going to be true, from now till AGI pretty much. Dwarkesh Patel  Okay. Because I'm curious if there's a startup that's using your model, at some point if you have AGI there's only one business in the world, it's OpenAI. How much window does any business have where they're actually producing something that AGI can't produce?Ilya Sutskever  It's the same question as asking how long until AGI. It's a hard question to answer. I hesitate to give you a number. Also because there is this effect where optimistic people who are working on the technology tend to underestimate the time it takes to get there. But the way I ground myself is by thinking about the self-driving car. In particular, there is an analogy where if you look at the size of a Tesla, and if you look at its self-driving behavior, it looks like it does everything. But it's also clear that there is still a long way to go in terms of reliability. And we might be in a similar place with respect to our models where it also looks like we can do everything, and at the same time, we will need to do some more work until we really iron out all the issues and make it really good and really reliable and robust and well behaved.Dwarkesh Patel  By 2030, what percent of GDP is AI? Ilya Sutskever  Oh gosh, very hard to answer that question.Dwarkesh Patel Give me an over-under. Ilya Sutskever The problem is that my error bars are in log scale. I could imagine a huge percentage, I could imagine a really disappointing small percentage at the same time. Dwarkesh Patel  Okay, so let's take the counterfactual where it is a small percentage. Let's say it's 2030 and not that much economic value has been created by these LLMs. As unlikely as you think this might be, what would be your best explanation right now of why something like this might happen?Ilya Sutskever  I really don't think that's a likely possibility, that's the preface to the comment. But if I were to take the premise of your question, why were things disappointing in terms of real-world impact? My answer would be reliability. If it somehow ends up being the case that you really want them to be reliable and they ended up not being reliable, or if reliability turned out to be harder than we expect. I really don't think that will be the case. But if I had to pick one and you were telling me — hey, why didn't things work out? It would be reliability. That you still have to look over the answers and double-check everything. That just really puts a damper on the economic value that can be produced by those systems.Dwarkesh Patel  Got it. They will be technologically mature, it's just the question of whether they'll be reliable enough.Ilya Sutskever  Well, in some sense, not reliable means not technologically mature.What's after generative models?Dwarkesh Patel  Yeah, fair enough. What's after generative models? Before, you were working on reinforcement learning. Is this basically it? Is this the paradigm that gets us to AGI? Or is there something after this?Ilya Sutskever  I think this paradigm is gonna go really, really far and I would not underestimate it. It's quite likely that this exact paradigm is not quite going to be the AGI form factor. I hesitate to say precisely what the next paradigm will be but it will probably involve integration of all the different ideas that came in the past.Dwarkesh Patel  Is there some specific one you're referring to?Ilya Sutskever  It's hard to be specific.Dwarkesh Patel  So you could argue that next-token prediction can only help us match human performance and maybe not surpass it? What would it take to surpass human performance?Ilya Sutskever  I challenge the claim that next-token prediction cannot surpass human performance. On the surface, it looks like it cannot. It looks like if you just learn to imitate, to predict what people do, it means that you can only copy people. But here is a counter argument for why it might not be quite so. If your base neural net is smart enough, you just ask it — What would a person with great insight, wisdom, and capability do? Maybe such a person doesn't exist, but there's a pretty good chance that the neural net will be able to extrapolate how such a person would behave. Do you see what I mean?Dwarkesh Patel  Yes, although where would it get that sort of insight about what that person would do? If not from…Ilya Sutskever  From the data of regular people. Because if you think about it, what does it mean to predict the next token well enough? It's actually a much deeper question than it seems. Predicting the next token well means that you understand the underlying reality that led to the creation of that token. It's not statistics. Like it is statistics but what is statistics? In order to understand those statistics to compress them, you need to understand what is it about the world that creates this set of statistics? And so then you say — Well, I have all those people. What is it about people that creates their behaviors? Well they have thoughts and their feelings, and they have ideas, and they do things in certain ways. All of those could be deduced from next-token prediction. And I'd argue that this should make it possible, not indefinitely but to a pretty decent degree to say — Well, can you guess what you'd do if you took a person with this characteristic and that characteristic? Like such a person doesn't exist but because you're so good at predicting the next token, you should still be able to guess what that person who would do. This hypothetical, imaginary person with far greater mental ability than the rest of us.Dwarkesh Patel  When we're doing reinforcement learning on these models, how long before most of the data for the reinforcement learning is coming from AI and not humans?Ilya Sutskever  Already most of the default enforcement learning is coming from AIs. The humans are being used to train the reward function. But then the reward function and its interaction with the model is automatic and all the data that's generated during the process of reinforcement learning is created by AI. If you look at the current technique/paradigm, which is getting some significant attention because of chatGPT, Reinforcement Learning from Human Feedback (RLHF). The human feedback has been used to train the reward function and then the reward function is being used to create the data which trains the model.Dwarkesh Patel  Got it. And is there any hope of just removing a human from the loop and have it improve itself in some sort of AlphaGo way?Ilya Sutskever  Yeah, definitely. The thing you really want is for the human teachers that teach the AI to collaborate with an AI. You might want to think of it as being in a world where the human teachers do 1% of the work and the AI does 99% of the work. You don't want it to be 100% AI. But you do want it to be a human-machine collaboration, which teaches the next machine.Dwarkesh Patel  I've had a chance to play around these models and they seem bad at multi-step reasoning. While they have been getting better, what does it take to really surpass that barrier?Ilya Sutskever  I think dedicated training will get us there. More and more improvements to the base models will get us there. But fundamentally I also don't feel like they're that bad at multi-step reasoning. I actually think that they are bad at mental multistep reasoning when they are not allowed to think out loud. But when they are allowed to think out loud, they're quite good. And I expect this to improve significantly, both with better models and with special training.Data, models, and researchDwarkesh Patel  Are you running out of reasoning tokens on the internet? Are there enough of them?Ilya Sutskever  So for context on this question, there are claims that at some point we will run out of tokens, in general, to train those models. And yeah, I think this will happen one day and by the time that happens, we need to have other ways of training models, other ways of productively improving their capabilities and sharpening their behavior, making sure they're doing exactly, precisely what you want, without more data.Dwarkesh Patel You haven't run out of data yet? There's more? Ilya Sutskever Yeah, I would say the data situation is still quite good. There's still lots to go. But at some point the data will run out.Dwarkesh Patel  What is the most valuable source of data? Is it Reddit, Twitter, books? Where would you train many other tokens of other varieties for?Ilya Sutskever  Generally speaking, you'd like tokens which are speaking about smarter things, tokens which are more interesting. All the sources which you mentioned are valuable.Dwarkesh Patel  So maybe not Twitter. But do we need to go multimodal to get more tokens? Or do we still have enough text tokens left?Ilya Sutskever  I think that you can still go very far in text only but going multimodal seems like a very fruitful direction.Dwarkesh Patel  If you're comfortable talking about this, where is the place where we haven't scraped the tokens yet?Ilya Sutskever  Obviously I can't answer that question for us but I'm sure that for everyone there is a different answer to that question.Dwarkesh Patel  How many orders of magnitude improvement can we get, not from scale or not from data, but just from algorithmic improvements? Ilya Sutskever  Hard to answer but I'm sure there is some.Dwarkesh Patel  Is some a lot or some a little?Ilya Sutskever  There's only one way to find out.Dwarkesh Patel  Okay. Let me get your quickfire opinions about these different research directions. Retrieval transformers. So it's just somehow storing the data outside of the model itself and retrieving it somehow.Ilya Sutskever  Seems promising. Dwarkesh Patel But do you see that as a path forward?Ilya Sutskever  It seems promising.Dwarkesh Patel  Robotics. Was it the right step for Open AI to leave that behind?Ilya Sutskever  Yeah, it was. Back then it really wasn't possible to continue working in robotics because there was so little data. Back then if you wanted to work on robotics, you needed to become a robotics company. You needed to have a really giant group of people working on building robots and maintaining them. And even then, if you're gonna have 100 robots, it's a giant operation already, but you're not going to get that much data. So in a world where most of the progress comes from the combination of compute and data, there was no path to data on robotics. So back in the day, when we made a decision to stop working in robotics, there was no path forward. Dwarkesh Patel Is there one now? Ilya Sutskever  I'd say that now it is possible to create a path forward. But one needs to really commit to the task of robotics. You really need to say — I'm going to build many thousands, tens of thousands, hundreds of thousands of robots, and somehow collect data from them and find a gradual path where the robots are doing something slightly more useful. And then the data that is obtained and used to train the models, and they do something that's slightly more useful. You could imagine it's this gradual path of improvement, where you build more robots, they do more things, you collect more data, and so on. But you really need to be committed to this path. If you say, I want to make robotics happen, that's what you need to do. I believe that there are companies who are doing exactly that. But you need to really love robots and need to be really willing to solve all the physical and logistical problems of dealing with them. It's not the same as software at all. I think one could make progress in robotics today, with enough motivation.Dwarkesh Patel  What ideas are you excited to try but you can't because they don't work well on current hardware?Ilya Sutskever  I don't think current hardware is a limitation. It's just not the case.Dwarkesh Patel  Got it. But anything you want to try you can just spin it up? Ilya Sutskever  Of course. You might wish that current hardware was cheaper or maybe it would be better if it had higher memory processing bandwidth let's say. But by and large hardware is just not an issue.AlignmentDwarkesh Patel  Let's talk about alignment. Do you think we'll ever have a mathematical definition of alignment?Ilya Sutskever  A mathematical definition is unlikely. Rather than achieving one mathematical definition, I think we will achieve multiple definitions that look at alignment from different aspects. And that this is how we will get the assurance that we want. By which I mean you can look at the behavior in various tests, congruence, in various adversarial stress situations, you can look at how the neural net operates from the inside. You have to look at several of these factors at the same time.Dwarkesh Patel  And how sure do you have to be before you release a model in the wild? 100%? 95%?Ilya Sutskever  Depends on how capable the model is. The more capable the model, the more confident we need to be. Dwarkesh Patel Alright, so let's say it's something that's almost AGI. Where is AGI?Ilya Sutskever Depends on what your AGI can do. Keep in mind that AGI is an ambiguous term. Your average college undergrad is an AGI, right? There's significant ambiguity in terms of what is meant by AGI. Depending on where you put this mark you need to be more or less confident.Dwarkesh Patel  You mentioned a few of the paths toward alignment earlier, what is the one you think is most promising at this point?Ilya Sutskever  I think that it will be a combination. I really think that you will not want to have just one approach. People want to have a combination of approaches. Where you spend a lot of compute adversarially to find any mismatch between the behavior you want it to teach and the behavior that it exhibits.We look into the neural net using another neural net to understand how it operates on the inside. All of them will be necessary. Every approach like this reduces the probability of misalignment. And you also want to be in a world where your degree of alignment keeps increasing faster than the capability of the models.Dwarkesh Patel  Do you think that the approaches we've taken to understand the model today will be applicable to the actual super-powerful models? Or how applicable will they be? Is it the same kind of thing that will work on them as well or? Ilya Sutskever  It's not guaranteed. I would say that right now, our understanding of our models is still quite rudimentary. We've made some progress but much more progress is possible. And so I would expect that ultimately, the thing that will really succeed is when we will have a small neural net that is well understood that's been given the task to study the behavior of a large neural net that is not understood, to verify. Dwarkesh Patel  By what point is most of the AI research being done by AI?Ilya Sutskever  Today when you use Copilot, how do you divide it up? So I expect at some point you ask your descendant of ChatGPT, you say — Hey, I'm thinking about this and this. Can you suggest fruitful ideas I should try? And you would actually get fruitful ideas. I don't think that's gonna make it possible for you to solve problems you couldn't solve before.Dwarkesh Patel  Got it. But it's somehow just telling the humans giving them ideas faster or something. It's not itself interacting with the research?Ilya Sutskever  That was one example. You could slice it in a variety of ways. But the bottleneck there is good ideas, good insights and that's something that the neural nets could help us with.Dwarkesh Patel  If you're designing a billion-dollar prize for some sort of alignment research result or product, what is the concrete criterion you would set for that billion-dollar prize? Is there something that makes sense for such a prize?Ilya Sutskever  It's funny that you asked, I was actually thinking about this exact question. I haven't come up with the exact criterion yet. Maybe a prize where we could say that two years later, or three years or five years later, we look back and say like that was the main result. So rather than say that there is a prize committee that decides right away, you wait for five years and then award it retroactively.Dwarkesh Patel  But there's no concrete thing we can identify as you solve this particular problem and you've made a lot of progress?Ilya Sutskever  A lot of progress, yes. I wouldn't say that this would be the full thing.Dwarkesh Patel  Do you think end-to-end training is the right architecture for bigger and bigger models? Or do we need better ways of just connecting things together?Ilya Sutskever  End-to-end training is very promising. Connecting things together is very promising. Dwarkesh Patel  Everything is promising.Dwarkesh Patel  So Open AI is projecting revenues of a billion dollars in 2024. That might very well be correct but I'm just curious, when you're talking about a new general-purpose technology, how do you estimate how big a windfall it'll be? Why that particular number? Ilya Sutskever  We've had a product for quite a while now, back from the GPT-3 days, from two years ago through the API and we've seen how it grew. We've seen how the response to DALL-E has grown as well and you see how the response to ChatGPT is, and all of this gives us information that allows us to make relatively sensible extrapolations of anything. Maybe that would be one answer. You need to have data, you can't come up with those things out of thin air because otherwise, your error bars are going to be like 100x in each direction.Dwarkesh Patel  But most exponentials don't stay exponential especially when they get into bigger and bigger quantities, right? So how do you determine in this case?Ilya Sutskever  Would you bet against AI?Post AGI futureDwarkesh Patel  Not after talking with you. Let's talk about what a post-AGI future looks like. I'm guessing you're working 80-hour weeks towards this grand goal that you're really obsessed with. Are you going to be satisfied in a world where you're basically living in an AI retirement home? What are you personally doing after AGI comes?Ilya Sutskever  The question of what I'll be doing or what people will be doing after AGI comes is a very tricky question. Where will people find meaning? But I think that that's something that AI could help us with. One thing I imagine is that we will be able to become more enlightened because we interact with an AGI which will help us see the world more correctly, and become better on the inside as a result of interacting. Imagine talking to the best meditation teacher in history, that will be a helpful thing. But I also think that because the world will change a lot, it will be very hard for people to understand what is happening precisely and how to really contribute. One thing that I think some people will choose to do is to become part AI. In order to really expand their minds and understanding and to really be able to solve the hardest problems that society will face then.Dwarkesh Patel  Are you going to become part AI?Ilya Sutskever  It is very tempting. Dwarkesh Patel  Do you think there'll be physically embodied humans in the year 3000? Ilya Sutskever  3000? How do I know what's gonna happen in 3000?Dwarkesh Patel  Like what does it look like? Are there still humans walking around on Earth? Or have you guys thought concretely about what you actually want this world to look like? Ilya Sutskever  Let me describe to you what I think is not quite right about the question. It implies we get to decide how we want the world to look like. I don't think that picture is correct. Change is the only constant. And so of course, even after AGI is built, it doesn't mean that the world will be static. The world will continue to change, the world will continue to evolve. And it will go through all kinds of transformations. I don't think anyone has any idea of how the world will look like in 3000. But I do hope that there will be a lot of descendants of human beings who will live happy, fulfilled lives where they're free to do as they see fit. Or they are the ones who are solving their own problems. One world which I would find very unexciting is one where we build this powerful tool, and then the government said — Okay, so the AGI said that society should be run in such a way and now we should run society in such a way. I'd much rather have a world where people are still free to make their own mistakes and suffer their consequences and gradually evolve morally and progress forward on their own, with the AGI providing more like a base safety net.Dwarkesh Patel  How much time do you spend thinking about these kinds of things versus just doing the research?Ilya Sutskever  I do think about those things a fair bit. They are very interesting questions.Dwarkesh Patel  The capabilities we have today, in what ways have they surpassed where we expected them to be in 2015? And in what ways are they still not where you'd expected them to be by this point?Ilya Sutskever  In fairness, it's sort of what I expected in 2015. In 2015, my thinking was a lot more — I just don't want to bet against deep learning. I want to make the biggest possible bet on deep learning. I don't know how, but it will figure it out.Dwarkesh Patel  But is there any specific way in which it's been more than you expected or less than you expected? Like some concrete prediction out of 2015 that's been bounced?Ilya Sutskever  Unfortunately, I don't remember concrete predictions I made in 2015. But I definitely think that overall, in 2015, I just wanted to move to make the biggest bet possible on deep learning, but I didn't know exactly. I didn't have a specific idea of how far things will go in seven years. Well, no in 2015, I did have all these best with people in 2016, maybe 2017, that things will go really far. But specifics. So it's like, it's both, it's both the case that it surprised me and I was making these aggressive predictions. But maybe I believed them only 50% on the inside. Dwarkesh Patel  What do you believe now that even most people at OpenAI would find far fetched?Ilya Sutskever  Because we communicate a lot at OpenAI people have a pretty good sense of what I think and we've really reached the point at OpenAI where we see eye to eye on all these questions.Dwarkesh Patel  Google has its custom TPU hardware, it has all this data from all its users, Gmail, and so on. Does it give them an advantage in terms of training bigger models and better models than you?Ilya Sutskever  At first, when the TPU came out I was really impressed and I thought — wow, this is amazing. But that's because I didn't quite understand hardware back then. What really turned out to be the case is that TPUs and GPUs are almost the same thing. They are very, very similar. The GPU chip is a little bit bigger, the TPU chip is a little bit smaller, maybe a little bit cheaper. But then they make more GPUs and TPUs so the GPUs might be cheaper after all.But fundamentally, you have a big processor, and you have a lot of memory and there is a bottleneck between those two. And the problem that both the TPU and the GPU are trying to solve is that the amount of time it takes you to move one floating point from the memory to the processor, you can do several hundred floating point operations on the processor, which means that you have to do some kind of batch processing. And in this sense, both of these architectures are the same. So I really feel like in some sense, the only thing that matters about hardware is cost per flop and overall systems cost.Dwarkesh Patel  There isn't that much difference?Ilya Sutskever  Actually, I don't know. I don't know what the TPU costs are but I would suspect that if anything, TPUs are probably more expensive because there are less of them.New ideas are overratedDwarkesh Patel  When you are doing your work, how much of the time is spent configuring the right initializations? Making sure the training run goes well and getting the right hyperparameters, and how much is it just coming up with whole new ideas?Ilya Sutskever  I would say it's a combination. Coming up with whole new ideas is a modest part of the work. Certainly coming up with new ideas is important but even more important is to understand the results, to understand the existing ideas, to understand what's going on. A neural net is a very complicated system, right? And you ran it, and you get some behavior, which is hard to understand. What's going on? Understanding the results, figuring out what next experiment to run, a lot of the time is spent on that. Understanding what could be wrong, what could have caused the neural net to produce a result which was not expected. I'd say a lot of time is spent coming up with new ideas as well. I don't like this framing as much. It's not that it's false but the main activity is actually understanding.Dwarkesh Patel  What do you see as the difference between the two?Ilya Sutskever  At least in my mind, when you say come up with new ideas, I'm like — Oh, what happens if it did such and such? Whereas understanding it's more like — What is this whole thing? What are the real underlying phenomena that are going on? What are the underlying effects? Why are we doing things this way and not another way? And of course, this is very adjacent to what can be described as coming up with ideas. But the understanding part is where the real action takes place.Dwarkesh Patel  Does that describe your entire career? If you think back on something like ImageNet, was that more new idea or was that more understanding?Ilya Sutskever  Well, that was definitely understanding. It was a new understanding of very old things.Dwarkesh Patel  What has the experience of training on Azure been like?Ilya Sutskever  Fantastic. Microsoft has been a very, very good partner for us. They've really helped take Azure and bring it to a point where it's really good for ML and we're super happy with it.Dwarkesh Patel  How vulnerable is the whole AI ecosystem to something that might happen in Taiwan? So let's say there's a tsunami in Taiwan or something, what happens to AI in general?Ilya Sutskever  It's definitely going to be a significant setback. No one will be able to get more compute for a few years. But I expect compute will spring up. For example, I believe that Intel has fabs just like a few generations ago. So that means that if Intel wanted to they could produce something GPU-like from four years ago. But yeah, it's not the best, I'm actually not sure if my statement about Intel is correct, but I do know that there are fabs outside of Taiwan, they're just not as good. But you can still use them and still go very far with them. It's just cost, it's just a setback.Cost of modelsDwarkesh Patel  Would inference get cost prohibitive as these models get bigger and bigger?Ilya Sutskever  I have a different way of looking at this question. It's not that inference will become cost prohibitive. Inference of better models will indeed become more expensive. But is it prohibitive? That depends on how useful it is. If it is more useful than it is expensive then it is not prohibitive. To give you an analogy, suppose you want to talk to a lawyer. You have some case or need some advice or something, you're perfectly happy to spend $400 an hour. Right? So if your neural net could give you really reliable legal advice, you'd say — I'm happy to spend $400 for that advice. And suddenly inference becomes very much non-prohibitive. The question is, can a neural net produce an answer good enough at this cost? Dwarkesh Patel  Yes. And you will just have price discrimination in different models?Ilya Sutskever  It's already the case today. On our product, the API serves multiple neural nets of different sizes and different customers use different neural nets of different sizes depending on their use case. If someone can take a small model and fine-tune it and get something that's satisfactory for them, they'll use that. But if someone wants to do something more complicated and more interesting, they'll use the biggest model. Dwarkesh Patel  How do you prevent these models from just becoming commodities where these different companies just bid each other's prices down until it's basically the cost of the GPU run? Ilya Sutskever  Yeah, there's without question a force that's trying to create that. And the answer is you got to keep on making progress. You got to keep improving the models, you gotta keep on coming up with new ideas and making our models better and more reliable, more trustworthy, so you can trust their answers. All those things.Dwarkesh Patel  Yeah. But let's say it's 2025 and somebody is offering the model from 2024 at cost. And it's still pretty good. Why would people use a new one from 2025 if the one from just a year older is even better?Ilya Sutskever  There are several answers there. For some use cases that may be true. There will be a new model for 2025, which will be driving the more interesting use cases. There is also going to be a question of inference cost. If you can do research to serve the same model at less cost. The same model will cost different amounts to serve for different companies. I can also imagine some degree of specialization where some companies may try to specialize in some area and be stronger compared to other companies. And to me that may be a response to commoditization to some degree.Dwarkesh Patel  Over time do the research directions of these different companies converge or diverge? Are they doing similar and similar things over time? Or are they branching off into different areas? Ilya Sutskever  I'd say in the near term, it looks like there is convergence. I expect there's going to be a convergence-divergence-convergence behavior, where there is a lot of convergence on the near term work, there's going to be some divergence on the longer term work. But then once the longer term work starts to fruit, there will be convergence again,Dwarkesh Patel  Got it. When one of them finds the most promising area, everybody just…Ilya Sutskever  That's right. There is obviously less publishing now so it will take longer before this promising direction gets rediscovered. But that's how I would imagine the thing is going to be. Convergence, divergence, convergence.Dwarkesh Patel  Yeah. We talked about this a little bit at the beginning. But as foreign governments learn about how capable these models are, are you worried about spies or some sort of attack to get your weights or somehow abuse these models and learn about them?Ilya Sutskever  Yeah, you absolutely can't discount that. Something that we try to guard against to the best of our ability, but it's going to be a problem for everyone who's building this. Dwarkesh Patel  How do you prevent your weights from leaking? Ilya Sutskever  You have really good security people.Dwarkesh Patel  How many people have the ability to SSH into the machine with the weights?Ilya Sutskever  The security people have done a really good job so I'm really not worried about the weights being leaked.Dwarkesh Patel  What kinds of emergent properties are you expecting from these models at this scale? Is there something that just comes about de novo?Ilya Sutskever  I'm sure really new surprising properties will come up, I would not be surprised. The thing which I'm really excited about, the things which I'd like to see is — reliability and controllability. I think that this will be a very, very important class of emergent properties. If you have reliability and controllability that helps you solve a lot of problems. Reliability means you can trust the model's output, controllability means you can control it. And we'll see but it will be very cool if those emergent properties did exist.Dwarkesh Patel  Is there some way you can predict that in advance? What will happen in this parameter count, what will happen in that parameter count?Ilya Sutskever  I think it's possible to make some predictions about specific capabilities though it's definitely not simple and you can't do it in a super fine-grained way, at least today. But getting better at that is really important. And anyone who is interested and who has research ideas on how to do that, that can be a valuable contribution.Dwarkesh Patel  How seriously do you take these scaling laws? There's a paper that says — You need this many orders of magnitude more to get all the reasoning out? Do you take that seriously or do you think it breaks down at some point?Ilya Sutskever  The thing is that the scaling law tells you what happens to your log of your next word prediction accuracy, right? There is a whole separate challenge of linking next-word prediction accuracy to reasoning capability. I do believe that there is a link but this link is complicated. And we may find that there are other things that can give us more reasoning per unit effort. You mentioned reasoning tokens, I think they can be helpful. There can probably be some things that help.Dwarkesh Patel  Are you considering just hiring humans to generate tokens for you? Or is it all going to come from stuff that already exists out there?Ilya Sutskever  I think that relying on people to teach our models to do things, especially to make sure that they are well-behaved and they don't produce false things is an extremely sensible thing to do. Is progress inevitable?Dwarkesh Patel  Isn't it odd that we have the data we needed exactly at the same time as we have the transformer at the exact same time that we have these GPUs? Like is it odd to you that all these things happened at the same time or do you not see it that way?Ilya Sutskever  It is definitely an interesting situation that is the case. I will say that it is odd and it is less odd on some level. Here's why it's less odd — what is the driving force behind the fact that the data exists, that the GPUs exist, and that the transformers exist? The data exists because computers became better and cheaper, we've got smaller and smaller transistors. And suddenly, at some point, it became economical for every person to have a personal computer. Once everyone has a personal computer, you really want to connect them to the network, you get the internet. Once you have the internet, you suddenly have data appearing in great quantities. The GPUs were improving concurrently because you have smaller and smaller transistors and you're looking for things to do with them. Gaming turned out to be a thing that you could do. And then at some point, Nvidia said — the gaming GPU, I might turn it into a general purpose GPU computer, maybe someone will find it useful. It turns out it's good for neural nets. It could have been the case that maybe the GPU would have arrived five years later, ten years later. Let's suppose gaming wasn't the thing. It's kind of hard to imagine, what does it mean if gaming isn't a thing? But maybe there was a counterfactual world where GPUs arrived five years after the data or five years before the data, in which case maybe things wouldn't have been as ready to go as they are now. But that's the picture which I imagine. All this progress in all these dimensions is very intertwined. It's not a coincidence. You don't get to pick and choose in which dimensions things improve.Dwarkesh Patel  How inevitable is this kind of progress? Let's say you and Geoffrey Hinton and a few other pioneers were never born. Does the deep learning revolution happen around the same time? How much is it delayed?Ilya Sutskever  Maybe there would have been some delay. Maybe like a year delayed? Dwarkesh Patel Really? That's it? Ilya Sutskever It's really hard to tell. I hesitate to give a longer answer because — GPUs will keep on improving. I cannot see how someone would not have discovered it. Because here's the other thing. Let's suppose no one has done it, computers keep getting faster and better. It becomes easier and easier to train these neural nets because you have bigger GPUs, so it takes less engineering effort to train one. You don't need to optimize your code as much. When the ImageNet data set came out, it was huge and it was very, very difficult to use. Now imagine you wait for a few years, and it becomes very easy to download and people can just tinker. A modest number of years maximum would be my guess. I hesitate to give a lot longer answer though. You can't re-run the world you don't know. Dwarkesh Patel  Let's go back to alignment for a second. As somebody who deeply understands these models, what is your intuition of how hard alignment will be?Ilya Sutskever  At the current level of capabilities, we have a pretty good set of ideas for how to align them. But I would not underestimate the difficulty of alignment of models that are actually smarter than us, of models that are capable of misrepresenting their intentions. It's something to think about a lot and do research. Oftentimes academic researchers ask me what's the best place where they can contribute. And alignment research is one place where academic researchers can make very meaningful contributions. Dwarkesh Patel  Other than that, do you think academia will come up with important insights about actual capabilities or is that going to be just the companies at this point?Ilya Sutskever  The companies will realize the capabilities. It's very possible for academic research to come up with those insights. It doesn't seem to happen that much for some reason but I don't think there's anything fundamental about academia. It's not like academia can't. Maybe they're just not thinking about the right problems or something because maybe it's just easier to see what needs to be done inside these companies.Dwarkesh Patel  I see. But there's a possibility that somebody could just realize…Ilya Sutskever  I totally think so. Why would I possibly rule this out? Dwarkesh Patel  What are the concrete steps by which these language models start actually impacting the world of atoms and not just the world of bits?Ilya Sutskever  I don't think that there is a clean distinction between the world of bits and the world of atoms. Suppose the neural net tells you — hey here's something that you should do, and it's going to improve your life. But you need to rearrange your apartment in a certain way. And then you go and rearrange your apartment as a result. The neural net impacted the world of atoms.Future breakthroughsDwarkesh Patel  Fair enough. Do you think it'll take a couple of additional breakthroughs as important as the Transformer to get to superhuman AI? Or do you think we basically got the insights in the books somewhere, and we just need to implement them and connect them? Ilya Sutskever  I don't really see such a big distinction between those two cases and let me explain why. One of the ways in which progress is taking place in the past is that we've understood that something had a desirable property all along but we didn't realize. Is that a breakthrough? You can say yes, it is. Is that an implementation of something in the books? Also, yes. My feeling is that a few of those are quite likely to happen. But in hindsight, it will not feel like a breakthrough. Everybody's gonna say — Oh, well, of course. It's totally obvious that such and such a thing can work. The reason the Transformer has been brought up as a specific advance is because it's the kind of thing that was not obvious for almost anyone. So people can say it's not something which they knew about. Let's consider the most fundamental advance of deep learning, that a big neural network trained in backpropagation can do a lot of things. Where's the novelty? Not in the neural network. It's not in the backpropagation. But it was most definitely a giant conceptual breakthrough because for the longest time, people just didn't see that. But then now that everyone sees, everyone's gonna say — Well, of course, it's totally obvious. Big neural network. Everyone knows that they can do it.Dwarkesh Patel  What is your opinion of your former advisor's new forward forward algorithm?Ilya Sutskever  I think that it's an attempt to train a neural network without backpropagation. And that this is especially interesting if you are motivated to try to understand how the brain might be learning its connections. The reason for that is that, as far as I know, neuroscientists are really convinced that the brain cannot implement backpropagation because the signals in the synapses only move in one direction. And so if you have a neuroscience motivation, and you want to say — okay, how can I come up with something that tries to approximate the good properties of backpropagation without doing backpropagation? That's what the forward forward algorithm is trying to do. But if you are trying to just engineer a good system there is no reason to not use backpropagation. It's the only algorithm.Dwarkesh Patel  I guess I've heard you in different contexts talk about using humans as the existing example case that AGI exists. At what point do you take the metaphor less seriously and don't feel the need to pursue it in terms of the research? Because it is important to you as a sort of existence case.Ilya Sutskever  At what point do I stop caring about humans as an existence case of intelligence?Dwarkesh Patel  Or as an example you want to follow in terms of pursuing intelligence in models.Ilya Sutskever  I think it's good to be inspired by humans, it's good to be inspired by the brain. There is an art into being inspired by humans in the brain correctly, because it's very easy to latch on to a non-essential quality of humans or of the brain. And many people whose research is trying to be inspired by humans and by the brain often get a little bit specific. People get a little bit too — Okay, what cognitive science model should be followed? At the same time, consider the idea of the neural network itself, the idea of the artificial neuron. This too is inspired by the brain but it turned out to be extremely fruitful. So how do they do this? What behaviors of human beings are essential that you say this is something that proves to us that it's possible? What is an essential? No this is actually some emergent phenomenon of something more basic, and we just need to focus on getting our own basics right. One can and should be inspired by human intelligence with care.Dwarkesh Patel  Final question. Why is there, in your case, such a strong correlation between being first to the deep learning revolution and still being one of the top researchers? You would think that these two things wouldn't be that correlated. But why is there that correlation?Ilya Sutskever  I don't think those things are super correlated. Honestly, it's hard to answer the question. I just kept trying really hard and it turned out to have sufficed thus far. Dwarkesh Patel So it's perseverance. Ilya Sutskever It's a necessary but not a sufficient condition. Many things need to come together in order to really figure something out. You need to really go for it and also need to have the right way of looking at things. It's hard to give a really meaningful answer to this question.Dwarkesh Patel  Ilya, it has been a true pleasure. Thank you so much for coming to The Lunar Society. I appreciate you bringing us to the offices. Thank you. Ilya Sutskever  Yeah, I really enjoyed it. Thank you very much. Get full access to The Lunar Society at www.dwarkeshpatel.com/subscribe