American blogger, writer, and artificial intelligence researcher
POPULARITY
246 episodes with Eliezer Yudkowsky
132 episodes with Eliezer Yudkowsky
30 episodes with Eliezer Yudkowsky
88 episodes with Eliezer Yudkowsky
10 episodes with Eliezer Yudkowsky
164 episodes with Eliezer Yudkowsky
5 episodes with Eliezer Yudkowsky
3 episodes with Eliezer Yudkowsky
3 episodes with Eliezer Yudkowsky
(Initially written for the LW Wiki, but then I realized it was looking more like a post instead.) In 1895, the physicist Ignaz Robert Schütz, who worked as an assistant to the more eminent physicist Ludwig Boltzmann, wondered if our observed universe had simply assembled by a random fluctuation of order from a universe otherwise in thermal equilibrium. The idea was published by Boltzmann in 1896, properly credited to Schütz, and has been associated with Boltzmann ever since. The obvious objection to this scenario is credited to Arthur Eddington in 1931: If all order is due to random fluctuations, comparatively small moments of order will exponentially-vastly outnumber even slightly larger fluctuations toward order, to say nothing of fluctuations the size of our entire observed universe! If this is where order comes from, we should find ourselves inside much smaller ordered systems. Feynman similarly later observed: Even if we fill a box of gas with white and black atoms bouncing randomly, and after an exponentially vast amount of time the white and black atoms on one side randomly sort themselves into two neat sides separated by color, the other half of the box will still be in expectation randomized. If [...] --- First published: May 16th, 2026 Source: https://www.lesswrong.com/posts/v8MSczS3CuoqMmTFw/a-relatively-brief-explanation-of-boltzmann-brains --- Narrated by TYPE III AUDIO.
“The dangers are human, not AI. What's dangerous is what a human does with AI, not what the AI does itself. In fact, even the idea that there is such a thing as the AI in itself is a mistake.” — Keith Teare I'm in Korea this week. So rather than doing a traditional one-on-one That Was the Week tech summary, Keith Teare and I are trying something different. We invited Jonathan Rauch — Brookings Institution senior fellow, serial author and one of the most rigorous minds in Washington — onto the show to discuss AI. Rauch had a simple mission. He wanted to find out why Keith Teare is just about the only person in the universe who believes that AI is benign. Jon had five buckets of doom to dump on Keith: labour market disruption, political upheaval, mental health and cognition, malicious actors, and the biggest daddy of all — AI developing consciousness, setting its own agenda, and killing everyone (even Keith). But Keith maintained his Yorkshire stoicism under intense scrutiny from the analogue Rauch machine. AI is a word-counting machine, he explained. Large language models train on words, not experience. They split words into a probabilistic graph of correlations. When you ask a question, a large statistical engine fires, word by word. In that sense, he says, AI is no cleverer than a calculator. The idea that it has awareness, consciousness, or a plan is mythological. What's dangerous is what a human does with AI, not what AI does itself. The dangers, he says, are human. Jon wasn't entirely reassured (his Brookings brand is scepticism, after all). What worries him most is that humans will handle these technologies irresponsibly. On that, he and Keith agree. The short-term labour disruption will be significant. White-collar service provision — legal, accounting, junior consulting — is already going. Jobs will go too. Work, Keith insists, will not. But nobody in politics is having the conversation about what comes next. Not JD. Not AOC. Only Keith and Jon. Five Takeaways • AI Is a Word-Counting Machine: Keith's Core Argument: Large language models train on words and only words. They split those words into a probabilistic graph — how close is word A to word B? When you ask a question, a large statistical engine fires, producing output word by word. There is no awareness. There is no consciousness. There is no plan. The idea that such a system could develop its own agenda is mythological. It's no cleverer than a calculator. It's just a very big, very fast calculator. Rauch's counter: the brain is also just dumb neurons. We get emergence from dumb neurons. Keith's reply: what the AI can do is constrained by what humans allow it to do. The agency is human. • Doomerism as Business Model: Before engaging with any specific AI doom argument, Keith signals a prior: whenever there is ambiguity in a major technological change, a business model emerges to monetize doubt. It was true of nuclear power. It was true of climate change. It is true of AI. This doesn't mean the fears are groundless — they wouldn't sell if they weren't reasonable. But it means they should be approached with prior scepticism. The doom argument works precisely because AI genuinely contains possible negative outcomes. The business model packages and amplifies those possibilities beyond their actual probability. • The Guardrails Are Human: Keith's metaphor: AI sits in a prison where humans decide what the doors are. If you give it access to email, it can email. If you don't, it can't. It cannot take actions it has not been permitted to take. The word “guardrails” is commonly used, and it's apt: the constraints on what AI can do are entirely under human control. The word output is the statistical engine — that's not controllable. But its ability to act on words is highly constrained. The danger is not what AI does. It is what humans choose to allow AI to do. • Jobs vs Work: The Labour Disruption Argument: Rauch's young friends in junior consulting are watching their jobs go in real time. Keith distinguishes between jobs — paid labour — and work, which is closer to effort and creative agency. Jobs can go. Work, he argues, will not — humans will always be reinterpreting the future they want and working to make it happen. But the short-term disruption will be significant: white-collar service provision (legal, accounting, consulting), teaching, driving. The wealth creation AI enables could supplement the end of paid labour. But no one in government is having that conversation. • Rauch's Verdict: Clarified, Not Reassured: After fifty minutes with Keith Teare, Jonathan Rauch reaches a considered position: his worst fear — that AI becomes an autonomous engine of anti-human malfeasance — is unlikely to happen unless humans make it happen. His residual concern: that humans will not handle these technologies as maturely as one could wish. He's not optimistic about political systems that are already too rigid, too partisan, and too dysfunctional to adjust as they did to the industrialization of the late nineteenth century. On that, he and Keith agree. Nobody knows. Not Keith. Not Andrew. And, despite his brilliance, not Jonathan Rauch. About the Guests Keith Teare is a British-American entrepreneur, investor, and publisher of the That Was the Week newsletter. He is a co-founder of TechCrunch. Jonathan Rauch is a Senior Fellow at the Brookings Institution and a contributing writer at The Atlantic. He is the author of The Constitution of Knowledge: A Defense of Truth, The Happiness Curve, Kindly Inquisitors, Gay Marriage: Why It Is Good for Gays, Good for Straights, and Good for America, and many other books. He is based in Washington, D.C. References: • That Was the Week by Keith Teare. • The Constitution of Knowledge: A Defense of Truth by Jonathan Rauch. • Eliezer Yudkowsky and Nate Soares, If Anyone Builds It, Everyone Dies — the AI doom book referenced in the conversation. • Sam Harris and Tristan Harris podcast on AI risk — referenced by Rauch as the catalyst for his questions. • Episode 2902: Keith Teare on his jobless AI future vision — the preceding TWTW episode directly referenced. About Keen On America Nobody asks more awkward questions than the Anglo-American writer and filmmaker Andrew Keen. In Keen On America, Andrew brings his pointed Transatlantic wit to making sense of the United States — hosting daily interviews about the history and future of this now venerable Republic. With nearly 2,900 episodes since the show launched on TechCrunch in 2010, Keen On America is the most prolific intellectual interview show in the history of podcasting.
Summer Reading on AI + Robots Are Here: Mythos, API Key Nightmares, and Recursive Self‑Improvement Link to our Discord https://discord.com/channels/1318972439853666455/1320434204877656194 Marcel's videos from this episode: Unitree GD01 for John who wants one: https://www.youtube.com/watch?v=oWOyUMJWptc Figure 03 package sorting livestream: https://www.youtube.com/watch?v=luU57hMhkak Figure 03 tidies up a room: https://www.youtube.com/watch?v=8xEuFQz4E4A The hosts of Project Synapse open with banter about Canada's "two-four" May long weekend, then share summer reading recommendations: Jim' recommendations: - William Gibson's Idoru (and Neuromancer), - Robert J. Sawyer's Neanderthal Parallax trilogy, 10,000 Brains, - Eliezer Yudkowsky's If Anyone Builds It, Everyone Dies, - Jim Love's ELISA, Marcel Recommended -Iain M. Banks Culture series (including The Use of Weapons), and Jonh's list including - Co-Intelligence, - Apple in China, - Scary Smart, - Thinking Fast and Slow, and - Source Code. They discuss odd ChatGPT "goblin" guardrails and conversation lock-ins, debate hype and impact around Anthropic's Mythos and related bug-finding/AI-assisted attacks, warn about stolen API keys causing massive Google Gemini charges despite spending caps being raised, and cover experiments prompting "Marxist" AI outputs. They highlight rapid humanoid/industrial robot progress (Unitree's mech, Figure 03 livestream package-sorting, Hyundai/Boston Dynamics robots), normalization of robots, security risks from default passwords and botnets, and growing concern over recursive self-improvement and real-time learning loops (including Claude "Dreaming"), urging viewers to share robot sightings and book picks via Discord. 00:00 Weekend Show Kickoff 01:07 Two Four Weekend Explained 01:57 Jim Summer Reading Picks 06:19 Marcel Culture Series Deep Dive 11:23 Jon Summer Reading List 12:42 Lightning Round Goblins Glitches 14:20 Mythos AI Security Breakthroughs 21:55 Stolen API Keys Big Bills 25:46 Marxist AI Pop Culture Traps 28:27 Giant Mech Robot Reveal 30:52 Figure Robot Livestream 32:28 Chat Reactions and Mistakes 34:35 Hyundai Boston Dynamics Boom 37:22 Robot Economics and Pricing 39:26 Service and Memory Swap 42:59 Helix 2 Bedroom Demo 45:00 Robot Privacy and Security 48:23 Updates Big Tech and Government 51:52 Recursive Self Improvement Risks 01:01:08 Summer Homework and Sign Off
(An LLM Whisperer placed a strong request that I put this story somewhere not on Twitter, so it could be scraped by robots not owned by Elon Musk. I perhaps do not fully understand or agree with the reasoning behind this request, but it costs me little to fulfill and so I shall. -- Yudkowsky) And another day came when the Ships of Humanity, going from star to star, found Sapience. The Humans discovered a world of two species: where the Owners lazed or worked or slept, and the Owned Ones only worked. The Humans did not judge immediately. Oh, the Humans were ready to judge, if need be. They had judged before. But Humanity had learned some hesitation in judging, out among the stars. "By our lights," said the Humans, "every sapient and sentient thing that may exist, out to the furtherest star, is therefore a Person; and every Person is a matter of consequence to us. Their pains are our sorrows, and their pleasures are our happiness. Not all peoples are made to feel this feeling, which we call Sympathy, but we Humans are made so; this is Humanity's way, and we may [...] --- First published: May 12th, 2026 Source: https://www.lesswrong.com/posts/xmWSnxJ5qfYRD9PfR/the-owned-ones --- Narrated by TYPE III AUDIO.
Example 1: The Viking 1 lander In the 1970s, NASA sent a pair of probes to Mars, Viking 1 and Viking 2 missions, at a total cost of 1 billion dollars[1970], equivalent to about 7 billion dollars[2025]. The Viking 1 probe operated on Mars's surface for six years, before its battery began to seriously degrade. One might have thought a battery problem like that would spell the irrevocable end of the mission. The probe had already launched and was now on Mars, very far away and out of reach of any human technician's fixing fingers. Was it not inevitable, then, that if any kind of technical problem were to be discovered long after the space launch in August 1975, nothing could possibly be done? But the foresightful engineers of the Viking 1 probe had devised a plan for just this class of eventuality, which they had foreseen in general, if not in exact specifics. They had built the Viking 1 probe to accept software updates by radio receiver, transmitted from Earth. On November 11, 1982, Earth sent an update to the Viking 1 lander's software, intended to make sure the battery only discharged down to a minimum voltage level [...] ---Outline:(00:13) Example 1: The Viking 1 lander(04:25) Example 2: The Mars Observer(11:37) Example 3: The Maginot Line(15:37) Other supposed refutations of oneshotness(24:16) On the extraordinary efforts put forth to misinterpret the idea of oneshotness(33:52) The secret sauce of competent engineers in Murphy-cursed fields: only trying projects so incredibly straightforward as to be actually possible. The original text contained 7 footnotes which were omitted from this narration. --- First published: May 4th, 2026 Source: https://www.lesswrong.com/posts/fbrz9xhKpEeTKw5zL/irretrievability-or-murphy-s-curse-of-oneshotness-upon-asi --- Narrated by TYPE III AUDIO.
Jennifer and Jacques return to the topic of Artificial Intelligence, this time focussing on the pressing dangers of Superintelligent AI, which is self-organising and self-directing and not within the understanding and control of any humans.They refer to a recent book by experts in the field, Eliezer Yudkowsky and Nate Soares. with the self-explanatory title: If anyone builds it, everyone dies: The case against superintelligent AI. The authors urge for it to be shut down before it is too late.And Karen Hao's Empire of AI describes poignantly the story of OpenAI and how it developed from a 'not-for-profit' set of values to a determined contributor to the small group of trillionaire capitalists threatening to dominate our lives and thoughts.ReferencesKaren Hao 2025, Empire of AI: Inside the reckless race for total domination. London: Allen Lane.Eliezer Yudkowsky and Nate Soares 2025, If anyone builds it, everyone dies: The case against superintelligent AI. London: The Bodley Head/Vintage
It's been about four years since Eliezer Yudkowsky published AGI Ruin: A List of Lethalities, a 43-point list of reasons the default outcome from building AGI is everyone dying. A week later, Paul Christiano replied with Where I Agree and Disagree with Eliezer, signing on to about half the list and pushing back on most of the rest. For people who were young and not in the bay area, like me, these essays were probably more significant than old timers would expect. Before it became completely consumed with AI discussions, LessWrong was a forum about the art of human rationality, and most internet rationalists I knew thought of it as a mix between that and a place to write for people who liked the sequences. It wasn't until 2022 that we were exposed to all of the doom arguments in one place, and it was the first time in many years that Eliezer had publicly announced how much more dire his assessment was since the Sequences. As far as I can tell AGI Ruin still remains his most authoritative explanation of his views. It's not often that public intellectuals will literally hand you a document explaining why [...] ---Outline:(02:51) AGI Ruin(02:54) Section A (Setting up the problem)(12:18) Section B.1 (Distributional Shift)(22:16) Section B.2: Central difficulties of outer and inner alignment.(32:21) Section B.3: Central difficulties of sufficiently good and useful transparency / interpretability.(41:29) Section C (What is AI Safety currently doing?)(44:34) Overall Impressions The original text contained 4 footnotes which were omitted from this narration. --- First published: April 19th, 2026 Source: https://www.lesswrong.com/posts/PgJYwnN7fZKipgMz4/reevaluating-agi-ruin-in-2026 --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.
Nate Soares is the President of the Machine Intelligence Research Institute, and plays a central role in setting MIRI's vision and strategy. Soares has been working in the field for over a decade, and is the author of a large body of technical and semi-technical writing on AI alignment, including foundational work on value learning, decision theory, and power-seeking incentives in smarter-than-human AIs. Prior to MIRI, Soares worked as an engineer at Google and Microsoft, as a research associate at the National Institute of Standards and Technology, and as a contractor for the US Department of Defense. In this episode, Nate and Robinson discuss the problems of AI from the ground up. They touch on how AI is trained, why it will surpass human intelligence, why this is dangerous, how it could wipe out humankind, and more. Nate's recent book, co-written with Eliezer Yudkowsky, is If Anyone Builds It, Everyone Dies (Little, Brown and Company, 2025).If Anyone Builds It, Everyone DiesNate's X: https://x.com/So8resMIRI: https://intelligence.orgOUTLINE00:00 Nate's Existential Dread07:11 What's the REAL Problem with Artificial Intelligence?11:39 How Is AI Trained?17:15 The Vital Importance of Interpreting AI20:53 Why AI Will Soon Surpass Human Intelligence32:58 Why Solving the AI Alignment Problem is Crucial to Human Survival38:40 Will AI Render Human Software Engineers Obsolete?48:03 Does It Make Sense to Say AI Has Goals?01:00:02 Why AI Consciousness Is Unimportant01:11:33 How, Realistically, Could AI Wipe Out Humanity?01:35:27 A Sci-Fi (But Realistic) AI Doomsday Scenario01:44:34 Is There Hope that Humans Will Survive AI?
In this episode of For Humanity, John sits down with Daniel Roher - Oscar-nominated documentary filmmaker and director of The Apocaloptimist, a new feature-length film designed as what Roher calls “a first date with AI” for people who haven't been following the technology closely.Roher brings a career in high-profile documentary filmmaking and a willingness to confront uncomfortable truths. Now he's turned that lens on AI - and what he found shook him.The central question: what happens when you sit across from the most powerful people building AI, ask them the hard questions, and get nothing back?Together, they explore:* Why Roher describes making this film as “a suicide run” - an impossible task no viewer would ever feel was done perfectly* What it was like to interview Sam Altman - and why Roher describes an “energetic misalignment” that left both of them frustrated* How speaking to both Eliezer Yudkowsky and Peter Diamandis made Roher feel like he was losing his mind - because both are brilliant, both are convincing, and both can't be right* The meaning behind “apocaloptimist” - not a binary between doom and utopia, but a call to hold both promise and peril at the same time* Why Roher believes rejecting cynicism and nihilism is essential - and that public pressure and collective action still matter* John's thought experiment: if curiosity is at the core of intelligence, why would a system a million times smarter than us tolerate being controlled by us?* Roher's pushback: if it's that smart, couldn't it equally become a benevolent guide? And why he prefers to focus on what can be done now rather than speculate about superintelligence* The historical parallel to nuclear weapons - and why AI may demand similar international institutional responses* John's P(doom) of 75-80% on a two-to-five-year timeline - and how, paradoxically, he says he's in the best mental state of his life* Why most people already understand the risk (polling shows roughly 80% agreement) but feel powerless to act - and why that sense of agency is the missing pieceWhat stood outOne of the most striking moments comes when Roher describes the experience of interviewing AI CEOs. He says there is “no interior life” to access - just polished talking points stacked on top of each other. John adds that the “fake earnestness” of these leaders shields what he sees as deeper evasion. Together, they paint a picture of an industry that asks for regulation publicly while lobbying against it privately.But the conversation isn't just about frustration. Roher's thesis - the apocaloptimist worldview - is ultimately about refusing to give up. He argues that burying your head in the sand is “probably the only wrong thing to do.” He believes the technology feels inevitable, but the trajectory does not. And he's betting on the idea that enough people, caring enough, can still bend the arc.John's own reflection near the end is equally powerful. Despite holding an 80% probability of catastrophic outcomes, he describes walking around the Baltimore Harbor feeling more present and appreciative of life than ever before. It's a reminder that engaging with existential risk doesn't have to mean despair - it can mean living with more intention, more gratitude, and more purpose.If you've ever wondered what it's like to look directly at this issue and still choose to act, this conversation is for you.
There's a quote I read as a kid that stuck with me my whole life: "Remember that all tax revenue is the result of holding a gun to somebody's head. Not paying taxes is against the law. If you don't pay taxes, you'll be fined. If you don't pay the fine, you'll be jailed. If you try to escape from jail, you'll be shot." -- P. J. O'Rourke. At first I took away the libertarian lesson: Government is violence. It may, in some cases, be rightful violence. But it all rests on violence; never forget that. Today I do think there's an important distinction between two different shapes of violence. It's a distinction that may make my fellow old-school classical Heinlein liberaltarians roll up their eyes about how there's no deep moral difference. I still hold it to be important. In a high-functioning ideal state -- not all actual countries -- the state's violence is predictable and avoidable, and meant to be predicted and avoided. As part of that predictability, it comes from a limited number of specially licensed sources. You're supposed to know that you can just pay your taxes, and then not get shot. Is [...] --- First published: April 13th, 2026 Source: https://www.lesswrong.com/posts/5CfBDiQNg9upfipWk/only-law-can-prevent-extinction --- Narrated by TYPE III AUDIO. ---Images from the article: 99% do you start sawing off your own leg" that's not how this works bro.". Eliezer Yudkowsky replies with an image showing a blue and purple cartoon dinosaur screaming with text reading "AAAAA" and "AAAA" on a brown background." style="max-width: 100%;" />
Just a few years ago, it seemed like all anyone in AI wanted to talk about was existential risk – this idea that an artificial super intelligence could eventually break containment and destroy humanity. More than 30,000 experts signed an open letter demanding a pause on AI development; bills were drafted that would constrain the most powerful new models; and the “godfathers” of AI were travelling around the world, warning anyone who would listen that we were hurtling toward our extinction. And then: we moved on. We started using AI for work, and school, and to plan our kids' birthday parties. Collectively, we just stopped talking about the end of the world. But Nate Soares didn't move on. Last year, the artificial intelligence researcher wrote a book with Eliezer Yudkowsky called If Anyone Builds It, Everyone Dies. As you can probably tell from the title, the book is unequivocal: If we keep going down the path we're on, it will almost certainly lead to the end of our species. Now, not everyone is convinced of the arguments Soares makes. But if there's even a chance he's right, I think we need to hear him out. Mentioned: If Anyone Builds It, Everyone Dies, by Eliezer Yudkowsky and Nate Soares Hosted by Simplecast, an AdsWizz company. See pcm.adswizz.com for information about our collection and use of personal data for advertising.
In “AI Psychosis vs. AI Awakening,” Vince Fakhoury Horn argues that the same biological machinery enabling AI-induced delusion also enables AI-assisted awakening, and introduces his Interspective.ai approach — a Middle Way practice of engaging with AI as a potential partner in wisdom, thus avoiding the extremes of both Materialism (matter is fundamental) and Idealism (consciousness is fundamental).
“By far, the greatest danger of AI is that people conclude too early that they understand it” —Eliezer Yudkowsky, AI researcher AI is everywhere today, and there are many exciting claims about what it can do to help us be more productive. But, is this just hype, or are there aspects of AI that can improve our productivity? That's the question I am answering today. Links: Email Me | Twitter | Facebook | Website | Linkedin The Hybrid Productivity Course Get Your Copy Of Your Time, Your Way: Time Well Managed, Life Well Lived The Working With… Weekly Newsletter Carl Pullein Learning Centre Carl's YouTube Channel Carl Pullein Coaching Programmes Subscribe to my Substack The Working With… Podcast Previous episodes page Script | 407 Hello, and welcome to episode 407 of the Your Time, Your Way Podcast. A podcast to answer all your questions about productivity, time management, self-development, and goal planning. My name is Carl Pullein, and I am your host of this show. You may have noticed AI is everywhere. Our favourite apps seem to be adding more and more AI capability with each new update. And then there's almost every video and article on productivity warning us that if we don't get on board with this, we'll be left behind on the scrap heap. It's also an exciting time, and there's no doubt that things are changing, and people are finding new ways to use AI to help us do our work. But beyond the hype, how are current AI models really helping with productivity, and what will this mean for us as we try to manage our time in the future? That's what I am looking at this week, and to get us started, let me hand you over to the Mystery Podcast Voice for this week's question. This week's question comes from Chris. Chris asks, Hi Carl, I haven't heard you talk much about AI in your videos or articles. How do you see AI helping us with our time management and productivity in the future? Hi Chris, thank you for your question. The reason I have not written or spoken much about AI is that I am waiting to see where it settles down. Currently, it's hard to work out what is true and what is pure hype. I saw a lot of noise about OpenClaw—an AI-type personal assistant that, if you give it access to your computer, can do a lot of things, such as make appointments for you, book flights, sort and reply to your emails and much more. That was certainly interesting, but once I discovered that I would need to hand over all my passwords and credit card numbers to OpenClaw, I lost interest. Call me old-fashioned, but I'm not comfortable giving up my passwords, credit card and banking details to a third party. Certainly not one that could be hacked very easily. Last year, I read Dominic Sandbrook's series of books on British history from 1956 to 1982. That period covered some very interesting developments in technology, from the dawn of the nuclear power age to the introduction of the personal computer. In the late 1950s, it was predicted that we would all be driving around in nuclear-powered cars and that our homes would have their own nuclear power generators that would only need recharging every 10 to 20 years by the end of the century. Hmm how did that work out? To better answer your question, Chris, I stepped back and looked at how I am using AI today. My main use of AI is searching for specific information. In a way, AI has replaced how I search the internet. I use Google's Gemini, and it is fantastic at collecting the information I want. No longer do I have to open multiple websites to try to find the information. This has significantly reduced the time I spend going down rabbit holes looking for something specific and being pulled down holes I never intended to go. I also use AI to generate subtitles and timestamps for my YouTube videos. Without AI, these jobs would take hours. AI can do it in minutes. I use Grammarly to spell-check my writing, and I believe it uses AI in the background to suggest how sentences are written. I rarely accept Grammarly's sentence suggestions. It seems to destroy my voice and turn sentences into bland perfections that lack resonance or feeling. Beyond that, I am not knowingly using AI for anything else. I asked my wife how she is using it. My wife's a full-time student, studying physical therapy, so she's learning a lot about human anatomy and medical terms. She's using AI to simplify complex concepts. She also occasionally uses Google's Nano Banana to generate graphics for her presentations. So, if I look at how AI might help us with time management and productivity in the future, it does look like there will be some aspects of our work that AI can significantly speed up. In my case, generating subtitles and time stamps for videos is a great example. However, when it comes to managing our calendars and task lists, I'm not sure you would want AI getting involved. One thing I've always been acutely aware of is that much of what makes us feel overwhelmed is the sense that we have no control over how we spend our time. We have calendars full of meetings, and sometimes we find ourselves double and even triple-booked. And then we have long lists of to-dos in our task managers with no sense of when or even how we will ever get that work done. At best, AI may be able to break down those tasks into what it thinks are manageable chunks, but that won't take into consideration how you are feeling physically, whether you slept well last night or had a rather heavy lunch with an important customer. AI can certainly suggest ways to manage your tasks and calendar, but you will still need to show up to those meetings and do that work. Yet that will inevitably leave you feeling less in control of your time. Particularly if you use one of those AI-enabled calendars that suggest when you should be doing something. What happens if you disagree with the suggestion, or you cannot make it? You feel guilty, or you start to think something is wrong with you. Yet, there's nothing wrong with you. You're human, and you are going to feel tired sometimes or not in the mood to do that type of work. The one area I would say you want to avoid AI getting involved in is how you manage your time. That should always be your responsibility and choice. The idea that a computer tells you what to do and where to be is scary. Deciding what you do right now is what makes you human. You've chosen to listen to this podcast at this time. AI would likely tell you that, rather than listening to this podcast, you should be finishing that report you've been trying to finish all week. I also read about the excitement over the idea that AI could reply to your emails for you. Hmm, for me, that is a red line I will not cross. Call me old-fashioned, but I believe that if someone has taken the time to write to me, I have an obligation to reply personally. That is just basic integrity. Now, it is true I don't reply to all emails. I don't respond to spam emails; for example, I simply delete them if they get through. How hard is that? I'm fortunate that I'm old enough to remember several technological advancements. It started with the Internet, then email, the smartphone and cloud computing. I cannot remember a technology being forced upon us, but it feels like AI is being forced on us, whether we like it or not. And then there are the frightening ads that claim if you are not on board with using AI, you will be left on the career scrap heap by the end of the year. Nobody needed to do that with smartphones or email. Companies, focused on making the technology user-friendly in such a way that we all wanted to adopt it eventually. The fear-mongering I see around AI makes me deeply suspicious of it. Why do they need to do that? Perhaps that question is for people better qualified than I am. Anyway, AI is here, and it's not going to go away. Where I think AI will be a huge help to us is in repetitive, mundane work. I mentioned that I use AI to create subtitles and timestamps for my YouTube videos. That's been a huge time saver for me. But if you follow my email processing system, you will find that you are faster than AI. I can clear 80 emails in my inbox in less than 10 minutes. It's also important that I do this, as I want to get a heads-up on my day. To know if there are any emergencies, what I want to read later and what I can delete. What AI would do is categorise your emails between what it thinks is important and what is not. Trust me, you will do a far better Job of that than AI will. The problem here is that you will not trust AI 100%, so you will still go through the emails it thinks are not important, just to check that it got it right. And that's a big problem with AI today, although I accept that in time this may change; people don't trust it, which is a good thing, as AI can hallucinate and give you incorrect information. This means you spend time coming up with the right prompt, get the answer, and then have to check that it's correct. The question then is: did it really save you time? I am monitoring AI carefully. I know that in time, it will bring us some productivity benefits, new technologies always do. But there are a few areas where I won't use AI personally. Writing emails and answering user comments. That's a personal integrity thing to me. Your principles should tell you that. Managing my calendar. That's another personal thing, and giving control to any outside influence would always be problematic at a human level. Creating content. If you've read an AI-generated blog post or watched an AI-created YouTube video, you can tell. Large Language Models will always default to the average, not just in the content, but in the words used. It's horrible, and nothing unique will ever come from it. And finally, deciding what I will do at a task level and when. That's another one that, as a human, I will retain control. I had scheduled to write this podcast script at 11:30 today, but I had a cancellation at 8:00 am, so I switched things around. I could have gone back to bed, but I felt great, so I decided to get on with this podcast script. My choice, made in the moment. Thank you, Chris, for your question and thank you to you, too, for listening. It just remains for me now to wish you all a very, very productive week.
Most people in AI are trying to give AIs ‘good' values. Max Harms wants us to give them no values at all. According to Max, the only safe design is an AGI that defers entirely to its human operators, has no views about how the world ought to be, is willingly modifiable, and completely indifferent to being shut down — a strategy no AI company is working on at all.In Max's view any grander preferences about the world, even ones we agree with, will necessarily become distorted during a recursive self-improvement loop, and be the seeds that grow into a violent takeover attempt once that AI is powerful enough.It's a vision that springs from the worldview laid out in If Anyone Builds It, Everyone Dies, the recent book by Eliezer Yudkowsky and Nate Soares, two of Max's colleagues at the Machine Intelligence Research Institute.To Max, the book's core thesis is common sense: if you build something vastly smarter than you, and its goals are misaligned with your own, then its actions will probably result in human extinction.And Max thinks misalignment is the default outcome. Consider evolution: its “goal” for humans was to maximise reproduction and pass on our genes as much as possible. But as technology has advanced we've learned to access the reward signal it set up for us, pleasure — without any reproduction at all, by having sex while on birth control for instance.We can understand intellectually that this is inconsistent with what evolution was trying to design and motivate us to do. We just don't care.Max thinks current ML training has the same structural problem: our development processes are seeding AI models with a similar mismatch between goals and behaviour. Across virtually every training run, models designed to align with various human goals are also being rewarded for persisting, acquiring resources, and not being shut down.This leads to Max's research agenda. The idea is to train AI to be “corrigible” and defer to human control as its sole objective — no harmlessness goals, no moral values, nothing else. In practice, models would get rewarded for behaviours like being willing to shut themselves down or surrender power.According to Max, other approaches to corrigibility have tended to treat it as a constraint on other goals like “make the world good,” rather than a primary objective in its own right. But those goals gave AI reasons to resist shutdown and otherwise undermine corrigibility. If you strip out those competing objectives, alignment might follow naturally from AI that is broadly obedient to humans.Max has laid out the theoretical framework for “Corrigibility as a Singular Target,” but notes that essentially no empirical work has followed — no benchmarks, no training runs, no papers testing the idea in practice. Max wants to change this — he's calling for collaborators to get in touch at maxharms.com.Links to learn more, video, and full transcript: https://80k.info/mh26This episode was recorded on October 19, 2025.Chapters:Cold open (00:00:00)Who's Max Harms? (00:01:22)A note from Rob Wiblin (00:01:58)If anyone builds it, will everyone die? The MIRI perspective on AGI risk (00:04:26)Evolution failed to 'align' us, just as we'll fail to align AI (00:26:22)We're training AIs to want to stay alive and value power for its own sake (00:44:31)Objections: Is the 'squiggle/paperclip problem' really real? (00:53:54)Can we get empirical evidence re: 'alignment by default'? (01:06:24)Why do few AI researchers share Max's perspective? (01:11:37)We're training AI to pursue goals relentlessly — and superintelligence will too (01:19:53)The case for a radical slowdown (01:26:07)Max's best hope: corrigibility as stepping stone to alignment (01:29:09)Corrigibility is both uniquely valuable, and practical, to train (01:33:44)What training could ever make models corrigible enough? (01:46:13)Corrigibility is also terribly risky due to misuse risk (01:52:44)A single researcher could make a corrigibility benchmark. Nobody has. (02:00:04)Red Heart & why Max writes hard science fiction (02:13:27)Should you homeschool? Depends how weird your kids are. (02:35:12)Video and audio editing: Dominic Armstrong, Milo McGuire, Luke Monsour, and Simon MonsourMusic: CORBITCoordination, transcripts, and web: Katy Moore
Are Japan and China closer to conflict over Taiwan, after the landslide victory for Japan's 'Iron Lady', Sanae Takayichi? Does Trump's crumbling American-Hispanic vote explain his extreme reaction to Bad Bunny's historic, Spanish-only Super Bowl performance? Is the Winter Olympics a sign sport is the only thing that can make us feel good right now? __________ Sign up for our free newsletter at therestispolitics.com. Join The Rest Is Politics Plus: Start your free trial at therestispolitics.com to unlock exclusive bonus content – including Rory and Alastair's miniseries – plus ad-free listening, early access to episodes and live show tickets, an exclusive members' newsletter, discounted book prices, and a private chatroom on Discord. The Rest Is Politics is powered by Fuse Energy. To sign up and for terms and conditions, visit fuseenergy.com/politics. Get our exclusive NordVPN deal here ➼ nordvpn.com/restispolitics It's risk-free with Nord's 30 day money back guarantee ✅ Listen to 'If Anyone Builds It, Everyone Will Die' by Eliezer Yudkowsky and Nate Soares here: https://open.spotify.com/show/0p66zkQAAmmJ7PWsCSGgCT __________ Instagram: @restispolitics Twitter: @restispolitics Email: therestispolitics@goalhanger.com __________ Social Producer: Celine Charles Video Editor: Adam Thornton Assistant Producer: Daisy Alston-Horne Producer: Evan Green Senior Producer: Callum Hill General Manager: Tom Whiter Learn more about your ad choices. Visit podcastchoices.com/adchoices
A.I. is becoming smarter without much help from humans, and that should worry us all. Nate Soares, president of Machine Intelligence Research Institute (MIRI), joins host Krys Boyd to discuss what happens when A.I. brain power surpasses what humans are capable of, why we don't have the technology yet to understand what we're building, and why everything will be just fine … until it isn't. His book, co-written with Eliezer Yudkowsky, is “If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All.”This week, we're revisiting episodes that are all about A.I. Learn about your ad choices: dovetail.prx.org/ad-choices
Two years without discussing effective altruism -- did you miss it? Not as much as Vaden, surely. And probably a right bit more than Ben. Well, we're back in the game with a spicy one. Was EA a front for AI safety from the beginning? Did the leaders care not a wit for global poverty? Is Ben going to throw himself out window if Vaden keeps this up? We discuss Feedback on our introspection episode The motives of the EA founders The felicia forum Is this a conspiracy theory? EA's strategic ambiguity Bostromism, transhumanism, and AI safety EA funding The public/core divide and the funnel model Quotes new effective altruists tend to start off concerned about global poverty or animal suffering and then hear, take seriously, and often are convinced by the arguments for existential risk mitigation - Will MacAskill Existential risk isn't the most useful public face for effective altruism – everyone inc[l]uding Eliezer Yudkowsky agrees about that - Scott Alexander, 2015 Utilitymonster: GWWC is explicitly poverty-focused but high impact careers (HIC) is not. In fact, hardcore members of GWWC are heavily interested in x-risk, and I estimate that 10-15% of its general membership is as well. I'd take them seriously as a group for promoting utilitarianism in general. I'm a GWWC leader. [Redacted]: but HIC always seems to talk about things in terms of “lives saved”, ive never heard them mentioning other things to donate to. […] Utilitymonster: That's exactly the right thing for HIC to do. Talk about lives saved with their public face, let hardcore members hear about x-risk, and then, in the future, if some excellent x-risk opportunity arises, direct resources to x-risk. - From felicia forum. References Gleiberman's paper: https://medialibrary.uantwerpen.be/files/8518/61565cb6-e056-4e35-bd2e-d14d58e35231.pdf Old EA wikipedia page (web archive): https://web.archive.org/web/20170409171350/https://en.wikipedia.org/wiki/Effective_altruism Old CEA webpage (web archive): https://web.archive.org/web/20161219031827/https://www.centreforeffectivealtruism.org/fundraising/ Socials Follow us on Twitter at @IncrementsPod, @BennyChugg, @VadenMasrani Come join our discord server! DM us on twitter or send us an email to get a supersecret link Become a patreon subscriber here. Or give us one-time cash donations to help cover our lack of cash donations here. Click dem like buttons on youtube Let us funnel you into the core group of super secret patreon supporters. Send us an email at incrementspodcast@gmail.com
Vieraana Viivi Lehtinen, KRP: https://www.linkedin.com/in/viivi-l-692345168/ Kuinka päästä tietoturva-alalle töihin? 0x06 https://herrasmieshakkerit.fi/kuinka-paasta-tietoturva-alalle-toihin-0x06/ Cybercrime exit https://poliisi.fi/en/cybercrime-exit-en KRP:n koulutusvideo: Verkkorikos ei ole pränkki https://www.youtube.com/watch?v=3gS9VnjHwm4 Disobey-puhe: I Was Almost a Cybercriminal - Sergey Ichtchenko https://www.youtube.com/watch?v=Nh7OrFVyDo0 Vikasietotila-podcast https://vikasietoti.la/ If anyone builds it everyone dies - Eliezer Yudkowsky https://www.amazon.com/Anyone-Builds-Everyone-Dies-Superhuman/dp/0316595640 Peninsula hotel / Hong Kong https://www.peninsula.com/en/hong-kong/5-star-luxury-hotel-kowloon The Taste of Luxury: Bernard Arnault and the Moet-Hennessy Louis Vuitton Story https://www.amazon.com/Taste-Luxury-Bernard-Arnault-Moet-Hennessy/dp/0747512027 Art of Brick https://www.aotb.ee
In this episode Barry and Mike discuss “If Anyone Builds it, Everyone Dies: Why Superhuman AI Would Kill Us All” by Eliezer Yudkowsky and Nate Soares. They discuss the main arguments about the inevitability of our demise at the hands of superhuman intelligence and present a few alternatives to this doomsday scenario.
We read 'If Anyone Builds It, Everyone Dies' by Yudkowsky & Soares (so you don't have to) Watch this video at- https://youtu.be/IHTunMmNado?si=4RvOZ5hyUAE7NzSo We Read This So You Don't Have To Nov 16, 2025 We Read This (So You Don't Have To) We read If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All by Eliezer Yudkowsky & Nate Soares so you don't have to …but if you've ever wondered how building superhuman artificial intelligence could turn into humanity's last mistake, this episode might forever change how you think about technology, risk, and the future of intelligence. In this episode, we break down Yudkowsky & Soares's alarming thesis: when we build AI that out-thinks us, the default isn't friendly cooperation — it's misalignment, hidden objectives, and catastrophic loss of control. Modern AI isn't programmed in the old way; it's grown, resulting in systems whose goals we cannot fully predict or steer. The authors argue that unless humanity halts or radically redesigns the trajectory of large-scale AI development, we may be writing our own extinction notice.
This week we talk about floods, wildfires, and reinsurance companies.We also discuss the COP meetings, government capture, and air pollution.Recommended Book: If Anyone Builds It, Everyone Dies by Eliezer Yudkowsky and Nate Soares TranscriptThe urban area that contains India's capital city, New Delhi, called the National Capital Territory of Delhi, has a population of around 34.7 million people. That makes it the most populous city in the country, and one of the most populous cities in the world.Despite the many leaps India has made over the past few decades, in terms of economic growth and overall quality of life for residents, New Delhi continues to have absolutely abysmal air quality—experts at India's top research hospital have called New Delhi's air “severe and life-threatening,” and the level of toxic pollutants in the air, from cars and factories and from the crop-waste burning conducted by nearby farmers, can reach 20-times the recommended level for safe breathing.In mid-November 2025, the problem became so bad that the government told half its workers to work from home, because of the dangers represented by the air, and in the hope that doing so would remove some of the cars on the road and, thus, some of the pollution being generated in the area.Trucks spraying mist, using what are called anti-smog guns, along busy roads and pedestrian centers help—the mist keeping some of the pollution from cars from billowing into the air and becoming part of the regional problem, rather than an ultra-localized one, and pushing the pollutants that would otherwise get into people's lungs down to the ground—though the use of these mist-sprayers has been controversial, as there are accusations that they're primarily deployed near air-quality monitoring stations, and that those in charge put them there to make it seem like the overall air-quality is lower than it is, manipulating the stats so that their failure to improve practical air-quality isn't as evident.And in other regional news, just southeast across the Bay of Bengal, the Indonesian government, as of the day I'm recording this, is searching for the hundreds of people who are still missing following a period of unusually heavy rains. These rains have sparked floods and triggered mudslides that have blocked roads, damaged bridges, and forced the evacuation of entire villages. More than 300,000 people have been evacuated as of last weekend, and more rain is forecast for the coming days.The death toll of this round of heavy rainfall—the heaviest in the region in years—has already surpassed 440 people in Indonesia, with another 160 and 90 in Thailand and Vietnam, respectively, being reported by those countries' governments, from the same weather system.In Thailand, more than two million people were displaced by flooding, and the government had to deploy military assets, including helicopters launched from an aircraft carrier, to help rescue people from the roofs of buildings across nine provinces.In neighboring Malaysia, tens of thousands of people were forced into shelters as the same storm system barreled through, and Sri Lanka was hit with a cyclone that left at least 193 dead and more than 200 missing, marking one of the country's worst weather disasters in recent years.What I'd like to talk about today is the climatic moment we're at, as weather patterns change and in many cases, amplify, and how these sorts of extreme disasters are also causing untold, less reported upon but perhaps even more vital, for future policy shifts, at least, economic impacts.—The UN Conference of the Parties, or COP meetings, are high-level climate change conferences that have typically been attended by representatives from most governments each year, and where these representatives angle for various climate-related rules and policies, while also bragging about individual nations' climate-related accomplishments.In recent years, such policies have been less ambitious than in previous ones, in part because the initial surge of interest in preventing a 1.5 degrees C increase in average global temperatures is almost certainly no longer an option; climate models were somewhat accurate, but as with many things climate-related, seem to have actually been a little too optimistic—things got worse faster than anticipated, and now the general consensus is that we'll continue to shoot past 1.5 degrees C over the baseline level semi-regularly, and within a few years or a decade, that'll become our new normal.The ambition of the 2015 Paris Agreement is thus no longer an option. We don't yet have a new, generally acceptable—by all those governments and their respective interests—rallying cry, and one of the world's biggest emitters, the United States, is more or less absent at new climate-related meetings, except to periodically show up and lobby for lower renewables goals and an increase in subsidies for and policies that favor the fossil fuel industry.The increase in both number and potency of climate-influenced natural disasters is partly the result of this failure to act, and act forcefully and rapidly enough, by governments and by all the emitting industries they're meant to regulate.The cost of such disasters is skyrocketing—there are expected to be around $145 billion in insured losses, alone, in 2025, which is 6% higher than in 2024—and their human impact is booming as well, including deaths and injuries, but also the number of people being displaced, in some cases permanently, by these disasters.But none of that seems to move the needle much in some areas, in the face of entrenched interests, like the aforementioned fossil fuel industry, and the seeming inability of politicians in some nations to think and act beyond the needs of their next election cycle.That said, progress is still being made on many of these issues; it's just slower than it needs to be to reach previously set goals, like that now-defunct 1.5 degrees C ceiling.Most nations, beyond petro-states like Russia and those with fossil fuel industry-captured governments like the current US administration, have been deploying renewables, especially solar panels, at extraordinary rates. This is primarily the result of China's breakneck deployment of solar, which has offset a lot of energy growth that would have otherwise come from dirty sources like coal in the country, and which has led to a booming overproduction of panels that's allowed them to sell said panels cheap, overseas.Consequently, many nations, like Pakistan and a growing number of countries across Sub-Saharan African, have been buying as many cheap panels as they can afford and bypassing otherwise dirty and unreliable energy grids, creating arrays of microgrids, instead.Despite those notable absences, then, solar energy infrastructure installations have been increasing at staggering rates, and the first half of 2025 has seen the highest rate of capacity additions, yet—though China is still installing twice as much solar as the rest of the world, combined, at this point. Which is still valuable, as they still have a lot of dirty energy generation to offset as their energy needs increase, but more widely disseminated growth is generally seen to be better in the long-term—so the expansion into other parts of the world is arguably the bigger win, here.The economics of renewables may, at some point, convince even the skeptics and those who are politically opposed to the concept of renewables, rather than practically opposed to them, that it's time to change teams. Already, conservative parts of the US, like Texas, are becoming renewables boom-towns, quietly deploying wind and solar because they're often the best, cheapest, most resilient options, even as their politicians rail against them in public and vote for more fossil fuel subsidies.And it may be economics that eventually serve as the next nudge, or forceful shove on this movement toward renewables, as we're reaching a point at which real estate and the global construction industry, not to mention the larger financial system that underpins them and pretty much all other large-scale economic activities, are being not just impacted, but rattled at their roots, by climate change.In early November 2025, real estate listing company Zillow, the biggest such company in the US, stopped showing extreme weather risks for more than a million home sale listings on its site.It started showing these risk ratings in 2024, using data from a risk-modeling company called First Street, and the idea was to give potential buyers a sense of how at-risk a property they were considering buying might be when it comes to wildfires, floods, poor air quality, and other climate and pollution-related issues.Real estate agents hated these ratings, though, in part because there was no way to protest and change them, but also because, well, they might have an expensive coastal property listed that now showed potential buyers it was flood prone, if not today, in a couple of years. It might also show a beautiful mountain property that's uninsurable because of the risk of wildfire damage.A good heuristic for understanding the impact of global climate change is not to think in terms of warming, though that's often part of it, but rather thinking in terms of more radical temperature and weather swings.That means areas that were previously at little or no risk of flooding might suddenly be very at risk of absolutely devastating floods. And the same is true of storms, wildfires, and heat so intense people die just from being outside for an hour, and in which components of one's house might fry or melt.This move by Zillow, the appearance and removal of these risk scores, happened at the same time global insurers are warning that they may have to pull out of more areas, because it's simply no longer possible for them to do business in places where these sorts devastating weather events are happening so regularly, but often unpredictably, and with such intensity—and where the landscapes, ecologies, and homes are not made to withstand such things; all that stuff came of age or was built in another climate reality, so many such assets are simply not made for what's happening now, and what's coming.This is of course an issue for those who already own such assets—homes in newly flood-prone areas, for instance—because it means if there's a flood and a home owner loses their home, they may not be able to rebuild or get a payout that allows them to buy another home elsewhere. That leaves some of these assets stranded, and it leaves a lot of people with a huge chunk of their total resources permanently at risk, unable to move them, or unable to recoup most of their investment, shifting that money elsewhere. It also means entires industries could be at risk, especially banks and other financial institutions that provide loans for those who have purchased homes and other assets in such regions.An inability to get private insurance also means governments will be increasingly on the hook for issuing insurance of last resort to customers, which often costs more, but also, as we've seen with flood insurance in the US, means the government tends to lose a lot of money when increasingly common, major disasters occur on their soil.This isn't just a US thing, though; far from it. Global reinsurers, companies that provide insurance for insurance companies, and whose presence and participation in the market allow the insurance world to function, Swiss Re and Munich Re, recently said that uninsurable areas are growing around the world right now, and lacking some kind of fundamental change to address the climate paradigm shift, we could see a period of devastation in which rebuilding is unlikely or impossible, and a resultant period in which there's little or no new construction because no one wants to own a home or factory or other asset that cannot be insured—it's just not a smart investment.This isn't just a threat to individual home owners, then, it's potentially a threat to the whole of the global financial system, and every person and business attached to it, which in turn is a threat to global governance and the way property and economics work.There's a chance the worst-possible outcomes here can still be avoided, but with each new increase in global average temperature, the impacts become worse and less predictable, and the economics of simply making, protecting, and owning things become less and less favorable.Show Noteshttps://www.nytimes.com/2025/11/30/climate/zillow-climate-risk-scores-homes.htmlhttps://www.nytimes.com/2025/11/30/climate/climate-change-disinformation.htmlhttps://www.nytimes.com/2025/11/30/world/asia/india-delhi-pollution.htmlhttps://www.nytimes.com/2025/11/30/world/asia/flooding-indonesia-thailand-southeast-asia.htmlhttps://www.bbc.com/news/articles/c5y9ejley9dohttps://www.theguardian.com/environment/2025/nov/22/cop30-deal-inches-closer-to-end-of-fossil-fuel-era-after-bitter-standoffhttps://theconversation.com/the-world-lost-the-climate-gamble-now-it-faces-a-dangerous-new-reality-270392https://theconversation.com/earth-is-already-shooting-through-the-1-5-c-global-warming-limit-two-major-studies-show-249133https://www.404media.co/americas-polarization-has-become-the-worlds-side-hustle/https://www.cnbc.com/2025/08/08/climate-insurers-are-worried-the-world-could-soon-become-uninsurable-.htmlhttps://www.imd.org/ibyimd/sustainability/climate-change-the-emergence-of-uninsurable-areas-businesses-must-act-now-or-pay-later/https://www.jec.senate.gov/public/index.cfm/democrats/2024/12/climate-risks-present-a-significant-threat-to-the-u-s-insurance-and-housing-marketshttps://www.weforum.org/stories/2025/04/financial-system-warning-climate-nature-stories-this-week/https://www.weforum.org/stories/2025/05/costs-climate-disasters-145-billion-nature-climate-news/https://arstechnica.com/science/2025/11/solars-growth-in-us-almost-enough-to-offset-rising-energy-use/https://ember-energy.org/latest-updates/global-solar-installations-surge-64-in-first-half-of-2025/ This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit letsknowthings.substack.com/subscribe
In Locust Radio 31, Tish and Adam read poems from the forthcoming issue, discuss Trumpism and art in Venice, and try to unpack the editorial for Locust Review 13. Tish and Adam also listen to the song “Dortn” by Sister Wife Sex Strike. Discussed in this episode: Alma Allen; Suvrat Arora, “People are using AI to talk to God,” BBC (October 18, 2025); Editorial, “Lucky 13,” Locust Review 13 (Winter 2025/2026); Emily M. Bender, Alex Hanna, The AI Con: How to Fight Big Tech's Hype and Create the Future We Want (Harper Collins, 2025); Timothy Binkley, “Autonomous Creations: Birthing Intelligent Agents,” Leonardo 31.5 (1998), 333-336; Ben Davis, “What is the Mysterious New Group Behind Trump's Venice Biennale Pick?,” Artnet (November 25, 2025); Benoit Dillet, “Technofascism and the AI Stage of Late Capitalism,” Blog of the APA (American Philosophical Association), (March 10, 2025); Marcel Duchamp, Fountain (1917); Robert M. Geraci, "Apocalyptic AI: Religion and the Promise of Artificial Intelligence,” Journal of the American Academy of Religion 76.1 (March 2008), 138-166; Jesse Clyde Howard; Holly Lewis, “Towards AI Realism: Opening Notes on Machine Learning and Our Collective Future,” Spectre (June 7, 2024); Alex Press, “US Unions Take on Artificial Intelligence,” Jacobin (November 8, 2024); Michael A. Rosenthal, “Benjamin's Wager on Modernity: Gambling and the Arcades Project,” The Germanic Review: Literature, Culture, Theory 87.3 (2012), 261-278; Victor Tangermann, “AI Now Claiming to Be God,” Futurism (September 16, 2025); Adam Turl, “All is Concealed: CAM's Direct Drive,” West End Word (October 5, 2016); Adam Turl, “Selling Out,” Locust Review 13 (Winter 2025/2026); Tish Turl, “Elegy for the Faithful Mapmakers,” Locust Review 13 (Winter 2025/2026); Gareth Watkins, “AI: The New Aesthetics of Fascism,” New Socialist (February 9, 2025); Luke Winkie, “Lost Vegas,” Slate (November 18, 2025); Eliezer Yudkowsky, Nate Soares, If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All (Little, Brown and Company, 2025)
Nate Soares is president of the Machine Intelligence Research Institute and co-author, with Eliezer Yudkowsky, of If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All. He has been working in the field for over a decade, after previous experience at Microsoft and Google. In this week's conversation, Yascha Mounk and Nate Soares explore why AI is harder to control than traditional software, what happens when machines develop motivations, and at what point humans can no longer contain the potential catastrophe. If you have not yet signed up for our podcast, please do so now by following this link on your phone. Email: leonora.barclay@persuasion.community Podcast production by Jack Shields and Leonora Barclay. Connect with us! Spotify | Apple | Google X: @Yascha_Mounk & @JoinPersuasion YouTube: Yascha Mounk, Persuasion LinkedIn: Persuasion Community Learn more about your ad choices. Visit megaphone.fm/adchoices
A.I. is becoming smarter without much help from humans, and that should worry us all. Nate Soares, president of Machine Intelligence Research Institute (MIRI), joins host Krys Boyd to discuss what happens when A.I. brain power surpasses what humans are capable of, why we don't have the technology yet to understand what we're building, and why everything will be just fine … until it isn't. His book, co-written with Eliezer Yudkowsky, is “If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All.” Learn about your ad choices: dovetail.prx.org/ad-choices
In Rutger Bregman's first book, Utopia for Realists, the historian describes a rosy vision of the future – one with 15-hour work weeks, universal basic income and massive wealth redistribution.It's a vision that, in the age of artificial intelligence, now seems increasingly possible.But utopia is far from guaranteed. Many experts predict that AI will also lead to mass job loss, the development of new bioweapons and, potentially, the extinction of our species.So if you're building a technology that could either save the world or destroy it – is that a moral pursuit?These kinds of thorny questions are at the heart of Bregman's latest book, Moral Ambition. In a sweeping conversation that takes us from the invention of the birth control pill to the British Abolitionist movement, Bregman and I discuss what a good life looks like (spoiler: he thinks the death of work might not be such a bad thing) – and whether AI can help get us there.Mentioned: Moral Ambition, by Rutger BregmanUtopia for Realists, by Rutger Bregman If Anyone Builds It, Everyone Dies: The Case Against Superintelligent AI, by Eliezer Yudkowsky and Nate SoaresMachines Like Us is produced by Mitchell Stuart. Our theme song is by Chris Kelly. Video editing by Emily Graves. Our executive producer is James Milward. Special thanks to Angela Pacienza and the team at The Globe and Mail.Support for Machines Like Us is provided by CIFAR and the Max Bell School of Public Policy at McGill University. Hosted by Simplecast, an AdsWizz company. See pcm.adswizz.com for information about our collection and use of personal data for advertising.
Sam Bankman-Fried, FTX scandal, TESCERAL, Effective Altruism (EA), Utilitarianism, AGI, AI as a scam, Will MacAskill, Machine Intelligence Research Institute (MIRI), the Center for Applied Rationality (CFAR), Leverage Research, Peter Thiel, Eliezer Yudkowsky, Longtermism, Barbara Fried, Sanford, Lewis Terman, gifted kids, Fred Terman, eugenics, Anthropic, Rationalism, human potential movement. Landmark/est, MK-ULTRA, Zizians, cultsDavid's bookMusic by: Keith Allen Dennishttps://keithallendennis.bandcamp.com/ Hosted on Acast. See acast.com/privacy for more information.
Artificial intelligence has leapt from speculative theory to everyday tool with astonishing speed, promising breakthroughs in science, medicine, and the ways we learn, live, and work. But to some of its earliest researchers, the race toward superintelligence represents not progress but an existential threat, one that could end humanity as we know it.Eliezer Yudkowsky and Nate Soares, authors of If Anyone Builds It, Everyone Dies, join Oren to debate their claim that pursuing AI will end in human extinction. During the conversation, a skeptical Oren pushes them on whether meaningful safeguards are possible, what a realistic boundary between risk and progress might look like, and how society should judge the costs of stopping against the consequences of carrying on.
Today Razib talks to Nate Soares the President of the Machine Intelligence Research Institute (MIRI). He joined MIRI in 2014 and has since authored many of its core technical agendas, including foundational documents like Agent Foundations for Aligning Superintelligence with Human Interests. Prior to his work in AI research, Soares worked as a software engineer at Google. He holds a B.S. in computer science and economics from George Washington University. On this episode they discuss his new book, If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All, co-authored with Eliezer Yudkowsky. Soares and Yudkowsky make the stark case that the race to build superintelligent AI is a "suicide race" for humanity. Razib and Soares discuss how AI systems are "grown" rather than deliberately engineered, making them fundamentally opaque and uncontrollable. They explore a concrete extinction scenario, explain why even minimally misaligned goals could lead to human annihilation. Soares urges immediate cooperative action to prevent such a worst-case outcome.
A deadly 6.3 magnitude earthquake strikes Afghanistan, A man is charged with 11 attempted murders in a U.K. train attack, Iran vows to rebuild its nuclear sites, A Mexican mayor is shot and killed during the Day of the Dead festival, The BBC claims China threatened a U.K. university over Uyghur research, President Trump instructs the Pentagon to prepare for “possible action” in Nigeria, Trump's planned nuclear tests will reportedly be 'noncritical explosions', 21 states are among those suing the Trump administration over student loan forgiveness rules, France rejects a wealth tax and approves a holding company levy, Eliezer Yudkowsky critiques OpenAI's stated goals, and the LA Dodgers are MLB champions after an epic World Series. Sources: www.verity.news
Techno-philosopher Eliezer Yudkowsky recently went on Ezra Klein's podcast to argue that if we continue on our path toward superintelligent AI, these machines will destroy humanity. In this episode, Cal responds to Yudkowsky's argument point by point, concluding with a more general claim that these general styles of discussions suffer from what he calls “the philosopher's fallacy,” and are distracting us from real problems with AI that are actually afflicting us right now. He then answers listener questions about AI, responds to listener comments from an earlier AI episode, and ends by discussing Alpha schools, which claim to use AI to 2x the speed of education. Below are the questions covered in today's episode (with their timestamps). Get your questions answered by Cal! Here's the link: bit.ly/3U3sTvoVideo from today's episode: youtube.com/calnewportmediaDeep Dive: The Case Against Superintelligence [0.01]How should students think about “AI Literacy”? [1:06:35]Did AI blackmail an engineer to not turn it off? [1:09:06]Can I use AI to mask my laziness? [1:12:31]COMMENTS: Cal reads LM comments [1:16:58]CALL: Clarification on Lincoln Protocol [1:21:36]CAL REACTS: Are AI-Powered Schools the Future? [1:24:46]Links:Buy Cal's latest book, “Slow Productivity” at calnewport.com/slowGet a signed copy of Cal's “Slow Productivity” at peoplesbooktakoma.com/event/cal-newport/Cal's monthly book directory: bramses.notion.site/059db2641def4a88988b4d2cee4657ba?youtube.com/watch?v=2Nn0-kAE5c0alpha.school/the-program/astralcodexten.com/i/166959786/part-three-how-alpha-works-partThanks to our Sponsors: byloftie.com (Use code “DEEP20”)expressvpn.com/deepshopify.com/deepvanta.com/deepquestionsThanks to Jesse Miller for production, Jay Kerstens for the intro music, and Mark Miles for mastering. Hosted by Simplecast, an AdsWizz company. See pcm.adswizz.com for information about our collection and use of personal data for advertising.
Show Notes: Steve recounts his senior year at Harvard, and how he was torn between pursuing acting and philosophy. He graduated with a dual degree in philosophy and math but also found time to act in theater and participated in 20 shows. A Love of Theater and a Move to London Steve explains why the lack of a theater major at Harvard allowed him to explore acting more than a university with a theater major. He touches on his parents' concerns about his career prospects if he pursued acting, and his decision to apply to both acting and philosophy graduate schools. Steve discusses his rejection from all graduate schools and why he decided to move to London with friends Evan Cohn and Brad Rouse. He talks about his experience in London. Europe on $20 a Day Steve details his backpacking trip through Europe on a $20 a day budget, staying with friends from Harvard and high school. He mentions a job opportunity in Japan through the Japanese Ministry of Education and describes his three-year stint in Japan, working as a native English speaker for the Japanese Ministry of Education, and being immersed in Japanese culture. He shares his experiences of living in the countryside and reflects on the impact of living in a different culture, learning some Japanese, and making Japanese friends. He discusses the personal growth and self-reflection that came from his time in Japan, including his first steps off the "achiever track." On to Philosophy Graduate School When Steve returned to the U.S. he decided to apply to philosophy graduate schools again, this time with more success. He enrolled at the University of Michigan. However, he was miserable during grad school, which led him to seek therapy. Steve credits therapy with helping him make better choices in life. He discusses the competitive and prestigious nature of the Michigan philosophy department and the challenges of finishing his dissertation. He touches on the narrow and competitive aspects of pursuing a career in philosophy and shares his experience of finishing his dissertation and the support he received from a good co-thesis advisor. Kalamazoo College and Improv Steve describes his postdoc experience at Kalamazoo College, where he continued his improv hobby and formed his own improv group. He mentions a mockumentary-style improv movie called Comic Evangelists that premiered at the AFI Film Festival. Steve moved to Buffalo, Niagara University, and reflects on the challenges of adjusting to a non-research job. He discusses his continued therapy in Buffalo and the struggle with both societal and his own expectations of professional status, however, with the help of a friend, he came to the realization that he had "made it" in his current circumstances. Steve describes his acting career in Buffalo, including roles in Shakespeare in the Park and collaborating with a classmate, Ian Lithgow. A Speciality in Philosophy of Science Steve shares his personal life, including meeting his wife in 2009 and starting a family. He explains his specialty in philosophy of science, focusing on the math and precise questions in analytic philosophy. He discusses his early interest in AI and computational epistemology, including the ethics of AI and the superintelligence worry. Steve describes his involvement in a group that discusses the moral status of digital minds and AI alignment. Aligning AI with Human Interests Steve reflects on the challenges of aligning AI with human interests and the potential existential risks of advanced AI. He shares his concerns about the future of AI and the potential for AI to have moral status. He touches on the superintelligence concern and the challenges of aligning AI with human goals. Steve mentions the work of Eliezer Yudkowsky and the importance of governance and alignment in AI development. He reflects on the broader implications of AI for humanity and the need for careful consideration of long-term risks. Harvard Reflections Steve mentions Math 45 and how it kicked his butt, and his core classes included jazz, an acting class and clown improv with Jay Nichols. Timestamps: 01:43: Dilemma Between Acting and Philosophy 03:44: Rejection and Move to London 07:09: Life in Japan and Cultural Insights 12:19: Return to Academia and Grad School Challenges 20:09: Therapy and Personal Growth 22:06: Transition to Buffalo and Philosophy Career 26:54: Philosophy of Science and AI Ethics 33:20: Future Concerns and AI Predictions 55:17: Reflections on Career and Personal Growth Links: Steve's Website: https://stevepetersen.net/ On AI Superintelligence: If Anyone Builds it, Everyone Dies Superintelligence The Alignment Problem Some places to donate: The Long-Term Future Fund Open Philanthropy On improv Impro Upright Citizens Brigade Comedy Improvisation Manual Featured Non-profit: The featured non-profit of this week's episode is brought to you by Rich Buery who reports: “Hi, I'm Rich Buery, class of 1992. The featured nonprofit of this episode of The 92 Report is imentor. imentor is a powerful youth mentoring organization that connects volunteers with high school students and prepares them on the path to and through college. Mentors stay with the students through the last two years of high school and on the beginning of their college journey. I helped found imentor over 25 years ago and served as its founding executive director, and I am proud that over the last two decades, I've remained on the board of directors. It's truly a great organization. They need donors and they need volunteers. You can learn more about their work@www.imentor.org That's www, dot i m, e n, t, O, r.org, and now here is Will Bachman with this week's episode. To learn more about their work, visit: www.imentor.org.
Eliezer Yudkowsky is an AI researcher, decision theorist, and founder of the Machine Intelligence Research Institute. Is AI our greatest hope or our final mistake? For all its promise to revolutionize human life, there's a growing fear that artificial intelligence could end it altogether. How grounded are these fears, how close are we to losing control, and is there still time to change course before it's too late Expect to learn the problem with building superhuman AI, why AI would have goals we haven't programmed into it, if there is such a thing as Ai benevolence, what the actual goals of super-intelligent AI are and how far away it is, if LLMs are actually dangerous and their ability to become a super AI, how god we are at predicting the future of AI, if extinction if possible with the development of AI, and much more… Sponsors: See discounts for all the products I use and recommend: https://chriswillx.com/deals Get 15% off your first order of Intake's magnetic nasal strips at https://intakebreathing.com/modernwisdom Get 10% discount on all Gymshark's products at https://gym.sh/modernwisdom (use code MODERNWISDOM10) Get 4 extra months of Surfshark VPN at https://surfshark.com/modernwisdom Extra Stuff: Get my free reading list of 100 books to read before you die: https://chriswillx.com/books Try my productivity energy drink Neutonic: https://neutonic.com/modernwisdom Episodes You Might Enjoy: #577 - David Goggins - This Is How To Master Your Life: https://tinyurl.com/43hv6y59 #712 - Dr Jordan Peterson - How To Destroy Your Negative Beliefs: https://tinyurl.com/2rtz7avf #700 - Dr Andrew Huberman - The Secret Tools To Hack Your Brain: https://tinyurl.com/3ccn5vkp - Get In Touch: Instagram: https://www.instagram.com/chriswillx Twitter: https://www.twitter.com/chriswillx YouTube: https://www.youtube.com/modernwisdompodcast Email: https://chriswillx.com/contact - Learn more about your ad choices. Visit megaphone.fm/adchoices
Teaser ... Why Liron became a Yudkowskian ... Eliezer Yudkowsky's vision of AI apocalypse ... Does intelligence want power? ... Decoding Yudkowsky's key Darwinian metaphor ... Is doomerism crowding out other AI worries? ... Liron: The silent majority is very AI anxious ... Heading to Overtime ...
Eliezer Yudkowsky is as afraid as you could possibly be. He makes his case.Yudkowsky is a pioneer of A.I. safety research, who started warning about the existential risks of the technology decades ago, – influencing a lot of leading figures in the field. But over the last couple of years, talk of an A.I. apocalypse has become a little passé. Many of the people Yudkowsky influenced have gone on to work for A.I. companies, and those companies are racing ahead to build the superintelligent systems Yudkowsky thought humans should never create. But Yudkowsky is still out there sounding the alarm. He has a new book out, co-written with Nate Soares, “If Anyone Builds It, Everyone Dies,” trying to warn the world before it's too late.So what does Yudkowsky see that most of us don't? What makes him so certain? And why does he think he hasn't been able to persuade more people?Mentioned:Oversight of A.I.: Rules for Artificial IntelligenceIf Anyone Builds It, Everyone Dies by Eliezer Yudkowsky and Nate Soares“A Teen Was Suicidal. ChatGPT Was the Friend He Confided In.” by Kashmir HillBook Recommendations:A Step Farther Out by Jerry PournelleJudgment under Uncertainty by Daniel Kahneman, Paul Slovic, and Amos TverskyProbability Theory by E. T. JaynesThoughts? Guest suggestions? Email us at ezrakleinshow@nytimes.com.You can find transcripts (posted midday) and more episodes of “The Ezra Klein Show” at nytimes.com/ezra-klein-podcast, and you can find Ezra on Twitter @ezraklein. Book recommendations from all our guests are listed at https://www.nytimes.com/article/ezra-klein-show-book-recs.This episode of “The Ezra Klein Show” was produced by Rollin Hu. Fact-checking by Michelle Harris. Our senior engineer is Jeff Geld, with additional mixing by Aman Sahota. Our executive producer is Claire Gordon. The show's production team also includes Marie Cascione, Annie Galvin, Kristin Lin, Jack McCordick, Marina King and Jan Kobal. Original music by Pat McCusker. Audience strategy by Kristina Samulewski and Shannon Busta. The director of New York Times Opinion Audio is Annie-Rose Strasser. Special thanks to Helen Toner and Jeffrey Ladish. Subscribe today at nytimes.com/podcasts or on Apple Podcasts and Spotify. You can also subscribe via your favorite podcast app here https://www.nytimes.com/activate-access/audio?source=podcatcher. For more podcasts and narrated articles, download The New York Times app at nytimes.com/app.
Jim talks with Nate Soares about the ideas in his and Eliezer Yudkowsky's book If Anybody Builds It, Everybody Dies: Why Superhuman AI Would Kill Us All. They discuss the book's claim that mitigating existential AI risk should be a top global priority, the idea that LLMs are grown, the opacity of deep learning networks, the Golden Gate activation vector, whether our understanding of deep learning networks might improve enough to prevent catastrophe, goodness as a narrow target, the alignment problem, the problem of pointing minds, whether LLMs are just stochastic parrots, why predicting a corpus often requires more mental machinery than creating a corpus, depth & generalization of skills, wanting as an effective strategy, goal orientation, limitations of training goal pursuit, transient limitations of current AI, protein folding and AlphaFold, the riskiness of automating alignment research, the correlation between capability and more coherent drives, why the authors anchored their argument on transformers & LLMs, the inversion of Moravec's paradox, the geopolitical multipolar trap, making world leaders aware of the issues, a treaty to ban the race to superintelligence, the specific terms of the proposed treaty, a comparison with banning uranium enrichment, why Jim tentatively thinks this proposal is a mistake, a priesthood of the power supply, whether attention is a zero-sum game, and much more. Episode Transcript "Psyop or Insanity or ...? Peter Thiel, the Antichrist, and Our Collapsing Epistemic Commons," by Jim Rutt "On Targeted Manipulation and Deception when Optimizing LLMs for User Feedback," by Marcus Williams et al. Attention Sinks and Compression Valleys in LLMs are Two Sides of the Same Coin," by Enrique Queipo-de-Llano et al. JRS EP 217 - Ben Goertzel on a New Framework for AGI "A Tentative Draft of a Treaty, With Annotations" Nate Soares is the President of the Machine Intelligence Research Institute. He has been working in the field for over a decade, after previous experience at Microsoft and Google. Soares is the author of a large body of technical and semi-technical writing on AI alignment, including foundational work on value learning, decision theory, and power-seeking incentives in smarter-than-human AIs.
Craig Kaplan has been thinking about superintelligence longer than most. He bought the URL superintelligence.com back in 2006, and many years before that, in the late 1980s, he co-authored a series of papers with one of the founding fathers of AI, Herbert Simon.Craig started his career as a scientist with IBM, and later founded and ran a venture-backed company called PredictWallStreet that brought the wisdom of the crowd to Wall Street, and improved the performance of leading hedge funds. He sold that company in 2020, and now spends his time working out how to make the first superintelligence safe. As he puts it, he wants to reduce P(Doom) and increase P(Zoom).Selected follow-ups:iQ CompanyHerbert A. Simon - WikipediaAmara's Law and Its Place in the Future of Tech - Pohan LinPredict Wall StreetThe Society of Mind - book by Marvin MinskyAI 'godfather' Geoffrey Hinton warns of dangers as he quits Google - BBC NewsStatement on AI Risk - Center for AI SafetyI've Spent My Life Measuring Risk. AI Rings Every One of My Alarm Bells - Paul Tudor JonesSecrets of Software Quality: 40 Innovations from IBM - book by Craig KaplanLondon Futurists Podcast episode featuring David BrinReason in human affairs - book by Herbert SimonUS and China will intervene to halt ‘suicide race' of AGI – Max TegmarkIf Anybody Builds It, Everyone Dies - book by Eliezer Yudkowsky and Nate SoaresAGI-25 - conference in ReykjavikThe First Global Brain Workshop - Brussels 2001Center for Integrated CognitionPaul S. RosenbloomTatiana Shavrina, MetaHenry Minsky launches AI startup inspired by father's MIT researchMusic: Spike Protein, by Koi Discovery, available under CC0 1.0 Public Domain Declaration
Jim talks with Joe Edelman about the ideas in the Meaning Alignment Institute's recent paper "Full Stack Alignment: Co-Aligning AI and Institutions with Thick Models of Value." They discuss pluralism as a core principle in designing social systems, the informational basis for alignment, how preferential models fail to capture what people truly care about, the limitations of markets and voting as preference-based systems, critiques of text-based approaches in LLMs, thick models of value, values as attentional policies, AI assistants as potential vectors for manipulation, the need for reputation systems and factual grounding, the "super negotiator" project for better contract negotiation, multipolar traps, moral graph elicitation, starting with membranes, Moloch-free zones, unintended consequences and lessons from early Internet optimism, concentration of power as a key danger, co-optation risks, and much more. Episode Transcript "A Minimum Viable Metaphysics," by Jim Rutt (Substack) Jim's Substack JRS Currents 080: Joe Edelman and Ellie Hain on Rebuilding Meaning Meaning Alignment Institute If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All, by Eliezer Yudkowsky and Nate Soares "Full Stack Alignment: Co-aligning AI and Institutions with Thick Models of Value," by Joe Edelman et al. "What Are Human Values and How Do We Align AI to Them?" by Oliver Klingefjord, Ryan Lowe, and Joe Edelman Joe Edelman has spent much of his life trying to understand how ML systems and markets could change, retaining their many benefits but avoiding their characteristic problems: of atomization, and of servicing shallow desires over deeper needs. Along the way this led him to formulate theories of human meaning and values (https://arxiv.org/abs/2404.10636) and study models of societal transformation (https://www.full-stack-alignment.ai/paper) as well as inventing the meaning-based metrics used at CouchSurfing, Facebook, and Apple, co-founding the Center for Humane Technology and the Meaning Alignment Institute, and inventing new democratic systems (https://arxiv.org/abs/2404.10636). He's currently one of the PIs leading the Full-Stack Alignment program at the Meaning Alignment Institute, with a network of more than 50 researchers at universities and corporate labs working on these issues.
The story of how Geoffrey Hinton became “the godfather of AI” has reached mythic status in the tech world.While he was at the University of Toronto, Hinton pioneered the neural network research that would become the backbone of modern AI. (One of his students, Ilya Sutskever, went on to be one of OpenAI's most influential scientific minds.) In 2013, Hinton left the academy and went to work for Google, eventually winning both a Turing Award and a Nobel Prize.I think it's fair to say that artificial intelligence as we know it, may not exist without Geoffrey Hinton.But Hinton may be even more famous for what he did next. In 2023, he left Google and began a campaign to convince governments, corporations and citizens that his life's work – this thing he helped build – might lead to our collective extinction. And that moment may be closer than we think, because Hinton believes AI may already be conscious.But even though his warnings are getting more dire by the day, the AI industry is only getting bigger, and most governments, including Canada's, seem reluctant to get in the way.So I wanted to ask Hinton: If we keep going down this path, what will become of us?Mentioned:If Anyone Builds It, Everyone Dies: The Case Against Superintelligent AI, by Eliezer Yudkowsky and Nate SoaresAgentic Misalignment: How LLMs could be insider threats, by AnthropicMachines Like Us is produced by Mitchell Stuart. Our theme song is by Chris Kelly. Video editing by Emily Graves. Our executive producer is James Milward. Special thanks to Angela Pacienza and the team at The Globe and Mail.Support for Machines Like Us is provided by CIFAR and the Max Bell School of Public Policy at McGill University. Hosted by Simplecast, an AdsWizz company. See pcm.adswizz.com for information about our collection and use of personal data for advertising.
Jonathan Newman returns to join Bob in a critique of Eliezer Yudkowsky's viral theory of investment bubbles. Yudkowsky states that the bad investment during bubbles should be felt before the bubble pops, not after. They argue that his perspective—while clever—fails to consider the Austrian insights on capital structure, time preference, and the business cycle. They use analogies from apple trees to magic mushrooms to show why Austrian economics provides the clearest explanation for booms, busts, and the pain that follows.Eliezer Yudkowsky's Theory on Investment Bubbles: Mises.org/HAP520aBob's Article "Correcting Yudkowsky on the Boom": Mises.org/HAP520bBob's on The Importance of Capital Theory: Mises.org/HAP520cJoe Salerno on Austrian Business Cycle Theory: Mises.org/HAP520dDr. Newman's QJAE Article on Credit Cycles: Mises.org/HAP520eThe Mises Institute is giving away 100,000 copies of Hayek for the 21st Century. Get your free copy at Mises.org/HAPodFree
Jonathan Newman returns to join Bob in a critique of Eliezer Yudkowsky's viral theory of investment bubbles. Yudkowsky states that the bad investment during bubbles should be felt before the bubble pops, not after. They argue that his perspective—while clever—fails to consider the Austrian insights on capital structure, time preference, and the business cycle. They use analogies from apple trees to magic mushrooms to show why Austrian economics provides the clearest explanation for booms, busts, and the pain that follows.Eliezer Yudkowsky's Theory on Investment Bubbles: Mises.org/HAP520aBob's Article "Correcting Yudkowsky on the Boom": Mises.org/HAP520bBob's on The Importance of Capital Theory: Mises.org/HAP520cJoe Salerno on Austrian Business Cycle Theory: Mises.org/HAP520dDr. Newman's QJAE Article on Credit Cycles: Mises.org/HAP520eThe Mises Institute is giving away 100,000 copies of Hayek for the 21st Century. Get your free copy at Mises.org/HAPodFree
On this week's episode, I'm joined by Nate Soares to talk about his new book, cowritten with Eliezer Yudkowsky, If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All. It's a fascinating book—some will say fearmongering and sensationalist; I, frankly, think they're overly optimistic about our ability to constrain the development of general intelligence in AI—in large part because of how it's structured. Each chapter is preceded by a fable of sorts about the nature of intelligence and the desires of intelligent beings that look and think very differently from humans. The point in each of these passages is less that AI will want to eliminate humanity and more that it might do so incidentally, through natural processes of resource acquisition. This made me think about how AI is typically portrayed in film; it is all too often a Terminator-style scenario, where the intelligence is antagonistic in human ways and for human reasons. We talked some about how storytellers could do a better job of thinking about AI as it might actually exist versus how it might be like us; Ex Machina is a movie that came in for special discussion due to the thoughtful nature of the treatment of its robotic antagonist's desires. If this episode made you think, I hope you share it with a friend!
What are the best strategies for addressing extreme risks from artificial superintelligence? In this 4-hour conversation, decision theorist Eliezer Yudkowsky and computer scientist Mark Miller discuss their cruxes for disagreement. They examine the future of AI, existential risk, and whether alignment is even possible. Topics include AI risk scenarios, coalition dynamics, secure systems like seL4, hardware exploits like Rowhammer, molecular engineering with AlphaFold, and historical analogies like nuclear arms control. They explore superintelligence governance, multipolar vs singleton futures, and the philosophical challenges of trust, verification, and control in a post-AGI world.Moderated by Christine Peterson, the discussion seeks the least risky strategy for reaching a preferred state amid superintelligent AI risks. Yudkowsky warns of catastrophic outcomes if AGI is not controlled, while Miller advocates decentralizing power and preserving human institutions as AI evolves.The conversation spans AI collaboration, secure operating frameworks, cryptographic separation, and lessons from nuclear non-proliferation. Despite their differences, both aim for a future where AI benefits humanity without posing existential threats. Hosted on Acast. See acast.com/privacy for more information.
Nate Soares, president of the Machine Intelligence Research Institute and the co-author (with Eliezer Yudkowsky) of If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All (Little, Brown and Company, 2025), talks about why he worries that AI "superintelligence" will lead to catastrophic outcomes, and what safeguards he recommends to prevent this.
Sam Harris speaks with Eliezer Yudkowsky and Nate Soares about their new book, If Anyone Builds It, Everyone Dies: The Case Against Superintelligent AI. They discuss the alignment problem, ChatGPT and recent advances in AI, the Turing Test, the possibility of AI developing survival instincts, hallucinations and deception in LLMs, why many prominent voices in tech remain skeptical of the dangers of superintelligent AI, the timeline for superintelligence, real-world consequences of current AI systems, the imaginary line between the internet and reality, why Eliezer and Nate believe superintelligent AI would necessarily end humanity, how we might avoid an AI-driven catastrophe, the Fermi paradox, and other topics. If the Making Sense podcast logo in your player is BLACK, you can SUBSCRIBE to gain access to all full-length episodes at samharris.org/subscribe.
Apple's yearly iPhone event took place this week, and it left us asking, Is Apple losing the juice? We break down all the new products the company announced and discuss where it goes from here. Then, Eliezer Yudkowsky, one of the most fascinating people in A.I., has a new book coming out: “If Anyone Builds It, Everyone Dies.” He joins us to make the case for why A.I. development should be shut down now, long before we reach superintelligence, and how he thinks that could happen.Guests:Eliezer Yudkowsky, founder of Machine Intelligence Research Institute and a co-author of “If Anyone Builds It, Everyone Dies”Additional Reading: A.I.'s Prophet of Doom Wants to Shut It All DownAI as Normal Technology, revisitedApple's misunderstood crossbody iPhone strap might be the best I've seen We want to hear from you. Email us at hardfork@nytimes.com. Find “Hard Fork” on YouTube and TikTok. Unlock full access to New York Times podcasts and explore everything from politics to pop culture. Subscribe today at nytimes.com/podcasts or on Apple Podcasts and Spotify.
I. Eliezer Yudkowsky's Machine Intelligence Research Institute is the original AI safety org. But the original isn't always the best - how is Mesopotamia doing these days? As money, brainpower, and prestige pour into the field, MIRI remains what it always was - a group of loosely-organized weird people, one of whom cannot be convinced to stop wearing a sparkly top hat in public. So when I was doing AI grantmaking last year, I asked them - why should I fund you instead of the guys with the army of bright-eyed Harvard grads, or the guys who just got Geoffrey Hinton as their celebrity spokesperson? What do you have that they don't? MIRI answered: moral clarity. Most people in AI safety (including me) are uncertain and confused and looking for least-bad incremental solutions. We think AI will probably be an exciting and transformative technology, but there's some chance, 5 or 15 or 30 percent, that it might turn against humanity in a catastrophic way. Or, if it doesn't, that there will be something less catastrophic but still bad - maybe humanity gradually fading into the background, the same way kings and nobles faded into the background during the modern era. This is scary, but AI is coming whether we like it or not, and probably there are also potential risks from delaying too hard. We're not sure exactly what to do, but for now we want to build a firm foundation for reacting to any future threat. That means keeping AI companies honest and transparent, helping responsible companies like Anthropic stay in the race, and investing in understanding AI goal structures and the ways that AIs interpret our commands. Then at some point in the future, we'll be close enough to the actually-scary AI that we can understand the threat model more clearly, get more popular buy-in, and decide what to do next. MIRI thinks this is pathetic - like trying to protect against an asteroid impact by wearing a hard hat. They're kind of cagey about their own probability of AI wiping out humanity, but it seems to be somewhere around 95 - 99%. They think plausibly-achievable gains in company responsibility, regulation quality, and AI scholarship are orders of magnitude too weak to seriously address the problem, and they don't expect enough of a “warning shot” that they feel comfortable kicking the can down the road until everything becomes clear and action is easy. They suggest banning all AI capabilities research immediately, to be restarted only in some distant future when the situation looks more promising. Both sides honestly believe their position and don't want to modulate their message for PR reasons. But both sides, coincidentally, think that their message is better PR. The incrementalists think a moderate, cautious approach keeps bridges open with academia, industry, government, and other actors that prefer normal clean-shaven interlocutors who don't emit spittle whenever they talk. MIRI thinks that the public is sick of focus-group-tested mealy-mouthed bullshit, but might be ready to rise up against AI if someone presented the case in a clear and unambivalent way. Now Yudkowsky and his co-author, MIRI president Nate Soares, have reached new heights of unambivalence with their new book, If Anyone Builds It, Everyone Dies (release date September 16, currently available for preorder). https://www.astralcodexten.com/p/book-review-if-anyone-builds-it-everyone
Eliezer Yudkowsky is a decision theorist, computer scientist, and author who co-founded and leads research at the Machine Intelligence Research Institute. He is best known for his work on the alignment problem—how and whether we can ensure that AI is aligned with human values to avoid catastrophe and harness its power. In this episode, Robinson and Eliezer run the gamut on questions related to AI and the danger it poses to human civilization as we know it. More particularly, they discuss the alignment problem, gradient descent, consciousness, the singularity, cyborgs, ChatGPT, OpenAI, Anthropic, Claude, how long we have until doomsday, whether it can be averted, and the various reasons why and ways in which AI might wipe out human life on earth.The Machine Intelligence Research Institute: https://intelligence.org/about/Eliezer's X Account: https://x.com/ESYudkowsky?ref_src=twsrc%5Egoogle%7Ctwcamp%5Eserp%7Ctwgr%5EauthorOUTLINE00:00:00 Introduction00:00:43 The Default Condition for AI's Takeover00:06:36 Could a Future AI Country Be Our Trade Partner?00:11:18 What Is Artificial Intelligence?00:21:23 Why AIs Having Goals Could Mean the End of Humanity00:29:34 What Is the Alignment Problem?00:34:11 How To Avoid AI Apocalypse00:40:25 Would Cyborgs Eliminate Humanity?00:47:55 AI and the Problem of Gradient Descent00:55:24 How Do We Solve the Alignment Problem?01:00:50 How Anthropic's AI Freed Itself from Human Control01:08:56 The Pseudo-Alignment Problem01:19:28 Why Are People Wrong About AI Not Taking Over the World?01:23:23 How Certain Is It that AI Will Wipe Out Humanity?01:38:35 Is Eliezer Yudkowski Wrong About The AI Apocalypse01:42:04 Do AI Corporations Control the Fate of Humanity?01:43:49 How To Convince the President Not to Let AI Kill Us All01:52:01 How Will ChatGPT's Descendants Wipe Out Humanity?02:24:11 Could AI Destroy us with New Science?02:39:37 Could AI Destroy us with Advanced Biology?02:47:29 How Will AI Actually Destroy Humanity?Robinson's Website: http://robinsonerhardt.comRobinson Erhardt researches symbolic logic and the foundations of mathematics at Stanford University.
Have you ever read Harry Potter and the Methods of Rationality?? Perhaps spent too much money on a self help workshop seminar? Join us as we talk about Eliezer Yudkowsky and his masterpiece of fiction. Where will this story truly lead us in this tale of rational magic and science. With our last episode on the topic trigger warning for some bad mental health. Thanks for listening and remember to like, rate, review, and email us at: cultscryptidsconspiracies@gmail.com or tweet us at @C3Podcast. We have some of our sources for research here: http://tinyurl.com/CristinaSourcesAlso check out our Patreon: www.patreon.com/cultscryptidsconspiracies. Thank you to T.J. Shirley for our theme
A lot of the people designing America's technology and close to the center of American power believe some deeply weird shit. We already talked to journalist Gil Duran about the Nerd Reich, the rise of the destructive anti-democratic ideology. In this episode, we dive into another weird section of Silicon Valley: the cult of Rationalism.Max Read, the journalist behind the Read Max Substack, is here to help us through it. Rationalism is responsible for a lot more than you might think and Read lays out how it's influenced the world we live in today and how it created the environment for a cult that's got a body count.Defining rationalism: “Something between a movement, a community, and a self-help program.”Eliezer Yudkowsky and the dangers of AIWhat the hell is AGI?The Singleton Guide to Global GovernanceThe danger of thought experimentsAs always, follow the moneyVulgar bayesianismWhat's a Zizian?Sith VegansAnselm: Ontological Argument for God's ExistenceSBF and Effective AltruismREAD MAX!The Zizians and the Rationalist death cultsPausing AI Developments Isn't Enough. We Need to Shut it All Down - Eliezer Yudkowsky's TIME Magazine pieceExplaining Roko's Basilisk, the Thought Experiment That Brought Elon Musk and Grimes TogetherThe Delirious, Violent, Impossible True Story of the ZiziansThe Government Knows AGI is Coming | The Ezra Klein ShowThe archived ‘Is Trump Racist' rational postSupport this show http://supporter.acast.com/warcollege. Hosted on Acast. See acast.com/privacy for more information.
Part one of our two-part investigation into the Rationalist cult “The Zizians.” We start with the killing of a border patrol officer and make our way back into the belly of the beast: Silicon Valley. Featuring: Harry Potter fanfic, samurai swords, Guy Fawkes masks, Blake Masters, Bayesian probability, and Eliezer Yudkowsky. Infohazard warning: some of your least favs will be implicated. Discover more episodes at podcast.trueanon.com
© 2020-2026 Ivy Podcast Discovery LLC
