Podcasts about Toby Ord

The 2020s have so far been marked by pandemic, war, and startling technological breakthroughs. Conversations around climate disaster, great-power conflict, and malicious AI are seemingly everywhere. It's enough to make anyone feel like the end might be near. Toby Ord has made it his mission to figure out just how close we are to catastrophe — and maybe not close at all!Ord is the author of the 2020 book, The Precipice: Existential Risk and the Future of Humanity. Back then, I interviewed Ord on the American Enterprise Institute's Political Economy podcast, and you can listen to that episode here. In 2024, he delivered his talk, The Precipice Revisited, in which he reassessed his outlook on the biggest threats facing humanity.Today on Faster, Please — The Podcast, Ord and I address the lessons of Covid, our risk of nuclear war, potential pathways for AI, and much more.Ord is a senior researcher at Oxford University. He has previously advised the UN, World Health Organization, World Economic Forum, and the office of the UK Prime Minister.In This Episode* Climate change (1:30)* Nuclear energy (6:14)* Nuclear war (8:00)* Pandemic (10:19)* Killer AI (15:07)* Artificial General Intelligence (21:01)Below is a lightly edited transcript of our conversation. Climate change (1:30). . . the two worst pathways, we're pretty clearly not on, and so that's pretty good news that we're kind of headed more towards one of the better pathways in terms of the emissions that we'll put out there.Pethokoukis: Let's just start out by taking a brief tour through the existential landscape and how you see it now versus when you first wrote the book The Precipice, which I've mentioned frequently in my writings. I love that book, love to see a sequel at some point, maybe one's in the works . . . but let's start with the existential risk, which has dominated many people's thinking for the past quarter-century, which is climate change.My sense is, not just you, but many people are somewhat less worried than they were five years ago, 10 years ago. Perhaps they see at least the most extreme outcomes less likely. How do you see it?Ord: I would agree with that. I'm not sure that everyone sees it that way, but there were two really big and good pieces of news on climate that were rarely reported in the media. One of them is that there's the question about how many emissions there'll be. We don't know how much carbon humanity will emit into the atmosphere before we get it under control, and there are these different emissions pathways, these RCP 4.5 and things like this you'll have heard of. And often, when people would give a sketch of how bad things could be, they would talk about RCP 8.5, which is the worst of these pathways, and we're very clearly not on that, and we're also, I think pretty clearly now, not on RCP 6, either. So the two worst pathways, we're pretty clearly not on, and so that's pretty good news that we're kind of headed more towards one of the better pathways in terms of the emissions that we'll put out there.What are we doing right?Ultimately, some of those pathways were based on business-as-usual ideas that there wouldn't be climate change as one of the biggest issues in the international political sphere over decades. So ultimately, nations have been switching over to renewables and low-carbon forms of power, which is good news. They could be doing it much more of it, but it's still good news. Back when we initially created these things, I think we would've been surprised and happy to find out that we were going to end up among the better two pathways instead of the worst ones.The other big one is that, as well as how much we'll admit, there's the question of how bad is it to have a certain amount of carbon in the atmosphere? In particular, how much warming does it produce? And this is something of which there's been massive uncertainty. The general idea is that we're trying to predict, if we were to double the amount of carbon in the atmosphere compared to pre-industrial times, how many degrees of warming would there be? The best guess since the year I was born, 1979, has been three degrees of warming, but the uncertainty has been somewhere between one and a half degrees and four and a half.Is that Celsius or Fahrenheit, by the way?This is all Celsius. The climate community has kept the same uncertainty from 1979 all the way up to 2020, and it's a wild level of uncertainty: Four and a half degrees of warming is three times one and a half degrees of warming, so the range is up to triple these levels of degrees of warming based on this amount of carbon. So massive uncertainty that hadn't changed over many decades.Now they've actually revised that and have actually brought in the range of uncertainty. Now they're pretty sure that it's somewhere between two and a half and four degrees, and this is based on better understanding of climate feedbacks. This is good news if you're concerned about worst-case climate change. It's saying it's closer to the central estimate than we'd previously thought, whereas previously we thought that there was a pretty high chance that it could even be higher than four and a half degrees of warming.When you hear these targets of one and a half degrees of warming or two degrees of warming, they sound quite precise, but in reality, we were just so uncertain of how much warming would follow from any particular amount of emissions that it was very hard to know. And that could mean that things are better than we'd thought, but it could also mean things could be much worse. And if you are concerned about existential risks from climate change, then those kind of tail events where it's much worse than we would've thought the things would really get, and we're now pretty sure that we're not on one of those extreme emissions pathways and also that we're not in a world where the temperature is extremely sensitive to those emissions.Nuclear energy (6:14)Ultimately, when it comes to the deaths caused by different power sources, coal . . . killed many more people than nuclear does — much, much more . . .What do you make of this emerging nuclear power revival you're seeing across Europe, Asia, and in the United States? At least the United States it's partially being driven by the need for more power for these AI data centers. How does it change your perception of risk in a world where many rich countries, or maybe even not-so-rich countries, start re-embracing nuclear energy?In terms of the local risks with the power plants, so risks of meltdown or other types of harmful radiation leak, I'm not too concerned about that. Ultimately, when it comes to the deaths caused by different power sources, coal, even setting aside global warming, just through particulates being produced in the soot, killed many more people than nuclear does — much, much more, and so nuclear is a pretty safe form of energy production as it happens, contrary to popular perception. So I'm in favor of that. But the proliferation concerns, if it is countries that didn't already have nuclear power, then the possibility that they would be able to use that to start a weapons program would be concerning.And as sort of a mechanism for more clean energy. Do you view nuclear as clean energy?Yes, I think so. It's certainly not carbon-producing energy. I think that it has various downsides, including the difficulty of knowing exactly what to do with the fuel, that will be a very long lasting problem. But I think it's become clear that the problems caused by other forms of energy are much larger and we should switch to the thing that has fewer problems, rather than more problems.Nuclear war (8:00)I do think that the Ukraine war, in particular, has created a lot of possible flashpoints.I recently finished a book called Nuclear War: A Scenario, which is kind of a minute-by-minute look at how a nuclear war could break out. If you read the book, the book is terrifying because it really goes into a lot of — and I live near Washington DC, so when it gives its various scenarios, certainly my house is included in the blast zone, so really a frightening book. But when it tried to explain how a war would start, I didn't find it a particularly compelling book. The scenarios for actually starting a conflict, I didn't think sounded particularly realistic.Do you feel — and obviously we have Russia invade Ukraine and loose talk by Vladimir Putin about nuclear weapons — do you feel more or less confident that we'll avoid a nuclear war than you did when you wrote the book?Much less confident, actually. I guess I should say, when I wrote the book, it came out in 2020, I finished the writing in 2019, and ultimately we were in a time of relatively low nuclear risk, and I feel that the risk has risen. That said, I was trying to provide estimates for the risk over the next hundred years, and so I wasn't assuming that the low-risk period would continue indefinitely, but it was quite a shock to end up so quickly back in this period of heightened tensions and threats of nuclear escalation, the type of thing I thought was really from my parents' generation. So yes, I do think that the Ukraine war, in particular, has created a lot of possible flashpoints. That said, the temperature has come down on the conversation in the last year, so that's something.Of course, the conversation might heat right back up if we see a Chinese invasion of Taiwan. I've been very bullish about the US economy and world economy over the rest of this decade, but the exception is as long as we don't have a war with China, from an economic point of view, but certainly also a nuclear point of view. Two nuclear armed powers in conflict? That would not be an insignificant event from the existential-risk perspective.It is good that China has a smaller nuclear arsenal than the US or Russia, but there could easily be a great tragedy.Pandemic (10:19)Overall, a lot of countries really just muddled through not very well, and the large institutions that were supposed to protect us from these things, like the CDC and the WHO, didn't do a great job either.The book comes out during the pandemic. Did our response to the pandemic make you more or less confident in our ability and willingness to confront that kind of outbreak? The worst one that saw in a hundred years?Yeah, overall, it made me much less confident. There'd been general thought by those who look at these large catastrophic risks that when the chips are down and the threat is imminent, that people will see it and will band together and put a lot of effort into it; that once you see the asteroid in your telescope and it's headed for you, then things will really get together — a bit like in the action movies or what have you.That's where I take my cue from, exactly.And with Covid, it was kind of staring us in the face. Those of us who followed these things closely were quite alarmed a long time before the national authorities were. Overall, a lot of countries really just muddled through not very well, and the large institutions that were supposed to protect us from these things, like the CDC and the WHO, didn't do a great job either. That said, scientists, particularly developing RNA vaccines, did better than I expected.In the years leading up to the pandemic, certainly we'd seen other outbreaks, they'd had the avian flu outbreak, and you know as well as I do, there were . . . how many white papers or scenario-planning exercises for just this sort of event. I think I recall a story where, in 2018, Bill Gates had a conversation with President Trump during his first term about the risk of just such an outbreak. So it's not as if this thing came out of the blue. In many ways we saw the asteroid, it was just pretty far away. But to me, that says something again about as humans, our ability to deal with severe, but infrequent, risks.And obviously, not having a true global, nasty outbreak in a hundred years, where should we focus our efforts? On preparation? Making sure we have enough ventilators? Or our ability to respond? Because it seems like the preparation route will only go so far, and the reason it wasn't a much worse outbreak is because we have a really strong ability to respond.I'm not sure if it's the same across all risks as to how preparation versus ability to respond, which one is better. In some risks, there's also other possibilities like avoiding an outbreak, say, an accidental outbreak happening at all, or avoiding a nuclear war starting and not needing to actually respond at all. I'm not sure if there's an overall rule as to which one was better.Do you have an opinion on the outbreak of Covid?I don't know whether it was a lab leak. I think it's a very plausible hypothesis, but plausible doesn't mean it's proven.And does the post-Covid reaction, at least in the United States, to vaccines, does that make you more or less confident in our ability to deal with . . . the kind of societal cohesion and confidence to tackle a big problem, to have enough trust? Maybe our leaders don't deserve that trust, but what do you make from this kind of pushback against vaccines and — at least in the United States — our medical authorities?When Covid was first really striking Europe and America, it was generally thought that, while China was locking down the Wuhan area, that Western countries wouldn't be able to lock down, that it wasn't something that we could really do, but then various governments did order lockdowns. That said, if you look at the data on movement of citizens, it turns out that citizens stopped moving around prior to the lockdowns, so the lockdown announcements were more kind of like the tail, rather than the dog.But over time, citizens wanted to kind of get back out and interact more, and the rules were preventing them, and if a large fraction of the citizens were under something like house arrest for the better part of a year, would that lead to some fairly extreme resentment and some backlash, some of which was fairly irrational? Yeah, that is actually exactly the kind of thing that you would expect. It was very difficult to get a whole lot of people to row together and take the same kind of response that we needed to coordinate the response to prevent the spread, and pushing for that had some of these bad consequences, which are also going to make it harder for next time. We haven't exactly learned the right lessons.Killer AI (15:07)If we make things that are smarter than us and are not inherently able to control their values or give them moral rules to work within, then we should expect them to ultimately be calling the shots.We're more than halfway through our chat and now we're going to get to the topic probably most people would like to hear about: After the robots take our jobs, are they going to kill us? What do you think? What is your concern about AI risk?I'm quite concerned about it. Ultimately, when I wrote my book, I put AI risk as the biggest existential risk, albeit the most uncertain, as well, and I would still say that. That said, some things have gotten better since then.I would assume what makes you less confident is one, what seems to be the rapid advance — not just the rapid advance of the technology, but you have the two leading countries in a geopolitical globalization also being the leaders in the technology and not wanting to slow it down. I would imagine that would make you more worried that we will move too quickly. What would make you more confident that we would avoid any serious existential downsides?I agree with your supposition that the attempts by the US and China to turn this into some kind of arms race are quite concerning. But here are a few things: Back when I was writing the book, the leading AI systems with things like AlphaGo, if you remember that, or the Atari plane systems.Quaint. Quite quaint.It was very zero-sum, reinforcement-learning-based game playing, where these systems were learning directly to behave adversarially to other systems, and they could only understand the kind of limited aspect about the world, and struggle, and overcoming your adversary. That was really all they could do, and the idea of teaching them about ethics, or how to treat people, and the diversity of human values seemed almost impossible: How do you tell a chess program about that?But then what we've ended up with is systems that are not inherently agents, they're not inherently trying to maximize something. Rather, you ask them questions and they blurt out some answers. These systems have read more books on ethics and moral philosophy than I have, and they've read all kinds of books about the human condition. Almost all novels that have ever been published, and pretty much every page of every novel involves people judging the actions of other people and having some kind of opinions about them, and so there's a huge amount of data about human values, and how we think about each other, and what's inappropriate behavior. And if you ask the systems about these things, they're pretty good at judging whether something's inappropriate behavior, if you describe it.The real challenge remaining is to get them to care about that, but at least the knowledge is in the system, and that's something that previously seemed extremely difficult to do. Also, these systems, there are versions that do reasoning and that spend longer with a private text stream where they think — it's kind of like sub-vocalizing thoughts to themselves before they answer. When they do that, these systems are thinking in plain English, and that's something that we really didn't expect. If you look at all of the weights of a neural network, it's quite inscrutable, famously difficult to know what it's doing, but somehow we've ended up with systems that are actually thinking in English and where that could be inspected by some oversight process. There are a number of ways in which things are better than I'd feared.So what is your actual existential risk scenario look like? This is what you're most concerned about happening with AI.I think it's quite hard to be all that concrete on it at the moment, partly because things change so quickly. I don't think that there's going to be some kind of existential catastrophe from AI in the next couple of years, partly because the current systems require so much compute in order to run them that they can only be run at very specialized and large places, of which there's only a few in the world. So that means the possibility that they break out and copy themselves into other systems is not really there, in which case, the possibility of turning them off is much possible as well.Also, they're not yet intelligent enough to be able to execute a lengthy plan. If you have some kind of complex task for them, that requires, say, 10 steps — for example, booking a flight on the internet by clicking through all of the appropriate pages, and finding out when the times are, and managing to book your ticket, and fill in the special codes they sent to your email, and things like that. That's a somewhat laborious task and the systems can't do things like that yet. There's still the case that, even if they've got a, say, 90 percent chance of completing any particular step, that the 10 percent chances of failure add up, and eventually it's likely to fail somewhere along the line and not be able to recover. They'll probably get better at that, but at the moment, the inability to actually execute any complex plans does provide some safety.Ultimately, the concern is that, at a more abstract level, we're building systems which are smarter than us at many things, and we're attempting to make them much more general and to be smarter than us across the board. If you know that one player is a better chess player than another, suppose Magnus Carlsen's playing me at chess, I can't predict exactly how he's going to beat me, but I can know with quite high likelihood that he will end up beating me. I'll end up in checkmate, even though I don't know what moves will happen in between here and there, and I think that it's similar with AI systems. If we make things that are smarter than us and are not inherently able to control their values or give them moral rules to work within, then we should expect them to ultimately be calling the shots.Artificial General Intelligence (21:01)Ultimately, existential risks are global public goods problems.I frequently check out the Metaculus online prediction platform, and I think currently on that platform, 2027 for what they would call “weak AGI,” artificial general intelligence — a date which has moved up two months in the past week as we're recording this, and then I think 2031 also has accelerated for “strong AGI,” so this is pretty soon, 2027 or 2031, quite soon. Is that kind of what you're assuming is going to happen, that we're going to have to deal with very powerful technologies quite quickly?Yeah, I think that those are good numbers for the typical case, what you should be expecting. I think that a lot of people wouldn't be shocked if it turns out that there is some kind of obstacle that slows down progress and takes longer before it gets overcome, but it's also wouldn't be surprising at this point if there are no more big obstacles and it's just a matter of scaling things up and doing fairly simple processes to get it to work.It's now a multi-billion dollar industry, so there's a lot of money focused on ironing out any kinks or overcoming any obstacles on the way. So I expect it to move pretty quickly and those timelines sound very realistic. Maybe even sooner.When you wrote the book, what did you put as the risk to human existence over the next a hundred years, and what is it now?When I wrote the book, I thought it was about one in six.So it's still one in six . . . ?Yeah, I think that's still about right, and I would say that most of that is coming from AI.This isn't, I guess, a specific risk, but, to the extent that being positive about our future means also being positive on our ability to work together, countries working together, what do you make of society going in the other direction where we seem more suspicious of other countries, or more even — in the United States — more suspicious of our allies, more suspicious of international agreements, whether they're trade or military alliances. To me, I would think that the Age of Globalization would've, on net, lowered that risk to one in six, and if we're going to have less globalization, to me, that would tend to increase that risk.That could be right. Certainly increased suspicion, to the point of paranoia or cynicism about other nations and their ability to form deals on these things, is not going to be helpful at all. Ultimately, existential risks are global public goods problems. This continued functioning of human civilization is this global public good and existential risk is the opposite. And so these are things where, one way to look at it is that the US has about four percent of the world's people, so one in 25 people live in the US, and so an existential risk is hitting 25 times as many people as. So if every country is just interested in themself, they'll undervalue it by a factor of 25 or so, and the countries need to work together in order to overcome that kind of problem. Ultimately, if one of us falls victim to these risks, then we all do, and so it definitely does call out for international cooperation. And I think that it has a strong basis for international cooperation. It is in all of our interests. There are also verification possibilities and so on, and I'm actually quite optimistic about treaties and other ways to move forward.On sale everywhere The Conservative Futurist: How To Create the Sci-Fi World We Were PromisedMicro Reads▶ Economics* Tech tycoons have got the economics of AI wrong - Economist* Progress in Artificial Intelligence and its Determinants - Arxiv* The role of personality traits in shaping economic returns amid technological change - CEPR▶ Business* Tech CEOs try to reassure Wall Street after DeepSeek shock - Wapo* DeepSeek Calls for Deep Breaths From Big Tech Over Earnings - Bberg Opinion* Apple's AI Moment Is Still a Ways Off - WSJ* Bill Gates Isn't Like Those Other Tech Billionaires - NYT* OpenAI's Sam Altman and SoftBank's Masayoshi Son Are AI's New Power Couple - WSJ* SoftBank Said to Be in Talks to Invest as Much as $25 Billion in OpenAI - NYT* Microsoft sheds $200bn in market value after cloud sales disappoint - FT▶ Policy/Politics* ‘High anxiety moment': Biden's NIH chief talks Trump 2.0 and the future of US science - Nature* Government Tech Workers Forced to Defend Projects to Random Elon Musk Bros - Wired* EXCLUSIVE: NSF starts vetting all grants to comply with Trump's orders - Science* Milei, Modi, Trump: an anti-red-tape revolution is under way - Economist* FDA Deregulation of E-Cigarettes Saved Lives and Spurred Innovation - Marginal Revolution* Donald Trump revives ideas of a Star Wars-like missile shield - Economist▶ AI/Digital* Is DeepSeek Really a Threat? - PS* ChatGPT vs. Claude vs. DeepSeek: The Battle to Be My AI Work Assistant - WSJ* OpenAI teases “new era” of AI in US, deepens ties with government - Ars* AI's Power Requirements Under Exponential Growth - Rand* How DeepSeek Took a Chunk Out of Big AI - Bberg* DeepSeek poses a challenge to Beijing as much as to Silicon Valley - Economist▶ Biotech/Health* Creatine shows promise for treating depression - NS* FDA approves new, non-opioid painkiller Journavx - Wapo▶ Clean Energy/Climate* Another Boffo Energy Forecast, Just in Time for DeepSeek - Heatmap News* Column: Nuclear revival puts uranium back in the critical spotlight - Mining* A Michigan nuclear plant is slated to restart, but Trump could complicate things - Grist▶ Robotics/AVs* AIs and Robots Should Sound Robotic - IEEE Spectrum* Robot beauticians touch down in California - FT Opinion▶ Space/Transportation* A Flag on Mars? Maybe Not So Soon. - NYT* Asteroid triggers global defence plan amid chance of collision with Earth in 2032 - The Guardian* Lurking Inside an Asteroid: Life's Ingredients - NYT▶ Up Wing/Down Wing* An Ancient 'Lost City' Is Uncovered in Mexico - NYT* Reflecting on Rome, London and Chicago after the Los Angeles fires - Wapo Opinion▶ Substacks/Newsletters* I spent two days testing DeepSeek R1 - Understanding AI* China's Technological Advantage -overlapping tech-industrial ecosystems - AI Supremacy* The state of decarbonization in five charts - Exponential View* The mistake of the century - Slow Boring* The Child Penalty: An International View - Conversable Economist* Deep Deepseek History and Impact on the Future of AI - next BIG futureFaster, Please! is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber. This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit fasterplease.substack.com/subscribe

Are you interested in existential hope instead of existential risk? Summary of the article titled Existential risk and existential hope: Definitions from 2015 by Owen Cotton-Barratt and Toby Ord, published by the Future of Humanity Institute. This is a great preparation to our next interview with Allison Duettmann in episode 252 talking optimism and existential hope. Since we are investigating the future of cities, I thought it would be interesting to see the opposite of existential risk in our doom and gloom world. This article presents the existential hope concept, the chance of something extremely good happening. As the most important things, I would like to highlight 3 aspects: The concept of "existential hope" is introduced, highlighting the importance of pursuing transformative events that can greatly improve humanity's future prospects. Existential risks are events that could either wipe out humanity or drastically limit its future potential. Existential catastrophes could be defined by the loss of expected value, which accounts for events that reduce humanity's future potential even without immediate extinction. You can find the article through this link. Abstract: We look at the strengths and weaknesses of two existing definitions of existential risk, and suggest a new definition based on expected value. This leads to a parallel concept: ‘existential hope', the chance of something extremely good happening. Connecting episode you might be intersted in: No.101R - What we owe the future (book summary) No.178R - Brighter (book summary) No.222 - Interview with Adam Dorr about a brighter future You can find the transcript through ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠this link⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠. What wast the most interesting part for you? What questions did arise for you? Let me know on Twitter ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠@WTF4Cities⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ or on the ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠wtf4cities.com⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ website where the ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠showno⁠t⁠es⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠ ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠are also available. I hope this was an interesting episode for you and thanks for tuning in. Music by ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Lesfm ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠from ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Pixabay⁠

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Humanity isn't remotely longtermist, so arguments for AGI x-risk should focus on the near term, published by Seth Herd on August 13, 2024 on LessWrong. Toby Ord recently published a nice piece On the Value of Advancing Progress about mathematical projections of far-future outcomes given different rates of progress and risk levels. The problem with that and many arguments for caution is that people usually barely care about possibilities even twenty years out. We could talk about sharp discounting curves in decision-making studies, and how that makes sense given evolutionary pressures in tribal environments. But I think this is pretty obvious from talking to people and watching our political and economic practices. Utilitarianism is a nicely self-consistent value system. Utilitarianism pretty clearly implies longtermism. Most people don't care that much about logical consistency,[1] so they are happily non-utilitarian and non-longtermist in a variety of ways. Many arguments for AGI safety are longtermist, or at least long-term, so they're not going to work well for most of humanity. This is a fairly obvious, but worth-keeping-in-mind point. One non-obvious lemma of this observation is that much skepticism about AGI x-risk is probably based on skepticism about AGI happening soon. This doesn't explain all skepticism, but it's a significant factor worth addressing. When people dig into their logic, that's often a central point. They start out saying "AGI wouldn't kill humans" then over the course of a conversation it turns out that they feel that way primarily because they don't think real AGI will happen in their lifetimes. Any discussion of AGI x-risks isn't productive, because they just don't care about it. The obvious counterpoint is "You're pretty sure it won't happen soon? I didn't know you were an expert in AI or cognition!" Please don't say this - nothing convinces your opponents to cling to their positions beyond all logic like calling them stupid.[2] Something like "well, a lot of people with the most relevant expertise think it will happen pretty soon. A bunch more think it will take longer. So I just assume I don't know which is right, and it might very well happen pretty soon". It looks to me like discussing whether AGI might threaten humans is pretty pointless if the person is still assuming it's not going to happen for a long time. Once you're past that, it might make sense to actually talk about why you think AGI would be risky for humans.[3] 1. ^ This is an aside, but you'll probably find that utilitarianism isn't that much more logical than other value systems anyway. Preferring what your brain wants you to prefer, while avoiding drastic inconsistency, has practical advantages over values that are more consistent but that clash with your felt emotions. So let's not assume humanity isn't utilitarian just because it's stupid. 2. ^ Making sure any discussions you have about x-risk are pleasant for all involved is probably actually the most important strategy. I strongly suspect that personal affinity weighs more heavily than logic on average, even for fairly intellectual people. (Rationalists are a special case; I think we're resistant but not immune to motivated reasoning). So making a few points in a pleasant way, then moving on to other topics they like is probably way better than making the perfect logical argument while even slightly irritating them. 3. ^ From there you might be having the actual discussion on why AGI might threaten humans. Here are some things I've seen be convincing. People seem to often think "okay fine it might happen soon, but surely AI smarter than us still won't have free will and make its own goals". From there you could point out that it needs goals to be useful, and if it misunderstands those goals even slightly, it might be...

Link to original articleWelcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Humanity isn't remotely longtermist, so arguments for AGI x-risk should focus on the near term, published by Seth Herd on August 13, 2024 on LessWrong. Toby Ord recently published a nice piece On the Value of Advancing Progress about mathematical projections of far-future outcomes given different rates of progress and risk levels. The problem with that and many arguments for caution is that people usually barely care about possibilities even twenty years out. We could talk about sharp discounting curves in decision-making studies, and how that makes sense given evolutionary pressures in tribal environments. But I think this is pretty obvious from talking to people and watching our political and economic practices. Utilitarianism is a nicely self-consistent value system. Utilitarianism pretty clearly implies longtermism. Most people don't care that much about logical consistency,[1] so they are happily non-utilitarian and non-longtermist in a variety of ways. Many arguments for AGI safety are longtermist, or at least long-term, so they're not going to work well for most of humanity. This is a fairly obvious, but worth-keeping-in-mind point. One non-obvious lemma of this observation is that much skepticism about AGI x-risk is probably based on skepticism about AGI happening soon. This doesn't explain all skepticism, but it's a significant factor worth addressing. When people dig into their logic, that's often a central point. They start out saying "AGI wouldn't kill humans" then over the course of a conversation it turns out that they feel that way primarily because they don't think real AGI will happen in their lifetimes. Any discussion of AGI x-risks isn't productive, because they just don't care about it. The obvious counterpoint is "You're pretty sure it won't happen soon? I didn't know you were an expert in AI or cognition!" Please don't say this - nothing convinces your opponents to cling to their positions beyond all logic like calling them stupid.[2] Something like "well, a lot of people with the most relevant expertise think it will happen pretty soon. A bunch more think it will take longer. So I just assume I don't know which is right, and it might very well happen pretty soon". It looks to me like discussing whether AGI might threaten humans is pretty pointless if the person is still assuming it's not going to happen for a long time. Once you're past that, it might make sense to actually talk about why you think AGI would be risky for humans.[3] 1. ^ This is an aside, but you'll probably find that utilitarianism isn't that much more logical than other value systems anyway. Preferring what your brain wants you to prefer, while avoiding drastic inconsistency, has practical advantages over values that are more consistent but that clash with your felt emotions. So let's not assume humanity isn't utilitarian just because it's stupid. 2. ^ Making sure any discussions you have about x-risk are pleasant for all involved is probably actually the most important strategy. I strongly suspect that personal affinity weighs more heavily than logic on average, even for fairly intellectual people. (Rationalists are a special case; I think we're resistant but not immune to motivated reasoning). So making a few points in a pleasant way, then moving on to other topics they like is probably way better than making the perfect logical argument while even slightly irritating them. 3. ^ From there you might be having the actual discussion on why AGI might threaten humans. Here are some things I've seen be convincing. People seem to often think "okay fine it might happen soon, but surely AI smarter than us still won't have free will and make its own goals". From there you could point out that it needs goals to be useful, and if it misunderstands those goals even slightly, it might be...

Leven we inderdaad op een historisch keerpunt waarop de mensheid gedwongen wordt heel anders te leven? Wat zal de opvolger van het antropoceen zijn waarin de mens centraal stond? Kan de intelligentie van de natuur ons helpen een nieuwe weg in te slaan? Kunnen we omgaan met de overweldigende complexiteit van de uitdagingen die op ons afkomen? In deze aflevering zetten we de hoogtepunten van een jaar lang gesprekken met pioniers en denkers voor Future Affairs op een rij. Hoe heeft deze podcastserie onszelf nou veranderd? En wat komt er de komende decennia op ons af?Met fragmenten uit de afleveringen met: Philipp Blom, Toby Ord, René ten Bos, Haroon Sheikh, Marleen Stikker, Bernardo Kastrup, Jalila Essaidi, Bob Hendrikx en Jeremy Lent.Abonneer je hier op de Future Affairs nieuwsbrief: nrc.nl/futureaffairsPresentatie: Jessica van der Schalk & Wouter van NoortProductie: Ruben PestMontage: Gal Tsadok-HaiZie het privacybeleid op https://art19.com/privacy en de privacyverklaring van Californië op https://art19.com/privacy#do-not-sell-my-info.

Future Affairs LIVE: De mensheid is in staat pandemieën te creëren, nucleaire oorlogen te starten en de natuur nog verder te ontregelen. Maar een moreel kader om dat níet te doen ontbreekt. De kans dat we deze eeuw als mensheid ten onder gaan is volgens de wereldberoemd moraalfilosoof Toby Ord zo'n 1 op 6. De hoogste tijd dat we een plan maken voor ons voortbestaan hier op aarde. Toby Ord is verbonden aan The Future of Humanity Institute van de universiteit van Oxford en hij is adviseur voor World Health Organization, the World Bank, the World Economic Forum, the US National Intelligence Council en the UK Prime Minister's Office. Zijn onderzoek richt zich op de rampen die ons voortbestaan op aarde bedreigen.Wij spraken hem tijdens Brainwash Festival in Amsterdam over de grote vraag: hoeveel toekomst hebben we nog?Gast: Toby OrdPresentatie: Jessica van der Schalk & Wouter van NoortProductie: Brainwash FestivalMontage: Gal Tsadok-HaiZie het privacybeleid op https://art19.com/privacy en de privacyverklaring van Californië op https://art19.com/privacy#do-not-sell-my-info.

Last week's episode covered a man-made existential risk to humanity—nuclear war. But what about natural risks? Could there, right now, be a vast asteroid sailing through space that'll collide with Earth, sending us go the way of the dinosaurs?In this rocky episode of The Studies Show, Tom and Stuart look at the data on how often we should expect civilisation-destroying asteroids to hit Earth - and what if anything we can do about it if one is approaching.The Studies Show is brought to you by Works in Progress magazine, the best place on the internet to find mind-changing essays on science, technology, and human progress. We've both written for WiP—one of Tom's articles there is the basis for this episode. You can find all their issues for free at worksinprogress.co.Show notes* Tom's Works in Progress article on the threat from asteroids, on which this episode is based* Toby Ord's book The Precipice, on existential risk (including discussion of asteroids)* Article from Finn Moorhouse on risks from asteroids* Analysis of moon craters to work out how often asteroids hit* And an equation to calculate the impact power of an asteroid hit, from the characteristics of the asteroid* Report from the 2013 US Congressional hearing on threats from outer space* NASA's explanation of how it scans space for asteroids* Carl Sagan's 1994 article on the “dual-use” propensity of asteroid-deflection technology* 2015 article on mining asteroids, and how nudging them closer could help* Just one example of a recent article (2024) on asteroid deflection techniques* 2023 Nature article about the successful DART mission to nudge an asteroid with kinetic force* NASA's DART page with extra news and infoCreditsThe Studies Show is produced by Julian Mayers at Yada Yada Productions. This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.thestudiesshowpod.com/subscribe

I'm often asked about how the existential risk landscape has changed in the years since I wrote The Precipice. Earlier this year, I gave a talk on exactly that, and I want to share it here. Here's a video of the talk and a full transcript. In the years since I wrote The Precipice, the question I'm asked most is how the risks have changed. It's now almost four years since the book came out, but the text has to be locked down a long time earlier, so we are really coming up on about five years of changes to the risk landscape. I'm going to dive into four of the biggest risks — climate change, nuclear, pandemics, and AI — to show how they've changed. Now a lot has happened over those years, and I don't want this to just be recapping the news in fast-forward. But [...] ---Outline:(01:30) Climate Change(01:58) Carbon Emissions(03:18) Climate Sensitivity(06:43) Nuclear(06:46) Heightened Chance of Onset(08:16) Likely New Arms Race(09:54) Funding Collapse(10:53) Pandemics(10:56) Covid(16:03) Protective technologies(18:59) AI in Biotech(20:32) AI(20:50) RL agents ⇒ language models(24:59) Racing(27:05) Governance(30:14) ConclusionsThe original text contained 7 images which were described by AI. --- First published: July 12th, 2024 Source: https://forum.effectivealtruism.org/posts/iKLLSYHvnhgcpoBxH/the-precipice-revisited --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

I'm often asked about how the existential risk landscape has changed in the years since I wrote The Precipice. Earlier this year, I gave a talk on exactly that, and I want to share it here. Here's a video of the talk and a full transcript. In the years since I wrote The Precipice, the question I'm asked most is how the risks have changed. It's now almost four years since the book came out, but the text has to be locked down a long time earlier, so we are really coming up on about five years of changes to the risk landscape. I'm going to dive into four of the biggest risks — climate change, nuclear, pandemics, and AI — to show how they've changed. Now a lot has happened over those years, and I don't want this to just be recapping the news in fast-forward. But [...] ---Outline:(01:30) Climate Change(01:58) Carbon Emissions(03:18) Climate Sensitivity(06:36) Nuclear(06:39) Heightened Chance of Onset(08:09) Likely New Arms Race(09:48) Funding Collapse(10:47) Pandemics(10:50) Covid(15:57) Protective technologies(18:53) AI in Biotech(20:27) AI(20:45) RL agents ⇒ language models(24:53) Racing(26:59) Governance(30:09) ConclusionsThe original text contained 7 images which were described by AI. --- First published: July 12th, 2024 Source: https://forum.effectivealtruism.org/posts/iKLLSYHvnhgcpoBxH/the-precipice-revisited --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Precipice Revisited, published by Toby Ord on July 12, 2024 on The Effective Altruism Forum. I'm often asked about how the existential risk landscape has changed in the years since I wrote The Precipice. Earlier this year, I gave a talk on exactly that, and I want to share it here. Here's a video of the talk and a full transcript. In the years since I wrote The Precipice, the question I'm asked most is how the risks have changed. It's now almost four years since the book came out, but the text has to be locked down a long time earlier, so we are really coming up on about five years of changes to the risk landscape. I'm going to dive into four of the biggest risks - climate change, nuclear, pandemics, and AI - to show how they've changed. Now a lot has happened over those years, and I don't want this to just be recapping the news in fast-forward. But luckily, for each of these risks I think there are some key insights and takeaways that one can distill from all that has happened. So I'm going to take you through them and tease out these key updates and why they matter. I'm going to focus on changes to the landscape of existential risk - which includes human extinction and other ways that humanity's entire potential could be permanently lost. For most of these areas, there are many other serious risks and ongoing harms that have also changed, but I won't be able to get into those. The point of this talk is to really narrow in on the changes to existential risk. Climate Change Let's start with climate change. We can estimate the potential damages from climate change in three steps: 1. how much carbon will we emit? 2. how much warming does that carbon produce? 3. how much damage does that warming do? And there are key updates on the first two of these, which have mostly flown under the radar for the general public. Carbon Emissions The question of how much carbon we will emit is often put in terms of Representative Concentration Pathways (RCPs). Initially there were 4 of these, with higher numbers meaning more greenhouse effect in the year 2100. They are all somewhat arbitrarily chosen - meant to represent broad possibilities for how our emissions might unfold over the century. Our lack of knowledge about which path we would take was a huge source of uncertainty about how bad climate change would be. Many of the more dire climate predictions are based on the worst of these paths, RCP 8.5. It is now clear that we are not at all on RCP 8.5, and that our own path is headed somewhere between the lower two paths. This isn't great news. Many people were hoping we could control our emissions faster than this. But for the purposes of existential risk from climate change, much of the risk comes from the worst case possibilities, so even just moving towards the middle of the range means lower existential risk - and the lower part of the middle is even better. Climate Sensitivity Now what about the second question of how much warming that carbon will produce? The key measure here is something called the equilibrium climate sensitivity. This is roughly defined as how many degrees of warming there would be if the concentrations of carbon in the atmosphere were to double from pre-industrial levels. If there were no feedbacks, this would be easy to estimate: doubling carbon dioxide while keeping everything else fixed produces about 1.2°C of warming. But the climate sensitivity also accounts for many climate feedbacks, including water vapour and cloud formation. These make it higher and also much harder to estimate. When I wrote The Precipice, the IPCC stated that climate sensitivity was likely to be somewhere between 1.5°C and 4.5°C. When it comes to estimating the impacts of warming, this is a vast range, with the top giving three times as much warming as the bottom. Moreover, the...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: On the Value of Advancing Progress, published by Toby Ord on July 11, 2024 on The Effective Altruism Forum. (PDF available) I show how a standard argument for advancing progress is extremely sensitive to how humanity's story eventually ends. Whether advancing progress is ultimately good or bad depends crucially on whether it also advances the end of humanity. Because we know so little about the answer to this crucial question, the case for advancing progress is undermined. I suggest we must either overcome this objection through improving our understanding of these connections between progress and human extinction or switch our focus to advancing certain kinds of progress relative to others - changing where we are going, rather than just how soon we get there.[1] Things are getting better. While there are substantial ups and downs, long-term progress in science, technology, and values have tended to make people's lives longer, freer, and more prosperous. We could represent this as a graph of quality of life over time, giving a curve that generally trends upwards. What would happen if we were to advance all kinds of progress by a year? Imagine a burst of faster progress, where after a short period, all forms of progress end up a year ahead of where they would have been. We might think of the future trajectory of quality of life as being primarily driven by science, technology, the economy, population, culture, societal norms, moral norms, and so forth. We're considering what would happen if we could move all of these features a year ahead of where they would have been. While the burst of faster progress may be temporary, we should expect its effect of getting a year ahead to endure.[2] If we'd only advanced some domains of progress, we might expect further progress in those areas to be held back by the domains that didn't advance - but here we're imagining moving the entire internal clock of civilisation forward a year. If we were to advance progress in this way, we'd be shifting the curve of quality of life a year to the left. Since the curve is generally increasing, this would mean the new trajectory of our future is generally higher than the old one. So the value of advancing progress isn't just a matter of impatience - wanting to get to the good bits sooner - but of overall improvement in people's quality of life across the future. Figure 1. Sooner is better. The solid green curve is the default trajectory of quality of life over time, while the dashed curve is the trajectory if progress were uniformly advanced by one year (shifting the default curve to the left). Because the trajectories trend upwards, quality of life is generally higher under the advanced curve. To help see this, I've shaded the improvements to quality of life green and the worsenings red. That's a standard story within economics: progress in science, technology, and values has been making the world a better place, so a burst of faster progress that brought this all forward by a year would provide a lasting benefit for humanity. But this story is missing a crucial piece. The trajectory of humanity's future is not literally infinite. One day it will come to an end. This might be a global civilisation dying of old age, lumbering under the weight of accumulated bureaucracy or decadence. It might be a civilisation running out of resources: either using them up prematurely or enduring until the sun itself burns out. It might be a civilisation that ends in sudden catastrophe - a natural calamity or one of its own making. If the trajectory must come to an end, what happens to the previous story of an advancement in progress being a permanent uplifting of the quality of life? The answer depends on the nature of the end time. There are two very natural possibilities. One is that the end time is fixed. It...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Robust longterm comparisons, published by Toby Ord on May 15, 2024 on The Effective Altruism Forum. (Cross-posted from http://www.tobyord.com/writing/robust-longterm-comparisons ) The choice of discount rate is crucially important when comparing options that could affect our entire future. Except when it isn't. Can we tease out a class of comparisons that everyone can agree on regardless of their views on discounting? Some of the actions we can take today may have longterm effects - permanent changes to humanity's longterm trajectory. For example, we may take risks that could lead to human extinction. Or we might irreversibly destroy parts of our environment, creating permanent reductions in the quality of life. Evaluating and comparing such effects is usually extremely sensitive to what economists call the pure rate of time preference, denoted ρ. This is a way of encapsulating how much less we should value a benefit simply because it occurs at a later time. There are other components of the overall discount rate that adjust for the fact that an extra dollar is worth less when people are richer, that later benefits may be less likely to occur - or that the entire society may have ceased to exist by then. But the pure rate of time preference is the amount by which we should discount future benefits even after all those things have been accounted for. Most attempts to evaluate or compare options with longterm effects get caught up in intractable disagreements about ρ. Philosophers almost uniformly think ρ should be set to zero, with any bias towards the present being seen as unfair. That is my usual approach, and I've developed a framework for making longterm comparisons without any pure time preference. While some prominent economists agree that ρ should be zero, the default in economic analysis is to use a higher rate, such as 1% per year. The difference between a rate of 0% and 1% is small for most things economists evaluate, where the time horizon is a generation or less. But it makes a world of difference to the value of longterm effects. For example, ρ = 1% implies that a stream of damages starting in 500 years time and lasting a billion years is less bad than a single year of such damages today. So when you see a big disagreement on how to make a tradeoff between, say, economic benefits and existential risk, you can almost always pinpoint the source to a disagreement about ρ. This is why it was so surprising to read Charles Jones's recent paper: 'The AI Dilemma: Growth versus Existential Risk'. In his examination of whether and when the economic gains from developing advanced AI could outweigh the resulting existential risk, the rate of pure time preference just cancels out. The value of ρ plays no role in his primary model. There were many other results in the paper, but it was this detail that grabbed my attention. Here was a question about trading off risk of human extinction against improved economic consumption that economists and philosophers might actually be able to agree on. After all, even better than picking the correct level of ρ, deriving the correct conclusion, and yet still having half the readers ignore your findings, is if there is a way of conducting the analysis such that you are not only correct - but that everyone else can see that too. Might we be able to generalise this happy result further? Is there are broader range of long run effects in which the discount rate still cancels out? Are there other disputed parameters (empirical or normative) that also cancel out in those cases? What I found is that this can indeed be greatly generalised, creating a domain in which we can robustly compare long run effects - where the comparisons are completely unaffected by different assumptions about discounting. Let's start by considering a basic model w...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Supervolcanoes tail risk has been exaggerated?, published by Vasco Grilo on March 6, 2024 on The Effective Altruism Forum. This is a linkpost for the peer-reviewed article "Severe Global Cooling After Volcanic Super-Eruptions? The Answer Hinges on Unknown Aerosol Size" ( McGraw 2024). Below are its abstract, my notes, my estimation of a nearterm annual extinction risk from supervolcanoes of 3.38*10^-14, and a brief discussion of it. At the end, I have a table comparing my extinction risk estimates with Toby Ord's existential risk guesses given in The Precipice. Abstract Here is the abstract from McGraw 2024 (emphasis mine): Volcanic super-eruptions have been theorized to cause severe global cooling, with the 74 kya Toba eruption purported to have driven humanity to near-extinction. However, this eruption left little physical evidence of its severity and models diverge greatly on the magnitude of post-eruption cooling. A key factor controlling the super-eruption climate response is the size of volcanic sulfate aerosol, a quantity that left no physical record and is poorly constrained by models. Here we show that this knowledge gap severely limits confidence in model-based estimates of super-volcanic cooling, and accounts for much of the disagreement among prior studies. By simulating super-eruptions over a range of aerosol sizes, we obtain global mean responses varying from extreme cooling all the way to the previously unexplored scenario of widespread warming. We also use an interactive aerosol model to evaluate the scaling between injected sulfur mass and aerosol size. Combining our model results with the available paleoclimate constraints applicable to large eruptions, we estimate that global volcanic cooling is unlikely to exceed 1.5°C no matter how massive the stratospheric injection. Super-eruptions, we conclude, may be incapable of altering global temperatures substantially more than the largest Common Era eruptions. This lack of exceptional cooling could explain why no single super-eruption event has resulted in firm evidence of widespread catastrophe for humans or ecosystems. My notes I have no expertise in volcanology, but I found McGraw 2024 to be quite rigorous. In particular, they are able to use their model to replicate the more pessimistic results of past studies tweeking just 2 input parameters (highlighted by me below): "We next evaluate if the assessed aerosol size spread is the likely cause of disagreement among past studies with interactive aerosol models. For this task, we interpolated the peak surface temperature responses from our ModelE simulations to the injected mass and peak global mean aerosol size from several recent interactive aerosol model simulations of large eruptions (Fig. 7, left panel). Accounting for these two values alone (left panel), our model experiments are able to reproduce remarkably similar peak temperature responses as the original studies found". By "reproduce remarkably well", they are referring to a coefficient of determination (R^2) of 0.87 (see Fig. 7). "By comparison, if only the injected masses of the prior studies are used, the peak surface temperature responses cannot be reproduced". By this, they are referring to an R^2 ranging from -1.82 to -0.04[1] (see Fig. 7). They agree with past studies on the injected mass, but not on the aerosol size[2]. Fig. 3a (see below) illustrates the importance of the peak mean aerosol size. The greater the size, the weaker the cooling. I think this is explained as follows: Primarily, smaller particles reflect more sunlight per mass due to having greater cross-sectional area per mass[3]. Secondarily, larger particles have less time to reflect sunlight due to falling down faster[4]. According to Fig. 2 (see below), aerosol size increases with injected mass, which makes intuitive sen...

Remember when the airwaves were full of people questioning the idea of man-made climate change? You don't hear much from them any more - in large part becuase the evidence that our CO2 emissions are altering the climate has become so overwhelming.After a recap on how we know that carbon warms the climate, Tom and Stuart use this episode of The Studies Show to discuss climate predictions—er, I mean, projections—and how accurate they've been. They ask whether the media always gets it right when discussing climate (spoiler: no), and whether we should be optimistic or panicked about what's happening to the environment.The Studies Show is sponsored by Works in Progress magazine. Ever wondered what people mean when they talk about “progress studies”? Works in Progress is what they mean. It's a magazine bursting with fascinating articles on how science and technology have improved our lives - and how they could be even better in future. There's a whole new February 2024 issue out now - read it at this link.Show notes* 2023: the hottest year on record, with surprising and anomalous melting of ice in Antarctica* NASA on how the presence of CO2 in the atmosphere raises the Earth's temperature* Carbon Brief explains how scientists estimate climate sensitivity, and discusses the complexities of the latest climate models* The most recent IPCC report, from March 2023* The IEA's forecast of solar power, with the incredible and very optimistic graph mentioned in the episode:* Tom's unfortunately-titled Unherd article on the unlikely but much-discussed “RCP 8.5” scenario* Zeke Hausfather's study on matching up the projections of climate models with what actually happened years and decades later* Response from the sceptics (they still exist!)* Website offering responses to all the most common claims by climate change sceptics (e.g. “the Earth hasn't warmed since 1998”; “CO2 is plant food”)* Toby Ord on how, whereas climate change could be extremely bad, it's tricky to argue that it's a truly “existential” riskCredits and acknowledgementsThe Studies Show is produced by Julian Mayers at Yada Yada Productions. We're grateful to Karsten Haustein for talking to us for this episode (any errors are our own). This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.thestudiesshowpod.com/subscribe

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Nuclear war tail risk has been exaggerated?, published by Vasco Grilo on February 26, 2024 on The Effective Altruism Forum. The views expressed here are my own, not those of Alliance to Feed the Earth in Disasters ( ALLFED), for which I work as a contractor. Summary I calculated a nearterm annual risk of human extinction from nuclear war of 5.93*10^-12 ( more). I consider grantmakers and donors interested in decreasing extinction risk had better focus on artificial intelligence ( AI) instead of nuclear war ( more). I would say the case for sometimes prioritising nuclear extinction risk over AI extinction risk is much weaker than the case for sometimes prioritising natural extinction risk over nuclear extinction risk ( more). I get a sense the extinction risk from nuclear war was massively overestimated in The Existential Risk Persuasion Tournament ( XPT) ( more). I have the impression Toby Ord greatly overestimated tail risk in The Precipice ( more). I believe interventions to decrease deaths from nuclear war should be assessed based on standard cost-benefit analysis ( more). I think increasing calorie production via new food sectors is less cost-effective to save lives than measures targeting distribution ( more). Extinction risk from nuclear war I calculated a nearterm annual risk of human extinction from nuclear war of 5.93*10^-12 (= (6.36*10^-14*5.53*10^-10)^0.5) from the geometric mean between[1]: My prior of 6.36*10^-14 for the annual probability of a war causing human extinction. My inside view estimate of 5.53*10^-10 for the nearterm annual probability of human extinction from nuclear war. By nearterm annual risk, I mean that in a randomly selected year from 2025 to 2050. I computed my inside view estimate of 5.53*10^-10 (= 0.0131*0.0422*10^-6) multiplying: 1.31 % annual probability of a nuclear weapon being detonated as an act of war. 4.22 % probability of insufficient calorie production given at least one nuclear detonation. 10^-6 probability of human extinction given insufficient calorie production. I explain the rationale for the above estimates in the next sections. Note nuclear war might have cascade effects which lead to civilisational collapse[2], which could increase longterm extinction risk while simultaneously having a negligible impact on the nearterm one I estimated. I do not explicitly assess this in the post, but I guess the nearterm annual risk of human extinction from nuclear war is a good proxy for the importance of decreasing nuclear risk from a longtermist perspective: My prior implicitly accounts for the cascade effects of wars. I derived it from historical data on the deaths of combatants due to not only fighting, but also disease and starvation, which are ever-present indirect effects of war. Nuclear war might have cascade effects, but so do other catastrophes. Global civilisational collapse due to nuclear war seems very unlikely to me. For instance, the maximum destroyable area by any country in a nuclear 1st strike was estimated to be 65.3 k km^2 in Suh 2023 (for a strike by Russia), which is just 70.8 % (= 65.3*10^3/( 92.2*10^3)) of the area of Portugal, or 3.42 % (= 65.3*10^3/( 1.91*10^6)) of the global urban area. Even if nuclear war causes a global civilisational collapse which eventually leads to extinction, I guess full recovery would be extremely likely. In contrast, an extinction caused by advanced AI would arguably not allow for a full recovery. I am open to the idea that nuclear war can have longterm implications even in the case of full recovery, but considerations along these lines would arguably be more pressing in the context of AI risk. For context, William MacAskill said the following on The 80,000 Hours Podcast. "It's quite plausible, actually, when we look to the very long-term future, that that's [whether artificial...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: On being an EA for decades, published by Michelle Hutchinson on February 12, 2024 on The Effective Altruism Forum. A friend sent me on a lovely trip down memory lane last week. She forwarded me an email chain from 12 years ago in which we all pretended we left the EA community, and explained why. We were partly thinking through what might cause us to drift in order that we could do something to make that less likely, and partly joking around. It was a really nice reminder of how uncertain things felt back then, and how far we've come. Although the emails hadn't led us to take any specific actions, almost everyone on the thread was still devoting their time to helping others as much as they can. We're also, for the most part, still supporting each other in doing so. Of the 10 or so people on there, for example, Niel Bowerman is now my boss - CEO of 80k. Will MacAskill and Toby Ord both made good on their plans to write books and donate their salaries above £20k[1] a year. And Holly Morgan is who I turned to a couple of weeks ago when I needed help thinking through work stress. Here's what I wrote speculating about why I might drift away from EA. Note that the email below was written quickly and just trying to gesture at things I might worry about in future, I was paying very little attention to the details. The partner referenced became my husband a year and a half later, and we now have a four year old. On 10 February 2012 18:14, Michelle Hutchinson wrote: Writing this was even sadder than I expected it to be. 1. Holly's and my taste in music drove the rest of the HIH[2] crazy, and they murdered us both. 2. The feeling that I was letting down my parents, husband and children got the better of me, and I sought better paid employment. 3. Toby, Will and Nick got replaced by trustees whose good judgement and intelligence I didn't have nearly as much faith in, so I no longer felt fully supportive of the organisation - I was worried it was soon going to become preachy rather than effective, or even dangerous (researching future technology without much foresight). 4. I realised there were jobs that wouldn't involve working (or even emailing co-workers) Friday evenings :p 5. I remembered how much I loved reading philosophy all day, and teaching tutorials, and unaccountably Oxford offered me a stipendiary lectureship, so I returned to academia. 6. [My partner] got lazy career-wise, and I wanted a nice big house and expensive food, so I got a higher paid job. 7. I accidentally emailed confidential member information to our whole mailing list / said the wrong thing to one of our big funders / made a wrong call that got us into legal trouble, and it was politely suggested to me that the best way I could help the EA movement was by being nowhere near it. 8. When it became clear that although I rationally agreed with the moral positions of GWWC and CEA, most of my emotional motivation for working for them came from not wanting to let down the other [team members], it was decided really I wasn't a nice enough person to have a position of responsibility in such a great organisation. 9. [My partner] got a job too far away from other CEA people for me to be able to work for CEA and be with him. I chose him. 10. When we had children I took maternity leave, and then couldn't bear to leave the children to return to work. 11. I tried to give a talk about GWWC to a large audience, fainted with fright, and am now in a coma. When I wrote the email, I thought it was really quite likely that in 10 years I'd have left the organisation and community. Looking around the world, it seemed like a lot of people become less idealistic as they grow older. And looking inside myself, it felt pretty contingent that I happened to fall in with a group of people who supp...

A friend sent me on a lovely trip down memory lane last week. She forwarded me an email chain from 12 years ago in which we all pretended we left the EA community, and explained why. We were partly thinking through what might cause us to drift in order that we could do something to make that less likely, and partly joking around. It was a really nice reminder of how uncertain things felt back then, and how far we've come. Although the emails hadn't led us to take any specific actions, almost everyone on the thread was still devoting their time to helping others as much as they can. We're also, for the most part, still supporting each other in doing so. Of the 10 or so people on there, for example, Niel Bowerman is now my boss - CEO of 80k. Will MacAskill and Toby Ord [...] The original text contained 3 footnotes which were omitted from this narration. --- First published: February 12th, 2024 Source: https://forum.effectivealtruism.org/posts/zEMvHK9Qa4pczWbJg/on-being-an-ea-for-decades --- Narrated by TYPE III AUDIO.

Rebroadcast: this episode was originally released in October 2021.Preventing the apocalypse may sound like an idiosyncratic activity, and it sometimes is justified on exotic grounds, such as the potential for humanity to become a galaxy-spanning civilisation.But the policy of US government agencies is already to spend up to $4 million to save the life of a citizen, making the death of all Americans a $1,300,000,000,000,000 disaster.According to Carl Shulman, research associate at Oxford University's Future of Humanity Institute, that means you don't need any fancy philosophical arguments about the value or size of the future to justify working to reduce existential risk — it passes a mundane cost-benefit analysis whether or not you place any value on the long-term future.Links to learn more, summary, and full transcript.The key reason to make it a top priority is factual, not philosophical. That is, the risk of a disaster that kills billions of people alive today is alarmingly high, and it can be reduced at a reasonable cost. A back-of-the-envelope version of the argument runs:The US government is willing to pay up to $4 million (depending on the agency) to save the life of an American.So saving all US citizens at any given point in time would be worth $1,300 trillion.If you believe that the risk of human extinction over the next century is something like one in six (as Toby Ord suggests is a reasonable figure in his book The Precipice), then it would be worth the US government spending up to $2.2 trillion to reduce that risk by just 1%, in terms of American lives saved alone.Carl thinks it would cost a lot less than that to achieve a 1% risk reduction if the money were spent intelligently. So it easily passes a government cost-benefit test, with a very big benefit-to-cost ratio — likely over 1000:1 today.This argument helped NASA get funding to scan the sky for any asteroids that might be on a collision course with Earth, and it was directly promoted by famous economists like Richard Posner, Larry Summers, and Cass Sunstein.If the case is clear enough, why hasn't it already motivated a lot more spending or regulations to limit existential risks — enough to drive down what any additional efforts would achieve?Carl thinks that one key barrier is that infrequent disasters are rarely politically salient. Research indicates that extra money is spent on flood defences in the years immediately following a massive flood — but as memories fade, that spending quickly dries up. Of course the annual probability of a disaster was the same the whole time; all that changed is what voters had on their minds.Carl suspects another reason is that it's difficult for the average voter to estimate and understand how large these respective risks are, and what responses would be appropriate rather than self-serving. If the public doesn't know what good performance looks like, politicians can't be given incentives to do the right thing.It's reasonable to assume that if we found out a giant asteroid were going to crash into the Earth one year from now, most of our resources would be quickly diverted into figuring out how to avert catastrophe.But even in the case of COVID-19, an event that massively disrupted the lives of everyone on Earth, we've still seen a substantial lack of investment in vaccine manufacturing capacity and other ways of controlling the spread of the virus, relative to what economists recommended.Carl expects that all the reasons we didn't adequately prepare for or respond to COVID-19 — with excess mortality over 15 million and costs well over $10 trillion — bite even harder when it comes to threats we've never faced before, such as engineered pandemics, risks from advanced artificial intelligence, and so on.Today's episode is in part our way of trying to improve this situation. In today's wide-ranging conversation, Carl and Rob also cover:A few reasons Carl isn't excited by ‘strong longtermism'How x-risk reduction compares to GiveWell recommendationsSolutions for asteroids, comets, supervolcanoes, nuclear war, pandemics, and climate changeThe history of bioweaponsWhether gain-of-function research is justifiableSuccesses and failures around COVID-19The history of existential riskAnd much moreProducer: Keiran HarrisAudio mastering: Ben CordellTranscriptions: Katy Moore

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Exaggerating the risks (Part 13: Ord on Biorisk), published by Vasco Grilo on December 31, 2023 on The Effective Altruism Forum. This is a crosspost to Exaggerating the risks (Part 13: Ord on Biorisk), as published by David Thorstad on 29 December 2023. This massive democratization of technology in biological sciences … is at some level fantastic. People are very excited about it. But this has this dark side, which is that the pool of people that could include someone who has … omnicidal tendencies grows many, many times larger, thousands or millions of times larger as this technology is democratized, and you have more chance that you get one of these people with this very rare set of motivations where they're so misanthropic as to try to cause … worldwide catastrophe. Toby Ord, 80,000 Hours Interview Listen to this post [there is an option for this in the original post] 1. Introduction This is Part 13 of my series Exaggerating the risks. In this series, I look at some places where leading estimates of existential risk look to have been exaggerated. Part 1 introduced the series. Parts 2-5 ( sub-series: "Climate risk") looked at climate risk. Parts 6-8 ( sub-series: "AI risk") looked at the Carlsmith report on power-seeking AI. Parts 9, 10 and 11 began a new sub-series on biorisk. In Part 9, we saw that many leading effective altruists give estimates between 1.0-3.3% for the risk of existential catastrophe from biological causes by 2100. I think these estimates are a bit too high. Because I have had a hard time getting effective altruists to tell me directly what the threat is supposed to be, my approach was to first survey the reasons why many biosecurity experts, public health experts, and policymakers are skeptical of high levels of near-term existential biorisk. Parts 9, 10 and 11 gave a dozen preliminary reasons for doubt, surveyed at the end of Part 11. The second half of my approach is to show that initial arguments by effective altruists do not overcome the case for skepticism. Part 12 examined a series of risk estimates by Piers Millett and Andrew Snyder-Beattie. We saw, first, that many of these estimates are orders of magnitude lower than those returned by leading effective altruists and second, that Millett and Snyder-Beattie provide little in the way of credible support for even these estimates. Today's post looks at Toby Ord's arguments in The Precipice for high levels of existential risk. Ord estimates the risk of irreversible existential catastrophe by 2100 from naturally occurring pandemics at 1/10,000, and the risk from engineered pandemics at a whopping 1/30. That is a very high number. In this post, I argue that Ord does not provide sufficient support for either of his estimates. 2. Natural pandemics Ord begins with a discussion of natural pandemics. I don't want to spend too much time on this issue, since Ord takes the risk of natural pandemics to be much lower than that of engineered pandemics. At the same time, it is worth asking how Ord arrives at a risk of 1/10,000. Effective altruists effectively stress that humans have trouble understanding how large certain future-related quantities can be. For example, there might be 1020, 1050 or even 10100 future humans. However, effective altruists do not equally stress how small future-related probabilities can be. Risk probabilities can be on the order of 10-2 or even 10-5, but they can also be a great deal lower than that: for example, 10-10, 10-20, or 10-50 [for example, a terrorist attack causing human extinction is astronomically unlikely on priors]. Most events pose existential risks of this magnitude or lower, so if Ord wants us to accept that natural pandemics have a 1/10,000 chance of leading to irreversible existential catastrophe by 2100, Ord owes us a solid argument for this conclusion. It ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: An exhaustive list of cosmic threats, published by JordanStone on December 5, 2023 on The Effective Altruism Forum. Toby Ord covers 'Asteroids and Comets' and 'Stellar Explosions' in The Precipice. But I thought it would be useful to provide an up-to-date and exhaustive list of all cosmic threats. I'm defining cosmic threat here as any existential risk potentially arising from space. I think this list may be useful for 3 main reasons: New cosmic threats are discovered frequently. So it's plausible that future cause areas could pop out of this space. I think that keeping an eye on it should help identify areas that may need research. Though it should be noted that some of the risks are totally impossible to protect against at this point (e.g. a rogue planet entering our solar system). Putting all of the cosmic threats together in one place could reveal that cosmic threats are more important than previously thought, or provide a good intro for someone interested in working in this space. There is momentum in existential risk reduction from outer space, with great powers (Russia, USA, China, India, Europe) already collaborating on asteroid impact risk. So harnessing that momentum to tackle some more of the risks on this list could be really tractable and may lead to collaboration on other x-risks like AI, biotech and nuclear. I will list each cosmic threat, provide a brief explanation, and find the best evidence I can to provide severity and probability estimates for each. Enjoy :) I'll use this format: Cosmic Threat [Severity of worst case scenario /10] [Probability of that scenario occurring in the next 100 years] Explanation of threat Explanation of rationale and approach Severity estimates For the severity, 10 is the extinction of all intelligent life on Earth, and 0 is a fart in the wind. It was difficult to pin down one number for threats with multiple outcomes (e.g. asteroids have different sizes). So the severity estimates are for the worst-case scenarios for each cosmic threat, and the probability estimate corresponds to that scenario. Probability estimates Probabilities are presented as % chance of that scenario occurring in the next 100 years. I have taken probabilities from the literature and converted values to normalise them as a probability of their occurrence within the next 100 years (as a %). This isn't a perfect way to do it, but I prioritised getting a general understanding of their probability, rather than numbers that are hard to imagine. When the severity or likelihood is unclear or not researched well enough, I've written 'unknown'. I'm trying my best to ignore reasoning along the lines of "if it hasn't happened before, then it very likely won't happen ever or is extremely rare" because of the anthropic principle. Our view of past events on Earth is biased towards a world that has allowed humanity to evolve, which likely required a few billion years of stable-ish conditions. So it is likely that we have just been lucky in the past, where no cosmic threats have disturbed Earth's habitability so extremely as to set back life's evolution by billions of years (not even the worst mass extinction ever at the Permian-Triassic boundary did this, as reptiles survived). An Exhaustive List of Cosmic Threats Format: Cosmic Threat [Severity of worst case scenario /10] [Probability of that scenario occurring in the next 100 years] Explanation of threat Solar flares [4/10] [1%]. Electromagnetic radiation erupts from the surface of the sun. Solar flares occur fairly regularly and cause minor impacts, mainly on communications. A large solar flare has the potential to cause electrical grids to fail, damage satellites, disrupt radio signals, cause increased radiation influx, destroy data storage devices, cause navigation errors, and permanently damage scientific eq...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: 10 years of Earning to Give, published by AGB on November 8, 2023 on The Effective Altruism Forum. General note: The bulk of this post was written a couple of months ago, but I am releasing it now to coincide with the Effective Giving Spotlight week. I shortly expect to release a second post documenting some observations on the community building funding landscape. Introduction Way back in 2010, I was sitting in my parents' house, watching one of my favourite TV shows, the UK's Daily Politics. That day's guest was an Oxford academic by the name of Toby Ord. He was donating everything above 18000 (26300 in today's money) to charity, and gently pushing others to give 10%. "Nice guy," I thought. "Pity it'll never catch on." Two years later, a couple of peers interned at Giving What We Can. At the same time, I did my own internship in finance, and my estimate of my earning potential quadrupled[1]. One year after that, I graduated and took the Giving What We Can pledge myself. While my pledge form read that I had committed to donate 20% of my income, my goal was to hit far higher percentages. How did that go? Post goals Earning To Give was one of EA's first ideas to get major mainstream attention, much of it negative. Some was mean-spirited, but some of it read to me as a genuine attempt to warn young people about what they were signing up for. For example, from the linked David Brooks piece: From the article, Trigg seems like an earnest, morally serious man... First, you might start down this course seeing finance as a convenient means to realize your deepest commitment: fighting malaria. But the brain is a malleable organ....Every hour you spend with others, you become more like the people around you. If there is a large gap between your daily conduct and your core commitment, you will become more like your daily activities and less attached to your original commitment. You will become more hedge fund, less malaria. There's nothing wrong with working at a hedge fund, but it's not the priority you started out with. At the time, EAs had little choice but to respond to such speculation with speculation of their own. At this point, I can at least answer how some things have played out for me personally. I have divided this post into reflections on my personal EtG path and on the EA community. My path First, some context. Over the past decade: My wife Denise and I have donated 1.5m.[2] This equates to 46% of our combined gross incomes.[2] The rest of the money is split 550k / 550k / 700k between spending / saving (incl. pension) / taxes.[2] We have three children (ages 13, 6, 2) and live in London. I work as a trader, formerly at a quant trading firm and now at a hedge fund. Work Many critics of EtG assume that we really want to be doing something meaningful, but have - with a heavy heart - intellectually conceded that money is what matters. I want to emphasise this: This is not me, and I doubt it applies to even 20% of people doing EtG. If you currently feel this way, I strongly suspect you should stop. I like my work. I get to work with incredibly sharp and motivated people. I get to work on a diverse array of intellectual challenges. Most of all, I've managed to land a career that bears an uncanny resemblance to what I do with my spare time; playing games, looking for inconsistencies in others' beliefs, and exploiting that to win. But prior to discovering EtG, I was wrestling with the fact that this natural choice just seemed very selfish. As I saw it, my choices were to do something directly useful and be miserable but valuable, or to work in finance and be happy but worthless. So a reminder that the money I have a comparative advantage in earning is itself of value was a relief, not a burden. My career pathway has not been smooth, with a major derailment in 2018, which ...

General note: The bulk of this post was written a couple of months ago, but I am releasing it now to coincide with the Effective Giving Spotlight week. I shortly expect to release a second post documenting some observations on the community building funding landscape. IntroductionWay back in 2010, I was sitting in my parents' house, watching one of my favourite TV shows, the UK's Daily Politics. That day's guest was an Oxford academic by the name of Toby Ord. He was donating everything above £18000 (£26300 in today's money) to charity, and gently pushing others to give 10%."Nice guy," I thought. "Pity it'll never catch on."Two years later, a couple of peers interned at Giving What We Can. At the same time, I did my own internship in finance, and my estimate of my earning potential quadrupled[1]. One year after that, I graduated and took the Giving What We [...] ---Outline:(01:13) Post goals(02:28) My path(03:07) Work(05:36) Lifestyle Inflation(07:48) Savings(09:27) Donations(10:11) Community(10:14) Why engage?(11:58) Why stop?(13:48) Closing ThoughtsThe original text contained 7 footnotes which were omitted from this narration. --- First published: November 7th, 2023 Source: https://forum.effectivealtruism.org/posts/gxppfWhx7ta2fkF3R/10-years-of-earning-to-give --- Narrated by TYPE III AUDIO.

Figure 1  (see full caption below)This post is a part of Rethink Priorities' Worldview Investigations Team's CURVE Sequence: "Causes and Uncertainty: Rethinking Value in Expectation." The aim of this sequence is twofold: first, to consider alternatives to expected value maximisation for cause prioritisation; second, to evaluate the claim that a commitment to expected value maximisation robustly supports the conclusion that we ought to prioritise existential risk mitigation over all else.Executive SummaryBackgroundThis report builds on the model originally introduced by Toby Ord on how to estimate the value of existential risk mitigation. The previous framework has several limitations, including:The inability to model anything requiring shorter time units than centuries, like AI timelines.A very limited range of scenarios considered. In the previous model, risk and value growth can take different forms, and each combination represents one scenarioNo explicit treatment of persistence –– how long the mitigation efforts' effects last for ––as a variable of interest.No easy way [...] ---Outline:(00:38) Executive Summary(05:26) Abridged Report(11:20) Generalised Model: Arbitrary Risk Profile(13:37) Value(19:00) Great Filters and the Time of Perils Hypothesis(21:06) Decaying Risk(21:55) Results(21:58) Convergence(25:35) The Expected Value of Mitigating Risk Visualised(31:59) Concluding Remarks(35:00) AcknowledgementsThe original text contained 24 footnotes which were omitted from this narration. --- First published: October 23rd, 2023 Source: https://forum.effectivealtruism.org/posts/S9H86osFKhfFBCday/how-bad-would-human-extinction-be --- Narrated by TYPE III AUDIO.

This is a selection of highlights from episode #163 of The 80,000 Hours Podcast.These aren't necessarily the most important, or even most entertaining parts of the interview — and if you enjoy this, we strongly recommend checking out the full episode:Toby Ord on the perils of maximising the good that you doAnd if you're finding these highlights episodes valuable, please let us know by emailing podcast@80000hours.org.Highlights put together by Simon Monsour, Milo McGuire, and Dominic Armstrong

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The possibility of an indefinite AI pause, published by Matthew Barnett on September 19, 2023 on The Effective Altruism Forum. This post is part of AI Pause Debate Week. Please see this sequence for other posts in the debate. tl;dr An indefinite AI pause is a somewhat plausible outcome and could be made more likely if EAs actively push for a generic pause. I think an indefinite pause proposal is substantially worse than a brief pause proposal, and would probably be net negative. I recommend that alternative policies with greater effectiveness and fewer downsides should be considered instead. Broadly speaking, there seem to be two types of moratoriums on technologies: (1) moratoriums that are quickly lifted, and (2) moratoriums that are later codified into law as indefinite bans. In the first category, we find the voluntary 1974 moratorium on recombinant DNA research, the 2014 moratorium on gain of function research, and the FDA's partial 2013 moratorium on genetic screening. In the second category, we find the 1958 moratorium on conducting nuclear tests above the ground (later codified in the 1963 Partial Nuclear Test Ban Treaty), and the various moratoriums worldwide on human cloning and germline editing of human genomes. In these cases, it is unclear whether the bans will ever be lifted - unless at some point it becomes infeasible to enforce them. Overall I'm quite uncertain about the costs and benefits of a brief AI pause. The foreseeable costs of a brief pause, such as the potential for a compute overhang, have been discussed at length by others, and I will not focus my attention on them here. I recommend reading this essay to find a perspective on brief pauses that I'm sympathetic to. However, I think it's also important to consider whether, conditional on us getting an AI pause at all, we're actually going to get a pause that quickly ends. I currently think there is a considerable chance that society will impose an indefinite de facto ban on AI development, and this scenario seems worth analyzing in closer detail. Note: in this essay, I am only considering the merits of a potential lengthy moratorium on AI, and I freely admit that there are many meaningful axes on which regulatory policy can vary other than "more" or "less". Many forms of AI regulation may be desirable even if we think a long pause is not a good policy. Nevertheless, it still seems worth discussing the long pause as a concrete proposal of its own. The possibility of an indefinite pause Since an "indefinite pause" is vague, let me be more concrete. I currently think there is between a 10% and 50% chance that our society will impose legal restrictions on the development of advanced AI systems that, Prevent the proliferation of advanced AI for more than 10 years beyond the counterfactual under laissez-faire Have no fixed, predictable expiration date (without necessarily lasting forever) Eliezer Yudkowsky, perhaps the most influential person in the AI risk community, has already demanded an "indefinite and worldwide" moratorium on large training runs. This sentiment isn't exactly new. Some effective altruists, such as Toby Ord, have argued that humanity should engage in a "long reflection" before embarking on ambitious and irreversible technological projects, including AGI. William MacAskill suggested that this pause should perhaps last "a million years". Two decades ago, Nick Bostrom considered the ethics of delaying new technologies in a utilitarian framework and concluded a delay of "over 10 million years" may be justified if it reduces existential risk by a single percentage point. I suspect there are approximately three ways that such a pause could come about. The first possibility is that governments could explicitly write such a pause into law, fearing the development of AI in a broad sense,...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A Love Letter to EA, published by utilistrutil on September 10, 2023 on The Effective Altruism Forum. Dear EA, I love you. In hindsight, my life before we met looks like it was designed to prepare me for you, like Fate left her fingerprints on so many formative books and conversations. Yet, for all that preparation, when my friend finally introduced us, I almost missed you! "Earning to give-me-a-break, am I right?" I didn't recognize you at first as the one I'd been looking for - I didn't even know I'd been looking. But that encounter made an indelible impression on me, one that eventually demanded a Proper Introduction, this time performed in your own words. At second sight, I could make out the form of the one I am now privileged to know so well. We survived the usual stages: talking, seeing each other casually, managing a bit of long (inferential) distance. It has been said that love is about changing and being changed - Lord, how I've changed. And you have too! I loved you when CEA was a basement, I loved you in your crypto era, and I'll love you through whatever changes the coming years may bring. I will admit, however, that thinking about the future scares me. I don't mean the far future (though I worry about that, too, of course); I mean our future. We're both so young, and there's so much that could go wrong. What if we make mistakes? What if I can't find an impactful job? What if your health (epistemic or otherwise) fails? In this light, our relationship appears quite fragile. All I can do is remind myself that we have weathered many crises together, and every day grants us greater understanding of the challenges that lie ahead and deeper humility with which to meet them. I also understand you better with each passing year. And that's a relief because, let's face it, loving you is complicated! There's so much to understand. Sometimes I feel like I'll never fully Get You, and then I feel a little jealous toward people who have known you longer or seem to know a different side of you. When I get to thinking this way, I am tempted to pronounce that I am only "adjacent" to you, but we both know that this would be true only in the sense that a wave is adjacent to the ocean. And you? You make me feel seen in a way I never thought was possible. We tend to talk the most when some Issue requires resolution, but I hope you know that for every day we argue, there are 99 days when I think of you with nothing but fondness. 99 days when I relish your companionship and delight in my memories with you: traveling the world, reading in the park, raising the next generation, talking late into the night, bouncing a spikeball off Toby Ord's window . . . I love your tweets, even when they make me cringe, I trust your judgement, even after you buy a castle, and I cherish a meal with you, even when it's a lukewarm bottle of Huel shared in an Oxford train station. When we disagree, I love how you challenge me to refine my map of the world. I am constantly impressed by your boundless empathy, and I am so thankful for everything you've taught me. I love your eccentric friends and rationalist neighbors. My parents like you too, by the way, even if they don't really get what I see in you. I love you x 99. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Effective altruism is associated with the slogan "do the most good." On one level, this has to be unobjectionable: What could be bad about helping people more and more?But in today's interview, Toby Ord — moral philosopher at the University of Oxford and one of the founding figures of effective altruism — lays out three reasons to be cautious about the idea of maximising the good that you do. He suggests that rather than “doing the most good that we can,” perhaps we should be happy with a more modest and manageable goal: “doing most of the good that we can.”Links to learn more, summary and full transcript.Toby was inspired to revisit these ideas by the possibility that Sam Bankman-Fried, who stands accused of committing severe fraud as CEO of the cryptocurrency exchange FTX, was motivated to break the law by a desire to give away as much money as possible to worthy causes.Toby's top reason not to fully maximise is the following: if the goal you're aiming at is subtly wrong or incomplete, then going all the way towards maximising it will usually cause you to start doing some very harmful things.This result can be shown mathematically, but can also be made intuitive, and may explain why we feel instinctively wary of going “all-in” on any idea, or goal, or way of living — even something as benign as helping other people as much as possible.Toby gives the example of someone pursuing a career as a professional swimmer. Initially, as our swimmer takes their training and performance more seriously, they adjust their diet, hire a better trainer, and pay more attention to their technique. While swimming is the main focus of their life, they feel fit and healthy and also enjoy other aspects of their life as well — family, friends, and personal projects.But if they decide to increase their commitment further and really go all-in on their swimming career, holding back nothing back, then this picture can radically change. Their effort was already substantial, so how can they shave those final few seconds off their racing time? The only remaining options are those which were so costly they were loath to consider them before.To eke out those final gains — and go from 80% effort to 100% — our swimmer must sacrifice other hobbies, deprioritise their relationships, neglect their career, ignore food preferences, accept a higher risk of injury, and maybe even consider using steroids.Now, if maximising one's speed at swimming really were the only goal they ought to be pursuing, there'd be no problem with this. But if it's the wrong goal, or only one of many things they should be aiming for, then the outcome is disastrous. In going from 80% to 100% effort, their swimming speed was only increased by a tiny amount, while everything else they were accomplishing dropped off a cliff.The bottom line is simple: a dash of moderation makes you much more robust to uncertainty and error.As Toby notes, this is similar to the observation that a sufficiently capable superintelligent AI, given any one goal, would ruin the world if it maximised it to the exclusion of everything else. And it follows a similar pattern to performance falling off a cliff when a statistical model is 'overfit' to its data.In the full interview, Toby also explains the “moral trade” argument against pursuing narrow goals at the expense of everything else, and how consequentialism changes if you judge not just outcomes or acts, but everything according to its impacts on the world.Toby and Rob also discuss:The rise and fall of FTX and some of its impactsWhat Toby hoped effective altruism would and wouldn't become when he helped to get it off the groundWhat utilitarianism has going for it, and what's wrong with it in Toby's viewHow to mathematically model the importance of personal integrityWhich AI labs Toby thinks have been acting more responsibly than othersHow having a young child affects Toby's feelings about AI riskWhether infinities present a fundamental problem for any theory of ethics that aspire to be fully impartialHow Toby ended up being the source of the highest quality images of the Earth from spaceGet this episode by subscribing to our podcast on the world's most pressing problems and how to solve them: type ‘80,000 Hours' into your podcasting app. Or read the transcript.Producer and editor: Keiran HarrisAudio Engineering Lead: Ben CordellTechnical editing: Simon MonsourTranscriptions: Katy Moore

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: New Princeton course on longtermism, published by Calvin Baker on September 2, 2023 on The Effective Altruism Forum. This semester (Fall 2023), Prof Adam Elga and I will be co-instructing Longtermism, Existential Risk, and the Future of Humanity, an upper div undergraduate philosophy seminar at Princeton. (Yes, I did shamelessly steal half of our title from The Precipice.) We are grateful for support from an Open Phil course development grant and share the reading list here for all who may be interested. Part 1: Setting the stage Week 1: Introduction to longtermism and existential risk Core Ord, Toby. 2020. The Precipice: Existential Risk and the Future of Humanity. London: Bloomsbury. Read introduction, chapter 1, and chapter 2 (pp. 49-56 optional); chapters 4-5 optional but highly recommended. Optional Roser (2022) "The Future is Vast: Longtermism's perspective on humanity's past, present, and future" Our World in Data Karnofsky (2021) 'This can't go on' Cold Takes (blog) Kurzgesagt (2022) "The Last Human - A Glimpse into the Far Future" Week 2: Introduction to decision theory Core Weisberg, J. (2021). Odds & Ends. Read chapters 8, 11, and 14. Ord, T., Hillerbrand, R., & Sandberg, A. (2010). "Probing the improbable: Methodological challenges for risks with low probabilities and high stakes." Journal of Risk Research, 13(2), 191-205. Read sections 1-2. Optional Weisberg, J. (2021). Odds & Ends chapters 5-7 (these may be helpful background for understanding chapter 8, if you don't have much background in probability). Titelbaum, M. G. (2020) Fundamentals of Bayesian Epistemology chapters 3-4 Week 3: Introduction to population ethics Core Parfit, Derek. 1984. Reasons and Persons. Oxford: Oxford University Press. Read sections 4.16.120-23, 125, and 127 (pp. 355-64; 366-71, and 377-79). Parfit, Derek. 1986. "Overpopulation and the Quality of Life." In Applied Ethics, ed. P. Singer, 145-164. Oxford: Oxford University Press. Read sections 1-3. Optional Remainders of Part IV of Reasons and Persons and "Overpopulation and the Quality of Life" Greaves (2017) "Population Axiology" Philosophy Compass McMahan (2022) "Creating People and Saving People" section 1, first page of section 4, and section 8 Temkin (2012) Rethinking the Good 12.2 pp. 416-17 and section 12.3 (esp. pp. 422-27) Harman (2004) "Can We Harm and Benefit in Creating?" Roberts (2019) "The Nonidentity Problem" SEP Frick (2022) "Context-Dependent Betterness and the Mere Addition Paradox" Mogensen (2019) "Staking our future: deontic long-termism and the non-identity problem" sections 4-5 Week 4: Longtermism: for and against Core Greaves, Hilary and William MacAskill. 2021. "The Case for Strong Longtermism." Global Priorities Institute Working Paper No.5-2021. Read sections 1-6 and 9. Curran, Emma J. 2023. "Longtermism and the Complaints of Future People". Forthcoming in Essays on Longtermism, ed. H. Greaves, J. Barrett, and D. Thorstad. Oxford: OUP. Read section 1. Optional Thorstad (2023) "High risk, low reward: A challenge to the astronomical value of existential risk mitigation." Focus on sections 1-3. Curran, E. J. (2022). "Longtermism, Aggregation, and Catastrophic Risk" (GPI Working Paper 18-2022). Global Priorities Institute. Beckstead (2013) "On the Overwhelming Importance of Shaping the Far Future" Chapter 3 "Toby Ord on why the long-term future of humanity matters more than anything else, and what we should do about it" 80,000 Hours podcast Frick (2015) "Contractualism and Social Risk" sections 7-8 Part 2: Philosophical problems Week 5: Fanaticism Core Bostrom, N. (2009). "Pascal's mugging." Analysis, 69 (3): 443-445. Russell, J. S. "On two arguments for fanaticism." Noûs, forthcoming. Read sections 1, 2.1, and 2.2. Temkin, L. S. (2022). "How Expected Utility Theory Can Drive Us Off the Rails." In L. S. ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: [Crosspost] An AI Pause Is Humanity's Best Bet For Preventing Extinction (TIME), published by Otto on July 25, 2023 on The Effective Altruism Forum. Otto Barten is director of the Existential Risk Observatory, a nonprofit aiming to reduce existential risk by informing the public debate. Joep Meindertsma is founder of PauseAI, a movement campaigning for an AI Pause.The existential risks posed by artificial intelligence (AI) are now widely recognized. After hundreds of industry and science leaders warned that "mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war," the U.N. Secretary-General recently echoed their concern. So did the prime minister of the U.K., who is also investing 100 million pounds into AI safety research that is mostly meant to prevent existential risk. Other leaders are likely to follow in recognizing AI's ultimate threat. In the scientific field of existential risk, which studies the most likely causes of human extinction, AI is consistently ranked at the top of the list. In The Precipice, a book by Oxford existential risk researcher Toby Ord that aims to quantify human extinction risks, the likeliness of AI leading to human extinction exceeds that of climate change, pandemics, asteroid strikes, supervolcanoes, and nuclear war combined. One would expect that even for severe global problems, the risk that they lead to full human extinction is relatively small, and this is indeed true for most of the above risks. AI, however, may cause human extinction if only a few conditions are met. Among them is human-level AI, defined as an AI that can perform a broad range of cognitive tasks at least as well as we can. Studies outlining these ideas were previously known, but new AI breakthroughs have underlined their urgency: AI may be getting close to human level already. Recursive self-improvement is one of the reasons why existential-risk academics think human-level AI is so dangerous. Because human-level AI could do almost all tasks at our level, and since doing AI research is one of those tasks, advanced AI should therefore be able to improve the state of AI. Constantly improving AI would create a positive feedback loop with no scientifically established limits: an intelligence explosion. The endpoint of this intelligence explosion could be a superintelligence: a godlike AI that outsmarts us the way humans often outsmart insects. We would be no match for it. A godlike, superintelligent AI A superintelligent AI could therefore likely execute any goal it is given. Such a goal would be initially introduced by humans, but might come from a malicious actor, or not have been thought through carefully, or might get corrupted during training or deployment. If the resulting goal conflicts with what is in the best interest of humanity, a superintelligence would aim to execute it regardless. To do so, it could first hack large parts of the internet and then use any hardware connected to it. Or it could use its intelligence to construct narratives that are extremely convincing to us. Combined with hacked access to our social media timelines, it could create a fake reality on a massive scale. As Yuval Harari recently put it: "If we are not careful, we might be trapped behind a curtain of illusions, which we could not tear away - or even realise is there." As a third option, after either legally making money or hacking our financial system, a superintelligence could simply pay us to perform any actions it needs from us. And these are just some of the strategies a superintelligent AI could use in order to achieve its goals. There are likely many more. Like playing chess against grandmaster Magnus Carlsen, we cannot predict the moves he will play, but we can predict the outcome: we los...

Since writing The Precipice, one of my aims has been to better understand how reducing existential risk compares with other ways of influencing the l…--- First published: July 18th, 2023 Source: https://forum.effectivealtruism.org/posts/Doa69pezbZBqrcucs/shaping-humanity-s-longterm-trajectory Linkpost URL:http://files.tobyord.com/shaping-humanity's-longterm-trajectory.pdf --- Narrated by TYPE III AUDIO. Share feedback on this narration.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Shaping Humanity's Longterm Trajectory, published by Toby Ord on July 18, 2023 on The Effective Altruism Forum. Since writing The Precipice, one of my aims has been to better understand how reducing existential risk compares with other ways of influencing the longterm future. Helping avert a catastrophe can have profound value due to the way that the short-run effects of our actions can have a systematic influence on the long-run future. But it isn't the only way that could happen. For example, if we advanced human progress by a year, perhaps we should expect to see us reach each subsequent milestone a year earlier. And if things are generally becoming better over time, then this may make all years across the whole future better on average. I've developed a clean mathematical framework in which possibilities like this can be made precise, the assumptions behind them can be clearly stated, and their value can be compared. The starting point is the longterm trajectory of humanity, understood as how the instantaneous value of humanity unfolds over time. In this framework, the value of our future is equal to the area under this curve and the value of altering our trajectory is equal to the area between the original curve and the altered curve. This allows us to compare the value of reducing existential risk to other ways our actions might improve the longterm future, such as improving the values that guide humanity, or advancing progress. Ultimately, I draw out and name 4 idealised ways our short-term actions could change the longterm trajectory: advancements speed-ups gains enhancements And I show how these compare to each other, and to reducing existential risk. While the framework is mathematical, the maths in these four cases turns out to simplify dramatically, so anyone should be able to follow it. My hope is that this framework, and this categorisation of some of the key ways we might hope to shape the longterm future, can improve our thinking about longtermism. Some upshots of the work: Some ways of altering our trajectory only scale with humanity's duration or its average value - but not both. There is a serious advantage to those that scale with both: speed-ups, enhancements, and reducing existential risk. When people talk about 'speed-ups', they are often conflating two different concepts. I disentangle these into advancements and speed-ups, showing that we mainly have advancements in mind, but that true speed-ups may yet be possible. The value of advancements and speed-ups depends crucially on whether they also bring forward the end of humanity. When they do, they have negative value. It is hard for pure advancements to compete with reducing existential risk as their value turns out not to scale with the duration of humanity's future. Advancements are competitive in outcomes where value increases exponentially up until the end time, but this isn't likely over the very long run. Work on creating longterm value via advancing progress is most likely to compete with reducing risk if the focus is on increasing the relative progress of some areas over others, in order to make a more radical change to the trajectory. The work is appearing as a chapter for the forthcoming book, Essays on Longtermism, but as of today, you can also read it online here. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: An Overview of the AI Safety Funding Situation, published by Stephen McAleese on July 12, 2023 on The Effective Altruism Forum. Introduction AI safety is a field concerned with preventing negative outcomes from AI systems and ensuring that AI is beneficial to humanity. The field does research on problems such as the AI alignment problem which is the problem of designing AI systems that follow user intentions and behave in a desirable and beneficial way. Understanding and solving AI safety problems may involve reading past research, producing new research in the form of papers or posts, running experiments with ML models, and so on. Producing research typically involves many different inputs such as research staff, compute, equipment, and office space. These inputs all require funding and therefore funding is a crucial input for enabling or accelerating AI safety research. Securing funding is usually a prerequisite for starting or continuing AI safety research in industry, in an academic setting, or independently. There are many barriers that could prevent people from working on AI safety. Funding is one of them. Even if someone is working on AI safety, a lack of funding may prevent them from continuing to work on it. It's not clear how hard AI safety problems like AI alignment are. But in any case, humanity is more likely to solve them if there are hundreds or thousands of brilliant minds working on them rather than one guy. I would like there to be a large and thriving community of people working on AI safety and I think funding is an important prerequisite for enabling that. The goal of this post is to give the reader a better understanding of funding opportunities in AI safety so that hopefully funding will be less of a barrier if they want to work on AI safety. The post starts with a high-level overview of the AI safety funding situation followed by a more in-depth description of various funding opportunities. Past work To get an overview of AI safety spending, we first need to find out how much is spent on it per year. We can use past work as a prior and then use grant data to find a more accurate estimate. Changes in funding in the AI safety field (2017) by the Center for Effective Altruism estimated the change in AI safety funding between 2014 - 2017. In 2017, the post estimated that total AI safety spending was about $9 million. How are resources in effective altruism allocated across issues? (2020) by 80,000 Hours estimated the amount of money spent by EA on AI safety in 2019. Using data from the Open Philanthropy grants database, the post says that EA spent about $40 million on AI safety globally in 2019. In The Precipice (2020), Toby Ord estimated that between $10 million and $50 million was spent on reducing AI risk in 2020. 2021 AI Alignment Literature Review and Charity Comparison is an in-depth review of AI safety organizations and grantmakers and has a lot of relevant information. Overview of global AI safety funding One way to estimate total global spending on AI safety is to aggregate the total donations of major AI safety funds such as Open Philanthropy (Open Phil). It's important to note that the definition of 'AI safety' I'm using is AI safety research that is focused on reducing risks from advanced AI (AGI) such as existential risks which is the type of AI safety research I think is more neglected and important than other research in the long term. Therefore my analysis will be focused on EA funds and top AI labs and I don't intend to measure investment on near-term AI safety concerns such as effects on the labor market, fairness, privacy, ethics, disinformation, etc. The results of this analysis are shown in the following bar chart which was created in Google Sheets (link) and is based on data from analyzing grant databases from Open Philanthro...

Adam Thierer profileAdam's work:Microsoft's New AI Regulatory Framework & the Coming Battle over Computational ControlWhat If Everything You've Heard about AI Policy is Wrong?Can We Predict the Jobs and Skills Needed for the AI Era?Flexible, Pro-Innovation Governance Strategies for Artificial IntelligenceU.S. Chamber AI Commission Report Offers Constructive Path ForwardThe Coming Onslaught of “Algorithmic Fairness” RegulationsCorbin's review of Toby Ord's book The Precipice:World to End; Experts Hardest HitAlso:An article taking aim at the “IAEA for AI” concept!

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: If you find EA conferences emotionally difficult, you're not alone, published by Amber Dawn on May 22, 2023 on The Effective Altruism Forum. I went to EAG London this weekend. I had some interesting chats, wrote some cryptic squiggles in my notebook (“Clockify” “the Easterlin paradox”, “functionalist eudaimonic theories”), and gave and received some hopefully-useful advice. Overall, the conference was fun and worthwhile for me. But at times, I also found the conference emotionally difficult. I think this is pretty common. After last year's EAG, Alastair Fraser-Urquhart wrote about how he burnt out at the conference and had to miss a retreat starting the next day. The post was popular, and many said they'd had similar experiences. The standard euphemism for this facet of EA conferences is ‘intense' or ‘tiring', but I suspect these adjectives are often a more socially-acceptable way of saying ‘I feel low/anxious/exhausted and want to curl up in a foetal position in a darkened room'. I want to write this post to: balance out the ‘woo EAG lfg!' hype, and help people who found it a bad or ambivalent experience to feel less alone dig into to why EAGs can be difficult: this might help attendees have better experiences themselves, and also create an environment where others are more likely to have good experiences help people who mostly enjoy EAGs understand what their more neurotic or introverted friends are going through Here are some reasons that EAGs might be emotionally difficult. Some of these I've experienced personally, others are based on comments I've heard, and others are plausible educated guesses. It's easy to compare oneself (negatively) to others EA conferences are attended by a bunch of “impressive” people: big-name EAs like Will MacAskill and Toby Ord, entrepreneurs, organisation leaders, politicians, and “inner-circle-y” people who are Forum- or Twitter-famous. You've probably scheduled meetings with people because they're impressive to you; perhaps you're seeking mentorship and advice from people who are more senior or advanced in your field, or you want to talk to someone because they have cool ideas. This can naturally inflame impostor syndrome, feelings of inadequacy, and negative comparisons. Everyone seems smarter, harder-working, more agentic, better informed. Everyone's got it all figured out, while you're still stuck at Stage 2 of 80k's career planning process. Everyone expects you to have a plan to save the world, and you don't even have a plan for how to start making a plan. Most EAs, I think, know that these thought patterns are counterproductive. But even if some rational part of you knows this, it can still be hard to fight them - especially if you're tired, scattered, or over-busy, since this makes it harder to employ therapeutic coping mechanisms. The stakes are high We're trying to solve immense, scary problems. We (and CEA) pour so much time and money into these conferences because we hope that they'll help us make progress on those problems. This can make the conferences anxiety-inducing - you really really hope that the conference pays off. This is especially true if you have some specific goal - such as finding a job, collaborators or funders - or if you think the conference has a high opportunity cost for you. You spend a lot of time talking about depressing things This is just part of being an EA, of course, but most of us don't spend all our time directly confronting the magnitude of these problems. Having multiple back-to-back conversations about ‘how can we solve [massive, seemingly-intractable problem]?' can be pretty discouraging. Everything is busy and frantic You're constantly rushing from meeting to meeting, trying not to bump into others who are doing the same. You see acquaintances but only have time to wave hello, because y...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: David Edmonds's biography of Parfit is out, published by Pablo on April 23, 2023 on The Effective Altruism Forum. Parfit: A Philosopher and His Mission to Save Morality, by David Edmonds, was published a few days ago (Amazon | Audible | Goodreads | Oxford). The book is worth reading in full, but here are some passages that focus specifically on Parfit's involvement with effective altruism: In his retirement, an organization with which he felt a natural affinity approached him for support. The Oxford-based charity Giving What We Can (GWWC) was set up by Toby Ord, an Australian-born philosopher, and another young philosopher, Will MacAskill. The charity's direct inspiration came from Peter Singer. Singer had devised a thought experiment that was simple but devastatingly effective in making people reflect on their behavior and their attitude to those in need. [...] This thought experiment has generated a significant secondary literature. Certainly, the GWWC initiators found it compelling. Their mission was to persuade people to give more of their income away and to donate to effective organizations that could make a real difference. [...] There were spinoff organizations from GWWC such as 80,000 Hours. The number refers to the rough number of hours we might have in our career, and 80,000 Hours was set up to research how people can most effectively devote their time rather than their money to tackling the world's most pressing problems. In 2012, the Centre for Effective Altruism was established to incorporate both GWWC and 80,000 Hours. Since its launch, the effective altruism movement has grown slowly but steadily. Most of the early backers were idealistic young postgraduates, many of them philosophers. If Singer was the intellectual father of the movement, Parfit was its grandfather. It became an in-joke among some members that anyone who came to work for GWWC had to possess a copy of Reasons and Persons. Some owned two copies: one for home, one for the office. But it took Parfit until 2014 to sign the GWWC pledge. And he agreed to do so only after wrangling over the wording. Initially, those who joined the GWWC campaign were required to make a public pledge to donate at least 10% of their income to charities that worked to relieve poverty. Parfit had several issues with this. For reasons the organizers never understood, he said that the participants had to make a promise rather than a pledge. He may have believed that a promise entailed a deeper level of commitment. Nor was he keen on the name "Giving What We Can". 10% of a person's income is certainly a generous sum, and in line with what adherents to some world religions are expected to give away. Nevertheless, Parfit pointed out, it was obvious that people could donate more. [...] Parfit also caviled at the word 'giving'. He believed this implied we are morally entitled to what we hand over, and morally entitled to our wealth and high incomes. This he rejected. Well-off people in the developed world were merely lucky that they were born into rich societies: they did not deserve their fortune. Linguistic quibbles aside, the issue that Parfit felt most strongly about was the movement's sole focus, initially, on poverty and development. While it was indeed pressing to relieve the suffering of people living today, Parfit argued, there should be an option that at least some of the money donated be earmarked for the problems of tomorrow. The human population has risen to a billion, and faces existential risks such as meteors, nuclear war, bioterrorism, pandemics, and climate change. Parfit claimed that between (A) peace, (B) a war which kills 7.5 billion people and (C) a war which killed everyone, the difference between (B) and (C) was much greater than the difference between (A) and (B). [...] Given how grim human exist...

SHOW THEME Even with no f***s to give, James and Catherine fill 38 minutes with movies and mummies before pondering humanity's precipice and agreeing that most art (other than "Jaws" props) is disposable.  SHOW NOTES 00:12 - Catherine is finally on mic 00:48 - Good sleep yields not giving a f*** 01:45 - Know your puppy limitations  03:25 - Sinking the coffee cup  04:35 - Have we got Ancient Egypt's mummies all wrong? (No) 05:30 - The perfect movie! 07:15 - Skimming the sarcophagus 08:30 - Fayium (portrait) vs faience (glaze technique) 10:10 - Remnants of a Roman Egypt  11:06 - Difference is... mummies are real 12:32 - Unwrapping colonial entitlement 15:00 - Art, anthropology, or morbid fascination? 16:25 - Humanity's eve of destruction? The Precipice: existential risk and the future of humanity - Toby Ord 21:35 - Ancient alien afterlife destruction? 22:40 - Ancient Egyptian Instagram Influencers 24:15 - Inherent Vice: What to do with Decaying Masterpiece? 25:00 - Miami Vice fashion rant 28:28 - Naum Gabo: "If it's gone it's gone" 29:30 - Artwork will eventually fail, so throw it out!  31:00 - Use of dead animals, chocolate and lard is not super edgy but is dumb and or, gross 32:25 - Recalling ASU's "Vague Art Show" and ditching painting at RISD 35:44 - Hirst "props" can't touch "Jaws" 37:00 - "Jaws" the musical 38:05 - Back to the garbage heap! 

Twenty years ago, Jason Matheny was a public health student who in his spare time was crusading to create a meat industry that would be less reliant on animals.  In 2004, after he founded New Harvest to popularize cultured meat, his fame grew. The New York Times profiled him in its annual “Ideas of the Year” feature in 2005. That same year Discover magazine named cultured meat one of the most notable tech stories. For the next several years, Jason was the face of the movement to grow real meat without animals, traveling the world to persuade governments and food companies alike that they should be investing in a future where people would eat meat, but not animals.  By 2009, now armed with his BA, MBA, MPH, and PhD, Jason began turning his attention toward preventing the more immediate and potentially catastrophic risks humanity faces. After leaving New Harvest, he eventually rose to become the director of Intelligence Advanced Research Projects Activity (IARPA), a federal agency that develops advanced technologies for national intelligence. Running the federal intelligence agency would eventually lead Jason to helm a national security center at Georgetown University, followed by a high-profile national security role in the Biden White House, to now being the CEO of the Rand Corporation. He was even named one of Foreign Policy's “Top 50 Global Thinkers.” As you'll hear in this interview, Jason shifted from his work on cultivated meat toward national security as he became convinced that technology can vastly improve both human and animal welfare, and that the only real threat to technological advancement is an apocalyptic catastrophe like a synthetic virus or asteroid. He still cares about the welfare of those of us living today—human and nonhuman alike—but Jason's primary preoccupation has become reducing civilization-threatening risks so that our species can keep progressing into the deep future. I think you'll find this conversation with this leading thinker as riveting as I did. Jason even talks about what technologies he hopes listeners will pursue to mitigate existential risks, so be sure to listen closely! Discussed in this episode Jason recommends reading The Precipice by Toby Ord. Jason passed the New Harvest torch onto Isha Datar, who was our guest on Episode 42. Our Episode 89 with Rep. Ro Khanna regarding his legislation relating to national security implications of losing the alt-meat race. Paul's thoughts in The Hill on government funding for alt-meat. More about Jason Matheny Jason Matheny is president and chief executive officer of the RAND Corporation, a nonprofit, nonpartisan research organization that helps improve policy and decisionmaking through research and analysis.  Prior to becoming RAND's president and CEO in July 2022, he led White House policy on technology and national security at the National Security Council and the Office of Science and Technology Policy. Previously, he was founding director of the Center for Security and Emerging Technology at Georgetown University and director of the Intelligence Advanced Research Projects Activity (IARPA), where he was responsible for developing advanced technologies for the U.S. intelligence community.  Before IARPA, he worked for Oxford University, the World Bank, the Applied Physics Laboratory, the Center for Biosecurity, and Princeton University. Matheny has served on many nonpartisan boards and committees, including the National Security Commission on Artificial Intelligence, to which he was appointed by Congress in 2018.  He is a recipient of the Intelligence Community's Award for Individual Achievement in Science and Technology, the National Intelligence Superior Service Medal, and the Presidential Early Career Award for Scientists and Engineers. He was also named one of Foreign Policy's “Top 50 Global Thinkers.”  Matheny holds a Ph.D. in applied economics from Johns Hopkins University, an M.P.H. from Johns Hopkins University, an M.B.A. from Duke University, and a B.A. in art history from the University of Chicago.

In this episode, we examine the topic of existential threat, focusing in particular on the subject of nuclear war. Sam opens the discussion by emphasizing the gravity of our ability to destroy life as we know it at any moment, and how shocking it is that nearly all of us perpetually ignore this fact. Philosopher Nick Bostrom expands on this idea by explaining how developing technologies like DNA synthesis could make humanity more vulnerable to malicious actors. Sam and historian Fred Kaplan then guide us through a hypothetical timeline of events following a nuclear first strike, highlighting the flaws in the concept of nuclear deterrence. Former Defense Secretary William J. Perry echoes these concerns, painting a grim picture of his "nuclear nightmare" scenario: a nuclear terrorist attack. Zooming out, Toby Ord outlines each potential extinction-level threat, and why he believes that, between all of them, we face a one in six chance of witnessing the downfall of our species. Our episode ends on a cautiously optimistic note, however, as Yuval Noah Harari shares his thoughts on "global myth-making" and its potential role in helping us navigate through these perilous times.   About the Series Filmmaker Jay Shapiro has produced The Essential Sam Harris, a new series of audio documentaries exploring the major topics that Sam has focused on over the course of his career. Each episode weaves together original analysis, critical perspective, and novel thought experiments with some of the most compelling exchanges from the Making Sense archive. Whether you are new to a particular topic, or think you have your mind made up about it, we think you'll find this series fascinating.  

In this episode, we examine the topic of existential threat, focusing in particular on the subject of nuclear war. Sam opens the discussion by emphasizing the gravity of our ability to destroy life as we know it at any moment, and how shocking it is that nearly all of us perpetually ignore this fact. Philosopher Nick Bostrom expands on this idea by explaining how developing technologies like DNA synthesis could make humanity more vulnerable to malicious actors. Sam and historian Fred Kaplan then guide us through a hypothetical timeline of events following a nuclear first strike, highlighting the flaws in the concept of nuclear deterrence. Former Defense Secretary William J. Perry echoes these concerns, painting a grim picture of his "nuclear nightmare" scenario: a nuclear terrorist attack. Zooming out, Toby Ord outlines each potential extinction-level threat, and why he believes that, between all of them, we face a one in six chance of witnessing the downfall of our species. Our episode ends on a cautiously optimistic note, however, as Yuval Noah Harari shares his thoughts on "global myth-making" and its potential role in helping us navigate through these perilous times. About the Series Filmmaker Jay Shapiro has produced The Essential Sam Harris, a new series of audio documentaries exploring the major topics that Sam has focused on over the course of his career. Each episode weaves together original analysis, critical perspective, and novel thought experiments with some of the most compelling exchanges from the Making Sense archive. Whether you are new to a particular topic, or think you have your mind made up about it, we think you’ll find this series fascinating.

Opening session: Toby Ord | Toby Ord | EAG Bay Area 23 - YouTubeEffective Altruism is a social movement dedicated to finding ways to do the most good possible, whether through charitable donations, career choices, or volunteer projects. EA Global conferences are gatherings for EAs to meet. You can also listen to this talk along with its accompanying video on YouTube.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Casting the Decisive Vote, published by Toby Ord on March 27, 2023 on The Effective Altruism Forum. The moral value of voting is a perennial topic in EA. This piece shows that in any election that isn't a forgone conclusion, the chance of your vote being decisive can't be much lower than 1 in the number of voters. So voting will be worth it around the point where the value your preferred candidate would bring to the average citizen exceeds the cost of you voting. What is the chance your vote changes the outcome of an election? We know it is low, but how low? In particular, how does it compare with an intuitive baseline of a 1 in n chance, where n is the number of voters? This baseline is an important landmark not only because it is so intuitive, but because it is roughly the threshold needed for voting to be justified in terms of the good it produces for the members of the community (since the total benefit is also going to be proportional to n). Some political scientists have tried to estimate it with simplified theoretical models involving random voting. Depending on their assumptions, this has suggested it is much higher than the baseline — roughly 1 in the square root of n (Banzhaf 1965) — or that it is extraordinarily lower — something like 1 in 10^2659 for a US presidential election (Brennan 2011). Statisticians have attempted to determine the chance of a vote being decisive for particular elections using detailed empirical modelling, with data from previous elections and contemporaneous polls. For example, Gelman et al (2010) use such a model to estimate that an average voter had a 1 in 60 million chance of changing the result of the 2008 US presidential election, which is about 3 times higher than the baseline. In contrast, I'll give a simple method that depends on almost no assumptions or data, and provides a floor for how low this probability can be. It will calculate this using just two inputs: the number of voters, n, and the probability of the underdog winning, p_u. The method works for any two-candidate election that uses simple majority. So it wouldn't work for the US presidential election, but would work for your chance of being decisive within your state, and could be combined with estimates that state is decisive nationally. It also applies for many minor ‘elections' you may encounter, such as the chance of your vote being decisive on a committee. We start by considering a probability distribution over what share of the vote a candidate will get, from 0% to 100%. In theory, this distribution could have any shape, but in practice it will almost always have a single peak (which could be at one end, or somewhere in between). We will assume that the probability distribution over vote share has this shape (that it is ‘unimodal') and this is the only substantive assumption we'll make. We will treat this as the probability distribution of the votes a candidate gets before factoring in your own vote. If there is an even number of votes (before yours) then your vote matters only if the vote shares are tied. In that case, which way you vote decides the election. If there is an odd number of votes (before yours), it is a little more complex, but works out about the same: Before your vote, one candidate has one fewer vote. Your vote decides whether they lose or tie, so is worth half an election. But because there are two different ways the candidates could be one vote apart (candidate A has one fewer or candidate B has one fewer), you are about twice as likely to end up in this situation, so have the same expected impact. For ease of presentation I'll assume there is an even number of voters other than you, but nothing turns on this. (In real elections, you may also have to worry about probabilistic recounts, but if you do the analysis, these don't substantivel...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Reflecting on the Last Year — Lessons for EA (opening keynote at EAG), published by Toby Ord on March 24, 2023 on The Effective Altruism Forum. I recently delivered the opening talk for EA Global: Bay Area 2023. I reflect on FTX, the differences between EA and utilitarianism, and the importance of character. Here's the recording and transcript. The Last Year Let's talk a bit about the last year. The spring and summer of 2022 were a time of rapid change. A second major donor had appeared, roughly doubling the amount of committed money. There was a plan to donate this new money more rapidly, and to use more of it directly on projects run by people in the EA community. Together, this meant much more funding for projects led by EAs or about effective altruism. It felt like a time of massive acceleration, with EA rapidly changing and growing in an attempt to find enough ways to use this money productively and avoid it going to waste. This caused a bunch of growing pains and distortions. When there was very little money in effective altruism, you always knew that the person next to you couldn't have been in it for the money — so they must have been in it because they were passionate about what they were doing for the world. But that became harder to tell, making trust harder. And the most famous person in crypto had become the most famous person in EA. So someone whose views and actions were quite radical and unrepresentative of most EAs, became the most public face of effective altruism, distorting public perception and even our self-perception of what it meant to be an EA. It also meant that EA became more closely connected to an industry that was widely perceived as sketchy. One that involved a product of disputed social value, and a lot of sharks. One thing that especially concerned me was a great deal of money going into politics. We'd tried very hard over the previous 10 years to avoid EA being seen as a left or right issue — immediately alienating half the population. But a single large donor had the potential to change that unilaterally. And EA became extremely visible: people who'd never heard of it all of a sudden couldn't get away from it, prompting a great deal of public criticism. From my perspective at the time, it was hard to tell whether or not the benefits of additional funding for good causes outweighed these costs — both were large and hard to compare. Even the people who thought it was worth the costs shared the feelings of visceral acceleration: like a white knuckled fairground ride, pushing us up to vertiginous heights faster than we were comfortable with. And that was just the ascent. Like many of us, I was paying attention to the problems involved in the rise, and was blindsided by the fall. As facts started to become more clear, we saw that the companies producing this newfound income had been very poorly governed, allowing behaviour that appears to me to have been both immoral and illegal — in particular, it seems that when the trading arm had foundered, customers' own deposits were raided to pay for an increasingly desperate series of bets to save the company. Even if that strategy had worked and the money was restored to the customers, I still think it would have been illegal and immoral. But it didn't work, so it also caused a truly vast amount of harm. Most directly and importantly to the customers, but also to a host of other parties, including the members of the EA community and thus all the people and animals we are trying to help. I'm sure most of your have thought a lot about this over last few months. I've come to think of my own attempts to process this as going through these four phases. First, there's: Understanding what happened. What were the facts on the ground? Were crimes committed? How much money have customer's lost? A lot of t...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The illusion of consensus about EA celebrities, published by Ben Millwood on March 17, 2023 on The Effective Altruism Forum. Epistemic status: speaking for myself and hoping it generalises I don't like everyone that I'm supposed to like: I've long thought that [redacted] was focused on all the wrong framings of the issues they discuss, [redacted] is on the wrong side of their disagreement with [redacted] and often seems to have kind of sloppy thinking about things like this, [redacted] says many sensible things but a writing style that I find intensely irritating and I struggle to get through; [redacted] is similar, but not as sensible, [redacted] is working on an important problem, but doing a kind of mediocre job of it, which might be crowding out better efforts. Why did I redact all those names? Well, my criticisms are often some mixture of: half-baked; I don't have time to evaluate everyone fairly and deeply, and don't need to in order to make choices about what to focus on, based on justifications that are not very legible or easy to communicate, not always totally central to their point or fatal to their work, kind of upsetting or discouraging to hear, often not that actionable. I want to highlight that criticisms like this will usually not surface, and while in individual instances this is sensible, in aggregate it may contribute to a misleading view of how we view our celebrities and leaders. We end up seeming more deferential and hero-worshipping than we really are. This is bad for two reasons: it harms our credibility in the eyes of outsiders (or insiders, even) who have negative views of those people, it projects the wrong expectation to newcomers who trust us and want to learn or adopt our attitudes. What to do about it? I think "just criticise people more" in isolation is not a good solution. People, even respected people in positions of leadership, often seem to already find posting on the Forum a stressful experience, and I think tipping that balance in the more brutal direction seems likely to cost more than it gains. I think you could imagine major cultural changes around how people give and receive feedback that could make this better, mitigate catastrophising about negative feedback, and ensure people feel safe to risk making mistakes or exposing their oversights. But those seem to me like heavy, ambitious pieces of cultural engineering that require a lot of buy-in to get going, and even if successful may incur ongoing frictional costs. Here's smaller, simpler things that could help: Write a forum post about it (this one's taken, sorry), Make disagreements more visible and more legible, especially among leaders or experts. I really enjoyed the debate between Will MacAskill and Toby Ord in the comments of Are we living at the most influential time in history? ­– you can't come away from that discussion thinking "oh, whatever the smart, respected people in EA think must be right", because either way at least one of them will disagree with you! There's a lot of disagreement on the Forum all the time, of course, but I have a (somewhat unfair) vibe of this as the famous people deposit their work into the forum and leave for higher pursuits, and then we in the peanut gallery argue over it. I'd love it if there were (say) a document out there that Redwood Research and Anthropic both endorsed, that described how their agendas differ and what underlying disagreements lead to those differences. Make sure people incoming to the community, or at the periphery of the community, are inoculated against this bias, if you spot it. Point out that people usually have a mix of good and bad ideas. Have some go-to examples of respected people's blind spots or mistakes, at least as they appear to you. (Even if you never end up explaining them to anyone, it's probably goo...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Against EA-Community-Received-Wisdom on Practical Sociological Questions, published by Michael Cohen on March 9, 2023 on The Effective Altruism Forum. In my view, there is a rot in the EA community that is so consequential that it inclines me to discourage effective altruists from putting much, if any, trust in EA community members, EA "leaders", the EA Forum, or LessWrong. But I think that it can be fixed, and the EA movement would become very good. In my view, this rot comes from incorrect answers to certain practical sociological questions, like: How important for success is having experience or having been apprenticed to someone experienced? Is the EA Forum a good tool for collaborative truth-seeking? How helpful is peer review for collaborative truth-seeking? Meta-1. Is "Defer to a consensus among EA community members" a good strategy for answering practical sociological questions? Meta-2. How accurate are conventional answers to practical sociological questions that many people want to get right? I'll spend a few sentences attempting to persuade EA readers that my position is not easily explained away by certain things they might call mistakes. Most of my recent friends are in the EA community. (I don't think EAs are cringe). I assign >10% probability to AI killing everyone, so I'm doing technical AI Safety research as a PhD student at FHI. (I don't think longtermism or sci-fi has corrupted the EA community). I've read the sequences, and I thought they were mostly good. (I'm not "inferentially distant"). I think quite highly of the philosophical and economic reasoning of Toby Ord, Will MacAskill, Nick Bostrom, Rob Wiblin, Holden Karnofsky, and Eliezer Yudkowsky. (I'm "value-aligned", although I object to this term). Let me begin with an observation about Amazon's organizational structure. From what I've heard, Team A at Amazon does not have to use the tool that Team B made for them. Team A is encouraged to look for alternatives elsewhere. And Team B is encouraged to make the tool into something that they can sell to other organizations. This is apparently how Amazon Web Services became a product. The lesson I want to draw from this is that wherever possible, Amazon outsources quality control to the market (external people) rather than having internal "value-aligned" people attempt to assess quality and issue a pass/fail verdict. This is an instance of the principle: "if there is a large group of people trying to answer a question correctly (like 'Is Amazon's tool X the best option available?'), and they are trying (almost) as hard as you to answer it correctly, defer to their answer." That is my claim; now let me defend it, not just by pointing at Amazon, and claiming that they agree with me. High-Level Claims Claim 1: If there is a large group of people trying to answer a question correctly, and they are trying (almost) as hard as you to answer it correctly, any consensus of theirs is more likely to be correct than you. There is extensive evidence (Surowiecki, 2004) that aggregating the estimates of many people produces a more accurate estimate as the number of people grows. It may matter in many cases that people are actually trying rather than just professing to try. If you have extensive and unique technical expertise, you might be able to say no one is trying as hard as you, because properly trying to answer the question correctly involves seeking to understand the implications of certain technical arguments, which only you have bothered to do. There is potentially plenty of gray area here, but hopefully, all of my applications of Claim 1 steer well clear of it. Let's now turn to Meta-2 from above. Claim 2: For practical sociological questions that many people want to get right, if there is a conventional answer, you should go with the conventional answer....

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Nelson Mandela's organization, The Elders, backing x risk prevention and longtermism, published by krohmal5 on February 1, 2023 on The Effective Altruism Forum. Excerpt from the web posting below. Strong endorsement of EA priorities from a decidedly not EA source—The Elders—an organization founded by Nelson Mandela in 2007. Although their reference to “approaching a precipice” and imagery that looks a lot like the Toby Ord book cover seems unlikely to be a coincidence? Also, thanks to Nathan for encouraging me to post this on the forum, a first for me. “For the next five years, our focus will be on the climate crisis, nuclear weapons, pandemics, and the ongoing scourge of conflict. The impact of these threats is already being seen on lives and livelihoods: a rapid rise in extreme weather events, a pandemic that killed millions and cost trillions, a war in which the use of nuclear weapons has been openly raised. Some of these threats jeopardise the very existence of human life on our planet, yet nations lack the ability or will to manage these risks. We are approaching a precipice. The urgency of the interconnected existential threats we face requires a crisis mindset from our leaders – one that puts our shared humanity centre-stage, leaves no one behind, and recognises the rights of future generations. We believe that when nations work together, these threats can all be addressed for the good of the whole world. But to do so, our political leaders must show purpose, moral courage and an urgency of action. This is why, over the next five years, we will work on three programmes that address existential threats to humanity requiring a collective response - the climate crisis, pandemics, and nuclear weapons - and also on conflict (a threat in itself, and a risk factor for other threats).” Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Holden Karnofsky is the co-CEO of Open Philanthropy and co-founder of GiveWell. He is also the author of one of the most interesting blogs on the internet, Cold Takes.We discuss:* Are we living in the most important century?* Does he regret OpenPhil's 30 million dollar grant to OpenAI in 2016?* How does he think about AI, progress, digital people, & ethics?Highly recommend!Watch on YouTube. Listen on Apple Podcasts, Spotify, or any other podcast platform. Read the full transcript here.Timestamps(0:00:00) - Intro(0:00:58) - The Most Important Century(0:06:44) - The Weirdness of Our Time(0:21:20) - The Industrial Revolution (0:35:40) - AI Success Scenario(0:52:36) - Competition, Innovation , & AGI Bottlenecks(1:00:14) - Lock-in & Weak Points(1:06:04) - Predicting the Future(1:20:40) - Choosing Which Problem To Solve(1:26:56) - $30M OpenAI Investment(1:30:22) - Future Proof Ethics(1:37:28) - Integrity vs Utilitarianism(1:40:46) - Bayesian Mindset & Governance(1:46:56) - Career AdviceTranscriptDwarkesh Patel  All right, today I have the pleasure of speaking with Holden Karnofsky who is the co-CEO of Open Philanthropy. In my opinion, Holden is one of the most interesting intellectuals alive. Holden, welcome to the Lunar Society. Holden Karnofsky Thanks for having me. The Most Important CenturyDwarkesh PatelLet's start off by talking about The Most Important Century thesis. Do you want to explain what this is for the audience? Holden Karnofsky My story is that I originally co-founded an organization called GiveWell that helps people decide where to give as effectively as possible. While I'm no longer as active as I once was there, I'm on its board. It's a website called GiveWell.org that makes good recommendations about where to give to charity to help a lot of people. As we were working at GiveWell, we met Cari Tuna and Dustin Moskovitz. Dustin is the co-founder of Facebook and Asana and we started a project that became Open Philanthropy to try to help them give away their large fortune and help as many people as possible. So I've spent my career looking for ways to do as much good as possible with a dollar, an hour, or basically whatever resources you have (especially with money).I've developed this professional specialization in looking for ideas that are underappreciated, underrated, and tremendously important because a lot of the time that's where I think you can find what you might call an “outsized return on investment.” There are opportunities to spend money and get an enormous impact because you're doing something very important that's being ignored by others. So it's through that kind of professional specialization that I've actively looked for interesting ideas that are not getting enough attention. Then I encountered the Effective Altruist Community, which is a community of people basically built around the idea of doing as much good as you can. It's through that community that I encountered the idea of the most important century. It's not my idea at all, I reached this conclusion with the help and input of a lot of people. The basic idea is that if we developed the right kind of AI systems this century (and that looks reasonably likely), this could make this century the most important of all time for humanity. So now let's talk about the basic mechanics of why that might be or how you might think about that. One thing is that if you look back at all of economic history ( the rate at which the world economy has grown), you see acceleration. You see that it's growing a lot faster today than it ever was. One theory of why that might be or one way of thinking about it through the lens of basic economic growth theory is that in normal circumstances, you can imagine a feedback loop where you have people coming up with ideas, and then the ideas lead to greater productivity and more resources.When you have more resources, you can also have more people, and then those people have more ideas. So you get this feedback loop that goes people, ideas, resources, people, ideas, resources. If you're starting a couple of hundred years ago and you run a feedback loop like that, standard economic theory says you'll get accelerating growth. You'll get a rate of economic growth that goes faster and faster. Basically, if you take the story of our economy to date and you plot it on a chart and do the kind of simplest thing you can to project it forward, you project that it will go that our economy will reach an infinite growth rate this century. The reason that I currently don't think that's a great thing to expect by default is that one of the steps of that feedback loop broke a couple hundred years ago. So it goes more people, more ideas, more resources, more people, more ideas, more resources. But, a couple hundred years ago, people stopped having more children when they had more resources. They got richer instead of more populous. This is all discussed on the Most Important Century page on my blog, Cold Takes. What happens right now is that when we have more ideas and we have more resources, we don't end up with more people as a result. We don't have that same accelerating feedback loop. If you had AI systems that could do all the things humans do to advance science and technology (meaning the AI systems could fill in that “more ideas” part of the loop), then you could get that feedback loop back. You could get sort of this unbounded, heavily accelerating, explosive growth in science and technology. That's the basic dynamic at the heart of it and a way of putting it that's trying to use familiar concepts from economic growth theory. Another way of putting it might just be, “Gosh, if we had AI systems that could do everything humans do to advance science and technology, that would be insane.” What if we were to take the things that humans do to create new technologies that have transformed the planet so radically and we were able to completely automate them so that every computer we have is potentially another mind working on advancing technology? So either way, when you think about it, you could imagine the world changing incredibly quickly and incredibly dramatically. I argue in the Most Important Century series that it looks reasonably likely, in my opinion, more than 50-50, that this century will see AI systems that can do all of the key tasks that humans do to advance science and technology. If that happens, we'll see explosive progress in science and technology. The world will quickly become extremely different from how it is today. You might think of it as if there was thousands of years of changes packed into a much shorter time period. If that happens, then I argue that you could end up in a deeply unfamiliar future. I give one example of what that might look like using this hypothetical technology idea called digital people. That would be sort of people that live in virtual environments that are kind of simulated, but also realistic and exactly like us. When you picture that kind of advanced world, I think there is a decent reason to think that if we did get that rate of scientific and technological advancement, we could basically hit the limits of science and technology. We could basically find most of what there is to find and end up with a civilization that expands well beyond this planet, has a lot of control over the environment, is very stable for very long periods of time, and looks sort of post-human in a lot of relevant ways. If you think that, then this is basically our last chance to shape how this happens. The most important century hypothesis in a nutshell is that if we develop AI that can do all the things humans do to advance science and technology, we could very quickly reach a very futuristic world that's very different from today's. It could be a very stable, very large world, and this is our last chance to shape it.The Weirdness of Our TimeDwarkesh Patel Gotcha. I and many other people are going to find that very wild. Could you walk us through the process by which you went from working in global development to thinking this way? In 2014, for example, you had an interview or a conversation and this is a quote from there. “I have looked at the situation in Africa, have an understanding of the situation in Africa, and see a path of doing a lot of good in Africa. I don't know how to look into the far future situation, don't understand the far future situation, and don't see a path to doing good on that front I feel good about.” Maybe you can walk me through how you got from there to where you are today.Holden Karnofsky Firstly, I want to connect this back to how this relates to the work I was doing at GiveWell, and why this all falls under one theme. If we are on the cusp for this century of creating these advanced AI systems, then we could be looking at a future that's either very good or very bad. I think there are decent arguments that if we move forward without caution and we develop sloppily designed AI systems, they could end up with goals of their own. We would end up with a universe that contains very little that humans value or a galaxy that does or a world where very powerful technologies are used by ill-meaning governments to create a world that isn't very good. We could also end up with a world where we manage to eliminate a lot of forms of material scarcity and have a planet that's much better off than today's. A lot of what I ask is how can we help the most people possible per dollar spent? If you ask how we can help the most people possible per dollar spent, then funding some work to help shape that transition and make sure that we don't move forward too incautiously, and that we do increase the odds that we do get like a good future world instead of a bad future one, is helping a huge number of people per dollar spent. That's the motivation. You're quoting an argument I was having where we posted a transcript back in 2014–– a time that was part of my journey of getting here. I was talking to people who were saying, “Holden, you want to help a lot of people with your resources. You should be focused on this massive event that could be coming this century that very few people are paying attention to, and there might be a chance to make this go well or poorly for humanity.” So I was saying, “Gosh, like that sure is interesting.” And I did think it was interesting. That's why I was spending the time and having those conversations. But I said, “When I look at global poverty and global health, I see what I can do. I see the evidence. I see the actions I can take. I'm not seeing that with this stuff.” So what changed? I would say a good chunk of what changed is maybe like the most boring answer possible. I just kept at it. I was sitting there in 2014 saying, “Gosh, this is really interesting, but it's all a bit overwhelming. It's all a bit crazy. I don't know how I would even think about this. I don't know how I would come up with a risk from AI that I actually believed was a risk and could do something about today.” Now, I've just been thinking about this for a much longer time period. I do believe that most things you could say about the far future are very unreliable and not worth taking action on, but I think there are a few things one might say about what a transition to very powerful AI systems could look like. There are some things I'm willing to say would be bad if AI systems were poorly designed, had goals of their own, and ended up kind of running the world instead of humans. That seems bad.I am more familiar today than I was then with the research and the work people can do to make that less likely and the actions people can take to make that less likely–– so that's probably more than half the answer. But another thing that would be close to half the answer is that I think there have been big changes in the world of AI since then. 2014 was the beginning of what's sometimes called the “deep learning revolution”. Since then, we've basically seen these very computationally intensive (but fundamentally simple) AI systems achieve a lot of progress on lots of different unrelated tasks. It's not crazy to imagine that the current way people are developing AI systems, cutting-edge AI systems, could take us all the way to the kind of extremely powerful AI systems that automate roughly everything humans do to advance science and technology. It's not so wild to imagine that we could just keep on going with these systems, make them bigger, put more work into them, but basically stay on the same path and you could get there. If you imagine doing that, it becomes a little bit less daunting to imagine the risks that might come up and the things we could do about them. So I don't think it's necessarily the leading possibility, but it's enough to start thinking concretely about the problem. Dwarkesh Patel Another quote from the interview that I found appealing was “Does the upper crust of humanity have a track record of being able to figure out the kinds of things MIRI claims to have figured out?” By the way, for context for the viewers, MIRI is the organization Eliezer was leading, which is who you were talking to at the time. Holden Karnofsky I don't remember exactly what kinds of things MIRI was trying to figure out and I'm not sure that I even understood what they were that well. I definitely think it is true that it is hard to predict the future, no matter who you are, no matter how hard you think, and no matter how much you've studied. I think parts of our “world” or memeplex or whatever you want to call it, overblow this at least a little bit. I think I was buying into that a little bit more than I could. In 2014, I would have said something like, “Gosh, no one's ever done something like making smart statements about what several decades out of our future could look like or making smart statements about what we would be doing today to prepare for it.” Since then, I think a bunch of people have looked into this and looked for historical examples of people making long-term predictions and long-term interventions. I don't think it's amazing, but I think I wrote a recent blog post entitled The Track Record of Future. It seems… fine. “Fine” is how I put it, where I don't think there's anyone who has demonstrated a real ability to predict the future with precision and know exactly what we should do. I also don't think humans' track record of this is so bad and so devastating that we shouldn't think we are capable of at least giving it a shot. If you enter into this endeavor with self-awareness about the fact that everything is less reliable than it appears and feels at first glance and you look for the few things that you would really bet on, I think it's worth doing. I think it's worth the bet. My job is to find 10 things we could do, and have nine of them fail embarrassingly, on the off chance that one of them becomes such a big hit that it makes up for everything else. I don't think it's totally crazy to think we could make meaningful statements about how things we do today could make these future events go better, especially if the future events are crazily far away (especially if they're within the next few decades.) That's something I've changed my mind on, at least to some degree. Dwarkesh Patel Gotcha. Okay, so we'll get to forecasting in a second, but let's continue on the object-level conversation about the most important century. I want to make sure I have the thesis right. Is the argument that because we're living in a weird time, we shouldn't be surprised if something transformative happens in a century or is the argument that something transformative could happen this century? Holden Karnofsky It's a weird time. So something we haven't covered yet, but I think is worth throwing in is that a significant part of the ‘Most Important Century series' is making the case that even if you ignore AI, there's a lot of things that are very strange about the time that our generation lives in. The reason I spent so much effort on this is because back in 2014, my number one objection to these stories about transformative AI wasn't anything about whether the specific claims about AI or economic models or alignment research made sense. This whole thing sounded crazy and was just suspicious. It's suspicious if someone says to you, “You know, this could be the most important century of all time for humanity.” I titled the series that way because I wanted people to know that I was saying something crazy and that I should have to defend it. I didn't want to be backpedaling or soft-pedaling or hiding what a big claim I was making. I think my biggest source of skepticism was how I didn't have any specific objection. It sounds crazy and suspicious to say that we might live in one of the most significant times of the most significant time for humanity ever. So a lot of my series is saying that it is weird to think that, but we already have a lot of evidence that we live in an extraordinarily weird time that would be on the short list of contenders for the most important time ever–– even before you get into anything about AI, and just used completely commonly accepted facts about the world. For example, if you chart the history of economic growth, you'll see that the last couple hundred years have seen faster growth by a lot than anything else in the history of humanity or the world. If you chart anything about scientific and technological developments, you'll see that everything significant is packed together in the recent past. There's almost no way to cut it. I've looked at many different cuts of this. There's almost no way to cut it that won't give you that conclusion. One way to put it is that the universe is something like 11 or 12 billion years old. Life on Earth is three billion years old. Human civilization is a blink of an eye compared to that. We're in this really tiny sliver of time, the couple hundred years when we've seen a huge amount of technological advancement and economic growth. So that's weird. I also talk about the fact that the current rate of economic growth seems high enough that we can't keep it going for that much longer. If it went for another 10,000 years, that's another blink of an eye and galactic time scales. It looks to me like we would run out of atoms in the galaxy and wouldn't have anywhere to go. So I think there are a lot of signs that we just live in a really strange time. One more thing that I'll just throw in there–– I think a lot of people who disagree with my take would say, “Look, I do believe eventually we will develop space colonization abilities. We could go to the stars, fill up the galaxy with life, and maybe have artificial general intelligence, but to say that this will happen in a century is crazy. I think it might be 500 years. I think it might be a thousand years. I think it might be 5000 years.” A big point I make in the series is how I say, “Well, even if it's 100000 years, that's still an extremely crazy time to be in in the scheme of things.” If you make a graphic timeline and you show my view versus yours, they look exactly the same down to the pixel. So there are already a lot of reasons to think we live in a very weird time. We're on this planet where there's no other sign of life anywhere in the galaxy. We believe that we could fill up the galaxy with life. That alone would make us among the earliest life that has ever existed in the galaxy–– a tiny fraction of it. So that's a lot of what the series is about. I'll answer this question explicitly. You ask, “Is this series about whether transformative AI will come and make this century weird?” or is it about “This century could be weird, and therefore transformative AI will come?” The central claim is that transformative AI could be developed in this century and the sections about ‘how weird a time we live in' are just a response to an objection. It's a response to a point of skepticism. It's a way of saying there are already a lot of reasons to think we live in a very weird time. So actually, this thing about AI is only a moderate quantitative update, not a complete revolution in the way you're thinking about things. Dwarkesh Patel There's a famous comedian who has a bit where he's imagining what it must have been like to live in 10BC. Let's say somebody comes with the proof that current deep learning techniques are not scalable for some reason and that transformative AI is very unlikely this century. I don't know if this is a hypothetical where that would happen, but let's just say that it is. Even if this is a weird time in terms of economic growth, does that have any implications other than transformative AI? Holden Karnofsky I encourage people to go to my series because I have a bunch of charts illustrating this and it could be a little bit hard to do concisely now. But having learned about just how strange the time we live in is when you look at it in context, I think the biggest thing I take away from this is how we should really look for the next big thing. If you'd been living 300 years ago and you'd been talking about the best way to help people, a lot of people might have been talking about various forms of helping low-income people. They probably would have been talking about spreading various religious beliefs. It would have seemed crazy to think that what you should be thinking about, for example, was the steam engine and how that might change the world, but I think the Industrial Revolution was actually an enormous deal and was probably the right thing to be thinking about if there's any way to be thinking about it how that would change the world and what one might do to make that a world that could be better.So that's basically where I'm at. I just think that as a world, as a global civilization, we should place a really high priority on saying that we live in a weird time. Growth has been exploding, accelerating over the last blink of an eye. We really need to be nervous and vigilant about what comes next and think about all the things that could radically transform the world. We should make a list of all the things that might radically transform the world, make sure we've done everything we can to think about them and identify the ways we might be able to do something today that would actually help. Maybe after we're done doing all that, we can have a lot of the world's brightest minds doing their best to think of stuff and when can't think of any more, then we can go back to all the other things that we worry aboutRight now the world invests so little in that kind of speculative, “Hey, what's the next big thing?” Even if it's not super productive to do so, even if there's not that much to learn, I feel the world should be investing more in that because the stakes are extremely high. I think it's a reasonable guess that we're living in a world that's recently been incredibly transformed by the Industrial Revolution and the future could be incredibly transformed by the next thing. I just don't think this gets a lot of discussion in basically any circles. If it got some, I would feel a lot more comfortable. I don't think the whole world should just obsess over what the next transformative event is, but I think right now there's so little attention to it. The Industrial Revolution Dwarkesh Patel I'm glad you brought up the Industrial Revolution because I feel like there are two implicit claims within the most important century thesis that don't seem perfectly compatible. One is that we live in an extremely wild time and that the transition here is potentially wilder than any other transition there has been before. The second is we have some sense of what we can be doing to make sure this transition goes well. Do you think that somebody at the beginning of the Industrial Revolution, knowing what they knew then, could have done something significant to make sure that it went as favorably as possible? Or do you think that that's a bad analogy for some reason? Holden Karnofsky It's a pretty good analogy for being thought-provoking and for thinking, “Gosh, if you had seen the Industrial Revolution coming in advance and this is when economic growth really reached a new level back in the 1700s and 1800s, what could you have done?” I think part of the answer is it's not that clear and I think that is a bit of an argument we shouldn't get too carried away today by thinking that we know exactly what we can do. But I don't think the answer is quite nothing. I have a goofy cold-taste post that I never published and may never publish because I lost track of it. What it basically says is “What if you'd been in that time and you had known the Industrial Revolution was coming or you had thought it might be?” You would ask yourself what you could be doing. One answer you might have given is you might have said, “Well, gosh, if this happens, whatever country it happens in might be disproportionately influential. What would be great is if I could help transform the thinking and the culture in that country to have a better handle on human rights and more value on human rights and individual liberties and a lot of other stuff–– and gosh, it kind of looks like people were doing that and it looks like it worked out.” So this is the Enlightenment.I even give this goofy example–– I could look it up and it's all kind of a trollish post. But the example is someone's thinking, “Hey, I'm thinking about this esoteric question about what a government owes to its citizens” or, “When does a citizen have a right to overthrow a government or when is it acceptable to enforce certain beliefs and not?” Then the other person in the dialogue is just like, “This is the weirdest, most esoteric question. Why does this matter? Why aren't you helping poor people?” But these are the questions that the Enlightenment thinkers were thinking about. I think there is a good case that they came up with a lot of stuff that really shaped the whole world since then because of the fact that the UK became so influential and really laid the groundwork for a lot of stuff about the rights of the governed, free speech, individual rights, and human rights. Then I go to the next analogy. It's like we're sitting here today and someone is saying, “Well, instead of working on global poverty, I'm studying this esoteric question about how you get an AI system to do what you want it to do instead of doing its own thing. I think it's not completely crazy to see them as analogous.” Now, I don't think this is what the Enlightenment thinkers are actually doing. I don't think they were saying this could be the most important millennium, but it is interesting that it doesn't look like there was nothing to be had there. It doesn't look like there's nothing you could have come up with. In many ways, it looks like what the Enlightenment thinkers were up to had the same esoteric, strange, overly cerebral feel at the time and ended up mattering a huge amount. So it doesn't feel like there's zero precedent either.Dwarkesh Patel Maybe I'm a bit more pessimistic about that because I think the people who are working on individual rights frameworks weren't anticipating an industrial revolution. I feel like the type of person who'd actually anticipate the industrial revolution would have a political philosophy that was actually probably a negative given, you know… Karl Marx. If you saw something like this happening, I don't think it would be totally not obvious. Holden Karnofsky I mean, I think my basic position here is that I'm not sitting here highly confident. I'm not saying there's tons of precedent and we know exactly what to do. That's not what I believe. I believe we should be giving it a shot. I think we should be trying and I don't think we should be totally defeatist and say, “Well, it's so obvious that there's never anything you could have come up with throughout history and humans have been helpless to predict the future.” I don't think that is true. I think that's enough of an example to kind of illustrate that. I mean, gosh, you could make the same statement today and say, “Look, doing research on how to get AI systems to behave as intended is a perfectly fine thing to do at any period in time.” It's not like a bad thing to do. I think John Locke was doing his stuff because he felt it was a good thing to do at any period in time, but the thing is that if we are at this crucial period of time, it becomes an even better thing to do and it becomes magnified to the point where it could be more important than other things. Dwarkesh Patel The one reason I might be skeptical of this theory is that I could say, “Oh, gosh, if you look throughout history, people were often convinced that they were living in the most important time,” or at least an especially important time. If you go back, everybody could be right about living in the most important time. Should you just have a very low prior that anybody is right about this kind of thing? How do you respond to that kind of logic?Holden Karnofsky First of all, I don't know if it's really true that it's that common for people to say that they're living in the most important time in history. This would be an interesting thing to look at. But just from stuff I've read about past works on political philosophy and stuff, I don't exactly see this claim all over the place. It definitely happens. It's definitely happened. I think a way of thinking about it is that there are two reasons you might think you are especially important. One is that you actually are and you've made reasonable observations about it. Another is that you want to be or you want to think you are so you're self-deceiving. So over the long sweep of history, a lot of people will come to this conclusion for the second reason. Most of the people who think they're the most important will be wrong. So that's all true. That certainly could apply to me and it certainly could apply to others. But I think that's just completely fine and completely true. I think we should have some skepticism when we find ourselves making these kinds of observations. At the same time, I think it would be a really bad rule or a really bad norm that every time you find yourself thinking the stakes are really high or that you're in a really important position, you just decide to ignore the thought. I think that would be very bad.If you imagine a universe where there actually are some people who live in an especially important time, and there are a bunch of other people who tell stories to themselves about whether they do, how would you want all those people to behave? To me, the worst possible rule is that all those people should just be like, “No, this is crazy, and forget about it.” I think that's the worst possible rule because the people who are living at the important time will then do the wrong thing. I think another bad rule would be that everyone should take themselves completely seriously and just promote their own interests ahead of everyone else's. A rule I would propose over either of them is that all these people should take their beliefs reasonably seriously and try to do the best thing according to their beliefs, but should also adhere to common sense standards of ethical conduct and not do too much “ends justify the means' reasoning.” It's totally good and fine to do research on alignment, but people shouldn't be telling lies or breaking the law in order to further their ends. That would be my proposed rules. When we have these high stake, crazy thoughts, we should do what we can about them and not go so crazy about them that we break all the rules of society. That seems like a better rule. That's the rule I'm trying to follow. Dwarkesh Patel Can you talk more about that? If for some reason, we can be convinced that the expected value calculation was immense, and you had to break some law in order to increase the odds that the AI goes well, I don't know how hypothetical this would be. Is it just that you're not sure whether you would be right and so you'd just want to err on the side of caution? Holden Karnofsky Yeah, I'm really not a fan of ends justify the means' reasoning. The thing that looks really, really bad is people saying it's worth doing horrible things and coercing each other and using force to accomplish these things because the ends we're trying to get to are more important than everything else. I'm against that stuff. I think that stuff looks a lot worse historically than people trying to break the future and do helpful things. So I see my main role in the world as trying to break the future and do helpful things. I can do that without doing a bunch of harmful, common sense, unethical stuff. Maybe someday there will be one of these intense tradeoffs. I haven't really felt like I've run into them yet. If I ever ran into one of those intense tradeoffs, I'd have to ask myself how confident I really am. The current level of information and confidence I have is, in my opinion, not enough to really justify the means. Dwarkesh Patel Okay, so let's talk about the potential implausibility of continued high growth. One thing somebody might think is, “OK, maybe 2 percent growth can't keep going on forever, but maybe the growth slows down to point five percent a year.” As you know, small differences in growth rates have big effects on the end result. So by the point that we've exhausted all the possible growth in the galaxy, we'll probably be able to expand to other galaxies. What's wrong with that kind of logic where there's point five percent growth that still doesn't imply a lock-in or would it be weird if that implied a lock-in?Holden Karnofsky I think we might want to give a little bit more context here. One of the key arguments of the most important century is that it's just part of one of the arguments that we live in a strange time. I'm also arguing that the current level of economic growth just looks too high to go on for another 10,000 years or so. One of the points I make, which is a point I got from Robin Hanson, is that if you just take the current level of economic growth and extrapolate it out 10,000 years, you end up having to conclude that we would need multiple stuff that is worth as much as the whole world economy is today–– multiple times that per atom in the galaxy. If you believe we can't break the speed of light, then we can't get further than that. We can't get outside the galaxy. So in some sense, we run out of material. So you're saying, “Alright but what if the growth rate falls to 0.5 percent?” Then I'm kind of like, “OK, well, so the growth rate now I ballparked it in the post is around 2 percent. That's the growth rate generally in the most developed countries. Let's say it falls to 0.5 percent.” Just like for how long? Did you calculate how long it would take to get to the same place? Dwarkesh Patel I think it was like 25,000 years. 0.5 percent gets you like one world-size economy. It's 10,000 versus 25,000, but 25,000 is the number of light years between us and like the next galaxy. Holden Karnofsky That doesn't sound right. I don't think this galaxy calculation is very close. There's also going to be a bunch of dead space. As you get to the outer reach of the galaxy, there's not going to be as much there. That doesn't sound super right, but let's just roll with it. I mean, sure, let's just say that you had 2 percent today and then growth went down to 0.5 percent and stayed there forever. I'm pretty sure that's still too big. I'm pretty sure you're still going to hit limits in some reasonable period of time, but that would still be weird on its own. It would just be like, “Well, we lived in the 200-year period when we had 2 percent growth and then we had 0.5 percent growth forever.” That would still make this a kind of an interesting time. It would be the most dynamic, fastest-changing time in all of human history. Not by a ton, but it's also like you pick the number that's the closest and the most perfectly optimized here. So if it went down to 0.1 percent or even down to 0.01 percent, then it would take longer to run out of stuff. But it would be even stranger with the 2 percent versus the 0.01 percent. So I don't really think there's any way out of “Gosh, this looks like this looks like it's probably going to end up looking like a very special time or a very weird time.”Dwarkesh Patel This is not worth getting hung up on, but from that perspective, then the century where we had 8 percent growth because of the Industrial Revolution–– would you say that maybe that's the most important century?Holden Karnofsky Oh, sure. Yeah, totally. No, the thing about rapid growth is it's not supposed to be on its own. By growth standards, this century looks less special than the last one or two. It's saying that the century is one of a handful or I think when I say “One of 80 of the most significant centuries,” or something by economic growth standards. That's only one argument, but then I look at a lot of other ways in which this century looks unusual. To say that something is the most important century of all time sounds totally nuts because there are so many centuries in the history of humanity, especially if you want to think about it on galactic time scales. Even once you narrow it down to 80, it's just much way less weird. If I've already convinced you using kind of non-controversial reasoning that we're one of the 80 most important centuries, it shouldn't take me nearly as much further evidence to say, actually, this one might be number one out of 80 because you're starting odds are more than 1 percent. So to get you up to 10 percent or 20 percent or 30 percent doesn't necessarily require a massive update the way that it would if we're just starting from nowhere. Dwarkesh Patel I guess I'm still not convinced that just because this is a weird century, this has any implications for why or whether we should see transformative AI this century. If we have a model about when transformative AI happens, is one of the variables that goes into that “What is the growth rate in 2080?” It just feels weird to have this as a parameter for when the specific technological development is going to happen. Holden Karnofsky It's just one argument in the series. I think the way that I would come at it is I would just say, “Hey, look at AI systems. Look at what they're doing. Look at how fast the rate of progress is. Look at these five different angles on imagining when I might be able to do all the things humans do to advance science and technology.” Just imagine that we get there this century. Wouldn't it be crazy to have AI that could do all the things humans do to advance science and technology? Wouldn't that lead to just a lot of crazy stuff happening? There's only ever been one species in the history of the universe that we know of that can do the kinds of things humans do. Wouldn't it be weird if there were two? That would be crazy. One of them was a new one we built that could be copied at will, and run at different speeds on any hardware you have. That would be crazy. Then you might come back and say, “Yeah, that would be crazy. This is too crazy. Like I'm ruling this out because this is too crazy.” Then I would say, “OK, well, we have a bunch of evidence that we live in an unusual, crazy time.” And you actually should think that there's a lot of signs that this century is not just a random century picked from a sample of millions of centuries. So that's the basic structure of the argument. As far as the growth rate in zero AD, I think it matters. I think you're asking the question, why do the dynamics of growth in zero AD matter at all for this argument? I think it's because it's just a question of, “How does economic growth work generally and what is the trend that we're on, and what happens if that trend continues?” If around zero AD growth was very low but accelerating, and if that was also true at one hundred AD and a thousand AD and negative a thousand or, you know, a thousand BC, then it starts to point to a general pattern. Growth is accelerating and maybe accelerating for a particular reason, and therefore you might expect more acceleration.AI Success ScenarioDwarkesh Patel Alright, let's talk about transformative AI then. Can you describe what success looks like concretely? Are humans part of the post-transformative AI world? Are we hoping that these AIs become enslaved gods that help us create a utopia? What does the concrete success scenario look like? Holden Karnofsky I mean, I think we've talked a lot about the difficulty of predicting the future, and I think I do want to emphasize that I really do believe in that. My attitude to the most important century is not at all, “Hey, I know exactly what's going to happen and I'm making a plan to get us through it.” It's much more like there's a general fuzzy outline of a big thing that might be approaching us. There are maybe two or three things we can come up with that seem good to do. Everything else we think about, we're not going to know if it's good to do or bad to do. So I'm just trying to find the things that are good to do so that I can make things go a little bit better or help things go a little bit better. That is my general attitude. It's like if you were on a ship in a storm and you saw some very large, fuzzy object obscured by the clouds, you might want to steer away from it. You might not want to say, “Well, I think that is an island and I think there's probably a tiger on it. So if we go and train the tiger in the right way, blah, blah, blah, blah, blah,” you don't want to get into that. Right? So that is the general attitude I'm taking.What does success look like to me? Success could look like a lot of things, but one thing success would look like to me would frankly just be that we get something not too different from the trajectory we're already on. So, in other words, if we can have systems that behaved as intended, acted as tools and amplifiers of humans, and did the things they're supposed to do. If we could avoid a world where those systems got sort of all controlled by one government or one person, we could avoid a world where that caused a huge concentration of power. If we could have a world where AI systems are just another technology that helps us do a lot of stuff, and we'd invent lots of other technologies and everything is relatively broadly distributed and everything works roughly as it's supposed to work, then you might be in a world where we continue the trend we've seen over the last couple of hundred years, which is that we're all getting richer. We're all getting more tools. We all hopefully get an increasing ability to understand ourselves, study ourselves, and understand what makes us happy, what makes us thrive. Hopefully, the world just gets better over time and we have more and more new ideas that thus hopefully make us wiser. I do think that in most respects, the world of today is a heck of a lot better than the world of 200 years ago. I don't think the only reason for that is wealth and technology, but I think they played a role. I think that if you'd gone back to 200 years ago and said, “Holden, how would you like the world to develop a bunch of new technologies as long as they're sort of evenly distributed and they behave roughly as intended and people mostly just get richer and discover new stuff?” I'd be like, “That sounds great!” I don't know exactly where we're going to land. I can't predict in advance whether we're going to decide that we want to treat our technologies as having their own rights. That's stuff that the world will figure out. But I'd like to avoid massive disasters that are identifiable because I think if we can, we might end up in a world where the future is wiser than we are and is able to do better things. Dwarkesh Patel The way you put it, AI enabling humans doesn't sound like something that could last for thousands of years. It almost sounds as weird as chimps saying “What we would like is humans to be our tools.” At best, maybe they could hope we would give them nice zoos. What is the role of humans in this in this future? Holden Karnofsky A world I could easily imagine, although that doesn't mean it's realistic at all, is a world where we build these AI systems. They do what they're supposed to do, and we use them to gain more intelligence and wisdom. I've talked a little bit about this hypothetical idea of digital people–– maybe we develop something like that. Then, after 100 years of this, we've been around and people have been having discussions in the public sphere, and people kind of start to talk about whether the AIs themselves do have rights of their own and should be sharing the world with us. Maybe then they do get rights. Maybe some AI systems end up voting or maybe we decide they shouldn't and they don't. Either way, you have this kind of world where there's a bunch of different beings that all have rights and interests that matter. They vote on how to set up the world so that we can all hopefully thrive and have a good time. We have less and less material scarcity. Fewer and fewer tradeoffs need to be made. That would be great. I don't know exactly where it ends or what it looks like. But I don't know. Does anything strike you as unimaginable about that? Dwarkesh Patel Yeah, the fact that you can have beings that can be copied at will, but also there's some method of voting..Holden Karnofsky Oh, yeah. That's a problem that would have to be solved. I mean, we have a lot of attention paid to how the voting system works, who gets to vote, and how we avoid things being unfair. I mean, it's definitely true that if we decided there was some kind of digital entity and it had the right to vote and that digital entity was able to copy itself–– you could definitely wreak some havoc right there. So you'd want to come up with some system that restricts how many copies you can make of yourself or restricts how many of those copies can vote. These are problems that I'm hoping can be handled in a way that, while not perfect, could be non-catastrophic by a society that hasn't been derailed by some huge concentration of power or misaligned systems. Dwarkesh Patel That sounds like that might take time. But let's say you didn't have time. Let's say you get a call and somebody says, “Holden, next month, my company is developing or deploying a model that might plausibly lead to AGI.” What does Open Philanthropy do? What do you do? Holden Karnofsky Well, I need to distinguish. You may not have time to avoid some of these catastrophes. A huge concentration of power or AI systems don't behave as intended and have their own goals. If you can prevent those catastrophes from happening, you might then get more time after you build the AIs to have these tools that help us invent new technologies and help us perhaps figure things out better and ask better questions. You could have a lot of time or you could figure out a lot in a little time if you had those things. But if someone said–– wait how long did you give me?Dwarkesh Patel A month. Let's say three months. So it's a little bit more. Holden Karnofsky Yeah, I would find that extremely scary. I kind of feel like that's one of the worlds in which I might not even be able to offer an enormous amount. My job is in philanthropy (and a lot of what philanthropists do historically or have done well historically), is we help fields grow. We help do things that operate on very long timescales. So an example of something Open Philanthropy does a lot of right now is we fund people who do research on alignment and we fund people who are thinking about what it would look like to get through the most important century successfully. A lot of these people right now are very early in their careers and just figuring stuff out. So a lot of the world I picture is like 10 years from now, 20 years from now, or 50 years from now. There's this whole field of expertise that got support when traditional institutions wouldn't support it. That was because of us. Then you come to me and you say, “We've got one week left. What do we do?” I'd be like, “I don't know. We did what we could do. We can't go back in time and try to prepare for this better.” So that would be an answer. I could say more specific things about what I'd say in the one to three-month time frame, but a lot of it would be flailing around and freaking out, frankly. Dwarkesh Patel Gotcha. Okay. Maybe we can reverse the question. Let's say you found out that AI actually is going to take much longer than you thought, and you have more than five decades. What changes? What are you able to do that you might not otherwise be able to do? Holden Karnofsky I think the further things are, the more I think it's valid to say that humans have trouble making predictions on long time frames. The more I'd be interested in focusing on other causes of very broad things we do, such as trying to grow the set of people who think about issues like this, rather than trying to specifically study how to get AI systems like today's to behave as intended. So I think that's a general shift, but I would say that I tend to feel a bit more optimistic on longer time frames because I do think that the world just isn't ready for this and isn't thinking seriously about this. A lot of what we're trying to do at Open Philanthropy is create support that doesn't exist in traditional institutions for people to think about these topics. That includes doing AI alignment research. That also includes thinking through what we want politically, and what regulations we might want to prevent disaster. I think those are a lot of things. It's kind of a spectrum. I would say, if it's in three months, I would probably be trying to hammer out a reasonable test of whether we can demonstrate that the AI system is either safe or dangerous.If we can demonstrate it's dangerous, use that demonstration to really advocate for a broad slowing of AI research to buy more time to figure out how to make it less dangerous. I don't know that I feel that much optimism. If this kind of AI is 500 years off, then I'm kind of inclined to just ignore it and just try and make the world better and more robust, and wiser. But I think if we've got 10 years, 20 years, 50 years, 80 years, something in that range, I think that is kind of the place where supporting early careers and supporting people who are going to spend their lives thinking about this would be beneficial. Then we flash forward to this crucial time and there are a lot more people who spent their lives thinking about it. I think that would be a big deal. Dwarkesh Patel Let's talk about the question of whether we can expect the AI to be smart enough to disempower humanity, but dumb enough to have that kind of goal. When I look out at smart people in the world, it seems like a lot of them have very complex, nuanced goals that they've thought a lot about what is good and how to do good. Holden Karnofsky A lot of them don't. Dwarkesh Patel Does that overall make you more optimistic about AIs? Holden Karnofsky I am not that comforted by that. This is a very, very old debate in the world of AI alignment. Eliezer Yudkowsky has something called the orthogonality thesis. I don't remember exactly what it says, but it's something like “You could be very intelligent about any goal. You could have the stupidest goal and be very intelligent about how to get it.” In many ways, a lot of human goals are pretty silly. A lot of the things that make me happy are not things that are profound or wonderful. They're just things that happen to make me happy. You could very intelligently try to get those things, but it doesn't give me a lot of comfort. I think basically my picture of how modern AI works is that you're basically training these systems by trial and error. You're basically taking an AI system, and you're encouraging some behaviors, while discouraging other behaviors. So you might end up with a system that's being encouraged to pursue something that you didn't mean to encourage. It does it very intelligently. I don't see any contradiction there. I think that if you were to design an AI system and you were kind of giving it encouragement every time it was getting more money into your bank account, you might get something that's very, very good at getting money into your bank account to the point where you'd going to disrupt the whole world to do that. You will not automatically get something that thinks, “Gosh, is this a good thing to do?” I think with a lot of human goals, there's not really a right answer about whether our goals actually make sense. They're just the goals we have.Dwarkesh Patel You've written elsewhere about how moral progress is something that's real, that's historically happened, and it corresponds to what actually counts as moral progress. Do you think there's a reason to think the same thing might happen with AI? Whatever the process is that creates moral progress?Holden Karnofsky I kind of don't in particular. I've used the term moral progress as just a term to refer to changes in morality that are good. I think there has been moral progress, but I don't think that means moral progress is something inevitable or something that happens every time you are intelligent. An example I use a lot is attitudes toward homosexuality. It's a lot more accepted today than it used to be. I call that moral progress because I think it's good. Some people will say, “Well, you know, I don't believe that morality is objectively good or bad. I don't believe there is any such thing as moral progress. I just think things change randomly.” That will often be an example I'll pull out and I'll say, “But do you think that was a neutral change?” I just think it was good. I think it was good, but that's not because I believe there's some underlying objective reality. It's just my way of tagging or using language to talk about moral changes that seem like they were positive to me. I don't particularly expect that an AI system would have the same evolution that I've had in reflecting on morality or would come to the same conclusions I've come to or would come up with moralities that seem good to me. I don't have any reason to think any of that. I do think that historically there have been some cases of moral progress. Dwarkesh Patel What do you think is the explanation for historical progress? Holden Karnofsky One thing that I would say is that humans have a lot in common with each other. I think some of history contains cases of humans learning more about the world, learning more about themselves, and debating each other. I think a lot of moral progress has just come from humans getting to know other humans who they previously were stereotyping and judging negatively and afraid of. So I think there's some way in which humans learning about the world and learning about themselves leads them to have kind of conclusions that are more reflective and more intelligent for their own goals. But, if you brought in something into the picture that was not a human at all, it might be very intelligent and reflective about its goals, but those goals might have zero value from our point of view. Dwarkesh Patel Recent developments in AI have made many people think that AI could happen much sooner than they otherwise thought. Has the release of these new models impacted your timelines? Holden Karnofsky Yeah, I definitely think that recent developments in AI have made me a bit more freaked out. Ever since I wrote The Most Important Century series and before that, there were years when Open Philanthropy was very interested in AI risk, but it's become more so as we've seen progress in AI. I think what we're seeing is we're seeing these very generic, simple systems that are able to do a lot of different tasks. I think people are interested in this. There are a lot of compilations of what GPT-3 is–– a very simple language model that, by the way, my wife and brother-in-law both worked on. This very simple language model just predicts the next word it's going to see in a stream of text. People have gotten it to tell stories. People got similar (though not identical) models to analyze and explain jokes. People have gotten it to play role-playing games, write poetry, write lyrics, answer multiple-choice questions, and answer trivia questions. One of the results that I found most ridiculous, strange and weird was this thing called Minerva, where people took one of these language models and with very little special intervention, they got it to do these difficult math problems and explain its reasoning and get them right about half the time. It wasn't really trained in a way that was very specialized for these math problems, so we just see AI systems having all these unpredictable human-like abilities just from having this very simple training procedure. That is something I find kind of wild and kind of scary. I don't know exactly where it's going or how fast. Dwarkesh Patel So if you think transformative AI might happen this century, what implications does that have for the traditional global health and well-being stuff that OpenPhilanthropy does? Will that have persistent effects of AI if it gets aligned? Will it create a utopia for us anyway? Holden Karnofsky I don't know about utopia. My general take is that anything could happen. I think my general take on this most important century stuff, and the reason it's so important is because it's easy to imagine a world that is really awesome and is free from scarcity and we see more of the progress we've seen over the last 200 years and we end up in a really great place. It's also easy to imagine a horrible dystopia. But my take is that the more likely you think all this is, the more likely you think transformative AI is, the more you should think that that should be the top priority, that we should be trying to make that go well instead of trying to solve more direct problems that are more short term. I'm not an extremist on this. So, OpenPhilanthropy does both.OpenPhilanthropy works on speculative far-off future risks and OpenPhil also does a bunch of more direct work. Again, we do direct and recommend a lot of money to give to those top charities, which do things like distributing bed nets in Africa to help prevent malaria and treat children for intestinal parasites. OpenPhilanthropy does a lot of advocacy for more money going to foreign aid or for better land use policies to have a stronger economy. We do a bunch of scientific research work that is more aimed at direct medical applications, especially in poor countries. So I support all that stuff. I'm glad we're doing it. It's just a matter of how real and how imminent do you think this transformative AI stuff is? The more real and more imminent, the more of our resources should go into it. Dwarkesh Patel Yeah, that makes sense to me. I'm curious, whatever work you do elsewhere, do those still have persistent effects after transformative AI comes? Or do you think they'll basically wash out in comparison to the really big stuff? Holden Karnofsky I mean, I think in some sense, the effects are permanent in the sense that if you cause someone to live a healthier, better life, that's a significant thing that happened. Nothing will ever erase that life or make that life unimportant, but I think in terms of the effects on the future, I do expect it mostly to wash out. I expect that mostly whatever we do to make the world better in that way will not persist in any kind of systematic, predictable manner past these crazy changes. I think that's probably how things look pre and post-industrial revolution. There are probably some exceptions, but that's my guess.Competition, Innovation , & AGI BottlenecksDwarkesh Patel You've expressed skepticism towards the competition frame around AI or you try to make capabilities go faster for the countries or companies you favor most. But elsewhere, you've used the “innovation as mining metaphor,” and maybe you can explain that when you're giving the answer. It seems like this frame should imply that the second most powerful AI company is probably right on the heels of the first most powerful. So if you think the first most powerful is going to take safety more seriously, you should try to boost them. How do you think about how these two different frames interact? Holden Karnofsky I think it's common for people who become convinced that AI could be really important to just jump straight to, “Well, I want to make sure that people I trust build it first.” That could mean my country, that could mean my friends, people I'm investing in. I have generally called that the competition frame which is “I want to win a competition to develop AI”, and I've contrasted it with a frame that I also think is important, called the caution frame, which is that we need to all work together to be careful to not build something that spins out of control and has all these properties and behaves in all these ways we didn't intend. I do think that if we do develop these very powerful AI systems, we're likely to end up in a world where there are multiple players trying to develop it and they're all hot on each other's heels. I am very interested in ways for us all to work together to avoid disaster as we're doing that. I am maybe less excited than the average person who first learns about this is and is like “I'm picking the one I like best and helping them race ahead.”Dwarkesh Patel Although I am someone interested in both, if you take the innovation as mining metaphor seriously, doesn't that imply that actually the competition is really a big factor here?Holden Karnofsky The innovation mining metaphor is from another bit of Cold Takes. It's an argument I make that you should think of ideas as being somewhat like natural resources in the sense of once someone discovers a scientific hypothesis or once someone writes a certain great symphony, that's something that can only be done once. That's an innovation that can only be done once. So it gets harder and harder over time to have revolutionary ideas because the most revolutionary, easiest-to-find ideas have already been found. So there's an analogy to mining. I don't think it applies super importantly to the AI thing because all I'm saying is that success by person one makes success by person two harder. I'm not saying that it has no impact or that it doesn't speed things up. Just to use a literal mining metaphor, let's say there's a bunch of gold in the ground. It is true that if you rush and go get all that gold, it'll be harder for me to now come in and find a bunch of gold. That is true. What's not true is that it doesn't matter if you do it. I mean, you might do it a lot faster than me. You might do it a lot ahead of me. Dwarkesh Patel Fair enough. Maybe one piece of skepticism that somebody could have about transformative AI is that all this is going to be bottlenecked by the non-automatable steps in the innovation sequence. So there won't be these feedback loops that speed up. What is your reaction? Holden Karnofsky I think the single best criticism and my biggest point of skepticism on this most important century stuff is the idea that you could build an AI system that's very impressive and could do pretty much everything humans can do. There might be one step that you still have to have humans do, and that could bottleneck everything. Then you could have the world not speed up that much and science and technology not advance that fast because they are doing almost everything. But humans are still slowing down this one step or the real world is slowing down one step. Let's say real-world experiments to invent new technologies take how long they take. I think this is the best objection to this whole thing and the one that I'd most like to look into more. I do ultimately think that there's enough reason to think that if you had AI systems that had human-like reasoning and analysis capabilities, you shouldn't count on this kind of bottleneck causing everything to go really slow. I write about that in this piece called Weak Point in the Most Important Century: Full Automation. Part of this is how you don't need to automate the entire economy to get this crazy growth loop. You can automate just a part of it that specifically has to do with very important tech like energy and AI itself. Those actually seem, in many ways, less bottlenecked than a lot of other parts of the economy. So you could be developing better AI algorithms and AI chips, manufacturing them, mostly using robots, and using those to come up with even better designs. Then you could also be designing more and more efficient solar panels, and using those to collect more and more energy to power your eyes. So a lot of the crucial pieces here just actually don't seem all that likely to be bottlenecked. You can be at the point wher

As the news that thirty year-old cryptocurrency baron, Sam Bankman-Fried‘s, FTX empire suddenly collapsed, the residual effects reverberated in the spheres of business, politics and philanthropy. Bankman-Fried was one of the largest donors to and a huge proponent of effective altruism, a social and philosophical movement started by academics Peter Singer, Toby Ord, and William … Continue reading Alexander Zaitchik on Effective Altruism + Longtermism → This article and podcast Alexander Zaitchik on Effective Altruism + Longtermism appeared first on Sea Change Radio.

General Visit Brett's website, where you can find his blog and much more: https://www.bretthall.org/ Follow Brett on Twitter: https://twitter.com/Tokteacher Subscribe to Brett's YouTube channel: https://youtube.com/channel/UCmP5H2rF-ER33a58ZD5jCig?sub_confirmation=1 References Iona's Substack essay, in which she previously described Brett as a philosopher—a description with which Brett disagreed: https://drionaitalia.substack.com/p/knots-gather-at-the-comb Karl Popper's philosophy: https://plato.stanford.edu/entries/popper/ Massimo Pigliucci's Two for Tea appearance: https://m.soundcloud.com/twoforteapodcast/55-massimo-pigliucci David Deutsch's ‘The Beginning of Infinity': https://www.amazon.com/gp/aw/d/0143121359/ref=tmm_pap_swatch_0?ie=UTF8&qid=1658005291&sr=8-1 Daniel James Sharp's Areo review of Ord's ‘The Precipice': https://areomagazine.com/2020/05/11/we-contain-multitudes-a-review-of-the-precipice-existential-risk-and-the-future-of-humanity-by-toby-ord/ David Hume and the problem of induction: https://plato.stanford.edu/entries/induction-problem/ Natural selection and the Neo-Darwinian synthesis: https://www.britannica.com/science/neo-Darwinism Richard Dawkins's ‘The Extended Selfish Gene': https://www.amazon.com/gp/aw/d/B01MYDYR6N/ref=tmm_pap_swatch_0?ie=UTF8&qid=1658008393&sr=8-3 Theory-ladenness: https://en.m.wikipedia.org/wiki/Theory-ladenness Ursula K. Le Guin's ‘The Left Hand of Darkness': https://www.amazon.com/gp/aw/d/1473221625/ref=tmm_pap_swatch_0?ie=UTF8&qid=1658010065&sr=8-1 The Popperian ‘paradox of tolerance' cartoon: https://images.app.goo.gl/MEbujAKv2VSp1m4B8 For the Steven Pinker Two for Tea interview on ‘Rationality', stay tuned to the Two for Tea podcast feed as it's coming soon for public listening: https://m.soundcloud.com/twoforteapodcast Brett's critique of Bayesianism: https://www.bretthall.org/bayesian-epistemology.html Brett on morality: https://www.bretthall.org/morality Steven Pinker's book ‘Rationality': https://www.amazon.com/gp/aw/d/0525561994/ref=tmm_hrd_swatch_0?ie=UTF8&qid=1658012700&sr=8-1 Timestamps 00:00 Opening and introduction. What, exactly, is Brett? What does he do? 4:58 Free speech and Popperian thought (and what is Popperian thought, anyway?). 12:24 Brett's view on existential risk and the future; how he differs from the likes of Martin Rees and Toby Ord. 22:38 How can we overcome ‘acts of God'? (With reference to Iona's syphilitic friend.) The dangers of the unknown and the necessity of progress. 26:50 The unpredictability of the nature of problems, with reference to fear of nuclear war and nuclear energy. The nature and history of problem solving, particularly as regards energy. 37:02 The Popperian/Deutschian theory of knowledge—guesswork, creativity, and the reduction of error. 46:50 William Paley's watch, Darwinism, selfish genes, and the embedding of knowledge into reality. 54:15 On theory-ladenness, the necessity of error correction, the power of science, and the impossibility of a final theory—all is approximation and continual improvement. 1:01:10 The nature of good explanations, with reference to the invocation of gods vs scientific accounts and the nature of the atom. 1:07:24 How the principle of the difficulty of variability is important in art as well as science, with reference to Ursula K. Le Guin's ‘The Left Hand of Darkness.' ‘Aha' vs ‘what the fuck?' surprise. 1:15:30 The nature of critical thinking and Brett on education: the misconceptions inherent in the current fashion for teaching critical thinking. 1:26:10 A question for Brett from Twitter: what did Popper really think about tolerance and intolerance (see the famous cartoon on the paradox of tolerance)? 1:36:24 Is there anything else Brett would like to add?

Paris Marx is joined by Émile Torres to discuss the ongoing effort to sell effective altruism and longtermism to the public, and why they're philosophies that won't solve the real problems we face.Émile Torres is a PhD candidate at Leibniz University Hannover and the author of the forthcoming book Human Extinction: A History of the Science and Ethics of Annihilation. Follow Émile on Twitter at @xriskology.Tech Won't Save Us offers a critical perspective on tech, its worldview, and wider society with the goal of inspiring people to demand better tech and a better world. Follow the podcast (@techwontsaveus) and host Paris Marx (@parismarx) on Twitter, support the show on Patreon, and sign up for the weekly newsletter.The podcast is produced by Eric Wickham and part of the Harbinger Media Network.Also mentioned in this episode:Émile recently wrote about the ongoing effort to sell longtermism and effective altruism to the public.Peter Singer wrote an article published in 1972 arguing that rich people need to give to charity, which went on to influence effective altruists.NYT recently opined on whether it's ethical for lawyers to defend climate villains.Nathan Robinson recently criticized effective altruism for Current Affairs.Support the show

Back on the podcast, today is sleep, nutrition, and metabolism expert, Greg Potter, PhD. Through his academic research, public speaking, consulting and writing, Greg empowers people to make simple and sustainable lifestyle changes that add years to their lives and life to their years. His work has been featured in dozens of international media outlets, including Reuters, TIME, and The Washington Post, and he regularly contributes to popular websites, blogs, and podcasts. In this podcast, Greg is talking about light, including the importance of getting out in the sun and also modern problems with artificial light. He discusses the impact of light on the circadian system along with up-to-date recommendations related to light hygiene. We discuss practical tips for reducing light at night (not all of which involve putting away your device), and why not getting the right kind of light might be keeping you from achieving your body composition goals. Here's the outline of this episode with Greg Potter:  [00:02:14] Wellics Corporate Wellness Software. [00:06:49] The importance of light. [00:08:30] The introduction of electric light. [00:09:55] myLuxRecorder (Satchin Panda's app, no longer available); Podcast: How to Use Time-Restricted Eating to Reverse Disease and Optimize Health, with Satchin Panda. [00:10:37] How light influences the circadian system. [00:15:34] Consensus paper with recommendations related to light hygiene; Study: Brown, Timothy M., et al. "Recommendations for daytime, evening, and nighttime indoor light exposure to best support physiology, sleep, and wakefulness in healthy adults." PLoS biology 20.3 (2022): e3001571. [00:19:13] Practical tips for reducing light at night. [00:22:44] Increasing prevalence of myopia. [00:23:46] Podcast: Getting Stronger, with Todd Becker. [00:26:01] Vitamin D synthesis; Podcast: The Pleiotropic Effects of Sunlight, with Megan Hall. [00:26:15] Effects of light on mood and cognition. [00:27:24] Effect of light exposure patterns on cognitive performance; Study: Grant, Leilah K., et al. "Daytime exposure to short wavelength-enriched light improves cognitive performance in sleep-restricted college-aged adults." Frontiers in neurology (2021): 197. [00:28:14] Effects of light on metabolic health. [00:28:20] Dan Pardi podcast featuring Peter Light: Sunlight And Fat Metabolism: A New Discovery. [00:28:52] Effect of bright and dim light on metabolism (Netherlands); Study: Harmsen, Jan-Frieder, et al. "The influence of bright and dim light on substrate metabolism, energy expenditure and thermoregulation in insulin-resistant individuals depends on time of day." Diabetologia 65.4 (2022): 721-732. [00:30:53] Effects of light on skin and immune function. [00:31:57] Highlights #15 (topics: Sun avoidance & exposure, increasing testosterone, Robert Sapolsky). [00:35:14] Skyglow. [00:36:48] Light at night and endocrine disruption. [00:37:45] Light at night and quality/duration of sleep. [00:38:19] Blue light in the evening interferes with sleep homeostasis; Study: Cajochen, Christian, et al. "Evidence that homeostatic sleep regulation depends on ambient lighting conditions during wakefulness." Clocks & Sleep 1.4 (2019): 517-531. [00:38:53] Effects of light at night on sympathetic nervous system/cortisol; Study: Rahman, Shadab A., et al. "Characterizing the temporal dynamics of melatonin and cortisol changes in response to nocturnal light exposure." Scientific reports 9.1 (2019): 1-12. [00:39:26] Effects of light at night on heart rate, HRV, insulin resistance; Study: Mason, Ivy C., et al. "Light exposure during sleep impairs cardiometabolic function." Proceedings of the National Academy of Sciences 119.12 (2022): e2113290119. [00:41:34] Effects of moon phases on sleep; Study: Casiraghi, Leandro, et al. "Moonstruck sleep: Synchronization of human sleep with the moon cycle under field conditions." Science advances 7.5 (2021): eabe0465. [00:45:40] Effects of individual sensitivity to light; Study: Phillips, Andrew JK, et al. "High sensitivity and interindividual variability in the response of the human circadian system to evening light." Proceedings of the National Academy of Sciences 116.24 (2019): 12019-12024. [00:47:55] Camping and melatonin synthesis across seasons; Study: Stothard, Ellen R., et al. "Circadian entrainment to the natural light-dark cycle across seasons and the weekend." Current Biology 27.4 (2017): 508-513. [00:48:40] Seasonal changes in thyroid hormones (meta-analysis): Kuzmenko, N. V., et al. "Seasonal variations in levels of human thyroid-stimulating hormone and thyroid hormones: a meta-analysis." Chronobiology International 38.3 (2021): 301-317. [00:53:24] Effect of location in the world; Podcast: Morning Larks and Night Owls: the Biology of Chronotypes, with Greg Potter, PhD. [00:54:30] Daylight Savings Time transition and  traffic accidents in the US; Study: Fritz, Josef, et al. "A chronobiological evaluation of the acute effects of daylight saving time on traffic accident risk." Current biology 30.4 (2020): 729-735. [00:56:08] Effects of Daylight Savings Time on cardiac events. [00:56:48] Daylight Savings Time and cyberloafing; Study: Wagner, David T., et al. "Lost sleep and cyberloafing: Evidence from the laboratory and a daylight saving time quasi-experiment." Journal of Applied psychology 97.5 (2012): 1068.  [00:57:26] Circadian clock disrupted by Daylight Savings Time; Study: Kantermann, Thomas, et al. "The human circadian clock's seasonal adjustment is disrupted by daylight saving time." Current Biology 17.22 (2007): 1996-2000.  [01:00:44] Implications of permanent daylight savings time. [01:03:37] Effects of light at night in animals; Study: Sanders, Dirk, et al. "A meta-analysis of biological impacts of artificial light at night." Nature Ecology & Evolution 5.1 (2021): 74-81. [01:09:14] Minimizing the impact of light at night on wildlife. [01:13:50] Human-centric lighting at hospitals; Study: Giménez, Marina C., et al. "Patient room lighting influences on sleep, appraisal and mood in hospitalized people." Journal of sleep research 26.2 (2017): 236-246. [01:14:51] Babies in a neonatal unit did better with light/dark cycle; Study: Vásquez-Ruiz, Samuel, et al. "A light/dark cycle in the NICU accelerates body weight gain and shortens time to discharge in preterm infants." Early human development 90.9 (2014): 535-540. [01:17:59] Effects of light at night on plants; Study: Ffrench-Constant, Richard H., et al. "Light pollution is associated with earlier tree budburst across the United Kingdom." Proceedings of the Royal Society B: Biological Sciences 283.1833 (2016): 20160813. [01:18:50] Maturation of soybeans shifted with  artificial light at night; Study: Palmer, Matthew, et al. Roadway lighting's impact on altering soybean growth. No. FHWA-ICT-17-010. 2017. [01:19:44] How to optimise your light environment. [01:19:54] Incandescent vs compact fluorescent bulbs. [01:21:58] LED lights. [01:25:33] Light-emitting devices with screens; metamerism. [01:26:20] Using metamerism to regulate impact of digital devices; Study: Allen, Annette E., et al. "Exploiting metamerism to regulate the impact of a visual display on alertness and melatonin suppression independent of visual appearance." Sleep 41.8 (2018): zsy100. [01:26:51] Software that reduces your exposure to short wavelengths: Nightshift (iPhone), Night Light/Blue Light Filter (Android), f.lux. [01:27:23] Apps to prevent short-wavelength light emissions do help; Study: Gringras, Paul, et al. "Bigger, brighter, bluer-better? Current light-emitting devices–adverse sleep properties and preventative strategies." Frontiers in public health 3 (2015): 233. [01:27:31] Blue-light blocking app did not improve sleep; Study: Smidt, Alec M., et al. "Effects of Automated Diurnal Variation in Electronic Screen Temperature on Sleep Quality in Young Adults: A Randomized Controlled Trial." Behavioral Sleep Medicine (2021): 1-17. [01:28:31] Blue-blockers. [01:31:31] Recommendations for shift workers. Greg's paper on this topic: Potter, Gregory DM, and Thomas R. Wood. "The future of shift work: Circadian biology meets personalised medicine and behavioural science." Frontiers in Nutrition 7 (2020): 116. [01:33:34] Jet lag: Jet Lag Rooster. [01:37:27] Find Greg on Instagram, TikTok; gregpotterphd.com [01:37:56] Book: When Brains Dream: Understanding the Science and Mystery of Our Dreaming Minds, by Antonio Zadra. [01:38:08] Book: The Beginning of Infinity: Explanations That Transform the World, by David Deutsch. [01:38:32] Book: The Precipice by Toby Ord.