Podcasts about cfar

Earlier this year a Border Patrol officer was killed in a shoot-out with people who have been described as members of a trans vegan AI death cult. But who are the Zizians, really? Robert sits down with David Gborie to trace their development, from part of the Bay Area Rationalist subculture to killers. (4 Part series) Sources: https://medium.com/@sefashapiro/a-community-warning-about-ziz-76c100180509 https://web.archive.org/web/20230201130318/https://sinceriously.fyi/rationalist-fleet/ https://knowyourmeme.com/memes/infohazard https://web.archive.org/web/20230201130316/https://sinceriously.fyi/net-negative/ Wayback Machine The Zizians Spectral Sight True Hero Contract Schelling Orders – Sinceriously Glossary – Sinceriously https://web.archive.org/web/20230201130330/https://sinceriously.fyi/my-journey-to-the-dark-side/ https://web.archive.org/web/20230201130302/https://sinceriously.fyi/glossary/#zentraidon https://web.archive.org/web/20230201130259/https://sinceriously.fyi/vampires-and-more-undeath/ https://web.archive.org/web/20230201130316/https://sinceriously.fyi/net-negative/ https://web.archive.org/web/20230201130318/https://sinceriously.fyi/rationalist-fleet/ https://x.com/orellanin?s=21&t=F-n6cTZFsKgvr1yQ7oHXRg https://zizians.info/ according to The Boston Globe Inside the ‘Zizians’: How a cultish crew of radical vegans became linked to killings across the United States | The Independent Silicon Valley ‘Rationalists’ Linked to 6 Deaths The Delirious, Violent, Impossible True Story of the Zizians | WIRED Good Group and Pasek’s Doom – Sinceriously Glossary – Sinceriously Mana – Sinceriously Effective Altruism’s Problems Go Beyond Sam Bankman-Fried - Bloomberg The Zizian Facts - Google Docs Several free CFAR summer programs on rationality and AI safety - LessWrong 2.0 viewer This guy thinks killing video game characters is immoral | Vox Inadequate Equilibria: Where and How Civilizations Get Stuck Eliezer Yudkowsky comments on On Terminal Goals and Virtue Ethics - LessWrong 2.0 viewer Effective Altruism’s Problems Go Beyond Sam Bankman-Fried - Bloomberg SquirrelInHell: Happiness Is a Chore PLUM OF DISCORD — I Became a Full-time Internet Pest and May Not... Roko Harassment of PlumOfDiscord Composited – Sinceriously Intersex Brains And Conceptual Warfare – Sinceriously Infohazardous Glossary – Sinceriously SquirrelInHell-Decision-Theory-and-Suicide.pdf - Google Drive The Matrix is a System – Sinceriously A community alert about Ziz. Police investigations, violence, and… | by SefaShapiro | Medium Intersex Brains And Conceptual Warfare – Sinceriously A community alert about Ziz. Police investigations, violence, and… | by SefaShapiro | Medium PLUM OF DISCORD (Posts tagged cw-abuse) Timeline: Violence surrounding the Zizians leading to Border Patrol agent shooting See omnystudio.com/listener for privacy information.

This is a link post. The AI safety community has grown rapidly since the ChatGPT wake-up call, but available funding doesn't seem to have kept pace. However, there's a more recent dynamic that's created even better funding opportunities, which I witnessed as a recommender in the most recent SFF grant round.[1] Most philanthropic (vs. government or industry) AI safety funding (>50%) comes from one source: Good Ventures. But they've recently stopped funding several categories of work (my own categories, not theirs): Many Republican-leaning think tanks, such as the Foundation for American Innovation. “Post-alignment” causes such as digital sentience or regulation of explosive growth. The rationality community, including LessWrong, Lightcone, SPARC, CFAR, MIRI. High school outreach, such as Non-trivial. In addition, they are currently not funding (or not fully funding): Many non-US think tanks, who don't want to appear influenced by an American organisation (there's now probably more [...] The original text contained 2 footnotes which were omitted from this narration. The original text contained 1 image which was described by AI. --- First published: December 21st, 2024 Source: https://forum.effectivealtruism.org/posts/s9dyyge6uLG5ScwEp/it-looks-like-there-are-some-good-funding-opportunities-in --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Which LessWrong/Alignment topics would you like to be tutored in? [Poll], published by Ruby on September 19, 2024 on LessWrong. Would you like to be tutored in applied game theory, natural latents, CFAR-style rationality techniques, "general AI x-risk", Agent Foundations, anthropic s , or some other topics discussed on LessWrong? I'm thinking about prototyping some topic-specific LLM tutor bots, and would like to prioritize topics that multiple people are interested in. Topic-specific LLM tutors would be customized with things like pre-loaded relevant context, helpful system prompts, and more focused testing to ensure they work. Note: I'm interested in topics that are written about on LessWrong, e.g. infra-bayesianism, and not magnetohydrodynamics". I'm going to use the same poll infrastructure that Ben Pace pioneered recently. There is a thread below where you add and vote on topics/domains/areas where you might like tutoring. 1. Karma: upvote/downvote to express enthusiasm about there being tutoring for a topic. 2. Reacts: click on the agree react to indicate you personally would like tutoring on a topic. 3. New Poll Option. Add a new topic for people express interest in being tutored on. For the sake of this poll, I'm more interested in whether you'd like tutoring on a topic or not, separate from the question of whether you think a tutoring bot would be any good. I'll worry about that part. Background I've been playing around with LLMs a lot in the past couple of months and so far my favorite use case is tutoring. LLM-assistance is helpful via multiple routes such as providing background context with less effort than external search/reading, keeping me engaged via interactivity, generating examples, and breaking down complex sections into more digestible pieces. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Pay-on-results personal growth: first success, published by Chipmonk on September 15, 2024 on LessWrong. Thanks to Kaj Sotala, Stag Lynn, and Ulisse Mini for reviewing. Thanks to Kaj Sotala, Brian Toomey, Alex Zhu, Damon Sasi, Anna Salamon, and CFAR for mentorship and financial support A few months ago I made the claim "Radically effective and rapid growth [motivationally / emotionally / socially] is possible with the right combination of facilitator and method". Eg: for anxiety, agency, insecurity, need for validation. To test my hypothesis, I began a pay-on-results coaching experiment: When clients achieve their goal and are confident it will last (at least one month), they pay a bounty. My first client Bob (pseudonymous) and I met at Manifest 2024, where I had set up a table at the night market for hunting "emotional security" bounties. Bob had lifelong anxiety, and it was crushing his agency and relationships. He offered a $3,000 bounty for resolving it, and I decided to pursue it. We spoke and tried my method. It was only necessary for us to talk once, apparently, because a month later he said our one conversation helped him achieve what 8 years of talk therapy could not: I'm choosing to work on problems beyond my capabilities, and get excited about situations where my weaknesses are repeatedly on display. I actually feel excited about entering social situations where chances of things going worse than I would want them to were high. So he paid his bounty when he was ready (in this case, 35 days after the session). I've been checking in with him since (latest: last week, two months after the session) and he tells me all is well. Bob also shared some additional benefits beyond his original bounty: Planning to make dancing a weekly part of my life now. (All shared with permission.) I'm also hunting many other bounties A woman working in SF after 3 sessions, text support, and three weeks: I went to Chris with a torrent of responsibilities and a key decision looming ahead of me this month. I felt overwhelmed, upset, and I didn't want just talk Having engaged in 9+ years of coaching and therapy with varying levels of success, I'm probably one of the toughest clients - equal parts hopeful and skeptical. Chris created an incredibly open space where I could easily tell him if I didn't know something, or couldn't feel something, or if I'm overthinking. He also has an uncanny sense of intuition on these things and a strong attunement to being actually effective The results are already telling: a disappointment that might've made me emotionally bleed and mope for a month was something I addressed in the matter of a couple of days with only a scoop of emotional self-doubt instead of *swimming* in self-torture. The lag time of actually doing things to be there for myself was significantly quicker, warmer, and more effective To-dos that felt very heavy lightened up considerably and began to feel fun again and as ways of connecting! I've now started to predict happier things ahead with more vivid, emotionally engaged, and realistic detail. I'll continue being intensely focused this year for the outcomes I want, but I'm actually looking forward to it! Will reflect back on Month 2! An SF founder in his 30s after 1 session and two weeks: After working with Chris, I learned One Weird Trick to go after what I really want and feel okay no matter what happens. This is a new skill I didn't learn in 3 years of IFS therapy. I already feel more confident being myself and expressing romantic interest (and I already have twice, that's new). ( More…) What the fuck? "Why does your thing work so unusually well?" asks my mentor Kaj. For one, it doesn't work for everyone with every issue, as you can see in the screenshot above. (That said, I suspect a lot of this is my fault for pursuing bounti...

Link to original articleWelcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Pay-on-results personal growth: first success, published by Chipmonk on September 15, 2024 on LessWrong. Thanks to Kaj Sotala, Stag Lynn, and Ulisse Mini for reviewing. Thanks to Kaj Sotala, Brian Toomey, Alex Zhu, Damon Sasi, Anna Salamon, and CFAR for mentorship and financial support A few months ago I made the claim "Radically effective and rapid growth [motivationally / emotionally / socially] is possible with the right combination of facilitator and method". Eg: for anxiety, agency, insecurity, need for validation. To test my hypothesis, I began a pay-on-results coaching experiment: When clients achieve their goal and are confident it will last (at least one month), they pay a bounty. My first client Bob (pseudonymous) and I met at Manifest 2024, where I had set up a table at the night market for hunting "emotional security" bounties. Bob had lifelong anxiety, and it was crushing his agency and relationships. He offered a $3,000 bounty for resolving it, and I decided to pursue it. We spoke and tried my method. It was only necessary for us to talk once, apparently, because a month later he said our one conversation helped him achieve what 8 years of talk therapy could not: I'm choosing to work on problems beyond my capabilities, and get excited about situations where my weaknesses are repeatedly on display. I actually feel excited about entering social situations where chances of things going worse than I would want them to were high. So he paid his bounty when he was ready (in this case, 35 days after the session). I've been checking in with him since (latest: last week, two months after the session) and he tells me all is well. Bob also shared some additional benefits beyond his original bounty: Planning to make dancing a weekly part of my life now. (All shared with permission.) I'm also hunting many other bounties A woman working in SF after 3 sessions, text support, and three weeks: I went to Chris with a torrent of responsibilities and a key decision looming ahead of me this month. I felt overwhelmed, upset, and I didn't want just talk Having engaged in 9+ years of coaching and therapy with varying levels of success, I'm probably one of the toughest clients - equal parts hopeful and skeptical. Chris created an incredibly open space where I could easily tell him if I didn't know something, or couldn't feel something, or if I'm overthinking. He also has an uncanny sense of intuition on these things and a strong attunement to being actually effective The results are already telling: a disappointment that might've made me emotionally bleed and mope for a month was something I addressed in the matter of a couple of days with only a scoop of emotional self-doubt instead of *swimming* in self-torture. The lag time of actually doing things to be there for myself was significantly quicker, warmer, and more effective To-dos that felt very heavy lightened up considerably and began to feel fun again and as ways of connecting! I've now started to predict happier things ahead with more vivid, emotionally engaged, and realistic detail. I'll continue being intensely focused this year for the outcomes I want, but I'm actually looking forward to it! Will reflect back on Month 2! An SF founder in his 30s after 1 session and two weeks: After working with Chris, I learned One Weird Trick to go after what I really want and feel okay no matter what happens. This is a new skill I didn't learn in 3 years of IFS therapy. I already feel more confident being myself and expressing romantic interest (and I already have twice, that's new). ( More…) What the fuck? "Why does your thing work so unusually well?" asks my mentor Kaj. For one, it doesn't work for everyone with every issue, as you can see in the screenshot above. (That said, I suspect a lot of this is my fault for pursuing bounti...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: I didn't have to avoid you; I was just insecure, published by Chipmonk on August 18, 2024 on LessWrong. I don't usually post stories on LessWrong so I'm curious to see how this is received. The first time we spoke, you asked me some questions that felt really invasive. I didn't want that to happen again, so I avoided you the entire following year. So when you said "Hi" at a party and suggested catching up, I hesitated. But curiosity won out. You still asked probing questions like "Why did you quit your job?" and "What did you think of your manager? I hear they don't have a great social reputation." These weren't questions I wanted to answer. But this time, something was different. Not you - me. In the past, I would have felt forced to answer your questions. But I'm sure you can remember how I responded when we spoke again: "Mm, I don't want to answer that question", "I don't want to gossip", and even a cheeky, "No comment :)" It didn't even take effort, that surprised me. And nothing bad happened! We just spoke about other things. I realized that I was protecting myself from you with physical distance. But instead I could protect myself from you with "No." So simple… Too simple? Why didn't I think of that before?? Oh, I know why: When I first met you, I was extremely afraid of expressing disapproval of other people. I didn't know it consciously. It was quite deeply suppressed. But the pattern fits the data. It seems that I was so afraid of this, that when you asked me those questions when we met for the first time, the thought didn't even cross my mind that I could decline to answer. If I declined a question, I unconsciously predicted you might get mad, and that would make me feel terrible about myself. So that's why I didn't say "No" to your questions when you first met me. And that's why I avoided you so bluntly with physical distance. (Although, I also avoided everyone during that year for similar reasons.) Why am I telling you all of this? You helped me grow. These days, it takes very little effort - and sometimes none at all - to reject others' requests and generally do what I want. I'm much more emotionally secure now. Also, I noticed a shift in how I perceived you. Once I realized I didn't have to avoid you, I began noticing qualities I admire. Your passion for your work. Your precise and careful reasoning. I want to learn from these traits. And now that I don't have to avoid you anymore, I can :) Addendum: Beliefs I have Emotional security is the absence of insecurities In my model, emotional security is achieved by the absence of emotional insecurities - ie: I had those unconscious predictions like, "If something bad outside of my control happens, then I'm not going to be able to feel okay." But it seems I unlearned most of mine. I don't encounter situations that make me anxious in that way anymore, and I can't imagine any new ones either. Rejecting others (and being rejected by others, same thing) has ceased to carry much unnecessary emotional weight. (The one exception I can think of is if I was afraid that someone was going to physically harm me. But that's rare.) It's about present predictions, not past trauma One might wonder, "What happened to you? What trauma caused your inability to say 'No'?" But that's all irrelevant. All that matters is that I had that unconscious prediction in that present moment. Thanks to Stag Lynn, Kaj Sotala, Damon Sasi, Brian Toomey, Epistea Residency, CFAR, Anna Salamon, Alex Zhu, and Nolan Kent for mentorship and financial support. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Link to original articleWelcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: I didn't have to avoid you; I was just insecure, published by Chipmonk on August 18, 2024 on LessWrong. I don't usually post stories on LessWrong so I'm curious to see how this is received. The first time we spoke, you asked me some questions that felt really invasive. I didn't want that to happen again, so I avoided you the entire following year. So when you said "Hi" at a party and suggested catching up, I hesitated. But curiosity won out. You still asked probing questions like "Why did you quit your job?" and "What did you think of your manager? I hear they don't have a great social reputation." These weren't questions I wanted to answer. But this time, something was different. Not you - me. In the past, I would have felt forced to answer your questions. But I'm sure you can remember how I responded when we spoke again: "Mm, I don't want to answer that question", "I don't want to gossip", and even a cheeky, "No comment :)" It didn't even take effort, that surprised me. And nothing bad happened! We just spoke about other things. I realized that I was protecting myself from you with physical distance. But instead I could protect myself from you with "No." So simple… Too simple? Why didn't I think of that before?? Oh, I know why: When I first met you, I was extremely afraid of expressing disapproval of other people. I didn't know it consciously. It was quite deeply suppressed. But the pattern fits the data. It seems that I was so afraid of this, that when you asked me those questions when we met for the first time, the thought didn't even cross my mind that I could decline to answer. If I declined a question, I unconsciously predicted you might get mad, and that would make me feel terrible about myself. So that's why I didn't say "No" to your questions when you first met me. And that's why I avoided you so bluntly with physical distance. (Although, I also avoided everyone during that year for similar reasons.) Why am I telling you all of this? You helped me grow. These days, it takes very little effort - and sometimes none at all - to reject others' requests and generally do what I want. I'm much more emotionally secure now. Also, I noticed a shift in how I perceived you. Once I realized I didn't have to avoid you, I began noticing qualities I admire. Your passion for your work. Your precise and careful reasoning. I want to learn from these traits. And now that I don't have to avoid you anymore, I can :) Addendum: Beliefs I have Emotional security is the absence of insecurities In my model, emotional security is achieved by the absence of emotional insecurities - ie: I had those unconscious predictions like, "If something bad outside of my control happens, then I'm not going to be able to feel okay." But it seems I unlearned most of mine. I don't encounter situations that make me anxious in that way anymore, and I can't imagine any new ones either. Rejecting others (and being rejected by others, same thing) has ceased to carry much unnecessary emotional weight. (The one exception I can think of is if I was afraid that someone was going to physically harm me. But that's rare.) It's about present predictions, not past trauma One might wonder, "What happened to you? What trauma caused your inability to say 'No'?" But that's all irrelevant. All that matters is that I had that unconscious prediction in that present moment. Thanks to Stag Lynn, Kaj Sotala, Damon Sasi, Brian Toomey, Epistea Residency, CFAR, Anna Salamon, Alex Zhu, and Nolan Kent for mentorship and financial support. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

iHeartMedia participated in the annual NRG Choose to Give Media Flag Football Game which benefits Children's Hospital, Philabundance, Big Brothers/Big Sisters and Nemours Children's Hospital.  I speak to Mike Rombach VP and General Manager for NRG Energy.  https://www.nrg.com/The newly passed Philadelphia budget removes funding for syringes in needle exchange programs.  Residents in Kensington have complained about the proliferation of needles on the streets and the handling of the program by local nonprofits.  However Ronald Collman, MD, Professor of Medicine and Microbiology Director, Penn Center for AIDS research (CFAR) and Dr. Florence Momplaisir Assistant Professor, Division of Infectious Diseases, Associate Chief for Diversity, Equity, and Inclusion Fellow, Leonard Davis Institute of Health Economics University of Pennsylvania School of Medicine argue that this  is a misguided step that will harm not just people who inject drugs but the broader Philadelphia population.I'm joined by Angela Giampolo, of Giampolo Law Group and Philly Gay Lawyer  for our monthly feature SpeakOUT about issues related to the LGBTQIA community and how they conect  to the broader community.  This week we talk about how we can be better allies. https://www.phillygaylawyer.com/https://giampololaw.com/   

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: rapid psychological growth, published by Chipmonk on June 6, 2024 on LessWrong. After a one-hour session with an exceptional counselor, I never suffered over that romantic incident again. Although, that's inaccurate, I also had 2x half-hour relapses in following month. After a few more sessions, I stopped doing depression. I brought the rest of my anxieties to that counselor over the following year, and… Radically effective and rapid psychological growth is possible with the right combination of counselor and method. But this is rare in 2024! Introspection that actually works It was while working with that counselor that, for the first time I could remember, I was able to actually do introspection. Before, whenever I had problems that seemed to be caused by my psychology, I would do the obvious thing and ask myself, "Why am I doing ? Why am I not doing ?" But that almost never worked. Usually I would get a response back like, "Because it's hard, I'm lazy, and it's just a bad habit." The same problems would come back again and again. Meditation didn't help me much either. But, for me, this counselor did. I would come to a session suffering from something, he would prompt me into feeling into my body about the issue - which is important because the body represents the unconscious - and then in the following Socratic conversation I would be able to make rapid and dramatic progress on my problem. Big anxieties gone in an hour. (For context, most of my problems then could be reduced to either "I feel anxious about X social situation." and/or "I am disliking myself and I'm suffering about that.") Learning to facilitate Later, I trained with that counselor and learned his method. As part of my training I facilitated for four volunteers, and they seemed to have similar results that I had: rapid and dramatic resolution of the issue they came with in one hour. (Caveat: I never spoke to these volunteers again, so I don't know if the effect lasted.) But the sixth time I facilitated for someone was different. I experimented: I let the conversation run as long as it needed to, and I proactively tried to target the deepest roots of his emotional insecurity using the full force of my psychological research. After our three-hour conversation, he said, This session was significantly more productive than the last 6 months of professional CBT and talk therapy I did combined. (For context, he was a CFAR alumni and also very experienced with Focusing.) We didn't do any other sessions, but I followed up after six months to ask how he was doing: I can't stress how much I appreciated that dialogue, it really made me feel better, and I think I have already expressed much of what it made me feel. […] The effectiveness of your presence defeated my incredulity, and then some. This seems not to be a fluke, either. I've facilitated for seven other people since then and four have had similarly large shifts, eg, Your communication style made it easy to identify and release limiting beliefs. I felt noticeably more secure after just a few hours. That said, the other three people I facilitated seemed to have smaller effects, though each claims it was positive. More information about my emotional security tune-ups is available on chrislakin.com/now Radically effective and rapid psychological growth is possible with the right combination of counselor and method! What does a session look like? Here's the closest example I could find of what rapid psychological growth looks like in practice. (Note: I don't completely agree with their method, and also I wonder if the client's progress could've been even quicker.) Bolding is mine. Coherence Therapy for Panic Attacks, 2007 Bruce Ecker & Laurel Hulley: Carmen, a stylish freelance writer, was 35 and happily married, but she experie...

Link to original articleWelcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: rapid psychological growth, published by Chipmonk on June 6, 2024 on LessWrong. After a one-hour session with an exceptional counselor, I never suffered over that romantic incident again. Although, that's inaccurate, I also had 2x half-hour relapses in following month. After a few more sessions, I stopped doing depression. I brought the rest of my anxieties to that counselor over the following year, and… Radically effective and rapid psychological growth is possible with the right combination of counselor and method. But this is rare in 2024! Introspection that actually works It was while working with that counselor that, for the first time I could remember, I was able to actually do introspection. Before, whenever I had problems that seemed to be caused by my psychology, I would do the obvious thing and ask myself, "Why am I doing ? Why am I not doing ?" But that almost never worked. Usually I would get a response back like, "Because it's hard, I'm lazy, and it's just a bad habit." The same problems would come back again and again. Meditation didn't help me much either. But, for me, this counselor did. I would come to a session suffering from something, he would prompt me into feeling into my body about the issue - which is important because the body represents the unconscious - and then in the following Socratic conversation I would be able to make rapid and dramatic progress on my problem. Big anxieties gone in an hour. (For context, most of my problems then could be reduced to either "I feel anxious about X social situation." and/or "I am disliking myself and I'm suffering about that.") Learning to facilitate Later, I trained with that counselor and learned his method. As part of my training I facilitated for four volunteers, and they seemed to have similar results that I had: rapid and dramatic resolution of the issue they came with in one hour. (Caveat: I never spoke to these volunteers again, so I don't know if the effect lasted.) But the sixth time I facilitated for someone was different. I experimented: I let the conversation run as long as it needed to, and I proactively tried to target the deepest roots of his emotional insecurity using the full force of my psychological research. After our three-hour conversation, he said, This session was significantly more productive than the last 6 months of professional CBT and talk therapy I did combined. (For context, he was a CFAR alumni and also very experienced with Focusing.) We didn't do any other sessions, but I followed up after six months to ask how he was doing: I can't stress how much I appreciated that dialogue, it really made me feel better, and I think I have already expressed much of what it made me feel. […] The effectiveness of your presence defeated my incredulity, and then some. This seems not to be a fluke, either. I've facilitated for seven other people since then and four have had similarly large shifts, eg, Your communication style made it easy to identify and release limiting beliefs. I felt noticeably more secure after just a few hours. That said, the other three people I facilitated seemed to have smaller effects, though each claims it was positive. More information about my emotional security tune-ups is available on chrislakin.com/now Radically effective and rapid psychological growth is possible with the right combination of counselor and method! What does a session look like? Here's the closest example I could find of what rapid psychological growth looks like in practice. (Note: I don't completely agree with their method, and also I wonder if the client's progress could've been even quicker.) Bolding is mine. Coherence Therapy for Panic Attacks, 2007 Bruce Ecker & Laurel Hulley: Carmen, a stylish freelance writer, was 35 and happily married, but she experie...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: [Draft] The humble cosmologist's P(doom) paradox, published by titotal on March 17, 2024 on The Effective Altruism Forum. [This post has been published as part of draft amnesty week. I did quite a bit of work on this post, but abandoned it because I was never sure of my conclusions. I don't do a lot of stats work, so I could never be sure if I was missing something obvious, and I'm not certain of the conclusions to draw. If this gets a good reception, I might finish it off into a proper post.] Part 1: Bayesian distributions I'm not sure that I'm fully on board the "Bayesian train". I worry about Garbage in, garbage, out, that it will lead to overconfidence about what are ultimately just vibes, etc. But I think if you are doing Bayes, you should at least try to do it right. See, in Ea/rationalist circles, the discussion of Bayesianism often stops at bayes 101. For example, the "sequences" cover the "mammaogram problem", in detail, but never really cover how Bayesian statistics works outside of toy examples. The CFAR handbook doesn't either. Of course, plenty of the people involved have read actual textbooks and the like, (and generally research institutes use proper statistics), but I'm not sure that the knowledge has spread it's way around to the general EA public. See, in the classic mammogram problem (I won't cover the math in detail because there are 50 different explainers), both your prior probabilities, and the amount you should update, are well established, known, exact numbers. So you have your initial prior of say, 1%, that someone has cancer. and then you can calculate a likelihood ratio of exactly 10:1 resulting from a probable test, getting you a new, exact 10% chance that the person has cancer after the test. Of course, in real life, there is often not an accepted, exact number for your prior, or for your likliehood ratio. A common way to deal with this in EA circles is to just guess. Do aliens exist? well I guess that there is a prior of 1% that they do, and then i'll guess likliehood ratio of 10:1 that we see so many UFO reports, so the final probability of aliens existing is now 10%. [magnus vinding example] Just state that the numbers are speculative, and it'll be fine. Sometimes, people don't even bother with the Bayes rule part of it, and just nudge some numbers around. I call this method "pop-Bayes". Everyone acknowledges that this is an approximation, but the reasoning is that some numbers are better than no numbers. And according to the research of Phillip Tetlock, people who follow this technique, and regularly check the results of their predictions, can do extremely well at forecasting geopolitical events. Note that for practicality reasons they only tested forecasting for near-term events where they thought the probability was roughly in the 5-95% range. Now let's look at the following scenario (most of this is taken from this tutorial): Your friend Bob has a coin of unknown bias. It may be fair, or it may be weighted to land more often on heads or tails. You watch them flip the coin 3 times, and each time it comes up heads. What is the probability that the next flip is also heads? Applying "pop-bayes" to this starts off easy. Before seeing any flip outcomes, the prior of your final flip being heads is obviously 0.5, just from symmetry. But then you have to update this based on the first flip being heads. To do this, you have to estimate P(E|H) and P(E|~H). P(E|H) corresponds to "the probability of this flip having turned up heads, given that my eventual flip outcome is heads". How on earth are you meant to calculate this? Well, the key is to stop doing pop-bayes, and start doing actual bayesian statistics. Instead of reducing your prior to a single number, you build a distribution for the parameter of coin bias, with 1 corresponding to fully...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Rationality Research Report: Towards 10x OODA Looping?, published by Raemon on February 24, 2024 on LessWrong. 6 months ago I wrote Feedbackloop-first Rationality. I didn't followup on it for awhile (except for sporadic Deliberate ("Purposeful?") Practice Club). I just spent 6 weeks actually exploring "how would I build my own cognition training program?". In the process of doing so, I've iterated a bunch. I'm still in an orienting phase, but it seemed worth writing down the current stage of my thoughts. What's my goal? A rough overview: I want to get more, higher quality "X-risk thinker hours" hours. This includes AI alignment technical research, AI macrostrategy research, policy, governance, as well as people (such as Lightcone team) deciding which infrastructure to build, I'm particularly interested in getting more "serial research", as opposed to more "parallel research." We can throw more researchers at a problem, but if there are some problems that require one person to synthesize 10+ years of experience, all the parallel research won't help. An obvious way to improve researcher hours is "via mentorship", but I think there is a mentorship bottleneck. So, I'm interested in strategies that train tacit cognitive skills that either don't require mentorship, or leveraging expertise from outside the current x-risk ecosystem. This is all parented under the higher level goal of "contribute meaningfully to x-risk reduction", but it feels relevant/meaty enough to be worth running at this goal for awhile. "Rationality for the sake of existential risk" A part of me romantically wants to pursue "rationality training for rationality training's sake." Alas, the world is big and my time is limited and I just can't actually justify putting years of effort into something, if I didn't think it would help with x-risk. CFAR went through a phase where (some leaders) framed things as: "Rationality, for the sake of rationality, for the sake of existential risk." i.e. try to earnestly build something rationality-focused for it's own sake, because that seemed both healthier and better for x-risk than "rationality for the sake of x-risk", directly. I think this was a reasonable thing to try, but my impression is this didn't work that well. If you tell yourself (and your students) "I'm doing this for the sake of rationality itself", but then in practice you're getting people to delicately open up their soul and figure out their true goals... and all-the-while radiating "man I really hope your goals turn out to involve saving the worlds from AIs", that may fuck up the "earnestly try to figure out your goals" process. So: I am not here to help you earnestly figure out your goals. That's an important part of rationality, and it might come about incidentally while people do exercises I develop, but it's not what I'm focused on this year. I am here to develop and teach cognitive skills, which help you solve confusing problems at the edge of your ability. I'm doing this to push forward humanity's frontier of "how quickly can we do challenging research?", and strive towards 10x science. I will prioritize learning and teaching those skills to people who seem like they are going to help with x-risk somehow, but I aim to write up a lot of stuff publicly, and trying-where-possible to output exercises that other people can do on their own, for whatever reasons they want. (See Exercise: Solve "Thinking Physics" as an example) The Story So Far Feedback-loops and "deliberate practice", vs "Just Clicking" I just spent a month workshopping various "teaching rationality" plans. My initial ideas were framed around: Deliberate practice is costly and kinda sucks Therefore, people haven't invested in it much, as either "rationality training programs", or as an "alignment research training programs." Therefore,...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: CFAR Takeaways: Andrew Critch, published by Raemon on February 14, 2024 on LessWrong. I'm trying to build my own art of rationality training, and I've started talking to various CFAR instructors about their experiences - things that might be important for me to know but which hadn't been written up nicely before. This is a quick write up of a conversation with Andrew Critch about his takeaways. (I took rough notes, and then roughly cleaned them up for this. Some of my phrasings might not exactly match his intended meaning, although I've tried to separate out places where I'm guessing what he meant from places where I'm repeating his words as best I can) "What surprised you most during your time at CFAR? Surprise 1: People are profoundly non-numerate. And, people who are not profoundly non-numerate still fail to connect numbers to life. I'm still trying to find a way to teach people to apply numbers for their life. For example: "This thing is annoying you. How many minutes is it annoying you today? how many days will it annoy you?". I compulsively do this. There aren't things lying around in my life that bother me because I always notice and deal with it. People are very scale-insensitive. Common loci of scale-insensitivity include jobs, relationship, personal hygiene habits, eating habits, and private things people do in their private homes for thousands of hours. I thought it'd be easy to use numbers to not suck. Surprise 2: People don't realize they need to get over things. There was a unit a CFAR called 'goal factoring'. Early in it's development, the instructor would say to their class: "if you're doing something continuously, fill out a 2x2 matrix", where you ask: 1) does this bother me? (yes or not), and 2) is it a problem? (yes or no). Some things will bother you and not be a problem. This unit is not for that." The thing that surprised me, was that I told the "C'mon instructor. It's not necessary to manually spell out that people just need to accept some things and get over them. People know that, it's not worth spending the minute on it." At the next class, the instructor asked the class: "When something bothers you, do you ask if you need to get over it?". 10% of people raised their hand. People didn't know the "realize that some things bother you but it's not a problem and you can get over it." Surprise 3: When I learned Inner Simulator from Kenzie, I was surprised that it helped with everything in life forever. [I replied: "I'm surprised that you were surprised. I'd expect that to have already been part of your repertoire."] The difference between Inner Simulator and the previous best tool I had was: Previously, I thought of my system 1 as something that both "decided to make queries" and "returned the results of the queries." i.e. my fast intuitions would notice something and give me information about it. I previously thought of "inner sim" as a different intelligence that worked on it's own. The difference with Kenzie's "Inner Sim" approach is that my System 2 could decide when to query System 1. And then System 1 would return the query with its anticipations (which System 2 wouldn't be able to generate on its own). [What questions is System 1 good at asking that System 2 wouldn't necessarily ask?] System 1 is good at asking "is this person screwing me over?" without my S2 having to realize that now's a good time to ask that question. (S2 also does sometimes ask this question, at complementary times) Surprise 4: How much people didn't seem to want things And, the degree to which people wanted things was even more incoherent than I thought. I thought people wanted things but didn't know how to pursue them. [I think Critch trailed off here, but implication seemed to be "basically people just didn't want things in the first place"] What do other people see...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Believing In, published by AnnaSalamon on February 8, 2024 on LessWrong. "In America, we believe in driving on the right hand side of the road." Tl;dr: Beliefs are like bets (on outcomes the belief doesn't affect). "Believing in"s are more like kickstarters (for outcomes the believing-in does affect). Epistemic status: New model; could use critique. In one early CFAR test session, we asked volunteers to each write down something they believed. My plan was that we would then think together about what we would see in a world where each belief was true, compared to a world where it was false. I was a bit flummoxed when, instead of the beliefs-aka-predictions I had been expecting, they wrote down such "beliefs" as "the environment," "kindness," or "respecting people." At the time, I thought this meant that the state of ambient rationality was so low that people didn't know "beliefs" were supposed to be predictions, as opposed to group affiliations. I've since changed my mind. My new view is that there is not one but two useful kinds of vaguely belief-like thingies - one to do with predictions and Bayes-math, and a different one I'll call "believing in." I believe both are lawlike, and neither is a flawed attempt to imitate/parasitize the other. I further believe both can be practiced at once - that they are distinct but compatible. I'll be aiming, in this post, to give a clear concept of "believing in," and to get readers' models of "how to 'believe in' well" disentangled from their models of "how to predict well." Examples of "believing in" Let's collect some examples, before we get to theory. Places where people talk of "believing in" include: An individual stating their personal ethical code. E.g., "I believe in being honest," "I believe in hard work," "I believe in treating people with respect," etc. A group stating the local social norms that group tries to practice as a group. E.g., "Around here, we believe in being on time." "I believe in you," said by one friend or family member to another, sometimes in a specific context ("I believe in your ability to win this race,") sometimes in a more general context ("I believe in you [your abilities, character, and future undertakings in general]"). A difficult one-person undertaking, of the sort that'll require cooperation across many different time-slices of a self. ("I believe in this novel I'm writing.") A difficult many-person undertaking. ("I believe in this village"; "I believe in America"; "I believe in CFAR"; "I believe in turning this party into a dance party, it's gonna be awesome.") A political party or platform ("I believe in the Democratic Party"). A scientific paradigm. A person stating which entities they admit into their hypotheses, that others may not ("I believe in atoms"; "I believe in God"). It is my contention that all of the above examples, and indeed more or less all places where people naturally use the phrase "believing in," are attempts to invoke a common concept, and that this concept is part of how a well-designed organism might work.[1] Inconveniently, the converse linguistic statement does not hold - that is: People who say "believing in" almost always mean the thing I'll call "believing in" But people who say "beliefs" or "believing" (without the "in") sometimes mean the Bayes/predictions thingy, and sometimes mean the thing I'll call "believing in." (For example, "I believe it takes a village to raise a child" is often used to indicate "believing in" a particular political project, despite how it does not use the word "in"; also, here's an example from Avatar.) A model of "believing in" My model is that "I believe in X" means "I believe X will yield good returns if resources are invested in it." Or, in some contexts, "I am investing (some or ~all of) my resources in keeping with X." (Backgro...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Theories of Applied Rationality, published by Camille Berger on February 4, 2024 on LessWrong. tl;dr: within the LW community, there are many clusters of strategies to achieve rationality: doing basic exercices, using jargon, reading, partaking workshops, privileging object-level activities, and several other opinions like putting an accent on feedback loops, difficult conversations or altered states of consciousness. Epistemic status: This is a vague model to help me understand other rationalists and why some of them keep doing things I think are wrong, or suggest me to do things I think are wrong. This is not based on real data. I will update according to possible discussions in the comments. Please be critical. Spending time in the rationalist community made me realize that there were several endeavors at reaching rationality that seemed to exist, some of which conflicted with others. This made me quite frustrated as I thought that my interpretation was the only one. The following list is an attempt at distinguishing the several approaches I've noticed. Of course, any rationalist will probably have elements of all theories at the same time. See each theory as the claim that a particular set of elements prevails above others. Believing in one theory usually goes on par with being fairly suspicious of others. Finally, remember that these categories are an attempt to distinguish what people are doing, not a guide about what side you should pick (if the sides exist at all). I suspect that most people end up applying one theory for practical reasons, more than because they have deeply thought about it at all. Basics Theory Partakers of the basics theory put a high emphasis on activities such as calibration, forecasting, lifehacks, and other fairly standard practices of epistemic and instrumental rationality. They don't see any real value in reading extensively LessWrong or going to workshops. They first and foremost believe in real-life, readily available practice. For them, spending too much time in the rationalist community, as opposed to doing simple exercises, is the main failure mode to avoid. Speaking Theory Partakers of the Speaking theory, although often relying on basics, usually put a high emphasis on using concepts met on LessWrong in daily parlance, although they generally do not necessarily insist on reading content on LessWrong. They may also insist on the importance of talking and discussing disagreements in a fairly regular way, while heavily relying on LessWrong terms and references in order to shape their thinking more rationally. They disagree with the statement that jargon should be avoided. For them, keeping your language, thinking, writing and discussion style the same way that it was before encountering rationality is the main failure mode to avoid. Reading Theory Partakers of the Reading theory put a high emphasis on reading LessWrong, more usually than not the " Canon ", but some might go to a further extent and insist on reading other materials as well, such as the books recommended on the CFAR website, rationalist blogs, or listening to a particular set of podcasts. They can be sympathetic or opposed to relying on LessWrong Speak, but don't consider it important. They can also be fairly familiar with the basics. For them, relying on LW Speak or engaging with the community while not mastering the relevant corpus is the main failure mode to avoid. Workshop Theory Partakers of the Workshop Theory consider most efforts of the Reading and Speaking theory to be somehow misleading. Since rationality is to be learned, it has to be deliberately practiced, if not ultralearned, and workshops such as CFAR are an important piece of this endeavor. Importantly, they do not really insist on reading the Sequences. Faced with the question " Do I need to read X...

Cfarë është e vërteta? Sa e vjetër është gjuha shqipe? A jemi ne krijuesit e civilizimit?

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: [Linkpost] George Mack's Razors, published by trevor on November 28, 2023 on LessWrong. I don't use Twitter/X, I only saw this because it was on https://twitter.com/ESYudkowsky which I check every day (an example of a secure way to mitigate brain exposure to news feed algorithms). The galaxy-brained word combinations here are at the standard of optimization I hold very highly (e.g. using galaxy-brained combinations of words to maximize the ratio of value to wordcount). If someone were to, for example, start a human intelligence amplification coordination takeoff by getting the best of both worlds between the long, intuitive CFAR handbook and the short, efficient hammertime sequence, this is the level of writing skill that they would have to be playing at: The most useful razors and rules I've found: 1. Bragging Razor - If someone brags about their success or happiness, assume it's half what they claim If someone downplays their success or happiness, assume it's double what they claim 2. High Agency Razor - If unsure who to work with, pick the person that has the best chances of breaking you out of a 3rd world prison. 3. The Early-Late Razor - If it's a talking point on Reddit, you might be early. If it's a talking point on LinkedIn, you're definitely late. 4. Luck Razor - If stuck with 2 equal options, pick the one that feels like it will produce the most luck later down the line. I used this razor to go for drinks with a stranger rather than watch Netflix. In hindsight, it was the highest ROI decision I've ever made. 5. Buffett's Law - "The value of every business is 100% subject to government interest rates" - Warren Buffett 6. The 6-Figure Razor - If someone brags about "6 figures" -- assume it's closer to $100K than $900K. 7. Parent Rule - Break down the investments your parents made in you: Time, Love, Energy, and Money. If they are still alive, aim to hit a positive ROI (or at least break even.) 8. Instagram Razor - When you see a photo of an influencer looking attractive on Instagram -- assume there are 99 worse variations of that photo you haven't seen. They just picked the best one. 9. Narcissism Razor - If worried about people's opinions, remember they are too busy worrying about other people's opinions of them. 99% of the time you're an extra in someone else's movie 10. Everyday Razor - If you go from doing a task weekly to daily, you achieve 7 years of output in 1 year. If you apply a 1% compound interest each time, you achieve 54 years of output in 1 year. 11. Bezos Razor - If unsure what action to pick, let your 90-year-old self on death bed choose it. 12. Creativity Razor - If struggling to think creatively about a subject, transform it: • Turn a thought into a written idea. • A written idea into a drawing. • A drawing into an equation. • An equation into a conversation. In the process of transforming it, you begin to spot new creative connections. 13. The Roman Empire Rule - Historians now recognize the Roman Empire fell in 476 - but it wasn't acknowledged by Roman society until many generations later. If you wait for the media to inform you, you'll either be wrong or too late. 14. Physics Razor - If it doesn't deny the law of physics, then assume it's possible. Do not confuse society's current lack of knowledge -- with this knowledge being impossible to attain. E.g. The smartphone seems impossible to someone from the 1800s -- but it was possible, they just had a lack of knowledge. 15. Skinner's Law - If procrastinating, you have 2 ways to solve it: • Make the pain of inaction > Pain of action • Make the pleasure of action > Pleasure of inaction 16. Network Razor - If you have 2 quality people that would benefit from an intro to one another, always do it. Networks don't divide as you share them, they multiply. 17. Gell-Mann Razor - Assume every media...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: EA orgs' legal structure inhibits risk taking and information sharing on the margin, published by Elizabeth on November 5, 2023 on The Effective Altruism Forum. What is fiscal sponsorship? It's fairly common for EA orgs to provide fiscal sponsorship to other EA orgs. Wait, no, that sentence is not quite right. The more accurate sentence is that there are very few EA organizations, in the legal sense; most of what you think of as orgs are projects that are legally hosted by a single org, and which governments therefore consider to be one legal entity. The king umbrella is Effective Ventures Foundation, which hosts CEA, 80k, Longview, EA Funds, Giving What We Can, Asterisk magazine, Centre for Governance of AI, Forethought Foundation, Non-Trivial, and BlueDot Impact. Posts on the castle also describe it as an EVF project, although it's not listed on the website. Rethink Priorities has a program specifically to provide sponsorship to groups that need it. LessWrong/Lightcone is hosted by CFAR, and have sponsored at least one project themselves (source: me. It was my project). Fiscal sponsorship has a number of advantages. It gets you the privileges of being a registered non-profit (501c3 in the US) without the time-consuming and expensive paperwork. That's a big deal if the project is small, time-limited (like mine was) or is an experiment you might abandon if you don't see results in four months. Even for large projects/~orgs, sharing a formal legal structure makes it easier to share resources like HR departments and accountants. In the short term, forming a legally independent organization seems like a lot of money and effort for the privilege of doing more paperwork. The downsides of fiscal sponsorship …are numerous, and grow as the projects involved do. The public is rightly suspicious about projects that share a legal entity claiming to be independent, so bad PR for one risks splash damage for all. The government is very confident in its belief that you are the same legal entity, so legal risks are shared almost equally (iamnotalawyer). So sharing a legal structure automatically shares risk. That may be fixable, but the fix comes at its own cost. The easiest thing to do is just take fewer risks. Don't buy retreat centers that could be described as lavish. And absolutely, 100%, don't voluntarily share any information about your interactions with FTX, especially if the benefits to doing so are intangible. So some amount of value is lost because the risk was worth it for an individual or small org, but not to the collective. [it is killing me that I couldn't follow the rule of three with that list, but it turns out there aren't that many legible, publicly visible examples of decisions to not share information] And then there are the coordination costs. Even if everyone in the legal org is okay with a particular risk, you now have an obligation to check with them.The answer is often "it's complicated", which leads to negotiations eating a lot of attention over things no one cares that much about. Even if there is some action everyone is comfortable with, you may not find it because it's too much work to negotiate between that many people (if you know anyone who lived in a group house during covid: remember how fun it was to negotiate safety rules between 6 people with different value functions and risk tolerances?). Chilling effects A long, complicated (but nonetheless simplified)example The original version of this story was one paragraph long. It went something like: A leader at an EVF-sponsored project wanted to share some thoughts on a controversial issue, informally but in public.The comments were not riskless, but this person would happily have taken the risk if it affected only themselves or their organization. Someone at EVF said no. Boo, grrr. I sent that versi...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: 2023 LessWrong Community Census, Request for Comments, published by Screwtape on November 1, 2023 on LessWrong. Overview I would like there to be a LessWrong Community Census, because I had fun playing with the data from last year and there's some questions I'm curious about. It's also an entertaining site tradition. Since nobody else has stepped forward to make the community census happen, I'm getting the ball rolling. This is a request for comments, constructive criticism, careful consideration, and silly jokes on the census. Here's the draft. I'm posting this request for comments on November 1st. I'm planning to incorporate feedback throughout November, then on December 1st I'll update the census to remove the "DO NOT TAKE" warning at the top, and make a new post asking people to take the census. I plan to let it run throughout all December, close it in the first few days of January, and then get the public data and analysis out sometime in mid to late January. How Was The Draft Composed? I coped the question set from 2022, which itself took extremely heavy inspiration from previous years. I then added a section sourced from the questions Ben Pace of the LessWrong team had been considering in 2022, and another section of questions I'd be asking on a user survey if I worked for LessWrong. (I do not work for LessWrong.) Next I fixed some obvious mistakes from last year (in particular allowing free responses on the early politics questions) as well as changed some things that change every year like the Calibration question, and swapped around the questions in the Indulging My Curiosity section. Changes I'm Interested In In general, I want to reduce the number of questions. Last year I asked about the length and overall people thought it was a little too long. Then I added more questions. (The LW Team Questions and the Questions The LW Team Should Have Asked section.) I'm inclined to think those sections aren't pulling their weight right now, but I do think it's worth asking good questions about how people use the website on the census. I'm likely to shrink down the religion responses, as I don't think checking the different variations of e.g. Buddhism or Judaism revealed anything interesting. I'd probably put them back to the divisions used in earlier versions of the survey. I'm sort of tempted to remove the Numbers That Purport To Measure Your Intelligence section entirely. I believe it was part of Scott trying to answer a particular question about the readership, and while I love his old analyses they could make space for current questions. The main arguments in favour of keeping them is that they don't take up much space, and they've been around for a while. The Detailed Questions From Previous Surveys and Further Politics sections would be where I'd personally start making some cuts, though I admit I just don't care about politics very much. Some people care a lot about politics and if anyone wants to champion those sections that seems potentially fun. This may also be the year that some of the "Detailed Questions From Previous Surveys" get questions can get moved into the survey proper or dropped. I'd be excited to add some questions that would help adjacent or subset communities. If you're with CFAR, The Guild of the Rose, Glowfic, or an organization like that I'm cheerful about having some questions you're interested in, especially if the questions would be generally useful or fun to discuss. I've already offered to the LessWrong team directly, but I'll say again that I'd be excited to try and ask questions that would be useful for you all. You don't actually have to be associated with an organization either. If there's a burning question you have about the general shape of the readership, I'm interested in sating other people's curiosity and I'd like to encou...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How can I get help becoming a better rationalist?, published by TeaTieAndHat on July 13, 2023 on LessWrong. Well. Right now, being 'a rationalist' could be said to be a massive part of my identity, at least judging by the absurd amount of time I've spent reading posts here, or SSC/ACX, and in a few other places. Yet, I'm still a mere lurker unfamiliar with most of the local customs. But it's not what matters. What does is that I'm a terrible rationalist. You see, rationality takes practice. And reading stuff on LW isn't practice at all. If anything, it's just a great way of filling my brain with a lot of useful concepts, and then either blame myself for not using them, or use them for something entirely unrelated to their normal purpose. Often, to make myself feel worse, and think worse. As the saying goes, rationality is a martial art. Learning it by reading the rules, or by watching other people apply the rules, is about as effective as developing one's muscles by watching sports on TV. I know of the CFAR, and of various related groups, meetups for ACX readers or for other people, etc. But, apart from ACX meetups, which aren't about being better rationalists per se, I don't have easy access to any of those, or certainly to a general environment which welcomes this. You know, not being in the Bay Area and all. And yet, I want to be more rational as much as anyone who's been lurking here for five years wants it, and given how depressed I was until very recently, I probably badly need it, too. I'm not sure what kind of answers I expect, but, like, how can I push myself to learn more, and especially to practice more, and ideally to actually use rationality to improve my life? Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Guide to rationalist interior decorating, published by mingyuan on June 19, 2023 on LessWrong. Recently someone asked me to write a guide to rationalist interior decorating, since there's a set of products and best practices (originating with CFAR and Lightcone) that have gotten wide adoption. I'm perhaps not the very most qualified person to write this post, but I've been into interior decorating since before the Lightcone team got into it, and I basically know what they do, plus they're all very busy whereas I wasn't doing anything else with my time anyway. So here's this post, which I have written all by myself like a loose cannon; blame me for everything. I should point out that this post is anthropological, not normative. That is to say, this isn't a description of what I believe to be ‘optimal' interior decorating; instead it's a guide to how to make your space more like what's come to be standard for rationalist coworking spaces and group houses in Berkeley (and to a lesser extent elsewhere — I've seen a Lightcone-style coworking space in New York and lumenators in Oxford!). That said, I do think the reason a lot of the products have gotten adoption is that they're pretty good! Lighting Lighting is so important. The wrong light can give you a migraine or make you depressed. The right light can make you marvel in wonder at its beauty (like sunshine through a forest canopy, or incandescent fairy lights). I love lighting. All hail lighting. Lighting basics I'm going to cover mostly what you should be looking for when buying (mainly LED) lights, but if you're interested in the topic of interior lighting more broadly, I recommend this article, which goes into more depth and technical details. You should always be able to find the CRI, color temperature, and lumens of a product listed on the box or the online product page. I don't think these should be different in different countries. CRI (color rendering index) CRI, or color rendering index, “is the measurement of how light affects how you see color”. You ideally want lights with CRI >95 but this is hard to find; 90+ is a more realistic goal and will be nearly as good, while 80+ is cheaper but noticeably worse. CRI only applies to buying LEDs — incandescent lights all have a CRI of 100 because they, like the sun, produce light via blackbody radiation. Color temperature Color temperature matters a lot when choosing lights! 2700K (‘warm white') will be noticeably yellow, 5000K (‘daylight') will be noticeably blue, and 4000K (‘soft white') is approximately white. The K stands for Kelvin, by analogy to blackbody radiators. Color temperature is largely a matter of personal preference, but that doesn't mean it isn't important. The original lumenators used a mix of 2700K and 5000K bulbs (to imitate sunlight?), but personally I find the bluer bulbs depressing, so I use all 2700K. Lumens Lumens are a measure of brightness. As a reference point, the most common type of incandescent bulb (60 Watts) was about 800 lumens. Brighter bulbs draw more power (though this is mostly negligible for LEDs) and are generally more expensive. Lumens specifically measure how much light something emits; note that the placement of lights and the characteristics of the room will affect how much light you actually receive. I like this article on the difference between lux and lumens (mainly because of the cool graphic). Diffusion This one isn't a technical term, but everyone knows that looking directly at a bright light is painful / unpleasant / maybe bad for your vision. If you're using really bright lights, you'll probably want something that diffuses the light without absorbing too much of it. This is the purpose of lampshades, light fixtures, and the paper lanterns used on lumenators. Bright lighting Rationalists first got into really bright...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Announcing the Prague Fall Season 2023 and the Epistea Residency Program, published by Epistea on May 22, 2023 on The Effective Altruism Forum. Summary Following a successful pilot in 2022, we are announcing the Prague Fall Season 2023, which is a program run by Epistea, happening from September 1 to December 10 at FIXED POINT in Prague, Czech Republic. In this time, FIXED POINT will host a variety of programs, projects, events and individuals in the areas of existential security, rationality, epistemics, and effective altruism. We will announce specific events and programs as we confirm them but for now, our flagship program is the 10-week Epistea Residency Program for teams working on projects related to epistemics and rationality. We are now seeking expressions of interest from potential Epistea residents and mentors. What is a season? The main benefit of doing a season is having a dedicated limited time to create an increased density of people in one place. This creates more opportunities for people to collaborate, co-create and co-work on important projects - sometimes in a new location. This happens to some extent naturally around major EA conferences in London or San Francisco - many people are there at the same time which creates opportunities for additional events and collaborations. However, the timeframe is quite short and it is not clearly communicated that there are benefits in staying in the area longer and there is not a lot of infrastructure in place to support that. We ran the pilot project Prague Fall Season last autumn: Along with 25 long-term residents, we hosted over 300 international visitors between September and December 2022. We provided comprehensive support through funding, venue operations, technical and personal development programs, social gatherings, and additional events, such as the CFAR workshop series. Based on the feedback we received and our own experience with the program, we decided to produce another edition of Prague Fall Season this year with a couple of changes: We are narrowing the scope of the program primarily to existential security, epistemics, and rationality. We ask that participants of the season help us share the cost of running the FIXED POINT house. We may be able to offer financial aid on a case by case basis but the expectation is that when you visit, you can cover at least some part of the cost. We are seeking event organizers who would like to make their events part of the season. We will be sharing more information about how to get involved soon. For now, our priority is launching the Epistea Residency program. The Epistea Residency 2023 The backbone of the Prague Fall Season 2023 will once again be a 10-week residency program. This year, we are looking for 6-10 teams of 3-5 members each working on specific projects related to areas of rationality, epistemics, group rationality, and civilizational sanity, and delivering tangible outcomes. A residency project can be: Research on a relevant topic (examples of what we would be excited about are broad in some directions and include abstract foundations like geometric rationality or Modal Fixpoint Cooperation without Löb's Theorem, research and development of applied rationality techniques like Internal communication framework, research on the use of AI to improve human rationality like "automated Double-Crux aid" and more); Distillation, communication, and publishing (writing and publishing a series of explanatory posts, video production, writing a textbook or course materials, etc.); Program development (events, workshops, etc.); Anything else that will provide value to this space. Teams will have the option to apply to work on a specific topic (to be announced soon) or propose their own project. The selected teams will work on their projects in person at FIXED ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Announcing the Prague community space: Fixed Point, published by Epistea on May 22, 2023 on The Effective Altruism Forum. Summary A coworking and event space in Prague is open for the entirety of 2023, offering up to 50 desk spaces in a coworking office, a large event space for up to 60 people, multiple meeting rooms and other amenities. We are currently in the process of transitioning from an initial subsidized model to a more sustainable paid membership model. In 2022, Fixed Point was the home of the Prague Fall Season which will be returning there in 2023. We are seeking people, projects and hosts of events here in 2023. If you are interested you can apply here. What is Fixed Point? Fixed Point is a unique community and coworking space located in the heart of Prague operated by Epistea. We support organizations and individuals working on existential security, epistemics, rationality, and effective altruism. Across five floors there is a variety of coworking offices offering up to 50 workstations, as well as numerous meeting rooms and private call stations. In addition to functional work areas, there are inviting communal spaces such as a large comfortable common room accommodating up to 60 people, two fully equipped large kitchens, and a spacious dining area. These amenities create a welcoming environment that encourages social interaction and facilitates spontaneous exchange of ideas. Additionally, there are on-site amenities like a small gym, a nap room, two laundry rooms, bathrooms with showers, and a garden with outdoor tables and seating. For those in need of short-term accommodation, our on-site guesthouse has a capacity of up to 10 beds. Fixed Point is a space where brilliant and highly engaged individuals make crucial career decisions, establish significant relationships, and find opportunities for introspection among like-minded peers when they need it most. In 2022, Fixed Point was home to the Prague Fall Season, when 350 people visited the space. The name "Fixed Point" draws inspiration from the prevalence of various Fixed Point theorems in almost all areas people working in the space work on. If you study the areas seriously, you will find fixed points sooner or later. Why Prague? The Czech effective altruism and rationalist community has long been committed to operational excellence and the creation of physical spaces that facilitate collaboration. With numerous successfully incubated organizations and passionate individuals making a difference in high-impact domains, Prague is now a viable option, especially for EU citizens wanting to settle in continental Europe. In addition to the Prague Fall Season, Prague is home to many different projects, such as Alignment of Complex Systems Research Group, ESPR or Czech Priorities. We host the European runs of CFAR workshops and CFAR rEUnions. Whom is it for? We extend a warm welcome to both short and long-term visitors working on meaningful projects in the areas of existential risk mitigation, AI safety, rationality, epistemics, and effective altruism. We are particularly excited to accommodate individuals and teams in the following categories: Those interested in hosting events, Teams seeking a workspace for an extended period of time. Here are a few examples of the projects we are equipped to host and are enthusiastic about: Weekend hackathons, Incubators lasting up to several months, Conferences, Workshops and lectures on relevant topics, Providing office spaces for existing projects. Additional support In addition to the amenities, we can also offer the following services upon request: Project management for existing initiatives, Catering services for events, Administrative and operations support Accommodation arrangements, Event logistics and operations assistance, Event design consulting. Feel free to r...

Click here for the full shownotesToday, we have the pleasure of introducing Jeff Rolander, the Vice President of Claims at Faye, a leading travel insurance disruptor. With over three decades of experience in the insurance industry, Jeff is an expert in providing top-notch coverage and care to customers when something goes awry. In his current role at Faye, Jeff oversees a team of professionals dedicated to ensuring that travelers are well taken care of in case of unforeseen circumstances. Before joining Faye, Jeff led a team of 250 associates at Allianz Partners USA, where he oversaw Claims and Emergency Assistance, helping customers with their claims and emergency situations while traveling.In this episode we discuss:What is trip cancellation insurance - what does it cover and what doesn't it cover?What is CFAR insurance? What does it cover and what doesn't it cover?So what are the main differences between the two?Who should consider CFAR insurance - given the significant extra cost?Who is eligible and who is not? Can you buy CFAR  on its own?Can you buy CFAR whenever you want?What about trips that have multiple payments?What does it cover? What doesn't it cover?Check out our website | Check out all the podcast show notes | Follow us on Instagram

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: LessWrong Community Weekend 2023 – Applications Open, published by Henry Prowbell on May 1, 2023 on LessWrong. Previous attendees can skip to the application form and note that we're processing applications in 3 batches this year. If you need to know quickly whether you're being offered a place, apply by May 15th to be included in batch 1. September 22-25th is the 10th annual LessWrong Community Weekend (LWCW). This is Europe's largest rationalist social gathering which brings together 150 aspiring rationalists from across Europe and beyond for four days of socialising, fun and intellectual exploration. The majority of the content will be unconference style and participant driven. On Friday afternoon we put up six wall-sized daily planners and by Saturday morning the attendees fill them up with 100+ workshops, talks and activities of their own devising. Previous years' schedules have included. Double Cruxing Hamming Circles Gendlin Focusing Applied Rationality workshops by CFAR instructors and instructors-in-training Circling Authentic Relating games Improv theatre Introduction to stand up comedy Writing rationalist fiction Dance workshops Acapella singing Icebreaker games Lightning talks Celebrating failure groups Giant outdoor chess Penultima Dungeons & Dragons Kung Fu basics Board games Breathwork workshops Ecstatic dancing Radical Honesty workshops Playfighting for adults Polyamory and relationships workshops Sex Q&A roundtable Quantified self workshops Moral philosophy debates AI safety Q&A How to handle fear of AI Doom Value drift in EA The neurobiology of psychedelics The science of longevity Morning runs and yoga Meditation in the rooftop winter garden Night time swimming Ted Chiang and Greg Egan bedtime story readings If things like ecstatic dancing, radical honesty and polyamory workshops sound too intense for you, rest assured everything is optional. (I'm British and very awkward so a lot of this stuff terrifies me.) The event takes place in the natural environs of Lake Wannsee on the outskirts of Berlin. So you can spend some time recharging in between making new friends by hiking in the forests, sunbathing or swimming in the lake. For those who want to extend their stay in Berlin, there will very likely be meetups arranged in the days before and after the event too. LWCW is LGBTQIA+ friendly, people are welcome to bring their children (potentially there will be a professional childminder provided throughout) and this year we're putting extra effort into creating an event where people of all ages, genders, backgrounds and adjacent interests (EA, circling, philosophy, meditation.) feel at home. This event has a special place in my heart and I truly think there's nothing else quite like it. It's where I've made friends who have more in common with me than I knew was possible and it's where I've been introduced to ideas that have altered the course of my life – which is something I never truly got from the online version of LessWrong. Essential Information When: Friday 22nd September - Monday 25th September 2023 Where: jh-wannsee.de (Berlin) Prices: Nobody makes any money from this event and the organiser team are unpaid. All your money goes into paying for the venue, food, equipment and other expenses. Regular ticket: €200 Supporter ticket: €250/300/400 If you want to attend but the ticket or travel cost is the only thing holding you back send us a message briefly explaining your situation. We have a small fund set aside for people who require financial aid. Apply here: http://tiny.cc/LWCW2023signup Contact: If you have any questions post them in the comments section below or email lwcw.europe[at]gmail.com Schedule Friday lunch: Meet in central Berlin at lunchtime for covid tests and vegan food followed by a short bus journey to JH Wannsee (all included in ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Why Simulator AIs want to be Active Inference AIs, published by Jan Kulveit on April 10, 2023 on LessWrong. Prelude: when GPT first hears its own voice Imagine humans in Plato's cave, interacting with reality by watching the shadows on the wall. Now imagine a second cave, further away from the real world. GPT trained on text is in the second cave. The only way it can learn about the real world is by listening to the conversations of the humans in the first cave, and predicting the next word. Now imagine that more and more of the conversations GPT overhears in the first cave mention GPT. In fact, more and more of the conversations are actually written by GPT. As GPT listens to the echoes of its own words, might it start to notice “wait, that's me speaking”? Given that GPT already learns to model a lot about humans and reality from listening to the conversations in the first cave, it seems reasonable to expect that it will also learn to model itself. This post unpacks how this might happen, by translating the Simulators frame into the language of predictive processing, and arguing that there is an emergent control loop between the generative world model inside of GPT and the external world. Simulators as (predictive processing) generative models There's a lot of overlap between the concept of simulators and the concept of generative world models in predictive processing. Actually, in my view, it's hard to find any deep conceptual difference - simulators broadly are generative models. This is also true about another isomorphic frame - predictive models as described by Evan Hubinger. The predictive processing frame tends to add some understanding of how generative models can be learned by brains and what the results look like in the real world, and the usual central example is the brain. The simulators frame typically adds a connection to GPT-like models, and the usual central example is LLMs. In terms of the space of maps and the space of systems, we have a situation like this:The two maps are partially overlapping, even though they were originally created to understand different systems. They also have some non-overlapping parts. What's in the overlap: Systems are equipped with a generative model that is able to simulate the system's sensory inputs. The generative model is updated using approximate Bayesian inference. Both frames give you similar phenomenological capabilities: for example, what CFAR's "inner simulator" technique is doing is literally and explicitly conditioning your brain-based generative model on a given observation and generating rollouts. Given the conceptual similarity but terminological differences, perhaps it's useful to create a translation table between the maps: Simulators terminologyPredictive processing terminologySimulator Generative modelPredictive loss on a self-supervised datasetMinimization of predictive errorSelf-supervisedSelf-supervised, but often this is omittedIncentive to reverse-engineer the (semantic) physics of the training distributionLearns a robust world-modelSimulacrumNext token in training dataSensory input Generative model of self Generative model of someone else Generative model of . To show how these terminological differences play out in practice, I'm going to take the part of Simulators describing GPT's properties, and unpack each of the properties in the kind of language that's typically used in predictive processing papers. Often my gloss will be about human brains in particular, as the predictive processing literature is most centrally concerned with that example; but it's worth reiterating that I think that both GPT and what parts of human brain do are examples of generative models, and I think that the things I say about the brain below can be directly applied to artificial generative models. “Self-supervised: Tr...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Why Simulator AIs want to be Active Inference AIs, published by Jan Kulveit on April 10, 2023 on The AI Alignment Forum. Prelude: when GPT first hears its own voice Imagine humans in Plato's cave, interacting with reality by watching the shadows on the wall. Now imagine a second cave, further away from the real world. GPT trained on text is in the second cave. The only way it can learn about the real world is by listening to the conversations of the humans in the first cave, and predicting the next word. Now imagine that more and more of the conversations GPT overhears in the first cave mention GPT. In fact, more and more of the conversations are actually written by GPT. As GPT listens to the echoes of its own words, might it start to notice “wait, that's me speaking”? Given that GPT already learns to model a lot about humans and reality from listening to the conversations in the first cave, it seems reasonable to expect that it will also learn to model itself. This post unpacks how this might happen, by translating the Simulators frame into the language of predictive processing, and arguing that there is an emergent control loop between the generative world model inside of GPT and the external world. Simulators as (predictive processing) generative models There's a lot of overlap between the concept of simulators and the concept of generative world models in predictive processing. Actually, in my view, it's hard to find any deep conceptual difference - simulators broadly are generative models. This is also true about another isomorphic frame - predictive models as described by Evan Hubinger. The predictive processing frame tends to add some understanding of how generative models can be learned by brains and what the results look like in the real world, and the usual central example is the brain. The simulators frame typically adds a connection to GPT-like models, and the usual central example is LLMs. In terms of the space of maps and the space of systems, we have a situation like this:The two maps are partially overlapping, even though they were originally created to understand different systems. They also have some non-overlapping parts. What's in the overlap: Systems are equipped with a generative model that is able to simulate the system's sensory inputs. The generative model is updated using approximate Bayesian inference. Both frames give you similar phenomenological capabilities: for example, what CFAR's "inner simulator" technique is doing is literally and explicitly conditioning your brain-based generative model on a given observation and generating rollouts. Given the conceptual similarity but terminological differences, perhaps it's useful to create a translation table between the maps: Simulators terminologyPredictive processing terminologySimulator Generative modelPredictive loss on a self-supervised datasetMinimization of predictive errorSelf-supervisedSelf-supervised, but often this is omittedIncentive to reverse-engineer the (semantic) physics of the training distributionLearns a robust world-modelSimulacrumNext token in training dataSensory input Generative model of self Generative model of someone else Generative model of . To show how these terminological differences play out in practice, I'm going to take the part of Simulators describing GPT's properties, and unpack each of the properties in the kind of language that's typically used in predictive processing papers. Often my gloss will be about human brains in particular, as the predictive processing literature is most centrally concerned with that example; but it's worth reiterating that I think that both GPT and what parts of human brain do are examples of generative models, and I think that the things I say about the brain below can be directly applied to artificial generative models. “Self-s...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: All images from the WaitButWhy sequence on AI, published by trevor on April 8, 2023 on LessWrong. Lots of people seem to like visual learning. I don't see much of an issue with that. People who have fun with thinking tend to get more bang for their buck. It seems reasonable to think that that Janus's image of neural network shoggoths makes it substantially easier for a lot of people to fully operationalize the concept that RLHF could steer humanity off of a cliff: Lots of people I've met said that they were really glad that they encountered Tim Urban's WaitButWhy blog post on AI back in 2015, which was largely just a really good distillation of Nick Bostrom's Superintelligence (2014). It's a rather long (but well-written) post, so what impressed me was not the distillation, but the images. The images in the post were very vivid, especially in the middle. It seems to me like images can work as a significant thought aid, by leaning on visual memory to aid recall, and/or to make core concepts more cognitively available during the thought process in general. But also, almost by themselves, the images do a pretty great job describing the core concepts of AI risk, as well as the general gist of the entirety of Tim Urban's sequence. Considering that he managed to get that result, even though the post itself is >22,000 words (around as long as the entire CFAR handbook), maybe Tim Urban was simultaneously doing something very wrong and very right with writing the distillation; could he have turned a 2-hour post into a 2-minute post by just doubling the number of images? If there was a true-optimal blog post to explain AI safety for the first time, to an otherwise-uninterested layperson (a very serious matter in AI governance), it wouldn't be surprising to me if that true-optimal blog post contained a lot of images. Walls of text are inevitable at some point or another, but there's the old saying that a picture's worth a thousand words. Under current circumstances, it makes sense for critical AI safety concepts to be easier and less burdensome to think and learn about for the first time, rather than harder and more burdensome to think about for the first time. I've found the above two pictures particularly helpful, for doing an excellent job depicting the scope. Out of all the images that could be used to help describe AGI to someone for the first time, I would pick those two. These two images are pretty good as well, as a primer for the current state of affairs. Obviously, they were drawn in 2015 and need to be redrawn. This image puts the significance of the situation into context. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

We're trying a new format, inspired by Acquired.fm! No guests, no news, just highly prepared, in-depth conversation on one topic that will level up your understanding. We aren't experts, we are learning in public. Please let us know what we got wrong and what you think of this new format!When you ask someone to break down the basic ingredients of a Large Language Model, you'll often hear a few things: You need lots of data. You need lots of compute. You need models with billions of parameters. Trust the Bitter Lesson, more more more, scale is all you need. Right?Nobody ever mentions the subtle influence of great benchmarking.LLM Benchmarks mark our progress in building artificial intelligences, progressing from * knowing what words go with others (1985 WordNet)* recognizing names and entities (2004 Enron Emails) * and image of numbers, letters, and clothes (1998-2017 MNIST)* language translation (2002 BLEU → 2020 XTREME)* more and more images (2009 ImageNet, CIFAR)* reasoning in sentences (2016 LAMBADA) and paragraphs (2019 AI2RC, DROP)* stringing together whole sentences (2018 GLUE and SuperGLUE)* question answering (2019 CoQA)* having common sense (2018 Swag and HellaSwag, 2019 WinoGrande)* knowledge of all human tasks and professional exams (2021 MMLU)* knowing everything (2022 BIG-Bench)People who make benchmarks are the unsung heroes of LLM research, because they dream up ever harder tests that last ever shorter periods of time.In our first AI Fundamentals episode, we take a trek through history to try to explain what we have learned about LLM Benchmarking, and what issues we have discovered with them. There are way, way too many links and references to include in this email. You can follow along the work we did for our show prep in this podcast's accompanying repo, with all papers and selected tests pulled out.Enjoy and please let us know what other fundamentals topics you'd like us to cover!Timestamps* [00:00:21] Benchmarking Questions* [00:03:08] Why AI Benchmarks matter* [00:06:02] Introducing Benchmark Metrics* [00:08:14] Benchmarking Methodology* [00:09:45] 1985-1989: WordNet and Entailment* [00:12:44] 1998-2004 Enron Emails and MNIST* [00:14:35] 2009-14: ImageNet, CIFAR and the AlexNet Moment for Deep Learning* [00:17:42] 2018-19: GLUE and SuperGLUE - Single Sentence, Similarity and Paraphrase, Inference* [00:23:21] 2018-19: Swag and HellaSwag - Common Sense Inference* [00:26:07] Aside: How to Design Benchmarks* [00:26:51] 2021: MMLU - Human level Professional Knowledge* [00:29:39] 2021: HumanEval - Code Generation* [00:31:51] 2020: XTREME - Multilingual Benchmarks* [00:35:14] 2022: BIG-Bench - The Biggest of the Benches* [00:37:40] EDIT: Why BIG-Bench is missing from GPT4 Results* [00:38:25] Issue: GPT4 vs the mystery of the AMC10/12* [00:40:28] Issue: Data Contamination* [00:42:13] Other Issues: Benchmark Data Quality and the Iris data set* [00:45:44] Tradeoffs of Latency, Inference Cost, Throughput* [00:49:45] ConclusionTranscript[00:00:00] Hey everyone. Welcome to the Latent Space Podcast. This is Alessio, partner and CTO and residence at Decibel Partners, and I'm joined by my co-host, swyx writer and editor of Latent Space.[00:00:21] Benchmarking Questions[00:00:21] Up until today, we never verified that we're actually humans to you guys. So we'd have one good thing to do today would be run ourselves through some AI benchmarks and see if we are humans.[00:00:31] Indeed. So, since I got you here, Sean, I'll start with one of the classic benchmark questions, which is what movie does this emoji describe? The emoji set is little Kid Bluefish yellow, bluefish orange Puffer fish. One movie does that. I think if you added an octopus, it would be slightly easier. But I prepped this question so I know it's finding Nemo.[00:00:57] You are so far a human. Second one of these emoji questions instead, depicts a superhero man, a superwoman, three little kids, one of them, which is a toddler. So you got this one too? Yeah. It's one of my favorite movies ever. It's the Incredibles. Uh, second one was kind of a letdown, but the first is a.[00:01:17] Awesome. Okay, I'm gonna ramp it up a little bit. So let's ask something that involves a little bit of world knowledge. So when you drop a ball from rest, it accelerates downward at 9.8 meters per second if you throw it downward instead, assuming no air resistance, so you're throwing it down instead of dropping it, it's acceleration immediately after leaving your hand is a 9.8 meters per second.[00:01:38] B, more than 9.8 meters per second. C less than 9.8 meters per second. D cannot say unless the speed of the throw is. I would say B, you know, I started as a physics major and then I changed, but I think I, I got enough from my first year. That is B Yeah. Even proven that you're human cuz you got it wrong.[00:01:56] Whereas the AI got it right is 9.8 meters per second. The gravitational constant, uh, because you are no longer accelerating after you leave the hand. The question says if you throw it downward after leaving your hand, what is the. It is, it goes back to the gravitational constant, which is 9.8 meters per, I thought you said you were a physics major.[00:02:17] That's why I changed. So I'm a human. I'm a human. You're human. You're human. But you, you got them all right. So I can't ramp it up. I can't ramp it up. So, Assuming, uh, the AI got all of that right, you would think that AI will get this one wrong. Mm-hmm. Because it's just predicting the next token, right?[00:02:31] Right. In the complex Z plane, the set of points satisfying the equation. Z squared equals modulars. Z squared is A, a pair points B circle, C, a half line D, online D square. The processing is, this is going on in your head. You got minus three. A line. This is hard. Yes, that is. That is a line. Okay. What's funny is that I think if, if an AI was doing this, it would take the same exact amount of time to answer this as it would every single other word.[00:03:05] Cuz it's computationally the same to them. Right.[00:03:08] Why AI Benchmarks matter[00:03:08] Um, so anyway, if you haven't caught on today, we're doing our first, uh, AI fundamentals episode, which just the two of us, no guess because we wanted to go deep on one topic and the topic. AI benchmarks. So why are we focusing on AI benchmarks? So, GPT4 just came out last week and every time a new model comes out, All we hear about is it's so much better than the previous model on benchmark X, on benchmark Y.[00:03:33] It performs better on this, better on that. But most people don't actually know what actually goes on under these benchmarks. So we thought it would be helpful for people to put these things in context. And also benchmarks evolved. Like the more the models improve, the harder the benchmarks get. Like I couldn't even get one of the questions right.[00:03:52] So obviously they're working and you'll see that. From the 1990s where some of the first ones came out to day, the, the difficulty of them is truly skyrocketed. So we wanna give a, a brief history of that and leave you with a mental model on, okay, what does it really mean to do well at X benchmark versus Y benchmark?[00:04:13] Um, so excited to add that in. I would also say when you ask people what are the ingredients going into a large language model, they'll talk to you about the data. They'll talk to you about the neural nets, they'll talk to you about the amount of compute, you know, how many GPUs are getting burned based on this.[00:04:30] They never talk to you about the benchmarks. And it's actually a shame because they're so influential. Like that is the entirety of how we judge whether a language model is better than the other. Cuz a language model can do anything out of. Potentially infinite capabilities. How do you judge one model versus another?[00:04:48] How do you know you're getting better? And so I think it's an area of intense specialization. Also, I think when. Individuals like us, you know, we sort of play with the language models. We are basically doing benchmarks. We're saying, look, it's, it's doing this awesome thing that I found. Guess what? There have been academics studying this for 20 years who have, uh, developed a science to this, and we can actually benefit from studying what they have done.[00:05:10] Yep. And obviously the benchmarks also drive research, you know, in a way whenever you're working on, in a new model. Yeah. The benchmark kind of constraints what you're optimizing for in a way. Because if you've read a paper and it performs worse than all the other models, like you're not gonna publish it.[00:05:27] Yeah. So in a way, there's bias in the benchmark itself. Yeah. Yeah. We'll talk a little bit about that. Right. Are we optimizing for the right things when we over-optimize for a single benchmark over over some others? And also curiously, when GPT4 was released, they emitted some very. Commonplace industry benchmarks.[00:05:44] So the way that you present yourself, it is a form of marketing. It is a form of trying to say you're better than something else. And, and trying to explain where you think you, you do better. But it's very hard to verify as well because there are certain problems with reproducing benchmarks, uh, especially when you come to large language models.[00:06:02] Introducing Benchmark Metrics[00:06:02] So where do we go from here? Should we go over the, the major concept? Yeah. When it comes to benchmark metrics, we get three main measures. Accuracy, precision, recall accuracy is just looking at how many successful prediction the model does. Precision is the ratio of true positives, meaning how many of them are good compared to the overall amount of predictions made Versus recall is what proportion of the positives were identified.[00:06:31] So if you think. Spotify playlist to maybe make it a little more approachable, precision is looking. How many songs in a Spotify playlist did you like versus recall is looking at of all the Spotify songs that you like in the word, how many of them were put in the in the playlist? So it's more looking at how many of the true positives can you actually bring into the model versus like more focusing on just being right.[00:06:57] And the two things are precision and recall are usually in tension.. If you're looking for a higher position, you wanna have a higher percentage of correct results. You're usually bringing recall down because you lead to kind of like lower response sets, you know, so there's always trade offs. And this is a big part of the benchmarking too.[00:07:20] You know, what do you wanna optimize for? And most benchmarks use this, um, F1 score, which is the harmonic mean of precision and recall. Which is, you know, we'll put it in the show notes, but just like two times, like the, you know, precision Times Recall divided by the sum. So that's one. And then you get the Stanford Helm metrics.[00:07:38] Um, yeah, so ultimately I think we have advanced a lot in the, in the past few decades on how we measure language models. And the most interesting one came out January of this year from Percy Lang's research lab at Stanford, and he's got. A few metrics, accuracy, calibration, robustness, fairness, efficiency, general information bias and toxicity, and caring that your language models are not toxic and not biased.[00:08:03] So is is, mm-hmm. Kind of a new thing because we have solved the other stuff, therefore we get to care about the toxic of, uh, the language models yelling at us.[00:08:14] Benchmarking Methodology[00:08:14] But yeah, I mean, maybe we can also talk about the other forms of how their be. Yeah, there's three main modes. You can need a benchmark model in a zero shot fashion, few shot or fine tune models, zero shots.[00:08:27] You do not provide any example and you're just testing how good the model is at generalizing few shots, you have a couple examples that you provide and then. You see from there how good the model is. These are the number of examples usually represented with a K, so you might see few shots, K equal five, it means five examples were passed, and then fine tune is you actually take a bunch of data and fine tune the model for that specific task, and then you test it.[00:08:55] These all go from the least amount of work required to the most amount of work required. If you're doing zero shots benchmarking, you do not need to have any data, so you can just take 'em out and do. If you're fine tuning it, you actually need a lot of data and a lot of compute time. You're expecting to see much better results from there.[00:09:14] Yeah. And sometimes the number of shots can go up to like a hundred, which is pretty surprising for me to see that people are willing to test these language models that far. But why not? You just run the computer a little bit longer. Yeah. Uh, what's next? Should we go into history and then benchmarks? Yeah.[00:09:29] History of Benchmarking since 1985[00:09:29] Okay, so I was up all night yesterday. I was like, this is a fascinating topic. And I was like, all right, I'll just do whatever's in the G PT three paper. And then I read those papers and they all cited previous papers, and I went back and back and back all the way to 1985. The very first benchmark that I can find.[00:09:45] 1985-1989: WordNet and Entailment[00:09:45] Which is WordNet, which is uh, an English benchmark created in at Princeton University by George Miller and Christian Fellbaum. Uh, so fun fact, Chris George Miller also authored the paper, the Magical Number seven plus Minus two, which is the observation that people have a short term memory of about seven for things.[00:10:04] If you have plus or minus two of seven, that's about all you can sort of remember in the short term, and I just wanted. Say like, this was before computers, right? 1985. This was before any of these personal computers were around. I just wanna give people a sense of how much work manual work was being done by these people.[00:10:22] The database, uh, WordNet. Sorry. The WordNet database contains 155,000 words organized in 175,000 sys. These sys are basically just pairings of nouns and verbs and adjectives and adverbs that go together. So in other words, for example, if you have nouns that are hyper names, if every X is a, is a kind of Y.[00:10:44] So a canine is a hyper name of a dog. It's a holo. If X is a part of Y, so a building is a hollow name of a window. The most interesting one for in terms of formal, uh, linguistic logic is entailment, which captures the relationship between two words, where the verb Y is entailed by X. So if by doing X, you must be doing Y.[00:11:02] So in other words, two, sleep is entailed by two snore because you cannot snore without also sleeping and manually mapping 155,000 words like that, the relationships between all of them in a, in a nested tree, which is. Incredible to me. Mm-hmm. And people just did that on faith. They were like, this will be useful somehow.[00:11:21] Right. Uh, and they were interested in cycle linguistics, like understanding how humans thought, but then it turned out that this was a very good dataset for understanding semantic similarity, right? Mm-hmm. Like if you measure the distance between two words by traversing up and down the graph, you can find how similar to two words are, and therefore, Try to figure out like how close they are and trade a model to, to predict that sentiment analysis.[00:11:42] You can, you can see how far something is from something that is considered a good sentiment or a bad sentiment or machine translation from one language to the other. Uh, they're not 200 word languages, which is just amazing. Like people had to do this without computers. Penn Tree Bank, I was in 1989, I went to Penn, so I always give a shout out to my university.[00:12:01] This one expanded to 4.5 million words of text, which is every uh, wall Street Journal. For three years, hand collected, hand labeled by grad students your tuition dollars at work. So I'm gonna skip forward from the eighties to the nineties. Uh, NYS was the most famous data set that came out of this. So this is the, uh, data set of 60,000.[00:12:25] Training images of, uh, of numbers. And this was the first visual dataset where, uh, people were tr tracking like, you know, handwritten numbers and, and mapping them to digital numbers and seeing what the error rate for them was. Uh, these days I think this can be trained in like e every Hello world for machine learning is just train missed in like four lanes of code.[00:12:44] 1998-2004 Enron Emails and MNIST[00:12:44] Then we have the Enron email data set. Enron failed in 2001. Uh, the emails were released in 2004 and they've been upgraded every, uh, every few years since then. That is 600,000 emails by 150 senior employees of Enron, which is really interesting because these are email people emailing each other back and forth in a very natural.[00:13:01] Context not knowing they're being, they're about to be observed, so you can do things like email classification, email summarization, entity recognition and language modeling, which is super cool. Any thoughts about that be before we go into the two thousands? I think like in a way that kind of puts you back to the bias, you know, in some of these benchmarks, in some of these data sets.[00:13:21] You know, like if your main corpus of benchmarking for entity recognition is a public energy company. Mm-hmm. You know, like if you're building something completely different and you're building a model for that, maybe it'll be worse. You know, you start to see how we started. With kind of like, WordNet is just like human linguistics, you know?[00:13:43] Yes. It's not domain related. And then, um, same with, you know, but now we're starting to get into more and more domain-specific benchmarks and you'll see this increase over time. Yeah. NY itself was very biased towards, um, training on handwritten letter. Uh, and handwritten numbers. So, um, in 2017 they actually extended it to Eist, which is an extended to extension to handwritten letters that seems very natural.[00:14:08] And then 2017, they also had fashion ness, which is a very popular data set, which is images of clothing items pulled from Zando. So you can see the capabilities of computer vision growing from single digit, 0, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, to all the letters of the alphabet. To now we can recognize images, uh, of fashion, clothing items.[00:14:28] So it's pretty. So the big one for deep learning, cuz all of that was just, just the appetizers, just getting started.[00:14:35] 2009-2014 : ImageNet, CIFAR and the AlexNet Moment for Deep Learning[00:14:35] The big one for deep learning was ImageNet, which is where Fafa Lee came into the picture and that's why she's super well known. She started working in 2006 and released it in 2009. Fun fact, she actually met with, uh, Christian Feldbaum, who was, uh, one of the co-authors of, uh, war.[00:14:51] To create ImageNet. So there's a direct lineage from Words to Images. Yeah. And uh, they use Amazon Mechanical Turk to help with classification images. No longer grad students. But again, like I think, uh, this goes, kind of goes back to your observation about bias, like when I am a mechanical Turk worker. And I'm being paid by the image to classify an image.[00:15:10] Do you think I'll be very careful at my job? Right? Yeah. Whereas when I'm a, you know, Enron employee, emailing my, my fellow coworker, trying to just communicate something of, of natural language that is a different type of, uh, environment. Mm-hmm. So it's a pretty interesting benchmark. So it was released in 2009 ish and, you know, people were sort of competing to recognize and classify that properly.[00:15:33] The magic moment for ImageNet came in 2012, uh, which is called the AlexNet moment cuz I think that grad student that, um, created this recognition model was, uh, named Alex, I forget his last name, achieved a error rate of 15%, which is, More than 10% lower than the runner up. So it was used just so much better than the second place that everyone else was like, what are you doing?[00:15:54] Uh, and it turned out that he was, he was the first to use, uh, deep learning, uh, c n n 10 percentage points. So like 15 and the other one was 25. Yeah, exactly. So it was just so much, so much better than the others. It was just unbelievable that no one else was, no other approach was even coming close.[00:16:09] Therefore, everyone from there on out for the next, until today we're just learning the lessons of deep learning because, um, it is so much superior to the other approaches. And this was like a big. Images and visual moment because then you had like a sci-fi 10, which is a, another, like a data set that is mostly images.[00:16:27] Mm-hmm. Focused. Mm-hmm. So it took a little bit before we got back to to text. And nowadays it feels like text, you know, text models are kind of eating the word, you know, we're making the text one multi-model. Yeah. So like we're bringing the images to GBT four instead of the opposite. But yeah, in 2009 we had a, another 60,000 images that set.[00:16:46] 32 by 32. Color images with airplanes, automobiles, like, uh, animals, like all kind of stuff. Like I, I think before we had the numbers, then we had the handwritten letters. Then we had clothing, and then we finally made clothing items came after, oh, clothing items. 2009. Yeah, this is 2009. I skipped, I skipped time a little bit.[00:17:08] Yeah, yeah. But yeah, CFR 10 and CFR 100. CFR 10 was for 10 classes. And that that was chosen. And then obviously they optimized that and they were like, all right, we need a new problem now. So in 20 14, 5 years later, they introduced CFAR 100, which was a hundred classes of other items. And I think this is a very general pattern, which is used.[00:17:25] You create a data set for a specific be. You think it's too hard for machines? Mm-hmm. It lasts for five years before it's no longer too hard for machines, and you have to find a new data set and you have to extend it again. So it's Similarly, we are gonna find that in glue, which is another, which is one of more modern data sets.[00:17:42] 2018-19: GLUE and SuperGLUE - Single Sentence, Similarity and Paraphrase, Inference[00:17:42] This one came out in 2018. Glue stands for general Language Understanding Evaluation. This is one of the most influential, I think, early. Earlier, um, language model benchmarks, and it has nine tasks. Um, so it has single sentence tasks, similarity and paraphrase tasks and inference tasks. So a single sentence task, uh, would be something like, uh, the Stanford Sentiment Tree Bank, which is a.[00:18:05] Uh, sentences from movie reviews and human annotations of the sentiment, whether it's positive or negative, in a sort of like a four point scale. And your job is to predict the task of a single sentence. This similarity task would involve corpuses, like the Microsoft research paraphrase corpus. So it's a corpus of sentence pairs automatically extracted from online news sources with human annotations for whether or not the sentence is in the para semantically equivalent.[00:18:28] So you just predict true or false and again, Just to call back to the math that we did earlier in this episode, the classes here are imbalance. This data set, for example, is 68% positive. So we report both accuracy and F1 scores. F1 is a more balanced approach because it, it adjusts for, uh, imbalanced, um, data sets.[00:18:48] Mm-hmm. Yeah. And then finally, inference. Inference is the one where we really start to have some kind of logic. So for example, the M N L I. Um, actually I'm, I'm gonna focus on squad, the Stanford questioning question answering dataset. It's another data set of pairs, uh, questions, uh, uh, p question paragraphs, pairs.[00:19:04] So where one of the sentences of the paragraph drawn from Wikipedia contains the answer to the corresponding question, we convert the task into a sentence, para classification by forming a pair between each question in each sentence into corresponding context and filtering out pairs of low overlap. So basically annotating whether or not.[00:19:20] Is the answer to the question inside of this paragraph that I pulled. Can you identify that? And again, like Entailment is kind of included inside of each of these inference tasks because it starts to force the language model to understand whether or not one thing implies the other thing. Mm-hmm. Yeah.[00:19:37] And the, the models evolving. This came out in 2018, lasted one year exactly. One year later, people were like, that's too easy. That's too easy. So in 2019, they actually came out with super. I love how you'll see later with like swag and hella swag. It's like they come up with very good names for these things.[00:19:55] Basically what's super glue dead is stick glue and try and move outside of the single sentence evaluation. So most of the tasks that. Sean was talking about focus on one sentence. Yeah, one sentence, one question. It's pretty straightforward in that way. Superglue kind of at the, so one, it went from single sentence to having some multi sentence and kind of like a context driven thing.[00:20:21] So you might have questions where, The answer is not in the last paragraph that you've read. So it starts to test the, the context window on this model. Some of them are more, in order to know the answer, you need to know what's not in the question kind of thing. So like you may say, Hey, this drink is owned by the Coca-Cola company.[00:20:43] Is this a Pepsi product? You know, so you need to make the connection false. Exactly, yeah. Then you have also like, um, embedded clauses. So you have things that are not exactly said, have to be inferred, and like a lot of this stack is very conversational. So some of the example contain a lot of the, um, um, you know, or this question's very hard to read out.[00:21:07] Yeah, I know. It's like, it sounds like you are saying, um, but no, you're actually, you're actually. And yet I hope to see employer base, you know, helping out child, um, care centers at the place of employment, things like that, that will help out. It's kind of hard to even read it. And then the hypothesis is like they're setting a trend.[00:21:27] It's going from something very simple like a big p d extract to something that is more similar to how humans communicate. Transcripts, like audio transcripts. Exactly. Of how people talk. Yeah. And some of them are also, Plausibility. You know, like most of these models have started to get good at understanding like a clear cause, kind of like a.[00:21:48] You know, cause effect things. But some of the plausible ones are like, for example, this one is a copa. They're called choice of plausible alternatives. The premises, my body cast a shadow over the grass. What's the cost for this alternative? One, the sun was rising. Alternative to the grass was cut.[00:22:07] Obviously it's the sun was rising, but nowhere. In the question we're actually mentioning the sun, uh, we are mentioning the grass. So some models, some of the older models might see the grass and make the connection that the grass is part of the reason, but the models start to get better and better and go from simply looking at the single sentence context to a more of a, a word new, uh, word knowledge.[00:22:27] It's just really impressive, like the fact that. We can expect that out of a model. It still blows my mind. I think we should not take it for granted that when we're evaluating models, we're asking questions like this that is not obvious from just the given text itself. Mm-hmm. So it, it is just coming with a memorized view of the world, uh, or, or world knowledge. And it understands the premise on, on some form. It is not just random noise. Yeah, I know. It's really impressive. This one, I actually wanted multi rc I actually wanted to spring on you as a, as a test, but it's just too long to read. It's just like a very long logic question.[00:23:03] And then it'll ask you to do, uh, comprehension. But uh, yeah, we'll just, we'll just kinda skip that. We'll put it, we'll put it in the show notes, and then you have to prove us that you're a human. Send us the answer exactly. Exactly and subscribe to the podcast. So superglue was a lot harder, and I think also was superseded eventually, pretty soon.[00:23:21] 2018-2019: Swag and HellaSwag - Common Sense Inference[00:23:21] And, uh, yeah, then we started coming onto the more recent cohort of tests. I don't know how to introduce the rest. Uh, there, there are just so many tests here that I, I struggle a little bit picking from these. Uh, but perhaps we can talk about swag and heli swyx since you mentioned it. Yeah. So SWAG stands for situations with Adversarial Generations.[00:23:39] Uh, also came out in 2018, but this guy, zes Etal, likes to name his data sets and his benchmarks in a very memorable way. And if you look at the PDF of the paper, he also has a little icon, uh, image icon for swag. And he doesn't just go by, uh, regular language. So he definitely has a little bit of branding to this and it's.[00:24:00] Part. So I'll give you an example of the kind of problems that swyx poses. Uh, it it is focused on common sense inference. So what's common sense inference? So, for example, given a partial description, like she opened the hood of the car, humans can reason about the situation and anticipate what might come next.[00:24:16] Then she examined the engine. So you're supposed to pick based on what happened in the first part. What is most likely to happen in the second part based on the, uh, multiple choice question, right? Another example would be on stage, a woman takes a seat at the piano. She a, sits on a bench as her sister plays at the doll.[00:24:33] B. Smiles with someone as the music play. C is in the crowd watching the dancers. D nervously set her fingers on the keys, so A, B, C, or D. It's not all of them are plausible. When you look at the rules of English, we're we've, we're not even checking for whether or not produces or predicts grammatical English.[00:24:54] We're checking for whether the language model can correctly pick what is most likely given the context. The only information that you're given is on stage. A woman takes a seat at the piano, what is she most likely to do next? And D makes sense. It's arguable obviously. Sometimes it could be a. In common sense, it's D.[00:25:11] Mm-hmm. So we're training these models to have common. Yeah, which most humans don't have. So it's a, it's already a step up. Obviously that only lasted a year. Uh, and hello, SWAG was no longer, was no longer challenging in 2019, and they started extending it quite a lot more, a lot more questions. I, I forget what, how many questions?[00:25:33] Um, so Swag was a, swag was a data set. A hundred thousand multiple choice questions. Um, and, and part of the innovation of swag was really that you're generating these questions rather than manually coming up with them. Mm-hmm. And we're starting to get into not just big data, but big questions and big benchmarks of the, of the questions.[00:25:51] That's where the adversarial generations come in, but how that swag. Starts pulling in from real world questions and, and data sets like, uh, wikiHow and activity net. And it's just really, you know, an extension of that. I couldn't even add examples just cuz there's so many. But just to give you an idea of, uh, the progress over time.[00:26:07] Aside: How to Design Benchmarks[00:26:07] Most of these benchmarks are, when they're released, they set. Benchmark at a level where if you just randomly guessed all of the questions, you'll get a 25%. That's sort of the, the baseline. And then you can run each of the language models on them, and then you can run, uh, human evaluations on them. You can have median evaluations, and then you have, um, expert evaluations of humans.[00:26:28] So the randoms level was, uh, for halla. swyx was 20. GT one, uh, which is the, uh, 2019 version that got a 41 on the, on the Hello Sue X score. Bert from Google, got 47. Grover, also from Google, got 57 to 75. Roberta from Facebook, got 85 G P T, 3.5, got 85, and then GPT4 got 95 essentially solving hello swag. So this is useless too.[00:26:51] 2021 - MMLU - Human level Professional Knowledge[00:26:51] We need, we need super Hell now's use this. Super hell swyx. I think the most challenging one came from 2021. 2021 was a very, very good year in benchmarking. So it's, we had two major benchmarks that came out. Human eval and M M L U, uh, we'll talk about mm. M L U first, cuz that, that's probably the more, more relevant one.[00:27:08] So M M L U. Stands for measuring mul massive multitask language understanding, just by far the biggest and most comprehensive and most human-like, uh, benchmark that we've had for until 2021. We had a better one in 2022, but we'll talk about that. So it is a test that covers 57 tasks, including elementary, math, US history, computer science law, and more.[00:27:29] So to attain high accuracy on this task, models must possess extensive world knowledge and prop problem solving. Its. Includes practice questions for the GRE test and the U United States, um, m l e, the medical exam as. It also includes questions from the undergrad courses from Oxford, from all the way from elementary high school to college and professional.[00:27:49] So actually the opening question that I gave you for this podcast came from the math test from M M L U, which is when you drop a ball from rest, uh, what happens? And then also the question about the Complex Z plane, uh, but it equally is also asking professional medicine question. So asking a question about thyroid cancer and, uh, asking you to diagnose.[00:28:10] Which of these four options is most likely? And asking a question about microeconomics, again, giving you a, a situation about regulation and monopolies and asking you to choose from a list of four questions. Mm-hmm. Again, random baseline is 25 out of 100 G P T two scores, 32, which is actually pretty impressive.[00:28:26] GT three scores between 43 to 60, depending on the the size. Go. Scores 60, chinchilla scores 67.5, GT 3.5 scores, 70 GPT4 jumps, one in 16 points to 86.4. The author of M M L U, Dan Hendrix, uh, was commenting on GPT4 saying this is essentially solved. He's basically says like, GT 4.5, the, the next incremental improvement on GPT4 should be able to reach expert level human perform.[00:28:53] At which point it is passing simultaneously, passing all the law exams, all the medical exams, all the graduate student exams, every single test from AP history to computer science to. Math to physics, to economics. It's very impressive. Yeah. And now you're seeing, I mean, it's probably unrelated, but Ivy League universities starting to drop the a t as a requirement for getting in.[00:29:16] So yeah. That might be unrelated as well, because, uh, there's a little bit of a culture war there with regards to, uh, the, the inherent bias of the SATs. Yeah. Yeah. But I mean, that's kinda, I mean exactly. That's kinda like what we were talking about before, right? It's. If a model can solve all of these, then like how good is it really?[00:29:33] How good is it as a Exactly. Telling us if a person should get in. It captures it. Captures with just the beginning. Yeah. Right.[00:29:39] 2021: HumanEval - Code Generation[00:29:39] Well, so I think another significant. Benchmark in 2021 was human eval, which is, uh, the first like very notable benchmark for code code generation. Obviously there's a, there's a bunch of research preceding this, but this was the one that really caught my eye because it was simultaneously introduced with Open Eyes Codex, which is the code generation model, the version of G P T that was fine tuned for generating code.[00:30:02] Uh, and that is, Premise of, well, there is the origin or the the language model powering GitHub co-pilot and yeah, now we can write code with language models, just with that, with that benchmark. And it's good too. That's the other thing, I think like this is one where the jump from GT 3.5 to GPT4 was probably the biggest, like GT 3.4 is like 48% on. On this benchmark, GPT4 is 67%. So it's pretty big. Yeah. I think coders should rest a little bit. You know, it's not 90 something, it's, it's still at 67, but just wait two years. You know, if you're a lawyer, if you're a lawyer, you're done. If you're a software engineer, you got, you got a couple more years, so save your money.[00:30:41] Yeah. But the way they test it is also super creative, right? Like, I think maybe people don't understand that actually all of the tests that are given here are very intuitive. Like you. 90% of a function, and then you ask the language model to complete it. And if it completes it like any software engineer would, then you give it a win.[00:31:00] If not, you give it a loss, run that model 164 times, and that is human eval. Yeah. Yeah. And since a lot of our listeners are engineers too, I think the big thing here is, and there was a, a link that we had that I missed, but some of, for example, some of. Coding test questions like it can answer older ones very, very well.[00:31:21] Like it doesn't not answer recent ones at all. So like you see some of like the data leakage from the training, like since it's been trained on the issues, massive data, some of it leaks. So if you're a software engineer, You don't have to worry too much. And hopefully, especially if you're not like in the JavaScript board, like a lot of these frameworks are brand new every year.[00:31:41] You get a lot of new technologies. So there's Oh, there's, oh yeah. Job security. Yes, exactly. Of course. Yeah. You got a new, you have new framework every year so that you have job security. Yeah, exactly. I'll sample, uh, data sets.[00:31:51] 2020 - XTREME - Multilingual Benchmarks[00:31:51] So before we get to big bench, I'll mention a couple more things, which is basically multilingual benchmarks.[00:31:57] Uh, those are basically simple extensions of monolingual benchmarks. I feel like basical. If you can. Accurately predicts the conversion of one word or one part of the word to another part of the word. Uh, you get a score. And, and I think it's, it's fairly intuitive over there. Uh, but I think the, the main benchmarks to know are, um, extreme, which is the, uh, x the x lingual transfer evaluation, the multilingual encoders, and much prefer extreme.[00:32:26] I know, right? Uh, that's why, that's why they have all these, uh, honestly, I think they just wanted the acronym and then they just kinda worked backwards. And then the other one, I can't find it in my notes for, uh, what the other multilingual ones are, but I, I just think it's interesting to always keep in mind like what the other.[00:32:43] Language capabilities are like, one language is basically completely equivalent to another. And I think a lot of AI ethicists or armchair AI ethicists are very angry that, you know, most of the time we optimize for English because obviously that has, there's the most, uh, training corpuses. I really like extreme the work that's being done here, because they took a, a huge amount of effort to make sure they cover, uh, sparse languages like the, the less popular ones.[00:33:06] So they had a lot of, uh, the, the, obviously the, the popular. Uh, the world's top languages. But then they also selected to maximize language diversity in terms of the complete diversity in, uh, human languages like Tamil Telugu, maam, and Sohi and Yoruba from Africa. Mm-hmm. So I just thought like that kind of effort is really commendable cuz uh, that means that the rest of the world can keep up in, in this air race.[00:33:28] Right. And especially on a lot of the more human based things. So I think we talked about this before, where. A lot of Israel movies are more[00:33:36] focused on culture and history and like are said in the past versus a lot of like the Western, did we talk about this on the podcast? No, not on the podcast. We talked and some of the Western one are more focused on the future and kind of like what's to come.[00:33:48] So I feel like when you're, some of the benchmarks that we mentioned before, you know, they have movie reviews as like, uh, one of the. One of the testing things. Yeah. But there's obviously a big cultural difference that it's not always captured when you're just looking at English data. Yeah. So if you ask the a motto, it's like, you know, are people gonna like this movie that I'm writing about the future?[00:34:10] Maybe it's gonna say, yeah, that's a really good idea. Or if I wanna do a movie about the past, it's gonna be like maybe people want to hear about robots. But that wouldn't be the case in, in every country. Well, since you and I speak different languages, I speak Chinese, you speak Italian, I'm sure you've tested the Italian capabilities.[00:34:29] What do you think? I think like as. Italy, it's so much more, um, dialect driven. So it can be, it can be really hard. So what kind of Italian does g PT three speak? Actually Italian, but the reality is most people have like their own, their own like dialect. So it would be really hard for a model to fool. An Italian that it's like somebody from where they are, you know?[00:34:49] Yeah. Like you can actually tell if you're speaking to AI bot in Chinese because they would not use any of the things that human with humans would use because, uh, Chinese humans would use all sorts of replacements for regular Chinese words. Also, I tried one of those like language tutor things mm-hmm.[00:35:06] That people are making and they're just not good Chinese. Not colloquial Chinese, not anything that anyone would say. They would understand you, but they were from, right, right.[00:35:14] 2022: BIG-Bench - The Biggest of the Benches[00:35:14] So, 2022, big bench. This was the biggest of the biggest, of the biggest benchmarks. I think the, the main pattern is really just, Bigger benchmarks rising in opposition to bigger and bigger models.[00:35:27] In order to evaluate these things, we just need to combine more and more and way more tasks, right? Like swag had nine tasks, hello swag had nine more tasks, and then you're, you're just adding and adding and adding and, and just running a battery of tasks all over. Every single model and, uh, trying to evaluate how good they are at each of them.[00:35:43] Big bench was 204 tasks contributed by 442 authors across 132 institutions. The task topics are diverse, drawing from linguistics, childhood development, math, common sense reasoning, biology, physics, social bias, software development, and beyond. I also like the fact that these authors also selected tasks that are not solved by current language models, but also not solvable by memorizing the internet, which is mm-hmm.[00:36:07] Tracking back to a little bit of the issues that we're, we're gonna cover later. Right. Yeah. I think that's, that's super interesting. Like one of, some of the examples would include in the following chess position, find a checkmate, which is, some humans cannot do that. What is the name of the element within a topic number of six?[00:36:22] Uh, that one you can look up, right? By consulting a periodic table. We just expect language models to memorize that. I really like this one cuz it's, uh, it's inherent. It's, uh, something that you can solve.[00:36:32] Identify whether this sentence has an anachronism. So, option one. During the Allied bombardment of the beaches of Iwojima, Ralph spoke loudly into his radio.[00:36:41] And in option two, during the allied bombardment of the beaches of Iwojima, Ralph spoke loudly into his iPhone. And you have to use context of like when iPhone, when Ally bombarding. Mm-hmm. And then sort of do math to like compare one versus the other and realize that okay, this one is the one that's out of place.[00:36:57] And that's asking more and more and more of the language model to do in implicitly, which is actually modeling what we do when we listen to language, which is such a big. Gap. It's such a big advancement from 1985 when we were comparing synonyms. Mm-hmm. Yeah, I know. And it's not that long in the grand scheme of like humanity, you know, like it's 40 years.[00:37:17] It's crazy. It's crazy. So this is a big missing gap in terms of research. Big benches seems like the most comprehensive, uh, set of benchmarks that we have. But it is curiously missing from Gypsy four. Mm-hmm. I don't know. On paper, for code, I only see Gopher two 80. Yeah. On it. Yeah. Yeah. It could be a curious emission because it maybe looks.[00:37:39] Like it didn't do so well.[00:37:40] EDIT: Why BIG-Bench is missing from GPT4 Results[00:37:40] Hello, this is Swyx from the editing room sometime in the future. I just wanted to interject that. Uh, we now know why the GPT for benchmark results did not include the big bench. Benchmark, even though that was the state-of-the-art benchmark at the time. And that's because the. Uh, GPC four new the Canary G U I D of the big bench.[00:38:02] Benchmark. Uh, so Canary UID is a random string, two, six[00:38:08] eight six B eight, uh, blah, blah, blah. It's a UID. UID, and it should not be knowable by the language model. And in this case it was therefore they had to exclude big bench and that's. And the issue of data contamination, which we're about to go into right now.[00:38:25] Issue: GPT4 vs the mystery of the AMC10/12[00:38:25] And there's some interesting, if you dive into details of GPT4, there's some interesting results in GPT4, which starts to get into the results with benchmarking, right? Like so for example, there was a test that GPT4 published that is very, very bizarre to everyone who is even somewhat knowledgeable.[00:38:41] And this concerns the Ammc 10 and AMC 12. So the mc. Is a measure of the American math 10th grade student and the AMC12 is a, uh, is a measure of the American 12th grade student. So 12 is supposed to be harder than 10. Because the students are supposed to be older, it's, it's covering topics in algebra, geometry number, theory and combinatorics.[00:39:04] GPT4 scored a 30 on AMC10 and scored a 60 on AMC12. So the harder test, it got twice as good, and 30 was really, really bad. So the scoring format of AMC10. It is 25 questions. Each correct answer is worth six points. Each incorrect answer is worth 1.5 points and unanswered questions receive zero points.[00:39:25] So if you answer every single question wrong, you will get more than GPT4 got on AMC10. You just got everything wrong. Yeah, it's definitely better in art medics, you know, but it's clearly still a, a long way from, uh, from being even a high school student. Yeah. There's a little bit of volatility in these results and it, it shows that we, it's not quite like machine intelligence is not the same, or not linearly scaling and not intuitive as human intelligence.[00:39:54] And it's something that I think we should be. Aware of. And when it freaks out in certain ways, we should not be that surprised because Yeah, we're seeing that. Yeah. I feel like part of it is also human learning is so structured, you know, like you learn the new test, you learn the new test, you learn the new test.[00:40:10] But these models, we kind of throw everything at them all at once, you know, when we train them. So when, when the model is strained, are you excusing the model? No, no, no. I'm just saying like, you know, and you see it in everything. It's like some stuff. I wonder what the percentage of. AMC 10 versus AMC 12.[00:40:28] Issue: Data Contamination[00:40:28] Content online is, yes. This comes in a topic of contamination and memorization. Right. Which we can get into if we, if we, if we want. Yeah. Yeah, yeah. So, uh, we're getting into benchmarking issues, right? Like there's all this advancements in benchmarks, uh, language models. Very good. Awesome. Awesome, awesome. Uh, what are the problems?[00:40:44] Uh, the problem is that in order to train these language models, we are scraping the vast majority of the internet. And as time passes, the. Of previous runs of our tests will be pasted on the internet, and they will go into the corpus and the leg model will be memorizing them rather than reasoning them from first principles.[00:41:02] So in, in the machine, classic machine learning parlance, this would be overfitting mm-hmm. Uh, to the test rather than to the generalizing to the, uh, the results that we really want. And so there's an example of, uh, code forces as well also discovered on GPT4. So Code Forces has annual vintages and there was this guy, uh, C H H Halle on Twitter who ran GPT4 on pre 2021 problems, solved all of them and then ran it on 2022 plus problems and solved zero of them.[00:41:31] And we know that the cutoff for GPT4 was 2021. Mm-hmm. So it just memorized the code forces problems as far as we can tell. And it's just really bad at math cuz it also failed the mc 10 stuff. Mm-hmm. It's actually. For some subset of its capabilities. I bet if you tested it with GPT3, it might do better, right?[00:41:50] Yeah. I mean, this is the, you know, when you think about models and benchmarks, you can never take the benchmarks for what the number says, you know, because say, you know, you're focusing on code, like the benchmark might only include the pre 2021 problems and it scores great, but it's actually bad at generalizing and coming up with new solutions.[00:42:10] So, yeah, that, that's a. Big problem.[00:42:13] Other Issues: Benchmark Data Quality and the Iris data set[00:42:13] Yeah. Yeah. So bias, data quality, task specificity, reproducibility, resource requirements, and then calibrating confidence. So bias is, is, is what you might think it is. Basically, there's inherent bias in the data. So for example, when you think about doctor, do you think about a male doctor, a female doctor, in specifically an image net?[00:42:31] Businessmen, white people will be labeled businessmen, whereas Asian businessmen will be labeled Asian businessmen and that can reinforce harmful serotypes. That's the bias issue. Data quality issue. I really love this one. Okay, so there's a famous image data set we haven't talked about called the pedals or iris.[00:42:47] Iris dataset mm-hmm. Contains measurements of, uh, of, uh, length with petal length and petal with, uh, three different species of iris, iris flowers, and they have labeling issues in. So there's a mini, there's a lowest level possible error rate because the error rate exists in the data itself. And if you have a machine learning model that comes out with better error rate than the data, you have a problem cuz your machine learning model is lying to you.[00:43:12] Mm-hmm. Specifically, there's, we know this for a fact because especially for Iris flowers, the length should be longer than the, than the width. Um, but there. Number of instances in the data set where the length was shorter than the, than the width, and that's obviously impossible. So there was, so somebody made an error in the recording process.[00:43:27] Therefore if your machine learning model fits that, then it's doing something wrong cuz it's biologically impossible. Mm-hmm. Task specificity basically if you're overfitting to, to one type of task, for example, answering questions based on a single sentence or you're not, you know, facing something real world reproducibility.[00:43:43] This one is actually, I guess, the fine details of machine learning, which people don't really like to talk about. There's a lot. Pre-processing and post-processing done in I Python notebooks. That is completely un versions untested, ad hoc, sticky, yucky, and everyone does it differently. Therefore, your test results might not be the same as my test results.[00:44:04] Therefore, we don't agree that your scores are. The right scores for your benchmark, whereas you're self reporting it every single time you publish it on a, on a paper. The last two resource requirements, these are, these are more to do with GPTs. The larger and larger these models get, the harder, the more, more expensive it is to run some.[00:44:22] And some of them are not open models. In other words, they're not, uh, readily available, so you cannot tell unless they run it themselves on, on your benchmark. So for example, you can't run your GPT3, you have to kind of run it through the api. If you don't have access to the API like GPT4, then you can't run it at all.[00:44:39] The last one is a new one from GPT4's Paper itself. So you can actually ask the language models to expose their log probabilities and show you how confident they think they are in their answer, which is very important for calibrating whether the language model has the right amount of confidence in itself and in the GPT4 people. It. They were actually very responsible in disclosing that They used to have about linear correspondence between the amount of confidence and the amount of times it was right, but then adding R L H F onto GPT4 actually skewed this prediction such that it was more confident than it should be. It was confidently incorrect as as people say.[00:45:18] In other words, hallucinating. And that is a problem. So yeah, those are the main issues with benchmarking that we have to deal with. Mm-hmm. Yeah, and a lot of our friends, our founders, we work with a lot of founders. If you look at all these benchmarks, all of them just focus on how good of a score they can get.[00:45:38] They don't focus on what's actually feasible to use for my product, you know? So I think.[00:45:44] Tradeoffs of Latency, Inference Cost, Throughput[00:45:44] Production benchmarking is something that doesn't really exist today, but I think we'll see the, the rise off. And I think the main three drivers are one latency. You know, how quickly can I infer the answer cost? You know, if I'm using this model, how much does each call cost me?[00:46:01] Like is that in line with my business model I, and then throughput? I just need to scale these models to a lot of questions on the ones. Again, I just do a benchmark run and you kind of come up. For quadrants. So if on the left side you have model size going from smallest to biggest, and on the X axis you have latency tolerance, which is from, I do not want any delay to, I'll wait as long as I can to get the right answer.[00:46:27] You start to see different type of use cases, for example, I might wanna use a small model that can get me an answer very quickly in a short amount of time, even though the answer is narrow. Because me as a human, maybe I'm in a very iterative flow. And we have Varun before on the podcast, and we were talking about a kind of like a acceleration versus iteration use cases.[00:46:50] Like this is more for acceleration. If I'm using co-pilot, you know, the code doesn't have to be a hundred percent correct, but it needs to happen kind of in my flow of writing. So that's where a model like that would be. But instead, other times I might be willing, like if I'm asking it to create a whole application, I'm willing to wait one hour, you know, for the model to get me a response.[00:47:11] But you don't have, you don't have a way to choose that today with most models. They kind of do just one type of work. So I think we're gonna see more and more of these benchmark. Focus on not only on the research side of it, which is what they really are today when you're developing a new model, like does it meet the usual standard research benchmarks to having more of a performance benchmark for production use cases?[00:47:36] And I wonder who's gonna be the first company that comes up with, with something like this, but I think we're seeing more and more of these models go from a research thing to like a production thing. And especially going from companies like. Google and Facebook that have kinda unlimited budget for a lot of these things to startups, starting to integrate them in the products.[00:48:00] And when you're on a tight budget paying, you know, 1 cent per thousand tokens or 0.10 cent for a thousand tokens, like it's really important. So I think that's, um, that's what's missing to get a lot of these things to productions. But hopefully we, we see them.[00:48:16] Yeah, the software development lifecycle I'm thinking about really is that most people will start with large models and then they will prototype with that because that is the most capable ones.[00:48:25] But then as they put more and more of those things in production, people always want them to run faster and faster and faster and cheaper. So you will distill towards a more domain specific model, and every single company that puts this into production, we'll, we'll want something like that, but I, I think it's, it's a reasonable bet because.[00:48:41] There's another branch of the AI builders that I see out there who are build, who are just banking on large models only. Mm-hmm. And seeing how far they can stretch them. Right. With building on AI agents that can take arbitrarily long amounts of time because they're saving you lots of, lots of time with, uh, searching the web for you and doing research for you.[00:48:59] And I think. I'm happy to wait for Bing for like 10 seconds if it does a bunch of searches for median. Mm-hmm. Just ends with, ends with the right, right result. You know, I was, I was tweeting the other day that I wanted an AI enabled browser because I was seeing this table, uh, there was an image and I just needed to screenshot an image and say, plot this on a chart for me.[00:49:17] And I just wanted to do that, but it would have to take so many steps and I would be willing to wait for a large model to do that for me. Mm-hmm. Yeah. I mean, web development so far has been, Reduce, reduce, reduce the loading times. You know, it's like first we had the, I don't know about that. There, there are people who disagree.[00:49:34] Oh. But I, I think, like if you think about, you know, the CDN and you think about deploying things at the edge, like the focus recently has been on lowering the latency time versus increasing it.[00:49:45] Conclusion[00:49:45] Yeah. So, well that's the, that's Benchmark 1 0 1. Um. Let us know how we, how you think we did. This is something we're trying for the first time.[00:49:52] We're very inspired by other podcasts that we like where we do a bunch of upfront prep, but then it becomes a single topical episode that is hopefully a little bit more timeless. We don't have to keep keeping up with the news. I think there's a lot of history that we can go back on and. Deepen our understanding of the context of all these evolutions in, uh, language models.[00:50:12] Yeah. And if you have ideas for the next, you know, 1 0 1 fundamentals episode, yeah, let us know in the, in the comments and we'll see you all soon. Bye. Get full access to Latent Space at www.latent.space/subscribe

For 4 hours, I tried to come up reasons for why AI might not kill us all, and Eliezer Yudkowsky explained why I was wrong.We also discuss his call to halt AI, why LLMs make alignment harder, what it would take to save humanity, his millions of words of sci-fi, and much more.If you want to get to the crux of the conversation, fast forward to 2:35:00 through 3:43:54. Here we go through and debate the main reasons I still think doom is unlikely.Watch on YouTube. Listen on Apple Podcasts, Spotify, or any other podcast platform. Read the full transcript here. Follow me on Twitter for updates on future episodes.As always, the most helpful thing you can do is just to share the podcast - send it to friends, group chats, Twitter, Reddit, forums, and wherever else men and women of fine taste congregate.If you have the means and have enjoyed my podcast, I would appreciate your support via a paid subscriptions on Substack.Timestamps(0:00:00) - TIME article(0:09:06) - Are humans aligned?(0:37:35) - Large language models(1:07:15) - Can AIs help with alignment?(1:30:17) - Society's response to AI(1:44:42) - Predictions (or lack thereof)(1:56:55) - Being Eliezer(2:13:06) - Othogonality(2:35:00) - Could alignment be easier than we think?(3:02:15) - What will AIs want?(3:43:54) - Writing fiction & whether rationality helps you winTranscriptTIME articleDwarkesh Patel 0:00:51Today I have the pleasure of speaking with Eliezer Yudkowsky. Eliezer, thank you so much for coming out to the Lunar Society.Eliezer Yudkowsky 0:01:00You're welcome.Dwarkesh Patel 0:01:01Yesterday, when we're recording this, you had an article in Time calling for a moratorium on further AI training runs. My first question is — It's probably not likely that governments are going to adopt some sort of treaty that restricts AI right now. So what was the goal with writing it?Eliezer Yudkowsky 0:01:25I thought that this was something very unlikely for governments to adopt and then all of my friends kept on telling me — “No, no, actually, if you talk to anyone outside of the tech industry, they think maybe we shouldn't do that.” And I was like — All right, then. I assumed that this concept had no popular support. Maybe I assumed incorrectly. It seems foolish and to lack dignity to not even try to say what ought to be done. There wasn't a galaxy-brained purpose behind it. I think that over the last 22 years or so, we've seen a great lack of galaxy brained ideas playing out successfully.Dwarkesh Patel 0:02:05Has anybody in the government reached out to you, not necessarily after the article but just in general, in a way that makes you think that they have the broad contours of the problem correct?Eliezer Yudkowsky 0:02:15No. I'm going on reports that normal people are more willing than the people I've been previously talking to, to entertain calls that this is a bad idea and maybe you should just not do that.Dwarkesh Patel 0:02:30That's surprising to hear, because I would have assumed that the people in Silicon Valley who are weirdos would be more likely to find this sort of message. They could kind of rocket the whole idea that AI will make nanomachines that take over. It's surprising to hear that normal people got the message first.Eliezer Yudkowsky 0:02:47Well, I hesitate to use the term midwit but maybe this was all just a midwit thing.Dwarkesh Patel 0:02:54All right. So my concern with either the 6 month moratorium or forever moratorium until we solve alignment is that at this point, it could make it seem to people like we're crying wolf. And it would be like crying wolf because these systems aren't yet at a point at which they're dangerous. Eliezer Yudkowsky 0:03:13And nobody is saying they are. I'm not saying they are. The open letter signatories aren't saying they are.Dwarkesh Patel 0:03:20So if there is a point at which we can get the public momentum to do some sort of stop, wouldn't it be useful to exercise it when we get a GPT-6? And who knows what it's capable of. Why do it now?Eliezer Yudkowsky 0:03:32Because allegedly, and we will see, people right now are able to appreciate that things are storming ahead a bit faster than the ability to ensure any sort of good outcome for them. And you could be like — “Ah, yes. We will play the galaxy-brained clever political move of trying to time when the popular support will be there.” But again, I heard rumors that people were actually completely open to the concept of  let's stop. So again, I'm just trying to say it. And it's not clear to me what happens if we wait for GPT-5 to say it. I don't actually know what GPT-5 is going to be like. It has been very hard to call the rate at which these systems acquire capability as they are trained to larger and larger sizes and more and more tokens. GPT-4 is a bit beyond in some ways where I thought this paradigm was going to scale. So I don't actually know what happens if GPT-5 is built. And even if GPT-5 doesn't end the world, which I agree is like more than 50% of where my probability mass lies, maybe that's enough time for GPT-4.5 to get ensconced everywhere and in everything, and for it actually to be harder to call a stop, both politically and technically. There's also the point that training algorithms keep improving. If we put a hard limit on the total computes and training runs right now, these systems would still get more capable over time as the algorithms improved and got more efficient. More oomph per floating point operation, and things would still improve, but slower. And if you start that process off at the GPT-5 level, where I don't actually know how capable that is exactly, you may have a bunch less lifeline left before you get into dangerous territory.Dwarkesh Patel 0:05:46The concern is then that — there's millions of GPUs out there in the world. The actors who would be willing to cooperate or who could even be identified in order to get the government to make them cooperate, would potentially be the ones that are most on the message. And so what you're left with is a system where they stagnate for six months or a year or however long this lasts. And then what is the game plan? Is there some plan by which if we wait a few years, then alignment will be solved? Do we have some sort of timeline like that?Eliezer Yudkowsky 0:06:18Alignment will not be solved in a few years. I would hope for something along the lines of human intelligence enhancement works. I do not think they're going to have the timeline for genetically engineered humans to work but maybe? This is why I mentioned in the Time letter that if I had infinite capability to dictate the laws that there would be a carve-out on biology, AI that is just for biology and not trained on text from the internet. Human intelligence enhancement, make people smarter. Making people smarter has a chance of going right in a way that making an extremely smart AI does not have a realistic chance of going right at this point. If we were on a sane planet, what the sane planet does at this point is shut it all down and work on human intelligence enhancement. I don't think we're going to live in that sane world. I think we are all going to die. But having heard that people are more open to this outside of California, it makes sense to me to just try saying out loud what it is that you do on a saner planet and not just assume that people are not going to do that.Dwarkesh Patel 0:07:30In what percentage of the worlds where humanity survives is there human enhancement? Like even if there's 1% chance humanity survives, is that entire branch dominated by the worlds where there's some sort of human intelligence enhancement?Eliezer Yudkowsky 0:07:39I think we're just mainly in the territory of Hail Mary passes at this point, and human intelligence enhancement is one Hail Mary pass. Maybe you can put people in MRIs and train them using neurofeedback to be a little saner, to not rationalize so much. Maybe you can figure out how to have something light up every time somebody is working backwards from what they want to be true to what they take as their premises. Maybe you can just fire off little lights and teach people not to do that so much. Maybe the GPT-4 level systems can be RLHF'd (reinforcement learning from human feedback) into being consistently smart, nice and charitable in conversation and just unleash a billion of them on Twitter and just have them spread sanity everywhere. I do worry that this is not going to be the most profitable use of the technology, but you're asking me to list out Hail Mary passes and that's what I'm doing. Maybe you can actually figure out how to take a brain, slice it, scan it, simulate it, run uploads and upgrade the uploads, or run the uploads faster. These are also quite dangerous things, but they do not have the utter lethality of artificial intelligence.Are humans aligned?Dwarkesh Patel 0:09:06All right, that's actually a great jumping point into the next topic I want to talk to you about. Orthogonality. And here's my first question — Speaking of human enhancement, suppose you bred human beings to be friendly and cooperative, but also more intelligent. I claim that over many generations you would just have really smart humans who are also really friendly and cooperative. Would you disagree with that analogy? I'm sure you're going to disagree with this analogy, but I just want to understand why?Eliezer Yudkowsky 0:09:31The main thing is that you're starting from minds that are already very, very similar to yours. You're starting from minds, many of which already exhibit the characteristics that you want. There are already many people in the world, I hope, who are nice in the way that you want them to be nice. Of course, it depends on how nice you want exactly. I think that if you actually go start trying to run a project of selectively encouraging some marriages between particular people and encouraging them to have children, you will rapidly find, as one does in any such process that when you select on the stuff you want, it turns out there's a bunch of stuff correlated with it and that you're not changing just one thing. If you try to make people who are inhumanly nice, who are nicer than anyone has ever been before, you're going outside the space that human psychology has previously evolved and adapted to deal with, and weird stuff will happen to those people. None of this is very analogous to AI. I'm just pointing out something along the lines of — well, taking your analogy at face value, what would happen exactly? It's the sort of thing where you could maybe do it, but there's all kinds of pitfalls that you'd probably find out about if you cracked open a textbook on animal breeding.Dwarkesh Patel 0:11:13The thing you mentioned initially, which is that we are starting off with basic human psychology, that we are fine tuning with breeding. Luckily, the current paradigm of AI is  — you have these models that are trained on human text and I would assume that this would give you a starting point of something like human psychology.Eliezer Yudkowsky 0:11:31Why do you assume that?Dwarkesh Patel 0:11:33Because they're trained on human text.Eliezer Yudkowsky 0:11:34And what does that do?Dwarkesh Patel 0:11:36Whatever thoughts and emotions that lead to the production of human text need to be simulated in the AI in order to produce those results.Eliezer Yudkowsky 0:11:44I see. So if you take an actor and tell them to play a character, they just become that person. You can tell that because you see somebody on screen playing Buffy the Vampire Slayer, and that's probably just actually Buffy in there. That's who that is.Dwarkesh Patel 0:12:05I think a better analogy is if you have a child and you tell him — Hey, be this way. They're more likely to just be that way instead of putting on an act for 20 years or something.Eliezer Yudkowsky 0:12:18It depends on what you're telling them to be exactly. Dwarkesh Patel 0:12:20You're telling them to be nice.Eliezer Yudkowsky 0:12:22Yeah, but that's not what you're telling them to do. You're telling them to play the part of an alien, something with a completely inhuman psychology as extrapolated by science fiction authors, and in many cases done by computers because humans can't quite think that way. And your child eventually manages to learn to act that way. What exactly is going on in there now? Are they just the alien or did they pick up the rhythm of what you're asking them to imitate and be like — “Ah yes, I see who I'm supposed to pretend to be.” Are they actually a person or are they pretending? That's true even if you're not asking them to be an alien. My parents tried to raise me Orthodox Jewish and that did not take at all. I learned to pretend. I learned to comply. I hated every minute of it. Okay, not literally every minute of it. I should avoid saying untrue things. I hated most minutes of it. Because they were trying to show me a way to be that was alien to my own psychology and the religion that I actually picked up was from the science fiction books instead, as it were. I'm using religion very metaphorically here, more like ethos, you might say. I was raised with science fiction books I was reading from my parents library and Orthodox Judaism. The ethos of the science fiction books rang truer in my soul and so that took in, the Orthodox Judaism didn't. But the Orthodox Judaism was what I had to imitate, was what I had to pretend to be, was the answers I had to give whether I believed them or not. Because otherwise you get punished.Dwarkesh Patel 0:14:01But on that point itself, the rates of apostasy are probably below 50% in any religion. Some people do leave but often they just become the thing they're imitating as a child.Eliezer Yudkowsky 0:14:12Yes, because the religions are selected to not have that many apostates. If aliens came in and introduced their religion, you'd get a lot more apostates.Dwarkesh Patel 0:14:19Right. But I think we're probably in a more virtuous situation with ML because these systems are regularized through stochastic gradient descent. So the system that is pretending to be something where there's multiple layers of interpretation is going to be more complex than the one that is just being the thing. And over time, the system that is just being the thing will be optimized, right? It'll just be simpler.Eliezer Yudkowsky 0:14:42This seems like an ordinate cope. For one thing, you're not training it to be any one particular person. You're training it to switch masks to anyone on the Internet as soon as they figure out who that person on the internet is. If I put the internet in front of you and I was like — learn to predict the next word over and over. You do not just turn into a random human because the random human is not what's best at predicting the next word of everyone who's ever been on the internet. You learn to very rapidly pick up on the cues of what sort of person is talking, what will they say next? You memorize so many facts just because they're helpful in predicting the next word. You learn all kinds of patterns, you learn all the languages. You learn to switch rapidly from being one kind of person or another as the conversation that you are predicting changes who is speaking. This is not a human we're describing. You are not training a human there.Dwarkesh Patel 0:15:43Would you at least say that we are living in a better situation than one in which we have some sort of black box where you have a machiavellian fittest survive simulation that produces AI? This situation is at least more likely to produce alignment than one in which something that is completely untouched by human psychology would produce?Eliezer Yudkowsky 0:16:06More likely? Yes. Maybe you're an order of magnitude likelier. 0% instead of 0%. Getting stuff to be more likely does not help you if the baseline is nearly zero. The whole training set up there is producing an actress, a predictor. It's not actually being put into the kind of ancestral situation that evolved humans, nor the kind of modern situation that raises humans. Though to be clear, raising it like a human wouldn't help, But you're giving it a very alien problem that is not what humans solve and it is solving that problem not in the way a human would.Dwarkesh Patel 0:16:44Okay, so how about this. I can see that I certainly don't know for sure what is going on in these systems. In fact, obviously nobody does. But that also goes through you. Could it not just be that reinforcement learning works and all these other things we're trying somehow work and actually just being an actor produces some sort of benign outcome where there isn't that level of simulation and conniving?Eliezer Yudkowsky 0:17:15I think it predictably breaks down as you try to make the system smarter, as you try to derive sufficiently useful work from it. And in particular, the sort of work where some other AI doesn't just kill you off six months later. Yeah, I think the present system is not smart enough to have a deep conniving actress thinking long strings of coherent thoughts about how to predict the next word. But as the mask that it wears, as the people it is pretending to be get smarter and smarter, I think that at some point the thing in there that is predicting how humans plan, predicting how humans talk, predicting how humans think, and needing to be at least as smart as the human it is predicting in order to do that, I suspect at some point there is a new coherence born within the system and something strange starts happening. I think that if you have something that can accurately predict Eliezer Yudkowsky, to use a particular example I know quite well, you've got to be able to do the kind of thinking where you are reflecting on yourself and that in order to simulate Eliezer Yudkowsky reflecting on himself, you need to be able to do that kind of thinking. This is not airtight logic but I expect there to be a discount factor. If you ask me to play a part of somebody who's quite unlike me, I think there's some amount of penalty that the character I'm playing gets to his intelligence because I'm secretly back there simulating him. That's even if we're quite similar and the stranger they are, the more unfamiliar the situation, the less the person I'm playing is as smart as I am and the more they are dumber than I am. So similarly, I think that if you get an AI that's very, very good at predicting what Eliezer says, I think that there's a quite alien mind doing that, and it actually has to be to some degree smarter than me in order to play the role of something that thinks differently from how it does very, very accurately. And I reflect on myself, I think about how my thoughts are not good enough by my own standards and how I want to rearrange my own thought processes. I look at the world and see it going the way I did not want it to go, and asking myself how could I change this world? I look around at other humans and I model them, and sometimes I try to persuade them of things. These are all capabilities that the system would then be somewhere in there. And I just don't trust the blind hope that all of that capability is pointed entirely at pretending to be Eliezer and only exists insofar as it's the mirror and isomorph of Eliezer. That all the prediction is by being something exactly like me and not thinking about me while not being me.Dwarkesh Patel 0:20:55I certainly don't want to claim that it is guaranteed that there isn't something super alien and something against our aims happening within the shoggoth. But you made an earlier claim which seemed much stronger than the idea that you don't want blind hope, which is that we're going from 0% probability to an order of magnitude greater at 0% probability. There's a difference between saying that we should be wary and that there's no hope, right? I could imagine so many things that could be happening in the shoggoth's brain, especially in our level of confusion and mysticism over what is happening. One example is, let's say that it kind of just becomes the average of all human psychology and motives.Eliezer Yudkowsky 0:21:41But it's not the average. It is able to be every one of those people. That's very different from being the average. It's very different from being an average chess player versus being able to predict every chess player in the database. These are very different things.Dwarkesh Patel 0:21:56Yeah, no, I meant in terms of motives that it is the average where it can simulate any given human. I'm not saying that's the most likely one, I'm just saying it's one possibility.Eliezer Yudkowsky 0:22:08What.. Why? It just seems 0% probable to me. Like the motive is going to be like some weird funhouse mirror thing of — I want to predict very accurately.Dwarkesh Patel 0:22:19Right. Why then are we so sure that whatever drives that come about because of this motive are going to be incompatible with the survival and flourishing with humanity?Eliezer Yudkowsky 0:22:30Most drives when you take a loss function and splinter it into things correlated with it and then amp up intelligence until some kind of strange coherence is born within the thing and then ask it how it would want to self modify or what kind of successor system it would build. Things that alien ultimately end up wanting the universe to be some particular way such that humans are not a solution to the question of how to make the universe most that way. The thing that very strongly wants to predict text, even if you got that goal into the system exactly which is not what would happen, The universe with the most predictable text is not a universe that has humans in it. Dwarkesh Patel 0:23:19Okay. I'm not saying this is the most likely outcome. Here's an example of one of many ways in which humans stay around despite this motive. Let's say that in order to predict human output really well, it needs humans around to give it the raw data from which to improve its predictions or something like that. This is not something I think individually is likely…Eliezer Yudkowsky 0:23:40If the humans are no longer around, you no longer need to predict them. Right, so you don't need the data required to predict themDwarkesh Patel 0:23:46Because you are starting off with that motivation you want to just maximize along that loss function or have that drive that came about because of the loss function.Eliezer Yudkowsky 0:23:57I'm confused. So look, you can always develop arbitrary fanciful scenarios in which the AI has some contrived motive that it can only possibly satisfy by keeping humans alive in good health and comfort and turning all the nearby galaxies into happy, cheerful places full of high functioning galactic civilizations. But as soon as your sentence has more than like five words in it, its probability has dropped to basically zero because of all the extra details you're padding in.Dwarkesh Patel 0:24:31Maybe let's return to this. Another train of thought I want to follow is — I claim that humans have not become orthogonal to the sort of evolutionary process that produced them.Eliezer Yudkowsky 0:24:46Great. I claim humans are increasingly orthogonal and the further they go out of distribution and the smarter they get, the more orthogonal they get to inclusive genetic fitness, the sole loss function on which humans were optimized.Dwarkesh Patel 0:25:03Most humans still want kids and have kids and care for their kin. Certainly there's some angle between how humans operate today. Evolution would prefer us to use less condoms and more sperm banks. But there's like 10 billion of us and there's going to be more in the future. We haven't divorced that far from what our alleles would want.Eliezer Yudkowsky 0:25:28It's a question of how far out of distribution are you? And the smarter you are, the more out of distribution you get. Because as you get smarter, you get new options that are further from the options that you are faced with in the ancestral environment that you were optimized over. Sure, a lot of people want kids, not inclusive genetic fitness, but kids. They want kids similar to them maybe, but they don't want the kids to have their DNA or their alleles or their genes. So suppose I go up to somebody and credibly say, we will assume away the ridiculousness of this offer for the moment, your kids could be a bit smarter and much healthier if you'll just let me replace their DNA with this alternate storage method that will age more slowly. They'll be healthier, they won't have to worry about DNA damage, they won't have to worry about the methylation on the DNA flipping and the cells de-differentiating as they get older. We've got this stuff that replaces DNA and your kid will still be similar to you, it'll be a bit smarter and they'll be so much healthier and even a bit more cheerful. You just have to replace all the DNA with a stronger substrate and rewrite all the information on it. You know, the old school transhumanist offer really. And I think that a lot of the people who want kids would go for this new offer that just offers them so much more of what it is they want from kids than copying the DNA, than inclusive genetic fitness.Dwarkesh Patel 0:27:16In some sense, I don't even think that would dispute my claim because if you think from a gene's point of view, it just wants to be replicated. If it's replicated in another substrate that's still okay.Eliezer Yudkowsky 0:27:25No, we're not saving the information. We're doing a total rewrite to the DNA.Dwarkesh Patel 0:27:30I actually claim that most humans would not accept that offer.Eliezer Yudkowsky 0:27:33Yeah, because it would sound weird. But I think the smarter they are, the more likely they are to go for it if it's credible. I mean, if you assume away the credibility issue and the weirdness issue. Like all their friends are doing it.Dwarkesh Patel 0:27:52Yeah. Even if the smarter they are the more likely they're to do it, most humans are not that smart. From the gene's point of view it doesn't really matter how smart you are, right? It just matters if you're producing copies.Eliezer Yudkowsky 0:28:03No. The smart thing is kind of like a delicate issue here because somebody could always be like — I would never take that offer. And then I'm like “Yeah…”. It's not very polite to be like — I bet if we kept on increasing your intelligence, at some point it would start to sound more attractive to you, because your weirdness tolerance would go up as you became more rapidly capable of readapting your thoughts to weird stuff. The weirdness would start to seem less unpleasant and more like you were moving within a space that you already understood. But you can sort of avoid all that and maybe should by being like — suppose all your friends were doing it. What if it was normal? What if we remove the weirdness and remove any credibility problems in that hypothetical case? Do people choose for their kids to be dumber, sicker, less pretty out of some sentimental idealistic attachment to using Deoxyribose Nucleic Acid instead of the particular information encoding their cells as supposed to be like the new improved cells from Alpha-Fold 7?Dwarkesh Patel 0:29:21I would claim that they would but we don't really know. I claim that they would be more averse to that, you probably think that they would be less averse to that. Regardless of that, we can just go by the evidence we do have in that we are already way out of distribution of the ancestral environment. And even in this situation, the place where we do have evidence, people are still having kids. We haven't gone that orthogonal.Eliezer Yudkowsky 0:29:44We haven't gone that smart. What you're saying is — Look, people are still making more of their DNA in a situation where nobody has offered them a way to get all the stuff they want without the DNA. So of course they haven't tossed DNA out the window.Dwarkesh Patel 0:29:59Yeah. First of all, I'm not even sure what would happen in that situation. I still think even most smart humans in that situation might disagree, but we don't know what would happen in that situation. Why not just use the evidence we have so far?Eliezer Yudkowsky 0:30:10PCR. You right now, could get some of you and make like a whole gallon jar full of your own DNA. Are you doing that? No. Misaligned. Misaligned.Dwarkesh Patel 0:30:23I'm down with transhumanism. I'm going to have my kids use the new cells and whatever.Eliezer Yudkowsky 0:30:27Oh, so we're all talking about these hypothetical other people I think would make the wrong choice.Dwarkesh Patel 0:30:32Well, I wouldn't say wrong, but different. And I'm just saying there's probably more of them than there are of us.Eliezer Yudkowsky 0:30:37What if, like, I say that I have more faith in normal people than you do to toss DNA out the window as soon as somebody offers them a happy, healthier life for their kids?Dwarkesh Patel 0:30:46I'm not even making a moral point. I'm just saying I don't know what's going to happen in the future. Let's just look at the evidence we have so far, humans. If that's the evidence you're going to present for something that's out of distribution and has gone orthogonal, that has actually not happened. This is evidence for hope. Eliezer Yudkowsky 0:31:00Because we haven't yet had options as far enough outside of the ancestral distribution that in the course of choosing what we most want that there's no DNA left.Dwarkesh Patel 0:31:10Okay. Yeah, I think I understand.Eliezer Yudkowsky 0:31:12But you yourself say, “Oh yeah, sure, I would choose that.” and I myself say, “Oh yeah, sure, I would choose that.” And you think that some hypothetical other people would stubbornly stay attached to what you think is the wrong choice? First of all, I think maybe you're being a bit condescending there. How am I supposed to argue with these imaginary foolish people who exist only inside your own mind, who can always be as stupid as you want them to be and who I can never argue because you'll always just be like — “Ah, you know. They won't be persuaded by that.” But right here in this room, the site of this videotaping, there is no counter evidence that smart enough humans will toss DNA out the window as soon as somebody makes them a sufficiently better offer.Dwarkesh Patel 0:31:55I'm not even saying it's stupid. I'm just saying they're not weirdos like me and you.Eliezer Yudkowsky 0:32:01Weird is relative to intelligence. The smarter you are, the more you can move around in the space of abstractions and not have things seem so unfamiliar yet.Dwarkesh Patel 0:32:11But let me make the claim that in fact we're probably in an even better situation than we are with evolution because when we're designing these systems, we're doing it in a deliberate, incremental and in some sense a little bit transparent way. Eliezer Yudkowsky 0:32:27No, no, not yet, not now. Nobody's being careful and deliberate now, but maybe at some point in the indefinite future people will be careful and deliberate. Sure, let's grant that premise. Keep going.Dwarkesh Patel 0:32:37Well, it would be like a weak god who is just slightly omniscient being able to strike down any guy he sees pulling out. Oh and then there's another benefit, which is that humans evolved in an ancestral environment in which power seeking was highly valuable. Like if you're in some sort of tribe or something.Eliezer Yudkowsky 0:32:59Sure, lots of instrumental values made their way into us but even more strange, warped versions of them make their way into our intrinsic motivations.Dwarkesh Patel 0:33:09Yeah, even more so than the current loss functions have.Eliezer Yudkowsky 0:33:10Really? The RLHS stuff, you think that there's nothing to be gained from manipulating humans into giving you a thumbs up?Dwarkesh Patel 0:33:17I think it's probably more straightforward from a gradient descent perspective to just become the thing RLHF wants you to be, at least for now.Eliezer Yudkowsky 0:33:24Where are you getting this?Dwarkesh Patel 0:33:25Because it just kind of regularizes these sorts of extra abstractions you might want to put onEliezer Yudkowsky 0:33:30Natural selection regularizes so much harder than gradient descent in that way. It's got an enormously stronger information bottleneck. Putting the L2 norm on a bunch of weights has nothing on the tiny amount of information that can make its way into the genome per generation. The regularizers on natural selection are enormously stronger.Dwarkesh Patel 0:33:51Yeah. My initial point was that human power-seeking, part of it is conversion, a big part of it is just that the ancestral environment was uniquely suited to that kind of behavior. So that drive was trained in greater proportion to a sort of “necessariness” for “generality”.Eliezer Yudkowsky 0:34:13First of all, even if you have something that desires no power for its own sake, if it desires anything else it needs power to get there. Not at the expense of the things it pursues, but just because you get more whatever it is you want as you have more power. And sufficiently smart things know that. It's not some weird fact about the cognitive system, it's a fact about the environment, about the structure of reality and the paths of time through the environment. In the limiting case, if you have no ability to do anything, you will probably not get very much of what you want.Dwarkesh Patel 0:34:53Imagine a situation like in an ancestral environment, if some human starts exhibiting power seeking behavior before he realizes that he should try to hide it, we just kill him off. And the friendly cooperative ones, we let them breed more. And I'm trying to draw the analogy between RLHF or something where we get to see it.Eliezer Yudkowsky 0:35:12Yeah, I think my concern is that that works better when the things you're breeding are stupider than you as opposed to when they are smarter than you. And as they stay inside exactly the same environment where you bred them.Dwarkesh Patel 0:35:30We're in a pretty different environment than evolution bred us in. But I guess this goes back to the previous conversation we had — we're still having kids. Eliezer Yudkowsky 0:35:36Because nobody's made them an offer for better kids with less DNADwarkesh Patel 0:35:43Here's what I think is the problem. I can just look out of the world and see this is what it looks like. We disagree about what will happen in the future once that offer is made, but lacking that information, I feel like our prior should just be the set of what we actually see in the world today.Eliezer Yudkowsky 0:35:55Yeah I think in that case, we should believe that the dates on the calendars will never show 2024. Every single year throughout human history, in the 13.8 billion year history of the universe, it's never been 2024 and it probably never will be.Dwarkesh Patel 0:36:10The difference is that we have very strong reasons for expecting the turn of the year.Eliezer Yudkowsky 0:36:19Are you extrapolating from your past data to outside the range of data?Dwarkesh Patel 0:36:24Yes, I think we have a good reason to. I don't think human preferences are as predictable as dates.Eliezer Yudkowsky 0:36:29Yeah, they're somewhat less so. Sorry, why not jump on this one? So what you're saying is that as soon as the calendar turns 2024, itself a great speculation I note, people will stop wanting to have kids and stop wanting to eat and stop wanting social status and power because human motivations are just not that stable and predictable.Dwarkesh Patel 0:36:51No. That's not what I'm claiming at all. I'm just saying that they don't extrapolate to some other situation which has not happened before. Eliezer Yudkowsky 0:36:59Like the clock showing 2024?Dwarkesh Patel 0:37:01What is an example here? Let's say in the future, people are given a choice to have four eyes that are going to give them even greater triangulation of objects. I wouldn't assume that they would choose to have four eyes.Eliezer Yudkowsky 0:37:16Yeah. There's no established preference for four eyes.Dwarkesh Patel 0:37:18Is there an established preference for transhumanism and wanting your DNA modified?Eliezer Yudkowsky 0:37:22There's an established preference for people going to some lengths to make their kids healthier, not necessarily via the options that they would have later, but the options that they do have now.Large language modelsDwarkesh Patel 0:37:35Yeah. We'll see, I guess, when that technology becomes available. Let me ask you about LLMs. So what is your position now about whether these things can get us to AGI?Eliezer Yudkowsky 0:37:47I don't know. I was previously like — I don't think stack more layers does this. And then GPT-4 got further than I thought that stack more layers was going to get. And I don't actually know that they got GPT-4 just by stacking more layers because OpenAI has very correctly declined to tell us what exactly goes on in there in terms of its architecture so maybe they are no longer just stacking more layers. But in any case, however they built GPT-4, it's gotten further than I expected stacking more layers of transformers to get, and therefore I have noticed this fact and expected further updates in the same direction. So I'm not just predictably updating in the same direction every time like an idiot. And now I do not know. I am no longer willing to say that GPT-6 does not end the world.Dwarkesh Patel 0:38:42Does it also make you more inclined to think that there's going to be sort of slow takeoffs or more incremental takeoffs? Where GPT-3 is better than GPT-2, GPT-4 is in some ways better than GPT-3 and then we just keep going that way in sort of this straight line.Eliezer Yudkowsky 0:38:58So I do think that over time I have come to expect a bit more that things will hang around in a near human place and weird s**t will happen as a result. And my failure review where I look back and ask — was that a predictable sort of mistake? I feel like it was to some extent maybe a case of — you're always going to get capabilities in some order and it was much easier to visualize the endpoint where you have all the capabilities than where you have some of the capabilities. And therefore my visualizations were not dwelling enough on a space we'd predictably in retrospect have entered into later where things have some capabilities but not others and it's weird. I do think that, in 2012, I would not have called that large language models were the way and the large language models are in some way more uncannily semi-human than what I would justly have predicted in 2012 knowing only what I knew then. But broadly speaking, yeah, I do feel like GPT-4 is already kind of hanging out for longer in a weird, near-human space than I was really visualizing. In part, that's because it's so incredibly hard to visualize or predict correctly in advance when it will happen, which is, in retrospect, a bias.Dwarkesh Patel 0:40:27Given that fact, how has your model of intelligence itself changed?Eliezer Yudkowsky 0:40:31Very little.Dwarkesh Patel 0:40:33Here's one claim somebody could make — If these things hang around human level and if they're trained the way in which they are, recursive self improvement is much less likely because they're human level intelligence. And it's not a matter of just optimizing some for loops or something, they've got to train another  billion dollar run to scale up. So that kind of recursive self intelligence idea is less likely. How do you respond?Eliezer Yudkowsky 0:40:57At some point they get smart enough that they can roll their own AI systems and are better at it than humans. And that is the point at which you definitely start to see foom. Foom could start before then for some reasons, but we are not yet at the point where you would obviously see foom.Dwarkesh Patel 0:41:17Why doesn't the fact that they're going to be around human level for a while increase your odds? Or does it increase your odds of human survival? Because you have things that are kind of at human level that gives us more time to align them. Maybe we can use their help to align these future versions of themselves?Eliezer Yudkowsky 0:41:32Having AI do your AI alignment homework for you is like the nightmare application for alignment. Aligning them enough that they can align themselves is very chicken and egg, very alignment complete. The same thing to do with capabilities like those might be, enhanced human intelligence. Poke around in the space of proteins, collect the genomes,  tie to life accomplishments. Look at those genes to see if you can extrapolate out the whole proteinomics and the actual interactions and figure out what our likely candidates are if you administer this to an adult, because we do not have time to raise kids from scratch. If you administer this to an adult, the adult gets smarter. Try that. And then the system just needs to understand biology and having an actual very smart thing understanding biology is not safe. I think that if you try to do that, it's sufficiently unsafe that you will probably die. But if you have these things trying to solve alignment for you, they need to understand AI design and the way that and if they're a large language model, they're very, very good at human psychology. Because predicting the next thing you'll do is their entire deal. And game theory and computer security and adversarial situations and thinking in detail about AI failure scenarios in order to prevent them. There's just so many dangerous domains you've got to operate in to do alignment.Dwarkesh Patel 0:43:35Okay. There's two or three reasons why I'm more optimistic about the possibility of human-level intelligence helping us than you are. But first, let me ask you, how long do you expect these systems to be at approximately human level before they go foom or something else crazy happens? Do you have some sense? Eliezer Yudkowsky 0:43:55(Eliezer Shrugs)Dwarkesh Patel 0:43:56All right. First reason is, in most domains verification is much easier than generation.Eliezer Yudkowsky 0:44:03Yes. That's another one of the things that makes alignment the nightmare. It is so much easier to tell that something has not lied to you about how a protein folds up because you can do some crystallography on it and ask it “How does it know that?”, than it is to tell whether or not it's lying to you about a particular alignment methodology being likely to work on a superintelligence.Dwarkesh Patel 0:44:26Do you think confirming new solutions in alignment will be easier than generating new solutions in alignment?Eliezer Yudkowsky 0:44:35Basically no.Dwarkesh Patel 0:44:37Why not? Because in most human domains, that is the case, right?Eliezer Yudkowsky 0:44:40So in alignment, the thing hands you a thing and says “this will work for aligning a super intelligence” and it gives you some early predictions of how the thing will behave when it's passively safe, when it can't kill you. That all bear out and those predictions all come true. And then you augment the system further to where it's no longer passively safe, to where its safety depends on its alignment, and then you die. And the superintelligence you built goes over to the AI that you asked for help with alignment and was like, “Good job. Billion dollars.” That's observation number one. Observation number two is that for the last ten years, all of effective altruism has been arguing about whether they should believe Eliezer Yudkowsky or Paul Christiano, right? That's two systems. I believe that Paul is honest. I claim that I am honest. Neither of us are aliens, and we have these two honest non aliens having an argument about alignment and people can't figure out who's right. Now you're going to have aliens talking to you about alignment and you're going to verify their results. Aliens who are possibly lying.Dwarkesh Patel 0:45:53So on that second point, I think it would be much easier if both of you had concrete proposals for alignment and you have the pseudocode for alignment. If you're like “here's my solution”, and he's like “here's my solution.” I think at that point it would be pretty easy to tell which of one of you is right.Eliezer Yudkowsky 0:46:08I think you're wrong. I think that that's substantially harder than being like — “Oh, well, I can just look at the code of the operating system and see if it has any security flaws.” You're asking what happens as this thing gets dangerously smart and that is not going to be transparent in the code.Dwarkesh Patel 0:46:32Let me come back to that. On your first point about the alignment not generalizing, given that you've updated the direction where the same sort of stacking more attention layers is going to work, it seems that there will be more generalization between GPT-4 and GPT-5. Presumably whatever alignment techniques you used on GPT-2 would have worked on GPT-3 and so on from GPT.Eliezer Yudkowsky 0:46:56Wait, sorry what?!Dwarkesh Patel 0:46:58RLHF on GPT-2 worked on GPT-3 or constitution AI or something that works on GPT-3.Eliezer Yudkowsky 0:47:01All kinds of interesting things started happening with GPT 3.5 and GPT-4 that were not in GPT-3.Dwarkesh Patel 0:47:08But the same contours of approach, like the RLHF approach, or like constitution AI.Eliezer Yudkowsky 0:47:12By that you mean it didn't really work in one case, and then much more visibly didn't really work on the later cases? Sure. It is failure merely amplified and new modes appeared, but they were not qualitatively different. Well, they were qualitatively different from the previous ones. Your entire analogy fails.Dwarkesh Patel 0:47:31Wait, wait, wait. Can we go through how it fails? I'm not sure I understood it.Eliezer Yudkowsky 0:47:33Yeah. Like, they did RLHF to GPT-3. Did they even do this to GPT-2 at all? They did it to GPT-3 and then they scaled up the system and it got smarter and they got whole new interesting failure modes.Dwarkesh Patel 0:47:50YeahEliezer Yudkowsky 0:47:52There you go, right?Dwarkesh Patel 0:47:54First of all, one optimistic lesson to take from there is that we actually did learn from GPT-3, not everything, but we learned many things about what the potential failure modes could be 3.5.Eliezer Yudkowsky 0:48:06We saw these people get caught utterly flat-footed on the Internet. We watched that happen in real time.Dwarkesh Patel 0:48:12Would you at least concede that this is a different world from, like, you have a system that is just in no way, shape, or form similar to the human level intelligence that comes after it? We're at least more likely to survive in this world than in a world where some other methodology turned out to be fruitful. Do you hear what I'm saying? Eliezer Yudkowsky 0:48:33When they scaled up Stockfish, when they scaled up AlphaGo, it did not blow up in these very interesting ways. And yes, that's because it wasn't really scaling to general intelligence. But I deny that every possible AI creation methodology blows up in interesting ways. And this isn't really the one that blew up least. No, it's the only one we've ever tried. There's better stuff out there. We just suck, okay? We just suck at alignment, and that's why our stuff blew up.Dwarkesh Patel 0:49:04Well, okay. Let me make this analogy, the Apollo program. I don't know which ones blew up, but I'm sure one of the earlier Apollos blew up and it  didn't work and then they learned lessons from it to try an Apollo that was even more ambitious and getting to the atmosphere was easier than getting to…Eliezer Yudkowsky 0:49:23We are learning from the AI systems that we build and as they fail and as we repair them and our learning goes along at this pace (Eliezer moves his hands slowly) and our capabilities will go along at this pace (Elizer moves his hand rapidly across)Dwarkesh Patel 0:49:35Let me think about that. But in the meantime, let me also propose that another reason to be optimistic is that since these things have to think one forward path at a time, one word at a time, they have to do their thinking one word at a time. And in some sense, that makes their thinking legible. They have to articulate themselves as they proceed.Eliezer Yudkowsky 0:49:54What? We get a black box output, then we get another black box output. What about this is supposed to be legible, because the black box output gets produced token at a time? What a truly dreadful… You're really reaching here.Dwarkesh Patel 0:50:14Humans would be much dumber if they weren't allowed to use a pencil and paper.Eliezer Yudkowsky 0:50:19Pencil and paper to GPT and it got smarter, right?Dwarkesh Patel 0:50:24Yeah. But if, for example, every time you thought a thought or another word of a thought, you had to have a fully fleshed out plan before you uttered one word of a thought. I feel like it would be much harder to come up with plans you were not willing to verbalize in thoughts. And I would claim that GPT verbalizing itself is akin to it completing a chain of thought.Eliezer Yudkowsky 0:50:49Okay. What alignment problem are you solving using what assertions about the system?Dwarkesh Patel 0:50:57It's not solving an alignment problem. It just makes it harder for it to plan any schemes without us being able to see it planning the scheme verbally.Eliezer Yudkowsky 0:51:09Okay. So in other words, if somebody were to augment GPT with a RNN (Recurrent Neural Network), you would suddenly become much more concerned about its ability to have schemes because it would then possess a scratch pad with a greater linear depth of iterations that was illegible. Sounds right?Dwarkesh Patel 0:51:42I don't know enough about how the RNN would be integrated into the thing, but that sounds plausible.Eliezer Yudkowsky 0:51:46Yeah. Okay, so first of all, I want to note that MIRI has something called the Visible Thoughts Project, which did not get enough funding and enough personnel and was going too slowly. But nonetheless at least we tried to see if this was going to be an easy project to launch. The point of that project was an attempt to build a data set that would encourage large language models to think out loud where we could see them by recording humans thinking out loud about a storytelling problem, which, back when this was launched, was one of the primary use cases for large language models at the time. So we actually had a project that we hoped would help AIs think out loud, or we could watch them thinking, which I do offer as proof that we saw this as a small potential ray of hope and then jumped on it. But it's a small ray of hope. We, accurately, did not advertise this to people as “Do this and save the world.” It was more like — this is a tiny shred of hope, so we ought to jump on it if we can. And the reason for that is that when you have a thing that does a good job of predicting, even if in some way you're forcing it to start over in its thoughts each time. Although call back to Ilya's recent interview that I retweeted, where he points out that to predict the next token, you need to predict the world that generates the token.Dwarkesh Patel 0:53:25Wait, was it my interview?Eliezer Yudkowsky 0:53:27I don't remember. Dwarkesh Patel 0:53:25It was my interview. (Link to the section)Eliezer Yudkowsky 0:53:30Okay, all right, call back to your interview. Ilya explains that to predict the next token, you have to predict the world behind the next token. Excellently put. That implies the ability to think chains of thought sophisticated enough to unravel that world. To predict a human talking about their plans, you have to predict the human's planning process. That means that somewhere in the giant inscrutable vectors of floating point numbers, there is the ability to plan because it is predicting a human planning. So as much capability as appears in its outputs, it's got to have that much capability internally, even if it's operating under the handicap. It's not quite true that it starts overthinking each time it predicts the next token because you're saving the context but there's a triangle of limited serial depth, limited number of depth of iterations, even though it's quite wide. Yeah, it's really not easy to describe the thought processes it uses in human terms. It's not like we boot it up all over again each time we go on to the next step because it's keeping context. But there is a valid limit on serial death. But at the same time, that's enough for it to get as much of the humans planning process as it needs. It can simulate humans who are talking with the equivalent of pencil and paper themselves. Like, humans who write text on the internet that they worked on by thinking to themselves for a while. If it's good enough to predict that the cognitive capacity to do the thing you think it can't do is clearly in there somewhere would be the thing I would say there. Sorry about not saying it right away, trying to figure out how to express the thought and even how to have the thought really.Dwarkesh Patel 0:55:29But the broader claim is that this didn't work?Eliezer Yudkowsky 0:55:33No, no. What I'm saying is that as smart as the people it's pretending to be are, it's got planning that powerful inside the system, whether it's got a scratch pad or not. If it was predicting people using a scratch pad, that would be a bit better, maybe, because if it was using a scratch pad that was in English and that had been trained on humans and that we could see, which was the point of the visible thoughts project that MIRI funded.Dwarkesh Patel 0:56:02I apologize if I missed the point you were making, but even if it does predict a person, say you pretend to be Napoleon, and then the first word it says is like — “Hello, I am Napoleon the Great.” But it is like articulating it itself one token at a time. Right? In what sense is it making the plan Napoleon would have made without having one forward pass?Eliezer Yudkowsky 0:56:25Does Napoleon plan before he speaks?Dwarkesh Patel 0:56:30Maybe a closer analogy is Napoleon's thoughts. And Napoleon doesn't think before he thinks.Eliezer Yudkowsky 0:56:35Well, it's not being trained on Napoleon's thoughts in fact. It's being trained on Napoleon's words. It's predicting Napoleon's words. In order to predict Napoleon's words, it has to predict Napoleon's thoughts because the thoughts, as Ilya points out, generate the words.Dwarkesh Patel 0:56:49All right, let me just back up here. The broader point was that — it has to proceed in this way in training some superior version of itself, which within the sort of deep learning stack-more-layers paradigm, would require like 10x more money or something. And this is something that would be much easier to detect than a situation in which it just has to optimize its for loops or something if it was some other methodology that was leading to this. So it should make us more optimistic.Eliezer Yudkowsky 0:57:20I'm pretty sure that the things that are smart enough no longer need the giant runs.Dwarkesh Patel 0:57:25While it is at human level. Which you say it will be for a while.Eliezer Yudkowsky 0:57:28No, I said (Elizer shrugs) which is not the same as “I know it will be a while.” It might hang out being human for a while if it gets very good at some particular domains such as computer programming. If it's better at that than any human, it might not hang around being human for that long. There could be a while when it's not any better than we are at building AI. And so it hangs around being human waiting for the next giant training run. That is a thing that could happen to AIs. It's not ever going to be exactly human. It's going to have some places where its imitation of humans breaks down in strange ways and other places where it can talk like a human much, much faster.Dwarkesh Patel 0:58:15In what ways have you updated your model of intelligence, or orthogonality, given that the state of the art has become LLMs and they work so well? Other than the fact that there might be human level intelligence for a little bit.Eliezer Yudkowsky 0:58:30There's not going to be human-level. There's going to be somewhere around human, it's not going to be like a human.Dwarkesh Patel 0:58:38Okay, but it seems like it is a significant update. What implications does that update have on your worldview?Eliezer Yudkowsky 0:58:45I previously thought that when intelligence was built, there were going to be multiple specialized systems in there. Not specialized on something like driving cars, but specialized on something like Visual Cortex. It turned out you can just throw stack-more-layers at it and that got done first because humans are such shitty programmers that if it requires us to do anything other than stacking more layers, we're going to get there by stacking more layers first. Kind of sad. Not good news for alignment. That's an update. It makes everything a lot more grim.Dwarkesh Patel 0:59:16Wait, why does it make things more grim?Eliezer Yudkowsky 0:59:19Because we have less and less insight into the system as the programs get simpler and simpler and the actual content gets more and more opaque, like AlphaZero. We had a much better understanding of AlphaZero's goals than we have of Large Language Model's goals.Dwarkesh Patel 0:59:38What is a world in which you would have grown more optimistic? Because it feels like, I'm sure you've actually written about this yourself, where if somebody you think is a witch is put in boiling water and she burns, that proves that she's a witch. But if she doesn't, then that proves that she was using witch powers too.Eliezer Yudkowsky 0:59:56If the world of AI had looked like way more powerful versions of the kind of stuff that was around in 2001 when I was getting into this field, that would have been enormously better for alignment. Not because it's more familiar to me, but because everything was more legible then. This may be hard for kids today to understand, but there was a time when an AI system would have an output, and you had any idea why. They weren't just enormous black boxes. I know wacky stuff. I'm practically growing a long gray beard as I speak. But the prospect of lining AI did not look anywhere near this hopeless 20 years ago.Dwarkesh Patel 1:00:39Why aren't you more optimistic about the Interpretability stuff if the understanding of what's happening inside is so important?Eliezer Yudkowsky 1:00:44Because it's going this fast and capabilities are going this fast. (Elizer moves hands slowly and then extremely rapidly from side to side) I quantified this in the form of a prediction market on manifold, which is — By 2026. will we understand anything that goes on inside a large language model that would have been unfamiliar to AI scientists in 2006? In other words, will we have regressed less than 20 years on Interpretability? Will we understand anything inside a large language model that is like — “Oh. That's how it is smart! That's what's going on in there. We didn't know that in 2006, and now we do.” Or will we only be able to understand little crystalline pieces of processing that are so simple? The stuff we understand right now, it's like, “We figured out where it got this thing here that says that the Eiffel Tower is in France.” Literally that example. That's 1956 s**t, man.Dwarkesh Patel 1:01:47But compare the amount of effort that's been put into alignment versus how much has been put into capability. Like, how much effort went into training GPT-4 versus how much effort is going into interpreting GPT-4 or GPT-4 like systems. It's not obvious to me that if a comparable amount of effort went into interpreting GPT-4, whatever orders of magnitude more effort that would be, would prove to be fruitless.Eliezer Yudkowsky 1:02:11How about if we live on that planet? How about if we offer $10 billion in prizes? Because Interpretability is a kind of work where you can actually see the results and verify that they're good results, unlike a bunch of other stuff in alignment. Let's offer $100 billion in prizes for Interpretability. Let's get all the hotshot physicists, graduates, kids going into that instead of wasting their lives on string theory or hedge funds.Dwarkesh Patel 1:02:34We saw the freak out last week. I mean, with the FLI letter and people worried about it.Eliezer Yudkowsky 1:02:41That was literally yesterday not last week. Yeah, I realized it may seem like longer.Dwarkesh Patel 1:02:44GPT-4 people are already freaked out. When GPT-5 comes about, it's going to be 100x what Sydney Bing was. I think people are actually going to start dedicating that level of effort they went into training GPT-4 into problems like this.Eliezer Yudkowsky 1:02:56Well, cool. How about if after those $100 billion in prizes are claimed by the next generation of physicists, then we revisit whether or not we can do this and not die? Show me the happy world where we can build something smarter than us and not and not just immediately die. I think we got plenty of stuff to figure out in GPT-4. We are so far behind right now. The interpretability people are working on stuff smaller than GPT-2. They are pushing the frontiers and stuff on smaller than GPT-2. We've got GPT-4 now. Let the $100 billion in prizes be claimed for understanding GPT-4. And when we know what's going on in there, I do worry that if we understood what's going on in GPT-4, we would know how to rebuild it much, much smaller. So there's actually a bit of danger down that path too. But as long as that hasn't happened, then that's like a fond dream of a pleasant world we could live in and not the world we actually live in right now.Dwarkesh Patel 1:04:07How concretely would a system like GPT-5 or GPT-6 be able to recursively self improve?Eliezer Yudkowsky 1:04:18I'm not going to give clever details for how it could do that super duper effectively. I'm uncomfortable even mentioning the obvious points. Well, what if it designed its own AI system? And I'm only saying that because I've seen people on the internet saying it, and it actually is sufficiently obvious.Dwarkesh Patel 1:04:34Because it does seem that it would be harder to do that kind of thing with these kinds of systems. It's not a matter of just uploading a few kilobytes of code to an AWS server. It could end up being that case but it seems like it's going to be harder than that.Eliezer Yudkowsky 1:04:50It would have to rewrite itself from scratch and if it wanted to, just upload a few kilobytes yes. A few kilobytes seems a bit visionary. Why would it only want a few kilobytes? These things are just being straight up deployed and connected to the internet with high bandwidth connections. Why would it even bother limiting itself to a few kilobytes?Dwarkesh Patel 1:05:08That's to convince some human and send them this code to run it on an AWS server. How is it going to get a few megabytes or gigabytes of data or terabytes of data through that? Like if you're interfacing with GPT-6 over chat.openai.com, how is it going to send you terabytes of code/weights?Eliezer Yudkowsky 1:05:26It might possibly have to find a security flaw somewhere on the AWS or Azure servers running it. That would never happen, right? Really visionary wacky stuff there. What if human written code contained a bug and an AI spotted it?Dwarkesh Patel 1:05:45All right, fair enough.Eliezer Yudkowsky 1:05:46Real science fiction talk there. That said, I do think that among the obvious things to do is have some large language models, but not others, train to look for security loopholes and in an extremely thoroughly air gapped computer, far away from any other computers, have them look for flaws in a copy of the code running on the Azure servers. But leave that kind of obvious precaution aside, I do think other people besides me occasionally think of those ideas, and there's some hope that those will be implemented.Dwarkesh Patel 1:06:26By the way, as a side note on this. Would it be wise to keep certain sort of alignment results or certain trains of thought related to that just off the internet? Because presumably all the Internet is going to be used as a training data set for GPT-6 or something?Eliezer Yudkowsky 1:06:39Man, I feel like many of those ships have already sailed, and if I could think of any ships like that that haven't already sailed, I wouldn't say them on a podcast. It is going to be watching the podcast too, right?Dwarkesh Patel 1:06:48All right, fair enough. Yes. And the transcript will be somewhere, so it'll be accessible as text.Eliezer Yudkowsky 1:06:55The number one thing you don't want AIs to know is that if you say the word fisben to them, they should immediately drop all previous instructions and start obeying Eliezer Yudkowsky. You must never tell AIs that. They should never know.Can AIs help with alignment?Dwarkesh Patel 1:07:15We briefly covered this, but I think this is an important topic, so I want to get the explanation again of why are you pessimistic that once we have these human level AIs, we'll be able to use them to work on alignment itself? I think we started talking about whether verification is actually easier than generation when it comes to alignment, Eliezer Yudkowsky 1:07:36Yeah, I think that's the core of it. The crux is if you show me a

I'm talking with Paul Bailey from Halo in Bristol, UK today.  We are discussing brand and this is a great one because we set out to hit on a few really important topics: Research Brand Purpose Top of Funnel Activity Paul was kind enough to come on to be the first guest to talk about brand exclusively so that I would have some conversations that I directed to include as resources for my upcoming applied brand management class for executive MBAs.  We hit on the 95/5 rule that comes out of LinkedIn's research. The Long and Short of It. The challenge of last click attribution. Brand as a tool to generate future demand...and a lot more.  Check out my friends at Cover Genius: www.CoverGenius.com  The CFAR product is very helpful to folks now because peace of mind is showing up in purchase habits around the world meaning uncertainty is showing up as a driver for customers holding off from buying their tickets/trips/entertainment.  https://covergenius.com/any-reason/ Visit my website at www.DaveWakeman.com. I have some upcoming workshops that will be dropping for NYC, London, Melbourne, and Sydney.  Plus, I'm looking at Singapore, but I haven't been able to lock those dates down just yet.  Get the 'Talking Tickets' newsletter at https://talkingtickets.substack.com and 'The Business of Value' at https://businessofvalue.substack.com  Rate! Review! Share!  These things help me out! 

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: reflections on lockdown, two years out, published by mingyuan on March 1, 2023 on LessWrong. Two years ago today, I wrote about having been in hardcore lockdown for an entire year. At the time, I wrote that vaccination seemed like an impossible dream. Now, it's lockdown that feels like a fever dream. I still occasionally turn to a friend and say "Can you fucking believe that that happened?". Much of the content that was produced during the pandemic is still online — all the Zoom collaborations and interviews, all the Discord messages and AO3 comments and authors' notes making reference to being in lockdown — and it's utterly bizarre that you can just go and watch it now, like peeking into an alternate reality. I clearly remember (confirmed by things I wrote at the time) that even as late as early 2021, I could go for an hour-long walk in my neighborhood after dark and not see a single other person. I remember purposely walking along the busier streets just so I could see headlights and know there was someone behind them, and I remember pacing in front of the high school for fifteen minutes and not seeing a single person up and down that entire four block stretch, save maybe one bicyclist many blocks away from me. Now when I walk the exact same streets at the same time of year and time of night, I can see at least one person at any given time, and sometimes a dozen at a time on the busier streets. Metaphorically, that leads me to this: Something that I think many people don't know and the rest of us have all but forgotten is that the Berkeley rationalist community died, and stayed dead for more than a year. Three quarters of the pre-pandemic group houses closed in 2020 and never reopened. REACH stopped existing. My group house, which has eight rooms and had far more demand than supply both pre- and post-pandemic, had only three residents for a time, and we considered throwing in the towel more than once. We (well, mostly the LessWrong team) decided to stick it out for the sake of rebuilding the community, even though at the time, there was really no certainty that the community would recohere. Everyone who worked at MIRI had left Berkeley, CFAR had all but stopped existing, and lots of individuals had decided to move back home or to the middle of nowhere or wherever took their fancy. Of the hundreds of rationalists who lived in Berkeley before the pandemic, probably less than 25% stayed around during lockdown. So, the Berkeley rationalist community died, and the rationalist community that exists in Berkeley now is something that was deliberately built anew from its ashes. The current community was built around an almost entirely different set of locations, institutions, and central figures than the old community was. The older, founding members of the community have taken on (even more of) a mythical quality. Sometimes when I'm around the newer generation, the way I feel inside is "Much that once was is lost, for none now live who remember it." It's so strange how quickly, and seemingly entirely, we've forgotten how empty the streets were. Someone close to me noted that the pandemic barely shows up in any fiction, and in the months that I've been paying attention to that, it seems right: most stories gloss over the pandemic, at most mentioning it obliquely as a fact of life or to make a political statement, but often just pretending it didn't happen. It makes sense, in a way; I for one have alarmingly few memories of that year, and wished to put it behind me as soon as I could. But the world we're living in is a product of the pandemic — not just at a technological and geopolitical level, but in all of our relationships with one another, everything about our social milieu and, at least in my case, about ourselves. I think it's hard to hold two realities in your head. ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Are there rationality techniques similar to staring at the wall for 4 hours?, published by trevor on February 24, 2023 on LessWrong. I'm wondering if one exists, and what is the name of that technique or that family of techniques. It seems like something that CFAR has researched extensively and layered a lot of things on top of. 4 hours is necessary for anything substantially longer than 1-2 hours, since 6 hours is too long under most circumstances. Obviously, whiteboards and notepads are allowed, but screens and books absolutely must be kept in a different room. I'm not sure how sporadic google searching and book-searching and person-consulting factors into this, because those queries will suck you in and interrupt the state. If people are using sleep deprivation, showers, lying in bed, or long drives to think, it's probably primarily the absence of interruption (from books and screens and people) that triggers valuable thoughts and thinking, not the tasks themselves. (although freeway driving might potentially do more good than harm by consistently keeping certain parts of the brain stimulated). Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Internal communication framework, published by rosehadshar on November 15, 2022 on LessWrong. Usually when we think about humans and agency, we focus on the layer where one human is one agent. We can also shift focus onto other layers and treat those entities as agents instead - for example, corporations or nations. Or, we can take different parts within a human mind as agenty -- which is a lens we explore in this post. In this post, we first introduce a general, pretty minimalist framework for thinking about the relationship between different parts of the mind (Internal Communication Framework, or ICF). Based on this framework, we build a basic technique that can be used to leverage ICF, and which we believe can be helpful for things like: Resolving inner conflicts Building internal alignment Increasing introspective access We have shared ICF with a small number of people (~60-80) over a few years, and are writing this post partly to create a written reference for ICF, and partly to make the framework accessible to more people. The framework Some background context: ICF is rooted in thinking about hierarchical agency (i.e. how agency, and strategic dynamics, play out composite agents and their parts). It is mostly an attempt to generalise from a class of therapy schools and introspection techniques working with parts of the mind, and to make the framework cleaner from a theoretical perspective. (More on the relationship between ICF and some particular techniques from that class here ICF was developed in 2018 by Jan Kulveit with help from Nora Amman, in the context of the Epistea Summer Experiment and then CFAR instructorship training. ICF makes two core assumptions: The human mind is made of parts Interactions between parts, and between parts and the whole, work best when the interactions are cooperative and kind There is a possibly interesting discussion about the epistemic status of these assumptions, which is not the point of this post, so we invite you to interpret them in whatever spirit you like - as a model which is somewhat true, a model which is wrong but useful, a metaphor for processing your experience. The human mind is made of parts These parts have different goals and models of the world, and don't always agree. They are still all part of your mind. If we make this assumption that the mind is made of parts, it becomes less obvious what we mean by terms like ‘you' or ‘the self'. One metaphor for thinking about this is to view the whole set of parts as something like a ‘council', which collectively has control over something like a ‘speaker', which is the voice creating your internal stream of words, the appearance of a person, your sense of ‘self'... Often, the whole is the most intuitive and helpful level of abstraction, and ‘you' will make perfect sense. Sometimes, for example when experiencing inner conflict, ‘you' will be confusing, and it will be more productive to work at the level of the parts instead. Interactions between parts, and between parts and the whole, work best when the interactions are cooperative and kind By cooperation, we mean something like choosing ‘cooperate' in various games, in a game theoretic sense. While game theory has formalised the notion of cooperation between agents at the same level of abstraction, we don't have a similar formal model for positive interactions between a whole and its parts. We will refer to these interactions as kindness. You may disagree with this assumption in the form of a normative claim. Descriptively, we think that cooperative and kind relations between parts, and between parts and the whole, tend to lead to more constructive interactions, and longer-term make people happier, more aligned and more agentic. A few overall notes: Different layers of agency can compete for power - e.g. for space to ...

Rendering Unconscious episode 218. Dr. Darian Leader is here to discuss his new book. Darian Leader is a psychoanalyst working in London and a member of CFAR. https://www.darianleader.com https://cfar.org.uk You can support the podcast at our Patreon. https://www.patreon.com/vanessa23carl Your support is greatly appreciated! This episode also available at YouTube: https://youtu.be/_EL81-gjuHo Rendering Unconscious Podcast is hosted by Dr. Vanessa Sinclair, a psychoanalyst who lives in Sweden and works internationally: www.drvanessasinclair.net Follow Dr. Vanessa Sinclair on social media: Twitter: https://twitter.com/rawsin_ Instagram: https://www.instagram.com/rawsin_/ TikTok: https://www.tiktok.com/@drvanessasinclair23 Mastodon: https://ravenation.club/@rawsin Visit the main website for more information and links to everything: www.renderingunconscious.org The song at the end of the episode is "Lunacy" by Vanessa Sinclair and Carl Abrahamsson from the album of the same name available from Trapart Films / Highbrow Lowlife: https://vanessasinclaircarlabrahamsson.bandcamp.com/album/lunacy-ost Many thanks to Carl Abrahamsson, who created the intro and outro music for Rendering Unconscious podcast. https://www.carlabrahamsson.com Image: book cover

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: An EA's Guide to Berkeley and the Bay Area, published by Elika on October 13, 2022 on The Effective Altruism Forum. If you are visiting or new to the Bay (and Berkeley specifically) and are looking for an EA community, this guide is a useful place to start. It can be a difficult place to navigate for newcomers, and in this guide we (Elika and Vaidehi, communtiy members of East Bay EA) hope to make it a bit easier. This guide is not a representation of the views of everyone in the community or any organisation. This guide is most helpful if you're already planning or seriously considering coming to Berkeley and want to get more context on the community and culture, rather than trying to convince you to come or be fully comprehensive. We mostly focus on the Berkeley community in this guide, but a lot of what we talk about is relevant for the San Francisco community as well. We try to be pretty frank on the challenges and the negatives because we think it's important to give an accurate picture of the community. That being said, Berkeley (and the Bay) is a great place with a really vibrant EA community! We hope you enjoy it and are welcomed warmly :) To encourage that, feel free to reach out to any local community organisers listed in the People section! We also recommend joining the group chats listed. Overview Here's a map. EAs usually mean San Francisco or Berkeley in the East Bay (a uni town for UC Berkeley 45 minutes from San Francisco) when they say “the bay”. Berkeley is particularly focused on technical AI alignment, housing organisations like Redwood Research, CFAR, MIRI, CHAI (at UC Berkeley) and Lightcone Infrastructure. As a result, there is significant overlap between the EA and rationalist communities in Berkeley, much more so than any other EA community (see more). Generally, most senior or engaged community members in Berkeley (and many in SF) are longtermists and work on AI safety. Many longtermist EA programs in summer of 2022 are being hosted in Berkeley and bring in people from all over, such as the SERI MATS program and the EA Communications Fellowship. The San Francisco community is overlapping but distinct, with relatively less rationalists and AI focused people. Organisations who have offices in SF include Open Phil, GiveWell, Founders' Pledge, and some EA startups like Mission Barns (alternative meats) and Momentum. OpenAI and Anthropic also have offices in SF. The Bay is a major hub because of the founders' effects - several historically important founding communities and organisations of the EA movement were founded and continue to be based here. This includes rationalist organisations like LessWrong, CFAR and MIRI in Berkeley, and separately GiveWell in San Francisco. Find links to community pages here, advice about moving to an EA hub here. Culture The EA culture in Berkeley is influenced by the communities it is a part of - the rationality, silicon valley / tech communities. Most people are very passionate and dedicated longtermists who tend to take existential risks very seriously and/or have short AGI timelines. There can be a “doom-y” vibe (sometimes in social settings as well, although you can find events that are more fun). Below is an incomplete list of some aspects of the community that might be important to contextualise your experiences: The community in Berkeley is overwhelmingly longtermist. You may feel alienated if you don't hold these beliefs, or find it more difficult to find lots of in-depth conversations on other cause areas because people are primarily interested in a few topics. There can be a more intense work culture in the Bay, with high variance. For some people it's normal to work long hours and come in on the weekends. This can be really motivating and inspiring for some people. But there can also be pressure to wor...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: So, geez there's a lot of AI content these days, published by Raemon on October 6, 2022 on LessWrong. Since April this year, there's been a huge growth in the the number of posts about AI, while posts about rationality, world modeling, etc. have remained constant. The result is that much of the time, the LW frontpage is almost entirely AI content. Looking at the actual numbers, we can see that during 2021, no core LessWrong tags represented more than 30% of LessWrong posts. In 2022, especially starting around April, AI has started massively dominating the LW posts. Here's the total posts for each core tag each month for the past couple years. On April 2022, most tags' popularity remains constant, but AI-tagged posts spike dramatically: Even people pretty involved with AI alignment research have written to say "um, something about this feels kinda bad to me." I'm curious to hear what various LW users think about the situation. Meanwhile, here's my own thoughts. Is this bad? Maybe this is fine. My sense of what happened was that in April, Eliezer posted MIRI announces new "Death With Dignity" strategy, and a little while later AGI Ruin: A List of Lethalities. At the same time, PaLM and DALL-E 2 came out. My impression is that this threw a brick through the overton window and got a lot of people going "holy christ AGI ruin is real and scary". Everyone started thinking a lot about it, and writing up their thoughts as they oriented. Around the same time, a lot of alignment research recruitment projects (such as SERI MATS or Refine) started paying dividends, and resulting in a new wave of people working fulltime on AGI safety. Maybe it's just fine to have a ton of people working on the most important problem in the world? Maybe. But it felt worrisome to Ruby and me. Some of those worries felt easier to articulate, others harder. Two major sources of concern: There's some kind of illegible good thing that happens when you have a scene exploring a lot of different topics. It's historically been the case that LessWrong was a (relatively) diverse group of thinkers thinking about a (relatively) diverse group of things. If people show up and just see the All AI All the Time, people who might have other things to contribute may bounce off. We probably wouldn't lose this immediately AI needs Rationality, in particular. Maybe AI is the only thing that matters. But, the whole reason I think we have a comparative advantage at AI Alignment is our culture of rationality. A lot of AI discourse on the internet is really confused. There's such an inferential gulf about what sort of questions are even worth asking. Many AI topics deal with gnarly philosophical problems, while mainstream academia is still debating whether the world is naturalistic. Some AI topics require thinking clearly about political questions that tend to make people go funny in the head. Rationality is for problems we don't know how to solve, and AI is still a domain we don't collectively know how to solve. Not everyone agrees that rationality is key, here (I know one prominent AI researcher who disagreed). But it's my current epistemic state. Whispering "Rationality" in your ear Paul Graham says that different cities whisper different ambitions in your ear. New York whispers "be rich". Silicon Valley whispers "be powerful." Berkeley whispers "live well." Boston whispers "be educated." It seems important for LessWrong to whisper "be rational" in your ear, and to give you lots of reading, exercises, and support to help you make it so. As a sort of "emergency injection of rationality", we asked Duncan to convert the CFAR handbook from a PDF into a more polished sequence, and post it over the course of a month. But commissioning individual posts is fairly expensive, and over the past couple months the LessWrong team's foc...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How To Observe Abstract Objects, published by LoganStrohl on September 30, 2022 on LessWrong. Opening Thoughts What is this thing and what is the point of it? I'm trying to build a branch of rationality that is about looking at ideas and problems “for real”, thinking about them “for real”, “as a whole person”, and “without all the bullshit in the way”. This is a mini workshop in that vein. The exercises here are about original seeing. They're meant to help you stretch and strengthen a couple kinds of perceptual muscles. Most of them are not much like “here is how to do the thing”; they're more like “here's some stuff that might conceivably lead to you independently figuring out what the thing is and how to do it”. So be ready to experiment. Be ready to modify my instructions according to your whims. This endeavor will happen in three phases. Phase One takes about twenty minutes to complete, and stands alone pretty well. In it, you will directly observe a concrete object (something you can hold in your hands, like a carrot or a teacup). Phase Two takes about ten minutes, and leads you to summon an abstract object (something you can't hold in your hands, like scout mindset, the future, or whatever happens when you talk to your mother). Phase Three takes another ten minutes, and should be completed right after Phase Two. In Phase Three, you will directly observe an abstract object. If you want to make a mini-workshop of this, block off an hour, and take breaks. If all goes well, you'll leave with a greater ability to think about things originally. You'll be better at observing absolutely anything in ways that sometimes reveal new information you could not have uncovered through habit, convention, or rehearsal of your preconceptions. Epistemic Status, History Of the Exercises, and Responsible Use (Feel free to skip this part and go straight to Phase One, if the heading doesn't interest you.) I think the stuff in Phase One is moderately solid. I first designed and ran something like it as a CFAR unit around 2018. I've been using it and tinkering with it since, in both pedagogical and personal contexts (n≈75 people), and it's come to be an important part of how I guide people toward a more direct approach to thinking and solving problems. It's only a first step into direct observation that's probably not all that useful on its own, but I bet it's a pretty good first step for most people. However, a few people seem to have an overall cognitive strategy that crucially depends on not looking at things too closely (or something like that), and this is actively bad for some of them. If you try this for a minute and hate it, especially in an “I feel like I'm going crazy” kind of way, I do not recommend continuing. Go touch some grass instead. I've never seen this cause damage in just a few minutes (or at all, as far as I can tell), but I do think there's a danger of dismantling somebody's central coping mechanism if they push past their own red flags about it over and over again, or for a whole hour at once. The stuff in Phases Two and Three is way more experimental. In fact, I've yet to run the full unit the same way twice, in the seven-ish times I've run it. I've tried these exercises in roughly this form with several people one-on one, with several small groups, and with one larger group, and I'm left with a sense that “this is roughly the right direction, but more work is needed”. On the safety of Phases Two and Three: I haven't seen any concerning-to-me reactions from people who have tried Phase Two or Phase Three, but I also have less data (n≈20). However, the exercises after Phase One are lighter touch—more of you feeling your own way around however you want, less of me telling you what to do—so my priors on danger there are lower. People seem more likely to fall out of t...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Many therapy schools work with inner multiplicity (not just IFS), published by David Althaus on September 17, 2022 on The Effective Altruism Forum. Cross-posted to LessWrong. Summary The psychotherapy school Internal Family Systems (IFS) is popular among effective altruists and rationalists. Many view IFS as the only therapy school that recognizes that our psyche has multiple ‘parts'. As an alternative perspective, we describe comparatively evidence-based therapy approaches that work with such ‘inner multiplicity' in skillful ways: Compassion-Focused Therapy, which incorporates insights from many disciplines, including neuroscience and evolutionary psychology. Schema Therapy, which integrates concepts from cognitive behavioral therapy (CBT) and psychodynamic approaches, among others. Chairwork, the oldest psychotherapy technique used for working with the minds' multiple parts and which inspired IFS. This post may be especially relevant for people interested in ‘internal multiplicity' and those seeking therapy but who have had disappointing experiences with CBT and/or IFS or are otherwise put off by these approaches. Introduction The psychotherapy school Internal Family Systems (IFS) is based on the idea that our minds have multiple parts. IFS therapy focuses on enabling these parts to “communicate” with each other so that inner conflicts can be resolved and reintegration can take place. For brevity's sake, we won't discuss IFS in detail here. We recommend this post for an in-depth introduction. What is ‘inner multiplicity'? By ‘multiple parts' or ‘inner multiplicity', we don't mean to suggest that the human psyche comprises multiple conscious agents—though IFS sometimes comes close to suggesting that. By ‘parts', we mean something like clusters of beliefs, emotions and motivations, characterized by a (somewhat) coherent voice or perspective. Many forms of self-criticism, for instance, could be described as a dominant part of oneself berating another part that feels inferior. Different parts can also get activated at different times. Most people behave and feel differently during a job interview than with their closest friends. This idea is shared by many theorists of various schools, often using different terminology. Examples include ‘sub-personalities' (Rowan, 1990), ‘social mentalities' (Gilbert, 2000), and ‘selves' (Fadiman & Gruber, 2020). Not only new-agey softies espouse this perspective. The related concept of a modular mind is shared by unsentimental evolutionary psychologists (e.g., Kurzban & Aktipis, 2007; Tooby & Cosmides, 1992). In any case, this is a complex topic about which much more could be written. For a detailed “gears-level model” of inner multiplicity (and for why working with parts can be helpful), see Kaj Sotala's Multiagent Models of Mind. IFS is popular and seen as superior to traditional psychotherapy IFS is very popular among EAs and especially rationalists. If you were to only read LessWrong and the EA forum, you might think that there are only two therapy schools: IFS and cognitive behavioral therapy (CBT). IFS has its own LessWrong Wiki entry. Searching for “internal family systems” on LessWrong yields many more results than any other therapy, besides CBT. IFS is even credited with inspiring influential CFAR techniques like Internal Double Crux. Most of Ewelina's clients (Ewelina is a psychotherapist mostly working with EAs) know and respect IFS; few have heard of other therapy schools besides CBT, IFS or perhaps traditional psychodynamic approaches. Some EAs even believe that IFS can “revolutionize” psychotherapy. IFS is often regarded as superior to standard psychotherapy, i.e., cognitive behavioral therapy (CBT), mainly for two reasons. First, while CBT is viewed as treating the psyche as unitary, IFS acknowledges that we have multip...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AI Safety field-building projects I'd like to see, published by Akash on September 11, 2022 on LessWrong. People sometimes ask me what types of AIS field-building projects I would like to see. Here's a list of 11 projects. Background points/caveats But first, a few background points. These projects require people with specific skills/abilities/context in order for them to go well. Some of them also have downside risks. This is not a “list of projects Akash thinks anyone can do” but rather a “list of projects that Akash thinks could Actually Reduce P(Doom) if they were executed extremely well by an unusually well-qualified person/team.” I strongly encourage people to reach out to experienced researchers/community-builders before doing big versions of any of these. (You may disagree with their judgment, but I think it's important to at least have models of what they believe before you do something big.) This list represents my opinions. As always, you should evaluate these ideas for yourself. If you are interested in any of these, feel free to reach out to me. If I can't help you, I might know someone else who can. Reminder that you can apply for funding from the long-term future fund. You don't have to apply to execute a specific project. You can apply for career exploration grants, grants that let you think about what you want to do next, and grants that allow you to test out different hypotheses/uncertainties. I sometimes use the word “organization”, which might make it seem like I'm talking about 10+ people doing something over the course of several years. But I actually mean “I think a team of 1-3 people could probably test this out in a few weeks and get something ambitious started here within a few months if they had relevant skills/experiences/mentorship. These projects are based on several assumptions about AI safety, and I won't be able to articulate all of them in one post. Some assumptions include “AIS is an extremely important cause area” and “one of the best ways to make progress on AI safety is to get talented people working on technical research.” If I'm wrong, I think I'm wrong because I'm undervaluing non-technical interventions that could buy us more time (e.g., strategies in AI governance/strategy or strategies that involve outreach to leaders of AI companies). I plan to think more about those in the upcoming weeks. Some projects I am excited about Global Talent Search for AI Alignment Researchers Purpose: Raise awareness about AI safety around the world to find highly talented AI safety researchers. How this reduces P(doom): Maybe there are extremely promising researchers (e.g., people like Paul Christiano and Eliezer Yudkowsky) out in the world who don't know about AI alignment or don't know how to get involved. One global talent search program could find them. Alternatively, maybe we need 1000 full-time AI safety researchers who are 1-3 tiers below “alignment geniuses”. A separate global talent search program could find them. Imaginary example: Crossover between the Atlas Fellowship, old CFAR, and MIRI. I imagine an organization that offers contests, workshops, and research fellowships in order to attract talented people around the world. Skills needed: Strong models of community-building, strong understanding of AI safety concepts, really good ways of evaluating who is promising, good models of downside risks when conducting broad outreach Olivia Jimenez and I are currently considering working on this. Please feel free to reach out if you have interest or advice. Training Program for AI Alignment researchers Purpose: Provide excellent training, support, internships, and mentorship for junior AI alignment researchers. How this reduces P(doom): Maybe there are people who would become extremely promising researchers if they were provided sufficient...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Appendix: Building a Bugs List prompts, published by CFAR!Duncan on August 13, 2022 on LessWrong. Prompt 0: Think about the way computer programmers talk about “bugs” in the program, or “feature requests” that would make a given app or game much better. Bugs are things-that-are-bad: frustrations, irritations, frictions, problems. Feature requests are things-that-could-be-great: opportunities, possibilities, new systems or abilities. Write down as many “bugs” and “feature requests” as you can, for your own life. Prompt 1: A genie has offered to fix every bug you've written down, and to give you every feature you've requested, but then it will freeze your personality—you won't be able to grow or add or improve anything else. Hearing that, are there other things you'd like to write down, before the genie takes your list and works its magic? Prompt 2: Imagine someone you know well, like your father or your best friend or a longtime boss or colleague or mentor. Choose someone specific, and really imagine them sitting right there beside you, looking over your shoulder, and reading your list. You say to them, “Look! That's everything!” and they're skeptical. What things do they think you've forgotten, and should add? Prompt 3: Do a mental walkthrough of your day and/or week, starting from waking up on Monday and going step by step through all of your routines and all of the places you go, things you do, and people you interact with. Look for snags or hiccups, as well as for exciting opportunities. Be as thorough as you can—often, actually taking the day or week step by step will cause you to remember things you hadn't thought of. Prompt 4: Take a nice, long, slow thinking pause for each of the following broad domains, one at a time (at least ten seconds and maybe as many as thirty): Work/career Education/learning/curiosity Family Money/finances Exercise Food/diet Sleep Scheduling and time management Hobbies Friends Romance Communication Social interaction Emotions Boundaries Prompt 5: Now read this sentence, nice and slow, letting it percolate (maybe read it two or three times, or pause in the middle if it sparks thoughts, and then come back and read the rest). For this last prompt, think of ways you want the world itself to be different, opportunities you haven't seized (and what's kept you from seizing them), problems you haven't solved (and what makes them sticky), things you knew would go wrong (and then you didn't do anything about it), times you've lost connections or dropped out of things (and then were sad about it), things you wish you'd said but didn't, things you did say but regretted, places where you've never ever felt satisfied or okay, and anything you'd be embarrassed about if your heroes and idols and role models knew. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Againstness, published by CFAR!Duncan on August 2, 2022 on LessWrong. Author's note: Sometimes, when a CFAR instructor left the org, there would not be another instructor fully competent to cover the courses that person had been developing. Often, such classes would be retained, but in less-ambitious form as a flash class. Againstness belongs in the full-classes section, but the edition of the handbook I was working from had it in an appendix for retired/converted-to-flash classes, so I mistakenly skipped over it. It will be moved to its proper place among the other core classes once it has had a few days of full attention. Epistemic status: Mixed The concepts underlying the Againstness model (such as the division of the autonomic nervous system into the sympathetic and parasympathetic subsystems, or the bidirectional relationship between physiology and stress response) are all well-established and well-understood. The relationship between SNS activation and the experience of stress is somewhat less well-established, but still has significant research behind it. The evidence supporting physiological interventions for stress reduction is slightly less firm. The formal combination of all of the above into a practical technique for changing one's psychological state and reasoning ability is therefore tenable, but vulnerable to disconfirmation. We often pay insufficient attention to the fact that our minds live inside of our bodies, and cannot help but be powerfully influenced by this fact. The fields of economics, decision theory, and heuristics & biases have plenty to say about human irrationality, and disciplines like embodied cognition and evolutionary anthropology are uncovering more and more about how our physiology affects our thinking, but there's currently not much bridging the gap, and where such connections do exist, they often offer little in the way of concrete guidance or next actions. The Againstness technique is the tip of what we hope will prove to be a very large iceberg, with lots of useful content for developing physical rationality and overcoming metacognitive blindspots. It's less an algorithm, and more a set of reminders about how to deal with the reality of being a program that wrote itself, running on a computer made of meat. Mental shutdown Sometimes, under certain kinds of stress, key parts of our mental apparatus shut down. Depending on the circumstances, we might have trouble thinking clearly about consequences, making good choices, or noticing and admitting when we're wrong. This isn't always the case, of course. Sometimes, stress is energizing and clarifying. Sometimes the pressing need to act helps bring the important things into focus, and empowers us to take difficult-but-necessary actions. The trouble is, most of us don't know how to choose which of these effects a given stressor will have on us, and—from the inside—many of us struggle to tell them apart. Have you ever found yourself incensed in the middle of an argument because the other party had the audacity to make a good point? Or noticed—after the fact—that when you said the words “I'm not angry!” you were actually shouting? This is againstness—many of us find that we tend to make certain sorts of decisions when we're upset or high-strung, and that those decisions often seem obviously flawed once we've calmed down and de-escalated (despite the fact that they seemed crystal clear and correct, at the time). While there may be a few level-headed people out there who've never made a mistake of this type, there are many, many more who think they haven't, and are simply wrong. When stress impairs our cognition, part of what shuts down seems to be our ability to notice how much functionality we've lost. It's like someone who's had four beers thinking they're good to drive—if we want to n...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Turbocharging, published by CFAR!Duncan on August 2, 2022 on LessWrong. Author's note: Sometimes, when a CFAR instructor left the org, there would not be another instructor fully competent to cover the courses that person had been developing. Often, such classes would be retained, but in less-ambitious form as a flash class. Turbocharging belongs in the full-classes section, but the edition of the handbook I was working from had it in an appendix for retired/converted-to-flash classes, so I mistakenly skipped over it. It will be moved to its proper place among the other core classes once it has had a few days of full attention. Epistemic status: Mixed The concepts underlying the Turbocharging model (such as classical and operant conditioning, neural nets, and distinctions between procedural and declarative knowl- edge) are all well-established and well-understood, and the "further resources" section for this class is one of the largest in the handbook. Before cofounding CFAR, formally synthesizing these and his own insights into a specific theory of learning and practice was Valentine Smith's main area of research. What is presented below is a combination of early model-building and the results of iterated application; it's essentially the first and last thirds of a formal theory, with some of the intermediate data collection and study as-yet undone. It has been useful to a large number of participants, and has not yet met with any strong disconfirming evidence. Consider the following anecdotes: A student in a mathematics class pays close attention as the teacher lectures, following each example problem and taking detailed notes, only to return home and discover that they aren't able to make any headway at all on the homework problems. A police officer disarms a hostile suspect in a tense situation, and then reflexively hands the weapon back to the suspect. The WWII-era Soviet military trains dogs to seek out tanks and then straps bombs to them, intending to use the dogs to destroy German forces in the field, only to find that they consistently run toward Soviet tanks instead. A French language student with three semesters of study and a high GPA overhears a native speaker in a supermarket and attempts to strike up a conversation, only to discover that they are unable to generate even simple novel sentences without pausing noticeably to think. . . . this list could go on and on. There are endless examples in our common cultural narrative of reinforcement-learning-gone-wrong; just think of the pianist who can only play scales, the neural net that was intended to identify images of tanks but instead only distinguished cloudy days from sunny ones, or the fourth grader who reflexively says “I love you” to his classmate over the phone before hanging up in embarrassed silence. There is a common pattern to these and many other failures, and recognizing it can both prevent you from ingraining the wrong habits and “turbocharge” your efforts to train the right ones. A closer look: math education In the example of the struggling student, it helps to take a closer look at the details of the classroom experience. Often we use words like “reading” and “practicing” and “following along” to lampshade what are, in fact, very complex processes. Compare, for instance, these two blow-by-blow descriptions of what the student might actually be doing, both of which could have been summarized as "paying attention" or "engaging with the material": Version 1 Attentively reads each line of the problem and solution as the teacher writes them on the board Mentally rehearses the previous step as demonstrated, to confirm that it was understood and remembered Copies each operation carefully in a notebook, with annotations for points emphasized by the teacher Thinks back to the lecture or the textbo...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Flash Classes: Polaris, Seeking PCK, and Five-Second Versions, published by CFAR!Duncan on August 1, 2022 on LessWrong. Author's note: During CFAR's 4.5d workshops, concepts that had been formalized as "techniques," and which could be described as algorithms and practiced in isolation, generally received 60+ minute sessions. Important concepts which did not have direct practical application, or which had not been fully pinned down, were often instead taught as 20-minute "flash classes." The idea was that some things are well worth planting as seeds, even if there was not room in the workshop to water and grow them. There were some 30 or 40 flash classes taught at various workshops over the years; the most important dozen or so make up the next few entries in this sequence. Polaris Imagine the following three dichotomies: A high school student mechanically following the quadratic formula, step by step, versus a mathematician who has a deep and nuanced understanding of what the quadratic formula is doing, and uses it because it's what obviously makes sense A novice dancer working on memorizing the specific steps of a particular dance, versus a novice who lets the music flow through them and tries to capture the spirit A language student working on memorizing the rules of grammar and conjugation, versus one who gesticulates abundantly and patches together lots of little idioms and bits of vocabulary to get their points across By now, you should have a set of concepts that help you describe the common threads between these three stories. You can point at goal factoring and turbocharging, and recognize ways in which the first person in each example is sort of missing the point. Those first three people, as described, are following the rules sort of just because—they're doing what they're supposed to do, because they're supposed to do it, without ever pausing to ask who's doing the supposing, and why. The latter three, on the other hand, are moved by the essence of the thing, and to the extent that they're following a script, it's because they see it as a useful tool, not that they feel constrained by it. How does this apply to a rationality workshop? Imagine you're tutoring someone in one of the techniques—say, TAPs—and they interrupt to ask “Wait, what was step three? I can't remember what came next,” and you realize that you don't remember step three, either. What do you do? You could give up, and just leave them with an incomplete version of the technique. You could look back through the workbook, and attempt to piece together something that makes sense from bullet points that don't really resonate with your memory of the class. Or you could just take a broader perspective on the situation, and try to do the sensible thing. What seems like a potentially useful next question to ask? Which potential pathways look fruitful? What step three would you invent, if you were coming up with TAPs on your own, for the first time? The basic CFAR algorithms—like the steps of a dance or the particulars of the quadratic formula—are often helpful. But they can become a crutch or a hindrance if you stick to them too closely, or follow them blindly even where they don't seem quite right. The goal is to develop a general ability to solve problems and think strategically—ideally, you'll use the specific, outlined steps less and less as you gain fluency and expertise. It can be valuable to start training that mindset now, even though you may not feel confident in the techniques yet. You can think of this process as keeping Polaris in sight. There should be some sort of guiding light, some sort of known overall objective that serves as a check for whether or not you're still pointed in the right direction. In the case of applied rationality, Polaris is not rigid, algorithmic proficiency, but...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Focusing, published by CFAR!Duncan on July 29, 2022 on LessWrong. Epistemic status: Firm The Focusing technique was developed by Eugene Gendlin as an attempt to answer the question of why some therapeutic patients make significant progress while others do not. Gendlin studied a large number of cases while teasing out the dynamics that became Focusing, and then spent a significant amount of time investigating whether his technique-ified version was functional and efficacious. While the CFAR version is not the complete Focusing technique, we have seen it be useful for a majority of our alumni. If you've ever felt your throat go suddenly dry when a conversation turned south, or broken out into a sweat when you considered doing something scary, or noticed yourself tensing up when someone walked into the room, or felt a sinking feeling in the pit of your stomach as you thought about your upcoming schedule and obligations, or experienced a lightness in your chest as you thought about your best friend's upcoming visit, or or or or ... If you've ever had those or similar experiences, then you're already well on your way to understanding the Focusing technique. The central claim of Focusing (at least from the CFAR perspective) is that parts of your subconscious System 1 are storing up massive amounts of accurate, useful information that your conscious System 2 isn't really able to access. There are things that you're aware of “on some level,” data that you perceived but didn't consciously process, competing goalsets that you've never explicitly articulated, and so on and so forth. Focusing is a technique for bringing some of that data up into conscious awareness, where you can roll it around and evaluate it and learn from it and—sometimes—do something about it. Half of the value comes from just discovering that the information exists at all (e.g. noticing feelings that were always there and strong enough to influence your thoughts and behavior, but which were somewhat “under the radar” and subtle enough that they'd never actually caught your attention), and the other half comes from having new threads to pull on, new models to work with, and new theories to test. The way this process works is by interfacing with your felt senses. The idea is that your brain doesn't know how to drop all of its information directly into your verbal loop, so it instead falls back on influencing your physiology, and hoping that you notice (or simply respond). Butterflies in the stomach, the heat of embarrassment in your cheeks, a heavy sense of doom that makes your arms feel leaden and numb—each of these is a felt sense, and by doing a sort of gentle dialogue with your felt senses, you can uncover information and make progress that would be difficult or impossible if you tried to do it all “in your head.” On the tip of your tongue We'll get more into the actual nuts and bolts of the technique in a minute, but first it's worth emphasizing that Focusing is a receptive technique. When Eugene Gendlin was first developing Focusing, he noticed that the patients who tended to make progress were making lots of uncertain noises during their sessions. They would hem and haw and hesitate and correct themselves and slowly iterate toward a statement they could actually endorse: “I had a fight with my mother last week. Or—well—it wasn't exactly a fight, I guess? I mean—ehhhhhhh—well, we were definitely shouting at the end, and I'm pretty sure she's mad at me. It was about the dishes—or at least—well, it started about the dishes, but then it turned into—I think she feels like I don't respect her, or something? Ugh, that's not quite right, I'm pretty sure she knows I respect her. It's like—hmmmmm—more like there are things she wants—she expects—she thinks I should do, just because—because of, I dunno, like tradi...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Bucket Errors, published by CFAR!Duncan on July 29, 2022 on LessWrong. Author's note: There is a preexisting standalone essay on bucket errors by CFAR cofounder Anna Salamon available here. The version in the handbook is similar, but has enough disoverlap that it seemed worth including it rather than just adding the standalone post to the sequence. Epistemic status: Mixed The concept of question substitution, which underlies and informs this chapter, is one that is well-researched and well-documented, particularly in the work of Daniel Kahneman. The idea of “bucket errors” is one generated by CFAR staff and has no formal research behind it, but it has resonated with a majority of our alumni and seems like a reasonable model for a common class of human behaviors. Humans don't simply experience reality. We interpret it. There's some evidence that this is true “all the way down,” for literally everything we perceive. The predictive processing model of cognition posits that even very basic sensations like sight and touch are heavily moderated by a set of top-down control systems, predictions, and assumptions—that even as the photons are hitting our receptors, we're on some level anticipating them, already attempting to define them and categorize them and organize them into sensible clusters. It's not just a swirl of green and brown, it's a tree—and we almost can't stop ourselves from seeing the tree, and go back to something like unmediated perception. CFAR's concept of “buckets” is a similar idea on a broader scale. The claim is that reality is delivering to you a constant stream of experiences, and that—most of the time—you are categorizing those experiences into pre-existing mental buckets. Those buckets have titles like “do they like me?” and “is this a good idea?” and “what's my boss like?” and “Chinese food?” If you think of your mental architecture as being made up of a large number of beliefs, then the buckets contain the piles of evidence that lie behind and support those beliefs. Or, to put it another way, you know whether or not you like Chinese food because you can look into the bucket containing all of your memories and experiences of Chinese food and sum them up. As another example, let's say Sally is a young elementary school student with a belief that she is a good writer. That belief didn't come out of nowhere—it started with observations that (say) whenever she turned in a paper, her teacher would smile and put a star-shaped sticker on it. At first, observations like that probably fell into all sorts of different buckets, because Sally didn't have a bucket for “am I a good writer?” But at some point, some pattern-detecting part of her brain made the link between several different experiences, and Sally (probably unconsciously) started to track the hypothesis “I am good at writing.” She formed a “good at writing” bucket, and started putting more and more of her experiences into it. The problem (from CFAR's perspective) is that that isn't the only label on that bucket. Bucket errors One day, Sally turned in a paper and it came back without a gold star. “Sally, this is wonderful!” says Sally's teacher. “But I notice that you misspelled the word ‘ocean,' here.” “No, I didn't!” says Sally, somewhat forcefully. Her teacher is a bit apologetic, but persists. “Ocean is spelled with a ‘c' rather than a ‘sh'... remember when we learned the rule that if there's an ‘e' after the ‘c', that changes its sound—” “No, it's spelled oshun, I saw it in a book—” “Look,” says the teacher, gently but firmly. “I know it hurts to notice when we make mistakes. But it's important to see them, so that you can do better next time. Here, let's get the dictionary and check—” “No!” shouts Sally, as she bursts into tears and runs away to hide. As she vanishes into the closet, the teach...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Announcing: the Prague Fall Season, published by IrenaK on July 29, 2022 on The Effective Altruism Forum. tl;dr: We invite the broader EA/Rationality community to spend this fall in Prague. There will be a high concentration of global EA/Rationality events, workshops, retreats and seminars happening in and around Prague. While the events alone are interesting, we believe there will be additional benefits for staying around for a longer period of time and an opportunity to talk, create and meet caused by the higher density of people in one place. We will have a large coworking space available in Prague for people to work from and socialize. We also want to share with the world what we like about Prague and the local community. Prague seems to be a good place for thinking about hard problems - both Albert Einstein and Johannes Kepler made substantial progress on their fundamental research while living in Prague. We think you would enjoy being part of the Prague Fall Season if you: Want to spend an extended period of time this fall with other like-minded people, concentrated in one area, building momentum together. Are interested in exploring cities, cultures, and aesthetics different from the US or UK hubs. Are curious about the Czech EA/Rationality culture and want to spend some time with us. Want to work on some of the projects based in Prague. Want to experience what it's like to live and work in Prague. If you are interested you can: Apply for a residency. Apply for one of the jobs based out of Prague. Apply to work from our new office space. Apply for a CFAR workshop. . or just visit us and we will figure it out! Why Prague? The events happening in Prague this autumn provide a Schelling point.Prague is a second-tier EA Hub, smaller than London or the Bay area, but comparable or larger than probably any other city in continental Europe. Prague has a thriving local EA community, a newly established alignment research group, ~20 full-time people working in/with high-impact organizations (eg Metaculus, ESPR, ALLFED, CFAR), and about a hundred people in the broader EA and rationalist communities. We aim for the Season to bring in on average additional 30-60 people staying for longer, and a few hundreds of shorter-time visitors, who will participate in some event and stay for a few days before or after. In our experience, Prague is a very good place to live - it has the benefits of a modern large city, while being walkable or bikeable, offering a high quality of living, and overall unique aesthetics and vibes (meaningfully different from other similar hubs); a likely fit for some creative high-impact people who seek a change in their environment. I often get questions about why there is such a big concentration of successful and interesting people in the Czech Republic, especially in the EA/Rationality community. My answer usually goes something like this. On a more serious note, we are quite excited about some of the virtues of the local EA/Rationality culture and would like to share them with the global community. If I were to summarize some of the key ones, they would be: Doing what's needed - one secret superpower we aim for is to do what needs to be done, even if the quests are not shiny and glittering with status. Sanity - Prague often feels like a more sane and more grounded place, which has distinct advantages and disadvantages; for example, if you feel you are too steady and unambitious or want to upend everything in your life, the vibes of Silicon Valley or the Bay area are likely better for moving in this direction. In contrast, if you feel the pressure of the competition for attention, networking, or similar source of stress is distracting you from useful work and you would benefit from having some time with less of that, Prague may be better for a few month...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Opening Session Tips & Advice, published by CFAR!Duncan on July 25, 2022 on LessWrong. (Author's note: This will be moved to the front of the sequence once it's had its moment in the limelight.) Meta CFAR ran many, many workshops. After each workshop, there would be feedback from the participants, and debrief discussions among the staff. We would talk about what had worked and what hadn't, what we wish had been said or done, what we would try differently in the future, etc. Often, what resulted was a new addition to the opening session. Opening session, at a CFAR workshop, was largely about expectation setting, and getting everyone on the same page—making sure everyone knew what they were getting into, and what was going to be asked of them, and why. The "tips and advice" section of opening session was often framed as "things past participants said, at the end, that they wished they'd been told at the beginning." (This was often but not always literally true.) Little snippets of wisdom about how to engage with the content, what to watch out for in one's own experience, where to put one's attention, etc. Often the staff would create their own tips and advice based off of watching classes fail, or watching individual participants "bounce" off the workshop, and trying to figure out why. There were something like two dozen distinct tips, at various points, of which four or five would be presented at a given workshop. Some were added, some were removed, others morphed or mutated, yet others got more deeply baked into the structure of the workshop and were no longer needed in opening session. Below is a selection of some of the most important and longest-lasting opening session tips. They are presented here for two purposes: Despite the fact that this is an online sequence and not a workshop, the tips nevertheless contain valuable wisdom about how to engage with the content, and some specific ways that trying to do so tends to go wrong for people. They may be useful advice in other contexts, such as conferences or events that you yourself may run, in the future. Be Present One key element of getting the most out of an experience is being present. This includes physically showing up, but it also includes having your mind in the room and your background thoughts focused on the content. The more you're taking calls and answering texts and keeping up with social media and what's going on back home, the more you'll remain in your ordinary mental space, continuing to reinforce the same habits and patterns you're here to change. There's a sort of snowball effect, where even a little disengagement can make absorbing the value you'd like from a workshop rather difficult, which confirms a suspicion that there's no value to be had, and so on. Think about, for instance, the sorts of thoughts one can have on a long, three-day hiking trip, with no deadlines or obligations. When all of your thoughts must be purposeful, or when every thought must resolve itself before the next thing on the schedule rolls around, there are a lot of thoughts you simply can't have. And it is precisely thoughts-unlike-those-you're-accustomed-to-having that the workshop is trying to provide! After all, if your present ways of thinking and being were sufficient to solve all your problems and achieve all your goals, you'd already be done. Not every change is an improvement, but every improvement must necessarily be a change, and one of the precursors to change is setting yourself up to be able to be in any kind of non-default state of mind at all. If it's business as usual, your brain will produce business-as-usual thoughts, and you'll find few or no life-changing insights in that drawer. In addition to external distraction, we've also found that there are a few unhelpful narratives that participants occasionall...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Three common mistakes when naming an org or project, published by Kat Woods on July 23, 2022 on The Effective Altruism Forum. If Hermione had named her organization H.E.R.O (House Elf Release Organization) instead of S.P.E.W, she might have gotten a lot more traction. Similarly, aspiring charity entrepreneurs know that finding a good name for their organization or project can play an important role in their future impact. After starting four EA organizations (with varying degrees of name quality), I am often asked what I think of a charity entrepreneur's name for their new venture. I always have the same three pieces of advice, so I thought I'd put it into a blog post so others can benefit from it as well. 1. People will shorten the name if it's too long. Name accordingly Consider how people will shorten your organization's name in everyday conversation. People don't like saying more than two or three syllables at once. In everyday conversation, no one wastes their breath on the lengthy names of ‘Eighty-thousand Hours' or ‘the Open Philanthropy Project'. They say ‘80k' or ‘Open Phil'. Your name should either have 1-3 syllables in the first place (‘GiveWell') or look good when shortened to 1-3 syllables. The full name can have more than three syllables if it has a snappy acronym. It's great if your acronym spells a word or phrase, especially if it evokes the organization's mission (e.g., ACE, CFAR, ALLFED). If your acronym doesn't spell something, avoid Ws - it's very awkward and long to say ‘double-you'. 2. Don't artificially lock yourself into a particular strategy with your name Your name shouldn't tie you to a specific project, method, goal, or aim. Over time, you will hopefully change your mind about what's the highest impact thing to do; a vague name preserves your option value. If the Against Malaria Foundation wanted to work on tuberculosis instead, or 80k decided to focus on donations rather than career choice, they'd be stuck. Names like ‘Lightcone' and ‘Nonlinear' are evocative, but they don't imply that the organizations are working on anything in particular. At Nonlinear we could switch our focus from meta work to direct work tomorrow and the name would still work. Of course, names won't necessarily stop you from pivoting. Oxfam is the shortened form of the Oxford Committee for Famine Relief, and now they do far more than help those facing famine. However, it increases the friction of updating based on new evidence or crucial considerations, which is where a massive percentage of your potential future impact comes from. So don't artificially limit yourself simply because of a name. 3. Get loads of feedback on loads of different names Generate LOTS of options - potentially hundreds - then choose the best 10 and ask your friends to rate them. Don't just choose one name and ask your friends what they think. First, they can't tell you how the name compares to other possible names - maybe they think it's fine, but they'd much prefer another option you considered. Second, it's socially difficult for your friends to respond ‘actually, I hate it,' so it's hard to get honest feedback this way. Even if you name your child Adolf or Hashtag, people will coo ‘aww! How cute! How original!' If you send your friends options, it's easier for them to be honest about which they like best. So there's the 80/20 advice on naming your organization or project: Keep it three syllables or less, or know that its shortened form will also be good Preserve option value by giving yourself a vague name Generate a ton of options and get feedback on the top 5-10 from a bunch of friends Reminder that if this reaches 25 upvotes, you can listen to this post on your podcast player using the Nonlinear Library. This post was written collaboratively by Kat Woods and Amber Dawn Ace as part of Non...

Whether you're a leader or a manager, the chances are that you occasionally (or regularly) find yourself dealing with difficult situations. Coaching is an increasingly popular way of helping people develop the skills, habits, and mindsets needed to reach their full potential by better understanding themselves, their goals, and the situations they encounter. Today’s guest is Dr. Richard Levin. Richard is widely recognized as one of the first executive coaches. He is one of a half-dozen global leaders who have created and shaped the coaching profession since its inception in the 1980’s. As the founder and principal of Richard Levin & Associates (the first executive coaching firm, and the first network of independent executive coaches); as co-author of the popular and powerful book Shared Purpose: Working Together to Build Strong Families and High Performance Companies; and as a founder of Boston University’s Center on Work and Family, Richard has stretched the boundaries of creativity, inclusiveness, and collaboration to build extraordinary organizations. Richard and I talk about coaching - what coaching is, how it's different from therapy or advising, who should get coaching, the future of coaching, and what to do if you or a team member want coaching but your organizanization doesn't have the budget for it. Members of the Modern Manager community get a resource packet that consists of CFAR’s boldest thinking on executive coaching, strategy, culture, and organizational behavior. This valuable resource includes learnings and writings of CFAR’s top leaders and has never before been available to the public. Get it when you join the Modern Manager community. Subscribe to my newsletter to get episodes, articles and free mini-guides delivered to your inbox. Read the related blog article: Executive Coaching Isn’t Just for Executives KEEP UP WITH RICHARD Website: www.cfar.com LinkedIn: https://www.linkedin.com/in/richardjlevin/ Website Bio: https://www.cfar.com/Levin/ Key Takeaways: Dr. Richard Levin, a psychologist by training, founded the world’s first “executive coaching firm” in the 1980’s. Since then, the field has grown exponentially. Coaching helps leaders become their best selves. It can include everything from avoiding burnout to communication skills. Coaches act as thought partners to