Podcasts about singularity institute

Play Episode Listen Later Jun 2, 2023 39:11

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Yudkowsky vs Hanson on FOOM: Whose Predictions Were Better?, published by 1a3orn on June 1, 2023 on LessWrong. TLDR Starting in 2008, Robin Hanson and Eliezer Yudkowsky debated the likelihood of FOOM: a rapid and localized increase in some AI's intelligence that occurs because an AI recursively improves itself. As Yudkowsky summarizes his position: I think that, at some point in the development of Artificial Intelligence, we are likely to see a fast, local increase in capability—“AI go FOOM.” Just to be clear on the claim, “fast” means on a timescale of weeks or hours rather than years or decades; and “FOOM” means way the hell smarter than anything else around, capable of delivering in short time periods technological advancements that would take humans decades, probably including full-scale molecular nanotechnology. (FOOM, 235) Over the course of this debate, both Hanson and Yudkowsky made a number of incidental predictions about things which could occur before the advent of artificial superintelligence -- or for which we could at the very least receive strong evidence before artificial superintelligence. On the object level, my conclusions is that when you examine these predictions, Hanson probably does a little better than Yudkowsky. Although depending on how you weigh different topics, I could see arguments from "they do about the same" to "Hanson does much better." On one meta level, my conclusion is that Hanson's view --- that we should try to use abstractions that have proven prior predictive power -- looks like a pretty good policy. On another meta level, my conclusion -- springing to a great degree from how painful seeking clear predictions in 700 pages of words has been -- is that if anyone says "I have a great track record" without pointing to specific predictions that they made, you should probably ignore them, or maybe point out their lack of epistemic virtue if you have the energy to spare for doing that kind of criticism productively. Intro There are number of difficulties involved in evaluating some public figure's track record. We want to avoid cherry-picking sets of particularly good or bad predictions. And we want to have some baseline to compare them to. We can mitigate both of these difficulties -- although not, alas, eliminate them -- by choosing one document to evaluate: "The Hanson-Yudkowsky Foom Debate". (All future page numbers refer to this PDF.) Note that the PDF includes the (1) debate-via-blogposts which took place on OvercomingBias, (2) an actual in-person debate that took place at Jane Street in 2011 and (3) further summary materials from Hanson (further blogposts) and Yudkowsky ("Intelligence Explosion Microeconomic"). This spans a period from 2008 to 2013. I do not intend this to be a complete review of everything in these arguments. The discussion spans the time from the big bang until hypothetical far future galactic civilizations. My review is a little more constrained: I am only going to look at predictions for which I think we've received strong evidence in the 15 or so years since the debate started. Note also that the context of this debate was quite different than it would be if it happened today. At the time of the debate, both Hanson and Yudkowsky believed that machine intelligence would be extremely important, but that the time of its arrival was uncertain. They thought that it would probably arrive this century, but neither had the very, certain short timelines which are common today. At this point Yudkowsky was interested in actually creating a recursively self-improving artificial intelligence, a "seed AI." For instance, in 2006 the Singularity Institute -- what MIRI was before it renamed -- had a website explicitly stating that they sought funding to create recursively self-improving AI. During the Jane Street debate Y...

ai predictions artificial intelligence speech ea hanson miri robin hanson eliezer yudkowsky rationalist jane street yudkowsky lesswrong foom singularity institute

LW - Yudkowsky vs Hanson on FOOM: Whose Predictions Were Better? by 1a3orn

Play Episode Listen Later Jun 2, 2023 39:11

Link to original articleWelcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Yudkowsky vs Hanson on FOOM: Whose Predictions Were Better?, published by 1a3orn on June 1, 2023 on LessWrong. TLDR Starting in 2008, Robin Hanson and Eliezer Yudkowsky debated the likelihood of FOOM: a rapid and localized increase in some AI's intelligence that occurs because an AI recursively improves itself. As Yudkowsky summarizes his position: I think that, at some point in the development of Artificial Intelligence, we are likely to see a fast, local increase in capability—“AI go FOOM.” Just to be clear on the claim, “fast” means on a timescale of weeks or hours rather than years or decades; and “FOOM” means way the hell smarter than anything else around, capable of delivering in short time periods technological advancements that would take humans decades, probably including full-scale molecular nanotechnology. (FOOM, 235) Over the course of this debate, both Hanson and Yudkowsky made a number of incidental predictions about things which could occur before the advent of artificial superintelligence -- or for which we could at the very least receive strong evidence before artificial superintelligence. On the object level, my conclusions is that when you examine these predictions, Hanson probably does a little better than Yudkowsky. Although depending on how you weigh different topics, I could see arguments from "they do about the same" to "Hanson does much better." On one meta level, my conclusion is that Hanson's view --- that we should try to use abstractions that have proven prior predictive power -- looks like a pretty good policy. On another meta level, my conclusion -- springing to a great degree from how painful seeking clear predictions in 700 pages of words has been -- is that if anyone says "I have a great track record" without pointing to specific predictions that they made, you should probably ignore them, or maybe point out their lack of epistemic virtue if you have the energy to spare for doing that kind of criticism productively. Intro There are number of difficulties involved in evaluating some public figure's track record. We want to avoid cherry-picking sets of particularly good or bad predictions. And we want to have some baseline to compare them to. We can mitigate both of these difficulties -- although not, alas, eliminate them -- by choosing one document to evaluate: "The Hanson-Yudkowsky Foom Debate". (All future page numbers refer to this PDF.) Note that the PDF includes the (1) debate-via-blogposts which took place on OvercomingBias, (2) an actual in-person debate that took place at Jane Street in 2011 and (3) further summary materials from Hanson (further blogposts) and Yudkowsky ("Intelligence Explosion Microeconomic"). This spans a period from 2008 to 2013. I do not intend this to be a complete review of everything in these arguments. The discussion spans the time from the big bang until hypothetical far future galactic civilizations. My review is a little more constrained: I am only going to look at predictions for which I think we've received strong evidence in the 15 or so years since the debate started. Note also that the context of this debate was quite different than it would be if it happened today. At the time of the debate, both Hanson and Yudkowsky believed that machine intelligence would be extremely important, but that the time of its arrival was uncertain. They thought that it would probably arrive this century, but neither had the very, certain short timelines which are common today. At this point Yudkowsky was interested in actually creating a recursively self-improving artificial intelligence, a "seed AI." For instance, in 2006 the Singularity Institute -- what MIRI was before it renamed -- had a website explicitly stating that they sought funding to create recursively self-improving AI. During the Jane Street debate Y...

ai predictions artificial intelligence speech ea hanson miri robin hanson eliezer yudkowsky rationalist jane street yudkowsky lesswrong foom singularity institute

Ep 63: Eliezer Yudkowsky (AI Safety Expert) Says It's Too Late to Save Humanity from AI

Three Cartoon Avatars

Play Episode Listen Later May 6, 2023 197:57

(0:00) Intro(1:18) Welcome Eliezer(6:27) How would you define artificial intelligence?(15:50) What is the purpose of a firm alarm?(19:29) Eliezer's background(29:28) The Singularity Institute for Artificial Intelligence(33:38) Maybe AI doesn't end up automatically doing the right thing(45:42) AI Safety Conference(51:15) Disaster Monkeys(1:02:15) Fast takeoff(1:10:29) Loss function(1:15:48) Protein folding(1:24:55) The deadly stuff(1:46:41) Why is it inevitable?(1:54:27) Can't we let tech develop AI and then fix the problems?(2:02:56) What were the big jumps between GPT3 and GPT4?(2:07:15) “The trajectory of AI is inevitable”(2:28:05) Elon Musk and OpenAI(2:37:41) Sam Altman Interview(2:50:38) The most optimistic path to us surviving(3:04:46) Why would anything super intelligent pursue ending humanity?(3:14:08) What role do VCs play in this? Show Notes:https://twitter.com/liron/status/1647443778524037121?s=20https://futureoflife.org/event/ai-safety-conference-in-puerto-rico/https://www.lesswrong.com/posts/j9Q8bRmwCgXRYAgcJ/miri-announces-new-death-with-dignity-strategyhttps://www.youtube.com/watch?v=q9Figerh89ghttps://www.vox.com/the-highlight/23447596/artificial-intelligence-agi-openai-gpt3-existential-risk-human-extinctionEliezer Yudkowsky – AI Alignment: Why It's Hard, and Where to Start Mixed and edited: Justin HrabovskyProduced: Rashad AssirExecutive Producer: Josh MachizMusic: Griff Lawson

EA - The FTX crisis highlights a deeper cultural problem within EA - we don't sufficiently value good governance by Fods12

Play Episode Listen Later Nov 14, 2022 7:28

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The FTX crisis highlights a deeper cultural problem within EA - we don't sufficiently value good governance, published by Fods12 on November 14, 2022 on The Effective Altruism Forum. Introduction In this piece, I will explain why I don't think the collapse of FTX and resulting fallout for Future Fund and EA community in general is a one-off or 'black swan' event as some have argued on this forum. Rather, I think that what happened was part of a broader pattern of failures and oversights that have been persistent within EA and EA-adjacent organisations since the beginning of the movement. As a disclaimer, I do not have any inside knowledge or special expertise about FTX or any of the other organisations I will mention in this post. I speak simply as a long-standing and concerned member of the EA community. Weak Norms of Governance The essential point I want to make in this post is that the EA community has not been very successful in fostering norms of transparency, accountability, and institutionalisation of decision-making. Many EA organisations began as ad hoc collections of like-minded individuals with very ambitions goals but relatively little career experience. This has often led to inadequate organisational structures and procedures being established for proper management of personal, financial oversight, external auditing, or accountability to stakeholders. Let me illustrate my point with some major examples I am aware of from EA and EA-adjacent organisations: Weak governance structures and financial oversight at the Singularity Institute, leading to the theft of over $100,000 in 2009. Inadequate record keeping, rapid executive turnover, and insufficient board oversight at the Centre for Effective Altruism over the period 2016-2019. Inadequate financial record keeping at 80,000 Hours during 2018. Insufficient oversight, unhealthy power dynamics, and other harmful practices reported at MIRI/CFAR during 2015-2017. Similar problems reported at the EA-adjacent organisation Leverage Research during 2017-2019. 'Loose norms around board of directors and conflicts of interests between funding orgs and grantees' at FTX and the Future Fund from 2021-2022. While these specific issues are somewhat diverse, I think what they have in common is an insufficient emphasis on principles of good organisational governance. This ranges from the most basic such as clear objectives and good record keeping, to more complex issues such as external auditing, good systems of accountability, transparency of the organisation to its stakeholders, avoiding conflicts of interest, and ensuring that systems exist to protect participants in asymmetric power relationships. I believe that these aspects of good governance and robust institution building have not been very highly valued in the broader EA community. In my experience, EAs like to talk about philosophy, outreach, career choice, and other nerdy stuff. Discussing best practise of organisational governance and systems of accountability doesn't seem very high status or 'sexy' in the EA space. There has been some discussion of such issues on this forum (e.g. this thoughtful post), but overall EA culture seems to have failed to properly absorb these lessons. EA projects are often run by small groups of young idealistic people who have similar educational and social backgrounds, who often socialise together, and (in many cases) participate in romantic relationships with one another - The case of Sam Bankman-Fried and Caroline Ellison is certainly not the only such example in the EA community. The EA culture seems to be heavily influenced by start-up culture and entrepreneurialism, with a focus on moving quickly and relying on finding highly-skilled and highly-aligned people and then providing them funding and space to work with minimal oversi...

crisis cultural speech weak deeper similar ea ftx inadequate insufficient effective altruism eas good governance sufficiently caroline ellison rationalist singularity institute

LW - My Recollection of How This All Got Started by G Gordon Worley III

Play Episode Listen Later Apr 6, 2022 6:51

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: My Recollection of How This All Got Started, published by G Gordon Worley III on April 6, 2022 on LessWrong. I've told this story to various folks one-on-one. They usually want to know something like "how did you get into AI safety" or "how did you get into EA". And although I expect to keep telling it one-off, I'll write it down for those of you I'll never get to meet. Why should you read it? Because my story is partially the story of how all this got started: LessWrong, AI safety, EA, and so on. I'm not saying it's the whole story, I'm just saying I've been hanging around what I think of as this community for over 20 years now, so my story is one facet of how we got here. My story starts in the late 90s. I was hanging around this mailserv called "extropians". How I ended up there I don't recall, but my best guess is I wandered over directly or indirectly from something I found on Slashdot. I'm pretty sure nanobots or cryonics were involved. This guy Eli-something-or-other wrote some posts about how most people are unable to think coherently about future tech that's too many "shock levels" above what they already know about. He started a mailing list that split off from extropians to talk coherently about the most shocking stuff: the Shock Level 4 stuff. Thus the community began to come into existence with the creation of the SL4 mailserv. We talked about all kinds of wild ideas on there, but the big one was AGI. An important topic in those days was figuring out if AGI was default safe or dangerous. Eliezer said default dangerous and made lots of arguments that this was the case. I was eventually convinced. Some remain unconvinced to this day. Big names I remember from this time include Hal Finney, Christine Peterson, Max Moore, Eric Drexler, Ben Goertzel, and Robin Hanson. I'm not sure who was on the list and who wasn't. I also remember some other names but we were not among the big players. The list started to die down after a couple years, but around this time we started hanging out on IRC. It was a lot of fun, but a huge time suck. This helped bring the community more together in real time, but everyone was still spread out. Somewhere along the way the Singularity Institute started. Around this time Eliezer started to get really into heuristics and biases and Bayes' Theorem, claiming it was the secret of the universe or something. After I studied a bunch of information theory and thermodynamics I basically believed it, although I still prefer to think in the cybernetic terms I picked up from my engineering education. We also all got interested in quantum physics and evolutionary psychology and some other stuff. Eliezer was really on about building Friendly AI and had been since about the start of the SL4 mailing list. What that meant got clearer over time. What also got clearer is that we were all too stupid to help even though many of us were convinced AGI was going to be dangerous (I remember a particular exchange where Eliezer got so frustrated at my inability to do some basic Bayesian reasoning that he ended up writing a whole guide to Bayes' Theorem). Part of the problem seemed to be not that we lacked intelligence, but that we just didn't know how to think very good. Our epistemics were garbage and it wasn't very clear why. Eliezer went off into the wilderness and stopped hanging out on IRC. The channel kind of died down and went out with a whimper. I got busy doing other stuff but tried to keep track of what was happening. The community felt like it was in a lull. Then Overcoming Bias started! Eliezer and Robin posted lots of great stuff! There was an AI foom debate. Sequences of posts were posted. It was fun! Then at some point LessWrong started. Things really picked up. New people emerged in the community. I found myself busy and sidelined at this time,...

ai started speech ea agi recollections eliezer irc bayesian sequences worley theorem bayes robin hanson rationalist hal finney ben goertzel slashdot lesswrong christine peterson singularity institute

LW - My Recollection of How This All Got Started by G Gordon Worley III

Play Episode Listen Later Apr 6, 2022 6:51

Link to original articleWelcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: My Recollection of How This All Got Started, published by G Gordon Worley III on April 6, 2022 on LessWrong. I've told this story to various folks one-on-one. They usually want to know something like "how did you get into AI safety" or "how did you get into EA". And although I expect to keep telling it one-off, I'll write it down for those of you I'll never get to meet. Why should you read it? Because my story is partially the story of how all this got started: LessWrong, AI safety, EA, and so on. I'm not saying it's the whole story, I'm just saying I've been hanging around what I think of as this community for over 20 years now, so my story is one facet of how we got here. My story starts in the late 90s. I was hanging around this mailserv called "extropians". How I ended up there I don't recall, but my best guess is I wandered over directly or indirectly from something I found on Slashdot. I'm pretty sure nanobots or cryonics were involved. This guy Eli-something-or-other wrote some posts about how most people are unable to think coherently about future tech that's too many "shock levels" above what they already know about. He started a mailing list that split off from extropians to talk coherently about the most shocking stuff: the Shock Level 4 stuff. Thus the community began to come into existence with the creation of the SL4 mailserv. We talked about all kinds of wild ideas on there, but the big one was AGI. An important topic in those days was figuring out if AGI was default safe or dangerous. Eliezer said default dangerous and made lots of arguments that this was the case. I was eventually convinced. Some remain unconvinced to this day. Big names I remember from this time include Hal Finney, Christine Peterson, Max Moore, Eric Drexler, Ben Goertzel, and Robin Hanson. I'm not sure who was on the list and who wasn't. I also remember some other names but we were not among the big players. The list started to die down after a couple years, but around this time we started hanging out on IRC. It was a lot of fun, but a huge time suck. This helped bring the community more together in real time, but everyone was still spread out. Somewhere along the way the Singularity Institute started. Around this time Eliezer started to get really into heuristics and biases and Bayes' Theorem, claiming it was the secret of the universe or something. After I studied a bunch of information theory and thermodynamics I basically believed it, although I still prefer to think in the cybernetic terms I picked up from my engineering education. We also all got interested in quantum physics and evolutionary psychology and some other stuff. Eliezer was really on about building Friendly AI and had been since about the start of the SL4 mailing list. What that meant got clearer over time. What also got clearer is that we were all too stupid to help even though many of us were convinced AGI was going to be dangerous (I remember a particular exchange where Eliezer got so frustrated at my inability to do some basic Bayesian reasoning that he ended up writing a whole guide to Bayes' Theorem). Part of the problem seemed to be not that we lacked intelligence, but that we just didn't know how to think very good. Our epistemics were garbage and it wasn't very clear why. Eliezer went off into the wilderness and stopped hanging out on IRC. The channel kind of died down and went out with a whimper. I got busy doing other stuff but tried to keep track of what was happening. The community felt like it was in a lull. Then Overcoming Bias started! Eliezer and Robin posted lots of great stuff! There was an AI foom debate. Sequences of posts were posted. It was fun! Then at some point LessWrong started. Things really picked up. New people emerged in the community. I found myself busy and sidelined at this time,...

ai started speech ea agi recollections eliezer irc bayesian sequences worley theorem bayes robin hanson rationalist hal finney ben goertzel slashdot lesswrong christine peterson singularity institute

LW - The Power of Reinforcement by lukeprog from The Science of Winning at Life

Play Episode Listen Later Dec 25, 2021 7:34

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is The Science of Winning at Life, Part 6: The Power of Reinforcement, published by lukeprog. Part of the sequence: The Science of Winning at Life Also see: Basics of Animal Reinforcement, Basics of Human Reinforcement, Physical and Mental Behavior, Wanting vs. Liking Revisited, Approving reinforces low-effort behaviors, Applying Behavioral Psychology on Myself. Story 1: On Skype with Eliezer, I said: "Eliezer, you've been unusually pleasant these past three weeks. I'm really happy to see that, and moreover, it increases my probability than an Eliezer-led FAI research team will work. What caused this change, do you think?" Eliezer replied: "Well, three weeks ago I was working with Anna and Alicorn, and every time I said something nice they fed me an M&M." Story 2: I once witnessed a worker who hated keeping a work log because it was only used "against" him. His supervisor would call to say "Why did you spend so much time on that?" or "Why isn't this done yet?" but never "I saw you handled X, great job!" Not surprisingly, he often "forgot" to fill out his worklog. Ever since I got everyone at the Singularity Institute to keep work logs, I've tried to avoid connections between "concerned" feedback and staff work logs, and instead take time to comment positively on things I see in those work logs. Story 3: Chatting with Eliezer, I said, "Eliezer, I get the sense that I've inadvertently caused you to be slightly averse to talking to me. Maybe because we disagree on so many things, or something?" Eliezer's reply was: "No, it's much simpler. Our conversations usually run longer than our previously set deadline, so whenever I finish talking with you I feel drained and slightly cranky." Now I finish our conversations on time. Story 4: A major Singularity Institute donor recently said to me: "By the way, I decided that every time I donate to the Singularity Institute, I'll set aside an additional 5% for myself to do fun things with, as a motivation to donate." The power of reinforcement It's amazing to me how consistently we fail to take advantage of the power of reinforcement. Maybe it's because behaviorist techniques like reinforcement feel like they don't respect human agency enough. But if you aren't treating humans more like animals than most people are, then you're modeling humans poorly. You are not an agenty homunculus "corrupted" by heuristics and biases. You just are heuristics and biases. And you respond to reinforcement, because most of your motivation systems still work like the motivation systems of other animals. A quick reminder of what you learned in high school A reinforcer is anything that, when it occurs in conjunction with an act, increases the probability that the act will occur again. A positive reinforcer is something the subject wants, such as food, petting, or praise. Positive reinforcement occurs when a target behavior is followed by something the subject wants, and this increases the probability that the behavior will occur again. A negative reinforcer is something the subject wants to avoid, such as a blow, a frown, or an unpleasant sound. Negative reinforcement occurs when a target behavior is followed by some relief from something the subject doesn't want, and this increases the probability that the behavior will happen again. What works Small reinforcers are fine, as long as there is a strong correlation between the behavior and the reinforcer (Schneider 1973; Todorov et al. 1984). All else equal, a large reinforcer is more effective than a small one (Christopher 1988; Ludvig et al. 2007; Wolfe 1936), but the more you increase the reinforcer magnitude, the less benefit you get from the increase (Frisch & Dickinson 1990). The reinforcer should immediately follow the target behavior (Escobar & Bruner 2007; Schlinger & Blakely 1994; Schneider 1990). P...

power science story winning positive basics speech negative chatting ea schneider mm wolfe fai eliezer reinforcement ludvig approving winning at life rationalist todorov alicorn singularity institute on skype

LW - The Good News of Situationist Psychology by lukeprog from The Science of Winning at Life

Play Episode Listen Later Dec 25, 2021 4:40

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is The Science of Winning at Life, Part 5: The Good News of Situationist Psychology, published by lukeprog. Part of the sequence: The Science of Winning at Life In 1961, Stanley Milgram began his famous obedience experiments. He found that ordinary people would deliver (what they believed to be) excruciatingly painful electric shocks to another person if instructed to do so by an authority figure. Milgram claimed these results showed that in certain cases, people are more heavily influenced by their situation than by their internal character. Fifty years and hundreds of studies later, this kind of situationism is widely accepted for broad domains of human action. People can inflict incredible cruelties upon each other in a prison simulation.b Hurried passersby step over a stricken person in their path, while unhurried passersby stop to help.a Willingness to help varies with the number of bystanders, and with proximity to a fragrant bakery or cofee shop.c The list goes on and on.d Our inability to realize how powerful the effect situation has on human action is so well-known that it has a name. Our tendency to over-value trait-based explanations of others' behavior and under-value situation-based explanations of their behavior is called the fundamental attribution error (aka correspondence bias). Recently, some have worried that this understanding undermines the traditional picture we have of ourselves as stable persons with robust characteristics. How can we trust others if their unpredictable situation may have so powerful an effect that it overwhelms the effect of their virtuous character traits? But as I see it, situationist psychology is wonderful news, for it means we can change! If situation has a powerful effect on behavior, then we have significant powers to improve our own behavior. It would be much worse to discover that our behavior was almost entirely determined by traits we were born with and cannot control. For example, drug addicts can be more successful in beating addiction if they change their peer group - if they stop spending recreational time with other addicts, and spend time with drug-free people instead, or in a treatment environment.e Improving rationality What about improving your rationality? Situationist psychology suggests it may be wise to surround yourself with fellow rationalists. Having now been a visiting fellow with the Singularity Institute for only two days, I can already tell that almost everyone I've met who is with the Singularity Institute or has been through its visiting fellows program is a level or two above me - not just in knowledge about Friendly AI and simulation arguments and so on, but in day-to-day rationality skills. It's fascinating to take part in a conversation with really trained rationalists. It might go something like this: Person One: "I suspect that P, though I know that cognitive bias A and B and C are probably influencing me here. However, I think that evidence X and Y offer fairly strong support for P." Person Two: "But what about Z? This provides evidence against P because blah blah blah..." Person One: "Huh. I hadn't thought that. Well, I'm going to downshift my probability that P." Person Three: "But what about W? The way Schmidhuber argues is this: blah blah blah." Person One: "No, that doesn't work because blah blah blah." Person Three: "Hmmm. Well, I have a lot of confusion and uncertainty about that." This kind of thing can go on for hours, and not just on abstract subjects like simulation arguments, but also on more personal issues like fears and dreams and dating. I've had several of these many-hours-long group conversations already - people arguing vigorously, often 'trashing' others' views (with logic and evidence), but with everybody apparently willing to update their beliefs, nobody getting mad or...

science winning psychology good news speech ea fifty willingness milgram hurried stanley milgram winning at life rationalist situationists schmidhuber singularity institute

LW - The Power of Reinforcement by lukeprog from The Science of Winning at Life

Play Episode Listen Later Dec 25, 2021 7:34

Link to original articleWelcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is The Science of Winning at Life, Part 6: The Power of Reinforcement, published by lukeprog. Part of the sequence: The Science of Winning at Life Also see: Basics of Animal Reinforcement, Basics of Human Reinforcement, Physical and Mental Behavior, Wanting vs. Liking Revisited, Approving reinforces low-effort behaviors, Applying Behavioral Psychology on Myself. Story 1: On Skype with Eliezer, I said: "Eliezer, you've been unusually pleasant these past three weeks. I'm really happy to see that, and moreover, it increases my probability than an Eliezer-led FAI research team will work. What caused this change, do you think?" Eliezer replied: "Well, three weeks ago I was working with Anna and Alicorn, and every time I said something nice they fed me an M&M." Story 2: I once witnessed a worker who hated keeping a work log because it was only used "against" him. His supervisor would call to say "Why did you spend so much time on that?" or "Why isn't this done yet?" but never "I saw you handled X, great job!" Not surprisingly, he often "forgot" to fill out his worklog. Ever since I got everyone at the Singularity Institute to keep work logs, I've tried to avoid connections between "concerned" feedback and staff work logs, and instead take time to comment positively on things I see in those work logs. Story 3: Chatting with Eliezer, I said, "Eliezer, I get the sense that I've inadvertently caused you to be slightly averse to talking to me. Maybe because we disagree on so many things, or something?" Eliezer's reply was: "No, it's much simpler. Our conversations usually run longer than our previously set deadline, so whenever I finish talking with you I feel drained and slightly cranky." Now I finish our conversations on time. Story 4: A major Singularity Institute donor recently said to me: "By the way, I decided that every time I donate to the Singularity Institute, I'll set aside an additional 5% for myself to do fun things with, as a motivation to donate." The power of reinforcement It's amazing to me how consistently we fail to take advantage of the power of reinforcement. Maybe it's because behaviorist techniques like reinforcement feel like they don't respect human agency enough. But if you aren't treating humans more like animals than most people are, then you're modeling humans poorly. You are not an agenty homunculus "corrupted" by heuristics and biases. You just are heuristics and biases. And you respond to reinforcement, because most of your motivation systems still work like the motivation systems of other animals. A quick reminder of what you learned in high school A reinforcer is anything that, when it occurs in conjunction with an act, increases the probability that the act will occur again. A positive reinforcer is something the subject wants, such as food, petting, or praise. Positive reinforcement occurs when a target behavior is followed by something the subject wants, and this increases the probability that the behavior will occur again. A negative reinforcer is something the subject wants to avoid, such as a blow, a frown, or an unpleasant sound. Negative reinforcement occurs when a target behavior is followed by some relief from something the subject doesn't want, and this increases the probability that the behavior will happen again. What works Small reinforcers are fine, as long as there is a strong correlation between the behavior and the reinforcer (Schneider 1973; Todorov et al. 1984). All else equal, a large reinforcer is more effective than a small one (Christopher 1988; Ludvig et al. 2007; Wolfe 1936), but the more you increase the reinforcer magnitude, the less benefit you get from the increase (Frisch & Dickinson 1990). The reinforcer should immediately follow the target behavior (Escobar & Bruner 2007; Schlinger & Blakely 1994; Schneider 1990). P...

power science story winning positive basics speech negative chatting ea schneider wolfe fai eliezer reinforcement ludvig approving winning at life rationalist todorov alicorn singularity institute on skype

LW - The Good News of Situationist Psychology by lukeprog from The Science of Winning at Life

Play Episode Listen Later Dec 25, 2021 4:40

Link to original articleWelcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is The Science of Winning at Life, Part 5: The Good News of Situationist Psychology, published by lukeprog. Part of the sequence: The Science of Winning at Life In 1961, Stanley Milgram began his famous obedience experiments. He found that ordinary people would deliver (what they believed to be) excruciatingly painful electric shocks to another person if instructed to do so by an authority figure. Milgram claimed these results showed that in certain cases, people are more heavily influenced by their situation than by their internal character. Fifty years and hundreds of studies later, this kind of situationism is widely accepted for broad domains of human action. People can inflict incredible cruelties upon each other in a prison simulation.b Hurried passersby step over a stricken person in their path, while unhurried passersby stop to help.a Willingness to help varies with the number of bystanders, and with proximity to a fragrant bakery or cofee shop.c The list goes on and on.d Our inability to realize how powerful the effect situation has on human action is so well-known that it has a name. Our tendency to over-value trait-based explanations of others' behavior and under-value situation-based explanations of their behavior is called the fundamental attribution error (aka correspondence bias). Recently, some have worried that this understanding undermines the traditional picture we have of ourselves as stable persons with robust characteristics. How can we trust others if their unpredictable situation may have so powerful an effect that it overwhelms the effect of their virtuous character traits? But as I see it, situationist psychology is wonderful news, for it means we can change! If situation has a powerful effect on behavior, then we have significant powers to improve our own behavior. It would be much worse to discover that our behavior was almost entirely determined by traits we were born with and cannot control. For example, drug addicts can be more successful in beating addiction if they change their peer group - if they stop spending recreational time with other addicts, and spend time with drug-free people instead, or in a treatment environment.e Improving rationality What about improving your rationality? Situationist psychology suggests it may be wise to surround yourself with fellow rationalists. Having now been a visiting fellow with the Singularity Institute for only two days, I can already tell that almost everyone I've met who is with the Singularity Institute or has been through its visiting fellows program is a level or two above me - not just in knowledge about Friendly AI and simulation arguments and so on, but in day-to-day rationality skills. It's fascinating to take part in a conversation with really trained rationalists. It might go something like this: Person One: "I suspect that P, though I know that cognitive bias A and B and C are probably influencing me here. However, I think that evidence X and Y offer fairly strong support for P." Person Two: "But what about Z? This provides evidence against P because blah blah blah..." Person One: "Huh. I hadn't thought that. Well, I'm going to downshift my probability that P." Person Three: "But what about W? The way Schmidhuber argues is this: blah blah blah." Person One: "No, that doesn't work because blah blah blah." Person Three: "Hmmm. Well, I have a lot of confusion and uncertainty about that." This kind of thing can go on for hours, and not just on abstract subjects like simulation arguments, but also on more personal issues like fears and dreams and dating. I've had several of these many-hours-long group conversations already - people arguing vigorously, often 'trashing' others' views (with logic and evidence), but with everybody apparently willing to update their beliefs, nobody getting mad or...

science winning psychology good news speech ea fifty willingness milgram hurried stanley milgram winning at life rationalist situationists schmidhuber singularity institute

Thoughts on the Singularity Institute (SI) by HoldenKarnofsky

Play Episode Listen Later Dec 12, 2021 44:55

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Thoughts on the Singularity Institute (SI), published by HoldenKarnofsky on LessWrong. This post presents thoughts on the Singularity Institute from Holden Karnofsky, Co-Executive Director of GiveWell. Note: Luke Muehlhauser, the Executive Director of the Singularity Institute, reviewed a draft of this post, and commented: "I do generally agree that your complaints are either correct (especially re: past organizational competence) or incorrect but not addressed by SI in clear argumentative writing (this includes the part on 'tool' AI). I am working to address both categories of issues." I take Luke's comment to be a significant mark in SI's favor, because it indicates an explicit recognition of the problems I raise, and thus increases my estimate of the likelihood that SI will work to address them. September 2012 update: responses have been posted by Luke and Eliezer (and I have responded in the comments of their posts). I have also added acknowledgements. The Singularity Institute (SI) is a charity that GiveWell has been repeatedly asked to evaluate. In the past, SI has been outside our scope (as we were focused on specific areas such as international aid). With GiveWell Labs we are open to any giving opportunity, no matter what form and what sector, but we still do not currently plan to recommend SI; given the amount of interest some of our audience has expressed, I feel it is important to explain why. Our views, of course, remain open to change. (Note: I am posting this only to Less Wrong, not to the GiveWell Blog, because I believe that everyone who would be interested in this post will see it here.) I am currently the GiveWell staff member who has put the most time and effort into engaging with and evaluating SI. Other GiveWell staff currently agree with my bottom-line view that we should not recommend SI, but this does not mean they have engaged with each of my specific arguments. Therefore, while the lack of recommendation of SI is something that GiveWell stands behind, the specific arguments in this post should be attributed only to me, not to GiveWell. Summary of my views The argument advanced by SI for why the work it's doing is beneficial and important seems both wrong and poorly argued to me. My sense at the moment is that the arguments SI is making would, if accepted, increase rather than decrease the risk of an AI-related catastrophe. More SI has, or has had, multiple properties that I associate with ineffective organizations, and I do not see any specific evidence that its personnel/organization are well-suited to the tasks it has set for itself. More A common argument for giving to SI is that "even an infinitesimal chance that it is right" would be sufficient given the stakes. I have written previously about why I reject this reasoning; in addition, prominent SI representatives seem to reject this particular argument as well (i.e., they believe that one should support SI only if one believes it is a strong organization making strong arguments). More My sense is that at this point, given SI's current financial state, withholding funds from SI is likely better for its mission than donating to it. (I would not take this view to the furthest extreme; the argument that SI should have some funding seems stronger to me than the argument that it should have as much as it currently has.) I find existential risk reduction to be a fairly promising area for philanthropy, and plan to investigate it further. More There are many things that could happen that would cause me to revise my view on SI. However, I do not plan to respond to all comment responses to this post. (Given the volume of responses we may receive, I may not be able to even read all the comments on this post.) I do not believe these two statements are inconsistent, and I lay out paths for getting me...

ai executive director speech ea eliezer co executive director givewell rationalist lesswrong holden karnofsky singularity institute

Power dynamics as a blind spot or blurry spot in our collective world-modeling, especially around AI by Andrew_Critch

Play Episode Listen Later Dec 12, 2021 10:23

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Power dynamics as a blind spot or blurry spot in our collective world-modeling, especially around AI, published by Andrew_Critch on LessWrong. Where I'm coming from Epistemic status: personal experience In a number of prior posts, and in ARCHES, I've argued that more existential safety consideration is needed on the topic of multi-principal/multi-agent (multi/multi) dynamics among powerful AI systems. In general, I have found it much more difficult to convince thinkers within and around LessWrong's readership base to attend to multi/multi dynamics, as opposed to, say, convincing generally morally conscious AI researchers who are not (yet) closely associated with the effective altruism or rationality communities. Because EA/rationality discourse is particularly concerned with maintaining good epistemic processes, I think it would be easy to conclude from this state of affairs that multi/multi dynamics are not important (because communities with great concern for epistemic process do not care about them much), and AI researchers who do care about multi/multi dynamics have “bad epistemics” (e.g., because they have been biased by institutionalized trends). In fact, more than one LessWrong reader has taken these positions with me in private conversation, in good faith (I'm almost certain). In this post, I wish to share an opposing concern: that the EA and rationality communities have become systematically biased to ignore multi/multi dynamics, and power dynamics more generally. A history of systemic avoidance Epistemic status: self-evidently important considerations based on somewhat-publicly verifiable facts/trends. Our neglect of multi/multi dynamics has not been coincidental. For a time, influential thinkers in the rationality community intentionally avoided discussions of multi/multi dynamics, so as to avoid contributing to the sentiment that the development and use of AI technology would be driven by competitive (imperfectly cooperative) motives. (FWIW, I also did this sometimes.) The idea was that we — the rationality community — should avoid developing narratives that could provoke businesses and state leaders into worrying about whose values would be most represented in powerful AI systems, because that might lead them to go to war with each other, ideologically or physically. Indeed, there was a time when this community — particularly the Singularity Institute — represented a significant share of public discourse on the future of AI technology, and it made sense to be thoughtful about how to use that influence. Eliezer recently wrote (in a semi-private group, but with permission to share): The vague sense of assumed common purpose, in the era of AGI-alignment thinking from before Musk, was a fragile equilibrium, one that I had to fight to support every time some wise fool sniffed and said "Friendly to who?". Maybe somebody much weaker than Elon Musk could and inevitably would have smashed that equilibrium with much less of a financial investment, reducing Musk's "counterfactual impact". Maybe I'm an optimistic fool for thinking that this axis didn't just go from 0%-in-practice to 0%-in-practice. But I am still inclined to consider people a little responsible for the thing that they seem to have proximally caused according to surface appearances. That vague sense of common purpose might have become stronger if it had been given more time to grow and be formalized, rather than being smashed. That ship has now sailed. Perhaps it was right to worry that our narratives could trigger competition between states and companies, or perhaps the competitive dynamic was bound to emerge anyway and it was hubristic to think ourselves so important. Either way, carefully avoiding questions about multi/multi dynamics on LessWrong or The Alignment Forum will not turn back the clock...

power ai elon musk speech collective friendly ea modeling blindspot agi eliezer power dynamics blurry arches rationalist fwiw lesswrong singularity institute andrew critch

SIAI - An Examination by BrandonReinhart

Play Episode Listen Later Dec 12, 2021 24:54

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: SIAI - An Examination, published by BrandonReinhart on the LessWrong. 12/13/2011 - A 2011 update with data from the 2010 fiscal year is in progress. Should be done by the end of the week or sooner. Disclaimer I am not affiliated with the Singularity Institute for Artificial Intelligence. I have not donated to the SIAI prior to writing this. I made this pledge prior to writing this document. Notes Images are now hosted on LessWrong.com. The 2010 Form 990 data will be available later this month. It is not my intent to propagate misinformation. Errors will be corrected as soon as they are identified. Introduction Acting on gwern's suggestion in his Girl Scout Cookie analysis, I decided to look at SIAI funding. After reading about the Visiting Fellows Program and more recently the Rationality Boot Camp, I decided that the SIAI might be something I would want to support. I am concerned with existential risk and grapple with the utility implications. I feel that I should do more. I wrote on the mini-boot camp page a pledge that I would donate enough to send someone to rationality mini-boot camp. This seemed to me a small cost for the potential benefit. The SIAI might get better at building rationalists. It might build a rationalist who goes on to solve a problem. Should I donate more? I wasn't sure. I read gwern's article and realized that I could easily get more information to clarify my thinking. So I downloaded the SIAI's Form 990 annual IRS filings and started to write down notes in a spreadsheet. As I gathered data and compared it to my expectations and my goals, my beliefs changed. I now believe that donating to the SIAI is valuable. I cannot hide this belief in my writing. I simply have it. My goal is not to convince you to donate to the SIAI. My goal is to provide you with information necessary for you to determine for yourself whether or not you should donate to the SIAI. Or, if not that, to provide you with some direction so that you can continue your investigation. The SIAI's Form 990's are available at GuideStar and Foundation Center. You must register in order to access the files at GuideStar. 2002 (Form 990-EZ) 2003 (Form 990-EZ) 2004 (Form 990-EZ) 2005 (Form 990) 2006 (Form 990) 2007 (Form 990) 2008 (Form 990-EZ) 2009 (Form 990) SIAI Financial Overview The Singularity Institute for Artificial Intelligence (SIAI) is a public organization working to reduce existential risk from future technologies, in particular artificial intelligence. "The Singularity Institute brings rational analysis and rational strategy to the challenges facing humanity as we develop cognitive technologies that will exceed the current upper bounds on human intelligence." The SIAI are also the founders of Less Wrong. The graphs above offer an accurate summary of SIAI financial state since 2002. Sometimes the end of year balances listed in the Form 990 doesn't match what you'd get if you did the math by hand. These are noted as discrepancies between the filed year end balance and the expected year end balance or between the filed year start balance and the expected year start balance. Filing Error 1 - There appears to be a minor typo to the effect of $4.86 in the end of year balance for the 2004 document. It appears that Part I, Line 18 has been summed incorrectly. $32,445.76 is listed, but the expected result is $32,450.41. The Part II balance sheet calculations which agree with the error so the source of the error is unclear. The start of year balance in 2005 reflects the expected value so this was probably just a typo in 2004. The following year's reported start of year balance does not contain the error. Filing Error 2 - The 2006 document reports a year start balance of $95,105.00 when the expected year start balance is $165,284.00, a discrepancy of $70,179.00. This amount is close to ...

artificial intelligence speech irs ea errors examination ez girl scout cookies rationalist guidestar lesswrong foundation center singularity institute

The curse of identity by Kaj_Sotala

Play Episode Listen Later Dec 12, 2021 9:17

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The curse of identity, published by Kaj_Sotala on the LessWrong. So what you probably mean is, "I intend to do school to improve my chances on the market". But this statement is still false, unless it is also true that "I intend to improve my chances on the market". Do you, in actual fact, intend to improve your chances on the market? I expect not. Rather, I expect that your motivation is to appear to be the sort of person who you think you would be if you were ambitiously attempting to improve your chances on the market... which is not really motivating enough to actually DO the work. However, by persistently trying to do so, and presenting yourself with enough suffering at your failure to do it, you get to feel as if you are that sort of person without having to actually do the work. This is actually a pretty optimal solution to the problem, if you think about it. (Or rather, if you DON'T think about it!) -- PJ Eby I have become convinced that problems of this kind are the number one problem humanity has. I'm also pretty sure that most people here, no matter how much they've been reading about signaling, still fail to appreciate the magnitude of the problem. Here are two major screw-ups and one narrowly averted screw-up that I've been guilty of. See if you can find the pattern. When I began my university studies back in 2006, I felt strongly motivated to do something about Singularity matters. I genuinely believed that this was the most important thing facing humanity, and that it needed to be urgently taken care of. So in order to become able to contribute, I tried to study as much as possible. I had had troubles with procrastination, and so, in what has to be one of the most idiotic and ill-thought-out acts of self-sabotage possible, I taught myself to feel guilty whenever I was relaxing and not working. Combine an inability to properly relax with an attempted course load that was twice the university's recommended pace, and you can guess the results: after a year or two, I had an extended burnout that I still haven't fully recovered from. I ended up completing my Bachelor's degree in five years, which is the official target time for doing both your Bachelor's and your Master's. A few years later, I became one of the founding members of the Finnish Pirate Party, and on the basis of some writings the others thought were pretty good, got myself elected as the spokesman. Unfortunately – and as I should have known before taking up the post – I was a pretty bad choice for this job. I'm good at expressing myself in writing, and when I have the time to think. I hate talking with strangers on the phone, find it distracting to look people in the eyes when I'm talking with them, and have a tendency to start a sentence over two or three times before hitting on a formulation I like. I'm also bad at thinking quickly on my feet and coming up with snappy answers in live conversation. The spokesman task involved things like giving quick statements to reporters ten seconds after I'd been woken up by their phone call, and live interviews where I had to reply to criticisms so foreign to my thinking that they would never have occurred to me naturally. I was pretty terrible at the job, and finally delegated most of it to other people until my term ran out – though not before I'd already done noticeable damage to our cause. Last year, I was a Visiting Fellow at the Singularity Institute. At one point, I ended up helping Eliezer in writing his book. Mostly this involved me just sitting next to him and making sure he did get writing done while I surfed the Internet or played a computer game. Occasionally I would offer some suggestion if asked. Although I did not actually do much, the multitasking required still made me unable to spend this time productively myself, and for some reason i...

master internet identity bachelor curse speech ea combine singularity eliezer visiting fellow rationalist lesswrong singularity institute kaj sotala

The Power of Reinforcement by lukeprog

Play Episode Listen Later Dec 11, 2021 7:35

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Power of Reinforcement, published by lukeprog on the LessWrong. Part of the sequence: The Science of Winning at Life Also see: Basics of Animal Reinforcement, Basics of Human Reinforcement, Physical and Mental Behavior, Wanting vs. Liking Revisited, Approving reinforces low-effort behaviors, Applying Behavioral Psychology on Myself. Story 1: On Skype with Eliezer, I said: "Eliezer, you've been unusually pleasant these past three weeks. I'm really happy to see that, and moreover, it increases my probability than an Eliezer-led FAI research team will work. What caused this change, do you think?" Eliezer replied: "Well, three weeks ago I was working with Anna and Alicorn, and every time I said something nice they fed me an M&M." Story 2: I once witnessed a worker who hated keeping a work log because it was only used "against" him. His supervisor would call to say "Why did you spend so much time on that?" or "Why isn't this done yet?" but never "I saw you handled X, great job!" Not surprisingly, he often "forgot" to fill out his worklog. Ever since I got everyone at the Singularity Institute to keep work logs, I've tried to avoid connections between "concerned" feedback and staff work logs, and instead take time to comment positively on things I see in those work logs. Story 3: Chatting with Eliezer, I said, "Eliezer, I get the sense that I've inadvertently caused you to be slightly averse to talking to me. Maybe because we disagree on so many things, or something?" Eliezer's reply was: "No, it's much simpler. Our conversations usually run longer than our previously set deadline, so whenever I finish talking with you I feel drained and slightly cranky." Now I finish our conversations on time. Story 4: A major Singularity Institute donor recently said to me: "By the way, I decided that every time I donate to the Singularity Institute, I'll set aside an additional 5% for myself to do fun things with, as a motivation to donate." The power of reinforcement It's amazing to me how consistently we fail to take advantage of the power of reinforcement. Maybe it's because behaviorist techniques like reinforcement feel like they don't respect human agency enough. But if you aren't treating humans more like animals than most people are, then you're modeling humans poorly. You are not an agenty homunculus "corrupted" by heuristics and biases. You just are heuristics and biases. And you respond to reinforcement, because most of your motivation systems still work like the motivation systems of other animals. A quick reminder of what you learned in high school A reinforcer is anything that, when it occurs in conjunction with an act, increases the probability that the act will occur again. A positive reinforcer is something the subject wants, such as food, petting, or praise. Positive reinforcement occurs when a target behavior is followed by something the subject wants, and this increases the probability that the behavior will occur again. A negative reinforcer is something the subject wants to avoid, such as a blow, a frown, or an unpleasant sound. Negative reinforcement occurs when a target behavior is followed by some relief from something the subject doesn't want, and this increases the probability that the behavior will happen again. What works Small reinforcers are fine, as long as there is a strong correlation between the behavior and the reinforcer (Schneider 1973; Todorov et al. 1984). All else equal, a large reinforcer is more effective than a small one (Christopher 1988; Ludvig et al. 2007; Wolfe 1936), but the more you increase the reinforcer magnitude, the less benefit you get from the increase (Frisch & Dickinson 1990). The reinforcer should immediately follow the target behavior (Escobar & Bruner 2007; Schlinger & Blakely 1994; Schneider 1990). Pryor (2007) notes that...

power science story winning positive basics speech negative chatting ea schneider mm wolfe pryor fai eliezer reinforcement ludvig approving rationalist todorov lesswrong alicorn singularity institute on skype

The Power of Reinforcement by lukeprog