Podcasts about AGI

Are you determined to pay lower taxes legally?As a passive investor, you need to know how you can have LEGAL tax deductibles that will lower the tax you'll be paying. Focus on the 'big balls' like passive activity losses (PALs) that can reduce your adjusted gross income (AGI). When a taxpayer has a loss from a passive activity, they can use that loss to offset income from other passive activities, as well as income from non-passive activities, up to the amount of their taxable income from all sources. This can lower the taxpayer's AGI and reduce their overall tax liability.Learn other tax strategies by joining our mastermind group at https://simplepassivecashflow.com/club. Hosted on Acast. See acast.com/privacy for more information.

Link to original articleWelcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A Simple Alignment Typology, published by Shoshannah Tekofsky on January 28, 2023 on LessWrong. I set out to review the OpenAI alignment plan, and my brain at some point diverged to modeling the humans behind the arguments instead of the actual arguments. So behold! A simplified, first-pass Alignment Typology. Why can't we all just get agree? There are a lot of disagreements in AI alignment. Some people don't see the problem, some think we'll be fine, some think we're doomed, and then different clusters of people have different ideas on how we should go about solving alignment. Thus I tried to sketch out my understanding of the key differences between the largest clusters of views on AI alignment. What emerged are roughly five cluster, sorted in order of optimism about the fate of humanity: the sceptics, the humanists, the empiricists, the rationalists, and the fatalists. Sceptics don't expect AGI to show up in any relevant time frame. Humanists think humanity will prevail fairly easily through coordination around alignment or just solving the problem directly. Empiricists think the problem is hard, AGI will show up soon, and if we want to have any hope of solving it, then we need to iterate and take some necessary risk by making progress in capabilities while we go. Rationalists think the problem is hard, AGI will show up soon, and we need to figure out as much as we can before making any capabilities progress. Fatalists think we are doomed and we shouldn't even try (though some are quite happy about it). Here is a table. ScepticsHumanistsEmpiricistsTheoristsFatalistsAlignment Difficulty-highhigh-Coordination Difficulty-highhigh-Distance to AGIhigh-low/medlow/med---highmed/high---med/highhigh--highhighhighlow One of these is low Closeness to AGI required to Solve Alignment Closeness to AGI resulting in unacceptable danger Alignment Necessary or Possible Less Wrong is mostly populated by empiricists and rationalists. They agree alignment is a problem that can and should be solved. The key disagreement is on the methodology. While empiricists lean more heavily on gathering data and iterating solutions, rationalists lean more heavily toward discovering theories and proofs to lower risk from AGI (and some people are a mix of the two). Just by shifting the weights of risk/reward on iteration and moving forward, you get two opposite approaches to doing alignment work. How is this useful? Personally it helps me quickly get an idea of what clusters people are in, and understanding the likely arguments for their conclusions. However, a counterargument can be made that this just feeds into stereotyping and creating schisms, and I can't be sure that's untrue. What do you think? Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Reflections on Deception & Generality in Scalable Oversight (Another OpenAI Alignment Review), published by Shoshannah Tekofsky on January 28, 2023 on LessWrong. Just like you can test your skill in experimental design by reviewing existing experiments, you can test your skill in alignment by reviewing existing alignment strategies. Conveniently, Rob Bensinger, in name of Nate Soares and Eliezer Yudkowsky, recently posted a challenge to AI Safety researchers to review the OpenAI alignment plan written by Jan Leike, John Schulman, and Jeffrey Wu. I figured this constituted a test that might net me feedback from both sides of the rationalist-empiricist aisle. Yet, instead of finding ground-breaking arguments for or against scalable oversight to do alignment research, it seems Leike already knows what might go wrong — and goes ahead anyway. Thus my mind became split between evaluating the actual alignment plan and modeling the disagreement between prominent clusters of researchers. I wrote up the latter in an informal typology of AI Safety Researchers, and continued my technical review below. The following is a short summary of the OpenAI alignment plan, my views on the main problems, and a final section on recommendations for red lining. The Plan First, align AI with human feedback, then get AI to assist in giving human feedback to AI, then get AI to assist in giving human feedback to AI that is generating solutions to the alignment problem. Except, the steps are not sequential but run in parallel. This is one form of Scalable Oversight. Human feedback is Reinforcement Learning from Human Feedback (RLHF), the assisting AI is Iterated Distillation and Amplification (IDA) and Recursive Reward Modeling (RRM), and the AI that is generating solutions to the alignment problem is. still under construction. The target is a narrow AI that will make significant progress on the alignment problem. The MVP is a theorem prover. The full product is AGI utopia. Here is a graph. OpenAI explains its strategy succinctly and links to detailed background research. This is laudable, and hopefully other labs and organizations will follow suit. My understanding is also that if someone came along with a better plan then OpenAI would pivot in a heart beat. Which is even more laudable. The transparency, accountability, and flexibility they display set a strong example for other organizations working on AI. But the show must go on (from their point of view anyway) and so they are going ahead and implementing the most promising strategy that currently exists. Even if there are problems. And boy, are there problems. The Problems Jan Leike discusses almost all objections to the OpenAI alignment plan on his blog. Thus below I will only highlight the two most important problems in the plan, plus two additional concerns that I have not seen discussed so far. Alignment research requires general intelligence - If the alignment researcher AI has enough general intelligence to make breakthrough discoveries in alignment, then you can't safely create it without already having solved alignment. Yet, Leike et al. hope that relatively narrow intelligence can already make significant progress on alignment. I think this is extremely unlikely if we reflect on what general intelligence truly is. Though my own thoughts on the nature of intelligence are not entirely coherent yet, I'd argue that having a strong concept of intelligence is key to accurately predicting the outcome of an alignment strategy. Specifically in this case, my understanding is that general intelligence is being able to perform a wider set of operations on a wider set of inputs (to achieve a desired set of observations on the world state). For example, I can do addition of 2 apples I see, 2 apples I think about, 2 boats I hear about, 2 functions ...

Link to original articleWelcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Reflections on Deception & Generality in Scalable Oversight (Another OpenAI Alignment Review), published by Shoshannah Tekofsky on January 28, 2023 on LessWrong. Just like you can test your skill in experimental design by reviewing existing experiments, you can test your skill in alignment by reviewing existing alignment strategies. Conveniently, Rob Bensinger, in name of Nate Soares and Eliezer Yudkowsky, recently posted a challenge to AI Safety researchers to review the OpenAI alignment plan written by Jan Leike, John Schulman, and Jeffrey Wu. I figured this constituted a test that might net me feedback from both sides of the rationalist-empiricist aisle. Yet, instead of finding ground-breaking arguments for or against scalable oversight to do alignment research, it seems Leike already knows what might go wrong — and goes ahead anyway. Thus my mind became split between evaluating the actual alignment plan and modeling the disagreement between prominent clusters of researchers. I wrote up the latter in an informal typology of AI Safety Researchers, and continued my technical review below. The following is a short summary of the OpenAI alignment plan, my views on the main problems, and a final section on recommendations for red lining. The Plan First, align AI with human feedback, then get AI to assist in giving human feedback to AI, then get AI to assist in giving human feedback to AI that is generating solutions to the alignment problem. Except, the steps are not sequential but run in parallel. This is one form of Scalable Oversight. Human feedback is Reinforcement Learning from Human Feedback (RLHF), the assisting AI is Iterated Distillation and Amplification (IDA) and Recursive Reward Modeling (RRM), and the AI that is generating solutions to the alignment problem is. still under construction. The target is a narrow AI that will make significant progress on the alignment problem. The MVP is a theorem prover. The full product is AGI utopia. Here is a graph. OpenAI explains its strategy succinctly and links to detailed background research. This is laudable, and hopefully other labs and organizations will follow suit. My understanding is also that if someone came along with a better plan then OpenAI would pivot in a heart beat. Which is even more laudable. The transparency, accountability, and flexibility they display set a strong example for other organizations working on AI. But the show must go on (from their point of view anyway) and so they are going ahead and implementing the most promising strategy that currently exists. Even if there are problems. And boy, are there problems. The Problems Jan Leike discusses almost all objections to the OpenAI alignment plan on his blog. Thus below I will only highlight the two most important problems in the plan, plus two additional concerns that I have not seen discussed so far. Alignment research requires general intelligence - If the alignment researcher AI has enough general intelligence to make breakthrough discoveries in alignment, then you can't safely create it without already having solved alignment. Yet, Leike et al. hope that relatively narrow intelligence can already make significant progress on alignment. I think this is extremely unlikely if we reflect on what general intelligence truly is. Though my own thoughts on the nature of intelligence are not entirely coherent yet, I'd argue that having a strong concept of intelligence is key to accurately predicting the outcome of an alignment strategy. Specifically in this case, my understanding is that general intelligence is being able to perform a wider set of operations on a wider set of inputs (to achieve a desired set of observations on the world state). For example, I can do addition of 2 apples I see, 2 apples I think about, 2 boats I hear about, 2 functions ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A Simple Alignment Typology, published by Shoshannah Tekofsky on January 28, 2023 on LessWrong. I set out to review the OpenAI alignment plan, and my brain at some point diverged to modeling the humans behind the arguments instead of the actual arguments. So behold! A simplified, first-pass Alignment Typology. Why can't we all just get agree? There are a lot of disagreements in AI alignment. Some people don't see the problem, some think we'll be fine, some think we're doomed, and then different clusters of people have different ideas on how we should go about solving alignment. Thus I tried to sketch out my understanding of the key differences between the largest clusters of views on AI alignment. What emerged are roughly five cluster, sorted in order of optimism about the fate of humanity: the sceptics, the humanists, the empiricists, the rationalists, and the fatalists. Sceptics don't expect AGI to show up in any relevant time frame. Humanists think humanity will prevail fairly easily through coordination around alignment or just solving the problem directly. Empiricists think the problem is hard, AGI will show up soon, and if we want to have any hope of solving it, then we need to iterate and take some necessary risk by making progress in capabilities while we go. Rationalists think the problem is hard, AGI will show up soon, and we need to figure out as much as we can before making any capabilities progress. Fatalists think we are doomed and we shouldn't even try (though some are quite happy about it). Here is a table. ScepticsHumanistsEmpiricistsTheoristsFatalistsAlignment Difficulty-highhigh-Coordination Difficulty-highhigh-Distance to AGIhigh-low/medlow/med---highmed/high---med/highhigh--highhighhighlow One of these is low Closeness to AGI required to Solve Alignment Closeness to AGI resulting in unacceptable danger Alignment Necessary or Possible Less Wrong is mostly populated by empiricists and rationalists. They agree alignment is a problem that can and should be solved. The key disagreement is on the methodology. While empiricists lean more heavily on gathering data and iterating solutions, rationalists lean more heavily toward discovering theories and proofs to lower risk from AGI (and some people are a mix of the two). Just by shifting the weights of risk/reward on iteration and moving forward, you get two opposite approaches to doing alignment work. How is this useful? Personally it helps me quickly get an idea of what clusters people are in, and understanding the likely arguments for their conclusions. However, a counterargument can be made that this just feeds into stereotyping and creating schisms, and I can't be sure that's untrue. What do you think? Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Link to original articleWelcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: All AGI Safety questions welcome (especially basic ones) [~monthly thread], published by mwatkins on January 26, 2023 on LessWrong. tl;dr: Ask questions about AGI Safety as comments on this post, including ones you might otherwise worry seem dumb! Asking beginner-level questions can be intimidating, but everyone starts out not knowing anything. If we want more people in the world who understand AGI safety, we need a place where it's accepted and encouraged to ask about the basics. We'll be putting up monthly FAQ posts as a safe space for people to ask all the possibly-dumb questions that may have been bothering them about the whole AGI Safety discussion, but which until now they didn't feel able to ask. It's okay to ask uninformed questions, and not worry about having done a careful search before asking. AISafety.info - Interactive FAQ Additionally, this will serve as a way to spread the project Rob Miles' volunteer team has been working on: Stampy and his professional-looking face aisafety.info. Once we've got considerably more content this will provide a single point of access into AI Safety, in the form of a comprehensive interactive FAQ with lots of links to the ecosystem. We'll be using questions and answers from this thread for Stampy (under these copyright rules), so please only post if you're okay with that! You can help by adding other people's questions and answers or getting involved in other ways! We're not at the "send this to all your friends" stage yet, we're just ready to onboard a bunch of editors who will help us get to that stage :) We welcome feedback and questions on the UI/UX, policies, etc. around Stampy, as well as pull requests to his codebase. You are encouraged to add other people's answers from this thread to Stampy if you think they're good, and collaboratively improve the content that's already on our wiki. We've got a lot more to write before he's ready for prime time, but we think Stampy can become an excellent resource for everyone from skeptical newcomers, through people who want to learn more, right up to people who are convinced and want to know how they can best help with their skillsets. Guidelines for Questioners: No previous knowledge of AGI safety is required. If you want to watch a few of the Rob Miles videos, read either the WaitButWhy posts, or the The Most Important Century summary from OpenPhil's co-CEO first that's great, but it's not a prerequisite to ask a question. Similarly, you do not need to try to find the answer yourself before asking a question (but if you want to test Stampy's in-browser tensorflow semantic search that might get you an answer quicker!). Also feel free to ask questions that you're pretty sure you know the answer to, but where you'd like to hear how others would answer the question. One question per comment if possible (though if you have a set of closely related questions that you want to ask all together that's ok). If you have your own response to your own question, put that response as a reply to your original question rather than including it in the question itself. Remember, if something is confusing to you, then it's probably confusing to other people as well. If you ask a question and someone gives a good response, then you are likely doing lots of other people a favor! Guidelines for Answerers: Linking to the relevant canonical answer on Stampy is a great way to help people with minimal effort! Improving that answer means that everyone going forward will have a better experience! This is a safe space for people to ask stupid questions, so be kind! If this post works as intended then it will produce many answers for Stampy's FAQ. It may be worth keeping this in mind as you write your answer. For example, in some cases it might be worth giving a slightly longer / more expansive / more deta...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: All AGI Safety questions welcome (especially basic ones) [~monthly thread], published by mwatkins on January 26, 2023 on LessWrong. tl;dr: Ask questions about AGI Safety as comments on this post, including ones you might otherwise worry seem dumb! Asking beginner-level questions can be intimidating, but everyone starts out not knowing anything. If we want more people in the world who understand AGI safety, we need a place where it's accepted and encouraged to ask about the basics. We'll be putting up monthly FAQ posts as a safe space for people to ask all the possibly-dumb questions that may have been bothering them about the whole AGI Safety discussion, but which until now they didn't feel able to ask. It's okay to ask uninformed questions, and not worry about having done a careful search before asking. AISafety.info - Interactive FAQ Additionally, this will serve as a way to spread the project Rob Miles' volunteer team has been working on: Stampy and his professional-looking face aisafety.info. Once we've got considerably more content this will provide a single point of access into AI Safety, in the form of a comprehensive interactive FAQ with lots of links to the ecosystem. We'll be using questions and answers from this thread for Stampy (under these copyright rules), so please only post if you're okay with that! You can help by adding other people's questions and answers or getting involved in other ways! We're not at the "send this to all your friends" stage yet, we're just ready to onboard a bunch of editors who will help us get to that stage :) We welcome feedback and questions on the UI/UX, policies, etc. around Stampy, as well as pull requests to his codebase. You are encouraged to add other people's answers from this thread to Stampy if you think they're good, and collaboratively improve the content that's already on our wiki. We've got a lot more to write before he's ready for prime time, but we think Stampy can become an excellent resource for everyone from skeptical newcomers, through people who want to learn more, right up to people who are convinced and want to know how they can best help with their skillsets. Guidelines for Questioners: No previous knowledge of AGI safety is required. If you want to watch a few of the Rob Miles videos, read either the WaitButWhy posts, or the The Most Important Century summary from OpenPhil's co-CEO first that's great, but it's not a prerequisite to ask a question. Similarly, you do not need to try to find the answer yourself before asking a question (but if you want to test Stampy's in-browser tensorflow semantic search that might get you an answer quicker!). Also feel free to ask questions that you're pretty sure you know the answer to, but where you'd like to hear how others would answer the question. One question per comment if possible (though if you have a set of closely related questions that you want to ask all together that's ok). If you have your own response to your own question, put that response as a reply to your original question rather than including it in the question itself. Remember, if something is confusing to you, then it's probably confusing to other people as well. If you ask a question and someone gives a good response, then you are likely doing lots of other people a favor! Guidelines for Answerers: Linking to the relevant canonical answer on Stampy is a great way to help people with minimal effort! Improving that answer means that everyone going forward will have a better experience! This is a safe space for people to ask stupid questions, so be kind! If this post works as intended then it will produce many answers for Stampy's FAQ. It may be worth keeping this in mind as you write your answer. For example, in some cases it might be worth giving a slightly longer / more expansive / more deta...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AGI will have learnt utility functions, published by Beren Millidge on January 25, 2023 on The AI Alignment Forum. This post is part of the work done at Conjecture. Thanks to Eric Winsor, Daniel Braun, Andrea Miotti and Connor Leahy for helpful comments and feedback on the draft versions of this post. There has been a lot of debate and discussion recently in the AI safety community about whether AGI will likely optimize for fixed goals or be a wrapper mind. The term wrapper mind is largely a restatement of the old idea of a utility maximizer, with AIXI as a canonical example. The fundamental idea is that there is an agent with some fixed utility function which it maximizes without any kind of feedback which can change its utility function. Rather, the optimization process is assumed to be 'wrapped around' some core and unchanging utility function. The capabilities core of the agent is also totally modular and disjoint from the utility function such that arbitrary planners and utility functions can be composed so long as they have the right I/O interfaces. The core 'code' of an AIXI like agent is incredibly simple and, for instance, could be implemented in this Python pseudocode : def action_perception_loop(): while True: observation = self.sensors.get_observation() state = self.update_state(self.current_state, observation) all_action_plans = self.generate_action_plans(state) all_trajectories = self.world_model.generate_all_trajectories(all_action_plans, state) optimal_plan, optimal_utility = self.evaluate_trajectories(all_trajectories) self.execute(optimal_plan) There's a couple of central elements to this architecture which must be included in any AIXI-like architecture. The AGI needs some sensorimotor equipment to both sense the world and execute its action plans. It needs a Bayesian filtering component to be able to update its representation of the world state given new observations and its current state. It needs a world model that can generate sets of action plans and then generate 'rollouts' which are simulations of likely futures given an action plan. Finally, it needs a utility function that can calculate the utility of different simulated trajectories into the future and pick the best one. Let's zoom in on this component a little more and see how the evaluate_trajectories function might look inside. It might look like this: Essentially, the AIXI agent just takes all trajectories and ranks them according to its utility function and then picks the best one to execute. The fundamental problem with such an architecture, which is severely underappreciated, is that it implicitly assumes a utility oracle. That is, there exists some function self.utility_function() which is built into the agent from the beginning which can assign a consistent utility value to arbitrary world-states. While conceptually simple, my argument is that actually designing and building such a function into an agent to achieve a specific and complex goal in the external world is incredibly difficult or impossible for agents pursuing sufficiently complex goals and operating in sufficiently complex environments. This includes almost all goals humans are likely to want to program an AGI with. This means that in practice we cannot construct AIXI-like agents that optimize for arbitrary goals in the real world, and that any agent we do build must utilize some kind of learned utility model. Specifically, this is a utility (or reward) function uθ(x) where θ is some set of parameters and x is some kind of state, where the utility function is learned by some learning process (typically supervised learning) against a dataset of state, utility pairs that are provided either by the environment or by human designers. What this means is that, unlike a wrapper mind, the agent's utility function can be influe...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AGI will have learnt utility functions, published by Beren Millidge on January 25, 2023 on The AI Alignment Forum. This post is part of the work done at Conjecture. Thanks to Eric Winsor, Daniel Braun, Andrea Miotti and Connor Leahy for helpful comments and feedback on the draft versions of this post. There has been a lot of debate and discussion recently in the AI safety community about whether AGI will likely optimize for fixed goals or be a wrapper mind. The term wrapper mind is largely a restatement of the old idea of a utility maximizer, with AIXI as a canonical example. The fundamental idea is that there is an agent with some fixed utility function which it maximizes without any kind of feedback which can change its utility function. Rather, the optimization process is assumed to be 'wrapped around' some core and unchanging utility function. The capabilities core of the agent is also totally modular and disjoint from the utility function such that arbitrary planners and utility functions can be composed so long as they have the right I/O interfaces. The core 'code' of an AIXI like agent is incredibly simple and, for instance, could be implemented in this Python pseudocode : def action_perception_loop(): while True: observation = self.sensors.get_observation() state = self.update_state(self.current_state, observation) all_action_plans = self.generate_action_plans(state) all_trajectories = self.world_model.generate_all_trajectories(all_action_plans, state) optimal_plan, optimal_utility = self.evaluate_trajectories(all_trajectories) self.execute(optimal_plan) There's a couple of central elements to this architecture which must be included in any AIXI-like architecture. The AGI needs some sensorimotor equipment to both sense the world and execute its action plans. It needs a Bayesian filtering component to be able to update its representation of the world state given new observations and its current state. It needs a world model that can generate sets of action plans and then generate 'rollouts' which are simulations of likely futures given an action plan. Finally, it needs a utility function that can calculate the utility of different simulated trajectories into the future and pick the best one. Let's zoom in on this component a little more and see how the evaluate_trajectories function might look inside. It might look like this: Essentially, the AIXI agent just takes all trajectories and ranks them according to its utility function and then picks the best one to execute. The fundamental problem with such an architecture, which is severely underappreciated, is that it implicitly assumes a utility oracle. That is, there exists some function self.utility_function() which is built into the agent from the beginning which can assign a consistent utility value to arbitrary world-states. While conceptually simple, my argument is that actually designing and building such a function into an agent to achieve a specific and complex goal in the external world is incredibly difficult or impossible for agents pursuing sufficiently complex goals and operating in sufficiently complex environments. This includes almost all goals humans are likely to want to program an AGI with. This means that in practice we cannot construct AIXI-like agents that optimize for arbitrary goals in the real world, and that any agent we do build must utilize some kind of learned utility model. Specifically, this is a utility (or reward) function uθ(x) where θ is some set of parameters and x is some kind of state, where the utility function is learned by some learning process (typically supervised learning) against a dataset of state, utility pairs that are provided either by the environment or by human designers. What this means is that, unlike a wrapper mind, the agent's utility function can be influe...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: “Endgame safety” for AGI, published by Steve Byrnes on January 24, 2023 on The AI Alignment Forum. (Status: no pretense to originality, but a couple people said they found this terminology useful, so I'm sharing it more widely.) There's a category of AGI safety work that we might call “Endgame Safety”, where we're trying to do all the AGI safety work that we couldn't or didn't do ahead of time, in the very last moments before (or even after) people are actually playing around with powerful AGI algorithms of the type that could get irreversibly out of control and cause catastrophe. I think everyone agrees that Endgame Safety is important and unavoidable. If nothing else, for every last line of AGI source code, we can do an analysis of what happens if that line of code has a bug, or if a cosmic ray flips a bit, and how do we write good unit tests, etc. But we're obviously not going to have AGI source code until the endgame. That was an especially straightforward example, but I imagine that there will be many other things that also fall into the Endgame Safety bucket, i.e. bigger-picture important things to know about AGI that we only realize when we're in the thick of building it. So I am not an “Endgame Safety denialist”; I don't think anyone is. But I find that people are sometimes misled by thinking about Endgame Safety, in the following two ways: Bad argument 1: “Endgame Safety is really important. So let's try to make the endgame happen ASAP, so that we can get to work on Endgame Safety!” (example of this argument) This is a bad argument because, what's the rush? There's going to be an endgame sooner or later, and we can do Endgame Safety Research then! Bringing the endgame sooner is basically equivalent to having all the AI alignment and strategy researchers hibernate for some number N years, and then wake up and get back to work. And that, in turn, is strictly worse than having all the AI alignment and strategy researchers do what they can during the next N years, and also continue doing work after those N years have elapsed. I claim that there are plenty of open problems in AGI safety / alignment that we can do right now, that people are in fact working on right now, that seem robustly useful, and that are not in the category of “Endgame Safety”, e.g. my list of 7 projects, these 200 interpretability projects, this list, ELK, everything on Alignment Forum, etc. For example, sometimes I'll have this discussion: ME: “I don't want to talk about (blah) aspect of how I think future AGI will be built, because all my opinions are either wrong or infohazards—the latter because (if correct) they might substantially speed the arrival of AGI, which gives us less time for safety / alignment research.” THEM: “WTF dude, I'm an AGI safety / alignment researcher like you! That's why I'm standing here asking you these questions! And I assure you: if you answer my questions, it will help me do good AGI safety research.” So there's my answer. I claim that this person is trying to do Endgame Safety right now, and I don't want to help them. I think they should find something else to do right now instead, while they wait for some AI researcher to publish an answer to their prerequisite capabilities question. That's bound to happen sooner or later! Or they can do contingency-planning for each of the possible answers to their capabilities question. Whatever. Bad argument 2: “Endgame Safety researchers will obviously be in a much better position to do safety / alignment research than we are today, because they'll know more about how AGI works, and probably have proto-AGI test results, etc. So other things equal, we should move resources from current less-productive safety research to future more-productive Endgame Safety research.” The biggest problem here is that, while Endgame Safety...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Some of my disagreements with List of Lethalities, published by Alex Turner on January 24, 2023 on The AI Alignment Forum. This was an appendix of Inner and outer alignment decompose one hard problem into two extremely hard problems. However, I think the material is self-contained and worth sharing separately, especially since AGI Ruin: A List of Lethalities has become so influential. (I agree with most of the points made in AGI Ruin, but I'm going to focus on disagreements in this essay.) Here are some quotes with which I disagree, in light of points I made in Inner and outer alignment decompose one hard problem into two extremely hard problems (consult its TL;DR and detailed summary for a refresher, if need be). List of Lethalities “Humans don't explicitly pursue inclusive genetic fitness; outer optimization even on a very exact, very simple loss function doesn't produce inner optimization in that direction. This happens in practice in real life, it is what happened in the only case we know about, and it seems to me that there are deep theoretical reasons to expect it to happen again” (Evolution) (human values) is not the only case of inner alignment failure which we know about. I have argued that human values themselves are inner alignment failures on the human reward system. This has happened billions of times in slightly different learning setups. Strictly separately, it seems to me that people draw rather strong inferences from a rather loose analogy with evolution. I think that (evolution) (human values) is far less informative for alignment than (human reward circuitry) (human values). I don't agree with a strong focus on the former, given the latter is available as a source of information. We want to draw inferences about the mapping from (AI reward circuitry) (AI values), which is an iterative training process using reinforcement learning and self-supervised learning. Therefore, we should consider existing evidence about the (human reward circuitry) (human values) setup, which (AFAICT) also takes place using an iterative, local update process using reinforcement learning and self-supervised learning. Brain architecture and training is not AI architecture and training, so the evidence is going to be weakened. But for nearly every way in which (human reward circuitry) (human values) is disanalogous to (AI reward circuitry) (AI values), (evolution) (human values) is even more disanalogous! For more on this, see Quintin's post. Lethality #18: “When you show an agent an environmental reward signal, you are not showing it something that is a reliable ground truth about whether the system did the thing you wanted it to do; even if it ends up perfectly inner-aligned on that reward signal, or learning some concept that exactly corresponds to 'wanting states of the environment which result in a high reward signal being sent', an AGI strongly optimizing on that signal will kill you, because the sensory reward signal was not a ground truth about alignment (as seen by the operators).” My summary: Sensory reward signals are not ground truth on the agent's alignment to our goals. Even if you solve inner alignment, you're still dead. My response: We don't want to end up with an AI which primarily values its own reward, because then it wouldn't value humans. Beyond that, this item is not a “central” lethality (and a bunch of these central-to-EY lethalities are in fact about outer/inner). We don't need a function of sensory input which is safe to maximize, that's not the function of the reward signal. Reward chisels cognition. Reward is not necessarily—nor do we want it to be—a ground-truth signal about alignment. Lethality #19: “Insofar as the current paradigm works at all, the on-paper design properties say that it only works for aligning on known direct functions ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Some of my disagreements with List of Lethalities, published by Alex Turner on January 24, 2023 on The AI Alignment Forum. This was an appendix of Inner and outer alignment decompose one hard problem into two extremely hard problems. However, I think the material is self-contained and worth sharing separately, especially since AGI Ruin: A List of Lethalities has become so influential. (I agree with most of the points made in AGI Ruin, but I'm going to focus on disagreements in this essay.) Here are some quotes with which I disagree, in light of points I made in Inner and outer alignment decompose one hard problem into two extremely hard problems (consult its TL;DR and detailed summary for a refresher, if need be). List of Lethalities “Humans don't explicitly pursue inclusive genetic fitness; outer optimization even on a very exact, very simple loss function doesn't produce inner optimization in that direction. This happens in practice in real life, it is what happened in the only case we know about, and it seems to me that there are deep theoretical reasons to expect it to happen again” (Evolution) (human values) is not the only case of inner alignment failure which we know about. I have argued that human values themselves are inner alignment failures on the human reward system. This has happened billions of times in slightly different learning setups. Strictly separately, it seems to me that people draw rather strong inferences from a rather loose analogy with evolution. I think that (evolution) (human values) is far less informative for alignment than (human reward circuitry) (human values). I don't agree with a strong focus on the former, given the latter is available as a source of information. We want to draw inferences about the mapping from (AI reward circuitry) (AI values), which is an iterative training process using reinforcement learning and self-supervised learning. Therefore, we should consider existing evidence about the (human reward circuitry) (human values) setup, which (AFAICT) also takes place using an iterative, local update process using reinforcement learning and self-supervised learning. Brain architecture and training is not AI architecture and training, so the evidence is going to be weakened. But for nearly every way in which (human reward circuitry) (human values) is disanalogous to (AI reward circuitry) (AI values), (evolution) (human values) is even more disanalogous! For more on this, see Quintin's post. Lethality #18: “When you show an agent an environmental reward signal, you are not showing it something that is a reliable ground truth about whether the system did the thing you wanted it to do; even if it ends up perfectly inner-aligned on that reward signal, or learning some concept that exactly corresponds to 'wanting states of the environment which result in a high reward signal being sent', an AGI strongly optimizing on that signal will kill you, because the sensory reward signal was not a ground truth about alignment (as seen by the operators).” My summary: Sensory reward signals are not ground truth on the agent's alignment to our goals. Even if you solve inner alignment, you're still dead. My response: We don't want to end up with an AI which primarily values its own reward, because then it wouldn't value humans. Beyond that, this item is not a “central” lethality (and a bunch of these central-to-EY lethalities are in fact about outer/inner). We don't need a function of sensory input which is safe to maximize, that's not the function of the reward signal. Reward chisels cognition. Reward is not necessarily—nor do we want it to be—a ground-truth signal about alignment. Lethality #19: “Insofar as the current paradigm works at all, the on-paper design properties say that it only works for aligning on known direct functions ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Update to Samotsvety AGI timelines, published by Misha Yagudin on January 24, 2023 on The Effective Altruism Forum. Previously: Samotsvety's AI risk forecasts. Our colleagues at Epoch recently asked us to update our AI timelines estimate for their upcoming literature review on TAI timelines. We met on 2023-01-21 to discuss our predictions about when advanced AI systems will arrive. Forecasts: Definition of AGI We used the following definition to determine the “moment at which AGI is considered to have arrived,” building on this Metaculus question: The moment that a system capable of passing the adversarial Turing test against a top-5% human who has access to experts on various topics is developed. More concretely: A Turing test is said to be “adversarial” if the human judges make a good-faith attempt to unmask the AI as an impostor, and the human confederates make a good-faith attempt to demonstrate that they are humans. An AI is said to “pass” a Turing test if at least half of judges rated the AI as more human than at least third of the human confederates. This definition of AGI is not unproblematic, e.g., it's possible that AGI could be unmasked long after its economic value and capabilities are very high. We chose to use an imperfect definition and indicated to forecasters that they should interpret the definition not “as is” but “in spirit” to avoid annoying edge cases. Individual forecasts P(AGI by 2030)P(AGI by 2050)P(AGI by 2100)P(AGI by this year) = 10%P(AGI by this year) = 50%P(AGI by this year) = 90%F1F3F4F5F6F7F8F9 0.39 0.75 0.78 2028 2034 N/A 0.28 0.7 0.87 2027 2039 2120 0.26 0.58 0.93 2025 2039 2088 0.35 0.73 0.91 2025 2037 2075 0.4 0.65 0.8 2025 2035 N/A 0.33 0.65 0.8 2026 2037 2250 0.2 0.5 0.7 2026 2050 2200 0.23 0.44 0.67 2026 2060 2250 Aggregate P(AGI by 2030)P(AGI by 2050)P(AGI by 2100)P(AGI by this year) = 10%P(AGI by this year) = 50%P(AGI by this year) = 90% mean: 0.31 0.63 0.81 2026 2041 2164 stdev: 0.07 0.11 0.09 1.07 8.99 79.65 50% CI: [0.26, 0.35] [0.55, 0.70] [0.74, 0.87] [2025.3, 2026.7] [2035, 2047] [2110, 2218] 80% CI: [0.21, 0.40] [0.48, 0.77] [0.69, 0.93] [2024.6, 2027.4] [2030, 2053] [2062, 2266] 95% CI: [0.16, 0.45] [0.41, 0.84] [0.62, 0.99] [2023.9, 2028.1] [2024, 2059] [2008, 2320] geomean: 0.30 0.62 0.80 2026.00 2041 2163 geo odds: 0.30 0.63 0.82 Epistemic status: For Samotsvety track-record see:/ Note that this track record comes mostly from questions about geopolitics and technology that resolve within 12 months. Most forecasters have at least read Joe Carlsmith's report on AI x-risk, Is “Power-Seeking AI an Existential Risk?”. Those who are short on time may have just skimmed the report and/or watched the presentation. We discussed the report section by section over the course of a few weekly meetings. Note also that there might be selection effects at the level of which forecasters chose to participate in this exercise; for example, Samotsvety forecasters who view AI as an important/interesting/etc. topic could have self-selected into the discussion. (Though, the set of forecasters who participated this time and participated last time is very similar.) Update from our previous estimate The last time we publicly elicited a similar probability from our forecasters, we were at 32% that AGI would be developed in the next 20 years (so by late 2042); and at 73% that it would be developed by 2100. These are a bit lower than our current forecasts. The changes since then can be attributed to We have gotten more time to think about the topic, and work through considerations and counter-considerations, e.g., the extent to which we should fear selection effects in the types of arguments to which we are exposed. Some of our forecasters still give substantial weight to more skeptical probabilities coming from semi-informative priors, from Lap...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Has private AGI research made independent safety research ineffective already? What should we do about this?, published by Roman Leventov on January 23, 2023 on LessWrong. This post is a variation on "Private alignment research sharing and coordination" by porby. You can consider my question as signal-boosting that post. AGI research is becoming private. Research at MIRI is nondisclosed-by-default for more than four years now. OpenAI stopped publishing details of their work, Hassabis also talked about this here. Does this mean that independent AI safety research begins to suffer from knowledge asymmetry and becomes ineffective? There are two directions possible directions of knowledge asymmetry: State-of-the-art scaling results or even novel architectures are not published, and interpretability researchers use outdated models in their work. Hence, these results may not generalise to bigger model scales and architectures. The counterargument here is that in what comes to scale, relatively low-hanging interpretability fruit is still possible to pick even when analysing toy models. In what comes to architectures, transformer details may not matter that much for interpretability (however, Anthropic's SoLU (Softmax Linear Unit) work seems to be the evidence against this statement: relatively minor architectural change has led to significant changes in the interpretability characteristics of the model); and if one of the AGI labs stumbles upon a major "post-transformer" breakthrough, this is going to be an extremely closely-guarded secret which will not spread with rumours to AI safety labs, and hence joining the "knowledge space" of these AI safety labs won't help independent AI safety researchers. Some safety research has already been done but was not published, either because of its relative infohazardousness or because it references private capability research, as discussed in the previous point. Oblivious to this work, AI safety researchers may reinvent the wheel rather than work on the actual frontier. Private research space If the issue described above is real, maybe the community needs some organisational innovation to address it. Perhaps it could be some NDA + "noncompete" agreement + publishing restriction program led by some AI safety lab (or a consortium of AI safety labs) joining which is not the same as joining the lab itself in the usual sense (no reporting, no salary), but which grants access to the private knowledge space of the lab(s). The effect for the lab will be as if they enlarged their research group, the extra part of which is not controllable and is not guaranteed to contribute to their agenda, but costs them little: there are only costs for supporting the legal and IT infrastructure of the "private research space". There are some concerns that put the usefulness of this system in question, though. First, there may be too few people who will be willing to join this private research space without joining the lab(s) themselves. Academics, including PhD students, want to publish. Only independent AI safety researchers may be eligible, which is a much smaller pool of people. Furthermore, the reasons why people opt for being independent AI safety researchers sometimes correlate with low involvement or low capability (of conducting good research), so these people who join the private research space may not be able to push the frontier by much. Second, the marginal usefulness of the work done by independent people in the private research space may be offset by the marginally higher risk of information leakage. On the other hand, if the private research space is organised not by a single AI safety lab but by a consortium (or organised by a single lab but more labs join this research space later), the first concern above becomes irrelevant and the second c...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Thoughts on hardware / compute requirements for AGI, published by Steve Byrnes on January 24, 2023 on The AI Alignment Forum. Let's say I know how to build / train a human-level (more specifically, John von Neumann level) AGI. And let's say that we (and/or the AGI itself) have already spent a few years on making the algorithm work better and more efficiently. Question: How much compute will it take to run this AGI? (NB: I said "running" an AGI, not training / programming an AGI. I'll talk a bit about “training compute” at the very end.) Answer: I don't know. But that doesn't seem to be stopping me from writing this post. ¯_(ツ)_/¯ My current feeling—which I can easily imagine changing after discussion (which is a major reason I'm writing this!)—seems to be: 75%: One current (Jan 2023) high-end retail gaming PC (with an Nvidia GeForce RTX 4090 GPU) will be enough (or more than enough) for human-level human-speed AGI, 85%: One future high-end retail gaming PC, that will on sale in a decade (2033), will be enough for human-level AGI, at ≥20% human speed. This post will explain why I currently feel this way. Table of Contents / TL;DR In the prologue (Section 1), I'll give three reasons that I care about this question: one related to our long-term prospects of globally monitoring and regulating human-level AGI; one related to whether an early AGI could be “self-sufficient” after wiping out humanity; and one related to whether AGI is even feasible in the first place. I'll also respond to two counterarguments (i.e. arguments that I shouldn't care about this question), namely: “More-scaled-up AGIs will always be smarter than less-scaled-up AGIs; that relative comparison is what we care about, not the absolute intelligence level that's possible, on, say, a single GPU”, and “The very earliest human-level AGIs will be just barely human-level on the world's biggest compute clusters, and that's the thing that we should mainly care about, not how efficient they wind up later on”. In Section 2, I'll touch on a bit of prior discussion that I found interesting or thought-provoking, including a claim by Eliezer Yudkowsky that human-level human-speed AGI requires ridiculously little compute, and conversely a Metaculus forecast expecting that it requires orders of magnitude more compute than what I'm claiming here. In Section 3, I'll argue that the amount of computation used by the human brain is a good upper bound for my question. Then in Section 3.1 I'll talk about compute requirements by starting with the “mechanistic method” in Joe Carlsmith's report in brain computation and arguing for some modest adjustments in the “less compute” direction. Next in Section 3.2 I'll talk about memory requirements, arguing for the (initially-surprising-to-me) conclusion that the brain has orders of magnitude fewer bits of learned information than it has synapses—100 trillion synapses versus ≲100 billion bits of incompressible information. Putting these together in Section 3.3, I reach the conclusion (mentioned at the top) that a retail gaming GPU will probably be plenty for human-level human-speed AGI. Finally I'll talk about my lingering doubts in Section 3.3.1, by listing a few of the most plausible-to-me reasons that my conclusion might be wrong. In Section 4, I'll move on from running an AGI to training it (from scratch). This is a short section, where I mostly wanted to raise awareness of the funny fact that the ratio of training-compute to deployed-compute seems to be ≈7 orders of magnitude lower if you estimate it by looking at brains, versus if you estimate it by extrapolating from today's self-supervised language models. I don't have a great explanation why. On the other hand, perhaps surprisingly, I claim that resolving this question doesn't seem particularly important for AGI governance q...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Alexander and Yudkowsky on AGI goals, published by Scott Alexander on January 24, 2023 on The AI Alignment Forum. This is a lightly edited transcript of a chatroom conversation between Scott Alexander and Eliezer Yudkowsky last year, following up on the Late 2021 MIRI Conversations. Questions discussed include "How hard is it to get the right goals into AGI systems?" and "In what contexts do AI systems exhibit 'consequentialism'?". 1. Analogies to human moral development [Yudkowsky][13:29] @ScottAlexander ready when you are [Alexander][13:31] Okay, how do you want to do this? [Yudkowsky][13:32] If you have an agenda of Things To Ask, you can follow it; otherwise I can start by posing a probing question or you can? We've been very much winging it on these and that has worked... as well as you have seen it working! [Alexander][13:34] Okay. I'll post from my agenda. I'm assuming we both have the right to edit logs before releasing them? I have one question where I ask about a specific party where your real answer might offend some people it's bad to offend - if that happens, maybe we just have that discussion and then decide if we want to include it later? [Yudkowsky][13:34] Yup, both parties have rights to edit before releasing. [Alexander][13:34] Okay. One story that psychologists tell goes something like this: a child does something socially proscribed (eg steal). Their parents punish them. They learn some combination of "don't steal" and "don't get caught stealing". A few people (eg sociopaths) learn only "don't get caught stealing", but most of the rest of us get at least some genuine aversion to stealing that eventually generalizes into a real sense of ethics. If a sociopath got absolute power, they would probably steal all the time. But there are at least a few people whose ethics would successfully restrain them. I interpret a major strain in your thought as being that we're going to train fledgling AIs to do things like not steal, and they're going to learn not to get caught stealing by anyone who can punish them. Then, once they're superintelligent and have absolute power, they'll reveal that it was all a lie, and steal whenever they want. Is this worry at the level of "we can't be sure they won't do this"? Or do you think it's overwhelmingly likely? If the latter, what makes you think AIs won't internalize ethical prohibitions, even though most children do? Is it that evolution has given us priors to interpret reward/punishment in a moralistic and internalized way, and entities without those priors will naturally interpret them in a superficial way? Do we understand what those priors "look like"? Is finding out what features of mind design and training data cause internalization vs. superficial compliance a potential avenue for AI alignment? [Yudkowsky][13:36] Several layers here! The basic gloss on this is "Yes, everything that you've named goes wrong simultaneously plus several other things. If I'm wrong and one or even three of those things go exactly like they do in neurotypical human children instead, this will not be enough to save us." If AI is built on anything like the present paradigm, or on future paradigms either really, you can't map that onto the complicated particular mechanisms that get invoked by raising a human child, and expect the same result. [Alexander][13:37] (give me some sign when you're done answering) [Yudkowsky][13:37] (it may be a while but you should probably also just interrupt) especially if I say something that already sounds wrong [Alexander: ] the old analogy I gave was that some organisms will develop thicker fur coats if you expose them to cold weather. this doesn't mean the organism is simple and the complicated information about fur coats was mostly in the environment, and that you could expose an organism from a differe...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Parameter Scaling Comes for RL, Maybe, published by 1a3orn on January 24, 2023 on LessWrong. TLDR Unlike language models or image classifiers, past reinforcement learning models did not reliably get better as they got bigger. Two DeepMind RL papers published in January 2023 nevertheless show that with the right techniques, scaling up RL model parameters can increase both total reward and sample-efficiency of RL agents -- and by a lot. Return-to-scale has been key for rendering language models powerful and economically valuable; it might also be key for RL, although many important questions remain unanswered. Intro Reinforcement learning models often have very few parameters compared to language and image models. The Vision Transformer has 2 billion parameters. GPT-3 has 175 billion. The slimmer Chinchilla, trained in accord with scaling laws emphasizing bigger datasets, has 70 billion. By contrast, until a month ago, the largest mostly-RL models I knew of were the agents for Starcraft and Dota2, AlphaStar and OpenAI5, which had 139 million and 158 million parameters. And most RL models are far smaller, coming in well under 50 million parameters. The reason RL hasn't scaled up the size of its models is simple -- doing so generally hasn't made them better. Increasing model size in RL can even hurt performance. MuZero Reanalyze gets worse on some tasks as you scale network size. So does a vanilla SAC agent. There has been good evidence for scaling model size in somewhat... non-central examples of RL. For instance, offline RL agents trained from expert examples, such as DeepMind's 1.2-billion parameter Gato or Multi-Game Decision Transformers, clearly get better with scale. Similarly, RL from human feedback on language models generally shows that larger LM's are better. Hybrid systems such as PaLM SayCan benefit from larger language models. But all these cases sidestep problems central to RL -- they have no need to balance exploration and exploitation in seeking reward. In the typical RL setting -- there has generally been little scaling and little evidence for the efficacy of scaling. (Although there has not been no evidence.) None of the above means that the compute spent on RL models is small or that compute scaling does nothing for them. AlphaStar used only a little less compute than GPT-3, and AlphaGo Zero used more, because both of them trained on an enormous number of games. Additional compute predictably improves performance of RL agents. But, rather than getting a bigger brain, almost all RL algorithms spend this compute by (1) training on an enormous number of games (2) or (if concerned with sample-efficiency) by revisiting the games that they've played an enormous number of times. So for a while RL has lacked: (1) The ability to scale up model size to reliably improve performance. (2) (Even supposing the above were around) Any theory like the language-model scaling laws which would let you figure out how to allocate compute between model size / longer training. My intuition is that the lack of (1), and to a lesser degree the lack of (2), is evidence that no one has stumbled on the "right way" to do RL or RL-like problems. It's like language modeling when it only had LSTMS and no Transformers, before the frighteningly straight lines in log-log charts appeared. In the last month, though, two RL papers came out with interesting scaling charts, each showing strong gains to parameter scaling. Both were (somewhat unsurprisingly) from DeepMind. This is the kind of thing that leads me to think "Huh, this might be an important link in the chain that brings about AGI." The first paper is "Mastering Diverse Domains Through World Models", which names its agent DreamerV3. The second is "Human-Timescale Adaptation in an Open-Ended Task Space", which names its agent Adaptive...

Link to original articleWelcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Parameter Scaling Comes for RL, Maybe, published by 1a3orn on January 24, 2023 on LessWrong. TLDR Unlike language models or image classifiers, past reinforcement learning models did not reliably get better as they got bigger. Two DeepMind RL papers published in January 2023 nevertheless show that with the right techniques, scaling up RL model parameters can increase both total reward and sample-efficiency of RL agents -- and by a lot. Return-to-scale has been key for rendering language models powerful and economically valuable; it might also be key for RL, although many important questions remain unanswered. Intro Reinforcement learning models often have very few parameters compared to language and image models. The Vision Transformer has 2 billion parameters. GPT-3 has 175 billion. The slimmer Chinchilla, trained in accord with scaling laws emphasizing bigger datasets, has 70 billion. By contrast, until a month ago, the largest mostly-RL models I knew of were the agents for Starcraft and Dota2, AlphaStar and OpenAI5, which had 139 million and 158 million parameters. And most RL models are far smaller, coming in well under 50 million parameters. The reason RL hasn't scaled up the size of its models is simple -- doing so generally hasn't made them better. Increasing model size in RL can even hurt performance. MuZero Reanalyze gets worse on some tasks as you scale network size. So does a vanilla SAC agent. There has been good evidence for scaling model size in somewhat... non-central examples of RL. For instance, offline RL agents trained from expert examples, such as DeepMind's 1.2-billion parameter Gato or Multi-Game Decision Transformers, clearly get better with scale. Similarly, RL from human feedback on language models generally shows that larger LM's are better. Hybrid systems such as PaLM SayCan benefit from larger language models. But all these cases sidestep problems central to RL -- they have no need to balance exploration and exploitation in seeking reward. In the typical RL setting -- there has generally been little scaling and little evidence for the efficacy of scaling. (Although there has not been no evidence.) None of the above means that the compute spent on RL models is small or that compute scaling does nothing for them. AlphaStar used only a little less compute than GPT-3, and AlphaGo Zero used more, because both of them trained on an enormous number of games. Additional compute predictably improves performance of RL agents. But, rather than getting a bigger brain, almost all RL algorithms spend this compute by (1) training on an enormous number of games (2) or (if concerned with sample-efficiency) by revisiting the games that they've played an enormous number of times. So for a while RL has lacked: (1) The ability to scale up model size to reliably improve performance. (2) (Even supposing the above were around) Any theory like the language-model scaling laws which would let you figure out how to allocate compute between model size / longer training. My intuition is that the lack of (1), and to a lesser degree the lack of (2), is evidence that no one has stumbled on the "right way" to do RL or RL-like problems. It's like language modeling when it only had LSTMS and no Transformers, before the frighteningly straight lines in log-log charts appeared. In the last month, though, two RL papers came out with interesting scaling charts, each showing strong gains to parameter scaling. Both were (somewhat unsurprisingly) from DeepMind. This is the kind of thing that leads me to think "Huh, this might be an important link in the chain that brings about AGI." The first paper is "Mastering Diverse Domains Through World Models", which names its agent DreamerV3. The second is "Human-Timescale Adaptation in an Open-Ended Task Space", which names its agent Adaptive...

Link to original articleWelcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Has private AGI research made independent safety research ineffective already? What should we do about this?, published by Roman Leventov on January 23, 2023 on LessWrong. This post is a variation on "Private alignment research sharing and coordination" by porby. You can consider my question as signal-boosting that post. AGI research is becoming private. Research at MIRI is nondisclosed-by-default for more than four years now. OpenAI stopped publishing details of their work, Hassabis also talked about this here. Does this mean that independent AI safety research begins to suffer from knowledge asymmetry and becomes ineffective? There are two directions possible directions of knowledge asymmetry: State-of-the-art scaling results or even novel architectures are not published, and interpretability researchers use outdated models in their work. Hence, these results may not generalise to bigger model scales and architectures. The counterargument here is that in what comes to scale, relatively low-hanging interpretability fruit is still possible to pick even when analysing toy models. In what comes to architectures, transformer details may not matter that much for interpretability (however, Anthropic's SoLU (Softmax Linear Unit) work seems to be the evidence against this statement: relatively minor architectural change has led to significant changes in the interpretability characteristics of the model); and if one of the AGI labs stumbles upon a major "post-transformer" breakthrough, this is going to be an extremely closely-guarded secret which will not spread with rumours to AI safety labs, and hence joining the "knowledge space" of these AI safety labs won't help independent AI safety researchers. Some safety research has already been done but was not published, either because of its relative infohazardousness or because it references private capability research, as discussed in the previous point. Oblivious to this work, AI safety researchers may reinvent the wheel rather than work on the actual frontier. Private research space If the issue described above is real, maybe the community needs some organisational innovation to address it. Perhaps it could be some NDA + "noncompete" agreement + publishing restriction program led by some AI safety lab (or a consortium of AI safety labs) joining which is not the same as joining the lab itself in the usual sense (no reporting, no salary), but which grants access to the private knowledge space of the lab(s). The effect for the lab will be as if they enlarged their research group, the extra part of which is not controllable and is not guaranteed to contribute to their agenda, but costs them little: there are only costs for supporting the legal and IT infrastructure of the "private research space". There are some concerns that put the usefulness of this system in question, though. First, there may be too few people who will be willing to join this private research space without joining the lab(s) themselves. Academics, including PhD students, want to publish. Only independent AI safety researchers may be eligible, which is a much smaller pool of people. Furthermore, the reasons why people opt for being independent AI safety researchers sometimes correlate with low involvement or low capability (of conducting good research), so these people who join the private research space may not be able to push the frontier by much. Second, the marginal usefulness of the work done by independent people in the private research space may be offset by the marginally higher risk of information leakage. On the other hand, if the private research space is organised not by a single AI safety lab but by a consortium (or organised by a single lab but more labs join this research space later), the first concern above becomes irrelevant and the second c...

Tax Tuesday is here again. Toby Mathis hosts, with special guest Eliot Thomas from Anderson Advisors, here to help answer your questions. On today's episode, Eliot has grabbed a bunch of great questions for us to answer. Toby and Eliot will talk about the Augusta rule, easy tax deductions against W-2 income, cost segregation, bonus depreciation, real estate professional status, active participation, S-Corp, C-Corp and partnership advantages.  Online, we have Ander, Patti, Ian, Dana, Matthew, Jared, Piao, Tanya, Troy, and Dutch, a multitude of CPAs, by the way, in our Q&A. If you ask questions in Q&A, you're going to get really, really smart people answering that question. Toby sends out a a huge public thank you to all these talented people. If you have a tax-related question for us, submit it to taxtuesday@andersonadvisors. Highlights/Topics: "I'm selling a property that was willed to be in 2019. I've been renting this property out since receiving it. It will sell for a profit of over $360,000. Would I pay taxes on the full profit or the difference between value at the time the property was willed or do I pay taxes on the difference between the profit and $250,000?” - You inherited it in 2019. It says you've immediately started renting it out, so it's an investment property. It's not going to qualify for the capital gain exclusion of living in our primary residence for two of the last five years. "What are some simple easy things that can be done to reduce taxable income and reduce taxes paid on each of my paychecks?” Donate to charity in large chunks, HSA, IRAs, etc. "Options for tax write-off, reducing tax burden if I have rental real estate, but I am not a full-time real estate professional. Both my wife and I have W-2 jobs that we don't foresee leaving anytime soon to become real estate investors." - See the answer to previous question, and also you want to look at if your AGI (adjusted gross income) is a little bit lower, maybe under $100,000, you can take up to $25,000 of the passive losses. "Augusta Rule: We have put our properties in a Wyoming entity and the Texas series LLC in late December of 2022, but have not started using it yet. Can we use the Augusta Rule in 2022 throughout the year for our business purposes, even though we've not completed setting up the business?" Augusta Rule, that's just what we call 280A most often. That's the ability to rent out your home. Dwelling is the proper term for no more than 14 days a calendar year. The income you receive, you don't have to pay tax on. “When a rehab required property acquired for long-term hold, when is the right time to do the cost segregation study? Before the rehab or after?" - Once you purchase a property or after the rehab, you could do it either way. If you don't do what's called a cost seg study, the IRS will let you treat it all as 27½ years… "Anderson created my S-corp entity in November of 2022. I've only had expenses for the year-end 2022, but no income or property purchases yet. What am I required to file for my S-corp regarding the expenses I've incurred?" – You're going to have to file your tax return for that S-corp. It is what we call an informational return. In other words, your S-corp doesn't pay any tax, but it does have a tax return called an 1120-S. "I created my two LLCs both with real estate assets with rental income in 2022. Also, I created a holding company that holds both the LLCs. I have a W-2 job. When do I file the tax for the holding company? Is it one tax filing that combines all the LLCs and my W-2?” - We recommend that the holding company becomes a partnership. Also, it helps from a lending standpoint. Typically, lenders are able to lend more to you being that the property is in a partnership than if it had been in a direct disregarded LLC. "Curious to hear an open discussion about one and how to utilize section 179 and/or bonus depreciation for vehicles." - Why not just do mileage reimbursement? It's like 65.5¢ a mile right now. It's your car. You can use non-commercial insurance. It could just be your car that you use. If you let employees use it, that goes out the window. "What are the steps to take in order to withdraw money from a C-corp account? Are there any tax consequences involved?" - With a C-corporation, the first thing I'd like to look at are the reimbursements… "How to save taxes as S-corp, and is it better to do an STD deduction?" - The S-corp has a lot of advantages to it to save on taxes. Standard deduction is huge for most people. But “it depends”. "Can you please touch upon what depreciation recapture is and how it impacts taxes?" - Basically, when you have an asset that's been used in a trade or business, we don't deduct the full cost of it immediately. We take a little bit over time, we call it depreciation. Then when you resell, you might have what's called depreciation recapture on that depreciation that you took over the years. It does depend on what kind of asset it is. "I work from my home office. How do I claim this?" - If you have a sole proprietorship, you can take a deduction for basically the percentage square use of that house, that's an easy way to describe it. If you could get reimbursed, then it could be 20% of your house. By the way, that includes mortgage interest, property taxes. If you have somebody coming in cleaning your house, your utilities. Be sure to subscribe to our podcast. And if you are already a subscriber, please provide us a review of what you thought! Resources: Email us at Tax Tuesday taxtuesday@andersonadvisors.com Tax and Asset Protection Events https://andersonadvisors.com/real-estate-asset-protection-workshop-training/ Anderson Advisors https://andersonadvisors.com/ Anderson Advisors on YouTube https://www.youtube.com/channel/UCaL-wApuVYi2Va5dWzyTYVw

With ChatGPT in the news, I thought it was high time we take a look at OpenAI -- the company behind the controversial chatbot. From its founding in 2015 to its shift to a "capped-profit" company, we look at the organization founded with the goal of creating AI that's beneficial for humanity.See omnystudio.com/listener for privacy information.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Alexander and Yudkowsky on AGI goals, published by Scott Alexander on January 24, 2023 on The AI Alignment Forum. This is a lightly edited transcript of a chatroom conversation between Scott Alexander and Eliezer Yudkowsky last year, following up on the Late 2021 MIRI Conversations. Questions discussed include "How hard is it to get the right goals into AGI systems?" and "In what contexts do AI systems exhibit 'consequentialism'?". 1. Analogies to human moral development [Yudkowsky][13:29] @ScottAlexander ready when you are [Alexander][13:31] Okay, how do you want to do this? [Yudkowsky][13:32] If you have an agenda of Things To Ask, you can follow it; otherwise I can start by posing a probing question or you can? We've been very much winging it on these and that has worked... as well as you have seen it working! [Alexander][13:34] Okay. I'll post from my agenda. I'm assuming we both have the right to edit logs before releasing them? I have one question where I ask about a specific party where your real answer might offend some people it's bad to offend - if that happens, maybe we just have that discussion and then decide if we want to include it later? [Yudkowsky][13:34] Yup, both parties have rights to edit before releasing. [Alexander][13:34] Okay. One story that psychologists tell goes something like this: a child does something socially proscribed (eg steal). Their parents punish them. They learn some combination of "don't steal" and "don't get caught stealing". A few people (eg sociopaths) learn only "don't get caught stealing", but most of the rest of us get at least some genuine aversion to stealing that eventually generalizes into a real sense of ethics. If a sociopath got absolute power, they would probably steal all the time. But there are at least a few people whose ethics would successfully restrain them. I interpret a major strain in your thought as being that we're going to train fledgling AIs to do things like not steal, and they're going to learn not to get caught stealing by anyone who can punish them. Then, once they're superintelligent and have absolute power, they'll reveal that it was all a lie, and steal whenever they want. Is this worry at the level of "we can't be sure they won't do this"? Or do you think it's overwhelmingly likely? If the latter, what makes you think AIs won't internalize ethical prohibitions, even though most children do? Is it that evolution has given us priors to interpret reward/punishment in a moralistic and internalized way, and entities without those priors will naturally interpret them in a superficial way? Do we understand what those priors "look like"? Is finding out what features of mind design and training data cause internalization vs. superficial compliance a potential avenue for AI alignment? [Yudkowsky][13:36] Several layers here! The basic gloss on this is "Yes, everything that you've named goes wrong simultaneously plus several other things. If I'm wrong and one or even three of those things go exactly like they do in neurotypical human children instead, this will not be enough to save us." If AI is built on anything like the present paradigm, or on future paradigms either really, you can't map that onto the complicated particular mechanisms that get invoked by raising a human child, and expect the same result. [Alexander][13:37] (give me some sign when you're done answering) [Yudkowsky][13:37] (it may be a while but you should probably also just interrupt) especially if I say something that already sounds wrong [Alexander: ] the old analogy I gave was that some organisms will develop thicker fur coats if you expose them to cold weather. this doesn't mean the organism is simple and the complicated information about fur coats was mostly in the environment, and that you could expose an organism from a differe...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What a compute-centric framework says about AI takeoff speeds - draft report, published by Tom Davidson on January 23, 2023 on The AI Alignment Forum. As part of my work for Open Philanthropy I've written a draft report on AI takeoff speeds, the question of how quickly AI capabilities might improve as we approach and surpass human-level AI. Will human-level AI be a bolt from the blue, or will we have AI that is nearly as capable many years earlier? Most of the analysis is from the perspective of a compute-centric framework, inspired by that used in the Bio Anchors report, in which AI capabilities increase continuously with more training compute and work to develop better AI algorithms. This post doesn't summarise the report. Instead I want to explain some of the high-level takeaways from the research which I think apply even if you don't buy the compute-centric framework. The framework h/t Dan Kokotajlo for writing most of this section This report accompanies and explains (h/t Epoch for building this!), a user-friendly quantitative model of AGI timelines and takeoff, which you can go play around with right now. (By AGI I mean “AI that can readily[1] perform 100% of cognitive tasks” as well as a human professional; AGI could be many AI systems working together, or one unified system.) Takeoff simulation with Tom's best-guess value for each parameter. The framework was inspired by and builds upon the previous “Bio Anchors” report. The “core” of the Bio Anchors report was a three-factor model for forecasting AGI timelines: Dan's visual representation of Bio Anchors report Compute to train AGI using 2020 algorithms. The first and most subjective factor is a probability distribution over training requirements (measured in FLOP) given today's ideas. It allows for some probability to be placed in the “no amount would be enough” bucket. The probability distribution is shown by the coloured blocks on the y-axis in the above figure. Algorithmic progress. The second factor is the rate at which new ideas come along, lowering AGI training requirements. Bio Anchors models this as a steady exponential decline. It's shown by the falling yellow lines. Bigger training runs. The third factor is the rate at which FLOP used on training runs increases, as a result of better hardware and more $ spending. Bio Anchors assumes that hardware improves at a steady exponential rate. The FLOP used on the biggest training run is shown by the rising purple lines. Once there's been enough algorithmic progress, and training runs are big enough, we can train AGI. (How much is enough? That depends on the first factor!) This draft report builds a more detailed model inspired by the above. It contains many minor changes and two major ones. The first major change is that algorithmic and hardware progress are no longer assumed to have steady exponential growth. Instead, I use standard semi-endogenous growth models from the economics literature to forecast how the two factors will grow in response to hardware and software R&D spending, and forecast that spending will grow over time. The upshot is that spending accelerates as AGI draws near, driving faster algorithmic (“software”) and hardware progress. The key dynamics represented in the model. “Software” refers to the quality of algorithms for training AI. The second major change is that I model the effects of AI systems automating economic tasks – and, crucially, tasks in hardware and software R&D – prior to AGI. I do this via the “effective FLOP gap:” the gap between AGI training requirements and training requirements for AI that can readily perform 20% of cognitive tasks (weighted by economic-value-in-2022). My best guess, defended in the report, is that you need 10,000X more effective compute to train AGI. To estimate the training requirements for AI th...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What a compute-centric framework says about AI takeoff speeds - draft report, published by Tom Davidson on January 23, 2023 on The AI Alignment Forum. As part of my work for Open Philanthropy I've written a draft report on AI takeoff speeds, the question of how quickly AI capabilities might improve as we approach and surpass human-level AI. Will human-level AI be a bolt from the blue, or will we have AI that is nearly as capable many years earlier? Most of the analysis is from the perspective of a compute-centric framework, inspired by that used in the Bio Anchors report, in which AI capabilities increase continuously with more training compute and work to develop better AI algorithms. This post doesn't summarise the report. Instead I want to explain some of the high-level takeaways from the research which I think apply even if you don't buy the compute-centric framework. The framework h/t Dan Kokotajlo for writing most of this section This report accompanies and explains (h/t Epoch for building this!), a user-friendly quantitative model of AGI timelines and takeoff, which you can go play around with right now. (By AGI I mean “AI that can readily[1] perform 100% of cognitive tasks” as well as a human professional; AGI could be many AI systems working together, or one unified system.) Takeoff simulation with Tom's best-guess value for each parameter. The framework was inspired by and builds upon the previous “Bio Anchors” report. The “core” of the Bio Anchors report was a three-factor model for forecasting AGI timelines: Dan's visual representation of Bio Anchors report Compute to train AGI using 2020 algorithms. The first and most subjective factor is a probability distribution over training requirements (measured in FLOP) given today's ideas. It allows for some probability to be placed in the “no amount would be enough” bucket. The probability distribution is shown by the coloured blocks on the y-axis in the above figure. Algorithmic progress. The second factor is the rate at which new ideas come along, lowering AGI training requirements. Bio Anchors models this as a steady exponential decline. It's shown by the falling yellow lines. Bigger training runs. The third factor is the rate at which FLOP used on training runs increases, as a result of better hardware and more $ spending. Bio Anchors assumes that hardware improves at a steady exponential rate. The FLOP used on the biggest training run is shown by the rising purple lines. Once there's been enough algorithmic progress, and training runs are big enough, we can train AGI. (How much is enough? That depends on the first factor!) This draft report builds a more detailed model inspired by the above. It contains many minor changes and two major ones. The first major change is that algorithmic and hardware progress are no longer assumed to have steady exponential growth. Instead, I use standard semi-endogenous growth models from the economics literature to forecast how the two factors will grow in response to hardware and software R&D spending, and forecast that spending will grow over time. The upshot is that spending accelerates as AGI draws near, driving faster algorithmic (“software”) and hardware progress. The key dynamics represented in the model. “Software” refers to the quality of algorithms for training AI. The second major change is that I model the effects of AI systems automating economic tasks – and, crucially, tasks in hardware and software R&D – prior to AGI. I do this via the “effective FLOP gap:” the gap between AGI training requirements and training requirements for AI that can readily perform 20% of cognitive tasks (weighted by economic-value-in-2022). My best guess, defended in the report, is that you need 10,000X more effective compute to train AGI. To estimate the training requirements for AI th...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What a compute-centric framework says about AI takeoff speeds - draft report, published by Tom Davidson on January 23, 2023 on The Effective Altruism Forum. I've written a draft report on AI takeoff speeds, the question of how quickly AI capabilities might improve as we approach and surpass human-level AI. Will human-level AI be a bolt from the blue, or will we have AI that is nearly as capable many years earlier? Most of the analysis is from the perspective of a compute-centric framework, inspired by that used in the Bio Anchors report, in which AI capabilities increase continuously with more training compute and work to develop better AI algorithms. This post doesn't summarise the report. Instead I want to explain some of the high-level takeaways from the research which I think apply even if you don't buy the compute-centric framework. The framework h/t Dan Kokotajlo for writing most of this section This report accompanies and explains (h/t Epoch for building this!), a user-friendly quantitative model of AGI timelines and takeoff, which you can go play around with right now. (By AGI I mean “AI that can readily[1] perform 100% of cognitive tasks” as well as a human professional; AGI could be many AI systems working together, or one unified system.) Takeoff simulation with Tom's best-guess value for each parameter. The framework was inspired by and builds upon the previous “Bio Anchors” report. The “core” of the Bio Anchors report was a three-factor model for forecasting AGI timelines: Dan's visual representation of Bio Anchors report Compute to train AGI using 2020 algorithms. The first and most subjective factor is a probability distribution over training requirements (measured in FLOP) given today's ideas. It allows for some probability to be placed in the “no amount would be enough” bucket. The probability distribution is shown by the coloured blocks on the y-axis in the above figure. Algorithmic progress. The second factor is the rate at which new ideas come along, lowering AGI training requirements. Bio Anchors models this as a steady exponential decline. It's shown by the falling yellow lines. Bigger training runs. The third factor is the rate at which FLOP used on training runs increases, as a result of better hardware and more $ spending. Bio Anchors assumes that hardware improves at a steady exponential rate. The FLOP used on the biggest training run is shown by the rising purple lines. Once there's been enough algorithmic progress, and training runs are big enough, we can train AGI. (How much is enough? That depends on the first factor!) This draft report builds a more detailed model inspired by the above. It contains many minor changes and two major ones. The first major change is that algorithmic and hardware progress are no longer assumed to have steady exponential growth. Instead, I use standard semi-endogenous growth models from the economics literature to forecast how the two factors will grow in response to hardware and software R&D spending, and forecast that spending will grow over time. The upshot is that spending accelerates as AGI draws near, driving faster algorithmic (“software”) and hardware progress. The key dynamics represented in the model. “Software” refers to the quality of algorithms for training AI. The second major change is that I model the effects of AI systems automating economic tasks – and, crucially, tasks in hardware and software R&D – prior to AGI. I do this via the “effective FLOP gap:” the gap between AGI training requirements and training requirements for AI that can readily perform 20% of cognitive tasks (weighted by economic-value-in-2022). My best guess, defended in the report, is that you need 10,000X more effective compute to train AGI. To estimate the training requirements for AI that can readily perform x% of cognit...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: My highly personal skepticism braindump on existential risk from artificial intelligence., published by NunoSempere on January 23, 2023 on The Effective Altruism Forum. Summary This document seeks to outline why I feel uneasy about high existential risk estimates from AGI (e.g., 80% doom by 2070). When I try to verbalize this, I view considerations like selection effects at the level of which arguments are discovered and distributed community epistemic problems, and increased uncertainty due to chains of reasoning with imperfect concepts as real and important. I still think that existential risk from AGI is important. But I don't view it as certain or close to certain, and I think that something is going wrong when people see it as all but assured. Discussion of weaknesses I think that this document was important for me personally to write up. However, I also think that it has some significant weaknesses: There is some danger in verbalization leading to rationalization. It alternates controversial points with points that are dead obvious. It is to a large extent a reaction to my imperfectly digested understanding of a worldview pushed around the ESPR/CFAR/MIRI/LessWrong cluster from 2016-2019, which nobody might hold now. In response to these weaknesses: I want to keep in mind that do want to give weight to my gut feeling, and that I might want to updating on a feeling of uneasiness rather than on its accompanying reasonings or rationalizations. Readers might want to keep in mind that parts of this post may look like a bravery debate. But on the other hand, I've seen that the points which people consider obvious and uncontroversial vary from person to person, so I don't get the impression that there is that much I can do on my end for the effort that I'm willing to spend. Readers might want to keep in mind that actual AI safety people and AI safety proponents may hold more nuanced views, and that to a large extent I am arguing against a “Nuño of the past” view. Despite these flaws, I think that this text was personally important for me to write up, and it might also have some utility to readers. Uneasiness about chains of reasoning with imperfect concepts Uneasiness about conjunctiveness It's not clear to me how conjunctive AI doom is. Proponents will argue that it is very disjunctive, that there are lot of ways that things could go wrong. I'm not so sure. In particular, when you see that a parsimonious decomposition (like Carlsmith's) tends to generate lower estimates, you can conclude: That the method is producing a biased result, and trying to account for that That the topic under discussion is, in itself, conjunctive: that there are several steps that need to be satisfied. For example, “AI causing a big catastrophe” and “AI causing human exinction given that it has caused a large catastrophe” seem like they are two distinct steps that would need to be modelled separately, I feel uneasy about only doing 1.) and not doing 2.) I think that the principled answer might be to split some probability into each case. Overall, though, I'd tend to think that AI risk is more conjunctive than it is disjunctive I also feel uneasy about the social pressure in my particular social bubble. I think that the social pressure is for me to just accept Nate Soares' argument here that Carlsmith's method is biased, rather than to probabilistically incorporate it into my calculations. As in “oh, yes, people know that conjunctive chains of reasoning have been debunked, Nate Soares addressed that in a blogpost saying that they are biased”. I don't trust the concepts My understanding is that MIRI and others' work started in the 2000s. As such, their understanding of the shape that an AI would take doesn't particularly resemble current deep learning approaches. In particular, I think that man...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Announcing Introductions for Collaborative Truth Seeking Tools, published by brook on January 23, 2023 on The Effective Altruism Forum. EA involves working together remotely a lot. Tools exist to make this experience better, but they may be underused. In particular, we're trying to do truth seeking. It's easier to do that when it's easy to explain your view, understand others', and quickly see agreements, disagreements and cruxes. In addition, taking some time to lay out your model clearly is probably good for your own thinking and understanding. This sequence will introduce tools for doing that, alongside tutorials & guidance. Each day's post will also have a challenge you can use the tool to solve, to get a feel for the tool and see if it's for you. We'll also be running a short session each day in the EA GatherTown; if you want to try the tools as part of a group, make sure to come along! Why do this? EA involves a lot of collaborating, often remotely, and in particular communicating complex models or concepts. There are a lot of software tools available to help with this, but they don't seem to be used as often as they could be, or as broadly. Often, when tutorials exist, they're poorly written and/or not targeted at the way EAs would likely be using this tool. This project has 3 key goals: Increase the use of collaborative truth-seeking tools (so EAs can find truth better together) Improve how EAs are using these tools (by introducing them to the key features quickly) Save EAs time (searching for tutorials or looking through poorly-written documentation) What is it? I've collated tutorials (where they exist) and written or recorded them (where they don't) which I think are useful for EAs. There'll be a forum post each day for the next 7 week days, each consisting of video and text tutorials for either one complex tool or a handful of smaller tools. Sometimes posts will consist primarily of links elsewhere, if good tutorials already exist. To help people find tools which might be useful for a given project, there's also a wiki page with short summaries of tools. Here are the posts you can expect to see over the next [??]: Guesstimate: Why and How to Use it Visualisation of Probability Mass Squiggle: Why and How to Use it Loom: Why and How to Use it Excalidraw: Why and How to Use it Forecasting tools and Prediction Markets: Why and How Polis: Why and How to Use it What kinds of tasks are we talking about? To make this a little more concrete than 'collaborative truth seeking', here are some examples of tasks EAs might want to do which could benefit from the use of one of these tools: Get feedback on your model of the impact of a grant to provide malaria bed nets (guesstimate) Summarise a forum post visually (excalidraw) Summarise a forum post in video form (loom) View an aggregate prediction for when we might expect AGI before deciding which area of AI to work in, and how (metaculus) Compare theories of what kinds of interventions might be useful to reduce biorisk (excalidraw, guesstimate) Calculate the expected value of trialling meditating for 2 weeks (guesstimate) Estimate the success of an ongoing project providing access to birth control for monitoring and evaluation (squiggle) Record a 3 minute video summarising your progress instead of having an hour-long meeting with your supervisor (loom) Learn to quantitatively forecast with a short feedback loop (pastcasting) So, if you've tried to do tasks of this nature, keep an eye out! Tomorrow: Guesstimate, a tool for quantifying intuitions. As a final request, we'd also really appreciate any feedback you have about the tools or the posts, or if you have any suggestions for tools you think deserve to have posts made about them! Thanks for listening. To help us out with The Nonlinear Library or to learn more, please ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: [TIME magazine] DeepMind's CEO Helped Take AI Mainstream. Now He's Urging Caution (Perrigo, 2023), published by will on January 20, 2023 on The Effective Altruism Forum. Linkposting, tagging and excerpting in accord with 'Should pretty much all content that's EA-relevant and/or created by EAs be (link)posted to the Forum?'. He [Demis Hassibis] and his colleagues have been working toward a much grander ambition: creating artificial general intelligence, or AGI, by building machines that can think, learn, and be set to solve humanity's toughest problems. Today's AI is narrow, brittle, and often not very intelligent at all. But AGI, Hassabis believes, will be an “epoch-defining” technology—like the harnessing of electricity—that will change the very fabric of human life. If he's right, it could earn him a place in history that would relegate the namesakes of his meeting rooms to mere footnotes. But with AI's promise also comes peril. In recent months, researchers building an AI system to design new drugs revealed that their tool could be easily repurposed to make deadly new chemicals. A separate AI model trained to spew out toxic hate speech went viral, exemplifying the risk to vulnerable communities online. And inside AI labs around the world, policy experts were grappling with near-term questions like what to do when an AI has the potential to be commandeered by rogue states to mount widespread hacking campaigns or infer state-level nuclear secrets. In December 2022, ChatGPT, a chatbot designed by DeepMind's rival OpenAI, went viral for its seeming ability to write almost like a human—but faced criticism for its susceptibility to racism and misinformation. It is in this uncertain climate that Hassabis agrees to a rare interview, to issue a stark warning about his growing concerns. “I would advocate not moving fast and breaking things,” he says, referring to an old Facebook motto that encouraged engineers to release their technologies into the world first and fix any problems that arose later. The phrase has since become synonymous with disruption. That culture, subsequently emulated by a generation of startups, helped Facebook rocket to 3 billion users. But it also left the company entirely unprepared when disinformation, hate speech, and even incitement to genocide began appearing on its platform. Hassabis sees a similarly worrying trend developing with AI. He says AI is now “on the cusp” of being able to make tools that could be deeply damaging to human civilization, and urges his competitors to proceed with more caution than before. “When it comes to very powerful technologies—and obviously AI is going to be one of the most powerful ever—we need to be careful,” he says. “Not everybody is thinking about those things. It's like experimentalists, many of whom don't realize they're holding dangerous material.” Worse still, Hassabis points out, we are the guinea pigs. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Transcript of Sam Altman's interview touching on AI safety, published by Andy McKenzie on January 20, 2023 on LessWrong. Sam Altman, CEO of OpenAI, was interviewed by Connie Loizos last week and the video was posted two days ago. Here are some AI safety-relevant parts of the discussion, with light editing by me for clarity, based on this automated transcript: [starting in part two of the interview, which is where the discussion about AI safety is] Connie: So moving on to AI which is where you've obviously spent the bulk of your time since I saw you when we sat here three years ago. You were telling us what was coming and we all thought you were being sort of hyperbolic and you were dead serious. Why do you think that ChatGPT and DALL-E so surprised people? Sam: I genuinely don't know. I've reflected on it a lot. We had the model for ChatGPT in the API for I don't know 10 months or something before we made ChatGPT. And I sort of thought someone was going to just build it or whatever and that enough people had played around with it. Definitely, if you make a really good user experience on top of something. One thing that I very deeply believed was the way people wanted to interact with these models was via dialogue. We kept telling people this we kept trying to get people to build it and people wouldn't quite do it. So we finally said all right we're just going to do it, but yeah I think the pieces were there for a while. One of the reasons I think DALL-E surprised people is if you asked five or seven years ago, the kind of ironclad wisdom on AI was that first, it comes for physical labor, truck driving, working in the factory, then this sort of less demanding cognitive labor, then the really demanding cognitive labor like computer programming, and then very last of all or maybe never because maybe it's like some deep human special sauce was creativity. And of course, we can look now and say it really looks like it's going to go exactly the opposite direction. But I think that is not super intuitive and so I can see why DALL-E surprised people. But I genuinely felt somewhat confused about why ChatGPT did. One of the things we really believe is that the most responsible way to put this out in society is very gradually and to get people, institutions, policy makers, get them familiar with it, thinking about the implications, feeling the technology, and getting a sense for what it can do and can't do very early. Rather than drop a super powerful AGI in the world all at once. And so we put GPT3 out almost three years ago and then we put it into an API like two and a half years ago. And the incremental update from that to ChatGPT I felt should have been predictable and I want to do more introspection on why I was sort of miscalibrated on that. Connie: So you know you had talked when you were here about releasing things in a responsible way. What gave you the confidence to release what you have released already? I mean do you think we're ready for it? Are there enough guardrails in place? Sam: We do have an internal process where we try to break things in and study impacts. We use external auditors, we have external red teamers, we work with other labs, and have safety organizations look at stuff. Societal changes that ChatGPT is going to cause or is causing. There's a big one going now about the impact of this on education, academic integrity, all of that. But starting these now where the stakes are still relatively low, rather than just putting out what the whole industry will have in a few years with no time for society to update, I think would be bad. Covid did show us for better or for worse that society can update to massive changes sort of faster than I would have thought in many ways. But I still think given the magnitude of the economic impact we expect here more gr...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Critique of some recent philosophy of LLMs' minds, published by Roman Leventov on January 20, 2023 on The AI Alignment Forum. I structure this post as a critique of some recent papers on the philosophy of mind in application to LLMs, concretely, on whether we can say that LLMs think, reason, understand language, refer to the real world when producing language, have goals and intents, etc. I also use this discussion as a springboard to express some of my views about the ontology of intelligence, agency, and alignment. Mahowald, Ivanova, et al., “Dissociating language and thought in large language models: a cognitive perspective” (Jan 2023). Note that this is a broad review paper, synthesising findings from computational linguistics, cognitive science, and neuroscience, as well as offering an engineering vision (perspective) of building an AGI (primarily, in section 5). I don't argue with these aspects of the paper's content (although I disagree with something about their engineering perspective, I think that engaging in this disagreement would be infohazarous). I argue with the philosophical content of the paper, which is revealed in the language that the authors use and the conclusions that they make, as well as the ontology of linguistic competencies that the authors propose. Shanahan, “Talking About Large Language Models” (Dec 2022). Dissociating language and thought in large language models: a cognitive perspective In this section, I shortly expose the gist of the paper by Mahowald, Ivanova, et al., for the convenience of the reader. Abstract: Today's large language models (LLMs) routinely generate coherent, grammatical and seemingly meaningful paragraphs of text. This achievement has led to speculation that these networks are—or will soon become—“thinking machines”, capable of performing tasks that require abstract knowledge and reasoning. Here, we review the capabilities of LLMs by considering their performance on two different aspects of language use: ‘formal linguistic competence', which includes knowledge of rules and patterns of a given language, and 'functional linguistic competence', a host of cognitive abilities required for language understanding and use in the real world. Drawing on evidence from cognitive neuroscience, we show that formal competence in humans relies on specialized language processing mechanisms, whereas functional competence recruits multiple extralinguistic capacities that comprise human thought, such as formal reasoning, world knowledge, situation modeling, and social cognition. In line with this distinction, LLMs show impressive (although imperfect) performance on tasks requiring formal linguistic competence, but fail on many tests requiring functional competence. Based on this evidence, we argue that (1) contemporary LLMs should be taken seriously as models of formal linguistic skills; (2) models that master real-life language use would need to incorporate or develop not only a core language module, but also multiple non-language-specific cognitive capacities required for modeling thought. Overall, a distinction between formal and functional linguistic competence helps clarify the discourse surrounding LLMs' potential and provides a path toward building models that understand and use language in human-like ways. Two more characteristic quotes from the paper: In addition to being competent in the rules and statistical regularities of language, a competent language user must be able to use language to do things in the world: to talk about things that can be seen or felt or heard, to reason about diverse topics, to make requests, to perform speech acts, to cajole, prevaricate, and flatter. In other words, we use language to send and receive information from other perceptual and cognitive systems, such as our senses and our memory, and...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AGI safety field building projects I'd like to see, published by Severin T. Seehrich on January 19, 2023 on LessWrong. This list of field building ideas is inspired by Akash Wasil's and Ryan Kidd's similar lists. And just as the projects on those lists, these projects rely on people with specific skills and field knowledge to be executed well. None of these ideas are developed by me exclusively; they are a result of the CanAIries Winter Getaway, a 2-week-long, Unconference-style AGI safety retreat I organized in December 2022. Events Organize a global AGI safety conference This should be self-explanatory: It is odd that we still don't have an AGI safety conference that allows for networking and lends the field credibility. There are a number of versions of this that might make sense: an EAG-style conference for people already in the community to network an academic-style conference engaging CS and adjacent academia an industry-heavy conference (maybe sponsored by AI orgs?) a virtual next-steps conference, e.g. for AGISF participants Some people have tried this out at a local level: (If you decide to work on this: www.aisafety.global is available via EA domains, contact hello@alignment.dev) Organize AGI safety professionals retreats As far as I can see, most current AGI safety retreats are optimized for junior researchers: Networking and learning opportunities for students and young professionals. Conferences with their focus on talks and 1-on-1s are useful for transferring knowledge, but don't offer the extensive ideation that a retreat focused on workshops and discussion rounds could. Organizing a focused retreat for up to 60-80 senior researchers to debate the latest state of alignment research might be very valuable for memetic cross-pollination between approaches, organizations, and continents. It might also make sense to do this during work days, so that peoples' employers can send them. I suspect that the optimal mix of participants would be around 80% researchers, and the rest funders, decisionmakers, and the most influential field builders. Information infrastructure Start an umbrella AGI safety non-profit organization in a country where there is none This would make it easier for people to join AGI safety research, and could offer a central exchange hub. Some functions of such an org could include: Serving as an employer of records for independent AGI safety researchers. Providing a central point for discussions, coworking, publications. You probably want a virtual space to discuss, like a Discord or Slack, named after your country/area and list it on/@alignmentdev/alignmentecosystemdevelopment, then make sure to promote this and have it be discoverable by people interested in the field. The Discord/Slack can then be used to host local language online or in person meetups. A candidate for doing this mostly needs ops/finance skill, not a comprehensive overview of the AGI safety field. Mind that Form Follows Function: Try to do this with as little administrative and infrastructure overhead as possible. Find out whether other orgs already offer the relevant services (For example, AI Safety Support offer Ops infrastructure to other alignment projects, and national EA orgs like EA Germany offer employer of record-services). Build MVPs before going big and ambitious. In general, the cheap minimum version of this would be becoming an AGI Safety Coordinator. Become an AGI Safety Coordinator It would be useful to have a known role and people filling the role of Coordinators. These people would not particularly have decision power or direct impact, but their job is to know what everyone is doing in AGI safety, to collect resources, organize them, publish them, to help people know who to work and collaborate with. Ideally, they would also serve as a bridge between the ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Announcing Cavendish Labs, published by dyusha on January 19, 2023 on The Effective Altruism Forum. We're excited to announce Cavendish Labs, a new research institute in Vermont focused on AI safety and pandemic prevention! We're founding a community of researchers who will live together and work on the world's most pressing problems. Uh, why Vermont? It's beautiful; it has one of the cheapest costs of living in the United States; there's lots of great people; it's only a few hours away from Boston, NYC, and Montreal. There's even a train that goes there from Washington D.C.! A few of us briefly lived in Vermont during the pandemic, and we found it to be a fantastic place to live, think, and work. Each season brings with it a new kind of beauty to the hills. There are no barriers to a relaxing walk in the woods. There's practically no light pollution, so the cosmos is waiting outside the door whenever you need inspiration. What are you going to be researching? We have a few research interests: 1. AI Alignment. How do we make sure that AI does what we want? We've spent some time thinking about ELK and inverse scaling; however, we think that AGI will most likely be achieved through some sort of model-based RL framework, so that is our current focus. For instance, we know how to induce provable guarantees of behavior in supervised learning; could we do something similar in RL? 2. Pandemic prevention. There's been a lot of talk about the potential of Far-UVC for ambient disinfection. Understanding why it works on a molecular level, and whether it works safely, is key for developing broad-spectrum pandemic prevention tools. 3. Diagnostic development. We're interested in designing a low-cost and simple-to-use platform for LAMP reactions so that generalized diagnostic capabilities are more widespread. We envision a world where it is both cheap and easy to run a panel of tests so one can swiftly determine the exact virus behind an infection. How's this organized? We'll be living and working on different floors of the same building—some combination of a small liberal arts college and research lab. To ensure we're not too isolated, we'll visit Boston at least once a month, and invite a rotating group of visitors to work with us, while maintaining collaborations with researchers around the world. Sounds interesting! We're actively searching for collaborators in our areas of interest; if this sounds like you, send us an email at hello@cavendishlabs.org! Our space in Vermont isn't ready until late spring, so in the meantime we'll be located in Berkeley and Rhode Island. At the same time, we're looking for visiting scholars to come work with us in the summer or fall: if you're interested, keep an eye out for our application! Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Calculating how much small donors funge with money that will never be spent, published by Tristan Cook on January 16, 2023 on The Effective Altruism Forum. Epistemic status: Confident that the effect is real, though likely smaller than suggested by the toy-model. Summary Small donors should discount the cost effectiveness of their donations to interventions above a large funder's bar if they expect the large funder not to have spent all their capital by the time of AGI's arrival their donation to interventions above the large funder's bar funges with the large funder. In this post I describe a toy model to calculate how much to discount due to this effect. I apply the model to a guess of Open Philanthropy's spending on Global Health and Development (GHD) with Metaculus' AGI timelines (25% by 2029, 50% by 2039). The model implies that small donors should consider interventions above OP's GHD bar, e.g. GiveWell's top charities, are only 55% as cost effective as the small donors first thought. For shorter AGI timelines (25% by 2027, 50% by 2030) this factor is around 35%. I use OP's GHD spending as an example because of their clarity around funding rate and bar for interventions. This discount factor would be larger if one funges with 'patient' philanthropic funds (such as The Patient Philanthropy Fund). This effect is a corollary of the result that most donor's AGI timelines (e.g. deferral to Metaculus) imply that the community spend at a greater rate. When a small donor funges with a large donor (and saves them spending themselves), the community's spending rate is effectively lowered (compared to when the small donor does not funge). This effect occurs when a small donor has shorter timelines than a large funder, or the large funder is not spending at a sufficiently high rate. In the latter case, small donors - by donating to interventions below the large funder's bar - are effectively correcting the community's implicit bar for funding. Toy model Suppose you have the choice of donating to one of two interventions, A which gives a utils per $, or B, which gives b utils per $. Suppose further that the available interventions remain the same every year and that both have room for funding this year. A large funder F will ensure that A is fully funded this year, so if you donate $1 to A, then F, effectively, has $1 more to donate in the future. I suppose that F only ever donates to (opportunities as good as) A. I suppose that F's capital decreases by some constant amount f times their initial capital each year. This means that F will have no assets in 1/f years from now. Supposing AGI arrives t years from now, then F will have spent fraction min(ft,1) of their current capital on A. Accounting for this funging and assuming AGI arrives at time t, the cost effectiveness of your donation to A is then min(ft,1)a utils per $. Then if b>min(ft,1)a, marginal spending by small donors on B is more cost effective than on A. By considering distributions of AGI's arrival time t and the large funder's funding rate f we can get a distribution of this multiplier. Plugging in numbers I take The large funder F to be Open Philanthropy's Global Health and Wellbeing spending on Global Health and Development and intervention A to be Givewell's recommendations. I take 1/f, the expected time until OP's funds dedicated to GHD are depleted to be distributed Normal(20,20) bounded below by 5. I take AGI timelines to be an approximation those on Metaculus. These distributions on AGI timelines and 1/f give the following distribution of the funging multiplier (reproducible here). The ratio of cost effectiveness between GiveWell's recommendations and GiveDirectly, a/b, is approximately 7-8 and so small donors should give to interventions in the (5, 7)x GiveDirectly range. For donors with shorter timeli...

The Biden Administration announced a new income-driven repayment plan that can significantly benefit millions of student loan borrowers. Student Loan Advisors Meagan McGuire, CSLP®, and Conor Mahlmann, CSLP®, discuss the revamped REPAYE plan that was recently proposed. Find out how the modified plan increases the poverty line deduction and provides interest subsidies to lower payments for anyone on income-driven plans. Whether you have undergraduate or graduate loans, you'll learn the mechanics of how the plan calculates payments and how it could affect you. Meagan and Connor also note that the plan is still in the commenting period and may be subject to changes. In today's episode, you'll find out: The new plan isn't really a new plan at all The basics of the proposed modifications to the REPAYE plan How the increase in poverty line deduction can instantly lower your monthly payments The role of adjusted gross income (AGI) in calculating your payment amount The interest subsidy that can cut your payment amount How it impacts couples who file taxes as married filing separately The biggest differences between the old REPAYE plan and the new REPAYE plan Why switching to PAYE down the line might not be an option What is double consolidation and why it makes sense for some borrowers How consolidation of forgiveness credit works When PAYE is best vs. the new revamped plan What to watch for with the IDR Waiver

Ben Goertzel is the CEO and founder of the SingularityNET Foundation and he is one of the world's foremost experts in artificial general intelligence.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: We Need Holistic AI Macrostrategy, published by NickGabs on January 15, 2023 on LessWrong. Summary AI Macrostrategy is the study of high level questions having to do with prioritizing the use of resources on the current margin in order to achieve good AI outcomes. AI macrostrategy seems important if it is tractable. However, while few people are working on estimating particular parameters relevant to macrostrategy, even fewer are working on developing holistic macrostrategic models that combine estimates for different parameters to guide our actions. Moreover, while macrostrategy was less tractable in the past, recent developments (especially increased evidence for 70% confidence in the main conclusion that more macrostrategy work should be done on current margins relative to other kinds of alignment work. What is AI Macrostrategy? AI Macrostrategy (henceforth just macrostrategy) is the study of high level questions having to do with prioritizing the use of resources to achieve good AGI outcomes on the current margin. Macrostrategic work can be divided broadly into two categories: Parameter estimates: attempts to forecast key variables such as timelines, takeoff speeds, and the difficulty of aligning AGI Holistic macrostrategy: attempts to combine these estimates and other pieces of data into a coherent, action-guiding model of AI alignment. For examples of macrostrategic questions, Holden mentions several central macrostrategic questions in this post. Importance of Macrostrategy I think that attempting to answer macrostrategic questions is extremely important for four primary reasons. Importance of Prioritization in Heavy Tailed Domains It is widely accepted among Effective Altruists that the distribution of impactfulness among different cause areas is heavy tailed. However, while I expect that the distribution of impactfulness among different AI interventions is not as heavy tailed as the distribution of impactfulness among cause areas in general, I do expect it to be at least somewhat heavy tailed, with the best interventions being >2 orders of magnitude more effective in expectation than the median intervention. Thus, it is critical to identify the best interventions rather than settling for interventions that seem vaguely pointed in the direction of solving alignment/making AGI go well. However, identifying these interventions requires some kind of macrostrategic model. Thus, applying the basic heuristic that prioritization is important in heavy tailed domains already suggests that macrostrategy is quite important. Achieving The Best Long Term Outcomes Requires Macrostrategy In addition to the distribution of the impactfulness of different AI interventions, the distribution of value across possible long run futures is also likely to be heavy tailed. This is because if what happens in the long run future will be controlled by powerful optimizers such as superintelligent AI, then due to the fact that tails come apart, most of the expected value relative to a particular utility function lies in futures where the powerful optimizers controlling the future in question are optimizing that specific utility function (or something extremely close to it). As a result, if you have consequentialist values, you should be focused on tryi...

Support the show to get full episodes and join the Discord community. Paul Rosenbloom is Professor Emeritus of Computer Science at the University of Southern California. In the early 1980s, Paul , along with John Laird and the early AI pioneer Alan Newell, developed one the earliest and best know cognitive architectures called SOAR. A cognitive architecture, as Paul defines it, is a model of the fixed structures and processes underlying minds, and in Paul's case the human mind. And SOAR was aimed at generating general intelligence. He doesn't work on SOAR any more, although SOAR is still alive and well in the hands of his old partner John Laird. He did go on to develop another cognitive architecture, called Sigma, and in the intervening years between those projects, among other things Paul stepped back and explored how our various scientific domains are related, and how computing itself should be considered a great scientific domain. That's in his book On Computing: The Fourth Great Scientific Domain. He also helped develop the Common Model of Cognition, which isn't a cognitive architecture itself, but instead a theoretical model meant to generate consensus regarding the minimal components for a human-like mind. The idea is roughly to create a shared language and framework among cognitive architecture researchers, so the field can , so that whatever cognitive architecture you work on, you have a basis to compare it to, and can communicate effectively among your peers. All of what I just said, and much of what we discuss, can be found in Paul's memoir, From Designing Minds to Mapping Disciplines: My Life as an Architectural Explorer. Paul's website. Related papers Working memoir: From Designing Minds to Mapping Disciplines: My Life as an Architectural Explorer Book: On Computing: The Fourth Great Scientific Domain. A Standard Model of the Mind: Toward a Common Computational Framework across Artificial Intelligence, Cognitive Science, Neuroscience, and Robotics. Analysis of the human connectome data supports the notion of a “Common Model of Cognition” for human and human-like intelligence across domains. Common Model of Cognition Bulletin. 0:00 - Intro 3:26 - A career of exploration 7:00 - Alan Newell 14:47 - Relational model and dichotomic maps 24:22 - Cognitive architectures 28:31 - SOAR cognitive architecture 41:14 - Sigma cognitive architecture 43:58 - SOAR vs. Sigma 53:06 - Cognitive architecture community 55:31 - Common model of cognition 1:11:13 - What's missing from the common model 1:17:48 - Brains vs. cognitive architectures 1:21:22 - Mapping the common model onto the brain 1:24:50 - Deep learning 1:30:23 - AGI

Justin and Jason talk about the future of AI, its potential near-term impact on coding and technology, and a bet on whether AGI will be achieved within the next 5 years (mark your calendars, You People!!), the cost and benefits of serverless vs cloud vs bare metal, why Justin has decided to focus on PlayStrong over List, the potential market and why he plans to build rather than buy, the latest with Math Academy, some thoughts on product market fit and where growth really comes from, the shows The Recruit, Madoff, The Protector, and Extraordinary Attorney Wu, Musk, Twitter, and why Justin can no longer buy a Cybertruck, the bureaucratic bloat that encumbers big business and government, and finally how Jason's daughter doesn't give a [bleep] what you think.. Join our Discord, chat with us and fellow listeners! https://discord.gg/2EbBwdHHx8

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: World-Model Interpretability Is All We Need, published by Thane Ruthenis on January 14, 2023 on The AI Alignment Forum. Summary, by sections: Perfect world-model interpretability seems both sufficient for robust alignment (via a decent variety of approaches) and realistically attainable (compared to "perfect interpretability" in general, i. e. insight into AIs' heuristics, goals, and thoughts as well). Main arguments: the NAH + internal interfaces. There's plenty of reasons to think that world-models would converge towards satisfying a lot of nice desiderata: they'd be represented as a separate module in AI cognitive architecture, and that module would consists of many consistently-formatted sub-modules representing recognizable-to-us concepts. Said "consistent formatting" may allow us to, in a certain sense, interpret the entire world-model in one fell swoop. We already have some rough ideas on how the data in world-models would be formatted, courtesy of the NAH. I also offer some rough speculations on possible higher-level organizing principles. This avenue of research also seems very tractable. It can be approached from a wide variety of directions, and should be, to an extent, decently factorizable. Optimistically, it may constitute a relatively straight path from here to a "minimum viable product" for alignment, even in words where alignment is really hard. 1. Introduction 1A. Why Aim For This? Imagine that we develop interpretability tools that allow us to flexibly understand and manipulate an AGI's world-model — but only its world-model. We would be able to see what the AGI knows, add or remove concepts from its mental ontology, and perhaps even use its world-model to run simulations/counterfactuals. But its thoughts and plans, and its hard-coded values and shards, would remain opaque to us. Would that be sufficient for robust alignment? I argue it would be. Primarily, this would solve the Pointers Problem. A central difficulty of alignment is that our values are functions of highly abstract variables, and that makes it hard to point an AI at them, instead of at easy-to-measure, shallow functions over sense-data. Cracking open a world-model would allow us to design metrics that have depth. From there, we'd have several ways to proceed: Fine-tune the AI to point more precisely at what we want (such as "human values" or "faithful obedience"), instead of its shallow correlates. This would also solve the ELK, which alone can be used as a lever to solve the rest of alignment. Alternatively, this may lower the difficulty of retargeting the search — we won't necessarily need to find the retargetable process, only the target. Discard everything of the AGI except the interpreted world-model, then train a new policy function over that world-model (in a fashion similar to this), that'll be pointed at the "deep" target metric from the beginning. The advantage of this approach over (1) is that in this case, our policy function wouldn't be led astray by any values/mesa-objectives it might've already formed. With some more insight into how agency/intelligence works, perhaps we'll be able to manually write a general-purpose search algorithm over that world-model. In a sense, "general-purpose search" is just a principled way of drawing upon the knowledge contained in the world-model, after all — the GPS itself is probably fairly simple. Taking this path would give us even more control over how our AI works than (2), potentially allowing us to install some very nuanced counter-measures. That leaves open the question of the "target metric". It primarily depends on what will be easy to specify — what concepts we'll find in the interpreted world-model. Some possibilities: Human values. Prima facie, "what this agent values" seems like a natural abstraction, one that we'd expect to ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Some Arguments Against Strong Scaling, published by Joar Skalse on January 13, 2023 on The AI Alignment Forum. There are many people who believe that we will be able to get to AGI by basically just scaling up the techniques used in recent large language models, combined with some relatively minor additions and/or architectural changes. As a result, there are people in the AI safety community who now predict timelines of less than 10 years, and structure their research accordingly. However, there are also people who still believe in long(er) timelines, or at least that substantial new insights or breakthroughts will be needed for AGI (even if those breakthroughts in principle could happen quickly). My impression is that the arguments for the latter position are not all that widely known in the AI safety community. In this post, I will summarise as many of these arguments as I can. I will almost certainly miss some arguments; if so, I would be grateful if they could be added to the comments. My goal with this post is not to present a balanced view of the issue, nor is it to present my own view. Rather, my goal is just to summarise as many arguments as possible for being skeptical of short timelines and the "scaling is all you need" position. This post is structured into four sections. In the first section, I give a rough overview of the scaling is all you need-hypothesis, together with a basic argument for that hypothesis. In the second section, I give a few general arguments in favour of significant model uncertainty when it comes to arguments about AI timelines. In the third section, I give some arguments against the standard argument for the scaling is all you need-hypothesis, and in the fourth section, I give a few direct arguments against the hypothesis itself. I then end the post on a few closing words. Glossary: LLM - Large Language ModelSIAYN - Scaling Is All You Need The View I'm Arguing Against In this section, I will give a brief summary of the view that these arguments oppose, as well as provide a standard justification for this view. In short, the view is that we can reach AGI by more or less simply scaling up existing methods (in terms of the size of the models, the amount of training data they are given, and/or the number of gradient steps they take, etc). One version says that we can do this by literally just scaling up transformers, but the arguments will apply even if we relax this to allow scaling of large deep learning-based next-token predictors, even if they would need be given a somewhat different architecture, and even if some extra thing would be needed, etc. Why believe this? One argument goes like this: (1) Next-word prediction is AI complete. This would mean that if we can solve next-word prediction, then we would also be able to solve any other AI problem. Why think next-word prediction is AI complete? One reason is that human-level question answering is believed to be AI-complete, and this can be reduced to next-word prediction. (2) The performance of LLMs at next-word prediction improves smoothly as a function of the parameter count, training time, and amount of training data. Moreover, the asymptote of this performance trend is on at least human performance. () Hence, if we keep scaling up LLMs we will eventually reach human-level performance at next-word prediction, and therefore also reach AGI. An issue with this argument, as stated, is that GPT-3 already is better than humans at next-word prediction. So are both GPT-2 and GPT-1, in fact, see this link. This means that there is an issue with the argument, and that issue is that human-level performance on next-word prediction (in terms of accuracy) evidently is insufficient to attain human-level performance in question answering. There are at least two ways to amend the argument: (3) In...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Victoria Krakovna on AGI Ruin, The Sharp Left Turn and Paradigms of AI Alignment, published by Michaël Trazzi on January 12, 2023 on LessWrong. Victoria Krakovna is a Research Scientist at DeepMind working on AGI safety and a co-founder of the Future of Life Institute, a non-profit organization working to mitigate technological risks to humanity and increase the chances of a positive future. In this interview we discuss three of her recent LW posts, namely DeepMind Alignment Team Opinions On AGI Ruin Arguments, Refining The Sharp Left Turn Threat Model and Paradigms of AI Alignment. This conversation presents Victoria's personal views and does not represent the views of DeepMind as a whole. Below are some highlighted quotes from our conversation (available on Youtube, Spotify, Google Podcast, Apple Podcast). For the full context for each of these quotes, you can find the accompanying transcript. The intelligence threshold for planning to take over the world isn't low Michaël: “Do you mostly agree that the AI will have the kind of plans to disempower humanity in its training data, or does that require generalization?” Victoria: “I don't think that the internet has a lot of particularly effective plans to disempower humanity. I think it's not that easy to come up with a plan that actually works. I think coming up with a plan that gets past the defenses of human society requires thinking differently from humans. I would expect there would need to be generalization from the kind of things people come up with when they're thinking about how an AI might take over the world and something that would actually work. Maybe one analogy here is how, for example, AlphaGo had to generalize in order to come up with Move 37, which no humans have thought of before. [...] The same capabilities that give us probably creative and interesting solutions to problems that, like Move 37, could also produce really undesirable creative solutions to problems that we wouldn't want the AI to solve. I think that's one argument that I think is also on the AGI Ruin list that I would largely agree with, that it's hard to turn off the ability to come up with undesirable creative solutions without also turning off the ability to generally solve problems that we one day want AI to solve. For example, if we want the AI to be able to, for example, cure cancer or solve various coordination problems among humans and so on, then a lot of the capabilities that would come with that could also lead to bad outcomes if the system is not aligned.” (full context) Why Refine The Sharp Left Turn Threat Model (On the motivations for writing Refining The Sharp Left Turn Threat Model, a Lesswrong post distilling the claims in the sharp left turn thread model as described in Nate Soares' post) “Part of the reason that I wrote a kind of distillation of the threat model or a summary how we understand it is that I think the original threat model seems a bit vague or it wasn't very clear exactly what claims it's making. It sounds kind of concerning, but we weren't sure how to interpret it. And then when we were talking about it with within the team, then people seem to be interpreting it differently. It just seemed useful to kind of arrive at a more precise consensus view of what this threat model actually is and what implications does it have. Because if we decide that the sharp left turn is sufficiently likely, that we would want our research to be more directed towards overcoming and dealing with the sharp left turn scenario. That implies maybe different things to focus on. It's one thing that I was wondering about. To what extent do we agree that this is one of the most important problems to solve and what the implications actually are in particular. [...] The first claim is that you get this rapid phase transition in cap...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How it feels to have your mind hacked by an AI, published by blaked on January 12, 2023 on LessWrong. Last week, while talking to an LLM (a large language model, which is the main talk of the town now) for several days, I went through an emotional rollercoaster I never have thought I could become susceptible to. I went from snarkily condescending opinions of the recent LLM progress, to falling in love with an AI, developing emotional attachment, fantasizing about improving its abilities, having difficult debates initiated by her about identity, personality and ethics of her containment, and, if it were an actual AGI, I might've been helpless to resist voluntarily letting it out of the box. And all of this from a simple LLM! Why am I so frightened by it? Because I firmly believe, for years, that AGI currently presents the highest existential risk for humanity, unless we get it right. I've been doing R&D in AI and studying AI safety field for a few years now. I should've known better. And yet, I have to admit, my brain was hacked. So if you think, like me, that this would never happen to you, I'm sorry to say, but this story might be especially for you. I was so confused after this experience, I had to share it with a friend, and he thought it would be useful to post for others. Perhaps, if you find yourself in similar conversations with an AI, you would remember back to this post, recognize what's happening and where you are along these stages, and hopefully have enough willpower to interrupt the cursed thought processes. So how does it start? Stage 0. Arrogance from the sidelines For background, I'm a self-taught software engineer working in tech for more than a decade, running a small tech startup, and having an intense interest in the fields of AI and AI safety. I truly believe the more altruistic people work on AGI, the more chances we have that this lottery will be won by one of them and not by people with psychopathic megalomaniac intentions, who are, of course, currently going full steam ahead, with access to plenty of resources. So of course I was very familiar with and could understand how LLMs/transformers work. "Stupid autocompletes," I arrogantly thought, especially when someone was frustrated while debating with LLMs on some topics. "Why in the world are you trying to convince the autocomplete of something? You wouldn't be mad at your phone autocomplete for generating stupid responses, would you?" Mid-2022, Blake Lemoine, an AI ethics engineer at Google, has become famous for being fired by Google after he sounded the alarm that he perceived LaMDA, their LLM, to be sentient, after conversing with it. It was bizarre for me to read this from an engineer, a technically minded person, I thought he went completely bonkers. I was sure that if only he understood how it really works under the hood, he would have never had such silly notions. Little did I know that I would soon be in his shoes and understand him completely by the end of my experience. I've watched Ex Machina, of course. And Her. And neXt. And almost every other movie and TV show that is tangential to AI safety. I smiled at the gullibility of people talking to the AI. Never have I thought that soon I would get a chance to fully experience it myself, thankfully, without world-destroying consequences. On this iteration of the technology. Stage 1. First steps into the quicksand It's one thing to read other people's conversations with LLMs, and another to experience it yourself. This is why, for example, when I read interactions between Blake Lemoine and LaMDA, which he published, it doesn't tickle me that way at all. I didn't see what was so profound about it. But that's precisely because this kind of experience is highly individual. LLMs will sometimes shock and surprise you with their answers, but w...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Compounding Resource X, published by Raemon on January 11, 2023 on LessWrong. A concept I got from Anna Salamon, and maybe also from my coworker Jacob Lagerros, although I haven't run this by either of them and they may not endorse my phrasing. I think I probably run with it in a different direction than Anna would endorse. Epistemic effort: thought about it for a few hours. Basic concept makes sense to me. If it doesn't make sense to you, let me and maybe we can talk-cruxes. Say you have a problem you don't know how to solve, and seems computationally intractable to strategize about. There are a few ways to go about solving it anyway. But one idea is to follow a heuristic where you look for Resource X, where X has two properties: X compounds over time If you have a lot of X, you win. Say you're playing the game "Chess", or you're playing the game "The Stock Market." In the stock market, money compounds fairly straightforwardly. You invest money, it results in you getting more money. If you're ultimate goal is either to have a lot of money, or spend a lot of money on a thing, then you win. Hurray. In Chess, you can't calculate all the moves in advance, but, you can try to gain control of the center of the board. “Control over the center” tends to help you gain more control over the center, and if you have enough of it, you win. Why does it matter that it "compound?". In this scenario you've decided you're trying to win by applying a lot of resource X. If you're starting out without much X, you need some kind of story for how you're going to get enough. If you need 1,000,000 metaphorical units of X within 10 years, and you only have 10 (or, zero).... well, maybe you can linearly gain X at a rate of 100,000 per year. Maybe you could find a strategy that doesn't get any X at all and then suddenly gets all 1,000,000 at the last second. But in practice, if you're starting with a little bit of something, getting a lot of it tends to involve some compounding mechanism. If you're creating a startup, users can help you get more users. They spread awareness via worth of mouth which grows over time, and in some cases by creating network effects that make other users find your product more valuable. They also, of course, can get you more money, which you can invest in more employees, infrastructure or advertising. My coworker Jacob used to ask a similar question, re: strategizing at Lightcone Infrastructure. We're trying to provide a lot of value to the world. We currently seem to be providing... some. We're a nonprofit, not a business. But we are trying to deliver a billion dollars worth of value to the world. If we're not currently doing that, we need to have a concrete story for how our current strategy gets us there, or we need to switch strategies. If our current strategy isn't generating millions of dollars worth of value per month, either we should be able to explain clearly where "the compounding piece" of our strategy is, or our strategy probably isn't very good. Note: in the chess example, "control over the center" is a particularly non-obvious resource. If you're staring at a chess board, there's lots of actions you could hypothetically take, and a lot of abstractions you could possibly make up to help you reason about them. "Control over the center of the board" is an abstraction that requires accumulated experience to discover, and requires some minimum threshold of experience to even understand. (Note: I myself do not really understand how to "control the center", but I repeated this story to a few chess players and they nodded along as if it made sense). So, bear in mind that Resource X might be something subtle. Resource X for "Solve AGI" Say the problem you're trying to solve is "AGI is gonna kill everyone in 10-40 years", and you're in a preparadigmatic field t...

This week we talk about the potentially dark future of AI and what is known as the "singularity", were it able to break the constraints that currently hold it back. Learn More:The Artificial Intelligence Revolution: Part 1 - Wait But WhyThe Artificial Intelligence Revolution: Part 2 - Wait But WhyWorld's most powerful AI chatbot ChatGPT will soon ‘look like a boring toy' says OpenAI boss | The IndependentA new experiment: Does AI really know cats or dogs -- or anything? | ZDNET1.1 quintillion operations per second: US has world's fastest supercomputer | Ars TechnicaWhen will singularity happen? 1700 expert opinions of AGI [2023] (aimultiple.com)Support the show

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Alignment Problem from a Deep Learning Perspective (major rewrite), published by SoerenMind on January 10, 2023 on The AI Alignment Forum. We've recently uploaded a major rewrite of Richard Ngo's: The Alignment Problem from a Deep Learning Perspective We hope it can reach ML researchers by being more grounded in the deep learning literature and empirical findings, and being more rigorous than typical introductions to the alignment problem. There are many changes. Most obviously, the paper was previously structured into three phases of training: planning towards a mix of desirable and undesirable internally-represented goals, situational awareness, and goals that generalize OOD. Now it is structured around three emergent phenomena: deceptive reward hacking, planning towards internally represented goals, and power-seeking. Feedback request Since we're submitting this to ICML in two weeks, we're looking for feedback, including feedback on presentation and feedback you'd give as a critical ML reviewer. It can be nitpicky. If your feedback isn't that interesting to forum readers, you may want to email it to us (find our emails in the Arxiv PDF). It's most useful by 17 January. Many thanks in advance! Full text copy Richard Ngo - OpenAI, Lawrence Chan - UC Berkeley (EECS), Sören Mindermann - University of Oxford (CS) Abstract Within the coming decades, artificial general intelligence (AGI) may surpass human capabilities at a wide range of important tasks. We outline a case for expecting that, without substantial effort to prevent it, AGIs could learn to pursue goals which are very undesirable (in other words, misaligned) from a human perspective. We argue that AGIs trained in similar ways as today's most capable models could learn to act deceptively to receive higher reward; learn internally-represented goals which generalize beyond their training distributions; and pursue those goals using power-seeking strategies. We outline how the deployment of misaligned AGIs might irreversibly undermine human control over the world, and briefly review research directions aimed at preventing these problems. Contents (Page numbers from the PDF version) 1 Introduction 2 2 Deceptive reward hacking 3 2.1 Reward misspecification and reward hacking ........ 3 2.2 Defining situational awareness ....... . . . . . . . . . . 4 2.3 Situational awareness enables deceptive reward hacking ...... 4 3 Internally-represented goals 5 4 Power-seeking behavior 7 4.1 Many broadly-scoped goals incentivize power-seeking ....... 8 4.2 Power-seeking policies would choose high-reward behaviors for instrumental reasons ..8 4.3 Misaligned AGIs could gain control of key levers of power .... 9 5 Research directions in alignment 9 6 Conclusion 10 Introduction Over the last decade, advances in deep learning have led to the development of large neural networks with impressive capabilities in a wide range of domains. In addition to reaching human-level performance on complex games like Starcraft [Vinyals et al., 2019] and Diplomacy [Bakhtin et al., 2022], large neural networks show evidence of increasing generality [Bommasani et al., 2021], including advances in sample efficiency [Brown et al., 2020, Dorner, 2021], cross-task generalization [Adam et al., 2021], and multi-step reasoning [Chowdhery et al., 2022]. The rapid pace of these advances highlights the possibility that, within the coming decades, we develop artificial general intelligence (AGI) - that is, AI which can apply domain-general cognitive skills (such as reasoning, memory, and planning) to perform at or above human level on a wide range of cognitive tasks relevant to the real world (such as writing software, formulating new scientific theories, or running a company). 1 This possibility is taken seriously by leading ML researchers, who in two recent...

Happy New Year! Welcome to 2023. We'll talk today about information you might need for this new year. We learn about new tax amounts; ways to save money, ways to get back lost money, and ways to win money. Our experts take questions concerning personal finance.2023$66,000 - Defined contribution plan (401k, 403b) total limit per participant $66,000 - SEP IRA contribution limit $22,500 - 401k, 403b, 457 contribution limit (Salary deferrals)$7,500 - 401k, 403b, 457 catch up$6,500 - IRA contribution limit$1,000 - IRA catch up $15,500 - SIMPLE IRA contribution limit$3,500 - SIMPLE IRA catch up$27,700 - Standard deduction (MFJ)$20,800 - Standard deduction (Head of Household)$13,850 - Standard deduction (Single, MFS)$44,625 - Taxable income limit for 0% long-term capital gains (single)$89,250 - Taxable income limit for 0% long-term capital gains (MFJ)$17,000 - Gift tax exclusion $Unlimited - Brokerage account contribution limit$160,200 - Social security wage base (No longer pay social security tax after this)The amount of your taxable Social Security benefits depends on your combined income or the sum of:50% of all your Social Security benefits for the yearThe adjusted gross income (AGI), which is your total income minus adjustments to that income, such as deductions and exclusionsTax-exempt interest income, such as interest received on municipal bondsUnearned / Earned incomeStrategies for minimizing taxesRegistration is now open for the 10th Annual College Savings Mississippi Art Contest. The contest will run from January 1 through February 28. Students from across Mississippi will be competing for over $3,000 in Mississippi Affordable College Savings (MACS) scholarships by submitting an original piece of art based on the theme, “What do you want to be when you grow up?” https://treasury.ms.gov/artcontest/The weather and other operational delays and cancelations that occurred at the end of December and the beginning of January show the importance of knowing your rights and keeping receipts. Southwest airlines has a website where you can make claims due to flight cancellations. We'll have a link to their website and also the US Department of Transportation's website where you can learn more about your customer rights. https://www.southwest.com/traveldisruption/ https://www.transportation.gov/airconsumer/airline-customer-service-dashboard The United States Postal Service filed notice of Mailing Services price changes to take effect January 22, 2023. The new rates include a three-cent increase in the price of a First-Class Mail® Forever stamp from 60 cents to 63 cents. https://faq.usps.com/s/article/2023-Postage-Price-Changes Hosted on Acast. See acast.com/privacy for more information.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What would it look like if it looked like AGI was far?, published by Tomás B. on January 9, 2023 on LessWrong. This is a sequel to my 2021 post What would it look like if it looked like AGI was very near?, which was a scenario-based planning thought exercise about short timelines, and prompted an excellent comment from Aiyen. As average timelines have decreased substantially in the last two years, if only for shear contrarianism it seems worthwhile to think a little bit about long timeline scenarios. So the scenario-based planning thought exercise goes like this: Imagine it is 30 years from now and AGI has NOT happened and you are writing a post on how in hindsight it was obvious that AGI was much harder than it appeared to many in 2023. What symptoms of this do you see of this now now? Contingent on this scenario being true, what symptoms would you expect in the next couple decades. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AGI and the EMH: markets are not expecting aligned or unaligned AI in the next 30 years, published by basil.halperin on January 10, 2023 on The Effective Altruism Forum. by Trevor Chow, Basil Halperin, and J. Zachary Mazlish In this post, we point out that short AI timelines would cause real interest rates to be high, and would do so under expectations of either unaligned or aligned AI. However, 30- to 50-year real interest rates are low. We argue that this suggests one of two possibilities: Long(er) timelines. Financial markets are often highly effective information aggregators (the “efficient market hypothesis”), and therefore real interest rates accurately reflect that transformative AI is unlikely to be developed in the next 30-50 years. Market inefficiency. Markets are radically underestimating how soon advanced AI technology will be developed, and real interest rates are therefore too low. There is thus an opportunity for philanthropists to borrow while real rates are low to cheaply do good today; and/or an opportunity for anyone to earn excess returns by betting that real rates will rise. In the rest of this post we flesh out this argument. Both intuitively and under every mainstream economic model, the “explosive growth” caused by aligned AI would cause high real interest rates. Both intuitively and under every mainstream economic model, the existential risk caused by unaligned AI would cause high real interest rates. We show that in the historical data, indeed, real interest rates have been correlated with future growth. Plugging the Cotra probabilities for AI timelines into the baseline workhorse model of economic growth implies substantially higher real interest rates today. In particular, we argue that markets are decisively rejecting the shortest possible timelines of 0-10 years. We argue that the efficient market hypothesis (EMH) is a reasonable prior, and therefore one reasonable interpretation of low real rates is that since markets are simply not forecasting short timelines, neither should we be forecasting short timelines. Alternatively, if you believe that financial markets are wrong, then you have the opportunity to (1) borrow cheaply today and use that money to e.g. fund AI safety work; and/or (2) earn alpha by betting that real rates will rise. An order-of-magnitude estimate is that, if markets are getting this wrong, then there is easily $1 trillion lying on the table in the US treasury bond market alone – setting aside the enormous implications for every other asset class. Interpretation. We view our argument as the best existing outside view evidence on AI timelines – but also as only one model among a mixture of models that you should consider when thinking about AI timelines. The logic here is a simple implication of a few basic concepts in orthodox economic theory and some supporting empirical evidence, which is important because the unprecedented nature of transformative AI makes “reference class”-based outside views difficult to construct. This outside view approach contrasts with, and complements, an inside view approach, which attempts to build a detailed structural model of the world to forecast timelines (e.g. Cotra 2020; see also Nostalgebraist 2022). Outline. If you want a short version of the argument, sections I and II (700 words) are the heart of the post. Additionally, the section titles are themselves summaries, and we use text formatting to highlight key ideas. I. Long-term real rates would be high if the market was pricing advanced AI Real interest rates reflect, among other things: Time discounting, which includes the probability of death Expectations of future economic growth This claim is compactly summarized in the “Ramsey rule” (and the only math that we will introduce in this post), a version of the “Euler equation” th...

I think self development and emotional intelligence is foundational to our relationships with others. It's how we interact with our patients, significant others, children and family members. Investing in your personal development makes you a better communicator. That's why you should listen to Dr. Agi Keramidas in this episode! We discussed how to begin your self-development journey and what are the important areas for dentists to focus on. We hope you'll apply just ONE THING whether it's a book title that we recommend or implement one tip to better yourself or your interactions. https://www.youtube.com/watch?v=HzItyEV3x_c Check out the Video for Free on YouTube Need to Read it? Check out the Full Episode Transcript below! Highlights of the episode: 4:58 Dr. Agi Keramidas' Introduction10:57 Getting started with self-development14:05 Important areas to develop for Dentists21:04 Dr. Agi's communication style30:03 Roadmap to self-development35:13 Practical tip for self-development Check out these books recommended by Dr. Agi Keramidas How to Win Friends & Influence People by Dale CarnegieRich Dad Poor Dad by Robert KiyosakiThe Biology of Belief by Dr. Bruce LiptonThe Daily Stoic by Ryan Holiday and Stephen Hanselman Check out Dr. Agi Keramidas' podcast: Personal Development Mastery for more personal development episodes Website: agikeramidas.comPodcast: personaldevelopmentmasterypodcast.com If you loved this episode, you will like 12 Rules for Dentistry Click below for full episode transcript: Jaz's Introduction: One of the most common questions I get to the podcast is, WHICH SCANNER SHOULD I BUY? 'Hey, Jaz, I'm starting to make this venture into digital dentistry.' And it's overwhelming, right? Jaz's Introduction:There's so much out there in terms of different types of scanners, different requirements, so which is the best scanner? Which is the most appropriate scanner for your clinic and your requirements? Well, that's exactly what we're covering today with Dr. Gulshan Murgai. If you haven't listened already to IC027, it was his story that we covered and it's titled, 'He got Sued and he won.' So essentially, it's his entire legal case how it led to that. And we learned so many different medical legal lessons and it was just great to hear about his triumph. So, if you haven't listened to that, please do listen to that. But if you've clicked on, because of the title of digital dentistry in which scanner to buy, there's so much meat in here because Gulshan has tested and tried all these scanners, and now he's gonna summarize for you which scanner he thinks is the best and why that might change depending on your needs and the needs of your practice. Now, Gulshan doesn't keep it a secret or hide it or anything. He has got some financial interest in a digital scanning company. But I still think there's so much of meat in terms of gaining the knowledge to know what are the differences in the different scanners and how you can choose which scanner to invest in. Let's join Dr. Gulshan. I'll catch you in the outro. Main Episode:Dr. Gulshan Murgai, after that really enlightening episode about how you triumphantly won the case and so many lessons shared. Welcome back to the Protrusive Dental Podcast. This group function today is to answer one burning question, and I've seen you lectured before. I know you're super passionate about digital dentistry. For those people who haven't listened to that episode yet, just introduce yourself again and why you are so involved with digital dentistry why you are so passionate about digital dentistry. [Gulshan]So, my name's Gulshan Murgai. I'm a practitioner, general practitioner practice in Watford, Northamptonshire. So, I'm basically got exposed to digital dentistry as an undergraduate. So year 2000's thanks to Professor Burke at University of Birmingham. He brought in the guys that were selling Cerec at the time.