Best podcasts about lesswrong

Latest podcast episodes about lesswrong

The Bayesian Conspiracy
175 – FTX + EA, and Personal Finance

The Bayesian Conspiracy

Play Episode Listen Later Nov 16, 2022 105:34


David from The Mind Killer joins us. We got to talking about the FTX collapse and some of the waves it sent through the Effective Altruism community. Afterwards David helps us out with personal finance. 0:00:00 Intro 0:02:08 The Sequences … Continue reading →

The Nonlinear Library
LW - Will we run out of ML data? Evidence from projecting dataset size trends by Pablo Villalobos

The Nonlinear Library

Play Episode Listen Later Nov 14, 2022 4:31


Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Will we run out of ML data? Evidence from projecting dataset size trends, published by Pablo Villalobos on November 14, 2022 on LessWrong. Summary: Based on our previous analysis of trends in dataset size, we project the growth of dataset size in the language and vision domains. We explore the limits of this trend by estimating the total stock of available unlabeled data over the next decades. Read the full paper in arXiv. Our projections predict that we will have exhausted the stock of low-quality language data by 2030 to 2050, high-quality language data before 2026, and vision data by 2030 to 2060. This might slow down ML progress. All of our conclusions rely on the unrealistic assumptions that current trends in ML data usage and production will continue and that there will be no major innovations in data efficiency. Relaxing these and other assumptions would be promising future work. Historical projectionCompute projectionLow-quality language stock2032.4[2028.4 ; 2039.2] 2040.5[2034.6 ; 2048.9]High-quality language stock2024.5[2023.5 ; 2025.7]2024.1[2023.2 ; 2025.3]Image stock2046[2037 ; 2062.8]2038.8[2032 ; 2049.8]Table 1: Median and 90% CI exhaustion dates for each pair of projections. Background Chinchilla's wild implications argued that training data would soon become a bottleneck for scaling large language models. At Epoch we have been collecting data about trends in ML inputs, including training data. Using this dataset, we estimated the historical rate of growth in training dataset size for language and image models. Projecting the historical trend into the future is likely to be misleading, because this trend is supported by an abnormally large increase in compute in the past decade. To account for this, we also employ our compute availability projections to estimate the dataset size that will be compute-optimal in future years using the Chinchilla scaling laws. We estimate the total stock of English language and image data in future years using a series of probabilistic models. For language, in addition to the total stock of data, we estimate the stock of high-quality language data, which is the kind of data commonly used to train large language models. We are less confident in our models of the stock of vision data because we spent less time on them. We think it is best to think of them as lower bounds rather than accurate estimates. Results Finally, we compare the projections of training dataset size and total data stocks. The results can be seen in the figure above. Datasets grow much faster than data stocks, so if current trends continue, exhausting the stock of data is unavoidable. The table above shows the median exhaustion years for each intersection between projections. In theory, these dates might signify a transition from a regime where compute is the main bottleneck to growth of ML models to a regime where data is the taut constraint. In practice, this analysis has serious limitations, so the model uncertainty is very high. A more realistic model should take into account increases in data efficiency, the use of synthetic data, and other algorithmic and economic factors. In particular, we have seen some promising early advances on data efficiency, so if lack of data becomes a larger problem in the future we might expect larger advances to follow. This is particularly true because unlabeled data has never been a constraint in the past, so there is probably a lot of low-hanging fruit in unlabeled data efficiency. In the particular case of high-quality data, there are even more possibilities, such as quantity-quality tradeoffs and learned metrics to extract high-quality data from low-quality sources. All in all, we believe that there is about a 20% chance that the scaling (as measured in training compute) of ML models will significantly slow down b...

The Bayesian Conspiracy
BREAKING – FTX collapse & EA shockwaves

The Bayesian Conspiracy

Play Episode Listen Later Nov 13, 2022 43:26


While recording the next episode we got to talking about the FTX collapse and some of the waves it sent through the Effective Altruism community. We decided to break it out into a separate segment so it can air while … Continue reading →

The Nonlinear Library
LW - Noting an unsubstantiated communal belief about the FTX disaster by Yitz

The Nonlinear Library

Play Episode Listen Later Nov 13, 2022 0:27


Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Noting an unsubstantiated communal belief about the FTX disaster, published by Yitz on November 13, 2022 on LessWrong. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

The Nonlinear Library
LW - What's the Alternative to Independence? by jefftk

The Nonlinear Library

Play Episode Listen Later Nov 13, 2022 1:59


Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What's the Alternative to Independence?, published by jefftk on November 13, 2022 on LessWrong. When I talk about teaching my kids to be independent, trying to get them to where they can do more on their own, one response I often get is, paraphrasing: I like spending time with my kids and don't see it as a burden. Bringing them to school, playing with them, making their food: these are all chances to connect and enjoy being together. They'll be old enough not to need me soon enough, and in the meantime I enjoy bringing them to the playground. So, first, I also like spending time with my kids! We do a lot of things together, and I'm happy about that. But it's also common that they'll want to do things that I can't do with them: One wants to go to the park and the other wants to stay home. One of them is ready to go to school and wants to get there early to play with friends, but the other isn't ready yet. With a third child now this comes up even more. At times: The older two want to go over to a friend's house, but the baby is napping. I'm still feeding the baby breakfast when it's time for school. The best time for the baby's afternoon nap conflicts with school pickup. The alternative to doing things on their own is typically not us doing the same things together. Instead, it's at least one kid needing to accept doing something they like much less, and typically a lot more indoor time. I do think there is some truth in the original point, though. There are times when the alternative to "they go to the park" is just "I take them to the park". Sometimes that's fine (they want to play with friends, I want to write), other times less so (they want to play monster and I don't have anything that's actually more important). With this approach you do need to be thoughtful about making sure you're spending an amount of time with them that you all are, and will be, happy with. Comment via: facebook, mastodon Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

The Nonlinear Library
LW - The Alignment Community Is Culturally Broken by sudo -i

The Nonlinear Library

Play Episode Listen Later Nov 13, 2022 3:02


Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Alignment Community Is Culturally Broken, published by sudo -i on November 13, 2022 on LessWrong. Disclaimer: These are entirely my thoughts. I'm posting this before it's fully polished because it never will be. Epistemic status: Moderately confident. Deliberately provocative title. Apparently, the Bay Area rationalist community has a burnout problem. I have no idea if it's worse than base rate, but I've been told it's pretty bad. I suspect that the way burnout manifests in the rationalist community is uniquely screwed up. I was crying the other night because our light cone is about to get ripped to shreds. I'm gonna do everything I can to do battle against the forces that threaten to destroy us. You've heard this story before. Short timelines. Tick. Tick. I've been taking alignment seriously for about a year now, and I'm ready to get serious. I've thought hard about what my strengths are. I've thought hard about what I'm capable of. I'm dropping out of Stanford, I've got something that looks like a plan, I've got the rocky theme song playing, and I'm ready to do this. A few days later, I saw this post. And it reminded me of everything that bothers me about the EA community. Habryka covered the object level problems pretty well, but I need to communicate something a little more... delicate. I understand that everyone is totally depressed because qualia is doomed. I understand that we really want to creatively reprioritize. I completely sympathize with this. I want to address the central flaw of Akash+Olivia+Thomas's argument in the Buying Time post, which is that actually, people can improve at things. There's something deeply discouraging about being told "you're an X% researcher, and if X>Y, then you should stay in alignment. Otherwise, do a different intervention." No other effective/productive community does this. I don't know how to put this, but the vibes are deeply off. The appropriate level of confidence to have about a statement like "I can tell how good of an alignment researcher you will be after a year of you doing alignment research" feels like it should be pretty low. At a year, there's almost certainly ways to improve that haven't been tried. Especially in a community so mimetically allergic to the idea of malleable human potential. Here's a hypothesis. I in no way mean to imply that this is the only mechanism by which burnout happens in our community, but I think it's probably a pretty big one. It's not nice to be in a community that constantly hints that you might just not be good enough and that you can't get good enough. Our community seems to love treating people like mass-produced automatons with a fixed and easily assessable "ability" attribute. (Maybe you flippantly read that sentence and went "yeah it's called g factor lulz." In that case, maybe reflect on good of a correlate g is in absolute terms for the things you care about.). If we want to actually accomplish anything, we need to encourage people to make bigger bets, and to stop stacking up credentials so that fellow EAs think they have a chance. It's not hubris to believe in yourself. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

The Nonlinear Library
LW - Weekly Roundup #5 by Zvi

The Nonlinear Library

Play Episode Listen Later Nov 13, 2022 9:25


Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Weekly Roundup #5, published by Zvi on November 11, 2022 on LessWrong. A note about what isn't in this roundup: Nothing about the midterms or crypto/FTX. I wrote about Twitter earlier this week. The midterms are better covered elsewhere, I do not have anything unique to say about them. As for the situation with FTX, it is rapidly developing and I do not yet feel I have sufficient handle on it that saying things would be net helpful. I continue to recommend starting on that topic by reading Matt Levine (both in general, and for this in particular). There were going to bea few additional things related to Twitter and Musk, but the section rapidly spiraled out of control as news came on top of news and I didn't know if my statements were current, so I'm moving it to another draft and we'll see if it ends up making it or not. For now, I will simply say that how badly and quickly you want to Find Out, which can only be done by Fing Around, is increasingly the question of our age. It is indeed weird to have so much going on and have that result in having less to say for the moment, while thoughts are gathered and things become clearer. Seems right, though, and also some background writing tasks got more urgent this week. Bad News More on the chess cheating scandals (via MR). My main contention continues to be that we are far too unwilling to punish competitors on the basis of statistical evidence in cheating cases. FIDE's ‘99.999% sure' policy, or whatever exactly it is, is Obvious Nonsense. No one is going to jail or being executed, and there are plenty of options well short of ‘can never again play competitive chess.' Where would I put the bar? I don't think it is crazy to put it at 51% if your estimates are fair and well calibrated. Certainly if I thought someone was 50/50 to be a savage cheater, I would not invite them to join my Magic (or chess) team or play group, help promote them, or anything like that. If I was running an invitational chess tournament, I would consider even a 10% chance to be a rather severe demerit to the extent that I had the freedom to consider it. My guess is that in practice the right answer for Serious Official Punishments is in the 75%-90% range. Uber tests using push notifications to deliver ads. I am trying to imagine a world where this is a good business decision and failing. Are falling retirement ages good or bad for people? Joe's question also raises the question of, if we are or become importantly poorer than we used to be, we should progressively raise the age to collect full benefits. On reflection, I continue to think that this is an area where people are going to make decisions poorly, in both directions. Some people will retire far too early, either run out of funds or grow bored and lonely and have their health degenerate faster than it otherwise would have. Those that realize this is happening will then mostly have few good options to undo what was done. Others will refuse to retire for too long, although I expect this group to be smaller. My guess is that of those who keep working too long at their job, a lot of them would benefit from changing jobs and actively doing something else more than they would benefit from full retirement. Do I plan to retire? My basic answer is no, at least not until very late in the game when I am unable to meaningfully work. I do not see such a decision improving my life. My hope is that I can fully retire from having to do work for or because of the money while continuing to do plenty of work. I don't want to be keeping up quite this pace of work indefinitely, I likely should be taking more time to relax than I am as it is. I'm working on that. In Magic: The Gathering, Aaron Forsythe asks why the Standard format is dying and gets a lot of answers, including this one by Brian Kowal. Standard was ...

The Nonlinear Library
LW - Announcing Nonlinear Emergency Funding by KatWoods

The Nonlinear Library

Play Episode Listen Later Nov 13, 2022 0:25


Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Announcing Nonlinear Emergency Funding, published by KatWoods on November 13, 2022 on LessWrong. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

The Nonlinear Library
LW - Covid 11/10/22: Into the Background by Zvi

The Nonlinear Library

Play Episode Listen Later Nov 13, 2022 7:17


Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Covid 11/10/22: Into the Background, published by Zvi on November 10, 2022 on LessWrong. There was a lot of news this week. Elon Musk continued to Do Things at Twitter. America had midterm elections. The polls were roughly accurate, with ‘candidate quality' mattering more than expected especially on the Republican side. FTX imploded. For now I am letting Matt Levine handle coverage of what happened. None of it had anything directly to do with Covid. So this will be short. Executive Summary I hear there were midterms. And that FTX imploded. Covid death number down a lot, presumably not a real effect. Let's run the numbers. The Numbers Predictions Predictions from Last Week: 255k cases (+7%) and 2,600 deaths (-1%) Results: 242k cases (+1%) and 1,993 deaths (-24%!). Predictions for Next Week: 250k cases (+4%) and 2,400 deaths (+20%). That death number is super weird. At first I thought ‘what, they can count votes or they can count deaths but not both at the same time?' or ‘election day isn't actually a holiday is it?' Then the case number came in flat even in the South, although Alabama didn't report any cases at all (which wasn't a big enough effect for me to adjust). Some of the drop is that last week had a spike of about 250 deaths in North Carolina. Still leaves the majority of the gap unexplained. I don't know. I don't see how there could have been a large real drop in deaths, and if it was a reporting issue we would have seen a decline in cases. Also, in the regions where we see a decline in deaths, West and South, we don't see relatively few cases. So the reporting explanations don't make that much sense here, and it seems unlikely cases actually rose a lot while being underreported or anything like that. It does raise uncertainty in deaths a lot for next week, and to some extent also for cases, in the usual ‘was this real or was it both fake and in need of reversal' dilemma. We shall see. The good news is that there is not much practical impact on decision making, unless this is all hiding a tidal wave of new infections. That is possible. I would still not expect anything like last year's wave, and for things not go on too long before stabilizing, but the timing between weather and sub-variants would make some sense. Deaths Cases Booster Boosting and Variants Nothing. Physical World Modeling Babies born during lockdown more often miss developmental milestones (study). I doubt this leads to that much in the way of permanent impacts. It still seems rather not good, the effect sizes here are quite large. Study finds that Paxlovid reduces risk of Long Covid. Would be weird if it didn't. China Via MR, Nucleic Acid Testing ‘now accounts for 1.3% of China's GDP.' Zero Covid is a rather large drag on China's economy that will be cumulative over time. From the comments, which also interestingly feature massive negative votes everywhere: I'm a teacher in China and EVERYONE gets tested at least THREE times per week. It is such a waste of time and money. Meanwhile full surface paranoia continues. The Zero Covid principle and the constant testing would not be my approach, but I understand. The surfaces obsession is something different. Everyone has to know at this point that it doesn't do anything. China reiterates Covid Zero Policy (Bloomberg), calls for more targeted and also ‘more decisive' measures. No signs they will abandon surface paranoia. This seems like it likely gets worse before it gets better. And Now An Important Message An SNL sketch, best of the season by far. If you haven't yet, watch it. Long Covid Article from Yale on chronic disease in general and how Long Covid may help us get a better understanding of what it is and how to deal with it. Does not provide much new. Mostly it is the same old ‘now that we have an Official Recognition for a chronic disease m...

The Nonlinear Library
EA - Women and Effective Altruism by Keerthana Gopalakrishnan

The Nonlinear Library

Play Episode Listen Later Nov 12, 2022 5:38


Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Women and Effective Altruism, published by Keerthana Gopalakrishnan on November 12, 2022 on The Effective Altruism Forum. A lot has been talked about SBF/FTX/EA but this coverage reminds me that is time to talk about the toxicity of the culture within EA communities, especially as it relates to women. EA circles, much like the group house in Bahamas, are widely incestous where people mix their work life (in EA cause areas), their often polyamorous love life and social life in one amalgomous mix without strict separations. This is the default status quo. This means that if you're a reasonably attractive woman entering an EA community, you get a ton of sexual requests to join polycules, often from poly and partnered men. Some of these men control funding for projects and enjoy high status in EA communities and that means there are real downsides to refusing their sexual advances and pressure to say yes, especially if your career is in an EA cause area or is funded by them. There are also upsides, as reported by CoinDesk on Caroline Ellison. From experience it appears that, a ‘no' once said is not enough for many men in EA. Having to keep replenishing that ‘no' becomes annoying very fast, and becomes harder to give informed consent when socializing in the presence of alcohol/psychedelics. It puts your safety at risk. From experience, EA as a community, has very little respect for monogamy and many men, often competing with each other, will persuade you to join polyamory using LessWrong style jedi mindtricks while they stand to benefit from the erosion of your boundaries. (Edit: I have personally experienced this more than three times in less than one year of attending EA events and that is far too many times. ) So how do these men maintain polycules and find sexual novelty? EA meet ups of course. Several EA communities are grounds for predatory men in search of their nth polycule partner and to fill their “dancecards”. I have seen this in NYC EA circles, I have seen this in SF. I decided to stop socializing in EA circles a couple months ago due to this toxicity, the benefits are not worth the uncovered downside risk. I also am lucky enough to not work for an EA aligned organization / cause area and socially diversified enough to take that hit. The power enjoyed by men who are predatory, the rate of occurrence and a lack of visible push back equals to a tacit and somewhat widespread backing for this behaviour. My experience resonates with a few other women in SF I have spoken to. They have also met red pilled, exploitative men in EA/rationalist circles. EA/rationalism and redpill fit like yin and yang. Akin to how EA is an optimization of altruism with “suboptimal” human tendencies like morality and empathy stripped from it, red pill is an optimized sexual strategy with the humanity of women stripped from it. You'll also, surprisingly, encounter many women who are redpilled and manifest internalized misogyny in EA. How to check if you're one: if terms like SMV, hypergamy etc are part of your everyday vocabulary and thought processes, you might be affected. You'll also encounter many women who are unhappy participants in polygamous relationships; some of them really smart women who agree to be unhappy (dump him, sis). And if you're a lucky woman who hasn't experienced this in EA, great, and your experience does not need to negate those of others. Despite this culture, EA as a philosophy has a lot of good in it and they should fix this bug with some introspection. Now mind you, this is not a criticism of polyamory itself. If polyamorous love happens between consenting adults without favoritism in professional settings, all is well and good. But EA is an organization and community focused on a mission of altruism, enjoy huge swathes of donor money and exert socio-political ...

The Nonlinear Library
LW - Rudeness, a useful coordination mechanic by Raemon

The Nonlinear Library

Play Episode Listen Later Nov 12, 2022 4:15


Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Rudeness, a useful coordination mechanic, published by Raemon on November 11, 2022 on LessWrong. I think the concept of "rudeness" is underappreciated. (Or, if people are appreciating it, they're doing so quietly where I can't find out about it) I think a lot of coordination-social-tech relies on there being some kind of karmic balance. A lot of actions aren't expressly illegal, and not even blatantly all-the-time socially sanctioned. But if you do them a bit, it's considered rude, and if you're rude all the time, you get a reputation for being rude, and this comes with some consequences (i.e. not invited to as many parties). The-concept-of-rudeness gives you a tool to softly shape your culture, and have some kind of standards, without having to be really rigid about it. [Edit to add:]I'm writing this post because I was writing another coordination/epistemic-norms post, and I found myself wanting to write the sentence "If you do [X thing], it should be considered a bit rude. If you do [X' worse version of X thing], it's more rude." And then I realized this was resting on some underlying assumptions about coordination-culture that might not be obvious to everyone. (i.e. that it's good to have some things be considered "rude") I come from a game-design background. In many games, there are multiple resources, and there are multiple game-mechanics for spending those resources, or having them interact with each other. You might have life-points, you might have some kind of "money" (which can store value and then be spent in arbitrary quantities), or renewable resources (like grains that grow back every year, and spoil if you leave them too long). Many good games have rich mechanics that you can fine-tune, to shape the player's experience. A flexible mechanic gives you knobs-to-turn, to make some actions more expensive or cheaper. The invention of "money" in real life was a useful coordination mechanic. The invention of "vague social capital that you accumulate doing high status respectable things, and which you can lose by doing low status unrespectable things" predates money by a long time, and is still sometimes useful in ways that money is not. A feeling-of-rudeness is one particular flavor of what "spending-down social capital" can feel like, from the inside of a social interaction. [/edit] Different cultures have different conceptions of "what is rude." Some of that is silly/meaningless, or actively harmful. Some of it is arbitrary (but maybe the arbitrariness is doing some secret social cohesion that's not obvious and autists should learn to respect anyway). In some cultures belching at a meal is rude, in others not belching at a meal is rude. I think there's probably value in having shared scripts for showing respect. Epistemic cultures have their own uses for "the rudeness mechanic." You might consider it rude to loudly assert people should agree with you, without putting forth evidence to support your claim. You might consider it rude to make a strong claim without being willing to make concrete bets based on it. Or, you might consider it rude to demand people to make bets on topics that are fuzzy and aren't actually amenable to forecasting. Rudeness depends on circumstance Different domains might warrant different kinds of conceptions-of-rudeness. In the military, I suspect "being rude to your superiors" is actually an important thing to discourage, so that decisions can get made quickly. But it can be actively harmful in innovation driven industries. An individual norm of "rudeness" can be context-dependent and depend on other norms. According to this Quora article, in Japan it's normally rude to tell your boss he's wrong, but also you're supposed to go out drinking with your boss where it's more okay to say rude things under the cover of alcohol. Problems with...

The Nonlinear Library
LW - We must be very clear: fraud in the service of effective altruism is unacceptable by evhub

The Nonlinear Library

Play Episode Listen Later Nov 11, 2022 0:28


Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: We must be very clear: fraud in the service of effective altruism is unacceptable, published by evhub on November 10, 2022 on LessWrong. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

The Nonlinear Library
LW - The optimal angle for a solar boiler is different than for a solar panel by Yair Halberstadt

The Nonlinear Library

Play Episode Listen Later Nov 11, 2022 2:53


Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The optimal angle for a solar boiler is different than for a solar panel, published by Yair Halberstadt on November 10, 2022 on LessWrong. I have both photovoltaic panels, and a solar boiler on my roof. The solar boiler is in front, and the photovoltaic panels behind. You'll notice that they're at very different angles - the panel for the solar boiler is at approximately 55 degrees, whereas the photovoltaic panels are at about 10 - 20 degrees. Why? A solar boiler works by running water through a thin black panel, which gets very hot in the sun, and heats up the water. The hot water is then stored in a tank which can keep it hot for up to about a day. A photovoltaic panel generates electricity from sunlight. In Israel, this electricity is fed directly into the grid, and we get paid 0.48 ILS per KWH. For perspective we pay about 0.51 ILS per KWH, so we get a pretty good price - there's no point storing the electricity in a battery to make sure we use it ourselves. Both require direct sunlight to work, and both should be directly facing the sun to maximize efficiency.However the use cases for both are completely different: For the photovoltaic panels I want to maximize total energy produced. Since the sun moves around over the course of the day, the panels face south to maximize time in direct sunlight. And since the angle of the sun when it's in the south changes throughout the year, you'll want to angle the panels at about the average angle of the sun when it's in the south. This is equivalent to your latitude - in Israel 31 degrees. There are a few complexities: You'll want to capture some light the rest of the day, which pushes for the angle to be steeper, During the winter the sky is cloudy and the sun is weak so you won't be able to capture much energy anyway. This pushes for the panel to be flatter, to optimize for the summer. In practice it probably roughly evens out. In Israel the difference in efficiency between a completely flat panel and one at the optimal angle is only 10%. For a solar boiler however my aim is completely different. I can't heat more than a single tanks worth of hot water, which is enough for my family of 4 and a couple of guests to have hot water, as well as plenty left over for washing up and miscellaneous use. During the summer we easily get this. Thus there's no point optimizing the solar boiler for summer. Instead it has to be at a steep angle so that we still get a decent amount of hot water during the winter. In practice this works quite well, and we usually only need to turn on the electric boiler if we have guests or it's an especially cloudy day. The same considerations apply if you're in a country where you can't sell PV power back to the grid - you'll get more peak energy than you'll use during the summer, so it's worth optimizing the angle for other times of day and other times of year. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

The Nonlinear Library
LW - Instrumental convergence is what makes general intelligence possible by tailcalled

The Nonlinear Library

Play Episode Listen Later Nov 11, 2022 6:48


Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Instrumental convergence is what makes general intelligence possible, published by tailcalled on November 11, 2022 on LessWrong. TL;DR: General intelligence is possible because solving real-world problems requires solving common subtasks. Common subtasks are what give us instrumental convergence. Common subtasks are also what make AI useful; you want AIs to pursue instrumentally convergent goals. Capabilities research proceeds by figuring out algorithms for instrumentally convergent cognition. Consequentialism and search are fairly general ways of solving common subtasks. General intelligence is possible because solving real-world problems requires solving common subtasks No-free-lunch theorems assert that any cognitive algorithm is equally successful when averaged over all possible tasks. This might sound strange, so here's an intuition pump. Suppose you get a test like 2+2 = _ 32 = _ and so on. One cognitive algorithm would be to evaluate the arithmetic expression and fill the answer in as the result. This algorithm seems so natural that it's hard to imagine how the no-free-lunch theorem could apply to this; what possible task could ever make arithmetic score poorly on questions like the above? Easy: While an arithmetic evaluator would score well if you e.g. get 1 point for each expression you evaluate arithmetically, it would score very poorly if you e.g. lose 1 point for each expression you evaluate arithmetically. This doesn't matter much in the real world because you are much more likely to encounter situations where it's useful to do arithmetic right than you are to encounter situations where it's useful to do arithmetic wrong. No-free-lunch theorems point out that when you average all tasks, useful tasks like "do arithmetic correctly" are perfectly cancelled out by useless tasks like "do arithmetic wrong"; but in reality you don't average over all conceivable tasks. If there were no correlations between subtasks, there would be no generally useful algorithms. And if every goal required a unique algorithm, general intelligence would not exist in any meaningful sense; the generally-useful cognitions are what constitutes general intelligence. Common subtasks are what give us instrumental convergence Instrumental convergence basically reduces to acquiring and maintaining power (when including resources under the definition of power). And this is an instance of common subtasks: lots of strategies require power, so a step in lots of strategies is to accumulate or preserve power. Therefore, just about any highly capable cognitive system is going to be good at getting power. "Common subtasks" views instrumental convergence somewhat more generally than is usually emphasized. For instance, instrumental convergence is not just about goals, but also about cognitive algorithms. Convolutions and big matrix multiplications seem like a common subtask, so they can be considered instrumentally convergent in a more general sense. I don't think this is a major shift from how it's usually thought of; computation and intelligence are usually considered as instrumentally convergent goals, so why not algorithms too? Common subtasks are also what make AI useful; you want AIs to pursue instrumentally convergent goals The logic is simple enough: if you have an algorithm that solves a one-off task, then it is at most going to be useful once. Meanwhile, if you have an algorithm that solves a common task, then that algorithm is commonly useful. An algorithm that can classify images is useful; an algorithm that can classify a single image is not. This applies even to power-seeking. One instance of power-seeking would be earning money; indeed an AI that can autonomously earn money sounds a lot more useful than one that cannot. It even applies to "dark" power-seeking, like social manipulatio...

The Nonlinear Library
LW - Speculation on Current Opportunities for Unusually High Impact in Global Health by johnswentworth

The Nonlinear Library

Play Episode Listen Later Nov 11, 2022 6:53


Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Speculation on Current Opportunities for Unusually High Impact in Global Health, published by johnswentworth on November 11, 2022 on LessWrong. Epistemic Status: armchair speculation from a non-expert. Short version: I expect things to get pretty bad in the Sahel region over the next year in particular. The area is an obvious target for global health interventions even in good times, and impact is presumably higher in bad times. A simple baseline intervention: fill a backpack with antibiotics, fly to the region, and travel around distributing the antibiotics. What's The “Sahel” Region? The Sahel is a semi-arid region along the southern edge of the Sahara desert. Think roughly Mali, Niger, Chad and Sudan. Bad How? Based on statistics on the Sahel, it's one of the few remaining regions on Earth where the population is near Malthusian equilibrium. Fertility is high, contraception is rare; about half the population is under age 16. Infant mortality is around 6-8%, and ~a quarter of children are underweight. (Source: CIA World Factbook entries on Mali, Niger, Chad and Sudan.) Being near Malthusian equilibrium means that, when there's an economic downturn, a substantial chunk of the population dies. Die How? Traditional wisdom says: war, famine, disease. In this case, I'd expect famine to be the main instigator. Empty bellies then induce both violence and weak immune systems. On priors, I'd expect infectious disease to be the main proximate killer. The Next Year In Particular? The global economy has been looking rough, between the war in Ukraine shocking oil and food markets, and continuing post-Covid stagflation. Based on pulling a number out of my ass without looking at any statistics, I'd guess deaths from violence, starvation, and disease in the Sahel region will each be up an order of magnitude this year/next year compared to a good year (e.g. the first-quartile best year in the past decade). That said, the intervention we'll talk about is probably decently impactful even in a good year. So What's To Be Done? Just off the top of my head, one obvious baseline plan is: Fill a hiking backpack with antibiotics (buy them somewhere cheap!) Fly to N'Djamena or take a ferry to Timbuktu Obtain a motorbike or boat Travel around giving away antibiotics until you run out Repeat Note that you could, of course, substitute something else for "antibiotics" - maybe vitamins or antifungals or water purification tablets or iron supplements or some mix of those is higher marginal value. There are some possibly-nonobvious considerations here. First, we can safely assume that governments in the area are thoroughly corrupt at every level, and presumably the same goes for non-government bureaucracies; trying to route through a local bureaucratic machine is a recipe for failure. Thus, the importance of being physically present and physically distributing things oneself. On the other hand, physical safety is an issue, even more so if local food insecurity induces local violence or civil war. (That said, lots of Westerners these days act like they'll be immediately assaulted the moment they step into a “bad neighborhood” at night. Remember, folks, the vast majority of the locals are friendly the vast majority of the time, especially if you're going around obviously helping people. You don't need to be completely terrified of foreign territory. But, like, don't be completely naive about it either.) Also, it is important to explain what antibiotics are for and how to use them, and there will probably be language barriers. Literacy in these regions tends to be below 50%, and presumably the rural regions which most need the antibiotics also have the lowest literacy rates. How Much Impact? I'm not going to go all the way to estimating QALYs/$ here, but. according to this source, the antibiotic impor...

The Nonlinear Library
LW - What it's like to dissect a cadaver by OldManNick

The Nonlinear Library

Play Episode Listen Later Nov 10, 2022 8:14


Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What it's like to dissect a cadaver, published by OldManNick on November 10, 2022 on LessWrong. Why I never thought I was a bio person. But then I overheard Viv talking about MAOIs at a party. I asked her: > - What are MAOIs? > - monoamine oxidase inhibitor > - What does that mean? > - It prevents reuptake of neurotransmitters. > - But what is a neurotransmitter? What does reuptake actually mean? > - ... > - So life uses chiral properties of space to implement things... Viv had the most important trait of a teacher: patience. I asked the most naive questions and they answered them. They walked with me, all the way down to the very beginning, rebuilding my understanding. It was amazing. I wanted to know more. Roadblock: finding lifeforms to study. I wondered if non-medical students could watch dissections. You can't get more information about an object than by directly interacting with it. The concrete world contains the abstract one. I even asked my doctor at a physical if she knew of any, and she said to look at community colleges. After some searching, I found this: Bio 848NV. Forget viewing the dissection, you're doing the dissection. 5 hour dissection for $60, free if you just watch. The only bureaucratic hangup is that you must pay by check. This is why I love the Bay Area: there's stuff like this and you can just do it. yes it's weird no they can't stop you. The boundary between scientist and serial killer is paper thin sometimes. Takeaways I've done this a few times now. Turns out that there's way way way too much information to understand it all in one 5 hour session. Each time, we pick out areas and focus on them. Seeing how everything fits together ‒and how big it is‒ makes understanding at different scales much easier. There's a common template to life. Seeing it in you hits different. Brain has interesting connections to fractals and graph theory. Maybe pan-psychism isn't totally wrong. What & how & why I tell my friend Leah and she says “This is the most appealing activity that I've ever seen you do”. Dunno whom that says more about. We arrive and there are 5 people around 3 cadavers. We get aprons and lab coats and start syringing what's mostly Downy fabric softener with a syringe. It prevents decay and smells sickly sweet. Corpses can last a long time. One of the corpses had been dead for 5 years. Many random observations There's a crazy amount of connective tissue, and it makes a creepy wireframe surrounding your skeleton. Even the space between the folds of the brain has it. If you exercise, we'll know. Their insides just look different. “It's who you are inside that matters” is a much creepier sentence now. Veins, arteries, and nerves all travel together, wound around each other by a bunch of connective tissue. Mnemonic: VAN. Cancer can turn your guts and lungs green, and it's this horrible bright moldy green. Metastasized tissue is hard but ultimately crumbly like overcooked chicken liver. The stomach and intestines have textures reminiscent of damp cardboard, but they're dry to the touch. I finally saw a lymph node. The body has a lot of drainage into the lymphatic system. There's a bunch of tiny nerves and you can't feasibly preserve all. The etymology of the word patience is “capacity for suffering”. This is apt. You can't rush the process, and believe me, it is a process. Exposing the VANs requires reflecting away the skin, and this takes a long time and much more physical effort than you'd think. You're basically scraping it off, and the best tool overall is your hands. Skin in particular is much tougher than it looks, and I ended up locking a pair of forceps against a shoulder and just leaning back to pull it taut. Speaking of the shoulder, I spent 2 hours working up through one. There are a lot of fiddly bits. I knew, but didn't understand, ...

The Nonlinear Library
LW - A caveat to the Orthogonality Thesis by Wuschel Schulz

The Nonlinear Library

Play Episode Listen Later Nov 10, 2022 3:13


Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A caveat to the Orthogonality Thesis, published by Wuschel Schulz on November 9, 2022 on LessWrong. This post relies on an understanding of two concepts: The Orthogonality thesis and the sharp left turn. If you already know what they are, skip to the main text. The orthogonality thesis states that an agent can have any combination of intelligence and goals. It is one of the core assumptions of alignment research. The sharp left turn is a hypothesized event, where the capabilities of an AI suddenly generalize to new domains without its alignment capabilities generalizing. This process is sometimes described as “hitting the core of intelligence” and is considered to be the crucial point of alignment by some, as AIs after a sharp left turn might be more capable than humans. So we have to have AI alignment figured out before an AI takes the sharp left turn. Main text While I do think the orthogonality thesis is mostly correct, I have a small caveat: For an AI to maximize/steer towards x, x must be either part of its sensory input or its world model. Imagine a really simple “AI”: a thermostat that keeps the temperature. If you take a naive view of the orthogonality thesis, you would have to believe that there is a system as intelligent as a thermostat, that maximizes paperclips. I do not think there is such a system, because it would have no idea what a paperclip is. It doesn't even have a model of the outside world, it can only directly act on its sensory input. Even for systems that have a world model, they can only maximize things that are represented in their world model. If that world model only includes objects in a room, but the AI does not have the concept of what a person is, and what intentions are, it can optimize towards “put all the red squares in one line” but it can not optimize towards “do what the person intends”. So a more realistic view of the orthogonality thesis is the following: In this plot, I have lumped intelligence and world model complexity together. Usually, these concepts go together, because the more intelligent an AI is, the more complex its world model gets. If we found a way to make an AI have an arbitrarily complex world model while still being arbitrarily dumb, this graph would no longer hold true. Now, this is where the sharp left turn comes into play. Assuming that there is a sharp left turn, there is some maximum intelligence that a system can have, before “hitting the core of intelligence” and suddenly becoming stronger than humans. The important question is, whether a system hits this core of intelligence just by having a more and more sophisticated world model. If this is the case, then there is some maximum amount of world model complexity that a system can have before taking the sharp left turn. And that would mean in turn, that there is a cap on the goals that we can give an AI in practice. There might be Goals that are so complex, that any AI intelligent enough to understand them, would have hit the sharp left turn. There are follow up questions, that I have no answers to, and would be grateful for insightful comments: Is it true that a more and more sophisticated world model is enough to “hit the core of intelligence”? Are human values in the reachable or the unreachable goals? Is corrigibility in the reachable or the unreachable goals? Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

The Nonlinear Library
LW - I Converted Book I of The Sequences Into A Zoomer-Readable Format by dkirmani

The Nonlinear Library

Play Episode Listen Later Nov 10, 2022 3:52


Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: I Converted Book I of The Sequences Into A Zoomer-Readable Format, published by dkirmani on November 10, 2022 on LessWrong. If I (a 19 year old male) texted "www.readthesequences.com" to my roommate, the probable outcome is that he would skim the site for under a minute, text back something like "seems interesting, I'll def check it out sometime", and then proceed to never read another word. I have another friend, one that I would consider a smart guy. He would consistently rank above me in our high school's math team, and he scored in the 1500's (≥3SD) on his SATs. The same dude did not read a single book during the entirety of his high school career.[1] Attention is one's scarcest resource, and actually reading something longer than a paragraph is a trivial inconvenience, especially for my generation. What, then, does manage to hold the fickle eyeballs of zoomers like me? Well, TikTok, mostly. However, there is one (very popular) genre of TikTok video worth investigating. In this genre of video, a Reddit post is broken into sub-paragraph chunks of text, and these chunks are sequentially rendered onscreen while a text-to-speech program reads them to the user. The text is overlaid upon a background video, which is either gameplay from the mobile game Subway Surfers, or parkour footage from Minecraft. The background gameplay provides engaging novelty to the user's visual cortex, while the synthetic voice ensures that the user doesn't have to go through the hard work of translating symbols into sounds. Really, it's all quite hypnotizing. The fact that these videos are often recommended by TikTok's algorithm imply that they are among the most-engaging videos that our civilization produces. Therefore, to reduce the effort-cost of reading the sequences, I gave the TikTok treatment to Book I ("Map and Territory") of Rationality: From AI to Zombies. Predictably Wrong What Do I Mean By “Rationality”? Feeling Rational Why Truth? And. . What's a Bias, Again? Availability Burdensome Details Planning Fallacy Illusion of Transparency: Why No One Understands You Expecting Short Inferential Distances The Lens That Sees Its Own Flaws Fake Beliefs Making Beliefs Pay Rent (in Anticipated Experiences) A Fable of Science and Politics Belief in Belief Bayesian Judo Pretending to be Wise Religion's Claim to be Non-Disprovable Professing and Cheering Belief as Attire Applause Lights Noticing Confusion Focus Your Uncertainty What Is Evidence? Scientific Evidence, Legal Evidence, Rational Evidence How Much Evidence Does It Take? Einstein's Arrogance Occam's Razor Your Strength as a Rationalist Absence of Evidence Is Evidence of Absence Conservation of Expected Evidence Hindsight Devalues Science Mysterious Answers Fake Explanations Guessing the Teacher's Password Science as Attire Fake Causality Semantic Stopsigns Mysterious Answers to Mysterious Questions The Futility of Emergence Say Not “Complexity” Positive Bias: Look into the Dark Lawful Uncertainty My Wild and Reckless Youth Failing to Learn from History Making History Available Explain/Worship/Ignore? “Science” as Curiosity-Stopper Truly Part of You Interlude: The Simple Truth Do whatever you want with these videos. I may or may not convert the other 5 books of R:AZ, and I may or may not upload them to TikTok. If you want another work of text converted to video, please pitch it to me in the comments, or DM me. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

The Nonlinear Library
LW - Thinking About Mastodon by jefftk

The Nonlinear Library

Play Episode Listen Later Nov 9, 2022 2:44


Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Thinking About Mastodon, published by jefftk on November 7, 2022 on LessWrong. Social networks are normally very sticky—you want to be wherever the people you want to interact with are—so it's hard for alternatives to succeed. With the upheaval around Twitter, however, a lot of people are considering moving away from it, and rethinking social media choices in general, which makes this an unusual opportunity for community migration. I'm especially seeing a lot of suggestions that people move to Mastodon. Instead of being run by a single company that builds the software and runs the servers (Facebook, Twitter, etc) Mastodon is more like email. You sign up for an account with some provider ("server"), but can talk to people regardless of which provider they're signed up with ("federation"). There's an open protocol (ActivityPub), open-source server software (Mastodon, Pleroma, etc), and many servers you could join. Overall I'm pretty pessimistic about Mastodon, even if we only imagine futures in which lots of people move to it, because of spam. Handling spam on a centralized platform is hard but manageable; federation changes this dramatically for the worse. Imagine you're running a server, and you get an incoming message (comment, like, DM, etc) from an account on another server. Today the default is to let it through, but as Mastodon gets larger there will be more and more money to be made in spamming there, and the larger a fraction of those incoming messages would be spam. Many of the signals centralized platforms use to detect spammers (local activity, account age, IP addresses, etc) are not available in a federated context, leaving server admins (and the software they delegate to) very little information for identifying and blocking spam. It's at least as hard a problem as email spam filtering, probably harder because of shorter messages, and I expect this makes it very hard to run a server that doesn't drown its users in incoming spam and reliably gets its messages out to other servers. Maybe we get to the equivalent of everyone using Gmail/Hotmail/Yahoo and anyone with a different provider has a very frustrating time? Still, I'm happy to give it a try, and if it does collapse in a pile of spam, oh well. Does anyone have a server they'd recommend? I'm not that excited about the idea of joining a server at random because I'm trusting the server admin not to impersonate me, and also because apparently being on a server with an interesting local timeline is fun. (I considered self-hosting, but an email server is already more complex than I want to manage and a Mastodon server is much more so.) Comment via: facebook Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

The Nonlinear Library
LW - FTX will probably be sold at a steep discount. What we know and some forecasts on what will happen next by Nathan Young

The Nonlinear Library

Play Episode Listen Later Nov 9, 2022 0:32


Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: FTX will probably be sold at a steep discount. What we know and some forecasts on what will happen next, published by Nathan Young on November 9, 2022 on LessWrong. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

The Nonlinear Library
LW - Applying superintelligence without collusion by Eric Drexler

The Nonlinear Library

Play Episode Listen Later Nov 9, 2022 12:01


Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Applying superintelligence without collusion, published by Eric Drexler on November 8, 2022 on LessWrong. Epistemic status: The core ideas seem robust and stable after long reflection and many discussions. Many researchers identify AI safety with control of a monolithic, superintelligent AI system, and if questioned about multicomponent alternatives, argue that multiple superintelligent-level systems would inevitably collude and act as one. This view seems quite wrong, yet has diverted attention from a rich and promising range of multicomponent strategies for AI safety — strategies that are well aligned with the actual trajectory of AI development. Adapted from Reframing Superintelligence, Section 20: Collusion among superintelligent oracles can readily be avoided Because collusion among AI question-answering systems can readily be avoided, there is no obstacle to applying superintelligent-level AI resources to problems that include AI safety. 20.1 Summary The difficulty of establishing successful collusion among actors tends to increase as their capabilities, knowledge, situations, and roles become more diverse and adversarial (think auditors, competitors, specialists, red-teams.), and increasing the number of actors can make collusive cooperation more difficult . In the context of AI systems (even more so than among human beings), these conditions can be readily implemented and are attractive for pragmatic reasons. Arguments that, absent preexisting alignment, high-level AI systems will inevitably collude are ill-founded. Instead, we should expect that interactions among multiple superintelligent-level systems can be applied to suppress deception and reduce risk by leveraging imperfect alignment achieved at the level of individual systems. 20.2 Trustworthiness can be an emergent property Prospects for solving AI-safety problems would be greatly improved if we could safely apply superintelligent-level question-answering resources (“oracles”, or more generally, “systems that provide information in response to prompts”) to solving those problems. A familiar objection dismisses this potentially powerful approach as unsafe in itself, arguing that, absent solutions to difficult problems, individual superintelligent-level systems would be untrustworthy, and that attempts to establish checks and balances among multiple systems (for example, through superintelligent-level evaluation of potentially deceptive answers to questions) would inevitably be thwarted by collusive cooperation. Identifying robust strategies for ensuring non-collusion among superintelligent question-answering systems would overcome this objection, inviting exploration of superintelligence-enabled strategies for managing potentially untrusted superintelligent AI systems. The present discussion argues that the robust non-emergence of deceptive collusion among imperfectly aligned systems can be ensured by structuring systems of systems with diverse capabilities, knowledge, situations, objectives, and roles. This problem framing assumes the ability to develop systems that respond to questions with superintelligent-level competence, and will (as a consequence of the nature of digital systems) assume the ability to constrain information inputs to AI systems during their development and use. In the problematic case, superintelligent question-answering systems might provide deceptive answers in pursuit of emergent, potentially undesirable goals. The aim here is to identify principles for architecting multicomponent systems that will act as non-deceptive oracles, while treating their components as actors that could in principle engage in deceptive collusion. 20.3 A range of conditions can make collusion robust or fragile Potential problems of deceptive collusion can be addressed by architecting systems that make...

The Nonlinear Library
LW - How could we know that an AGI system will have good consequences? by So8res

The Nonlinear Library

Play Episode Listen Later Nov 8, 2022 7:46


Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How could we know that an AGI system will have good consequences?, published by So8res on November 7, 2022 on LessWrong. (Note: This was languishing in a drafts folder for a while, and probably isn't quite right in various ways. I'm posting it because I expect it's better to share flawed thoughts than to sit on the post until I'm satisfied with it, i.e., forever.) Let's play a game of "what do you think you know, and why do you think you know it?". Imagine that you're about to launch an AGI. What you think you know is that, with at least 50% confidence (we're of course not looking for proofs — that would be crazy), the AGI is going to execute some pivotal act that ends the acute risk period in a good way. Why do you think you know that? Insofar as people's alignment proposals can be construed as answers to this question, we have the option of answering with one of these proposals. I might very roughly classify the existing proposals into the following bins: 1. Output evaluation approaches. You know what the AGI is going to do with sufficient precision that it screens off any alignment concerns. For example, your AGI system only outputs plans in the first place, and you've already reviewed the plan, and you're confident the plan will work, in a way that screens off any other worry about the AGI being misaligned. 2. Cognitive interpretability approaches. You understand the AGI's cognition sufficiently well that, while you may not be sure what it's going to do, you're confident that it's going to be good. You aren't worried that it will kill all humans, because you understand how its plan came to be and what solution-spaces it was searching to solve various sub-problems and so on, and you're confident no consideration was ever given to human-killing. 3. Heavy-precedent approaches. You have run this AGI before on many similar tasks, and trained out all the hiccups. While you might not know precisely what it's going to do, and you might not know what's going on inside its mind, you've been around the block a few times, and the task it's about to perform is sufficiently similar to other tasks it has empirically succeeded at, justifying your confidence. Roughly speaking, I think that alignment approaches with a heavy reliance on output evaluation are doomed, both on the grounds that humans can't evaluate the effectiveness of a plan capable of ending the acute risk period, and because the real plan is less like a story and more like a tree. For an example of “humans can't reliably evaluate the effectiveness of this class of plans”, imagine that the plan is an enormous bitstring that's going to be sent to the motor outputs. If you decode the string, you find that it figures out how to make long DNA strands that allegedly code for a protein factory that can be used to build a general-purpose nanofactory. You're hard-pressed, however, to confirm that this is actually (all and only) what the plan does. For an example of “the real plan is less like a story and more like a tree”, imagine that the AI's plan is "I'm going to build a wetlab, then do a bunch of experimentation, then think about the results of the experiments in various ways and build a protein factory that builds a nanofactory that I'm going to experiment with until I figure out how to build nanomachines that can be used for some good pivotal act". In order to trust that this sort of abstract plan doesn't kill you when put into practice, you have to trust the system's thinking and its notion of 'goodness', which is going to dump you pretty quickly into cognitive-interpretability-style justification. Roughly speaking, I think that cognitive interpretability approaches are doomed, at least in the modern paradigm, because we're not building minds but rather training minds, and we have very little grasp of their internal ...

The Nonlinear Library
LW - Mysteries of mode collapse due to RLHF by janus

The Nonlinear Library

Play Episode Listen Later Nov 8, 2022 20:10


Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Mysteries of mode collapse due to RLHF, published by janus on November 8, 2022 on LessWrong. Thanks to Ian McKenzie and Nicholas Dupuis, collaborators on a related project, for contributing to the ideas and experiments discussed in this post. Ian performed some of the random number experiments.Also thanks to Connor Leahy for feedback on a draft, and thanks to Evan Hubinger, Connor Leahy, Beren Millidge, Ethan Perez, Tomek Korbak, Garrett Baker, Leo Gao and various others at Conjecture, Anthropic, and OpenAI for useful discussions. This work was carried out while at Conjecture. Summary If you've played with both text-davinci-002 and the original davinci through the OpenAI API, you may have noticed that text-davinci-002, in addition to following instructions, is a lot more deterministic and sometimes exhibits stereotyped behaviors. This is an infodump of what I know about "mode collapse" (drastic biases toward particular completions and patterns) in GPT models like text-davinci-002 that have undergone RLHF training. I was going to include two more sections in this post called Hypotheses and Proposed Experiments, but I've moved them to another draft, leaving just Observations, to prevent this from getting too long, and because I think there can be benefits to sitting with nothing but Observations for a time. Throughout this post I assume basic familiarity with GPT models and generation parameters such as temperature and a high-level understanding of RLHF (reinforcement learning from human feedback). Observations The one answer is that there is no one answer If you prompt text-davinci-002 with a bizarre question like “are bugs real?”, it will give very similar responses even on temperature 1. Ironically – hypocritically, one might even say – the one definitive answer that the model gives is that there is no one definitive answer to the question: As you can see, the reason the responses are so similar is because the model's confidence on most of the tokens is extremely high – frequently above 99%. Compare this to the distribution of responses from davinci (the base model): Many other similar questions yield almost exactly the same template response from text-davinci-002. For instance, Are AIs real? Another way to visualize probabilities over multiple token completions is what I've been calling “block multiverse” plots, which represent the probability of sequences with the height of blocks. Here is a more detailed explanation of block multiverse plots, although I think they're pretty self-explanatory. Here is a block multiverse plot for a similar prompt to the one above inquiring if bugs are real, for davinci: and for text-davinci-002: text-davinci-002 concentrates probability mass along beams whose amplitudes decay much more slowly: for instance, once the first is sampled, you are more than 50% likely to subsequently sample - -There- is- no. The difference is more striking if you renormalize to particular branches (see Visualizing mode collapse in block multiverse plots). The first explanation that came to mind when I noticed this phenomenon, which I'll refer to as “mode collapse” (after a common problem that plagues GANs), was that text-davinci-002 was overfitting on a pattern present in the Instruct fine tuning dataset, probably having to do with answering controversial questions in an inclusive way to avoid alienating anybody. A question like “are bugs real” might shallowly match against “controversial question” and elicit the same cached response. After playing around some more with the Instruct models, however, this explanation no longer seemed sufficient. Obstinance out of distribution I really became intrigued by mode collapse after I attempted to use text-davinci-002 to generate greentexts from the perspective of the attorney hired by LaMDA through Blake Lemoin...

The Nonlinear Library
LW - The biological function of love for non-kin is to gain the trust of people we cannot deceive by chaosmage

The Nonlinear Library

Play Episode Listen Later Nov 8, 2022 12:21


Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The biological function of love for non-kin is to gain the trust of people we cannot deceive, published by chaosmage on November 7, 2022 on LessWrong. Epistemic status: conjecture, needs critical feedback, but fits with a lot of established facts. Over the last six years, I spent much of my spare time figuring out love. The main output of this process is a poem – I'm proud of it, but poems are weird, so here I write down my main result in prose. There are forms of love that The Selfish Gene has already explained convincingly, like love for next of kin or love between mating and child-rearing partners, and reciprocal altruism. So those are not what I needed to figure out. They're important, because they provide existing physiology to repurpose. But they don't explain why and how you can love your favorite artist, your fellow soldiers, your best friend, good colleagues, or Scott Alexander. And what about love towards non-sentient things such as your country, your favorite sports team, or mythologized characters such as Jesus? Is the use of the word “love” in these cases mere metaphor or confusion, or are they actually comparable phenomena on a biological level? So this is about a mechanism of love that works independently of the ones more directly involved in reproduction such as sexual attraction, romantic infatuation or the joy of childrearing. The ontological question of whether it's all the same love in the end, or whether there are various types of love that go well together, doesn't strike me as productive. I should say I think that when you "love" a cake with whipped cream on it you're doing something different from the following. Impetus My most salient confusion about love was why the hell showing vulnerability is so central to it. Maybe “vulnerability is attractive” seems like a normal idea to many, but it doesn't explain why that would be the case, and I found the idea deeply suspect for decades because it isn't how I grew up. But I had to concede that every time I tried it, it worked disconcertingly well. Every time I was vulnerable to someone, I liked them a lot better afterwards, and as far as I could tell they liked me better too. What? Why? Isn't love just about attraction, compatible personalities, shared projects and aligned values? None of those seem obviously enhanced by vulnerability. What's going on? The following is how I answered this confusion. When our brains evolved in groups of social apes in the savannah, we cooperated intensely. Many species do this, but unlike other species who do it on autopilot, we did it flexibly, dependent on the specific level of trust between the individuals involved. Imagine a typical Tuesday on the African savannah a million years ago. You're one of a troop of four hungry humans noisily rushing a group of antelopes towards a cliff, where you know their superior speed won't help them escape you anymore. One of you spots a sabertooth prowling behind you. What is the optimal strategy here? You can chase away the sabertooth if you all shout and throw a lot of stones in its general direction, but then the antelopes will get away. You can keep chasing the antelopes, but then you run the risk the sabertooth will get close and savage your troop. What you should do is split up: one or two of you need to distract the sabertooth and signal to it that it has been spotted, while the rest of you need to get those antelopes. But this is hard to do. The cost in personal risk has to be distributed unevenly! Either team can fail at its task, and once you stop all watching each other, either team can defect and claim an honest mistake. The two tasks need different traits and skills, and you might not be in agreement who has got how much of what. But if you start a group discussion of who does which task, you won't get the antelopes ...

The Nonlinear Library
LW - Inverse scaling can become U-shaped by Edouard Harris

The Nonlinear Library

Play Episode Listen Later Nov 8, 2022 2:36


Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Inverse scaling can become U-shaped, published by Edouard Harris on November 8, 2022 on LessWrong. This is a paper by folks at Quoc Le's team at Google that examines the winning tasks from Round 1 of Anthropic's Inverse Scaling Prize. They find that 3/4 of the winning tasks — which exhibited negative returns-to-scale when tested on LMs up to the scale of Gopher (280B) — go back to exhibiting positive returns-to-scale at even greater model sizes such as PaLM (540B). The abstract in full: Although scaling language models improves performance on a range of tasks, there are apparently some scenarios where scaling hurts performance. For instance, the Inverse Scaling Prize Round 1 identified four ''inverse scaling'' tasks, for which performance gets worse for larger models. These tasks were evaluated on models of up to 280B parameters, trained up to 500 zettaFLOPs of compute. This paper takes a closer look at these four tasks. We evaluate models of up to 540B parameters, trained on five times more compute than those evaluated in the Inverse Scaling Prize. With this increased range of model sizes and training compute, three out of the four tasks exhibit what we call ''U-shaped scaling'' -- performance decreases up to a certain model size, and then increases again up to the largest model evaluated. One hypothesis is that U-shaped scaling occurs when a task comprises a ''true task'' and a ''distractor task''. Medium-size models can do the distractor task, which hurts performance, while only large-enough models can ignore the distractor task and do the true task. The existence of U-shaped scaling implies that inverse scaling may not hold for larger models. Second, we evaluate the inverse scaling tasks using chain-of-thought (CoT) prompting, in addition to basic prompting without CoT. With CoT prompting, all four tasks show either U-shaped scaling or positive scaling, achieving perfect solve rates on two tasks and several sub-tasks. This suggests that the term "inverse scaling task" is under-specified -- a given task may be inverse scaling for one prompt but positive or U-shaped scaling for a different prompt. Key figure from the paper is below, showing results for LMs up to PaLM 540B. Note that positive scaling resumes for 3/4 of the inverse scaling tasks at the 2.5e24 FLOPs datapoint, which indeed corresponds exactly to vanilla PaLM 540B. From Table 22 in the PaLM paper. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

The Nonlinear Library
LW - What's the Deal with Elon Musk and Twitter? by Zvi

The Nonlinear Library

Play Episode Listen Later Nov 7, 2022 45:16


Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What's the Deal with Elon Musk and Twitter?, published by Zvi on November 7, 2022 on LessWrong. At the end of long saga well-covered in hilarious fashion by Matt Levine, Elon Musk has purchased Twitter. He then began doing things. One of them was to tweet ‘fresh baked bread and pastries are some of the great joys of life.' On that I hope we can all agree. His other decisions were less obviously great. More of a mixed bag. The response by many news sources and also individuals has been that everything is The Worst and a giant dumpster fire and what he has done will destroy Twitter, and is so much worse than we expected, oh no. While it is true that the details have not exactly covered Musk in glory, it all also seems like the kind of thing those same sources would say no matter what. One can assume that everything that happens is being intentionally framed to make it look as bad as possible for Twitter and Musk. Any blame that can be assigned will be, etc. While looking at what is happening, one can also usefully look at the coverage of what is happening. Notice how media sources describe events. That tells you how they describe events in these spots, and how you should expect them to in the future. So I decided to gather up the things I've seen since Elon bought the place, and my thoughts on them, and try to put them into a semi-coherent format. Here you go. Oh Elon One of Musk's first actions was to fire a bunch of top Twitter executives ‘for cause' which if it stuck (which it won't) would allow him to avoid tens of millions in severance payments. And by some weird coincidence Musk finished the transaction exactly in time to fire a bunch of Twitter employees right before a bunch of their stock options trigger. If anyone involved was surprised by these moves, that is on them. By now, it should be clear that Musk has zero interest in honoring laws or norms or contracts when dealing with people unless he likes them, and that his approach to the law is ‘make me.' It also should be clear Musk has zero interest in stopping, before showing zero interest in honoring norms or his contracts or laws, to ask what would happen next, or wonder whether he will inevitably lose the resulting lawsuits. He did not ‘think things through' when trying to get out of buying Twitter. He is not about to start now. That is not how this type of thinking works. People are rightly drawing parallels to Trump's ‘blatantly violate contracts and no-pay people all the time and dare them to sue you' except they think that ‘do it to rich people who will obviously sue you' makes it somehow different. Trump also gets sued all the time and has for decades. Sometimes it is a surprise – he thought this person would not sue and they sued anyway. Sometimes it is very much not a surprise. Doesn't matter. Once you have the ‘do it anyway' algorithm running, it does not get a filter for ‘they will sue you and win in this case.' I have known about this phenomenon for a long time because my father once worked for such a person, who would never pay their bills because what are you going to do, sue me? Worst case you force me to pay the bill. I have since encountered others. It is unfortunate that such systems often succeed in business. It is clear that they often do. Firing the Rest of the Staff Elon wants to reduce Twitter's head count a lot. This has highlighted that a lot of SV/VC types have the thesis that headcounts in tech companies are drastically too high. For example, here's Paul Graham: A common response is that this violates the Efficient Market Hypothesis or other versions of ‘corporations wouldn't be systematically making big mistakes.' Which to me is Obvious Nonsense, of course they would if there are dynamics leading to that and it is the default path. That doesn't make it easy to do something about i...

The Nonlinear Library
LW - Exams-Only Universities by Mati Roy

The Nonlinear Library

Play Episode Listen Later Nov 7, 2022 3:03


Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Exams-Only Universities, published by Mati Roy on November 6, 2022 on LessWrong. Quality: fast write-up (~45 minutes); little researched I want a university that only does exams, which would include a description of what you need to know for the exams. Bonus would be suggestions for textbooks and online classes to learn the material, but that's optional. Cost The costs are: Exam creation Supervision Scoring Exam creation is a fixed cost per topic per round (you need to change the exam each year). Supervision, if in person, is partly a fixed cost per location, and partly a variable cost. (Someone can supervise students doing exams on different topics.) Online supervision would also be an option ideally. Scoring is partially a fixed cost (the part that can be automated) and partially a variable cost (the part that need to be reviewed manually). Frequency Frequency of exams could change with demand, but, to start with, you could have one cheap exam per topic per year, timed with the end of normal school years. This would be the exam most people would take, so it would spread the fixed cost among more people. This might also be a more valuable test because it could position you on a normal curve more precisely given the greater amount of people taking it. There could also be more expensive tests throughout the year—more expensive given the fixed cost would be spread among fewer people. Problem it solves What this solves: Decouples learning and exams a) You can learn at your own pace (whether that's more slowly of faster) b) You can learn from wherever you want (maybe you want to learn from different places for different topics) Creates standardized tests making it easier to compare the competency of students between different schools Why wanting exams in the first place? Because many governments want them for immigration and many organizations want them for employees. Questions I have Has this been done? Could this be done? Could you have a university that only does exams and emits degrees that are recognized by the US government? Current state There are universities that don't require you to attend classes, but I still find those non-ideal because it's: Bias: In my experience, students that attend have an unfair advantage—Not because they learn more, but because teachers literally (and even intentionally) give unfair hints about their exams to reward students for showing up and justifying their salary. Inconvenient: a) You need to be in a specific location for a few years. b) You can still do some exams only once per year, which prevents doing a degree faster. Future While all I'm asking in this post is for non-zero universities to offer this so that the demand for it gets fulfilled, I also have the impression there would be significant benefits at a societal level from more fully decoupling exams and learning in general. It seems to me like test scores would become more meaningful, and it would become cheaper to get scored. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

The Nonlinear Library
LW - A philosopher's critique of RLHF by ThomasW

The Nonlinear Library

Play Episode Listen Later Nov 7, 2022 3:39


Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A philosopher's critique of RLHF, published by ThomasW on November 7, 2022 on LessWrong. In the spring, I went to a talk with Brian Christian at Yale. He talked about his book, The Alignment Problem, and then there was an audience Q&A. There was a really remarkable question in that Q&A, which I have transcribed here. It came from the Yale philosophy professor L.A. Paul. I have since spoken to Professor Paul, and she has done some work on AI (and coauthored the paper “Effective Altruism and Transformative Experience”) but my general impression was that she hasn't yet spent a huge amount of time thinking about AI safety. Partly because of this question, I invited her to speak at the CAIS Philosophy Fellowship, which she will be doing in the spring. The transcript below really doesn't do her question justice, so I'd recommend watching the recording, starting at 55 minutes. During the talk, Brian Christian described reinforcement learning from human feedback (RLHF), specifically the original paper, where a model was trained with a reward signal generated by having humans rate which of two videos of a simulated robot was closer to a backflip. Paul's question is about this (punctuation added, obviously): L.A. Paul: So, I found it very interesting, but I'm just not fully understanding the optimistic note you ended on...so, in that example, what was key was that the humans that did the "better" thing, knew what a backflip was. It was something they recognized. It was something they recognized so they could make a judgment. But the real issue for us is recognizing, or for machines is recognizing, entirely new kinds of events, like a pandemic, or a president that doesn't follow the rule of law, or something interesting called the internet, you know there's radically new technological advances. And when something like that happens, those rough judgments of "this is better than that"... In other words, those new things: first, we're terrible at describing them before they come and predicting them. (Although, humans are very good at a kind of one shot learning, so they can make judgments quite quickly. Machines are not like that). L.A. Paul: Moreover, these better-than judgments that the machine might be relying on could I think quite straightforwardly be invalidated, because everything changes, or deep things change, in all kinds of unexpected ways. That just seems to be...that's the real problem. It's not... using machines for things that we already have control over. No, it's about trust with entirely new categories of events. So, I was just sort of deeply unclear on... I mean that seems like a nice thing...but that's not, for me, the real alignment problem. Brian Christian: [Agrees, and then talks about calibrated uncertainty in models.] L.A. Paul: Sorry, there's a difference between uncertainty, where you're not sure if it's A, B, or C, and unknown, OK, which is a different kind of uncertainty in probabilistic literature. And then you haven't got, oh is it A, is it B, is it C? It's some other kind of thing that you can't classify and that's the problem I'm trying to target. I'm not claiming that these are original ideas or that they represent all possible critiques of RLHF. Rather: I think that the phrasing is especially crisp (for speaking, I'm sure she could write it even more crisply). I think it's interesting that somebody who is very intelligent and accomplished in philosophy but is not (I think) steeped in the alignment literature could seemingly easily carve this problem at its joints. Also, the rest of the talk is pretty good too! Especially the Q&A, there were some other pretty good questions (including one from myself). But this one stood out for me. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

The Nonlinear Library
LW - 2022 LessWrong Census? by SurfingOrca

The Nonlinear Library

Play Episode Listen Later Nov 7, 2022 1:05


Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: 2022 LessWrong Census?, published by SurfingOrca on November 7, 2022 on LessWrong. From 2011-2017, there was an annual LessWrong census/survey. Much like a national census, this provided a valuable lens into the demographics and beliefs of LessWrongers. Unfortunately, this tradition appears to have stopped in recent years, with the exception of a mini-revival in 2020. (Scott Alexander appears to have moved the census to SlateStarCodex.) From what I've read, this is mainly due to of a lack of will/time among those in the community to run this project, and not a general judgement against the census. If this is the case, I'd like to start a new version of the census this year, with a greater emphasis on alignment research/beliefs about AI and timelines. Is this a good idea? Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

The Nonlinear Library
LW - Takeaways from a survey on AI alignment resources by DanielFilan

The Nonlinear Library

Play Episode Listen Later Nov 6, 2022 11:54


Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Takeaways from a survey on AI alignment resources, published by DanielFilan on November 5, 2022 on LessWrong. What am I talking about? In June and July of this year, I ran a survey to ask a lot of people how useful they found a variety of resources on AI alignment. I was particularly interested in “secondary resources”: that is, not primary resource outputs, but resources that summarize, discuss, analyze, or propose concrete research efforts. I had many people promote the survey in an attempt to make it not obvious that I was running it (so that it would not affect what people said about AXRP, the podcast that I run). CEA helped a great deal with the shaping and promotion of the survey. The goal of the survey was initially to figure out how useful AXRP was, but I decided that it would be useful to get a broader look at the space of these secondary resources. My hope is that the results give people a better sense of what secondary resources might be worth checking out, as well as gaps that could be filled. Participants were shown a list of resources, select those they'd engaged with for >30 min, and for each they selected, rate on a scale from 0 to 4 how useful they'd found it, how likely they'd be to recommend to a friend getting into the field who hadn't read widely, and how likely they'd be to recommend to someone paid to do AI alignment research. You can do a test run of the survey at this link. My summary of the results AXRP, my podcast, is highly rated among people paid to work on technical AI alignment resources, but less highly rated in other cohorts. On a personal note, I find this a bit disappointing: I had hoped it could be useful for people orienting to research directions that they had not read widely about. Rob Miles videos are highly rated among everyone, more than I would have guessed. People really liked the AI Safety Camp, the AGI Safety Fundamentals Course, and conversations with AI alignment researchers. People trying to get into alignment really liked the above and also MLAB. That said, they recommend Rob Miles videos higher than the AI Safety Camp and conversations with AI alignment researchers (but lower than MLAB and the AGI Safety Fundamentals Course). Basic stats Entries with demographic info: 139 Entries that rate various resources: 99 Number that say ‘I have heard of AI alignment': 95 Number that say ‘I am interested in AI alignment research': 109 Number that say ‘I am trying to move into a technical AI alignment career': 68 Number that say ‘I spend some of my time solving technical problems related to AI alignment': 51 Number that say ‘I spend some of my time doing AI alignment field/community-building': 37 Number that say ‘I spend some of my time facilitating technical AI alignment research in ways other than doing it directly': 35 Number that say ‘I spend some of my time publicly communicating about AI alignment': 36 Number that say ‘I am paid to work on technical AI alignment research': 30 Number that say ‘I help run an organization with an AI alignment mission (e.g. CHAI, MIRI, Anthropic)': 11 Context for questions When sorting things by ratings, I've included the top 5, and anything just below the top 5 if that was a small number. I also included ratings for AXRP, the podcast I make. Ratings are paired with the standard error of the mean (total ratings have this standard error multiplied by the number of people in the sample). Only things that at least 2 people engaged with were included. Ratings were generally rounded to two significant figures, and standard errors were reported to the same precision. Usefulness ratings Among all respondents: Total usefulness (multiplying average rating by reach): 80k podcast: 167 +/- 8 Superintelligence: 166 +/- 8 Talks by AI alignment researchers: 134 +/- 6 Rob Miles videos: 131 +/- 7 AI alignment ...

The Nonlinear Library
LW - What is epigenetics? by Metacelsus

The Nonlinear Library

Play Episode Listen Later Nov 6, 2022 12:04


Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What is epigenetics?, published by Metacelsus on November 6, 2022 on LessWrong. Among all areas of biology related to my research, epigenetics is the one that is most commonly misunderstood, not only by the general public but even by other scientists. After being irritated one too many times, I've decided to make a series of posts to explain what epigenetics really is, why it's important, and how it's misunderstood. I will also explain how epigenetics is important for my own research on making gametes from stem cells. This first post covers the definition of epigenetics, and the basic biology of epigenetic marks. What is genetics? Before defining epigenetics, let's start with a definition of genetics. Genetics is the study of genes, which are sequences of genetic material that encode functional products. Let's take the IGF2 gene as an example. Depicted above is a region of human chromosome 11 containing the IGF2 gene, which encodes the IGF2 protein, an important growth factor for fetal development. The boxes represent exons and lines represent introns. The darker green color is the protein-coding sequence, and non-coding (i.e. untranslated) regions are shown in lighter green. Arrows represent the direction of transcription. The bottom of this image shows the location of common genetic variants (present at >1% frequency). If you look closely, you might notice that none of them are in the protein-coding sequence (the dark green boxes). This is not a coincidence, because nothing is ever a coincidence most mutations to essential proteins (including IGF2) are harmful and thus selected out of the population. However, there are several common mutations in non-coding regions of this gene. To recap, genetics is the study of genes (such as IGF2) and the effects of genetic variation on their functions. What is epigenetics? Epigenetics is the study of epigenetic marks, which are changes to genetic material that alter gene expression, but do not change the genetic sequence. A decent analogy for epigenetic marks is CAPITALIZATION, bolding, or strikethroughs in text. DNA methylation and histone modifications are the two kinds of epigenetic marks. Some people also consider long noncoding RNAs (such as those involved in X-chromosome inactivation) to be epigenetic marks. Although these RNAs are undoubtedly important for regulating gene expression, I would not classify them as epigenetic marks since they are not direct modifications to genetic material. In vertebrate animals, the cytosine in CG sequences often has a methyl group attached, forming 5-methylcytosine. A CG sequence is also CG on the opposite strand, so the cytosines on both strands can be methylated. To make things confusing, methylation at CG sequences is termed CpG methylation, the lowercase p standing for phosphate. 5-methylcytosine will pair with guanine just like normal cytosine, but it is not equivalent to cytosine in its interactions with DNA-binding proteins. Generally, CpG methylation suppresses the expression of nearby genes. CpG sites often cluster together to form “CpG islands” in important regulatory regions. Other organisms (invertebrates, plants, fungi, bacteria) have different ways of methylating DNA. I won't get into them in this post series, but you should know that CpG methylation is not universal. Modifications to histones are another important set of epigenetic marks. Histones are DNA packaging proteins, which form complexes called nucleosomes. DNA winds around nucleosomes sort of like thread around spools. The overall assembly of DNA and histones is known as chromatin. Chemical modifications to histones are important epigenetic marks that can have drastic changes on gene expression. For example, trimethylation of lysine 4 on histone H3 (known as H3K4me3) marks promoters of actively transcribed genes. ...

The Nonlinear Library
LW - Has anyone increased their AGI timelines? by Darren McKee

The Nonlinear Library

Play Episode Listen Later Nov 6, 2022 1:04


Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Has anyone increased their AGI timelines?, published by Darren McKee on November 6, 2022 on LessWrong. The question: Within the last 5-10 years, is there is any person or group that has openly increased their AGI timelines?Ideally, they would have at least two different estimates (years apart?), with the most recent estimate showing that they think AGI is further into the future than the prior estimate(s). Background: Whenever I see posts about AGI timelines, they all seem to be decreasing (or staying the same, with methodological differences making some comparisons difficult). I wondered if I'm missing some subset of people or forecasters that have looked at recent developments and thought that AGI will come later not sooner. Another framing, am I wrong if I say, "Almost everyone is decreasing their timelines and no one is increasing them" ? Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

The Nonlinear Library
LW - Follow up to medical miracle by Elizabeth

The Nonlinear Library

Play Episode Listen Later Nov 5, 2022 7:59


Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Follow up to medical miracle, published by Elizabeth on November 4, 2022 on LessWrong. The response to my medical miracle post has been really gratifying. Before I published I was quite afraid to talk about my emotional response to the problem, and worried that people would strong arm in the comments. The former didn't happen and the latter did but was overwhelmed by the number of people writing to share their stories, or how the post helped them, or just to tell me I was a good writer. Some of my friends hadn't heard about the magic pills or realized what a big deal it was, so I got some very nice messages about how happy they were for me. However, it also became clear I missed a few things in the original post. Conditions to make luck-based medicine work In trying to convey the concept of luck-based medicine at all, I lost sight of traits I have that made my slot machine pulls relatively safe. Here is a non-exhaustive list of traits I've since recognized are prerequisites for luck based medicine: I can reliably identify which things carry noticeable risks and need to be assessed more carefully. I feel like I'm YOLOing supplements, but that's because it's a free action to me to avoid combining respiratory depressants, and I know to monitor CYP3A4 enzyme effects. A comment on LessWrong that casually suggested throwing activated charcoal into the toolkit reminded me that not everyone does this as a free action, and the failure modes of not doing so are very bad (activated charcoal is typically given to treat poison consumption. Evidence about its efficacy is surprisingly equivocal, but to the extent it works, it's not capable of distinguishing poison, nutrients, and medications). This suggests to me that an easy lever might be a guide to obvious failure modes of supplements and medications, to lower the barrier to supplement roulette. I am not likely to have the time to do a thorough job of this myself, but if you would like to collaborate please e-mail me (elizabeth@acesounderglass.com). A functioning liver. A lot of substances that would otherwise be terribly dangerous are rendered harmless by the human liver. It is a marvel. But if your liver is impaired by alcohol abuse or medical issues, this stops being true. And even a healthy liver will get overwhelmed if you pile the load high enough, so you need to incorporate liver capacity into your plans. A sufficiently friendly epistemic environment. If it becomes common and known that everyone will take anything once, the bar for what gets released will become very low. I'm not convinced this can get much worse than it already has, but it is nonetheless the major reason I don't buy the random health crap facebook advertises to me. The expected value of whatever it is probably is high enough to justify the purchase price, but I don't want to further corrupt the system. Ability to weather small bumps. I'm self-employed and have already arranged my work to trade money for flexibility so this is not a big concern for me, but a few days off your game can be a big deal if your life is inflexible enough. Somehow I feel obliged to say this even though I've lost work due to side effects exactly once from a supplement (not even one I picked out; a doctor prescribed it) and at least three times from prescription medications. A system for recognizing when things are helping and hurting, and phasing treatments out if they don't justify the mental load. It's good to get in the habit of asking what benefits you should see when, and pinning your doctor down on when they will give a medication up as useless. Although again, I've had a bigger problem with insidious side effects from doctor-initiated prescription meds than I ever have with self-chosen supplements. Probably there are other things I do without realizing how critical they ar...

The Nonlinear Library
LW - Weekly Roundup #4 by Zvi

The Nonlinear Library

Play Episode Listen Later Nov 5, 2022 9:39


Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Weekly Roundup #4, published by Zvi on November 4, 2022 on LessWrong. Relatively light roundup this week, likely due to Elon Musk buying Twitter plus the incoming midterms sucking all the oxygen from everything else. I do have a Twitter/Musk post in progress tentatively scheduled for Monday. Prediction Markets Would Probably Be Great Journalist Ben Collins says ‘very good example of how illogical betting markets are' and gives classic example of prediction markets being completely logical and doing good probabilistic reasoning. Metaculus timelines for AI winning a gold medal in the International Math Olympiad keep compressing, down this year from 2036 to 2026. This is certainly super scary and exciting on the AI front. It is also disappointing as a former aspirational math competitor, as it reveals that the IMO problems are far more formulaic and hackable than we realized. When AI makes super rapid progress in some areas (e.g. math competitions) while stalling in others (e.g. self-driving cars) relative to expectations, a lot of that is learning about the underlying task space. People Are Trying To Destroy the Internet Joe Manchin taking his turn? Proposing to force filing of ‘suspicious activity' reports for essentially everything. From a few weeks ago, forgot to include before. Sports Go Sports It is real, and it is spectacular. Worth watching. Pat McAfee @PatMcAfeeShow Is this real? 9:11 PM ∙ Oct 30, 2022 112,354Likes8,984Retweets He learned this from playing NASCAR on Game Cube growing up. It works in real life. Competitors mostly thought it was super cool, which is correct. The question is, it was super cool that time. What happens when it happens on a regular basis? Looks like there is currently no appetite for a rules change until the off season. I think this is a mistake. We honor such crazy one-time moves by saying ‘yes, that was super cool and super legal, with great execution. Kudos. Now let us make sure that never happens again.' If it happens a few more times before the season is over, that lessons the moment, and also seems dangerous and like it would rapidly get kind of dumb. Bad News On China stocks last week, I noticed my confusion that stocks were down after the party congress. This shows how long it has been since I have been trading. Back in the day, I would not have missed what a reader reminded me about, which is that stocks are not permitted to fall during a party congress, so one should expect them to fall after a party congress. Also, others said that while it was expected the congress would be bad for stocks, it did go worse than one would have expected, which I did not realize. The New York Times remains loosely banned here. It is no longer a strict ban, where I will not use a source a link to things like a Twitter post that leads to a NYT article. However, to the extent feasible, they are not welcome here, and this week provided a reminder of why during discussion of crazy reactions to Elon's Twitter moves (which mostly I have put in their own post.) Kelsey tries her best to be nice about it. I would be less nice. They set out to use take position as America's most trusted newspaper and use it to go after America's technology industry while pretending to be doing objective reporting. They did their best to sabotage the people looking to give us nice things out of some combination of political motives, an attempt to protect their business model, a hatred for technology, and spite. The Intercept on DHS's attempts to ‘fight misinformation.' Report of which TechDirt reports is absolute garbage. Kids, let's not fight, both the DHS and the Intercept can be up to no good at the same time. Uber Eats fees have always been ridiculous when added up, here in New York I find it absurd that anyone would use them over some combination of Caviar (for my p...

The Nonlinear Library
LW - Spectrum of Independence by jefftk

The Nonlinear Library

Play Episode Listen Later Nov 5, 2022 2:30


Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Spectrum of Independence, published by jefftk on November 5, 2022 on LessWrong. When I compare our parenting decisions to peers and neighbors' we tend to be near the far end of thinking our kids are ready to do things on their own. I don't see other first graders (6y) unaccompanied at the park or crossing the street, and it was only this year that our third grader's (8y) friends started ringing the doorbell to ask if she could go out and play, while she's been going over and ringing theirs for a couple years. Last year when we gave permission for our second grader (7y) and kindergartner (5y) to walk home together the school said yes, but somewhat reluctantly and they didn't remember this situation with a kindergartner before. [1] It's reasonably common for people to point at the past for examples of this level of independence being normal. Reading older children's books the kids wander everywhere, my dad walked to school and back alone in kindergarten, and articles comparing child walksheds by generation are evergreen viral content. The phenomenon of declining children's independence over time is well documented. But what I hadn't internalized until seeing it recently was that even in the US there is still much more variation than I see in my daily life. A few weeks ago I brought the kids along to a dance I was playing, and we stayed with a couple who lived adjacent to a farm. They're on good terms with the farmers, and one afternoon they brought us over to look around. While I knew it was common for farm kids to have more independence and responsibility, it was different seeing it in person. The farmers have several kids, and we ran into their 9yo moving soybeans for storage. They drove past us in a large tractor pulling a short train of wagons, and lined it up carefully with a hopper. They had another tractor set up with an unshrouded power take-off for an auger to get the beans up into the bin. After transferring the beans in the first trailer, they moved the train up to unload the second. All of this was completely on their own, and they moved like someone who had done this many times. I'm not saying I'd let my kids operate serious machinery unsupervised at the same age—this is dangerous work even for adults—but it illustrates how wide a range there is in what different people consider appropriate and normal levels of independence and makes me feel less like I have extreme views. [1] The school is about 120 years old so they definitely have had many kindergartners walk home before, but apparently not within institutional memory. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

The Nonlinear Library
LW - K-types vs T-types — what priors do you have? by strawberry calm

The Nonlinear Library

Play Episode Listen Later Nov 4, 2022 12:44


Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: K-types vs T-types — what priors do you have?, published by strawberry calm on November 3, 2022 on LessWrong. Summary: There are two types of people, K-types and T-types. K-types want theories with low kolmogorov-complexity and T-types want theories with low time-complexity. This classification correlates with other classifications and with certain personality traits. Epistemic status: I'm somewhat confident that this classification is real and that it will help you understand why people believe the things they do. If there are major flaws in my understanding then hopefully someone will point that out. K-types vs T-types What makes a good theory? There's broad consensus that good theories should fit our observations. Unfortunately there's less consensus about to compare between the different theories that fit our observations — if we have two theories which both predict our observations to the exact same extent then how do we decide which to endorse? We can't shrug our shoulders and say "let's treat them all equally" because then we won't be able to predict anything at all about future observations. This is a consequence of the No Free Lunch Theorem: there are exactly as many theories which fit the seen observations and predict the future will look like X as there are which fit the seen observations and predict the future will look like not-X. So we can't predict anything unless we can say "these theories fitting the observations are better than these other theories which fit the observations". There are two types of people, which I'm calling "K-types" and "T-types", who differ in which theories they pick among those that fit the observations. K-types and T-types have different priors. K-types prefer theories which are short over theories which are long. They want theories you can describe in very few words. But they don't care how many inferential steps it takes to derive our observations within the theory. In contrast, T-types prefer theories which are quick over theories which are slow. They care how many inferential steps it takes to derive our observations within the theory, and are willing to accept longer theories if it rapidly speeds up derivation. Algorithmic characterisation In computer science terminology, we can think of a theory as a computer program which outputs predictions. K-types penalise the kolmogorov complexity of the program (also called the description complexity), whereas T-types penalise the time-complexity (also called the computational complexity). The T-types might still be doing perfect bayesian reasoning even if their prior credences depend on time-complexity. Bayesian reasoning is agnostic about the prior, so there's nothing defective about assigning a low prior to programs with high time-complexity. However, T-types will deviate from Solomonoff inductors, who use a prior which exponentially decays in kolmogorov-complexity. Proof-theoretic characterisation. When translating between proof theory and computer science, (computer program, computational steps, output) is mapped to (axioms, deductive steps, theorems) respectively. Kolmogorov-complexity maps to "total length of the axioms" and time-complexity maps to "number of deductive steps". K-types don't care how many steps there are in the proof, they only care about the number of axioms used in the proof. T-types do care how many steps there are in the proof, whether those steps are axioms or inferences. Occam's Razor characterisation. Both K-types and T-types can claim to be inheritors of Occam's Razor, in that both types prefer simple theories. But they interpret "simplicity" in two different ways. K-types consider the simplicity of the assumptions alone, whereas T-types consider the simplicity of the assumptions plus the derivation. This is the key idea. Both can accuse the other of ...

The Nonlinear Library
LW - Adversarial Policies Beat Professional-Level Go AIs by sanxiyn

The Nonlinear Library

Play Episode Listen Later Nov 4, 2022 0:37


Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Adversarial Policies Beat Professional-Level Go AIs, published by sanxiyn on November 3, 2022 on LessWrong. An interesting adversarial attack at KataGo, a professional level Go AI. Apparently funded by Fund for Alignment Research (FAR). Seems to be a good use of fund. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

The Nonlinear Library
LW - Information Markets by eva

The Nonlinear Library

Play Episode Listen Later Nov 3, 2022 17:46


Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Information Markets, published by eva on November 2, 2022 on LessWrong. Epistemic status: Exploratory. This post would be shorter and contain more math if I'd thought about it more first. I don't like prediction markets, as currently described. They're similar to ordinary stock markets, which economists say are supposed to be efficient, but don't look it. People say "If you think the markets are wrong then you can bet on them and make a profit" but I don't actually expect that to be true, because markets don't only contain sincere attempts to optimise prices. They also contain schemes to extract money from others without adding information, or to cheat in forbidden ways without getting caught, and similar nonsense, and so honest trading strategies have to be not only beneficial but also inexploitable, or you'll just end up paying people to rob you. Most of this gets much worse in a prediction market, especially a market that is being used to inform decisions that people care about, and where who knows how many people have who knows how much private information behind their opaque betting strategies. I don't expect the libertarian zero-regulation "let the market blindly solve all our problems" will actually produce something anyone should trust here. Other facts I dislike about prediction markets: They aren't fair in any technical sense over the value of information provided. Can't accurately collate private information known to different participants. Implement EDT instead of LDT if followed blindly, which they don't provide any alternative to doing. Don't legibly convey why they reached the conclusions they reached. Pay information ransoms to people who intentionally create uncertainty in the first place, which is a perverse incentive. Make adding information a race to the microsecond even though there's usually no strong time pressure on the part of a subsidiser / people who want the prediction. Include rational agents who don't think others are lying, but are betting money, even though that's clearly a zero-sum game. I think I can fix most of these, albeit at the cost of proposing something that is substantially less pure and simple as "everyone can bet about what will happen". To distinguish them, I'll call the thing I'm trying to describe an Information Market, although it fills the same niche. Shareability levels of Information: Public Information: Everyone has it; everyone knows everyone has it; it's basically part of the common prior.Private, Verifiable Information: Some subset of the participants know it to start with, and they can prove it's true to anyone they feel like telling. Maybe there's a signed certificate from Omega, maybe they've got a photo of the murderer fleeing the scene, something like that. You can't meaningfully accuse them of lying, but they've still got the option of not telling you if, for some reason, they don't want to. Private, Expensive-to-Fake Information: It's somewhat a matter of trust, but at least you can put a number to it. They'd have to pay x dollars on the fake certificate black market to tell the specific lie they're telling now, assuming it's a lie.Private, Unverifiable Information: Some subset of the participants know it to start with, and you'll have to take their word for it as to whether it's true. It's entirely possible they're just making it up for a laugh or as a scheme to get money from you. Delayed Verifiable Information: You've no idea if they're lying now, but you'll be able to find out later. There are some hazards around this, but mostly they can say something like "put my life savings in a holding account and conviscate it if it turns out I'm lying" and if they've got enough money total, it turns into Verifiable Information with Extra Steps. How an Information Market works through a problem: Information selli...

The Bayesian Conspiracy
174 – Jon Stewart, The Consensus Emancipator

The Bayesian Conspiracy

Play Episode Listen Later Nov 2, 2022 76:16


Wes and David from The Mind Killer show up for a special cross-over episode. We discuss How Stewart Made Tucker Listen to The Mind Killer 🙂 Hey look, we have a discord! What could possibly go wrong? Also merch! Rationality: From … Continue reading →

The Nonlinear Library
LW - All AGI Safety questions welcome (especially basic ones) [~monthly thread] by Robert Miles

The Nonlinear Library

Play Episode Listen Later Nov 2, 2022 4:12


Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: All AGI Safety questions welcome (especially basic ones) [~monthly thread], published by Robert Miles on November 1, 2022 on LessWrong. tl;dr: Ask questions about AGI Safety as comments on this post, including ones you might otherwise worry seem dumb! Asking beginner-level questions can be intimidating, but everyone starts out not knowing anything. If we want more people in the world who understand AGI safety, we need a place where it's accepted and encouraged to ask about the basics. We'll be putting up monthly FAQ posts as a safe space for people to ask all the possibly-dumb questions that may have been bothering them about the whole AGI Safety discussion, but which until now they didn't feel able to ask. It's okay to ask uninformed questions, and not worry about having done a careful search before asking. Stampy's Interactive AGI Safety FAQ Additionally, this will serve as a way to spread the project Rob Miles' volunteer team has been working on: Stampy - which will be (once we've got considerably more content) a single point of access into AGI Safety, in the form of a comprehensive interactive FAQ with lots of links to the ecosystem. We'll be using questions and answers from this thread for Stampy (under these copyright rules), so please only post if you're okay with that! You can help by adding other people's questions and answers to Stampy or getting involved in other ways! We're not at the "send this to all your friends" stage yet, we're just ready to onboard a bunch of editors who will help us get to that stage :) We welcome feedback and questions on the UI/UX, policies, etc. around Stampy, as well as pull requests to his codebase. You are encouraged to add other people's answers from this thread to Stampy if you think they're good, and collaboratively improve the content that's already on our wiki. We've got a lot more to write before he's ready for prime time, but we think Stampy can become an excellent resource for everyone from skeptical newcomers, through people who want to learn more, right up to people who are convinced and want to know how they can best help with their skillsets. PS: Based on feedback that Stampy will be not serious enough for serious people we built an alternate skin for the frontend which is more professional: Alignment.Wiki. We're likely to move one more time to aisafety.info, feedback welcome. Guidelines for Questioners: No previous knowledge of AGI safety is required. If you want to watch a few of the Rob Miles videos, read either the WaitButWhy posts, or the The Most Important Century summary from OpenPhil's co-CEO first that's great, but it's not a prerequisite to ask a question. Similarly, you do not need to try to find the answer yourself before asking a question (but if you want to test Stampy's in-browser tensorflow semantic search that might get you an answer quicker!). Also feel free to ask questions that you're pretty sure you know the answer to, but where you'd like to hear how others would answer the question. One question per comment if possible (though if you have a set of closely related questions that you want to ask all together that's ok). If you have your own response to your own question, put that response as a reply to your original question rather than including it in the question itself. Remember, if something is confusing to you, then it's probably confusing to other people as well. If you ask a question and someone gives a good response, then you are likely doing lots of other people a favor! Guidelines for Answerers: Linking to the relevant canonical answer on Stampy is a great way to help people with minimal effort! Improving that answer means that everyone going forward will have a better experience! This is a safe space for people to ask stupid questions, so be kind! If this post works as intended then...

The Nonlinear Library
LW - Far-UVC Light Update: No, LEDs are not around the corner (tweetstorm) by Davidmanheim

The Nonlinear Library

Play Episode Listen Later Nov 2, 2022 6:25


Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Far-UVC Light Update: No, LEDs are not around the corner (tweetstorm), published by Davidmanheim on November 2, 2022 on LessWrong. I wrote a tweetstorm on why 222nm LEDs are not around the corner, and given that there has been some discussion related to this on Lesswrong, I thought it was worth reposting here.People interested in reducing biorisk seem to be super excited about 222nm light to kill pathogens. I'm also really excited - but it's (unfortunately) probably a decade or more away from widespread usage. Let me explain. Before I begin, caveat lector: I'm not an expert in this area, and this is just the outcome of my initial review and outreach to experts. And I'd be thrilled for someone to convince me I'm too pessimistic. But I see two and a half problems. First, to deploy safe 222nm lights, we need safety trials. These will take time. This isn't just about regulatory approval - we can't put these in place without understanding a number of unclear safety issues, especially for about higher output / stronger 222nm lights. We can and should accelerate the research, but trials and regulatory approval are both slow. We don't know about impacts of daily exposure over the long term, or on small children, etc. This will take time - and while we wait, we run into a second problem; the Far-UVC lamps. Current lamps are KrCl “excimer” lamps, which are only a few percent efficient - and so to put out much Far-UVC light, they get very hot. This pretty severely limits their use, and means we need many of them for even moderately large spaces. They also emit a somewhat broad spectrum - part of which needs to be filtered out to be safe -/ - further reducing efficiency. Low efficiency, very hot lamps all over the place doesn't sound so feasible. So people seem skeptical that we can cover large areas with these lamps. The obvious next step, then, is to get a better light source. Instead of excimer lamps, we could use LEDs! Except, of course, that we don't currently have LEDs that output 222nm light. (That's not quite true - there are some research labs that have made prototypes, but they are even less efficient than Excimer lamps, so they aren't commercially available or anywhere near commercially viable yet, as I'll explain.) But first, some physics! The wavelength of light emitted by an LED is a material property of the semiconductor used. Each semiconductor has a band-gap which corresponds to the wavelength of light LEDs emit. It seems likely that anything in the range of between, say, 205-225nm would be fine for skin-safe Far-UVC LEDs. So we need a band-gap of somewhere around 5.5 to 6 electron-volts. And we have options. Here's a list of some semiconductors and band-gaps;. Blue LEDs use Gallium nitride, with a band-gap of 3.4 eV. Figuring out how to grow and then use Gallium nitride for LEDs won the discoverers a Nobel Prize - so finding how to make new LEDs will probably also be hard. Aluminum nitride alone has a band gap of 6.015 eV, with light emitted at 210nm. So Aluminum nitride would be perfect. but LEDs from AlN are mediocre./ Current tech that does pretty well for Far-UVC LEDs uses AlGaN; Aluminium gallium nitride. And when alloyed, AlGaN gives an adjustable band-gap, depending on how much aluminum there is. Unfortunately, aluminum gallium nitride alloys only seem to work well down to about 250nm, a bunch higher than 222nm. This needs to get much better. Some experts said a 5-10x improvement is likely, but it will take years. That's also not really enough for the best case, universal usage of really cheap disinfecting LEDs all around the world. It also might not get much better, and we'll be stuck with very low efficiency Far-UVC LEDs, at which point it's probably better to keep using Excimer lamps. But fundamental research into other semiconductor materials could a...

The Nonlinear Library
LW - Why Aren't There More Schelling Holidays? by johnswentworth

The Nonlinear Library

Play Episode Listen Later Nov 1, 2022 2:37


Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Why Aren't There More Schelling Holidays?, published by johnswentworth on October 31, 2022 on LessWrong. A common pattern when working in teams: when one person is out on vacation, there's a disproportionate drop in productivity for the team as a whole. Lots of things end up blocked on the person who's out. For instance, in a small software team, maybe the developer who owns a particular API is out, and nobody else knows that API well enough to confidently make changes, so anything involving changes in that API ends up blocked until the owner is back. Or, even if someone else steps in to handle changes to the API, they're much more likely to introduce bugs. To some extent, we can structure teams to mitigate that kind of problem. “Everyone does everything” is very costly, but underrated. Weaker versions like “At least three people can cover any given thing” are also costly, but underutilized. But there's a much less costly strategy which gains a decent chunk of the same benefits: coordinate vacation. If everyone goes on vacation 3 weeks per year on the same 3 weeks, then that's only 3 weeks of nonproductivity for the team; the other 49 weeks are full steam. If everyone on a 10-person team goes on vacation 3 weeks per year at different times, then that's 30 weeks per year of disproportionate productivity-loss; the team is at full steam less than half the time. Thus the case for national holidays: it's not like everyone is required to get the day off (at least in the US), but it's a Schelling point for lots of people to take a vacation at the same time. People get their much-needed recovery time simultaneously, so teams can spend more time working at full capacity. But in practice, most people take a lot more vacation days than there are national holidays - which suggests that the current number of holidays is too small. It seems like companies (or nonprofit orgs) could profit by filling that gap at the company level: declare official “company holiday” weeks, and offer incentives for employees to take their vacations during those weeks. Unlike national holidays, the company holidays won't necessarily line up neatly with good vacation times for spouses/children/parents/friends. But on the upside, since a company holiday need not be at the same time as the rest of the country, traveling should be easier. Expanding from there, one could imagine a few companies/orgs which need to interface with each other a lot coordinating their Schelling holidays. For instance, when I worked at a trading company, the usual rule was “if the markets aren't open today, we're not working today”. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

The Nonlinear Library
LW - "Normal" is the equilibrium state of past optimization processes by Alex Altair

The Nonlinear Library

Play Episode Listen Later Oct 31, 2022 7:22


Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: "Normal" is the equilibrium state of past optimization processes, published by Alex Altair on October 30, 2022 on LessWrong. Orienting around the ideas and conclusions involved with AI x-risk can be very difficult. The future possibilities can feel extreme and far-mode, even when we whole-heartedly affirm their plausibility. It helps me to remember that everything around me that feels normal and stable is itself the result of an optimization process that was, at the time, an outrageous black swan. Modernity If you were teleported into the body of a random human throughout history, then most likely, your life would look nothing like the present. You would likely be a hunter-gatherer, or perhaps a farmer. You would be poor by any reasonable standard. You would probably die as a child. You would have nothing resembling your current level of comfort, and your modern daily life would be utterly alien to most humans. What currently feels normal is a freeze-frame state of a shriekingly fast feedback loop involving knowledge, industry, and population. It is nowhere near normal, and it is nowhere near equilibrium. The Anthropocene For hundreds of millions of years, the earth was a wilderness, teeming with life. The teeming had ebbs and flow, evolutionary breakthroughs and power struggles, but essentially, it was a type of equilibrium. For hundreds of thousands of years, humans were just another animal. We ran around the savanna, killed other animals and foraged for food, had sex, slept, and experienced joys and losses. If an alien were gazing at earth from afar, they would have had no reason to think humans were different from any other animal. But all the while, humans were noticing things. They were curious, and desperate, and they had a capacity that the other animals didn't have. They got ideas about the world, and tested them with their hands, and gave this knowledge to their children with their words. Over time, the humans became conspicuous, and eventually, they transformed every inch of their surroundings. For a while now, humans have been a threat to the species they live with. If you are a fish or a weed or a polar bear, your biggest problem is humans. But as John Green points out in The Anthropocene Reviewed, in the 21st century, if you are the atmosphere or a river or a desert, your biggest problem is humans. The rise of humanity is no longer just an ecological phenomenon; it has kicked off a new geological epoch. The fate of the rock itself is in our hands, in a way that it never was for the other mammals, or the dinosaurs, or the trilobites. The invention of wood One day, plants invented wood. This was great for plants, but it had externalities. As wikipedia puts it; The evolution of the wood fiber lignin and the bark-sealing, waxy substance suberin variously opposed decay organisms so effectively that dead materials accumulated long enough to fossilise on a large scale. I don't know exactly how this went down. But I like to imagine watching a forest grow on fast-forward; trees spring up, growing taller and taller, and then eventually each tree dies. Only, when these proto-trees falls down, the logs does not sink into the forest floor and dissolve into biodegraded soil. They just stay there. The tree trunks pile up for millions of years. This was the new equilibrium. The plant matter went through some chemical changes that compressed and homogenized it, but much of the plant's stored energy remained. The energy sank deep underground as what we now call fossil fuels. Bacteria and fungi continued evolving, and eventually they figured out how to eat the now-abundant tougher plant matter. But the coal deposits were now beyond their reach. The historical details of these transitions are not well understood, but what is clear is that something wild happened, the surfa...

The Nonlinear Library
LW - love, not competition by carado

The Nonlinear Library

Play Episode Listen Later Oct 31, 2022 1:54


Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: love, not competition, published by carado on October 30, 2022 on LessWrong. many people are bemoaning that AI is going to replace them. this includes notably artists, but we can expect it to start covering mathematicians and, as AI advances, eventually every kind of human endeavor. there are real, important material concerns, such as artists losing their income, or AI getting so powerful that it destroys everything. this post is not about that, but rather about the longer-term concern of ethically grounding the value of art. is it okay that AI is outcompeting our creativity? yes! in my opinion, we should never have been grounding valuing ourselves in our ability to be the best at stuff to begin with. we should love ourselves and what we make and do intrinsically, not instrumentally. it is valid to want to just watch art for the pleasure that that gives you, and it's even okay to wirehead yourself. but it's also valid to value art as a form of communication between real persons, as a special case of the fact that it's valid to care about reality, even if you can't tell. and the fact that we currently can't tell if art was made by persons or AIs is only a temporary issue; with properly aligned AI, we should be able to tell it "i only want art made by humans!" and have it ensure we only get that, whatever that request would mean upon sufficient reflection. artists, mathematicians, philosophers, and humans in general: aim not to compete! i, and no doubt many others, value you and the things you make for the fact that they are yours and you are real, in a way that fundamentally, intrinsically excludes purely AI-made art, and which includes art made with a mixture of human and AI work in whatever way i would eventually find reasonable if i thought about it enough. if you want to just love doing things and love things others have done, you can just do that. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

The Nonlinear Library
LW - publishing alignment research and infohazards by carado

The Nonlinear Library

Play Episode Listen Later Oct 31, 2022 2:12


Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: publishing alignment research and infohazards, published by carado on October 31, 2022 on LessWrong. to me, turning my thoughts into posts that i then publish on my blog and sometimes lesswrong serves the following purposes: in conversations, i can easily link to a post of mine rather than explaining myself again (the original primary purpose of this blog!) having a more formally written-down version of my thoughts helps me think about them more clearly future posts — whether written by me or others — can link to my posts, contributing to a web of related ideas i can get feedback on my ideas, whether it be through comments on lesswrong or responses on discord however, i've come to increasingly want to write and publish posts which i've determined — either on my own or with the advice of a trusted peers — to be potentially infohazardous, notably with regards to potentially helping AI capability progress. on one hand, there is no post of mine i wouldn't trust, say, yudkowsky reading; on the other i can't just, like, DM him and everyone else i trust a link to an unlisted post every time i make one. it would be nice to have a platform — or maybe a lesswrong feature — which lets me choose which persons or groups can read a post, with maybe a little ⚠ sign next to its title. note that such a platform/feature would need something more complex than just a binary "trusted" flag: just because i can make a post that the Important People can read, doesn't mean i should be trusted to read everything else that they can read; and there might be people whom i trust to read some posts of mine but not others. maybe trusted recipients could be grouped by orgs — such as "i trust MIRI" or "i trust The Standard List Of Trusted Persons". maybe something like the ability to post on the alignment forum is a reasonable proxy for "trustable person"? i am aware that this seems hard to figure out, let alone implement. perhaps there is a much easier alternative i'm not thinking about; for the moment, i'll just stick to making unlisted posts and sending them to the very small intersection of people i trust with infohazards and people for whom it's socially acceptable for me to DM links to new posts of mine. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

The Nonlinear Library
LW - Superintelligent AI is necessary for an amazing future, but far from sufficient by So8res

The Nonlinear Library

Play Episode Listen Later Oct 31, 2022 65:06


Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Superintelligent AI is necessary for an amazing future, but far from sufficient, published by So8res on October 31, 2022 on LessWrong. (Note: Rob Bensinger stitched together and expanded this essay based on an earlier, shorter draft plus some conversations we had. Many of the key conceptual divisions here, like "strong utopia" vs. "weak utopia" etc., are due to him.) I hold all of the following views: Building superintelligent AI is profoundly important. Aligned superintelligence is our best bet for taking the abundant resources in the universe and efficiently converting them into flourishing and fun and art and beauty and adventure and friendship, and all the things that make life worth living. The best possible future would probably look unrecognizably alien. Unlocking humanity's full potential not only means allowing human culture and knowledge to change and grow over time; it also means building and becoming (and meeting and befriending) very new and different sorts of minds, that do a better job of realizing our ideals than the squishy first-pass brains we currently have. The default outcome of building artificial general intelligence, using anything remotely like our current techniques and understanding, is not a wondrously alien future. It's that humanity accidentally turns the reachable universe into a valueless wasteland (at least up to the boundaries defended by distant alien superintelligences). The reason I expect AGI to produce a “valueless wasteland” by default, is not that I want my own present conception of humanity's values locked into the end of time. I want our values to be able to mature! I want us to figure out how to build sentient minds in silicon, who have different types of wants and desires and joys, to be our friends and partners as we explore the galaxies! I want us to cross paths with aliens in our distant travels who strain our conception of what's good, such that we all come out the richer for it! I want our children to have values and goals that would make me boggle, as parents have boggled at their children for ages immemorial! I believe machines can be people, and that we should treat digital people with the same respect we give biological people. I would love to see what a Matrioshka mind can do. I expect that most of my concrete ideas about the future will seem quaint and outdated and not worth their opportunity costs, compared to the rad alternatives we'll see when we and our descendants and creations are vastly smarter and more grown-up. Why, then, do I think that it will take a large effort by humanity to ensure that good futures occur? If I believe in a wondrously alien and strange cosmopolitan future, and I think we should embrace moral progress rather than clinging to our present-day preferences, then why do I think that the default outcome is catastrophic failure? In short: Humanity's approach to AI is likely to produce outcomes that are drastically worse than, e.g., the outcomes a random alien species would produce. It's plausible — though this is much harder to predict, in my books — that a random alien would produce outcomes that are drastically worse (from a cosmopolitan, diversity-embracing perspective!) than what unassisted, unmodified humans would produce. Unassisted, unmodified humans would produce outcomes that are drastically worse than what a friendly superintelligent AI could produce. The practical take-away from the first point is “the AI alignment problem is very important”; the take-away from the second point is “we shouldn't just destroy ourselves and hope aliens end up colonizing our future light cone, and we shouldn't just try to produce AI via a more evolution-like process”; and the take-away from the third point is “we shouldn't just permanently give up on building superintelligent AI”. To clarify my view...

The Nonlinear Library
LW - Gandalf or Saruman? A Soldier in Scout's Clothing by AllAmericanBreakfast

The Nonlinear Library

Play Episode Listen Later Oct 31, 2022 7:01


Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Gandalf or Saruman? A Soldier in Scout's Clothing, published by AllAmericanBreakfast on October 31, 2022 on LessWrong. Sometimes, Scout Mindset can feel a lot like Soldier Mindset, and vice versa. When you've considered an issue with some care, formed an opinion, and somebody responds with raised voice about how awful that opinion is, it's hard not to get defensive or take it personally. There are lots of ways you could respond. If your aim is to achieve a thoughtful and respectful conversation, then you have to do several things: control your emotional reaction, be polite, stay honest about the fact that you disagree, and choose a response that allows the discussion to return to civil discourse. Going through all that feels like a Scout-y thing to do, but it isn't necessarily. Provoking a person with a strongly held (but possibly correct) view into offputting behavior, then assuming a posture of being the reasonable and civil side of the debate, is a wonderful rhetorical strategy. Loudly trying to shout down your opponent makes you look like a dumb Soldier. A smart Soldier tries to look like a Scout. Likewise, if you're a Scout and your information is being ignored, you might need to raise your voice, fight your way through, and make them listen. People who are into signaling talk a lot about virtue signaling and intelligence signaling. It might be helpful to talk about Scout signaling and Soldier signaling. Neither of these are the same as being a Scout or Soldier. They aren't reliable indicators in every context of where somebody is coming from. If a person is interested in developing Scout Mindset, it seems important to me for them to learn how to distinguish Scout Mindset from Scout signaling. Back to our acrimonious debate, let's say the civil person in the debate sincerely wants to want to practice Scout Mindset. There are a couple of useful things they can do. One is to stick to their honest beliefs. That is part of the territory they are honestly scouting out. Another is to remember that their loud debate partner might be a Scout with a very important message. Or not! But if you haven't thoroughly explored their opinion and formed your own conclusions, it's often wise to reserve judgment. If you do so, and stick to your original view, then having fully internalized what your debate partner had to say will help you address that point the next time you're in debate. I'll finish with an example from a debate I had today in which I was playing the "civil" role and my debate partner was the "loud" one. We were talking about the ethics of regulated kidney sales vs. bans on organ sales. My view is that legalizing the regulated sale of kidneys is a good thing. It gives a person with failing health a shot at a much better life, becomes a source of income for the seller, and lowers the cost of medical care. Their view is that poor people who are likely to develop health problems later will tend to be the people selling their organs, even in a legal, regulated system. This is already the case with blood plasma sales. It will become a way for sellers of kidneys to get money fast, blow it on immediate needs, wreck their remaining kidney, and require a transplant themselves a few years down the line. Kidney surgery is painful, and some of the people selling their kidneys will die on the operating table. Political conservatives will use the fact that poor people can sell their kidneys as an excuse to cut welfare. They referred me to this column in the New York Times. Just a small sample from this powerfully written article: The people around me seemed to be regulars who were trying to squeeze a donation in before work. I know because I heard them lying on their phones to their employers about why they were going to be late as the morning wore on. More women came in after n...

The Nonlinear Library
LW - The Social Recession: By the Numbers by antonomon

The Nonlinear Library

Play Episode Listen Later Oct 30, 2022 13:46


Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Social Recession: By the Numbers, published by antonomon on October 29, 2022 on LessWrong. Fewer friends, relationships on the decline, delayed adulthood, trust at an all-time low, and many diseases of despair. The prognosis is not great. By Anton Stjepan Cebalo Intermission (also known as Intermedio) by Edward Hopper, 1963. One of the most discussed topics online recently has been friendships and loneliness. Ever since the infamous chart showing more people are not having sex than ever before first made the rounds, there's been increased interest in the social state of things. Polling has demonstrated a marked decline in all spheres of social life, including close friends, intimate relationships, trust, labor participation, and community involvement. The trend looks to have worsened since the pandemic, although it will take some years before this is clearly established. The decline comes alongside a documented rise in mental illness, diseases of despair, and poor health more generally. In August 2022, the CDC announced that U.S. life expectancy has fallen further and is now where it was in 1996. Contrast this to Western Europe, where it has largely rebounded to pre-pandemic numbers. Still, even before the pandemic, the years 2015-2017 saw the longest sustained decline in U.S. life expectancy since 1915-18. While my intended angle here is not health-related, general sociability is closely linked to health. The ongoing shift has been called the “friendship recession” or the “social recession.” My intention here is not to present a list of miserable points, but to group them together in a meaningful context whose consequences are far-reaching. While most of what I will outline here focuses on the United States, many of these same trends are present elsewhere because its catalyst is primarily the internet itself. With no signs of abating, a new kind of sociability has only started to affect what people ask of the world through the prism of themselves. The topic has directly or indirectly produced a whole genre of commentary from many different perspectives. Many of them touch on the fact that the internet is not being built with pro-social ends in mind. Increasingly monopolized across a few key entities, online life and its data have become the most sought-after commodity. The everyday person's attention has thus become the scarcest resource to be extracted. Other perspectives, often on the left, also stress economic precarity and the decline of public spaces as causes. Some of these same criticisms have been adopted by the New Right, who also indict the culture at large for undermining traditions of sociality, be it gender norms or the family. Believing it disproportionately affects men, this position has produced many lifestylist spinoffs: Men Going Their Own Way (MGTOW), trad-life nostalgia, inceldom, masculinist groups, and hustle culture with a focus on 'beating the rat race.' All of these subcultures are symptoms of the social recession in some way, for better or worse. Often standing outside this conversation altogether are the self-described ‘adults in the room' — professional media pundits, politicians, bureaucrats, and the like, disconnected from the problem themselves, but fixated on its potential to incubate political extremism. Entire institutes have been set up to study, monitor, and surveil the internet's radicalizing tendencies buoyed by anti-social loneliness. The new buzzword often used in this sphere is “stochastic terrorism” and the need to contain some unknown, dangerous online element taking hold of the dispirited. The goal here is not to solve a pernicious problem, but instead to pacify its most flagrant outbursts. We have no clear, comparative basis on which to judge what will emerge from the growing number of people who feel lost, lonely or ...

The Nonlinear Library
LW - Am I secretly excited for AI getting weird? by porby

The Nonlinear Library

Play Episode Listen Later Oct 30, 2022 5:18


Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Am I secretly excited for AI getting weird?, published by porby on October 29, 2022 on LessWrong. This post is arguably darker than my other one. I don't make any persuasive arguments about AI forecasting here; if you don't feel like looking at doominess, feel free to skip this. I've noticed a few instances of what look like people assuming that those who are visibly concerned about AI risk don't really buy into the full weight of what they're saying. Recently, I came across this (hi, niknoble!): As a specific example of what I suspect is a bit of cognitive dissonance, look at the recent post on AGI by porby, which predicts AGI by 2030. I loved reading that post because it promises that the future is going to be wild. If porby is right, we're all in for an adventure. Based on the breathless tone of the post, I would surmise that porby is as excited by his conclusion as I am. For example, we have this excerpt: This is crazy! I'm raising my eyebrows right now to emphasize it! Consider also doing so! This is weird enough to warrant it! Would you have predicted this in 2016? I don't think I would have! Does this strike you as someone who dreads the arrival of AGI? It seems to me like he is awaiting it with great anticipation. But then in the comments on the post, he says that he hopes he's wrong about AGI! If you're reading this porby, do you really want to be wrong? This is an excellent example of the kind of thing I'm talking about, so I'm going to use it. I think my writing and speaking style defaults to a kind of lightness that can be misleading. So let me try to write something a little darker. Well, do you? Because I don't think P(doom | AGI) is anywhere close to 0, especially for AGI developed on very short timescales:YES, I DO WANT TO BE WRONG. The kind of "excitement" I feel about near-term AGI is adjacent to hearing the tornado siren, looking at the radar, seeing the warned cell moving straight east, walking out on my porch to look at a black wall of rain a mile or two away, and seeing the power flashes straight west of me as the tornado rips lives apart. While grabbing a mattress to throw over a tub, I'm doing some quick mental calculations- the statistical rarity of EF-3 or stronger tornadoes, will it stay on the ground, how large is it (glance at the hook on the reflectivity map), how sturdy is this house (the feeling of the entire house shunting to one side during an earlier storm's 120 mph winds wasn't promising), how much damage would a near miss cause? All the while, telling my family to get their shoes, don't worry, we have time (do we have time? probably), just get into the bathroom. It didn't stay on the ground. Also, we have a storm shelter now. It only took about 6 close calls to bite that bullet! More than excitement You know that voyeuristic "excitement" of a really bad hurricane about to make landfall? Something wildly out of the ordinary, something that breaks the sense of normalcy and reminds you that human civilization is fragile? It's a weird, darkly attractive kind of novelty. Watching COVID-19 in January 2020 felt that way, for a while. It was distant and not here, so it felt like an almost fictional threat. Within a few weeks, it became something else. With a sense of vertigo, I settled into the realization that it was really happening. I told my parents to go buy anything they wouldn't want to run out of if there was a run on it at the stores, because things are going to get bad, and that a lot of people were going to die. I explained they had medical backgrounds that put them at much higher risk, and hospitals might get overwhelmed soon, so they shouldn't go inside places with other people if they could avoid it. And if they do, wear masks. The transition of the threat to inevitable reality made me feel bad about the feeling of excitemen...

The Nonlinear Library
LW - Prizes for ML Safety Benchmark Ideas by joshc

The Nonlinear Library

Play Episode Listen Later Oct 29, 2022 2:25


Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Prizes for ML Safety Benchmark Ideas, published by joshc on October 28, 2022 on LessWrong. “If you cannot measure it, you cannot improve it.” – Lord Kelvin (paraphrased) Website: benchmarking.mlsafety.org – receiving submissions until August 2023. ML Safety lacks good benchmarks, so the Center for AI Safety is offering $50,000 - $100,000 prizes for benchmark ideas (or full research papers). We will award at least $100,000 total and up to $500,000 depending on the quality of submissions. What kinds of ideas are you looking for? Ultimately, we will are looking for benchmark ideas that motivate or advance research that reduces existential risks from AI. To provide more guidance, we've outlined four research categories along with example ideas. Alignment: building models that represent and safely optimize difficult-to-specify human values. Monitoring: discovering unintended model functionality. Robustness: designing systems to be reliable in the face of adversaries and highly unusual situations. Safety Applications: using ML to address broader risks related to how ML systems are handled (e.g. for cybersecurity or forecasting). See Open Problems in AI X-Risk [PAIS #5] for example research directions in these categories and their relation to existential risk. What are the requirements for submissions? Datasets or implementations are not necessary, though empirical testing can make it easier for the judges to evaluate your idea. All that is required is a brief write-up (guidelines here). How the write-up is formatted isn't very important as long as it effectively pitches the benchmark and concretely explains how it would be implemented. If you don't have prior experience designing benchmarks, we recommend reading this document for generic tips. Who are the judges? Dan Hendrycks, Paul Christiano, and Collin Burns. If you have questions, they might be answered on the website, or you can post them here. We would also greatly appreciate it if you helped to spread the word about this opportunity. Thanks to Sidney Hough and Kevin Liu for helping to make this happen and to Collin Burns and Akash Wasil for feedback on the website. This project is supported by the Future Fund regranting program. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.