Podcasts about ltff

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Work with me on agent foundations: independent fellowship, published by Alex Altair on September 21, 2024 on LessWrong. Summary: I am an independent researcher in agent foundations, and I've recently received an LTFF grant to fund someone to do research with me. This is a rolling application; I'll close it whenever I'm no longer interested in taking another person. If you're not familiar with agent foundations, you can read about my views in this post. What the role might be like This role is extremely flexible. Depending on who you are, it could end up resembling an internship, a research assistant position, a postdoc or even as a mentor/advisor to me. Below, I've listed out the parameters of the fellowship that I am using as a baseline of what it could be. All of these parameters are negotiable! $25 per hour. This is not a lot for people who live in the SF Bay area, or who are used to industry salaries, but it looks to me like this is comparable to a typical grad student salary. 20 hours per week. I'd like this fellowship to be one of your main projects, and I think it can take quite a lot of "deep work" focus before one can make progress on the research problems.[1] 3 months, with a decent chance of extension. During my AI safety camp project, it took about 6 weeks to get people up to speed on all the parts of the agent structure problem. Ideally I could find someone for this role who is already closer to caught up (though I don't necessarily anticipate that). I'm thinking of this fellowship as something like an extended work-trial for potentially working together longer-term. That said, I think we should at least aim to get results by the end of it. Whether I'll decide to invite you to continue working with me afterwards depends on how our collaboration went (both technically and socially), how many other people I'm collaborating with at that time, and whether I think I have enough funds to support it. Remote, but I'm happy to meet in person. Since I'm independent, I don't have anything like an office for you to make use of. But if you happen to be in the SF Bay area, I'd be more than happy to have our meetings in person. I wake up early, so US eastern and European time zones work well for me (and other time zones too). Meeting 2-5 times per week. Especially in the beginning, I'd like to do a pretty large amount of syncing up. It can take a long time to convey all the aspects of the research problems. I also find that real-time meetings regularly generate new ideas. That said, some people find meetings worse for their productivity, and so I'll be responsive to your particular work style. An end-of-term write-up. It seems to take longer than three months to get results in the types of questions I'm interested in, but I think it's good practice to commit to producing a write-up of how the fellowship goes. If it goes especially well, we could produce a paper. What this role ends up looking like mostly depends on your experience level relative to mine. Though I now do research, I haven't gone through the typical academic path. I'm in my mid-thirties and have a proportional amount of life and career experience, but in terms of mathematics, I consider myself the equivalent of a second year grad student. So I'm comfortable leading this project and am confident in my research taste, but you might know more math than me. The research problems Like all researchers in agent foundations, I find it quite difficult to concisely communicate what my research is about. Probably the best way to tell if you will be interested in my research problems is to read other things I've written, and then have a conversation with me about it. All my research is purely mathematical,[2] rather than experimental or empirical. None of it involves machine learning per se, but the theorems should ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Long-Term Future Fund: March 2024 Payout recommendations, published by Linch on June 12, 2024 on The Effective Altruism Forum. Introduction This payout report covers the Long-Term Future Fund's grantmaking from May 1 2023 to March 31 2024 (11 months). It follows our previous April 2023 payout report. Total funding recommended: $6,290,550 Total funding paid out: $5,363,105 Number of grants paid out: 141 Acceptance rate (excluding desk rejections): 159/672 = 23.7% Acceptance rate (including desk rejections): 159/825 = 19.3% Report authors: Linchuan Zhang (primary author), Caleb Parikh (fund chair), Oliver Habryka, Lawrence Chan, Clara Collier, Daniel Eth, Lauro Langosco, Thomas Larsen, Eli Lifland 25 of our grantees, who received a total of $790,251, requested that our public reports for their grants are anonymized (the table below includes those grants). 13 grantees, who received a total of $529, 819, requested that we not include public reports for their grants. You can read our policy on public reporting here. We referred at least 2 grants to other funders for evaluation. Highlighted Grants (The following grants writeups were written by me, Linch Zhang. They were reviewed by the primary investigators of each grant). Below, we highlighted some grants that we thought were interesting and covered a relatively wide scope of LTFF's activities. We hope that reading the highlighted grants can help donors make more informed decisions about whether to donate to LTFF.[1] Gabriel Mukobi ($40,680) - 9-month university tuition support for technical AI safety research focused on empowering AI governance interventions The Long-Term Future Fund provided a $40,680 grant to Gabriel Mukobi from September 2023 to June 2024, originally for 9 months of university tuition support. The grant enabled Gabe to pursue his master's program in Computer Science at Stanford, with a focus on technical AI governance. Several factors favored funding Gabe, including his strong academic background (4.0 GPA in Stanford CS undergrad with 6 graduate-level courses), experience in difficult technical AI alignment internships (e.g., at the Krueger lab), and leadership skills demonstrated by starting and leading the Stanford AI alignment group. However, some fund managers were skeptical about the specific proposed technical research directions, although this was not considered critical for a skill-building and career-development grant. The fund managers also had some uncertainty about the overall value of funding Master's degrees. Ultimately, the fund managers compared Gabe to marginal MATS graduates and concluded that funding him was favorable. They believed Gabe was better at independently generating strategic directions and being self-motivated for his work, compared to the median MATS graduate. They also considered the downside risks and personal costs of being a Master's student to be lower than those of independent research, as academia tends to provide more social support and mental health safeguards, especially for Master's degrees (compared to PhDs). Additionally, Gabe's familiarity with Stanford from his undergraduate studies was seen as beneficial on that axis. The fund managers also recognized the value of a Master's degree credential for several potential career paths, such as pursuing a PhD or working in policy. However, a caveat is that Gabe might have less direct mentorship relevant to alignment compared to MATS extension grantees. Outcomes: In a recent progress report, Gabe noted that the grant allowed him to dedicate more time to schoolwork and research instead of taking on part-time jobs. He produced several new publications that received favorable media coverage and was accepted to 4 out of 6 PhD programs he applied to. The grant also allowed him to finish graduating in March instead of Ju...

Agreement78 % of my donations so far have gone to the Long-Term Future Fund[1] (LTFF), which mainly supports AI safety interventions. However, I have become increasingly sceptical about the value of existential risk mitigation, and currently think the best interventions are in the area of animal welfare[2]. As a result, I realised it made sense for me to arrange a bet with someone very worried about AI in order to increase my donations to animal welfare interventions. Gregory Colbourn (Greg) was the 1st person I thought of. He said: I think AGI [artificial general intelligence] is 0-5 years away and p(doom|AGI) is ~90% I doubt doom in the sense of human extinction is anywhere as likely as suggested by the above. I guess the annual extinction risk over the next 10 years is 10^-7, so I proposed a bet to Greg similar to the end-of-the-world bet between [...] ---Outline:(00:07) Agreement(03:53) Impact(05:18) AcknowledgementsThe original text contained 5 footnotes which were omitted from this narration. --- First published: June 4th, 2024 Source: https://forum.effectivealtruism.org/posts/GfGxaPBAMGcYjv8Xd/i-bet-greg-colbourn-10-keur-that-ai-will-not-kill-us-all-by --- Narrated by TYPE III AUDIO.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: I bet Greg Colbourn 10 k€ that AI will not kill us all by the end of 2027, published by Vasco Grilo on June 4, 2024 on The Effective Altruism Forum. Agreement 78 % of my donations so far have gone to the Long-Term Future Fund[1] (LTFF), which mainly supports AI safety interventions. However, I have become increasingly sceptical about the value of existential risk mitigation, and currently think the best interventions are in the area of animal welfare[2]. As a result, I realised it made sense for me to arrange a bet with someone very worried about AI in order to increase my donations to animal welfare interventions. Gregory Colbourn (Greg) was the 1st person I thought of. He said: I think AGI [artificial general intelligence] is 0-5 years away and p(doom|AGI) is ~90% I doubt doom in the sense of human extinction is anywhere as likely as suggested by the above. I guess the annual extinction risk over the next 10 years is 10^-7, so I proposed a bet to Greg similar to the end-of-the-world bet between Bryan Caplan and Eliezer Yudkowsky. Meanwhile, I transferred 10 k€ to PauseAI[3], which is supported by Greg, and he agreed to the following. If Greg or any of his heirs are still alive by the end of 2027, they transfer to me or an organisation of my choice 20 k€ times the ratio between the consumer price index for all urban consumers and items in the United States, as reported by the Federal Reserve Economic Data (FRED), in December 2027 and April 2024. I expect inflation in this period, i.e. a ratio higher than 1. Some more details: The transfer must be made in January 2028. I will decide in December 2027 whether the transfer should go to me or an organisation of choice. My current preference is for it to go directly to an organisation, such that 10 % of it is not lost in taxes. If for some reason I am not able to decide (e.g. if I die before 2028), the transfer must be made to my lastly stated organisation of choice, currently The Humane League (THL). As Founders Pledge's Patient Philanthropy Fund, I have my investments in Vanguard FTSE All-World UCITS ETF USD Acc. This is an exchange-traded fund (ETF) tracking global stocks, which have provided annual real returns of 5.0 % since 1900. In addition, Lewis Bollard expects the marginal cost-effectiveness of Open Philanthropy's (OP's) farmed animal welfare grantmaking "will only decrease slightly, if at all, through January 2028"[4], so I suppose I do not have to worry much about donating less over the period of the bet of 3.67 years (= 2028 + 1/12 - (2024 + 5/12)). Consequently, I think my bet is worth it if its benefit-to-cost ratio is higher than 1.20 (= (1 + 0.050)^3.67). It would be 2 (= 20*10^3/(10*10^3)) if the transfer to me or an organisation of my choice was fully made, and Person X fulfils the agreement, so I need 60 % (= 1.20/2) of the transfer to be made and agreement with Person X to be fulfilled. I expect this to be the case based on what I know about Greg and Person X, and information Greg shared, so I went ahead with the bet. Here are my and Greg's informal signatures: Me: Vasco Henrique Amaral Grilo. Greg: Gregory Hamish Colbourn. Impact I expect 90 % of the potential benefits of the bet to be realised. So I believe the bet will lead to additional donations of 8 k€ (= (0.9*20 - 10)*10^3). Saulius estimated corporate campaigns for chicken welfare improve 41 chicken-years per $, and OP thinks "the marginal FAW [farmed animal welfare] funding opportunity is ~1/5th as cost-effective as the average from Saulius' analysis", which means my donations will affect 8.20 chicken-years per $ (= 41/5). Therefore I expect my bet to improve 65.6 k chicken-years (= 8*10^3*8.20). I also estimate corporate campaigns for chicken welfare have a cost-effectiveness of 14.3 DALY/$[5]. So I expect the benefits of the bet to be equiv...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Wild animal welfare? Stable totalitarianism? Predict which new EA cause area will go mainstream!, published by Jackson Wagner on March 13, 2024 on The Effective Altruism Forum. Long have I idly whiled away the hours browsing Manifold Markets, trading on trivialities like videogame review scores or NASA mission launch dates. It's fun, sure -- but I am a prediction market advocate, who believes that prediction markets have great potential to aggregate societally useful information and improve decision-making! I should stop fooling around, and instead put my Manifold $Mana to some socially-productive use!! So, I've decided to create twenty subsidized markets about new EA cause areas. Each one asks if the nascent cause area (like promoting climate geoengineering, or researching space governance) will receive $10,000,000+ from EA funders before the year 2030. My hope is that that these markets can help create common knowledge around the most promising up-and-coming "cause area candidates", and help spark conversations about the relative merits of each cause. If some causes are deemed likely-to-be-funded-by-2030, but little work is being done today, that could even be a good signal for you to start your own new project in the space! Without further ado, here are the markets: Animal Welfare Will farmed-invertebrate welfare (shrimp, insects, octopi, etc) get $10m+ from EA funders before 2030? Will wild-animal welfare interventions get $10m+ from EA funders before 2030? [embed most popular market] Global Health & Development Will alcohol, tobacco, & sugar taxation... ? Mental-health / subjective-wellbeing interventions in developing countries? Institutional improvements Approval voting, quadratic funding, liquid democracy, and related democratic mechanisms? Georgism (aka land value taxes)? Charter Cities / Affinity Cities / Network States? Investing (Note that the resolution criteria on these markets is different than for the other questions, since investments are different from grants.) Will the Patient Philanthropy Fund grow to $10m+ before 2030? Will "impact markets" distribute more than $10m of grant funding before 2030? X-Risk Civilizational bunkers? Climate geoengineering? Preventing stable totalitarianism? Preventing S-risks? Artificial Intelligence Mass-movement political advocacy for AI regulation (ie, "PauseAI")? Mitigation of AI propaganda / "botpocalypse" impacts? Transhumanism Cryonics & brain-emulation research? Human intelligence augmentation / embryo selection? Space governance / space colonization? Moral philosophy Research into digital sentience or the nature of consciousness? Interventions primarily motivated by anthropic reasoning, acausal trade with parallel universes, alien civilizations, simulation arguments, etc? I encourage you to trade on these markets, comment on them, and boost/share them -- put your Manifold mana to a good use by trying to predict the future trajectory of the EA movement! Here is one final market I created, asking which three of the cause areas above will receive the most support between now and 2030. Resolution details & other thoughts The resolution criteria for most of these questions involves looking at publicly-available grantmaking documentation (like this Openphil website, for example), adding up all the grants that I believe qualify as going towards the stated cause area, and seeing if the grand total exceeds ten million dollars. Since I'm specifically interested in how the EA movement will grow and change over time, I will only be counting money from "EA funders" -- stuff like OpenPhil, LTFF, SFF, Longview Philanthropy, Founders Fund, GiveWell, etc, will count for this, while money from "EA-adjacent" sources (like, say, Patrick Collison, Yuri Milner, the Bill & Melinda Gates Foundation, Elon Musk, Vitalik Buterin, Peter T...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Understanding SAE Features with the Logit Lens, published by Joseph Isaac Bloom on March 11, 2024 on The AI Alignment Forum. This work was produced as part of the ML Alignment & Theory Scholars Program - Winter 2023-24 Cohort, with support from Neel Nanda and Arthur Conmy. Joseph Bloom is funded by the LTFF, Manifund Regranting Program, donors and LightSpeed Grants. This post makes extensive use of Neuronpedia, a platform for interpretability focusing on accelerating interpretability researchers working with SAEs. Links: SAEs on HuggingFace, Analysis Code Executive Summary This is an informal post sharing statistical methods which can be used to quickly / cheaply better understand Sparse Autoencoder (SAE) features. Firstly, we use statistics (standard deviation, skewness and kurtosis) of the logit weight distributions of features (WuWdec[feature]) to characterize classes of features, showing that many features can be understood as promoting / suppressing interpretable classes of tokens. We propose 3 different kinds of features, analogous to previously characterized " universal neurons": Partition Features, which (somewhat) promote half the tokens and suppress the other half according to capitalization and spaces (example pictured below) Suppression Features, which act like partition features but are more asymmetric. Prediction Features which promote tokens in classes of varying sizes, ranging from promoting tokens that have a close bracket to promoting all present tense verbs. Secondly, we propose a statistical test for whether a feature's output direction is trying to distinguish tokens in some set (eg: "all caps tokens") from the rest. We borrowed this technique from systems biology where it is used at scale frequently. The key limitation here is that we need to know in advance which sets of tokens are promoted / inhibited. Lastly, we demonstrate the utility of the set-based technique by using it to locate features which enrich token categories of interest (defined by regex formulas, NLTK toolkit parts of speech tagger and common baby names for boys/girls). Feature 4467. Above: Feature Dashboard Screenshot from Neuronpedia. It is not immediately obvious from the dashboard what this feature does. Below: Logit Weight distribution classified by whether the token starts with a space, clearly indicating that this feature promotes tokens which lack an initial space character. Introduction In previous work, we trained and open-sourced a set of sparse autoencoders (SAEs) on the residual stream of GPT2 small. In collaboration with Neuronpedia, we've produced feature dashboards, auto-interpretability explanations and interfaces for browsing for ~300k+ features. The analysis in this post is performed on features from the layer 8 residual stream of GPT2 small (for no particular reason). SAEs might enable us to decompose model internals into interpretable components. Currently, we don't have a good way to measure interpretability at scale, but we can generate feature dashboards which show things like how often the feature fires, its direct effect on tokens being sampled (the logit weight distribution) and when it fires (see examples of feature dashboards below). Interpreting the logit weight distribution in feature dashboards for multi-layer models is implicitly using Logit Lens, a very popular technique in mechanistic interpretability. Applying the logit lens to features means that we compute the product of a feature direction and the unembed (WuWdec[feature]), referred to as the "logit weight distribution". Since SAEs haven't been around for very long, we don't yet know what the logit weight distributions typically look like for SAE features. Moreover, we find that the form of logit weight distribution can vary greatly. In most cases we see a vaguely normal distribution and s...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Impact Assessment of AI Safety Camp (Arb Research), published by Sam Holton on January 24, 2024 on The Effective Altruism Forum. Authors: Sam Holton, Misha Yagudin Data collection: David Mathers, Patricia Lim Note: Arb Research was commissioned to produce this impact assessment by the AISC organizers. Summary AI Safety Camp (AISC) connects people interested in AI safety (AIS) to a research mentor, forming project teams that last for a few weeks and go on to write up their findings. To assess the impact of AISC, we first consider how the organization might increase the productivity of the Safety field as a whole. Given its short duration and focus on introducing new people to AIS, we conclude that AISC's largest contribution is in producing new AIS researchers that otherwise wouldn't have joined the field. We gather survey data and track participants in order to estimate how many researchers AISC has produced, finding that 5-10% of participants plausibly become AIS researchers (see "Typical AIS researchers produced by AISC" for examples) that otherwise would not have joined the field. AISC spends roughly $12-30K per researcher. We could not find estimates for counterfactual researcher production in similar programs such as (SERI) MATS. However, we used the LTFF grants database to estimate that the cost of researcher upskilling in AI safety for 1 year is $53K. Even assuming all researchers with this amount of training become safety researchers that wouldn't otherwise have joined the field, AISC still recruits new researchers at a similar or lower cost (though note that training programs at different stages of a career pipeline are compliments). We then consider the relevant counterfactuals for a nonprofit organization interested in supporting AIS researchers and tentatively conclude that funding the creation of new researchers in this way is slightly more impactful than funding a typical AIS project. However, this conclusion is highly dependent on one's particular views about AI safety and could also change based on an assessment of the quality of researchers produced by AISC. We also review what other impacts AISC has in terms of producing publications and helping participants get a position in AIS organizations. Approach To assess impact, we focus on AISC's rate of net-new researcher production. We believe this is the largest contribution of the camp given their focus on introducing researchers to the field and given the short duration of projects. In the appendix, we justify this and explain why new researcher production is one of the most important contributions to the productivity of a research field. For completeness, we also attempt to quantify other impacts such as: Direct research outputs from AISC and follow-on research. Network effects leading to further AIS and non-AIS research. AISC leading to future positions. AISC plausibly has several positive impacts that we were unable to measure, such as increasing researcher effort, increasing research productivity, and improving resource allocation. We are also unable to measure the quality of AIS research due to the difficulty of assessing such work. Data collected We used 2 sources of data for this assessment: Survey. We surveyed AISC participants from all camps, receiving 24 responses (~10% of all participants). Questions aimed to determine the participants' AIS involvement before and after camp as well as identify areas for improvement. To ensure honest answers, we promised respondents that anecdotes would not be shared without their direct permission. Instead, we will summarize common lessons from these responses where possible. Participant tracking. To counter response biases in survey data, we independently researched the career path of 101 participants from AISC 4-6, looking at involvement in AI safety rese...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: EA Infrastructure Fund Ask Us Anything (January 2024), published by Tom Barnes on January 17, 2024 on The Effective Altruism Forum. The EA Infrastructure Fund (EAIF) is running an Ask Us Anything! This is a time where EAIF grantmakers have set aside some time to answer questions on the Forum. I (Tom) will aim to answer most questions next weekend (~January 20th), so please submit questions by the 19th. Please note: We believe the next three weeks are an especially good time to donate to EAIF, because: We continue to face signficant funding constraints, leading to many great projects going either unfunded or underfunded Your donation will be matched at a 2:1 ratio until Feb 2. EAIF has ~$2m remaining in available matching funds, meaning that (unlike LTFF) this match is unlikely to be utilised without your support If you agree, you can donate to us here. About the Fund The EA Infrastructure Fund aims to increase the impact of projects that use the principles of effective altruism, by increasing their access to talent, capital, and knowledge. Over 2022 and H1 2023, we made 347 grants totalling $13.4m in dispersement. You can see our public grants database here. Related posts EA Infrastructure Fund's Plan to Focus on Principles-First EA LTFF and EAIF are unusually funding-constrained right now EA Funds organizational update: Open Philanthropy matching and distancing EA Infrastructure Fund: June 2023 grant recommendations What do Marginal Grants at EAIF Look Like? Funding Priorities and Grantmaking Thresholds at the EA Infrastructure Fund About the Team Tom Barnes: Tom is currently a Guest Fund Manager at EA Infrastructure Fund (previously an Assistant Fund Manager since ~Oct 2022). He also works as an Applied Researcher at Founders Pledge, currently on secondment to the UK Government to work on AI policy. Previously, he was a visiting fellow at Rethink Priorities, and was involved in EA uni group organizing. Caleb Parikh: Caleb is the project lead of EA Funds. Caleb has previously worked on global priorities research as a research assistant at GPI, EA community building (as a contractor to the community health team at CEA), and global health policy. Caleb currently leads EAIF as interim chair. Linchuan Zhang: Linchuan (Linch) Zhang currnetly works full-time at EA Funds. He was previously a Senior Researcher at Rethink Priorities working on existential security research. Before joining RP, he worked on time-sensitive forecasting projects around COVID-19. Previously, he programmed for Impossible Foods and Google and has led several EA local groups. Ask Us Anything We're happy to answer any questions - marginal uses of money, how we approach grants, questions/critiques/concerns you have in general, what reservations you have as a potential donor or applicant, etc. There's no hard deadline for questions, but I would recommend submitting by the 19th January as I aim to respond from the 20th As a reminder, we remain funding-constrained, and your donation will be matched (for every $1 you donate, EAIF will receive $3). Please consider donating! If you have projects relevant to builiding up the EA community's infrastructure, you can also apply for funding here. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Some more marginal Long-Term Future Fund grants, published by calebp on November 16, 2023 on The Effective Altruism Forum. These fictional grants represent the most promising applications we turn down due to insufficient funding. Throughout the text, 'I' refers to Caleb Parikh, and 'we' refers to both Caleb Parikh and Linch Zhang. This reflects the perspectives of two individuals who are very familiar with the Long-Term Future Fund (LTFF). However, others associated with the LTFF might not agree that this accurately represents their impression of the LTFF's marginal (rejected) grants. Fictional grants that we rejected but were very close to our funding bar Each grant is based on 1-3 real applications we have received in the past ~three months. You can see our original LTFF marginal funding post here, and our post on the usefulness of funding the EAIF and LTFF here.[1] Please note that these are a few of the most promising grants we've recently turned down - not the average rejected grant. [2] ($25,000)~ Funding to continue research on a multi-modal chess language model, focusing on alignment and interpretability. The project involves optimizing a data extraction pipeline, refining the model's behaviour to be less aggressive, and exploring ways to modify the model training. Additional tasks include developing a simple Encoder-Decoder chess language model as a benchmark and writing an article about AI safety. The primary objective is to develop methods ensuring that multi-modal models act according to high-level behavioural priorities. The applicant's background includes experience as a machine learning engineer and chess, competing and developing predictive models. The past year's work under a previous LTFF grant resulted in a training dataset and some initial analysis, laying the groundwork for this continued research. ($25,000) ~ Four months' salary for a former academic to tackle some unusually tractable research problems in disaster resilience after large-scale GCRs. Their work would focus on researching Australia's resilience to a northern hemisphere nuclear war. Their track record included several papers in high-impact factor journals, and their past experiences and networks made them well-positioned for further work in this area. The grantee would also work on public outreach to inform the Australian public about nuclear risks and resilience strategies. ($50,000)~ Six months of career transition funding to help the applicant enter a technical AI safety role. The applicant has seven years of software engineering experience at prominent tech companies and aims to pivot his career towards AI safety. They'll focus on interpretability experiments with Leela Go Zero during the grant. The grant covers 50% of his previous salary and will facilitate upskilling in AI safety, completion of technical courses, and preparation for interviews with AI safety organizations. He has pivoted his career successfully in the past and has been actively engaged in the effective altruism community, co-running a local group and attending international conferences. This is his first funding request. ($40,000)~ Six months dedicated to exploring and contributing to AI governance initiatives, focusing on policy development and lobbying in Washington, D.C. The applicant seeks to build expertise and networks in AI governance, aiming to talk with over 50 professionals in the field and apply to multiple roles in this domain. The grant will support efforts to increase the probability of the U.S. government enacting legislation to manage the development of frontier AI technologies. The applicant's background includes some experience in AI policy and a strong commitment to effective altruism principles. The applicant has fewer than three years of professional experience and an undergraduate degree ...

We speak with Jamie Bernardi, co-founder & AI Safety Lead at not-for-profit BlueDot Impact, who host the biggest and most up-to-date courses on AI safety & alignment at AI Safety Fundamentals (https://aisafetyfundamentals.com/). Jamie completed his Bachelors (Physical Natural Sciences) and Masters (Physics) at the U. Cambridge and worked as an ML Engineer before co-founding BlueDot Impact.The free courses they offer are created in collaboration with people on the cutting edge of AI safety, like Richard Ngo at OpenAI and Prof David Kreuger at U. Cambridge. These courses have been one of the most powerful ways for new people to enter the field of AI safety, and I myself (Soroush) have taken AGI Safety Fundamentals 101 — an exceptional course that was crucial to my understanding of the field and can highly recommend. Jamie shares why he got into AI safety, some recent history of the field, an overview of the current field, and how listeners can get involved and start contributing to a ensure a safe & positive world with advanced AI and AGI.Hosted by Soroush Pour. Follow me for more AGI content:Twitter: https://twitter.com/soroushjpLinkedIn: https://www.linkedin.com/in/soroushjp/== Show links ==-- About Jamie --* Website: https://jamiebernardi.com/* Twitter: https://twitter.com/The_JBernardi* BlueDot Impact: https://www.bluedotimpact.org/-- Further resources --* AI Safety Fundamentals courses: https://aisafetyfundamentals.com/* Donate to LTFF to support AI safety initiatives: https://funds.effectivealtruism.org/funds/far-future* Jobs + opportunities in AI safety:  * https://aisafetyfundamentals.com/opportunities  * https://jobs.80000hours.org* Horizon Fellowship for policy training in AI safety: https://www.horizonpublicservice.org/fellowshipRecorded Sep 7, 2023

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Long-Term Future Fund Ask Us Anything (September 2023), published by Linch on August 31, 2023 on The Effective Altruism Forum. LTFF is running an Ask Us Anything! Most of the grantmakers at LTFF have agreed to set aside some time to answer questions on the Forum. I (Linch) will make a soft commitment to answer one round of questions this coming Monday (September 4th) and another round the Friday after (September 8th). We think that right now could be an unusually good time to donate. If you agree, you can donate to us here. About the Fund The Long-Term Future Fund aims to positively influence the long-term trajectory of civilization by making grants that address global catastrophic risks, especially potential risks from advanced artificial intelligence and pandemics. In addition, we seek to promote, implement, and advocate for longtermist ideas and to otherwise increase the likelihood that future generations will flourish. In 2022, we dispersed ~250 grants worth ~10 million. You can see our public grants database here. Related posts LTFF and EAIF are unusually funding-constrained right now EA Funds organizational update: Open Philanthropy matching and distancing Long-Term Future Fund: April 2023 grant recommendations What Does a Marginal Grant at LTFF Look Like? Asya Bergal's Reflections on my time on the Long-Term Future Fund Linch Zhang's Select examples of adverse selection in longtermist grantmaking About the Team Asya Bergal: Asya is the current chair of the Long-Term Future Fund. She also works as a Program Associate at Open Philanthropy. Previously, she worked as a researcher at AI Impacts and as a trader and software engineer for a crypto hedgefund. She's also written for the AI alignment newsletter and been a research fellow at the Centre for the Governance of AI at the Future of Humanity Institute (FHI). She has a BA in Computer Science and Engineering from MIT. Caleb Parikh: Caleb is the project lead of EA Funds. Caleb has previously worked on global priorities research as a research assistant at GPI, EA community building (as a contractor to the community health team at CEA), and global health policy. Linchuan Zhang: Linchuan (Linch) Zhang is a Senior Researcher at Rethink Priorities working on existential security research. Before joining RP, he worked on time-sensitive forecasting projects around COVID-19. Previously, he programmed for Impossible Foods and Google and has led several EA local groups. Oliver Habryka: Oliver runs Lightcone Infrastructure, whose main product is Lesswrong. Lesswrong has significantly influenced conversations around rationality and AGI risk, and the LWits community is often credited with having realized the importance of topics such as AGI (and AGI risk), COVID-19, existential risk and crypto much earlier than other comparable communities. You can find a list of our fund managers in our request for funding here. Ask Us Anything We're happy to answer any questions - marginal uses of money, how we approach grants, questions/critiques/concerns you have in general, what reservations you have as a potential donor or applicant, etc. There's no real deadline for questions, but let's say we have a soft commitment to focus on questions asked on or before September 8th. Because we're unusually funding-constrained right now, I'm going to shill again for donating to us. If you have projects relevant to mitigating global catastrophic risks, you can also apply for funding here. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: LTFF and EAIF are unusually funding-constrained right now, published by Linch on August 30, 2023 on The Effective Altruism Forum. Summary EA Funds aims to empower thoughtful individuals and small groups to carry out altruistically impactful projects - in particular, enabling and accelerating small/medium-sized projects (with grants

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Select examples of adverse selection in longtermist grantmaking, published by Linch on August 23, 2023 on The Effective Altruism Forum. Sometimes, there is a reason other grantmakers aren't funding a fairly well-known EA (-adjacent) project. This post is written in a professional capacity, as a volunteer/sometimes contractor for EA Funds' Long-Term Future Fund (LTFF), which is a fiscally sponsored project of Effective Ventures Foundation (UK) and Effective Ventures Foundation USA Inc. I am not and have never been an employee at either Effective Ventures entity. Opinions are my own and do not necessarily represent that of any of my employers or of either Effective Ventures entity. I originally wanted to make this post a personal shortform, but Caleb Parikh encouraged me to make it a top-level post instead. There is an increasing number of new grantmakers popping up, and also some fairly rich donors in longtermist EA that are thinking of playing a more active role in their own giving (instead of deferring). I am broadly excited about the diversification of funding in longtermist EA. There are many advantages of having a diverse pool of funding: Potentially increases financial stability of projects and charities Allows for a diversification of worldviews Encourages accountability, particularly of donors and grantmakers - if there's only one or a few funders, people might be scared of offering justified criticisms Access to more or better networks - more diverse grantmakers might mean access to a greater diversity of networks, allowing otherwise overlooked and potentially extremely high-impact projects to be funded Greater competition and race to excellence and speed among grantmakers - I've personally been on both sides of being faster and much slower than other grantmakers, and it's helpful to have a competitive ecosystem to improve grantee and/or donor experience However, this comment will mostly talk about the disadvantages. I want to address adverse selection: In particular, if a project that you've heard of through normal EA channels hasn't been funded by existing grantmakers like LTFF, there is a decently high likelihood that other grantmakers have already evaluated the grant and (sometimes for sensitive private reasons) have decided it is not worth funding. Reasons against broadly sharing reasons for rejection From my perspective as an LTFF grantmaker, it is frequently imprudent, impractical, or straightforwardly unethical to directly make public our reasons for rejection. For example: Our assessments may include private information that we are not able to share with other funders. Writing up our reasons for rejection of specific projects may be time-consuming, politically unwise, and/or encourage additional ire ("punching down"). We don't want to reify our highly subjective choices too much, and public writeups of rejections can cause informational cascades. Often other funders don't even think to ask about whether the project has already been rejected by us, and why (and/or rejected grantees don't pass on that they've been rejected by another funder). Sharing negative information about applicants would make applying to EA Funds more costly and could discourage promising applicants. Select examples Here are some (highly) anonymized examples of grants I have personally observed being rejected by a centralized grantmaker. For further anonymization, in some cases I've switched details around or collapsed multiple examples into one. Most, although not all, of the examples are personal experiences from working on the LTFF. Many of these examples are grants that have later been funded by other grantmakers or private donors. An academic wants funding for a promising sounding existential safety research intervention in an area of study that none of the LTFF grantmakers ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What Does a Marginal Grant at LTFF Look Like? Funding Priorities and Grantmaking Thresholds at the Long-Term Future Fund, published by Linch on August 10, 2023 on The Effective Altruism Forum. The Long-Term Future Fund (LTFF) makes small, targeted grants with the aim of improving the long-term trajectory of humanity. We are currently fundraising to cover our grantmaking budget for the next 6 months. We would like to give donors more insight into how we prioritize different projects, so they have a better sense of how we plan to spend their marginal dollar. Below, we've compiled fictional but representative grants to illustrate what sort of projects we might fund depending on how much we raise for the next 6 months, assuming we receive grant applications at a similar rate and quality to the recent past. Our motivations for presenting this information are a) to provide transparency about how the LTFF works, and b) to move the EA and longtermist donor communities towards a more accurate understanding of what their donations are used for. Sometimes, when people donate to charities (EA or otherwise), they may wrongly assume that their donations go towards funding the average, or even more optimistically, the best work of those charities. However, it is usually more useful to consider the marginal impact for the world that additional dollars would buy. By offering illustrative examples of the sort of projects we might fund at different levels of funding, we hope to give potential donors a better sense of what their donations might buy, depending on how much funding has already been committed. We hope that this post will help improve the quality of thinking and discussions about charities in the EA and longtermist communities. For donors who believe that the current marginal LTFF grants are better than marginal funding of all other organizations, please consider donating! Compared to the last 3 years, we now have both a) unusually high quality and quantity of applications and b) unusually low amount of donations, which means we'll have to raise our bar substantially if we do not receive additional donations. This is an especially good time to donate, as donations are matched 2:1 by Open Philanthropy (OP donates $2 for every $1 you donate). That said, if you instead believe that marginal funding of another organization is (between 1x and 3x, depending on how you view marginal OP money) better than current marginal LTFF grants, then please do not donate to us, and instead donate to them and/or save the money for later. Background on the LTFF We are committed to improving the long-term trajectory of civilization, with a particular focus on reducing global catastrophic risks. We specialize in funding early stage projects rather than established organizations. From March 2022 to March 2023, we received 878 applications and funded 263 as grants, worth ~$9.1M dollars total (average $34.6k/grant). To our knowledge, we have made more small grants in this time period than any other longtermist- or EA- motivated funder. Other funders in this space include Open Philanthropy, Survival and Flourishing Fund, and recently Lightspeed Grants and Manifund. Historically, ~40% of our funding has come from Open Phil. However, we are trying to become more independent of Open Phil. As a temporary stopgap measure, Open Phil is matching donations to LTFF 2:1 instead of granting to us directly. 100% of money we fundraise for LTFF qua LTFF goes to grantees; we fundraise separately and privately for operational costs. We try to be very willing to fund weird things that the grantmakers' inside views believe are really impactful for the long-term future. You can read more about our work at our website here, or in our accompanying payout report here. Methodology for this analysis At the LTFF, we assign ea...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Manifund: What we're funding (weeks 2-4), published by Austin on August 4, 2023 on The Effective Altruism Forum. Overall reflections Very happy with the volume and quality of grants we've been making $600k+ newly committed across 12 projects Regrantors have been initiating grants and coordinating on large projects Independent donors have committed $35k+ of their own money! We plan to start fundraising soon, based on this pace of distribution Happy to be coordinating with funders at LTFF, Lightspeed, Nonlinear and OpenPhil We now have a common Slack channel to share knowledge and plans Currently floating the idea of setting up a common app between us. Happy with our experimentation! Some things we've been trying: Equity investments, loans, dominant assurance contracts and retroactive funding Grantathon, office hours, feedback on Discord & site comments Less happy with our operations (wrt feedback and response times to applicants) Taking longer to support to individual grantees, or start new Manifund initiatives Please ping us if it's been a week and you haven't heard anything! Wise deactivated our account, making international payments more difficult/expensive. In cases where multiple regrantors may fund a project, we've observed a bit of "funding chicken" Grant of the month [$310k] Apollo Research This is our largest grant to date! Many of our regrantors were independently excited about Apollo; in the end, we coordinated between Tristan Hume, Evan Hubinger and Marcus Abramovitch to fund this. From Tristan: I'm very excited about Apollo based on a combination of the track record of it's founding employees and the research agenda they've articulated. Marius and Lee have published work that's significantly contributed to Anthropic's work on dictionary learning. I've also met both Marius and Lee and have confidence in them to do a good job with Apollo. Additionally, I'm very much a fan of alignment and dangerous capability evals as an area of research and think there's lots of room for more people to work on them. In terms of cost-effectiveness I like these research areas because they're ones I think are very tractable to approach from outside a major lab in a helpful way, while not taking large amounts of compute. I also think Apollo existing in London will allow them to hire underutilized talent that would have trouble getting a U.S. visa. New grants [$112k] Jesse Hoogland: Scoping Developmental Interpretability Jesse posted this through our open call: We propose a 6-month research project to assess the viability of Developmental Interpretability, a new AI alignment research agenda. "DevInterp" studies how phase transitions give rise to computational structure in neural networks, and offers a possible path to scalable interpretability tools. Though we have both empirical and theoretical reasons to believe that phase transitions dominate the training process, the details remain unclear. We plan to clarify the role of phase transitions by studying them in a variety of models combining techniques from Singular Learning Theory and Mechanistic Interpretability. In six months, we expect to have gathered enough evidence to confirm that DevInterp is a viable research program. If successful, we expect Developmental Interpretability to become one of the main branches of technical alignment research over the next few years. Rachel was excited about this project and considered setting up a dominance assurance contract to encourage regrants, but instead offered 10% matching; Evan took her up on this! [$60k] Dam and Pietro: Writeup on Agency and (Dis)Empowerment A regrant initiated by Evan: 6 months support for two people, Damiano and Pietro, to write a paper about (dis)empowerment. Its ultimate aim is to offer formal and operational notions of (dis)empowerment. For example, an inte...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: EA Funds organisational update: Open Philanthropy matching and distancing, published by calebp on August 2, 2023 on The Effective Altruism Forum. We want to communicate some changes that are happening at EA Funds, particularly on the EA Infrastructure Fund and the Long-Term Future Fund. In summary: EA Funds (particularly the EAIF and LTFF) and Open Philanthropy have historically had overlapping staff, and Open Phil has supported EA Funds, but we (staff at EA Funds and Open Philanthropy) are now trying to increase the separation between EA Funds and Open Philanthropy. In particular: The current chairs of the LTFF and the EAIF, who have also joined as staff members at Open Philanthropy, are planning to step down from their respective chair positions over the next several months. Max Daniel is going to step down as the EAIF's chair on August 2nd, and Asya Bergal is planning to step down as the LTFF's chair in October. To help transition EA Funds away from reliance on Open Philanthropy's financial support, Open Philanthropy is planning to match donations to the EA Infrastructure and Long-Term Future Fund at 2:1 rates, up to $3.5M each, over the next six months. The EAIF and LTFF have substantial funding gaps - we are looking to raise an additional $3.84M for the LTFF and $4.83M for the EAIF. over the next six months. By default, I expect, the LTFF to have ~$720k, and the EAIF to have ~$400k by default. Our relationship with Open Philanthropy EA Funds started in 2017 and was largely developed during CEA's time at Y Combinator. It spun out of CEA in 2020, though both CEA and EA Funds are part of the Effective Ventures Foundation. Last year, EA Funds moved over $35M towards high-impact projects through the Animal Welfare Fund (AWF), EA Infrastructure Fund (EAIF), Global Health and Development Fund (GHDF), and Long-Term Future Fund (LTFF). Over the last two years, the EAIF and LTFF used some overlapping resources with Open Philanthropy in the following ways: Over the last year, Open Philanthropy has contributed a substantial proportion of EAIF and LTFF budgets and has covered our entire operations budget.[1] They also made a sizable grant in February 2022. (You can see more detail on Open Philanthropy's website.) The chairs of the EAIF and LTFF both joined the Longtermist EA Community Growth team at Open Philanthropy and have worked in positions at EA Funds and Open Philanthropy simultaneously. (Asya Bergal joined the LTFF in June 2020, has been chair since February 2021, and joined Open Philanthropy in April 2021; Max Daniel joined the EAIF in March 2021, has been chair since mid-2021, and joined Open Philanthropy in November 2022.) As a board member of the Effective Ventures Foundation (UK), Claire Zabel, who is also the Senior Program Officer for EA Community Growth (Longtermism) at Open Philanthropy and supervises both Asya and Max, has regularly met with me throughout my tenure at EA Funds to hear updates on EA Funds and offer advice on various topics related to EA Funds (both day-to-day issues and higher-level organisation strategy). That said, I think it is worth noting that: The majority of funding for the LTFF has come from non-Open Philanthropy sources. Open Philanthropy as an organisation has limited visibility into our activities, though certain Open Philanthropy employees, particularly Max Daniel and Asya Bergal, have a lot of visibility into certain parts of EA Funds. Our grants supporting our operations and LTFF/EAIF grantmaking funds have had minimal restrictions. Since the shutdown of the FTX Future Fund, Open Phil and I have both felt more excited about building a grantmaking organisation that is legibly independent from Open Phil. Earlier this year, Open Phil staff reached out to me proposing some steps to make this happen, and have worked with me closely ...

The Lightspeed application asks:  “What impact will [your project] have on the world? What is your project's goal, how will you know if you've achieved it, and what is the path to impact?”LTFF uses an identical question, and SFF puts it even more strongly (“What is your organization's plan for improving humanity's long term prospects for survival and flourishing?”). I've applied to all three grants of these at various points, and I've never liked this question. It feels like it wants a grand narrative of an amazing, systemic project that will measurably move the needle on x-risk. But I'm typically applying for narrowly defined projects, like “Give nutrition tests to EA vegans and see if there's a problem”. I think this was a good project. I think this project is substantially more likely to pay off than underspecified alignment strategy research, and arguably has as good a long tail.  But when I look at “What impact will [my project] have on the world?” the project feels small and sad. I feel an urge to make things up, and express far more certainty for far more impact than I  believe. Then I want to quit, because lying is bad but listing my true beliefs feels untenable.I've gotten better at this over time, but I know other people with similar feelings, and I suspect it's a widespread issue (I encourage you to share your experience in the comments so we can start figuring that out).I should note that the pressure for grand narratives has good points; funders are in fact looking for VC-style megabits. I think that narrow projects are underappreciated, but for purposes of this post that's beside the point: I think many grantmakers are undercutting their own preferred outcomes by using questions that implicitly push for a grand narrative. I think they should probably change the form, but I also think we applicants can partially solve the problem by changing how we interact with the current forms.My goal here is to outline the problem, gesture at some possible solutions, and create a space for other people to share data. I didn't think about my solutions very long, I am undoubtedly missing a bunch and what I do have still needs workshopping, but it's a place to start. Source:https://www.lesswrong.com/posts/FNPXbwKGFvXWZxHGE/grant-applications-and-grand-narrativesNarrated for LessWrong by TYPE III AUDIO.Share feedback on this narration.[Curated Post] ✓

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Alignment Megaprojects: You're Not Even Trying to Have Ideas, published by NicholasKross on July 12, 2023 on LessWrong. Consider the state of funding for AI alignment. Is the field more talent-constrained, or funding-constrained? I think most existing researchers, if they take the AI-based extinction-risk seriously, think it's talent constrained. I think the bar-for-useful-contribution could be so high, that we loop back around to "we need to spend more money (and effort) on finding (and making) more talent". And the programs to do those may themselves be more funding-constrained than talent-constrained. Like, the 20th century had some really good mathematicians and physicists, and the US government spared little expense towards getting them what they needed, finding them, and so forth. Top basketball teams will "check up on anyone over 7 feet that's breathing". Consider how huge Von Neumann's expense account must've been, between all the consulting and flight tickets and car accidents. Now consider that we don't seem to have Von Neumanns anymore. There are caveats to at least that second point, but the overall problem structure still hasn't been "fixed". Things an entity with absurdly-greater funding (e.g. the US Department of Defense) could probably do, with their absurdly-greater funding and probably coordination power: Indefinitely-long-timespan basic minimum income for everyone who Coordinating, possibly by force, every AI alignment researcher and aspiring alignment researcher on Earth to move to one place that doesn't have high rents like the Bay. Possibly up to and including creating that place and making rent free for those who are accepted in. Enforce a global large-ML-training shutdown. An entire school system (or at least an entire network of universities, with university-level funding) focused on Sequences-style rationality in general and AI alignment in particular. Genetic engineering, focused-training-from-a-young-age, or other extreme "talent development" setups. Deeper, higher-budget investigations into how "unteachable" things like security mindset really are, and how deeply / quickly you can teach them. All of these at once. I think the big logistical barrier here is something like "LTFF is not the U,S government", or more precisely "nothing as crazy as these can be done 'on-the-margin' or with any less than the full funding". However, I think some of these could be scaled down into mere megaprojects or less. Like, if the training infrastructure is bottlenecked on trainers, then we need to fund indirect "training" work just to remove the bottleneck on the bottleneck of the problem. (Also, the bottleneck is going to move at least when you solve the current bottleneck, and also "on its own" as the entire world changes around you). Also... this might be the first list of ideas-in-precisely-this-category, on all of LessWrong/the EA Forum. (By which I mean "technical AI alignment research projects that you could fund, without having to think about the alignment problem itself in much detail beyond agreeing with 'doom could actually happen in my lifetime', if funding really wasn't the constraint".) Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Manifund is launching a new regranting program! We will allocate ~$2 million over the next six months based on the recommendations of our regrantors. Grantees can apply for funding through our site; we're also looking for additional regrantors and donors to join.What is regranting?Regranting is a funding model where a donor delegates grantmaking budgets to different individuals known as “regrantors”. Regrantors are then empowered to make grant decisions based on the objectives of the original donor.This model was pioneered by the FTX Future Fund; in a 2022 retro they considered regranting to be very promising at finding new projects and people to fund. More recently, Will MacAskill cited regranting as one way to diversify EA funding.What is Manifund?Manifund is the charitable arm of Manifold Markets. Some of our past work:Impact certificates, with Astral Codex Ten and the OpenPhil AI Worldviews ContestForecasting tournaments, with Charity Entrepreneurship and Clearer ThinkingDonating prediction market winnings to charity, funded by the Future FundHow does regranting on Manifund work?Our website makes the process simple, transparent, and fast:A donor contributes money to Manifold for Charity, our registered 501c3 nonprofitThe donor then allocates the money between regrantors of their choice. They can increase budgets for regrantors doing a good job, or pick out new regrantors who share the donor's values.Regrantors choose which opportunities (eg existing charities, new projects, or individuals) to spend their budgets on, writing up an explanation for each grant made.We expect most regrants to start with a conversation between the recipient and the regrantor, and after that, for the process to take less than two weeks.Alternatively, people looking for funding can post their project on the Manifund site. Donors and regrantors can then decide whether to fund it, similar to Kickstarter.The Manifund team screens the grant to make sure it is legitimate, legal, and aligned with our mission. If so, we approve the grant, which sends money to the recipient's Manifund account.The recipient withdraws money from their Manifund account to be used for their project.Differences from the Future Fund's regranting programAnyone can donate to regrantors. Part of what inspired us to start this program is how hard it is to figure out where to give as a longtermist donor—there's no GiveWell, no ACE, just a mass of opaque, hard-to-evaluate research orgs. Manifund's regranting infrastructure lets individual donors outsource their giving decisions to people they trust, who may be more specialized and more qualified at grantmaking.All grant information is public. This includes the identity of the regrantor and grant recipient, the project description, the grant size, and the regrantor's writeup. We strongly believe in transparency as it allows for meaningful public feedback, accountability of decisions, and establishment of regrantor track records.Almost everything is done through our website. This lets us move faster, act transparently, set good defaults, and encourage discourse about the projects in comment sections.We recognize that not all grants are suited for publishing; for now, we recommend sensitive grants apply to other donors (such as LTFF, SFF, OpenPhil).We're starting with less money. The Future [...]--- First published: July 5th, 2023 Source: https://forum.effectivealtruism.org/posts/RMXctNAksBgXgoszY/announcing-manifund-regrants Linkpost URL:https://manifund.org/rounds/regrants --- Narrated by TYPE III AUDIO. Share feedback on this narration.

Link to original articleWelcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Alignment Megaprojects: You're Not Even Trying to Have Ideas, published by NicholasKross on July 12, 2023 on LessWrong. Consider the state of funding for AI alignment. Is the field more talent-constrained, or funding-constrained? I think most existing researchers, if they take the AI-based extinction-risk seriously, think it's talent constrained. I think the bar-for-useful-contribution could be so high, that we loop back around to "we need to spend more money (and effort) on finding (and making) more talent". And the programs to do those may themselves be more funding-constrained than talent-constrained. Like, the 20th century had some really good mathematicians and physicists, and the US government spared little expense towards getting them what they needed, finding them, and so forth. Top basketball teams will "check up on anyone over 7 feet that's breathing". Consider how huge Von Neumann's expense account must've been, between all the consulting and flight tickets and car accidents. Now consider that we don't seem to have Von Neumanns anymore. There are caveats to at least that second point, but the overall problem structure still hasn't been "fixed". Things an entity with absurdly-greater funding (e.g. the US Department of Defense) could probably do, with their absurdly-greater funding and probably coordination power: Indefinitely-long-timespan basic minimum income for everyone who Coordinating, possibly by force, every AI alignment researcher and aspiring alignment researcher on Earth to move to one place that doesn't have high rents like the Bay. Possibly up to and including creating that place and making rent free for those who are accepted in. Enforce a global large-ML-training shutdown. An entire school system (or at least an entire network of universities, with university-level funding) focused on Sequences-style rationality in general and AI alignment in particular. Genetic engineering, focused-training-from-a-young-age, or other extreme "talent development" setups. Deeper, higher-budget investigations into how "unteachable" things like security mindset really are, and how deeply / quickly you can teach them. All of these at once. I think the big logistical barrier here is something like "LTFF is not the U,S government", or more precisely "nothing as crazy as these can be done 'on-the-margin' or with any less than the full funding". However, I think some of these could be scaled down into mere megaprojects or less. Like, if the training infrastructure is bottlenecked on trainers, then we need to fund indirect "training" work just to remove the bottleneck on the bottleneck of the problem. (Also, the bottleneck is going to move at least when you solve the current bottleneck, and also "on its own" as the entire world changes around you). Also... this might be the first list of ideas-in-precisely-this-category, on all of LessWrong/the EA Forum. (By which I mean "technical AI alignment research projects that you could fund, without having to think about the alignment problem itself in much detail beyond agreeing with 'doom could actually happen in my lifetime', if funding really wasn't the constraint".) Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Benefits of being rejected from CEA's Online team, published by Ben West on July 3, 2023 on The Effective Altruism Forum. Summary: If you apply and are rejected, then I (and presumably other hiring managers) will often either not remember you — or actually be enthusiastic about you (e.g if you made it to later hiring rounds) if anyone asks in the future. People who have been rejected from the CEA Online team have gotten concrete job opportunities that they would not have gotten if they had never applied If you're rejected, I can also give you advice and connect you to other opportunities. Rejection is hard, but fear of it shouldn't stop you from applying. Note: I wrote this year ago and temporal references like “the most recent” should be interpreted relative to that. Importantly: I no longer am on the Online team, although I think this post is still roughly accurate for them. I sometimes talk to people who are nervous about applying to EA organizations because they think a rejection could damage their chances at not just the organization they apply to but all EA organizations. This fear is not completely ungrounded – EA is a small community, and hiring managers do occasionally talk to each other about candidates. As with most worries with how others think of you though, "You probably wouldn't worry about what people think of you if you could know how seldom they do": 100+ people apply to CEA per month, and my memory is pretty bad. The people grading applications will probably not remember people whose applications they reject, especially if that happens early on, and if it happens later, that likely means that they saw something promising. (There are also other costs to applying, like your time and energy.) I wanted to point out though that being rejected as a candidate, particularly if you make it to the final round of a hiring process, can actually be substantially positive. Here are some things that happened to rejected candidates in some of the hiring rounds I ran: Hired by CEA as a contractor for a position similar to what they applied to (the position we were hiring for ended up needing slightly more than 1 FTE of work, so we hired them to do some of the overfill) Received a grant to quit their job and work independently on something similar to what they applied to after I encouraged them to do so and recommended the grant Repeatedly consulted CEA on their area of expertise as a volunteer (though we offered to pay them, they just declined payment) Received a grant from LTFF to skill up after I endorsed them, based on what I learned during their hiring process I also try to give useful feedback to candidates who are rejected. Here is an email I recently received from a rejected applicant, which Carly (the applicant) kindly agreed to share publicly: Hi Ben, Hope you are well! I was in the Project Coordinator search at CEA a few months ago and wanted to drop you a quick note of gratitude. I want to thank you for your part in helping to transition my career into EA. I was very hopeful about getting the role at CEA and can easily imagine a scenario where a typical rejection letter - short, generic, or dismissive - may have dampened my enthusiasm and at worst lowered my spirits enough to not go to EAGx Boston the following day. Your rejection letter, however, was so insightful and encouraging that it had the opposite effect. You motivated me to keep learning and networking and to go to the conference which started the chain of events that led me to a position that I'm very excited to start at Alvea in a few weeks. All of this is just to say thanks and no need for a reply! I know those letters take time and are not expected or necessary for unsuccessful candidates but they make an impact! Carly Tryens People in the EA network tend to be both inexperienced and self-mo...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Grant applications and grand narratives, published by Elizabeth on July 2, 2023 on LessWrong. The Lightspeed application asks: “What impact will [your project] have on the world? What is your project's goal, how will you know if you've achieved it, and what is the path to impact?” LTFF uses an identical question, and SFF puts it even more strongly (“What is your organization's plan for improving humanity's long term prospects for survival and flourishing?”). I've applied to all three grants of these at various points, and I've never liked this question. It feels like it wants a grand narrative of an amazing, systemic project that will measurably move the needle on x-risk. But I'm typically applying for narrowly defined projects, like “Give nutrition tests to EA vegans and see if there's a problem”. I think this was a good project. I think this project is substantially more likely to pay off than underspecified alignment strategy research, and arguably has as good a long tail. But when I look at “What impact will [my project] have on the world?” the project feels small and sad. I feel an urge to make things up, and express far more certainty for far more impact than I believe. Then I want to quit, because lying is bad but listing my true beliefs feels untenable. I've gotten better at this over time, but I know other people with similar feelings, and I suspect it's a widespread issue (I encourage you to share your experience in the comments so we can start figuring that out). I should note that the pressure for grand narratives has good points; funders are in fact looking for VC-style megabits. I think that narrow projects are underappreciated, but for purposes of this post that's beside the point: I think many grantmakers are undercutting their own preferred outcomes by using questions that implicitly push for a grand narrative. I think they should probably change the form, but I also think we applicants can partially solve the problem by changing how we interact with the current forms. My goal here is to outline the problem, gesture at some possible solutions, and create a space for other people to share data. I didn't think about my solutions very long, I am undoubtedly missing a bunch and what I do have still needs workshopping, but it's a place to start. More on the costs of the question Pushes away the most motivated people Even if you only care about subgoal G instrumentally, G may be best accomplished by people who care about it for its own sake. Community building (real building, not a euphemism for recruitment) benefits from knowing the organizer cares about participants and the community as people and not just as potential future grist for the x-risk mines. People repeatedly recommended a community builder friend of mine apply for funding, but they struggled because they liked organizing for its own sake, and justifying it in x-risk terms felt bad. [Although there are also downsides to organizers with sufficiently bad epistemics.] Additionally, if G is done by someone who cares about it for its own sake, then it doesn't need to be done by someone whose motivated by x-risk. Highly competent, x-risk motivated people are rare and busy, and we should be delighted by opportunities to take things off their plate. Vulnerable to grift You know who's really good at creating exactly the grand narrative a grantmaker wants to hear? People who feel no constraint to be truthful. You can try to compensate for this by looking for costly signals of loyalty or care, but those have their own problems. Punishes underconfidence Sometimes people aren't grifting, they really really believe in their project, but they're wrong. Hopefully grantmakers are pretty good at filtering out those people. But it's fairly hard to correct for people who are underconfident, and impossible to cor...

Link to original articleWelcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Grant applications and grand narratives, published by Elizabeth on July 2, 2023 on LessWrong. The Lightspeed application asks: “What impact will [your project] have on the world? What is your project's goal, how will you know if you've achieved it, and what is the path to impact?” LTFF uses an identical question, and SFF puts it even more strongly (“What is your organization's plan for improving humanity's long term prospects for survival and flourishing?”). I've applied to all three grants of these at various points, and I've never liked this question. It feels like it wants a grand narrative of an amazing, systemic project that will measurably move the needle on x-risk. But I'm typically applying for narrowly defined projects, like “Give nutrition tests to EA vegans and see if there's a problem”. I think this was a good project. I think this project is substantially more likely to pay off than underspecified alignment strategy research, and arguably has as good a long tail. But when I look at “What impact will [my project] have on the world?” the project feels small and sad. I feel an urge to make things up, and express far more certainty for far more impact than I believe. Then I want to quit, because lying is bad but listing my true beliefs feels untenable. I've gotten better at this over time, but I know other people with similar feelings, and I suspect it's a widespread issue (I encourage you to share your experience in the comments so we can start figuring that out). I should note that the pressure for grand narratives has good points; funders are in fact looking for VC-style megabits. I think that narrow projects are underappreciated, but for purposes of this post that's beside the point: I think many grantmakers are undercutting their own preferred outcomes by using questions that implicitly push for a grand narrative. I think they should probably change the form, but I also think we applicants can partially solve the problem by changing how we interact with the current forms. My goal here is to outline the problem, gesture at some possible solutions, and create a space for other people to share data. I didn't think about my solutions very long, I am undoubtedly missing a bunch and what I do have still needs workshopping, but it's a place to start. More on the costs of the question Pushes away the most motivated people Even if you only care about subgoal G instrumentally, G may be best accomplished by people who care about it for its own sake. Community building (real building, not a euphemism for recruitment) benefits from knowing the organizer cares about participants and the community as people and not just as potential future grist for the x-risk mines. People repeatedly recommended a community builder friend of mine apply for funding, but they struggled because they liked organizing for its own sake, and justifying it in x-risk terms felt bad. [Although there are also downsides to organizers with sufficiently bad epistemics.] Additionally, if G is done by someone who cares about it for its own sake, then it doesn't need to be done by someone whose motivated by x-risk. Highly competent, x-risk motivated people are rare and busy, and we should be delighted by opportunities to take things off their plate. Vulnerable to grift You know who's really good at creating exactly the grand narrative a grantmaker wants to hear? People who feel no constraint to be truthful. You can try to compensate for this by looking for costly signals of loyalty or care, but those have their own problems. Punishes underconfidence Sometimes people aren't grifting, they really really believe in their project, but they're wrong. Hopefully grantmakers are pretty good at filtering out those people. But it's fairly hard to correct for people who are underconfident, and impossible to cor...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Launching Lightspeed Grants (Apply by July 6th), published by habryka on June 7, 2023 on LessWrong. Lightspeed Grants provides fast funding for projects that help humanity flourish among the stars. The application is minimal and grant requests of any size ($5k - $5M) are welcome. Budget is $5M for this grant round, and (probably) more in future rounds. Applications close in 30 days (July 6th). Opt into our venture grants program to get a response within 14 days (otherwise get a response in 30-60 days, around the start of August). Apply here. The application should only take 1-2 hours! If you want to join as a funder, send us an email at funds@lightspeedgrants.org. Is the application really only 2 hours though? Often, applicants get nervous about grant applications and spend a lot more time than they need to on them, or get overwhelmed and procrastinate on applying. We really just want you to spell out some basic information about your project in a plain way and think this is doable in the 1-2 hour timeframe. If you're worried about overthinking things, we'll have application co-working sessions and office hours every Thursday of July between noon and 2PM PT. If you think you might procrastinate on the application or get stuck in the weeds and spend a ton of unnecessary time on it, you can join one and fill out the application on the call, plus ask questions. Add the co-working to your calendar here! Who runs Lightspeed Grants? Lightspeed grants is run by Lightcone Infrastructure. Applications are evaluated by ~5 evaluators selected for their general reasoning ability and networks including applicants/references, and are chosen in collaboration with our funders. Our primary funder for this round is Jaan Tallinn. Applications are open to individuals, nonprofits, and projects that don't have a charitable sponsor. When necessary, Hack Club Bank provides fiscal sponsorship for successful applications. Why? Improved grantee experience I've been doing various forms of grantmaking for 5+ years, both on the Long Term Future Fund and the Survival and Flourishing Fund, and I think it's possible to do better, both in grant quality and applicant-experience. Applications tend to be unnecessarily complicated to fill out, and it can take months to get a response from existing grantmakers, often without any intermediate updates. Different donors also often end up playing donor-chicken where donors wait to fund an organization to see whether other donors will fund it first, delaying decisions further. This period of funding uncertainty can have large effects on organizational strategy, and also makes experimenting with smaller projects much more costly, since each grant application might be associated with weeks to months of funding uncertainty, meaning it takes months to go from an idea to execution, or to go from "the beta test turned out well" to moving forward with the project. My goal is to have an application process that requires minimal additional work beyond "explain why your project is a good idea and you are a good fit for it" and where most responses happen within 2 weeks. This round, we're aiming for something somewhat less ambitious while we find our footing, and are planning to get back to people reliably in less than 60 days, with the ability to opt-into a 14-day response process. Improved funder experience Currently funders either have to find hyper-local grant opportunities among their friends and acquaintances and fund them directly, start something like their own foundation, or give up control over their funds and donate money to something like the Long Term Future Fund, which will then fund some portfolio of grants that the funder has relatively little insight into (especially with the decrease in grant writeups from the LTFF and EAIF). The Lightspeed Grants proce...

Link to original articleWelcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Launching Lightspeed Grants (Apply by July 6th), published by habryka on June 7, 2023 on LessWrong. Lightspeed Grants provides fast funding for projects that help humanity flourish among the stars. The application is minimal and grant requests of any size ($5k - $5M) are welcome. Budget is $5M for this grant round, and (probably) more in future rounds. Applications close in 30 days (July 6th). Opt into our venture grants program to get a response within 14 days (otherwise get a response in 30-60 days, around the start of August). Apply here. The application should only take 1-2 hours! If you want to join as a funder, send us an email at funds@lightspeedgrants.org. Is the application really only 2 hours though? Often, applicants get nervous about grant applications and spend a lot more time than they need to on them, or get overwhelmed and procrastinate on applying. We really just want you to spell out some basic information about your project in a plain way and think this is doable in the 1-2 hour timeframe. If you're worried about overthinking things, we'll have application co-working sessions and office hours every Thursday of July between noon and 2PM PT. If you think you might procrastinate on the application or get stuck in the weeds and spend a ton of unnecessary time on it, you can join one and fill out the application on the call, plus ask questions. Add the co-working to your calendar here! Who runs Lightspeed Grants? Lightspeed grants is run by Lightcone Infrastructure. Applications are evaluated by ~5 evaluators selected for their general reasoning ability and networks including applicants/references, and are chosen in collaboration with our funders. Our primary funder for this round is Jaan Tallinn. Applications are open to individuals, nonprofits, and projects that don't have a charitable sponsor. When necessary, Hack Club Bank provides fiscal sponsorship for successful applications. Why? Improved grantee experience I've been doing various forms of grantmaking for 5+ years, both on the Long Term Future Fund and the Survival and Flourishing Fund, and I think it's possible to do better, both in grant quality and applicant-experience. Applications tend to be unnecessarily complicated to fill out, and it can take months to get a response from existing grantmakers, often without any intermediate updates. Different donors also often end up playing donor-chicken where donors wait to fund an organization to see whether other donors will fund it first, delaying decisions further. This period of funding uncertainty can have large effects on organizational strategy, and also makes experimenting with smaller projects much more costly, since each grant application might be associated with weeks to months of funding uncertainty, meaning it takes months to go from an idea to execution, or to go from "the beta test turned out well" to moving forward with the project. My goal is to have an application process that requires minimal additional work beyond "explain why your project is a good idea and you are a good fit for it" and where most responses happen within 2 weeks. This round, we're aiming for something somewhat less ambitious while we find our footing, and are planning to get back to people reliably in less than 60 days, with the ability to opt-into a 14-day response process. Improved funder experience Currently funders either have to find hyper-local grant opportunities among their friends and acquaintances and fund them directly, start something like their own foundation, or give up control over their funds and donate money to something like the Long Term Future Fund, which will then fund some portfolio of grants that the funder has relatively little insight into (especially with the decrease in grant writeups from the LTFF and EAIF). The Lightspeed Grants proce...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Japan AI Alignment Conference Postmortem, published by Chris Scammell on April 20, 2023 on LessWrong. The goal Conjecture collaborated with Araya to host a two day AI Safety conference in Japan, the first Japan AI Alignment Conference (“JAC2023”). Our aim was to put together a small 30-40 person event to generate excitement around alignment for researchers in Japan and fuel new ideas for research topics. Wired Japan covered the event and interviewed Ryota Kanai (CEO of ARAYA), who co-organized it with us, here (original in JP). The conference agenda was broken into four sections that aimed to progress deeper into alignment as the weekend went on (full agenda available here): Saturday morning focused on creating common knowledge about AI safety and alignment. Saturday afternoon focused on clarifying unaddressed questions participants had about AI alignment and moving towards thematic discussions. Sunday morning focused on participant-driven content, with research talks in one room and opportunities for open discussion and networking in the other. Sunday afternoon focused on bringing all participants together to discuss concrete takeaways. While AI Safety is a discussion subject in Japan, AI alignment ideas have received very little attention. We organized the conference because we were optimistic about the reception to alignment ideas in Japan, having found on previous trips to Japan that researchers there were receptive and interested in learning more. In the best case, we hoped the conference could plant seeds for an organic AI alignment conversation to start in Japan. In the median case, we hoped to meet 2-3 sharp researchers who were eager to work directly on the alignment problem and contribute new ideas to the field. Now that the conference is over, we're left wondering how successful we were in raising awareness of alignment issues in Japan and fostering new research directions. What went well? By the aims above, the event was a success. We had a total of 65 participants, including 21 from the West, 27 from Japan, and 17 online attendees. We were pleasantly surprised by the amount of interest generated by the event, and had to turn down several participants as we reached capacity. We are grateful to LTFF for having supported the event via a grant, which allowed us to cover event costs and reimburse travel and accommodation for some participants who would not otherwise have come. While it is too early to know whether or not the conference had a lasting impact, there seems to be some traction. CEA organizers Anneke Pogarell and Moon Nagai and other conference participants created the AI Alignment Japan Slack channel, which has nearly 150 members. Some participants have begun working on translating alignment-related texts into Japanese. Others have begun to share more alignment-related content on social media, or indicated that they are discussing the subject with their organizations. Some participants are planning to apply for grant funding to continue independent research. Conjecture is in talks with two researchers interested in pursuing research projects we think are helpful, and ARAYA has hired at least one researcher to continue working on alignment full-time. As for the event itself, we conducted a survey after the event and found that 91% of respondents would recommend the conference to a friend, and that overall participant satisfaction was high. The "networking" aspect of the conference was rated as the most valuable component, but all other sections received a majority score of 4 out of 5, indicating that the content was received positively. Nearly all respondents from Japan indicated their knowledge of alignment had improved from the event. When asked how the conference had impacted their thoughts on the subject, the majority expressed a sense of urgen...

Link to original articleWelcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Japan AI Alignment Conference Postmortem, published by Chris Scammell on April 20, 2023 on LessWrong. The goal Conjecture collaborated with Araya to host a two day AI Safety conference in Japan, the first Japan AI Alignment Conference (“JAC2023”). Our aim was to put together a small 30-40 person event to generate excitement around alignment for researchers in Japan and fuel new ideas for research topics. Wired Japan covered the event and interviewed Ryota Kanai (CEO of ARAYA), who co-organized it with us, here (original in JP). The conference agenda was broken into four sections that aimed to progress deeper into alignment as the weekend went on (full agenda available here): Saturday morning focused on creating common knowledge about AI safety and alignment. Saturday afternoon focused on clarifying unaddressed questions participants had about AI alignment and moving towards thematic discussions. Sunday morning focused on participant-driven content, with research talks in one room and opportunities for open discussion and networking in the other. Sunday afternoon focused on bringing all participants together to discuss concrete takeaways. While AI Safety is a discussion subject in Japan, AI alignment ideas have received very little attention. We organized the conference because we were optimistic about the reception to alignment ideas in Japan, having found on previous trips to Japan that researchers there were receptive and interested in learning more. In the best case, we hoped the conference could plant seeds for an organic AI alignment conversation to start in Japan. In the median case, we hoped to meet 2-3 sharp researchers who were eager to work directly on the alignment problem and contribute new ideas to the field. Now that the conference is over, we're left wondering how successful we were in raising awareness of alignment issues in Japan and fostering new research directions. What went well? By the aims above, the event was a success. We had a total of 65 participants, including 21 from the West, 27 from Japan, and 17 online attendees. We were pleasantly surprised by the amount of interest generated by the event, and had to turn down several participants as we reached capacity. We are grateful to LTFF for having supported the event via a grant, which allowed us to cover event costs and reimburse travel and accommodation for some participants who would not otherwise have come. While it is too early to know whether or not the conference had a lasting impact, there seems to be some traction. CEA organizers Anneke Pogarell and Moon Nagai and other conference participants created the AI Alignment Japan Slack channel, which has nearly 150 members. Some participants have begun working on translating alignment-related texts into Japanese. Others have begun to share more alignment-related content on social media, or indicated that they are discussing the subject with their organizations. Some participants are planning to apply for grant funding to continue independent research. Conjecture is in talks with two researchers interested in pursuing research projects we think are helpful, and ARAYA has hired at least one researcher to continue working on alignment full-time. As for the event itself, we conducted a survey after the event and found that 91% of respondents would recommend the conference to a friend, and that overall participant satisfaction was high. The "networking" aspect of the conference was rated as the most valuable component, but all other sections received a majority score of 4 out of 5, indicating that the content was received positively. Nearly all respondents from Japan indicated their knowledge of alignment had improved from the event. When asked how the conference had impacted their thoughts on the subject, the majority expressed a sense of urgen...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Orthogonal: A new agent foundations alignment organization, published by carado on April 19, 2023 on LessWrong. We are putting together Orthogonal, a non-profit alignment research organization focused on agent foundations, based in Europe. We are pursuing the formal alignment flavor of agent foundations in order to solve alignment in a manner which would scale to superintelligence in order to robustly overcome AI risk. If we can afford to, we also intend to hire agent foundations researchers which, while not directly aimed at such an agenda, produce output which is likely to be instrumental to it, such as finding useful "true names". Within this framework, our foremost agenda for the moment is QACI, and we expect to make significant progress on ambitious alignment within short timelines (months to years) and produce a bunch of dignity in the face of high existential risk. Our goal is to be the kind of object-level research which cyborgism would want to accelerate. And when other AI organizations attempt to "buy time" by restraining their AI systems, we intend to be the research that this time is being bought for. We intend to exercise significant caution with regards to AI capability exfohazards: Conjecture's policy document offers a sensible precedent for handling matters of internal sharing, and locked posts are a reasonable default for publishing our content to the outside. Furthermore, we would like to communicate about research and strategy with MIRI, whose model of AI risk we largely share and who we percieve to have the most experience with non-independent agent foundations research. Including myself — Tamsin Leake, founder of Orthogonal and LTFF-funded AI alignment researcher — we have several promising researchers intending to work fulltime, and several more who are considering that option. I expect that we will find more researchers excited to join our efforts in solving ambitious alignment. If you are interested in such a position, we encourage you to get acquainted with our research agenda — provided we get adequate funding, we hope to run a fellowship where people who have demonstrated interest in this research can work alongside us in order to test their fit as a fellow researcher at Orthogonal. We might also be interested in people who could help us with engineering, management, and operations. And, in order to make all of that happen, we are looking for funding. For these matters or any other inquiries, you can get in touch with us at contact@orxl.org. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Link to original articleWelcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Orthogonal: A new agent foundations alignment organization, published by carado on April 19, 2023 on LessWrong. We are putting together Orthogonal, a non-profit alignment research organization focused on agent foundations, based in Europe. We are pursuing the formal alignment flavor of agent foundations in order to solve alignment in a manner which would scale to superintelligence in order to robustly overcome AI risk. If we can afford to, we also intend to hire agent foundations researchers which, while not directly aimed at such an agenda, produce output which is likely to be instrumental to it, such as finding useful "true names". Within this framework, our foremost agenda for the moment is QACI, and we expect to make significant progress on ambitious alignment within short timelines (months to years) and produce a bunch of dignity in the face of high existential risk. Our goal is to be the kind of object-level research which cyborgism would want to accelerate. And when other AI organizations attempt to "buy time" by restraining their AI systems, we intend to be the research that this time is being bought for. We intend to exercise significant caution with regards to AI capability exfohazards: Conjecture's policy document offers a sensible precedent for handling matters of internal sharing, and locked posts are a reasonable default for publishing our content to the outside. Furthermore, we would like to communicate about research and strategy with MIRI, whose model of AI risk we largely share and who we percieve to have the most experience with non-independent agent foundations research. Including myself — Tamsin Leake, founder of Orthogonal and LTFF-funded AI alignment researcher — we have several promising researchers intending to work fulltime, and several more who are considering that option. I expect that we will find more researchers excited to join our efforts in solving ambitious alignment. If you are interested in such a position, we encourage you to get acquainted with our research agenda — provided we get adequate funding, we hope to run a fellowship where people who have demonstrated interest in this research can work alongside us in order to test their fit as a fellow researcher at Orthogonal. We might also be interested in people who could help us with engineering, management, and operations. And, in order to make all of that happen, we are looking for funding. For these matters or any other inquiries, you can get in touch with us at contact@orxl.org. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Apply to >30 AI safety funders in one application with the Nonlinear Network, published by Drew Spartz on April 12, 2023 on The Effective Altruism Forum. Nonlinear spoke to dozens of earn-to-givers and a common sentiment was, "I want to fund good AI safety-related projects, but I don't know where to find them." At the same time, applicants don't know how to find them either. And would-be applicants are often aware of just one or two funders - some think it's “LTFF or bust” - causing many to give up before they've started, demoralized, because fundraising seems too hard. As a result, we're trying an experiment to help folks get in front of donors and vice versa. In brief: Looking for funding? Why apply to just one funder when you can apply to dozens? If you've already applied for EA funding, simply paste your existing application. We'll share it with relevant funders (~30 so far) in our network. You can apply if you're still waiting to hear from other funders. This way, instead of having to awkwardly ask dozens of people and get rejected dozens of times (if you can even find the funders), you can just send in the application you already made. We're also accepting non-technical projects relevant to AI safety (e.g. meta, forecasting, field-building, etc.) Application deadline: May 17, 2023. Looking for projects to fund? Apply to join the funding round by May 17, 2023. Soon after, we'll share access to a database of applications relevant to your interests (e.g. interpretability, moonshots, forecasting, field-building, novel research directions, etc). If you'd like to fund any projects, you can reach out to applicants directly, or we can help coordinate. This way, you avoid the awkwardness of directly rejecting applicants, and don't get inundated by people trying to “sell” you. Inspiration for this project When the FTX crisis broke, we quickly spun up the Nonlinear Emergency Fund to help provide bridge grants to tide people over until the larger funders could step in. Instead of making all the funding decisions ourselves, we put out a call to other funders/earn-to-givers. Scott Alexander connected us with a few dozen funders who reached out to help, and we created a Slack to collaborate. We shared every application (that consented) in an Airtable with around 30 other donors. This led to a flurry of activity as funders investigated applications. They collaborated on diligence, and grants were made that otherwise wouldn't have happened. Some funders, like Scott, after seeing our recommendations, preferred to delegate decisions to us, but others preferred to make their own decisions. Collectively, we rapidly deployed roughly $500,000 - far more than we initially expected. The biggest lesson we learned: openly sharing applications with funders was high leverage - possibly leading to four times as many people receiving funding and 10 times more donations than would have happened if we hadn't shared. If you've been thinking about raising money for your project idea, we encourage you to do it now. Push through your imposter syndrome because, as Leopold Aschenbrenner said, nobody's on the ball on AGI alignment. Another reason to apply: we've heard from EA funders that they don't get enough applications, so you should have a low bar for applying - many fund over 50% of applications they receive (SFF, LTFF, EAIF). Since the Nonlinear Network is a diverse set of funders, you can apply for a grant size anywhere between single digit thousands to single digit millions. Note: We're aware of many valid critiques of this idea, but we're keeping this post short so we actually ship it. We're starting with projects related to AI safety because our timelines are short, but if this is successful, we plan to expand to the other cause areas. Apply here. Reminder that you can listen to LessWrong ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Reflections on my 5-month AI alignment upskilling grant, published by Jay Bailey on December 28, 2022 on The Effective Altruism Forum. Five months ago, I received a grant from the Long Term Future Fund to upskill in AI alignment. As of a few days ago, I was invited to Berkeley for two months of full-time alignment research under Owain Evans's stream in the SERIMATS program. This post is about how I got there. The post is partially a retrospective for myself, and partially a sketch of the path I took so that others can decide if it's right for them. This post was written relatively quickly - I'm happy to answer more questions via PM or in the comments. Summary I was a software engineer for 3-4 years with little to no ML experience before I was accepted for my grant. I did a bunch of stuff around fundamental ML maths, understanding RL and transformers, and improving my alignment understanding. Having tutors, getting feedback on my plan early on, and being able to pivot as I went were all very useful for not getting stuck doing stuff that was no longer useful. I probably wouldn't have gotten into SERIMATS without that ability to pivot midway through. After SERIMATS, I want to finish off the last part of the grant while I find work, then start work as a Research Engineer at an alignment organisation. If in doubt, put in an application! My Background My background is more professional and less academic than most. Until I was 23, I didn't do much of anything - then I got a Bachelor of Computer Science from a university ranked around 1,000th, with little maths and no intent to study ML at all, let alone alignment. It was known for strong graduate employment though, so I went straight into industry from there. I had 3.5 years of software engineering experience (1.5 at Amazon, 2 as a senior engineer at other jobs) before applying for the LTFF grant. I had no ML experience at the time, besides being halfway through doing the fast.ai course in my spare time. Not going to lie, seeing how many Top-20 university PhD students I was sharing my cohort with (At least three!) was a tad intimidating - but I made it in the end, so industry experience clearly has a role to play as well. Grant The details of the grant are one of the main reasons I wrote this - I've been asked for 1:1's and details on this at least three times in the last six months, and if you get asked something from at least three different people, it might be worth writing it up and sharing it around. Firstly, the process. Applying for the grant is pretty painless. As long as you have a learning plan already in place, the official guidance is to take 1-2 hours on it. I took a bit longer, polishing it more than required. I later found out my plan was more detailed than it probably had to be. In retrospect, I think my level of detail was good, but I spent too much time editing. AI Safety Support helped me with administration. The main benefit that I got from it was that the tutoring and compute money was tax free (since I didn't get the money personally, rather I used a card they provided me) and I didn't have to worry about tax withholding throughout the year. Secondly, the money. I agonized over how much money to ask for. This took me days. I asked myself how much I really needed, then I asked myself how much I would actually accept gladly with no regrets, then I balked at those numbers, even knowing that most people ask for too little, not too much. I still balk at the numbers, to be honest, but it would have been so much easier to write this if I had other grants to go off. So, in the interest of transparency and hopefully preventing someone else going through the same level of anguish, I'm sharing the full text of my grant request, including money requested (in Australian dollars, but you can always convert it) here....

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Reflections on my 5-month alignment upskilling grant, published by Jay Bailey on December 27, 2022 on LessWrong. Five months ago, I received a grant from the Long Term Future Fund to upskill in AI alignment. As of a few days ago, I was invited to Berkeley for two months of full-time alignment research under Owain Evans's stream in the SERIMATS program. This post is about how I got there. The post is partially a retrospective for myself, and partially a sketch of the path I took so that others can decide if it's right for them. This post was written relatively quickly - I'm happy to answer more questions via PM or in the comments. Summary I was a software engineer for 3-4 years with little to no ML experience before I was accepted for my grant. I did a bunch of stuff around fundamental ML maths, understanding RL and transformers, and improving my alignment understanding. Having tutors, getting feedback on my plan early on, and being able to pivot as I went were all very useful for not getting stuck doing stuff that was no longer useful. I probably wouldn't have gotten into SERIMATS without that ability to pivot midway through. After SERIMATS, I want to finish off the last part of the grant while I find work, then start work as a Research Engineer at an alignment organisation. If in doubt, put in an application! My Background My background is more professional and less academic than most. Until I was 23, I didn't do much of anything - then I got a Bachelor of Computer Science from a university ranked around 1,000th, with little maths and no intent to study ML at all, let alone alignment. It was known for strong graduate employment though, so I went straight into industry from there. I had 3.5 years of software engineering experience (1.5 at Amazon, 2 as a senior engineer at other jobs) before applying for the LTFF grant. I had no ML experience at the time, besides being halfway through doing the fast.ai course in my spare time. Not going to lie, seeing how many Top-20 university PhD students I was sharing my cohort with (At least three!) was a tad intimidating - but I made it in the end, so industry experience clearly has a role to play as well. Grant The details of the grant are one of the main reasons I wrote this - I've been asked for 1:1's and details on this at least three times in the last six months, and if you get asked something from at least three different people, it might be worth writing it up and sharing it around. Firstly, the process. Applying for the grant is pretty painless. As long as you have a learning plan already in place, the official guidance is to take 1-2 hours on it. I took a bit longer, polishing it more than required. I later found out my plan was more detailed than it probably had to be. In retrospect, I think my level of detail was good, but I spent too much time editing. AI Safety Support helped me with administration. The main benefit that I got from it was that the tutoring and compute money was tax free (since I didn't get the money personally, rather I used a card they provided me) and I didn't have to worry about tax withholding throughout the year. Secondly, the money. I agonized over how much money to ask for. This took me days. I asked myself how much I really needed, then I asked myself how much I would actually accept gladly with no regrets, then I balked at those numbers, even knowing that most people ask for too little, not too much. I still balk at the numbers, to be honest, but it would have been so much easier to write this if I had other grants to go off. So, in the interest of transparency and hopefully preventing someone else going through the same level of anguish, I'm sharing the full text of my grant request, including money requested (in Australian dollars, but you can always convert it) here. Personal embarrassmen...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Diversification is Underrated, published by Justis on November 17, 2022 on The Effective Altruism Forum. Note: This is not an FTX post, and I don't think its content hinges on current events. Also - though this is probably obvious - I'm speaking in a strictly personal capacity. Formal optimization problems often avail themselves of one solution - there can be multiple optima, but by default there tends to be one optimum for any given problem setup, and the highest expected value move is just to dump everything into that optimum. As a community, we tend to enjoy framing things as formal optimization problems. This is pretty good! But the thing about formal problem setups is they encode lots of assumptions, and those assumptions can have several degrees of freedom. Sometimes the assumptions are just plain qualitative, where quantifying them misses the point; the key isn't to just add another order-of-magnitude (or three) variable to express uncertainty. Rather, the key is to adopt a portfolio approach such that you're hitting optima or near-optima under a variety of plausible assumptions, even mutually exclusive ones. This isn't a new idea. In various guises and on various scales, it's called moral parliament, buckets, cluster thinking, or even just plain hedging. As a community, to our credit, we do a lot of this stuff. But I think we could do more, and be more confident and happy about it. Case study: me I do/have done the following things, that are likely EA-related: Every month, I donate 10% of my pre-tax income to the Against Malaria Foundation. I also donate $100 to Compassion in World Farming, mostly because I feel bad about eating meat. In my spare time, I provide editing services to various organizations as a contractor. The content I edit is often informed by a longtermist perspective, and the modal topic is probably AI safety. I once was awarded (part of a) LTFF (not FTX, the EA Funds one) grant, editing writeups on current cutting-edge AI safety research and researchers. Case study from a causes perspective On a typical longtermist view, my financial donations don't make that much sense - they're morally fine, but it'd be dramatically better in expectation to donate toward reducing x-risk. On a longtermist-skeptical view, the bulk of my editing doesn't accomplish much for altruistic purposes. It's morally fine, but it'd be better to polish general outreach communications for the more legible global poverty and health sector. And depending on how you feel about farmed animals, that smaller piece of the pie could dwarf everything else (even just the $100 a month is plausibly saving more chickens from bad lives than my AMF donations save human lives), or irrelevant (if you don't care about chicken welfare basically at all). I much prefer my situation to a more "aligned" situation, where all my efforts go the same single direction. It's totally plausible to me that work being done right now on AI safety makes a really big difference for how well things go in the next couple decades. It's also plausible to me that none of it matters, either because we're doomed in any case or because our current trajectory is just basically fine. Similarly, it's plausible to me (though I think unlikely) that I learn that AMF's numbers are super inflated somehow, or that its effectiveness collapsed and nobody bothered to check. And it's plausible that in 20 years, we will have made sufficient progress in global poverty and health that there no longer exist donation opportunities in the space as high leverage as there are right now, and so now is a really important time. So I'm really happy to just do both. I don't have quantitative credences here, though I'm normally a huge fan of those. I just don't think they work that well for the outside view of the portfolio approach - I've ...

As we head into the 2022 Midterm Election we invited Kelly Dittmar to talk to us about the status of women in politics. Kelly is an Associate Professor of Political Science at Rutgers University–Camden; and Director of Research and Scholar at the Center for American Women and Politics (CAWP) at the Eagleton Institute of Politics. She is the co-author of A Seat at the Table: Congresswomen's Perspectives on Why Their Representation Matters and author of Navigating Gendered Terrain: Stereotypes and Strategy in Political Campaigns . Dittmar's research focuses on gender and American political institutions. Dittmar was an American Political Science Association (APSA) Congressional Fellow from 2011 to 2012. At CAWP, she manages national research projects, helps to develop and implement CAWP's research agenda, and contributes to CAWP reports, publications, and analyses. She also works with CAWP's programs for women's public leadership and has been an expert source and commentator for media outlets including MSNBC , NPR, PBS, the New York Times, and the Washington Post. Dittmar earned her B.A. from Aquinas College in Grand Rapids, MI and her Ph.D. from Rutgers University-New Brunswick.Join us as we talk to Kelly about the status of women in politics, and how we can harness women's political power together. Special thanks to VEST Member Shagah Zakerion for moderating this session. Shagah Zakerion, is the Executive Director of the Lobeck Taylor Operating Foundation (LTOF). Prior to joining the LTFF team, Shagah served as Executive Director of Tulsa's Young Professionals (TYPROS) and The Forge business incubator. She also served as Diversity and Inclusion Program Manager for Williams, a Fortune 500 company headquartered in Tulsa.She serves on the board of advisors for Resonance Center for Women and the board of directors for the Khalid Jabara Foundation, Sally's List, and Gaining Ground Literacy. As a three time cancer survivor, Shagah is passionate about eliminating cancer and serves several organizations on that mission, including sitting on Stephenson Cancer Research Center's board of advocates and community advisory board.If you enjoyed the episode share it with a friend. Stay up to date with all of our news by following us on Linkedin or better yet, apply to become a VEST Member at www.VESTHer.co

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: List of donation opportunities (focus: non-US longtermist policy work), published by weeatquince on September 30, 2022 on The Effective Altruism Forum. Introduction In the past I have written on the EA Forum about where I am donating (2019, 2021). This year I have a dilemma. I have too many places I am excited to investigate and potentially donate too. I have a document where I have been listing opportunities I am excited by and thought why not share my list with others in the run up to the giving season. In my opinion there are a lot of EA projects lacking funding. I believe donors (especially medium-size non-US donors) who could evaluate and fund some of these would have an outsized impact, more impact than just giving to various EA Funds (in fact this post is a follow up of a red team post on the LTFF). Also I am also interested in feedback and criticism of my donation list as I plan to donate to at least some of the places listed below (to be decided). The primary focus is on longtermist/ EA policy work. By policy work I am considering organisations that directly influence current policy so it is better for the long run future (not just indirectly doing policy-adjacent academic research or supporting individuals policy careers, etc). I list a few ideas on other cause areas at the end. Note that not every org/person listed considers themselves to be EA affiliated. This list has about 25-30 funding ideas, ranging $15,000 to $4m, of which I estimate perhaps 66% are worth funding if investigated. I want to caveat this with some warnings: Expect inaccuracies: Not every org has had a chance to input. Also some details might be out of date – this list has been growing for a while now and some orgs may now have funding (or have run out of funding). Expect conflicts of interest: I have worked with and/or helped and/or am friends with the staff at and/or am an adviser/affiliate at a lot of the organisations I list. The only organisations listed that have paid me for work are the APPG for Future Generations and Charity Entrepreneurship. As a rule of thumb, assume I have some bias towards funding this kind of work. Background reasoning – why policy? Why fund policy work? Influencing policy is an effective way to drive change in the world. It is the key focus of advocacy groups and campaigns around the globe and seen as one of the most high impact ways to affect society for the better. This applies to existential risks. 80000 Hours research suggests there are two ways to protect the future from anthropogenic risks: technical work and governance/policy work (e.g. see here on AI). There are many things to advocate for that would protect the future. See here for a list of 250 longtermist policy ideas and see (mostly UK focused) collections of policy ideas on long-term institutions and biosecurity and ensuring AI regulation goes well and malevolent actors. Furthermore EA policy change work which is targeted and impact focused and carefully measured can be extremely effective. Analysis of 100s of historical policy change campaigns (look across various reports here) suggests a new EA charity spending around $1-1.5m, has a 10-50% chance driving a major policy change. And existing EA charities seem even more effective than that. The APPG for Future Generations seemed to consistently drive a policy change for every ~$50k. LEEP seems to have driven their first policy change for under ~$50k and seems on track to keep driving changes at that level. Why might policy work be underfunded? In short there is a lack of funders with the motivation and capability to fund this work. Some funders are avoiding funding policy work. The Long Term Future Fund (LTFF) does not fund any new policy work and is sceptical of policy work (see here). As far as I can tell OpenPhil's longtermist teams appear to...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Long-Term Future Fund: December 2021 grant recommendations, published by abergal on August 18, 2022 on The Effective Altruism Forum. Introduction The Long-Term Future Fund made the following grants as part of its 2021 Q4 grant cycle (grants paid out sometime between August and December 2021): Total funding distributed: 2,081,577 Number of grantees: 34 Acceptance rate (excluding desk rejections): 54% Payout date: July - December 2021 Report authors: Asya Bergal (Chair), Oliver Habryka, Adam Gleave, Evan Hubinger 2 of our grantees requested that we not include public reports for their grants. (You can read our policy on public reporting here). We also referred 2 grants, totalling $110,000, to private funders, and approved 3 grants, totalling $102,000, that were later withdrawn by grantees.If you're interested in getting funding from the Long-Term Future Fund, apply here.(Note: The initial sections of this post were written by me, Asya Bergal.) Other updates Our grant volume and overall giving increased significantly in 2021 (and in 2022 – to be featured in a later payout report). In the second half of 2021, we applied for funding from larger institutional funders to make sure we could make all the grants that we thought were above the bar for longtermist spending. We received two large grants at the end of 2021: $1,417,000 from the Survival and Flourishing Fund's 2021-H2 S-process round $2,583,000 from Open Philanthropy Going forward, my guess is that donations from smaller funders will be insufficient to support our grantmaking, and we'll mainly be relying on larger funders. More grants and limited fund manager time mean that the write-ups in this report are shorter than our write-ups have been traditionally. I think communicating publicly about our decision-making process continues to be valuable for the overall ecosystem, so in future reports, we're likely to continue writing short one-sentence summaries for most of our grants, and more for larger grants or grants that we think are particularly interesting. Highlights Here are some of the public grants from this round that I thought looked most exciting ex ante: $50,000 to support John Wentworth's AI alignment research. We've written about John Wentworth's work in the past here. (Note: We recommended this grant to a private funder, rather than funding it through LTFF donations.) $18,000 to support Nicholas Whitaker doing blogging and movement building at the intersection of EA / longtermism and Progress Studies. The Progress Studies community is adjacent to the longtermism community, and is one of a small number of communities thinking carefully about the long-term future. I think having more connections between the two is likely to be good both from an epistemic and a talent pipeline perspective. Nick had strong references and seemed well-positioned to do this work, as the co-founder and editor of the Works in Progress magazine. $60,000 to support Peter Hartree pursuing independent study, plus a few "special projects". Peter has done good work for 80K for several years, received very strong references, and has an impressive history of independent projects, including Inbox When Ready. Grant Recipients In addition to the grants described below, 2 grants have been excluded from this report at the request of the applicants. Note: Some of the grants below include detailed descriptions of our grantees. Public reports are optional for our grantees, and we run all of our payout reports by grantees before publishing them. We think carefully about what information to include to maximize transparency while respecting grantees' preferences. We encourage anyone who thinks they could use funding to positively influence the long-term trajectory of humanity to apply for funding. Grants evaluated by Evan Hubinger EA Switzerland/PIB...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Brain-like AGI project "aintelope", published by Gunnar Zarncke on August 14, 2022 on LessWrong. Steven Byrnes called for a brain-like AGI research agenda, and three guys from the Hamburg EA community listened. We are excited about Steven's five-star program 15.2.1.2 "Reverse-engineer human social instincts," and kicked off work a few weeks ago in June. We familiarized ourselves with Steven's brain-like AGI framework, and meet weekly now. This post is an announcement and a request for feedback and collaboration. Why us? We have a great skill fit: A professional data scientist with the required machine learning experience to implement RL agents, A professional Python developer with game-programming background to implement the world mode, and visualization. A seasoned software engineer with startup CTO experience who takes care of everything else, e.g., this blog post (me). What have we done so far? We have already implemented a toy world and simple RL agent in a first iteration of Steven's framework. We build on top of the Python framework PettingZoo. Our code is in a private Github repo that we believe should stay private given the potential impact. Looking for thoughts on this. We have collected a list of more than 60 candidate instincts from neuroscience and other sources that we can implement and experiments with. The project website will be here:/ (effectively empty right now). The project and our progress so far were presented at the Human-aligned AI Summer School in Prague on August 5th, where we got feedback about the project, brain-like AGI in general, and found the three participants who wanted to collaborate. What do we want to do? Implementing models and running tests is a proven way to test theories and check our understanding of the models. Better quick success/failure on something where it is too easy to build a big theory. Specifically, we want to: Show that a relatively simple set of instincts can shape complex behaviors of a single agent. Show whether the instincts lead to significantly reduced training time compared to agents without such instincts. Extend the simulation to groups of agents. Show that prosocial behavior can be shaped with few instincts. See if we can get Ersatz Interpretability working. In the ideal case, the simulated agents show behavior consistent with having values like altruism or honesty. Immediate next steps: Implement food-related thought-assessors. Implement different types of social cues to get to collaborative behavior. Probably not the three types suggested by Steven as motivating examples. How can you help? Please give us feedback. Review what we already have (requires an invite, please request). Help with funding and growth of the project (I am applying to LTFF). Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Some concerns about policy work funding and the Long Term Future Fund, published by weeatquince on August 12, 2022 on The Effective Altruism Forum. Situation As far as I can tell The Long-Term Future Fund (LTFF) wants to fund work that will influence public policy. They say they are looking to fund “policy analysis, advocacy ...” work and post in policy Facebook groups asking for applications. However, as far as I can tell, in practice, the LTFF appears to never (or only very rarely) fund such projects that apply for funding. Especially new projects.My view that such funding is rare is based on the following pieces of evidence: Very few policy grants have been made. Looking at the payout reports of the LTFF, they funded a grant for policy influencing work in 2019 (to me, for a project that was not a new project). Upon asking them they say they have funded at least one more policy grant that has not been written up. Very many policy people have applied for LTFF grants. Without actively looking for it I know of: someone in an established think tank looking for funds for EA work, 3 groups wanting to start EA think tank type projects, a group wanting to do mass campaigning work. All these groups looked competent and were rejected. I am sure many others apply too. I know one of these has gone on to get funding elsewhere (FTX). Comments from staff at leading EA orgs. In January last year, a senior staff member at a leading EA institution mentioned, to my surprise, that EA funders tend not to fund any new longtermist policy projects (except perhaps with very senior trusted people like OpenPhil funding CSET). Recently I spoke to someone at CEA about this and asked if it matched their views too and they said they do think there is a problem here. Note this was about EA funders in general, not specifically the LTFF. Comments from EA policy folk looking for funding. There seems to be (at least there was in 2021) a general view from EAs working in the policy space that it has been very hard to find funding for policy work. Note this is about EA funders in general, not the LTFF. Odd lines of questioning. When I applied, the line of questioning was very odd. I faced an hour of: Why should we do any policy stuff? Isn't all policy work a waste of time? Didn't [random unconnected policy thing] not work? Etc. Of course it can be useful to check applicants have a good understanding of what they are doing, but it made me question whether they wanted to fund policy work at all. Odd feedback. Multiple applicants to the LTFF have reported receiving feedback along the lines of: We see high downside risks but cannot clarify what those risks are. Or: We want to fund you but an anonymous person vetoed you and we cannot say who or why. Or: Start-up policy projects are too risky. Or: We worry you might be successful and hard to shut down if we decided we don't like you in future. This makes me worry that the fund managers do not think through the risks and reasons for funding or not funding policy work in as much depth as I would like, and that they maybe do not fund any new/start-up policy projects. Acknowledgment that they do apply a higher bar for policy work. Staff at the LTFF have told me that they apply a higher bar for policy work than for other grants. Of course, this is all circumstantial, and not necessarily a criticism of the LTFF. The fund managers might argue they never get any policy projects worth funding and that all the promising projects I happened to hear of were actually net negative and it was good not to fund them. It is also possible that things have improved in the last year (the notes that make up this post have been sitting in an email chain for a long while now). Relevance and recommendation That said I thought it was worth me writing this up publicly as the possibilit...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How to Diversify Conceptual AI Alignment: the Model Behind Refine, published by adamShimi on July 20, 2022 on The Effective Altruism Forum. This work has been done while at Conjecture Tl;dr: We need far more conceptual AI alignment research approaches than we have now if we want to increase our chances to solve the alignment problem. However, the conceptual alignment field remains hard to access, and what feedback and mentorship there is focuses around few existing research directions rather than stimulating new ideas. This model lead to the creation of Refine, a research incubator for potential conceptual alignment researchers funded by the LTFF and hosted by Conjecture. Its goal is to help conceptual alignment research grow in both number and variety, through some minimal teaching and a lot of iteration and feedback on incubatees' ideas. The first cohort has been selected, and will run from August to October 2022. In the bigger picture, Refine is an experiment within Conjecture to find ways of increasing the number of conceptual researchers and improve the rate at which the field is making productive mistakes. The Problem: Not Enough Varied Conceptual Research I believe that in order to solve the alignment problem, we need significantly more people attacking it from a lot different angles. Why? First because none of the current approaches appears to yield a full solution. I expect many of them to be productive mistakes we can and should build on, but they don't appear sufficient, especially with shorter timelines. In addition, the history of science teaches us that for many important discoveries, especially in difficult epistemic situations, the answers don't come from one lone genius seeing through the irrelevant details, but instead from bits of evidence revealed by many different takes and operationalizations (possibly unified and compressed together at the end). And we should expect alignment to be hard based on epistemological vigilance. So if we accept that we need more people tackling alignment in more varied ways, why are we falling short of that ideal? Note that I will focus here on conceptual researchers, as they are the source of most variations on the problem, and because they are so hard to come by. I see three broad issues with getting more conceptual alignment researchers working on wildly different approaches: (Built-in Ontological Commitments) Almost all current attempts to create more conceptual alignment researchers (SERI MATS, independent mentoring...) rely significantly on mentorship by current conceptual researchers. Although this obviously comes with many benefits, it also leads to many ontological commitments being internalized when one is learning the field. As such, it's hard to go explore a vastly different approach because the way you see the problem has been moulded by this early mentorship. (Misguided Requirements) I see many incorrect assumptions about what it takes to be a good conceptual researcher floating around, both from field-builders and from potential candidates. Here's a non-exhaustive list of the most frustrating ones You need to know all previous literature on alignment (the field has more breadth than depth, and so getting a few key ideas is more important than knowing everything) You need to master maths and philosophy (a lot of good conceptual work only uses basic maths and philosophy) You need to have an ML background (you can pick up the relevant part and just work on approaches different to pure prosaic alignment) (No Feedback) If you want to start on your own, you will have trouble getting any feedback at all. The AF doesn't provide much feedback even for established researchers, and it has almost nothing in store for newcomers. Really, the main source of feedback in the field is asking other researchers, but when y...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How to Diversify Conceptual Alignment: the Model Behind Refine, published by Adam Shimi on July 20, 2022 on The AI Alignment Forum. This work has been done while at Conjecture Tl;dr: We need far more conceptual AI alignment research approaches than we have now if we want to increase our chances to solve the alignment problem. However, the conceptual alignment field remains hard to access, and what feedback and mentorship there is focuses around few existing research directions rather than stimulating new ideas. This model lead to the creation of Refine, a research incubator for potential conceptual alignment researchers funded by the LTFF and hosted by Conjecture. Its goal is to help conceptual alignment research grow in both number and variety, through some minimal teaching and a lot of iteration and feedback on incubatees' ideas. The first cohort has been selected, and will run from August to October 2022. In the bigger picture, Refine is an experiment within Conjecture to find ways of increasing the number of conceptual researchers and improve the rate at which the field is making productive mistakes. The Problem: Not Enough Varied Conceptual Research I believe that in order to solve the alignment problem, we need significantly more people attacking it from a lot different angles. Why? First because none of the current approaches appears to yield a full solution. I expect many of them to be productive mistakes we can and should build on, but they don't appear sufficient, especially with shorter timelines. In addition, the history of science teaches us that for many important discoveries, especially in difficult epistemic situations, the answers don't come from one lone genius seeing through the irrelevant details, but instead from bits of evidence revealed by many different takes and operationalizations (possibly unified and compressed together at the end). And we should expect alignment to be hard based on epistemological vigilance. So if we accept that we need more people tackling alignment in more varied ways, why are we falling short of that ideal? Note that I will focus here on conceptual researchers, as they are the source of most variations on the problem, and because they are so hard to come by. I see three broad issues with getting more conceptual alignment researchers working on wildly different approaches: (Built-in Ontological Commitments) Almost all current attempts to create more conceptual alignment researchers (SERI MATS, independent mentoring...) rely significantly on mentorship by current conceptual researchers. Although this obviously comes with many benefits, it also leads to many ontological commitments being internalized when one is learning the field. As such, it's hard to go explore a vastly different approach because the way you see the problem has been moulded by this early mentorship. (Misguided Requirements) I see many incorrect assumptions about what it takes to be a good conceptual researcher floating around, both from field-builders and from potential candidates. Here's a non-exhaustive list of the most frustrating ones You need to know all previous literature on alignment (the field has more breadth than depth, and so getting a few key ideas is more important than knowing everything) You need to master maths and philosophy (a lot of good conceptual work only uses basic maths and philosophy) You need to have an ML background (you can pick up the relevant part and just work on approaches different to pure prosaic alignment) (No Feedback) If you want to start on your own, you will have trouble getting any feedback at all. The AF doesn't provide much feedback even for established researchers, and it has almost nothing in store for newcomers. Really, the main source of feedback in the field is asking other researchers, but when you start...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Jaan Tallinn's 2021 Philanthropy Overview, published by jaan on April 28, 2022 on LessWrong. to follow up my philantropic pledge from 2020, i've updated my philanthropy page with 2021 results. in 2021 i made $22M worth of endpoint grants — exceeding my commitment of $14.4M (20k times $718.11 — the minimum price of ETH in 2021). notes: this number includes $1.9M to orgs that do re-granting (LTFF, EAIF, impetus grants, and PPF) — so it's likely that some of that $1.9M should not be included in the "endpoint grants in 2021" total. regardless, i'm comfortably above my commitment level for that not to matter; i have an ongoing substantial charitable project that's not reflected in the 2021 numbers — it's possible (and likely if ETH price holds, as SFF's s-process alone can't handle such amount) that i will report it retroactively next year or in 2024. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Will EAIF and LTFF publish its reports of recent grants?, published by throwaway129103 on April 23, 2022 on The Effective Altruism Forum. It's been a while since EAIF and LTFF have published grant reports and I wonder if they plan on longer doing so because of capacity bottlenecks? Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: EA Houses: Live or Stay with EAs Around The World, published by Kat Woods on April 15, 2022 on The Effective Altruism Forum. Would you like to live with EAs? Want to connect with EAs when you travel? If someone “granted” you free rent, freeing up your time, what EA project could you complete? Airbnb changed the world by making productive use of people's spare bedrooms and second homes. We at Nonlinear are trying a simple experiment to see if we can do something similar in EA - connecting EAs who have extra space with EAs who need space. Picture an EA who works a regular job to earn money to pay rent. You might not have enough money to provide them a $50,000 grant, but you might be able to provide your spare room, which frees up the EA to work on something high impact. You're like an angel investor, but for “impact housing”. We made a spreadsheet where you can either seek roommates, find EA couchsurfing opportunities, or become a host, providing space for EAs. Patrons can offer accommodations either for free, at a reduced rental fee, or against some service (such as pet sitting). Some benefits: Enable positive impact that would otherwise not happen Support a better use of otherwise vacant or under-utilized housing Fostering cultural, social, and intellectual exchange between groups of people Provide actually affordable housing to people all across the world Examples of housing that might be available A room that just became available in your friends' flat and they would love to have EAs around You or your parents have a second home which sits empty much of the year A city apartment or studio that stays vacant for a few weeks/months a year and needs regular check-ups Your parents look for help with garden work and walking their dog in exchange for a room You're wanting to rent a house with a bunch of EAs and you want to find other like-minded people to join you. Your parents are empty-nesters and would love to have something like an exchange student around, but with a higher impact. A group of EAs in a city link up to rent a house together Example use cases for guests Conducting an EA research project (e.g. if you received an LTFF grant or similar) Upskilling (e.g. learning technical ML skills) Engaging in local community building, particularly in areas where there is little EA presence (e.g. taking a leading role in forming a group) Launching a venture (e.g. startup, charity) Increased runway to give you the space you need to experiment or think about things Living with value-aligned people Opportunity to participate in or grow the local EA community Benefits for hosts Create positive impact by enabling EAs to to work on projects they wouldn't otherwise be able to Contribute growing the local EA community and be around more EAs Engage in stimulating conversations with EAs Find someone to take care of your pet, garden, or similar Examples of houses available Emerson Spartz's parents are Midwestern-wholesome empty nesters who would be thrilled to host a nice EA in one of their spare bedrooms. They live an hour from Chicago in a beautiful home surrounded by lush woodlands with a friendly dog and cat. His father, Tom, is a retired entrepreneur and mother, Maggi, runs a local nonprofit. This is a cozy retreat from the chaos of the world to work on something big. Kat and Emerson's Caribbean House Emerson and I have a house on the beach in downtown San Juan, Puerto Rico, USA that we live in for half the year. While we're gone we want to fill it with EAs doing important work on longtermism. There's room for ten people and you could potentially stay for one week to six months. Perfect for somebody who wants to be surrounded by an EA community and escape the winter months. OK, how does it work exactly? If you have space, list it on the spreadsheet. Potential applicants can reac...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Being an individual alignment grantmaker, published by A donor on February 28, 2022 on LessWrong. I am an earlyish crypto investor who has accumulated enough to be a mid-sized grantmaker, and I intend to donate most of my money over the next 5-10 years to try and increase the chances that humanity has a wonderful future. My best guess is that this is mostly decided by whether we pass the test of AI alignment, so that's my primary focus. AI alignment has lots of money flowing into it, with some major organizations not running fundraisers, Zvi characterizing SFF as having “too much money”, OpenPhil expanding its grantmaking for the cause, FTX setting themselves up as another major grantmaker, and ACX reporting the LTFF's position as: what actually happened was that the Long Term Future Fund approached me and said “we will fund every single good AI-related proposal you get, just hand them to us, you don't have to worry about it” So the challenge is to find high-value funding opportunities in a crowded space. One option would be to trust that the LTFF or whichever organization I pick will do something useful with the money, and I think this is a perfectly valid default choice. However, I suspect that as the major grantmakers are well-funded, I have a specific comparative advantage over them in allocating my funds: I have much more time per unit money to assess, advise, and mentor my grantees. It helps that I have enough of an inside view of what kinds of things might be valuable that I have some hope of noticing gold when I strike it. Additionally, I can approach people who would not normally apply to a fund. What is my grantmaking strategy? First, I decided what parts of the cause to focus on. I'm most interested in supporting alignment infrastructure, because I feel relatively more qualified to judge the effectiveness of interventions to improve the funnel which takes in people who don't know about alignment in one end, takes them through increasing levels of involvement, and (when successful) ends with people who make notable contributions. I'm also excited about funding frugal people to study or do research which seems potentially promising to my inside view. Next, I increased my surface area with places which might have good giving opportunities by involving myself with many parts of the movement. This includes Rob Miles's Discord, AI Safety Support's Slack, in-person communities, EleutherAI, and the LW/EA investing Discord, where there are high concentrations of relevant people, and exploring my non-LW social networks for promising people. I also fund myself to spend most of my time helping out with projects, advising people, and learning about what it takes to build things. Then, I put out feelers towards people who are either already doing valuable work unfunded or appear to have the potential and drive to do so if they were freed of financial constraints. This generally involves getting to know them well enough that I have a decent picture of their skills, motivation structure, and life circumstances. I put some thought into the kind of work I would be most excited to see them do, then discuss this with them and offer them a ~1 year grant (usually $14k-25k, so far) as a trial. I also keep an eye open for larger projects which I might be able to kickstart. When an impact certificate market comes into being (several promising signs on the horizon!), I intend to sell the impact of funding the successful projects and use the proceeds to continue grantmaking for longer. Alongside sharing my models of how to grantmake in this area for others in a similar position and getting advice on it, the secondary purpose of this post is to pre-register my intent to sell impact in order to strengthen the connection between future people buying my impact and my current decisions. ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Implications of automated ontology identification, published by Alex Flint on February 18, 2022 on The AI Alignment Forum. Financial status: supported by individual donors and a grant from LTFF. Epistemic status: early-stage technical work. This write-up benefited from conversations with John Wentworth. Outline This write-up is a response to ARC's request for feedback on ontology identification, described in the ELK technical report. We suppose that a solution to ELK is found, and explore the technical implications of that. In order to do this we operationalize "automated ontology identification" in terms of a safety guarantee and a generalization guarantee. For some choices of safety guarantee and generalization guarantee we show that ontology identification can be iterated, leading to a fixed point that has strange properties. We explore properties of this fixed point informally, with a view towards a possible future impossibility result. We speculate that a range of safety and generalization guarantees would give rise to the same basic iteration scheme. In an appendix we confirm that impossibility of automated ontology identification would not imply impossibility of interpretability in general or statistical learning in general. Introduction In this write-up we consider the implications of a solution to the ontology identification problem described in the ELK technical report. We proceed in three steps. First, we define ontology identification as a method for finding a reporter, given a predictor and a labeled dataset, subject to a certain generalization guarantee and a certain safety guarantee. Second, we show that, due to the generalization and safety guarantee, ontology identification can be iterated to construct a powerful oracle using only a finite narrow dataset. We find no formal inconsistency here, though the result seems counter-intuitive to us. Third, we explore the powers of the oracle by asking whether it could solve unreasonably difficult problems in value learning. The crux of our framework is an operationalization of automated ontology identification. We define an "automated ontology identifier" as meeting two formal requirements: (Safety) Given an error-free training set, an automated ontology identifier must find a reporter that never answers "YES" when the true answer is "NO" (though the converse is permissible). This mirrors the emphasis on worst-case performance in the ELK report. We say that a reporter meeting this requirement is ‘conservative'. (Generalization) Given a question/answer dataset drawn from a limited "easy set", an automated ontology identifier must find a reporter that answers "YES" for at least one case outside of the easy set. This mirrors the emphasis on answering cases that humans cannot label manually in the ELK report. We say that a reporter meeting this requirement is ‘helpful relative to the easy set'. The departure between generalization in this write-up and generalization as studied in statistical learning is the safety guarantee. We require automated ontology identifiers to be absolutely trustworthy when they answer "YES" to a question, although they are allowed to be wrong when answering "NO". We believe that any automated ontology identifier ought to make some formal safety guarantee, because we are ultimately considering plans that have consequences we don't understand, and we must eventually decide whether to press "Execute" or not. We suspect that this safety guarantee could be weakened considerably while remaining susceptible to the iteration scheme that we propose. Automated ontology identifiers as we have defined them are not required to answer all possible questions. We might limit ourselves to questions of the form "Is it 99% likely that X?" or "Excluding the possibility of nearby extraterrestrials, does X h...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Alignment versus AI Alignment, published by Alex Flint on February 4, 2022 on The AI Alignment Forum. Financial status: This work is supported by individual donors and a grant from LTFF. Epistemic status: This post contains many inside-view stories about the difficulty of alignment. Thanks to Adam Shimi, John Wentworth, and Rob Miles for comments on this essay. What exactly is difficult about AI alignment that is not also difficult about alignment of governments, economies, companies, and other non-AI systems? Is it merely that the fast speed of AI makes the AI alignment problem quantitatively more acute than other alignment problems, or are there deeper qualitative differences? Is there a real connection between alignment of AI and non-AI systems at all? In this essay we attempt to clarify which difficulties of AI alignment show up similarly in non-AI systems, and which do not. Our goal is to provide a frame for importing and exporting insights from and to other fields without losing sight of the difficulties of AI alignment that are unique. Clarifying which difficulties are shared should clarify the difficulties that are truly unusual about AI alignment. We begin with a series of examples of aligning different kinds of systems, then we seek explanations for the relative difficulty of AI and non-AI alignment. Alignment in general In general, we take actions in the world in order to steer the future in a certain direction. One particular approach to steering the future is to take actions that influence the constitutions of some intelligent system in the world. A general property of intelligent systems seems to be that there are interventions one can execute on them that have robustly long-lasting effects, such as changing the genome of a bacterium, or the trade regulations of a market economy. These are the aspects of the respective intelligent systems that persist through time and dictate their equilibrium behavior. In contrast, although plucking a single hair from a human head or adding a single barrel of oil to a market does have an impact on the future, the self-correcting mechanisms of the respective intelligent systems negate rather than propagate such changes. Furthermore, we will take alignment in general to be about utilizing such interventions on intelligent systems to realize our true terminal values. Therefore we will adopt the following working definition of alignment: Successfully taking an action that steers the future in the direction of our true terminal values by influencing the part of an intelligent system that dictates its equilibrium behavior. Our question is: in what ways is the difficulty of alignment of AI systems different from that of non-AI systems? Example: Aligning an economic society by establishing property rights Suppose the thing we are trying to align is a human society and that we view that thing as a collection of households and firms making purchasing decisions that maximize their individual utilities. Suppose that we take as a working operationalization of our terminal values the maximization of the sum of individual utilities of the people in the society. Then we might proceed by creating the conditions for the free exchange of goods and services between the households and firms, perhaps by setting up a government that enforces property rights. This is one particular approach to aligning a thing (a human society) with an operationalization of The Good (maximization of the sum of individual utilities). This particular approach works by structuring the environment in which the humans live in such a way that the equilibrium behavior of the society brings about accomplishment of the goal. We have: An intelligent system being aligned, which in this case is a human society. A model of that system, which in this case is a collection o...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Challenges with Breaking into MIRI-Style Research, published by Chris Leong on January 17, 2022 on The AI Alignment Forum. Trying to break into MIRI-style research seems to be much, much harder than trying to break into ML-style safety research. This is worrying if you believe this research to be important. I'll examine two kinds of causes: those which come from MIRI-style research being a niche area and those which go beyond this: Challenges beyond MIRI-style research being a niche area: MIRI doesn't seem to be running internships or running their AI safety for computer scientists workshops If you try to break into ML-style Safety and fail, you can always be reuse at least part of what you've learned to obtain a highly-compensated role in industry. Agent foundations knowledge is highly niche and unlikely to be used elsewhere. You can park in a standard industry job for a while in order to earn career capital for ML-style safety. Not so for MIRI-style research. MIRI publishes a lot less material these days. I support this decision I support as infohazards deserve to be taken seriously, but it also makes it harder to contribute. There are well-crafted materials for learning a lot of the prerequisites for ML-style safety. There seems to be a natural pathway of studying a masters then pursuing a PhD to break into ML-style safety. There are a large number of scholarships available and many countries offer loans or income support. The above opportunities mean that there are more ways to gauge fit for ML-style safety research. There's no equivalent to submitting a paper. If a paper passes review, then it gains a certain level of credibility. There are upvotes, but this signaling mechanism is more distorted by popularity or accessibility. Further, unlike writing an academic paper, writing alignment forum posts won't provide credibility outside of the field. Challenges that come from being a niche area I think this probably should be a niche area. It would be a bit strange if foundations work were the majority of the research. Nonetheless, it's worth highlighting some of the implications: General AI safety programs and support - ie. AI Safety Fundamentals Course, AI Safety Support, AI Safety Camp, Alignment Newsletter, ect. are naturally going to strongly focus on ML-style research and might not even have the capability to vet MIRI-style research. It is much harder to find people with similar interests to collaborate with or mentor you. Compare to how easy it is to meet a bunch of people interested in ML-style research by attending EA meetups or EAGx. If you want to feel part of the AI safety community and join in the conversations are having, you will have to spend time learning about ML-style research. While is likely valuable to broaden your scope as this can lead to cross-pollination, it also sucks up time when you could be learning about MIRI-style research. Further Thoughts I think it's worth thinking about what this looks like overall. If you want to try breaking into MIRI-style research, the most likely path looks like saving up 3-12 months runway. 3 months might be possible if you've been consistently working on things in your free time and you've already read a lot of the material that you need to read + made substantial research progress. That said, even if you're able to produce material to prove yourself in 3 months, you'd probably need an extra month or two to obtain funding and you always need more runway than the minimum possible time. It would be possible to apply for an LTFF grant to support this research, but it's probably easier to build up the credibility for ML-style research. Further, if you fail, then you haven't learned skills nor gained credibility that would assist you for any other paths.I suspect that these considerations not only significantly c...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Two Podcast Opportunities, published by finm on December 29, 2021 on The Effective Altruism Forum. Background I think there's a strong case for recording more existing EA content as podcasts. The main reason is that this will help more people engage with more content, because there are times when it's much easier to listen to versus read the same writing. I'm glad to see some recent progress. The Nonlinear Library popped up a couple months ago, which uses text-to-speech software to create an automatically updating repository of audio content. The EA Forum Podcast is also great, especially because genuine spoken word still beats TTS. Two Opportunities At some point it would be good to discuss the general theme of geting even more really high-quality readouts of EA content into the world. But this post is about two specific opportunities which seem especially impactful and time-sensitive to me. The first is the series of conversations MIRI recently released. Most of these are dialogues, and I think there's about 14 hours of content there in total. I expect a lot of people would love to read them, but don't have the time to sit down and trawl through that much text. So it could be really worthwhile if we could produce some (high-quality) readouts in podcast format. It'll be important to distinguish between multiple voices here, so 'table reads' of multiple people would be best (virtual is fine). An 80/20 version could involve recording only the most notable two or three conversations. The second opportunity is the recent EA Forum creative writing prize. The winners were announced a couple days ago, and they're really excellent. Between the prizes and the 'honorable mentions', there are 15 stories; a small (audio)book's worth. With some care, e.g. tasteful sound effects, these could be turned into a really nice little package; even for folks outside of EA. Again, I think a diversity of voices would make this better than one reader; and different voices for different characters could be a bonus. End Products Where could these end up? One idea would be to put these on existing feeds — probably the Nonlinear Library for the MIRI conversations, and the EA Forum Podcast for the fiction (though maybe both feeds for both series). But if these end up being high enough quality, I think they could end up as their own podcast feeds, perhaps with a bit of branding and a website (e.g.). I think this format of 'effectively an audiobook living in a podcast feed' has been well exemplified by The Life You Can Save and this recording of Rationality: From AI to Zombies. The creative writing feed could even be extended indefinitely, by adding more entrants from this year's prize, or by putting out a call for more pieces. What Next? If either of these projects are going to happen, we'll need a few people with time and interest in recording some of these pieces or parts. We'll also need an audio editor (or perhaps more than one). So if you're interested in either reading or editing, please let me know! You could leave a comment on this post, or message me directly. I would be happy to project manage this if there's sufficient interest. For example, I could help coordinate who reads what, help send out mics, and set up the podcast feeds. I think it's very likely that funding would be available for this. Here is Michael Aird in a comment about the creative writing prizes: I'm willing to personally guarantee (say) at least $500 for a mic (if the person doesn't have one already and it costs that much) and $20/hr for up to 5 times as many hours as the cumulative total run time across all the episodes produced, with the guarantee being capped at $1500 total. I also expect the LTFF might be interested in the MIRI conversations, and the EAIF in the creative writing pieces. One another person has also...

Elizabeth Frame Ellison had endless professional opportunities upon graduating from law school, but she chose the less obvious one. Instead of the "typical" corporate or big firm career path, she chose to serve those who needed it the most. From her experiences in our broken justice system to her travels abroad to her love for her Tulsa community, Elizabeth is changing lives one dish at a time. Join us as we talk about:Everyone deserves a fighting chanceDifferent is great and deliciousEquity is inherently valuable and builds good businessFaking it til you make it... sort ofThe world is your oyster Get ready to pack your bags to head to Tulsa, Oklahoma.About ElizabethElizabeth Frame Ellison is the President and CEO of Lobeck Taylor Family Foundation (LTFF) in Tulsa, Oklahoma. In her eleven years leading LTFF, Ellison has founded Oklahoma's FIRST food hall, Mother Road Market (2018) as well as Tulsa's kickstart kitchen incubator, Kitchen 66 (2016). Ellison is also a founding partner of 36 Degrees North (2016), a co-workspace and basecamp for entrepreneurs in Tulsa's Arts District and a founding board member of Vest (2020).Ellison received recognition as one of Oklahoma's 40 under 40 and several awards for small business and entrepreneurial support. She has given several keynote addresses and served as a panelist at a Google conference on the future of food in 2017.Ellison received her Bachelor of Arts in Political Science and Classical Culture in 2004 and worked for Boren for Congress as the deputy finance director before joining Congressman Boren (OK Dist. 2) as a Legislative Assistant. In 2006, Ellison served as Political Director when her mother, Kathy Taylor, decided to run for Mayor and asked for campaign help. After a successful campaign, Ellison entered Law School at The University of Oklahoma. As the class President, Ellison was honored to give the commencement address at her Law School graduation. In 2012, Ellison was elected to serve as a school board representative for Tulsa Technology Center.When she isn't working, Ellison enjoys travel, culinary exploration, true crime novels and athletic activity alongside her husband Chris and their boys Taylor (9) and Wyatt (6). Ellison lives in Tulsa and San Francisco.Social: @elizabethframeellison

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: I'm leaving AI alignment – you better stay, published by rmoehn on the LessWrong. Write a Review Requirements for independent AI alignment research and how they are connected This diagram summarizes the requirements for independent AI alignment research and how they are connected. In this post I'll outline my four-year-long attempt at becoming an AI alignment researcher. It's an ‘I did X [including what I did wrong], and here's how it went' post (see also jefftk's More writeups!). I'm not complaining about how people treated me – they treated me well. And I'm not trying to convince you to abandon AI alignment research – you shouldn't. I'm not saying that anyone should have done anything differently – except myself. Requirements Funding Funding is the main requirement, because it enables everything else. Thanks to Paul Christiano I had funding for nine months between January 2019 and January 2020. Thereafter I applied to the EA Foundation Fund (now Center on Long-Term Risk Fund) and Long-Term Future Fund for a grant and they rejected my applications. Now I don't know of any other promising sources of funding. I also don't know of any AI alignment research organisation that would hire me as a remote worker. How much funding you need varies. I settled on 5 kUSD per month, which sounds like a lot when you're a student, and which sounds like not a lot when you look at market rates for software developers/ML engineers/ML researchers. On top of that, I'm essentially a freelancer who has to pay social insurance by himself, take time off to do accounting and taxes, and build runway for dry periods. Results and relationships In any job you must get results and build relationships. If you don't, you don't earn your pay. (Manager Tools talks about results and relationships all the time. See for example What You've Been Taught About Management is Wrong or First Job Fundamentals.) The results I generated weren't obviously good enough to compel Paul to continue to fund me. And I didn't build good enough relationships with people who could have convinced the LTFF and EAFF fund managers that I have the potential they're looking for. Time Funding buys time, which I used for study and research. Another aspect of time is how effectively and efficiently you use it. I'm good at effective, not so good at efficient. – I spend much time on non-research, mostly studying Japanese and doing sports. And dawdling. I noticed the dawdling problem at the end of last year and got it under control at the beginning of this year (see my time tracking). Too late. Added 2020-03-16: I also need a lot of sleep in order to do this kind of work. – About 8.5 h per day. Travel and location I live in Kagoshima City in southern Japan, which is far away from the AI alignment research hubs. This means that I don't naturally meet AI alignment researchers and build relationships with them. I could have compensated for this by travelling to summer schools, conferences etc. But I missed the best opportunities and I felt that I didn't have the time and money to take the second-best opportunities. Of course, I could also relocate to one of the research hubs. But I don't want to do that for family reasons. I did start maintaining the Predicted AI alignment event/meeting calendar in order to avoid missing opportunities again. And I did apply and get accepted to the AI Safety Camp Toronto 2020. They even chose my research proposal for one of the teams. But I failed to procure the funding that would have supported me from March through May when the camp takes place. Knowledge I know more than most young AI alignment researchers about how to make good software, how to write well and how to work professionally. I know less than most young AI alignment researchers about maths, ML and how to do research. The latter appear to be ...