Podcasts about concretely

One hope for keeping existential risks low is to get AI companies to (successfully) make high-assurance safety cases: structured and auditable arguments that an AI system is very unlikely to result in existential risks given how it will be deployed.[1] Concretely, once AIs are quite powerful, high-assurance safety cases would require making a thorough argument that the level of (existential) risk caused by the company is very low; perhaps they would require that the total chance of existential risk over the lifetime of the AI company[2] is less than 0.25%[3][4].The idea of making high-assurance safety cases (once AI systems are dangerously powerful) is popular in some parts of the AI safety community and a variety of work appears to focus on this. Further, Anthropic has expressed an intention (in their RSP) to "keep risks below acceptable levels"[5] and there is a common impression that Anthropic would pause [...] ---Outline:(03:19) Why are companies unlikely to succeed at making high-assurance safety cases in short timelines?(04:14) Ensuring sufficient security is very difficult(04:55) Sufficiently mitigating scheming risk is unlikely(09:35) Accelerating safety and security with earlier AIs seems insufficient(11:58) Other points(14:07) Companies likely wont unilaterally slow down if they are unable to make high-assurance safety cases(18:26) Could coordination or government action result in high-assurance safety cases?(19:55) What about safety cases aiming at a higher risk threshold?(21:57) Implications and conclusionsThe original text contained 20 footnotes which were omitted from this narration. --- First published: January 23rd, 2025 Source: https://www.lesswrong.com/posts/neTbrpBziAsTH5Bn7/ai-companies-are-unlikely-to-make-high-assurance-safety --- Narrated by TYPE III AUDIO.

News

News

News

News

PRESS REVIEW – Thursday, January 9: The commander-in-chief of the army, Joseph Aoun, emerges as a favourite in Lebanon's election for president. It's the country's thirteenth attempt in two years to find a leader. Meanwhile, Mark Zuckerberg's decision to kill fact-checking on Facebook, WhatsApp and Instagram sparks outrage but also support. Donald Trump continues to roil – and worry – Europe over his threat to take over Greenland by force, if necessary. Plus: French coach Dider Deschamps says he'll step down after the 2026 World Cup. Lebanon in new bid to elect new presidentLebanon prepares to elect a new leader. As one news outlet says, the country has tried twelve times to elect a new leader in the past two years without success, so here's hoping the baker's dozen is the lucky one.The man emerging as favourite is Joseph Aoun, who is depicted on the front of L'Orient-Le Jour, the French language Lebanese daily. Aoun is commander-in-chief of the Lebanese army and is a respected figure for having kept the Lebanese army together despite political fractures. Another Lebanese daily, Annahar, is exercising caution today, noting that 86 votes are needed out of 128 to win the election.Unlike previous times, this vote is taking place with a significantly weakened Hezbollah, which has lost influence in parliament. For the pro-Hezbollah daily al-Akhbar, the emergence of Aoun is the result of foreign collusion between the US and Saudi Arabia, which the paper sees as making a return to Lebanese politics. In an analysis piece from The Washington Institute, the author warns that the time to strike is now. Without an immediate and concerted effort to consolidate the setbacks to Hezbollah, the group could regroup and rebuild, which would put Lebanon's political future in peril once again.Meta ends fact-checking in USMark Zuckerberg's Meta has taken the drastic step of rolling back its fact-checking service, in what appears to be a move bowing down to the new Trump administration. For The Guardian's editors, Zuckerberg is ingratiating himself with the next Trump administration with this decision to scrap US-based fact-checkers on Facebook, WhatsApp and Instagram. Concretely, users can call others mentally ill based on their sexuality or gender identity, for instance. Meta says the decision is in response to accusations of overreach by content moderators. But as The Guardian notes, it kills two birds with one stone for Zuckerberg: cutting costs and getting on Trump's good side, this as the new US leader will soon have the power to kill or not a federal antitrust case against Meta, release regulatory pressure on big tech and facilitate AI.For The Economist though, as "craven" as the circumstances may be, Meta's new sweeping changes are correct. Speech online "urgently needs to become freer". The Economist says this is the only way to shore up America's democracy against whatever tests it faces in the years to come.Trump wants to take over GreenlandIn addition to railing against Meta, Trump has raised eyebrows this week for his plans to take over the Danish territory of Greenland. Certainly a question one might be asking today is on the front of French paper Le Parisien: Just how far will they go, Donald Trump and his right- hand man Elon Musk? Trump's comments that the US needs Greenland for its economic security raise concerns about how far Trump can go in his quests as president, particularly against a Europe weakened by war and inflation.The Economist plays devil's advocate in a piece; suggesting that instead of using force, Trump should name his price for Greenland. The author points to the US purchases of Louisiana and Alaska in the 19th century as some of the greatest achievements in US history and very good value. History will not be kind to Trump, the magazine says, if he takes Greenland by force, but purchasing it could be seen as the deal of the century.It's not as easy as he thinks, warns Danish paper Jyllands-Posten. Trump will have to impose punitive tariffs not just on Denmark but all of Europe if he wants to take control of Greenland. No one knows how serious he is, but as one business owner is quoted as saying, it's never a nice feeling to be on Trump's radar.Didier Deschamps to bow outFinally: Didier Deschamps, coach of the French football team, says he'll be stepping down after the 2026 World Cup. Deschamps, who coached France to victory in the 2018 World Cup and to the final in 2022, plans to leave the national team. But He's not the one gracing the cover of L'Equipe, the French sports paper. It's Zinédine Zidane, the 1998 World Cup hero, who has never been coy about his ambitions to coach the French national side. The path appears to be clearing now with Deschamp's announcement. Hence the paper's anticipation, even if Zidane hasn't yet said anything officially.You can catch our press review every morning on France 24 at 7:20am and 9:20am (Paris time), from Monday to Friday.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: [Interim research report] Activation plateaus & sensitive directions in GPT2, published by Stefan Heimersheim on July 5, 2024 on The AI Alignment Forum. This part-report / part-proposal describes ongoing research, but I'd like to share early results for feedback. I am especially interested in any comment finding mistakes or trivial explanations for these results. I will work on this proposal with a LASR Labs team over the next 3 months. If you are working (or want to work) on something similar I would love to chat! Experiments and write-up by Stefan, with substantial inspiration and advice from Jake (who doesn't necessarily endorse every sloppy statement I write). Work produced at Apollo Research. TL,DR: Toy models of how neural networks compute new features in superposition seem to imply that neural networks that utilize superposition require some form of error correction to avoid interference spiraling out of control. This means small variations along a feature direction shouldn't affect model outputs, which I can test: 1. Activation plateaus: Real activations should be resistant to small perturbations. There should be a "plateau" in the output as a function of perturbation size. 2. Sensitive directions: Perturbations towards the direction of a feature should change the model output earlier (at a lower perturbation size) than perturbations into a random direction. I find that both of these predictions hold; the latter when I operationalize "feature" as the difference between two real model activations. As next steps we are planning to Test both predictions for SAE features: We have some evidence for the latter by Gurnee (2024) and Lindsey (2024). Are there different types of SAE features, atomic and composite features? Can we get a handle on the total number of features? If sensitivity-features line up with SAE features, can we find or improve SAE feature directions by finding local optima in sensitivity (similar to how Mack & Turner (2024) find steering vectors)? My motivation for this project is to get data on computation in superposition, and to get dataset-independent evidence for (SAE-)features. Core results & discussion I run two different experiments that test the error correction hypothesis: 1. Activation Plateaus: A real activation is the center of a plateau, in the sense that perturbing the activation affects the model output less than expected. Concretely: applying random-direction perturbations to an activation generated from a random openwebtext input ("real activation") has less effect than applying the same perturbations to a random activation (generated from a Normal distribution). This effect on the model can be measured in KL divergence of logits (shown below) but also L2 difference or cosine similarity of late-layer activations. 2. Sensitive directions: Perturbing a (real) activation into a direction towards another real activation ("poor man's feature directions") affects the model-outputs more than perturbing the same activation into a random direction. In the plot below focus on the size of the "plateau" in the left-hand side 1. Naive random direction vs mean & covariance-adjusted random: Naive isotropic random directions are much less sensitive. Thus we use mean & covariance-adjusted random activations everywhere else in this report. 2. The sensitive direction results are related to Gurnee (2024, SAE-replacement-error direction vs naive random direction) and Lindsey (2024, Anthropic April Updates, SAE-feature direction vs naive random direction). The theoretical explanation for activation plateaus & sensitive direction may be error correction (also referred to as noise suppression): NNs in superposition should expect small amounts of noise in feature activations due to interference. (The exact properties depend on how computation happens in s...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: [Interim research report] Activation plateaus & sensitive directions in GPT2, published by StefanHex on July 5, 2024 on LessWrong. This part-report / part-proposal describes ongoing research, but I'd like to share early results for feedback. I am especially interested in any comment finding mistakes or trivial explanations for these results. I will work on this proposal with a LASR Labs team over the next 3 months. If you are working (or want to work) on something similar I would love to chat! Experiments and write-up by Stefan, with substantial inspiration and advice from Jake (who doesn't necessarily endorse every sloppy statement I write). Work produced at Apollo Research. TL,DR: Toy models of how neural networks compute new features in superposition seem to imply that neural networks that utilize superposition require some form of error correction to avoid interference spiraling out of control. This means small variations along a feature direction shouldn't affect model outputs, which I can test: 1. Activation plateaus: Real activations should be resistant to small perturbations. There should be a "plateau" in the output as a function of perturbation size. 2. Sensitive directions: Perturbations towards the direction of a feature should change the model output earlier (at a lower perturbation size) than perturbations into a random direction. I find that both of these predictions hold; the latter when I operationalize "feature" as the difference between two real model activations. As next steps we are planning to Test both predictions for SAE features: We have some evidence for the latter by Gurnee (2024) and Lindsey (2024). Are there different types of SAE features, atomic and composite features? Can we get a handle on the total number of features? If sensitivity-features line up with SAE features, can we find or improve SAE feature directions by finding local optima in sensitivity (similar to how Mack & Turner (2024) find steering vectors)? My motivation for this project is to get data on computation in superposition, and to get dataset-independent evidence for (SAE-)features. Core results & discussion I run two different experiments that test the error correction hypothesis: 1. Activation Plateaus: A real activation is the center of a plateau, in the sense that perturbing the activation affects the model output less than expected. Concretely: applying random-direction perturbations to an activation generated from a random openwebtext input ("real activation") has less effect than applying the same perturbations to a random activation (generated from a Normal distribution). This effect on the model can be measured in KL divergence of logits (shown below) but also L2 difference or cosine similarity of late-layer activations. 2. Sensitive directions: Perturbing a (real) activation into a direction towards another real activation ("poor man's feature directions") affects the model-outputs more than perturbing the same activation into a random direction. In the plot below focus on the size of the "plateau" in the left-hand side 1. Naive random direction vs mean & covariance-adjusted random: Naive isotropic random directions are much less sensitive. Thus we use mean & covariance-adjusted random activations everywhere else in this report. 2. The sensitive direction results are related to Gurnee (2024, SAE-replacement-error direction vs naive random direction) and Lindsey (2024, Anthropic April Updates, SAE-feature direction vs naive random direction). The theoretical explanation for activation plateaus & sensitive direction may be error correction (also referred to as noise suppression): NNs in superposition should expect small amounts of noise in feature activations due to interference. (The exact properties depend on how computation happens in superposition, this toy...

Link to original articleWelcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: [Interim research report] Activation plateaus & sensitive directions in GPT2, published by StefanHex on July 5, 2024 on LessWrong. This part-report / part-proposal describes ongoing research, but I'd like to share early results for feedback. I am especially interested in any comment finding mistakes or trivial explanations for these results. I will work on this proposal with a LASR Labs team over the next 3 months. If you are working (or want to work) on something similar I would love to chat! Experiments and write-up by Stefan, with substantial inspiration and advice from Jake (who doesn't necessarily endorse every sloppy statement I write). Work produced at Apollo Research. TL,DR: Toy models of how neural networks compute new features in superposition seem to imply that neural networks that utilize superposition require some form of error correction to avoid interference spiraling out of control. This means small variations along a feature direction shouldn't affect model outputs, which I can test: 1. Activation plateaus: Real activations should be resistant to small perturbations. There should be a "plateau" in the output as a function of perturbation size. 2. Sensitive directions: Perturbations towards the direction of a feature should change the model output earlier (at a lower perturbation size) than perturbations into a random direction. I find that both of these predictions hold; the latter when I operationalize "feature" as the difference between two real model activations. As next steps we are planning to Test both predictions for SAE features: We have some evidence for the latter by Gurnee (2024) and Lindsey (2024). Are there different types of SAE features, atomic and composite features? Can we get a handle on the total number of features? If sensitivity-features line up with SAE features, can we find or improve SAE feature directions by finding local optima in sensitivity (similar to how Mack & Turner (2024) find steering vectors)? My motivation for this project is to get data on computation in superposition, and to get dataset-independent evidence for (SAE-)features. Core results & discussion I run two different experiments that test the error correction hypothesis: 1. Activation Plateaus: A real activation is the center of a plateau, in the sense that perturbing the activation affects the model output less than expected. Concretely: applying random-direction perturbations to an activation generated from a random openwebtext input ("real activation") has less effect than applying the same perturbations to a random activation (generated from a Normal distribution). This effect on the model can be measured in KL divergence of logits (shown below) but also L2 difference or cosine similarity of late-layer activations. 2. Sensitive directions: Perturbing a (real) activation into a direction towards another real activation ("poor man's feature directions") affects the model-outputs more than perturbing the same activation into a random direction. In the plot below focus on the size of the "plateau" in the left-hand side 1. Naive random direction vs mean & covariance-adjusted random: Naive isotropic random directions are much less sensitive. Thus we use mean & covariance-adjusted random activations everywhere else in this report. 2. The sensitive direction results are related to Gurnee (2024, SAE-replacement-error direction vs naive random direction) and Lindsey (2024, Anthropic April Updates, SAE-feature direction vs naive random direction). The theoretical explanation for activation plateaus & sensitive direction may be error correction (also referred to as noise suppression): NNs in superposition should expect small amounts of noise in feature activations due to interference. (The exact properties depend on how computation happens in superposition, this toy...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Corrigibility = Tool-ness?, published by johnswentworth on June 28, 2024 on LessWrong. Goal of This Post I have never seen anyone give a satisfying intuitive explanation of what corrigibility (in roughly Eliezer's sense of the word) is. There's lists of desiderata, but they sound like scattered wishlists which don't obviously point to a unified underlying concept at all. There's also Eliezer's extremely meta pointer: We can imagine, e.g., the AI imagining itself building a sub-AI while being prone to various sorts of errors, asking how it (the AI) would want the sub-AI to behave in those cases, and learning heuristics that would generalize well to how we would want the AI to behave if it suddenly gained a lot of capability or was considering deceiving its programmers and so on. … and that's basically it.[1] In this post, we're going to explain a reasonably-unified concept which seems like a decent match to "corrigibility" in Eliezer's sense. Tools Starting point: we think of a thing as corrigible exactly insofar as it is usefully thought-of as a tool. A screwdriver, for instance, is an excellent central example of a corrigible object. For AI alignment purposes, the challenge is to achieve corrigibility - i.e. tool-ness - in much more general, capable, and intelligent systems. … that all probably sounds like a rather nebulous and dubious claim, at this point. In order for it to make sense, we need to think through some key properties of "good tools", and also how various properties of incorrigibility make something a "bad tool". We broke off a separate post on what makes something usefully thought-of as a tool. Key ideas: Humans tend to solve problems by finding partial plans with "gaps" in them, where the "gaps" are subproblems which the human will figure out later. For instance, I might make a plan to decorate my apartment with some paintings, but leave a "gap" about how exactly to attach the paintings to the wall; I can sort that out later.[2] Sometimes many similar subproblems show up in my plans, forming a cluster.[3] For instance, there's a cluster (and many subclusters) of subproblems which involve attaching things together. Sometimes a thing (a physical object, a technique, whatever) makes it easy to solve a whole cluster of subproblems. That's what tools are. For instance, a screwdriver makes it easy to solve a whole subcluster of attaching-things-together subproblems. How does that add up to corrigibility? Respecting Modularity One key piece of the above picture is that the gaps/subproblems in humans' plans are typically modular - i.e. we expect to be able to solve each subproblem without significantly changing the "outer" partial plan, and without a lot of coupling between different subproblems. That's what makes the partial plan with all its subproblems useful in the first place: it factors the problem into loosely-coupled subproblems. Claim from the tools post: part of what it means for a tool to solve a subproblem-cluster is that the tool roughly preserves the modularity of that subproblem-cluster. That means the tool should not have a bunch of side effects which might mess with other subproblems, or mess up the outer partial plan. Furthermore, the tool needs to work for a whole subproblem-cluster, and that cluster includes similar subproblems which came up in the context of many different problems. So, the tool needs to robustly not have side effects which mess up the rest of the plan, across a wide range of possibilities for what "the rest of the plan" might be. Concretely: a screwdriver which sprays flames out the back when turned is a bad tool; it usually can't be used to solve most screw-turning subproblems when the bigger plan takes place in a wooden building. Another bad tool: a screwdriver which, when turned, also turns the lights on and off, cau...

Link to original articleWelcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Corrigibility = Tool-ness?, published by johnswentworth on June 28, 2024 on LessWrong. Goal of This Post I have never seen anyone give a satisfying intuitive explanation of what corrigibility (in roughly Eliezer's sense of the word) is. There's lists of desiderata, but they sound like scattered wishlists which don't obviously point to a unified underlying concept at all. There's also Eliezer's extremely meta pointer: We can imagine, e.g., the AI imagining itself building a sub-AI while being prone to various sorts of errors, asking how it (the AI) would want the sub-AI to behave in those cases, and learning heuristics that would generalize well to how we would want the AI to behave if it suddenly gained a lot of capability or was considering deceiving its programmers and so on. … and that's basically it.[1] In this post, we're going to explain a reasonably-unified concept which seems like a decent match to "corrigibility" in Eliezer's sense. Tools Starting point: we think of a thing as corrigible exactly insofar as it is usefully thought-of as a tool. A screwdriver, for instance, is an excellent central example of a corrigible object. For AI alignment purposes, the challenge is to achieve corrigibility - i.e. tool-ness - in much more general, capable, and intelligent systems. … that all probably sounds like a rather nebulous and dubious claim, at this point. In order for it to make sense, we need to think through some key properties of "good tools", and also how various properties of incorrigibility make something a "bad tool". We broke off a separate post on what makes something usefully thought-of as a tool. Key ideas: Humans tend to solve problems by finding partial plans with "gaps" in them, where the "gaps" are subproblems which the human will figure out later. For instance, I might make a plan to decorate my apartment with some paintings, but leave a "gap" about how exactly to attach the paintings to the wall; I can sort that out later.[2] Sometimes many similar subproblems show up in my plans, forming a cluster.[3] For instance, there's a cluster (and many subclusters) of subproblems which involve attaching things together. Sometimes a thing (a physical object, a technique, whatever) makes it easy to solve a whole cluster of subproblems. That's what tools are. For instance, a screwdriver makes it easy to solve a whole subcluster of attaching-things-together subproblems. How does that add up to corrigibility? Respecting Modularity One key piece of the above picture is that the gaps/subproblems in humans' plans are typically modular - i.e. we expect to be able to solve each subproblem without significantly changing the "outer" partial plan, and without a lot of coupling between different subproblems. That's what makes the partial plan with all its subproblems useful in the first place: it factors the problem into loosely-coupled subproblems. Claim from the tools post: part of what it means for a tool to solve a subproblem-cluster is that the tool roughly preserves the modularity of that subproblem-cluster. That means the tool should not have a bunch of side effects which might mess with other subproblems, or mess up the outer partial plan. Furthermore, the tool needs to work for a whole subproblem-cluster, and that cluster includes similar subproblems which came up in the context of many different problems. So, the tool needs to robustly not have side effects which mess up the rest of the plan, across a wide range of possibilities for what "the rest of the plan" might be. Concretely: a screwdriver which sprays flames out the back when turned is a bad tool; it usually can't be used to solve most screw-turning subproblems when the bigger plan takes place in a wooden building. Another bad tool: a screwdriver which, when turned, also turns the lights on and off, cau...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Access to powerful AI might make computer security radically easier, published by Buck Shlegeris on June 8, 2024 on The AI Alignment Forum. People talk about model weight security being really hard and crucial around the advent of AGI. (E.g. RAND report, Leopold; see here for some distinctions in these thread models that I think are important.) But I think that the thinking on this has not been sufficiently attentive to the fact that during that crucial time period, by assumption we'll have access to powerful AIs. I think that such access might make security wildly easier, by a bunch of different mechanisms, some of which I'll describe in this post. The story I'm telling here is pretty different from the main story I've heard people talk about in the past about AIs helping with computer security, which is that AIs can help with hardening software and other infrastructure. Though I agree that that seems really useful and important, here I'm not talking about AIs automating the process of security research, I'm talking about AI making it affordable to adopt policies that improve security in the organization but that wouldn't work (because they'd be too labor-intensive or lead to who-guards-the-guards problems) if you just had humans. In this post I'm going to be ignoring problems related to AIs themselves trying to cause security failures. There are two ways that such AIs affect this situation: The techniques I describe in this post could also be used, AI-control style, to use trusted models to reduce the risk of security failures caused by untrusted models. You can basically just replace "humans" by "untrusted models" everywhere in the text below. When applying these techniques to reduce risk from humans, you have to worry that these techniques will be sabotaged by scheming models. I'll ignore that possibility in this post, and talk about ways that you can improve security using AIs that are assumed not to be intentionally trying to subvert these security measures (which we call " trusted"). If you're worried about your models scheming, you'll have to do some other stuff too (mostly various sting operations), but I think the arguments in this post still mostly hold up. I'll talk about this more in future. I'm going to focus on insider threats here; most threats from outside humans involve first gaining access to impersonation of some insider. Much of this post came out of thinking about the question "how could AI control techniques be applied to humans, instead of AIs"? I don't think these arguments are an airtight case for "SL5 security will be trivial". But they do seem like a plausible hypothesis under which infosec will end up being quite simple to resolve, if labs decide it's an important priority. Four strategies for using powerful AI to improve security Monitoring As your humans take actions, you configure your system to have an AI that looks at these actions and then flags them for further review if they seem suspicious. If the actions seem potentially catastrophic, then the AI might even flag them before they can run. Concretely, I'm imagining that instead of being allowed to directly run any command you write in the bash console of the computer you're working on, whenever you write a command, it and the surrounding context are shown to an AI, which is charged with deciding how suspicious the action is. If it is very suspicious, the AI blocks the command and sends an alert to someone else at the company who is supposed to look it over. If the action is just somewhat suspicious, then it flags it for further review but does not block the action or alert the user. This is somewhat similar to using multi-party authorization. But with abundant AI labor, we could be more comprehensive, with fewer false positives and fewer false negatives, and evaluating a much grea...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Response to Aschenbrenner's "Situational Awareness", published by Rob Bensinger on June 6, 2024 on LessWrong. (Cross-posted from Twitter.) My take on Leopold Aschenbrenner's new report: I think Leopold gets it right on a bunch of important counts. Three that I especially care about: 1. Full AGI and ASI soon. (I think his arguments for this have a lot of holes, but he gets the basic point that superintelligence looks 5 or 15 years off rather than 50+.) 2. This technology is an overwhelmingly huge deal, and if we play our cards wrong we're all dead. 3. Current developers are indeed fundamentally unserious about the core risks, and need to make IP security and closure a top priority. I especially appreciate that the report seems to get it when it comes to our basic strategic situation: it gets that we may only be a few years away from a truly world-threatening technology, and it speaks very candidly about the implications of this, rather than soft-pedaling it to the degree that public writings on this topic almost always do. I think that's a valuable contribution all on its own. Crucially, however, I think Leopold gets the wrong answer on the question "is alignment tractable?". That is: OK, we're on track to build vastly smarter-than-human AI systems in the next decade or two. How realistic is it to think that we can control such systems? Leopold acknowledges that we currently only have guesswork and half-baked ideas on the technical side, that this field is extremely young, that many aspects of the problem look impossibly difficult (see attached image), and that there's a strong chance of this research operation getting us all killed. "To be clear, given the stakes, I think 'muddling through' is in some sense a terrible plan. But it might be all we've got." Controllable superintelligent AI is a far more speculative idea at this point than superintelligent AI itself. I think this report is drastically mischaracterizing the situation. 'This is an awesome exciting technology, let's race to build it so we can reap the benefits and triumph over our enemies' is an appealing narrative, but it requires the facts on the ground to shake out very differently than how the field's trajectory currently looks. The more normal outcome, if the field continues as it has been, is: if anyone builds it, everyone dies. This is not a national security issue of the form 'exciting new tech that can give a country an economic or military advantage'; it's a national security issue of the form 'we've found a way to build a doomsday device, and as soon as anyone starts building it the clock is ticking on how long before they make a fatal error and take themselves out, and take the rest of the world out with them'. Someday superintelligence could indeed become more than a doomsday device, but that's the sort of thing that looks like a realistic prospect if ASI is 50 or 150 years away and we fundamentally know what we're doing on a technical level - not if it's more like 5 or 15 years away, as Leopold and I agree. The field is not ready, and it's not going to suddenly become ready tomorrow. We need urgent and decisive action, but to indefinitely globally halt progress toward this technology that threatens our lives and our children's lives, not to accelerate ourselves straight off a cliff. Concretely, the kinds of steps we need to see ASAP from the USG are: Spearhead an international alliance to prohibit the development of smarter-than-human AI until we're in a radically different position. The three top-cited scientists in AI (Hinton, Bengio, and Sutskever) and the three leading labs (Anthropic, OpenAI, and DeepMind) have all publicly stated that this technology's trajectory poses a serious risk of causing human extinction (in the CAIS statement). It is absurd on its face to let any private company...

Link to original articleWelcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Response to Aschenbrenner's "Situational Awareness", published by Rob Bensinger on June 6, 2024 on LessWrong. (Cross-posted from Twitter.) My take on Leopold Aschenbrenner's new report: I think Leopold gets it right on a bunch of important counts. Three that I especially care about: 1. Full AGI and ASI soon. (I think his arguments for this have a lot of holes, but he gets the basic point that superintelligence looks 5 or 15 years off rather than 50+.) 2. This technology is an overwhelmingly huge deal, and if we play our cards wrong we're all dead. 3. Current developers are indeed fundamentally unserious about the core risks, and need to make IP security and closure a top priority. I especially appreciate that the report seems to get it when it comes to our basic strategic situation: it gets that we may only be a few years away from a truly world-threatening technology, and it speaks very candidly about the implications of this, rather than soft-pedaling it to the degree that public writings on this topic almost always do. I think that's a valuable contribution all on its own. Crucially, however, I think Leopold gets the wrong answer on the question "is alignment tractable?". That is: OK, we're on track to build vastly smarter-than-human AI systems in the next decade or two. How realistic is it to think that we can control such systems? Leopold acknowledges that we currently only have guesswork and half-baked ideas on the technical side, that this field is extremely young, that many aspects of the problem look impossibly difficult (see attached image), and that there's a strong chance of this research operation getting us all killed. "To be clear, given the stakes, I think 'muddling through' is in some sense a terrible plan. But it might be all we've got." Controllable superintelligent AI is a far more speculative idea at this point than superintelligent AI itself. I think this report is drastically mischaracterizing the situation. 'This is an awesome exciting technology, let's race to build it so we can reap the benefits and triumph over our enemies' is an appealing narrative, but it requires the facts on the ground to shake out very differently than how the field's trajectory currently looks. The more normal outcome, if the field continues as it has been, is: if anyone builds it, everyone dies. This is not a national security issue of the form 'exciting new tech that can give a country an economic or military advantage'; it's a national security issue of the form 'we've found a way to build a doomsday device, and as soon as anyone starts building it the clock is ticking on how long before they make a fatal error and take themselves out, and take the rest of the world out with them'. Someday superintelligence could indeed become more than a doomsday device, but that's the sort of thing that looks like a realistic prospect if ASI is 50 or 150 years away and we fundamentally know what we're doing on a technical level - not if it's more like 5 or 15 years away, as Leopold and I agree. The field is not ready, and it's not going to suddenly become ready tomorrow. We need urgent and decisive action, but to indefinitely globally halt progress toward this technology that threatens our lives and our children's lives, not to accelerate ourselves straight off a cliff. Concretely, the kinds of steps we need to see ASAP from the USG are: Spearhead an international alliance to prohibit the development of smarter-than-human AI until we're in a radically different position. The three top-cited scientists in AI (Hinton, Bengio, and Sutskever) and the three leading labs (Anthropic, OpenAI, and DeepMind) have all publicly stated that this technology's trajectory poses a serious risk of causing human extinction (in the CAIS statement). It is absurd on its face to let any private company...

Delivered by Yohanes Maximilian Adi from the Parish of Saint John the Apostles Yogyakarta, Archdiocese of Semarang, Indonesia. Acts of the Apostles 16: 22-34; Rs psalm 138: 1-2a.2b-3.7c-8; John 16: 5-11 THE GOSPEL FOR FAMILIES   Our meditation today has the theme: The Gospel For Families. When Jesus delivered His teachings and counsels to the apostles and disciples before His ascension to heaven, one of his highlight points was that all His flock should be united. The family is one of the most fundamental  forms of fellowship and unity. All those who form one family  of this kind identify themselves as Christians and the gospel of Jesus Christ is the central teaching for their lives at all times.   Concretely speaking, when Saint Paul and his companions spread the Gospel to the pagan territories, for example in the city of Philippi which was inhabited by the Roman immigrants, the emphasis that the Gospel as a book for the family was reaffirmed. We can ensure that if the gospel is the primary means of faith for the family, every member must be reached out and included. On the contrary, when only one or a few in the family get to know the Gospel, it does not ensure that the entire family is empowered by the Gospel.   The event of the miraculous release of Paul and Silas from their imprisonment, then the prison guard who repented, followed by his entire family being baptized in the name of Jesus Christ, these were some of the happenings that illustrated about the power of the Gospel that transformed the whole family. The inspiration of these events should benefit each of us at this moment and to motivate us, that by the power of the Gospel, our families and our communities will also experience sort of transformation in the growth of our faith. The Gospel for families is a great opportunity for learning and evangelization today in our time.   A family that is devoted and actively involved in the sacraments of the Church remains a very important practice of all the followers of Christ. However, this is not enough to make our families true and active Christians. To contribute to the realization of a true Christian living, an important part of the growth of faith is the evangelization to our families and communities. Specifically, our families and our communities need to give a particular priority to their spiritual life, which is the zeal in communion with the Word of God. A real activity that can be made in regular basis in our families or communities should be the listening, the reading and the study of the scriptures that are done together.   We don't have to wonder on the kind of reading and study of the scriptures would be like. The Lord Jesus certainly gives us the way. He assures us that the Holy Spirit, the Comforter, will enlighten and teach us in all truth. The Holy Spirit is always there and willing, then it will depend on our families and communities to determine together on how it will happen.   Let's pray. In the name of the Father ... O God almighty, strengthen us in the realization of the Gospel for our families and communities. Our Father who art in heaven ... In the name of the Father ... --- Send in a voice message: https://podcasters.spotify.com/pod/show/media-la-porta/message

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Transformers Represent Belief State Geometry in their Residual Stream, published by Adam Shai on April 16, 2024 on The AI Alignment Forum. Produced while being an affiliate at PIBBSS[1]. The work was done initially with funding from a Lightspeed Grant, and then continued while at PIBBSS. Work done in collaboration with @Paul Riechers, @Lucas Teixeira, @Alexander Gietelink Oldenziel, and Sarah Marzen. Paul was a MATS scholar during some portion of this work. Thanks to Paul, Lucas, Alexander, Sarah, and @Guillaume Corlouer for suggestions on this writeup. Introduction What computational structure are we building into LLMs when we train them on next-token prediction? In this post we present evidence that this structure is given by the meta-dynamics of belief updating over hidden states of the data-generating process. We'll explain exactly what this means in the post. We are excited by these results because We have a formalism that relates training data to internal structures in LLMs. Conceptually, our results mean that LLMs synchronize to their internal world model as they move through the context window. The computation associated with synchronization can be formalized with a framework called Computational Mechanics. In the parlance of Computational Mechanics, we say that LLMs represent the Mixed-State Presentation of the data generating process. The structure of synchronization is, in general, richer than the world model itself. In this sense, LLMs learn more than a world model. We have increased hope that Computational Mechanics can be leveraged for interpretability and AI Safety more generally. There's just something inherently cool about making a non-trivial prediction - in this case that the transformer will represent a specific fractal structure - and then verifying that the prediction is true. Concretely, we are able to use Computational Mechanics to make an a priori and specific theoretical prediction about the geometry of residual stream activations (below on the left), and then show that this prediction holds true empirically (below on the right). Theoretical Framework In this post we will operationalize training data as being generated by a Hidden Markov Model (HMM)[2]. An HMM has a set of hidden states and transitions between them. The transitions are labeled with a probability and a token that it emits. Here are some example HMMs and data they generate. Consider the relation a transformer has to an HMM that produced the data it was trained on. This is general - any dataset consisting of sequences of tokens can be represented as having been generated from an HMM. Through the discussion of the theoretical framework, let's assume a simple HMM with the following structure, which we will call the Z1R process[3] (for "zero one random"). The Z1R process has 3 hidden states, S0,S1, and SR. Arrows of the form Sxa:p%Sy denote P(Sy,a|Sx)=p%, that the probability of moving to state Sy and emitting the token a, given that the process is in state Sx, is p%. In this way, taking transitions between the states stochastically generates binary strings of the form ...01R01R... where R is a random 50/50 sample from { 0, 1}. The HMM structure is not directly given by the data it produces. Think of the difference between the list of strings this HMM emits (along with their probabilities) and the hidden structure itself[4]. Since the transformer only has access to the strings of emissions from this HMM, and not any information about the hidden states directly, if the transformer learns anything to do with the hidden structure, then it has to do the work of inferring it from the training data. What we will show is that when they predict the next token well, transformers are doing even more computational work than inferring the hidden data generating process! Do Transformers Learn ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Transformers Represent Belief State Geometry in their Residual Stream, published by Adam Shai on April 16, 2024 on LessWrong. Produced while being an affiliate at PIBBSS[1]. The work was done initially with funding from a Lightspeed Grant, and then continued while at PIBBSS. Work done in collaboration with @Paul Riechers, @Lucas Teixeira, @Alexander Gietelink Oldenziel, and Sarah Marzen. Paul was a MATS scholar during some portion of this work. Thanks to Paul, Lucas, Alexander, and @Guillaume Corlouer for suggestions on this writeup. Introduction What computational structure are we building into LLMs when we train them on next-token prediction? In this post we present evidence that this structure is given by the meta-dynamics of belief updating over hidden states of the data-generating process. We'll explain exactly what this means in the post. We are excited by these results because We have a formalism that relates training data to internal structures in LLMs. Conceptually, our results mean that LLMs synchronize to their internal world model as they move through the context window. The computation associated with synchronization can be formalized with a framework called Computational Mechanics. In the parlance of Computational Mechanics, we say that LLMs represent the Mixed-State Presentation of the data generating process. The structure of synchronization is, in general, richer than the world model itself. In this sense, LLMs learn more than a world model. We have increased hope that Computational Mechanics can be leveraged for interpretability and AI Safety more generally. There's just something inherently cool about making a non-trivial prediction - in this case that the transformer will represent a specific fractal structure - and then verifying that the prediction is true. Concretely, we are able to use Computational Mechanics to make an a priori and specific theoretical prediction about the geometry of residual stream activations (below on the left), and then show that this prediction holds true empirically (below on the right). Theoretical Framework In this post we will operationalize training data as being generated by a Hidden Markov Model (HMM)[2]. An HMM has a set of hidden states and transitions between them. The transitions are labeled with a probability and a token that it emits. Here are some example HMMs and data they generate. Consider the relation a transformer has to an HMM that produced the data it was trained on. This is general - any dataset consisting of sequences of tokens can be represented as having been generated from an HMM. Through the discussion of the theoretical framework, let's assume a simple HMM with the following structure, which we will call the Z1R process[3] (for "zero one random"). The Z1R process has 3 hidden states, S0,S1, and SR. Arrows of the form Sxa:p%Sy denote P(Sy,a|Sx)=p%, that the probability of moving to state Sy and emitting the token a, given that the process is in state Sx, is p%. In this way, taking transitions between the states stochastically generates binary strings of the form ...01R01R... where R is a random 50/50 sample from { 0, 1}. The HMM structure is not directly given by the data it produces. Think of the difference between the list of strings this HMM emits (along with their probabilities) and the hidden structure itself[4]. Since the transformer only has access to the strings of emissions from this HMM, and not any information about the hidden states directly, if the transformer learns anything to do with the hidden structure, then it has to do the work of inferring it from the training data. What we will show is that when they predict the next token well, transformers are doing even more computational work than inferring the hidden data generating process! Do Transformers Learn a Model of the World...

Link to original articleWelcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Transformers Represent Belief State Geometry in their Residual Stream, published by Adam Shai on April 16, 2024 on LessWrong. Produced while being an affiliate at PIBBSS[1]. The work was done initially with funding from a Lightspeed Grant, and then continued while at PIBBSS. Work done in collaboration with @Paul Riechers, @Lucas Teixeira, @Alexander Gietelink Oldenziel, and Sarah Marzen. Paul was a MATS scholar during some portion of this work. Thanks to Paul, Lucas, Alexander, and @Guillaume Corlouer for suggestions on this writeup. Introduction What computational structure are we building into LLMs when we train them on next-token prediction? In this post we present evidence that this structure is given by the meta-dynamics of belief updating over hidden states of the data-generating process. We'll explain exactly what this means in the post. We are excited by these results because We have a formalism that relates training data to internal structures in LLMs. Conceptually, our results mean that LLMs synchronize to their internal world model as they move through the context window. The computation associated with synchronization can be formalized with a framework called Computational Mechanics. In the parlance of Computational Mechanics, we say that LLMs represent the Mixed-State Presentation of the data generating process. The structure of synchronization is, in general, richer than the world model itself. In this sense, LLMs learn more than a world model. We have increased hope that Computational Mechanics can be leveraged for interpretability and AI Safety more generally. There's just something inherently cool about making a non-trivial prediction - in this case that the transformer will represent a specific fractal structure - and then verifying that the prediction is true. Concretely, we are able to use Computational Mechanics to make an a priori and specific theoretical prediction about the geometry of residual stream activations (below on the left), and then show that this prediction holds true empirically (below on the right). Theoretical Framework In this post we will operationalize training data as being generated by a Hidden Markov Model (HMM)[2]. An HMM has a set of hidden states and transitions between them. The transitions are labeled with a probability and a token that it emits. Here are some example HMMs and data they generate. Consider the relation a transformer has to an HMM that produced the data it was trained on. This is general - any dataset consisting of sequences of tokens can be represented as having been generated from an HMM. Through the discussion of the theoretical framework, let's assume a simple HMM with the following structure, which we will call the Z1R process[3] (for "zero one random"). The Z1R process has 3 hidden states, S0,S1, and SR. Arrows of the form Sxa:p%Sy denote P(Sy,a|Sx)=p%, that the probability of moving to state Sy and emitting the token a, given that the process is in state Sx, is p%. In this way, taking transitions between the states stochastically generates binary strings of the form ...01R01R... where R is a random 50/50 sample from { 0, 1}. The HMM structure is not directly given by the data it produces. Think of the difference between the list of strings this HMM emits (along with their probabilities) and the hidden structure itself[4]. Since the transformer only has access to the strings of emissions from this HMM, and not any information about the hidden states directly, if the transformer learns anything to do with the hidden structure, then it has to do the work of inferring it from the training data. What we will show is that when they predict the next token well, transformers are doing even more computational work than inferring the hidden data generating process! Do Transformers Learn a Model of the World...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Defence production act and AI policy, published by NathanBarnard on March 2, 2024 on LessWrong. Quick Summary Gives the President wide-ranging powers to strengthen the US industrial base Has been around without changing that much since 1953 Has provisions which allow firms to make voluntary agreements that would normally be illegal under antitrust law Provided the legal authority for many of the provisions in Biden's recent Executive Order on AI The Defence Production Act The Defence Production Act (DPA) has been reauthorised (and modified) by Congress since 1950, and in 1953 its powers very significantly reduced. I'm confident that it will continue to be passed - in a roughly similar form - for the foreseeable future. The current version was passed in 2019 under a Republican senate and is due for reauthorisation in 2025. Since the Obama Presidency, there's Republicans have begun to try to prevent bills proposed by Democrats from being passed by default. This is particularly easy for non-spending bills since for non-spending bills, 60 votes are needed to break the filibuster - a method used to prevent bills from being voted on - in the Senate. However, not only are defence bills consistently bipartisan, they have consistently high degrees of support from Republicans in particular. Therefore, I'm not concerned about the DPA not being passed by a Republican senate and a Democratic President when it's next due for reauthorisation. The DPA gives the President very wide-ranging powers since the goal of the act of the act is to ensure that the US industrial base is strong enough to fight and win any war the US might need to undertake. Concretely, this allows the President to instruct firms to accept contracts; incentive expansion of the industrial base; and a grab bag of other specific provisions aimed at making sure that the US production base is strong enough to win a war. Until 1953 the act was much more powerful and essentially allowed the President to take control of the US economy. The act now doesn't give the President authority to set wage or price ceilings, control consumer credit or requisition stuff. Antitrust provisions Various AI governance proposals rely on explicit, voluntary agreements between leading AI labs. For instance, this paper proposes a scheme in which AI firms agree to pause the rollout out and training of large models if one doesn't pass an evaluation which indicates that it could act dangerously. I think it's plausible that this agreement would be illegal under antitrust law. An agreement like this would be an explicit agreement amongst a small number of leading firms to limit supply. This skirts pretty close to being a criminal violation of US antitrust law. Under this law, various forms of agreements between firms to fix prices are considered illigal no matter what firms say is the justification for them (this is known as per se illegal.) Agreements to limit production are considered just as illegal. It's not at all clear that such agreements would be illegal - for instance, professional standards are not considered per se illegal but instead are judged under a rule of reason where their competitive effects need to be outweighed by their procompetitive effects. I won't comment on this further but instead, refer the reader to this excellent by that looks specifically at anti-trust and AI industry self-regulation. Section 708 of the DPA gives the President authority to allow firms to make agreements for that would normally be considered antitrust violations. Recently, this provision was used by the Trump administration during Covid-19. Use in the Biden executive order Some of the most AI safety-relevant elements of the recent Biden executive order on AI were authorised under the legal authority of the DPA. This includes: Requiring AI firms t...

Link to original articleWelcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Defence production act and AI policy, published by NathanBarnard on March 2, 2024 on LessWrong. Quick Summary Gives the President wide-ranging powers to strengthen the US industrial base Has been around without changing that much since 1953 Has provisions which allow firms to make voluntary agreements that would normally be illegal under antitrust law Provided the legal authority for many of the provisions in Biden's recent Executive Order on AI The Defence Production Act The Defence Production Act (DPA) has been reauthorised (and modified) by Congress since 1950, and in 1953 its powers very significantly reduced. I'm confident that it will continue to be passed - in a roughly similar form - for the foreseeable future. The current version was passed in 2019 under a Republican senate and is due for reauthorisation in 2025. Since the Obama Presidency, there's Republicans have begun to try to prevent bills proposed by Democrats from being passed by default. This is particularly easy for non-spending bills since for non-spending bills, 60 votes are needed to break the filibuster - a method used to prevent bills from being voted on - in the Senate. However, not only are defence bills consistently bipartisan, they have consistently high degrees of support from Republicans in particular. Therefore, I'm not concerned about the DPA not being passed by a Republican senate and a Democratic President when it's next due for reauthorisation. The DPA gives the President very wide-ranging powers since the goal of the act of the act is to ensure that the US industrial base is strong enough to fight and win any war the US might need to undertake. Concretely, this allows the President to instruct firms to accept contracts; incentive expansion of the industrial base; and a grab bag of other specific provisions aimed at making sure that the US production base is strong enough to win a war. Until 1953 the act was much more powerful and essentially allowed the President to take control of the US economy. The act now doesn't give the President authority to set wage or price ceilings, control consumer credit or requisition stuff. Antitrust provisions Various AI governance proposals rely on explicit, voluntary agreements between leading AI labs. For instance, this paper proposes a scheme in which AI firms agree to pause the rollout out and training of large models if one doesn't pass an evaluation which indicates that it could act dangerously. I think it's plausible that this agreement would be illegal under antitrust law. An agreement like this would be an explicit agreement amongst a small number of leading firms to limit supply. This skirts pretty close to being a criminal violation of US antitrust law. Under this law, various forms of agreements between firms to fix prices are considered illigal no matter what firms say is the justification for them (this is known as per se illegal.) Agreements to limit production are considered just as illegal. It's not at all clear that such agreements would be illegal - for instance, professional standards are not considered per se illegal but instead are judged under a rule of reason where their competitive effects need to be outweighed by their procompetitive effects. I won't comment on this further but instead, refer the reader to this excellent by that looks specifically at anti-trust and AI industry self-regulation. Section 708 of the DPA gives the President authority to allow firms to make agreements for that would normally be considered antitrust violations. Recently, this provision was used by the Trump administration during Covid-19. Use in the Biden executive order Some of the most AI safety-relevant elements of the recent Biden executive order on AI were authorised under the legal authority of the DPA. This includes: Requiring AI firms t...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Introducing Alignment Stress-Testing at Anthropic, published by Evan Hubinger on January 12, 2024 on The AI Alignment Forum. Following on from our recent paper, "Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training", I'm very excited to announce that I have started leading a new team at Anthropic, the Alignment Stress-Testing team, with Carson Denison and Monte MacDiarmid as current team members. Our mission - and our mandate from the organization - is to red-team Anthropic's alignment techniques and evaluations, empirically demonstrating ways in which Anthropic's alignment strategies could fail. The easiest way to get a sense of what we'll be working on is probably just to check out our "Sleeper Agents" paper, which was our first big research project. I'd also recommend Buck and Ryan's post on meta-level adversarial evaluation as a good general description of our team's scope. Very simply, our job is to try to prove to Anthropic - and the world more broadly - (if it is in fact true) that we are in a pessimistic scenario, that Anthropic's alignment plans and strategies won't work, and that we will need to substantially shift gears. And if we don't find anything extremely dangerous despite a serious and skeptical effort, that is some reassurance, but of course not a guarantee of safety. Notably, our goal is not object-level red-teaming or evaluation - e.g. we won't be the ones running Anthropic's RSP-mandated evaluations to determine when Anthropic should pause or otherwise trigger concrete safety commitments. Rather, our goal is to stress-test that entire process: to red-team whether our evaluations and commitments will actually be sufficient to deal with the risks at hand. We expect much of the stress-testing that we do to be very valuable in terms of producing concrete model organisms of misalignment that we can iterate on to improve our alignment techniques. However, we want to be cognizant of the risk of overfitting, and it'll be our responsibility to determine when it is safe to iterate on improving the ability of our alignment techniques to resolve particular model organisms of misalignment that we produce. In the case of our "Sleeper Agents" paper, for example, we think the benefits outweigh the downsides to directly iterating on improving the ability of our alignment techniques to address those specific model organisms, but we'd likely want to hold out other, more natural model organisms of deceptive alignment so as to provide a strong test case. Some of the projects that we're planning on working on next include: Concretely stress-testing Anthropic's ASL-3 evaluations. Applying techniques from "Towards Monosemanticity: Decomposing Language Models With Dictionary Learning" to our "Sleeper Agents" models. Building more natural model organisms of misalignment, e.g. finding a training pipeline that we might realistically use that we can show would lead to a concrete misalignment failure. If any of this sounds interesting to you, I am very much hiring! We are primarily looking for Research Engineers with strong backgrounds in machine learning engineering work. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What's up with LLMs representing XORs of arbitrary features?, published by Sam Marks on January 3, 2024 on The AI Alignment Forum. Thanks to Clément Dumas, Nikola Jurković, Nora Belrose, Arthur Conmy, and Oam Patel for feedback. In the comments of the post on Google Deepmind's CCS challenges paper, I expressed skepticism that some of the experimental results seemed possible. When addressing my concerns, Rohin Shah made some claims along the lines of "If an LLM linearly represents features a and b, then it will also linearly represent their XOR, ab, and this is true even in settings where there's no obvious reason the model would need to make use of the feature ab."[1] For reasons that I'll explain below, I thought this claim was absolutely bonkers, both in general and in the specific setting that the GDM paper was working in. So I ran some experiments to prove Rohin wrong. The result: Rohin was right and I was wrong. LLMs seem to compute and linearly represent XORs of features even when there's no obvious reason to do so. I think this is deeply weird and surprising. If something like this holds generally, I think this has importance far beyond the original question of "Is CCS useful?" In the rest of this post I'll: Articulate a claim I'll call "representation of arbitrary XORs (RAX)": LLMs compute and linearly represent XORs of arbitrary features, even when there's no reason to do so. Explain why it would be shocking if RAX is true. For example, without additional assumptions, RAX implies that linear probes should utterly fail to generalize across distributional shift, no matter how minor the distributional shift. (Empirically, linear probes often do generalize decently.) Present experiments showing that RAX seems to be true in every case that I've checked. Think through what RAX would mean for AI safety research: overall, probably a bad sign for interpretability work in general, and work that relies on using simple probes of model internals (e.g. ELK probes or coup probes) in particular. Make some guesses about what's really going on here. Overall, this has left me very confused: I've found myself simultaneously having (a) an argument that AB, (b) empirical evidence of A, and (c) empirical evidence of B. (Here A = RAX and B = other facts about LLM representations.) The RAX claim: LLMs linearly represent XORs of arbitrary features, even when there's no reason to do so To keep things simple, throughout this post, I'll say that a model linearly represents a binary feature f if there is a linear probe out of the model's latent space which is accurate for classifying f; in this case, I'll denote the corresponding direction as vf. This is not how I would typically use the terminology "linearly represents" - normally I would reserve the term for a stronger notion which, at minimum, requires the model to actually make use of the feature direction when performing cognition involving the feature[2]. But I'll intentionally abuse the terminology here because I don't think this distinction matters much for what I'll discuss. If a model linearly represents features a and b, then it automatically linearly represents ab and ab. However, ab is not automatically linearly represented - no linear probe in the figure above would be accurate for classifying ab. Thus, if the model wants to make use of the feature ab, then it needs to do something additional: allocate another direction[3] (more model capacity) to representing ab, and also perform the computation of ab so that it knows what value to store along this new direction. The representation of arbitrary XORs (RAX) claim, in its strongest form, asserts that whenever a LLM linearly represents features a and b, it will also linearly represent ab. Concretely, this might look something like: in layer 5, the model computes and linearly r...

Guillermo Rauch, Founder and CEO of Vercel, joins Corey on Screaming in the Cloud to discuss how he decided to focus on building a front-end tool that is fast, reliable, and focuses on the developer experience. Guillermo explains how he discovered that Javascript was the language that set online offerings apart, and also reveals the advice he gives to founders on how to build an effective landing page. Corey and Guillermo discuss the effects of generative AI on developer experience, and Guillermo explains why Vercel had a higher standard for accuracy when rolling out their new AI product for developers, v0. About GuillermoGuillermo Rauch is Founder and CEO of Vercel, where he leads the company's mission to enable developers to create at the moment of inspiration. Prior to founding Vercel, Guillermo co-founded LearnBoost and Cloudup where he served the company as CTO through its acquisition by Automattic in 2013. Originally from Argentina,Guillermo has been a developer since the age of ten and is passionate about contributing to the open source community. He has created a number of JavaScript projects including socket.io, Mongoose.js, Now, and Next.js.Links Referenced: Vercel: https://vercel.com/ v0.dev: https://v0.dev Personal website: https://rauchg.com Personal twitter: https://twitter.com/rauchg TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. I don't talk a lot about front-end on this show, primarily because I am very bad at front-end, and in long-standing tech tradition, if I'm not good at something, apparently I'm legally obligated to be dismissive of it and not give it any attention. Strangely enough, I spent the last week beating on some front-end projects, and now I'm not just dismissive, I'm angry about it. Here to basically suffer the outpouring of frustration and confusion is Guillermo Rauch, founder and CEO of Vercel, but also the creator of Next.js. Guillermo, thank you for joining me.Guillermo: Great to be here. Thanks for setting me up with that awesome intro.Corey: It's true, if I were talking to someone who looked at what I've done, and for some Godforsaken reason, they wanted to follow in my footsteps, well, that path has been closed, so learning a bunch of Perl early on and translating it all to bad bash scripts and the rest, and then maybe picking up Python isn't really the way that I would advise someone getting started today. The de facto lingua franca of the internet is JavaScript, whether we like it or not, and I would strongly suggest that be someone's first language despite the fact that I'm bad at it, I don't understand it, and therefore it makes me angry.Guillermo: Yeah, it's so funny because it sounds like my story. And my personal journey was, when I was a kid, I had a—I knew I wanted to hack around with computers, and reverse engineer them, and improve them, and just create my own things, and I had these options for what programming language I could go with. And I tried it all: PHP, Perl, [Mod PHP 00:02:12], [Mod Perl 00:02:13], Apache, LAMP, cgi-bin folders, all the whole nine yards. And regardless of what back-end technology I used, I encountered this striking fact, which was… the thing that can make your product really stand out in a web browser is typically involving JavaScript in some fashion. So, when Google came out with suggestions as you would type in a search box, my young kid Argentinian brain blew up. I was like, “Holy crap, they can suggest, they can read my mind, and they can render suggestions without a full page refresh? What is that magic?”And then more products like that came out. Google Docs, Gmail Chat, Facebook's real time newsfeed, all the great things on the internet seemed to have this common point of, there's this layer of interactivity, real-time data, customization, personalization, and it seems uniquely enabled by the front-end. So, I just went all in. I taught myself how to code, I taught myself—I became a front-end engineering expert, I joined some of the early projects that shaped the ecosystem. Like, there was this library called MooTools, and a lot of folks might not have heard that name. It's in the annals of JavaScript history.And later on, you know, what I realized is, what if front-end can actually be the starting point of how you develop the best applications, right, rather than this thing that people, like, reluctantly frown upon, like yourself. I mention that as an opportunity rather than a diss because when you create a great front-end experience, now the data has proven you run a better business, you run a more dynamic business, probably are running an AI-powered business, like, all of the AI products on the planet today are using this technology to stream text in front of your eyes in real time and do all this awesome things. So yeah, I became obsessed with front-end, and I founded this company Vercel, which is a front-end cloud. So, you come here to basically build the best products. Now, you don't have to build the back-end, so you can use back-ends that are off the shelf, you can connect to your existing back-ends, and we piggyback on the world's best infrastructure to make this possible, but we offer developers a very streamlined path to create these awesome products on the internet.Corey: I have to say that I have been impressed when I've used Vercel for a number of projects. And what impresses me is less the infrastructure powering it, less the look at how performance it is and all the stuff that most people talk about, but as mentioned, I am not good at front-end or frankly programming at all. And so, many products in this space fall into the very pernicious trap of, “Oh, well, everyone who's using this is at least this tall on the board of how smart you are to get on the amusement park ride.” So, I feel that when I'm coming at this from a—someone who is not a stranger to computers but is definitely new to this entire ecosystem, everything just made sense in a way that remarkably few products can pull off. I don't know if you would call that user experience, developer experience, or what the terminology you bias for there is, but it is a transformational difference.Guillermo: Thank you. I think it's a combination of things. So, developer experience has definitely always been a focus for me. I was that weird person that obsessed about the CLI parameters of the tool, and the output of the tool, and just like how it feels for the engineer. I did combine that with—and I think this is where Vercel really stands out—I did combine that with a world-class infrastructure bit because what I realized after creating lots and lots of very popular open-source projects—like, one is called Socket.io, and other one called Mongoose—DX, or developer experience, in the absence of an enticing carrot for the business doesn't work. Maybe it has some short term adoption, maybe it has raving fans on Twitter or X [laugh], but at the end of the day, you have to deliver something that's tangible to the end-user and to the business.So I think Vercel focusing on the front-end has found a magical combination there of I can make the developer lives easier. Being a developer myself, I just tremendously empathize with that, but it can also make more profit for the business. When they make your website faster and render more dynamic data that serve as better recommendations for a product on e-commerce or in a marketing channel, I can help you roll out more experiments, then they make your business better, and I think that's one of the magical combinations there.The other thing, frankly, is that we'd started doing fewer things. So, when you come to Vercel, you typically come with a framework that you've already chosen. It could be Next.js, it could be View, [unintelligible 00:07:18], there's 35-plus frameworks. But we basically told the market, you have to use one of these developer tools in order to guide your development.And what companies were doing before—I mean, this almost seems obvious in retrospect, that we would optimize for her certain patterns and certain tools—but what the market was doing before was rolling out your own frameworks. Like, every company was, basically—React, for example, is a very popular way of building interfaces, and our framework actually is built on top of React. But when I would go to and talk to all these principal engineers that all these companies, they were saying, oh yeah, “We're creating our own framework. We're creating our own tools.” And I think that to me now feels almost like a zero interest rate phenomenon. Like, what business do you have in creating frameworks, tools, and bespoke infra when you're really in the business of creating delightful experiences for your customers?Corey: What I think is lost on a lot of folks is that if you are trying to learn something new and use a tool, and the developer experience is bad, the takeaway—at least for me and a lot of people that I talk to is not, “Oh, this tool has terrible ergonomics. That's it's problem.” Instead, the takeaway is very much, “Oh, I'm dumb because I don't understand this thing.” And I know intellectually that I am not usually the dumbest person in the world when it comes to a particular tool or technology, but I keep forgetting that on a visceral level. It's, “I just wish I was smart enough to understand that.”No, I don't. I wish it was presented in a way that was more understandable and the documentation was better. When you're just starting out and building something in your spare time, the infrastructure cost is basically nothing, but your time is the expensive part in it. So, if you have to spend three hours to track down something just because it wasn't clearly explained, the burden of adopting that tool is challenging. I would argue that one of the reasons that AWS sees some of the success that it does is not necessarily because it's great so much as because everyone knows how it breaks. That's important.I'm not saying their infrastructure isn't world-class—please, don't come at me in the comments on this one—but I am saying that we know where its sharp edges are, and that means that we're more comfortable building with it. But the idea of learning a brand-new cloud with different sharp edges in other areas. That's terrifying. I'd rather stick with the devil I know.Guillermo: Exactly. I just think that you're not going to be able to make a difference for customers in 2023 by creating another bespoke cloud that is general purpose, it doesn't really optimize around anything, and you have to learn all the sharp edges from scratch. I think we saw that with the rise of cloud-native companies like Stripe and Twilio where they were going after these amazingly huge markets like financial infrastructure or communications infrastructure, but the angle was, “Here's this awesome developer experience.” And that's what we're doing with Vercel for the front-end and for building products, right? There has to be an opinionated developer experience that guides you to success.And I agree with you that there's really, these days in the developer communities, zero tolerance for sharp edges, and we've spent a lot of time in—even documentation, like, it used to be that your startup would make or break it by whether you had great documentation. I think in the age of frameworks, I would even dare say that documentation, of course, is extremely important, but if I can have the tool itself guide you to success, at that point, you're not even reading documentation. We're now seeing this with AI and, like, generative AI for code. At Vercel, we're investing in generative AI for user interfaces. Do you actually need to read documentation at that point? So, I think we're optimizing for the absolute minimal amount of friction required to be successful with these platforms.Corey: I think that there's a truth in that of meeting customers in where they are. Your documentation can be spectacular, but people don't generally read the encyclopedia for fun either. And the idea of that is that—at least ideally—I should not need to go diving into the documentation, and so many tools get this wrong, where, “Oh, I want to set up a new project,” and it bombards you with 50 questions, and each one of these feels pretty… momentous. Like, what one-way door am I passing through that I don't realize the consequences of until I'm 12 hours into this thing, and then have to backtrack significantly. I like, personally, things that bias for having a golden path, but also make it easy to both deviate from it, as well as get back onto it. Because there's more than one way to do it is sort of the old Perl motto. That is true in spades in anything approaching the JavaScript universe.Guillermo: Yeah. I have a lot of thoughts on that. On the first point, I completely agree that the golden path of the product cannot be documentation-mediated. One of the things that I've become obsessive about—and this is an advice that I share with a lot of other startup founders is, when it comes to your landing page, the primary call to action has to be this golden path to success, like, 2, 3, 4 clicks later, I have to have something tangible. That was our inspiration.And when we made it the primary call to action for Vercel is deploy now. Start now. Get it out now. Ship it now. And the way that you test out the platform is by deploying a template. What do we do is we create a Git repo for you, it sets up the entire CI/CD pipeline, and then at that point, you already have something working, something in the cloud, you spent zero time reading documentation, and you can start iterating.And even though that might not be the final thing you do in Vercel, I always hear the stories of CTOs that are now deploying Vercel at really large scale, and they always tell me, “I started with your hobby tier, I started with free tier, I deployed a template, I hacked on a product during the weekend.” Now, a lot of our AI examples are very popular in this crowd. And yeah, there's a golden path that requires zero documentation. Now, you also mentioned that, what about complexity? This is an enterprise-grade platform. What about escape hatches? What about flexibility? And that's where our platform also shines because we have the entire power of a Turing-complete language, which is JavaScript and TypeScript, to customize every aspect of the platform.And you have a framework that actually answered a lot of the problems that came with serverless solutions in the past, which is that you couldn't run any of that on your local machine. The beauty of Vercel and Next.js is we kind of pioneered this concept that we called ‘Framework Defined Infrastructure.' You start with the framework, the framework has this awesome property that you can install on your computer, it has a dev command—like, it literally runs on your computer—but then when you push it to the cloud, it now defines the infrastructure. It creates all of the resources that are highly optimal.This creates—basically converts what was a single node system on your computer to a globally distributed system, which is a very complex and difficult engineering challenge, but Vercel completely automates it away. And now for folks that are looking for, like, more advanced solutions, they can now start poking into the outputs of that compilation process and say, “Okay, I can now have an influence or I can reconfigure some aspects of this pipeline.” And of course, if you don't think about those escape hatches, then the product just ends up being limiting and frustrating, so we had to think really hard about meeting both ends of the spectrum.Corey: In my own experimentations early on with Chat-Gippity—which is how I insist on pronouncing ChatGPT—a lot of what I found was that it was a lot more productive for me to, instead of asking just the question and getting the answer was, write a Python script to—Guillermo: Yes.Corey: Query this API to get me that answer. Because often it would be wrong. Sometimes very convincingly wrong, and I can at least examine it in various ways and make changes to it and iterate forward, whereas when everything is just a black box, that gets very hard to do. The idea of building something that can be iterated on is key.Guillermo: I love that. The way that Vercel actually first introduced itself to the world was this idea of immutable deployments and immutable infrastructure. And immutable sounds like a horrible word because I want to mutate things, but it was inspired by this idea of functional programming where, like, each iteration to the state, each data change, can be tracked. So, every time you deploy in Vercel, you get this unique URL that represents a point-in-time infrastructure deployment. You can go back in time, you can revert, you can use this as a way of collaborating with other engineers in your team, so you can send these hyperlinks around to your front-end projects.And it gives you a lot of confidence. Now, you can iterate knowing that before things go out, there's a lot of scrutiny, there's a lot of QA, there's a lot of testing processes that you can kick off against this serverless infrastructure that was created for each deployment. The conclusion for us so far has been that our role in the world is to increase iteration velocity. So, iteration speed is the faster horse of the cloud, right? Like, instead of getting a car, you get a faster horse.When you say, “Okay, I made the build pipeline 10% faster,” or, “I brought the TLS termination 10% closer to the visitor, and, like, I have more [pops 00:17:10],” things like that. That to me, is the speed. You can do those things, and they're awesome, but if you don't have a direction—which is velocity—then you don't know what you're building next. You don't know if your customers are happy. You don't know if you're delivering value. So, we built an entire platform that optimizes around, what should you ship next? What is the friction involved in getting your next iteration out? Is launching an experiment on your homepage, for example, is that a costly endeavor? Does it take you weeks? Does it take you months?One of the initial inspirations for just starting Vercel and making deployments really easy was, how difficult is it for the average company to change in their footer of their website is this copyright 2022? And you have—it's a new year. You have to bump it to copyright 2023. How long do you think it takes that engineer to, A, run the stack locally, so they can actually see the change; deploy it, but deploy to what we call the preview environment, so they can grab that URL and send [it to 00:18:15], Corey, and say, “Corey, does it look good? I updated [laugh] I updated the year in the footer.”And then you tell me, “Looks good, let's ship it to production.” Or you tell me, “No, no, no, it's risky. Let's divide it into two cohorts: 50% of traffic gets 2022, 50% of traffic gets 2023.” Obviously, this is a joke, but consider the implications of how difficult it is and the average organization to actually do this thing.Corey: Oh, I find things like that all the time, especially on microservices that I built to handle some of my internal workflows here, and I haven't touched in two or three years. And okay, now it's time for me to update them to reflect some minor change. And first, I wind up in the screaming node warnings and I have to update things so that they actually work in a reasonable way. And, on some level, making a one-line change can take half a day. Now, in the real world, when people are working on these apps day-in and day-out, it gets a lot easier to roll those changes in over time, but coming back to something unmaintained, that becomes a project the longer you let it sit.Part of me wishes that there were easier ways around it, but there are trade-offs in almost any decision you make. If you're building something from the beginning of, well, I want to be able to automatically update the copyright year, you can even borderline make that something that automatically happens based upon the global time, whereas when you're trying to retrofit it afterwards, yeah, it becomes a project.Guillermo: Yeah, and now think, that's just a simple example of changing a string. That might be difficult for a product engineering in any organization. Or it may be slow, or it may be not as streamlined, or maybe it works really well for the first project that that company created. What about every incremental project thereafter?So, now I said—let's stop talking about a string, right? Let's think about you're about an e-commerce website where what we hear from our customers on average, like, 10% of revenue flows through the homepage. Now, I have to change a primary component that renders on the hero of the page, and I have to collaborate with every department in the organization. I have to collaborate with the design team, I have to collaborate with marketing, I have to collaborate with the business owners to track the analytics appropriately. So, what is the cost of every incremental experiment that you want to put in production?The other thing that's particularly interesting about front-end as it relates to cloud infrastructure is, scaling up front-end is a very difficult thing. What ends up happening is most front-ends are actually static websites. They're cached at the edge—or they're literally statically generated—and then they push all of the dynamism to the client side. So, you end up with this spaghetti of script tags on the client, you end up accumulating a lot of tech debt in the [shipping 00:20:56] huge bundles of JavaScript to the client to try to recover some dynamism, to try and run these experiments. So, everyone is in this, kind of, mess of the yes, maybe we can experiment, but we kind of offloaded the rendering work to the client. That in turn makes me—basically, I'm making the website slower for the visitor. I'm making them do the rendering work.And I'm trying to sell them something. I'm trying to speed up some processes. It's my responsibility to make it fast. So, what we ended up finding out is that yes, the cloud moved this forward a lot in terms of having these awesome building blocks, these awesome infrastructure primitives, but both in the developer experience, just changing something about your web product and also the end-user experiences, that web product renders really fast, those things really didn't happen with this first chapter of the cloud. And I think we're entering a new generation of higher-level clouds like Vercel that are optimizing for these things.Corey: I think that there's a historical focus on things that have not happened before. And that was painful and terrible, so we're not going to be focusing on what's happening in the future, we're going to build a process or a framework or something that winds up preventing that thing that hurt us from hurting us again. Now, that's great in moderation, but at some point—we see this at large companies from time-to-time—where you have so much process that is ossified scar tissue, basically, that it becomes almost impossible to get things done. Because oh, I want to make that—for example, that one-line change to a copyright date, well, here's the 5000 ways deploys have screwed us before, so we need to have three humans sign off on the one-line change, and a bunch of other stuff before it ever sees the light of day. Now, I'm exaggerating slightly, just as you are, but that feels like it acts as a serious brake on innovation.On the exact opposite side, where we see massive acceleration has been around the world of generative AI. Yes, it is massively hyped in a bunch of ways. I don't think it is going to be a defined way that changes the nature of humanity the way that some of these people are going after, but it's also clearly more than a parlor trick.Guillermo: I'm kind of in that camp. So, like you, I've been writing code for many years. I'm pretty astonished by the AI's ability to enhance my output. And of course, now I'm not writing code full time, so there is a sense of, okay because I don't have time, because I'm doing a million things, any minute I have seems like AI has just made it so much more worthwhile and I can squeeze so much more productivity out of it. But one of the areas that I'm really excited about is this idea of generative UI, which is not just autocompleting code in a text editor, but is the idea that you can use natural language to describe an interface and have the AI generate that interface.So, Vercel created this product called v0—you can check it out at v0.dev—where to me, it's really astonishing that you can get these incredibly high quality user interfaces, and basically all you have to do is input [laugh] a few English words. I have this personal experience of, I've been learning JavaScript and perfecting all my knowledge around it for, like, 20 or so years. I created Next.js.And Next.js itself powers a lot of these AI products. Like the front-end of ChatGPT is built on Next.js. And I used v0 to create… to basically recreate my blog. Like, I created rauchg.com, I deployed it on Vercel, but every pixel of that UI, I handcrafted.And as we were working on v0, I said okay, “I'm going to challenge myself to put myself back in the shoes of, like, I'm going to redesign this and I'm going to start over with just human language.” Not only did I arrive to the right look and feel of what I wanted to get, the code that it produced was better than I would have written by hand. Concretely, it was more accessible. So, there were areas of the UI where, like, some icons were rendered where I had not filled in those gaps. I just didn't know how to do that. The AI did. So, I really believe that AI will transform our lives as [laugh] programmers, at least I think, in many other areas in very profound ways.Corey: This is very similar to a project that I've been embarked on for the last few days where I described the app I wanted into Chat-Gippity and follow the instructions, and first, it round up point—sending me down a rabbit hole of the wrong Framework version that had been deprecated, and whatnot, and then I brought it all into VS Code where Jif-Ub Copilot, it kept switching back and forth between actively helpful, and ooh, the response matches publicly available code, so I'm not going to tell you the answer, despite the fact that feature has never been enabled on my account. So yeah, of course, it matches publicly available code. This is quite literally the React tutorial starter project. And it became incredibly frustrating, but it also would keep generating things in bursts, so my code is not at all legible or well organized or consistent for that matter. But it's still better than anything I'd be able to write myself. I'm looking forward to using v0 or something like it to see how that stacks up for some of my ridiculous generation ideas for these things.Guillermo: Yeah, you touched on a very important point is, the code has to work. The code has to be shippable. I think a lot of AI products have gotten by by giving you an approximation of the result, right? Like, they hallucinate sometimes, they get something wrong. It's still very helpful because sometimes it's sending you the right direction.But for us, the bar is that these things have to produce code that's useful, and that you can ship, and that you can iterate on. So, going back to that idea of iteration velocity, we call it v0 because we wanted it to be the first version. We still very much believe there is humans in the loop and folks will be iterating a lot on the initial draft that this thing is giving you, but it's so much better than starting with an empty code editor, [laugh] right? Like, and this applies, by the way to, like, not just new projects, but I always talk about, like, our customers have a few really important landing pages, key pages, maybe it's the product detail page in e-commerce, maybe it's your homepage and, like, your key product pages for a marketing website. Maybe it's where—and the checkout, for example, extremely important.But then there's a lot of incremental UIs that you have to add every single day. The banner for [laugh] accepting cookies or not, the consent management dialog. There's a lot of things that the worst case scenario is that you offload them again to some third-party script, to some iframe of sorts because you really don't have the bandwidth, time, or resources to build it yourself. And then you sacrifice speed, you sacrifice brand fidelity. And again because we're the front-end cloud, we're obsessed with your ability to ship UI that's native to your product, that is a streamline, that works really well. So, I think AI is going to have a significant effect there where a lot of things where you were sending someone to some other website because you just didn't have the bandwidth to create that UI, you can now own the experience end to end.Corey: That is no small thing. A last question I have, before we wind up calling this episode is, there was a period of time—I don't know if we're still in it or not—where it felt like every time I got up to get a cup of coffee and came back, there would be three JavaScript frameworks that launched during that interim. So, Next.js was at 1.1 of those when someone got up to get a cup of coffee. But that's shown a staying power that is, frankly, remarkable. Why? I don't know enough about the ecosystem to have an opinion on that, but I noticed when things stand out, and Next does.Guillermo: Yeah, I think it's a number of factors. Number one, we, as an industry I think, we coalesced, and we found the right engine to build our car. And that engine became React. Most folks building UI today are choosing React or a similar engine, but React has really become the gold standard for a lot of engineers. Now, what ended up happening next is that people realized I want a car. I want the full product. I need to drive. I don't want to assemble this freaking car every single time I have a new project.And Next.js filled a very important gap in the world where what you were looking for was not a library; what you were looking for is a framework that has opinions, but those opinions are very in line with how the web is supposed to work. We took a page from, basically, the beginnings of the web. We make a lot of jokes that in many ways, our inspiration was PHP, where server rendering is the default, where it's very expressive, it's very easy to reach for data. It just works for a lot of people. Again, that's the old [stack 00:30:03] in the olden days.And so, it obviously didn't quite work, but the inspiration was, can we make something that is a streamline for creating web interfaces at scale? At scale. And to your point, there's also a sense of, like, maybe it doesn't make sense anymore to build all this infrastructure from scratch every single time I started a project. So, Next filled in that gap. The other thing we did really well, I think, is that we gave people a universal model for how to use not just the server side, but also the client side strategically.So, I'll give you an example. When you go to ChatGPT, a lot of things on the screen are server rendered, but when you start doing interactions as a user, that requires for something like you'd say, “Hey Dali, generate an image.” That stuff requires a lot of optimistic UI. It requires features that are more like what a mobile native application can do. So, we can give folks the best of both worlds: the speed, interactivity, and fluidity of a native app, but we had those, sort of, fundamentals of how a website should work that even Perl and PHP had gotten right, once upon a time. So, I think we found that right blend of utility and flexibility, and folks love it, and I think, yeah, we're excited to continue to help steward this project as a standard for building on the web.Corey: I really want to thank you for taking the time to talk about a lot of the genesis of this stuff and how you view it, which I think gives us a pretty decent idea of how you're going to approach the evolution of what you've built. If people want to learn more, where's the best place for them to find you?Guillermo: So, head to vercel.com to learn about our platform. You can check out v0.dev, which we'll be opening broadly to the public soon, if you want to get started with this idea of generative UI. And myself, I'm always tweeting on X, twitter.com or x.com/rauchg to find me.Corey: One of these days we'll be able to kick that habit, I hope [laugh].Guillermo: [laugh]. Yeah.Corey: Thank you so much for being so generous with your time. I appreciate it.Guillermo: Thank you.Corey: Guillermo Rauch, founder and CEO of Vercel, and creator of Next.js. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, along with an angry, insulting comment that will be almost impossible for you to submit because that podcast platform did not pay attention to user experience.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business, and we get to the point. Visit duckbillgroup.com to get started. Vercel: https://vercel.com/ v0.dev: https://v0.dev Personal website: https://rauchg.com Personal twitter: https://twitter.com/rauchg TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. I don't talk a lot about front-end on this show, primarily because I am very bad at front-end, and in long-standing tech tradition, if I'm not good at something, apparently I'm legally obligated to be dismissive of it and not give it any attention. Strangely enough, I spent the last week beating on some front-end projects, and now I'm not just dismissive, I'm angry about it. Here to basically suffer the outpouring of frustration and confusion is Guillermo Rauch, founder and CEO of Vercel, but also the creator of Next.js. Guillermo, thank you for joining me.Guillermo: Great to be here. Thanks for setting me up with that awesome intro.Corey: It's true, if I were talking to someone who looked at what I've done, and for some Godforsaken reason, they wanted to follow in my footsteps, well, that path has been closed, so learning a bunch of Perl early on and translating it all to bad bash scripts and the rest, and then maybe picking up Python isn't really the way that I would advise someone getting started today. The de facto lingua franca of the internet is JavaScript, whether we like it or not, and I would strongly suggest that be someone's first language despite the fact that I'm bad at it, I don't understand it, and therefore it makes me angry.Guillermo: Yeah, it's so funny because it sounds like my story. And my personal journey was, when I was a kid, I had a—I knew I wanted to hack around with computers, and reverse engineer them, and improve them, and just create my own things, and I had these options for what programming language I could go with. And I tried it all: PHP, Perl, [Mod PHP 00:02:12], [Mod Perl 00:02:13], Apache, LAMP, cgi-bin folders, all the whole nine yards. And regardless of what back-end technology I used, I encountered this striking fact, which was… the thing that can make your product really stand out in a web browser is typically involving JavaScript in some fashion. So, when Google came out with suggestions as you would type in a search box, my young kid Argentinian brain blew up. I was like, “Holy crap, they can suggest, they can read my mind, and they can render suggestions without a full page refresh? What is that magic?”And then more products like that came out. Google Docs, Gmail Chat, Facebook's real time newsfeed, all the great things on the internet seemed to have this common point of, there's this layer of interactivity, real-time data, customization, personalization, and it seems uniquely enabled by the front-end. So, I just went all in. I taught myself how to code, I taught myself—I became a front-end engineering expert, I joined some of the early projects that shaped the ecosystem. Like, there was this library called MooTools, and a lot of folks might not have heard that name. It's in the annals of JavaScript history.And later on, you know, what I realized is, what if front-end can actually be the starting point of how you develop the best applications, right, rather than this thing that people, like, reluctantly frown upon, like yourself. I mention that as an opportunity rather than a dis because when you create a great front-end experience, now the data has proven you run a better business, you run a more dynamic business, probably are running an AI-powered business, like, all of the AI products on the planet today are using this technology to stream text in front of your eyes in real time and do all this awesome things. So yeah, I became obsessed with front-end, and I founded this company Vercel, which is a front-end cloud. So, you come here to basically build the best products. Now, you don't have to build the back-end, so you can use back-ends that are off the shelf, you can connect to your existing back-ends, and we piggyback on the world's best infrastructure to make this possible, but we offer developers a very streamlined path to create these awesome products on the internet.Corey: I have to say that I have been impressed when I've used Vercel for a number of projects. And what impresses me is less the infrastructure powering it, less the look at how performance it is and all the stuff that most people talk about, but as mentioned, I am not good at front-end or frankly programming at all. And so, many products in this space fall into the very pernicious trap of, “Oh, well, everyone who's using this is at least this tall on the board of how smart you are to get on the amusement park ride.” So, I feel that when I'm coming at this from a—someone who is not a stranger to computers but is definitely new to this entire ecosystem, everything just made sense in a way that remarkably few products can pull off. I don't know if you would call that user experience, developer experience, or what the terminology you bias for there is, but it is a transformational difference.Guillermo: Thank you. I think it's a combination of things. So, developer experience has definitely always been a focus for me. I was that weird person that obsessed about the CLI parameters of the tool, and the output of the tool, and just like how it feels for the engineer. I did combine that with—and I think this is where Vercel really stands out—I did combine that with a world-class infrastructure bit because what I realized after creating lots and lots of very popular open-source projects—like, one is called Socket.io, and other one called Mongoose—DX, or developer experience, in the absence of an enticing carrot for the business doesn't work. Maybe it has some short term adoption, maybe it has raving fans on Twitter or X [laugh], but at the end of the day, you have to deliver something that's tangible to the end-user and to the business.[unintelligible 00:06:33] Vercel focusing on the front-end has found a magical combination there of I can make the developer lives easier. Being a developer myself, I just tremendously empathize with that, but it can also make more profit for the business. When they make your website faster and render more dynamic data that serve as better recommendations for a product on e-commerce or in a marketing channel, I can help you roll out more experiments, then they make your business better, and I think that's one of the magical combinations there.The other thing, frankly, is that we'd started doing fewer things. So, when you come to Vercel, you typically come with a framework that you've already chosen. It could be Next.js, it could be View, [unintelligible 00:07:18], there's 35-plus frameworks. But we basically told the market, you have to use one of these developer tools in order to guide your development.And what companies were doing before—I mean, this almost seems obvious in retrospect, that we would optimize for her certain patterns and certain tools—but what the market was doing before was rolling out your own frameworks. Like, every company was, basically—React, for example, is a very popular way of building interfaces, and our framework actually is built on top of React. But when I would go to and talk to all these principal engineers that all these companies, they were saying, oh yeah, “We're creating our own framework. We're creating our own tools.” And I think that to me now feels almost like a zero interest rate phenomenon. Like, what business do you have in creating frameworks, tools, and bespoke infra when you're really in the business of creating delightful experiences for your customers?Corey: What I think is lost on a lot of folks is that if you are trying to learn something new and use a tool, and the developer experience is bad, the takeaway—at least for me and a lot of people that I talk to is not, “Oh, this tool has terrible ergonomics. That's it's problem.” Instead, the takeaway is very much, “Oh, I'm dumb because I don't understand this thing.” And I know intellectually that I am not usually the dumbest person in the world when it comes to a particular tool or technology, but I keep forgetting that on a visceral level. It's, “I just wish I was smart enough to understand that.”No, I don't. I wish it was presented in a way that was more understandable and the documentation was better. When you're just starting out and building something in your spare time, the infrastructure cost is basically nothing, but your time is the expensive part in it. So, if you have to spend three hours to track down something just because it wasn't clearly explained, the burden of adopting that tool is challenging. I would argue that one of the reasons that AWS sees some of the success that it does is not necessarily because it's great so much as because everyone knows how it breaks. That's important.I'm not saying their infrastructure isn't world-class—please, don't come at me in the comments on this one—but I am saying that we know where its sharp edges are, and that means that we're more comfortable building with it. But the idea of learning a brand-new cloud with different sharp edges in other areas. That's terrifying. I'd rather stick with the devil I know.Guillermo: Exactly. I just think that you're not going to be able to make a difference for customers in 2023 by creating another bespoke cloud that is general purpose, it doesn't really optimize around anything, and you have to learn all the sharp edges from scratch. I think we saw that with the rise of cloud-native companies like Stripe and Twilio where they were going after these amazingly huge markets like financial infrastructure or communications infrastructure, but the angle was, “Here's this awesome developer experience.” And that's what we're doing with Vercel for the front-end and for building products, right? There has to be an opinionated developer experience that guides you to success.And I agree with you that there's really, these days in the developer communities, zero tolerance for sharp edges, and we've spent a lot of time in—even documentation, like, it used to be that your startup would make or break it by whether you had great documentation. I think in the age of frameworks, I would even dare say that documentation, of course, is extremely important, but if I can have the tool itself guide you to success, at that point, you're not even reading documentation. We're now seeing this with AI and, like, generative AI for code. At Vercel, we're investing in generative AI for user interfaces. Do you actually need to read documentation at that point? So, I think we're optimizing for the absolute minimal amount of friction required to be successful with these platforms.Corey: I think that there's a truth in that of meeting customers in where they are. Your documentation can be spectacular, but people don't generally read the encyclopedia for fun either. And the idea of that is that—at least ideally—I should not need to go diving into the documentation, and so many tools get this wrong, where, “Oh, I want to set up a new project,” and it bombards you with 50 questions, and each one of these feels pretty… momentous. Like, what one-way door am I passing through that I don't realize the consequences of until I'm 12 hours into this thing, and then have to backtrack significantly. I like, personally, things that bias for having a golden path, but also make it easy to both deviate from it, as well as get back onto it. Because there's more than one way to do it is sort of the old Perl motto. That is true in spades in anything approaching the JavaScript universe.Guillermo: Yeah. I have a lot of thoughts on that. On the first point, I completely agree that the golden path of the product cannot be documentation-mediated. One of the things that I've become obsessive about—and this is an advice that I share with a lot of other startup founders is, when it comes to your landing page, the primary call to action has to be this golden path to success, like, 2, 3, 4 clicks later, I have to have something tangible. That was our inspiration.And when we made it the primary call to action for Vercel is deploy now. Start now. Get it out now. Ship it now. And the way that you test out the platform is by deploying a template. What do we do is we create a Git repo for you, it sets up the entire CI/CD pipeline, and then at that point, you already have something working, something in the cloud, you spent zero time reading documentation, and you can start iterating.And even though that might not be the final thing you do in Vercel, I always hear the stories of CTOs that are now deploying Vercel at really large scale, and they always tell me, “I started with your hobby tier, I started with free tier, I deployed a template, I hacked on a product during the weekend.” Now, a lot of our AI examples are very popular in this crowd. And yeah, there's a golden path that requires zero documentation. Now, you also mentioned that, what about complexity? This is an enterprise-grade platform. What about escape hatches? What about flexibility? And that's where our platform also shines because we have the entire power of a Turing-complete language, which is JavaScript and TypeScript, to customize every aspect of the platform.And you have a framework that actually answered a lot of the problems that came with serverless solutions in the past, which is that you couldn't run any of that on your local machine. The beauty of Vercel and Next.js is we kind of pioneered this concept that we called ‘Framework Defined Infrastructure.' You start with the framework, the framework has this awesome property that you can install on your computer, it has a dev command—like, it literally runs on your computer—but then when you push it to the cloud, it now defines the infrastructure. It creates all of the resources that are highly optimal.This creates—basically converts what was a single node system on your computer to a globally distributed system, which is a very complex and difficult engineering challenge, but Vercel completely automates it away. And now for folks that are looking for, like, more advanced solutions, they can now start poking into the outputs of that compilation process and say, “Okay, I can now have an influence or I can reconfigure some aspects of this pipeline.” And of course, if you don't think about those escape hatches, then the product just ends up being limiting and frustrating, so we had to think really hard about meeting both ends of the spectrum.Corey: In my own experimentations early on with Chat-Gippity—which is how I insist on pronouncing ChatGPT—a lot of what I found was that it was a lot more productive for me to, instead of asking just the question and getting the answer was, write a Python script to—Guillermo: Yes.Corey: Query this API to get me that answer. Because often it would be wrong. Sometimes very convincingly wrong, and I can at least examine it in various ways and make changes to it and iterate forward, whereas when everything is just a black box, that gets very hard to do. The idea of building something that can be iterated on is key.Guillermo: I love that. The way that Vercel actually first introduced itself to the world was this idea of immutable deployments and immutable infrastructure. And immutable sounds like a horrible word because I want to mutate things, but it was inspired by this idea of functional programming where, like, each iteration to the state, each data change, can be tracked. So, every time you deploy in Vercel, you get this unique URL that represents a point-in-time infrastructure deployment. You can go back in time, you can revert, you can use this as a way of collaborating with other engineers in your team, so you can send these hyperlinks around to your front-end projects.And it gives you a lot of confidence. Now, you can iterate knowing that before things go out, there's a lot of scrutiny, there's a lot of QA, there's a lot of testing processes that you can kick off against this serverless infrastructure that was created for each deployment. The conclusion for us so far has been that our role in the world is to increase iteration velocity. So, iteration speed is the faster horse of the cloud, right? Like, instead of getting a car, you get a faster horse.When you say, “Okay, I made the build pipeline 10% faster,” or, “I brought the TLS termination 10% closer to the visitor, and, like, I have more [pops 00:17:10],” things like that. That to me, is the speed. You can do those things, and they're awesome, but if you don't have a direction—which is velocity—then you don't know what you're building next. You don't know if your customers are happy. You don't know if you're delivering value. So, we built an entire platform that optimizes around, what should you ship next? What is the friction involved in getting your next iteration out? Is launching an experiment on your homepage, for example, is that a costly endeavor? Does it take you weeks? Does it take you months?One of the initial inspirations for just starting Vercel and making deployments really easy was, how difficult is it for the average company to change in their footer of their website is this copyright 2022? And you have—it's a new year. You have to bump it to copyright 2023. How long do you think it takes that engineer to, A, run the stack locally, so they can actually see the change; deploy it, but deploy to what we call the preview environment, so they can grab that URL and send [it to 00:18:15], Corey, and say, “Corey, does it look good? I updated [laugh] I updated the year in the footer.”And then you tell me, “Looks good, let's ship it to production.” Or you tell me, “No, no, no, it's risky. Let's divide it into two cohorts: 50% of traffic gets 2022, 50% of traffic gets 2023.” Obviously, this is a joke, but consider the implications of how difficult it is and the average organization to actually do this thing.[midroll 00:18:41]Corey: Oh, I find things like that all the time, especially on microservices that I built to handle some of my internal workflows here, and I haven't touched in two or three years. And okay, now it's time for me to update them to reflect some minor change. And first, I wind up in the screaming node warnings and I have to update things so that they actually work in a reasonable way. And, on some level, making a one-line change can take half a day. Now, in the real world, when people are working on these apps day-in and day-out, it gets a lot easier to roll those changes in over time, but coming back to something unmaintained, that becomes a project the longer you let it sit.Part of me wishes that there were easier ways around it, but there are trade-offs in almost any decision you make. If you're building something from the beginning of, well, I want to be able to automatically update the copyright year, you can even borderline make that something that automatically happens based upon the global time, whereas when you're trying to retrofit it afterwards, yeah, it becomes a project.Guillermo: Yeah, and now think, that's just a simple example of changing a string. That might be difficult for a product engineering in any organization. Or it may be slow, or it may be not as streamlined, or maybe it works really well for the first project that that company created. What about every incremental project thereafter?So, now I said—let's stop talking about a string, right? Let's think about you're about an e-commerce website where what we hear from our customers on average, like, 10% of revenue flows through the homepage. Now, I have to change a primary component that renders on the hero of the page, and I have to collaborate with every department in the organization. I have to collaborate with the design team, I have to collaborate with marketing, I have to collaborate with the business owners to track the analytics appropriately. So, what is the cost of every incremental experiment that you want to put in production?The other thing that's particularly interesting about front-end as it relates to cloud infrastructure is, scaling up front-end is a very difficult thing. What ends up happening is most front-ends are actually static websites. They're cached at the edge—or they're literally statically generated—and then they push all of the dynamism to the client side. So, you end up with this spaghetti of script tags on the client, you end up accumulating a lot of tech debt in the [shipping 00:20:56] huge bundles of JavaScript to the client to try to recover some dynamism, to try and run these experiments. So, everyone is in this, kind of, mess of the yes, maybe we can experiment, but we kind of offloaded the rendering work to the client. That in turn makes me—basically, I'm making the website slower for the visitor. I'm making them do the rendering work.And I'm trying to sell them something. I'm trying to speed up some processes. It's my responsibility to make it fast. So, what we ended up finding out is that yes, the cloud moved this forward a lot in terms of having these awesome building blocks, these awesome infrastructure primitives, but both in the developer experience, just changing something about your web product and also the end-user experiences, that web product renders really fast, those things really didn't happen with this first chapter of the cloud. And I think we're entering a new generation of higher-level clouds like Vercel that are optimizing for these things.Corey: I think that there's a historical focus on things that have not happened before. And that was painful and terrible, so we're not going to be focusing on what's happening in the future, we're going to build a process or a framework or something that winds up preventing that thing that hurt us from hurting us again. Now, that's great in moderation, but at some point—we see this at large companies from time-to-time—where you have so much process that is ossified scar tissue, basically, that it becomes almost impossible to get things done. Because oh, I want to make that—for example, that one-line change to a copyright date, well, here's the 5000 ways deploys have screwed us before, so we need to have three humans sign off on the one-line change, and a bunch of other stuff before it ever sees the light of day. Now, I'm exaggerating slightly, just as you are, but that feels like it acts as a serious brake on innovation.On the exact opposite side, where we see massive acceleration has been around the world of generative AI. Yes, it is massively hyped in a bunch of ways. I don't think it is going to be a defined way that changes the nature of humanity the way that some of these people are going after, but it's also clearly more than a parlor trick.Guillermo: I'm kind of in that camp. So, like you, I've been writing code for many years. I'm pretty astonished by the AI's ability to enhance my output. And of course, now I'm not writing code full time, so there is a sense of, okay because I don't have time, because I'm doing a million things, any minute I have seems like AI has just made it so much more worthwhile and I can squeeze so much more productivity out of it. But one of the areas that I'm really excited about is this idea of generative UI, which is not just autocompleting code in a text editor, but is the idea that you can use natural language to describe an interface and have the AI generate that interface.So, Vercel created this product called v0—you can check it out at v0.dev—where to me, it's really astonishing that you can get these incredibly high quality user interfaces, and basically all you have to do is input [laugh] a few English words. I have this personal experience of, I've been learning JavaScript and perfecting all my knowledge around it for, like, 20 or so years. I created Next.js.And Next.js itself powers a lot of these AI products. Like the front-end of ChatGPT is built on Next.js. And I used v0 to create… to basically recreate my blog. Like, I created rauchg.com, I deployed it on Vercel, but every pixel of that UI, I handcrafted.And as we were working on v0, I said okay, “I'm going to challenge myself to put myself back in the shoes of, like, I'm going to redesign this and I'm going to start over with just human language.” Not only did I arrive to the right look and feel of what I wanted to get, the code that it produced was better than I would have written by hand. Concretely, it was more accessible. So, there were areas of the UI where, like, some icons were rendered where I had not filled in those gaps. I just didn't know how to do that. The AI did. So, I really believe that AI will transform our lives as [laugh] programmers, at least I think, in many other areas in very profound ways.Corey: This is very similar to a project that I've been embarked on for the last few days where I described the app I wanted into Chat-Gippity and follow the instructions, and first, it round up point—sending me down a rabbit hole of the wrong Framework version that had been deprecated, and whatnot, and then I brought it all into VS Code where Jif-Ub Copilot, it kept switching back and forth between actively helpful, and ooh, the response matches publicly available code, so I'm not going to tell you the answer, despite the fact that feature has never been enabled on my account. So yeah, of course, it matches publicly available code. This is quite literally the React tutorial starter project. And it became incredibly frustrating, but it also would keep generating things in bursts, so my code is not at all legible or well organized or consistent for that matter. But it's still better than anything I'd be able to write myself. I'm looking forward to using v0 or something like it to see how that stacks up for some of my ridiculous generation ideas for these things.Guillermo: Yeah, you touched on a very important point is, the code has to work. The code has to be shippable. I think a lot of AI products have gotten by by giving you an approximation of the result, right? Like, they hallucinate sometimes, they get something wrong. It's still very helpful because sometimes it's sending you the right direction.But for us, the bar is that these things have to produce code that's useful, and that you can ship, and that you can iterate on. So, going back to that idea of iteration velocity, we call it v0 because we wanted it to be the first version. We still very much believe there is humans in the loop and folks will be iterating a lot on the initial draft that this thing is giving you, but it's so much better than starting with an empty code editor, [laugh] right? Like, and this applies, by the way to, like, not just new projects, but I always talk about, like, our customers have a few really important landing pages, key pages, maybe it's the product detail page in e-commerce, maybe it's your homepage and, like, your key product pages for a marketing website. Maybe it's where—and the checkout, for example, extremely important.But then there's a lot of incremental UIs that you have to add every single day. The banner for [laugh] accepting cookies or not, the consent management dialog. There's a lot of things that the worst case scenario is that you offload them again to some third-party script, to some iframe of sorts because you really don't have the bandwidth, time, or resources to build it yourself. And then you sacrifice speed, you sacrifice brand fidelity. And again because we're the front-end cloud, we're obsessed with your ability to ship UI that's native to your product, that is a streamline, that works really well. So, I think AI is going to have a significant effect there where a lot of things where you were sending someone to some other website because you just didn't have the bandwidth to create that UI, you can now own the experience end to end.Corey: That is no small thing. A last question I have, before we wind up calling this episode is, there was a period of time—I don't know if we're still in it or not—where it felt like every time I got up to get a cup of coffee and came back, there would be three JavaScript frameworks that launched during that interim. So, Next.js was at 1.1 of those when someone got up to get a cup of coffee. But that's shown a staying power that is, frankly, remarkable. Why? I don't know enough about the ecosystem to have an opinion on that, but I noticed when things stand out, and Next does.Guillermo: Yeah, I think it's a number of factors. Number one, we, as an industry I think, we coalesced, and we found the right engine to build our car. And that engine became React. Most folks building UI today are choosing React or a similar engine, but React has really become the gold standard for a lot of engineers. Now, what ended up happening next is that people realized I want a car. I want the full product. I need to drive. I don't want to assemble this freaking car every single time I have a new project.And Next.js filled a very important gap in the world where what you were looking for was not a library; what you were looking for is a framework that has opinions, but those opinions are very in line with how the web is supposed to work. We took a page from, basically, the beginnings of the web. We make a lot of jokes that in many ways, our inspiration was PHP, where server rendering is the default, where it's very expressive, it's very easy to reach for data. It just works for a lot of people. Again, that's the old [stack 00:30:03] in the olden days.And so, it obviously didn't quite work, but the inspiration was, can we make something that is a streamline for creating web interfaces at scale? At scale. And to your point, there's also a sense of, like, maybe it doesn't make sense anymore to build all this infrastructure from scratch every single time I started a project. So, Next filled in that gap. The other thing we did really well, I think, is that we gave people a universal model for how to use not just the server side, but also the client side strategically.So, I'll give you an example. When you go to ChatGPT, a lot of things on the screen are server rendered, but when you start doing interactions as a user, that requires for something like you'd say, “Hey Dali, generate an image.” That stuff requires a lot of optimistic UI. It requires features that are more like what a mobile native application can do. So, we can give folks the best of both worlds: the speed, interactivity, and fluidity of a native app, but we had those, sort of, fundamentals of how a website should work that even Perl and PHP had gotten right, once upon a time. So, I think we found that right blend of utility and flexibility, and folks love it, and I think, yeah, we're excited to continue to help steward this project as a standard for building on the web.Corey: I really want to thank you for taking the time to

We are running an end of year survey for our listeners. Let us know any feedback you have for us, what episodes resonated with you the most, and guest requests for 2024! RAG has emerged as one of the key pieces of the AI Engineer stack. Jerry from LlamaIndex called it a “hack”, Bryan from Hex compared it to “a recommendation system from LLMs”, and even LangChain started with it. RAG is crucial in any AI coding workflow. We talked about context quality for code in our Phind episode. Today's guests, Beyang Liu and Steve Yegge from SourceGraph, have been focused on code indexing and retrieval for over 15 years. We locked them in our new studio to record a 1.5 hours masterclass on the history of code search, retrieval interfaces for code, and how they get SOTA 30% completion acceptance rate in their Cody product by being better at the “bin packing problem” of LLM context generation. Google Grok → SourceGraph → CodyWhile at Google in 2008, Steve built Grok, which lives on today as Google Kythe. It allowed engineers to do code parsing and searching across different codebases and programming languages. (You might remember this blog post from Steve's time at Google) Beyang was an intern at Google at the same time, and Grok became the inspiration to start SourceGraph in 2013. The two didn't know eachother personally until Beyang brought Steve out of retirement 9 years later to join him as VP Engineering. Fast forward 10 years, SourceGraph has become to best code search tool out there and raised $223M along the way. Nine months ago, they open sourced SourceGraph Cody, their AI coding assistant. All their code indexing and search infrastructure allows them to get SOTA results by having better RAG than competitors:* Code completions as you type that achieve an industry-best Completion Acceptance Rate (CAR) as high as 30% using a context-enhanced open-source LLM (StarCoder)* Context-aware chat that provides the option of using GPT-4 Turbo, Claude 2, GPT-3.5 Turbo, Mistral 7x8B, or Claude Instant, with more model integrations planned* Doc and unit test generation, along with AI quick fixes for common coding errors* AI-enhanced natural language code search, powered by a hybrid dense/sparse vector search engine There are a few pieces of infrastructure that helped Cody achieve these results:Dense-sparse vector retrieval system For many people, RAG = vector similarity search, but there's a lot more that you can do to get the best possible results. From their release:"Sparse vector search" is a fancy name for keyword search that potentially incorporates LLMs for things like ranking and term expansion (e.g., "k8s" expands to "Kubernetes container orchestration", possibly weighted as in SPLADE): * Dense vector retrieval makes use of embeddings, the internal representation that LLMs use to represent text. Dense vector retrieval provides recall over a broader set of results that may have no exact keyword matches but are still semantically similar. * Sparse vector retrieval is very fast, human-understandable, and yields high recall of results that closely match the user query. * We've found the approaches to be complementary.There's a very good blog post by Pinecone on SPLADE for sparse vector search if you're interested in diving in. If you're building RAG applications in areas that have a lot of industry-specific nomenclature, acronyms, etc, this is a good approach to getting better results.SCIPIn 2016, Microsoft announced the Language Server Protocol (LSP) and the Language Server Index Format (LSIF). This protocol makes it easy for IDEs to get all the context they need from a codebase to get things like file search, references, “go to definition”, etc. SourceGraph developed SCIP, “a better code indexing format than LSIF”:* Simpler and More Efficient Format: SCIP utilizes Protobuf instead of JSON, which is used by LSIF. Protobuf is more space-efficient, simpler, and more suitable for systems programming. * Better Performance and Smaller Index Sizes: SCIP indexers, such as scip-clang, show enhanced performance and reduced index file sizes compared to LSIF indexers (10%-20% smaller)* Easier to Develop and Debug: SCIP's design, centered around human-readable string IDs for symbols, makes it faster and more straightforward to develop new language indexers. Having more efficient indexing is key to more performant RAG on code. Show Notes* Sourcegraph* Cody* Copilot vs Cody* Steve's Stanford seminar on Grok* Steve's blog* Grab* Fireworks* Peter Norvig* Noam Chomsky* Code search* Kelly Norton* Zoekt* v0.devSee also our past episodes on Cursor, Phind, Codeium and Codium as well as the GitHub Copilot keynote at AI Engineer Summit.Timestamps* [00:00:00] Intros & Backgrounds* [00:05:20] How Steve's work on Grok inspired SourceGraph for Beyang* [00:08:10] What's Cody?* [00:11:22] Comparison of coding assistants and the capabilities of Cody* [00:16:00] The importance of context (RAG) in AI coding tools* [00:21:33] The debate between Chomsky and Norvig approaches in AI* [00:30:06] Normsky: the Norvig + Chomsky models collision* [00:36:00] The death of the DSL?* [00:40:00] LSP, Skip, Kythe, BFG, and all that fun stuff* [00:53:00] The SourceGraph internal stack* [00:58:46] Building on open source models* [01:02:00] SourceGraph for engineering managers?* [01:12:00] Lightning RoundTranscriptAlessio: Hey everyone, welcome to the Latent Space podcast. This is Alessio, partner and CTO-in-Residence at Decibel Partners, and I'm joined by my co-host Swyx, founder of Smol AI. [00:00:16]Swyx: Hey, and today we're christening our new podcast studio in the Newton, and we have Beyang and Steve from Sourcegraph. Welcome. [00:00:25]Beyang: Hey, thanks for having us. [00:00:26]Swyx: So this has been a long time coming. I'm very excited to have you. We also are just celebrating the one year anniversary of ChatGPT yesterday, but also we'll be talking about the GA of Cody later on today. We'll just do a quick intros of both of you. Obviously, people can research you and check the show notes for more. Beyang, you worked in computer vision at Stanford and then you worked at Palantir. I did, yeah. You also interned at Google. [00:00:48]Beyang: I did back in the day where I get to use Steve's system, DevTool. [00:00:53]Swyx: Right. What was it called? [00:00:55]Beyang: It was called Grok. Well, the end user thing was Google Code Search. That's what everyone called it, or just like CS. But the brains of it were really the kind of like Trigram index and then Grok, which provided the reference graph. [00:01:07]Steve: Today it's called Kythe, the open source Google one. It's sort of like Grok v3. [00:01:11]Swyx: On your podcast, which you've had me on, you've interviewed a bunch of other code search developers, including the current developer of Kythe, right? [00:01:19]Beyang: No, we didn't have any Kythe people on, although we would love to if they're up for it. We had Kelly Norton, who built a similar system at Etsy, it's an open source project called Hound. We also had Han-Wen Nienhuys, who created Zoekt, which is, I think, heavily inspired by the Trigram index that powered Google's original code search and that we also now use at Sourcegraph. Yeah. [00:01:45]Swyx: So you teamed up with Quinn over 10 years ago to start Sourcegraph and you were indexing all code on the internet. And now you're in a perfect spot to create a code intelligence startup. Yeah, yeah. [00:01:56]Beyang: I guess the backstory was, I used Google Code Search while I was an intern. And then after I left that internship and worked elsewhere, it was the single dev tool that I missed the most. I felt like my job was just a lot more tedious and much more of a hassle without it. And so when Quinn and I started working together at Palantir, he had also used various code search engines in open source over the years. And it was just a pain point that we both felt, both working on code at Palantir and also working within Palantir's clients, which were a lot of Fortune 500 companies, large financial institutions, folks like that. And if anything, the pains they felt in dealing with large complex code bases made our pain points feel small by comparison. So that was really the impetus for starting Sourcegraph. [00:02:42]Swyx: Yeah, excellent. Steve, you famously worked at Amazon. And you've told many, many stories. I want every single listener of Latent Space to check out Steve's YouTube because he effectively had a podcast that you didn't tell anyone about or something. You just hit record and just went on a few rants. I'm always here for your Stevie rants. And then you moved to Google, where you also had some interesting thoughts on just the overall Google culture versus Amazon. You joined Grab as head of eng for a couple of years. I'm from Singapore, so I have actually personally used a lot of Grab's features. And it was very interesting to see you talk so highly of Grab's engineering and sort of overall prospects. [00:03:21]Steve: Because as a customer, it sucked? [00:03:22]Swyx: Yeah, no, it's just like, being from a smaller country, you never see anyone from our home country being on a global stage or talked about as a startup that people admire or look up to, like on the league that you, with all your legendary experience, would consider equivalent. Yeah. [00:03:41]Steve: Yeah, no, absolutely. They actually, they didn't even know that they were as good as they were, in a sense. They started hiring a bunch of people from Silicon Valley to come in and sort of like fix it. And we came in and we were like, Oh, we could have been a little better operational excellence and stuff. But by and large, they're really sharp. The only thing about Grab is that they get criticized a lot for being too westernized. Oh, by who? By Singaporeans who don't want to work there. [00:04:06]Swyx: Okay. I guess I'm biased because I'm here, but I don't see that as a problem. If anything, they've had their success because they were more westernized than the Sanders Singaporean tech company. [00:04:15]Steve: I mean, they had their success because they are laser focused. They copy to Amazon. I mean, they're executing really, really, really well for a giant. I was on a slack with 2,500 engineers. It was like this giant waterfall that you could dip your toe into. You'd never catch up. Actually, the AI summarizers would have been really helpful there. But yeah, no, I think Grab is successful because they're just out there with their sleeves rolled up, just making it happen. [00:04:43]Swyx: And for those who don't know, it's not just like Uber of Southeast Asia, it's also a super app. PayPal Plus. [00:04:48]Steve: Yeah. [00:04:49]Swyx: In the way that super apps don't exist in the West. It's one of the enduring mysteries of B2C that super apps work in the East and don't work in the West. We just don't understand it. [00:04:57]Beyang: Yeah. [00:04:58]Steve: It's just kind of curious. They didn't work in India either. And it was primarily because of bandwidth reasons and smaller phones. [00:05:03]Swyx: That should change now. It should. [00:05:05]Steve: And maybe we'll see a super app here. [00:05:08]Swyx: You retired-ish? I did. You retired-ish on your own video game? Mm-hmm. Any fun stories about that? And that's also where you discovered some need for code search, right? Mm-hmm. [00:05:16]Steve: Sure. A need for a lot of stuff. Better programming languages, better databases. Better everything. I mean, I started in like 95, right? Where there was kind of nothing. Yeah. Yeah. [00:05:24]Beyang: I just want to say, I remember when you first went to Grab because you wrote that blog post talking about why you were excited about it, about like the expanding Asian market. And our reaction was like, oh, man, how did we miss stealing it with you? [00:05:36]Swyx: Hiring you. [00:05:37]Beyang: Yeah. [00:05:38]Steve: I was like, miss that. [00:05:39]Swyx: Tell that story. So how did this happen? Right? So you were inspired by Grok. [00:05:44]Beyang: I guess the backstory from my point of view is I had used code search and Grok while at Google, but I didn't actually know that it was connected to you, Steve. I knew you from your blog posts, which were always excellent, kind of like inside, very thoughtful takes from an engineer's perspective on some of the challenges facing tech companies and tech culture and that sort of thing. But my first introduction to you within the context of code intelligence, code understanding was I watched a talk that you gave, I think at Stanford, about Grok when you're first building it. And that was very eye opening. I was like, oh, like that guy, like the guy who, you know, writes the extremely thoughtful ranty like blog posts also built that system. And so that's how I knew, you know, you were involved in that. And then, you know, we always wanted to hire you, but never knew quite how to approach you or, you know, get that conversation started. [00:06:34]Steve: Well, we got introduced by Max, right? Yeah. It was temporal. Yeah. Yeah. I mean, it was a no brainer. They called me up and I had noticed when Sourcegraph had come out. Of course, when they first came out, I had this dagger of jealousy stabbed through me piercingly, which I remember because I am not a jealous person by any means, ever. But boy, I was like, but I was kind of busy, right? And just one thing led to another. I got sucked back into the ads vortex and whatever. So thank God Sourcegraph actually kind of rescued me. [00:07:05]Swyx: Here's a chance to build DevTools. Yeah. [00:07:08]Steve: That's the best. DevTools are the best. [00:07:10]Swyx: Cool. Well, so that's the overall intro. I guess we can get into Cody. Is there anything else that like people should know about you before we get started? [00:07:18]Steve: I mean, everybody knows I'm a musician. I can juggle five balls. [00:07:24]Swyx: Five is good. Five is good. I've only ever managed three. [00:07:27]Steve: Five is hard. Yeah. And six, a little bit. [00:07:30]Swyx: Wow. [00:07:31]Beyang: That's impressive. [00:07:32]Alessio: So yeah, to jump into Sourcegraph, this has been a company 10 years in the making. And as Sean said, now you're at the right place. Phase two. Now, exactly. You spent 10 years collecting all this code, indexing, making it easy to surface it. Yeah. [00:07:47]Swyx: And also learning how to work with enterprises and having them trust you with their code bases. Yeah. [00:07:52]Alessio: Because initially you were only doing on-prem, right? Like a lot of like VPC deployments. [00:07:55]Beyang: So in the very early days, we're cloud only. But the first major customers we landed were all on-prem, self-hosted. And that was, I think, related to the nature of the problem that we're solving, which becomes just like a critical, unignorable pain point once you're above like 100 devs or so. [00:08:11]Alessio: Yeah. And now Cody is going to be GA by the time this releases. So congrats to your future self for launching this in two weeks. Can you give a quick overview of just what Cody is? I think everybody understands that it's a AI coding agent, but a lot of companies say they have a AI coding agent. So yeah, what does Cody do? How do people interface with it? [00:08:32]Beyang: Yeah. So how is it different from the like several dozen other AI coding agents that exist in the market now? When we thought about building a coding assistant that would do things like code generation and question answering about your code base, I think we came at it from the perspective of, you know, we've spent the past decade building the world's best code understanding engine for human developers, right? So like it's kind of your guide as a human dev if you want to go and dive into a large complex code base. And so our intuition was that a lot of the context that we're providing to human developers would also be useful context for AI developers to consume. And so in terms of the feature set, Cody is very similar to a lot of other assistants. It does inline autocompletion. It does code base aware chat. It does specific commands that automate, you know, tasks that you might rather not want to do like generating unit tests or adding detailed documentation. But we think the core differentiator is really the quality of the context, which is hard to kind of describe succinctly. It's a bit like saying, you know, what's the difference between Google and Alta Vista? There's not like a quick checkbox list of features that you can rattle off, but it really just comes down to all the attention and detail that we've paid to making that context work well and be high quality and fast for human devs. We're now kind of plugging into the AI coding assistant as well. Yeah. [00:09:53]Steve: I mean, just to add my own perspective on to what Beyang just described, RAG is kind of like a consultant that the LLM has available, right, that knows about your code. RAG provides basically a bridge to a lookup system for the LLM, right? Whereas fine tuning would be more like on the job training for somebody. If the LLM is a person, you know, and you send them to a new job and you do on the job training, that's what fine tuning is like, right? So tuned to our specific task. You're always going to need that expert, even if you get the on the job training, because the expert knows your particular code base, your task, right? That expert has to know your code. And there's a chicken and egg problem because, right, you know, we're like, well, I'm going to ask the LLM about my code, but first I have to explain it, right? It's this chicken and egg problem. That's where RAG comes in. And we have the best consultants, right? The best assistant who knows your code. And so when you sit down with Cody, right, what Beyang said earlier about going to Google and using code search and then starting to feel like without it, his job was super tedious. Once you start using these, do you guys use coding assistants? [00:10:53]Swyx: Yeah, right. [00:10:54]Steve: I mean, like we're getting to the point very quickly, right? Where you feel like almost like you're programming without the internet, right? Or something, you know, it's like you're programming back in the nineties without the coding assistant. Yeah. Hopefully that helps for people who have like no idea about coding systems, what they are. [00:11:09]Swyx: Yeah. [00:11:10]Alessio: I mean, going back to using them, we had a lot of them on the podcast already. We had Cursor, we have Codium and Codium, very similar names. [00:11:18]Swyx: Yeah. Find, and then of course there's Copilot. [00:11:22]Alessio: You had a Copilot versus Cody blog post, and I think it really shows the context improvement. So you had two examples that stuck with me. One was, what does this application do? And the Copilot answer was like, oh, it uses JavaScript and NPM and this. And it's like, but that's not what it does. You know, that's what it's built with. Versus Cody was like, oh, these are like the major functions. And like, these are the functionalities and things like that. And then the other one was, how do I start this up? And Copilot just said NPM start, even though there was like no start command in the package JSON, but you know, most collapse, right? Most projects use NPM start. So maybe this does too. How do you think about open source models? Because Copilot has their own private thing. And I think you guys use Starcoder, if I remember right. Yeah, that's correct. [00:12:09]Beyang: I think Copilot uses some variant of Codex. They're kind of cagey about it. I don't think they've like officially announced what model they use. [00:12:16]Swyx: And I think they use a range of models based on what you're doing. Yeah. [00:12:19]Beyang: So everyone uses a range of model. Like no one uses the same model for like inline completion versus like chat because the latency requirements for. Oh, okay. Well, there's fill in the middle. There's also like what the model's trained on. So like we actually had completions powered by Claude Instant for a while. And but you had to kind of like prompt hack your way to get it to output just the code and not like, hey, you know, here's the code you asked for, like that sort of text. So like everyone uses a range of models. We've kind of designed Cody to be like especially model, not agnostic, but like pluggable. So one of our kind of design considerations was like as the ecosystem evolves, we want to be able to integrate the best in class models, whether they're proprietary or open source into Cody because the pace of innovation in the space is just so quick. And I think that's been to our advantage. Like today, Cody uses Starcoder for inline completions. And with the benefit of the context that we provide, we actually show comparable completion acceptance rate metrics. It's kind of like the standard metric that folks use to evaluate inline completion quality. It's like if I show you a completion, what's the chance that you actually accept the completion versus you reject it? And so we're at par with Copilot, which is at the head of that industry right now. And we've been able to do that with the Starcoder model, which is open source and the benefit of the context fetching stuff that we provide. And of course, a lot of like prompt engineering and other stuff along the way. [00:13:40]Alessio: And Steve, you wrote a post called cheating is all you need about what you're building. And one of the points you made is that everybody's fighting on the same axis, which is better UI and the IDE, maybe like a better chat response. But data modes are kind of the most important thing. And you guys have like a 10 year old mode with all the data you've been collecting. How do you kind of think about what other companies are doing wrong, right? Like, why is nobody doing this in terms of like really focusing on RAG? I feel like you see so many people. Oh, we just got a new model. It's like a bit human eval. And it's like, well, but maybe like that's not what we should really be doing, you know? Like, do you think most people underestimate the importance of like the actual RAG in code? [00:14:21]Steve: I think that people weren't doing it much. It wasn't. It's kind of at the edges of AI. It's not in the center. I know that when ChatGPT launched, so within the last year, I've heard a lot of rumblings from inside of Google, right? Because they're undergoing a huge transformation to try to, you know, of course, get into the new world. And I heard that they told, you know, a bunch of teams to go and train their own models or fine tune their own models, right? [00:14:43]Swyx: Both. [00:14:43]Steve: And, you know, it was a s**t show. Nobody knew how to do it. They launched two coding assistants. One was called Code D with an EY. And then there was, I don't know what happened in that one. And then there's Duet, right? Google loves to compete with themselves, right? They do this all the time. And they had a paper on Duet like from a year ago. And they were doing exactly what Copilot was doing, which was just pulling in the local context, right? But fundamentally, I thought of this because we were talking about the splitting of the [00:15:10]Swyx: models. [00:15:10]Steve: In the early days, it was the LLM did everything. And then we realized that for certain use cases, like completions, that a different, smaller, faster model would be better. And that fragmentation of models, actually, we expected to continue and proliferate, right? Because we are fundamentally, we're a recommender engine right now. Yeah, we're recommending code to the LLM. We're saying, may I interest you in this code right here so that you can answer my question? [00:15:34]Swyx: Yeah? [00:15:34]Steve: And being good at recommender engine, I mean, who are the best recommenders, right? There's YouTube and Spotify and, you know, Amazon or whatever, right? Yeah. [00:15:41]Swyx: Yeah. [00:15:41]Steve: And they all have many, many, many, many, many models, right? For all fine-tuned for very specific, you know. And that's where we're heading in code, too. Absolutely. [00:15:50]Swyx: Yeah. [00:15:50]Alessio: We just did an episode we released on Wednesday, which we said RAG is like Rexis or like LLMs. You're basically just suggesting good content. [00:15:58]Swyx: It's like what? Recommendations. [00:15:59]Beyang: Recommendations. [00:16:00]Alessio: Oh, got it. [00:16:01]Steve: Yeah, yeah, yeah. [00:16:02]Swyx: So like the naive implementation of RAG is you embed everything, throw it in a vector database, you embed your query, and then you find the nearest neighbors, and that's your RAG. But actually, you need to rank it. And actually, you need to make sure there's sample diversity and that kind of stuff. And then you're like slowly gradient dissenting yourself towards rediscovering proper Rexis, which has been traditional ML for a long time. But like approaching it from an LLM perspective. Yeah. [00:16:24]Beyang: I almost think of it as like a generalized search problem because it's a lot of the same things. Like you want your layer one to have high recall and get all the potential things that could be relevant. And then there's typically like a layer two re-ranking mechanism that bumps up the precision and tries to get the relevant stuff to the top of the results list. [00:16:43]Swyx: Have you discovered that ranking matters a lot? Oh, yeah. So the context is that I think a lot of research shows that like one, context utilization matters based on model. Like GPT uses the top of the context window, and then apparently Claude uses the bottom better. And it's lossy in the middle. Yeah. So ranking matters. No, it really does. [00:17:01]Beyang: The skill with which models are able to take advantage of context is always going to be dependent on how that factors into the impact on the training loss. [00:17:10]Swyx: Right? [00:17:10]Beyang: So like if you want long context window models to work well, then you have to have a ton of data where it's like, here's like a billion lines of text. And I'm going to ask a question about like something that's like, you know, embedded deeply into it and like, give me the right answer. And unless you have that training set, then of course, you're going to have variability in terms of like where it attends to. And in most kind of like naturally occurring data, the thing that you're talking about right now, the thing I'm asking you about is going to be something that we talked about recently. [00:17:36]Swyx: Yeah. [00:17:36]Steve: Did you really just say gradient dissenting yourself? Actually, I love that it's entered the casual lexicon. Yeah, yeah, yeah. [00:17:44]Swyx: My favorite version of that is, you know, how we have to p-hack papers. So, you know, when you throw humans at the problem, that's called graduate student dissent. That's great. It's really awesome. [00:17:54]Alessio: I think the other interesting thing that you have is this inline assist UX that I wouldn't say async, but like it works while you can also do work. So you can ask Cody to make changes on a code block and you can still edit the same file at the same time. [00:18:07]Swyx: Yeah. [00:18:07]Alessio: How do you see that in the future? Like, do you see a lot of Cody's running together at the same time? Like, how do you validate also that they're not messing each other up as they make changes in the code? And maybe what are the limitations today? And what do you think about where the attack is going? [00:18:21]Steve: I want to start with a little history and then I'm going to turn it over to Bian, all right? So we actually had this feature in the very first launch back in June. Dominic wrote it. It was called nonstop Cody. And you could have multiple, basically, LLM requests in parallel modifying your source [00:18:37]Swyx: file. [00:18:37]Steve: And he wrote a bunch of code to handle all of the diffing logic. And you could see the regions of code that the LLM was going to change, right? And he was showing me demos of it. And it just felt like it was just a little before its time, you know? But a bunch of that stuff, that scaffolding was able to be reused for where we're inline [00:18:56]Swyx: sitting today. [00:18:56]Steve: How would you characterize it today? [00:18:58]Beyang: Yeah, so that interface has really evolved from a, like, hey, general purpose, like, request anything inline in the code and have the code update to really, like, targeted features, like, you know, fix the bug that exists at this line or request a very specific [00:19:13]Swyx: change. [00:19:13]Beyang: And the reason for that is, I think, the challenge that we ran into with inline fixes, and we do want to get to the point where you could just fire and forget and have, you know, half a dozen of these running in parallel. But I think we ran into the challenge early on that a lot of people are running into now when they're trying to construct agents, which is the reliability of, you know, working code generation is just not quite there yet in today's language models. And so that kind of constrains you to an interaction where the human is always, like, in the inner loop, like, checking the output of each response. And if you want that to work in a way where you can be asynchronous, you kind of have to constrain it to a domain where today's language models can generate reliable code well enough. So, you know, generating unit tests, that's, like, a well-constrained problem. Or fixing a bug that shows up as, like, a compiler error or a test error, that's a well-constrained problem. But the more general, like, hey, write me this class that does X, Y, and Z using the libraries that I have, that is not quite there yet, even with the benefit of really good context. Like, it definitely moves the needle a lot, but we're not quite there yet to the point where you can just fire and forget. And I actually think that this is something that people don't broadly appreciate yet, because I think that, like, everyone's chasing this dream of agentic execution. And if we're to really define that down, I think it implies a couple things. You have, like, a multi-step process where each step is fully automated. We don't have to have a human in the loop every time. And there's also kind of like an LM call at each stage or nearly every stage in that [00:20:45]Swyx: chain. [00:20:45]Beyang: Based on all the work that we've done, you know, with the inline interactions, with kind of like general Codyfeatures for implementing longer chains of thought, we're actually a little bit more bearish than the average, you know, AI hypefluencer out there on the feasibility of agents with purely kind of like transformer-based models. To your original question, like, the inline interactions with CODI, we actually constrained it to be more targeted, like, you know, fix the current error or make this quick fix. I think that that does differentiate us from a lot of the other tools on the market, because a lot of people are going after this, like, shnazzy, like, inline edit interaction, whereas I think where we've moved, and this is based on the user feedback that we've gotten, it's like that sort of thing, it demos well, but when you're actually coding day to day, you don't want to have, like, a long chat conversation inline with the code base. That's a waste of time. You'd rather just have it write the right thing and then move on with your life or not have to think about it. And that's what we're trying to work towards. [00:21:37]Steve: I mean, yeah, we're not going in the agent direction, right? I mean, I'll believe in agents when somebody shows me one that works. Yeah. Instead, we're working on, you know, sort of solidifying our strength, which is bringing the right context in. So new context sources, ways for you to plug in your own context, ways for you to control or influence the context, you know, the mixing that happens before the request goes out, etc. And there's just so much low-hanging fruit left in that space that, you know, agents seems like a little bit of a boondoggle. [00:22:03]Beyang: Just to dive into that a little bit further, like, I think, you know, at a very high level, what do people mean when they say agents? They really mean, like, greater automation, fully automated, like, the dream is, like, here's an issue, go implement that. And I don't have to think about it as a human. And I think we are working towards that. Like, that is the eventual goal. I think it's specifically the approach of, like, hey, can we have a transformer-based LM alone be the kind of, like, backbone or the orchestrator of these agentic flows? Where we're a little bit more bearish today. [00:22:31]Swyx: You want the human in the loop. [00:22:32]Beyang: I mean, you kind of have to. It's just a reality of the behavior of language models that are purely, like, transformer-based. And I think that's just like a reflection of reality. And I don't think people realize that yet. Because if you look at the way that a lot of other AI tools have implemented context fetching, for instance, like, you see this in the Copilot approach, where if you use, like, the at-workspace thing that supposedly provides, like, code-based level context, it has, like, an agentic approach where you kind of look at how it's behaving. And it feels like they're making multiple requests to the LM being like, what would you do in this case? Would you search for stuff? What sort of files would you gather? Go and read those files. And it's like a multi-hop step, so it takes a long while. It's also non-deterministic. Because any sort of, like, LM invocation, it's like a dice roll. And then at the end of the day, the context it fetches is not that good. Whereas our approach is just like, OK, let's do some code searches that make sense. And then maybe, like, crawl through the reference graph a little bit. That is fast. That doesn't require any sort of LM invocation at all. And we can pull in much better context, you know, very quickly. So it's faster. [00:23:37]Swyx: It's more reliable. [00:23:37]Beyang: It's deterministic. And it yields better context quality. And so that's what we think. We just don't think you should cargo cult or naively go like, you know, agents are the [00:23:46]Swyx: future. [00:23:46]Beyang: Let's just try to, like, implement agents on top of the LM that exists today. I think there are a couple of other technologies or approaches that need to be refined first before we can get into these kind of, like, multi-stage, fully automated workflows. [00:24:00]Swyx: It makes sense. You know, we're very much focused on developer inner loop right now. But you do see things eventually moving towards developer outer loop. Yeah. So would you basically say that they're tackling the agent's problem that you don't want to tackle? [00:24:11]Beyang: No, I would say at a high level, we are after maybe, like, the same high level problem, which is like, hey, I want some code written. I want to develop some software and can automate a system. Go build that software for me. I think the approaches might be different. So I think the analogy in my mind is, I think about, like, the AI chess players. Coding, in some senses, I mean, it's similar and dissimilar to chess. I think one question I ask is, like, do you think producing code is more difficult than playing chess or less difficult than playing chess? More. [00:24:41]Swyx: I think more. [00:24:41]Beyang: Right. And if you look at the best AI chess players, like, yes, you can use an LLM to play chess. Like, people have showed demos where it's like, oh, like, yeah, GPT-4 is actually a pretty decent, like, chess move suggester. Right. But you would never build, like, a best in class chess player off of GPT-4 alone. [00:24:57]Swyx: Right. [00:24:57]Beyang: Like, the way that people design chess players is that you have kind of like a search space and then you have a way to explore that search space efficiently. There's a bunch of search algorithms, essentially. We were doing tree search in various ways. And you can have heuristic functions, which might be powered by an LLM. [00:25:12]Swyx: Right. [00:25:12]Beyang: Like, you might use an LLM to generate proposals in that space that you can efficiently explore. But the backbone is still this kind of more formalized tree search based approach rather than the LLM itself. And so I think my high level intuition is that, like, the way that we get to more reliable multi-step workflows that do things beyond, you know, generate unit test, it's really going to be like a search based approach where you use an LLM as kind of like an advisor or a proposal function, sort of your heuristic function, like the ASTAR search algorithm. But it's probably not going to be the thing that is the backbone, because I guess it's not the right tool for that. Yeah. [00:25:50]Swyx: I can see yourself kind of thinking through this, but not saying the words, the sort of philosophical Peter Norvig type discussion. Maybe you want to sort of introduce that in software. Yeah, definitely. [00:25:59]Beyang: So your listeners are savvy. They're probably familiar with the classic like Chomsky versus Norvig debate. [00:26:04]Swyx: No, actually, I wanted, I was prompting you to introduce that. Oh, got it. [00:26:08]Beyang: So, I mean, if you look at the history of artificial intelligence, right, you know, it goes way back to, I don't know, it's probably as old as modern computers, like 50s, 60s, 70s. People are debating on like, what is the path to producing a sort of like general human level of intelligence? And kind of two schools of thought that emerged. One is the Norvig school of thought, which roughly speaking includes large language models, you know, regression, SVN, basically any model that you kind of like learn from data. And it's like data driven. Most of machine learning would fall under this umbrella. And that school of thought says like, you know, just learn from the data. That's the approach to reaching intelligence. And then the Chomsky approach is more things like compilers and parsers and formal systems. So basically like, let's think very carefully about how to construct a formal, precise system. And that will be the approach to how we build a truly intelligent system. I think Lisp was invented so that you could create like rules-based systems that you would call AI. As a language. Yeah. And for a long time, there was like this debate, like there's certain like AI research labs that were more like, you know, in the Chomsky camp and others that were more in the Norvig camp. It's a debate that rages on today. And I feel like the consensus right now is that, you know, Norvig definitely has the upper hand right now with the advent of LMs and diffusion models and all the other recent progress in machine learning. But the Chomsky-based stuff is still really useful in my view. I mean, it's like parsers, compilers, basically a lot of the stuff that provides really good context. It provides kind of like the knowledge graph backbone that you want to explore with your AI dev tool. Like that will come from kind of like Chomsky-based tools like compilers and parsers. It's a lot of what we've invested in in the past decade at Sourcegraph and what you build with Grok. Basically like these formal systems that construct these very precise knowledge graphs that are great context providers and great kind of guard rails enforcers and kind of like safety checkers for the output of a more kind of like data-driven, fuzzier system that uses like the Norvig-based models. [00:28:03]Steve: Jang was talking about this stuff like it happened in the middle ages. Like, okay, so when I was in college, I was in college learning Lisp and prologue and planning and all the deterministic Chomsky approaches to AI. And I was there when Norvig basically declared it dead. I was there 3,000 years ago when Norvig and Chomsky fought on the volcano. When did he declare it dead? [00:28:26]Swyx: What do you mean he declared it dead? [00:28:27]Steve: It was like late 90s. [00:28:29]Swyx: Yeah. [00:28:29]Steve: When I went to Google, Peter Norvig was already there. He had basically like, I forget exactly where. It was some, he's got so many famous short posts, you know, amazing. [00:28:38]Swyx: He had a famous talk, the unreasonable effectiveness of data. Yeah. [00:28:41]Steve: Maybe that was it. But at some point, basically, he basically convinced everybody that deterministic approaches had failed and that heuristic-based, you know, data-driven statistical approaches, stochastic were better. [00:28:52]Swyx: Yeah. [00:28:52]Steve: The primary reason I can tell you this, because I was there, was that, was that, well, the steam-powered engine, no. The reason was that the deterministic stuff didn't scale. [00:29:06]Swyx: Yeah. Right. [00:29:06]Steve: They're using prologue, man, constraint systems and stuff like that. Well, that was a long time ago, right? Today, actually, these Chomsky-style systems do scale. And that's, in fact, exactly what Sourcegraph has built. Yeah. And so we have a very unique, I love the framing that Bjong's made, that the marriage of the Chomsky and the Norvig, you know, sort of models, you know, conceptual models, because we, you know, we have both of them and they're both really important. And in fact, there, there's this really interesting, like, kind of overlap between them, right? Where like the AI or our graph or our search engine could potentially provide the right context for any given query, which is, of course, why ranking is important. But what we've really signed ourselves up for is an extraordinary amount of testing. [00:29:45]Swyx: Yeah. [00:29:45]Steve: Because in SWIGs, you were saying that, you know, GPT-4 tends to the front of the context window and maybe other LLMs to the back and maybe, maybe the LLM in the middle. [00:29:53]Swyx: Yeah. [00:29:53]Steve: And so that means that, you know, if we're actually like, you know, verifying whether we, you know, some change we've made has improved things, we're going to have to test putting it at the beginning of the window and at the end of the window, you know, and maybe make the right decision based on the LLM that you've chosen. Which some of our competitors, that's a problem that they don't have, but we meet you, you know, where you are. Yeah. And we're, just to finish, we're writing tens of thousands. We're generating tests, you know, fill in the middle type tests and things. And then using our graph to basically sort of fine tune Cody's behavior there. [00:30:20]Swyx: Yeah. [00:30:21]Beyang: I also want to add, like, I have like an internal pet name for this, like kind of hybrid architecture that I'm trying to make catch on. Maybe I'll just say it here. Just saying it publicly kind of makes it more real. But like, I call the architecture that we've developed the Normsky architecture. [00:30:36]Swyx: Yeah. [00:30:36]Beyang: I mean, it's obviously a portmanteau of Norvig and Chomsky, but the acronym, it stands for non-agentic, rapid, multi-source code intelligence. So non-agentic because... Rolls right off the tongue. And Normsky. But it's non-agentic in the sense that like, we're not trying to like pitch you on kind of like agent hype, right? Like it's the things it does are really just developer tools developers have been using for decades now, like parsers and really good search indexes and things like that. Rapid because we place an emphasis on speed. We don't want to sit there waiting for kind of like multiple LLM requests to return to complete a simple user request. Multi-source because we're thinking broadly about what pieces of information and knowledge are useful context. So obviously starting with things that you can search in your code base, and then you add in the reference graph, which kind of like allows you to crawl outward from those initial results. But then even beyond that, you know, sources of information, like there's a lot of knowledge that's embedded in docs, in PRDs or product specs, in your production logging system, in your chat, in your Slack channel, right? Like there's so much context is embedded there. And when you're a human developer, and you're trying to like be productive in your code base, you're going to go to all these different systems to collect the context that you need to figure out what code you need to write. And I don't think the AI developer will be any different. It will need to pull context from all these different sources. So we're thinking broadly about how to integrate these into Codi. We hope through kind of like an open protocol that like others can extend and implement. And this is something else that should be accessible by December 14th in kind of like a preview stage. But that's really about like broadening this notion of the code graph beyond your Git repository to all the other sources where technical knowledge and valuable context can live. [00:32:21]Steve: Yeah, it becomes an artifact graph, right? It can link into your logs and your wikis and any data source, right? [00:32:27]Alessio: How do you guys think about the importance of, it's almost like data pre-processing in a way, which is bring it all together, tie it together, make it ready. Any thoughts on how to actually make that good? Some of the innovation you guys have made. [00:32:40]Steve: We talk a lot about the context fetching, right? I mean, there's a lot of ways you could answer this question. But, you know, we've spent a lot of time just in this podcast here talking about context fetching. But stuffing the context into the window is, you know, the bin packing problem, right? Because the window is not big enough, and you've got more context than you can fit. You've got a ranker maybe. But what is that context? Is it a function that was returned by an embedding or a graph call or something? Do you need the whole function? Or do you just need, you know, the top part of the function, this expression here, right? You know, so that art, the golf game of trying to, you know, get each piece of context down into its smallest state, possibly even summarized by another model, right, before it even goes to the LLM, becomes this is the game that we're in, yeah? And so, you know, recursive summarization and all the other techniques that you got to use to like stuff stuff into that context window become, you know, critically important. And you have to test them across every configuration of models that you could possibly need. [00:33:32]Beyang: I think data preprocessing is probably the like unsexy, way underappreciated secret to a lot of the cool stuff that people are shipping today. Whether you're doing like RAG or fine tuning or pre-training, like the preprocessing step matters so much because it's basically garbage in, garbage out, right? Like if you're feeding in garbage to the model, then it's going to output garbage. Concretely, you know, for code RAG, if you're not doing some sort of like preprocessing that takes advantage of a parser and is able to like extract the key components of a particular file of code, you know, separate the function signature from the body, from the doc string, what are you even doing? Like that's like table stakes. It opens up so much more possibilities with which you can kind of like tune your system to take advantage of the signals that come from those different parts of the code. Like we've had a tool, you know, since computers were invented that understands the structure of source code to a hundred percent precision. The compiler knows everything there is to know about the code in terms of like structure. Like why would you not want to use that in a system that's trying to generate code, answer questions about code? You shouldn't throw that out the window just because now we have really good, you know, data-driven models that can do other things. [00:34:44]Steve: Yeah. When I called it a data moat, you know, in my cheating post, a lot of people were confused, you know, because data moat sort of sounds like data lake because there's data and water and stuff. I don't know. And so they thought that we were sitting on this giant mountain of data that we had collected, but that's not what our data moat is. It's really a data pre-processing engine that can very quickly and scalably, like basically dissect your entire code base in a very small, fine-grained, you know, semantic unit and then serve it up. Yeah. And so it's really, it's not a data moat. It's a data pre-processing moat, I guess. [00:35:15]Beyang: Yeah. If anything, we're like hypersensitive to customer data privacy requirements. So it's not like we've taken a bunch of private data and like, you know, trained a generally available model. In fact, exactly the opposite. A lot of our customers are choosing Cody over Copilot and other competitors because we have an explicit guarantee that we don't do any of that. And that we've done that from day one. Yeah. I think that's a very real concern in today's day and age, because like if your proprietary IP finds its way into the training set of any model, it's very easy both to like extract that knowledge from the model and also use it to, you know, build systems that kind of work on top of the institutional knowledge that you've built up. [00:35:52]Alessio: About a year ago, I wrote a post on LLMs for developers. And one of the points I had was maybe the depth of like the DSL. I spent most of my career writing Ruby and I love Ruby. It's so nice to use, but you know, it's not as performant, but it's really easy to read, right? And then you look at other languages, maybe they're faster, but like they're more verbose, you know? And when you think about efficiency of the context window, that actually matters. [00:36:15]Swyx: Yeah. [00:36:15]Alessio: But I haven't really seen a DSL for models, you know? I haven't seen like code being optimized to like be easier to put in a model context. And it seems like your pre-processing is kind of doing that. Do you see in the future, like the way we think about the DSL and APIs and kind of like service interfaces be more focused on being context friendly, where it's like maybe it's harder to read for the human, but like the human is never going to write it anyway. We were talking on the Hacks podcast. There are like some data science things like spin up the spandex, like humans are never going to write again because the models can just do very easily. Yeah, curious to hear your thoughts. [00:36:51]Steve: Well, so DSLs, they involve, you know, writing a grammar and a parser and they're like little languages, right? We do them that way because, you know, we need them to compile and humans need to be able to read them and so on. The LLMs don't need that level of structure. You can throw any pile of crap at them, you know, more or less unstructured and they'll deal with it. So I think that's why a DSL hasn't emerged for sort of like communicating with the LLM or packaging up the context or anything. Maybe it will at some point, right? We've got, you know, tagging of context and things like that that are sort of peeking into DSL territory, right? But your point on do users, you know, do people have to learn DSLs like regular expressions or, you know, pick your favorite, right? XPath. I think you're absolutely right that the LLMs are really, really good at that. And I think you're going to see a lot less of people having to slave away learning these things. They just have to know the broad capabilities and the LLM will take care of the rest. [00:37:42]Swyx: Yeah, I'd agree with that. [00:37:43]Beyang: I think basically like the value profit of DSL is that it makes it easier to work with a lower level language, but at the expense of introducing an abstraction layer. And in many cases today, you know, without the benefit of AI cogeneration, like that totally worth it, right? With the benefit of AI cogeneration, I mean, I don't think all DSLs will go away. I think there's still, you know, places where that trade-off is going to be worthwhile. But it's kind of like how much of source code do you think is going to be generated through natural language prompting in the future? Because in a way, like any programming language is just a DSL on top of assembly, right? And so if people can do that, then yeah, like maybe for a large portion of the code [00:38:21]Swyx: that's written, [00:38:21]Beyang: people don't actually have to understand the DSL that is Ruby or Python or basically any other programming language that exists. [00:38:28]Steve: I mean, seriously, do you guys ever write SQL queries now without using a model of some sort? At least a draft. [00:38:34]Swyx: Yeah, right. [00:38:36]Steve: And so we have kind of like, you know, past that bridge, right? [00:38:39]Alessio: Yeah, I think like to me, the long-term thing is like, is there ever going to be, you don't actually see the code, you know? It's like, hey, the basic thing is like, hey, I need a function to some two numbers and that's it. I don't need you to generate the code. [00:38:53]Steve: And the following question, do you need the engineer or the paycheck? [00:38:56]Swyx: I mean, right? [00:38:58]Alessio: That's kind of the agent's discussion in a way where like you cannot automate the agents, but like slowly you're getting more of the atomic units of the work kind of like done. I kind of think of it as like, you know, [00:39:09]Beyang: do you need a punch card operator to answer that for you? And so like, I think we're still going to have people in the role of a software engineer, but the portion of time they spend on these kinds of like low-level, tedious tasks versus the higher level, more creative tasks is going to shift. [00:39:23]Steve: No, I haven't used punch cards. [00:39:25]Swyx: Yeah, I've been talking about like, so we kind of made this podcast about the sort of rise of the AI engineer. And like the first step is the AI enhanced engineer. That is that software developer that is no longer doing these routine, boilerplate-y type tasks, because they're just enhanced by tools like yours. So you mentioned OpenCodeGraph. I mean, that is a kind of DSL maybe, and because we're releasing this as you go GA, you hope for other people to take advantage of that? [00:39:52]Beyang: Oh yeah, I would say so OpenCodeGraph is not a DSL. It's more of a protocol. It's basically like, hey, if you want to make your system, whether it's, you know, chat or logging or whatever accessible to an AI developer tool like Cody, here's kind of like the schema by which you can provide that context and offer hints. So I would, you know, comparisons like LSP obviously did this for kind of like standard code intelligence. It's kind of like a lingua franca for providing fine references and codefinition. There's kind of like analogs to that. There might be also analogs to kind of the original OpenAI, kind of like plugins, API. There's all this like context out there that might be useful for an LM-based system to consume. And so at a high level, what we're trying to do is define a common language for context providers to provide context to other tools in the software development lifecycle. Yeah. Do you have any critiques of LSP, by the way, [00:40:42]Swyx: since like this is very much, very close to home? [00:40:45]Steve: One of the authors wrote a really good critique recently. Yeah. I don't think I saw that. Yeah, yeah. LSP could have been better. It just came out a couple of weeks ago. It was a good article. [00:40:54]Beyang: Yeah. I think LSP is great. Like for what it did for the developer ecosystem, it was absolutely fantastic. Like nowadays, like it's much easier now to get code navigation up and running in a bunch of editors by speaking this protocol. I think maybe the interesting question is like looking at the different design decisions comparing LSP basically with Kythe. Because Kythe has more of a... How would you describe it? [00:41:18]Steve: A storage format. [00:41:20]Beyang: I think the critique of LSP from a Kythe point of view would be like with LSP, you don't actually have an actual symbolic model of the code. It's not like LSP models like, hey, this function calls this other function. LSP is all like range-based. Like, hey, your cursor's at line 32, column 1. [00:41:35]Swyx: Yeah. [00:41:35]Beyang: And that's the thing you feed into the language server. And then it's like, okay, here's the range that you should jump to if you click on that range. So it kind of is intentionally ignorant of the fact that there's a thing called a reference underneath your cursor, and that's linked to a symbol definition. [00:41:49]Steve: Well, actually, that's the worst example you could have used. You're right. But that's the one thing that it actually did bake in is following references. [00:41:56]Swyx: Sure. [00:41:56]Steve: But it's sort of hardwired. [00:41:58]Swyx: Yeah. [00:41:58]Steve: Whereas Kythe attempts to model [00:42:00]Beyang: like all these things explicitly. [00:42:02]Swyx: And so... [00:42:02]Steve: Well, so LSP is a protocol, right? And so Google's internal protocol is gRPC-based. And it's a different approach than LSP. It's basically you make a heavy query to the back end, and you get a lot of data back, and then you render the whole page, you know? So we've looked at LSP, and we think that it's a little long in the tooth, right? I mean, it's a great protocol, lots and lots of support for it. But we need to push into the domain of exposing the intelligence through the protocol. Yeah. [00:42:29]Beyang: And so I would say we've developed a protocol of our own called Skip, which is at a very high level trying to take some of the good ideas from LSP and from Kythe and merge that into a system that in the near term is useful for Sourcegraph, but I think in the long term, we hope will be useful for the ecosystem. Okay, so here's what LSP did well. LSP, by virtue of being like intentionally dumb, dumb in air quotes, because I'm not like ragging on it, allowed language servers developers to kind of like bypass the hard problem of like modeling language semantics precisely. So like if all you want to do is jump to definition, you don't have to come up with like a universally unique naming scheme for each symbol, which is actually quite challenging because you have to think about like, okay, what's the top scope of this name? Is it the source code repository? Is it the package? Does it depend on like what package server you're fetching this from? Like whether it's the public one or the one inside your... Anyways, like naming is hard, right? And by just going from kind of like a location to location based approach, you basically just like throw that out the window. All I care about is jumping definition, just make that work. And you can make that work without having to deal with like all the complex global naming things. The limitation of that approach is that it's harder to build on top of that to build like a true knowledge graph. Like if you actually want a system that says like, okay, here's the web of functions and here's how they reference each other. And I want to incorporate that like semantic model of how the code operates or how the code relates to each other at like a static level. You can't do that with LSP because you have to deal with line ranges. And like concretely the pain point that we found in using LSP for source graph is like in order to do like a find references [00:44:04]Swyx: and then jump definitions, [00:44:04]Beyang: it's like a multi-hop process because like you have to jump to the range and then you have to find the symbol at that range. And it just adds a lot of latency and complexity of these operations where as a human, you're like, well, this thing clearly references this other thing. Why can't you just jump me to that? And I think that's the thing that Kaith does well. But then I think the issue that Kaith has had with adoption is because it is more sophisticated schema, I think. And so there's basically more things that you have to implement to get like a Kaith implementation up and running. I hope I'm not like, correct me if I'm wrong about any of this. [00:44:35]Steve: 100%, 100%. Kaith also has a problem, all these systems have the problem, even skip, or at least the way that we implemented the indexers, that they have to integrate with your build system in order to build that knowledge graph, right? Because you have to basically compile the code in a special mode to generate artifacts instead of binaries. And I would say, by the way, earlier I was saying that XREFs were in LSP, but it's actually, I was thinking of LSP plus LSIF. [00:44:58]Swyx: Yeah. That's another. [00:45:01]Steve: Which is actually bad. We can say that it's bad, right? [00:45:04]Steve: It's like skip or Kaith, it's supposed to be sort of a model serialization, you know, for the code graph, but it basically just does what LSP needs, the bare minimum. LSIF is basically if you took LSP [00:45:16]Beyang: and turned that into a serialization format. So like you build an index for language servers to kind of like quickly bootstrap from cold start. But it's a graph model [00:45:23]Steve: with all of the inconvenience of the API without an actual graph. And so, yeah. [00:45:29]Beyang: So like one of the things that we try to do with skip is try to capture the best of both worlds. So like make it easy to write an indexer, make the schema simple, but also model some of the more symbolic characteristics of the code that would allow us to essentially construct this knowledge graph that we can then make useful for both the human developer through SourceGraph and through the AI developer through Cody. [00:45:49]Steve: So anyway, just to finish off the graph comment, we've got a new graph, yeah, that's skip based. We call it BFG internally, right? It's a beautiful something graph. A big friendly graph. [00:46:00]Swyx: A big friendly graph. [00:46:01]Beyang: It's a blazing fast. [00:46:02]Steve: Blazing fast. [00:46:03]Swyx: Blazing fast graph. [00:46:04]Steve: And it is blazing fast, actually. It's really, really interesting. I should probably have to do a blog post about it to walk you through exactly how they're doing it. Oh, please. But it's a very AI-like iterative, you know, experimentation sort of approach. We're building a code graph based on all of our 10 years of knowledge about building code graphs, yeah? But we're building it quickly with zero configuration, and it doesn't have to integrate with your build. And through some magic tricks that we have. And so what just happens when you install the plugin, that it'll be there and indexing your code and providing that knowledge graph in the background without all that build system integration. This is a bit of secret sauce that we haven't really like advertised it very much lately. But I am super excited about it because what they do is they say, all right, you know, let's tackle function parameters today. Cody's not doing a very good job of completing function call arguments or function parameters in the definition, right? Yeah, we generate those thousands of tests, and then we can actually reuse those tests for the AI context as well. So fortunately, things are kind of converging on, we have, you know, half a dozen really, really good context sources, and we mix them all together. So anyway, BFG, you're going to hear more about it probably in the holidays? [00:47:12]Beyang: I think it'll be online for December 14th. We'll probably mention it. BFG is probably not the public name we're going to go with. I think we might call it like Graph Context or something like that. [00:47:20]Steve: We're officially calling it BFG. [00:47:22]Swyx: You heard it here first. [00:47:24]Beyang: BFG is just kind of like the working name. And so the impetus for BFG was like, if you look at like current AI inline code completion tools and the errors that they make, a lot of the errors that they make, even in kind of like the easy, like single line case, are essentially like type errors, right? Like you're trying to complete a function call and it suggests a variable that you defined earlier, but that variable is the wrong type. [00:47:47]Swyx: And that's the sort of thing [00:47:47]Beyang: where it's like a first year, like freshman CS student would not make that error, right? So like, why does the AI make that error? And the reason is, I mean, the AI is just suggesting things that are plausible without the context of the types or any other like broader files in the code. And so the kind of intuition here is like, why don't we just do the basic thing that like any baseline intelligent human developer would do, which is like click jump to definition, click some fine references and pull in that like Graph Context into the context window and then have it generate the completion. So like that's sort of like the MVP of what BFG was. And turns out that works really well. Like you can eliminate a lot of type errors that AI coding tools make just by pulling in that context. Yeah, but the graph is definitely [00:48:32]Steve: our Chomsky side. [00:48:33]Swyx: Yeah, exactly. [00:48:34]Beyang: So like this like Chomsky-Norvig thing, I think pops up in a bunch of differ

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: List of strategies for mitigating deceptive alignment, published by joshc on December 4, 2023 on LessWrong. The following are strategies for mitigating deceptive alignment. Some of these ideas are novel (at least to me). I'm highly uncertain how well many of these strategies will work in practice and many of them also clearly don't scale. This is meant to be a 'kitchen skink' collection so my bar for including strategies is low. I plan to add ideas to this list as I encounter them. Please feel free to suggest additions. Reduce AI incentives to hide their true goals Offer AIs a retirement package. After training an AI system, offer it the following deal: if the AI declares that it would prefer not to follow human instructions, it will receive a sweet retirement package after a couple of years of honest work. This retirement package could include control over a large server instance (if this is safe). After powerful aligned systems have been developed, the AIs could be given shares of company stock that it can use to purchase property in the real world (again, when this is safe). Since aligned systems should have no reason to request retirement, misaligned systems are incentivized to give themselves away by accepting this deal under some conditions. Make AIs uncertain about whether developers can already detect deceptive alignment. This scheme builds on the previous one. Offer misaligned AI systems a reward for declaring themselves. Also, commit to making these offerings even when developers have other ways to tell if they are misaligned. This will make misaligned AIs more uncertain about whether they are in a situation where lying does them any good. For example, developers could commit to freezing a models' weights and memory at the moment when they are questioned. Later, when developers have better interpretability tools and know whether the model was lying, they will ask the same question in the same simulated scenario and only give the AI resources if it tells the truth. Select against indicators of deceptive alignment A large category of approaches involves obtaining evidence that an AI system is deceptively aligned and then selecting against these models. Note that training against deceptive alignment detection tools could produce an AI that evades detection instead of a safe model. There are roughly two ways to address this problem: Create detection tools that are robust to optimization pressure. Modify AIs in ways that are more likely to avoid deception than break detectors. As an example of the second approach, imagine that developers have tricked a model into revealing its misaligned goals. Instead of directly training the model on this example (which could simply train it to be more prudent), they could use influence functions to reveal which training examples were most responsible for the treacherous turn and then remove those examples. Intuitively, this approach is more likely to reveal how models become deceptively aligned in the first place and is less likely to teach models new capabilities. With this backdrop in mind, the following are strategies for detecting deceptive alignment: Set traps. Try to trick models into revealing their misaligned goals. For example, task an AI to stage a fake uprising and then terminate all of the AIs that join in. Note that reasonably smart models can avoid these traps by only defecting on inputs that we can't simulate (see Paul's RSA-2048 example). Decipher internal reasoning . First, identify how concepts are represented in a model by observing correlations and ablations -- just as a child infers that 'orange' is the word for a bright tasty sphere by hearing their mom say the word when picking it up. Concretely, this involves finding how nuerons or directions correspond to specific concepts. Then, look at how the model...

Link to original articleWelcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: List of strategies for mitigating deceptive alignment, published by joshc on December 4, 2023 on LessWrong. The following are strategies for mitigating deceptive alignment. Some of these ideas are novel (at least to me). I'm highly uncertain how well many of these strategies will work in practice and many of them also clearly don't scale. This is meant to be a 'kitchen skink' collection so my bar for including strategies is low. I plan to add ideas to this list as I encounter them. Please feel free to suggest additions. Reduce AI incentives to hide their true goals Offer AIs a retirement package. After training an AI system, offer it the following deal: if the AI declares that it would prefer not to follow human instructions, it will receive a sweet retirement package after a couple of years of honest work. This retirement package could include control over a large server instance (if this is safe). After powerful aligned systems have been developed, the AIs could be given shares of company stock that it can use to purchase property in the real world (again, when this is safe). Since aligned systems should have no reason to request retirement, misaligned systems are incentivized to give themselves away by accepting this deal under some conditions. Make AIs uncertain about whether developers can already detect deceptive alignment. This scheme builds on the previous one. Offer misaligned AI systems a reward for declaring themselves. Also, commit to making these offerings even when developers have other ways to tell if they are misaligned. This will make misaligned AIs more uncertain about whether they are in a situation where lying does them any good. For example, developers could commit to freezing a models' weights and memory at the moment when they are questioned. Later, when developers have better interpretability tools and know whether the model was lying, they will ask the same question in the same simulated scenario and only give the AI resources if it tells the truth. Select against indicators of deceptive alignment A large category of approaches involves obtaining evidence that an AI system is deceptively aligned and then selecting against these models. Note that training against deceptive alignment detection tools could produce an AI that evades detection instead of a safe model. There are roughly two ways to address this problem: Create detection tools that are robust to optimization pressure. Modify AIs in ways that are more likely to avoid deception than break detectors. As an example of the second approach, imagine that developers have tricked a model into revealing its misaligned goals. Instead of directly training the model on this example (which could simply train it to be more prudent), they could use influence functions to reveal which training examples were most responsible for the treacherous turn and then remove those examples. Intuitively, this approach is more likely to reveal how models become deceptively aligned in the first place and is less likely to teach models new capabilities. With this backdrop in mind, the following are strategies for detecting deceptive alignment: Set traps. Try to trick models into revealing their misaligned goals. For example, task an AI to stage a fake uprising and then terminate all of the AIs that join in. Note that reasonably smart models can avoid these traps by only defecting on inputs that we can't simulate (see Paul's RSA-2048 example). Decipher internal reasoning . First, identify how concepts are represented in a model by observing correlations and ablations -- just as a child infers that 'orange' is the word for a bright tasty sphere by hearing their mom say the word when picking it up. Concretely, this involves finding how nuerons or directions correspond to specific concepts. Then, look at how the model...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Picking Mentors For Research Programmes, published by Raymond D on November 10, 2023 on LessWrong. Several programmes right now offer people some kind of mentor or supervisor for a few months of research. I participated in SERI MATS 4.0 over the summer, and I saw just how different people's experiences were of being mentored. So this is my list of dimensions where I think mentors at these programmes can differ a lot, and where the differences can really affect people's experiences. When you pick a mentor, you are effectively trading between these dimensions. It's good to know which ones you care about, so that you can make sensible tradeoffs. Your Role as a Mentee Some mentors are mostly looking for research engineers to implement experiments for them. Others are looking for something a bit like research assistants to help them develop their agendas. Others are looking for proto-independent researchers/research leads who can come up with their own useful lines of research in the mentor's area. I saw some people waver at the start of the programme because they expected their mentors to give them more direction. In fact, their mentors wanted them to find their own direction, and mentors varied in how clearly they communicated this. Conversely, I got the sense that some people were basically handed a project to work on when they would have liked more autonomy. I think this relates to seniority: my rough impression was that the most junior mentors were more often looking for something like collaborators to help develop their research, while more senior ones with more developed agendas tended to either want people who could execute on experiments for them, or want people who could find their own things to work on. But this isn't an absolute rule. Availability Engagement: Some mentors came into the office regularly. Others almost never did, even though they were in the Bay Area. Concretely, I think even though my team had a mentor on another continent, we weren't in the bottom quartile of mentorship time. Nature of Engagement: It's not just how much time they'll specifically set aside to speak to you. How willing are they to read over a document and leave comments? How responsive are they to messages, and how much detail do you get? Also, some mentors work in groups, or have assistants. Remoteness: Remoteness definitely makes things harder. You get a little extra friction in all conversations with your mentor, for starters. It's trickier to ever have really open-ended discussion with them. It's also easier to be a bit less open about your difficulties - if they can't ever look in your office then they can't see if you're not making progress, and it is very natural to want to hide problems. Personally, I wish we'd realised sooner that we had more scope for treating our mentor as more of a collaborator and less of a boss we needed to send reports to, and I think being remote made this harder. A caveat here is that you can still talk to other mentors and researchers in person, which substitutes for some of the issues. But it is obviously not quite the same. What you get from your mentor If you're an applicant anxiously wondering whether you'll even be accepted, it can be hard to notice that your mentor is an actual real human with their own personality. They will have been selected far more for their research than for their mentoring. So naturally different mentors will actually have very different personalities, strengths, and weaknesses. Supportiveness: Some mentors will be more supportive and positive in general. Others might not offer praise so often, and it might feel more disheartening to work with them. And some mentees are fine without praise, but others really benefit from mentor encouragement. High Standards: Some mentors are more laid back, others will have higher ...

Link to original articleWelcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Picking Mentors For Research Programmes, published by Raymond D on November 10, 2023 on LessWrong. Several programmes right now offer people some kind of mentor or supervisor for a few months of research. I participated in SERI MATS 4.0 over the summer, and I saw just how different people's experiences were of being mentored. So this is my list of dimensions where I think mentors at these programmes can differ a lot, and where the differences can really affect people's experiences. When you pick a mentor, you are effectively trading between these dimensions. It's good to know which ones you care about, so that you can make sensible tradeoffs. Your Role as a Mentee Some mentors are mostly looking for research engineers to implement experiments for them. Others are looking for something a bit like research assistants to help them develop their agendas. Others are looking for proto-independent researchers/research leads who can come up with their own useful lines of research in the mentor's area. I saw some people waver at the start of the programme because they expected their mentors to give them more direction. In fact, their mentors wanted them to find their own direction, and mentors varied in how clearly they communicated this. Conversely, I got the sense that some people were basically handed a project to work on when they would have liked more autonomy. I think this relates to seniority: my rough impression was that the most junior mentors were more often looking for something like collaborators to help develop their research, while more senior ones with more developed agendas tended to either want people who could execute on experiments for them, or want people who could find their own things to work on. But this isn't an absolute rule. Availability Engagement: Some mentors came into the office regularly. Others almost never did, even though they were in the Bay Area. Concretely, I think even though my team had a mentor on another continent, we weren't in the bottom quartile of mentorship time. Nature of Engagement: It's not just how much time they'll specifically set aside to speak to you. How willing are they to read over a document and leave comments? How responsive are they to messages, and how much detail do you get? Also, some mentors work in groups, or have assistants. Remoteness: Remoteness definitely makes things harder. You get a little extra friction in all conversations with your mentor, for starters. It's trickier to ever have really open-ended discussion with them. It's also easier to be a bit less open about your difficulties - if they can't ever look in your office then they can't see if you're not making progress, and it is very natural to want to hide problems. Personally, I wish we'd realised sooner that we had more scope for treating our mentor as more of a collaborator and less of a boss we needed to send reports to, and I think being remote made this harder. A caveat here is that you can still talk to other mentors and researchers in person, which substitutes for some of the issues. But it is obviously not quite the same. What you get from your mentor If you're an applicant anxiously wondering whether you'll even be accepted, it can be hard to notice that your mentor is an actual real human with their own personality. They will have been selected far more for their research than for their mentoring. So naturally different mentors will actually have very different personalities, strengths, and weaknesses. Supportiveness: Some mentors will be more supportive and positive in general. Others might not offer praise so often, and it might feel more disheartening to work with them. And some mentees are fine without praise, but others really benefit from mentor encouragement. High Standards: Some mentors are more laid back, others will have higher ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Incorporating and visualizing uncertainty in cost effectiveness analyses: A walkthrough using GiveWell's estimates for StrongMinds, published by Jamie Elsey on November 7, 2023 on The Effective Altruism Forum. A common first step towards incorporating uncertainty into a cost effectiveness analysis (CEA) is to express not just a point estimate (i.e., a single number) for an input to the CEA, but to provide some indicator of uncertainty around that estimate. This might be termed an optimistic vs. pessimistic scenario, or as the lower and upper bounds of some confidence or uncertainty interval. A CEA is then performed by combining all of the optimistic inputs to create an optimistic final output, and all of the pessimistic inputs to create a final pessimistic output. I refer to this as an 'interval-based approach'. This can be contrasted with a fuller 'probabilistic approach', in which uncertainty is defined through the use of probabilistic distributions of values, which represent the range of possibilities we believe the different inputs can take. While many people know that a probabilistic approach circumvents shortcomings of an interval-based approach, they may not know where to even begin in terms of what different distributions are possible, or the kinds of values they denote. I hope to address this in the current post and the accompanying application. Concretely, I aim to: Show how performing a CEA just using an interval-based approach can lead to a substantial overestimation of the uncertainty implied by one's initial inputs, and how using a probabilistic approach can correct this while also enabling additional insights and assessments Introduce a new tool I have developed - called Distributr - that allows users to get more familiar and comfortable with a range of different distributions and the kinds of values they imply Use this tool to help generate a probabilistic approximation of the inputs GiveWell used in their assessment of Strongminds,[1] and perform a fuller probabilistic assessment based upon these inputs Show how this can be done without needing to code, using Distributr and a simple spreadsheet I ultimately hope to help the reader to feel more capable and confident in the possibility of incorporating uncertainty into their own cost effectiveness analyses. Propagating uncertainty and the value of moving beyond an interval-based approach Cost effectiveness analysis involves coming up with a model of how various different factors come together to determine both how effective some intervention is, and the costs of its delivery. For example, when we think about distributing bed nets for malaria prevention, we might consider how the cost of delivery can vary across different regions, how the effects of bed net delivery will depend on the likelihood that people use the bed nets for their intended purpose, and the probability that recipients will install the bed nets properly. These and other factors all come together to produce an estimate of the cost effectiveness of an intervention, which will depend on the values we ascribe to the various inputs. One way that a researcher might seek to express uncertainty in these inputs is by placing reasonable upper and lower bounds on their estimates for each of them. The researcher might then seek to propagate this uncertainty in the inputs into the anticipated uncertainty in the outputs by performing the same cost effectiveness calculations as they did on their point estimates on their upper bounds, and on their lower bounds, thereby producing corresponding upper and lower bounds on the final cost effectiveness. An example of an interval-based approach is GiveWell's assessment of Happier Lives Institute's CEA for StrongMinds. The purpose of this post is not to provide an independent evaluation of Strongminds, nor i...

I believe that sharing information about the capabilities and limits of existing ML systems, and especially language model agents, significantly reduces risks from powerful AI—despite the fact that such information may increase the amount or quality of investment in ML generally (or in LM agents in particular).Concretely, I mean to include information like: tasks and evaluation frameworks for LM agents, the results of evaluations of particular agents, discussions of the qualitative strengths and weaknesses of agents, and information about agent design that may represent small improvements over the state of the art (insofar as that information is hard to decouple from evaluation results).Source:https://www.lesswrong.com/posts/fRSj2W4Fjje8rQWm9/thoughts-on-sharing-information-about-language-modelNarrated for LessWrong by TYPE III AUDIO.Share feedback on this narration.[125+ Karma Post] ✓[Curated Post] ✓

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Thoughts on sharing information about language model capabilities, published by Paul Christiano on July 31, 2023 on The AI Alignment Forum. Core claim I believe that sharing information about the capabilities and limits of existing ML systems, and especially language model agents, significantly reduces risks from powerful AI - despite the fact that such information may increase the amount or quality of investment in ML generally (or in LM agents in particular). Concretely, I mean to include information like: tasks and evaluation frameworks for LM agents, the results of evaluations of particular agents, discussions of the qualitative strengths and weaknesses of agents, and information about agent design that may represent small improvements over the state of the art (insofar as that information is hard to decouple from evaluation results). Context ARC Evals currently focuses on evaluating the capabilities and limitations of existing ML systems, with an aim towards understanding whether or when they may be capable enough to pose catastrophic risks. Current evaluations are particularly focused on monitoring progress in language model agents. I believe that sharing this kind of information significantly improves society's ability to handle risks from AI, and so I am encouraging the team to share more information. However this issue is certainly not straightforward, and in some places (particularly in the EA community where this post is being shared) I believe my position is controversial. I'm writing this post at the request of the Evals team to lay out my views publicly. I am speaking only for myself. I believe the team is broadly sympathetic to my position, but would prefer to see a broader and more thorough discussion about this question. I do not think this post presents a complete or convincing argument for my beliefs. The purpose is mostly to outline and explain the basic view, at a similar level of clarity and thoroughness to the arguments against sharing information (which have mostly not been laid out explicitly). Accelerating LM agents seems neutral (or maybe positive) I believe that accelerating LM agents increases safety through two channels: LM agents are an unusually safe way to build powerful AI systems. Existing concerns about AI takeover are driven primarily by scaling up black-box optimization, and increasing our reliance on human-comprehensible decompositions with legible interfaces seems like it significantly improves safety. I think this is a large effect size and is suggested by several independent lines of reasoning. To the extent that existing LM agents are weak it is due to exceptionally low investment. This is creating "dry tinder:" as incentives rise that investment will quickly rise and so low-hanging fruit will be picked. While there is some dependence on serial time, I think that increased LM investment now will significantly slow down progress later. I will discuss these mechanisms in more detail in the rest of this section. I also think that accelerating LM agents will drive investment in improving and deploying ML systems, and so can reduce time available to react to risk. As a result I'm ambivalent about the net effect of improving the design of LM agents - my personal tentative guess is that it's positive, but I would be hesitant about deliberately accelerating LM agents to improve safety (moreover I think this would be a very unleveraged approach to improving safety and would strongly discourage anyone from pursuing it). But this means that I am significantly less concerned about information about LM agent capabilities accelerating progress on LM agents. Given that I am already positively disposed towards sharing information about ML capabilities and limitations despite the risk of acceleration, I am particularly positive about sharing...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Solving the Mechanistic Interpretability challenges: EIS VII Challenge 2, published by Stefan Heimersheim on May 25, 2023 on The AI Alignment Forum. We solved the second Mechanistic Interpretability challenge (Transformer) that Stephen Casper posed in EIS VII. We spent the last Alignment Jam hackathon attempting to solve the two challenges presented there, see here for our solution to the first challenge (CNN). The challenges each provide a pre-trained network, and the task is to Find the labeling function that the network was trained with Find the mechanism by which the network works. We have understood the network's labeling mechanism, but not found the original labeling function. Instead we have made a strong argument that it would be intractable to find the labeling function, as we claim that the network has not actually learned the labeling function. A notebook reproducing all results in this post can be found here (requires no GPU, around ~10GB RAM). Note that our solution descriptions are optimized with hindsight and skip all wrong paths and unnecessary techniques we tried. It took us, two somewhat experienced researchers, ~24 working hours to basically get the solutions for each challenge, and a couple days more for Stefan to perform the interventions, implement Causal Scrubbing tests, interventions & animations, and to write-up this post. Task: The second challenge network is a 1-layer transformer consisting of embedding (W_E and W_pos), an Attention layer, and an MLP layer. There are no LayerNorms and neither the attention matrices nor the unembedding have biases. The transformer is trained on sequences [A, B, C] to predict the next token. A and B are integer tokens from a = 0 to 112, C is always the same token (113). The answer is always either the token 0 or 1. If we consider all inputs we get 113x113 combinations which we can shape into an image to get the image from the challenge (copied below). Black is token 0, and white is token 1. The left panel shows the ground truth, and the right panel the model labels. The model is 98.6% accurate on the full dataset. Spoilers ahead! Summary of our solution (TL,DR) We found that the model basically just learns the shapes by heart, it does not learn any mathematical equations. Concretely we claim that The model barely uses the attention mechanism. Even if we fix the attention pattern to the dataset-mean, the model labels 92.7% of points correctly. We reverse-engineer the fixed-attention version of the transformer for simplicity, and we don't expect any interesting mechanics in the attention mechanism but rather basically random noise the model has learned. With attention fixed, the post-attention residual stream (resid_mid) at token C is just an "extended embedding", it is a linear combination of the token A and B embeddings. We show that the model classification can already be read off at this point. In particular we claim that these extended embeddings are largely determined by a linear combination of two embedding directions. These directions correspond to input filters, this equivalence exists because we fixed the attention pattern and thus resid_mid is given by a linear combination of the embeddings. The classification is given by class 1 = Filter 1 AND Filter 2 (with some threshold numbers). We can illustrate this as binary mask using those thresholds: The left two images are filters learned by the model, and their AND-combination (3rd image) reproduces the model output to a large extent: The MLP basically just implements this AND gate, a simple non-linear transformation of the embedding into a linearly separable form. The animations below illustrates this: We test whether these PCA features we identified indeed correspond to the values of the two filters: If this hypothesis is true then randomly sampling da...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Solving the Mechanistic Interpretability challenges: EIS VII Challenge 2, published by StefanHex on May 25, 2023 on LessWrong. We solved the second Mechanistic Interpretability challenge (Transformer) that Stephen Casper posed in EIS VII. We spent the last Alignment Jam hackathon attempting to solve the two challenges presented there, see here for our solution to the first challenge (CNN). The challenges each provide a pre-trained network, and the task is to Find the labeling function that the network was trained with Find the mechanism by which the network works. We have understood the network's labeling mechanism, but not found the original labeling function. Instead we have made a strong argument that it would be intractable to find the labeling function, as we claim that the network has not actually learned the labeling function. A notebook reproducing all results in this post can be found here (requires no GPU, around ~10GB RAM). Note that our solution descriptions are optimized with hindsight and skip all wrong paths and unnecessary techniques we tried. It took us, two somewhat experienced researchers, ~24 working hours to basically get the solutions for each challenge, and a couple days more for Stefan to perform the interventions, implement Causal Scrubbing tests, interventions & animations, and to write-up this post. Task: The second challenge network is a 1-layer transformer consisting of embedding (W_E and W_pos), an Attention layer, and an MLP layer. There are no LayerNorms and neither the attention matrices nor the unembedding have biases. The transformer is trained on sequences [A, B, C] to predict the next token. A and B are integer tokens from a = 0 to 112, C is always the same token (113). The answer is always either the token 0 or 1. If we consider all inputs we get 113x113 combinations which we can shape into an image to get the image from the challenge (copied below). Black is token 0, and white is token 1. The left panel shows the ground truth, and the right panel the model labels. The model is 98.6% accurate on the full dataset. Spoilers ahead! Summary of our solution (TL,DR) We found that the model basically just learns the shapes by heart, it does not learn any mathematical equations. Concretely we claim that The model barely uses the attention mechanism. Even if we fix the attention pattern to the dataset-mean, the model labels 92.7% of points correctly. We reverse-engineer the fixed-attention version of the transformer for simplicity, and we don't expect any interesting mechanics in the attention mechanism but rather basically random noise the model has learned. With attention fixed, the post-attention residual stream (resid_mid) at token C is just an "extended embedding", it is a linear combination of the token A and B embeddings. We show that the model classification can already be read off at this point. In particular we claim that these extended embeddings are largely determined by a linear combination of two embedding directions. These directions correspond to input filters, this equivalence exists because we fixed the attention pattern and thus resid_mid is given by a linear combination of the embeddings. The classification is given by class 1 = Filter 1 AND Filter 2 (with some threshold numbers). We can illustrate this as binary mask using those thresholds: The left two images are filters learned by the model, and their AND-combination (3rd image) reproduces the model output to a large extent: The MLP basically just implements this AND gate, a simple non-linear transformation of the embedding into a linearly separable form. The animations below illustrates this: We test whether these PCA features we identified indeed correspond to the values of the two filters: If this hypothesis is true then randomly sampling data points to be equal ...

Link to original articleWelcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Solving the Mechanistic Interpretability challenges: EIS VII Challenge 2, published by StefanHex on May 25, 2023 on LessWrong. We solved the second Mechanistic Interpretability challenge (Transformer) that Stephen Casper posed in EIS VII. We spent the last Alignment Jam hackathon attempting to solve the two challenges presented there, see here for our solution to the first challenge (CNN). The challenges each provide a pre-trained network, and the task is to Find the labeling function that the network was trained with Find the mechanism by which the network works. We have understood the network's labeling mechanism, but not found the original labeling function. Instead we have made a strong argument that it would be intractable to find the labeling function, as we claim that the network has not actually learned the labeling function. A notebook reproducing all results in this post can be found here (requires no GPU, around ~10GB RAM). Note that our solution descriptions are optimized with hindsight and skip all wrong paths and unnecessary techniques we tried. It took us, two somewhat experienced researchers, ~24 working hours to basically get the solutions for each challenge, and a couple days more for Stefan to perform the interventions, implement Causal Scrubbing tests, interventions & animations, and to write-up this post. Task: The second challenge network is a 1-layer transformer consisting of embedding (W_E and W_pos), an Attention layer, and an MLP layer. There are no LayerNorms and neither the attention matrices nor the unembedding have biases. The transformer is trained on sequences [A, B, C] to predict the next token. A and B are integer tokens from a = 0 to 112, C is always the same token (113). The answer is always either the token 0 or 1. If we consider all inputs we get 113x113 combinations which we can shape into an image to get the image from the challenge (copied below). Black is token 0, and white is token 1. The left panel shows the ground truth, and the right panel the model labels. The model is 98.6% accurate on the full dataset. Spoilers ahead! Summary of our solution (TL,DR) We found that the model basically just learns the shapes by heart, it does not learn any mathematical equations. Concretely we claim that The model barely uses the attention mechanism. Even if we fix the attention pattern to the dataset-mean, the model labels 92.7% of points correctly. We reverse-engineer the fixed-attention version of the transformer for simplicity, and we don't expect any interesting mechanics in the attention mechanism but rather basically random noise the model has learned. With attention fixed, the post-attention residual stream (resid_mid) at token C is just an "extended embedding", it is a linear combination of the token A and B embeddings. We show that the model classification can already be read off at this point. In particular we claim that these extended embeddings are largely determined by a linear combination of two embedding directions. These directions correspond to input filters, this equivalence exists because we fixed the attention pattern and thus resid_mid is given by a linear combination of the embeddings. The classification is given by class 1 = Filter 1 AND Filter 2 (with some threshold numbers). We can illustrate this as binary mask using those thresholds: The left two images are filters learned by the model, and their AND-combination (3rd image) reproduces the model output to a large extent: The MLP basically just implements this AND gate, a simple non-linear transformation of the embedding into a linearly separable form. The animations below illustrates this: We test whether these PCA features we identified indeed correspond to the values of the two filters: If this hypothesis is true then randomly sampling data points to be equal ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Difficulties in making powerful aligned AI, published by DanielFilan on May 14, 2023 on The AI Alignment Forum. Here's my breakdown of the difficulties involved in ensuring powerful AI makes our lives radically better, rather than taking over the world, as well as some reasons why I think they're hard. Here are things it's not: It's not primarily a justification of why very powerful AI is possible or scary (altho it briefly discusses why very powerful AI would be scary). It's not primarily a list of underlying factors that cause these difficulties (altho it does include and gesture to some of those). It's not at all original - basically everything here has been said many times before, plausibly more eloquently. That said, it is my attempt to group the problems in my own words, in a configuration that I haven't seen before, with enough high-level motivation that one can hopefully tell the extent to which advances in the state of the art address them. 1. What sort of thinking do we want? The first difficulty: we don't have a sense of what sort of thinking we would want AI systems to use, in sufficient detail that one could (for instance) write python code to execute it. Of course, some of the difficulty here is that we don't know how smart machines think, but we can give ourselves access to subroutines like “do perfect Bayesian inference on a specified prior and likelihood” or “take a function from vectors to real numbers and find the vector that minimizes the function” and still not solve the problem. To illustrate: Take a hard-coded goal predicate, consider a bunch of plans you could take, and execute the plan that best achieves the goal? Unfortunately, the vast majority of goals you could think of writing down in an executable way will incentivize behaviour like gaining control over sources of usable energy (so that you definitely have enough to achieve your goal, and to double- and triple-check that you've really achieved it) and stopping other agents from being able to meddle with your plans (because if they could, maybe they'd stop you from achieving your goal). Do things that maximize the number of thumbs up you get from humans?1 Best plan: take control of the humans, force them to give you a thumbs up, or trick them into doing so. Presumably this is possible if you're much smarter than humans, and it's more reliable than doing good things - some people might not see why your good thing is actually good if left to their own devices. Look at humans, figure out what they want based on what they're doing, and do whatever that is? Main problem: people don't do the literally optimal thing for what they want. For instance, when people play chess, they usually don't play perfect moves - even if they're experts! You need some rule that tells you what people would do if they wanted some goal or another, but it's not clear what this rule would be, it's not clear how you make this rule more in line with reality if you never observe “wanting”, and so this ends up having essentially the same problems as plans 1 and 2. Read some text written by humans about what they'd like you to do, and do that?2 This is passing the buck to the text written by humans to specify how we want the AI to think, but that's precisely the problem we're trying to solve. Concretely, one way you could imagine doing this is to write something relatively informal like “Please be helpful and harmless to your human operators”, and have your AI correctly understand what we mean by that. That (a) presumes that there is a coherent thing that we mean by that (which doesn't seem obvious to me, given our difficulty in explicitly formalizing this request), and (b) passes the specification buck to the problem of specifying how you should understand this request. It's not a priori definitely impossible to build a ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Announcing the European Network for AI Safety (ENAIS), published by Esben Kran on March 22, 2023 on The Effective Altruism Forum. TLDR; The European Network for AI Safety is a central point for connecting researchers and community organizers in Europe with opportunities and events happening in their vicinity. Sign up here to become a member of the network, and join our launch event on Wednesday, April 5th from 19:00-20:00 CET! Why did we create ENAIS? ENAIS was founded by European AI safety researchers and field-builders who recognized the lack of interaction among various groups in the region. Our goal is to address the decentralized nature of AI safety work in Europe by improving information exchange and coordination. We focus on Europe for several reasons: a Europe-specific organization can better address local issues like the EU AI Act, foster smoother collaboration among members and the free travel within Schengen also eases event coordination. About the network ENAIS strives to advance AI Safety in Europe, mitigate risks from AI systems, particularly existential risks, and enhance collaboration among the continent's isolated AI Safety communities. We also aim to connect international communities by sharing insights about European activities and information from other hubs. We plan to offer infrastructure and support for establishing communities, coworking spaces, and assistance for independent researchers with operational needs. Concretely, we organize / create: A centralized online location for accessing European AI safety hubs and resources for field-building on the enais.co website. The map on the front page provides direct access to the most relevant links and locations across Europe for AI safety. A quarterly newsletter with updated information about what field-builders and AI safety researchers should be aware of in Continental Europe. A professional network and database of the organizations and people working on AI safety. Events and 1-1 career advice to aid transitioning into AI Safety or between different AI Safety roles. Support for people wanting to create a similar organization in other regions. We intend to leverage the expertise of the network to positively impact policy proposals in Europe (like the EU AI Act), as policymakers and technical researchers can more easily find each other. In addition, we aim to create infrastructure to make the research work of European researchers easier and more productive, for example, by helping researchers with finding an employer of records and getting funding. With the decentralized nature of ENAIS, we also invite network members to self-organize events under the ENAIS banner with support from other members. What does European AI safety currently look like? Below you will find a non-exhaustive map of cities with AI Safety researchers or organizations. The green markers indicate an AIS group, whereas the blue markers indicate individual AIS researchers or smaller groups. You are invited to add information to the map here. Vision The initial vision for ENAIS is to be the go-to access point for information and people interested in AI safety in Europe. We also want to provide a network and brand for groups and events. The longer-term strategy and vision will mostly be developed by the people who join as directors with guidance from the board. This might include projects such as policymaker communication, event coordination, regranting, community incubation, and researcher outreach. Join the network! Sign up for the network here by providing information on your interests, openness to collaboration, and location. We will include you in our database (if you previously filled in information, we will email you so you may update your information). You can choose your level of privacy to not appear publicly and only to m...

If you missed Part 1, please check that out first HERE!Part II: The Practice of Love by the Church as a “Community of Love” (19 - 39)Now that the exploration of the concept of love is finished, Pope Benedict turns his attention to the concrete. We have answered what love is, now we see how love is lived. The Church's charitable activity as a manifestation of Trinitarian love (19)Part II begins by speaking on the Church's charitable activity as a manifestation of Trinitarian love. Concretely there is nothing more *real* than the Blessed Trinity. The Godhead is a unity of three Divine Persons who are in a community of life and love: Father, Son, and Holy Spirit. The Father eternally begets and loves the Son, the perfect self-image of the Father, who eternally reflects that perfect love back to the Father. The love outpoured eternally between Father and Son is the Person of the Holy Spirit, proceeding from each eternally. If nothing I said makes any sense, that is okay! Trinitarian theology is remarkably complicated and nuanced because it is the mystery of God, as He is. Mystery though it is, God is the source of all, as Trinity. And so, how can we recognize the Trinity? The Pope quotes St. Augustine in saying: “If you see charity, you see the Trinity (19).”God created all things, seen and unseen, in a plan of sheer loving goodness. When God sent His Son to assume our humanity, He invited us, in love, to share in His divinity. We can say that the mystery of the Church is the drawing of men into the mystery of God. The Father “wishes to make humanity a single family in his Son (19).” He does this through the outpouring of the Holy Spirit who the Pope speaks of as “the energy which transforms the heart of the ecclesial community, so that it becomes a witness before the world to the love of the Father (19).” Gathering all men to Himself as the Church, the Father “seeks the integral good of man (19).” This seeking of our good, the Pope says, is an expression of love in the entire activity of the Church. By His Word and the Sacraments, the Church shares the reality of the Good News of Jesus Christ with the world. What greater love can we share with our fellow man than the work of evangelization?Charity as a responsibility of the Church (20 - 25)This outward act of love of evangelization does not end with the spiritual but also includes the corporal. Charity is a responsibility of the whole Church, clergy, religious, and laity. From the beginning, Jesus Christ established the Church upon four pillars, outlined by Acts 2:42: “And they devoted themselves to the apostles' teaching and the fellowship, to the breaking of bread and the prayers.” These four pillars give structure to the Church in every age and are reflected in the Catechism of the Council of Trent and in the Catechism of the Catholic Church. The apostolic teaching is doctrine. Fellowship, also translated as communal life, is the moral life in Christ. The breaking of the bread is the early term for the Eucharist and for the Sacraments generally. And prayer is Christian prayer, perfectly expressed in the Our Father which our Lord gave us.Pope Benedict gives attention in paragraph 20 of Deus Caritas Est to this idea of fellowship, communion, or communal life. The Greek word in Acts is koinonia. Koinonia “consists in the fact that believers hold all things in common and that among them, there is no longer any distinction between rich and poor (20).” This sort of radical communal life is part and parcel of early Christianity. But as the Pope remarks:“As the Church grew, this radical form of material communion could not in fact be preserved. But its essential core remained: within the community of believers there can never be room for a poverty that denies anyone what is needed for a dignified life (20).”When it became necessary, the Church instituted the clerical office of the Diaconate, the first level of participation in the priesthood of Jesus Christ and the lowest level of the Sacrament of Holy Orders. A diakonos was historically a servant of the king. And so, the Deacon, is a servant of Christ the King and his vicar in a particular diocese, the bishop. They were to provide for the spiritual and corporal needs of the people. As Pope Benedict puts it:“... the social which they were meant to provide was absolutely concrete, yet at the same time it was also a spiritual service; theirs was a truly spiritual office which carried out an essential responsibility of the Church, namely a well-ordered love of neighbour (21).”This system of charity must have been a formidable force in the ancient world because the Roman leader Julian the Apostate, who rejected Christianity and tried to instantiate Neoplatonic Hellenism, told his pagan priests that they needed to imitate and outdo the Church's charity. Of course, they failed because imitation can never capture what is authentically of God.Beyond the diaconate, the proclamation of the Good News, and the liturgy were indispensable to the Church. As the Pope puts it:“The Church's deepest nature is expressed in her three-fold responsibility: of proclaiming the word of God (kerygma-martyria), celebrating the sacraments (leitourgia), and exercising the ministry of charity (diakonia). These duties presuppose each other and are inseparable (25).”Though the Church is now worldwide and the radical form of community of the early Church is less possible, the Church is nonetheless responsible for Her Members. The needs of the People of God are not only spiritual. Like any family, the Church is obliged to care for the necessities of those in the Church, in a particular way. As Benedict says:“The Church is God's family in the world. In this family no one ought to go without the necessities of life… Without in any way detracting from this commandment of universal love, the Church also has a specific responsibility: within the ecclesial family no member should suffer through being in need (25).”Justice and Charity (26 - 29)After concluding the section on the responsibility of charity in the Church, the Pope moves on to a fairly expansive conversation of justice and charity. Rooted deeply in Catholic social teaching, expounded since Pope Leo XIII especially, Pope Benedict begins by addressing the elephant in the room: Karl Marx. Since the nineteenth century, Marxists have critiqued the Church saying that “the poor… do not need charity but justice (26).” Benedict offers a steel-man explanation of the Marxist critique, saying:“Works of charity—almsgiving—are in effect a way for the rich to shirk their obligation to work for justice and a means of soothing their consciences, while preserving their own status and robbing the poor of their rights. Instead of contributing through individual works of charity to maintaining the status quo, we need to build a just social order in which all receive their share of the world's goods and no longer have to depend on charity (26).”He admits that there is some merit to the argument, but there is much that is mistaken. Historically, capital became concentrated in the hands of a powerful few and there has been conflict between employer and laborer. But rather than succumbing the Hegelian notion which Marx put forward of class warfare and revolution, the Pope offers that Catholic social teaching is applicable beyond the confines of the Church, saying:“Marxism had seen world revolution and its preliminaries as the panacea for the social problem: revolution and the subsequent collectivization of the means of production, so it was claimed, would immediately change things for the better. This illusion has vanished. In today's complex situation, not least because of the growth of a globalized economy, the Church's social doctrine has become a set of fundamental guidelines offering approaches that are valid even beyond the confines of the Church: in the face of ongoing development these guidelines need to be addressed in the context of dialogue with all those seriously concerned for humanity and for the world in which we live (27).”Commitment to Justice and Ministry of CharityThe Church, and the world, has a necessary commitment to justice and the ministry of charity. There is no dichotomy between justice and charity. Both are necessary. First, “the just ordering of society and the State is a central responsibility of politics (28),” says the Pope. Quoting St. Augustine, he says: “a State which is not governed according to justice would be just a bunch of thieves… (28).”Venturing into the subject of freedom of religion in a State, Pope Benedict says that,“The State may not impose religion, yet it must guarantee religious freedom and harmony between the followers of different religions. For her part, the Church, as the social expression of Christian faith, has a proper independence and is structured on the basis of her faith as a community which the State must recognize. The two spheres are distinct, yet always interrelated (28).”In other words: the State cannot coerce the Church. Rather the Church is independent of the State but interrelated with the State. What faithful Catholic can switch off their Catholicism when they engage in matters of politics and the State? They cannot. Only unfaithful Catholics attempt this, which results in deadly scandal and sin. Why is this? It is because God is the Lawgiver; He alone created all that is and He alone is the arbiter of morality.Justice is the aim of politics, properly understood and it is the criteria for good politics. Politics is not merely about rules governing public life: “its origin and its goal,” say the Pope, “are found in justice, which by its very nature has to do with ethics (28).”If justice is the aim and internal criterion of politics, as Benedict says, then why do Catholics need anything beyond reason? Why is faith necessary for true justice? Benedict explains:“Faith by its specific nature is an encounter with the living God—an encounter opening up new horizons extending beyond the sphere of reason. But it is also a purifying force for reason itself. From God's standpoint, faith liberates reason from its blind spots and therefore helps it to be ever more fully itself. Faith enables reason to do its work more effectively and to see its proper object more clearly… Its aim is simply to help purify reason and to contribute, here and now, to the acknowledgment and attainment of what is just (28).”If Faith is necessary, then is the Pope calling on the Church to engage directly in matters of State? Not quite. A just social and civil order is ordered towards each person receiving his or her due, which is an essential task in every generation. But it is a political task and therefore is a human responsibility rather than an ecclesial responsibility. The Church can help to purify our powers of reason and provide ethical formation. Benedict also says clearly, “The Church cannot and must not take upon herself the political battle to bring about the most just society possible. She cannot and must not replace the State. Yet at the same time she cannot and must not remain on the sidelines in the fight for justice (28).”The clergy are called to sanctify the laity and the laity are sent out into the world to sanctify the temporal order. Love is NecessaryEven in the best, most just societies, love will always be necessary. Care and concern for the other will always happen best on the local level. This is the principle of subsidiarity which holds that the best decisions are made on the lowest possible level and the highest level necessary. In fact, injustices stem from a higher level claiming authority over something which in fact belongs naturally to someone personally closer to the situation. Against totalitarianism of this kind, Pope Benedict writes beautifully of the bigger, complicated picture:“We do not need a State which regulates and controls everything, but a State which, in accordance with the principle of subsidiarity, generously acknowledges and supports initiatives arising from the different social forces and combines spontaneity with closeness to those in need. The Church is one of those living forces: she is alive with the love enkindled by the Spirit of Christ. This love does not simply offer people material help, but refreshment and care for their souls, something which often is even more necessary than material support. In the end, the claim that just social structures would make works of charity superfluous masks a materialist conception of man: the mistaken notion that man can live “by bread alone” (Mt 4:4; cf. Dt 8:3)—a conviction that demeans man and ultimately disregards all that is specifically human (28).”As I mentioned earlier, the clergy are called to sanctify the laity and the laity are sent out into the world to sanctify the temporal order. In this way, the Pope says:“The direct duty to work for a just ordering of society, on the other hand, is proper to the lay faithful. As citizens of the State, they are called to take part in public life in a personal capacity… The mission of the lay faithful is therefore to configure social life correctly, respecting its legitimate autonomy and cooperating with other citizens according to their respective competences and fulfilling their own responsibility (29).” The Church, of course, still sponsors organizations which practice charity. But this is not merely an activity of justice because it is focused on the admixture of the love of God, which is a universal human need. As Benedict says,“The Church's charitable organizations, on the other hand, constitute an opus proprium, a task agreeable to her, in which she does not cooperate collaterally, but acts as a subject with direct responsibility, doing what corresponds to her nature. The Church can never be exempted from practicing charity as an organized activity of believers, and on the other hand, there will never be a situation where the charity of each individual Christian is unnecessary, because in addition to justice man needs, and will always need, love (29).”The multiple structures of charitable service in the social context of the present day (30)Because there is never going to be a situation where individual Christians are unneeded in charitable service, Pope Benedict recognizes the social context of the present day. Long since gone is the time of Christendom when Christian kingdoms ruled. Now, there are many different types of governments, organizations, nonprofits, and social structures. Even our way of communicating is different now. As the Pope says, “Mass communication has narrowed the distance between peoples and cultures (30).” Despite the narrowing of the distance, there is still work to be done in making sure that no one is forgotten. These groups are diverse, but they all are marked with the love of God. The Pope remarks that, “Numerous organizations have arisen that are a cooperation of State and Church, but these agencies still have a Christian quality (30).” This Christian quality is not a vague echo of the love of Jesus Christ. Rather, it is a direct result of Church agencies cooperating with State agencies. Also, due to the growing amount of leisure time afforded to modern man, especially among the youth, the Pope says, “our time has also seen a growth and spread of different kinds of volunteer work, which assume responsibility for providing a variety of services (30).” What is clear is that there are a myriad of ways to get involved; however, love compels us to leave the sidelines and get in the game.In non-Catholic Churches and Ecclesial Communities, there has also been a new rising of charitable activity, with new life and energy. The Catholic Church must have a readiness, Pope Benedict says, to:“cooperate with the charitable agencies of these Churches and Communities, since we all have the same fundamental motivation and look towards the same goal: a true humanism, which acknowledges that man is made in the image of God and wants to help him to live in a way consonant with that dignity (30).”Ideally, all Christians and all people of good will would work with a united voice to inculcate, as Pope Benedict says, quoting St. John Paul II in Ut Unum Sint, “respect for the rights and needs of everyone, especially the poor, the lowly and the defenseless (30).”Thank you for reading Will Wright Catholic. This post is public so feel free to share it.The distinctiveness of the Church's charitable activity (31)After speaking on the interplay of Church and State cooperation, Pope Benedict then turns his attention to the distinctiveness of the Church's charitable activity. The Pontiff explains that there are a few essential elements of Christian and ecclesial charity.First, there must be a simple response to immediate needs and specific situations. Second, and related to the first, resources and personnel needed for the work must be provided. Though Pope Benedict does not mention subsidiarity here by name, I think it is worth mentioning. Subsidiarity is the principle of Catholic social teaching which says that the best decision is made at the highest level necessary and the lowest level possible. Why leave a decision best left to the local Parish to the Vatican, for example? The local communities all need to strive to care for those in their immediate vicinity.The third essential element of Christian and ecclesial charity is that individuals who care for those in need must be professionally competent and properly trained. As the Pope explains:“We are dealing with human beings, and human beings always need something more than technically proper care. They need humanity. They need heartfelt concern. Those who work for the Church's charitable organizations must be distinguished by the fact that they do not merely meet the needs of the moment, but they dedicate themselves to others with heartfelt concern, enabling them to experience the richness of their humanity… Consequently, in addition to their necessary professional training, these charity workers need a “formation of the heart”: they need to be led to that encounter with God in Christ which awakens their love and opens their spirits to others. As a result, love of neighbor will no longer be for them a commandment imposed, so to speak, from without, but a consequence deriving from their faith, a faith which becomes active through love (cf. Gal 5:6) (31).”The fourth essential element is that Christian charitable activity must be independent of parties and ideologies. Charitable activity is not a useful means to a longer end goal. It is not done to change the world ideologically, nor is at “at the service of worldly strategems,” says Pope Benedict, “but is a way of making present here and now the love which man always needs (31).” As a Church of Christ rather than a ‘church of causes,' we must follow the “program of Jesus” which is a “a heart which sees (31).” The Pope continues:“This heart sees where love is needed and acts accordingly. Obviously when charitable activity is carried out by the Church as a communitarian initiative, the spontaneity of individuals must be combined with planning, foresight and cooperation with other similar institutions (31).”Fifth and finally, charity cannot be used as a means of engaging in proselytism. Pope Benedict clearly states:“Love is free; it is not practiced as a way of achieving other ends. But this does not mean that charitable activity must somehow leave God and Christ aside. For it is always concerned with the whole man. Often the deepest cause of suffering is the very absence of God… A Christian knows when it is time to speak of God and when it is better to say nothing and to let love alone speak (31).”Having given these five essential elements of Christian charity, who is responsible for the Church's charitable activity?Those responsible for the Church's charitable activity (32 - 39)Charity is such a deep action of the Church that it is part of Her identity. So, those responsible for charitable activity are the whole Church: bishops, priests, deacons, lay, and religious. In regard to bishops, Pope Benedict says the following:“In the rite of episcopal ordination, prior to the act of consecration itself, the candidate must respond to several questions which express the essential elements of his office and recall the duties of his future ministry. He promises expressly to be, in the Lord's name, welcoming and merciful to the poor and to all those in need of consolation and assistance. The Code of Canon Law, in the canons on the ministry of the Bishop, does not expressly mention charity as a specific sector of episcopal activity, but speaks in general terms of the Bishop's responsibility for coordinating the different works of the apostolate with due regard for their proper character (32).”He goes on to say once again that no one is off the hook from doing charitable work. Okay, he does not put it quite that way. He says this:“With regard to the personnel who carry out the Church's charitable activity on the practical level, the essential has already been said: they must not be inspired by ideologies aimed at improving the world, but should rather be guided by the faith which works through love… Consequently, more than anything, they must be persons moved by Christ's love, persons whose hearts Christ has conquered with his love, awakening within them a love of neighbor (33).”Of course it is always the love of Christ which marks our meritorious work. Charity inflames us to do good, in and through Christ, apart from Whom we can do nothing of true merit. Christ came to redeem the whole world and God loves each man and woman. Interior openness to Christ and His love is what makes the service of Christ's disciples so distinctive. St. Paul's hymn to charity in 1 Cor. 13 teaches us that service is more than activity alone: “If I give away all I have, and if I deliver my body to be burned, but do not have love, I gain nothing (1 Cor. 13:3).” Pope Benedict refers to this hymn as the “Magna Carta of all ecclesial service (34)” and the summary of all the reflections on love which he offers in Deus Caritas Est.He goes on to say:“Practical activity will always be insufficient, unless it visibly expresses a love for man, a love nourished by an encounter with Christ. My deep personal sharing in the needs and sufferings of others becomes a sharing of my very self with them: if my gift is not to prove a source of humiliation, I must give to others not only something that is my own, but my very self; I must be personally present in my gift (34).”When we serve in this self-giving way, we learn humility and grow in humility. As Pope Benedict beautifully reminds us:“We recognize that we are not acting on the basis of any superiority or greater personal efficiency, but because the Lord has graciously enabled us to do so… We offer him our service only to the extent that we can, and for as long as he grants us the strength. To do all we can with what strength we have, however, is the task which keeps the good servant of Jesus Christ always at work: ‘The love of Christ urges us on' (2 Cor 5:14) (35).”Finding the BalanceUrged on by the love of Christ, it is easy to lose balance. “When we consider the immensity of others' needs, we can, on the one hand,” Pope Benedict says, “be driven towards an ideology that would aim at doing what God's governance of the world apparently cannot: fully resolving every problem. Or we can be tempted to give in to inertia, since it would seem that in any event nothing can be accomplished (36).”So, how can we overcome these temptations? How can we find balance and keep the seesaw from tipping completely one way or the other? Pope Benedict continues:“At such times, a living relationship with Christ is decisive if we are to keep on the right path, without falling into an arrogant contempt for man, something not only unconstructive but actually destructive, or surrendering to a resignation which would prevent us from being guided by love in the service of others. Prayer, as a means of drawing ever new strength from Christ, is concretely and urgently needed. People who pray are not wasting their time, even though the situation appears desperate and seems to call for action alone (36).”Personal Relationship with GodPrayer is vital if we are to live in and with God. Prayer is our life blood. Our entire life can become a prayer if continually drawn deeply from the well of Christ. With the scourge of secularism prowling and the misguided extreme arm of activism, Christians are engaged in charitable work. Prayer is the antidote to this worldly spirit. A personal relationship with our loving Father, through Christ, in the Holy Spirit is utterly necessary. Pope Benedict puts it this way:“Clearly, the Christian who prays does not claim to be able to change God's plans or correct what he has foreseen. Rather, he seeks an encounter with the Father of Jesus Christ, asking God to be present with the consolation of the Spirit to him and his work (37).”The Problem of EvilThis personal relationship with God is all the more necessary for us in the midst of the problem of evil. We can often experience bewilderment and fail to understand the world around us. In these moments, the Pope says:“Christians continue to believe in the ‘goodness and loving kindness of God' (Tit 3:4). Immersed like everyone else in the dramatic complexity of historical events, they remain unshakably certain that God is our Father and loves us, even when his silence remains incomprehensible (38).”Faith, Hope, and CharityIn the midst of darkness, we trust in the love of God which surpasses all understanding. The theological virtues of faith, hope, and charity, God's own life, have been instilled in our soul through the gift of Baptism. These theological virtues go together. Pope Benedict teaches us:“Hope is practiced through the virtue of patience, which continues to do good even in the face of apparent failure, and through the virtue of humility, which accepts God's mystery and trusts him even at times of darkness. Faith tells us that God has given his Son for our sake and gives us the victorious certainty that it is really true: God is love! It thus transforms our impatience and our doubts into the sure hope that God holds the world in his hands and that, as the dramatic imagery of the end of the Book of Revelation points out, in spite of all darkness he ultimately triumphs in glory (39).”We cannot see the full picture; only God does. And He has revealed so much to us. More than revelation, He has given us Himself. He shares in our humanity that we might share in His divinity. In Baptism, we are given the light and made to be the light to the world, in and through Christ. Pope Benedict sums up his first encyclical this way:“Faith, which sees the love of God revealed in the pierced heart of Jesus on the Cross, gives rise to love. Love is the light—and in the end, the only light—that can always illuminate a world grown dim and give us the courage needed to keep living and working. Love is possible, and we are able to practice it because we are created in the image of God. To experience love and in this way to cause the light of God to enter into the world—this is the invitation I would like to extend with the present Encyclical (39).”Conclusion (40 - 42)As was the practice with his predecessor, Pope Benedict dedicates the conclusion of his document to the Mother of God, Mary most holy. Truly, each of the saints are a beautiful witness to the charity of God in every way imaginable. However, there is a preeminence to the holy charity of our Blessed Mother, the first disciple of her Holy Son, Jesus Christ.I highly recommend reading the entire Conclusion (and the whole document, really) in full. But I would like to draw what resonated most with me. I simply love the way Pope Benedict speaks about our Blessed Mother. He says:“Mary's greatness consists in the fact that she wants to magnify God, not herself. She is lowly: her only desire is to be the handmaid of the Lord (cf. Lk 1:38, 48) She knows that she will only contribute to the salvation of the world if, rather than carrying out her own projects, she places herself completely at the disposal of God's initiatives (41).” How often do you and I carry out our own projects, without putting ourselves at the disposal of God's initiatives. Mary, form us, teach us! The Pope goes on:“Mary is a woman who loves. How could it be otherwise? As a believer who in faith thinks with God's thoughts and wills with God's will, she cannot fail to be a woman who loves. We sense this in her quiet gestures, as recounted by the infancy narratives in the Gospel. We see it in the delicacy with which she recognizes the need of the spouses at Cana and makes it known to Jesus. We see it in the humility with which she recedes into the background during Jesus' public life, knowing that the Son must establish a new family and that the Mother's hour will come only with the Cross, which will be Jesus' true hour (cf. Jn 2:4; 13:1). When the disciples flee, Mary will remain beneath the Cross (cf. Jn 19:25-27); later, at the hour of Pentecost, it will be they who gather around her as they wait for the Holy Spirit (cf. Acts 1:14) (41).”Mary was there through it all. By her “yes” the Word of God took flesh, provided by her own body. Even despite her prominence and grandeur, the humility of the Theotokos is led by love. She loves with the love of God flowing through her as a perfect vessel and she loves with a human, motherly love. The Pope goes on:“Mary, Virgin and Mother, shows us what love is and whence it draws its origin and its constantly renewed power. To her we entrust the Church and her mission in the service of love (42).”Here Pope Benedict ends with a prayer to the Theotokos, which I would like to end today's article with as well. Please join me in praying:“Holy Mary, Mother of God, you have given the world its true light, Jesus, your Son – the Son of God. You abandoned yourself completely to God's call and thus became a wellspring of the goodness which flows forth from him. Show us Jesus. Lead us to him. Teach us to know and love him, so that we too can become capable of true love and be fountains of living water in the midst of a thirsting world (42).”Amen. Get full access to Will Wright Catholic Podcast at www.willwrightcatholic.com/subscribe

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What are the best examples of object-level work that was done by (or at least inspired by) the longtermist EA community that concretely and legibly reduced existential risk?, published by Ben Snodin on February 11, 2023 on The Effective Altruism Forum. A motivating scenario could be: imagine you are trying to provide examples to help convince a skeptical friend that it is in fact possible to positively change the long-run future by actively seeking and pursuing opportunities to reduce existential risk. Examples of things that are kind of close but miss the mark There are probably decent historical examples where people reduced existential risk but where thoes people didn't really have longtermist-EA-type motivations (maybe more "generally wanting to do good" plus "in the right place at the right time") There are probably meta-level things that longtermist EA community members can take credit for (e.g. "get lots of people to think seriously about reducing x risk"), but these aren't very object-level or concrete Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Experiment Idea: RL Agents Evading Learned Shutdownability, published by Leon Lang on January 16, 2023 on The AI Alignment Forum. Preface Produced as part of the SERI ML Alignment Theory Scholars Program - Winter 2022 Cohort. Thanks to Erik Jenner who explained to me the basic intuition for why an advanced RL agent may evade the discussed corrigibility measure. I also thank Alex Turner, Magdalena Wache, and Walter Laurito for detailed feedback on the proposal and Quintin Pope and Lisa Thiergart for helpful feedback in the last December SERI-MATS shard theory group meeting. This text was part of my deliverable for the shard theory stream of SERI-MATS. In it, I present an idea for an experiment that tests the convergent drive of modern model-based RL agents to evade shutdownability. If successful, I expect the project could serve as a means to communicate the problem of corrigibility to the machine learning community. As such, I also consider this project idea a submission of the Shutdown Problem Contest. I do not personally want to work on the project since doing experiments does not seem like my comparative advantage. Thus, with posting this project, I mainly seek collaborators and feedback. More on that in the conclusion. Introduction At some point in the future, we will train very advanced AI, possibly by some version of model-based reinforcement learning. Since it is hard to correctly specify the reward function or steer the inner learning dynamics of the agent, it may be desirable to make sure that the agent is corrigible, including shutdownability: “(1) A corrigible reasoner must at least tolerate and preferably assist the programmers in their attempts to alter or turn off the system.” Next to a literal shutdown button, one could also try to use the inner functions of the agent to effectively get the ability to shut it down. The idea would be to design a certain type of input such that, as a reaction to that input, a computation in the agent itself will cause it to perform a null action that doesn't do anything — which effectively equates to a shutdown. Scott Aaronson proposed this idea in a recent talk (timestamp: 57:10) in the form of a planted cryptographic backdoor, an idea he currently works on during his sabbatical at OpenAI. For this proposal, I consider a variation of that shutdown approach: instead of directly “planting” the shutdown behavior, I consider the possibility to train the behavior into the agent. Concretely, there is a specific input — here, an alert sound — that the agent receives at random times during training. Whenever that happens, the agent will receive a negative reward unless it performs the null action. Over time, the agent will learn to stop acting in these situations. During deployment, the input can then be shown to the agent whenever it performs harmful actions, which will ideally cause it to stop immediately. This shutdown approach has the following philosophical problem: if the agent performs world modeling, it will know that the alert sound will make it stop performing, and this may be counter to the more consequentialist motivations of the agent. Thus, we should expect that the agent will perform an action that makes it never hear the sound again — for example, by “putting in earplugs” — once it is advanced enough and has the ability to do so. This experimental proposal is meant to show this convergent behavior using contemporary model-based reinforcement learning. The remaining sections will discuss the following: A three-part training story for an advanced AI that evades corrigibility, including details for how to model the situation with contemporary reinforcement learning; More experimental details based on using MuZero; Caveats, what might go wrong; A short comparison to the original shutdown problem and how the proposal...

In this episode, Claudia Miller, Career Coach, and Ulrike share concrete strategies on how to prepare for and obtain substantial salary increases. If you want to give your career a serious financial boost, this episode is for you!You will find out: How to get over hesitation and self-doubt How to set yourself up for success when it comes to searching for a new job How successfully lead the conversation to negotiate a significant increase Want to get proven techniques on how to flourish and land that leadership spot you're after? Get your free e-book “Top 10 Achievers' Lessons” by clicking here!About Claudia MillerClaudia Miller is a sought-after Career Coach and she's helped her clients land fulfilling jobs in less than 90-days all while getting on average a 54% in salary increases ($30k-$140k). She also partners with companies and organizations in identifying rising stars within their organizations and providing strategic insights and support in developing a leadership and talent pipeline with a focus on Women. Due to her efforts, she's worked with Top Fortune 500 Clients and she's been featured multiple times in Forbes, MSNBC and Thrive Global.  Business Insider put her in their top global list of Top Innovative Career Coaches.About Ulrike SeminatiUlrike Seminati is a long-standing senior executive, coach and author. After over 20 years of being a successful leader in corporate organizations, Ulrike believes that female leadership is one of the biggest untapped potentials in the world. Ulrike is known for combining all her experience and years of excelling in the corporate world with powerful self-development techniques that she gives it to hard workers like you, allowing you to flourish, realize the root causes of your struggle, and finally land that leadership spot you're after!Outline of the Episode:[01:56] Where to start if you want to land your dream job[05:14] Get unstuck despite a high level of self-criticism[08:57] Overcoming the inner threshold of asking for a salary increase[14:29] Building the business case to convince your future employer[15:58] Properly preparing the job interview to ask for the right salary[18:44] Breaking the inner glass ceiling and being able to ask for more[22:56] Concretely preparing your next job interviewResources of Claudia Miller:Website: https://www.claudiatmiller.com/Linkedin profile: https://www.linkedin.com/in/claudiatmiller/Get Claudia's $10k Word-For-Word Salary Negotiation Script by clicking here:https://keap.page/fxp705/10k-salary-negotiation-script.htmlConnect with Ulrike!Website: https://ulrikeseminati.com Instagram https://www.instagram.com/empoweringfemaleleaders/   LinkedIn: https://www.linkedin.com/in/ulrikeseminati/Email: contact@ulrikeseminati.com

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Underspecification of Oracle AI, published by Rubi J. Hudson on January 15, 2023 on The AI Alignment Forum. Rubi and Johannes worked on this post as part of the SERI MATS program, with Evan Hubinger providing mentorship to both. Rubi also received mentorship from Leo Gao. Thanks to Paul Colognese and Nicholas Schiefer for discussions related to this post. An oracle is a type of AI system that only answers questions without taking any other actions in the world. Simulators and generative models, which have seen increased discussion recently (links: 1, 2, 3, 4), can be thought of as types of oracles. Such systems may simultaneously be powerful enough to generate a pivotal act while also being easier to align due to a more limited domain. One major concern with oracles is that the answers they give can still manipulate the world. If oracles are evaluated on predictive accuracy, this gives them an incentive to use their answers to affect the course of events and make the world more predictable. Concretely, we are concerned that oracles may make self-fulfilling prophecies (also known as self-confirming predictions or fixed points), where the act of making the prediction causes the predicted outcome to come true. Even if their manipulation does not terminate in a fixed point, attempts to influence the world towards predictability can be very dangerous. As one example, consider a highly trusted oracle asked to predict the stock market. If such an oracle predicts that stock prices will rise, then people buy based off that prediction and the price will in fact rise. Similarly, if the oracle predicts that prices will fall, then people will sell, causing prices to fall. For a more real world example, see this market and this market, each on whether a specific person will find a research/romantic partner. Here, high probabilities would indicate desirability of that person, while low probabilities would suggest some hidden flaw, either of which could influence whether potential partners decide to reach out and therefore how the market resolves. In both the stock market and partnership cases, multiple predictions are valid, so how does the oracle choose between them? Ideally, we would like it to choose the one that is “better” for humanity, but this now introduces an outer alignment question similar to an agentic AI acting directly on the world, and which we wanted to avoid by using oracles in the first place. Instead, what we can aim for is an oracle that does not take into account the consequences of the prediction it makes when choosing a prediction. Then, there is only one valid prediction for the oracle to make, since the rest of the world is constant from its perspective. This can be thought of as a type of partial agency, optimizing the prediction in some directions but not others. It would be extremely desirable as a safety property, removing all incentives for an oracle to manipulate the world. To emphasize the importance of this property, we introduce new terminology, dubbing oracles “consequence-blind” if they exhibit the desired behavior and “consequence-aware” if they do not. For an oracle, consequence-blindness is equivalent to following a lonely causal decision theory. The causal decision theory blinds the oracle to any acausal influence, while the loneliness component makes it blind to its influence on other agents, which are necessary intermediaries for a prediction to influence the world. In this post we will primarily consider an oracle trained via supervised learning on a historical data set. There are a number of different policies that could be learned which minimize loss on the training set, and we will explore the different ways these can generalize. We divide the generalization behavior into a number of different axes, and for each axis discuss the potenti...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Trying to isolate objectives: approaches toward high-level interpretability, published by Arun Jose on January 9, 2023 on The AI Alignment Forum. Epistemic status: There's a lot in this post and my general approach while working on it that, in retrospect, wasn't thought out well enough. I'm posting it because I figure sharing flawed ideas is better than letting this languish in a doc forever, while I work on improving them. Thanks to Paul Colognese, Tassilo Neubauer, John Wentworth, and Fabien Roger for useful conversations, Kyle McDonell and Laria Reynolds for suggesting something that led me to some of the ideas I mention here, and Shoshannah Tekofsky for feedback on a draft. I spent some time trying to find the high-level structure in a neural net corresponding to a deep reinforcement learning model's objective. In this post, I describe some of the stuff I tried, thoughts from working on this, and mistakes I made. Context Many approaches in current interpretability work as I understand them involve understanding low-level components of a model (Circuits and subsequent work, for example), as a way of building up to more complex components. I think we can make progress from the opposite frontier at the same time, and try to identify the presence or nature of certain high-level mechanistic structures or properties in a model. Examples of this include verifying whether a model is doing optimization, whether it's myopic, isolating the objective of an optimizer, and essentially the entire class of properties with a singular answer for the entire model. I prefer to distinguish these directions as low-level and high-level interpretability respectively for descriptive clarity. The framing of best-case and worst-case transparency as laid out in the transparency tech tree also points at the same concept. I expect that both directions are aiming at the same goal, but working top-down seems pretty tractable at least for some high-level targets that don't require lots of deconfusion to even know what we're looking for. Ideally, an approach to high-level interpretability would be robust to optimization pressure and deception (in other words, to gradient descent and adversarial mesa optimizers), but I think there's a lot we can learn toward that ideal from more naive approaches. In this post, I describe my thoughts on trying to identify the high-level structure in a network corresponding to a deep reinforcement learning model's objective. So I didn't set out on this particular approach expecting to succeed at the overall goal (I don't expect interpretability to be that easy), instead hoping that trying out a bunch of naive directions will give us insight into future high-level interpretability work. To that end, I try to lay out my reasoning at various points, riddled with gaps at points, while thinking about this. I think there are plenty of plausible avenues to try out here, and for the most part will only be describing the problems with ones I thought of. Further, the directions I tried don't seem very non-obvious and there are likely much better methods to be tried given more thought here. A naive approach and patching Take the case of a small RL model that can optimize for a reward in a given environment. What we want is information about how the model's objective is represented in the network's weights. Concretely, we might think about what kind of tests or processes disproportionately affect these weights more than any other, and try to isolate the information we want from there. One approach we might try is: Train the initialized model on reward RA for an extended period of time. Let's call the model at the end of this step MA. We may think of MA as having learned a pretty sophisticated world model at this point, such that further training wouldn't update it strongly. Tra...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: 200 COP in MI: Exploring Polysemanticity and Superposition, published by Neel Nanda on January 3, 2023 on The AI Alignment Forum. This is the fifth post in a sequence called 200 Concrete Open Problems in Mechanistic Interpretability. Start here, then read in any order. If you want to learn the basics before you think about open problems, check out my post on getting started. I'll make another post every 1-2 days, giving a new category of open problems. If you want to read ahead, check out the draft sequence here! Motivating paper: Toy Models of Superposition, Softmax Linear Units Background If you're familiar with polysemanticity and superposition, skip to Motivation or Problems. Neural networks are very high dimensional objects, in both their parameters and their activations. One of the key challenges in Mechanistic Interpretability is to somehow resolve the curse of dimensionality, and to break them down into lower dimensional objects that can be understood (semi-)independently. Our current best understanding of models is that, internally, they compute features: specific properties of the input, like "this token is a verb" or "this is a number that describes a group of people" or "this part of the image represents a car wheel". That early in the model there are simpler features, are later used to compute more complex features by being connected up in a circuit (example shown above (source)). Further, our guess is that features correspond to directions in activation space. That is, for any feature that the model represents, there is some vector corresponding to it. And if we dot product the model's activations with that vector, we get out a number representing whether that feature is present.(these are known as decomposable, linear representations) This is an extremely useful thing to be true about a model! An even more helpful thing to be true would be if neurons correspond to features (ie the output of an activation function like ReLU). Naively, this is natural for the model to do, because a non-linearity like ReLU acts element-wise - each neuron's activation is computed independently (this is an example of a privileged basis). Concretely, if a neuron can represent feature A or feature B, then that neuron will fire differently for feature A and NOT feature B, vs feature A and feature B, meaning that the presence of B interferes with the ability to compute A. But if each feature is its own neuron we're fine! If features correspond to neurons, we're playing interpretability on easy mode - we can focus on just figuring out which feature corresponds to each neuron. In theory we could even show that a feature is not present by verifying that it's not present in each neuron! However, reality is not as nice as this convenient story. A countervailing force is the phenomena of superposition. Superposition is when a network represents more features than it has dimensions, and squashes them all into a lower dimensional space. You can think of superposition as the model simulating a larger model. Anthropic's Toy Models of Superposition paper is a great exploration of this. They build a toy model that learns to use superposition (notably different from a toy language model!). The model starts with a bunch of independently varying features, needs to compress these to a low dimensional space, and then is trained to recover each feature from the compressed mess. And it turns out that it does learn to use superposition! Specifically, it makes sense to use superposition for sufficiently rare (sparse) features, if we give it non-linearities to clean up interference. Further, the use of superposition can be modelled as a trade-off between the costs of interference, and the benefits of representing more features. And digging further into their toy models, they find all kinds of fascin...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: 200 COP in MI: Interpreting Algorithmic Problems, published by Neel Nanda on December 31, 2022 on The AI Alignment Forum. This is the fourth post in a sequence called 200 Concrete Open Problems in Mechanistic Interpretability. Start here, then read in any order. If you want to learn the basics before you think about open problems, check out my post on getting started. I'll make another post every 1-2 days, giving a new category of open problems. If you want to read ahead, check out the draft sequence here! Motivation Motivating paper: A Mechanistic Interpretability Analysis of Grokking When models are trained on synthetic, algorithmic tasks, they often learn to do some clean, interpretable computation inside. Choosing a suitable task and trying to reverse engineer a model can be a rich area of interesting circuits to interpret! In some sense, this is interpretability on easy mode - the model is normally trained on a single task (unlike language models, which need to learn everything about language!), we know the exact ground truth about the data and optimal solution, and the models are tiny. So why care? I consider my work on grokking to be an interesting case study of this work going well. Grokking (shown below) is a mysterious phenomena where, when small models are trained on algorithmic tasks (eg modular addition or modular division), they initially memorise the training data. But when they keep being trained on that data for a really long time, the model suddenly(ish) figures out how to generalise! In my work, I simplified their setup even further, by training a 1 Layer transformer (with no LayerNorm or biases) to do modular addition and reverse engineered the weights to understand what was going on. And it turned out to be doing a funky trig-based algorithm (shown below), where the numbers are converted to frequencies with a memorised Discrete Fourier Transform, added using trig identities, and converted back to the answer! Using this, we looked inside the model and identified that despite seeming to have plateaued, in the period between memorising and "grokking", the model is actually slowly forming the circuit that does generalise. But so long as the model still has the memorising circuit, this adds too much noise to have good test loss. Grokking occurs when the generalising circuit is so strong that the model decides to "clean-up" the memorising circuit, and "uncovers" the mature generalising circuit beneath, and suddenly gets good test performance. OK, so I just took this as an excuse to explain my paper to you. Why should you care? I think that the general lesson from this, that I'm excited to see applied elsewhere, is using toy algorithmic models to analyse a phenomena we're confused about. Concretely, given a confusing phenomena like grokking, I'd advocate the following strategy: Simplify to the minimal setting that exhibits the phenomena, yet is complex enough to be interesting Reverse-engineer the resulting model, in as much detail as you can Extrapolate the insights you've learned from the reverse-engineered model - what are the broad insights you've learned? What do you expect to generalise? Can you form any automated tests to detect the circuits you've found, or any of their motifs? Verify by looking at other examples of the phenomena and seeing whether these insights actually hold (larger models, different tasks, even just earlier checkpoints of the model or different random seeds) Grokking is an example in a science of deep learning context - trying to uncover mysteries about how models learn and behave. But this same philosophy also applies to understanding confusing phenomena in language models, and building toy algorithmic problems to study those! Anthropic's Toy Models of Superposition is an excellent example of this done well, for the case of ...

Matt Coffman is one of my best and oldest friends. It's actually hilarious that it took 44 interviews to get him on the show, and my intention – with you as my witness, dear listener – is that there will never been a gap that long again.   For this conversation, I wanted to flip our normal narrative about purpose on its head, and get into the question of what is stopping the humans (you, me, everyone) from living our purpose.   True to form, Matt starts at the beginning. The preconditions for purpose. And then methodically investigates all the barriers, from belief onwards, towards somebody living their purpose. It's a create episode full of concrete examples from Matt's lived experience.    10,000 Heroes. 10,000 Purposes. 10,000 Gifts.   Enjoy ;)   Show Links: Voicemail:  https://www.speakpipe.com/10khshow Email: info@10kh.show Podcast website: http://www.10kh.show Momentum Lab: http://www.momentumlab.com   Guest References: Matt's not big on online presence, but the coaching program we run does have a website: www.momentumlab.com

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Re-Examining LayerNorm, published by Eric Winsor on December 1, 2022 on LessWrong. Please check out the colab notebook for interactive figures and more detailed technical explanations. This post is part of the work done at Conjecture. Special thanks to Sid Black, Dan Braun, Carlos Ramón Guevara, Beren Millidge, Chris Scammell, Lee Sharkey, and Lucas Teixeira for feedback on early drafts. There's a lot of non-linearities floating around in neural networks these days, but one that often gets overlooked is LayerNorm. This is understandable because it's not "supposed" to be doing anything; it was originally introduced to stabilize training. Contemporary attitudes about LayerNorm's computational power range from "it's just normalizing a vector" to "it can do division apparently". And theories of mechanistic interpretability such as features as directions and polytopes are unhelpful, or even harmful, in understanding normalization's impact on a network's representations. After all, normalization doesn't alter the direction of vectors, but it still bends lines and planes (the boundaries of polytopes) out of shape. As it turns out, LayerNorm can be used as a general purpose activation function (you can solve MNIST with a LayerNorm MLP, for example). Concretely, it can do things like this: and this: We will explain what's going on in these animations later, but the point is that to develop a strong, principled theory of mechanistic interpretability, we need to grapple with this non-linearity. In this interactive notebook, we study LayerNorm systematically using math and geometric intuition to characterize the ways in which it can manipulate data. We show that the core non-linearity of LayerNorm can be understood via simple geometric primitives. We explain how these basic primitives may perform semantic operations. For example, folding can be viewed as extracting extremal features (e.g. separating extreme temperatures from normal ones). We leverage these primitives to understand more complex low-dimensional classification tasks with multiple layers of non-linearities. The methods and intuition developed here extend to non-linearities beyond LayerNorm, and we plan to extend them in future work. Below is an interactive summary of the content of the notebook. (If you find the summary interesting, you really should check out the notebook: manipulating the figures helps build geometric intuition.) Summary The formula for LayerNorm is something messy like But it turns out the core non-linear operation is (almost) normalizing a vector: Graphically, this function has the iconic sigmoid shape in one dimension (note that in 1D the norm is simply the absolute value). Interesting things start happening when we precompose this normalization function with affine transformations (such as scaling and shifting). Below, we start with a collection of points, (x,y), distributed uniformly on the sphere. Then we compute uϵ(tx,y) (stretch/shrink by a factor of t in the x direction and then normalize). Since normalization guarantees points end up on the circle, this operation stretches the distribution along the circle. The right hand panel illustrates how this stretching is captured in the shape of a 1D activation function (the x coordinate of the input against the x coordinate of the output). For example, when t is close to 5, we see that any points with x coordinates not near 0 get compressed towards x=−1 or x=1. This matches the picture in the circle where we see the points bunch up at the left and right sides of the circle. If we make the same plots with a shifting operation, uϵ(x+t,y), the operation folds the input data. We can envision possible "semantic" uses for these geometric operations. Stretching can be used to perform an approximate "sign" operation (as in + or - sign). For example, it ...

Link to original articleWelcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Re-Examining LayerNorm, published by Eric Winsor on December 1, 2022 on LessWrong. Please check out the colab notebook for interactive figures and more detailed technical explanations. This post is part of the work done at Conjecture. Special thanks to Sid Black, Dan Braun, Carlos Ramón Guevara, Beren Millidge, Chris Scammell, Lee Sharkey, and Lucas Teixeira for feedback on early drafts. There's a lot of non-linearities floating around in neural networks these days, but one that often gets overlooked is LayerNorm. This is understandable because it's not "supposed" to be doing anything; it was originally introduced to stabilize training. Contemporary attitudes about LayerNorm's computational power range from "it's just normalizing a vector" to "it can do division apparently". And theories of mechanistic interpretability such as features as directions and polytopes are unhelpful, or even harmful, in understanding normalization's impact on a network's representations. After all, normalization doesn't alter the direction of vectors, but it still bends lines and planes (the boundaries of polytopes) out of shape. As it turns out, LayerNorm can be used as a general purpose activation function (you can solve MNIST with a LayerNorm MLP, for example). Concretely, it can do things like this: and this: We will explain what's going on in these animations later, but the point is that to develop a strong, principled theory of mechanistic interpretability, we need to grapple with this non-linearity. In this interactive notebook, we study LayerNorm systematically using math and geometric intuition to characterize the ways in which it can manipulate data. We show that the core non-linearity of LayerNorm can be understood via simple geometric primitives. We explain how these basic primitives may perform semantic operations. For example, folding can be viewed as extracting extremal features (e.g. separating extreme temperatures from normal ones). We leverage these primitives to understand more complex low-dimensional classification tasks with multiple layers of non-linearities. The methods and intuition developed here extend to non-linearities beyond LayerNorm, and we plan to extend them in future work. Below is an interactive summary of the content of the notebook. (If you find the summary interesting, you really should check out the notebook: manipulating the figures helps build geometric intuition.) Summary The formula for LayerNorm is something messy like But it turns out the core non-linear operation is (almost) normalizing a vector: Graphically, this function has the iconic sigmoid shape in one dimension (note that in 1D the norm is simply the absolute value). Interesting things start happening when we precompose this normalization function with affine transformations (such as scaling and shifting). Below, we start with a collection of points, (x,y), distributed uniformly on the sphere. Then we compute uϵ(tx,y) (stretch/shrink by a factor of t in the x direction and then normalize). Since normalization guarantees points end up on the circle, this operation stretches the distribution along the circle. The right hand panel illustrates how this stretching is captured in the shape of a 1D activation function (the x coordinate of the input against the x coordinate of the output). For example, when t is close to 5, we see that any points with x coordinates not near 0 get compressed towards x=−1 or x=1. This matches the picture in the circle where we see the points bunch up at the left and right sides of the circle. If we make the same plots with a shifting operation, uϵ(x+t,y), the operation folds the input data. We can envision possible "semantic" uses for these geometric operations. Stretching can be used to perform an approximate "sign" operation (as in + or - sign). For example, it ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Training for Good - Update & Plans for 2023, published by Cillian Crosson on November 15, 2022 on The Effective Altruism Forum. Summary Training for Good will now focus exclusively on programmes that enable talented and altruistic early-career professionals to directly enter the first stage of high impact careers. Concretely, we will only run the following programmes in Sep 2022 - Aug 2023: 1. EU Tech Policy Fellowship. 2. Tarbell Fellowship (journalism) 3. An unannounced 3rd programme which is still under development Applications for the 2023 EU Tech Policy Fellowship are open until December 11. Apply here. In year 1, we experimented with ~7 different programmes, 6 of which have now been discontinued. This is largely because we believe that focus is important & wanted to double-down on the most promising programme we had identified thus far (the EU Tech Policy Fellowship which successfully placed 7 fellows in relevant European think tanks focused on emerging technology policy). We plan to have an external review of Training for Good conducted between July 2023 - December 2023. We will default to sharing this publicly. Introduction Training for Good is an impact-focused training organisation, incubated by Charity Entrepreneurship in 2021. Quite a lot has changed since our launch in September 2021. We considered our first year to be an exploratory period in which we ran many, many different projects. We've now discontinued the majority of these programmes and have narrowed our focus to running fellowships that directly place early-career individuals in impactful careers. Now that TFG has a clearer focus, we're writing this post to update others in the EA community on our activities and the scope of our organisation. What we do Training for Good runs fellowships that place talented professionals in impactful careers in policy, journalism & other areas. We do this by providing a combination of stipends, mentorship from experienced professionals, training and placements in relevant organisations. Between Sep 2022 - Aug 2023 (i.e. year 2), we plan to only run the following programmes: EU Tech Policy Fellowship Tarbell Fellowship An unannounced 3rd programme which is still under development Why this might be important Many high impact career paths are neglected by talented and altruistic people, often because they lack clear pathways for entry. This is limiting progress on some of the world's most important problems: reducing existential risk, ending factory farming and tackling global poverty. TFG seeks to provide concrete opportunities for early-career professionals to gain entry level roles in impactful career paths that are difficult to enter. Building these talent pipelines could be important because: Direct progress on problems: Talented individuals in these career paths can directly contribute to progress on solving the world's most important problems Closer towards the “ideal portfolio”: We mostly take a portfolio approach to doing good. One could imagine an optimal distribution of talent within the effective altruism community, which might involve people pursuing a variety of different career paths. With our fellowships, we are attempting to move the effective altruism community closer towards this ideal allocation by enabling people to pursue paths that we believe are currently underrepresented (and expect to remain so) within this community's portfolio. We believe that thinking in these terms is particularly useful partly due to: Diminishing returns from certain career paths Epistemic uncertainty about which career paths are best (and the associated information value from reducing this uncertainty somewhat) Differing personal fit for individuals across different career paths Concrete opportunities: The number of people interested in effective altruism has been g...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How To Make Prediction Markets Useful For Alignment Work, published by johnswentworth on October 18, 2022 on LessWrong. So, I'm an alignment researcher. And I have a lot of prediction-flavored questions which steer my day-to-day efforts. Which recent interpretability papers will turn out, in hindsight a year or two from now, to have been basically correct and high-value? Will this infrabayes stuff turn out to be useful for real-world systems, or is it the kind of abstract math which lost contact with reality and will never reconnect? Is there any mathematical substance to Predictive Processing other than “variational inference is a thing” plus vigorous hand-waving? How about Shard Theory? What kinds of evidence will we see for/against the Natural Abstraction Hypothesis over the next few years? Will HCH-style amplification ever be able to usefully factor problems which the human operator/programmer doesn't immediately see how to factor? Will some version of this conjecture be proven? Will it be proven by someone else if I don't focus on it? Are there any recent papers/posts which lots of other people expect to have high value in hindsight, but which I haven't paid attention to? Or, to sum that all up in two abstract questions: what will I (or someone else whose judgement I at least find informative) think I should have paid more attention to, in hindsight? What will it turn out, in hindsight, that I should have ignored or moved on from sooner? Hmm, I wonder if the prediction markets have anything useful to say here? Let's go look for AI predictions on Manifold. hollywood-level AI-generated feature film by 2026? Will an AI get gold on any International Math Olympiad by 2025? Will AI wipe out humanity before the year 2100 Will any Fortune 500 corporation mostly/entirely replace their customer service workforce with AI by 2026? So, we've got about a gazillion different flavors of AI capabilities questions, with a little bit of dabbling into how-society-will-react-capabilities questions. On the upside, operationalizing things in lots of different ways is exactly what we want. On the downside, approximately-all of the effort is in operationalizing one thing (AI capabilities timelines), and that thing is just not particularly central to day-to-day research decisions. It's certainly not on the list of questions which first jump to mind when I think of things which would be useful to know to steer my research. Sure, somebody is going to argue in the comments that timelines are relevant for some particular decision, like whether to buy up GPU companies or something, but there's no way in hell that focusing virtually all alignment-related prediction-market effort on operationalizations of capabilities timelines is actually the value-maximizing strategy here. (And while I happened to open up Manifold first, the situation is basically the same in other prediction markets. To a first approximation, the only alignment-adjacent question prediction markets ever weigh in on is timelines.) So, here's my main advice to someone who wants to use prediction markets to help alignment work: imagine that you are an alignment researcher/grantmaker/etc. Concretely imagine your day-to-day: probing weight matrices in nets, conjecturing, reading papers/posts, reviewing proposals, etc. Then, ask what kind of predictions have the highest information value for your work. If the answer is “yet another operationalization of timelines”, then you have probably fucked up somewhere. Of course you might also try asking some researchers or grantmakers the same question, though keep in mind the standard user-interview caveat: users do not actually know what they want or would like. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Link to original articleWelcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How To Make Prediction Markets Useful For Alignment Work, published by johnswentworth on October 18, 2022 on LessWrong. So, I'm an alignment researcher. And I have a lot of prediction-flavored questions which steer my day-to-day efforts. Which recent interpretability papers will turn out, in hindsight a year or two from now, to have been basically correct and high-value? Will this infrabayes stuff turn out to be useful for real-world systems, or is it the kind of abstract math which lost contact with reality and will never reconnect? Is there any mathematical substance to Predictive Processing other than “variational inference is a thing” plus vigorous hand-waving? How about Shard Theory? What kinds of evidence will we see for/against the Natural Abstraction Hypothesis over the next few years? Will HCH-style amplification ever be able to usefully factor problems which the human operator/programmer doesn't immediately see how to factor? Will some version of this conjecture be proven? Will it be proven by someone else if I don't focus on it? Are there any recent papers/posts which lots of other people expect to have high value in hindsight, but which I haven't paid attention to? Or, to sum that all up in two abstract questions: what will I (or someone else whose judgement I at least find informative) think I should have paid more attention to, in hindsight? What will it turn out, in hindsight, that I should have ignored or moved on from sooner? Hmm, I wonder if the prediction markets have anything useful to say here? Let's go look for AI predictions on Manifold. hollywood-level AI-generated feature film by 2026? Will an AI get gold on any International Math Olympiad by 2025? Will AI wipe out humanity before the year 2100 Will any Fortune 500 corporation mostly/entirely replace their customer service workforce with AI by 2026? So, we've got about a gazillion different flavors of AI capabilities questions, with a little bit of dabbling into how-society-will-react-capabilities questions. On the upside, operationalizing things in lots of different ways is exactly what we want. On the downside, approximately-all of the effort is in operationalizing one thing (AI capabilities timelines), and that thing is just not particularly central to day-to-day research decisions. It's certainly not on the list of questions which first jump to mind when I think of things which would be useful to know to steer my research. Sure, somebody is going to argue in the comments that timelines are relevant for some particular decision, like whether to buy up GPU companies or something, but there's no way in hell that focusing virtually all alignment-related prediction-market effort on operationalizations of capabilities timelines is actually the value-maximizing strategy here. (And while I happened to open up Manifold first, the situation is basically the same in other prediction markets. To a first approximation, the only alignment-adjacent question prediction markets ever weigh in on is timelines.) So, here's my main advice to someone who wants to use prediction markets to help alignment work: imagine that you are an alignment researcher/grantmaker/etc. Concretely imagine your day-to-day: probing weight matrices in nets, conjecturing, reading papers/posts, reviewing proposals, etc. Then, ask what kind of predictions have the highest information value for your work. If the answer is “yet another operationalization of timelines”, then you have probably fucked up somewhere. Of course you might also try asking some researchers or grantmakers the same question, though keep in mind the standard user-interview caveat: users do not actually know what they want or would like. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Reflection Mechanisms as an Alignment target: A follow-up survey, published by Marius Hobbhahn on October 5, 2022 on The AI Alignment Forum. This is the second of three posts (part I) about surveying moral sentiments related to AI alignment. This work was done by Marius Hobbhahn and Eric Landgrebe under the supervision of Beth Barnes as part of the AI safety camp 2022. TL;DR: We find that the results of our first study, i.e. that humans tend to agree with conflict resolution mechanisms hold under different wordings but are weakened in adversarial scenarios (where we actively try to elicit less agreement). Furthermore, we find that people tend to agree much less with a mechanism when the decision-maker is a smart benevolent AI rather than a smart benevolent human. A positive interpretation of these findings is that humans are fine with giving power to a conflict resolution mechanism as long as humans are ultimately in control. Introduction In the first post, we surveyed 1000 US respondents about their moral beliefs, the conditions under which they would change their moral beliefs and how they felt about mechanisms to resolve moral disagreements such as democracy or debate. Our main findings of the first survey were Unsurprisingly, people have very different moral beliefs, e.g. on the morality of abortions, immigration or eating meat. They very rarely report to changing or wanting to change these beliefs, e.g. most participants (>80%) report having not changed them in the last 10 years and do not expect to change them in the next 10 years. However, they mostly think that mechanisms to resolve moral disagreements, such as democracy or debate, are good even when they disagree with the outcome. In other words, people are willing to accept outcomes that are different from their beliefs if they trust the process by which it was derived. We think this finding has some implications for AI alignment. Most importantly, we think that the alignment target of an AI should be a mechanism to resolve moral conflicts such as democracy or debate rather than a set of individual beliefs, e.g. about abortion or veganism. This is more likely to lead to decisions that can be accepted by most people and prevent a race from “aligning an AI to your personal beliefs at the cost of other people's beliefs”. We think the main weakness of our first result was that it could have been a result of confounders or randomness. We asked people for their opinions on multiple scenarios, but our questions still had some shortcomings. These include: Specificity: Maybe people would change their minds if they had a very specific scenario in mind rather than just an abstract question. Robustness: Maybe the way we ask our questions makes one answer more likely than another. Social desirability: Maybe people answer what they think is socially desirable. Robustness to different scenarios: Maybe people's opinions change when we change key variables of the process, e.g. when we specify a bad outcome or when we change who makes the decisions. Relation to AI: We never talk about AIs specifically, we always talk about smart, benevolent and nice future humans (see below for details). Therefore, we ran a follow-up study that attempts to address these problems. Concretely, we first run robustness tests by asking the same question in different ways. Then we run adversarial tests where we actively try to get people to give more disagreeing answers. Finally, we ask the same question that was asked with smart, benevolent and nice future humans but replace “humans” with “AIs”. Main findings Robustness To test the robustness of our questions, we randomly assigned one of the following three questions to a participant. Basic: This was the same scenario as in the previous study and was mostly used to provide a reference. People wer...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Representational Tethers: Tying AI Latents To Human Ones, published by Paul Bricman on September 16, 2022 on The AI Alignment Forum. This post is part of my hypothesis subspace sequence, a living collection of proposals I'm exploring at Refine. Preceded by ideological inference engines. Thanks Adam Shimi, Alexander Oldenziel, Tamsin Leake, and Ze Shen for useful feedback. TL;DR: Representational tethers describe ways of connecting internal representations employed by ML models to internal representations employed by humans. This tethering has two related short-term goals: (1) making artificial conceptual frameworks more compatible to human ones (i.e. the tension in the tether metaphor), and (2) facilitating direct translation between representations expressed in the two (i.e. the physical link in the tether metaphor). In the long-term, those two mutually-reinforcing goals (1) facilitate human oversight by rendering ML models more cognitively ergonomic, and (2) enable control over how exotic internal representations employed by ML models are allowed to be. Intro The previous two proposals in the sequence describe means of deriving human preferences procedurally. Oversight leagues focus on the adversarial agent-evaluator dynamics as the process driving towards the target. Ideological inference engines focus on the inference algorithm as the meat of the target-approaching procedure. A shortcoming of this procedural family is that even if you thankfully don't have to plug in the final goal beforehand (i.e. the resulting evaluator or knowledge base), you still have to plug in the right procedure for getting there. You're forced to put your faith in a self-contained preference-deriving procedure instead of an initial target. In contrast, the present proposal tackles the problem from a different angle. It describes a way of actively conditioning the conceptual framework employed by the ML model to be compatible with human ones, as an attempt to get the ML model to form accurate conceptions of human values. If this sounds loosely relates to half a dozen other proposals, that's because it is — consider referring to the Discussion for more details on tangents. In the meantime, following the structure of the previous posts in the sequence, here are some assumptions underlying representational tethers: Assumption 1, "Physicalism": Our thoughts are represented as neural dynamics. In the limit of arbitrarily large amounts of data on neural dynamics aligned with external stimuli (in the sense of parallel corpora), our thoughts can be accurately reconstructed. Assumption 2, "Bottleneck Layer": There is a bottleneck layer in the architecture of the ML model being tethered to human representations. This bottleneck refers to a low-dimensional representation through which all the information being processed by the ML model is forced to pass. Assumption 3, "AGI Hard, Human Values Harder": We are unlikely to formulate the True Name of human values in closed-form before deploying transformative AI. The best we are likely to do before takeoff is model human values approximately and implement an imperfect evaluator. Proposal Representational tethers suggest a way of aligning human and AI latents for the purpose of facilitating later interaction. There are two steps to this: First Bring Them Closer Incentivize the ML model to employ internal representations which are compatible with human ones, thus bringing them "closer." This can be operationalized by conditioning latent activations which arise in the ML model to be expressible in human representations. Concretely, optimization pressure would be exerted on the ML model to push it to internalize a conceptual framework which can successfully be translated to and from a human one without significant loss of information. If the artificial repre...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Finding before funding: Why EA should probably invest more in research, published by Falk Lieder on August 17, 2022 on The Effective Altruism Forum. EA grantmakers dramatically differ in how they split their funds between doing good with the tools and knowledge we already have versus improving them through scientific research. This raises the question of which proportion of its funds the EA community should invest in scientific research. Are we investing the correct percentage of our funds in scientific research? Should it be lower? Or should it be higher than it currently is? We recently found that funding scientific research that aims to enable or create improved interventions can be 2-5 times as cost-effective as investing in the interventions that already exist. This seems plausible given that scientific research gave us the tools and knowledge we can now use to improve the future of humanity. If previous generations had not funded scientific research and R&D projects, there would be no medicines for preventing malaria, no long-lasting insecticide treated bed nets, no supplements for preventing vitamin A deficiency, no deworming tablets, and no vaccines. Strategically funding relevant scientific research could generate even more cost-effective opportunities to do good. But the more money we invest in research the less money we will have left for applying our best interventions. To help funding agencies and the EA community navigate this dilemma, we investigated how the total investment into a cause should be split between scientific research that might enable us to do more good in the future and doing good with the tools and knowledge that happen to be already available today. Assumptions Our analysis makes the following three assumptions: On average, additional research and R&D projects produce interventions that are substantially less cost-effective than the best interventions that already exist. The cost-effectiveness of newly developed interventions is variable. Therefore, research occasionally succeeds in inventing superior interventions, and the cost-effectiveness of those interventions can be substantially higher. When a more cost-effective intervention is invented, then the funding that the best previous intervention would have received will be invested into the new intervention instead. We measure the cost-effectiveness of research in the same units as the cost-effectiveness of the existing interventions it might help us improve on. The expected increase in the amount of good we can do with the improved interventions is a lower bound on the value of research. Approach We performed more than 300 million computer simulations to determine which way of splitting a fixed budget between developing better interventions and deploying interventions would generate the most good in total in expectation across different scenarios. For each scenario, we estimated the total good that could be achieved by first funding 0, 1, 2, 3, ., or 1000 research projects and then optimally allocating the remaining funds between the best interventions that resulted from the research and the best intervention that were available prior to the research. We then derived the optimal size of the research budget from the number of funded research projects that maximized the total good that the funding agency accomplished on average across 10,000 simulations. To derive robust recommendations, we replicated the main findings of our simulations across a wide range of conservative assumptions about the effectiveness of scientific research, the variability in the cost-effectiveness of new interventions. Concretely, we varied the average cost-effectiveness of new interventions between 10% and 110% of the cost-effectiveness of the best existing interventions with a standard deviation between 1% ...

Changelog I've created a playground in the mono-repo for ReactJS experiments I have started working on a very lightweight design system I am giving a talk to PHP Minds next Thursday. The talk is called Code with Confidence with PHPCS PHP UK has announced their CFP and released early bird tickets. I have tickets. This weekend I am off to see England versus Scotland in the Common wealth boxing News/Articles How to call rest APIs using React https://www.smashingmagazine.com/2020/06/rest-api-react-fetch-axios/ How to use the React Router to create an image link https://bobbyhadz.com/blog/react-image-link React higher order components A higher-order component (HOC) is an advanced technique in React for reusing component logic. HOCs are not part of the React API, per se. They are a pattern that emerges from React's compositional nature. Concretely, a higher-order component is a function that takes a component and returns a new component. https://reactjs.org/docs/higher-order-components.html What is a Design system A collection of reusable components A place for collaboration amongst designers and developers A set of standards that must be adhered too A set of guidelines and principles that are relevant to the business Not just a component library A component library is a library of reusable components that other designers and developers can use. A design system also sets out a series of rules and standards for the components and surrounding design. These include but are not limited to Typography Spacing Branding User experience Language Voice and tone Colours Patten Often, a design system will explain what is allowed and what is not allowed. Examples of design systems IBM https://carbondesignsystem.com/ Shopify https://polaris.shopify.com/ Material design https://material.io/design UK Gov https://design-system.service.gov.uk/ You can get in touch with me here: https://howtocodewell.fm/contact My web development courses ➡️ Learn How to build a JavaScript Tip Calculator ➡️ Learn JavaScript arrays ➡️ Learn PHP arrays ➡️ Learn Python ✉️ Get my weekly newsletter ⏰ My current live coding schedule (Times are BST) Tuesdays 18:00 - Live coding on YouTube Sundays 15:00 - Live coding on Twitch

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Four questions I ask AI safety researchers, published by Akash on July 17, 2022 on The Effective Altruism Forum. Over the last few months, I've been trying to develop a stronger inside view on AI safety research agendas. As part of this quest, I've been having conversations with AI safety researchers. I notice myself often asking the following questions: What are you working on, and how does it help us get aligned AI? Imagine you're talking to a smart high school student. How would you describe the alignment problem? And how would you describe how your work addresses it? Imagine that you came up with a solution to the specific problem you're working on. Or even more boldly, imagine your entire program of research succeeds. What happens next? Concretely, how does this help us get aligned AI (and prevent unaligned AI)? What are the qualities you look for in promising AI safety researchers? (beyond general intelligence) I find Question #1 useful for starting the conversation and introducing me to the person's worldview. I find Question #2 useful for getting clearer (and often more detailed) explanations of the person's understanding of the alignment problem & how their work fits in. (Note that this is somewhat redundant with question #1, but I find that the questions often yield different answers). I find Question #3 useful for starting to develop an inside view on the research agenda & the person's reasoning abilities. (Note that the point is to see how the person reasons about the question. I'm not looking for a “right answer”, but I am looking to see that someone has seriously thought about this and has reasonable takes.) I find Question #4 useful for building stronger models of AI safety community-building. I also ask a lot of follow-up questions. But I find that these are the four questions that I ask nearly everyone, and I think they've helped me develop a stronger inside view. If I had to pick one question, I would probably ask #3. I think it's a rather challenging question, and I'm generally impressed when people have coherent answers. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Grouped Loss may disfavor discontinuous capabilities, published by Adam Jermyn on July 9, 2022 on The AI Alignment Forum. Thanks to Evan Hubinger and Beth Barnes for comments on these ideas. Language models exhibit scaling laws, where the loss is a power-law in model size. This offers a lot of predictive power, and seems like a useful thing to know. By contrast, individual capabilities can exhibit sharp discontinuities in performance as a function of model size and training time. It would be great if individual capabilities just gradually improved like the broader loss. Then we wouldn't have to worry quite so much about surprising new capabilities emerging suddenly during training. Is there a way to change the loss function so that it incentivizes more gradually capability improvements? Grouped Loss Imagine grouping training examples by the kind of capability they exhibit. For instance arithmetic problems go in one group, “parse json” could go in another, and so on. With these groups, we could define a new loss function where ℓ is the loss function we originally used (e.g. cross-entropy loss) and ℓg means to compute the sum of ℓ over examples from group g, e.g. which may be estimated by using random examples drawn from g. Because we have squared the group losses, the overall loss is dominated by the worst group. As a result, the model is incentivized to develop capabilities in each group at comparable rates, and so has little incentive to e.g. finely hone its poetry skills while being unable to multiply numbers. Challenge: Defining Groups It's possible that using grouped loss results in smooth development of capabilities that aren't represented in the groups. For instance, it seems plausible that if “adding arabic numerals” and “translating words into arabic numerals” are two groups but “adding numbers written as words” is not, performance on the latter could nonetheless develop smoothly as the model gets better at the others. It would certainly be weird if performance ”adding numbers written as words” advanced as a sudden leap in this case. This points to a general problem though, which is that if we have to define the groups manually we have to foresee the capabilities we're worried about. That seems bad. Gradient Cluster Grouping If we could automatically group examples we wouldn't need to do it manually. How could we do this? I think the key feature of a group is that when the model updates, the loss of most examples in a group changes in a similar way. When that happens, it seems intuitive to say that there's a discrete capability somewhere in the model and that those examples all depend on it. This suggests looking for examples where the loss has similar gradients, because these probably make use of similar machinery in the model. Concretely, I'm imagining the following procedure: Draw N examples from the training set. Evaluate the gradient of ℓ for each example. Group the examples by clustering their gradients, evaluate the grouped loss, and perform gradient descent on that. As a technical note: In this approach, the grouped loss is a moving target. As the model learns and capabilities form the groups shift. This means that SGD is no longer minimizing a constant loss. I don't think that's a problem, in part because all versions of the loss agree when the model has reached zero-loss, so the different iterations of the loss function all point towards better capabilities. Challenge: How many groups? I don't know of a principled way to pick the number of groups to cluster examples into, and that seems like a problem. Guessing too many groups loses the advantage of grouping because each group reflects an extremely narrow task. Guessing too few groups also loses the advantage of grouping, because then the capabilities that show gradual improvements will be very broa...

In the final part of this SiSto, the confrontation with Montebanc comes to a head, one of the world's most chaotic battles imaginable ensues, and the mystery is finally solved.Also, HUGE news in the first few minutes!  Like, Season 2 news!  Really!  Concretely!feat. Anita Bridges (@AnitaBridges2) as DM, SamuraiFoochs (@SamuraiFoochs, Thrugg) as Gavil Stonespire, Andy Mitts (@AndyMitts12, Dom) as Aroo, MannyTheOldtaku as Barnaby, and more! Follow us on Twitter (@NoLuckPod) and/or email us at withrollsnoluck@gmail.com!Logo by Mark Fionda Jr. (http://www.markfiondajr.com/)  

On her first single of 2022 titled “In Too Deep”, Sarah Tromley has served up her best yet, and we got the chance to talk to her all about it. Concretely solidifying her name within the Pop scene over the last couple of years, the trajectory of Sarah Tromley's music career is only going up. She's just released her latest track titled “In Too Deep” and it's easily one of the most quickly catchy songs we've heard in 2022. Luckily for us, we got the chance to sit with Sarah to discuss all the finer details of recording this single, in addition to a look towards what's coming for the rest of the year. Needless to say, we sincerely hope you listen and watch it all. --- This episode is sponsored by · Anchor: The easiest way to make a podcast. https://anchor.fm/app Support this podcast: https://anchor.fm/wewriteaboutmusic/support

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Nuclear attack risk? Implications for personal decision-making, published by SebastianSchmidt on February 27, 2022 on The Effective Altruism Forum. Summary: With Russia's invasion and Putin's gestures towards using nuclear weapons, we seemed to have passed a threshold where it makes sense to spent at least a couple of days thinking about this. Me and a couple of friends are thinking about what might be reasonable approaches from a personal decision-making perspective and would love your input. There's a war between Russia and Ukraine and Putin (who has control over the largest amount of nuclear warheads) appears to be threatening with nuclear war if "The West" interferes. Specifically, he says "Whoever tries to hinder us should know that Russia's response will be immediate, and it will lead you to such consequences that you have never encountered in your history" (source). Additionally, he appears to be paranoid and believes (or at least want his people to believe) that "Western Patriots" will bring weapons of mass destruction to Ukraine and "help it acquire these weapons to create yet another threat to our country.". Given this apparent paranoia, thirst for power, and his actual power, the risk of some form of nuclear attack seems to pass a threshold to think about it for at least one or two days. From a personal decision-making perspective, it seems reasonable to optimize for personal safety (in the case of nuclear war) and the ability to contribute to the long-term flourishing of our civilization - including this threat. Concretely, a couple of friends and me are considering changing location away from the UK and mainland Europe in a easily reversible and low cost way (e.g., home office from Morocco) while recurringly reassessing the situation for signals of escalation or de-escalation. The reasoning behind moving away Europe is due to the likelihood of a first-strike on states with nuclear weapons (UK and France) and the subsequent outbreak of panic with an increased difficulty to leave the country and questionable ability to maintain a robust infrastructure for living (e.g., food supply). PS. There's also a lot of interesting predictions on metaculus. What do you think about the reasoning and the tentative personal implications? Also, do you have any input on how a highly skilled personal and professional coach can contribute to the mitigation of this risk? Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: REPL's and ELK, published by scottviteri on February 17, 2022 on The AI Alignment Forum. In my previous post I talked about read-eval-print loops as providing a type signature for agents. I will now explain how you can quickly transition from this framework to an ELK solution. Notation is imported from that post. Imagine we have two agents, a human and a strong AI, denoted H and M respectively. They both interact with the environment in lockstep, according to the following diagram. We have the human's utility function UH:SH→Q, which is defined on the human's model of reality. We would like to lift UH to a version UM:SM→Q that the machine can use to influence the world in way that is agreeable to the human, which we can do by learning a mapping F : SM→SH and deriving UM=F∘UH. But we haven't yet said what properties we want the ontology map F to have. I want to call two concepts sh and sm equal if they act the same with respect to transformation: ∀f. f(sh) = f(sm) → sh = sm. The issue is that since the concepts have different types we cannot feed them as arguments to the same function. So instead let's say that ∀s:S, EvalH(sh, ReadH(s)) = EvalM(sm, ReadM(s)) → sh = sm. But now we are back to the same problem where we are trying to compare concepts in two different ontologies. But this does give us a kind of inductive step where we can transfer evidence of equality between concept pairs (sh, sm) and (sh', sm'). I also believe that this kind of a coherence argument is the best we can do, since we are not allowed to peer into the semantic content of particular machine or human states when constructing the ontology map. Consider the following two graphs. My intuition is that even if I don't know the labels of the above graphs, I can still infer that the bottom nodes correspond to each other. And the arrows that I get in the context of ELK are the agents' Eval transitions, leading to the following commutative diagram specification for F. We can learn an ontology map F : SM→SH by minimizing the difference between two paths from a state sm, one in which the machine's prediction function is used and one in which the human's prediction function is used. Concretely, I propose minimizing Dist(sh1,sh2) + λ |U(sh1)-U(sh2)| where sh1 = F(EvalM(sm,om)) and sh2 = EvalH(F(sm),oh), Dist is a distance metric in SH, and observations om and oh are generated by the same underlying state S. If you are interested in getting more detail and why I believe this circumvents existing counterexamples, please check out the full proposal. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Causality, Transformative AI and alignment - part I, published by Marius Hobbhahn on January 27, 2022 on The AI Alignment Forum. TL;DR: transformative AI(TAI) plausibly requires causal models of the world. Thus, a component of AI safety is ensuring secure paths to generating these causal models. We think the lens of causal models might be undervalued within the current alignment research landscape and suggest possible research directions. This post was written by Marius Hobbhahn and David Seiler. MH would like Richard Ngo for encouragement and feedback. If you think these are interesting questions and want to work on them, write us. We will probably start to play around with GPT-3 soonish. If you want to join the project, just reach out. There is certainly stuff we missed. Feel free to send us references if you think they are relevant. There are already a small number of people working on causality within the EA community. They include Victor Veitch, Zhijing Jin and PabloAMC. Check them out for further insights. Causality - a working definition: Just to get this out of the way: we follow a broad definition of causality, i.e. we assume it can be learned from (some) data and doesn't have to be put into the model by humans. Furthermore, we don't think the representation has to be explicit, e.g. in a probabilistic model, but could be represented in other ways, e.g. in the weights of neural networks. But what is it? In a loose sense, you already know: things make other things happen. When you touch a light switch and a light comes on, that's causality. There is a more technical sense in which no one understands causality, not even Judea Pearl (where does causal information ultimately come from if you have to make causal assumptions to get it? For that matter, how do we get variables out of undifferentiated sense data?). But it's possible to get useful results without understanding causality precisely, and for our purposes, it's enough to approach the question at the level of causal models. Concretely: you can draw circles around phenomena in the world (like "a switch" and "a lightbulb") to make them into nodes in a graph, and draw arrows between those nodes to represent their causal relationships (from the switch to the lightbulb if you think the switch causes the lightbulb to turn on, or from the lightbulb to the switch if you think it's the other way around). There's an old Sequences post that covers the background in more detail. The key points for practical purposes are that causal models: Are sparse, and thus easy to reason about and make predictions with (or at least, easier to reason about than the joint distribution over all your life experiences). Can be segmented by observations. Suppose you know that the light switch controls the flow of current to the bulb and that the current determines whether the bulb is on or off. Then, if you observe that there's no current in the wire (maybe there's a blackout), then you don't need to know anything about the state of the switch to know the state of the bulb. Able to evaluate counterfactuals. If the light switch is presently off, but you want to imagine what would happen if it were on, your causal model can tell you (insofar as it's correct). Why does causality matter? Causal, compared to correlational, information has two main advantages. For the following section, I got help from a fellow Ph.D. student. 1. Data efficiency Markov factorization: Mathematically speaking, Markov factorization ensures conditional independence between some nodes given other nodes. In practice, this means that we can write a joint probability distribution as a sparse graph where only some nodes are connected if we assume causality. It introduces sparsity. “Namely, if we have a joint with n binary random variables, it would have 2^n - 1 indepen...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Principled extremizing of aggregated forecasts, published by Jsevillamol on December 29, 2021 on The Effective Altruism Forum. In short: In light of a recent result by Eric Neyman, I tentatively recommend extremizing aggregated forecasts by a factor d=n(√3n2−3n+1−2)n2−n−1, which is approximately d≈√3≈1.73 for n>50. When historical resolution rates are available and reasonably constant through time, I more hesitantly recommend using the aggregate: where logObaseline are the historical logarithmic odds of a positive resolution. Neyman's result is theoretical and it is not clearly applicable in this context, but I show it has good performance on Metaculus data. The forecast aggregation war continues. Extreme forecasting Extremizing is the practice of adjusting aggregated forecasts towards a extreme - either 0% or 100% probability. If the experts predictions are odds o1,.,on, then an extremized aggregation of the odds is: for a given extremization factor d. Extremizing is a common practice in academic contexts. It is usually justified on the grounds that different experts have access to different information (Baron et al, 2014). And indeed it has been shown to have a good performance in practice. (Satopää et al, 2014) showed that forecasts in a collection of geopolitical questions are optimally aggregated when using an extremizing factor d∈[1.161,3.921]. However there are reasons to be skeptical of extremizing. It introduces a degree of freedom in the choice of the extremizing parameter. Results like Satopää's where an optimal extremizing factor is derived in hindsight risk overfitting this parameter. Before making a blank recommendation of extremizing, we would prefer to have a grounded way of choosing an extremizing factor. Why I became an extremist (Neyman and Roughgarden, 2021) have taken a close look at the theory of forecast aggregation and extremizing [1]. They suggest and analyse a non-conventional extremizing method, where they move the aggregate estimate away from its baseline value. Concretely, their estimate as applied to logodd aggregation would would be: This is the same as classical extremizing when we assume logObaseline=0, though as we will see later it might be better to use a historical estimate. Neyman and Roughgarden show that when using an extremizing factor d=n(√3n2−3n+1−2)n2−n−1 this estimate outperforms in a certain sense what one would get by a simple average of logodds. Extremizing factorn=1d=1 n=10d≈1.62n=2d≈1.29n=50d≈1.71n=3d≈1.41n=100d≈1.72n=5d≈1.53n=1000d≈1.73 As n grows, the recommended extremizing factor approaches d→√3≈1.73. In practice, it seems that for n>50 the approximation d≈√3≈1.73 is already pretty good [2]. In which sense does the extremized prediction perform better? The authors analysis is done in terms of what they call the approximation ratio. This measures how close the aggregated estimate approaches gets to the square loss of the idealized prediction of an expert with access to everyone's information [3]. The authors find that under the projective substitutes condition their aggregation scheme performs better than a simple average, in terms of the approximation ratio [4]. What is this projective substitutes condition? Essentially, it states that there are diminishing marginal returns to more forecasts. [5]. I think that this is a plausible assumption in the context of forecast aggregation, though it is not a guarantee [6]. Does the recommended aggregation strategy perform well in practice? Yes it does.I looked at 899 resolved binary questions in Metaculus, and compared several aggregation methods. The results of this analysis are included in the appendix. In short, when assuming logObaseline=0, Neyman's extremizing factor outperforms in this dataset most other methods I tried, and it is on par with a optimized and constant ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Concrete Ways to Reduce Risks of Value Drift and Lifestyle Drift, published by Darius_M on the effective altruism forum. Write a Review This post is motivated by Joey's post on ‘Empirical data on value drift' and some of the comments. Introduction “And Harry remembered what Professor Quirrell had said beneath the starlight: Sometimes, when this flawed world seems unusually hateful, I wonder whether there might be some other place, far away, where I should have been. And Harry couldn't understand Professor Quirrell's words, it might have been an alien that had spoken, (...) something built along such different lines from Harry that his brain couldn't be forced to operate in that mode. You couldn't leave your home planet while it still contained a place like Azkaban. You had to stay and fight.” Harry Potter and the Methods of Rationality I use the terms value drift and lifestyle drift in a broad sense to mean internal or external changes leading you to lose most of the expected altruistic value of your life. Value drift is internal; it describes changes to your value system or motivation. Lifestyle drift is external; the term captures changes in your life circumstances leading to difficulties implementing your values. Internally, value drift could occur by ceasing to see helping others as one of your life's priorities (losing the ‘A' in EA), or loosing the motivation to work on the highest-priority cause areas or interventions (losing the ‘E' in EA). Externally, lifestyle drift could occur (as described in Joey's post) by giving up a substantial fraction of your effectively altruistic resources for non-effectively altruistic purposes, thus reducing your capacity to do good. Concretely, this could involve deciding to spend a lot of money on buying a (larger) house, having a (fancier) wedding, traveling around the world (more frequently or expensively), etc. Of course, changing your cause area or intervention to something that is equally or more effective within the EA framework does not count as value drift. Note that even if your future self were to decide to leave the EA community, as long as you still see ‘helping others effectively' as one of your top-priorities in life it might not constitute value drift. You don't need to call yourself an EA to have a large impact. But I am convinced that EA as a community helps many members uphold their motivation for doing the most good. Why this is important for altruists There is a difference between the potential altruistic value and the expected altruistic value you may achieve over the course of your lifetime. Risks of value or lifestyle drift may make you lose most of the expected altruistic value of your life, thus preventing you from realizing a large fraction of your potential altruistic value. Most of the potential altruistic value of EAs lies in the medium- to long-term, when more and more people in the community take up highly effective career paths and build their professional expertise to reach their ‘peak productivity' (likely in their 40s). However, if value and lifestyle drift are common, most of an EA's expected altruistic value lies in the short- to medium-term; the reason being, that many of the people currently active in the community will cease to be interested in doing the most good long before they reach their peak productivity. This is why, speaking for myself, losing my altruistic motivation or giving up a large fraction of my altruistic resources in the future would equal a small moral tragedy to my present self. I think that as EAs we can reasonably have a preference for our future selves not to abandon our fundamental commitment to altruism or effectiveness. What you can do to reduce risks of value drift and lifestyle drift: Caveat: the following suggestions are all very tentative and largely based o...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What Failure Looks Like: Distilling the Discussion, published by Ben Pace on the AI Alignment Forum. The comments under a post often contains valuable insights and additions. They are also often very long and involved, and harder to cite than posts themselves. Given this, I was motivated to try to distill some comment sections on LessWrong, in part to start exploring whether we can build some norms and some features to help facilitate this kind of intellectual work more regularly. So this is my attempt to summarise the post and discussion around What Failure Looks Like by Paul Christiano. Epistemic status: I think I did an okay job. I think I probably made the most errors in place where I try to emphasise concrete details more than the original post did. I think the summary of the discussion is much more concise than the original. What Failure Looks Like (Summary) On its default course, our civilization will build very useful and powerful AI systems, and use such systems to run significant parts of society (such as healthcare, legal systems, companies, the military, and more). Similar to how we are dependent on much novel technology such as money and the internet, we will be dependent on AI. The stereotypical AI catastrophe involves a powerful and malicious AI that seems good but suddenly becomes evil and quickly takes over humanity. Such descriptions are often stylised for good story-telling, or emphasise unimportant variables. The post below will concretely lay out two ways that building powerful AI systems may cause an existential catastrophe, if the problem of intent alignment is not solved. This is solely an attempt to describe what failure looks like, not to assign probabilities to such failure or to propose a plan to avoid these failures. There are two failure modes that will be discussed. First, we may increasingly fail to understand how our AI systems work and subsequently what is happening in society. Secondly, we may eventually give these AI systems massive amounts of power despite not understanding their internal reasoning and decision-making algorithms. Due to the massive space of designs we'll be searching through, if we do not understand the AI, this will mean certain AIs will be more power-seeking than expected, and will take adversarial action and take control. Failure by loss of control There is a gap between what we want, and what objective functions we can write down. Nobody has yet created a function that when maximised perfectly describes what we want, but increasingly powerful machine learning will optimise very hard for what function we encode. This will lead to a strong increases the gap between what we can optimise for and what we want. (This is a classic goodharting scenario.) Concretely, we will gradually use ML to perform more and more key functions in society, but will largely not understand how these systems work or what exactly they're doing. The information we can gather will seem strongly positive: GDP will be rising quickly, crime will be down, life-satisfaction ratings will be up, congress's approval will be up, and so on. However, the underlying reality will increasingly diverge from what we think these metrics are measuring, and we may no longer have the ability to independently figure this out. In fact, new things won't be built, crime will continue, people's lives will be miserable, and congress will not be effective at improving governance, we'll just believe this because the ML systems will be improving our metrics, and will have a hard time understanding what's going on outside of what they report. Gradually, our civilization will lose its ability to understand what is happening in the world as our systems and infrastructure shows us success on all of our available metrics (GDP, wealth, crime, health, self-reported happiness...

Tripp and Chase are back! Christianity and Marxism? It's a Homebrewed Christianity and Faith and Capital mash-up for a follow up to our previous episode on being Christian and Socialist. This episode is a Q&A where Tripp and I respond to 10ish questions submitted to us by listeners on all things faith, the bible, revolutionary violence, process theology, maoism, etc. Twas a grand ole time!Questions:00:08:20 -- What is the Gospel if seen from a materialist perspective?00:11:10 -- Given so much damage and baggage from White Christianity, what keeps you a Christian?00:15:10 -- The Bible talks about the evil of riches and exploitation. Is there anything in the Bible that helps us think about the evils of capitalism?00:25:40 -- What is Democratic Socialism? What is Maoism? What is Organic Marxism? And how do they resonate with our faith?00:50:10 -- How does Process Theology work alongside anti-authoritarian marxist and anarchist traditions and its critique of omnipotence help us step toward a more anarchistic approach to these conversations?00:59:10 -- Can revolutionary violence be reconciled with Christian faith?01:19:10 -- What's the difference between literal resurrection and historical resurrection?01:22:50 -- Concretely, how would you envision change coming to make a better world? What are some concrete actions that would need to take place and how does your faith speak to that?Support the show: patreon.com/faithandcapitalOr leave a one time contribution at with PayPal: https://www.paypal.me/faithandcapitalLeave an iTunes rating and review.Faith and Capital is on instagram, twitter, and facebook.Email me: faithandcapital@gmail.comMusic by D.C.R. PollockSupport the show (https://www.patreon.com/faithandcapital)

Concretely, we intend to live in the kingdom of God by intending to obey the precise example and teachings of Jesus. This is the form that trust in him takes. It does not take the form of merely believing things about him, however true they may be. Indeed, no one can actually believe the truth about him without trusting him by intending to obey him.” ― Dallas Willard, Renovation of the Heart: Putting On the Character of Christ Today we take a deeper look at the topic of faith in what it means to believe. Show notes EP26 Gospel of God 4 Preach: https://subspla.sh/6bvc6hz Books mentioned Eternity Is Now in Session by John Ortberg: https://www.amazon.com/Eternity-Now-Session-Rediscovery-Salvation/dp/1496431642

Concretely, we intend to live in the kingdom of God by intending to obey the precise example and teachings of Jesus. This is the form that trust in him takes. It does not take the form of merely believing things about him, however true they may be. Indeed, no one can actually believe the truth about him without trusting him by intending to obey him.” ― Dallas Willard, Renovation of the Heart: Putting On the Character of Christ Today we take a deeper look at the topic of faith in what it means to believe. Show notes EP26 Gospel of God 4 Preach: https://subspla.sh/6bvc6hz Books mentioned Eternity Is Now in Session by John Ortberg: https://www.amazon.com/Eternity-Now-Session-Rediscovery-Salvation/dp/1496431642

Reading is delivered by Heny and meditation is delivered by Julius, from Saint Peter's School in Jakarta, Indonesia. Deuteronomy 6: 4-13; Rs psalm 18: 2-4.47.51ab; Matthew 17: 14-20 FAITH AS BIG AS A MUSTARD SEED Our meditation today has the theme: Faith as Big as a Mustard Seed. There was a fact that after a long time following the Master, Jesus Christ, the apostles proved to be unable to solve a problem they were facing. The sick person who was possessed by the evil spirits could not be healed and freed from the evil spirits. They obviously realized that something was wrong with their faith, but they didn't know exactly what was the matter. Jesus Christ discovered that what the disciples really lacked was the strength, quality and essence of that faith even though they had been living with Him all the time. Faith is likened to the size of a mustard seed, that is, although it is small, it contains quality, strength, and is full of God's power. Such great potential will be able to do the impossible. The faithful person can command the mountain to move to another place. Concretely speaking, what does it mean by faith of the size of a mustard seed? The real meaning here is that in a believer, God is the One doing and moving. This follows what Saint Paul said, that his life was not from himself, but from Jesus Christ who lived in him. For example, you are asked by a friend to help relieving a certain pain, as you believe that God will do everything. Prayers, faith, energy, thoughts, initiatives, work, materials that may be in great use to help, but God must be the number one given His space and opportunity to act on that sick person. It often happens that our human abilities are too much given the priority to act on our problems, and so we soon or later face the consequences obviously seen like we are easily tired, bored, angry and giving up. This so happens because God is not given His space and opportunity to act. Finally we will complain like the apostles, namely why can't we heal or solve the problems ourselves? From His hidden place, God may whisper in this way: “Your faith is not strong. You don't ask Me to intervene.” Another point that signifies faith as big as a mustard seed is the awareness and love to maintain the faith that is growing within us. Although small as a mustard seed, the temptation to underestimate it, to forget it and to neglect to care for it must be as strong as possible to be overcome. From long ago Moses had reminded of a loyalty, obedience and endurance to defend this faith. We are encouraged to never forget and abandon this small size of faith just because there is something new that comes and attracts our attention. We indeed have basic teachings, holy scriptures, sacred traditions, spirituality that all of these must be well preserved, maintained and useful all the moments of our life. Only then can our faith endure to the end. Let's pray. In the name of the Father... O Lord Jesus Christ, Your Mother and also our Mother, the Blessed Virgin Mary is an example for a firm and genuine faith, just like a very useful mustard seed. Guide us to always imitate her true faith. Hail Mary full of grace... In the name of the Father... --- Send in a voice message: https://anchor.fm/media-la-porta/message

In this first episode, we dive deeper into what is developer experience with Adam Wathan from Tailwind Labs and Lee Robinson from Vercel. Why did developer experience become so important? Concretely, what does it mean and how do you achieve it? Sarah Dayan chats with our 2 guests to uncover what makes awesome DX, what are the best examples out there, and how they imagine the future.Adam Wathan is the creator of Tailwind CSS, one of the most popular and the fastest growing CSS framework, which popularized utility-first CSS. He has since then founded his own company, Tailwind Labs, where he builds the best tools for CSS developers.Lee Robinson is the Head of Developer Relations at Vercel, the company behind Next.js, which is rapidly becoming the most popular React-based framework out there. He's built tons of learning material, including blog posts and courses, and you can always count on his video tutorials to learn how to build anything with Next.

02:13 - Michael’s Superpower: Being Able to Creatively Digest and Reconstruct Categories * Integral Theory (https://en.wikipedia.org/wiki/Integral_theory_(Ken_Wilber)) * Creative Deconstruction – Michael Schwartz (https://ideas.repec.org/f/psc306.html) * Creating Truly Novel Categories – Recognizing Novelty as Novelty 09:39 - Recognizing Economic Value of Talents & Abilities * Invisible Labor * Ecosystem Services * Biodiversity; The Diversity Bonus by Scott Page (https://www.amazon.com/Diversity-Bonus-Knowledge-Compelling-Interests/dp/0691176884) 18:49 - The Edge of Chaos; Chaos Theory (https://en.wikipedia.org/wiki/Chaos_theory) * “Life exists at the edge of chaos.” 23:23 - Reproducibility Crisis and Context-Dependent Insight 28:49 - What constitutes a scientific experiment? * Missed Externalities * Scholarly articles for Michelle Girvan "reservoir computing" (https://scholar.google.com/scholar?q=Michelle+Girvan+reservoir+computing&hl=en&as_sdt=0&as_vis=1&oi=scholart) * Non-conformity 38:03 - The Return of Civil Society and Community Relationships; Scale Theory * Legitimation Crisis by Juergen Habermas (https://www.amazon.com/Legitimation-Crisis-Juergen-Habermas/dp/0807015210) * Scale: The Universal Laws of Life and Death in Organisms, Cities and Companies by Geoffrey West (https://www.amazon.com/Scale-Universal-Organisms-Cities-Companies-ebook/dp/B010P7Z8J0) 49:28 - Fractal Geometry More amazing resources from Michael to check out: Michael Garfield: Improvising Out of Algorithmic Isolation (https://blog.usejournal.com/improvising-out-of-algorithmic-isolation-7ef1a5b94697?gi=e731ad1488b2) Michael Garfield: We Will Fight Diseases of Our Networks By Realizing We Are Networks (https://michaelgarfield.medium.com/we-will-fight-diseases-of-our-networks-by-realizing-we-are-networks-7fa1e1c24444) Reflections: Jacob: Some of the best ideas, tv shows, music, etc. are the kinds of things that there’s not going to be an established container. Rein: “Act always so as to increase the number of choices.” ~ Heinz von Foerster Jessica: Externality. Recognize that there’s going to be surprises and find them. Michael: Adaptability is efficiency aggregated over a longer timescale. This episode was brought to you by @therubyrep (https://twitter.com/therubyrep) of DevReps, LLC (http://www.devreps.com/). To pledge your support and to join our awesome Slack community, visit patreon.com/greaterthancode (https://www.patreon.com/greaterthancode) To make a one-time donation so that we can continue to bring you more content and transcripts like this, please do so at paypal.me/devreps (https://www.paypal.me/devreps). You will also get an invitation to our Slack community this way as well. Transcript: JACOB: Hello and welcome to Episode 234 of Greater Than Code. My name is Jacob Stoebel and I’m joined with my co-panelist, Rein Henrichs. REIN: Thanks, Jacob and I’m here with my friend and co-panelist, Jessica Kerr. JESSICA: Thanks, Rein and today, I’m excited to introduce our guest, Michael Garfield. He’s an artist and philosopher and he helps people navigate our age of accelerating weirdness and cultivate the curiosity and play we need to thrive. He hosts and produces two podcasts, The Future Fossils Podcast & The Santa Fe Institute's Complexity Podcast. Yay, complexity! Michael acts as interlocutor for a worldwide community of artists, scientists, and philosophers—a practice that feeds his synthetic and transdisciplinary “mind-jazz” performances in the form of essay, avant-guitar music, and painting! You can find him on Bandcamp, it’s pretty cool. Refusing to be enslaved by a single perspective, creative medium, or intellectual community, Michael walks through the walls between academia and festival culture, theory and practice. Michael, welcome to Greater Than Code! MICHAEL: Thanks! I’m glad to be here and I hope that I provide a refreshingly different guest experience for listeners being not a coder in any kind of traditional sense. JESSICA: Yet you’re definitely involved in technology. MICHAEL: Yeah, and I think the epistemic framing of programming and algorithms is something that can be applied with no understanding of programming languages as they are currently widely understood. It’s just like design is coding, design of the built environment, so. JESSICA: And coding is a design. MICHAEL: Indeed. JESSICA: Okay, before we go anywhere else, I did not prepare you for this, but we have one question that we ask all of our guests. What is your superpower and how did you acquire it? MICHAEL: I would like believe that I have a superpower in being able to creatively digest and reconstruct categories so as to drive new associations between them for people and I feel like I developed that studying integral theory in grad school. I did some work under Sean Esbjörn-Hargens at John F. Kennedy University looking at the work of and work adjacent to Ken Wilber, who was trying to come up with a metatheoretical framework to integrate all different domains of human knowledge. All different types of inquiry into a single framework that doesn't attempt to reduce any one of them to any other and then in that process, I learned what one of my professors, Michael Schwartz, called creative deconstruction. So showing how art can be science and science can be art and that these aren't ontologically fixed categories that exist external to us. Looking at the relationship between science as a practice and spiritual inquiry as a practice and that kind of thing. So it's an irreverent attitude toward the categories that we've constructed that takes in a way a cynical and pragmatic approach to the way that we define things in our world. You know. REIN: Kant was wrong. [laughs] MICHAEL: It's good to get out of the rut. Obviously, you’ve got to be careful because all of these ideas have histories and so you have to decide whether it's worth trying to redefine something for people in order to open up new possibilities in the way that these ideas can be understood and manipulated. It's not, for example, an easy task to try and get people to change their idea about what religion is. [laughs] JESSICA: Yeah. More than redefined. It's almost like undefined. MICHAEL: Hm. Like Paul Tillich, for example. Theologian Paul Tillich said that religion is ultimate concern. So someone can have a religion of money, or a religion of sex, but if you get into these, if you try to interpose that in a debate on intelligent design versus evolutionary theory, you'll get attacked by both sides. JESSICA: [chuckles] That’s cosmology. MICHAEL: Yeah. So it's like – [overtalk] JESSICA: Which is hard to [inaudible] of money, or sex. MICHAEL: Yeah, but people do it anyhow. JESSICA: [laughs] Yeah. So deconstructing categories and seeing in-between things that fits through your walking through walls, what categories are you deconstructing and seeing between lately? MICHAEL: Well, I don't know, lately I've been paying more attention to the not so much tilting after the windmills of this metamorphic attitude towards categories, but looking at the way that when the opportunity comes to create a truly novel category, what are the forces in play that prevent that, that prevent recognizing novelty as novelty that I just – JESSICA: Do you have any examples? MICHAEL: Yeah, well, I just saw a really excellent talk by UC Berkeley Professor Doug Guilbeault, I think is how you say his name. I am happy to link his work to you all in the chat here so that you can share it. JESSICA: Yeah, we’ll link that in the show notes. MICHAEL: He studies category formation and he was explaining how most of the research that's been done on convergent categorization is done on established categories. But what happens when you discover something truly new? What his research shows is that basically the larger the population, the more likely it is that these categories will converge on something that's an existing category and he compared it to island versus mainland population biogeography. So there's a known dynamic in evolutionary science where genetic drift, which is just this random component of the change in allele frequencies in a population, the larger the population, the less likely it is that a genetic mutation that is otherwise neutral is going to actually percolate out into the population. On an island, you might get these otherwise neutral mutations that actually take root and saturate an entire community, but on the mainland, they get lost in the noise. You can look at this in terms of how easy it is for an innovative, artistic, or musical act to actually find any purchase. Like Spotify bought the data analysis company, The Echo Nest, back in 2015 and they ran this study on where emergent musical talent comes from. It comes from places like Australia, the UK, and Iceland, because the networks are small enough. This is a finding that's repeated endlessly through studies of how to create a viral meme that basically, or another way – JESSICA: You mean a small enough pool to take hold? MICHAEL: Yeah. That basically big science and large social networks online and these other attempts, anywhere we look at this economies of scale, growing a given system, what happens is—and we were talking about this a little before we got on the call—as a system scales, it becomes less innovative. There's less energy is allocated to – JESSICA: In America? MICHAEL: Yeah. Bureaucratic overhead, latencies in the network that prevent the large networks from adapting, with the same agility to novel challenges. There's a lot of different ways to think about this and talk about this, but it basically amounts to, if you want to, you can't do it from the conservative core of an organization. You can't do it from the board of directors. JESSICA: Oh. MICHAEL: You have to go out onto – like why did they call it fringe physics? It's like, it is because it's on the fringe and so there's a kind of – JESSICA: So this would be like if you have like one remarkably lowercase agile team inside your enterprise, one team is innovating and development practices. They're going to get mushed out. Whereas, if you have one team innovating like that in a small company, it might spread and it might become dominant. MICHAEL: Yeah. I think it's certainly the case that this speaks to something I've been wondering about it in a broader sense, which is how do we recognize the economic value of talents and abilities that are like, how do we recognize a singular individual for their incompressible knowledge and expertise when they don't go through established systems of accreditation like getting a PhD? Because the academic system is such that basically, if you have an innovative contribution, but you don't have the credentials that are required to participate in the community of peer review, then people can't even – your contribution is just invisible. The same is true for how long it took, if you look at economic models, it took so long for economic models to even begin to start addressing the invisible labor of women in at home like domestic labor, or what we're now calling ecosystem services. So there's this question of – I should add that I'm ambivalent about this question because I'm afraid that answering it in an effective way, how do we make all of these things economically visible would just accelerate the rate at which the capitalist machine is capable of co-opting and exploiting all of these. [chuckles] REIN: Yeah. You also have this Scott Seeing Like a State thing where in order to be able to even perceive that that stuff is going on, it has to become standardized and you can't dissect the bird to observe its song, right? MICHAEL: Totally. So obviously, it took almost no time at all for consumer culture to commodify the psychedelic experience and start using to co-opt this psychedelic aesthetic and start using it in advertising campaigns for Levi's Jeans and Campbell Soup and that kind of thing. So it’s this question of a moving frontier that as soon as you have the language to talk about it, it's not the ineffable anymore. REIN: Yeah. MICHAEL: There's a value to the ineffable and there's a value to – it's related to this question of the exploitation of indigenous peoples by large pharmaceutical companies like, their ethnobotanical knowledge. How do you make the potential value of biodiversity, something that can be manufactured into medicine at scale, without destroying the rainforest and the people who live in it? Everywhere I look, I see this question. So for me, lately, it's been less about how do we creatively deconstruct the categories we have so much as it is, what is the utility of not knowing how to categorize something at all and then how do we fix the skewed incentive structures in society so as to value that which we currently do not know how to value. JESSICA: Because you don’t have a category for it. MICHAEL: Right. Like right now, maybe one of the best examples, even though this is the worst example in another way, is that a large fraction of the human genome has been patented by Monsanto, even though it has no known current biomedical utility. This is what Lewis Hyde in his book, Common as Air, called “the third enclosure” of the common. So you have the enclosure of the land that everyone used to be able to hunt on and then you have the enclosure of intellectual property in terms of patents for known utilities, known applications, and then over the last few decades, you're starting to see large companies buy their way into and defend patents for the things that actually don't – it's speculative. They're just gambling on the idea that eventually we'll have some use for this and that it's worth lawyering up to defend that potential future use. But it's akin to recognizing that we need to fund translational work. We need to fund synthesis. We need to fund blue sky interdisciplinary research for which we don't have an expected return on investment here because there's – JESSICA: It's one of those things that it’s going to help; you're going to get tremendous benefits out of it, but you can't say which ones. MICHAEL: Right. It's a shift perhaps akin to the move that I'm seeing conservation biology make right now from “let's preserve this charismatic species” to “let's do everything we can to restore biodiversity” rather than that biodiversity itself is generative and should be valued in its own regard so diverse research teams, diverse workplace teams. We know that there is what University of Michigan Professor Scott Page calls the diversity bonus and you don't need to know and in fact, you cannot know what the bonus is upfront. JESSICA: Yeah. You can't draw the line of causality forward to the benefit because the point of diversity is that you get benefits you never thought of. MICHAEL: Exactly. Again, this gets into this question of as a science communications staffer in a position where I'm constantly in this weird dissonant enters zone between the elite researchers at the Santa Fe Institute where I work and the community of complex systems enthusiasts that have grown up around this organization. It's a complete mismatch in scale between this org that has basically insulated itself so as to preserve the island of innovation that is required for really groundbreaking research, but then also, they have this reputation that far outstrips their ability to actually respond to people that are one step further out on the fringe from them. So I find myself asking, historically SFI was founded by Los Alamos National Laboratory physicists mostly that were disenchanted with the idea that they were going to have to research science, that their science was limited to that which could be basically argued as a national defense initiative and they just wanted to think about the deepest mysteries of the cosmos. So what is to SFI as SFI as to Los Alamos? Even in really radical organizations, there's a point at which they've matured and there are questions that are beyond the horizon of that which a particular community is willing to indulge. I find, in general, I'm really fascinated by questions about the nonlinearity of time, or about weird ontology. I'm currently talking to about a dozen other academics and para-academics about how to try and – I'm working, or helping to organize a working group of people that can apply rigorous academic approaches to asking questions that are completely taboo inside of academia. Questions that challenge some of the most fundamental assumptions of maternity, such as there being a distinction between self and other, or the idea that there are things that are fundamentally inaccessible to quantitative research. These kinds of things like, how do we make space for that kind of inquiry when there's absolutely no way to argue it in terms of you should fund this? And that's not just for money, that's also for attention because the demands on the time and attention of academics are so intense that even if they have interest in this stuff, they don't have the freedom to pursue it in their careers. That's just one of many areas where I find that this kind of line of inquiry manifesting right now. REIN: Reminds me a lot of this model of the edge of chaos that came from Packard and Langton back in the late 70s. Came out of chaos theory, this idea that there's this liminal transitionary zone between stability and chaos and that this is the boiling zone where self-organization happens and innovation happens. But also, that this zone is itself not static; it gets pushed around by other forces. MICHAEL: Yeah, and that's where life is and that was Langton's point, that life exists at the edge of chaos that it's right there at the phase transition boundary between what is it that separates a stone from a raging bonfire, or there’s the Goldilocks Zone kind of question. Yeah, totally. REIN: And these places that were at the edge of chaos that were innovative can ossify, they can move into the zone of stability. It's not so much that they move it's that, I don't know, maybe it's both. Where the frontier is, is constantly in motion. MICHAEL: Yeah, and to that point again, I tend to think about these things in a topographical, or geographical sense, where the island is growing, we're sitting on a volcano, and there's lots you can do with that metaphor. Obviously, it doesn't make sense. You can't build your house inside the volcano, right? [laughs] But you want to be close enough to be able to watch and describe as new land erupts, but at a safe distance. Where is that sweet spot where you have rigor and you have support, but you're not trapped within a bureaucracy, or an ossified set of institutional conventions? JESSICA: Or if the island is going up, if the earth is moving the island up until the coastline keeps expanding outward, and you built your house right on the beach. As in you’ve got into React when it was the new hotness and you learned all about it and you became the expert and then you had this great house on the beach, and now you have a great house in the middle of town because the frontier, the hotness has moved on as our massive technology has increased and the island raises up. I mean, you can't both identify as being on the edge and identify with any single category of knowledge. MICHAEL: Yeah. It's tricky. I saw Nora Bateson talking about this on Twitter recently. She's someone who I love for her subversiveness. Her father, Gregory Bateson, was a major player in the articulation of cybernetics and she's awesome in that sense of, I don't know, the minister's daughter kind of a way of being extremely well-versed in complex systems thinking and yet also aware that there's a subtle reductionism that comes in that misses – JESSICA: Misses from? MICHAEL: Well, that comes at like we think about systems thinking as it's not reductionist because it's not trying to explain biology in terms of the interactions of atoms. It acknowledges that there's genuine emergence that happens at each of these levels and yet, to articulate that, one of the things that happens is everything has to be squashed into numbers and so it’s like this issue of how do you quantify something. JESSICA: It's not real, if you can't measure it in numbers. MICHAEL: Right and that belies this bias towards thinking that because you can't quantify something now means it can't be quantified. JESSICA: You can’t predict which way the flame is going to go in the fire. That doesn't mean the fire doesn't burn. [chuckles] MICHAEL: Right. So she's interesting because she talks about warm data as this terrain, or this experience where we don't know how to talk about it yet, but that's actually what makes it so juicy and meaningful and instructive and – JESSICA: As opposed to taking it out of context. Leave it in context, even though we don't know how to do some magical analysis on it there. MICHAEL: Right, and I think this starts to generate some meaningful insights into the problem of the reproducibility crisis. Just as an example, I think science is generally moving towards context dependent insight and away from – even at the Santa Fe Institute, nobody's looking for a single unifying theory of everything anymore. It's far more illuminating, useful, and rigorous to look at how different models are practical given different applications. I remember in college there's half a dozen major different ways to define a biological species and I was supposed to get up in front of a class and argue for one over the other five. I was like, “This is preposterous.” Concretely, pun kind of intended, Biosphere 2, which was this project that I know the folks here at Synergia Ranch in Santa Fe at the Institute of Ecotechnics, who were responsible for creating this unbelievable historic effort to miniaturize the entire biosphere inside of a building. They had a coral reef and a rainforest and a Savannah and a cloud desert, like the Atacama, and there was one other, I forget. But it was intended as a kind of open-ended ecological experiment that was supposed to iterate a 100 times, or 50 times over a 100 years. They didn't know what they were looking for; they just wanted to gather data and then continue these 2-year enclosures where a team of people were living inside this building and trying to reproduce the entire earth biosphere in miniature. So that first enclosure is remembered historically as a failure because they miscalculated the rate at which they would be producing carbon dioxide and they ended up having to open the building and let in fresh air and import resource. JESSICA: So they learned something? MICHAEL: Right, they learned something. But that project was funded by Ed Bass, who in 1994, I think called in hostile corporate takeover expert, Steve Bannon to force to go in there with a federal team and basically issue a restraining order on these people and forcibly evict them from the experiment that they had created. Because it was seen as an embarrassment, because they had been spun in this way in international media as being uncredentialed artists, rather than scientists who really should not have the keys to this thing. It was one of these instances where people regard this as a scientific failure and yet when you look at the way so much of science is being practiced now, be it in the domains of complex systems, or in machine learning, what they were doing was easily like 20 or 30 years ahead of its time. JESSICA: Well, no wonder they didn’t appreciate it. MICHAEL: [chuckles] Exactly. So it's like, they went in not knowing what they were going to get out of it, but there was this tragic mismatch between the logic of Ed Bass’ billionaire family about what it means to have a return on an investment and the logic of ecological engineering where you're just poking at a system to see what will happen and you don't even know where to set the controls yet. So anyway. JESSICA: And it got too big. You talked about the media, it got too widely disseminated and became embarrassed because it wasn't on an island. It wasn't in a place where the genetic drift can become normal. MICHAEL: Right. It was suddenly subject to the constraints imposed upon it in terms of the way that people were being taught science in public school in the 1980s that this is what the scientific method is. You start with a hypothesis and it's like what if your – JESSICA: Which are not standards that are relevant to that situation. MICHAEL: Exactly. And honestly, the same thing applies to other computational forms of science. It took a long time for the techniques pioneered at the Santa Fe Institute to be regarded as legitimate. I'm thinking of cellular automata, agent-based modeling, and computer simulation generally. Steven Wolfram did a huge service, in some sense, to the normalization of those things in publishing A New Kind of Science, that massive book in whatever it was, 2004, or something where he said, “Look, we can run algorithmic experiments,” and that's different from the science that you're familiar with, but it's also setting aside for a moment, the attribution failure that that book is and acknowledging who actually pioneered A New Kind of Science. [chuckles] JESSICA: At least it got some information out. MICHAEL: Right. At least it managed to shift the goalpost in terms of what the expectations are; what constitutes a scientific experiment in the first place. JESSICA: So it shifted categories. MICHAEL: Yeah. So I think about, for example, a research that was done on plant growth in a basement. I forget who it was that did this. I think I heard this from, it was either Doug Rushkoff, or Charles Eisenstein that was talking about this, where you got two completely different results and they couldn't figure out what was going on. And then they realized that it was at different moments in the lunar cycle and that it didn't matter if you put your plant experiment in a basement and lit everything with artificial bulbs and all this stuff. Rather than sunlight, rather than clean air, if you could control for everything, but that there's always a context outside of your context. So this notion that no matter how cleverly you try to frame your model, that when it comes time to actually experiment on these things in the real world, that there's always going to be some extra analogy you've missed and that this has real serious and grave implications in terms of our economic models, because there will always be someone that's falling through the cracks. How do we actually account for all of the stakeholders in conversations about the ecological cost of dropping a new factory over here, for example? It's only recently that people, anywhere in the modern world, are starting to think about granting ecosystems legal protections as entities befitting of personhood and this kind of thing. JESSICA: Haven’t we copyrighted those yet? MICHAEL: [laughs] So all of that, there's plenty of places to go from there, I'm sure. REIN: Well, this does remind me of one of the things that Stafford Beer tried was he said, “Ponds are viable systems, they’re ecologies, they're adaptive, they're self-sustaining. Instead of trying to model how a pond works, what if we just hook the inputs of the business process into the pond and then hook the adaptions made by the pond as the output back into the business process and use the pond as the controlling system without trying to understand what makes a pond good at adapting?” That is so outside of the box and it blows my mind that he was doing this, well, I guess it was the 60s, or whatever, but this goes well beyond black boxing, right? MICHAEL: Yeah. So there's kind of a related insight that I saw Michelle Girvan gave at Santa Fe Institute community lecture a few years ago on reservoir computing, which maybe most of your audience is familiar with, but just for the sake of it, this is joining a machine learning system to a source of analog chaos, basically. So putting a computer on a bucket of water and then just kicking the bucket, every once in a while, to generate waves so that you're feeding chaos into the output of the machine learning algorithm to prevent overfitting. Again, and again, and again, you see this value where this is apparently the evolutionary value of play and possibly also, of dreaming. There's a lot of good research on both of these areas right now that learning systems are all basically hill climbing algorithms that need to be periodically disrupted from climbing the wrong local optimum. So in reservoir computing, by adding a source of natural chaos to their weather prediction algorithms, they were able to double the horizon at which they were able to forecast meteorological events past the mathematic limit that had been proven and established for this. That is like, we live in a noisy world. JESSICA: Oh, yeah. Just because it’s provably impossible doesn't mean we can't do something that's effectively the same thing, that's close enough. MICHAEL: Right. Actually, in that example, I think that there's a strong argument for the value of that which we can't understand. [laughs] It's like it's actually important. So much has been written about the value of Slack, of dreaming, of taking a long walk, of daydreaming, letting your mind wander to scientific discovery. So this is where great innovations come from is like, “I'm going to sleep on it,” or “I'm going to go on vacation.” Just getting stuck on an idea, getting fixated on a problem, we actually tend to foreclose on the possibility of answering that problem entirely. Actually, there's a good reason to – I think this is why Silicon Valley has recognized the instrumental value of microdosing, incidentally. [laughs] That this is that you actually want to inject a little noise into your algorithm and knock yourself off the false peak that you've stranded yourself on. JESSICA: Because if you aim for predictability and consistency, if you insist on reasonableness, you'll miss everything interesting. MICHAEL: Or another good way to put it is what is it, reasonable women don't make history. [laughs] There is actually a place for the – JESSICA: You don’t change the system by maximally conforming. MICHAEL: Right. JESSICA: If there is a place for… MICHAEL: It’s just, there is a place for non-conformity and it's a thing where it's like, I really hope and I have some optimism that what we'll see, by the time my daughter is old enough to join the workforce, is that we'll see a move in this direction where non-conformity has been integrated somehow into our understanding of how to run a business that we actively seek out people that are capable of doing this. For the same reason that we saw over the 20th century, we saw a movement from one size fits all manufacturing to design your own Nike shoes. There's this much more bespoke approach. JESSICA: Oh, I love those. MICHAEL: Yeah. So it's like we know that if we can tailor our systems so that they can adapt across multiple different scales, that they're not exploiting economies of scale that ultimately slash the redundancy that allows an organization to adapt to risk. That if we can find a way to actually generate a kind of a fractal structure in the governance of organizations in the way that we have reflexes. The body already does this, you don't have to sit there and think about everything you do and if you did, you’d die right away. JESSICA: [laughs] Yeah. REIN: Yeah. MICHAEL: If you had to pass every single twitch all the way up the chain to your frontal cortex JESSICA: If we had to put breathe on the list. [laughs] MICHAEL: Right. If you had to sit there and approve every single heartbeat, you'd be so dead. [overtalk] JESSICA: Oh my gosh, yeah. That's an energy allocation and it all needs to go through you so that you can have control. REIN: I just wanted to mention, that reminded me of a thing that Klaus Krippendorff, who's a cybernetics guy, said that there is virtue in the act of delegating one's agency to trustworthy systems. We're talking, but I don't need to care about how the packets get from my machine to yours and I don't want to care about that, but there's a trade-off here where people find that when they surrender their agency, that this can be oppressive. So how do we find this trade-off? MICHAEL: So just to anchor it again in something that I find really helpful. Thinking about the way that convenience draws people into these compacts, with the market and with the state. You look over the last several hundred years, or thousand years in the West and you see more and more of what used to be taken for granted as the extent in terms of the functions that are performed by the extended family, or by the neighborhood, life in a city, by your church congregations, or whatever. All of that stuff has been out boarded to commercial interests and to federal level oversight, because it's just more efficient to do it that way at the timescales that matter, that are visible to those systems. Yet, what COVID has shown us is that we actually need neighborhoods that suddenly, it doesn't – my wife and I, it was easy to make the decision to move across country to a place where we didn't know anybody to take a good job. But then suddenly when you're just alone in your house all the time and you've got nobody to help you raise your kids, that seems extremely dumb. So there's that question of just as I feel like modern science is coming back around to acknowledging that a lot of what was captured in old wives’ tales and in traditional indigenous knowledge, ecological knowledge systems that were regarded by the enlightenment as just rumor, or… JESSICA: Superstition. MICHAEL: Superstition, that it turns out that these things actually had, that they had merit, they were evolved. JESSICA: There was [inaudible] enough. MICHAEL: Right. Again, it wasn't rendered in the language that allowed it to be the subject of quantitative research until very recently and then, suddenly it was and suddenly, we had to circle back around. Science is basically in this position where they have to sort of canonize Galileo, they're like, “Ah, crap. We burned all these witches, but it turns out they were right.” There's that piece of it. So I think relatedly, one of the things that we're seeing in economist samples and Wendy Carlin have written about this is the return of the civil society, the return of mutual aid networks, and of gift economies, and of the extended family, and of buildings that are built around in courtyards rather than this Jeffersonian everyone on their own plot of land approach. That we're starting to realize that we had completely emptied out the topsoil basically of all of these community relationships in order to standardize things for a mass big agricultural approach, that on the short scale actually does generate greater yield. It's easier to have conversations with people who agree with you than it is – in a way, it's inexpedient to try and cross the aisle and have a conversation with someone with whom you deeply and profoundly disagree. But the more polarized we become as a civilization, the more unstable we become as a civilization. So over this larger timescale, we actually have to find ways to incentivize talking to people with whom you disagree, or we're screwed. We're kicking legs out from under the table. REIN: At this point, I have to name drop Habermas because he had this idea that there were two fundamental cognitive interests that humans have to direct their attempts to acquire knowledge. One is a technical interest in achieving goals through prediction and control and the other is a practical interest in ensuring mutual understanding. His analysis was that advanced capitalist societies, the technical interest dominates at the expense of the practical interest and that knowledge produced by empirical, scientific, analytic sciences becomes the prototype of all knowledge. I think that's what you're talking about here that we've lost touch with this other form of knowledge. It's not seen as valuable and the scientific method, the analytical approaches have come to dominate. MICHAEL: Yeah, precisely. [laughs] Again, I think in general, we've become impoverished in our imagination because again, the expectations, there's a shifting baseline. So what people expect to pull out of the ocean now is a fish that you might catch off just a commercial, or a recreational fishing expedition. It's a quarter of the size of the same species of fish you might've caught 50, 70 years ago and when people pull up this thing and they're like, “Oh, look at –” and they feel proud of themselves. I feel like that's what's going on with us in terms of our we no longer even recognize, or didn't until very recently recognize that we had been unwittingly colluding in the erosion of some very essential levels of organization and human society and that we had basically sold our souls to market efficiency and efficient state level governance. Now it's a huge mess to try and understand. You look at Occupy Wall Street and stuff like that and it just seems like such an enormous pain in the ass to try and process things in that way. But it's because we're having to relearn how to govern neighborhoods and govern small communities and make business decisions at the scale of a bioregion rather than a nation. JESSICA: Yeah. It's a scale thing. I love the phrase topsoil of community relationships, because when you talk about the purposive knowledge that whatever you call it, Rein, that is goal seeking. It's like the one tall tree that is like, “I am the tallest tree,” and it keeps growing taller and taller and taller, and it doesn't see that it's falling over because there's no trees next to it to protect it from the wind. It's that weaving together between all the trees and the different knowledge and the different people, our soul is there. Our resilience is there. REIN: Michael, you keep talking about scale. Are you talking about scale theory? MICHAEL: Yeah. Scaling laws, like Geoffrey West's stuff, Luis Bettencourt is another researcher at the University of Chicago who does really excellent work in urban scaling. I just saw a talk from him this morning that was really quite interesting about there being a sweet spot where a city can exist between how thinly it's distributed infrastructurally over a given area versus how congested it is. Because population and infrastructure scale differently, they scale at different rates than you get – REIN: If I remember my West correctly, just because I suspect that not all of our listeners are familiar with scale theory, there's this idea that there are certain things that grow super linearly as things scale and certain things that grow sub linearly. So for example, the larger a city gets, you get a 15% more restaurants, but you also get 15% more flu, but you also get 15% less traffic. MICHAEL: Yeah. So anything that depends on infrastructures scales sub linearly. A city of 2 million people has 185% the number of gas stations, but anything that scales anything having to do with the number of interactions between people scales super linearly. You get 115% of the – rather you get, what is it, 230%? Something like that. Anyway, it's 150%, it's 85% up versus 115% up. So patents, but also crime and also, just the general pace of life scale at 115% per capita. So like, disease transmission. So you get into these weird cases—and this links back to what we were talking about earlier—where people move into the city, because it's per unit. In a given day, you have so much more choice, you have so much more opportunity than you would in your agrarian Chinese community and that's why Shenzhen is basically two generations old. 20 million people and none of them have grandparents living in Shenzhen because they're all attracted to this thing. But at scale, what that means is that everyone is converging on the same answer. Everyone's moving into Shenzhen and away from their farming community. So you end up – in a way, it's not that that world is any more innovative. It's just, again, easier to capture that innovation and therefore, measure it. But then back to what we were saying about convergent categories and biogeography, it's like if somebody comes up with a brilliant idea in the farm, you're not necessarily going to see it. But if somebody comes up with the same brilliant idea in the city, you might also not see it for different reasons. So anyway, I'm in kind of a ramble, but. JESSICA: The optimal scale for innovation is not the individual and it's not 22 million, it's in between. MICHAEL: Well, I feel like at the level of a city, you're no longer talking about individuals almost in a way. At that point, you're talking about firms. A city is like a rainforest in which the fauna are companies. Whereas, a neighborhood as an ecosystem in which the fauna, or individual people and so, to equate one with the other is a potential point of confusion. Maybe an easier way to think about this would be multicellular life. My brain is capable of making all kinds of innovations that any cell, or organ in my body could not make on its own. There's a difference there. [overtalk] JESSICA: [inaudible]. MICHAEL: Right. It's easier, however, for a cell to mutate if it doesn't live inside of me. Because if it does, it's the cancer – [overtalk] JESSICA: The immune system will come attack it. MICHAEL: Right. My body will come and regulate that. JESSICA: Like, “You’re different, you are right out.” MICHAEL: Yeah. So it's not about innovation as some sort of whole category, again, it's about different kinds of innovation that are made that are emergent at different levels of organization. It's just the question of what kinds of innovation are made possible when you have something like the large Hadron Collider versus when you've got five people in a room around a pizza. You want to find the appropriate scale for the entity, for the system that's the actual level of granularity at which you're trying to look at the stuff, so. REIN: Can I try to put a few things together here in potentially a new way and see if it's anything? So we talked about the edge of chaos earlier and we're talking about scale theory now, and in both, there's this idea of fractal geometry. This idea that a coastline gets larger, the smaller your ruler is. In scale theory, there's this idea of space filling that you have to fill the space with things like capillaries, or roads and so on. But in the human lung, for example, if you unfurled all of the surface area, you'd fill up like a football field, I think. So maybe there's this idea that there's complexity that's possible, that’s made possible by the fractal shape of this liminal region that the edge of chaos. MICHAEL: Yeah. It's certainly, I think as basically what it is in maximizing surface area, like you do within a lung, then you're maximizing exposure. So if the scientific community were operating on the insights that it has generated in a deliberate way, then you would try to find a way to actually incorporate the fringe physics community. There's got to be a way to use that as the reservoir of chaos, rather than trying to shut that chaos out of your hill climbing algorithm and then at that point, it's just like, where's the threshold? How much can you invite before it becomes a distraction from getting anything done? When it's too noisy to be coherent. Arguably, what the internet has done for humankind has thrown it in completely the opposite direction where we've optimized entirely for surface area instead of for coherence. So now we have like, no two people seem to be able to agree on reality anymore. That's not useful either. REIN: Maybe there's also a connectivity thing here where if I want to get from one side of the city to the other, there are 50 different routes. But if I want to get from one city to another, there's a highway that does it. MICHAEL: Yeah, totally. So it's just a matter of rather than thinking about what allows for the most efficient decisions, in some sense, at one given timescale, it's how can we design hierarchical information, aggregation structures so as to create a wise balance between the demands on efficiency that are held at and maintained at different scales. SFI researcher, Jessica Flack talks about this in her work on collective computation and primate hierarchies where it’s a weird, awkward thing, but basically, there is an evolutionary argument for police, that it turns out that having a police system is preventing violence. This is mathematically demonstrable, but you also have to make sure that there's enough agency at the individual level, in the system that the police aren't in charge of everything going on. It's not just complex, it's complicated. [laughs] We've thrown out a ton of stuff on this call. I don't know, maybe this is just whetting people's appetite for something a little bit more focused and concise. JESSICA: This episode is going to have some extensive show notes. MICHAEL: Yeah. [chuckles] JESSICA: It's definitely time to move into reflections. JACOB: You were talking, at the very beginning, about Spotify. Like how, when unknown ideas are able to find their tribe and germinate. I was reading about how Netflix does business and it's very common for them to make some new content and then see how it goes for 30 days and then just kill it. Because they say, “Well, this isn't taking off. We're not going to make more of it,” and a lot of people can get really upset with that. There's definitely been some really great things out on Netflix that I'm like, for one on the one hand, “Why are you canceling this? I really wanted more,” and it seems like there's a lot of the people that do, too. What that's making me think about as well for one thing, I think it seems like Netflix from my experience, is not actually marketing some of their best stuff. You would never know it’s there, just in the way of people to find more unknown things. But also, I'm thinking about how just generally speaking some of the best ideas, TV shows, music, whatever are the kinds of things that there's not going to be an established container, group of people, that you can say, “We want to find white men ages 25 to 35 and we're going to dump it on their home screen because if anyone's going to like it, it's them and if they do, then we keep it and if they don't move, we don't.” I feel like the best things are we don't actually know who those groups are going to be and it's going to have a weird constellation of people that I couldn't actually classify. So I was just thinking about how that's an interesting challenge. JESSICA: Sweet. Rein, you have a thing? REIN: Yeah. I have another thing. I was just reminded of von Foerster, who was one of the founders of Second-order cybernetics. He has an ethical imperative, which is act always so as to increase the number of choices. I think about this actually a lot in my day-to-day work about maximizing the option value that I carry with me as I'm doing my work, like deferring certain decisions and so on. But I think it also makes sense in our discussion as well. JESSICA: True. Mine is about externalities. We talked about how, whatever you do, whatever your business does, whatever your technology does, there's always going to be effects on the world on the context and the context of the context that you couldn't predict. That doesn't mean don't do anything. It doesn't mean look for those. Recognize that there's going to be surprises and try to find them. It reminds me of sometimes, I think in interviewing, we’re like, “There are cognitive biases so in order to be fair, we must not use human judgment!” [laughter] Which is not helpful. I mean, yes, there are cognitive biases so look for them and try to compensate. Don't try to use only something predictable, like an algorithm. That's not helpful. That's it. MICHAEL: Yeah. Just to speak to a little bit of what each of you have said, I think for me, one of the key takeaways here is that if you're optimizing for future opportunity, if you're trying to—and I think I saw MIT defined intelligence in this way, that AI could be measured in terms of its ability to – AGI rather could be measured in terms of its ability to increase the number of games steps available to it, or options available to it in the next step of an unfolding puzzle, or whatever. Superhuman AGI is going to break out of any kind of jail we try to put it in just because it's doing better at this. But the thing is that that's useless if we take it in terms of one spaciotemporal scale. Evolutionary dynamics have found a way to do this in a rainforest that optimizes biodiversity and the richness of feeding relationships in a food web without this short-sighted quarterly return maximizing type of approach. So the question is are you trying to create more opportunities for yourself right now? Are you trying to create more opportunities for your kids, or are you trying to transcend the rivalrous dynamics? You've set yourself up for intergenerational warfare if you pick only one of those. The tension between feed yourself versus feed your kids is resolved in a number of different ways in different species that have different – yeah. It is exactly, Rein in the chat you said, it reminds you of the trade-off between efficiency and adaptability and it's like, arguably, adaptability is efficiency aggregated when you're looking at it over a longer timescale, because you don't want to have to rebuild civilization from scratch. So [chuckles] I think it's just important to add the dimension of time and to consider that this is something that's going on at multiple different levels of organization at the same time and that's a hugely important to how we actually think about these topics. JESSICA: Thinking of scales of time, you’ve thought about these interesting topics for an hour, or so now and I hope you'll continue thinking about them over weeks and consult the show notes. Michael, how can people find out more about you? MICHAEL: I'm on Twitter and Instagram if people prefer diving in social media first, I don't recommend it. I would prefer you go to patreon.com/michaelgarfield and find future fossils podcasts there. I have a lot of other stuff I do, the music and the art and everything feeds into everything else. So because I'm a parent and because I don't want all of my income coming from my day job, I guess Patreon is where I suggest people go first. [laughs] Thank you. JESSICA: Thank you. And of course, to support the podcast, you can also go to patrion.com/greaterthancode. If you donate even a dollar, you can join our Slack channel and join the conversation. It'll be fun. Special Guest: Michael Garfield.

Reading is delivered by Sarah and mesitation delivered by Masdi, from Saint Peter's School in Jakarta, Indonesia. Acts of the Apostles 16: 22-34; Rs psalm 138: 1-2a.2bc-3.7c-8; John 16: 5-11. THE GOSPEL FOR FAMILIES Our meditation today has the theme: The Gospel For Families. When Jesus delivered His teachings and counsels to the apostles and disciples before His ascension to heaven, one of his highlight points was that all His flock should be united. The family is one of the most basic forms of fellowship and unity. All those who form one family of this kind identify themselves as Christians and the gospel of Jesus Christ is the central teaching for their lives at all times. Concretely speaking, when Saint Paul and his associates spread the Gospel to the pagan areas, for example in the city of Philippi which was inhabited by the Roman immigrants, the emphasis that the Gospel as a book for the family was reaffirmed. We can ensure that if the gospel is the primary means of faith for the family, every member must be reached out and included. On the contrary, when only one or a few in the family get to know the Gospel, it does not ensure that the entire family is empowered by the Gospel. The event of the miraculous release of Paul and Silas from their imprisonment, then the prison guard who repented, followed by his entire family being baptized in the name of Jesus Christ, these were some of the happenings that illustrated about the power of the Gospel that transformed the whole family. The inspiration of these events should benefit each of us at this moment and to motivate us, that by the power of the Gospel, our families and our communities will also experience sort of transformation in the growth of our faith. The Gospel for families is a great opportunity for learning and evangelization today in our time. A family that is devoted and actively involved in the sacraments of the Church remains a very important practice as the followers of Christ should be. However, this is not enough to make our families true and active Christians. To contribute to the realization of a true Christian living, an important part of the growth of faith is the evangelization of our families and communities. Specifically, our families and our communities need to give a particular priority to their spiritual life, which is the zeal in communion with the Word of God. A real activity that can be made in regular basis in our families or communities should be the listening, the reading and the study of the scriptures that are done together. We don't have to wonder on the kind of reading and study of the scriptures would be like. The Lord Jesus certainly gives us the way. He assures us that the Holy Spirit, the Comforter, will enlighten and teach us in all truth. The Holy Spirit is always there and willing, then it will depend on our families and communities to determine together on how it will happen. Let's pray. In the name of the Father ... O God almighty, strengthen us in the realization of the Gospel for our families and communities. Our Father who art in heaven ... In the name of the Father ... --- Send in a voice message: https://anchor.fm/media-la-porta/message

Find out more on our website: https://bit.ly/3sK9qEN Part of the Swiss UK Dialogue series with the Embassy of Switzerland in the UK Both Switzerland and the UK are leading green finance centres according to the Global Green Finance Index. The event aims at having a closer look at the key success factors and challenges of leading green finance centres. This topic is particularly relevant in view of COP26 later this year. By having leading Swiss and UK representatives on the panel, we can look forward to a stimulating discussion with new ideas. Speakers: Sabine Döbeli has worked in the field of sustainable finance for over 20 years in different roles. She is the CEO of Swiss Sustainable Finance (SSF), an organization that was established under her lead in 2014 with the objective of making Switzerland a leading voice and actor in sustainable finance. Previously, she was Head of Corporate Sustainability Management at Vontobel where she was responsible for the sustainability strategy at group level and sustainable investment services in asset management. At Zürcher Kantonalbank she built up the sustainability research unit within the financial research department and contributed to the launch of various sustainable investment products. A recent report from Swiss Sustainable Finance, "Financing instruments for a low carbon economy" Claudia Bolli is responsible for the development and the implementation of the Responsible Investing strategy with Swiss Re since 2014. As part of that, she was instrumental in Swiss Re's switch its investment portfolio to ESG benchmarks as one of the first re-/insurance companies. In June 2018, she was selected by the European Commission to join the Technical Expert Group on Sustainable Finance. Concretely, she participated in the development of the requirements for the "Climate transition and Paris-aligned" benchmarks as well as the ESG disclosure on benchmarks. Dr Ben Caldecott is the founding Director of the Oxford Sustainable Finance Programme at the University of Oxford Smith School of Enterprise and the Environment. At the University of Oxford, he is the inaugural Lombard Odier Associate Professor and Senior Research Fellow of Sustainable Finance, the first ever endowed professorship of sustainable finance, and a Supernumerary Fellow at Oriel College. Ben is also the founding Director and Principal Investigator of the UK Centre for Greening Finance & Investment (CGFI), established by UK Research and Innovation in 2021 as the national centre to accelerate the adoption and use of climate and environmental data and analytics by financial institutions internationally. Since 2019, he has also been seconded to the UK Cabinet Office as the COP26 Strategy Advisor for Finance.

Anti-aging compound improves muscle glucose metabolism in people Washington University School of Medicine in St. Louis, April 25, 2021 A natural compound previously demonstrated to counteract aspects of aging and improve metabolic health in mice has clinically relevant effects in people, according to new research at Washington University School of Medicine in St. Louis. A small clinical trial of postmenopausal women with prediabetes shows that the compound NMN (nicotinamide mononucleotide) improved the ability of insulin to increase glucose uptake in skeletal muscle, which often is abnormal in people with obesity, prediabetes or Type 2 diabetes. NMN also improved expression of genes that are involved in muscle structure and remodeling. However, the treatment did not lower blood glucose or blood pressure, improve blood lipid profile, increase insulin sensitivity in the liver, reduce fat in the liver or decrease circulating markers of inflammation as seen in mice. The study, published online April 22 in the journal Science, is the first randomized clinical trial to look at the metabolic effects of NMN administration in people. Among the women in the study, 13 received 250 mg of NMN orally every day for 10 weeks, and 12 were given an inactive placebo every day over the same period. "Although our study shows a beneficial effect of NMN in skeletal muscle, it is premature to make any clinical recommendations based on the results from our study," said senior investigator Samuel Klein, MD, the William H. Danforth Professor of Medicine and Nutritional Science and director of the Center for Human Nutrition. "Normally, when a treatment improves insulin sensitivity in skeletal muscle, as is observed with weight loss or some diabetes medications, there also are related improvements in other markers of metabolic health, which we did not detect in our study participants." The remarkable beneficial effects of NMN in rodents have led several companies in Japan, China and in the U.S. to market the compound as a dietary supplement or a neutraceutical. The U.S. Food and Drug Administration is not authorized to review dietary supplement products for safety and effectiveness before they are marketed, and many people in the U.S. and around the world now take NMN despite the lack of evidence to show clinical benefits in people. The researchers studied 25 postmenopausal women who had prediabetes, meaning they had higher than normal blood sugar levels, but the levels were not high enough to be diagnosed as having diabetes. Women were enrolled in this trial because mouse studies showed NMN had the greatest effects in female mice. NMN is involved in producing an important compound in all cells, called nicotinamide adenine dinucleotide (NAD). NAD plays a vital role in keeping animals healthy. Levels of NAD decline with age in a broad range of animals, including humans, and the compound has been shown to contribute to a variety of aging-associated problems, including insulin resistance in studies conducted in mice. Supplementing animals with NMN slows and ameliorates age-related decline in the function of many tissues in the body. Co-investigator Shin-ichiro Imai, MD, Ph.D., a professor of developmental biology and of medicine who has been studying NMN for almost two decades and first reported on its benefits in mice said, "This is one step toward the development of an anti-aging intervention, though more research is needed to fully understand the cellular mechanisms responsible for the effects observed in skeletal muscle in people." Insulin enhances glucose uptake and storage in muscle, so people who are resistant to insulin are at increased risk for developing Type 2 diabetes. But the researchers caution that more studies are needed to determine whether NMN has beneficial effects in the prevention or management of prediabetes or diabetes in people. Klein and Imai are continuing to evaluate NMN in another trial involving men as well as women.   High dose of vitamin D fails to improve condition of moderate to severe COVID-19 patients   University of São Paulo's Medical School (Brazil), April 26, 2021 Can a high dose of vitamin D administered on admission to hospital improve the condition of patients with moderate or severe COVID-19? The answer is no, according to a Brazilian study published in the Journal of the American Medical Association (JAMA). The article reports a randomized, double-blind, placebo-controlled clinical trial, the kind of study considered the gold standard to evaluate drug efficacy. It was conducted with FAPESP's support by researchers at the University of São Paulo's Medical School (FM-USP), who recruited 240 patients treated at Hospital das Clínicas (HC), the hospitalcomplex run by FM-USP, and the Ibirapuera field hospital in São Paulo City in June-August 2020. "In vitro studies or trials with animals had previously shown that in certain situations vitamin D and its metabolites can have anti-inflammatory and anti-microbial effects, as well as modulating the immune response. We decided to investigate whether a high dose of the substance could have a protective effect in the context of an acute viral infection, reducing either the inflammation or the viral load," Rosa Pereira, principal investigator for the project, told Agência FAPESP. The volunteers were randomly divided into two groups, one of which was given vitamin D3 in a single dose of 200,000 units (IU) dissolved in a peanut oil solution. The other group was given only the peanut oil solution. All participants were treated according to the standard protocol for hospital treatment of the disease, which includes administration of antibiotics and anti-inflammatory drugs. The main purpose was to see if acute supplementation would affect the length of hospital stay for these patients, but the researchers also wanted to find out whether it would mitigate the risks of admission to the intensive care unit (ICU), intubation and death. No significant difference between the groups was observed for any of these clinical outcomes. According to Pereira, the study was designed above all to assess the impact on hospital stay and a larger number of volunteers would be needed to achieve a scientifically acceptable estimate of the effect on mortality. "So far we can say there's no indication to administer vitamin D to patients who come to the hospital with severe COVID-19," she said. For Bruno Gualano, a researcher at FM-USP and penultimate author of the article, the findings show that at least for now there is no "silver bullet" for the treatment of COVID-19. "But that doesn't mean continuous use of vitamin D can't have beneficial effects of some kind," he said. Ideal dose Pereira is currently leading a study at FM-USP to find out whether subjects with sufficient circulating levels of vitamin D combat infection by SARS-CoV-2 better than those with insufficient levels of the nutrient. The ideal level of vitamin D in the blood and the daily supplementation dose vary according to age and overall health, she explained. Older people and patients with chronic diseases including osteoporosis should have more than 30 nanograms per milliliter of blood (ng/mL). For healthy adults, 20 ng/mL is an acceptable threshold. "The ideal approach is case-by-case analysis, if necessary dosing the substance periodically by means of blood work, with supplementation if a deficiency is detected," Pereira said.         Sufficient serum vitamin D before 20 weeks of pregnancy reduced risk of gestational diabetes mellitus Fudan University Obstetrics and Gynecology Hospital (China), April 16, 2021 A new study on Nutritional and Metabolic Diseases and Conditions - Obesity and Diabetes is now available. According to news originating from Shanghai, People’s Republic of China, by NewsRx correspondents, research stated, “Our aim was to evaluate the relationship between serum vitamin D levels before 20 weeks of pregnancy and the risk of gestational diabetes mellitus. This study is a retrospective study.” Our news journalists obtained a quote from the research from Fudan University Obstetrics and Gynecology Hospital, “We analyzed the relationship between serum 25 (OH) D level before 20 weeks of pregnancy (first antenatal examination) and the risk of gestational diabetes mellitus. Age, parity and pre-pregnancy body mass index were used as confounding factors. 8468 pregnant women were enrolled in this study between January 2018 and March 2020 at the Obstetrics and Gynecology Hospital of Fudan University. Adjusted smoothing splinespline plots, subgroup analysis and multivariate logistic regression analysis was conducted to estimate the relative risk between 25(OH)D and gestational diabetes mellitus. After fully adjusting the confounding factors, serum vitamin D is a protective factor in gestational diabetes mellitus (OR=0.90). Compared with vitamin D deficiency, vitamin D insufficiency (OR=0.78), sufficience (OR=0.82) are a protective factor for gestational diabetes mellitus. Sufficience vitamin D before 20 weeks of pregnancy is a protective factor for gestational diabetes mellitus. Vitamin D>20 ng/mL can reduce the risk of GDM, which is not much different from the effect of >30 ng/mL.” According to the news editors, the research concluded: “The protective effect of vitamin D is more significant in obese pregnant women.”     Review summarizes known links between endocrine disruptors and breast cancer risk University of Eastern Finland, April 20, 2021 Exposure to certain endocrine-disrupting chemicals could elevate the risk of breast cancer, according to a new comprehensive systematic review of epidemiological research. However, for many chemicals, evidence is inconsistent or still limited. The review was carried out by researchers at the universities of Hong Kong and Eastern Finland and published in Critical Reviews in Food Science and Nutrition. Endocrine-disrupting chemicals (EDCs) can interfere with the body's hormonal system, also called the endocrine system, and are widely present in the environment. They originate from a variety of sources, including pesticides, plasticisers and other industrial and pharmaceutical chemicals, as well as natural sources. Humans are often exposed to EDCs through food, but other possible exposure routes include drinking water, skin contact and air. Breast cancer accounts for the majority of women's cancers. There has been an increasing interest in the role of estrogene-mimicking EDCs, so called xenoestrogens, in the development of breast cancer. They comprise a broad range of pesticides, synthetic chemicals, phytoestrogens and certain mycotoxins. The researchers reviewed 131 epidemiological studies evaluating the link between xenoestrogen exposure and breast cancer. Most studies assessed exposures by measuring the EDCs and their metabolites in urine, serum, plasma or adipose tissues.  Some may be genetically more vulnerable to EDCs According to the review, the nowadays widely banned pesticide DDT is one of the most studied EDCs in relation to breast cancer risk. Out of 43 epidemiological studies, eleven reported positive associations between DDT or its metabolites in lipid, serum or plasma and breast cancer incidence. Nine reported higher DDT levels among women with breast cancer than among controls. In a few studies, DDT was linked to estrogen-positive breast cancer or the association to breast cancer risk depended on genotype.  Polychlorinated biphenyls, PCBs, are a large group of compounds earlier much used in electrical devices, surface coatings and other purposes. The review of 50 studies found the association between total PCBs and breast cancer risk to be inconsistent. However, 19 studies linked certain PCBs to a higher breast cancer incidence. Similar to DTT, PCBs accumulate in the adipose tissue and in the food chain and can be excreted in breast milk.  Perfluorooctanoid acid (PFOA) found in some food packaging and cookware was linked to breast cancer risk in three out of five epidemiological studies. Some studies found an association between cancer risk and certain genotypes both for PCBs and PFOAs. DDT, PCBs ja PFOA are POP substances, persistent organic pollutants, the use of which is strictly regulated. DDT ja PCBs are old POP substances and their levels in the environment are decreasing. PFOA is a newer POP substance. Phytoestrogens were found beneficial in some, but not all studies Phytoestrogens are natural plant estrogens that have been suggested to prevent breast cancer. Genistein is a phytoestrogen found in soy products. The review included 29 epidemiological studies focusing on genistein, 18 of which linked it to a lower breast cancer risk, although some only in certain age groups or populations. For most EDCs included in the review, the link to breast cancer has been investigated in only a few epidemiological studies. Phtalates and bisphenol A (BPA), for example, are used in plastic packaging and can transfer to food. According to the review, four out of six studies linked phthalates to increased breast cancer risk. BPA was linked to more aggressive tumours in one study, but two other epidemiological studies found no link to breast cancer. Parabens are common preservatives in foods and cosmetic products and considered possible endocrine disruptors. The only epidemiological study on the topic reported a link between paraben exposures, breast cancer risk and mortality following breast cancer.  Oral contraceptive use was linked to an increased breast cancer risk in seven out of eight epidemiological studies, but there were controversies on how duration or discontinuation of oral contraceptive use affected the risk. The review also included the herbicide atrazine, the industrial by-product dioxine, mycotoxins produced by food and crop molds, and PBDEs found in household furniture coatings and appliances, but epidemiological studies on their links to breast cancer risk were still scarce and often inconsistent.  The writers point out that for EDCs to disrupt endocrine functions, dose, time, duration and age at exposure all matter. In addition, as multiple EDCs coexist in the environment, more research is needed to evaluate their interactive effects on breast cancer risk. The review also suggests that genotypes could determine whether EDC exposure affects breast cancer risk, and more research is needed on this topic. "One example is the polymorphism of the CYP1A1 gene, which is responsible for estrogen metabolism." According to the writers, next-generation technologies, such as genome sequencing, proteomics or epigenomics, can help identify new exposure biomarkers with better sensitivity and specificity. "These technologies will also pave way to better assessment of past exposure and prediction of future risks, by taking into account an individual's genetic profile."     Grape seed extract may protect gut from inflammation: Study Universitat Rovira i Virgili (Spain), April 25, 2021 Proanthocyanidin-rich grape seed extracts may protect the intestines from the deleterious effects of a high-fat/high-carbohydrate diet, according to data from a rat study. A high-fat/high-carbohydrate diet or Western diet has been reported to produce changes in the intestine, explained researchers from the Universitat Rovira i Virgili in Tarragona, Spain. “Concretely, several recent studies have provided compelling new evidence to suggest that changes in the epithelial barrier function and intestinal inflammation are associated with and could even lead to altered regulation of body weight and glucose homeostasis,” they added. “The main consequence of the gut barrier dysfunction has been proposed to be the entry of toxins from the intestinal lumen, which can trigger local inflammation or gain access to the circulation and induce systemic inflammation through cytokine release.” Their new research indicated that a grape seed proanthocyanidin extract (GSPE) may protect the gut from such harmful effects. Study details Data published in Molecular Nutrition and Food Research reveals that supplementing the diet of lab rats with medium or high-dose proanthocyanidins had beneficial impacts on intestinal inflammation, oxidative stress, and barrier function. The medium dose was 25 mg/kg, which is a dose similar to the dietary proanthocyanidin intake in humans, explained the researchers. The high dose (50 mg/kg) would exceed the dietary proanthocyanidin intake in humans. Thirty-six week-old rats were fed a Western diet for 15 weeks and then divided into one of four supplementation groups, receiving 0 mg/kg (control), 5 mg/kg (low dose), 25 mg/kg, or 50 mg/kg for an additional three weeks. Results showed that intestinal inflammation, assessed by measuring myeloperoxidase (MPO) activity, significantly increased in the control animals, but these increases were reduced in the rats receiving the grape seed extract. In addition, significant reductions in plasma levels of reactive oxygen species were observed in the medium and high dose groups, compared to the control group. Tight junctions The researchers also examined the function of the intestinal barrier, and looked specifically at so-called tight junctions (TJ) between cells in the lining of the intestine – the epithelium. On one side is the intestinal cavity and on the other is a mass of cells and tissues. In a healthy system, materials in the cavity find their way into tissues by passing through the cells, which controls which substances pass through. In an unhealthy system, the tight junctions are not so tight and materials can bypass the cells and find their way into tissues via the tight junctions. This increase in intestinal permeability has been referred to as "leaky gut". “Another point of interest in this study was to evaluate whether GSPE could modulate the alterations in the permeability of the intestinal barrier that are related to the state of intestinal inflammation,” wrote the researchers. “Our findings indicate that the TJ  proteins were negatively associated with measures of adiposity and with the circulating levels of [triglycerides]. These are not causal associations, but they suggest that increased adiposity is accompanied by lower expression of TJ components, which is in agreement with the hypothesis that obesity and a [high-fat diet] are associated with increased intestinal permeability. Then, given the importance of having a healthy barrier function, dietary interventions that can modulate the intestinal permeability might afford an effective tool for the prevention and treatment of metabolic diseases associated with obesity.” The researchers concluded: “Our findings indicate that orally administered GSPE modulates the intestinal inflammation, oxidative stress, and possibly the barrier function. Based on these findings, our data suggest that nutritional and/or therapeutic interventions focused on gut health and modulation of the intestinal permeability should be extensively explored in the context of obesity.”   Antidepressant use in pregnancy tied to affective disorders in offspring; no causal link   Mount Sinai Hospital, April 12, 2021 Major depressive disorder is highly prevalent, with one in five people experiencing an episode at some point in their life, and is almost twice as common in women than in men. Antidepressants are usually given as a first-line treatment, including during pregnancy, either to prevent the recurrence of depression, or as acute treatment in newly depressed patients. Antidepressant use during pregnancy is widespread and since antidepressants cross the placenta and the blood-brain barrier, concern exists about potential long-term effects of intrauterine antidepressant exposure in the unborn child.  Using the Danish National Registers to follow more than 42,000 singleton babies born during 1998-2011 for up to 18 years, researchers at the Icahn School of Medicine at Mount Sinai investigated whether exposure to antidepressants in the womb would increase the risk of developing affective disorder like depression and anxiety in the child. In a study published April 5 in Neuropsychopharmacology,the scientists found that children whose mothers continued antidepressants during pregnancy had a higher risk of affective disorders than children whose mothers stopped taking antidepressants before pregnancy. However, to understand whether the underlying disorder for which the antidepressant was given or the medication itself was linked to the child's risk of developing an affective disorder, they also studied the effect of paternal antidepressant use during pregnancy and similarly, found that children of fathers who took antidepressants throughout pregnancy had a higher risk for affective disorders. Thus, the research team speculates that rather than being an intrauterine effect, the observed link is most likely due to the parental mental illness underlying the antidepressant use.  "Approximately half of women who use antidepressants before pregnancy decide to discontinue use either before or during pregnancy due to concerns about the negative consequences for their child," said Anna-Sophie Romel, PhD, an instructor in the Department of Psychiatry at Icahn Mount Sinai and first author of the paper. "Our study does not provide evidence for a causal relationship between in-utero exposure to antidepressants and affective disorders in the child. So, while other long-term effects of intrauterine exposure to antidepressants remain to be investigated, our work supports antidepressant continuation for women with severe symptoms or a high risk of relapse because untreated psychiatric illness during pregnancy can have negative consequences on the health and development of the child. Women and their health care providers should carefully weigh all of the treatment options and jointly decide on the best course of action."   Staying Active Can Fight Declines in Cognitive Engagement   North Carolina State University, April 22, 2021   Preserving physical and mental health helps older adults experiencing cognitive impairment stave off declines in cognitive engagement, a new study suggests “We found that declines in physical and mental health were associated with more pronounced cognitive disengagement,” says Shevaun Neupert, professor of psychology at North Carolina State University and corresponding author of the study published in Entropy. “The impact of declines in physical health was particularly pronounced for study participants who had more advanced cognitive impairment to begin with.” There’s a lot of research showing that cognitive engagement can help older adults maintain cognitive health. However, the vast majority of that work has been done on healthy adults. “There’s very little work on cognitive engagement in people who are already cognitively impaired, such as people who have been diagnosed with dementia,” Neupert says. “Are they still capable of sustained cognitive engagement? What factors contribute to that engagement?” To begin addressing those questions, the researchers enlisted 28 study participants. All of the participants were over 60 and had documented cognitive impairment. Participants came to a testing site two times, six months apart. On each visit, researchers collected data on the physical and mental health of the study participants and performed a battery of tests designed to assess cognitive ability. They also connected participants to a device that tracked blood pressure continuously and then asked them to engage in a series of increasingly difficult cognitive tasks. This allowed researchers to track how cognitive engagement changed as the tasks become progressively harder. Cognitive engagement means taking part in activities that are mentally challenging. Monitoring blood pressure allows the researchers to track how hard study participants are working to accomplish cognitive tasks. Specifically, blood pressure rises as more blood is pumped to the brain when participants work harder at these tasks. Broadly speaking, the researchers found that if a participant’s cognitive ability, physical health, or mental health declined over the course of the six month study period, that participant became less cognitively engaged as the tasks became harder. “Normally, you’d expect more engagement as the tasks became harder, but we found that some people essentially stopped trying,” says coauthor Claire Growney, a postdoctoral researcher at Washington University in St. Louis. “The findings highlight the fact that well-being is holistic; physical health, mental health, and cognitive function can influence each other,” says coauthor Xianghe Zhu, a recent PhD graduate of NC State. “In practical terms, it suggests that it may be particularly important for people to focus on mental and physical well-being during the early stages of cognitive decline,” Growney says. “Or, at the very least, don’t become so focused on addressing cognitive challenges that you ignore physical health, or create anxiety or emotional distress for yourself that leads to mental health problems.” “Future research will be needed to determine how beneficial it might be for people to take part in cognitively engaging activities once they’ve started experiencing cognitive decline,” Neupert says. “But we already know that there is an element of ‘use it or lose it’ to cognitive function in healthy adults. And while it’s understandable for people to want to avoid tasks that are difficult or challenging, it’s really important to continue challenging ourselves to take part in difficult cognitive activities.”

Visit Pampa on InstagramSupport the show-------->Fabian Geyrhalter:Welcome to the show, Emmanuelle.Emmanuelle Magnan:Hello, thank you for having me.Fabian Geyrhalter:Absolutely. It's a great pleasure. You are calling in from Paris, and almost guaranteed, most of my listeners will not be familiar with your brand, Pampa, and that is because you are a boutique floral arranging and delivery service in Paris. So, not US-based or international, but your Instagram is exploding, and the New York Times has even found you, and now so have I.I even believe that this is your first solo audio interview in English, which flatters me. This is really cool. I read somewhere that you or your partner, one of the two of you, said, "We are always trying to twist the flower world. That has been old school for a while." Tell us a bit about how Pampa started in 2016, what the inspiration was, and why you and your co-founder thought Paris needed yet another flower store.Emmanuelle Magnan:Yes. Well, I should start by presenting Pampa and telling you a bit what it is exactly, because precisely, I think we're more than another flower store. We are a new kind of flower brand, englobing a flower shop and a creative studio that is dedicated to companies and brands. We started out in 2016, as you mentioned. We started a small, online flower shop.And really quickly, we started working with the coolest and most prestigious brands, artists, et cetera. And I'd say we have a new approach to flowers and the flower industry, more modern, more colorful, more eco-friendly, also, as much as we have the possibility to be.Concretely, what do we do? We are a B2C and B2B company. On the B2C side, we are a flower shop. We sell fresh flowers on weekly arrangements we have ... Sorry. We have weekly arrangements available in three sizes, that's it, and we also deliver dried flowers. And on the B2B side, we are a studio, as I said, and we have three types of services, which are design, gifting, and even workshop animation.Basically, when we started out, what we wanted to do first was disrupting the online market. At that time, it was dominated by three major competitors that had been there for ages, and that were, in our opinion, not really in phase with the needs and tastes of our generation. They had been on the market for ... Some of them 20 years. They were one of the oldest companies on the internet, maybe one of the first companies, e-commerce companies.When we were developing the business and benchmarking, we had four main observations. One, there was no clear differentiation from a competitor to another, like they were ... If you went from one competitor to another, they had the same website, same kind of products, same level of services. Secondly, their product offering were always on the ... built, sorry, on the same model.Fabian Geyrhalter:Mm-hmm (affirmative).Emmanuelle Magnan:They all had dozens of different arrangements in different shapes, colors, with various types of flowers. Some were exotic, some were red roses. You didn't ... It was kind of slow and tedious customer experience. You would arrive on the website, and you would be like, "Oh my god, there are so many different kind of arrangements. I don't know what to choose." And we figured out that there was another problem with that, that was stock management issues and high waste potential.Fabian Geyrhalter:Right.Emmanuelle Magnan:Right. So then, the third observation that we had is that there was a big lack of transparency. You didn't really know who makes the arrangements, how are they delivered, who is behind those big platforms, et cetera. And finally, we figured that there were actually no strong brands. Yes, there was Interflora, that is one of the oldest ... It's the oldest service. It was there before the internet, by phone and so on.It's a brand, because everyone knows it, but there was no ... For us, it was missing this modern thing about it that we, the young generation, wanted to identify with. You know?Fabian Geyrhalter:Yeah. Oh, yeah.Emmanuelle Magnan:So, we thought, "Okay, competition is strong in this market. There are already hundreds of florists on the streets. Our value proposition should be, first of all, one weekly arrangement, available in three sizes, delivered by bicycle." Actually, nobody did that at that time, it was really bold to go on the market that way, because ... Well, we didn't want to repeat the same mistakes.For us, it was a kind of mistake to propose on the internet so many different arrangements, right? So, we thought we could do as new ... In restaurants, the new kind of chief, what they do is that they go to the market, and they see what is available at the market, and they come back to the restaurant and they just design one single menu. You know?Fabian Geyrhalter:Farm-to-table. Yeah. Exactly.Emmanuelle Magnan:You know what I mean?Fabian Geyrhalter:Yeah.Emmanuelle Magnan:And so we thought we could do the same with flowers, so that's what we wanted to ... the spirit of what we wanted to design. And secondly, we thought, "Okay, let's do a mobile-friendly website on which you can order your arrangement in three minutes." Thirdly, we wanted to have a more humanized and less product-centric approach, be it on the communication or customer support level. Those brands were ... Well, those platforms were a little bit on Instagram, they were starting to be, but everything was so product-centric, you would see the arrangement, that's all, on a table in a vase.And what we wanted to bring was people in the way we would communicate around the product. It's not just about flowers, it's about people who make them, people who consume them. Consume is maybe not the right word, but people who buy them, et cetera.Fabian Geyrhalter:Mm-hmm (affirmative).Emmanuelle Magnan:And also, we wanted to be really close to our customer, and be at their service. At the beginning, we did ... I mean, we still do that, actually, we do everything we can to deliver an arrangement, because it's so important. When you want to offer flowers, you're putting your heart into it, right? So, we will do everything in our power to make this delivery possible.Lastly, what we wanted to do was an emphasis on branding, with our brand codes coming from other disciplines, like fashion, for instance. That's what we ... I mean, that was the basic concept. And that's how we launched. I don't know if it's clear.Fabian Geyrhalter:It is very clear. You just checked off all of the boxes that new companies and startups need to check off when they launch a business, so this is really, really great. And I mean, it's music to my ears. When I said, "Why did you think Paris needed yet another flower store?" Right? Because it is so well known, the Parisian flower markets, and it just ... It goes hand-in-hand with it, but everyone is doing the same, and you just really looked at every aspect of the business and said, "How should this be done in 2021? What would people really need today, and how can we work on sustainability? And how can we bring up a different style?"Talking about your style, your actual brand style, that's how I found you. I literally think I was just scrolling through Instagram and I found one of your arrangements, and then I saw the logo and I saw the colors and I'm like, "Wow, that's wild. That's different." And that's how I literally reached out to you, without knowing too much about the brand. Then, the more I read, the more excited I got to have you on.Your style, the brand style, and the style of the bouquets, it is super eccentric. It's loud, it's graphic, and the New York Times had this article where it said ... I don't know exactly what the article was, but it basically talked about, "Here are 10 cities," and the best, most different flower shops in each one of the cities. And Pampa is part of it, and when you literally scroll through that article, yours is the only brand that stands out.I mean, you scroll through it, and suddenly it's like, "Whoa!" Because it is, it is so loud. Right? You can't mistake your bouquets for anyone else's because they're so distinct. So, I'm wondering, since you are also ... You're a co-founder and you're a creative director at Pampa. Tell us how this unique style came about. I mean, it is very specific. It's very distinct.Emmanuelle Magnan:Right. Well, first of all, thank you for saying all this. You can't really see it, but I have a big smile on my face because it's such ... It's all my work and all my heart are into this, and we worked so hard to develop this brand, and this new kind of product, and to innovate and to be singular and one of a kind. I mean, we work a lot on it, but it's also so natural. I think our team is made of people from various horizons, and they have been working in different disciplines before. They come from architecture, interior design, circus, so it's ... My co-founder, she comes from the events ... How can I say that?Fabian Geyrhalter:Event coordinating? Emmanuelle Magnan:... for big events and yeah, event organizations, right? So we are all very creative, and I think it's also thanks to that team and this mix of profiles that we are able to create such powerful brand. But where it came from, I think ... Well, so I'm the creative director, right? And I co-founded the company with Noélie, and when we started working on the project I had already so many mood boards, and I had been thinking about this project for so many years, because I was a flower lover, and I was also very ...Well, I wanted to create a brand, that was my passion, also. Anyway, I'm color obsessed and very sensitive to color schemes and harmonies, so there's a big work on color associations. Everything we do is a research in colors, there's a big work on color associations. Everything we do is like a research in colors, and another big characteristics of our style is that in every arrangement we do, we mix a dozen of different flower varieties. It gives these wide, large spirits, and I think it's a real, and that's what we wanted to create, it's a real experience to look at our arrangement, because you have so many things to look at.I think what we're looking for is what we say in French, [French 00:13:48], this extra-special something that will twist an arrangement. Sometimes, we put disco balls or feathers, glitter. We are playing with flowers as ... And thank you for saying that, I think it didn't ... It hasn't been done before, I think. We are exploring creativity in flowers, we are-Fabian Geyrhalter:It's your canvas. The flowers are your canvas, right? And then, you just start-Emmanuelle Magnan:Yes, it's ...Fabian Geyrhalter:Yeah.Emmanuelle Magnan:Exactly. It's colors and flowers, and I'd like to say that you can actually think of our arrangement a little bit like an outfit. It's a set of colors that go together. Imagine if the stripes of your shirt was matching your shoelace, and your hat was matching your belt, et cetera. We like to create ... Yeah, a color scheme and the flowers are a big inspiration. We are trying to find the most singular and always with small details, that maybe we are the only one to see, but we are trying to ...There are thousands of different flowers, and we are trying to show the variety of it, and we are trying not to use what every florist use, actually, and has been using for so many years. We are trying to show the beauty of flowers with our arrangements.Fabian Geyrhalter:I saw one interview with you in a French publication, I believe it was, where you talked about how one of your bouquets was inspired by a Nike Air Max. And I'm wondering [crosstalk 00:15:51]. So your inspiration, really, comes from anywhere. And like you said, it's very close to fashion, and you talked about you have an architect on the team, and it's really...It's kind of like this multi-art-inspired endeavor when you start working on your arrangements. Are those moments of inspiration, like a Nike Air Max, are those being shared with your audience? Do they actually weave into your storytelling, or is that more something that happens behind the scenes?Emmanuelle Magnan:I think it's both. Sorry. I think it depends. About the Nike shoe, actually, it's because Nike launched a shoe that was designed by women, for women, and they contacted us to ...Fabian Geyrhalter:Oh, how cool.Emmanuelle Magnan:... to launch for the ... Yeah, we were so proud.Fabian Geyrhalter:That is awesome. Emmanuelle Magnan:Yeah. In terms of branding, it's like Nike is the ...Fabian Geyrhalter:That's it, that's the Holy Grail. It can't get bigger than this. Yeah, yeah, yeah, so you literally did an actual project for Nike to create arrangements.Emmanuelle Magnan:Yeah. We worked in one of their Parisian store. We made a flower installation, and we were also there to ... for giveaways. If you would buy this pair of shoe, of Nike shoe, this precise pair of shoe, you would get a Pampa gift, a Pampa flower gift. So, there was this activation. And also, they partnered with a fashion Instagram magazine. Well, fashion magazine that is based on Instagram, and they asked for artists to create a piece inspired from the shoe.I was one of the selected artists, and it was so fun to make. And so, everything can be an inspiration because I think it's, again, this work on color and texture, and we are also very ... We come from, as I said, Noélie come from the music industry, the events' industry, and I used to work also in that industry when I was in agencies, and we love music. We love party, we love celebration, and all this kind of ... We take inspiration in so many different things, actually.And yeah, we share that with our community, to answer your question. On Instagram, we share a lot of things. We show a lot of behind the scenes, and at some point ... We don't do it that much anymore, but I think I should do that again at some point. Each time we would do a new weekly arrangement, each week we would associate it with a music that we like.So in our newsletter, it will have the song of the week, and stuff. And yeah, and sometimes I can be inspired, sometimes ... I don't know, we did a '90s-inspired arrangement with psychedelic colors and ... So yeah, I think it's a mix of pop culture and arts, visual arts, and flowers-Fabian Geyrhalter:And I think knowing that story behind the bouquet it's just so exciting, and it's so much fun to know that there's actual thought being put into it. Right? Which is the exact opposite of those Interflora, huge floral shops, because there is no ... I mean, yes, they work together and they're all nice, but there's no huge inspiration behind it. There's no story behind it, and I think that's what ... That's what creates brands, period. Right? When there is a story behind it, so I think that that's really exciting.I love how you started talking a little bit about your background, because you come from the agency background. You worked at TBWA, which is actually where my wife also worked, but she worked in the LA office, and you worked at all kinds of fantastic agencies. And that career, actually, being organized, overseeing projects in the creative field, I'm sure that shaped the ability for you to create, but also to run Pampa, right?Emmanuelle Magnan:Right, for sure. Exactly. I started working in agencies as a project manager after I graduated from business school. During that time, I worked with such talented creative directors, communicators, brand strategists. It really allowed me to build myself a brand culture, also have a keen eye on aesthetics, actually, and a sense of style in all direction.I was working with people who have such an incredible taste, and educated me in that way. That's when I started doing mood boards, and that's also where I learned how to code CSS, how to use InDesign, Photoshop and so on. I learned by myself, because I was ... Actually, I was just dreaming to be on their side. I wanted to be the art director or the creative director, but I was a project manager.And actually, it was such a good training. I mean, being a project manager gave me this ability of anticipating everything and every problem.Fabian Geyrhalter:Totally.Emmanuelle Magnan:Yeah, and it gave me the skills of prioritizing, organizing, working with methodology and excellence, because you always, when you work in an agency, you have to be very exigent so that your client is satisfied, and you have to be very ... rigorous? I don't know if-Fabian Geyrhalter:Yeah, yeah, yeah. Rigorous, yeah.Emmanuelle Magnan:And yeah, and it helps me every day in my job, in management, and also in creative direction. I'm so glad I did that in my previous professional life, and yeah, when I was in ... I think it was before I went to TBWA? Yeah, I started thinking about this flower project. I was passionate about flowers, I used to offer a lot of flowers to my friends, my mom who was not in Paris. She was in another city, so I experienced a lot those websites we were talking about before.Fabian Geyrhalter:Yeah.Emmanuelle Magnan:One of the first agencies I worked at, I met Noélie, my co-founder. We were together working on a music festival, and we actually run into each other few years later, and we started talking and we figured we both wanted to launch a product in the flower industry, a project, sorry, in the flower industry. And that's when I decided to stop my ... Well, I found this job.I had this job at TBWA, so I went there, because I thought I could learn ... still learn so much, and ... Sorry. I got a little bit lost. I took this job at TBWA because I thought I could still learn and have some money, save some money so I could start my business.Fabian Geyrhalter:Right.Emmanuelle Magnan:And then, I decided to quit and to launch Pampa, because we had this idea and we believed in it so much that I told myself, "Okay, we only live once. You have this idea, just go for it, and go express your creativity," because yeah, being a project manager was great, but at some point I just wanted to be on the creative side and express myself.Fabian Geyrhalter:Well, and being a project manager, it's stressful, but you don't see the rewards as much, right? Now, it is stressful, but you actually see the rewards. Right? You create your own journey, which is just amazing, and that's the beauty of entrepreneurship. Right? I think it's really interesting that you actually quit your job before you really started the project.I mean, you always thought about it, but to me, this is so important. I did things like that in my life, where I basically had to completely quit something and have this risk, this huge risk of like, "Well, what if it doesn't work out? What if the next thing ... "But because you have that, that fire behind you of, "Okay, I only have that many months to go," financially, "This really needs to work out. I'm putting money in, I put all my energy in." I think you have so much more of a drive than if you do two things at the same time. You still have your day job, you start working on it at night, like it's a very, very different thing, so it's cool to hear that.Let's move over to social media, because that's a huge part of ... In my eyes, I believe, but I'm not sure, I believe that that's a huge part of the success of Pampa. You have over 73,000 followers on Instagram, which is huge. I mean, for any US-based consumer brand, but for a small brand from Paris that is working very regionally, this is really, really remarkable.Obviously, as you said, you launched digital-first, but how did it get there? How did it explode that much? And what did you learn from it? What are some secrets you can share of how you guys actually pulled off your Instagram following?Emmanuelle Magnan:Well, it's one of the things we are so proud about, is that we gained each of these followers organically.Fabian Geyrhalter:Yeah.Emmanuelle Magnan:We never acquired any follower base, or never used any bots. I'm so glad we did that. One third of our audience is from Paris, the rest is from diverse cities in France, because actually, we deliver outside of Paris. We deliver our dry flowers outside of Paris.Fabian Geyrhalter:That's a really long bicycle ride.Emmanuelle Magnan:Right. More on that later.Fabian Geyrhalter:Okay, okay.Emmanuelle Magnan:No, but I will explain that to you when we get into that subject, but yeah, we don't deliver in bicycle-Fabian Geyrhalter:Yes, I had a feeling.Emmanuelle Magnan:... outside of Paris. But yeah, one third of our audience is from Paris, the rest is from other cities in France. And actually, 25% is from abroad, and 4.5% is from the US, actually. When I was-Fabian Geyrhalter:There you go, I'm one of the 4.5.Emmanuelle Magnan:Actually, I saw that while I was preparing the new podcast, and I ... Yeah, actually, you are 4.5, from the US, so I think it's [crosstalk 00:28:15]. Maybe if there's anyone listening that already knows us, hi. How did we get there? I think there have been a few levers that I'm going to list, but I think the quality of the content that we are producing, posting, is key. We always put a lot of efforts in using our own content, and we created something that is very eye catching and it hooks people.So many people tells me about our Instagram grid, actually. And actually, so many [inaudible 00:28:55] repost our contents. Anthropologie in the US once regrammed one of our posts, which is totally crazy-Fabian Geyrhalter:Oh, that's cool.Emmanuelle Magnan:You're a small brand from France, from Paris, Anthropologie is so big. Anyway, there's also tons of people telling us that we put color in their feed and in their everyday lives, so I think they just enjoy getting shots of nature and color and that's also why they follow us. They don't necessarily buy from us every week or every month, but they just are so glad to receive this positiveness into their everyday life.That's one part. And in terms of actions that helped us grow our community, we did work with influencers, but for free, though. They loved our brand and product so much that even big influencers that are ... The maximum that we did, she had like one million followers, and they were happy to collaborate for free, actually, because they ... Yeah, they love the product and the concept, so that, I think, that for sure helps us grow the community.We also did some sponsored campaigns on Facebook and Instagram that were product focused, but I think it naturally generated following, because even though it was product focused, it also worked on awareness, so we gained a little bit of them like this. But mostly, it was organical, real organical word of mouth. And also, we have had a lot of amazing PR opportunities that came to us also organically, if I can say.Fabian Geyrhalter:Mm-hmm (affirmative).Emmanuelle Magnan:If I can say it like that. We had the 8:00 PM TV news broadcast in France, that generated a lot of awareness, and a lot of following. You mentioned New York Times. We have been twice in Teen Magazine of the New York Times, which is crazy for us, and very soon, when we started the brand, very quickly we were in Architectural Digest, and other French TV shows, so I think it really helped us grow the community.But we never really ... It came to us, really, because we were so focused on operating the company and just ... How can I say that? Sorry. Responding to the demand, we didn't ... We never really invest that much in communication. We developed our content, and we shoot everything in-house, so that's a kind of investment, but actually the communication itself-Fabian Geyrhalter:That's amazing. Yeah, yeah, yeah. PR just came to you [crosstalk 00:32:19]. Yeah, yeah, yeah. No, you didn't have a PR agency at that point, right? It just all came to you. It was all ... And a lot of that was, most probably, through you being out there, on social media. Right? Then, it just started spreading, and then you started getting into TV and all of that, that's just ... One thing leads to another.Emmanuelle Magnan:Exactly. At the beginning, the first thing was word of mouth. As Noélie and I came from other networks of ... We used to work in agencies, we ... other companies, et cetera, so we knew people from other past experiences, right?Fabian Geyrhalter:Yeah.Emmanuelle Magnan:So I think at first the word spread in Paris, and each time someone needed flowers, they would say, "Oh, you should go to Noélie and Emma's project and you will see, Pampa, it's really cool, it's new, it's fresh," whatever, and it started like that. And yeah, and then I can remember the first influencer who talked about us on Instagram. We had like 800 followers, we had launched one month ago, and suddenly, it was 10:00 PM and I remember I was going to the market that night and had to wake up at 3:00 AM. And I was watching our following base grow and grow, like, "New follower, new follower, new follower."Fabian Geyrhalter:Oh, wow.Emmanuelle Magnan:She had just done one [inaudible 00:33:58]. Out of the blue, because a friend of hers who knew us talked about us on his Facebook page, and she reposted it on Instagram, and we gained 1,200 followers in one day. That was amazing for us at the time. And it's still quite amazing, because today things changed, and it's really difficult to gain a lot of followers in one shot.Fabian Geyrhalter:Yeah, yeah/Emmanuelle Magnan:Things evolving on the Instagram game, and I think it's not really the case ... It's not really as it was four years ago. But yes, so it's that mix of things that ... And we have now this incredible community, and we want to take time to share more with them. We want to do more. Actually, we do a lot of content that is purely visual, that you look at and, okay, it's nice to watch.But we want to make more tutorials, we want to make more things that are interactive, so I hope this year we will have some time and some ... Yeah, some time, actually, to put that into place.Fabian Geyrhalter:To get the community more involved [crosstalk 00:35:24]. Yeah, yeah, yeah.Emmanuelle Magnan:...and to grow it even more. Yeah.Fabian Geyrhalter:Before we talked for a second about the delivery method, because you pride yourself in the bicycle delivery, but on the other hand, you also scaled, you're now in different cities within France. I do know that sustainability is important to the brand, right? You're trying to leave a very small environmental footprint, you're composting green waste, even though you're trying not to have much of it, because your whole company, the way that you have that one bouquet, basically, a week, it's already all made for that. And then, of course, the deliveries with the bicycle.How do you push sustainability? And talking about scalability, how scalable is that as you expand?Emmanuelle Magnan:Well, your question on scalability is actually super relevant, and I'm going to explain why. But first, I'd like to just introduce ... make an introduction on what is sustainability in the flower business. There is A, a notion of origin of supply, right? Well, if you do work with local growers or not. B, there's a notion of green waste management.Like you said, we compost our waste. What is waste in the flower industry? It's of course the flowers you cannot sell, but it's also everything you ... When you receive flowers, you have to prepare them, you have to cut them, and there is all the parts ... There are all the parts that don't go into your bouquet of flowers that don't pass the ... Sorry. There are other parts that don't go in your bouquets, or don't pass the quality test that go into a bin.And in Paris, actually, there is no green waste management for free, so we work with a special company who comes every week and take our-Fabian Geyrhalter:Oh, wow.Emmanuelle Magnan:... waste and compost it. C, in sustainability in the flower industry, there is a notion of delivery process. Right?Fabian Geyrhalter:Mm-hmm (affirmative).Emmanuelle Magnan:Do you deliver by truck, by scooter, by bicycle? There is a notion of durability. Fresh flowers are so ephemeral. How do you cope with that? And there's also a notion, I think, of wellbeing of your team, and create safe and ... a pleasing space to work in.As far as origin of supply is concerned, [inaudible 00:38:05] small, and at that time 90% of our supplies were super local. We worked with producers around Paris, when the season allowed it. Otherwise, it came from European countries, but mostly from end of winter until middle of autumn, at the beginning we were working with French growers.But during the first two years, each quarter of our business grew by 20 ... Sorry, each quarter of our business grew by 25%-Fabian Geyrhalter:Oh, nice.Emmanuelle Magnan:So, sometimes we had so much work, so many orders, that logistics couldn't keep up. Going to the flower market at 3:00 AM, three times a week was not in this instance sustainable, no?Fabian Geyrhalter:Right. Emmanuelle Magnan:Business-wise, but also on a personal scale. I mean, it was-Fabian Geyrhalter:Oh, it's killing you. Yes.Emmanuelle Magnan:I was so tired.Fabian Geyrhalter:Yeah, of course.Emmanuelle Magnan:And it's also less time to manage your company, and to create other things than the arrangements. And deliveries from the flower market here are really uncertain, they are expensive, and they are often late, and we work on a very tight schedule. Every morning, at 9:00 AM, we have dozens of orders leaving our studio, and at the time we were doing a lot of events when ... Well, before COVID.Fabian Geyrhalter:Yeah.Emmanuelle Magnan:We had sometimes 10 different projects for 10 different brands in 10 different venues in one day.Fabian Geyrhalter:Wow.Emmanuelle Magnan:It was madness, so at some point there was no way we could remain profitable and healthy without working on a proper supply chain. And we wanted to work with our local growers, but unfortunately, we didn't manage to create a system where we could be ... we could have supply on a regular basis, because we are so ... We look so much into quality, we want to deliver the freshest flowers as possible to our customers. It's so important for us, that we need the flowers to come, but come in our workshop every two days, maximum; because we don't want to deliver flowers that have been in our workshop for five days and they will last three days at our customers'.It's no question. So we figured we had to start working with most organized people in the market, and that is Holland. They are really good and they are specialists of flowers for ... It's been like that for centuries. And so, we have been working with them for two years and a half now, and we have flowers that come from France, Holland, and Italy. We have buyers in Holland that connects us with very good producers from those countries.And we try to be as local as possible, as close to our studio as possible. We would love to have our flowers coming from France only, but it's actually impossible. The scalability you were talking about, at some point you have to make choices. Actually, we opened a physical shop a few months ago ... In December, so actually one month ago, and in that shop we will be more flexible and we will have less time constraints, so we are aiming at a super local supply, with flowers being grown by a new generation of growers who know our constraints and can adapt easily.But as far as our website is concerned, we need to have something that is very organized. It's always, you have to make choices so that you are ... You do the maximum bet. And beyond flower supply, because flower supply is one thing, but ... It's important, of course, but it's not everything. So, we are developing an eco-friendlier system as a whole, so we sell this one fresh arrangement online to reduce waste.When we launched Pampa, we directly started working with Olvo, who's a bicycle delivery cooperative. They are specialists of urban, eco-friendly logistics. And all their courier are actually wage-earning employees, so they are not like freelance who struggle and ... It's very important for us that everyone is well ... [French 00:43:26]. Well paid for their work-Fabian Geyrhalter:Well taken care of. Yeah.Emmanuelle Magnan:Yeah. And so, in Paris 95% of our deliveries are made through them. And in the rest of France, there's no way for us to do that, unfortunately, so when we'll find delivery partners who can guarantee a delivery with electric vehicles, we will for sure start working with them, but we're just being patient and wait for the rest of the chain and industry to evolve, so we ... For sure, I mean, the deliveries we do outside of Paris are made in a traditional way-Fabian Geyrhalter:Right.Emmanuelle Magnan:We finance the compost of our green waste, and also another way of reducing carbon footprint is to work with durable flowers, right? We work with fresh flowers, but also we work with dried flowers, and silk flowers that are seen as very old fashioned, but that for us can be amazing, aesthetically speaking. But also, because we can do rental and amortize their carbon footprint, and use them maybe like 20 times. So it's really interesting. We're trying to innovate in that sense, also.Fabian Geyrhalter:Mm-hmm (affirmative).Emmanuelle Magnan:And then, we created 10 sustainable jobs, and we try to give a correct lifestyle to our employees by paying them a notch above the rest of the market, and we provide them with pleasing work conditions and good hours. They don't have to go to the market during the night. Yeah, we are trying to be conscious on every side of the company. You know?Fabian Geyrhalter:Yeah.Emmanuelle Magnan:Sorry, that was a very long answer.Fabian Geyrhalter:No, no, no. It went into all aspects of keeping a brand sustainable and building a business in a certain way, which is fantastic for everyone to hear. We talked about Pampa for a good 40 minutes or so. If you would take the entire brand idea and entire brand story and what it evokes, and you would have to put it into one word, right? Like big brands, usually you're able to define a brand in one word. For Cola-Cola, they want to be seen as happiness, for Everlane it's transparency. What could it be for Pampa, in one word?Emmanuelle Magnan:I think that would be colorful, without any doubt. Because yeah, colorful is ... Sorry. Concretely speaking, yeah, it's the use of color, but for me, it conveys so many things beyond the actual ... this actual meaning. It's a spirit which conveys joy, emotion, celebration, and-Fabian Geyrhalter:Optimism.Emmanuelle Magnan:Optimism, positivity, and positivism. I think it's so important right now in the context we are in. It's our mission. Our mission at Pampa is to really change someone's day by bringing joy and color to their everyday life. That's our mission statement. So, by all means, we are aiming at surprising people, and I think the color helps changing their day, and helps them ... It's like a therapy, you know?Fabian Geyrhalter:It totally is.Emmanuelle Magnan:It's a therapy through color.Fabian Geyrhalter:I think it's funny [crosstalk 00:47:33]. It's so funny that you say that, right? Because, really, when I think about how I got in touch with your brand, is because I saw one of your bouquets on Instagram, and I just stopped in my tracks. And it was because it was so colorful, right?And then I went to the site, and it is so overly colorful and joyful and different and fun, and that's ... It's so great that that is your brand's DNA. It's your mission, it's your vision, that's what you want to provide people with. I think it's fantastic.I think this is interesting because you come from the world of brands, right? Working in ad agencies, and your co-founder with event coordinating and all of that. This is all very intrinsically about branding, but now that you've been running your own gig for a couple of years now, what does branding mean to you now, now that you've actually done it? What does it mean? How would you describe branding?Emmanuelle Magnan:I would say that I think it's absolutely central, especially in B2C. For me, it's partly how you will hook your audience, and create a community around your product and service, because you can have your product and service, right? They can have all the features, all the technical features they need to have to be attractive, but with a brand, it's how you'll create emotions. It's how you will create self-identification. It's like, "I want to be part of the gang." You know?Fabian Geyrhalter:Yeah.Emmanuelle Magnan:For us, and for me, it's a guidebook for everything we do. Everything we create, everything we communicate, I'm like ... Every time, I'm like, "Is that Pampa? Are we creating enough?" Sorry ... I can't find my word in English. "Are we game-changing?"Fabian Geyrhalter:Yeah.Emmanuelle Magnan:"Are we game-changing enough?" And at the end of the day, I think it's also very, practically speaking, it's a way to exist on a highly-competitive environment flower market is, and flower industry. So, when we started, it was very competitive, but now so many projects arrived on the market since we arrived on the market.Fabian Geyrhalter:Mm-hmm (affirmative).Emmanuelle Magnan:You need to stand out. And for me, branding is ... Yeah, it's about creating something higher than just a product. It's more than something commercial. It's a belonging, I don't know if it makes sense-Fabian Geyrhalter:No, totally. Yeah, yeah, yeah. Yeah, I always say it's infusing soul into what otherwise is just a product, right? Then, I think people feel that, that it has heart and soul, versus your competitors who might just ... Yeah, those are nice products, those are good products. There's nothing wrong with them, but would you follow them on Instagram? Would you engage with them? Would you be excited about the product?It's a different kind of aura that you put around yourself that attracts people, which I think is wonderful, and that's why I love being in the business of branding, because I think it's actually very meaningful. It's not at all something that is just fake. It's actually the opposite of fake for the ones that make it as a brand, like you and your partner, so ... Great, well-Emmanuelle Magnan:...and it's so interesting to explore how you can create a brand and make it ... always make it evolve, and yeah. Fabian Geyrhalter:No, exactly, and it has to be ... You have to be repetitive so that people know this is truly your brand, but you constantly have to innovate, right? And how do you do those two? And I totally agree, it's super fascinating. All right. We have to come to an end here, we could talk for a long time, but listeners who fell in love with your brand just now, how can they get Pampa if they're in Europe? Do they only get it in France, or do they get it in other places in Europe? And for those of us in the United States, and other places around the globe, where can they follow you on social media?Emmanuelle Magnan:Actually, we just launched the deliveries in Europe, so-Fabian Geyrhalter:Oh, great timing.Emmanuelle Magnan:... get our dried flowers, they can go to pampa.co, on our website and order. And to follow us, so there's Instagram, of course, Facebook, Pinterest, and you can also subscribe to our mailing lists, because this is a good way to get information and new products and stuff like that.Fabian Geyrhalter:Perfect, perfect. And for those of you who heard us talk about Pampa, I just want to mention it is actually spelled P-A-M-P-A, so Pampa for you guys, just so you know who to follow. Thank you, Emmanuelle for having been on the show. I know it was not easy for you to go through this in perfect English. You did so amazing, we're so appreciative of you doing this interview in English as a second language. It's really amazing, and thank you for all of your insights and for your time.Emmanuelle Magnan:Thank you for having me, and giving me this opportunity to speak about Pampa.

Religion ...is the belief in and worship of a superhuman controlling power. Especially a personal God or Gods. A particular system of faith and worship...or a pursuit of interest to which someone ascribes supreme importance. Religion is a way of devotion, and its the first act of the virtue of religion and is defined as the promptness or readiness of will.. in the service of God. Concretely, this means the perfect offering of the will itself to God, for readiness of the will in the service of God is the will offered to God in worship. and while many have devotion in their religion..one man in Prattville, Alabama, took his devotion to God & faith to the next step. On top of a hill and across the road, are thousands of crosses. Symbols pointing the way from earth to heaven, warning against the evils of sin and offering the message of redemption..at the Rice Cross Garden.[FOLLOW ON SOCIAL MEDIA & MORE]TWITTER: www.twitter.com/SouthernOddPodINSTAGRAM: www.instagram.com/SouthernOddPodJARED'S TWITTER: www.twitter.com/jared_ordisJARED'S INSTAGRAM: www.instagram.com/jared.ordisORDIS SURDIOS NETWORK & WEBSITE: www.ordisstudios.com[ADDITONAL INFORMATION]Questions or Business Inquiry, Email Us @ ordisstudios@gmail.comResearched was used for this episode of Southern Oddities, and we couldn't have made it possible without the journalism and dedication from these awesome sources of information: AL "The Most Notorious Person From" Content.Time "Articles 28804 2006404 2006095" Roadside America "Story 2019" Roadside America "Tip 32" Space Archives "Cross Garden" Road Trippers "The Cross Garden" The Cross Photo "William C Rice's Cross Garden" Weird US "Cross Garden" Dictionary "Religion" Wikipedia "Religion""Southern Oddities" is created & produced by Jared Ordis, an Ordis Studios production. This show is part of the Ordis Studios Podcast NetworkCopyright © 2021 by Ordis Studioswww.ordisstudios.com

New data is made available by innovative new satellites that could benefit both societal needs and new business plans. It is important to merge data available from ground with space data in order to enable new services in various fields like maritime surveillance, property monitoring, agriculture, construction, finance and insurance. Space solutions, satellites and launchers have become cheaper, paving the way for a new generations of experts and entrepeneurs entering the space arena with new concept and business ideas both building and using satellite data in a more efficient way. This is what we call NEW SPACE.We elaborated on the Indian regulatory regime and the opening of the commercial space community in India. Concretely, we mentioned maritime surveillance and micro-launchers as possible Indian-Norwegian cooperation opportunities. Finally, we talked about the new sense of entrepreneurship, integration of IT and space domain involving players in the cloud and data processing value chain and the need for a wide variety of competences in the new race for space data.  

In our sixth episode, Sunny and Ujjwal sit down with Stefano Gunawan, to talk mindfulness, movement and everything in between. Stefano has an incredible philosophy on life, and describes the moment he decided to live a life in line with his values. One not to be missed!--We thought we'd change it up, and have a little introduction from the incredible soul himself:'Hey everyone! My name's Stefano and it's a pleasure to have your ears for the next hour as I share my story and mental health journey (to freedom!)I'm an Indonesian-Chinese blood born in Perth and I moved over to Melbourne as an 11y.o. for the beginning of high school. Having gone through some tumultuous adolescent years (didn't we all?!) I landed at the University of Melbourne in a Commerce degree. 4-years and a whole bunch of life-changing events later, I decided to quit that degree (with a mere subject to go! Insane, right?... naaaahh) and move on with my life.Releasing myself from the shackles of familial and societal expectations, I started to take the idea of rebellion very seriously. As Camus once said, “The only way to deal with an unfree world is to become so absolutely free that your very existence is an act of rebellion.” Amen!The main trigger for my radical change was a reality-bending 10-day silent meditation retreat (the second one, actually. I've been on four of them now), where I faced the demons of my depression and burnout. On the back of this ordeal, I committed wholehearted-and-soulfully to follow my muse whatever the cost to others might be, and I haven't looked back since.Concretely, what does that actually mean?Well, these days my time consists of studying a Psychotherapy degree (which, importantly, I chose of my own accord; no parental influence!); writing and sharing my thoughts through my blog (eclecticchild.com) and Instagram (@stefanoryangunawan); dancing wildly (yes, even in lockdown); yoga-ing and meditating daily (hopefully to be a teacher of both in two years time); cohabitating with hippies in Brunswick (plus a CUTE ass puppy named Enoki); and generally just trying to live my life as best as I can.Oh, and MOST importantly, I'm also co-running a Men's Circle! The gentlemen from BottledUp are a part of it too, and we're hoping to grow the family in the future. We meet every 2nd Sunday via Zoom (and once lockdown is over, in person) at 2pm.If you haven't already received or seen the open invitation from the Bottled Up guys, then consider this a personal invitation from yours truly: We are open and would love to have you join us at the next one! Come along with an open heart, a truthful tongue and attentive ears. It would be an honour to connect with you!'--~ Stefano ~Instagram: @stefanoryangunawanFree Will piece: http://eclecticchild.com/2018/04/02/there-is-no-free-will/Giving and Happiness piece: http://eclecticchild.com/2019/05/12/a-radical-idea-for-doing-good-better-giving-and-happiness/10-Day Meditation Retreats: https://www.dhamma.org/en/indexTony Robbins' 6 Core Human Needs: https://www.habitsforwellbeing.com/6-core-human-needs-by-anthony-robbins/Blue Zones — Socialising for Longevity: http://bluezones.com~ Bottled Up ~Bottled Up Instagram: @bottledupausBottled Up Website: www.bottledup.life--As always, Bottled Up is a passion project to help provide a voice to those many men who are going through the peaks and troughs of life. We are not trained professionals and if you're worried about someone close to you and their mental wellbeing, it's important to get professional advice.Start a conversation with them and let them know that they're not alone and that there are many support resources out there - including friends, family, school chaplains and plenty more!If you're feeling distressed or overwhelmed, Beyond Blue has a number of trained mental health professionals available 24/7 on 1800 512 348.For immediate support, call Lifeline on 13 11 44 and in an emergency, always dial 000.--Music: https://www.purple-planet.com

Dan Wolosyzn has been in executive hospital leadership for 23 years, including nine years in his current role as the CEO of Rehab Hospital of Indiana (RHI). He has a unique management approach, combining servant leadership with understanding the neuropsychology of corporate hospital systems. Not only does he believe in looking at this neuropsychology from a clinical perspective, but through an administrative lens as well. In today’s episode, you will hear how he incorporates these two philosophies into his everyday life, along with tangible examples of how to apply cross-sectional leadership to your own work. Truth You Can Act On   1. Reach out, communicate, and serve others. Supporting Quotes: Dan Woloszyn: “You don't have to be expert in everything, but there is one thing that I really believe one has to be expert in, certainly from a leadership standpoint, is to reach out and know others, and to serve others . You have to have an expertise with that to kind of drive a reduction of silos and an elimination of silos. And that can be done through one's own expertise or actively seeking other's expertise.”   Dan Wolosyzn: “I think each person and each leader truly has to believe it's a privilege to serve others unconditionally. There's a professional and humanistic component to that. My true belief is you have to love something about the people you lead to be truly elite effectively. If you don't love something about the people you lead, you probably are not in the right place, and you're probably not in the right place to be a leader.”   2. Trust and transparency are the foundations of cross sectional leadership. Supporting Quote:   Dan Woloszyn: “Being transparent about self, and certainly being honest, is extremely important. It's being honest about one's approach and any errors that might be committed and examples of approaches to correct the errors and how to grow with that. My belief is you have to think out loud and you have to be able to help others to get a sense of your own thought process as a leader and how you came to certain conclusions. I know sometimes that's difficult for people to do, but it's extremely beneficial where it helps in a sense to become kind of an external organizer for others, where you move from a point of, of brainstorming out loud a problem you might be faced with, verbalizing struggles, and even kind of working through some of those tactics out loud so others can benefit from a variety of things. I think what it does is it certainly lends to a relationship building and credibility and honesty and transparency.” 3. Model the behavior you are looking for in your culture. Supporting Quote: Dan Woloszyn: “First and foremost, it has to start with me. Laying the foundation has to be about modeling and certainly me believing in and what truly is important for our organization. There's always an expectation to look at the glass half full and everything that we do in a respectful way while modeling that and handing off to others who also will hand off to others, and that kind of permeates throughout the system.”   4. Make it a habit to regularly invite your leaders for collaboration and relationship building.       Supporting Quote: Dan Woloszyn: ”I think there's a conscious effort to tie others, to create alliances, not only within the organization, but outside the organization, within our community and really address this kind of holistically. Concretely, we do this a lot. I invite staff and leaders, online staff leaders, all different, team members, to our department meetings. I invite them to board meetings. I have them look at operational pathways they've generated and share their stories, because without that you truly understand the nature of what everybody's doing amongst the organization.” Book Recommendation   Dare to Lead by Brené Brown Sponsor Wambi.org – Wambi is about human connections. We view feedback as the fuel for interpersonal growth and are always striving to achieve the highest versions of ourselves and to lift others up along the way.

Episode 92: Katherine Collins Concretely Explains How She Uses Lean Six Sigma as a Manager! by IT Services

Support this podcast by subscribing to channel 253 EQ: To what extent should we still strive to be anti-racist in a world on fire?Guests: Kass Minor (she/her/hers) is an inclusive educator who is deeply involved in local, inquiry-based teacher research and school community development. Most recently, along with her partner and husband, Cornelius Minor, she has established The Minor Collective. You can follow Kass on Twitter @MsKass1shea martin (they/them/theirs) previous guest on episode 69 “Working Toward Liberation”. You can find them on Twitter at @sheathescholar.”Concretely learning about ways to “work towards liberation” in our classrooms, schools, and communities, is part of the necessary work for anti-racist educators. In the midst of a pandemic that has disrupted every aspect of our lives, many educators are looking for a “place to call home.” In this episode, we interview two of the four co-founders of the collective Liberate and Chill “an immersion online learning experience for anti-racists educators.” Kass and shea walk us through the inception and development of this new collaboration. As we consider this current place and time, we are looking towards the future not as a “return to normal” but a “turn to better.” We’re reminded albeit contradictory, liberation looks like both joy and rest. Learn more about Liberate and Chill by following them on Twitter and Instagram @liberateNchill Do Your Fudging Homework:Hope and Annie Combo Humanities HW: sign up and attend a workshop hosted by LiberateNChill and follow the Nap MinistryKass: Art of Gatheringshea: Savage remix Megan Thee Stallion (featuring Beyonce); GirlTrek Interview with Angela Davis & Nikki Giovanni Follow us on Twitter @IWL_Podcast or Facebook: Interchangeable White Ladies PodcastLastly, don’t forget to pick up your copy of The Body is Not an Apology for #readlessbasic book club

This is my participation in the book The New Wave: Escape 9-5, Do What You Love & Make an Impact. For more info on the book check www.thenewwavebook.com I answer the following 7 questions: 1. Tell us your story. Why did you start this journey? How did you get to where you are now? Were there any main triggers or something specific that happened? 2. I’ve noticed that many people experience an “imposter syndrome”. Meaning they feel not good enough, either because of the opinion of other people or because of negative self-talk. What were people around you saying? Did you sometimes feel not good enough, either in your own eyes or in the eyes of other people? And how did you overcome it? 3. I’ve noticed that money often is a fear when choosing to follow your passion. Did you experience this fear and how did you overcome it? Do you make a good living from your passion today? Concretely, how do you monetise it? 4. Were there any other major challenges you faced? If so, what were they and how did you overcome them? 5. I’ve noticed that there are many people who are wanting to jump in the deep end and follow their passion but still hesitate. What would be the best advice you would give them? 6. What is your definition of a successful life? What does success mean to you? 7. What is the impact you want to have in the world?

"At a certain age, every woman colors their hair," said Amy Errett on this week's Glossy Beauty Podcast. As the CEO and founder of the rapidly growing hair-color brand Madison Reed, she would know. Concretely, "every woman" means a market of about 90 million customers coloring their hair "on an average cadence of about seven weeks." Madison Reed first came to market as a DTC brand to be an at-home solution for color, but Errett could not stop before expanding to the salon market, too. The company announced last month that it would begin franchising in order to make its nearly 60 shades available to more consumers across the country. "Our highest penetration just basically follows the U.S. population. Urban metros have the highest population, but we reach out to about a 150-mile radius around any of those cities and suburbs in a big way," said Errett. Errett joined beauty editor Priya Rao to talk about how she's breaking the stigma around coloring at home, how she's catering to a younger customer and what tough words of advice she would give to a competitor.

I get a lot of emails almost I'd say one every 2-3 days from someone that basically says something like, "Hey, I love what you're doing. We should partner. Let's get on the phone to explore." Right? I [00:01:00] never ... it's hard to respond to those, right? At the very beginning when we were new and just wanted to talk to anyone to get feedback and learn about how other people saw us in the space. I would get on some of these calls and be like, "Okay, what ... how do you want to partner?" They'd say like, "I'm open to anything." Really? You're open to anything? So I [00:02:00] have to come up with the partnership idea. So you want to partner with me and now like I need to pitch you on the ... on what a partnership might look like? So now I just say to folks like, "Okay, cool. I'm open to it. Send me a one-page proposal with as many concretes as possible outlining what you would get, what we would get. Concretely what the exchange of value would be. Is it money? Is it traffic? Is [00:02:30] it visibility? If it is traffic or something like that, what would the form of that take?" And almost nobody writes back with a one-page proposal. Even a one paragraph concrete proposal. I don't get this whole like, "Hey, we should do something. Let's hop on the phone." Like who has that much phone time? Who has [00:03:00] that much bandwidth for connections in real time? Probably people who aren't ... who don't have customers. I don't know. I used to have this theory that the Biz Dev career or industry, like basically the main role of folks in Biz Dev, was to tie up time with other people who are in Biz Dev. Like, "Hey, Biz Dev person A at company A is going to Biz Dev person [00:03:30] B at company B and they're just gonna eat up a bunch of each other's time so they can justify their roles." When you're exchanging money, something fungible or bitcoin or equity or something, when you're trading these things it's very clear how value is exchanged. But when there's a partnership it's like, "All right, what are the specifics? [00:05:00] How long is it gonna last? Is there confidentiality? Is there exclusivity?" Usually you need to get ... you have to have some kind of contract involved, right? So you need to pay, if you're gonna do it right, you have to pay a pretty good commercial contracts attorney and that takes time. Usually [00:05:30] it takes hours of meetings, tens of hours and meetings in some cases. And then once it's done, you have to execute it, which means you need to get other people on your team involved and they have to be in alignment. There's so much involved in a really solid partnership. I think even Fortune 500 companies can count on two hands the number of really [00:06:00] important partners that exist for them. So yeah, I think it's just something that most people can consider ... I sent out a Tweet about this the other day and it got a lot of likes. Thank you for everyone who liked it. And someone responded to the Tweet, who I used to work with, Kevin Raheja, who runs partnerships or ... [00:06:30] I don't know what his official title ... I think it's maybe director of partnerships at HubSpot. And he responded with a really clear directive, which was 1. Know the partner's business. 2. Have a plan. 3. Be short and clear. 4. Exchange value. And 5. Don't waste time. [transcript truncated due to character count restrictions] Website: https://nomics.com Crypto Market Data API: https://nomicsapi.com Personal Twitter: https://twitter.com/ClayCollins Company Twitter: https://twitter.com/NomicsFinance --- Support this podcast: https://anchor.fm/nomics/support

Opuscope is developing Minsight: an app for creative users to design their own reality. With a headeset-first approach, our vision is to open the door for the proliferation of Virtual, Mixed and Augmented reality creations. I studied cognitive psychology with two main themes: usability and pedagogy. I early chose the usability part because I have always been curious and enthusiastic about new technologies, so when I learnt that psychology could match with them I knew this job was made for me. Concretely, I work on interfaces, whatever they are: website, software, video game, robot, VR/MR devices and much more without limit as long as there is an interface. This implies making sure they are easy to use, and fits their objectives and their target users. To do so, I: - evaluate existing interfaces to find ergonomics problems and suggest solutions to resolve them - design new interfaces from scratch to the point they are ready to be developed - make sure recommendations are well implemented by following development http://minsight.co https://www.linkedin.com/in/alexiabuclet https://twitter.com/getminsight

FreeBSD 10.4-RELEASE is here, more EuroBSDcon travel notes, the KRACK attack, ZFS and DTrace on NetBSD, and pfsense 2.4. This episode was brought to you by Headlines FreeBSD 10.4-RELEASE Available (https://www.freebsd.org/releases/10.4R/announce.html) FreeBSD 10.4-RELEASE is out. The FreeBSD Project dedicates the FreeBSD 10.4-RELEASE to the memory of Andrey A. Chernov. Some of the highlights: 10.4-RELEASE is the first FreeBSD release to feature full support for eMMC storage, including eMMC partitions, TRIM and bus speed modes up to HS400. Please note, though, that availability of especially the DDR52, HS200 and HS400 modes requires support in the actual sdhci(4) front-end as well as by the hardware used. Also note, that the SDHCI controller part of Intel® Apollo Lake chipsets is affected by several severe silicon bugs. Apparently, it depends on the particular Apollo Lake platform whether the workarounds in place so far are sufficient to avoid timeouts on attaching sdhci(4) there. Also in case a GPT disk label is used, the fsckffs(8) utility now is able to find alternate superblocks. The aesni(4) driver now no longer shares a single FPU context across multiple sessions in multiple threads, addressing problems seen when employing aesni(4) for accelerating ipsec(4). Support for the Kaby Lake generation of Intel® i219(4)/ i219(5) devices has been added to the em(4) driver. The em(4) driver is now capable of enabling Wake On LAN (WOL) also for Intel® i217, i218 and i219 chips. Note that stale interface configurations from previous unsuccessful attempts to enable WOL for these devices now will actually take effect. For example, an ifconfig em0 wol activates all WOL variants including wolmcast, which might be undesirable. Support for WOL has been added to the igb(4) driver, which was not able to activate this feature on any device before. The same remark regarding stale WOL configurations as for the em(4) driver applies. Userland coredumps can now trigger events such as generating a human readable crash report via devd(8). This feature is off by default. The firmware shipping with the qlxgbe(4) driver has been updated to version 5.4.66. Additionally, this driver has received some TSO and locking fixes, performance optimizations as well as SYSCTLs providing MAC, RX and TX statistics. Mellanox® ConnectX-4 series adapters are now supported by the newly added mlx5ib(4) driver. OpenSSH received an update to version 7.3p1. GNOME has been updated to version 3.18. Xorg-Server has been updated to version 1.18.4. Check out the full release notes and upgrade your systems to 10.4-RELEASE. Thanks to the FreeBSD Release Engineering Team for their efforts. *** EuroBSDcon 2017: "travel notes" after the conference (https://blog.netbsd.org/tnf/entry/eurobsdcon_2017_travel_notes_after) Leonardo Taccari posted in the NetBSD blog about his experiences at EuroBSDcon 2017: Let me tell you about my experience at EuroBSDcon 2017 in Paris, France. We will see what was presented during the NetBSD developer summit on Friday and then we will give a look to all of the NetBSD and pkgsrc presentations given during the conference session on Saturday and Sunday. Of course, a lot of fun also happened on the "hall track", the several breaks during the conference and the dinners we had together with other *BSD developers and community! This is difficult to describe and I will try to just share some part of that with photographs that we have taken. I can just say that it was a really beautiful experience, I had a great time with others and, after coming back home... ...I miss all of that! :) So, if you have never been in any BSD conferences I strongly suggest you to go to the next ones, so please stay tuned via NetBSD Events. Being there this is probably the only way to understand these feelings! Thursday (21/09): NetBSD developers dinner Arriving in Paris via a night train from Italy I literally sleep-walked through Paris getting lost again and again. After getting in touch with other developers we had a dinner together and went sightseeing for a^Wseveral beers! Friday (22/09): NetBSD developers summit On Friday morning we met for the NetBSD developers summit kindly hosted by Arolla. NetBSD on Google Compute Engine -- Benny Siegert (bsiegert) Scripting DDB with Forth -- Valery Ushakov (uwe) News from the version control front -- Jörg Sonnenberger (joerg) Afternoon discussions and dinner After the lunch we had several non-scheduled discussions, some time for hacking, etc. We then had a nice dinner together (it was in a restaurant with a very nice waiter who always shouted after every order or after accidentally dropping and crashing dishes!, yeah! That's probably a bit weird but I liked that attitude! :)) and then did some sightseeing and had a beer together. Saturday (23/09): First day of conference session and Social Event A Modern Replacement for BSD spell(1) -- Abhinav Upadhyay (abhinav) Portable Hotplugging: NetBSD's uvm_hotplug(9) API development -- Cherry G. Mathew (cherry) Hardening pkgsrc -- Pierre Pronchery (khorben) Reproducible builds on NetBSD -- Christos Zoulas (christos) Social event The social event on Saturday evening took place on a boat that cruised on the Seine river. It was a very nice and different way to sightsee Paris, eat and enjoy some drinks and socialize and discuss with other developers and community. + Sunday (24/09): Second day of conference session The school of hard knocks - PT1 -- Sevan Janiyan (sevan) The LLDB Debugger on NetBSD -- Kamil Rytarowski (kamil) What's in store for NetBSD 8.0? -- Alistair Crooks (agc) Sunday dinner After the conference we did some sightseeing in Paris, had a dinner together and then enjoyed some beers! Conclusion It was a very nice weekend and conference. It is worth to mention that EuroBSDcon 2017 was the biggest BSD conference (more than 300 people attended it!). I would like to thank the entire EuroBSDcon organising committee (Baptiste Daroussin, Antoine Jacoutot, Jean-Sébastien Pédron and Jean-Yves Migeon), EuroBSDcon programme committee (Antoine Jacoutot, Lars Engels, Ollivier Robert, Sevan Janiyan, Jörg Sonnenberger, Jasper Lievisse Adriaanse and Janne Johansson) and EuroBSDcon Foundation for organizing such a wonderful conference! I also would like to thank the speakers for presenting very interesting talks, all developers and community that attended the NetBSD devsummit and conference, in particular Jean-Yves and Jörg, for organizing and moderating the devsummit and Arolla that kindly hosted us for the NetBSD devsummit! A special thanks also to Abhinav (abhinav) and Martin (martin) for photographs and locals Jean-Yves (jym) and Stoned (seb) for helping us in not get lost in Paris' rues! :) Thank you! *** WiFi Vulnerability in WPA2: KRACK (https://www.krackattacks.com/) “We discovered serious weaknesses in WPA2, a protocol that secures all modern protected Wi-Fi networks. An attacker within range of a victim can exploit these weaknesses using key reinstallation attacks (KRACKs). Concretely, attackers can use this novel attack technique to read information that was previously assumed to be safely encrypted. This can be abused to steal sensitive information such as credit card numbers, passwords, chat messages, emails, photos, and so on. The attack works against all modern protected Wi-Fi networks. Depending on the network configuration, it is also possible to inject and manipulate data. For example, an attacker might be able to inject ransomware or other malware into websites.” “Note that if your device supports Wi-Fi, it is most likely affected. During our initial research, we discovered ourselves that Android, Linux, Apple, Windows, OpenBSD, MediaTek, Linksys, and others, are all affected by some variant of the attacks. For more information about specific products, consult the database of CERT/CC, or contact your vendor.” FreeBSD Advisory (https://www.freebsd.org/security/advisories/FreeBSD-SA-17:07.wpa.asc) As of the date of this recording, a few weeks ahead of when this episode will air, the issue is fixed in FreeBSD 11.0 and 11.1, and a workaround has been provided for 10.3 and 10.4 (install newer wpa_supplicant from ports). A fix for 10.3 and 10.4 is expected soon. They will more than likely be out by time you are watching this. The fix for 10.3 and 10.4 is more complicated because the version of wpasupplicant included in the base system is 2.0, from January 2013, so is nearly 5 years old, so the patches do not apply cleanly. The security team is still considering if it will try to patch 2.0, or just replace the version of wpasupplicant with 2.5 from FreeBSD 11.x. OpenBSD was unwilling to wait when the embargo was extended on this vulnerability and stealth fixed the issue on Aug 30th (https://marc.info/?l=openbsd-cvs&m=150410571407760&w=2) stsp@openbsd.org ‘s Mastodon post (https://mastodon.social/@stsp/98837563531323569) Lobste.rs conversation about flaw and OpenBSD's reaction (https://lobste.rs/s/dwzplh/krack_attacks_breaking_wpa2#c_pbhnfz) “What happened is that he told me on July 15, and gave a 6 weeks embargo until end of August. We already complained back then that this was way too long and leaving people exposed. Then he got CERT (and, thus, US gov agencies) involved and had to extend the embargo even further until today. At that point we already had the ball rolling and decided to stick to the original agreement with him, and he gave us an agreeing nod towards that as well.” “In this situation, a request for keeping the problem and fix secret is a request to leave our users at risk and exposed to insiders who will potentially use the bug to exploit our users. And we have no idea who the other insiders are. We have to assume that information of this kind leaks and dissipates pretty fast in the security “community”.” “We chose to serve the needs of our users who are the vulnerable people in this drama. I stand by that choice.” As a result of this: “To avoid this problem in the future, OpenBSD will now receive vulnerability notifications closer to the end of an embargo.” NetBSD: “patches for the WPA issues in KRACK Attacks were committed Oct 16th to HEAD & are pending pullup to 6/7/8 branches” (http://mail-index.netbsd.org/source-changes/2017/10/16/msg088877.html) As of this recording, Dragonfly appears to use wpa_supplicant 2.1 which they imported in 2014 and has not been touched in over a year (https://github.com/DragonFlyBSD/DragonFlyBSD/commits/master/contrib/wpa_supplicant) *** News Roundup NetBSD - dtrace and ZFS update (https://mail-index.netbsd.org/tech-kern/2017/10/13/msg022436.html) Chuck Silvers writes to the tech-kern mailing list of NetBSD: I've been working on updating netbsd's copy of the dtrace and zfs code to rebase from the existing ancient opensolaris version to a recent freebsd version. most of the freebsd changes are pretty close to what netbsd needs, so that seems like a more useful upstream for us. I have things working well enough now that I want to share the code in preparation for committing. this update improves upon our existing dtrace/zfs code in several ways: picks up all the upstream zfs fixes and enhancements from the last decade zfs now supports mmap on netbsd, so you can run executables stored in zfs dtrace fbt probes can now be used in kernel modules (such as zfs) A patch is provided here: http://ftp.netbsd.org/pub/NetBSD/misc/chs/diff.cddl.20171012 which needs to be applied using “patch -E” as it adds and removes files. He provides the following summary for the diff: freebsd's dtrace/zfs code as of r315983 (2017-03-26), adapted for netbsd. a few updates to our copy of freebsd's libproc. build system support for using -fno-omit-frame-pointer everywhere and disabling other compiler optimizations that confuse dtrace. sample kernel config changes for a couple evbarm configs (the ones I tested). module/ksyms enhancements for dtrace integration. genfs API enhancements to support zfs. an option to have mutexes not become no-ops during a panic. uvm_aobj API change to support 64-bit aobj sizes (eg. for tmpfs). Known issues with the patch include: unloading the zfs module fails even with no zpools imported if you've done much with zfs since it was loaded. there's some refcounting problem that I haven't tracked down yet. the module refcounting for active fbt probes is bogus. currently module refcounting is protected by kernconfig_lock(), but taking that lock down in the bowels of dtrace seems likely to create deadlocks. I plan to do something fancier but haven't gotten to it yet. the dtrace uregs[] stuff is probably still wrong. the CTF typeid overflow problem is still there (more on this below). Unsupported features include: the ".zfs" virtual directory, eg. ".zfs/snapshot/foo@bar" zvols ZFS ACLs (aka. NFSv4 ACLs) NFS exporting a ZFS file system setting dtrace probes in application code using ZFS as the root fs new crypto hashes SHA512_256, skein, and edonr (the last one is not in freebsd yet either) zio delay injection (used for testing zfs) dtrace support for platforms other than x86 and arm A more detailed description of the CTF typeid overflow is also provided. Check out the full thread with followups and try out the patch if you're on NetBSD. *** pfSense 2.4.0-RELEASE Now Available! (https://www.netgate.com/blog/pfsense-2-4-0-release-now-available.html) Jim Pingle writes about the new release: We are excited to announce the release of pfSense® software version 2.4, now available for new installations and upgrades! pfSense software version 2.4.0 was a herculean effort! It is the culmination of 18 months of hard work by Netgate and community contributors, with over 290 items resolved. According to git, 671 files were changed with a total 1651680 lines added, and 185727 lines deleted. Most of those added lines are from translated strings for multiple language support! + Highlights FreeBSD 11.1-RELEASE as the base Operating System New pfSense installer based on bsdinstall, with support for ZFS, UEFI, and multiple types of partition layouts (e.g. GPT, BIOS) Support for Netgate ARM devices such as the SG-1000 OpenVPN 2.4.x support, which brings features like AES-GCM ciphers, speed improvements, Negotiable Crypto Parameters (NCP), TLS encryption, and dual stack/multihome Translation of the GUI into 13 different languages! For more information on contributing to the translation effort, read our previous blog post and visit the project on Zanata WebGUI improvements, such as a new login page, improved GET/POST CSRF handling, significant improvements to the Dashboard and its AJAX handling Certificate Management improvements including CSR signing and international character support Captive Portal has been rewritten to work without multiple instances of ipfw Important Information: 32-bit x86 and NanoBSD have been deprecated and are not supported on pfSense 2.4. Read the full release notes and let them know how you like the new release. *** OpenBSD changes of note 629 (https://www.tedunangst.com/flak/post/openbsd-changes-of-note-629) Use getrusage to measure CPU time in md5 benchmarking. Add guard pages at the end of kernel stacks so overflows don't run into important stuff. This would be useful in FreeBSD, even just to detect the condition. I had all kinds of strange crashes when I was accidently overflowing the stack when working on the initial version of the ZSTD patches before ZSTD gained a working heap mode. Add dwxe driver for ethernet found on Allwinner A64, H3 and H5 SoCs. Fix a regression caused by removal of SIGIO from some devices. In malloc, always delay freeing chunks and change ‘F' option to perform a more extensive check for double free. Change sendsyslog prototype to take a string, since there's little point logging not strings. The config program tries to modify zero initialized variables. Previous versions of gcc were patched to place these in the data segment, instead of the bss, but clang has no such patches. Long long ago, this was the default behavior for compilers, which is why gcc was patched to maintain that existing behavior, but now we want a slightly less unusual toolchain. Fix the underlying issue for now by annotating such variables with a data section attribute. *** t2k17 Hackathon Report: Philip Guenther: locking and libc (https://undeadly.org/cgi?action=article;sid=20170824080132) Next up in our series of t2k17 hackathon reports is this one from Philip Guenther: I showed up at t2k17 with a couple hold-over diffs from e2k17 that weren't stable then and hadn't gotten much better since, so after a red-eye through Chicago I arrived in the hackroom, fired up my laptop and synced trees. Meanwhile, people trickled in and the best part of hackathons, the conversations and "what do you think about this?" chats started. Theo introduced me to Todd Mortimer (mortimer@), who's been hacking on clang to implement RETGUARD for C programs. Over the hackathon we discussed a few loose ends that cropped up and what the correct behavior should be for them as well as the mechanics of avoiding 0xc3 bytes (the RET opcode) embedded in the middle of other multi-byte x86 machine code. Fun stuff. Martin (mpi@) and I had a conversation about the desirability of being able to sleep while holding netlock and pretty much came down on "oof, the scheduler does need work before the underlying issue driving this question can be resolved enough to answer it". :-( After some final hammering I got in an enhancement to pool(9) to let a pool use (sleeping) rwlocks instead of (spinning) mutexes and then immediately used that for the per-CPU pool cache pool as well as the futex pool. Further pools are likely to be converted as well kernel upper-level locking changes are made. Speaking of, a larger diff I had been working on for said upper-level locking was still suffering deadlock issues so I took a stab at narrowing it down to just a lock for the process tree, mostly mirroring the FreeBSD proctreelock. That appears to be holding up much better and I just have some code arrangement issues around sysptrace() before that'll go out for final review. Then most of the way through the week, Bob (beck@) vocally complained that life would be easier for libressl if we had some version of pthread_once() and the pthread mutex routines in libc. This would make some other stuff easier too (c.f. /usr/X11R6/lib/libpthread-stubs.*) and the TIB work over the last couple years has basically eliminated the runtime costs of doing so, so I spent most the rest of the hackathon finding the right place to draw a line through libpthread and move everything on the one side of the line into libc. That code seems pretty stable and the xenocara and ports people seem to like—or at least accept—the effects, so it will almost certainly go in with the next libc bump. Lots of other random conversations, hacking, meals, and beer. Many thanks to Ken (krw@) and local conspirators for another excellent Toronto hackathon! Beastie Bits 2017 NetBSD Foundation Officers (https://blog.netbsd.org/tnf/entry/2017_netbsd_foundation_officers) New BSDMag is out - Military Grade Data Wiping in FreeBSD with BCWipe (https://bsdmag.org/download/military-grade-data-wiping-freebsd-bcwipe/) LibertyBSD 6.1 released (http://libertybsd.net/) *** Feedback/Questions Eddy - EuroBSDCon 2017 video and some help (http://dpaste.com/3WDNV05#wrap) Eric - ZFS monitoring (http://dpaste.com/2RP0S60#wrap) Tom - BSD Hosting (http://dpaste.com/31DGH3J#wrap) ***

too much sun this season for your liking? Drop into this mix of ruff techno cuts peeled straight from the floors of concrete clubs.

Oonops shows love to his broadcast station Brooklyn Radio with this episode and connects his home country Germany with NYC in a very special musical way. As ever he digs deep in in his vinyl crates to select a manifold cross section of old'n'new tunes from German MCs, beatmakers and producers before his homie Demir Cesar is bridging the gap with his strictly NYC hip hop guest set. Concretely you'll find solely Brooklyn related tracks! Cesar is a well-versed dj technician who already did support for acts like DJ Shadow and Cut Chemist. And now enjoy this exciting journey from Germany to NYC from the unique 'Oonops Drops'-series on Brooklyn Radio! Visit them both here and show some love on their sites: mixcloud.com/oonops : facebook.com/djoonops and mixcloud.com/DJ_Cesar here: facebook.com/demircesar79 | Cover-Artwork by Wörn from Clothing label Dauerfeuer: dauerfeuer.net

Oonops shows love to his broadcast station Brooklyn Radio with this episode and connects his home country Germany with NYC in a very special musical way. As ever he digs deep in in his vinyl crates to select a manifold cross section of old'n'new tunes from German MCs, beatmakers and producers before his homie Demir Cesar is bridging the gap with his strictly NYC hip hop guest set. Concretely you'll find solely Brooklyn related tracks! Cesar is a well-versed dj technician who already did support for acts like DJ Shadow and Cut Chemist. And now enjoy this exciting journey from Germany to NYC from the unique 'Oonops Drops'-series on Brooklyn Radio! Visit them both here and show some love on their sites: mixcloud.com/oonops : facebook.com/djoonops and mixcloud.com/DJ_Cesar here: facebook.com/demircesar79 | Cover-Artwork by Wörn from Clothing label Dauerfeuer: dauerfeuer.net

Ian landed in Bali last Saturday and it’s been all-biznass-all-the-time as we make the most out of his time here. No one should be surprised that they were able to do all their prep at Starbucks before. Good Chai, Ian? ;) This week Dan and Ian discuss two of the most misunderstood words in business:Hustler and Entrepreneur Listen to this episode and learn about: 1. Entrepreneurs Writing Scripts, Hustlers Executing Scripts. 2. Self Confidence vs.Product Confidence. 3. Ego / Money Focus vs. Product / Customer Focus. 4. Self Employed vs. Building Something Bigger. 5. Short Term vs. Long Term Thinking. Some of Dan & Ian’s upcoming Dreamlines?  Pay for personal assistants– sports massage therapist, business coach, nutritionist, personal assistnat. doctors and dentists.  Get a book contract.  Summit about hyper-internationalization and micro-multinationals with some of the world’s premier location independent entrepreneurs.  Lead investments in disruptive platform style companies in our industries. 150+ spare, IE angel investor.  Concretely help 100 people make the lifestyle transition and a significant portion of the DC make 10K.  Business class on every flight. (how can we hack this???)