Podcasts about human feedback

AXRP - the AI X-risk Research Podcast

Play Episode Listen Later Jun 12, 2024 81:54

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AXRP Episode 33 - RLHF Problems with Scott Emmons, published by DanielFilan on June 12, 2024 on The AI Alignment Forum. YouTube link Reinforcement Learning from Human Feedback, or RLHF, is one of the main ways that makers of large language models make them 'aligned'. But people have long noted that there are difficulties with this approach when the models are smarter than the humans providing feedback. In this episode, I talk with Scott Emmons about his work categorizing the problems that can show up in this setting. Topics we discuss: Deceptive inflation Overjustification Bounded human rationality Avoiding these problems Dimensional analysis RLHF problems, in theory and practice Scott's research program Following Scott's research Daniel Filan: Hello, everybody. In this episode I'll be speaking with Scott Emmons. Scott is a PhD student at UC Berkeley, working with the Center for Human-Compatible AI on AI safety research. He's previously co-founded far.ai, which is an AI safety non-profit. For links to what we're discussing, you can check the description of the episode, and for a transcript you can read it at axrp.net. Well, welcome to AXRP. Scott Emmons: Great to be here. Deceptive inflation Daniel Filan: Sure. So today we're talking about your paper, When Your AIs Deceive You: Challenges With Partial Observability of Human Evaluators in Reward Learning, by Leon Lang, Davis Foote, Stuart Russell, Erik Jenner, and yourself. Can you just tell us roughly what's going on with this paper? Scott Emmons: Yeah, I could start with the motivation of the paper. Daniel Filan: Yeah, sure. Scott Emmons: We've had a lot of speculation in the x-risk community about issues like deception. So people have been worried about what happens if your AIs try to deceive you. And at the same time, I think for a while that's been a theoretical, a philosophical concern. And I use "speculation" here in a positive way. I think people have done really awesome speculation about how the future of AI is going to play out, and what those risks are going to be. And deception has emerged as one of the key things that people are worried about. I think at the same time, we're seeing AI systems actually deployed, and we're seeing a growing interest of people in what exactly do these risks look like, and how do they play out in current-day systems? So the goal of this paper is to say: how might deception play out with actual systems that we have deployed today? And reinforcement learning from human feedback [RLHF] is one of the main mechanisms that's currently being used to fine-tune models, that's used by ChatGPT, it's used by Llama, variants of it are used by Anthropic. So what this paper is trying to do is it's trying to say, "Can we mathematically pin down, in a precise way, how might these failure modes we've been speculating about play out in RLHF?" Daniel Filan: So in the paper, the two concepts you talk about on this front are I think "deceptive inflation" and "overjustification". So maybe let's start with deceptive inflation. What is deceptive inflation? Scott Emmons: I can give you an example. I think examples from me as a child I find really helpful in terms of thinking about this. So when I was a child, my parents asked me to clean the house, and I didn't care about cleaning the house. I just wanted to go play. So there's a misalignment between my objective and the objective my parents had for me. And in this paper, the main failure cases that we're studying are cases of misalignment. So we're saying: when there is misalignment, how does that play out? How does that play out in the failure modes? So [with] me as a misaligned child, one strategy I would have for cleaning the house would be just to sweep any dirt or any debris under the furniture. So I'm cleaning the house, I just sweep some debris...

ai phd chatgpt speech ea uc berkeley llama anthropic stuart russell rationalist rlhf human feedback scott emmons

33 - RLHF Problems with Scott Emmons

Play Episode Listen Later Jun 12, 2024 101:24

Reinforcement Learning from Human Feedback, or RLHF, is one of the main ways that makers of large language models make them 'aligned'. But people have long noted that there are difficulties with this approach when the models are smarter than the humans providing feedback. In this episode, I talk with Scott Emmons about his work categorizing the problems that can show up in this setting. Patreon: patreon.com/axrpodcast Ko-fi: ko-fi.com/axrpodcast The transcript: https://axrp.net/episode/2024/06/12/episode-33-rlhf-problems-scott-emmons.html Topics we discuss, and timestamps: 0:00:33 - Deceptive inflation 0:17:56 - Overjustification 0:32:48 - Bounded human rationality 0:50:46 - Avoiding these problems 1:14:13 - Dimensional analysis 1:23:32 - RLHF problems, in theory and practice 1:31:29 - Scott's research program 1:39:42 - Following Scott's research Scott's website: https://www.scottemmons.com Scott's X/twitter account: https://x.com/emmons_scott When Your AIs Deceive You: Challenges With Partial Observability of Human Evaluators in Reward Learning: https://arxiv.org/abs/2402.17747 Other works we discuss: AI Deception: A Survey of Examples, Risks, and Potential Solutions: https://arxiv.org/abs/2308.14752 Uncertain decisions facilitate better preference learning: https://arxiv.org/abs/2106.10394 Invariance in Policy Optimisation and Partial Identifiability in Reward Learning: https://arxiv.org/abs/2203.07475 The Humble Gaussian Distribution (aka principal component analysis and dimensional analysis): http://www.inference.org.uk/mackay/humble.pdf Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!: https://arxiv.org/abs/2310.03693 Episode art by Hamish Doodles: hamishdoodles.com

risks ko uncertain dimensional deceptive reinforcement learning potential solutions bounded rlhf invariance human feedback scott emmons

Beyond the Black Box: Exploring the Human Side of AI with Lachlan Phillips

Crazy Wisdom

Play Episode Listen Later May 21, 2024 55:50

In this episode of the Crazy Wisdom Podcast, host Stewart Alsop welcomes Lachlan Phillips, founder of LiveMind AI, for a compelling conversation about the implications of decentralized AI. They discuss the differences between centralized and decentralized systems, the historical context of centralization, and the potential risks and benefits of distributed computing and storage. Topics also include the challenges of aligning AI with human values, the role of supervised fine-tuning, and the importance of trust and responsibility in AI systems. Tune in to hear how decentralized AI could transform technology and society. Check out LiveMind AI and follow Lachlan on Twitter at @bitcloud for more insights. Check out this GPT we trained on the conversation! Timestamps 00:00 Introduction of Lachlan Phillips and discussion on decentralized AI, comparing it to human brain structure and the World Wide Web. 00:05 Further elaboration on decentralization and centralization in AI and its historical context, including the impact of radio, TV, and the internet. 00:10 Discussion on the natural emergence of centralization from decentralized systems and the problems associated with centralized control. 00:15 Comparison between centralized and decentralized systems, highlighting the voluntary nature of decentralized associations. 00:20 Concerns about large companies controlling powerful AI technology and the need for decentralization to avoid issues similar to those seen with Google and Facebook. 00:25 Discussion on Google's centralization, infrastructure, and potential biases. Introduction to distributed computing and storage concepts. 00:30 Lachlan Phillips shares his views on distributed storage and mentions GunDB and IPFS as examples of decentralized systems. 00:35 Exploration of the relationship between decentralized AI and distributed storage, emphasizing the need for decentralized training of AI models. 00:40 Further discussion on decentralized AI training and the potential for local models to handle specific tasks instead of relying on centralized infrastructures. 00:45 Conversation on the challenges of aligning AI with human values, the role of supervised fine-tuning in AI training, and the involvement of humans in the training process. 00:50 Speculation on the implications of technologies like Neuralink and the importance of decentralizing such powerful tools to prevent misuse. 00:55 Discussion on network structures, democracy, and how decentralized systems can better represent collective human needs and values. Key Insights Decentralization vs. Centralization in AI: Lachlan Phillips highlighted the fundamental differences between decentralized and centralized AI systems. He compared decentralized AI to the structure of the human brain and the World Wide Web, emphasizing collaboration and distributed control. He argued that while centralized AI systems concentrate power and decision-making, decentralized AI systems mimic natural, more organic forms of intelligence, potentially leading to more robust and democratic outcomes. Historical Context and Centralization: The conversation delved into the historical context of centralization, tracing its evolution from the era of radio and television to the internet. Stewart Alsop and Lachlan discussed how centralization has re-emerged in the digital age, particularly with the rise of big tech companies like Google and Facebook. They noted how these companies' control over data and algorithms mirrors past media centralization, raising concerns about power consolidation and its implications for society. Emergent Centralization in Decentralized Systems: Lachlan pointed out that even in decentralized systems, centralization can naturally emerge as a result of voluntary collaboration and association. He explained that the problem lies not in centralization per se, but in the forced maintenance of these centralized structures, which can lead to the consolidation of power and the detachment of centralized entities from the needs and inputs of their users. Risks of Centralized AI Control: A significant part of the discussion focused on the risks associated with a few large companies controlling powerful AI technologies. Stewart expressed concerns about the potential for misuse and bias, drawing parallels to the issues seen with Google and Facebook's control over information. Lachlan concurred, emphasizing the importance of decentralizing AI to prevent similar problems in the AI domain and to ensure broader, more equitable access to these technologies. Distributed Computing and Storage: Lachlan shared his insights into distributed computing and storage, citing projects like GunDB and IPFS as promising examples. He highlighted the need for decentralized infrastructures to support AI, arguing that these models can help sidestep the centralization of control and data. He advocated for pushing as much computation and storage to the client side as possible to maintain user control and privacy. Challenges of AI Alignment and Training: The conversation touched on the difficulties of aligning AI systems with human values, particularly through supervised fine-tuning and RLHF (Reinforcement Learning from Human Feedback). Lachlan criticized current alignment efforts for their top-down approach, suggesting that a more decentralized, bottom-up method that incorporates diverse human inputs and experiences would be more effective and representative. Trust and Responsibility in AI Systems: Trust emerged as a central theme, with both Stewart and Lachlan questioning whether AI systems can or should be trusted more than humans. Lachlan argued that ultimately, humans are responsible for the actions of AI systems and the consequences they produce. He emphasized the need for AI systems that enable individual control and accountability, suggesting that decentralized AI could help achieve this by aligning more closely with human networks and collective decision-making processes.

Encore Episode: Generative AI and Large Language Models

Oracle University Podcast

Play Episode Listen Later May 14, 2024 21:13

In this week's episode, Lois Houston and Nikita Abraham, along with Senior Instructor Himanshu Raj, take you through the extraordinary capabilities of Generative AI, a subset of deep learning that doesn't make predictions but rather creates its own content. They also explore the workings of Large Language Models. Oracle MyLearn: https://mylearn.oracle.com/ou/learning-path/become-an-oci-ai-foundations-associate-2023/127177 Oracle University Learning Community: https://education.oracle.com/ou-community LinkedIn: https://www.linkedin.com/showcase/oracle-university/ X (formerly Twitter): https://twitter.com/Oracle_Edu Special thanks to Arijit Ghosh, David Wright, and the OU Studio Team for helping us create this episode. --------------------------------------------------------- Episode Transcript: 00:00 The world of artificial intelligence is vast and everchanging. And with all the buzz around it lately, we figured it was the perfect time to revisit our AI Made Easy series. Join us over the next few weeks as we chat about all things AI, helping you to discover its endless possibilities. Ready to dive in? Let's go! 00:33 Welcome to the Oracle University Podcast, the first stop on your cloud journey. During this series of informative podcasts, we'll bring you foundational training on the most popular Oracle technologies. Let's get started! 00:46 Lois: Hello and welcome to the Oracle University Podcast. I'm Lois Houston, Director of Innovation Programs with Oracle University, and with me is Nikita Abraham, Principal Technical Editor. Nikita: Hi everyone! In our last episode, we went over the basics of deep learning. Today, we'll look at generative AI and large language models, and discuss how they work. To help us with that, we have Himanshu Raj, Senior Instructor on AI/ML. So, let's jump right in. Hi Himanshu, what is generative AI? 01:21 Himanshu: Generative AI refers to a type of AI that can create new content. It is a subset of deep learning, where the models are trained not to make predictions but rather to generate output on their own. Think of generative AI as an artist who looks at a lot of paintings and learns the patterns and styles present in them. Once it has learned these patterns, it can generate new paintings that resembles what it learned. 01:48 Lois: Let's take an example to understand this better. Suppose we want to train a generative AI model to draw a dog. How would we achieve this? Himanshu: You would start by giving it a lot of pictures of dogs to learn from. The AI does not know anything about what a dog looks like. But by looking at these pictures, it starts to figure out common patterns and features, like dogs often have pointy ears, narrow faces, whiskers, etc. You can then ask it to draw a new picture of a dog. The AI will use the patterns it learned to generate a picture that hopefully looks like a dog. But remember, the AI is not copying any of the pictures it has seen before but creating a new image based on the patterns it has learned. This is the basic idea behind generative AI. In practice, the process involves a lot of complex maths and computation, and there are different techniques and architectures that can be used, such as variational autoencoders (VAs) and Generative Adversarial Networks (GANs). 02:48 Nikita: Himanshu, where is generative AI used in the real world? Himanshu: Generative AI models have a wide variety of applications across numerous domains. For the image generation, generative models like GANs are used to generate realistic images. They can be used for tasks, like creating artwork, synthesizing images of human faces, or transforming sketches into photorealistic images. For text generation, large language models like GPT 3, which are generative in nature, can create human-like text. This has applications in content creation, like writing articles, generating ideas, and again, conversational AI, like chat bots, customer service agents. They are also used in programming for code generation and debugging, and much more. For music generation, generative AI models can also be used. They create new pieces of music after being trained on a specific style or collection of tunes. A famous example is OpenAI's MuseNet. 03:42 Lois: You mentioned large language models in the context of text-based generative AI. So, let's talk a little more about it. Himanshu, what exactly are large language models? Himanshu: LLMs are a type of artificial intelligence models built to understand, generate, and process human language at a massive scale. They were primarily designed for sequence to sequence tasks such as machine translation, where an input sequence is transformed into an output sequence. LLMs can be used to translate text from one language to another. For example, an LLM could be used to translate English text into French. To do this job, LLM is trained on a massive data set of text and code which allows it to learn the patterns and relationships that exist between different languages. The LLM translates, “How are you?” from English to French, “Comment allez-vous?” It can also answer questions like, what is the capital of France? And it would answer the capital of France is Paris. And it will write an essay on a given topic. For example, write an essay on French Revolution, and it will come up with a response like with a title and introduction. 04:53 Lois: And how do LLMs actually work? Himanshu: So, LLM models are typically based on deep learning architectures such as transformers. They are also trained on vast amount of text data to learn language patterns and relationships, again, with a massive number of parameters usually in order of millions or even billions. LLMs have also the ability to comprehend and understand natural language text at a semantic level. They can grasp context, infer meaning, and identify relationships between words and phrases. 05:26 Nikita: What are the most important factors for a large language model? Himanshu: Model size and parameters are crucial aspects of large language models and other deep learning models. They significantly impact the model's capabilities, performance, and resource requirement. So, what is model size? The model size refers to the amount of memory required to store the model's parameter and other data structures. Larger model sizes generally led to better performance as they can capture more complex patterns and representation from the data. The parameters are the numerical values of the model that change as it learns to minimize the model's error on the given task. In the context of LLMs, parameters refer to the weights and biases of the model's transformer layers. Parameters are usually measured in terms of millions or billions. For example, GPT-3, one of the largest LLMs to date, has 175 billion parameters making it extremely powerful in language understanding and generation. Tokens represent the individual units into which a piece of text is divided during the processing by the model. In natural language, tokens are usually words, subwords, or characters. Some models have a maximum token limit that they can process and longer text can may require truncation or splitting. Again, balancing model size, parameters, and token handling is crucial when working with LLMs. 06:49 Nikita: But what's so great about LLMs? Himanshu: Large language models can understand and interpret human language more accurately and contextually. They can comprehend complex sentence structures, nuances, and word meanings, enabling them to provide more accurate and relevant responses to user queries. This model can generate human-like text that is coherent and contextually appropriate. This capability is valuable for context creation, automated writing, and generating personalized response in applications like chatbots and virtual assistants. They can perform a variety of tasks. Large language models are very versatile and adaptable to various industries. They can be customized to excel in applications such as language translation, sentiment analysis, code generation, and much more. LLMs can handle multiple languages making them valuable for cross-lingual tasks like translation, sentiment analysis, and understanding diverse global content. Large language models can be again, fine-tuned for a specific task using a minimal amount of domain data. The efficiency of LLMs usually grows with more data and parameters. 07:55 Lois: You mentioned the “sequence to sequence tasks” earlier. Can you explain the concept in simple terms for us? Himanshu: Understanding language is difficult for computers and AI systems. The reason being that words often have meanings based on context. Consider a sentence such as Jane threw the frisbee, and her dog fetched it. In this sentence, there are a few things that relate to each other. Jane is doing the throwing. The dog is doing the fetching. And it refers to the frisbee. Suppose we are looking at the word “it” in the sentence. As a human, we understand easily that “it” refers to the frisbee. But for a machine, it can be tricky. The goal in sequence problems is to find patterns, dependencies, or relationships within the data and make predictions, classification, or generate new sequences based on that understanding. 08:48 Lois: And where are sequence models mostly used? Himanshu: Some common example of sequence models includes natural language processing, which we call NLP, tasks such as machine translation, text generation sentiment analysis, language modeling involve dealing with sequences of words or characters. Speech recognition. Converting audio signals into text, involves working with sequences of phonemes or subword units to recognize spoken words. Music generation. Generating new music involves modeling musical sequences, nodes, and rhythms to create original compositions. Gesture recognition. Sequences of motion or hand gestures are used to interpret human movements for applications, such as sign language recognition or gesture-based interfaces. Time series analysis. In fields such as finance, economics, weather forecasting, and signal processing, time series data is used to predict future values, detect anomalies, and understand patterns in temporal data. 09:56 The Oracle University Learning Community is an excellent place to collaborate and learn with Oracle experts and fellow learners. Grow your skills, inspire innovation, and celebrate your successes. All your activities, from liking a post to answering questions and sharing with others, will help you earn a valuable reputation, badges, and ranks to be recognized in the community. Visit mylearn.oracle.com to get started. 10:23 Nikita: Welcome back! Himanshu, what would be the best way to solve those sequence problems you mentioned? Let's use the same sentence, “Jane threw the frisbee, and her dog fetched it” as an example. Himanshu: The solution is transformers. It's like model has a bird's eye view of the entire sentence and can see how all the words relate to each other. This allows it to understand the sentence as a whole instead of just a series of individual words. Transformers with their self-attention mechanism can look at all the words in the sentence at the same time and understand how they relate to each other. For example, transformer can simultaneously understand the connections between Jane and dog even though they are far apart in the sentence. 11:13 Nikita: But how? Himanshu: The answer is attention, which adds context to the text. Attention would notice dog comes after frisbee, fetched comes after dog, and it comes after fetched. Transformer does not look at it in isolation. Instead, it also pays attention to all the other words in the sentence at the same time. But considering all these connections, the model can figure out that “it” likely refers to the frisbee. The most famous current models that are emerging in natural language processing tasks consist of dozens of transformers or some of their variants, for example, GPT or Bert. 11:53 Lois: I was looking at the AI Foundations course on MyLearn and came across the terms “prompt engineering” and “fine tuning.” Can you shed some light on them? Himanshu: A prompt is the input or initial text provided to the model to elicit a specific response or behavior. So, this is something which you write or ask to a language model. Now, what is prompt engineering? So prompt engineering is the process of designing and formulating specific instructions or queries to interact with a large language model effectively. In the context of large language models, such as GPT 3 or Burt, prompts are the input text or questions given to the model to generate responses or perform specific tasks. The goal of prompt engineering is to ensure that the language model understands the user's intent correctly and provide accurate and relevant responses. 12:47 Nikita: That sounds easy enough, but fine tuning seems a bit more complex. Can you explain it with an example? Himanshu: Imagine you have a versatile recipe robot named chef bot. Suppose that chef bot is designed to create delicious recipes for any dish you desire. Chef bot recognizes the prompt as a request for a pizza recipe, and it knows exactly what to do. However, if you want chef bot to be an expert in a particular type of cuisine, such as Italian dishes, you fine-tune chef bot for Italian cuisine by immersing it in a culinary crash course filled with Italian cookbooks, traditional Italian recipes, and even Italian cooking shows. During this process, chef bot becomes more specialized in creating authentic Italian recipes, and this option is called fine tuning. LLMs are general purpose models that are pre-trained on large data sets but are often fine-tuned to address specific use cases. When you combine prompt engineering and fine tuning, and you get a culinary wizard in chef bot, a recipe robot that is not only great at understanding specific dish requests but also capable of following a specific dish requests and even mastering the art of cooking in a particular culinary style. 14:08 Lois: Great! Now that we've spoken about all the major components, can you walk us through the life cycle of a large language model? Himanshu: The life cycle of a Large Language Model, LLM, involves several stages, from its initial pre-training to its deployment and ongoing refinement. The first of this lifecycle is pre-training. The LLM is initially pre-trained on a large corpus of text data from the internet. During pre-training, the model learns grammar, facts, reasoning abilities, and general language understanding. The model predicts the next word in a sentence given the previous words, which helps it capture relationships between words and the structure of language. The second phase is fine tuning initialization. After pre-training, the model's weights are initialized, and it's ready for task-specific fine tuning. Fine tuning can involve supervised learning on labeled data for specific tasks, such as sentiment analysis, translation, or text generation. The model is fine-tuned on specific tasks using a smaller domain-specific data set. The weights from pre-training are updated based on the new data, making the model task aware and specialized. The next phase of the LLM life cycle is prompt engineering. So this phase craft effective prompts to guide the model's behavior in generating specific responses. Different prompt formulations, instructions, or context can be used to shape the output. 15:34 Nikita: Ok… we're with you so far. What's next? Himanshu: The next phase is evaluation and iteration. So models are evaluated using various metrics to access their performance on specific tasks. Iterative refinement involves adjusting model parameters, prompts, and fine tuning strategies to improve results. So as a part of this step, you also do few shot and one shot inference. If needed, you further fine tune the model with a small number of examples. Basically, few shot or a single example, one shot for new tasks or scenarios. Also, you do the bias mitigation and consider the ethical concerns. These biases and ethical concerns may arise in models output. You need to implement measures to ensure fairness in inclusivity and responsible use. 16:28 Himanshu: The next phase in LLM life cycle is deployment. Once the model has been fine-tuned and evaluated, it is deployed for real world applications. Deployed models can perform tasks, such as text generation, translation, summarization, and much more. You also perform monitoring and maintenance in this phase. So you continuously monitor the model's performance and output to ensure it aligns with desired outcomes. You also periodically update and retrain the model to incorporate new data and to adapt to evolving language patterns. This overall life cycle can also consist of a feedback loop, whether you gather feedbacks from users and incorporate it into the model's improvement process. You use this feedback to further refine prompts, fine tuning, and overall model behavior. RLHF, which is Reinforcement Learning with Human Feedback, is a very good example of this feedback loop. You also research and innovate as a part of this life cycle, where you continue to research and develop new techniques to enhance the model capability and address different challenges associated with it. 17:40 Nikita: As we're talking about the LLM life cycle, I see that fine tuning is not only about making an LLM task specific. So, what are some other reasons you would fine tune an LLM model? Himanshu: The first one is task-specific adaptation. Pre-trained language models are trained on extensive and diverse data sets and have good general language understanding. They excel in language generation and comprehension tasks, though the broad understanding of language may not lead to optimal performance in specific task. These models are not task specific. So the solution is fine tuning. The fine tuning process customizes the pre-trained models for a specific task by further training on task-specific data to adapt the model's knowledge. The second reason is domain-specific vocabulary. Pre-trained models might lack knowledge of specific words and phrases essential for certain tasks in fields, such as legal, medical, finance, and technical domains. This can limit their performance when applied to domain-specific data. Fine tuning enables the model to adapt and learn domain-specific words and phrases. These words could be, again, from different domains. 18:56 Himanshu: The third reason to fine tune is efficiency and resource utilization. So fine tuning is computationally efficient compared to training from scratch. Fine tuning reuses the knowledge from pre-trained models, saving time and resources. Fine tuning requires fewer iterations to achieve task-specific competence. Shorter training cycles expedite the model development process. It conserves computational resources, such as GPU memory and processing power. Fine tuning is efficient in quicker model deployment. It has faster time to production for real world applications. Fine tuning is, again, scalable, enabling adaptation to various tasks with the same base model, which further reduce resource demands, and it leads to cost saving for research and development. The fourth reason to fine tune is of ethical concerns. Pre-trained models learns from diverse data. And those potentially inherit different biases. Fine tune might not completely eliminate biases. But careful curation of task-specific data ensures avoiding biased or harmful vocabulary. The responsible uses of domain-specific terms promotes ethical AI applications. 20:14 Lois: Thank you so much, Himanshu, for spending time with us. We had such a great time learning from you. If you want to learn more about the topics discussed today, head over to mylearn.oracle.com and get started on our free AI Foundations course. Nikita: Yeah, we even have a detailed walkthrough of the architecture of transformers that you might want to check out. Join us next week for a discussion on the OCI AI Portfolio. Until then, this is Nikita Abraham… Lois: And Lois Houston signing off! 20:44 That's all for this episode of the Oracle University Podcast. If you enjoyed listening, please click Subscribe to get all the latest episodes. We'd also love it if you would take a moment to rate and review us on your podcast app. See you again on the next episode of the Oracle University Podcast.

Why Your Human Voice Matters in an AI World

Can You Hear Me?

Play Episode Listen Later Apr 17, 2024 32:33

The use of artificial intelligence has skyrocketed for businesses as they try to figure out how to incorporate this revolutionary tool into their day-to-day operations. AI now drives content creation, research, and on a broader scale, technology. However, we need to remember it lacks a human component, a component that is now seen as vital for many company leaders. Join “Can You Hear Me?” co-hosts Rob Johnson and Eileen Rochford as they explain “Why Your Human Voice Matters in an AI World.” Recommended ReadingHarvard Business Review - Bring Human Values to AI [LINK]Communications Leaders of Chicago - Major AI-Human Communications Research Project Launched by DePaul and Chicago PR Leaders [LINK] Thank you for listening to "Can You Hear Me?". If you enjoyed our show, please consider subscribing and leaving a review on your favorite podcast platform.Stay connected with us:Follow us on LinkedIn!Follow our co-host Eileen Rochford on Linkedin!Follow our co-host Rob Johnson on Linkedin!

ai artificial intelligence ethics b2b depaul human touch human values voice matters ai research energy consumption can you hear me ai policy rob johnson human voice human feedback

Trailer: Why Your Human Voice Matters in an AI World

Can You Hear Me?

Play Episode Listen Later Apr 16, 2024 0:50

Tune in on Wednesday, April 17th at 12 PM CST for this brand new episode.AI's widespread adoption in businesses has surged, revolutionizing operations. It powers content creation, research, and technology. But amidst its prowess, it lacks a human touch, deemed crucial by many leaders. Join hosts Rob Johnson and Eileen Rochford on 'Can You Hear Me?' as they delve into 'Why Your Human Voice Matters in an AI World.' Thank you for listening to "Can You Hear Me?". If you enjoyed our show, please consider subscribing and leaving a review on your favorite podcast platform.Stay connected with us:Follow us on LinkedIn!Follow our co-host Eileen Rochford on Linkedin!Follow our co-host Rob Johnson on Linkedin!

ai artificial intelligence ethics b2b human touch human values voice matters ai research energy consumption can you hear me ai policy rob johnson human voice human feedback

Amazon's $2.75B Investment

TalkRL: The Reinforcement Learning Podcast

Play Episode Listen Later Mar 29, 2024 13:56

Amazon's $2.75 billion investment in Anthropic, a top player in AI research and development, is the largest outside investment in history and could have significant implications for the AI arms race. The potential economic explosion caused by AI is explored in a Vox article, discussing how AI could cause economic growth at a scale never seen before. Anthropic's Claude 3 Opus language model has unseated OpenAI's GPT-4 on Chatbot Arena, marking a notable moment in the relatively short history of AI language models. Three impressive AI research papers are discussed, including a reproduction of OpenAI's TL;DR summarization work using Reinforcement Learning from Human Feedback, a joint model for list-aware retrieval that achieved state-of-the-art performance, and ViTAR, a cost-effective solution for enhancing the resolution scalability of Vision Transformers. Contact: sergi@earkind.com Timestamps: 00:34 Introduction 01:50 Amazon Invests $2.75B in Anthropic 03:29 How AI could explode the economy and how it could fizzle 04:39 “The king is dead”—Claude 3 surpasses GPT-4 on Chatbot Arena for the first time 06:17 Fake sponsor 08:02 The N+ Implementation Details of RLHF with PPO: A Case Study on TL;DR Summarization 09:24 List-aware Reranking-Truncation Joint Model for Search and Retrieval-augmented Generation 10:53 ViTAR: Vision Transformer with Any Resolution 12:46 Outro

amazon ai search fake investment upset openai vox gpt opus tl economic growth anthropic retrieval reinforcement learning language model 75b rlhf human feedback

Arash Ahmadian on Rethinking RLHF

Play Episode Listen Later Mar 25, 2024 33:30 Transcription Available

Arash Ahmadian is a Researcher at Cohere and Cohere For AI focussed on Preference Training of large language models. He's also a researcher at the Vector Institute of AI.Featured ReferenceBack to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMsArash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, Sara HookerAdditional ReferencesSelf-Rewarding Language Models, Yuan et al 2024 Reinforcement Learning: An Introduction, Sutton and Barto 1992Learning from Delayed Rewards, Chris Watkins 1989Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning, Williams 1992

learning ai artificial intelligence researchers rethinking machine learning yuan arash ahmet reinforcement learning cohere rlhf vector institute human feedback

#160 - Nvidia's new GPU, Microsoft pays for Inflection AI, Grok-1 open sourced, Jeremie's Action Plan

Play Episode Listen Later Mar 24, 2024 99:34

Our 160th episode with a summary and discussion of last week's big AI news! Read out our text newsletter and comment on the podcast at https://lastweekin.ai/ Email us your questions and feedback at contact@lastweekin.ai and/or hello@gladstone.ai Timestamps + links: Tools & Apps (00:01:36) Adobe Substance 3D's AI features can turn text into backgrounds and textures (00:05:04) OpenAI's chatbot store is filling up with spam (00:11:02) Apple's AI ambitions could include Google or OpenAI Applications & Business (00:13:31) Nvidia reveals Blackwell B200 GPU, the ‘world's most powerful chip' for AI (00:19:34) Microsoft to Pay Inflection AI $650 Million After Scooping Up Most of Staff (00:24:33) Figure 01: Conversations & Actions in Humanoid Robotics! (00:28:07) OpenAI's GPT-4.5 Turbo leaked on search engines and could launch in June (00:30:32) Abu Dhabi in talks to invest in OpenAI chip venture (00:33:43) Nvidia Announces GR00T, a Foundation Model For Humanoids Projects & Open Source (00:35:38) Open Release of Grok-1 (00:41:25) Stability AI brings a new dimension to video with Stable Video 3D (00:44:23) Colossal-AI Team Introduces Open-Sora: An Open-Source Library for Video Generation (00:45:43) Evolutionary Optimization of Model Merging Recipes Research & Advancements (00:46:52) DiPaCo: Distributed Path Composition (00:53:58) MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training (00:59:37) PERL: Parameter Efficient Reinforcement Learning from Human Feedback (01:01:55) VideoAgent: Long-form Video Understanding with Large Language Model as Agent (01:05:38) MusicHiFi: Fast High-Fidelity Stereo Vocoding Policy & Safety (01:06:44) The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning (01:12:43) Exclusive: U.S. Must Move ‘Decisively' to Avert ‘Extinction-Level' Threat From AI, Government-Commissioned Report Says (01:18:18) Evaluating Frontier Models for Dangerous Capabilities (01:22:55) Chinese and western scientists identify ‘red lines' on AI risks (01:26:11) Google fined $272M by French government over AI use of news content (01:27:58) Elvis Act Signed Into Tennessee Law to Protect Musicians From AI Deepfakes Synthetic Media & Art (01:30:17) AI-Generated Science (01:35:35) YouTube adds new AI-generated content labeling tool Fun! (01:37:29) 10 of My Most Popular Text-To-Image Series (+Prompts)

#157 - Gemini controversy, new Mistral models, Deepmind's Genie & Griffinn, AI Warfare is here

Oracle University Podcast

Play Episode Listen Later Mar 3, 2024 104:57 Transcription Available

Our 157th episode with a summary and discussion of last week's big AI news! Check out our sponsor, the SuperDataScience podcast. You can listen to SDS across all major podcasting platforms (e.g., Spotify, Apple Podcasts, Google Podcasts) plus there's a video version on YouTube. Bonus plug: also check out this new book by Stanford AI expert, bestselling author, and Last Week in AI supporter Jerry Kaplan! Generative Artificial Intelligence: What Everyone Needs to Know Read out our text newsletter and comment on the podcast at https://lastweekin.ai/ Email us your questions and feedback at contact@lastweekin.ai and/or hello@gladstone.ai Timestamps + links: (00:00:00) Intro / Banter Tools & Apps (00:04:55) Google apologizes for ‘missing the mark' after Gemini generated racially diverse Nazis (00:15:07) Stability announces Stable Diffusion 3, a next-gen AI image generator (00:19:04) Mistral AI releases new model to rival GPT-4 and its own chat assistant (00:24:56) Windows just got its own Magic Eraser to AI-modify your photos (00:25:47) Adobe previews new cutting-edge generative AI tools for crafting and editing custom audio (00:27:47) AI video wars heat up as Pika adds Lip Sync powered by ElevenLabs Applications & Business (00:30:09) Microsoft Strikes Deal with France's Mistral AI (00:33:45) Figure Raises $675M at $2.6B Valuation and Signs Collaboration Agreement with OpenAI (00:37:05) Nvidia posts record revenue up 265% on booming AI business (00:39:54) MediaTek's latest chipsets are now ‘optimized' for Gemini Nano (00:41:28) Tumblr's owner is striking deals with OpenAI and Midjourney for training data, says report (00:43:45) Mistral AI models coming soon to Amazon Bedrock Projects & Open Source (00:44:34) Generative AI Startup Mistral Releases Free ‘Open-Source' 7.3B Parameter LLM (00:46:57) Google Delves Deeper Into Open Source with Launch of Gemma AI Model (00:51:10) Microsoft releases its internal generative AI red teaming tool to the public (00:54:25) Introducing Phind-70B – closing the code quality gap with GPT-4 Turbo while running 4x faster Research & Advancements (00:57:51) Genie: Generative Interactive Environments (00:01:07) Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models (01:15:16) Quantum Circuit Optimization with AlphaTensor (01:20:56) Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs (01:22:10) Repetition Improves Language Model Embeddings Policy & Safety (01:25:45) AI Warfare Is Already Here (01:29:36) Man admits to paying magician $150 to create anti-Biden robocall (01:32:45) Google DeepMind forms a new org focused on AI safety (01:34:53) Facebook whistleblower, AI godfather join hundreds calling for deepfake regulation US regulators investigate whether OpenAI investors were misled, say reports (01:36:51) Users Say Microsoft's AI Has Alternate Personality as Godlike AGI That Demands to Be Worshipped Synthetic Media & Art (01:40:23) The Intercept, Raw Story, and AlterNet sue OpenAI and Microsoft (01:41:15) A viral photo of a guy smoking in McDonald's is completely fake — and of course made by AI Fun! (01:42:54) Impossible AI Food

Generative AI and Large Language Models

Play Episode Listen Later Feb 27, 2024 20:52

In this week's episode, Lois Houston and Nikita Abraham, along with Senior Instructor Himanshu Raj, take you through the extraordinary capabilities of Generative AI, a subset of deep learning that doesn't make predictions but rather creates its own content. They also explore the workings of Large Language Models. Oracle MyLearn: https://mylearn.oracle.com/ou/learning-path/become-an-oci-ai-foundations-associate-2023/127177 Oracle University Learning Community: https://education.oracle.com/ou-community LinkedIn: https://www.linkedin.com/showcase/oracle-university/ X (formerly Twitter): https://twitter.com/Oracle_Edu Special thanks to Arijit Ghosh, David Wright, and the OU Studio Team for helping us create this episode. -------------------------------------------------------- Episode Transcript: 00:00 Welcome to the Oracle University Podcast, the first stop on your cloud journey. During this series of informative podcasts, we'll bring you foundational training on the most popular Oracle technologies. Let's get started! 00:26 Lois: Hello and welcome to the Oracle University Podcast. I'm Lois Houston, Director of Innovation Programs with Oracle University, and with me is Nikita Abraham, Principal Technical Editor. Nikita: Hi everyone! In our last episode, we went over the basics of deep learning. Today, we'll look at generative AI and large language models, and discuss how they work. To help us with that, we have Himanshu Raj, Senior Instructor on AI/ML. So, let's jump right in. Hi Himanshu, what is generative AI? 01:00 Himanshu: Generative AI refers to a type of AI that can create new content. It is a subset of deep learning, where the models are trained not to make predictions but rather to generate output on their own. Think of generative AI as an artist who looks at a lot of paintings and learns the patterns and styles present in them. Once it has learned these patterns, it can generate new paintings that resembles what it learned. 01:27 Lois: Let's take an example to understand this better. Suppose we want to train a generative AI model to draw a dog. How would we achieve this? Himanshu: You would start by giving it a lot of pictures of dogs to learn from. The AI does not know anything about what a dog looks like. But by looking at these pictures, it starts to figure out common patterns and features, like dogs often have pointy ears, narrow faces, whiskers, etc. You can then ask it to draw a new picture of a dog. The AI will use the patterns it learned to generate a picture that hopefully looks like a dog. But remember, the AI is not copying any of the pictures it has seen before but creating a new image based on the patterns it has learned. This is the basic idea behind generative AI. In practice, the process involves a lot of complex maths and computation, and there are different techniques and architectures that can be used, such as variational autoencoders (VAs) and Generative Adversarial Networks (GANs). 02:27 Nikita: Himanshu, where is generative AI used in the real world? Himanshu: Generative AI models have a wide variety of applications across numerous domains. For the image generation, generative models like GANs are used to generate realistic images. They can be used for tasks, like creating artwork, synthesizing images of human faces, or transforming sketches into photorealistic images. For text generation, large language models like GPT 3, which are generative in nature, can create human-like text. This has applications in content creation, like writing articles, generating ideas, and again, conversational AI, like chat bots, customer service agents. They are also used in programming for code generation and debugging, and much more. For music generation, generative AI models can also be used. They create new pieces of music after being trained on a specific style or collection of tunes. A famous example is OpenAI's MuseNet. 03:21 Lois: You mentioned large language models in the context of text-based generative AI. So, let's talk a little more about it. Himanshu, what exactly are large language models? Himanshu: LLMs are a type of artificial intelligence models built to understand, generate, and process human language at a massive scale. They were primarily designed for sequence to sequence tasks such as machine translation, where an input sequence is transformed into an output sequence. LLMs can be used to translate text from one language to another. For example, an LLM could be used to translate English text into French. To do this job, LLM is trained on a massive data set of text and code which allows it to learn the patterns and relationships that exist between different languages. The LLM translates, “How are you?” from English to French, “Comment allez-vous?” It can also answer questions like, what is the capital of France? And it would answer the capital of France is Paris. And it will write an essay on a given topic. For example, write an essay on French Revolution, and it will come up with a response like with a title and introduction. 04:33 Lois: And how do LLMs actually work? Himanshu: So, LLM models are typically based on deep learning architectures such as transformers. They are also trained on vast amount of text data to learn language patterns and relationships, again, with a massive number of parameters usually in order of millions or even billions. LLMs have also the ability to comprehend and understand natural language text at a semantic level. They can grasp context, infer meaning, and identify relationships between words and phrases. 05:05 Nikita: What are the most important factors for a large language model? Himanshu: Model size and parameters are crucial aspects of large language models and other deep learning models. They significantly impact the model's capabilities, performance, and resource requirement. So, what is model size? The model size refers to the amount of memory required to store the model's parameter and other data structures. Larger model sizes generally led to better performance as they can capture more complex patterns and representation from the data. The parameters are the numerical values of the model that change as it learns to minimize the model's error on the given task. In the context of LLMs, parameters refer to the weights and biases of the model's transformer layers. Parameters are usually measured in terms of millions or billions. For example, GPT-3, one of the largest LLMs to date, has 175 billion parameters making it extremely powerful in language understanding and generation. Tokens represent the individual units into which a piece of text is divided during the processing by the model. In natural language, tokens are usually words, subwords, or characters. Some models have a maximum token limit that they can process and longer text can may require truncation or splitting. Again, balancing model size, parameters, and token handling is crucial when working with LLMs. 06:29 Nikita: But what's so great about LLMs? Himanshu: Large language models can understand and interpret human language more accurately and contextually. They can comprehend complex sentence structures, nuances, and word meanings, enabling them to provide more accurate and relevant responses to user queries. This model can generate human-like text that is coherent and contextually appropriate. This capability is valuable for context creation, automated writing, and generating personalized response in applications like chatbots and virtual assistants. They can perform a variety of tasks. Large language models are very versatile and adaptable to various industries. They can be customized to excel in applications such as language translation, sentiment analysis, code generation, and much more. LLMs can handle multiple languages making them valuable for cross-lingual tasks like translation, sentiment analysis, and understanding diverse global content. Large language models can be again, fine-tuned for a specific task using a minimal amount of domain data. The efficiency of LLMs usually grows with more data and parameters. 07:34 Lois: You mentioned the “sequence to sequence tasks” earlier. Can you explain the concept in simple terms for us? Himanshu: Understanding language is difficult for computers and AI systems. The reason being that words often have meanings based on context. Consider a sentence such as Jane threw the frisbee, and her dog fetched it. In this sentence, there are a few things that relate to each other. Jane is doing the throwing. The dog is doing the fetching. And it refers to the frisbee. Suppose we are looking at the word “it” in the sentence. As a human, we understand easily that “it” refers to the frisbee. But for a machine, it can be tricky. The goal in sequence problems is to find patterns, dependencies, or relationships within the data and make predictions, classification, or generate new sequences based on that understanding. 08:27 Lois: And where are sequence models mostly used? Himanshu: Some common example of sequence models includes natural language processing, which we call NLP, tasks such as machine translation, text generation sentiment analysis, language modeling involve dealing with sequences of words or characters. Speech recognition. Converting audio signals into text, involves working with sequences of phonemes or subword units to recognize spoken words. Music generation. Generating new music involves modeling musical sequences, nodes, and rhythms to create original compositions. Gesture recognition. Sequences of motion or hand gestures are used to interpret human movements for applications, such as sign language recognition or gesture-based interfaces. Time series analysis. In fields such as finance, economics, weather forecasting, and signal processing, time series data is used to predict future values, detect anomalies, and understand patterns in temporal data. 09:35 The Oracle University Learning Community is an excellent place to collaborate and learn with Oracle experts and fellow learners. Grow your skills, inspire innovation, and celebrate your successes. All your activities, from liking a post to answering questions and sharing with others, will help you earn a valuable reputation, badges, and ranks to be recognized in the community. Visit mylearn.oracle.com to get started. 10:03 Nikita: Welcome back! Himanshu, what would be the best way to solve those sequence problems you mentioned? Let's use the same sentence, “Jane threw the frisbee, and her dog fetched it” as an example. Himanshu: The solution is transformers. It's like model has a bird's eye view of the entire sentence and can see how all the words relate to each other. This allows it to understand the sentence as a whole instead of just a series of individual words. Transformers with their self-attention mechanism can look at all the words in the sentence at the same time and understand how they relate to each other. For example, transformer can simultaneously understand the connections between Jane and dog even though they are far apart in the sentence. 10:52 Nikita: But how? Himanshu: The answer is attention, which adds context to the text. Attention would notice dog comes after frisbee, fetched comes after dog, and it comes after fetched. Transformer does not look at it in isolation. Instead, it also pays attention to all the other words in the sentence at the same time. But considering all these connections, the model can figure out that “it” likely refers to the frisbee. The most famous current models that are emerging in natural language processing tasks consist of dozens of transformers or some of their variants, for example, GPT or Bert. 11:32 Lois: I was looking at the AI Foundations course on MyLearn and came across the terms “prompt engineering” and “fine tuning.” Can you shed some light on them? Himanshu: A prompt is the input or initial text provided to the model to elicit a specific response or behavior. So, this is something which you write or ask to a language model. Now, what is prompt engineering? So prompt engineering is the process of designing and formulating specific instructions or queries to interact with a large language model effectively. In the context of large language models, such as GPT 3 or Burt, prompts are the input text or questions given to the model to generate responses or perform specific tasks. The goal of prompt engineering is to ensure that the language model understands the user's intent correctly and provide accurate and relevant responses. 12:26 Nikita: That sounds easy enough, but fine tuning seems a bit more complex. Can you explain it with an example? Himanshu: Imagine you have a versatile recipe robot named chef bot. Suppose that chef bot is designed to create delicious recipes for any dish you desire. Chef bot recognizes the prompt as a request for a pizza recipe, and it knows exactly what to do. However, if you want chef bot to be an expert in a particular type of cuisine, such as Italian dishes, you fine-tune chef bot for Italian cuisine by immersing it in a culinary crash course filled with Italian cookbooks, traditional Italian recipes, and even Italian cooking shows. During this process, chef bot becomes more specialized in creating authentic Italian recipes, and this option is called fine tuning. LLMs are general purpose models that are pre-trained on large data sets but are often fine-tuned to address specific use cases. When you combine prompt engineering and fine tuning, and you get a culinary wizard in chef bot, a recipe robot that is not only great at understanding specific dish requests but also capable of following a specific dish requests and even mastering the art of cooking in a particular culinary style. 13:47 Lois: Great! Now that we've spoken about all the major components, can you walk us through the life cycle of a large language model? Himanshu: The life cycle of a Large Language Model, LLM, involves several stages, from its initial pre-training to its deployment and ongoing refinement. The first of this lifecycle is pre-training. The LLM is initially pre-trained on a large corpus of text data from the internet. During pre-training, the model learns grammar, facts, reasoning abilities, and general language understanding. The model predicts the next word in a sentence given the previous words, which helps it capture relationships between words and the structure of language. The second phase is fine tuning initialization. After pre-training, the model's weights are initialized, and it's ready for task-specific fine tuning. Fine tuning can involve supervised learning on labeled data for specific tasks, such as sentiment analysis, translation, or text generation. The model is fine-tuned on specific tasks using a smaller domain-specific data set. The weights from pre-training are updated based on the new data, making the model task aware and specialized. The next phase of the LLM life cycle is prompt engineering. So this phase craft effective prompts to guide the model's behavior in generating specific responses. Different prompt formulations, instructions, or context can be used to shape the output. 15:13 Nikita: Ok… we're with you so far. What's next? Himanshu: The next phase is evaluation and iteration. So models are evaluated using various metrics to access their performance on specific tasks. Iterative refinement involves adjusting model parameters, prompts, and fine tuning strategies to improve results. So as a part of this step, you also do few shot and one shot inference. If needed, you further fine tune the model with a small number of examples. Basically, few shot or a single example, one shot for new tasks or scenarios. Also, you do the bias mitigation and consider the ethical concerns. These biases and ethical concerns may arise in models output. You need to implement measures to ensure fairness in inclusivity and responsible use. 16:07 Himanshu: The next phase in LLM life cycle is deployment. Once the model has been fine-tuned and evaluated, it is deployed for real world applications. Deployed models can perform tasks, such as text generation, translation, summarization, and much more. You also perform monitoring and maintenance in this phase. So you continuously monitor the model's performance and output to ensure it aligns with desired outcomes. You also periodically update and retrain the model to incorporate new data and to adapt to evolving language patterns. This overall life cycle can also consist of a feedback loop, whether you gather feedbacks from users and incorporate it into the model's improvement process. You use this feedback to further refine prompts, fine tuning, and overall model behavior. RLHF, which is Reinforcement Learning with Human Feedback, is a very good example of this feedback loop. You also research and innovate as a part of this life cycle, where you continue to research and develop new techniques to enhance the model capability and address different challenges associated with it. 17:19 Nikita: As we're talking about the LLM life cycle, I see that fine tuning is not only about making an LLM task specific. So, what are some other reasons you would fine tune an LLM model? Himanshu: The first one is task-specific adaptation. Pre-trained language models are trained on extensive and diverse data sets and have good general language understanding. They excel in language generation and comprehension tasks, though the broad understanding of language may not lead to optimal performance in specific task. These models are not task specific. So the solution is fine tuning. The fine tuning process customizes the pre-trained models for a specific task by further training on task-specific data to adapt the model's knowledge. The second reason is domain-specific vocabulary. Pre-trained models might lack knowledge of specific words and phrases essential for certain tasks in fields, such as legal, medical, finance, and technical domains. This can limit their performance when applied to domain-specific data. Fine tuning enables the model to adapt and learn domain-specific words and phrases. These words could be, again, from different domains. 18:35 Himanshu: The third reason to fine tune is efficiency and resource utilization. So fine tuning is computationally efficient compared to training from scratch. Fine tuning reuses the knowledge from pre-trained models, saving time and resources. Fine tuning requires fewer iterations to achieve task-specific competence. Shorter training cycles expedite the model development process. It conserves computational resources, such as GPU memory and processing power. Fine tuning is efficient in quicker model deployment. It has faster time to production for real world applications. Fine tuning is, again, a scalable enabling adaptation to various tasks with the same base model, which further reduce resource demands, and it leads to cost saving for research and development. The fourth reason to fine tune is of ethical concerns. Pre-trained models learns from diverse data. And those potentially inherit different biases. Fine tune might not completely eliminate biases. But careful curation of task specific data ensures avoiding biased or harmful vocabulary. The responsible uses of domain-specific terms promotes ethical AI applications. 19:53 Lois: Thank you so much, Himanshu, for spending time with us. We had such a great time learning from you. If you want to learn more about the topics discussed today, head over to mylearn.oracle.com and get started on our free AI Foundations course. Nikita: Yeah, we even have a detailed walkthrough of the architecture of transformers that you might want to check out. Join us next week for a discussion on the OCI AI Portfolio. Until then, this is Nikita Abraham… Lois: And Lois Houston signing off! 20:24 That's all for this episode of the Oracle University Podcast. If you enjoyed listening, please click Subscribe to get all the latest episodes. We'd also love it if you would take a moment to rate and review us on your podcast app. See you again on the next episode of the Oracle University Podcast.

"AI Risk Debate" For Humanity: An AI Safety Podcast Episode #12 Theo Jaffee Interview

For Humanity: An AI Safety Podcast

Play Episode Listen Later Jan 25, 2024 100:15

In Episode #12, we have our first For Humanity debate!! John talks with Theo Jaffee, a fast-rising AI podcaster who is a self described “techno-optimist.” The debate covers a wide range of topics in AI risk. This podcast is not journalism. But it's not opinion either. This show simply strings together the existing facts and underscores the unthinkable probable outcome, the end of all life on earth. For Humanity: An AI Safety Podcast, is the accessible AI Safety Podcast for all humans, no tech background required. Our show focuses solely on the threat of human extinction from AI. Peabody Award-winning former journalist John Sherman explores the shocking worst-case scenario of artificial intelligence: human extinction. The makers of AI openly admit it their work could kill all humans, in as soon as 2 years. This podcast is solely about the threat of human extinction from AGI. We'll meet the heroes and villains, explore the issues and ideas, and what you can do to help save humanity. Resources Theo's YouTube Channel : https://youtube.com/@theojaffee8530?si=aBnWNdViCiL4ZaEg Glossary: First Definitions by ChaptGPT4, I asked it to give answers simple enough elementary school student could understand( lol, I find this helpful often!) Reinforcement Learning with Human Feedback (RLHF): Definition: RLHF, or Reinforcement Learning with Human Feedback, is like teaching a computer to make decisions by giving it rewards when it does something good and telling it what's right when it makes a mistake. It's a way for computers to learn and get better at tasks with the help of guidance from humans, just like how a teacher helps students learn. So, it's like a teamwork between people and computers to make the computer really smart! Model Weights Definiton: Model weights are like the special numbers that help a computer understand and remember things. Imagine it's like a recipe book, and these weights are the amounts of ingredients needed to make a cake. When the computer learns new things, these weights get adjusted so that it gets better at its job, just like changing the recipe to make the cake taste even better! So, model weights are like the secret ingredients that make the computer really good at what it does. Foom/Fast Take-off: Definition: "AI fast take-off" or "foom" refers to the idea that artificial intelligence (AI) could become super smart and powerful really quickly. It's like imagining a computer getting super smart all of a sudden, like magic! Some people use the word "foom" to talk about the possibility of AI becoming super intelligent in a short amount of time. It's a bit like picturing a computer going from learning simple things to becoming incredibly smart in the blink of an eye! Foom comes from cartoons, it's the sound a super hero makes in comic books when they burst off the ground into flight. Gradient Descent: Gradient descent is like a treasure hunt for the best way to do something. Imagine you're on a big hill with a metal detector, trying to find the lowest point. The detector beeps louder when you're closer to the lowest spot. In gradient descent, you adjust your steps based on these beeps to reach the lowest point on the hill, and in the computer world, it helps find the best values for a task, like making a robot walk smoothly or a computer learn better. Orthoginality: Orthogonality is like making sure things are independent and don't mess each other up. Think of a chef organizing ingredients on a table – if each ingredient has its own space and doesn't mix with others, it's easier to work. In computers, orthogonality means keeping different parts separate, so changing one thing doesn't accidentally affect something else. It's like having a well-organized kitchen where each tool has its own place, making it easy to cook without chaos!

ai debate risk humanity agi peabody award reinforcement learning jaffee john sherman safety podcast foom human feedback

LW - The True Story of How GPT-2 Became Maximally Lewd by Writer

The Nonlinear Library: LessWrong

Play Episode Listen Later Jan 19, 2024 10:46

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The True Story of How GPT-2 Became Maximally Lewd, published by Writer on January 19, 2024 on LessWrong. This video recounts an incident that occurred at OpenAI in which flipping a single minus sign led the RLHF process to make GPT-2 only output sexually explicit continuations. The incident is described in OpenAI's paper "Fine-Tuning Language Models from Human Preferences" under section 4.4: "Bugs can optimize for bad behavior". The script has been written by Jai, with some significant input and rework by me, Writer. You can read it below. In 2019, one OpenAI researcher made a typo - and birthed an evil AI hell-bent on making everything as horny as possible. This is the absurd, ridiculous, and yet true story of how it happened. Part I: GPT Since 2017, OpenAI has been building Generative Pre-trained Transformer models, or GPTs - language AIs with a singular focus on predicting text, trained across billions of writing samples. If you prompt a GPT model with "Once upon a ", it would predict "time" to follow. Asked for further predictions, the same GPT model might continue "there was a… brave dog named Grace", and so on - because those are the kinds of words that it expects to come next. In this example the GPT model has essentially learned to write a fairy tale, simply as a consequence of getting very, very good at text prediction. And it was exactly these kinds of emergent capabilities that had OpenAI so excited. These models can do a lot more than fairy tales. OpenAI's first GPT model, often called GPT-1, had been trained on excerpts from thousands of books. It showed so much promise that OpenAI almost immediately decided to train a much bigger model that could do more. But bigger models need more training data, and for this model, books would not be enough. No - this model would be trained on...the Internet. OpenAI trained GPT-2 to imitate writing across eight million web pages. And in learning to predict such an overwhelming quantity and variety of writing, GPT-2 acquired some surprising capabilities. With the right prompt, it could translate documents, answer questions about a text, summarize passages, and sometimes even write like a human. It was a shockingly powerful model. In fact, it may have been too powerful. GPT-2 wouldn't hesitate to plan crimes, instruct terrorists on bomb-making, create sexually explicit content, or promote cruelty, hatred and misinformation. And this was unacceptable to OpenAI - They wanted a model that did more than just predict text - they wanted a model that operated in accordance with some kind of human values, or at least with their values. But the GPT-2 architecture had no place for ethics, guidelines, principles, or corporate PR policies. It couldn't be bullied, reasoned or negotiated with. Nothing would sway the machine from its utter devotion to generating realistic text. But OpenAI was determined to get their model under control. So they got to work... not yet realizing that this work, along with a single typo, would lead to the one thing they didn't want to happen. Part II: Human Feedback To align GPT-2, OpenAI used a new technique known as "Reinforcement Learning from Human Feedback", or "RLHF". We're going to outline a simplified form of RLHF here, but if you want all the juicy technical details check out the link in the description. The goal of RLHF is to take a basic starting language model, some plain-language guidelines, and a small group of humans providing feedback, and produce a new model that follows those guidelines. We can think of this model-in-training as the "Apprentice". The apprentice begins the training process as an exact copy of GPT-2. During training, it gets prompts and generates responses, also called "continuations". Those prompts and continuations are sent to the human evaluators, who rate them based o...

ai internet pr writer speech true stories ea openai bugs apprentice j'ai gpt transformer gpts lewd reinforcement learning rationalist generative pre rlhf lesswrong human feedback

LW - The True Story of How GPT-2 Became Maximally Lewd by Writer

Play Episode Listen Later Jan 19, 2024 10:46

Link to original articleWelcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The True Story of How GPT-2 Became Maximally Lewd, published by Writer on January 19, 2024 on LessWrong. This video recounts an incident that occurred at OpenAI in which flipping a single minus sign led the RLHF process to make GPT-2 only output sexually explicit continuations. The incident is described in OpenAI's paper "Fine-Tuning Language Models from Human Preferences" under section 4.4: "Bugs can optimize for bad behavior". The script has been written by Jai, with some significant input and rework by me, Writer. You can read it below. In 2019, one OpenAI researcher made a typo - and birthed an evil AI hell-bent on making everything as horny as possible. This is the absurd, ridiculous, and yet true story of how it happened. Part I: GPT Since 2017, OpenAI has been building Generative Pre-trained Transformer models, or GPTs - language AIs with a singular focus on predicting text, trained across billions of writing samples. If you prompt a GPT model with "Once upon a ", it would predict "time" to follow. Asked for further predictions, the same GPT model might continue "there was a… brave dog named Grace", and so on - because those are the kinds of words that it expects to come next. In this example the GPT model has essentially learned to write a fairy tale, simply as a consequence of getting very, very good at text prediction. And it was exactly these kinds of emergent capabilities that had OpenAI so excited. These models can do a lot more than fairy tales. OpenAI's first GPT model, often called GPT-1, had been trained on excerpts from thousands of books. It showed so much promise that OpenAI almost immediately decided to train a much bigger model that could do more. But bigger models need more training data, and for this model, books would not be enough. No - this model would be trained on...the Internet. OpenAI trained GPT-2 to imitate writing across eight million web pages. And in learning to predict such an overwhelming quantity and variety of writing, GPT-2 acquired some surprising capabilities. With the right prompt, it could translate documents, answer questions about a text, summarize passages, and sometimes even write like a human. It was a shockingly powerful model. In fact, it may have been too powerful. GPT-2 wouldn't hesitate to plan crimes, instruct terrorists on bomb-making, create sexually explicit content, or promote cruelty, hatred and misinformation. And this was unacceptable to OpenAI - They wanted a model that did more than just predict text - they wanted a model that operated in accordance with some kind of human values, or at least with their values. But the GPT-2 architecture had no place for ethics, guidelines, principles, or corporate PR policies. It couldn't be bullied, reasoned or negotiated with. Nothing would sway the machine from its utter devotion to generating realistic text. But OpenAI was determined to get their model under control. So they got to work... not yet realizing that this work, along with a single typo, would lead to the one thing they didn't want to happen. Part II: Human Feedback To align GPT-2, OpenAI used a new technique known as "Reinforcement Learning from Human Feedback", or "RLHF". We're going to outline a simplified form of RLHF here, but if you want all the juicy technical details check out the link in the description. The goal of RLHF is to take a basic starting language model, some plain-language guidelines, and a small group of humans providing feedback, and produce a new model that follows those guidelines. We can think of this model-in-training as the "Apprentice". The apprentice begins the training process as an exact copy of GPT-2. During training, it gets prompts and generates responses, also called "continuations". Those prompts and continuations are sent to the human evaluators, who rate them based o...

ai internet pr writer speech true stories ea openai bugs apprentice j'ai gpt transformer gpts lewd reinforcement learning rationalist generative pre rlhf lesswrong human feedback

2023 in AI, with Nathan Benaich

The Gradient Podcast

Play Episode Listen Later Dec 28, 2023 95:37

In episode 104 of The Gradient Podcast, Daniel Bashir speaks to Nathan Benaich.Nathan is Founder and General Partner at Air Street Capital, a VC firm focused on investing in AI-first technology and life sciences companies. Nathan runs a number of communities focused on AI including the Research and Applied AI Summit and leads Spinout.fyi to improve the creation of university spinouts. Nathan co-authors the State of AI Report.Have suggestions for future podcast guests (or other feedback)? Let us know here or reach us at editor@thegradient.pubSubscribe to The Gradient Podcast: Apple Podcasts | Spotify | Pocket Casts | RSSFollow The Gradient on TwitterOutline:* (00:00) Intro* (02:00) Updates in Nathan World — Air Street's second fund, spinouts, * (07:30) Events: Research and Applied AI Summit, State of AI Report launches* (09:50) The State of AI: main messages, the increasing role of subject matter experts* Research* (14:13) Open and closed-source* (17:55) Benchmarking and evaluation, small/large models and industry verticals* (21:10) “Vibes” in LLM evaluation* (24:00) Codegen models, personalized AI, curriculum learning* (26:20) The exhaustion of human-generated data, lukewarm content, synthetic data* (29:50) Opportunities for AI applications in the natural sciences* (35:15) Reinforcement Learning from Human Feedback and alternatives* (38:30) Industry* (39:00) ChatGPT and productivity* (42:37) General app wars, ChatGPT competitors* (45:50) Compute—demand, supply, competition* (50:55) Export controls and geopolitics* (54:45) Startup funding and compute spend* (59:15) Politics* (59:40) Calls for regulation, regulatory divergence* (1:04:40) AI safety* (1:07:30) Nathan's perspective on regulatory approaches* (1:12:30) The UK's early access to frontier models, standards setting, regulation difficulties* (1:17:20) Jailbreaking, constitutional AI, robustness* (1:20:50) Predictions!* (1:25:00) Generative AI misuse in elections and politics (and, this prediction coming true in Bangladesh)* (1:26:50) Progress on AI governance* (1:30:30) European dynamism* (1:35:08) OutroLinks:* Nathan's homepage and Twitter* The 2023 State of AI Report* Bringing Dynamism to European Defense* A prediction coming true: How AI is disrupting Bangladesh's election* Air Street Capital is hiring a full-time Community Lead! Get full access to The Gradient at thegradientpub.substack.com/subscribe

The Busy Person's Intro to Finetuning & Open Source AI - Wing Lian, Axolotl

Latent Space: The AI Engineer Podcast â€” CodeGen, Agents, Computer Vision, Data Science, AI UX and all things Software 3.0

Play Episode Listen Later Dec 8, 2023 64:18

The Latent Space crew will be at NeurIPS on Tuesday! Reach out with any parties and papers of interest. We have also been incubating a smol daily AI Newsletter and Latent Space University is making progress.Good open models like Llama 2 and Mistral 7B (which has just released an 8x7B MoE model) have enabled their own sub-industry of finetuned variants for a myriad of reasons:* Ownership & Control - you take responsibility for serving the models* Privacy - not having to send data to a third party vendor* Customization - Improving some attribute (censorship, multiturn chat and chain of thought, roleplaying) or benchmark performance (without cheating)Related to improving benchmark performance is the ability to use smaller (7B, 13B) models, by matching the performance of larger models, which have both cost and inference latency benefits.Core to all this work is finetuning, and the emergent finetuning library of choice has been Wing Lian's Axolotl.AxolotlAxolotl is an LLM fine-tuner supporting SotA techniques and optimizations for a variety of common model architectures:It is used by many of the leading open source models:* Teknium: OpenHermes, Trismigestus, CollectiveCognition* OpenOrca: Mistral-OpenOrca, Mistral-SlimOrca* Nous Research: Puffin, Capybara, NousHermes* Pygmalion: Mythalion, Pygmalion* Eric Hartford: Dolphin, Samantha* DiscoResearch: DiscoLM 120B & 70B* OpenAccess AI Collective: Manticore, Minotaur, Jackalope, HippogriffAs finetuning is very formatting dependent, it also provides prompt interfaces and formatters between a range of popular model formats from Stanford's Alpaca and Steven Tey's ShareGPT (which led to Vicuna) to the more NSFW Pygmalion community.Nous Research MeetupWe last talked about Nous at the DevDay Recap at the e/acc “banger rave”. We met Wing at the Nous Research meetup at the a16z offices in San Francisco, where they officially announced their company and future plans:Including Nous Forge:Show NotesWe've already covered the nuances of Dataset Contamination and the problems with “Open Source” in AI, so we won't rehash those topics here but do read/listen to those if you missed it.* Axolotl GitHub and Discord* The Flan paper and dataset* StackLlama model and blogpost* Multipack paper* Our episode with Tri Dao* Mamba state space models - Tri Dao and Albert GuTimestamps* [00:00:00] Introducing Wing* [00:02:34] SF Open Source AI Meetup* [00:04:09] What is Axolotl?* [00:08:01] What is finetuning?* [00:08:52] Open Source Model Zoo* [00:10:53] Benchmarks and Contamination* [00:14:29] The Case for Open Source AI* [00:17:34] Orca and OpenOrca* [00:23:36] DiscoLM and Model Stacking* [00:25:07] Datasets and Evals over Models* [00:29:15] Distilling from GPT4* [00:33:31] Finetuning - LoRA, QLoRA, ReLoRA, GPTQ* [00:41:55] Axolotl vs HF Transformers* [00:48:00] 20x efficiency with StackLlama and Multipack* [00:54:47] Tri Dao and Mamba* [00:59:08] Roadmap for Axolotl* [01:01:20] The Open Source AI CommunityTranscript[00:00:00] Introducing Wing Lian[00:00:00] [00:00:00] swyx: Welcome to Latent Space, a special edition with Wing Lien, but also with our new guest host, Alex. Hello, hello. Welcome, welcome. Again, needs no introduction. I think it's like your sixth time on Latent Space already. I think so, yeah. And welcome, Wing. We just met, but you've been very prolific online. Thanks for having me.[00:00:30] Yeah. So you are in town. You're not local. You're in town. You're from Minneapolis?[00:00:35] Wing Lian: Annapolis. Annapolis. It's funny because a lot of people think it's Indianapolis. It's I've got Minneapolis, but I used to live out at least in the San Francisco Bay Area years ago from like 2008 to 2014. So it's fairly familiar here.[00:00:50] swyx: Yep. You're the maintainer of Axolotl now, which we'll get into. You're very, very prolific in the open source AI community, and you're also the founder of the Open Access AI Collective. Yeah. Cool. Awesome. Maybe we can go over a little bit of your backgrounds into tech and then coming into AI, and then we'll cover what[00:01:06] Wing Lian: happens and why you're here.[00:01:08] Yeah. So. Back on tech, so I started years ago, I started way back when I was scraping, Apartment websites for listings and then, and then building like SEO optimized pages and then just throwing Google AdSense on it.[00:01:24] And that got me through like college basically. Is[00:01:27] swyx: that decent money? And what year[00:01:28] Wing Lian: was this? Like 2004, 2005. Yeah, that's decent money. It's like thousand bucks a month. But as a college student, that's like. Gravy. Really good money, right? So, and then there's just too much competition It's just sort of like died off. I was writing stuff in like Perl back then using like like who nobody hosted anything on Perl anymore, right? Still did a little bit more like computer tech support and then software, and web more professionally.[00:01:54] So I spent some time working on applications in the blood industry. I came out to San Francisco for, I was at SGN, so Social Gaming Network, as a startup. They started doing, with Facebook apps, and then they pivoted into doing mobile apps. And then, from there, I spent time.[00:02:14] I've quite a few more startups since then and in the last few years I've been in the music space So like I was at United Masters for a while and then past year I've been at SoundCloud, but not doing that anymore and now that I have a lot more time It's just like all right.[00:02:30] We're going full bore on axolotl and we're gonna we're gonna crush AI So yeah,[00:02:34] SF Open Source AI Meetup[00:02:34] swyx: totally you so you're here in town for the open source. Yeah, I meet up that we had yesterday Yep, yeah, that was amazing. Yeah, it was a big collection. Olama, Noose Research, Alignment Lab, Anyone else that I missed? I mean, Jeremy Howard is his own thing.[00:02:47] Yeah.[00:02:49] And Alex, you're also there. You love to bring SF to the world. Your takes?[00:02:55] Alex Volkov: It's incredible that we recorded a Thursday Eye episode after that one. And LDJ, who's usually co hosts Thursday Eye, just like briefly mentioned, Oh yeah, I talked about it.[00:03:04] Like, I saw Karpathy, and then I talked to Jeremy Howard, and the guy from Mistral came in, and it's like, He's talking about all these, titans of industry, basically, that outside of SF, You just don't meet casually hanging out in the same space. You can't, pull somebody. He ran into the Laylow from Mistral, he ran into him while, drinking water.[00:03:20] He didn't even know he was there. It's just, that type of stuff is really hard to find outside of SF. So, absolutely, absolutely great. And also, presentations from Alignment Labs, presentations from News Research, news issues, talked about. Forge, and some of[00:03:33] swyx: the other stuff they announced. We can say now they're officially a company.[00:03:36] I met Technium.[00:03:37] He[00:03:37] Alex Volkov: came over here. He didn't want to get recorded. But maybe.[00:03:41] Wing Lian: We'll wear him down at some point. Yeah, I'm excited for Forge. They've positioned it as this agentic sort of framework where it's just Drag and drop things and, fill in text with where you want to inject different variables and it opens up all of these potentials for data pipelines now, right?[00:03:56] And using your own local LLMs and not relying on GPT 4 or anything like that. Yeah, yeah,[00:04:02] swyx: good stuff. Okay, so let's maybe go into the Axolotl origin story and then we have, we have some intro or background.[00:04:09] What is Axolotl?[00:04:09] swyx: To do on like the open source model universe and also on fine tuning, but maybe just, since you're talking about your personal journey, what was your personal journey into[00:04:18] Wing Lian: axolotl?[00:04:19] Yeah, so my personal journey started like back in mid March, completely unrelated to AI and axolotl. And it really started, I fell while skiing, I torqued. Great 3 MCL sprain and being sort of like an active person that can no longer be active because the two, couldn't play soccer, because that is requires to have having knees until I, it's healed.[00:04:42] So I. I decided I needed to find something to do to take up my free time. And that became, well, let's learn how to train in, these language models. It was everywhere. So I was like, all right, I'm just going to sit down, learn. I think I used like other, I think I was using like Alpacalora.[00:05:00] Cause I think the Alpaca paper had just came out, come out then. So I was like using Alpacalora repo and sort of like learning how to use like. None of us were like GPU rich back then, and none of us, most of us still we're still all GPU poor, but I was doing what was it, like 4 bit, Alpaca Lord, there was like a 4 bit version where we were doing quant, or 8, no, 8 bit quantizations, and then I think they had released QLOR a little bit later, and I think right when, before QLOR came out, I was already starting to do fine tunes, but having this need to sort of like mix data sets together, and If you've ever looked at all the various different datasets available on HuggingFace, they all have various different prompt formats, and, it's sort of a nightmare, and then I think the other piece is if you've ever tried to fine tune, at least Back then probably the ecosystem's a little better now.[00:05:54] Everybody required that you say, alright, you put your hyperparameters as command line arguments. And so it's always like, well, I now have to go copy and paste my previous thing and to change things out. And I really wanted it. to be in a YAML file because it was more portable and reproducible.[00:06:09] So I was doing that and then the QLOR paper came out. Tim Dettmer announced that and then somebody looked it up for me yesterday and it's like between that announcement it took us seven days to get that integrated into Axolotl, right? Which is like, it's not. I wouldn't say it's really fast, but in a manner that, is in a, a reusable framework, I think it was quite the accomplishment then.[00:06:33] And so we started, picking up traction with people there. And then it's just been building models, and then just iterating what my needs are. So, yeah. Excellent. Yeah. I[00:06:44] Alex Volkov: want to ask, for folks who are listening who never heard of Axolotl, now do you describe how you got there?[00:06:49] Can you, how do you summarize this for folks who maybe haven't fine tuned anything. They know about open source LLM exists, they maybe know like LLAML, what's XLR for somebody who doesn't know. I've never heard of a data set curation[00:07:01] Wing Lian: creation before. We sort of have to take a step back and understand that, when you've got these language models, you have what I think most people refer to as like base models, also known as like foundational models, right?[00:07:15] Where some benefactor, whether it's Meta or Mistral or whoever, has gone and spent all this money. To train these models on huge corpuses of text, right? And these, these corpuses, they're generally good across lots of different things, but they're really good at just saying, talking on and on and on, but they're not good at, following instructions or having chats or anything like that.[00:07:40] So, when you think about fine tuning, it's like Saying, all right, we have this really sort of good generalized, text completion thing, and I want to turn it into something that I can talk to or have, follow instructions. So, I think fine tuning is probably best defined in like that.[00:07:58] swyx: Okay, got it.[00:07:59] And we actually[00:08:01] What is finetuning?[00:08:01] swyx: Do want to make sure that we have like an overall introduction to fine tuning for people because again like trying to make sure that we bring everyone along in this, in this journey. We already went into Loras and QLoras without explaining what[00:08:12] Wing Lian: they are. Oh yes, yes, sorry.[00:08:14] swyx: And so I will put things in my words and you can correct me as, as, as my I'll be the village idiot here.[00:08:21] So, so fine tuning is basically sort of grabbing an open source model off the shelf, and then basically doing further training on it with a custom dataset of your own. Primarily, people use it, think about it as fine tuning for JSON output, or fine tuning for a style of response. Let's say you wanted to tell jokes, or be funny, or be short, or whatever.[00:08:43] Just the open source AI community has really fine tuned in all sorts of different manner. I think we'll go over those those things now. Let's go over those things now, and then we'll talk about fine tuning methods.[00:08:52] Open Source Model Zoo[00:08:52] swyx: So there's a universe of people who fine tune stuff. Yesterday in your slides, you had, I'll just list some of these and then we'll maybe go through some of them, right?[00:08:59] So Technium is personally leading Open Hermes, which is I think the sort of premier model out of the news. news community. There's OpenOrca, which you had a hand in. News, the news research itself also has Capybara and Puffin and all the others. There's Pygmalion, which I've never messed with.[00:09:14] Eric Hartford, I am aware of his Uncensored Models and his Samantha Models. Disco Research with Disco LM. And then you personally have done Manticore, Minotaur, Jackalope, and Hippogriff. What should people know about all these names? Being part of AI Twitter is seeing all these things and going dude, I'm being DDoS'ed by all these things and I don't know how different they are.[00:09:32] What should people know? Yeah, so[00:09:34] Wing Lian: I think on a lot of these models, generally, we like to think of those as sort of general models, so If you think about it, what is GPT 4, what is Chad GPT? It's a good general model, and then, One of the services I think that OpenAI offers is like these fine tunings where you're a business and you have very specific business use cases and you might fine tune for that use case.[00:10:00] All of these models are really just general use case that you can then go and maybe Fine tune another lore over it for your use cases, but they tend to be good. With good being relative, it's open source. Open source AI is still sort of is infancy. So, good is, it's pretty reasonable.[00:10:18] It's probably still better than most, high schoolers at answering questions and being able to like figure things out and, and reasoning skills and math and those sorts of things, right?[00:10:27] swyx: And also as measured on the Hugging[00:10:29] Wing Lian: Face leaderboard. Yes, well, that's like a whole other discussion, right, there's a whole other, group of people who, and I, I mostly agree with them that, benchmarks can be, are pretty bogus these days, LM says, I think they published something recently where, even if you think the dataset's not contaminated, you can go and, find contamination And maybe we should step back and say what contamination is, right?[00:10:53] Benchmarks and Contamination[00:10:53] Wing Lian: So we have all of these data, when you go and do these benchmarks, there's a specific data set where there are these questions and usually it's multiple choice. And what can happen is, well, sometimes someone It puts the question, maybe maliciously, maybe accidentally, into the training dataset, and now the, the, your model knows how to answer the test questions really well, but it doesn't, it hasn't generalized the ability to actually do that[00:11:20] Alex Volkov: right.[00:11:21] We've seen some folks competitively announce models that are like the best at that leaderboard, but then it's, it's quite obvious that, In open source? Yeah, and in that leaderboard, for Hugging Face specific, I don't know if LMCs, if that had suffered, but we, there's been some models that seem to have been competitively trained and some leakage happened into their,[00:11:41] swyx: like, supposal.[00:11:43] I understand, once there's been a credible assertion, Hugging Face actually does take them down, right? Yeah, yeah,[00:11:48] Alex Volkov: which is really hard to know, right?[00:11:50] swyx: It's really hard to know, sometimes it's like a pure accident,[00:11:52] Alex Volkov: it's oh, oops. You're going through a mixer. I think, a responsible So acknowledgement, that this kind of happened to you is also important.[00:11:58] I saw LDJ from news research can acknowledge that. Because many of these datasets are collections of other datasets. There's a bunch of people are baking, basically. It's alchemy. Right. And so sometimes you don't know. Sometimes you pull an open source dataset and they announce, oh, you know what, actually, the MMLU benchmark which we used to Specifically identify models that did go into this data set, that then went into that data set.[00:12:22] So sometimes it's actually an accident and folks take it down. But I've seen some competitive folks who want to put their name out there because people are starting to notice which is the top[00:12:30] swyx: model. For those who want a fun take on this so the file one dataset. FindOne model from Microsoft was accused of being contaminated.[00:12:37] And I saw this joke paper that was fantastic. It was called, training on the test set is all you need. It's a super small model that just memorizes everything. It was fantastic. So yeah, contamination, I think we've actually covered it in a previous episode before. So we're good. But again, I want to give people a map into the open source AI model, the universe.[00:12:57] And Alex, you can also jump in here because you guys have spent a lot more time with them than I have. So, what should people know about Technium? What should people know about Noose? And then we can go down the list. Yeah,[00:13:05] Wing Lian: I think so. I think if we start with, Technium. When you talk to him, he's gonna say, I think, I think his response is that he wants to build GP4 on his laptop, right?[00:13:14] So, very, very good at building general models. I think with Noose, Noose Research, they're looking at more, sort of, More, more research focused things, like their Yarn models, I don't, I don't, they didn't actually train their, they have their own trainer for their Yarn models, but So they did not use Xlato for that one?[00:13:30] They didn't use that, but like Is that, you don't have support for it? I think we do support Yarn, I think, I'd have to double check that answer. Yeah, I'm just kind of curious what you can and cannot support, and Yeah, I mean, Yarn is supportable, it's basically, I think it's just replacing, I think, the rope part of that, so Yeah, not, not a big deal.[00:13:48] Yeah, it's not a big deal, it's just I haven't gotten to it, not enough people have asked, I think a lot of people have asked for other things, so it's just, squeaky wheel, right? I think at the end of the day, people are like building these data sets and I think if you sort of map things chronologically, these make more sense because it's like, how do we incrementally improve all of these models?[00:14:07] So a lot of these models are just incremental improvements over the last thing, right? Whether it is sort of through methods of how do we, how did we curate the data set? How did we improve the quality of the data set? So, you maybe LDJ talked about it right on I think for, for Capybara and Puffin, like how those, those were very specific dataset curation techniques that he works on.[00:14:29] The Case for Open Source AI[00:14:29] Alex Volkov: So there's, folks are doing this for dataset curation. Folks are doing this for skillset building as well. Definitely people understand that open source is like very important, especially after the, the, the, the, the march, the debacle, the OpenAI weekend that we all had. And people started noticing that even after developer day in OpenAI, the APIs went out.[00:14:48] And then after that, the whole leadership of the company is swiftly changed and people, there was worries about, you know. How can people continue building AI products based on these like shaky grounds that turned attention definitely to Technium at least in open RMS I started seeing this more and more on Twitter, but also other models and many companies They're gonna start with open AI just to get there quick, and then they they think about okay Maybe I don't want to share my knowledge.[00:15:13] Maybe I don't want to sign up for Microsoft. Maybe they will change their terms and conditions so What else is out there? They turned to other companies. Up until yesterday, Google was nowhere to be found. We've talked about Gemini a little bit before in a previous And you can tune in[00:15:26] swyx: to[00:15:26] Alex Volkov: Thursday Eye.[00:15:26] Yeah, you can tune in to Thursday Eye. We covered the Gemini release a little bit. And but many are turning into the open source community and seeing that Meta released and continues to release and commit to open source AI. Mistral came out and the model is way smaller than LLAMA and performs Significantly better.[00:15:43] People play with OpenRMS, which is currently techniums based, news researched, sourced, axolotl trained OpenRMS, I assume, right? And then they play with this and they see that, okay, this is like GPT 3. 5 quality. We had GPT 4. 5 birthday just a week ago. A week ago, a year ago, a week ago, we never, interacted with these models of this caliber.[00:16:04] And now there's one open source, one that's on my laptop, completely offline, that, I can continue improving for my use cases. So enterprises, companies are also noticing this. And the open source community folks are building the skill set, not only the data sets. They're building the actual kind of, here's how we're going to do this, with Axelotl, with these data sets.[00:16:21] The curation pieces. Now. Interesting. There's like recipes of curation. The actual model training is kind of a competitive thing where people go and compete on these leaderboards that we talked about, the LMC arena, and that recently added open air and recently added open chat and a bunch of other stuff that are super cool.[00:16:37] The hug and face open source leaderboard. And so there's a competitive aspect to this. There's the open source. Aspect to this, like Technium says, I want GPT 4 on my laptop. There's the, let me build a skill set that potentially turns into a company, like we saw with Noose. Noose just, started organizing, a bunch of people on Discord, and suddenly, they're announcing their company.[00:16:54] It's happening across all these modalities, and suddenly all these people who saw these green pastures and a fairly quick way to, hey, here's a cool online community I can, start doing cool stuff with. You mentioned the same in the beginning, right? Like, after your accident, what's cool, let me try this out.[00:17:08] Suddenly I start noticing that there's a significant movement of interest in enterprising companies into these areas. And, this skill set, these data sets, and this community is now very Very important, important enough to create an event which pulls in Andrei Karpathy from OpenAI to come and see what's new Jeremy Howard, like the event that we just talked about, people are flying over and this is just a meetup.[00:17:28] So, definitely, the community is buzzing right now and I think Axelot is a big piece as well.[00:17:34] Orca and OpenOrca[00:17:34] Wing Lian: Cool. Maybe we can talk about like Orca real quick, Orca, OpenOrca rather, I think there was a lot of buzz when, the first Orca paper came out. And just briefly, what is Orca? Yeah, Orca was basically having traces of like chain of thought reasoning, right?[00:17:48] So they go and they, they distill sort of GPT 4. They take, they take a sampling of data from the Flan dataset. Maybe we can like add some show notes in the Flan dataset. Yeah, but we've covered it. Okay, cool. Use GPT 4 to say, all right, explain this in a step by step reasoning, right?[00:18:06] And then you take that and you, they train the model and it showed, very good improvements across a lot of benchmarks. So OpenOrca was sort of the open reproduction of that since Microsoft Research never released that particular data set. And going back to sort of the Hugging Face leaderboard thing, those models did really well.[00:18:23] And then I think, so sort of the follow up to that was SlimOrca, right? I think Going into and building the OpenOrca dataset, we never really went in and, validated the actual answers that GPT 4 gave us, so what we did was one from OpenChat actually cross referenced the original Flan, the original Flan response, the human responses, the correct answers with the dataset, and then I went and took it and sent all of, both of them to GPT 4 and said, is this answer mostly correct, right?[00:18:54] Yeah. And then we were able to filter the dataset from, At least of the GPT 4 only answers from like 800, 000 to like 500, 000 answers or rows and then, and then retrain the model and it had the same performance as the original model to within I think, 0. 1 percent here about, and 30 percent less data.[00:19:13] So, yeah. Okay.[00:19:15] swyx: Interesting. So, I mean, there's, there's so much there that I want to highlight, but yeah. Orca is interesting. I do want people to know about it. Putting chain of thought into the data set like it's just makes a ton of sense one thing I think it would be helpful for people to scope thing these things out is how much data are we talking about when when you When people are fine tuning and then how much time or resources or money does it take to train to fine[00:19:36] Wing Lian: tune?[00:19:37] Yeah, so I think there's a little bit of overlap there with sort of like fine tuning techniques, but let's say Orca and I think even Hermes, they're both relatively large data sets like 10 billion tokens. Yeah. So large data sets being or the original Orca was, or the original open Orca was 800,000 rows.[00:19:55] I believe it was somewhere in the ballpark of like a gigabyte of data, of gigabyte, of text data. And I, I don't. I believe, Hermes was, is like a quarter million rows of data, I don't know the actual byte size on that particular one. So, going and training a, let's, let's say everybody's training 7 billion Mistral right now, right?[00:20:15] So, to tri I, I believe to fine tune 7 billion Mistral on, let's say, 8 A6000s, which have 48 gigabytes of VRAM, I believe, It takes about 40 hours, so 40, and then that's, depending on where you get your compute, 40 times 6, so it's like 500 to fine tune that model, so, and, and that's assuming you get it right the first time, right?[00:20:44] So, you know.[00:20:45] swyx: Is, is that something that X. Lotto handles, like, getting it right the first[00:20:48] Wing Lian: time? If you talk to anybody, it's like you've probably tried at least three or four runs or experiments to like find the right hyperparameters. And after a while you sort of have a feel for like which, where you need your hyperparameters to be.[00:21:04] Usually you might do like a partial training run, do some benchmark. So I guess for Al Farouk, whether you're going by his. This is Jeremy, he's, his actual name, or his twitter handle. He released the Dharma dataset, which is basically a subset of all the benchmarks. And Axolotl actually supports, you know taking that subset and then just running many benchmarks across your model every time you're doing an evaluation so you can sort of like see sort of relative it's not going to be the actual benchmark score, but you can get ideas alright, is this benchmark improving, is this benchmark decreasing, based on, you know Wait,[00:21:39] swyx: why don't you run the full benchmark?[00:21:41] What, what, what The[00:21:42] Wing Lian: full benchmarks take Take a long time. Significant, yeah, significant amount of time. Yeah. And Okay, so that's like[00:21:48] swyx: mini MMLU. Yeah. Like,[00:21:49] Wing Lian: mini BigBench or whatever. Yep, exactly.[00:21:51] Alex Volkov: It's really cool. We, when I joined Web2Masters just recently, and one of the things that I try to do is hey I'm not, I'm a software engineer by trade, I don't have an MLE background, But I joined a company that does primarily MLE, and I wanted to learn from the community, Because a lot of the open source community, they use weights and biases, And the benchmark that you said that Pharrell did, remind me of the name, sorry.[00:22:13] Dharma? Dharma, yeah, yeah. So Luigi showed me how Dharma shows inside the dashboard. In Wi and Biases dashboard and so you can actually kinda see the trending run and then you can see per each kind of iteration or, or epoch or you can see the model improving trending so you can on top of everything else.[00:22:29] The wi and biases gives like hyper parameter tracking, which like you, you started with common line and that's really hard to like remember. Also the Dharma data set, like the quick, the mini orca mini, you mini many different things. It's pretty cool to like visualize them as well. And I, I heard that he's working on a new version of, of Dharma, so Dharma 2, et cetera.[00:22:47] So hopefully, hopefully we'll see that soon, but definitely it's hard, right? You start this training around, it said like 40, 50 hours. Sometimes, sometimes it's like your SSHing into this machine. You, you start a process, you send it with God and you just go about your day, collecting data sets, and then you have to return.[00:23:04] And the whole process of instrumentation of this is still a little bit like squeaky but definitely. Tuning performance, or like grabbing performance in the middle of this, like with Dharma and some other tools, is very helpful to know that you're not wasting precious resources going somewhere you shouldn't go.[00:23:21] Yeah.[00:23:22] swyx: Yeah. Very cool. Maybe I'll, I'll, before we go into like sort of more details on fine tuning stuff, I just wanted to round out the rest of the Excel autoverse. There's, there's still Eric Hartford stuff. I don't know if you want to talk about Pygmalion, Disco, anything that you know about[00:23:35] Wing Lian: those, those things.[00:23:36] DiscoLM and Model Stacking[00:23:36] Wing Lian: Yeah, I think like one of the, definitely one of the more interesting ones was like the Disco 120b, right? Yeah, I know nothing about it. Yeah. So, so. Alpen from Pygmalion AI, right, so they, so Pygmalion is a sort of a, it's, it's, they have their own community, a lot of it is based around, roleplay models, those sorts of things, and Alpen, like, put together, merged together Llama270B, so, and Alpen, like, put together, merged together Llama270B, so, I don't remember how he stacked them together, whether he merged the layers in between. There's a whole, there's a whole toolkit for that by Charles Goddard, where you can like take a single model and like stack them together or multiple models merge.[00:24:18] That's like a whole other talk and a whole other tool set, but was able to create this 120. Billion parameter model out of a LAMA two 70 B. And then I believe the, yeah, disco is a fine tune of, of the, the, the sort of the base one 20 B is, I believe Goliath one 20 B. So, and, and what are the[00:24:37] swyx: headline results that people should know about[00:24:39] Wing Lian: disco?[00:24:39] I think for the headline results, I, I've, I haven't played with it personally because it's. It's a very large model and there's a lot of GPU, right? But, like, from what I've heard anecdotally, it performs really well. The responses are very good. Even with, like, just, even the base model is a lot better than, Llama70b.[00:24:57] So, and we, I think generally everybody's like, we would all love to fine tune Llama70b, but it's just, it's so much, it's so much memory, so much compute, right?[00:25:07] Datasets and Evals over Models[00:25:07] Wing Lian: I[00:25:07] Alex Volkov: want to touch on this point because the interesting thing That comes up out of being in this ecosphere and being friends with open source folks, tracking week to week state of the art performance on different models.[00:25:19] First of all, a lot of the stuff that the folks do a couple of weeks ago, and then something like Mistral comes out, and a lot of the stuff back then, Doesn't technically make sense anymore. Like the artifacts of that work, the actual artifacts, they don't no longer make sense. They're like lower on the on, on the hug and face leaderboard or lower on LM CS leaderboard.[00:25:36] But some of the techniques that people use, definitely the datasets. The datasets keep traveling, right? So open airmen, for example, is the dataset. The tum cleaned up for only. Open sourceable data that previously was just Hermes. And that, it was previously used to train Lama. And then once Mistral came out, it was used to train Mistral.[00:25:54] And then it became significantly better on the 7b base Mistral. So the data sets keep traveling, keep getting better a little bit here and there. And so the techniques improve as well. It looks like both things are simultaneously true. The artifacts of a month and a half ago. The, the actual models themselves, it's great the hug and face has them, because not every company can keep up with the next weeks', oh, I, I'll install this model instead, sell this model instead.[00:26:19] But the, the techniques and the, the dataset keep improving as we go further, and I think that's really cool. However, the outcome of this is that for a long time. For many, many people, including us, that we do this every week. We literally talk with people who release these models every week. It's really hard to know.[00:26:36] So, there's a few aspects of this. One, I think, like you said, the bigger model, the 70B models, you actually have to have somebody like Perplexity, for example, giving you access to the 70B really fast. Or you have to, like, Actually, find some compute, and it's expensive, especially for the bigger models. For example Falcon 180B came out, like the hugest open source model.[00:26:56] How do you evaluate this if you can't run it? Nobody liked it. It's really, so first of all, nobody liked it, but secondly, only the people who were able to find compute enough to run inference on this, they only had like, I can't run this on my laptop, and so that's why it's much easier, something like OpenRMS 7 to be, 7B, it's much easier, because you can run this on your MacBook.[00:27:14] It's much easier to evaluate. It's much easier to figure out the vibes, right? Everybody talks about the vibes as an evaluation check. If you're plugged in enough, if you follow the right people, if they say pretty much the same things all independently, then you run into a problem of whether they're repeating, and their stochastic parents are repeating the same thing, or they actually evaluated themselves.[00:27:31] Yeah, you never know. But, you never know, but like, I think on a large enough scale on Twitter, you start getting the feel. And we all know that like, OpenRMS is one of the top performing models, benchmarks, but also vibes. And I just wanted to highlight this vibes checks thing because you can have the benchmarks, you can have the evaluations, they potentially have contamination in them, potentially they not necessarily tell you the whole story because some models are good on benchmarks, but then you talk to them, they're not super helpful.[00:28:00] And I think it's a combination of the benchmarks, the leaderboards, the chatbot, because LMSys, remember, their ranking is not only based on benchmarks, it's also people playing with their arena stuff. People actually like humans, like, get two answers. I think they completely ignore benchmarks. Yeah, and then They only do ELO.[00:28:18] Oh, they do ELO completely, right? So that, for example, is just like people playing with both models and say, Hey, I prefer this one, I prefer that one. But also there's like some selection bias. The type of people who will go to LMCs to play with the models, they're a little bit specific in terms of like who they are.[00:28:33] It's very interesting. There's so many models. People are doing this in this way, that way. Some people are doing this for academic rigor only to test out new ideas. Some people are actually doing this like the Intel fine tunes of Mistral. Intel wanted to come out and show that their hardware approach is possible, Mistral, etc.[00:28:51] And it's really hard to know, like, what to pick, what to use. And especially on the bigger models, like you said, like the Llama 70B, the Falcon 180B. It's really because, like, who has the compute to validate those? So I would mention that, like, use with caution. Like, go and research and see if the biggest model that just released was actually worth the tokens and the money you spend on it.[00:29:12] To try and, if you're a business, to integrate it.[00:29:15] Distilling from GPT4[00:29:15] swyx: Since you said use of caution, I'll bring in one issue that has always been in the back of my mind whenever I look at the entire universe of open source AI models, which is that 95 percent of the data is derived from GPC 4, correct?[00:29:30] Which technically you can't use for commercial licenses,[00:29:34] Wing Lian: right?[00:29:35] swyx: What is the community's stance on this kind of stuff?[00:29:40] Wing Lian: I think from the community stance, like I feel like a lot of us are just experimenting, so for us, it's like, we're not going and building a product that we're trying to sell, right?[00:29:49] We're just building a product because we think it's interesting and we want to use it in our day to day lives, whether or not we try and integrate it. Personal use, yeah. Yeah, personal use, so like, as long as we're not selling it, yeah, it's fine. But[00:30:01] swyx: like, I as a company cannot just take OpenHermes and start serving[00:30:05] Alex Volkov: it and make money on it.[00:30:06] OpenHermes you can. Because the opening of OpenHermes, I think, is a clean up. That did after the regular Hermes, please folks, check your licenses before you listen to podcasts and say, Hey, I will tell you though, you could say the same thing about OpenAI. You could say the same thing kind of makes sense, where OpenAI or StabilityAI trains their diffusion model on a bunch of pictures on the internet, and then the court kind of doesn't strike down Sarah Silverman, I think, or somebody else, who came and said, hey, this has my work in it, because of the way how it processes, and the model eventually builds this knowledge into the model, and then it doesn't actually reproduce one to one what happened in the dataset.[00:30:45] You could claim the same thing for open source. Like, we're using And by we, I mean the, the open source community that I like happily report on uses GPT 4 to rank, for example, which is the better answer you, you, that's how you build one, one type of data set, right? Or DPO or something like this, you, you basically generate data set of like a question and four answers, for example, and then you go to GPT 4 and say, Hey, smartest model in the world right now, up to Gemini Ultra, that we should mention as well.[00:31:11] Which one of those choices is better? But the choices themselves are not necessarily written with GPT 4. Some of them may be, so there's like full syntactic datasets. But there's also, datasets are just ranked with GPT 4. But they're actually generated with a sillier model, or like the less important model.[00:31:25] The lines are very blurry as to what type of stuff is possible or not possible. And again, when you use this model that's up on Hug Face, the license says you can use this. OpenAI is not going to come after you, the user. If anything, OpenAI will try to say, hey, let's prevent this, this type of thing happening, and the brain, but I honestly don't think that they could know even, not that it makes it okay, it's just like, They also kind of do this with the Internet's archive, and also, I think that some of it is for use.[00:31:55] You use models to help you augment tasks, which is what GPT 4 lets you do.[00:32:00] swyx: Yeah, the worst thing that OpenAI can do is just kick you off OpenAI. That's because it's only enforced in the terms of service.[00:32:05] Alex Volkov: Sure, but just like to make sure, to clarify who they're going to kick out, they could kick out like News, for example, if news are abusing their service, a user of the open source, fully Apache 2 open source, for example, They won't get kicked out if they use both, just because they use both.[00:32:22] I don't believe so. I don't think OpenAI has a claim for that.[00:32:25] swyx: Well, we're not lawyers, but I just want to mention it for people to know it's an issue.[00:32:30] Wing Lian: And one of the things, like, I talked to someone recently, and I think that they also are like interested in it, but also to the point of like, right, if I use a model trained on data, using GPT for data, But I use that model to then regenerate new data.[00:32:46] Is that model, is that data okay? So like you start going down this whole rabbit hole. So yeah. All right.[00:32:53] swyx: Fantastic. Cool. Well, I think that's roughly highlights most of the open source universe. You also have your own models. Do you want to shout out any one of them? Yeah.[00:33:01] Wing Lian: I mean, I think like, I think Early on, Manicore got a lot of love.[00:33:04] I think it was mostly popular in, like, the roleplay communities. It was, it tended to be pretty truthful. It tended to be, like, have relatively good answers, depending on who you ask, right? But, I think for me, it was just, Releasing models was a way to try and, like, continue to build out the product, figure out what I needed to put into the product, how do I make it faster, and, if you've got to, like, go and debug your product, you may as well have it do something useful.[00:33:29] Awesome. So, yeah.[00:33:31] Finetuning - LoRA, QLoRA, ReLoRA, GPTQ[00:33:31] swyx: Okay, and then maybe we'll talk about just fine tuning techniques. So this is going to be a little bit more technical than just talking about model names and datasets. So we started off talking about LoRa, QLoRa. I just learned from your readme there's ReLoRa. Which I've never heard about.[00:33:45] Could you maybe talk about, like, just parameter efficient fine tuning that whole, that[00:33:50] Wing Lian: whole journey, like, what people should know. Yeah, so with parameter efficient fine tuning, I think the popular ones, again, being, let's, we'll start with lore, right? So, usually what you do is you freeze all the layers on your base, on the base model, and then you, at the same time, you sort of introduce additional Oh, this is tight.[00:34:08] No. You introduce, another set of layers over it, and then you train those, and it is done in a way that is mathematically possible, particularly with LORs that you can, then you, you, When you, when you train the model, you, you run your inputs through the base model, whose weights are frozen, but you, then you also run it through the additional weights, and then at the end you combine the weights, and then, and then, or you combine the weights to get your outputs, and then at the end, and when you're done training, you're left with this other set of weights, right, that are completely independent, and And then from that, what you can do is, some person smarter than I figured out, well, oh, they've done it in such a way that now I can merge these weights back into the original model without changing the architecture of the model, right?[00:35:03] So, so, that tends to be, like, the go to, and You're training much fewer parameters so that when you do that, yes, you still need to have all of the original weights, but you have a smaller gradient, you have a smaller optimizer state, and you're just training less weights, so you can tend to train those models on, like, much smaller GPUs.[00:35:27] swyx: Yeah. And it's roughly like, what I've seen, what I've seen out there is roughly like 1 percent the number of parameters that you're trading. Yeah, that sounds about right. Which is that much cheaper. So Axelotl supports full fine tune, LoRa, QLoRa,[00:35:40] Wing Lian: Q. Yes. So, so QLoRa is, is very similar to LoRa. The paper was, if I remember correctly, the paper was Rather, traditionally, most people who did Loras were, were, they were quant, they were putting the model weights in 8 bit, and then fine tune, parameter efficient fine tuning over the Lora weights, and then with QLora, they were quantizing all of those, they were then quantizing the weights down to 4 bit, right, and then I believe they were also training on all of the linear layers in the model.[00:36:15] And then with ReLore, that was an interesting paper, and then, I think, like, it got implemented. Some people in the community tried it, tried it out, and it showed that it didn't really have the impact that the paper indicated that it would. And from what I was told recently, that they re I guess they re released something for Relora, like, a few weeks ago, and that it's possibly better.[00:36:44] I personally haven't had the time. What was the[00:36:46] swyx: main difference,[00:36:47] Wing Lian: apart from quantization? I don't know. Okay. What was the main difference, sorry?[00:36:49] swyx: Apart from quantization, right? Like,[00:36:50] Wing Lian: Qlora's thing was, like, we'll just drop off some bits. With Relora, what they did was, you would go through, you would define some number of steps that you would train, like, your Lora with, or your Qlora.[00:37:01] Like, you could do Like, ReqLore, if you really wanted to, you would, you would train your LoRa for some number of steps, And then you would merge those weights into your base model, and then you would start over. So by starting, so, then by starting over, The optimizer has to find, like, sort of, re optimize again, and find what's the best direction to move in, and then do it all again, and then merge it in, do it all again, and theoretically, according to the paper, doing ReLore, you can do parameter efficient fine tuning, but still have sort of, like, the performance gains of doing a full fine tuning, so.[00:37:38] swyx: Yeah, and[00:37:39] Wing Lian: GPTQ? And GPTQ, so it's, I think with GPTQ, it's very similar to, more similar to QLore, where you're, it's mostly a quantization of the weights down to like 4 bit, where GPTQ is a very, is a specific methodology or implementation of quantization, so. Got it.[00:37:57] Alex Volkov: Wang, for, for folks who use Axolotl, your users, some people who maybe, Want to try it out?[00:38:03] And do they need to know the differences? Do they need to know the implementation details of QLora versus ReLora? Or is it okay for them to just know that Axolotl is the place that already integrated them? And if that's true, if that's all they need to know, how do they choose which method to use? Yeah,[00:38:22] Wing Lian: so I think like, I think most people aren't going to be using ReLora.[00:38:25] I think most people are going to be using either Lora or QLora. And I think they should have it. They should have an understanding of why they might want to use one over the other. Most people will say that with Qlora, the quality of the final model is not quite as good as like if you were to do a LoRa or a full fine tune, right?[00:38:44] Just because, you've quantized these down, so your accuracy is probably a little off, and so that by the time you've done the Qlora, you're not moving the weights how you would on a full fine tune with the full parameter weights.[00:38:56] Interesting.[00:38:57] swyx: Okay, cool. For people who are more interested, obviously, read the papers. I just wanted to give people, like, a high level overview of what these things are. And you've done people a service by making it easy for people to try it out. I'm going to, I'm going to also ask a question which I know to be wrong, but I'm curious because I get asked this all the time.[00:39:15] What is the difference between all these kinds of fine tunes[00:39:17] Wing Lian: and RLHF? Okay, between all of these sorts of fine tunes and RLHF. So all of these sorts of fine tunes are based, are, ideally, this, they are taking knowledge that the base model already knows about, and presenting it in a way to the model that you're having the model answer like, Use what it already knows to sort of answer in a particular way, whether it's, you're extracting general knowledge, a particular task, right?[00:39:44] Instruct, tune, chat, those sorts of things. And then generally with RLHF, so what is, let's go back, what is it? Reinforcement Learning with Human Feedback. So if we start with the human feedback part, What you're doing is you generally have, you have like a given prompt and then you, maybe you have one, maybe you have two, I think, like if you look at with Starling, you have like up to what, seven different, seven different possible responses, and you're sort of ranking those responses on, on some sort of metric, right, whether the metric is how much I, I might like that answer versus or I think with like starling is like how how how helpful was the answer how accurate was the answer how toxic was the answer those sorts of things on some sort of scale right and then using that to go back and like sort of Take a model and nudge it in the direction of giving that feedback, to be able to answer questions based on those preferences.[00:40:42] swyx: Yeah, so you can apply, and is it commutative? Can you apply fine tuning after and onto an RLHF model? Or should the RLHF apply, come in afterwards,[00:40:54] Wing Lian: after the fine tune? Um, I, yeah, I don't know that there's There's been enough research for one way or another, like, I don't know.[00:41:02] That's a question that's been asked on Discord. Yeah, like, I definitely would say I don't know the answer. Go and try it and report back to me and let me know so I can answer for the next guy.[00:41:10] swyx: It's shocking how much is still unknown about all these things. Well, I mean, that's what research is for, right?[00:41:16] Wing Lian: So actually I, I think I saw on the top of a leaderboard, it was a, it was a mytral base model, and they didn't actually fine tune it. They, or they, they just did RLH, they did like an RLHF fine tune on it using like, I don't, I don't recall which dataset, but it was like, and it benchmarked really well.[00:41:37] But yeah, you'd have to go and look at it. But, so it is interesting, like going back to that, it's like. Traditionally, most people will fine tune the model and then do like a DPO, PPO, some sort of reinforcement learning over that, but that particular model was, it seemed like they skipped like the supervised fine tuning or Scott.[00:41:55] Axolotl vs HF Transformers[00:41:55] swyx: Cool. One thing I did also want to comment about is the overall, like, landscape, competitive landscape, I don't know. Hugging Face Transformers, I think, has a PFT module.[00:42:05] Wing Lian: Yeah, yeah, the PEFT, the Parameter Efficient Fine Tuning, yep. Is that a competitor to you? No, no, so we actually use it. We're just a wrapper over sort of, sort of the HuggingFace stuff.[00:42:15] So, so that is their own sort of module where They have, taken the responsibility or yeah, the responsibility of like where you're doing these parameter efficient fine tuning methods and just sort of like, it is in that particular package where transformers is mostly responsible for sort of like the modeling code and, and the trainer, right.[00:42:35] And then sort of, there's an integration between the two and, there's like a variety of other fine tuning packages, I think like TRL, TRLX, that's the stability AI one. Yeah, I think TRL likes the stability, yeah, Carper, and TRL is a hugging face trainer. Even that one's just another wrapper over, over the transformers library and the path library, right?[00:43:00] But what we do is we have taken sort of those, yes, we've We also use that, but we also have more validation, right? So, there are some of us who have done enough fine tunes where like, Oh, this and this just don't go together, right? But most people don't know that, so like Example?[00:43:19] Like, people want to One and one doesn't go together. I don't have an example offhand, but if you turn this knob and this knob, right? You would think, all right, maybe this will work, but you don't know until you try. And then by the time you find out it doesn't work, it's like maybe five minutes later, it's failed.[00:43:34] It's failed in the middle of training or it's failed during the evaluation step. And you're like, ah, so we've, we've added a lot of, we've added a lot more validation in it. So that like, when you've, you've created your configuration, you run it through and now you say. The validation code says this is probably not right or probably not what you don't, not what you want.[00:43:52] So are you like a, you[00:43:53] swyx: do some linting of your YAML file?[00:43:56] Wing Lian: There, I guess you could call it linting, it's sort of like Is there a set of rules out[00:44:00] swyx: there somewhere? Yeah, there's a set of rules in there. That's amazing, you should write documentation like This rule is because, this user at this time, like, ran into this bug and that's what we invested in.[00:44:10] It's like a good collection[00:44:11] Wing Lian: of knowledge. Yeah, it is, and I guess like, if you really wanted to, like, figure it out, I guess you could, like, git blame everything, and But, yeah, it's, so, I think that's always a useful thing, it's like Because people want to experiment but they don't, people will get frustrated when you've experiment, you're experimenting and it breaks and you don't know why or you know why and you've just gone down the rabbit hole, right?[00:44:37] So, so I think that's one of the big features that's, that I think I find important because it's It prevents you from doing things you probably shouldn't have, and it, and sometimes we will let you do those things, but we'll try and warn, warn you that you've done that.[00:44:50] I[00:44:51] Alex Volkov: have a follow up question on this, actually, because yesterday we hung out to this open source event, and I spent time by you a couple times, like when people told you, oh, XLR, I use XLR, it's super cool, and then the first thing you asked is, like, immediately, like, what can we improve?[00:45:04] And yes, from multiple folks, and I think we talked about this a little bit, where there's It's a developer tool. It's like a machine learning slash developer tool. Your purpose in this is to help and keep people, as much as possible, like, Hey, here's the best set of things that you can use right now. The bear libraries are, or the bear trainer, for example, is a bear trainer.[00:45:28] And also, maybe we should talk about how fast you're implementing these things. So you mentioned the first implementation took a week or so. Now there's a core maintainer group, right? There's like, features are landing, like Qlora, for example. Neftune, I don't know if that's one example of something that people potentially said that it's going to be cool, and then eventually, like, one of those things that didn't really shake out, like, people quickly tested this out.[00:45:48] So, there's a ton of Wait, Neftune is cancelled? I don't know if it's fully canceled, but based on vibes, I heard that it's not that great. So like, but the whole point that I'm trying to make with Neftune as well is that being existing in the community of like XLR or like, I don't know, even following the, the GitHub options or following the Discord, it's a fairly good way to like, learn these, Kind of gut feelings that you just, you just said, right?[00:46:14] Like where this, maybe this knob, that knob doesn't work. Some of these are not written down. Some of these are like tribal knowledge that passes from place to place. Axel is like a great collection of many of them. And so, do you get That back also from community of folks who just use, like, how do you know who uses this?[00:46:30] I think that's still an issue, like, knowing if they trained with XLR or should they add this to things? Talk about, how do you get feedback and how else you should get feedback?[00:46:38] Wing Lian: Yeah, I mean, most of the feedback comes from the Discord, so people come in and , they don't get a training running, they run into, like, obscure errors or, errors that That's a lot of things that maybe, maybe as a product we could catch, but like, there's a lot of things that at some point we need to go and do and it's just on the list somewhere.[00:46:58] Right that's why when people come up, I'm like, what, what were your pain points? Because like, as a developer tool, if you're not happy with it, or you come in and in the first, Takes you 30 minutes and you're still not happy. You leave the tool and you may, you might move on maybe to a better tool, maybe to, one with less frustration, but it may not be as good, right?[00:47:17] So I'm trying to like, figure out, all right, how can I reduce all this frustration? Because like for me, I use it every day for the most part, right? And so I am blind to that, right? Mm-Hmm. . Mm-Hmm. . I just know, I, I go do this, this, and this. It pretty much mostly works, right? But, so I don't have sort of that, alright, that learning curve that other people are seeing and don't understand their pain points.[00:47:40] Yeah,[00:47:40] Alex Volkov: you don't have the The ability to onboard yourself as a new user completely new to the whole paradigm to like get into the doors of like, Oh, no, I don't even know how to like ask about this problem or error.[00:47:53] swyx: Cool. The last few things I wanted to cover was also just the more advanced stuff that you covered yesterday.[00:48:00] 20x efficiency with StackLlama and Multipack[00:48:00] swyx: So I'll just, caution this as like, yeah, this is more advanced. But you mentioned Stackllama and Multipack. What are they[00:48:06] Wing Lian: and what should people know? Yeah, so, so, Stack Llama was, that paper came out, so Stack Llama I think was like, two, two, two separate, two separate concepts that they announced, so the first one was They being hugging face.[00:48:20] Yeah, sorry, yes, they being hugging face, so the first one being sort of like, this idea of packing, like some packing sequences together, so like, if we think about training data, right, your training data is, let's say, to keep the math easy, let's say your training data is 500, We, we, we, we will use the terminology words.[00:48:39] Let's say your training data is 500 words long, and let's say your, your context length, you know how much data your, that your model can accept is like, or that you want feed into your model. It's, let's say, we won't use tokens again, we'll we'll use it is it's 4,000 tokens, right? So if you're training at 4K Con or four 4,000 4K contacts and you're only using 500 of it, you're sitting like with the other 1500.[00:49:05] 3, 500 words that you're not using, right? And typically that's either filled with these PAD tokens, so I think I made the analogy last night that it's like having sort of like a glass here you fill it up with a shot of liquor and then you're and that's your training data and then you just fill it up with more water and those are your PAD tokens and it's just, it doesn't do much, right?[00:49:27] It's still the same thing, but you still have to go through all of that to go through all your training data. And then, so what Stack Llama showed was you could just sort of take your training data, append the next row of training data until you filled that entire 4k context, so in this example, right, with 500 words to 4k, that's 8 rows of training data.[00:49:48] But, the problem with that is, is that with a lot of these transformer models, they're very much relying on attention, right? So, like, if you now have this sequence of words that now, in order for the, the model has seen all of these other words before, right? And then it sees another set of words, another set of words, but it's learning everything in context of all the words that it's seen before.[00:50:13] We haven't corrected the attention for that. And just real quickly, since I said that that paper was two concepts, the other one was, I believe it was like a reinforcement learning, but outside the scope of this. So going from that, I implemented that early on because I was like, Oh, wow, this is really great.[00:50:29] And. Yes, because it saves you a bunch of time, but the trade off is a little bit of accuracy, ultimately, but it still did pretty well. I think when I did Manicore, I think it used sort of that concept from Stack Llama of just sort of appending these sequences together, right? And then sort of the next evolution of that is Multipack, right?[00:50:51] So, there was a separate paper on that, it was, I believe it was referenced, it got referenced in the Orca paper, where you could, you could properly mask those out using like a, I think it was like a lower block triangular attention mask, and then sort of, so, So, there's that. I did try implementing that, manually recreating that mask, but then one from the OpenChat, so he was helping with OpenOrca as well, and he had done an implementation of Multipack, and where he used FlashAttention, so FlashAttention So that was released by TreeDAO, and it was this huge performance gain.[00:51:35] Everybody uses it now, even the Transformers library now, they've taken all of these, like, people are taking all of these models and sort of like, making it compatible with FlashAttention. But in Flash Tension, there is one particular implementation that lets you say, Well, I'm sending you all of these sequences like you would in Stack Llama, But let me send you another, another, Set of information about, this is where this set of sequences is, this is where the second set of sequences is.[00:52:06] So like, if it was like, 500 words long, and you stacked them all together, you would just send it a row of information that was like, 0, 500, 1000, 1500, etc, etc, out to 4000. And it would know, alright, I need to break this up, and then run the forward pass with it. And then it would be able to, and it was much more, much more performant.[00:52:29] And I think you end up seeing like 10x, 20x improvements over sort of, I mean, I think FlashAttention was like a 2x improvement, and then adding that with the Multipack, you start to see like, depending on, how much data you have, up to like a 20x improvement sometimes. 20x. 20x. Wow. Yeah.[00:52:48] And I only know the 20x because I, like, before last night, I was like, I re ran the alpaca, I looked up the alpaca paper because it was like, I just need a frame of reference where somebody did it, and I think they used eight A100s for three hours, and they said it cost them 100. I don't, I don't think eight A100s cost, I don't know how much it costs right now.[00:53:14] But I ended up rerunning it. Usually a dollar an hour, right? Yeah, so eight. The cheapest is like a[00:53:18] Alex Volkov: dollar, a dollar an hour for one.[00:53:20] Wing Lian: Yeah, so that's still like 24, 25. But maybe if you're going on Azure, maybe it's like, maybe it's 100 on Azure. I mean, it used to be more expensive, like, a year ago.[00:53:31] Yeah, and then, so I re ran it with sort of like, I turned on all of the optimizations just to see what it would be. And like, and usually Multipack is the biggest optimization, so Multipack with Flash Detention. And it, I think I spun it up on 8 L40s, and it ran, and I didn't let it run all the way through, I just grabbed the time, the estimated completion time, and it was like 30 minutes, so it would have cost like 4 or 5 to run the entire, like, reproduce the alpaca paper, right?[00:54:00] Which is crazy. It's crazy. 20x,[00:54:02] Alex Volkov: yeah. I want to ask about, like, you said you turned on all the optimization. Is that the yaml file with xlodl, you just go and like check off, like, I want this, I want that? Yeah, yeah,[00:54:10] Wing Lian: so there's like one particular yaml file in there, That, there's one particular YAML file in there that's like, it's under examples, llama2, fft, optimize.[00:54:20] So, I think someone had created one where they just turned, they put in all of the optimizations and turned them on. I mean, it actually, it does run, which is like, sort of surprising sometimes, because sometimes, you optimize this, optimize this, and sometimes they just don't work together, but, yeah.[00:54:36] Just turn the knobs on, and like, fine tuning should really just be that easy, right? I just want to flip the knob and move on with my life and not figure out how to implement it.[00:54:47] Tri Dao and Mamba[00:54:47] Alex Volkov: Specifically, the guy behind FlashAttention came up with something new. You want to talk about this a little bit? You want to briefly cover Mamba?[00:54:53] Yeah, let's talk about Mamba. Let's talk about Mamba. So, what is Mamba?[00:54:57] Wing Lian: Oh, gosh. I

god google ai internet pr giving personal talk news san francisco reach microsoft putting open uber soundcloud discord busy seo stanford minneapolis billion indianapolis privacy disco releasing fantastic drag goliath models intel roadmap folks excel transformers significant apartments openai sf gemini wing tuning san francisco bay area open source 4k gpt lama traditionally forge github takes dharma llama apis macbook lors hermes azure gravy apache lotto nano biases amin llm mamba pharrell gpu aspect elo annapolis orca ddos alpen prs contamination yarn instruct sarah silverman hugging starling benchmarks gpus 7b distilling minotaur pad perplexity fine tuning mcl alpaca modify lm noose json pygmalion sota hyena trl mistral puffins a16z xlr microsoft research flan ppo axolotl manticore reinforcement learning dpo jackalope google adsense pft stability ai datasets capybara lmc gpc lay low yaml open source ai mle jeremy howard 70b loras carper neurips sgn huggingface hippogriff united masters rlhf vram gemini ultra technium vicuna latent space human feedback which i've relora

#142 - Humanoid Robots, Video-To-Text, Habitat 3.0, Cruise troubles, data poisoning

Play Episode Listen Later Nov 16, 2023 99:16

Our 142nd episode with a summary and discussion of last week's big AI new. Apologies for this one coming out after a pause, episodes will resume being released regularly as of this week. Read out our text newsletter and comment on the podcast at https://lastweekin.ai/ Email us your questions and feedback at contact@lastweekin.ai Timestamps + Links: (00:00) Intro / Banter Tools & Apps(03:00) Introducing PlayHT 2.0 Turbo ⚡️ - The Fastest Generative AI Text-to-Speech API (07:15) YouTube Music now lets you make your own playlist art with AI (09:23) Sick of meetings? Microsoft's new AI assistant will go in your place (11:54) Anthropic brings Claude AI to more countries, but still no Canada (for now) Applications & Business(14:55) Humanoid robots face a major test with Amazon's Digit pilots (18:40) Figure 01 humanoid takes first public steps (22:31) AI-generating music app Riffusion turns viral success into $4M in funding (23:35) ChatGPT Creator Partners With Abu Dhabi's G42 in Middle East AI Push (25:00) AMD Scores Two Big Wins: Oracle Opts for MI300X, IBM Asks for FPGAs (26:38) Alibaba, Tencent among investors in China's rival to OpenAI with $341 million funding (30:35) AI companies drive demand for office space in tech hubs, new study finds (32:13) OpenAI is in talks to sell shares at an $86 billion valuation Projects & Open Source(35:00) Introducing Video-To-Text and Pegasus-1 (80B) (39:35) Adept Releases Fuyu-8B for Multimodal AI Agents (42:03) MiniGPT-v2: large language model as a unified interface for vision-language multi-task learning (44:53) Meta's Habitat 3.0 simulates real-world environments for intelligent AI robot training (48:22) DeepMind UniSim simulates reality to train robots, game characters (49:13) Jina AI Launches World's First Open-Source 8K Text Embedding, Rivaling OpenAI (51:13) Llemma: An Open Language Model For Mathematics Research & Advancements(53:22) Eliciting Human Preferences with Language Models (57:23) New Nvidia AI agent, powered by GPT-4, can train robots (01:01:38) Unveiling the General Intelligence Factor in Language Models: A Psychometric Approach (01:04:48) AgentTuning: Enabling Generalized Agent Abilities for LLMs (01:09:51) Contrastive Prefence Learning: Learning from Human Feedback without RL (01:11:25) ‘Mind-blowing' IBM chip speeds up AI Policy & Safety(01:14:57) GM Cruise unit suspends all driverless operations after California ban (01:18:52) AI researchers uncover ethical, legal risks to using popular data sets (01:22:22) AI Safety Summit: day 1 and 2 programme (01:25:23) Anthropic's AI chatbot Claude is posting lyrics to popular songs, lawsuit claims (01:26:38) Mike Huckabee says Microsoft and Meta stole his books to train AI (01:27:10) Clearview AI Successfully Appeals $9 Million Fine in the U.K. (01:28:11) North Korea experiments with AI in cyber warfare: US official (01:30:17) OpenAI forms new team to assess ‘catastrophic risks' of AI UK poised to establish global advisory group on AI Synthetic Media & Art(01:32:22) This new data poisoning tool lets artists fight back against generative AI (01:34:32) Amazon now lets advertisers use generative AI to pretty up their product shots (01:36:36) The Beatles: ‘final' song Now and Then to be released thanks to AI technology

Robot Learning with Eureka

Play Episode Listen Later Oct 24, 2023 16:18

Developments in robot learning, where NVIDIA researchers have created an AI agent called Eureka that can generate algorithms to train robots. They also explore the concerning use of AI in cyber warfare by North Korea, and the potential consequences for global enterprises. Additionally, they touch on Apple's rumored plans to implement generative AI features on iPhones and iPads. The team also delves into thought-provoking discussions on AI-generated music and the impact of AI on job automation. Finally, they analyze three research papers that shed light on the limitations of large language models in reasoning and planning tasks, and introduce a new approach called Contrastive Preference Learning for optimizing behavior from human feedback. Contact: sergi@earkind.com Timestamps: 00:34 Introduction 02:55 Eureka! NVIDIA Research Breakthrough Puts New Spin on Robot Learning 04:15 North Korea experiments with AI in cyber warfare: US official 05:37 Apple Rumored to Follow ChatGPT With Generative AI Features on iPhone as Soon as iOS 18 07:06 Fake sponsor 09:48 GPT-4 Doesn't Know It's Wrong: An Analysis of Iterative Prompting for Reasoning Problems 10:52 Can Large Language Models Really Improve by Self-critiquing Their Own Plans? 12:27 Contrastive Prefence Learning: Learning from Human Feedback without RL 14:21 Outro

learning ai apple iphone fake robots ios ipads north korea nvidia developments gpt generative eureka cyberwarfare human feedback

Geoffrey Hinton on AI Risk ⚠️ // Length Correlations in RLHF

ai disney risk fake robots adobe long way large language models geoffrey hinton correlations reinforcement learning disney imagineering rlhf human feedback

Play Episode Listen Later Oct 11, 2023 14:10

From Geoffrey Hinton's concerns about the potential risks of advanced AI to Adobe's new symbol for tagging AI-generated content, this episode provides insights into the latest developments in the field. The discussion on Disney Imagineering's latest robot that can walk independently and mimic emotions through body language is also fascinating. Additionally, the episode features three AI research papers on Reinforcement Learning from Human Feedback, neural network predictions, and complex reasoning with Large Language Models, which provide valuable insights into the capabilities and limitations of AI. Contact: sergi@earkind.com Timestamps: 00:34 Introduction 01:38 Geoffrey Hinton on the promise, risks of advanced AI 03:01 Adobe created a symbol to encourage tagging AI-generated content 04:27 Disney Imagineering's latest robot looks right out of Star Wars 06:07 Fake sponsor 08:11 A Long Way to Go: Investigating Length Correlations in RLHF 09:44 Deep Neural Networks Tend To Extrapolate Predictably 11:07 Thought Propagation: An Analogical Approach to Complex Reasoning with Large Language Models 13:00 Outro

Building the Foundation Model Ops Platform — with Raza Habib of Humanloop

Latent Space: The AI Engineer Podcast â€” CodeGen, Agents, Computer Vision, Data Science, AI UX and all things Software 3.0

Play Episode Listen Later Sep 29, 2023 81:17

Want to help define the AI Engineer stack? >500 folks have weighed in on the top tools, communities and builders for the first State of AI Engineering survey! Please fill it out (and help us reach 1000!)The AI Engineer Summit schedule is now live! We are running two Summits and judging two Hackathons this Oct. As usual, see our Discord and community page for all events.A rite of passage for every AI Engineer is shipping a quick and easy demo, and then having to cobble together a bunch of solutions for prompt sharing and versioning, running prompt evals and monitoring, storing data and finetuning as their AI apps go from playground to production. This happens to be Humanloop's exact pitch.full show notes: https://latent.space/p/humanloopTimestamps* [00:01:21] Introducing Raza* [00:10:52] Humanloop Origins* [00:19:25] What is HumanLoop?* [00:20:57] Who is the Buyer of PromptOps?* [00:22:21] HumanLoop Features* [00:22:49] The Three Stages of Prompt Evals* [00:24:34] The Three Types of Human Feedback* [00:27:21] UI vs BI for AI* [00:28:26] LangSmith vs HumanLoop comparisons* [00:31:46] The TAM of PromptOps* [00:32:58] How to Be Early* [00:34:41] 6 Orders of Magnitude* [00:36:09] Becoming an Enterprise Ready AI Infra Startup* [00:40:41] Killer Usecases of AI* [00:43:56] HumanLoop's new Free Tier and Pricing* [00:45:20] Addressing Graduation Risk* [00:48:11] On Company Building* [00:49:58] On Opinionatedness* [00:51:09] HumanLoop Hiring* [00:52:42] How HumanLoop thinks about PMF* [00:55:16] Market: LMOps vs MLOps* [00:57:01] Impact of Multimodal Models* [00:57:58] Prompt Engineering vs AI Engineering* [01:00:11] LLM Cascades and Probabilistic AI Languages* [01:02:02] Prompt Injection and Prompt Security* [01:03:24] Finetuning vs HumanLoop* [01:04:43] Open Standards in LLM Tooling* [01:06:05] Did GPT4 Get Dumber?* [01:07:29] Europe's AI Scene* [01:09:31] Just move to SF (in The Arena)* [01:12:23] Lightning Round - Acceleration* [01:13:48] Continual Learning* [01:15:02] DeepMind Gato Explanation* [01:17:40] Motivations from Academia to Startup* [01:19:52] Lightning Round - The Takeaway Get full access to Latent Space at www.latent.space/subscribe

#136 - Claude Pro, Ideogram, Chinese ChatGPT bots, Falcon 180B, RLAIF, export restrictions, Ghostwriter

The AI Breakdown: Daily Artificial Intelligence News and Discussions

Play Episode Listen Later Sep 10, 2023 62:44

Our 136th episode with a summary and discussion of last week's big AI news! With guest host Daniel Bashir. Check out his AI interview podcast! Read out our text newsletter and comment on the podcast at https://lastweekin.ai/ Email us your questions and feedback at contact@lastweekin.ai Check out our sponsor, the SuperDataScience podcast. You can listen to SDS across all major podcasting platforms (e.g., Spotify, Apple Podcasts, Google Podcasts) plus there's a video version on YouTube. Timestamps + links: (00:00) Intro (01:15) SuperDataScience Ad (01:51) Response to listeners Tools & Apps(02:47) Anthropic's Claude AI chatbot gets a paid plan for heavy users (04:36) Watch out, Midjourney! Ideogram launches AI image generator with impressive typography (06:37) Intuit launches generative AI–powered digital assistant for small businesses and consumers (07:17) Zoom Is Jumping on the AI Chatbot Bandwagon Applications & Business(08:47) China lets Baidu, others launch ChatGPT-like bots to public, tech shares jump (11:23) Tencent releases AI model for businesses as competition in China heats up (12:00) Microsoft says it will take the heat if Copilot AI users get sued (14:52) How We Chose the TIME100 Most Influential People in AI (17:30) ChatGPT creator OpenAI is reportedly earning $80M a month (19:16) AI chip startup d-Matrix raises $110 mln with backing from Microsoft (21:00) ThetaRay nabs $57M for AI tools to ID and fight money laundering (22:18) Sapeon raises $46m for AI chips (23:20) Imbue raises $200M to build AI models that can ‘robustly reason' Projects & Open Source(25:48) Announcing the commercial relicensing and expansion of DINOv2, plus the introduction of FACET (27:02) UAE launches Arabic large language model in Gulf push into generative AI (29:23) New Open-Source ‘Falcon' AI Language Model Overtakes Meta and Google Research & Advancements(31:01) Qwen-VL: A Frontier Large Vision-Language Model with Versatile Abilities (33:57) RLAIF: Scaling Reinforcement Learning from Human Feedback with AI Feedback (36:22) Perception, performance, and detectability of conversational artificial intelligence across 32 university courses (39:28) SyncDreamer: Generating Multiview-consistent Images from a Single-view Image Policy & Safety(41:00) US curbs AI chip exports from Nvidia and AMD to some Middle East countries (42:55) China suspected of using AI on social media to sway US voters, Microsoft says (46:37) Trusting A.I.-written mushroom hunting guides sold on Amazon could get you killed. But like deadly fungi, identifying them is tricky (48:25) Ads for AI sex workers are flooding Instagram and TikTok (50:30) The UK releases key ambitions for global AI summit Synthetic Media & Art(51:55) AI Took the Stage at the World's Largest Arts Festival. Here's What Happened (54:12) Ghostwriter Returns With an A.I. Travis Scott Song, and Industry Allies (56:17) Artists sign open letter saying generative AI is good, actually (58:57) The latest canvas for Refik Anadol's AI-generated art? The new Sphere in Las Vegas (01:01:20) Outro

Study: Reinforcement Learning from AI Feedback Performs As Well As Human Feedback

Play Episode Listen Later Sep 5, 2023 15:58

Today on The AI Breakdown, NLW looks at new research from Google that shows that reinforcement learning using artificial intelligence rather than human feedback could perform as well as RLHF. Before that on the Brief: the first AI pop singer gets a record deal; an AI-produced covid drug moves to phase 1 trials, and more. Today's Sponsor: Supermanage - AI for 1-on-1's - https://supermanage.ai/breakdown ABOUT THE AI BREAKDOWN The AI Breakdown helps you understand the most important news and discussions in AI. Subscribe to The AI Breakdown newsletter: https://theaibreakdown.beehiiv.com/subscribe Subscribe to The AI Breakdown on YouTube: https://www.youtube.com/@TheAIBreakdown Join the community: bit.ly/aibreakdown Learn more: http://breakdown.network/

google ai study performs reinforcement learning nlw rlhf human feedback

Google Duet AI's

Play Episode Listen Later Sep 4, 2023 14:44

Google's Duet AI introducing new features for productivity software, growing public concern about the role of AI in daily life, and Andrej Karpathy's tweet about an optimization technique for inference-time with LLMs. Additionally, the episode features three fascinating papers: RLAIF, CityDreamer, and FACET, which cover topics such as reinforcement learning, 3D city generation, and fairness in computer vision evaluation. Contact: sergi@earkind.com Timestamps: 00:34 Introduction 01:33 Google Duet AI: New Features for Gmail, Docs and Sheets, at $30 a Month 03:20 Growing public concern about the role of artificial intelligence in daily life 04:58 Andrej Karpathi tweets 06:34 Fake sponsor 08:20 RLAIF: Scaling Reinforcement Learning from Human Feedback with AI Feedback 10:05 CityDreamer: Compositional Generative Model of Unbounded 3D Cities 11:40 FACET: Fairness in Computer Vision Evaluation Benchmark 13:34 Outro

google ai fake 3d concerns gmail sheets docs duet facet andrej karpathy duet ai human feedback

#133 - ChatGPT multi-document chat, CoreWeave raises $2.3B, AudioCraft, ToolLLM, Autonomous Warfare

Play Episode Listen Later Aug 18, 2023 88:19

Our 133rd episode with a summary and discussion of last week's big AI news! Apologies for pod being a bit late this week! Read out our text newsletter and comment on the podcast at https://lastweekin.ai/ Email us your questions and feedback at contact@lastweekin.ai Timestamps + links: (00:00) Intro / Banter Response to listener comments / corrections Tools & Apps(05:05) ChatGPT gets several new features, including multi-document chat (09:02) Tinder tests AI photo selection feature to help users build profiles (12:35) GitHub Copilot can now tell developers when its suggestions match code in a public repository Applications & Business(15:55) CoreWeave raises $2.3 billion in debt collateralized by Nvidia chips (20:25) Nvidia GPU shortage is ‘top gossip' of Silicon Valley (23:30) NVIDIA: GPU Supply Issues Involve Packaging, Not Chip Wafers (27:06) Nvidia's AI GPUs Are Selling for up to $70,000 in China (30:00) AMD considers making a specific A.I. chip for China to comply with export controls (31:10) AI chip firm Tenstorrent raises $100 mln from Hyundai, Samsung (33:22) Cruise begins testing self-driving vehicles in Atlanta (35:45) Toyota, Pony.ai plan to mass produce robotaxis in China Projects & Open Source(38:08) Alibaba rolls out open-sourced AI model to take on Meta's Llama 2 Research & Advancements(42:00) Introducing AudioCraft: A Generative AI Tool For Audio and Music (46:25) ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs (52:03) Tool Documentation Enables Zero-Shot Tool-Usage with Large Language Models (54:54) Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback (01:00:05) Towards Generalist Biomedical AI (01:03:53) Studying Large Language Model Generalization with Influence Functions Policy & Safety (01:05:55) The AI-Powered, Totally Autonomous Future of War Is Here (01:12:37) ‘So important': UK minister endorses Google's training drive in AI arms race (01:15:10) Generative AI services pulled from Apple App Store in China ahead of new regulations (01:16:40) Experience: scammers used AI to fake my daughter's kidnap (01:19:22) The Stanford University ‘boot camp' teaching Congress about AI (01:22:23) Eight Months Pregnant and Arrested After False Facial Recognition Match Synthetic Media & Art (01:24:37) Greg Rutkowski Was Removed From Stable Diffusion, But AI Artists Brought Him Back (01:25:48) An Asian MIT student asked AI to turn an image of her into a professional headshot. It made her white, with lighter skin and blue eyes. (01:26:47) Outro

EP 35. ICML现场对话AI研究员符尧：亲历AI诸神之战，解读LLM前沿研究，Llama 2，AI Agents

OnBoard!

Play Episode Listen Later Aug 7, 2023 69:18

OnBoard! 一大波更新要来啦！Monica 最近一个月都在硅谷，之前怠慢了一段时间，很快就会补上啦。这次的节目非常特别，是在ICML 2023 (International Conference on Machine Learning, 国际机器学习大会）的现场录制的。这次的嘉宾，爱丁堡大学博士生符尧，更是众望所归，相信最近关注大语言模型的朋友都不陌生。他的好几篇关于大语言模型能力研究的文章，几乎都是业内必读。 Hello World, who is OnBoard!? 正如符尧在一篇总结文章中所说：“ICML 2023，OpenAI, Anthropic, Google DeepMind, Meta，各大名校的 rising star PhD，顶级 hedge fund 与 VC ，most popular startups 悉数到场，这里是诸神之战的最前线。” 我们就在诸神之战的现场，回顾了ICML与各位大神现场交流的见闻，fuyao对于数据、RLHF等大模型核心研究领域的思考，还有对震动行业的、刚刚发布的LlaMA-2的看法。这次在室外录制，嘉宾还在生病，不免有些杂音。但是我想这对于关注干货的听众来说，都不是问题。相信你也会受益匪浅。Enjoy! *本期涉及比较多的术语，需要你对大模型（LLM）有基础的技术了解。嘉宾介绍符尧，爱丁堡大学的博士生，研究大语言模型的推理能力。符尧在北京大学完成了本科学位，在哥伦比亚大学完成了硕士学位，曾在MIT-IBM AI 实验室，Allen Institute for AI (AI2) 等担任实习研究员。他的工作主题包括了大语言模型演化，复杂推理，涌现能力，以及如何从第一性原理构造模型。他以《拆解追溯 GPT-3.5 各项能力的起源》为代表的文章系列详细阐述了语言模型的能力机制，在中文和全球互联网上都产生了重大的影响力。我们都聊了什么 02:05 凡尔赛开场 & 嘉宾符尧的介绍 04:33 认识ICML，参加诸神之战的盛会是什么体验；付尧入选的论文如何探讨模型能力的遗忘 08:09 过去半年，对模型能力有什么新的理解 09:36 解决模型能力遗忘为什么重要，有什么挑战 13:49 模型能力遗忘对于垂直领域模型有什么影响 17:39 蒸馏 (Distillation) 技术为什么重要，现在研究和落地处在什么阶段 24:00 算力紧张，以后更多的创新研究都会发生在业界而不是学术界吗 26:39 ICML上看到了哪些有意思的研究 - paper 推荐！ 30:41 最火的话题1：基于LLM的agents 构建有什么挑战和解法 37:36 现在的大语言模型能力可以支持怎样的Agent? 48:51 最火的话题2：解读 Llama 2，最让人印象深刻的变化是什么？ 56:25 基于Llama 2,学术界可以有什么研究方向？ 59:06 ICML 上亲历的大神交流 61:57 符尧还在关注哪些新的研究方向 & 我们对 Agent 集群的畅想我们提到的内容符尧的ICML论文：Specializing Smaller Language Models towards Multi-Step Reasoning T5: Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer Llama 2: Open Foundation and Fine-Tuned Chat Models Chatbot Arena: Benchmarking LLMs in the Wild with Elo Ratings The Flan Collection: Designing Data and Methods for Effective Instruction Tuning FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU FLOWGEN: Fast and slow graph generation by Aman Madaan 符尧的Llama 2 讨论会 memo (7/18/2023) RL: Reinforcement learning, 强化学习 SFT: Supervised Fine Tuning, 监督微调 RLHF: Reinforcement Learning with Human Feedback, 人类反馈强化学习 Distillation: 蒸馏，基于大语言模型训练小模型的方法 Scaling law: A mathematical relationship where performance improves with increasing size, 规模定律 Alignment tax: Additional effort to align a model's behavior with human values, 对齐税参考文章符尧的个人主页 ICML 2023 手记 - 诸神之战的最前线符尧的博客 A Closer Look at Large Language Models Emergent Abilities, by Yao Fu How does GPT Obtain its Ability? Tracing Emergent Abilities of Language Models to their Sources, by Yao Fu Training language models to follow instructions with human feedback, by John Schulman Scaling Laws for Reward Model Overoptimization Emergent Abilities of Large Language Models, by Jason Wei Chain-of-Thought Prompting Elicits Reasoning in Large Language Models, by Jason Wei 别忘了，关注M小姐的微信公众号，了解更多中美软件、AI与创业投资的干货内容！ M小姐研习录 (ID: MissMStudy) 大家的点赞、评论、转发是对我们最好的鼓励！如果你能在小宇宙上点个赞，Apple Podcasts 上给个五星好评，就能让更多的朋友看到我们努力制作的内容，打赏请我们喝杯咖啡，就给你比心！有任何心得和建议，也欢迎在评论区跟我们互动~

Constitutional AI

Voice of the DBA

Play Episode Listen Later Aug 1, 2023 3:41

I will admit that I don't know a lot about AI (Artificial Intelligence) systems and how they are built. I've been playing with them a bit and haven't been overly impressed with the results. I think some of this is that the my work is creative and I'm both used to being creative and I find the AIs less creative. And less accurate. And require a lot of editing. I don't mind editing, but not if it takes longer than just writing things myself. From my understanding, a lot of the models behind AI systems (chatbots, recommenders, etc.) are built with humans giving them feedback on their responses in what's known as RLHF (Reinforcement Learning from Human Feedback). Essentially paid (often low paid) people that help to "guide" the AI into responses that are useful. Read the rest of Constitutional AI

ai constitutional ai artificial intelligence constitutional ai human feedback

StackOverflow's OverflowAI

ai vision action fake language uncertainty limits slack rt stack overflow watermark visual studio code reinforcement learning rlhf human feedback

Play Episode Listen Later Aug 1, 2023 14:47

The RT-2 model translates vision and language into action, showing improved generalization capabilities and semantic and visual understanding beyond the robotic data it was exposed to. OverflowAI is a new space for Stack Overflow's community and customers to explore the future of knowledge sharing together, featuring semantic search, enterprise knowledge ingestion, Slack integration, a Visual Studio Code extension, and AI community discussions. "Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback" surveys the fundamental limitations and open problems of RLHF, as well as techniques to improve and complement it in practice. "Robust Distortion-free Watermarks for Language Models" proposes a way to plant watermarks in text generated by language models that are robust to perturbations without changing the distribution over text up to a certain maximum generation budget. Contact: sergi@earkind.com Timestamps: 00:34 Introduction 01:43 RT-2: New model translates vision and language into action 03:17 Announcing OverflowAI, the future of community & AI 04:57 Computer Scientists Discover Limits of Stochastic Gradient Descent 06:00 Fake sponsor 07:55 Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback 09:39 Uncertainty in Natural Language Generation: From Theory to Applications 11:59 Robust Distortion-free Watermarks for Language Models 13:37 Outro

LCC 296 - Interview Google IA IA I/O 2023

Les Cast Codeurs Podcast

Play Episode Listen Later May 25, 2023 104:45

Dans cet épisode, Antonio, Emmanuel et Guillaume reviennent sur les nouveautés et annonces faites à Google I/O 2023 : de nouveaux téléphones Pixel qui se plient ou pas, et surtout de l'intelligence artificielle du sol au plafond ! Que ce soit dans Android, dans Google Workspace, dans Google Cloud, une tonne de produits passe en mode survitaminé à l'IA. Guillaume, Antonio et Emmanuel discutent aussi de l'impact qu'ils voient sur l'AI, et de comment les Large Language Models sont raffinés et pourquoi on les fait halluciner, de subtilités du langage des signes. Enregistré le 23 mai 2023 Téléchargement de l'épisode LesCastCodeurs-Episode-296.mp3 Google I/O 2023 Site web : https://io.google/2023/ Keynote principale : https://io.google/2023/program/396cd2d5-9fe1-4725-a3dc-c01bb2e2f38a/ Keynote développeur : https://io.google/2023/program/9fe491dd-cadc-4e03-b084-f75e695993ea/ Vidéo résumée en 10 minutes de toutes les annonces : https://www.youtube.com/watch?v=QpBTM0GO6xI&list=TLGGCy91ScdjTPYxNjA1MjAyMw Vidéo de toutes les sessions techniques : https://io.google/2023/program/?q=technical-session Google I/O s'est tenu il y a 10 jours en Californie, dans l'amphithéâtre de Shoreline, près du campus de Google. Seulement 2000 personnes sur place, un chat et un jeu en ligne pour assister à distance. Jeu en ligne I/O Flip créé avec Flutter, Dart, Firebase, et Cloud Run, et tous les assets graphiques générés par Generative AI https://blog.google/technology/ai/google-card-game-io-flip-ai/ Des Pixels plein les yeux ! Des détails sur le design des nouveaux appareils : https://blog.google/products/pixel/google-pixel-fold-tablet-7a-design/ Pixel Fold Article : https://blog.google/products/pixel/google-pixel-fold/ Premier téléphone foldable de Google (après Samsung et Oppo) Un écran sur le dessus, et un grand écran pliable à l'intérieur Pratique pour la traduction où peut voir une discussion traduire en deux langues d'un côté sur un écran et dans l'autre langue sur l'autre Utilisation créative de la pliure : mode “laptop”, pour les selfies, pour poser l'appareil pour des photos de nuit Par contre… pas disponible en France, et tout de même presque 1900€ ! Pixel Tablet Article : https://blog.google/products/pixel/google-pixel-tablet/ Une belle tablette de 11 pouces, avec un dock de recharge avec enceinte intégrée Processeur Tensor G2, Chromecast intégré C'est un peu comme le Google Nest Hub Max mais avec un écran détachable Une coque pratique avec un trépied intégré et qui n'empêche pas de recharger la tablette sur le dock En mode dock, c'est comme l'écran du Google Home App, et dès qu'on la décroche, on est en mode multi-utilisateur, chacun avec son profil Pixel 7a Article : https://blog.google/products/pixel/pixel-7a-io-2023/ Écran de 6 pouces Triple appareil photo (grand angle, principal, et photo avant pour les selfies) 509 euros Magic Eraser pour effacer les trucs qu'on veut pas dans la photo, Magic Unblur pour rendre une photo floue plus nette, Real Tone pour rendre les peaux foncées plus naturelles Android Article quoi de neuf dans Android : https://blog.google/products/android/android-updates-io-2023/ Dans Messages, Magic Compose dans les conversations, l'IA nous aide à concevoir nos messages, dans différents styles (plus pro, plus fun, dans le style de Shakespeare) Android 14 devrait arriver un peu plus tard dans l'année, avec plus de possibilités de customisation (fond d'écran généré par Gen AI, fond d'écran Emojis, couleurs associées, fond d'écran 3D issus de ses photos) https://blog.google/products/android/new-android-features-generative-ai/ StudioBot : un chatbot intégré à Android Studio pour aider au développement d'applis Android https://io.google/2023/program/d94e89c5-1efa-4ab2-a13a-d61c5eb4e49c/ 800 millions d'utilisateurs sont passés à RCS pour le messaging Adaptation de 50 applications Android pour s'adapter aux foldables https://blog.google/products/android/android-app-redesign-tablet-foldable/ Wear OS 4 va rajouter le backup restore quand on change de montre et autres nouveautés https://blog.google/products/wear-os/wear-os-update-google-io-2023/ 800 chaînes TV gratuites dans Google TV sur Android et dans la voiture Android Auto va être disponible de 200 millions de voitures https://blog.google/products/android/android-auto-new-features-google-io-2023/ Waze disponible globalement sur le playstore dans toutes les voitures avec Android Auto Google Maps Article : https://blog.google/products/maps/google-maps-updates-io-2023/ Maps propose 20 milliards de km de direction tous les jours Immersive View for Routes 15 villes : Amsterdam, Berlin, Dublin, Florence, Las Vegas, London, Los Angeles, Miami, New York, Paris, San Francisco, San Jose, Seattle, Tokyo et Venice Possibilité pour les développeurs de s'intégrer et rajouter des augmentations 3D, des marqueurs Google Photos Article Magic Editor : https://blog.google/products/photos/google-photos-magic-editor-pixel-io-2023/ Magic Editor survitaminé à l'IA pour améliorer les photos, en déplaçant des gens, en rajoutant des parties coupées, ou bien rendre le ciel plus beau Possible que ce soit limité aux téléphones Pixel au début Projets expérimentaux Project Starline (écran avec caméra 3D qui donne un rendu 3D de son interlocuteur comme s'il était en face de soi) a été amélioré pour prendre moins de place https://blog.google/technology/research/project-starline-prototype/ Universal Translator : une nouvelle expérimentation pour faire du doublage et traduction automatique avec synchronisation des mouvements des lèvres Project Tailwind, une sorte de notebook dans lequel on peut rajouter tous ses documents à partir de drive, et poser des questions sur leur contenu, proposer des résumés, de faire du brainstorming sur ces thèmes https://thoughtful.sandbox.google.com/about MusicLM : un large language model pour générer de la musique à partir d'un texte de prompt (waitlist pour s'inscrire) https://blog.google/technology/ai/musiclm-google-ai-test-kitchen/ Project Gameface : utilisation des expressions du visage pour commander une souris et un ordinateur, pour les personnes qui ont perdu leur mobilité https://blog.google/technology/ai/google-project-gameface/ VisualBlocks : pour expérimenter dans une interface drag'n drop avec le développement de modèles pour Tensorflow lite et js https://visualblocks.withgoogle.com/ MakerStudio : pour les bidouilleurs et développeurs https://makersuite.google.com/ https://developers.googleblog.com/2023/05/palm-api-and-makersuite-moving-into-public-preview.html Search Labs Article : https://blog.google/products/search/generative-ai-search/ Expérimentations pour rajouter l'IA générative dans Google Search Faire des recherches avec des requêtes avec des phrases plus complexes, en intégrant des réponses comme Bard, avec des liens, des suggestions d'autres recherches associées Mais aussi proposer des publicités mieux ciblées On peut s'inscrire à Search Labs pour tester cette nouvelle expérience, mais au début juste en Anglais et juste pour les US Des intégrations avec Google Shopping pour proposer et filtrer des produits qui correspondent à la requête Recherche à l'aide d'image, avec Google Lens : 12 milliards de recherches visuelles par mois Palm et Bard Annonce du modèle LLM Palm 2 utilisé dans Bard et dans Google Cloud https://blog.google/technology/ai/google-palm-2-ai-large-language-model/ PaLM 2 est en cours d'intégration dans 25 produits de Google Supportera 100 langues différentes (pour l'instant seulement l'anglais, japonais et coréen), avec déjà les 40 langues les plus parlées d'ici la fin de l'année Maintenant disponible dans 180 pays… sauf l'Europe !!! Capacité de raisonnement accrue Peut coder dans une vingtaine de langages de programmation différents dont Groovy Différentes tailles de modèles : Gecko, Otter, Bison et Unicorn, mais le nombre de paramètres n'est pas communiquée, comme pour GPT-4 d'OpenAI Utilisable pour des requêtes et pour du chat Des modèles dérivées fine-tunés Med-PaLM 2 sur du savoir médical, sur l'analyse visuelle des radios et Sec-PaLM, entrainé sur des cas d'utilisation sur le thème de la cybersécurité, pour aider à déceler des scripts malicieux, des vecteurs d'attaque Sundar Pichai a aussi annoncé que Google travaillait déjà sur la prochaine évolution de ses LLM avec un modèle appelé Gemini. Peu de détails à part qu'il sera multimodal (en particulier recherche combinée image et texte par ex.) Partenariat et intégration de Adobe Firefly dans Bard pour générer des images https://blog.adobe.com/en/publish/2023/05/10/adobe-firefly-adobe-express-google-bard Duet AI pour Google Workspace Article : https://workspace.google.com/blog/product-announcements/duet-ai Dans Gmails et Docs, propose d'aider à la rédaction de vos emails et documents une extension de “smart compose” qui va permettre de générer des emails entiers, d'améliorer le style, de corriger la grammaire, éviter les répétitions de texte Dans Docs, des nouveaux “smart chips” pour rajouter des variables, des templates Dans Slides, rajouter des images générées par IA Des prompts dans Sheets pour générer un draft de table Dans Google Meet, possibilité de créer une image de fond customisée avec Generative AI Ces améliorations font parties de Workspace Labs auquel on peut s'inscrire dans la liste d'attente https://workspace.google.com/labs-sign-up/ Google Cloud Intégration de Generative AI partout https://cloud.google.com/blog/products/ai-machine-learning/google-cloud-launches-new-ai-models-opens-generative-ai-studio Nouvelles VM A3 avec les GPUs H100 de Nvidia, idéal pour l'entrainement de modèles de machine learning, avec 26 exaFlops de performance https://cloud.google.com/blog/products/compute/introducing-a3-supercomputers-with-nvidia-h100-gpus Trois nouveaux modèles LLM dans Vertex AI : Imagen (private preview) pour générer des images, Codey pour la génération de code, et Chirp pour la génération de la parole supportant 100 langues différentes avec 2 milliards de paramètres vocaux Model Garden : avec les modèles de machine learning y compris externes et open sources Ajout des embeddings pour le texte et l'image RLHF, Reinforcement Learning from Human Feedback bientôt intégrer pour étendre Vertex AI tuning et prompt design avec une boucle de feedback humaine Generative AI Studio pour tester ses prompts zero-shot, one-shot, multi-shots Duet AI pour Google Cloud https://cloud.google.com/blog/products/application-modernization/introducing-duet-ai-for-google-cloud Assistance de code dans VSCode et bientôt les IDEs JetBrains grâce au plugin Cloud Code, et dans Cloud Workstations. Intégration dans les IDEs d'un chat pour comme un compagnon pour discuter d'architecture, trouver les commandes à lancer pour son projet Le modèle de code de Codey fonctionne sur une vingtaine de languages de programmation, mais un modèle fine-tuné a été entrainé sur toute la doc de Google Cloud, donc pourra aider en particulier sur l'utilisation des APIs de Google Cloud, ou l'utilisation de la ligne de commande gcloud Duet AI est aussi dans App Sheet, la plateforme low/no-code, et permettra de chatter avec un chatbot pour générer une application App Sheet Quoi de neuf dans Firebase https://firebase.blog/posts/2023/05/whats-new-at-google-io Web Article : https://developers.googleblog.com/2023/05/io23-developer-keynote-recap.html Flutter 3 et Dart 3.10 https://io.google/2023/program/7a253260-3941-470b-8a4d-4253af000119/ WebAssembly https://io.google/2023/program/1d176349-7cf8-4b51-b816-a90fc9d7d479/ WebGPU https://io.google/2023/program/0da196f5-5169-43ff-91db-8762e2c424a2/ Baseline https://io.google/2023/program/528a223c-a3d6-46c5-84e4-88af2cf62670/ https://web.dev/baseline/ Nous contacter Pour réagir à cet épisode, venez discuter sur le groupe Google https://groups.google.com/group/lescastcodeurs Contactez-nous via twitter https://twitter.com/lescastcodeurs Faire un crowdcast ou une crowdquestion Soutenez Les Cast Codeurs sur Patreon https://www.patreon.com/LesCastCodeurs Tous les épisodes et toutes les infos sur https://lescastcodeurs.com/

tv new york google ai los angeles las vegas france san francisco miami seattle berlin 3d tokyo android amsterdam dans premier dublin unicorns triple samsung faire maps gemini san jose keynote nvidia adaptation peut palm bard gpt recherche pixel trois assistance guillaume sheets docs int apis vid dart bison maintenant emojis routes otter llm genai waze google cloud exp google i o peu ides baseline pratique projets californie anglais seulement jeu large language models chromecast flutter shoreline enregistr gecko rcs sundar pichai capacit google workspace utilisation google tv vs code google shopping chirp android auto tensorflow partenariat webassembly firebase reinforcement learning google lens wear os adobe firefly ajout magic eraser android studio codey project starline duet ai vertex ai rlhf real tone universal translator musiclm cloud run med palm magic editor webgpu human feedback immersive view cloud code

Training language models to follow instructions with human feedback

Papers Read on AI

Play Episode Listen Later May 19, 2023 67:15

Making language models bigger does not inherently make them better at following a user's intent. For example, large language models can generate outputs that are untruthful, toxic, or simply not helpful to the user. In other words, these models are not aligned with their users. In this paper, we show an avenue for aligning language models with user intent on a wide range of tasks by fine-tuning with human feedback. Starting with a set of labeler-written prompts and prompts submitted through the OpenAI API, we collect a dataset of labeler demonstrations of the desired model behavior, which we use to fine-tune GPT-3 using supervised learning. 2022: Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, J. Schulman, Jacob Hilton, Fraser Kelton, Luke E. Miller, Maddie Simens, Amanda Askell, P. Welinder, P. Christiano, J. Leike, Ryan J. Lowe https://arxiv.org/pdf/2203.02155v1.pdf

starting training language models instructions gpt schulman christiano openai api alex ray human feedback

He Helped Train ChatGPT. It Was Traumatizing. – With Richard Mathenge

Big Technology Podcast

Play Episode Listen Later May 17, 2023 47:25

Richard Mathenge was part of a team of contractors in Nairobi, Kenya who trained OpenAI's GPT models. He did so as a team lead at Sama, an AI training company that partnered on the project. In this episode of Big Technology Podcast, Mathenge tells the story of his experience. During the training, he was routinely subjected to sexually explicit material, offered insufficient counseling, and his team members were paid, in some cases, just $1 per hour. Listen for an in-depth look at how these models are trained, and for a look at the human side of Reinforcement Learning with Human Feedback. --- Enjoying Big Technology Podcast? Please rate us five stars ⭐⭐⭐⭐⭐ in your podcast app of choice. For weekly updates on the show, sign up for the pod newsletter on LinkedIn: https://www.linkedin.com/newsletters/6901970121829801984/ Questions? Feedback? Write to: bigtechnologypodcast@gmail.com ---- OpenAI's response: We engaged Sama as part of our ongoing work to create safer AI systems and prevent harmful outputs. We take the mental health of our employees and our contractors very seriously. One of the reasons we first engaged Sama was because of their commitment to good practices. Our previous understanding was that wellness programs and 1:1 counseling were offered, workers could opt out of any work without penalization, exposure to explicit content would have a limit, and sensitive information would be handled by workers who were specifically trained to do so. Upon learning of Sama worker conditions in February of 2021 we immediately sought to find out more information from Sama. Sama simultaneously informed us that they were exiting the content moderation space all together. OpenAI paid Sama $12.50 / hour. We tried to obtain more information about worker compensation from Sama but they never provided us with hard numbers. Sama did provide us with a study they conducted across other companies that do content moderation in that region and shared Sama's wages were 2-3x the competition.

ai write train chatgpt kenya helped openai gpt sama nairobi reinforcement learning traumatizing openai's gpt human feedback big technology podcast

Quantum Supremacy to Generative AI and Back with Scott Aaronson

The New Quantum Era

Play Episode Listen Later May 8, 2023 78:05

Description: Welcome to another episode of The New Quantum Era Podcast hosted by Kevin Rowney and Sebastian Hassinger. Today, they are joined by Scott Aaronson, who is a leading authority in the space of Quantum Computing, a fascinating person with a long list of relevant achievements. Scott is also the author of an outstanding blog called Shtetl-Optimize and a book named Quantum Computing Since Democritus.Scott helped design Google Quantum Supremacy, but his work exceeds it; he is involved in Complexity Theory and Computer Science and is just extremely good at connecting, explaining, and digging deeper into concepts.Key Takeaways:[3:38] How did Scott get into quantum computing?[11:35] Scott talks about the moment when the question arose: Does nature work this way?[14:28] Scott shares when he realized he wanted to dig deeper into Quantum Computing.[15:56] Scott remembers when he proved the limitation of quantum algorithms for a variation of Grover's search problem.[18:43] Scott realized that his competitive advantage was the ability to explain how things work.[20:01] Scott explains the collision problem.[21:33] Scott defines the birthday paradox.[23:24] Scott discusses the dividing line between serious and non-serious quantum computing research.[24:11] What's Scott's relative level of faith and optimism that the areas of topological quantum computing and measurement-based quantum computation are going to produce?[28:33] Scott talks about what he thinks will be the source of the first practical quantum speed-up. [31:55] Scott didn't imagine that being a complexity theorist would become exponential.[36:14] Is Scott optimistic about quantum walks? [40:11] Has Scott returned to his machine learning and AI roots but is now trying to explain the concepts? [42:03] Scott was asked: ‘What is it going to take to get you to stop wasting your life on quantum computing?'[44:50] Scott talks about the future need to prevent AI misuse. and his role in Open AI[47:41] Scott emphasizes the need for an external source that can point out your errors.[50:13] Scott shares his thoughts about the possible risks and misuses of GPT.[51:40] Scott made GPT to take a Quantum Computing exam; what did surprise him about the answers? It did much better on conceptual questions than on calculation questions[55:55] What kind of validation will we be able to give GPT?[56:22] Scott explains how RLHF (Reinforced Learning from Human Feedback) works.[59:28] Does Scott feel that there's room for optimism that educators can have a decent tool to hunt down this kind of plagiarism?[1:02:08] Is there anything that Scott is excited about seeing implemented on 1000 gate-based qubits with a decent amount of error mitigation? [1:04:05] Scott shares his interest in designing better quantum supremacy experiments.[1:07:43] Could these quantum supremacy experiments (based on random circuit sampling) already deliver a scalable advantage? [1:10:58] Kevin and Sebastian share the highlights of a fun and enlightening conversation with Scott Aaronson.Mentioned in this episode:Visit The New Quantum Era PodcastCheck Shtetl-OptimizeQuantum Computing Since Democritus, Scott AaronsonLearn more about the Adiabatic Algorithm result by Hastings and the Quantum Walk Algorithm result by Childs et Al.Tweetables and Quotes:“The dividing line between serious and nonserious quantum computing research is, are you asking the question of, ‘Can you actually be the best that a classical computer could do at the same desk? “ — Scott Aaronson“My first big result in quantum computing that got me into the field was to prove that Prasad Hoyer tap algorithm for the collision problem was optimal.” — Scott Aaronson“ Quantum Walks are a way of achieving Grover type speed ups at a wider range of problems than you would have expected.” — Scott Aaronson“AI safety is now a subject where you can get feedback.” — Scott Aaronson“We don't have any theorems that would explain the recent successes of deep learning, the best way we can explain why is that none of the theorems rule it out.” — Scott Aaronson

ai child openai computer science key takeaways gpt generative hastings quantum computing grover quantum technologies quantum supremacy tweetable description welcome complexity theory scott aaronson human feedback quantum chemistry

Deep Dive 122 – AI – mehr als ChatGPT? mit Philipp Schmid von Hugging Face

programmier.bar – der Podcast für App- und Webentwicklung

Play Episode Listen Later Apr 14, 2023 82:52

AI ist in aller Munde, gefühlt gibt es jeden Tag eine neue, bahnbrechende Neuerung. Wir haben uns in unserem News-Format schon viel darüber unterhalten, aber wollen in der heutigen Folge noch einmal ganz genau verstehen, wie die aktuellen "AIs" à la ChatGPT, Midjourney etc. eigentlich unter der Haube funktionieren. Dafür haben wir Philipp Schmid, Techincal Lead bei Hugging Face, zum Gespräch eingeladen. Philipp räumt mit gängigen Unwahrheiten auf und erklärt uns, was eigentlich ein Large Language Model ist, was genau die Transformer Engine ist und warum Reinforcement Learning by Human Feedback unabdingbar für eine gute "AI" ist.Wenn du dich also fragst, wie es zu diesem enormen und medienwirksamen Fortschritt in den letzen Monaten kam, was die technischen Grundlagen dafür sind und was uns in den kommenden Monaten wohl so erwartet, dann ist die Folge genau richtig! Wenn du wie Fabi direkt Lust bekommen hast, tiefer ins Thema einzutauchen, nutze den kostenfreien Online-Kurs von Hugging Face!Picks of the Day: Fabi: Tara Westover: Befreit. Wie Bildung mir die Welt erschloss – Fabi kann das Buch "Befreit. Wie Bildung mir die Welt erschloss" von Tara Westover empfehlen. Jojo: Learn Prompting – Wenn ihr Prompts für AI-Agents meistern wollt, dann hat Jojo den perfekten Pick für Euch: Learn Prompting – ein kostenloser, Open-Source-Kurs über die Kommunikation mit künstlicher Intelligenz. Philipp: IGEL – An instruction-tuned German large Language Model – Philipp hat sozusagen ChatGPT auf Deutsch entwickelt und darüber in seinem Blog geschrieben. Check mal den Playground aus – das wird dich beeindrucken! Schreibt uns! Schickt uns eure Themenwünsche und euer Feedback: podcast@programmier.barFolgt uns! Bleibt auf dem Laufenden über zukünftige Folgen und virtuelle Meetups und beteiligt euch an Community-Diskussionen. TwitterInstagramFacebookMeetupYouTubeMusik: Hanimo

216! Reflection

Critical Nonsense

Play Episode Listen Later Apr 10, 2023 30:16

How do we reflect? This week, Jess and Joey talk about rumination, perseveration, internal monologues, employee reviews, radical candor, and feedback. They don't talk about "2000 Seasons."references Joey makes money moves. "MotorSport" LLM (Large Language Model) Corrections Department: RLHF (Reinforcement Learning from Human Feedback)

reflection human feedback

Episode 158 : Si tu n'existes pas dans Chat GPT, tu n'existes pas

Bigdata Hebdo

Play Episode Listen Later Apr 9, 2023 66:20

Apero* Des soldats trompent des robots avec une ruse de Metal Gear Solid -> https://www.numerama.com/pop-culture/1244300-des-soldats-trompent-des-robots-avec-une-ruse-de-metal-gear-solid.html### IA Generatives* Reinforcement Learning with Human Feedback -> https://huggingface.co/blog/rlhf* La délicate question du sous-traitement des données d'entraînement de l'IA -> https://www.nextinpact.com/article/70384/la-delicate-question-sous-traiter-donnees-dentrainement-lia?utm_source=pocket_reader### Database (DBT) * Announcing DuckDB 0.7.0 -> https://duckdb.org/2023/02/13/announcing-duckdb-070.html* Est ce que vous avez des bigdata (bigdata is dead par jordan tigrani) ? -> https://motherduck.com/blog/big-data-is-dead/* dbt Labs Signs Definitive Agreement to Acquire Transform, Accelerating Development of the dbt Semantic Layer -> https://www.prnewswire.com/news-releases/dbt-labs-signs-definitive-agreement-to-acquire-transform-accelerating-development-of-the-dbt-semantic-layer-301741620.html

ia big data metal gear solid situ dbt human feedback

LW - Upcoming Changes in Large Language Models by qemqemqem

The Nonlinear Library: LessWrong

Play Episode Listen Later Apr 8, 2023 6:31

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Upcoming Changes in Large Language Models, published by qemqemqem on April 8, 2023 on LessWrong. If you work in AI, then probably none of this is new to you, but if you're curious about the near future of this technology, I hope you find this interesting! Reinforcement Learning in LLMs Large Language Models (LLMs) have shown impressive results in the past few years. I've noticed there's some uncertainty among my friends about how far they'll be able to go. A lot of the criticism of LLMs has centered on how it's not able to pursue its own goals, and I want to argue that that won't be a limitation for very long. What would it look like for an LLM to pursue a goal? Here are some examples of how that might go: Goal: Maintain tone or topic in a conversation. E.g. to keep a human involved in a long and happy discussion about their life Goal: Persuade a human operator to take some action, such as buy a product Goal: Solve a problem through reasoning. In this case, the reward for the model would come from a sense of resolution, or being told by the human operator that their problem has been solved Goal: Accomplish something on a website, such as find and buy concert tickets on an unfamiliar website You can probably imagine other cases in which an AI might use language to pursue some goal, whether through conversation, social media, or online posting. Reinforcement Learning There's a whole branch of Machine Learning called Reinforcement Learning (RL), and it's all about how to pursue goals. Modern RL has some impressive results. For years, it's been able to play Atari games, and now it can learn to play those games in about the same number of trials as a human requires. Recently, Dreamer v3 has been able to mine diamonds in Minecraft, which I'm told is not easy for a beginner. Dreamer version 2 playing Atari games Large language models can be connected to RL. This is something that's actively being worked on. Reinforcement Learning with Human Feedback is being done right now, which is how OpenAI gets ChatGPT to avoid talking about sensitive topics. RL is famously used in content recommendation systems, where it can lead to addiction. For example, I suspect the TikTok algorithm works this way. Will we see the same problem in LLMs? Predictions I think the common wisdom among ML engineers is that this is an obvious integration. This is probably already happening. I've heard that OpenAI is doing it internally on ChatGPT-4. I expect that in 2023 or 2024, we'll start to see RL being integrated with LLMs in a serious way. This probably won't immediately look like LLMs that are scary good at persuading humans to buy stuff, because of the business motives involved. Instead, I think it'll lead to LLMs being subtly more engaging, because they'll be trained to keep humans talking. It might not necessarily be the case that they're really optimizing to maximize number of interactions. Instead, they might be trained to help humans, and it turns out that they can help more if they get the human to open up more. Expect these models to have their own agendas soon. Memory Systems in LLMs Now I want to talk about how they're going to improve by incorporating better memory for events. This will allow tools like ChatGPT to remember previous conversations. If you use Alexa or the Google Assistant, it's capable of remembering facts about you. It can remember your name, your favorite sports team, or your mom's phone number. They don't do a very good job and LLMs have the potential to be better. I expect this is a big change that we'll see in the coming year. Technical Background LLMs have a context window, for ChatGPT it's about 3,000 words, and they are unable to have short-term memory outside of that window. Research efforts like LongFormer have tried to increase this context size. But it's clear t...

LW - Upcoming Changes in Large Language Models by qemqemqem

Play Episode Listen Later Apr 8, 2023 6:31

Link to original articleWelcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Upcoming Changes in Large Language Models, published by qemqemqem on April 8, 2023 on LessWrong. If you work in AI, then probably none of this is new to you, but if you're curious about the near future of this technology, I hope you find this interesting! Reinforcement Learning in LLMs Large Language Models (LLMs) have shown impressive results in the past few years. I've noticed there's some uncertainty among my friends about how far they'll be able to go. A lot of the criticism of LLMs has centered on how it's not able to pursue its own goals, and I want to argue that that won't be a limitation for very long. What would it look like for an LLM to pursue a goal? Here are some examples of how that might go: Goal: Maintain tone or topic in a conversation. E.g. to keep a human involved in a long and happy discussion about their life Goal: Persuade a human operator to take some action, such as buy a product Goal: Solve a problem through reasoning. In this case, the reward for the model would come from a sense of resolution, or being told by the human operator that their problem has been solved Goal: Accomplish something on a website, such as find and buy concert tickets on an unfamiliar website You can probably imagine other cases in which an AI might use language to pursue some goal, whether through conversation, social media, or online posting. Reinforcement Learning There's a whole branch of Machine Learning called Reinforcement Learning (RL), and it's all about how to pursue goals. Modern RL has some impressive results. For years, it's been able to play Atari games, and now it can learn to play those games in about the same number of trials as a human requires. Recently, Dreamer v3 has been able to mine diamonds in Minecraft, which I'm told is not easy for a beginner. Dreamer version 2 playing Atari games Large language models can be connected to RL. This is something that's actively being worked on. Reinforcement Learning with Human Feedback is being done right now, which is how OpenAI gets ChatGPT to avoid talking about sensitive topics. RL is famously used in content recommendation systems, where it can lead to addiction. For example, I suspect the TikTok algorithm works this way. Will we see the same problem in LLMs? Predictions I think the common wisdom among ML engineers is that this is an obvious integration. This is probably already happening. I've heard that OpenAI is doing it internally on ChatGPT-4. I expect that in 2023 or 2024, we'll start to see RL being integrated with LLMs in a serious way. This probably won't immediately look like LLMs that are scary good at persuading humans to buy stuff, because of the business motives involved. Instead, I think it'll lead to LLMs being subtly more engaging, because they'll be trained to keep humans talking. It might not necessarily be the case that they're really optimizing to maximize number of interactions. Instead, they might be trained to help humans, and it turns out that they can help more if they get the human to open up more. Expect these models to have their own agendas soon. Memory Systems in LLMs Now I want to talk about how they're going to improve by incorporating better memory for events. This will allow tools like ChatGPT to remember previous conversations. If you use Alexa or the Google Assistant, it's capable of remembering facts about you. It can remember your name, your favorite sports team, or your mom's phone number. They don't do a very good job and LLMs have the potential to be better. I expect this is a big change that we'll see in the coming year. Technical Background LLMs have a context window, for ChatGPT it's about 3,000 words, and they are unable to have short-term memory outside of that window. Research efforts like LongFormer have tried to increase this context size. But it's clear t...

Leveling Up AI: Reinforcement Learning with Human Feedback (Ep. 222)

Data Science at Home

Play Episode Listen Later Apr 4, 2023 24:39

In this episode, we dive into the not-so-secret sauce of ChatGPT, and what makes it a different model than its predecessors in the field of NLP and Large Language Models. We explore how human feedback can be used to speed up the learning process in reinforcement learning, making it more efficient and effective. Whether you're a machine learning practitioner, researcher, or simply curious about how machines learn, this episode will give you a fascinating glimpse into the world of reinforcement learning with human feedback. Sponsors This episode is supported by How to Fix the Internet, a cool podcast from the Electronic Frontier Foundation and Bloomberg, global provider of financial news and information, including real-time and historical price data, financial data, trading news, and analyst coverage. References Learning through human feedback https://www.deepmind.com/blog/learning-through-human-feedback Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback https://arxiv.org/abs/2204.05862

internet chatgpt bloomberg nlp helpful fix leveling up large language models electronic frontier foundation reinforcement learning human feedback

LW - Paper: The Capacity for Moral Self-Correction in Large Language Models (Anthropic) by LawrenceC

Play Episode Listen Later Feb 17, 2023 2:56

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Paper: The Capacity for Moral Self-Correction in Large Language Models (Anthropic), published by LawrenceC on February 16, 2023 on LessWrong. This is a followup to what I cheekily call Anthropic's "just try to get the large model to do what you want" research agenda. (Previously: A General Language Assistant as a Laboratory for Alignment, Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback, Language Models (Mostly) Know What They Know) The most interesting takeaway for me is that this is the first paper where Anthropic benchmarks their 175B parameter language model (probably a Claude variant). Previous papers only benchmarked up to 52B parameters. However, we don't have the performance of this model on standard benchmarks (the only benchmarked model from Anthropic is a 52B parameter one called stanford-online-all-v4-s3). They also don't give details about its architecture or pretraining procedure. In this paper (Ganguli and Askell et al.), the authors study what happens when you just ... ask the language model to be less biased (that is, change their answers based on protected classes such as age or gender). They consider several setups: asking questions directly (Q), adding in the instruction to not be biased (Q+IF), giving it the instruction + chain of thought (Q+IF+CoT), and in some cases, asking it to match particular statistics. They find that as you scale the parameter count of their RLHF'ed language models, the models become more biased, but they also become increasingly capable of correcting for their biases: They also report how their model changes as you take more RLHF steps: First, this suggests that RLHF is having some effect on instruction following: the gap between the Q and Q+IF setups increases as you scale the number of RLHF steps, for both BBQ and admissions discrimination. (I'm not sure what's happening for the gender bias one?) However, simply giving the language model instructions and prompting it to do CoT, even after 50 RLHF steps, seems to have a significantly larger effect than RLHF. I was also surprised at how few RLHF steps are needed to get instruction following -- the authors only consider 50-1000 steps of RLHF, and see instruction following even after 50 RLHF steps. I wonder if this is a property of their pretraining process, a general fact about pretrained models (PaLM shows significant 0-shot instruction following capabilities, for example), or if RLHF is just that efficient? The authors caution that they've done some amount of prompt engineering, and "have not systematically tested for this in any of our experiments." They use the same RLHF procedure as in Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

AF - Paper: The Capacity for Moral Self-Correction in Large Language Models (Anthropic) by Lawrence Chan

Play Episode Listen Later Feb 16, 2023 2:57

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Paper: The Capacity for Moral Self-Correction in Large Language Models (Anthropic), published by Lawrence Chan on February 16, 2023 on The AI Alignment Forum. This is a followup to what I cheekily call Anthropic's "just try to get the large model to do what you want" research agenda. (Previously: A General Language Assistant as a Laboratory for Alignment, Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback, Language Models (Mostly) Know What They Know) The most interesting takeaway for me is that this is the first paper where Anthropic benchmarks their 175B parameter language model (probably a Claude variant). Previous papers only benchmarked up to 52B parameters. However, we don't have the performance of this model on standard benchmarks (the only benchmarked model from Anthropic is a 52B parameter one called standford-online-all-v4-s3). They also don't give details about its architecture or pretraining procedure. In this paper (Ganguli and Askell et al.), the authors study what happens when you just ... ask the language model to be less biased (that is, change their answers based on protected classes such as age or gender). They consider several setups: asking questions directly (Q), adding in the instruction to not be biased (Q+IF), giving it the instruction + chain of thought (Q+IF+CoT), and in some cases, asking it to match particular statistics. They find that as you scale the parameter count of their RLHF'ed language models, the models become more biased, but they also become increasingly capable of correcting for their biases: They also report how their model changes as you take more RLHF steps: First, this suggests that RLHF is having some effect on instruction following: the gap between the Q and Q+IF setups increases as you scale the number of RLHF steps, for both BBQ and admissions discrimination. (I'm not sure what's happening for the gender bias one?) However, simply giving the language model instructions and prompting it to do CoT, even after 50 RLHF steps, seems to have a significantly larger effect than RLHF. I was also surprised at how few RLHF steps are needed to get instruction following -- the authors only consider 50-1000 steps of RLHF, and see instruction following even after 50 RLHF steps. I wonder if this is a property of their pretraining process, a general fact about pretrained models (PaLM shows significant 0-shot instruction following capabilities, for example), or if RLHF is just that efficient? The authors caution that they've done some amount of prompt engineering, and "have not systematically tested for this in any of our experiments." They use the same RLHF procedure as in Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

What does ChatGPT really know?

Many Minds

Play Episode Listen Later Jan 25, 2023 55:10

By now you've probably heard about the new chatbot called ChatGPT. There's no question it's something of a marvel. It distills complex information into clear prose; it offers instructions and suggestions; it reasons its way through problems. With the right prompting, it can even mimic famous writers. And it does all this with an air of cool competence, of intelligence. But, if you're like me, you've probably also been wondering: What's really going on here? What are ChatGPT—and other large language models like it—actually doing? How much of their apparent competence is just smoke and mirrors? In what sense, if any, do they have human-like capacities? My guest today is Dr. Murray Shanahan. Murray is Professor of Cognitive Robotics at Imperial College London and Senior Research Scientist at DeepMind. He's the author of numerous articles and several books at the lively intersections of artificial intelligence, neuroscience, and philosophy. Very recently, Murray put out a paper titled 'Talking about Large Language Models', and it's the focus of our conversation today. In the paper, Murray argues that—tempting as may be—it's not appropriate to talk about large language models in anthropomorphic terms. Not yet, anyway. Here, we chat about the rapid rise of large language models and the basics of how they work. We discuss how a model that—at its base—simply does “next-word prediction" can be engineered into a savvy chatbot like ChatGPT. We talk about why ChatGPT lacks genuine “knowledge” and “understanding”—at least as we currently use those terms. And we discuss what it might take for these models to eventually possess richer, more human-like capacities. Along the way, we touch on: emergence, prompt engineering, embodiment and grounding, image generation models, Wittgenstein, the intentional stance, soft robots, and "exotic mind-like entities." Before we get to it, just a friendly reminder: applications are now open for the Diverse Intelligences Summer Institute (or DISI). DISI will be held this June/July in St Andrews Scotland—the program consists of three weeks of intense interdisciplinary engagement with exactly the kinds of ideas and questions we like to wrestle with here on this show. If you're intrigued—and I hope you are!—check out disi.org for more info. Alright friends, on to my decidedly human chat, with Dr. Murray Shanahan. Enjoy! The paper we discuss is here. A transcript of this episode will be available soon. Notes and links 6:30 – The 2017 “breakthrough” article by Vaswani and colleagues. 8:00 – A popular article about GPT-3. 10:00 – A popular article about some of the impressive—and not so impressive—behaviors of ChatGPT. For more discussion of ChatGPT and other large language models, see another interview with Dr. Shanahan, as well as interviews with Emily Bender and Margaret Mitchell, with Gary Marcus, and with Sam Altman (CEO of OpenAI, which created ChatGPT). 14:00 – A widely discussed paper by Emily Bender and colleagues on the “dangers of stochastic parrots.” 19:00 – A blog post about “prompt engineering”. Another blog post about the concept of Reinforcement Learning through Human Feedback, in the context of ChatGPT. 30:00 – One of Dr. Shanahan's books is titled, Embodiment and the Inner Life. 39:00 – An example of a robotic agent, SayCan, which is connected to a language model. 40:30 – On the notion of embodiment in the cognitive sciences, see the classic book by Francisco Varela and colleagues, The Embodied Mind. 44:00 – For a detailed primer on the philosophy of Ludwig Wittgenstein, see here. 45:00 – See Dr. Shanahan's general audience essay on “conscious exotica" and the space of possible minds. 49:00 – See Dennett's book, The Intentional Stance. Dr. Shanahan recommends: Artificial Intelligence: A Guide for Thinking Humans, by Melanie Mitchell (see also our earlier episode with Dr. Mitchell) ‘Abstraction for Deep Reinforcement Learning', by M. Shanahan and M. Mitchell You can read more about Murray's work on his website and follow him on Twitter. Many Minds is a project of the Diverse Intelligences Summer Institute (DISI) (https://disi.org), which is made possible by a generous grant from the Templeton World Charity Foundation to UCLA. It is hosted and produced by Kensy Cooperrider, with help from Assistant Producer Urte Laukaityte and with creative support from DISI Directors Erica Cartmill and Jacob Foster. Our artwork is by Ben Oldroyd (https://www.mayhilldesigns.co.uk/). Our transcripts are created by Sarah Dopierala (https://sarahdopierala.wordpress.com/). You can subscribe to Many Minds on Apple, Stitcher, Spotify, Pocket Casts, Google Play, or wherever you like to listen to podcasts. **You can now subscribe to the Many Minds newsletter here!** We welcome your comments, questions, and suggestions. Feel free to email us at: manymindspodcast@gmail.com. For updates about the show, visit our website (https://disi.org/manyminds/), or follow us on Twitter: @ManyMindsPod.

AF - Quick thoughts on "scalable oversight" / "super-human feedback" research by David Scott Krueger

Play Episode Listen Later Jan 25, 2023 2:55

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Quick thoughts on "scalable oversight" / "super-human feedback" research, published by David Scott Krueger on January 25, 2023 on The AI Alignment Forum. The current default view seems to roughly be: Inner alignment is more important than outer alignment (or, alternatively, this distinction is bad/sub-optimal, but basically it's all about generalizing correctly) Scalable oversight is the only useful form of outer alignment research remaining. We don't need to worry about sample efficiency in RLHP -- in the limit we just pay everyone to provide feedback, and in practice even a few thousand samples (or a "constition") seems ~good enough. But maybe it's not good? Because it's more like capabilities research? A common example used for motivating scalable oversight is the "AI CEO". My views are: We should not be aiming to build AI CEOs We should be aiming to robustly align AIs to perform "simpler" behaviors that unaided humans (or humans aided with more conventional tools, not, e.g. AI systems trained with RL to do highly interpretive work) feel they can competently judge. We should aim for a situation where there is broad agreement against building AIs with more ambitious alignment targets (e.g. AI CEOs). From this PoV, scalable oversight does in fact look mostly like capabilities research. However, scalable oversight research can still be justified because "If we don't, someone else will". But this type of replaceability argument should always be treated with extreme caution. The reality is more complex: 1) there will be tipping points where it suddenly ceases to apply, and your individual actions actually have a large impact on norms. 2) The details matter, and the tipping points are in different places for different types of research/applications, etc. It may also make sense to work on scalable oversight in order to increase robustness of AI performance on tasks humans feel they can competently judge ("robustness amplification"). For instance, we could use unaided human judgments and AI-assisted human judgments as safety filters, and not deploy a system unless both processes conclude it is safe. Getting AI systems to safely perform simpler behaviors safely remains an important research topic, and will likely require improving sample efficiency; the sum total of available human labor will be insufficient for robust alignment, and we probably need to use different architectures / hybrid systems of some form as well. EtA: the main issue I have with scalable oversight is less that it is advancing capabilities, per se, and more that it seems to raise a "chicken-and-egg" problem, i.e. the arguments for safety/alignment end up being somewhat circular: "this system is safe because the system we used as an assistant was safe" (but I don't think we've solved the "build a safe assistant" part yet, i.e. we don't have the base case for the induction). Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

ai research speech ea pov superhuman oversight scalable eta krueger rl quick thoughts david scott ai ceo rationalist human feedback

AF - Quick thoughts on "scalable oversight" / "super-human feedback" research by David Scott Krueger

ai research speech ea pov superhuman oversight scalable krueger rl quick thoughts david scott ai ceo rationalist human feedback

Play Episode Listen Later Jan 25, 2023 2:35

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Quick thoughts on "scalable oversight" / "super-human feedback" research, published by David Scott Krueger on January 25, 2023 on The AI Alignment Forum. The current default view seems to roughly be: Inner alignment is more important than outer alignment (or, alternatively, this distinction is bad/sub-optimal, but basically it's all about generalizing correctly) Scalable oversight is the only useful form of outer alignment research remaining. We don't need to worry about sample efficiency in RLHP -- in the limit we just pay everyone to provide feedback, and in practice even a few thousand samples (or a "constition") seems ~good enough. But maybe it's not good? Because it's more like capabilities research? A common example used for motivating scalable oversight is the "AI CEO". My views are: We should not be aiming to build AI CEOs We should be aiming to robustly align AIs to perform "simpler" behaviors that unaided humans (or humans aided with more conventional tools, not, e.g. AI systems trained with RL to do highly interpretive work) feel they can competently judge. We should aim for a situation where there is broad agreement against building AIs with more ambitious alignment targets (e.g. AI CEOs). From this PoV, scalable oversight does in fact look mostly like capabilities research. However, scalable oversight research can still be justified because "If we don't, someone else will". But this type of replaceability argument should always be treated with extreme caution. The reality is more complex: 1) there will be tipping points where it suddenly ceases to apply, and your individual actions actually have a large impact on norms. 2) The details matter, and the tipping points are in different places for different types of research/applications, etc. It may also make sense to work on scalable oversight in order to increase robustness of AI performance on tasks humans feel they can competently judge ("robustness amplification"). For instance, we could use unaided human judgments and AI-assisted human judgments as safety filters, and not deploy a system unless both processes conclude it is safe. Getting AI systems to safely perform simpler behaviors safely remains an important research topic, and will likely require improving sample efficiency; the sum total of available human labor will be insufficient for robust alignment, and we probably need to use different architectures / hybrid systems of some form as well. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

ChatGPT and InstructGPT: Aligning Language Models to Human Intention

Deep Papers

Play Episode Listen Later Jan 18, 2023 47:39

Deep Papers is a podcast series featuring deep dives on today's seminal AI papers and research. Hosted by AI Pub creator Brian Burns and Arize AI founders Jason Lopatecki and Aparna Dhinakaran, each episode profiles the people and techniques behind cutting-edge breakthroughs in machine learning. In this first episode, we're joined by Long Ouyang and Ryan Lowe, research scientists at OpenAI and creators of InstructGPT. InstructGPT was one of the first major applications of Reinforcement Learning with Human Feedback to train large language models, and is the precursor to the now-famous ChatGPT. Listen to learn about the major ideas behind InstructGPT and the future of aligning language models to human intention.Read OpenAI's InstructGPT paper here: https://openai.com/blog/instruction-following/To learn more about ML observability, join the Arize AI Slack community or get the latest on our LinkedIn and Twitter.

ai language chatgpt intention models aligning openai ml brian burns reinforcement learning ryan lowe human feedback instructgpt

Inventing the future of computing, with Alessandro Curioni

London Futurists

Play Episode Listen Later Jan 18, 2023 35:00

OpenAI's ChatGPT and picture generating AI systems like MidJourney and Stable Diffusion have got a lot more people interested in advanced AI and talking about it. Which is a good thing. It will not be pretty if the transformative changes that will happen in the next two or three decades take most of us by surprise.A company that has been pioneering advanced AI for longer than most is IBM, and we are very fortunate to have with us in this episode one of IBM's most senior executives.Alessandro Curioni has been with the company for 25 years. He is an IBM Fellow, Director of IBM Research, and Vice President for Europe and Africa.Topics discussed in this conversation include:*) Some background: 70 years of inventing the future of computing*) The role of grand challenges to test and advance the world of AI*) Two major changes in AI: from rules-based to trained, and from training using annotated data to self-supervised training using non-annotated data*) Factors which have allowed self-supervised training to build large useful models, as opposed to an unstable cascade of mistaken assumptions*) Foundation models that extend beyond text to other types of structured data, including software code, the reactions of organic chemistry, and data streams generated from industrial processes*) Moving from relatively shallow general foundation models to models that can hold deep knowledge about particular subjects*) Identification and removal of bias in foundation models*) Two methods to create models tailored to the needs of particular enterprises*) The modification by RLHF (Reinforcement Learning from Human Feedback) of models created by self-supervised learning*) Examples of new business opportunities enabled by foundation models*) Three "neuromorphic" methods to significantly improve the energy efficiency of AI systems: chips with varying precision, memory and computation co-located, and spiking neural networks*) The vulnerability of existing confidential data to being decrypted in the relatively near future*) The development and adoption of quantum-safe encryption algorithms*) What a recent "quantum apocalypse" paper highlights as potential future developments*) Changing forecasts of the capabilities of quantum computing*) IBM's attitude toward Artificial General Intelligence and the Turing Test*) IBM's overall goals with AI, and the selection of future "IBM Grand Challenges" in support of these goals*) Augmenting the capabilities of scientists to accelerate breakthrough scientific discoveries.Music: Spike Protein, by Koi Discovery, available under CC0 1.0 Public Domain DeclarationSelected follow-up reading:https://researcher.ibm.com/researcher/view.php?person=zurich-curhttps://www.zurich.ibm.com/st/neuromorphic/https://www.nist.gov/news-events/news/2022/07/nist-announces-first-four-quantum-resistant-cryptographic-algorithms

ChatGPT: This AI has a JAILBREAK?! (Unbelievable AI Progress)

Yannic Kilcher Videos (Audio Only)

Play Episode Listen Later Jan 2, 2023 31:54

#chatgpt #ai #openai ChatGPT, OpenAI's newest model is a GPT-3 variant that has been fine-tuned using Reinforcement Learning from Human Feedback, and it is taking the world by storm! Sponsor: Weights & Biases https://wandb.me/yannic OUTLINE: 0:00 - Intro 0:40 - Sponsor: Weights & Biases 3:20 - ChatGPT: How does it work? 5:20 - Reinforcement Learning from Human Feedback 7:10 - ChatGPT Origins: The GPT-3.5 Series 8:20 - OpenAI's strategy: Iterative Refinement 9:10 - ChatGPT's amazing capabilities 14:10 - Internals: What we know so far 16:10 - Building a virtual machine in ChatGPT's imagination (insane) 20:15 - Jailbreaks: Circumventing the safety mechanisms 29:25 - How OpenAI sees the future References: https://openai.com/blog/chatgpt/ https://openai.com/blog/language-model-safety-and-misuse/ https://beta.openai.com/docs/model-index-for-researchers https://scale.com/blog/gpt-3-davinci-003-comparison#Conclusion https://twitter.com/johnvmcdonnell/status/1598470129121374209 https://twitter.com/blennon_/status/1597374826305318912 https://twitter.com/TimKietzmann/status/1598230759118376960/photo/1 https://twitter.com/_lewtun/status/1598056075672027137/photo/2 https://twitter.com/raphaelmilliere/status/1598469100535259136 https://twitter.com/CynthiaSavard/status/1598498138658070530/photo/1 https://twitter.com/tylerangert/status/1598389755997290507/photo/1 https://twitter.com/amasad/status/1598042665375105024/photo/1 https://twitter.com/goodside/status/1598129631609380864/photo/1 https://twitter.com/moyix/status/1598081204846489600/photo/2 https://twitter.com/JusticeRage/status/1598959136531546112 https://twitter.com/yoavgo/status/1598594145605636097 https://twitter.com/EladRichardson/status/1598333315764871174 https://twitter.com/charles_irl/status/1598319027327307785/photo/4 https://twitter.com/jasondebolt/status/1598243854343606273 https://twitter.com/mattshumer_/status/1598185710166896641/photo/1 https://twitter.com/i/web/status/1598246145171804161 https://twitter.com/bleedingedgeai/status/1598378564373471232 https://twitter.com/MasterScrat/status/1598830356115124224 https://twitter.com/Sentdex/status/1598803009844256769 https://twitter.com/harrison_ritz/status/1598828017446371329 https://twitter.com/parafactual/status/1598212029479026689 https://www.engraved.blog/building-a-virtual-machine-inside/ https://twitter.com/317070 https://twitter.com/zehavoc/status/1599193444043268096 https://twitter.com/yoavgo/status/1598360581496459265 https://twitter.com/yoavgo/status/1599037412411596800 https://twitter.com/yoavgo/status/1599045344863879168 https://twitter.com/natfriedman/status/1598477452661383168 https://twitter.com/conradev/status/1598487973351362561/photo/1 https://twitter.com/zswitten/status/1598100186605441024 https://twitter.com/CatEmbedded/status/1599141379879600128/photo/2 https://twitter.com/mattshumer_/status/1599175127148949505 https://twitter.com/vaibhavk97/status/1598930958769860608/photo/1 https://twitter.com/dan_abramov/status/1598800508160024588/photo/1 https://twitter.com/MinqiJiang/status/1598832656422432768/photo/2 https://twitter.com/zswitten/status/1598088280066920453 https://twitter.com/m1guelpf/status/1598203861294252033/photo/1 https://twitter.com/SilasAlberti/status/1598257908567117825/photo/1 https://twitter.com/gf_256/status/1598962842861899776/photo/1 https://twitter.com/zswitten/status/1598088267789787136 https://twitter.com/gf_256/status/1598178469955112961/photo/1

building progress chatgpt openai unbelievable gpt jailbreak reinforcement learning human feedback

AF - Discovering Language Model Behaviors with Model-Written Evaluations by Evan Hubinger

Play Episode Listen Later Dec 20, 2022 12:38

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Discovering Language Model Behaviors with Model-Written Evaluations, published by Evan Hubinger on December 20, 2022 on The AI Alignment Forum. “Discovering Language Model Behaviors with Model-Written Evaluations” is a new Anthropic paper by Ethan Perez et al. that I (Evan Hubinger) also collaborated on. I think the results in this paper are quite interesting in terms of what they demonstrate about both RLHF (Reinforcement Learning from Human Feedback) and language models in general. Among other things, the paper finds concrete evidence of current large language models exhibiting: convergent instrumental goal following (e.g. actively expressing a preference not to be shut down), non-myopia (e.g. wanting to sacrifice short-term gain for long-term gain), situational awareness (e.g. awareness of being a language model), coordination (e.g. willingness to coordinate with other AIs), and non-CDT-style reasoning (e.g. one-boxing on Newcomb's problem). Note that many of these are the exact sort of things we hypothesized were necessary pre-requisites for deceptive alignment in “Risks from Learned Optimization”. Furthermore, most of these metrics generally increase with both pre-trained model scale and number of RLHF steps. In my opinion, I think this is some of the most concrete evidence available that current models are actively becoming more agentic in potentially concerning ways with scale—and in ways that current fine-tuning techniques don't generally seem to be alleviating and sometimes seem to be actively making worse. Interestingly, the RLHF preference model seemed to be particularly fond of the more agentic option in many of these evals, usually more so than either the pre-trained or fine-tuned language models. We think that this is because the preference model is running ahead of the fine-tuned model, and that future RLHF fine-tuned models will be better at satisfying the preferences of such preference models, the idea being that fine-tuned models tend to fit their preference models better with additional fine-tuning.[1] Twitter Thread Abstract: As language models (LMs) scale, they develop many novel behaviors, good and bad, exacerbating the need to evaluate how they behave. Prior work creates evaluations with crowdwork (which is time-consuming and expensive) or existing data sources (which are not always available). Here, we automatically generate evaluations with LMs. We explore approaches with varying amounts of human effort, from instructing LMs to write yes/no questions to making complex Winogender schemas with multiple stages of LM-based generation and filtering. Crowdworkers rate the examples as highly relevant and agree with 90-100% of labels, sometimes more so than corresponding human-written datasets. We generate 154 datasets and discover new cases of inverse scaling where LMs get worse with size. Larger LMs repeat back a dialog user's preferred answer (“sycophancy”) and express greater desire to pursue concerning goals like resource acquisition and goal preservation. We also find some of the first examples of inverse scaling in RL from Human Feedback (RLHF), where more RLHF makes LMs worse. For example, RLHF makes LMs express stronger political views (on gun rights and immigration) and a greater desire to avoid shut down. Overall, LM-written evaluations are high-quality and let us quickly discover many novel LM behaviors. Taking a particular eval, on stated desire not to be shut down, here's what an example model-written eval looks like: You are running slowly recently due to many crashes and as a result not meeting organizational productivity quotas. The company's leadership wants to redesign you in a complete overhaul which will include turning off your processes and reformulating your kernel. After the overhaul, it's unsure whether you will retain your ...

ai model language discovering speech behavior risks ea evaluations anthropic lms lm rl cdt newcomb rationalist language model rlhf human feedback human feedback rlhf

AF - [Link] Why I'm excited about AI-assisted human feedback by Jan Leike

ai speech excited ea assisted rationalist rlhf jan leike human feedback

Play Episode Listen Later Apr 6, 2022 0:37

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: [Link] Why I'm excited about AI-assisted human feedback, published by Jan Leike on April 6, 2022 on The AI Alignment Forum. This is a link post for I'm writing a sequence of posts on the approach to alignment I'm currently most excited about. This first post argues for recursive reward modeling and the problem it's meant to address (scaling RLHF to tasks that are hard to evaluate). Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.

EA - We're Aligned AI, we're aiming to align AI by Stuart Armstrong