Podcasts about christiano

In this episode of the Motiv8d Mindset Podcast, host Bobby Stein speaks with Christian O'Rourke about his journey in bodybuilding, the importance of mindset and focus, and the evolution of coaching in the fitness industry. They discuss the power of thoughts and manifestation, influential figures in bodybuilding, and the significance of building healthy relationships with food. Christian shares his personal experiences with faith, music, and his passion for cars, concluding with insights on personal growth and future aspirations.

In this bonus episode, we look at the results of Radio Ratings Survey 3 that took place today. Jen Seyderhelm dissects the numbers and we hear from the 5 main commercial radio group content directors. The overall results for the five metros: 2GB increased their lead as the top station with smooth fm gaining ground also in second. This was 2Day FM’s first full survey with their tailored to youth format and they’ll be happy with their increases, but bizarrely possibly not in the 40-64 age group which isn’t where they’re targeting. Kyle and Jackie O were steady and top FM breakfast behind Ben Fordham at 2GB. Triple M Sydney struggled the most, I suspect because their new lineup to start 2025 lost Tarsh Belling before the ink was dry on her contract. They have announced show anchor Cat Lynch to replace her today and that stability should help improve things from here. Melbourne - 3AW is still No1 and Gold 104.3 No 2, but both saw drops and it was a second significant one for Nine Radio even though Ross and Russ’s breakfast cumes improved. Ross and Russ were No 1 in breakfast but Jase and Lauren at Nova 100 regained FM No 1. They’re audience remained the same, Christian O’Connell’s dropped a percent. The biggest gaines overall though were for ABC Melbourne and Sharnelle Vella and Bob Murphy on breakfast which will be a relief to the pair and the broadcaster. Their 2% improvement saw them overtake Kyle and Jackie O who lost their gains, and more, from last survey. It was also not a great result for Mick Molloy and his crew on Triple M Melbourne. I thought that the sporting season and a bit of time would have seen them improve but not yet. Brisbane. The top four stations all lost a little ground with Triple M staying No 1 overall and Ash Luttsy and Nikki Osborne on Nova 106.9 in breakfast. The big winner however was 4BH and BBQ Gallagher on breakfast. I feel this also happened around this time last year. Both were up to a 9.3 audience share. Can he crack the top 3 this time around? 4BC also continued its steady improvement. Adelaide saw a changing of the guard on breakfast in one of the tightest races and markets. Mix 102.3 was No 1 station overall, increasing its lead over triple m and Nova but on breakfast, despite Roo, Ditts and Loz on Triple M increasing their audience share, Nova 91.9 Jodie and Hayesy has a bigger gain to pinch No 1which saw them end Roo, Ditts and Loz’s 20 survey winning streak. Finally Perth and it was no surprise that Nath, Nat and Shaun at Nova 93.7 remained No 1 on breakfast (but down 2%) however ARN’s 96FM took over as No 1 station. Clairsy and Lisa had good gains on 96FM breakfast to close the gap and this was the one market where triple m breakfast duo Xav and Katie have bedded in nicely and overtook 6PR to sit fourth in that market now. Addition mention to Gary Adshead who departed 6PR last year to join the ABC. ABC had a good survey in Perth and Gary was up 1.2% on Drive. More from the Content Directors at https://radiotoday.com.au/ratings Ratings charts at https://radioinfo.com.au/news/ratings-results-survey-3-2025 See omnystudio.com/listener for privacy information.

In this episode, queer transformation coach Christiano Green shares an intimate story about coming out and the long-lasting impact of a single sentence: “Life will be harder for you now.”For almost 20 years, Christiano carried that belief—until he discovered that queerness doesn't have to equal pain. It just means our path is different… not worse.Listen as Christiano explores:The emotional weight of coming outHow internalised beliefs shape our livesWhy comparison keeps us stuckAnd how queer people can reclaim their power and rewrite their storiesThis is a must-listen for any queer person navigating identity, healing, or self-worth.

The following article of the Cybersecurity industry is: “The Fight Against Identity Fraud: A Million-Dollar Challenge” by  Christiano Lucena, VP and General Manager Latin America, DocuSign.

Polisi menetapkan Christiano, pengemudi BMW yang men4br4k mahasiswa Fakultas Hukum Universitas Gadjah Mada, Argo Eriko Achfandi, sebagai tersangka. Christiano diketahui juga berstatus sebagai mahasiswa UGM.

In this episode of the Radio Today Podcast, Jen Seyderhelm recaps the week in the radio and podcast industries: Sandon stirs the pot at SCA Court clears Nova of ASIC issue and the company's ongoing ATO audit 2TM flicks the switch to FM in Tamworth The Nova Network’s Ben, Liam and Belle head home to Adelaide Jack Post departs the Christian O’Connell show and the National Radio Airplay Chart song of the week, G Flip's Disco Cowgirl. If you make podcasts in Asia, don't forget to enter the Asia Podcast Awards. More information at www.radiotoday.com.au/asia-podcast-awards-2025-now-open www.radiotoday.com.au See omnystudio.com/listener for privacy information.

Christian O'Connell had everything he (thought he) ever wanted, but something just didn't feel right. Most of us have been there too. Christian O’Connell is one of the most beloved voices in radio and currently on Gold FM, an award-winning broadcaster who has reached millions across the UK and Australia. But behind the microphone and the confident persona is a man who has wrestled with fear, doubt, and a profound loss of identity. Sarah Grynberg sits down for a conversation with Christian, as he shares the highs and lows of his incredible journey. He opens up about the emotional weight of navigating anxiety, public scrutiny, and the false rumours that shook the foundation of his personal life. You’ll learn: *How Christian rebuilt his identity after hitting emotional rock bottom. *How he coped with debilitating panic attacks *How he dealt with hurtful fake news being written about his family *Why courage isn’t the absence of fear, but the willingness to keep going *How to make friends at any stage of life This is a story of resilience, rebirth, and having the strength it takes to be fully, fearlessly yourself while enjoying a healthy dose of hilarity along the way. Purchase Sarah's book: Living A Life Of Greatness here. To purchase Living A Life of Greatness outside Australia here or here. Watch A Life of Greatness Episodes On Youtube here. Sign up for Sarah’s newsletter (Greatness Guide) here. Purchase Sarah's Meditations here. Instagram: @sarahgrynberg Website: https://sarahgrynberg.com/ Facebook: facebook.com/sarahgrynberg Twitter: twitter.com/sarahgrynbergSee omnystudio.com/listener for privacy information.

From dominating the UK airwaves to starting over on the other side of the world, Christian O'Connell is a broadcaster and writer who has built a career out of connection, candour, and comedy.In this episode of Full Disclosure, James O'Brien sits down with Christian to trace the path from his childhood in Hampshire to becoming one of the UK's most celebrated radio voices- and the breakdown that forced him to confront what success really meant. They explore the roots of Christian's anxiety, the healing power of storytelling, and the courage it took to walk away from it all to begin again in Australia.Warm, unfiltered and often profound, Christian opens up about family, fame, fear and finding clarity in chaos- a conversation about reinvention, honesty, and what it takes to truly come home to yourself.Discover more about The Heart of Speaking here

Talk of Patsy's daughter Audie's school camp got us thinking.. should we take the show to a camp site for a whole week?See omnystudio.com/listener for privacy information.

Christiano Sacha Fornaciari"La sedia del diavolo"Violenza, tortura e morte nel Brasile dei generaliEdizioni Lindauwww.lindau.itScritto con il rigore di un saggio storico e il ritmo incalzante di un romanzo, "La sedia del drago" racconta un Brasile poco noto e molto distante dallo stereotipo turistico «spiagge, calcio e samba», ancora oggi molto diffuso: è il Brasile dei generali e di una brutale dittatura che, nel quadro geopolitico disegnato dalla Guerra Fredda, a partire dal 1964 avrebbe governato il paese per oltre vent'anni, torturando e assassinando gli oppositori politici, perseguitando intellettuali e artisti, discriminando il mondo LBGTQ, ed esercitando una censura pervasiva su ogni mezzo di comunicazione. Facendo ricorso a fonti testimoniali, agli archivi dei quotidiani e delle riviste dell'epoca, a documenti originali reperiti negli archivi di Stato brasiliani e negli archivi digitali desecretati della CIA e del Dipartimento di Stato USA, nonché ai ricordi personali dell'autore, le pagine di questo libro descrivono con straordinaria efficacia eventi e atmosfere di quegli anni cupi, l'organizzazione e i principi dell'apparato repressivo, il coraggio dei dissidenti, le sofferenze delle vittime e la crudeltà dei loro carnefici. Un angosciante viaggio nel cuore di tenebra dell'uomo.Christiano Sacha Fornaciari, architetto, è nato a São Paulo del Brasile nel 1962. Si è laureato all'Istituto universitario di architettura di Venezia, dove è stato allievo di Massimo Cacciari per gli studi di estetica e di Franco Rella per gli studi di letteratura artistica. Componente della Consulta per l'arte sacra dell'arcidiocesi di Udine, si è perfezionato in Architettura e arte per la liturgia presso la Facoltà di sacra liturgia del Pontificio ateneo Sant'Anselmo in Roma.IL POSTO DELLE PAROLEascoltare fa pensarewww.ilpostodelleparole.itDiventa un supporter di questo podcast: https://www.spreaker.com/podcast/il-posto-delle-parole--1487855/support.

In Sydney 2GB is No 1 station just ahead of smooth 95.3. Ben Fordham is No 1 for GB on breakfast, Kyle and Jackie O No 1 FM. Smooth’s Bogart Torelli and Jonesy and Amanda on Gold 101.7 saw the biggest losses and the biggest gains were for triple j off the back of the Hottest 100 and for Nova 96.9, bearing in mind that Kate Ritchie has been on personal leave from breakfast with Fitzy and Wippa since early Feb. Mark Levy who took over from Ray Hadley on 2GB Mornings when Ray retired last year held on to No 1 spot. Melbourne, 3AW is a long way ahead at No 1 and for Ross and Russ on breakfast. No 1 FM breakfast belonged to Christian O’Connell who was up most of all and overtook Fifi Fev and Nick at the fox and Jase and Lauren at Nova 100 along the way. RSN 927 apparently will not feature in GfK surveys in 2025. Radioinfo has reached out to the station for comment. Brisbane – b105 retains No 1 overall and on breakfast. Marto Margaux and Dan at triple M have closed the gap though. Adelaide Triple M moved further in front at the top but strangely had a 2.1% drop on breakfast for Roo, Ditts and Loz. I expect that to change when survey 2 and AFL seasons are included. A massive congrats to Sonya and Jules on ABC Adelaide breakfast, up 2.8 and going from 5th to second behind the triple M team. And finally Perth where nova 93.7 and Nath, Nat and Shaun on breakfast are as unassailable as Russ and Ross on 3AW in Melbourne. Nath, Nat and Shaun were up 3.6% to a 19.8% audience share. 96FM’s Clairsy and Lisa were also up 1.5 to 12.6 in second. Jen Seyderhelm also explores the weekly reach results (cume) and the digital station results in this special boinus podcast episode of the Radio Today Podcast.See omnystudio.com/listener for privacy information.

Building a successful blog takes more than just compelling writing and eye-catching photos. To truly expand your reach and attract consistent traffic, you need to harness the power of Search Engine Optimization (SEO). In this episode, we're joined by Anne Zirkle and AnnMarie Christiano to explore actionable strategies to help you grow your blog with SEO, making it more visible, valuable, and sustainable. Resources Mentioned Shift Boldfluence Simply2moms.com Connect with Simply2moms Simply2moms.com Find them on Facebook! Check out their Pinterest Page They're on TikTok! Follow their Youtube

No 43º episódio do B IS THE NEW A - The Podcast, Davi Cury sentou pra conversar com o Christiano Coelho. Executivo e consultor especialista em gestão de marca, o Christiano tem em sua bagagem nomes como Asics, Nike, Under Armor e Veja (antiga Vert no Brasil).   Se você é um(a) sneakerhead ou apaixonado(a) por marcas esportivas, não perca esse episódio!

On today's episode, Clay is joined by Cristiano Souza to discuss his three primary criteria in identifying an excellent business. Cristiano Souza is the founder of Zeno Equity Partners, a firm he started in 2022 after spending 28 years at Dynamo. From 1994 through 2015, he was on the team that managed the Dynamo Cougar fund, which invested in high-quality equities in Brazil. During that time, the fund compounded at 24% per year in US dollar terms. IN THIS EPISODE YOU'LL LEARN: 00:00 - Intro 03:33 - The three things that Cristiano looks for in a business. 13:43 - What the Founder's Mindset is, and why it is so difficult to find. 18:53 - Why Cristiano believes that LVMH has a significant opportunity to continue to grow. 33:53 - The mistakes Cristiano has made in assessing management teams. 39:57 - Why Linde PLC is a core position in Christiano's portfolio. 51:17 - Why AppFolio is well-positioned to continue growing in the property management space. And so much more! Disclaimer: Slight discrepancies in the timestamps may occur due to podcast platform differences. BOOKS AND RESOURCES Join the exclusive TIP Mastermind Community to engage in meaningful stock investing discussions with Stig, Clay, Kyle, and the other community members. Cristiano's firm: Zeno Equity Partners. Book mentioned: Mastery by Robert Greene, The Nvidia Way by Tae Kim. Email Shawn at shawn@theinvestorspodcast.com to attend our free events in Omaha or visit this page. Follow Cristiano on LinkedIn. Follow Clay on X. Check out all the books mentioned and discussed in our podcast episodes here. Enjoy ad-free episodes when you subscribe to our Premium Feed. NEW TO THE SHOW? Get smarter about valuing businesses in just a few minutes each week through our newsletter, The Intrinsic Value Newsletter. Check out our We Study Billionaires Starter Packs. Follow our official social media accounts: X (Twitter) | LinkedIn | Instagram | Facebook | TikTok. Browse through all our episodes (complete with transcripts) here. Try our tool for picking stock winners and managing our portfolios: TIP Finance Tool. Enjoy exclusive perks from our favorite Apps and Services. Learn how to better start, manage, and grow your business with the best business podcasts. SPONSORS Support our free podcast by supporting our sponsors: Hardblock Found SimpleMining CFI Education Netsuite Unchained The Bitcoin Way Vanta Shopify Fintool Onramp TurboTax Vanta Fundrise HELP US OUT! Help us reach new listeners by leaving us a rating and review on Spotify! It takes less than 30 seconds, and really helps our show grow, which allows us to bring on even better guests for you all! Thank you – we really appreciate it! Support our show by becoming a premium member! https://theinvestorspodcastnetwork.supportingcast.fm Learn more about your ad choices. Visit megaphone.fm/adchoices Support our show by becoming a premium member! https://theinvestorspodcastnetwork.supportingcast.fm

In this week's episode of The Queer Quest Podcast, Christiano shares a podcast episode from the Be Alive podcast where he was interviewed.Christiano discusses his journey from a high-figure corporate career in Australia to becoming a transformational coach for the queer community. The conversation covers Christiano's personal experiences with bullying, coming out as gay, and overcoming substance abuse, as well as the formation of his movement, Queer Quest.He also shares insights on journaling and self-acceptance, emphasizing the importance of self-awareness, self-worth, self-confidence, and self-care. The episode underscores the power of community, leadership, and radical self-acceptance in personal growth.To check out Queer Quest PRO visit https://courses.queerquest.vip/queer-quest-pro/Check it out and don't forget to like, comment, and subscribe so someone else in the queer community who needs it can see this video.

Toda sexta-feira a partir das 21:00, o canal do YouTube "Rock Doido do Argoth" entrevista bandas e músicos ligados a cena Rock/Metal do nosso país, sempre em um papo descontraído e cheio de informações para você.Apresentado por Argoth Rodrigues, o conteúdo dinâmico e acessível rende aos espectadores boas doses de entretenimento e também diversão.Neste segundo EP a entrevista foi com o músico Christiano Guerra da banda "Spectrummm".Gostou? Curta e compartilhe!Pesquise "Rock Doido do Argoth" nas redes sociais e fique ligado nos mais variados conteúdos semanais.Todos os direitos reservados ao canal "Rock Doido do Argoth", favor não reproduzir o conteúdo sem prévia autorização.

This week on 'The Queer Quest Podcast', host Christiano Green shares his brand new webinar 'How to Love Your Queer Self Unapologetically Without the Fears of Society's Judgments' with the queer community. Christiano runs this webinar live every few weeks and there are countless questions as to if there would be a replay available so Christiano has made that wish a possibility and shares the full webinar for his amazing followers. In the 'How to Love Your Queer Self Unapologetically Without the Fears of Society's Judgments' webinar you will uncover the 3 secrets of queer excellence: 1. How Generic Personal Development is Failing You: Discover the Queer-Centric Approach That Transforms Lives 2. How Fear Is a Sign You're On the Right Path: How to Embrace It and Move Forward 3. How Waiting for the ‘Right Moment' is Stopping You: Why Now is the Time to Start Your Queer Quest If you're curious to find out how queer-centric personal development can help you further on your journey then this episode is for you. Check out Queer Quest PRO today here: https://courses.queerquest.vip/queer-quest-pro

In de nieuwe TVOK aflevering zitten Bjorn en Rob samen met Kerkradenaren Keyanu Schiffelers en Brent Bemelen van De Muurliggers podcast om ze te bevragen over het hoe en waarom omtrent hun eigen (internationale) voetbalpodcast en om samen 5 stellingen te bespreken. De stellingen zijn deze keer; Heeft Roda JC de juiste stappen gezet in de wintertransferperiode? Gaat Michael Breij de sleutel worden om Cukur en Baeten aan het scoren te krijgen? Is Wim Frijns totaal niet meer houdbaar en moet hij per direct weg als stadionspeaker? Moet degradatie uit en promotie naar de 1e Divisie op korte termijn verplicht gesteld worden? Gaat de club waar ik in de premiership sympathie voor heb de doelstelling voor dit seizoen zonder meer halen? Verder wordt er nog gepraat over de toekomst van Roda, het trio Sergio, Christiano en Kone, Jong ploegen, Walker en Hakimi, waarom geen seizoenskaart meer, gemist momentum en nog veel meer. Check 'm en volg ook podcast De Muurliggers daar waar je ook TVOK volgt! Geproduceerd en gepresenteerd door: www.southxvi.com Gesponsord door: Jegers Advocaten: www.jegersadvocaten.nl Next Door Hair Kapsalon, nagel- en beautysalon: www.facebook.com/Hairenmore Hotel Restaurant Vijlerhof: www.vijlerhof.nl Bernardushoeve: www.bernardushoeve.nl Van Ooyen Glashandel: vanooyen.com Wiertz Company: www.wiertz.com Roda Support: www.rodasupport.nl PC Data: www.pcdata-logistics.com Metaalgieterij Van Gilst: www.vangilst.net Roda JC Artic Front Wullenweber Keukens: wullenweberkeukens.jouwweb.nl Stock Grondverzet Stichting Vrienden van Roda Voetbaltrips.com Ostheopathie Daamen: osteopathiedaamen.nl Sportcafe De Aftrap: www.aftrap-kerkrade.nl Bovens Bouwadvies: www.bovens-bouwadvies.nl Maessen & Houben Strafrechtadvocaten: www.maessenhouben.nl Barberroad: barberroad.nl Vakgarage Dorscheidt: www.vakgaragedorscheidt.nl Hoveniersbedrijf van Davy van Loo: www.davyvanloo.nl

It's been a big year 2024, so come join the producers Caitlin and Rio as they recount some of the best moments of the year and share some behind-the-scenes information you won't hear anywhere else! See omnystudio.com/listener for privacy information.

Dużym sukcesem zakończył się mecz Polski z Portugalią. Nasi zawodnicy zrobili sobie fotkę z Christiano Ronaldo. Nie jest łatwo o zdjęcie z Ronaldem. Bo jak jego drużyna przegra to jest zły, i z nikim nie rozmawia. Gdyby Polska wygrała z Portugalią, a Christiano nie strzeliłby gola, to byłby zły i nie dałby sobie zrobić fotki, a tak to… jest focisze. Można oprawić, powiesić na ścianie w stołowym. Tak, nasi piłkarze nie są takim ofiarami, jak się o nich po meczu mówi. Lewandowski nie przyjechał na zgrupowanie, bo on już ma zdjęcie z Ronaldo. Poza tym Robert ma dobre plecy. I plecy Lewandowskiego wyczuły jaki będzie wynik meczu i doradziły Lewemu niewyjazd. Lewy, powiedziały plecy. Co? Jajo. Ogarnij uwarunkowania. Takich pleców nie kupisz za pieniądze. Niektórzy byli kadrowicze i publicyści, komentując foty naszych z Ronaldo, mówią coś o braku honoru, o opadaniu rąk, o braku złości sportowej. Ale o co chodzi? Przegrali to przegrali. Co mieli popełnić honorowe sudoku?

While Christian is away sick, we're going through some of our favourite bits from the show so far!See omnystudio.com/listener for privacy information.

While Christian is away sick, we're going through some of our favourite bits from the show so far!See omnystudio.com/listener for privacy information.

In this week's episode of The Queer Quest Podcast, host Christiano Green talks about the tragic loss of one of his queer friends who passed away recently from a drug overdose. The tragedy hit hard for Christiano for a number of reasons and the main topic of discussion was how easily this could have been him and how these tragic stories are sadly not as rare as they should be. Come join the conversation to the end and if you're struggling then don't be afraid to reach out to someone as they just might be able to help you.

In this episode of ClassHorrorCast, we sit down with Cristiano Crawford, the visionary behind the UK's renowned Psychopath Scream Park.From teaching soccer in the States to running nightclubs and events back in the UK, Cristiano's journey is nothing short of extraordinary.We explore how his diverse experiences led to the creation of one of the biggest and most terrifying haunt attractions in the country. Cristiano opens up about the behind-the-scenes work involved in staging such a massive event, the creative inspirations behind the scares, and some of his all-time favourite moments from past attractions.He also shares valuable insights on the importance of being true to yourself, following your passions, and chasing your dreams, no matter where they lead you.Get ready for an in-depth conversation with the mind behind the screams!If you enjoyed this - Check out my other content here - https://linktr.ee/FirstClassHorrorCheck out more from PsychoPath here - https://www.psycho-path.co.uk/Become a supporter of this podcast: https://www.spreaker.com/podcast/classhorrorcast--4295531/support.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Conflating value alignment and intent alignment is causing confusion, published by Seth Herd on September 5, 2024 on The AI Alignment Forum. Submitted to the Alignment Forum. Contains more technical jargon than usual. Epistemic status: I think something like this confusion is happening often. I'm not saying these are the only differences in what people mean by "AGI alignment". Summary: Value alignment is better but probably harder to achieve than personal intent alignment to the short-term wants of some person(s). Different groups and people tend to primarily address one of these alignment targets when they discuss alignment. Confusion abounds. One important confusion stems from an assumption that the type of AI defines the alignment target: strong goal-directed AGI must be value aligned or misaligned, while personal intent alignment is only viable for relatively weak AI. I think this assumption is important but false. While value alignment is categorically better, intent alignment seems easier, safer, and more appealing in the short term, so AGI project leaders are likely to try it.[1] Overview Clarifying what people mean by alignment should dispel some illusory disagreement, and clarify alignment theory and predictions of AGI outcomes. Caption: Venn diagram of three types of alignment targets. Value alignment and Personal intent alignment are both subsets of Evan Hubinger's definition of intent alignment: AGI aligned with human intent in the broadest sense. Prosaic alignment work usually seems to be addressing a target somewhere in the neighborhood of personal intent alignment (following instructions or doing what this person wants now), while agent foundations and other conceptual alignment work usually seems to be addressing value alignment. Those two clusters have different strengths and weaknesses as alignment targets, so lumping them together produces confusion. People mean different things when they say alignment. Some are mostly thinking about value alignment (VA): creating sovereign AGI that has values close enough to humans' for our liking. Others are talking about making AGI that is corrigible (in the Christiano or Harms sense)[2] or follows instructions from its designated principal human(s). I'm going to use the term personal intent alignment (PIA) until someone has a better term for that type of alignment target. Different arguments and intuitions apply to these two alignment goals, so talking about them without differentiation is creating illusory disagreements. Value alignment is better almost by definition, but personal intent alignment seems to avoid some of the biggest difficulties of value alignment. Max Harms' recent sequence on corrigibility as a singular target (CAST) gives both a nice summary and detailed arguments. We do not need us to point to or define values, just short term preferences or instructions. The principal advantage is that an AGI that follows instructions can be used as a collaborator in improving its alignment over time; you don't need to get it exactly right on the first try. This is more helpful in slower and more continuous takeoffs. This means that PI alignment has a larger basin of attraction than value alignment does.[3] Most people who think alignment is fairly achievable seem to be thinking of PIA, while critics often respond thinking of value alignment. It would help to be explicit. PIA is probably easier and more likely than full VA for our first stabs at AGI, but there are reasons to wonder if it's adequate for real success. In particular, there are intuitions and arguments that PIA doesn't address the real problem of AGI alignment. I think PIA does address the real problem, but in a non-obvious and counterintuitive way. Another unstated divide There's another important clustering around these two conceptions of al...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Conflating value alignment and intent alignment is causing confusion, published by Seth Herd on September 5, 2024 on LessWrong. Submitted to the Alignment Forum. Contains more technical jargon than usual. Epistemic status: I think something like this confusion is happening often. I'm not saying these are the only differences in what people mean by "AGI alignment". Summary: Value alignment is better but probably harder to achieve than personal intent alignment to the short-term wants of some person(s). Different groups and people tend to primarily address one of these alignment targets when they discuss alignment. Confusion abounds. One important confusion stems from an assumption that the type of AI defines the alignment target: strong goal-directed AGI must be value aligned or misaligned, while personal intent alignment is only viable for relatively weak AI. I think this assumption is important but false. While value alignment is categorically better, intent alignment seems easier, safer, and more appealing in the short term, so AGI project leaders are likely to try it.[1] Overview Clarifying what people mean by alignment should dispel some illusory disagreement, and clarify alignment theory and predictions of AGI outcomes. Caption: Venn diagram of three types of alignment targets. Value alignment and Personal intent alignment are both subsets of Evan Hubinger's definition of intent alignment: AGI aligned with human intent in the broadest sense. Prosaic alignment work usually seems to be addressing a target somewhere in the neighborhood of personal intent alignment (following instructions or doing what this person wants now), while agent foundations and other conceptual alignment work usually seems to be addressing value alignment. Those two clusters have different strengths and weaknesses as alignment targets, so lumping them together produces confusion. People mean different things when they say alignment. Some are mostly thinking about value alignment (VA): creating sovereign AGI that has values close enough to humans' for our liking. Others are talking about making AGI that is corrigible (in the Christiano or Harms sense)[2] or follows instructions from its designated principal human(s). I'm going to use the term personal intent alignment (PIA) until someone has a better term for that type of alignment target. Different arguments and intuitions apply to these two alignment goals, so talking about them without differentiation is creating illusory disagreements. Value alignment is better almost by definition, but personal intent alignment seems to avoid some of the biggest difficulties of value alignment. Max Harms' recent sequence on corrigibility as a singular target (CAST) gives both a nice summary and detailed arguments. We do not need us to point to or define values, just short term preferences or instructions. The principal advantage is that an AGI that follows instructions can be used as a collaborator in improving its alignment over time; you don't need to get it exactly right on the first try. This is more helpful in slower and more continuous takeoffs. This means that PI alignment has a larger basin of attraction than value alignment does.[3] Most people who think alignment is fairly achievable seem to be thinking of PIA, while critics often respond thinking of value alignment. It would help to be explicit. PIA is probably easier and more likely than full VA for our first stabs at AGI, but there are reasons to wonder if it's adequate for real success. In particular, there are intuitions and arguments that PIA doesn't address the real problem of AGI alignment. I think PIA does address the real problem, but in a non-obvious and counterintuitive way. Another unstated divide There's another important clustering around these two conceptions of alignment. Peop...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Am I confused about the "malign universal prior" argument?, published by nostalgebraist on August 28, 2024 on LessWrong. In a 2016 blog post, Paul Christiano argued that the universal prior (hereafter "UP") may be "malign." His argument has received a lot of follow-up discussion, e.g. in Mark Xu's The Solomonoff Prior is Malign Charlie Steiner's The Solomonoff prior is malign. It's not a big deal. among other posts. This argument never made sense to me. The reason it doesn't make sense to me is pretty simple, but I haven't seen it mentioned explicitly in any of the ensuing discussion. This leaves me feeling like either I am misunderstanding the argument in a pretty fundamental way, or that there is a problem with the argument that has gotten little attention from the argument's critics (in which case I don't understand why). I would like to know which of these is the case, and correct my misunderstanding if it exists, hence this post. (Note: In 2018 I wrote a comment on the original post where I tried to state one of my objections to my argument, though I don't feel I expressed myself especially well there.) UP-using "universes" and simulatable "universes" The argument for malignity involves reasoning beings, instantiated in Turing machines (TMs), which try to influence the content of the UP in order to affect other beings who are making decisions using the UP. Famously, the UP is uncomputable. This means the TMs (and reasoning beings inside the TMs) will not be able to use[1] the UP themselves, or simulate anyone else using the UP. At least not if we take "using the UP" in a strict and literal sense. Thus, I am unsure how to interpret claims (which are common in presentations of the argument) about TMs "searching for universes where the UP is used" or the like. For example, from Mark Xu's "The Solomonoff Prior is Malign": In particular, this suggests a good strategy for consequentialists: find a universe that is using a version of the Solomonoff prior that has a very short description of the particular universe the consequentialists find themselves in. Or, from Christiano's original post: So the first step is getting our foot in the door - having control over the parts of the universal prior that are being used to make important decisions. This means looking across the universes we care about, and searching for spots within those universe where someone is using the universal prior to make important decisions. In particular, we want to find places where someone is using a version of the universal prior that puts a lot of mass on the particular universe that we are living in, because those are the places where we have the most leverage. Then the strategy is to implement a distribution over all of those spots, weighted by something like their importance to us (times the fraction of mass they give to the particular universe we are in and the particular channel we are using). That is, we pick one of those spots at random and then read off our subjective distribution over the sequence of bits that will be observed at that spot (which is likely to involve running actual simulations). What exactly are these "universes" that are being searched over? We have two options: 1. They are not computable universes. They permit hypercomputation that can leverage the "actual" UP, in its full uncomputable glory, without approximation. 2. They are computible universes. Thus the UP cannot be used in them. But maybe there is some computible thing that resembles or approximates the UP, and gets used in these universes. Option 1 seems hard to square with the talk about TMs "searching for" universes or "simulating" universes. A TM can't do such things to the universes of option 1. Hence, the argument is presumably about option 2. That is, although we are trying to reason about the content of...

Join Christiano Green in this deeply personal and informative solo episode of The Queer Quest Podcast. Christiano shares his journey with substance abuse, shedding light on the challenges and triumphs he faced along the way. Dive into the latest statistics revealing the disproportionate impact of substance abuse on the LGBTQIA+ community and understand the crucial differences between substance abuse and addiction. We'll explore the societal pressures, mental health issues, and unique stressors that contribute to higher rates of substance use in the queer community. Learn about the historical context, the role of intersectionality, and how art and expression can aid in recovery. Discover the ongoing advocacy efforts and resources available for those in need. Whether you're personally affected by substance abuse, know someone who is, or simply want to be an ally, this episode is packed with insights and encouragement. Tune in for a heartfelt discussion that aims to foster understanding, hope, and community support.

Join us on this profound episode of The Queer Quest Podcast as Christiano Green delves into the spiritual dimensions of the queer coming out experience. In "Embracing Your Truth: Spiritual Insights on the Queer Coming Out Journey," Christiano shares his personal narrative and unpacks the transformative power of embracing one's queer identity. This episode is an intimate exploration of the trials and triumphs on the path to spiritual self-realization within the queer community. What You'll Discover: Personal Reflections: Hear Christiano recount the pivotal moments of his own coming out journey, and the spiritual awakening that followed. Overcoming Challenges: Learn how facing societal and personal hurdles can catalyze profound spiritual growth and resilience. Healing and Forgiveness: Gain insights into the healing practices that can mend the spiritual scars left by rejection and misunderstanding. Community Impact: Discover the crucial role that a supportive queer community plays in individual spiritual journeys. Practical Spiritual Practices: Get actionable advice on practices that foster spiritual well-being and connection to one's true self. This episode is not just a narrative but a resource, offering guidance and support to those navigating their own coming out journey, or anyone seeking to deepen their understanding of spiritual growth in the queer context. Whether you're seeking solace, inspiration, or companionship on your journey, this conversation aims to light the way. Tune in and let Christiano help you navigate the beautiful, complex journey of spiritual and personal discovery. Don't forget to subscribe and share this episode with anyone who might find it enlightening! --- Send in a voice message: https://podcasters.spotify.com/pod/show/queer-quest/message

Christian O'Connell – the radio presenter who has opened up about his crippling panic attacks, life changing therapy and the danger zone of men's mental health – shares his three crisis comforts.  Full episode https://www.crisiswhatcrisis.com/podcasts/christian-oconnell-on-panic-attacks-a-professional-unravelling-and-how-story-telling-saved-him/  Links    https://christianoconnell.com/  Stream/buy ‘Allies' by Some Velvet Morning: https://ampl.ink/qp6bm    Some Velvet Morning Website: www.somevelvetmorning.co.uk      Your Daily Practice: Sleep by Myndstream: https://open.spotify.com/track/5OX9XgJufFz9g63o2Dv2i5?si=b2f9397c92084682    Host – Andy Coulson   CWC team: Jane Sankey, Louise Difford, Mabel Pickering   With special thanks to Ioana Barbu and the brilliant people at Global      For all PR and guest approaches please contact – podcast@coulsonpartners.com

Radio presenter Christian O'Connell talks to Andy about his crippling panic attacks, life changing therapy and the danger zone of men's mental health. As a young teenager Christian was hit with anxiety - beginning, as he puts it ‘as a whisper' - until it forced him off air … leading him to leave Britain's most popular breakfast show and risk it all on a life in Australia.  -------------------------------------------------------------------------------------------------------------------From working as a dustman, being fired from hospital radio and working in telesales - Christian O'Connell went on to host the Number 1 breakfast show on Absolute Radio and pick up 11 Sony Radio Academy Gold Awards. He was at the peak of his career, yet behind the scenes he was at his lowest point.  In 2018, after suffering a series of acute panic attacks, he finally sought help with the support of his wife, and then took a huge risk and uprooted his family to the other side of the world. This move to Australia however was met with more challenges of its own as Christian struggled to be accepted.  He talks candidly and with real emotion about how he had to build a new life, take care of his family and reach another peak in his career. Christian is arguably a national treasure both here and Down Under – his is a story of remarkable resilience and one we can all learn from.  Links    https://christianoconnell.com/  Stream/buy ‘Allies' by Some Velvet Morning: https://ampl.ink/qp6bm    Some Velvet Morning Website: www.somevelvetmorning.co.uk      Your Daily Practice: Sleep by Myndstream: https://open.spotify.com/track/5OX9XgJufFz9g63o2Dv2i5?si=b2f9397c92084682    Host – Andy Coulson   CWC team: Jane Sankey, Louise Difford, Mabel Pickering   With special thanks to Ioana Barbu and the brilliant people at Global      For all PR and guest approaches please contact – podcast@coulsonpartners.com   

In this solo episode of 'The Queer Quest Podcast,' join Christiano Green as they dive deep into the heart and soul of Queer Quest.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: My AI Model Delta Compared To Christiano, published by johnswentworth on June 12, 2024 on LessWrong. Preamble: Delta vs Crux This section is redundant if you already read My AI Model Delta Compared To Yudkowsky. I don't natively think in terms of cruxes. But there's a similar concept which is more natural for me, which I'll call a delta. Imagine that you and I each model the world (or some part of it) as implementing some program. Very oversimplified example: if I learn that e.g. it's cloudy today, that means the "weather" variable in my program at a particular time[1] takes on the value "cloudy". Now, suppose your program and my program are exactly the same, except that somewhere in there I think a certain parameter has value 5 and you think it has value 0.3. Even though our programs differ in only that one little spot, we might still expect very different values of lots of variables during execution - in other words, we might have very different beliefs about lots of stuff in the world. If your model and my model differ in that way, and we're trying to discuss our different beliefs, then the obvious useful thing-to-do is figure out where that one-parameter difference is. That's a delta: one or a few relatively "small"/local differences in belief, which when propagated through our models account for most of the differences in our beliefs. For those familiar with Pearl-style causal models: think of a delta as one or a few do() operations which suffice to make my model basically match somebody else's model, or vice versa. This post is about my current best guesses at the delta between my AI models and Paul Christiano's AI models. When I apply the delta outlined here to my models, and propagate the implications, my models mostly look like Paul's as far as I can tell. That said, note that this is not an attempt to pass Paul's Intellectual Turing Test; I'll still be using my own usual frames. My AI Model Delta Compared To Christiano Best guess: Paul thinks that verifying solutions to problems is generally "easy" in some sense. He's sometimes summarized this as " verification is easier than generation", but I think his underlying intuition is somewhat stronger than that. What do my models look like if I propagate that delta? Well, it implies that delegation is fundamentally viable in some deep, general sense. That propagates into a huge difference in worldviews. Like, I walk around my house and look at all the random goods I've paid for - the keyboard and monitor I'm using right now, a stack of books, a tupperware, waterbottle, flip-flops, carpet, desk and chair, refrigerator, sink, etc. Under my models, if I pick one of these objects at random and do a deep dive researching that object, it will usually turn out to be bad in ways which were either nonobvious or nonsalient to me, but unambiguously make my life worse and would unambiguously have been worth-to-me the cost to make better. But because the badness is nonobvious/nonsalient, it doesn't influence my decision-to-buy, and therefore companies producing the good are incentivized not to spend the effort to make it better. It's a failure of ease of verification: because I don't know what to pay attention to, I can't easily notice the ways in which the product is bad. (For a more game-theoretic angle, see When Hindsight Isn't 20/20.) On (my model of) Paul's worldview, that sort of thing is rare; at most it's the exception to the rule. On my worldview, it's the norm for most goods most of the time. See e.g. the whole air conditioner episode for us debating the badness of single-hose portable air conditioners specifically, along with a large sidebar on the badness of portable air conditioner energy ratings. How does the ease-of-verification delta propagate to AI? Well, most obviously, Paul expects AI to go well mostly via ...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: My AI Model Delta Compared To Yudkowsky, published by johnswentworth on June 10, 2024 on LessWrong. Preamble: Delta vs Crux I don't natively think in terms of cruxes. But there's a similar concept which is more natural for me, which I'll call a delta. Imagine that you and I each model the world (or some part of it) as implementing some program. Very oversimplified example: if I learn that e.g. it's cloudy today, that means the "weather" variable in my program at a particular time[1] takes on the value "cloudy". Now, suppose your program and my program are exactly the same, except that somewhere in there I think a certain parameter has value 5 and you think it has value 0.3. Even though our programs differ in only that one little spot, we might still expect very different values of lots of variables during execution - in other words, we might have very different beliefs about lots of stuff in the world. If your model and my model differ in that way, and we're trying to discuss our different beliefs, then the obvious useful thing-to-do is figure out where that one-parameter difference is. That's a delta: one or a few relatively "small"/local differences in belief, which when propagated through our models account for most of the differences in our beliefs. For those familiar with Pearl-style causal models: think of a delta as one or a few do() operations which suffice to make my model basically match somebody else's model, or vice versa. This post is about my current best guesses at the delta between my AI models and Yudkowsky's AI models. When I apply the delta outlined here to my models, and propagate the implications, my models basically look like Yukowsky's as far as I can tell. This post might turn into a sequence if there's interest; I already have another one written for Christiano, and people are welcome to suggest others they'd be interested in. My AI Model Delta Compared To Yudkowsky Best guess: Eliezer basically rejects the natural abstraction hypothesis. He mostly expects AI to use internal ontologies fundamentally alien to the ontologies of humans, at least in the places which matter. Lethality #33 lays it out succinctly: 33. The AI does not think like you do, the AI doesn't have thoughts built up from the same concepts you use, it is utterly alien on a staggering scale. Nobody knows what the hell GPT-3 is thinking, not only because the matrices are opaque, but because the stuff within that opaque container is, very likely, incredibly alien - nothing that would translate well into comprehensible human thinking, even if we could see past the giant wall of floating-point numbers to what lay behind. What do my models look like if I propagate that delta? In worlds where natural abstraction basically fails, we are thoroughly and utterly fucked, and a 99% probability of doom strikes me as entirely reasonable and justified. Here's one oversimplified doom argument/story in a world where natural abstraction fails hard: 1. Humanity is going to build superhuman goal-optimizing agents. ('Cause, like, obviously somebody's going to do that, there's no shortage of capabilities researchers loudly advertising that they're aiming to do that exact thing.) These will be so vastly more powerful than humans that we have basically-zero bargaining power except insofar as AIs are aligned to our interests. 2. We're assuming natural abstraction basically fails, so those AI systems will have fundamentally alien internal ontologies. For purposes of this overcompressed version of the argument, we'll assume a very extreme failure of natural abstraction, such that human concepts cannot be faithfully and robustly translated into the system's internal ontology at all. (For instance, maybe a faithful and robust translation would be so long in the system's "internal language" that the transla...

This week we have a father & daughter combo with Christiano and Francisca Van Zeller. The Van Zellers left the Netherlands for political and religious reasons, came to Portugal and settled in Porto as port wine merchants, and have been bound to the Douro and the production of wines since 1620. Cristiano and Francisca are nowadays at the wheel of Van Zellers & Co., a family business that has been going on for 15 generations. Listen in to learn what they love about Portuguese wines, especially Port Wine, and enjoy living the simple life in the Douro region of Portugal.FOLLOW OUR GUESTSVan Zellers & Co WebsiteVan Zellers & Co on InstagramVan Zellers & Co on FacebookVan Zellers & Co on LinkedInABOUT PORTUGAL THE SIMPLE LIFE PODCAST:   "Portugal - The simple life”, an insider's perspective to Portugal.  We already know about Portugal's fantastic weather, food and people. In this podcast, we go deeper to meet the people who make this country so wonderful.Dylan, who has made his life in Portugal, shares an insider's perspective on what makes Portugal the unique, beautiful and fantastic country it is. Join him and his guests weekly as they shed light on the incredible people, culture, history and lifestyle that make Portugal so appealing. A country where everyone feels like they belong.    Don't forget to subscribe to our Podcast to receive more stories about living and moving to Portugal!   SPONSOR:Portugal Realty, a Leisure Launch group company, sponsors this episode. 

Why is Australian radio so risqué? Matt Deegan welcomes Craig Bruce, who launched Kyle and Jackie O's First Breakfast Show before becoming head of content for SCA and a radio consultant.We also talk to James Corden's podcast producer, Geoff Jein, who started out at BBC Radio 2 before moving to Perth to work on breakfast radio.Standby for lessons for broadcasters everywhere from how producers manage big talen, the 10 year deal to lock them in, how Britain's Christian O'Connell wowed Australians and bagged a $5 million a year deal.Content warning: strong language Get bonus content on Patreon Hosted on Acast. See acast.com/privacy for more information.

Tune in to a very special episode of 'The Queer Quest Podcast,' where we bring you an exclusive crossover from the Thrive Network. In this poignant discussion, Christiano Green, our cherished host, sits down with Andro to explore the transformative and healing power of love within the queer community.

In an especially intimate and inspiring episode of 'The Queer Quest Podcast,' we take a moment to pause, reflect, and envision the future we desire and deserve as queer individuals.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AI #60: Oh the Humanity, published by Zvi on April 18, 2024 on LessWrong. Many things this week did not go as planned. Humane AI premiered its AI pin. Reviewers noticed it was, at best, not ready. Devin turns out to have not been entirely forthright with its demos. OpenAI fired two employees who had been on its superalignment team, Leopold Aschenbrenner and Pavel Izmailov for allegedly leaking information, and also more troubliningly lost Daniel Kokotajlo, who expects AGI very soon, does not expect it to by default go well, and says he quit 'due to losing confidence that [OpenAI] would behave responsibly around the time of AGI.' That's not good. Nor is the Gab system prompt, although that is not a surprise. And several more. On the plus side, my 80,000 Hours podcast finally saw the light of day, and Ezra Klein had an excellent (although troubling) podcast with Dario Amodei. And we got the usual mix of incremental useful improvements and other nice touches. Table of Contents Introduction. Table of Contents. Language Models Offer Mundane Utility. Ask all your stupid questions. Language Models Don't Offer Mundane Utility. That won't stop social media. Oh the Humanity. It will, however, stop the Humane AI pin, at least for now. GPT-4 Real This Time. The new version continues to look slightly better. Fun With Image Generation. There is remarkably little porn of it. Deepfaketown and Botpocalypse Soon. Audio plus face equals talking head. Devin in the Details. To what extent was the Devin demo a fake? Another Supposed System Prompt. The gift of Gab. Not what we wanted. They Took Our Jobs. A model of firm employment as a function of productivity. Introducing. The quest to make context no longer be that which is scarce. In Other AI News. Respecting and disrespecting the rules of the game. Quiet Speculations. Spending some time wondering whether you should. The Quest for Sane Regulations. Senators get serious, Christiano is appointed. The Week in Audio. I spend 3.5 of my 80,000 hours, and several more. Rhetorical Innovation. Words that do not on reflection bring comfort. Don't Be That Guy. Also known as the only law of morality. Aligning a Smarter Than Human Intelligence is Difficult. Subproblems anyone? Please Speak Directly Into the Microphone. Thanks, everyone. People Are Worried About AI Killing Everyone. They are no longer at OpenAI. Other People Are Not As Worried About AI Killing Everyone. Mundane visions. The Lighter Side. The art of fixing it. Language Models Offer Mundane Utility The best use of LLMs continues to be 'ask stupid questions.' Ashwin Sharma: reading zen and the art of motorcycle maintenance changed the way I looked at the inner workings of my mind. It was like unlocking a secret level of a video game. what are you reading today? Tom Crean: Tried to read Zen… as a teenager and felt disoriented by it. I kept wondering who "Phaedrus" was. But I liked the general atmosphere of freedom. The philosophy went over my head. Now I'm reading Akenfield by Ronald Blythe. A portrait of a Suffolk Village in the 1960s. Ashwin Sharma: use GPT to help analyse the sections you're stuck on. Seriously, try it again and i promise you it'll be worth it. Joe Weisenthal: I've found this to be a great ChatGPT use case. Understanding terms in context while I'm reading. When I was a kid, my dad told me when reading to immediately stop and grab a dictionary every time I got to a word I didn't understand. Not really feasible. But AI solves this well. It's still a bit cumbersome, because with kindle or physical, no quick way to copy/paste a section into an AI or just ask the book what it means. But even with those hurdles, I've found the tools to be a great reading augment. Patrick McKenzie: It's surprisingly reliable to just point phone camera at screen and then ask questions about t...

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Paul Christiano named as US AI Safety Institute Head of AI Safety, published by Joel Burget on April 16, 2024 on LessWrong. U.S. Secretary of Commerce Gina Raimondo announced today additional members of the executive leadership team of the U.S. AI Safety Institute (AISI), which is housed at the National Institute of Standards and Technology (NIST). Raimondo named Paul Christiano as Head of AI Safety, Adam Russell as Chief Vision Officer, Mara Campbell as Acting Chief Operating Officer and Chief of Staff, Rob Reich as Senior Advisor, and Mark Latonero as Head of International Engagement. They will join AISI Director Elizabeth Kelly and Chief Technology Officer Elham Tabassi, who were announced in February. The AISI was established within NIST at the direction of President Biden, including to support the responsibilities assigned to the Department of Commerce under the President's landmark Executive Order. Paul Christiano, Head of AI Safety, will design and conduct tests of frontier AI models, focusing on model evaluations for capabilities of national security concern. Christiano will also contribute guidance on conducting these evaluations, as well as on the implementation of risk mitigations to enhance frontier model safety and security. Christiano founded the Alignment Research Center, a non-profit research organization that seeks to align future machine learning systems with human interests by furthering theoretical research. He also launched a leading initiative to conduct third-party evaluations of frontier models, now housed at Model Evaluation and Threat Research (METR). He previously ran the language model alignment team at OpenAI, where he pioneered work on reinforcement learning from human feedback (RLHF), a foundational technical AI safety technique. He holds a PhD in computer science from the University of California, Berkeley, and a B.S. in mathematics from the Massachusetts Institute of Technology. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: U.S. Commerce Secretary Gina Raimondo Announces Expansion of U.S. AI Safety Institute Leadership Team [and Paul Christiano update], published by Phib on April 16, 2024 on The Effective Altruism Forum. U.S. Secretary of Commerce Gina Raimondo announced today additional members of the executive leadership team of the U.S. AI Safety Institute (AISI), which is housed at the National Institute of Standards and Technology (NIST). Raimondo named Paul Christiano as Head of AI Safety, Adam Russell as Chief Vision Officer, Mara Campbell as Acting Chief Operating Officer and Chief of Staff, Rob Reich as Senior Advisor, and Mark Latonero as Head of International Engagement. They will join AISI Director Elizabeth Kelly and Chief Technology Officer Elham Tabassi, who were announced in February. The AISI was established within NIST at the direction of President Biden, including to support the responsibilities assigned to the Department of Commerce under the President's landmark Executive Order. ... Paul Christiano, Head of AI Safety, will design and conduct tests of frontier AI models, focusing on model evaluations for capabilities of national security concern. Christiano will also contribute guidance on conducting these evaluations, as well as on the implementation of risk mitigations to enhance frontier model safety and security. Christiano founded the Alignment Research Center, a non-profit research organization that seeks to align future machine learning systems with human interests by furthering theoretical research. He also launched a leading initiative to conduct third-party evaluations of frontier models, now housed at Model Evaluation and Threat Research (METR). He previously ran the language model alignment team at OpenAI, where he pioneered work on reinforcement learning from human feedback (RLHF), a foundational technical AI safety technique. He holds a PhD in computer science from the University of California, Berkeley, and a B.S. in mathematics from the Massachusetts Institute of Technology. Following up from previous news post: https://forum.effectivealtruism.org/posts/9QLJgRMmnD6adzvAE/nist-staffers-revolt-against-expected-appointment-of Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

While Christian is away on his break, we're going through some of our favourite bits from the show so far.  We can't go past some of these calls we got off the back of Christian's story about coming to this wife's rescue when she ran out of fuel. Do you do run the fuel gauntlet? Got a funny story about running out of petrol? Email us christian@christianoconnell.com.au  See omnystudio.com/listener for privacy information.

While Christian is away on his break, we're going through some of our favourite bits from the show so far! Like this heated debate about CROCS. Jack yearns for the day when these plastic shoes will be widely accepted in society. Have your say! Email us at: christian@christianoconnell.com.auSee omnystudio.com/listener for privacy information.

Dive deep into the heart of queer relationships and intimacy with Christiano Green, your guide and confidant on "The Queer Quest Podcast." In this enlightening solo episode, Christiano peels back the layers of what it means to navigate the beautiful, complex world of queer love and connection. With vulnerability and insight, we explore the diversity within queer relationships, the cornerstone of effective communication, and the multifaceted nature of intimacy beyond the physical. This episode is an essential listen for anyone yearning to understand the dynamics of queer relationships better, from the initial spark to nurturing a deep, enduring bond amid societal pressures. Christiano shares personal anecdotes, practical communication strategies, and ways to foster intimacy, offering a beacon of understanding and support. We also tackle how to confront challenges together, transforming obstacles into opportunities for growth and deeper connection. Whether you're navigating the dating scene, deep in a long-term partnership, or simply curious about the unique experiences of queer intimacy, this episode provides a compassionate, comprehensive guide. Join us on a journey to elevate your relationship and intimacy understanding, equipped with tips, empathy, and the affirmation that love, in its essence, knows no bounds. Tune in, feel seen, and be inspired as we embark on this quest together. Don't forget to subscribe for more insights, share your own stories, and become part of the vibrant Queer Quest community. Your adventure in love and self-discovery awaits.

In 2018, Christian O Connell was one of the most successful people in British Radio, he was on top of his world, and then…the panic attacks began. The attacks got so bad and so regular that he couldn't perform live and it nearly cost him his career. In this episode, we revisit Christian's experience, the physical and mental toll of his panic attacks as well as the societal pressures and expectations he felt as a man - his need to appear strong and provide for his family. And we hear how he eventually sought for help and started rebuilding his life. To hear the full episode with Christian, follow this link: https://link.chtbl.com/UunNrODW You can purchase a copy of Christian's book here: https://bit.ly/3IKdFqI  See omnystudio.com/listener for privacy information.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: NIST staffers revolt against expected appointment of 'effective altruist' AI researcher to US AI Safety Institute, published by Phib on March 9, 2024 on The Effective Altruism Forum. "The appointment of Christiano, which was said to come directly from Secretary of Commerce Gina Raimondo (NIST is an agency under the US Department of Commerce) has sparked outrage among NIST employees who fear that Christiano's association with EA and longtermism could compromise the institute's objectivity and integrity." "The AISI was established in November 2023 to "support the responsibilities assigned to the Department of Commerce" under the AI Executive Order. Earlier today, US Senate Majority Leader Chuck Schumer (D-NY) announced that the NIST will receive up to $10 million to establish the US AI Safety Institute." Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org

In this heart-to-heart episode of 'The Queer Quest Podcast,' your host, Christiano Green, peels back the curtain on their own technicolor journey.

In 2023 we did a few Fundamentals episodes covering Benchmarks 101, Datasets 101, FlashAttention, and Transformers Math, and it turns out those were some of your evergreen favorites! So we are experimenting with more educational/survey content in the mix alongside our regular founder and event coverage. Pls request more!We have a new calendar for events; join to be notified of upcoming things in 2024!Today we visit the shoggoth mask factory: how do transformer models go from trawling a deeply learned latent space for next-token prediction to a helpful, honest, harmless chat assistant? Our guest “lecturer” today is ; you might know him from his prolific online writing on and Twitter, or from his previous work leading RLHF at HuggingFace and now at the Allen Institute for AI (AI2) which recently released the open source GPT3.5-class Tulu 2 model which was trained with DPO. He's widely considered one of the most knowledgeable people on RLHF and RLAIF. He recently gave an “RLHF 201” lecture at Stanford, so we invited him on the show to re-record it for everyone to enjoy! You can find the full slides here, which you can use as reference through this episode. Full video with synced slidesFor audio-only listeners, this episode comes with slide presentation along our discussion. You can find it on our YouTube (like, subscribe, tell a friend, et al).Theoretical foundations of RLHFThe foundation and assumptions that go into RLHF go back all the way to Aristotle (and you can find guidance for further research in the slide below) but there are two key concepts that will be helpful in thinking through this topic and LLMs in general:* Von Neumann–Morgenstern utility theorem: you can dive into the math here, but the TLDR is that when humans make decision there's usually a “maximum utility” function that measures what the best decision would be; the fact that this function exists, makes it possible for RLHF to model human preferences and decision making.* Bradley-Terry model: given two items A and B from a population, you can model the probability that A will be preferred to B (or vice-versa). In our world, A and B are usually two outputs from an LLM (or at the lowest level, the next token). It turns out that from this minimal set of assumptions, you can build up the mathematical foundations supporting the modern RLHF paradigm!The RLHF loopOne important point Nathan makes is that "for many tasks we want to solve, evaluation of outcomes is easier than producing the correct behavior". For example, it might be difficult for you to write a poem, but it's really easy to say if you like or dislike a poem someone else wrote. Going back to the Bradley-Terry Model we mentioned, the core idea behind RLHF is that when given two outputs from a model, you will be able to say which of the two you prefer, and we'll then re-encode that preference into the model.An important point that Nathan mentions is that when you use these preferences to change model behavior "it doesn't mean that the model believes these things. It's just trained to prioritize these things". When you have preference for a model to not return instructions on how to write a computer virus for example, you're not erasing the weights that have that knowledge, but you're simply making it hard for that information to surface by prioritizing answers that don't return it. We'll talk more about this in our future Fine Tuning 101 episode as we break down how information is stored in models and how fine-tuning affects it.At a high level, the loop looks something like this:For many RLHF use cases today, we can assume the model we're training is already instruction-tuned for chat or whatever behavior the model is looking to achieve. In the "Reward Model & Other Infrastructure" we have multiple pieces:Reward + Preference ModelThe reward model is trying to signal to the model how much it should change its behavior based on the human preference, subject to a KL constraint. The preference model itself scores the pairwise preferences from the same prompt (worked better than scalar rewards).One way to think about it is that the reward model tells the model how big of a change this new preference should make in the behavior in absolute terms, while the preference model calculates how big of a difference there is between the two outputs in relative terms. A lot of this derives from John Schulman's work on PPO:We recommend watching him talk about it in the video above, and also Nathan's pseudocode distillation of the process:Feedback InterfacesUnlike the "thumbs up/down" buttons in ChatGPT, data annotation from labelers is much more thorough and has many axis of judgement. At a simple level, the LLM generates two outputs, A and B, for a given human conversation. It then asks the labeler to use a Likert scale to score which one it preferred, and by how much:Through the labeling process, there are many other ways to judge a generation:We then use all of this data to train a model from the preference pairs we have. We start from the base instruction-tuned model, and then run training in which the loss of our gradient descent is the difference between the good and the bad prompt.Constitutional AI (RLAIF, model-as-judge)As these models have gotten more sophisticated, people started asking the question of whether or not humans are actually a better judge of harmfulness, bias, etc, especially at the current price of data labeling. Anthropic's work on the "Constitutional AI" paper is using models to judge models. This is part of a broader "RLAIF" space: Reinforcement Learning from AI Feedback.By using a "constitution" that the model has to follow, you are able to generate fine-tuning data for a new model that will be RLHF'd on this constitution principles. The RLHF model will then be able to judge outputs of models to make sure that they follow its principles:Emerging ResearchRLHF is still a nascent field, and there are a lot of different research directions teams are taking; some of the newest and most promising / hyped ones:* Rejection sampling / Best of N Sampling: the core idea here is that rather than just scoring pairwise generations, you are generating a lot more outputs (= more inference cost), score them all with your reward model and then pick the top N results. LLaMA2 used this approach, amongst many others.* Process reward models: in Chain of Thought generation, scoring each step in the chain and treating it like its own state rather than just scoring the full output. This is most effective in fields like math that inherently require step-by-step reasoning.* Direct Preference Optimization (DPO): We covered DPO in our NeurIPS Best Papers recap, and Nathan has a whole blog post on this; DPO isn't technically RLHF as it doesn't have the RL part, but it's the “GPU Poor” version of it. Mistral-Instruct was a DPO model, as do Intel's Neural Chat and StableLM Zephyr. Expect to see a lot more variants in 2024 given how “easy” this was.* Superalignment: OpenAI launched research on weak-to-strong generalization which we briefly discuss at the 1hr mark.Note: Nathan also followed up this post with RLHF resources from his and peers' work:Show Notes* Full RLHF Slides* Interconnects* Retort (podcast)* von Neumann-Morgenstern utility theorem* Bradley-Terry model (pairwise preferences model)* Constitutional AI* Tamer (2008 paper by Bradley Knox and Peter Stone)* Paul Christiano et al. RLHF paper* InstructGPT* Eureka by Jim Fan* ByteDance / OpenAI lawsuit* AlpacaEval* MTBench* TruthfulQA (evaluation tool)* Self-Instruct Paper* Open Assistant* Louis Castricato* Nazneen Rajani* Tulu (DPO model from the Allen Institute)Timestamps* [00:00:00] Introductions and background on the lecture origins* [00:05:17] History of RL and its applications* [00:10:09] Intellectual history of RLHF* [00:13:47] RLHF for decision-making and pre-deep RL vs deep RL* [00:20:19] Initial papers and intuitions around RLHF* [00:27:57] The three phases of RLHF* [00:31:09] Overfitting issues* [00:34:47] How preferences get defined* [00:40:35] Ballpark on LLaMA2 costs* [00:42:50] Synthetic data for training* [00:47:25] Technical deep dive in the RLHF process* [00:54:34] Projection / best event sampling* [00:57:49] Constitutional AI* [01:04:13] DPO* [01:08:54] What's the Allen Institute for AI?* [01:13:43] Benchmarks and models comparisonsTranscriptAlessio [00:00:00]: Hey everyone, welcome to the Latent Space podcast. This is Alessio, partner and CTO in Residence at Decibel Partners, and I'm joined by my co-host Swyx, founder of Smol AI.Swyx [00:00:15]: Hey, and today we have Dr. Nathan Lambert in the house. Welcome.Nathan [00:00:18]: Thanks guys.Swyx [00:00:19]: You didn't have to come too far. You got your PhD in Berkeley, and it seems like you've lived there most of the time in recent years. You worked on robotics and model-based reinforcement learning on your PhD, and you also interned at FAIR and DeepMind. You bootstrapped the RLHF team at Hugging Face, and you recently joined the Allen Institute as a research scientist. So that's your quick bio. What should people know about you that maybe is not super obvious about you on New LinkedIn?Nathan [00:00:43]: I stay sane in various insane sport and ultra-endurance sport activities that I do.Swyx [00:00:50]: What's an ultra-endurance sport activity?Nathan [00:00:52]: Long-distance trail running or gravel biking. Try to unplug sometimes, although it's harder these days. Yeah.Swyx [00:00:59]: Well, you know, just the Bay Area is just really good for that stuff, right?Nathan [00:01:02]: Oh, yeah. You can't beat it. I have a trailhead like 1.2 miles from my house, which is pretty unmatchable in any other urban area.Swyx [00:01:11]: Pretty excellent. You also have an incredible blog, Interconnects, which I'm a fan of. And I also just recently discovered that you have a new podcast, Retort.Nathan [00:01:20]: Yeah, we do. I've been writing for a while, and I feel like I've finally started to write things that are understandable and fun. After a few years lost in the wilderness, if you ask some of my friends that I made read the earlier blogs, they're like, oh, this is yikes, but it's coming along. And the podcast is with my friend Tom, and we just kind of like riff on what's actually happening on AI and not really do news recaps, but just what it all means and have a more critical perspective on the things that really are kind of funny, but still very serious happening in the world of machine learning.Swyx [00:01:52]: Yeah. Awesome. So let's talk about your work. What would you highlight as your greatest hits so far on Interconnects, at least?Nathan [00:01:59]: So the ones that are most popular are timely and or opinion pieces. So the first real breakout piece was when April and I also just wrote down the thing that everyone in AI was feeling, which is we're all feeling stressed, that we're going to get scooped, and that we're overworked, which is behind the curtain, what it feels to work in AI. And then a similar one, which we might touch on later in this, was about my recent job search, which wasn't the first time I wrote a job search post. People always love that stuff. It's so open. I mean, it's easy for me to do in a way that it's very on-brand, and it's very helpful. I understand that until you've done it, it's hard to share this information. And then the other popular ones are various model training techniques or fine tuning. There's an early one on RLHF, which is, this stuff is all just like when I figure it out in my brain. So I wrote an article that's like how RLHF actually works, which is just the intuitions that I had put together in the summer about RLHF, and that was pretty well. And then I opportunistically wrote about QSTAR, which I hate that you have to do it, but it is pretty funny. From a literature perspective, I'm like, open AI publishes on work that is very related to mathematical reasoning. So it's like, oh, you just poke a little around what they've already published, and it seems pretty reasonable. But we don't know. They probably just got like a moderate bump on one of their benchmarks, and then everyone lost their minds. It doesn't really matter.Swyx [00:03:15]: You're like, this is why Sam Altman was fired. I don't know. Anyway, we're here to talk about RLHF 101. You did a presentation, and I think you expressed some desire to rerecord it. And that's why I reached out on Twitter saying, like, why not rerecord it with us, and then we can ask questions and talk about it. Yeah, sounds good.Nathan [00:03:30]: I try to do it every six or 12 months is my estimated cadence, just to refine the ways that I say things. And people will see that we don't know that much more, but we have a bit of better way of saying what we don't know.Swyx [00:03:43]: Awesome. We can dive right in. I don't know if there's any other topics that we want to lay out as groundwork.Alessio [00:03:48]: No, you have some awesome slides. So for people listening on podcast only, we're going to have the slides on our show notes, and then we're going to have a YouTube version where we run through everything together.Nathan [00:03:59]: Sounds good. Yeah. I think to start skipping a lot of the, like, what is a language model stuff, everyone knows that at this point. I think the quote from the Llama 2 paper is a great kind of tidbit on RLHF becoming like a real deal. There was some uncertainty earlier in the year about whether or not RLHF was really going to be important. I think it was not that surprising that it is. I mean, with recent models still using it, the signs were there, but the Llama 2 paper essentially reads like a bunch of NLP researchers that were skeptical and surprised. So the quote from the paper was, meanwhile, reinforcement learning known for its instability seemed a somewhat shadowy field for those in the NLP research community. However, reinforcement learning proved highly effective, particularly given its cost and time effectiveness. So you don't really know exactly what the costs and time that Meta is looking at, because they have a huge team and a pretty good amount of money here to release these Llama models. This is just the kind of thing that we're seeing now. I think any major company that wasn't doing RLHF is now realizing they have to have a team around this. At the same time, we don't have a lot of that in the open and research communities at the same scale. I think seeing that converge would be great, but it's still very early days. And the other thing on the slide is some of Anthropic's work, but everyone knows Anthropic is kind of the masters of this, and they have some of their own techniques that we're going to talk about later on, but that's kind of where we start.Alessio [00:05:17]: Can we do just a one-second RL version? So you come from a robotics background, which RL used to be, or maybe still is, state-of-the-art. And then now you're seeing a lot of LLM plus RL, so you have the gym fans, Eureka, you have MPU, which we had on the podcast when they started with RL. Now they're doing RL plus LLMs. Yeah. Any thoughts there on how we got here? Maybe how the pendulum will keep swinging?Nathan [00:05:46]: I really think RL is about a framing of viewing the world through trial and error learning and feedback, and really just one that's focused on thinking about decision-making and inputs in the world and how inputs have reactions. And in that, a lot of people come from a lot of different backgrounds, whether it's physics, electrical engineering, mechanical engineering. There are obviously computer scientists, but compared to other fields of CS, I do think it's a much more diverse background of people. My background was in electrical engineering and doing robotics and things like that. It really just changes the worldview. I think that reinforcement learning as it was back then, so to say, is really different. You're looking at these toy problems and the numbers are totally different, and everyone went kind of zero to one at scaling these things up, but people like Jim Phan and other people that were... You saw this transition in the decision transformer and papers and when people are trying to use transformers to do decision-making for things like offline RL, and I think that was kind of like the early days. But then once language models were so proven, it's like everyone is using this tool for their research. I think in the long run, it will still settle out, or RL will still be a field that people work on just because of these kind of fundamental things that I talked about. It's just viewing the whole problem formulation different than predicting text, and so there needs to be that separation. And the view of RL in language models is pretty contrived already, so it's not like we're doing real RL. I think the last slide that I have here is a way to make RLHF more like what people would think of with RL, so actually running things over time, but a weird lineage of tools that happen to get us to where we are, so that's why the name takes up so much space, but it could have gone a lot of different ways. Cool.Alessio [00:07:29]: We made it one slide before going on a tangent.Nathan [00:07:31]: Yeah, I mean, it's kind of related. This is a...Swyx [00:07:35]: Yeah, so we have a history of RL.Nathan [00:07:37]: Yeah, so to give the context, this paper really started because I have this more diverse background than some computer scientists, such as trying to understand what the difference of a cost function or a reward function and a preference function would be without going into all of the details. Costs are normally things that control theorists would work with in these kind of closed domains, and then reinforcement learning has always worked with rewards that's central to the formulation that we'll see, and then the idea was like, okay, we now are at preferences, and each step along the way there's kind of different assumptions that you're making. We'll get into these, and those assumptions are built on other fields of work. So that's what this slide is going to say, it's like RLHF, while directly building on tools from RL and language models, is really implicitly impacted and built on theories and philosophies spanning tons of human history. I think we cite Aristotle in this paper, which is fun. It's like going pre-BC, it's like 2,300 years old or something like that. So that's the reason to do this, I think. We kind of list some things in the paper about summarizing what different presumptions of RLHF could be. I think going through these is actually kind of funny. It's fun to talk about these, because they're kind of grab bags of things that you'll see return throughout this podcast that we're talking about it. The core thing of RLHF that, in order to be a believer in this, is that RL actually works. It's like, if you have a reward function, you can optimize it in some way and get a different performance out of it, and you could do this at scale, and you could do this in really complex environments, which is, I don't know how to do that in all the domains. I don't know how to exactly make chat GPT. So it's kind of, we'll overshadow everything. And then there's, go from something kind of obvious like that, and then you read the von Neumann-Morgenstern utility theorem, which is essentially an economic theory that says you can weight different probabilities of different people, which is a theoretical piece of work that is the foundation of utilitarianism, and trying to quantify preferences is crucial to doing any sort of RLHF. And if you look into this, all of these things, there's way more you could go into if you're interested in any of these. So this is kind of like grabbing a few random things, and then kind of similar to that is the Bradley-Terry model, which is the fancy name for the pairwise preferences that everyone is doing. And then all the things that are like, that Anthropic and OpenAI figured out that you can do, which is that you can aggregate preferences from a bunch of different people and different sources. And then when you actually do RLHF, you extract things from that data, and then you train a model that works somehow. And we don't know, there's a lot of complex links there, but if you want to be a believer in doing this at scale, these are the sorts of things that you have to accept as preconditions for doing RLHF. Yeah.Swyx [00:10:09]: You have a nice chart of like the sort of intellectual history of RLHF that we'll send people to refer to either in your paper or in the YouTube video for this podcast. But I like the other slide that you have on like the presumptions that you need to have for RLHF to work. You already mentioned some of those. Which one's underappreciated? Like, this is the first time I've come across the VNM Utility Theorem.Nathan [00:10:29]: Yeah, I know. This is what you get from working with people like to my co-host on the podcast, the rhetoric is that sociologist by training. So he knows all these things and like who the philosophers are that found these different things like utilitarianism. But there's a lot that goes into this. Like essentially there's even economic theories that like there's debate whether or not preferences exist at all. And there's like different types of math you can use with whether or not you actually can model preferences at all. So it's pretty obvious that RLHF is built on the math that thinks that you can actually model any human preference. But this is the sort of thing that's been debated for a long time. So all the work that's here is like, and people hear about in their AI classes. So like Jeremy Bentham, like hedonic calculus and all these things like these are the side of work where people assume that preferences can be measured. And this is like, I don't really know, like, this is what I kind of go on a rant and I say that in RLHF calling things a preference model is a little annoying because there's no inductive bias of what a preference is. It's like if you were to learn a robotic system and you learned a dynamics model, like hopefully that actually mirrors the world in some way of the dynamics. But with a preference model, it's like, Oh my God, I don't know what this model, like I don't know what chat GPT encodes as any sort of preference or what I would want it to be in a fair way. Anthropic has done more work on trying to write these things down. But even like if you look at Claude's constitution, like that doesn't mean the model believes these things. It's just trained to prioritize these things. And that's kind of what the later points I'm looking at, like what RLHF is doing and if it's actually like a repeatable process in the data and in the training, that's just unknown. And we have a long way to go before we understand what this is and the link between preference data and any notion of like writing down a specific value.Alessio [00:12:05]: The disconnect between more sociology work versus computer work already exists, or is it like a recent cross contamination? Because when we had Tri Dao on the podcast, he said FlashAttention came to be because at Hazy they have so much overlap between systems engineer and like deep learning engineers. Is it the same in this field?Nathan [00:12:26]: So I've gone to a couple of workshops for the populations of people who you'd want to include this like R. I think the reason why it's not really talked about is just because the RLHF techniques that people use were built in labs like OpenAI and DeepMind where there are some of these people. These places do a pretty good job of trying to get these people in the door when you compare them to like normal startups. But like they're not bringing in academics from economics, like social choice theory. There's just too much. Like the criticism of this paper that this is based on is like, oh, you're missing these things in RL or at least this decade of RL and it's like it would be literally be bigger than the Sutton and Barto book if you were to include everyone. So it's really hard to include everyone in a principled manner when you're designing this. It's just a good way to understand and improve the communication of what RLHF is and like what is a good reward model for society. It really probably comes down to what an individual wants and it'll probably motivate models to move more in that direction and just be a little bit better about the communication, which is a recurring theme and kind of my work is like I just get frustrated when people say things that don't really make sense, especially when it's going to manipulate individual's values or manipulate the general view of AI or anything like this. So that's kind of why RLHF is so interesting. It's very vague in what it's actually doing while the problem specification is very general.Swyx [00:13:42]: Shall we go to the, I guess, the diagram here on the reinforcement learning basics? Yeah.Nathan [00:13:47]: So reinforcement learning, I kind of mentioned this, it's a trial and error type of system. The diagram and the slides is really this classic thing where you have an agent interacting with an environment. So it's kind of this agent has some input to the environment, which is called the action. The environment returns a state and a reward and that repeats over time and the agent learns based on these states and these rewards that it's seeing and it should learn a policy that makes the rewards go up. That seems pretty simple than if you try to mentally map what this looks like in language, which is that like the language models don't make this easy. I think with the language model, it's very hard to define what an environment is. So if the language model is the policy and it's generating, it's like the environment should be a human, but setting up the infrastructure to take tens of thousands of prompts and generate them and then show them to a human and collect the human responses and then shove that into your training architecture is very far away from working. So we don't really have an environment. We just have a reward model that returns a reward and the state doesn't really exist when you look at it like an RL problem. What happens is the state is a prompt and then you do a completion and then you throw it away and you grab a new prompt. We're really in as an RL researcher, you would think of this as being like you take a state, you get some completion from it and then you look at what that is and you keep kind of iterating on it and all of that isn't here, which is why you'll hear RLHF referred to as bandits problem, which is kind of like you choose one action and then you watch the dynamics play out. There's many more debates that you can have in this. If you get the right RL people in the room, then kind of like this is an RL even when you zoom into what RLHF is doing.Alessio [00:15:22]: Does this change as you think about a chain of thought reasoning and things like that? Like does the state become part of the chain that you're going through?Nathan [00:15:29]: There's work that I've mentioned on one slide called process reward models that essentially rewards each step in the chain of thought reasoning. It doesn't really give the part of interaction, but it does make it a little bit more fine grained where you can think about like calling it at least you have many states from your initial state. That formulation I don't think people have fully settled on. I think there's a bunch of great work out there, like even OpenAI is releasing a lot of this and let's verify step by step is there pretty great paper on the matter. I think in the next year that'll probably get made more concrete by the community on like if you can easily draw out like if chain of thought reasoning is more like RL, we can talk about that more later. That's a kind of a more advanced topic than we probably should spend all the time on.Swyx [00:16:13]: RLHF for decision making. You have a slide here that compares pre-deep RL versus deep RL.Nathan [00:16:19]: This is getting into the history of things, which is showing that the work that people are using now really came from well outside of NLP and it came before deep learning was big. Next up from this paper, Tamer, which is from 2008. Some names that are still really relevant in kind of human centric RL, Bradley Knox and Peter Stone. If you have an agent take an action, you would just have a human give a score from zero to one as a reward rather than having a reward function. And then with that classifier, you can do something with a policy that learns to take actions to maximize that reward. It's a pretty simple setup. It works in simple domains. And then the reason why this is interesting is you compare it to the paper that everyone knows, which is this Paul Christiano et al. Deep Reinforced Learning from Human Preferences paper, which is where they showed that learning from human preferences, you can solve like the basic RL tasks at the time. So various control problems and simulation and this kind of like human preferences approach had higher rewards in some environments than if you just threw RL at the environment that returned a reward. So the preferences thing was you took two trajectories. So in this case, it was like complete trajectories of the agent and the human was labeling which one is better. You can see how this kind of comes to be like the pairwise preferences that are used today that we'll talk about. And there's also a really kind of interesting nugget that is the trajectory that the humans were labeling over has a lot more information than the RL algorithm would see if you just had one state, which is kind of why people think that it's why the performance in this paper was so strong. But I still think that it's surprising that there isn't more RL work of this style happening now. This paper is in 2017. So it's like six years later and I haven't seen things that are exactly similar, but it's a great paper to understand where stuff that's happening now kind of came from.Swyx [00:17:58]: Just on the Christiano paper, you mentioned the performance being strong. I don't remember what results should I have in mind when I think about that paper?Nathan [00:18:04]: It's mostly like if you think about an RL learning curve, which is like on the X axis, you have environment interactions on the Y axis, you have performance. You can think about different like ablation studies of between algorithms. So I think they use like A2C, which I don't even remember what that stands for as their baseline. But if you do the human preference version on a bunch of environments, like the human preference labels, the agent was able to learn faster than if it just learned from the signal from the environment, which means like it's happening because the reward model has more information than the agent would. But like the fact that it can do better, I was like, that's pretty surprising to me because RL algorithms are pretty sensitive. So I was like, okay.Swyx [00:18:41]: It's just one thing I do want to establish as a baseline for our listeners. We are updating all the weights. In some sense, the next token prediction task of training a language model is a form of reinforcement learning. Except that it's not from human feedback. It's just self-supervised learning from a general corpus. There's one distinction which I love, which is that you can actually give negative feedback. Whereas in a general sort of pre-training situation, you cannot. And maybe like the order of magnitude of feedback, like the Likert scale that you're going to talk about, that actually just gives more signal than a typical training process would do in a language model setting. Yeah.Nathan [00:19:15]: I don't think I'm the right person to comment exactly, but like you can make analogies that reinforcement learning is self-supervised learning as well. Like there are a lot of things that will point to that. I don't know whether or not it's a richer signal. I think that could be seen in the results. It's a good thing for people to look into more. As reinforcement learning is so much less compute, like it is a richer signal in terms of its impact. Because if they could do what RLHF is doing at pre-training, they would, but they don't know how to have that effect in like a stable manner. Otherwise everyone would do it.Swyx [00:19:45]: On a practical basis, as someone fine-tuning models, I have often wished for negative fine-tuning, which pretty much doesn't exist in OpenAI land. And it's not the default setup in open-source land.Nathan [00:19:57]: How does this work in like diffusion models and stuff? Because you can give negative prompts to something to like stable diffusion or whatever. It's for guidance.Swyx [00:20:04]: That's for clip guidance.Nathan [00:20:05]: Is that just from like how they prompt it then? I'm just wondering if we could do something similar. It's another tangent.Swyx [00:20:10]: I do want to sort of spell that out for people in case they haven't made the connection between RLHF and the rest of the training process. They might have some familiarity with it.Nathan [00:20:19]: Yeah. The upcoming slides can really dig into this, which is like this in 2018 paper, there was a position paper from a bunch of the same authors from the Christiano paper and from the OpenAI work that everyone knows, which is like, they write a position paper on what a preference reward model could do to solve alignment for agents. That's kind of based on two assumptions. The first assumption is that we can learn user intentions to a sufficiently high accuracy. That doesn't last with me because I don't know what that means. But the second one is pretty telling in the context of RLHF, which is for many tasks we want to solve, evaluation of outcomes is easier than producing the correct behavior. And this is the whole thing. It's like we can compare two poems that the model generates and it can be viewed as liking a positive example, or it could be viewed as really disliking a negative example. And that's what I think a lot of people are doing in like the harm space is like a harmful response to a language model, whether or not you agree with the company's definition of harms is that it's a really bad negative example and they downweight them by preferring something more benign in the RLHF process, among other ways of dealing with safety. So that's a good way of saying it's like this is core, this kind of like comparison and positive or negative example is core to all of the RLHF work that has continued.Swyx [00:21:29]: People often say, I don't know what I want, but I'll know when I see it. This is that expressed in reinforcement learning tools.Nathan [00:21:35]: Yeah, it is. Yeah, it is. That's what everyone's doing in the preference modeling stage that we'll get to. Yeah. Yeah. And you can see there are more papers. This is really just to have all the links for people that go deeper. There's a Ziegler et al. paper in 2019, which shows that you can do this RLHF process on language models. This familiar diagram starts to emerge in 2019, and it's just to show that this goes really far back. I think we can kind of breeze through some of these. And then 2020 is the first open AI experiment that I think caught people's eyes, which is this learning to summarize experiment. It has this three-step process that we'll go to into more when I kind of go into the main concepts. But this is like the first time you see this diagram that they reuse with InstructGPT, they reuse with ChatGPT. And the types of examples that they would have, I don't think I need to read these exactly, but one that I have read a whole bunch of times is like, they took these prompts from Reddit that was like, explain like I'm five or get career advice, and people really pour their heart and soul into these. So these are like multi-paragraph pieces of writing. And then they essentially do comparisons between a vanilla language model, like I think it was either GPT-2 or GPT-3, I don't always get the exact years.Swyx [00:22:42]: 3 was early 2020. So that's about right.Nathan [00:22:45]: Yeah. So this is probably done with GPT-2. It doesn't really matter. But the language model does normal things when you do few shot, which is like it repeats itself. It doesn't have nice text. And what they did is that this was the first time where the language model would generate like pretty nice text from an output. It was restricted to the summarization domain. But I think that I guess this is where I wish I was paying attention more because I would see the paper, but I didn't know to read the language model outputs and kind of understand this qualitative sense of the models very well then. Because you look at the plots in the papers, these Learning to Summarize and Destruct GPT have incredibly pretty plots, just like nicely separated lines with error bars and they're like superfine tuning works, the RL step works. But if you were early to see like how different the language that was written by these models was, I think you could have been early to like things like ChatGPT and knowing RLHF would matter. And now I think the good people know to chat with language models, but not even everyone does this. Like people are still looking at numbers. And I think OpenAI probably figured it out when they were doing this, how important that could be. And then they had years to kind of chisel away at that and that's why they're doing so well now. Yeah.Swyx [00:23:56]: I mean, arguably, you know, it's well known that ChatGPT was kind of an accident that they didn't think it would be that big of a deal. Yeah.Nathan [00:24:02]: So maybe they didn't. Maybe they didn't, but they were getting the proxy that they needed.Swyx [00:24:06]: I've heard off the record from other labs that it was in the air. If OpenAI didn't do it, someone else would have done it. So you've mentioned a couple of other papers that are very seminal to this period. And I love how you say way back when in referring to 2019.Nathan [00:24:19]: It feels like it in my life.Swyx [00:24:21]: So how much should people understand the relationship between RLHF, instruction tuning, PPO, KL divergence, anything like that? Like how would you construct the level of knowledge that people should dive into? What should people know at the high level? And then if people want to dive in deeper, where do they go? Is instruct tuning important here or is that part of the overall process towards modern RLHF?Nathan [00:24:44]: I think for most people, instruction tuning is probably still more important in their day to day life. I think instruction tuning works very well. You can write samples by hand that make sense. You can get the model to learn from them. You could do this with very low compute. It's easy to do almost in like no code solutions at this point. And the loss function is really straightforward. And then if you're interested in RLHF, you can kind of learn from it from a different perspective, which is like how the instruction tuning distribution makes it easier for your RLHF model to learn. There's a lot of details depending on your preference data, if it's close to your instruction model or not, if that matters. But that's really at the RLHF stage. So I think it's nice to segment and just kind of understand what your level of investment and goals are. I think instruction tuning still can do most of what you want to do. And it's like, if you want to think about RLHF, at least before DPO really had taken off at all, it would be like, do you want to have a team of at least like five people if you're really thinking about doing RLHF? I think DPO makes it a little bit easier, but that's still really limited to kind of one data set that everyone's using at this point. Like everyone's using this ultra feedback data set and it boosts AlpacaVal, MTBench, TruthfulQA and like the qualitative model a bit. We don't really know why. It's like, it might just be a data set combined with the method, but you've got to be ready for a bumpy ride if you're wanting to try to do RLHF. I don't really recommend most startups to do it unless it's like going to provide them a clear competitive advantage in their kind of niche, because you're not going to make your model chat GPT like better than OpenAI or anything like that. You've got to accept that there's some exploration there and you might get a vein of benefit in your specific domain, but I'm still like, oh, be careful going into the RLHF can of worms. You probably don't need to.Swyx [00:26:27]: Okay. So there's a bit of a time skip in what you mentioned. DPO is like a couple months old, so we'll leave that towards the end. I think the main result that I think most people talk about at this stage, we're talking about September 2020 and then going into, I guess maybe last year was Vicuña as one of the more interesting applications of instruction tuning that pushed LLAMA1 from, let's say a GPT 3-ish model to a GPT 3.5 model in pure open source with not a lot of resources. I think, I mean, they said something like, you know, they use like under $100 to makeNathan [00:26:58]: this. Yeah. Like instruction tuning can really go a long way. I think the claims of chat GPT level are long overblown in most of the things in open source. I think it's not to say, like Vicuña was a huge step and it's just kind of showing that instruction tuning with the right data will completely change what it feels like to talk with your model. Yeah.Swyx [00:27:19]: From text completion to actually chatting back and forth. Yeah. Yeah.Nathan [00:27:23]: Instruction tuning can be multi-turn. Just having a little bit of data that's like a couple of turns can go a really long way. That was like the story of the whole first part of the year is like people would be surprised by how far you can take instruction tuning on a small model. I think the things that people see now is like the small models don't really handle nuance as well and they could be more repetitive even if they have really good instruction tuning. But if you take that kind of 7 to 70 billion parameter jump, like the instruction tuning at the bigger model is like robustness, little things make more sense. So that's still just with instruction tuning and scale more than anything else.Swyx [00:27:56]: Excellent. Shall we go to technical overview?Nathan [00:27:58]: Yeah. This is kind of where we go through my own version of this like three phase process. You can talk about instruction tuning, which we've talked about a lot. It's funny because all these things, instruction tuning has the fewest slides, even though it's the most practical thing for most people. We could save the debate for like if the big labs still do instruction tuning for later, but that's a coming wave for people. And then like preference data and training and then kind of like what does reinforce learning optimization actually mean? We talk about these sequentially because you really have to be able to do each of them to be able to do the next one. You need to be able to have a model that's chatty or helpful instruction following. Every company has their own word that they like to assign to what instructions mean. And then once you have that, you can collect preference data and do some sort of optimization.Swyx [00:28:39]: When you say word, you mean like angle bracket inst or do you mean something else?Nathan [00:28:42]: Oh, I don't even know what inst means, but just saying like they use their adjective that they like. I think Entropic also like steerable is another one.Swyx [00:28:51]: Just the way they describe it. Yeah.Nathan [00:28:53]: So like instruction tuning, we've covered most of this is really about like you should try to adapt your models to specific needs. It makes models that were only okay, extremely comprehensible. A lot of the times it's where you start to get things like chat templates. So if you want to do system prompts, if you want to ask your model, like act like a pirate, that's one of the ones I always do, which is always funny, but like whatever you like act like a chef, like anything, this is where those types of things that people really know in language models start to get applied. So it's good as a kind of starting point because this chat template is used in our early childhood and all of these things down the line, but it was a basic pointer. It's like, once you see this with instruction tuning, you really know it, which is like you take things like stack overflow where you have a question and an answer. You format that data really nicely. There's much more tricky things that people do, but I still think the vast majority of it is question answer. Please explain this topic to me, generate this thing for me. That hasn't changed that much this year. I think people have just gotten better at scaling up the data that they need. Yeah, this is where this talk will kind of take a whole left turn into more technical detail land. I put a slide with the RLHF objective, which I think is good for people to know. I've started going back to this more, just kind of understand what is trying to happen here and what type of math people could do. I think because of this algorithm, we've mentioned this, it's in the air, direct preference optimization, but everything kind of comes from an equation of trying to learn a policy that maximizes the reward. The reward is some learned metric. A lot can be said about what the reward should be subject to some constraint. The most popular constraint is the KL distraint, which is just a distributional distance. Essentially in language models, that means if you have a completion from your instruction or RLHF model, you can compare that completion to a base model. And looking at the log probs from the model, which are essentially how likely each token is, you can see a rough calculation of the distance between these two models, just as a scalar number. I think what that actually looks like in code, you can look at it. It'd be like a sum of log probs that you get right from the model. It'll look much more simpler than it sounds, but it is just to make the optimization kind of stay on tracks.Make sure it doesn't overfit to the RLHF data. Because we have so little data in RLHF, overfitting is really something that could happen. I think it'll fit to specific features that labelers like to see, that the model likes to generate, punctuation, weird tokens like calculator tokens. It could overfit to anything if it's in the data a lot and it happens to be in a specific format. And the KL constraint prevents that. There's not that much documented work on that, but there's a lot of people that know if you take that away, it just doesn't work at all. I think it's something that people don't focus on too much. But the objective, as I said, it's just kind of, you optimize the reward. The reward is where the human part of this comes in. We'll talk about that next. And then subject to a constraint, don't change the model too much. The real questions are, how do you implement the reward? And then how do you make the reward go up in a meaningful way? So like a preference model, the task is kind of to design a human reward. I think the equation that most of the stuff is based on right now is something called a Bradley-Terry model, which is like a pairwise preference model where you compare two completions and you say which one you like better. I'll show an interface that Anthropic uses here. And the Bradley-Terry model is really a fancy probability between two selections. And what's happening in the math is that you're looking at the probability that the chosen completion, the one you like better, is actually the better completion over the rejected completion. And what these preference models do is they assume this probability is correlated to reward. So if you just sample from this probability, it'll give you a scalar. And then you use that reward later on to signify what piece of text is better. I'm kind of inclined to breeze through the math stuff because otherwise, it's going to be not as good to listen to.Alessio [00:32:49]: I think people want to hear it. I think there's a lot of higher level explanations out there. Yeah.Nathan [00:32:55]: So the real thing is you need to assign a scalar reward of how good a response is. And that's not necessarily that easy to understand. Because if we take back to one of the first works, I mentioned this tamer thing for decision making. People tried that with language models, which is if you have a prompt in a completion and you just have someone rate it from 0 to 10, could you then train a reward model on all of these completions in 0 to 10 ratings and see if you can get chat2BT with that? And the answer is really kind of no. Like a lot of people tried that. It didn't really work. And then that's why they tried this pairwise preference thing. And it happened to work. And this Bradley Terry model comes from the 50s. It's from these fields that I was mentioning earlier. And it's wild how much this happens. I mean, this screenshot I have in the slides is from the DPO paper. I think it might be the appendix. But it's still really around in the literature of what people are doing for RLHF.Alessio [00:33:45]: Yeah.Nathan [00:33:45]: So it's a fun one to know.Swyx [00:33:46]: I'll point out one presumption that this heavily relies on. You mentioned this as part of your six presumptions that we covered earlier, which is that you can aggregate these preferences. This is not exactly true among all humans, right? I have a preference for one thing. You have a preference for a different thing. And actually coming from economics, you mentioned economics earlier. There's a theorem or a name for this called error impossibility, which I'm sure you've come across..Nathan [00:34:07]: It's one of the many kind of things we throw around in the paper.Swyx [00:34:10]: Right. Do we just ignore it?Nathan [00:34:14]: We just, yeah, just aggregate. Yeah. I think the reason this really is done on a deep level is that you're not actually trying to model any contestable preference in this. You're not trying to go into things that are controversial or anything. It's really the notion of preference is trying to stay around correctness and style rather than any meaningful notion of preference. Because otherwise these companies, they don't want to do this at all. I think that's just how it is. And it's like, if you look at what people actually do. So I have a bunch of slides on the feedback interface. And they all publish this.Swyx [00:34:43]: It's always at the appendices of every paper.Nathan [00:34:47]: There's something later on in this talk, which is like, but it's good to mention. And this is when you're doing this preference collection, you write out a very long document of instructions to people that are collecting this data. And it's like, this is the hierarchy of what we want to prioritize. Something amount like factuality, helpfulness, honestness, harmlessness. These are all different things. Every company will rank these in different ways, provide extensive examples. It's like, if you see these two answers, you should select this one and why. And all of this stuff. And then my kind of like head scratching is like, why don't we check if the models actually do these things that we tell the data annotators to collect? But I think it's because it's hard to make that attribution. And it's hard to test if a model is honest and stuff. It would just be nice to understand the kind of causal mechanisms as a researcher or like if our goals are met. But at a simple level, what it boils down to, I have a lot more images than I need. It's like you're having a conversation with an AI, something like type GPT. You get shown two responses or more in some papers, and then you have to choose which one is better. I think something you'll hear a lot in this space is something called a Likert scale. Likert is a name. It's a name for probably some research in economics, decision theory, something. But essentially, it's a type of scale where if you have integers from like one to eight, the middle numbers will represent something close to a tie. And the smallest numbers will represent one model being way better than the other. And the biggest numbers will be like the other models better. So in the case of one to eight, if you're comparing models A to B, if you return a one, if you really liked option A, you return eight if you really like B, and then like a four or five if they were close. There's other ways to collect this data. This one's become really popular. We played with it a bit at Hugging Face. It's hard to use. Filling out this preference data is really hard. You have to read like multiple paragraphs. It's not for me. Some people really like it. I hear I'm like, I can't imagine sitting there and reading AI-generated text and like having to do that for my job. But a lot of these early papers in RLHF have good examples of what was done. The one I have here is from Anthropic's collection demo because it was from slides that I did with Anthropic. But you can look up these in the various papers. It looks like Chat2BT with two responses, and then you have an option to say which one is better. It's nothing crazy. The infrastructure is almost exactly the same, but they just log which one you think is better. I think places like Scale are also really big in this where a lot of the labeler companies will help control like who's doing how many samples. You have multiple people go over the same sample once and like what happens if there's disagreement. I don't really think this disagreement data is used for anything, but it's good to know like what the distribution of prompts is, who's doing it, how many samples you have, controlling the workforce. All of this is very hard. A last thing to add is that a lot of these companies do collect optional metadata. I think the Anthropic example shows a rating of like how good was the prompt or the conversation from good to bad because things matter. Like there's kind of a quadrant of preference data in my mind, which is you're comparing a good answer to a good answer, which is like really interesting signal. And then there's kind of the option of you're comparing a bad answer to a bad answer, which is like you don't want to train your model on two different issues. This is like, we did this at Hugging Base and it was like, our data was like, we don't know if we can use this because a lot of it was just bad answer to bad answer because you're like rushing to try to do this real contract. And then there's also good answer to bad answer, which I think is probably pretty reasonable to include. You just prefer the good one and move on with your life. But those are very different scenarios. I think open AIs of the world are all in good answer, good answer, and have learned to eliminate everything else. But when people try to do this in open source, it's probably like what Open Assistance saw is like, there's just a lot of bad answers in your preference data. And you're like, what do I do with this? Metadata flags can help. I threw in the instruct GPT metadata. You can see how much they collect here. And like everything from the model fails to actually complete the task, hallucinations, different types of offensive or dangerous content, moral judgment, expresses opinion. Like, I don't know exactly if they're doing this now, but you can kind of see why doing RLHF at scale and prioritizing a lot of different endpoints would be hard because these are all things I'd be interested in if I was scaling up a big team to do RLHF and like what is going into the preference data. You do an experiment and you're like, okay, we're going to remove all the data where they said the model hallucinates like just that and then retrain everything. Like, what does that do?Swyx [00:38:59]: Yeah, so hallucination is big, but some of these other metadata categories, and I've seen this in a lot of papers, it's like, does it contain sexual content? Does it express a moral judgment? Does it denigrate a protected class? That kind of stuff, very binary. Should people try to adjust for this at the RLHF layer or should they put it as a pipeline where they have a classifier as a separate model that grades the model output?Nathan [00:39:20]: Do you mean for training or like a deployment? Deployment. I do think that people are doing it at deployment. I think we've seen safety and other things in the RLHF pipeline. Like Lama 2 is famous for kind of having this like helpfulness and safety reward models. Deep in the Gemini report is something that Gemini has like four things, which is like helpfulness, factuality, maybe safety, maybe something else. But places like Anthropic and Chattopadhyay and Bard almost surely have a classifier after, which is like, is this text good? Is this text bad? That's not that surprising, I think, because you could use like a hundred times smaller language model and do much better at filtering than RLHF. But I do think it's still so deeply intertwined with the motivation of RLHF to be for safety that some of these categories still persist. I think that's something I'll kind of settle out, I think.Swyx [00:40:11]: I'm just wondering if it's worth collecting this data for the RLHF purpose, if you're not going to use it in any way, separate model to-Nathan [00:40:18]: Yeah, I don't think OpenAI will collect all of this anymore, but I think for research perspectives, it's very insightful to know, but it's also expensive. So essentially your preference data scales with how many minutes it takes for you to do each task and every button is like, it scales pretty linearly. So it's not cheap stuff.Swyx [00:40:35]: Can we, since you mentioned expensiveness, I think you may have joined one of our spaces back in Lama 2 was released. We had an estimate from you that was something on the order of Lama 2 costs $3 to $6 million to train GPU-wise, and then it was something like $20 to $30 million in preference data. Is that something that's still in the ballpark? I don't need precise numbers.Nathan [00:40:56]: I think it's still a ballpark. I know that the 20 million was off by a factor of four because I was converting from a prompt number to a total data point. So essentially when you do this, if you have multi-turn setting, each turn will be one data point and the Lama 2 paper reports like 1.5 million data points, which could be like 400,000 prompts. So I would say it's still say like 6 to 8 million is safe to say that they're spending, if not more, they're probably also buying other types of data and or throwing out data that they don't like, but it's very comparable to compute costs. But the compute costs listed in the paper always are way lower because all they have to say is like, what does one run cost? But they're running tens or hundreds of runs. So it's like, okay, like... Yeah, it's just kind of a meaningless number. Yeah, the data number would be more interesting.Alessio [00:41:42]: What's the depreciation of this data?Nathan [00:41:46]: It depends on the method. Like some methods, people think that it's more sensitive to the, this is what I was saying. It was like, does the type of instruction tuning you do matter for RLHF? So like, depending on the method, some people are trying to figure out if you need to have like what is called like, this is very confusing. It's called like on policy data, which is like your RLHF data is from your instruction model. I really think people in open source and academics are going to figure out how to use any preference data on any model just because they're scrappy. But there's been an intuition that to do like PPO well and keep improving the model over time and do like what Meta did and what people think that OpenAI does is that you need to collect new preference data to kind of edge the distribution of capabilities forward. So there's a depreciation where like the first batch of data you collect isn't really useful for training the model when you have the fifth batch. We don't really know, but it's a good question. And I do think that if we had all the LLAMA data, we wouldn't know what to do with all of it. Like probably like 20 to 40% would be pretty useful for people, but not the whole data set. Like a lot of it's probably kind of gibberish because they had a lot of data in there.Alessio [00:42:51]: So do you think like the open source community should spend more time figuring out how to reuse the data that we have or like generate more data? I think that's one of the-Nathan [00:43:02]: I think if the people are kind of locked into using synthetic data, people also think that synthetic data is like GPT-4 is more accurate than humans at labeling preferences. So if you look at these diagrams, like humans are about 60 to 70% agreement. And we're like, that's what the models get to. And if humans are about 70% agreement or accuracy, like GPT-4 is like 80%. So it is a bit better, which is like in one way of saying it.Swyx [00:43:24]: Humans don't even agree with humans 50% of the time.Nathan [00:43:27]: Yeah, so like that's the thing. It's like the human disagreement or the lack of accuracy should be like a signal, but how do you incorporate that? It's really tricky to actually do that. I think that people just keep using GPT-4 because it's really cheap. It's one of my like go-to, like I just say this over and over again is like GPT-4 for data generation, all terms and conditions aside because we know OpenAI has this stuff is like very cheap for getting pretty good data compared to compute or salary of any engineer or anything. So it's like tell people to go crazy generating GPT-4 data if you're willing to take the organizational like cloud of should we be doing this? But I think most people have accepted that you kind of do this, especially at individuals. Like they're not gonna come after individuals. I do think more companies should think twice before doing tons of OpenAI outputs. Also just because the data contamination and what it does to your workflow is probably hard to control at scale.Swyx [00:44:21]: And we should just mention at the time of recording, we've seen the first example of OpenAI enforcing their terms of service. ByteDance was caught, reported to be training on GPT-4 data and they got their access to OpenAI revoked. So that was one example.Nathan [00:44:36]: Yeah, I don't expect OpenAI to go too crazy on this cause they're just gonna, there's gonna be so much backlash against them. And like, everyone's gonna do it anyways.Swyx [00:44:46]: And what's at stake here to spell it out is like, okay, that's like cost $10 to collect one data point from a human. It's gonna cost you like a 10th of a cent with OpenAI, right? So like it's just orders of magnitude cheaper. And therefore people-Nathan [00:44:58]: Yeah, and it's like the signal you get from humans is from preferences isn't that high. The signal that you get from humans for instructions is pretty high, but it is also very expensive. So like the human instructions are definitely like by far and away the best ones out there compared to the synthetic data. But I think like the synthetic preferences are just so much easier to get some sort of signal running with and you can work in other, I think people will start working in other goals there between safety and whatever. That's something that's taking off and we'll kind of see that. I think in 2024, at some point, people will start doing things like constitutional AI for preferences, which will be pretty interesting. I think we saw how long it took RLHF to get started in open source. Instruction tuning was like the only thing that was really happening until maybe like August, really. I think Zephyr was the first model that showed success with RLHF in the public, but that's a long time from everyone knowing that it was something that people are interested in to having any like check mark. So I accept that and think the same will happen with constitutional AI. But once people show that you can do it once, they continue to explore.Alessio [00:46:01]: Excellent.Swyx [00:46:01]: Just in the domain of human preference data suppliers, Scale.ai very happily will tell you that they supplied all that data for Lama 2. The other one is probably interesting, LMSYS from Berkeley. What they're running with Chaterina is perhaps a good store of human preference data.Nathan [00:46:17]: Yeah, they released some toxicity data. They, I think, are generally worried about releasing data because they have to process it and make sure everything is safe and they're really lightweight work. I think they're trying to release the preference data. I have, if we make it to evaluation, I'd pretty much say that Chaterina is the best limited evaluation that people have to learn how to use language models. And like, it's very valuable data. They also may share some data with people that they host models from. So like if your model is hosted there and you pay for the hosting, you can get the prompts because you're pointing the endpoint at it and that gets pinged to you and you're any real LLM inference stack saves the prompts tha

Paul Christiano runs the Alignment Research Center, a non-profit research organization whose mission is to align future machine learning systems with human interests. Paul previously ran the language model alignment team at OpenAI, the creators of ChatGPT.  Today, we're hoping to explore the solution-landscape to the AI Alignment problem, and hoping Paul can guide us on that journey.  ------ ✨ DEBRIEF | Unpacking the episode:  https://www.bankless.com/debrief-paul-christiano    ------ ✨ COLLECTIBLES | Collect this episode:  https://collectibles.bankless.com/mint  ------ ✨ Always wanted to become a Token Analyst? Bankless Citizens get exclusive access to Token Hub. Join Them. https://bankless.cc/TokenHubRSS   ------ In today's episode, Paul answers many questions, but the overarching ones are:  1) How BIG is the AI Alignment problem?  2) How HARD is the AI Alighment problem? 3) How SOLVABLE is the AI Alignment problem?  Does humanity have a chance? Tune in to hear Paul's thoughts.  ------ BANKLESS SPONSOR TOOLS:  ⚖️ ARBITRUM | SCALING ETHEREUM https://bankless.cc/Arbitrum