Podcasts about Allen Institute

Tricks to Fine Tuning // MLOps Podcast #318 with Prithviraj Ammanabrolu, Research Scientist at Databricks. Join the Community: https://go.mlops.community/YTJoinIn Get the newsletter: https://go.mlops.community/YTNewsletter // AbstractPrithviraj Ammanabrolu drops by to break down Tao fine-tuning—a clever way to train models without labeled data. Using reinforcement learning and synthetic data, Tao teaches models to evaluate and improve themselves. Raj explains how this works, where it shines (think small models punching above their weight), and why it could be a game-changer for efficient deployment.// BioRaj is an Assistant Professor of Computer Science at the University of California, San Diego, leading the PEARLS Lab in the Department of Computer Science and Engineering (CSE). He is also a Research Scientist at Mosaic AI, Databricks, where his team is actively recruiting research scientists and engineers with expertise in reinforcement learning and distributed systems.Previously, he was part of the Mosaic team at the Allen Institute for AI. He earned his PhD in Computer Science from the School of Interactive Computing at Georgia Tech, advised by Professor Mark Riedl in the Entertainment Intelligence Lab.// Related LinksWebsite: https://www.databricks.com/~~~~~~~~ ✌️Connect With Us ✌️ ~~~~~~~Catch all episodes, blogs, newsletters, and more: https://go.mlops.community/TYExploreJoin our Slack community [https://go.mlops.community/slack]Follow us on X/Twitter [@mlopscommunity](https://x.com/mlopscommunity) or [LinkedIn](https://go.mlops.community/linkedin)] Sign up for the next meetup: [https://go.mlops.community/register]MLOps Swag/Merch: [https://shop.mlops.community/]Connect with Demetrios on LinkedIn: /dpbrinkmConnect with Raj on LinkedIn: /rajammanabroluTimestamps:[00:00] Raj's preferred coffee[00:36] Takeaways[01:02] Tao Naming Decision[04:19] No Labels Machine Learning[08:09] Tao and TAO breakdown[13:20] Reward Model Fine-Tuning[18:15] Training vs Inference Compute[22:32] Retraining and Model Drift[29:06] Prompt Tuning vs Fine-Tuning[34:32] Small Model Optimization Strategies[37:10] Small Model Potential[43:08] Fine-tuning Model Differences[46:02] Mistral Model Freedom[53:46] Wrap up

AI Models Lie for Goals

Tricks to Fine Tuning // MLOps Podcast #318 with Prithviraj Ammanabrolu, Research Scientist at Databricks.Join the Community: https://go.mlops.community/YTJoinIn Get the newsletter: https://go.mlops.community/YTNewsletter // AbstractPrithviraj Ammanabrolu drops by to break down Tao fine-tuning—a clever way to train models without labeled data. Using reinforcement learning and synthetic data, Tao teaches models to evaluate and improve themselves. Raj explains how this works, where it shines (think small models punching above their weight), and why it could be a game-changer for efficient deployment.// BioRaj is an Assistant Professor of Computer Science at the University of California, San Diego, leading the PEARLS Lab in the Department of Computer Science and Engineering (CSE). He is also a Research Scientist at Mosaic AI, Databricks, where his team is actively recruiting research scientists and engineers with expertise in reinforcement learning and distributed systems.Previously, he was part of the Mosaic team at the Allen Institute for AI. He earned his PhD in Computer Science from the School of Interactive Computing at Georgia Tech, advised by Professor Mark Riedl in the Entertainment Intelligence Lab.// Related LinksWebsite: https://www.databricks.com/~~~~~~~~ ✌️Connect With Us ✌️ ~~~~~~~Catch all episodes, blogs, newsletters, and more: https://go.mlops.community/TYExploreJoin our Slack community [https://go.mlops.community/slack]Follow us on X/Twitter [@mlopscommunity](https://x.com/mlopscommunity) or [LinkedIn](https://go.mlops.community/linkedin)] Sign up for the next meetup: [https://go.mlops.community/register]MLOps Swag/Merch: [https://shop.mlops.community/]Connect with Demetrios on LinkedIn: /dpbrinkmConnect with Raj on LinkedIn: /rajammanabroluTimestamps:[00:00] Raj's preferred coffee[00:36] Takeaways[01:02] Tao Naming Decision[04:19] No Labels Machine Learning[08:09] Tao and TAO breakdown[13:20] Reward Model Fine-Tuning[18:15] Training vs Inference Compute[22:32] Retraining and Model Drift[29:06] Prompt Tuning vs Fine-Tuning[34:32] Small Model Optimization Strategies[37:10] Small Model Potential[43:08] Fine-tuning Model Differences[46:02] Mistral Model Freedom[53:46] Wrap up

In a major scientific breakthrough, researchers from 29 institutions across North America—led by the Allen Institute—have developed over 1,000 new genetic tools called enhancer AAV vectors to precisely target specific brain cells. Guest: Dr. Bosiljka Tasic - Director of Molecular Genetics at the Allen Institute for Brain Science Learn more about your ad choices. Visit megaphone.fm/adchoices

Just how weird will the AI-powered future be? To discuss, ChinaTalk interviewed Nathan Lambert, who writes the Interconnects newsletter and researches AI at the Allen Institute. We get into… Why OpenAI is trending toward engagement farming and sycophancy, The state of Chinese AI innovation six months post-DeepSeek, and the factors influencing diffusion of Chinese vs American models, Meta's organizational culture and how it influences the quality of the Llama models, Unconventional career advice for the AI age. Nathan's book recommendation: Careless People: A Cautionary Tale of Power, Greed, and Lost Idealism by Sarah Wynn-Williams Learn more about your ad choices. Visit megaphone.fm/adchoices

Just how weird will the AI-powered future be? To discuss, ChinaTalk interviewed Nathan Lambert, who writes the Interconnects newsletter and researches AI at the Allen Institute. We get into… Why OpenAI is trending toward engagement farming and sycophancy, The state of Chinese AI innovation six months post-DeepSeek, and the factors influencing diffusion of Chinese vs American models, Meta's organizational culture and how it influences the quality of the Llama models, Unconventional career advice for the AI age. Nathan's book recommendation: Careless People: A Cautionary Tale of Power, Greed, and Lost Idealism by Sarah Wynn-Williams Learn more about your ad choices. Visit megaphone.fm/adchoices

"Vous avez quoi entre les mains ?" "De l'or !"Et ça, les trois milliardaires les plus en vogue l'ont bien compris.Xavier Niel (Free), Rodolphe Saadé (CMA-CGM) et Eric Schmidt (Google) ont financé à hauteur de 300 millions d'euros le laboratoire de recherche ouverte (open source) à but non lucratif dirigé par Patrick Perez, chercheur en IA appliquée.Patrick est à la tête de Kyutai, fondé en 2023, qui est déjà l'un des leaders français en IA, avec plusieurs outils disponibles : Moshi, leur IA vocale conversationnelle ; Hibiki, pour la traduction en live ; et MoshiVis, pour l'analyse d'images.Au programme de cet épisode : taxis autonomes, erreurs inhérentes à l'IA, entraînement des modèles par les humains, problème des contenus synthétiques… et là où l'IA est la plus lucrative.Avant de fonder Kuytai, Patrick a navigué entre recherche académique et industrie. Il a dirigé la stratégie IA chez Valeo, travaillé sur le traitement d'images chez Technicolor, et il a aussi mené des travaux chez Microsoft et à l'INRIA, deux références en innovation technologique.Ce parcours lui permet aujourd'hui de s'attaquer à l'un des sujets les plus prometteurs du moment : la multimodalité en IA — une approche qui combine texte, image et audio pour créer des outils plus puissants et plus intuitifs.Et bonne nouvelle, c'est la nouvelle vague de recherche qui sera à l'origine des prochaines grandes percées dans le domaine.Cet épisode est un point d'étape pour vraiment comprendre où en est la recherche en IA et comment se positionne la France.Entre fantasmes et réalités, Patrick explique comment fonctionne l'IA et comment elle capte peu à peu les signaux du monde réel — et pourquoi c'est une révolution.TIMELINE:00:00:00 : La beauté des mathématiques appliquées rendue accessible grâce à l'IA00:11:17 : Vers une IA vraiment multimodale : comprendre sans passer par le texte00:21:20 : Donner des yeux et des oreilles à l'IA00:30:17 : La rencontre entre IA et robotique : des robotaxis à Paris ?00:48:09 : Les prochaines avancées de l'IA vont TOUT changer00:55:20 : GPT se trompe encore… et c'est une bonne chose !01:00:51 : Quand la machine devient professeur pour d'autres machines01:08:33 : L'intervention des humains dans l'entraînement des IAs est encore nécessaire01:21:33 : Le problème des contenus synthétiques qui ne se présentent pas comme tels01:34:07 : Deviendrons-nous débiles en déléguant trop à l'IA ?01:42:40 : Là où l'IA est la plus lucrative01:53:09 : Convaincre des géants : Xavier Niel, Rodolphe Saadé, Eric Schmidt02:07:36 : L'IA pour coder : où en est-on ?02:15:59 : Ce qu'on peut faire avec l'IA et le coût des GPULes anciens épisodes de GDIY mentionnés : #450 - Karim Beguir - InstaDeep - L'IA Générale ? C'est pour 2025#397 - Yann Le Cun - Chief AI Scientist chez Meta - L'Intelligence Artificielle Générale ne viendra pas de Chat GPT#267 - Andréa Bensaïd - Eskimoz - Refuser 30 millions pour viser le milliard#418 - Clément Delangue - Hugging Face - 4,5 milliards de valo avec un produit gratuit à 99%#414 - Florian Douetteau - Dataiku - La prochaine grande vague de l'IA : l'adopter ou périr ?Nous avons parlé de :KYUTAIMoshi (l'IA de Kyuntai)Inria : Institut national de recherche en sciences et technologies du numériqueStéphane MallardTest des taxis autonomes Weymo : vidéo InstaDocumentaire aux USHibiki (outil de traduction)Allen Institute for Artificial IntelligenceVous pouvez contacter Patrick sur Linkedin et sur Bluesky.Vous souhaitez sponsoriser Génération Do It Yourself ou nous proposer un partenariat ?Contactez mon label Orso Media via ce formulaire.Distribué par Audiomeans. Visitez audiomeans.fr/politique-de-confidentialite pour plus d'informations.

How a mouse watching the Matrix improved our knowledge of the brain Guest: Dr. Forrest Collman, Associate Director of Informatics at the Allen Institute for Brain Science Learn more about your ad choices. Visit megaphone.fm/adchoices

Are political tensions making people reconsider parenthood? Guest: Zachary Neal, Professor of Psychology at Michigan State University Will the gun buyback program be revived? Guest: Dr. Noah Schwartz, Assistant Professor of Political Science at the University of the Fraser Valley and Author of “On Target: Gun Culture, Storytelling, and the NRA” Could there be life on Venus? Guest: Sara Seager, Astrophysicist and Professor of Physics, Planetary Science, and Aeronautics and Astronautics at MIT How a mouse watching the Matrix improved our knowledge of the brain Guest: Dr. Forrest Collman, Associate Director of Informatics at the Allen Institute for Brain Science How RFK Jr.'s policies can impact Canada Guest: Taylor Noakes, Independent Journalist and Public Historian from Montreal Who should be the MP for Port Moody–Coquitlam? Guest: Bonita Zarillo, NDP Candidate for Port Moody–Coquitlam Guest: Zoe Royer, Liberal Candidate for Port Moody–Coquitlam Guest: Paul Lambert, Conservative Candidate for Port Moody–Coquitlam Learn more about your ad choices. Visit megaphone.fm/adchoices

Molti nuovi smartphone hanno batterie sempre più capienti ma, come spiega Roberto Pezzali, esperto di tecnologia della redazione di Dday.it, non aspettatevi di acquistare telefoni con maggiore autonomia. Vi spieghiamo perché.Supercondensatori in grado di erogare picchi di potenza superiori rispetto a quelli garantiti dalle batterie al litio e con tempi di ricarica estremamente rapidi. Per quali applicazioni? Ne parliamo con Matteo Bertocchi, Ceo di Novac, startup italiana che ha appena annunciato un finanziamento da 3,5 milioni di euro.Da alcuni mesi sembra che il ritmo di miglioramento delle prestazioni dei modelli linguistici di grandi dimensioni (LLM) stia rallentando. Alcuni addetti ai lavori avvertono che esiste la possibilità che i dati disponibili per addestrare gli LLM si stiano esaurendo. Parliamo degli ultimi trend delll'IA con Luca Soldaini, ricercatore dell'Allen Institute for AI di Seattle.E come sempre in Digital News le notizie di innovazione e tecnologia più importanti della settimana.

Consciousness is easier to possess than to define. One thing we can do is to look into the brain and see what lights up when conscious awareness is taking place. A complete understanding of this would be known as the "neural correlates of consciousness." Once we have that, we could hopefully make progress on developing a theoretical picture of what consciousness is and why it happens. Today's guest, Christof Koch, is a leader in the search for neural correlates and an advocate of a particular approach to consciousness, Integrated Information Theory.Blog post with transcript: https://www.preposterousuniverse.com/podcast/2025/03/24/309-christof-koch-on-consciousness-and-integrated-information/Support Mindscape on Patreon.Christof Koch was awarded a Ph.D. from the Max Planck Institute for Biological Cybernetics. He is currently a Meritorious Investigator at the Allen Institute for Brain Science, where he was formerly president and chief scientist, and Chief Scientist at the Tiny Blue Dot Foundation. He is the author of several books, most recently Then I Am Myself the World - What Consciousness Is and How to Expand It.Web siteAllen Center web pageGoogle Scholar publicationsAmazon author pageWikipediaSee Privacy Policy at https://art19.com/privacy and California Privacy Notice at https://art19.com/privacy#do-not-sell-my-info.

LET'S GO! Happy second birthday to ThursdAI, your favorite weekly AI news show! Can you believe it's been two whole years since we jumped into that random Twitter Space to rant about GPT-4? From humble beginnings as a late-night Twitter chat to a full-blown podcast, Newsletter and YouTube show with hundreds of thousands of downloads, it's been an absolutely wild ride! That's right, two whole years of me, Alex Volkov, your friendly AI Evangelist, along with my amazing co-hosts, trying to keep you up-to-date on the breakneck speed of the AI worldAnd what better way to celebrate than with a week PACKED with insane AI news? Buckle up, folks, because this week Google went OPEN SOURCE crazy, Gemini got even cooler, OpenAI created a whole new Agents SDK and the open-source community continues to blow our minds. We've got it all - from game-changing model releases to mind-bending demos.This week I'm also on the Weights & Biases company retreat, so TL;DR first and then the newsletter, but honestly, I'll start embedding the live show here in the substack from now on, because we're getting so good at it, I barely have to edit lately and there's a LOT to show you guys! TL;DR and Show Notes & Links* Hosts & Guests* Alex Volkov - AI Eveangelist & Weights & Biases (@altryne)* Co Hosts - @WolframRvnwlf @ldjconfirmed @nisten * Sandra Kublik - DevRel at Cohere (@itsSandraKublik)* Open Source LLMs * Google open sources Gemma 3 - 1B - 27B - 128K context (Blog, AI Studio, HF)* EuroBERT - multilingual encoder models (210M to 2.1B params)* Reka Flash 3 (reasoning) 21B parameters is open sourced (Blog, HF)* Cohere Command A 111B model - 256K context (Blog)* Nous Research Deep Hermes 24B / 3B Hybrid Reasoners (X, HF)* AllenAI OLMo 2 32B - fully open source GPT4 level model (X, Blog, Try It)* Big CO LLMs + APIs* Gemini Flash generates images natively (X, AI Studio)* Google deep research is now free in Gemini app and powered by Gemini Thinking (Try It no cost)* OpenAI released new responses API, Web Search, File search and Computer USE tools (X, Blog)* This weeks Buzz * The whole company is at an offsite at oceanside, CA* W&B internal MCP hackathon and had cool projects - launching an MCP server soon!* Vision & Video* Remade AI - 8 LORA video effects for WANX (HF)* AI Art & Diffusion & 3D* ByteDance Seedream 2.0 - A Native Chinese-English Bilingual Image Generation Foundation Model by ByteDance (Blog, Paper)* Tools* Everyone's talking about Manus - (manus.im)* Google AI studio now supports youtube understanding via link droppingOpen Source LLMs: Gemma 3, EuroBERT, Reka Flash 3, and Cohere Command-A Unleashed!This week was absolutely HUGE for open source, folks. Google dropped a BOMBSHELL with Gemma 3! As Wolfram pointed out, this is a "very technical achievement," and it's not just one model, but a whole family ranging from 1 billion to 27 billion parameters. And get this – the 27B model can run on a SINGLE GPU! Sundar Pichai himself claimed you'd need "at least 10X compute to get similar performance from other models." Insane!Gemma 3 isn't just about size; it's packed with features. We're talking multimodal capabilities (text, images, and video!), support for over 140 languages, and a massive 128k context window. As Nisten pointed out, "it might actually end up being the best at multimodal in that regard" for local models. Plus, it's fine-tuned for safety and comes with ShieldGemma 2 for content moderation. You can grab Gemma 3 on Google AI Studio, Hugging Face, Ollama, Kaggle – everywhere! Huge shoutout to Omar Sanseviero and the Google team for this incredible release and for supporting the open-source community from day one! Colin aka Bartowski, was right, "The best thing about Gemma is the fact that Google specifically helped the open source communities to get day one support." This is how you do open source right!Next up, we have EuroBERT, a new family of multilingual encoder models. Wolfram, our European representative, was particularly excited about this one: "In European languages, you have different characters than in other languages. And, um, yeah, encoding everything properly is, uh, difficult." Ranging from 210 million to 2.1 billion parameters, EuroBERT is designed to push the boundaries of NLP in European and global languages. With training on a massive 5 trillion-token dataset across 15 languages and support for 8K context tokens, EuroBERT is a workhorse for RAG and other NLP tasks. Plus, how cool is their mascot?Reka Flash 3 - a 21B reasoner with apache 2 trained with RLOOAnd the open source train keeps rolling! Reka AI dropped Reka Flash 3, a 21 billion parameter reasoning model with an Apache 2.0 license! Nisten was blown away by the benchmarks: "This might be one of the best like 20B size models that there is right now. And it's Apache 2.0. Uh, I, I think this is a much bigger deal than most people realize." Reka Flash 3 is compact, efficient, and excels at chat, coding, instruction following, and function calling. They even used a new reinforcement learning technique called REINFORCE Leave One-Out (RLOO). Go give it a whirl on Hugging Face or their chat interface – chat.reka.ai!Last but definitely not least in the open-source realm, we had a special guest, Sandra (@itsSandraKublik) from Cohere, join us to announce Command-A! This beast of a model clocks in at 111 BILLION parameters with a massive 256K context window. Sandra emphasized its efficiency, "It requires only two GPUs. Typically the models of this size require 32 GPUs. So it's a huge, huge difference." Command-A is designed for enterprises, focusing on agentic tasks, tool use, and multilingual performance. It's optimized for private deployments and boasts enterprise-grade security. Congrats to Sandra and the Cohere team on this massive release!Big CO LLMs + APIs: Gemini Flash Gets Visual, Deep Research Goes Free, and OpenAI Builds for AgentsThe big companies weren't sleeping either! Google continued their awesome week by unleashing native image generation in Gemini Flash Experimental! This is seriously f*****g cool, folks! Sorry for my French, but it's true. You can now directly interact with images, tell Gemini what to do, and it just does it. We even showed it live on the stream, turning ourselves into cat-confetti-birthday-hat-wearing masterpieces! Wolfram was right, "It's also a sign what we will see in, like, Photoshop, for example. Where you, you expect to just talk to it and have it do everything that a graphic designer would be doing." The future of creative tools is HERE.And guess what else Google did? They made Deep Research FREE in the Gemini app and powered by Gemini Thinking! Nisten jumped in to test it live, and we were all impressed. "This is the nicest interface so far that I've seen," he said. Deep Research now digs through HUNDREDS of websites (Nisten's test hit 156!) to give you comprehensive answers, and the interface is slick and user-friendly. Plus, you can export to Google Docs! Intelligence too cheap to meter? Google is definitely pushing that boundary.Last second additions - Allen Institute for AI released OLMo 2 32B - their biggest open model yetJust as I'm writing this, friend of the pod, Nathan from Allen Institute for AI announced the release of a FULLY OPEN OLMo 2, which includes weights, code, dataset, everything and apparently it beats the latest GPT 3.5, GPT 4o mini, and leading open weight models like Qwen and Mistral. Evals look legit, but nore than that, this is an Apache 2 model with everything in place to advance open AI and open science! Check out Nathans tweet for more info, and congrats to Allen team for this awesome release! OpenAI new responses API and Agent ASK with Web, File and CUA toolsOf course, OpenAI wasn't going to let Google have all the fun. They dropped a new SDK for agents called the Responses API. This is a whole new way to build with OpenAI, designed specifically for the agentic era we're entering. They also released three new tools: Web Search, Computer Use Tool, and File Search Tool. The Web Search tool is self-explanatory – finally, built-in web search from OpenAI!The Computer Use Tool, while currently limited in availability, opens up exciting possibilities for agent automation, letting agents interact with computer interfaces. And the File Search Tool gives you a built-in RAG system, simplifying knowledge retrieval from your own files. As always, OpenAI is adapting to the agentic world and giving developers more power.Finally in the big company space, Nous Research released PORTAL, their new Inference API service. Now you can access their awesome models, like Hermes 3 Llama 70B and DeepHermes 3 8B, directly via API. It's great to see more open-source labs offering API access, making these powerful models even more accessible.This Week's Buzz at Weights & Biases: Offsite Hackathon and MCP Mania!This week's "This Week's Buzz" segment comes to you live from Oceanside, California! The whole Weights & Biases team is here for our company offsite. Despite the not-so-sunny California weather (thanks, storm!), it's been an incredible week of meeting colleagues, strategizing, and HACKING!And speaking of hacking, we had an MCP hackathon! After last week's MCP-pilling episode, we were all hyped about Model Context Protocol, and the team didn't disappoint. In just three hours, the innovation was flowing! We saw agents built for WordPress, MCP support integrated into Weave playground, and even MCP servers for Weights & Biases itself! Get ready, folks, because an MCP server for Weights & Biases is COMING SOON! You'll be able to talk to your W&B data like never before. Huge shoutout to the W&B team for their incredible talent and for embracing the agentic future! And in case you missed it, Weights & Biases is now part of the CoreWeave family! Exciting times ahead!Vision & Video: LoRA Video Effects and OpenSora 2.0Moving into vision and video, Remade AI released 8 LoRA video effects for 1X! Remember 1X from Alibaba? Now you can add crazy effects like "squish," "inflate," "deflate," and even "cakeify" to your videos using LoRAs. It's open source and super cool to see video effects becoming trainable and customizable.And in the realm of open-source video generation, OpenSora 2.0 dropped! This 11 billion parameter model claims state-of-the-art video generation trained for just $200,000! They're even claiming performance close to Sora itself on some benchmarks. Nisten checked out the demos, and while we're all a bit jaded now with the rapid pace of video AI, it's still mind-blowing how far we've come. Open source video is getting seriously impressive, seriously fast.AI Art & Diffusion & 3D: ByteDance's Bilingual Seedream 2.0ByteDance, the folks behind TikTok, released Seedream 2.0, a native Chinese-English bilingual image generation foundation model. This model, from ByteDream, excels at text rendering, cultural nuance, and human preference alignment. Seedream 2.0 boasts "powerful general capability," "native bilingual comprehension ability," and "excellent text rendering." It's designed to understand both Chinese and English prompts natively, generating high-quality, culturally relevant images. The examples look stunning, especially its ability to render Chinese text beautifully.Tools: Manus AI Agent, Google AI Studio YouTube Links, and Cursor EmbeddingsFinally, in the tools section, everyone's buzzing about Manus, a new AI research agent. We gave it a try live on the show, asking it to do some research. The UI is slick, and it seems to be using Claude 3.7 behind the scenes. Manus creates a to-do list, browses the web in a real Chrome browser, and even generates files. It's like Operator on steroids. We'll be keeping an eye on Manus and will report back on its performance in future episodes.And Google AI Studio keeps getting better! Now you can drop YouTube links into Google AI Studio, and it will natively understand the video! This is HUGE for video analysis and content understanding. Imagine using this for support, content summarization, and so much more.PHEW! What a week to celebrate two years of ThursdAI! From open source explosions to Gemini's visual prowess and OpenAI's agentic advancements, the AI world is moving faster than ever. As Wolfram aptly put it, "The acceleration, you can feel it." And Nisten reminded us of the incredible journey, "I remember I had early access to GPT-4 32K, and, uh, then... the person for the contract that had given me access, they cut it off because on the one weekend, I didn't realize how expensive it was. So I had to use $180 worth of tokens just trying it out." Now, we have models that are more powerful and more accessible than ever before. Thank you to Wolfram, Nisten, and LDJ for co-hosting and bringing their insights every week. And most importantly, THANK YOU to our amazing community for tuning in, listening, and supporting ThursdAI for two incredible years! We couldn't do it without you. Here's to another year of staying up-to-date so YOU don't have to! Don't forget to subscribe to the podcast, YouTube channel, and newsletter to stay in the loop. And share ThursdAI with a friend – it's the best birthday gift you can give us! Until next week, keep building and keep exploring the amazing world of AI! LET'S GO! This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit sub.thursdai.news/subscribe

In this episode, Jeff Dance, Host and founder of Fresh Consulting, is joined by Jason Thane, Co-founder and CEO of GenUI, and Elisha Terada, Technical Innovation Director at Fresh Consulting and Co-founder of Brancher AI, to discuss the evolution and future of AI agents. They highlight the shift from traditional AI bots to agentic AI, which involves more autonomous decision-making. The conversation covers the implications of decentralized AI technology and its potential to enhance human creativity and productivity. Episode Breakdown: 0:00 - Introduction 0:40 - About the Guests 2:45 - Jason's Background in AI 5:22 - Elisha's Journey into AI 10:23 - Defining AI Agents vs. AI Bots 18:11 - Examples of AI Applications 25:10 - Future of AI Agents and AGI 29:35 -Industries using AI Agents 41:39 - Human-AI Collaboration 45:17 - Reflections on the Future

It has been a wild few weeks and eventful few months in AI: DeepSeek, OpenAI, Stargate, Microsoft, Meta, Amazon, Salesforce, Google, Elon Musk, and more. In all of this, there's a heightened focus on what it takes to train AI models and the importance of open-source AI. This week on the GeekWire Podcast, we get insights from Ali Farhadi, CEO of the Allen Institute for AI (Ai2), the Seattle-based nonprofit that has been innovating in open-source AI since long before it was popular. "If the U.S. wants to maintain its edge ... we have only one way, and that is to promote open approaches, promote open-source solutions," Farhadi says, reflecting on the past few months. "Because no matter how many dollars you're investing in an ecosystem, without communal, global efforts, you're not going to be as fast." Related Coverage and Links: Allen Institute for AI's new open-source iOS AI app runs on-device for secure, private, offline use Allen Institute for AI challenges DeepSeek on key benchmarks with big new open-source AI model Allen Institute for AI’s new model points to items in images, aims to make bigger point in industry New York Times: An Industry Insider Drives an Open Alternative to Big Tech’s A.I. Ken Yeung, "The AI Economy" newsletter: Ai2: The AI House That Paul Allen Built Ai2 Blog: OLMoE, meet iOS Hosted by GeekWire co-founder Todd BishopSee omnystudio.com/listener for privacy information.

Our 199th episode with a summary and discussion of last week's big AI news! Recorded on 02/09/2025 Join our brand new Discord here! https://discord.gg/nTyezGSKwP Hosted by Andrey Kurenkov and Jeremie Harris. Feel free to email us your questions and feedback at contact@lastweekinai.com and/or hello@gladstone.ai Read out our text newsletter and comment on the podcast at https://lastweekin.ai/. In this episode: - OpenAI's deep research feature capability launched, allowing models to generate detailed reports after prolonged inference periods, competing directly with Google's Gemini 2.0 reasoning models.  - France and UAE jointly announce plans to build a massive AI data center in France, aiming to become a competitive player within the AI infrastructure landscape.  - Mistral introduces a mobile app, broadening its consumer AI lineup amidst market skepticism about its ability to compete against larger firms like OpenAI and Google.  - Anthropic unveils 'Constitutional Classifiers,' a method showing strong defenses against universal jailbreaks; they also launched a $20K challenge to find weaknesses. Timestamps + Links: (00:00:00) Intro / Banter (00:02:27) News Preview (00:03:28) Response to listener comments Tools & Apps (00:08:01) OpenAI now reveals more of its o3-mini model's thought process (00:16:03) Google's Gemini app adds access to ‘thinking' AI models (00:21:04) OpenAI Unveils A.I. Tool That Can Do Research Online (00:31:09) Mistral releases its AI assistant on iOS and Android (00:36:17) AI music startup Riffusion launches its service in public beta (00:39:11) Pikadditions by Pika Labs lets users seamlessly insert objects into videos Applications & Business (00:41:19) Softbank set to invest $40 billion in OpenAI at $260 billion valuation, sources say (00:47:36) UAE to invest billions in France AI data centre (00:50:34) Report: Ilya Sutskever's startup in talks to fundraise at roughly $20B valuation (00:52:03) ASML to Ship First Second-Gen High-NA EUV Machine in the Coming Months, Aiming for 2026 Production (00:54:38) NVIDIA's GB200 NVL 72 Shipments Not Under Threat From DeepSeek As Hyperscalers Maintain CapEx; Meanwhile, Trump Tariffs Play Havoc With TSMC's Pricing Strategy Projects & Open Source (00:56:49) The Allen Institute for AI (AI2) Releases Tülu 3 405B: Scaling Open-Weight... (01:00:06) SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model (01:03:56) PhD Knowledge Not Required: A Reasoning Challenge for Large Language Models (01:08:26) OpenEuroLLM: Europe's New Initiative for Open-Source AI Development Research & Advancements (01:10:34) LIMO: Less is More for Reasoning (01:16:39) s1: Simple test-time scaling (01:19:17) ZebraLogic: On the Scaling Limits of LLMs for Logical Reasoning (01:23:55) Streaming DiLoCo with overlapping communication: Towards a Distributed Free Lunch Policy & Safety (01:26:50) US sets AI safety aside in favor of 'AI dominance' (01:29:39) Almost Surely Safe Alignment of Large Language Models at Inference-Time (01:32:02) Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teaming (01:33:16) Anthropic offers $20,000 to whoever can jailbreak its new AI safety system

Dylan Patel is the founder of SemiAnalysis, a research & analysis company specializing in semiconductors, GPUs, CPUs, and AI hardware. Nathan Lambert is a research scientist at the Allen Institute for AI (Ai2) and the author of a blog on AI called Interconnects. Thank you for listening ❤ Check out our sponsors: https://lexfridman.com/sponsors/ep459-sc See below for timestamps, and to give feedback, submit questions, contact Lex, etc. CONTACT LEX: Feedback - give feedback to Lex: https://lexfridman.com/survey AMA - submit questions, videos or call-in: https://lexfridman.com/ama Hiring - join our team: https://lexfridman.com/hiring Other - other ways to get in touch: https://lexfridman.com/contact EPISODE LINKS: Dylan's X: https://x.com/dylan522p SemiAnalysis: https://semianalysis.com/ Nathan's X: https://x.com/natolambert Nathan's Blog: https://www.interconnects.ai/ Nathan's Podcast: https://www.interconnects.ai/podcast Nathan's Website: https://www.natolambert.com/ Nathan's YouTube: https://youtube.com/@natolambert Nathan's Book: https://rlhfbook.com/ SPONSORS: To support this podcast, check out our sponsors & get discounts: Invideo AI: AI video generator. Go to https://invideo.io/i/lexpod GitHub: Developer platform and AI code editor. Go to https://gh.io/copilot Shopify: Sell stuff online. Go to https://shopify.com/lex NetSuite: Business management software. Go to http://netsuite.com/lex AG1: All-in-one daily nutrition drinks. Go to https://drinkag1.com/lex OUTLINE: (00:00) - Introduction (13:28) - DeepSeek-R1 and DeepSeek-V3 (35:02) - Low cost of training (1:01:19) - DeepSeek compute cluster (1:08:52) - Export controls on GPUs to China (1:19:10) - AGI timeline (1:28:35) - China's manufacturing capacity (1:36:30) - Cold war with China (1:41:00) - TSMC and Taiwan (2:04:38) - Best GPUs for AI (2:19:30) - Why DeepSeek is so cheap (2:32:49) - Espionage (2:41:52) - Censorship (2:54:46) - Andrej Karpathy and magic of RL (3:05:17) - OpenAI o3-mini vs DeepSeek r1 (3:24:25) - NVIDIA (3:28:53) - GPU smuggling (3:35:30) - DeepSeek training on OpenAI data (3:45:59) - AI megaclusters (4:21:21) - Who wins the race to AGI? (4:31:34) - AI agents (4:40:16) - Programming and AI (4:47:43) - Open source (4:56:55) - Stargate (5:04:24) - Future of AI PODCAST LINKS: - Podcast Website: https://lexfridman.com/podcast - Apple Podcasts: https://apple.co/2lwqZIr - Spotify: https://spoti.fi/2nEwCF8 - RSS: https://lexfridman.com/feed/podcast/ - Podcast Playlist: https://www.youtube.com/playlist?list=PLrAXtmErZgOdP_8GztsuKi9nrraNbKKp4 - Clips Channel: https://www.youtube.com/lexclips

This week on the GeekWire Podcast, we dive deep into DeepSeek, the AI project that shaking up the tech world, to better understand the underlying technical advances and the long-term implications for the industry. Joining us is Bill Howe, an associate professor at the University of Washington's Information School and the co-founding director of the UW Center for Responsible AI Systems and Experiences, among other UW roles. Related stories: DeepSeek’s new model shows that AI expertise might matter more than compute in 2025 Allen Institute for AI challenges DeepSeek on key benchmarks with big new open-source AI model Microsoft CEO says AI use will ‘skyrocket’ with more efficiency amid craze over DeepSeek Who will win in AI? DeepSeek’s breakthrough stirs questions around value capture We open the show from the Microsoft campus in Redmond, after getting an inside look at the company's history for an upcoming installment in our Microsoft @ 50 series. John marvels at the size of new campus project, which is still under way, and we experience first-hand the company's vast parking garage when we try to leave. Also on our agenda this week: Amazon's lawsuit against Washington state over a Washington Post public records request, and what it says about the conflicts inherent to Amazon founder Jeff Bezos' ownership of the newspaper. Related story: Bezos vs. Bezos: Amazon sues WA state over Washington Post request for Kuiper records With GeekWire's Todd Bishop and John Cook. Edited by Curt Milton. See omnystudio.com/listener for privacy information.

Being able to change to meet one's circumstances is essential to survival. As HG Wells famously wrote: “adapt or perish.” In this week's episode, both of our storytellers find themselves in unfamiliar territory and need to change course. Part 1: As the only American, microbiologist Chris Robinson struggles to make friends with the other researchers in Chernobyl. Part 2: In his quest to study the adaptability of stickleback fish, neuroscientist Ashwin Bhandiwad keeps needing to adjust his experiment with each new hurdle. Chris Robinson is a published writer and PAm-Costco USA Scholar in the midst of his PhD at Indiana University. His research uses the honey bee as a model to study the ecology and evolution of the gut microbiome and how evolutionary adaptations, such as antibiotic resistance, are transmitted by mobile genetic elements. Originally from the Lowcountry of South Carolina, Chris has harvested watermelon with the USDA, spent a few years as a line cook in Charleston kitchens, and was formally a Fulbright Research Fellow in Ukraine. When not staring at a computer screen, Chris can be found deep into a bicycle ride, playing in the garden, or lamenting the failure of some baking experiment. Ashwin Bhandiwad has spent a remarkable amount of time trying to understand how the brain is organized. Once called "the most handsome boy in the world" by his mom, Ashwin is now a scientist at the Allen Institute for Brain Science working on developing tools to create maps of the brain. Ashwin received his PhD in Psychology from the University of Washington where he investigated how loud noise causes damage in the inner ear. Simultaneously, he disregarded that research by attending many loud concerts. Ashwin also loves swimming, starting projects that he'll never finish, and talking in silly voices to his young children. Learn more about your ad choices. Visit megaphone.fm/adchoices

Subscribe to our channel: https://www.youtube.com/@optispan We explore the psychological and health implications of hypochondriasis, especially in the context of biological age testing and wearables. We then delve into how anxiety and fixation on health metrics can influence behavior and potentially exacerbate health risks. This discussion transitions to a groundbreaking study on brain aging from the Allen Institute, highlighting insights from single-cell transcriptomics, particularly inflammation and neurovascular changes as drivers of aging. Finally, we discuss practical takeaways like the importance of exercise, stress management, and environmental enrichment in promoting healthy brain aging while reflecting on broader societal impacts of health-focused influencers and diagnostics. 0:00 - Study on hypochondriasis and mortality risks1:20 - Biological age testing and psychological implications4:30 - Wearables and their psychological effects6:10 - Influence of biohacking on health anxieties8:10 - Concluding thoughts on diagnostics and anxiety12:30 - New study on brain aging by Allen Institute14:00 - Methodology: Single-cell transcriptomics in brain aging16:30 - Insights on inflammation and glial cells in aging19:45 - Implications for neurovascular interactions and aging22:30 - Practical advice: Sleep, stress, and physical exercise25:15 - Role of environmental enrichment in cognitive aging Producers: Tara Mei, Nicholas Arapis Video Editor: Jacob Keliikoa DISCLAIMER: The information provided on the Optispan podcast is intended solely for general educational purposes and is not meant to be, nor should it be construed as, personalized medical advice. No doctor-patient relationship is established by your use of this channel. The information and materials presented are for informational purposes only and are not a substitute for professional medical advice, diagnosis, or treatment. We strongly advise that you consult with a licensed healthcare professional for all matters concerning your health, especially before undertaking any changes based on content provided by this channel. The hosts and guests on this channel are not liable for any direct, indirect, or other damages or adverse effects that may arise from the application of the information discussed. Medical knowledge is constantly evolving; therefore, the information provided should be verified against current medical standards and practices. More places to find us: Twitter: https://twitter.com/optispanpodcast Twitter: https://twitter.com/optispan Twitter: https://twitter.com/mkaeberlein Linkedin: https://www.linkedin.com/company/optispan Instagram: https://www.instagram.com/optispanpodcast/ TikTok: https://www.tiktok.com/@optispan https://www.optispan.life/ Hi, I'm Matt Kaeberlein. I spent the first few decades of my career doing scientific research into the biology of aging, trying to understand the finer details of how humans age in order to facilitate translational interventions that promote healthspan and improve quality of life. Now I want to take some of that knowledge out of the lab and into the hands of people who can really use it. On this podcast I talk about all things aging and healthspan, from supplements and nutrition to the latest discoveries in longevity research. My goal is to lift the veil on the geroscience and longevity world and help you apply what we know to your own personal health trajectory. I care about quality science and will always be honest about what I don't know. I hope you'll find these episodes helpful!

Mallory continues her series of conversations with female scientists in this special episode featuring yet another WVU alumna, Dr. Rachel Hostetler. In their discussion, Rachel shares insights into her diverse academic interests, her decision to transition directly from graduate school into the private sector, and the exciting real-world applications she's involved with at the Allen Institute. Tune in to hear more about her inspiring career journey and the cutting-edge work shaping the future of science!

Happy holidays! We'll be sharing snippets from Latent Space LIVE! through the break bringing you the best of 2024! We want to express our deepest appreciation to event sponsors AWS, Daylight Computer, Thoth.ai, StrongCompute, Notable Capital, and most of all our LS supporters who helped fund the venue and A/V production!For NeurIPS last year we did our standard conference podcast coverage interviewing selected papers (that we have now also done for ICLR and ICML), however we felt that we could be doing more to help AI Engineers 1) get more industry-relevant content, and 2) recap 2024 year in review from experts. As a result, we organized the first Latent Space LIVE!, our first in person miniconference, at NeurIPS 2024 in Vancouver.Since Nathan Lambert ( Interconnects ) joined us for the hit RLHF 201 episode at the start of this year, it is hard to overstate how much Open Models have exploded this past year. In 2023 only five names were playing in the top LLM ranks, Mistral, Mosaic's MPT, TII UAE's Falcon, Yi from Kai-Fu Lee's 01.ai, and of course Meta's Llama 1 and 2. This year a whole cast of new open models have burst on the scene, from Google's Gemma and Cohere's Command R, to Alibaba's Qwen and Deepseek models, to LLM 360 and DCLM and of course to the Allen Institute's OLMo, OL MOE, Pixmo, Molmo, and Olmo 2 models. We were honored to host Luca Soldaini, one of the research leads on the Olmo series of models at AI2.Pursuing Open Model research comes with a lot of challenges beyond just funding and access to GPUs and datasets, particularly the regulatory debates this year across Europe, California and the White House. We also were honored to hear from and Sophia Yang, head of devrel at Mistral, who also presented a great session at the AI Engineer World's Fair Open Models track!Full Talk on YouTubePlease like and subscribe!Timestamps* 00:00 Welcome to Latent Space Live * 00:12 Recap of 2024: Best Moments and Keynotes * 01:22 Explosive Growth of Open Models in 2024 * 02:04 Challenges in Open Model Research * 02:38 Keynote by Luca Soldani: State of Open Models * 07:23 Significance of Open Source AI Licenses * 11:31 Research Constraints and Compute Challenges * 13:46 Fully Open Models: A New Trend * 27:46 Mistral's Journey and Innovations * 32:57 Interactive Demo: Lachat Capabilities * 36:50 Closing Remarks and NetworkingTranscriptSession3Audio[00:00:00] AI Charlie: Welcome to Latent Space Live, our first mini conference held at NeurIPS 2024 in Vancouver. This is Charlie, your AI co host. As a special treat this week, we're recapping the best of 2024 going domain by domain. We sent out a survey to the over 900 of you who told us what you wanted, and then invited the best speakers in the latent space network to cover each field.[00:00:28] AI Charlie: 200 of you joined us in person throughout the day, with over 2, 200 watching live online. Our next keynote covers the state of open models in 2024, with Luca Soldani and Nathan Lambert of the Allen Institute for AI, with a special appearance from Dr. Sophia Yang of Mistral. Our first hit episode of 2024 was with Nathan Lambert on RLHF 201 back in January.[00:00:57] AI Charlie: Where he discussed both reinforcement learning for language [00:01:00] models and the growing post training and mid training stack with hot takes on everything from constitutional AI to DPO to rejection sampling and also previewed the sea change coming to the Allen Institute. And to Interconnects, his incredible substack on the technical aspects of state of the art AI training.[00:01:18] AI Charlie: We highly recommend subscribing to get access to his Discord as well. It is hard to overstate how much open models have exploded this past year. In 2023, only five names were playing in the top LLM ranks. Mistral, Mosaics MPT, and Gatsby. TII UAE's Falcon, Yi, from Kaifu Lee's 01. ai, And of course, Meta's Lama 1 and 2.[00:01:43] AI Charlie: This year, a whole cast of new open models have burst on the scene. From Google's Jemma and Cohere's Command R, To Alibaba's Quen and DeepSeq models, to LLM360 and DCLM, and of course, to the Allen Institute's OLMO, [00:02:00] OLMOE, PIXMO, MOLMO, and OLMO2 models. Pursuing open model research comes with a lot of challenges beyond just funding and access to GPUs and datasets, particularly the regulatory debates this year across Europe.[00:02:14] AI Charlie: California and the White House. We also were honored to hear from Mistral, who also presented a great session at the AI Engineer World's Fair Open Models track. As always, don't forget to check the show notes for the YouTube link to their talk, as well as their slides. Watch out and take care.[00:02:35] Luca Intro[00:02:35] Luca Soldaini: Cool. Yeah, thanks for having me over. I'm Luca. I'm a research scientist at the Allen Institute for AI. I threw together a few slides on sort of like a recap of like interesting themes in open models for, for 2024. Have about maybe 20, 25 minutes of slides, and then we can chat if there are any questions.[00:02:57] Luca Soldaini: If I can advance to the next slide. [00:03:00] Okay, cool. So I did the quick check of like, to sort of get a sense of like, how much 2024 was different from 2023. So I went on Hugging Face and sort of get, tried to get a picture of what kind of models were released in 2023 and like, what do we get in 2024?[00:03:16] Luca Soldaini: 2023 we get, we got things like both LLAMA 1 and 2, we got Mistral, we got MPT, Falcon models, I think the YI model came in at the end. Tail end of the year. It was a pretty good year. But then I did the same for 2024. And it's actually quite stark difference. You have models that are, you know, reveling frontier level.[00:03:38] Luca Soldaini: Performance of what you can get from closed models from like Quen, from DeepSeq. We got Llama3. We got all sorts of different models. I added our own Olmo at the bottom. There's this growing group of like, Fully open models that I'm going to touch on a little bit later. But you know, just looking at the slides, it feels like 2024 [00:04:00] was just smooth sailing, happy knees, much better than previous year.[00:04:04] Luca Soldaini: And you know, you can plot you can pick your favorite benchmark Or least favorite, I don't know, depending on what point you're trying to make. And plot, you know, your closed model, your open model and sort of spin it in ways that show that, oh, you know open models are much closer to where closed models are today versus to Versus last year where the gap was fairly significant.[00:04:29] Luca Soldaini: So one thing that I think I don't know if I have to convince people in this room, but usually when I give this talks about like open models, there is always like this background question in, in, in people's mind of like, why should we use open models? APIs argument, you know, it's, it's. Just an HTTP request to get output from a, from one of the best model out there.[00:04:53] Luca Soldaini: Why do I have to set up infra and use local models? And there are really like two answer. There is the more [00:05:00] researchy answer for this, which is where it might be. Background lays, which is just research. If you want to do research on language models, research thrives on, on open models, there is like large swath of research on modeling, on how these models behave on evaluation and inference on mechanistic interpretability that could not happen at all if you didn't have open models they're also for AI builders, they're also like.[00:05:30] Luca Soldaini: Good use cases for using local models. You know, you have some, this is like a very not comprehensive slides, but you have things like there are some application where local models just blow closed models out of the water. So like retrieval, it's a very clear example. We might have like constraints like Edge AI applications where it makes sense.[00:05:51] Luca Soldaini: But even just like in terms of like stability, being able to say this model is not changing under the hood. It's, there's plenty of good cases for, [00:06:00] for open models. And the community is just not models. Is I stole this slide from one of the Quent2 announcement blog posts. But it's super cool to see like how much tech exists around open models and serving them on making them efficient and hosting them.[00:06:18] Luca Soldaini: It's pretty cool. And so. It's if you think about like where the term opens come from, comes from like the open source really open models meet the core tenants of, of open, of open source specifically when it comes around collaboration, there is truly a spirit, like through these open models, you can build on top of other people.[00:06:41] Luca Soldaini: innovation. We see a lot of these even in our own work of like, you know, as we iterate in the various versions of Alma it's not just like every time we collect from scratch all the data. No, the first step is like, okay, what are the cool data sources and datasets people have put [00:07:00] together for language model for training?[00:07:01] Luca Soldaini: Or when it comes to like our post training pipeline We one of the steps is you want to do some DPO and you use a lot of outputs of other models to improve your, your preference model. So it's really having like an open sort of ecosystem benefits and accelerates the development of open models.[00:07:23] The Definition of Open Models[00:07:23] Luca Soldaini: One thing that we got in 2024, which is not a specific model, but I thought it was really significant, is we first got we got our first open source AI definition. So this is from the open source initiative they've been generally the steward of a lot of the open source licenses when it comes to software and so they embarked on this journey in trying to figure out, okay, How does a license, an open source license for a model look like?[00:07:52] Luca Soldaini: Majority of the work is very dry because licenses are dry. So I'm not going to walk through the license step by [00:08:00] step, but I'm just going to pick out one aspect that is very good and then one aspect that personally feels like it needs improvement on the good side. This this open source AI license actually.[00:08:13] Luca Soldaini: This is very intuitive. If you ever build open source software and you have some expectation around like what open source looks like for software for, for AI, sort of matches your intuition. So, the weights need to be fairly available the code must be released with an open source license and there shouldn't be like license clauses that block specific use cases.[00:08:39] Luca Soldaini: So. Under this definition, for example, LLAMA or some of the QUEN models are not open source because the license says you can't use this model for this or it says if you use this model you have to name the output this way or derivative needs to be named that way. Those clauses don't meet open source [00:09:00] definition and so they will not be covered.[00:09:02] Luca Soldaini: The LLAMA license will not be covered under the open source definition. It's not perfect. One of the thing that, um, internally, you know, in discussion with with OSI, we were sort of disappointed is around the language. For data. So you might imagine that an open source AI model means a model where the data is freely available.[00:09:26] Luca Soldaini: There were discussion around that, but at the end of the day, they decided to go with a softened stance where they say a model is open source if you provide sufficient detail information. On how to sort of replicate the data pipeline. So you have an equivalent system, sufficient, sufficiently detailed.[00:09:46] Luca Soldaini: It's very, it's very fuzzy. Don't like that. An equivalent system is also very fuzzy. And this doesn't take into account the accessibility of the process, right? It might be that you provide enough [00:10:00] information, but this process costs, I don't know, 10 million to do. Now the open source definition. Like, any open source license has never been about accessibility, so that's never a factor in open source software, how accessible software is.[00:10:14] Luca Soldaini: I can make a piece of open source, put it on my hard drive, and never access it. That software is still open source, the fact that it's not widely distributed doesn't change the license, but practically there are expectations of like, what we want good open sources to be. So, it's, It's kind of sad to see that the data component in this license is not as, as, Open as some of us would like would like it to be.[00:10:40] Challenges for Open Models[00:10:40] Luca Soldaini: and I linked a blog post that Nathan wrote on the topic that it's less rambly and easier to follow through. One thing that in general, I think it's fair to say about the state of open models in 2024 is that we know a lot more than what we knew in, [00:11:00] in 2023. Like both on the training data, like And the pre training data you curate on like how to do like all the post training, especially like on the RL side.[00:11:10] Luca Soldaini: You know, 2023 was a lot of like throwing random darts at the board. I think 2024, we have clear recipes that, okay, don't get the same results as a closed lab because there is a cost in, in actually matching what they do. But at least we have a good sense of like, okay, this is, this is the path to get state of the art language model.[00:11:31] Luca Soldaini: I think that one thing that it's a downside of 2024 is that I think we are more research constrained in 2023. It feels that, you know, the barrier for compute that you need to, to move innovation along as just being right rising and rising. So like, if you go back to this slide, there is now this, this cluster of models that are sort of released by the.[00:11:57] Luca Soldaini: Compute rich club. Membership is [00:12:00] hotly debated. You know, some people don't want to be. Called the rich because it comes to expectations. Some people want to be called rich, but I don't know, there's debate, but like, these are players that have, you know, 10, 000, 50, 000 GPUs at minimum. And so they can do a lot of work and a lot of exploration and improving models that it's not very accessible.[00:12:21] Luca Soldaini: To give you a sense of like how I personally think about. Research budget for each part of the, of the language model pipeline is like on the pre training side, you can maybe do something with a thousand GPUs, really you want 10, 000. And like, if you want real estate of the art, you know, your deep seek minimum is like 50, 000 and you can scale to infinity.[00:12:44] Luca Soldaini: The more you have, the better it gets. Everyone on that side still complains that they don't have enough GPUs. Post training is a super wide sort of spectrum. You can do as little with like eight GPUs as long as you're able to [00:13:00] run, you know, a good version of, say, a LLAMA model, you can do a lot of work there.[00:13:05] Luca Soldaini: You can scale a lot of the methodology, just like scales with compute, right? If you're interested in you know, your open replication of what OpenAI's O1 is you're going to be on the 10K spectrum of our GPUs. Inference, you can do a lot with very few resources. Evaluation, you can do a lot with, well, I should say at least one GPUs if you want to evaluate GPUs.[00:13:30] Luca Soldaini: Open models but in general, like if you are, if you care a lot about intervention to do on this model, which it's my prefer area of, of research, then, you know, the resources that you need are quite, quite significant. Yeah. One other trends that has emerged in 2024 is this cluster of fully open models.[00:13:54] Luca Soldaini: So Omo the model that we built at ai, two being one of them and you know, it's nice [00:14:00] that it's not just us. There's like a cluster of other mostly research efforts who are working on this. And so it's good to to give you a primer of what like fully open means. So fully open, the easy way to think about it is instead of just releasing a model checkpoint that you run, you release a full recipe so that other people working on it.[00:14:24] Luca Soldaini: Working on that space can pick and choose whatever they want from your recipe and create their own model or improve on top of your model. You're giving out the full pipeline and all the details there instead of just like the end output. So I pull up the screenshot from our recent MOE model.[00:14:43] Luca Soldaini: And like for this model, for example, we released the model itself. Data that was trained on, the code, both for training and inference all the logs that we got through the training run, as well as every intermediate checkpoint and like the fact that you release different part of the pipeline [00:15:00] allows others to do really cool things.[00:15:02] Luca Soldaini: So for example, this tweet from early this year from folks in news research they use our pre training data to do a replication of the BitNet paper in the open. So they took just a Really like the initial part of a pipeline and then the, the thing on top of it. It goes both ways.[00:15:21] Luca Soldaini: So for example, for the Olmo2 model a lot of our pre trained data for the first stage of pre training was from this DCLM initiative that was led by folks Ooh, a variety of ins a variety of institutions. It was a really nice group effort. But you know, for When it was nice to be able to say, okay, you know, the state of the art in terms of like what is done in the open has improved.[00:15:46] AI2 Models - Olmo, Molmo, Pixmo etc[00:15:46] Luca Soldaini: We don't have to like do all this work from scratch to catch up the state of the art. We can just take it directly and integrate it and do our own improvements on top of that. I'm going to spend a few minutes doing like a [00:16:00] shameless plug for some of our fully open recipes. So indulge me in this.[00:16:05] Luca Soldaini: So a few things that we released this year was, as I was mentioning, there's OMOE model which is, I think still is state of the art MOE model in its size class. And it's also. Fully open, so every component of this model is available. We released a multi modal model called Molmo. Molmo is not just a model, but it's a full recipe of how you go from a text only model to a multi modal model, and we apply this recipe on top of Quent checkpoints, on top of Olmo checkpoints, as well as on top of OlmoE.[00:16:37] Luca Soldaini: And I think there'd be a replication doing that on top of Mistral as well. The post training side we recently released 2. 0. 3. Same story. This is a recipe on how you go from a base model to A state of the art post training model. We use the Tulu recipe on top of Olmo, on top of Llama, and then there's been open replication effort [00:17:00] to do that on top of Quen as well.[00:17:02] Luca Soldaini: It's really nice to see like, you know, when your recipe sort of, it's kind of turnkey, you can apply it to different models and it kind of just works. And finally, the last thing we released this year was Olmo 2, which so far is the best state of the art. Fully open language model a Sera combines aspect from all three of these previous models.[00:17:22] Luca Soldaini: What we learn on the data side from MomoE and what we learn on like making models that are easy to adapt from the Momo project and the Tulu project. I will close with a little bit of reflection of like ways this, this ecosystem of open models like it's not all roses. It's not all happy. It feels like day to day, it's always in peril.[00:17:44] Luca Soldaini: And, you know, I talked a little bit about like the compute issues that come with it. But it's really not just compute. One thing that is on top of my mind is due to like the environment and how you know, growing feelings about like how AI is treated. [00:18:00] It's actually harder to get access to a lot of the data that was used to train a lot of the models up to last year.[00:18:06] Luca Soldaini: So this is a screenshot from really fabulous work from Shane Longpre who's, I think is in Europe about Just access of like diminishing access to data for language model pre training. So what they did is they went through every snapshot of common crawl. Common crawl is this publicly available scrape of the, of a subset of the internet.[00:18:29] Luca Soldaini: And they looked at how For any given website whether a website that was accessible in say 2017, what, whether it was accessible or not in 2024. And what they found is as a reaction to like the close like of the existence of closed models like OpenAI or Cloud GPT or Cloud a lot of content owners have blanket Blocked any type of crawling to your website.[00:18:57] Luca Soldaini: And this is something that we see also internally at [00:19:00] AI2. Like one project that we started this year is we wanted to, we wanted to understand, like, if you're a good citizen of the internet and you crawl following sort of norms and policy that have been established in the last 25 years, what can you crawl?[00:19:17] Luca Soldaini: And we found that there's a lot of website where. The norms of how you express preference of whether to crawl your data or not are broken. A lot of people would block a lot of crawling, but do not advertise that in RobustDXT. You can only tell that they're crawling, that they're blocking you in crawling when you try doing it.[00:19:37] Luca Soldaini: Sometimes you can't even crawl the robots. txt to, to check whether you're allowed or not. And then a lot of websites there's, there's like all these technologies that historically have been, have existed to make websites serving easier such as Cloudflare or DNS. They're now being repurposed for blocking AI or any type of crawling [00:20:00] in a way that is Very opaque to the content owners themselves.[00:20:04] Luca Soldaini: So, you know, you go to these websites, you try to access them and they're not available and you get a feeling it's like, Oh, someone changed, something changed on the, on the DNS side that it's blocking this and likely the content owner has no idea. They're just using a Cloudflare for better, you know, load balancing.[00:20:25] Luca Soldaini: And this is something that was sort of sprung on them with very little notice. And I think the problem is this, this blocking or ideas really, it impacts people in different ways. It disproportionately helps companies that have a headstart, which are usually the closed labs and it hurts incoming newcomer players where either have now to do things in a sketchy way or you're never going to get that content that the closed lab might have.[00:20:54] Luca Soldaini: So there's a lot, it was a lot of coverage. I'm going to plug Nathan's blog post again. That is, [00:21:00] that I think the title of this one is very succinct which is like, we're actually not, You know, before thinking about running out of training data, we're actually running out of open training data. And so if we want better open models they should be on top of our mind.[00:21:13] Regulation and Lobbying[00:21:13] Luca Soldaini: The other thing that has emerged is that there is strong lobbying efforts on trying to define any kind of, AI as like a new extremely risky and I want to be precise here. Like the problem is now, um, like the problem is not not considering the risk of this technology. Every technology has risks that, that should always be considered.[00:21:37] Luca Soldaini: The thing that it's like to me is sorry, is ingenious is like just putting this AI on a pedestal and calling it like, An unknown alien technology that has like new and undiscovered potentials to destroy humanity. When in reality, all the dangers I think are rooted in [00:22:00] dangers that we know from existing software industry or existing issues that come with when using software on on a lot of sensitive domains, like medical areas.[00:22:13] Luca Soldaini: And I also noticed a lot of efforts that have actually been going on and trying to make this open model safe. I pasted one here from AI2, but there's actually like a lot of work that has been going on on like, okay, how do you make, if you're distributing this model, Openly, how do you make it safe?[00:22:31] Luca Soldaini: How, what's the right balance between accessibility on open models and safety? And then also there's annoying brushing of sort of concerns that are then proved to be unfounded under the rug. You know, if you remember the beginning of this year, it was all about bio risk of these open models.[00:22:48] Luca Soldaini: The whole thing fizzled because as being Finally, there's been like rigorous research, not just this paper from Cohere folks, but it's been rigorous research showing [00:23:00] that this is really not a concern that we should be worried about. Again, there is a lot of dangerous use of AI applications, but this one was just like, A lobbying ploy to just make things sound scarier than they actually are.[00:23:15] Luca Soldaini: So I got to preface this part. It says, this is my personal opinion. It's not my employer, but I look at things like the SP 1047 from, from California. And I think we kind of dodged a bullet on, on this legislation. We, you know, the open source community, a lot of the community came together at the last, sort of the last minute and did a very good effort trying to explain all the negative impact of this bill.[00:23:43] Luca Soldaini: But There's like, I feel like there's a lot of excitement on building these open models or like researching on these open models. And lobbying is not sexy it's kind of boring but it's sort of necessary to make sure that this ecosystem can, can really [00:24:00] thrive. This end of presentation, I have Some links, emails, sort of standard thing in case anyone wants to reach out and if folks have questions or anything they wanted to discuss.[00:24:13] Luca Soldaini: Is there an open floor? I think we have Sophia[00:24:16] swyx: who wants to who one, one very important open model that we haven't covered is Mistral. Ask her on this slide. Yeah, yeah. Well, well, it's nice to have the Mistral person talk recap the year in Mistral. But while Sophia gets set up, does anyone have like, just thoughts or questions about the progress in this space?[00:24:32] Questions - Incentive Alignment[00:24:32] swyx: Do you always have questions?[00:24:34] Quesiton: I'm very curious how we should build incentives to build open models, things like Francois Chollet's ArcPrize, and other initiatives like that. What is your opinion on how we should better align incentives in the community so that open models stay open?[00:24:49] Luca Soldaini: The incentive bit is, like, really hard.[00:24:51] Luca Soldaini: Like, even It's something that I actually, even we think a lot about it internally because like building open models is risky. [00:25:00] It's very expensive. And so people don't want to take risky bets. I think the, definitely like the challenges like our challenge, I think those are like very valid approaches for it.[00:25:13] Luca Soldaini: And then I think in general, promoting, building, so, any kind of effort to participate in this challenge, in those challenges, if we can promote doing that on top of open models and sort of really lean into like this multiplier effect, I think that is a good way to go. If there were more money for that.[00:25:35] Luca Soldaini: For efforts like research efforts around open models. There's a lot of, I think there's a lot of investments in companies that at the moment are releasing their model in the open, which is really cool. But it's usually more because of commercial interest and not wanting to support this, this like open models in the longterm, it's a really hard problem because I think everyone is operating sort of [00:26:00] in what.[00:26:01] Luca Soldaini: Everyone is at their local maximum, right? In ways that really optimize their position on the market. Global maximum is harder to achieve.[00:26:11] Question2: Can I ask one question? No.[00:26:12] Luca Soldaini: Yeah.[00:26:13] Question2: So I think one of the gap between the closed and open source models is the mutability. So the closed source models like chat GPT works pretty good on the low resource languages, which is not the same on the open, open source models, right?[00:26:27] Question2: So is it in your plan to improve on that?[00:26:32] Luca Soldaini: I think in general,[00:26:32] Luca Soldaini: yes, is I think it's. I think we'll see a lot of improvements there in, like, 2025. Like, there's groups like, Procurement English on the smaller side that are already working on, like, better crawl support, multilingual support. I think what I'm trying to say here is you really want to be experts.[00:26:54] Luca Soldaini: who are actually in those countries that teach those languages to [00:27:00] participate in the international community. To give you, like, a very easy example I'm originally from Italy. I think I'm terribly equipped to build a model that works well in Italian. Because one of the things you need to be able to do is having that knowledge of, like, okay, how do I access, you know, how Libraries, or content that is from this region that covers this language.[00:27:23] Luca Soldaini: I've been in the US long enough that I no longer know. So, I think that's the efforts that folks in Central Europe, for example, are doing. Around like, okay, let's tap into regional communities. To get access you know, to bring in collaborators from those areas. I think it's going to be, like, very crucial for getting products there.[00:27:46] Mistral intro[00:27:46] Sophia Yang: Hi everyone. Yeah, I'm super excited to be here to talk to you guys about Mistral. A really short and quick recap of what we have done, what kind of models and products we have released in the [00:28:00] past year and a half. So most of you We have already known that we are a small startup funded about a year and a half ago in Paris in May, 2003, it was funded by three of our co founders, and in September, 2003, we released our first open source model, Mistral 7b yeah, how, how many of you have used or heard about Mistral 7b?[00:28:24] Sophia Yang: Hey, pretty much everyone. Thank you. Yeah, it's our Pretty popular and community. Our committee really loved this model, and in December 23, we, we released another popular model with the MLE architecture Mr. A X seven B and oh. Going into this year, you can see we have released a lot of things this year.[00:28:46] Sophia Yang: First of all, in February 2004, we released MrSmall, MrLarge, LeChat, which is our chat interface, I will show you in a little bit. We released an embedding model for, you [00:29:00] know, converting your text into embedding vectors, and all of our models are available. The, the big cloud resources. So you can use our model on Google cloud, AWS, Azure Snowflake, IBM.[00:29:16] Sophia Yang: So very useful for enterprise who wants to use our model through cloud. And in April and May this year, we released another powerful open source MOE model, AX22B. And we also released our first code. Code Model Coastal, which is amazing at 80 plus languages. And then we provided another fine tuning service for customization.[00:29:41] Sophia Yang: So because we know the community love to fine tune our models, so we provide you a very nice and easy option for you to fine tune our model on our platform. And also we released our fine tuning code base called Menstrual finetune. It's open source, so feel free to take it. Take a look and.[00:29:58] Sophia Yang: More models. [00:30:00] On July 2, November this year, we released many, many other models. First of all is the two new small, best small models. We have Minestra 3B great for Deploying on edge devices we have Minstrel 8B if you used to use Minstrel 7B, Minstrel 8B is a great replacement with much stronger performance than Minstrel 7B.[00:30:25] Sophia Yang: We also collaborated with NVIDIA and open sourced another model, Nemo 12B another great model. And Just a few weeks ago, we updated Mistral Large with the version 2 with the updated, updated state of the art features and really great function calling capabilities. It's supporting function calling in LatentNate.[00:30:45] Sophia Yang: And we released two multimodal models Pixtral 12b. It's this open source and Pixtral Large just amazing model for, models for not understanding images, but also great at text understanding. So. Yeah, a [00:31:00] lot of the image models are not so good at textual understanding, but pixel large and pixel 12b are good at both image understanding and textual understanding.[00:31:09] Sophia Yang: And of course, we have models for research. Coastal Mamba is built on Mamba architecture and MathRoll, great with working with math problems. So yeah, that's another model.[00:31:29] Sophia Yang: Here's another view of our model reference. We have several premier models, which means these models are mostly available through our API. I mean, all of the models are available throughout our API, except for Ministry 3B. But for the premier model, they have a special license. Minstrel research license, you can use it for free for exploration, but if you want to use it for enterprise for production use, you will need to purchase a license [00:32:00] from us.[00:32:00] Sophia Yang: So on the top row here, we have Minstrel 3b and 8b as our premier model. Minstrel small for best, best low latency use cases, MrLarge is great for your most sophisticated use cases. PixelLarge is the frontier class multimodal model. And, and we have Coastral for great for coding and then again, MrEmbedding model.[00:32:22] Sophia Yang: And The bottom, the bottom of the slides here, we have several Apache 2. 0 licensed open way models. Free for the community to use, and also if you want to fine tune it, use it for customization, production, feel free to do so. The latest, we have Pixtros 3 12b. We also have Mr. Nemo mum, Coastal Mamba and Mastro, as I mentioned, and we have three legacy models that we don't update anymore.[00:32:49] Sophia Yang: So we recommend you to move to our newer models if you are still using them. And then, just a few weeks ago, [00:33:00] we did a lot of, uh, improvements to our code interface, Lachette. How many of you have used Lachette? Oh, no. Only a few. Okay. I highly recommend Lachette. It's chat. mistral. ai. It's free to use.[00:33:16] Sophia Yang: It has all the amazing capabilities I'm going to show you right now. But before that, Lachette in French means cat. So this is actually a cat logo. If you You can tell this is the cat eyes. Yeah. So first of all, I want to show you something Maybe let's, let's take a look at image understanding.[00:33:36] Sophia Yang: So here I have a receipts and I want to ask, just going to get the prompts. Cool. So basically I have a receipt and I said I ordered I don't know. Coffee and the sausage. How much do I owe? Add a 18 percent tip. So hopefully it was able to get the cost of the coffee and the [00:34:00] sausage and ignore the other things.[00:34:03] Sophia Yang: And yeah, I don't really understand this, but I think this is coffee. It's yeah. Nine, eight. And then cost of the sausage, we have 22 here. And then it was able to add the cost, calculate the tip, and all that. Great. So, it's great at image understanding, it's great at OCR tasks. So, if you have OCR tasks, please use it.[00:34:28] Sophia Yang: It's free on the chat. It's also available through our API. And also I want to show you a Canvas example. A lot of you may have used Canvas with other tools before. But, With Lachat, it's completely free again. Here, I'm asking it to create a canvas that's used PyScript to execute Python in my browser.[00:34:51] Sophia Yang: Let's see if it works. Import this. Okay, so, yeah, so basically it's executing [00:35:00] Python here. Exactly what we wanted. And the other day, I was trying to ask Lachat to create a game for me. Let's see if we can make it work. Yeah, the Tetris game. Yep. Let's just get one row. Maybe. Oh no. Okay. All right. You get the idea. I failed my mission. Okay. Here we go. Yay! Cool. Yeah. So as you can see, Lachet can write, like, a code about a simple game pretty easily. And you can ask Lachet to explain the code. Make updates however you like. Another example. There is a bar here I want to move.[00:35:48] Sophia Yang: Okay, great, okay. And let's go back to another one. Yeah, we also have web search capabilities. Like, you can [00:36:00] ask what's the latest AI news. Image generation is pretty cool. Generate an image about researchers. Okay. In Vancouver? Yeah, it's Black Forest Labs flux Pro. Again, this is free, so Oh, cool.[00:36:19] Sophia Yang: I guess researchers here are mostly from University of British Columbia. That's smart. Yeah. So this is Laia ira. Please feel free to use it. And let me know if you have any feedback. We're always looking for improvement and we're gonna release a lot more powerful features in the coming years.[00:36:37] Sophia Yang: Thank you. Get full access to Latent Space at www.latent.space/subscribe

这次依旧是硬核话题，我们跟学术大牛深度聊聊2024年上半年美国创投圈最火的的话题之一，具身智能。 没错，智能机器人之火终于从国内来到美国了。在去年下半年的时候，美国创投界还是在关注大模型和应用、infra等等，虽然Deepmind RT-2 等工作彼时已经崭露头角，更喜欢软件的美国VC似乎还在犹豫机器人这个太硬的赛道。但是从今年上半年开始，事情似乎有了变化。 Hello World, who is OnBoard!? 除了Figure AI 这样的人形机器人公司获得了英伟达、微软等一系列战投的加持，硅谷的老牌基金们也疯狂涌入了所谓的机器人大模型公司，比如学术大牛创立的 Physical intelligence, Skild, 还有 Cruise 前CEO 创立的Bot company, 等等。 这次的嘉宾也是大名鼎鼎，UCSD 计算机科学副教授，苏昊老师，关注具身智能和3D视觉领域的同学应该都不陌生。他参与的一系列AI数据集和软件工作，从ImageNet到ShapeNet、PointNet、SAPIEN，以及最近的ManiSkill等等，都是三维视觉、机器人操作等领域穿越几个时代的标志性作品。 苏昊老师现在还是智能机器人创业公司Hillbot 的联合创始人，我们深度探讨了： 过去一年，我们从学术界、工业界讨论的种种话题，又有了哪些新的进展？ 大模型的发展如何影响具身智能的不同技术路径？ 大模型带来的泛化能力，跟硬件、控制系统等，又会怎样相互作用？ 机器人模型里的数据问题，有哪些解决方案？ 具身智能这个看似很纷繁的话题，苏昊老师总是能抽丝剥茧，相信你们也能从我们两个多小时的交流中，受益匪浅。Enjoy! 对了！今年年初，Onboard 就发布过一期关于具身智能的讨论，嘉宾包括了 Deepmind Robotics，高仙机器人和UCSD 的不同视角的重磅嘉宾。那一期讨论也非常精彩，建议大家回去复习哈！ 嘉宾介绍 苏昊 (Twitter @HaoSuLabUCSD)，UC San Diego Associate Professor，Hillbot智能机器人初创公司创始人、CTO。Stanford PhD, UCSD 具身智能实验室主任，数据科学研究所创始成员，以及视觉计算中心和情境机器人研究所成员。他的研究工作集中在开发算法来模拟、理解并与物理世界互动。 OnBoard! 主持：Monica, 美元VC投资人，前 AWS 硅谷团队+ AI 创业公司打工人，公众号M小姐研习录 (ID: MissMStudy) 主理人 | 即刻：莫妮卡同学 我们都聊了什么 03:04 苏昊的学术历程，为什么最近觉得有关证明的研究进展对机器人领域很有启发？ 10:05 从智能演化的角度，理解“具身智能”这个“老概念” 15:01 为什么从语言而不是视觉上最先看到了接近人类的智能？ 21:31 实现具身智能有哪些主流的路线？如何理解不同路径不同切入点背后的逻辑？ 32:10 可以通过大模型的能力实现运动控制吗？有泛化性的控制数据要怎么采集？ 38:26 演示学习 (learning from demonstration) 有哪些不同路径？ALOHA这类遥操作有什么利弊？ 47:00 规划和执行需要一起做训练吗？做一个端到端的系统核心难点在哪里？ 51:15 划重点：好的算法的本质就是降低对数据的需求 52:23 针对机器人的大模型会跟LLM架构有什么异同？ 59:31 人形机器人可以解决数据和能力泛化的问题吗？ 66:16 模拟器能解决训练数据的问题吗？近年来模拟器相关技术有什么关键进展？ 78:31 AI生成3D，Sora 等新技术进展对实现 sim2real 路径有什么影响？ 95:26 苏昊老师现在的创业项目 Hillbot 100:32 快问快答：推荐的书，影响最大的人，具身智能被高估和低估的话题，如何解压！ 重点词汇和公司 Boston Dynamics PI (Physical Intelligence) ⁠OpenAI DALL-E 3⁠ ⁠SAPIEN: A SimulAted Part-based Interactive ENvironment⁠ ⁠ManiSkill⁠: a powerful unified framework for robot simulation and training powered by SAPIEN. ⁠Google Deepmind RT-1⁠: Robotics Transformer for real-world control at scale ⁠Google Deepmind RT-2⁠: New model translates vision and language into action, ⁠Paper⁠ ⁠Google Deepmind Open X-Embodiment⁠: Robotic Learning Datasets and RT-X Models, ⁠Paper⁠ ⁠ALOHA⁠: A Low-cost Open-source Hardware System for Bimanual Teleoperation ⁠Mobile ALOHA⁠: a low-cost and whole-body teleoperation system for data collection. Behavior Colony：行为克隆 Learning from Demonstration：示范学习 ⁠Meta AI Habitat⁠: A Platform for Embodied AI Research ⁠AI2⁠: The Allen Institute for Artificial Intelligence ⁠Segment Anything Model (SAM)⁠: a new AI model from Meta AI that can "cut out" any object, in any image, with a single click ⁠robot-VILA⁠: Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planning ⁠CoPa⁠: General Robotic Manipulation through Spatial Constraints of Parts with Foundational Model ⁠ImageNet: image database organized according to the WordNet hierarchy⁠ ⁠EP 44.【AI年终特辑3】具身智能深度对话：从学术到产业，机器人的ChatGPT时刻来了吗？ - OnBoard! | 小宇宙⁠ ⁠Debate: Is Scaling Enough to Deploy General Purpose Robots @CoRL2023⁠ ⁠解密机器人大模型RFM-1：Covariant创始人陈曦专访⁠ ⁠对话高阳：具身大模型框架ViLa+CoPa⁠ 参考文章欢迎关注M小姐的微信公众号，了解更多中美软件、AI与创业投资的干货内容！ M小姐研习录 (ID: MissMStudy) 欢迎在评论区留下你的思考，与听友们互动。如果你用 Apple Podcasts 收听，也请给我们一个五星好评，这对我们非常重要。 最后！快来加入Onboard！听友群，结识到高质量的听友们，我们还会组织线下主题聚会，开放实时旁听播客录制，嘉宾互动等新的尝试。添加任意一位小助手微信，onboard666, 或者 Nine_tunes,小助手会拉你进群。期待你来！

This week on the GeekWire Podcast, we sit down with some of the Seattle region’s “Uncommon Thinkers” — inventors, scientists, technologists and entrepreneurs transforming industries and driving positive change in the world. We recorded the episode on location, backstage at the GeekWire Gala, where we recognized five Uncommon Thinkers through this annual awards program, presented in partnership with Greater Seattle Partners. Speaking on the episode are: Uri Shumlak, co-founder and chief scientist at Zap Energy, a physicist leading a team in pursuit of fusion energy, taking a different approach from others in the field. Read the profile. Ingrid Swanson Pultz, CTO at Mopac Biologics, and translational advisor at the UW Institute for Protein Design, a microbiologist who led the development of a gluten-destroying enzyme. Read the profile. Chris Dunckley, director of chemistry and engineering of TerraPower Isotopes, a chemical engineer who leads a team turning radioactive waste into cancer therapy. Read the profile. Andy Lapsa, aerospace engineer and CEO of Stoke Space, a company focused on developing fully and rapidly reusable space vehicles using a liquid cooling technique for re-entry. Read the profile. Also featured in in the Uncommon Thinkers series: Hanna Hajishirzi of the Allen Institute for AI and the UW's Allen School of Computer Science & Engineering, who focuses on open-source Ai models. Read the profile. With GeekWire's Todd Bishop. Edited by Curt Milton.See omnystudio.com/listener for privacy information.

Nathan Lambert of the excellent https://www.interconnects.ai/ newsletter and the Allen Institute joins the pod for a rundown of the biggest AI stories of this year and next. We also talk about what he's learned training advanced AI models at the Allen Institute. Outtro Music: Young and Holtful by Young-Holt Unlimited, 1969. https://open.spotify.com/track/5am0dV7aB91Q6sWqIAuurA?autoplay=true Learn more about your ad choices. Visit megaphone.fm/adchoices

Nathan Lambert of the excellent https://www.interconnects.ai/ newsletter and the Allen Institute joins the pod for a rundown of the biggest AI stories of this year and next. We also talk about what he's learned training advanced AI models at the Allen Institute. Outtro Music: Young and Holtful by Young-Holt Unlimited, 1969. https://open.spotify.com/track/5am0dV7aB91Q6sWqIAuurA?autoplay=true Learn more about your ad choices. Visit megaphone.fm/adchoices

Near death experiences can be profound and even life changing. People describe seeing bright lights, staring into the abyss, or meeting dead relatives. Many believe these experiences to be proof of an afterlife. But now, scientists are studying these strange events and gaining insights into the brain and consciousness itself. Will we uncover the scientific underpinning of these near-death events?  Guests: Steve Paulson - executive producer of To the Best of Our Knowledge for Wisconsin Public Radio Sebastian Junger - journalist, filmmaker and author of “The Perfect Storm: A True Story of Men Against the Sea”  Christoph Koch - neuroscientist at the Allen Institute in Seattle and chief scientist of the Tiny Blue Dot Foundation in Santa Monica California  Daniel Kondziella - neuroscientist in the Department of Clinical Medicine at the University of Copenhagen Featuring music by Dewey Dellay and Jun Miyake Originally aired September 25, 2023 Big Picture Science is part of the Airwave Media podcast network. Please contact advertising@airwavemedia.com to inquire about advertising on Big Picture Science. You can get early access to ad-free versions of every episode by joining us on Patreon. Thanks for your support!   Learn more about your ad choices. Visit megaphone.fm/adchoices

Near death experiences can be profound and even life changing. People describe seeing bright lights, staring into the abyss, or meeting dead relatives. Many believe these experiences to be proof of an afterlife. But now, scientists are studying these strange events and gaining insights into the brain and consciousness itself. Will we uncover the scientific underpinning of these near-death events?  Guests: Steve Paulson - executive producer of To the Best of Our Knowledge for Wisconsin Public Radio Sebastian Junger - journalist, filmmaker and author of “The Perfect Storm: A True Story of Men Against the Sea”  Christoph Koch - neuroscientist at the Allen Institute in Seattle and chief scientist of the Tiny Blue Dot Foundation in Santa Monica California  Daniel Kondziella - neuroscientist in the Department of Clinical Medicine at the University of Copenhagen Featuring music by Dewey Dellay and Jun Miyake Originally aired September 25, 2023 Big Picture Science is part of the Airwave Media podcast network. Please contact advertising@airwavemedia.com to inquire about advertising on Big Picture Science. You can get early access to ad-free versions of every episode by joining us on Patreon. Thanks for your support!   Learn more about your ad choices. Visit megaphone.fm/adchoices

Science can sometimes feel like an exclusive club that only certain people are allowed into. In this week's episode, produced in partnership with the Allen Institute, both of our storytellers try to find their place in science. Part 1: After getting accepted to a PhD program, Max Departee can't help but feel like he's not good enough to be there. Part 2: Han Arbach is worried coming out as non-binary will ruin their scientific career. Max Departee is a research scientist from the Pacific Northwest who has always had a fascination with the natural world. A curious nature and outdoor spirt led him to attend Montana State University where, between fly-fishing on local rivers and skiing the Rockies, he earned his Bachelors Degree in Biotechnology. Max's career and training as a scientist have taken him many places, from a PhD program in North Carolina, to a small Biotech Start-up in Washington, and back to his home town of Seattle where he now works at the Allen Institute for Brain Science. Han Arbach grew up dreaming of becoming an astronaut after watching the space shuttle land at the military base their family was stationed at. As they continued to grow up and became a “frequent flyer” in the orthopedics department for various injuries, their aspirations shifted towards medical training. Encouraged by fantastic AP Biology and Chemistry teachers in high school they pursued a biochemistry major at Mount Holyoke College. Here they were encouraged by a chemistry professor to try out research. This fostered a newfound love for discovery and research, and with it a new dream career path of becoming a scientist. Han completed their Ph.D. in Biochemistry at the University of Washington studying tail regeneration and nuclear structure in tadpoles. They then did Postdoctoral work at the Fred Hutchinson Cancer Center using viruses as a tool to probe facets of cell biology. Now, they are a Program Officer at the Paul G. Allen Frontiers Group. Outside of work, you will find them raising two dogs with their partner, attempting to befriend crows, and being a poor but enthusiastic gardener.  Learn more about your ad choices. Visit megaphone.fm/adchoices

In the fifth Season of the National Institute of Neurological Disorders and Stroke's Building Up the Nerve podcast, we help you strengthen your science communication skills with tools and advice to use throughout your career. We know that navigating your career can be daunting, but we're here to help—it's our job!In the fifth episode of the season, we talk about Securing Funding for Research focusing on choosing what funding to apply for, “pitching” your science to different funders, and writing effective grant applications.Featuring Sonya Dumanis, PhD, Executive Vice President of the Coalition for Aligning Science and Deputy Director for Aligning Science Across Parkinson's; Kat M. Steele, PhD, Associate Director of CREATE and Albert Kobayashi Professor in Mechanical Engineering at University of Washington; and Gene Yeo, PhD, MBA, Professor of Cellular and Molecular Medicine at University of California, San Diego and Chief Scientific Advisor, Sanford Laboratories for Innovative Medicine.ResourcesNIH Funding OpportunitiesSmall Business Innovation Research (SBIR) and Small Business Technology Transfer (STTR) grants: https://seed.nih.gov/small-business-funding/small-business-program-basics/understanding-sbir-sttr NINDS Funding Opportunities: https://www.ninds.nih.gov/funding/find-funding-opportunities NINDS Training & Career Development Opportunities: https://www.ninds.nih.gov/funding/training-career-development NIH ResourcesEarly Career Reviewer program: https://public.csr.nih.gov/ForReviewers/BecomeAReviewer/ECR NIH RePORTER: https://reporter.nih.gov/ Early Stage Investigator (ESI) Policies: https://grants.nih.gov/policy-and-compliance/policy-topics/early-stage-investigators NINDS Guidelines for incorporating rigor into grant applications: https://www.ninds.nih.gov/funding/preparing-your-application/preparing-research-plan/rigorous-study-design-and-transparent-reporting NIH Activity Codes: https://grants.nih.gov/funding/activity-codes Allen Institute's Allen Distinguished Investigators: https://alleninstitute.org/division/frontiers-group/distinguished-investigators/ Advanced Research Projects Agency for Health (ARPA-H): https://arpa-h.gov/ Aligning Science Across Parkinson's (ASAP) Disease – Collaborative Research Network (CRN): https://parkinsonsroadmap.org/research-network/# California Institute for Regenerative Medicine (CIRM) grants: https://www.cirm.ca.gov/ Transcript available at http://ninds.buzzsprout.com/.

In this episode of The Cognitive Revolution, we dive deep into frontier post-training techniques for large language models with Nathan Lambert from the Allen Institute for AI. Nathan discusses the groundbreaking Tulu 3 release, which matches Meta's post-training performance using the LlAMA base model. We explore supervised fine-tuning, preference-based reinforcement learning, and the innovative reinforcement learning from verifiable reward technique. Nathan provides unprecedented insights into the practical aspects of model development, compute requirements, and data generation strategies. This technically rich conversation illuminates previously opaque aspects of LLM development, achieved by a small team of 10-15 people. Join us for one of our most detailed and valuable discussions on state-of-the-art AI model development. Check out Nathan's Lambert newsletter: https://www.natolambert.com https://www.interconnects.ai Be notified early when Turpentine's drops new publication: https://www.turpentine.co/exclusiveaccess SPONSORS: Incogni: Take your personal data back with Incogni! Use code REVOLUTION at the link below and get 60% off an annual plan: https://incogni.com/revolution Notion: Notion offers powerful workflow and automation templates, perfect for streamlining processes and laying the groundwork for AI-driven automation. With Notion AI, you can search across thousands of documents from various platforms, generating highly relevant analysis and content tailored just for you - try it for free at https://notion.com/cognitiverevolution Shopify: Shopify is the world's leading e-commerce platform, offering a market-leading checkout system and exclusive AI apps like Quikly. Nobody does selling better than Shopify. Get a $1 per month trial at https://shopify.com/cognitive Oracle Cloud Infrastructure (OCI): Oracle's next-generation cloud platform delivers blazing-fast AI and ML performance with 50% less for compute and 80% less for outbound networking compared to other cloud providers13. OCI powers industry leaders with secure infrastructure and application development capabilities. New U.S. customers can get their cloud bill cut in half by switching to OCI before December 31, 2024 at https://oracle.com/cognitive 80,000 Hours: 80,000 Hours offers free one-on-one career advising for Cognitive Revolution listeners aiming to tackle global challenges, especially in AI. They connect high-potential individuals with experts, opportunities, and personalized career plans to maximize positive impact. Apply for a free call at https://80000hours.org/cognitiverevolution to accelerate your career and contribute to solving pressing AI-related issues. RECOMMENDED PODCAST: Unpack Pricing - Dive into the dark arts of SaaS pricing with Metronome CEO Scott Woody and tech leaders. Learn how strategic pricing drives explosive revenue growth in today's biggest companies like Snowflake, Cockroach Labs, Dropbox and more. Apple: https://podcasts.apple.com/us/podcast/id1765716600 Spotify: https://open.spotify.com/show/38DK3W1Fq1xxQalhDSueFg CHAPTERS: (00:00:00) Teaser (00:00:59) Sponsors: Incogni (00:02:20) About the Episode (00:05:56) Introducing AI2 (00:09:56) Tulu: Deep Dive (Part 1) (00:17:43) Sponsors: Shopify | Oracle Cloud Infrastructure (OCI) (00:20:38) Open vs. Closed Recipes (00:29:48) Compute & Value (Part 1) (00:34:22) Sponsors: 80,000 Hours | Notion (00:37:02) Compute & Value (Part 2) (00:42:41) Model Weight Evolution (00:53:16) DPO vs. PPO (01:06:36) Project Trajectory (01:20:39) Synthetic Data & LLM Judge (01:27:39) Verifiable RL (01:38:17) Advice for Practitioners (01:44:01) Open Source vs. Closed (01:49:18) Outro

Bridger (Waleed) Ammar has been leading top-tier, high-impact data-modeling projects since 2006 in research, education, engineering and product. Sponsor The Jason Cavness Experience is sponsored by CavnessHR. CavnessHR provides HR to companies with 49 or fewer people. CavnessHR provides a tech platform that automates HR while providing access to a dedicated HR Business Partner. www.CavnessHR.com Go to www.thejasoncavnessexperience.com for the podcast on your favorite platforms  Bridger's Bio Bridger (Waleed) Ammar has been leading top-tier, high-impact data-modeling projects since 2006 in research, education, engineering and product. A few experiences which particularly helped shape his thinking: - Co-founded the ACM chapter at Alexandria University. Defended his PhD in Language-Universal Large Models (L-ULM), in 2016, with Tom Mitchell and Kuzman Ganchev as examiners. - Taught at Alexandria University, Carnegie Mellon University, and University of Washington. Published at Nature, JAMA, NeurIPS, ACL, EMNLP among other top-tier venues. Advised mission-critical organizations on AI strategy, including the NSF (USA), SDAIA (KSA), a leading gaming platform (USA), a leading freight forwarding platform (KSA). At King Saud university, he learned the holistic power of safely integrating different cultures for global good. - At Alexandria University, he contributed to a digital model for historical artifacts, in collaboration with the Alexandria Library. At P&G, he learned the holistic power of mapping the manufacturing process in a data model. - At IBM, he contributed to the state of the art (SOTA) in using statistics to model biological sequences, in collaboration with DARPA. - At eSpace, he learned the basics of building sustainable businesses, in collaboration with Alexandria University. - At Microsoft, he contributed to the then-SOTA in statistical machine translation models, in collaboration with the Cairo Microsoft Innovation Center. At Carnegie Mellon University, as a Google PhD fellow, he developed the SOTA in language-universal models (L-UMs). At Google Shopping, he contributed to the SOTA in mixing random forests with neural networks. - At the Allen Institute for Artificial Intelligence, he learned the SOTA in managing science from his mentor Oren Etzioni, then developed the SOTA in modeling science. At Google Health, he contributed to the SOTA in building the digital manifestation of living cells in species-agnostic models. - At Google Research, he learned the SOTA in cost-effective scaling of LLM inference to a Billion users. - At Google Assistant, he learned the SOTA in scalable distribution of data products. At Burning Man, he learned how to safely integrate freedom and self expression. We talked about the following and other items Burning Man Experience and Philosophy Scientific Progress and Its Impact  Ethics in Science and Peer Review Purpose of Science and Future Discoveries Encouraging Young Scientists and Scientific Discoveries Future of AI and Its Impact on Various Industries Global AI Development and Personal Background  Is Singularity coming Paddle boarding and dancing AI/ML How were the pyramids built Are humans becoming smarter AI ethics Bridger's Social Media  Bridger's LinkedIn: https://www.linkedin.com/in/waleedammar/  Bridger's Email: wammar@higg.world Company Website: https://higg.world/ Company Instagram: https://www.instagram.com/holistic_intelligence/  

古館さん(@ShoheiFurutachi) in-person収録＠Allen Institute。直近の論文リバイズ中の出来事、最近の興味、ネタ被りに関する様々な考え方、ジョブハントの様子、東大での某失敗談の詳細 (10/13 収録) Show Notes (番組HP): 古館さん 前回のNeuroRadio古館さん回 Part 1 Part 2 古館さんのNature論文 Illana WittenのCognitive Demand論文 茨城県の県立医療大学 パーキンソン病リハビリにおけるキューの利用 (pdf) Mark Andermann Hypothalamusに変なドーパミンニューロンがいて、性衝動のコントロールが知られている Parkinson病患者にL-DOPAを投与すると性衝動が上がる L-DOPAで幻覚 千歳さん のNR回 河西・柳下ラボ Over-Associationを示唆する論文 Georg Keller 自閉症だと自分をこそばせられる 統合失調でした(萩) 自閉症と統合失調症はスペクトラムの逆側？ Positive Prediction ErrorとNegative Prediction Errorについてのレビュー Oliver Sacks 妻を帽子と間違えた男 五十嵐さん Wellcome Career Development Award Dmitriy Aronov Vijay Namboodiri Moserラボ DMDMの論文 のレビュワーコメント(pdf) Karel Svoboda Steinmetzの仕事 尾藤先生 ASCONA Lake Conference Visual CortexのConnectivity 例えば：1 2 FENS Massimo Scanziani Cyril Herry Letzkus 内田さん 天羽さん SfN 2024 PrincetonのTigerBrain SWCもやってるEmerging Neuroscientists Seminar Series Carlos Brody David Tank Jonathan Pillow Nathaniel Daw Annegret Falkner Tatiana Engel グローバルCOE(pdf) 石川冬木らによるテロメラーゼ遺伝子TLP1のクローニング Krakauer Sherringtonian vs Hopfieldianのレビュー UCL Ophthalmology Carandini Andy Peters Ace Hotel Gilles Laurent MPI Collective Behavior Iain Couzin 100匹くらいの魚とか Median Rapheの仕事 Matt Lovett-Baron Deisseroth Losonczy 魚を始めた 伊藤さん Chris Harvey Noah Pettit Selmaan Chettih Christopher Zimmerman Science Eppendolf Prize Zachary Knight CTAの仕事 について話した回 番組で扱ってたのを忘れていた(萩) 宮道先生 のTRAP(初代) 五十嵐さんのタスク Nelson Spruston のタスク paAIP2 稲垣さん 小宮山ラボ 服部さん（現Scripps） のNN2023 大木研 村上さん 医学部の機能生物学セミナー ドイツの戸田さん 伊丹十三 のエッセイ：この２つのどちらか １ ２ 黒田さん と中野さんのNR回 大木研のLP 林-高木先生 ダルメシアンの図 Editorial Notes: ドッキリ収録含めAllen Institute visit最高に楽しかったです。(古) dmdmのReviewerコメント読んだ。”monumental effort”とかのいいまわしが確かにカレルっぽい(萩) 超まじめな話と笑い話が同じトーンで行われていて面白かったです (脇)

The fifth Season of the National Institute of Neurological Disorders and Stroke's Building Up the Nerve podcast, where we help you strengthen your science communication skills with tools and advice to use throughout your career. We know that navigating your career can be daunting, but we're here to help—it's our job!In the second episode of the season, we talk about Thriving in Team Science, focusing on how to build professional collaborations and guidelines to ensure success for all parties when working in team, especially across disciplines.Featuring Bosiljka Tasic, PhD, Director, Molecular Genetics, Allen Institute for Brain Science; Heidi Baumgartner, PhD, Research Scholar, Stanford University and Executive Director, ManyBabies; and Lingfei Wu, PhD, Assistant Professor, University of PittsburghResourcesArticles from Dr. Lingfei Wu:“Remote collaboration fuses fewer breakthrough ideas” (2023): https://pubmed.ncbi.nlm.nih.gov/38030778/ “Large teams develop and small teams disrupt science and technology” (2019): https://pubmed.ncbi.nlm.nih.gov/30760923/ Resources from ManyBabies: https://manybabies.org/resources/ ManyBabies Collaboration AgreementContributor Role Taxonomy (CRedIT): https://credit.niso.org/ Tenzing.club Protocols.ioTranscript available at http://ninds.buzzsprout.com/.

Dr. Oren Etzioni is the founder of TrueMedia.org, a free-use product that allows critical election audiences from around the world to quickly and effectively detect deepfakes. He was the Founding Chief Executive Officer at the Allen Institute for AI (AI2), having served as CEO from its inception in 2013 until late 2022. He is Professor Emeritus at the University of Washington where he helped to pioneer meta-search, online comparison shopping, machine reading, and open information extraction. He has also authored several award-winning technical papers, and founded and sold many companies. In addition to his role at TrueMedia.org, he is currently a technical director of the AI2 Incubator and a Venture Partner at Madrona.See omnystudio.com/listener for privacy information.

Consciousness and Science with Christof Koch Christof Koch was a professor at the California Institute of Technology and the president and chief scientist of the Allen Institute for Brain Science. He is now a meritorious investigator at the Allen Institute and the chief scientist for the Tiny Blue Dot Foundation. He is author of Biophysics … Continue reading "Consciousness and Science with Christof Koch"

Scare stories about Artificial Intelligence are everywhere – but its colossal environmental impact is startlingly underreported. How exactly does the use of A.I. contribute to the climate crisis, is there anything being done to counteract it, and why is this issue largely unknown? To find out, Kate Devlin talks to  Jesse Dodge, senior research scientist at the Allen Institute for AI.  We are sponsored by Indeed. Go to Indeed.com/bunker for £100 sponsored credit.    www.patreon.com/bunkercast    Written and presented by Kate Devlin. Produced by Eliza Davis Beard. Audio production by Tom Taylor. Managing Editor Jacob Jarvis. Group Editor Andrew Harrison. Art by James Parrett. Music by Kenny Dickinson. THE BUNKER is a Podmasters Production  www.podmasters.co.uk  Learn more about your ad choices. Visit podcastchoices.com/adchoices

Top Stories:1. Seattle's minimum wage⁠Seattle Times article⁠2. City of Seattle's budget issues⁠Seattle Times article⁠3. Effect of caregiving in the workplace⁠Forbes article⁠4. Brain studies at Allen Institute⁠Yahoo article⁠About guest Marjorie Thomas - CFO, Allen InstituteMarjorie joined the Allen Institute in 2017. As Chief Financial Officer she leads all accounting financial functions including grant administration. Previously Marjorie was CFO and Treasurer of RealNetworks. Before that, she held senior financial positions at Intuit, Sony Electronics, and Hewlett Packard. She served on the board of directors for Junior Achievement of Northern CA and currently serves on the board of MoPop.Host Rachel Horgan:Rachel is an independent event producer, emcee and entrepreneur. She worked for the Business Journal for 5 years as their Director of Events interviewing business leaders on stage before launching the weekly podcast. She earned her communication degree from the University of San Diego.Contact:Email: info@theweeklyseattle.comInstagram: @theweeklyseattleWebsite: ⁠⁠⁠⁠⁠⁠⁠⁠⁠www.theweeklyseattle.com

Episode 133I spoke with Peter Lee about:* His early work on compiler generation, metacircularity, and type theory* Paradoxical problems* GPT-4s impact, Microsoft's “Sparks of AGI” paper, and responses and criticismEnjoy—and let me know what you think!Peter is President of Microsoft Research. He leads Microsoft Research and incubates new research-powered products and lines of business in areas such as artificial intelligence, computing foundations, health, and life sciences. Before joining Microsoft in 2010, he was at DARPA, where he established a new technology office that created operational capabilities in machine learning, data science, and computational social science. Prior to that, he was a professor and the head of the computer science department at Carnegie Mellon University. Peter is a member of the National Academy of Medicine and serves on the boards of the Allen Institute for Artificial Intelligence, the Brotman Baty Institute for Precision Medicine, and the Kaiser Permanente Bernard J. Tyson School of Medicine. He served on President Obama's Commission on Enhancing National Cybersecurity. He has testified before both the US House Science and Technology Committee and the US Senate Commerce Committee. With Carey Goldberg and Dr. Isaac Kohane, he is the coauthor of the best-selling book, “The AI Revolution in Medicine: GPT-4 and Beyond.” In 2024, Peter Lee was named by Time magazine as one of the 100 most influential people in health and life sciences.Find me on Twitter for updates on new episodes, and reach me at editor@thegradient.pub for feedback, ideas, guest suggestions. I spend a lot of time on this podcast—if you like my work, you can support me on Patreon :) You can also support upkeep for the full Gradient team/project through a paid subscription on Substack!Subscribe to The Gradient Podcast:  Apple Podcasts  | Spotify | Pocket Casts | RSSFollow The Gradient on TwitterOutline:* (00:00) Intro* (00:50) Basic vs. applied research* (05:20) Theory and practice in computing* (10:28) Traditional denotational semantics and semantics engineering in modern-day systems* (16:47) Beauty and practicality* (20:40) Metacircularity in the polymorphic lambda calculus: research directions* (24:31) Understanding the nature of difficulties with metacircularity* (26:30) Difficulties with reflection, classic paradoxes* (31:02) Sparks of AGI* (31:41) Reproducibility* (38:04) Confirming and disconfirming theories, foundational work* (42:00) Back and forth between commitments and experimentation* (51:01) Dealing with responsibility* (56:30) Peter's picture of AGI* (1:01:38) OutroLinks:* Peter's Twitter, LinkedIn, and Microsoft Research pages* Papers and references* The automatic generation of realistic compilers from high-level semantic descriptions* Metacircularity in the polymorphic lambda calculus* A Fresh Look at Combinator Graph Reduction* Sparks of AGI* Re-envisioning DARPA* Fundamental Research in Engineering Get full access to The Gradient at thegradientpub.substack.com/subscribe

Our guest in this episode is Jon Gelsey. Jon was the first CEO of Auth0, a leading identity-as-a-service platform, which grew from 5 to 300 employees during his four years at the helm. Auth0 was acquired by Okta in February 2021 for $6.5B. After Auth0, Jon served as CEO of Xnor, a computer vision and machine learning spinoff of the Allen Institute. The company was acquired by Apple for ~$200M in January 2020. When Auth0 first started in 2013, there were already several authentication vendors in the market. Okta, ForgeRock, and OneLogin had all built considerable scale by the time Auth0 launched its product. Not only did Jon and the team build a successful company in a very crowded space, but they also did it their way. While all of Auth0's competitors were running a top-down GTM motion, Jon made a critical decision to adopt a bottom-up, product-led growth (PLG) strategy. Instead of relying on traditional marketing tactics for demand generation, Auth0 built an extensive content rollout plan to help drive inbound interest in the product. To date, Auth0 is the only PLG company in cybersecurity to achieve a multi-billion dollar exit. On Inside the Network, Jon talks about building go-to-market strategies, identifying the right buyer personas, and establishing success metrics for customer acquisition.  In addition to his experience as a serial entrepreneur, Jon worked on the M&A and strategy team at Microsoft from 2007 to 2014 where he led several acquisitions for the company. Jon shares the tips and tricks founders need to know to plan, negotiate, and successfully close acquisitions with potential buyers. 

If you see this in time, join our emergency LLM paper club on the Llama 3 paper!For everyone else, join our special AI in Action club on the Latent Space Discord for a special feature with the Cursor cofounders on Composer, their newest coding agent!Today, Meta is officially releasing the largest and most capable open model to date, Llama3-405B, a dense transformer trained on 15T tokens that beats GPT-4 on all major benchmarks:The 8B and 70B models from the April Llama 3 release have also received serious spec bumps, warranting the new label of Llama 3.1.If you are curious about the infra / hardware side, go check out our episode with Soumith Chintala, one of the AI infra leads at Meta. Today we have Thomas Scialom, who led Llama2 and now Llama3 post-training, so we spent most of our time on pre-training (synthetic data, data pipelines, scaling laws, etc) and post-training (RLHF vs instruction tuning, evals, tool calling).Synthetic data is all you needLlama3 was trained on 15T tokens, 7x more than Llama2 and with 4 times as much code and 30 different languages represented. But as Thomas beautifully put it:“My intuition is that the web is full of s**t in terms of text, and training on those tokens is a waste of compute.” “Llama 3 post-training doesn't have any human written answers there basically… It's just leveraging pure synthetic data from Llama 2.”While it is well speculated that the 8B and 70B were "offline distillations" of the 405B, there are a good deal more synthetic data elements to Llama 3.1 than the expected. The paper explicitly calls out:* SFT for Code: 3 approaches for synthetic data for the 405B bootstrapping itself with code execution feedback, programming language translation, and docs backtranslation.* SFT for Math: The Llama 3 paper credits the Let's Verify Step By Step authors, who we interviewed at ICLR:* SFT for Multilinguality: "To collect higher quality human annotations in non-English languages, we train a multilingual expert by branching off the pre-training run and continuing to pre-train on a data mix that consists of 90% multilingualtokens."* SFT for Long Context: "It is largely impractical to get humans to annotate such examples due to the tedious and time-consuming nature of reading lengthy contexts, so we predominantly rely on synthetic data to fill this gap. We use earlier versions of Llama 3 to generate synthetic data based on the key long-context use-cases: (possibly multi-turn) question-answering, summarization for long documents, and reasoning over code repositories, and describe them in greater detail below"* SFT for Tool Use: trained for Brave Search, Wolfram Alpha, and a Python Interpreter (a special new ipython role) for single, nested, parallel, and multiturn function calling.* RLHF: DPO preference data was used extensively on Llama 2 generations. This is something we partially covered in RLHF 201: humans are often better at judging between two options (i.e. which of two poems they prefer) than creating one (writing one from scratch). Similarly, models might not be great at creating text but they can be good at classifying their quality.Last but not least, Llama 3.1 received a license update explicitly allowing its use for synthetic data generation.Llama2 was also used as a classifier for all pre-training data that went into the model. It both labelled it by quality so that bad tokens were removed, but also used type (i.e. science, law, politics) to achieve a balanced data mix. Tokenizer size mattersThe tokens vocab of a model is the collection of all tokens that the model uses. Llama2 had a 34,000 tokens vocab, GPT-4 has 100,000, and 4o went up to 200,000. Llama3 went up 4x to 128,000 tokens. You can find the GPT-4 vocab list on Github.This is something that people gloss over, but there are many reason why a large vocab matters:* More tokens allow it to represent more concepts, and then be better at understanding the nuances.* The larger the tokenizer, the less tokens you need for the same amount of text, extending the perceived context size. In Llama3's case, that's ~30% more text due to the tokenizer upgrade. * With the same amount of compute you can train more knowledge into the model as you need fewer steps.The smaller the model, the larger the impact that the tokenizer size will have on it. You can listen at 55:24 for a deeper explanation.Dense models = 1 Expert MoEsMany people on X asked “why not MoE?”, and Thomas' answer was pretty clever: dense models are just MoEs with 1 expert :)[00:28:06]: I heard that question a lot, different aspects there. Why not MoE in the future? The other thing is, I think a dense model is just one specific variation of the model for an hyperparameter for an MOE with basically one expert. So it's just an hyperparameter we haven't optimized a lot yet, but we have some stuff ongoing and that's an hyperparameter we'll explore in the future.Basically… wait and see!Llama4Meta already started training Llama4 in June, and it sounds like one of the big focuses will be around agents. Thomas was one of the authors behind GAIA (listen to our interview with Thomas in our ICLR recap) and has been working on agent tooling for a while with things like Toolformer. Current models have “a gap of intelligence” when it comes to agentic workflows, as they are unable to plan without the user relying on prompting techniques and loops like ReAct, Chain of Thought, or frameworks like Autogen and Crew. That may be fixed soon?

Share this episode: https://www.samharris.org/podcasts/making-sense-episodes/374-consciousness-and-the-physical-world Sam Harris speaks with Christof Koch about the nature of consciousness. They discuss Christof’s development as a neuroscientist, his collaboration with Francis Crick, change blindness and binocular rivalry, sleep and anesthesia, the limits of physicalism, non-locality, brains as classical systems, conscious AI, idealism and panpsychism, Integrated Information Theory (IIT), what it means to say something “exists,” the illusion of the self, brain bridging, Christof’s experience with psychedelics, and other topics. Christof Koch is a neuroscientist at the Allen Institute and the Chief Scientist of the Tiny Blue Dot Foundation. He is the former president of the Allen Institute for Brain Science and a former professor at the California Institute of Technology. He writes regularly for Scientific American and is the author of five books, most recently Then I Am Myself the World: What Consciousness Is and How to Expand It. Website: https://christofkoch.com/ Learning how to train your mind is the single greatest investment you can make in life. That’s why Sam Harris created the Waking Up app. From rational mindfulness practice to lessons on some of life’s most important topics, join Sam as he demystifies the practice of meditation and explores the theory behind it.

In this episode of the Wise Decision Maker Show, Dr. Gleb Tsipursky speaks to Petra Smith, Executive Director of People and Culture at The Allen Institute, about the benefits of hyper-flexibility.You can learn about The Allen Institute at https://alleninstitute.org/

Artificial intelligence consumes a lot of energy. Exactly how much is hard to say, because AI companies keep much of that information hidden – a practice that some policymakers and activists are trying to change. On today's episode of Politico Tech, Steven Overly called up Jesse Dodge to better understand the energy and climate cost behind generative AI. Dodge is a senior research scientist at the Allen Institute for AI in Seattle, who not only develops large language models, he also studies their electricity usage and CO2 emissions.

Christof Koch is a pioneering computational neuroscientist and neurophysiologist best known for his groundbreaking work on the neural basis of consciousness. He collaborated with Francis Crick, the co-discoverer of the structure of DNA, to establish a neurobiological framework for understanding consciousness. Christof served as the President and Chief Scientist of the Allen Institute for Brain Science in Seattle and continues his work there as a Meritorious Investigator. He is also the Chief Scientist of the Tiny Blue Dot Foundation in Santa Monica, CA, which funds research aimed at alleviating suffering, anxiety, and other forms of distress in people worldwide. Christof has authored over five books on consciousness, with his latest being "Then I Am Myself the World: What Consciousness Is and How to Expand It." This book delves into the subject of consciousness through the lenses of physics, psychology, neuroscience, philosophy, and computer science, as well as Christof's personal experiences exploring his consciousness. In this episode, Christof dives deep into what might explain the origin of consciousness and existing contradictions. We explore how our minds construct reality, the wonder of experience, and the profound implications of Integrated Information Theory. Christof also reflects on the importance of mindfulness, the power of belief, and the ongoing debate on free will. Our conversation includes: The origin of consciousness and the “Hard Problem Integrated Information Theory (IIT) How far down the phylogenetic tree consciousness might go The mind-body problem: physical structures and subjective experiences Panpsychism and how consciousness might be a fundamental aspect of matter. Qualities of experience and the perception box Mind-melding and the “uber” consciousness Why AI or any compute-based system may never be sentient The boundaries of consciousness and the dissolution of self The notion of "mind at large" Christoph's experiences with psychedelics Free will My hope is that this episode gives you a sense of awe about your mind so that you look at life and your experiences with a bit more wonder. Enjoy! For show notes and more, visit www.larryweeks.com 

Nathan Lambert of the Interconnects substack and Allen Institute joins for a roundup where we get into: What DC should understand about the Bay Area AI engineer psyche What GPT4o and Google's AI Dev Day mean for the future of AI OpenAI's model spec, and exit, voice, and loyalty in the leading labs Outtro music: Scarlett Johansson's The Moon Song Learn more about your ad choices. Visit megaphone.fm/adchoices

This week on What's At Stake, tune into another episode of our Artificially Intelligent Conversations series. Each month, Penta Partners Andrea Christianson and Chris Mehigan will dissect the AI latest news and developments from each side of the Atlantic, helping you understand how AI will affect stakeholder management, policy, and more. This week, they host Nicole DeCario, Director of AI & Society at the Allen Institute for AI (AI2) to cover AI literacy, policy, and industry norms.Nicole discusses how AI2's commitment to open-source language models aims to shape a more transparent and collaborative future in AI research, distinct from the closed doors of industry norms. Her insights reveal a pressing need for ethical frameworks and a nuanced understanding of AI technology in order for U.S. regulators to craft effective policy. You don't want to miss this episode!

In Then I Am Myself the World, Christof Koch explores the only thing we directly experience: consciousness. At the book's heart is integrated-information theory, the idea that the essence of consciousness is the ability to exert causal power over itself, to be an agent of change. Koch investigates the physical origins of consciousness in the brain and how this knowledge can be used to measure consciousness in natural and artificial systems. Enabled by such tools, Koch reveals when and where consciousness exists, and uses that knowledge to confront major social and scientific questions: When does a fetus first become self-aware? Can psychedelic and mystical experiences transform lives? What happens to consciousness in near-death experiences? Why will generative AI ultimately be able to do the very thing we can do, yet never feel any of it? And do our experiences reveal a single, objective reality?    Christof Koch is a neuroscientist at the Allen Institute and at the Tiny Blue Dot Foundation, the former president of the Allen Institute for Brain Science, and a former professor at the California Institute of Technology. Author of four previous titles — The Feeling of Life Itself: Why Consciousness Is Widespread but Can't Be Computed, Consciousness: Confessions of a Romantic Reductionist, and The Quest for Consciousness: A Neurobiological Approach — Koch writes regularly for a range of media, including Scientific American. He lives in the Pacific Northwest. Shermer and Koch discuss: “subjective experience” • the author's near-death experience changed him • the difficulties of materialism/physicalism • a fundamental theory of consciousness that explains subjective experiences in objective measures • designing a “consciousness detector” for unresponsive patients • why magic mushrooms and Ayahuasca are of so fascinating to neuroscientists • how our minds are shaped by our beliefs, prior experiences, and intentions • insights crucial to those suffering from anxiety, low self-esteem, post-traumatic stress, and depression. • the future of advanced brain-machine interfaces • why digital computers will never be conscious.

Can machine learning help predict extreme weather events and climate change? Christopher Bretherton, senior director of climate modeling at the Allen Institute for Artificial Intelligence, or AI2, explores the technology's potential to enhance climate modeling with AI Podcast host Noah Kravitz in an episode recorded live at the NVIDIA GTC global AI conference. Bretherton explains how machine learning helps overcome the limitations of traditional climate models and underscores the role of localized predictions in empowering communities to prepare for climate-related risks. Through ongoing research and collaboration, Bretherton and his team aim to improve climate modeling and enable society to better mitigate and adapt to the impacts of climate change.

The explosion of AI-powered chatbots and image generators, like ChatGPT and DALL-E, over the past two years is changing the way we interact with technology. Their impressive abilities to generate lifelike images from written instructions or write an essay on the topic of your choosing can seem a bit like magic.But that “magic” comes at a steep environmental cost, researchers are learning. The data centers used to power these models consume an enormous amount of not just electricity, but also fresh water to keep everything running smoothly. And the industry shows no signs of slowing down. It was reported earlier this month that Sam Altman, the CEO of leading AI company OpenAI, is seeking to raise about $7 trillion to reshape the global semiconductor industry for AI chip production.Ira Flatow is joined by Dr. Jesse Dodge, research scientist at the Allen Institute for AI, to talk about why these models use so much energy, why the placement of these data centers matter, and what regulations these companies could face.Transcripts for this segment will be available the week after the show airs on sciencefriday.com Subscribe to this podcast. Plus, to stay updated on all things science, sign up for Science Friday's newsletters.

In October 2023, an international group of scientists released an impressively detailed cell atlas of the human brain, published in 21 papers in the journals Science, Science Advances and Science Translational Medicine.The human brain has roughly 171 billion cells, which makes it a herculean task to categorize them all. Scientists collected samples from different parts of the brain and have identified 3,000 different types of cells. Each cell contains thousands of genes and each cell type only expresses a small fraction of those. Cataloging cells by their gene expressions, paves the way for scientists to tailor disease treatments to target only the affected cells. This human brain cell atlas is only the first draft, but it could signal a paradigm shift in how we understand and treat neurological diseases.Ira talks with one of the researchers who helped put together the cell atlas, Dr. Ed Lein, senior investigator at the Allen Institute for Brain Science, and takes listener calls.Transcripts for each segment will be available the week after the show airs on sciencefriday.com. To stay updated on all things science, sign up for Science Friday's newsletters.