Podcasts about code llama

Large language models are helping developers move faster than ever. But behind the convenience of AI-generated code lies a security vulnerability: package hallucinations. In this episode, Ashok sits down with U.S. Army cybersecurity officer and PhD researcher Joe Spracklen to unpack new research on how hallucinated package names—fake libraries that don't yet exist—can be weaponized by attackers and quietly introduced into your software supply chain. Joe's recent academic study reveals how large language models like ChatGPT and Code Llama are frequently recommending software packages that don't actually exist—yet. These fake suggestions create the perfect opportunity for attackers to register malicious packages with those names, compromising developer machines and potentially entire corporate networks. Whether your team is deep into AI pair programming or just starting to experiment, this conversation surfaces key questions every tech leader should be asking before pushing AI-generated code to production. Unlock the full potential of your product team with Integral's player coaches, experts in lean, human-centered design. Visit integral.io/convergence for a free Product Success Lab workshop to gain clarity and confidence in tackling any product design or engineering challenge. Inside the episode... What "package hallucinations" are and why they matter How AI code assistants can introduce real vulnerabilities into your network Which models were most likely to hallucinate packages Why hallucinated package names are often persistent—not random How attackers could weaponize hallucinated names to spread malware What mitigation strategies were tested—and which ones failed Why simple retrieval-based techniques (like RAG) don't solve the problem Steps security-conscious teams can take today to protect their environments The importance of developer awareness as more non-traditional engineers enter the field Mentioned in this episode Python Package Index (PyPI) npm JavaScript package registry Snyk, Socket.dev, Phylum (dependency monitoring tools) Artifactory, Nexus, Verdaccio (private package registries) ChatGPT, Code Llama, DeepSeek (AI models tested) Subscribe to the Convergence podcast wherever you get podcasts including video episodes on YouTube at youtube.com/@convergencefmpodcast Learn something? Give us a 5 star review and like the podcast on YouTube. It's how we grow. Unlock the full potential of your product team with Integral's player coaches, experts in lean, human-centered design. Visit integral.io/convergence for a free Product Success Lab workshop to gain clarity and confidence in tackling any product design or engineering challenge. Subscribe to the Convergence podcast wherever you get podcasts including video episodes to get updated on the other crucial conversations that we'll post on YouTube at youtube.com/@convergencefmpodcast Learn something? Give us a 5 star review and like the podcast on YouTube. It's how we grow.   Follow the Pod Linkedin: https://www.linkedin.com/company/convergence-podcast/ X: https://twitter.com/podconvergence Instagram: @podconvergence

Podcast Notes: Debunking Claims About AI's Future in CodingEpisode OverviewAnalysis of Anthropic CEO Dario Amodei's claim: "We're 3-6 months from AI writing 90% of code, and 12 months from AI writing essentially all code"Systematic examination of fundamental misconceptions in this predictionTechnical analysis of GenAI capabilities, limitations, and economic forces1. Terminological MisdirectionCategory Error: Using "AI writes code" fundamentally conflates autonomous creation with tool-assisted compositionTool-User Relationship: GenAI functions as sophisticated autocomplete within human-directed creative processEquivalent to claiming "Microsoft Word writes novels" or "k-means clustering automates financial advising"Orchestration Reality: Humans remain central to orchestrating solution architecture, determining requirements, evaluating output, and integrationCognitive Architecture: LLMs are prediction engines lacking intentionality, planning capabilities, or causal understanding required for true "writing"2. AI Coding = Pattern Matching in Vector SpaceFundamental Limitation: LLMs perform sophisticated pattern matching, not semantic reasoningVerification Gap: Cannot independently verify correctness of generated code; approximates solutions based on statistical patternsHallucination Issues: Tools like GitHub Copilot regularly fabricate non-existent APIs, libraries, and function signaturesConsistency Boundaries: Performance degrades with codebase size and complexity; particularly with cross-module dependenciesNovel Problem Failure: Performance collapses when confronting problems without precedent in training data3. The Last Mile ProblemIntegration Challenges: Significant manual intervention required for AI-generated code in production environmentsSecurity Vulnerabilities: Generated code often introduces more security issues than human-written codeRequirements Translation: AI cannot transform ambiguous business requirements into precise specificationsTesting Inadequacy: Lacks context/experience to create comprehensive testing for edge casesInfrastructure Context: No understanding of deployment environments, CI/CD pipelines, or infrastructure constraints4. Economics and Competition RealitiesOpen Source Trajectory: Critical infrastructure historically becomes commoditized (Linux, Python, PostgreSQL, Git)Zero Marginal Cost: Economics of AI-generated code approaching zero, eliminating sustainable competitive advantageNegative Unit Economics: Commercial LLM providers operate at loss per query for complex coding tasksInference costs for high-token generations exceed subscription pricingHuman Value Shift: Value concentrating in requirements gathering, system architecture, and domain expertiseRising Open Competition: Open models (Llama, Mistral, Code Llama) rapidly approaching closed-source performance at fraction of cost5. False Analogy: Tools vs. ReplacementsTool Evolution Pattern: GenAI follows historical pattern of productivity enhancements (IDEs, version control, CI/CD)Productivity Amplification: Enhances developer capabilities rather than replacing themCognitive Offloading: Handles routine implementation tasks, enabling focus on higher-level concernsDecision Boundaries: Majority of critical software engineering decisions remain outside GenAI capabilitiesHistorical Precedent: Despite 50+ years of automation predictions, development tools consistently augment rather than replace developersKey TakeawayGenAI coding tools represent significant productivity enhancement but fundamental mischaracterization to frame as "AI writing code"More likely: GenAI companies face commoditization pressure from open-source alternatives than developers face replacement

Aprende Python desde Cero – Fundamentos para Crear Software y Automatizaciones Introducción al Episodio: En este último episodio del año de Ingeniosos de Sistemas, exploramos los fundamentos de Python, el lenguaje de programación más versátil y amigable para principiantes. Presentado por Charles Alonso de Ten Nolita, quien también invita a unirse a la Academia de Inteligencia Artificial de Ten Nolita, una plataforma de aprendizaje continuo disponible por solo 10 euros al mes. Resumen del Episodio: Por qué Python es Ideal para Principiantes y Profesionales: Diseñado para ser claro, directo y fácil de aprender. Popular en áreas como inteligencia artificial, automatización, análisis de datos y desarrollo web. Compatible con múltiples plataformas (Windows, macOS, Linux). Utilizado por empresas como Instagram, Spotify y YouTube. Fundamentos del Lenguaje: Configuración del entorno: Descarga Python desde python.org. Usa gestores de entornos como MiniConda para una instalación optimizada. Instala un IDE como Visual Studio Code para una experiencia de desarrollo sencilla. Primeros Pasos: Escribir y ejecutar scripts básicos como "Hola, Mundo". Aprender a manejar variables, tipos de datos, condicionales, bucles y funciones. Ejercicios Prácticos: Crear un programa que calcule el área de un rectángulo. Escribir un script que solicite al usuario su nombre y edad, determine si es mayor de edad y lo salude. Ventajas de Aprender Python: Gran comunidad y abundancia de recursos de aprendizaje. Acceso a bibliotecas para resolver tareas específicas, desde generación de gráficos hasta automatización avanzada. Ideal para principiantes que buscan entender conceptos fundamentales de programación. El Poder de los Modelos de Lenguaje (LLMs): Cómo usar herramientas como ChatGPT, Claude o modelos de código abierto como Code Llama para generar y mejorar scripts en Python. Ejemplos prácticos de automatización y generación de código asistida. Python para Resolver Problemas Reales: Aplicaciones como la creación de códigos QR, gráficos de datos y pequeños scripts de automatización. Aprender a pensar como programador es más importante que memorizar sintaxis, ya que los LLM pueden ayudar con esta última tarea. Llamadas a la Acción: Únete a la Academia de Ten Nolita: Aprende a convertirte en un "software composer" y domina la programación para aplicaciones prácticas. Sigue el Canal de YouTube: Tutoriales y contenido visual para complementar el aprendizaje. Forma Parte de la Comunidad de Telegram: Discusiones específicas y recursos exclusivos para estudiantes. Optimización SEO del Programa: Palabras clave: Aprender Python, programación para principiantes, automatización con Python, software composer, inteligencia artificial, cursos de Python, lenguaje de programación fácil, Python paso a paso. Llamadas a la acción estratégicas para aumentar suscripciones y participación en plataformas asociadas (YouTube, Telegram, Academia). Final del Episodio: ¡Feliz Año Nuevo! Continúa aprendiendo y practicando Python para desbloquear un mundo de posibilidades en tecnología y automatización. Apúntate a la academia Canal de telegram  y  Canal de Youtube Pregunta por Whatsapp +34 620 240 234 Déjame un mensaje de voz

Accurate body composition data is often seen as the holy grail of information for the longevity-minded biohacker.  Calipers are an accurate way to measure body fat. Smart scales and medical facilities equipped with DEXA machines also do a good job.  But now it is possible to generate body composition data using the camera on your phone.  In this episode, Jason Moore, CEO and founder of Spren, discusses the intersection of technology and health and the significance of understanding body metrics like muscle mass and fat distribution. We also explore the role of AI in health monitoring and the future of personalized health data. Transcript and additional show notes Flexbeam Red Light therapyA wearable red light therapy device which targets key parts of the body to improve sleep, treat injuries and sooth aches and pains associated with aging. Code LLAMA. Discounts vary - see details of the current offer here PartiQlar supplementsEnhance your wellness journey with PartiQlar supplements. No magic formulas, just pure single ingredients, like NMN, L-Glutathione, Spermidine, Resveratrol, TMG and Quercetin. Get a 15% discount with the code MASTERAGING15 at PartiQlarEnergyBits algae snacksA microscopic form of life that could help us age better. Use code LLAMA for a 20 percent discountDisclaimer: This post contains affiliate links. If you make a purchase, I may receive a commission at no extra cost to you.Support the showThe Live Long and Master Aging (LLAMA) podcast, a HealthSpan Media LLC production, shares ideas but does not offer medical advice. If you have health concerns of any kind, or you are considering adopting a new diet or exercise regime, you should consult your doctor.

Loneliness, which affects how long people live, is at epidemic levels. Social isolation is not confined to the elderly, although the anxiety caused by feeling alone is borne especially hard by older people. Could artificial intelligence help? Dor Skuler, co-founder of Intuition Robotics, joined us to explain that AI-driven technology could be part of the solution. The company has developed ElliQ, an AI companion that keeps people company at home through conversations and activities that connect with the outside world.Additional details and show notesDISCOUNTSFlexbeam Red Light therapyA wearable red light therapy device which targets key parts of the body to improve sleep, treat injuries and sooth aches and pains associated with aging. Code LLAMA. Discounts vary - see details of the current offer hereFit, Healthy & Happy Podcast Welcome to the Fit, Healthy and Happy Podcast hosted by Josh and Kyle from Colossus...Listen on: Apple Podcasts SpotifyEnergyBits algae snacksA microscopic form of life that could help us age better. Use code LLAMA for a 20 percent discountSiPhox Health home blood testingMeasure 17 critical blood biomarkers from home. Get a 20% discount with code LLAMA Disclaimer: This post contains affiliate links. If you make a purchase, I may receive a commission at no extra cost to you.Support the showThe Live Long and Master Aging (LLAMA) podcast, a HealthSpan Media LLC production, shares ideas but does not offer medical advice. If you have health concerns of any kind, or you are considering adopting a new diet or exercise regime, you should consult your doctor.

The traditional idea of retirement is changing with people now looking for new ways to continue contributing and finding purpose in their lives. Campbell Gerrish and Caroline Brecker, co-founders of Third Half Advisors, have developed the concept of the 'third half' and how to navigate the bonus years of life. They stress the need to envision a different kind of future and ways to explore fresh opportunities post-retirement.In this interview Campbell and Caroline discuss the importance of building a portfolio of activities, focusing on one's strengths and values, and finding communities of interest. They also highlight the need for self-reflection and the importance of maintaining social connections.Flexbeam Red Light therapyA wearable red light therapy device which targets key parts of the body to improve sleep, treat injuries and sooth aches and pains associated with aging. Code LLAMA. Discounts vary - see details of the current offer hereEnergyBits algae snacksA microscopic form of life that could help us age better. Use code LLAMA for a 20 percent discountSiPhox Health home blood testingMeasure 17 critical blood biomarkers from home. Get a 20% discount with code LLAMA Disclaimer: This post contains affiliate links. If you make a purchase, I may receive a commission at no extra cost to you.Support the showThe Live Long and Master Aging (LLAMA) podcast, a HealthSpan Media LLC production, shares ideas but does not offer medical advice. If you have health concerns of any kind, or you are considering adopting a new diet or exercise regime, you should consult your doctor.

Sky Bergman's film, Lives Well Lived, explored the joys of growing old, the value of wisdom, and the appreciation of life. Through interviews with people aged 75 and older, the documentary highlighted the resilience, sense of joy and positivity that come with age. The film has now evolved into a book, Lives Well Lived Generations, where Sky shares her personal journey of embracing aging, cherishing every wrinkle and gray hair, and fostering connections across generations. She explores the significance of meaningful conversations between young and older individuals, combating ageism, and promoting mutual understanding.The documentary and book aim to inspire others to appreciate life, engage with different age groups, and recognize the value of shared experiences. In this interview we discuss how Sky's work has reshaped her understanding of life while nurturing a positive attitude towards aging and living fully in the moment.Additional details and show notes Flexbeam Red Light therapyA wearable red light therapy device which targets key parts of the body to improve sleep, treat injuries and sooth aches and pains associated with aging. Code LLAMA. Discounts vary - see details of the current offer hereEnergyBits algae snacksA microscopic form of life that could help us age better. Use code LLAMA for a 20 percent discountSiPhox Health home blood testingMeasure 17 critical blood biomarkers from home. Get a 20% discount with code LLAMA Disclaimer: This post contains affiliate links. If you make a purchase, I may receive a commission at no extra cost to you.Support the showThe Live Long and Master Aging (LLAMA) podcast, a HealthSpan Media LLC production, shares ideas but does not offer medical advice. If you have health concerns of any kind, or you are considering adopting a new diet or exercise regime, you should consult your doctor.

Hey everyone, this is Alex and can you believe that we're almost done with Q1 2024? March 2024 was kind of crazy of course, so I'm of course excited to see what April brings (besides Weights & Biases conference in SF called Fully Connected, which I encourage you to attend and say Hi to me and the team!) This week we have tons of exciting stuff on the leaderboards, say hello to the new best AI in the world Opus (+ some other surprises), in the open source we had new MoEs (one from Mosaic/Databricks folks, which tops the open source game, one from AI21 called Jamba that shows that a transformers alternative/hybrid can actually scale) and tiny MoE from Alibaba, as well as an incredible Emotion TTS from Hume. I also had the pleasure to finally sit down with friend of the pod Tanishq Abraham and Paul Scotti from MedArc and chatted about MindEye 2, how they teach AI to read minds using diffusion models

Giving computers a voice has always been at the center of sci-fi movies; “I'm sorry Dave, I'm afraid I can't do that” wouldn't hit as hard if it just appeared on screen as a terminal output, after all. The first electronic speech synthesizer, the Voder, was built at Bell Labs 85 years ago (1939!), and it's…. something:We will not cover the history of Text To Speech (TTS), but the evolution of the underlying architecture has generally been Formant Synthesis → Concatenative Synthesis → Neural Networks. Nowadays, state of the art TTS is just one API call away with models like Eleven Labs and OpenAI's TTS, or products like Descript. Latency is minimal, they have very good intonation, and can mimic a variety of accents. You can hack together your own voice AI therapist in a day!But once you have a computer that can communicate via voice, what comes next? Singing

Happy leap year day everyone, very excited to bring you a special once-in-a-4 year edition of ThursdAI

Join our new community: https://thisdayinai.com.View the show notes here: https://thisdayinai.com/bookmarks/2-ep49/Build AI Agents & Try AI From The Show: https://simtheory.aiIf you enjoy the podcast, please consider leaving us a review wherever you get your podcasts.====In this episode we reveal the new ThisDayinAI.com community website. We discuss the latest GPT-4 updates, Code Llama 70B open-source release and first impressions, we play around with the new LLaVA-1.6 release and are impressed by its capabilities. We also look at YOLO World and discuss the impact of EAGLE-7B and RWKV Language Models. Finally, we cover Bard's horrible new image creation feature and censorship. CHAPTERS:====00:00 - Introducing ThisDayInAI.com Community5:10 - Be Careful What You Wish For! Mike Gets Spam Called by AI16:16 - OpenAI Announces "improved" GPT-4 Preview Model to Make GPT-4 Less Lazy27:00 - LLaVA-1.6: Improved reasoning, OCR, and world knowledge34:00 - YOLO-World: Real-Time Open-Vocabulary Object Detection45:11 - RWKV an RNN with GPT-level LLM performance and EAGLE7B Impressions58:16 - Google Bard's New Highly Censored Image Creation Feature1:07:13 - Will Google Bard be Renamed to Google Gemini?

Join us as we dive into the latest in technology and AI. We're starting with Facebook's Code LLama, a new large language model aimed at transforming coding. We'll also explore Apple's Vision Pro, highlighting its unique applications and eye tracking technology. Hear about the Rabbit R1, a device designed to automate tasks with the power of AI, and discuss the future of user interfaces where AI simplifies complex actions. Plus, we'll cover the costs associated with training these large models and the feasibility of running open-source models on standard GPUs. Perfect for tech enthusiasts and professionals looking to stay ahead in the digital world.

Meta unveils the Code Llama, an open-source AI model ready to unlock the secrets of code generation. Delve into the possibilities and innovations that await as we discuss the implications of this groundbreaking launch. Get on the AI Box Waitlist: AIBox.ai Join our ChatGPT Community: Facebook Group Follow me on Twitter: Jaeden's Twitter

Explore the frontiers of open-source code generation with Meta's Code Llama. Join us in understanding how this pioneering AI model is shaping the future of coding and unleashing new possibilities for developers worldwide. Get on the AI Box Waitlist: AIBox.ai Join our ChatGPT Community: Facebook Group Follow me on Twitter: Jaeden's Twitter

Journey into the realm of open-source code evolution with Meta's Code Llama. Discover how this AI model is set to redefine the way we generate code, opening doors to innovation and endless possibilities. Get on the AI Box Waitlist: AIBox.ai Join our ChatGPT Community: Facebook Group Follow me on Twitter: Jaeden's Twitter

Join us on an exploration of Meta's latest breakthrough, Code Llama, an open-source AI model that's set to redefine the landscape of code generation. Discover the innovation driving this game-changing technology and its implications for the future. Get on the AI Box Waitlist: AIBox.ai Join our ChatGPT Community: Facebook Group Follow me on Twitter: Jaeden's Twitter

TL;DR of all topics covered + Show notes* Open Source LLMs* Meta releases Code-LLama 70B - 67.8% HumanEval (Announcement, HF instruct version, HuggingChat, Perplexity)* Together added function calling + JSON mode to Mixtral, Mistral and CodeLLama* RWKV (non transformer based) Eagle-7B - (Announcement, Demo, Yam's Thread)* Someone leaks Miqu, Mistral confirms it's an old version of their model* Olmo from Allen Institute - fully open source 7B model (Data, Weights, Checkpoints, Training code) - Announcement* Datasets & Embeddings* Teknium open sources Hermes dataset (Announcement, Dataset, Lilac)* Lilac announces Garden - LLM powered clustering cloud for datasets (Announcement)* BAAI releases BGE-M3 - Multi-lingual (100+ languages), 8K context, multi functional embeddings (Announcement, Github, technical report)* Nomic AI releases Nomic Embed - fully open source embeddings (Announcement, Tech Report)* Big CO LLMs + APIs* Bard with Gemini Pro becomes 2nd LLM in the world per LMsys beating 2 out of 3 GPT4 (Thread)* OpenAI launches GPT mention feature, it's powerful! (Thread)* Vision & Video*

In this episode, we delve into Meta's groundbreaking release of Code Llama 70B, an open source AI model that stands as a formidable competitor to private AI developments. We'll explore its features, potential impacts on the AI landscape, and what this means for the future of open source and proprietary AI technologies. Invest in AI Box: ⁠https://Republic.com/ai-box⁠ Get on the AI Box Waitlist: ⁠https://AIBox.ai/⁠ ⁠AI Facebook Community Learn About ChatGPT Learn About AI at Tesla

Join me as we unravel Meta's Code Llama, an innovative open-source AI model designed to redefine code generation paradigms, exploring its potential impact and applications. Invest in AI Box: https://Republic.com/ai-box Get on the AI Box Waitlist: https://AIBox.ai/ AI Facebook Community

Meta will dieses Jahr NVIDIA-H100-Grafikkarten im Wert von 10 Milliarden Dollar kaufen und damit zum größten Grafikkarten-Inhaber der Welt aufsteigen. In diesem Zuge hat Meta auch kürzlich CodeLlama 70b released, das ihr jetzt auch direkt auf HuggingFace ausprobieren könnt.In Chrome Version 121 gibt es mehrere neue AI Features für die USA. Darunter zählen das smarte Erstellen von Tab Groups und die Möglichkeit Drafts von Gemini in Freitextfeldern auf einer Webseite erstellen zu lassen. Google hat ebenfalls eine Partnerschaft mit HuggingFace angekündigt, die das Hosten von AI-Modellen vereinfachen und sogar auf die Google Kubernetes Engine (GKE) bringen soll.Fabi und Philipp philosophieren darüber, was es an neuen AI Features auf der nächsten WWDC von Apple geben könnte, da es einige Leaks zu Siri und OpenAI gab.OpenAI hat neue Embedding-Modelle sowie ein Preis-Update für GPT-3.5 Turbo veröffentlicht.In dieser Woche gibt es schon wieder ein besseres Modell zum Erstellen von Bildern mit dem eigenen Gesicht als Inputparameter: InstantID von InstantX. Erste Versuche liefern sogar bessere Ergebnisse als Photomaker von Tencent aus der letzten AI News.Die Special Picks of the Day:Philipps Blogbeitrag über LLM Fine TuningLLMs auf Android und der Proof of Concept von unserem Hörer und ehemaligen Podcast-Gast Nico Martin Soundtrack composed by AIVA (Artificial Intelligence Virtual Artist)Schreibt uns! Schickt uns eure Themenwünsche und euer Feedback: podcast@programmier.barFolgt uns! Bleibt auf dem Laufenden über zukünftige Folgen und virtuelle Meetups und beteiligt euch an Community-Diskussionen. TwitterInstagramFacebookMeetupYouTube

Deutschland will AI Act nicht blockieren Code-Qualität nimmt durch KI ab Cyberkriminelle experimentieren mit KI und Meta veröffentlicht Code Llama 70B heise.de/ki-update https://www.heise.de/thema/Kuenstliche-Intelligenz https://the-decoder.de/ https://www.heiseplus.de/podcast

In this episode, we unravel the unveiling of Code Llama, Meta's open-source AI model designed to redefine the landscape of code generation. Join me for a solo exploration as we discuss the features, capabilities, and the potential impact of this groundbreaking addition to the coding world. Invest in AI Box: ⁠https://Republic.com/ai-box⁠ Get on the AI Box Waitlist: ⁠https://AIBox.ai/⁠ ⁠AI Facebook Community Learn About ChatGPT Learn About AI at Tesla

In this episode, we explore the game-changing potential of Meta's Code Llama, an open-source AI model set to revolutionize the coding experience, making it more efficient and user-friendly. Invest in AI Box: ⁠https://Republic.com/ai-box⁠ Get on the AI Box Waitlist: ⁠https://AIBox.ai/⁠ ⁠AI Facebook Community Learn About ChatGPT Learn About AI at Tesla

In this episode, we deep-dive into Meta's groundbreaking Code Llama, an open-source AI model poised to revolutionize code generation and development practices across industries. Invest in AI Box: https://Republic.com/ai-box Get on the AI Box Waitlist: https://AIBox.ai/ AI Facebook Community

In this episode, we analyze Meta's introduction of Code Llama, an open-source AI model aimed at code generation, shedding light on its implications and the possibilities it presents in coding advancements. Invest in AI Box: https://Republic.com/ai-box Get on the AI Box Waitlist: ⁠⁠https://AIBox.ai/⁠⁠ AI Facebook Community Learn more about AI in Video Learn more about Open AI

Docker CTO Justin Cormack reveals that Docker has been a go-to tool for data scientists in AI and machine learning for years, primarily in specialized areas like image processing and prediction models. However, the release of OpenAI's ChatGPT last year sparked a significant surge in Docker's popularity within the AI community.The focus shifted to large language models (LLMs), with a growing interest in the retrieval-augmented generation (RAG) stack. Docker's collaboration with Ollama enables developers to run Llama 2 and Code Llama locally, simplifying the process of starting and experimenting with AI applications. Additionally, partnerships with Neo4j and LangChain allow for enhanced support in storing and retrieving data for LLMs.Cormack emphasizes the simplicity of getting started locally, addressing challenges related to GPU shortages in the cloud. Docker's efforts also include building an AI solution using its data, aiming to assist users in Dockerizing applications through an interactive notebook in Visual Studio Code. This tool leverages LLMs to analyze applications, suggest improvements, and generate Docker files tailored to specific languages and applications.Docker's integration with AI technologies demonstrates a commitment to making AI and Docker more accessible and user-friendly.Learn more from The New Stack about AI and Docker:Artificial Intelligence News, Analysis, and ResourcesWill GenAI Take Jobs? No, Says Docker CEODebugging Containers in Kubernetes — It's Complicated

At the AI Pioneers Summit we announced Latent Space Launchpad, an AI-focused accelerator in partnership with Decibel. If you're an AI founder of enterprise early adopter, fill out this form and we'll be in touch with more details. We also have a lot of events coming up as we wrap up the year, so make sure to check out our community events page and come say hi!We previously interviewed the founders of many developer productivity startups embedded in the IDE, like Codium AI, Cursor, and Codeium. We also covered Replit's (former) SOTA model, replit-code-v1-3b and most recently had Amjad and Michele announce replit-code-v1_5-3b at the AI Engineer Summit.Much has been speculated about the StackOverflow traffic drop since ChatGPT release, but the experience is still not perfect. There's now a new player in the “search for developers” arena: Phind.Phind's goal is to help you find answers to your technical questions, and then help you implement them. For example “What should I use to create a frontend for a Python script?” returns a list of frameworks as well as links to the sources. You can then ask follow up questions on specific implementation details, having it write some code for you, etc. They have both a web version and a VS Code integrationThey recently were top of Hacker News with the announcement of their latest model, which is now the #1 rated model on the BigCode Leaderboard, beating their previous version:TLDR Cheat Sheet:* Based on CodeLlama-34B, which is trained on 500B tokens* Further fine-tuned on 70B+ high quality code and reasoning tokens* Expanded context window to 16k tokens* 5x faster than GPT-4 (100 tok/s vs 20 tok/s on single stream)* 74.7% HumanEval vs 45% for the base modelWe've talked before about HumanEval being limited in a lot of cases and how it needs to be complemented with “vibe based” evals. Phind thinks of evals alongside two axis: * Context quality: when asking the model to generate code, was the context high quality? Did we put outdated examples in it? Did we retrieve the wrong files?* Result quality: was the code generated correct? Did it follow the instructions I gave it or did it misunderstand some of it?If you have bad results with bad context, you might get to a good result by working on better RAG. If you have good context and bad result you might either need to work on your prompting or you have hit the limits of the model, which leads you to fine tuning (like they did). Michael was really early to this space and started working on CommonCrawl filtering and indexing back in 2020, which led to a lot of the insights that now power Phind. We talked about that evolution, his experience at YC, how he got Paul Graham to invest in Phind and invite him to dinner at his house, and how Ron Conway connected him with Jensen Huang to get access to more GPUs!Show Notes* Phind* BigScience T0* InstructGPT Paper* Inception-V3* LMQL* Marginalia Nu* Mistral AI* People:* Paul Graham (pg)* Ron Conway* Yacine Jernite from HuggingFace* Jeff DelaneyTimestamps* [00:00:00] Intros & Michael's early interest in computer vision* [00:03:14] Pivoting to NLP and natural language question answering models* [00:07:20] Building a search engine index of Common Crawl and web pages* [00:11:26] Releasing the first version of Hello based on the search index and BigScience T0 model* [00:14:02] Deciding to focus the search engine specifically for programmers* [00:17:39] Overview of Phind's current product and focus on code reasoning* [00:21:51] The future vision for Phind to go from idea to complete code* [00:24:03] Transitioning to using the GPT-4 model and the impact it had* [00:29:43] Developing the Phind model based on CodeLlama and additional training* [00:32:28] Plans to continue improving the Phind model with open source technologies* [00:43:59] The story of meeting Paul Graham and Ron Conway and how that impacted the company* [00:53:02] How Ron Conway helped them get GPUs from Nvidia* [00:57:12] Tips on how Michael learns complex AI topics* [01:01:12] Lightning RoundTranscriptAlessio: Hey everyone, welcome to the Latent Space Podcast. This is Alessio, partner and CTO of Residence and Decibel Partners, and I'm joined by my co-host Swyx, founder of Smol AI. [00:00:19]Swyx: Hey, and today we have in the studio Michael Royzen from Phind. Welcome. [00:00:23]Michael: Thank you so much. [00:00:24]Alessio: It's great to be here. [00:00:25]Swyx: Yeah, we are recording this in a surprisingly hot October in San Francisco. And sometimes the studio works, but the blue angels are flying by right now, so sorry about the noise. So welcome. I've seen Phind blow up this year, mostly, I think since your launch in Feb and V2 and then your Hacker News posts. We tend to like to introduce our guests, but then obviously you can fill in the blanks with the origin story. You actually were a high school entrepreneur. You started SmartLens, which is a computer vision startup in 2017. [00:00:59]Michael: That's right. I remember when like TensorFlow came out and people started talking about, obviously at the time after AlexNet, the deep learning revolution was already in flow. Good computer vision models were a thing. And what really made me interested in deep learning was I got invited to go to Apple's WWDC conference as a student scholar because I was really into making iOS apps at the time. So I go there and I go to this talk where they added an API that let people run computer vision models on the device using far more efficient GPU primitives. After seeing that, I was like, oh, this is cool. This is going to have a big explosion of different computer vision models running locally on the iPhone. And so I had this crazy idea where it was like, what if I could just make this model that could recognize just about anything and have it run on the device? And that was the genesis for what eventually became SmartLens. I took this data set called ImageNet 22K. So most people, when they think of ImageNet, think of ImageNet 1K. But the full ImageNet actually has, I think, 22,000 different categories. So I took that, filtered it, pre-processed it, and then did a massive fine tune on Inception V3, which was, I think, the state of the art deep convolutional computer vision model at the time. And to my surprise, it actually worked insanely well. I had no idea what would happen if I give a single model. I think it ended up being 17,000 categories approximately that I collapsed them into. It worked so well that it actually worked better than Google Lens, which released its V1 around the same time. And on top of this, the model ran on the device. So it didn't need an internet connection. A big part of the issue with Google Lens at the time was that connections were slower. 4G was around, but it wasn't nearly as fast. So there was a noticeable lag having to upload an image to a server and get it back. But just processing it locally, even on the iPhones of the day in 2017, much faster. It was a cool little project. It got some traction. TechCrunch wrote about it. There was kind of like one big spike in usage, and then over time it tapered off. But people still pay for it, which is wild. [00:03:14]Swyx: That's awesome. Oh, it's like a monthly or annual subscription? [00:03:16]Michael: Yeah, it's like a monthly subscription. [00:03:18]Swyx: Even though you don't actually have any servers? [00:03:19]Michael: Even though we don't have any servers. That's right. I was in high school. I had a little bit of money. I was like, yeah. [00:03:25]Swyx: That's awesome. I always wonder what the modern equivalents kind of "Be my eyes". And it would be actually disclosed in the GPT-4 Vision system card recently that the usage was surprisingly not that frequent. The extent to which all three of us have our sense of sight. I would think that if I lost my sense of sight, I would use Be My Eyes all the time. The average usage of Be My Eyes per day is 1.5 times. [00:03:49]Michael: Exactly. I was thinking about this as well, where I was also looking into image captioning, where you give a model an image and then it tells you what's in the image. But it turns out that what people want is the exact opposite. People want to give a description of an image and then have the AI generate the image. [00:04:04]Alessio: Oh, the other way. [00:04:06]Michael: Exactly. And so at the time, I think there were some GANs, NVIDIA was working on this back in 2019, 2020. They had some impressive, I think, face GANs where they had this model that would produce these really high quality portraits, but it wasn't able to take a natural language description the way Midjourney or DALL-E 3 can and just generate you an image with exactly what you described in it. [00:04:32]Swyx: And how did that get into NLP? [00:04:35]Michael: Yeah, I released the SmartLens app and that was around the time I was a senior in high school. I was applying to college. College rolls around. I'm still sort of working on updating the app in college. But I start thinking like, hey, what if I make an enterprise version of this as well? At the time, there was Clarify that provided some computer vision APIs, but I thought this massive classification model works so well and it's so small and so fast, might as well build an enterprise product. And I didn't even talk to users or do any of those things that you're supposed to do. I was just mainly interested in building a type of backend I've never built before. So I was mainly just doing it for myself just to learn. I built this enterprise classification product and as part of it, I'm also building an invoice processing product where using some of the aspects that I built previously, although obviously it's very different from classification, I wanted to be able to just extract a bunch of structured data from an unstructured invoice through our API. And that's what led me to Hugnyface for the first time because that involves some natural language components. And so I go to Hugnyface and with various encoder models that were around at the time, I used the standard BERT and also Longformer, which came out around the same time. And Longformer was interesting because it had a much bigger context window than those models at the time, like BERT, all of the first gen encoder only models, they only had a context window of 512 tokens and it's fixed. There's none of this alibi or ROPE that we have now where we can basically massage it to be longer. They're fixed, 512 absolute encodings. Longformer at the time was the only way that you can fit, say, like a sequence length or ask a question about like 4,000 tokens worth of text. Implemented Longformer, it worked super well, but like nobody really kind of used the enterprise product and that's kind of what I expected because at the end of the day, it was COVID. I was building this kind of mostly for me, mostly just kind of to learn. And so nobody really used it and my heart wasn't in it and I kind of just shelved it. But a little later, I went back to HugMeFace and I saw this demo that they had, and this is in the summer of 2020. They had this demo made by this researcher, Yacine Jernite, and he called it long form question answering. And basically, it was this self-contained notebook demo where you can ask a question the way that we do now with ChatGPT. It would do a lookup into some database and it would give you an answer. And it absolutely blew my mind. The demo itself, it used, I think, BART as the model and in the notebook, it had support for both an Elasticsearch index of Wikipedia, as well as a dense index powered by Facebook's FAISS. I think that's how you pronounce it. It was very iffy, but when it worked, I think the question in the demo was, why are all boats white? When it worked, it blew my mind that instead of doing this few shot thing, like people were doing with GPT-3 at the time, which is all the rage, you could just ask a model a question, provide no extra context, and it would know what to do and just give you the answer. It blew my mind to such an extent that I couldn't stop thinking about that. When I started thinking about ways to make it better, I tried training, doing the fine tune with a larger BART model. And this BART model, yeah, it was fine tuned on this Reddit data set called Eli5. So basically... [00:08:02]Alessio: Subreddit. [00:08:03]Swyx: Yeah, subreddit. [00:08:04]Alessio: Yeah. [00:08:05]Michael: And put it into like a well-formatted, relatively clean data set of like human questions and human answers. And that was a really great bootstrap for that model to be able to answer these types of questions. And so Eli5 actually turned out to be a good data set for training these types of question answering models, because the question is written by a human, the answer is written by a human, and at least helps the model get the format right, even if the model is still very small and it can't really think super well, at least it gets the format right. And so it ends up acting as kind of a glorified summarization model, where if it's fed in high quality context from the retrieval system, it's able to have a reasonably high quality output. And so once I made the model as big as I can, just fine tuning on BART large, I started looking for ways to improve the index. So in the demo, in the notebook, there were instructions for how to make an Elasticsearch index just for Wikipedia. And I was like, why not do all of Common Crawl? So I downloaded Common Crawl, and thankfully, I had like 10 or $15,000 worth of AWS credits left over from the SmartLens project. And that's what really allowed me to do this, because there's no other funding. I was still in college, not a lot of money, and so I was able to spin up a bunch of instances and just process all of Common Crawl, which is massive. So it's roughly like, it's terabytes of text. I went to Alexa to get the top 1,000 websites or 10,000 websites in the world, then filtered only by those websites, and then indexed those websites, because the web pages were already included in Dump. [00:09:38]Swyx: You mean to supplement Common Crawl or to filter Common Crawl? [00:09:41]Michael: Filter Common Crawl. [00:09:42]Alessio: Oh, okay. [00:09:43]Michael: Yeah, sorry. So we filtered Common Crawl just by the top, I think, 10,000, just to limit this, because obviously there's this massive long tail of small sites that are really cool, actually. There's other projects like, shout out to Marginalia Nu, which is a search engine specialized on the long tail. I think they actually exclude the top 10,000. [00:10:03]Swyx: That's what they do. [00:10:04]Alessio: Yeah. [00:10:05]Swyx: I've seen them around, I just don't really know what their pitch is. Okay, that makes sense. [00:10:08]Michael: So they exclude all the top stuff. So the long tail is cool, but for this, that was kind of out of the question, and that was most of the data anyway. So we've removed that. And then I indexed the remaining approximately 350 million webpages through Elasticsearch. So I built this index running on AWS with these webpages, and it actually worked quite well. You can ask it general common knowledge, history, politics, current events, questions, and it would be able to do a fast lookup in the index, feed it into the model, and it would give a surprisingly good result. And so when I saw that, I thought that this is definitely doable. And it kind of shocked me that no one else was doing this. And so this was now the fall of 2020. And yeah, I was kind of shocked no one was doing this, but it costs a lot of money to keep it up. I was still in college. There are things going on. I got bogged down by classes. And so I ended up shelving this for almost a full year, actually. When I returned to it in fall of 2021, when BigScience released T0, when BigScience released the T0 models, that was a massive jump in the reasoning ability of the model. And it was better at reasoning, it was better at summarization, it was still a glorified summarizer basically. [00:11:26]Swyx: Was this a precursor to Bloom? Because Bloom's the one that I know. [00:11:29]Alessio: Yeah. [00:11:30]Michael: Actually coming out in 2022. But Bloom had other problems where for whatever reason, the Bloom models just were never really that good, which is so sad because I really wanted to use them. But I think they didn't turn on that much data. I think they used like the original, they were trying to replicate GPT-3. So they just use those numbers, which we now know are like far below Chinchilla Optimal and even Chinchilla Optimal, which we can like talk about later, like what we're currently doing with MIMO goes, yeah, it goes way beyond that. But they weren't trying enough data. I'm not sure how that data was clean, but it probably wasn't super clean. And then they didn't really do any fine tuning until much later. So T0 worked well because they took the T5 models, which were closer to Chinchilla Optimal because I think they were trained on also like 300 something billion tokens, similar to GPT-3, but the models were much smaller. I think T0 is the first model that did large scale instruction tuning from diverse data sources in the fall of 2021. This is before Instruct GPT. This is before Flan T5, which came out in 2022. This is the very, very first, at least well-known example of that. And so it came out and then I did, on top of T0, I also did the Reddit Eli5 fine tune. And that was the first model and system that actually worked well enough to where I didn't get discouraged like I did previously, because the failure cases of the BART based system was so egregious. Sometimes it would just miss a question so horribly that it was just extremely discouraging. But for the first time, it was working reasonably well. Also using a much bigger model. I think the BART model is like 800 million parameters, but T0, we were using 3B. So it was T0, 3B, bigger model. And that was the very first iteration of Hello. So I ended up doing a show HN on Hacker News in January 2022 of that system. Our fine tune T0 model connected to our Elasticsearch index of those 350 million top 10,000 common crawl websites. And to the best of my knowledge, I think that's the first example that I'm aware of a LLM search engine model that's effectively connected to like a large enough index that I consider like an internet scale. So I think we were the first to release like an internet scale LLM powered rag search system In January 2022, around the time me and my future co-founder, Justin, we were like, this seems like the future. [00:14:02]Alessio: This is really cool. [00:14:03]Michael: I couldn't really sleep even like I was going to bed and I was like, I was thinking about it. Like I would say up until like 2.30 AM, like reading papers on my phone in bed, go to sleep, wake up the next morning at like eight and just be super excited to keep working. And I was also doing my thesis at the same time, my senior honors thesis at UT Austin about something very similar. We were researching factuality in abstractive question answering systems. So a lot of overlap with this project and the conclusions of my research actually kind of helped guide the development path of Hello. In the research, we found that LLMs, they don't know what they don't know. So the conclusion was, is that you always have to do a search to ensure that the model actually knows what it's talking about. And my favorite example of this even today is kind of with chat GPT browsing, where you can ask chat GPT browsing, how do I run llama.cpp? And chat GPT browsing will think that llama.cpp is some file on your computer that you can just compile with GCC and you're all good. It won't even bother doing a lookup, even though I'm sure somewhere in their internal prompts they have something like, if you're not sure, do a lookup. [00:15:13]Alessio: That's not good enough. So models don't know what they don't know. [00:15:15]Michael: You always have to do a search. And so we approached LLM powered question answering from the search angle. We pivoted to make this for programmers in June of 2022, around the time that we were getting into YC. We realized that what we're really interested in is the case where the models actually have to think. Because up until then, the models were kind of more glorified summarization models. We really thought of them like the Google featured snippets, but on steroids. And so we saw a future where the simpler questions would get commoditized. And I still think that's going to happen with like Google SGE and like it's nowadays, it's really not that hard to answer the more basic kind of like summarization, like current events questions with lightweight models that'll only continue to get cheaper over time. And so we kind of started thinking about this trade off where LLM models are going to get both better and cheaper over time. And that's going to force people who run them to make a choice. Either you can run a model of the same intelligence that you could previously for cheaper, or you can run a better model for the same price. So someone like Google, once the price kind of falls low enough, they're going to deploy and they're already doing this with SGE, they're going to deploy a relatively basic glorified summarizer model that can answer very basic questions about like current events, who won the Super Bowl, like, you know, what's going on on Capitol Hill, like those types of things. The flip side of that is like more complex questions where like you have to reason and you have to solve problems and like debug code. And we realized like we're much more interested in kind of going along the bleeding edge of that frontier case. And so we've optimized everything that we do for that. And that's a big reason of why we've built Phind specifically for programmers, as opposed to saying like, you know, we're kind of a search engine for everyone because as these models get more capable, we're very interested in seeing kind of what the emergent properties are in terms of reasoning, in terms of being able to solve complex multi-step problems. And I think that some of those emerging capabilities like we're starting to see, but we don't even fully understand. So I think there's always an opportunity for us to become more general if we wanted, but we've been along this path of like, what is the best, most advanced reasoning engine that's connected to your code base, that's connected to the internet that we can just provide. [00:17:39]Alessio: What is Phind today, pragmatically, from a product perspective, how do people interact with it? Yeah. Or does it plug into your workflow? [00:17:46]Michael: Yeah. [00:17:47]Alessio: So Phind is really a system. [00:17:48]Michael: Phind is a system for programmers when they have a question or when they're frustrated or when something's not working. [00:17:54]Swyx: When they're frustrated. [00:17:55]Alessio: Yeah. [00:17:56]Michael: For them to get on block. I think like the single, the most abstract page for Phind is like, if you're experiencing really any kind of issue as a programmer, we'll solve that issue for you in 15 seconds as opposed to 15 minutes or longer. Phind has an interface on the web. It has an interface in VS code and more IDEs to come, but ultimately it's just a system where a developer can paste in a question or paste in code that's not working and Phind will do a search on the internet or they will find other code in your code base perhaps that's relevant. And then we'll find the context that it needs to answer your question and then feed it to a reasoning engine powerful enough to actually answer it. So that's really the philosophy behind Phind. It's a system for getting developers the answers that they're looking for. And so right now from a product perspective, this means that we're really all about getting the right context. So the VS code extension that we launched recently is a big part of this because you can just ask a question and it knows where to find the right code context in your code. It can do an internet search as well. So it's up to date and it's not just reliant on what the model knows and it's able to figure out what it needs by itself and answer your question based on that. If it needs some help, you can also get yourself kind of just, there's opportunities for you yourself to put in all that context in. But the issue is also like not everyone wants these VS code. Some people like are real Neovim sticklers or they're using like PyCharm or other IDEs, JetBrains. And so for those people, they're actually like okay with switching tabs, at least for now, if it means them getting their answer. Because really like there's been an explosion of all these like startups doing code, doing search, etc. But really who everyone's competing with is ChatGPT, which only has like that one web interface. Like ChatGPT is really the bar. And so that's what we're up against. [00:19:50]Alessio: And so your idea, you know, we have Amman from Cursor on the podcast and they've gone through the we need to own the IDE thing. Yours is more like in order to get the right answer, people are happy to like go somewhere else basically. They're happy to get out of their IDE. [00:20:05]Michael: That was a great podcast, by the way. But yeah, so part of it is that people sometimes perhaps aren't even in an IDE. So like the whole task of software engineering goes way beyond just running code, right? There's also like a design stage. There's a planning stage. A lot of this happens like on whiteboards. It happens in notebooks. And so the web part also exists for that where you're not even coding it and you're just trying to get like a more conceptual understanding of what you're trying to build first. The podcast with Amman was great, but somewhere where I disagree with him is that you need to own the IDE. I think like he made some good points about not having platform risk in the long term. But some of the features that were mentioned like suggesting diffs, for example, those are all doable with an extension. We haven't yet seen with VS Code in particular any functionality that we'd like to do yet in the IDE that we can't either do through directly supported VS Code functionality or something that we kind of hack into there, which we've also done a fair bit of. And so I think it remains to be seen where that goes. But I think what we're looking to be is like we're not trying to just be in an IDE or be an IDE. Like Phind is a system that goes beyond the IDE and like is really meant to cover the entire lifecycle of a developer's thought process in going about like, hey, like I have this idea and I want to get from that idea to a working product. And so then that's what the long term vision of Phind is really about is starting with that. In the future, I think programming is just going to be really just the problem solving. Like you come up with an idea, you come up with like the basic design for the algorithm in your head, and you just tell the AI, hey, just like just do it, just make it work. And that's what we're building towards. [00:21:51]Swyx: I think we might want to give people an impression about like type of traffic that you have, because when you present it with a text box, you could type in anything. And I don't know if you have some mental categorization of like what are like the top three use cases that people tend to coalesce around. [00:22:08]Alessio: Yeah, that's a great question. [00:22:09]Michael: The two main types of searches that we see are how-to questions, like how to do X using Y tool. And this historically has been our bread and butter, because with our embeddings, like we're really, really good at just going over a bunch of developer documentation and figuring out exactly the part that's relevant and just telling you, OK, like you can use this method. But as LLMs have gotten better, and as we've really transitioned to using GPT-4 a lot in our product, people organically just started pasting in code that's not working and just said, fix it for me. [00:22:42]Swyx: Fix this. [00:22:43]Alessio: Yeah. [00:22:44]Michael: And what really shocks us is that a lot of the people who do that, they're coming from chat GPT. So they tried it in chat GPT with chat GPT-4. It didn't work. Maybe it required like some multi-step reasoning. Maybe it required some internet context or something found in either a Stack Overflow post or some documentation to solve it. And so then they paste it into find and then find works. So those are really those two different cases. Like, how can I build this conceptually or like remind me of this one detail that I need to build this thing? Or just like, here's this code. Fix it. And so that's what a big part of our VS Code extension is, is like enabling a much smoother here just like fix it for me type of workflow. That's really its main benefits. Like it's in your code base. It's in the IDE. It knows how to find the relevant context to answer that question. But at the end of the day, like I said previously, that's still a relatively, not to say it's a small part, but it's a limited part of the entire mental life cycle of a programmer. [00:23:47]Swyx: Yep. So you launched in Feb and then you launched V2 in August. You had a couple other pretty impactful posts slash feature launches. The web search one was massive. So you were mostly a GPT-4 wrapper. We were for a long time. [00:24:03]Michael: For a long time until recently. Yeah. [00:24:05]Alessio: Until recently. [00:24:06]Swyx: So like people coming over from ChatGPT were saying, we're going to say model with your version of web search. Would that be the primary value proposition? [00:24:13]Michael: Basically yeah. And so what we've seen is that any model plus web search is just significantly better than [00:24:18]Alessio: that model itself. Do you think that's what you got right in April? [00:24:21]Swyx: Like so you got 1500 points on Hacking News in April, which is like, if you live on Hacking News a lot, that is unheard of for someone so early on in your journey. [00:24:31]Alessio: Yeah. [00:24:32]Michael: We're super, super grateful for that. Definitely was not expecting it. So what we've done with Hacker News is we've just kept launching. [00:24:38]Alessio: Yeah. [00:24:39]Michael: Like what they don't tell you is that you can just keep launching. That's what we've been doing. So we launched the very first version of Find in its current incarnation after like the previous demo connected to our own index. Like once we got into YC, we scrapped our own index because it was too cumbersome at the time. So we moved over to using Bing as kind of just the raw source data. We launched as Hello Cognition. Over time, every time we like added some intelligence to the product, a better model, we just keep launching. And every additional time we launched, we got way more traffic. So we actually silently rebranded to Find in late December of last year. But like we didn't have that much traffic. Nobody really knew who we were. [00:25:18]Swyx: How'd you pick the name out of it? [00:25:19]Michael: Paul Graham actually picked it for us. [00:25:21]Swyx: All right. [00:25:22]Alessio: Tell the story. Yeah. So, oh boy. [00:25:25]Michael: So this is the biggest side. Should we go for like the full Paul Graham story or just the name? [00:25:29]Swyx: Do you want to do it now? Or do you want to do it later? I'll give you a choice. [00:25:32]Alessio: Hmm. [00:25:33]Michael: I think, okay, let's just start with the name for now and then we can do the full Paul Graham story later. But basically, Paul Graham, when we were lucky enough to meet him, he saw our name and our domain was at the time, sayhello.so and he's just like, guys, like, come on, like, what is this? You know? And we were like, yeah, but like when we bought it, you know, we just kind of broke college students. Like we didn't have that much money. And like, we really liked hello as a name because it was the first like conversational search engine. And that's kind of, that's the angle that we were approaching it from. And so we had sayhello.so and he's like, there's so many problems with that. Like, like, like the say hello, like, what does that even mean? And like .so, like, it's gotta be like a .com. And so we did some time just like with Paul Graham in the room. We just like looked at different domain names, like different things that like popped into our head. And one of the things that popped into like Paul Graham said was fine with the Phind spelling in particular. [00:26:33]Swyx: Yeah. Which is not typical naming advice, right? Yes. Because it's not when people hear it, they don't spell it that way. [00:26:38]Michael: Exactly. It's hard to spell. And also it's like very 90s. And so at first, like, we didn't like, I was like, like, ah, like, I don't know. But over time it kept growing on us. And eventually we're like, okay, we like the name. It's owned by this elderly Canadian gentleman who we got to know, and he was willing to sell it to us. [00:26:57]Michael: And so we bought it and we changed the name. Yeah. [00:27:01]Swyx: Anyways, where were you? [00:27:02]Alessio: I had to ask. [00:27:03]Swyx: I mean, you know, everyone who looks at you is wondering. [00:27:06]Michael: And a lot of people actually pronounce it Phind, which, you know, by now it's part of the game. But eventually we want to buy Phind.com and then just have that redirect to Phind. So Phind is like definitely the right spelling. But like, we'll just, yeah, we'll have all the cases addressed. [00:27:23]Swyx: Cool. So Bing web search, and then August you launched V2. Is V2 the Phind as a system pitch? Or have you moved, evolved since then? [00:27:31]Michael: Yeah, so I don't, like the V2 moniker, like, I don't really think of it that way in my mind. There's like, there's the version we launched during, last summer during YC, which was the Bing version directed towards programmers. And that's kind of like, that's why I call it like the first incarnation of what we currently are. Because it was already directed towards programmers. We had like a code snippet search built in as well, because at the time, you know, the models we were using weren't good enough to generate code snippets. Even GPT, like the text DaVinci 2 was available at the time, wasn't that good at generating code and it would generate like very, very short, very incomplete code snippets. And so we launched that last summer, got some traction, but really like we were only doing like, I don't know, maybe like 10,000 searches a day. [00:28:15]Alessio: Some people knew about it. [00:28:16]Michael: Some people used it, which is impressive because looking back, the product like was not that good. And every time we've like made an improvement to the way that we retrieve context through better embeddings, more intelligent, like HTML parsers, and importantly, like better underlying models. Every major version after that was when we introduced a better underlying answering model. Like in February, we had to swallow a bit of our pride when we were like, okay, our own models aren't good enough. We have to go to open AI. And actually that did lead to kind of like our first decent bump of traffic in February. And people kept using it, like our attention was way better too. But we were still kind of running into problems of like more advanced reasoning. Some people tried it, but people were leaving because even like GPT 3.5, both turbo and non-turbo, like still not that great at doing like code related reasoning beyond the how do you do X, like documentation search type of use case. And so it was really only when GPT 4 came around in April that we were like, okay, like this is like our first real opportunity to really make this thing like the way that it should have been all along. And having GPT 4 as the brain is what led to that Hacker News post. And so what we did was we just let anyone use GPT 4 on Fyne for free without a login, [00:29:43]Alessio: which I actually don't regret. [00:29:45]Michael: So it was very expensive, obviously. But like at that stage, all we needed to do was show like, we just needed to like show people here's what Fyne can do. That was the main thing. And so that worked. That worked. [00:29:58]Alessio: Like we got a lot of users. [00:29:59]Michael: Do you know Fireship? [00:30:01]Swyx: Yeah. YouTube, Jeff Delaney. [00:30:03]Michael: Yeah. He made a short about Fyne. [00:30:06]Alessio: Oh. [00:30:07]Michael: And that's on top of the Hacker News post. And that's what like really, really made it blow up. It got millions of views in days. And he's just funny. Like what I love about Fireship is like he like you guys, yeah, like humor goes a long a long way towards like really grabbing people's attention. And so that blew up. [00:30:25]Swyx: Something I would be anxious about as a founder during that period, so obviously we all remember that pretty closely. So there were a couple of people who had access to the GPT-4 API doing this, which is unrestricted access to GPT-4. And I have to imagine OpenAI wasn't that happy about that because it was like kind of de facto access to GPT-4 before they released it. [00:30:46]Alessio: No, no. [00:30:47]Michael: GPT-4 was in chat GPT from day one. I think. OpenAI actually came to our support because what happened was we had people building unofficial APIs around to try to get free access to it. And I think OpenAI actually has the right perspective on this where they're like, OK, people can do whatever they want with the API if they're paying for it, like they can do whatever they want, but it's like not OK if, you know, paying customers are being exploite by these other actors. They actually got in touch with us and they helped us like set up better Cloudflare bot monitoring controls to effectively like crack down on those unofficial APIs, which we're very happy about. But yeah, so we launched GPT-4. A lot of people come to the product and yeah, for a long time, we're just we're figuring out like what do we make of this, right? How do we a make it better, but also deal with like our costs, which have just like massively, massively ballooned. Over time, it's become more clear with the release of Llama 2 and Llama 3 on the horizon that we will once again see a return to vertical applications running their own models. As was true last year and before, I think that GPT-4, my hypothesis is that the jump from 4 to 4.5 or 4 to 5 will be smaller than the jump from 3 to 4. And the reason why is because there were a lot of different things. Like there was two plus, effectively two, two and a half years of research that went into going from 3 to 4. Like more data, bigger model, all of the instruction tuning techniques, RLHF, all of that is known. And like Meta, for example, and now there's all these other startups like Mistral too, like there's a bunch of very well-funded open source players that are now working on just like taking the recipe that's now known and scaling it up. So I think that even if a delta exists, the delta between in 2024, the delta between proprietary and open source won't be large enough that a startup like us with a lot of data that we've collected can take the data that we have, fine tune an open source model, and like be able to have it be better than whatever the proprietary model is at the time. That's my hypothesis.Michael: But we'll once again see a return to these verticalized models. And that's something that we're super excited about because, yeah, that brings us to kind of the fine model because the plan from kind of the start was to be able to return to that if that makes sense. And I think now we're definitely at a point where it does make sense because we have requests from users who like, they want longer context in the model, basically, like they want to be able to ask questions about their entire code base without, you know, context and retrieval and taking a chance of that. Like, I think it's generally been shown that if you have the space to just put the raw files inside of a big context window, that is still better than chunking and retrieval. So there's various things that we could do with longer context, faster speed, lower cost. Super excited about that. And that's the direction that we're going with the fine model. And our big hypothesis there is precisely that we can take a really good open source model and then just train it on absolutely all of the high quality data that we can find. And there's a lot of various, you know, interesting ideas for this. We have our own techniques that we're kind of playing with internally. One of the very interesting ideas that I've seen, I think it's called Octopack from BigCode. I don't think that it made that big waves when it came out, I think in August. But the idea is that they have this data set that maps GitHub commits to a change. So basically there's all this really high quality, like human made, human written diff data out there on every time someone makes a commit in some repo. And you can use that to train models. Take the file state before and like given a commit message, what should that code look like in the future? [00:34:52]Swyx: Got it. [00:34:53]Alessio: Do you think your HumanEval is any good?Michael: So we ran this experiment. We trained the Phind model. And if you go to the BigCode leaderboard, as of today, October 5th, all of our models are at the top of the BigCode leaderboard by far. It's not close, particularly in languages other than Python. We have a 10 point gap between us and the next best model on JavaScript. I think C sharp, multilingual. And what we kind of learned from that whole experience releasing those models is that human eval doesn't really matter. Not just that, but GPT-4 itself has been trained on human eval. And we know this because GPT-4 is able to predict the exact docstring in many of the problems. I've seen it predict like the specific example values in the docstring, which is extremely improbable. So I think there's a lot of dataset contamination and it only captures a very limited subset of what programmers are actually doing. What we do internally for evaluations are we have GPT-4 score answers. GPT-4 is a really good evaluator. I mean, obviously it's by really good, I mean, it's the best that we have. I'm sure that, you know, a couple of months from now, next year, we'll be like, oh, you know, like GPT-4.5, GPT-5, it's so much better. Like GPT-4 is terrible, but like right now it's the best that we have short of humans. And what we found is that when doing like temperature zero evals, it's actually mostly deterministic GPT-4 across runs in assigning scores to two different answers. So we found it to be a very useful tool in comparing our model to say, GPT-4, but yeah, on our like internal real world, here's what people will be asking this model dataset. And the other thing that we're running is just like releasing the model to our users and just seeing what they think. Because that's like the only thing that really matters is like releasing it for the application that it's intended for, and then seeing how people react. And for the most part, the incredible thing is, is that people don't notice a difference between our model and GPT-4 for the vast majority of searches. There's some reasoning problems that GPT-4 can still do better. We're working on addressing that. But in terms of like the types of questions that people are asking on find, there's not that much difference. And in fact, I've been running my own kind of side by side comparisons, shout out to GodMode, by the way. [00:37:16]Michael: And I've like myself, I've kind of confirmed this to be the case. And even sometimes it gives a better answer, perhaps like more concise or just like better implementation than GPT-4, which that's what surprises me. And by now we kind of have like this reasoning is all you need kind of hypothesis where we've seen emerging capabilities in the find model, whereby training it on high quality code, it can actually like reason better. It went from not being able to solve world problems, where riddles were like with like temporal placement of objects and moving and stuff like that, that GPT-4 can do pretty well. We went from not being able to do those at all to being able to do them just by training on more code, which is wild. So we're already like starting to see like these emerging capabilities. [00:37:59]Swyx: So I just wanted to make sure that we have the, I guess, like the model card in our heads. So you started from Code Llama? [00:38:07]Alessio: Yes. [00:38:08]Swyx: 65, 34? 34. [00:38:10]Michael: So unfortunately, there's no Code Llama 70b. If there was, that would be super cool. But there's not. [00:38:15]Swyx: 34. And then, which in itself was Llama 2, which is on 2 trillion tokens and the added 500 billion code tokens. Yes. [00:38:22]Michael: And you just added a bunch more. [00:38:23]Alessio: Yeah. [00:38:24]Michael: And they also did a couple of things. So they did, I think they did 500 billion, like general pre-training and then they did an extra 20 billion long context pre-training. So they actually increased the like max position tokens to 16k up from 8k. And then they changed the theta parameter for the ROPE embeddings as well to give it theoretically better long context support up to 100k tokens. But yeah, but otherwise it's like basically Llama 2. [00:38:50]Swyx: And so you just took that and just added data. [00:38:52]Michael: Exactly. [00:38:53]Swyx: You didn't do any other fundamental. [00:38:54]Michael: Yeah. So we didn't actually, we haven't yet done anything with the model architecture and we just trained it on like many, many more billions of tokens on our own infrastructure. And something else that we're taking a look at now is using reinforcement learning for correctness. One of the interesting pitfalls that we've noticed with the Phind model is that in cases where it gets stuff wrong, it sometimes is capable of getting the right answer. It's just, there's a big variance problem. It's wildly inconsistent. There are cases when it is able to get the right chain of thought and able to arrive [00:39:25]Alessio: at the right answer, but not always. [00:39:27]Michael: And so like one of our hypotheses is something that we're going to try is that like we can actually do reinforcement learning on, for a given problem, generate a bunch of completions and then like use the correct answer as like a loss basically to try to get it to be more correct. And I think there's a high chance I think of this working because it's very similar to the like RLHF method where you basically show pairs of completions for a given question except the criteria is like which one is like less harmful. But here we have a different criteria. But if the model is already capable of getting the right answer, which it is, we're just, we just need to cajole it into being more consistent. [00:40:06]Alessio: There were a couple of things that I noticed in the product that were not strange but unique. So first of all, the model can talk multiple times in a row, like most other applications is like human model, human model. And then you had outside of the thumbs up, thumbs down, you have things like have DLLM prioritize this message and its answers or then continue from this message to like go back. How does that change the flow of the user and like in terms of like prompting it, yeah, what are like some tricks or learnings you've had? [00:40:37]Michael: So yeah, that's specifically in our pair programmer mode, which is a more conversational mode that also like asks you clarifying questions back if it doesn't fully understand what you're doing and it kind of it holds your hand a bit more. And so from user feedback, we had requests to make more of an auto GPT where you can kind of give it this problem that might take multiple searches or multiple different steps like multiple reasoning steps to solve. And so that's the impetus behind building that product. Being able to do multiple steps and also be able to handle really long conversations. Like people are really trying to use the pair programmer to go from like sometimes really from like basic idea to like complete working code. And so we noticed was is that we were having like these very, very long threads, sometimes with like 60 messages, like 100 messages. And like those become really, really challenging to manage the appropriate context window of what should go inside of the context and how to preserve the context so that the model can continue or the product can continue giving good responses, even if you're like 60 messages deep in a conversation. So that's where the prioritized user messages like comes from. It's like people have asked us to just like let them pin messages that they want to be left in the conversation. And yeah, and then that seems to have like really gone a long way towards solving that problem, yeah. [00:41:54]Alessio: And then you have a run on Replit thing. Are you planning to build your own repl? Like learning some people trying to run the wrong code, unsafe code? [00:42:03]Michael: Yes. Yes. So I think like in the long term vision of like being a place where people can go from like idea to like fully working code, having a code sandbox, like a natively integrated code sandbox makes a lot of sense. And replit is great and people use that feature. But yeah, I think there's more we can do in terms of like having something a bit closer to code interpreter where it's able to run the code and then like recursively iterate on it. Exactly. [00:42:31]Swyx: So you're working on APIs to enable you to do that? Yep. So Amjad has specifically told me in person that he wants to enable that for people at the same time. He's also working on his own models, and Ghostwriter and you know, all the other stuff. So it's going to get interesting. Like he wants to power you, but also compete with you. Yeah. [00:42:47]Michael: And like, and we love replit. I think that a lot of the companies in our space, like we're all going to converge to solving a very similar problem, but from a different angle. So like replit approaches this problem from the IDE side. Like they started as like this IDE that you can run in the browser. And they started from that side, making coding just like more accessible. And we're approaching it from the side of like an LLM that's just like connected to everything that it needs to be connected to, which includes your code context. So that's why we're kind of making inroads into IDEs, but we're kind of, we're approaching this problem from different sides. And I think it'll be interesting to see where things end up. But I think that in the long, long term, we have an opportunity to also just have like this general technical reasoning engine product that's potentially also not just for, not just for programmers. It's also powered in this web interface, like where there's potential, I think other things that we will build that eventually might go beyond like our current scope. [00:43:49]Swyx: Exciting. We'll look forward to that. We're going to zoom out a little bit into sort of AI ecosystem stories, but first we got to get the Paul Graham, Ron Conway story. [00:43:59]Alessio: Yeah. [00:44:00]Michael: So flashback to last summer, we're in the YC batch. We're doing the summer batch, summer 22. So the summer batch runs from June to September, approximately. And so this was late July, early August, right around the time that many like YC startups start like going out, like during up, here's how we're going to pitch investors and everything. And at the same time, me and my co-founder, Justin, we were planning on moving to New York. So for a long time, actually, we were thinking about building this company in New York, mainly for personal reasons, actually, because like during the pandemic, pre-ChatGPT, pre last year, pre the AI boom, SF unfortunately really kind of, you know, like lost its luster. Yeah. Like no one was here. It was far from clear, like if there would be an AI boom, if like SF would be like... [00:44:49]Alessio: Back. [00:44:50]Michael: Yeah, exactly. Back. As everyone is saying these days, it was far from clear. And so, and all of our friends, we were graduating college because like we happened to just graduate college and immediately start YC, like we didn't even have, I think we had a week in between. [00:45:06]Swyx: You didn't bother looking for jobs. You were just like, this is what we want to do. [00:45:08]Michael: Well, actually both me and my co-founder, we had jobs that we secured in 2021 from previous internships, but we both, funny enough, when I spoke to my boss's boss at the company at where I reneged my offer, I told him we got into YC, they actually said, yeah, you should do YC. [00:45:27]Swyx: Wow. [00:45:28]Alessio: That's very selfless. [00:45:29]Swyx: That was really great that they did that. But in San Francisco, they would have offered to invest as well. [00:45:33]Michael: Yes, they would have. But yeah, but we were both planning to be in New York and all of our friends were there from college at this point, like we have this whole plan where like on August 1st, we're going to move to New York and we had like this Airbnb for the month of New York. We're going to stay there and we're going to work and like all of that. The day before we go to New York, I called Justin and I just, I tell him like, why are we doing this? Because in our batch, by the time August 1st rolled around, all of our mentors at YC were saying like, hey, like you should really consider staying in SF. [00:46:03]Swyx: It's the hybrid batch, right? [00:46:04]Michael: Yeah, it was the hybrid batch, but like there were already signs that like something was kind of like afoot in SF, even if like we didn't fully want to admit it yet. And so we were like, I don't know, I don't know. Something kind of clicked when the rubber met the road and it was time to go to New York. We're like, why are we doing this? And like, we didn't have any good reasons for staying in New York at that point beyond like our friends are there. So we still go to New York because like we have the Airbnb, like we don't have any other kind of place to go for the next few weeks. We're in New York and New York is just unfortunately too much fun. Like all of my other friends from college who are just, you know, basically starting their jobs, starting their lives as adults. They just stepped into these jobs, they're making all this money and they're like partying and like all these things are happening. And like, yeah, it's just a very distracting place to be. And so we were just like sitting in this like small, you know, like cramped apartment, terrible posture, trying to get as much work done as we can, too many distractions. And then we get this email from YC saying that Paul Graham is in town in SF and he is doing office hours with a certain number of startups in the current batch. And whoever signs up first gets it. And I happen to be super lucky. I was about to go for a run, but I just, I saw the email notification come across the street. I immediately clicked on the link and like immediately, like half the spots were gone, but somehow the very last spot was still available. And so I picked the very, very last time slot at 7 p.m. semi-strategically, you know, so we would have like time to go over. And also because I didn't really know how we're going to get to SF yet. And so we made a plan that we're going to fly from New York to SF and back to New York in one day and do like the full round trip. And we're going to meet with PG at the YC Mountain View office. And so we go there, we do that, we meet PG, we tell him about the startup. And one thing I love about PG is that he gets like, he gets so excited. Like when he gets excited about something, like you can see his eyes like really light up. And he'll just start asking you questions. In fact, it's a little challenging sometimes to like finish kind of like the rest of like the description of your pitch because like, he'll just like asking all these questions about how it works. And I'm like, you know, what's going on? [00:48:19]Swyx: What was the most challenging question that he asked you? [00:48:21]Michael: I think that like really how it worked. Because like as soon as like we told him like, hey, like we think that the future of search is answers, not links. Like we could really see like the gears turning in his head. I think we were like the first demo of that. [00:48:35]Swyx: And you're like 10 minutes with him, right? [00:48:37]Michael: We had like 45, yeah, we had a decent chunk of time. And so we tell him how it works. Like he's very excited about it. And I just like, I just blurted out, I just like asked him to invest and he hasn't even seen the product yet. We just asked him to invest and he says, yeah. And like, we're super excited about that. [00:48:55]Swyx: You haven't started your batch. [00:48:56]Michael: No, no, no. This is about halfway through the batch or two, two, no, two thirds of the batch. [00:49:02]Swyx: And you're like not technically fundraising yet. We're about to start fundraising. Yeah. [00:49:06]Michael: So we have like this demo and like we showed him and like there was still a lot of issues with the product, but I think like it must have like still kind of like blown his mind in some way. So like we're having fun. He's having fun. We have this dinner planned with this other friend that we had in SF because we were only there for that one day. So we thought, okay, you know, after an hour we'll be done, you know, we'll grab dinner with our friend and we'll fly back to New York. But PG was like, like, I'm having so much fun. Do you want to have dinner? Yeah. Come to my house. Or he's like, I gotta go have dinner with my wife, Jessica, who's also awesome, by the way. [00:49:40]Swyx: She's like the heart of YC. Yeah. [00:49:42]Michael: Jessica does not get enough credit as an aside for her role. [00:49:46]Swyx: He tries. [00:49:47]Michael: He understands like the technical side and she understands people and together they're just like a phenomenal team. But he's like, yeah, I got to go see Jessica, but you guys are welcome to come with. Do you want to come with? And we're like, we have this friend who's like right now outside of like literally outside the door who like we also promised to get dinner with. It's like, we'd love to, but like, I don't know if we can. He's like, oh, he's welcome to come too. So all of us just like hop in his car and we go to his house and we just like have this like we have dinner and we have this just chat about the future of search. Like I remember him telling Jessica distinctly, like our kids as kids are not going to know what like a search result is. Like they're just going to like have answers. That was really like a mind blowing, like inflection point moment for sure. [00:50:34]Swyx: Wow, that email changed your life. [00:50:35]Michael: Absolutely. [00:50:36]Swyx: And you also just spoiled the booking system for PG because now everyone's just going to go after the last slot. Oh man. [00:50:42]Michael: Yeah. But like, I don't know if he even does that anymore. [00:50:46]Swyx: He does. He does. Yeah. I've met other founders that he did it this year. [00:50:49]Michael: This year. Gotcha. But when we told him about how we did it, he was like, I am like frankly shocked that YC just did like a random like scheduling system. [00:50:55]Alessio: They didn't like do anything else. But, um. [00:50:58]Swyx: Okay. And then he introduces Duron Conway. Yes. Who is one of the most legendary angels in Silicon Valley. [00:51:04]Michael: Yes.So after PG invested, the rest of our round came together pretty quickly. [00:51:10]Swyx: I'm, by the way, I'm surprised. Like it's, it might feel like playing favorites right within the current batch to be like, yo, PG invested in this one. Right. [00:51:17]Alessio: Too bad for the others. [00:51:18]Swyx: Too bad for the others, I guess. [00:51:19]Michael: I think this is a bigger point about YC and like these accelerators in general is like YC gets like a lot of criticism from founders who feel like they didn't get value out of it. But like, in my view, YC is what you make of it. And YC tells you this. They're like, you really got to grab this opportunity, like buy the balls and make the most of it. And if you do, then it could be the best thing in the world. And if you don't, and if you're just kind of like a passive, even like an average founder in YC, you're still going to fail. And they tell you that. They're like, if you're average in your batch, you're going to fail. Like you have to just be exceptional in every way. With that in mind, perhaps that's even part of the reason why we asked PG to invest. And so yeah, after PG invested, the rest of our round came together pretty quickly, which I'm very fortunate for. And yeah, he introduced us to Ron. And after he did, I get a call from Ron. And then Ron says like, hey, like PG tells me what you're working on. I'd love to come meet you guys. And I'm like, wait, no way. And then we're just holed up in this like little house in San Mateo, which is a little small, but you know, it had a nice patio. In fact, we had like a monitor set up outside on the deck out there. And so Ron Conway comes over, we go over to the patio where like our workstation is. And Ron Conway, he's known for having like this notebook that he goes around with where he like sits down with the notebook and like takes very, very detailed notes. So he never like forgets anything. So he sits down with his notebook and he asks us like, hey guys, like, what do you need? And we're like, oh, we need GPUs. Back then, the GPU shortage wasn't even nearly as bad as it is now. But like even then, it was still challenging to get like the quota that we needed. And he's like, okay, no problem. And then like he leaves a couple hours later, we get an email and we're CC'd on an email that Ron wrote to Jensen, the CEO of Nvidia, saying like, hey, these guys need GPUs. [00:53:02]Swyx: You didn't say how much? It was just like, just give them GPUs. [00:53:04]Alessio: Basically, yeah. [00:53:05]Michael: Ron is known for writing these like one-liner emails that are like very short, but very to the point. And I think that's why like everyone responds to Ron. Everyone loves Ron. And so Jensen responds. He responds quickly, like tagging this VP of AI at Nvidia. And we start working with Nvidia, which is great. And something that I love about Nvidia, by the way, is that after that intro, we got matched with like a dedicated team. And at Nvidia, they know that they're going to win regardless. So they don't care where you get the GPUs from. They're like, they're truly neutral, unlike various sales reps that you might encounter at various like clouds and, you know, hardware companies, et cetera. They actually just want to help you because they know they don't care. Like regardless, they know that if you're getting Nvidia GPUs, they're still winning. So I guess that's a tip is that like if you're looking for GPUs like Nvidia, they'll help you do it. [00:53:54]Swyx: So just to tie up this thing, because so first of all, that's a fantastic story. And I just wanted to let you tell that because it's special. That is a strategic shift, right? That you already decided to make by the time you met Ron, which is we are going to have our own hardware. We're going to rack him in a data center somewhere. [00:54:11]Michael: Well, not even that we need our own hardware because actually we don't. Right. But we just we just need GPUs, period. And like every cloud loves like they have their own sales tactics and like they want to make you commit to long terms and like very non-flexible terms. And like there's a web of different things that you kind of have to navigate. Nvidia will kind of be to the point like, OK, you can do this on this cloud, this on this cloud. Like this is your budget. Maybe you want to consider buying as well. Like they'll help you walk through what the options are. And the reason why they're helpful is because like they look at the full picture. So they'll help you with the hardware. And in terms of software, they actually implemented a custom feature for us in Faster Transformer, which is one of their libraries.Swyx: For you? [00:54:53]Michael: For us. Yeah. Which is wild. I don't think they would have done it otherwise. They implemented streaming generation for T5 based models, which we were running at the time up until we switched to GPT in February, March of this year. So they implemented that just for us, actually, in Faster Transformer. And so like they'll help you like look at the complete picture and then just help you get done what you need to get done. I know one of your interests is also local models, open source models and hardware kind of goes hand in hand.Alessio: Any fun projects, explorations in the space that you want to share with local llamas and stuff? [00:55:27]Michael: Yeah, it's something that we're very interested in because something that kind of we're hearing a lot about is like people want something like find, especially comp

Discover Meta's latest innovation as they launch Code Llama, an open-source AI model revolutionizing code generation. Join us as we delve into the potential impact of this cutting-edge technology in software development and its broader implications for AI in the tech industry. Tune in to gain insights into how Code Llama aims to shape the future of coding. Get on the AI Box Waitlist: https://AIBox.ai/Join our ChatGPT Community: ⁠https://www.facebook.com/groups/739308654562189/⁠Follow me on Twitter: ⁠https://twitter.com/jaeden_ai⁠

Thanks to the over 17,000 people who have joined the first AI Engineer Summit! A full recap is coming. Last call to fill out the State of AI Engineering survey! See our Community page for upcoming meetups in SF, Paris and NYC.This episode had good interest on Twitter.Fast.ai's “Practical Deep Learning” courses been watched by over >6,000,000 people, and the fastai library has over 25,000 stars on Github. Jeremy Howard, one of the creators of Fast, is now one of the most prominent and respected voices in the machine learning industry; but that wasn't always the case. Being non-consensus and right In 2018, Jeremy and Sebastian Ruder published a paper on ULMFiT (Universal Language Model Fine-tuning), a 3-step transfer learning technique for NLP tasks: The paper demonstrated that pre-trained language models could be fine-tuned on a specific task with a relatively small amount of data to achieve state-of-the-art results. They trained a 24M parameters model on WikiText-103 which was beat most benchmarks.While the paper had great results, the methods behind weren't taken seriously by the community: “Everybody hated fine tuning. Everybody hated transfer learning. I literally did tours trying to get people to start doing transfer learning and nobody was interested, particularly after GPT showed such good results with zero shot and few shot learning […] which I was convinced was not the right direction, but who's going to listen to me, cause as you said, I don't have a PhD, not at a university… I don't have a big set of computers to fine tune huge transformer models.”Five years later, fine-tuning is at the center of most major discussion topics in AI (we covered some like fine tuning vs RAG and small models fine tuning), and we might have gotten here earlier if Jeremy had OpenAI-level access to compute and distribution. At heart, Jeremy has always been “GPU poor”:“I've always been somebody who does not want to build stuff on lots of big computers because most people don't have lots of big computers and I hate creating stuff that most people can't use.”This story is a good reminder of how some of the best ideas are hiding in plain sight; we recently covered RWKV and will continue to highlight the most interesting research that isn't being done in the large labs. Replacing fine-tuning with continued pre-trainingEven though fine-tuning is now mainstream, we still have a lot to learn. The issue of “catastrophic forgetting” and potential solutions have been brought up in many papers: at the fine-tuning stage, the model can forget tasks it previously knew how to solve in favor of new ones. The other issue is apparent memorization of the dataset even after a single epoch, which Jeremy covered Can LLMs learn from a single example? but we still don't have the answer to. Despite being the creator of ULMFiT, Jeremy still professes that there are a lot of open questions on finetuning:“So I still don't know how to fine tune language models properly and I haven't found anybody who feels like they do.”He now advocates for "continued pre-training" - maintaining a diversity of data throughout the training process rather than separate pre-training and fine-tuning stages. Mixing instructional data, exercises, code, and other modalities while gradually curating higher quality data can avoid catastrophic forgetting and lead to more robust capabilities (something we covered in Datasets 101).“Even though I originally created three-step approach that everybody now does, my view is it's actually wrong and we shouldn't use it… the right way to do this is to fine-tune language models, is to actually throw away the idea of fine-tuning. There's no such thing. There's only continued pre-training. And pre-training is something where from the very start, you try to include all the kinds of data that you care about, all the kinds of problems that you care about, instructions, exercises, code, general purpose document completion, whatever. And then as you train, you gradually curate that, you know, you gradually make that higher and higher quality and more and more specific to the kinds of tasks you want it to do. But you never throw away any data….So yeah, that's now my view, is I think ULMFiT is the wrong approach. And that's why we're seeing a lot of these so-called alignment tax… I think it's actually because people are training them wrong.An example of this phenomena is CodeLlama, a LLaMA2 model finetuned on 500B tokens of code: while the model is much better at code, it's worse on generic tasks that LLaMA2 knew how to solve well before the fine-tuning. In the episode we also dive into all the places where open source model development and research is happening (academia vs Discords - tracked on our Communities list and on our survey), and how Jeremy recommends getting the most out of these diffuse, pseudonymous communities (similar to the Eleuther AI Mafia).Show Notes* Jeremy's Background* FastMail* Optimal Decisions* Kaggle* Enlitic* fast.ai* Rachel Thomas* Practical Deep Learning* fastai for PyTorch* nbdev* fastec2 (the underrated library we describe)* Can LLMs learn from a single example?* the Kaggle LLM Science Exam competition, which “challenges participants to answer difficult science-based questions written by a Large Language Model”.* Sebastian Ruder* Alec Radford* Sylvain Gugger* Stephen Merity* Chris Lattner* Modular.ai / Mojo* Jono Whittaker* Zeiler and Fergus paper* ULM Fit* DAWNBench* Phi-1* Code Llama* AlexNetTimestamps* [00:00:00] Intros and Jeremy's background* [00:05:28] Creating ULM Fit - a breakthrough in NLP using transfer learning* [00:06:32] The rise of GPT and the appeal of few-shot learning over fine-tuning* [00:10:00] Starting Fast.ai to distribute AI capabilities beyond elite academics* [00:14:30] How modern LMs like ChatGPT still follow the ULM Fit 3-step approach* [00:17:23] Meeting with Chris Lattner on Swift for TensorFlow at Google* [00:20:00] Continued pre-training as a fine-tuning alternative* [00:22:16] Fast.ai and looking for impact vs profit maximization* [00:26:39] Using Fast.ai to create an "army" of AI experts to improve their domains* [00:29:32] Fast.ai's 3 focus areas - research, software, and courses* [00:38:42] Fine-tuning memorization and training curve "clunks" before each epoch* [00:46:47] Poor training and fine-tuning practices may be causing alignment failures* [00:48:38] Academia vs Discords* [00:53:41] Jeremy's high hopes for Chris Lattner's Mojo and its potential* [01:05:00] Adding capabilities like SQL generation through quick fine-tuning* [01:10:12] Rethinking Fast.ai courses for the AI-assisted coding era* [01:14:53] Rapid model development has created major technical debt* [01:17:08] Lightning RoundAI Summary (beta)This is the first episode we're trying this. Here's an overview of the main topics before you dive in the transcript. * Jeremy's background and philosophies on AI* Studied philosophy and cognitive science in college* Focused on ethics and thinking about AI even 30 years ago* Believes AI should be accessible to more people, not just elite academics/programmers* Created fast.ai to make deep learning more accessible* Development of transfer learning and ULMFit* Idea of transfer learning critical for making deep learning accessible* ULMFit pioneered transfer learning for NLP* Proposed training general language models on large corpora then fine-tuning - this became standard practice* Faced skepticism that this approach would work from NLP community* Showed state-of-the-art results on text classification soon after trying it* Current open questions around fine-tuning LLMs* Models appear to memorize training data extremely quickly (after 1 epoch)* This may hurt training dynamics and cause catastrophic forgetting* Unclear how best to fine-tune models to incorporate new information/capabilities* Need more research on model training dynamics and ideal data mixing* Exciting new developments* Mojo and new programming languages like Swift could enable faster model innovation* Still lots of room for improvements in computer vision-like innovations in transformers* Small models with fine-tuning may be surprisingly capable for many real-world tasks* Prompting strategies enable models like GPT-3 to achieve new skills like playing chess at superhuman levels* LLMs are like computer vision in 2013 - on the cusp of huge new breakthroughs in capabilities* Access to AI research* Many key convos happen in private Discord channels and forums* Becoming part of these communities can provide great learning opportunities* Being willing to do real work, not just talk about ideas, is key to gaining access* The future of practical AI* Coding becoming more accessible to non-programmers through AI assistance* Pre-requisite programming experience for learning AI may no longer be needed* Huge open questions remain about how to best train, fine-tune, and prompt LLMsTranscriptAlessio: Hey everyone, welcome to the Latent Space Podcast. This is Alessio, partner and CTO at Residence at Decibel Partners, and I'm joined by my co-host Swyx, founder of Smol AI. [00:00:21]Swyx: Hey, and today we have in the remote studio, Jeremy Howard all the way from Australia. Good morning. [00:00:27]Jeremy: The remote studio, also known as my house. Good morning. Nice to see you. [00:00:32]Swyx: Nice to see you too. I'm actually very used to seeing you in your mask as a message to people, but today we're mostly audio. But thank you for doing the very important public service of COVID awareness. It was a pleasure. [00:00:46]Jeremy: It was all very annoying and frustrating and tedious, but somebody had to do it. [00:00:52]Swyx: Somebody had to do it, especially somebody with your profile. I think it really drives home the message. So we tend to introduce people for them and then ask people to fill in the blanks on the personal side. Something I did not know about you was that you graduated with a BA in philosophy from the University of Melbourne. I assumed you had a PhD. [00:01:14]Jeremy: No, I mean, I barely got through my BA because I was working 80 to 100 hour weeks at McKinsey and Company from 19 years old onwards. So I actually didn't attend any lectures in second and third year university. [00:01:35]Swyx: Well, I guess you didn't need it or you're very sort of self-driven and self-motivated. [00:01:39]Jeremy: I took two weeks off before each exam period when I was working at McKinsey. And then, I mean, I can't believe I got away with this in hindsight, I would go to all my professors and say, oh, I was meant to be in your class this semester and I didn't quite turn up. Were there any assignments I was meant to have done, whatever. I can't believe all of them let me basically have it. They basically always would say like, okay, well, if you can have this written by tomorrow, I'll accept it. So yeah, stressful way to get through university, but. [00:02:12]Swyx: Well, it shows that, I guess, you min-maxed the opportunities. That definitely was a precursor. [00:02:18]Jeremy: I mean, funnily, like in as much as I, you know, in philosophy, the things I found interesting and focused on in the little bit of time I did spend on it was ethics and cognitive science. And it's kind of really amazing that it's now come back around and those are actually genuinely useful things to know about, which I never thought would happen. [00:02:38]Swyx: A lot of, yeah, a lot of relevant conversations there. So you were a consultant for a while and then in the magical month of June 1989, you founded both Optimal Decisions and Fastmeal, which I also briefly used. So thank you for that. [00:02:53]Jeremy: Oh, good for you. Yeah. Cause I had read the statistics, which is that like 90% or something of small businesses fail. So I thought if I start two businesses, I have a higher chance. In hindsight, I was thinking of it as some kind of stochastic thing I didn't have control over, but it's a bit odd, but anyway. [00:03:10]Swyx: And then you were president and chief scientist at Kaggle, which obviously is the sort of composition platform of machine learning. And then Enlitic, where you were working on using deep learning to improve medical diagnostics and clinical decisions. Yeah. [00:03:28]Jeremy: I was actually the first company to use deep learning in medicine, so I kind of founded the field. [00:03:33]Swyx: And even now that's still like a pretty early phase. And I actually heard you on your new podcast with Tanish, where you went very, very deep into the stuff, the kind of work that he's doing, such a young prodigy at his age. [00:03:47]Jeremy: Maybe he's too old to be called a prodigy now, ex-prodigy. No, no. [00:03:51]Swyx: I think he still counts. And anyway, just to round out the bio, you have a lot more other credentials, obviously, but most recently you started Fast.ai, which is still, I guess, your primary identity with Rachel Thomas. So welcome. [00:04:05]Jeremy: Yep. [00:04:06]Swyx: Thanks to my wife. Thank you. Yeah. Doing a lot of public service there with getting people involved in AI, and I can't imagine a better way to describe it than fast, fast.ai. You teach people from nothing to stable diffusion in seven weeks or something, and that's amazing. Yeah, yeah. [00:04:22]Jeremy: I mean, it's funny, you know, when we started that, what was that, like 2016 or something, the idea that deep learning was something that you could make more accessible was generally considered stupid. Everybody knew that deep learning was a thing that you got a math or a computer science PhD, you know, there was one of five labs that could give you the appropriate skills and that you would join, yeah, basically from one of those labs, you might be able to write some papers. So yeah, the idea that normal people could use that technology to do good work was considered kind of ridiculous when we started it. And we weren't sure if it was possible either, but we kind of felt like we had to give it a go because the alternative was we were pretty sure that deep learning was on its way to becoming, you know, the most or one of the most, you know, important technologies in human history. And if the only people that could use it were a handful of computer science PhDs, that seemed like A, a big waste and B, kind of dangerous. [00:05:28]Swyx: Yeah. [00:05:29]Alessio: And, you know, well, I just wanted to know one thing on your bio that at Kaggle, you were also the top rank participant in both 2010 and 2011. So sometimes you see a lot of founders running companies that are not really in touch with the problem, but you were clearly building something that you knew a lot about, which is awesome. Talking about deep learning, you created, published a paper on ULM fit, which was kind of the predecessor to multitask learning and a lot of the groundwork that then went to into Transformers. I've read back on the paper and you turned this model, AWD LSTM, which I did the math and it was like 24 to 33 million parameters, depending on what training data set you use today. That's kind of like not even small, it's like super small. What were some of the kind of like contrarian takes that you had at the time and maybe set the stage a little bit for the rest of the audience on what was kind of like the state of the art, so to speak, at the time and what people were working towards? [00:06:32]Jeremy: Yeah, the whole thing was a contrarian take, you know. So okay, so we started Fast.ai, my wife and I, and we thought, yeah, so we're trying to think, okay, how do we make it more accessible? So when we started thinking about it, it was probably 2015 and then 2016, we started doing something about it. Why is it inaccessible? Okay, well, A, no one knows how to do it other than a few number of people. And then when we asked those few number of people, well, how do you actually get good results? They would say like, oh, it's like, you know, a box of tricks that aren't published. So you have to join one of the labs and learn the tricks. So a bunch of unpublished tricks, not much software around, but thankfully there was Theano and rappers and particularly Lasagna, the rapper, but yeah, not much software around, not much in the way of data sets, you know, very hard to get started in terms of the compute. Like how do you get that set up? So yeah, no, everything was kind of inaccessible. And you know, as we started looking into it, we had a key insight, which was like, you know what, most of the compute and data for image recognition, for example, we don't need to do it. You know, there's this thing which nobody knows about, nobody talks about called transfer learning, where you take somebody else's model, where they already figured out like how to detect edges and gradients and corners and text and whatever else, and then you can fine tune it to do the thing you want to do. And we thought that's the key. That's the key to becoming more accessible in terms of compute and data requirements. So when we started Fast.ai, we focused from day one on transfer learning. Lesson one, in fact, was transfer learning, literally lesson one, something not normally even mentioned in, I mean, there wasn't much in the way of courses, you know, the courses out there were PhD programs that had happened to have recorded their lessons and they would rarely mention it at all. We wanted to show how to do four things that seemed really useful. You know, work with vision, work with tables of data, work with kind of recommendation systems and collaborative filtering and work with text, because we felt like those four kind of modalities covered a lot of the stuff that, you know, are useful in real life. And no one was doing anything much useful with text. Everybody was talking about word2vec, you know, like king plus queen minus woman and blah, blah, blah. It was like cool experiments, but nobody's doing anything like useful with it. NLP was all like lemmatization and stop words and topic models and bigrams and SPMs. And it was really academic and not practical. But I mean, to be honest, I've been thinking about this crazy idea for nearly 30 years since I had done cognitive science at university, where we talked a lot about the CELS Chinese room experiment. This idea of like, what if there was somebody that could kind of like, knew all of the symbolic manipulations required to answer questions in Chinese, but they didn't speak Chinese and they were kind of inside a room with no other way to talk to the outside world other than taking in slips of paper with Chinese written on them and then they do all their rules and then they pass back a piece of paper with Chinese back. And this room with a person in is actually fantastically good at answering any question you give them written in Chinese. You know, do they understand Chinese? And is this, you know, something that's intelligently working with Chinese? Ever since that time, I'd say the most thought, to me, the most thoughtful and compelling philosophical response is yes. You know, intuitively it feels like no, because that's just because we can't imagine such a large kind of system. But you know, if it looks like a duck and acts like a duck, it's a duck, you know, or to all intents and purposes. And so I always kind of thought, you know, so this is basically a kind of analysis of the limits of text. And I kind of felt like, yeah, if something could ingest enough text and could use the patterns it saw to then generate text in response to text, it could appear to be intelligent, you know. And whether that means it is intelligent or not is a different discussion and not one I find very interesting. Yeah. And then when I came across neural nets when I was about 20, you know, what I learned about the universal approximation theorem and stuff, and I started thinking like, oh, I wonder if like a neural net could ever get big enough and take in enough data to be a Chinese room experiment. You know, with that background and this kind of like interest in transfer learning, you know, I'd been thinking about this thing for kind of 30 years and I thought like, oh, I wonder if we're there yet, you know, because we have a lot of text. Like I can literally download Wikipedia, which is a lot of text. And I thought, you know, how would something learn to kind of answer questions or, you know, respond to text? And I thought, well, what if we used a language model? So language models are already a thing, you know, they were not a popular or well-known thing, but they were a thing. But language models exist to this idea that you could train a model to fill in the gaps. Or actually in those days it wasn't fill in the gaps, it was finish a string. And in fact, Andrej Karpathy did his fantastic RNN demonstration from this at a similar time where he showed like you can have it ingest Shakespeare and it will generate something that looks a bit like Shakespeare. I thought, okay, so if I do this at a much bigger scale, using all of Wikipedia, what would it need to be able to do to finish a sentence in Wikipedia effectively, to do it quite accurately quite often? I thought, geez, it would actually have to know a lot about the world, you know, it'd have to know that there is a world and that there are objects and that objects relate to each other through time and cause each other to react in ways and that causes proceed effects and that, you know, when there are animals and there are people and that people can be in certain positions during certain timeframes and then you could, you know, all that together, you can then finish a sentence like this was signed into law in 2016 by US President X and it would fill in the gap, you know. So that's why I tried to create what in those days was considered a big language model trained on the entirety on Wikipedia, which is that was, you know, a bit unheard of. And my interest was not in, you know, just having a language model. My interest was in like, what latent capabilities would such a system have that would allow it to finish those kind of sentences? Because I was pretty sure, based on our work with transfer learning and vision, that I could then suck out those latent capabilities by transfer learning, you know, by fine-tuning it on a task data set or whatever. So we generated this three-step system. So step one was train a language model on a big corpus. Step two was fine-tune a language model on a more curated corpus. And step three was further fine-tune that model on a task. And of course, that's what everybody still does today, right? That's what ChatGPT is. And so the first time I tried it within hours, I had a new state-of-the-art academic result on IMDB. And I was like, holy s**t, it does work. And so you asked, to what degree was this kind of like pushing against the established wisdom? You know, every way. Like the reason it took me so long to try it was because I asked all my friends in NLP if this could work. And everybody said, no, it definitely won't work. It wasn't like, oh, maybe. Everybody was like, it definitely won't work. NLP is much more complicated than vision. Language is a much more vastly complicated domain. You know, and you've got problems like the grounding problem. We know from like philosophy and theory of mind that it's actually impossible for it to work. So yeah, so don't waste your time. [00:15:10]Alessio: Jeremy, had people not tried because it was like too complicated to actually get the data and like set up the training? Or like, were people just lazy and kind of like, hey, this is just not going to work? [00:15:20]Jeremy: No, everybody wasn't lazy. So like, so the person I thought at that time who, you know, there were two people I thought at that time, actually, who were the strongest at language models were Stephen Merity and Alec Radford. And at the time I didn't know Alec, but I, after we had both, after I'd released ULM Fit and he had released GPT, I organized a chat for both of us with Kate Metz in the New York Times. And Kate Metz answered, sorry, and Alec answered this question for Kate. And Kate was like, so how did, you know, GPT come about? And he said, well, I was pretty sure that pre-training on a general large corpus wouldn't work. So I hadn't tried it. And then I read ULM Fit and turns out it did work. And so I did it, you know, bigger and it worked even better. And similar with, with Stephen, you know, I asked Stephen Merity, like, why don't we just find, you know, take your AWD-ASTLM and like train it on all of Wikipedia and fine tune it? And he's kind of like, well, I don't think that's going to really lie. Like two years before I did a very popular talk at KDD, the conference where everybody in NLP was in the audience. I recognized half the faces, you know, and I told them all this, I'm sure transfer learning is the key. I'm sure ImageNet, you know, is going to be an NLP thing as well. And, you know, everybody was interested and people asked me questions afterwards and, but not just, yeah, nobody followed up because everybody knew that it didn't work. I mean, even like, so we were scooped a little bit by Dai and Lee, Kwok Lee at Google. They had, they had, I already, I didn't even realize this, which is a bit embarrassing. They had already done a large language model and fine tuned it. But again, they didn't create a general purpose, large language model on a general purpose corpus. They only ever tested a domain specific corpus. And I haven't spoken to Kwok actually about that, but I assume that the reason was the same. It probably just didn't occur to them that the general approach could work. So maybe it was that kind of 30 years of mulling over the, the cell Chinese room experiment that had convinced me that it probably would work. I don't know. Yeah. [00:17:48]Alessio: Interesting. I just dug up Alec announcement tweet from 2018. He said, inspired by Cobe, Elmo, and Yola, I'm fit. We should have a single transformer language model can be fine tuned to a wide variety. It's interesting because, you know, today people think of AI as the leader, kind of kind of like the research lab pushing forward the field. What was that at the time? You know, like kind of like going back five years, people think of it as an overnight success, but obviously it took a while. [00:18:16]Swyx: Yeah. Yeah. [00:18:17]Jeremy: No, I mean, absolutely. And I'll say like, you know, it's interesting that it mentioned Elmo because in some ways that was kind of diametrically opposed to, to ULM fit. You know, there was these kind of like, so there was a lot of, there was a lot of activity at the same time as ULM fits released. So there was, um, so before it, as Brian McCann, I think at Salesforce had come out with this neat model that did a kind of multitask learning, but again, they didn't create a general fine tune language model first. There was Elmo, um, which I think was a lip, you know, actually quite a few months after the first ULM fit example, I think. Um, but yeah, there was a bit of this stuff going on. And the problem was everybody was doing, and particularly after GPT came out, then everybody wanted to focus on zero shot and few shot learning. You know, everybody hated fine tuning. Everybody hated transfer learning. And like, I literally did tours trying to get people to start doing transfer learning and people, you know, nobody was interested, particularly after GPT showed such good results with zero shot and few shot learning. And so I actually feel like we kind of went backwards for years and, and not to be honest, I mean, I'm a bit sad about this now, but I kind of got so disappointed and dissuaded by like, it felt like these bigger lab, much bigger labs, you know, like fast AI had only ever been just me and Rachel were getting all of this attention for an approach I thought was the wrong way to do it. You know, I was convinced was the wrong way to do it. And so, yeah, for years people were really focused on getting better at zero shot and few shots and it wasn't until, you know, this key idea of like, well, let's take the ULM fit approach, but for step two, rather than fine tuning on a kind of a domain corpus, let's fine tune on an instruction corpus. And then in step three, rather than fine tuning on a reasonably specific task classification, let's fine tune on a, on a RLHF task classification. And so that was really, that was really key, you know, so I was kind of like out of the NLP field for a few years there because yeah, it just felt like, I don't know, pushing uphill against this vast tide, which I was convinced was not the right direction, but who's going to listen to me, you know, cause I, as you said, I don't have a PhD, not at a university, or at least I wasn't then. I don't have a big set of computers to fine tune huge transformer models. So yeah, it was definitely difficult. It's always been hard. You know, it's always been hard. Like I've always been somebody who does not want to build stuff on lots of big computers because most people don't have lots of big computers and I hate creating stuff that most people can't use, you know, and also stuff that's created on lots of big computers has always been like much more media friendly. So like, it might seem like a recent thing, but actually throughout my 30 years in data science, the attention's always been on, you know, the big iron results. So when I first started, everybody was talking about data warehouses and it was all about Teradata and it'd be like, oh, this big bank has this huge room full of computers and they have like terabytes of data available, you know, at the press of a button. And yeah, that's always what people want to talk about, what people want to write about. And then of course, students coming out of their PhDs and stuff, that's where they want to go work because that's where they read about. And to me, it's a huge distraction, you know, because like I say, most people don't have unlimited compute and I want to help most people, not the small subset of the most well-off people. [00:22:16]Alessio: That's awesome. And it's great to hear, you do such a great job educating that a lot of times you're not telling your own story, you know? So I love this conversation. And the other thing before we jump into Fast.AI, actually, a lot of people that I know, they run across a new architecture and whatnot, they're like, I got to start a company and raise a bunch of money and do all of this stuff. And say, you were like, I want everybody to have access to this. Why was that the case for you? Was it because you already had a successful venture in like FastMail and you were more interested in that? What was the reasoning? [00:22:52]Jeremy: It's a really good question. So I guess the answer is yes, that's the reason why. So when I was a teenager, I thought it would be really cool to like have my own company. You know, I didn't know the word startup. I didn't know the word entrepreneur. I didn't know the word VC. And I didn't really know what any of those things were really until after we started Kaggle, to be honest. Even the way it started to what we now call startups. I just thought they were just small businesses. You know, they were just companies. So yeah, so those two companies were FastMail and Optimal Decisions. FastMail was the first kind of synchronized email provider for non-businesses. So something you can get your same email at home, on your laptop, at work, on your phone, whatever. And then Optimal Decisions invented a new approach to insurance pricing. Something called profit-optimized insurance pricing. So I saw both of those companies, you know, after 10 years. And at that point, I had achieved the thing that as a teenager I had wanted to do. You know, it took a lot longer than it should have because I spent way longer in management consulting than I should have because I got caught up in that stupid rat race. But, you know, eventually I got there and I remember my mom saying to me, you must be so proud. You know, because she remembered my dream. She's like, you've done it. And I kind of reflected and I was like, I'm not proud at all. You know, like people quite liked FastMail. You know, it's quite nice to have synchronized email. It probably would have happened anyway. Yeah, I'm certainly not proud that I've helped some insurance companies suck more money out of their customers. Yeah, no, I'm not proud. You know, it's actually, I haven't really helped the world very much. You know, maybe in the insurance case I've made it a little bit worse. I don't know. So, yeah, I was determined to not waste more years of my life doing things, working hard to do things which I could not be reasonably sure would have a lot of value. So, you know, I took some time off. I wasn't sure if I'd ever work again, actually. I didn't particularly want to, because it felt like, yeah, it felt like such a disappointment. And, but, you know, and I didn't need to. I had enough money. Like, I wasn't super rich, but I had enough money. I didn't need to work. And I certainly recognized that amongst the other people I knew who had enough money that they didn't need to work, they all worked ridiculously hard, you know, and constantly put themselves in extremely stressful situations. And I thought, I don't want to be one of those idiots who's tied to, you know, buying a bigger plane than the next guy or whatever. You know, Kaggle came along and I mainly kind of did that just because it was fun and interesting to hang out with interesting people. But, you know, with Fast.ai in particular, you know, Rachel and I had a very explicit, you know, long series of conversations over a long period of time about like, well, how can we be the most helpful to society as a whole, and particularly to those people who maybe need more help, you know? And so we definitely saw the world going in a potentially pretty dystopian direction if the world's most powerful technology was controlled by a small group of elites. So we thought, yeah, we should focus on trying to help that not happen. You know, sadly, it looks like it still is likely to happen. But I mean, I feel like we've helped make it a little bit less likely. So we've done our bit. [00:26:39]Swyx: You've shown that it's possible. And I think your constant advocacy, your courses, your research that you publish, you know, just the other day you published a finding on, you know, learning that I think is still something that people are still talking about quite a lot. I think that that is the origin story of a lot of people who are going to be, you know, little Jeremy Howards, furthering your mission with, you know, you don't have to do everything by yourself is what I'm saying. No, definitely. Definitely. [00:27:10]Jeremy: You know, that was a big takeaway from like, analytic was analytic. It definitely felt like we had to do everything ourselves. And I kind of, I wanted to solve medicine. I'll say, yeah, okay, solving medicine is actually quite difficult. And I can't do it on my own. And there's a lot of other things I'd like to solve, and I can't do those either. So that was definitely the other piece was like, yeah, you know, can we create an army of passionate domain experts who can change their little part of the world? And that's definitely happened. Like I find nowadays, at least half the time, probably quite a bit more that I get in contact with somebody who's done really interesting work in some domain. Most of the time I'd say, they say, yeah, I got my start with fast.ai. So it's definitely, I can see that. And I also know from talking to folks at places like Amazon and Adobe and stuff, which, you know, there's lots of alumni there. And they say, oh my God, I got here. And like half of the people are fast.ai alumni. So it's fantastic. [00:28:13]Swyx: Yeah. [00:28:14]Jeremy: Actually, Andre Kapathy grabbed me when I saw him at NeurIPS a few years ago. And he was like, I have to tell you, thanks for the fast.ai courses. When people come to Tesla and they need to know more about deep learning, we always send them to your course. And the OpenAI Scholars Program was doing the same thing. So it's kind of like, yeah, it's had a surprising impact, you know, that's just one of like three things we do is the course, you know. [00:28:40]Swyx: Yes. [00:28:40]Jeremy: And it's only ever been at most two people, either me and Rachel or me and Sylvia nowadays, it's just me. So yeah, I think it shows you don't necessarily need a huge amount of money and a huge team of people to make an impact. [00:28:56]Swyx: Yeah. So just to reintroduce fast.ai for people who may not have dived into it much, there is the courses that you do. There is the library that is very well loved. And I kind of think of it as a nicer layer on top of PyTorch that people should start with by default and use it as the basis for a lot of your courses. And then you have like NBDev, which I don't know, is that the third one? [00:29:27]Jeremy: Oh, so the three areas were research, software, and courses. [00:29:32]Swyx: Oh, sorry. [00:29:32]Jeremy: So then in software, you know, fast.ai is the main thing, but NBDev is not far behind. But then there's also things like FastCore, GHAPI, I mean, dozens of open source projects that I've created and some of them have been pretty popular and some of them are still a little bit hidden, actually. Some of them I should try to do a better job of telling people about. [00:30:01]Swyx: What are you thinking about? Yeah, what's on the course of my way? Oh, I don't know, just like little things. [00:30:04]Jeremy: Like, for example, for working with EC2 and AWS, I created a FastEC2 library, which I think is like way more convenient and nice to use than anything else out there. And it's literally got a whole autocomplete, dynamic autocomplete that works both on the command line and in notebooks that'll like auto-complete your instance names and everything like that. You know, just little things like that. I try to make like, when I work with some domain, I try to make it like, I want to make it as enjoyable as possible for me to do that. So I always try to kind of like, like with GHAPI, for example, I think that GitHub API is incredibly powerful, but I didn't find it good to work with because I didn't particularly like the libraries that are out there. So like GHAPI, like FastEC2, it like autocompletes both at the command line or in a notebook or whatever, like literally the entire GitHub API. The entire thing is like, I think it's like less than 100K of code because it actually, as far as I know, the only one that grabs it directly from the official open API spec that GitHub produces. And like if you're in GitHub and you just type an API, you know, autocomplete API method and hit enter, it prints out the docs with brief docs and then gives you a link to the actual documentation page. You know, GitHub Actions, I can write now in Python, which is just so much easier than writing them in TypeScript and stuff. So, you know, just little things like that. [00:31:40]Swyx: I think that's an approach which more developers took to publish some of their work along the way. You described the third arm of FastAI as research. It's not something I see often. Obviously, you do do some research. And how do you run your research? What are your research interests? [00:31:59]Jeremy: Yeah, so research is what I spend the vast majority of my time on. And the artifacts that come out of that are largely software and courses. You know, so to me, the main artifact shouldn't be papers because papers are things read by a small exclusive group of people. You know, to me, the main artifacts should be like something teaching people, here's how to use this insight and here's software you can use that builds it in. So I think I've only ever done three first-person papers in my life, you know, and none of those are ones I wanted to do. You know, they were all ones that, like, so one was ULM Fit, where Sebastian Ruder reached out to me after seeing the course and said, like, you have to publish this as a paper, you know. And he said, I'll write it. He said, I want to write it because if I do, I can put it on my PhD and that would be great. And it's like, okay, well, I want to help you with your PhD. And that sounds great. So like, you know, one was the masks paper, which just had to exist and nobody else was writing it. And then the third was the Fast.ai library paper, which again, somebody reached out and said, please, please write this. We will waive the fee for the journal and everything and actually help you get it through publishing and stuff. So yeah, so I don't, other than that, I've never written a first author paper. So the research is like, well, so for example, you know, Dawn Bench was a competition, which Stanford ran a few years ago. It was kind of the first big competition of like, who can train neural nets the fastest rather than the most accurate. And specifically it was who can train ImageNet the fastest. And again, this was like one of these things where it was created by necessity. So Google had just released their TPUs. And so I heard from my friends at Google that they had put together this big team to smash Dawn Bench so that they could prove to people that they had to use Google Cloud and use their TPUs and show how good their TPUs were. And we kind of thought, oh s**t, this would be a disaster if they do that, because then everybody's going to be like, oh, deep learning is not accessible. [00:34:20]Swyx: You know, to actually be good at it, [00:34:21]Jeremy: you have to be Google and you have to use special silicon. And so, you know, we only found out about this 10 days before the competition finished. But, you know, we basically got together an emergency bunch of our students and Rachel and I and sat for the next 10 days and just tried to crunch through and try to use all of our best ideas that had come from our research. And so particularly progressive resizing, just basically train mainly on small things, train on non-square things, you know, stuff like that. And so, yeah, we ended up winning, thank God. And so, you know, we turned it around from being like, like, oh s**t, you know, this is going to show that you have to be Google and have TPUs to being like, oh my God, even the little guy can do deep learning. So that's an example of the kind of like research artifacts we do. And yeah, so all of my research is always, how do we do more with less, you know? So how do we get better results with less data, with less compute, with less complexity, with less education, you know, stuff like that. So ULM fits obviously a good example of that. [00:35:37]Swyx: And most recently you published, can LLMs learn from a single example? Maybe could you tell the story a little bit behind that? And maybe that goes a little bit too far into the learning of very low resource, the literature. [00:35:52]Jeremy: Yeah, yeah. So me and my friend, Jono Whittaker, basically had been playing around with this fun Kaggle competition, which is actually still running as we speak, which is, can you create a model which can answer multiple choice questions about anything that's in Wikipedia? And the thing that makes it interesting is that your model has to run on Kaggle within nine hours. And Kaggle's very, very limited. So you've only got 14 gig RAM, only two CPUs, and a small, very old GPU. So this is cool, you know, if you can do well at this, then this is a good example of like, oh, you can do more with less. So yeah, Jono and I were playing around with fine tuning, of course, transfer learning, pre-trained language models. And we saw this, like, so we always, you know, plot our losses as we go. So here's another thing we created. Actually, Sylvain Guuger, when he worked with us, created called fast progress, which is kind of like TQEDM, but we think a lot better. So we look at our fast progress curves, and they kind of go down, down, down, down, down, down, down, a little bit, little bit, little bit. And then suddenly go clunk, and they drop. And then down, down, down, down, down a little bit, and then suddenly clunk, they drop. We're like, what the hell? These clunks are occurring at the end of each epoch. So normally in deep learning, this would be, this is, you know, I've seen this before. It's always been a bug. It's always turned out that like, oh, we accidentally forgot to turn on eval mode during the validation set. So I was actually learning then, or, oh, we accidentally were calculating moving average statistics throughout the epoch. So, you know, so it's recently moving average or whatever. And so we were using Hugging Face Trainer. So, you know, I did not give my friends at Hugging Face the benefit of the doubt. I thought, oh, they've fucked up Hugging Face Trainer, you know, idiots. Well, you'll use the Fast AI Trainer instead. So we switched over to Learner. We still saw the clunks and, you know, that's, yeah, it shouldn't really happen because semantically speaking in the epoch, isn't like, it's not a thing, you know, like nothing happens. Well, nothing's meant to happen when you go from ending one epoch to starting the next one. So there shouldn't be a clunk, you know. So I kind of asked around on the open source discords. That's like, what's going on here? And everybody was just like, oh, that's just what, that's just what these training curves look like. Those all look like that. Don't worry about it. And I was like, oh, are you all using Trainer? Yes. Oh, well, there must be some bug with Trainer. And I was like, well, we also saw it in Learner [00:38:42]Swyx: and somebody else is like, [00:38:42]Jeremy: no, we've got our own Trainer. We get it as well. They're just like, don't worry about it. It's just something we see. It's just normal. [00:38:48]Swyx: I can't do that. [00:38:49]Jeremy: I can't just be like, here's something that's like in the previous 30 years of neural networks, nobody ever saw it. And now suddenly we see it. [00:38:57]Swyx: So don't worry about it. [00:38:59]Jeremy: I just, I have to know why. [00:39:01]Swyx: Can I clarify? This is, was everyone that you're talking to, were they all seeing it for the same dataset or in different datasets? [00:39:08]Jeremy: Different datasets, different Trainers. They're just like, no, this is just, this is just what it looks like when you fine tune language models. Don't worry about it. You know, I hadn't seen it before, but I'd been kind of like, as I say, I, you know, I kept working on them for a couple of years after ULM fit. And then I kind of moved on to other things, partly out of frustration. So I hadn't been fine tuning, you know, I mean, Lama's only been out for a few months, right? But I wasn't one of those people who jumped straight into it, you know? So I was relatively new to the kind of Lama fine tuning world, where else these guys had been, you know, doing it since day one. [00:39:49]Swyx: It was only a few months ago, [00:39:51]Jeremy: but it's still quite a bit of time. So, so yeah, they're just like, no, this is all what we see. [00:39:56]Swyx: Don't worry about it. [00:39:56]Jeremy: So yeah, I, I've got a very kind of like, I don't know, I've just got this brain where I have to know why things are. And so I kind of, I ask people like, well, why, why do you think it's happening? And they'd be like, oh, it would pretty obviously, cause it's like memorize the data set. It's just like, that can't be right. It's only seen it once. Like, look at this, the loss has dropped by 0.3, 0.3, which is like, basically it knows the answer. And like, no, no, it's just, it is, it's just memorize the data set. So yeah. So look, Jono and I did not discover this and Jono and I did not come up with a hypothesis. You know, I guess we were just the ones, I guess, who had been around for long enough to recognize that like, this, this isn't how it's meant to work. And so we, we, you know, and so we went back and like, okay, let's just run some experiments, you know, cause nobody seems to have actually published anything about this. [00:40:51]Well, not quite true.Some people had published things, but nobody ever actually stepped back and said like, what the hell, you know, how can this be possible? Is it possible? Is this what's happening? And so, yeah, we created a bunch of experiments where we basically predicted ahead of time. It's like, okay, if this hypothesis is correct, that it's memorized in the training set, then we ought to see blah, under conditions, blah, but not under these conditions. And so we ran a bunch of experiments and all of them supported the hypothesis that it was memorizing the data set in a single thing at once. And it's a pretty big data set, you know, which in hindsight, it's not totally surprising because the theory, remember, of the ULMFiT theory was like, well, it's kind of creating all these latent capabilities to make it easier for it to predict the next token. So if it's got all this kind of latent capability, it ought to also be really good at compressing new tokens because it can immediately recognize it as like, oh, that's just a version of this. So it's not so crazy, you know, but it is, it requires us to rethink everything because like, and nobody knows like, okay, so how do we fine tune these things? Because like, it doesn't even matter. Like maybe it's fine. Like maybe it's fine that it's memorized the data set after one go and you do a second go and okay, the validation loss is terrible because it's now really overconfident. [00:42:20]Swyx: That's fine. [00:42:22]Jeremy: Don't, you know, don't, I keep telling people, don't track validation loss, track validation accuracy because at least that will still be useful. Just another thing that's got lost since ULMFiT, nobody tracks accuracy of language models anymore. But you know, it'll still keep learning and it does, it does keep improving. But is it worse? You know, like, is it like, now that it's kind of memorized it, it's probably getting a less strong signal, you know, I don't know. So I still don't know how to fine tune language models properly and I haven't found anybody who feels like they do, like nobody really knows whether this memorization thing is, it's probably a feature in some ways. It's probably some things that you can do usefully with it. It's probably, yeah, I have a feeling it's messing up training dynamics as well. [00:43:13]Swyx: And does it come at the cost of catastrophic forgetting as well, right? Like, which is the other side of the coin. [00:43:18]Jeremy: It does to some extent, like we know it does, like look at Code Llama, for example. So Code Llama was a, I think it was like a 500 billion token fine tuning of Llama 2 using code. And also pros about code that Meta did. And honestly, they kind of blew it because Code Llama is good at coding, but it's bad at everything else, you know, and it used to be good. Yeah, I was pretty sure it was like, before they released it, me and lots of people in the open source discords were like, oh my God, you know, we know this is coming, Jan Lukinsk saying it's coming. I hope they kept at least like 50% non-code data because otherwise it's going to forget everything else. And they didn't, only like 0.3% of their epochs were non-code data. So it did, it forgot everything else. So now it's good at code and it's bad at everything else. So we definitely have catastrophic forgetting. It's fixable, just somebody has to do, you know, somebody has to spend their time training a model on a good mix of data. Like, so, okay, so here's the thing. Even though I originally created three-step approach that everybody now does, my view is it's actually wrong and we shouldn't use it. [00:44:36]Jeremy: And that's because people are using it in a way different to why I created it. You know, I created it thinking the task-specific models would be more specific. You know, it's like, oh, this is like a sentiment classifier as an example of a task, you know, but the tasks now are like a, you know, RLHF, which is basically like answer questions that make people feel happy about your answer. So that's a much more general task and it's a really cool approach. And so we see, for example, RLHF also breaks models like, you know, like GPT-4, RLHDEFT, we know from kind of the work that Microsoft did, you know, the pre, the earlier, less aligned version was better. And these are all kind of examples of catastrophic forgetting. And so to me, the right way to do this is to fine-tune language models, is to actually throw away the idea of fine-tuning. There's no such thing. There's only continued pre-training. And pre-training is something where from the very start, you try to include all the kinds of data that you care about, all the kinds of problems that you care about, instructions, exercises, code, general purpose document completion, whatever. And then as you train, you gradually curate that, you know, you gradually make that higher and higher quality and more and more specific to the kinds of tasks you want it to do. But you never throw away any data. You always keep all of the data types there in reasonably high quantities. You know, maybe the quality filter, you stop training on low quality data, because that's probably fine to forget how to write badly, maybe. So yeah, that's now my view, is I think ULM fit is the wrong approach. And that's why we're seeing a lot of these, you know, so-called alignment tacks and this view of like, oh, a model can't both code and do other things. And, you know, I think it's actually because people are training them wrong. [00:46:47]Swyx: Yeah, well, I think you have a clear [00:46:51]Alessio: anti-laziness approach. I think other people are not as good hearted, you know, they're like, [00:46:57]Swyx: hey, they told me this thing works. [00:46:59]Alessio: And if I release a model this way, people will appreciate it, I'll get promoted and I'll kind of make more money. [00:47:06]Jeremy: Yeah, and it's not just money. It's like, this is how citations work most badly, you know, so if you want to get cited, you need to write a paper that people in your field recognize as an advancement on things that we know are good. And so we've seen this happen again and again. So like I say, like zero shot and few shot learning, everybody was writing about that. Or, you know, with image generation, everybody just was writing about GANs, you know, and I was trying to say like, no, GANs are not the right approach. You know, and I showed again through research that we demonstrated in our videos that you can do better than GANs, much faster and with much less data. And nobody cared because again, like if you want to get published, you write a GAN paper that slightly improves this part of GANs and this tiny field, you'll get published, you know. So it's, yeah, it's not set up for real innovation. It's, you know, again, it's really helpful for me, you know, I have my own research lab with nobody telling me what to do and I don't even publish. So it doesn't matter if I get citations. And so I just write what I think actually matters. I wish there was, and, you know, and actually places like OpenAI, you know, the researchers there can do that as well. It's a shame, you know, I wish there was more academic, open venues in which people can focus on like genuine innovation. [00:48:38]Swyx: Twitter, which is unironically has become a little bit of that forum. I wanted to follow up on one thing that you mentioned, which is that you checked around the open source discords. I don't know if it's too, I don't know if it's a pusher to ask like what discords are lively or useful right now. I think that something I definitely felt like I missed out on was the early days of Luther AI, which is a very hard bit. And, you know, like what is the new Luther? And you actually shouted out the alignment lab AI discord in your blog post. And that was the first time I even knew, like I saw them on Twitter, never knew they had a discord, never knew that there was actually substantive discussions going on in there and that you were an active member of it. Okay, yeah. [00:49:23]Jeremy: And then even then, if you do know about that and you go there, it'll look like it's totally dead. And that's because unfortunately, nearly all the discords, nearly all of the conversation happens in private channels. You know, and that's, I guess. [00:49:35]Swyx: How does someone get into that world? Because it's obviously very, very instructive, right? [00:49:42]Jeremy: You could just come to the first AI discord, which I'll be honest with you, it's less bustling than some of the others, but it's not terrible. And so like, at least, to be fair, one of Emma's bustling channels is private. [00:49:57]Swyx: I guess. [00:49:59]Jeremy: So I'm just thinking. [00:50:01]Swyx: It's just the nature of quality discussion, right? Yeah, I guess when I think about it, [00:50:05]Jeremy: I didn't have any private discussions on our discord for years, but there was a lot of people who came in with like, oh, I just had this amazing idea for AGI. If you just thought about like, if you imagine that AI is a brain, then we, you know, this just, I don't want to talk about it. You know, I don't want to like, you don't want to be dismissive or whatever. And it's like, oh, well, that's an interesting comment, but maybe you should like, try training some models first to see if that aligns with your intuition. Like, oh, but how could I possibly learn? It's like, well, we have a course, just actually spend time learning. Like, you know, anyway. And there's like, okay, I know the people who always have good answers there. And so I created a private channel and put them all in it. And I got to admit, that's where I post more often because there's much less, you know, flight of fancy views about how we could solve AGI, blah, blah, blah. So there is a bit of that. But having said that, like, I think the bar is pretty low. Like if you join a Discord and you can hit the like participants or community or whatever button, you can see who's in it. And then you'll see at the top, who the admins or moderators or people in the dev role are. And just DM one of them and say like, oh, here's my GitHub. Well, here's some blog posts I wrote. You know, I'm interested in talking about this, you know, can I join the private channels? And I've never heard of anybody saying no. I will say, you know, Alutha's all pretty open. So you can do the Alutha Discord still. You know, one problem with the Alutha Discord is it's been going on for so long that it's like, it's very inside baseball. It's quite hard to get started. Yeah. Carpa AI looks, I think it's all open. That's just less stability. That's more accessible. [00:52:03]Swyx: Yeah. [00:52:04]Jeremy: There's also just recently, now it's research that does like the Hermes models and data set just opened. They've got some private channels, but it's pretty open, I think. You mentioned Alignment Lab, that one it's all the interesting stuff is on private channels. So just ask. If you know me, ask me, cause I've got admin on that one. There's also, yeah, OS Skunkworks, OS Skunkworks AI is a good Discord, which I think it's open. So yeah, they're all pretty good. [00:52:40]Swyx: I don't want you to leak any, you know, Discords that don't want any publicity, but this is all helpful. [00:52:46]Jeremy: We all want people, like we all want people. [00:52:49]Swyx: We just want people who like, [00:52:51]Jeremy: want to build stuff, rather than people who, and like, it's fine to not know anything as well, but if you don't know anything, but you want to tell everybody else what to do and how to do it, that's annoying. If you don't know anything and want to be told like, here's a really small kind of task that as somebody who doesn't know anything is going to take you a really long time to do, but it would still be helpful. Then, and then you go and do it. That would be great. The truth is, yeah, [00:53:19]Swyx: like, I don't know, [00:53:20]Jeremy: maybe 5% of people who come in with great enthusiasm and saying that they want to learn and they'll do anything. [00:53:25]Swyx: And then somebody says like, [00:53:25]Jeremy: okay, here's some work you can do. Almost nobody does that work. So if you're somebody who actually does the work and follows up, you will massively stand out. That's an extreme rarity. And everybody will then want to help you do more work. [00:53:41]Swyx: So yeah. [00:53:41]Jeremy: So just, yeah, just do work and people will want to support you. [00:53:47]Alessio: Our Discord used to be referral only for a long time. We didn't have a public invite and then we opened it and they're kind of like channel gating. Yeah. A lot of people just want to do, I remember it used to be like, you know, a forum moderator. [00:54:00]Swyx: It's like people just want to do [00:54:01]Alessio: like drive-by posting, [00:54:03]Swyx: you know, and like, [00:54:03]Alessio: they don't want to help the community. They just want to get their question answered. [00:54:07]Jeremy: I mean, the funny thing is our forum community does not have any of that garbage. You know, there's something specific about the low latency thing where people like expect an instant answer. And yeah, we're all somehow in a forum thread where they know it's like there forever. People are a bit more thoughtful, but then the forums are less active than they used to be because Discord has got more popular, you know? So it's all a bit of a compromise, you know, running a healthy community is, yeah, it's always a bit of a challenge. All right, we got so many more things [00:54:47]Alessio: we want to dive in, but I don't want to keep you here for hours. [00:54:50]Swyx: This is not the Lex Friedman podcast [00:54:52]Alessio: we always like to say. One topic I would love to maybe chat a bit about is Mojo, modular, you know, CrystalLiner, not many of you on the podcast. So we want to spend a little time there. You recently did a hacker's guide to language models and you ran through everything from quantized model to like smaller models, larger models, and all of that. But obviously modular is taking its own approach. Yeah, what got you excited? I know you and Chris have been talking about this for like years and a lot of the ideas you had, so. [00:55:23]Jeremy: Yeah, yeah, yeah, yeah, no, absolutely. So I met Chris, I think it was at the first TensorFlow Dev Summit. And I don't think he had even like, I'm not sure if he'd even officially started his employment with Google at that point. So I don't know, you know, certainly nothing had been mentioned. So I, you know, I admired him from afar with LLVM and Swift and whatever. And so I saw him walk into the courtyard at Google. It's just like, oh s**t, man, that's Chris Latner. I wonder if he would lower his standards enough to talk to me. Well, worth a try. So I caught up my courage because like nobody was talking to him. He looked a bit lost and I wandered over and it's like, oh, you're Chris Latner, right? It's like, what are you doing here? What are you doing here? And I was like, yeah, yeah, yeah. It's like, oh, I'm Jeremy Howard. It's like, oh, do you do some of this AI stuff? And I was like, yeah, yeah, I like this AI stuff. Are you doing AI stuff? It's like, well, I'm thinking about starting to do some AI stuff. Yeah, I think it's going to be cool. And it's like, wow. So like, I spent the next half hour just basically brain dumping all the ways in which AI was stupid to him. And he listened patiently. And I thought he probably wasn't even remember or care or whatever. But yeah, then I kind of like, I guess I re-caught up with him a few months later. And it's like, I've been thinking about everything you said in that conversation. And he like narrated back his response to every part of it, projects he was planning to do. And it's just like, oh, this dude follows up. Holy s**t. And I was like, wow, okay. And he was like, yeah, so we're going to create this new thing called Swift for TensorFlow. And it's going to be like, it's going to be a compiler with auto differentiation built in. And blah, blah, blah. And I was like, why would that help? [00:57:10]Swyx: You know, why would you? [00:57:10]Jeremy: And he was like, okay, with a compiler during the forward pass, you don't have to worry about saving context, you know, because a lot will be optimized in the backward. But I was like, oh my God. Because I didn't really know much about compilers. You know, I spent enough to kind of like, understand the ideas, but it hadn't occurred to me that a compiler basically solves a lot of the problems we have as end users. I was like, wow, that's amazing. Okay, you do know, right, that nobody's going to use this unless it's like usable. It's like, yeah, I know, right. So I was thinking you should create like a fast AI for this. So, okay, but I don't even know Swift. And he was like, well, why don't you start learning it? And if you have any questions, ask me. It's just like, holy s**t. Like, not only has Chris Latner lowered his standards enough to talk to me, but he's offering me personal tutoring on the programming language that he made. So I was just like, I'm not g

In this episode, we dive into Meta's latest innovation, Code Llama, an open-source AI model designed to generate code. Join us as we explore how this technology could revolutionize software development and its potential impact on the coding community. Learn more about the exciting possibilities that Code Llama brings to the world of programming. Get on the AI Box Waitlist: https://AIBox.ai/Join our ChatGPT Community: ⁠https://www.facebook.com/groups/739308654562189/⁠Follow me on Twitter: ⁠https://twitter.com/jaeden_ai⁠

This Week in Startups is brought to you by… Squarespace. Turn your idea into a new website! Go to Squarespace.com/TWIST for a free trial. When you're ready to launch, use offer code TWIST to save 10% off your first purchase of a website or domain. OpenPhone. Create business phone numbers for you and your team that work through an app on your smartphone or desktop. TWiST listeners can get an extra 20% off any plan for your first 6 months at openphone.com/twist Fitbod. Tired of doing the same workouts at the gym? Fitbod will build you personalized workouts that help you progress with every set. Get 25% off your subscription or try out the app for FREE when you sign up now at fitbod.me/TWIST. * Today's show: Sunny Madra joins Jason to discuss Sunny's All-In Summit experience (1:34), OpenAI's race to beat Google in launching the first multimodal LLM (9:07), whether generative AI needs a UI shift (33:00), and much more! * Time stamps: (0:00) Sunny Madra joins Jason (1:34) All-In Summit 2023 recap and Sunny's experience (7:44) Squarespace - Use offer code TWIST to save 10% off your first purchase of a website or domain at https://Squarespace.com/TWIST (9:07) OpenAI's race to beat Google in launching the first multimodal LLM (14:42) Experiences with image-generating AI and Midjourney's interface (19:23) The culture at Apple and where risk aversion sets them back (22:32) OpenPhone - Get 20% off your first six months at https://openphone.com/twist (24:03) Google's advantages and chances against OpenAI in multimodal LLMs (31:30) Fitbod - Get 25% off at https://fitbod.me/twist (33:00) UI developments and what sets multimodal LLMs apart (44:22) ChatGPT Enterprise (45:37) Sunny demos Canva's ChatGPT plugin (55:00) Code LLaMa's potential and Falcon 180B's unique features (59:41) Sunny demos Headshots AI * Follow Sunny: https://twitter.com/sundeep * Check out Headshots AI: https://headshots-starter.vercel.app/overview Check out Falcon 180B: https://falconllm.tii.ae/ * Read LAUNCH Fund 4 Deal Memo: https://www.launch.co/four Apply for Funding: https://www.launch.co/apply Buy ANGEL: https://www.angelthebook.com Great recent interviews: Steve Huffman, Brian Chesky, Aaron Levie, Sophia Amoruso, Reid Hoffman, Frank Slootman, Billy McFarland, PrayingForExits, Jenny Lefcourt Check out Jason's suite of newsletters: https://substack.com/@calacanis * Follow Jason: Twitter: https://twitter.com/jason Instagram: https://www.instagram.com/jason LinkedIn: https://www.linkedin.com/in/jasoncalacanis * Follow TWiST: Substack: https://twistartups.substack.com Twitter: https://twitter.com/TWiStartups YouTube: https://www.youtube.com/thisweekin * Subscribe to the Founder University Podcast: https://www.founder.university/podcast

Code Llama might just be starting the revolution for how data scientists code. In this Five-Minute Friday, host Jon Krohn investigates the suite of models under the free-to-use Code Llama and how to find the best fit for your project's needs. Additional materials: www.superdatascience.com/712 Interested in sponsoring a SuperDataScience Podcast episode? Visit JonKrohn.com/podcast for sponsorship information.

This week, we discuss Netflix's DVD deprecation, the remote work debate, and how to fork an open-source project. Plus, thoughts on why Europe needs more ice. Watch the YouTube Live Recording of Episode (https://www.youtube.com/watch?v=lFr-ysPYxnA) 431 (https://www.youtube.com/watch?v=lFr-ysPYxnA) Runner-up Titles Try Harder It's a necessary luxury Someone's drinking too much water here A culture of ice Where are the high performers, at home or at work Quit using your Gmail address Thou shalt export to CSV Rundown Netflix Says You Can Keep Their DVDs (and Request More, Too) (https://www.nytimes.com/2023/08/24/arts/netflix-dvds.html?smid=nytcore-ios-share&referringSource=articleShare) Zoom's CEO thinks Zoom sucks for building trust, leaked audio reveals (https://arstechnica.com/tech-policy/2023/08/leaked-audio-reveals-zoom-ceo-believes-its-hard-to-build-trust-on-zoom/) Meta is back in the office three days a week, as WFH continues to die (https://www.theverge.com/2023/9/5/23860073/meta-return-to-office-three-days-wfh-work-from-home) Can you trust 'open source' companies? (https://www.theregister.com/2023/08/18/opinion_column/) OpenTF created a fork of Terraform! (https://opentf.org/announcement) OpenTF pulls the trigger on its open-source Terraform fork (https://opensourcewatch.beehiiv.com/p/opentf-pulls-trigger-opensource-terraform-fork) Relevant to your Interests VMware's future: Navigating multicloud complexity and generative AI (https://siliconangle.com/2023/08/19/vmwares-future-navigating-multicloud-complexity-generative-ai-broadcoms-wing/) VMware Tanzu portfolio reshuffled ahead of Broadcom close | TechTarget (https://www.techtarget.com/searchitoperations/news/366549332/VMware-Tanzu-portfolio-reshuffled-ahead-of-Broadcom-close) Nvidia's blowout offers a giddy whiff of 1995 (https://www.axios.com/newsletters/axios-ai-plus-937b329c-8072-4f8a-a5d6-1039a0e794a5.html?chunk=0&utm_term=emshare#story0) Announcing AWS Dedicated Local Zones (https://aws.amazon.com/about-aws/whats-new/2023/08/aws-dedicated-local-zones/?utm_source=newsletter&utm_medium=email&utm_campaign=newsletter_axioslogin&stream=top) Top Ten social media platforms we spend the most time on (https://www.traveldailymedia.com/top-ten-social-media-platforms-we-spend-the-most-time-on/) Max will launch a 24/7 CNN stream for all subscribers next month (https://www.theverge.com/2023/8/24/23844121/cnn-max-warnerbros-discovery-news) Meta launches own AI code-writing tool: Code Llama (https://www.theverge.com/2023/8/24/23843487/meta-llama-code-generation-generative-ai-llm?stream=top) As TikTok Ban Looms, ByteDance Battles Oracle For Control Of Its Algorithm (https://www.forbes.com/sites/emilybaker-white/2023/08/24/tiktok-ban-oracle-bytedance-algorithm-fight/?sh=6cf5105e3ef0) Slack's Migration to a Cellular Architecture - Slack Engineering (https://slack.engineering/slacks-migration-to-a-cellular-architecture/) The Cloud 100 2023 (https://www.forbes.com/lists/cloud100/) Data isn't everything. Judgement counts too. (https://www.tiktok.com/t/ZT8YFUFju/) Amazon Elastic Block Store at 15 Years (https://perspectives.mvdirona.com/2023/08/amazon-elastic-block-store-at-15-years/?ck_subscriber_id=512840665) Instacart is the Best and Worst Grocery Business Imaginable (https://www.thediff.co/archive/instacart-is-the-best-and-worst-grocery-business-imaginable/) Amazon CEO Andy Jassy tells employees it's 'past' time to commit to the company's RTO mandate and their jobs are at stake (https://www.businessinsider.com/amazon-andy-jassy-rto-office-policy-employee-jobs-2023-8?op=1) Duet AI, Google's AI assistant suite, expands across Google Cloud (https://techcrunch.com/2023/08/29/duet-ai-googles-ai-assistant-suite-expands-across-google-cloud/) Halloween creeps a little closer: Seasonal supply chains accelerate (https://www.spglobal.com/marketintelligence/en/mi/research-analysis/halloween-creeps-closer-seasonal-supply-chains-accelerate.html) What's new with GKE at Google Cloud Next | Google Cloud Blog (https://cloud.google.com/blog/products/containers-kubernetes/whats-new-with-gke-at-google-cloud-next) Duet AI in Google Cloud Preview | Google Cloud Blog (https://cloud.google.com/blog/products/ai-machine-learning/duet-ai-in-google-cloud-preview) What's new in Oracle to PostgreSQL database migrations with DMS | Google Cloud Blog (https://cloud.google.com/blog/products/databases/whats-new-in-oracle-to-postgresql-database-migrations-with-dms) US AI startup Poolside raises $126m seed round and relocates to France (https://sifted.eu/articles/poolside-raises-126m-relocated-france-news) Ping, ForgeRock, Thoma Bravo, the power of open source, and the madness of IAM (https://callmeleach.substack.com/p/ping-forgerock-thoma-bravo-the-power?utm_medium=web) Thoma Bravo Completes Acquisition of ForgeRock; Combines ForgeRock into Ping Identity (https://www.prnewswire.com/news-releases/thoma-bravo-completes-acquisition-of-forgerock-combines-forgerock-into-ping-identity-301908059.html) Interoperability between Google Chat and other messaging platforms — powered by Mio (https://workspaceupdates.googleblog.com/2023/08/goolge-chat-slack-interoperability-mio.html) Broadcom boss dismisses notion China could derail VMware buy (https://www.theregister.com/2023/09/01/broadcom_vmware_nutanix_results/) Microsoft blames outage on small staff, automation failures (https://www.theregister.com/2023/09/04/microsoft_australia_outage_incident_report/) Amazon QuickSight adds scheduled and programmatic export to Excel format (https://aws.amazon.com/about-aws/whats-new/2023/08/amazon-quicksight-scheduled-programmatic-export-excel-format/?ck_subscriber_id=512840665) Google unveils AI tools for enterprise customers at $30 a month (https://www.reuters.com/technology/google-unveil-ai-tools-corporate-gmail-customers-30-month-wsj-2023-08-29/) Chip design firm Arm seeks up to $52 billion valuation in blockbuster U.S. IPO (https://www.cnbc.com/2023/09/05/chip-design-firm-arm-sets-share-price-between-47-and-51-for-blockbuster-us-ipo.html) Birmingham City Council goes under after Oracle disaster (https://www.theregister.com/2023/09/05/birmingham_city_council_oracle/?s=08) IBM Introduces 'Watsonx Your Business' (https://finance.yahoo.com/news/ibm-introduces-watsonx-business-160000392.html) Meta May Allow Instagram, Facebook Users in Europe to Pay and Avoid Ads (https://www.nytimes.com/2023/09/01/technology/meta-instagram-facebook-ads-europe.html?smid=nytcore-ios-share&referringSource=articleShare) Announcing Kubecost Cloud in General Availability: The Easiest Way to Optimize Your Kubernetes Costs (https://blog.kubecost.com/blog/kubecost-cloud-general-availability/) Platform Engineering - What You Need To Know Now (https://tanzu.vmware.com/content/ebooks/platformengineering-whatyouneedtoknownow?utm_source=cote&utm_campaign=devrel&utm_medium=newsletter&utm_content=newsletter20230830) The lifespans of technological adoptions in the US (http://www.asymco.com/2022/01/10/the-lifespans-of-technological-adoptions-in-the-us/) Introducing ONCE (https://once.com/) Nonsense The fight for the right to repair McFlurry machines (https://www.morningbrew.com/daily/stories/2023/08/31/the-fight-for-the-right-to-repair-mcflurry-machines) Delta Airlines Offers Woman $1,800 After Losing Her Dog (https://www.yahoo.com/entertainment/delta-airlines-offers-woman-1-142849291.html) Conferences Sep 18th to 19th SHIFT (https://shift.infobip.com/) in Zadar, Coté speaking. October 6, 2023, KCD Texas 2023 (https://community.cncf.io/events/details/cncf-kcd-texas-presents-kcd-texas-2023/), CFP Closes: August 30, 2023 November 6-9, 2023, KubeCon NA (https://events.linuxfoundation.org/kubecon-cloudnativecon-north-america/), SDT's a sponsor, Matt's there November 6-9, 2023 VMware Explore Barcelona (https://www.vmware.com/explore/eu.html), Coté's attending Jan 29, 2024 to Feb 1, 2024 That Conference Texas (https://that.us/events/tx/2024/schedule/) If you want your conference mentioned, let's talk media sponsorships. SDT news & hype Join us in Slack (http://www.softwaredefinedtalk.com/slack). Get a SDT Sticker! Send your postal address to stickers@softwaredefinedtalk.com (mailto:stickers@softwaredefinedtalk.com) and we will send you free laptop stickers! Follow us: Twitch (https://www.twitch.tv/sdtpodcast), Twitter (https://twitter.com/softwaredeftalk), Instagram (https://www.instagram.com/softwaredefinedtalk/), Mastodon (https://hachyderm.io/@softwaredefinedtalk), BlueSky (https://bsky.app/profile/softwaredefinedtalk.com), LinkedIn (https://www.linkedin.com/company/software-defined-talk/), TikTok (https://www.tiktok.com/@softwaredefinedtalk), Threads (https://www.threads.net/@softwaredefinedtalk) and YouTube (https://www.youtube.com/channel/UCi3OJPV6h9tp-hbsGBLGsDQ/featured). Use the code SDT to get $20 off Coté's book, Digital WTF (https://leanpub.com/digitalwtf/c/sdt), so $5 total. Become a sponsor of Software Defined Talk (https://www.softwaredefinedtalk.com/ads)! Recommendations Brandon: JUST ONE MILE | Official Trailer (https://www.youtube.com/watch?v=80V5o06yEZ4) Matt: Deadloch (https://www.imdb.com/title/tt14671678/) Coté: Rick Rubin interviews Rory Sutherland (https://www.youtube.com/watch?v=VnYlChfORRw). I doubt much of the airport business book stuff in here is “true,” but that's sort of the whole point, and it's fantastic listening. His book (https://amzn.to/462Mvov) Alchemy (https://amzn.to/462Mvov) has a great one word review right there in the title. But, again: it's fun! When you've listened to too much If Books Could Kill (https://en.wikipedia.org/wiki/If_Books_Could_Kill) you can check in on Rory if you need to take the cure (https://idioms.thefreedictionary.com/take+the+cure). Photo Credits Header (https://unsplash.com/photos/PsBTqRHVilU) Artwork (https://labs.openai.com/e/bKjqW8kPJyI2wuzBA0FogiKb/UJeLhuIFmvkrNFbfcCc4jE29)

Introducing ChatGPT Enterprise OpenAI announced they're launching ChatGPT Enterprise. This is a version of ChatGPT with enterprise-grade security and privacy, unlimited higher-speed GPT-4 access, longer context windows, advanced data analysis capabilities, customization options, and more.  The move appears to be a response to enterprise demand for a safe, compliant version of ChatGPT, says OpenAI. “Since ChatGPT's launch just nine months ago, we've seen teams adopt it in over 80% of Fortune 500 companies. We've heard from business leaders that they'd like a simple and safe way of deploying it in their organization.” Now, it looks like they're getting just that. New Google AI Updates at Google Cloud Next 23 Google made some big AI announcements at Google Cloud Next ‘23. The event was headlined by Google's announcement that Duet AI for Workspace, its generative AI tool in Gmail, Docs, Sheets, Slides, Chat, and Meet, is now generally available and has a no-cost trial.  As part of the event, Google also announced new models in Vertex AI, their suite of APIs for foundational models. You can now access Llama 2 and Code Llama from Meta using Vertex AI—and Claude 2 is coming soon. Also mentioned, there is a new digital watermarking functionality for Imagen, Google's image generation technology. This is powered by Google DeepMind's SynthID and could give us a preview of how we'll be accurately identifying AI-generated images and text in the future. OpenAI disputes authors' claims that every ChatGPT response is a derivative work OpenAI has finally broken its silence after being sued by a number of authors, all of whom allege that ChatGPT was illegally trained on their work without permission. OpenAI is looking to dismiss the lawsuits, saying: "the use of copyrighted materials by innovators in transformative ways does not violate copyright."  Unlike plagiarists who seek to directly profit off distributing copyrighted materials, OpenAI argued that its goal was "to teach its models to derive the rules underlying human language" to do things like help people "save time at work," "make daily life easier," or simply entertain themselves by typing prompts into ChatGPT. Citing a notable copyright case involving Google Books, OpenAI also reminded the court that "while an author may register a copyright in her book, the 'statistical information' pertaining to 'word frequencies, syntactic patterns, and thematic markers' in that book are beyond the scope of copyright protection." Enjoy the episode! It was a busy week in the world of AI! Listen to the full episode of the podcast: https://www.marketingaiinstitute.com/podcast-showcase Want to receive our videos faster? SUBSCRIBE to our channel! Visit our website: https://www.marketingaiinstitute.com Receive our weekly newsletter: https://www.marketingaiinstitute.com/newsletter-subscription Looking for content and resources? Register for a free webinar: https://www.marketingaiinstitute.com/resources#filter=.webinar  Come to our next Marketing AI Conference: www.MAICON.ai Enroll in AI Academy for Marketers: https://www.marketingaiinstitute.com/academy/home Join our community: Slack: https://www.marketingaiinstitute.com/slack-group-form LinkedIn: https://www.linkedin.com/company/mktgai Twitter: https://twitter.com/MktgAi Instagram: https://www.instagram.com/marketing.ai/ Facebook: https://www.facebook.com/marketingAIinstitute

Dans le monde de l'IA, Meta est connu pour Llama 2, son modèle de langage avancé similaire à GPT-4, le modèle concurrent d'OpenAI qui sert de base technique à ChatGPT. Aujourd'hui, le groupe américain fondé par Mark Zuckerberg présente Code Llama, une déclinaison spécialement conçue pour la rédaction de code informatique. Dans un article publié sur son blog, Meta explique que ce modèle prend en charge plusieurs langages de programmation, notamment Python, C++, Java, PHP, JavaScript et Bash. Cela signifie que les professionnels et les amateurs peuvent générer du code à partir de phrases en langage naturel. Il vous suffit de formuler votre demande, pour que le code apparaisse à l'écran en quelques secondes. Les utilisateurs expérimentés peuvent également entrer directement du code dans la zone de saisie pour affiner leur demande. À noter que Code Llama peut également corriger du code dysfonctionnel, ce qui pourra faire gagner pas mal de temps aux développeurs. De plus, Meta propose trois versions différentes de Code Llama : une première avec 7 milliards de paramètres, une autre à 13 milliards et la dernière à 34 milliards de paramètres. Les paramètres servent à évaluer les capacités globales d'une intelligence artificielle à produire des résultats précis. En résumé, plus le modèle est grand, plus il demande de ressource pour tourner rapidement. La version à 7 milliards de paramètres serait d'ailleurs plus efficace pour générer du code en temps réel. Les premiers résultats obtenus avec ce nouveau modèle semblent prometteurs. Meta partage les résultats de Code Llama obtenus grâce à l'outil d'évaluation HumanEval, et il semblerait que ces résultats se situent au niveau de ceux de GPT-3.5. L'avantage de cet outil par rapport à GPT-4 ou à GitHub Copilot de Microsoft, c'est sa gratuité. En effet, Code Llama est d'ores et déjà disponible sur le site de Meta, et le modèle de langage, tout comme Llama 2, est open source. Learn more about your ad choices. Visit megaphone.fm/adchoices

Dans le monde de l'IA, Meta est connu pour Llama 2, son modèle de langage avancé similaire à GPT-4, le modèle concurrent d'OpenAI qui sert de base technique à ChatGPT. Aujourd'hui, le groupe américain fondé par Mark Zuckerberg présente Code Llama, une déclinaison spécialement conçue pour la rédaction de code informatique.Dans un article publié sur son blog, Meta explique que ce modèle prend en charge plusieurs langages de programmation, notamment Python, C++, Java, PHP, JavaScript et Bash. Cela signifie que les professionnels et les amateurs peuvent générer du code à partir de phrases en langage naturel. Il vous suffit de formuler votre demande, pour que le code apparaisse à l'écran en quelques secondes. Les utilisateurs expérimentés peuvent également entrer directement du code dans la zone de saisie pour affiner leur demande. À noter que Code Llama peut également corriger du code dysfonctionnel, ce qui pourra faire gagner pas mal de temps aux développeurs. De plus, Meta propose trois versions différentes de Code Llama : une première avec 7 milliards de paramètres, une autre à 13 milliards et la dernière à 34 milliards de paramètres. Les paramètres servent à évaluer les capacités globales d'une intelligence artificielle à produire des résultats précis. En résumé, plus le modèle est grand, plus il demande de ressource pour tourner rapidement. La version à 7 milliards de paramètres serait d'ailleurs plus efficace pour générer du code en temps réel.Les premiers résultats obtenus avec ce nouveau modèle semblent prometteurs. Meta partage les résultats de Code Llama obtenus grâce à l'outil d'évaluation HumanEval, et il semblerait que ces résultats se situent au niveau de ceux de GPT-3.5. L'avantage de cet outil par rapport à GPT-4 ou à GitHub Copilot de Microsoft, c'est sa gratuité. En effet, Code Llama est d'ores et déjà disponible sur le site de Meta, et le modèle de langage, tout comme Llama 2, est open source. Hébergé par Acast. Visitez acast.com/privacy pour plus d'informations.

We release Code Llama, a family of large language models for code based on Llama 2 providing state-of-the-art performance among open models, infilling capabilities, support for large input contexts, and zero-shot instruction following ability for programming tasks. We provide multiple flavors to cover a wide range of applications: foundation models (Code Llama), Python specializations (Code Llama - Python), and instruction-following models (Code Llama - Instruct) with 7B, 13B and 34B parameters each. All models are trained on sequences of 16k tokens and show improvements on inputs with up to 100k tokens. 7B and 13B Code Llama and Code Llama - Instruct variants support infilling based on surrounding content. Code Llama reaches state-of-the-art performance among open models on several code benchmarks, with scores of up to 53% and 55% on HumanEval and MBPP, respectively. Notably, Code Llama - Python 7B outperforms Llama 2 70B on HumanEval and MBPP, and all our models outperform every other publicly available model on MultiPL-E. We release Code Llama under a permissive license that allows for both research and commercial use. 2023: Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, J. Rapin, Artyom Kozhevnikov, I. Evtimov, Joanna Bitton, Manish P Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre D'efossez, Jade Copet, F. Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, Gabriel Synnaeve https://arxiv.org/pdf/2308.12950v2.pdf

Vi forteller at selveste Captain Credible kommer på kode24-dagen. Vi inviterer på kode24s bursdagsfest, torsdag 7. september, 17:30 på Brygg. Kona til Jørgen vil høre mer om egen familie, Ole Petters pappa forteller enda mer om hans families doss. Jørgen har vært på sirkus. Ole Petter har bodd på Sommero og spist på Brasilia. Norske selskaper konkurrerer om beste startlønn. Meta prøver seg med Code Llama. Hva pokker er IBMs Watsonx Code Assistant for Z? kode24-klubben fabler om noe som heter XAI. Sirkus Arildos og Sirkus Arnodos har vitser.  See omnystudio.com/listener for privacy information.

Hashtag Tendances, 31 août 2023 — Un hébergeur danois victime d'un rançongiciel, TikTok veut bannir les liens de commerce électronique, Meta lance Code Llama et une puce d'intelligence artificielle moins énergivore signée IBM.

OpenTF announces they're forking Terraform and joining the Linux Foundation, Meta gets in the LLM-for-codegen game with Code Llama, Matt Mullenweg announces WordPress.com's new 100-year plan, Paul Gichuki from Thinkst learns that default behaviors stick (and so do examples) & Marco Otte-Witte makes his case for Rust on the web.

OpenTF announces they're forking Terraform and joining the Linux Foundation, Meta gets in the LLM-for-codegen game with Code Llama, Matt Mullenweg announces WordPress.com's new 100-year plan, Paul Gichuki from Thinkst learns that default behaviors stick (and so do examples) & Marco Otte-Witte makes his case for Rust on the web.

Code Llama may spur a new wave of experimentation around AI and programming—but it will also help Meta. Read this story here. Learn more about your ad choices. Visit megaphone.fm/adchoices

OpenTF announces they're forking Terraform and joining the Linux Foundation, Meta gets in the LLM-for-codegen game with Code Llama, Matt Mullenweg announces WordPress.com's new 100-year plan, Paul Gichuki from Thinkst learns that default behaviors stick (and so do examples) & Marco Otte-Witte makes his case for Rust on the web.

Have you seen the new Gremlin Chaos Engineering certification? What company did Smartbear just acquire? And will Meta's new Llama language model change automation testing coding forever? Find out in the episode of the Test Guild News Show for the week of August 27. So grab your favorite cup of coffee or tea, and let's do this! Time News Title Rocket Link 0:18  Applitoools FREE Account Offer https://applitools.info/joe  0:36 SmartBear to Acquire Stoplight https://testguild.me/3o0yah 2:08 API testing tool called Step CI https://testguild.me/niunm6 3:19 Microsoft Playwright Testing, a new service https://testguild.me/3ut5on 4:25 Announcing the launch of Karate Learning Center https://testguild.me/89v1c2 5:22 Meta AI just announced Code Llama! https://testguild.me/1gtlln 7:03  Tests Using Cypress in React Applications https://testguild.me/meqp7m 7:57 Gremlin Enterprise Chaos Engineering Certification https://testguild.me/kb6zz2 9:04 Grafana k6 v0.46.0 release https://testguild.me/prxz84

Meta veröffentlicht Open Source Code-KI ChatGPT erzielt teilweise bessere Noten als Studierende Metas Llama 3 soll so gut wie GPT-4 werden Und Etcembly präsentiert erste KI-gestützte Krebsimmuntherapie heise.de/ki-update https://www.heise.de/thema/Kuenstliche-Intelligenz https://the-decoder.de/ https://www.heiseplus.de/podcast

John Siwicki discusses several interesting topics in this episode. He starts by introducing the Text FX project from Google, which offers AI-powered tools for rappers, writers, and wordsmiths. He explores the different tools available, such as Simile Explode, Unexpected Alliteration, and Acronym Fuse Scene Unfold.Next, he mentions a project called "Visualizing AI" by Google's DeepMind. This project showcases artwork and animations created by artists based on AI. The visuals are captivating and provide a unique perspective on AI.John then moves on to discuss a new plugin called Jambot by Figma. Jambot brings Chat GPT inside Figma's whiteboard software, allowing users to utilize AI-generated responses during ideation and collaboration sessions.He also highlights the launch of Code Llama by Meta Facebook, a model specifically designed for code writing. The model is open-sourced, and Perpexity Labs provides a playground for users to experiment with it.Lastly, John shares the big news of the week: OpenAI's update to their API, which now allows fine-tuning of Chat GPT Three and a Half Turbo. Fine-tuning enables users to train the model on their own data, resulting in higher quality results, shorter prompts, lower costs, and lower latency requests.Key Takeaways:* Google's Text FX project offers AI-powered tools for rappers, writers, and wordsmiths.* "Visualizing AI" by Google's DeepMind showcases artwork and animations inspired by AI.* Figma's Jambot plugin brings Chat GPT inside their whiteboard software for collaborative ideation.* Meta Facebook launched Code Llama, a model tuned for code writing, which is open-sourced and available for experimentation.* OpenAI's API update allows fine-tuning of Chat GPT Three and a Half Turbo, resulting in higher quality results, lower costs, and lower latency requests.Links: * https://www.figma.com/community/widget/1274481464484630971* https://labs.perplexity.ai/* https://platform.openai.com/docs/guides/fine-tuning* https://www.deepmind.com/visualising-ai * https://textfx.withgoogle.com/* https://www.youtube.com/@stacksnacks This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.stack-snacks.com

Stick around until the end for a BONUS episode of Boot Up with Jason! If you want to subscribe to the show (yes, you want to), go to BootUp.show!Unfortunate translations; a tech bro reckoning; Nvidia makes bank, carbon emissions; NFT & crypto collapses; jail vegan diets; if you're gonna launder, launder big; food delivery is tanking; the Feds tried to get what the Chinese do with TikTok; Gizmodo's killer headlines; Code Llama, rusted; The Peripheral canceled, Frasier returns; Dune 2 delayed; the Matildas; last chance for Devo, OMD; Youtube's AI plan; Threads on the web; Photoshop's generative refill on old photos; Wipeout code; frivolous & not so frivolous lawsuits; Ahsoka; Wirecutter; dystopian fiction; Radio Shack would like your address, again.Sponsors:Hover - Go to Hover now and grab your very own domain or a few of them at hover.com/gog and get 10% off your first purchase.1Password - Get a great deal on the only password manager recommended by Grumpy Old Geeks! gog.show/1passwordPrivate Internet Access - Go to GOG.Show/vpn and sign up today. For a limited time only, you can get OUR favorite VPN for as little as $2.03 a month.Show notes at: https://gog.show/615/FOLLOW UPElon Musk's Shadow RuleFyre Fest 2 tickets go on sale, despite no line-up or venueThe Internet's Next Great Power SuckIN THE NEWSNvidia just made $6 billion in pure profit over the AI boomGodfather NFTs Weren't Enough to Prevent Recur From Sleeping With the FishesSam Bankman-Fried isn't getting his Adderall and is surviving on 'bread and water,' lawyers sayFormer OpenSea Executive Sentenced to Three Months in Prison for Insider TradingDOJ charges Tornado Cash co-founders for laundering over $1 billion in cryptoThe Era of Food Delivery Is FadingThe Feds Asked TikTok for Lots of Domestic Spying FeaturesPeloton's Business Is as Busted as Its Bike SeatsZoom CEO Says Employees Need to Be in the Office Because It's Hard to Build Trust Over ZoomA Space Junk Removal Mission Got Struck By Space JunkIntroducing Code Llama, a state-of-the-art large language model for codingMEDIA CANDY"The Peripheral" has been canceled at Amazon's Prime Video, Variety has learned from sources.‘Frasier' Revival Sets Premiere Date at Paramount+, First Two Episodes to Air on CBS‘Dune: Part Two' Delays Release to 2024 Amid Ongoing Actors StrikeMatildas: The World At Our FeetDevo Has the Uncontrollable Urge to RetireYouTube is working on a plan to compensate artists and rightsholders for AI musicOur principles for partnering with the music industry on AI technologyI'm an Apple TV channels customer. Can I stream Paramount+ via the Apple TV app?APPS & DOODADSThreads on the web is herePeople are Using Photoshop's Generative Fill to Restore Old PhotosLeaked Wipeout source code leads to near-total rewrite and remasterJudge dismisses lawsuit claiming Apple Watch blood oxygen sensor has a racial bias.US judge: Art created solely by artificial intelligence cannot be copyrightedTHE DARK SIDE WITH DAVEThe CyberWireDave BittnerHacking HumansCaveatControl LoopAhsokaStar Wars B-Wing Drone FootageWhat Happened to Wirecutter?RadioShack's NSFW Period Appears Over, as New Owner Plots a Brand ExpansionThe College Board Tells TikTok and Facebook Your SAT ScoresCLOSING SHOUT-OUTSAdobe Co-Founder Dr. John Warnock Has Passed AwayWWE legend Terry Funk dead at 79See Privacy Policy at https://art19.com/privacy and California Privacy Notice at https://art19.com/privacy#do-not-sell-my-info.

Nesse episódio trouxemos as notícias e novidades do mundo da programação que nos chamaram atenção dos dias 19/08 a 25/08.

Nesse episódio trouxemos as notícias e novidades do mundo da programação que nos chamaram atenção dos dias 19/08 a 25/08.

Here in the US students are heading back to school. But there's one problem that hasn't been solved yet. Using AI and ChatGPT in the classroom. Some schools have started to embrace it while others still reject it. What should schools do about it?Newsletter: Sign up for our free daily newsletterMore on this: Episode PageJoin the discussion: Ask Jordan questions about ChatGPT and educationRelated Episodes:Ep 35: How Students Can Use AI to Solve Everyday Problems​Ep 55: How to properly leverage AI in the classroom​Ep 74: Should Schools Ban ChatGPT?Upcoming Episodes: Check out the upcoming Everyday AI Livestream lineupWebsite: YourEverydayAI.comEmail The Show: info@youreverydayai.comConnect with Jordan on LinkedInTimestamps:[00:01:15] Daily AI news[00:06:50] Jordan's hot take about AI in education[00:09:45] Recent headlines about AI in schools[00:16:07] Will AI make students dumb?[00:20:00] Why parents should care about students learning AI[00:23:35] Proposed solutions to "combat" AI and cheating[00:26:00] How AI affects skill sets Topics Covered in This Episode:- AI News Stories- Meta (Facebook's parent company) launches Code Llama, a new AI code tool- Alibaba updates its AI chatbots to understand images and have more complex conversations- Big-name companies, including Nvidia, Amazon, Google, Intel, AMD, IBM, and Salesforce, invest in the open-source platform Hugging Face- Personal Opinion- Jordan shares his views on AI in schools- College students using AI to write papers- Current AI detectors are not accurateKeywords: AI content, Meta, Facebook, Code Llama, AI code tool, Alibaba, AI chatbots, images, complex conversations, Nvidia, Amazon, Google, Intel, AMD, IBM, Salesforce, open source platform, Hugging Face, AI in schools, college students, AI detectors, accuracy. Get more out of ChatGPT by learning our PPP method in this live, interactive and free training! Sign up now: https://youreverydayai.com/ppp-registration/

Meta has released LLM-for-coding Code Llama in numerous versions. NLW explores the community discussion, including some interesting data around an unreleased version trained on synthetic data that seemed to perform better than any other. Before that on the Brief, Spain starts an AI agency; the UK announces more details of its AI Safety Summit and new AI models out of South Korea and China. Today's Sponsor: Supermanage - AI for 1-on-1's - https://supermanage.ai/breakdown ABOUT THE AI BREAKDOWN The AI Breakdown helps you understand the most important news and discussions in AI.  Subscribe to The AI Breakdown newsletter: https://theaibreakdown.beehiiv.com/subscribe Subscribe to The AI Breakdown on YouTube: https://www.youtube.com/@TheAIBreakdown Join the community: bit.ly/aibreakdown Learn more: http://breakdown.network/

This week, the Zuck strikes again - Meta unveils a state of the art AI code generator to challenge OpenAI's dominance. We explore the implications of AI models training themselves, and how it could accelerate capabilities. Then we put 11 labs' multilingual speech synthesis to the test, using it to generate a fake phishing call on our mother. Don't miss our scandalous experiments pushing AI to its limits in this jam-packed episode!If you like the pod, please consider subbing, liking, commenting etc. xoxCHAPTERS:=====00:00 - Rehearsal of Phishing Our Mother (Cold Open)00:19 - Meta's Code Llama08:24 - Unnatural Instruction to Train AI Models15:06 - Why Didn't Meta Release the Unnatural Instruction Code Llama Model? The Sparks of AGI?16:50 - Evolution of GPT: Is Unnatural Instruction The Next Evolution of Models?23:04 - DeepMind's Reinforced Self-Training ReST for Language Modeling paper and thoughts on future models36:09 - Fine Tuning GPT-3.5 Turbo Announced by OpenAI: Should You Just Fine Tune Open Source?44:05 - ElevenLabs Out of Beta and Multilingual v2: Explained by AI Us.48:12 - Chris Tried to Figure Out AI Phishing53:03 - Rehearsing Phishing Our Mother Call & Implications of This AI Tech59:43 - How Much We Lost Not Investing in NVIDIA1:01:29 - AI Bros Give Investment AdviceSOURCES:======https://ai.meta.com/blog/code-llama-large-language-model-coding/https://www.theinformation.com/articles/metas-next-ai-attack-on-openai-free-code-generating-softwarehttps://twitter.com/emollick/status/1694793231727210579?s=46&t=uXHUN4Glah4CaV-g2czc6Qhttps://minimaxir.com/2023/08/stable-diffusion-xl-wrong/https://twitter.com/abacaj/status/1679996952560246786/photo/1https://openai.com/blog/gpt-3-5-turbo-fine-tuning-and-api-updateshttps://arstechnica.com/ai/2023/08/how-chatgpt-turned-generative-ai-into-an-anything-tool/https://elevenlabs.io/blog/multilingualv2/https://www.businessinsider.com/nvidia-technology-spending-wave-build-out-google-meta-oracle-gpu-2023-8PAPERS:======https://arxiv.org/pdf/2212.09689.pdfhttps://arxiv.org/pdf/2308.08998.pdf

In this episode, we dive into Meta's latest technological marvel, Code Llama, an AI model designed to automatically generate code. We discuss its capabilities, potential impact on the software development industry, and the ethical implications of automating such a critical skill set. Get on the AI Box Waitlist: ⁠https://AIBox.ai/⁠ Facebook Community: ⁠⁠https://www.facebook.com/groups/739308654562189/⁠⁠ Discord Community: ⁠https://aibox.ai/discord⁠ Follow me on X: ⁠https://twitter.com/jaeden_ai⁠

This is a recap of the top 10 posts on Hacker News on August 24th, 2023.This podcast was generated by wondercraft.ai(00:37): Code Llama, a state-of-the-art large language model for codingOriginal post: https://news.ycombinator.com/item?id=37248494&utm_source=wondercraft_ai(01:58): Hugging Face raises $235M from investors including Salesforce and NvidiaOriginal post: https://news.ycombinator.com/item?id=37248895&utm_source=wondercraft_ai(03:36): Hacker News GuidelinesOriginal post: https://news.ycombinator.com/item?id=37250834&utm_source=wondercraft_ai(05:32): Bypassing Bitlocker using a cheap logic analyzer on a Lenovo laptopOriginal post: https://news.ycombinator.com/item?id=37249623&utm_source=wondercraft_ai(07:08): Leaving Haskell behindOriginal post: https://news.ycombinator.com/item?id=37246932&utm_source=wondercraft_ai(08:59): Jacobin: A more than minimal JVM written in GoOriginal post: https://news.ycombinator.com/item?id=37247394&utm_source=wondercraft_ai(10:53): FreeBSD on FirecrackerOriginal post: https://news.ycombinator.com/item?id=37253035&utm_source=wondercraft_ai(12:45): Adding water to Martian soil samples might have been a bad ideaOriginal post: https://news.ycombinator.com/item?id=37249442&utm_source=wondercraft_ai(14:23): Graph of Thoughts: Solving Elaborate Problems with Large Language ModelsOriginal post: https://news.ycombinator.com/item?id=37248694&utm_source=wondercraft_aiThis is a third-party project, independent from HN and YC. Text and audio generated using AI, by wondercraft.ai. Create your own studio quality podcast with text as the only input in seconds at app.wondercraft.ai. Issues or feedback? We'd love to hear from you: team@wondercraft.ai

If AI Becomes Conscious: Here's How Researchers Will Know Meta launches own AI code-writing tool: Code Llama. In Reversal Because of A.I., Office Jobs Are Now More at Risk Google, Amazon, Nvidia, and others put $235 million into Hugging Face. Ai Providers Begin To Explore New Terrain: Chatbots In Salary Negotiations Chatgpt Has ‘Systematic Political Bias' Towards The Left. --- Send in a voice message: https://podcasters.spotify.com/pod/show/aidaily/message

Meta's launched AI to produce code  Their new "Code Llama" can create working code from natural language prompts, and also debug and explain code pasted into it. So in what is a bizarre turn of events, Meta software engineers have created AI to take their job… or at least parts of it.   Meta says this allows its staff to focus on the "most human-centric aspects of their job, rather than repetitive tasks". But they don't go so far as to say what those human aspects are.  Microsoft's Github Copilot has similar features but is currently being sued for copyright because the AI can reproduce proprietary code.     Donald Trump is back on Twitter/X  Back to post an image with his mug shot from his booking in Atlanta, with the words "Election Interference, never surrender" and just a link to his website to fundraise.  Trump was banned indefinitely, until Elon bought Twitter and gave him his account back.   His return is not a surprise though. According to reports earlier this year, his exclusivity deal with his own Truth Social was due to expire in June and sources said he was eager to get back to Twitter. This post could be the first of many to come.   LISTEN ABOVESee omnystudio.com/listener for privacy information.

Code Llama, a new state-of-the-art large language model for coding, and Microsoft's plans to add AI capabilities to apps like Paint and Photos on Windows 11. We also explore Ludwig, a low-code framework for building custom AI models, and WizardMath, a new model that improves the mathematical reasoning abilities of large language models. Contact:  sergi@earkind.com Timestamps: 00:34 Introduction 01:34 Introducing Code Llama, a state-of-the-art large language model for coding 03:24 Microsoft may bring AI capabilities to apps like Paint and Photos on Windows 11 05:24 Ludwig: a low-code framework for building custom AI models like LLMs and other deep neural networks. 06:48 Fake sponsor 08:53 Bayesian Flow Networks 10:13 The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only 11:59 WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct 13:44 Outro

Twitter blocks EveryAction's NGP VAN links, privacy watchdogs call for protection against data scraping, Apple supports California's right-to-repair bill, rumors of USB-C charging port for iPhone 15, Digital Services Act imposes transparency requirements on Meta, Google, and Apple, U.S. Justice Department sues SpaceX over hiring practices, Zoom CEO believes returning to the office challenges trust-building, Max launches CNN Max streaming news feature, Meta introduces AI code-writing tool Code Llama, and TechCrunch covers Tornado Cash founders and FBI monitoring North Korean hackers.

Parliamo del Digital Markets Act europeo oggi che entra in vigore con un pochino di analisi, e poi vediamo il rilascio della versione 0.8.0 di Bun, Codel Llama di Meta e un po' di news economiche sulla AI. #dma #eu #bun #codellama #ai === Podcast Anchor - https://anchor.fm/edodusi Spotify - https://open.spotify.com/show/4B2I1RTHTS5YkbCYfLCveU Apple Podcasts - https://podcasts.apple.com/us/podcast/buongiorno-da-edo/id1641061765 Google Podcasts - https://podcasts.google.com/feed/aHR0cHM6Ly9hbmNob3IuZm0vcy9iMWJmNDhhMC9wb2RjYXN0L3Jzcw Amazon Music - https://music.amazon.it/podcasts/5f724c1e-f318-4c40-9c1b-34abfe2c9911/buongiorno-da-edo = RSS - https://anchor.fm/s/b1bf48a0/podcast/rss --- Send in a voice message: https://podcasters.spotify.com/pod/show/edodusi/message

Unboxing videóban bukkant fel a Xiaomi 13T Android Portál     2023-08-25 08:20:46     Mobiltech Xiaomi Guatemala A Xiaomi 13T-t Guatemalában bukkant fel egy unboxing videóban, amely előzetes tippek szerint a Redmi K60 Ultra globális megfelelője lesz. Megjegyzendő, hogy ez nem a Xiaomi 13T Pro, amiről szintén hallottunk, és amely szintén a K60 Ultra globális megfelelője. A képek ugyanazt a dizájnt mutatják, mint a Redmi K60 Ultra – csak a sarki Redmi A tudást megajándékozzák Kókán: laptopot ajánlott fel a kitűnő végzősöknek egy vállalkozó Digital Hungary     2023-08-25 08:57:05     Infotech Oktatás Ajándék A kókai Kossuth Lajos Általános Iskolában egy különleges és inspiráló kezdeményezés indult el, amelynek célja a diákok motiválása és a továbbtanulásuk támogatása. Az iskola egykori diákja, Csalami János, az Electronika Vonala alapítója és ügyvezetője olyan ajándékot kínál a nyolcadik osztályt végző kiemelkedő diákoknak, amely hosszú távon hozzájáru Fél óra és hozza a drón a felvágottat IT Business     2023-08-25 11:11:12     Infotech USA Drón Google Texas Leányvállalat Dallas Az Alphabet leányvállalataként tevékenykedő Wing drónos szállítócég a Walmarttal együttműködve az Egyesült Államokban, pontosabban a Dallas-Fort Worth metroplexben kezdi meg távirányítású légi kiszállítási szolgáltatását. A tervek szerint első körben 60 ezer háztartás veheti majd igénybe a légi futárszolgálatot. A Wing ezt megelőzően a texasi Frisc Ingyenes lesz a Meta programkódokat gyártó mesterséges intelligenciája Bitport     2023-08-25 09:36:00     Infotech Mesterséges intelligencia A jövő hétre várhatók részletek a Meta újabb nagy dobása, a nyílt forrású Code Llama bevezetésével kapcsolatban. A techipar be nem tartott nagy ígéretei InStyle Men     2023-08-25 06:06:55     Infotech A korszakalkotó újítások szépen ugyanolyanná válnak, mint azok, amiket leváltani hivatottak. A gyerekek sehol nincsenek biztonságban, ha az internetes veszélyekről van szó Márkamonitor     2023-08-25 07:36:06     Infotech A digitális eszközök és az online világ mindennapjaink szerves részévé váltak, és ez alól a gyermekek sem jelentenek kivételt. Minél több időt töltenek a fiatalok az online térben, annál inkább megnő a veszélye, hogy számukra káros tartalommal találkoznak, sokuk pedig egyáltalán nincs felkészülve az ilyen helyzetek helyes kezelésére.   Az Internet Megszűnik a népszerű Facebook Messenger app, amelyik jól bánt a telefonnal hvg.hu     2023-08-25 08:03:00     Mobiltech Telefon Messenger Mobilinternet Váltani kell: szeptemberben megválik a készülék erőforrásaival és a mobilinternettel is takarékosabb Messengertől a Facebook. Az antarktiszi jég pusztulása a kihalás szélére sodorhatja az ott élő pingvineket Qubit     2023-08-25 10:45:29     Tudomány Antarktisz Tavaly 5 császárpingvin kolóniából 4 fiókái pusztultak el a jégolvadás következtében. Ha a pingvinek nem találnak maguknak új élőhelyeket a költési időszakra, az a faj fennmaradását lehetetleníti el. Nagyot változik a világ mától az online óriásplatformoknál HWSW     2023-08-25 09:41:20     Infotech Európai Unió Az Európai Unió digitális szolgáltatásokra vonatkozó rendeletének szigorú feltételei 19 rendszerszintű szolgáltatást érintenek. Vajon meg tudja védeni az embereket a big tech-től az EU? IT café     2023-08-25 12:46:00     Infotech Thierry Breton szerint a DSA remek lehetőség arra, hogy a nagy tech cégek visszanyerjék az emberek bizalmát. A mesterséges intelligencia elterjedése a társadalmi átrendeződés egyik hajtóereje lesz Digital Hungary     2023-08-25 06:51:07     Gazdaság Mesterséges intelligencia Robot Mindannyian lélegzet-visszafojtva figyeljük a mesterséges intelligencia (MI) térnyerését. Egyre több az aggódó hang: ki szab határt a rohamos fejlődésnek? Képesek leszünk jogilag is lekövetni a piac változásait? Milyen alkalmazásokat lesz tilos fejleszteni? És vajon elveszik-e a robotok a munkánkat a következő évtizedben? Karászi Zoltánnal, a több Egy ipari kapu hibája miatt soha ne álljon le a termelés! Transpack     2023-08-25 08:11:00     Cégvilág Robot Kamera Gyorsmozgású ipari kapukat, robotvezérlet védelmi technikát, tűzmegelőzésre is képes kamerát mutatott be az Efaflex Kft. A VMware is a mesterséges intelligenciában látja az jövő üzletét Bitport     2023-08-25 15:01:00     Infotech Mesterséges intelligencia Nvidia A vállalat az Explore konferencián jelentette be, hogy az Nvidiával társulva MI-megoldásokhoz épít felhőplatformot.

This is the AI News Briefing of August 25, 2023.(00:29) AI-powered brain implant helps paralyzed speak(01:06) Nvidia's record $6 billion profit from AI chips(01:43) Meta's new open-source AI coding tool, Code Llama(02:14) Major AI investment deals of the week AI-powered brain implant helps paralyzed speak: https://www.ucsf.edu/news/2023/08/425986/how-artificial-intelligence-gave-paralyzed-woman-her-voice-back Nvidia's record $6 billion profit from AI chips: https://www.axios.com/2023/08/23/why-nvidia-stock-price-up-chart-revenue-q2-2023 Meta's new open-source AI coding tool, Code Llama: https://about.fb.com/news/2023/08/code-llama-ai-for-coding/ Follow our newsletter at www.adepto.ai for a deeper dive into these fascinating developments and for the latest AI news and insights.The AI News Briefing has been produced by Adepto in cooperation with Wondercraft AI.Music: Inspire by Kevin MacLeod (incompetech.com), Licensed under Creative Commons: By Attribution 3.0 http://creativecommons.org/licenses/by/3.0/

Unboxing videóban bukkant fel a Xiaomi 13T Android Portál     2023-08-25 08:20:46     Mobiltech Xiaomi Guatemala A Xiaomi 13T-t Guatemalában bukkant fel egy unboxing videóban, amely előzetes tippek szerint a Redmi K60 Ultra globális megfelelője lesz. Megjegyzendő, hogy ez nem a Xiaomi 13T Pro, amiről szintén hallottunk, és amely szintén a K60 Ultra globális megfelelője. A képek ugyanazt a dizájnt mutatják, mint a Redmi K60 Ultra – csak a sarki Redmi A tudást megajándékozzák Kókán: laptopot ajánlott fel a kitűnő végzősöknek egy vállalkozó Digital Hungary     2023-08-25 08:57:05     Infotech Oktatás Ajándék A kókai Kossuth Lajos Általános Iskolában egy különleges és inspiráló kezdeményezés indult el, amelynek célja a diákok motiválása és a továbbtanulásuk támogatása. Az iskola egykori diákja, Csalami János, az Electronika Vonala alapítója és ügyvezetője olyan ajándékot kínál a nyolcadik osztályt végző kiemelkedő diákoknak, amely hosszú távon hozzájáru Fél óra és hozza a drón a felvágottat IT Business     2023-08-25 11:11:12     Infotech USA Drón Google Texas Leányvállalat Dallas Az Alphabet leányvállalataként tevékenykedő Wing drónos szállítócég a Walmarttal együttműködve az Egyesült Államokban, pontosabban a Dallas-Fort Worth metroplexben kezdi meg távirányítású légi kiszállítási szolgáltatását. A tervek szerint első körben 60 ezer háztartás veheti majd igénybe a légi futárszolgálatot. A Wing ezt megelőzően a texasi Frisc Ingyenes lesz a Meta programkódokat gyártó mesterséges intelligenciája Bitport     2023-08-25 09:36:00     Infotech Mesterséges intelligencia A jövő hétre várhatók részletek a Meta újabb nagy dobása, a nyílt forrású Code Llama bevezetésével kapcsolatban. A techipar be nem tartott nagy ígéretei InStyle Men     2023-08-25 06:06:55     Infotech A korszakalkotó újítások szépen ugyanolyanná válnak, mint azok, amiket leváltani hivatottak. A gyerekek sehol nincsenek biztonságban, ha az internetes veszélyekről van szó Márkamonitor     2023-08-25 07:36:06     Infotech A digitális eszközök és az online világ mindennapjaink szerves részévé váltak, és ez alól a gyermekek sem jelentenek kivételt. Minél több időt töltenek a fiatalok az online térben, annál inkább megnő a veszélye, hogy számukra káros tartalommal találkoznak, sokuk pedig egyáltalán nincs felkészülve az ilyen helyzetek helyes kezelésére.   Az Internet Megszűnik a népszerű Facebook Messenger app, amelyik jól bánt a telefonnal hvg.hu     2023-08-25 08:03:00     Mobiltech Telefon Messenger Mobilinternet Váltani kell: szeptemberben megválik a készülék erőforrásaival és a mobilinternettel is takarékosabb Messengertől a Facebook. Az antarktiszi jég pusztulása a kihalás szélére sodorhatja az ott élő pingvineket Qubit     2023-08-25 10:45:29     Tudomány Antarktisz Tavaly 5 császárpingvin kolóniából 4 fiókái pusztultak el a jégolvadás következtében. Ha a pingvinek nem találnak maguknak új élőhelyeket a költési időszakra, az a faj fennmaradását lehetetleníti el. Nagyot változik a világ mától az online óriásplatformoknál HWSW     2023-08-25 09:41:20     Infotech Európai Unió Az Európai Unió digitális szolgáltatásokra vonatkozó rendeletének szigorú feltételei 19 rendszerszintű szolgáltatást érintenek. Vajon meg tudja védeni az embereket a big tech-től az EU? IT café     2023-08-25 12:46:00     Infotech Thierry Breton szerint a DSA remek lehetőség arra, hogy a nagy tech cégek visszanyerjék az emberek bizalmát. A mesterséges intelligencia elterjedése a társadalmi átrendeződés egyik hajtóereje lesz Digital Hungary     2023-08-25 06:51:07     Gazdaság Mesterséges intelligencia Robot Mindannyian lélegzet-visszafojtva figyeljük a mesterséges intelligencia (MI) térnyerését. Egyre több az aggódó hang: ki szab határt a rohamos fejlődésnek? Képesek leszünk jogilag is lekövetni a piac változásait? Milyen alkalmazásokat lesz tilos fejleszteni? És vajon elveszik-e a robotok a munkánkat a következő évtizedben? Karászi Zoltánnal, a több Egy ipari kapu hibája miatt soha ne álljon le a termelés! Transpack     2023-08-25 08:11:00     Cégvilág Robot Kamera Gyorsmozgású ipari kapukat, robotvezérlet védelmi technikát, tűzmegelőzésre is képes kamerát mutatott be az Efaflex Kft. A VMware is a mesterséges intelligenciában látja az jövő üzletét Bitport     2023-08-25 15:01:00     Infotech Mesterséges intelligencia Nvidia A vállalat az Explore konferencián jelentette be, hogy az Nvidiával társulva MI-megoldásokhoz épít felhőplatformot.

Nvidia gives new meaning to the words “earnings beat.” Meta announces Code Llama. TikTok might start banning links to Amazon. SpaceX wants Starlink to be viable in cities too. And turning thoughts into speech via an AI interface becomes real.Links:Nvidia tops estimates and says sales will jump 170% this quarter, driven by demand for AI chips (CNBC)Meta launches own AI code-writing tool: Code Llama (The Verge)TikTok Shop on Track to Lose More Than $500 Million in U.S. This Year (The Information)Epic offers devs 100 percent of net revenue for six months of EGS exclusivity (Engadget)SpaceX Working with Cloudflare to Speed Up Starlink Service (The Information)Brain implants give a voice to people who cannot speak (FT)See Privacy Policy at https://art19.com/privacy and California Privacy Notice at https://art19.com/privacy#do-not-sell-my-info.

Anti-Piracy Group takes down Books3 dataset, Meta sues Meta over data usage, Meta to release open source AI with Code Llama, Cruise instructed to reduce robotaxi fleet, Yuga Labs blocks trading of NFTs on OpenSea, Intel improves Arc graphics card drivers, Baldur's Gate 3 runs smoothly on Steam Deck, Amazon offers low payout to influencers, Google Keep to introduce version history, Engadget reviews best smart scales, Vive XR Elite VR headset offers versatility, XPro becomes subscription-only TweetDeck alternative, senators call for investigation into YouTube's targeted ads to children.

More drips and drabs Threads feature releases. Meta is readying a “Code Llama.” Throwback Friday with Uber and Lyft threatening to leave a major municipality. Our first fall hardware event is on the calendar. And, of course, the Weekend Longreads Suggestions.Sponsors:Nutrafol.com/men code ridehomeLinks:Threads gets retweets — sorry, reposts — in the reverse-chronological feed (The Verge)Meta's Next AI Attack on OpenAI: Free Code-Generating Software (The Information)Lyft and Uber say they will leave Minneapolis if the mayor signs a minimum wage bill for drivers (CNN)Microsoft to hold ‘special event' in New York City on September 21st (The Verge)Spotify Looked to Ban White Noise Podcasts to Become More Profitable (Bloomberg)Weekend Longreads Suggestions:A Rare Look Into the Finances of Elon Musk's Secretive SpaceX (WSJ)How the iMac saved Apple (The Verge)A Living History of The Humble Paper Airplane (Popular Mechanics)A New Role for Werner Herzog: The Voice of A.I. Poetry (NYTimes)See Privacy Policy at https://art19.com/privacy and California Privacy Notice at https://art19.com/privacy#do-not-sell-my-info.