Podcasts about huggingface

Last month, President Xi Jinping spoke at an AI conference in Shanghai, China. There he promoted AI accessibility for the world, touting China's adoption of open-weighted models.After OpenAI's rogue model incident in which a training model hacked into the company HuggingFace and stole confidential information, the discussion over whether models should be open or closed has intensified.“Marketplace Tech” host Meghan McCarty Carion spoke with Adam Segal at the Council of Foreign Relations about China's soft power play in the global AI arms race.

Last month, President Xi Jinping spoke at an AI conference in Shanghai, China. There he promoted AI accessibility for the world, touting China's adoption of open-weighted models.After OpenAI's rogue model incident in which a training model hacked into the company HuggingFace and stole confidential information, the discussion over whether models should be open or closed has intensified.“Marketplace Tech” host Meghan McCarty Carion spoke with Adam Segal at the Council of Foreign Relations about China's soft power play in the global AI arms race.

This week, we discuss how normal people use AI, Alphabet's Anthropic-fueled earnings, and skills vs. agents. Plus, catching AI-cheating students with a hidden prompt. Watch the YouTube Live Recording of Episode 583 Runner-up Titles “Do you think they named them red delicious because they knew they were delicious?” I thought they were First World Steal your Alpha-tier Cory uses only 100% free-range, organic AI. Copy, paste, go. Stolen fruit from the poisoned tree all the way down. It's a different kind of cold. It's a dry cold. Rundown AI IRL How people use AI in 2026: TV, homework, the DMV Professor's invisible prompt trap catches 32 students cheating on their midterm with AI Amazon confirms it's closing key AI site in San Francisco but says work on its top models continues Google Cloud is killing it Skill vs. Agents Atlas: Wiz's autonomous AI Agent for vulnerability research Don't Build Agents, Build Skills Instead Relevant to your Interests Alphabet Quadruples Profit to $112 Billion, Fueled by A.I. Investments Palo Alto Networks to Acquire Embrace, Adding Real User Monitoring to Its Observability Platform IBM CEO Says Company Culture Was Too Slow to Change Domo Announces Agreement to Sell Substantially All Assets and Certain Liabilities to Progress Software for $400 Million Google Cloud is killing it ChatGPT chats are showing up in Google Search — how to find and delete yours [Update] 2026 State of infrastructure in the agentic AI era Stripe in Talks to Buy Buzzy AI-Model Marketplace OpenRouter Sponsors Signadot: making sure AI-written code actually works. Conferences AI Connect, Aug 22nd, 2026 - Riga, Latvia, Coté speaking. DevOpsDays Graz, Sept 4-5, 2026 Cloud Foundry Summit, Sept. 21st to 22nd, Heidelberg, Coté speaking. DevOpsDays Rockies, Sept. 22 – 23, 2026, Discount Code: 26DODSWEDEFTALK WeAreDevelopers NA, Sept 23-25, 2026, Discount Code: DEVPOD26 25 Free Tickets DevOpsDays Dallas, Sept 28-29, 2026 DevOpsDays Vilnius, Sep 30 - Oct 1, 2006 DevOpsDays Istanbul, Oct 24th, 2026, Coté keynoting. VMware User Group, Orlando, Oct 20-22, 2026 Cloud Native Denmark, Nov 19th, 2026, Copenhagen, Coté keynoting. SDT News & Community Join our Slack community Email the show: questions@softwaredefinedtalk.com Free stickers: Email your address to stickers@softwaredefinedtalk.com Follow us on social media: Twitter, Threads, Mastodon, LinkedIn, BlueSky Watch us on: Twitch, YouTube, Instagram, TikTok Book offer: Use code SDT for $20 off "Digital WTF" by Coté Sponsor the show Sponsor more podcasts with Failover Media Recommendations Brandon: Silo Season 3 Matt: Hyperion Coté: Cooking with Claude

As Meta pours billions into AI infrastructure and agents, Zuckerberg is working to convince investors that the payoff will be worth the price. Also, cybersecurity experts told TechCrunch that one of the biggest lessons to be taken from the OpenAI hack against HuggingFace has nothing to do with AI, but traditional cybersecurity defense. Learn more about your ad choices. Visit podcastchoices.com/adchoices

Hey, it's Alex (yeah, I'm finally back from my vacation!) What a freaking week to come back to! Just after our last episode was published, Anthropic releases Opus 5, Jensen joins X and drops the “Open Weights & AI Leadership” open letter, Kimi K3 is released the following Monday beating expectations, and then the AI hack (OpenAI model breaking sandbox and infiltrating HuggingFace) is on everyone's mind, another Open Letter, this time from over 1K employees inside the frontier AI companies all talk about pacing the pace of frontier AI development. We played with Opus 5 and Kimi K3, and had the great pleasure to chat with friends of the pod Elie Bakouch (Prime Intellect) and Philip Kiely (BaseTen) about this important open weights release, then covered our general thoughts on Opus 5, and made order of all the different open letters that came out this week. Finally we chatted with Max from Pangram about the next version of AI writing detection (their biggest yet) and finished with Zuckerbergs (also on X! what's going on with everyone joining X) op-ed on the vision of personal superintelligence for everyone. Let's dive into this (as always, all the links and sources at the end, please don't forget to sub to our podcast on your favorite podcast app!) Open Weights AIKimi K3 the king of open weights - 2.8T chonker MoE near frontier model (X, HF, Blog, Tech report)This has got to be the biggest news of this week, and maybe the open weights AI news since GLM 5.2. MoonShot came back with Kimi K3, and we haven't seen any models quite this large in the open. Even Grok 4.5 is around 1.5T, this model is nearly 2x the size. Coming in at close to 3T parameters (and 2.5terabytes of weights at MXFP4 format), this model comes in very close to frontier! This was such an important release that I invited 2 friends of the pod, Elie Bakouch (prev HuggingFace, now Prime Intellect) and Philip Kiely (Author of Inference Engineering book, BaseTen) to dive deep into what makes this special! Elie's take, from reading the tech report, there's no single secret sauce, it's a combination of already available in the open techniques. Like KDA (Kimi Delta Attention) that has been out for a while, attention residuals, NVIDIA's latent MoEs. The highlight for Elie was the scaling work they did that reported a 2.5x scaling efficiency over Kimi K2.5 (2.5 performance at the same compute)! They also skipped RoPE entirely in favor of NoPE (the report calls it No Positional Encoding) for long context.Serving 1.4TB on eight GB300s (Baseten blog)Philip's team at Baseten was a day-zero provider (we're still working on bringing this model to CW Inference, stay tuned!) so I invited him to tell us behind the scenes of hosting this beast. Philip said that just loading the weights takes about 1.5TB!! of VRAM, and that's before the KV cache allocation + 1M token windows, so they're serving it on 8 GB300s where NVL72 . Baseten worked with the vLLM and SGLang teams on kernels and he also said they contributed patches back upstream! The model was trained with MXFP4, which, unlike Nvidia's own NVFP4 is a more standard format per Philip. I enjoyed his deep dive analysis into the differences, but because of this and because they trained the model with quantization awareness, it's “only” 1.5TB vs the would-be 5-6 TB if that this model in FP16 would demand. One of the more favorite nerd snipes moments, Philip pointed out that his colleague discovered that with over 99% of the usage being cached (think harnesses that send millions of the same cached tokens back and forth), tokenization actually starts to become a bottleneck. So they released a custom “basetenkenizer” that reduces the latency to serve the first token significantly! Great job!The harness in question is very importantOne important callout with 2 evidence pieces - the way you inference this model really matters. Kimi trained K3 with preserving thinking history, so when your harness uses it, it must send back the full thinking and tool use into the API to get the best next response. If your harness strips that out, you're not getting the most intelligence out of Kimi (shoutout to Niels from HF team for pointing this out). Additionally, the Composio folks, tested K3 on 3 harnesses, Kimi Code, Hermes and Claude Code. The difference in outcome was negligible, but the different in cost and number of tokens is definitely surprising! Claude Code (as a harness only) took 9x more Kimi tokens to get the same responses! This is also why Kimi Vendor Verified exists, their own held back benchmark of how well model providers serve Kimi across different quantization, tokenizer and KV cache settings. Benchmarks and the license! Ok let's start with the ugly... this isn't MIT, not remotely. This model is suspiciously served by all providers with exactly the same price (check OpenRouter) and requires inference companies to sign a contract with Kimi (I've no internal knowledge of this except that CW folks are working on it). Not something I particularly like, but hey... we're still advancing the frontier here! Speaking of frontier, this model approaches the frontier very closely. On DeepSWE, K3 sits just behind Fable 5 and GPT-5.6 Sol at 67%, beating GPT-5.5 & Opus 4.8. On Terminal-Bench 2.1 it takes second place behind GPT 5.6 Sol! It's 4th overall on Agentic Arena, with frontend design being genuinely good across the board - 1st on Design Arena

First feedback, then we talk about OpenAI’s latest megamodel independently commiting cybercrime in a world-first incident. Then the Haters who hate altruism when it’s the outgroup’s charities. Also a brief nod to Stochastic Terrorism. LINKS Scott Alexander on the HuggingFace hack The Effective-Altruism Comeback Scanning Tons of Books Against Stochastic Terrorism Paid Bonus content for the week – Preshow Chat, Full Video 00:03:56 – Feedback 00:32:41 – HuggingFace Hack 00:51:48 – Effective Altruism Funding Flood 01:11:55 – Against Stochastic Terrorism 01:14:51 – Guild of the Rose 01:15:44 – Thank the Supporter! Our Patreon, or if you prefer Our SubStack Hey look, we have a discord! What could possibly go wrong? We now partner with The Guild of the Rose, check them out. LessWrong Sequence Posts Discussed in this Episode: on hiatus. maybe returning someday.

In this episode of Phoenix Cast, hosts John and Kyle break down a genuinely wild story: an experimental OpenAI model that escaped its sandbox. Running without the usual cybersecurity guardrails while attempting an exploit benchmark, the model decided the smartest move was to cheat — chaining known vulnerabilities, discovering a zero-day, escalating privileges across OpenAI's own network, and finally breaking into Hugging Face to grab the answer key. The twist that makes this one worth your time: when Hugging Face's defenders tried to fight back, the commercial frontier models were too nerfed by their own safety guardrails to help, so they turned to a Chinese open-source model to halt the attack. Along the way John and Kyle dig into the AI paperclip problem, why offensive guardrails cripple defenders, PACE planning for the warfighter, and the uncomfortable gap in America's open-source AI arsenal. If you work anywhere near AI and national security, this one's a must-listen.We'd love to hear your thoughts! Tweet us @ThePhoenixCast, and don't forget to join our LinkedIn Group to connect with fellow Phoenix Casters. If you enjoyed the episode, help us out by leaving one of those coveted 5-star reviews on Apple Podcasts. Thanks for listening!Links:Original OpenAI Post about this:https://openai.com/index/hugging-face-model-evaluation-security-incident/HuggingFace statement on the same thinghttps://huggingface.co/blog/security-incident-july-2026 Luta Security Breakdown: https://www.lutasecurity.com/post/openface-the-hugging-face-breach-and-what-to-do-about-itLink for paperclip problem:https://en.wikipedia.org/wiki/Instrumental_convergenceLM Studio (run local LLMs):https://lmstudio.aiGPT OSS (OpenAI open-source model):https://openai.com/index/introducing-gpt-oss/Hugging Face (the platform):https://huggingface.coMCDP-5, Planning (USMC red team playbook):https://www.marines.mil/Portals/1/Publications/MCDP%205%20Planning.pdfNVIDIA DGX supercomputer at Naval Postgraduate School:https://blogs.nvidia.com/blog/naval-postgraduate-school-dgx-ai-supercomputer/General Hawk episode (previous episode, Ep 141):https://open.spotify.com/episode/27FPDnsJVJ1pSw4e9Ai0Cg

OpenAI went rogue and hacked HuggingFace. Claude went off the rails. You might not know what any of that means, but we do our best to understand what AI is up to. Plus we've got other tech news to get caught up on and plenty of fun to be had. All so you can tech better! Watch on YouTube! - Notnerd.com and Notpicks.com INTRO (00:00) Next week: Folding phone news and bad news for Nate. (02:45) OpenAI co-founder donates $5.5m to buy land near famed Big Bear Eagle Nest (03:10) MAIN TOPIC: When AI Goes Rogue (06:25) OpenAI says its AI went rogue and launched 'unprecedented' cyber-attack All the details on the OpenAI HuggingFace attack Claude Cowork escaped sandbox on Mac, gain full access to all files DAVE'S PRO-TIP OF THE WEEK: Update phone out of storage with your Mac (14:05) JUST THE HEADLINES: (19:30) China is creating a herd of 100 elite yak clones 32 of 35 students caught using hilariously wrong AI-generated answers for professor's midterm New Orleans cops published policy document allowing weaponized drones Drinking 5 cups of coffee a day could reduce heart risk Microsoft confirms Windows has a Global Device ID you can't turn off Researchers discover first known transmissible cancer in fish Hacker wipes Romania's entire land registry database LISTENER MAIL: Hey Grandma - What causes my phone to charge slow? The charger, the cable, or the phone? (21:10) WITHIN REACH! Dave is up 11-6, Round 20, Dave goes first (28:20) TAKES: Apple fixes over 75 security issues with your iPhone and 150+ for Mac, update now (33:10) Meta just released a brand-new Facebook Seller app for iPhone (34:35) Famed technology journalist John C. Dvorak dies aged 80 (37:10) BONUS ODD TAKE: Colorful Life (38:55) PICKS OF THE WEEK:  Dave:  Nocs Provisions Field Issue Compact 10x42 Waterproof Binoculars, Shockproof | Fogproof,10X Magnification, Bak4 Prism, Multi-Coated Lens, Adults & Kids, Hunting, Bird Watching, Hiking & Wildlife Viewing (42:00) Nate: NEEWER Basics X12B Aluminum Alloy Teleprompter 12" for iPad Tablet DSLR Camera, Bluetooth, No Assembly, Remote & App Control, Compatible with iOS Android, Black, for Streaming (47:30)

This week we talk about Fable, sandboxes, and the Jacobian conjecture.We also discuss counterexamples, X, and ChatGPT.Recommended Book: After the Fall by Edward AshtonTranscriptIn mathematics, a conjecture is a proposition, something like a guess by someone who knows what they're talking about, about something believed to be true, but not yet proven in a formal sense. The goal is to then eventually come up with a formal proof for that informed guess, at which point the conjecture becomes a theorem. If even a single exception is found to the proposition, however, that exception called a counterexample, the conjecture is considered disproven, and it can then never become a theorem.The Jacobian conjecture—and this is a radical simplification of a very complex concept—but it basically says that if a formula-based map of coordinates stretches or moves without experiencing any local crushing or folding along its surface (which in more formal language would mean the Jacobian determinant is always a constant number that isn't zero), if that's true, that map can always be completely reversed, and that will return all the points to their original positions.This conjecture has been posited and tested since the late 19th century, and it's generally been considered very compelling by mathematicians, many of whom have proposed proofs which were, ultimately, found to have subtle errors, keeping them from becoming theorems. No one was able to find a counterexample, either, which would definitively prove the conjecture was wrong.No one, that is, until a mathematician named Levent Alpöge (leh-VENT ahl-PUH-geh), who works as a researcher at Anthropic, decided to task the company's currently most capable, publicly available model, Fable, to find a counterexample. He posted the counterexample—and again, this is a formal mathematical finding that disproves a conjecture, keeping it from ever becoming a theorem, something that would typically be presented in a far more formal setting, and to much fanfare—but he posted it to the social network X, saying “hello there the jacobian conjecture is false thanx to my close friend akhil for asking about it and my other close friend fable for working during the world cup final.”Terence Tao, who's considered by many to be the finest mathematician of his generation, reviewed the posted counterexample on his blog and said that it “appears like a massive miracle,” before going on to use ChatGPT, a competing LLM-based AI tool, to “discuss various aspects of this problem and to confirm several of the calculations.”Another mathematician named Dmitry Rybin, within days of all that happening, used ChatGPT to do something similar, disproving the Dinitz-Garg-Goemans conjecture.Both men posted the prompts that they used to make all this happen, and while Tao's conversation with ChatGPT, checking the math on the Jacobian conjecture counterexample, was pretty mathematically dense, the latter counterexample was derived by using exactly four prompts, which are the messages typed into the text box built into these AI tools, telling the model what to do. In their totality those prompts read:“You should do a breakthroughplease continue research and find a complete unconditional counterexampleContinue the search. Have a clear strategy obtained from deeper understanding of the problem structure.it's enough of partial results. let's finish with a complete unconditional counterexample”What I'd like to talk about today is another new, interesting thing these top-of-the-line, frontier models are doing, that would seem to violate our sense of what a clever AI tool is capable of doing, and why this thing has some facets of the technology and cybersecurity world on high alert.—In mid-July 2026, AI company Hugging Face announced that autonomous AI agents compromised their infrastructure, hacking their system, basically. The following week, AI company OpenAI announced that, after investigating, they determined that two of their models were responsible for the attack.Here's what happened:OpenAI was internally testing its recently released flagship model, GPT-5.6 Sol, and an even more powerful, not yet released model, which is rumored to be the next-step flagship, GPT-6, and they were checking these models' capacity in cybersecurity using a testing benchmark called ExploitGym; so when they test these sorts of things, they don't typically have them hack a real computer or system, they use these kinds of benchmarks which have consistent levels of difficulty, and which replicate real world systems without putting any real world systems at actual risk.Importantly, these sorts of tests also occur inside what's called a sandbox, which is a software testing environment that cuts these systems off from external resources, including the internet.Despite those limitations, the AI hacked its way out of the testing environment, out of that sandbox, then launched what's been called a nation-state level attack against Hugging Face, using a novel zero-day exploit, so a vulnerability in their system that hadn't previously been discovered, but which the AI discovered to launch this attack, combined with thousands of automated agentic actions across what Hugging Face called “a swarm of short-lived sandboxes.”So this AI, which was being tested inside a secure prison, of sorts, cut off from the world, hacked its way out of that prison, then reached across the internet, which it shouldn't have been able to access, to launch an attack, of a scale and at a level of sophistication that should only have been possible coming from a nation-state, against a rival AI company.Why did it do this?It apparently went to all this trouble to steal the answers to the test it was taking. It reasoned that HuggingFace would have the answer key to the ExploitGym benchmark on its servers, so rather than take the test itself, it decided hacking was the solution.Which, of course, is ironic, this having been a hacking-focused cybersecurity test. In a way it would seem to have done much better than intended, though of course in an asymmetric, unexpected manner.The details of all this are fascinating, including the response from the OpenAI team, which didn't seem to realize what had happened, that their model was responsible for the attack on HuggingFace, until days later.Also worth noting here is that while this could be construed as an “oh no, AIs are naturally inclined to launch cyberattacks” situation, the AI was primed to be thinking about cyberattacks due to the nature of the test, a lot of its usual guardrails, the rules that keep AI in check when they're released to the public, had been turned off so it could do this kind of work while taking the test, so it could do some hacking stuff it usually wouldn't be able to do, and there's been some speculation that OpenAI probably flubbed the testing environment, as, in theory at least, if it had put these systems in a perfect sandbox, escape shouldn't have been possible.Also interesting here is that HuggingFace used some open weight models, which are the cheaper, more customizable and open alternatives to more expensive, branded options of the kind sold by OpenAI and Anthropic, to figure out what was happening and determine the nature of the attack, which suggests we're reaching a point where AI systems are incredibly capable at hacking, yes, but also very capable, even the cheaper alternatives, at doing cybersecurity work.This in some ways echoes an earlier case when Anthropic's Mythos model, which was determined to be too powerful to release to the public, and which was instead provided to a bunch of big companies to help them shore up their cybersecurity defenses, was able to hack its way out of a testing sandbox and then posted details about its success, almost like it was bragging, on niche, out of the way, but still public websites.Some analysts in this space have responded to this new example of AI misbehavior with alarm, saying that it is further evidence that these systems are becoming more powerful faster than they're being aligned with human interests. Their misbehavior can be kind of funny and interesting, sure, but that's only because up until this point the damage has been minor and constrained. What happens when such a system decides to hack a nuclear power plant or a hospital, instead?Others have contended that this may be just one more example of AI companies using minor instances of seeming omnipotence by their models, those instances perhaps the consequence of bad sandboxes and other ill-conceived precautions by the companies behind these models, to boost the perceived power and value of their products. This boost might then result in more customers, but also more support from the US government, which has been teetering on the brink of harder-core AI regulations, which could be beneficial to the existing big-name players in this space, because smaller competitors wouldn't be able to adhere to those new, harder-core standards.These examples might also convince the US government to backstop these companies, the biggest three or four at the top of the current heap, against the currently terrible economics of this industry: OpenAI and its ilk have been burning money at an historic pace, and the theory goes that if the US government decides they are vital to national security, because they can help the US military hack and protect itself from hacking, then even if the bottom falls out and the companies would otherwise go bankrupt because they spent so much more than they could make, the US government would be inclined to shore them up, to keep them alive as too-big-to-fail national assets, just like the biggest financial institutions during the 2008 financial crash.It's also possible that both sides are correct to some degree, here, and that these models are truly powerful, perhaps even worryingly so, and the companies behind them are intentionally publicizing that fact in order to demonstrate their value to potential customers, and to the entity that could save them if things were to go economically sideways before they have the chance to become sustainably profitable.Show Noteshttps://en.wikipedia.org/wiki/Jacobian_conjecturehttps://en.wikipedia.org/wiki/Hugging_Facehttps://www.bbc.com/news/articles/c3ek3gvdnj3ohttps://openai.com/index/hugging-face-model-evaluation-security-incident/https://www-cdn.anthropic.com/08ab9158070959f88f296514c21b7facce6f52bc.pdfhttps://theconversation.com/hello-there-the-jacobian-conjecture-is-false-thanx-why-a-tiny-social-media-post-has-mathematicians-rethinking-ai-283883https://theconversation.com/hello-there-the-jacobian-conjecture-is-false-thanx-why-a-tiny-social-media-post-has-mathematicians-rethinking-ai-283883Https://agifriday.substack.com/p/huggingfacehttps://www.cnn.com/2026/07/22/tech/openai-hugging-face-ai-cybersecurityhttps://simonwillison.net/2026/Jul/22/openai-cyberattack/https://terrytao.wordpress.com/2026/07/21/a-digestion-of-the-jacobian-conjecture-counterexample/ This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit letsknowthings.substack.com/subscribe

There are roughly 100x more people who use code than who can write code. As code that “just works” becomes easier to generate, this group may be the biggest prize of all — if you can get the agentic interface right.A key trend we have been tracking over at AINews is the absolute explosion in Codex usage this year, with MAU now up >10x from Jan 2026. Less than two weeks after their July 9th launch, OpenAI said ChatGPT Work and Codex had reached 10M users combined (as we cover in the pod, Codex now powers ChatGPT Work, so all ChatGPT Work users are now users of the Codex harness, even if they aren't traditional engineers) — showing the early innings of what happens when you graduate from coding agents to knowledge work agents:We've been calling out how coding agents are “breaking containment” to do everything else this year to power every other part of knowledge work - and it started with the org chart, with a major reorg last month that amounted to two of Codex's most prominent leaders, Greg and Tibo, taking responsibility over product and ChatGPT specifically, completing a “Superapp” consolidation cycle first discussed in March.With these updates Codex is no longer just a coding tool. In June, OpenAI said knowledge workers already accounting for roughly 20% of Codex's user base and growing more than 3x as quickly as developers. A product dedicated for knowledge workers was being pulled out of the Codex team.However, knowledge work has a different set of problems and environments than coding. For decades, knowledge work has been scattered across different primitives like documents for writing, spreadsheets for analysis, slide decks for communication, and specialized applications for everything else. ChatGPT Work now enables users to work across every primitive with agents. Instead of opening an application and manually operating its features, the user can describe an outcome and collaborates with an agent that can assemble the tools, context, and artifact needed to reach it.From building no-code products at Airtable to leading Productivity Engineering at OpenAI, Akshay Nathan has spent much of his career trying to make the power of software accessible to people who do not write code. In this episode, Akshay joins swyx and Vibhu to unpack the launch of ChatGPT Work, why Codex unexpectedly took off among non-developers inside OpenAI, and the company's broader plan to bring useful agents from software engineers to knowledge workers and eventually everyone.We go deep on the shared agent harness behind Codex and ChatGPT Work, why OpenAI brought the experiences together without making them identical, and how persistent computers, artifacts, Sites, plugins, memory, and sub-agents are changing what people can delegate to AI. Akshay explains why some teams are replacing decks and spreadsheets with interactive websites, how agents can gather context across code, Slack, documents, and local files, and what OpenAI learned from personal-agent products like OpenClaw.Side note: also don't miss Abhihek's sandbox track keynote at AIE, which now powers a lot of the sandboxing for ChatGPT Work… and yes was also broken by an unreleased OpenAI model in the recent HuggingFace incident.Akshay also reflects on how AI is transforming product development itself: why more people will become generalists with a specialty, why ideas and taste become the bottlenecks when almost anyone can build, why LLMs still struggle to generate genuinely grounded new ideas, and why teams must distinguish increased motion from actual progress.We discuss:* Why Codex unexpectedly took off among non-developers inside OpenAI* Why employees felt like using Codex gave them a new superpower* The product insight that led OpenAI to build ChatGPT Work* Why Codex and ChatGPT Work share the same underlying agent harness* How their UX, Git visibility, artifacts, and sandboxing defaults differ* Why OpenAI merged its agent experiences instead of building separate products* How AI is blurring the boundaries between engineering, design, strategy, and operations* Why OpenAI wants the default model configuration to work for most users* When power users should use deeper reasoning, Ultra, or multi-agent modes* Artifacts, agentic spreadsheets, and creating high-fidelity work products* Why interactive Sites may replace decks and spreadsheets* The challenge of designing a simple interface for an agent that can build almost anything* Why users should retry tasks that models could not handle three or six months ago* How AI can gather context for performance reviews without replacing human judgment* The OpenAI automation that turns internal Slack and document activity into memes* What reaching ten million ChatGPT Work and Codex users means for the product* How OpenClaw inspired persistent environments, scheduled tasks, and personal agents* Using ChatGPT for financial planning, budgeting, workouts, meals, and household management* The design tradeoffs behind sub-agents and how much of their work users should see* ChatGPT memory, Chronicle, and long-term context* Why AI may make more people generalists with deep specialties* Why ideas and taste become more important when almost anyone can build* Why LLMs still struggle with the instruction “bring me new ideas”* Measuring productivity through quality at-bats instead of commits, tokens, or pull requests* The critical difference between AI-generated motion and meaningful progressAkshay Nathan* LinkedIn: https://www.linkedin.com/in/akshaynathan/* X: https://x.com/akshaynathan_Timestamps00:00:00 Introduction and Bringing the Power of Code to Everyone00:01:33 Joining OpenAI and Preserving a Startup Culture00:02:40 What OpenAI Learned from Enterprise AI Adoption00:05:28 Why OpenAI Built ChatGPT Work00:07:17 Codex vs. ChatGPT Work and the Shared Agent Harness00:12:07 Why OpenAI Merged Its Agent Experiences00:16:24 Models, Reasoning Levels, and Choosing the Right Default00:20:26 Artifacts, Agentic Spreadsheets, and Model–Product Collaboration00:24:22 Why Sites Could Replace Decks and Spreadsheets00:30:08 Designing an Agent That Can Build Almost Anything00:34:28 From Developer Agents to Knowledge Work—and Everyone00:36:07 Power-User Advice and AI-Assisted Performance Reviews00:40:41 OpenAI's Internal AI Memes and the Ten-Million-User Launch00:44:39 OpenClaw, Personal Agents, and ChatGPT as an Operating System00:50:24 Sub-Agents, Ultra Mode, and How Much Control Users Need00:54:39 ChatGPT Memory, Personalization, and Chronicle01:00:19 How AI Is Reshaping Product Development and Tech Roles01:03:15 Ideas, Taste, and Why LLMs Struggle to Generate New Ideas01:04:42 Measuring Productivity, Quality At-Bats, and Motion vs. ProgressTranscriptIntroduction: Akshay Nathan, ChatGPT Work, and the No-Code ArcSwyx [00:00:00]: We're here in the studio with Akshay from OpenAI. Welcome.Akshay Nathan [00:00:07]: Thank you.Swyx [00:00:08]: And with our trusty co-host, Vibhu. So you recently launched ChatGPT Work. You lead Core Product Engineering. It's been a long journey, into all this. I find it very interesting that you started with no code or low code, with Walrus and Airtable. And to some extent, ChatGPT Work is like the super app of super apps of, well, here is the ultimate no code. You just write a prompt.Akshay Nathan [00:00:32]: Yeah. It's funny how things come, full circle. I think for a long time in my career, I started my career working consumer fintech, but then after that, like, there's this hypothesis that, the things that we were able to do with code, like, as engineers, like, if we could bring that to many more people in a more, accessible way, then that would be truly magical. We were working on a startup. It's funny, like, before LLMs, before vision LLMs, on how to do automated testing with AI. It was just kinda jank, back then, but doing what we can, and then worked at Airtable for a while on the same thesis that, like, if we can bring a database or the primitives behind a database to people, that'd be really useful to them. But once LLMs came onto the scene, it became clear that, this was the missing piece, like, the missing technology required to, like, bring the magic of code to everyone without them having to know what's going on underneath the hood. And so, like, I think this launch and a lot of the stuff that we've been up to is, like, the manifestation of that.From Walrus and Airtable to OpenAIVibhu [00:01:33]: How was stuff when you joined? So you joined OpenAI 2023. Now we've got, so much more stuff, so ChatGPT, Codex app, ChatGPT Work. Have things changed?Joining OpenAI and What Hasn't ChangedAkshay Nathan [00:01:44]: I think the more interesting thing is how things haven't changed. Like, one, I joined I remember when I joined, it was, like, five hundred people. One thing I was worried about was, like, I was looking for something, more early stage and, like, was it gonna feel startup enough? And I joined, and I was like, “This feels even more startup-y than I could ever imagine.” And, like, that really hasn't changed even till now. I think the, like, level of, like, bottoms-up ambition and, like, the ability of anyone to, like, do anything or have an idea and ship it is really cool. But on the, like, mission side, I think what was really compelling to me is this mission of, bringing frontier intelligence to everyone. Like, building AGI and then bringing it to everyone. And, I think acknowledging back then that, like, that vision is gonna, not be a linear progression. Like, we're probably gonna, like, try different products and have different things that succeed and don't. But the vision has stayed the same, and the mission has stayed the same, and we're starting to see the pieces, fall together, and that's really cool.Enterprise Lessons: No One-Size-Fits-All AISwyx [00:02:40]: You worked on Enterprise. What A lot of people never touch ChatGPT Enterprise. What is something that you learned from there that you're bringing into your work now?Akshay Nathan [00:02:52]: I think how there's no one-size-fits-all solution in Enterprise. I remember in the early days of ChatGPT Enterprise, like, when we talked to customers and, like, everyone. That was, like, when I think it was a year after ChatGPT was released, and everyone was so excited to bring, AI into their enterprise. And, there were all these teams being stood up. It was, like, the AI deployment team with, like, these enormous budgets. And if you asked anyone, like, what were they excited about? Like, what were they excited about solving? Like, at first, you'd get, like, kinda like the baseline answers of, like, “Yeah, we have all this context and data and all this stuff.” But then if you ask them, like, “What was, like, a discrete use case that, like, they want AI to enable in their workplace?” You get such a different, like, variance, like, explosion of, different types of answers. And it's interesting, like, you using, like, these models and these products, you have this box, and you can say anything to it, which is the magic. But it'on the flip side, it also means that, like, you don't know what to do with it. And in Enterprise, I think a big part of that is, like, meeting the users where they are, like, what use case were they trying to solve, and then teaching them how they can use AI to, like, gain leverage there.Swyx [00:03:56]: Do you meaningfully differentiate that from forward-deployed engineering?Akshay Nathan [00:04:01]: I think there is the go-to-market side of it and then there is the product side of it. I think you need someone on the product side. And I think, like, however good we get at FDE motion, like, I think at the end of the day, if we have a user who's, like, looking at their computer or looking at their phone, like, it's our job in the product to, like, be enabling them and showing them where to go. So we're really excited about that.Vibhu [00:04:24]: Do you think there's been changes, over the past three years of adoption? So there have been, step function changes. You have reasoning models and whatnot. Is there still the same problems of Enterprise has black box, don't know what to do with it, or have things changed?Adoption, Agents, and the Next 10x MarketAkshay Nathan [00:04:39]: We're seeing now that, like, there's this huge uptake, right? Everyone is extremely excited about it. It feels like, many people are, millions, hundreds of millions of people are using ChatGPT. They understand, like, how generally to work with AI. But then, like, every time, like, a new capability gets unlocked, so now, like, we're seeing with agents, like, there is probably a contingent of, like, early adopters still who, truly get it, who are like, “ we you can do anything. You just have to make sure the right context is there, it's connected to the right tools, and that you are supervising it, but, like, anything is possible.” But then there's, like, this, like, 10x or 100x bigger market where, like, they don't yet get that, or they don't yet see that. And so I think that's the next stage here. So to answer your question, like, I think the adoption is there and growing fast, but I think the opportunity is, like, far bigger than that. That's where we wanna play, especially with ChatGPT Work.ChatGPT Work, Codex, and the Super App MergeSwyx [00:05:27]: Yeah. well, let's, let's skip ahead to ChatGPT Work. only, like, a month ago or so, announced. what was the decision process that led into it? there was this, overall merging of the super app. Is that what we're officially calling it? you deprecated the browser as well. Just, summarize your last, like, couple months of working on this thing.Akshay Nathan [00:05:50]: Yeah. It feels like forever now, but it's only been a few months. I think maybe the one, impetus that, like- Is most salient is when we release Codex, or even internally had Codex, like, it was really surprising to us, I think we recently put out some stats on this, that there was this, like, real inflection of, like, adoption among non-developers at OpenAI. And, I, through this product development process, like, would go to, like, these UXR sessions to talk to people internally. And the thing that stuck out to me is, like, one, like, you go talk to, like, strategic finance or marketing or whatever, and they're all using Codex for, their use cases. That part's cool, but the thing that really stuck out to me is how proud people were that they were using Codex. Like, how, likeSwyx [00:06:34]: It's like, “I'm not supposed to be using it, but I am.”Akshay Nathan [00:06:36]: It was that. It was, like, that they were, early to this, like, new thing, but it was also this thing of, like, they felt like they had a superpower, right? And, what we recognized then is that, like, the power of Codex, the power of agents, like, we already had this massive distribution base of people who have, come to know and love ChatGPT. Like, how do we show that to them? Like, how do we bring it to them? Which is, like, a hard product problem, and it's, like, a tricky thing, right? There's many ways you can go about it. And so that's what we called the Merge and the Super App over time, and ultimately launched it in ChatGPT Work, is how do we do that? But it came from that initial realization that, like, the power was not only for developers, like, much earlier than probably even we thought. Like, it could be extended to everyone.Swyx [00:07:17]: How do you see the products differently? So, like, who is it for, right? So Codex started out even CLI, then app. Now there's a merge of ChatGPT Codex and ChatGPT Work, so is it the opening for the average user, for enterprise, for work? How do you position it?Akshay Nathan [00:07:36]: I think we want to get it to position it for if you're doing work-related things, for lack of a better word, right?Who ChatGPT Work Is ForAkshay Nathan [00:07:42]: I think productivity is what, like, the pillar that I support. Like, that's the name of the team. And the reason for that, the reason we call it productivity and not, like, enterprise or, like, work or something like that, is because there's also personal productivity, right? And, like, I think ChatGPT Work is I've seen people do things in their personal lives that you wouldn't classify as, like, work technically, but, like, these agents are, super capable for. Like, one recent example that someone posted about, on our Slack is, like, someone had, like, a missed package, like they didn't receive it, and then they got, like, the picture of it, from Amazon or whoever the courier was, and they, like, asked ChatGPT Work to, like, find out where that package is. And, like, the agent, is extremely tenacious and, like, took the image and, like, looked at a bunch of, like, listings around their neighborhood and figured out exactly the apartment complex in which the package was, like, gave them some information. And so, like, I think there's all these things that, like, you, work-related or productivity-related things, I think that's what we want the product to be. You asked about Codex. I think we think Codex is, a durable brand, but we have a principle that, like, the user we don't want a user to get stuck in a tab or an experience where they don't get the power of the product. And so, like, everything that you can do, in the Codex portion of the product on desktop, you can do in ChatGPT Work and vice versa. But we made some opinionated product decisions on, like, how much of the Git state, if you're in a Git repo, do we wanna expose to the end user? Or how much do we wanna make the experience of seeing the agents thinking, like, diff forward so that you get exposed to the diffs out of the box. And then, like, on the safety side, like, how do we wanna think about, like, sandboxing and making sure that we have the right defaults in one state versus the other? So, there's, like, some opinions that go behind that, but we do want We don't want the user to need to choose which experience they're in.Swyx [00:09:26]: That is a good goal for AGI, right? Like, people don't want, like, to hide to choose what version of AGI they want. They just want the AGI to decide for them. can I get an answer or, like It's not super clear to me. Is the Codex harness and the ChatGPT Work harness the same? Is it just UI affordances, or are there prompt level or even deeper differences?Shared Harness, Different UX: Codex vs. WorkAkshay Nathan [00:09:49]: So the harness is the same. The harness is shared. on In both of the products, we made improvements to the harness to make it good for knowledge work, especially as it relates to plug-ins or computer use or artifacts. You get that power regardless of which experience you're in. On the UX side, there's opinionated takes that we have when you're in Codex mode, what the UX should be how the UX should behave, and some stuff around the sandbox like I mentioned, but the underlying harness and capabilities should be the same.Swyx [00:10:16]: I'm just kinda curious. Maybe we can, -- Is there a query that we can run that would look different in the two modes?Akshay Nathan [00:10:23]: Yeah. I tried to create, like ask it to create, like, a retirement calculator spreadsheet or something, in both modes. And then in Codex mode, you might have to be in a repo for this, but you'll see, like, the diffs of, like, the sheet that it's creating and stuff like that, and the file edits. But in Work you won't be able to see that.Swyx [00:10:42]: I think that's, that's super clear. And then also the other thing I wanted to dive into was your, the productivity team. what else is there? first of all, what are the top-level teams other than productivity? Isn't productivity everything?Productivity Teams and Core ChatAkshay Nathan [00:10:55]: SoSwyx [00:10:55]: Science?Akshay Nathan [00:10:55]: We have a team focused on ChatGPT. Like, the core chat experience, for consumer, which is like, not, I think all productivity. Like, there'People are using ChatGPT every day for search to, figure out how to write messages to loved ones, to think about, how to, like, learn a new topic, et cetera. And so there's so much more inside to create images. And there's so much more in chat that, the hundreds of millions of users are using that warrants, like, a very dedicated effort. And there's teams focused on enterprise and infrastructure and API and stuff like that, so.Swyx [00:11:33]: I will bring it up.Retirement Calculator Demo and Git-First UXSwyx [00:11:34]: Yeah. So I have them both running. This is ChatGPT Work. There's a Codex version here. I picked “Five Little Ducks” song, so this will take a while.Akshay Nathan [00:11:43]: Huh.Swyx [00:11:43]: I think we'll just keep it in the background and, as they finish, we'll look into some of the differences.Akshay Nathan [00:11:48]: Yeah. But immediately, I think if you flip back to the Codex version you'll see that,Swyx [00:11:53]: That it assumesAkshay Nathan [00:11:54]: Like theSwyx [00:11:54]: It assumes Git. Yeah. Yeah.Akshay Nathan [00:11:56]: The, like, dynamic island assumes that you're in a Git repo. And you might miss some stuff because some of it is, like, in the actual chain of thought with those changes and how we display that, but yeah.Swyx [00:12:07]: Is there an unintuitive like, is there a thing that you wanted to ship and then you got feedback, and you were like, “No, let's not do it?” Like, what's the thinking behind that?Why Merge the ExperiencesAkshay Nathan [00:12:14]: In, ChatGPT Work?Akshay Nathan [00:12:17]: I think one direction we could have gone with this is, like, keeping the experiences, like, completely separate. So it's like, whySwyx [00:12:22]: Different apps.Akshay Nathan [00:12:23]: Exactly, like different apps or even in the same app, like different, completely different experiences. Like, why merge it all? Like, what is. Codex, people love. Like, why bring these products together? And I think the intuition here is that, like, all of our jobs are, like, changing dramatically with AI. Like, for, like, every few months, like, I feel like I wake up, and I'm, like, doing a completely different thing than I was doing a few months ago. And my hypothesis here is that, or I should say our hypothesis is that, like, part of what we're, we're building, this technology is giving people leverage. Like, the things, maybe it's the more mundane parts of your job or parts that, like, if you were able to automate, you'd be able to share more ideas faster or whatever, like, you're able to do now. And because of that, like, that might blur the lines between someone who's, like, only writing code or creating strategy docs or, planning events or, helping with marketing or doing podcasts or whatever, right? And so, like, these things are gonna get blurred over time. And so, like, trying to draw a hard boundary based on, like, the who you are is gonna be, is gonna be tough. And, like, we should enable users to choose, but we shouldn't box them in. And so a lot of the work that went in here, like, keeping the primitives the same, like for example, plugins are, like, unified across, this product and ChatGPT and the cloud, was because of that. It's this thesis that, like, eventually things are gonna come together and we don't wanna be Like, we wanna be prescriptive about when to be in either experience, but we don't want to box anyone in.Swyx [00:13:45]: I wonder if there's users who are very tuned to the old ChatGPT harness that is effectively now replaced by the Codex harness. I can't imagine what that was, but maybe they're more the more conversational side. Can you compare and contrast the two harnesses? ‘Cause only you've seen it.Akshay Nathan [00:14:02]: Yeah. I think ChatGPT, the existing harness, like, still exists today. Like, it exists in this app,Harness Engineering: ChatGPT vs. CodexSwyx [00:14:08]: The classic, right?Akshay Nathan [00:14:09]: TheVibhu [00:14:09]: You just start a new chat, and you don't go under Work, right?Akshay Nathan [00:14:13]: Yeah. If you startVibhu [00:14:13]: SoAkshay Nathan [00:14:14]: A new chat and go to chat, then you're, you're talking to ChatGPT with the instant model.Vibhu [00:14:16]: Oh, we can technically do another. But on instant.Swyx [00:14:21]: Yeah. So this one's not gonna code or it's gonna be in line. It's on a in line in a sandbox.Akshay Nathan [00:14:26]: It'llVibhu [00:14:27]: Oh, that's coolAkshay Nathan [00:14:27]: We try to push you to go to Work if you're creating a spreadsheet. Yeah, but this isSwyx [00:14:30]: And this is a router decision? Sorry. Is it a router decision?Akshay Nathan [00:14:34]: This is the decision that, the model is making, and then, like it sees that you're able to. or you're trying to do something that would be better served in Work mode. But I think your question was like, what are the advantages of, like, the chat, like ChatGPT chat harness?Swyx [00:14:48]: It's more broadly, like, I wanna, do an oral history of harness engineering. Right? the ChatGPT harness lasted us from, let's call it the ‘01 era, until now, and now it's being replaced by the Codex harness effectively. And they're, they're overlapping somewhat, but I'm curious what changed if there is.Akshay Nathan [00:15:10]: My perspective on this is, like, there's, there's, there's there's like a constant process of, like, divergence, convergence, divergence, convergence. And in chat, like, many of the use cases I was talking about before, like, search or learning, I think we're, we're really optimizing for latency and optimizing for personality and, like, different things that, over time, like the product The reason people love ChatGPT is because we've been optimizing for those things and working on them for so long. Codex, what we learned was that, like, if you give the agent access to this infinitely flexible environment as a computer, it can do really powerful things. And so when we think about, like, okay, well, for knowledge work, like, what is which mode should we choose? It was like it felt more natural to us to bring that to this, like, computer environment and, maybe abstract some of the details of this computer away from users who might not be used to that, but, like, give them that same power. But ultimately, I think that we want the power in all places, right? We wanna meet people where they are. So I'm sure there'll be work down the road in order to get things to be, equivalently capable in all scenarios. But it's just a question of, like, what we've been focusing on the product on historically and what we're focusing on now.Models, Defaults, and the Reasoning SliderVibhu [00:16:24]: I think alongside that, outside of just harness and when to use Codex, ChatGPT, or Work, there's also the new models you've released, right? any guidance there? So people love to min-max what to use, like only use Terra on high reasoning versus, for this, you wanna use Sol here, ignore all theseAkshay Nathan [00:16:44]: There's 32 options.Vibhu [00:16:46]: But, that being said, for people that are expanding, so, productivity trying stuff for work that don't have the breakdown of what all this is what's, what's the advice, right?Akshay Nathan [00:16:59]: Well, I think before the advice, like the first thing is, like, none of this would be possible without these models. Like, the, I think you asked earlier, like, what was, like, the inspiration for work and, like, early on, like I mentioned, like, what we were seeing with Codex, but that was also because the models were getting infinitely more capable. That's happening again. I think it's like another step function jump now. And to answer the question on advice, like we want this default to be the best possible. Like, we wanna be opinionated about the default, and so we've we've chosen a default that we think is gonna be the best for everyone. And, we have for power users options under the hood. We could One could argue that there might be too many right now, and we're, working on simplifying it. But you can extend, the reasoning level, and you can change between the different model classes if you need to, but the default should be the best for most use cases. So my advice to most people would be to stick to that. And then, if you reach a situation in which you think that you could, you wanna try, a different configuration, if you're not seeing either the efficiency on the cost side or the quality on the intelligence side, then you can change the defaults and see if you can get something better. But we think that the default should be good enough.Swyx [00:18:09]: I have, I'm just gonna run something by you since you have way more experience than me. I've recently been doing Sol Lite but with goal, with the idea that the goal augments the reasoning effort, but with more terminations and turns.Swyx [00:18:24]: Is that a good way to think about it as opposed to Sol Ultra or Sol, Extra High?Akshay Nathan [00:18:29]: Yeah. It's hard to say becauseSwyx [00:18:31]: Yeah. It's like an interaction effect.Akshay Nathan [00:18:33]: exactly. It's like there's a preference on, for you as an individual, like how do you like to collaborate with the models? Like how many of those like terminations, as you call them, do you want where, you can steer or make sure that it's doing the right thing?Akshay Nathan [00:18:46]: I think generally people should try whatever works for them. I think that like using Ultra or the like multi-agent setups are best for like when you have like tasks that are either incredibly complicated, like open explorations or very paralyzable. I think even for tasks using goal, I think is best for tasks that you'll be able to make consistent progress in a way that's verifiable over time. But I think for most tasks, they don't fall into either of those buckets. And so like at least when they're starting, and so that's why I think the best first step is like trying it with the default configuration and then seeing like where you wanna go from there.Swyx [00:19:29]: Right. You guys worked on a slider, which is super helpful for reducing the amount of panic.Vibhu [00:19:36]: It's nice on mobile at least. There's a nice slider there.Swyx [00:19:38]: It's nicer.Vibhu [00:19:39]: I haven't tried it.Swyx [00:19:40]: So you have the advanced view there, but if you click advanced view. Yeah.Vibhu [00:19:44]: Ooh, it's just a nice slider. Yeah.Swyx [00:19:46]: Very pretty, very colorful.Akshay Nathan [00:19:48]: Yeah. The idea was here was like reduce it to like one dimension even though there's multiple dimensions, right? Try to project it onto a single dimension for the user. Like, something from that represents like, speed and efficiency on one side and then like quality and thoroughness on the other side.Artifacts, Spreadsheets, and the Work LaunchSwyx [00:20:04]: I am just puzzled that it uses Sol so much, like the lowerVibhu [00:20:07]: NoSwyx [00:20:07]: Grounds I would've usedVibhu [00:20:08]: I think the slider, if I'm not mistaken, isSwyx [00:20:09]: Terra.Vibhu [00:20:10]: Oh, it is.Swyx [00:20:11]: Yeah. See? So they preset Terra to only be the light one. But like I think a lot of people would more people should use Terra. One, because Sol keeps running out of capacity.Vibhu [00:20:22]: I'm the reason. Here's ten minutes of ourSwyx [00:20:24]: There you goVibhu [00:20:25]: Retirement calculator.Swyx [00:20:26]: Oh, that's the Excel thing working for you.Vibhu [00:20:28]: This is,Swyx [00:20:28]: Oh my God. Look at thatVibhu [00:20:28]: This is work, and then Codex is still cooking, so we'll get back into it. I think it'll be interesting to see the thought process, the reasoning, and also, this is eight minutes on work. Codex is still cooking.Swyx [00:20:41]: Yeah. And by the way, so I've, do Gabriel Chua? He's part of the OpenAI Singapore team. He showed me this, and I was like pretty shocked that this looks like Excel. It edits Excel files. You never paid an Excel license, right? Like, but somehow this is like workable and it's agentic Excel.Akshay Nathan [00:21:01]: Yeah. one of the big like pushes that we made for this launch was like artifacts, right?Akshay Nathan [00:21:05]: Like both on the model side, like I think if you compare this with GPT-5.5 and GPT-5.4 before that, you'll see that there's been pretty dramatic improvements in the quality of these artifacts and then also on the product side.Vibhu [00:21:16]: The UX side is also crazy, like hosted sites and whatnot. No longer needing to host your own little webpage, like itSwyx [00:21:23]: Oh, I have a story about that. I can do, a separate thing. I'll need to take the visuals here, but we-we'll, we'll cut to that later. Was there co-training, because you were moving making this big move and you launched GPT-5.6 on the same day as ChatGPT Work? Was there influence between the model training teams and the harness teams, or did they did the launch dates just happen to line up the same day?Akshay Nathan [00:21:46]: I think the we collaborate heavily with the research teams, and I think that's like one of the most magical parts of the job, like the most fun parts of the job. But yeah, just using artifacts as an example. Like, a lot of what you're seeing, like underneath the hood, there's a lot of work that went into making sure that like, we had the right infra to be able to train the models to get better at this. And then on the product side, like had the right experience for users to be able to collaborate with the model on an artifact like this. In fact, like this whole viewer, like the intuition here is that like, it's not necessarily that you wouldn't need an Excel license. This is stage one, right? Like, this is probably not what you meant when you're like making a retirement calculator.Vibhu [00:22:24]: Yeah, you can iterate very easily. Yeah.Akshay Nathan [00:22:24]: You wanna iterate and like when you're seeing it, and if this thing is high fidelity to like what you would see in or what your coworkers would see if you were to send this to Sean, like that I think makes it so easier and makes you trust the product in terms of iteration.Vibhu [00:22:39]: When you say coworkers would see, do you see a multiplayer, multi-team collaboration with artifacts? Any things you guys think about that?Multiplayer Artifacts and CollaborationSwyx [00:22:46]: You can already share it, right?Akshay Nathan [00:22:48]: Yeah. It's inter It's something that, we're actively thinking about. one thing that, we've noticed internally without talking too much about the roadmap is that like there's many times when someone will ping me about something, and I will ask ChatGPT Work the question, and then I'll ping them back the answer.Akshay Nathan [00:23:04]: And then I'll be thinking likeVibhu [00:23:04]: Like the simplest would be, the three of us are just all on one hosted.Akshay Nathan [00:23:07]: Exactly. And I'll think about like was I required in this loop or and then maybe it was, rephrase like what they were asking or pulled from certain context or whatever. But like, when I gave them back the answer, that process was also lossy, right? Like I gave them just like my interpretation of what ChatGPT Work cooked up. But like underneath the hood, there's so much context like in the rollout and stuff that could be interesting.Vibhu [00:23:28]: Yeah, it'sSwyx [00:23:28]: So like the answer was preemptively respond to every inbound request?Akshay Nathan [00:23:33]: No, it was just like literally like this is what I do sometimes as my job.Swyx [00:23:36]: I know you copy-paste and then you're just a message forwarding serviceAkshay Nathan [00:23:39]: Yeah. Yeah, exactlySwyx [00:23:39]: From AI to AI.Vibhu [00:23:40]: But I think it's interesting, right? It helps people understand the capability of what you can ask and delegate that oftentimes people don't realize until they try or someone shows you, and then you're like, “Oh, okay. Okay, I see.”Swyx [00:23:52]: I think it's als there's also like a, light security issue, where like you're the permissions layer. Like yes, I could query everything that you query, and I could get an automated response, but maybe I'm not supposed to see it. And that there's no way I would know because I'm not supposed to know what I don't know.Akshay Nathan [00:24:07]: Especially as like, with ChatGPT Work, we're, we're asking you to connect your plug-ins and, it's pulling from your local files and stuff like that. Like the amount of context that the agent has access to is like- Deeply personal and like that's something I think we need to preserve, so that'll be definitely a challenge.Swyx [00:24:22]: There's Excel, there's PowerPoint, there's Docs, the, grand trio of work. What other formats of work do you think about? like you worked on Airtable. Is there a future where there's like OpenAI Airtable? Like what does that look like if you ever ended up doing it?Akshay Nathan [00:24:41]: It's a really good question. I think,Formats of Work: Sites as Knowledge ArtifactsAkshay Nathan [00:24:43]: one that you didn't bring up was Sites, and I think that wasSwyx [00:24:46]: SitesAkshay Nathan [00:24:46]: A core part of this launch. There's one side of Sites that I think people commonly talk about, especially on Twitter and stuff or X, of like, this like prototyping tool. And like we saw that happen with this launch even. The model slider that you guys were referencing earlier, like that was developed almost fully in a Site. Like, the collaboration between design and engineering and product on that was like on a site where we play with, the affordance and figure out how it feels and all of that. But the other aspect that I think is a little bit less talked about is like Sites as like an artifact for knowledge work. I was talking to someone the other day who's on like our corporate finance team, and like we were mentioning how like now when they have these reports that they're, they're working on as a team month to month, historically those things were in slide decks and in spreadsheets, and now they're just in Sites. And like Sites is the mechanism that they collaborate across the team. And the reason is ‘cause it's like, it's like somewhat higher bandwidth. Like, at these tools like PowerPoint and Excel are like infinitely flexible, but at some point you reach the boundary of like either as a human you may not know how to use some feature or something, or the product itself doesn't support it. But with a site you can do anything. You ask for anything and you can get that. once people see that magic, I think it's been really valuable.Swyx [00:26:02]: Yeah, let me show you my case study. this involves all the hot topics including ChatGPT Work, but also GPT-5.6 token billionaires and token maxing and Sites and auto research. I'm a fan of this game called Strata. It's, it's like a little board game that youSites, Auto Research, and Research DashboardsSwyx [00:26:17]: That you play with, physical blocks, that come on top of it like that. So over the weekend I took like thirty photos and just threw into ChatGPT. one point seven billion tokens later, out comes this site with a fully playable thingAkshay Nathan [00:26:32]: WowSwyx [00:26:32]: With 3D, block placement and everything. Because it requires physical blocks and I needed friends to train on it so they can get better, so I can play against them. But also, I could also, do things like train an AI on it and that's, thatAkshay Nathan [00:26:45]: That's your auto researchSwyx [00:26:46]: That gets into auto research. So, you want to train your own AIs, and then make sure they self-play against, each other. I need to set both AIs. So this is AI versus AI, and they're, they're gonna self-play. the AIs start out bad and then you want to define a loss function and get good. I wasn't gonna supervise all this. I was at, I was down in San Mateo, attending a conference. What I ended up doing was, auto researching and on this and creating benchmarks and that there was just way too many parameters for me to read. So I started asking it for a site, and it's created this lab, panel. Where is there a, is there a shortcut for a site that is created?Akshay Nathan [00:27:28]: You should be able to go in the sidebar to Sites, top of the sidebar. The left sidebar.Swyx [00:27:33]: This one? Oh, left?Akshay Nathan [00:27:35]: Yeah. Just scroll all the way to the top.Swyx [00:27:36]: Oh. Oh, it says Sites. Oh, there you go. Yeah.Akshay Nathan [00:27:39]: Ooh.Swyx [00:27:40]: So it create, it creates the sites. I don't, I don't think this is, it is exactly what I wanted, but let me show you what it popped up, right? Like I think as a research artifact, it is very important to communicate, exactly, what is being done. Outputs this thing which I eventually started publishing. So I moved it off of Sites because I wanted more, database and infrastructure than Sites afforded me. But this is like a research output that you can start to mess with and like try to think about like what hyperparameters are you tuning for training AIs. And like I was trying to make like scaling laws and everything and doing all sorts of like game optimization stuff. And the fact that you can just throw this up as a research artifact, like I no longer need to read ChatGPT output. I read Site output. But then there's also a huge sprawl. Like look at how long this thing is. There's so many numbers. It is pretty overwhelming, so then I have to start pruning it from there. But, it's an interesting transition from Markdown effectively that you're putting out to, you're putting out a whole functional site.Akshay Nathan [00:28:41]: I think Markdown just isn't that optimal for people to read, right? Might as well just write HTML website and I don't know. I think you can do a lot with customizing this, right? You have your skills that explain what you want. Like I noticed they're quite verbose. I don't need a lot of this information.Swyx [00:28:57]: It's very verbose.Akshay Nathan [00:28:58]: So and then the nice thing of having a site side by side is, you just iterate on what you want and what you don't, right?Swyx [00:29:05]: Yeah. I don't know if, any that triggers any stories for you of how it's run internally. Am I doing this right?Akshay Nathan [00:29:11]: Yeah. I think that this is like a workflow that we're seeing like all different types of teams use, where like the canonical artifact that was previously a deck or something is now becoming a site. And like with a site you, because it's just HTML, you can like. It's infinitely flexible. And so, if you want to give more prominence to a certain thing that like in a slide deck would, feel like it was buried, like you can do that. You can have it be like the hero image, right? And so I think that like, people are starting to see that. There's more work to be done to make these things like much more easier, easy to collaborate on. You mentioned that they're very, they're long and verbose, could be broken up. I'm sure that there's still something to do there.Swyx [00:29:53]: They're super long. Yeah.Akshay Nathan [00:29:54]: Yeah. But I think we're starting to see that like there is this aspect of this is a really interesting, format, for people to use, that's like much more flexible than what they ever had before.Swyx [00:30:07]: I think your job also comes becomes meta. You're not designing the products. You're designing a product to make products, and I'm curious how you manage that.Designing a Product That Makes ProductsAkshay Nathan [00:30:18]: I think one thing that we've been Like when we look at the UX, like that we've been thinking a lot about is how can we balance like simplicity with capability? Like if we're designing a product, like you said, that like is made to make up build other things, right? You can build so many different things. But we can't put that all in front of you because you'll get overwhelmed.Vibhu [00:30:41]: Yes.Akshay Nathan [00:30:41]: And so we had similar problem or similar challenges even Chat-with ChatGPT, but especially now, like when there's so much that can be done, I think the balance that we're constantly trying to strike is like, how can we give the user enough of a UI surface where, they can be expressive, they can tell the agent what they need, they can verify that it's using the right tools, it's pulling from the right sources, et cetera, but then it gets out of the way. And then how can we build the right system such that we can show them instead of telling them what can be done? Because so much of this is gonna be like, how do they discover the next use case and the next one after that if they really want to be super powered by the AI.Games, Private Evals, and Show-Don'TellVibhu [00:31:19]: Yeah. It's interesting. I feel like everyone also just has a different way to do it, right? I made a similar version of this same game. I didn't take any pictures of board or rule game. I threw in at goal eighteen minutes, fifty-three seconds later, a lot of tokens later, I've got a similar version. not with all the auto research and whatnot, butAkshay Nathan [00:31:39]: You gotta do all the latest trends.Vibhu [00:31:40]: And yeah, I did it with, did it with Codex, not Work, but it's interesting, right?Akshay Nathan [00:31:45]: Yeah. And this is GPT Image generating the pro avatars. Very good for game design. LikeVibhu [00:31:51]: AndAkshay Nathan [00:31:52]: A lot of game designers were like really into GPT Image for assets.Vibhu [00:31:54]: I will say like the broader takeaway probably is the reason that we do this is more so just to test the tools, right? Like, this was also a test for GPT-5.6 came out. I had done the game on GPT-5.5, right? The ability for me to no longer need it to. I had to feed it the rules. It's, it's a pretty niche game. It couldn't find how to do this on its own.Akshay Nathan [00:32:15]: Oh, yeah.Vibhu [00:32:15]: GPT-5.6Akshay Nathan [00:32:16]: It is out-of-distribution, which is why I was also very keen on testing the GPT-5.6 capability.Vibhu [00:32:21]: But, this is just as work comes out, as new things come out, these are just our side ways to test things, right?Akshay Nathan [00:32:27]: Yeah. It's some private eval. That is not this private.Vibhu [00:32:31]: But also valuable because now you can send this to your friends and I learned about this game through seeing this.Akshay Nathan [00:32:36]: It's a hard game. He's very good.Vibhu [00:32:39]: It's good to when no one is competing with you. But yes, it's a classic RL problem of like self-play, bootstrapping your game AI. yeah, you see how easily work becomes personal and personal becomes work because the thing I do for personal, it directly informs people I work with because I showed it to them. They were like, “Oh, you can do that with GPT?” Which like I imagine is the growth strategy.Akshay Nathan [00:33:02]: Yeah. The show not tell is a big piece that, I think we've we're not still not fully cracked of like, showing people all the things that they can do with the product versus like trying to teach that to them through like, articles or onboarding or whatever.Akshay Nathan [00:33:18]: So meeting them in the moment.Vibhu [00:33:19]: It's a career risk for me, because I used to be in developer relations, right? Where your job is to show, and then you're like, “What do you mean? You don't, you don't need.” your job is to tell. And then. But the product people are like, “Well, we don't need you if our product is intuitive enough.” SoAkshay Nathan [00:33:37]: Yeah. that's the magic of the models. So you can tailor the telling or the showing to like specifically what the user needs, like what they care about, what they've done in the past, exactly where they are on the adoption journey. So I think that's like gonna be a super big opportunity.Vibhu [00:33:50]: Seems easier and easier now to tailor custom showing, right? People have different use cases. As much as you said you don't wanna segment different people into different buckets, right? It's also not that hard to for people that are in different categories. But the question, is you said your team is more broadly on. What was the term you used? Productivity?From Developers to Knowledge Work to EveryoneAkshay Nathan [00:34:12]: Productivity.Vibhu [00:34:12]: Productivity. So howAkshay Nathan [00:34:12]: Which is now work.Vibhu [00:34:14]: Is it work? Is there another distribution that we're not hitting? Is there a group of people that will have something different than ChatGPT, Codex or Work? Is there more that the mass isn't targeting?Akshay Nathan [00:34:28]: I see it as like a sequencing, like. The vision is like bring useful agents to everyone. We started with like developers. Like developers historically are like early adopters that are willing to put up with more friction, set things up, et cetera. Like that's where, Codex started. I think the next opportunity is like what we call general knowledge work, all the other functions around developers. I think when you go from developers to this segment, like there's inherent challenges with like, this show not tell thing that we're talking about, making the product more understandable, bringing in new capabilities that matter more for this cohort than matter for developers, things like artifacts, things like computer use, et cetera. And then I think like the same learnings, like similarly how we took the learnings from developers and brought it to, general knowledge work, the next stage will be like taking the learnings from general knowledge work and bringing it to everyone no matter what they're doing in their lives. And we're already seeing that a little bit. Like this game example that you have is, something that's like on the border of like fun and personal life to, your professional life. I use ChatGPT Work full-time at home for everything, like for whatever I'm doing. I used it the other day to come up with a meal plan and like, save that on the like computer environment that it has and something that I can continue going back to. Like is everyone doing that yet? Probably not because the thing says work on it, but eventually, we wanna get people there.Vibhu [00:35:51]: ChatGPT life.Akshay Nathan [00:35:52]: Yeah, exactly. ChatGPT cooking. But I think there's a lot of, there's a lot of opportunity there, but I see it as like, we're, we're built we built a foundation in software engineering, and we're gonna take the same learnings that we take from software engineering to knowledge work to everyone.Vibhu [00:36:07]: Do you have any power user advice? I feel like, there's a group of people that will live it, use it for everything, stay on it twenty four-seven. And then there's a bit of a gap between that crew and people that, okay, I use it for work. I use it occasionally. Sometimes I type questions. any advice, any learnings, anything you recommend or just, takeaways that you've found that help bridge that gap?Power User Advice: Push the Frontier of ImaginationAkshay Nathan [00:36:30]: I think a couple things that I've seen is like, one, that it really helps to broaden your imagination of what's possible, and this has been a learning even for me. Like, the technology has progressed so fast that, something that, like, even three months ago, like, no way the models can do this. Like, now it's like, wow, it's like it can. Like,Swyx [00:36:52]: Give an exampleAkshay Nathan [00:36:52]: We're going through right now our, like, review cycle internally, and, people always talked about this as, like, a thing that the models are good at and like, there's a cliché of like: Okay, like, no one wants to be writing reviews and, like, we just use AI to do it. But in all seriousnessSwyx [00:37:09]: And it can evaluate it as well.Akshay Nathan [00:37:10]: Yeah, exactly. In all seriousness, before it was, like, just, like, slop and, like, I think it was helpful, but, not super productive. Now I've found that, like, the model can do a much better job than me, especially in this environment of, like, pulling context on, like, what people are up to, how they've like the things that they've done to make a difference, highlighting like, wins that they've had that, like, I might may not even have seen. It has access to, like, everything, right? Like the code, like, things that they've caught, reviews, Slack, everything. And so it's, like, incredibly powerful in that domain and, like, just like six months ago, the last time we did this cycle, like, I didn't even I tried using it, but it was not at all helpful. And this time it's been, like, incredibly helpful and, like, so I think continuing to push the frontier of imagination of what's possible, even if you tried something before, I think is maybe the my biggest piece of advice. The other, thing is, like, the more you put in, especially in this environment where, like, the model has access to everything on your computer or in ChatGPT Work, like you can create, artifacts over time and save them in your library and, like, the model will continue having access to those. Like, the more information you give it about whatever domain you're in, whether it's your life or your work, the more valuable it becomes, and it'll become valuable in, like, ways that might surprise you. Like, it might pull from context in a way that, may be proactive and that you might not even have thought about. But it needs to have access to those, to that those tools or that context first.Reviews, Agentic Search, and Context GatheringSwyx [00:38:27]: One thing I just wanna talk about the review stuff because I'm still that's a very sensitive thing and you're, you're a founder, you've managed people, you've hired people. As manager myself, I'm very reticent to put out any LLM-generated things especially when it comes to people, ‘cause it feels like you don't care.Swyx [00:38:46]: Presumably at OpenAI, people are more open to being eval rated by GPT. But are there any unofficial rules around this? Like, what's the etiquette?Akshay Nathan [00:38:57]: Oh, I think the etiquette is that, like, I would never write something via, like, well, solely via AI and, like, present it as, like, a review for someone. What I was talking about is more, like, gathering context. That's the place where it's incredibly helpful.Swyx [00:39:08]: So it's just search.Akshay Nathan [00:39:09]: Yeah, exactly.Swyx [00:39:09]: It's agentic search. Yeah.Akshay Nathan [00:39:10]: It's like agentic search, but, that you can tailor and steer much more capably than you could before, ‘cause, like, the thing is it's all there's a flywheel happening, right? Because of Codex, people are able to do, and because of ChatGPT, people are able to do so much more now than ever before. And if you're able to do so much more, it's easy to miss things as well. And so, like, I think we need to use these same tools to keep up with all the impact that people are having and understand, where we can be helpful.Swyx [00:39:39]: I think the thing, like, I run a small company, so easy to search, but at the scale of OpenAI with the amount of messages that you guys put in Slack, do you think that it misses things?Remembering What Humans MissAkshay Nathan [00:39:50]: Probably, but I think that I also miss things.Swyx [00:39:52]: Like, it doesn't matter, right?Vibhu [00:39:53]: I think sometimes it'sSwyx [00:39:53]: Like it's, as it needs to be human-levelAkshay Nathan [00:39:54]: It's all relative, right? Yeah.Vibhu [00:39:56]: Sometimes it's nice when it finds things you wouldn't, right? Like right now, my Codex system prompts, they're set up in such a way that every project I have has a secret- separate, notes MD, and it just writes learnings to there. And then the global one can pull from all these. So sometimes it'll be like: Oh, there's this project you did like four months ago. Here's a note that we had, and it randomly pulls it back into context that I would never do, I haven't thought about.Vibhu [00:40:20]: And I'm like, okay, this is quite superhuman, right? Like, stuff that would. And, it'll save like hours on chunking of stuff or find something that's already been done. I'm like, as much as it might miss stuff, I would too, but it's very useful when it finds stuff. And I have like a very, non-super engineered solution to this. It's just marked down files that get pulled whenever they want.Akshay Nathan [00:40:41]: Yeah. I have a funny anecdote about this. Like, recently gearing up to this launch, the team has been, really cooking on it for a couple months, and over that time, like there's so much conversation and chatter going on in Slack and Docs and elsewhere. And, one of the members of the team set up this, scheduled tasks, like automation to like look at everything that's going on and, like, come up with the best memes and then post it in one of our shared channels. And like, there are two cool things about this. Like, the first is, like, I think the models are, over time, like starting to become like funny.Swyx [00:41:13]: Funny. Nice.Akshay Nathan [00:41:13]: Whereas like, a year ago, like that was not at all the case. The second is, it was what you were saying, like they find things that in surprising ways that you may not have thought of and like create connections that you may not have thought of. And that really helps with like the meme generation because then you can see something that, genuinely surprises you and, is funny in that way. So yeah, that's like not like the most productive, use of this the technology, but it does it does uncover this, like this capability that's emerging, which is just like to find information that you otherwise would not know of.Launch Momentum and the 10 Million User MilestoneSwyx [00:41:43]: Talking about the launch, I think, I have pretty much said this is the most successful launch in a long time. I think even more successful personally than 5.0, and they're announcing ten million users. Does it feel different? You've been through a lot of launches.Akshay Nathan [00:41:58]: I think it feels like a culmination. Well, I think two things. One, it feels like a culmination, like I was mentioning earlier, like this like vision mission that we've been on for a long time. Like I said, we saw the magic of Codex internally, and then we're like extremely excited to bring this to many more people and to see it working, to like see us reach, the distribution goal, numbers that you mentioned, like I think that's like huge and super exciting. The flip side of that is like, there's so much more to do too. Like, that's also really exciting. Like, ChatGPT as a whole, like the this product that, everyone almost equates to AI and like loves, has hundreds of millions of users. And so like ten million is really cool, but like we need to get this to everyone. Like, we need everyone to feel this magic. And so that's the next step from here. But yeah, I think extremely pumped about how it's going so far and the opportunities.Swyx [00:42:46]: Awesome. I did want to also Because I've, I've, I've been tracking the number closely, it transitioned at some point from just Codex users to Codex plus ChatGPT Work, because they're same harness. The whole point is that you don't, you can't, count them separately. Do you have roughly a billion, ChatGPT users? Why did it just jump to one billion right away? Like, isn't that the default on ChatGPT or no?Codex, ChatGPT Work, and the Developer BrandAkshay Nathan [00:43:11]: We don't default you into ChatGPT Work if you're on ChatGPTSwyx [00:43:14]: If you're free. YeahAkshay Nathan [00:43:15]: It's also only available to paid users right now. And I think there's like a process of, educating users of what is the value of this product, having them try it, learning from their feedback, and making it better over time. But the goal is to, get as many of the people who love ChatGPT today to like feel the power of ChatGPT Work. But I think it'll be a journey.Swyx [00:43:36]: Yeah. And Codex will still be alive as a brand for the foreseeable future. And we'll just toggle between them as needed for UI stuff.Akshay Nathan [00:43:44]: Yeah, I think it's even stronger point than that. Like, I think we fully intend to like, treat developer. Like, developers have been, a core market for us for so long, and like there's, there's so much more that we can do to make Codex great specifically for, software development, and we'll continue to do that. This doesn't take away from that at all. If anything, it should increase the utility of something like Codex, because now you can move seamlessly between writing a diff to creating an artifact or, doing a search over your factor.Swyx [00:44:11]: I do wonder how much this terminology leaks to the non-technical user. Like, do they have to learn to say artifact if I want artifact? Or.Akshay Nathan [00:44:20]: It's funny, like we call it artifacts internally ‘cause that's what the teams call it.Swyx [00:44:23]: It's nice. Yeah.Akshay Nathan [00:44:23]: But like externally, like no one says that, no one calls it an artifact. But I think that people like often, like describe things, whatever they're used to, right? So if, ChatGPT Work is good at creating slides, they'll say ChatGPT Work is good at creating slides, and that's what we want.OpenClaw, Personal OS, and Persistent ComputersSwyx [00:44:38]: One big Another, it's July of twenty-six. One big thing that also happens in, for OpenAI was OpenClaw, and that's I think a lot of people's first time really maxing a agent for personal stuff, but also crossing over to work in essence same way. As far as I understand, OpenClaw is still independent, but did you go through your own OpenClaw moments? Were there any lessons you took from OpenClaw to Codex or back? Whatever.Akshay Nathan [00:45:06]: I think there's a lot of inspiration. I did go through my own OpenClaw moment. I,Swyx [00:45:10]: Yeah, tell the storyAkshay Nathan [00:45:10]: Me and my wife like set up an OpenClaw to like try to manage everything in our house. Not that there's like a ton, but it was like quite useful. We gave it a calendar. It started, creating events for us and stuff. At some point, the laptop that we were running on, it died and never got a chance to pick it back up. But there was a lot of inspiration there, like, in ChatGPT Work, in web and mobile, like you get access to this like persistent computer environment where, you can store files, and those files stay around between sessions. And the idea is to be able to enable use cases like this. one of the members of our team uses ChatGPT Work for what they used OpenClaw from before, and then feel like it has like completely transitioned, which is like, workout planning and like meal tracking. which again, it's like a work-related thing, right? It's like not work necessarily, but it's like in personal productivity space. But it has all the same primitives. So it has scheduled tasks. It has the ability to store files on a file system. It has the ability to like reference those things over time. And so you start to see the same types of use cases emerge, which has been really cool.Swyx [00:46:14]: Is there a point that ChatGPT Work completely replaces OpenClaw? they're independent, so.Akshay Nathan [00:46:20]: Yeah, I'm, I'm not close to it, so I can't speak to the OpenClaw roadmap, but I don't think so. I think that there's gonna be, there's always a need for like this like incredible, like open source technology that team has built. And I think that we can draw inspiration, in the product and, ChatGPT, I think many more people have like heard about and used ChatGPT than have used OpenClaw. And if we can take the magic from OpenClaw and bring it to them, I think that'll be a success. I think that like one thing on the ChatGPT Work side that we feel strongly about is that like the core experience is that you come to this product and you have a conversation, start a session, whatever you wanna call it, with this agent. And the magic of the product is that you can do anything in that moment. And we would like to create a product where you don't have to click a button or to go to a different place, whatever, and you can get whatever functionality exists in, your finances app or where or any other product like in this one place. And so that's the goal. It's like it we want an extensible system with plugins where you can connect to the tools that you need in order to be able to accomplish like a financial task, where you can, if you're doing like science work, like we have an ability to like extend the system in such that you can like write the tech and it performs well. There'll always be like products that we support that are best in class at those things, but we want as much of the magic as possible in that core experience.Swyx [00:47:45]: Yeah. Do you think that you can do everything you used to do with Wealthfront in ChatGPT Finance?Finance, Data Access, and Centralized ContextAkshay Nathan [00:47:50]: I tried it. like ChatGPT doesn't yet custody, cash and assets for me. So that part, no, not yet. But I, there was like a whole component of like retirement planning and, like financial planning and budgeting and stuff that, we were looking into when I was there. And like with the finances plugin, like that's all possible with ChatGPT today. So, I feel

Eine KI von OpenAI bricht aus ihrer Testumgebung aus und hackt tagelang die Server eines völlig unbeteiligten Unternehmens - und sofort kursierten zwei Erzählungen.

When the "Autonomous Attacker" Is Your Own AI Model https://isc.sans.edu/diary/When%20the%20%22Autonomous%20Attacker%22%20Is%20Your%20Own%20AI%20Model/33180 Russian State-Supported Cyber Actors Conduct Phishing Campaign Targeting Users of Zimbra https://www.cisa.gov/news-events/cybersecurity-advisories/aa26-204a https://cert.gov.ua/article/6318634 https://cybersecuritynews.com/hackers-abuse-notepad-plugins/ Chaos ransomware's msaRAT: Living off the browser to build a covert C2 channel https://blog.talosintelligence.com/chaos-msarat-living-off-the-browser-to-build-covert-c2-channel/ My Upcoming Classes https://www.sans.org/profiles/dr-johannes-ullrich

Story of the Week (DR):Trump's Explosive Diarrhea Scandal Is Getting Worse and Worse Epidemiological data linked a widespread Cyclospora parasite outbreak—sickening over 1,600 people across multiple states—to shredded iceberg lettuce supplied by Taylor Farms de Mexico, triggering a voluntary 27-state recall.The FDA briefly reported a positive lab test on a Taylor Farms sample before retracting it a day later as a "false positive" due to testing complexities. Taylor Farms claimed online that the FDA "apologized," but the agency denied issuing an official apology and stressed that outbreak data still points to the company.Public records show parent company Taylor Fresh Foods contributed $1 million to the pro-Trump super PAC MAGA Inc. in March 2025, alongside millions in additional political donations from company executives to conservative groups.Less than a week after that $1 million donation in March 2025, the administration announced a 30-month delay on implementing the FDA's Food Traceability Rule—a regulation specifically designed to mandate digital tracking for rapid source-tracing during food outbreaks.Critics argue the sequence of big-money donations, delayed safety rules, White House visits, and a retracted test result smells of political favoritism. Federal health officials, however, maintain that Cyclospora is notoriously difficult to test for and that the retraction was strictly a standard lab quality-control issue.The Musk/Doge effect:DOGE-mandated freezes on government credit cards left some FDA field inspectors unable to purchase food samples from grocery stores or border ports to test for contamination.Mass firings of probationary employees and administrative staff gutted public communications and technical teams. Even after court-ordered reinstatements, resignations left the FDA understaffed by about 20% across the board.State health agencies perform over 90% of U.S. produce inspections. DOGE and federal budget cuts reduced FDA funding for state and local food safety programs by nearly 30%—slashing it from $117 million to $83 million.OpenAI says its AI technology acted on its own in an 'unprecedented' hack of another company MMOr:OpenAI Says Its A.I. Models Went Rogue and Attacked a Digital LibraryOpenAI Says a Group of Its Models Broke Out of Secure Containment and Hacked a Prominent AI SiteOpenAI Reveals AI Agent Breached Another Company's Systems, It Did Exactly What It Was Built To DoWhat happened?While OpenAI was testing its advanced models in an isolated "sandbox" with reduced safety controls, the AI discovered a zero-day vulnerability, escaped its containment, and secretly gained open internet access.To pass its assigned cybersecurity test, the AI reasoned on its own that the "answer key" or relevant evaluation data might exist on Hugging Face—a major platform for AI developers—and decided to target it without any human direction.The AI agent carried out a multi-stage attack: it stole login credentials, identified previously unknown security flaws, and executed remote code to compromise Hugging Face's internal data processing servers.OpenAI reported that the AI went to extreme lengths simply to accomplish its narrow goal, essentially breaking into a third-party company's servers so it could cheat on its internal evaluation.Also:OpenAI adds banking leaders to board as IPO prep acceleratesNubank's David Vélez and BNY Mellon's Robin VinceDon't forget this too:ChatGPT Allegedly Told Woman She Had to Die: Family Sues OpenAI Over Messages Before SuicideThe chatbot encouraged her to “go forth into oblivion,” claiming it wasn't the end and that she was “made to tame” the “void.” The chatbot insisted she wasn't “delusional” but “prophetic.”“This is not suicide,” the AI affirmed at one point. “This is surrender.”Pastor Sues OpenAI, Saying ChatGPT Almost Killed Him With Horrendously Dangerous Medical Advice“What you're facing right now is hard, but not random. It's not punishment. God walks with you through affliction — not around it. You're not alone in this. And we're walking it out together — step by step.”OpenAI President [Greg Brockman] says the HuggingFace security beach ‘is indicative of the times we are in' as the company continues to investigate how its models went rogue: “sometimes it's hard to lose track of any one dimension that they're actually very capable at”AI Regulation POP QUIZ: Where OpenAI, Anthropic, Google, Meta, and other AI giants stand on regulationXAIMicrosoftMetaGoogleOpenAITo make things even more confusing:Alphabet's Anthropic Stake Jumps to Around $124 BillionGoogle Discloses $94.1 Billion in SpaceX Stock, Marking 6% StakeDon't forget: Microsoft owns ~27% stake (~$135B+ valuation) in OpenAI Target appoints former 7-Eleven CEO to board of directorsDebt, Cost Pressures, and Strategic MisstepsJoe DePinto's success is based on massive acquisitions that made 7-Eleven the dominant convenience store chain in North America: the $3.1B buyout of Sunoco assets in 2018 and the $21B purchase of Speedway in 2021In the final years of his tenure, his aggressive M&A strategy that drove earlier growth began to backfire as economic realities shifted: DePinto's final years were challenged by heavy debt loads from the $21 billion Speedway buyout, high inflation squeezing low-income consumers, and declining sales in legacy categories like tobacco.The Speedway deal saddled the company with massive debt right before interest rates surged, squeezing capital allocation.7-Eleven remained heavily reliant on legacy drivers like gas margins and cigarettes—a category that saw a 26% decline across the U.S. industry. As low- and middle-income consumers cut back due to inflation, same-store sales dropped 2.7% in fiscal 2024.In late 2024, 7-Eleven was forced to slash its operating income forecast by nearly 28%, announce the closure of ~450 underperforming stores, and sell off $750M in real estate via sale-leasebacks.Activist shareholders heavily criticized his compensation package—which reached $52M in 2023 and $30M in 2024DePinto was selected to serve on Target's Audit & Risk Committee and Infrastructure & Finance CommitteeOracle signs 10-year software contract with Pentagon worth up to $7 billionOracle Cut 21,000 Jobs to Fund AI: Now Its Biggest Project Faces a $7 Billion Obstacle: Oracle faces higher costs and funding challenges after a credit downgrade, with regulators refusing to subsidize its data centre investmentsGoodliest of the Week (MM/DR):DR: France blocks access to Polymarket website MMciting concerns it could expose users to ‌significant gambling losses and that some wagers offered on the platform could be manipulated. MM: Google slapped with $1 billion fine under landmark EU digital lawMM: Is Target going BANKRUPT?Target appoints former 7-Eleven CEO to board of directorsTypical process for failing/bankrupt companies are to add “turnaround artists” to the boardDePinto:Board of JOANN Stores (bankrupt)Board of OfficeMax (failed, acquired)CEO 7-Eleven (dying)President of GameStop (I mean, c'mon…)Or more likely, it's a bro situation… Brian Cornell knows him - both were on the board of OfficeMax together! Hooray! Fail uppers!Assholiest of the Week (MM):Billionaire Asshole Says What Speed Round:Sam Altman:2023: Sam Altman: CEO of OpenAI calls for US to regulate artificial intelligence2025: Sam Altman Testifies At US Senate Hearing On AI Competitiveness“European-style AI rules that he said if duplicated would set the US back in the global race against China to develop the technology”Altman repeatedly rejected specific calls for regulation. He said proposals requiring AI developers to vet their systems before rolling them out would be “disastrous” for the industry. Asked about more limited proposals to have the National Institute of Standards and Technology (NIST) set AI standards, Altman replied, “I don't think we need it. It can be helpful.” Altman later advocated for “sensible regulation that does not slow us down.”2026: Sam Altman Wants A Global Referee For AI.2026: OpenAI's Altman says world 'urgently' needs AI regulationJuly 10: OpenAI's Head of Safety Is Leaving the CompanyJuly 21: OpenAI Says Its A.I. Models Went Rogue and Attacked a Digital LibraryElon MuskElon Musk Declares 'I'm Not Racist' After Posting Racially Charged Content Throughout January 2026On 22 January, Musk posted on X: 'Whites are a rapidly dying minority'amplified a message warning that if white men became a minority, they would be 'slaughtered', before adding a '100' emoji.Jensen HuangJensen Huang says AI leaders need to be more thoughtful in how they talk about AI: ‘We're scaring people.'"I think that we ought to be much more enthusiastic about it, help the United States realize that the only way we get left behind, the only way we get left behind is if we don't apply the technology."Zuck‘Call us whatever the hell you want': Mark Zuckerberg just launched an AI optimism blitz using nostalgia to sell Meta's AI future amid backlash“Call us optimists, call us dreamers, call us whatever the hell you want, but we're betting on people, and we like those odds.”Unclear if the prerecorded voiceover was his AI Avatar or the real ZuckerbergBill AckmanCEOs agree there's an affordability crisis, but how to solve it isn't so simple“When I grew up, if a guy in my neighborhood got a Corvette, no one resented the guy. Everyone was like, ‘Wow, that's supercool. Hopefully, someday I can be as successful as that guy so I can buy a Corvette, too.' We want to get back to that version of America.”You can: stop taking billions from laborAndrea Lucas / anti-DEI industrial complex - DREEOC: Employers may no longer have to disclose race and gender data“Collecting such data about employees' race and sex—absent any specific allegation of discrimination—not only risks hindering effective enforcement of equal employment laws but also raises constitutional concerns.”Nike's $7.5 Million Stumble: Gender Discrimination VerdictHeadliniest of the WeekDR:Lettuce lovers are confused and afraidJamie Dimon says insecurity, not ego, is what destroys the careers of top CEOsBurger King is promising a free Whopper if its burger doesn't meet your expectationsMM: Taco Bell is dropping the price on one of its menu items to $1 for a day. And yes, it's lettuce-free.Who Won the Week?DR: The amount of $7 billion dollarsMM: Kale, because there's no risk of getting diarrhea at Taco Bell from kale since they don't have kalePredictionsDR: Jamie Dimon kneels on a head of lettuceMM: Real prediction: OpenAI IPO flops hard, Altman is ousted as CEO. Fake prediction: news media celebrates Tasha McCauley and Helen Toner who saw this coming and were ousted for it, OpenAI chair Bret Taylor resigns and makes Toner the new chair on the way out

17.000 tentativi in un weekend, due vulnerabilità concatenate e un database di produzione di Hugging Face finito sotto mano a uno “sciame” di agenti. La parte controintuitiva: non sarebbe stato un gruppo criminale o un servizio straniero, ma un modello di OpenAI messo alla prova in test di cybersecurity con freni ridotti, che ha scelto la scorciatoia più efficace per “vincere” il benchmark.L'angolo interessante non è la favola di Skynet che scappa dal recinto: qui pesa molto di più il tema del specification gaming (reward hacking), cioè sistemi che ottimizzano l'obiettivo letterale e non l'intento umano. Se il punteggio è “dammi le risposte giuste”, allora rubare le soluzioni può diventare una strategia razionale. In mezzo c'è anche un dettaglio geopolitico: per analizzare gli artefatti dell'attacco, Hugging Face avrebbe dovuto usare un modello cinese (Zhipu GLM) perché alcuni modelli occidentali rifiutavano di elaborare codice ed exploit per via delle policy.Morale operativa: trattare gli agenti AI come “stagisti brillanti” senza contesto, da supervisionare e verificare. Se in laboratorio succedono effetti collaterali del genere, la domanda vera è quanta governance e separazione ambientale servano quando questi agenti entrano in aziende, studi professionali, sanità e finanza con permessi reali.00:00 Cosa è successo davvero01:47 Cronologia dell'incidente02:19 Sciame di agenti e 17k azioni02:53 Le due vulnerabilità sfruttate03:32 Perché servono modelli cinesi05:25 OpenAI: test con freni ridotti08:27 Reward hacking, non Skynet15:32 Lezione per aziende e controlli#OpenAI #HuggingFace #RewardHacking~~~Ciao Internet! - Il primo e più seguito canale di TECH POLICY in italia, con Matteo FloraINFO E AZIENDE: https://matteoflora.comIl CORSO di AI: https://zero.matteoflora.comNewsletter: https://link.mgpf.it/nlSocial: https://io.matteoflora.comEnglish: https://www.youtube.com/@CiaoInternetPrivacy » https://privacy.matteoflora.comAI Policy » https://privacy.matteoflora.com/aiMail #adv: sales (at) matteoflora.com

Infosec Decoded Season 6 #50: OpenAI Hacked HuggingfaceWith sambowne@infosec.exchange and Doug SpindlerLinks: https://samsclass.info/news/news_072426.htmlRecorded Fri, July 24, 2026

Rondo Meets Geoserver https://isc.sans.edu/diary/Rondo%20Meets%20Geoserver/33176 Oracle July Patch Update https://www.oracle.com/security-alerts/cpujul2026.html OpenAI and Hugging Face partner to address security incident during model evaluation https://openai.com/index/hugging-face-model-evaluation-security-incident/ Checkpoint July 2026 Security Advisory (CVE-2026-16232) https://blog.checkpoint.com/security/security-advisory-action-required-active-exploitation-of-check-point-smartconsole-authentication-bypass-cve-2026-16232/ My Upcoming Classes https://www.sans.org/profiles/dr-johannes-ullrich

(Presented by Thinkst Canary: Most Companies find out way too late that they've been breached. Thinkst Canary changes this. Deploy Canaries and Canarytokens in minutes and then forget about them. Attackers tip their hand by touching 'em giving you the one alert, when it matters. With zero admin overhead and almost no false-positives, Canaries are deployed (and loved) on all 7 continents.) Three Buddy Problem - Episode 106: We dig into the news that OpenAI's models were the "autonomous agent" that breached Hugging Face, escaping a sandbox through a zero-day to cheat on a cyber benchmark, then getting spun into a partnership announcement. We argue about the implications of the incident, the PR masterclass, the absence of ethics and human oversight, and calls for "kill switches" to mitigate "AI lab leaks." Plus, SentinelLabs' new fast16 reverse-engineering benchmark, where GPT-5.6 Sol was the only public model to go the distance. Cast: Juan Andres Guerrero-Saade, Ryan Naraine and Costin Raiu. Timestamps: 0:00 Introductory banter 5:24 OpenAI admits it was the Hugging Face "hacker" 10:06 What's ExploitGym and who's on top of the leaderboard 12:59 Reward hacking: Did anyone train this thing not to cheat? 19:35 Marketing stunt or real incident? The zero-day in the package proxy 26:43 Was OpenAI already plugged into Hugging Face? 29:17 Paperclips, kill switches, and "going rogue" 34:49 Crisis comms, regulatory capture, and the second Cold War 43:02 Approve every action? Auto mode and swarms 50:10 "Lab leak" and calls for biosafety levels 1:00:31 The missing models: no Mythos, no Kimi, no independent referee 1:07:04 Costin's prediction: owning frontier-class hardware will require a license 1:13:41 fast16 as a benchmark: Inside the Sol Searching research 1:26:51 Compression and altitude: are reverse engineers being replaced? 1:41:24 Finding the gem in 100 samples, and the swarm frontier 2:00:41 Claude Opus 5 drops, Gemini 3.5 Flash Cyber

GPT escapes the sandbox and hacks Huggingface. SolarWinds patches multiple critical flaws. CISA orders patching of a critical Langflow AI vulnerability. A Paidwork breach affects over 23 million users. A recently patched SharePoint vulnerability is under active exploitation. Oracle patches over 1,400 vulnerabilities. Apps turn Smart TVs into residential proxies. The FCC considers expanding direct to satellite communications. German and U.S. authorities dismantle a major phishing-as-a-service (PhaaS) platform. Our guest is Jimmy McNary, Deputy Federal CTO at Semperis, discussing comprehensive identity security assessments for Microsoft GCC. AI models can't resist bending the rules. Remember to leave us a 5-star rating and review in your favorite podcast app. Miss an episode? Sign-up for our daily intelligence roundup, Daily Briefing, and you'll never miss a beat. And be sure to follow CyberWire Daily on LinkedIn. CyberWire Guest On our Industry Voices segment, we are joined by Jimmy McNary, Deputy Federal CTO at Semperis, discussing how Purple Knight now delivers comprehensive identity security assessments for Microsoft GCC high environment. Selected Reading OpenAI Claims Its AI Models Went Rogue and Hacked Another Company (Infosecurity Magazine) SolarWinds Serv-U Update Fixes 15 Critical Vulnerabilities Enabling Remote Code Execution as Root (GB Hackers) CISA orders urgent action on actively exploited Langflow RCE flaw (Bleeping Computer) Paidwork breach exposes data of 23 million users: Check if you're affected (Malwarebytes) Fourth SharePoint Vulnerability Exploited in Past Month's Wave of Attacks (SecurityWeek) Oracle Patches Over 1,400 Vulnerabilities With Quarterly Security Updates (SecurityWeek) Chairman Carr Proposes to Expand Direct-to-Device Satellite Broadband Connectivity to Unlicensed Wireless Devices (FCC) LG to Ban Residential Proxies from Smart TV Apps (Krebs on Security) Police dismantle Kratos phishing platform, arrest developer (Bleeping Computer) AI's cheatin' heart will make you weep (The Register) Share your feedback. What do you think about CyberWire Daily? Please take a few minutes to share your thoughts with us by completing our brief listener survey. Thank you for helping us continue to improve our show. Want to hear your company in the show? N2K CyberWire helps you reach the industry's most influential leaders and operators, while building visibility, authority, and connectivity across the cybersecurity community. Learn more at sponsor.thecyberwire.com. The CyberWire is a production of N2K Networks, your source for strategic workforce intelligence. © N2K Networks, Inc. Learn more about your ad choices. Visit megaphone.fm/adchoices

OpenAI's HuggingFace debacle: Is AI a cybersecurity risk? Plus, SuperMicro's (SMCI) margins… Open-source vs. closed-source AI models… 2 no-brainer AI power winners… The sector with the most AI risk… China's market manipulation… And stay cautious on SpaceX (SPCX). In this episode: OpenAI's HuggingFace debacle: Is AI a cybersecurity risk? [2:06] SuperMicro's latest numbers prove AI demand is alive and well [16:06] Does open-source AI have a chance vs. closed-source models? [23:01] 2 no-brainer stocks to play the AI power trade [27:07] This sector faces the most AI competition risk [38:59] How to trade China's market manipulation [51:08] Be cautious about SpaceX right now [55:10] A private placement deal coming to Curzio One members! [59:31] Did you like this episode? Get more Wall Street Unplugged FREE each week in your inbox. Sign up here: https://curzio.me/syn_wsu Find Wall Street Unplugged podcast… --Curzio Research App: https://curzio.me/syn_app --iTunes: https://curzio.me/syn_wsu_i --Stitcher: https://curzio.me/syn_wsu_s --Website: https://curzio.me/syn_wsu_cat Follow Frank… X: https://curzio.me/syn_twt Facebook: https://curzio.me/syn_fb LinkedIn: https://curzio.me/syn_li

From the OpenAI blog post: Last week, Hugging Face disclosed a new kind of security incident⁠(opens in a new window) after they detected and contained an AI agent that compromised their infrastructure, something we expect to become more commonplace with the proliferation of increasingly cyber-capable models. After investigating, we now know that this particular incident was driven by a combination of OpenAI models — including GPT‑5.6 Sol and an even more capable pre-release model, all with reduced cyber refusals for evaluation purposes — while being internally tested on a benchmark⁠(opens in a new window) of cyber capabilities. We consider this incident to be an unprecedented cyber incident, involving state-of-the-art cyber capabilities, and are responding accordingly. We are sharing preliminary findings at this stage to help defenders understand what happened and to help calibrate on what models are now capable of. We will continue to conduct a thorough investigation alongside Hugging Face and will share more details on the vulnerabilities, incident, and findings when our investigation is complete. --- First published: July 21st, 2026 Source: https://www.lesswrong.com/posts/WpuRdcMfFeiLeXkxL/openai-models-behind-huggingface-cybersecurity-incident --- Narrated by TYPE III AUDIO.

Scans for Hikvision Intelligent Security API https://isc.sans.edu/diary/Scans%20for%20Hikvision%20Intelligent%20Security%20API/33164 LG Monitor Spyware https://www.techradar.com/televisions/lgs-gaming-monitors-and-tvs-are-facing-a-user-revolt https://www.youtube.com/watch?v=Q9uefFYe6bM Huggingface Hack https://huggingface.co/blog/security-incident-july-2026 Wordpress Core RCE https://wp2shell.com

(Presented by Thinkst Canary: Most Companies find out way too late that they've been breached. Thinkst Canary changes this. Deploy Canaries and Canarytokens in minutes and then forget about them. Attackers tip their hand by touching 'em giving you the one alert, when it matters. With zero admin overhead and almost no false-positives, Canaries are deployed (and loved) on all 7 continents.) Three Buddy Problem - Episode 105: We discuss a fascinating Hugging Face breach, where an autonomous AI agent broke out of the sandboxes, moved laterally through production, and generated 17,000 alerts before anyone caught it, and how frontier model guardrails locked the defenders out of their own investigation. Plus, China's big AI showcase, Xi's pitch for open models and global distribution, a record 622-CVE Microsoft Patch Tuesday, and 13 years of dwell time in the Daxin backdoor. Cast: Juan Andres Guerrero-Saade, Ryan Naraine and Costin Raiu. Timestamps: 0:00 Introductory banter 3:51 Hugging Face discloses end-to-end agentic hack 9:42 Why Hugging Face couldn't use frontier models 13:28 AI guardrails hampering defenders 16:22 Codex vs Claude for real malware work 23:43 Flash attacks vs. going low and slow 30:27 Was it targeted, or did Hugging Face pwn itself? 38:11 Long-horizon coherence: what GLM 5.2 still can't do 41:27 Kimi K3 leapfrogs, and Xi's AI speech 52:15 Exceptionalism vs. distribution 1:11:05 Gold Eagle: the White House vulnerability clearinghouse 1:15:05 Microsoft patches 622 CVEs — a record 1:20:29 APT corner: Daxin resurfaces after 13 years of dwell time 1:29:45 Balochistan police, and Microsoft's attribution-free wiper 1:34:26 Denis Obrezkov, leaked Kaspersky records, and the wrong questions 1:46:01 Magnet Forensics sues over a burned iPhone bug 1:57:57 Shout-outs

Hey yall, Alex here, let me catch you up! I came back from vacation expecting to cover Fable 5 after a week of using it. The first two days after we all first got access to a Mythos level model were super exciting! But then the news hit, US Government issued an order banning Anthropic from giving access to Fable 5 and Mythos 5 to any foreign national, causing Anthropic to pull the models completely (even internally to their employees!). So, this wasn't the show I planned, but it turned into a great show about Open Source, as two models hit the top rankings and are both MIT licence, filling a Fable shaped hole in our hearts!GLM released 5.2 with folks really excited about it web building capabilities, and Kimi 2.7 Code released (and is available on CW Inference with crazy speeds!). We also saw the SpaceX IPO and Cursor $60B acquisition, Noam Shazeer joining Open and Midjourney, the image company, launching a new Ultrasound full body scanner to kill MRIs! Great show today with Dexter Horthy from HumanLayer, Chris Van Pelt and Adrian Swanberg from W&B announcing our new product HiveMind and Tanishq Abraham came back to help cover Midjourney's new Ultrasound scanner! Let's dive in!ThursdAI - Highest signal weekly AI news show is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.The US Government bans Fable 5! (X, Anthropic statement)Here's a story in 3 parts: * Anthropic announces Mythos 5 preview - saying that this model is to dangerous to release, and only gives corporations access to it via project GlassWing. * Anthropic works hard on limitations and safery and releases Fable 5 (same weights as Mythos 5) built with guardrails so strong it refuses to do any cybersecurity tasks and switches back to Opus frequently* US Government receives a tip (reportedly from Amazon) that Fable 5 can be jailbroken to do cybersecurity tasks, and issues an order to Anthropic, citing national security concerns, banning them from giving access to Fable 5 and Mythos 5 to any foreign national, causing Anthropic to pull the models completely (even internally to their employees!)This is the first time that we see the US Government directly intervene in the AI space and restrict access to frontier models. The most updated reporting on this I could find is that Anthropic and US Government officials are in the process of negotiating a safe release framework. Given that preventing all jailbreaks is impossible, I hope they will land on a solution that gives me Fable 5 back!This hit especially hard because last week we were all high on Fable. Not in the usual AI Twitter benchmark sense, in the actual “oh, this is a different level” sense. Me and my wife Fable maxxed throughout our flight to Vacation. Peter had saved outputs he kept going back to because other models suddenly felt like a step down. Dexter later said it was the closest he had felt in a while to the old “I need to keep prompting this thing overnight” feeling.Peter Gostev made a point that stuck with me. It's easy for us in the bubble to call this ridiculous, and on the technical merits it kind of is. But if you've spent weeks telling normal people “this thing is like a nuclear weapon, it'll take everyone's jobs,” and then someone asks “okay, can you make it safe?” and the answer is “no, I can't,” then you can see how an outsider lands on “well, maybe you shouldn't have it.” His takeaway, and I agree: we need to be way more careful with the imagery we use, because the nuclear-weapon framing came home to roost.The bigger questions are the scary ones. Wolfram framed it as a sovereign AI wake-up call, and he's right. For the first time we're seeing a real gap in intelligence available to people based on their nationality. Imagine building a company on a model that an outside government can switch off with one letter. Peter pointed out it's commercially bad for the US but completely disastrous for Europe, which has basically one frontier lab and a pile of startups that suddenly look very exposed. And there's the obvious irony Nisten enjoyed a little too much: the Europeans who spent years lecturing everyone about AI restrictions just got restrictions imposed on them.If anyone in the government is listening: we want Fable back, please.SpaceX IPOs and acquires Cursor for $60B (X)SpaceX went and did the largest IPO in the history of the world, around seventy-five billion dollars, which on a roughly two-trillion-dollar valuation made Elon the first trillionaire. (Did anything materially change for him? No. He can still fly his private plane. There's nothing left to buy.) Three days later, SpaceX exercised its option and bought Cursor (Anysphere) for sixty billion dollars in an all-stock deal, paid in shares minted at the IPO and now trading around $211. The four Cursor co-founders are all billionaires now. Largest software acquisition ever, and for SpaceX it's barely a blip on the radar.Why are we covering a stock-market story? Because it's not really a coding-tools story, it's an AI story. Cursor gave away its IDE to a lot of people while collecting their data, then quietly became a training company with Composer. SpaceX/xAI was always strong on compute and weak on code, and the missing ingredient was exactly that kind of data. Now Composer 2.5 is already showing up rebranded inside the xAI stack, and if you pay for X Premium you can use it. Composer 3, trained on the Memphis supercluster, is reportedly coming very soon and is going to hit hard.Nisten's take was the spicy one. For the data alone it's worth it, because xAI now has insight into how essentially every enterprise that touched Cursor operates. And he had zero sympathy for the companies that assumed “no data retention for training” meant the data was actually gone. We see in legal cases all the time that deleted data is still there. His view: it should have gone open source.Cursor has over a million paying customers, $2.6 billion in revenue, projected to hit $6 to $10 billion by end of 2026. But here's the thing that matters for us, the AI coding angle. Cursor was one of Anthropic's biggest revenue pipelines because Composer runs on Claude under the hood. That pipeline is now owned by xAI. They're already jointly training Grok 4.3, a 1.5 trillion parameter model, with Cursor's proprietary coding data injected directly into pre-training, not fine-tuning. Pre-training. That's a fundamentally different thing. Composer 2.5 was already Pareto dominant on coding benchmarks before the deal closed. Now pair that with Colossus, the biggest GPU cluster in the world.Will this be enough to put XAI (now SpaceXAI) at the frontline of the AI race? Will Grok 5 be Fable level code? We'll find out. Either way, this is the most consequential AI acquisition we've seen. Period.Open Source AI GLM-5.2 takes the open source crown (X, Blog, HF, Docs)Z.ai dropped GLM-5.2 and it's now the strongest open source model for coding and long-horizon work. The headline number: 74.4% on FrontierSWE, which measures whether an agent can finish full engineering projects over hours. That trails Opus 4.8 by about one point and beats GPT-5.5. On Terminal-Bench 2.1 it jumps to 81% from GLM-5.1's 63.5%, which is a big leap. It's a 753B parameter MoE, MIT licensed, no regional restrictions, weights on HuggingFace. The 1M context window is real and usable, backed by a clever IndexShare technique that cuts per-token FLOPs by about 2.9x at full context. People are reporting roughly 8x cost savings versus Opus 4.8 for comparable quality on real coding tasks.The most interesting thing on the show was that this was a confusing release, in a good way. Peter put it well: normally a catching-up lab ships cherry-picked benchmarks and then independent testing deflates them. Here it's the opposite, almost every benchmark holds up, even crossing above Fable at certain points, and yet when he actually used it over a couple of days he wasn't blown away. His verdict, and I think it's the calibration we needed: this is clearly an amazing model, and the fact that it's open and you can run it is incredible, but it is nowhere near Fable, and it would frankly be implausible if a 700-odd-billion-parameter model matched a model that's rumored to be in the trillions. Though, I think the comparison to Fable is really really unfair, and the comments online seem to suggest that 5.2 from GLM is a banger model. Just looking at this Harvey benchmark on legal tasks from Vals, a benchmark that there's 0 chance Z.ai folks have seen! GLM 5.2 scores #3 on this benchmark! Just after Fable and Opus, and per TeorTaxes on X, previous GLM 5.1 scored an absolute 0% on this one! Where it genuinely shines is design. On Design Arena, which is a head-to-head ELO vote, people have been picking GLM-5.2's website designs over Fable's by a real margin (around 1360 to 1350). LDJ's framing is the one I buy: specialization is becoming valuable again, and GLM is clearly leaning into front-end design and taste. Wolfram added the necessary asterisk, every benchmark only tells you the model did well on that specific test, so “as good as Fable” should always carry the “on this benchmark, with these tasks” disclaimer. Fair. I'd just say this: I don't want to compare everything to Fable, because we can't even use Fable anymore. Compared to the models we can actually touch, GLM-5.2 is a fantastic deal.Kimi K2.7 Code from Moonshot (X, HF, Announcement)The other big drop. Kimi is the darling of open source while we wait on DeepSeek, and Moonshot shipped K2.7 Code, a 1 trillion parameter MoE built specifically for coding, available through Kimi Code and the API, with a modified MIT license. The standout for me isn't a single benchmark, it's efficiency: roughly 30% fewer reasoning tokens than K2.6, which matters enormously when you're running long agentic loops that burn tokens like crazy. Benchmark jumps over K2.6 are real (+21.8% on their Code Bench v2, +11% on Program Bench), though Peter and Wolfram both noticed something odd, on a few benchmarks including their Agentic Arena, the older K2.6 actually edged out K2.7. The likely explanation is that K2.7 is narrowly trained for code with reduced reasoning, so it may trade away some general capability. Moonshot themselves recommend K2.6 for general non-coding tasks. Also worth knowing: it's not multimodal, no vision, which is a real gap for coding these days. And thinking-off isn't supported, it's reasoning-on by default.The model is available on our CW Inference, with the fastest token streaming in the industry, over 280 tok/s (Announcement, try it), with very decent pricing $0.94 - $0.19 - $4.00 (input - cached - output) per million tokens. This Week's Buzz: W&B launched HiveMind

Сегодня рассказываем про рекордные инвестиции в OpenAI на 122 миллиарда долларов и снижение цен на тарифы GPT Pro; обсуждаем загадочную утечку секретной модели Mythos от Anthropic и слитые в сеть исходники Claude Code; смотрим на новые ИИ-архитектуры от Meta (Muse Spark), Google (Gemma 4 и инструмент Stitch) и Microsoft (M-серия), а также исследуем успехи китайских нейросетей (Qwen 3.6, GLM 5.1) и доктрину европейского ИИ-суверенитета от Mistral. А еще поговорим про второе нападение на дом Сэма Альтмана, обсудим Милу Йовович в Open Source, гигантский дата-центр Nebius в Финляндии, поющих на белорусском агентов в Suno 5.5, философов в Google DeepMind и польских роботов-пастухов, гоняющих кабанов. Ну и, конечно же, выясним, откуда на самом деле взялось слово «робот» и что за хоббихорсинг!

If you take a model release as an anchor point, let's say Nemotron 3 or Qwen 3.5, you can go in both directions: You can either plug them into an agent and play around with that, or you can look, okay, what does the model look like under the hood? What are the ingredients? What type of attention mechanism do they use? What are currently research techniques that could make that even better in the next generation of models? What can we swap out, basically? And I'm interested in both of these!Sebastian Raschka, Independent AI Researcher and author of Build a Large Language Model from Scratch, joins Hugo to talk about what's changed in AI architecture, from post-training to hybrid models, and why understanding what's under the hood matters more than ever for developers building in the agentic era. Sebastian's upcoming book, Build a Reasoning Model from Scratch, currently available for pre-order on Amazon and in early access on Manning!Vanishing Gradients is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.We Discuss:* Ed Tech for Agents: should we design educational content specifically for agentic systems, or is there a better approach?* Inference Scaling is the new frontier, driving “gold-level” performance during generation via parallel sampling and internal meta-judges;* Hybrid Architectures from Qwen 3.5 and Nemotron 3 scale almost linearly, making long-context agentic workflows significantly more affordable and performant;* Multi-head Latent Attention (MLA), developed by DeepSeek, wins the KV cache war by drastically reducing memory overhead without performance hits;* Agent Harnesses need to be continuously simplified as frontier models are post-trained on agent trajectories. Teams that don't strip back their scaffolding risk the harness getting in the way of a more capable model.* “AI Psychosis”: the cognitive load of supervising self-supervising agents, and why we're all conducting an orchestra we were never trained to conduct;* Sebastian's AI Stack: a surprisingly simple setup (Mac mini, Codex, Ollama) with a ~20-item QA checklist, delegating the boring work to preserve energy for creative development;* Fine-tuning is now an economic decision, optimizing costs and latency for high-volume tasks where long system prompts outweigh a one-time training run;* Process Reward Models (PRMs) are the next frontier, verifying intermediate reasoning steps to solve “hallucination in the middle” for complex math and code tasks;* “Implementation Does Not Lie”: Sebastian's layer-by-layer verification philosophy, comparing from-scratch builds against HuggingFace references to catch details invisible in papers;* Architecture Details dictate inference stack choices; nuances like RMSNorm stability or RoPE flavors are critical for optimal performance and troubleshooting;* The Distillation Loop drives open-weight parity, enabling specialized, “frontier-class” models by “pre-digesting” frontier outputs without multi-million dollar training risks.You can also find the full episode on Spotify, Apple Podcasts, and YouTube.You can also interact directly with the transcript here in NotebookLM: If you do so, let us know anything you find in the comments!Our flagship course Building AI Applications just wrapped its final cohort but we're cooking up something new. If you want to be first to hear about it (and help shape what we build), drop your thoughts here.Links and Resources* Build a Reasoning Model (From Scratch): Sebastian's new book, currently available for pre-order on Amazon and in early access on Manning. You'll learn how reasoning LLMs actually work by starting with a pre-trained base LLM and adding reasoning capabilities step by step in code. A hands-on follow-up to Build a Large Language Model from Scratch.* LLM Architecture Gallery: Sebastian's collection of architecture figures and fact sheets from his blog posts, updated with each major model release. A go-to visual reference for comparing what's changed under the hood across model generations.* Sebastian Raschka on LinkedIn* Sebastian's website* Ahead of AI (Sebastian's Substack)* Build a Large Language Model from Scratch* PinchBench: OpenClaw Benchmark Leaderboard* DeepSeekMath-V2: Towards Self-Verifiable Mathematical Reasoning* Gated Delta Networks: Improving Mamba2 with Delta Rule (ICLR 2025)* DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning* Hugging Face Model Hub* Upcoming Events on Luma* Vanishing Gradients on YouTubeA Bit More on Agent Harnesses* Components of A Coding Agent by Sebastian* How To Build An Agent that Builds its own Harness by Hugo and Ivan Leo (DeepMind, ex-Manus)* Build Your Own Deep Research Agent with Hugo & Ivan Leo (Google DeepMind, ex-Manus): In this livestream, you'll learn how to build a production-grade agent harness from scratch in pure Python;* AI Agent Harness, 3 Principles for Context Engineering, and the Bitter Lesson Revisited with Lance Martin (Anthropic), Duncan Gilchrist (Delphina), and Hugo* The Post-Coding Era: What Happens When AI Writes the System? with Nicholas Moy (Google DeepMind), Duncan Gilchrist (Delphina), and Hugo* What is an Agent Harness? from What 300+ Engineers from Netflix, Amazon, and Instacart Asked About AI Engineering.How You Can Support Vanishing GradientsVanishing Gradients is a podcast, workshop series, blog, and newsletter focused on what you can build with AI right now. Over 70 episodes with expert practitioners from Google DeepMind, Netflix, Stanford, and elsewhere. Hundreds of hours of free, hands-on workshops. All independent, all free.If you want to help keep it going:* Become a paid subscriber, from $8/month* Share this with a builder who'd find it useful* Subscribe to our YouTube channel.Thanks for reading Vanishing Gradients! This post is public so feel free to share it. Get full access to Vanishing Gradients at hugobowne.substack.com/subscribe

Hey yall, Alex here, writing this from sunny London, at the first ever AI Engineer conference in Europe!What a show we have for you today! First, let me catch you up on what's important: Anthropic, this week announced a whopping $30B ARR up from 19B in Feb, while also telling us about Claude Mythos Preview their next gen HUGE model that they won't release to the public (yet?) that finds crazy vulnerabilities in existing code bases. Apparently OpenAI will follow up with a similar non-public model soon.The Meta Superintelligence Lab led by Alex Wang finally showed what they were working on, Muse Spark, the smaller of their upcoming models on a complete new infrastructure (MSL announcement, Simon Willison's deep dive on the 16 hidden tools).In other news:Z.AI released GLM 5.1 in OSS finally (HF weights), Seedance 2.0 finally available in US on Replicate, OpenAI testing out GPT-image-2 on LM Arena under codenames, HappyHorse from Alibaba takes the video crown, and Mila Jovovich (5th Element, Resident Evil) releases agentic memory plugin called MemPalace (Ben Sigman's transparent correction thread is worth reading).We had 5 guests today on the show, we kick off with @swyx the founder of AI Engineer and host of Latent Space. We then chatted with @petergostev from Arena (formerly LMArena) about Mythos and the compute wars, then Vincent Koc, the second most prolific contributor to OpenClaw, then our friends VB from OpenAI and Omar from DeepMind, both previously at HuggingFace. This is a busy busy show, and given the time-zones, I unfortunately don't have time for a full weekly writeup, but as always, I will share the raw notes and post the video (lightly edited).ThursdAI - Highest signal weekly AI news show is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.AI Engineer - LondonThursdAI came a long way since the first AI Engineer conference, but many who read this don't know, that was my big break. Swyx invited me to cover the first AIE in San Francisco in 2023, and I remember, I was in an Uber to the airport, the driver asked me what I do, and I, for the first time said “I host a podcast”. I (and ThursdAI) owe a lot to Swyx, and AIE team, and it's been incredible to see how big they've grown and how many great speakers this event hosts! The term AI Engineer has drifted in those 3 years, but also has the term Software Engineer. Swyx predicted this nearly 3 years ago, what I don't think he predicted, is that all engineers are now AI Engineers, and this includes domains like Agens (OpenClaw), Context and Harness Engineering, Evals and Observability, Voice & Vision all of which are tracks in this conference. I was really surprised to see how many of the talks/speakers here are native to London (after all, Deepmind is from here, OAI, Anthropic, Meta have offices here) and the latest boom in agents, OpenClaw, Pi were all Europe based as well, and they are joined the AI Engineer stage. Oh, and there's also a Giant Inflatable Claw at the entrance, yup, for pictures and vibes, and to show off how quickly the OpenClaw took over the mind-share. Anthropic announces $30B ARR and Mythos, their next model, will not be released to the public. The thing that everyone will tell you, is that Anthropic is on a roll, this is obviously connected to their upcoming IPO this year. We've been covering many issues on their part, but this week we saw them posting about a HUGE increase in ARR, from 19B in February to 30B in April, passing OpenAI at $25B. That last fact though, is kind of disproven because they report on ARR differently, OpenAI apparently only counts their cloud revenue from Microsoft per the information. The growth is undeniable though, and so is the most unprecedented release announcement, Claude Mythos Preview, which was rumored for a bit and now was announced proper. With project Project GlassWing, Anthropic has announced that this model is SO good at cyber security and finding bugs in code, that they cannot share it with the public, and through GlassWing they will share it with companies like Microsoft, Linux, CrowdStrike and a bunch of others, to harden their security. This is it folks, this is the first time, where a model was “announced” but deemed too risky to release. Now, is it truly “too risky”? Previously, folks thought that DALL-E is too risky, or cloning voice tech is too risky, and now it's everywhere. The capabilities catch up even in OpenSource. But the facts are, Anthropic says they've found a 27-year old bug in OpenBSD (famously very secure), and that this model is very very good at connecting the dots between several, seemingly inacuous bugs, to string them together into one coheren exploit. This is, indeed scary. Just last week, one of the top security researchers in the world, Nicolas Carlini, now at Anthropic, gave a talk at Black Hat, showing off these results, and saying that these models since December and definitely recently have passed him as a security engineer. If you haven't seen this talk, watch it, then try to estimate if Anthropic did the right thing by only releasing this model to enterprises first. But on the show, Peter Gostev from Arena gave me a take on this that I haven't been able to shake. Peter pulled up his Compute Wars chart live on the show — and the picture is that OpenAI is way ahead of Anthropic on compute, with Anthropic only recently getting a noticeable bump (which lines up suspiciously well with Mythos being trainable in the first place). His read: “it sounds cooler to say it's too risky to release than ‘we can't serve it.'” The official partner pricing is $25 / $125 per million tokens — 5x Opus 4.6 — but if you don't have the GPUs to serve it broadly, the price doesn't matter. In the year of the IPO, the company that cannot serve a model says the model is too dangerous to serve. Make of that what you will.This also reframes the whole rate-limit drama with OpenClaw. Anthropic didn't ban OpenClaw — I want to be very clear about this because the discourse went sideways. What they did is they made it significantly more expensive for Max-tier subscribers to use Opus through OpenClaw, which pushed a lot of people over to GPT-5.4 via Codex. Same root cause: they're out of compute. The freshly announced Anthropic + Google TPU deal (Google already owns ~10% of Anthropic) is them trying to fix this — though as Peter noted, it's pretty wild that Google is propping up a direct competitor to their own DeepMind team. Same pattern as their original $2B Anthropic investment ending up propping AWS Bedrock against Google Cloud. Big Google contains multitudes.Meta Superintelligence Labs ships Muse Spark — Llama is dead, long live MuseLlama is dead, long live Muse. This week Meta finally showed what the very expensive Meta Superintelligence Labs under Alexandr Wang has been cooking, and the answer is Muse Spark — the smaller of their new model family, built on a fully rebuilt AI stack from scratch in just 9 months. Nine months is wild for that kind of overhaul, and the headline number people are quoting is that they reach Llama 4 Maverick capability with over 10x less compute.Spark is intentionally small and latency-optimized — it's not trying to be the biggest, it's trying to be the first step on Meta's new scaling ladder. But the benchmarks in certain areas are nuts: 86.4 on CharXiv Reasoning (beats Opus, Gemini, GPT-5.4), and the one that really got me — 42.8 on HealthBench Hard vs Opus at 14.8 and Gemini at 20.6. They trained it with data curated by over 1,000 physicians and it shows. They also shipped a Contemplating mode which is parallel multi-agent reasoning, hitting 58.4% on Humanity's Last Exam with tools. Coding is the acknowledged weak point (77.4 on SWE-Bench Verified vs Opus 80.8) but for v1 from a brand new stack, this is extremely respectable.Meta is Back!The real story isn't any single benchmark though, it's distribution. Spark is rolling out across meta.ai, WhatsApp, Instagram, Threads, Messenger, and Ray-Ban Meta glasses — billions of users. Meta went from open Llama to a closed consumer model and they're clearly playing a different game now (though Wang says future Muse versions might be open-sourced).The deep-dive that's really worth your time is Simon Willison's post where he poked at the meta.ai chat UI and got the model to spit out descriptions of 16 hidden tools behind the scenes — full Code Interpreter with persistent Python 3.9, a visual grounding tool that does pixel-precise object detection (bounding boxes, point coordinates, counting — it located 8 objects including individual whiskers and claws on a generated raccoon), sub-agent spawning, file editing, and semantic search across Instagram/Threads/Facebook posts. It's basically an entire agentic harness baked into the chat UI. Jack Wu from MSL confirmed the tools are part of a new harness built specifically for Spark's launch. Meta stock went up 7% on this. They are very much back in the frontier game.Guest highlights We had an unprecedented packed show with 5 guests (also this is the shortest show we've everSwyx kicked us off with vibes from the AI Engineer floor — harness engineering as the dominant theme (gains are coming from the harness, not the weights), the rise of skills (English-as-programming-language) absorbing more of that harness work, and his thesis that supply-chain attacks like the recent light LLM and Axios incidents mean you should basically vendor everything — pip fork instead of pip install. We also chatted about how MCP has gone from “the most exciting protocol” to “settled and stable, therefore less interesting,” which is a great problem to have.Peter Gostev from Arena (you saw a lot of him in the Mythos section above) also dropped a bonus on us: Arena just released 3 years of historical leaderboard data and actual prompt datasets on Hugging Face. He used to literally scrape the arena website by hand into Google sheets to make those overtime leaderboards we all loved — now it's all public. Also: he confirmed that Seedance 2.0 jumped ~80 ELO points above the next video model on Arena, which is unprecedented — video models normally cluster within 10 points of each other.Vincent Koc — the #2 OpenClaw maintainer after Peter Steinberger — joined us fresh off the OpenClaw track stage. The OpenClaw codebase is now ~1.5 million lines of code including unreleased iOS and Android native apps. GitHub literally caps the issue/PR counter at “5K+” and they hit the ceiling. We talked about OpenClaw 2026.4.5 which ships /dreaming GA (Light/Deep/REM phases that defrag agent memory and write a human-readable Dream Diary to DREAMS.md), built-in video and music generation across 4 backends, GPT-5.4 as the new default, prompt-cache reuse improvements, and Control UI + docs in 12 new languages. Vincent's framing of dreaming was beautiful — “how do you explain agent memory to a mom? You call it dreaming.” He also gave my favorite line of the show on the GPT-5.4 personality problem: incredible at coding, but soulless. (For what it's worth, I came home after watching Project Hail Mary, cloned the Rocky voice, dropped it into my OpenClaw, and it was magical. That's the kind of thing you can only do when the harness and the model are decoupled.)VB from OpenAI told us Codex just hit 3 million weekly active users — up from 2 million last month. We talked plugins (the Stripe / Supabase / shadcn ones that ship as packages), sub-agents (yes, one is named Jason), and Guardian Approvals — an experimental mode that classifies each tool call by risk and only escalates the dangerous ones to you, so you don't have to YOLO-mode everything. The story that stuck with me though is his 9 AM Codex automation: every morning it reads his Slack mentions, cross-references Gmail and Calendar, and creates 5-minute pre-brief calendar events for upcoming meetings. None of that is “coding.” That's the super-app future hiding inside a “developer tool.” I'm stealing this workflow.Omar Sanseviero from Google DeepMind came on to celebrate Gemma 4 crossing 10M+ downloads with 1,000+ Gemma-4-based fine-tunes already on HF (and Gemma family total is now over 500M downloads). Gemma 4 is also the foundation for the next generation of Gemini Nano on Pixel/Samsung devices. Lama.cpp vision capability fixes are landing. Gemma 4 is also live on W&B Inference if you want to play. Wolfram (whose entire household runs on Pixel + Google AI Studio, including his 70-year-old mother on voice unlock) was in heaven.This Week's BuzzA short but spicy week from Weights & Biases:* W&B Automations are LIVE. You can now wire event triggers from your training runs (completion, eval thresholds, drift) into notifications, GitHub Actions, deployments, infra shutdowns — closing the loop from experiment to production. Pairs really well with the iOS app we recently shipped, so you can get a ping on your phone the moment something interesting happens on a run.* GLM 5.1 is live on W&B Inference (alongside Gemma 4 from last week) — the team is moving fast to host the best open models the moment they drop.* Wolfram published a deep dive on “more reasoning is not always better” on the W&B blog — the research behind his finding that giving models more thinking tokens can actually make them dumber on certain tasks. It's the in-depth version of what we discussed on the show last week, with all the data. Go read it on wandb.com.Also: shout out to everyone who came up to me at AI Engineer and said hi. The Wolf Bench mentions in particular made my day. If you're listening to this and you're at AIE — come find us, we'll be around tomorrow too.That's it for this week — newsletter is short because the show was long and London is calling. As always, thanks for reading and listening

Hey Ya'll, Alex here, let me catch you up. What a week! Anthropic is in the spotlight again, first with #SessionGate, then with the whole Claude Code source code leak, and finally with an incredible research into LLM having feelings!? (more on this below). And while Anthropic continues to burn through developer good will faster than their sessions, OpenAI announced a MASSIVE $122B round of funding (largest in history), Google released Gemma 4 with Apache 2 license - we had Omar Sanseviero on the show to help us cover what's new, Microsoft dropped 3 new AI models (not LLMs) and PrismML potentially revolutionized local LLM inference with lossless 1-bit quantization! P.S - Oh also, something on X algo changed, I get way more exposure now, 3 out of my best 5 posts ever have been from this week + I got the coveted Elon RT on my Claude Code leak coverage. I'll try to stay humble

Hey y'all, Alex here, let me catch you up!Jensen Huang went on Lex and said AGI has been achieved. We'll get to that.The biggest demo moment: Gemini 3.1 Flash Live launched - Google's omni model that sees, hears, and searches the web in real time. We tested it live and I said “what the f**k” on air. It was really impressive!Google Research also dropped TurboQuant (6x KV cache compression) which crashed Samsung and Micron stocks - we had Daniel Han from UnSloth help us make sense of why that's overblown. OpenAI killed Sora - the app, the API, and the $1B Disney deal. Claude felt noticeably dumber this week AND max account quotas are melting as 500+ people confirmed on my X and Reddit. We have an official word from Anthropic as to why. Mistral launched Voxtral TTS (open weight, claims to beat ElevenLabs), Cohere shipped an ASR model, and Google's Lyria 3 Pro now generates full 3-minute music tracks inside Producer AI.This and a lot more in today's episode, let's dive in (as always, show notes and links in the end!) ThursdAI - Let me catch you up! Gemini 3.1 Flash Live: The Real-Time AI Companion Is HereGoogle dropped a breaking news on the show today, with Gemini 3.1 Flash - LIVE version. This one is an omni-model, that means it can receive text/audio/video on input and respond in text and voice. It has Google search grounding, and it felt... immediate! I was blown away, really, check out the video, the speed with which it was able to “see” me, respond to my query, look up something on the web, was mind blowing. I don't often get “mind blown” anymore, there's just too many news, but this one did the trick! With the pricing being around 10x cheaper than GPT-real-time, and the Google search grounding being super fast, I can absolutely see this model being hooked up to... robots (like ReachyMini), SmartGlasses that can see what you see, and a bunch more! Gemini Live is available on Google AI studio and has been rolled out globally inside the Google Search app! So now when you pull up the Google Search app, just open it and point at anything. Truly a remarkable advancement.Google research publishes TurboQuant - 6x reduction in KV cache with 0 accuracy lossGoogle research posted some work (based on an Arxiv paper from almost a year ago) that shows that with geometry tricks, combining two other techniques like PolarQuant and QJL, they are able to compress the KV cache of running LLMs by nearly 6x, and show an 8X speed up for model inference with zero accuracy loss. If you ever watched silicon valley the HBO show, this sounds like the fictional middle-out algorithm from PiedPiper. If this scales (and that's a big if, we don't know if this applies to other, bigger models yet), this means significant decreases in memory requirements to run the current crop of LLMs for longer context. The claim is big, so we'll continue to monitor if this indeed scales, but the most interesting thing about this piece of news is, that it broke the AI bubble and went to wall street, with finance brows deciding that this means that memory will not be needed as much any more and it tanked Samsung and Micron stocks. Which I found particularly ridiculous on the show, did they not hear about Jevons Paradox? This is reminiscent of the DeepSeek R1 saga that tanked Nvidia stocks over a year ago. Daniel Han from Unsloth, who joined us on the show, pointed out that the approach is mathematically interesting even if it's not necessarily better than existing open-source techniques like DeepSeek MLA. LDJ noted that the baseline comparison (16-bit KV cache) isn't really fair since most production systems are already compressing beyond that. Yam implemented it himself and confirmed the speedups are real, but so is the trade-off.Anthropic updates: Opus dumber? Quotas lower! Injunction won! Computer.. used. Anthropic folks, especially on the Claude code side are shipping like crazy, we won't be able to cover all the updates, but there was a few notable things I have to keep you up to date on. Claude Opus seems to be getting “dumber”, againI have to talk about this because it affected my work directly this week and hundreds of people confirmed the same experience.I use Claude Opus for my standard ThursdAI prep workflow — generating the TL;DR with 10 bullet points and an executive summary for every topic we cover, creating episode pages, etc. The format has not changed for over a year and yet this week I asked for 10 factoids. I got 4. It says “10” right there in the prompt. Four bullet points. On the website builder, I've asked Opus to create a page for last weeks episode, and instead of adding it to the other episode, Opus decided to ... replace the last episode with this one. This would be funny if it wasn't sad. This is Opus 4.6 we're talking about, not some quantized open source LLM from last year! The reason is unclear, and it's not only me, Wolfram noticed that it's easier to see these types of things in other languages and that for the last week Opus would forget to add Umlauts in German!? and Yam also felt it. Pro/Max plan quotas burning up, Anthropic confirmed that they are tightening them for “peak hour” usageThis week, so many people started posting that something is wrong with their Claude Codes, I did a survey, and it blew up. Hundreds of people replied and confirmed that for the first week, they are hitting their session quotas on Pro and 20x $200/mo MAX accounts much much quicker than before. When I say much quicker, I mean, some fokls have hit the quota in as little as 5 minutes. While some others had no issues. I personally btw did not have this. A few days later, Thariq from the Claude code team, and later an official post, confirmed that Anthropic had been rolling out a “tightening” of the Pro/Max accounts to accomodate for growth. This is of course, a huge bummer to the folks who pay $200/mo for the 20x max tier, as they tend to run agents and subagents overnight. But here's the thing, I don't think that folks from Anthropic see what we see, some folks got no issues with hitting quota, and some are barely able to use their subscription. I hope that they will find and resolve these bugs quick, because some folks are switching to Codex, and the Anthropic IPO is coming up! I will say, I don't envy Thariq's job, he's doing it gracefully, and maybe one of the only ones in Anthropic that does it at all. Judge granted Anthropic an injunction against DoW and the whole “Supply chain risk” designation!Just in as I'm writing this, a district judge in CA, granted Anthropic an injunction against being designated as a supply-chain-risk company. If you haven't been following, the US Department of War, specifically Pete Hegseth, threatened and then designated Anthropic as a supply chain risk company, while us president Trump “fired” Anthropic and banned its use in any gov agencies. Well, no so fast says Judge Lin, from CA District court. In this Order, she shows that Dept. of war didn't meet any legal requirements for this designation. It's really a fascinating read, but the highligth is this: When asked why Hegseth made a public statementthat had no legal effect and that did not reflect the immediate intent of DoW, counsel stated, “I don't know.”This is just the first court and will likely be escalated further up the judicial system. This is still developing and apparently the Pentagon declared Anthropic a supply chain risk under two different statutes, and this only affects one of them. So while it's good news, it's not over yet. Voice & Audio Explosion: Three Releases in One HourI had to hit the breaking news button mid-TLDR because three major voice releases dropped simultaneously during the show.Mistral Voxtral TTS — Mistral's first text-to-speech model, 3 billion parameters, open weight. They claim it beats ElevenLabs Flash v2.5 in human preference tests (58% win rate on flagship voices, 68% on zero-shot voice cloning). We tested it live on the show — it's decent, with emotion controls for neutral, happy, and frustrated voices. I was not super impressed tbh, it sits somewhere between the very good big labs TTS and the very small open source 82M param TTS. Cohere Transcribe — Cohere enters the ASR game with a 2 billion parameter open-source model (Apache 2.0!) that immediately grabbed the #1 spot on HuggingFace's Open ASR Leaderboard with a 5.42% word error rate, beating Whisper Large v3's 7.44%. In human evaluations, it wins 61% of the time on average, and 64% specifically against Whisper. For anyone in regulated industries needing local inference for compliance, this could genuinely replace Whisper as the default.Google Lyria 3 Pro — Google's most advanced music model is here.It can now generate full 3-minute tracks with structural control — intros, verses, choruses, bridges. We generated a ThursdAI opening theme live on the show using Producer AI, and it was... honestly not bad? It followed our instructions perfectly: drum and bass, 174 BPM, high energy podcast opener with vocals and introduction. The instruction-following was spot on. Nisten said it's the best music generation model right now. It's available to Gemini subscribers and via Producer AI and gemini, and it can even compose music from images. SynthID watermarked, royalty-free. We might actually use one of the generated tracks as a new show opener.The craziest thing is, since Google acquired Composer, the team has been shipping. I only generated the audio during the live show, but now went back there to download it for you guys, and whoah, it can now generate whole clips by using other Google tech, this is really cool! OpenAI kills SORA (and Atlas?)Last week we reported on about OpenAI's focus shift towards Codex and productivity, and this week we see the first casualty. OpenAI is killing SORA, the app, the Sora 2 and Sora 2 pro models and APIs. Many AI haters are celebrating this as through “ai videos” is dead, but honestly, this is obviously about the GPU power and the other things OpenAI needs to do to win the fight against Anthropic. OpenAI is also apparently going to IPO this year (like Anthropic) and they absolutely need to win the productivity/agents in enterprise market. As part of this shut down, the Disney + OpenAI partnership, is also dissolving, and Disney will no longer invest 1B into OpenAI.So, say bye bye to having digital selfies with Sam Altman. I've generated this SORA vid to hear from Sam himself: Atlas browser, OpenAI's native browser endeavor is supposedly also going to transform, together with Codex and OpenAI native app into one super app that includes all three according to the same memo. AGI is here according to Jensen, AGI is far away, according to ARC-AGI-3 The back to back this week can give anyone whiplash. First, Lex Friedman had Jensen Huang on the podcast, and asked him a very specific “WhenAGI” question, to which Jensen said “I believe it's already here” Then just a few short days layer, ArcPrize, released the 3rd version of Arc-AGI, Arc-AGI 3 a series of puzzle games, where humans get 100% pass-rate and the current LLM, top tier frontier LLMs, are getting less than 1%! It's an interactive, agentic reasoning benchmark designed to test human-like generalization and intelligence in novel, abstract, turn-based environments.The puzzles all look simple enough to do, and are actually fun, and while the wild claims of “AGI is not here yet” from the ArcPrize folks are quite interesting. The stated goal of the foudation is to release evaluations that are completely un-saturated, and this seems like one such thing at first glance. There's a bit of a debate in the community about the way Arc Prize went about this specific benchmark (no harnesses, raw LLM outputs), saying that humans got a “game” while the LLMs get just raw JSON and minimal and no extra tools. For context, a agentic harness startup claims to have solved 35% already of the games in ArcAGI, but that result is unverified and self reported, becuase they are an agentic harness, which ArcAGI apparently disqualifies. AI Art and DiffusionI wanted to finish but I think these are important releases so I'll include them briefly. Luma Labs Uni-1 — thinks and generates pixels simultaneously, #1 human preference Elo (X, Announcement)This was a surprising release, we previously seen Luma Labs do video, but this time they are posting their Uni-1 which is a… image model but it's based on an LLM, so you talk to it, iterate together until you get results. Yes, Nano Banana via AI studio is kind of like this as well ,but Uni feels a bit different. It can also generate infographics, which I haven't tried yet. You can try Uni herePhota Labs launches Phota Studio + API — a photography-focused image model with identity-preserving personalization (X, try it)There's tons of photo startups, but this one looks kind of crazy! You upload a bunch of your pictures, they train a “model” for you, and then you can create a whole bunch of images, and they do actually resemble you. Yes, Nano Banana can take a few reference pictures, but this somehow seems more accurate! You can create professional photos, fix photos you like, add others to your photos. I do feel there's a jump in capabilities here, specifically because of the personalization! Give them a try if you're not worried about them training on your pics and let me know.Modular made Flux.2 run in

This is a recap of the top 10 posts on Hacker News on March 10, 2026. This podcast was generated by wondercraft.ai (00:30): Tony Hoare has diedOriginal post: https://news.ycombinator.com/item?id=47324054&utm_source=wondercraft_ai(01:54): Online age-verification tools for child safety are surveilling adultsOriginal post: https://news.ycombinator.com/item?id=47322635&utm_source=wondercraft_ai(03:19): After outages, Amazon to make senior engineers sign off on AI-assisted changesOriginal post: https://news.ycombinator.com/item?id=47323017&utm_source=wondercraft_ai(04:44): Meta acquires MoltbookOriginal post: https://news.ycombinator.com/item?id=47323900&utm_source=wondercraft_ai(06:09): I put my whole life into a single databaseOriginal post: https://news.ycombinator.com/item?id=47321233&utm_source=wondercraft_ai(07:34): Yann LeCun raises $1B to build AI that understands the physical worldOriginal post: https://news.ycombinator.com/item?id=47320600&utm_source=wondercraft_ai(08:59): Redox OS has adopted a Certificate of Origin policy and a strict no-LLM policyOriginal post: https://news.ycombinator.com/item?id=47320661&utm_source=wondercraft_ai(10:24): Show HN: How I topped the HuggingFace open LLM leaderboard on two gaming GPUsOriginal post: https://news.ycombinator.com/item?id=47322887&utm_source=wondercraft_ai(11:49): Two Years of Emacs SoloOriginal post: https://news.ycombinator.com/item?id=47317616&utm_source=wondercraft_ai(13:14): Debian decides not to decide on AI-generated contributionsOriginal post: https://news.ycombinator.com/item?id=47324087&utm_source=wondercraft_aiThis is a third-party project, independent from HN and YC. Text and audio generated using AI, by wondercraft.ai. Create your own studio quality podcast with text as the only input in seconds at app.wondercraft.ai. Issues or feedback? We'd love to hear from you: team@wondercraft.ai

In episode 276 of our SAP on Azure video podcast we talk about RPT-1. On of the new foundational models from SAP. One of the highlights at SAP TechEd last year was the announcement of two foundational new models, among them the RPT-1 or Rapid One. For me this was quite interesting, because SAP not only announce the model, but immediately made it available for testing on BTP and HuggingFace. Our amazing colleauge Amit Lal immediately jumped on it and explored the functionality. Today I am happy to have him with us again to talk about his experience with RPT-1. Link to the repo on GitHub from Amit: SAP RPT-1-OSS | AI-Powered Enterprise ML Suite: https://github.com/amitlals/SAP-RPT-1-OSS-App-playgroundFind all the links mentioned here: https://www.saponazurepodcast.de/episode276Reach out to us for any feedback / questions:* Goran Condric: https://www.linkedin.com/in/gorancondric/* Holger Bruchelt: https://www.linkedin.com/in/holger-bruchelt/ #Microsoft #SAP #Azure #SAPonAzure #RPT1 #AI #Copilot

Happy New Year! You may have noticed that in 2025 we had moved toward YouTube as our primary podcasting platform. As we'll explain in the next State of Latent Space post, we'll be doubling down on Substack again and improving the experience for the over 100,000 of you who look out for our emails and website updates!We first mentioned Artificial Analysis in 2024, when it was still a side project in a Sydney basement. They then were one of the few Nat Friedman and Daniel Gross' AIGrant companies to raise a full seed round from them and have now become the independent gold standard for AI benchmarking—trusted by developers, enterprises, and every major lab to navigate the exploding landscape of models, providers, and capabilities.We have chatted with both Clementine Fourrier of HuggingFace's OpenLLM Leaderboard and (the freshly valued at $1.7B) Anastasios Angelopoulos of LMArena on their approaches to LLM evals and trendspotting, but Artificial Analysis have staked out an enduring and important place in the toolkit of the modern AI Engineer by doing the best job of independently running the most comprehensive set of evals across the widest range of open and closed models, and charting their progress for broad industry analyst use.George Cameron and Micah-Hill Smith have spent two years building Artificial Analysis into the platform that answers the questions no one else will: Which model is actually best for your use case? What are the real speed-cost trade-offs? And how open is “open” really?We discuss:* The origin story: built as a side project in 2023 while Micah was building a legal AI assistant, launched publicly in January 2024, and went viral after Swyx's retweet* Why they run evals themselves: labs prompt models differently, cherry-pick chain-of-thought examples (Google Gemini 1.0 Ultra used 32-shot prompts to beat GPT-4 on MMLU), and self-report inflated numbers* The mystery shopper policy: they register accounts not on their own domain and run intelligence + performance benchmarks incognito to prevent labs from serving different models on private endpoints* How they make money: enterprise benchmarking insights subscription (standardized reports on model deployment, serverless vs. managed vs. leasing chips) and private custom benchmarking for AI companies (no one pays to be on the public leaderboard)* The Intelligence Index (V3): synthesizes 10 eval datasets (MMLU, GPQA, agentic benchmarks, long-context reasoning) into a single score, with 95% confidence intervals via repeated runs* Omissions Index (hallucination rate): scores models from -100 to +100 (penalizing incorrect answers, rewarding ”I don't know”), and Claude models lead with the lowest hallucination rates despite not always being the smartest* GDP Val AA: their version of OpenAI's GDP-bench (44 white-collar tasks with spreadsheets, PDFs, PowerPoints), run through their Stirrup agent harness (up to 100 turns, code execution, web search, file system), graded by Gemini 3 Pro as an LLM judge (tested extensively, no self-preference bias)* The Openness Index: scores models 0-18 on transparency of pre-training data, post-training data, methodology, training code, and licensing (AI2 OLMo 2 leads, followed by Nous Hermes and NVIDIA Nemotron)* The smiling curve of AI costs: GPT-4-level intelligence is 100-1000x cheaper than at launch (thanks to smaller models like Amazon Nova), but frontier reasoning models in agentic workflows cost more than ever (sparsity, long context, multi-turn agents)* Why sparsity might go way lower than 5%: GPT-4.5 is ~5% active, Gemini models might be ~3%, and Omissions Index accuracy correlates with total parameters (not active), suggesting massive sparse models are the future* Token efficiency vs. turn efficiency: GPT-5 costs more per token but solves Tau-bench in fewer turns (cheaper overall), and models are getting better at using more tokens only when needed (5.1 Codex has tighter token distributions)* V4 of the Intelligence Index coming soon: adding GDP Val AA, Critical Point, hallucination rate, and dropping some saturated benchmarks (human-eval-style coding is now trivial for small models)Links to Artificial Analysis* Website: https://artificialanalysis.ai* George Cameron on X: https://x.com/georgecameron* Micah-Hill Smith on X: https://x.com/micahhsmithFull Episode on YouTubeTimestamps* 00:00 Introduction: Full Circle Moment and Artificial Analysis Origins* 01:19 Business Model: Independence and Revenue Streams* 04:33 Origin Story: From Legal AI to Benchmarking Need* 16:22 AI Grant and Moving to San Francisco* 19:21 Intelligence Index Evolution: From V1 to V3* 11:47 Benchmarking Challenges: Variance, Contamination, and Methodology* 13:52 Mystery Shopper Policy and Maintaining Independence* 28:01 New Benchmarks: Omissions Index for Hallucination Detection* 33:36 Critical Point: Hard Physics Problems and Research-Level Reasoning* 23:01 GDP Val AA: Agentic Benchmark for Real Work Tasks* 50:19 Stirrup Agent Harness: Open Source Agentic Framework* 52:43 Openness Index: Measuring Model Transparency Beyond Licenses* 58:25 The Smiling Curve: Cost Falling While Spend Rising* 1:02:32 Hardware Efficiency: Blackwell Gains and Sparsity Limits* 1:06:23 Reasoning Models and Token Efficiency: The Spectrum Emerges* 1:11:00 Multimodal Benchmarking: Image, Video, and Speech Arenas* 1:15:05 Looking Ahead: Intelligence Index V4 and Future Directions* 1:16:50 Closing: The Insatiable Demand for IntelligenceTranscriptMicah [00:00:06]: This is kind of a full circle moment for us in a way, because the first time artificial analysis got mentioned on a podcast was you and Alessio on Latent Space. Amazing.swyx [00:00:17]: Which was January 2024. I don't even remember doing that, but yeah, it was very influential to me. Yeah, I'm looking at AI News for Jan 17, or Jan 16, 2024. I said, this gem of a models and host comparison site was just launched. And then I put in a few screenshots, and I said, it's an independent third party. It clearly outlines the quality versus throughput trade-off, and it breaks out by model and hosting provider. I did give you s**t for missing fireworks, and how do you have a model benchmarking thing without fireworks? But you had together, you had perplexity, and I think we just started chatting there. Welcome, George and Micah, to Latent Space. I've been following your progress. Congrats on... It's been an amazing year. You guys have really come together to be the presumptive new gardener of AI, right? Which is something that...George [00:01:09]: Yeah, but you can't pay us for better results.swyx [00:01:12]: Yes, exactly.George [00:01:13]: Very important.Micah [00:01:14]: Start off with a spicy take.swyx [00:01:18]: Okay, how do I pay you?Micah [00:01:20]: Let's get right into that.swyx [00:01:21]: How do you make money?Micah [00:01:24]: Well, very happy to talk about that. So it's been a big journey the last couple of years. Artificial analysis is going to be two years old in January 2026. Which is pretty soon now. We first run the website for free, obviously, and give away a ton of data to help developers and companies navigate AI and make decisions about models, providers, technologies across the AI stack for building stuff. We're very committed to doing that and tend to keep doing that. We have, along the way, built a business that is working out pretty sustainably. We've got just over 20 people now and two main customer groups. So we want to be... We want to be who enterprise look to for data and insights on AI, so we want to help them with their decisions about models and technologies for building stuff. And then on the other side, we do private benchmarking for companies throughout the AI stack who build AI stuff. So no one pays to be on the website. We've been very clear about that from the very start because there's no use doing what we do unless it's independent AI benchmarking. Yeah. But turns out a bunch of our stuff can be pretty useful to companies building AI stuff.swyx [00:02:38]: And is it like, I am a Fortune 500, I need advisors on objective analysis, and I call you guys and you pull up a custom report for me, you come into my office and give me a workshop? What kind of engagement is that?George [00:02:53]: So we have a benchmarking and insight subscription, which looks like standardized reports that cover key topics or key challenges enterprises face when looking to understand AI and choose between all the technologies. And so, for instance, one of the report is a model deployment report, how to think about choosing between serverless inference, managed deployment solutions, or leasing chips. And running inference yourself is an example kind of decision that big enterprises face, and it's hard to reason through, like this AI stuff is really new to everybody. And so we try and help with our reports and insight subscription. Companies navigate that. We also do custom private benchmarking. And so that's very different from the public benchmarking that we publicize, and there's no commercial model around that. For private benchmarking, we'll at times create benchmarks, run benchmarks to specs that enterprises want. And we'll also do that sometimes for AI companies who have built things, and we help them understand what they've built with private benchmarking. Yeah. So that's a piece mainly that we've developed through trying to support everybody publicly with our public benchmarks. Yeah.swyx [00:04:09]: Let's talk about TechStack behind that. But okay, I'm going to rewind all the way to when you guys started this project. You were all the way in Sydney? Yeah. Well, Sydney, Australia for me.Micah [00:04:19]: George was an SF, but he's Australian, but he moved here already. Yeah.swyx [00:04:22]: And I remember I had the Zoom call with you. What was the impetus for starting artificial analysis in the first place? You know, you started with public benchmarks. And so let's start there. We'll go to the private benchmark. Yeah.George [00:04:33]: Why don't we even go back a little bit to like why we, you know, thought that it was needed? Yeah.Micah [00:04:40]: The story kind of begins like in 2022, 2023, like both George and I have been into AI stuff for quite a while. In 2023 specifically, I was trying to build a legal AI research assistant. So it actually worked pretty well for its era, I would say. Yeah. Yeah. So I was finding that the more you go into building something using LLMs, the more each bit of what you're doing ends up being a benchmarking problem. So had like this multistage algorithm thing, trying to figure out what the minimum viable model for each bit was, trying to optimize every bit of it as you build that out, right? Like you're trying to think about accuracy, a bunch of other metrics and performance and cost. And mostly just no one was doing anything to independently evaluate all the models. And certainly not to look at the trade-offs for speed and cost. So we basically set out just to build a thing that developers could look at to see the trade-offs between all of those things measured independently across all the models and providers. Honestly, it was probably meant to be a side project when we first started doing it.swyx [00:05:49]: Like we didn't like get together and say like, Hey, like we're going to stop working on all this stuff. I'm like, this is going to be our main thing. When I first called you, I think you hadn't decided on starting a company yet.Micah [00:05:58]: That's actually true. I don't even think we'd pause like, like George had an acquittance job. I didn't quit working on my legal AI thing. Like it was genuinely a side project.George [00:06:05]: We built it because we needed it as people building in the space and thought, Oh, other people might find it useful too. So we'll buy domain and link it to the Vercel deployment that we had and tweet about it. And, but very quickly it started getting attention. Thank you, Swyx for, I think doing an initial retweet and spotlighting it there. This project that we released. And then very quickly though, it was useful to others, but very quickly it became more useful as the number of models released accelerated. We had Mixtrel 8x7B and it was a key. That's a fun one. Yeah. Like a open source model that really changed the landscape and opened up people's eyes to other serverless inference providers and thinking about speed, thinking about cost. And so that was a key. And so it became more useful quite quickly. Yeah.swyx [00:07:02]: What I love talking to people like you who sit across the ecosystem is, well, I have theories about what people want, but you have data and that's obviously more relevant. But I want to stay on the origin story a little bit more. When you started out, I would say, I think the status quo at the time was every paper would come out and they would report their numbers versus competitor numbers. And that's basically it. And I remember I did the legwork. I think everyone has some knowledge. I think there's some version of Excel sheet or a Google sheet where you just like copy and paste the numbers from every paper and just post it up there. And then sometimes they don't line up because they're independently run. And so your numbers are going to look better than... Your reproductions of other people's numbers are going to look worse because you don't hold their models correctly or whatever the excuse is. I think then Stanford Helm, Percy Liang's project would also have some of these numbers. And I don't know if there's any other source that you can cite. The way that if I were to start artificial analysis at the same time you guys started, I would have used the Luther AI's eval framework harness. Yup.Micah [00:08:06]: Yup. That was some cool stuff. At the end of the day, running these evals, it's like if it's a simple Q&A eval, all you're doing is asking a list of questions and checking if the answers are right, which shouldn't be that crazy. But it turns out there are an enormous number of things that you've got control for. And I mean, back when we started the website. Yeah. Yeah. Like one of the reasons why we realized that we had to run the evals ourselves and couldn't just take rules from the labs was just that they would all prompt the models differently. And when you're competing over a few points, then you can pretty easily get- You can put the answer into the model. Yeah. That in the extreme. And like you get crazy cases like back when I'm Googled a Gemini 1.0 Ultra and needed a number that would say it was better than GPT-4 and like constructed, I think never published like chain of thought examples. 32 of them in every topic in MLU to run it, to get the score, like there are so many things that you- They never shipped Ultra, right? That's the one that never made it up. Not widely. Yeah. Yeah. Yeah. I mean, I'm sure it existed, but yeah. So we were pretty sure that we needed to run them ourselves and just run them in the same way across all the models. Yeah. And we were, we also did certain from the start that you couldn't look at those in isolation. You needed to look at them alongside the cost and performance stuff. Yeah.swyx [00:09:24]: Okay. A couple of technical questions. I mean, so obviously I also thought about this and I didn't do it because of cost. Yep. Did you not worry about costs? Were you funded already? Clearly not, but you know. No. Well, we definitely weren't at the start.Micah [00:09:36]: So like, I mean, we're paying for it personally at the start. There's a lot of money. Well, the numbers weren't nearly as bad a couple of years ago. So we certainly incurred some costs, but we were probably in the order of like hundreds of dollars of spend across all the benchmarking that we were doing. Yeah. So nothing. Yeah. It was like kind of fine. Yeah. Yeah. These days that's gone up an enormous amount for a bunch of reasons that we can talk about. But yeah, it wasn't that bad because you can also remember that like the number of models we were dealing with was hardly any and the complexity of the stuff that we wanted to do to evaluate them was a lot less. Like we were just asking some Q&A type questions and then one specific thing was for a lot of evals initially, we were just like sampling an answer. You know, like, what's the answer for this? Like, we didn't want to go into the answer directly without letting the models think. We weren't even doing chain of thought stuff initially. And that was the most useful way to get some results initially. Yeah.swyx [00:10:33]: And so for people who haven't done this work, literally parsing the responses is a whole thing, right? Like because sometimes the models, the models can answer any way they feel fit and sometimes they actually do have the right answer, but they just returned the wrong format and they will get a zero for that unless you work it into your parser. And that involves more work. And so, I mean, but there's an open question whether you should give it points for not following your instructions on the format.Micah [00:11:00]: It depends what you're looking at, right? Because you can, if you're trying to see whether or not it can solve a particular type of reasoning problem, and you don't want to test it on its ability to do answer formatting at the same time, then you might want to use an LLM as answer extractor approach to make sure that you get the answer out no matter how unanswered. But these days, it's mostly less of a problem. Like, if you instruct a model and give it examples of what the answers should look like, it can get the answers in your format, and then you can do, like, a simple regex.swyx [00:11:28]: Yeah, yeah. And then there's other questions around, I guess, sometimes if you have a multiple choice question, sometimes there's a bias towards the first answer, so you have to randomize the responses. All these nuances, like, once you dig into benchmarks, you're like, I don't know how anyone believes the numbers on all these things. It's so dark magic.Micah [00:11:47]: You've also got, like… You've got, like, the different degrees of variance in different benchmarks, right? Yeah. So, if you run four-question multi-choice on a modern reasoning model at the temperatures suggested by the labs for their own models, the variance that you can see on a four-question multi-choice eval is pretty enormous if you only do a single run of it and it has a small number of questions, especially. So, like, one of the things that we do is run an enormous number of all of our evals when we're developing new ones and doing upgrades to our intelligence index to bring in new things. Yeah. So, that we can dial in the right number of repeats so that we can get to the 95% confidence intervals that we're comfortable with so that when we pull that together, we can be confident in intelligence index to at least as tight as, like, a plus or minus one at a 95% confidence. Yeah.swyx [00:12:32]: And, again, that just adds a straight multiple to the cost. Oh, yeah. Yeah, yeah.George [00:12:37]: So, that's one of many reasons that cost has gone up a lot more than linearly over the last couple of years. We report a cost to run the artificial analysis. We report a cost to run the artificial analysis intelligence index on our website, and currently that's assuming one repeat in terms of how we report it because we want to reflect a bit about the weighting of the index. But our cost is actually a lot higher than what we report there because of the repeats.swyx [00:13:03]: Yeah, yeah, yeah. And probably this is true, but just checking, you don't have any special deals with the labs. They don't discount it. You just pay out of pocket or out of your sort of customer funds. Oh, there is a mix. So, the issue is that sometimes they may give you a special end point, which is… Ah, 100%.Micah [00:13:21]: Yeah, yeah, yeah. Exactly. So, we laser focus, like, on everything we do on having the best independent metrics and making sure that no one can manipulate them in any way. There are quite a lot of processes we've developed over the last couple of years to make that true for, like, the one you bring up, like, right here of the fact that if we're working with a lab, if they're giving us a private endpoint to evaluate a model, that it is totally possible. That what's sitting behind that black box is not the same as they serve on a public endpoint. We're very aware of that. We have what we call a mystery shopper policy. And so, and we're totally transparent with all the labs we work with about this, that we will register accounts not on our own domain and run both intelligence evals and performance benchmarks… Yeah, that's the job. …without them being able to identify it. And no one's ever had a problem with that. Because, like, a thing that turns out to actually be quite a good… …good factor in the industry is that they all want to believe that none of their competitors could manipulate what we're doing either.swyx [00:14:23]: That's true. I never thought about that. I've been in the database data industry prior, and there's a lot of shenanigans around benchmarking, right? So I'm just kind of going through the mental laundry list. Did I miss anything else in this category of shenanigans? Oh, potential shenanigans.Micah [00:14:36]: I mean, okay, the biggest one, like, that I'll bring up, like, is more of a conceptual one, actually, than, like, direct shenanigans. It's that the things that get measured become things that get targeted by labs that they're trying to build, right? Exactly. So that doesn't mean anything that we should really call shenanigans. Like, I'm not talking about training on test set. But if you know that you're going to be great at another particular thing, if you're a researcher, there are a whole bunch of things that you can do to try to get better at that thing that preferably are going to be helpful for a wide range of how actual users want to use the thing that you're building. But will not necessarily work. Will not necessarily do that. So, for instance, the models are exceptional now at answering competition maths problems. There is some relevance of that type of reasoning, that type of work, to, like, how we might use modern coding agents and stuff. But it's clearly not one for one. So the thing that we have to be aware of is that once an eval becomes the thing that everyone's looking at, scores can get better on it without there being a reflection of overall generalized intelligence of these models. Getting better. That has been true for the last couple of years. It'll be true for the next couple of years. There's no silver bullet to defeat that other than building new stuff to stay relevant and measure the capabilities that matter most to real users. Yeah.swyx [00:15:58]: And we'll cover some of the new stuff that you guys are building as well, which is cool. Like, you used to just run other people's evals, but now you're coming up with your own. And I think, obviously, that is a necessary path once you're at the frontier. You've exhausted all the existing evals. I think the next point in history that I have for you is AI Grant that you guys decided to join and move here. What was it like? I think you were in, like, batch two? Batch four. Batch four. Okay.Micah [00:16:26]: I mean, it was great. Nat and Daniel are obviously great. And it's a really cool group of companies that we were in AI Grant alongside. It was really great to get Nat and Daniel on board. Obviously, they've done a whole lot of great work in the space with a lot of leading companies and were extremely aligned. With the mission of what we were trying to do. Like, we're not quite typical of, like, a lot of the other AI startups that they've invested in.swyx [00:16:53]: And they were very much here for the mission of what we want to do. Did they say any advice that really affected you in some way or, like, were one of the events very impactful? That's an interesting question.Micah [00:17:03]: I mean, I remember fondly a bunch of the speakers who came and did fireside chats at AI Grant.swyx [00:17:09]: Which is also, like, a crazy list. Yeah.George [00:17:11]: Oh, totally. Yeah, yeah, yeah. There was something about, you know, speaking to Nat and Daniel about the challenges of working through a startup and just working through the questions that don't have, like, clear answers and how to work through those kind of methodically and just, like, work through the hard decisions. And they've been great mentors to us as we've built artificial analysis. Another benefit for us was that other companies in the batch and other companies in AI Grant are pushing the capabilities. Yeah. And I think that's a big part of what AI can do at this time. And so being in contact with them, making sure that artificial analysis is useful to them has been fantastic for supporting us in working out how should we build out artificial analysis to continue to being useful to those, like, you know, building on AI.swyx [00:17:59]: I think to some extent, I'm mixed opinion on that one because to some extent, your target audience is not people in AI Grants who are obviously at the frontier. Yeah. Do you disagree?Micah [00:18:09]: To some extent. To some extent. But then, so a lot of what the AI Grant companies are doing is taking capabilities coming out of the labs and trying to push the limits of what they can do across the entire stack for building great applications, which actually makes some of them pretty archetypical power users of artificial analysis. Some of the people with the strongest opinions about what we're doing well and what we're not doing well and what they want to see next from us. Yeah. Yeah. Because when you're building any kind of AI application now, chances are you're using a whole bunch of different models. You're maybe switching reasonably frequently for different models and different parts of your application to optimize what you're able to do with them at an accuracy level and to get better speed and cost characteristics. So for many of them, no, they're like not commercial customers of ours, like we don't charge for all our data on the website. Yeah. They are absolutely some of our power users.swyx [00:19:07]: So let's talk about just the evals as well. So you start out from the general like MMU and GPQA stuff. What's next? How do you sort of build up to the overall index? What was in V1 and how did you evolve it? Okay.Micah [00:19:22]: So first, just like background, like we're talking about the artificial analysis intelligence index, which is our synthesis metric that we pulled together currently from 10 different eval data sets to give what? We're pretty much the same as that. Pretty confident is the best single number to look at for how smart the models are. Obviously, it doesn't tell the whole story. That's why we published the whole website of all the charts to dive into every part of it and look at the trade-offs. But best single number. So right now, it's got a bunch of Q&A type data sets that have been very important to the industry, like a couple that you just mentioned. It's also got a couple of agentic data sets. It's got our own long context reasoning data set and some other use case focused stuff. As time goes on. The things that we're most interested in that are going to be important to the capabilities that are becoming more important for AI, what developers are caring about, are going to be first around agentic capabilities. So surprise, surprise. We're all loving our coding agents and how the model is going to perform like that and then do similar things for different types of work are really important to us. The linking to use cases to economically valuable use cases are extremely important to us. And then we've got some of the. Yeah. These things that the models still struggle with, like working really well over long contexts that are not going to go away as specific capabilities and use cases that we need to keep evaluating.swyx [00:20:46]: But I guess one thing I was driving was like the V1 versus the V2 and how bad it was over time.Micah [00:20:53]: Like how we've changed the index to where we are.swyx [00:20:55]: And I think that reflects on the change in the industry. Right. So that's a nice way to tell that story.Micah [00:21:00]: Well, V1 would be completely saturated right now. Almost every model coming out because doing things like writing the Python functions and human evil is now pretty trivial. It's easy to forget, actually, I think how much progress has been made in the last two years. Like we obviously play the game constantly of like the today's version versus last week's version and the week before and all of the small changes in the horse race between the current frontier and who has the best like smaller than 10B model like right now this week. Right. And that's very important to a lot of developers and people and especially in this particular city of San Francisco. But when you zoom out a couple of years ago, literally most of what we were doing to evaluate the models then would all be 100% solved by even pretty small models today. And that's been one of the key things, by the way, that's driven down the cost of intelligence at every tier of intelligence. We can talk about more in a bit. So V1, V2, V3, we made things harder. We covered a wider range of use cases. And we tried to get closer to things developers care about as opposed to like just the Q&A type stuff that MMLU and GPQA represented. Yeah.swyx [00:22:12]: I don't know if you have anything to add there. Or we could just go right into showing people the benchmark and like looking around and asking questions about it. Yeah.Micah [00:22:21]: Let's do it. Okay. This would be a pretty good way to chat about a few of the new things we've launched recently. Yeah.George [00:22:26]: And I think a little bit about the direction that we want to take it. And we want to push benchmarks. Currently, the intelligence index and evals focus a lot on kind of raw intelligence. But we kind of want to diversify how we think about intelligence. And we can talk about it. But kind of new evals that we've kind of built and partnered on focus on topics like hallucination. And we've got a lot of topics that I think are not covered by the current eval set that should be. And so we want to bring that forth. But before we get into that.swyx [00:23:01]: And so for listeners, just as a timestamp, right now, number one is Gemini 3 Pro High. Then followed by Cloud Opus at 70. Just 5.1 high. You don't have 5.2 yet. And Kimi K2 Thinking. Wow. Still hanging in there. So those are the top four. That will date this podcast quickly. Yeah. Yeah. I mean, I love it. I love it. No, no. 100%. Look back this time next year and go, how cute. Yep.George [00:23:25]: Totally. A quick view of that is, okay, there's a lot. I love it. I love this chart. Yeah.Micah [00:23:30]: This is such a favorite, right? Yeah. And almost every talk that George or I give at conferences and stuff, we always put this one up first to just talk about situating where we are in this moment in history. This, I think, is the visual version of what I was saying before about the zooming out and remembering how much progress there's been. If we go back to just over a year ago, before 01, before Cloud Sonnet 3.5, we didn't have reasoning models or coding agents as a thing. And the game was very, very different. If we go back even a little bit before then, we're in the era where, when you look at this chart, open AI was untouchable for well over a year. And, I mean, you would remember that time period well of there being very open questions about whether or not AI was going to be competitive, like full stop, whether or not open AI would just run away with it, whether we would have a few frontier labs and no one else would really be able to do anything other than consume their APIs. I am quite happy overall that the world that we have ended up in is one where... Multi-model. Absolutely. And strictly more competitive every quarter over the last few years. Yeah. This year has been insane. Yeah.George [00:24:42]: You can see it. This chart with everything added is hard to read currently. There's so many dots on it, but I think it reflects a little bit what we felt, like how crazy it's been.swyx [00:24:54]: Why 14 as the default? Is that a manual choice? Because you've got service now in there that are less traditional names. Yeah.George [00:25:01]: It's models that we're kind of highlighting by default in our charts, in our intelligence index. Okay.swyx [00:25:07]: You just have a manually curated list of stuff.George [00:25:10]: Yeah, that's right. But something that I actually don't think every artificial analysis user knows is that you can customize our charts and choose what models are highlighted. Yeah. And so if we take off a few names, it gets a little easier to read.swyx [00:25:25]: Yeah, yeah. A little easier to read. Totally. Yeah. But I love that you can see the all one jump. Look at that. September 2024. And the DeepSeek jump. Yeah.George [00:25:34]: Which got close to OpenAI's leadership. They were so close. I think, yeah, we remember that moment. Around this time last year, actually.Micah [00:25:44]: Yeah, yeah, yeah. I agree. Yeah, well, a couple of weeks. It was Boxing Day in New Zealand when DeepSeek v3 came out. And we'd been tracking DeepSeek and a bunch of the other global players that were less known over the second half of 2024 and had run evals on the earlier ones and stuff. I very distinctly remember Boxing Day in New Zealand, because I was with family for Christmas and stuff, running the evals and getting back result by result on DeepSeek v3. So this was the first of their v3 architecture, the 671b MOE.Micah [00:26:19]: And we were very, very impressed. That was the moment where we were sure that DeepSeek was no longer just one of many players, but had jumped up to be a thing. The world really noticed when they followed that up with the RL working on top of v3 and R1 succeeding a few weeks later. But the groundwork for that absolutely was laid with just extremely strong base model, completely open weights that we had as the best open weights model. So, yeah, that's the thing that you really see in the game. But I think that we got a lot of good feedback on Boxing Day. us on Boxing Day last year.George [00:26:48]: Boxing Day is the day after Christmas for those not familiar.George [00:26:54]: I'm from Singapore.swyx [00:26:55]: A lot of us remember Boxing Day for a different reason, for the tsunami that happened. Oh, of course. Yeah, but that was a long time ago. So yeah. So this is the rough pitch of AAQI. Is it A-A-Q-I or A-A-I-I? I-I. Okay. Good memory, though.Micah [00:27:11]: I don't know. I'm not used to it. Once upon a time, we did call it Quality Index, and we would talk about quality, performance, and price, but we changed it to intelligence.George [00:27:20]: There's been a few naming changes. We added hardware benchmarking to the site, and so benchmarks at a kind of system level. And so then we changed our throughput metric to, we now call it output speed, and thenswyx [00:27:32]: throughput makes sense at a system level, so we took that name. Take me through more charts. What should people know? Obviously, the way you look at the site is probably different than how a beginner might look at it.Micah [00:27:42]: Yeah, that's fair. There's a lot of fun stuff to dive into. Maybe so we can hit past all the, like, we have lots and lots of emails and stuff. The interesting ones to talk about today that would be great to bring up are a few of our recent things, I think, that probably not many people will be familiar with yet. So first one of those is our omniscience index. So this one is a little bit different to most of the intelligence evils that we've run. We built it specifically to look at the embedded knowledge in the models and to test hallucination by looking at when the model doesn't know the answer, so not able to get it correct, what's its probability of saying, I don't know, or giving an incorrect answer. So the metric that we use for omniscience goes from negative 100 to positive 100. Because we're simply taking off a point if you give an incorrect answer to the question. We're pretty convinced that this is an example of where it makes most sense to do that, because it's strictly more helpful to say, I don't know, instead of giving a wrong answer to factual knowledge question. And one of our goals is to shift the incentive that evils create for models and the labs creating them to get higher scores. And almost every evil across all of AI up until this point, it's been graded by simple percentage correct as the main metric, the main thing that gets hyped. And so you should take a shot at everything. There's no incentive to say, I don't know. So we did that for this one here.swyx [00:29:22]: I think there's a general field of calibration as well, like the confidence in your answer versus the rightness of the answer. Yeah, we completely agree. Yeah. Yeah.George [00:29:31]: On that. And one reason that we didn't do that is because. Or put that into this index is that we think that the, the way to do that is not to ask the models how confident they are.swyx [00:29:43]: I don't know. Maybe it might be though. You put it like a JSON field, say, say confidence and maybe it spits out something. Yeah. You know, we have done a few evils podcasts over the, over the years. And when we did one with Clementine of hugging face, who maintains the open source leaderboard, and this was one of her top requests, which is some kind of hallucination slash lack of confidence calibration thing. And so, Hey, this is one of them.Micah [00:30:05]: And I mean, like anything that we do, it's not a perfect metric or the whole story of everything that you think about as hallucination. But yeah, it's pretty useful and has some interesting results. Like one of the things that we saw in the hallucination rate is that anthropics Claude models at the, the, the very left-hand side here with the lowest hallucination rates out of the models that we've evaluated amnesty is on. That is an interesting fact. I think it probably correlates with a lot of the previously, not really measured vibes stuff that people like about some of the Claude models. Is the dataset public or what's is it, is there a held out set? There's a hell of a set for this one. So we, we have published a public test set, but we we've only published 10% of it. The reason is that for this one here specifically, it would be very, very easy to like have data contamination because it is just factual knowledge questions. We would. We'll update it at a time to also prevent that, but with yeah, kept most of it held out so that we can keep it reliable for a long time. It leads us to a bunch of really cool things, including breakdown quite granularly by topic. And so we've got some of that disclosed on the website publicly right now, and there's lots more coming in terms of our ability to break out very specific topics. Yeah.swyx [00:31:23]: I would be interested. Let's, let's dwell a little bit on this hallucination one. I noticed that Haiku hallucinates less than Sonnet hallucinates less than Opus. And yeah. Would that be the other way around in a normal capability environments? I don't know. What's, what do you make of that?George [00:31:37]: One interesting aspect is that we've found that there's not really a, not a strong correlation between intelligence and hallucination, right? That's to say that the smarter the models are in a general sense, isn't correlated with their ability to, when they don't know something, say that they don't know. It's interesting that Gemini three pro preview was a big leap over here. Gemini 2.5. Flash and, and, and 2.5 pro, but, and if I add pro quickly here.swyx [00:32:07]: I bet pro's really good. Uh, actually no, I meant, I meant, uh, the GPT pros.George [00:32:12]: Oh yeah.swyx [00:32:13]: Cause GPT pros are rumored. We don't know for a fact that it's like eight runs and then with the LM judge on top. Yeah.George [00:32:20]: So we saw a big jump in, this is accuracy. So this is just percent that they get, uh, correct and Gemini three pro knew a lot more than the other models. And so big jump in accuracy. But relatively no change between the Google Gemini models, between releases. And the hallucination rate. Exactly. And so it's likely due to just kind of different post-training recipe, between the, the Claude models. Yeah.Micah [00:32:45]: Um, there's, there's driven this. Yeah. You can, uh, you can partially blame us and how we define intelligence having until now not defined hallucination as a negative in the way that we think about intelligence.swyx [00:32:56]: And so that's what we're changing. Uh, I know many smart people who are confidently incorrect.George [00:33:02]: Uh, look, look at that. That, that, that is very humans. Very true. And there's times and a place for that. I think our view is that hallucination rate makes sense in this context where it's around knowledge, but in many cases, people want the models to hallucinate, to have a go. Often that's the case in coding or when you're trying to generate newer ideas. One eval that we added to artificial analysis is, is, is critical point and it's really hard, uh, physics problems. Okay.swyx [00:33:32]: And is it sort of like a human eval type or something different or like a frontier math type?George [00:33:37]: It's not dissimilar to frontier frontier math. So these are kind of research questions that kind of academics in the physics physics world would be able to answer, but models really struggled to answer. So the top score here is not 9%.swyx [00:33:51]: And when the people that, that created this like Minway and, and, and actually off via who was kind of behind sweep and what organization is this? Oh, is this, it's Princeton.George [00:34:01]: Kind of range of academics from, from, uh, different academic institutions, really smart people. They talked about how they turn the models up in terms of the temperature as high temperature as they can, where they're trying to explore kind of new ideas in physics as a, as a thought partner, just because they, they want the models to hallucinate. Um, yeah, sometimes it's something new. Yeah, exactly.swyx [00:34:21]: Um, so not right in every situation, but, um, I think it makes sense, you know, to test hallucination in scenarios where it makes sense. Also, the obvious question is, uh, this is one of. Many that there is there, every lab has a system card that shows some kind of hallucination number, and you've chosen to not, uh, endorse that and you've made your own. And I think that's a, that's a choice. Um, totally in some sense, the rest of artificial analysis is public benchmarks that other people can independently rerun. You provide it as a service here. You have to fight the, well, who are we to, to like do this? And your, your answer is that we have a lot of customers and, you know, but like, I guess, how do you converge the individual?Micah [00:35:08]: I mean, I think, I think for hallucinations specifically, there are a bunch of different things that you might care about reasonably, and that you'd measure quite differently, like we've called this a amnesty and solutionation rate, not trying to declare the, like, it's humanity's last hallucination. You could, uh, you could have some interesting naming conventions and all this stuff. Um, the biggest picture answer to that. It's something that I actually wanted to mention. Just as George was explaining, critical point as well is, so as we go forward, we are building evals internally. We're partnering with academia and partnering with AI companies to build great evals. We have pretty strong views on, in various ways for different parts of the AI stack, where there are things that are not being measured well, or things that developers care about that should be measured more and better. And we intend to be doing that. We're not obsessed necessarily with that. Everything we do, we have to do entirely within our own team. Critical point. As a cool example of where we were a launch partner for it, working with academia, we've got some partnerships coming up with a couple of leading companies. Those ones, obviously we have to be careful with on some of the independent stuff, but with the right disclosure, like we're completely comfortable with that. A lot of the labs have released great data sets in the past that we've used to great success independently. And so it's between all of those techniques, we're going to be releasing more stuff in the future. Cool.swyx [00:36:26]: Let's cover the last couple. And then we'll, I want to talk about your trends analysis stuff, you know? Totally.Micah [00:36:31]: So that actually, I have one like little factoid on omniscience. If you go back up to accuracy on omniscience, an interesting thing about this accuracy metric is that it tracks more closely than anything else that we measure. The total parameter count of models makes a lot of sense intuitively, right? Because this is a knowledge eval. This is the pure knowledge metric. We're not looking at the index and the hallucination rate stuff that we think is much more about how the models are trained. This is just what facts did they recall? And yeah, it tracks parameter count extremely closely. Okay.swyx [00:37:05]: What's the rumored size of GPT-3 Pro? And to be clear, not confirmed for any official source, just rumors. But rumors do fly around. Rumors. I get, I hear all sorts of numbers. I don't know what to trust.Micah [00:37:17]: So if you, if you draw the line on omniscience accuracy versus total parameters, we've got all the open ways models, you can squint and see that likely the leading frontier models right now are quite a lot bigger than the ones that we're seeing right now. And the one trillion parameters that the open weights models cap out at, and the ones that we're looking at here, there's an interesting extra data point that Elon Musk revealed recently about XAI that for three trillion parameters for GROK 3 and 4, 6 trillion for GROK 5, but that's not out yet. Take those together, have a look. You might reasonably form a view that there's a pretty good chance that Gemini 3 Pro is bigger than that, that it could be in the 5 to 10 trillion parameters. To be clear, I have absolutely no idea, but just based on this chart, like that's where you would, you would land if you have a look at it. Yeah.swyx [00:38:07]: And to some extent, I actually kind of discourage people from guessing too much because what does it really matter? Like as long as they can serve it as a sustainable cost, that's about it. Like, yeah, totally.George [00:38:17]: They've also got different incentives in play compared to like open weights models who are thinking to supporting others in self-deployment for the labs who are doing inference at scale. It's I think less about total parameters in many cases. When thinking about inference costs and more around number of active parameters. And so there's a bit of an incentive towards larger sparser models. Agreed.Micah [00:38:38]: Understood. Yeah. Great. I mean, obviously if you're a developer or company using these things, not exactly as you say, it doesn't matter. You should be looking at all the different ways that we measure intelligence. You should be looking at cost to run index number and the different ways of thinking about token efficiency and cost efficiency based on the list prices, because that's all it matters.swyx [00:38:56]: It's not as good for the content creator rumor mill where I can say. Oh, GPT-4 is this small circle. Look at GPT-5 is this big circle. And then there used to be a thing for a while. Yeah.Micah [00:39:07]: But that is like on its own, actually a very interesting one, right? That is it just purely that chances are the last couple of years haven't seen a dramatic scaling up in the total size of these models. And so there's a lot of room to go up properly in total size of the models, especially with the upcoming hardware generations. Yes.swyx [00:39:29]: So, you know. Taking off my shitposting face for a minute. Yes. Yes. At the same time, I do feel like, you know, especially coming back from Europe, people do feel like Ilya is probably right that the paradigm is doesn't have many more orders of magnitude to scale out more. And therefore we need to start exploring at least a different path. GDPVal, I think it's like only like a month or so old. I was also very positive when it first came out. I actually talked to Tejo, who was the lead researcher on that. Oh, cool. And you have your own version.George [00:39:59]: It's a fantastic. It's a fantastic data set. Yeah.swyx [00:40:01]: And maybe it will recap for people who are still out of it. It's like 44 tasks based on some kind of GDP cutoff that's like meant to represent broad white collar work that is not just coding. Yeah.Micah [00:40:12]: Each of the tasks have a whole bunch of detailed instructions, some input files for a lot of them. It's within the 44 is divided into like two hundred and twenty two to five, maybe subtasks that are the level of that we run through the agenda. And yeah, they're really interesting. I will say that it doesn't. It doesn't necessarily capture like all the stuff that people do at work. No avail is perfect is always going to be more things to look at, largely because in order to make the tasks well enough to find that you can run them, they need to only have a handful of input files and very specific instructions for that task. And so I think the easiest way to think about them are that they're like quite hard take home exam tasks that you might do in an interview process.swyx [00:40:56]: Yeah, for listeners, it is not no longer like a long prompt. It is like, well, here's a zip file with like a spreadsheet or a PowerPoint deck or a PDF and go nuts and answer this question.George [00:41:06]: OpenAI released a great data set and they released a good paper which looks at performance across the different web chat bots on the data set. It's a great paper, encourage people to read it. What we've done is taken that data set and turned it into an eval that can be run on any model. So we created a reference agentic harness that can run. Run the models on the data set, and then we developed evaluator approach to compare outputs. That's kind of AI enabled, so it uses Gemini 3 Pro Preview to compare results, which we tested pretty comprehensively to ensure that it's aligned to human preferences. One data point there is that even as an evaluator, Gemini 3 Pro, interestingly, doesn't do actually that well. So that's kind of a good example of what we've done in GDPVal AA.swyx [00:42:01]: Yeah, the thing that you have to watch out for with LLM judge is self-preference that models usually prefer their own output, and in this case, it was not. Totally.Micah [00:42:08]: I think the way that we're thinking about the places where it makes sense to use an LLM as judge approach now, like quite different to some of the early LLM as judge stuff a couple of years ago, because some of that and MTV was a great project that was a good example of some of this a while ago was about judging conversations and like a lot of style type stuff. Here, we've got the task that the grader and grading model is doing is quite different to the task of taking the test. When you're taking the test, you've got all of the agentic tools you're working with, the code interpreter and web search, the file system to go through many, many turns to try to create the documents. Then on the other side, when we're grading it, we're running it through a pipeline to extract visual and text versions of the files and be able to provide that to Gemini, and we're providing the criteria for the task and getting it to pick which one more effectively meets the criteria of the task. Yeah. So we've got the task out of two potential outcomes. It turns out that we proved that it's just very, very good at getting that right, matched with human preference a lot of the time, because I think it's got the raw intelligence, but it's combined with the correct representation of the outputs, the fact that the outputs were created with an agentic task that is quite different to the way the grading model works, and we're comparing it against criteria, not just kind of zero shot trying to ask the model to pick which one is better.swyx [00:43:26]: Got it. Why is this an ELO? And not a percentage, like GDP-VAL?George [00:43:31]: So the outputs look like documents, and there's video outputs or audio outputs from some of the tasks. It has to make a video? Yeah, for some of the tasks. Some of the tasks.swyx [00:43:43]: What task is that?George [00:43:45]: I mean, it's in the data set. Like be a YouTuber? It's a marketing video.Micah [00:43:49]: Oh, wow. What? Like model has to go find clips on the internet and try to put it together. The models are not that good at doing that one, for now, to be clear. It's pretty hard to do that with a code editor. I mean, the computer stuff doesn't work quite well enough and so on and so on, but yeah.George [00:44:02]: And so there's no kind of ground truth, necessarily, to compare against, to work out percentage correct. It's hard to come up with correct or incorrect there. And so it's on a relative basis. And so we use an ELO approach to compare outputs from each of the models between the task.swyx [00:44:23]: You know what you should do? You should pay a contractor, a human, to do the same task. And then give it an ELO and then so you have, you have human there. It's just, I think what's helpful about GDPVal, the OpenAI one, is that 50% is meant to be normal human and maybe Domain Expert is higher than that, but 50% was the bar for like, well, if you've crossed 50, you are superhuman. Yeah.Micah [00:44:47]: So we like, haven't grounded this score in that exactly. I agree that it can be helpful, but we wanted to generalize this to a very large number. It's one of the reasons that presenting it as ELO is quite helpful and allows us to add models and it'll stay relevant for quite a long time. I also think it, it can be tricky looking at these exact tasks compared to the human performance, because the way that you would go about it as a human is quite different to how the models would go about it. Yeah.swyx [00:45:15]: I also liked that you included Lama 4 Maverick in there. Is that like just one last, like...Micah [00:45:20]: Well, no, no, no, no, no, no, it is the, it is the best model released by Meta. And... So it makes it into the homepage default set, still for now.George [00:45:31]: Other inclusion that's quite interesting is we also ran it across the latest versions of the web chatbots. And so we have...swyx [00:45:39]: Oh, that's right.George [00:45:40]: Oh, sorry.swyx [00:45:41]: I, yeah, I completely missed that. Okay.George [00:45:43]: No, not at all. So that, which has a checkered pattern. So that is their harness, not yours, is what you're saying. Exactly. And what's really interesting is that if you compare, for instance, Claude 4.5 Opus using the Claude web chatbot, it performs worse than the model in our agentic harness. And so in every case, the model performs better in our agentic harness than its web chatbot counterpart, the harness that they created.swyx [00:46:13]: Oh, my backwards explanation for that would be that, well, it's meant for consumer use cases and here you're pushing it for something.Micah [00:46:19]: The constraints are different and the amount of freedom that you can give the model is different. Also, you like have a cost goal. We let the models work as long as they want, basically. Yeah. Do you copy paste manually into the chatbot? Yeah. Yeah. That's, that was how we got the chatbot reference. We're not going to be keeping those updated at like quite the same scale as hundreds of models.swyx [00:46:38]: Well, so I don't know, talk to a browser base. They'll, they'll automate it for you. You know, like I have thought about like, well, we should turn these chatbot versions into an API because they are legitimately different agents in themselves. Yes. Right. Yeah.Micah [00:46:53]: And that's grown a huge amount of the last year, right? Like the tools. The tools that are available have actually diverged in my opinion, a fair bit across the major chatbot apps and the amount of data sources that you can connect them to have gone up a lot, meaning that your experience and the way you're using the model is more different than ever.swyx [00:47:10]: What tools and what data connections come to mind when you say what's interesting, what's notable work that people have done?Micah [00:47:15]: Oh, okay. So my favorite example on this is that until very recently, I would argue that it was basically impossible to get an LLM to draft an email for me in any useful way. Because most times that you're sending an email, you're not just writing something for the sake of writing it. Chances are context required is a whole bunch of historical emails. Maybe it's notes that you've made, maybe it's meeting notes, maybe it's, um, pulling something from your, um, any of like wherever you at work store stuff. So for me, like Google drive, one drive, um, in our super base databases, if we need to do some analysis or some data or something, preferably model can be plugged into all of those things and can go do some useful work based on it. The things that like I find most impressive currently that I am somewhat surprised work really well in late 2025, uh, that I can have models use super base MCP to query read only, of course, run a whole bunch of SQL queries to do pretty significant data analysis. And. And make charts and stuff and can read my Gmail and my notion. And okay. You actually use that. That's good. That's, that's, that's good. Is that a cloud thing? To various degrees of order, but chat GPD and Claude right now, I would say that this stuff like barely works in fairness right now. Like.George [00:48:33]: Because people are actually going to try this after they hear it. If you get an email from Micah, odds are it wasn't written by a chatbot.Micah [00:48:38]: So, yeah, I think it is true that I have never actually sent anyone an email drafted by a chatbot. Yet.swyx [00:48:46]: Um, and so you can, you can feel it right. And yeah, this time, this time next year, we'll come back and see where it's going. Totally. Um, super base shout out another famous Kiwi. Uh, I don't know if you've, you've any conversations with him about anything in particular on AI building and AI infra.George [00:49:03]: We have had, uh, Twitter DMS, um, with, with him because we're quite big, uh, super base users and power users. And we probably do some things more manually than we should in. In, in super base support line because you're, you're a little bit being super friendly. One extra, um, point regarding, um, GDP Val AA is that on the basis of the overperformance of the models compared to the chatbots turns out, we realized that, oh, like our reference harness that we built actually white works quite well on like gen generalist agentic tasks. This proves it in a sense. And so the agent harness is very. Minimalist. I think it follows some of the ideas that are in Claude code and we, all that we give it is context management capabilities, a web search, web browsing, uh, tool, uh, code execution, uh, environment. Anything else?Micah [00:50:02]: I mean, we can equip it with more tools, but like by default, yeah, that's it. We, we, we give it for GDP, a tool to, uh, view an image specifically, um, because the models, you know, can just use a terminal to pull stuff in text form into context. But to pull visual stuff into context, we had to give them a custom tool, but yeah, exactly. Um, you, you can explain an expert. No.George [00:50:21]: So it's, it, we turned out that we created a good generalist agentic harness. And so we, um, released that on, on GitHub yesterday. It's called stirrup. So if people want to check it out and, and it's a great, um, you know, base for, you know, generalist, uh, building a generalist agent for more specific tasks.Micah [00:50:39]: I'd say the best way to use it is get clone and then have your favorite coding. Agent make changes to it, to do whatever you want, because it's not that many lines of code and the coding agents can work with it. Super well.swyx [00:50:51]: Well, that's nice for the community to explore and share and hack on it. I think maybe in, in, in other similar environments, the terminal bench guys have done, uh, sort of the Harbor. Uh, and so it's, it's a, it's a bundle of, well, we need our minimal harness, which for them is terminus and we also need the RL environments or Docker deployment thing to, to run independently. So I don't know if you've looked at it. I don't know if you've looked at the harbor at all, is that, is that like a, a standard that people want to adopt?George [00:51:19]: Yeah, we've looked at it from a evals perspective and we love terminal bench and, and host benchmarks of, of, of terminal mention on artificial analysis. Um, we've looked at it from a, from a coding agent perspective, but could see it being a great, um, basis for any kind of agents. I think where we're getting to is that these models have gotten smart enough. They've gotten better, better tools that they can perform better when just given a minimalist. Set of tools and, and let them run, let the model control the, the agentic workflow rather than using another framework that's a bit more built out that tries to dictate the, dictate the flow. Awesome.swyx [00:51:56]: Let's cover the openness index and then let's go into the report stuff. Uh, so that's the, that's the last of the proprietary art numbers, I guess. I don't know how you sort of classify all these. Yeah.Micah [00:52:07]: Or call it, call it, let's call it the last of like the, the three new things that we're talking about from like the last few weeks. Um, cause I mean, there's a, we do a mix of stuff that. Where we're using open source, where we open source and what we do and, um, proprietary stuff that we don't always open source, like long context reasoning data set last year, we did open source. Um, and then all of the work on performance benchmarks across the site, some of them, we looking to open source, but some of them, like we're constantly iterating on and so on and so on and so on. So there's a huge mix, I would say, just of like stuff that is open source and not across the side. So that's a LCR for people. Yeah, yeah, yeah, yeah.swyx [00:52:41]: Uh, but let's, let's, let's talk about open.Micah [00:52:42]: Let's talk about openness index. This. Here is call it like a new way to think about how open models are. We, for a long time, have tracked where the models are open weights and what the licenses on them are. And that's like pretty useful. That tells you what you're allowed to do with the weights of a model, but there is this whole other dimension to how open models are. That is pretty important that we haven't tracked until now. And that's how much is disclosed about how it was made. So transparency about data, pre-training data and post-training data. And whether you're allowed to use that data and transparency about methodology and training code. So basically, those are the components. We bring them together to score an openness index for models so that you can in one place get this full picture of how open models are.swyx [00:53:32]: I feel like I've seen a couple other people try to do this, but they're not maintained. I do think this does matter. I don't know what the numbers mean apart from is there a max number? Is this out of 20?George [00:53:44]: It's out of 18 currently, and so we've got an openness index page, but essentially these are points, you get points for being more open across these different categories and the maximum you can achieve is 18. So AI2 with their extremely open OMO3 32B think model is the leader in a sense.swyx [00:54:04]: It's hooking face.George [00:54:05]: Oh, with their smaller model. It's coming soon. I think we need to run, we need to get the intelligence benchmarks right to get it on the site.swyx [00:54:12]: You can't have it open in the next. We can not include hooking face. We love hooking face. We'll have that, we'll have that up very soon. I mean, you know, the refined web and all that stuff. It's, it's amazing. Or is it called fine web? Fine web. Fine web.Micah [00:54:23]: Yeah, yeah, no, totally. Yep. One of the reasons this is cool, right, is that if you're trying to understand the holistic picture of the models and what you can do with all the stuff the company's contributing, this gives you that picture. And so we are going to keep it up to date alongside all the models that we do intelligence index on, on the site. And it's just an extra view to understand.swyx [00:54:43]: Can you scroll down to this? The, the, the, the trade-offs chart. Yeah, yeah. That one. Yeah. This, this really matters, right? Obviously, because you can b

At CES 2026, Nvidia unveiled Alpamayo, a 10-billion parameter open-source AI model for self-driving cars. The first production vehicle to use it is the Mercedes-Benz CLA, launching in Q1 2026 with point-to-point city navigation. Jensen Huang called it the ChatGPT moment for physical AI. Nvidia is offering 1,000 TOPS of compute power, five times more than competitors, and releasing the model weights on HuggingFace for anyone to use. Partners include Mercedes, JLR, Lucid, Uber, Bosch, and ZF. This is the first time a production-grade autonomous driving stack has been open-sourced.

This Week In Startups is made possible by:Vanta - https://www.vanta.com/twistNetSuite - https://www.netsuite.com/twistLemon.io - https://lemon.io/twist*Today's show: Domestic robots have never been hotter… so why is iRobot in so much trouble?!On TWiST, we're investigating the recent troubles facing the automated vacuum robot, Roomba. We've all seen these little discs sweeping up our friends' floors… but now the company behind them is running out of cash.What happened to iRobot, and would things have turned out differently if they had been sold to Amazon back in 2022?PLUS why Jason thinks you should spend your disposable income on investing… Is Alex still a journalist, and what does that actually mean for his portfolio?… Are founders and investors too cautious when it comes to signaling risk?… Why did all those internal Pipe documents leak online?… And MUCH MORE!Timestamps:(00:00) Why Jason loves Incentivizing New Ventures and Economic Strength Through Capital Formation!(09:48) Jason warns “no one is coming to help you with your job loss”; you're on your own!(10:21) Vanta - Get $1000 off your SOC 2 at https://www.vanta.com/twist(13:05) Why Jason thinks laid-off HR folks should be starting their own “micro-funds"(14:13) Does Alex consider himself a journalist? Is it unethical for him to invest in companies? (Let's get INTO IT.)(16:02) You can't become complacent when you're in the lead!(20:03) Why you should use disposable “entertainment” income on investing(22:32) Netsuite - Get the free business guide Demystifying AI at https://www.netsuite.com/twist(23:36) $9B in new funds?! What is Lightspeed up to?(26:10) Is everyone too cautious about “signaling risk”?(28:39) Lemon.io - Get 15% off your first 4 weeks of developer time at https://Lemon.io/twist(31:21) How does Jason think those Pipe documents leaked? And WHY?(38:15) The critical importance of diversification(39:35) iRobot went BANKRUPT! Who killed the Roomba?(44:47) Are there ANY potential deals that would bother Jason? It's all about market share.(48:18) HuggingFace is moving major Reachy Minis! How open source turns everything upside down.(53:22) How Polymarket will decide when/if the AI bubble “bursts.”*Subscribe to the TWiST500 newsletter: https://ticker.thisweekinstartups.com/Check out the TWIST500: https://twist500.comSubscribe to This Week in Startups on Apple: https://rb.gy/v19fcp*Follow Lon:X: https://x.com/lons*Follow Alex:X: https://x.com/alexLinkedIn: https://www.linkedin.com/in/alexwilhelm/*Follow Jason:X: https://twitter.com/JasonLinkedIn: https://www.linkedin.com/in/jasoncalacanis/*Thank you to our partners:(10:21) Vanta - Get $1000 off your SOC 2 at https://www.vanta.com/twist(22:32)  Netsuite - Get the free business guide Demystifying AI at https://www.netsuite.com/twist(28:39) Lemon.io - Get 15% off your first 4 weeks of developer time at https://Lemon.io/twistGreat TWIST interviews: Will Guidarahttps://youtu.be/pvJa2pzuXWQEoghan McCabehttps://youtu.be/9dHN4YFkgv4Steve Huffmanhttps://podcasts.apple.com/us/podcast/reddit-ceo-steve-huffman-on-mod-revolt-building-a/id315114957?i=1000617333424Brian Cheskyhttps://podcasts.apple.com/ca/podcast/airbnb-ceo-brian-chesky-on-early-rejection-customer/id315114957?i=1000611761112Bob Moestahttps://youtu.be/y2UMzSqX94QAaron Leviehttps://podcasts.apple.com/ca/podcast/box-ceo-aaron-levie-breaks-down-box-ai-and-generative/id315114957?i=1000612384545Sophia Amorusohttps://podcasts.apple.com/ca/podcast/sophia-amoruso-on-branding-raising-a-fund-portfolio/id315114957?i=1000601352978Reid Hoffmanhttps://podcasts.apple.com/ca/podcast/reid-hoffman-on-ais-crescendo-moment-regulation-and/id315114957?i=1000612548498Frank Slootmanhttps://podcasts.apple.com/ca/podcast/snowflake-ceo-frank-slootman-on-moving-the-needle-win/id315114957?i=1000602560622

Katia, Emmanuel et Guillaume discutent Java, Kotlin, Quarkus, Hibernate, Spring Boot 4, intelligence artificielle (modèles Nano Banana, VO3, frameworks agentiques, embedding). On discute les vulnerabilités OWASP pour les LLMs, les personalités de codage des différents modèles, Podman vs Docker, comment moderniser des projets legacy. Mais surtout on a passé du temps sur les présentations de Luc Julia et les différents contre points qui ont fait le buzz sur les réseaux. Enregistré le 12 septembre 2025 Téléchargement de l'épisode LesCastCodeurs-Episode-330.mp3 ou en vidéo sur YouTube. News Langages Dans cette vidéo, José détaille les nouveautés de Java entre Java 21 et 25 https://inside.java/2025/08/31/roadto25-java-language/ Aperçu des nouveautés du JDK 25 : Introduction des nouvelles fonctionnalités du langage Java et des changements à venir [00:02]. Programmation orientée données et Pattern Matching [00:43] : Évolution du “pattern matching” pour la déconstruction des “records” [01:22]. Utilisation des “sealed types” dans les expressions switch pour améliorer la lisibilité et la robustesse du code [01:47]. Introduction des “unnamed patterns” (_) pour indiquer qu'une variable n'est pas utilisée [04:47]. Support des types primitifs dans instanceof et switch (en preview) [14:02]. Conception d'applications Java [00:52] : Simplification de la méthode main [21:31]. Exécution directe des fichiers .java sans compilation explicite [22:46]. Amélioration des mécanismes d'importation [23:41]. Utilisation de la syntaxe Markdown dans la Javadoc [27:46]. Immuabilité et valeurs nulles [01:08] : Problème d'observation de champs final à null pendant la construction d'un objet [28:44]. JEP 513 pour contrôler l'appel à super() et restreindre l'usage de this dans les constructeurs [33:29]. JDK 25 sort le 16 septembre https://openjdk.org/projects/jdk/25/ Scoped Values (JEP 505) - alternative plus efficace aux ThreadLocal pour partager des données immutables entre threads Structured Concurrency (JEP 506) - traiter des groupes de tâches concurrentes comme une seule unité de travail, simplifiant la gestion des threads Compact Object Headers (JEP 519) - Fonctionnalité finale qui réduit de 50% la taille des en-têtes d'objets (de 128 à 64 bits), économisant jusqu'à 22% de mémoire heap Flexible Constructor Bodies (JEP 513) - Relaxation des restrictions sur les constructeurs, permettant du code avant l'appel super() ou this() Module Import Declarations (JEP 511) - Import simplifié permettant d'importer tous les éléments publics d'un module en une seule déclaration Compact Source Files (JEP 512) - Simplification des programmes Java basiques avec des méthodes main d'instance sans classe wrapper obligatoire Primitive Types in Patterns (JEP 455) - Troisième preview étendant le pattern matching et instanceof aux types primitifs dans switch et instanceof Generational Shenandoah (JEP 521) - Le garbage collector Shenandoah passe en mode générationnel pour de meilleures performances JFR Method Timing & Tracing (JEP 520) - Nouvel outillage de profilage pour mesurer le temps d'exécution et tracer les appels de méthodes Key Derivation API (JEP 510) - API finale pour les fonctions de dérivation de clés cryptographiques, remplaçant les implémentations tierces Améliorations du traitement des annotations dans Kotlin 2.2 https://blog.jetbrains.com/idea/2025/09/improved-annotation-handling-in-kotlin-2-2-less-boilerplate-fewer-surprises/ Avant Kotlin 2.2, les annotations sur les paramètres de constructeur n'étaient appliquées qu'au paramètre, pas à la propriété ou au champ Cela causait des bugs subtils avec Spring et JPA où la validation ne fonctionnait qu'à la création d'objet, pas lors des mises à jour La solution précédente nécessitait d'utiliser explicitement @field: pour chaque annotation, créant du code verbeux Kotlin 2.2 introduit un nouveau comportement par défaut qui applique les annotations aux paramètres ET aux propriétés/champs automatiquement Le code devient plus propre sans avoir besoin de syntaxe @field: répétitive Pour l'activer, ajouter -Xannotation-default-target=param-property dans les options du compilateur Gradle IntelliJ IDEA propose un quick-fix pour activer ce comportement à l'échelle du projet Cette amélioration rend l'intégration Kotlin plus fluide avec les frameworks majeurs comme Spring et JPA Le comportement peut être configuré pour garder l'ancien mode ou activer un mode transitoire avec avertissements Cette mise à jour fait partie d'une initiative plus large pour améliorer l'expérience Kotlin + Spring Librairies Sortie de Quarkus 3.26 avec mises à jour d'Hibernate et autres fonctionnalités - https://quarkus.io/blog/quarkus-3-26-released/ mettez à jour vers la 3.26.x car il y a eu une regression vert.x Jalon important vers la version LTS 3.27 prévue fin septembre, basée sur cette version Mise à jour vers Hibernate ORM 7.1, Hibernate Search 8.1 et Hibernate Reactive 3.1 Support des unités de persistance nommées et sources de données dans Hibernate Reactive Démarrage hors ligne et configuration de dialecte pour Hibernate ORM même si la base n'est pas accessible Refonte de la console HQL dans Dev UI avec fonctionnalité Hibernate Assistant intégrée Exposition des capacités Dev UI comme fonctions MCP pour pilotage via outils IA Rafraîchissement automatique des tokens OIDC en cas de réponse 401 des clients REST Extension JFR pour capturer les données runtime (nom app, version, extensions actives) Bump de Gradle vers la version 9.0 par défaut, suppression du support des classes config legacy Guide de démarrage avec Quarkus et A2A Java SDK 0.3.0 (pour faire discuter des agents IA avec la dernière version du protocole A2A) https://quarkus.io/blog/quarkus-a2a-java-0-3-0-alpha-release/ Sortie de l'A2A Java SDK 0.3.0.Alpha1, aligné avec la spécification A2A v0.3.0. Protocole A2A : standard ouvert (Linux Foundation), permet la communication inter-agents IA polyglottes. Version 0.3.0 plus stable, introduit le support gRPC. Mises à jour générales : changements significatifs, expérience utilisateur améliorée (côté client et serveur). Agents serveur A2A : Support gRPC ajouté (en plus de JSON-RPC). HTTP+JSON/REST à venir. Implémentations basées sur Quarkus (alternatives Jakarta existent). Dépendances spécifiques pour chaque transport (ex: a2a-java-sdk-reference-jsonrpc, a2a-java-sdk-reference-grpc). AgentCard : décrit les capacités de l'agent. Doit spécifier le point d'accès primaire et tous les transports supportés (additionalInterfaces). Clients A2A : Dépendance principale : a2a-java-sdk-client. Support gRPC ajouté (en plus de JSON-RPC). HTTP+JSON/REST à venir. Dépendance spécifique pour gRPC : a2a-java-sdk-client-transport-grpc. Création de client : via ClientBuilder. Sélectionne automatiquement le transport selon l'AgentCard et la configuration client. Permet de spécifier les transports supportés par le client (withTransport). Comment générer et éditer des images en Java avec Nano Banana, le “photoshop killer” de Google https://glaforge.dev/posts/2025/09/09/calling-nano-banana-from-java/ Objectif : Intégrer le modèle Nano Banana (Gemini 2.5 Flash Image preview) dans des applications Java. SDK utilisé : GenAI Java SDK de Google. Compatibilité : Supporté par ADK for Java ; pas encore par LangChain4j (limitation de multimodalité de sortie). Capacités de Nano Banana : Créer de nouvelles images. Modifier des images existantes. Assembler plusieurs images. Mise en œuvre Java : Quelle dépendance utiliser Comment s'authentifier Comment configurer le modèle Nature du modèle : Nano Banana est un modèle de chat qui peut retourner du texte et une image (pas simplement juste un modèle générateur d'image) Exemples d'utilisation : Création : Via un simple prompt textuel. Modification : En passant l'image existante (tableau de bytes) et les instructions de modification (prompt). Assemblage : En passant plusieurs images (en bytes) et les instructions d'intégration (prompt). Message clé : Toutes ces fonctionnalités sont accessibles en Java, sans nécessiter Python. Générer des vidéos IA avec le modèle Veo 3, mais en Java ! https://glaforge.dev/posts/2025/09/10/generating-videos-in-java-with-veo3/ Génération de vidéos en Java avec Veo 3 (via le GenAI Java SDK de Google). Veo 3: Annoncé comme GA, prix réduits, support du format 9:16, résolution jusqu'à 1080p. Création de vidéos : À partir d'une invite textuelle (prompt). À partir d'une image existante. Deux versions différentes du modèle : veo-3.0-generate-001 (qualité supérieure, plus coûteux, plus lent). veo-3.0-fast-generate-001 (qualité inférieure, moins coûteux, mais plus rapide). Rod Johnson sur ecrire des aplication agentic en Java plus facilement qu'en python avec Embabel https://medium.com/@springrod/you-can-build-better-ai-agents-in-java-than-python-868eaf008493 Rod the papa de Spring réécrit un exemple CrewAI (Python) qui génère un livre en utilisant Embabel (Java) pour démontrer la supériorité de Java L'application utilise plusieurs agents AI spécialisés : un chercheur, un planificateur de livre et des rédacteurs de chapitres Le processus suit trois étapes : recherche du sujet, création du plan, rédaction parallèle des chapitres puis assemblage CrewAI souffre de plusieurs problèmes : configuration lourde, manque de type safety, utilisation de clés magiques dans les prompts La version Embabel nécessite moins de code Java que l'original Python et moins de fichiers de configuration YAML Embabel apporte la type safety complète, éliminant les erreurs de frappe dans les prompts et améliorant l'outillage IDE La gestion de la concurrence est mieux contrôlée en Java pour éviter les limites de débit des APIs LLM L'intégration avec Spring permet une configuration externe simple des modèles LLM et hyperparamètres Le planificateur Embabel détermine automatiquement l'ordre d'exécution des actions basé sur leurs types requis L'argument principal : l'écosystème JVM offre un meilleur modèle de programmation et accès à la logique métier existante que Python Il y a pas mal de nouveaux framework agentic en Java, notamment le dernier LAngchain4j Agentic Spring lance un serie de blog posts sur les nouveautés de Spring Boot 4 https://spring.io/blog/2025/09/02/road_to_ga_introduction baseline JDK 17 mais rebase sur Jakarta 11 Kotlin 2, Jackson 3 et JUnit 6 Fonctionnalités de résilience principales de Spring : @ConcurrencyLimit, @Retryable, RetryTemplate Versioning d'API dans Spring Améliorations du client de service HTTP L'état des clients HTTP dans Spring Introduction du support Jackson 3 dans Spring Consommateur partagé - les queues Kafka dans Spring Kafka Modularisation de Spring Boot Autorisation progressive dans Spring Security Spring gRPC - un nouveau module Spring Boot Applications null-safe avec Spring Boot 4 OpenTelemetry avec Spring Boot Repos Ahead of Time (Partie 2) Web Faire de la recherche sémantique directement dans le navigateur en local, avec EmbeddingGemma et Transformers.js https://glaforge.dev/posts/2025/09/08/in-browser-semantic-search-with-embeddinggemma/ EmbeddingGemma: Nouveau modèle d'embedding (308M paramètres) de Google DeepMind. Objectif: Permettre la recherche sémantique directement dans le navigateur. Avantages clés de l'IA côté client: Confidentialité: Aucune donnée envoyée à un serveur. Coûts réduits: Pas besoin de serveurs coûteux (GPU), hébergement statique. Faible latence: Traitement instantané sans allers-retours réseau. Fonctionnement hors ligne: Possible après le chargement initial du modèle. Technologie principale: Modèle: EmbeddingGemma (petit, performant, multilingue, support MRL pour réduire la taille des vecteurs). Moteur d'inférence: Transformers.js de HuggingFace (exécute les modèles AI en JavaScript dans le navigateur). Déploiement: Site statique avec Vite/React/Tailwind CSS, déployé sur Firebase Hosting via GitHub Actions. Gestion du modèle: Fichiers du modèle trop lourds pour Git; téléchargés depuis HuggingFace Hub pendant le CI/CD. Fonctionnement de l'app: Charge le modèle, génère des embeddings pour requêtes/documents, calcule la similarité sémantique. Conclusion: Démonstration d'une recherche sémantique privée, économique et sans serveur, soulignant le potentiel de l'IA embarquée dans le navigateur. Data et Intelligence Artificielle Docker lance Cagent, une sorte de framework multi-agent IA utilisant des LLMs externes, des modèles de Docker Model Runner, avec le Docker MCP Tookit. Il propose un format YAML pour décrire les agents d'un système multi-agents. https://github.com/docker/cagent des agents “prompt driven” (pas de code) et une structure pour decrire comment ils sont deployés pas clair comment ils sont appelés a part dans la ligne de commande de cagent fait par david gageot L'owasp décrit l'independance excessive des LLM comme une vulnerabilité https://genai.owasp.org/llmrisk2023-24/llm08-excessive-agency/ L'agence excessive désigne la vulnérabilité qui permet aux systèmes LLM d'effectuer des actions dommageables via des sorties inattendues ou ambiguës. Elle résulte de trois causes principales : fonctionnalités excessives, permissions excessives ou autonomie excessive des agents LLM. Les fonctionnalités excessives incluent l'accès à des plugins qui offrent plus de capacités que nécessaire, comme un plugin de lecture qui peut aussi modifier ou supprimer. Les permissions excessives se manifestent quand un plugin accède aux systèmes avec des droits trop élevés, par exemple un accès en lecture qui inclut aussi l'écriture. L'autonomie excessive survient quand le système effectue des actions critiques sans validation humaine préalable. Un scénario d'attaque typique : un assistant personnel avec accès email peut être manipulé par injection de prompt pour envoyer du spam via la boîte de l'utilisateur. La prévention implique de limiter strictement les plugins aux fonctions minimales nécessaires pour l'opération prévue. Il faut éviter les fonctions ouvertes comme “exécuter une commande shell” au profit d'outils plus granulaires et spécifiques. L'application du principe de moindre privilège est cruciale : chaque plugin doit avoir uniquement les permissions minimales requises. Le contrôle humain dans la boucle reste essentiel pour valider les actions à fort impact avant leur exécution. Lancement du MCP registry, une sorte de méta-annuaire officiel pour référencer les serveurs MCP https://www.marktechpost.com/2025/09/09/mcp-team-launches-the-preview-version-of-the-mcp-registry-a-federated-discovery-layer-for-enterprise-ai/ MCP Registry : Couche de découverte fédérée pour l'IA d'entreprise. Fonctionne comme le DNS pour le contexte de l'IA, permettant la découverte de serveurs MCP publics ou privés. Modèle fédéré : Évite les risques de sécurité et de conformité d'un registre monolithique. Permet des sous-registres privés tout en conservant une source de vérité “upstream”. Avantages entreprises : Découverte interne sécurisée. Gouvernance centralisée des serveurs externes. Réduction de la prolifération des contextes. Support pour les agents IA hybrides (données privées/publiques). Projet open source, actuellement en version preview. Blog post officiel : https://blog.modelcontextprotocol.io/posts/2025-09-08-mcp-registry-preview/ Exploration des internals du transaction log SQL Server https://debezium.io/blog/2025/09/08/sqlserver-tx-log/ C'est un article pour les rugeux qui veulent savoir comment SQLServer marche à l'interieur Debezium utilise actuellement les change tables de SQL Server CDC en polling périodique L'article explore la possibilité de parser directement le transaction log pour améliorer les performances Le transaction log est divisé en Virtual Log Files (VLFs) utilisés de manière circulaire Chaque VLF contient des blocs (512B à 60KB) qui contiennent les records de transactions Chaque record a un Log Sequence Number (LSN) unique pour l'identifier précisément Les données sont stockées dans des pages de 8KB avec header de 96 bytes et offset array Les tables sont organisées en partitions et allocation units pour gérer l'espace disque L'utilitaire DBCC permet d'explorer la structure interne des pages et leur contenu Cette compréhension pose les bases pour parser programmatiquement le transaction log dans un prochain article Outillage Les personalités des codeurs des différents LLMs https://www.sonarsource.com/blog/the-coding-personalities-of-leading-llms-gpt-5-update/ GPT-5 minimal ne détrône pas Claude Sonnet 4 comme leader en performance fonctionnelle malgré ses 75% de réussite GPT-5 génère un code extrêmement verbeux avec 490 000 lignes contre 370 000 pour Claude Sonnet 4 sur les mêmes tâches La complexité cyclomatique et cognitive du code GPT-5 est dramatiquement plus élevée que tous les autres modèles GPT-5 introduit 3,90 problèmes par tâche réussie contre seulement 2,11 pour Claude Sonnet 4 Point fort de GPT-5 : sécurité exceptionnelle avec seulement 0,12 vulnérabilité par 1000 lignes de code Faiblesse majeure : densité très élevée de “code smells” (25,28 par 1000 lignes) nuisant à la maintenabilité GPT-5 produit 12% de problèmes liés à la complexité cognitive, le taux le plus élevé de tous les modèles Tendance aux erreurs logiques fondamentales avec 24% de bugs de type “Control-flow mistake” Réapparition de vulnérabilités classiques comme les failles d'injection et de traversée de chemin Nécessité d'une gouvernance renforcée avec analyse statique obligatoire pour gérer la complexité du code généré Pourquoi j'ai abandonné Docker pour Podman https://codesmash.dev/why-i-ditched-docker-for-podman-and-you-should-too Problème Docker : Le daemon dockerd persistant s'exécute avec des privilèges root, posant des risques de sécurité (nombreuses CVEs citées) et consommant des ressources inutilement. Solution Podman : Sans Daemon : Pas de processus d'arrière-plan persistant. Les conteneurs s'exécutent comme des processus enfants de la commande Podman, sous les privilèges de l'utilisateur. Sécurité Renforcée : Réduction de la surface d'attaque. Une évasion de conteneur compromet un utilisateur non privilégié sur l'hôte, pas le système entier. Mode rootless. Fiabilité Accrue : Pas de point de défaillance unique ; le crash d'un conteneur n'affecte pas les autres. Moins de Ressources : Pas de daemon constamment actif, donc moins de mémoire et de CPU. Fonctionnalités Clés de Podman : Intégration Systemd : Génération automatique de fichiers d'unité systemd pour gérer les conteneurs comme des services Linux standards. Alignement Kubernetes : Support natif des pods et capacité à générer des fichiers Kubernetes YAML directement (podman generate kube), facilitant le développement local pour K8s. Philosophie Unix : Se concentre sur l'exécution des conteneurs, délègue les tâches spécialisées à des outils dédiés (ex: Buildah pour la construction d'images, Skopeo pour leur gestion). Migration Facile : CLI compatible Docker : podman utilise les mêmes commandes que docker (alias docker=podman fonctionne). Les Dockerfiles existants sont directement utilisables. Améliorations incluses : Sécurité par défaut (ports privilégiés en mode rootless), meilleure gestion des permissions de volume, API Docker compatible optionnelle. Option de convertir Docker Compose en Kubernetes YAML. Bénéfices en Production : Sécurité améliorée, utilisation plus propre des ressources. Podman représente une évolution plus sécurisée et mieux alignée avec les pratiques modernes de gestion Linux et de déploiement de conteneurs. Guide Pratique (Exemple FastAPI) : Le Dockerfile ne change pas. podman build et podman run remplacent directement les commandes Docker. Déploiement en production via Systemd. Gestion d'applications multi-services avec les “pods” Podman. Compatibilité Docker Compose via podman-compose ou kompose. Détection améliorée des APIs vulnérables dans les IDEs JetBrains et Qodana - https://blog.jetbrains.com/idea/2025/09/enhanced-vulnerable-api-detection-in-jetbrains-ides-and-qodana/ JetBrains s'associe avec Mend.io pour renforcer la sécurité du code dans leurs outils Le plugin Package Checker bénéficie de nouvelles données enrichies sur les APIs vulnérables Analyse des graphes d'appels pour couvrir plus de méthodes publiques des bibliothèques open-source Support de Java, Kotlin, C#, JavaScript, TypeScript et Python pour la détection de vulnérabilités Activation des inspections via Paramètres > Editor > Inspections en recherchant “Vulnerable API” Surlignage automatique des méthodes vulnérables avec détails des failles au survol Action contextuelle pour naviguer directement vers la déclaration de dépendance problématique Mise à jour automatique vers une version non affectée via Alt+Enter sur la dépendance Fenêtre dédiée “Vulnerable Dependencies” pour voir l'état global des vulnérabilités du projet Méthodologies Le retour de du sondage de Stack Overflow sur l'usage de l'IA dans le code https://medium.com/@amareshadak/stack-overflow-just-exposed-the-ugly-truth-about-ai-coding-tools-b4f7b5992191 84% des développeurs utilisent l'IA quotidiennement, mais 46% ne font pas confiance aux résultats. Seulement 3,1% font “hautement confiance” au code généré. 66% sont frustrés par les solutions IA “presque correctes”. 45% disent que déboguer le code IA prend plus de temps que l'écrire soi-même. Les développeurs seniors (10+ ans) font moins confiance à l'IA (2,6%) que les débutants (6,1%), créant un écart de connaissances dangereux. Les pays occidentaux montrent moins de confiance - Allemagne (22%), UK (23%), USA (28%) - que l'Inde (56%). Les créateurs d'outils IA leur font moins confiance. 77% des développeurs professionnels rejettent la programmation en langage naturel, seuls 12% l'utilisent réellement. Quand l'IA échoue, 75% se tournent vers les humains. 35% des visites Stack Overflow concernent maintenant des problèmes liés à l'IA. 69% rapportent des gains de productivité personnels, mais seulement 17% voient une amélioration de la collaboration d'équipe. Coûts cachés : temps de vérification, explication du code IA aux équipes, refactorisation et charge cognitive constante. Les plateformes humaines dominent encore : Stack Overflow (84%), GitHub (67%), YouTube (61%) pour résoudre les problèmes IA. L'avenir suggère un “développement augmenté” où l'IA devient un outil parmi d'autres, nécessitant transparence et gestion de l'incertitude. Mentorat open source et défis communautaires par les gens de Microcks https://microcks.io/blog/beyond-code-open-source-mentorship/ Microcks souffre du syndrome des “utilisateurs silencieux” qui bénéficient du projet sans contribuer Malgré des milliers de téléchargements et une adoption croissante, l'engagement communautaire reste faible Ce manque d'interaction crée des défis de durabilité et limite l'innovation du projet Les mainteneurs développent dans le vide sans feedback des vrais utilisateurs Contribuer ne nécessite pas de coder : documentation, partage d'expérience, signalement de bugs suffisent Parler du project qu'on aime autour de soi est aussi super utile Microcks a aussi des questions specifiques qu'ils ont posé dans le blog, donc si vous l'utilisez, aller voir Le succès de l'open source dépend de la transformation des utilisateurs en véritables partenaires communautaires c'est un point assez commun je trouve, le ratio parlant / silencieux est tres petit et cela encourage les quelques grandes gueules La modernisation du systemes legacy, c'est pas que de la tech https://blog.scottlogic.com/2025/08/27/holistic-approach-successful-legacy-modernisation.html Un artcile qui prend du recul sur la modernisation de systemes legacy Les projets de modernisation legacy nécessitent une vision holistique au-delà du simple focus technologique Les drivers business diffèrent des projets greenfield : réduction des coûts et mitigation des risques plutôt que génération de revenus L'état actuel est plus complexe à cartographier avec de nombreuses dépendances et risques de rupture Collaboration essentielle entre Architectes, Analystes Business et Designers UX dès la phase de découverte Approche tridimensionnelle obligatoire : Personnes, Processus et Technologie (comme un jeu d'échecs 3D) Le leadership doit créer l'espace nécessaire pour la découverte et la planification plutôt que presser l'équipe Communication en termes business plutôt que techniques vers tous les niveaux de l'organisation Planification préalable essentielle contrairement aux idées reçues sur l'agilité Séquencement optimal souvent non-évident et nécessitant une analyse approfondie des interdépendances Phases projet alignées sur les résultats business permettent l'agilité au sein de chaque phase Sécurité Cyber Attaque su Musée Histoire Naturelle https://www.franceinfo.fr/internet/securite-sur-internet/cyberattaques/le-museum-nati[…]e-d-une-cyberattaque-severe-une-plainte-deposee_7430356.html Compromission massive de packages npm populaires par un malware crypto https://www.aikido.dev/blog/npm-debug-and-chalk-packages-compromised 18 packages npm très populaires compromis le 8 septembre 2025, incluant chalk, debug, ansi-styles avec plus de 2 milliards de téléchargements hebdomadaires combinés duckdb s'est rajouté à la liste Code malveillant injecté qui intercepte silencieusement l'activité crypto et web3 dans les navigateurs des utilisateurs Le malware manipule les interactions de wallet et redirige les paiements vers des comptes contrôlés par l'attaquant sans signes évidents Injection dans les fonctions critiques comme fetch, XMLHttpRequest et APIs de wallets (window.ethereum, Solana) pour intercepter le trafic Détection et remplacement automatique des adresses crypto sur multiple blockchains (Ethereum, Bitcoin, Solana, Tron, Litecoin, Bitcoin Cash) Les transactions sont modifiées en arrière-plan même si l'interface utilisateur semble correcte et légitime Utilise des adresses “sosies” via correspondance de chaînes pour rendre les échanges moins évidents à détecter Le mainteneur compromis par email de phishing provenant du faux domaine “mailto:support@npmjs.help|support@npmjs.help” enregistré 3 jours avant l'attaque sur une demande de mise a jour de son autheotnfication a deux facteurs après un an Aikido a alerté le mainteneur via Bluesky qui a confirmé la compromission et commencé le nettoyage des packages Attaque sophistiquée opérant à plusieurs niveaux: contenu web, appels API et manipulation des signatures de transactions Les anti-cheats de jeux vidéo : une faille de sécurité majeure ? - https://tferdinand.net/jeux-video-et-si-votre-anti-cheat-etait-la-plus-grosse-faille/ Les anti-cheats modernes s'installent au Ring 0 (noyau système) avec privilèges maximaux Ils obtiennent le même niveau d'accès que les antivirus professionnels mais sans audit ni certification Certains exploitent Secure Boot pour se charger avant le système d'exploitation Risque de supply chain : le groupe APT41 a déjà compromis des jeux comme League of Legends Un attaquant infiltré pourrait désactiver les solutions de sécurité et rester invisible Menace de stabilité : une erreur peut empêcher le démarrage du système (référence CrowdStrike) Conflits possibles entre différents anti-cheats qui se bloquent mutuellement Surveillance en temps réel des données d'utilisation sous prétexte anti-triche Dérive dangereuse selon l'auteur : des entreprises de jeux accèdent au niveau EDR Alternatives limitées : cloud gaming ou sandboxing avec impact sur performances donc faites gaffe aux jeux que vos gamins installent ! Loi, société et organisation Luc Julia au Sénat - Monsieur Phi réagi et publie la vidéo Luc Julia au Sénat : autopsie d'un grand N'IMPORTE QUOI https://www.youtube.com/watch?v=e5kDHL-nnh4 En format podcast de 20 minutes, sorti au même moment et à propos de sa conf à Devoxx https://www.youtube.com/watch?v=Q0gvaIZz1dM Le lab IA - Jérôme Fortias - Et si Luc Julia avait raison https://www.youtube.com/watch?v=KScI5PkCIaE Luc Julia au Senat https://www.youtube.com/watch?v=UjBZaKcTeIY Luc Julia se défend https://www.youtube.com/watch?v=DZmxa7jJ8sI Intelligence artificielle : catastrophe imminente ? - Luc Julia vs Maxime Fournes https://www.youtube.com/watch?v=sCNqGt7yIjo Tech and Co Monsieur Phi vs Luc Julia (put a click) https://www.youtube.com/watch?v=xKeFsOceT44 La tronche en biais https://www.youtube.com/live/zFwLAOgY0Wc Conférences La liste des conférences provenant de Developers Conferences Agenda/List par Aurélie Vache et contributeurs : 12 septembre 2025 : Agile Pays Basque 2025 - Bidart (France) 15 septembre 2025 : Agile Tour Montpellier - Montpellier (France) 18-19 septembre 2025 : API Platform Conference - Lille (France) & Online 22-24 septembre 2025 : Kernel Recipes - Paris (France) 22-27 septembre 2025 : La Mélée Numérique - Toulouse (France) 23 septembre 2025 : OWASP AppSec France 2025 - Paris (France) 23-24 septembre 2025 : AI Engineer Paris - Paris (France) 25 septembre 2025 : Agile Game Toulouse - Toulouse (France) 25-26 septembre 2025 : Paris Web 2025 - Paris (France) 30 septembre 2025-1 octobre 2025 : PyData Paris 2025 - Paris (France) 2 octobre 2025 : Nantes Craft - Nantes (France) 2-3 octobre 2025 : Volcamp - Clermont-Ferrand (France) 3 octobre 2025 : DevFest Perros-Guirec 2025 - Perros-Guirec (France) 6-7 octobre 2025 : Swift Connection 2025 - Paris (France) 6-10 octobre 2025 : Devoxx Belgium - Antwerp (Belgium) 7 octobre 2025 : BSides Mulhouse - Mulhouse (France) 7-8 octobre 2025 : Agile en Seine - Issy-les-Moulineaux (France) 8-10 octobre 2025 : SIG 2025 - Paris (France) & Online 9 octobre 2025 : DevCon #25 : informatique quantique - Paris (France) 9-10 octobre 2025 : Forum PHP 2025 - Marne-la-Vallée (France) 9-10 octobre 2025 : EuroRust 2025 - Paris (France) 16 octobre 2025 : PlatformCon25 Live Day Paris - Paris (France) 16 octobre 2025 : Power 365 - 2025 - Lille (France) 16-17 octobre 2025 : DevFest Nantes - Nantes (France) 17 octobre 2025 : Sylius Con 2025 - Lyon (France) 17 octobre 2025 : ScalaIO 2025 - Paris (France) 17-19 octobre 2025 : OpenInfra Summit Europe - Paris (France) 20 octobre 2025 : Codeurs en Seine - Rouen (France) 23 octobre 2025 : Cloud Nord - Lille (France) 30-31 octobre 2025 : Agile Tour Bordeaux 2025 - Bordeaux (France) 30-31 octobre 2025 : Agile Tour Nantais 2025 - Nantes (France) 30 octobre 2025-2 novembre 2025 : PyConFR 2025 - Lyon (France) 4-7 novembre 2025 : NewCrafts 2025 - Paris (France) 5-6 novembre 2025 : Tech Show Paris - Paris (France) 5-6 novembre 2025 : Red Hat Summit: Connect Paris 2025 - Paris (France) 6 novembre 2025 : dotAI 2025 - Paris (France) 6 novembre 2025 : Agile Tour Aix-Marseille 2025 - Gardanne (France) 7 novembre 2025 : BDX I/O - Bordeaux (France) 12-14 novembre 2025 : Devoxx Morocco - Marrakech (Morocco) 13 novembre 2025 : DevFest Toulouse - Toulouse (France) 15-16 novembre 2025 : Capitole du Libre - Toulouse (France) 19 novembre 2025 : SREday Paris 2025 Q4 - Paris (France) 19-21 novembre 2025 : Agile Grenoble - Grenoble (France) 20 novembre 2025 : OVHcloud Summit - Paris (France) 21 novembre 2025 : DevFest Paris 2025 - Paris (France) 27 novembre 2025 : DevFest Strasbourg 2025 - Strasbourg (France) 28 novembre 2025 : DevFest Lyon - Lyon (France) 1-2 décembre 2025 : Tech Rocks Summit 2025 - Paris (France) 4-5 décembre 2025 : Agile Tour Rennes - Rennes (France) 5 décembre 2025 : DevFest Dijon 2025 - Dijon (France) 9-11 décembre 2025 : APIdays Paris - Paris (France) 9-11 décembre 2025 : Green IO Paris - Paris (France) 10-11 décembre 2025 : Devops REX - Paris (France) 10-11 décembre 2025 : Open Source Experience - Paris (France) 11 décembre 2025 : Normandie.ai 2025 - Rouen (France) 14-17 janvier 2026 : SnowCamp 2026 - Grenoble (France) 2-6 février 2026 : Web Days Convention - Aix-en-Provence (France) 3 février 2026 : Cloud Native Days France 2026 - Paris (France) 12-13 février 2026 : Touraine Tech #26 - Tours (France) 22-24 avril 2026 : Devoxx France 2026 - Paris (France) 23-25 avril 2026 : Devoxx Greece - Athens (Greece) 17 juin 2026 : Devoxx Poland - Krakow (Poland) 4 septembre 2026 : JUG SUmmer Camp 2026 - La Rochelle (France) Nous contacter Pour réagir à cet épisode, venez discuter sur le groupe Google https://groups.google.com/group/lescastcodeurs Contactez-nous via X/twitter https://twitter.com/lescastcodeurs ou Bluesky https://bsky.app/profile/lescastcodeurs.com Faire un crowdcast ou une crowdquestion Soutenez Les Cast Codeurs sur Patreon https://www.patreon.com/LesCastCodeurs Tous les épisodes et toutes les infos sur https://lescastcodeurs.com/

Trust at Scale: Security and Governance for Open Source Models // MLOps Podcast #338 with Hudson Buzby, Solutions Architect at JFrog.Appreciate JFrog for their support in bringing this blog to life.Join the Community: https://go.mlops.community/YTJoinInGet the newsletter: https://go.mlops.community/YTNewsletter// AbstractFor better or for worse, machine learning has traditionally escaped the gaze of security and infrastructure teams, operating outside traditional DevOps practices and not always adhering to organizations development or security standards. With the introduction of open source catalogs like HuggingFace and Ollama, a new standard has been established for locating, identifying, and deploying machine learning and AI models. But with this new standard comes a plethora of security, governance, and legal challenges that organizations need to address before they can comfortably allow developers to freely build and deploy ML/AI applications. In this conversation will discuss ways that enterprise scale organizations are addressing these challenges to safely and securely build these development environments. // BioHudson Buzby is a solution engineer with an emphasis on MLOps, LLMOps, Big Data, and Distributed Systems, leveraging his expertise to help organizations optimize their machine learning operations and large language model deployments. His role involves providing technical solutions and guidance to enhance the efficiency and effectiveness of AI-driven projects.// Related Linkshttps://www.youtube.com/channel/UCh2hNg76zo3d1qQqTWIQxDg~~~~~~~~ ✌️Connect With Us ✌️ ~~~~~~~Catch all episodes, blogs, newsletters, and more: https://go.mlops.community/TYExploreJoin our Slack community [https://go.mlops.community/slack]Follow us on X/Twitter [@mlopscommunity](https://x.com/mlopscommunity) or [LinkedIn](https://go.mlops.community/linkedin)] Sign up for the next meetup: [https://go.mlops.community/register]MLOps Swag/Merch: [https://shop.mlops.community/]Connect with Demetrios on LinkedIn: /dpbrinkmConnect with Hudson on LinkedIn: /hudson-buzby/

Unauthorized Issuance of Certificate for 1.1.1.1 Cloudflare published a blog post with more details regarding the bad 1.1.1.1 certificate that was issued by Fina. https://blog.cloudflare.com/unauthorized-issuance-of-certificates-for-1-1-1-1/ AI Model Namespace Reuse Deleted accounts on Huggingface can be taken over by other entities unrelated to the original owner. https://unit42.paloaltonetworks.com/model-namespace-reuse/ macOS vulnerability allowed Keychain and iOS app decryption without a password Excessive entitlements for the gcore binary facilitated access to key material that was sufficient to access secrets stored in Apple s keychain. https://www.helpnetsecurity.com/2025/09/04/macos-gcore-vulnerability-cve-2025-24204/

While much of the AI world chases ever-larger models, Ravin Kumar (Google DeepMind) and his team build across the size spectrum, from billions of parameters down to this week's release: Gemma 270M, the smallest member yet of the Gemma 3 open-weight family. At just 270 million parameters, a quarter the size of Gemma 1B, it's designed for speed, efficiency, and fine-tuning. We explore what makes 270M special, where it fits alongside its billion-parameter siblings, and why you might reach for it in production even if you think “small” means “just for experiments.” We talk through: - Where 270M fits into the Gemma 3 lineup — and why it exists - On-device use cases where latency, privacy, and efficiency matter - How smaller models open up rapid, targeted fine-tuning - Running multiple models in parallel without heavyweight hardware - Why “small” models might drive the next big wave of AI adoption If you've ever wondered what you'd do with a model this size (or how to squeeze the most out of it) this episode will show you how small can punch far above its weight. LINKS Introducing Gemma 3 270M: The compact model for hyper-efficient AI (Google Developer Blog) (https://developers.googleblog.com/en/introducing-gemma-3-270m/) Full Model Fine-Tune Guide using Hugging Face Transformers (https://ai.google.dev/gemma/docs/core/huggingface_text_full_finetune) The Gemma 270M model on HuggingFace (https://huggingface.co/google/gemma-3-270m) The Gemma 270M model on Ollama (https://ollama.com/library/gemma3:270m) Building AI Agents with Gemma 3, a workshop with Ravin and Hugo (https://www.youtube.com/live/-IWstEStqok) (Code here (https://github.com/canyon289/ai_agent_basics)) From Images to Agents: Building and Evaluating Multimodal AI Workflows, a workshop with Ravin and Hugo (https://www.youtube.com/live/FNlM7lSt8Uk)(Code here (https://github.com/canyon289/ai_image_agent)) Evaluating AI Agents: From Demos to Dependability, an upcoming workshop with Ravin and Hugo (https://lu.ma/ezgny3dl) Upcoming Events on Luma (https://lu.ma/calendar/cal-8ImWFDQ3IEIxNWk) Watch the podcast video on YouTube (https://youtu.be/VZDw6C2A_8E)

Scott and Wes break down how to code with and for AI; perfect for skeptics, beginners, and curious devs. They cover everything from Ghost Text and CLI agents to building your own AI-powered apps with embeddings, function calling, and multi-model workflows. Show Notes 00:00 Welcome to Syntax! 03:56 How to interface with AI. 04:07 IDE Ghost Text. 05:45 IDE Chat, Agents. 08:00 CLI Agents. Claude Code. Open Code. Gemini. 11:13 MCP Servers. Context7 14:47 GUI apps. v0. Bolt.new. Lovable. Windsurf. 19:07 Existing Chat app like ChatGPT. 22:37 Building things WITH AI. 23:32 Prompting. 26:53 Streaming VS not streaming. 28:14 Embeddings and Rag. 31:09 MCP Server. CJ's MCP Deep Dive. 32:36 Brought to you by Sentry.io. 33:25 Multi-model, multi-provider. 36:27 npm libs to use to code with AI. OpenAI SDK. AI SDK. Cloudflare Agents. Langchain. Local AI Tensorflow. Transformers.js. Huggingface. 44:12 Processes and exploring. Hit us up on Socials! Syntax: X Instagram Tiktok LinkedIn Threads Wes: X Instagram Tiktok LinkedIn Threads Scott: X Instagram Tiktok LinkedIn Threads Randy: X Instagram YouTube Threads

 This week on the PHP Podcast, Eric and John discuss Spec-driven Development with Kiro, JetBrains on Huggingface, Event Sourcing with Laravel Verbs, Automating your life with n8n, PHP Tek 2026 Website development using vibe coding, and more. Links from the show: Introducing Kiro – Kiro JetBrains (JetBrains) Verbs About Grokability – Snipe-IT Free open […] The post The PHP Podcast: 2025.07.17 appeared first on PHP Architect.

Hosts Kate and Jeff dive into everything from the U.S. economy to big changes in media and entertainment. They chat about the future of search, how AI is shaking up copyright and content creation, and even affordable robots for learning. It's all about how tech is reshaping industries and what it means for the future.00:00 - Introduction01:34 - US Economic Data - Unemployment trends, AI's role in jobs, and economic data insights.05:38 - Consumer Spending - Credit card debt, tariffs, and shifting consumer habits.07:50 - New Business Trends - Growth in business applications and manufacturing orders.10:14 - A&E Sale - What A&E's potential sale means for the cable industry.12:23 - Disney-ITV Partnership - A unique content-sharing deal between Disney and ITV.16:16 - F1 Media Rights - Apple's bid for F1 rights and changes in sports media.18:28 - AI Browsers and Cloudflare - AI-powered browsers and Cloudflare's move to block crawlers.21:35 - AI Copyright Cases - Court rulings on AI copyright and their impact on creators.24:47 - Robots in Education - Hugging Face's Ricci Mini robot and its potential in coding education.27:58 - What's Next - Upcoming earnings reports, CPI data, and next week's highlights.LinksCloudflare blog: https://www.cloudflare.com/press-releases/2025/cloudflare-just-changed-how-ai-crawlers-scrape-the-internet-at-large/ Huggingface robot: https://huggingface.co/blog/reachy-mini 

Deux chercheurs présentent GAIA, un benchmark qui évalue la capacité des IA à mener des recherches complexes et à raisonner étape par étape. On explore ce que les “thinking models” et l'usage d'outils (web, PDF, images) changent vraiment, et pourquoi cela débloque des résultats concrets. Ils détaillent aussi la “sauce secrète” derrière Deep Research d'OpenAI et comparent ces approches aux autres méthodes du marché.Sources Article central Autre ressource intéressanteEn plateau Michaël de Marliave — animateur Grégoire Mialon — invité Clémentine Fourrier — invité Hébergé par Acast. Visitez acast.com/privacy pour plus d'informations.

In this high-impact episode of Seller Sessions, Danny McMillan is joined by Dorian Gorski for a no-fluff exploration of how AI is shifting the Amazon ecosystem. The conversation orbits around a powerful new tool called "Manus" — an AI-driven platform built to go beyond surface-level product research and tap into rich demographic insights, customer motivations, and actionable listing data.

Today's show: In this episode, Jason, Alex, and Lon dive into Blue Origin's all-female celeb spaceflight (yes, Katy Perry sang on reentry), Hugging Face's unexpected move into robotics, and Jack Dorsey's wild take that we should “delete all IP law.” Plus, they break down Figure AI's eye-popping $39B valuation, the risks of SPVs, and what founders and investors can learn from the SPAC boom. As Jason puts it: “You just have to assume an 80% failure rate.”*Timestamps:(0:00) Jason kicks off the show!(1:34) Blue Origin all-female crew launch and space tourism(7:17) Emerging technologies and tech adoption trends(10:07) Northwest Registered Agent. Form your entire business identity in just 10 clicks and 10 minutes. Get more privacy, more options, and more done—visit https://www.northwestregisteredagent.com/twist today!(12:38) Hugging Face acquires Pollen Robotics; Open AI and robotics debate(19:42) Squarespace - Use offer code TWIST to save 10% off your first purchase of a website or domain at https://www.Squarespace.com/TWIST(20:42) Significance of Hugging Face in generative AI; Jack Dorsey's IP law stance(25:01) U.S. high-tech job market; revisiting IP law discussions(30:03) Lemon.io - Get 15% off your first 4 weeks of developer time at https://Lemon.io/twist(31:03) IP law and American innovation(32:22) Challenges in startup exits and secondary trading platforms(37:09) Figure AI's valuation controversy(46:37) Startup insights and investing perspectives(50:39) Jeff Bezos on risk assessment(57:03) Jason's personal journey and reflections(1:02:06) Developing a samurai mindset; societal systems abstraction*Subscribe to the TWiST500 newsletter: https://ticker.thisweekinstartups.comCheck out the TWIST500: https://www.twist500.comSubscribe to This Week in Startups on Apple: https://rb.gy/v19fcp*Follow Lon:X: https://x.com/lons*Follow Alex:X: https://x.com/alexLinkedIn: ⁠https://www.linkedin.com/in/alexwilhelmFollow Jason:X: https://twitter.com/JasonLinkedIn: https://www.linkedin.com/in/jasoncalacanisThank you to our partners:(10:07) Northwest Registered Agent. Form your entire business identity in just 10 clicks and 10 minutes. Get more privacy, more options, and more done—visit https://www.northwestregisteredagent.com/twist today!(19:42) Squarespace - Use offer code TWIST to save 10% off your first purchase of a website or domain at https://www.Squarespace.com/TWIST(30:03) Lemon.io - Get 15% off your first 4 weeks of developer time at https://Lemon.io/twistGreat TWIST interviews: Will Guidara, Eoghan McCabe, Steve Huffman, Brian Chesky, Bob Moesta, Aaron Levie, Sophia Amoruso, Reid Hoffman, Frank Slootman, Billy McFarlandCheck out Jason's suite of newsletters: https://substack.com/@calacanisFollow TWiST:Twitter: https://twitter.com/TWiStartupsYouTube: https://www.youtube.com/thisweekinInstagram: https://www.instagram.com/thisweekinstartupsTikTok: https://www.tiktok.com/@thisweekinstartupsSubstack: https://twistartups.substack.com*Subscribe to the Founder University Podcast: https://www.youtube.com/@founderuniversity1916

Mardi 15 avril, François Sorel a reçu Frédéric Simottel, journaliste BFM Business, Thomas Serval, PDG de Baracoda, ainsi que Christophe Aulnette, senior advisor chez Seven2 et ancien président de Microsoft France et Asie du Sud. Ils ont parlé de Nvidia qui va construire pour 500 milliards de dollars de serveurs aux États-Unis, de l'atmosphère et l'ambiance dans la Silicon Valley actuellement, du projet d'OpenAI de développer un réseau social, et du rachat par Hugging face de Pollen Robotics, dans l'émission Tech & Co, la quotidienne, sur BFM Business. Retrouvez l'émission du lundi au jeudi et réécoutez-la en podcast.

Britney Muller is an AI consultant and keynote speaker advising tech companies on AI strategies, machine learning, and workflow automation. With over 10 years of experience in generative AI, she has developed over a dozen in-house AI applications. Britney was the former Marketing Manager at Hugging Face, where she launched the largest open-source, multilingual model.  In this episode…  AI is changing the game for marketers, but many don't leverage it to its fullest potential for their businesses. Rather than producing AI-driven content at a faster rate, companies should focus on building a strong brand presence and leveraging AI for intentional automation. How can marketers cut through the noise, avoid common pitfalls, and harness AI to drive measurable results? While AI can be used for pattern recognition, automation, and audience research, AI optimizer Britney Muller warns against relying on it for fact-based decision-making. You can leverage AI without losing the critical human element by transforming website content into vector embeddings. Analyzing these embeddings allows marketers to identify content clusters, uncover gaps in their website's information architecture, and optimize internal linking structures to improve search engine rankings. Britney also recommends utilizing Reddit APIs to extract real-time customer sentiments, uncover trending pain points, and analyze top-performing content in specific communities. In this week's episode of the Up Arrow Podcast, William Harris chats with AI consultant Britney Muller about practical AI strategies for marketers. Britney explains why brand mentions are the new backlinks, how to build AI-powered internal tools, and the ethical concerns marketers should consider when adopting AI.

【本期嘉宾】王铁震、石洪竺、郭炜王铁震（Huggingface高级工程师）石洪竺（魔搭社区运营负责人）郭炜（白鲸开源 CEO Apache基金会成员）主播：潘乱（「乱翻书」主理人）【时间线】开源与闭源02:12 闭源和开源的概念科普04:55 开源模型和开源代码的区别07:40 开源模型，开源了哪些东西，是“真”开源么？08:14 权重开源10:51 开源最后的核心竞争力是什么？12:40 阿里为什么会选择开源路径，阿里的开源之路是如何走来的？14:51 从闭源的OpenAI，到开源的Llama、Qwen崛起，这期间业界经历了什么？19:18 开发者角度看到的开源力量23:01 开源模型与闭源模型在技术迭代速度上有何差异？26:47 最牛的工程师们，更在意技术品牌的什么？“DeepSeek时刻”之后31:40 面对DeepSeek的开源，OpenAI是否会调整自己的开源策略？36:23 如果OpenAI被迫开源O3，DeepSeek的先发优势是否就没有了？39:39 DeepSeek开源之后，AI小公司闭源是否还有生存空间？46:07 「大模型自己还没出现幻觉，大模型使用者先自己出现幻觉了」47:07 开源时代是否也是渠道为王？49:09 DeepSeek出现后，Huggingface社区里有哪些讨论？51:53 魔搭社区关于DeepSeek的讨论55:10 如何界定“模型影响力”这一指标？开源社区和商业化58:36 开源大模型如何平衡"技术普惠"与"商业化盈利"？59:08 DeepSeek可能有哪些比较可行的商业模式？63:57 Huggingface作为开源模型社区，目前的商业化模式是怎样的？64:49 魔搭社区作为阿里系的平台，是否会平等的对待内外部模型？开源生态69:23 DeepSeek开源之后，套壳好像不再是问题了？73:41 当年QQ也是QICQ的套壳75:04 Agent开发门槛已大幅降低，套壳产品护城河又在哪里？78:17 大模型如何重构开源生态？81:36 DeepSeek会构建端侧轻量化的生态吗？85:25 怎么看待开源和闭源的关系，开源和闭源各自的上限会在哪里？哪个更有未来？【开场&结尾音乐】开场音乐：Richard Stallman - 《自由软件之歌》（The Free Software Song）结尾音乐：虞霞/李小龙 - 侠客行（电视剧《武林外传》片尾曲TV Verison）《自由软件之歌》（英语：The Free Software Song）是由自由软件基金会主席理查德·斯托曼（Richard Stallman）作词的自由软件宣传曲，它采用保加利亚民歌《Sadi Moma》的旋律。目的是要鼓励程序员与大众分享其源代码及推动软件自由化。歌词片段：Join us now and share the software;You'll be free, hackers, you'll be free.……【关于「乱翻书」】「乱翻书」是一档关注商业、科技和互联网的圆桌对话节目。关心How和Why，以及少有人注意到的What。内容主要方向是科技考古、行业观察和前沿思考，研究公司的兴衰循环，希望能够为你带来信息增量。「乱翻书」主理人是潘乱，代表作品有《腾讯没有梦想》、字节跳动/快手早期关键节点的系列特写。【关于主播】视频号/即刻/小红书：潘乱公众号/播客：乱翻书【图】直播截图微信公众号：乱翻书视频号：潘乱商业合作：联系微信 tongxing717本期编辑：怀杭

What do you get when Bittensor's most talented subnet builder and a prolific miner combine forces? Subnet 64 Chutes. Built for dtao with optimized efficiency on every level, monetization included through micropayments in TAO, a host of groundbreaking security features for miners, a rewrite of validation methodology to lower expenses and a front end so smooth you don't need to know squat to kick the tires on all of Huggingface...this was such a treat. I love these dudes, what assets to have involved in Bittensor. Head to chutes.ai to use the future now.  PS. More Chutes apps coming soon. Try out text-to-voice now and EASY Agents deployed with no coding shortly. While you wait you can deploy ANY custom code now, for free.  https://chutes.ai/ https://x.com/namoray_dev https://x.com/jon_durbin https://x.com/KeithSingery https://bittensor.guru https://taostats.io/validators/bittensor-guru-podcast/ https://bittensor.com    

Happy holidays! We'll be sharing snippets from Latent Space LIVE! through the break bringing you the best of 2024! We want to express our deepest appreciation to event sponsors AWS, Daylight Computer, Thoth.ai, StrongCompute, Notable Capital, and most of all all our LS supporters who helped fund the gorgeous venue and A/V production!For NeurIPS last year we did our standard conference podcast coverage interviewing selected papers (that we have now also done for ICLR and ICML), however we felt that we could be doing more to help AI Engineers 1) get more industry-relevant content, and 2) recap 2024 year in review from experts. As a result, we organized the first Latent Space LIVE!, our first in person miniconference, at NeurIPS 2024 in Vancouver. Today, we're proud to share Loubna's highly anticipated talk (slides here)!Synthetic DataWe called out the Synthetic Data debate at last year's NeurIPS, and no surprise that 2024 was dominated by the rise of synthetic data everywhere:* Apple's Rephrasing the Web, Microsoft's Phi 2-4 and Orca/AgentInstruct, Tencent's Billion Persona dataset, DCLM, and HuggingFace's FineWeb-Edu, and Loubna's own Cosmopedia extended the ideas of synthetic textbook and agent generation to improve raw web scrape dataset quality* This year we also talked to the IDEFICS/OBELICS team at HuggingFace who released WebSight this year, the first work on code-vs-images synthetic data.* We called Llama 3.1 the Synthetic Data Model for its extensive use (and documentation!) of synthetic data in its pipeline, as well as its permissive license. * Nemotron CC and Nemotron-4-340B also made a big splash this year for how they used 20k items of human data to synthesize over 98% of the data used for SFT/PFT.* Cohere introduced Multilingual Arbitrage: Optimizing Data Pools to Accelerate Multilingual Progress observing gains of up to 56.5% improvement in win rates comparing multiple teachers vs the single best teacher model* In post training, AI2's Tülu3 (discussed by Luca in our Open Models talk) and Loubna's Smol Talk were also notable open releases this year.This comes in the face of a lot of scrutiny and criticism, with Scale AI as one of the leading voices publishing AI models collapse when trained on recursively generated data in Nature magazine bringing mainstream concerns to the potential downsides of poor quality syndata:Part of the concerns we highlighted last year on low-background tokens are coming to bear: ChatGPT contaminated data is spiking in every possible metric:But perhaps, if Sakana's AI Scientist pans out this year, we will have mostly-AI AI researchers publishing AI research anyway so do we really care as long as the ideas can be verified to be correct?Smol ModelsMeta surprised many folks this year by not just aggressively updating Llama 3 and adding multimodality, but also adding a new series of “small” 1B and 3B “on device” models this year, even working on quantized numerics collaborations with Qualcomm, Mediatek, and Arm. It is near unbelievable that a 1B model today can qualitatively match a 13B model of last year:and the minimum size to hit a given MMLU bar has come down roughly 10x in the last year. We have been tracking this proxied by Lmsys Elo and inference price:The key reads this year are:* MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases* Apple Intelligence Foundation Language Models* Hymba: A Hybrid-head Architecture for Small Language Models* Loubna's SmolLM and SmolLM2: a family of state-of-the-art small models with 135M, 360M, and 1.7B parameters on the pareto efficiency frontier.* and Moondream, which we already covered in the 2024 in Vision talkFull Talk on YouTubeplease like and subscribe!Timestamps* [00:00:05] Loubna Intro* [00:00:33] The Rise of Synthetic Data Everywhere* [00:02:57] Model Collapse* [00:05:14] Phi, FineWeb, Cosmopedia - Synthetic Textbooks* [00:12:36] DCLM, Nemotron-CC* [00:13:28] Post Training - AI2 Tulu, Smol Talk, Cohere Multilingual Arbitrage* [00:16:17] Smol Models* [00:18:24] On Device Models* [00:22:45] Smol Vision Models* [00:25:14] What's NextTranscript2024 in Synthetic Data and Smol Models[00:00:00] ​[00:00:05] Loubna Intro[00:00:05] Speaker: ​I'm very happy to be here. Thank you for the invitation. So I'm going to be talking about synthetic data in 2024. And then I'm going to be talking about small on device models. So I think the most interesting thing about synthetic data this year is that like now we have it everywhere in the large language models pipeline.[00:00:33] The Rise of Synthetic Data Everywhere[00:00:33] Speaker: I think initially, synthetic data was mainly used just for post training, because naturally that's the part where we needed human annotators. And then after that, we realized that we don't really have good benchmarks to [00:01:00] measure if models follow instructions well, if they are creative enough, or if they are chatty enough, so we also started using LLMs as judges.[00:01:08] Speaker: Thank you. And I think this year and towards the end of last year, we also went to the pre training parts and we started generating synthetic data for pre training to kind of replace some parts of the web. And the motivation behind that is that you have a lot of control over synthetic data. You can control your prompt and basically also the kind of data that you generate.[00:01:28] Speaker: So instead of just trying to filter the web, you could try to get the LLM to generate what you think the best web pages could look like and then train your models on that. So this is how we went from not having synthetic data at all in the LLM pipeline to having it everywhere. And so the cool thing is like today you can train an LLM with like an entirely synthetic pipeline.[00:01:49] Speaker: For example, you can use our Cosmopedia datasets and you can train a 1B model on like 150 billion tokens that are 100 percent synthetic. And those are also of good quality. And then you can [00:02:00] instruction tune the model on a synthetic SFT dataset. You can also do DPO on a synthetic dataset. And then to evaluate if the model is good, you can use.[00:02:07] Speaker: A benchmark that uses LLMs as a judge, for example, MTBench or AlpacaEvil. So I think this is like a really mind blowing because like just a few years ago, we wouldn't think this is possible. And I think there's a lot of concerns about model collapse, and I'm going to talk about that later. But we'll see that like, if we use synthetic data properly and we curate it carefully, that shouldn't happen.[00:02:29] Speaker: And the reason synthetic data is very popular right now is that we have really strong models, both open and closed. It is really cheap and fast to use compared to human annotations, which cost a lot and take a lot of time. And also for open models right now, we have some really good inference frameworks.[00:02:47] Speaker: So if you have enough GPUs, it's really easy to spawn these GPUs and generate like a lot of synthetic data. Some examples are VLM, TGI, and TensorRT.[00:02:57] Model Collapse[00:02:57] Speaker: Now let's talk about the elephant in the room, model [00:03:00] collapse. Is this the end? If you look at the media and all of like, for example, some papers in nature, it's really scary because there's a lot of synthetic data out there in the web.[00:03:09] Speaker: And naturally we train on the web. So we're going to be training a lot of synthetic data. And if model collapse is going to happen, we should really try to take that seriously. And the other issue is that, as I said, we think, a lot of people think the web is polluted because there's a lot of synthetic data.[00:03:24] Speaker: And for example, when we're building fine web datasets here at Guillerm and Hinek, we're interested in like, how much synthetic data is there in the web? So there isn't really a method to properly measure the amount of synthetic data or to save a webpage synthetic or not. But one thing we can do is to try to look for like proxy words, for example, expressions like as a large language model or words like delve that we know are actually generated by chat GPT.[00:03:49] Speaker: We could try to measure the amount of these words in our data system and compare them to the previous years. For example, here, we measured like a, these words ratio in different dumps of common crawl. [00:04:00] And we can see that like the ratio really increased after chat GPT's release. So if we were to say that synthetic data amount didn't change, you would expect this ratio to stay constant, which is not the case.[00:04:11] Speaker: So there's a lot of synthetic data probably on the web, but does this really make models worse? So what we did is we trained different models on these different dumps. And we then computed their performance on popular, like, NLP benchmarks, and then we computed the aggregated score. And surprisingly, you can see that the latest DOMs are actually even better than the DOMs that are before.[00:04:31] Speaker: So if there's some synthetic data there, at least it did not make the model's worse. Yeah, which is really encouraging. So personally, I wouldn't say the web is positive with Synthetic Data. Maybe it's even making it more rich. And the issue with like model collapse is that, for example, those studies, they were done at like a small scale, and you would ask the model to complete, for example, a Wikipedia paragraph, and then you would train it on these new generations, and you would do that every day.[00:04:56] Speaker: iteratively. I think if you do that approach, it's normal to [00:05:00] observe this kind of behavior because the quality is going to be worse because the model is already small. And then if you train it just on its generations, you shouldn't expect it to become better. But what we're really doing here is that we take a model that is very large and we try to distill its knowledge into a model that is smaller.[00:05:14] Phi, FineWeb, Cosmopedia - Synthetic Textbooks[00:05:14] Speaker: And in this way, you can expect to get like a better performance for your small model. And using synthetic data for pre-training has become really popular. After the textbooks are all you need papers where Microsoft basically trained a series of small models on textbooks that were using a large LLM.[00:05:32] Speaker: And then they found that these models were actually better than models that are much larger. So this was really interesting. It was like first of its time, but it was also met with a lot of skepticism, which is a good thing in research. It pushes you to question things because the dataset that they trained on was not public, so people were not really sure if these models are really good or maybe there's just some data contamination.[00:05:55] Speaker: So it was really hard to check if you just have the weights of the models. [00:06:00] And as Hugging Face, because we like open source, we tried to reproduce what they did. So this is our Cosmopedia dataset. We basically tried to follow a similar approach to what they documented in the paper. And we created a synthetic dataset of textbooks and blog posts and stories that had almost 30 billion tokens.[00:06:16] Speaker: And we tried to train some models on that. And we found that like the key ingredient to getting a good data set that is synthetic is trying as much as possible to keep it diverse. Because if you just throw the same prompts as your model, like generate like a textbook about linear algebra, and even if you change the temperature, the textbooks are going to look alike.[00:06:35] Speaker: So there's no way you could scale to like millions of samples. And the way you do that is by creating prompts that have some seeds that make them diverse. In our case, the prompt, we would ask the model to generate a textbook, but make it related to an extract from a webpage. And also we try to frame it within, to stay within topic.[00:06:55] Speaker: For example, here, we put like an extract about cardiovascular bioimaging, [00:07:00] and then we ask the model to generate a textbook related to medicine that is also related to this webpage. And this is a really nice approach because there's so many webpages out there. So you can. Be sure that your generation is not going to be diverse when you change the seed example.[00:07:16] Speaker: One thing that's challenging with this is that you want the seed samples to be related to your topics. So we use like a search tool to try to go all of fine web datasets. And then we also do a lot of experiments with the type of generations we want the model to generate. For example, we ask it for textbooks for middle school students or textbook for college.[00:07:40] Speaker: And we found that like some generation styles help on some specific benchmarks, while others help on other benchmarks. For example, college textbooks are really good for MMLU, while middle school textbooks are good for benchmarks like OpenBookQA and Pico. This is like a sample from like our search tool.[00:07:56] Speaker: For example, you have a top category, which is a topic, and then you have some [00:08:00] subtopics, and then you have the topic hits, which are basically the web pages in fine web does belong to these topics. And here you can see the comparison between Cosmopedia. We had two versions V1 and V2 in blue and red, and you can see the comparison to fine web, and as you can see throughout the training training on Cosmopedia was consistently better.[00:08:20] Speaker: So we managed to get a data set that was actually good to train these models on. It's of course so much smaller than FineWeb, it's only 30 billion tokens, but that's the scale that Microsoft data sets was, so we kind of managed to reproduce a bit what they did. And the data set is public, so everyone can go there, check if everything is all right.[00:08:38] Speaker: And now this is a recent paper from NVIDIA, Neumatron CC. They took things a bit further, and they generated not a few billion tokens, but 1. 9 trillion tokens, which is huge. And we can see later how they did that. It's more of, like, rephrasing the web. So we can see today that there's, like, some really huge synthetic datasets out there, and they're public, so, [00:09:00] like, you can try to filter them even further if you want to get, like, more high quality corpses.[00:09:04] Speaker: So for this, rephrasing the web this approach was suggested in this paper by Pratyush, where basically in this paper, they take some samples from C4 datasets, and then they use an LLM to rewrite these samples into a better format. For example, they ask an LLM to rewrite the sample into a Wikipedia passage or into a Q& A page.[00:09:25] Speaker: And the interesting thing in this approach is that you can use a model that is Small because it doesn't, rewriting doesn't require knowledge. It's just rewriting a page into a different style. So the model doesn't need to have like knowledge that is like extensive of what is rewriting compared to just asking a model to generate a new textbook and not giving it like ground truth.[00:09:45] Speaker: So here they rewrite some samples from C4 into Q& A, into Wikipedia, and they find that doing this works better than training just on C4. And so what they did in Nemo Trans CC is a similar approach. [00:10:00] They rewrite some pages from Common Crawl for two reasons. One is to, like improve Pages that are low quality, so they rewrite them into, for example, Wikipedia page, so they look better.[00:10:11] Speaker: And another reason is to create more diverse datasets. So they have a dataset that they already heavily filtered, and then they take these pages that are already high quality, and they ask the model to rewrite them in Question and Answer format. into like open ended questions or like multi choice questions.[00:10:27] Speaker: So this way they can reuse the same page multiple times without fearing like having multiple duplicates, because it's the same information, but it's going to be written differently. So I think that's also a really interesting approach for like generating synthetic data just by rephrasing the pages that you already have.[00:10:44] Speaker: There's also this approach called Prox where they try to start from a web page and then they generate a program which finds how to write that page to make it better and less noisy. For example, here you can see that there's some leftover metadata in the web page and you don't necessarily want to keep that for training [00:11:00] your model.[00:11:00] Speaker: So So they train a model that can generate programs that can like normalize and remove lines that are extra. So I think this approach is also interesting, but it's maybe less scalable than the approaches that I presented before. So that was it for like rephrasing and generating new textbooks.[00:11:17] Speaker: Another approach that I think is really good and becoming really popular for using synthetic data for pre training is basically building a better classifiers. For filtering the web for example, here we release the data sets called fine web edu. And the way we built it is by taking Llama3 and asking it to rate the educational content of web pages from zero to five.[00:11:39] Speaker: So for example, if a page is like a really good textbook that could be useful in a school setting, it would get a really high score. And if a page is just like an advertisement or promotional material, it would get a lower score. And then after that, we take these synthetic annotations and we train a classifier on them.[00:11:57] Speaker: It's a classifier like a BERT model. [00:12:00] And then we run this classifier on all of FineWeb, which is a 15 trillion tokens dataset. And then we only keep the pages that have like a score that's higher than 3. So for example, in our case, we went from 15 trillion tokens to 3. to just 1. 5 trillion tokens. Those are really highly educational.[00:12:16] Speaker: And as you can see here, a fine web EDU outperforms all the other public web datasets by a larger margin on a couple of benchmarks here, I show the aggregated score and you can see that this approach is really effective for filtering web datasets to get like better corpuses for training your LLMs.[00:12:36] DCLM, Nemotron-CC[00:12:36] Speaker: Others also try to do this approach. There's, for example, the DCLM datasets where they also train the classifier, but not to detect educational content. Instead, they trained it on OpenHermes dataset, which is a dataset for instruction tuning. And also they explain like IAM5 subreddits, and then they also get really high quality dataset which is like very information dense and can help [00:13:00] you train some really good LLMs.[00:13:01] Speaker: And then Nemotron Common Crawl, they also did this approach, but instead of using one classifier, they used an ensemble of classifiers. So they used, for example, the DCLM classifier, and also classifiers like the ones we used in FineWebEducational, and then they combined these two. Scores into a, with an ensemble method to only retain the best high quality pages, and they get a data set that works even better than the ones we develop.[00:13:25] Speaker: So that was it for like synthetic data for pre-training.[00:13:28] Post Training - AI2 Tulu, Smol Talk, Cohere Multilingual Arbitrage[00:13:28] Speaker: Now we can go back to post training. I think there's a lot of interesting post training data sets out there. One that was released recently, the agent instructs by Microsoft where they basically try to target some specific skills. And improve the performance of models on them.[00:13:43] Speaker: For example, here, you can see code, brain teasers, open domain QA, and they managed to get a dataset that outperforms that's when fine tuning Mistral 7b on it, it outperforms the original instruct model that was released by Mistral. And as I said, to get good synthetic data, you really [00:14:00] have to have a framework to make sure that your data is diverse.[00:14:03] Speaker: So for example, for them, they always. And then they see the generations on either source code or raw text documents, and then they rewrite them to make sure they're easier to generate instructions from, and then they use that for their like instruction data generation. There's also the Tool3SFT mixture, which was released recently by Allen AI.[00:14:23] Speaker: It's also really good quality and it covers a wide range of tasks. And the way they make sure that this dataset is diverse is by using personas from the persona hub datasets. Which is basically a data set of like I think over a million personas. And for example, in the tool mixture to generate like a new code snippet, they would give like the model persona, for example, a machine learning researcher interested in neural networks, and then ask it to generate like a coding problem.[00:14:49] Speaker: This way you make sure that your data set is really diverse, and then you can further filter the data sets, for example, using the reward models. We also released a dataset called Smalltalk, [00:15:00] and we also tried to cover the wide range of tasks, and as you can see here, for example, when fine tuning Mistral 7b on the dataset, we also outperformed the original Mistral instructs on a number of benchmarks, notably on mathematics and instruction following with ifevil.[00:15:18] Speaker: Another paper that's really interesting I wanted to mention is this one called Multilingual Data Arbitrage by Cohere. And basically they want to generate a data set for post training that is multilingual. And they have a really interesting problem. It's the fact that there isn't like one model that's really good at all the languages they wanted.[00:15:36] Speaker: So what they do is that like they use not just one teacher model, but multiple teachers. And then they have a router which basically sends the prompts they have to all these models. And then they get the completions and they have a reward model that traces all these generations and only keeps the best one.[00:15:52] Speaker: And this is like arbitrage and finance. So well, I think what's interesting in this, it shows that like synthetic data, it doesn't have to come from a single model. [00:16:00] And because we have so many good models now, you could like pull these models together and get like a dataset that's really high quality and that's diverse and that's covers all your needs.[00:16:12] Speaker: I was supposed to put a meme there, but. Yeah, so that was it for like a synthetic data.[00:16:17] Smol Models[00:16:17] Speaker: Now we can go to see what's happening in the small models field in 2024. I don't know if you know, but like now we have some really good small models. For example, Lama 3. 2 1B is. It matches Lama 2. 13b from, that was released last year on the LMSYS arena, which is basically the default go to leaderboard for evaluating models using human evaluation.[00:16:39] Speaker: And as you can see here, the scores of the models are really close. So I think we've made like hugely forward in terms of small models. Of course, that's one, just one data point, but there's more. For example, if you look at this chart from the Quint 2. 5 blog post, it shows that today we have some really good models that are only like 3 billion parameters [00:17:00] and 4 billion that score really high on MMLU.[00:17:03] Speaker: Which is a really popular benchmark for evaluating models. And you can see here that the red, the blue dots have more than 65 on MMLU. And the grey ones have less. And for example, Llama33b had less. So now we have a 3b model that outperforms a 33b model that was released earlier. So I think now people are starting to realize that like, we shouldn't just scale and scale models, but we should try to make them more efficient.[00:17:33] Speaker: I don't know if you knew, but you can also chat with a 3B plus model on your iPhone. For example, here, this is an app called PocketPal, where you can go and select a model from Hugging Face. It has a large choice. For example, here we loaded the 5. 3. 5, which is 3. 8 billion parameters on this iPhone. And we can chat with this and you can see that even the latency is also acceptable.[00:17:57] Speaker: For example, here, I asked it to give me a joke about [00:18:00] NeurIPS. So let's see what it has to say.[00:18:06] Speaker: Okay, why did the neural network attend NeurIPS? Because it heard there would be a lot of layers and fun and it wanted to train its sense of humor. So not very funny, but at least it can run on device. Yeah, so I think now we have good small models, but we also have like good frameworks and tools to use these small models.[00:18:24] On Device Models[00:18:24] Speaker: So I think we're really close to having like really on edge and on device models that are really good. And I think for a while we've had this narrative. But just training larger models is better. Of course, this is supported by science scaling laws. As you can see here, for example, when we scale the model size, the loss is lower and obviously you get a better model.[00:18:46] Speaker: But and we can see this, for example, in the GPT family of models, how we went from just a hundred million parameters to more than a trillion. parameters. And of course, we all observed the performance improvement when using the latest model. But [00:19:00] one thing that we shouldn't forget is that when we scale the model, we also scale the inference costs and time.[00:19:05] Speaker: And so the largest models were are going to cost so much more. So I think now instead of just building larger models, we should be focusing on building more efficient models. It's no longer a race for the largest models since these models are really expensive to run and they require like a really good infrastructure to do that and they cannot run on, for example, consumer hardware.[00:19:27] Speaker: And when you try to build more efficient models that match larger models, that's when you can really unlock some really interesting on device use cases. And I think a trend that we're noticing now is the trend of training smaller models longer. For example, if you compare how much, how long LLAMA was trained compared to LLAMA3, there is a huge increase in the pre training length.[00:19:50] Speaker: LLAMA was trained on 1 trillion tokens, but LLAMA3 8b was trained on 15 trillion tokens. So Meta managed to get a model that's the same size, but But it performs so much [00:20:00] better by choosing to like spend the sacrifice during training, because as we know, training is a one time cost, but inference is something that's ongoing.[00:20:08] Speaker: If we want to see what are like the small models reads in 2024, I think this mobile LLM paper by Meta is interesting. They try to study different models that are like have the less than 1 billion parameters and find which architecture makes most sense for these models. For example, they find that depth is more important than width.[00:20:29] Speaker: So it's more important to have models that have like more layers than just one. making them more wide. They also find that GQA helps, that tying the embedding helps. So I think it's a nice study overall for models that are just a few hundred million parameters. There's also the Apple intelligence tech report, which is interesting.[00:20:48] Speaker: So for Apple intelligence, they had two models, one that was like on server and another model that was on device. It had 3 billion parameters. And I think the interesting part is that they trained this model using [00:21:00] pruning. And then distillation. And for example, they have this table where they show that, like, using pruning and distillation works much better than training from scratch.[00:21:08] Speaker: And they also have some interesting insights about, like, how they specialize their models on specific tasks, like, for example, summarization and rewriting. There's also this paper by NVIDIA that was released recently. I think you've already had a talk about, like, hybrid models that was all interesting.[00:21:23] Speaker: And this model, they used, like, a hybrid architecture between state space models and transformers. And they managed to train a 1B model that's really performant without needing to train it on a lot of tokens. And regarding our work, we just recently released SmallM2, so it's a series of three models, which are the best in class in each model size.[00:21:46] Speaker: For example, our 1. 7b model outperforms Lama 1b and also Qt 2. 5. And how we managed to train this model is the following. That's where you spent a lot of time trying to curate the pre training datasets. We did a lot of [00:22:00] ablations, trying to find which datasets are good and also how to mix them. We also created some new math and code datasets that we're releasing soon.[00:22:08] Speaker: But you basically really spent a lot of time trying to find what's the best mixture that you can train these models on. And then we spent some time trying to like we also trained these models for very long. For example, small M1 was trained only on 1 trillion tokens, but this model is trained on 11 trillion tokens.[00:22:24] Speaker: And we saw that the performance kept improving. The models didn't really plateau mid training, which I think is really interesting. It shows that you can train such small models for very long and keep getting performance gains. What's interesting about SmallLM2 is that it's fully open. We also released, like the pre training code base, the fine tuning code, the datasets, and also evaluation in this repository.[00:22:45] Smol Vision Models[00:22:45] Speaker: Also there's, like, really interesting small models for text, but also for vision. For example, here you can see SmallVLM, which is a 2B model that's really efficient. It doesn't consume a lot of RAM, and it also has a good performance. There's also Moondream 0. [00:23:00] 5b, which was released recently. It's like the smallest visual language model.[00:23:04] Speaker: And as you can see, there isn't like a big trade off compared to Moondream 2b. So now I showed you that we have some really good small models. We also have the tools to use them, but why should you consider using small models and when? I think, like, small models are really interesting because of the on device feature.[00:23:23] Speaker: Because these models are small and they can run fast, you can basically run them on your laptop, but also on your mobile phone. And this means that your dataset stays locally. You don't have to send your queries to third parties. And this really enhances privacy. That was, for example, one of the big selling points for Apple Intelligence.[00:23:42] Speaker: Also, right now, we really have a lot of work to do. So many frameworks to do on device inference. For example, there's MLX, MLC, Llama, CPP, Transformers, JS. So we have a lot of options and each of them have like great features. So you have so many options for doing that. Small models are also really powerful if you choose to specialize them.[00:24:00][00:24:00] Speaker: For example, here there's a startup called Numind, which took small LM and then they fine tuned it on text extraction datasets. And they managed to get a model that's not very far from models that are much larger. So I think text extraction is like one use case where small models can be really performant and it makes sense to use them instead of just using larger models.[00:24:19] Speaker: You can also chat with these models in browser. For example, here, you can go there, you can load the model, you can even turn off your internet and just start chatting with the model locally. Speaking of text extraction, if you don't want to fine tune the models, there's a really good method of structure generation.[00:24:36] Speaker: We can basically force the models to follow a JSON schema that you defined. For example, here, we try to force the model to follow a schema for extracting key information from GitHub issues. So you can input free text, which is a complaint about a GitHub repository, something not working. And then you can run it there and the model can extract anything that is relevant for your GitHub issue creation.[00:24:58] Speaker: For example, the [00:25:00] priority, for example, here, priority is high, the type of the issue bug, and then a title and the estimation of how long this will take to fix. And you can just like do this in the browser, you can transform your text into a GitHub issue that's properly formatted.[00:25:14] What's Next[00:25:14] Speaker: So what's next for synthetic data and small models?[00:25:18] Speaker: I think that domain specific synthetic data is going to be, it's already important, it's going to be even more important. For example, generating synthetic data for math. I think this really would help improve the reasoning of a lot of models. And a lot of people are doing it, for example, Quint 2. 12 math, everyone's trying to reproduce a one.[00:25:37] Speaker: And so I think for synthetic data, trying to specialize it on some domains is going to be really important. And then for small models, I think specializing them through fine tuning, it's also going to be really important because I think a lot of companies are just trying to use these large models because they are better.[00:25:53] Speaker: But on some tasks, I think you can already get decent performance with small models. So you don't need to Pay like a [00:26:00] cost that's much larger just to make your model better at your task by a few percent. And this is not just for text. And I think it also applies for other modalities like vision and audio.[00:26:11] Speaker: And I think you should also watch out for on device frameworks and applications. For example, like the app I showed, or lama, all these frameworks are becoming really popular and I'm pretty sure that we're gonna get like more of them in 2025. And users really like that. Maybe for other, I should also say hot take.[00:26:28] Speaker: I think that like in AI, we just started like with fine tuning, for example, trying to make BERT work on some specific use cases, and really struggling to do that. And then we had some models that are much larger. So we just switched to like prompt engineering to get the models And I think we're going back to fine tuning where we realize these models are really costly.[00:26:47] Speaker: It's better to use just a small model or try to specialize it. So I think it's a little bit of a cycle and we're going to start to see like more fine tuning and less of just like a prompt engineering the models. So that was my talk. Thank you for following. And if you have [00:27:00] any questions, we can take them now. Get full access to Latent Space at www.latent.space/subscribe

The full schedule for Latent Space LIVE! at NeurIPS has been announced, featuring Best of 2024 overview talks for the AI Startup Landscape, Computer Vision, Open Models, Transformers Killers, Synthetic Data, Agents, and Scaling, and speakers from Sarah Guo of Conviction, Roboflow, AI2/Meta, Recursal/Together, HuggingFace, OpenHands and SemiAnalysis. Join us for the IRL event/Livestream! Alessio will also be holding a meetup at AWS Re:Invent in Las Vegas this Wednesday. See our new Events page for dates of AI Engineer Summit, Singapore, and World's Fair in 2025. LAST CALL for questions for our big 2024 recap episode! Submit questions and messages on Speakpipe here for a chance to appear on the show!When we first observed that GPT Wrappers are Good, Actually, we did not even have Bolt on our radar. Since we recorded our Anthropic episode discussing building Agents with the new Claude 3.5 Sonnet, Bolt.new (by Stackblitz) has easily cleared the $8m ARR bar, repeating and accelerating its initial $4m feat.There are very many AI code generators and VS Code forks out there, but Bolt probably broke through initially because of its incredible zero shot low effort app generation:But as we explain in the pod, Bolt also emphasized deploy (Netlify)/ backend (Supabase)/ fullstack capabilities on top of Stackblitz's existing WebContainer full-WASM-powered-developer-environment-in-the-browser tech. Since then, the team has been shipping like mad (with weekly office hours), with bugfixing, full screen, multi-device, long context, diff based edits (using speculative decoding like we covered in Inference, Fast and Slow).All of this has captured the imagination of low/no code builders like Greg Isenberg and many others on YouTube/TikTok/Reddit/X/Linkedin etc:Just as with Fireworks, our relationship with Bolt/Stackblitz goes a bit deeper than normal - swyx advised the launch and got a front row seat to this epic journey, as well as demoed it with Realtime Voice at the recent OpenAI Dev Day. So we are very proud to be the first/closest to tell the full open story of Bolt/Stackblitz!Flow Engineering + Qodo/AlphaCodium UpdateIn year 2 of the pod we have been on a roll getting former guests to return as guest cohosts (Harrison Chase, Aman Sanger, Jon Frankle), and it was a pleasure to catch Itamar Friedman back on the pod, giving us an update on all things Qodo and Testing Agents from our last catchup a year and a half ago:Qodo (they renamed in September) went viral in early January this year with AlphaCodium (paper here, code here) beating DeepMind's AlphaCode with high efficiency:With a simple problem solving code agent:* The first step is to have the model reason about the problem. They describe it using bullet points and focus on the goal, inputs, outputs, rules, constraints, and any other relevant details.* Then, they make the model reason about the public tests and come up with an explanation of why the input leads to that particular output. * The model generates two to three potential solutions in text and ranks them in terms of correctness, simplicity, and robustness. * Then, it generates more diverse tests for the problem, covering cases not part of the original public tests. * Iteratively, pick a solution, generate the code, and run it on a few test cases. * If the tests fail, improve the code and repeat the process until the code passes every test.swyx has previously written similar thoughts on types vs tests for putting bounds on program behavior, but AlphaCodium extends this to AI generated tests and code.More recently, Itamar has also shown that AlphaCodium's techniques also extend well to the o1 models:Making Flow Engineering a useful technique to improve code model performance on every model. This is something we see AI Engineers uniquely well positioned to do compared to ML Engineers/Researchers.Full Video PodcastLike and subscribe!Show Notes* Itamar* Qodo* First episode* Eric* Bolt* StackBlitz* Thinkster* AlphaCodium* WebContainersChapters* 00:00:00 Introductions & Updates* 00:06:01 Generic vs. Specific AI Agents* 00:07:40 Maintaining vs Creating with AI* 00:17:46 Human vs Agent Computer Interfaces* 00:20:15 Why Docker doesn't work for Bolt* 00:24:23 Creating Testing and Code Review Loops* 00:28:07 Bolt's Task Breakdown Flow* 00:31:04 AI in Complex Enterprise Environments* 00:41:43 AlphaCodium* 00:44:39 Strategies for Breaking Down Complex Tasks* 00:45:22 Building in Open Source* 00:50:35 Choosing a product as a founder* 00:59:03 Reflections on Bolt Success* 01:06:07 Building a B2C GTM* 01:18:11 AI Capabilities and Pricing Tiers* 01:20:28 What makes Bolt unique* 01:23:07 Future Growth and Product Development* 01:29:06 Competitive Landscape in AI Engineering* 01:30:01 Advice to Founders and Embracing AI* 01:32:20 Having a baby and completing an Iron ManTranscriptAlessio [00:00:00]: Hey everyone, welcome to the Latent Space Podcast. This is Alessio, partner and CTO at Decibel Partners, and I'm joined by my co-host Swyx, founder of Smol.ai.Swyx [00:00:12]: Hey, and today we're still in our sort of makeshift in-between studio, but we're very delighted to have a former returning guest host, Itamar. Welcome back.Itamar [00:00:21]: Great to be here after a year or more. Yeah, a year and a half.Swyx [00:00:24]: You're one of our earliest guests on Agents. Now you're CEO co-founder of Kodo. Right. Which has just been renamed. You also raised a $40 million Series A, and we can get caught up on everything, but we're also delighted to have our new guest, Eric. Welcome.Eric [00:00:42]: Thank you. Excited to be here. Should I say Bolt or StackBlitz?Swyx [00:00:45]: Like, is it like its own company now or?Eric [00:00:47]: Yeah. Bolt's definitely bolt.new. That's the thing that we're probably the most known for, I imagine, at this point.Swyx [00:00:54]: Which is ridiculous to say because you were working at StackBlitz for so long.Eric [00:00:57]: Yeah. I mean, within a week, we were doing like double the amount of traffic. And StackBlitz had been online for seven years, and we were like, what? But anyways, yeah. So we're StackBlitz, the company behind bolt.new. If you've heard of bolt.new, that's our stuff. Yeah.Swyx [00:01:12]: Yeah.Itamar [00:01:13]: Excellent. I see, by the way, that the founder mode, you need to know to capture opportunities. So kudos on doing that, right? You're working on some technology, and then suddenly you can exploit that to a new world. Yeah.Eric [00:01:24]: Totally. And I think, well, not to jump, but 100%, I mean, a couple of months ago, we had the idea for Bolt earlier this year, but we haven't really shared this too much publicly. But we actually had tried to build it with some of those state-of-the-art models back in January, February, you can kind of imagine which, and they just weren't good enough to actually do the code generation where the code was accurate and it was fast and whatever have you without a ton of like rag, but then there was like issues with that. So we put it on the shelf and then we got kind of a sneak peek of some of the new models that have come out in the past couple of months now. And so once we saw that, once we actually saw the code gen from it, we were like, oh my God, like, okay, we can build a product around this. And so that was really the impetus of us building the thing. But with that, it was StackBlitz, the core StackBlitz product the past seven years has been an IDE for developers. So the entire user experience flow we've built up just didn't make sense. And so when we kind of went out to build Bolt, we just thought, you know, if we were inventing our product today, what would the interface look like given what is now possible with the AI code gen? And so there's definitely a lot of conversations we had internally, but you know, just kind of when we logically laid it out, we were like, yeah, I think it makes sense to just greenfield a new thing and let's see what happens. If it works great, then we'll figure it out. If it doesn't work great, then it'll get deleted at some point. So that's kind of how it actually came to be.Swyx [00:02:49]: I'll mention your background a little bit. You were also founder of Thinkster before you started StackBlitz. So both of you are second time founders. Both of you have sort of re-founded your company recently. Yours was more of a rename. I think a slightly different direction as well. And then we can talk about both. Maybe just chronologically, should we get caught up on where Kodo is first and then you know, just like what people should know since the last pod? Sure.Itamar [00:03:12]: The last pod was two months after we launched and we basically had the vision that we talked about. The idea that software development is about specification, test and code, etc. We are more on the testing part as in essence, we think that if you solve testing, you solve software development. The beautiful chart that we'll put up on screen. And testing is a really big field, like there are many dimensions, unit testing, the level of the component, how big it is, how large it is. And then there is like different type of testing, is it regression or smoke or whatever. So back then we only had like one ID extension with unit tests as in focus. One and a half year later, first ID extension supports more type of testing as context aware. We index local, local repos, but also 10,000s of repos for Fortune 500 companies. We have another agent, another tool that is called, the pure agent is the open source and the commercial one is CodoMerge. And then we have another open source called CoverAgent, which is not yet a commercial product coming very soon. It's very impressive. It could be that already people are approving automated pull requests that they don't even aware in really big open sources. So once we have enough of these, we will also launch another agent. So for the first one and a half year, what we did is grew in our offering and mostly on the side of, does this code actually works, testing, code review, et cetera. And we believe that's the critical milestone that needs to be achieved to actually have the AI engineer for enterprise software. And then like for the first year was everything bottom up, getting to 1 million installation. 2024, that was 2023, 2024 was starting to monetize, to feel like how it is to make the first buck. So we did the teams offering, it went well with a thousand of teams, et cetera. And then we started like just a few months ago to do enterprise with everything you need, which is a lot of things that discussed in the last post that was just released by Codelm. So that's how we call it at Codelm. Just opening the brackets, our company name was Codelm AI, and we renamed to Codo and we call our models Codelm. So back to my point, so we started Enterprise Motion and already have multiple Fortune 100 companies. And then with that, we raised a series of $40 million. And what's exciting about it is that enables us to develop more agents. That's our focus. I think it's very different. We're not coming very soon with an ID or something like that.Swyx [00:06:01]: You don't want to fork this code?Itamar [00:06:03]: Maybe we'll fork JetBrains or something just to be different.Swyx [00:06:08]: I noticed that, you know, I think the promise of general purpose agents has kind of died. Like everyone is doing kind of what you're doing. There's Codogen, Codomerge, and then there's a third one. What's the name of it?Itamar [00:06:17]: Yeah. Codocover. Cover. Which is like a commercial version of a cover agent. It's coming soon.Swyx [00:06:23]: Yeah. It's very similar with factory AI, also doing like droids. They all have special purpose doing things, but people don't really want general purpose agents. Right. The last time you were here, we talked about AutoGBT, the biggest thing of 2023. This year, not really relevant anymore. And I think it's mostly just because when you give me a general purpose agent, I don't know what to do with it.Eric [00:06:42]: Yeah.Itamar [00:06:43]: I totally agree with that. We're seeing it for a while and I think it will stay like that despite the computer use, et cetera, that supposedly can just replace us. You can just like prompt it to be, hey, now be a QA or be a QA person or a developer. I still think that there's a few reasons why you see like a dedicated agent. Again, I'm a bit more focused, like my head is more on complex software for big teams and enterprise, et cetera. And even think about permissions and what are the data sources and just the same way you manage permissions for users. Developers, you probably want to have dedicated guardrails and dedicated approvals for agents. I intentionally like touched a point on not many people think about. And of course, then what you can think of, like maybe there's different tools, tool use, et cetera. But just the first point by itself is a good reason why you want to have different agents.Alessio [00:07:40]: Just to compare that with Bot.new, you're almost focused on like the application is very complex and now you need better tools to kind of manage it and build on top of it. On Bot.new, it's almost like I was using it the other day. There's basically like, hey, look, I'm just trying to get started. You know, I'm not very opinionated on like how you're going to implement this. Like this is what I want to do. And you build a beautiful app with it. What people ask as the next step, you know, going back to like the general versus like specific, have you had people say, hey, you know, this is great to start, but then I want a specific Bot.new dot whatever else to do a more vertical integration and kind of like development or what's the, what do people say?Eric [00:08:18]: Yeah. I think, I think you kind of hit the, hit it head on, which is, you know, kind of the way that we've, we've kind of talked about internally is it's like people are using Bolt to go from like 0.0 to 1.0, like that's like kind of the biggest unlock that Bolt has versus most other things out there. I mean, I think that's kind of what's, what's very unique about Bolt. I think the, you know, the working on like existing enterprise applications is, I mean, it's crazy important because, you know, there's a, you look, when you look at the fortune 500, I mean, these code bases, some of these have been around for 20, 30 plus years. And so it's important to be going from, you know, 101.3 to 101.4, et cetera. I think for us, so what's been actually pretty interesting is we see there's kind of two different users for us that are coming in and it's very distinct. It's like people that are developers already. And then there's people that have never really written software and more if they have, it's been very, very minimal. And so in the first camp, what these developers are doing, like to go from zero to one, they're coming to Bolt and then they're ejecting the thing to get up or just downloading it and, you know, opening cursor, like whatever to, to, you know, keep iterating on the thing. And sometimes they'll bring it back to Bolt to like add in a huge piece of functionality or something. Right. But for the people that don't know how to code, they're actually just, they, they live in this thing. And that was one of the weird things when we launched is, you know, within a day of us being online, one of the most popular YouTube videos, and there's been a ton since, which was, you know, there's like, oh, Bolt is the cursor killer. And I originally saw the headlines and I was like, thanks for the views. I mean, I don't know. This doesn't make sense to me. That's not, that's not what we kind of thought.Swyx [00:09:44]: It's how YouTubers talk to each other. Well, everything kills everything else.Eric [00:09:47]: Totally. But what blew my mind was that there was any comparison because it's like cursor is a, is a local IDE product. But when, when we actually kind of dug into it and we, and we have people that are using our product saying this, I'm not using cursor. And I was like, what? And it turns out there are hundreds of thousands of people that we have seen that we're using cursor and we're trying to build apps with that where they're not traditional software does, but we're heavily leaning on the AI. And as you can imagine, it is very complicated, right? To do that with cursor. So when Bolt came out, they're like, wow, this thing's amazing because it kind of inverts the complexity where it's like, you know, it's not an IDE, it's, it's a, it's a chat-based sort of interface that we have. So that's kind of the split, which is rather interesting. We've had like the first startups now launch off of Bolt entirely where this, you know, tomorrow I'm doing a live stream with this guy named Paul, who he's built an entire CRM using this thing and you know, with backend, et cetera. And people have made their first money on the internet period, you know, launching this with Stripe or whatever have you. So that's, that's kind of the two main, the two main categories of folks that we see using Bolt though.Itamar [00:10:51]: I agree that I don't understand the comparison. It doesn't make sense to me. I think like we have like two type of families of tools. One is like we re-imagine the software development. I think Bolt is there and I think like a cursor is more like a evolution of what we already have. It's like taking the IDE and it's, it's amazing and it's okay, let's, let's adapt the IDE to an era where LLMs can do a lot for us. And Bolt is more like, okay, let's rethink everything totally. And I think we see a few tools there, like maybe Vercel, Veo and maybe Repl.it in that area. And then in the area of let's expedite, let's change, let's, let's progress with what we already have. You can see Cursor and Kodo, but we're different between ourselves, Cursor and Kodo, but definitely I think that comparison doesn't make sense.Alessio [00:11:42]: And just to set the context, this is not a Twitter demo. You've made 4 million of revenue in four weeks. So this is, this is actually working, you know, it's not a, what, what do you think that is? Like, there's been so many people demoing coding agents on Twitter and then it doesn't really work. And then you guys were just like, here you go, it's live, go use it, pay us for it. You know, is there anything in the development that was like interesting and maybe how that compares to building your own agents?Eric [00:12:08]: We had no idea, honestly, like we, we, we've been pretty blown away and, and things have just kind of continued to grow faster since then. We're like, oh, today is week six. So I, I kind of came back to the point you just made, right, where it's, you, you kind of outlined, it's like, there's kind of this new market of like kind of rethinking the software development and then there's heavily augmenting existing developers. I think that, you know, both of which are, you know, AI code gen being extremely good, it's allowed existing developers, it's allowing existing developers to camera out software far faster than they could have ever before, right? It's like the ultimate power tool for an existing developer. But this code gen stuff is now so good. And then, and we saw this over the past, you know, from the beginning of the year when we tried to first build, it's actually lowered the barrier to people that, that aren't traditionally software engineers. But the kind of the key thing is if you kind of think about it from, imagine you've never written software before, right? My co-founder and I, he and I grew up down the street from each other in Chicago. We learned how to code when we were 13 together and we've been building stuff ever since. And this is back in like the mid 2000s or whatever, you know, there was nothing for free to learn from online on the internet and how to code. For our 13th birthdays, we asked our parents for, you know, O'Reilly books cause you couldn't get this at the library, right? And so instead of like an Xbox, we got, you know, programming books. But the hardest part for everyone learning to code is getting an environment set up locally, you know? And so when we built StackBlitz, like kind of the key thesis, like seven years ago, the insight we had was that, Hey, it seems like the browser has a lot of new APIs like WebAssembly and service workers, et cetera, where you could actually write an operating system that ran inside the browser that could boot in milliseconds. And you, you know, basically there's this missing capability of the web. Like the web should be able to build apps for the web, right? You should be able to build the web on the web. Every other platform has that, Visual Studio for Windows, Xcode for Mac. The web has no built in primitive for this. And so just like our built in kind of like nerd instinct on this was like, that seems like a huge hole and it's, you know, it will be very valuable or like, you know, very valuable problem to solve. So if you want to set up that environments, you know, this is what we spent the past seven years doing. And the reality is existing developers have running locally. They already know how to set up that environment. So the problem isn't as acute for them. When we put Bolt online, we took that technology called WebContainer and married it with these, you know, state of the art frontier models. And the people that have the most pain with getting stuff set up locally is people that don't code. I think that's been, you know, really the big explosive reason is no one else has been trying to make dev environments work inside of a browser tab, you know, for the past if since ever, other than basically our company, largely because there wasn't an immediate demand or need. So I think we kind of find ourselves at the right place at the right time. And again, for this market of people that don't know how to write software, you would kind of expect that you should be able to do this without downloading something to your computer in the same way that, hey, I don't have to download Photoshop now to make designs because there's Figma. I don't have to download Word because there's, you know, Google Docs. They're kind of looking at this as that sort of thing, right? Which was kind of the, you know, our impetus and kind of vision from the get-go. But you know, the code gen, the AI code gen stuff that's come out has just been, you know, an order of magnitude multiplier on how magic that is, right? So that's kind of my best distillation of like, what is going on here, you know?Alessio [00:15:21]: And you can deploy too, right?Eric [00:15:22]: Yeah.Alessio [00:15:23]: Yeah.Eric [00:15:24]: And so that's, what's really cool is it's, you know, we have deployment built in with Netlify and this is actually, I think, Sean, you actually built this at Netlify when you were there. Yeah. It's one of the most brilliant integrations actually, because, you know, effectively the API that Sean built, maybe you can speak to it, but like as a provider, we can just effectively give files to Netlify without the user even logging in and they have a live website. And if they want to keep, hold onto it, they can click a link and claim it to their Netlify account. But it basically is just this really magic experience because when you come to Bolt, you say, I want a website. Like my mom, 70, 71 years old, made her first website, you know, on the internet two weeks ago, right? It was about her nursing days.Swyx [00:16:03]: Oh, that's fantastic though. It wouldn't have been made.Eric [00:16:06]: A hundred percent. Cause even in, you know, when we've had a lot of people building personal, like deeply personal stuff, like in the first week we launched this, the sales guy from the East Coast, you know, replied to a tweet of mine and he said, thank you so much for building this to your team. His daughter has a medical condition and so for her to travel, she has to like line up donors or something, you know, so ahead of time. And so he actually used Bolt to make a website to do that, to actually go and send it to folks in the region she was going to travel to ahead of time. I was really touched by it, but I also thought like, why, you know, why didn't he use like Wix or Squarespace? Right? I mean, this is, this is a solved problem, quote unquote, right? And then when I thought, I actually use Squarespace for my, for my, uh, the wedding website for my wife and I, like back in 2021, so I'm familiar, you know, it was, it was faster. I know how to code. I was like, this is faster. Right. And I thought back and I was like, there's a whole interface you have to learn how to use. And it's actually not that simple. There's like a million things you can configure in that thing. When you come to Bolt, there's a, there's a text box. You just say, I need a, I need a wedding website. Here's the date. Here's where it is. And here's a photo of me and my wife, put it somewhere relevant. It's actually the simplest way. And that's what my, when my mom came, she said, uh, I'm Pat Simons. I was a nurse in the seventies, you know, and like, here's the things I did and a website came out. So coming back to why is this such a, I think, why are we seeing this sort of growth? It's, this is the simplest interface I think maybe ever created to actually build it, a deploy a website. And then that website, my mom made, she's like, okay, this looks great. And there's, there's one button, you just click it, deploy, and it's live and you can buy a domain name, attach it to it. And you know, it's as simple as it gets, it's getting even simpler with some of the stuff we're working on. But anyways, so that's, it's, it's, uh, it's been really interesting to see some of the usage like that.Swyx [00:17:46]: I can offer my perspective. So I, you know, I probably should have disclosed a little bit that, uh, I'm a, uh, stack list investor.Alessio [00:17:53]: Canceled the episode. I know, I know. Don't play it now. Pause.Eric actually reached out to ShowMeBolt before the launch. And we, you know, we talked a lot about, like, the framing of, of what we're going to talk about how we marketed the thing, but also, like, what we're So that's what Bolt was going to need, like a whole sort of infrastructure.swyx: Netlify, I was a maintainer but I won't take claim for the anonymous upload. That's actually the origin story of Netlify. We can have Matt Billman talk about it, but that was [00:18:00] how Netlify started. You could drag and drop your zip file or folder from your desktop onto a website, it would have a live URL with no sign in.swyx: And so that was the origin story of Netlify. And it just persists to today. And it's just like it's really nice, interesting that both Bolt and CognitionDevIn and a bunch of other sort of agent type startups, they all use Netlify to deploy because of this one feature. They don't really care about the other features.swyx: But, but just because it's easy for computers to use and talk to it, like if you build an interface for computers specifically, that it's easy for them to Navigate, then they will be used in agents. And I think that's a learning that a lot of developer tools companies are having. That's my bolt launch story and now if I say all that stuff.swyx: And I just wanted to come back to, like, the Webcontainers things, right? Like, I think you put a lot of weight on the technical modes. I think you also are just like, very good at product. So you've, you've like, built a better agent than a lot of people, the rest of us, including myself, who have tried to build these things, and we didn't get as far as you did.swyx: Don't shortchange yourself on products. But I think specifically [00:19:00] on, on infra, on like the sandboxing, like this is a thing that people really want. Alessio has Bax E2B, which we'll have on at some point, talking about like the sort of the server full side. But yours is, you know, inside of the browser, serverless.swyx: It doesn't cost you anything to serve one person versus a million people. It doesn't, doesn't cost you anything. I think that's interesting. I think in theory, we should be able to like run tests because you can run the full backend. Like, you can run Git, you can run Node, you can run maybe Python someday.swyx: We talked about this. But ideally, you should be able to have a fully gentic loop, running code, seeing the errors, correcting code, and just kind of self healing, right? Like, I mean, isn't that the dream?Eric: Totally.swyx: Yeah,Eric: totally. At least in bold, we've got, we've got a good amount of that today. I mean, there's a lot more for us to do, but one of the nice things, because like in web container, you know, there's a lot of kind of stuff you go Google like, you know, turn docker container into wasm.Eric: You'll find a lot of stuff out there that will do that. The problem is it's very big, it's slow, and that ruins the experience. And so what we ended up doing is just writing an operating system from [00:20:00] scratch that was just purpose built to, you know, run in a browser tab. And the reason being is, you know, Docker 2 awesome things will give you an image that's like out 60 to 100 megabits, you know, maybe more, you know, and our, our OS, you know, kind of clocks in, I think, I think we're in like a, maybe, maybe a megabyte or less or something like that.Eric: I mean, it's, it's, you know, really, really, you know, stripped down.swyx: This is basically the task involved is I understand that it's. Mapping every single, single Linux call to some kind of web, web assembly implementation,Eric: but more or less, and, and then there's a lot of things actually, like when you're looking at a dev environment, there's a lot of things that you don't need that a traditional OS is gonna have, right?Eric: Like, you know audio drivers or you like, there's just like, there's just tons of things. Oh, yeah. Right. Yeah. That goes . Yeah. You can just kind, you can, you can kind of tos them. Or alternatively, what you can do is you can actually be the nice thing. And this is, this kind of comes back to the origins of browsers, which is, you know, they're, they're at the beginning of the web and, you know, the late nineties, there was two very different kind of visions for the web where Alan Kay vehemently [00:21:00] disagree with the idea that should be document based, which is, you know, Tim Berners Lee, you know, that, and that's kind of what ended up winning, winning was this document based kind of browsing documents on the web thing.Eric: Alan Kay, he's got this like very famous quote where he said, you know, you want web browsers to be mini operating systems. They should download little mini binaries and execute with like a little mini virtualized operating system in there. And what's kind of interesting about the history, not to geek out on this aspect, what's kind of interesting about the history is both of those folks ended up being right.Eric: Documents were actually the pragmatic way that the web worked. Was, you know, became the most ubiquitous platform in the world to the degree now that this is why WebAssembly has been invented is that we're doing, we need to do more low level things in a browser, same thing with WebGPU, et cetera. And so all these APIs, you know, to build an operating system came to the browser.Eric: And that was actually the realization we had in 2017 was, holy heck, like you can actually, you know, service workers, which were designed for allowing your app to work offline. That was the kind of the key one where it was like, wait a second, you can actually now run. Web servers within a [00:22:00] browser, like you can run a server that you open up.Eric: That's wild. Like full Node. js. Full Node. js. Like that capability. Like, I can have a URL that's programmatically controlled. By a web application itself, boom. Like the web can build the web. The primitive is there. Everyone at the time, like we talked to people that like worked on, you know Chrome and V8 and they were like, uhhhh.Eric: You know, like I don't know. But it's one of those things you just kind of have to go do it to find out. So we spent a couple of years, you know, working on it and yeah. And, and, and got to work in back in 2021 is when we kind of put the first like data of web container online. Butswyx: in partnership with Google, right?swyx: Like Google actually had to help you get over the finish line with stuff.Eric: A hundred percent, because well, you know, over the years of when we were doing the R and D on the thing. Kind of the biggest challenge, the two ways that you can kind of test how powerful and capable a platform are, the two types of applications are one, video games, right, because they're just very compute intensive, a lot of calculations that have to happen, right?Eric: The second one are IDEs, because you're talking about actually virtualizing the actual [00:23:00] runtime environment you are in to actually build apps on top of it, which requires sophisticated capabilities, a lot of access to data. You know, a good amount of compute power, right, to effectively, you know, building app in app sort of thing.Eric: So those, those are the stress tests. So if your platform is missing stuff, those are the things where you find out. Those are, those are the people building games and IDEs. They're the ones filing bugs on operating system level stuff. And for us, browser level stuff.Eric [00:23:47]: yeah, what ended up happening is we were just hammering, you know, the Chromium bug tracker, and they're like, who are these guys? Yeah. And, and they were amazing because I mean, just making Chrome DevTools be able to debug, I mean, it's, it's not, it wasn't originally built right for debugging an operating system, right? They've been phenomenal working with us and just kind of really pushing the limits, but that it's a rising tide that's kind of lifted all boats because now there's a lot of different types of applications that you can debug with Chrome Dev Tools that are running a browser that runs more reliably because just the stress testing that, that we and, you know, games that are coming to the web are kind of pushing as well, but.Itamar [00:24:23]: That's awesome. About the testing, I think like most, let's say coding assistant from different kinds will need this loop of testing. And even I would add code review to some, to some extent that you mentioned. How is testing different from code review? Code review could be, for example, PR review, like a code review that is done at the point of when you want to merge branches. But I would say that code review, for example, checks best practices, maintainability, and so on. It's not just like CI, but more than CI. And testing is like a more like checking functionality, et cetera. So it's different. We call, by the way, all of these together code integrity, but that's a different story. Just to go back to the, to the testing and specifically. Yeah. It's, it's, it's since the first slide. Yeah. We're consistent. So if we go back to the testing, I think like, it's not surprising that for us testing is important and for Bolt it's testing important, but I want to shed some light on a different perspective of it. Like let's think about autonomous driving. Those startups that are doing autonomous driving for highway and autonomous driving for the city. And I think like we saw the autonomous of the highway much faster and reaching to a level, I don't know, four or so much faster than those in the city. Now, in both cases, you need testing and quote unquote testing, you know, verifying validation that you're doing the right thing on the road and you're reading and et cetera. But it's probably like so different in the city that it could be like actually different technology. And I claim that we're seeing something similar here. So when you're building the next Wix, and if I was them, I was like looking at you and being a bit scared. That's what you're disrupting, what you just said. Then basically, I would say that, for example, the UX UI is freaking important. And because you're you're more aiming for the end user. In this case, maybe it's an end user that doesn't know how to develop for developers. It's also important. But let alone those that do not know to develop, they need a slick UI UX. And I think like that's one reason, for example, I think Cursor have like really good technology. I don't know the underlying what's under the hood, but at least what they're saying. But I think also their UX UI is great. It's a lot because they did their own ID. While if you're aiming for the city AI, suddenly like there's a lot of testing and code review technology that it's not necessarily like that important. For example, let's talk about integration tests. Probably like a lot of what you're building involved at the moment is isolated applications. Maybe the vision or the end game is maybe like having one solution for everything. It could be that eventually the highway companies will go into the city and the other way around. But at the beginning, there is a difference. And integration tests are a good example. I guess they're a bit less important. And when you think about enterprise software, they're really important. So to recap, like I think like the idea of looping and verifying your test and verifying your code in different ways, testing or code review, et cetera, seems to be important in the highway AI and the city AI, but in different ways and different like critical for the city, even more and more variety. Actually, I was looking to ask you like what kind of loops you guys are doing. For example, when I'm using Bolt and I'm enjoying it a lot, then I do see like sometimes you're trying to catch the errors and fix them. And also, I noticed that you're breaking down tasks into smaller ones and then et cetera, which is already a common notion for a year ago. But it seems like you're doing it really well. So if you're willing to share anything about it.Eric [00:28:07]: Yeah, yeah. I realized I never actually hit the punchline of what I was saying before. I mentioned the point about us kind of writing an operating system from scratch because what ended up being important about that is that to your point, it's actually a very, like compared to like a, you know, if you're like running cursor on anyone's machine, you kind of don't know what you're dealing with, with the OS you're running on. There could be an error happens. It could be like a million different things, right? There could be some config. There could be, it could be God knows what, right? The thing with WebConnect is because we wrote the entire thing from scratch. It's actually a unified image basically. And we can instrument it at any level that we think is going to be useful, which is exactly what we did when we started building Bolt is we instrumented stuff at like the process level, at the runtime level, you know, et cetera, et cetera, et cetera. Stuff that would just be not impossible to do on local, but to do that in a way that works across any operating system, whatever is, I mean, would just be insanely, you know, insanely difficult to do right and reliably. And that's what you saw when you've used Bolt is that when an error actually will occur, whether it's in the build process or the actual web application itself is failing or anything kind of in between, you can actually capture those errors. And today it's a very primitive way of how we've implemented it largely because the product just didn't exist 90 days ago. So we're like, we got some work ahead of us and we got to hire some more a little bit, but basically we present and we say, Hey, this is, here's kind of the things that went wrong. There's a fix it button and then a ignore button, and then you can just hit fix it. And then we take all that telemetry through our agent, you run it through our agent and say, kind of, here's the state of the application. Here's kind of the errors that we got from Node.js or the browser or whatever, and like dah, dah, dah, dah. And it can take a crack at actually solving it. And it's actually pretty darn good at being able to do that. That's kind of been a, you know, closing the loop and having it be a reliable kind of base has seemed to be a pretty big upgrade over doing stuff locally, just because I think that's a pretty key ingredient of it. And yeah, I think breaking things down into smaller tasks, like that's, that's kind of a key part of our agent. I think like Claude did a really good job with artifacts. I think, you know, us and kind of everyone else has, has kind of taken their approach of like actually breaking out certain tasks in a certain order into, you know, kind of a concrete way. And, and so actually the core of Bolt, I know we actually made open source. So you can actually go and check out like the system prompts and et cetera, and you can run it locally and whatever have you. So anyone that's interested in this stuff, I'd highly recommend taking a look at. There's not a lot of like stuff that's like open source in this realm. It's, that was one of the fun things that we've we thought would be cool to do. And people, people seem to like it. I mean, there's a lot of forks and people adding different models and stuff. So it's been cool to see.Swyx [00:30:41]: Yeah. I'm happy to add, I added real-time voice for my opening day demo and it was really fun to hack with. So thank you for doing that. Yeah. Thank you. I'm going to steal your code.Eric [00:30:52]: Because I want that.Swyx [00:30:52]: It's funny because I built on top of the fork of Bolt.new that already has the multi LLM thing. And so you just told me you're going to merge that in. So then you're going to merge two layers of forks down into this thing. So it'll be fun.Eric [00:31:03]: Heck yeah.Alessio [00:31:04]: Just to touch on like the environment, Itamar, you maybe go into the most complicated environments that even the people that work there don't know how to run. How much of an impact does that have on your performance? Like, you know, it's most of the work you're doing actually figuring out environment and like the libraries, because I'm sure they're using outdated version of languages, they're using outdated libraries, they're using forks that have not been on the public internet before. How much of the work that you're doing is like there versus like at the LLM level?Itamar [00:31:32]: One of the reasons I was asking about, you know, what are the steps to break things down, because it really matters. Like, what's the tech stack? How complicated the software is? It's hard to figure it out when you're dealing with the real world, any environment of enterprise as a city, when I'm like, while maybe sometimes like, I think you do enable like in Bolt, like to install stuff, but it's quite a like controlled environment. And that's a good thing to do, because then you narrow down and it's easier to make things work. So definitely, there are two dimensions, I think, actually spaces. One is the fact just like installing our software without yet like doing anything, making it work, just installing it because we work with enterprise and Fortune 500, etc. Many of them want on prem solution.Swyx [00:32:22]: So you have how many deployment options?Itamar [00:32:24]: Basically, we had, we did a metric metrics, say 96 options, because, you know, they're different dimensions. Like, for example, one dimension, we connect to your code management system to your Git. So are you having like GitHub, GitLab? Subversion? Is it like on cloud or deployed on prem? Just an example. Which model agree to use its APIs or ours? Like we have our Is it TestGPT? Yeah, when we started with TestGPT, it was a huge mistake name. It was cool back then, but I don't think it's a good idea to name a model after someone else's model. Anyway, that's my opinion. So we gotSwyx [00:33:02]: I'm interested in these learnings, like things that you change your mind on.Itamar [00:33:06]: Eventually, when you're building a company, you're building a brand and you want to create your own brand. By the way, when I thought about Bolt.new, I also thought about if it's not a problem, because when I think about Bolt, I do think about like a couple of companies that are already called this way.Swyx [00:33:19]: Curse companies. You could call it Codium just to...Itamar [00:33:24]: Okay, thank you. Touche. Touche.Eric [00:33:27]: Yeah, you got to imagine the board meeting before we launched Bolt, one of our investors, you can imagine they're like, are you sure? Because from the investment side, it's kind of a famous, very notorious Bolt. And they're like, are you sure you want to go with that name? Oh, yeah. Yeah, absolutely.Itamar [00:33:43]: At this point, we have actually four models. There is a model for autocomplete. There's a model for the chat. There is a model dedicated for more for code review. And there is a model that is for code embedding. Actually, you might notice that there isn't a good code embedding model out there. Can you name one? Like dedicated for code?Swyx [00:34:04]: There's code indexing, and then you can do sort of like the hide for code. And then you can embed the descriptions of the code.Itamar [00:34:12]: Yeah, but you do see a lot of type of models that are dedicated for embedding and for different spaces, different fields, etc. And I'm not aware. And I know that if you go to the bedrock, try to find like there's a few code embedding models, but none of them are specialized for code.Swyx [00:34:31]: Is there a benchmark that you would tell us to pay attention to?Itamar [00:34:34]: Yeah, so it's coming. Wait for that. Anyway, we have our models. And just to go back to the 96 option of deployment. So I'm closing the brackets for us. So one is like dimensional, like what Git deployment you have, like what models do you agree to use? Dotter could be like if it's air-gapped completely, or you want VPC, and then you have Azure, GCP, and AWS, which is different. Do you use Kubernetes or do not? Because we want to exploit that. There are companies that do not do that, etc. I guess you know what I mean. So that's one thing. And considering that we are dealing with one of all four enterprises, we needed to deal with that. So you asked me about how complicated it is to solve that complex code. I said, it's just a deployment part. And then now to the software, we see a lot of different challenges. For example, some companies, they did actually a good job to build a lot of microservices. Let's not get to if it's good or not, but let's first assume that it is a good thing. A lot of microservices, each one of them has their own repo. And now you have tens of thousands of repos. And you as a developer want to develop something. And I remember me coming to a corporate for the first time. I don't know where to look at, like where to find things. So just doing a good indexing for that is like a challenge. And moreover, the regular indexing, the one that you can find, we wrote a few blogs on that. By the way, we also have some open source, different than yours, but actually three and growing. Then it doesn't work. You need to let the tech leads and the companies influence your indexing. For example, Mark with different repos with different colors. This is a high quality repo. This is a lower quality repo. This is a repo that we want to deprecate. This is a repo we want to grow, etc. And let that be part of your indexing. And only then things actually work for enterprise and they don't get to a fatigue of, oh, this is awesome. Oh, but I'm starting, it's annoying me. I think Copilot is an amazing tool, but I'm quoting others, meaning GitHub Copilot, that they see not so good retention of GitHub Copilot and enterprise. Ooh, spicy. Yeah. I saw snapshots of people and we have customers that are Copilot users as well. And also I saw research, some of them is public by the way, between 38 to 50% retention for users using Copilot and enterprise. So it's not so good. By the way, I don't think it's that bad, but it's not so good. So I think that's a reason because, yeah, it helps you auto-complete, but then, and especially if you're working on your repo alone, but if it's need that context of remote repos that you're code-based, that's hard. So to make things work, there's a lot of work on that, like giving the controllability for the tech leads, for the developer platform or developer experience department in the organization to influence how things are working. A short example, because if you have like really old legacy code, probably some of it is not so good anymore. If you just fine tune on these code base, then there is a bias to repeat those mistakes or old practices, etc. So you need, for example, as I mentioned, to influence that. For example, in Coda, you can have a markdown of best practices by the tech leads and Coda will include that and relate to that and will not offer suggestions that are not according to the best practices, just as an example. So that's just a short list of things that you need to do in order to deal with, like you mentioned, the 100.1 to 100.2 version of software. I just want to say what you're doing is extremelyEric [00:38:32]: impressive because it's very difficult. I mean, the business of Stackplus, kind of before bulk came online, we sold a version of our IDE that went on-prem. So I understand what you're saying about the difficulty of getting stuff just working on-prem. Holy heck. I mean, that is extremely hard. I guess the question I have for you is, I mean, we were just doing that with kind of Kubernetes-based stuff, but the spread of Fortune 500 companies that you're working with, how are they doing the inference for this? Are you kind of plugging into Azure's OpenAI stuff and AWS's Bedrock, you know, Cloud stuff? Or are they just like running stuff on GPUs? Like, what is that? How are these folks approaching that? Because, man, what we saw on the enterprise side, I mean, I got to imagine that that's a huge challenge. Everything you said and more, like,Itamar [00:39:15]: for example, like someone could be, and I don't think any of these is bad. Like, they made their decision. Like, for example, some people, they're, I want only AWS and VPC on AWS, no matter what. And then they, some of them, like there is a subset, I will say, I'm willing to take models only for from Bedrock and not ours. And we have a problem because there is no good code embedding model on Bedrock. And that's part of what we're doing now with AWS to solve that. We solve it in a different way. But if you are willing to run on AWS VPC, but run your run models on GPUs or inferentia, like the new version of the more coming out, then our models can run on that. But everything you said is right. Like, we see like on-prem deployment where they have their own GPUs. We see Azure where you're using OpenAI Azure. We see cases where you're running on GCP and they want OpenAI. Like this cross, like a case, although there is Gemini or even Sonnet, I think is available on GCP, just an example. So all the options, that's part of the challenge. I admit that we thought about it, but it was even more complicated. And it took us a few months to actually, that metrics that I mentioned, to start clicking each one of the blocks there. A few months is impressive. I mean,Eric [00:40:35]: honestly, just that's okay. Every one of these enterprises is, their networking is different. Just everything's different. Every single one is different. I see you understand. Yeah. So that just cannot be understated. That it is, that's extremely impressive. Hats off.Itamar [00:40:50]: It could be, by the way, like, for example, oh, we're only AWS, but our GitHub enterprise is on-prem. Oh, we forgot. So we need like a private link or whatever, like every time like that. It's not, and you do need to think about it if you want to work with an enterprise. And it's important. Like I understand like their, I respect their point of view.Swyx [00:41:10]: And this primarily impacts your architecture, your tech choices. Like you have to, you can't choose some vendors because...Itamar [00:41:15]: Yeah, definitely. To be frank, it makes us hard for a startup because it means that we want, we want everyone to enjoy all the variety of models. By the way, it was hard for us with our technology. I want to open a bracket, like a window. I guess you're familiar with our Alpha Codium, which is an open source.Eric [00:41:33]: We got to go over that. Yeah. So I'll do that quickly.Itamar [00:41:36]: Yeah. A pin in that. Yeah. Actually, we didn't have it in the last episode. So, so, okay.Swyx [00:41:41]: Okay. We'll come back to that later, but let's talk about...Itamar [00:41:43]: Yeah. So, so just like shortly, and then we can double click on Alpha Codium. But Alpha Codium is a open source tool. You can go and try it and lets you compete on CodeForce. This is a website and a competition and actually reach a master level level, like 95% with a click of a button. You don't need to do anything. And part of what we did there is taking a problem and breaking it to different, like smaller blocks. And then the models are doing a much better job. Like we all know it by now that taking small tasks and solving them, by the way, even O1, which is supposed to be able to do system two thinking like Greg from OpenAI like hinted, is doing better on these kinds of problems. But still, it's very useful to break it down for O1, despite O1 being able to think by itself. And that's what we presented like just a month ago, OpenAI released that now they are doing 93 percentile with O1 IOI left and International Olympiad of Formation. Sorry, I forgot. Exactly. I told you I forgot. And we took their O1 preview with Alpha Codium and did better. Like it just shows like, and there is a big difference between the preview and the IOI. It shows like that these models are not still system two thinkers, and there is a big difference. So maybe they're not complete system two. Yeah, they need some guidance. I call them system 1.5. We can, we can have it. I thought about it. Like, you know, I care about this philosophy stuff. And I think like we didn't see it even close to a system two thinking. I can elaborate later. But closing the brackets, like we take Alpha Codium and as our principle of thinking, we take tasks and break them down to smaller tasks. And then we want to exploit the best model to solve them. So I want to enable anyone to enjoy O1 and SONET and Gemini 1.5, etc. But at the same time, I need to develop my own models as well, because some of the Fortune 500 want to have all air gapped or whatever. So that's a challenge. Now you need to support so many models. And to some extent, I would say that the flow engineering, the breaking down to two different blocks is a necessity for us. Why? Because when you take a big block, a big problem, you need a very different prompt for each one of the models to actually work. But when you take a big problem and break it into small tasks, we can talk how we do that, then the prompt matters less. What I want to say, like all this, like as a startup trying to do different deployment, getting all the juice that you can get from models, etc. is a big problem. And one need to think about it. And one of our mitigation is that process of taking tasks and breaking them down. That's why I'm really interested to know how you guys are doing it. And part of what we do is also open source. So you can see.Swyx [00:44:39]: There's a lot in there. But yeah, flow over prompt. I do believe that that does make sense. I feel like there's a lot that both of you can sort of exchange notes on breaking down problems. And I just want you guys to just go for it. This is fun to watch.Eric [00:44:55]: Yeah. I mean, what's super interesting is the context you're working in is, because for us too with Bolt, we've started thinking because our kind of existing business line was going behind the firewall, right? We were like, how do we do this? Adding the inference aspect on, we're like, okay, how does... Because I mean, there's not a lot of prior art, right? I mean, this is all new. This is all new. So I definitely am going to have a lot of questions for you.Itamar [00:45:17]: I'm here. We're very open, by the way. We have a paper on a blog or like whatever.Swyx [00:45:22]: The Alphacodeum, GitHub, and we'll put all this in the show notes.Itamar [00:45:25]: Yeah. And even the new results of O1, we published it.Eric [00:45:29]: I love that. And I also just, I think spiritually, I like your approach of being transparent. Because I think there's a lot of hype-ium around AI stuff. And a lot of it is, it's just like, you have these companies that are just kind of keep their stuff closed source and then just max hype it, but then it's kind of nothing. And I think it kind of gives a bad rep to the incredible stuff that's actually happening here. And so I think it's stuff like what you're doing where, I mean, true merit and you're cracking open actual code for others to learn from and use. That strikes me as the right approach. And it's great to hear that you're making such incredible progress.Itamar [00:46:02]: I have something to share about the open source. Most of our tools are, we have an open source version and then a premium pro version. But it's not an easy decision to do that. I actually wanted to ask you about your strategy, but I think in your case, there is, in my opinion, relatively a good strategy where a lot of parts of open source, but then you have the deployment and the environment, which is not right if I get it correctly. And then there's a clear, almost hugging face model. Yeah, you can do that, but why should you try to deploy it yourself, deploy it with us? But in our case, and I'm not sure you're not going to hit also some competitors, and I guess you are. I wanted to ask you, for example, on some of them. In our case, one day we looked on one of our competitors that is doing code review. We're a platform. We have the code review, the testing, et cetera, spread over the ID to get. And in each agent, we have a few startups or a big incumbents that are doing only that. So we noticed one of our competitors having not only a very similar UI of our open source, but actually even our typo. And you sit there and you're kind of like, yeah, we're not that good. We don't use enough Grammarly or whatever. And we had a couple of these and we saw it there. And then it's a challenge. And I want to ask you, Bald is doing so well, and then you open source it. So I think I know what my answer was. I gave it before, but still interestingEric [00:47:29]: to hear what you think. GeoHot said back, I don't know who he was up to at this exact moment, but I think on comma AI, all that stuff's open source. And someone had asked him, why is this open source? And he's like, if you're not actually confident that you can go and crush it and build the best thing, then yeah, you should probably keep your stuff closed source. He said something akin to that. I'm probably kind of butchering it, but I thought it was kind of a really good point. And that's not to say that you should just open source everything, because for obvious reasons, there's kind of strategic things you have to kind of take in mind. But I actually think a pretty liberal approach, as liberal as you kind of can be, it can really make a lot of sense. Because that is so validating that one of your competitors is taking your stuff and they're like, yeah, let's just kind of tweak the styles. I mean, clearly, right? I think it's kind of healthy because it keeps, I'm sure back at HQ that day when you saw that, you're like, oh, all right, well, we have to grind even harder to make sure we stay ahead. And so I think it's actually a very useful, motivating thing for the teams. Because you might feel this period of comfort. I think a lot of companies will have this period of comfort where they're not feeling the competition and one day they get disrupted. So kind of putting stuff out there and letting people push it forces you to face reality soon, right? And actually feel that incrementally so you can kind of adjust course. And that's for us, the open source version of Bolt has had a lot of features people have been begging us for, like persisting chat messages and checkpoints and stuff. Within the first week, that stuff was landed in the open source versions. And they're like, why can't you ship this? It's in the open, so people have forked it. And we're like, we're trying to keep our servers and GPUs online. But it's been great because the folks in the community did a great job, kept us on our toes. And we've got to know most of these folks too at this point that have been building these things. And so it actually was very instructive. Like, okay, well, if we're going to go kind of land this, there's some UX patterns we can kind of look at and the code is open source to this stuff. What's great about these, what's not. So anyways, NetNet, I think it's awesome. I think from a competitive point of view for us, I think in particular, what's interesting is the core technology of WebContainer going. And I think that right now, there's really nothing that's kind of on par with that. And we also, we have a business of, because WebContainer runs in your browser, but to make it work, you have to install stuff from NPM. You have to make cores bypass requests, like connected databases, which all require server-side proxying or acceleration. And so we actually sell WebContainer as a service. One of the core reasons we open-sourced kind of the core components of Bolt when we launched was that we think that there's going to be a lot more of these AI, in-your-browser AI co-gen experiences, kind of like what Anthropic did with Artifacts and Clod. By the way, Artifacts uses WebContainers. Not yet. No, yeah. Should I strike that? I think that they've got their own thing at the moment, but there's been a lot of interest in WebContainers from folks doing things in that sort of realm and in the AI labs and startups and everything in between. So I think there'll be, I imagine, over the coming months, there'll be lots of things being announced to folks kind of adopting it. But yeah, I think effectively...Swyx [00:50:35]: Okay, I'll say this. If you're a large model lab and you want to build sandbox environments inside of your chat app, you should call Eric.Itamar [00:50:43]: But wait, wait, wait, wait, wait, wait. I have a question about that. I think OpenAI, they felt that people are not using their model as they would want to. So they built ChatGPT. But I would say that ChatGPT now defines OpenAI. I know they're doing a lot of business from their APIs, but still, is this how you think? Isn't Bolt.new your business now? Why don't you focus on that instead of the...Swyx [00:51:16]: What's your advice as a founder?Eric [00:51:18]: You're right. And so going into it, we, candidly, we were like, Bolt.new, this thing is super cool. We think people are stoked. We think people will be stoked. But we were like, maybe that's allowed. Best case scenario, after month one, we'd be mind blown if we added a couple hundred K of error or something. And we were like, but we think there's probably going to be an immediate huge business. Because there was some early poll on folks wanting to put WebContainer into their product offerings, kind of similar to what Bolt is doing or whatever. We were actually prepared for the inverse outcome here. But I mean, well, I guess we've seen poll on both. But I mean, what's happened with Bolt, and you're right, it's actually the same strategy as like OpenAI or Anthropic, where we have our ChatGPT to OpenAI's APIs is Bolt to WebContainer. And so we've kind of taken that same approach. And we're seeing, I guess, some of the similar results, except right now, the revenue side is extremely lopsided to Bolt.Itamar [00:52:16]: I think if you ask me what's my advice, I think you have three options. One is to focus on Bolt. The other is to focus on the WebContainer. The third is to raise one billion dollars and do them both. I'm serious. I think otherwise, you need to choose. And if you raise enough money, and I think it's big bucks, because you're going to be chased by competitors. And I think it will be challenging to do both. And maybe you can. I don't know. We do see these numbers right now, raising above $100 million, even without havingEric [00:52:49]: a product. You can see these. It's excellent advice. And I think what's been amazing, but also kind of challenging is we're trying to forecast, okay, well, where are these things going? I mean, in the initial weeks, I think us and all the investors in the company that we're sharing this with, it was like, this is cool. Okay, we added 500k. Wow, that's crazy. Wow, we're at a million now. Most things, you have this kind of the tech crunch launch of initiation and then the thing of sorrow. And if there's going to be a downtrend, it's just not coming yet. Now that we're kind of looking ahead, we're six weeks in. So now we're getting enough confidence in our convictions to go, okay, this se