Podcasts about Observable

In this episode of R Weekly Highlights: We have a six-month follow-up perspective from an early Positron user, how the current landscape of AI tools perform when learning the ropes with the Tidyverse, and how you can create your first Observable plot while using R for data munging.Episode LinksThis week's curator: Jon Carroll - @jonocarroll@fosstodon.org (Mastodon) & @jonocarroll.fosstodon.org.ap.brid.gy (Bluesky) & @carroll_jono (X/Twitter)Positron: current joys and painsLearning the tidyverse with the help of AI toolsObservable for R usersEntire issue available at rweekly.org/2025-W15Supplement ResourcesPositron +1e https://open-vsx.org/extension/grrrck/positron-plus-1-eVanishing Gradients episode 47 (The Great Pacific Garbage Patch of Code Slop with Joe Reis) https://vanishinggradients.fireside.fm/47Observable color palette viewer https://observablehq.com/plot/features/scales#color-scalesObservable Plots (R/Pharma 2024 Workshop Series) https://www.youtube.com/watch?v=M6fP68XnacMSupporting the showUse the contact page at https://serve.podhome.fm/custompage/r-weekly-highlights/contact to send us your feedbackR-Weekly Highlights on the Podcastindex.org - You can send a boost into the show directly in the Podcast Index. First, top-up with Alby, and then head over to the R-Weekly Highlights podcast entry on the index.A new way to think about value: https://value4value.infoGet in touch with us on social mediaEric Nantz: @rpodcast@podcastindex.social (Mastodon), @rpodcast.bsky.social (BlueSky) and @theRcast (X/Twitter)Mike Thomas: @mike_thomas@fosstodon.org (Mastodon), @mike-thomas.bsky.social (BlueSky), and @mike_ketchbrook (X/Twitter) Music credits powered by OCRemixSunny Side Up - Yoshi's Island DS - ZackParrish - https://ocremix.org/remix/OCR04558Costa Del Sol DANCE - Final Fantasy VII - Posu Yan - https://ocremix.org/remix/OCR00095

User research is an underappreciated art - we in tech are so used to being immersed in an ocean of quantitative data that we can forget that on the other side of the screen are real humans who want to solve very specific problems. And often times, their problems are extremely hard to put a number on. Why did they abandon their cart right before checkout? What made them start creating a new newsletter but then abandon it but come back a month later? Not everything can be answered with a SQL query against the telemetry database. Marisa Morby, a Principal Researcher at Observable, sat down with me to help me better understand what it means to be great (not just good) at user research, and how that can help produce a whole new range of unexpected product insights. And Marisa definitely knows what the impact of great user research can be on the product - she previously worked at such notable companies like Netlify, Gatsby, and Accenture Song, where she honed her skills and UX instincts.

Today's episode is with Paul Klein, founder of Browserbase. We talked about building browser infrastructure for AI agents, the future of agent authentication, and their open source framework Stagehand.* [00:00:00] Introductions* [00:04:46] AI-specific challenges in browser infrastructure* [00:07:05] Multimodality in AI-Powered Browsing* [00:12:26] Running headless browsers at scale* [00:18:46] Geolocation when proxying* [00:21:25] CAPTCHAs and Agent Auth* [00:28:21] Building “User take over” functionality* [00:33:43] Stagehand: AI web browsing framework* [00:38:58] OpenAI's Operator and computer use agents* [00:44:44] Surprising use cases of Browserbase* [00:47:18] Future of browser automation and market competition* [00:53:11] Being a solo founderTranscriptAlessio [00:00:04]: Hey everyone, welcome to the Latent Space podcast. This is Alessio, partner and CTO at Decibel Partners, and I'm joined by my co-host Swyx, founder of Smol.ai.swyx [00:00:12]: Hey, and today we are very blessed to have our friends, Paul Klein, for the fourth, the fourth, CEO of Browserbase. Welcome.Paul [00:00:21]: Thanks guys. Yeah, I'm happy to be here. I've been lucky to know both of you for like a couple of years now, I think. So it's just like we're hanging out, you know, with three ginormous microphones in front of our face. It's totally normal hangout.swyx [00:00:34]: Yeah. We've actually mentioned you on the podcast, I think, more often than any other Solaris tenant. Just because like you're one of the, you know, best performing, I think, LLM tool companies that have started up in the last couple of years.Paul [00:00:50]: Yeah, I mean, it's been a whirlwind of a year, like Browserbase is actually pretty close to our first birthday. So we are one years old. And going from, you know, starting a company as a solo founder to... To, you know, having a team of 20 people, you know, a series A, but also being able to support hundreds of AI companies that are building AI applications that go out and automate the web. It's just been like, really cool. It's been happening a little too fast. I think like collectively as an AI industry, let's just take a week off together. I took my first vacation actually two weeks ago, and Operator came out on the first day, and then a week later, DeepSeat came out. And I'm like on vacation trying to chill. I'm like, we got to build with this stuff, right? So it's been a breakneck year. But I'm super happy to be here and like talk more about all the stuff we're seeing. And I'd love to hear kind of what you guys are excited about too, and share with it, you know?swyx [00:01:39]: Where to start? So people, you've done a bunch of podcasts. I think I strongly recommend Jack Bridger's Scaling DevTools, as well as Turner Novak's The Peel. And, you know, I'm sure there's others. So you covered your Twilio story in the past, talked about StreamClub, you got acquired to Mux, and then you left to start Browserbase. So maybe we just start with what is Browserbase? Yeah.Paul [00:02:02]: Browserbase is the web browser for your AI. We're building headless browser infrastructure, which are browsers that run in a server environment that's accessible to developers via APIs and SDKs. It's really hard to run a web browser in the cloud. You guys are probably running Chrome on your computers, and that's using a lot of resources, right? So if you want to run a web browser or thousands of web browsers, you can't just spin up a bunch of lambdas. You actually need to use a secure containerized environment. You have to scale it up and down. It's a stateful system. And that infrastructure is, like, super painful. And I know that firsthand, because at my last company, StreamClub, I was CTO, and I was building our own internal headless browser infrastructure. That's actually why we sold the company, is because Mux really wanted to buy our headless browser infrastructure that we'd built. And it's just a super hard problem. And I actually told my co-founders, I would never start another company unless it was a browser infrastructure company. And it turns out that's really necessary in the age of AI, when AI can actually go out and interact with websites, click on buttons, fill in forms. You need AI to do all of that work in an actual browser running somewhere on a server. And BrowserBase powers that.swyx [00:03:08]: While you're talking about it, it occurred to me, not that you're going to be acquired or anything, but it occurred to me that it would be really funny if you became the Nikita Beer of headless browser companies. You just have one trick, and you make browser companies that get acquired.Paul [00:03:23]: I truly do only have one trick. I'm screwed if it's not for headless browsers. I'm not a Go programmer. You know, I'm in AI grant. You know, browsers is an AI grant. But we were the only company in that AI grant batch that used zero dollars on AI spend. You know, we're purely an infrastructure company. So as much as people want to ask me about reinforcement learning, I might not be the best guy to talk about that. But if you want to ask about headless browser infrastructure at scale, I can talk your ear off. So that's really my area of expertise. And it's a pretty niche thing. Like, nobody has done what we're doing at scale before. So we're happy to be the experts.swyx [00:03:59]: You do have an AI thing, stagehand. We can talk about the sort of core of browser-based first, and then maybe stagehand. Yeah, stagehand is kind of the web browsing framework. Yeah.What is Browserbase? Headless Browser Infrastructure ExplainedAlessio [00:04:10]: Yeah. Yeah. And maybe how you got to browser-based and what problems you saw. So one of the first things I worked on as a software engineer was integration testing. Sauce Labs was kind of like the main thing at the time. And then we had Selenium, we had Playbrite, we had all these different browser things. But it's always been super hard to do. So obviously you've worked on this before. When you started browser-based, what were the challenges? What were the AI-specific challenges that you saw versus, there's kind of like all the usual running browser at scale in the cloud, which has been a problem for years. What are like the AI unique things that you saw that like traditional purchase just didn't cover? Yeah.AI-specific challenges in browser infrastructurePaul [00:04:46]: First and foremost, I think back to like the first thing I did as a developer, like as a kid when I was writing code, I wanted to write code that did stuff for me. You know, I wanted to write code to automate my life. And I do that probably by using curl or beautiful soup to fetch data from a web browser. And I think I still do that now that I'm in the cloud. And the other thing that I think is a huge challenge for me is that you can't just create a web site and parse that data. And we all know that now like, you know, taking HTML and plugging that into an LLM, you can extract insights, you can summarize. So it was very clear that now like dynamic web scraping became very possible with the rise of large language models or a lot easier. And that was like a clear reason why there's been more usage of headless browsers, which are necessary because a lot of modern websites don't expose all of their page content via a simple HTTP request. You know, they actually do require you to run this type of code for a specific time. JavaScript on the page to hydrate this. Airbnb is a great example. You go to airbnb.com. A lot of that content on the page isn't there until after they run the initial hydration. So you can't just scrape it with a curl. You need to have some JavaScript run. And a browser is that JavaScript engine that's going to actually run all those requests on the page. So web data retrieval was definitely one driver of starting BrowserBase and the rise of being able to summarize that within LLM. Also, I was familiar with if I wanted to automate a website, I could write one script and that would work for one website. It was very static and deterministic. But the web is non-deterministic. The web is always changing. And until we had LLMs, there was no way to write scripts that you could write once that would run on any website. That would change with the structure of the website. Click the login button. It could mean something different on many different websites. And LLMs allow us to generate code on the fly to actually control that. So I think that rise of writing the generic automation scripts that can work on many different websites, to me, made it clear that browsers are going to be a lot more useful because now you can automate a lot more things without writing. If you wanted to write a script to book a demo call on 100 websites, previously, you had to write 100 scripts. Now you write one script that uses LLMs to generate that script. That's why we built our web browsing framework, StageHand, which does a lot of that work for you. But those two things, web data collection and then enhanced automation of many different websites, it just felt like big drivers for more browser infrastructure that would be required to power these kinds of features.Alessio [00:07:05]: And was multimodality also a big thing?Paul [00:07:08]: Now you can use the LLMs to look, even though the text in the dome might not be as friendly. Maybe my hot take is I was always kind of like, I didn't think vision would be as big of a driver. For UI automation, I felt like, you know, HTML is structured text and large language models are good with structured text. But it's clear that these computer use models are often vision driven, and they've been really pushing things forward. So definitely being multimodal, like rendering the page is required to take a screenshot to give that to a computer use model to take actions on a website. And it's just another win for browser. But I'll be honest, that wasn't what I was thinking early on. I didn't even think that we'd get here so fast with multimodality. I think we're going to have to get back to multimodal and vision models.swyx [00:07:50]: This is one of those things where I forgot to mention in my intro that I'm an investor in Browserbase. And I remember that when you pitched to me, like a lot of the stuff that we have today, we like wasn't on the original conversation. But I did have my original thesis was something that we've talked about on the podcast before, which is take the GPT store, the custom GPT store, all the every single checkbox and plugin is effectively a startup. And this was the browser one. I think the main hesitation, I think I actually took a while to get back to you. The main hesitation was that there were others. Like you're not the first hit list browser startup. It's not even your first hit list browser startup. There's always a question of like, will you be the category winner in a place where there's a bunch of incumbents, to be honest, that are bigger than you? They're just not targeted at the AI space. They don't have the backing of Nat Friedman. And there's a bunch of like, you're here in Silicon Valley. They're not. I don't know.Paul [00:08:47]: I don't know if that's, that was it, but like, there was a, yeah, I mean, like, I think I tried all the other ones and I was like, really disappointed. Like my background is from working at great developer tools, companies, and nothing had like the Vercel like experience. Um, like our biggest competitor actually is partly owned by private equity and they just jacked up their prices quite a bit. And the dashboard hasn't changed in five years. And I actually used them at my last company and tried them and I was like, oh man, like there really just needs to be something that's like the experience of these great infrastructure companies, like Stripe, like clerk, like Vercel that I use in love, but oriented towards this kind of like more specific category, which is browser infrastructure, which is really technically complex. Like a lot of stuff can go wrong on the internet when you're running a browser. The internet is very vast. There's a lot of different configurations. Like there's still websites that only work with internet explorer out there. How do you handle that when you're running your own browser infrastructure? These are the problems that we have to think about and solve at BrowserBase. And it's, it's certainly a labor of love, but I built this for me, first and foremost, I know it's super cheesy and everyone says that for like their startups, but it really, truly was for me. If you look at like the talks I've done even before BrowserBase, and I'm just like really excited to try and build a category defining infrastructure company. And it's, it's rare to have a new category of infrastructure exists. We're here in the Chroma offices and like, you know, vector databases is a new category of infrastructure. Is it, is it, I mean, we can, we're in their office, so, you know, we can, we can debate that one later. That is one.Multimodality in AI-Powered Browsingswyx [00:10:16]: That's one of the industry debates.Paul [00:10:17]: I guess we go back to the LLMOS talk that Karpathy gave way long ago. And like the browser box was very clearly there and it seemed like the people who were building in this space also agreed that browsers are a core primitive of infrastructure for the LLMOS that's going to exist in the future. And nobody was building something there that I wanted to use. So I had to go build it myself.swyx [00:10:38]: Yeah. I mean, exactly that talk that, that honestly, that diagram, every box is a startup and there's the code box and then there's the. The browser box. I think at some point they will start clashing there. There's always the question of the, are you a point solution or are you the sort of all in one? And I think the point solutions tend to win quickly, but then the only ones have a very tight cohesive experience. Yeah. Let's talk about just the hard problems of browser base you have on your website, which is beautiful. Thank you. Was there an agency that you used for that? Yeah. Herb.paris.Paul [00:11:11]: They're amazing. Herb.paris. Yeah. It's H-E-R-V-E. I highly recommend for developers. Developer tools, founders to work with consumer agencies because they end up building beautiful things and the Parisians know how to build beautiful interfaces. So I got to give prep.swyx [00:11:24]: And chat apps, apparently are, they are very fast. Oh yeah. The Mistral chat. Yeah. Mistral. Yeah.Paul [00:11:31]: Late chat.swyx [00:11:31]: Late chat. And then your videos as well, it was professionally shot, right? The series A video. Yeah.Alessio [00:11:36]: Nico did the videos. He's amazing. Not the initial video that you shot at the new one. First one was Austin.Paul [00:11:41]: Another, another video pretty surprised. But yeah, I mean, like, I think when you think about how you talk about your company. You have to think about the way you present yourself. It's, you know, as a developer, you think you evaluate a company based on like the API reliability and the P 95, but a lot of developers say, is the website good? Is the message clear? Do I like trust this founder? I'm building my whole feature on. So I've tried to nail that as well as like the reliability of the infrastructure. You're right. It's very hard. And there's a lot of kind of foot guns that you run into when running headless browsers at scale. Right.Competing with Existing Headless Browser Solutionsswyx [00:12:10]: So let's pick one. You have eight features here. Seamless integration. Scalability. Fast or speed. Secure. Observable. Stealth. That's interesting. Extensible and developer first. What comes to your mind as like the top two, three hardest ones? Yeah.Running headless browsers at scalePaul [00:12:26]: I think just running headless browsers at scale is like the hardest one. And maybe can I nerd out for a second? Is that okay? I heard this is a technical audience, so I'll talk to the other nerds. Whoa. They were listening. Yeah. They're upset. They're ready. The AGI is angry. Okay. So. So how do you run a browser in the cloud? Let's start with that, right? So let's say you're using a popular browser automation framework like Puppeteer, Playwright, and Selenium. Maybe you've written a code, some code locally on your computer that opens up Google. It finds the search bar and then types in, you know, search for Latent Space and hits the search button. That script works great locally. You can see the little browser open up. You want to take that to production. You want to run the script in a cloud environment. So when your laptop is closed, your browser is doing something. The browser is doing something. Well, I, we use Amazon. You can see the little browser open up. You know, the first thing I'd reach for is probably like some sort of serverless infrastructure. I would probably try and deploy on a Lambda. But Chrome itself is too big to run on a Lambda. It's over 250 megabytes. So you can't easily start it on a Lambda. So you maybe have to use something like Lambda layers to squeeze it in there. Maybe use a different Chromium build that's lighter. And you get it on the Lambda. Great. It works. But it runs super slowly. It's because Lambdas are very like resource limited. They only run like with one vCPU. You can run one process at a time. Remember, Chromium is super beefy. It's barely running on my MacBook Air. I'm still downloading it from a pre-run. Yeah, from the test earlier, right? I'm joking. But it's big, you know? So like Lambda, it just won't work really well. Maybe it'll work, but you need something faster. Your users want something faster. Okay. Well, let's put it on a beefier instance. Let's get an EC2 server running. Let's throw Chromium on there. Great. Okay. I can, that works well with one user. But what if I want to run like 10 Chromium instances, one for each of my users? Okay. Well, I might need two EC2 instances. Maybe 10. All of a sudden, you have multiple EC2 instances. This sounds like a problem for Kubernetes and Docker, right? Now, all of a sudden, you're using ECS or EKS, the Kubernetes or container solutions by Amazon. You're spending up and down containers, and you're spending a whole engineer's time on kind of maintaining this stateful distributed system. Those are some of the worst systems to run because when it's a stateful distributed system, it means that you are bound by the connections to that thing. You have to keep the browser open while someone is working with it, right? That's just a painful architecture to run. And there's all this other little gotchas with Chromium, like Chromium, which is the open source version of Chrome, by the way. You have to install all these fonts. You want emojis working in your browsers because your vision model is looking for the emoji. You need to make sure you have the emoji fonts. You need to make sure you have all the right extensions configured, like, oh, do you want ad blocking? How do you configure that? How do you actually record all these browser sessions? Like it's a headless browser. You can't look at it. So you need to have some sort of observability. Maybe you're recording videos and storing those somewhere. It all kind of adds up to be this just giant monster piece of your project when all you wanted to do was run a lot of browsers in production for this little script to go to google.com and search. And when I see a complex distributed system, I see an opportunity to build a great infrastructure company. And we really abstract that away with Browserbase where our customers can use these existing frameworks, Playwright, Publisher, Selenium, or our own stagehand and connect to our browsers in a serverless-like way. And control them, and then just disconnect when they're done. And they don't have to think about the complex distributed system behind all of that. They just get a browser running anywhere, anytime. Really easy to connect to.swyx [00:15:55]: I'm sure you have questions. My standard question with anything, so essentially you're a serverless browser company, and there's been other serverless things that I'm familiar with in the past, serverless GPUs, serverless website hosting. That's where I come from with Netlify. One question is just like, you promised to spin up thousands of servers. You promised to spin up thousands of browsers in milliseconds. I feel like there's no real solution that does that yet. And I'm just kind of curious how. The only solution I know, which is to kind of keep a kind of warm pool of servers around, which is expensive, but maybe not so expensive because it's just CPUs. So I'm just like, you know. Yeah.Browsers as a Core Primitive in AI InfrastructurePaul [00:16:36]: You nailed it, right? I mean, how do you offer a serverless-like experience with something that is clearly not serverless, right? And the answer is, you need to be able to run... We run many browsers on single nodes. We use Kubernetes at browser base. So we have many pods that are being scheduled. We have to predictably schedule them up or down. Yes, thousands of browsers in milliseconds is the best case scenario. If you hit us with 10,000 requests, you may hit a slower cold start, right? So we've done a lot of work on predictive scaling and being able to kind of route stuff to different regions where we have multiple regions of browser base where we have different pools available. You can also pick the region you want to go to based on like lower latency, round trip, time latency. It's very important with these types of things. There's a lot of requests going over the wire. So for us, like having a VM like Firecracker powering everything under the hood allows us to be super nimble and spin things up or down really quickly with strong multi-tenancy. But in the end, this is like the complex infrastructural challenges that we have to kind of deal with at browser base. And we have a lot more stuff on our roadmap to allow customers to have more levers to pull to exchange, do you want really fast browser startup times or do you want really low costs? And if you're willing to be more flexible on that, we may be able to kind of like work better for your use cases.swyx [00:17:44]: Since you used Firecracker, shouldn't Fargate do that for you or did you have to go lower level than that? We had to go lower level than that.Paul [00:17:51]: I find this a lot with Fargate customers, which is alarming for Fargate. We used to be a giant Fargate customer. Actually, the first version of browser base was ECS and Fargate. And unfortunately, it's a great product. I think we were actually the largest Fargate customer in our region for a little while. No, what? Yeah, seriously. And unfortunately, it's a great product, but I think if you're an infrastructure company, you actually have to have a deeper level of control over these primitives. I think it's the same thing is true with databases. We've used other database providers and I think-swyx [00:18:21]: Yeah, serverless Postgres.Paul [00:18:23]: Shocker. When you're an infrastructure company, you're on the hook if any provider has an outage. And I can't tell my customers like, hey, we went down because so-and-so went down. That's not acceptable. So for us, we've really moved to bringing things internally. It's kind of opposite of what we preach. We tell our customers, don't build this in-house, but then we're like, we build a lot of stuff in-house. But I think it just really depends on what is in the critical path. We try and have deep ownership of that.Alessio [00:18:46]: On the distributed location side, how does that work for the web where you might get sort of different content in different locations, but the customer is expecting, you know, if you're in the US, I'm expecting the US version. But if you're spinning up my browser in France, I might get the French version. Yeah.Paul [00:19:02]: Yeah. That's a good question. Well, generally, like on the localization, there is a thing called locale in the browser. You can set like what your locale is. If you're like in the ENUS browser or not, but some things do IP, IP based routing. And in that case, you may want to have a proxy. Like let's say you're running something in the, in Europe, but you want to make sure you're showing up from the US. You may want to use one of our proxy features so you can turn on proxies to say like, make sure these connections always come from the United States, which is necessary too, because when you're browsing the web, you're coming from like a, you know, data center IP, and that can make things a lot harder to browse web. So we do have kind of like this proxy super network. Yeah. We have a proxy for you based on where you're going, so you can reliably automate the web. But if you get scheduled in Europe, that doesn't happen as much. We try and schedule you as close to, you know, your origin that you're trying to go to. But generally you have control over the regions you can put your browsers in. So you can specify West one or East one or Europe. We only have one region of Europe right now, actually. Yeah.Alessio [00:19:55]: What's harder, the browser or the proxy? I feel like to me, it feels like actually proxying reliably at scale. It's much harder than spending up browsers at scale. I'm curious. It's all hard.Paul [00:20:06]: It's layers of hard, right? Yeah. I think it's different levels of hard. I think the thing with the proxy infrastructure is that we work with many different web proxy providers and some are better than others. Some have good days, some have bad days. And our customers who've built browser infrastructure on their own, they have to go and deal with sketchy actors. Like first they figure out their own browser infrastructure and then they got to go buy a proxy. And it's like you can pay in Bitcoin and it just kind of feels a little sus, right? It's like you're buying drugs when you're trying to get a proxy online. We have like deep relationships with these counterparties. We're able to audit them and say, is this proxy being sourced ethically? Like it's not running on someone's TV somewhere. Is it free range? Yeah. Free range organic proxies, right? Right. We do a level of diligence. We're SOC 2. So we have to understand what is going on here. But then we're able to make sure that like we route around proxy providers not working. There's proxy providers who will just, the proxy will stop working all of a sudden. And then if you don't have redundant proxying on your own browsers, that's hard down for you or you may get some serious impacts there. With us, like we intelligently know, hey, this proxy is not working. Let's go to this one. And you can kind of build a network of multiple providers to really guarantee the best uptime for our customers. Yeah. So you don't own any proxies? We don't own any proxies. You're right. The team has been saying who wants to like take home a little proxy server, but not yet. We're not there yet. You know?swyx [00:21:25]: It's a very mature market. I don't think you should build that yourself. Like you should just be a super customer of them. Yeah. Scraping, I think, is the main use case for that. I guess. Well, that leads us into CAPTCHAs and also off, but let's talk about CAPTCHAs. You had a little spiel that you wanted to talk about CAPTCHA stuff.Challenges of Scaling Browser InfrastructurePaul [00:21:43]: Oh, yeah. I was just, I think a lot of people ask, if you're thinking about proxies, you're thinking about CAPTCHAs too. I think it's the same thing. You can go buy CAPTCHA solvers online, but it's the same buying experience. It's some sketchy website, you have to integrate it. It's not fun to buy these things and you can't really trust that the docs are bad. What Browserbase does is we integrate a bunch of different CAPTCHAs. We do some stuff in-house, but generally we just integrate with a bunch of known vendors and continually monitor and maintain these things and say, is this working or not? Can we route around it or not? These are CAPTCHA solvers. CAPTCHA solvers, yeah. Not CAPTCHA providers, CAPTCHA solvers. Yeah, sorry. CAPTCHA solvers. We really try and make sure all of that works for you. I think as a dev, if I'm buying infrastructure, I want it all to work all the time and it's important for us to provide that experience by making sure everything does work and monitoring it on our own. Yeah. Right now, the world of CAPTCHAs is tricky. I think AI agents in particular are very much ahead of the internet infrastructure. CAPTCHAs are designed to block all types of bots, but there are now good bots and bad bots. I think in the future, CAPTCHAs will be able to identify who a good bot is, hopefully via some sort of KYC. For us, we've been very lucky. We have very little to no known abuse of Browserbase because we really look into who we work with. And for certain types of CAPTCHA solving, we only allow them on certain types of plans because we want to make sure that we can know what people are doing, what their use cases are. And that's really allowed us to try and be an arbiter of good bots, which is our long term goal. I want to build great relationships with people like Cloudflare so we can agree, hey, here are these acceptable bots. We'll identify them for you and make sure we flag when they come to your website. This is a good bot, you know?Alessio [00:23:23]: I see. And Cloudflare said they want to do more of this. So they're going to set by default, if they think you're an AI bot, they're going to reject. I'm curious if you think this is something that is going to be at the browser level or I mean, the DNS level with Cloudflare seems more where it should belong. But I'm curious how you think about it.Paul [00:23:40]: I think the web's going to change. You know, I think that the Internet as we have it right now is going to change. And we all need to just accept that the cat is out of the bag. And instead of kind of like wishing the Internet was like it was in the 2000s, we can have free content line that wouldn't be scraped. It's just it's not going to happen. And instead, we should think about like, one, how can we change? How can we change the models of, you know, information being published online so people can adequately commercialize it? But two, how do we rebuild applications that expect that AI agents are going to log in on their behalf? Those are the things that are going to allow us to kind of like identify good and bad bots. And I think the team at Clerk has been doing a really good job with this on the authentication side. I actually think that auth is the biggest thing that will prevent agents from accessing stuff, not captchas. And I think there will be agent auth in the future. I don't know if it's going to happen from an individual company, but actually authentication providers that have a, you know, hidden login as agent feature, which will then you put in your email, you'll get a push notification, say like, hey, your browser-based agent wants to log into your Airbnb. You can approve that and then the agent can proceed. That really circumvents the need for captchas or logging in as you and sharing your password. I think agent auth is going to be one way we identify good bots going forward. And I think a lot of this captcha solving stuff is really short-term problems as the internet kind of reorients itself around how it's going to work with agents browsing the web, just like people do. Yeah.Managing Distributed Browser Locations and Proxiesswyx [00:24:59]: Stitch recently was on Hacker News for talking about agent experience, AX, which is a thing that Netlify is also trying to clone and coin and talk about. And we've talked about this on our previous episodes before in a sense that I actually think that's like maybe the only part of the tech stack that needs to be kind of reinvented for agents. Everything else can stay the same, CLIs, APIs, whatever. But auth, yeah, we need agent auth. And it's mostly like short-lived, like it should not, it should be a distinct, identity from the human, but paired. I almost think like in the same way that every social network should have your main profile and then your alt accounts or your Finsta, it's almost like, you know, every, every human token should be paired with the agent token and the agent token can go and do stuff on behalf of the human token, but not be presumed to be the human. Yeah.Paul [00:25:48]: It's like, it's, it's actually very similar to OAuth is what I'm thinking. And, you know, Thread from Stitch is an investor, Colin from Clerk, Octaventures, all investors in browser-based because like, I hope they solve this because they'll make browser-based submission more possible. So we don't have to overcome all these hurdles, but I think it will be an OAuth-like flow where an agent will ask to log in as you, you'll approve the scopes. Like it can book an apartment on Airbnb, but it can't like message anybody. And then, you know, the agent will have some sort of like role-based access control within an application. Yeah. I'm excited for that.swyx [00:26:16]: The tricky part is just, there's one, one layer of delegation here, which is like, you're authoring my user's user or something like that. I don't know if that's tricky or not. Does that make sense? Yeah.Paul [00:26:25]: You know, actually at Twilio, I worked on the login identity and access. Management teams, right? So like I built Twilio's login page.swyx [00:26:31]: You were an intern on that team and then you became the lead in two years? Yeah.Paul [00:26:34]: Yeah. I started as an intern in 2016 and then I was the tech lead of that team. How? That's not normal. I didn't have a life. He's not normal. Look at this guy. I didn't have a girlfriend. I just loved my job. I don't know. I applied to 500 internships for my first job and I got rejected from every single one of them except for Twilio and then eventually Amazon. And they took a shot on me and like, I was getting paid money to write code, which was my dream. Yeah. Yeah. I'm very lucky that like this coding thing worked out because I was going to be doing it regardless. And yeah, I was able to kind of spend a lot of time on a team that was growing at a company that was growing. So it informed a lot of this stuff here. I think these are problems that have been solved with like the SAML protocol with SSO. I think it's a really interesting stuff with like WebAuthn, like these different types of authentication, like schemes that you can use to authenticate people. The tooling is all there. It just needs to be tweaked a little bit to work for agents. And I think the fact that there are companies that are already. Providing authentication as a service really sets it up. Well, the thing that's hard is like reinventing the internet for agents. We don't want to rebuild the internet. That's an impossible task. And I think people often say like, well, we'll have this second layer of APIs built for agents. I'm like, we will for the top use cases, but instead of we can just tweak the internet as is, which is on the authentication side, I think we're going to be the dumb ones going forward. Unfortunately, I think AI is going to be able to do a lot of the tasks that we do online, which means that it will be able to go to websites, click buttons on our behalf and log in on our behalf too. So with this kind of like web agent future happening, I think with some small structural changes, like you said, it feels like it could all slot in really nicely with the existing internet.Handling CAPTCHAs and Agent Authenticationswyx [00:28:08]: There's one more thing, which is the, your live view iframe, which lets you take, take control. Yeah. Obviously very key for operator now, but like, was, is there anything interesting technically there or that the people like, well, people always want this.Paul [00:28:21]: It was really hard to build, you know, like, so, okay. Headless browsers, you don't see them, right. They're running. They're running in a cloud somewhere. You can't like look at them. And I just want to really make, it's a weird name. I wish we came up with a better name for this thing, but you can't see them. Right. But customers don't trust AI agents, right. At least the first pass. So what we do with our live view is that, you know, when you use browser base, you can actually embed a live view of the browser running in the cloud for your customer to see it working. And that's what the first reason is the build trust, like, okay, so I have this script. That's going to go automate a website. I can embed it into my web application via an iframe and my customer can watch. I think. And then we added two way communication. So now not only can you watch the browser kind of being operated by AI, if you want to pause and actually click around type within this iframe that's controlling a browser, that's also possible. And this is all thanks to some of the lower level protocol, which is called the Chrome DevTools protocol. It has a API called start screencast, and you can also send mouse clicks and button clicks to a remote browser. And this is all embeddable within iframes. You have a browser within a browser, yo. And then you simulate the screen, the click on the other side. Exactly. And this is really nice often for, like, let's say, a capture that can't be solved. You saw this with Operator, you know, Operator actually uses a different approach. They use VNC. So, you know, you're able to see, like, you're seeing the whole window here. What we're doing is something a little lower level with the Chrome DevTools protocol. It's just PNGs being streamed over the wire. But the same thing is true, right? Like, hey, I'm running a window. Pause. Can you do something in this window? Human. Okay, great. Resume. Like sometimes 2FA tokens. Like if you get that text message, you might need a person to type that in. Web agents need human-in-the-loop type workflows still. You still need a person to interact with the browser. And building a UI to proxy that is kind of hard. You may as well just show them the whole browser and say, hey, can you finish this up for me? And then let the AI proceed on afterwards. Is there a future where I stream my current desktop to browser base? I don't think so. I think we're very much cloud infrastructure. Yeah. You know, but I think a lot of the stuff we're doing, we do want to, like, build tools. Like, you know, we'll talk about the stage and, you know, web agent framework in a second. But, like, there's a case where a lot of people are going desktop first for, you know, consumer use. And I think cloud is doing a lot of this, where I expect to see, you know, MCPs really oriented around the cloud desktop app for a reason, right? Like, I think a lot of these tools are going to run on your computer because it makes... I think it's breaking out. People are putting it on a server. Oh, really? Okay. Well, sweet. We'll see. We'll see that. I was surprised, though, wasn't I? I think that the browser company, too, with Dia Browser, it runs on your machine. You know, it's going to be...swyx [00:30:50]: What is it?Paul [00:30:51]: So, Dia Browser, as far as I understand... I used to use Arc. Yeah. I haven't used Arc. But I'm a big fan of the browser company. I think they're doing a lot of cool stuff in consumer. As far as I understand, it's a browser where you have a sidebar where you can, like, chat with it and it can control the local browser on your machine. So, if you imagine, like, what a consumer web agent is, which it lives alongside your browser, I think Google Chrome has Project Marina, I think. I almost call it Project Marinara for some reason. I don't know why. It's...swyx [00:31:17]: No, I think it's someone really likes the Waterworld. Oh, I see. The classic Kevin Costner. Yeah.Paul [00:31:22]: Okay. Project Marinara is a similar thing to the Dia Browser, in my mind, as far as I understand it. You have a browser that has an AI interface that will take over your mouse and keyboard and control the browser for you. Great for consumer use cases. But if you're building applications that rely on a browser and it's more part of a greater, like, AI app experience, you probably need something that's more like infrastructure, not a consumer app.swyx [00:31:44]: Just because I have explored a little bit in this area, do people want branching? So, I have the state. Of whatever my browser's in. And then I want, like, 100 clones of this state. Do people do that? Or...Paul [00:31:56]: People don't do it currently. Yeah. But it's definitely something we're thinking about. I think the idea of forking a browser is really cool. Technically, kind of hard. We're starting to see this in code execution, where people are, like, forking some, like, code execution, like, processes or forking some tool calls or branching tool calls. Haven't seen it at the browser level yet. But it makes sense. Like, if an AI agent is, like, using a website and it's not sure what path it wants to take to crawl this website. To find the information it's looking for. It would make sense for it to explore both paths in parallel. And that'd be a very, like... A road not taken. Yeah. And hopefully find the right answer. And then say, okay, this was actually the right one. And memorize that. And go there in the future. On the roadmap. For sure. Don't make my roadmap, please. You know?Alessio [00:32:37]: How do you actually do that? Yeah. How do you fork? I feel like the browser is so stateful for so many things.swyx [00:32:42]: Serialize the state. Restore the state. I don't know.Paul [00:32:44]: So, it's one of the reasons why we haven't done it yet. It's hard. You know? Like, to truly fork, it's actually quite difficult. The naive way is to open the same page in a new tab and then, like, hope that it's at the same thing. But if you have a form halfway filled, you may have to, like, take the whole, you know, container. Pause it. All the memory. Duplicate it. Restart it from there. It could be very slow. So, we haven't found a thing. Like, the easy thing to fork is just, like, copy the page object. You know? But I think there needs to be something a little bit more robust there. Yeah.swyx [00:33:12]: So, MorphLabs has this infinite branch thing. Like, wrote a custom fork of Linux or something that let them save the system state and clone it. MorphLabs, hit me up. I'll be a customer. Yeah. That's the only. I think that's the only way to do it. Yeah. Like, unless Chrome has some special API for you. Yeah.Paul [00:33:29]: There's probably something we'll reverse engineer one day. I don't know. Yeah.Alessio [00:33:32]: Let's talk about StageHand, the AI web browsing framework. You have three core components, Observe, Extract, and Act. Pretty clean landing page. What was the idea behind making a framework? Yeah.Stagehand: AI web browsing frameworkPaul [00:33:43]: So, there's three frameworks that are very popular or already exist, right? Puppeteer, Playwright, Selenium. Those are for building hard-coded scripts to control websites. And as soon as I started to play with LLMs plus browsing, I caught myself, you know, code-genning Playwright code to control a website. I would, like, take the DOM. I'd pass it to an LLM. I'd say, can you generate the Playwright code to click the appropriate button here? And it would do that. And I was like, this really should be part of the frameworks themselves. And I became really obsessed with SDKs that take natural language as part of, like, the API input. And that's what StageHand is. StageHand exposes three APIs, and it's a super set of Playwright. So, if you go to a page, you may want to take an action, click on the button, fill in the form, etc. That's what the act command is for. You may want to extract some data. This one takes a natural language, like, extract the winner of the Super Bowl from this page. You can give it a Zod schema, so it returns a structured output. And then maybe you're building an API. You can do an agent loop, and you want to kind of see what actions are possible on this page before taking one. You can do observe. So, you can observe the actions on the page, and it will generate a list of actions. You can guide it, like, give me actions on this page related to buying an item. And you can, like, buy it now, add to cart, view shipping options, and pass that to an LLM, an agent loop, to say, what's the appropriate action given this high-level goal? So, StageHand isn't a web agent. It's a framework for building web agents. And we think that agent loops are actually pretty close to the application layer because every application probably has different goals or different ways it wants to take steps. I don't think I've seen a generic. Maybe you guys are the experts here. I haven't seen, like, a really good AI agent framework here. Everyone kind of has their own special sauce, right? I see a lot of developers building their own agent loops, and they're using tools. And I view StageHand as the browser tool. So, we expose act, extract, observe. Your agent can call these tools. And from that, you don't have to worry about it. You don't have to worry about generating playwright code performantly. You don't have to worry about running it. You can kind of just integrate these three tool calls into your agent loop and reliably automate the web.swyx [00:35:48]: A special shout-out to Anirudh, who I met at your dinner, who I think listens to the pod. Yeah. Hey, Anirudh.Paul [00:35:54]: Anirudh's a man. He's a StageHand guy.swyx [00:35:56]: I mean, the interesting thing about each of these APIs is they're kind of each startup. Like, specifically extract, you know, Firecrawler is extract. There's, like, Expand AI. There's a whole bunch of, like, extract companies. They just focus on extract. I'm curious. Like, I feel like you guys are going to collide at some point. Like, right now, it's friendly. Everyone's in a blue ocean. At some point, it's going to be valuable enough that there's some turf battle here. I don't think you have a dog in a fight. I think you can mock extract to use an external service if they're better at it than you. But it's just an observation that, like, in the same way that I see each option, each checkbox in the side of custom GBTs becoming a startup or each box in the Karpathy chart being a startup. Like, this is also becoming a thing. Yeah.Paul [00:36:41]: I mean, like, so the way StageHand works is that it's MIT-licensed, completely open source. You bring your own API key to your LLM of choice. You could choose your LLM. We don't make any money off of the extract or really. We only really make money if you choose to run it with our browser. You don't have to. You can actually use your own browser, a local browser. You know, StageHand is completely open source for that reason. And, yeah, like, I think if you're building really complex web scraping workflows, I don't know if StageHand is the tool for you. I think it's really more if you're building an AI agent that needs a few general tools or if it's doing a lot of, like, web automation-intensive work. But if you're building a scraping company, StageHand is not your thing. You probably want something that's going to, like, get HTML content, you know, convert that to Markdown, query it. That's not what StageHand does. StageHand is more about reliability. I think we focus a lot on reliability and less so on cost optimization and speed at this point.swyx [00:37:33]: I actually feel like StageHand, so the way that StageHand works, it's like, you know, page.act, click on the quick start. Yeah. It's kind of the integration test for the code that you would have to write anyway, like the Puppeteer code that you have to write anyway. And when the page structure changes, because it always does, then this is still the test. This is still the test that I would have to write. Yeah. So it's kind of like a testing framework that doesn't need implementation detail.Paul [00:37:56]: Well, yeah. I mean, Puppeteer, Playwright, and Slenderman were all designed as testing frameworks, right? Yeah. And now people are, like, hacking them together to automate the web. I would say, and, like, maybe this is, like, me being too specific. But, like, when I write tests, if the page structure changes. Without me knowing, I want that test to fail. So I don't know if, like, AI, like, regenerating that. Like, people are using StageHand for testing. But it's more for, like, usability testing, not, like, testing of, like, does the front end, like, has it changed or not. Okay. But generally where we've seen people, like, really, like, take off is, like, if they're using, you know, something. If they want to build a feature in their application that's kind of like Operator or Deep Research, they're using StageHand to kind of power that tool calling in their own agent loop. Okay. Cool.swyx [00:38:37]: So let's go into Operator, the first big agent launch of the year from OpenAI. Seems like they have a whole bunch scheduled. You were on break and your phone blew up. What's your just general view of computer use agents is what they're calling it. The overall category before we go into Open Operator, just the overall promise of Operator. I will observe that I tried it once. It was okay. And I never tried it again.OpenAI's Operator and computer use agentsPaul [00:38:58]: That tracks with my experience, too. Like, I'm a huge fan of the OpenAI team. Like, I think that I do not view Operator as the company. I'm not a company killer for browser base at all. I think it actually shows people what's possible. I think, like, computer use models make a lot of sense. And I'm actually most excited about computer use models is, like, their ability to, like, really take screenshots and reasoning and output steps. I think that using mouse click or mouse coordinates, I've seen that proved to be less reliable than I would like. And I just wonder if that's the right form factor. What we've done with our framework is anchor it to the DOM itself, anchor it to the actual item. So, like, if it's clicking on something, it's clicking on that thing, you know? Like, it's more accurate. No matter where it is. Yeah, exactly. Because it really ties in nicely. And it can handle, like, the whole viewport in one go, whereas, like, Operator can only handle what it sees. Can you hover? Is hovering a thing that you can do? I don't know if we expose it as a tool directly, but I'm sure there's, like, an API for hovering. Like, move mouse to this position. Yeah, yeah, yeah. I think you can trigger hover, like, via, like, the JavaScript on the DOM itself. But, no, I think, like, when we saw computer use, everyone's eyes lit up because they realized, like, wow, like, AI is going to actually automate work for people. And I think seeing that kind of happen from both of the labs, and I'm sure we're going to see more labs launch computer use models, I'm excited to see all the stuff that people build with it. I think that I'd love to see computer use power, like, controlling a browser on browser base. And I think, like, Open Operator, which was, like, our open source version of OpenAI's Operator, was our first take on, like, how can we integrate these models into browser base? And we handle the infrastructure and let the labs do the models. I don't have a sense that Operator will be released as an API. I don't know. Maybe it will. I'm curious to see how well that works because I think it's going to be really hard for a company like OpenAI to do things like support CAPTCHA solving or, like, have proxies. Like, I think it's hard for them structurally. Imagine this New York Times headline, OpenAI CAPTCHA solving. Like, that would be a pretty bad headline, this New York Times headline. Browser base solves CAPTCHAs. No one cares. No one cares. And, like, our investors are bored. Like, we're all okay with this, you know? We're building this company knowing that the CAPTCHA solving is short-lived until we figure out how to authenticate good bots. I think it's really hard for a company like OpenAI, who has this brand that's so, so good, to balance with, like, the icky parts of web automation, which it can be kind of complex to solve. I'm sure OpenAI knows who to call whenever they need you. Yeah, right. I'm sure they'll have a great partnership.Alessio [00:41:23]: And is Open Operator just, like, a marketing thing for you? Like, how do you think about resource allocation? So, you can spin this up very quickly. And now there's all this, like, open deep research, just open all these things that people are building. We started it, you know. You're the original Open. We're the original Open operator, you know? Is it just, hey, look, this is a demo, but, like, we'll help you build out an actual product for yourself? Like, are you interested in going more of a product route? That's kind of the OpenAI way, right? They started as a model provider and then…Paul [00:41:53]: Yeah, we're not interested in going the product route yet. I view Open Operator as a model provider. It's a reference project, you know? Let's show people how to build these things using the infrastructure and models that are out there. And that's what it is. It's, like, Open Operator is very simple. It's an agent loop. It says, like, take a high-level goal, break it down into steps, use tool calling to accomplish those steps. It takes screenshots and feeds those screenshots into an LLM with the step to generate the right action. It uses stagehand under the hood to actually execute this action. It doesn't use a computer use model. And it, like, has a nice interface using the live view that we talked about, the iframe, to embed that into an application. So I felt like people on launch day wanted to figure out how to build their own version of this. And we turned that around really quickly to show them. And I hope we do that with other things like deep research. We don't have a deep research launch yet. I think David from AOMNI actually has an amazing open deep research that he launched. It has, like, 10K GitHub stars now. So he's crushing that. But I think if people want to build these features natively into their application, they need good reference projects. And I think Open Operator is a good example of that.swyx [00:42:52]: I don't know. Actually, I'm actually pretty bullish on API-driven operator. Because that's the only way that you can sort of, like, once it's reliable enough, obviously. And now we're nowhere near. But, like, give it five years. It'll happen, you know. And then you can sort of spin this up and browsers are working in the background and you don't necessarily have to know. And it just is booking restaurants for you, whatever. I can definitely see that future happening. I had this on the landing page here. This might be a slightly out of order. But, you know, you have, like, sort of three use cases for browser base. Open Operator. Or this is the operator sort of use case. It's kind of like the workflow automation use case. And it completes with UiPath in the sort of RPA category. Would you agree with that? Yeah, I would agree with that. And then there's Agents we talked about already. And web scraping, which I imagine would be the bulk of your workload right now, right?Paul [00:43:40]: No, not at all. I'd say actually, like, the majority is browser automation. We're kind of expensive for web scraping. Like, I think that if you're building a web scraping product, if you need to do occasional web scraping or you have to do web scraping that works every single time, you want to use browser automation. Yeah. You want to use browser-based. But if you're building web scraping workflows, what you should do is have a waterfall. You should have the first request is a curl to the website. See if you can get it without even using a browser. And then the second request may be, like, a scraping-specific API. There's, like, a thousand scraping APIs out there that you can use to try and get data. Scraping B. Scraping B is a great example, right? Yeah. And then, like, if those two don't work, bring out the heavy hitter. Like, browser-based will 100% work, right? It will load the page in a real browser, hydrate it. I see.swyx [00:44:21]: Because a lot of people don't render to JS.swyx [00:44:25]: Yeah, exactly.Paul [00:44:26]: So, I mean, the three big use cases, right? Like, you know, automation, web data collection, and then, you know, if you're building anything agentic that needs, like, a browser tool, you want to use browser-based.Alessio [00:44:35]: Is there any use case that, like, you were super surprised by that people might not even think about? Oh, yeah. Or is it, yeah, anything that you can share? The long tail is crazy. Yeah.Surprising use cases of BrowserbasePaul [00:44:44]: One of the case studies on our website that I think is the most interesting is this company called Benny. So, the way that it works is if you're on food stamps in the United States, you can actually get rebates if you buy certain things. Yeah. You buy some vegetables. You submit your receipt to the government. They'll give you a little rebate back. Say, hey, thanks for buying vegetables. It's good for you. That process of submitting that receipt is very painful. And the way Benny works is you use their app to take a photo of your receipt, and then Benny will go submit that receipt for you and then deposit the money into your account. That's actually using no AI at all. It's all, like, hard-coded scripts. They maintain the scripts. They've been doing a great job. And they build this amazing consumer app. But it's an example of, like, all these, like, tedious workflows that people have to do to kind of go about their business. And they're doing it for the sake of their day-to-day lives. And I had never known about, like, food stamp rebates or the complex forms you have to do to fill them. But the world is powered by millions and millions of tedious forms, visas. You know, Emirate Lighthouse is a customer, right? You know, they do the O1 visa. Millions and millions of forms are taking away humans' time. And I hope that Browserbase can help power software that automates away the web forms that we don't need anymore. Yeah.swyx [00:45:49]: I mean, I'm very supportive of that. I mean, forms. I do think, like, government itself is a big part of it. I think the government itself should embrace AI more to do more sort of human-friendly form filling. Mm-hmm. But I'm not optimistic. I'm not holding my breath. Yeah. We'll see. Okay. I think I'm about to zoom out. I have a little brief thing on computer use, and then we can talk about founder stuff, which is, I tend to think of developer tooling markets in impossible triangles, where everyone starts in a niche, and then they start to branch out. So I already hinted at a little bit of this, right? We mentioned more. We mentioned E2B. We mentioned Firecrawl. And then there's Browserbase. So there's, like, all this stuff of, like, have serverless virtual computer that you give to an agent and let them do stuff with it. And there's various ways of connecting it to the internet. You can just connect to a search API, like SERP API, whatever other, like, EXA is another one. That's what you're searching. You can also have a JSON markdown extractor, which is Firecrawl. Or you can have a virtual browser like Browserbase, or you can have a virtual machine like Morph. And then there's also maybe, like, a virtual sort of code environment, like Code Interpreter. So, like, there's just, like, a bunch of different ways to tackle the problem of give a computer to an agent. And I'm just kind of wondering if you see, like, everyone's just, like, happily coexisting in their respective niches. And as a developer, I just go and pick, like, a shopping basket of one of each. Or do you think that you eventually, people will collide?Future of browser automation and market competitionPaul [00:47:18]: I think that currently it's not a zero-sum market. Like, I think we're talking about... I think we're talking about all of knowledge work that people do that can be automated online. All of these, like, trillions of hours that happen online where people are working. And I think that there's so much software to be built that, like, I tend not to think about how these companies will collide. I just try to solve the problem as best as I can and make this specific piece of infrastructure, which I think is an important primitive, the best I possibly can. And yeah. I think there's players that are actually going to like it. I think there's players that are going to launch, like, over-the-top, you know, platforms, like agent platforms that have all these tools built in, right? Like, who's building the rippling for agent tools that has the search tool, the browser tool, the operating system tool, right? There are some. There are some. There are some, right? And I think in the end, what I have seen as my time as a developer, and I look at all the favorite tools that I have, is that, like, for tools and primitives with sufficient levels of complexity, you need to have a solution that's really bespoke to that primitive, you know? And I am sufficiently convinced that the browser is complex enough to deserve a primitive. Obviously, I have to. I'm the founder of BrowserBase, right? I'm talking my book. But, like, I think maybe I can give you one spicy take against, like, maybe just whole OS running. I think that when I look at computer use when it first came out, I saw that the majority of use cases for computer use were controlling a browser. And do we really need to run an entire operating system just to control a browser? I don't think so. I don't think that's necessary. You know, BrowserBase can run browsers for way cheaper than you can if you're running a full-fledged OS with a GUI, you know, operating system. And I think that's just an advantage of the browser. It is, like, browsers are little OSs, and you can run them very efficiently if you orchestrate it well. And I think that allows us to offer 90% of the, you know, functionality in the platform needed at 10% of the cost of running a full OS. Yeah.Open Operator: Browserbase's Open-Source Alternativeswyx [00:49:16]: I definitely see the logic in that. There's a Mark Andreessen quote. I don't know if you know this one. Where he basically observed that the browser is turning the operating system into a poorly debugged set of device drivers, because most of the apps are moved from the OS to the browser. So you can just run browsers.Paul [00:49:31]: There's a place for OSs, too. Like, I think that there are some applications that only run on Windows operating systems. And Eric from pig.dev in this upcoming YC batch, or last YC batch, like, he's building all run tons of Windows operating systems for you to control with your agent. And like, there's some legacy EHR systems that only run on Internet-controlled systems. Yeah.Paul [00:49:54]: I think that's it. I think, like, there are use cases for specific operating systems for specific legacy software. And like, I'm excited to see what he does with that. I just wanted to give a shout out to the pig.dev website.swyx [00:50:06]: The pigs jump when you click on them. Yeah. That's great.Paul [00:50:08]: Eric, he's the former co-founder of banana.dev, too.swyx [00:50:11]: Oh, that Eric. Yeah. That Eric. Okay. Well, he abandoned bananas for pigs. I hope he doesn't start going around with pigs now.Alessio [00:50:18]: Like he was going around with bananas. A little toy pig. Yeah. Yeah. I love that. What else are we missing? I think we covered a lot of, like, the browser-based product history, but. What do you wish people asked you? Yeah.Paul [00:50:29]: I wish people asked me more about, like, what will the future of software look like? Because I think that's really where I've spent a lot of time about why do browser-based. Like, for me, starting a company is like a means of last resort. Like, you shouldn't start a company unless you absolutely have to. And I remain convinced that the future of software is software that you're going to click a button and it's going to do stuff on your behalf. Right now, software. You click a button and it maybe, like, calls it back an API and, like, computes some numbers. It, like, modifies some text, whatever. But the future of software is software using software. So, I may log into my accounting website for my business, click a button, and it's going to go load up my Gmail, search my emails, find the thing, upload the receipt, and then comment it for me. Right? And it may use it using APIs, maybe a browser. I don't know. I think it's a little bit of both. But that's completely different from how we've built software so far. And that's. I think that future of software has different infrastructure requirements. It's going to require different UIs. It's going to require different pieces of infrastructure. I think the browser infrastructure is one piece that fits into that, along with all the other categories you mentioned. So, I think that it's going to require developers to think differently about how they've built software for, you know

ABOUT MELODY MECKFESSELMelody Meckfessel is the Chief Technology Officer (CTO) at Jasper.ai, the world's leading AI marketing platform. In her role, Melody shapes the technical vision of the company, oversees product delivery, and spearheads AI research to develop new capabilities that accelerate business outcomes for enterprise marketers.Before joining Jasper.ai, Melody co-founded and served as CEO of Observable, a data visualization platform that empowers teams to understand their businesses through data. She also spent over a decade at Google as Vice President of Engineering, where she led core infrastructure, Search, and DevOps teams for Google and Google Cloud Platform, impacting millions of users worldwide.Melody is recognized for her hands-on approach to engineering leadership and her expertise in building large-scale distributed systems. Her work is crucial in solving complex problems at scale for enterprise companies. She is passionate about defining the future of work with AI, where humans come first.This episode is brought to you by Clipboard HealthClipboard Health is looking for the next generation of exceptional software engineering leaders, not just managers. They're a profitable unicorn, backed by top-tier investors, and they take the craft of engineering management seriously.Clipboard Health matches highly qualified healthcare workers with nearby facilities to fulfill millions of shifts a year - revolutionizing healthcare staffing with a fast, flexible, and user-friendly platform.Learn more & browse their open roles at clipboardhealth.com/engineeringSHOW NOTES:Melody's perspective on the tech industry's rapid rate of change right now (2:53)Critical questions to guide investment decisions on “what to build & how“ in a rapidly evolving market (5:57)Strategies for navigating rapid change internally within eng teams (10:07)What it means to be an AI-first engineering organization (12:30)Changes in goals, metrics, and processes to shape your org and guide you through rapid change (15:33)Developing agile communication processes (18:39)Navigating ambiguity as a learned skill - practical ways to strengthen your ability to navigate uncertainty (20:09)Implementing a framework of curiosity & openness within eng teams (22:40)Why great things can't be planned (26:21)Becoming dynamic and resilient - how to thrive amid uncertainty and constant industry shifts (28:57)How to shift from prescriptive to inspirational - using illustrated inspiration to empower teams (32:00)Breaking through self-imposed limitations - understanding where eng leaders may limit themselves (33:48)Melody's perspective on fostering a culture of creativity within eng teams (35:06)Rapid fire questions (37:13)LINKS AND RESOURCESThe Qualified Sales Leader: Proven Lessons from a Five Time CRO - John McMahon shares valuable lessons for sales leaders and sales reps selling enterprise software solutions. In a conversational and easy to read narrative style, this must-read book provides learnings on how sales leaders can help their reps sell more for higher average deal sizes to executive level buyers.This episode wouldn't have been possible without the help of our incredible production team:Patrick Gallagher - Producer & Co-HostJerry Li - Co-HostNoah Olberding - Associate Producer, Audio & Video Editor https://www.linkedin.com/in/noah-olberding/Dan Overheim - Audio Engineer, Dan's also an avid 3D printer - https://www.bnd3d.com/Ellie Coggins Angus - Copywriter, Check out her other work at https://elliecoggins.com/about/

For episode 196 of De Facto Leaders, I share a Q & A session where I talk through how to write language therapy goals that are both trackable and functional.This is just one of many Q & A sessions I'm planning on sharing where I talk through how to find the balance between focusing on external behaviors that allow us to document progress and internal cognitive processes.I also talk about when to focus on “observable” language skills vs. strategy-based goals; especially when addressing both language and executive functioning skills. Throughout the session, you'll hear examples related to working on skills like syntax, semantic feature study, vocabulary, and cognitive processes that support language comprehension. This Q & A session was done in the member's group for Language Therapy Advance Foundations, my program that helps SLPs build a system for language therapy. You can learn more about Language Therapy Advance Foundations here : https://drkarenspeech.com/languagetherapy We're thrilled to be sponsored by IXL. IXL's comprehensive teaching and learning platform for math, language arts, science, and social studies is accelerating achievement in 95 of the top 100 U.S. school districts. Loved by teachers and backed by independent research from Johns Hopkins University, IXL can help you do the following and more:Simplify and streamline technologySave teachers' timeReliably meet Tier 1 standardsImprove student performance on state assessments

The Observer and Trapper swap stories. On the origins of Christmas Traditions. You'll never bake anything good again. The mouldering remains of last year.The First Supplemental Frequency from Observable Radio, a found footage podcast from Cameron Suey. Phil van Hest, and Purpurina.Content Warnings: Explicitly Depicted Violence/SquelchingWritten by Purpurina, Cameron Suey, and Wendy HectorProduced by Cameron Suey, Phil van Hest, and PurpurinaEdited by Cameron SueyThe EnsemblePhil van HestJason SmithKatie SkovholtPurpurinaArt by Karrin FletcherPsychology Consultant - Elisa Leal, Psy.D (CA PSY28330)Our Theme Music is: The Backrooms - MyuuAdditional Music provided by Tim Kulig, Katie Skovholt, and the artists at Epidemic SoundAncient Lands - FlouwDiabolic Gaze - Luella GrenCold Nights - KikoruStrangled by Piano Strings Part 1 - Ludvig MoulinBlue Spaces - Oakwood StationAbyssal Hibernation (Delta Drone L144Hz R147Hz) - OokeanGoth Christmas - parA Cradle Song - Mary RiddleSilent Night (Ambient Version) (Instrumental Version) - Ingrid WittAdditional Sound Design by PurpurinaSFX provided by Epidemic Sound and the artists at Freesound.orgAdditional SFX and Music covered under the following licenses:creativecommons.org/licenses/by/3.0/creativecommons.org/licenses/by/4.0/Special Thanks to Cathleen, Jon, Tid, Russ, Kalasin, Rick, Brianna, Zach and all our patrons and listeners.Thank you for listening, and stay tuned.

At Redeemer Presbyterian Church in Charleston, SC, our senior pastor Rev. Craig Bailey preached for the second Sunday of Advent.

If we need to fully understand space we need more time - lots of it. Beyond the hubblesphere things gets lumpy. drkarl.com

Alan is joined by Lou Elizondo, former military intelligence officer and author of Imminent: Inside the Pentagon's Hunt for UFOs. Lou shares his journey from joining JROTC to his work in the Pentagon's classified UFO program. He describes Unidentified Aerial Phenomena (UAP) with extraordinary capabilities like hypersonic speeds and movement without visible propulsion, raising national security concerns. Lou explains that UAP sightings date back to the 1950s, and the government once stigmatized discussions to avoid panic. Now, bipartisan efforts push for transparency, supported by emerging legislation and international cooperation on UAP research.     Guest Bio Lou Elizondo is a former military intelligence officer who served in various classified roles, including a key position in the Pentagon's UFO program. After his resignation in 2017, Lou became an advocate for transparency about Unidentified Aerial Phenomena (UAPs), revealing shocking insights into advanced aerospace technologies observed by military pilots. His New York Times bestselling book, Imminent: Inside the Pentagon's Hunt for UFOs, uncovers the hidden world of UAP investigations and challenges our understanding of reality. Lou's work has sparked global conversations about science, security, and the future of human knowledge.     Show Highlights (1:29) What led to Luis' career in military and intelligence services (5:26) What remote sensing is (11:03) How Luis' became in involved with UAPs from a military perspective (25:09) How Luis' deals with the lack of acceptance of the data (29:42) What led Louis to resign from the Pentagon (34:04) Observable traits of UAPs based on famous filmed cases (40:48) Why the government's attitude toward public transparency is changing (46:03) Next steps for people as UAPs are more openly discussed (52:56) The importance of keeping an open mind moving forward     Links Referenced Imminent: Inside the Pentagon's Hunt for UFO's https://www.amazon.com/Imminent-Pentagons-Investigating-UAPs_Featured-Experience/dp/0063235560

10.6.24 - Acts 11:19-30, 12:25-13:3 - “Observable Signs of God's Grace” - Alex Gailey

Ben Scheirman is back for part 2 of our interview on SwiftUI Migration. In this episode we focus on navigation, data handling and Swift packages.GuestBen Scheirman | Ben is an experienced software engineer from Houston, TX. Currently focused on Swift, iOS, Ruby, and Rust.Ben Scheirman (@bens@mastodon.xyz) - Mastodonsubdigital (Ben Scheirman)NSScreencast: Bite-sized Screencasts for iOS DevelopmentCombine SwiftAnnouncementsNeed help with your projects this year? BrightDigit has openings.Join Bushel BetaJoin our Brand New Patreon Page!LinksEpisode #288: Modern UIKit: Stack Navigation, Part 2pointfreeco/swift-perception: Observable tools, backported.brightdigit/Sublimation: Enable automatic discovery of your local development server on the fly. Turn your Server-Side Swift app from a mysterious vapor to a tangible solid server.krzysztofzablocki/LifetimeTracker: Find retain cycles / memory leaks sooner.siteline/swiftui-introspect: Introspect underlying UIKit/AppKit components from SwiftUIPresenting Coordinators - Soroush Khanlou on VimeoRelated EpisodesThe Great SwiftUI Migration - Part 1 with Ben ScheirmanSwiftUI Field Guide with Chris EidhofSOTU 2024 with Peter WithamSwiftUI Tips and Tricks with Craig ClaytonSwiftly Tooling with Pol Piella AbadiaIt Depends with Brandon WilliamsMy Taylor Deep Dish Swift Heroes World TourMobile System Design with Tjeerd in 't VeenThe Composable Architecture with Zev EisenbergBehind the Scenes of SwiftUI with Aviel GrossWWDC 2022 - SwiftUI and UIKit with Evan StoneSocial MediaEmailleo@brightdigit.comGitHub - @brightdigitTwitter BrightDigit - @brightdigitLeo - @leogdionLinkedInBrightDigitLeoPatreon - brightdigitCreditsMusic from https://filmmusic.io"Blippy Trance" by Kevin MacLeod (https://incompetech.com)License: CC BY (http://creativecommons.org/licenses/by/4.0/) (00:00) - Discussing Data Handling in Swift UI (01:22) - Observable Objects and View Models (04:20) - The Power of Previews in Swift UI (06:36) - Combining Combine and Async/Await (10:29) - Interfacing Between UIKit and Swift UI (17:12) - Challenges with Swift Package Manager Thanks to our monthly supporters Bertram Eber Edward Sanchez Satoshi Mitsumori Danielle Lewis Steven Lipton ★ Support this podcast on Patreon ★

Ben Scheirman of NSScreenCast comes on to talk about migrating apps such as a Nike's Sneakers app from UIKit to SwiftUI and all the little things you don't think about. This is part 1 of a 2 part interview.GuestBen Scheirman | Ben is an experienced software engineer from Houston, TX. Currently focused on Swift, iOS, Ruby, and Rust.Ben Scheirman (@bens@mastodon.xyz) - Mastodonsubdigital (Ben Scheirman)NSScreencast: Bite-sized Screencasts for iOS DevelopmentCombine SwiftAnnouncementsNeed help with your projects this year? BrightDigit has openings.Join Bushel BetaJoin our Brand New Patreon Page!LinksEpisode #288: Modern UIKit: Stack Navigation, Part 2pointfreeco/swift-perception: Observable tools, backported.brightdigit/Sublimation: Enable automatic discovery of your local development server on the fly. Turn your Server-Side Swift app from a mysterious vapor to a tangible solid server.krzysztofzablocki/LifetimeTracker: Find retain cycles / memory leaks sooner.siteline/swiftui-introspect: Introspect underlying UIKit/AppKit components from SwiftUIPresenting Coordinators - Soroush Khanlou on VimeoRelated EpisodesSwiftUI Field Guide with Chris EidhofSOTU 2024 with Peter WithamSwiftUI Tips and Tricks with Craig ClaytonSwiftly Tooling with Pol Piella AbadiaIt Depends with Brandon WilliamsMy Taylor Deep Dish Swift Heroes World TourMobile System Design with Tjeerd in 't VeenThe Composable Architecture with Zev EisenbergBehind the Scenes of SwiftUI with Aviel GrossWWDC 2022 - SwiftUI and UIKit with Evan StoneSocial MediaEmailleo@brightdigit.comGitHub - @brightdigitTwitter BrightDigit - @brightdigitLeo - @leogdionLinkedInBrightDigitLeoPatreon - brightdigitCreditsMusic from https://filmmusic.io"Blippy Trance" by Kevin MacLeod (https://incompetech.com)License: CC BY (http://creativecommons.org/licenses/by/4.0/) (00:00) - Who is Ben Scherman (02:38) - Migrating Apps to Swift UI (07:03) - Challenges with Swift UI and iOS Versions (10:24) - Using Introspect for Swift UI (16:44) - Implementing Collection View in Swift UI (25:05) - Exploring iOS 18 Scroll View API (25:30) - SwiftUI vs UIKit: Productivity and Constraints (26:38) - Design and Engineering Collaboration (29:43) - Stages of Migrating to SwiftUI (34:14) - SwiftUI Navigation and Environment Bindings (39:44) - Retain Cycles and Memory Management Thanks to our monthly supporters Bertram Eber Edward Sanchez Satoshi Mitsumori Danielle Lewis Steven Lipton ★ Support this podcast on Patreon ★

COVID Vax Sperm and Egg Disaster, a Weapon of Mass Depopulation? BREAKING: Naonostructures in COVID-19 Injectables ALERT: The mRNA Vaccines Contained A Secret Weapon of Mass Depopulation For Globalists To Trigger At Will - Top Scientists Warn Christiane Northrup: Infertility clinics are reporting that The sperm of inoculated men does not swim and the eggs of inoculated women won't grow into embryos. Vaccine Nanobots: The Conspiracy That Could Be True BREAKING: New peer reviewed paper confirms presence of nanostructures in COVID-19 injectables   Post Alex Jones @RealAlexJones ALERT: The mRNA Vaccines Contained A Secret Weapon of Mass Depopulation For Globalists To Trigger At Will - Top Scientists Warn 17:17 / 19:16 6:43 PM · Sep 7, 2024 6M Views   Post Blue Sky    @Anpo_Star Dr. Christiane Northrup: Infertility clinics are reporting that The sperm of inoculated men does not swim and the eggs of inoculated women won't grow into embryos. 3:31 PM · Sep 7, 2024 16.4K Views   Vaccine Nanobots: The Conspiracy That Could Be True Watch this video at- https://www.youtube.com/live/o4IU71zHnCQ?si=vxqGsTnw1RCrGQsm Vejon Health 141K subscribers 59,866 views Streamed live on Sep 7, 2024 #covid #medicine #research In this incredible video, we dive deep into the controversial topic of vaccine nanobots. Are they a ground-breaking scientific advancement or just a conspiracy theory? ******************************************************************************************** Join Vejon Health to get access to Members Only videos and posts:    / @vejonhealth   ******************************************************************************************** Join us as we explore a recent paper on the presence of nanotechnology in vaccines. ========================================================== Composition and Potential Cause of Embalmers' Clots Webinar - Thursday 12th September at 7PM UK time https://www.eventbrite.ca/e/101120607... ========================================================== We'll separate fact from fiction and uncover the shocking truths that will change your perspective on vaccines forever. Don't miss out on this critical discussion that affects you and your health! Remember to like, subscribe, and hit the notification bell for more insightful content! ========================================================== Disease X: Are you prepared? A Comprehensive Guide to Pandemic Preparedness Join our Kickstarter here: https://www.kickstarter.com/projects/... ========================================================== Advanced Covid 360 Course Register for Pre-launch DISCOUNTED Course Here! Limited Time! https://vejonhealth.learnworlds.com/c... ========================================================== Help "Humming Heroes" get to Number ONE on Amazon! Preorder with a reduced price here: https://mcmillanresearch.com/humming_... ======================================================== Alternative Links Here: Substack - COVID-19: https://philipmcmillan.substack.com/ Patreon:   / vejonhealth   Videos: https://mcmillanresearch.com/media/ Courses: https://mcmillanresearch.com/mr-educa... Rumble: https://rumble.com/user/vejonhealth Substack - Long Covid: https://drphilipmcmillan.substack.com/ More info: https://vejonhealth.com/ Twitter - Vejon Health:   / vejon_health   Twitter - Dr Philip A McMillan:   / philamillan   #covid #medicine #research   Post BREAKING: New peer reviewed paper confirms presence of nanostructures in COVID-19 injectables Shaun Rickard @ShaunRickard67   Watch this video on Rumble at- https://rumble.com/v5e07d1-breaking-new-peer-reviewed-paper-confirms-presence-of-nanostructures-in-cov.html ***WARNING - Those who have been injected with the experimental mRNA Gene Therapies may to want to sit down before watching this video Dr. John Campbell presents new peer reviewed papers which confirm the presence of mRNA nanostructures in the COVID-19 injectables NOTE: I have screen recorded this video because YouTube will very likely censor it and take it down shortly. I have also uploaded it to my censorship free Rumble channel for those who wish to share it on other SM platforms, or with friends and family who are not on X: https://rumble.com/v5e07d1-breaking-new-peer-reviewed-paper-confirms-presence-of-nanostructures-in-cov.html… For those wishing to conduct their own research, and for all the sceptics out there, I have listed all of Dr. Campbell's notes, research and source links below... "Real-Time Self-Assembly of Stereomicroscopically Visible Artificial Constructions in Incubated Specimens of mRNA Products Mainly from Pfizer and Moderna: A Comprehensive Longitudinal Study https://ijvtpr.com/index.php/IJVTPR/article/view/102… Our observations suggest the presence of some kind of nanotechnology in the COVID-19 injectables. International Journal of Vaccine Theory, Practice, and Research https://ijvtpr.com/index.php/IJVTPR/index… Full version of the journal Vol. 3 No. 2 (2024): Injuries, Causes, and Treatments, Part 2 https://ijvtpr.com/index.php/IJVTPR/issue/view/6… Creative Commons link https://creativecommons.org/licenses/by-nc-nd/4.0/… Observable real-time injuries at the cellular level in recipients of the “safe and effective” COVID-19 injectables are documented here for the first time, with the presentation of a comprehensive description and analysis of observed phenomena. The global administration of these often-mandated products from late 2020 triggered a plethora of independent research studies of the modified RNA injectable gene therapies, most notably those manufactured by Pfizer and Moderna. Analyses reported here consist of precise laboratory “bench science” aiming to understand why serious debilitating, prolonged injuries (and many deaths) occurred increasingly without any measurable protective effect The contents of COVID-19 injectables were examined under a stereomicroscope at up to 400X magnification. Carefully preserved specimens were cultured in a range of distinct media to observe immediate and long-term cause-and-effect relationships between the injectables and living cells under carefully controlled conditions. From such research, reasonable inferences can be drawn about observed injuries worldwide that have occurred since the injectables were pressed upon billions of individuals. In addition to cellular toxicity, our findings reveal numerous — on the order of 3~4 x 106 per milliliter of the injectable — visible artificial self-assembling entities ranging from about 1 to 100 µm, or greater, of many different shapes. There were animated worm-like entities, discs, chains, spirals, tubes, right-angle structures containing other artificial entities within them All these are exceedingly beyond any expected and acceptable levels of contamination of the COVID-19 injectables, and incubation studies revealed the progressive self-assembly of many artifactual structures. As time progressed during incubation, simple one- and two-dimensional structures over two or three weeks became more complex in shape and size developing into stereoscopically visible entities in three-dimensions. They resembled carbon nanotube filaments, ribbons, and tapes, some appearing as transparent, thin, flat membranes, and others as three-dimensional spirals, and beaded chains. Some of these seemed to appear and then disappear over time. Our observations suggest the presence of some kind of nanotechnology in the COVID-19 injectables. Dr. Young Mi Lee, Jeju, Jejudo, 63098, Republic of Korea (South Korea)" @TuckerCarlson @natalimorris @ReginaWatteel @KarlDHarrison @jordanbpeterson @elonmusk @WoodReporting @NChartierET @Inquiry_Canada @PierrePoilievre @CPC_HQ @ColinCarrieCPC @DrJBhattacharya @DavidKrayden @brianlilley @rupasubramanya @rustyrockets @therationalpost @ryangerritsen @TheRedactedInc @dbongino @bennyjohnson @charliekirk11 @realDonaldTrump 8:44 AM · Sep 7, 2024 1.2M Views

In this episode, Jimmy shares valuable examples of Culturize Checkpoints to help bring clarity to what being a Champion for Students truly looks like in our classrooms, and more importantly, what it leads to when it comes to observable impact. 

Known for co-creating Django and Datasette, as well as his thoughtful writing on LLMs, Simon Willison joins the show to chat about blogging as an accountability mechanism, how to build intuition with LLMs, building a startup with his partner on their honeymoon, and more.   Segments: (00:00:00) The weird intern (00:01:50) The early days of LLMs (00:04:59) Blogging as an accountability mechanism (00:09:24) The low-pressure approach to blogging (00:11:47) GitHub issues as a system of records (00:16:15) Temporal documentation and design docs (00:18:19) GitHub issues for team collaboration (00:21:53) Copy-paste as an API (00:26:54) Observable notebooks (00:28:50) pip install LLM (00:32:26) The evolution of using LLMs daily (00:34:47) Building intuition with LLMs (00:43:24) Democratizing access to automation (00:47:45) Alternative interfaces for language models (00:53:39) Is prompt engineering really engineering? (00:58:39) The frustrations of working with LLMs (01:01:59) Structured data extraction with LLMs (01:06:08) How Simon would go about building a LLM app (01:09:49) LLMs making developers more ambitious (01:13:32) Typical workflow with LLMs (01:19:58) Vibes-based evaluation (01:23:25) Staying up-to-date with LLMs (01:27:49) The impact of LLMs on new programmers (01:29:37) The rise of 'Goop' and the future of software development (01:40:20) Being an independent developer (01:42:26) Staying focused and accountable (01:47:30) Building a startup with your partner on the honeymoon (01:51:30) The responsibility of AI practitioners (01:53:07) The hidden dangers of prompt injection (01:53:44) “Artificial intelligence” is really “imitation intelligence”   Show Notes: Simon's blog: https://simonwillison.net/ Natalie's post on them building a startup together: https://blog.natbat.net/post/61658401806/lanyrd-from-idea-to-exit Simon's talk from DjangoCon: https://www.youtube.com/watch?v=GLkRK2rJGB0 Simon on twitter: https://x.com/simonw Datasette: https://github.com/simonw/datasette   Stay in touch:

Fear of death often stems from the unknown, yet this very uncertainty can be a source of comfort rather than dread. Since we lack concrete knowledge about what lies beyond life, it's worth considering that our fear might be misplaced. Death is a universal experience, and its mystery can remind us to embrace the present, […]

We've hoped you enjoyed this episode of the Morbid Forest Spotlight series. On today's episode, you've listened to our neighbors- Observable Radio.    Observable Radio is a found footage anthology retro sci fi analogy podcast. Follow the Observe as he and his colleague decipher mysterious signals from other worlds.  https://www.observableradio.com/ https://twitter.com/observableradio https://www.youtube.com/@observableradio https://www.instagram.com/observableradio See you very soon, Dear Travelers!!  

Join Hugh Ross in this breaking News of the Day episode of Stars, Cells, and God. Hugh describes a discovery that may resolve a long-standing mystery about dark matter. Primordial Black Holes Resolve Dark Matter Mystery? Dark matter is matter that doesn't interact with light or interacts at an extremely weak level. The quantity of dark matter that exists and its locations in the universe are not mysteries. Dark matter's composition is a mystery that has stymied astronomers and physicists for 5 decades. Leading candidates for dark matter's composition are axions and sterile neutrinos, but neither of these particles has been detected. Physicists Elba Alonso-Monsalve and David Kaiser propose that primordial black holes (PBHs) could make up all or a large fraction of dark matter if they formed previous to a tenth of a quadrillionth of a second after the cosmic creation event. These PBHs would take two forms: atom-sized bodies with masses equal to that Deimos and Phobos; bodies a ten thousandth the diameter of a proton with masses equal to a ton, with only the first form possibly existing to the present time. Observable tests for these PBHs include the degree to which they would 1) shift the balance between protons and neutrons, 2) cause ripples in the cosmic spacetime fabric, and 3) affect the amount of helium produced during the universe's first 3 minutes. Resources:  Primordial Black Holes with QCD Color Change Quantum Gravity Constraints Affirm Cosmic Creator

Scott and Wes serve up top tools and tricks for rapid idea execution, from JavaScript services like Valtown and Observable to database solutions including LowDB and Google Sheets integration. Get ready to streamline your development ideation process with these tasty insights! Show Notes 00:00 Welcome to Syntax! 02:16 Brought to you by Sentry.io. 03:14 JavaScript Services. 03:43 Valtown. 05:44 Observable. 06:35 Notebooks. 08:23 Deno Juypter Notebooks. 09:51 Svelte Repl. 10:32 Playgrounds: TypeScript, Tailwind, etc… 11:05 CSS Services. 11:10 CodePen. 13:14 Full stack services. 13:47 Your own stack. Hot Tips & Cool Treats. Wes's Hot Tips. Scott's Cool Treats. 21:01 Bun file routing. 24:25 Tooling and tips. 26:30 Database. 26:51 Write to a file. 27:40 LowDB. 29:00 SQLite + Drizzle. 29:40 Google Sheets. 30:06 Sheet DB. Hit us up on Socials! Syntax: X Instagram Tiktok LinkedIn Threads Wes: X Instagram Tiktok LinkedIn Threads Scott:X Instagram Tiktok LinkedIn Threads Randy: X Instagram YouTube Threads

The Observer catalogues his first signal. A mysterious curfew. An annual celebration. The pursuit of stasis.   This program is intended for mature audiences only. Content Warnings Mention of Suicide Implied harm of a Child   Performed by The Ensemble Phil van Hest Katie Skovholt Orion Kellogg Jason Smith Xalavier Nelson Jr. Kris Straub Purpurina

Welcome to the latest edition of Talking Data. Our Talking Data series seeks to offer timely insights into macro market themes along with macro data and its impact on the economy and markets. I am your host Kristen Radosh of Arbor Research and Trading. Our commentator is Jim Bianco of Bianco Research. Today Jim discusses managing fixed income in a world with yield. • What is going on with interest rates? • What is your outlook? • How hard is it to beat a benchmark? • How has the Index done? • What is the current positioning? Thank you for joining us today. We are client driven, if you have any questions or feedback on future topics, please let us know. For further information on Arbor Research, Bianco Research and Arbor Data Science, please contact Gus Handler at gus.handler@arborresearch.com.

The Multiverse Theory isn't just something out of science fiction anymore—it's real! Scientists have been diving deep into this mind-bending concept, and the evidence is stacking up. Multiple universes are coexisting alongside our own, each with its own set of rules and possibilities. It's like a cosmic buffet of alternate realities! From quantum mechanics to astrophysics, researchers are uncovering clues that suggest the existence of parallel universes, opening up a whole new realm of exploration and discovery. So, the Multiverse Theory isn't just a wild idea—it's a tantalizing glimpse into the vastness of the cosmos. Credit: CC BY-SA 3.0 https://creativecommons.org/licenses/... Observable universe: Unmismoobjetivo, https://commons.wikimedia.org/wiki/Fi... History of the Universe: Drbogdan, Yinweichen, https://commons.wikimedia.org/wiki/Fi... Wave-particle duality: Jubobroff, https://commons.wikimedia.org/wiki/Fi... CC BY-SA 4.0 https://creativecommons.org/licenses/... Big Bang Singularity: Waterced, https://commons.wikimedia.org/wiki/Fi... String Vibrations: SriVrushank(1840372), https://commons.wikimedia.org/wiki/Fi... Quantum Fluctuations: Derek Leinweber, https://commons.wikimedia.org/wiki/Fi... Double Slit Particle: Sean Kelley/The Quantum Atlas, https://commons.wikimedia.org/wiki/Fi... Animation is created by Bright Side. ---------------------------------------------------------------------------------------- Music from TheSoul Sound: https://thesoul-sound.com/ Check our Bright Side podcast on Spotify and leave a positive review! https://open.spotify.com/show/0hUkPxD... Subscribe to Bright Side: https://goo.gl/rQTJZz ---------------------------------------------------------------------------------------- Our Social Media: Facebook:   / brightside   Instagram:   / brightside.official   TikTok: https://www.tiktok.com/@brightside.of... Stock materials (photos, footages and other): https://www.depositphotos.com https://www.shutterstock.com https://www.eastnews.ru ---------------------------------------------------------------------------------------- For more videos and articles visit: http://www.brightside.me Learn more about your ad choices. Visit megaphone.fm/adchoices

Preached by Nathan Bayly; Acts 4:13-17

This is a recap of the top 10 posts on Hacker News on February 15th, 2024.This podcast was generated by wondercraft.ai(00:37): Sora: Creating video from textOriginal post: https://news.ycombinator.com/item?id=39386156&utm_source=wondercraft_ai(02:25): Our next-generation model: Gemini 1.5Original post: https://news.ycombinator.com/item?id=39383446&utm_source=wondercraft_ai(04:07): OpenAI – Application for US trademark “GPT” has failedOriginal post: https://news.ycombinator.com/item?id=39380165&utm_source=wondercraft_ai(05:35): Apple confirms it's breaking iPhone web apps in the EU on purposeOriginal post: https://news.ycombinator.com/item?id=39388218&utm_source=wondercraft_ai(07:36): Observable 2.0, a static site generator for data appsOriginal post: https://news.ycombinator.com/item?id=39383386&utm_source=wondercraft_ai(09:19): Uv: Python packaging in RustOriginal post: https://news.ycombinator.com/item?id=39387641&utm_source=wondercraft_ai(11:02): Every default macOS wallpaperOriginal post: https://news.ycombinator.com/item?id=39384731&utm_source=wondercraft_ai(12:57): Asahi Linux project's OpenGL support on Apple Silicon officially surpasses AppleOriginal post: https://news.ycombinator.com/item?id=39383798&utm_source=wondercraft_ai(14:46): Goodbye Auth0Original post: https://news.ycombinator.com/item?id=39380790&utm_source=wondercraft_ai(16:35): Feds want to ban the Flipper Zero – Experts say it's a scapegoatOriginal post: https://news.ycombinator.com/item?id=39385301&utm_source=wondercraft_aiThis is a third-party project, independent from HN and YC. Text and audio generated using AI, by wondercraft.ai. Create your own studio quality podcast with text as the only input in seconds at app.wondercraft.ai. Issues or feedback? We'd love to hear from you: team@wondercraft.ai

Subscriber-Only: Today's episode is available only to subscribers. If you are a Point-Free subscriber you can access your private podcast feed by visiting https://www.pointfree.co/account. --- So what's the point of observation in the Composable Architecture? While we have seemingly simplified nearly every inch of the library as it interfaces with SwiftUI, let's zoom out a bit, explore how some integration tests that benchmark certain aspects of the library have changed, and migrate the Todos application we built in the very first tour of this library to the new tools.

Why can't God find a group of people to say, 'That's my pursuit, I'm coming after your face, I want to behold Your glory?!?'" Pastor Todd implores church members to seek the face of God, to pray to behold His glory, and to ask for the Lord to reveal anything in our hearts that grieves Him, repent of it, and remove it out of our lives. This allows the weight of the glory of God to settle and dwell.

Subscriber-Only: Today's episode is available only to subscribers. If you are a Point-Free subscriber you can access your private podcast feed by visiting https://www.pointfree.co/account. --- We have iterated on how bindings work in the Composable Architecture many times, but have never been fully happy with the results. With Observation, that all changes. By eliminating view stores and observing store state directly, we are free to totally reimagine bindings in the Composable Architecture, and get rid of even more concepts in the process.

Subscriber-Only: Today's episode is available only to subscribers. If you are a Point-Free subscriber you can access your private podcast feed by visiting https://www.pointfree.co/account. --- Observation has allowed us to get rid of a number of view wrappers the Composable Architecture used to require in favor of vanilla SwiftUI views, instead, but we still depend on a zoo of view modifiers to drive navigation. Let's rethink all of these helpers and see if we can trade them out for simpler, vanilla SwiftUI view modifiers, instead.

Subscriber-Only: Today's episode is available only to subscribers. If you are a Point-Free subscriber you can access your private podcast feed by visiting https://www.pointfree.co/account. --- We can now observe struct, optional, and enum state in the Composable Architecture, but what about collections? Let's explore what it takes to get rid of the `ForEachStore` wrapper view for a vanilla `ForEach` view instead, while still observing updates to collection state in the most minimal way possible.

Bienvenue pour un sacré voyage depuis la Terre jusqu'aux confins de l'univers dans l'œil des télescopes géants du désert d'Atacama au Chili, avec l'astrophysicien François Hammer, responsable scientifique de ces instruments astronomiques. (Rediffusion du 20 février 2023). Retrouvons-nous la tête dans les étoiles aujourd'hui, dans les milliards d'étoiles de notre galaxie. Galaxie explorée et cartographiée par le plus puissant télescope terrestre Le VLT (very large télescope) situé dans le désert d'Atacama au Chili.Embarquement immédiat pour un sacré voyage de la Terre jusqu'aux confins de l'Univers. Pour reprendre le titre de l'ouvrage de notre invité l'astrophysicien Francois Hammer, responsable scientifique des grands spectrographes installés au Chili sur le site du VLT qui va nous en faire voir de toutes les couleurs, mais aussi de toutes les formes stellaires et planétaires voire exoplanétaires...Avec l'astrophysicien à l'Observatoire de Paris, François Hammer, pour son ouvrage Voyage de la Terre aux confins de l'Univers paru chez Odile Jacob.Et notre éphéméride mensuel Ciel d'Afrique.

Bienvenue pour un sacré voyage depuis la Terre jusqu'aux confins de l'univers dans l'œil des télescopes géants du désert d'Atacama au Chili, avec l'astrophysicien François Hammer, responsable scientifique de ces instruments astronomiques. (Rediffusion du 20 février 2023). Retrouvons-nous la tête dans les étoiles aujourd'hui, dans les milliards d'étoiles de notre galaxie. Galaxie explorée et cartographiée par le plus puissant télescope terrestre Le VLT (very large télescope) situé dans le désert d'Atacama au Chili.Embarquement immédiat pour un sacré voyage de la Terre jusqu'aux confins de l'Univers. Pour reprendre le titre de l'ouvrage de notre invité l'astrophysicien Francois Hammer, responsable scientifique des grands spectrographes installés au Chili sur le site du VLT qui va nous en faire voir de toutes les couleurs, mais aussi de toutes les formes stellaires et planétaires voire exoplanétaires...Avec l'astrophysicien à l'Observatoire de Paris, François Hammer, pour son ouvrage Voyage de la Terre aux confins de l'Univers paru chez Odile Jacob.Et notre éphéméride mensuel Ciel d'Afrique.

Subscriber-Only: Today's episode is available only to subscribers. If you are a Point-Free subscriber you can access your private podcast feed by visiting https://www.pointfree.co/account. --- We've made structs and optionals observable in the Composable Architecture, eliminating the need for `ViewStore`s and `IfLetStore`s, so what about enums? If we can make enums observable, we could further eliminate the concept of the `SwitchStore`, greatly improving the ergonomics of working with enums in the library.

Catch us at Modular's ModCon next week with Chris Lattner, and join our community!Due to Bryan's very wide ranging experience in data science and AI across Blue Bottle (!), StitchFix, Weights & Biases, and now Hex Magic, this episode can be considered a two-parter.Notebooks = Chat++We've talked a lot about AI UX (in our meetups, writeups, and guest posts), and today we're excited to dive into a new old player in AI interfaces: notebooks! Depending on your background, you either Don't Like or you Like notebooks — they are the most popular example of Knuth's Literate Programming concept, basically a collection of cells; each cell can execute code, display it, and share its state with all the other cells in a notebook. They can also simply be Markdown cells to add commentary to the analysis. Notebooks have a long history but most recently became popular from iPython evolving into Project Jupyter, and a wave of notebook based startups from Observable to DeepNote and Databricks sprung up for the modern data stack.The first wave of AI applications has been very chat focused (ChatGPT, Character.ai, Perplexity, etc). Chat as a user interface has a few shortcomings, the major one being the inability to edit previous messages. We enjoyed Bryan's takes on why notebooks feel like “Chat++” and how they are building Hex Magic:* Atomic actions vs Stream of consciousness: in a chat interface, you make corrections by adding more messages to a conversation (i.e. “Can you try again by doing X instead?” or “I actually meant XYZ”). The context can easily get messy and confusing for models (and humans!) to follow. Notebooks' cell structure on the other hand allows users to go back to any previous cells and make edits without having to add new ones at the bottom. * “Airlocks” for repeatability: one of the ideas they came up with at Hex is “airlocks”, a collection of cells that depend on each other and keep each other in sync. If you have a task like “Create a summary of my customers' recent purchases”, there are many sub-tasks to be done (look up the data, sum the amounts, write the text, etc). Each sub-task will be in its own cell, and the airlock will keep them all in sync together.* Technical + Non-Technical users: previously you had to use Python / R / Julia to write notebooks code, but with models like GPT-4, natural language is usually enough. Hex is also working on lowering the barrier of entry for non-technical users into notebooks, similar to how Code Interpreter is doing the same in ChatGPT. Obviously notebooks aren't new for developers (OpenAI Cookbooks are a good example), but haven't had much adoption in less technical spheres. Some of the shortcomings of chat UIs + LLMs lowering the barrier of entry to creating code cells might make them a much more popular UX going forward.RAG = RecSys!We also talked about the LLMOps landscape and why it's an “iron mine” rather than a “gold rush”: I'll shamelessly steal [this] from a friend, Adam Azzam from Prefect. He says that [LLMOps] is more of like an iron mine than a gold mine in the sense of there is a lot of work to extract this precious, precious resource. Don't expect to just go down to the stream and do a little panning. There's a lot of work to be done. And frankly, the steps to go from this resource to something valuable is significant.Some of my favorite takeaways:* RAG as RecSys for LLMs: at its core, the goal of a RAG pipeline is finding the most relevant documents based on a task. This isn't very different from traditional recommendation system products that surface things for users. How can we apply old lessons to this new problem? Bryan cites fellow AIE Summit speaker and Latent Space Paper Club host Eugene Yan in decomposing the retrieval problem into retrieval, filtering, and scoring/ranking/ordering:As AI Engineers increasingly find that long context has tradeoffs, they will also have to relearn age old lessons that vector search is NOT all you need and a good systems not models approach is essential to scalable/debuggable RAG. Good thing Bryan has just written the first O'Reilly book about modern RecSys, eh?* Narrowing down evaluation: while “hallucination” is a easy term to throw around, the reality is more nuanced. A lot of times, model errors can be automatically fixed: is this JSON valid? If not, why? Is it just missing a closing brace? These smaller issues can be checked and fixed before returning the response to the user, which is easier than fixing the model.* Fine-tuning isn't all you need: when they first started building Magic, one of the discussions was around fine-tuning a model. In our episode with Jeremy Howard we talked about how fine-tuning leads to loss of capabilities as well. In notebooks, you are often dealing with domain-specific data (i.e. purchases, orders, wardrobe composition, household items, etc); the fact that the model understands that “items” are probably part of an “order” is really helpful. They have found that GPT-4 + 3.5-turbo were everything they needed to ship a great product rather than having to fine-tune on notebooks specifically.Definitely recommend listening to this one if you are interested in getting a better understanding of how to think about AI, data, and how we can use traditional machine learning lessons in large language models. The AI PivotFor more Bryan, don't miss his fireside chat at the AI Engineer Summit:Show Notes* Hex Magic* Bryan's new book: Building Recommendation Systems in Python and JAX* Bryan's whitepaper about MLOps* “Kitbashing in ML”, slides from his talk on building on top of foundation models* “Bayesian Statistics The Fun Way” by Will Kurt* Bryan's Twitter* “Berkeley man determined to walk every street in his city”* People:* Adam Azzam* Graham Neubig* Eugene Yan* Even OldridgeTimestamps* [00:00:00] Bryan's background* [00:02:34] Overview of Hex and the Magic product* [00:05:57] How Magic handles the complex notebook format to integrate cleanly with Hex* [00:08:37] Discussion of whether to build vs buy models - why Hex uses GPT-4 vs fine-tuning* [00:13:06] UX design for Magic with Hex's notebook format (aka “Chat++”)* [00:18:37] Expanding notebooks to less technical users* [00:23:46] The "Memex" as an exciting underexplored area - personal knowledge graph and memory augmentation* [00:27:02] What makes for good LLMops vs MLOps* [00:34:53] Building rigorous evaluators for Magic and best practices* [00:36:52] Different types of metrics for LLM evaluation beyond just end task accuracy* [00:39:19] Evaluation strategy when you don't own the core model that's being evaluated* [00:41:49] All the places you can make improvements outside of retraining the core LLM* [00:45:00] Lightning RoundTranscriptAlessio: Hey everyone, welcome to the Latent Space Podcast. This is Alessio, Partner and CTO-in-Residence of Decibel Partners, and today I'm joining by Bryan Bischof. [00:00:15]Bryan: Hey, nice to meet you. [00:00:17]Alessio: So Bryan has one of the most thorough and impressive backgrounds we had on the show so far. Lead software engineer at Blue Bottle Coffee, which if you live in San Francisco, you know a lot about. And maybe you'll tell us 30 seconds on what that actually means. You worked as a data scientist at Stitch Fix, which used to be one of the premier data science teams out there. [00:00:38]Bryan: It used to be. Ouch. [00:00:39]Alessio: Well, no, no. Well, you left, you know, so how good can it still be? Then head of data science at Weights and Biases. You're also a professor at Rutgers and you're just wrapping up a new O'Reilly book as well. So a lot, a lot going on. Yeah. [00:00:52]Bryan: And currently head of AI at Hex. [00:00:54]Alessio: Let's do the Blue Bottle thing because I definitely want to hear what's the, what's that like? [00:00:58]Bryan: So I was leading data at Blue Bottle. I was the first data hire. I came in to kind of get the data warehouse in order and then see what we could build on top of it. But ultimately I mostly focused on demand forecasting, a little bit of recsys, a little bit of sort of like website optimization and analytics. But ultimately anything that you could imagine sort of like a retail company needing to do with their data, we had to do. I sort of like led that team, hired a few people, expanded it out. One interesting thing was I was part of the Nestle acquisition. So there was a period of time where we were sort of preparing for that and didn't know, which was a really interesting dynamic. Being acquired is a very not necessarily fun experience for the data team. [00:01:37]Alessio: I build a lot of internal tools for sourcing at the firm and we have a small VCs and data community of like other people doing it. And I feel like if you had a data feed into like the Blue Bottle in South Park, the Blue Bottle at the Hanahaus in Palo Alto, you can get a lot of secondhand information on the state of VC funding. [00:01:54]Bryan: Oh yeah. I feel like the real source of alpha is just bugging a Blue Bottle. [00:01:58]Alessio: Exactly. And what's your latest book about? [00:02:02]Bryan: I just wrapped up a book with a coauthor Hector Yee called Building Production Recommendation Systems. I'll give you the rest of the title because it's fun. It's in Python and JAX. And so for those of you that are like eagerly awaiting the first O'Reilly book that focuses on JAX, here you go. [00:02:17]Alessio: Awesome. And we'll chat about that later on. But let's maybe talk about Hex and Magic before. I've known Hex for a while, I've used it as a notebook provider and you've been working on a lot of amazing AI enabled experiences. So maybe run us through that. [00:02:34]Bryan: So I too, before I sort of like joined Hex, saw it as this like really incredible notebook platform, sort of a great place to do data science workflows, quite complicated, quite ad hoc interactive ones. And before I joined, I thought it was the best place to do data science workflows. And so when I heard about the possibility of building AI tools on top of that platform, that seemed like a huge opportunity. In particular, I lead the product called Magic. Magic is really like a suite of sort of capabilities as opposed to its own independent product. What I mean by that is they are sort of AI enhancements to the existing product. And that's a really important difference from sort of building something totally new that just uses AI. It's really important to us to enhance the already incredible platform with AI capabilities. So these are things like the sort of obvious like co-pilot-esque vibes, but also more interesting and dynamic ways of integrating AI into the product. And ultimately the goal is just to make people even more effective with the platform. [00:03:38]Alessio: How do you think about the evolution of the product and the AI component? You know, even if you think about 10 months ago, some of these models were not really good on very math based tasks. Now they're getting a lot better. I'm guessing a lot of your workloads and use cases is data analysis and whatnot. [00:03:53]Bryan: When I joined, it was pre 4 and it was pre the sort of like new chat API and all that. But when I joined, it was already clear that GPT was pretty good at writing code. And so when I joined, they had already executed on the vision of what if we allowed the user to ask a natural language prompt to an AI and have the AI assist them with writing code. So what that looked like when I first joined was it had some capability of writing SQL and it had some capability of writing Python and it had the ability to explain and describe code that was already written. Those very, what feel like now primitive capabilities, believe it or not, were already quite cool. It's easy to look back and think, oh, it's like kind of like Stone Age in these timelines. But to be clear, when you're building on such an incredible platform, adding a little bit of these capabilities feels really effective. And so almost immediately I started noticing how it affected my own workflow because ultimately as sort of like an engineering lead and a lot of my responsibility is to be doing analytics to make data driven decisions about what products we build. And so I'm actually using Hex quite a bit in the process of like iterating on our product. When I'm using Hex to do that, I'm using Magic all the time. And even in those early days, the amount that it sped me up, that it enabled me to very quickly like execute was really impressive. And so even though the models weren't that good at certain things back then, that capability was not to be underestimated. But to your point, the models have evolved between 3.5 Turbo and 4. We've actually seen quite a big enhancement in the kinds of tasks that we can ask Magic and even more so with things like function calling and understanding a little bit more of the landscape of agent workflows, we've been able to really accelerate. [00:05:57]Alessio: You know, I tried using some of the early models in notebooks and it actually didn't like the IPyNB formatting, kind of like a JSON plus XML plus all these weird things. How have you kind of tackled that? Do you have some magic behind the scenes to make it easier for models? Like, are you still using completely off the shelf models? Do you have some proprietary ones? [00:06:19]Bryan: We are using at the moment in production 3.5 Turbo and GPT-4. I would say for a large number of our applications, GPT-4 is pretty much required. To your question about, does it understand the structure of the notebook? And does it understand all of this somewhat complicated wrappers around the content that you want to show? We do our very best to abstract that away from the model and make sure that the model doesn't have to think about what the cell wrapper code looks like. Or for our Magic charts, it doesn't have to speak the language of Vega. These are things that we put a lot of work in on the engineering side, to the AI engineer profile. This is the AI engineering work to get all of that out of the way so that the model can speak in the languages that it's best at. The model is quite good at SQL. So let's ensure that it's speaking the language of SQL and that we are doing the engineering work to get the output of that model, the generations, into our notebook format. So too for other cell types that we support, including charts, and just in general, understanding the flow of different cells, understanding what a notebook is, all of that is hard work that we've done to ensure that the model doesn't have to learn anything like that. I remember early on, people asked the question, are you going to fine tune a model to understand Hex cells? And almost immediately, my answer was no. No we're not. Using fine-tuned models in 2022, I was already aware that there are some limitations of that approach and frankly, even using GPT-3 and GPT-2 back in the day in Stitch Fix, I had already seen a lot of instances where putting more effort into pre- and post-processing can avoid some of these larger lifts. [00:08:14]Alessio: You mentioned Stitch Fix and GPT-2. How has the balance between build versus buy, so to speak, evolved? So GPT-2 was a model that was not super advanced, so for a lot of use cases it was worth building your own thing. Is with GPT-4 and the likes, is there a reason to still build your own models for a lot of this stuff? Or should most people be fine-tuning? How do you think about that? [00:08:37]Bryan: Sometimes people ask, why are you using GPT-4 and why aren't you going down the avenue of fine-tuning today? I can get into fine-tuning specifically, but I do want to talk a little bit about the good old days of GPT-2. Shout out to Reza. Reza introduced me to GPT-2. I still remember him explaining the difference between general transformers and GPT. I remember one of the tasks that we wanted to solve with transformer-based generative models at Stitch Fix were writing descriptions of clothing. You might think, ooh, that's a multi-modal problem. The answer is, not necessarily. We actually have a lot of features about the clothes that are almost already enough to generate some reasonable text. I remember at that time, that was one of the first applications that we had considered. There was a really great team of NLP scientists at Stitch Fix who worked on a lot of applications like this. I still remember being exposed to the GPT endpoint back in the days of 2. If I'm not mistaken, and feel free to fact check this, I'm pretty sure Stitch Fix was the first OpenAI customer, unlike their true enterprise application. Long story short, I ultimately think that depending on your task, using the most cutting-edge general model has some advantages. If those are advantages that you can reap, then go for it. So at Hex, why GPT-4? Why do we need such a general model for writing code, writing SQL, doing data analysis? Shouldn't a fine-tuned model just on Kaggle notebooks be good enough? I'd argue no. And ultimately, because we don't have one specific sphere of data that we need to write great data analysis workbooks for, we actually want to provide a platform for anyone to do data analysis about their business. To do that, you actually need to entertain an extremely general universe of concepts. So as an example, if you work at Hex and you want to do data analysis, our projects are called Hexes. That's relatively straightforward to teach it. There's a concept of a notebook. These are data science notebooks, and you want to ask analytics questions about notebooks. Maybe if you trained on notebooks, you could answer those questions, but let's come back to Blue Bottle. If I'm at Blue Bottle and I have data science work to do, I have to ask it questions about coffee. I have to ask it questions about pastries, doing demand forecasting. And so very quickly, you can see that just by serving just those two customers, a model purely fine-tuned on like Kaggle competitions may not actually fit the bill. And so the more and more that you want to build a platform that is sufficiently general for your customer base, the more I think that these large general models really pack a lot of additional opportunity in. [00:11:21]Alessio: With a lot of our companies, we talked about stuff that you used to have to extract features for, now you have out of the box. So say you're a travel company, you want to do a query, like show me all the hotels and places that are warm during spring break. It would be just literally like impossible to do before these models, you know? But now the model knows, okay, spring break is like usually these dates and like these locations are usually warm. So you get so much out of it for free. And in terms of Magic integrating into Hex, I think AI UX is one of our favorite topics and how do you actually make that seamless. In traditional code editors, the line of code is like kind of the atomic unit and HEX, you have the code, but then you have the cell also. [00:12:04]Bryan: I think the first time I saw Copilot and really like fell in love with Copilot, I thought finally, fancy auto-complete. And that felt so good. It felt so elegant. It felt so right sized for the task. But as a data scientist, a lot of the work that you do previous to the ML engineering part of the house, you're working in these cells and these cells are atomic. They're expressing one idea. And so ultimately, if you want to make the transition from something like this code, where you've got like a large amount of code and there's a large amount of files and they kind of need to have awareness of one another, and that's a long story and we can talk about that. But in this atomic, somewhat linear flow through the notebook, what you ultimately want to do is you want to reason with the agent at the level of these individual thoughts, these atomic ideas. Usually it's good practice in say Jupyter notebook to not let your cells get too big. If your cell doesn't fit on one page, that's like kind of a code smell, like why is it so damn big? What are you doing in this cell? That also lends some hints as to what the UI should feel like. I want to ask questions about this one atomic thing. So you ask the agent, take this data frame and strip out this prefix from all the strings in this column. That's an atomic task. It's probably about two lines of pandas. I can write it, but it's actually very natural to ask magic to do that for me. And what I promise you is that it is faster to ask magic to do that for me. At this point, that kind of code, I never write. And so then you ask the next question, which is what should the UI be to do chains, to do multiple cells that work together? Because ultimately a notebook is a chain of cells and actually it's a first class citizen for Hex. So we have a DAG and the DAG is the execution DAG for the individual cells. This is one of the reasons that Hex is reactive and kind of dynamic in that way. And so the very next question is, what is the sort of like AI UI for these collections of cells? And back in June and July, we thought really hard about what does it feel like to ask magic a question and get a short chain of cells back that execute on that task. And so we've thought a lot about sort of like how that breaks down into individual atomic units and how those are tied together. We introduced something which is kind of an internal name, but it's called the airlock. And the airlock is exactly a sequence of cells that refer to one another, understand one another, use things that are happening in other cells. And it gives you a chance to sort of preview what magic has generated for you. Then you can accept or reject as an entire group. And that's one of the reasons we call it an airlock, because at any time you can sort of eject the airlock and see it in the space. But to come back to your question about how the AI UX fits into this notebook, ultimately a notebook is very conversational in its structure. I've got a series of thoughts that I'm going to express as a series of cells. And sometimes if I'm a kind data scientist, I'll put some text in between them too, explaining what on earth I'm doing. And that feels, in my opinion, and I think this is quite shared amongst exons, that feels like a really nice refinement of the chat UI. I've been saying for several months now, like, please stop building chat UIs. There is some irony because I think what the notebook allows is like chat plus plus. [00:15:36]Alessio: Yeah, I think the first wave of everything was like chat with X. So it was like chat with your data, chat with your documents and all of this. But people want to code, you know, at the end of the day. And I think that goes into the end user. I think most people that use notebooks are software engineer, data scientists. I think the cool things about these models is like people that are not traditionally technical can do a lot of very advanced things. And that's why people like code interpreter and chat GBT. How do you think about the evolution of that persona? Do you see a lot of non-technical people also now coming to Hex to like collaborate with like their technical folks? [00:16:13]Bryan: Yeah, I would say there might even be more enthusiasm than we're prepared for. We're obviously like very excited to bring what we call the like low floor user into this world and give more people the opportunity to self-serve on their data. We wanted to start by focusing on users who are already familiar with Hex and really make magic fantastic for them. One of the sort of like internal, I would say almost North Stars is our team's charter is to make Hex feel more magical. That is true for all of our users, but that's easiest to do on users that are already able to use Hex in a great way. What we're hearing from some customers in particular is sort of like, I'm excited for some of my less technical stakeholders to get in there and start asking questions. And so that raises a lot of really deep questions. If you immediately enable self-service for data, which is almost like a joke over the last like maybe like eight years, if you immediately enabled self-service, what challenges does that bring with it? What risks does that bring with it? And so it has given us the opportunity to think about things like governance and to think about things like alignment with the data team and making sure that the data team has clear visibility into what the self-service looks like. Having been leading a data team, trying to provide answers for stakeholders and hearing that they really want to self-serve, a question that we often found ourselves asking is, what is the easiest way that we can keep them on the rails? What is the easiest way that we can set up the data warehouse and set up our tools such that they can ask and answer their own questions without coming away with like false answers? Because that is such a priority for data teams, it becomes an important focus of my team, which is, okay, magic may be an enabler. And if it is, what do we also have to respect? We recently introduced the data manager and the data manager is an auxiliary sort of like tool on the Hex platform to allow people to write more like relevant metadata about their data warehouse to make sure that magic has access to the best information. And there are some things coming to kind of even further that story around governance and understanding. [00:18:37]Alessio: You know, you mentioned self-serve data. And when I was like a joke, you know, the whole rush to the modern data stack was something to behold. Do you think AI is like in a similar space where it's like a bit of a gold rush? [00:18:51]Bryan: I have like sort of two comments here. One I'll shamelessly steal from a friend, Adam Azzam from Prefect. He says that this is more of like an iron mine than a gold mine in the sense of there is a lot of work to extract this precious, precious resource. And that's the first one is I think, don't expect to just go down to the stream and do a little panning. There's a lot of work to be done. And frankly, the steps to go from this like gold to, or this resource to something valuable is significant. I think people have gotten a little carried away with the old maxim of like, don't go pan for gold, sell pickaxes and shovels. It's a much stronger business model. At this point, I feel like I look around and I see more pickaxe salesmen and shovel salesmen than I do prospectors. And that scares me a little bit. Metagame where people are starting to think about how they can build tools for people building tools for AI. And that starts to give me a little bit of like pause in terms of like, how confident are we that we can even extract this resource into something valuable? I got a text message from a VC earlier today, and I won't name the VC or the fund, but the question was, what are some medium or large size companies that have integrated AI into their platform in a way that you're really impressed by? And I looked at the text message for a few minutes and I was finding myself thinking and thinking, and I responded, maybe only co-pilot. It's been a couple hours now, and I don't think I've thought of another one. And I think that's where I reflect again on this, like iron versus gold. If it was really gold, I feel like I'd be more blown away by other AI integrations. And I'm not yet. [00:20:40]Alessio: I feel like all the people finding gold are the ones building things that traditionally we didn't focus on. So like mid-journey. I've talked to a company yesterday, which I'm not going to name, but they do agents for some use case, let's call it. They are 11 months old. They're making like 8 million a month in revenue, but in a space that you wouldn't even think about selling to. If you were like a shovel builder, you wouldn't even go sell to those people. And Swix talks about this a bunch, about like actually trying to go application first for some things. Let's actually see what people want to use and what works. What do you think are the most maybe underexplored areas in AI? Is there anything that you wish people were actually trying to shovel? [00:21:23]Bryan: I've been saying for a couple of months now, if I had unlimited resources and I was just sort of like truly like, you know, on my own building whatever I wanted, I think the thing that I'd be most excited about is building sort of like the personal Memex. The Memex is something that I've wanted since I was a kid. And are you familiar with the Memex? It's the memory extender. And it's this idea that sort of like human memory is quite weak. And so if we can extend that, then that's a big opportunity. So I think one of the things that I've always found to be one of the limiting cases here is access. How do you access that data? Even if you did build that data like out, how would you quickly access it? And one of the things I think there's a constellation of technologies that have come together in the last couple of years that now make this quite feasible. Like information retrieval has really improved and we have a lot more simple systems for getting started with information retrieval to natural language is ultimately the interface that you'd really like these systems to work on, both in terms of sort of like structuring the data and preparing the data, but also on the retrieval side. So what keys off the query for retrieval, probably ultimately natural language. And third, if you really want to go into like the purely futuristic aspect of this, it is latent voice to text. And that is also something that has quite recently become possible. I did talk to a company recently called gather, which seems to have some cool ideas in this direction, but I haven't seen yet what I, what I really want, which is I want something that is sort of like every time I listen to a podcast or I watch a movie or I read a book, it sort of like has a great vector index built on top of all that information that's contained within. And then when I'm having my next conversation and I can't quite remember the name of this person who did this amazing thing, for example, if we're talking about the Memex, it'd be really nice to have Vannevar Bush like pop up on my, you know, on my Memex display, because I always forget Vannevar Bush's name. This is one time that I didn't, but I often do. This is something that I think is only recently enabled and maybe we're still five years out before it can be good, but I think it's one of the most exciting projects that has become possible in the last three years that I think generally wasn't possible before. [00:23:46]Alessio: Would you wear one of those AI pendants that record everything? [00:23:50]Bryan: I think I'm just going to do it because I just like support the idea. I'm also admittedly someone who, when Google Glass first came out, thought that seems awesome. I know that there's like a lot of like challenges about the privacy aspect of it, but it is something that I did feel was like a disappointment to lose some of that technology. Fun fact, one of the early Google Glass developers was this MIT computer scientist who basically built the first wearable computer while he was at MIT. And he like took notes about all of his conversations in real time on his wearable and then he would have real time access to them. Ended up being kind of a scandal because he wanted to use a computer during his defense and they like tried to prevent him from doing it. So pretty interesting story. [00:24:35]Alessio: I don't know but the future is going to be weird. I can tell you that much. Talking about pickaxes, what do you think about the pickaxes that people built before? Like all the whole MLOps space, which has its own like startup graveyard in there. How are those products evolving? You know, you were at Wits and Biases before, which is now doing a big AI push as well. [00:24:57]Bryan: If you really want to like sort of like rub my face in it, you can go look at my white paper on MLOps from 2022. It's interesting. I don't think there's many things in that that I would these days think are like wrong or even sort of like naive. But what I would say is there are both a lot of analogies between MLOps and LLMops, but there are also a lot of like key differences. So like leading an engineering team at the moment, I think a lot more about good engineering practices than I do about good ML practices. That being said, it's been very convenient to be able to see around corners in a few of the like ML places. One of the first things I did at Hex was work on evals. This was in February. I hadn't yet been overwhelmed by people talking about evals until about May. And the reason that I was able to be a couple of months early on that is because I've been building evals for ML systems for years. I don't know how else to build an ML system other than start with the evals. I teach my students at Rutgers like objective framing is one of the most important steps in starting a new data science project. If you can't clearly state what your objective function is and you can't clearly state how that relates to the problem framing, you've got no hope. And I think that is a very shared reality with LLM applications. Coming back to one thing you mentioned from earlier about sort of like the applications of these LLMs. To that end, I think what pickaxes I think are still very valuable is understanding systems that are inherently less predictable, that are inherently sort of experimental. On my engineering team, we have an experimentalist. So one of the AI engineers, his focus is experiments. That's something that you wouldn't normally expect to see on an engineering team. But it's important on an AI engineering team to have one person whose entire focus is just experimenting, trying, okay, this is a hypothesis that we have about how the model will behave. Or this is a hypothesis we have about how we can improve the model's performance on this. And then going in, running experiments, augmenting our evals to test it, et cetera. What I really respect are pickaxes that recognize the hybrid nature of the sort of engineering tasks. They are ultimately engineering tasks with a flavor of ML. And so when systems respect that, I tend to have a very high opinion. One thing that I was very, very aligned with Weights and Biases on is sort of composability. These systems like ML systems need to be extremely composable to make them much more iterative. If you don't build these systems in composable ways, then your integration hell is just magnified. When you're trying to iterate as fast as people need to be iterating these days, I think integration hell is a tax not worth paying. [00:27:51]Alessio: Let's talk about some of the LLM native pickaxes, so to speak. So RAG is one. One thing is doing RAG on text data. One thing is doing RAG on tabular data. We're releasing tomorrow our episode with Kube, the semantic layer company. Curious to hear your thoughts on it. How are you doing RAG, pros, cons? [00:28:11]Bryan: It became pretty obvious to me almost immediately that RAG was going to be important. Because ultimately, you never expect your model to have access to all of the things necessary to respond to a user's request. So as an example, Magic users would like to write SQL that's relevant to their business. And it's important then to have the right data objects that they need to query. We can't expect any LLM to understand our user's data warehouse topology. So what we can expect is that we can build a RAG system that is data warehouse aware, data topology aware, and use that to provide really great information to the model. If you ask the model, how are my customers trending over time? And you ask it to write SQL to do that. What is it going to do? Well, ultimately, it's going to hallucinate the structure of that data warehouse that it needs to write a general query. Most likely what it's going to do is it's going to look in its sort of memory of Stack Overflow responses to customer queries, and it's going to say, oh, it's probably a customer stable and we're in the age of DBT, so it might be even called, you know, dim customers or something like that. And what's interesting is, and I encourage you to try, chatGBT will do an okay job of like hallucinating up some tables. It might even hallucinate up some columns. But what it won't do is it won't understand the joins in that data warehouse that it needs, and it won't understand the data caveats or the sort of where clauses that need to be there. And so how do you get it to understand those things? Well, this is textbook RAG. This is the exact kind of thing that you expect RAG to be good at augmenting. But I think where people who have done a lot of thinking about RAG for the document case, they think of it as chunking and sort of like the MapReduce and the sort of like these approaches. But I think people haven't followed this train of thought quite far enough yet. Jerry Liu was on the show and he talked a little bit about thinking of this as like information retrieval. And I would push that even further. And I would say that ultimately RAG is just RecSys for LLM. As I kind of already mentioned, I'm a little bit recommendation systems heavy. And so from the beginning, RAG has always felt like RecSys to me. It has always felt like you're building a recommendation system. And what are you trying to recommend? The best possible resources for the LLM to execute on a task. And so most of my approach to RAG and the way that we've improved magic via retrieval is by building a recommendation system. [00:30:49]Alessio: It's funny, as you mentioned that you spent three years writing the book, the O'Reilly book. Things must have changed as you wrote the book. I don't want to bring out any nightmares from there, but what are the tips for people who want to stay on top of this stuff? Do you have any other favorite newsletters, like Twitter accounts that you follow, communities you spend time in? [00:31:10]Bryan: I am sort of an aggressive reader of technical books. I think I'm almost never disappointed by time that I've invested in reading technical manuscripts. I find that most people write O'Reilly or similar books because they've sort of got this itch that they need to scratch, which is that I have some ideas, I have some understanding that we're hard won, I need to tell other people. And there's something that, from my experience, correlates between that itch and sort of like useful information. As an example, one of the people on my team, his name is Will Kurt, he wrote a book sort of Bayesian statistics the fun way. I knew some Bayesian statistics, but I read his book anyway. And the reason was because I was like, if someone feels motivated to write a book called Bayesian statistics the fun way, they've got something to say about Bayesian statistics. I learned so much from that book. That book is like technically like targeted at someone with less knowledge and experience than me. And boy, did it humble me about my understanding of Bayesian statistics. And so I think this is a very boring answer, but ultimately like I read a lot of books and I think that they're a really valuable way to learn these things. I also regrettably still read a lot of Twitter. There is plenty of noise in that signal, but ultimately it is still usually like one of the first directions to get sort of an instinct for what's valuable. The other comment that I want to make is we are in this age of sort of like archive is becoming more of like an ad platform. I think that's a little challenging right now to kind of use it the way that I used to use it, which is for like higher signal. I've chatted a lot with a CMU professor, Graham Neubig, and he's been doing LLM evaluation and LLM enhancements for about five years and know that I didn't misspeak. And I think talking to him has provided me a lot of like directionality for more believable sources. Trying to cut through the hype. I know that there's a lot of other things that I could mention in terms of like just channels, but ultimately right now I think there's almost an abundance of channels and I'm a little bit more keen on high signal. [00:33:18]Alessio: The other side of it is like, I see so many people say, Oh, I just wrote a paper on X and it's like an article. And I'm like, an article is not a paper, but it's just funny how I know we were kind of chatting before about terms being reinvented and like people that are not from this space kind of getting into AI engineering now. [00:33:36]Bryan: I also don't want to be gatekeepy. Actually I used to say a lot to people, don't be shy about putting your ideas down on paper. I think it's okay to just like kind of go for it. And I, I myself have something on archive that is like comically naive. It's intentionally naive. Right now I'm less concerned by more naive approaches to things than I am by the purely like advertising approach to sort of writing these short notes and articles. I think blogging still has a good place. And I remember getting feedback during my PhD thesis that like my thesis sounded more like a long blog post. And I now feel like that curmudgeonly professor who's also like, yeah, maybe just keep this to the blogs. That's funny.Alessio: Uh, yeah, I think one of the things that Swyx said when he was opening the AI engineer summit a couple of weeks ago was like, look, most people here don't know much about the space because it's so new and like being open and welcoming. I think it's one of the goals. And that's why we try and keep every episode at a level that it's like, you know, the experts can understand and learn something, but also the novices can kind of like follow along. You mentioned evals before. I think that's one of the hottest topics obviously out there right now. What are evals? How do we know if they work? Yeah. What are some of the fun learnings from building them into X? [00:34:53]Bryan: I said something at the AI engineer summit that I think a few people have already called out, which is like, if you can't get your evals to be sort of like objective, then you're not trying hard enough. I stand by that statement. I'm not going to, I'm not going to walk it back. I know that that doesn't feel super good because people, people want to think that like their unique snowflake of a problem is too nuanced. But I think this is actually one area where, you know, in this dichotomy of like, who can do AI engineering? And the answer is kind of everybody. Software engineering can become AI engineering and ML engineering can become AI engineering. One thing that I think the more data science minded folk have an advantage here is we've gotten more practice in taking very vague notions and trying to put a like objective function around that. And so ultimately I would just encourage everybody who wants to build evals, just work incredibly hard on codifying what is good and bad in terms of these objective metrics. As far as like how you go about turning those into evals, I think it's kind of like sweat equity. Unfortunately, I told the CEO of gantry several months ago, I think it's been like six months now that I was sort of like looking at every single internal Hex request to magic by hand with my eyes and sort of like thinking, how can I turn this into an eval? Is there a way that I can take this real request during this dog foodie, not very developed stage? How can I make that into an evaluation? That was a lot of sweat equity that I put in a lot of like boring evenings, but I do think ultimately it gave me a lot of understanding for the way that the model was misbehaving. Another thing is how can you start to understand these misbehaviors as like auxiliary evaluation metrics? So there's not just one evaluation that you want to do for every request. It's easy to say like, did this work? Did this not work? Did the response satisfy the task? But there's a lot of other metrics that you can pull off these questions. And so like, let me give you an example. If it writes SQL that doesn't reference a table in the database that it's supposed to be querying against, we would think of that as a hallucination. You could separately consider, is it a hallucination as a valuable metric? You could separately consider, does it get the right answer? The right answer is this sort of like all in one shot, like evaluation that I think people jump to. But these intermediary steps are really important. I remember hearing that GitHub had thousands of lines of post-processing code around Copilot to make sure that their responses were sort of correct or in the right place. And that kind of sort of defensive programming against bad responses is the kind of thing that you can build by looking at many different types of evaluation metrics. Because you can say like, oh, you know, the Copilot completion here is mostly right, but it doesn't close the brace. Well, that's the thing you can check for. Or, oh, this completion is quite good, but it defines a variable that was like already defined in the file. Like that's going to have a problem. That's an evaluation that you could check separately. And so this is where I think it's easy to convince yourself that all that matters is does it get the right answer? But the more that you think about production use cases of these things, the more you find a lot of this kind of stuff. One simple example is like sometimes the model names the output of a cell, a variable that's already in scope. Okay. Like we can just detect that and like we can just fix that. And this is the kind of thing that like evaluations over time and as you build these evaluations over time, you really can expand the robustness in which you trust these models. And for a company like Hex, who we need to put this stuff in GA, we can't just sort of like get to demo stage or even like private beta stage. We really hunting GA on all of these capabilities. Did it get the right answer on some cases is not good enough. [00:38:57]Alessio: I think the follow up question to that is in your past roles, you own the model that you're evaluating against. Here you don't actually have control into how the model evolves. How do you think about the model will just need to improve or we'll use another model versus like we can build kind of like engineering post-processing on top of it. How do you make the choice? [00:39:19]Bryan: So I want to say two things here. One like Jerry Liu talked a little bit about in his episode, he talked a little bit about sort of like you don't always want to retrain the weights to serve certain use cases. Rag is another tool that you can use to kind of like soft tune. I think that's right. And I want to go back to my favorite analogy here, which is like recommendation systems. When you build a recommendation system, you build the objective function. You think about like what kind of recs you want to provide, what kind of features you're allowed to use, et cetera, et cetera. But there's always another step. There's this really wonderful collection of blog posts from Eugene Yon and then ultimately like even Oldridge kind of like iterated on that for the Merlin project where there's this multi-stage recommender. And the multi-stage recommender says the first step is to do great retrieval. Once you've done great retrieval, you then need to do great ranking. Once you've done great ranking, you need to then do a good job serving. And so what's the analogy here? Rag is retrieval. You can build different embedding models to encode different features in your latent space to ensure that your ranking model has the best opportunity. Now you might say, oh, well, my ranking model is something that I've got a lot of capability to adjust. I've got full access to my ranking model. I'm going to retrain it. And that's great. And you should. And over time you will. But there's one more step and that's downstream and that's the serving. Serving often sounds like I just show the s**t to the user, but ultimately serving is things like, did I provide diverse recommendations? Going back to Stitch Fix days, I can't just recommend them five shirts of the same silhouette and cut. I need to serve them a diversity of recommendations. Have I respected their requirements? They clicked on something that got them to this place. Is the recommendations relevant to that query? Are there any hard rules? Do we maybe not have this in stock? These are all things that you put downstream. And so much like the recommendations use case, there's a lot of knobs to pull outside of retraining the model. And even in recommendation systems, when do you retrain your model for ranking? Not nearly as much as you do other s**t. And even this like embedding model, you might fiddle with more often than the true ranking model. And so I think the only piece of the puzzle that you don't have access to in the LLM case is that sort of like middle step. That's okay. We've got plenty of other work to do. So right now I feel pretty enabled. [00:41:56]Alessio: That's great. You obviously wrote a book on RecSys. What are some of the key concepts that maybe people that don't have a data science background, ML background should keep in mind as they work in this area? [00:42:07]Bryan: It's easy to first think these models are stochastic. They're unpredictable. Oh, well, what are we going to do? I think of this almost like gaseous type question of like, if you've got this entropy, where can you put the entropy? Where can you let it be entropic and where can you constrain it? And so what I want to say here is think about the cases where you need it to be really tightly constrained. So why are people so excited about function calling? Because function calling feels like a way to constrict it. Where can you let it be more gaseous? Well, maybe in the way that it talks about what it wants to do. Maybe for planning, if you're building agents and you want to do sort of something chain of thoughty. Well, that's a place where the entropy can happily live. When you're building applications of these models, I think it's really important as part of the problem framing to be super clear upfront. These are the things that can be entropic. These are the things that cannot be. These are the things that need to be super rigid and really, really aligned to a particular schema. We've had a lot of success in making specific the parts that need to be precise and tightly schemified, and that has really paid dividends. And so other analogies from data science that I think are very valuable is there's the sort of like human in the loop analogy, which has been around for quite a while. And I have gone on record a couple of times saying that like, I don't really love human in the loop. One of the things that I think we can learn from human in the loop is that the user is the best judge of what is good. And the user is pretty motivated to sort of like interact and give you kind of like additional nudges in the direction that you want. I think what I'd like to flip though, is instead of human in the loop, I'd like it to be AI in the loop. I'd rather center the user. I'd rather keep the user as the like core item at the center of this universe. And the AI is a tool. By switching that analogy a little bit, what it allows you to do is think about where are the places in which the user can reach for this as a tool, execute some task with this tool, and then go back to doing their workflow. It still gets this back and forth between things that computers are good at and things that humans are good at, which has been valuable in the human loop paradigm. But it allows us to be a little bit more, I would say, like the designers talk about like user-centered. And I think that's really powerful for AI applications. And it's one of the things that I've been trying really hard with Magic to make that feel like the workflow as the AI is right there. It's right where you're doing your work. It's ready for you anytime you need it. But ultimately you're in charge at all times and your workflow is what we care the most about. [00:44:56]Alessio: Awesome. Let's jump into lightning round. What's something that is not on your LinkedIn that you're passionate about or, you know, what's something you would give a TED talk on that is not work related? [00:45:05]Bryan: So I walk a lot. [00:45:07]Bryan: I have walked every road in Berkeley. And I mean like every part of every road even, not just like the binary question of, have you been on this road? I have this little app that I use called Wanderer, which just lets me like kind of keep track of everywhere I've been. And so I'm like a little bit obsessed. My wife would say a lot a bit obsessed with like what I call new roads. I'm actually more motivated by trails even than roads, but like I'm a maximalist. So kind of like everything and anything. Yeah. Believe it or not, I was even like in the like local Berkeley paper just talking about walking every road. So yeah, that's something that I'm like surprisingly passionate about. [00:45:45]Alessio: Is there a most underrated road in Berkeley? [00:45:49]Bryan: What I would say is like underrated is Kensington. So Kensington is like a little town just a teeny bit north of Berkeley, but still in the Berkeley hills. And Kensington is so quirky and beautiful. And it's a really like, you know, don't sleep on Kensington. That being said, one of my original motivations for doing all this walking was people always tell me like, Berkeley's so quirky. And I was like, how quirky is Berkeley? Turn it out. It's quite, quite quirky. It's also hard to say quirky and Berkeley in the same sentence I've learned as of now. [00:46:20]Alessio: That's a, that's a good podcast warmup for our next guests. All right. The actual lightning ground. So we usually have three questions, acceleration, exploration, then a takeaway acceleration. What's, what's something that's already here today that you thought would take much longer to arrive in AI and machine learning? [00:46:39]Bryan: So I invited the CEO of Hugging Face to my seminar when I worked at Stitch Fix and his talk at the time, honestly, like really annoyed me. The talk was titled like something to the effect of like LLMs are going to be the like technology advancement of the next decade. It's on YouTube. You can find it. I don't remember exactly the title, but regardless, it was something like LLMs for the next decade. And I was like, okay, they're like one modality of model, like whatever. His talk was fine. Like, I don't think it was like particularly amazing or particularly poor, but what I will say is damn, he was right. Like I, I don't think I quite was on board during that talk where I was like, ah, maybe, you know, like there's a lot of other modalities that are like moving pretty quick. I thought things like RL were going to be the like real like breakout success. And there's a little pun with Atari and breakout there, but yeah, like I, man, I was sleeping on LLMs and I feel a little embarrassed. I, yeah. [00:47:44]Alessio: Yeah. No, I mean, that's a good point. It's like sometimes the, we just had Jeremy Howard on the podcast and he was saying when he was talking about fine tuning, everybody thought it was dumb, you know, and then later people realize, and there's something to be said about messaging, especially like in technical audiences where there's kind of like the metagame, you know, which is like, oh, these are like the cool ideas people are exploring. I don't know where I want to align myself yet, you know, or whatnot. So it's cool exploration. So it's kind of like the opposite of that. You mentioned RL, right? That's something that was kind of like up and up and up. And then now it's people are like, oh, I don't know. Are there any other areas if you weren't working on, on magic that you want to go work on? [00:48:25]Bryan: Well, I did mention that, like, I think this like Memex product is just like incredibly exciting to me. And I think it's really opportunistic. I think it's very, very feasible, but I would maybe even extend that a little bit, which is I don't see enough people getting really enthusiastic about hardware with advanced AI built in. You're hearing whispering of it here and there, put on the whisper, but like you're starting to see people putting whisper into pieces of hardware and making that really powerful. I joked with, I can't think of her name. Oh, Sasha, who I know is a friend of the pod. Like I joked with Sasha that I wanted to make the big mouth Billy Bass as a babble fish, because at this point it's pretty easy to connect that up to whisper and talk to it in one language and have it talk in the other language. And I was like, this is the kind of s**t I want people building is like silly integrations between hardware and these new capabilities. And as much as I'm starting to hear whisperings here and there, it's not enough. I think I want to see more people going down this track because I think ultimately like these things need to be in our like physical space. And even though the margins are good on software, I want to see more like integration into my daily life. Awesome. [00:49:47]Alessio: And then, yeah, a takeaway, what's one message idea you want everyone to remember and think about? [00:49:54]Bryan: Even though earlier I was talking about sort of like, maybe like not reinventing things and being respectful of the sort of like ML and data science, like ideas. I do want to say that I think everybody should be experimenting with these tools as much as they possibly can. I've heard a lot of professors, frankly, express concern about their students using GPT to do their homework. And I took a completely opposite approach, which is in the first 15 minutes of the first class of my semester this year, I brought up GPT on screen and we talked about what GPT was good at. And we talked about like how the students can sort of like use it. I showed them an example of it doing data analysis work quite well. And then I showed them an example of it doing quite poorly. I think however much you're integrating with these tools or interacting with these tools, and this audience is probably going to be pretty high on that distribution. I would really encourage you to sort of like push this into the other people in your life. My wife is very technical. She's a product manager and she's using chat GPT almost every day for communication or for understanding concepts that are like outside of her sphere of excellence. And recently my mom and my sister have been sort of like onboarded onto the chat GPT train. And so ultimately I just, I think that like it is our duty to help other people see like how much of a paradigm shift this is. We should really be preparing people for what life is going to be like when these are everywhere. [00:51:25]Alessio: Awesome. Thank you so much for coming on, Bryan. This was fun. [00:51:29]Bryan: Yeah. Thanks for having me. And use Hex magic. [00:51:31] Get full access to Latent Space at www.latent.space/subscribe

Cale Birk is a former teacher, high school principal, and District Head of Innovation from British Columbia, Canada. In 2015, one of his schools was named one of the first Model PLC Schools in Canada. Feeling like he was only scratching the surface with collaboration, Cale wrote the book PLC 2.0 – Collaborating for Observable […]

Observable Radio is a found footage anthology podcast of retro sci-fi and analog horror from Cameron Suey and Phil van Hest. When he discovers something beneath the static of the worlds's communication network, an unnamed Observer begins to catalog and record the strange signals that should not exist… Link: https://www.observableradio.com/ RSS Feed: https://feeds.acast.com/public/shows/observableradio

Cam shows up, mysteriously and suspiciously alone, with an update about his new show, Observable Radio:When he discovers something beneath the static of the worlds's communication network, an unnamed observer begins to catalog and record the strange signals that should not exist…Observable Radio is a found footage anthology podcast from Cameron Suey and Phil van Hest.Art by Karrin FletcherFind us online:observableradio.comListen and Subscribe:APPLESPOTIFYGOOGLEFollow:INSTAGRAMBLUESKYTWITTERObservable Radio is listener supported. If you would like to contribute towards our production costs and payment for our voice actors, you can do so at:patreon.com/observableradioko-fi.com/observableradioThe following music was used for this media project: Music: The Empire by Alexander NakaradaFree downloadLicense (CC BY 4.0)Artist websiteThe Backrooms by Myuu(CC BY 3.0) Drowning Monas by Tim Kulig(Free download)(CC BY 4.0)Ain't She Sweet, The Opener, Jada, and Let's Dance by Hansag Big BandLicense (CC BY 3.0) Bullets and Bayonets by the United States Marine BandPublic Domain Sound Effects courtesy of the artists at Freesound.org, including:amliebschERHhenrique85nHybrid_VInspectorJLeoctiursmorgantjnaught101newagesoupPatrickLieberkindPorphyrTetrisrockerthatjeffcarterZabuhailoAdditional sound effects from zapsplat.comSFX covered under the following licenses:creativecommons.org/licenses/by/3.0/creativecommons.org/licenses/by/4.0/Chat with us on Twitter!

Join AWS SA Mahesh Biradar as he discusses Observability in the Cloud. Observability is the capability to continuously generate and discover actionable insights based on signals from the system under observation. In other words, it allows users to understand a system's state from its external output and take (corrective) action. The observability of a system has a significant impact on its operating and development costs. Observable systems yield meaningful, actionable data to their operators, allowing them to achieve favorable outcomes (faster incident response, increased developer productivity) and less toil and downtime.AWS Hosts: Nolan Chen & Malini Chatterjee

Welcome to Hunters and Unicorns: The Playbook Universe. Today we welcome Dan Miller! Key takeaways from this episode are: • What it took to close a deal worth just short of $100m • Why he never missed his number • Why allies and mentors in GTM community are essential Dan Miller is a GTM Advisor at Loft Lab sand Observable, focusing on revenue strategies. In this Hunters and Unicorns podcast, Dan reflects on his career-defining moments at prestigious companies such as Splunk, Nimble Storage, Sumo Logic and SignalFX. He shares his experience when transitioning from an IC to a Leadership role and the challenges he faced. Dan is a force of nature within the playbook space. His list of achievements is extensive and includes; • Exceeding 100% every year • Closed the larges deal in Splunk's history • Exposed to the playbooks of both John McMahon and Mark Cranny We loved understanding more from Dan about identifying early-stage companies and the right strategies needed for implementation. How can champions really be identified consistently? What are the real impactful metrics when it comes to sales? Dan shares with us his insight pertaining to sales as a science as opposed to an art. He also shares the inspirational story of Mark Cranney and his journey including competing with the Bladelogic team at Opsware and driving a $1.6 billion sale to HP. Dan gives us real insight to the SignalFX story! This episode is not to be missed!

Questions covered in this episode: 1:25 Opening Question: Can you tell me a little bit about yourself and your path into product marketing?6:32 How you handle release marketing at Observable and how you think about organizing and planning for releases at your current company??11:09 how did you work with leadership and your product team to kind of get that alignment initially upfront?13:27 How you've thought about this and how you've even empowered product marketers on your team at observable?18:37 How have you thought about as a manager, as a leader, how do you communicate that kind of across the org and up and down the org respectively as well? 22:43 How you've kind of advised pmms on your teams respectively to address some areas that you've seen launches go wrong?28:21 Can you share a little bit more about your journey, how you thought about it and how you actually took that or ather made the jump to being a PMM leader?33:34 Can you share one piece of insight that has served you well throughout your product marketing career?Want more insights from LaShaun? Check out her Sharebird Profile.Looking to connect? You can find LaShaun here on LinkedIn.

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Announcing Squiggle Hub, published by Ozzie Gooen on August 5, 2023 on The Effective Altruism Forum. Overview Squiggle Hub is a platform for the creation and sharing of code written in Squiggle. As with Squiggle, Squiggle Hub is free and open-source. As a refresher, Squiggle is a simple programming language for probabilistic estimation that runs on Javascript. It begins with the syntax of Guesstimate, but generally adds a lot more functionality. See its launch post here for more information, or the website for the full documentation. Squiggle Hub is a lot like a more powerful, but less visual, version of Guesstimate. We hope that it will eventually be much more valuable than Guesstimate is now. If you can use Guesstimate, you can basically use Squiggle. If you already use Guesstimate, try using the same syntax in Squiggle. It should mostly work. All models on Squiggle Hub are public. We've produced several small ones so far, and a few friends have written some as well. We're looking forward to seeing what others make! Looking for Squiggle examples? We've organized some in the docs. The Squiggle EA Forum Tag also has an updating list. Key Links Squiggle Hub Squiggle Discord The Squiggle Language Homepage Squiggle Newsletter (Part of the QURI Newsletter) Functionality Squiggle (the language) Write functions that accept and return probability distributions. Squiggle generates automatic plots for these. You can provide explicit ranges for functions. This helps with the visualization, and ensures they won't be called outside that range. Like, appleStockPrice(t:[2023, 2060]). Custom plots for distributions and functions. Scales include linear, log, and symlog (like log, but with support for negative values). Symlog scales accept a parameter constant that you can use for adjusting the scale. Make custom tables of any data and functions, with Table.make({data, columns}). Automatic conversion of Monte Carlo samples to distribution plots, using KDE. In the cases where the distribution is heavily skewed, Squiggle does this with a log transformation. The result of this is often more accurate than using histograms. Combined with custom scales, Squiggle much better supports highly skewed distributions (i.e. "5 to 5M") than Guesstimate does. Squiggle supports most of JSON. You can copy & paste JSON data and begin using it in Squiggle. Squiggle runs on Javascript. You can simply take Squiggle code and run it in your website, Observable, Obsidian, and more.[1] If you are making an application that uses probability distributions, you can use the Squiggle components directly. Lots of performance enhancements, library additions, and bug fixes, since Squiggle's initial release. All the docs and grammar are consolidated here. You can use this to feed into Claude, for some Squiggle generation and assistance. Squiggle Editor (The window on the left) With "Autorun", the output will update as you type. Turn this off if you want to run it manually. Adjust the number of Monte Carlo samples. It defaults to 1000, but you can change this in the settings. Also, there are a bunch of other toggles there to play with. A built-in Squiggle code formatter. Useful for lengthy files and for storing data. Function and variable autocomplete. Squiggle Viewer (The window on the right) Click the arrows to open and close visualizations. Select any variable in the right view to "zoom in" on it. Very handy when working on particular estimates of a file! On hover, there's a "show in editor" button, that finds the source code to the variable in question. If your Squiggle file ends with a variable, this will be shown as the "Result." You'll still be able to see all of the other defined variables, but these will be collapsed when you open the model. Useful for selecting a few key findings for display purposes. You can...

I answer a listener question about how to address school boards and city councils; I discuss the CON-INC establishment, Chris Ruffo and how what they say and don't say give them away as being owned and maintaining the status quo.  Observable jab stories too.

This week Tim sits down with Terry Latanville, fellow goalie, Staff iOS Engineer at DoorDash. Terry will be speaking at Swift TO Conference in August 2023. They discuss some innovations, such as Reactive VIP, Observable, Unity and Smurf Berries.https://latanville.ca/https://twitter.com/terrylatanvilleTerry Latanville (@RndmTsk@mastodon.social)Wayne Gretsky 100 yard dashHow to Speed Up SwiftUI Development and Testing Using PreviewSnapshotsAdopting SwiftUI with a Bottom-Up Approach to Minimize RiskSwift TO 2023 Become a member at https://plus.acast.com/s/mtjc. Hosted on Acast. See acast.com/privacy for more information.

What if you could only describe yourself using non-observable descriptions? You would fall IN LOVE with yourself

Peter's talking about an imaginary town, Mikey's engaging in some local customs, and Ben's seeing how many dicks can fit in things. Donate £3 or more to join next episode's Pod Squad: http://podiots.com And check our website and store: http://vidiotsofficial.com -------------------   Subscribe for more and TELL YOUR FRIENDS!   Support Ben and Peter: https://www.youtube.com/teamtriplejump   YouTube: https://youtube.com/vidiotsofficial Podiots: https://vidiotsofficial.com Pod Squad: https://podiots.com Shop: https://vidiotsofficial.com/shop Twitch: https://twitch.tv/vidiotsofficial Twitter: https://twitter.com/VidiotsOfficial Facebook: https://facebook.com/vidiotsofficial Discord: https://vidiotsofficial.com/discord/ Site: https://vidiotsofficial.com/ Learn more about your ad choices. Visit megaphone.fm/adchoices

Analizamos el evento de la WWDC bajo el prisma de los desarrolladores, comentando algunas de las novedades más importantes como Swift Data, las macros de Swift o el nuevo protocolo Observable que cambia por completo la arquitectura de funcionamiento en SwiftUI. Ha sido una WWDC de perfil bajo en novedades para los usuarios, ya que el margen de mejora de los sistemas operativos es menor, pero muy alto para los desarrolladores que han visto cumplidos algunos de sus sueños más recurrentes este año. Además de todas las novedades de desarrollo para la nueva plataforma visionOS que son una maravilla. Episodio con la colaboración de BP. Infórmate en bp.es. Promoción válida hasta el 30 de junio. Descubre nuestro canal de Twitch en: twitch.tv/applecoding. Descubre nuestras ofertas para oyentes: - Cursos en Udemy (con código de oferta) - Apple Coding Academy - Suscríbete a Apple Coding en nuestro Patreon. - Canal de Telegram de Swift. Acceso al canal. --------------- Consigue las camisetas oficiales de Apple Coding con los logos de Swift y Apple Coding así como todo tipo de merchadising como tazas o fundas. - Tienda de merchandising de Apple Coding. --------------- Tema musical: "For the Win" de "Two Steps from Hell", compuesto por Thomas Bergensen. Usado con permisos de fair use. Escúchalo en Apple Music o Spotify.

APIs are powering and empowering software innovation as they enable new use cases on top of existing services. Observability into API usage to answer questions like: how APIs are called, what APIs do, where APIs fail, where APIs are slow, where APIs are misused … has to be on top of mind for architects that decide to build or use APIs.In this episode we welcome Sonja Chevre, Group Product Manager at Tyk, who recently gave a captivating talk at KubeCon about using OpenTelemetry to get insights into popular API frameworks such as GraphQL. We are discussing common challenges for SREs such as that APIs often hide the status of a call behind an HTTP 200 or that debugging individual calls is really hard as details of the call are not exposed by default to telemetry data. We also cover topics such as API-led growth, API as a product as well as open standards such as OpenTelemetry and OpenAPI. Here the list of discussed links during the show:KubeCon Talk: https://kccnceu2023.sched.com/event/1HyVc/what-could-go-wrong-with-a-graphql-query-and-can-opentelemetry-help-sonja-chevre-ahmet-soormally-tyk-technologiesAPI Management vs Gateway discussion: https://www.linkedin.com/posts/sonjachevre_apimanagement-apigateway-apisecurity-activity-7061596404854521857-N_cO/API as a Product: https://tyk.io/blog/unlocking-the-potential-of-api-as-a-product/OpenAPI: https://www.openapis.org/OpenTelemetry: https://opentelemetry.io/

For this episode of the eCom Logistics Podcast, we welcome Kevin Lawton, Founder of The New Warehouse. Today, he talks about the innovations that are starting to take over warehousing operations today.Kevin shares how he started his podcast and how the warehousing space has changed over time with the emergence of technology. From AI and computer vision being utilized in training to more practical applications on the job. He also discusses how he got into the micro-fulfillment space and what he expects moving forward. ABOUT KEVINKevin has been involved in the distribution industry for 11 years, primarily focusing on inventory control. He started The New Warehouse Podcast in 2019 with over 350 episodes published. He is also an adjunct professor in supply chain at Rider University.He has grown The New Warehouse into a 3PL services business with a recently opened Micro-fulfillment center in Philadelphia. HIGHLIGHTS02:28 How Kevin started The New Warehouse podcast11:22 Understanding demographic replacement today15:55 Using technology for warehouse training and operations26:52 What excites Kevin about the micro-fulfillment space35:45 Observable changes in micro-fulfillment over the years QUOTES21:10 Using virtual reality for warehouse training - Kevin: "I love the training stuff just with the safety aspect overall. Training in the warehouse should be the same every time. I think programs like that allow you to do that same type of training every time.24:07 Any tools that help understand what's happening in the warehouse are extremely impactful - Kevin: "When you think about reducing staff by bringing in automation and robotics and just having member shortages in general with less people wanting to work in warehouses. Think about 4-5 managers having to cover half a million square feet, you can't be everywhere at the same time. Find out more about Kevin in the links below:LinkedIn: https://www.linkedin.com/in/kevinclawton/Website: https://www.thenewwarehouse.com/

Thank you for joining us for our Sunday morning gathering with our special guest pastor, Paul Cook.Our vision is to build a church that is “Encountering Jesus. Empowering People. To change Everything.”  Thank you so much for watching our videos.      If you'd like to tithe or give, please do so at our website. - LifeChapelToledo.com