Podcasts about so google

If you're in SF, join us tomorrow for a fun meetup at CodeGen Night!If you're in NYC, join us for AI Engineer Summit! The Agent Engineering track is now sold out, but 25 tickets remain for AI Leadership and 5 tickets for the workshops. You can see the full schedule of speakers and workshops at https://ai.engineer!It's exceedingly hard to introduce someone like Bret Taylor. We could recite his Wikipedia page, or his extensive work history through Silicon Valley's greatest companies, but everyone else already does that.As a podcast by AI engineers for AI engineers, we had the opportunity to do something a little different. We wanted to dig into what Bret sees from his vantage point at the top of our industry for the last 2 decades, and how that explains the rise of the AI Architect at Sierra, the leading conversational AI/CX platform.“Across our customer base, we are seeing a new role emerge - the role of the AI architect. These leaders are responsible for helping define, manage and evolve their company's AI agent over time. They come from a variety of both technical and business backgrounds, and we think that every company will have one or many AI architects managing their AI agent and related experience.”In our conversation, Bret Taylor confirms the Paul Buchheit legend that he rewrote Google Maps in a weekend, armed with only the help of a then-nascent Google Closure Compiler and no other modern tooling. But what we find remarkable is that he was the PM of Maps, not an engineer, though of course he still identifies as one. We find this theme recurring throughout Bret's career and worldview. We think it is plain as day that AI leadership will have to be hands-on and technical, especially when the ground is shifting as quickly as it is today:“There's a lot of power in combining product and engineering into as few people as possible… few great things have been created by committee.”“If engineering is an order taking organization for product you can sometimes make meaningful things, but rarely will you create extremely well crafted breakthrough products. Those tend to be small teams who deeply understand the customer need that they're solving, who have a maniacal focus on outcomes.”“And I think the reason why is if you look at like software as a service five years ago, maybe you can have a separation of product and engineering because most software as a service created five years ago. I wouldn't say there's like a lot of technological breakthroughs required for most business applications. And if you're making expense reporting software or whatever, it's useful… You kind of know how databases work, how to build auto scaling with your AWS cluster, whatever, you know, it's just, you're just applying best practices to yet another problem. "When you have areas like the early days of mobile development or the early days of interactive web applications, which I think Google Maps and Gmail represent, or now AI agents, you're in this constant conversation with what the requirements of your customers and stakeholders are and all the different people interacting with it and the capabilities of the technology. And it's almost impossible to specify the requirements of a product when you're not sure of the limitations of the technology itself.”This is the first time the difference between technical leadership for “normal” software and for “AI” software was articulated this clearly for us, and we'll be thinking a lot about this going forward. We left a lot of nuggets in the conversation, so we hope you'll just dive in with us (and thank Bret for joining the pod!)Timestamps* 00:00:02 Introductions and Bret Taylor's background* 00:01:23 Bret's experience at Stanford and the dot-com era* 00:04:04 The story of rewriting Google Maps backend* 00:11:06 Early days of interactive web applications at Google* 00:15:26 Discussion on product management and engineering roles* 00:21:00 AI and the future of software development* 00:26:42 Bret's approach to identifying customer needs and building AI companies* 00:32:09 The evolution of business models in the AI era* 00:41:00 The future of programming languages and software development* 00:49:38 Challenges in precisely communicating human intent to machines* 00:56:44 Discussion on Artificial General Intelligence (AGI) and its impact* 01:08:51 The future of agent-to-agent communication* 01:14:03 Bret's involvement in the OpenAI leadership crisis* 01:22:11 OpenAI's relationship with Microsoft* 01:23:23 OpenAI's mission and priorities* 01:27:40 Bret's guiding principles for career choices* 01:29:12 Brief discussion on pasta-making* 01:30:47 How Bret keeps up with AI developments* 01:32:15 Exciting research directions in AI* 01:35:19 Closing remarks and hiring at Sierra Transcript[00:02:05] Introduction and Guest Welcome[00:02:05] Alessio: Hey everyone, welcome to the Latent Space Podcast. This is Alessio, partner and CTO at Decibel Partners, and I'm joined by my co host swyx, founder of smol.ai.[00:02:17] swyx: Hey, and today we're super excited to have Bret Taylor join us. Welcome. Thanks for having me. It's a little unreal to have you in the studio.[00:02:25] swyx: I've read about you so much over the years, like even before. Open AI effectively. I mean, I use Google Maps to get here. So like, thank you for everything that you've done. Like, like your story history, like, you know, I think people can find out what your greatest hits have been.[00:02:40] Bret Taylor's Early Career and Education[00:02:40] swyx: How do you usually like to introduce yourself when, you know, you talk about, you summarize your career, like, how do you look at yourself?[00:02:47] Bret: Yeah, it's a great question. You know, we, before we went on the mics here, we're talking about the audience for this podcast being more engineering. And I do think depending on the audience, I'll introduce myself differently because I've had a lot of [00:03:00] corporate and board roles. I probably self identify as an engineer more than anything else though.[00:03:04] Bret: So even when I was. Salesforce, I was coding on the weekends. So I think of myself as an engineer and then all the roles that I do in my career sort of start with that just because I do feel like engineering is sort of a mindset and how I approach most of my life. So I'm an engineer first and that's how I describe myself.[00:03:24] Bret: You majored in computer[00:03:25] swyx: science, like 1998. And, and I was high[00:03:28] Bret: school, actually my, my college degree was Oh, two undergrad. Oh, three masters. Right. That old.[00:03:33] swyx: Yeah. I mean, no, I was going, I was going like 1998 to 2003, but like engineering wasn't as, wasn't a thing back then. Like we didn't have the title of senior engineer, you know, kind of like, it was just.[00:03:44] swyx: You were a programmer, you were a developer, maybe. What was it like in Stanford? Like, what was that feeling like? You know, was it, were you feeling like on the cusp of a great computer revolution? Or was it just like a niche, you know, interest at the time?[00:03:57] Stanford and the Dot-Com Bubble[00:03:57] Bret: Well, I was at Stanford, as you said, from 1998 to [00:04:00] 2002.[00:04:02] Bret: 1998 was near the peak of the dot com bubble. So. This is back in the day where most people that they're coding in the computer lab, just because there was these sun microsystems, Unix boxes there that most of us had to do our assignments on. And every single day there was a. com like buying pizza for everybody.[00:04:20] Bret: I didn't have to like, I got. Free food, like my first two years of university and then the dot com bubble burst in the middle of my college career. And so by the end there was like tumbleweed going to the job fair, you know, it was like, cause it was hard to describe unless you were there at the time, the like level of hype and being a computer science major at Stanford was like, A thousand opportunities.[00:04:45] Bret: And then, and then when I left, it was like Microsoft, IBM.[00:04:49] Joining Google and Early Projects[00:04:49] Bret: And then the two startups that I applied to were VMware and Google. And I ended up going to Google in large part because a woman named Marissa Meyer, who had been a teaching [00:05:00] assistant when I was, what was called a section leader, which was like a junior teaching assistant kind of for one of the big interest.[00:05:05] Bret: Yes. Classes. She had gone there. And she was recruiting me and I knew her and it was sort of felt safe, you know, like, I don't know. I thought about it much, but it turned out to be a real blessing. I realized like, you know, you always want to think you'd pick Google if given the option, but no one knew at the time.[00:05:20] Bret: And I wonder if I'd graduated in like 1999 where I've been like, mom, I just got a job at pets. com. It's good. But you know, at the end I just didn't have any options. So I was like, do I want to go like make kernel software at VMware? Do I want to go build search at Google? And I chose Google. 50, 50 ball.[00:05:36] Bret: I'm not really a 50, 50 ball. So I feel very fortunate in retrospect that the economy collapsed because in some ways it forced me into like one of the greatest companies of all time, but I kind of lucked into it, I think.[00:05:47] The Google Maps Rewrite Story[00:05:47] Alessio: So the famous story about Google is that you rewrote the Google maps back in, in one week after the map quest quest maps acquisition, what was the story there?[00:05:57] Alessio: Is it. Actually true. Is it [00:06:00] being glorified? Like how, how did that come to be? And is there any detail that maybe Paul hasn't shared before?[00:06:06] Bret: It's largely true, but I'll give the color commentary. So it was actually the front end, not the back end, but it turns out for Google maps, the front end was sort of the hard part just because Google maps was.[00:06:17] Bret: Largely the first ish kind of really interactive web application, say first ish. I think Gmail certainly was though Gmail, probably a lot of people then who weren't engineers probably didn't appreciate its level of interactivity. It was just fast, but. Google maps, because you could drag the map and it was sort of graphical.[00:06:38] Bret: My, it really in the mainstream, I think, was it a map[00:06:41] swyx: quest back then that was, you had the arrows up and down, it[00:06:44] Bret: was up and down arrows. Each map was a single image and you just click left and then wait for a few seconds to the new map to let it was really small too, because generating a big image was kind of expensive on computers that day.[00:06:57] Bret: So Google maps was truly innovative in that [00:07:00] regard. The story on it. There was a small company called where two technologies started by two Danish brothers, Lars and Jens Rasmussen, who are two of my closest friends now. They had made a windows app called expedition, which had beautiful maps. Even in 2000.[00:07:18] Bret: For whenever we acquired or sort of acquired their company, Windows software was not particularly fashionable, but they were really passionate about mapping and we had made a local search product that was kind of middling in terms of popularity, sort of like a yellow page of search product. So we wanted to really go into mapping.[00:07:36] Bret: We'd started working on it. Their small team seemed passionate about it. So we're like, come join us. We can build this together.[00:07:42] Technical Challenges and Innovations[00:07:42] Bret: It turned out to be a great blessing that they had built a windows app because you're less technically constrained when you're doing native code than you are building a web browser, particularly back then when there weren't really interactive web apps and it ended up.[00:07:56] Bret: Changing the level of quality that we [00:08:00] wanted to hit with the app because we were shooting for something that felt like a native windows application. So it was a really good fortune that we sort of, you know, their unusual technical choices turned out to be the greatest blessing. So we spent a lot of time basically saying, how can you make a interactive draggable map in a web browser?[00:08:18] Bret: How do you progressively load, you know, new map tiles, you know, as you're dragging even things like down in the weeds of the browser at the time, most browsers like Internet Explorer, which was dominant at the time would only load two images at a time from the same domain. So we ended up making our map tile servers have like.[00:08:37] Bret: Forty different subdomains so we could load maps and parallels like lots of hacks. I'm happy to go into as much as like[00:08:44] swyx: HTTP connections and stuff.[00:08:46] Bret: They just like, there was just maximum parallelism of two. And so if you had a map, set of map tiles, like eight of them, so So we just, we were down in the weeds of the browser anyway.[00:08:56] Bret: So it was lots of plumbing. I can, I know a lot more about browsers than [00:09:00] most people, but then by the end of it, it was fairly, it was a lot of duct tape on that code. If you've ever done an engineering project where you're not really sure the path from point A to point B, it's almost like. Building a house by building one room at a time.[00:09:14] Bret: The, there's not a lot of architectural cohesion at the end. And then we acquired a company called Keyhole, which became Google earth, which was like that three, it was a native windows app as well, separate app, great app, but with that, we got licenses to all this satellite imagery. And so in August of 2005, we added.[00:09:33] Bret: Satellite imagery to Google Maps, which added even more complexity in the code base. And then we decided we wanted to support Safari. There was no mobile phones yet. So Safari was this like nascent browser on, on the Mac. And it turns out there's like a lot of decisions behind the scenes, sort of inspired by this windows app, like heavy use of XML and XSLT and all these like.[00:09:54] Bret: Technologies that were like briefly fashionable in the early two thousands and everyone hates now for good [00:10:00] reason. And it turns out that all of the XML functionality and Internet Explorer wasn't supporting Safari. So people are like re implementing like XML parsers. And it was just like this like pile of s**t.[00:10:11] Bret: And I had to say a s**t on your part. Yeah, of[00:10:12] Alessio: course.[00:10:13] Bret: So. It went from this like beautifully elegant application that everyone was proud of to something that probably had hundreds of K of JavaScript, which sounds like nothing. Now we're talking like people have modems, you know, not all modems, but it was a big deal.[00:10:29] Bret: So it was like slow. It took a while to load and just, it wasn't like a great code base. Like everything was fragile. So I just got. Super frustrated by it. And then one weekend I did rewrite all of it. And at the time the word JSON hadn't been coined yet too, just to give you a sense. So it's all XML.[00:10:47] swyx: Yeah.[00:10:47] Bret: So we used what is now you would call JSON, but I just said like, let's use eval so that we can parse the data fast. And, and again, that's, it would literally as JSON, but at the time there was no name for it. So we [00:11:00] just said, let's. Pass on JavaScript from the server and eval it. And then somebody just refactored the whole thing.[00:11:05] Bret: And, and it wasn't like I was some genius. It was just like, you know, if you knew everything you wished you had known at the beginning and I knew all the functionality, cause I was the primary, one of the primary authors of the JavaScript. And I just like, I just drank a lot of coffee and just stayed up all weekend.[00:11:22] Bret: And then I, I guess I developed a bit of reputation and no one knew about this for a long time. And then Paul who created Gmail and I ended up starting a company with him too, after all of this told this on a podcast and now it's large, but it's largely true. I did rewrite it and it, my proudest thing.[00:11:38] Bret: And I think JavaScript people appreciate this. Like the un G zipped bundle size for all of Google maps. When I rewrote, it was 20 K G zipped. It was like much smaller for the entire application. It went down by like 10 X. So. What happened on Google? Google is a pretty mainstream company. And so like our usage is shot up because it turns out like it's faster.[00:11:57] Bret: Just being faster is worth a lot of [00:12:00] percentage points of growth at a scale of Google. So how[00:12:03] swyx: much modern tooling did you have? Like test suites no compilers.[00:12:07] Bret: Actually, that's not true. We did it one thing. So I actually think Google, I, you can. Download it. There's a, Google has a closure compiler, a closure compiler.[00:12:15] Bret: I don't know if anyone still uses it. It's gone. Yeah. Yeah. It's sort of gone out of favor. Yeah. Well, even until recently it was better than most JavaScript minifiers because it was more like it did a lot more renaming of variables and things. Most people use ES build now just cause it's fast and closure compilers built on Java and super slow and stuff like that.[00:12:37] Bret: But, so we did have that, that was it. Okay.[00:12:39] The Evolution of Web Applications[00:12:39] Bret: So and that was treated internally, you know, it was a really interesting time at Google at the time because there's a lot of teams working on fairly advanced JavaScript when no one was. So Google suggest, which Kevin Gibbs was the tech lead for, was the first kind of type ahead, autocomplete, I believe in a web browser, and now it's just pervasive in search boxes that you sort of [00:13:00] see a type ahead there.[00:13:01] Bret: I mean, chat, dbt[00:13:01] swyx: just added it. It's kind of like a round trip.[00:13:03] Bret: Totally. No, it's now pervasive as a UI affordance, but that was like Kevin's 20 percent project. And then Gmail, Paul you know, he tells the story better than anyone, but he's like, you know, basically was scratching his own itch, but what was really neat about it is email, because it's such a productivity tool, just needed to be faster.[00:13:21] Bret: So, you know, he was scratching his own itch of just making more stuff work on the client side. And then we, because of Lars and Yen sort of like setting the bar of this windows app or like we need our maps to be draggable. So we ended up. Not only innovate in terms of having a big sync, what would be called a single page application today, but also all the graphical stuff you know, we were crashing Firefox, like it was going out of style because, you know, when you make a document object model with the idea that it's a document and then you layer on some JavaScript and then we're essentially abusing all of this, it just was running into code paths that were not.[00:13:56] Bret: Well, it's rotten, you know, at this time. And so it was [00:14:00] super fun. And, and, you know, in the building you had, so you had compilers, people helping minify JavaScript just practically, but there is a great engineering team. So they were like, that's why Closure Compiler is so good. It was like a. Person who actually knew about programming languages doing it, not just, you know, writing regular expressions.[00:14:17] Bret: And then the team that is now the Chrome team believe, and I, I don't know this for a fact, but I'm pretty sure Google is the main contributor to Firefox for a long time in terms of code. And a lot of browser people were there. So every time we would crash Firefox, we'd like walk up two floors and say like, what the hell is going on here?[00:14:35] Bret: And they would load their browser, like in a debugger. And we could like figure out exactly what was breaking. And you can't change the code, right? Cause it's the browser. It's like slow, right? I mean, slow to update. So, but we could figure out exactly where the bug was and then work around it in our JavaScript.[00:14:52] Bret: So it was just like new territory. Like so super, super fun time, just like a lot of, a lot of great engineers figuring out [00:15:00] new things. And And now, you know, the word, this term is no longer in fashion, but the word Ajax, which was asynchronous JavaScript and XML cause I'm telling you XML, but see the word XML there, to be fair, the way you made HTTP requests from a client to server was this.[00:15:18] Bret: Object called XML HTTP request because Microsoft and making Outlook web access back in the day made this and it turns out to have nothing to do with XML. It's just a way of making HTTP requests because XML was like the fashionable thing. It was like that was the way you, you know, you did it. But the JSON came out of that, you know, and then a lot of the best practices around building JavaScript applications is pre React.[00:15:44] Bret: I think React was probably the big conceptual step forward that we needed. Even my first social network after Google, we used a lot of like HTML injection and. Making real time updates was still very hand coded and it's really neat when you [00:16:00] see conceptual breakthroughs like react because it's, I just love those things where it's like obvious once you see it, but it's so not obvious until you do.[00:16:07] Bret: And actually, well, I'm sure we'll get into AI, but I, I sort of feel like we'll go through that evolution with AI agents as well that I feel like we're missing a lot of the core abstractions that I think in 10 years we'll be like, gosh, how'd you make agents? Before that, you know, but it was kind of that early days of web applications.[00:16:22] swyx: There's a lot of contenders for the reactive jobs of of AI, but no clear winner yet. I would say one thing I was there for, I mean, there's so much we can go into there. You just covered so much.[00:16:32] Product Management and Engineering Synergy[00:16:32] swyx: One thing I just, I just observe is that I think the early Google days had this interesting mix of PM and engineer, which I think you are, you didn't, you didn't wait for PM to tell you these are my, this is my PRD.[00:16:42] swyx: This is my requirements.[00:16:44] mix: Oh,[00:16:44] Bret: okay.[00:16:45] swyx: I wasn't technically a software engineer. I mean,[00:16:48] Bret: by title, obviously. Right, right, right.[00:16:51] swyx: It's like a blend. And I feel like these days, product is its own discipline and its own lore and own industry and engineering is its own thing. And there's this process [00:17:00] that happens and they're kind of separated, but you don't produce as good of a product as if they were the same person.[00:17:06] swyx: And I'm curious, you know, if, if that, if that sort of resonates in, in, in terms of like comparing early Google versus modern startups that you see out there,[00:17:16] Bret: I certainly like wear a lot of hats. So, you know, sort of biased in this, but I really agree that there's a lot of power and combining product design engineering into as few people as possible because, you know few great things have been created by committee, you know, and so.[00:17:33] Bret: If engineering is an order taking organization for product you can sometimes make meaningful things, but rarely will you create extremely well crafted breakthrough products. Those tend to be small teams who deeply understand the customer need that they're solving, who have a. Maniacal focus on outcomes.[00:17:53] Bret: And I think the reason why it's, I think for some areas, if you look at like software as a service five years ago, maybe you can have a [00:18:00] separation of product and engineering because most software as a service created five years ago. I wouldn't say there's like a lot of like. Technological breakthroughs required for most, you know, business applications.[00:18:11] Bret: And if you're making expense reporting software or whatever, it's useful. I don't mean to be dismissive of expense reporting software, but you probably just want to understand like, what are the requirements of the finance department? What are the requirements of an individual file expense report? Okay.[00:18:25] Bret: Go implement that. And you kind of know how web applications are implemented. You kind of know how to. How databases work, how to build auto scaling with your AWS cluster, whatever, you know, it's just, you're just applying best practices to yet another problem when you have areas like the early days of mobile development or the early days of interactive web applications, which I think Google Maps and Gmail represent, or now AI agents, you're in this constant conversation with what the requirements of your customers and stakeholders are and all the different people interacting with it.[00:18:58] Bret: And the capabilities of the [00:19:00] technology. And it's almost impossible to specify the requirements of a product when you're not sure of the limitations of the technology itself. And that's why I use the word conversation. It's not literal. That's sort of funny to use that word in the age of conversational AI.[00:19:15] Bret: You're constantly sort of saying, like, ideally, you could sprinkle some magic AI pixie dust and solve all the world's problems, but it's not the way it works. And it turns out that actually, I'll just give an interesting example.[00:19:26] AI Agents and Modern Tooling[00:19:26] Bret: I think most people listening probably use co pilots to code like Cursor or Devon or Microsoft Copilot or whatever.[00:19:34] Bret: Most of those tools are, they're remarkable. I'm, I couldn't, you know, imagine development without them now, but they're not autonomous yet. Like I wouldn't let it just write most code without my interactively inspecting it. We just are somewhere between it's an amazing co pilot and it's an autonomous software engineer.[00:19:53] Bret: As a product manager, like your aspirations for what the product is are like kind of meaningful. But [00:20:00] if you're a product person, yeah, of course you'd say it should be autonomous. You should click a button and program should come out the other side. The requirements meaningless. Like what matters is like, what is based on the like very nuanced limitations of the technology.[00:20:14] Bret: What is it capable of? And then how do you maximize the leverage? It gives a software engineering team, given those very nuanced trade offs. Coupled with the fact that those nuanced trade offs are changing more rapidly than any technology in my memory, meaning every few months you'll have new models with new capabilities.[00:20:34] Bret: So how do you construct a product that can absorb those new capabilities as rapidly as possible as well? That requires such a combination of technical depth and understanding the customer that you really need more integration. Of product design and engineering. And so I think it's why with these big technology waves, I think startups have a bit of a leg up relative to incumbents because they [00:21:00] tend to be sort of more self actualized in terms of just like bringing those disciplines closer together.[00:21:06] Bret: And in particular, I think entrepreneurs, the proverbial full stack engineers, you know, have a leg up as well because. I think most breakthroughs happen when you have someone who can understand those extremely nuanced technical trade offs, have a vision for a product. And then in the process of building it, have that, as I said, like metaphorical conversation with the technology, right?[00:21:30] Bret: Gosh, I ran into a technical limit that I didn't expect. It's not just like changing that feature. You might need to refactor the whole product based on that. And I think that's, that it's particularly important right now. So I don't, you know, if you, if you're building a big ERP system, probably there's a great reason to have product and engineering.[00:21:51] Bret: I think in general, the disciplines are there for a reason. I think when you're dealing with something as nuanced as the like technologies, like large language models today, there's a ton of [00:22:00] advantage of having. Individuals or organizations that integrate the disciplines more formally.[00:22:05] Alessio: That makes a lot of sense.[00:22:06] Alessio: I've run a lot of engineering teams in the past, and I think the product versus engineering tension has always been more about effort than like whether or not the feature is buildable. But I think, yeah, today you see a lot more of like. Models actually cannot do that. And I think the most interesting thing is on the startup side, people don't yet know where a lot of the AI value is going to accrue.[00:22:26] Alessio: So you have this rush of people building frameworks, building infrastructure, layered things, but we don't really know the shape of the compute. I'm curious that Sierra, like how you thought about building an house, a lot of the tooling for evals or like just, you know, building the agents and all of that.[00:22:41] Alessio: Versus how you see some of the startup opportunities that is maybe still out there.[00:22:46] Bret: We build most of our tooling in house at Sierra, not all. It's, we don't, it's not like not invented here syndrome necessarily, though, maybe slightly guilty of that in some ways, but because we're trying to build a platform [00:23:00] that's in Dorian, you know, we really want to have control over our own destiny.[00:23:03] Bret: And you had made a comment earlier that like. We're still trying to figure out who like the reactive agents are and the jury is still out. I would argue it hasn't been created yet. I don't think the jury is still out to go use that metaphor. We're sort of in the jQuery era of agents, not the react era.[00:23:19] Bret: And, and that's like a throwback for people listening,[00:23:22] swyx: we shouldn't rush it. You know?[00:23:23] Bret: No, yeah, that's my point is. And so. Because we're trying to create an enduring company at Sierra that outlives us, you know, I'm not sure we want to like attach our cart to some like to a horse where it's not clear that like we've figured out and I actually want as a company, we're trying to enable just at a high level and I'll, I'll quickly go back to tech at Sierra, we help consumer brands build customer facing AI agents.[00:23:48] Bret: So. Everyone from Sonos to ADT home security to Sirius XM, you know, if you call them on the phone and AI will pick up with you, you know, chat with them on the Sirius XM homepage. It's an AI agent called Harmony [00:24:00] that they've built on our platform. We're what are the contours of what it means for someone to build an end to end complete customer experience with AI with conversational AI.[00:24:09] Bret: You know, we really want to dive into the deep end of, of all the trade offs to do it. You know, where do you use fine tuning? Where do you string models together? You know, where do you use reasoning? Where do you use generation? How do you use reasoning? How do you express the guardrails of an agentic process?[00:24:25] Bret: How do you impose determinism on a fundamentally non deterministic technology? There's just a lot of really like as an important design space. And I could sit here and tell you, we have the best approach. Every entrepreneur will, you know. But I hope that in two years, we look back at our platform and laugh at how naive we were, because that's the pace of change broadly.[00:24:45] Bret: If you talk about like the startup opportunities, I'm not wholly skeptical of tools companies, but I'm fairly skeptical. There's always an exception for every role, but I believe that certainly there's a big market for [00:25:00] frontier models, but largely for companies with huge CapEx budgets. So. Open AI and Microsoft's Anthropic and Amazon Web Services, Google Cloud XAI, which is very well capitalized now, but I think the, the idea that a company can make money sort of pre training a foundation model is probably not true.[00:25:20] Bret: It's hard to, you're competing with just, you know, unreasonably large CapEx budgets. And I just like the cloud infrastructure market, I think will be largely there. I also really believe in the applications of AI. And I define that not as like building agents or things like that. I define it much more as like, you're actually solving a problem for a business.[00:25:40] Bret: So it's what Harvey is doing in legal profession or what cursor is doing for software engineering or what we're doing for customer experience and customer service. The reason I believe in that is I do think that in the age of AI, what's really interesting about software is it can actually complete a task.[00:25:56] Bret: It can actually do a job, which is very different than the value proposition of [00:26:00] software was to ancient history two years ago. And as a consequence, I think the way you build a solution and For a domain is very different than you would have before, which means that it's not obvious, like the incumbent incumbents have like a leg up, you know, necessarily, they certainly have some advantages, but there's just such a different form factor, you know, for providing a solution and it's just really valuable.[00:26:23] Bret: You know, it's. Like just think of how much money cursor is saving software engineering teams or the alternative, how much revenue it can produce tool making is really challenging. If you look at the cloud market, just as a analog, there are a lot of like interesting tools, companies, you know, Confluent, Monetized Kafka, Snowflake, Hortonworks, you know, there's a, there's a bunch of them.[00:26:48] Bret: A lot of them, you know, have that mix of sort of like like confluence or have the open source or open core or whatever you call it. I, I, I'm not an expert in this area. You know, I do think [00:27:00] that developers are fickle. I think that in the tool space, I probably like. Default towards open source being like the area that will win.[00:27:09] Bret: It's hard to build a company around this and then you end up with companies sort of built around open source to that can work. Don't get me wrong, but I just think that it's nowadays the tools are changing so rapidly that I'm like, not totally skeptical of tool makers, but I just think that open source will broadly win, but I think that the CapEx required for building frontier models is such that it will go to a handful of big companies.[00:27:33] Bret: And then I really believe in agents for specific domains which I think will, it's sort of the analog to software as a service in this new era. You know, it's like, if you just think of the cloud. You can lease a server. It's just a low level primitive, or you can buy an app like you know, Shopify or whatever.[00:27:51] Bret: And most people building a storefront would prefer Shopify over hand rolling their e commerce storefront. I think the same thing will be true of AI. So [00:28:00] I've. I tend to like, if I have a, like an entrepreneur asked me for advice, I'm like, you know, move up the stack as far as you can towards a customer need.[00:28:09] Bret: Broadly, but I, but it doesn't reduce my excitement about what is the reactive building agents kind of thing, just because it is, it is the right question to ask, but I think we'll probably play out probably an open source space more than anything else.[00:28:21] swyx: Yeah, and it's not a priority for you. There's a lot in there.[00:28:24] swyx: I'm kind of curious about your idea maze towards, there are many customer needs. You happen to identify customer experience as yours, but it could equally have been coding assistance or whatever. I think for some, I'm just kind of curious at the top down, how do you look at the world in terms of the potential problem space?[00:28:44] swyx: Because there are many people out there who are very smart and pick the wrong problem.[00:28:47] Bret: Yeah, that's a great question.[00:28:48] Future of Software Development[00:28:48] Bret: By the way, I would love to talk about the future of software, too, because despite the fact it didn't pick coding, I have a lot of that, but I can talk to I can answer your question, though, you know I think when a technology is as [00:29:00] cool as large language models.[00:29:02] Bret: You just see a lot of people starting from the technology and searching for a problem to solve. And I think it's why you see a lot of tools companies, because as a software engineer, you start building an app or a demo and you, you encounter some pain points. You're like,[00:29:17] swyx: a lot of[00:29:17] Bret: people are experiencing the same pain point.[00:29:19] Bret: What if I make it? That it's just very incremental. And you know, I always like to use the metaphor, like you can sell coffee beans, roasted coffee beans. You can add some value. You took coffee beans and you roasted them and roasted coffee beans largely, you know, are priced relative to the cost of the beans.[00:29:39] Bret: Or you can sell a latte and a latte. Is rarely priced directly like as a percentage of coffee bean prices. In fact, if you buy a latte at the airport, it's a captive audience. So it's a really expensive latte. And there's just a lot that goes into like. How much does a latte cost? And I bring it up because there's a supply chain from growing [00:30:00] coffee beans to roasting coffee beans to like, you know, you could make one at home or you could be in the airport and buy one and the margins of the company selling lattes in the airport is a lot higher than the, you know, people roasting the coffee beans and it's because you've actually solved a much more acute human problem in the airport.[00:30:19] Bret: And, and it's just worth a lot more to that person in that moment. It's kind of the way I think about technology too. It sounds funny to liken it to coffee beans, but you're selling tools on top of a large language model yet in some ways your market is big, but you're probably going to like be price compressed just because you're sort of a piece of infrastructure and then you have open source and all these other things competing with you naturally.[00:30:43] Bret: If you go and solve a really big business problem for somebody, that's actually like a meaningful business problem that AI facilitates, they will value it according to the value of that business problem. And so I actually feel like people should just stop. You're like, no, that's, that's [00:31:00] unfair. If you're searching for an idea of people, I, I love people trying things, even if, I mean, most of the, a lot of the greatest ideas have been things no one believed in.[00:31:07] Bret: So I like, if you're passionate about something, go do it. Like who am I to say, yeah, a hundred percent. Or Gmail, like Paul as far, I mean I, some of it's Laura at this point, but like Gmail is Paul's own email for a long time. , and then I amusingly and Paul can't correct me, I'm pretty sure he sent her in a link and like the first comment was like, this is really neat.[00:31:26] Bret: It would be great. It was not your email, but my own . I don't know if it's a true story. I'm pretty sure it's, yeah, I've read that before. So scratch your own niche. Fine. Like it depends on what your goal is. If you wanna do like a venture backed company, if its a. Passion project, f*****g passion, do it like don't listen to anybody.[00:31:41] Bret: In fact, but if you're trying to start, you know an enduring company, solve an important business problem. And I, and I do think that in the world of agents, the software industries has shifted where you're not just helping people more. People be more productive, but you're actually accomplishing tasks autonomously.[00:31:58] Bret: And as a consequence, I think the [00:32:00] addressable market has just greatly expanded just because software can actually do things now and actually accomplish tasks and how much is coding autocomplete worth. A fair amount. How much is the eventual, I'm certain we'll have it, the software agent that actually writes the code and delivers it to you, that's worth a lot.[00:32:20] Bret: And so, you know, I would just maybe look up from the large language models and start thinking about the economy and, you know, think from first principles. I don't wanna get too far afield, but just think about which parts of the economy. We'll benefit most from this intelligence and which parts can absorb it most easily.[00:32:38] Bret: And what would an agent in this space look like? Who's the customer of it is the technology feasible. And I would just start with these business problems more. And I think, you know, the best companies tend to have great engineers who happen to have great insight into a market. And it's that last part that I think some people.[00:32:56] Bret: Whether or not they have, it's like people start so much in the technology, they [00:33:00] lose the forest for the trees a little bit.[00:33:02] Alessio: How do you think about the model of still selling some sort of software versus selling more package labor? I feel like when people are selling the package labor, it's almost more stateless, you know, like it's easier to swap out if you're just putting an input and getting an output.[00:33:16] Alessio: If you think about coding, if there's no ID, you're just putting a prompt and getting back an app. It doesn't really matter. Who generates the app, you know, you have less of a buy in versus the platform you're building, I'm sure on the backend customers have to like put on their documentation and they have, you know, different workflows that they can tie in what's kind of like the line to draw there versus like going full where you're managed customer support team as a service outsource versus.[00:33:40] Alessio: This is the Sierra platform that you can build on. What was that decision? I'll sort of[00:33:44] Bret: like decouple the question in some ways, which is when you have something that's an agent, who is the person using it and what do they want to do with it? So let's just take your coding agent for a second. I will talk about Sierra as well.[00:33:59] Bret: Who's the [00:34:00] customer of a, an agent that actually produces software? Is it a software engineering manager? Is it a software engineer? And it's there, you know, intern so to speak. I don't know. I mean, we'll figure this out over the next few years. Like what is that? And is it generating code that you then review?[00:34:16] Bret: Is it generating code with a set of unit tests that pass, what is the actual. For lack of a better word contract, like, how do you know that it did what you wanted it to do? And then I would say like the product and the pricing, the packaging model sort of emerged from that. And I don't think the world's figured out.[00:34:33] Bret: I think it'll be different for every agent. You know, in our customer base, we do what's called outcome based pricing. So essentially every time the AI agent. Solves the problem or saves a customer or whatever it might be. There's a pre negotiated rate for that. We do that. Cause it's, we think that that's sort of the correct way agents, you know, should be packaged.[00:34:53] Bret: I look back at the history of like cloud software and notably the introduction of the browser, which led to [00:35:00] software being delivered in a browser, like Salesforce to. Famously invented sort of software as a service, which is both a technical delivery model through the browser, but also a business model, which is you subscribe to it rather than pay for a perpetual license.[00:35:13] Bret: Those two things are somewhat orthogonal, but not really. If you think about the idea of software running in a browser, that's hosted. Data center that you don't own, you sort of needed to change the business model because you don't, you can't really buy a perpetual license or something otherwise like, how do you afford making changes to it?[00:35:31] Bret: So it only worked when you were buying like a new version every year or whatever. So to some degree, but then the business model shift actually changed business as we know it, because now like. Things like Adobe Photoshop. Now you subscribe to rather than purchase. So it ended up where you had a technical shift and a business model shift that were very logically intertwined that actually the business model shift was turned out to be as significant as the technical as the shift.[00:35:59] Bret: And I think with [00:36:00] agents, because they actually accomplish a job, I do think that it doesn't make sense to me that you'd pay for the privilege of like. Using the software like that coding agent, like if it writes really bad code, like fire it, you know, I don't know what the right metaphor is like you should pay for a job.[00:36:17] Bret: Well done in my opinion. I mean, that's how you pay your software engineers, right? And[00:36:20] swyx: and well, not really. We paid to put them on salary and give them options and they vest over time. That's fair.[00:36:26] Bret: But my point is that you don't pay them for how many characters they write, which is sort of the token based, you know, whatever, like, There's a, that famous Apple story where we're like asking for a report of how many lines of code you wrote.[00:36:40] Bret: And one of the engineers showed up with like a negative number cause he had just like done a big refactoring. There was like a big F you to management who didn't understand how software is written. You know, my sense is like the traditional usage based or seat based thing. It's just going to look really antiquated.[00:36:55] Bret: Cause it's like asking your software engineer, how many lines of code did you write today? Like who cares? Like, cause [00:37:00] absolutely no correlation. So my old view is I don't think it's be different in every category, but I do think that that is the, if an agent is doing a job, you should, I think it properly incentivizes the maker of that agent and the customer of, of your pain for the job well done.[00:37:16] Bret: It's not always perfect to measure. It's hard to measure engineering productivity, but you can, you should do something other than how many keys you typed, you know Talk about perverse incentives for AI, right? Like I can write really long functions to do the same thing, right? So broadly speaking, you know, I do think that we're going to see a change in business models of software towards outcomes.[00:37:36] Bret: And I think you'll see a change in delivery models too. And, and, you know, in our customer base you know, we empower our customers to really have their hands on the steering wheel of what the agent does they, they want and need that. But the role is different. You know, at a lot of our customers, the customer experience operations folks have renamed themselves the AI architects, which I think is really cool.[00:37:55] Bret: And, you know, it's like in the early days of the Internet, there's the role of the webmaster. [00:38:00] And I don't know whether your webmaster is not a fashionable, you know, Term, nor is it a job anymore? I just, I don't know. Will they, our tech stand the test of time? Maybe, maybe not. But I do think that again, I like, you know, because everyone listening right now is a software engineer.[00:38:14] Bret: Like what is the form factor of a coding agent? And actually I'll, I'll take a breath. Cause actually I have a bunch of pins on them. Like I wrote a blog post right before Christmas, just on the future of software development. And one of the things that's interesting is like, if you look at the way I use cursor today, as an example, it's inside of.[00:38:31] Bret: A repackaged visual studio code environment. I sometimes use the sort of agentic parts of it, but it's largely, you know, I've sort of gotten a good routine of making it auto complete code in the way I want through tuning it properly when it actually can write. I do wonder what like the future of development environments will look like.[00:38:55] Bret: And to your point on what is a software product, I think it's going to change a lot in [00:39:00] ways that will surprise us. But I always use, I use the metaphor in my blog post of, have you all driven around in a way, Mo around here? Yeah, everyone has. And there are these Jaguars, the really nice cars, but it's funny because it still has a steering wheel, even though there's no one sitting there and the steering wheels like turning and stuff clearly in the future.[00:39:16] Bret: If once we get to that, be more ubiquitous, like why have the steering wheel and also why have all the seats facing forward? Maybe just for car sickness. I don't know, but you could totally rearrange the car. I mean, so much of the car is oriented around the driver, so. It stands to reason to me that like, well, autonomous agents for software engineering run through visual studio code.[00:39:37] Bret: That seems a little bit silly because having a single source code file open one at a time is kind of a goofy form factor for when like the code isn't being written primarily by you, but it begs the question of what's your relationship with that agent. And I think the same is true in our industry of customer experience, which is like.[00:39:55] Bret: Who are the people managing this agent? What are the tools do they need? And they definitely need [00:40:00] tools, but it's probably pretty different than the tools we had before. It's certainly different than training a contact center team. And as software engineers, I think that I would like to see particularly like on the passion project side or research side.[00:40:14] Bret: More innovation in programming languages. I think that we're bringing the cost of writing code down to zero. So the fact that we're still writing Python with AI cracks me up just cause it's like literally was designed to be ergonomic to write, not safe to run or fast to run. I would love to see more innovation and how we verify program correctness.[00:40:37] Bret: I studied for formal verification in college a little bit and. It's not very fashionable because it's really like tedious and slow and doesn't work very well. If a lot of code is being written by a machine, you know, one of the primary values we can provide is verifying that it actually does what we intend that it does.[00:40:56] Bret: I think there should be lots of interesting things in the software development life cycle, like how [00:41:00] we think of testing and everything else, because. If you think about if we have to manually read every line of code that's coming out as machines, it will just rate limit how much the machines can do. The alternative is totally unsafe.[00:41:13] Bret: So I wouldn't want to put code in production that didn't go through proper code review and inspection. So my whole view is like, I actually think there's like an AI native I don't think the coding agents don't work well enough to do this yet, but once they do, what is sort of an AI native software development life cycle and how do you actually.[00:41:31] Bret: Enable the creators of software to produce the highest quality, most robust, fastest software and know that it's correct. And I think that's an incredible opportunity. I mean, how much C code can we rewrite and rust and make it safe so that there's fewer security vulnerabilities. Can we like have more efficient, safer code than ever before?[00:41:53] Bret: And can you have someone who's like that guy in the matrix, you know, like staring at the little green things, like where could you have an operator [00:42:00] of a code generating machine be like superhuman? I think that's a cool vision. And I think too many people are focused on like. Autocomplete, you know, right now, I'm not, I'm not even, I'm guilty as charged.[00:42:10] Bret: I guess in some ways, but I just like, I'd like to see some bolder ideas. And that's why when you were joking, you know, talking about what's the react of whatever, I think we're clearly in a local maximum, you know, metaphor, like sort of conceptual local maximum, obviously it's moving really fast. I think we're moving out of it.[00:42:26] Alessio: Yeah. At the end of 23, I've read this blog post from syntax to semantics. Like if you think about Python. It's taking C and making it more semantic and LLMs are like the ultimate semantic program, right? You can just talk to them and they can generate any type of syntax from your language. But again, the languages that they have to use were made for us, not for them.[00:42:46] Alessio: But the problem is like, as long as you will ever need a human to intervene, you cannot change the language under it. You know what I mean? So I'm curious at what point of automation we'll need to get, we're going to be okay making changes. To the underlying languages, [00:43:00] like the programming languages versus just saying, Hey, you just got to write Python because I understand Python and I'm more important at the end of the day than the model.[00:43:08] Alessio: But I think that will change, but I don't know if it's like two years or five years. I think it's more nuanced actually.[00:43:13] Bret: So I think there's a, some of the more interesting programming languages bring semantics into syntax. So let me, that's a little reductive, but like Rust as an example, Rust is memory safe.[00:43:25] Bret: Statically, and that was a really interesting conceptual, but it's why it's hard to write rust. It's why most people write python instead of rust. I think rust programs are safer and faster than python, probably slower to compile. But like broadly speaking, like given the option, if you didn't have to care about the labor that went into it.[00:43:45] Bret: You should prefer a program written in Rust over a program written in Python, just because it will run more efficiently. It's almost certainly safer, et cetera, et cetera, depending on how you define safe, but most people don't write Rust because it's kind of a pain in the ass. And [00:44:00] the audience of people who can is smaller, but it's sort of better in most, most ways.[00:44:05] Bret: And again, let's say you're making a web service and you didn't have to care about how hard it was to write. If you just got the output of the web service, the rest one would be cheaper to operate. It's certainly cheaper and probably more correct just because there's so much in the static analysis implied by the rest programming language that it probably will have fewer runtime errors and things like that as well.[00:44:25] Bret: So I just give that as an example, because so rust, at least my understanding that came out of the Mozilla team, because. There's lots of security vulnerabilities in the browser and it needs to be really fast. They said, okay, we want to put more of a burden at the authorship time to have fewer issues at runtime.[00:44:43] Bret: And we need the constraint that it has to be done statically because browsers need to be really fast. My sense is if you just think about like the, the needs of a programming language today, where the role of a software engineer is [00:45:00] to use an AI to generate functionality and audit that it does in fact work as intended, maybe functionally, maybe from like a correctness standpoint, some combination thereof, how would you create a programming system that facilitated that?[00:45:15] Bret: And, you know, I bring up Rust is because I think it's a good example of like, I think given a choice of writing in C or Rust, you should choose Rust today. I think most people would say that, even C aficionados, just because. C is largely less safe for very similar, you know, trade offs, you know, for the, the system and now with AI, it's like, okay, well, that just changes the game on writing these things.[00:45:36] Bret: And so like, I just wonder if a combination of programming languages that are more structurally oriented towards the values that we need from an AI generated program, verifiable correctness and all of that. If it's tedious to produce for a person, that maybe doesn't matter. But one thing, like if I asked you, is this rest program memory safe?[00:45:58] Bret: You wouldn't have to read it, you just have [00:46:00] to compile it. So that's interesting. I mean, that's like an, that's one example of a very modest form of formal verification. So I bring that up because I do think you have AI inspect AI, you can have AI reviewed. Do AI code reviews. It would disappoint me if the best we could get was AI reviewing Python and having scaled a few very large.[00:46:21] Bret: Websites that were written on Python. It's just like, you know, expensive and it's like every, trust me, every team who's written a big web service in Python has experimented with like Pi Pi and all these things just to make it slightly more efficient than it naturally is. You don't really have true multi threading anyway.[00:46:36] Bret: It's just like clearly that you do it just because it's convenient to write. And I just feel like we're, I don't want to say it's insane. I just mean. I do think we're at a local maximum. And I would hope that we create a programming system, a combination of programming languages, formal verification, testing, automated code reviews, where you can use AI to generate software in a high scale way and trust it.[00:46:59] Bret: And you're [00:47:00] not limited by your ability to read it necessarily. I don't know exactly what form that would take, but I feel like that would be a pretty cool world to live in.[00:47:08] Alessio: Yeah. We had Chris Lanner on the podcast. He's doing great work with modular. I mean, I love. LVM. Yeah. Basically merging rust in and Python.[00:47:15] Alessio: That's kind of the idea. Should be, but I'm curious is like, for them a big use case was like making it compatible with Python, same APIs so that Python developers could use it. Yeah. And so I, I wonder at what point, well, yeah.[00:47:26] Bret: At least my understanding is they're targeting the data science Yeah. Machine learning crowd, which is all written in Python, so still feels like a local maximum.[00:47:34] Bret: Yeah.[00:47:34] swyx: Yeah, exactly. I'll force you to make a prediction. You know, Python's roughly 30 years old. In 30 years from now, is Rust going to be bigger than Python?[00:47:42] Bret: I don't know this, but just, I don't even know this is a prediction. I just am sort of like saying stuff I hope is true. I would like to see an AI native programming language and programming system, and I use language because I'm not sure language is even the right thing, but I hope in 30 years, there's an AI native way we make [00:48:00] software that is wholly uncorrelated with the current set of programming languages.[00:48:04] Bret: or not uncorrelated, but I think most programming languages today were designed to be efficiently authored by people and some have different trade offs.[00:48:15] Evolution of Programming Languages[00:48:15] Bret: You know, you have Haskell and others that were designed for abstractions for parallelism and things like that. You have programming languages like Python, which are designed to be very easily written, sort of like Perl and Python lineage, which is why data scientists use it.[00:48:31] Bret: It's it can, it has a. Interactive mode, things like that. And I love, I'm a huge Python fan. So despite all my Python trash talk, a huge Python fan wrote at least two of my three companies were exclusively written in Python and then C came out of the birth of Unix and it wasn't the first, but certainly the most prominent first step after assembly language, right?[00:48:54] Bret: Where you had higher level abstractions rather than and going beyond go to, to like abstractions, [00:49:00] like the for loop and the while loop.[00:49:01] The Future of Software Engineering[00:49:01] Bret: So I just think that if the act of writing code is no longer a meaningful human exercise, maybe it will be, I don't know. I'm just saying it sort of feels like maybe it's one of those parts of history that just will sort of like go away, but there's still the role of this offer engineer, like the person actually building the system.[00:49:20] Bret: Right. And. What does a programming system for that form factor look like?[00:49:25] React and Front-End Development[00:49:25] Bret: And I, I just have a, I hope to be just like I mentioned, I remember I was at Facebook in the very early days when, when, what is now react was being created. And I remember when the, it was like released open source I had left by that time and I was just like, this is so f*****g cool.[00:49:42] Bret: Like, you know, to basically model your app independent of the data flowing through it, just made everything easier. And then now. You know, I can create, like there's a lot of the front end software gym play is like a little chaotic for me, to be honest with you. It is like, it's sort of like [00:50:00] abstraction soup right now for me, but like some of those core ideas felt really ergonomic.[00:50:04] Bret: I just wanna, I'm just looking forward to the day when someone comes up with a programming system that feels both really like an aha moment, but completely foreign to me at the same time. Because they created it with sort of like from first principles recognizing that like. Authoring code in an editor is maybe not like the primary like reason why a programming system exists anymore.[00:50:26] Bret: And I think that's like, that would be a very exciting day for me.[00:50:28] The Role of AI in Programming[00:50:28] swyx: Yeah, I would say like the various versions of this discussion have happened at the end of the day, you still need to precisely communicate what you want. As a manager of people, as someone who has done many, many legal contracts, you know how hard that is.[00:50:42] swyx: And then now we have to talk to machines doing that and AIs interpreting what we mean and reading our minds effectively. I don't know how to get across that barrier of translating human intent to instructions. And yes, it can be more declarative, but I don't know if it'll ever Crossover from being [00:51:00] a programming language to something more than that.[00:51:02] Bret: I agree with you. And I actually do think if you look at like a legal contract, you know, the imprecision of the English language, it's like a flaw in the system. How many[00:51:12] swyx: holes there are.[00:51:13] Bret: And I do think that when you're making a mission critical software system, I don't think it should be English language prompts.[00:51:19] Bret: I think that is silly because you want the precision of a a programming language. My point was less about that and more about if the actual act of authoring it, like if you.[00:51:32] Formal Verification in Software[00:51:32] Bret: I'll think of some embedded systems do use formal verification. I know it's very common in like security protocols now so that you can, because the importance of correctness is so great.[00:51:41] Bret: My intellectual exercise is like, why not do that for all software? I mean, probably that's silly just literally to do what we literally do for. These low level security protocols, but the only reason we don't is because it's hard and tedious and hard and tedious are no longer factors. So, like, if I could, I mean, [00:52:00] just think of, like, the silliest app on your phone right now, the idea that that app should be, like, formally verified for its correctness feels laughable right now because, like, God, why would you spend the time on it?[00:52:10] Bret: But if it's zero costs, like, yeah, I guess so. I mean, it never crashed. That's probably good. You know, why not? I just want to, like, set our bars really high. Like. We should make, software has been amazing. Like there's a Mark Andreessen blog post, software is eating the world. And you know, our whole life is, is mediated digitally.[00:52:26] Bret: And that's just increasing with AI. And now we'll have our personal agents talking to the agents on the CRO platform and it's agents all the way down, you know, our core infrastructure is running on these digital systems. We now have like, and we've had a shortage of software developers for my entire life.[00:52:45] Bret: And as a consequence, you know if you look, remember like health care, got healthcare. gov that fiasco security vulnerabilities leading to state actors getting access to critical infrastructure. I'm like. We now have like created this like amazing system that can [00:53:00] like, we can fix this, you know, and I, I just want to, I'm both excited about the productivity gains in the economy, but I just think as software engineers, we should be bolder.[00:53:08] Bret: Like we should have aspirations to fix these systems so that like in general, as you said, as precise as we want to be in the specification of the system. We can make it work correctly now, and I'm being a little bit hand wavy, and I think we need some systems. I think that's where we should set the bar, especially when so much of our life depends on this critical digital infrastructure.[00:53:28] Bret: So I'm I'm just like super optimistic about it. But actually, let's go to w

Applications close Monday for the NYC AI Engineer Summit focusing on AI Leadership and Agent Engineering! If you applied, invites should be rolling out shortly.The search landscape is experiencing a fundamental shift. Google built a >$2T company with the “10 blue links” experience, driven by PageRank as the core innovation for ranking. This was a big improvement from the previous directory-based experiences of AltaVista and Yahoo. Almost 4 decades later, Google is now stuck in this links-based experience, especially from a business model perspective. This legacy architecture creates fundamental constraints:* Must return results in ~400 milliseconds* Required to maintain comprehensive web coverage* Tied to keyword-based matching algorithms* Cost structures optimized for traditional indexingAs we move from the era of links to the era of answers, the way search works is changing. You're not showing a user links, but the goal is to provide context to an LLM. This means moving from keyword based search to more semantic understanding of the content:The link prediction objective can be seen as like a neural PageRank because what you're doing is you're predicting the links people share... but it's more powerful than PageRank. It's strictly more powerful because people might refer to that Paul Graham fundraising essay in like a thousand different ways. And so our model learns all the different ways.All of this is now powered by a $5M cluster with 144 H200s:This architectural choice enables entirely new search capabilities:* Comprehensive result sets instead of approximations* Deep semantic understanding of queries* Ability to process complex, natural language requestsAs search becomes more complex, time to results becomes a variable:People think of searches as like, oh, it takes 500 milliseconds because we've been conditioned... But what if searches can take like a minute or 10 minutes or a whole day, what can you then do?Unlike traditional search engines' fixed-cost indexing, Exa employs a hybrid approach:* Front-loaded compute for indexing and embeddings* Variable inference costs based on query complexity* Mix of owned infrastructure ($5M H200 cluster) and cloud resourcesExa sees a lot of competition from products like Perplexity and ChatGPT Search which layer AI on top of traditional search backends, but Exa is betting that true innovation requires rethinking search from the ground up. For example, the recently launched Websets, a way to turn searches into structured output in grid format, allowing you to create lists and databases out of web pages. The company raised a $17M Series A to build towards this mission, so keep an eye out for them in 2025. Chapters* 00:00:00 Introductions* 00:01:12 ExaAI's initial pitch and concept* 00:02:33 Will's background at SpaceX and Zoox* 00:03:45 Evolution of ExaAI (formerly Metaphor Systems)* 00:05:38 Exa's link prediction technology* 00:09:20 Meaning of the name "Exa"* 00:10:36 ExaAI's new product launch and capabilities* 00:13:33 Compute budgets and variable compute products* 00:14:43 Websets as a B2B offering* 00:19:28 How do you build a search engine?* 00:22:43 What is Neural PageRank?* 00:27:58 Exa use cases * 00:35:00 Auto-prompting* 00:38:42 Building agentic search* 00:44:19 Is o1 on the path to AGI?* 00:49:59 Company culture and nap pods* 00:54:52 Economics of AI search and the future of search technologyFull YouTube TranscriptPlease like and subscribe!Show Notes* ExaAI* Web Search Product* Websets* Series A Announcement* Exa Nap Pods* Perplexity AI* Character.AITranscriptAlessio [00:00:00]: Hey, everyone. Welcome to the Latent Space podcast. This is Alessio, partner and CTO at Decibel Partners, and I'm joined by my co-host Swyx, founder of Smol.ai.Swyx [00:00:10]: Hey, and today we're in the studio with my good friend and former landlord, Will Bryk. Roommate. How you doing? Will, you're now CEO co-founder of ExaAI, used to be Metaphor Systems. What's your background, your story?Will [00:00:30]: Yeah, sure. So, yeah, I'm CEO of Exa. I've been doing it for three years. I guess I've always been interested in search, whether I knew it or not. Like, since I was a kid, I've always been interested in, like, high-quality information. And, like, you know, even in high school, wanted to improve the way we get information from news. And then in college, built a mini search engine. And then with Exa, like, you know, it's kind of like fulfilling the dream of actually being able to solve all the information needs I wanted as a kid. Yeah, I guess. I would say my entire life has kind of been rotating around this problem, which is pretty cool. Yeah.Swyx [00:00:50]: What'd you enter YC with?Will [00:00:53]: We entered YC with, uh, we are better than Google. Like, Google 2.0.Swyx [00:01:12]: What makes you say that? Like, that's so audacious to come out of the box with.Will [00:01:16]: Yeah, okay, so you have to remember the time. This was summer 2021. And, uh, GPT-3 had come out. Like, here was this magical thing that you could talk to, you could enter a whole paragraph, and it understands what you mean, understands the subtlety of your language. And then there was Google. Uh, which felt like it hadn't changed in a decade, uh, because it really hadn't. And it, like, you would give it a simple query, like, I don't know, uh, shirts without stripes, and it would give you a bunch of results for the shirts with stripes. And so, like, Google could barely understand you, and GBD3 could. And the theory was, what if you could make a search engine that actually understood you? What if you could apply the insights from LLMs to a search engine? And it's really been the same idea ever since. And we're actually a lot closer now, uh, to doing that. Yeah.Alessio [00:01:55]: Did you have any trouble making people believe? Obviously, there's the same element. I mean, YC overlap, was YC pretty AI forward, even 2021, or?Will [00:02:03]: It's nothing like it is today. But, um, uh, there were a few AI companies, but, uh, we were definitely, like, bold. And I think people, VCs generally like boldness, and we definitely had some AI background, and we had a working demo. So there was evidence that we could build something that was going to work. But yeah, I think, like, the fundamentals were there. I think people at the time were talking about how, you know, Google was failing in a lot of ways. And so there was a bit of conversation about it, but AI was not a big, big thing at the time. Yeah. Yeah.Alessio [00:02:33]: Before we jump into Exa, any fun background stories? I know you interned at SpaceX, any Elon, uh, stories? I know you were at Zoox as well, you know, kind of like robotics at Harvard. Any stuff that you saw early that you thought was going to get solved that maybe it's not solved today?Will [00:02:48]: Oh yeah. I mean, lots of things like that. Like, uh, I never really learned how to drive because I believed Elon that self-driving cars would happen. It did happen. And I take them every night to get home. But it took like 10 more years than I thought. Do you still not know how to drive? I know how to drive now. I learned it like two years ago. That would have been great to like, just, you know, Yeah, yeah, yeah. You know? Um, I was obsessed with Elon. Yeah. I mean, I worked at SpaceX because I really just wanted to work at one of his companies. And I remember they had a rule, like interns cannot touch Elon. And, um, that rule actually influenced my actions.Swyx [00:03:18]: Is it, can Elon touch interns? Ooh, like physically?Will [00:03:22]: Or like talk? Physically, physically, yeah, yeah, yeah, yeah. Okay, interesting. He's changed a lot, but, um, I mean, his companies are amazing. Um,Swyx [00:03:28]: What if you beat him at Diablo 2, Diablo 4, you know, like, Ah, maybe.Alessio [00:03:34]: I want to jump into, I know there's a lot of backstory used to be called metaphor system. So, um, and it, you've always been kind of like a prominent company, maybe at least RAI circles in the NSF.Swyx [00:03:45]: I'm actually curious how Metaphor got its initial aura. You launched with like, very little. We launched very little. Like there was, there was this like big splash image of like, this is Aurora or something. Yeah. Right. And then I was like, okay, what this thing, like the vibes are good, but I don't know what it is. And I think, I think it was much more sort of maybe consumer facing than what you are today. Would you say that's true?Will [00:04:06]: No, it's always been about building a better search algorithm, like search, like, just like the vision has always been perfect search. And if you do that, uh, we will figure out the downstream use cases later. It started on this fundamental belief that you could have perfect search over the web and we could talk about what that means. And like the initial thing we released was really just like our first search engine, like trying to get it out there. Kind of like, you know, an open source. So when OpenAI released, uh, ChachBt, like they didn't, I don't know how, how much of a game plan they had. They kind of just wanted to get something out there.Swyx [00:04:33]: Spooky research preview.Will [00:04:34]: Yeah, exactly. And it kind of morphed from a research company to a product company at that point. And I think similarly for us, like we were research, we started as a research endeavor with a, you know, clear eyes that like, if we succeed, it will be a massive business to make out of it. And that's kind of basically what happened. I think there are actually a lot of parallels to, of w between Exa and OpenAI. I often say we're the OpenAI of search. Um, because. Because we're a research company, we're a research startup that does like fundamental research into, uh, making like AGI for search in a, in a way. Uh, and then we have all these like, uh, business products that come out of that.Swyx [00:05:08]: Interesting. I want to ask a little bit more about Metaforesight and then we can go full Exa. When I first met you, which was really funny, cause like literally I stayed in your house in a very historic, uh, Hayes, Hayes Valley place. You said you were building sort of like link prediction foundation model, and I think there's still a lot of foundation model work. I mean, within Exa today, but what does that even mean? I cannot be the only person confused by that because like there's a limited vocabulary or tokens you're telling me, like the tokens are the links or, you know, like it's not, it's not clear. Yeah.Will [00:05:38]: Uh, what we meant by link prediction is that you are literally predicting, like given some texts, you're predicting the links that follow. Yes. That refers to like, it's how we describe the training procedure, which is that we find links on the web. Uh, we take the text surrounding the link. And then we predict. Which link follows you, like, uh, you know, similar to transformers where, uh, you're trying to predict the next token here, you're trying to predict the next link. And so you kind of like hide the link from the transformer. So if someone writes, you know, imagine some article where someone says, Hey, check out this really cool aerospace startup. And they, they say spacex.com afterwards, uh, we hide the spacex.com and ask the model, like what link came next. And by doing that many, many times, you know, billions of times, you could actually build a search engine out of that because then, uh, at query time at search time. Uh, you type in, uh, a query that's like really cool aerospace startup and the model will then try to predict what are the most likely links. So there's a lot of analogs to transformers, but like to actually make this work, it does require like a different architecture than, but it's transformer inspired. Yeah.Alessio [00:06:41]: What's the design decision between doing that versus extracting the link and the description and then embedding the description and then using, um, yeah. What do you need to predict the URL versus like just describing, because you're kind of do a similar thing in a way. Right. It's kind of like based on this description, it was like the closest link for it. So one thing is like predicting the link. The other approach is like I extract the link and the description, and then based on the query, I searched the closest description to it more. Yeah.Will [00:07:09]: That, that, by the way, that is, that is the link refers here to a document. It's not, I think one confusing thing is it's not, you're not actually predicting the URL, the URL itself that would require like the, the system to have memorized URLs. You're actually like getting the actual document, a more accurate name could be document prediction. I see. This was the initial like base model that Exo was trained on, but we've moved beyond that similar to like how, you know, uh, to train a really good like language model, you might start with this like self-supervised objective of predicting the next token and then, uh, just from random stuff on the web. But then you, you want to, uh, add a bunch of like synthetic data and like supervised fine tuning, um, stuff like that to make it really like controllable and robust. Yeah.Alessio [00:07:48]: Yeah. We just have flow from Lindy and, uh, their Lindy started to like hallucinate recrolling YouTube links instead of like, uh, something. Yeah. Support guide. So. Oh, interesting. Yeah.Swyx [00:07:57]: So round about January, you announced your series A and renamed to Exo. I didn't like the name at the, at the initial, but it's grown on me. I liked metaphor, but apparently people can spell metaphor. What would you say are the major components of Exo today? Right? Like, I feel like it used to be very model heavy. Then at the AI engineer conference, Shreyas gave a really good talk on the vector database that you guys have. What are the other major moving parts of Exo? Okay.Will [00:08:23]: So Exo overall is a search engine. Yeah. We're trying to make it like a perfect search engine. And to do that, you have to build lots of, and we're doing it from scratch, right? So to do that, you have to build lots of different. The crawler. Yeah. You have to crawl a bunch of the web. First of all, you have to find the URLs to crawl. Uh, it's connected to the crawler, but yeah, you find URLs, you crawl those URLs. Then you have to process them with some, you know, it could be an embedding model. It could be something more complex, but you need to take, you know, or like, you know, in the past it was like a keyword inverted index. Like you would process all these documents you gather into some processed index, and then you have to serve that. Uh, you had high throughput at low latency. And so that, and that's like the vector database. And so it's like the crawling system, the AI processing system, and then the serving system. Those are all like, you know, teams of like hundreds, maybe thousands of people at Google. Um, but for us, it's like one or two people each typically, but yeah.Alessio [00:09:13]: Can you explain the meaning of, uh, Exo, just the story 10 to the 16th, uh, 18, 18.Will [00:09:20]: Yeah, yeah, yeah, sure. So. Exo means 10 to the 18th, which is in stark contrast to. To Google, which is 10 to the hundredth. Uh, we actually have these like awesome shirts that are like 10th to 18th is greater than 10th to the hundredth. Yeah, it's great. And it's great because it's provocative. It's like every engineer in Silicon Valley is like, what? No, it's not true. Um, like, yeah. And, uh, and then you, you ask them, okay, what does it actually mean? And like the creative ones will, will recognize it. But yeah, I mean, 10 to the 18th is better than 10 to the hundredth when it comes to search, because with search, you want like the actual list of, of things that match what you're asking for. You don't want like the whole web. You want to basically with search filter, the, like everything that humanity has ever created to exactly what you want. And so the idea is like smaller is better there. You want like the best 10th to the 18th and not the 10th to the hundredth. I'm like, one way to say this is like, you know how Google often says at the top, uh, like, you know, 30 million results found. And it's like crazy. Cause you're looking for like the first startups in San Francisco that work on hardware or something. And like, they're not 30 million results like that. What you want is like 325 results found. And those are all the results. That's what you really want with search. And that's, that's our vision. It's like, it just gives you. Perfectly what you asked for.Swyx [00:10:24]: We're recording this ahead of your launch. Uh, we haven't released, we haven't figured out the, the, the name of the launch yet, but what is the product that you're launching? I guess now that we're coinciding this podcast with. Yeah.Will [00:10:36]: So we've basically developed the next version of Exa, which is the ability to get a near perfect list of results of whatever you want. And what that means is you can make a complex query now to Exa, for example, startups working on hardware in SF, and then just get a huge list of all the things that match. And, you know, our goal is if there are 325 startups that match that we find you all of them. And this is just like, there's just like a new experience that's never existed before. It's really like, I don't know how you would go about that right now with current tools and you can apply this same type of like technology to anything. Like, let's say you want, uh, you want to find all the blog posts that talk about Alessio's podcast, um, that have come out in the past year. That is 30 million results. Yeah. Right.Will [00:11:24]: But that, I mean, that would, I'm sure that would be extremely useful to you guys. And like, I don't really know how you would get that full comprehensive list.Swyx [00:11:29]: I just like, how do you, well, there's so many questions with regards to how do you know it's complete, right? Cause you're saying there's only 30 million, 325, whatever. And then how do you do the semantic understanding that it might take, right? So working in hardware, like I might not use the words hardware. I might use the words robotics. I might use the words wearables. I might use like whatever. Yes. So yeah, just tell us more. Yeah. Yeah. Sure. Sure.Will [00:11:53]: So one aspect of this, it's a little subjective. So like certainly providing, you know, at some point we'll provide parameters to the user to like, you know, some sort of threshold to like, uh, gauge like, okay, like this is a cutoff. Like, this is actually not what I mean, because sometimes it's subjective and there needs to be a feedback loop. Like, oh, like it might give you like a few examples and you say, yeah, exactly. And so like, you're, you're kind of like creating a classifier on the fly, but like, that's ultimately how you solve the problem. So the subject, there's a subjectivity problem and then there's a comprehensiveness problem. Those are two different problems. So. Yeah. So you have the comprehensiveness problem. What you basically have to do is you have to put more compute into the query, into the search until you get the full comprehensiveness. Yeah. And I think there's an interesting point here, which is that not all queries are made equal. Some queries just like this blog post one might require scanning, like scavenging, like throughout the whole web in a way that just, just simply requires more compute. You know, at some point there's some amount of compute where you will just be comprehensive. You could imagine, for example, running GPT-4 over the internet. You could imagine running GPT-4 over the entire web and saying like, is this a blog post about Alessio's podcast, like, is this a blog post about Alessio's podcast? And then that would work, right? It would take, you know, a year, maybe cost like a million dollars, but, or many more, but, um, it would work. Uh, the point is that like, given sufficient compute, you can solve the query. And so it's really a question of like, how comprehensive do you want it given your compute budget? I think it's very similar to O1, by the way. And one way of thinking about what we built is like O1 for search, uh, because O1 is all about like, you know, some, some, some questions require more compute than others, and we'll put as much compute into the question as we need to solve it. So similarly with our search, we will put as much compute into the query in order to get comprehensiveness. Yeah.Swyx [00:13:33]: Does that mean you have like some kind of compute budget that I can specify? Yes. Yes. Okay. And like, what are the upper and lower bounds?Will [00:13:42]: Yeah, there's something we're still figuring out. I think like, like everyone is a new paradigm of like variable compute products. Yeah. How do you specify the amount of compute? Like what happens when you. Run out? Do you just like, ah, do you, can you like keep going with it? Like, do you just put in more credits to get more, um, for some, like this can get complex at like the really large compute queries. And like, one thing we do is we give you a preview of what you're going to get, and then you could then spin up like a much larger job, uh, to get like way more results. But yes, there is some compute limit, um, at, at least right now. Yeah. People think of searches as like, oh, it takes 500 milliseconds because we've been conditioned, uh, to have search that takes 500 milliseconds. But like search engines like Google, right. No matter how complex your query to Google, it will take like, you know, roughly 400 milliseconds. But what if searches can take like a minute or 10 minutes or a whole day, what can you then do? And you can do very powerful things. Um, you know, you can imagine, you know, writing a search, going and get a cup of coffee, coming back and you have a perfect list. Like that's okay for a lot of use cases. Yeah.Alessio [00:14:43]: Yeah. I mean, the use case closest to me is venture capital, right? So, uh, no, I mean, eight years ago, I built one of the first like data driven sourcing platforms. So we were. You look at GitHub, Twitter, Product Hunt, all these things, look at interesting things, evaluate them. If you think about some jobs that people have, it's like literally just make a list. If you're like an analyst at a venture firm, your job is to make a list of interesting companies. And then you reach out to them. How do you think about being infrastructure versus like a product you could say, Hey, this is like a product to find companies. This is a product to find things versus like offering more as a blank canvas that people can build on top of. Oh, right. Right.Will [00:15:20]: Uh, we are. We are a search infrastructure company. So we want people to build, uh, on top of us, uh, build amazing products on top of us. But with this one, we try to build something that makes it really easy for users to just log in, put a few, you know, put some credits in and just get like amazing results right away and not have to wait to build some API integration. So we're kind of doing both. Uh, we, we want, we want people to integrate this into all their applications at the same time. We want to just make it really easy to use very similar again to open AI. Like they'll have, they have an API, but they also have. Like a ChatGPT interface so that you could, it's really easy to use, but you could also build it in your applications. Yeah.Alessio [00:15:56]: I'm still trying to wrap my head around a lot of the implications. So, so many businesses run on like information arbitrage, you know, like I know this thing that you don't, especially in investment and financial services. So yeah, now all of a sudden you have these tools for like, oh, actually everybody can get the same information at the same time, the same quality level as an API call. You know, it just kind of changes a lot of things. Yeah.Will [00:16:19]: I think, I think what we're grappling with here. What, what you're just thinking about is like, what is the world like if knowledge is kind of solved, if like any knowledge request you want is just like right there on your computer, it's kind of different from when intelligence is solved. There's like a good, I've written before about like a different super intelligence, super knowledge. Yeah. Like I think that the, the distinction between intelligence and knowledge is actually a pretty good one. They're definitely connected and related in all sorts of ways, but there is a distinction. You could have a world and we are going to have this world where you have like GP five level systems and beyond that could like answer any complex request. Um, unless it requires some. Like, if you say like, uh, you know, give me a list of all the PhDs in New York city who, I don't know, have thought about search before. And even though this, this super intelligence is going to be like, I can't find it on Google, right. Which is kind of crazy. Like we're literally going to have like super intelligences that are using Google. And so if Google can't find them information, there's nothing they could do. They can't find it. So, but if you also have a super knowledge system where it's like, you know, I'm calling this term super knowledge where you just get whatever knowledge you want, then you can pair with a super intelligence system. And then the super intelligence can, we'll never. Be blocked by lack of knowledge.Alessio [00:17:23]: Yeah. You told me this, uh, when we had lunch, I forget how it came out, but we were talking about AGI and whatnot. And you were like, even AGI is going to need search. Yeah.Swyx [00:17:32]: Yeah. Right. Yeah. Um, so we're actually referencing a blog post that you wrote super intelligence and super knowledge. Uh, so I would refer people to that. And this is actually a discussion we've had on the podcast a couple of times. Um, there's so much of model weights that are just memorizing facts. Some of the, some of those might be outdated. Some of them are incomplete or not. Yeah. So like you just need search. So I do wonder, like, is there a maximum language model size that will be the intelligence layer and then the rest is just search, right? Like maybe we should just always use search. And then that sort of workhorse model is just like, and it like, like, like one B or three B parameter model that just drives everything. Yes.Will [00:18:13]: I believe this is a much more optimal system to have a smaller LM. That's really just like an intelligence module. And it makes a call to a search. Tool that's way more efficient because if, okay, I mean the, the opposite of that would be like the LM is so big that can memorize the whole web. That would be like way, but you know, it's not practical at all. I don't, it's not possible to train that at least right now. And Carpathy has actually written about this, how like he could, he could see models moving more and more towards like intelligence modules using various tools. Yeah.Swyx [00:18:39]: So for listeners, that's the, that was him on the no priors podcast. And for us, we talked about this and the, on the Shin Yu and Harrison chase podcasts. I'm doing search in my head. I told you 30 million results. I forgot about our neural link integration. Self-hosted exit.Will [00:18:54]: Yeah. Yeah. No, I do see that that is a much more, much more efficient world. Yeah. I mean, you could also have GB four level systems calling search, but it's just because of the cost of inference. It's just better to have a very efficient search tool and a very efficient LM and they're built for different things. Yeah.Swyx [00:19:09]: I'm just kind of curious. Like it is still something so audacious that I don't want to elide, which is you're, you're, you're building a search engine. Where do you start? How do you, like, are there any reference papers or implementation? That would really influence your thinking, anything like that? Because I don't even know where to start apart from just crawl a bunch of s**t, but there's gotta be more insight than that.Will [00:19:28]: I mean, yeah, there's more insight, but I'm always surprised by like, if you have a group of people who are really focused on solving a problem, um, with the tools today, like there's some in, in software, like there are all sorts of creative solutions that just haven't been thought of before, particularly in the information retrieval field. Yeah. I think a lot of the techniques are just very old, frankly. Like I know how Google and Bing work and. They're just not using new methods. There are all sorts of reasons for that. Like one, like Google has to be comprehensive over the web. So they're, and they have to return in 400 milliseconds. And those two things combined means they are kind of limit and it can't cost too much. They're kind of limited in, uh, what kinds of algorithms they could even deploy at scale. So they end up using like a limited keyword based algorithm. Also like Google was built in a time where like in, you know, in 1998, where we didn't have LMS, we didn't have embeddings. And so they never thought to build those things. And so now they have this like gigantic system that is built on old technology. Yeah. And so a lot of the information retrieval field we found just like thinks in terms of that framework. Yeah. Whereas we came in as like newcomers just thinking like, okay, there here's GB three. It's magical. Obviously we're going to build search that is using that technology. And we never even thought about using keywords really ever. Uh, like we were neural all the way we're building an end to end neural search engine. And just that whole framing just makes us ask different questions, like pursue different lines of work. And there's just a lot of low hanging fruit because no one else is thinking about it. We're just on the frontier of neural search. We just are, um, for, for at web scale, um, because there's just not a lot of people thinking that way about it.Swyx [00:20:57]: Yeah. Maybe let's spell this out since, uh, we're already on this topic, elephants in the room are Perplexity and SearchGPT. That's the, I think that it's all, it's no longer called SearchGPT. I think they call it ChatGPT Search. How would you contrast your approaches to them based on what we know of how they work and yeah, just any, anything in that, in that area? Yeah.Will [00:21:15]: So these systems, there are a few of them now, uh, they basically rely on like traditional search engines like Google or Bing, and then they combine them with like LLMs at the end to, you know, output some power graphics, uh, answering your question. So they like search GPT perplexity. I think they have their own crawlers. No. So there's this important distinction between like having your own search system and like having your own cache of the web. Like for example, so you could create, you could crawl a bunch of the web. Imagine you crawl a hundred billion URLs, and then you create a key value store of like mapping from URL to the document that is technically called an index, but it's not a search algorithm. So then to actually like, when you make a query to search GPT, for example, what is it actually doing it? Let's say it's, it's, it could, it's using the Bing API, uh, getting a list of results and then it could go, it has this cache of like all the contents of those results and then could like bring in the cache, like the index cache, but it's not actually like, it's not like they've built a search engine from scratch over, you know, hundreds of billions of pages. It's like, does that distinction clear? It's like, yeah, you could have like a mapping from URL to documents, but then rely on traditional search engines to actually get the list of results because it's a very hard problem to take. It's not hard. It's not hard to use DynamoDB and, and, and map URLs to documents. It's a very hard problem to take a hundred billion or more documents and given a query, like instantly get the list of results that match. That's a much harder problem that very few entities on, in, on the planet have done. Like there's Google, there's Bing, uh, you know, there's Yandex, but you know, there are not that many companies that are, that are crazy enough to actually build their search engine from scratch when you could just use traditional search APIs.Alessio [00:22:43]: So Google had PageRank as like the big thing. Is there a LLM equivalent or like any. Stuff that you're working on that you want to highlight?Will [00:22:51]: The link prediction objective can be seen as like a neural PageRank because what you're doing is you're predicting the links people share. And so if everyone is sharing some Paul Graham essay about fundraising, then like our model is more likely to predict it. So like inherent in our training objective is this, uh, a sense of like high canonicity and like high quality, but it's more powerful than PageRank. It's strictly more powerful because people might refer to that Paul Graham fundraising essay in like a thousand different ways. And so our model learns all the different ways. That someone refers that Paul Graham, I say, while also learning how important that Paul Graham essay is. Um, so it's like, it's like PageRank on steroids kind of thing. Yeah.Alessio [00:23:26]: I think to me, that's the most interesting thing about search today, like with Google and whatnot, it's like, it's mostly like domain authority. So like if you get back playing, like if you search any AI term, you get this like SEO slop websites with like a bunch of things in them. So this is interesting, but then how do you think about more timeless maybe content? So if you think about, yeah. You know, maybe the founder mode essay, right. It gets shared by like a lot of people, but then you might have a lot of other essays that are also good, but they just don't really get a lot of traction. Even though maybe the people that share them are high quality. How do you kind of solve that thing when you don't have the people authority, so to speak of who's sharing, whether or not they're worth kind of like bumping up? Yeah.Will [00:24:10]: I mean, you do have a lot of control over the training data, so you could like make sure that the training data contains like high quality sources so that, okay. Like if you, if you're. Training data, I mean, it's very similar to like language, language model training. Like if you train on like a bunch of crap, your prediction will be crap. Our model will match the training distribution is trained on. And so we could like, there are lots of ways to tweak the training data to refer to high quality content that we want. Yeah. I would say also this, like this slop that is returned by, by traditional search engines, like Google and Bing, you have the slop is then, uh, transferred into the, these LLMs in like a search GBT or, you know, our other systems like that. Like if slop comes in, slop will go out. And so, yeah, that's another answer to how we're different is like, we're not like traditional search engines. We want to give like the highest quality results and like have full control over whatever you want. If you don't want slop, you get that. And then if you put an LM on top of that, which our customers do, then you just get higher quality results or high quality output.Alessio [00:25:06]: And I use Excel search very often and it's very good. Especially.Swyx [00:25:09]: Wave uses it too.Alessio [00:25:10]: Yeah. Yeah. Yeah. Yeah. Yeah. Like the slop is everywhere, especially when it comes to AI, when it comes to investment. When it comes to all of these things for like, it's valuable to be at the top. And this problem is only going to get worse because. Yeah, no, it's totally. What else is in the toolkit? So you have search API, you have ExaSearch, kind of like the web version. Now you have the list builder. I think you also have web scraping. Maybe just touch on that. Like, I guess maybe people, they want to search and then they want to scrape. Right. So is that kind of the use case that people have? Yeah.Will [00:25:41]: A lot of our customers, they don't just want, because they're building AI applications on top of Exa, they don't just want a list of URLs. They actually want. Like the full content, like cleans, parsed. Markdown. Markdown, maybe chunked, whatever they want, we'll give it to them. And so that's been like huge for customers. Just like getting the URLs and instantly getting the content for each URL is like, and you can do this for 10 or 100 or 1,000 URLs, wherever you want. That's very powerful.Swyx [00:26:05]: Yeah. I think this is the first thing I asked you for when I tried using Exa.Will [00:26:09]: Funny story is like when I built the first version of Exa, it's like, we just happened to store the content. Yes. Like the first 1,024 tokens. Because I just kind of like kept it because I thought of, you know, I don't know why. Really for debugging purposes. And so then when people started asking for content, it was actually pretty easy to serve it. But then, and then we did that, like Exa took off. So the computer's content was so useful. So that was kind of cool.Swyx [00:26:30]: It is. I would say there are other players like Gina, I think is in this space. Firecrawl is in this space. There's a bunch of scraper companies. And obviously scraper is just one part of your stack, but you might as well offer it since you already do it.Will [00:26:43]: Yeah, it makes sense. It's just easy to have an all-in-one solution. And like. We are, you know, building the best scraper in the world. So scraping is a hard problem and it's easy to get like, you know, a good scraper. It's very hard to get a great scraper and it's super hard to get a perfect scraper. So like, and, and scraping really matters to people. Do you have a perfect scraper? Not yet. Okay.Swyx [00:27:05]: The web is increasingly closing to the bots and the scrapers, Twitter, Reddit, Quora, Stack Overflow. I don't know what else. How are you dealing with that? How are you navigating those things? Like, you know. You know, opening your eyes, like just paying them money.Will [00:27:19]: Yeah, no, I mean, I think it definitely makes it harder for search engines. One response is just that there's so much value in the long tail of sites that are open. Okay. Um, and just like, even just searching over those well gets you most of the value. But I mean, there, there is definitely a lot of content that is increasingly not unavailable. And so you could get through that through data partnerships. The bigger we get as a company, the more, the easier it is to just like, uh, make partnerships. But I, I mean, I do see the world as like the future where the. The data, the, the data producers, the content creators will make partnerships with the entities that find that data.Alessio [00:27:53]: Any other fun use case that maybe people are not thinking about? Yeah.Will [00:27:58]: Oh, I mean, uh, there are so many customers. Yeah. What are people doing on AXA? Well, I think dating is a really interesting, uh, application of search that is completely underserved because there's a lot of profiles on the web and a lot of people who want to find love and that I'll use it. They give me. Like, you know, age boundaries, you know, education level location. Yeah. I mean, you want to, what, what do you want to do with data? You want to find like a partner who matches this education level, who like, you know, maybe has written about these types of topics before. Like if you could get a list of all the people like that, like, I think you will unblock a lot of people. I mean, there, I mean, I think this is a very Silicon Valley view of dating for sure. And I'm, I'm well aware of that, but it's just an interesting application of like, you know, I would love to meet like an intellectual partner, um, who like shares a lot of ideas. Yeah. Like if you could do that through better search and yeah.Swyx [00:28:48]: But what is it with Jeff? Jeff has already set me up with a few people. So like Jeff, I think it's my personal exit.Will [00:28:55]: my mom's actually a matchmaker and has got a lot of married. Yeah. No kidding. Yeah. Yeah. Search is built into the book. It's in your jeans. Yeah. Yeah.Swyx [00:29:02]: Yeah. Other than dating, like I know you're having quite some success in colleges. I would just love to map out some more use cases so that our listeners can just use those examples to think about use cases for XR, right? Because it's such a general technology that it's hard to. Uh, really pin down, like, what should I use it for and what kind of products can I build with it?Will [00:29:20]: Yeah, sure. So, I mean, there are so many applications of XR and we have, you know, many, many companies using us for very diverse range of use cases, but I'll just highlight some interesting ones. Like one customer, a big customer is using us to, um, basically build like a, a writing assistant for students who want to write, uh, research papers. And basically like XR will search for, uh, like a list of research papers related to what the student is writing. And then this product has. Has like an LLM that like summarizes the papers to basically it's like a next word prediction, but in, uh, you know, prompted by like, you know, 20 research papers that X has returned. It's like literally just doing their homework for them. Yeah. Yeah. the key point is like, it's, it's, uh, you know, it's, it's, you know, research is, is a really hard thing to do and you need like high quality content as input.Swyx [00:30:08]: Oh, so we've had illicit on the podcast. I think it's pretty similar. Uh, they, they do focus pretty much on just, just research papers and, and that research. Basically, I think dating, uh, research, like I just wanted to like spell out more things, like just the big verticals.Will [00:30:23]: Yeah, yeah, no, I mean, there, there are so many use cases. So finance we talked about, yeah. I mean, one big vertical is just finding a list of companies, uh, so it's useful for VCs, like you said, who want to find like a list of competitors to a specific company they're investigating or just a list of companies in some field. Like, uh, there was one VC that told me that him and his team, like we're using XR for like eight hours straight. Like, like that. For many days on end, just like, like, uh, doing like lots of different queries of different types, like, oh, like all the companies in AI for law or, uh, all the companies for AI for, uh, construction and just like getting lists of things because you just can't find this information with, with traditional search engines. And then, you know, finding companies is also useful for, for selling. If you want to find, you know, like if we want to find a list of, uh, writing assistants to sell to, then we can just, we just use XR ourselves to find that is actually how we found a lot of our customers. Ooh, you can find your own customers using XR. Oh my God. I, in the spirit of. Uh, using XR to bolster XR, like recruiting is really helpful. It is really great use case of XR, um, because we can just get like a list of, you know, people who thought about search and just get like a long list and then, you know, reach out to those people.Swyx [00:31:29]: When you say thought about, are you, are you thinking LinkedIn, Twitter, or are you thinking just blogs?Will [00:31:33]: Or they've written, I mean, it's pretty general. So in that case, like ideally XR would return like the, the really blogs written by people who have just. So if I don't blog, I don't show up to XR, right? Like I have to blog. well, I mean, you could show up. That's like an incentive for people to blog.Swyx [00:31:47]: Well, if you've written about, uh, search in on Twitter and we, we do, we do index a bunch of tweets and then we, we should be able to service that. Yeah. Um, I mean, this is something I tell people, like you have to make yourself discoverable to the web, uh, you know, it's called learning in public, but like, it's even more imperative now because otherwise you don't exist at all.Will [00:32:07]: Yeah, no, no, this is a huge, uh, thing, which is like search engines completely influence. They have downstream effects. They influence the internet itself. They influence what people. Choose to create. And so Google, because they're a keyword based search engine, people like kind of like keyword stuff. Yeah. They're, they're, they're incentivized to create things that just match a lot of keywords, which is not very high quality. Uh, whereas XR is a search algorithm that, uh, optimizes for like high quality and actually like matching what you mean. And so people are incentivized to create content that is high quality, that like the create content that they know will be found by the right person. So like, you know, if I am a search researcher and I want to be found. By XR, I should blog about search and all the things I'm building because, because now we have a search engine like XR that's powerful enough to find them. And so the search engine will influence like the downstream internet in all sorts of amazing ways. Yeah. Uh, whatever the search engine optimizes for is what the internet looks like. Yeah.Swyx [00:33:01]: Are you familiar with the term? McLuhanism? No, it's not. Uh, it's this concept that, uh, like first we shape tools and then the tools shape us. Okay. Yeah. Uh, so there's like this reflexive connection between the things we search for and the things that get searched. Yes. So like once you change the tool. The tool that searches the, the, the things that get searched also change. Yes.Will [00:33:18]: I mean, there was a clear example of that with 30 years of Google. Yeah, exactly. Google has basically trained us to think of search and Google has Google is search like in people's heads. Right. It's one, uh, hard part about XR is like, uh, ripping people away from that notion of search and expanding their sense of what search could be. Because like when people think search, they think like a few keywords, or at least they used to, they think of a few keywords and that's it. They don't think to make these like really complex paragraph long requests for information and get a perfect list. ChatGPT was an interesting like thing that expanded people's understanding of search because you start using ChatGPT for a few hours and you go back to Google and you like paste in your code and Google just doesn't work and you're like, oh, wait, it, Google doesn't do work that way. So like ChatGPT expanded our understanding of what search can be. And I think XR is, uh, is part of that. We want to expand people's notion, like, Hey, you could actually get whatever you want. Yeah.Alessio [00:34:06]: I search on XR right now, people writing about learning in public. I was like, is it gonna come out with Alessio? Am I, am I there? You're not because. Bro. It's. So, no, it's, it's so about, because it thinks about learning, like in public, like public schools and like focuses more on that. You know, it's like how, when there are like these highly overlapping things, like this is like a good result based on the query, you know, but like, how do I get to Alessio? Right. So if you're like in these subcultures, I don't think this would work in Google well either, you know, but I, I don't know if you have any learnings.Swyx [00:34:40]: No, I'm the first result on Google.Alessio [00:34:42]: People writing about learning. In public, you're not first result anymore, I guess.Swyx [00:34:48]: Just type learning public in Google.Alessio [00:34:49]: Well, yeah, yeah, yeah, yeah. But this is also like, this is in Google, it doesn't work either. That's what I'm saying. It's like how, when you have like a movement.Will [00:34:56]: There's confusion about the, like what you mean, like your intention is a little, uh. Yeah.Alessio [00:35:00]: It's like, yeah, I'm using, I'm using a term that like I didn't invent, but I'm kind of taking over, but like, they're just so much about that term already that it's hard to overcome. If that makes sense, because public schools is like, well, it's, it's hard to overcome.Will [00:35:14]: Public schools, you know, so there's the right solution to this, which is to specify more clearly what you mean. And I'm not expecting you to do that, but so the, the right interface to search is actually an LLM.Swyx [00:35:25]: Like you should be talking to an LLM about what you want and the LLM translates its knowledge of you or knowledge of what people usually mean into a query that excellent uses, which you have called auto prompts, right?Will [00:35:35]: Or, yeah, but it's like a very light version of that. And really it's just basically the right answer is it's the wrong interface and like very soon interface to search and really to everything will be LLM. And the LLM just has a full knowledge of you, right? So we're kind of building for that world. We're skating to where the puck is going to be. And so since we're moving to a world where like LLMs are interfaced to everything, you should build a search engine that can handle complex LLM queries, queries that come from LLMs. Because you're probably too lazy, I'm too lazy too, to write like a whole paragraph explaining, okay, this is what I mean by this word. But an LLM is not lazy. And so like the LLM will spit out like a paragraph or more explaining exactly what it wants. You need a search engine that can handle that. Traditional search engines like Google or Bing, they're actually... Designed for humans typing keywords. If you give a paragraph to Google or Bing, they just completely fail. And so Exa can handle paragraphs and we want to be able to handle it more and more until it's like perfect.Alessio [00:36:24]: What about opinions? Do you have lists? When you think about the list product, do you think about just finding entries? Do you think about ranking entries? I'll give you a dumb example. So on Lindy, I've been building the spot that every week gives me like the top fantasy football waiver pickups. But every website is like different opinions. I'm like, you should pick up. These five players, these five players. When you're making lists, do you want to be kind of like also ranking and like telling people what's best? Or like, are you mostly focused on just surfacing information?Will [00:36:56]: There's a really good distinction between filtering to like things that match your query and then ranking based on like what is like your preferences. And ranking is like filtering is objective. It's like, does this document match what you asked for? Whereas ranking is more subjective. It's like, what is the best? Well, it depends what you mean by best, right? So first, first table stakes is let's get the filtering into a perfect place where you actually like every document matches what you asked for. No surgeon can do that today. And then ranking, you know, there are all sorts of interesting ways to do that where like you've maybe for, you know, have the user like specify more clearly what they mean by best. You could do it. And if the user doesn't specify, you do your best, you do your best based on what people typically mean by best. But ideally, like the user can specify, oh, when I mean best, I actually mean ranked by the, you know, the number of people who visited that site. Let's say is, is one example ranking or, oh, what I mean by best, let's say you're listing companies. What I mean by best is like the ones that have, uh, you know, have the most employees or something like that. Like there are all sorts of ways to rank a list of results that are not captured by something as subjective as best. Yeah. Yeah.Alessio [00:38:00]: I mean, it's like, who are the best NBA players in the history? It's like everybody has their own. Right.Will [00:38:06]: Right. But I mean, the, the, the search engine should definitely like, even if you don't specify it, it should do as good of a job as possible. Yeah. Yeah. No, no, totally. Yeah. Yeah. Yeah. Yeah. It's a new topic to people because we're not used to a search engine that can handle like a very complex ranking system. Like you think to type in best basketball players and not something more specific because you know, that's the only thing Google could handle. But if Google could handle like, oh, basketball players ranked by like number of shots scored on average per game, then you would do that. But you know, they can't do that. So.Swyx [00:38:32]: Yeah. That's fascinating. So you haven't used the word agents, but you're kind of building a search agent. Do you believe that that is agentic in feature? Do you think that term is distracting?Will [00:38:42]: I think it's a good term. I do think everything will eventually become agentic. And so then the term will lose power, but yes, like what we're building is agentic it in a sense that it takes actions. It decides when to go deeper into something, it has a loop, right? It feels different from traditional search, which is like an algorithm, not an agent. Ours is a combination of an algorithm and an agent.Swyx [00:39:05]: I think my reflection from seeing this in the coding space where there's basically sort of classic. Framework for thinking about this stuff is the self-driving levels of autonomy, right? Level one to five, typically the level five ones all failed because there's full autonomy and we're not, we're not there yet. And people like control. People like to be in the loop. So the, the, the level ones was co-pilot first and now it's like cursor and whatever. So I feel like if it's too agentic, it's too magical, like, like a, like a one shot, I stick a, stick a paragraph into the text box and then it spits it back to me. It might feel like I'm too disconnected from the process and I don't trust it. As opposed to something where I'm more intimately involved with the research product. I see. So like, uh, wait, so the earlier versions are, so if trying to stick to the example of the basketball thing, like best basketball player, but instead of best, you, you actually get to customize it with like, whatever the metric is that you, you guys care about. Yeah. I'm still not a basketballer, but, uh, but, but, you know, like, like B people like to be in my, my thesis is that agents level five agents failed because people like to. To kind of have drive assist rather than full self-driving.Will [00:40:15]: I mean, a lot of this has to do with how good agents are. Like at some point, if agents for coding are better than humans at all tests and then humans block, yeah, we're not there yet.Swyx [00:40:25]: So like in a world where we're not there yet, what you're pitching us is like, you're, you're kind of saying you're going all the way there. Like I kind of, I think all one is also very full, full self-driving. You don't get to see the plan. You don't get to affect the plan yet. You just fire off a query and then it goes away for a couple of minutes and comes back. Right. Which is effectively what you're saying you're going to do too. And you think there's.Will [00:40:42]: There's a, there's an in-between. I saw. Okay. So in building this product, we're exploring new interfaces because what does it mean to kick off a search that goes and takes 10 minutes? Like, is that a good interface? Because what if the search is actually wrong or it's not exactly, exactly specified to what you mean, which is why you get previews. Yeah. You get previews. So it is iterative, but ultimately once you've specified exactly what you mean, then you kind of do just want to kick off a batch job. Right. So perhaps what you're getting at is like, uh, there's this barrier with agents where you have to like explain the full context of what you mean, and a lot of failure modes happen when you have, when you don't. Yeah. There's failure modes from the agent, just not being smart enough. And then there's failure modes from the agent, not understanding exactly what you mean. And there's a lot of context that is shared between humans that is like lost between like humans and, and this like new creature.Alessio [00:41:32]: Yeah. Yeah. Because people don't know what's going on. I mean, to me, the best example of like system prompts is like, why are you writing? You're a helpful assistant. Like. Of course you should be an awful, but people don't yet know, like, can I assume that, you know, that, you know, it's like, why did the, and now people write, oh, you're a very smart software engineer, but like, you never made, you never make mistakes. Like, were you going to try and make mistakes before? So I think people don't yet have an understanding, like with, with driving people know what good driving is. It's like, don't crash, stay within kind of like a certain speed range. It's like, follow the directions. It's like, I don't really have to explain all of those things. I hope. But with. AI and like models and like search, people are like, okay, what do you actually know? What are like your assumptions about how search, how you're going to do search? And like, can I trust it? You know, can I influence it? So I think that's kind of the, the middle ground, like before you go ahead and like do all the search, it's like, can I see how you're doing it? And then maybe help show your work kind of like, yeah, steer you. Yeah. Yeah.Will [00:42:32]: No, I mean, yeah. Sure. Saying, even if you've crafted a great system prompt, you want to be part of the process itself. Uh, because the system prompt doesn't, it doesn't capture everything. Right. So yeah. A system prompt is like, you get to choose the person you work with. It's like, oh, like I want, I want a software engineer who thinks this way about code. But then even once you've chosen that person, you can't just give them a high level command and they go do it perfectly. You have to be part of that process. So yeah, I agree.Swyx [00:42:58]: Just a side note for my system, my favorite system, prompt programming anecdote now is the Apple intelligence system prompt that someone, someone's a prompt injected it and seen it. And like the Apple. Intelligence has the words, like, please don't, don't hallucinate. And it's like, of course we don't want you to hallucinate. Right. Like, so it's exactly that, that what you're talking about, like we should train this behavior into the model, but somehow we still feel the need to inject into the prompt. And I still don't even think that we are very scientific about it. Like it, I think it's almost like cargo culting. Like we have this like magical, like turn around three times, throw salt over your shoulder before you do something. And like, it worked the last time. So let's just do it the same time now. And like, we do, there's no science to this.Will [00:43:35]: I do think a lot of these problems might be ironed out in future versions. Right. So, and like, they might, they might hide the details from you. So it's like, they actually, all of them have a system prompt. That's like, you are a helpful assistant. You don't actually have to include it, even though it might actually be the way they've implemented in the backend. It should be done in RLE AF.Swyx [00:43:52]: Okay. Uh, one question I was just kind of curious about this episode is I'm going to try to frame this in terms of this, the general AI search wars, you know, you're, you're one player in that, um, there's perplexity, chat, GPT, search, and Google, but there's also like the B2B side, uh, we had. Drew Houston from Dropbox on, and he's competing with Glean, who've, uh, we've also had DD from, from Glean on, is there an appetite for Exa for my company's documents?Will [00:44:19]: There is appetite, but I think we have to be disciplined, focused, disciplined. I mean, we're already taking on like perfect web search, which is a lot. Um, but I mean, ultimately we want to build a perfect search engine, which definitely for a lot of queries involves your, your personal information, your company's information. And so, yeah, I mean, the grandest vision of Exa is perfect search really over everything, every domain, you know, we're going to have an Exa satellite, uh, because, because satellites can gather information that, uh, is not available publicly. Uh, gotcha. Yeah.Alessio [00:44:51]: Can we talk about AGI? We never, we never talk about AGI, but you had, uh, this whole tweet about, oh, one being the biggest kind of like AI step function towards it. Why does it feel so important to you? I know there's kind of like always criticism and saying, Hey, it's not the smartest son is better. It's like, blah, blah, blah. What? You choose C. So you say, this is what Ilias see or Sam see what they will see.Will [00:45:13]: I've just, I've just, you know, been connecting the dots. I mean, this was the key thing that a bunch of labs were working on, which is like, can you create a reward signal? Can you teach yourself based on a reward signal? Whether you're, if you're trying to learn coding or math, if you could have one model say, uh, be a grading system that says like you have successfully solved this programming assessment and then one model, like be the generative system. That's like, here are a bunch of programming assessments. You could train on that. It's basically whenever you could create a reward signal for some task, you could just generate a bunch of tasks for yourself. See that like, oh, on two of these thousand, you did well. And then you just train on that data. It's basically like, I mean, creating your own data for yourself and like, you know, all the labs working on that opening, I built the most impressive product doing that. And it's just very, it's very easy now to see how that could like scale to just solving, like, like solving programming or solving mathematics, which sounds crazy, but everything about our world right now is crazy.Alessio [00:46:07]: Um, and so I think if you remove that whole, like, oh, that's impossible, and you just think really clearly about like, what's now possible with like what, what they've done with O1, it's easy to see how that scales. How do you think about older GPT models then? Should people still work on them? You know, if like, obviously they just had the new Haiku, like, is it even worth spending time, like making these models better versus just, you know, Sam talked about O2 at that day. So obviously they're, they're spending a lot of time in it, but then you have maybe. The GPU poor, which are still working on making Lama good. Uh, and then you have the follower labs that do not have an O1 like model out yet. Yeah.Will [00:46:47]: This kind of gets into like, uh, what will the ecosystem of, of models be like in the future? And is there room is, is everything just gonna be O1 like models? I think, well, I mean, there's definitely a question of like inference speed and if certain things like O1 takes a long time, because that's the thing. Well, I mean, O1 is, is two things. It's like one it's it's use it's bootstrapping itself. It's teaching itself. And so the base model is smarter. But then it also has this like inference time compute where it could like spend like many minutes or many hours thinking. And so even the base model, which is also fast, it doesn't have to take minutes. It could take is, is better, smarter. I believe all models will be trained with this paradigm. Like you'll want to train on the best data, but there will be many different size models from different, very many different like companies, I believe. Yeah. Because like, I don't, yeah, I mean, it's hard, hard to predict, but I don't think opening eye is going to dominate like every possible LLM for every possible. Use case. I think for a lot of things, like you just want the fastest model and that might not involve O1 methods at all.Swyx [00:47:42]: I would say if you were to take the exit being O1 for search, literally, you really need to prioritize search trajectories, like almost maybe paying a bunch of grad students to go research things. And then you kind of track what they search and what the sequence of searching is, because it seems like that is the gold mine here, like the chain of thought or the thinking trajectory. Yeah.Will [00:48:05]: When it comes to search, I've always been skeptical. I've always been skeptical of human labeled data. Okay. Yeah, please. We tried something at our company at Exa recently where me and a bunch of engineers on the team like labeled a bunch of queries and it was really hard. Like, you know, you have all these niche queries and you're looking at a bunch of results and you're trying to identify which is matched to query. It's talking about, you know, the intricacies of like some biological experiment or something. I have no idea. Like, I don't know what matches and what, what labelers like me tend to do is just match by keyword. I'm like, oh, I don't know. Oh, like this document matches a bunch of keywords, so it must be good. But then you're actually completely missing the meaning of the document. Whereas an LLM like GB4 is really good at labeling. And so I actually think like you just we get by, which we are right now doing using like LLM

In this episode, we're giving you the best Amazon and Walmart strategy clips of 2024 so that you can start off 2025 with a leg up on your competition. ► Instagram: instagram.com/serioussellerspodcast ► Free Amazon Seller Chrome Extension: https://h10.me/extension ► Sign Up For Helium 10: https://h10.me/signup  (Use SSP10 To Save 10% For Life) ► Learn How To Sell on Amazon: https://h10.me/ft ► Watch The Podcasts On YouTube: youtube.com/@Helium10/videos Welcome to this special annual recap episode of the Serious Sellers Podcast, where we bring you the most impactful strategies from the past year to give your e-commerce business a competitive edge in 2025. Join us as we explore the essentials of selecting verified manufacturers and the importance of third-party verification in ensuring accurate information. We'll discuss the advantages of trade assurance for payment protection and the significance of management certifications like BSCI and ISO, which indicate high-quality factory standards and social compliance. Additionally, we touch on regional manufacturing specializations, exemplified by the production of egg dispensers, and the importance of measuring the halo impact of ad strategies on total sales and rankings using metrics like TACoS and cost per customer acquisition. Listen in as we discuss strategies for international Amazon success, highlighting a thriving American brand's expansion into Amazon Japan. We'll explore the strategic benefits of entering the Japanese market, such as lower PPC costs and favorable tax conditions, which contribute to higher profit margins. Patience, quality products, and strong supplier relationships are emphasized as key differentiators from competitors. We also explore optimizing Amazon PPC campaigns with lifecycle-based rules and the power of using index images with numbered benefits to effectively communicate value in product listings. Discover effective strategies for online marketplaces as we recount past challenges and successes in sourcing and selling products in the U.S. market. Learn about creative approaches to finding less visible suppliers and the importance of clear communication and relationship-building. We also highlight the effectiveness of Target's marketplace and the strategic advantages of being indexed on Google to enhance Amazon rankings. Finally, we'll cover the critical importance of using correct HTS codes to avoid costly import tariff mistakes, sharing a personal experience that led to significant cost savings. Tune in and equip yourself with these valuable insights to kick off 2025 strong. In episode 628 of the Serious Sellers Podcast, we discuss: 00:00 - SSP Top 20 Strategies of 2024 02:02 - Selecting Verified Manufacturers for Trust 09:53 - Keyword Analysis and Visibility Tracking 12:25 - Strategies for International Amazon Success 19:36 - Effective Strategies for Online Marketplaces 20:06 - Leveraging Google for Business Growth 23:43 - Optimizing Amazon Listings for Google Images 25:40 - Optimizing Amazon Listings for Sales 32:12 - Enhancing Amazon Listings With COSMO 33:29 - Avoiding Costly Import Tariff Mistakes Transcript   Bradley Sutton: Today we're giving you the best strategy clips of the year so that you can start off 2025 with a leg up on your competition. How cool is that? Pretty cool, I think. Hello everybody, and welcome to another episode of the Serious Sellers Podcast by Helium 10. I'm your host, Bradley Sutton, and this is the show. That's a completely BS-free, unscripted and unrehearsed organic conversation about serious strategies for serious sellers of any level in the e-commerce world. And, like we do every year and we have been doing this since we started in what was it 2018, 2019? we do a recap episode where I handpick some of the best strategies of the entire year. Every year, we go through about 100 episodes a year, not including the weekly buzz, where we have a lot of guests and everybody has great strategies. It's really hard to pick some of the top ones, but what I did is me and the team got together and pulled out some of the top strategies that you guys had talked about in social media and such that you liked, and we put it together so that you could get a leg up on the competition now that we're at the beginning here of 2025.   Bradley Sutton: And so these are strategies that are not out of date. They're still valid. There are some that already, within a few months, became out of date. They're still valid. There are some that already, within a few months, became out of date. We're not including them here. So, guys, I hope you enjoy this episode. Get your pen and paper out. I want each and every one of you to make it your homework to pick five, at least five of these that we're about to get. I think we're doing about 20 here, but do five that most apply to you and your business. Not everything applies to everybody equally. Pick five out of these and implement it this month in your business, or at least make a plan for it. All right, so let's go ahead and see the top strategies of the year.   Kian Golzari: So the first thing you did was you selected verified manufacturers. And what's that for? It means any information that they provide on their listing, whether it be number of years in business, how many staff they have, what certificates they have, what patents they have, what products they have, what does their production line look like, the images and videos in the factory. That's all been verified by a third party, meaning InterTech, SGS, TUV. One of these very reputable companies have gone in and verified all the information is true, whereas if we didn't work with verified suppliers, then whatever information they want to put there, we just have to sort of take their word for it. So verified is the most important thing to search for first. Then, on the left-hand side of the page, you'll see trade assurance right, I would always click that as well and trade assurance just means that your payment is protected. So if you've ordered an egg dispenser which holds, you know, 20 eggs and you do the production and you receive one which only holds 10 eggs, then the trade assurance will protect you and it will refund your order because you've selected that right. That's just a little bit of a safety net important for, like you know, new sellers, right. And then, as you scroll down on the left-hand side of the page, you'll see something that says management certification, right. And if you scroll down a little bit more, yeah. So you see like BSCI and you see Zedek, you see ISO. I always like to select BSCI and ISO. So BSCI is your business social compliance initiative and ISO is just a really high-quality standard and this just basically means these are factory certificates that they have. So, uh, BSCI will go in and they'll check, like you know, um how many years you've been in business. Do you have, like, fire extinguishers? Do you have adequate lighting? Do you have safety exits? Like we've checked the dormitories, we've checked like the canteen where the workers eat. So it's kind of like gives you confidence that you're working for a very, very good factory, right. So now, if we go back to the top of the list, right, we've. Now we've searched by manufacturers, we've got verified manufacturers, we've got trade assurance and we've got factories which have, you know, BSCI and ISO certification. So now, as I'm scrolling down the list, like if you zoom in on the company names, like the first word in the company name is always the city or the province in which that factory is located.   Kian Golzari: So sometimes, like the factories, like electronics are made in Shenzhen, backpacks are normally made in like Shenzhou. Like furniture, like steel tubing for furniture, chairs is made like Yongkang. So I'm just trying to get familiar. Is there an area which specializes in egg dispensers? Maybe not because it's such a niche product, right, that maybe you could make it, make it anywhere. But as I scroll down, I'm trying to see, like, is there one name that pops up more frequent than others and in that area which specializes in that product? But I see Ningbo has probably popped up a few times, right? So, but anyway, it doesn't matter. If Ningbo had popped out like eight out of nine times, I would say, right, well, that's the region we need to be ordering from.   Bradley Sutton: Interesting.   Gefen Laredo: You know ACOS is great, but obviously this is TACoS Tuesday and TACoS is the metric of your total sales.   Carrie Miller: Yes.   Gefen Laredo: And so when we're looking at total sales something that we brought in and I know it's a little vague, but we really looked at the halo impact of ad strategies and how they impacted ranking and total sales, right. And so when we focused our ad strategy, maybe on a cost per customer acquisition model, maybe on a TACoS model, and we look to really prioritize, hey, where are we showing up, right? So, if, if, if we're driving all this traffic and we have a 20 percent conversion rate, let's say, on this keyword, are we tracking using, using uh, using a Helium 10, of course, um, are we tracking that ranking properly? To say, hey, we started running these ads aggressively on August 1st and if we have been tracking ranking on that keyword for the last two months since going aggressive on that term, where are we ranking now and how have sales changed? and are there broader KPIs that we're measuring outside of just direct ad revenue? And that worked really well for us because we centered that around tentpole events and this is a really big strategy of ours. That is incredibly complex, it takes a whole village to actually execute. But when we focus our customer acquisition and ranking models around major times in the year so think Prime Day, think Fall, Prime Day, Black Friday, Cyber Monday, holiday and then, of course, if you're a one-off brand, if you I don't know are ski related, then obviously your season is January to March. You know like there are differences, but really peak seasons. If you're able to focus your growth model around the times that are going to give you the most reward, then that worked really well for us last year and we expect to see a lot more of that this year, especially as we all expect people are going to be more deal oriented. It's a constant battle for margins, so the better rank you are, the more organic sales you drive, the better your TACoS is.   Ben Webber:                           Several years ago we were about to stock out of as you know, we sell a lot of fourth quarter products and kind of joke toy products and we're about to stock out of one that we sold between 800 and 1000 units a day of which is a fairly substantial issue. So we actually loaded up a cargo van and drove the cargo van to Amazon, talked our way through the front gates to deliver it and they took it, and so we did that once, then we did it again and we got through again. The third time they're like no, you can't do this, and so like okay, but somehow, like no, you, you can't do this, and so like okay, but somehow, we have to be able to do this. So we looked into carrier central and figure out how we could become a last mile rider, which is incredibly easy it takes about 15 minutes to fill out a form and then you have to show that you can back in and out of a parking spot incredibly, incredibly easy. But so in that January we bought a truck and the rest is history from there. But it came about because we were about to stock out and panicked and we're like, well, what's the worst that can happen?   Silas Moestrup Pedersen: And one of the things that I recommend to every time that we have a new client or meet someone is to narrow in on fewer skills. It sounds quite simple, right, but what we do every time is that if you have a big catalog A, B, C and D products and then A products they get a special treatment compared to B, C and D. It could even be, if your catalog is massive, you only focus your ad spend on A products. Same thing from a content perspective. Those are the ones that get the most love in terms of title, bullet point, backend attributes, et cetera, descriptions. So it's just having that focus on fewer products, I think, is number one. Then, if you can automate your reporting, we have that in Looker automated so that you don't have to necessarily sit and look at the data and pull Excel spreadsheets et cetera it just saves you so much time. If you're capable of doing it and spending time on it, then I think. Thirdly, we talked a little bit about it, but I think taking the time to do super solid keyword research from the get-go Like get into Magnet, get into Amazon's data sources, get into Cerebro, look for all your competitors' keywords et cetera understand what those A keywords are, and those A keywords are the only thing that you focus on in the start. Those are the ones that go into your rank campaigns, that they go into your manual campaigns, et cetera, and that those are the ones that just like where you track everything through Like a little hack could be for your A products. Every week you use a repro. Every other week you put in your A product and then you export all the data for that. You take a spreadsheet. In column A you say this is the date when I pulled the. This is the date of either. I pulled the data, this is the ASIN you put in the ASIN that you pulled the data for. Then you make a formula.   Silas Moestrup Pedersen: You can just ask ChatGPT where, based on the paid and organic rank, you say whether you classified the keyword as being on page one, two, three or four, and then you pull this data in this way every single week for maybe two months when you're running a new test or something like that. You take all the data, you put it into a pivot table and boom, then you would have an overview and a graph of how many like your all your page one, two, three, four positions across your entire catalog and you could even put a filter on up in the top and then you can sort by ASIN and then you basically have your own visibility tool where you can see your paid on your organic visibility on a weekly level at an Asian level. And you can use that to take all those keywords If you're ranking let's say page three or two or something like that put them into a rank campaign. If you feel like they're good, you can take all the keywords where you're on page two, maybe put them in the title, et cetera. So, like building those systems, that allows you to scale something consistently.   Bradley Sutton:                           What was your gross sales yesterday, last week, last year? More importantly, what are your profits after all your cost of selling on Amazon? Did you pay any storage charges to Amazon? How much did you spend on PPC? Find out these key metrics and more by using the Helium 10 tool Profits. For more information, go to h10.me forward slash profits.   Cara Sayer: So one of the biggest things was the fact that I do think a lot of Amazon sellers don't really have a brand. They just have a name for a business or a name for something that they use and they don't really have a what I'd call a true brand. And they don't always. I think sometimes also, existing only on Amazon makes you lose perspective on you know how normal businesses work, like businesses that aren't based on Amazon, and so you know a lot of businesses. I mean, I think throughout life, people buy from people and I think that's so important to remember that, even on Amazon, one of the reasons why Amazon focuses so heavily on A plus listings and now they're bringing in the premium A plus and all the rest of it because Amazon knows right. You know me quite a few years now and I've always banged on about brand. I've always banged on about having a story. Tell your story. It doesn't have to be your story necessarily, it could be the product story, but you need to have something that differentiates you. And even then, I was chatting to someone at the conference earlier on and I was saying the thing is that sometimes it's not even the fact that you're selling different products, it's the way that you curate them right. So it's the collection of products that you've chosen to sell under your brand name says something.   Nick Katz: So one of our clients is an international brand. They're an American registered company and they last year they cleared seven figures and we're definitely looking to do a lot more this year. That's in two years. They're doing very, very well in America, they sell in Europe and they sell in Canada. But the Japanese sales are now almost comparable to the to the us sales, but the profit margins are a lot higher.   Bradley Sutton:                           That was about my second question.   Nick Katz: Yeah, because you know things like the PPC is a hell of a lot cheaper. The ACOS for the account is about uh, I think it's about eight, nine percent now. The TACoS is about three or four percent. It's the kind of figures you can't really get in the US. So actually in theory you could sell a lot less in Japan and still end up with the same kind of profit as you could in the US. But obviously if you're getting sales close to the US you're probably going to have much, much higher margins. Japan generally is cheaper. It's cheaper tax as well if you are off the threshold to pay tax. But if you're under 10 million yen, which is probably about 60,000, 70,000 US, if you're under that in sales, you don't have to pay consumption tax. There is no tax. So anybody like me selling in Europe who gets absolutely lost by the tax authorities there, paying 19, 20, 21, 23% in some of the regions in Europe, you could be selling 50, 60,000 US in Japan and not have to pay any consumption tax whatsoever. So there are definite advantages to selling in Japan.   Bradley Sutton: What are some of the things that set you apart from maybe the 10 other matcha people who maybe have started and gone out of business, you know, because they didn't have your strategy? What do you think set you apart from others?   Sam: Well, I think a handful of things. The first one is okay, so I think you can use. You can rely on Amazon PPC. You can look at your search term impression share reports, you can look at your keyword ranking and all that kind of stuff and that will help you in the short run. But honestly, the thing that really helped us the most was patience and making sure that your product is on a sensory level it's actually good and people like it. Once you have those two things covered, then you just need to get people to try it, get them to tell their friends, and then their friends who are interested in Marchable buy. Then they are buying again and then this whole thing kind of grows by itself. Your PPC and all of these other tools that you have are really just like fuel that you add to this engine.   Singchuen: And on the other side of things is, obviously you kind of need to make sure that you treat your suppliers well as well. Make sure that they understand what you're going through and make sure that you try to understand what they're going through. If language is a barrier, hire an interpreter, right, it's not too difficult. Decency goes both ways. So you may be pressed, but you've got to recognize that the factories themselves, they are pressed as well. So working together for a compromise, understanding each other and not throwing too much Just to be a little bit more understanding towards each other, goes a long way. A bit more understanding towards each other goes a long way. I think what tends to happen is that if you're not patient, as Sam has mentioned, you may cut off communications with factories that may help you in the future, and you don't want to do that.   Destaney Wishon: I think the biggest things that we look at is we create rules for the different outcomes we want. If we're launching a brand-new product, then we're creating rules that are based off sales. So we're going to be taking a deep dive into, hey, what is the conversion rate and what is the sales? And we're going to build rules for maximizing that increased bid when I have a certain conversion rate. On the flip side, if our goal is profitability, we're going to work backwards from our ACOS or RoAS goal. We're going to say, hey, let's build rules that are based on lowering bids when our ACOS is too high, and maybe layering in our conversion rates also low, let's go even lower, right. So those are the two simplest ones that we look at, but it really needs to be strategic. You can create rules that are based off the phase your product's in, whether it's launch, consistency, profitability, organic rank. You can create rules based off your overall business outcomes. Which is always an important one is what is that key RoAS that you're going to optimize for all of your campaigns, but just making sure not to overcomplicate it in the beginning, right. Once you start to understand the correlation between CPC and RoAS, then you can start building in a little bit more customization around lifecycle and things like that.   Kevin King: This is how you been converting like crazy with what? what do you call an index image? This he calls it the uh, it's the image in your listing that will be the top reasons why your product is the best. This is not your main photo. This is not your photo number one. This is what he calls this photo number two and it's an index of of your products is why I think it's why he calls it the index image, and what he says is you need to number the benefits. A lot of of people are using call-outs, they use infographics, but they don't number them. So you want to actually have numbers like this. So this should be something like this should be your second image the five reasons you love, or the seven reasons or the three reasons.   Odd numbers are always better than even numbers. Three, five or seven or nine always work the best. But here he's got the five and look, there's big, there's numbers. That's important. He just doesn't list them. People like order and when they see numbers, their mind can sort it and they can read it quickly and it makes sense to them. So the numbering system here is critical, not just the fact that he put the main point, the main benefit and capital, and then explained it in. I mean in bold and a little bit larger than explained everything else below it in light blue, but he's got these numbers. That's the critical thing is numbering it.   Bradley Sutton: Maybe this is a little bit of the sexy side of patents, but you've talked before about how patents doing patent searches can actually be a form of product research and finding a product to sell on Amazon. How in the world is that possible?   Rich Goldstein: Yeah, absolutely it's true, because the way that the patent system works, once a patent expires, it's fair game for anyone to use it. So a utility patent lasts for 20 years and a design patent lasts for 15. But once that patent expires, anyone can make that product and, at the same time, keep in mind that a lot of people have an idea for a product, they get it patented, but they never do the research, they never learn about the process enough to actually get that product launched, and so there are a lot of great ideas that have been patented that are just in the patent archives and they've never actually been put on the market. There are some lousy ideas, but there are also some great ideas, and so if you know how and you search the patent record for expired patents, you can find ideas for really great potential products.   Tom - Honest FBA: We dabbled with the US a few times in the past and Thomas Net is really popular. You see, it's spoken about quite a lot as a place, as a resource. Honestly, we never had any success there. There was a time when we were the MOQs are always insanely high and there was a product previously that we agreed to the MOQ. It was something like 10 or 20,000 units. It was pretty high. And there was a product previously that we agreed to the MOQ. It was something like 10 or 20,000 units. It was pretty big. And we were like, okay, we'll go for it, but can you just repackage them into a different kind of mix? And they just said, nah, nah, don't fancy it. And we were like, right, okay. So we kind of banged our head against the wall. So now a little-known site called Google is honestly the best bet, so like, but I'm not talking page one at Google. You've got to dig. So put on a VPN. If you're somewhere like we are, like in Spain, put on a US VPN and then get down to like pages five, six, seven, eight, get in there. And then I just hammer a lot of emails out, but a lot of the websites that you find down in those stages or those pages. They're not good at SEO, they're generally kind of old sites, but you're finding older, established businesses so and often you'll find a phone number. So one of the best lessons I say is like get on the phone and just ring them up and you can save months of time, like the guy who ended up.   Tom - Honest FBA: One of the guys who ended up working with had a phone call with him on the first day. I found it and we ended up. We're now doing two products with him already. We've got another three lined up and he had nothing to do with the niche we're in. He was in so we're in pets. He was in humans. He was in food. I just gave him a call, explained the brand vision, what we're trying to do. He got really excited. He's now helping us source new ingredients. He's coming to me with product ideas. He's now going to do a whole range of products for us. So that was one of the beauties is like having that communication line and being able to really explain yourself has been massive. We are still sourcing in China, by the way. We still think it's a really viable option, but having this US option as well, there's so many benefits to it.   Grace Kopplin: In terms of Walmart, that's always been a strategy for us. Transparently, Walmart just hasn't been a volume driver for us. It's been steady but it hasn't really been a place that's warranted a ton of focus for us. But another marketplace that has been great for us is actually Target's marketplace, target Plus and that's been a key, key piece of our success, especially with working with brands who are looking for store placement at Target. For example, we've had a few items that we've listed on Target's marketplace that have done really well, that have gotten the attention of a buyer and actually got store placement, which is really exciting. And, at the end of the day, getting an item placed on shelves most of the time can drive more volume than a mid-tier listing on Amazon. So we tend to try to use that strategy.   Bradley Sutton: How do you get on target these days? Wasn't it invite only back in the day or now that Target is adding that 360 or some kind of like yeah.   Grace Kopplin: I think it might still be invite only, but I know they've been actively adding a lot of sellers. I know that their backend is still quite archaic compared to what Amazon is. It's probably what Walmart was like four years ago. But I think it is still invite only, but definitely something to reach out to your connections and see if you can get a connect with a Walmart e-comm buyer.   Leo Sgovio: So there are a few reasons why you want to be indexed on Google, and for the most, let's start from the most advanced ones, right? Advanced sellers they normally try to send traffic to Amazon, especially during the launch period, using external traffic, right? So Google, we know, is a good referral that tends to help your rankings, and so Amazon tends to reward you if they see traffic coming from Google. So if you're not indexed, you lose a chance to show Amazon that you are getting traffic from Google. Now, I have a theory that paid traffic has a little bit more weight than organic, but the reason why you want to be indexed and the reason why you might want to be indexed for certain keywords is so that when you drive traffic through the URL to Amazon, you can actually give attribution to that keyword. That's number one, right? So you can actually use these URLs as your two-step.   Leo Sgovio: Number two if you do a good job with your indexation and your listing is optimized, you actually also appear in the images, right? And so if people are looking for specific products, sometimes I search on Google using images because I'm looking for specific products that might be hard to find on Amazon. But if I look through the Google images and I find the product, then I go to Amazon and so if you're not indexed, you're also not going to be able to be found there, and Google images actually gets a ton of traffic. So here are some of the reasons why, two of the reasons why. I can think of many more, but the most important are these ones. Google is still one of the largest search engine, and so missing out on that opportunity search engine and so missing out on that opportunity, I'm afraid it causes a lot of missed visibility for an Amazon seller at a listing level.   Carrie Miller: I think one of the things that sets us apart is that when I've created our listings, or whenever I create our photos, I think about what are the main benefits of the product, the main selling points of it, and I realized this isn't something that everyone can easily do, and so the way I kind of have been teaching it is that you can take your competitor's listing, download their reviews, download their best reviews, their five-star reviews, and say ask ChatGPT, like, what do people like most about this product? What are the benefits of this product according to reviews? What do people like? Basically, ask a bunch of questions to ChatGPT and you'll get a bunch of kind of selling points and you'll kind of see a trend of like the top selling points or top benefits of your product. And that's what you want to focus on is like what's in it for the customer? You've got to kind of appeal to their emotions. How is it going to make their life better, easier, are easier, are they going to be more beautiful? Are they going to you know what? What is it, what's in it for them?   Carrie Miller: And I think that that is going to be the key that sets you apart, and I know it's. It sounds pretty basic, but I've actually been doing some looking at different listings. People have been asking me hey, can you take a look at my listing? And when I look at the listing, I'm like, well, these aren't, these are not actually selling points or benefits. Like, these are features of the product. Right, you can always put the features in right later on, but how are you appealing to the person when you were? If you're telling somebody about your product, are you being like oh hey, the dimensions are 14 by 14. Like that's, that's like an afterthought, right? You, you want to. However, you would even just sell to a person, like talking face to face. That's how you're going to do that. Your first image shouldn't be a dimension photo. It should be a selling point, your main, like best selling point, main benefit in that first image. So I think that's a huge thing that a lot of people are kind of missing.   Bradley Sutton: What would you say is the most actionable things from search career performance? That kind of closes out like, hey, this is actually something that is not just, oh, it's good to know, but hey, I'm actually going to take action, uh, on this.    Mansour Norouzi: Taking action. I would say, even when I look at my own brand one is that for the main keywords, what I actually I do this on a weekly basis I have a list of the main keywords which is for my, for one of my aces are like 10 uh, 10 uh keywords and actually I go into the detail of week over week what is happening to my click share for those keywords, because they are very important for me and I want to be on the top and like top five for these turns. I want to be aware of what is going on with my competitors and what's my need. So if I see I have a track of my click share for the keywords, if I see it is going down, right away I'll figure out what's going on and maybe push with my advertising, for that for me would be our main keywords and what's going happening for my click share rate, conversion rate and click share just on my top keywords. Honestly, I will go, I think, by myself going with all for all the keywords, just like top five to 10 keywords, what they are, and I'll keep it very close overview and monitor them to see exactly what's going on, because you see that search volume going up or down, but I want my click share and my conversion share that I have I'm generating. Either they are consistent or going up. So if I see this trend is down, right away I start doing maybe I run coupon code or I push with my advertising to make sure I'm getting them back into track.   Bradley Sutton: What is your favorite? Helium 10 tool Ksenia or function of a tool.   Kseniia Reidel: Probably the audience. That's the one that I use all the time. Is it called audience?   Bradley Sutton: Yeah, the split where you ask the questions to the people and say, how are you using that Like for your images, or just for product ideas, or what are you using that?   Kseniia Reidel: Honestly for everything. For both for the product ideas, for your images, or just for product ideas or what are you using that? Honestly for everything. But both for the product ideas, for the images, because I just think it's so easy. You know, when you're thinking about like the product we find, then I usually do um, like the drawing and uh, 3d, you know the 3d image of the product that doesn't exist yet. Then usually all my products are like, really designed differently, that's what's on the market right now, and I just upload the image there and I see what people say and ask them would you buy this product? And if you wouldn't buy this product, why, why not? Or what would you change in this product? And sometimes I see the things that I didn't even you know, I didn't even think about that.   Bradley Sutton: So you're launching just the 3d rendering and just asking a question on that image, or you're launching it like, or you're launching it, you're putting it in a poll next to like existing products and asking them, or which one are you doing?   Kseniia Reidel: I'm doing both. Actually, the first, I just do the rendering and ask them would you buy this product? And if you would not buy this product, what would you change Like? How would you make it better for you? And then sometimes I also compare it to the other products that are on the market and ask them which one would they buy?   Bradley Sutton: Interesting.   Kseniia Reidel: And a lot of times I do the changes on the product based on what the people say.   Bradley Sutton: What was the results of those search, find, buy in order to send those relevancy signals? Again, not for rank, but to send those relevancy signals to Amazon. Take a look at this when I ran in Cerebro on June 19th, just three days after they did that relevancy single, you know, push those three coworkers here at Helium 10,. Take a look now at the Amazon recommended rank. Remember how it was only showing two keywords for Amazon recommended rank. Now it was showing multiple ones and it put that keyword that I sent the relevancy signal for egg holder countertop. It had Amazon recommended rank number three, which basically means that that was the third most important keyword according to Amazon for this product. Now do you remember what I was getting for impressions in PPC? Like 200 total impressions over three days. What did sending those relevancy signals to Amazon do for my PPC impressions? Take a look at this. To amazon, do for my PPC impressions. Take a look at this.   The next three day period from June 19th when my relevancy got fixed to June 21st instead of 200 impressions, 5 000 impressions, 4 000 of that. How? What keyword was it for? Egg holder countertop, that one that I sent those relevancy signals to Amazon for? This works, guys.   Ryan King:                          So Walmart has the equivalent would be brand portal, and I would absolutely recommend, if you're the seller, if you're the brand, to register through brand portal, and the main reasons are there are certain advertising opportunities that are only available to brand registered brands, so sponsored brand videos, sponsored brand ads that go across as banner displays. Another major one would be brand shops, brand shelves we can talk about later as well and then IP protection, and so the advantage of being registered in Brand Portal is that you can file IP infringement claims, and in this case, the most successful one to do is to file claims against those alternate listings for using your copyrighted imagery, and so we see success of getting those pulled down within 48 hours, typically when that happens. Now you can still file that IP claim even if you're not registered through Brand Portal. There's a link to file that claim, but you can't track its progress, you can't see the history, all those kinds of things. So it just gives you greater credibility in those and greater ability to look back at the progress. And the last one I'd say is if you're a registered brand, it's going to give you the highest content ranking for your listing. So even if there are other sellers that have tried to change that listing content. You're going to outrank them as the registered brand and chances are you're not going to have to deal with things changing on your listing in that regard.   Kevin Dolan: Cosmo is a specific tool and I think that the function that it performs is valuable to enhancing Amazon's understanding of a listing. So I certainly would not be surprised to see Amazon implementing this in a production capacity on a large swath of searches. That would not be surprising to me, but it's not as massive as the shift that we've seen into semantic-focused search. Cosmo in particular discusses essentially a mechanism for enhancing Amazon's understanding of a product by taking into consideration things that aren't expressed in the query and things that aren't expressed in the listing. The example that they use in the paper, the canonical example, is if you're looking for shoes for pregnant women, a listing might not literally say shoes for pregnant women. It might produce a specific type of open toed shoe that has good support, good comfort. That might not literally be listed as a keyword in the listing, but it might be something that the system can infer based on its knowledge of the universe, about what it's like to be a pregnant woman and the types of products that they might benefit from.   Norm Farrar: Out of everybody that we've looked at, it was up to 80. But 70% of Amazon sellers do not have the proper HTS code. They let their Chinese seller set an HS code and it's wrong. So when they get in here and guess what, nobody, nobody is calculating that as a part of your cost of goods. So they're going out, they're sourcing in China, they're not calculating, and this could be as high as 400%. Now, I've never seen it that, but it can be. So you know you're 25, 40% of your cost of goods. Is that not something that should be calculated? And like for me, I was doing natural soaps and I was paying 17%. So we were taking a look at it and Afolabi says can you consider this Castile soap? And I said yeah, it's olive based. And he goes well, how about I give you some good news. Pay zero. I just stuffed 17% back in my pocket. So out of the 70% of people that are missing the boat, they don't have the proper tariff code and the average person that gets the proper tariff code on an order the average that we've been able to calculate has been $7,800.

So Google just debuted a special quantum chip and everyone's FREAKING out. There's lots of implications at play here, not just for computing, but for reality as we know it. Pretty cool. Pretty scary. This week's bonus is a real doozy. https://benandemilshow.com BOSTON! we are coming to you! https://thewilbur.com/armory/artist/ben-and-emil/ code: benandemilboston LINK TO OUR DISCORD: https://discord.gg/CjujBt8g CHECK OUT OUR BONUS EPISODES: https://benandemilshow.com Subscribe to Emil's Substack: https://substack.com/@emilderosa Leave a comment to be featured as the comment of the week next week! And also, like this video, please! Thank you! __ AURA FRAMES: For a limited time, visit https://auraframes.com and get $35 off Aura's best-selling Carver Mat frames by using promo code BAES at checkout! It's their best Black Friday Cyber Monday deal of the year so don't miss it! SHOPIFY: Upgrade your business and get the same checkout we use! Sign up for your one dollar per month trial period at https://shopify.com/baes to upgrade your selling today. MOOMOO: Important: The creator is a paid influencer and not affiliated with Moomoo Financial Inc. ("MFI") or its affiliates. Content outside of the moomoo ad has not been reviewed by MFI and reflects the influencer's own views. MFI does not endorse any strategies mentioned and is not responsible for the influencer's services. Click this link https://j.moomoo.com/BAES to get up to 15 free stocks from moomoo U.S when you make a qualified deposit + earn 8.1% on uninvested cash for a limited time for new users!! Terms & Conditions Apply __ Latest MEATBALL SPECIAL HERE: https://youtu.be/Euyfzwmq8WY Last week's episode HERE: https://youtu.be/rXA-Tg-VS-M We bought suits HERE: https://youtu.be/_cM1XqA9n2U This episode was shot and edited by Connor Rousseau / @ conrad_roussrad Follow us on instagram! @ benandemilshow @ bencahn @ emilderosa Learn more about your ad choices. Visit megaphone.fm/adchoices

OpenAI DevDay is almost here! Per tradition, we are hosting a DevDay pregame event for everyone coming to town! Join us with demos and gossip!Also sign up for related events across San Francisco: the AI DevTools Night, the xAI open house, the Replicate art show, the DevDay Watch Party (for non-attendees), Hack Night with OpenAI at Cloudflare. For everyone else, join the Latent Space Discord for our online watch party and find fellow AI Engineers in your city.OpenAI's recent o1 release (and Reflection 70b debacle) has reignited broad interest in agentic general reasoning and tree search methods.While we have covered some of the self-taught reasoning literature on the Latent Space Paper Club, it is notable that the Eric Zelikman ended up at xAI, whereas OpenAI's hiring of Noam Brown and now Shunyu suggests more interest in tool-using chain of thought/tree of thought/generator-verifier architectures for Level 3 Agents.We were more than delighted to learn that Shunyu is a fellow Latent Space enjoyer, and invited him back (after his first appearance on our NeurIPS 2023 pod) for a look through his academic career with Harrison Chase (one year after his first LS show).ReAct: Synergizing Reasoning and Acting in Language Modelspaper linkFollowing seminal Chain of Thought papers from Wei et al and Kojima et al, and reflecting on lessons from building the WebShop human ecommerce trajectory benchmark, Shunyu's first big hit, the ReAct paper showed that using LLMs to “generate both reasoning traces and task-specific actions in an interleaved manner” achieved remarkably greater performance (less hallucination/error propagation, higher ALFWorld/WebShop benchmark success) than CoT alone. In even better news, ReAct scales fabulously with finetuning:As a member of the elite Princeton NLP group, Shunyu was also a coauthor of the Reflexion paper, which we discuss in this pod.Tree of Thoughtspaper link hereShunyu's next major improvement on the CoT literature was Tree of Thoughts:Language models are increasingly being deployed for general problem solving across a wide range of tasks, but are still confined to token-level, left-to-right decision-making processes during inference. This means they can fall short in tasks that require exploration, strategic lookahead, or where initial decisions play a pivotal role…ToT allows LMs to perform deliberate decision making by considering multiple different reasoning paths and self-evaluating choices to decide the next course of action, as well as looking ahead or backtracking when necessary to make global choices.The beauty of ToT is it doesnt require pretraining with exotic methods like backspace tokens or other MCTS architectures. You can listen to Shunyu explain ToT in his own words on our NeurIPS pod, but also the ineffable Yannic Kilcher:Other WorkWe don't have the space to summarize the rest of Shunyu's work, you can listen to our pod with him now, and recommend the CoALA paper and his initial hit webinar with Harrison, today's guest cohost:as well as Shunyu's PhD Defense Lecture:as well as Shunyu's latest lecture covering a Brief History of LLM Agents:As usual, we are live on YouTube! Show Notes* Harrison Chase* LangChain, LangSmith, LangGraph* Shunyu Yao* Alec Radford* ReAct Paper* Hotpot QA* Tau Bench* WebShop* SWE-Agent* SWE-Bench* Trees of Thought* CoALA Paper* Related Episodes* Our Thomas Scialom (Meta) episode* Shunyu on our NeurIPS 2023 Best Papers episode* Harrison on our LangChain episode* Mentions* Sierra* Voyager* Jason Wei* Tavily* SERP API* ExaTimestamps* [00:00:00] Opening Song by Suno* [00:03:00] Introductions* [00:06:16] The ReAct paper* [00:12:09] Early applications of ReAct in LangChain* [00:17:15] Discussion of the Reflection paper* [00:22:35] Tree of Thoughts paper and search algorithms in language models* [00:27:21] SWE-Agent and SWE-Bench for coding benchmarks* [00:39:21] CoALA: Cognitive Architectures for Language Agents* [00:45:24] Agent-Computer Interfaces (ACI) and tool design for agents* [00:49:24] Designing frameworks for agents vs humans* [00:53:52] UX design for AI applications and agents* [00:59:53] Data and model improvements for agent capabilities* [01:19:10] TauBench* [01:23:09] Promising areas for AITranscriptAlessio [00:00:01]: Hey, everyone, welcome to the Latent Space podcast. This is Alessio, partner and CTO of Residence at Decibel Partners, and I'm joined by my co-host Swyx, founder of Small AI.Swyx [00:00:12]: Hey, and today we have a super special episode. I actually always wanted to take like a selfie and go like, you know, POV, you're about to revolutionize the world of agents because we have two of the most awesome hiring agents in the house. So first, we're going to welcome back Harrison Chase. Welcome. Excited to be here. What's new with you recently in sort of like the 10, 20 second recap?Harrison [00:00:34]: Linkchain, Linksmith, Lingraph, pushing on all of them. Lots of cool stuff related to a lot of the stuff that we're going to talk about today, probably.Swyx [00:00:42]: Yeah.Alessio [00:00:43]: We'll mention it in there. And the Celtics won the title.Swyx [00:00:45]: And the Celtics won the title. You got that going on for you. I don't know. Is that like floorball? Handball? Baseball? Basketball.Alessio [00:00:52]: Basketball, basketball.Harrison [00:00:53]: Patriots aren't looking good though, so that's...Swyx [00:00:56]: And then Xun Yu, you've also been on the pod, but only in like a sort of oral paper presentation capacity. But welcome officially to the LinkedSpace pod.Shunyu [00:01:03]: Yeah, I've been a huge fan. So thanks for the invitation. Thanks.Swyx [00:01:07]: Well, it's an honor to have you on. You're one of like, you're maybe the first PhD thesis defense I've ever watched in like this AI world, because most people just publish single papers, but every paper of yours is a banger. So congrats.Shunyu [00:01:22]: Thanks.Swyx [00:01:24]: Yeah, maybe we'll just kick it off with, you know, what was your journey into using language models for agents? I like that your thesis advisor, I didn't catch his name, but he was like, you know... Karthik. Yeah. It's like, this guy just wanted to use language models and it was such a controversial pick at the time. Right.Shunyu [00:01:39]: The full story is that in undergrad, I did some computer vision research and that's how I got into AI. But at the time, I feel like, you know, you're just composing all the GAN or 3D perception or whatever together and it's not exciting anymore. And one day I just see this transformer paper and that's really cool. But I really got into language model only when I entered my PhD and met my advisor Karthik. So he was actually the second author of GPT-1 when he was like a visiting scientist at OpenAI. With Alec Redford?Swyx [00:02:10]: Yes.Shunyu [00:02:11]: Wow. That's what he told me. It's like back in OpenAI, they did this GPT-1 together and Ilya just said, Karthik, you should stay because we just solved the language. But apparently Karthik is not fully convinced. So he went to Princeton, started his professorship and I'm really grateful. So he accepted me as a student, even though I have no prior knowledge in NLP. And you know, we just met for the first time and he's like, you know, what do you want to do? And I'm like, you know, you have done those test game scenes. That's really cool. I wonder if we can just redo them with language models. And that's how the whole journey began. Awesome.Alessio [00:02:46]: So GPT-2 was out at the time? Yes, that was 2019.Shunyu [00:02:48]: Yeah.Alessio [00:02:49]: Way too dangerous to release. And then I guess the first work of yours that I came across was React, which was a big part of your defense. But also Harrison, when you came on The Pockets last year, you said that was one of the first papers that you saw when you were getting inspired for BlankChain. So maybe give a recap of why you thought it was cool, because you were already working in AI and machine learning. And then, yeah, you can kind of like intro the paper formally. What was that interesting to you specifically?Harrison [00:03:16]: Yeah, I mean, I think the interesting part was using these language models to interact with the outside world in some form. And I think in the paper, you mostly deal with Wikipedia. And I think there's some other data sets as well. But the outside world is the outside world. And so interacting with things that weren't present in the LLM and APIs and calling into them and thinking about the React reasoning and acting and kind of like combining those together and getting better results. I'd been playing around with LLMs, been talking with people who were playing around with LLMs. People were trying to get LLMs to call into APIs, do things, and it was always, how can they do it more reliably and better? And so this paper was basically a step in that direction. And I think really interesting and also really general as well. Like I think that's part of the appeal is just how general and simple in a good way, I think the idea was. So that it was really appealing for all those reasons.Shunyu [00:04:07]: Simple is always good. Yeah.Alessio [00:04:09]: Do you have a favorite part? Because I have one favorite part from your PhD defense, which I didn't understand when I read the paper, but you said something along the lines, React doesn't change the outside or the environment, but it does change the insight through the context, putting more things in the context. You're not actually changing any of the tools around you to work for you, but you're changing how the model thinks. And I think that was like a very profound thing when I, not that I've been using these tools for like 18 months. I'm like, I understand what you meant, but like to say that at the time you did the PhD defense was not trivial. Yeah.Shunyu [00:04:41]: Another way to put it is like thinking can be an extra tool that's useful.Alessio [00:04:47]: Makes sense. Checks out.Swyx [00:04:49]: Who would have thought? I think it's also more controversial within his world because everyone was trying to use RL for agents. And this is like the first kind of zero gradient type approach. Yeah.Shunyu [00:05:01]: I think the bigger kind of historical context is that we have this two big branches of AI. So if you think about RL, right, that's pretty much the equivalent of agent at a time. And it's like agent is equivalent to reinforcement learning and reinforcement learning is equivalent to whatever game environment they're using, right? Atari game or go or whatever. So you have like a pretty much, you know, you have a biased kind of like set of methodologies in terms of reinforcement learning and represents agents. On the other hand, I think NLP is like a historical kind of subject. It's not really into agents, right? It's more about reasoning. It's more about solving those concrete tasks. And if you look at SEL, right, like each task has its own track, right? Summarization has a track, question answering has a track. So I think really it's about rethinking agents in terms of what could be the new environments that we came to have is not just Atari games or whatever video games, but also those text games or language games. And also thinking about, could there be like a more general kind of methodology beyond just designing specific pipelines for each NLP task? That's like the bigger kind of context, I would say.Alessio [00:06:14]: Is there an inspiration spark moment that you remember or how did you come to this? We had Trida on the podcast and he mentioned he was really inspired working with like systems people to think about Flash Attention. What was your inspiration journey?Shunyu [00:06:27]: So actually before React, I spent the first two years of my PhD focusing on text-based games, or in other words, text adventure games. It's a very kind of small kind of research area and quite ad hoc, I would say. And there are like, I don't know, like 10 people working on that at the time. And have you guys heard of Zork 1, for example? So basically the idea is you have this game and you have text observations, like you see a monster, you see a dragon.Swyx [00:06:57]: You're eaten by a grue.Shunyu [00:06:58]: Yeah, you're eaten by a grue. And you have actions like kill the grue with a sword or whatever. And that's like a very typical setup of a text game. So I think one day after I've seen all the GPT-3 stuff, I just think about, you know, how can I solve the game? Like why those AI, you know, machine learning methods are pretty stupid, but we are pretty good at solving the game relatively, right? So for the context, the predominant method to solve this text game is obviously reinforcement learning. And the idea is you just try out an arrow in those games for like millions of steps and you kind of just overfit to the game. But there's no language understanding at all. And I'm like, why can't I solve the game better? And it's kind of like, because we think about the game, right? Like when we see this very complex text observation, like you see a grue and you might see a sword, you know, in the right of the room and you have to go through the wooden door to go to that room. You will think, you know, oh, I have to kill the monster and to kill that monster, I have to get the sword, I have to get the sword, I have to go, right? And this kind of thinking actually helps us kind of throw shots off the game. And it's like, why don't we also enable the text agents to think? And that's kind of the prototype of React. And I think that's actually very interesting because the prototype, I think, was around November of 2021. So that's even before like chain of thought or whatever came up. So we did a bunch of experiments in the text game, but it was not really working that well. Like those text games are just too hard. I think today it's still very hard. Like if you use GPD 4 to solve it, it's still very hard. So the change came when I started the internship in Google. And apparently Google care less about text game, they care more about what's more practical. So pretty much I just reapplied the idea, but to more practical kind of environments like Wikipedia or simpler text games like Alphard, and it just worked. It's kind of like you first have the idea and then you try to find the domains and the problems to demonstrate the idea, which is, I would say, different from most of the AI research, but it kind of worked out for me in that case.Swyx [00:09:09]: For Harrison, when you were implementing React, what were people applying React to in the early days?Harrison [00:09:14]: I think the first demo we did probably had like a calculator tool and a search tool. So like general things, we tried to make it pretty easy to write your own tools and plug in your own things. And so this is one of the things that we've seen in LangChain is people who build their own applications generally write their own tools. Like there are a few common ones. I'd say like the three common ones might be like a browser, a search tool, and a code interpreter. But then other than that-Swyx [00:09:37]: The LMS. Yep.Harrison [00:09:39]: Yeah, exactly. It matches up very nice with that. And we actually just redid like our integrations docs page, and if you go to the tool section, they like highlight those three, and then there's a bunch of like other ones. And there's such a long tail of other ones. But in practice, like when people go to production, they generally have their own tools or maybe one of those three, maybe some other ones, but like very, very few other ones. So yeah, I think the first demos was a search and a calculator one. And there's- What's the data set?Shunyu [00:10:04]: Hotpot QA.Harrison [00:10:05]: Yeah. Oh, so there's that one. And then there's like the celebrity one by the same author, I think.Swyx [00:10:09]: Olivier Wilde's boyfriend squared. Yeah. 0.23. Yeah. Right, right, right.Harrison [00:10:16]: I'm forgetting the name of the author, but there's-Swyx [00:10:17]: I was like, we're going to over-optimize for Olivier Wilde's boyfriend, and it's going to change next year or something.Harrison [00:10:21]: There's a few data sets kind of like in that vein that require multi-step kind of like reasoning and thinking. So one of the questions I actually had for you in this vein, like the React paper, there's a few things in there, or at least when I think of that, there's a few things that I think of. There's kind of like the specific prompting strategy. Then there's like this general idea of kind of like thinking and then taking an action. And then there's just even more general idea of just like taking actions in a loop. Today, like obviously language models have changed a lot. We have tool calling. The specific prompting strategy probably isn't used super heavily anymore. Would you say that like the concept of React is still used though? Or like do you think that tool calling and running tool calling in a loop, is that ReactSwyx [00:11:02]: in your mind?Shunyu [00:11:03]: I would say like it's like more implicitly used than explicitly used. To be fair, I think the contribution of React is actually twofold. So first is this idea of, you know, we should be able to use calls in a very general way. Like there should be a single kind of general method to handle interaction with various environments. I think React is the first paper to demonstrate the idea. But then I think later there are two form or whatever, and this becomes like a trivial idea. But I think at the time, that's like a pretty non-trivial thing. And I think the second contribution is this idea of what people call like inner monologue or thinking or reasoning or whatever, to be paired with tool use. I think that's still non-trivial because if you look at the default function calling or whatever, like there's no inner monologue. And in practice, that actually is important, especially if the tool that you use is pretty different from the training distribution of the language model. I think those are the two main things that are kind of inherited.Harrison [00:12:10]: On that note, I think OpenAI even recommended when you're doing tool calling, it's sometimes helpful to put a thought field in the tool, along with all the actual acquired arguments,Swyx [00:12:19]: and then have that one first.Harrison [00:12:20]: So it fills out that first, and they've shown that that's yielded better results. The reason I ask is just like this same concept is still alive, and I don't know whether to call it a React agent or not. I don't know what to call it. I think of it as React, like it's the same ideas that were in the paper, but it's obviously a very different implementation at this point in time. And so I just don't know what to call it.Shunyu [00:12:40]: I feel like people will sometimes think more in terms of different tools, right? Because if you think about a web agent versus, you know, like a function calling agent, calling a Python API, you would think of them as very different. But in some sense, the methodology is the same. It depends on how you view them, right? I think people will tend to think more in terms of the environment and the tools rather than the methodology. Or, in other words, I think the methodology is kind of trivial and simple, so people will try to focus more on the different tools. But I think it's good to have a single underlying principle of those things.Alessio [00:13:17]: How do you see the surface of React getting molded into the model? So a function calling is a good example of like, now the model does it. What about the thinking? Now most models that you use kind of do chain of thought on their own, they kind of produce steps. Do you think that more and more of this logic will be in the model? Or do you think the context window will still be the main driver of reasoning and thinking?Shunyu [00:13:39]: I think it's already default, right? You do some chain of thought and you do some tool call, the cost of adding the chain of thought is kind of relatively low compared to other things. So it's not hurting to do that. And I think it's already kind of common practice, I would say.Swyx [00:13:56]: This is a good place to bring in either Tree of Thought or Reflection, your pick.Shunyu [00:14:01]: Maybe Reflection, to respect the time order, I would say.Swyx [00:14:05]: Any backstory as well, like the people involved with NOAA and the Princeton group. We talked about this offline, but people don't understand how these research pieces come together and this ideation.Shunyu [00:14:15]: I think Reflection is mostly NOAA's work, I'm more like advising kind of role. The story is, I don't remember the time, but one day we just see this pre-print that's like Reflection and Autonomous Agent with memory or whatever. And it's kind of like an extension to React, which uses this self-reflection. I'm like, oh, somehow you've become very popular. And NOAA reached out to me, it's like, do you want to collaborate on this and make this from an archive pre-print to something more solid, like a conference submission? I'm like, sure. We started collaborating and we remain good friends today. And I think another interesting backstory is NOAA was contacted by OpenAI at the time. It's like, this is pretty cool, do you want to just work at OpenAI? And I think Sierra also reached out at the same time. It's like, this is pretty cool, do you want to work at Sierra? And I think NOAA chose Sierra, but it's pretty cool because he was still like a second year undergrad and he's a very smart kid.Swyx [00:15:16]: Based on one paper. Oh my god.Shunyu [00:15:19]: He's done some other research based on programming language or chemistry or whatever, but I think that's the paper that got the attention of OpenAI and Sierra.Swyx [00:15:28]: For those who haven't gone too deep on it, the way that you present the inside of React, can you do that also for reflection? Yeah.Shunyu [00:15:35]: I think one way to think of reflection is that the traditional idea of reinforcement learning is you have a scalar reward and then you somehow back-propagate the signal of the scalar reward to the rest of your neural network through whatever algorithm, like policy grading or A2C or whatever. And if you think about the real life, most of the reward signal is not scalar. It's like your boss told you, you should have done a better job in this, but you could jump on that or whatever. It's not like a scalar reward, like 29 or something. I think in general, humans deal more with long scalar reward, or you can say language feedback. And the way that they deal with language feedback also has this back-propagation process, right? Because you start from this, you did a good job on job B, and then you reflect what could have been done differently to change to make it better. And you kind of change your prompt, right? Basically, you change your prompt on how to do job A and how to do job B, and then you do the whole thing again. So it's really like a pipeline of language where in self-graded descent, you have something like text reasoning to replace those gradient descent algorithms. I think that's one way to think of reflection.Harrison [00:16:47]: One question I have about reflection is how general do you think the algorithm there is? And so for context, I think at LangChain and at other places as well, we found it pretty easy to implement React in a standard way. You plug in any tools and it kind of works off the shelf, can get it up and running. I don't think we have an off-the-shelf kind of implementation of reflection and kind of the general sense. I think the concepts, absolutely, we see used in different kind of specific cognitive architectures, but I don't think we have one that comes off the shelf. I don't think any of the other frameworks have one that comes off the shelf. And I'm curious whether that's because it's not general enough or it's complex as well, because it also requires running it more times.Swyx [00:17:28]: Maybe that's not feasible.Harrison [00:17:30]: I'm curious how you think about the generality, complexity. Should we have one that comes off the shelf?Shunyu [00:17:36]: I think the algorithm is general in the sense that it's just as general as other algorithms, if you think about policy grading or whatever, but it's not applicable to all tasks, just like other algorithms. So you can argue PPO is also general, but it works better for those set of tasks, but not on those set of tasks. I think it's the same situation for reflection. And I think a key bottleneck is the evaluator, right? Basically, you need to have a good sense of the signal. So for example, if you are trying to do a very hard reasoning task, say mathematics, for example, and you don't have any tools, you're operating in this chain of thought setup, then reflection will be pretty hard because in order to reflect upon your thoughts, you have to have a very good evaluator to judge whether your thought is good or not. But that might be as hard as solving the problem itself or even harder. The principle of self-reflection is probably more applicable if you have a good evaluator, for example, in the case of coding. If you have those arrows, then you can just reflect on that and how to solve the bug andSwyx [00:18:37]: stuff.Shunyu [00:18:38]: So I think another criteria is that it depends on the application, right? If you have this latency or whatever need for an actual application with an end-user, the end-user wouldn't let you do two hours of tree-of-thought or reflection, right? You need something as soon as possible. So in that case, maybe this is better to be used as a training time technique, right? You do those reflection or tree-of-thought or whatever, you get a lot of data, and then you try to use the data to train your model better. And then in test time, you still use something as simple as React, but that's already improved.Alessio [00:19:11]: And if you think of the Voyager paper as a way to store skills and then reuse them, how would you compare this reflective memory and at what point it's just ragging on the memory versus you want to start to fine-tune some of them or what's the next step once you get a very long reflective corpus? Yeah.Shunyu [00:19:30]: So I think there are two questions here. The first question is, what type of information or memory are you considering, right? Is it like semantic memory that stores knowledge about the word, or is it the episodic memory that stores trajectories or behaviors, or is it more of a procedural memory like in Voyager's case, like skills or code snippets that you can use to do actions, right?Swyx [00:19:54]: That's one dimension.Shunyu [00:19:55]: And the second dimension is obviously how you use the memory, either retrieving from it, using it in the context, or fine-tuning it. I think the Cognitive Architecture for Language Agents paper has a good categorization of all the different combinations. And of course, which way you use depends on the concrete application and the concrete need and the concrete task. But I think in general, it's good to think of those systematic dimensions and all the possible options there.Swyx [00:20:25]: Harrison also has in LangMEM, I think you did a presentation in my meetup, and I think you've done it at a couple other venues as well. User state, semantic memory, and append-only state, I think kind of maps to what you just said.Shunyu [00:20:38]: What is LangMEM? Can I give it like a quick...Harrison [00:20:40]: One of the modules of LangChain for a long time has been something around memory. And I think we're still obviously figuring out what that means, as is everyone kind of in the space. But one of the experiments that we did, and one of the proof of concepts that we did was, technically what it was is you would basically create threads, you'd push messages to those threads in the background, we process the data in a few ways. One, we put it into some semantic store, that's the semantic memory. And then two, we do some extraction and reasoning over the memories to extract. And we let the user define this, but extract key facts or anything that's of interest to the user. Those aren't exactly trajectories, they're maybe more closer to the procedural memory. Is that how you'd think about it or classify it?Shunyu [00:21:22]: Is it like about knowledge about the word, or is it more like how to do something?Swyx [00:21:27]: It's reflections, basically.Harrison [00:21:28]: So in generative worlds.Shunyu [00:21:30]: Generative agents.Swyx [00:21:31]: The Smallville. Yeah, the Smallville one.Harrison [00:21:33]: So the way that they had their memory there was they had the sequence of events, and that's kind of like the raw events that happened. But then every N events, they'd run some synthesis over those events for the LLM to insert its own memory, basically. It's that type of memory.Swyx [00:21:49]: I don't know how that would be classified.Shunyu [00:21:50]: I think of that as more of the semantic memory, but to be fair, I think it's just one way to think of that. But whether it's semantic memory or procedural memory or whatever memory, that's like an abstraction layer. But in terms of implementation, you can choose whatever implementation for whatever memory. So they're totally kind of orthogonal. I think it's more of a good way to think of the things, because from the history of cognitive science and cognitive architecture and how people study even neuroscience, that's the way people think of how the human brain organizes memory. And I think it's more useful as a way to think of things. But it's not like for semantic memory, you have to do this kind of way to retrieve or fine-tune, and for procedural memory, you have to do that. I think those are totally orthogonal kind of dimensions.Harrison [00:22:34]: How much background do you have in cognitive sciences, and how much do you model some of your thoughts on?Shunyu [00:22:40]: That's a great question, actually. I think one of the undergrad influences for my follow-up research is I was doing an internship at MIT's Computational Cognitive Science Lab with Josh Tannenbaum, and he's a very famous cognitive scientist. And I think a lot of his ideas still influence me today, like thinking of things in computational terms and getting interested in language and a lot of stuff, or even developing psychology kind of stuff. So I think it still influences me today.Swyx [00:23:14]: As a developer that tried out LangMEM, the way I view it is just it's a materialized view of a stream of logs. And if anything, that's just useful for context compression. I don't have to use the full context to run it over everything. But also it's kind of debuggable. If it's wrong, I can show it to the user, the user can manually fix it, and I can carry on. That's a really good analogy. I like that. I'm going to steal that. Sure. Please, please. You know I'm bullish on memory databases. I guess, Tree of Thoughts? Yeah, Tree of Thoughts.Shunyu [00:23:39]: I feel like I'm relieving the defense in like a podcast format. Yeah, no.Alessio [00:23:45]: I mean, you had a banger. Well, this is the one where you're already successful and we just highlight the glory. It was really good. You mentioned that since thinking is kind of like taking an action, you can use action searching algorithms to think of thinking. So just like you will use Tree Search to find the next thing. And the idea behind Tree of Thought is that you generate all these possible outcomes and then find the best tree to get to the end. Maybe back to the latency question, you can't really do that if you have to respond in real time. So what are maybe some of the most helpful use cases for things like this? Where have you seen people adopt it where the high latency is actually worth the wait?Shunyu [00:24:21]: For things that you don't care about latency, obviously. For example, if you're trying to do math, if you're just trying to come up with a proof. But I feel like one type of task is more about searching for a solution. You can try a hundred times, but if you find one solution, that's good. For example, if you're finding a math proof or if you're finding a good code to solve a problem or whatever, I think another type of task is more like reacting. For example, if you're doing customer service, you're like a web agent booking a ticket for an end user. Those are more reactive kind of tasks, or more real-time tasks. You have to do things fast. They might be easy, but you have to do it reliably. And you care more about can you solve 99% of the time out of a hundred. But for the type of search type of tasks, then you care more about can I find one solution out of a hundred. So it's kind of symmetric and different.Alessio [00:25:11]: Do you have any data or intuition from your user base? What's the split of these type of use cases? How many people are doing more reactive things and how many people are experimenting with deep, long search?Harrison [00:25:23]: I would say React's probably the most popular. I think there's aspects of reflection that get used. Tree of thought, probably the least so. There's a great tweet from Jason Wei, I think you're now a colleague, and he was talking about prompting strategies and how he thinks about them. And I think the four things that he had was, one, how easy is it to implement? How much compute does it take? How many tasks does it solve? And how much does it improve on those tasks? And I'd add a fifth, which is how likely is it to be relevant when the next generation of models come out? And I think if you look at those axes and then you look at React, reflection, tree of thought, it tracks that the ones that score better are used more. React is pretty easy to implement. Tree of thought's pretty hard to implement. The amount of compute, yeah, a lot more for tree of thought. The tasks and how much it improves, I don't have amazing visibility there. But I think if we're comparing React versus tree of thought, React just dominates the first two axes so much that my question around that was going to be like, how do you think about these prompting strategies, cognitive architectures, whatever you want to call them? When you're thinking of them, what are the axes that you're judging them on in your head when you're thinking whether it's a good one or a less good one?Swyx [00:26:38]: Right.Shunyu [00:26:39]: Right. I think there is a difference between a prompting method versus research, in the sense that for research, you don't really even care about does it actually work on practical tasks or does it help? Whatever. I think it's more about the idea or the principle, right? What is the direction that you're unblocking and whatever. And I think for an actual prompting method to solve a concrete problem, I would say simplicity is very important because the simpler it is, the less decision you have to make about it. And it's easier to design. It's easier to propagate. And it's easier to do stuff. So always try to be as simple as possible. And I think latency obviously is important. If you can do things fast and you don't want to do things slow. And I think in terms of the actual prompting method to use for a particular problem, I think we should all be in the minimalist kind of camp, right? You should try the minimum thing and see if it works. And if it doesn't work and there's absolute reason to add something, then you add something, right? If there's absolute reason that you need some tool, then you should add the tool thing. If there's absolute reason to add reflection or whatever, you should add that. Otherwise, if a chain of thought can already solve something, then you don't even need to use any of that.Harrison [00:27:57]: Yeah. Or if it's just better prompting can solve it. Like, you know, you could add a reflection step or you could make your instructions a little bit clearer.Swyx [00:28:03]: And it's a lot easier to do that.Shunyu [00:28:04]: I think another interesting thing is like, I personally have never done those kind of like weird tricks. I think all the prompts that I write are kind of like just talking to a human, right? It's like, I don't know. I never say something like, your grandma is dying and you have to solve it. I mean, those are cool, but I feel like we should all try to solve things in a very intuitive way. Just like talking to your co-worker. That should work 99% of the time. That's my personal take.Swyx [00:28:29]: The problem with how language models, at least in the GPC 3 era, was that they over-optimized to some sets of tokens in sequence. So like reading the Kojima et al. paper that was listing step-by-step, like he tried a bunch of them and they had wildly different results. It should not be the case, but it is the case. And hopefully we're getting better there.Shunyu [00:28:51]: Yeah. I think it's also like a timing thing in the sense that if you think about this whole line of language model, right? Like at the time it was just like a text generator. We don't have any idea how it's going to be used, right? And obviously at the time you will find all kinds of weird issues because it's not trained to do any of that, right? But then I think we have this loop where once we realize chain of thought is important or agent is important or tool using is important, what we see is today's language models are heavily optimized towards those things. So I think in some sense they become more reliable and robust over those use cases. And you don't need to do as much prompt engineering tricks anymore to solve those things. I feel like in some sense, I feel like prompt engineering even is like a slightly negative word at the time because it refers to all those kind of weird tricks that you have to apply. But I think we don't have to do that anymore. Like given today's progress, you should just be able to talk to like a coworker. And if you're clear and concrete and being reasonable, then it should do reasonable things for you.Swyx [00:29:51]: Yeah. The way I put this is you should not be a prompt engineer because it is the goal of the big labs to put you out of a job.Shunyu [00:29:58]: You should just be a good communicator. Like if you're a good communicator to humans, you should be a good communicator to languageSwyx [00:30:02]: models.Harrison [00:30:03]: That's the key though, because oftentimes people aren't good communicators to these language models and that is a very important skill and that's still messing around with the prompt. And so it depends what you're talking about when you're saying prompt engineer.Shunyu [00:30:14]: But do you think it's like very correlated with like, are they like a good communicator to humans? You know, it's like.Harrison [00:30:20]: It may be, but I also think I would say on average, people are probably worse at communicating with language models than to humans right now, at least, because I think we're still figuring out how to do it. You kind of expect it to be magical and there's probably some correlation, but I'd say there's also just like, people are worse at it right now than talking to humans.Shunyu [00:30:36]: We should make it like a, you know, like an elementary school class or whatever, how toSwyx [00:30:41]: talk to language models. Yeah. I don't know. Very pro that. Yeah. Before we leave the topic of trees and searching, not specific about QSTAR, but there's a lot of questions about MCTS and this combination of tree search and language models. And I just had to get in a question there about how seriously should people take this?Shunyu [00:30:59]: Again, I think it depends on the tasks, right? So MCTS was magical for Go, but it's probably not as magical for robotics, right? So I think right now the problem is not even that we don't have good methodologies, it's more about we don't have good tasks. It's also very interesting, right? Because if you look at my citation, it's like, obviously the most cited are React, Refraction and Tree of Thought. Those are methodologies. But I think like equally important, if not more important line of my work is like benchmarks and environments, right? Like WebShop or SuiteVenture or whatever. And I think in general, what people do in academia that I think is not good is they choose a very simple task, like Alford, and then they apply overly complex methods to show they improve 2%. I think you should probably match the level of complexity of your task and your method. I feel like where tasks are kind of far behind the method in some sense, right? Because we have some good test-time approaches, like whatever, React or Refraction or Tree of Thought, or like there are many, many more complicated test-time methods afterwards. But on the benchmark side, we have made a lot of good progress this year, last year. But I think we still need more progress towards that, like better coding benchmark, better web agent benchmark, better agent benchmark, not even for web or code. I think in general, we need to catch up with tasks.Harrison [00:32:27]: What are the biggest reasons in your mind why it lags behind?Shunyu [00:32:31]: I think incentive is one big reason. Like if you see, you know, all the master paper are cited like a hundred times more than the task paper. And also making a good benchmark is actually quite hard. It's almost like a different set of skills in some sense, right? I feel like if you want to build a good benchmark, you need to be like a good kind of product manager kind of mindset, right? You need to think about why people should use your benchmark, why it's challenging, why it's useful. If you think about like a PhD going into like a school, right? The prior skill that expected to have is more about, you know, can they code this method and can they just run experiments and can solve that? I think building a benchmark is not the typical prior skill that we have, but I think things are getting better. I think more and more people are starting to build benchmarks and people are saying that it's like a way to get more impact in some sense, right? Because like if you have a really good benchmark, a lot of people are going to use it. But if you have a super complicated test time method, like it's very hard for people to use it.Harrison [00:33:35]: Are evaluation metrics also part of the reason? Like for some of these tasks that we might want to ask these agents or language models to do, is it hard to evaluate them? And so it's hard to get an automated benchmark. Obviously with SweetBench you can, and with coding, it's easier, but.Shunyu [00:33:50]: I think that's part of the skillset thing that I mentioned, because I feel like it's like a product manager because there are many dimensions and you need to strike a balance and it's really hard, right? If you want to make sense, very easy to autogradable, like automatically gradable, like either to grade or either to evaluate, then you might lose some of the realness or practicality. Or like it might be practical, but it might not be as scalable, right? For example, if you think about text game, human have pre-annotated all the rewards and all the language are real. So it's pretty good on autogradable dimension and the practical dimension. If you think about, you know, practical, like actual English being practical, but it's not scalable, right? It takes like a year for experts to build that game. So it's not really that scalable. And I think part of the reason that SweetBench is so popular now is it kind of hits the balance between these three dimensions, right? Easy to evaluate and being actually practical and being scalable. Like if I were to criticize upon some of my prior work, I think webshop, like it's my initial attempt to get into benchmark world and I'm trying to do a good job striking the balance. But obviously we make it all gradable and it's really scalable, but then I think the practicality is not as high as actually just using GitHub issues, right? Because you're just creating those like synthetic tasks.Harrison [00:35:13]: Are there other areas besides coding that jump to mind as being really good for being autogradable?Shunyu [00:35:20]: Maybe mathematics.Swyx [00:35:21]: Classic. Yeah. Do you have thoughts on alpha proof, the new DeepMind paper? I think it's pretty cool.Shunyu [00:35:29]: I think it's more of a, you know, it's more of like a confidence boost or like sometimes, you know, the work is not even about, you know, the technical details or the methodology that it chooses or the concrete results. I think it's more about a signal, right?Swyx [00:35:47]: Yeah. Existence proof. Yeah.Shunyu [00:35:50]: Yeah. It can be done. This direction is exciting. It kind of encourages people to work more towards that direction. I think it's more like a boost of confidence, I would say.Swyx [00:35:59]: Yeah. So we're going to focus more on agents now and, you know, all of us have a special interest in coding agents. I would consider Devin to be the sort of biggest launch of the year as far as AI startups go. And you guys in the Princeton group worked on Suiagents alongside of Suibench. Tell us the story about Suiagent. Sure.Shunyu [00:36:21]: I think it's kind of like a triology, it's actually a series of three works now. So actually the first work is called Intercode, but it's not as famous, I know. And the second work is called Suibench and the third work is called Suiagent. And I'm just really confused why nobody is working on coding. You know, it's like a year ago, but I mean, not everybody's working on coding, obviously, but a year ago, like literally nobody was working on coding. I was really confused. And the people that were working on coding are, you know, trying to solve human evil in like a sick-to-sick way. There's no agent, there's no chain of thought, there's no anything, they're just, you know, fine tuning the model and improve some points and whatever, like, I was really confused because obviously coding is the best application for agents because it's autogradable, it's super important, you can make everything like API or code action, right? So I was confused and I collaborated with some of the students in Princeton and we have this work called Intercode and the idea is, first, if you care about coding, then you should solve coding in an interactive way, meaning more like a Jupyter Notebook kind of way than just writing a program and seeing if it fails or succeeds and stop, right? You should solve it in an interactive way because that's exactly how humans solve it, right? You don't have to, you know, write a program like next token, next token, next token and stop and never do any edits and you cannot really use any terminal or whatever tool. It doesn't make sense, right? And that's the way people are solving coding at the time, basically like sampling a program from a language model without chain of thought, without tool call, without refactoring, without anything. So the first point is we should solve coding in a very interactive way and that's a very general principle that applies for various coding benchmarks. And also, I think you can make a lot of the agent task kind of like interactive coding. If you have Python and you can call any package, then you can literally also browse internet or do whatever you want, like control a robot or whatever. So that seems to be a very general paradigm. But obviously I think a bottleneck is at the time we're still doing, you know, very simple tasks like human eval or whatever coding benchmark people proposed. They were super hard in 2021, like 20%, but they're like 95% already in 2023. So obviously the next step is we need a better benchmark. And Carlos and John, which are the first authors of Swaybench, I think they come up with this great idea that we should just script GitHub and solve whatever human engineers are solving. And I think it's actually pretty easy to come up with the idea. And I think in the first week, they already made a lot of progress. They script the GitHub and they make all the same, but then there's a lot of painful info work and whatever, you know. I think the idea is super easy, but the engineering is super hard. And I feel like that's a very typical signal of a good work in the AI era now.Swyx [00:39:17]: I think also, I think the filtering was challenging, because if you look at open source PRs, a lot of them are just like, you know, fixing typos. I think it's challenging.Shunyu [00:39:27]: And to be honest, we didn't do a perfect job at the time. So if you look at the recent blog post with OpenAI, we improved the filtering so that it's more solvable.Swyx [00:39:36]: I think OpenAI was just like, look, this is a thing now. We have to fix this. These students just rushed it.Shunyu [00:39:45]: It's a good convergence of interests for me.Alessio [00:39:48]: Was that tied to you joining OpenAI? Or was that just unrelated?Shunyu [00:39:52]: It's a coincidence for me, but it's a good coincidence.Swyx [00:39:55]: There is a history of anytime a big lab adopts a benchmark, they fix it. Otherwise, it's a broken benchmark.Shunyu [00:40:03]: So naturally, once we propose swimmage, the next step is to solve it. But I think the typical way you solve something now is you collect some training samples, or you design some complicated agent method, and then you try to solve it. Either super complicated prompt, or you build a better model with more training data. But I think at the time, we realized that even before those things, there's a fundamental problem with the interface or the tool that you're supposed to use. Because that's like an ignored problem in some sense. What your tool is, or how that matters for your task. So what we found concretely is that if you just use the text terminal off the shelf as a tool for those agents, there's a lot of problems. For example, if you edit something, there's no feedback. So you don't know whether your edit is good or not. That makes the agent very confused and makes a lot of mistakes. There are a lot of small problems, you would say. Well, you can try to do prompt engineering and improve that, but it turns out to be actually very hard. We realized that the interface design is actually a very omitted part of agent design. So we did this switch agent work. And the key idea is just, even before you talk about what the agent is, you should talk about what the environment is. You should make sure that the environment is actually friendly to whatever agent you're trying to apply. That's the same idea for humans. Text terminal is good for some tasks, like git, pool, or whatever. But it's not good if you want to look at browser and whatever. Also, browser is a good tool for some tasks, but it's not a good tool for other tasks. We need to talk about how design interface, in some sense, where we should treat agents as our customers. It's like when we treat humans as a customer, we design human computer interfaces. We design those beautiful desktops or browsers or whatever, so that it's very intuitive and easy for humans to use. And this whole great subject of HCI is all about that. I think now the research idea of switch agent is just, we should treat agents as our customers. And we should do like, you know… AICI.Swyx [00:42:16]: AICI, exactly.Harrison [00:42:18]: So what are the tools that a suite agent should have, or a coding agent in general should have?Shunyu [00:42:24]: For suite agent, it's like a modified text terminal, which kind of adapts to a lot of the patterns of language models to make it easier for language models to use. For example, now for edit, instead of having no feedback, it will actually have a feedback of, you know, actually here you introduced like a syntax error, and you should probably want to fix that, and there's an ended error there. And that makes it super easy for the model to actually do that. And there's other small things, like how exactly you write arguments, right? Like, do you want to write like a multi-line edit, or do you want to write a single line edit? I think it's more interesting to think about the way of the development process of an ACI rather than the actual ACI for like a concrete application. Because I think the general paradigm is very similar to HCI and psychology, right? Basically, for how people develop HCIs, they do behavior experiments on humans, right? I do every test, right? Like, which interface is actually better? And I do those behavior experiments, kind of like psychology experiments to humans, and I change things. And I think what's really interesting for me, for this three-agent paper, is we can probably do the same thing for agents, right? We can do every test for those agents and do behavior tests. And through the process, we not only invent better interfaces for those agents, that's the practical value, but we also better understand agents. Just like when we do those A-B tests, we do those HCI, we better understand humans. Doing those ACI experiments, we actually better understand agents. And that's pretty cool.Harrison [00:43:51]: Besides that A-B testing, what are other processes that people can use to think about this in a good way?Swyx [00:43:57]: That's a great question.Shunyu [00:43:58]: And I think three-agent is an initial work. And what we do is the kind of the naive approach, right? You just try some interface, and you see what's going wrong, and then you try to fix that. We do this kind of iterative fixing. But I think what's really interesting is there'll be a lot of future directions that's very promising if we can apply some of the HCI principles more systematically into the interface design. I think that would be a very cool interdisciplinary research opportunity.Harrison [00:44:26]: You talked a lot about agent-computer interfaces and interactions. What about human-to-agent UX patterns? Curious for any thoughts there that you might have.Swyx [00:44:38]: That's a great question.Shunyu [00:44:39]: And in some sense, I feel like prompt engineering is about human-to-agent interface. But I think there can be a lot of interesting research done about... So prompting is about how humans can better communicate with the agent. But I think there could be interesting research on how agents can better communicate with humans, right? When to ask questions, how to ask questions, what's the frequency of asking questions. And I think those kinds of stuff could be very cool research.Harrison [00:45:07]: Yeah, I think some of the most interesting stuff that I saw here was also related to coding with Devin from Cognition. And they had the three or four different panels where you had the chat, the browser, the terminal, and I guess the code editor as well.Swyx [00:45:19]: There's more now.Harrison [00:45:19]: There's more. Okay, I'm not up to date. Yeah, I think they also did a good job on ACI.Swyx [00:45:25]: I think that's the main learning I have from Devin. They cracked that. Actually, there was no foundational planning breakthrough. The planner is actually pretty simple, but ACI that they broke through on.Shunyu [00:45:35]: I think making the tool good and reliable is probably like 90% of the whole agent. Once the tool is actually good, then the agent design can be much, much simpler. On the other hand, if the tool is bad, then no matter how much you put into the agent design, planning or search or whatever, it's still going to be trash.Harrison [00:45:53]: Yeah, I'd argue the same. Same with like context and instructions. Like, yeah, go hand in hand.Alessio [00:46:00]: On the tool, how do you think about the tension of like, for both of you, I mean, you're building a library, so even more for you. The tension between making now a language or a library that is like easy for the agent to grasp and write versus one that is easy for like the human to grasp and write. Because, you know, the trend is like more and more code gets written by the agent. So why wouldn't you optimize the framework to be as easy as possible for the model versus for the person?Swyx [00:46:24]: I think it's possible to design an interfaceShunyu [00:46:25]: that's both friendly to humans and agents. But what do you think?Harrison [00:46:29]: We haven't thought about that from the perspective, like we're not trying to design LangChain or LangGraph to be friendly. But I mean, I think to be friendly for agents to write.Swyx [00:46:42]: But I mean, I think we see this with like,Harrison [00:46:43]: I saw some paper that used TypeScript notation instead of JSON notation for tool calling and it got a lot better performance. So it's definitely a thing. I haven't really heard of anyone designing like a syntax or a language explicitly for agents, but there's clearly syntaxes that are better.Shunyu [00:46:59]: I think function calling is a good example where it's like a good interface for both human programmers and for agents, right? Like for developers, it's actually a very friendly interface because it's very concrete and you don't have to do prompt engineering anymore. You can be very systematic. And for models, it's also pretty good, right? Like it can use all the existing coding content. So I think we need more of those kinds of designs.Swyx [00:47:21]: I will mostly agree and I'll slightly disagree in terms of this, which is like, whether designing for humans also overlaps with designing for AI. So Malte Ubo, who's the CTO of Vercel, who is creating basically JavaScript's competitor to LangChain, they're observing that basically, like if the API is easy to understand for humans, it's actually much easier to understand for LLMs, for example, because they're not overloaded functions. They don't behave differently under different contexts. They do one thing and they always work the same way. It's easy for humans, it's easy for LLMs. And like that makes a lot of sense. And obviously adding types is another one. Like type annotations only help give extra context, which is really great. So that's the agreement. And then a disagreement is that when I use structured output to do my chain of thought, I have found that I change my field names to hint to the LLM of what the field is supposed to do. So instead of saying topics, I'll say candidate topics. And that gives me a better result because the LLM was like, ah, this is just a draft thing I can use for chain of thought. And instead of like summaries, I'll say topic summaries to link the previous field to the current field. So like little stuff like that, I find myself optimizing for the LLM where I, as a human, would never do that. Interesting.Shunyu [00:48:32]: It's kind of like the way you optimize the prompt, it might be different for humans and for machines. You can have a common ground that's both clear for humans and agents, but to improve the human performance versus improving the agent performance, they might move to different directions.Swyx [00:48:48]: Might move different directions. There's a lot more use of metadata as well, like descriptions, comments, code comments, annotations and stuff like that. Yeah.Harrison [00:48:56]: I would argue that's just you communicatingSwyx [00:48:58]: to the agent what it should do.Harrison [00:49:00]: And maybe you need to communicate a little bit more than to humans because models aren't quite good enough yet.Swyx [00:49:06]: But like, I don't think that's crazy.Harrison [00:49:07]: I don't think that's like- It's not crazy.Swyx [00:49:09]: I will bring this in because it just happened to me yesterday. I was at the cursor office. They held their first user meetup and I was telling them about the LLM OS concept and why basically every interface, every tool was being redesigned for AIs to use rather than humans. And they're like, why? Like, can we just use Bing and Google for LLM search? Why must I use Exa? Or what's the other one that you guys work with?Harrison [00:49:32]: Tavilli.Swyx [00:49:33]: Tavilli. Web Search API dedicated for LLMs. What's the difference?Shunyu [00:49:36]: Exactly. To Bing API.Swyx [00:49:38]: Exactly.Harrison [00:49:38]: There weren't great APIs for search. Like the best one, like the one that we used initially in LangChain was SERP API, which is like maybe illegal. I'm not sure.Swyx [00:49:49]: And like, you know,Harrison [00:49:52]: and now there are like venture-backed companies.Swyx [00:49:53]: Shout out to DuckDuckGo, which is free.Harrison [00:49:55]: Yes, yes.Swyx [00:49:56]: Yeah.Harrison [00:49:56]: I do think there are some differences though. I think you want, like, I think generally these APIs try to return small amounts of text information, clear legible field. It's not a massive JSON blob. And I think that matters. I think like when you talk about designing tools, it's not only the, it's the interface in the entirety, not only the inputs, but also the outputs that really matter. And so I think they try to make the outputs.Shunyu [00:50:18]: They're doing ACI.Swyx [00:50:19]: Yeah, yeah, absolutely.Harrison [00:50:20]: Really?Swyx [00:50:21]: Like there's a whole set of industries that are just being redone for ACI. It's weird. And so my simple answer to them was like the error messages. When you give error messages, they should be basically prompts for the LLM to take and then self-correct. Then your error messages get more verbose, actually, than you normally would with a human. Stuff like that. Like a little, honestly, it's not that big. Again, like, is this worth a venture-backed industry? Unless you can tell us. But like, I think Code Interpreter, I think is a new thing. I hope so.Alessio [00:50:52]: We invested in it to be so.Shunyu [00:50:53]: I think that's a very interesting point. You're trying to optimize to the extreme, then obviously they're going to be different. For example, the error—Swyx [00:51:00]: Because we take it very seriously. Right.Shunyu [00:51:01]: The error for like language model, the longer the better. But for humans, that will make them very nervous and very tired, right? But I guess the point is more like, maybe we should try to find a co-optimized common ground as much as possible. And then if we have divergence, then we should try to diverge. But it's more philosophical now.Alessio [00:51:19]: But I think like part of it is like how you use it. So Google invented the PageRank because ideally you only click on one link, you know, like the top three should have the answer. But with models, it's like, well, you can get 20. So those searches are more like semantic grouping in a way. It's like for this query, I'll return you like 20, 30 things that are kind of good, you know? So it's less about ranking and it's more about grouping.Shunyu [00:51:42]: Another fundamental thing about HCI is the difference between human and machine's kind of memory limit, right? So I think what's really interesting about this concept HCI versus HCI is interfaces that's optimized for them. You can kind of understand some of the fundamental characteristics, differences of humans and machines, right? Why, you know, if you look at find or whatever terminal command, you know, you can only look at one thing at a time or that's because we have a very small working memory. You can only deal with one thing at a time. You can only look at one paragraph of text at the same time. So the interface for us is by design, you know, a small piece of information, but more temporal steps. But for machines, that should be the opposite, right? You should just give them a hundred different results and they should just decide in context what's the most relevant stuff and trade off the context for temporal steps. That's actually also better for language models because like the cost is smaller or whatever. So it's interesting to connect those interfaces to the fundamental kind of differences of those.Harrison [00:52:43]: When you said earlier, you know, we should try to design these to maybe be similar as possible and diverge if we need to.Swyx [00:52:49]: I actually don't have a problem with them diverging nowHarrison [00:52:51]: and seeing venture-backed startups emerging now because we are different from machines code AI. And it's just so early on, like they may still look kind of similar and they may still be small differences, but it's still just so early. And I think we'll only discover more ways that they differ. And so I'm totally fine with them kind of like diverging earlySwyx [00:53:10]: and optimizing for the...Harrison [00:53:11]: I agree. I think it's more like, you know,Shunyu [00:53:14]: we should obviously try to optimize human interface just for humans. We're already doing that for 50 years. We should optimize agent interface just for agents, but we might also try to co-optimize both and see how far we can get. There's enough people to try all three directions. Yeah.Swyx [00:53:31]: There's a thesis I sometimes push, which is the sour lesson as opposed to the bitter lesson, which we're always inspired by human development, but actually AI develops its own path.Shunyu [00:53:40]: Right. We need to understand better, you know, what are the fundamental differences between those creatures.Swyx [00:53:45]: It's funny when really early on this pod, you were like, how much grounding do you have in cognitive development and human brain stuff? And I'm like

Tiff and Dana drill down on how practices can avoid their hygiene schedules falling apart, including the benefit of ASAP lists and their opinions on re-care. Episode resources: Reach out to Tiff and Dana Tune Into DAT's Monthly Webinar Practice Momentum Group Consulting Subscribe to The Dental A-Team podcast Become Dental A-Team Platinum! Review the podcast Transcript: The Dental A Team (00:06.99) Hey, Dental A Team podcast listeners, we are back at you again. We are so excited to be here. I have Miss Dana with me today. You guys know she's one of my favorites to podcast with. Dana, I have to say, I know a couple podcasts ago, you guys may remember, I asked her about some legging recommendations. She gave me some hiking, like legging recommendations. I wasn't able to find the specific pair that you talked about yet, but I did find a Lulu pair.   that I used in Yosemite last week. And honestly, I've used them for a couple of hikes here in the Valley as well in Phoenix. Amazing. I was so happy because they were specifically that material. It was a different kind of material than regular leggings. And I'm so happy. And I was thinking of you the whole time. And I was like, my gosh, she was right. And it had so many pockets for my phone and all the things. So thank you, Dana. And I know you've also been really, you've been working really, really hard.   in personal life in your world of fitness. And I think it's really cool. And I wanted to acknowledge that today. And I wanted to let the world know that Dana is a freaking rock star. She was so strong. saw, I don't even remember the rec the other day that I saw you, you posted something and I was like, holy cow, like the things that this woman is capable of doing are incredible. So kudos to you, Dana. I love it. What is your, I always like to ask you a question. What's the most fun   thing in CrossFit that you're doing right now. Like what's something that you're working to overcome, accomplish, you're killing it on, something that you're really having fun with right   Dana (01:40.331) Yeah, I mean, I feel like I'm always working on something, right? And I think this year I turned 40. It was a big birthday. And my goal was to be the strongest that I've ever been. So I've been focusing a lot on strength, but I always am practicing skill work. And I finally got my rope climbs down. So I can actually do those like consecutively. So that's exciting. And I've been working on my handstand pushups to the point where I can do five in a row. So I'll take that.   The Dental A Team (02:02.895) So cool.   Nice.   The Dental A Team (02:09.935) Okay. Well, I mean, I struggle with five regular pushups, not on my head. So that's impressive to me. I know that in CrossFit there's people that are doing a lot of amazing, wild things, but to me that is amazing and wild. So congratulations. That's huge. Awesome. I love it. I love it. Well, you work really hard in and out of the quote unquote office. So thank you for always being here and thank you for always being willing   Dana (02:24.629) Thanks.   The Dental A Team (02:39.722) record these podcasts with me for our listeners. I know they're super beneficial. Today's content is actually really fun for me because by trade in case you guys did not know, Dana is a licensed and trained professional hygienist. That is what she did. I mean, she's done it all. She's done all the positions really most of them I should say not dentist, none of us has been a dentist, whatever, it's fine. But she was a hygienist, you guys for a long time and she still is a hygienist.   I think it's really cool. think it's always fun for me to have these conversations with you to gain your perspective from your side as well. Today I wanted to talk about really keeping the hygiene schedule full. So I know from my perspective from dental assistant and then from my perspective as a front office, which you also have that perspective, it's kind of like a free for all a lot of the times. And it's really just not always first or high priority.   But from the hygiene standpoint, Dana, what is that like? Let's just go through, what does it feel like when you look at a schedule and you're like, great, this is a killer day, it's gonna be awesome. You get through your first patient and then you look back at your schedule and it's totally falling apart. Or you look ahead and you're like, gosh, I only have scheduled, I'm only scheduled out a week and a half. Like, what is that like as a hygienist looking at your schedule and seeing   Dana (03:55.689) Yeah, I mean, it can be stressful because really for me, I mean, you probably know this about me, Tiff, but I'm pretty goal oriented. And so obviously, I know when I don't have patients on my schedule, that affects my ability to get to goal. then to like, I am there, the reason why I'm there and what I love is that patient care and being able to take care of my patients and do the things that they need when they're in my chair. And so if my schedule isn't full or isn't ideal, I don't get to do that as much as I ideally would love. And so it can be stressful, it   The Dental A Team (04:01.997) Yeah.   Dana (04:25.633) be frustrating, right? Because so many times, like hygiene is just like your routine care, right? And so it's, it doesn't, it is stressful as far as like, did I not build enough importance for that visit? Or where did maybe, you know, my communication with that patient make it so that they didn't value their time with me? And so I think that it can be frustrating and stressful for every hygienist.   The Dental A Team (04:49.836) Yeah, I think that makes sense. And I think that's important what you just said, because you just took it from a spin that I didn't even expect. And I think most probably team members for an office workers and assistants don't expect that either because it feels, or we allow it to feel, we take it very personally when a hygienist is like what happens to our schedule. But you have taken it personally when a patient falls off of your schedule. So you're already in a space   unrest of like upset of gosh, what could I have done differently? Or how could I have been better? And then we come out I'm gonna say we for the front office, we come out at oftentimes of like, hey, I did the best that I could, what do you expect from me? So then you're already you're already in a space of upset and unrest and, and trying to figure it out. And then we come at you like, don't come at me. And now you're you know, the hygiene team   then maybe take offense to that and become defensive. So that makes sense. I'm actually really glad I asked you that question, because I'm not sure I've ever asked that or dove into that. And I hope the listeners heard what Dana said as well as, and then can kind of see that picture painted of how an upset could occur between two people or two groups of people very easily because of one misunderstanding. We're both kind of taking offense to a situation without understanding the other person's point of view.   That was actually incredible. Thank you. I knew you would say something really great, but I wasn't even expecting as far as that went. So that was really fantastic. Thank you. I see within that a space of communication. So making sure clear expectations are made from the get -go. That we're always asking more questions and assuming less. That's a reminder I love to give everyone in my life. Ask more questions, assume less.   and assume the only assumption can be that you don't know the answer.   The Dental A Team (06:50.57) ask more questions and assume less and assume the only thing you can assume is that you don't actually know the answer because you're not the other person. So that's huge right there because I think that's a space where people can really say, hey, let's talk about this. And if we have solid systems set up from the get -go, a good foundation of what do we do if something falls off the schedule? What do we do when the schedule falls apart?   That's a really easy space to meet expectations and hopefully refill a schedule before a hygienist even notices. That's my favorite thing to do is to make sure that he or she has a full schedule and doesn't even know it fell apart. So with that, some common, common ways to do that, we're gonna dive into a few for you guys today so that you can really be prepared to meet those expectations and really clear up a lot of communication between the two of those two departments or three departments.   My favorite and Dana, think we work on this a lot with practices, blocked scheduling and ASAP lists. Those are my two favorites. I love blocked scheduling. We talk about this in a lot of different podcasts for a lot of different reasons, but within hygiene, it really helps set and build the value and ensures that your new patients are scheduled where they need to be. When those new patients schedule blocks and those SRP blocks are in place correctly, we're scheduling   out into the future. as they come off, it's very easy if someone falls off the schedule, maybe an SRP got called into work and can't come later today. Well, guess what, I've got another SRP that's very easy to find, that's probably about the same time, if not exactly the same time of day, that can likely come in and I can choose that one to call and bring them in earlier. The ASAP list is super similar, like what time of day are they looking to come in, making sure we're notating all of those things, every system as far as I know,   Has the capability of an ASAP list. know Dentrix does, Open Dental does, Dana's saying yes, so that means Eagle Soft does. She knew that was my crux. Ascend, I can't answer that question. And then Curve does, and I bet Care Stack does as well. So utilize those ASAP lists. If you don't know how to do it within your system, Google you guys. I know we get a lot of those questions and I'm happy to answer them. So you can reach out, Hello@TheDentalATeam.com as   The Dental A Team (09:08.058) But usually if we don't know the answer, we're typing it in Google for you guys. So Google at first reach out to us if you need it or if you need help creating the system around it. But that ASAP space with those blocked schedules really helped to solidify a grouping of people to pull into those holes. Dana, how do you help practices see the system behind a good ASAP list? Because I know a lot of practices really struggle to see the value   creating it and can sometimes feel like it's redundant or they're like, gosh, I've called all these people. What kind of systems do you put in place for   Dana (09:46.707) And I think this keeps coming up more and more right now because a lot of offices are short on hygiene hours and so the last thing we want to do is have an opening and there have been an easy person to put in there but it's not easy because we rely on our memory and let's face it as much as our memory is great, right? Four or five even close to six months later not so great if we're all being honest with ourselves and so I think the Where I see a lot of the pushback on the ASAP list is if they've tried it and it's messy,   The Dental A Team (10:09.57) Mm -hmm.   Dana (10:16.373) There's no cleanup system. There's no like we just keep calling the same people. There's no anything There's not great notes in there So my first thing is who do you put on the ASAP list and really the ASAP list is anyone who rescheduled who canceled right or rescheduled and didn't get back on our schedule I guess so they just canceled or   The Dental A Team (10:30.012) sorry.   Dana (10:35.749) They didn't and they didn't reappoint so putting those people on your ASAP list because you know They were due or our past do and need to get in fairly quickly Anyone who we had to schedule out a little bit farther than we'd like to and who wanted sooner or anyone who we put in a not ideal time for them and they would actually prefer something different so those are who goes on your ASAP list and then it's a really it is just when do we call them and When do we clean them   Right? And so it's just kind of creating those rules of thumb. So when we call them, obviously when we have a cancellation or we have somebody who even we anticipate calling side booking if we need to. So that would be when you would use the list and then it's just an easy cleanup   how many times are we gonna reach out to this patient and offer them something sooner before we take them off and we just give them their original appointment because we've tried to make it within our schedule and we don't wanna keep also bugging them to a certain extent. So it's just kind of coming up with that cutoff. I usually say like three or four times and then they just come up the ASAP list or making sure that we're checking our ASAP list and is somebody on there already have a sooner appointment? Like did we already do something and we just left them?   The Dental A Team (11:23.946) Okay.   Dana (11:49.039) on there. So it's just having those strategies that make ASAP list work. just oftentimes can get clunky because we don't have those strategies in place when we start it.   The Dental A Team (11:57.04) Mm -hmm. I totally agree. I totally agree. And there have been so many times when I was in practice that I would say to a patient, like, gosh, I know for me it would be really hard, you know, to move my appointment up in my life, but I like to think that it would be easy. So I know we've offered a couple of times, but I don't want to keep bugging you. Do you me to keep calling if they come up, or do you want me to take you off the list? And oftentimes patients will be like, no, it's not a bother at all. And I know it's annoying. Like, I can't make this one, but please keep trying. And I'd be like, totally fine. It's not a problem whatsoever. I just want to make sure I   driving you crazy with the phone calls or patients would be like, yeah, you know what, you're right. Like, I'm probably never going to be able to come in early. So just take me off. I'd rather just keep the plan that I have. So it helps to clean it up. Just asking a simple question of, is this beneficial for you? Or is it a nuisance? Because either way, I want to make sure that you're satisfied. So and I think we think we're invading, right? That's always the issue. We always feel like we're invading somebody's bubble or we're bothering them. And we don't want to be a   So then we shy away from calling them or like, gosh, I just called them two days ago. I'm not going to call them again. But if we ask them upfront and they tell us it is, or is not a bother, now we can make that decision that goes back to not making assumptions. So I think that's huge. Now, Dana, I preach this to all of my clients, all of my practices I work with, re care, make her re care, re care. When I was first new into dentistry, I was a few years into dentistry, my   dearest doctor they worked for for a long time. He was a re -care maniac and he used to drive me insane until I saw, okay, I get it. He's like, we're not six months booked, we've got a problem and he was just hammering this re -care constantly and I was like, what is the deal? But when we stop calling people and we stop being like front of mind, right? Top of mind, we're not in front of them anymore is when they forget about us. So whether they're scheduling or not,   us sending the reminders, the text messages, the emails, the quick phone calls, it's just like, hey, checking in on you just wanted to see where you're at in life. That's putting us top of mind so that when they do have a second to schedule that appointment, we are the place that they're calling. I say, call until call and text and remind until someone says I'm not coming back. Dana, what's your philosophy on re care, especially coming from a hygiene standpoint, right from a hygienist?   The Dental A Team (14:21.601) who needs to see these patients and wants to see these patients again in your chair, how do you feel about the repair space?   Dana (14:27.531) I mean, I 100 % agree with you. I consider myself a fairly good patient, but I'll be honest that it's most important when I'm in your office. And so if I don't schedule when I'm in your office, I'm going to need those touch points because life happens. And as much as I want to stay on top of my dental care, I want to be there every six months. Sometimes I just do that friendly nudge of like, hey, don't forget, you never did schedule with us, give us a call. And so I agree with you. think putting it in automated and then also reaching out personally, I think hitting it from both sides.   really can help keep people consistent and keep you, like you said, top of mind.   The Dental A Team (15:02.156) Totally agree Dana. I think that was beautiful. You have such an amazing perspective coming from the hygiene side and really just seeing how the dental assisting side, the hygiene side, the front office side, like all of these departments just combined together to get the job done on keeping a hygiene schedule full. It's just so important. It's so imperative. And I love what you said there. And just gathering all of those pieces, hygiene department, this is coming from you guys in my opinion. And I hope you guys share in that because it was a really cool perspective. Now,   I just want to recap, Dana, thank you so much for being here with me today and for allowing me to pick your brain and put you on the spot. I know you're never expecting all of the questions that I ask you and you take them with grace and with love and dignity and all of the pieces and you showed us some aspects of hygiene and that I wasn't expecting today. So was really, really cool. So thank you. And you guys out there, I really want you to put this powerful message into play. I want you to take all of these bits and pieces and really help keep your own hygiene schedule.   super full, that's what we're here to do today. That's what we're hoping for is to just have this amazingly full hygiene schedule. Now, here are the things you guys ASAP list if you're not using it within your system, do it now. I know a lot of people like the handwritten notes, like we said, but put it in place. Always, always, always be calling your re -care. You guys, I know this can be scary when you don't have anywhere to put them or you're worried about where you're going to put them,   When we back off is when we start to see in the future, the schedule really start to slow down. Blocking out new patient spots. And then you guys just have a plan for what happens, what you're going to do, what your team is supposed to do. What's the system for when things fall off the schedule? Who is responsible? How are they going to fill it? And who are they going to enlist to help if they can't do it themselves? So make sure that you have all of these systems in play, you guys, and in place.   The plan is going to be my number one spot because all those other pieces fall into that plan. You guys go do it, go implement it and let's see what you can do with this hygiene schedule. As always, thank you for being here with Dana and I today. Thank you so much for listening to this podcast. And if you have ideas that you're open to sharing to the dental industry and the dental world, drop them below in a five star review so people can see them. They do read the reviews, especially when it comes to individual   The Dental A Team (02:26.671) podcast so if you've got some ideas pop them in there otherwise find us on our social platforms as well the dental a team and then Hello@TheDentalATeam.com is where you can submit any emails over to us if you have ideas if you have questions if you want some systems if you want to bounce ideas off of us we're here for that and open to it thank you so much guys and we'll catch you next time. 

Join us in this episode as we sit down with Leo Segovio, a top expert in the space, to discuss a wide range of topics that are essential for E-commerce sellers. Leo shares his unique insights on how optimizing Amazon images can significantly impact indexing and ranking. He also opens up about his recent ventures, including a software project for influencer and affiliate marketing, and an intriguing Airbnb project in Italy. Additionally, Leo provides valuable tips for Amazon sellers looking to diversify their income by investing in real estate, highlighting the importance of strategic investments to complement a thriving Amazon business. Listen in as we explore the evolving landscape of influencer and affiliate marketing strategies. We discuss how leveraging platforms like TikTok, Instagram, and YouTube can empower brands by building robust affiliate networks. We highlight successful brands and share advanced techniques for optimizing listings to ensure better visibility on Google and Amazon. Practical tips for using press releases on high-authority domains to improve Google indexing are also discussed, offering listeners actionable advice to enhance their marketing efforts. Finally, we talk about the significance of Google indexing for Amazon sellers and the benefits of driving traffic from Google to boost Amazon rankings. We discuss the theory that paid traffic may hold more weight and the value of optimizing images with keywords to enhance discoverability. Additionally, we examine Amazon's evolving search algorithms and how intent-based optimization is changing the way products are discovered on the platform. This episode is packed with valuable insights and strategies to help Amazon sellers navigate the complexities of e-commerce and achieve greater success. In episode 568 of the Serious Sellers Podcast, Bradley and Leo discuss: 04:14 - Investing In Real Estate Investments 09:41 - Leveraging Creator Marketplace for Affiliate Networks 15:56 - Google Indexing for Amazon Sellers 18:16 - Google Traffic Boosts Amazon Ranking 24:01 - Google Indexing Boosts Product Visibility 26:46 - Search Algorithm Evolution and Intent-Based Optimization 29:13 - Optimizing Amazon Listings for Intent-Based Search Transcript Bradley Sutton: Today we've got one of the top minds in the entire Amazon game back on the show, Leo Segovia. He's going to be talking about a wide variety of topics, such as the impact on indexing and ranking by optimizing your Amazon images, and much, much more. How cool is that? Pretty cool, I think.   Bradley Sutton: Hello everybody, welcome to another episode of the Serious Sellers Podcast by Helium 10. I'm your host, Bradley Sutton, and this is the show. That's a completely BS-free, unscripted and unrehearsed organic conversation about serious strategies for serious sellers of any level in the e-commerce world. And speaking of the e-commerce world, I'm on the other side of the world right now. For those of you listening, maybe I sound a little different. We are in the AVASK office here in Madrid, Spain, right in the middle of our Elite Workshop, and just about 15 minutes ago we had our very first speaker. All the other speakers are very mad at him because he started off and he set the bar really high with his talk, but we've got no stranger to the show, Leo Segovia. Leo, how's it going?   Leo: Bradley, good morning. How are you doing?   Bradley Sutton: Doing great, doing great.   Leo: Awesome. Yeah, this morning was great. I'm actually happy this is my first time in Madrid. Yes, I actually just stopped once. I think I was on my way to Puerto Rico, but yes, I got to enjoy the city. Today I'm here at the AVASK office in Madrid, so happy to be here and happy to be your guest again.   Bradley Sutton: Awesome, awesome. So now you know it's been a while since Leo's been on the show, so let's first just catch up with what you've been up to. Have you been launching products on Amazon? You've just been focused on building software. What have you been up to the last couple of years since you've been on the show?   Leo: Yeah, it's been a crazy year for me actually. I've been involved in a couple of different projects. We are obviously always looking for new products to launch. What kept me very busy in the past year has been software that I've been working on for influencer and affiliate marketing, and actually this Airbnb project in the south of Italy, which has been kind of a roller coaster. Yeah.   Bradley Sutton: So, you actually moved, I remember you went from Canada to Florida and then a few months ago you moved back to your home, uh, country of Italy, but then this was always meant to be kind of just like a like a winter, uh or summer home for you.   Leo: Yeah, that is correct. Uh, I have a family in Italy. So, and recently their area of Italy is called Puglia, it's in the southeast was becoming more and more popular and more expensive, and so I decided to buy a property there so that we could spend a week or a month in the summertime, perhaps, when in Florida is too hot, you know, go inside of Italy. Invite some of my Amazon friends, you know, mastermind, and so that's the plan. Now, I was supposed to be there only for a couple of months, just to see what was going on, but when I got the keys, I realized that the place needed a lot of work, and so I've been stuck in Italy since November, actually, of last year, and I'll probably stay there until for two more months before going back to Miami.   Bradley Sutton: What passport do you have? What country passports? I have Italian and Canadian passport. Okay, so then, when you bought this house, you use your like Italian citizenship?   Leo: No, actually I well, I could participate to an auction because I bought this place at an auction. Not the $1 ones, it was more than that. But yes, because of my Italian citizenship it allowed me to participate to an auction. But everything that I'm doing is as a Canadian citizen. It works out better from tax perspective and all that.   Bradley Sutton: Okay. So that's why I was asking about this, because I think this is, you know, like somebody might be jumping on the show. What are we talking about? Airbnb here? But as e-commerce entrepreneurs, Amazon sellers, maybe we make a little money, maybe we're not interested in exiting our business, but now we have extra money, like do we start other businesses? You know, maybe something that has nothing to do with Amazon, but I hear of more Amazon sellers doing something similar. Where they go you know, not necessarily Italy, but another country, buy a house and then so, as a Canadian citizen or as an American citizen I would assume it's about the same. What's the process of participating, like in this Italian auction to be able to buy this house?   Leo: I think you need to have someone in Italy or a friend, someone with an Italian citizenship, in order to buy a place at auction. Otherwise, you just have to go to a real estate agent and buy a regular place. The reason for me it was convenient is because it was a good deal. If I was able to win the auction, and so in real estate, you make money when you buy, not when you sell. Right, if you buy for less, that's most likely guaranteed revenue or earnings whenever you sell, and so that's the reason why I did this. Now, I don't know exactly the process if I didn't have any Italian citizenship, but yeah, a lot of entrepreneurs you know, especially Amazon sellers whether, when they exit or you know if they're already doing quite well and they have good cashflow, they normally tend to invest in real estate Airbnb's. You secure yourself passive income from that, and it's always a good investment.   Bradley Sutton: So then would I have to have all cash though to once the auction closed, I can finance over there.   Leo: Okay. So that's an interesting thing. I was going to finance the project. I ended up buying a cash because it just made more sense for me, but in Italy they actually give you a mortgage as long as you can prove that you have income outside of Italy.   Bradley Sutton: Okay. And then so you calculated out, like how much you can maybe get an Airbnb. And then so have you calculated hey, for the other months of the year where I'm not staying here, I need to rent this place out x amount of time of the year. And it's going to be worth it, have you like, and did that analysis?   Leo: Yeah, so in this specific region of Italy and the location of what I bought in August for a place with a pool and four or five bedrooms, you can charge 5,000 to 6,000 euros a week. So you make your money in the summertime. Ideally, as an investor, you don't want to go and spend time in the summertime there, but you want to go, perhaps either early, like May or September, when the season starts to kind of slow down and so you don't take out money from your profits, right? So my plan is to rent it out June, July and August. If I have some good offers in September, maybe I'll rent it out, otherwise I'll go myself there in September or May, but, yeah, normally throughout the year.   Leo: You know Italy is a destination where you have a lot of tourism during summertime, unless you're in Rome or Venice or Florence, which is always busy throughout the year. You know south it's a summer destination, right. So you get a lot of tourism summertime. Wintertime dies down, so you probably can get us or what you can get in the summer. But you know it works out well because if you have a small apartment, for example, in a big city, and you are charging, you know, 200, 300 a night? Um, at the end of the year you make the same money. So with this kind of properties is a little bit of a different um investment. I went more on the luxury kind of market, hoping to work only with Americans. You know foreign tourism, but in my opinion it's a great one.   Bradley Sutton: Okay, so there you have it, guys. You know, like, maybe you've had some success on Amazon and you're thinking of what kind of things to invest in. You know, getting a property at a low price and maybe fixing it up even though it's a headache a little bit, you know could be the route that you want to go. Now you know we're going to talk a lot about some really cool Amazon strategy coming up, but you've been developing some software lately for a while now. That's not necessarily for on Amazon, but it helps Amazon sellers. Can you talk about that a little bit?   Leo: Yeah, I've been working. The software is called Spliced. I've been working on this for about two years now and I was supposed to be already in market, and the reason why I'm late is because of what I just explained. It took me, you know, it took resources and energy a little bit off the other project, but now we're ready to go, and the reason why I built this software is, you know, Bradley, you know I have Convomat, which was my first software that I built, and then Amazon changed it to iOS, and so I had to find a way to pivot. But I already knew that influencer and affiliate marketing was the way to go, also for us Amazon sellers, in order to have a little bit more control over the traffic, over the business and the revenue that we drive to our brands. And so, with Spliced, my goal is to leverage the creator's marketplace, which is huge between TikTok and Instagram and YouTube, and leverage that to build affiliate networks for your own brand.   Leo: There are already a lot of sellers out there that are doing a good job when it comes to affiliate marketing. Look at a brand like Goalie. Goalie, one of the key strategies for Goalie was actually the affiliate marketing, and so with Spliced, my goal is to allow brands to look up into our marketplace, which has already been built with we have over 20 million creators and then approach them with an affiliate partnership instead of just UGC content. This is the reason why we didn't build in Spliced, just UGC campaigns. There's already plenty out there of softwares that you can use for UGC, but in my opinion, if you have a solid affiliate network, you can keep launching new products, relaunching the same products. We use it for reviews. If you need something like that and you have more control over your business, and if you decide to launch your D2C website, you can leverage the same network and start pushing traffic off of Amazon. So there are a lot of reasons why I believe people should use a platform like that. It's like building an email list, but instead you're actually leveraging the creator marketplace.   Bradley Sutton: Okay, yeah, interesting, I think TikTok how it works, people's eyes are really open to more of influencer marketing. I mean, it's basically influencer marketing. There's not really SEO on TikTok, or it's not even. Even if you understand the hashtags, it doesn't necessarily guarantee the rally. It's a numbers game like getting you know. You get out to 25 influencers and maybe 24 do nothing, and then one person, even though they're small, they get on the For you page or something like that and literally can bring thousands and thousands of dollars. There's somebody I've been helping with, you know, shipping their products and they're you know they're doing me sometimes some days a thousand fifteen hundred units of sales for like this planner and it was a hundred. They didn't spend any for PPC on TikTok 100%. You know they just push the product to influencers and then one here and one there just goes viral and it just means a lot of business.   Leo: Yeah, I think you know it's probably right now a big hype. I mean the TikTok shops everyone is talking about them working with affiliates and it's probably one of the oldest marketing strategies, if we want to call it that way. You know the affiliate marketing works because of the power that the individual creator in this case has to influence people right, and so people want to buy from people, and if you, as a brand, do a good job in recruiting a few super affiliates in this case we're talking about good creators that will turn into affiliates, then you have to worry less about that promotion part of you know launching new products.   Bradley Sutton: Interesting, interesting. Okay, now let's move a little bit back towards the Amazon world and actually I'm going to go a little bit off of Amazon, but it's something that you talked about today in your speech and we're not going to go too deep into it. If you guys want to really hear his presentation, you have to be an Elite member. So you Elite members definitely make sure to look out for the recording on that. But one thing you were talking about when it comes to images, but the way you discovered this was you said you were checking indexing on Google. So we know, on Amazon, if you want to check indexing, you just use Helium 10 index checker, right? Or if you don't have Helium 10, you can use the old school method of put the ASIN plus the keyword and then search and see if it comes up. For just rudimentary index checking for keywords. If you want to see if your Amazon product is indexed on Google, how do you even see if you are?   Leo: Yeah, so normally on Google you will copy your URL, search it on Google. You can also do a site column with your URL and then Google will show only search results that are related to the domain you're searching. But if you type the whole domain, the whole URL, the canonical URL of your Amazon listing, if you are indexed, it will show there.   Bradley Sutton: But what about if you're? Can you look if you're indexed for a specific keyword?   Leo: So if you're indexed for a specific keyword, then you want to put that URL plus the keyword and then or amazon.com/dp/your ASIN, or you can also do ASIN in quotes plus the keyword and then you will see if you get, if you're indexed on Google from that keyword. It works in a similar way. Um, but yeah, the presentation we touched a lot on you know the details of what was going on Google which was dependent on, uh, the way that the listing was optimized on Amazon.   Bradley Sutton: You talked about some advanced strategy. We'll talk a little bit about that, about like images and stuff. But without the images, is there a way to force yourself to be indexed on Google, like, for example, if you create a custom canonical URL, just insert the keywords and then if you actually happen to you know, like maybe run some Google ads, get some conversions on that, will that index you for that keyword on Google?   Leo: Yeah, so, based on some experiments that I've done, the easiest way to get indexed on Google is to publish some press releases on domains with good authority score domain rank and have your you know pointing a link to your listing with the anchor text that you potentially also want to rank for using that specific canonical tag that you get from your Amazon listing. So the reason why this works better is because normally Google indexes across these websites. Like you know, if you publish through PR, news or something like that, they will be crawled, and so Google will find these links and then follow your Amazon listing, which obviously, as a consequence, would be indexed.   Bradley Sutton: Okay, interesting. Now, taking it a step further, why should an Amazon seller even be concerned about indexing on Google like, um? Obviously, if you're running Google ads, you know your goal is to get direct sales from it. But just being ranked organically, um, what kind of bumps do you see on sales? Or how does it help a to be ranked high? I'm not just index. I mean index doesn't do much if you're ranked on page 30 or something, but how does ranking organically for a keyword? What's the potential there for helping sales?   Leo: So there, are a few reasons why you want to be indexed on Google, and for the most, let's start from the most advanced ones, right? Advanced sellers they normally try to send traffic to Amazon, especially during the launch period, using external traffic, right? So Google, we know, is a good referral that tends to help your rankings, and so Amazon tends to reward you if they see traffic coming from Google. So if you're not indexed, you lose a chance to show Amazon that you are getting traffic from Google. Now, I have a theory that paid traffic has a little bit more weight than organic, but the reason why you want to be indexed and the reason why you might want to be indexed for certain keywords is so that when you drive traffic through the URL to Amazon, you can actually give attribution to that keyword. That's number one, right? So you can actually use these URLs as your two-step.   Leo: Number two if you do a good job with your indexation and your listing is optimized, you actually also appear in the images, right? And so if people are looking for specific products, sometimes I search on Google using images because I'm looking for specific products that might be hard to find on Amazon, but if I look through the Google images and I find the product, then I go to Amazon, and so if you're not indexed, you're also not going to be able to be found there, and Google images actually gets a ton of traffic. So here are some of the reasons why, two of the reasons why. I can think of many more, but the most important are these ones. Google is still one of the largest search engine, and so missing out on that opportunity, I'm afraid it causes a lot of missed visibility for an Amazon seller at a listing level.   Bradley Sutton: And then you've done some tests before where you noticed that if that Amazon can read what the search was from Google, so that when you get sales from a keyword in Google, it also potentially could help your Amazon ranking for that keyword, right?   Leo: Yeah, that is correct. There was a test that we have done two years ago where everyone was talking about Google traffic and so we drove traffic straight from Google paid to Amazon without using any keyword in the URL, and then we noticed that for the keywords that we were actually bidding on, we saw a lift in ranking. I remember going from position I think it was 35 or so to position seven or five. So surprisingly we saw that Amazon was able to attribute that search query on Google and then the ranking as a result for the keyword was actually improving on Amazon as well.   Bradley Sutton: Okay, interesting, interesting. Now let's switch and talk a little bit about images, because this also has to do with ranking on Amazon. It has to do with ranking on Google, getting indexed in Google. What has more of an impact with getting discovered or being able to be read by Google? Is it if you have an infographic and the actual words appear in the infographic, you know on the actual image, or is it the metadata, or it only works the best if you're doing both?   Leo: In my opinion, you have to do both, and the reason is that right now, every search engine uses AI to detect subjects, text and everything on an image. You know, if you look, if you're a Facebook advertiser, you probably know that they've had this for a long time. If you add more text on an image than the image, the visual itself, your ad wouldn't have been approved, and so AI detection for images has been going on for a while. But now, since you know, ChatGPT came out and you know Lama from Facebook, we have, you know. We know we have a lot more information about this topic, and what we found is that the search engines, including Amazon and Google, they scan the content of your image and they're able to rank these images based on the content of that image, including subjects, context and in the subjects and text. Did I say that?   Leo: So, basically, what Google cares the most on top of that is also the metadata, because the metadata helps the search engine classify that image. So, while the content itself helps them understanding okay, this is what this image is about the actual metadata is more technical for the crawlers, the engine themselves so that they can place you in certain categories. And so when it comes to Amazon, the content on the image right now, I noticed that through some different experiments, that is being used for ranking reasons. And so if you look at some products that don't have, for example, keywords on the images, they are less. You know there are multiple factors. Obviously, they play when it comes to rankings, but if you put two products side by side same ranking, same ratings, same being on market for the same time period, timeframe and same price one has text on images and keywords and one doesn't. Most likely the one with keywords on images is going to rank better.   Bradley Sutton: Okay. So then what Amazon sellers should be doing is for their main images, or you know, the in their image carousel and their A+ content is I mean, obviously you can't have text in your main image. You know that's against terms of surface, although if you can have the packaging there, that's a good, that's a good opportunity. But then to get, hey, you use the right keywords, but then also, if you're using like photoshop or something you have and we're not going to go into detail, it's like there's a bunch of crazy stuff about copyright and there's fields there that he talks about in his presentation. You'll have to watch the Elite workshop for that. But you've been doing testing where it one has, it one doesn't, and then it gets you indexed on Google. You've actually seen where the ones who did it, their Amazon sales were like way higher than the ones who did it.   Leo: Yeah, that is correct. We analyzed an e-brace on Amazon and this is, you know work that I was doing with a friend of mine, and we were trying to understand why these competitors were actually indexed on Google and they were indexed for certain keywords. Not the main keyword, but a variation of them. And so what I did I created this Google sheet where I was helping me understand which ones were indexed and for what keywords they were indexed and that led me to see that the ASINs that actually were indexed on Google were indexed for keywords that were present on the A+ banners. And so when we did that, what happened, this happened within 48 hours, we noticed that Google indexed that specific product image and they were actually featuring it as a search result on Google for the main search query, so that image wasn't used as a snippet or thumbnail for the listing itself. So the URL wasn't amazon.com/dp/ASIN, it was amazon.com/ the search you know embrace.   Leo: So it took me to the search results page, but the image that they took as a featured image was actually the one of my client, and so that was very interesting because Google detected a refinement and it detected an update in that listing. It saw that that image was very relevant for the search query because of the way that we optimized it using metadata and then they used it as the main image on the Google search results. Now this, to me, is fascinating and is very important, because if you are a shopper and you're searching on Google for an e-brace and then you see this image, most likely that's psychological,   most likely when you land on the Amazon search results page, you're going to go and find a product, you're going to go and click that product. So that added traffic, that added conversion rate, helped us recover the racing and the sales that we were losing. But that was a very interesting experiment that we did.   Bradley Sutton: Interesting, okay. So again, if you guys want to get more information about that, that almost might be worth it just to subscribe to Helium 10 Elite for one week, just to get that presentation. So if you guys want to look into that, go to h10.me forward slash Elite and see it's only $99 extra, so make sure to sign up for that. Now, another thing that I think a lot of people have been talking about not just you, but you were one of the first ones to talk about semantic search and Cosmo and things like this, and we'll talk about what that means. But I think, just to set some groundwork, I think everybody understands that any search algorithm will evolve over time. That's the whole purpose. Like the companies who don't want to do well, they'll just keep their algorithm the same right. But anybody you know whether we're talking about Google, Facebook, TikTok has an amazing algorithm, Amazon. It changes over time and we've seen that.   Bradley Sutton: You know, if we were searching five years ago on Amazon, it's different. And now if you've bought some how many of you who have bought something you search for a keyword that has to do with that and that thing that you bought is now at the top on your Amazon maybe not somebody else. That didn't happen like five, six years ago. Last year we showed an example of how you search a keyword that doesn't really exist. It's called noodle camera and no listing has the word noodle camera in it. But there was like maybe 30 listings that came up and it was like a stethoscope camera it looks like a noodle. So five years ago you put noodle camera it would say zero results because nobody has that in their listing and these listings don't have that keyword in there. But it's showing up because Amazon shows history that, oh, people don't know what this is called stethoscope camera but then they think it looks like a noodle. So now it's showing listing. So we've seen this even for a year. Now, first of all, Amazon science documents we've talked about it, but maybe 80%, 90%, never actually is 100% in production. Sometimes it goes into production, sometimes it doesn't. But what was it that made Cosmo so interesting these documents that talked about it, that you're like man. This is something that you think that Amazon is going to move towards.   Leo: Yeah, the reason why, I think is something that would be applied at scale across the marketplace is because, as searchers, as buyers, as shoppers, our goal when we use a search engine is to find a product or information that we need in order to solve a problem. And so, as a technology company in this case we're talking about Amazon their goal is to improve, like Bradley said, the algorithm in order to simplify that search result and give you exactly what you're looking for, by burning some steps in the middle, right. And so that's what Cosmo is designed for. Cosmo is designed to be a man in the middle, between yourself and the search results, right, when you work together with it to give feedback back and forth. And so what they do right now they learn. You type a search query, they give you some result, you refine that result by clicking on some products that you think are relevant. And what they do with this information? They start building this knowledge graph, right. So a classic example if you go on Wikipedia and for something, Wikipedia normally links to other relevant sources. That's what they call the knowledge graph, right? They know that this is relevant to that right. And so what cosmo is trying to do, instead of you having to refine the search. They're refining it for you.   Leo: So the example that I give in my presentation this morning is that, if someone is searching for winter coat, we saw a product that ranks number one on Amazon that doesn't have the word winter coat in the title. But yeah, they're ranked number one, and so this is shocking, right, like everyone's like oh come. Title is supposed to be the most important element on the page when it comes to optimization and some SEOs, but this time Amazon understood that you are looking for something that keeps you, to keep you warm, right. So now we're shifting from a keyword-based search to intent-based search, and so, as sellers, right now, what we need to do is understand what is the actual intent behind the person. What am I selling this to? I'm selling this to someone that wants to stay warm, right, that's what the purpose of a winter coat is, and so, with that intent in mind, we need to optimize listings so that we can convey the message through images, through the title, bullet points and description, so that Amazon, the new Cosmo, understands that this product is something that helps people stay warm.   Leo: And what I think is going to happen also because of the shift in the way that these search results are built, which is more intent-based, is that Amazon then will start recommending also related products. So if you're looking for, if you type in winter coat, they say, okay, well, this person is trying to stay warm and so let me show them also some winter gloves and winter socks and maybe some winter boots, and that will change everything right. They will change the way we advertise, they will change the way we try to be associated with other products. They will change the way we also promote our listings. So that's very interesting and fascinating, but I think it's a good thing for the buyer, right, while for our sellers might be challenging to figure out again, how do we optimize our listings keeping this semantic concept in mind for the buyer? And they've already proven. If you look at the Amazon science document in the research papers, they're already saying that they're seeing a lift in conversion rate when Cosmo is applied to a search result page. So we must pay attention to these and monitor certain. It's challenging right now to understand where this is applied, but we need to monitor better the Amazon marketplace and then evolve and adapt as Cosmo gets released into more categories.   Bradley Sutton: Not to be controversial here, but to me it's almost it's different, but it's not different. Like, at the end of the day, Amazon wants to make money, right, so that winter coat that became number one. It's not number one necessarily because of new algorithm, because it would not be number one unless that is one of the best converting ones, because that's what gives Amazon the best chance to make money. But I think where the difference here is, or what's something that's quote unquote new, is it gives people more at bats. Like maybe I never. Even if I didn't have winter coat in my title, it might've been almost impossible for me to get on page one. But now Amazon is all right, let's just throw it here. Oh, shoot, look at that, how well it's converting. Let's go ahead and push it all the way to the top, whereas maybe you know, four years ago, you know, unless you were super optimized for a certain keyword, you would never even have the ad back. Like you would never even be able to get on page one, you know, outside of PPC or something. So to me that's like the difference, but something also. Again, I keep saying I don't want to be controversial, but it's going to be because there's a lot of people I respect in the industry who have been talking a lot about things that and I agree mostly with them. But I completely disagree when they say things like, oh, tools like maybe Helium 10, if they don't change it's going to be out of date. To me, I cannot see a world where the traditional forms of keyword research, are going to be not as important In the future, if Amazon is super intuitive, of course that's going to evolve.   Bradley Sutton: But the main reason we do keyword research is to get indexed and to also make our listing. Initially because the Amazon algorithm is based on buyer interaction, right. So once it's been out there for three, four weeks, they have so many data points and how people searched and what they clicked on and stuff that. Okay, now we can start doing advanced algorithms. But to even get it in the right pages you had to have done the regular keyword research to show Amazon. Because when you're brand new, day zero of your listing, Amazon has no idea what it is. It goes by the image, it goes by what you have in the title and how you have it. So my personal opinion is that no like. Of course, little things are going to change with keyword research here or there, but the main core of hey, let me find the most important keywords. That's not going to change because you have to tell Amazon on day one what is your product.   Leo: So, Bradley, I agree with you and I think there is one important detail that is the link between what you're saying and what this all semantic stuff is about. Right, the reason why that winter coat might be ranking number one, even though the winter coat is not in the title is attributes of the winter coat. You know Amazon right now, which before they probably weren't doing before Cosmo, right, they're looking at the attributes. So most likely they are ranking this one very well because it contains, uh, goose feathers, or they have 300 grams of goose feathers per square meter or whatever foot, and so they now are using these attributes to understand is this product warmer than this one? So, while the keyword research tools are always going to be needed, what I think is an opportunity for companies like Helium 10 is now provide additional information to the seller together with the main keywords. That helps also the listing be more relevant for Cosmo, using attributes related to those keywords. So, if the keyword is winter coat, what are the main attributes of coats? Right? What does a coat have to have? Waterproof, has to be warm. What kind of feeling? Is it polyester? Is it goose feathers? Also, is it long or short? Things like that are going to be the difference between the traditional keyword research tools and the semantic powered keyword research tools. If you guys give the sellers the same list of keywords and, by the way, here are some attributes related to these keywords, that will help Amazon Cosmo understand more about your product. I think that's the winner, in my opinion.   Bradley Sutton: Yeah, and in his presentation he talked a lot about different things you can do to be more semantically relevant and you know, using ChatGPT, so some really good features there. But that's important because you know, the it's not just, we're not just talking about Amazon SEO, it's also going to help you on Google and Bing and these, these other things and there's things that just the human mind we can't process, but a computer can process and tell you hey, this is, this is the keywords with the buyer intent and this is the most important, this is how you can relate yourself. So, regardless of how much of this Amazon develops, it's already important now for outside of Amazon indexing. Now, before we get into your last strategy, and I have just a couple of questions for you if people want to get more information, reach out to you, find out about your new project you're working on, or just reach out to you. How can they find you out there?   Leo: I have my own website right now. It's leosgovio.com, so you can reach out to me on through my website.   Bradley Sutton: And spell that, because it's not spelled exactly as you might think.   Leo: It's l-e-o and then s as in Sam, g as in George, o, v as in Victor, i o. Yeah, over there I have some information also about the semantic SEO stuff. So if you're more interested about this, I'd be happy to share my knowledge in depth, and LinkedIn is one of the platforms that I use the most.   Bradley Sutton: Excellent. All right Favorite Helium 10 tool?   Leo: Magnet   Bradley Sutton: If you were a head of product at Helium 10, what is one tool or function that you would bring that we do not have currently?   Leo: I believe I will combine what we just discussed about into one tool, and so it's an hybrid between a listing analyzer powered with recommendation based on the semantic stuff.   Bradley Sutton: And your 30 to 60 second tip can be about anything for sellers out there.   Leo: Leverage. Try to think about your current strategy when it comes to product inserts. To leverage it for UGC.   Bradley Sutton: All right guys. If you want more information, go to leosgovia.com. Check them out in the Helium 10 Elite, the Q2 workshop replay. But thank you, guys, so much for joining us and we'll definitely be reaching out to Leo next year to see what he's been up to.   Leo: Thanks, Bradley, I appreciate you having me again and, yeah, looking forward to the next one.   Bradley Sutton: Adios desde España.

In this week's episode, Riccardo Cosentino and guest co-host, Corail Bourrelier Fabiani, sit down with fellow alumnus Brandon de León to discuss his Oxford Saïd Business School dissertation on electric aviation.Brandon emphasizes the importance of public perception in adopting new technologies and explores the potential of electric aviation to transform urban mobility. The discussion covers technical advancements, regulatory challenges, and the necessary societal embrace for successful integration of electric aircraft into daily transportation. Brandon's insights highlight the intersection of technology, society, and the future of urban air mobility.“But what really validated my research was that social acceptance came up as an interesting issue already, before the first vehicle flies. And like I mentioned before, the next plans are also around big events, the World Expo in Osaka. Next year in 2025, this was to be flights. And in 2028 in Los Angeles for the Olympics there. Other companies from the US are also planning to fly. So yeah, social acceptance is already showing itself as a key risk.” – Brandon de León  Key Takeaways:Defining the pre-commercialization of electric aviationThe critical role of societal acceptance in the adoption of electric aviationThe potential impact of electric aviation on urban infrastructureInsights into the interplay of technological advancements and regulatory frameworksDistributed and decentralized mega projects If you enjoyed this episode, make sure and give us a five star rating and leave us a review on iTunes, Podcast Addict, Podchaser or Castbox. The conversation doesn't stop here—connect and converse with our LinkedIn community: Follow Brandon de León on LinkedInFollow Corail Bourrelier Fabiani on LinkedInFollow Riccardo Cosentino on LinkedInFollow Navigating Major Programmes on LinkedInRead Riccardo's latest at wwww.riccardocosentino.com Transcript:Riccardo Cosentino  0:05  You're listening to Navigating Major Programmes, a podcast that aims to elevate the conversations happening in the infrastructure industry and inspire you to have a more efficient approach within it. I'm your host, Riccardo Cosentino. I bring over 20 years of Major Programme Management experience. Most recently, I graduated from Oxford University Saïd Business School, which shook my belief when it comes to navigating major programmes. Now it's time to shake yours. Join me in each episode as I press the industry experts about the complexity of Major Programme Management, emerging digital trends and the critical leadership required to approach these multibillion-dollar projects. Let's see where the conversation takes us.  Riccardo Cosentino  0:54  Hello, everyone. Welcome to a new episode of Navigating Major Programmes. I'm here today with my co-host, Corail, and we have a special guest, a good old friend of ours joining us today on the podcast. I will pass the mic to Corail who's gonna co-host the podcast today and help me asking questions to this really great guest that has agreed to join us today. How are you doing, Corail? Corail  1:19  Hi, Riccardo, I'm doing really good. Thank you for having me co-hosting this great episode. And I'm excited to talk to Brandon. So maybe a little bit of background, we met doing a major programme management master in Oxford. And during that two years of our lives we met Brandon who was a superstar in our cohort because at the time, he was working for Tesla. And he was talking to us about this really exciting industry and how he's been part of the founders of Tesla. And he's been part of the team that made it a great company that we know today or the big adventure. And then he moved on to work for Rizwan and developing this into a great new enterprise that goes beyond Tesla by working on all different types of trucks and other things that I don't know enough about. But so I'm really excited to hear the story of Brandon. And I think Brandon, it would be great to start with you maybe introducing yourself telling us a little bit about how you ended up in the electric car industry and what drove you to that really expanding field. And yeah, to know a bit more about your background.   Brandon de León  2:37  Wow, what an introduction. Thank you guys so much for having me here on the podcast with you. It's been incredible to see what you guys have produced after the Master's course and hard to follow what you shared Corail. But I think, generally speaking, it's been an amazing learning experience. The Oxford Masters in Major Program Management, learned a lot from you guys, as well as the content. So happy to be here and share a little bit of what I've been working on and some of my background. It's been about 12 years since I've been in the electric car space, but maybe just a step back and give a little personal context about how I even ended up in that. I, my voice betrays me, I'm from the States originally, even though I'm coming to you from Holland today, I'm living in Rotterdam and working in Amsterdam. I grew up on the east coast of the U.S. mainly in Georgia in Florida. And then that's where I connected with Tesla. And what brought me down to Florida originally was my pursuit of International Business Studies. I wanted to study that because I had grown up hearing about how my parents met in Germany, in the army, and so early on, I was ingrained with these ideas of a world beyond the bubble that I lived in. So even though I grew up in small town, Georgia, I knew that I wanted to go and at least traveled to these places maybe even work in these places, if that were possible. And so in high school, when I discovered international business classes, I couldn't get enough. I also learned in those classes that there's a lot of ways society progresses, and probably the fastest mechanism to make that happen is commerce. So you know, regulation and government takes a long time. Nonprofits are phenomenal. But also some of them have limited impact. So it wasn't really clear what the best path was. But having parents that had done service for the country, as it were, and then pursued their careers and more local service type of roles and social worker and studying pre-med and things like that, I knew that I wanted to find a way to make the world a better place in my own little way. Right? So international business was my chosen vessel that brought me to Miami to study undergrad. And then in university there I had a chance to actually work in a company that I had admired for what I would consider a great engineering design and that was BMW their local retailer in the south Florida area. I persuaded them to let me take an entry level job that they didn't have at the time. But I was really eager to get out of the department store I was working at, and to go work in the company that I admired so much down the street and regularly saw the employees from that office. And so thanks to some persuasion and friendliness on their side, I was able to take a very administrative basic role. And I spent two years with BMW, but I realized that 18, 19, 20, there wasn't a lot of career options for someone that young, in the automotive retail space, at least not the traditional automotive, even with really innovative products like BMW. So I went on the search for career path, and I ended up in a bank and I thought, wow, this is terrific, financial district, maybe I peaked early, there's a career advancement ladder, it's all planned out, pension, etc, all the trappings of a great career, but then my interest in engineering and technology kind of clashed at a certain point, because new payment technologies came out into the market. And some people will be familiar with these dongles that you'd plug into an iPhone, which is relatively new back then. And you could swipe credit cards, and it was a revolution for small merchants and mom and pop shops, and I thought, this is wonderful, it's gonna be great for getting them better cash flow, they'll grow faster, it'll really helped a lot of small businesses out. But banking is a very conservative culture and does not run to embrace new technology. So at the time, we were working on a laptop that had an operating system that was three generations old, because the security patches were all well-developed and stuff, there was a clash there. And I figured, okay, I need to find a career path that's also aligned with my interest in technology or automotive or something along those lines. And after a lot of soul searching, and job board searching, I came across Tesla, and they were starting a store locally in South Florida. And through a lot of discussions for different roles. I ended up joining the launch team for the Model S, which was the car that really established the brand as a large volume producer of vehicles. And of course, its focus was electric cars. So that was 2012. I joined right after the vehicle launched. And it's been a wild ride since then. But at that time, because of my international orientation, I thought the wildest dream I have right now is that Tesla will do great. And then we'll get the chance to go and launch in other countries. And maybe I can be part of that. Happily, I had that chance. So although I started in California, and then after we launched the vehicle there, and I gained a lot of experience integrating, delivering the vehicles, the first vehicles and integrating it directly into the lives of people and families across California, I had the chance to bring that back to the East Coast. And then there was an opportunity to join a different department back at headquarters in California. And I jumped at that, because I knew if international opportunities emerge, there'll be people from there, they're good to go. And so that was a strategic step, it was not clear that it would work out. But happily, I was in the right place at the right time, there was an assignment to go help the team in Canada kind of get find their feet, if you will. And then after doing that a couple of times going out to Toronto, and doing what I would call international, but doesn't necessarily appear to be very international experiences going from California to Canada, I also had the chance to join a very small team of four or five that came out to Europe for a few months to do the same, essentially to help train the first model as launch teams here. And that was quite a privilege. When I came home to San Francisco, I thought oh, wow, this is it. Everything I could dream has happened. This is fantastic. What do I do now? And I didn't have to wait long for an answer because being at the headquarters in the beehive of activity, there was an opportunity that was presented to me to come over and help build critical partnership networks because we were starting to deliver vehicles but didn't really have solutions in place for if people were driving from the Netherlands or Norway down to Spain for holiday, what happens if they run out of charge or if the car breaks, and we had a very small footprint in Europe, within three weeks, I was on a one-way flight back to Europe. And I haven't looked back since. So that's a little bit about me in a very long-winded way and how I've gotten here to you. Corail  8:58  No, that was fantastic. Brandon, thank you so much for sharing your passion. It's really fascinating how you're constantly growing, reinventing yourself, but yet you seem to have such a drive. And that brings me to something a little picture that you shared of yourself recently that was you, yourself. I don't know how old you were maybe five or six. And you were in this beautiful little plane. And I think we talked about how you progressed from different career paths, but always with a certain drive and in that journey now I feel like you also have great interest into the future of electric aviation. And you decided to write your dissertation on this specific topic, and the social acceptance of electric aviation. Can you tell us a little bit more about this interest of yours and how you came to write about this in your dissertation in Oxford? Brandon de León  10:00  Yeah, sure. Thanks for reminding me of that picture. It's my haircut was terrible. But yeah, I was very young. And that was, it was a fun picture of me as a very small child in this mock-up of what must have been like a pretend F-16 for children at an Air Force stand at some air show in the U.S. So that was quite a throwback. Thanks for that. The, that picture I think really reflects my interest from the earliest of memories. And I call it transport now because it seems more appropriate. But it's really cars, planes, things that move fast. They're exciting, or have always been exciting to me. And I know that's sometimes cliche and certainly not exclusive to me. But that's where my fascinations were as a kid and that really hasn't died. But my career being mainly in automotive and electric automotive for more recent decade or so, maybe it's worth sharing, it's quick middle steps. So after about 10 years at Tesla, scaling the core product and ecosystems around it in North America and Europe, I thought, okay, what do I, the recipe is pretty much set at Tesla. So we have gigafactories opening, launched four or five different vehicle programs, how can I best use all this wonderful experience? And in 2021, I joined a company called Rivian, which essentially, is, for those who don't know, it a lot like Tesla in that it's a new company that makes cars. But their plan was to electrify totally new vehicle types, still ground vehicles, right? So trucks, which are hugely popular in North America, also SUVs, which are growing in popularity globally. Perhaps, if you look at the Tesla Model Y, the best selling car globally, right now in 2023, I think it was. And then for me sitting in Europe, perhaps most importantly, commercial vans, so they have or we have a huge order with Amazon for 100,000 delivery vans. And that was super exciting to me, because being in Europe, I know that trucks are not a big deal here. SUVs are typically on the smaller side or middle size, definitely not the large American scale. But I knew that if they produced the vans, then we would have a tremendous success on our hands. And that's gone really well. We've delivered over 16,000 vans now it's super exciting to see that happening. So essentially, why join Rivian was to extend electrification. So when looking for a dissertation topic, during our master's degree, I really wanted to take that opportunity to explore the other side of my fascination. One, because there wouldn't be any conflict of interest. So it was a lot cleaner to not do electric vehicles. And then the other side is there was a really interesting ecosystem emerging that was ripe for research. And that's electric aviation. It obviously aligns with my fascinations, but also super timely.   Brandon de León  12:32  So when I started looking into how can I use a dissertation to add some value, however minuscule to what's going on in this ecosystem that fascinated me so much, I started to reach out to people and one of the people I reached out to was someone I would consider a founding father, a modern time founding father in electric aviation. And he had spent three or four decades at NASA researching electric propulsion. And it really caught on towards the 2010s. And we'll get into that later. But essentially, I was asking people like him who are highly technical, unlike me, who's a non engineer, how can a non-engineer contribute to the conversation into the development of this space, and in our discussions that came out that acceptance is really interesting, because it is a known concern. But it's kind of a fuzzy topic, a fluffy topic, it's ambiguous, people aren't really quite sure what to make of it, how to define it, how to grapple with it. And there's not a unified message around it. That's, that seems ripe for Social Sciences dissertation. And that's what led me into it because there weren't any other spaces that weren't mostly other parts, or aspects of the ecosystem today, are highly technical, or regulation-oriented. And this was a space where someone coming from social sciences point of view could really add value. So that's what led me into it, happy to document it more. But that's the background and how I got there. Corail  13:51  I think it's so interesting that you're bringing, as you say, a non-engineer background into a field that is highly engineering-heavy. And we see in Oxford, we talk a lot about the work of Kahneman, for example, and how it mixes psychology and economy and what amazing ideas that created and I feel you coming from a different background is also generating discussion that we don't think the regular engineer doesn't necessarily think about. And I think it's quite beautiful. You talked about the social interest of your parents early on that kind of inspired you. And it's interesting that you went into social acceptance and which encompasses I think many things but also the how people receive what we're producing. Right? And I wanted to ask you a bit more about this because when I think about social acceptance of electric vehicles or electric planes, as of, I don't know, kind of French bias, (inaudible) we talk about how planes are terrible for the environment and we are always thinking about shaming each other in France for how much we travel? I know my aunt for example, is constantly telling me you shouldn't take the plane so often, etc. And so for me, I only see positive outcome, right, for electrifying planes. So why did you, how did you identify social acceptance as a risk? How is it perceived in the industry? Brandon de León  15:19  Yes, it's a great question. And actually, thanks for the chance to add more background because it's not, it wasn't something I was able to include in the dissertation itself, I had to really shrink down that context and generally referred to the study as a study into the acceptance of electric aviation without giving a lot of detail and color. So essentially, in order to understand that better, it's helpful to describe the 2010s and the emergence of the ideas around electric aviation and how it was going to look and feel what the vision was, and who was articulating it. So although there have been decades of research at NASA, in particular from the guy, Mark Moore, is the gentleman I talked to and brainstormed with around ideas, potentially, that could be useful to the ecosystem. So there was quite a lot of work done on the physics and the engineering aspect. But what was interesting is that it didn't come from a lot of technology seemed to come from the defense side, right, where you have the internet, GPS, other things that are developed for military or defense purposes, and then they become commercialized. This is a rare instance where, even though NASA had done prior work, and really help manifest the technology, or the idea around how to use it, it was actually technology, commercial minds, technology and commercial minds that were leading the development of this vision, a particular vision of electric aviation, and they called it urban air mobility, mostly. There were many different names and the names of all the increase since then, in the early 2010s, essentially, you had Google printing tons of money and so just to pick, cherry-pick a specific example, this is not the origin story for the whole ecosystem, but it's a major part of the background. So Google is just minting money, right? And Larry Page starts to make bets. And they're called Alphabet. Now, there's a play on words there. But essentially, Silicon Valley companies that make it that big start to then have to find new avenues to create growth. And these are the bets that they're making. And one of them was autonomous vehicles, right? And today, that's Waymo. And another one that was backed by Larry Page in particular was a company called Kitty Hawk. And it had different names, as in its predecessor phases, but essentially, they were making a two/four passenger air vehicle, and it was all electric. And it looked like nothing you've ever seen before. If I had to describe the inspiration, I think that in many cases, these air vehicles developed by the organization he was backing, or Google was backing. I guess it's more him than Google to be honest, on the on the electric aviation side. And other pioneers of electric aviation in the same timeframe, they kind of looked like scaled up drones, toys, essentially, they're called multicopters in that format. But essentially, the vision was that these were going to be flying taxis. And they were going to be in cities. Now I'm not old enough to remember this in person. But I've read stories about how Delta and United used to have these phenomenal helicopter services where you could catch a helicopter from the top of the Pan Am building in downtown Manhattan, or Midtown and then fly over to JFK, or whatever airport. And that was the heyday of aviation, right when it was really a VIP experience. And this wasn't just New York City, this was San Francisco, tons of other cities have this helicopter service. And it's not really the case anymore outside of a couple non-airline, independent helicopter services in, let's say, New York City for example. And enter Uber, another emerging tech company, who was really ambitious and wanting to really reinvent mobility, not just on the ground, but they also saw an opportunity to play a role in this airspace as well, if you will. And so they took what they knew about ride-hailing and the app and the data that they had seen, all the trips people were taking around urban areas like L.A. and New York and probably better than anyone they fully understood and had the data and the data orientation that a Silicon Valley company would do to understand how there's a huge amount of traffic between this origin and destination. And so airport, if we look back at this helicopter services presents an interesting option. And so they started to, they started a sub-organization or department called Uber Elevate, and they issued a white paper, I think it was 2016, maybe 2017. But the white paper basically articulated a really grand vision for all these air vehicles doing thousands of movements in urban areas a year. So it's a whole new kind of flying, not the wing and tube that we're used to going between over long distances or medium range distances.   Brandon de León  19:49  This was a whole different layer of air transport that hadn't been seen before, because presumably, existing small airplanes were, with the capacity of a ground taxi, four, five, six seats or whatever, were too noisy, not comfortable and outdated designs and they couldn't vertically take off. And that's a big difference too is that these new vehicles were supposed to take off and land like a helicopter. So that then unlocked a lot of opportunities to land in urban areas without a massive airfield and runway. And so that was the lower end division in that Uber Elevate white paper. These days, that evolution of that vision has evolved quite a lot and become a lot more mild. To give you one example, there were images circulating around the time of that white paper, where you would imagine a high-rise tower and different levels that would have open bays that the small car-sized air vehicle could fly into horizontally and land or land at a top and then the elevator would move it around. But essentially, it was beehive for these. And that's where social acceptance really became a question. Because if you have that many vehicles flying around in the airspace that's not really used today and they're potentially making a lot of noise because helicopters are super noisy. And that's the best benchmark that we have, even if they're electric and quieter, they're not going to be in silent, then how are people going to react to the noise? How are people going to react to the visual pollution or obstruction to whatever view they have, if you enjoy the city view of Manhattan, it's now going to have a lot of air vehicles in it. If you enjoy the Coastal View, perhaps you'll see a lot of vehicles above the beach, that sort of thing. Social acceptance was early on identified as a risk, something that needed to be dealt with. But how to deal with that wasn't really clear. Riccardo Cosentino  21:29  Brandon, I have a quick follow-up on that. Because it's very interesting how this was a dissertation. So was the final project for the master's degree or for a master in major program management. Can you articulate how you end up picking an industry as a major programme? I'm assuming, I'm paraphrasing a little bit because your study is not about one particular project, one particular company, it's really just societal, and how society is going to who's going to embrace this new technology or not. And so when you were discussing with your supervisor about this topic, how was it received from the academic side because we're all educated, and we're all told my major project is a project about 1 billion dollar/pounds, whatever. But I'm not a believer in that metric. To me, it's, major programmes are about complexity and I think your dissertation fits perfectly that definition, but you must have had some back and forth with your supervisor, or even with some other academics. Brandon de León  22:38  Yeah, it's a great question. And I really thought this was a risk to my dissertation to be fair open to the point of marking, I didn't know if it was going to be received well, that how to articulate this as a mega project or giga project, as I called it. But basically, I think that if we look at the way we presented this content in the course, just to give the listeners an idea, that for most of history, or let's say the last century, there has been increasing focus on these growing, the projects of growing scale and complexity, and cost getting into the billions getting into this, they totally changed traffic patterns in the city or they, if there are huge new bridge or something like that and it's just visually imposing huge civil infrastructure or digital systems that cost a ton or aerospace programs like an A380 Airbus, which is just a mega behemoth of an airplane, right? And if the complexity is clear, super tangible. But I think that's the school of thought that are when we had the great fortune of I think straddling two eras of the faculty at Oxford. And the first chair that we encountered was Bent Flyvbjerg. And he literally wrote the book on this stuff. And so far as the Oxford Handbook for Major or Mega Project Management, and in that, through that lens, or what he helped us understand, it was this more centralized type of project. And then later, we actually had another generation of leadership for the faculty come in with the new chair, Daniel Armanios, and he was very interesting in that he introduced the concept of it not necessarily having to be a centralized, that's a singular entity, the mega project could be distributed, decentralized, even. Right? And so after reading both of their research, I'd actually found that Flyvbjerg and contemporary said, coined the phrase of an array of projects. And I thought that fits this. This is actually exactly what I need to articulate how this is a large, complex project, although it's effectively being built in a decentralized manner and actually, quite extremely decentralized manner. There are over 800 different organizations that have released a concept for an electric aviation or electric air vehicle of some kind or another. There's this nonprofit that tracks the industry and most of the funding is with a handful and most ofthe technical progress is at a handful, but the reality is that there is a massive number of companies that intend to enter this space. And essentially, by building these vehicles, they're having to also engage regulators and build the regulatory envelope for this to actually happen. And then also go out and entertain cities and get them on side. They even let it fly. So ultimately, what they're all building towards is a central vision, even though it's moderated a bit since the over white paper in the beehive towers in the city, what they're actually when you step back and look at it all, what's actually being constructed, is something quite central. And that is a layer of air transportation, a new air transportation system that doesn't exist today. Because electrics, there's no charging out there. It's also and this is the part I didn't really get to yet is that a lot of the companies want to get towards autonomy. Some people might know already, there's a pilot shortage historically, pilots are now being paid very well, after having years and years of declining. That's not the case anymore. There's a vast shortage of pilots today. But also, if you're looking at technically looking at these vehicles through a technical lens, from a physics point of view, the energy density in lithium ion batteries or automotive grade, especially. But even research batteries, they're still limited compared to typical combustion fuels, hydrocarbon fuels, in so much in how much energy they can carry per kilogram. And if you're in aerospace where every gram matters, it's critical that you lighten the vehicle, because it's a trade-off for payload and revenue. And so although electric vehicles have started to scale up the production of lithium-ion batteries and automotive grade electric batteries have really gotten cheaper and better energy density so they're improving every day, in labs across the world they're still just crossing the threshold where they're useful in the air and just unlocking short-range missions. So this is a new, this is a new transport layer that is just becoming feasible in the late 2010s  and still in development. So that's where I basically come back to your point, which is it's not a central program, it's definitely super distributed and decentralized, but they're all building in essentially a common vision of electric air transport that doesn't exist today. Riccardo Cosentino  27:21  Okay, one more question that on that note, and then I'll pass it back to Corail. As an industry, I mean, where would you position it in the developmental phases of an industry? And maybe, if you could make a comparison, we always go back to the internet, right? So every time there's a new revolutionary technology, we always say, yeah, think of the internet in 1995 or finger the internet in the 2000. On that basis, so with that in mind, where would you place this industry in the developmental arch? Brandon de León  27:59  Yeah, maybe if I could go a few years before the internet just for a comparison that rings harder in my mind is mobile phones. I think we're at the place, there's a famous study from McKinsey that I'll get the number wrong slightly. But I think that they hint here McKinsey did a study for AT&T, I think it was where they predicted in the early 90s roundabout then that the maximum total addressable market for mobile phones is 900,000 Americans. I think we're at that stage with electric aviation. And I don't mean that in the, to poke fun at our friends at McKinsey, I know we all have some, anyway, consulting generally. But I think that it is impossible to anticipate the actual scale that this will be deployed at over time. And I say that because if you look at this technology, the business model for many companies is not clear yet. So I think that's, once the technology is ready, we're at the point where the technology is only just becoming certified. And even with helping hands from governments that are eager to be technology leaders in Q4, right about Q4 last year, the first electric vertical takeoff and landing vehicle certified anywhere in the world was certified in China. And just this month, the first one was transacted to a Japanese customer from another company in China for demonstration flights at the upcoming expo, World Expo in Osaka. Nothing's actually, there's no revenue yet. Unless you talk about small revenue regimes from Defense Departments and things like that to help with the testing and helping R&D funds. So we're really pre-commercialization. And that's precisely why I wanted to jump in for the dissertation into this space. And I thought it was really rich picking for that study-wise because what we hear in the program and all the things we've learned about mega project management and so on, is that when did they go wrong, if not in execution mostly in the planning phases in the earliest phases. So this was a huge opportunity to talk to people across the G7 really across OEMs, regulators, infrastructure companies and so on. Even NGOs. And to get a sense for okay, where's everyone's head at individually and collectively. What's the sense for how they're all thinking about this particular aspect? Social acceptance of a new technology? Yeah. And so that's, I think lends itself to the study, but super early is the answer to the question in a couple of words. Riccardo Cosentino  30:19  Thank you.   Corail  30:20  Brandon, that's really interesting. And to go back to your dissertation, I think you were planning to interview 10 leaders and you ended up with 29 interviews. I think it shows the real interest that it sparked amongst the leader in that industry. And do you feel like they got interested because this is a topic that they didn't necessarily so much sought about? And they wanted to discuss more with you? Or was it very much a risk that was very present in their minds? And you just found that they had already thought about a lot of solutions to raising social acceptance? Brandon de León  30:59  Yeah, it's a fair question. You're absolutely right. My ambition was 10. I accidentally overshot that by three times. I paid for that on the back end, when it came to actually giving the proper level of attention to the data analysis and cleaning the data from the interviews. Yeah, that was, it was quite a heavy fall. But it was really a pleasure to, because once I started to talk to people in the space, Dr. Mark Moore and I had engaged over LinkedIn, of course, and email and then had a call. But I think that really, I realized early on, and one of the things that I picked up on from one of the faculty members, Dr. Harvey Mahler, was that observation can actually be a part of your research. And I thought, let me, let me go to at least one event where these people gather and just see what's the level of discourse? Is this really a risk? Or is it just something that I see in their social media content or things like that, and I was really lucky, because there were three major industry events, if I can call them that, that were happening right around the time I was doing my dissertation and or the early stages of it. And so I went to London to eVTOL Insights London Conference and it was very much inside baseball, you had the top leaders from the companies that were trying to develop and certify these air vehicles. But you also had the leading regulators, globally for aerospace were there, it was really interesting to just, fascinating to hear the conversation between them. But what I picked up on was that acceptance came up, it was, in some cases a footnote. In other cases, it was a panel topic. But it was never, there was only such a limited depth that could be accomplished in that format. And having chatted with Dr. Mark Moore, having seen that in person in London, but also at Revolution Aero, which is another major event in Dublin, I realized that there, there's not a lot of exploration of this topic, if this is essentially the limit of it. And there are other podcasts in the industry too, that I've listened to where it maybe it gets explored a little bit more, but usually, it's pretty, pretty limited how much people can talk about this, because the overwhelming focus right now is to use every dollar of investment. And right now there's over $15 billion, I think it's over 18 at last count, invested in this space, mainly in the vehicle developers, that will in the future produce these electric air vehicles, essentially, the ones that are just planning to actually produce the vehicles, a lot of them aren't necessarily interested in acceptance, that's something they consider a responsibility of the operator to go out and develop acceptance wherever they plan to operate the vehicles. The operator meaning like airline, essentially. And then the other case, some vehicle developers or pureplay operators, they see the acceptance risk a lot more clearly. And in some cases, they've experienced it before with their traditional air vehicles.   Brandon de León  33:42  So I think, for me, it became clearer and clearer that this was both interesting for me, and potentially helpful for them to have a longer form conversation, the average interview was something like 45 minutes to an hour, someone as long as two hours a couple of them, when as long as two hours, I made the coding quite a long process. But it was super insightful for me. And I felt really privileged. As I was reaching out to people, the reception I was getting was quite strong. I thought 10 was going to be the high end and also a significant enough sample that would make the research worthwhile and meaningful. But then actually, I started to realize that if there's greater interest, I'm happy to expand that to a larger number, especially if it allows me to get perspectives from multiple people representing the different sides of the ecosystem. So like I mentioned, regulators, not just in Europe, but also in North America. And also OEMs, not just in Europe, but from North America as well. So a lot of the funding sits in North America right now. And depending on who you ask the technological leaders, some of them are in Germany, some of them are also in California and Silicon Valley, and so on. So I didn't want to represent just one small pocket of the ecosystem because again, it's a larger array globally. If I could do a better job of capturing those points of view from a European point of view as well as an American point of view, I wanted to do that. And so that ended up getting me to nearly 30 interviews pretty quickly. That's how it grew so fast. Corail  33:44  I think it's fantastic. And there must have been so much work to just code this amount of interview, I just cannot imagine in the limited amount of time we have to do this dissertation. It's a lot. So congratulations. Brandon de León  35:26  Thank you. Corail  35:27  So can you share with us then how so I wanted to ask you, Brandon, how did this leader define the risk? And what were the solution that they were putting forward?   Brandon de León  35:39  It's a great question. I think maybe the step back as a precursor, or the best example of what they were trying to do before was helicopter services from decades ago. And if you live in New York City, or Sao Paulo, or Hong Kong, helicopter services are not an infrequent site. So there are places in the world where it's still quite common. It's just that in the U.S. we, being American, that's sort of my bias, those services had dwindled. After there was a famous incident in New York City at the top of the Pan Am building, I think it was bad weather that affected the helicopter landing. Long story short, one helicopter did a particularly bad job landing, and crashed onto the rooftop. And when it did, a propeller went this way. Another one fell to the ground, I believe it was or some debris fell to the ground and killed the young lady. The other one might have injured someone when it flew into a nearby building. This was, I didn't read the entire history of this industry, the helicopter service industry. But what I can tell you is that if you look at the old timetables and the brochures, being a historical geek and an aviation geek have done more than my fair share that there are very clearly helicopter services advertised in most, in a lot of major metropolitan areas from these mainstream airline names we all know and love today, or despise today, depending on what you think of it. But anyways, the reality is that those services dwindling, I think, in part happened, because there were restrictions put in place, when you had an incident like that it captured the attention of the public around, probably not just that city, probably not just the U.S., perhaps major cities around the world, especially as news could spread wherever the American newspapers are read. So I think that that put a little bit of ice on helicopter services. And so today, if you look at Blade, which is an operator that works does fly from Manhattan over to JFK, for example, to do the airport shuttle type use case, I believe they take off on the perimeter of Manhattan, they're not, they're just off on those waterfront, they're not on top of some building in the middle of the city. So things have definitely changed. And so when it came to acceptance and how they view it, one was, there was this precedent for things going wrong. And if things go wrong, it can really pause an industry. So making sure perceptions are warmed up to the idea of this happening again, because what they're talking about doing is literally lending in many different places across downtown Manhattan, for example of the island of Manhattan, actually being able to pop down on different buildings, but also perhaps green spaces or whatever, wherever they can place what they call a vertiport, which looks a lot to normal person, like a heliport, small helipad with a V instead of an H. There are other things there. The industry insiders will tell you, there's a lot more to it. And there is charging equipment and storage and things like that. But all that to say what the vision was in 2016-17, when Uber was hosting these huge industry segments with 72 experts one year and hundreds the next year to try and really build steam around this vision of urban air mobility. They knew they had an uphill battle. And then on top of that helicopters are famously extremely noisy. That's part of why they have limited routes that they can fly. The other part, of course, is safety and things like that there. If you look at London, I think there's one main helicopter route through the whole of London that goes, basically follows the river, for the most part. And then I think the only operational heliport inside core London, that's not a hospital for an air ambulance is essentially on the reverse side, too. So helicopters are really limited in where they could go, partly because of the noise, but other you know, fears, safety and things like that. And so that's essentially, what captivated the interests of the industry participants most was how do we reduce noise through technical innovation, better propeller design, electric motors are inherently quiet, they're not jet engines. Even if you hear things build as for marketing purposes, and electric jet, it's quite different. It's more of a fan. So I think that they saw an opportunity with electric propulsion to be much quieter, and also more safer, ironically, because you can put many more electric motors and propellers. So if one goes down, you're not worried about that you can still safely land the vehicle and then yes, I think basically centered around noise primarily because the industry insiders knew that, fundamentally, the vehicle was safer, more resilient, more robust, more redundant, if you will, with different electric motors and propellers, a higher number, some have six, some have eight, some have 12 propellers built into the vehicle design. So if one fails, it's really not a major issue for most of the format's of these electric air vehicles. But getting people to warm up to the idea of it was a real risk the way they see it. Riccardo Cosentino  40:29  So Brandon, obviously, this is a podcast about your dissertation. And you wouldn't, you wouldn't have a dissertation without a conclusion and some findings. What were your key findings? Brandon de León  40:40  It's a good, I think that so if I, my research question largely centered around how do these executives from all over the ecosystem, all sides of it, essentially define social acceptance? Who and what do they think drives it? And then also, effectively how they plan to approach it? Right? So how do they think that they can maximize social acceptance and minimize social rejection? And the primary finding I found in the case of the first question was, there is no single definition for acceptance. People describe it differently. You'll hear things like regulatory acceptance, social acceptance, of course, public acceptance, community acceptance, market acceptance. So it depends on the mentality of the person and what they're responsible for, and what they're interested in. So if you're looking holistically, you could argue that it's social acceptance, but some of them, a lot of them necessarily focus in on the stakeholders that are closest to the activities that are proposed. And first and foremost, these vehicles have to be certified in a very rigorous process, the organization's have to, as well. So regulators are front of mind. And then market acceptance, of course, they think there has to be some demand, whatever their chosen business model, whether it's airport shuttles or other things. So they look at it through those different lenses. But when you're at a conference, those are sometimes thrown around as synonyms. And people innately understand the acceptance, that means other people being okay with this, but who they're concerned with. And the degree of embrace is something that I found varies quite broadly. And I think what's interesting is, even with that said, it's kind of there's a structure, there's always a question of who are they talking about we're concerned with, and then what's the degree of embrace, and that was a common thread, and their different phraseology, if I can call it that. And then so far as who and what drives it, if you look at a template stakeholder map, this is a lot of the literature around stakeholder management is written by Dr. Friedman. And Dr. Friedman has multiple books on the topic, he's the most cited guy in the field. And I tried to stick to these bedrock, most cited folks in these different disciplines because I felt like you said, this is quite an ambiguous space I'm diving into, I need to anchor myself to really key literature here. And so there's a beautiful map of stakeholders, and he breaks them into primary and secondary. So we call primary stakeholders, essentially, everyone who's in the value chain, plus government and community. So the people, we're directly interfacing with whatever you're doing, plus the people helping you produce it, and finance essentially. So most of the industry is focused on primary stakeholders. And I think operators are a little bit more aware of the secondary stakeholders, but through the interviews, the 29 different executives, we touched on every one of them got covered at least once. So although there was an overwhelming focus on primary stakeholders, naturally, there was an awareness of an interest in getting all stakeholders on board. And so they defined it very differently. But when you ask them who they needed to actually get to accept, it was pretty comprehensive. So no surprise, these people were executives in this industry, or in adjacent industries that made them relevant for joining these jobs for decades, right? They have, I think, on average, almost two decades of experience, many have masters and doctorates. They've done this before, or at least led businesses before and are aware of the spectrum of stakeholders that they need to talk to. What I had hoped to get into, and maybe this is because I was just coming out of academia with that hat on was the nuts and bolts of human thinking and decision making around taking this vehicle or this airport shuttle or not. We didn't quite get into that. I think that what I quickly understood was that the level of discourse didn't go that deep yet. And so I was asking you about which bias do you think plays a role in the decision to take this air shuttle or not to the airport? And after a couple of interviews, I realized, okay, let me bring it back up a level and further define, really who's involved and who's being mentioned the most, who's most important or seen as most important? And that's about as far as I could go in that space. There was a fourth question, I omitted it earlier, but essentially it was to what degree is acceptance a risk and simple to say most of them surprisingly, there was a lot of actual alignment here, social acceptance was considered a risk but also a high risk, I think partly because of the helicopter service example, in Manhattan. And also just generally helicopter services being so restricted over decades that they, everyone in this space has seen, made it really clear that they need to do a lot of work on the side. But what was interesting to me is a few of them went further and said it was existential to the industry. And again, thinking back to that Manhattan rooftop, you can imagine why they might think that because if public opinion turns against the industry, there's no writers, there's no financing. And then it's not a great day for the participants in the industry. So that one was pretty clear. And then the other one was sort of how to maximize social acceptance. And that was really fascinating for me to hear. Because again, I was talking to people on all sides, there were some people who were in marketing, communications, leadership roles. There were other people that were in, essentially engineering leadership roles. What was fascinating is that, essentially, they all largely saw the demonstration flights as a major win that were happening. There were limited demonstration flights happening by a couple of companies that were making sure that they were being seen as leaders in this space, and then also taking advantage of being first mover at certain major events. So for example, last was it, I forget the month, I think it was June, I was able to go to the Paris Air Show. And there was a company from Germany called Volocopter, who was led by a former Airbus executive. And they were flying their two seater prototype called the VoloCity. And this is the one that's supposed to appear during the Olympic Games this year and do some flights over Paris as well. On this day, it was flying over the airfield Le Bourget in north of Paris, which is in aviation history, it's a fascinating place tons of history, museums there, Charles Lindbergh landed there when he did this transatlantic flight. Anyways, long story short, to see this electric multicopter. Aircraft take off and fly over the airfield was really cool for me, because of my research. But also, it was stunning, because even though I have worked for decade-plus in electric vehicles, and I know just how quiet electric transport can be, I was shocked that I couldn't hear it, it was inaudible, from a very short distance away. Doesn't make any sense in the mind. It doesn't compute, it should be audible. It's not once it's maybe a football field away, in my personal sense. And so I think that what, what they were getting on to is what I experienced at Tesla, which is the technology, if it's really good, is convincing on its own, all you have to do is show people allow them to drive an electric car allow them to go to an air show and see this thing flying. And understand that it's, it feels silent from most places. And I think they're definitely onto something with that. Others went further to say we need to do education campaigns, I think that generally the spirit is roughly the same. But when asked to diagnose the state of acceptance building, most of them agreed that not much has been done or not enough has been done there. Some companies have gone on like 60 minutes and other major news shows for a segment to talk about flying cars, or flying taxis and these sorts of things. Because it's interesting and cool that there's some new innovative air vehicle. But other than that, and social media content, which really only gets to their followers, few had gone out of those. And more is happening now happy to talk about that in a minute. But essentially, that those were the four areas that I asked about, and was able to get concrete answers and learn what their perspectives were. Corail  48:32  That's great. Brandon, I wanted to ask you, like you said that a lot of them flagged this risk as a critical risk. And yet one of your one of your notes in your dissertation is that yeah, there is very little that is done about it. And as you're saying it's starting to increase, and we have the Olympic Games coming in Paris, and potentially, I'd love you to talk more about this and what will happen during the games. But first, why do you think so little is done if it's seen as this important risk that needs to be managed early on?   Brandon de León  49:07  Yeah, it's a fair one. And also, it was the thing that perplex me coming out of the dissertation. Obviously, in the month since doing the dissertation. I've had more time to digest it and think about it. And to factor in more of their point of view, I think. But essentially, and also last week I was able to join one last conference in my roadshow, if you will, to see what had changed since I had done the conferences about a year ago. And I think the short answer, if I were to speak for them, what they would say is that acceptance is important, valuable, meaningful and critical when we get to commercialization. But right now, the reality is that most companies don't have enough money to make it to commercialization. They're staring down their coffers and they don't see enough financial runway and funding left to potentially even get through certification. Some of them have just enough to get there. But it's very clear that almost all them if not all of them are going to have to go back and raise more funds. So when the funds are that precious, they're looking at how do we maximize every dollar, or euro or pound, right? And in those cases, essentially the critical milestone they need to get to to show that they have a viable product and business insofar as at least producing these vehicles, if not, to operate themselves to sell to someone else to operate, is to get certification, or to show meaningful certification path progress, and it's no small task. So just to give you a taste, they have to prove that they can, they're certified design organization, that they have a production method that can make exact copies over and over again, and this has to be signed off by the regulator, this is not something they can self certify, like in much of the automotive space, which is also highly regulated, it's still a fraction of the regulation level of aviation. And then even once they get the design, organization approval, and the production organization approval, and I might be getting my words a little bit wrong here. So aviation experts don't scare me. But essentially, they also have to be able to get an approval that they have processes in place that are certified for maintenance, repair, and overhaul, just to name a few. There's other things that they actually have to get certified for. But essentially, getting those things, those ducks in a row is billions of dollars. And again, if the whole industry has, let's call it 18 billion and counting, and there's over 800 players, you can imagine most of them aren't going to make that. And even the ones that have raised money, they've burned billions in many cases already. There are major, let's say, some of the companies that have raised the most funds in Europe, for example, have about 12 months of runway, but they still have more than 12 months to get to certification potentially. So I think that they're resource-constrained and focused on the core next milestone but, to your point, I think it's also because it's a fuzzy topic. It's not really clear who's responsible for it and then who should be spending money on it, and if one company alone can do it. And there's other interesting things that I uncovered into the research in the financial filings of some of the companies that have gone public through IPOs, or specs in recent years. Some of them consider developing public acceptance as a potential risk to their first mover advantage, because it'll benefit the whole industry and their competitors too, in that subset, right or in that collective. So while they see it as a good thing to do, from a social point of view. And maybe even from a business point of view, they can appreciate that it would be helpful to reduce some friction in the future. I think they're betting that it's overcomable. And they're biasing towards maintaining a first mover advantage if they can do. Our research from literature and social sciences would argue that maybe that's not the best balance, happy to talk about that more. But essentially, they're taking a pretty big bet there that they're going to launch. And then be able to build awareness, convert people to believers, and interested customers, at least as fast as they can produce vehicles and put them into servers and build capacity. So I think that's where it's a bit of a risk is that if they don't start to build awareness, early, the lag, there's a time lag between building awareness and first awareness and actually being willing to use a service. Not everyone's an innovator, early adopter. And I think they're counting on the fact that they're going to have a slow ramp. So they're not going to be over capacity. They're going to have more than enough innovators and early adopters that are willing to take their services, or use these vehicles. And they rather maintain the first mover advantage, largely not everyone, but most people seem to be acting in that way. Corail  53:40  Okay, I guess I have one final question. I'm really intrigued about what you're thinking about the opportunity that the Olympic Games are representing in Paris for this industry? Are you excited to see something in the air at that time? Please, tell me what are your thoughts on the games coming? Brandon de León  54:01  Oh, yes, sorry. I missed that point entirely. Thanks for making sure I answered. So I think, yeah, it's a fantastic point. Because, for better or for worse, you can hate or love the Olympics, right? There's a lot of debate around that. But I think that the reality is major sporting events of other types, and just major events generally, whether it's a Swiftie concert, or whatever, that is a prime opportunity to build awareness and plant those seeds if you can get your product in front of that audience. It's massive for any business, right? This is why in the U.S., you see companies paying millions and millions and millions for 30 seconds during the Super Bowl, which is our American Football Championship, right? Every year. And it's the same thing is at play here. And so, the Paris Olympics are very interesting because Paris as of late, especially, has been a city that is very intent, with the city leadership on improving quality of life, introducing better transport, a lot more biking paths and making it just easier to use, to a more livable city, let's say it that way., I'm living here in the Netherlands bicycles are a way of life. And the people who are pushing the bicycle culture and infrastructure and urban planning from the Netherlands point of view at the universities and Amsterdam and other places, Paris is one of their favorite cases to point to. I think more people this week or this month, it was reported, more people were biking than driving in Paris for the first time in known history since I guess the advent of automotive. So I think it's really exciting time in Paris, but also Paris is also known for and France, too, for being unabashed in protecting their culture and also making sure that their perspectives are respected. And so you see a lot of this in sort of the way from the space I work in. Now with SUVs, one of the things I've noticed and seen is a policy around SUVs, where I think it's a proposal or it's gone into effect now, where SUVs will pay more for parking in the city. So what happens and where this comes into play with the Olympics is that for years, people have been in the industry targeting the Paris Olympics as a launch point some other some companies that were planning on doing flights at the Paris Olympics in this summer in 2024 realize they weren't going to make it in time technically, to be ready to fly. But this particular company called Volocopter, that I've mentioned before, out of Germany, they're very keen on demonstrating again, they were the ones that flew the Paris Air Show last time, and they've since done a massive amount of flights in the U.S. going around different cities and stuff on a roadshow. So they're very eager to build awareness, which results from this researcher's point of view, of course, and they see the Olympics as an iconic moment, because they're European company. They're very much proud of that. And also, if you look at the history of Airbus, Airbus was a European project, Pan European right parts come from all over Europe to build those planes. And this is maybe a second coming of Airbus in so many ways, in this new air transport world. And so it's super symbolic to be able to fly at Paris, in front of the crowds of Olympic spectators, not just at Paris airshow where you have a lot of aviation, aware or interested or geeky type folks, or people who work in the industry. It's a home field advantage when you're flying above that crowd. But when you put it in front of the Olympic audience, that's a whole nother level of magnitude and exposure and media coverage. And so that can do wonders for the company and change its fundraising prospects, it's runway and its ability to develop future products and launch into other markets and really, potentially accelerated and develop its first mover advantage, too. So it's huge. What's interesting is in September, the Paris city council acting on complaints from citizens about this plan of that air vehicle flying there now, I would say negotiations is not very clear what conversations are happening. But it was brought into question whether they're actually going to be allowed to fly over the city, whether or not they can get certified in time to do it. And that last check, I believe the CEO was reported as saying that they might not launch in July as originally hoped if the certification doesn't come on time. But they're hoping at least to be able to do it in August for the Paralympics. So there's a nonzero chance that they don't get to fly. That could happen. And that would be for them, I think they would class that as a really big disappointment, a missed opportunity, and so on. And also an opportunity for Europe and Paris, the show itself as a showcase for innovation in the space and air transport. So I think it's really interesting when you look at these big events, because they present such an opportunity. It's clear to the commercial side that they're chasing it. But what really validated my research was that social acceptance came up as an interesting issue already, before the first vehicle flies. And like I mentioned before, the next plans are also around big events, the World Expo in Osaka. Next year in 2025, this was to be flights. And in 2028 in Los Angeles for the Olympics there. Other companies from the US are also planning to fly. So yeah, social acceptance is already showing itself as a key risk. Corail  58:58  Yeah, that's crazy. It's kind of a live case study. For your (inaudible). The images you put in your executive summary of this electric planes flying were incredible. I have to admit, I didn't even know that it was already existing. and they were already flying planes, electric flying planes. So that was great. And I will be in Paris this summer, and I crossed all my fingers, that social acceptance is not blocking this line from playing because I want to be there and look at them.   Brandon de León  59:34  Same here.   Corail  59:35  Well, thank you so much, Brandon. I think I don't know Riccardo, if you have a closing question, or, but I think... Riccardo Cosentino  59:42  No, that's no, I think no, I'll leave it with you. Close.   Corail  59:46  Yeah. I think Brandon, that was fantastic. We learned so much. Although I read your entire dissertation. It was super interesting and fascinating. And I feel that you gave us even more explanations and stories in thepodcast. So thank you so much for being generous with all your knowledge. And yeah, I wish you the best in your career, really. Brandon de León  1:00:07  Thank you guys. Thanks for having me. Riccardo Cosentino  1:00:08  Thank you, Brandon. And thank you, Corail, for co-hosting the episode today. It's always an honor having you as my co-host, and there'll be hopefully more opportunities. And Brandon it's always a pleasure chatting with you. Brandon de León  1:00:21  Likewise. Take care, guys. Riccardo Co

What if your everyday mom moments could fill the pages of a book? What if motherhood has taught you leadership skills unmatched by any book or top-tier business school? What if the only thing holding you back from sharing your message with the world was simply not recognizing and appreciating your own value and the support others can offer once you do? These personal topics, especially if you're a bookworm or entrepreneurial mom, are discussed in today's episode with Zibby Owens, bestselling author of "Blank". Zibby is also the award-winning podcast host of "Moms Don't Have Time to Read Books" and the founder and CEO of Zibby Media. With four children and a Harvard Business School graduate, Zibby's journey is nothing short of inspiring. Check out her full bio below. In the meantime, don't forget to say hi on Instagram: https://www.instagram.com/melissallarena/ to let me know who else you'd like me to have as a guest. I love guest and topic ideas. This episode is sponsored by my book, "Fertile Imagination," A Guide for Stretching Every Mom's Superpower for Maximum Impact. In this episode, I realized that Zibby Owens possesses a fertile imagination, the ability to conceive of something that a mom may not have done before. I mean, how many moms do you know who, after a decade of being a stay-at-home mom, decide to disrupt entire industries? That's precisely what Zibby is doing, and that's why I was so inspired to have her on Unimaginable Wellness. Through my podcast and the guests featured in my book, I aim to showcase individuals who harness their imagination for good, making an impact that can reshape opportunities for their children and future generations. Helping moms ignite their imagination to solve meaningful problems is what truly ignites my passion. As your host, Melissa, I urge you to consider: Do you know what truly ignites you as a mom, a human, a woman? If not, and if you're unsure how to rediscover your interests, then "Fertile Imagination" is here to guide you. As a mom of three young boys, I've experienced the Groundhog Day feeling and the endless cycle of sibling squabbles. That's why I wrote "Fertile Imagination," to help readers like you embark on a journey of activating your imagination, listening to its whispers, and integrating what excites you into your daily life. So, whether it's Zibby's story or your own journey of exploration, "Fertile Imagination" is your roadmap to unlocking your inner powerhouse and designing a life aligned with your passions. This is an amazing conversation for you to absolutely take notes and maybe even look some stuff up because if you want to be an entrepreneur, Zibby does drop some nuggets of wisdom in terms of some tools you might want to consider. But at the same time, it's important to appreciate that Zibby was able to create and is still in the process of building an empire. She's authored several books based on everyday life moments that many moms, myself included, often overlook as potential sources of inspiration and revenue as entrepreneurs. There's undeniable value in our daily experiences, even if we don't always recognize it. My book, "Fertile Imagination," guides you in harnessing these moments that ignite your passion and shows you how to share, sell, or leverage them to make your maximum impact on the world. Visit http://www.melissallarena.com/fertileideas or fertileideas.com to grab a free chapter of "Fertile Imagination." Now, let's delve into the official bio of Zibby Owens. Official Bio Zibby Owens — like Pippa Jones — wears a lot of hats. She is the award-winning podcast host of Moms Don't Have Time to Read Books; founder and CEO of Zibby Media, which includes the publishing house Zibby Books, a book club, retreats, classes, and events; and is the proud owner of Zibby's Bookshop, an independent bookstore in Santa Monica. Her previous books include Bookends: A Memoir of Love, Loss, and Literature, children's book Princess Charming, and two anthologies that she edited. A frequent contributor to Good Morning America, Katie Couric Media, and other outlets, she is – most of all! – the mother of four fabulous kids ages 9 to 16 — and wife to Kyle Owens, founder of Morning Moon Productions. Follow her on Instagram and Substack where she tells it like it is. Learn More ·        https://www.zibbyowens.com ·       https://www.instagram.com/zibbyowens/  TRANSCRIPT How are you? I'm great. Thank you so much for having me.  I'm so excited. So I just got through your book, Blank, and oh my goodness, talk about like the turns and tosses and like the silliness. It was really entertaining. And as someone that reads nonfiction, for the most part, I was like, Oh, wait a minute, I'm missing out.   Thank you.  Thank you so much. I appreciate it.  I'm so curious, Zibby. I mean, I asked my listeners kind of what was on their mind, right? What did they want to learn about Zibby? Mom of four, someone that is disrupting the entire publishing industry. I mean, you're making us feel like rock stars. Let me just say any author out there.   But what is it about Zibby? You that makes you so like inventive and innovative in terms of the way that you approach business, mom, life, and this book tour right now with blank. What is it about you? Where did you get that spark?  That is a good question. I don't know where it came from, but  I will say I like to have fun with anything I'm doing.   So I feel like, for example, the book tour, which I'm calling the Zippyverse tour and going to all these stops and I have tour t shirts and friendship bracelets and sunglasses. The point of that is to make it feel fun for the end user.  to make authors, to make myself an example of how an author could be treated as a rock star.   I'm just like doing it myself.  But also, I think most things that I'm doing are in response to something else. So this tour is in response to my last tour where I had a few stops, well one stop in particular, where literally two people showed up to an event and I'm like, I cannot have that happen again.   So instead of just blindly walking into places, I decided to be more in control of it and have my own tour website and then have people so I can be aware of which events are doing well and which aren't and monitor my own marketing. So I guess the tour is an example of how I do most things, which is get a lot of data, figure out what works and what doesn't work, try something new, do something different, have fun, and see if it works or not.   And it might not work, and if not, then I switch gears and try something else. And as to where it came from,  I don't totally know. I think it's just who I am. I love it. I love it. And I think it's, it's so interesting because we physically met at Mom 2. 0, and I was just like smitten beyond belief. I was like, oh my god, just like, it's like you're like Shakira for us at this point.   Also on tour. By the way, like you guys should parallel share tour buses and such  you're living parallel lives. Right. But I was so excited that you were there and I thought it was really, really interesting that you were sharing your perspective on the publishing industry. And you just mentioned data.   And I will say as someone who is excited about building a community of moms that have this like Fertile imagination, like really like expansive thinking approach to problem solving in their lives. I was thinking to myself. Okay. So what shifts are coming up? What do you know? Is there some sort of like magic eight ball in terms of what's happening in publishing?   Based on your right now real time experience on this Zibby tour  Is there any are there any shifts in publishing?  I'm sure like During other times in this industry, there are different genres that, for whatever reason, have been rising to the top. Right? Like, romanticy is huge right now, and I look at some of these festivals I'm going to, and the lines for the fantasy authors are wrapped around them.   So, um. The buildings. I mean, it's crazy. Why does that happen? I don't know. Like, why is, why did the rom com market take off right now? Not sure. Did anyone predict it? No.  So what's coming after this?  It's hard to say. I think that, People are looking more and more for escapes  than more hard stuff. I mean, I love a good hard memoir that'll make me cry, I really do.   But I think for the most part, life is so heavy that people Need an outlet and books are becoming that obviously, as you well know, TikTok and Instagram have changed how books are consumed and found and all of that, which is really interesting. So I'm definitely watching all of that really closely and I find it very interesting and always wondering like, what else can I be doing?   Definitely have not mastered TikTok, so open to any. Anything, any suggestions, but the way that other people have adopted it and promoted books and have books and industries take off just points to the fact that sometimes you don't know where the next thing is going to come from, but the point is to be aware as it's coming.   Everybody bemoans the state of the industry and will people stop reading and I find it incredibly encouraging that so many younger readers are just totally into these things and I, these types of books and it speaks overall to the need for community around books and connection and that's what books provide and  having people um, It doesn't have to be books that are on the bestseller list.   It can be books that are taking off on TikTok or that somebody you really respect likes or something like that. But there will always be that as a way to connect, which I think is great.  And I think in terms of that idea of community, that is something that I secretly have always wanted to ask you, because I sense that the reason, or one of the reasons why Blank is doing so well right now, I mean, bestseller, is that because of the support that the community has been giving you, but then you've been nourishing this community too.   And so I'm just curious, like any stories of like meeting people in person who are part of your community and how we can continue to help this book, like keep skyrocketing. Oh, that's so sweet. And yes, I think you're right in, in part. I mean, I have been doing the podcast, Moms Don't Have Time to Read Books, since 2018.   And there are a lot of people who have sort of been on this crazy ride with me. I mean, in 2017, I went on social media for the first time. Like, I didn't even have an account. And I had been at home with my kids for 11 years, and I had been out of, I've just been out of it, even though prior to that I had been working in different things, marketing and writing, and I had ghostwritten a book and things like that.   But. I had been out of it for a while, and as I've gotten back into it, I've done what I always do, which is sort of share from the heart and share the backstory and share  my deepest feelings, which I've been doing since I was 14 and started writing for Seventeen magazine about like how I felt about my body and gaining weight and all of that.   So I've been really open about like how excited I am when good things happen, how  dejected I feel when like, years ago when I first lost the Webby Award. So now here I am like six years later and this year I won the Webby Award. So like there have been people who have been along the way. So for me, it's more a symbol of things than the award itself, although I'm very excited, but it's like My kids helped me that first year try to campaign and get their friends to vote and all that stuff and then they were there when I didn't win like comforting me and now this year I could tell them that like we won and they could all like hug me and now they're older and so anyway  people I think are rooting for me because I've been really open and yes I help lots of other people but I love that I mean it helps me just as much but I love getting to know authors and talking to people and and and you Just because other people have helped me so much by sharing in their memoirs or their essays or whatever, I give back by doing the same thing and hope that it, you know, I'm paying it forward and that it helps other people.   And I'm, I, I do think that my community has turned out to read blank and that makes me really, really happy. I think it's also a quick read. I think people are enjoying it and it's more, it's more fun. I think, I just think people are looking forward to it. Now for something like that, that's just a little more fun, and  I totally agree with you in the sense that a lot of the moms around me, like I've noticed, there's like this like book club mentality.   Either you're in a book club to like escape family life, and it's just like chit chat with girlfriends about like really random things. Or like a book club and you're an aspiring author. And like, you want to get in the minds of perspective readers. And so the escape society, I'm just going to like name them that.   I wonder like, okay, like fact and fiction are so blurry right now. And I was reading blank and I couldn't help but think like. Does Zibby have like a secret Instagram where she's like going to open houses? So what's the secret there? What's fact? What's fiction? And, and I loved the book. It was very unexpected.   Thank you. Yeah. I love going to open houses. I have gone to 8 million open houses. I always like pull the car over and pop in. It's something like the kids are like, I'll be like, open house. They'll be like, no.  So I just love doing that. I love seeing, I love design. I love just seeing how people live their lives too.   Like  I, I don't know the fact that like people just open up their homes. It's amazing. So I do not have a secret Instagram account, but, but I do love open houses. So that part is absolutely true.  That part. And then there were So many other moments in the book, and I'm not going to give any spoilers away, but there were so many moments that I was like, yep, that visual.   I know it. I've seen it. Like, let's just talk about, for example, one, like the Benihana  Onion Volcano, like it. Yeah. When you said that, or when I heard it in the book, because I have the audio version, I was like, yep, I know what that looks like. I know those volcanoes. And it's like one of those mom moments, like, should we get like a fire extinguisher kind of a thing?   But where did you pull out these different sort of like visuals that were like, so quote unquote, gettable?  I mean, that's my life as a mom, like, I, I mean, I, I am doing all these things, I am in it, like, I have four kids, like, this is my life, and it has been for a very long time, so, I'm just putting in all the things, they just come out, like, I'm, I wasn't like, oh, I need to be sure to put the Benny Hanna image in, in fact, I barely even remember where that is, but,  It's, it's just like, I've been, I've seen that so many times, and I have, I could fill endless books with just the kid stuff, so I think that I've noticed as I write fiction,  which is new for me, I mean, this type of process,  I, I, things are coming out that I don't even know.   I mean, I'm writing my next book now, and I have this huge outline, so I really just have to like, put the words in, like I've already figured out the whole story, and it's, I actually made it kind of a little less fun for myself, I think, because I already know everything, and now I just have to, like, write it and make it fun, but.   Anyway, I started writing it with this detailed outline and like, I wrote 10, 000 words about something totally not even in the outline. Like where is this going? But it's like I had to, I invented this whole backstory. Point is like my subconscious kind of takes over, I think, or whatever happens. But fiction is something I don't totally understand how I'm doing it.   But I know that it's like a mishmash of all these things that are in my brain, like a. Okay. The endless trips to Benihana and like the fact that my daughter's hydroflask like never fits in the car and like they pushed me out of the way to like get to the radio thing and I'm like always about to get crash my car like all those things are just my life that's just my day to day life so and I know I don't always post  that stuff like on social media and in my newsletters like I'm very intentional about not including my kids and things that are too specific to them.   Um, But that is my life. I mean, I post the glamorous parts and I post, I mean, I hope to, I hope, I hope I post like enough real stuff that people know, but I don't want to reveal personal things about the kids. So anyway, all to say, yes, I've been there many times.  You have like best customer out of Benihana, right?   So it's, yeah, totally like, okay, it was just hilarious to me. Like when that mention happened, cause I was just like, taken there immediately and then wanting teriyaki chicken. But anyway, that withstanding, you're actually making me hungry right now. I know it's, it's early, but like I could, I could have a turkey right now.   So this, this, so this is something that I find really common with so many women that Have been either out of the workplace or they're like starting brand new. Like you just mentioned these tiny, regular, banal moments of motherhood. And what you managed to do, my interpretation of it is make it into a story, make it into something that matters to a reader and like engage the reader.   Like, why is it that. In, in your situation, I assume like that that's enough for a story, enough to engage a reader. And I found so many other moms are like, but I have nothing to write about. And you can write a whole like 10, 000 word thing about that.  Well, I think it's how much we value those stories. I think a lot of moms don't put worth into what they're learning, but we're learning lots of things every day as parents.   I mean,  I learned more from being a mom than through all my education and I've gone to lots of schools and da da da, but it's an on the job, constant learning, constant changing,  like full body experience and no one can tell you how to do it and you can't study for it  In that way, it's like a nightmare, right?   You have to just, it's the most important thing in the world. And, you know, for people who like to prepare, there's not that much you can do that will actually help, right? Until you get to know what you have, what you're dealing with, the sleep books. Do they work? I don't know. I mean, I've read a lot of them.   Did they help me? Not so much. So I think that as we are dealing with our kids, and learning and ingesting information. It's, it's information that others are ingesting in an equally  unsure point of their lives. And by reflecting it back, I think we're reminding each other that none of us are in it alone, even though it can feel very isolating.   I mean, a night at home, my kids are older now, like my youngest is now nine, but Like the the tantrum not ending  at home at bedtime. I mean,  I have felt like well, is this ever gonna end? Will this kid ever stop crying or will this child ever fall asleep or how am I ever going to get out of the situation?   And There's nobody to call, like, it doesn't matter, you can't, so, anyway, it can feel very  lonely and isolating and terrifying, but really we're, so many of us are going through similar things, so as long as we share it, it makes it, and add some humor to it, I think it will help get through those day to day moments that can feel really difficult.   I agree. And I also think about my kids almost like imagination experts in residence. It's like, they help me imagine how a situation can be more fun. They help me imagine how I can like, lighten up. Mom, I've heard that before.  Like they, they really do help me navigate the, the, the stuff that is impossible to just kind of navigate with just my thinking, rational brain.   Right. Cause things don't always make sense. And, and what you mentioned right now, as far as motherhood and loneliness and just like solitude, that's how I felt when I was writing my book. And when I was reading your book, I was like, wait a minute, like, how is it like, I almost feel like we all need an outlet, like if we do embark on writing a book, right?   And so maybe I haven't found one yet. And like, I need to go get one. But at this point, it's It's going on artist dates for me, so I'm all about like Julia Cameron's like idea of like going and doing fun things. So as you're on this Zibby tour, I'm just wondering like, do you have an outlet? Maybe one where you just kind of step it up in your enthusiasm and feel energized and things like that.   Hmm.  Outlets for me.  I really enjoy design, like graphic design and all that and like Canva and creating things and I'm not good enough at it. Like it's very frustrating to me. I want to be and I keep trying, but I seem to, I know what looks good, but sometimes I don't know quite how to get there. So I have a lot of fun.   Doing that and trying always to improve and also teaching myself new things like this morning for instance Like I had a bunch of files on Dropbox that I'm like, I need to get these files onto YouTube There must be a better way So like next thing, you know I'm like learning how to use Zapier and like trying to schedule zaps and I'm like, I don't even know what I'm doing I'm probably about to make a huge mistake.   It ended up not totally working now I have to go back and figure it out But like Okay, that's fun for me. I, I, I just enjoy a challenge. And even if it's something as simple as that, so  I realize that's not as lofty as Julia Cameron, but also I have so many things that I do. I mean, I have my podcast, which I have been doing daily for four plus years, and I still have tons of episodes each week because even though I've gone to three times a week,  I had booked so many that, uh, the ones I'm doing now are coming out so late, which is ridiculous.   Like, I really have to, now I might have to change it all again. But anyway, I have all these different things that I am constantly kind of multitasking. Doing the podcast or having a meeting about, like, the marketing of a particular book or helping another author or writing an essay or maybe I should do a gift guide for Mother's Day or like, I mean, like, I, I am always like that.   So I feel like my life is one procrastination from something else all the time. But it sounds like productive procrastination, which I want to make a thing, like, I think, I think we need to celebrate productive procrastination, because if you have so many ideas in your mind, it's kind of like, all right, follow this thread and then go follow this other thread and something's going to work for sure.   And. And I'm wondering, like, when it came to your own business and when it came to, I'm not sure if it was like a clean transition from podcasting to publishing, but when it came to your own business, how did you like follow the thread in terms of like brick by brick building out this Zibby verse, which is like, wow.   I mean, I didn't try to start a Zibby verse. Like, I'm literally wearing a sweater today that says Evyverse, that these two amazing women, Susie and Andrea, who live in Minneapolis, who I've gotten to know, and I went, and they offered then when Blank came out, to your point, to like host me in their town and introduce me to all their friends.   And I was like, I'll go to Minneapolis. So I went, and they gave me this fabulous event, and then they gave me this sweater as a gift that like a friend of theirs like sewed on.  I mean, it's so nice. I, that was not the plan. Like, I really started out, to be perfectly honest, just trying to sell a book. I didn't have a platform.   I wanted to write a book forever. I'd written, I'd ghost written a book. I'd had a novel rejected in my 20s and I wanted to get back to it. And I had a bunch of parenting essays I'd been writing. That I wanted to turn into a book and so I started a podcast and I just, I think I'm just good at taking something and growing it and turning it into something else or just seeing opportunities of what comes next or if somebody else suggests something like testing it out or trying it or like mulling it over and figuring out how I would do it and just trying.   I mean, it was so frustrating at the beginning. I was like, okay, no one is ever going to download this podcast. I'm going to be stuck at 86 downloads an episode forever. And I was like, but you know what? I still love what I'm doing. So even if I only get 86 downloads an episode, I'm not going to stop. And even if I have like five different books rejected, I'm not going to stop.   Like, this is what I want to do. I, I, I'm very persistent when it comes to things that I enjoy and that I feel like I'm on the right path, even if it's not being validated to me externally, so  I'm just going to keep doing what I'm doing and see and that's how I've gotten here and  The publishing, obviously, was a different level because it took a lot of investment and hiring and all of that.   So I did a ton of research. It took me a couple years to decide to do it. Like, I thought about it for a while before I did it. I had lots of calls. I had, there, there is some method to the madness. But in the end, I just thought, what's the worst that happens? I try to start a publishing company to help authors and it fails.   Still, still cool. So, that's sort of my attitude. What's the worst that's going to happen? Yeah, and I think, I think a lot of times we just like assume like the worst is like total demise, the end of, of all our finances and just like, it's very catastrophic, I think, or maybe it's just my anxiety or something, but I, I see that, I see that, and so right now, just to kind of like put it out there, as I think about it.   Cool. My book, and I think about, okay, how can I bring together a community of moms who want to think expansively about what's possible for themselves, who want to use their skills in ways that have never been used by other moms around them before, like What would you suggest would be, like, my first step?   Okay, so, developing a community.  Well, you already are doing a community. You have a podcast. You wrote a book. I mean, you don't need You don't need my help. I should be asking you, like, what has worked for you? Seriously, you know, for me, what worked for me is really taking like activating my chutzpah, honestly, that is what has worked for me so far, like talking to people who I know the book will help feel empowered, like one of the people that read my book, for example, she's like, I felt like I was back in fourth grade when I read it.   Fertile imagination, like that teacher who's like, you can do it. And a little bit like Navy seal ish too, but you know, more like the teacher that's kind and generous. And like, that is what I was going for. So I wanted to just like, put myself out there intentionally to your point. Right. So it was kind of like, okay, like, where are these moms who need this information and who are super educated too?   And I think. That's like a tiny little nuance, but it's like, I don't know about you. I know you went to HBS, I went to Tuck. We have our MBAs and I can't say that doing a ZAP on Zapier was something we learned in our relative business schools. I'm just saying. Right. Do you know how to do zap since Zap, zap year?   I know. I'm a zapper. Yeah. I zap. Yeah. Oh yeah. We zap. We zap. We zap a lot. I get my emails with all my little zaps and this is how Zapier saved me time, right? So yeah, I'm happy to go in there and like help you zap stuff. Yeah, I might need that after this call.  For sure. For sure. But like, this is not what we were taught like in business school.   It was more about like how to plug into an organization and how to lead and how to like look at P& Ls and stuff. And granted, of course, we have our respective PNLs, but I'm just wondering, like, what would you say is the most valuable thing that you've learned as an entrepreneur, like doing it here in your space that you think should be taught in a business school?   That's a great question. Because yes, I did take these things in business school and they, did they help me? Right, exactly. Like,  I mean, honestly,  I think they should teach you that You can watch a quick YouTube tutorial about literally anything and teach yourself how to do it. That is how I started a podcast.   I was like, Google, how to start a podcast. You know, how do I? I watch videos all the time. Like we got locked out of the car the other day and everyone's like, what should we do? And I was like, I'm sure there's a YouTube video about this. And like, there was. So,  I think it's, it's knowing that like, there are so many experts in so many things that anything you want to do, there is a guide.   And now it's easier than ever to take people's advice and have them teach you. I think being open to learning and innovating, and this is something we learned is,  The ability to sort of shift gears and  be, pay attention to market trends and da, da, da. Yeah, yeah, yeah. Like we learned that, but you know, I feel like my cases in business school were like operational challenges and, you know, the cranberry manufacturing and like, I was like, what?   And like, then in like our one  conversation or one class, it was called leadership about like all the ways that people led wrong. And,  and so I think about that, but I'm also like, I don't want to ever lead as the professional only version of myself. Like, I've decided that the best way to lead is just to use myself, even if it's my mom's skills.   in the workplace. Like, I feel like I lead as a mom and I feel like it has only made me a better leader and they don't talk about that at business school at all. There is something about like taking care of people and gathering people and the softer skills or moments. Like with my company, I decided like three days a week, we all, Like take a break and go and have a nice lunch together  every week.   That's what we do. It's part of our culture. It's just like what we do. And I have them to my home and we sit around my dining room table and have lunch. And it's so nice. And it's  like, I want to take care of them. I want to  my teammates, like  my most important dinner guests, because that's what they are.   They are so important. And why would you treat an employee any differently than that? Like, if you don't want to have them. If you don't feel like they deserve that, they probably then should not be working for you. Do you know what I mean? So,  and, and leaving, like I leave every day at 2. 30 and go pick up my kids.   And I get it all done the rest of the day, but like, do I need to be in the office? So I just think there are so many things we could have learned about there are ways to make it work and  finding flexible environments and  leading by example and Making it work. And, and then I guess just always, like, I think that businesses, leading a business is like raising a kid in that as soon as you feel like you have it down, everything changes.   Right? I got it. And now, suddenly, they're a year older and I don't know what I'm doing again. And none of the clothes fit. Like, I have to start again? Are you kidding me? And that's what it's like in business. Like, things are constantly changing. Oh, okay, maybe it's an external change that's impacting the organization.   Maybe it's an internal growth change. But we have to be ready to, like, run to the gap and get new clothes.  It's so funny. I was just telling a client yesterday that using your imagination is like the perfect change management tool.  Like, I mean, so many people that have businesses are able to just go from one thing to another quite quickly, just really edit, change, refocus, and then, and then not feel like, kind of like, I don't know, maybe it's just me.   Like for me, I would kind of feel like men. I did it wrong. Embarrassed. Like I was going this way and now I'm like, no longer doing this thing. What people are going to think. And so I think that's something else that stops you. And the same could be said when you're at a target and your kids have a meltdown and you have a choice, be the mom that you intended to be in front of other people or care what they think.   And then be that another kind of mom. It's. The parallels are striking between motherhood and entrepreneurship for sure. So on that point, I'm just curious, like, can we name the superpower that you got from being a mom that you apply in business? What, what would you name that superpower?  Being humble.  100%.   The superpower is  the ability to constantly learn and change. Honestly, I mean, you have to constantly adapt  to changing conditions as a mom. So adaptability.  Yeah, I think  constantly being able to adaptability. Yeah. Regroup to do lots of things at once and to deal with things that are ultimately out of your control even when you care Yeah, and that's when it's like the hardest, right?   It's like when it's your heart when it's your name when it's the Zibby verse, right? It's different than if it were this third party entity and I think I think we can, we can close the conversation there because I think right now a lot of listeners are probably wondering to themselves like, holy cow, maybe there is value in my like super regular mom life moments.   Maybe I could write a story, right? Much value, so much value, so much value. And I think now the onus might be on people that have these like MBAs and things to put that value in like an income statement, like  goodwill, at least, right? Like just like find a place. To, to demonstrate that we, if we use the success markers of the world, like finances and money and all that, like, how, how can we translate that to like real income financial security for moms?   I mean, I think, I think the value is huge.  And so my hope, my intention is through your story, Zibby, through this conversation that anyone listening feels like, what? Heck yeah, I matter. I matter. And I think that's so true. Thank you so, so much, Zibby. This was amazing. Where can people buy your book? Blank.  Yes.   Please buy blank. You can buy it literally anywhere. Go to your local independent bookstore, order it online. My website is zibbyowens. com. You can, there are links to lots of places to buy it there. You can get a signed copy at Zibby's bookshop through my website. You can. Get it, I don't know, wherever you get books or you can listen or you can read it on Kindle or whatever and you can follow me on Instagram at Zibby Owens or and or my sub stack zibbyowens.   substack. com and you know, I tell it like it is. Absolutely. Thank you again, Zibby. Thank you.  Here are the three things that I would love for you to just really reflect upon after this wonderful conversation with Zibby Owens. Point number one, if you are a mom, even if you have not, I don't even know, use social media for the last decade, I want you to realize that it doesn't matter.   Everything that you want to know about launching a business, you can just Google it, you could even use AI. to help you get started. There is no reason not to explore your entrepreneurial muscles. Absolutely zero reason. So Google it. If you want to build an empire, launch a community or write your own book.   Point number two, We underappreciate ourselves. Let's just imagine what it takes in order to run a house. Imagine what it takes to raise a wonderful human who is going to contribute to society after we leave this planet. There's so much value in what a mom does and no, we do not need to see that value demonstrated in the bank account in order to appreciate it.   You have value that I promise you can absolutely change the world, but we need bravery. That might be what's missing. Bravery. Just be brave to demonstrate to the world that you can do A, B, or C. Be brave to actually start taking the steps and figuring out how you want to express your skills and talents with the world.   And realize that whatever it is that you're doing at home, someone else is paying someone to do that too. Please be aware. So. Underappreciating what you're doing at home, even if it feels like it's, you know, what every mom would do, of course, is, is something that needs to be re evaluated because you have so much value, and I think it's very obvious in the conversation we had here between myself and Zibby.   Point number three. So here's a very big call to action. If you went to a business school anywhere in the world, right, I would invite you to actually speak to a dean and ask that person, whoever they are, if they're open to having you teach a class, maybe on how to be a humble leader, maybe on how to be Empathetic as a leader, maybe on how to really, um, be a more nurturing leader.   These are the skills that you already have built in, in terms of being a mom. And these are skills that you can a hundred percent teach at a business school, and I would invite you to have that conversation with a Dean. As a matter of fact, I plan to do that myself. And so there you go. That is my action item.   Other than that, be sure to buy Zibby's book, Blank. It is available everywhere that you would ordinarily shop for books. Also buy my book, Fertile Imagination. Imagine double fisting two books by moms who might not have time to read a lot of books, but made the time to write books. And I would encourage you to think about your own life experience as worthy of filling the pages of any book.   Thank you again. Next episode drops on Tuesday. Make sure that you follow Unimaginable Wellness.

Today I want to talk about how to look up from your own blog with fresh eyes to see the changing landscape and new trends. And then I want to share the best ways to pivot I recommend you start implementing now to grow your blog income. The Changing World of Blogging Is Blogging Dead? The first question I get asked is “Is blogging dead?” And I'm going to say no, it's not dead but the blogging world is changing. Drop in Google Traffic As you know I talk to professional bloggers all the time, in all different niches, and recently I talked to four who all lost between 70-90% of their site traffic during these March Google updates.  Now we all know that when Google pushes out updates we shouldn't change anything for about a month to see where the dust settles, but I don't think any of these bloggers have seen such big swings before. So to be honest they were all a bit shell-shocked.  Show Notes: MiloTreeCart Book a FREE 20-minute strategy call with me MiloTreeCart Affiliate Program Catch My Party Anca from The Butter Table MiloTree Pop-Up App Join My Blogger Genius Email List Become a Blogger Genius Facebook GroupAll Blogger Genius Podcast Episodes Subscribe to the Blogger Genius Podcast: iTunes YouTube Spotify Google's Influence and SEO Shifts in Search Results So the first trend I expect to see is a decrease in overall traffic from Google for publishers or bloggers who are creating valuable content using best practices. Will we all be losing 70-90% of our traffic, my gut says no. In my experience, when Google makes changes they typically don't reverse course. For example, when we launched Catch My Party, a lot of our users were Etsy sellers so we used their API and imported their products to our site, each product got a separate page. We did this for ease. We weren't monetizing it. Google wasn't seeing these pages as duplicate content so if people were searching for these products they could end up on our site. We were making really good affiliate income, the Etsy sellers were happy because of all the exposure they were getting, it was growing our traffic, and then one day it all went away. Google de-indexed those pages and that traffic and income were gone. My point is that Google changes usually move in one direction. And what is Google facing right now? Existential threats. Remember, all they care about is people coming to Google in need information and end up finding it and having a good experience. Recently though, the overall feel is the search experience has been deteriorating so Google has been making changes. Showing more results from Reddit and Quora. This is because they want answers from “real people.” Google has found that sites have been gaming SEO to make money with affiliate marketing, for example, and Google has been cracking down.  The Introduction of AI in Search Plus how is google going to implement AI in search. They are calling this the Google Search Generative Experience or SGE. Google is working on what this is going to look like, but one thing Google has said is their SGE will quickly provide users with the answers they need and not necessarily direct users to other sites or content.  If their AI, Gemini, can give you better results on their GSE page than sending you off to a site they don't control, you better believe they will do that. In fact, I think it's already happening in the travel space. Plus for successful bloggers monetizing with display ads, as 3rd party cookies get phased out,  advertisers not being able to track visitors, there will probably be downward pressure on ad revenue. Adapting to New Realities I was on a call with a food blogger this past week and I started painting this new picture for her and she couldn't believe what I was saying. She's an old school blogger. She doesn't really use AI in her and feels like her readers could definitely tell the difference.  When I told her it's all about her prompts and she could prompt her AI to sound like her, she got really upset. I told her she should start getting serious about learning how to use ChatGPT, she was shocked. When I told her that some people will be using AI for recipes and not going to blogs she didn't believe me. She said, “Who's going to get their recipe from a machine?” That's not high-quality content. Then over the next two days I heard of two people who get their recipes from ChatGPT and think it's so easy. Do I believe everyone will be getting their recipes from AI, no. But it will become more common. I don't mean to be painting a bleak picture for bloggers and they should not abandon creating great blog content. I just mean to point out that change is a foot and the more willing you are to embrace it, the more success you'll have. What this really means is opening the aperture of how we see our business, our target audience, and our opportunities to serve them. I wouldn't be surprised if the goal of cranking out blog posts, and getting them to rank in Google so you can make a full-time income from ad networks MediaVine or Raptive will shrink in the long run.  So I have 3 recommendations: 1. Diversify Your Traffic Sources Start investing in Pinterest again if you had given up on it in the past. It might take a couple of months to see ROI on it, but Pinterest static pins still drive traffic. I've heard of people going back to Facebook and using it to drive page views. I think this is especially true and platforms like Facebook are deprioritizing news. If you can build engagement with what you post on Facebook, you can drive traffic. I've even heard that people are starting to see traffic coming from other search engines like Bing. Bing is growing in search. So Google should not be your only play. And another place to invest in is growing your email list. Besides your blog, your email list is one of your most valuable assets. Because you own your email list. And all these people have said yes, to you showing up in their inbox. Think of that as them inviting you into their living room, so use it wisely. Give them value and let them get to know you. Focus in Email Marketing And remember, email marketing is the best and easiest way to sell. This also includes showing up on social media. Even if these platforms don't drive a lot of traffic, right now, it's about getting know. For you. Which leads to my next recommendation.  2.) Start Building Relationships With Your Audience Don't think of them as nameless, faceless, “traffic.” These are real people. Like you are a real person. And this is where your competitive advantage moving forward comes into play. With AI, even great content is becoming a commodity. I can get a recipe or DIY from AI now. But YOU are not a commodity. You are a problem solver.  Start Focusing on Your Brand I did a solo episode on how you are now not just a content provider. You are an expert, selling your “vibe.” A few of the bloggers I spoke to this past week, didn't like this idea because they're used to hiding behind their blogs. But I'm going to say, this is the new reality. Get used to putting yourself out there, so people can get to know you. This is why you want to figure out your personal brand based on your expertise and vibe and start sharing it. And finally, my last recommendation: 3. Sell Solutions to Problems This is all about finding new income streams. We're moving from an information economy (with AI, information is becoming a commodity) to a people-focused economy. Of course selling digital products is probably the easiest way to get started. If you want to talk to me about how to do it go to milotree.com and book a free 20-minute Zoo call with me. I'll help you readjust your thinking as a blogger. I feel like this is what I do a lot of. And yes, starting with an ebook or guide or other types of digital downloads is a great place to start because they're easy to create and they provide passive income. But what we see in our data at MiloTreeCart is the people making the most money are selling memberships, coaching, live 1-hour workshops. And notice something about each one of those products. It's you showing up live with other people. Sure, if you want to create online courses go for it, but we're seeing mixed success with that. I feel like people are a little coursed out. Book a Call with Me to Talk About Digital Products And at MiloTreeCart, we have all the resources and step-by-step guides to launch these products with ease and without burnout. Just talk to me about it. There are opportunities out there. You just have to look for them. This is your time to connect with other bloggers in your niche, join Facebook groups, watch what bloggers in other niches are doing.  In addition to selling digital products, you could also look into selling physical products like if you're a crafter selling products on Etsy. And another thing you can sell are affiliate products. If you don't want to start creating your own products and services, find affiliate products you can promote that will make you serious money. I'm not talking Amazon affiliate links.  But I recommend you find affiliate products that pay a lot of money. I'm talking about software products that have generous affiliate programs, so think about the tools you use and maybe you could be promoting them to other bloggers. In fact, we sell military card, as you may know, for a lifetime deal of $349 and we pay our affiliates $100 per sale. For example. We sell MiloTreeCart for $349 right now, lifetime deal. And we pay our affiliates $100 a sale.  April MiloTreeCart Affiliate Challenge And I'm even running a challenge in the month of April awarding an additional $250 bonus and a free hour-long coaching call for people who make 8 sales. You make 8 sales during the month you get over $1,350 in value. Here's how to join our MiloTreeCart affiliate program.  Look for Other Income Opportunities And there are other opportunities out there. For example, I had my friend Anca from The Butter Table on the show. She's the food blogger who, like many new bloggers, didn't want to grow her blog the traditional way and wait two years to get any traction, so she figured out how to create user generated content for brands, how to pitch to them, and she's making a killing. She's thinking outside the box. And btw, she's teaching people how to do this. And if you say, I don't have any marketable skills, I”m going to say, go learn a new skill that you can monetize. And if you know how to do anything with AI, you can sell that as a skill. Maybe you can teach other bloggers how to use AI to write blog posts or emails. How Blogging Is Changing and What to Do Now Yes, the world of blogging is changing. What's worked so far, might not work as well in the future. Lean into your brand and your vibe. Your blog is not your brand. You are. Diversify your traffic sources and show up on social media even if it doesn't drive traffic because you want people to know you and you get to know people. It's no longer about traffic, it's about people. Invest in growing your mailing list. Sending emails is the easy way to create your superfans who trust you and will buy from you. Diversify your revenue streams and start selling solutions. You don't need millions of people. You just need a loyal audience who is interested in your blog niche. Find problems you can solve and charge for the solutions. It's a good idea to teach yourself new marketable skills that you can monetize. Find the opportunities (like great affiliate programs like MiloTreeCart) because there will be many new ways people will be making good money on the internet. And this is all good news. And show up as an empathetic human people can relate to. If you have any questions, reach out to me. I love hearing from you. If you're having a rough time, reach out to me. If you want to start selling digital products, definitely reach out to me and we'll get on a free call. Now look, no one knows what the future holds if they say they do, do not trust them.  But I'm actually really bullish about the future of online business because it might not be about our blogs at a certain point. I'm not sure in the future people are going to be going to static web pages for information. But if you can show up as a real person with real solutions and willing to put in the hard work, you will win. I hope you found this episode helpful. And just so you know, there are many nights when I'm lying there wide awake thinking about all of this stuff. So maybe I recorded this episode as much for me as I did for you. If you're liking this show, please subscribe. Please leave a review on iTunes. I would so greatly appreciate it and I will see you here again next week. Other Related Blogger Genius Podcast episodes You'll Enjoy: Capitalize on Your “Vibe,” Use It To Grow Your Blog with Jillian Leslie A New, Exciting Way to Make Money with Brand Collaborations with Anca Toderic How to Embrace Imperfection and Authenticity as a Blogger with Brooke Riley MiloTreeCart, the Best Tool for Non-Techies to Sell Digital Products I also want to introduce you to the MiloTreeCart, a tool designed for non-techies to sell digital products easily. It comes with features like fill-in-the-blank sales pages, check-out pages, a sales dashboard, upsells, and customer support. MiloTreeCart is currently available for a lifetime deal of $349 or three easy installments of $116.33.

In this episode of the Nomad Futurist, hosts Nabeel Mahmood and Phillip Koblence welcome Daniel Pope, Chief Technology Officer of Submer, an immersion cooling technology company. Pope embarked on his journey in the web hosting business at just 16, transforming a single-bedroom server into a multi-million dollar data center. Boasting over 20 years of experience in data center design and operations, his expertise and vision are focused on enabling the transition toward sustainable and future-proofed digital infrastructure.As Pope starts to share his journey, he explains that Submer, the company he helped found, is revolutionizing the way the industry approaches cooling systems for high-density chips. Headquartered in Barcelona, it stands at the forefront of the industry and has attained global reach.“Our products and our technology are mostly focused on large-scale crypto mining operations and the data center industry, including anything from enterprise customers to hyperscale companies. So Google, Amazon, AWS, etc.”Pope goes on to describe his innate curiosity and penchant for experimenting that began in childhood. He describes himself as a “tinkerer,” and recounts a story of taking apart an expensive PC computer his parents had bought for him. Years later, at 16, he embarked on his first data center venture, which started with a humble server in his bedroom.“In a decade, we went from one server in my bedroom to 18,000 servers in a big data center.”From there, he transitioned into roles as a solution architect and later overseeing professional services teams. Through all this, Pope remained deeply connected to the data center industry. Finally, in 2015, he ventured into immersion cooling.“People [started] uncovering the amount of water that was being used in the data center industry, which has been this deep, dark little secret of the industry for a long time.”As he concluded the podcast, Pope left listeners with a resounding message: “Embrace the slime.”Daniel Pope's episode unfolds as a journey through opportunities made from obstacles. Embracing a connection to the data center industry, he now stands as the Chief Technology Officer of Submer, where immersion cooling represents not just a technological breakthrough but a paradigm shift—an opportunity to embrace innovation and sustainability in equal measure. To continue this journey, connect with Pope on Linkedin and follow Submer on Linkedin for updated news and insights.

They say March comes in like a lion and goes out like a lamb, except for those times when March comes in like a lion and takes out a wide swath of sheep. Then they just say, "whoa!". So Google rode into March like a hungry lion with a staggeringly complicated Core Update, a very serious shellacking of manual penalties, changes to who can monetize content with AdSense, a multi-sweep set of algorithm updates, and changes to guidelines given to search result quality raters. Various Google spokespersons have outlined Google's goals for this update while also trying to explain aspects of the update in their opaque and sometimes uptight Google coded ways. It will take at least four weeks for the March 2024 Core Update to fully play out, during which SEOs and webmasters can expect several sweeps over the course of the next four weeks. Expect a massive house cleaning as Google reacts to a year or more of deep seeded consumer complaints about declining search quality. Google says it intends to eliminate about 40% of spammy content. It remains to be seen if March goes out like a lamb or a lion but one week into the month there's already virtual carnage throughout the Colosseum.Support this podcast at — https://redcircle.com/webcology/donationsAdvertising Inquiries: https://redcircle.com/brandsPrivacy & Opt-Out: https://redcircle.com/privacy

Chuck is back for a return appearance and we dig into the successes of CRM's that pair with their weaknesses, Sales Funnels, and much more.Charles Max Wood is a coach and podcast host at Top End Devs. He's building a system to help developers advance their careers by building their skills, personal brands, and networks. He's been podcasting about programming since 2008. He lives in Utah with his wife and 5 children.Chuck is also the author of “The MaxCoders Guide to Finding Your Dream Developer Job.” The podcasts he produces reach more than 80,000 developers every week. -Today's Top 3 Takeaways: Branded domain name emails.Integrating CRM's including ZOHO, Salesforce, Zapier Automation, Click FunnelsOnline Scheduling including Schedule Once and Acuity Scheduling Today's Guest & Resource Links: https://topenddevs.com/Book: The MaxCoders Guide To Finding Your Dream Developer Job – https://amzn.to/2Jrh5A8https://instagram.com/charlesmaxwood Watch us on YouTube: https://youtu.be/hk6DIZvzjSM Timestamped Show Notes: 07:50 – Mail servers on their own are pretty stupid. It's the stuff that you build on top of it that matters. So Google built this application layer on top of it, that sorts it all out and allows for that custom domain email universal platform for businesses. They have two advantages. One is that they've been doing this for a really, really long time. So they they've gotten pretty good just on their own building algorithms to say for example, hey that looks like spam! The second thing is now we're getting into the age where you have Machine Learning aka AI aka Artificial Intelligence. What it is, is they are taking this giant data model, and you train it against the systems that build basically mathematical formulas that replicate the result you want. 09:10 – This is all in software. So effectively, what you do is you set up your system and then you feed it. You say, here's an email, it's spam. Here's another email, it's spam. Here's an email, it's not spam. You send millions and billions of emails through to the end, what happens is when you send an email that you don't know, it can come back and it can say this is 90% chance, it's spam. So then you just have to decide how sensitive you want to be. The more you train it, the better it gets.20:20 – The issue that I had was this CRM, it did a bunch of stuff, I could go I could cancel like four or five services, because they all could now get consolidated. So I was super excited. I start using it, and then it seemed okay, but the email deliverability was in the toilet. So they're like, hey, look, you know, we have this tech expert that will go in and fix it for you. So I paid the extra to have the guy come in and fix it for me. So he comes in and fixes it and the deliverability gets better. It's not where it was using the other system. But it wasn't so bad that I just threw up my hands in disgust and walked away. 30:30 – So I went back to Schedule Once, and you know that they're cheap. So it's not a big deal. But they'll let you set up booking pages that all go to different calendars. That's the kicker, and I have to set up like 10 of them for all the podcasts and everything else. Then I can use the Zoho calendar thing, because it's already built into everything else, just for the stuff that involves just me.42:15 – Final Words Our Final Words of the Show: The thing...

Hey, this is Alex,Ok let's start with the big news, holy crap this week was a breakthrough week for speed! We had both Groq explode in popularity, and ByteDance release an updated SDXL model called Lightning, able to generate full blown SDXL 1024 images in 300ms. I've been excited about seeing what real time LLM/Diffusion can bring, and with both of these news release the same week, I just had to go and test them out together: Additionally, we had Google step into a big open weights role, and give us Gemma, 2 open weights models 2B and 7B (which is closer to 9B per Junyang) and it was great to see google committing to releasing at least some models in the open. We also had breaking news, Emad from Stability announced SD3, which looks really great, Google to pay Reddit 200M for AI training on their data & a few more things. TL;DR of all topics covered: * Big CO LLMs + APIs* Groq custom LPU inference does 400T/s Llama/Mistral generation (X, Demo)* Google image generation is in Hot Waters and was reportedly paused (refuses to generate white people)* Gemini 1.5 long context is very impressive to folks (Matt Shumer, Ethan Mollick)* Open Weights LLMs * Google releases GEMMA, open weights 2B and 7B models (Announcement, Models)* Teknium releases Nous Hermes DPO (Announcement, HF)* Vision & Video* YoLo V9 - SOTA real time object detector is out (Announcement, Code)* This weeks Buzz (What I learned in WandB this week)* Went to SF to cohost an event with A16Z, Nous, Mistral (Thread, My Report)* AI Art & Diffusion & 3D* ByteDance presents SDXL-Lightning (Try here, Model)* Stability announces Stable Diffusion 3 (Announcement)* Tools* Replit releases a new experimental Figma plugin for UI → Code (Announcement)* Arc browser adds "AI pinch to understand" summarization (Announcement)Big CO LLMs + APIsGroq's new LPU show extreme performance for LLMs - up to 400T/s (example)* Groq created a novel processing unit known as the Tensor Streaming Processor (TSP) which they categorize as a Linear Processor Unit (LPU). Unlike traditional GPUs that are parallel processors with hundreds of cores designed for graphics rendering, LPUs are architected to deliver deterministic performance for AI computations.* Analogy: They know where all the cars are going when everyone wakes up for work (when they compile) and how fast they all drive (compute latency) so they can get rid of traffic lights (routers) and turn lanes (backpressure) by telling everyone when to leave the house.* Why would we need something like this? Some folks are saying that average human reading is only 30T/s, I created an example that uses near instant Groq Mixtral + Lightning SDXL to just create images with Mixtral as my prompt managerOpen Source Weights LLMs Google Gemma - 2B and 7B open weights models (demo)* 4 hours after release, Llama.cpp added support, Ollama and LM Studio added support, Tri dao added Flash attention support* Vocab size is 256K* 8K context window* Tokenizer similar to LLama* Folks are... not that impressed as far as I've seen* Trained on 6 trillion tokens* Google also released Gemma.cpp (local CPU inference) - AnnouncementNous/Teknium re-release Nous Hermes with DPO finetune (Announcement)* DPO RLHF is performing better than previous models* Models are GGUF and can be found here* DPO enables Improvements across the boardThis weeks Buzz (What I learned with WandB this week)* Alex was in SF last week* A16Z + 20 something cohosts including Weights & Biases talked about importance of open source* Huge Shoutout Rajko and Marco from A16Z, and tons of open source folks who joined* Nous, Ollama, LLamaIndex, LMSys folks, Replicate, Perplexity, Mistral, Github, as well as Eric Hartford, Jon Durbin, Haotian Liu, HuggingFace, tons of other great folks from Mozilla, linux foundation and Percy from Together/StanfordAlso had a chance to checkout one of the smol dinners in SF, they go really hard, had a great time showing folks the Vision Pro, chatting about AI, seeing incredible demos and chat about meditation and spirituality all at the same time! AI Art & DiffusionByteDance presents SDXL-Lightning (Try here)* Lightning fast SDXL with 2, 4 or 8 steps* Results much closer to original SDXL than turbo version from a few months agoStability announces Stable Diffusion 3 (waitlist)Uses a Diffusion Transformer architecture (like SORA)Impressive multi subject prompt following: "Prompt: a painting of an astronaut riding a pig wearing a tutu holding a pink umbrella, on the ground next to the pig is a robin bird wearing a top hat, in the corner are the words "stable diffusion"Tools* Replit announces a new Figma design→ code plugin That's it for today, definitely check out the full conversation with Mark Heaps from Groq on the pod, and see you next week!

Pablos: Here's one of the things I think is a critical area of invention that remains unsolved, but it's definitely a part of the future. So if you're using an iPhone anywhere in the world, cultures vary. I've been working with this guy in Venezuela on a project. I text him on WhatsApp and then he replies with a voice memo like every time and so his, culture and worldview is just like talking to the phone and probably because I know Venezuelans do a lot more talking or something. Whereas I never use voice memo. I'm texting, but a lot of that is like, I'm in public around other people and I don't want to disturb them and, disturbing people is considered uncool where I come from, but in Venezuela, like everybody's chattering all the time, probably because they're all Latinos. Talking to your computer will become more and more common. And you can see that some people are more comfortable with it than others. I see it a lot more in people from other countries than I do in Americans. Right now, talking to Siri kind of sucks, and Alexa. These things are kind of stunted because, they're very one shot oriented. If you take your iPhone and start using the voice interface for ChatGPT, wow, it gets pretty exciting. Because now you're having this, two way, audible conversation that builds on itself over time. And if you haven't done that, I think everybody should try it because that will give you a sense of where these things are going. Once you get that going and realize, oh, I can just do this while I'm driving or walking, and I don't have to be staring at my phone. It starts to get compelling. And so it's not hard to imagine being, a few years down the road where ChatGPT is just listening all the time and piping in when it has the answers for you . So that's just laying the groundwork, hopefully all that makes sense. But where I think this goes is that we need to solve one really big problem that remains, which is sub vocal input. Ash: Okay. Pablos: And what that means is, right now, if I'm talking, I don't want to talk to my phone, I don't even want to dictate text messages or do voice memo, because there's people around listening, I don't want them here in my business. We're in this situation where the eavesdropping potential, even if you're not talking about something super secret, it could be private or whatever. I don't want to play a message from you out loud and I want other people hearing things that I haven't screened yet, who knows what you're talking about. So, what sub vocal input would do is give you the ability to just essentially whisper and have your phone pick it up. People around you wouldn't hear you, wouldn't understand you but you would still use the same machinery that you have and we all have the ability to whisper, and and quietly. If you're trying to whisper for someone else to hear you, maybe it gets kind of loud, but if you're just trying to whisper to yourself, it can be super quiet. We know that this should be possible, and we know that because deaf people are able to train themselves to do lip reading pretty well. So a deaf person who's, got nothing, bothering them audibly can sometimes, apply enough focus to the task of learning how to read lips that they can do a really good job of it. So there's enough of a signal in what your phone could see. So you know with Face ID there's a tiny little LiDAR sensor that's doing depth, and it can see your face. It can see the, minute details about your face. That's why it can tell, the difference between your face and a photo of you and your twin brother or sister, whatever. So it might be possible right now. With the hardware that's in an iPhone, even though you probably don't have access to the right APIs for this to work, but maybe in a equivalent Android phone or something, maybe this could be prototyped. Where you could just use that machinery, train a giant, model, just a machine learning model on, lip reading. Ash: Yeah. Pablos: And so you would be able to just look at your phone and whisper, and it would transcribe. Ash: There's a couple of things on this. Three GSM world, before GSM, 2000 or so. So we'll go back in time. One of the big conversations that we would have was, I was a proponent saying that we just don't have enough bandwidth and People are like, "yeah, but we're going to have 3G & 4G & 5G & 6G." And I said, "no, no, you're missing the point." The bandwidth to your device is not the issue, it's between the device and the human. It's your conversation. It's, this is where we're stuck. We're stuck because we type, we could try Dvorak, we could try QWERTY, we can pick the keyboard, we can have sideways keyboards, we can speak to it, but I still think all of these are terrible. Whispering, could be very interesting. There was a MIT headset, Alter Ego. So Alter Ego, if you look at this thing up, it's a mind reading, reading device. Sub vocalization signals through EEG, brain activity. He can actually make it work. Pablos: Well, I've played with some of these things. I have NeuroSky headset emotive, but I think what you have to do with them... Ash: This one you wear. It's bone conducting. It's wild. You just put it on and say, Pablos: Oh, it's bone conducting. So it's picking up speech, it's not EEG. Ash: No, no, no. The bone conducting is how it tells you things back. So it even whispers it back. Like, into your head. Pablos: Oh, but you could just do that with headphones. Ash: No, that's how it whispers back. You think it and then it tells you things. Anyway, it's called alter ego, we'll link to Alter Ego. To me, it goes back to what you're saying, which is, is there a way? Otherwise, we just look like, we're murmuring to ourselves, right? We'll just look completely crazy. Like sometimes I get a little bit annoyed with people on conversations with AirPods. You just have no idea what's going on, right? There's a little hairdryer sticking out of their head, and they're like, just walking around, and we just are fully, we're already like, isolating ourselves and now we're, we're conversing. I think what you're saying though is that the sub vocalization stuff needs to be in a way where it's, Almost so discreet that it is a relationship between you and a listening device, right? It's almost like the pixie on your shoulder. Pablos: Yes. Ash: It's like the little angel devils whatever the animated version was. Pablos: Yeah, and I think there could be other technologies. I don't know if you could fit it in something like an AirPod. Maybe like a Compton backscatter detector, one of these terahertz imagers, like the thing at the airport that you do the HOVA signal to, and then it's you. Without a lot of radiation, you know, those things are low impact. You could do something like that to see the tongue through the side of the mouth. Ash: My belief is closer to the way that you were trying to tackle this problem, which is, hey, it listens in and jumps in. But what if I could prompt it to jump in, right? So for example, let's assume that instead of having to build anything new, it's now just listening to me. Constant in real time. Imagine a natural language parsing system with a, engine underneath. We used to call these things While Aware. This was actually the name of our company from years ago. And While Aware was intercepting SMS messages in real time on the SMSC. And the idea was that, it would detect what the conversation was, but because it knows who you are, it would evoke different things at different moments, right? So let's pick, for example, Bitcoin share price, Bitcoin's falling as a price. And that message was coming to you or that data was somehow coming to you. It might say, do you want to open up, your trading account and you can go sell it. And for me, it might, immediately tell me, do you want to book, tickets to Belize in a non extradition country, because my capital call is too high,. Whatever it is, if I have a margin call, because it knows what's happening. It's contextual, understanding. And I think one of the big things that we're missing in all of these little support things that you allude to that ChatGPT brings to the table is contextual. We fail because It doesn't understand us. Siri doesn't know. Pablos: This is a separate conversation. Fundamentally, you are right. The whole future of AI requires that it know you, it needs to know you, it needs to know every conversation you've had, not only every SMS but text message and email, it needs to have 100 percent of that so it understands you. It knows what you know, it knows what you care about, it sees what you do, it sees what you say, it has to have all that and I want the AI to have all that. We need to architect for that and right now we're not doing that because we're building giant centralized AI's. Ash: That's when you're, different technologies, whether it's the backscatter or it's the, lip reader or the whisper detector. All of those become a lot easier when you have context. I don't know if you remember Google's evolution, 2009, 2010, Google suddenly, not as creepy as Facebook, but its searches were just better, its searches were just better. Why were they better? Oh, you're standing in New York city. So obviously maybe it's contextual to what's around you. Maybe the weather is cold. So Google's original cookie, which they're now getting rid of, was so laden with data. If you could mine that sucker, you won. It knew all of the signals. And I used to call it, signal gathering in terms of the more signal you had, the more accurate you became. And the more you look like sort of a savant. So our AI, like you said, isn't really smart and Siri's terrible because it doesn't know much. It doesn't even know intent. So as humans, why is it that we can speak with somebody with a very heavy accent sometimes? Because we know the context of what's happening and why we got there. It's not just lip reading. It's because when we're with them, we do our own interpretive dance. I think that if you tie the two together, what you just said about, you know, these other little signal things, you could pull it off. Pablos: I assume we're gonna get the latter for free. That's gonna happen. AIs will be stunted until they start to have access to everything and know everything about me and my context in real time. So that's all gonna happen anyway, and there's such momentum around that. So I think we get that for free and even if you didn't, having a conversation with ChatGPT right now will probably convince you that it's, like, good enough that we're going this direction one way or another. Ash: The reason I bring all this up is, can you imagine if, instead of having to whisper, what if all I have to do is have my phone out, and I just say yes or no, or I say more? Go back to my Starship Trooper obsession of, "would you like to know more?" What's interesting is, imagine in your scenario, you're having this sub vocal conversation, but instead of you having to have any conversation, ChatGPT has heard you and it's like, " oh, alter ego, Pablos: No, no, I get it. One of my friends, figured out that you could get through life with only four words, fuck, man, dude, and totally. If you just have those four words, you can get through life because you can express a multitude of things with just those four words. Totally. Ash: Totally. Your response, totally. Funny enough though, right? That may solve some of your problems because you could whisper a little Pablos: Yeah, yeah. Ash: And not have to do long things. Pablos: Yeah. Right. Exactly. No, you're totally right. And that's what you do with your friends. And the closer you are to your friends, like if you're just hanging out with somebody you've known for a long time, you can have a lot of communication with very little actual content. If I watch my daughter and her best friend hanging out, they're incomprehensible because they have like, shortcodes for memes, everything they see or talk about or discuss is related to some other thing that I wasn't part of and like they're foreign objects to me. I think that is kind of what you're describing. Like at some point, Ash: So go back to your Venezuelan, right? If you go back to that conversation and they're sending you a voice note. Now, let's say that voice notes processed and parsed and read by our GPT friend, and it comes back and gives you a summary, five sentence. So you don't even have to look. It just whispers it in your head. Like he wants to know, should he edit the podcast? I don't know, whatever it is. And you could just go back and be like, just hit the yes button, right? I mean, you could go back and say, totally. You could do one of your four words. Pablos: Yeah, totally. No, you got to try it. I tried it. You can go for days without using any other words. But yeah, I think that gets more possible. Like with a human, the more shared experience you have, the more shared context, shared vocabulary, the more concise you can be in your interactions. And so it stands to reason that an AI that knows you really well could get to the point where. All you gotta do is nod or wink and you're done, on a lot of things cause it knows how to set you up to make a quick decision. Ash: If it can formulate the outbound response in long form, and all you have to say is totally... Pablos: Mm hmm, yep. Ash: Then you're good, right? That's usually the problem with these voices, with getting those voices. I've got those too, where people, it is the Latin America thing. They just love, like, I don't know what's going on. It was Brazil too, just, people just go off. And they have a recording. I'm like, you do understand, if I could listen to this, I wouldn't be texting you. That's like, I would pick up the phone and just phone you if I can, if I could have a dialogue, I would have one. When I saw that, I was like, well, can you just tell me like what's in the voice recording? That's what we're looking for. The other thing to think of, and I thought this is where you were going before, you were talking about the sub vocal thing, It's almost like the Babelfish thing, for all the fans of Hitchhiker's Guide. I just had this crazy problem happen, which was, I'd ordered an Uber, and I'm sending information to the Uber driver in English, and the Uber driver is replying in Spanish, but I have a little translate button, but I don't think they had a translate button. And at some point they just simply just said, no hable ingles. I tried to give the directions to my house, finally, I had to run into the street. I sent my daughter out into the street, like someone went out and we're trying to tell them like, go to the yellow house. And I'm like, does anyone remember the word yellow? I realized that I was getting translate and they just didn't speak English. I think that maybe there's this universal input concept. If someone sends you a voice message, it not just transcribes it, but maybe it automatically just dumps it into like concise format. Or to the other person, it reads it to them. So you pick your poison of consumption, like the way you like to consume it, and you just build a proxy in the sky that just It just takes care of all this. There's like a universal proxy, like a little babble bot that sits in the world. And I think you could get pretty far with that. And then you use that to feed ChatGPT. And then you use that to go with the totally man, dude, fuck, right? That's your sequence to that. And then you add your sort of exotic input mechanisms for your sub vocal and everything else. So I could like, you know. Whisper. Pablos: So job one is all the people making AIs need to figure out how to make them mine so that I have my own that I can love and trust and have for life. Job two is they need to make that thing know everything about me, I'm not just a lowest common denominator, I'm me and I need, I need my AI to really know me. Job three is we've got to come up with some clever hardware for doing sub vocal input and it could be something that you wear like a headset that just see through the side of your face and see what's going on in your mouth and your tongue and your embouchure Ash: Well, it could be like a body cam, just clip it on. Pablos: It could be something like that, something that looks up at you. I don't know, it's hard to mount something that sees the front of your face very well, a phone does, though. And even if you had to just aim the phone at your face for it to work. That would be a good start. And I think you could do that today without making any hardware. Ash: Yeah, well, you could put it into your Apple watch. Just hold it up. it's like Dick Tracy. Pablos: There's no camera yet, but next apple Watch will. Ash: Yeah, next Apple will have a little camera, so you just hold that up. It doesn't even have to, you just have, you don't even have to hold it up because if you're using your little radar or LIDAR thing, you just have to have your hand out a little bit. Gesture control on steroids. Pablos: Did you see they put like a gesture control in the new Apple Watch, but it only knows one gesture, which is you pinch your fingers together and it can detect that. I haven't tried it yet. Ash: The other thing I was going to say is I wanted to add what you said about your daughter's thing is that if the AI becomes your buddy, then the total bandwidth between your AI and you will start to decrease. The requirement will decrease because you'll just be able to speak in your own code. You'll be able to be like, yeah, that thing that we worked on last week, dude. Pablos: Mm hmm. Ash: And then it'll just know, Pablos: Exactly. Right. Ash: the other way that it's going to help. So it all starts with that first step, though. It's got to twin you a little bit. Little little scary on the privacy side. Pablos: That's where, some of these, some of these folks working on OpenAI competitors have certainly, gotten onto that notion. Allegedly Apple is trying to figure out how to make the LLM's local, so they run on your device and presumably that's part of the rationale beyond just, justifying you having to buy a faster device and also, make it low latency.

This episode came together at ~4 hrs notice since Dylan had just landed in SF and we had to setup quickly; you might notice some small audio issues in some segments, we apologize. We're currently building our own podcast studio for 2024!

Thanks to the over 17,000 people who have joined the first AI Engineer Summit! A full recap is coming. Last call to fill out the State of AI Engineering survey! See our Community page for upcoming meetups in SF, Paris and NYC.This episode had good interest on Twitter.Fast.ai's “Practical Deep Learning” courses been watched by over >6,000,000 people, and the fastai library has over 25,000 stars on Github. Jeremy Howard, one of the creators of Fast, is now one of the most prominent and respected voices in the machine learning industry; but that wasn't always the case. Being non-consensus and right In 2018, Jeremy and Sebastian Ruder published a paper on ULMFiT (Universal Language Model Fine-tuning), a 3-step transfer learning technique for NLP tasks: The paper demonstrated that pre-trained language models could be fine-tuned on a specific task with a relatively small amount of data to achieve state-of-the-art results. They trained a 24M parameters model on WikiText-103 which was beat most benchmarks.While the paper had great results, the methods behind weren't taken seriously by the community: “Everybody hated fine tuning. Everybody hated transfer learning. I literally did tours trying to get people to start doing transfer learning and nobody was interested, particularly after GPT showed such good results with zero shot and few shot learning […] which I was convinced was not the right direction, but who's going to listen to me, cause as you said, I don't have a PhD, not at a university… I don't have a big set of computers to fine tune huge transformer models.”Five years later, fine-tuning is at the center of most major discussion topics in AI (we covered some like fine tuning vs RAG and small models fine tuning), and we might have gotten here earlier if Jeremy had OpenAI-level access to compute and distribution. At heart, Jeremy has always been “GPU poor”:“I've always been somebody who does not want to build stuff on lots of big computers because most people don't have lots of big computers and I hate creating stuff that most people can't use.”This story is a good reminder of how some of the best ideas are hiding in plain sight; we recently covered RWKV and will continue to highlight the most interesting research that isn't being done in the large labs. Replacing fine-tuning with continued pre-trainingEven though fine-tuning is now mainstream, we still have a lot to learn. The issue of “catastrophic forgetting” and potential solutions have been brought up in many papers: at the fine-tuning stage, the model can forget tasks it previously knew how to solve in favor of new ones. The other issue is apparent memorization of the dataset even after a single epoch, which Jeremy covered Can LLMs learn from a single example? but we still don't have the answer to. Despite being the creator of ULMFiT, Jeremy still professes that there are a lot of open questions on finetuning:“So I still don't know how to fine tune language models properly and I haven't found anybody who feels like they do.”He now advocates for "continued pre-training" - maintaining a diversity of data throughout the training process rather than separate pre-training and fine-tuning stages. Mixing instructional data, exercises, code, and other modalities while gradually curating higher quality data can avoid catastrophic forgetting and lead to more robust capabilities (something we covered in Datasets 101).“Even though I originally created three-step approach that everybody now does, my view is it's actually wrong and we shouldn't use it… the right way to do this is to fine-tune language models, is to actually throw away the idea of fine-tuning. There's no such thing. There's only continued pre-training. And pre-training is something where from the very start, you try to include all the kinds of data that you care about, all the kinds of problems that you care about, instructions, exercises, code, general purpose document completion, whatever. And then as you train, you gradually curate that, you know, you gradually make that higher and higher quality and more and more specific to the kinds of tasks you want it to do. But you never throw away any data….So yeah, that's now my view, is I think ULMFiT is the wrong approach. And that's why we're seeing a lot of these so-called alignment tax… I think it's actually because people are training them wrong.An example of this phenomena is CodeLlama, a LLaMA2 model finetuned on 500B tokens of code: while the model is much better at code, it's worse on generic tasks that LLaMA2 knew how to solve well before the fine-tuning. In the episode we also dive into all the places where open source model development and research is happening (academia vs Discords - tracked on our Communities list and on our survey), and how Jeremy recommends getting the most out of these diffuse, pseudonymous communities (similar to the Eleuther AI Mafia).Show Notes* Jeremy's Background* FastMail* Optimal Decisions* Kaggle* Enlitic* fast.ai* Rachel Thomas* Practical Deep Learning* fastai for PyTorch* nbdev* fastec2 (the underrated library we describe)* Can LLMs learn from a single example?* the Kaggle LLM Science Exam competition, which “challenges participants to answer difficult science-based questions written by a Large Language Model”.* Sebastian Ruder* Alec Radford* Sylvain Gugger* Stephen Merity* Chris Lattner* Modular.ai / Mojo* Jono Whittaker* Zeiler and Fergus paper* ULM Fit* DAWNBench* Phi-1* Code Llama* AlexNetTimestamps* [00:00:00] Intros and Jeremy's background* [00:05:28] Creating ULM Fit - a breakthrough in NLP using transfer learning* [00:06:32] The rise of GPT and the appeal of few-shot learning over fine-tuning* [00:10:00] Starting Fast.ai to distribute AI capabilities beyond elite academics* [00:14:30] How modern LMs like ChatGPT still follow the ULM Fit 3-step approach* [00:17:23] Meeting with Chris Lattner on Swift for TensorFlow at Google* [00:20:00] Continued pre-training as a fine-tuning alternative* [00:22:16] Fast.ai and looking for impact vs profit maximization* [00:26:39] Using Fast.ai to create an "army" of AI experts to improve their domains* [00:29:32] Fast.ai's 3 focus areas - research, software, and courses* [00:38:42] Fine-tuning memorization and training curve "clunks" before each epoch* [00:46:47] Poor training and fine-tuning practices may be causing alignment failures* [00:48:38] Academia vs Discords* [00:53:41] Jeremy's high hopes for Chris Lattner's Mojo and its potential* [01:05:00] Adding capabilities like SQL generation through quick fine-tuning* [01:10:12] Rethinking Fast.ai courses for the AI-assisted coding era* [01:14:53] Rapid model development has created major technical debt* [01:17:08] Lightning RoundAI Summary (beta)This is the first episode we're trying this. Here's an overview of the main topics before you dive in the transcript. * Jeremy's background and philosophies on AI* Studied philosophy and cognitive science in college* Focused on ethics and thinking about AI even 30 years ago* Believes AI should be accessible to more people, not just elite academics/programmers* Created fast.ai to make deep learning more accessible* Development of transfer learning and ULMFit* Idea of transfer learning critical for making deep learning accessible* ULMFit pioneered transfer learning for NLP* Proposed training general language models on large corpora then fine-tuning - this became standard practice* Faced skepticism that this approach would work from NLP community* Showed state-of-the-art results on text classification soon after trying it* Current open questions around fine-tuning LLMs* Models appear to memorize training data extremely quickly (after 1 epoch)* This may hurt training dynamics and cause catastrophic forgetting* Unclear how best to fine-tune models to incorporate new information/capabilities* Need more research on model training dynamics and ideal data mixing* Exciting new developments* Mojo and new programming languages like Swift could enable faster model innovation* Still lots of room for improvements in computer vision-like innovations in transformers* Small models with fine-tuning may be surprisingly capable for many real-world tasks* Prompting strategies enable models like GPT-3 to achieve new skills like playing chess at superhuman levels* LLMs are like computer vision in 2013 - on the cusp of huge new breakthroughs in capabilities* Access to AI research* Many key convos happen in private Discord channels and forums* Becoming part of these communities can provide great learning opportunities* Being willing to do real work, not just talk about ideas, is key to gaining access* The future of practical AI* Coding becoming more accessible to non-programmers through AI assistance* Pre-requisite programming experience for learning AI may no longer be needed* Huge open questions remain about how to best train, fine-tune, and prompt LLMsTranscriptAlessio: Hey everyone, welcome to the Latent Space Podcast. This is Alessio, partner and CTO at Residence at Decibel Partners, and I'm joined by my co-host Swyx, founder of Smol AI. [00:00:21]Swyx: Hey, and today we have in the remote studio, Jeremy Howard all the way from Australia. Good morning. [00:00:27]Jeremy: The remote studio, also known as my house. Good morning. Nice to see you. [00:00:32]Swyx: Nice to see you too. I'm actually very used to seeing you in your mask as a message to people, but today we're mostly audio. But thank you for doing the very important public service of COVID awareness. It was a pleasure. [00:00:46]Jeremy: It was all very annoying and frustrating and tedious, but somebody had to do it. [00:00:52]Swyx: Somebody had to do it, especially somebody with your profile. I think it really drives home the message. So we tend to introduce people for them and then ask people to fill in the blanks on the personal side. Something I did not know about you was that you graduated with a BA in philosophy from the University of Melbourne. I assumed you had a PhD. [00:01:14]Jeremy: No, I mean, I barely got through my BA because I was working 80 to 100 hour weeks at McKinsey and Company from 19 years old onwards. So I actually didn't attend any lectures in second and third year university. [00:01:35]Swyx: Well, I guess you didn't need it or you're very sort of self-driven and self-motivated. [00:01:39]Jeremy: I took two weeks off before each exam period when I was working at McKinsey. And then, I mean, I can't believe I got away with this in hindsight, I would go to all my professors and say, oh, I was meant to be in your class this semester and I didn't quite turn up. Were there any assignments I was meant to have done, whatever. I can't believe all of them let me basically have it. They basically always would say like, okay, well, if you can have this written by tomorrow, I'll accept it. So yeah, stressful way to get through university, but. [00:02:12]Swyx: Well, it shows that, I guess, you min-maxed the opportunities. That definitely was a precursor. [00:02:18]Jeremy: I mean, funnily, like in as much as I, you know, in philosophy, the things I found interesting and focused on in the little bit of time I did spend on it was ethics and cognitive science. And it's kind of really amazing that it's now come back around and those are actually genuinely useful things to know about, which I never thought would happen. [00:02:38]Swyx: A lot of, yeah, a lot of relevant conversations there. So you were a consultant for a while and then in the magical month of June 1989, you founded both Optimal Decisions and Fastmeal, which I also briefly used. So thank you for that. [00:02:53]Jeremy: Oh, good for you. Yeah. Cause I had read the statistics, which is that like 90% or something of small businesses fail. So I thought if I start two businesses, I have a higher chance. In hindsight, I was thinking of it as some kind of stochastic thing I didn't have control over, but it's a bit odd, but anyway. [00:03:10]Swyx: And then you were president and chief scientist at Kaggle, which obviously is the sort of composition platform of machine learning. And then Enlitic, where you were working on using deep learning to improve medical diagnostics and clinical decisions. Yeah. [00:03:28]Jeremy: I was actually the first company to use deep learning in medicine, so I kind of founded the field. [00:03:33]Swyx: And even now that's still like a pretty early phase. And I actually heard you on your new podcast with Tanish, where you went very, very deep into the stuff, the kind of work that he's doing, such a young prodigy at his age. [00:03:47]Jeremy: Maybe he's too old to be called a prodigy now, ex-prodigy. No, no. [00:03:51]Swyx: I think he still counts. And anyway, just to round out the bio, you have a lot more other credentials, obviously, but most recently you started Fast.ai, which is still, I guess, your primary identity with Rachel Thomas. So welcome. [00:04:05]Jeremy: Yep. [00:04:06]Swyx: Thanks to my wife. Thank you. Yeah. Doing a lot of public service there with getting people involved in AI, and I can't imagine a better way to describe it than fast, fast.ai. You teach people from nothing to stable diffusion in seven weeks or something, and that's amazing. Yeah, yeah. [00:04:22]Jeremy: I mean, it's funny, you know, when we started that, what was that, like 2016 or something, the idea that deep learning was something that you could make more accessible was generally considered stupid. Everybody knew that deep learning was a thing that you got a math or a computer science PhD, you know, there was one of five labs that could give you the appropriate skills and that you would join, yeah, basically from one of those labs, you might be able to write some papers. So yeah, the idea that normal people could use that technology to do good work was considered kind of ridiculous when we started it. And we weren't sure if it was possible either, but we kind of felt like we had to give it a go because the alternative was we were pretty sure that deep learning was on its way to becoming, you know, the most or one of the most, you know, important technologies in human history. And if the only people that could use it were a handful of computer science PhDs, that seemed like A, a big waste and B, kind of dangerous. [00:05:28]Swyx: Yeah. [00:05:29]Alessio: And, you know, well, I just wanted to know one thing on your bio that at Kaggle, you were also the top rank participant in both 2010 and 2011. So sometimes you see a lot of founders running companies that are not really in touch with the problem, but you were clearly building something that you knew a lot about, which is awesome. Talking about deep learning, you created, published a paper on ULM fit, which was kind of the predecessor to multitask learning and a lot of the groundwork that then went to into Transformers. I've read back on the paper and you turned this model, AWD LSTM, which I did the math and it was like 24 to 33 million parameters, depending on what training data set you use today. That's kind of like not even small, it's like super small. What were some of the kind of like contrarian takes that you had at the time and maybe set the stage a little bit for the rest of the audience on what was kind of like the state of the art, so to speak, at the time and what people were working towards? [00:06:32]Jeremy: Yeah, the whole thing was a contrarian take, you know. So okay, so we started Fast.ai, my wife and I, and we thought, yeah, so we're trying to think, okay, how do we make it more accessible? So when we started thinking about it, it was probably 2015 and then 2016, we started doing something about it. Why is it inaccessible? Okay, well, A, no one knows how to do it other than a few number of people. And then when we asked those few number of people, well, how do you actually get good results? They would say like, oh, it's like, you know, a box of tricks that aren't published. So you have to join one of the labs and learn the tricks. So a bunch of unpublished tricks, not much software around, but thankfully there was Theano and rappers and particularly Lasagna, the rapper, but yeah, not much software around, not much in the way of data sets, you know, very hard to get started in terms of the compute. Like how do you get that set up? So yeah, no, everything was kind of inaccessible. And you know, as we started looking into it, we had a key insight, which was like, you know what, most of the compute and data for image recognition, for example, we don't need to do it. You know, there's this thing which nobody knows about, nobody talks about called transfer learning, where you take somebody else's model, where they already figured out like how to detect edges and gradients and corners and text and whatever else, and then you can fine tune it to do the thing you want to do. And we thought that's the key. That's the key to becoming more accessible in terms of compute and data requirements. So when we started Fast.ai, we focused from day one on transfer learning. Lesson one, in fact, was transfer learning, literally lesson one, something not normally even mentioned in, I mean, there wasn't much in the way of courses, you know, the courses out there were PhD programs that had happened to have recorded their lessons and they would rarely mention it at all. We wanted to show how to do four things that seemed really useful. You know, work with vision, work with tables of data, work with kind of recommendation systems and collaborative filtering and work with text, because we felt like those four kind of modalities covered a lot of the stuff that, you know, are useful in real life. And no one was doing anything much useful with text. Everybody was talking about word2vec, you know, like king plus queen minus woman and blah, blah, blah. It was like cool experiments, but nobody's doing anything like useful with it. NLP was all like lemmatization and stop words and topic models and bigrams and SPMs. And it was really academic and not practical. But I mean, to be honest, I've been thinking about this crazy idea for nearly 30 years since I had done cognitive science at university, where we talked a lot about the CELS Chinese room experiment. This idea of like, what if there was somebody that could kind of like, knew all of the symbolic manipulations required to answer questions in Chinese, but they didn't speak Chinese and they were kind of inside a room with no other way to talk to the outside world other than taking in slips of paper with Chinese written on them and then they do all their rules and then they pass back a piece of paper with Chinese back. And this room with a person in is actually fantastically good at answering any question you give them written in Chinese. You know, do they understand Chinese? And is this, you know, something that's intelligently working with Chinese? Ever since that time, I'd say the most thought, to me, the most thoughtful and compelling philosophical response is yes. You know, intuitively it feels like no, because that's just because we can't imagine such a large kind of system. But you know, if it looks like a duck and acts like a duck, it's a duck, you know, or to all intents and purposes. And so I always kind of thought, you know, so this is basically a kind of analysis of the limits of text. And I kind of felt like, yeah, if something could ingest enough text and could use the patterns it saw to then generate text in response to text, it could appear to be intelligent, you know. And whether that means it is intelligent or not is a different discussion and not one I find very interesting. Yeah. And then when I came across neural nets when I was about 20, you know, what I learned about the universal approximation theorem and stuff, and I started thinking like, oh, I wonder if like a neural net could ever get big enough and take in enough data to be a Chinese room experiment. You know, with that background and this kind of like interest in transfer learning, you know, I'd been thinking about this thing for kind of 30 years and I thought like, oh, I wonder if we're there yet, you know, because we have a lot of text. Like I can literally download Wikipedia, which is a lot of text. And I thought, you know, how would something learn to kind of answer questions or, you know, respond to text? And I thought, well, what if we used a language model? So language models are already a thing, you know, they were not a popular or well-known thing, but they were a thing. But language models exist to this idea that you could train a model to fill in the gaps. Or actually in those days it wasn't fill in the gaps, it was finish a string. And in fact, Andrej Karpathy did his fantastic RNN demonstration from this at a similar time where he showed like you can have it ingest Shakespeare and it will generate something that looks a bit like Shakespeare. I thought, okay, so if I do this at a much bigger scale, using all of Wikipedia, what would it need to be able to do to finish a sentence in Wikipedia effectively, to do it quite accurately quite often? I thought, geez, it would actually have to know a lot about the world, you know, it'd have to know that there is a world and that there are objects and that objects relate to each other through time and cause each other to react in ways and that causes proceed effects and that, you know, when there are animals and there are people and that people can be in certain positions during certain timeframes and then you could, you know, all that together, you can then finish a sentence like this was signed into law in 2016 by US President X and it would fill in the gap, you know. So that's why I tried to create what in those days was considered a big language model trained on the entirety on Wikipedia, which is that was, you know, a bit unheard of. And my interest was not in, you know, just having a language model. My interest was in like, what latent capabilities would such a system have that would allow it to finish those kind of sentences? Because I was pretty sure, based on our work with transfer learning and vision, that I could then suck out those latent capabilities by transfer learning, you know, by fine-tuning it on a task data set or whatever. So we generated this three-step system. So step one was train a language model on a big corpus. Step two was fine-tune a language model on a more curated corpus. And step three was further fine-tune that model on a task. And of course, that's what everybody still does today, right? That's what ChatGPT is. And so the first time I tried it within hours, I had a new state-of-the-art academic result on IMDB. And I was like, holy s**t, it does work. And so you asked, to what degree was this kind of like pushing against the established wisdom? You know, every way. Like the reason it took me so long to try it was because I asked all my friends in NLP if this could work. And everybody said, no, it definitely won't work. It wasn't like, oh, maybe. Everybody was like, it definitely won't work. NLP is much more complicated than vision. Language is a much more vastly complicated domain. You know, and you've got problems like the grounding problem. We know from like philosophy and theory of mind that it's actually impossible for it to work. So yeah, so don't waste your time. [00:15:10]Alessio: Jeremy, had people not tried because it was like too complicated to actually get the data and like set up the training? Or like, were people just lazy and kind of like, hey, this is just not going to work? [00:15:20]Jeremy: No, everybody wasn't lazy. So like, so the person I thought at that time who, you know, there were two people I thought at that time, actually, who were the strongest at language models were Stephen Merity and Alec Radford. And at the time I didn't know Alec, but I, after we had both, after I'd released ULM Fit and he had released GPT, I organized a chat for both of us with Kate Metz in the New York Times. And Kate Metz answered, sorry, and Alec answered this question for Kate. And Kate was like, so how did, you know, GPT come about? And he said, well, I was pretty sure that pre-training on a general large corpus wouldn't work. So I hadn't tried it. And then I read ULM Fit and turns out it did work. And so I did it, you know, bigger and it worked even better. And similar with, with Stephen, you know, I asked Stephen Merity, like, why don't we just find, you know, take your AWD-ASTLM and like train it on all of Wikipedia and fine tune it? And he's kind of like, well, I don't think that's going to really lie. Like two years before I did a very popular talk at KDD, the conference where everybody in NLP was in the audience. I recognized half the faces, you know, and I told them all this, I'm sure transfer learning is the key. I'm sure ImageNet, you know, is going to be an NLP thing as well. And, you know, everybody was interested and people asked me questions afterwards and, but not just, yeah, nobody followed up because everybody knew that it didn't work. I mean, even like, so we were scooped a little bit by Dai and Lee, Kwok Lee at Google. They had, they had, I already, I didn't even realize this, which is a bit embarrassing. They had already done a large language model and fine tuned it. But again, they didn't create a general purpose, large language model on a general purpose corpus. They only ever tested a domain specific corpus. And I haven't spoken to Kwok actually about that, but I assume that the reason was the same. It probably just didn't occur to them that the general approach could work. So maybe it was that kind of 30 years of mulling over the, the cell Chinese room experiment that had convinced me that it probably would work. I don't know. Yeah. [00:17:48]Alessio: Interesting. I just dug up Alec announcement tweet from 2018. He said, inspired by Cobe, Elmo, and Yola, I'm fit. We should have a single transformer language model can be fine tuned to a wide variety. It's interesting because, you know, today people think of AI as the leader, kind of kind of like the research lab pushing forward the field. What was that at the time? You know, like kind of like going back five years, people think of it as an overnight success, but obviously it took a while. [00:18:16]Swyx: Yeah. Yeah. [00:18:17]Jeremy: No, I mean, absolutely. And I'll say like, you know, it's interesting that it mentioned Elmo because in some ways that was kind of diametrically opposed to, to ULM fit. You know, there was these kind of like, so there was a lot of, there was a lot of activity at the same time as ULM fits released. So there was, um, so before it, as Brian McCann, I think at Salesforce had come out with this neat model that did a kind of multitask learning, but again, they didn't create a general fine tune language model first. There was Elmo, um, which I think was a lip, you know, actually quite a few months after the first ULM fit example, I think. Um, but yeah, there was a bit of this stuff going on. And the problem was everybody was doing, and particularly after GPT came out, then everybody wanted to focus on zero shot and few shot learning. You know, everybody hated fine tuning. Everybody hated transfer learning. And like, I literally did tours trying to get people to start doing transfer learning and people, you know, nobody was interested, particularly after GPT showed such good results with zero shot and few shot learning. And so I actually feel like we kind of went backwards for years and, and not to be honest, I mean, I'm a bit sad about this now, but I kind of got so disappointed and dissuaded by like, it felt like these bigger lab, much bigger labs, you know, like fast AI had only ever been just me and Rachel were getting all of this attention for an approach I thought was the wrong way to do it. You know, I was convinced was the wrong way to do it. And so, yeah, for years people were really focused on getting better at zero shot and few shots and it wasn't until, you know, this key idea of like, well, let's take the ULM fit approach, but for step two, rather than fine tuning on a kind of a domain corpus, let's fine tune on an instruction corpus. And then in step three, rather than fine tuning on a reasonably specific task classification, let's fine tune on a, on a RLHF task classification. And so that was really, that was really key, you know, so I was kind of like out of the NLP field for a few years there because yeah, it just felt like, I don't know, pushing uphill against this vast tide, which I was convinced was not the right direction, but who's going to listen to me, you know, cause I, as you said, I don't have a PhD, not at a university, or at least I wasn't then. I don't have a big set of computers to fine tune huge transformer models. So yeah, it was definitely difficult. It's always been hard. You know, it's always been hard. Like I've always been somebody who does not want to build stuff on lots of big computers because most people don't have lots of big computers and I hate creating stuff that most people can't use, you know, and also stuff that's created on lots of big computers has always been like much more media friendly. So like, it might seem like a recent thing, but actually throughout my 30 years in data science, the attention's always been on, you know, the big iron results. So when I first started, everybody was talking about data warehouses and it was all about Teradata and it'd be like, oh, this big bank has this huge room full of computers and they have like terabytes of data available, you know, at the press of a button. And yeah, that's always what people want to talk about, what people want to write about. And then of course, students coming out of their PhDs and stuff, that's where they want to go work because that's where they read about. And to me, it's a huge distraction, you know, because like I say, most people don't have unlimited compute and I want to help most people, not the small subset of the most well-off people. [00:22:16]Alessio: That's awesome. And it's great to hear, you do such a great job educating that a lot of times you're not telling your own story, you know? So I love this conversation. And the other thing before we jump into Fast.AI, actually, a lot of people that I know, they run across a new architecture and whatnot, they're like, I got to start a company and raise a bunch of money and do all of this stuff. And say, you were like, I want everybody to have access to this. Why was that the case for you? Was it because you already had a successful venture in like FastMail and you were more interested in that? What was the reasoning? [00:22:52]Jeremy: It's a really good question. So I guess the answer is yes, that's the reason why. So when I was a teenager, I thought it would be really cool to like have my own company. You know, I didn't know the word startup. I didn't know the word entrepreneur. I didn't know the word VC. And I didn't really know what any of those things were really until after we started Kaggle, to be honest. Even the way it started to what we now call startups. I just thought they were just small businesses. You know, they were just companies. So yeah, so those two companies were FastMail and Optimal Decisions. FastMail was the first kind of synchronized email provider for non-businesses. So something you can get your same email at home, on your laptop, at work, on your phone, whatever. And then Optimal Decisions invented a new approach to insurance pricing. Something called profit-optimized insurance pricing. So I saw both of those companies, you know, after 10 years. And at that point, I had achieved the thing that as a teenager I had wanted to do. You know, it took a lot longer than it should have because I spent way longer in management consulting than I should have because I got caught up in that stupid rat race. But, you know, eventually I got there and I remember my mom saying to me, you must be so proud. You know, because she remembered my dream. She's like, you've done it. And I kind of reflected and I was like, I'm not proud at all. You know, like people quite liked FastMail. You know, it's quite nice to have synchronized email. It probably would have happened anyway. Yeah, I'm certainly not proud that I've helped some insurance companies suck more money out of their customers. Yeah, no, I'm not proud. You know, it's actually, I haven't really helped the world very much. You know, maybe in the insurance case I've made it a little bit worse. I don't know. So, yeah, I was determined to not waste more years of my life doing things, working hard to do things which I could not be reasonably sure would have a lot of value. So, you know, I took some time off. I wasn't sure if I'd ever work again, actually. I didn't particularly want to, because it felt like, yeah, it felt like such a disappointment. And, but, you know, and I didn't need to. I had enough money. Like, I wasn't super rich, but I had enough money. I didn't need to work. And I certainly recognized that amongst the other people I knew who had enough money that they didn't need to work, they all worked ridiculously hard, you know, and constantly put themselves in extremely stressful situations. And I thought, I don't want to be one of those idiots who's tied to, you know, buying a bigger plane than the next guy or whatever. You know, Kaggle came along and I mainly kind of did that just because it was fun and interesting to hang out with interesting people. But, you know, with Fast.ai in particular, you know, Rachel and I had a very explicit, you know, long series of conversations over a long period of time about like, well, how can we be the most helpful to society as a whole, and particularly to those people who maybe need more help, you know? And so we definitely saw the world going in a potentially pretty dystopian direction if the world's most powerful technology was controlled by a small group of elites. So we thought, yeah, we should focus on trying to help that not happen. You know, sadly, it looks like it still is likely to happen. But I mean, I feel like we've helped make it a little bit less likely. So we've done our bit. [00:26:39]Swyx: You've shown that it's possible. And I think your constant advocacy, your courses, your research that you publish, you know, just the other day you published a finding on, you know, learning that I think is still something that people are still talking about quite a lot. I think that that is the origin story of a lot of people who are going to be, you know, little Jeremy Howards, furthering your mission with, you know, you don't have to do everything by yourself is what I'm saying. No, definitely. Definitely. [00:27:10]Jeremy: You know, that was a big takeaway from like, analytic was analytic. It definitely felt like we had to do everything ourselves. And I kind of, I wanted to solve medicine. I'll say, yeah, okay, solving medicine is actually quite difficult. And I can't do it on my own. And there's a lot of other things I'd like to solve, and I can't do those either. So that was definitely the other piece was like, yeah, you know, can we create an army of passionate domain experts who can change their little part of the world? And that's definitely happened. Like I find nowadays, at least half the time, probably quite a bit more that I get in contact with somebody who's done really interesting work in some domain. Most of the time I'd say, they say, yeah, I got my start with fast.ai. So it's definitely, I can see that. And I also know from talking to folks at places like Amazon and Adobe and stuff, which, you know, there's lots of alumni there. And they say, oh my God, I got here. And like half of the people are fast.ai alumni. So it's fantastic. [00:28:13]Swyx: Yeah. [00:28:14]Jeremy: Actually, Andre Kapathy grabbed me when I saw him at NeurIPS a few years ago. And he was like, I have to tell you, thanks for the fast.ai courses. When people come to Tesla and they need to know more about deep learning, we always send them to your course. And the OpenAI Scholars Program was doing the same thing. So it's kind of like, yeah, it's had a surprising impact, you know, that's just one of like three things we do is the course, you know. [00:28:40]Swyx: Yes. [00:28:40]Jeremy: And it's only ever been at most two people, either me and Rachel or me and Sylvia nowadays, it's just me. So yeah, I think it shows you don't necessarily need a huge amount of money and a huge team of people to make an impact. [00:28:56]Swyx: Yeah. So just to reintroduce fast.ai for people who may not have dived into it much, there is the courses that you do. There is the library that is very well loved. And I kind of think of it as a nicer layer on top of PyTorch that people should start with by default and use it as the basis for a lot of your courses. And then you have like NBDev, which I don't know, is that the third one? [00:29:27]Jeremy: Oh, so the three areas were research, software, and courses. [00:29:32]Swyx: Oh, sorry. [00:29:32]Jeremy: So then in software, you know, fast.ai is the main thing, but NBDev is not far behind. But then there's also things like FastCore, GHAPI, I mean, dozens of open source projects that I've created and some of them have been pretty popular and some of them are still a little bit hidden, actually. Some of them I should try to do a better job of telling people about. [00:30:01]Swyx: What are you thinking about? Yeah, what's on the course of my way? Oh, I don't know, just like little things. [00:30:04]Jeremy: Like, for example, for working with EC2 and AWS, I created a FastEC2 library, which I think is like way more convenient and nice to use than anything else out there. And it's literally got a whole autocomplete, dynamic autocomplete that works both on the command line and in notebooks that'll like auto-complete your instance names and everything like that. You know, just little things like that. I try to make like, when I work with some domain, I try to make it like, I want to make it as enjoyable as possible for me to do that. So I always try to kind of like, like with GHAPI, for example, I think that GitHub API is incredibly powerful, but I didn't find it good to work with because I didn't particularly like the libraries that are out there. So like GHAPI, like FastEC2, it like autocompletes both at the command line or in a notebook or whatever, like literally the entire GitHub API. The entire thing is like, I think it's like less than 100K of code because it actually, as far as I know, the only one that grabs it directly from the official open API spec that GitHub produces. And like if you're in GitHub and you just type an API, you know, autocomplete API method and hit enter, it prints out the docs with brief docs and then gives you a link to the actual documentation page. You know, GitHub Actions, I can write now in Python, which is just so much easier than writing them in TypeScript and stuff. So, you know, just little things like that. [00:31:40]Swyx: I think that's an approach which more developers took to publish some of their work along the way. You described the third arm of FastAI as research. It's not something I see often. Obviously, you do do some research. And how do you run your research? What are your research interests? [00:31:59]Jeremy: Yeah, so research is what I spend the vast majority of my time on. And the artifacts that come out of that are largely software and courses. You know, so to me, the main artifact shouldn't be papers because papers are things read by a small exclusive group of people. You know, to me, the main artifacts should be like something teaching people, here's how to use this insight and here's software you can use that builds it in. So I think I've only ever done three first-person papers in my life, you know, and none of those are ones I wanted to do. You know, they were all ones that, like, so one was ULM Fit, where Sebastian Ruder reached out to me after seeing the course and said, like, you have to publish this as a paper, you know. And he said, I'll write it. He said, I want to write it because if I do, I can put it on my PhD and that would be great. And it's like, okay, well, I want to help you with your PhD. And that sounds great. So like, you know, one was the masks paper, which just had to exist and nobody else was writing it. And then the third was the Fast.ai library paper, which again, somebody reached out and said, please, please write this. We will waive the fee for the journal and everything and actually help you get it through publishing and stuff. So yeah, so I don't, other than that, I've never written a first author paper. So the research is like, well, so for example, you know, Dawn Bench was a competition, which Stanford ran a few years ago. It was kind of the first big competition of like, who can train neural nets the fastest rather than the most accurate. And specifically it was who can train ImageNet the fastest. And again, this was like one of these things where it was created by necessity. So Google had just released their TPUs. And so I heard from my friends at Google that they had put together this big team to smash Dawn Bench so that they could prove to people that they had to use Google Cloud and use their TPUs and show how good their TPUs were. And we kind of thought, oh s**t, this would be a disaster if they do that, because then everybody's going to be like, oh, deep learning is not accessible. [00:34:20]Swyx: You know, to actually be good at it, [00:34:21]Jeremy: you have to be Google and you have to use special silicon. And so, you know, we only found out about this 10 days before the competition finished. But, you know, we basically got together an emergency bunch of our students and Rachel and I and sat for the next 10 days and just tried to crunch through and try to use all of our best ideas that had come from our research. And so particularly progressive resizing, just basically train mainly on small things, train on non-square things, you know, stuff like that. And so, yeah, we ended up winning, thank God. And so, you know, we turned it around from being like, like, oh s**t, you know, this is going to show that you have to be Google and have TPUs to being like, oh my God, even the little guy can do deep learning. So that's an example of the kind of like research artifacts we do. And yeah, so all of my research is always, how do we do more with less, you know? So how do we get better results with less data, with less compute, with less complexity, with less education, you know, stuff like that. So ULM fits obviously a good example of that. [00:35:37]Swyx: And most recently you published, can LLMs learn from a single example? Maybe could you tell the story a little bit behind that? And maybe that goes a little bit too far into the learning of very low resource, the literature. [00:35:52]Jeremy: Yeah, yeah. So me and my friend, Jono Whittaker, basically had been playing around with this fun Kaggle competition, which is actually still running as we speak, which is, can you create a model which can answer multiple choice questions about anything that's in Wikipedia? And the thing that makes it interesting is that your model has to run on Kaggle within nine hours. And Kaggle's very, very limited. So you've only got 14 gig RAM, only two CPUs, and a small, very old GPU. So this is cool, you know, if you can do well at this, then this is a good example of like, oh, you can do more with less. So yeah, Jono and I were playing around with fine tuning, of course, transfer learning, pre-trained language models. And we saw this, like, so we always, you know, plot our losses as we go. So here's another thing we created. Actually, Sylvain Guuger, when he worked with us, created called fast progress, which is kind of like TQEDM, but we think a lot better. So we look at our fast progress curves, and they kind of go down, down, down, down, down, down, down, a little bit, little bit, little bit. And then suddenly go clunk, and they drop. And then down, down, down, down, down a little bit, and then suddenly clunk, they drop. We're like, what the hell? These clunks are occurring at the end of each epoch. So normally in deep learning, this would be, this is, you know, I've seen this before. It's always been a bug. It's always turned out that like, oh, we accidentally forgot to turn on eval mode during the validation set. So I was actually learning then, or, oh, we accidentally were calculating moving average statistics throughout the epoch. So, you know, so it's recently moving average or whatever. And so we were using Hugging Face Trainer. So, you know, I did not give my friends at Hugging Face the benefit of the doubt. I thought, oh, they've fucked up Hugging Face Trainer, you know, idiots. Well, you'll use the Fast AI Trainer instead. So we switched over to Learner. We still saw the clunks and, you know, that's, yeah, it shouldn't really happen because semantically speaking in the epoch, isn't like, it's not a thing, you know, like nothing happens. Well, nothing's meant to happen when you go from ending one epoch to starting the next one. So there shouldn't be a clunk, you know. So I kind of asked around on the open source discords. That's like, what's going on here? And everybody was just like, oh, that's just what, that's just what these training curves look like. Those all look like that. Don't worry about it. And I was like, oh, are you all using Trainer? Yes. Oh, well, there must be some bug with Trainer. And I was like, well, we also saw it in Learner [00:38:42]Swyx: and somebody else is like, [00:38:42]Jeremy: no, we've got our own Trainer. We get it as well. They're just like, don't worry about it. It's just something we see. It's just normal. [00:38:48]Swyx: I can't do that. [00:38:49]Jeremy: I can't just be like, here's something that's like in the previous 30 years of neural networks, nobody ever saw it. And now suddenly we see it. [00:38:57]Swyx: So don't worry about it. [00:38:59]Jeremy: I just, I have to know why. [00:39:01]Swyx: Can I clarify? This is, was everyone that you're talking to, were they all seeing it for the same dataset or in different datasets? [00:39:08]Jeremy: Different datasets, different Trainers. They're just like, no, this is just, this is just what it looks like when you fine tune language models. Don't worry about it. You know, I hadn't seen it before, but I'd been kind of like, as I say, I, you know, I kept working on them for a couple of years after ULM fit. And then I kind of moved on to other things, partly out of frustration. So I hadn't been fine tuning, you know, I mean, Lama's only been out for a few months, right? But I wasn't one of those people who jumped straight into it, you know? So I was relatively new to the kind of Lama fine tuning world, where else these guys had been, you know, doing it since day one. [00:39:49]Swyx: It was only a few months ago, [00:39:51]Jeremy: but it's still quite a bit of time. So, so yeah, they're just like, no, this is all what we see. [00:39:56]Swyx: Don't worry about it. [00:39:56]Jeremy: So yeah, I, I've got a very kind of like, I don't know, I've just got this brain where I have to know why things are. And so I kind of, I ask people like, well, why, why do you think it's happening? And they'd be like, oh, it would pretty obviously, cause it's like memorize the data set. It's just like, that can't be right. It's only seen it once. Like, look at this, the loss has dropped by 0.3, 0.3, which is like, basically it knows the answer. And like, no, no, it's just, it is, it's just memorize the data set. So yeah. So look, Jono and I did not discover this and Jono and I did not come up with a hypothesis. You know, I guess we were just the ones, I guess, who had been around for long enough to recognize that like, this, this isn't how it's meant to work. And so we, we, you know, and so we went back and like, okay, let's just run some experiments, you know, cause nobody seems to have actually published anything about this. [00:40:51]Well, not quite true.Some people had published things, but nobody ever actually stepped back and said like, what the hell, you know, how can this be possible? Is it possible? Is this what's happening? And so, yeah, we created a bunch of experiments where we basically predicted ahead of time. It's like, okay, if this hypothesis is correct, that it's memorized in the training set, then we ought to see blah, under conditions, blah, but not under these conditions. And so we ran a bunch of experiments and all of them supported the hypothesis that it was memorizing the data set in a single thing at once. And it's a pretty big data set, you know, which in hindsight, it's not totally surprising because the theory, remember, of the ULMFiT theory was like, well, it's kind of creating all these latent capabilities to make it easier for it to predict the next token. So if it's got all this kind of latent capability, it ought to also be really good at compressing new tokens because it can immediately recognize it as like, oh, that's just a version of this. So it's not so crazy, you know, but it is, it requires us to rethink everything because like, and nobody knows like, okay, so how do we fine tune these things? Because like, it doesn't even matter. Like maybe it's fine. Like maybe it's fine that it's memorized the data set after one go and you do a second go and okay, the validation loss is terrible because it's now really overconfident. [00:42:20]Swyx: That's fine. [00:42:22]Jeremy: Don't, you know, don't, I keep telling people, don't track validation loss, track validation accuracy because at least that will still be useful. Just another thing that's got lost since ULMFiT, nobody tracks accuracy of language models anymore. But you know, it'll still keep learning and it does, it does keep improving. But is it worse? You know, like, is it like, now that it's kind of memorized it, it's probably getting a less strong signal, you know, I don't know. So I still don't know how to fine tune language models properly and I haven't found anybody who feels like they do, like nobody really knows whether this memorization thing is, it's probably a feature in some ways. It's probably some things that you can do usefully with it. It's probably, yeah, I have a feeling it's messing up training dynamics as well. [00:43:13]Swyx: And does it come at the cost of catastrophic forgetting as well, right? Like, which is the other side of the coin. [00:43:18]Jeremy: It does to some extent, like we know it does, like look at Code Llama, for example. So Code Llama was a, I think it was like a 500 billion token fine tuning of Llama 2 using code. And also pros about code that Meta did. And honestly, they kind of blew it because Code Llama is good at coding, but it's bad at everything else, you know, and it used to be good. Yeah, I was pretty sure it was like, before they released it, me and lots of people in the open source discords were like, oh my God, you know, we know this is coming, Jan Lukinsk saying it's coming. I hope they kept at least like 50% non-code data because otherwise it's going to forget everything else. And they didn't, only like 0.3% of their epochs were non-code data. So it did, it forgot everything else. So now it's good at code and it's bad at everything else. So we definitely have catastrophic forgetting. It's fixable, just somebody has to do, you know, somebody has to spend their time training a model on a good mix of data. Like, so, okay, so here's the thing. Even though I originally created three-step approach that everybody now does, my view is it's actually wrong and we shouldn't use it. [00:44:36]Jeremy: And that's because people are using it in a way different to why I created it. You know, I created it thinking the task-specific models would be more specific. You know, it's like, oh, this is like a sentiment classifier as an example of a task, you know, but the tasks now are like a, you know, RLHF, which is basically like answer questions that make people feel happy about your answer. So that's a much more general task and it's a really cool approach. And so we see, for example, RLHF also breaks models like, you know, like GPT-4, RLHDEFT, we know from kind of the work that Microsoft did, you know, the pre, the earlier, less aligned version was better. And these are all kind of examples of catastrophic forgetting. And so to me, the right way to do this is to fine-tune language models, is to actually throw away the idea of fine-tuning. There's no such thing. There's only continued pre-training. And pre-training is something where from the very start, you try to include all the kinds of data that you care about, all the kinds of problems that you care about, instructions, exercises, code, general purpose document completion, whatever. And then as you train, you gradually curate that, you know, you gradually make that higher and higher quality and more and more specific to the kinds of tasks you want it to do. But you never throw away any data. You always keep all of the data types there in reasonably high quantities. You know, maybe the quality filter, you stop training on low quality data, because that's probably fine to forget how to write badly, maybe. So yeah, that's now my view, is I think ULM fit is the wrong approach. And that's why we're seeing a lot of these, you know, so-called alignment tacks and this view of like, oh, a model can't both code and do other things. And, you know, I think it's actually because people are training them wrong. [00:46:47]Swyx: Yeah, well, I think you have a clear [00:46:51]Alessio: anti-laziness approach. I think other people are not as good hearted, you know, they're like, [00:46:57]Swyx: hey, they told me this thing works. [00:46:59]Alessio: And if I release a model this way, people will appreciate it, I'll get promoted and I'll kind of make more money. [00:47:06]Jeremy: Yeah, and it's not just money. It's like, this is how citations work most badly, you know, so if you want to get cited, you need to write a paper that people in your field recognize as an advancement on things that we know are good. And so we've seen this happen again and again. So like I say, like zero shot and few shot learning, everybody was writing about that. Or, you know, with image generation, everybody just was writing about GANs, you know, and I was trying to say like, no, GANs are not the right approach. You know, and I showed again through research that we demonstrated in our videos that you can do better than GANs, much faster and with much less data. And nobody cared because again, like if you want to get published, you write a GAN paper that slightly improves this part of GANs and this tiny field, you'll get published, you know. So it's, yeah, it's not set up for real innovation. It's, you know, again, it's really helpful for me, you know, I have my own research lab with nobody telling me what to do and I don't even publish. So it doesn't matter if I get citations. And so I just write what I think actually matters. I wish there was, and, you know, and actually places like OpenAI, you know, the researchers there can do that as well. It's a shame, you know, I wish there was more academic, open venues in which people can focus on like genuine innovation. [00:48:38]Swyx: Twitter, which is unironically has become a little bit of that forum. I wanted to follow up on one thing that you mentioned, which is that you checked around the open source discords. I don't know if it's too, I don't know if it's a pusher to ask like what discords are lively or useful right now. I think that something I definitely felt like I missed out on was the early days of Luther AI, which is a very hard bit. And, you know, like what is the new Luther? And you actually shouted out the alignment lab AI discord in your blog post. And that was the first time I even knew, like I saw them on Twitter, never knew they had a discord, never knew that there was actually substantive discussions going on in there and that you were an active member of it. Okay, yeah. [00:49:23]Jeremy: And then even then, if you do know about that and you go there, it'll look like it's totally dead. And that's because unfortunately, nearly all the discords, nearly all of the conversation happens in private channels. You know, and that's, I guess. [00:49:35]Swyx: How does someone get into that world? Because it's obviously very, very instructive, right? [00:49:42]Jeremy: You could just come to the first AI discord, which I'll be honest with you, it's less bustling than some of the others, but it's not terrible. And so like, at least, to be fair, one of Emma's bustling channels is private. [00:49:57]Swyx: I guess. [00:49:59]Jeremy: So I'm just thinking. [00:50:01]Swyx: It's just the nature of quality discussion, right? Yeah, I guess when I think about it, [00:50:05]Jeremy: I didn't have any private discussions on our discord for years, but there was a lot of people who came in with like, oh, I just had this amazing idea for AGI. If you just thought about like, if you imagine that AI is a brain, then we, you know, this just, I don't want to talk about it. You know, I don't want to like, you don't want to be dismissive or whatever. And it's like, oh, well, that's an interesting comment, but maybe you should like, try training some models first to see if that aligns with your intuition. Like, oh, but how could I possibly learn? It's like, well, we have a course, just actually spend time learning. Like, you know, anyway. And there's like, okay, I know the people who always have good answers there. And so I created a private channel and put them all in it. And I got to admit, that's where I post more often because there's much less, you know, flight of fancy views about how we could solve AGI, blah, blah, blah. So there is a bit of that. But having said that, like, I think the bar is pretty low. Like if you join a Discord and you can hit the like participants or community or whatever button, you can see who's in it. And then you'll see at the top, who the admins or moderators or people in the dev role are. And just DM one of them and say like, oh, here's my GitHub. Well, here's some blog posts I wrote. You know, I'm interested in talking about this, you know, can I join the private channels? And I've never heard of anybody saying no. I will say, you know, Alutha's all pretty open. So you can do the Alutha Discord still. You know, one problem with the Alutha Discord is it's been going on for so long that it's like, it's very inside baseball. It's quite hard to get started. Yeah. Carpa AI looks, I think it's all open. That's just less stability. That's more accessible. [00:52:03]Swyx: Yeah. [00:52:04]Jeremy: There's also just recently, now it's research that does like the Hermes models and data set just opened. They've got some private channels, but it's pretty open, I think. You mentioned Alignment Lab, that one it's all the interesting stuff is on private channels. So just ask. If you know me, ask me, cause I've got admin on that one. There's also, yeah, OS Skunkworks, OS Skunkworks AI is a good Discord, which I think it's open. So yeah, they're all pretty good. [00:52:40]Swyx: I don't want you to leak any, you know, Discords that don't want any publicity, but this is all helpful. [00:52:46]Jeremy: We all want people, like we all want people. [00:52:49]Swyx: We just want people who like, [00:52:51]Jeremy: want to build stuff, rather than people who, and like, it's fine to not know anything as well, but if you don't know anything, but you want to tell everybody else what to do and how to do it, that's annoying. If you don't know anything and want to be told like, here's a really small kind of task that as somebody who doesn't know anything is going to take you a really long time to do, but it would still be helpful. Then, and then you go and do it. That would be great. The truth is, yeah, [00:53:19]Swyx: like, I don't know, [00:53:20]Jeremy: maybe 5% of people who come in with great enthusiasm and saying that they want to learn and they'll do anything. [00:53:25]Swyx: And then somebody says like, [00:53:25]Jeremy: okay, here's some work you can do. Almost nobody does that work. So if you're somebody who actually does the work and follows up, you will massively stand out. That's an extreme rarity. And everybody will then want to help you do more work. [00:53:41]Swyx: So yeah. [00:53:41]Jeremy: So just, yeah, just do work and people will want to support you. [00:53:47]Alessio: Our Discord used to be referral only for a long time. We didn't have a public invite and then we opened it and they're kind of like channel gating. Yeah. A lot of people just want to do, I remember it used to be like, you know, a forum moderator. [00:54:00]Swyx: It's like people just want to do [00:54:01]Alessio: like drive-by posting, [00:54:03]Swyx: you know, and like, [00:54:03]Alessio: they don't want to help the community. They just want to get their question answered. [00:54:07]Jeremy: I mean, the funny thing is our forum community does not have any of that garbage. You know, there's something specific about the low latency thing where people like expect an instant answer. And yeah, we're all somehow in a forum thread where they know it's like there forever. People are a bit more thoughtful, but then the forums are less active than they used to be because Discord has got more popular, you know? So it's all a bit of a compromise, you know, running a healthy community is, yeah, it's always a bit of a challenge. All right, we got so many more things [00:54:47]Alessio: we want to dive in, but I don't want to keep you here for hours. [00:54:50]Swyx: This is not the Lex Friedman podcast [00:54:52]Alessio: we always like to say. One topic I would love to maybe chat a bit about is Mojo, modular, you know, CrystalLiner, not many of you on the podcast. So we want to spend a little time there. You recently did a hacker's guide to language models and you ran through everything from quantized model to like smaller models, larger models, and all of that. But obviously modular is taking its own approach. Yeah, what got you excited? I know you and Chris have been talking about this for like years and a lot of the ideas you had, so. [00:55:23]Jeremy: Yeah, yeah, yeah, yeah, no, absolutely. So I met Chris, I think it was at the first TensorFlow Dev Summit. And I don't think he had even like, I'm not sure if he'd even officially started his employment with Google at that point. So I don't know, you know, certainly nothing had been mentioned. So I, you know, I admired him from afar with LLVM and Swift and whatever. And so I saw him walk into the courtyard at Google. It's just like, oh s**t, man, that's Chris Latner. I wonder if he would lower his standards enough to talk to me. Well, worth a try. So I caught up my courage because like nobody was talking to him. He looked a bit lost and I wandered over and it's like, oh, you're Chris Latner, right? It's like, what are you doing here? What are you doing here? And I was like, yeah, yeah, yeah. It's like, oh, I'm Jeremy Howard. It's like, oh, do you do some of this AI stuff? And I was like, yeah, yeah, I like this AI stuff. Are you doing AI stuff? It's like, well, I'm thinking about starting to do some AI stuff. Yeah, I think it's going to be cool. And it's like, wow. So like, I spent the next half hour just basically brain dumping all the ways in which AI was stupid to him. And he listened patiently. And I thought he probably wasn't even remember or care or whatever. But yeah, then I kind of like, I guess I re-caught up with him a few months later. And it's like, I've been thinking about everything you said in that conversation. And he like narrated back his response to every part of it, projects he was planning to do. And it's just like, oh, this dude follows up. Holy s**t. And I was like, wow, okay. And he was like, yeah, so we're going to create this new thing called Swift for TensorFlow. And it's going to be like, it's going to be a compiler with auto differentiation built in. And blah, blah, blah. And I was like, why would that help? [00:57:10]Swyx: You know, why would you? [00:57:10]Jeremy: And he was like, okay, with a compiler during the forward pass, you don't have to worry about saving context, you know, because a lot will be optimized in the backward. But I was like, oh my God. Because I didn't really know much about compilers. You know, I spent enough to kind of like, understand the ideas, but it hadn't occurred to me that a compiler basically solves a lot of the problems we have as end users. I was like, wow, that's amazing. Okay, you do know, right, that nobody's going to use this unless it's like usable. It's like, yeah, I know, right. So I was thinking you should create like a fast AI for this. So, okay, but I don't even know Swift. And he was like, well, why don't you start learning it? And if you have any questions, ask me. It's just like, holy s**t. Like, not only has Chris Latner lowered his standards enough to talk to me, but he's offering me personal tutoring on the programming language that he made. So I was just like, I'm not g

While it may still be early fall, the time to figure out your holiday travel is here. From flights to hotel stays to finally taking that trip to Europe to visit the Christmas markets, travel costs can add up. On the latest episode of PennyWise, host Nat Cardona is joined by Sally French  of NerdWallet with tips when to start booking your travel for the holidays. Read more on NerdWallet here! About this program Nat Cardona is host of PennyWise as well as Lee Enterprise's true-crime podcast Late Edition: Crime Beat Chronicles. Lee Enterprises produces many national, regional and sports podcasts. Learn more here. Episode transcript Note: The following transcript was created by Adobe Premiere and may contain misspellings and other inaccuracies as it was generated automatically: Welcome to Pennywise Lee Enterprises podcast. I'm your host, Nat Cardona.  It may be the very start of fall, but if you plan to travel for the holidays, now is the time to do your research and pull out your wallet. We have Nerdwallet travel rewards expert Sally French here with some tips to save on holiday travel this year. Okay, so talking about holiday travel, you know, as people look towards the next couple of months, you've got Thanksgiving and then it seems around Christmas, people are flying here, they're everywhere. Is there any advantage to last minute flights? I know that last year that was a big thing. Yes. In general, booking last minute flights is not something you should bank on. We typically see last minute hotel deals because hotels just want to fill up whatever available rooms that they have. But that's not really the case with airfares because people would be desperate to book a last minute flight. We actually see airfares get more expensive last minute. So I don't want to completely write off the idea that you won't find a great last minute airfare. But the reality is you should plan to book your airfare now, especially for holiday travel there. So a lot of that would entail tracking prices. So let's go into that. Yes. So there are a number of apps that make it pretty easy to track your prices. One of the great ones is Google flights. And Google flights has a bunch of individual features that can make this helpful. So Google flights has a calendar display that will show the flight prices across all the airlines for any day of that month. This is great for people who want to fly, let's say, for the holidays, but are flexible. If they fly out on Tuesday or Thursday, they don't really care. Google Flights Calendar View makes this really easy. Another thing that you can do is if you know you want to fly a specific airport, which is pretty common. If you're going to visit family for the holidays, you can enter in that airport versus your home airport. And Google flights will send you an alert when it sees that airfares are cheaper than usual to that destination. I am a huge believer in that calendar layout that you just mentioned. There's nothing more beautiful in my like, planning mind when it's you know, this says 380 on Tuesday and then a week later on a Friday, it's whatever, 300. I just love the breakdown. It's just all it's so neat how they do it right. And, you know, you would be so surprised at how much the price is different just by adjusting your flight by a couple of days. And there are just a lot of people who automatically assume they're going to fly out one day because they think that's most convenient for them. But when you realize maybe extending by one day could save you $200 times all for people in your family, then this can be a huge savings. So don't overlook that calendar view and see if you can change your trip gates by just one or two days to possibly save money. Sure. And I just kind of want a ground back here. So for Thanksgiving, when would be a good time to book for that long weekend? Yes. So definitely I would recommend booking that Thanksgiving travel by mid-October at the latest. However, Nerdwallet in general recommends booking travel 1 to 3 months out, and the reality is for the holidays, people should probably. But even earlier, we know that more people are planning to travel for the holidays this year versus last. And holidays are one of the busiest days to fly. In fact, in each of the past four years, the Sunday after Thanksgiving was the most crowded day to fly of that week. So you want to avoid traveling that week. And perhaps most interestingly, is that every year in the past four years, aside from 2020, the Sunday after, after Thanksgiving was also the busiest single day to fly of the entire year. Well, my gosh, everyone. I mean, that's the classic camera shot. And see ABC, CBS, whatever that reporter in the airport. And it's utter chaos happening in the background as people try to go home for Monday to start. Right? Right. Everyone thinks that they need to fly homeless sunday after thanksgiving. But that's exactly it is. Use that calendar view and realize that there's going to be way lighter traffic if you change your travel itinerary by just a couple of days. In fact, the best day to travel post holiday is giving. Tuesday has among the lightest crowds. So is the Wednesday after and so does Black Friday. If you want to do that Thanksgiving dinner and just fly home the next day, that's also a great day to travel. Okay. And then as far as the Christmas holiday goes, that end of December or last week or two. What's the suggestion with purchasing or booking your flight any time soon? Yes. Again, it's never too early to book Christmas travel, especially when we're talking about right now. It's just, again, a busy day to fly. That said, if you want to avoid crowds, travel on Christmas Day itself is a pretty light day. So if you are in a pinch, you've booked last minute, you don't know what to do. Maybe except that you travel on Christmas Day. It'll be less crowded, probably cheaper, and everyone will probably be a lot more merry on the flight that day anyway. Just fly and show up for dinner. That's fine. You think that's your only obligation? I agree. Open presents at home, fly and then have family dinner over at your next place. And make it all in. Let's talk about European travel. Whether you're a family or you're being adventurous. This one's often a little bit more difficult because not necessarily direct flights. You usually have two legs of a flight that you have to do. Let's get into that. Yes. So we're seeing a lot of interest in European travel, especially for the holidays. You can go abroad to the Christmas market. People love that. It's so magical. But the reality is, even though we're seen flight prices drop domestically, we're seeing a high interest in European travel. Everyone wants to go there to make up for lost time during the pandemic. So what we're seeing is that European travel is past its 2019 travel levels and the best time to book an international flight. So that would be the European flight from the U.S. is 72 days or more before departure. Once it's within that ten weeks window, we see airfares start to rise significantly. Unfortunately, we are sort of past that booking window for holiday travel to Europe. But again, that's where you start to employ those tricks, like being open to the exact day that you travel, maybe booking that basic economy airfare instead to save money. Sure. And that just goes back to this next point for the article. Being flexible, we've really kind of beat the dead horse on that throughout everything that we just talked about. So, I mean, anything you want to add as far as holiday travel goes as people are planning the next couple of months? One of the big things that I think some people almost forget about traveling to Europe is that Europe is not celebrating Thanksgiving. So if you do want to go to the Christmas markets, a lot of people want to go for Christmas, but consider traveling for Thanksgiving because you won't see that huge spike in not only airfares but hotel prices, because for them, the end of November is pretty much just a normal day. So stay home for Christmas and maybe consider doing in Europe for Thanksgiving. Yeah. Kick off the holiday season that way. That's that's a really good tip. Yeah. And so be that Christmas. Shopping at the Christmas markets. And then you have your gifts ready to go. Yeah, go to Germany. That would be really cool. That's on my bucket list. Same.Support the show: https://omny.fm/shows/pennywiseSee omnystudio.com/listener for privacy information.

Hey dear ThursdAI friends, as always I'm very excited to bring you this edition of ThursdAI, September 21st, which is packed full of goodness updates, great conversations with experts, breaking AI news and not 1 but 2 interviewsThursdAI - hey, psst, if you got here from X, dont' worry, I don't spam, but def. subscribe, you'll be the coolest most up to date AI person you know!TL;DR of all topics covered* AI Art & Diffusion*

This episode continues the conversation with Dianne Berg, author of What's behind our enduring fascination with wives and mothers who kill. In this episode, Nat Cardona and Diane talk about what causes wives and mothers to commit murder and how the public, judicial system and medical fields contribute and/or react to these criminal events. To listen to the first half of the interview with Dianne, click here.  To learn more about Dianne Berg, click here.  Episode transcript Note: The following transcript was created by Slack and may contain misspellings and other inaccuracies as it was generated automatically: Welcome to Lee Enterprise's Late Edition Crime Beat Chronicles. I'm your host Nat Cardona. In this episode, we're continuing the exploration of a niche area of true crime stories, the obsession that fans seem to have with killer wives and mothers. We're back with Diane Berg, a professor at Clark University and author of the article What's behind our enduring fascination with wives and mothers who kill? She is very much an expert on this complex topic. If you haven't listened to the first episode, go back and listen to the first half of my interview, please. Otherwise, we pick up the conversation back up by discussing some causes of what makes mothers and wives kill. I'm a mother myself and I'm, I'm actually pregnant. So I'm gonna have a baby in four weeks. I'm actually pregnant, but I'm having a baby in four weeks. Thank you. So I'm kind of like, you know, going through all these things and, you know, very much in the, you know, what makes me different from these other women who have done this historically or in more recent history. But the thing that comes to mind is there is just something so grabbing about when women do this, because you carry the child for so long and you birth the child and it's so much more intimate than the father who's removed and can kind of clean his hands in the sense of when there is a murder, you can go. You know, and that's because he's not involved. So in the natural process of pregnancy and birth. So, yeah, when there are these women historically or modern day that do this, it's like, well, you know, you just sit there and go, how how could this happen? How could this happen? And you do. The next point I want to get at is the openness nowadays that we have about talking about postpartum depression, because there seems to be a link with that postpartum psychosis. And you mentioned it's Lynsey Clancy who's kind of the most recent with that. So in your research and I is something you mentioned, I just want to clarify. Have you you've seen a difference between, let's say, 20, 30 years ago media coverage and nowadays media coverage of like like just jump into that. Okay. I mean, I kind of want to take those in order, if I may. So, yeah. First, going back to what you were talking about, how okay, when a man does it. Yes, that's terrible and bad and they're they're bad people. But when a woman does it, when a mother does it, especially, there's all this kind of language of the unnatural and the monstrous. And again, going back to, you know, right now, I've been rereading Euripides Medea all week to get ready for this class, because Medea is like the her murderous mother. Right. And a lot of times these these early modern mothers who kill their children, who, as you point out statistically are fewer than men who kill their children. It is then is now like men commit way more domestic violence than women do. But women do it. It gets more attention. And it's because of this unnatural list. Right? Women mothers are supposed to be, as you say, it's the natural process whereby we actually think we incubate the child. And there's a lot of that kind of language of like, how could she like a bloody like a bloody tiger? A tiger wouldn't do a thing like this. A snake wouldn't do a thing like this. The child that she nursed in her body for 40 weeks and fed with her breasts, and there's all this kind of language of like how unnatural this is that you would destroy your own creation in this way. And I think that's really deep. Obviously, that plays it. I think at a really macro level, it plays into fears about like God destroying the earth. But I think on the more kind of social and cultural level, it just flies in the face of everything that women are supposed to be. We're supposed to be kind and gentle and nurturing and giving and selfless, and all of these things are intimately tied up with our concept of the mother, right? The mother just gives and gives and gives. The mother is is a a you know, a vessel that never runs dry. Right. That's what it's supposed to be. And so if a mother not only fails to deliver on all those counts, but actually turns on her children and even destroys them, this like, taps into, I think, some really elemental fears. And I think that's why we're so interested in it. And I think that's why we stay interested in it. And as a mother, I'm a mother as well. I think it's it strikes a particular chord because it's that on the one hand, yes, there's that schadenfreude or. Right, There's that. Well, I didn't do that. You didn't do that? Yeah. My, my, my kid cried all day, too, but I didn't, you know, throw him out a window. There's that. But there's also the more interesting thing is that on some level, I think anyone who has ever had to care for a small child, an infant, especially if you have recently given birth and your own body and your own mind are still you know, you're not yourself yet. I think anyone who's been in that position has been that exhausted, that frustrated, felt that inadequate, felt how hard it is to live up to all those things. I just enumerated that mothers are supposed to be can understand how it happens. And that's terrifying that there but for the grace of God go. I write that if I hadn't had my support network, if I hadn't had my level of education, if I hadn't known how to find help. Right. That the I might have done a thing like that. And I think that's why we can't look away. I think that's a big part of it. Yeah, that is actually one of the notes that I was just rereading here is that it's hard to make peace with that because, you know, whether it be it's like take guys who who commit murder, there's often the you find out that they had childhood trauma they were abused but then there's plenty of people say, well, so was I. But I didn't it you know kill five people. It's kind of the same thing here. It's there's there's so many women who deal with postpartum depression and then it's very easy to say, well, I didn't do that and I would never think of doing that. But it's exactly what you say. It's when you stare in the face, it's like, well, it's a really thin line of what, you know, the possibility of it. It's just it's a weird thing to kind of I just grapple with an iron out. Yeah. And if there's actually, you know, things out of whack that would respond to medication, this isn't just even a this goes beyond just being exhaustion of being overwhelmed, feeling inadequate, all of which are incredibly legitimate things that, you know, I certainly experienced as a mother of three children. But then you actually add in some sort of, you know, chemical balance or mental illness or, you know, various factors. Women have no resources. They have no help, they have no money, no one cares about them. We have a government that cares very much about fetuses or at least claims to care very much about fetuses. It doesn't care so much about babies or their mothers. You know, if they wind up needing extra help. So in answer to your question about the sort of coverage of these things, I do think and I hope I'm not being optimistic, I do think that I'm seeing a shift in the coverage. It's not that there wasn't any mention when the when the Yates murders happened in 2001 or maybe it was. Yes, it was one. There was talk of the fact that this woman hadn't for one reason or another, she didn't get the care that she needed. And there were a lot of factors at play there. She and her husband were evangelical Christians. They were part of this quiver full movement, which basically they want you to have as many children for Jesus as possible. It's God's will. You just keep having children as long as God sends them to you. She was homeschooling them all she had already had. I can't remember now if it was after her second or third child. She'd had a pretty serious case of postpartum depression to the point where her her gynecologist said she shouldn't have any more children. This is going to happen again. It's going to get worse. But they had, I think, two more children after that. Anyway. She was being insufficiently monitored. I mean, there was a lot of talk about the fact that this woman was, in her own way, a victim. And there was a lot of finger pointing at the husband. His name was Rusty. Rusty Yates for continuing to, you know, have children with her and allowing her to homeschool the children. She had five children under the age of seven and, you know, wasn't taking her medication. And there was a lot going on there. So it wasn't that the coverage of her was completely unsympathetic, but there was an awful lot of she's a monster. She she couldn't have done it if she because the insanity defense, they're doing same thing with Lindsey Clancy. The prosecution is saying, well, no, no, she can't have been insane because she knew what she was doing. She was able to make a plan and carried out both Lindsey Clancy, Andrea Yates and Margaret Robinson, for that matter, wait until their husbands were away and they knew they had a window in order to commit the crimes. And the prosecution in Clancy's case and in this case have argued that that's impossible, because if she was insane, she couldn't have made a plan, she couldn't have carried it out, etc., etc.. Of course, we know that's not true. People, people suffering from psychosis can commit, make plans and carry them out all time. And it was initially charged with first degree murder and found guilty. And the jury didn't. They could have given her the death penalty. They they didn't, but they sent a sort of life in prison initially. And then they appealed several years later using an insanity defense, which which succeeded I don't actually think I don't have a crystal ball, but I think that the passage of 22 years is going to have made a difference in the Lindsey Clancy case. She is, you know, remains in a psychiatric facility. I, I think that there would be a great outcry if she actually were brought to trial for murder charges. And I think that there's been so much more in the press about postpartum depression and postpartum psychosis in the wake of Lindsey Clancy's. I mean, it's a crime. I but I hesitate to use that word. But in the wake of this very tragic incident, there's been so much more coverage of that and a lot more people coming forward, a lot of kind of op ed pieces, people saying, I have postpartum depression. This is what it's like. I you know, again, that could have been me. So I feel like there's a broader discussion about it. And you know, it was just I think two weeks ago that the government approved this medication for women with postpartum depression to be more widely distributed, which I feel is like a huge step forward. And, you know, I mean, I can talk obviously, I can talk about this all day. At the root of this, the fact that it's 2023 and we're only now it seems like having a really serious conversation about this just speaks to the degree to which women's issues are always pushed down the list. Right. Women's health, women's wellbeing, women's mental health. It's always bumped down the list. And of course, again, we're going back to mothers, right? Mothers aren't supposed to need anything. We're supposed to take care of everyone all the time and no one takes care of us. So I feel like, yes, progress. But wow, it's, you know, the 21st century. I know. We need to take a quick break, so don't go too far. It's fascinating to me. And I wonder if it's repeatedly fascinating to you just if this small increment of time is where we're starting to see that little switch turn to, you know, more in favor of the other possibilities that could be at play here. But 2023 compared to, you know, 16, 16, we're not you know, how how in the media things are typecasting with these types of crimes. Is it for it to be so not that much different? Is, oh, you know, how many how many things can you count, How many topics can you say are like that? Yeah, I mean, that's a great money generator. So I mean, I mean. Margaret Vincent, you know, I mean, she said ultimately that she had been, you know, she had fallen under evil influences and basically the devil made her do it. And you know, there's this great woodcut on the cover of the pamphlet about her, which is called The Pity Lost Mother Goes on, but we'll just call it a pity loss. Mother, for the sake of brevity that shows her with her children and she's strangling them and the devil is standing behind her. And he's got horns and claws and and he's he's basically making her do it. And after she had been in prison when she was apprehended, she said that she had been, you know, laboring under this terrible delusion. And there had been, like Roman Catholic neighbors who were trying to persuade her to become a Catholic. And that's like a bad influence at this time. And once she had been spoken to at length by a proper, you know, Protestant minister, she repented and recanted. And obviously she had to be hanged for it, but she at least was able to repent and make her peace. And so, like the the the end game of the pamphlet is that since she was truly repentant, you know, maybe she can be saved, right? Like, her body has to die, but maybe her soul can still be saved. But the important part is the repentance, right? Kind of say, yes, I did that. Yes, it was wrong in those days, you know, like, you know, I like to say yesterday's demonic possession might be today's postpartum psychosis or the other way around. Right. That, you know, these behaviors, there's got to be some kind of a just be an explanation as to be a reason. So, you know, if it's that, you know, I have a chemical imbalance and I need to, you know, take medication and be treated for it or like, oh, like I was actually possessed by it by a demon when this happened. There has to be some kind of resolution and you have to be sorry. Do you know off the top of your head with Lindsay Clancy if she said anything like in. Yeah, she said at her arraignment or I guess her she didn't speak at her arraignment, but her her counsel said that she said that she heard a voice in her head when her husband was gone. She sent her husband out on an errand. He was working from home because she was that she was sick. He had been working from home and she was doing well, apparently seemed to be doing well and hadn't had a good day with the children playing outside in the snow. And he was working from his home office and she texted him, recalled him and said, let's get takeout. And he said, Yeah. And so he sent him to a place that was about a half hour's drive away. And she said she heard a voice in her head telling her that she had to do it now, because if she didn't do it now, she wouldn't have another chance. That sounds pretty psychotic to me right? Andrea Yates said something pretty similar that, you know, she she knew that she would have to do it. Now. This was the chance and she had to take it. And something would have prevented them from doing these things. If, you know, if they hadn't taken these these opportunities, created, you know, these opportunities and and taken them. That's all we've really heard from her thus far. But apparently, she you know, she told her husband that you've done it. The husband has argued very movingly. I think that she deserves compassion and not condemnation, and that if he can forgive her, then, you know, then the people, the people on Facebook comment threads should probably, you know, dig deep and either find compassion or find the ability to get off that Facebook comment thread. Right. Oh, my gosh. Amen to that. I mean, and that that kind of brings me to my my parting thoughts here was how you ended your piece was there. It seems to be that there's two lanes of thought here when someone's digesting all of the true crime that they can, especially when it comes to wives of mothers. It seems it's the what did you call it, the shattered fruit. I can't. It just means that kind of it's a nasty word and there isn't a word in English that means this. Exactly. It basically means that the sort of pleasure, often a kind of guilty pleasure. We take in the misfortunes of others. But yeah, exactly. Exactly. So, you know, when when the Lindsay Clancey situation occurred, I know my immediate thought was like, oh my goodness, like, you know, social media, like, that's going to be an absolute pit of despair. You know, if you do the things that people the people's hot takes. Right. But but I feel like that visceral reaction that people have where they feel like they have to get in there and say, look, she's a monster and she should go to hell. Oh, those poor little angels, etc., etc.. That's very much part and parcel of that. Pushing it away. That can't be me. I'm not like that where you know, I'm not like that. I'm not a person who with my children, I'm not a person who would kill my husband. I'm not a person who, you know, would do X, Y, Z, terrible thing. And so I have to jump in here and do this very kind of like performative public condemnation of this thing to kind of distance myself from it, but also kind of reassure myself that, you know, that's not me, I'm different than that. I'm better than that. Right? That's actually the flip side of things. The other lane is what you had mentioned is that the appeal might lie in the fact that, oh, that light bulb thing, we might be capable of these things. It's kind of funny. And the thought that immediately came to me and this is always how I've felt about true crime, and especially on this topic, is like it's better to what is it the devil you know versus the devil you don't know, right? Yeah, that's just right. Well, I've been, you know, again, I've sort of been down this kind of classical tragedy rabbit hole this week. You know, I come back to what do we get out of this kind of stuff, Like, you know, here you are. You confess to being like you're constantly devouring this material, right? I do it. Lots of people I know do it. True crime, you know, has been so massive in recent years. Right. People just devour this stuff. I mean, it's always been very popular. It does seem like it's really kind of having a moment culturally. There's what we get from this stuff is is catharsis, Right? I mean, it's the same thing as as classical tragedy, right? We we watched the terrible thing happen, but the terrible thing hasn't happened to us. Now, if we're talking about a drama, if we're talking about Medea or Oedipus Rex or even Hamlet, yet the body, you know, the bodies are littering the stage and all these terrible things have happened. We have the the purging of pity and terror that comes. But no one has actually died. Nothing terrible has actually happened. We leave the theater feeling kind of scoured out and then we go and we we get a coffee and we chat about it. Right. But with the true crime stuff, someone has died. Something a real tragedy has occurred. And yet I still feel like it's that catharsis that you know, we see it, we watch it. You know, people watch to watch these trials when they can. Right? They need to see how it ends. And then they can walk away from it and it hasn't happened to us. Mhm. Right. We sort of had the, the, the purging of pity and terror but something terrible really has happened and still it's not like when a play is over and now the play is over. As you say, these stories happen over and over again. It's so, so accessible. Yeah, yeah, yeah. It's, I mean yeah that, yeah, yeah. And then there's that other thing. Go ahead, Go ahead. No, no, it's just kind of. I just feel like this. This appeal is kind of timeless, and it speaks to something in in like, the human condition. And I'm not sure it's a very nice thing in the human condition, but it certainly is. There. That's my thinking. Exactly. Yeah. It really it all ties in together. It's just. Yeah, definitely something to chew on, to use. I mean, what's next for you in this grand scheme of things. And I think going forward, I mean that's kind of a really open ended question. I, I mean, I'm excited for this, this course. I'm going to start teaching on Monday, which is again, we're going to start with with Medea and we're moving on to so then we're moving on to some everything inside of mothers and we're going to move on to some some women who kill, too. I don't know. We're moving on to petty tyrants after that. So we'll have some texts about fathers who abuse their authority by killing their wives and or children. And we're going to end up with wives who who kill their their husbands sort of petty traitors. And I will be putting kind of early modern texts in conversation with more modern cases throughout the semester. So I think it's going to be really fun and interesting. And I'm hoping my my I have I have every intention of writing a book, which is I have a title. It's going to be the same title as my seminar are actually Pulp Pulp nonfiction, Oh, True Crime and Fake News and Early Modern England. So that's that's my next big project. I'm currently working on a of what I think is going to be more public facing piece which is kind of different but kind of not. It's actually about Barbie and Paradise Lost. Milton's Paradise Lost, which I think is kind of interesting, is sort of Barbie Land as a kind of Eden and Ken as a kind of Adam figure. But that's that's what I'm kind of working on right now on the side. We'll see what happens with that. But yeah, I think going forward, you know, it's going to kind of be more murder and mayhem for me. I really safe to say that's the life, right. I hasten to add, I'm actually a very nice person. And it's funny that I know. I mean, I have three children of my own. And I think they they think it's they're a little bemused that this is kind of like my my reputation. I was once at a conference and I was introduced to someone and he said, Oh, you're the infanticide woman. And I was like, Please don't call me that. But, you know, yeah, I have children, I have children, I have dogs and cats. I, you know, I, I'm, I'm a nice person. I swear to God, you know, I'm vegan. I've been begging for for a very long time. So, yeah, this is all purely intellectual, I assure you. Yeah. Oh, my goodness. Yeah, Well, these are wrenching, all of that. And is there any way, if a listener is interested in following you and is not obviously at one of your classes at university, one of your son winners at university, is there a way that people can follow what you're doing or publishing. A I'm not really very I I'm not on Twitter or whatever it's called this week, so I have to go. Yeah, right. Perhaps going forward at this at this point, mainly, you know, just through, through what I publish. Yeah. And up to Clark University. I, I teach English at Clark University in Massachusetts. Okay. Okay. So Google search, people. And that is that, my friends, special thanks to Diane Berg for joining the show and then giving us a look at what's mesmerized true crime fans for centuries. Thanks for listening to Late Edition Crime Beat Chronicles. Hit that subscribe button so you don't miss what's coming next. See you later on.See omnystudio.com/listener for privacy information.

‍(If you DO want to work with Crimson Media Group, be sure to mention this podcast/ episode!)Reach out to Crimson Media Group here: https://thedentalmarketer.lpages.co/crimsonmediafree/Shane always gives all our listeners a: Free Marketing Analysis (just mention our podcast name)‍‍In this week's Monday Morning Marketing episode, join us as we cultivate a deeper understanding of the world of Google Ads with our guest, Shane Simmons. Discover how a laser-focused Google Ad campaign will help you truly stand out in today's fiercely competitive market. We unravel the art of campaign precision, emphasizing the value of targeting specific keywords related to one core service, as opposed to casting a wide net. Shane guides us through the essential steps, from implementing call tracking to calculate your cost per patient, to setting a reasonable budget (hint: expect to invest AT LEAST $1,000 per month for significant impact, depending on your location). We dissect the metrics that matter most, including cost per click, click-through rate, and conversion tracking. Plus, learn the secret sauce of creating a dedicated landing page with key details prominently displayed. ‍Tune in and gain the insights you need to dominate your Google Ads game and elevate your marketing strategy to new heights. Don't miss out on this invaluable episode with Shane!‍You can reach out to Shane Simmons here:Website: https://www.crimsonmediagroup.com/‍Other Mentions and Links:Google MapsGoogle AdsGoogle Local Service AdsFacebook Ads‍If you want your questions answered on Monday Morning Marketing, ask me on these platforms:My Newsletter: https://thedentalmarketer.lpages.co/newsletter/The Dental Marketer Society Facebook Group: https://www.facebook.com/groups/2031814726927041‍Episode Transcript (Auto-Generated - Please Excuse Errors)‍Michael: Hey, Shane. So talk to us about Google ads. How can we utilize this or what advice suggestions or methods can you give us that will help actually attract new patients through Google ads? Shane: Yeah, michael, so with google ads, these are great because these are people, Doing a search looking for a dentist or a dental service in their community So you're really looking at for the most part pretty serious in market patients at this point, so With that being said the number one piece of advice I could give on launching a successful google ads campaign is making sure that your Targeted campaigns are built out individually, and what I mean by that is, you know, if you're running a an emergency campaign, you you should be specifically bidding on and targeting the emergency dental terms in the area.So emergency dentist, same day dentist, 24 7 dentist, whatever the case is. Having a campaign geared just around emergency dentistry and then if you're wanting to run just a general patient campaign to build the hygiene, um, side of the practice, then those should be general terms that people look at such as, dentist near me, best dental office, dental cleaning, so on and so forth.So that's the number one thing that I think I've seen in the past is a lot of people try to Build one campaign and then bid on like all the different keywords that are general related. And if anybody who's ran Google ads knows, it can get really expensive really quick if you do it that way. Google ads isn't necessarily like, running an ad on Facebook or boosting a post on Facebook.Uh, this is something that you have to have a really, solid strategy in. Otherwise, you're going to blow your budget really quickly and not get any results. So... That's number one. I mean, for us, every new client that we take on, we run two campaigns right out the get go.First is a general campaign. Second is that emergency campaign. And especially for you startups out there listening to this, the emergency campaign is great because a lot of times what we have found is, you're going to have some openings in your schedule as a startup. So people come in, they feel that real personalized service that you're able to give them.Thank you. You get them out of pain get them taken care of and then what a lot of our startups have found is Many of those patients end up getting put on the hygiene schedule afterwards because they had you know Such a great experience. So that's something for you startup listeners out there I would highly recommend right out.The gate is launching an emergency campaign. So general campaign being second now One question we get is, all right, when we run these campaigns, what's next? Where are we sending them? Are we sending them to the website? You need to have a dedicated landing page for every ad campaign that you're running when it comes to Google.So if it's an emergency campaign, having an emergency specific landing page that you're sending that traffic. And what makes up a good landing page? First and foremost, the information that the patient's looking for they shouldn't have to scroll down to get to it when they go to the page So if you have an offer for a 79 emergency exam Then you need to have that right at the top of the landing page with a call to action button to call now Schedule online, whatever the case is have the call to action with the offer right there Um, the next thing you need to make sure that that landing page has is the proper tracking in place And this is kind of a theme every time I do these, uh interviews with you michael But it's so important is making sure you have call tracking set up so you can know where those patients came from which ad which campaign And whether or not they booked or not and then also having the proper form tracking too.So if you're collecting that patient's information on that landing page, or maybe they're entering their phone number, their name, their email address, and you know, what insurance they have, for example, um, you need to make sure that you have the proper tracking set up for that because Google ultimately in these ad campaigns is going to look at.How people clicked on the ad, what they did when they clicked on the ad, and whether or not they converted. So if you don't have conversion set up properly, it's going to mess up your whole bidding strategy, and you're going to end up never being able to actually dial in, a cost for new patient.ultimately when you're running an ad, after 3, 4, 5 months of doing that, you want to be able to get to a point where you say, All right, I know if we put two thousand dollars in ads. We're going to get 10 new patients out of that and that puts our cost per new patient at 200 so that's where we have to get to but if we don't have that proper tracking in place You're not going to ever be able to get to that because the campaign and Google and the algorithm is not going to know who responded to the ad and who didn't. So that's the second thing. Landing page and making sure that you have that proper tracking set up there. And then the third thing is come in with a realistic budget. too many times... I'll see practices say, you know, they're in southern california I spent you know, five hundred dollars a month in google ads, but i'm not getting nothing You're not going to get anything with five hundred dollars a month in google ads It's just the nature of that platform, you need to be able to dedicate at least 1200 a month in google ads spend if not more than that, especially if you're targeting things like implants and higher end procedures.so just going into it with realistic expectations and knowing that it's an investment and there's other people out there bidding for the same keywords. So you have to make yourself stand out one way or another. And that's where good ad copy comes into play, proper tracking. And of course, the landing page is everything when it comes to making sure that those patients actually convert.Michael: Real quick, what does good ad copy look Shane: like? Yeah, so it really is based around a couple of things with Google ads. One is the keywords that people are actually searching for. So if you can, for example, if somebody searches dentist near me, you know, you would ideally have dentist near me in that ad copy and then something that makes you unique.So maybe that's, we specialize in patients with dental anxieties or maybe it's something around Be seen same week or maybe it's we're a network with most insurances. So having something that's not just You know dentist in fontana and your best dentist in fontana.It's like Everybody's going to have that so whether it's highlighting the number of google reviews that you have whether it's the sedation Options you have whether it's that the fact that you're in network with most insurances that is what we call effective copy and With a google ad for any of those, people listening who've been into the ads manager themselves You'll notice that there's different keywords and ad copy that you can write based on What the person searched and so that's a big thing there is google's trying to match up with the keywords that you're specifically targeting with what the person searched and then trying to mix that into An ad that matches up with what that person searched.So that's the key right there for good ad copy is making sure that you actually point out something that makes the practice unique and would make you stop and click on it. To follow up with that, Michael, the other thing is always be testing. So, there's some key numbers that you need to look at to determine whether or not your Google Ad campaign is performing.certainly clicks is kind of a vanity number. but the big ones are, your average cost per click. Your click through rate and your conversions. So your average cost per click, pretty, self explanatory. How much are we spending, um, in these ads every time someone clicks on the ad, because that's ultimately how you're being charged.Click through rate is the percentage of times that someone actually Clicked on the ad when it was seen and so with google they typically say, you know anything above two to three percent click through rate is a healthy Click through rate, uh, meaning, you know, you're targeting the right people you have engaging ad copy things like that but for like an emergency campaign we recommend, 30 percent click through rate and then for a general campaign being closer to 10 So that's something you can look at If you notice like your click through rate on a Google Ads campaign is, under a percent, for example, and then your average cost per click is like 15, then you can look at a few things, but most likely it probably has to do with either your targeting or the ad copy that you have in place.And then, of course, the final piece of that is conversions. How many people... Called the practice how many people submitted a contact form? So then ultimately like I mentioned we want to get to a point where we can be pretty predictable and say All right We know for every a thousand dollars we put in we get this amount of new patients out of it And that's the key to making sure that you're scaling a campaign Responsibly and in a way that's going to give you the best return.Michael: Gotcha. Okay. At the beginning of the episode, you mentioned like a lot of people try to do one campaign and then stuff in a bunch of keywords in that one campaign. do we do one campaign for each keyword then? Or what? How does that work? Shane: No, that's a good question. Um, so think of it in terms of service.So, you want to bid for multiple keywords for each campaign, but you want to make sure it's geared around one service. So, if we're running, a emergency dental campaign, then that's where we would, target keywords in that one campaign, such as emergency dentist, uh, emergency dentist near me 24 7 dentist urgent care dentist, you know these type of terms Anything that revolves around emergency dentistry and those would all be keywords that you would bid on in that campaign What you don't want to do is have one campaign and be bidding on emergency dentistry keywords dental implant keywords root canals You know all of these where you kind of throw all the services into one campaign Because ultimately, it's too hard to create that many ads that's going to match up with what the person is searching for and for Google to actually show that.So if you're running a specific ad on implants. Only bid on implant related keywords. Don't bid on, hygiene keywords. If you're running a campaign on, clear aligners or orthodontics, don't also bid on implant keywords in that campaign. So making sure it's very service based. Michael: Gotcha. So really niche, like hone in on that specific keyword and then...Give it what you got kind of thing Shane: Exactly go all in on that one service and you can have multiple campaigns running at once you could have four campaigns But each campaign you said it's a budget you said it's targeting you could run an implant campaign an emergency campaign and a general campaign all at the same time You just want to make sure they're Individually built out and not just kind of, throwing all the keywords into one basket.Yeah. Michael: Okay, that makes sense. And then you mentioned coming with a realistic budget. What's the realistic Shane: budget? So it's gonna, it's not a great answer for this, but it's gonna depend where you're at. what I can tell you based on our experience is if you're in, you know, competitive area, Southern California, South Florida, areas like that.I mean, honestly, coming in at a bare minimum of 1500 a month is probably what you're going to need to spend, in more rural areas, you can get away with, somewhere between 1000 a month, 1200 a month is kind of a starter. but, Michael, we have offices that. Have 3, 500 a month, 4, 000 a month that they spend in Google ads, single offices, and there are bigger offices out there that spend 10, 15, 20, 000 a month.So it's not necessarily about how much you spend. Certainly that has an impact because you're paying. 10 every time someone clicks on your ad and you have a 50 a day, threshold, you're only going to get five clicks on your ad a day. some of that has to do with it, but it's not necessarily the person who has the biggest budget that's going to win.In this case, it's going to be making sure you're doing everything that you possibly can to make those clicks count. And that's. Having the proper landing page set up having the conversion tracking set up writing that engaging ad copy We talked about using some of the tips we discussed so you can be more efficient and kind of be the the underdog in the fight and still get some of those patients, but you have to come into it realistically You can't think if I spend 500 a month at Google Ads I want to get 10 new patients out of that because that's most likely just not realistic Michael: Gotcha. Awesome. So any final pieces of advice you want to give to our listeners on this? Shane: One final piece of advice I'll give is there's something recently that came out, in dentistry in the last two to three months or so, called Google local service ads. And I don't know if anybody's talked about these on here yet or not, but these are ads that used to only be available for like the home service industry.So you're like HVAC companies, heating, you know, air and plumbing, things like that. They opened it up to dentistry. And so right now what I would do is I would recommend everybody go through the verification process Uh, it's called google local service ads. You can create an account on there. You have to submit information Such as your dental license.There's a few things there that you have to do But once you do that, you can run google local service ads Which if you go right now, and if you type plumber near me at the very top You'll see a few businesses that'll be listed above every ad on the page It has the google review rating. It'll have a phone number.It'll say the business name and then it will usually have a green Circle with like a check mark in it. It says google guaranteed So when you submit this information Your office can get this google guaranteed badge And then you can start running these local service ads and a lot of dental practices are starting to pick up on this now But there's still a lot of people who aren't running them.So it's way less competitive and the other cool thing about this michael is You only pay per lead with Google local service ads. So they only charge you when somebody actually calls and schedules an appointment. And if they call and they don't schedule an appointment, you can actually dispute it and let Google know that, hey, this person didn't actually schedule and they won't charge you for that.So Google local service ads, you're gonna start hearing more about this. And I wouldn't be surprised, Michael, if we do an episode on those here in the coming months. Michael: Nice. Awesome, Shane. I appreciate your time. And if anyone has further questions, you can definitely find them on the Dental Marketer Society Facebook group, or where can Shane: they reach out to you directly?Yep. They can always, uh, deal with marketer Facebook group of course. And, uh, crimson media group.com is the other place that you can reach us, and fill out a contact form there. And, uh, we also always like to help our friend Michael out do a free marketing analysis for any of the listeners, of this podcast.So if you wanna do a marketing analysis, us give you some data feedback, you'll go to crimson media group.com and, uh, we'll help you out. Michael: Awesome. So, Shane, thank you so much for being with me on this Monday morning marketing episode. Thanks, Michael.‍

Want some advice? You'll find it in this episode where we interview Financial Advisor Jon Cunningham on best practices to teach kids financial skills and tips to start investing early. Don't miss out! This episode is especially great for parents and kids to listen together. This is a two part series so stay tuned! Learn more about your ad choices. Visit megaphone.fm/adchoices Transcript Advisor Advice Part 2 Welcome back to another episode of the Cash Kid Podcast. If you haven’t already, be sure to visit our website at cashkidpodcast.com for more resources and links to past episodes. Follow us on Instagram as well. Today, is part 2 of our “Advisor Advice” interview with financial advisor Mr. Jon Cunningham. If you missed the first part of my interview with him, definitely go back and listen to it first. So much great advice… and he’s got more to share. The Cash Kid Podcast is underway. (music) Intro tease: So you’ve got some cash. Maybe from an allowance, or that money your grandma gave you for your 7th birthday (Here you go sweetie.) Thanks grandma. Whatever it is, what are you going to do with it? Spend it, hide it away… or maybe invest it? Let’s start learning how to make that money grow. Time to learn how to be a cash kid. (cash register) Alright, let’s jump right back into part 2 of our interview with Mr. Jon Cunningham. (music) Cash Kid - So what are some ways that kids can learn about personal finance in the stock market? Jon - Yeah, another great question. You know, I think it's very important someone once told me to always preserve precious capital. And so as you're working as a young person and you're working hard and you're sweating outside maybe mowing grass or doing chores for your parents, the last thing you want to do is earn that money and then lose it. So it's important that you really understand how investments and how the stock market works before you just put money in an account and buy a stock and hope for the best, right? Cash Kid - Definitely. Jon - You know, even when I was in high school, we had a stock market simulation game. And so there are programs out there that will basically give you plain money. So it's not real money. You're not subjecting your own money to risk, but you're buying stocks with this, you know, fake money and not real stocks. But it's a way to simulate how stocks work in the buying and the selling of stocks and researching them without putting money at risk. So that's an easy and safe way for a young person to really understand and learn the market without actually subjecting their money to risk of loss. Cash Kid- Yeah. In our previous episode we've talked about what is the stock market game. And so we talked with one of my teachers that introduced that to me. Jon - Oh cool. Cash Kid - That's how I learned about the stock market. And so that's what our previous episode was about. What are some ways kids can start earning, saving and investing their money? Jon - Yeah, this is going to require a little bit of involvement from from Mom and Dad. But until you reach the age of adulthood, you really can't open an investment account without the involvement of a parent or guardian. So their accounts called up UTMAs Uniformed Transfer of Minors Accounts, and these are accounts that are owned by a guardian or a parent for your benefit. And money can go into these investment accounts and certainly can be invested for your benefit. However, the parent owns this account, so the funds have to be used for you and for your benefit and someday would have to be transferred to you when you became an adult. So this would be a safe way. I'd say safe. This could be a way to open an account with parental oversight that you can invest some of your money in the market that eventually could be for your benefit. Cash Kid - Yeah. Do you think there are any, like, jobs out there that kids could do? Jon - Oh, absolutely. I mean, I have three kids. I love it when they work for me. And I think I think oftentimes kids try to find creative ways of forming a business early, whether it be mowing grass or everything from that to pressure washing, things like that. And oftentimes it's tough to find prospects. It's tough to find customers. It takes time and energy and money and overhead to to make that income. When oftentimes you can find things in your own backyard that you can do for your parents that they would really appreciate and also be willing to pay you some money to do it. And so certainly, I think it's a good thing for kids to have chores and allowances, but then also to work for for that money. So that could be something you talk to your parents about and say, hey, I want to make, you know, $25 a month, you know, what can I do around the house to make that kind of money? And I'm sure, you know, most parents would be happy to oblige. Cash Kid - Yes. Now, I assume you would agree that parents can play a big role in the financial success of kids. I mean, kids are always thinking of a way to pick up spending habits. What ways can parents be involved in helping set their kids on a healthy financial path? Jon - Yeah Cash Kid another great question. One of the biggest issues with with our economy right now is just that more with the amount of student loan debt that's out there. To understand what a student loan is, when a young person wants to go to college and they don't have the the cash or the resources to pay for tuition, room and board and all those types of things, they'll go to lending and lending institutions to get a loan to cover those costs. And often time, oftentimes interest and payments are deferred on those loans until that person graduates. So the risk associated with that is the student goes to college, incurs this debt and they graduate and number one maybe they can't find a job or they can't find a job that pays enough money for them to cover the student loan debt, plus all their other living expenses that's out there. So part of the way that parents can help their their kids be on a on a good financial trajectory is to make sure that they're making little decisions early on in their children's lives by setting money aside in a college related accounts like 529 plans or or other investment accounts that are earmarked for for their kids college. So with some of the listeners that you might be having here, a question could be you know, what can I be doing to set money aside that could help my parents pay for my college someday? Or maybe it's, you know, what can I be doing to do really well in school that I get what's called a scholarship and some of my tuition might be covered and all that could be helpful and ways of setting yourselves up financially for the future. But then also to I think it's important for parents just to be good examples and show good stewardship and be good stewards of the cash flow and educate kids on just simply what's a credit card mean or what's a checking account, and then what can we be doing to ensure I'm building a good credit score, even at a young age. Just kind of having transparent conversations like that I think are really important. Cash Kid - Yes. So, um, I'm sure you tell this to your partners in a lot of like work that you do. So when you educate others about early investing, why do you think early investing is important? Jon - Well, that's a great question. So I was prepared to answer that one. So yeah, when you look at the cost of waiting and waiting on investing, it's very expensive. So take an example of a ten year old child that says, hey, I have $600 a year or $50 a month that I want to invest and I'm going to invest that money and hopefully earn 6% growth for the next 55 years. So you're looking at starting at ten years old to age 65 to 55 years. Guess how much money that person has in 55 years? They have $236,503. So they started at age ten and saved the exact same amount between age ten and age 65. So now let's fast forward and say, hey, a 15 year old says, Hey, I want to do the exact same thing. I want to set aside $50 a month or $600 per year for the next 50 years until I'm 65 years old. Guess how much money that person has? They have $174,201.54, assuming they can make 6% every single year. The cost difference of a little over $62,000. So you see there's a significant cost to waiting and delaying, saving and investing at a young age. Now, there's not many ten year olds that can set aside 600 bucks a year. You know, consistently. That's a lot of work and that's a lot of chores. And and that's understandable. But just that exercise shows you that it's very important to start early, especially when you get out of college and you start making an income to really begin diligently saving and setting money aside for the future and not delaying because it is expensive. Cash Kid - Right? So what age would you say somebody could start investing? Jon - Well, really, it could it could start at any time with the assistance of Mom and Dad or guardian. But typically you have to be at age 18 to open an account by yourself and have an account individually owned by by just yourself. So you have to be 18 years old to have your own account. But again, going back to the previous point, you can open what's called UTMA with the assistance of Mom and Dad. Cash Kid - Right. So what resources would you suggest about researching companies to invest or find an advisor like yourself to get help? Jon - Yeah, that's a great question. You know, oftentimes if you just simply ask your Mom and Dad say, hey, you know, do we have a financial advisor? And chances are they're going to say yes. And oftentimes they'd be happy to talk to people like you Cash Kid and your little listeners for sure, especially if they have good questions. So I think that's step one just to kind of say, hey, is there an existing relationship that I can take advantage of and ask them questions, number one. Number two, if that's not available, you know, Google Finance is something that's a great tool. You can go on and check any stock and you can see all the publicly offered information like revenue and expenses and have it done all of these types of things for the marketplace. So Google finance is a very good and free resource to research and look into the stocks that you have interest in. Cash Kid - That's it for today. We appreciate your time and your expertise. Thank you for joining us on the call today and boosting the financial knowledge of the fellow cash kids everywhere. Remember that anybody can be a cash kid. You just have to learn how to become one. Jon - Thanks, Cash Kid (music) Thank you Mr. Jon. Wow, I’ve got some more homework to do for sure as I learned a lot from Mr. Jon and excited to think of the payout to investing early and knowing how to be financial smart early in life. More great interviews like this one in future episodes. Remember to visit us at the cashkidpodcast.com for more helpful tools, information, and past episodes. Cash Kid, out! Disclaimer: The information presented represents the views and opinions of the guests. This show does not intend to provide personal investment advice through this podcast. This content has been made for informational and educational purposes only. To make a full and informed investment decision, we advise you to speak with a financial advisor and for kids, definitely your parents first before investing.

Transcript Eric Topol (00:00):This is a real great opportunity to speak to one of the most impressive medical informaticists and leaders in AI in the United States and worldwide. Dr. John Halamka, just by way of background, John, his baccalaureate in Stanford was at U C S F/Berkeley for combined MD PhD trained in emergency medicine at U C L A. He went on to Harvard where he, for 20 years was the Chief Information Officer at Beth Israel Deaconess. And then in 2020 he joined Mayo Clinic to head its platform to help transform Mayo Clinic to be the global leader in digital healthcare. So welcome, John. It's so great to have you. And by the way, I want to mention your recent book came out in April, one of many books you've written, redefining the Boundaries of Medicine, the High Tech High Touch Path into the Future.John Halamka (01:00):Well, a thrilled to be with you today, and you and I need to spend more time together very clearly.Eric Topol (01:06):Yeah, I really think so. Because this is the first time we've had a one-on-one conversation. We've been on panels together, but that's not enough. We've got to really do some brainstorming, the two of us. But first I wanted to get into, because you have been on a leading edge of ai and Mayo is doing big things in this space, what are you excited about? Where do you think things are right now?John Halamka (01:35):So you and I have been in academic healthcare for decades, and we know there's some brilliant people, well-meaning people, but sometimes the agility to innovate isn't quite there, whether it's a fear of failure, it's the process of getting things approved. So the question of course is can you build to scale the technology and the processes and change policies so that anyone can do what they want much more rapidly? And so what's been exciting over these last couple of years at Mayo is we started with the data and we know that anything we do, whether it's predictive or regenerative, starts with high quality curated data. And so by de-identifying all the multimodal data of Mayo and then working with other partners around the world to create a distributed federated approach for anyone to train anything, suddenly you're empowering a very large number of innovators. And then you've seen what's happened in society. I mean, culturally, people are starting to say, wow, this ai, it could actually reduce burden, it could democratize access to knowledge. I actually think that yes, there need to be guidelines and guardrails, but on the whole, this could be very good. So here we have a perfect storm, the technology, the policy, the cultural change, and therefore these next couple of years are going to be really productive.Implementing a Mayo Randomized AI TrialEric Topol (02:59):Well, and especially at Mayo, the reason I say that is not only do they recruit you, having had a couple of decades of experience in a Harvard program, but Mayo's depth of patient care is extraordinary. And so that gets me to, for example, you did a randomized trial at Mayo Clinic, which there aren't that many of by the way in AI where you gave E C G reading power of AI to half the primary care doctors and the other half you didn't for determining whether the patients had poor cardiac function that is low ejection fraction. And now as I understand it, having done that randomized trial published it, you've implemented that throughout the Mayo Clinic system as far as this AI ECG support. Is that true?John Halamka (03:56):Well, right, and let me just give you a personal example that shows you how it's used. So I have an SVT [supraventricular tachycardia] , and that means at times my resting heart rate of 55 goes to one 70. It's uncomfortable. It's not life-threatening. I was really concerned, oh, may I have underlying cardiomyopathy, valvular disease, coronary artery disease. So Paul Friedman and Peter Newsworthy said, Hey, we're going to take a six lead ECG wearable, send it to your home and just record a bunch of data and your activities of daily living. And then we buy 5G cell phone. We'll be collecting those six leads and we'll run it through all of our various validated AI systems. And then we'll tell you based on what the AI suggests, whether you're at high risk or not for various disease states. So it says your ejection fraction 70%. Oh, good. Don't have to worry about that. Your likelihood of developing AFib 3% cardiomyopathy, 2% valvular disease, 1%. So bottom line is without even going to a bricks and mortar facility here, I have these validated algorithms, at least doing a screen to see where maybe I should get additional evaluation and not.Eric Topol (05:12):Yeah, well see what you're bringing up is a whole other dimension. So on the one hand that what we talked about was you could give the primary care doctors who don't read electrocardiograms very well, you give them supercharged by having a deep learning interpretation set for them. But on the other, now you're bringing up this other patient facing story where you're taking a cardiogram when somebody's perfectly fine. But from that, from having deep learning of cardiograms, millions of cardiograms, you're telling what their risks are that they could develop things like atrial fibrillation. So this is starting to span the gamut of what the phase that we went through or still going through, which is taking medical images, whether it's a cardiogram or a scan of some sort, and seeing things with machines that humanize really can't detect or perceive. So yeah, we're just starting to get out of the block here, John. And you've already brought up a couple of major applications that we were not even potentially used three, four or five years ago that Mayo Clinics leading the charge, right?The Power of Machine EyesJohn Halamka (06:26):Well, yeah, and let me just give you two quick other examples of these are in studies now, right? So they're not ready for active patient use. The animate GI product does an overread of endoscopy. And what we're finding is that the expert human, I mean anywhere in the world, expert humans miss about 15% of small polyps. They're just hard to see. Prep may not be perfect, et cetera. The machine misses about 3%. So that's to say a human augmented with overread is five times better than a human alone pancreatic cancer, my father-in-law died about 11 years ago of stage four pancreatic cancer. So this is something that I'm very sensitive about, very often diagnosed late, and you can't do much. What we've been able to see is looking at pancreatic cancer, early films that were taken, abdominal CT scans and these sorts of things, algorithms can detect pancreatic cancer  two years before it is manifested clinically. And so here's the ethical question I'll pose to you. I know you think about a lot of this Scripps Mayo, UCSF, Stanford, we probably have thousands and thousands of abdominal CTs that were read normal. Is it an ethical imperative as these things go through clinical trials and are validated and FDA approved to rerun algorithms on previous patients to diagnose disease we didn't see?Eric Topol (08:03):Well, that is a really big important question because basically we're relieving all this stuff on the table that doesn't get diagnosed, can't be predicted because we're not even looking for it. And now whether it's retina, that is a gateway to so many systems of the body, or as you're mentioning various scans like an abdominal CT and many others that like mammography for heart disease risk and all sorts of things that weren't even contemplated that machine eyes can do. So it's really pretty striking and upending cancer diagnosis, being able to understand the risk of any individual for particular types of cancer so that you can catch it at the earliest possible time when it's microscopic before it spreads. This, of course, is a cardinal objective. People don't die of cancer per se. They die of its metastasis, of course, for the most part. So that gets me now to the next phase of ai because what we've been talking for mostly so far has been what has been brewing culminating for the last five years, which is medical images and what, there's so many things we can glean from them that humans can't including expert humans in whatever discipline of medicine.Multimodal AI and Social Determinants of Health(09:19):But the next phase, which you are starting to get at is the multimodal phase where you're not just taking the images, you're taking the medical records, the EHRs, you're getting the genomics, the gut microbiome, the sensors. You mentioned one, an ECGs, a cardiogram sensor, but other sensors like on the wrist, you're getting the environmental things like air pollution, air quality and various things. You're getting the whole ball of wax any given individual. Now, that's kind of where we're headed. Are you doing multimodal ai? Have you already embarked in that new path? Now that we have these large language modelsJohn Halamka (10:02):And we have, and so like anything we do in healthcare innovation, you need a Pareto diagram to say, what do you start with and where do you go? So in 2020, we started with all of the structured data problems, meds, allergies, labs. Then we went to the unstructured data, billions of notes, op reports, H and Ps, and then we moved to telemetry, and then we moved to CT, MRI, PET. Then we move to radiation oncology and looking at all the auto contouring profiles used in linear accelerators and then to omic, and now we're moving to an inferred social determinants of health. And let me explain that for a minute.(10:45):Exposome, as you point out, is really critical. Now, do you know if you live in a Superfund site area, do you know what risks you might have from the PM 2.5 particulates that are blowing through San Diego? Probably you don't. So you're not going to self-report this stuff. And so we have created something called the house Index where we've taken every address in the United States, and based on the latitude and longitude of where you live, we have mapped air, water, land, pollution, access to primary care, crime, education, grocery stores, stores, and therefore we can infer about 40 different things about your expose em just from where you live. And that's a mode. And then as you say, now, starting to gather remote patient monitoring. We have this acute advanced care in the home program where we're taking serious and complex illness, caring for the patient in the home, starting to instrument homes and gather a lot more telemetry. All of that multimodal data is now available to any one of the 76,000 employees of Mayo and our partners for use in algorithm development.Eric Topol (11:58):Yeah, no, that's extraordinary. And I also would say the social determinants of health, which you've really gotten into as its importance. There are so many papers now over the last several years that have emphasized that your zip code is one of the most important things of your health. And it's not even just a zip code. It's your neighborhood within that zip code for the reasons that you've mentioned. And inferring that and imputing that with other sources of data is vital. Now, this multimodal, you've again anticipated one of my questions, the possibility that we can gut hospitals as we know them today. Yes, preserving the ICUs, the emergency departments, the operating rooms, but those other people that occupy the vast majority of beds in the hospital that are not very sick, critically Ill. Do you think we're going to move as you're innovating at Mayo whereby we'll be able to keep those people at home for the most part in the years ahead? I mean, this isn't going to happen overnight, but do you think that's where we're headed?The Hospital-at-HomeJohn Halamka (13:08):So to date, Mayo and its partners have discharged about 23,000 patients from their homes. And as you can guess, we have done clinical trials and deep dive studies on every one of the patient's journeys. And what have we seen across 23,000 patients? Well, so generally, about 30% of patients that present for acute care to an emergency department come in by ambulance are appropriate for care in non-traditional settings. I mean, I think you would agree, somebody with episodic ventricular tachycardia, you're probably not going to put in a home setting, but somebody with congestive heart failure, COPD, pneumonia, I mean, these are things that, as you say, if they're going to get sicker, it will be over hours, not minutes. And therefore you can adjust in these molar than 20,000 patients. What we've seen is the outcomes are the same, the quality is the same safety, the same patient satisfaction. You get net promoter scores in the mid-nineties. You find me a hospital with a net promoter score in the mid nineties. You're eating your own food, slipping your own bed. Oh, your granddaughter's coming at 2:00 AM on a Sunday, whatever. And then ask yourself this other question, nosocomial infections,Eric Topol (14:31):Right?John Halamka (14:31):How many methicillin resistant staph infections do you have in your office? You're like, none, right? So you're infections in fall, so okay, better, stronger, cheaper, faster. And the safety of the quality are that for about 30% of the population should be a standard of care.Eric Topol (14:56):That's really big. So you don't think we have to do randomized trials to prove it?John Halamka (15:01):I mean, we have done enough studies to date, and there are organizations, Kaiser Permanente, Cleveland Clinic, all these folks who are joining us in investigating these areas. And the data is very compelling.Patients Asking Questions to LLMsEric Topol (15:17):Yeah, that's really exciting. And we may be able to jump past having to go through the large trials to prove what you just reviewed. So that's one thing of course that we're looking for in store. Another is the patient doing advanced large language model searches. So as you and everyone knows, we've done Google searches for years about symptoms, and inevitably people come up with hypochondria because they have some horrible disease that they looked up that is not a very good match specific for their condition and their background. But soon already today, we have people going into being creative mode, G P T four and other searches, and they're getting searches about their diagnosis and about what's the best literature and best treatments and expectations. That won't be FDA regulated. We don't have regulation of Google searches. So how do you see the democratization of large language models with patients having conversations with these chatbots?John Halamka (16:32):And of course, you ask a question no one has answered yet, but here are a few threads. So we know the challenge with existent commercial models as they're trained on the public internet. Some are trained on additional literature like PubMed or a mimic dataset, but none are trained on the rich clinical experience of millions and millions of patients. So therefore, they don't have the mastery of the care journey. So question, we are all asking, and again, no one knows. Then you take a GPT, BARD, a MedPaLM and additional pre-training with rich de-identified clinical experience and make it a better model for patients who are going to ask questions. We've got to try and we've got to try within guardrails and guidelines, but we definitely want to explore that. Can you or should you train a foundational model from scratch so that it doesn't have the bias of Reddit and all of the various kinds of chaff you find on the public internet? Could be very expensive, could be very time consuming. Probably society should look at doing it.Eric Topol (17:50):So this is just a review for those who are not up to speed on this, this means setting up a base model, which could be 20 to 30,000 graphic processing units, big expense. We're talking about tens of millions, but to do it right, so it isn't just a specialized fine tuning of a base model for medical purposes, but something that's de novo intended that no one's done yet. Yeah, that's I think a great idea if someone were to go down that path. Now you, early on when we were talking, you mentioned partners, not just other health systems, but one of the important partners you've established that's been out there as Google, which I think set up shop right in Rochester, Minnesota, so it could work closely with you. And obviously they have MedPaLM2, they have BARD, they published a lot in this space. They're obviously competing with Microsoft and others, but seems like it's mainly an arms race between those two and a few others. But how is that relationship going? And you also were very right spot on about the concerns of privacy, federated ai, privacy computing. Can you tell us about Mayo and Google?What is the Collaboration Between Mayo and Google?John Halamka (19:06):Well, absolutely. So Google provides storage, compute, various kinds of tools like their fire engine for moving data between various sources. Google does not have independent access to any of Mayo's data. So this isn't a situation of we have a challenging medical or engineering problem, bring 60 Google engineers to work on it. No, what they mean is they help us create the tooling and the environment so that then those with permission, Mayo employees or Mayo's partners can work through some of these things and build new models, validate models. So Google has been a great enabler on the tool set and building scale. You probably saw that Eric Horvitz gave a recent grand rounds at Stanford where he explained scale makes a difference, and that you start to see these unexpected behaviors, this emerging goodness, when you start dealing with vast amounts of multimodal data, vast amounts of compute. And so working with a cloud provider is going to give you that vast amounts of compute. So again, privacy, absolutely essential, de-identify the data, protect it, control it, but you can't as an institution, get enough computing power locally to develop some of these more.Towards Keyboard Liberation and Machine Chart ReviewEric Topol (20:36):Well, that goes back to the dilemma about building a base model with just the capital costs no less. You can't even get these GPUs scale because their supply and demand mismatch is profound. Well, the other thing, there's two other areas I want to get your impressions about. One of course is the change of interactions with patients. So today, as you well know, having all these years overseeing the informatics, Beth Israel now Mayo, the issue of the keyboard and the interference that it provides, not just as a data clerk burden to clinicians, which is horrible for morale and all the hours even after seeing patients that have to be put into charting through the EHRs and these clunky software systems that we are stuck with, but also the lack of even having face-to-face eye contact with patients in that limited time they have together. Now, there are many of these so-called ambient AI language, natural language processing, using large language models that are of course turning that conversation not just to a remarkable note, but also of course any part of the note, you could go back to the raw conversation. So it has trust embedded as what was really said. And then you have all these downstream functions like prescriptions, follow-up appointments, nudges to the patients about whatever, like their blood pressure or things that were discussed in the visit. You have translation to the patient at their level of education so they can understand the note you have things that we never had before. You have orders for the test or follow up appointment pre-authorization. What about these, John, are these the real deal or are we headed to this in the near term?John Halamka (22:41):So 10 years ago, I said all of these meaningful use criteria, all the keyboarded data entry, structured data and vocabularies. What if you had the doctor and the patient had a conversation and the conversation was the record? That was the legal record. And then AI systems extracted the structured data from the conversation. And there you would have satisfaction by both patient and doctor and a very easy source of truth. Go back to what was said. And of course, 10 years ago everyone said, that'll never happen. That's too far.(23:20):And so I'll give you a case. My mom was diagnosed with a brain abscess about a year ago. She's a cure of the brain abscess. I with ambient listening, had a conversation with my mother and it went something like this. Yes, I started to develop a fever. I said, oh, and you live alone, right? Oh, yes. My husband died 13 years ago. The note comes out, the patient is an 81 year old widow. So we're having a conversation about my father dying and she lives alone. And I didn't use the word widow, she didn't use the word widow. And so what it shows you is these systems can take detailed conversation, turn them into abstract concepts and record them in a way that's summarized and meaningful. Last example I'll give you recently, I did grand rounds at Mayo and I said, here's a challenge for all of us. It's Sunday at three in the morning. Mrs. Smith has just come in. She has a 3000 page chart, 75 hospitalizations and four or visits. Her complaint tonight is, I feel weak,Eric Topol (24:38):Right? That's a classic.John Halamka (24:43):How are you going to approach that? So we have an instance of MedPaLM2 that is containerized. So that I was able to put a prompt in it with some background data without, and it was all de-identified, but it was all very secure. So I put the 3000 pages into this MedPaLM2 container and said, audience, ask any question that you want. Oh, well, what medication should she be taking? What's her follow-up plan? Were there any complications in any of her surgeries? And within seconds, every answer to every question just appears. They say, oh my God, I can now treat the patient. And so this is real. It is absolutely. It's not perfect, but give us a couple of quarters.Eric Topol (25:31):Yeah, quarters not even years. I think you're putting the finger on something that a lot of people are not aware, which is when you have complex patients like what you just described, that woman, and you have so much information to review, no less the corpus of the medical literature, and you have help with diagnosis treatments that you might not otherwise thought of. It also gets me back to a point I was going to make the machine vision during colonoscopy where it does pick up these polyps, but it was shown that at the end of the day in the afternoon for gastroenterologists that are doing colonoscopies all day, their pickup rate drops down. They get tired, their eyes are just not working as well. And here your machines, they don't get tired. So these things are augmenting the performance of physicians, clinicians across the board potentially.(26:28):And yes, there's a concern as you touched on about confabulation or hallucinations, whatever, but this is a work in progress. There will be GPT-X, BARD-15 or whatever else right now, another area that is hot, which is still very in the earliest nascent stage, is the virtual medical coach. Whereby any of us with all our data, every visit we've ever had, plus our data that's in real time accruing or scans or slides or whatever it is, is all being fed in process with the medical literature and helping us to prevent a condition that we would have high risk to develop or manifest or better management of the various things we do have that we've already declared. What about that, John? Are we going to see virtual medical coaches like the kind we see for going to the airport, or you have an appointment such and such about your daily life, or is that something that is way out there in time?John Halamka (27:37):I know you're going to hate this answer. It depends.Eric Topol (27:41):Okay. I don't hate that. I like it actually. Yeah.John Halamka (27:44):So some years ago, one of my graduate students formed a virtual coaching company, and what he found was patients would often start with a virtual coach, but they wouldn't stick with it because the value add wasn't necessarily there. And that is it wasn't then every day there was something new or actionable. And so if it's few and far between, why do you want to go through the effort of engaging in this? So I think our answer there is we need to make sure that the person who uses it is getting something of value for using it. Reduced insurance rates, free club memberships to a gym, whatever, something of value. So it gets some stickiness.Virtual AI CoachingEric Topol (28:33):Yeah. Well, it's still early and right now, as you well know, it's really confined to certain conditions like diabetes or depression or high blood pressure. But it certainly has the chance in the years ahead to become broad for any individual. And that gets back to the patient scenario that you presented where you had all the data of that woman who presented with weakness as the inputs. And just think about that happening in real time, giving feedback to any given individual, always thinking that it's optional. And as you say, maybe it'd be more elective. There were incentives, and if people don't want it, they don't have to use it, but it's something that's out there dangling as a potential. Well, of the things we've discussed, there are many potential ways that AI can be transformative in the future, both for clinicians, for health systems, for patients. Have I missed anything that you're onto?John Halamka (29:40):Just that in predictive AI, we can judge performance against ground truth. Did you have the disease or not? Did you get a recommendation that was followed up on and it was positive? With generative ai measuring quality and accuracy, doing follow up and oversight is much harder. So I think what you're going to see is FDA and the office of the national coordinator and the White House work through generative AI oversight. It's going to start with, as we've seen voluntary oversight from some of the companies themselves. And it will evolve into maybe some use cases that are considered reasonable practices and others that we defer reasonable practices. Hey, you want an agent that will pre-draft your email and then you just edit it, that's fine. And Mayo is live with that in Epic inbox. How about help you write a letter or help you take, as you say, a very complex medical condition, explain it in eighth grade English or a foreign language. Very good at all of that differential diagnosis, not quite ready yet. And so I think we'll start with the administrative use cases, the things that reduce burden. We'll experiment with differential diagnosis. And I don't think we yet have line of sight to say, actually, we're going to have the generative ai do your diagnosis(31:09):Not there yet,Machines Promoting EmpathyEric Topol (31:10):Right? Perhaps we'll never be, particularly for important diagnoses, maybe for routine things that are not a serious matter. One thing that I didn't anticipate, and I want to get your view. When I wrote deep medicine, I was talking about restoring the patient-doctor relationship and the gift of time that could be garnered from having this machine support. But now we're seeing the evidence that the AI can promote empathy. So for example, reviewing a doctor's note and telling the doctor, you didn't show you're very sensitive. You weren't listening, making suggestions for being a more empathic physician or nurse. Did you foresee that too? Because you've been ahead of the curve on all this stuff.John Halamka (32:04):So here's an interesting question. You and I are physician, scientist, writers. How many physician scientist writers are there? Not so many. So what you get are brilliant math or brilliant science, and it is communicated very badly. So I did not anticipate this, but I'm saying the same thing you are, which is you can take a generative AI and take something that is not very digestible and turn it to something highly readable. And whether that's empathy or clarity or whatever, it actually works really well.Eric Topol (32:43):Yeah. Yeah. I mean, I kind of stunned by this because the machines don't know empathy. They can't feel empathy, but they can promote it. And that's really fascinating. So this has been an uplifting discussion. A lot of the things that's happening now give credit to you that you saw coming long before others, and it's a real joy. So we got to keep up with each other. We got to do some more brainstorming on the things that we haven't discussed today. But thanks so much, John, for joining me and for being such a bright light for the work you're doing with Mayo Clinic as a president of its platform. That's no question. Transforming the future of healthcare.John Halamka (33:25):Well, hey, thanks for having me. And I would say both you and I have taken the digital Hippocratic Oath. We will do no digital harm.Eric Topol (33:33):Love it. Get full access to Ground Truths at erictopol.substack.com/subscribe

This episode is available on Spotify and Apple PodcastsIn this "Ask Us Anything" (AUA) session, we answer questions from a Linkedin live about SEO & AI, the state of the economy, careers and more.Topics* Breakfast Routines* Predictions on Economic Recovery* AI and SEO* Preparing for AI and SGE* Eli shares the best career advice he didn't take* The Importance of Working at Big Brands Early in Your Career* Best Practices for Prioritizing and Implementing SEO Initiatives in Companies* Impact of Generative AI on Jobs and Medical Professionals* Building a Nerd Wallet Competitor in 2023* Impressive AI SEO Marketing Tools* Workspace Labs beta and Google's Keyword blog* Eli and Kevin Discuss Marketing Evolution and Tactics in Today's Digital Landscape* Companies Collaborating with Influencers* Discussing AI advancements and future implicationsTranscript[00:00:05] Kevin and Eli Go Live: A Friday Conversation[00:00:05] kevin: We're live.[00:00:06] eli: We're actually live.[00:00:07] kevin: Is this real?[00:00:08] eli: Yes, it is real. We did it. Oh, my God. Eli, what's going on this Friday? How are you?[00:00:14] kevin: Let's enjoy the podcast for the sake of all the people, all the millions of people that are not watching us live.[00:00:20] Contrary Marketing Podcast: Live AMA[00:00:20] eli: All right, sounds good.[00:00:21] kevin: Hello.[00:00:21] eli: Welcome to the Contrary marketing podcast, where we give you ideas you might not be thinking about today. Eli and I are doing a live AMA. Not MMA live. AMA we're beating ourselves up only verbally.[00:00:33] kevin: No, AMU sorry. Aua ask us anything.[00:00:38] eli: Ask us a good point where you can already tell by this highlevel. We ask each other questions, and we want questions from the live crowd that we're going to ask each other as well. Eli, you want to hit it off?[00:00:49] kevin: Yeah, but Kevin, I hate to interrupt the flow here, but I'm not sure that we're live.[00:00:52] eli: I'm pretty sure we have 33 viewers.[00:00:54] kevin: We do. Okay, so they're there on my LinkedIn event. I just see our logo.[00:01:00] eli: It's not a multiverse or what do you call it? Metaverse.[00:01:03] kevin: Can't see myself. Okay, well, we have 33 viewers. Welcome, everyone. Welcome.[00:01:06] Morning Q&A and Discussing Breakfast Routines[00:01:06] eli: Eli, hit me with a question here.[00:01:08] kevin: What do you have for breakfast?[00:01:09] eli: I had egg beaters, cherry cheese, and two slices of bread. Kind of my standard breakfast. I have that almost every morning, and I don't mind it at all. And of course, coffee. Can't forget the coffee shopify, mark. Exclusive for the growth team.[00:01:24] kevin: How about you? So it's a little bit earlier in the morning here on the West Coast, so I had to skip breakfast today. But I did have my coffee, and it's the first thing I do is wake up, think about the coffee, and run to go get it. But let's do a real question here.[00:01:35] eli: Let's do a real question.[00:01:37] kevin: I think we're full of SEO questions. One day we won't have SEO questions. Let's do a non SEO question.[00:01:43] Predictions on Economic Recovery and AI-related Stocks Driving Bull Market[00:01:43] kevin: When do you think the economy will get better?[00:01:45] eli: Man, if I knew that, I would put all my money into the stock market at some point in time. Look, I'm not an economist, but I have very deep insights into a lot of companies right now and some companies who make a lot that make a lot of money. And I don't yet think we have bottomed out. I think we have seen some of the worst B, two B businesses that sell to other companies and that might slowly turn around or at least flatten out, but I think consumer businesses might get hit even harder. And so, again, this is not financial advice, and I cannot see the future. But if I had to make a guess, I think Christmas this year or maybe even summer next year. That's kind of a span where I think the economy will get better. But I can only tell by how good business is going for some of the companies that are working with. And right now, demand is down. Sales cycles are still very long. Revenue is down year over year. So that's my prediction. What do you think?[00:02:44] kevin: I actually think the stock market has probably bottomed out. I don't know if you saw this, but we're in a bull market, which is crazy, because I always tell people, you can't pull your money out of the market because you don't know the bottom until you're months past the bottom. I forget the number, but this financial advisor told me that if you would have invested in the top of the market in 2008, right before the market, like, plummeted. So if you invested in the top of the market in 2008, and then you would have pulled your money out at the bottom of the market in March of 2020, when the market had dropped like 40% or something, you would have still made like 180%. So that's the market, right? So you look at trends and you think everything's terrible, but you have no idea. The only way you can really make money in the market is by keeping your money there. So I didn't realize it, but we're in a bull market. So a bear market is when the market is down 20% overall, and a bull market is when it's up 20%. So somehow within all of this, in 2023, we're actually in a bull market, and the market has turned. So I think the market will probably improve because a lot of the layoffs, a lot of the bad things, things have sort of already been baked in. Whether the bad things continue is a whole different story. So again, the market is psychology of the market is maybe things aren't as bad. So people invest, and there's money being made there. But I do think economically, there's probably a lot more layoffs. There's way too much spending for the amount of bad economic news and the amount of layoffs, unemployed amount of people. I think we're probably still a ways away from there. Maybe things turn around next year when there's an election, because markets are psychology, economics or psychology, and candidates can mess things up, messing around with things and make it look a little bit better. Stimulus might not be a good thing. Creates inflation. It created inflation. But hey, if they stimulate the economy and they start giving that money, people might spend, there might be more jobs. And we go back to those 2021 days, 2022 days, where companies just way overhired. I think around the marketing layoffs in general, there were just too many hires, so companies are just ramping down. I don't think it's necessarily a function of the employees. It's even a function of the companies. They just way overhired, and they need to pull that back. In Google's case, Google had never really fired anybody or laid anybody off. So they each had too many employees over two decades of growing, and they need to pull back. No one knows whether Google will do more layoffs, but they're not talking about it. Facebook continues or Meta continues to talk about doing layouts.[00:05:10] eli: The one thing that you mentioned that's super interesting is that we're in a bull market. And when you look at the stocks that are actually up driving that increase, it's all stocks related to AI. So Nvidia is going absolutely nuts right now. Apple after their announcements, google, Microsoft is really those stocks driving the market, and it's all about AI.[00:05:31] Kevin and Eli Discuss AI and Answer Questions[00:05:31] eli: And that's one topic that we got a lot of questions for, and I.[00:05:34] kevin: Think we have to drink now. Now. You just said AI. Oh.[00:05:37] eli: Every time you say, oh, okay, here we go.[00:05:39] kevin: I forgot.[00:05:39] eli: Here we go. Coffee, a zip of water.[00:05:41] kevin: There you go.[00:05:42] Preparing for AI and SGE: Impact on Search Traffic and the Future of SEO[00:05:42] kevin: All right, we made it five minutes without using the word AI.[00:05:45] eli: It's a new record, I think.[00:05:47] kevin: Yeah, it is a new record here.[00:05:50] eli: The question I have for you is you wrote this series of posts about AI and SGE, google's new search generic experience. Let's cut straight to the chase. Most people tuning in here know what's going on and have heard about AI and Se. What do you tell people or companies who ask you what they can do right now to prepare themselves for SGE when it rolls out to the broader public and for AI in general.[00:06:16] kevin: So I've been preparing for this for many years by focusing on users, by focusing on product led SEO. I was never really algo centric. So this is obviously AI and SGE is an algo adjustment. Well, not an algo, but it's an adjustment in this search layout and it's adjustment of the search page. So my focus has always been on building great things for the search user. So that's just a user that happens to come from a search engine, regardless of whether Google Bing, I don't know, Yandex or whatever. So I think that that is what everyone should continue to do, build for that search user. Now, the makeup of that search user is going to change. And I think the big way that's going to change is that search user is going to move from top of funnel, which is very generic searches, to more mid funnel. And I'll give you an example. I'm in the middle of planning a trip to Europe this summer, and I'm doing a lot of googling most of the queries that I Google, they are bringing up SGE. So obviously that didn't exist a month ago. So if I would have done that Googling, there wouldn't have been any generative responses unless I did it on Chat WT. But so there's now I'm Googling most of the names of hotels. When I Google, there is an SGE response. Now, is the SGE great? I don't think so. When I Google the name of hotel, there's a couple of things that I'm looking for around the hotel. One is where is it located? And two is what are the ratings? So where it's located? Obviously. That's Google. That's a Maps query. And what it's rated? That is TripAdvisor or Booking.com or anybody that's collected a significant mass amount of reviews. Google telling me that it is 176 room hotel built in 1992 and is in some neighborhood which I don't know the name of the neighborhood is not very helpful. However, the fact that that exists, that that content is there does mean that the people that are looking for that information are no longer going to go to TripAdvisor. So I'm still going to TripAdvisor because it has information that I need. But I think this SG is going to disrupt the top of the funnel and that may be a good thing. Maybe it's a good thing that there were so many sites out there before that created aggregated content that didn't give so much helpful information and they just ranked on Google and then captured the click and it wasn't helpful for the user. But by being number one, they got the click and now you won't get the click. So maybe that's a good thing for the users. They just capture that information right away. So I think everything about search is changing and traffic is going to change. And the interesting thing is I keep polling this on LinkedIn. I love LinkedIn polls because I get a lot of responses. Most people in my polls have not seen SGE. Another one I did last week is 53% of people are not preparing for anything related to generative in the rest of this year. And then the insight that I took out of this is that many, many marketers don't know what it is and they just assume it's another Google algo change. This is something I just heard from a CMO of a company who reached out and said, hey, our SEO agency says don't worry about this thing. It's another algo change, we'll be fine. And I don't think that's correct. I think this is a fundamental layout change and traffic will absolutely change. And just assuming that it's no big deal is a huge mistake. Now, will it crush most sites? I don't know. Like, depends on the vertical, depends on where Google lands with it. If you're an informational vertical, wikipedia as an example is getting crushed. I think that marketers should be using this, should be paying attention, and things will change. What do you think? Are you in my Pessimistic camp or you're like, this is Google, this is Google being Google. You don't even use Google. You use chat GBT. You tell me this all the time or even this Google.[00:09:50] eli: Yeah, it's funny. I do think things will change. So I'm halfway with 1ft. I'm in your camp.[00:09:57] kevin: I do think don't agree. You're not supposed to agree.[00:10:00] eli: No, it's just 1ft.[00:10:02] kevin: Half agree, half a foot.[00:10:06] eli: I agree with one toe and my other toe. Tell me that. No, it's it's a middle toe. It's a little longer than a piggy toe. I don't see SG going live in its current form. There was this interesting article that the chief Editor of Tom's Hardware published a couple of days ago fundamentally criticizing the accuracy and information in SGE and two, how it's still lacking references and how much it's copying, how borderline plagiarism it is. I do agree with that argument, directionally, but not to the same intensity, if that makes sense. But I think there's something there, it's in beta. It is great that it's in beta and it should stay in beta until Google has found a really good fit. I still also think that Bing AI search answers are much, much better.[00:10:55] kevin: Right?[00:10:55] Consequences of AI Integration in Search Results[00:10:55] eli: So little sneak peek. I'm publishing this article on Monday, but one thing that I did so this article, the chief editor Thomas Pilch last time is Pilch of Tom's Hardware. He calls out one example which is best GPU and in SGE. SGE basically word for word copies a large part articles by Tom's Hardware, PC gamer and other authorities and then links to them like barely maybe the Corroborated results, but doesn't even give a good attribution. And I tried that same query, best GPU in Edge on Bing in the conversational AI search and the results were much, much better. Basically, I got a very quick list, clear attribution and citation and the accuracy was so much more on point. So, long story short, I don't think this train is stoppable, train is moving, AI is going to go in the search results. The question is how? I think the current form of SGE is too aggressive to go live and not have everybody up in arms. I think we'll see a more toned down version of that. But we still need to think about the question of how do we thrive in this world and what can we do to stand out? And there are a bunch of things depending on the vertical you're in, but it's a new game or 50% of the game is new. Let's put it this way.[00:12:14] kevin: Totally agree with that. It's new and I also think it will keep changing. I think you're wrong about when it's going to launch or what form it's going to launch. I think that Google is rushing to launch things like maybe Google wants to wait a little bit and put it out next year. But every time OpenAI launches something new and they launch something new this week where it relates to functionality and they improved the capabilities of Chat GBT, I think that panics Google and someone's fingers hovering over that button that just puts it out onto the whole world, man.[00:12:47] eli: I don't think they can launch like that. I think they said in the blog article they're aiming for December of this year. They already said that they made some improvements to SG. I think Denny Sullivan, the search liaison at Google confirmed that. But I still think it's way off these references. It's a small detail, but it's an important one. I think citations or references need to be an SG and they need to be clearer than where they are right now. But I'm willing to gamble and I'm willing to bet here. But it's a whole new skill set that we have to learn and some things that I've already noticed. Again, we cannot make deterministic statements about how it works before we see the change in traffic, before we see the actual public life version or the final version. But one thing that I've already noticed is that this idea of third party reviews for your products has become so much more important. And it makes sense why Google is launching product review updates for its current algorithm left and right. I mean, every year you see four or five of these and they are all incredibly impactful. I mean, I'm talking to some businesses who see decreases of 30, 40, sometimes 50% of organic traffic due to a product review algorithm, and they suffer because they don't have a clear indication of methodology, clear authorship, clear evaluation system for products. And so it makes sense now because in part, Google leads so hard on third party reviews for its SGE experience when it comes to ecommerce related searches or local business searches. So this whole idea of a review footprint and influencing reviews and getting reviews out there is just so much more important, and I don't see that going away. That's my take.[00:14:23] kevin: Yeah. All right, let's move on to the next one. Let me ask you one. Go ahead. We're totally switching topics here and then we'll come back to SEO.[00:14:31] Eli shares the best career advice he didn't take[00:14:31] kevin: What's the best career advice you ever got and didn't take?[00:14:35] eli: Wow, dude, I got no warning about this one. That comes out left field, so let me see if I can stand up a great answer that doesn't make me lay in bed tonight and think about, oh, I wish I would have said that instead of this. So, look, I'm going to be going to be completely honest here and completely transparent. Someone who I look up to and still do gave me the advice that I wasn't ready enough yet to go out on my own to advise companies at that level. I'm not aiming to toot my own horn here, but I'm catering to executives. I'm catering to CEOs CMO CTOs Coos C suite.[00:15:13] kevin: Right. Head offs.[00:15:15] eli: Somebody gave me the advice to say, you need to do a couple more rounds of years at a company in house, get to a higher rank, broaden your scope to be ready to really advise companies at that level. And I didn't take it. And it worked out for me. There is an alternate universe where I took that advice and where it worked out even better for me. So I cannot A B test it, but I decided to go out on my own. I'm very happy with it, it works really well and I'm learning a ton. So that's advice I didn't take.[00:15:43] The Importance of Working at Big Brands Early in Your Career[00:15:43] eli: Let me ask you that same question because it's a really good question.[00:15:45] kevin: Well, first of all, I told you to go out on your own much earlier, so I'm glad you finally took my advice. You just didn't take it early enough and I'm glad you spent some time at some big companies. The advice I didn't take was I should have spent more time working at big companies. And I'll tell you why. This came from someone it's funny when you talk to like an older person, let's say someone in there, we're not going to go and blame people in their 40s or being old. None of you are old, I'm over 40. But it's more when you talk to an 80 year old and they give you career advice. And recently we interviewed someone on our podcast who is a couple of generations older than us, does not experience digital the way we did, but he was very insightful and we'll talk more about that in future posts and in future podcasts. But early in my career, an older person gave me advice that I should work for big brands. And at the time I wanted to work for startups because startups are cool and startups are how you get rich and all the cool stuff, work at Unicorns and write good things on your LinkedIn and writing your work at some major company and have some not so exciting job and boring title and boring job. Didn't seem exciting to me. But the advice I got was that I should work at big companies because big companies teach you things that you could then bring to smaller companies. I think that's true. And I should have spent more time at big companies. I should have when I got my first job, I took the first one offering a job because they didn't want to not have a job. It was a big ish kind of company, a few hundred employees. Eventually IPOed was not a startup at all. And then from there I went to a startup. I really wish at that point in time I had gone to a major brand, because there's something about being at a major brand where you learn how those things work. Yes, it's boring. Yes, you're probably not in control of a lot of things, but you see how those things work. You can bring those experiences to a smaller company because ultimately small companies want to be big companies. And then there's the brand cachet. So the company I worked at after that first job no one ever heard of, it was a startup got swallowed up by another company that no one ever heard of. And I learned an amazing amount, but it doesn't really make my resume look that exciting. I was fortunate that after that company I went to SurveyMonkey, which was a well known brand, which is more of a well known brand than it deserves because it's not a huge company, didn't have a huge valuation. However, so many people have taken surveys from SurveyMonkey, just in their mind it appeared like a big brand. For all we know, Pepsi or Coke couldn't be that. There might not be massive brands, but you just see it everywhere. Obviously there are big brands, but something you see everywhere, even if it's not huge, it's perception. So I was fortunate that it ended up being at a big brand, but I think I wish I would have stayed more in big brands and gone to companies like that. And I know we talk about Google a lot, a lot of people will say, oh, Googlers are not that smart. In 2023, maybe in 2005 they were very smart, but in 2023 they're not that smart. It doesn't really matter because everyone thinks they're that smart and that's all that matters. So when they go to their next jobs, when they go to do consulting, I wish I could say I'm an ex Googler and then go and try and do SEO consulting, but I can't. So that's advice to anybody listening that's early in your career, if you have a choice between some really sexy startup and Apple, take Apple, do it.[00:18:52] eli: Yeah, it depends on where you are in career and stuff, but I broadly agree, which I shouldn't.[00:18:56] Best Practices for Prioritizing and Implementing SEO Initiatives in Companies[00:18:56] eli: So I'm going to move on to the other question. I did ask some people earlier this week for questions that I can ask you, and I want to bring some of those up. So one of them is from Clay Kramer. Thanks, Clay, for submitting a question. And it is, what are some good practices for prioritizing and implementing SEO initiatives in your company?[00:19:16] kevin: So the best practice is one that's not followed in most companies, but is followed in the companies where SEO is the most successful, which is to think of it as an initiative instead of thinking of it as a thing to do. So in my career, and I'm fortunate that now my job is talking to many companies. Most of them I don't end up working for, but I talk to founders and I talk to C suite executives and learn about how SEO is done. In most companies, SEO is done as a tactic, as a thing to do. It does not elevate up to the level of executives. There's very little revenue reporting related to it. It's very black box. We're like, well, we don't know what's happening here, but it's magic, and we just fund the magic. And I think that's wrong. The approach is not that, it's this big strategic initiative that's tied into other strategic initiatives and it's part of a product plan, and everything we do needs to have some sort of SEO lens on it. Just like, well, there's the SEO person in the corner working their SEO black magic. And this is a tactic. And that is why a lot of the things that are talked about around SEO, they're just kind of considered tactics. And it's unfortunate that in a time like now where there's economic contraction and layoffs, the SEO who couldn't vocalize what they're doing couldn't communicate how all the things they're doing laddered into the broader picture could be on the chopping block. And I had a recent post on LinkedIn where I talked about how laying off an SEO team during a generational change in SEO makes no sense. That post was driven by how many SEO really, really smart SEO thinkers have reached out to me saying, I just found myself without a job. I was working at this big company, I'm doing these important things, and I was just laid off, and there's nobody behind it. There's nobody else doing SEO. I guess from our perspective as SEO consultants, this is good for us because when everything breaks and there's no budget to hire a full time employee, they will seek out a consultant to help them. But I think the first thing any company wants is a full time SEO employee that owns and drives and communicates what's happening with SEO. To me, that's the biggest myth is that it's not a strategic initiative. It's just a tactic, like build some links, like what we what'd you do on SEO this week? I built some links. Or what you do in SEO this week? I changed some title tags. Like, why? Like, how does that ladder into something? I just talked to a chief product officer at I don't know if they're fortune 500, but they're a really, really big public company. I asked them, like, what their roadmap was for SEO, and they're like, what would an SEO roadmap look like? They have like eight SEO people. And I had to talk to this CPO about how SEO should be important. And they're like, well, we're changing title tags. Then we're moving on to updating our XML sitemap. They're just things to do, and it's not an initiative. And they have eight employees, eight full time employees are spending a lot of money on and they have agencies and a bunch of other stuff. So millions of dollars a year, and it doesn't really tie to anything. It's not like we spend millions of dollars a year on SEO, and here's how we spend it on paid, and here's how it all ties together and CRM. No, we just do SEO. So that's my thing. What about you?[00:22:17] eli: I want to offer a different take. Of course. This is a contrary marketing podcast. I have to disagree.[00:22:22] kevin: Yeah, you just think SEO is dead. I'm with you.[00:22:25] eli: Yeah, I'm post SEO. Now. What is the news? Words AI optimization. Who even knows semantic optimization.[00:22:34] kevin: I don't know what's going on.[00:22:35] eli: So look, here's the thing, right? I think you should have both. You should have very clearly prioritized SEO projects. Where I agree with you, we have to agree with you is that most SEO strategies are actually tactics, not strategies. But what I think makes other sense is you have your top three SEO initiatives, and then you have your top three bets. And this is my contrarian stance here. I think SEO is so much of a black box now that you cannot expect everything to be properly projected and estimated. It's just not possible. There are some things that you don't know will work and will work out, but if they do, you get a competitive advantage. So the only way to move these things forward is to actually take bets. And so I've started working with clients, basically. We did that back at g two, and it worked out wonderfully. We had our big bets. Most of them actually worked out, but not all of them were based on perfect data, good logical constructs and argumentation. Of course, it's not just like licking your finger and putting it in the air and see where the wind is coming from, but it doesn't have to be properly projected and estimated by agreeing on a bet. You basically ask people to take a gamble. You get around all of these questions of, oh, how can we test it at a smaller scale? Like, how can we derisk it? How can we polish the stone so much until it's not sharp anymore? Or the knife, right? You tone it down. You tone it down. You tone it down, you launch it, and then it fails. And so instead, again, I'm pushing companies to take bets, not betting the farm, right? Not life or death type of bets, but let's allocate some capacity on things that we don't know will work out, but have good reason to believe that they might and try it. And that has proven to be very effective in my mind when it comes to prioritizing stuff.[00:24:22] kevin: Right.[00:24:22] eli: It's not just the numbers, but you also need to take a few bets.[00:24:25] kevin: I like that.[00:24:26] Impact of Generative AI on Jobs and Medical Professionals[00:24:26] kevin: All right. Yeah. Let me ask you a question related to this. So I got an email from someone, a medical doctor, a urologist. His profile picture had a stethoscope around his neck, and I checked out his LinkedIn, and he went to medical school as a real doctor. He's panicking about generative AI. Says gen of AI is cutting into his business. He didn't follow up yet. I'm curious why he's bothered by gender of AI, but that question generative AI is cutting into his business and is a urologist. It's interesting. What do you think doctors and service providers that don't provide a traditional service that you would think would be disrupted by generative AI should do about generative AI? I mean, ultimately, I think the doctor shouldn't worry is that doctor, and people come in and pay him to get treated physically, get treated. So cares. But I'm curious.[00:25:16] eli: You basically want to look. Out for tasks that you do completely virtually or completely digitally and that are legwork, right? Like, for example, there are some accounting tasks that you can replace with even chat GPT or AI. Want to be careful? You want to double check this, right? Let me tell you before I give any advice here. None of the technology is good enough yet for you to blindly trust it. Anything you do has to be double checked and viewed carefully.[00:25:44] kevin: Read the rest of the disclaimer. We're not medical advisors, lawyers, financial advisors. Thanks. And this has not been tested on animals. Okay?[00:25:52] eli: But in all honesty, for example, my dad is actually a urologist, and I know that he does a lot of work. What do you call this? Where my English is leaving me, where we talk into a machine and then somebody else types it out for you.[00:26:04] kevin: Dictation. Dictation.[00:26:05] eli: Thank you. Dictation. That you can perfectly replace it with AI. There are even tools. There's a tool. I'm giving you a recommendation right now. I'm not affiliated or anything. They're called audio pen. I think the purpose is more on the journalism side, where you can just ramble and speak and speak, and then audio Pen will kind of transcribe it for you and summarize it. Sorry, I mean journaling, not journalism. However, you can use that tool however you want. So if I was a doctor, I would use that for dictation so that there's not a poor soul that has to listen to it and type it down. And the reason doctors do that is because their handwriting is unreadable. And I know for a fact, but no, in all seriousness, I would use it for those kind of mundane tasks. Mundane, legwork, completely digital. That's where AI can already help you right now. I'm convinced that we will get to a point where AI will help you with diagnosis. There are interesting studies where AI can detect cancer and MRIs much more efficiently than doctors. Fun fact, the best results actually come from a combination of doctors and AI. Not just AI or just doctors. I'm not sure if we're there yet for the broader masses, that might take years, maybe decades. But I think right now the application is for very mundane tasks, virtually. And then in the future, I think AI will flow into every profession and into every job and make a lot of things a lot easier. I'm also in the camp of people who truly believe that AI is not going to replace more jobs than it creates. I think it will create a tremendous amount of new jobs. It will be a net positive impact. Do you agree or disagree?[00:27:40] kevin: No, I agree on that last point, that AI is not going to necessarily destroy jobs. It's like saying we don't want to have cashierless checkout at supermarkets because you got rid of the cashier's job. I don't know. Do you use this? There are sort of efficient, but then you have to call over the person, every other thing you check out to help you. So they're not great yet, but once they get a lot better, it's just a different job. I think AI is going to create a lot of new jobs. It's definitely going to take away old jobs. What you were saying around AI, and I think we should be clear about the difference between AI and generative AI. So AI has existed for a really long time. Like, there's AI that goes into cars, right? So like, obviously a Tesla is self driving that uses a lot of AI. I wrote an article about that recently. I think it creates a gigabyte of data every second. That's how much it's incorporating that's AI. It's like putting in all the sensors. But my car, it's not a Tesla. It's not a self driving car, but it has AI. That when I'm getting too close to a car in front of me and I'm driving too fast, it beeps and reminds me that I should hit the brake. So that's AI. It's just like processing information. So AI in general has been around for a really long time. Generative AI is newer, but the generative AI piece is that users can now access it and we can play with it and we can see how AI works. But it is actually not that complex. And we've talked about this before, it's just doing predictive statistics on future words and I think that is actually not that disruptive. It's just creating answers and sometimes it creates answers don't exist. Did you see the story about the lawyers who went to court with a chat chunk?[00:29:14] eli: Yes.[00:29:14] kevin: Those lawyers should be disbarred anyways because they didn't proofread their work. So I don't think all of a sudden it was like AI AI, but this is just generative AI. It's like, obviously uses neural networks and it's AI, but it is not certainly not taking doctors work.[00:29:27] eli: Yeah, I agree a little bit.[00:29:30] Building a Nerd Wallet Competitor in 2023[00:29:30] eli: But anyway, I wanted to take some life questions because we got some really cool life questions. Thank you all for submitting them. So, question for you, Eli Bujal Patel asks, what are your thoughts on building another behemoth like Nerd Wallet starting now? Would you start a Nerd Wallet like site, basically an affiliate on steroids in 2023?[00:29:50] kevin: Absolutely not. So I think that if you look at all the big affiliate sites, they stumbled upon something by accident and then it became massive and it worked out really well. I don't think you can go and look at something somebody else created and then say, I'm going to make a better Nerd Wallet or I'm going to make a better book site than Amazon and I'm going to sell my traffic back to Amazon. I think if you stumble upon something that there's like an open niche for and you create a bunch of content, there's potentially an opportunity. Again, it probably has to be more midfunnel than top of funnel, but I wouldn't go and say, oh, Nerd Wallet's missing this, and I'm going out Nerd Wallet. Nerd Wallet. One interesting stat I heard around the finance space in particular is Motley Fool. So now it's fool.com. I think it's been around for 20 years. They produce upwards of 100 pieces of content per day. So it's not a great content, but it's content. So if you were to create your own competitor to Fool.com today, you have to catch up with 20 years of 100 pieces of content. That's expensive. So don't disrupt something that already exists that is successful, because you have to outdo them. You have to outbrand them and show up in ranking. So I wouldn't go there at all. What do you think?[00:30:59] eli: I kind of agree. I wouldn't build a copy or another Nerd Wallet. I would be much more curious about how can I build a chat bot that replaces Nerd Wallet? We are at the verge of a huge technological shift that opens up a lot of opportunities. And so instead of thinking about these SEO models, which are under severe threat from AI, I would much rather think about what does the next evolution of Nerdwall It look like? What are they working on right now that I can compete with them on? I would wonder, what APIs can I use to train and feed a chat bot to become incredibly good at giving credit card advice?[00:31:34] kevin: Yeah, an interesting thing is I like this space. I like messing around with finance and seeing what's out there. So Google is they're pulling back from some of those queries. There was SGE on it. So if you look for Best credit card, there's no SGE on it anymore. But I think where Google falls, and I like your chatbot idea is Google can't do that next step. So if you did a query and again, it's gone. Right? But let's say it still exists, best Business credit card. And Google had an SGU and list out like, hey, here's this Chase card, and here's this Wells Fargo card, and here's this Discover card. Those aren't clickable results. So they basically replicate Google in an SGE response. It's not fulfilling. So I still think you go down to like, Nerd Wallet and then you click because from Google's little SGE summary, it's not enough to be like, that's it I'm Googling the Chase business card because that's what I want. So maybe that's why Google got rid of it. Maybe they got rid of it because they wanted to put the ads back. But I do think, yeah, chatbot is the way to go. Get more information. You don't need to read a 1500 word article to get a decision. Again, this requires huge behavior change. But you say, here's me, this is what I want. What's best credit card in response? And then there's your affiliate link.[00:32:39] eli: Agreed. Man, let's do a couple more.[00:32:41] Impressive AI SEO Marketing Tools[00:32:41] eli: Igal staultner also asks live thanks for your question. Igal hey guys, if you're already speaking about AI, what's the best use of AI you have seen in SEO marketing tools? To this day, I haven't been that.[00:32:53] kevin: Impressed with any SEO tool. What about you?[00:32:55] eli: There are two that are really interesting, and I'm probably going to get comments from all the other ones that I didn't mention. So I think one that's really impressive is by Word by W-O-R-D. It can first of all create content at scale across many different keywords and it can also find programmatic SEO place just with a URL.[00:33:16] kevin: Pretty impressive.[00:33:17] eli: The other one is write Sonic. They just published their fifth version of their Writer and I tried it out. I threw it into clear scope and I got a B plus out of it. Not saying that clear scope is the ultimate indicator for great content. There's more than that, obviously, but I'm noticing myself changing my mind where I thought for a long time that AI content is always going to be trash. And I'm starting to actually see counter evidence that some AI content is getting really, really good. So I've spent a lot of time over the last couple of days rethinking what the human contribution to content is and what content even means on the web and for SEO in general. So I would mention those too. There are a bunch of really cool other AI tools. We're using Summarize for the podcast. In part it creates really good descriptions and intros and Summaries. And then one app that I also like, based on AI, is called Poised PO. I sed. It gives you live feedback on how you talk, if you use a lot of filler words, if you say a lot, if you ask empathetic questions, if you're confident, assertive, et cetera. There are some really, really good tools sprouting left and right based on AI. And if you think about the fact that it's only been six months since Chat GPT got so popular, I feel like the next six months are going to be wild and I feel like the next twelve months are going to be even wilder.[00:34:37] kevin: So I thought you were going to say that around writing. I'm not impressed with the writing tools because I think most of them again, I haven't used byte words, but I think most of them are driven towards creating spam outside of SEO. Google just launched some new stuff. Did you see this? They improved Google Lens. Again. I use Pixels and use Android. But they improved Google Lens and got Now Google Lens can detect the skin condition. Just really cool. That's AI. And then I'm in the workspace labs. So there's a new thing in Gmail which is write for me so you can say what you're trying to say. It uses you and all the things you've written in the past to write an email for you. I did test it out, and it signed my name the way I wouldn't. Have a great day. Best, Eli. Right? I don't like that, so that's kind of weird. But again, it's kind of cool. Like, if you want to write a really long email, I hereby resign from this job. I hate this company, and all of you should burn in hell, or something like that. It can smooth that out for you. In general, I like Smart Compose, which I think most people use in Gmail already, which is it figures out what you're trying to say and just finishes your sentence. So this is right for me. Is that on steroids? So lots of cool AI stuff out there. When it comes to SEO, I think the approach to SEO has been more about, like, let's create a lot of high quantity content that may be of dubious quality. So not impressed yet, but please reach out if you have a really cool use case of AI and SEO.[00:35:58] Workspace Labs beta and Google's Keyword blog[00:35:58] eli: All right, I just signed up for the Workspace Labs beta while we were talking. Just Google workspace.[00:36:04] kevin: Lab beta.[00:36:05] eli: I thought I would get it automatically by being a customer.[00:36:07] kevin: Anyway, actually, a funny note here on how you could find out about these things. So Google's product blog is called the Keyword. Have you seen this? Of course. Okay, so it's called the keyword. That's the name of it. It's like blog. Google.com. Or actually, it's Blog Google. They don't dot the blog post right there on the homepage is virtually try on clothes with a new AI shopping feature. Like, if Google had a drinking contest for how often they say AI, everyone would be drunk.[00:36:34] Eli and Kevin Discuss Marketing Evolution and Tactics in Today's Digital Landscape[00:36:34] eli: One question comes from Charlie Williams. Actually like this question. I'm curious to know how Eli's approach to marketing has evolved over the years and what he thinks are the most important tactics for success in today's digital landscape.[00:36:47] kevin: I think my marketing has improved because I just learned more marketing, and that would be my advice for anyone new in their career, which is learn from what you're doing. I think that in today's digital landscape, it's just a digital representation of regular marketing. I love looking at old ads. Kevin, you ever go to museums and they show you like, I don't know, this is what it looked like in the 1920s.[00:37:08] eli: I do sometimes, too.[00:37:09] kevin: They were really good. They did good marketing. We're just doing it digitally. Something works on LinkedIn. Like when you have a viral post on LinkedIn, it's copywriting. So they did copywriting back in the day about, like, the invention of a washing machine or a car that had windows that you could roll down. I think marketing is the same. You're appealing to users and you're tracking them, and you're convincing them to trust you and give you money. Digital just allows you more tools. I don't know that tactics necessarily change. You want good copy, a great product, great message that resonates think the thing that many marketers potentially miss and actually working on a new book on this topic is they don't understand their users enough. So they understand themselves, they understand what they think will work, and they understand best practices about marketing, but they don't put themselves in the user's shoes. And sometimes that comes from, like, being a user, and sometimes that comes from good surveys. I had a friend who was he did market research for Skype. Part of his role was he went to all the places where people use Skype. So we went to India and Nepal and Bangladesh. His job was to not interview users, but he went to the users houses and he ate dinner with them. So props to Microsoft for doing that. Ate dinner, and he learned about them and how they were Skype users and how they use Skype to connect to people. Whatever it is you're marketing, understand the users motivations and why they want to pay you, then don't. Just like I watched a video on how to write good copy, and I watched a video on how to use TikTok for messaging. Ultimately, it comes down to humans buying things. So understand those humans. What about you? What's your evolution of marketing, man?[00:38:47] eli: What's the evolution? So my evolution of marketing has become a better understanding of the right playbook for the right business. I grew up in this very Silicon Valley type growth world where everything is highly measured as a strong product lens, the rigidity of testing, validating, and launching. And I think that's mostly applicable for certain types of companies, usually marketplaces, user generated content platforms. But I think there's this whole other cohort of companies who might be even bigger than the first. They're not able to measure most things. One example are enterprise companies who have long sales cycles, right? I'm talking about more than three months, sometimes six months, or even a year, who sell highly priced software to other enterprises. And they just need to play by different playbooks. They're not going to be able to test and validate everything as much as these other companies. For them, it's much more a before and after type of situation. So my lens has become more refined based on the business that I'm working with and picking the right playbook for the right business. I think we're getting really close on time.[00:39:55] Is SEO Dead?[00:39:55] eli: Let's do one more short question. E G live?[00:39:59] kevin: Is SEO dead? Sorry, that's not short. It's a short question. That's the long answer.[00:40:05] eli: It's fair. You caught me on this one.[00:40:07] kevin: You know what?[00:40:08] eli: Yes and no. I know it's not the answer that everybody wants, but SEO in its old form, I think, is going away. And out of it comes a new type of SEO that has maybe that has a core, maybe 50% of it is similar to what we did before, and that 50% is different. And I'm personally very excited. We're basically coming out of an exploit cycle and we're going back into an explore cycle and I'm all here for it. So I would say SEO is dead. 50%. That's my quick answer.[00:40:36] kevin: What do you think? No, it's not dead at all. It just changes. SEO doesn't die until search engines die. And I don't think search engines will die like Chat, GBT and generate. AI doesn't replace search, it just changes outwork. It's like saying SEO is dead because featured Snippets and Knowledge graph, I mean, I think links are going to probably go away in some way or another because links matter less in generative AI. However, brand matters and links and mentions and brand visibility matters. So just change what you're doing and change where you're going. Obviously, all those useless websites with guest posts that don't exist, those were a waste of money to begin with. So that probably have to go in. That's my quick answer. We'll do a whole Is SEO dead episode once it dies. So stay tuned.[00:41:22] Kevin and Eli's Final Conversation Topic[00:41:22] eli: Eli, you got one last one. All right, you did it in this one. Sorry, it's my turn.[00:41:27] kevin: Last one. Quick. Short one.[00:41:29] eli: Short one.[00:41:29] kevin: Now I have to pick.[00:41:30] Fears and Questions Clients Have About the Impact of Generative AI[00:41:30] eli: What is the biggest question that all your clients asking right now? What are some common questions that we haven't covered yet that you see bubbling up amongst your clients?[00:41:41] kevin: Do you have a discount available for startups? No, I don't, because I need to say bandwidth for the people that don't ask for a discount. I'd say a lot of companies are really freaked out by genera AI. I'm hearing it everywhere. If there's any listeners on the podcast that want us to do, like, a deep dive at your company on what we see in genera AI and our predictions, because I don't think anybody really knows we're available for that, we'll put a link in the show notes to how to contact us for that. Everyone's freaked out because there's change on the horizon. It's the same way when presidential elections happen, investment banks put out their statements of like, this is what we think Trump would do as president. This is what we think Biden would do as president. We're just prepared for all eventualities. I think when it comes to generative AI, it's really unpredictable and it's unknowable. So you have pessimists like me saying, everything's changing. You have optimists like Kevin. They say, don't worry, just keep doing what you're doing. And they don't know. They don't know. Do they hire for it? Do they fire for it? How do they plan? How do they message things to investors and board members and all stakeholders? So that's the biggest what about you? What's your biggest question?[00:42:51] eli: Man it's also related to AI, but I'm trying to not make it related because we're talking about this so much and I feel like I'm just going in circles here.[00:42:59] Companies Collaborating with Influencers like Sports Teams[00:42:59] eli: So the other really big question is what should we invest in that we haven't invested in right now? One really cool thing that I'm seeing is companies collaborating with influencers, more like sports teams. They're almost on full time payroll. Influencers, that is, for specific companies, that creates a lot of amazing content and they build real audiences, they build great engagement. And it's this amazing partnership between people who stand out in a space or have a lot of experience and expertise in a space and companies who get a real benefit from them. It's a great win win mix. If I had to tie that back to the whole AI discussion. We've seen this new Perspectives tabs roll out on Google, and I personally have a huge wish and a huge hope that it will be a new ground where influencers or creators, whatever you want to call them, can get a lot more traffic and a stronger voice. And it's kind of a way for companies to find a new playing field and forge these win win situations together with influencers and audiences. So I'm going to keep it to that one.[00:43:59] kevin: Love it.[00:44:00] Discussing AI advancements and future implications[00:44:00] kevin: All right, well, thank you, everyone. This has been epic. First time of recording live. Let us know if you want to do this again. And for everyone else that didn't record Live, well, follow us on LinkedIn and, you'll know, or listen to us live, follow us on LinkedIn, you'll know, next time we do it. That's a wrap.[00:44:15] eli: Thank you, Eli, for being a good thought partner, as always. And thank you all for tuning in. Happy weekend, and we'll hear you next week.[00:44:20] kevin: Thank you.[00:44:21] The Contrarian Marketing Podcast: Exploring Unconventional Business Strategies[00:44:21] eli: And now it's your turn. Head over to Contrarianmarketingpodcast.com and subscribe to the free weekly newsletter to get a summary of today's episode, key takeaways and community content. And while you're there, go to today's episode and leave your opinion in the comments. We'll feature the best thoughts in the newsletter and on the podcast. Also, if you like today's episode, please feel free to leave five stars on Spotify and Apple podcasts or wherever you listen to podcast. As always. This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.contrarianmarketingpodcast.com

Aaron Torres and Ala Shiban are from Klotho, which powers Infrastructure Copilot, the most advanced infrastructure design tool that understands how to define, connect, and scale your infrastructure-as-code. Victoria talks to Aaron and Ala about the Klotho engine, Klotho the CLI tool, and InfraCopilot and how they work together to help enable developer teams to iterate on applications and features quickly. Klotho (https://klo.dev/) Infrastructure Copilot (https://infracopilot.io/) Follow Klotho on Github (https://github.com/klothoplatform/klotho), Discord (https://discord.com/invite/4wwBRqqysY), Twitter (https://twitter.com/GetKlotho), or LinkedIn (https://www.linkedin.com/company/klothoplatform/). Follow Aaron Torres on LinkedIn (https://www.linkedin.com/in/torresaaron/), or Twitter (https://twitter.com/aarontorres). Follow Ala Shiban on LinkedIn (https://www.linkedin.com/in/alashiban/) or Twitter (https://twitter.com/AlaShiban). Follow thoughtbot on Twitter (https://twitter.com/thoughtbot) or LinkedIn (https://www.linkedin.com/company/150727/). Become a Sponsor (https://thoughtbot.com/sponsorship) of Giant Robots! Transcript: VICTORIA: This is the Giant Robots Smashing Into Other Giant Robots Podcast, where we explore the design, development, and business of great products. I'm your host, Victoria Guido. And with me today is Aaron Torres and Ala Shiban from Klotho, which powers Infrastructure Copilot, the most advanced infrastructure design tool that understands how to define, connect, and scale your infrastructure-as-code. Aaron and Ala, thank you for joining me. ALA: Thank you for having us. AARON: Yeah, thank you very much. VICTORIA: Well, great. I wanted to just start with a little bit of a icebreaker; maybe tell me a little bit more about what the weather is like where you're currently at. AARON: So I'm in St. Louis, Missouri. Right now, it is definitely...it feels like summer finally. So we're getting some nice, warm days and clear skies. ALA: And I'm in LA. And it's gloomier than I would like compared to what it's been in the last few years. But I'll take it if this means we're getting closer to summer. VICTORIA: Right. And I'm not too far from you, Ala, in San Diego, and it's a little chillier than I would prefer as well. But that's what we get for living close to the beach. So there's always trade-offs. Well, wonderful. I'm so excited to talk to you about your product here today. Let me start with a question about, let's say, I'm a non-technical founder, and I've just heard about your product. What's your pitch to someone in that position on the value of your tool? ALA: For somebody who isn't technical, I would say you can enable your team, your developer team, to quickly iterate on their applications or features and let InfraCopilot and Klotho take care of taking that application or features and deploy them and getting them running on the cloud. VICTORIA: Okay. So maybe I've been thinking about having to hire an AWS engineer or someone who's an infrastructure engineer. I could consider getting a tool like Klotho and Infrastructure Copilot to allow my developers to take on more of that responsibility themselves. ALA: Absolutely. VICTORIA: Gotcha. Okay, well, great. So let me ask about how did it all get started? What was the impetus that set you on this journey ALA: Both Aaron and I used to work at Riot Games, and I used to lead the cloud services org at Riot. I had about 50 people, 40 engineers, as part of a larger 120-person org, infrastructure platform org, which was tasked with building the platform that runs League of Legends, VALORANT, for 200 million people all around the world, in China. Full DevOps mode for Riot developers and full ops mode for running in China. It took us three years, a lot of effort. And by the time we were done, it was already legacy, and that seemed broken to me. We were already getting started to do another round of upgrades and iterations. At that point, I decided to leave. But I couldn't let go of this feeling that we shouldn't have had to spend so many years solving a problem only for it not to be solved. And based on research and conversations, it was clear that this was an industry-wide phenomena. And so I went about trying to figure out why that happens and then how we can solve it, and that's how Klotho came about. VICTORIA: That's so interesting. And I've certainly been a part of similar situations where you spend so much time solving a big problem and infrastructure only to get to the end of it and realize now you have a whole nother set of problems. [laughs] And you get upgrade. And they've also invented new ways of doing things in the cloud that you want to be able to take advantage of. So you had that time with Riot Games and League of Legends and building this globally responsive infrastructure. What lessons learned did you take from that into building Klotho and building your product, Infrastructure Copilot? AARON: We learned a bunch of things. One of the more difficult problems to solve isn't technical at all; it's organizational and understanding how the organization flows and how the different teams interact with each other. So we really endeavor to solve that problem. I mean, our product is a technical product, but it is meant to help bridge that gulf and make that problem a little bit easier as well. Otherwise, yeah, exactly to your point, part of the problem with these migrations is that new technology comes along. And there's definitely a feeling of when you hire new developers, they are excited about the new thing, and there's other reasons as well. But you get this kind of natural, eternal migration going to the newer technology. VICTORIA: That makes sense. And you bring up a great point on some of the issues, not being technical but organizational. And when I look at a lot of infrastructure-as-code tools, when we get to security, I wonder how it fits in with the organizational requirements for security, right? Like, you have to have defined groups who have defined access to different levels and have the tools in place to be able to manage identities in your organization. So I'm curious how that fits into what you built with Klotho and the Infrastructure Copilot. ALA: The way we think about infrastructure is as a set of intents or things that developers, and operators, cloud engineers, infrastructure engineers are trying to satisfy or to do. So you have tasks. You're trying to build a solution. You're trying to build an architecture or add something to it. And organizations have constraints, whether it's their own Terraform, or their own ruleset, or security expectations, or compliance expectations. And the way we look at this dynamic is those rules are encoded in a way that Klotho, which is a cloud compiler, it has the ability to reason about both the application and the infrastructure-as-code and enforce or at least warn about mismatches between the constraints that the organization sets, and what the developer or operator are trying to do, or the intent that is being described high level or low level within the tools. And then that is reflected both visually and in code and in the infrastructure-as-code, one or more. And so it's very much rooted in how the entire set of technologies and product and tools are designed. VICTORIA: Got it. So do you see the tool will be more fit for the market of larger development shops who maybe have existing infrastructure but want to experiment with a different way of managing it for their developers? ALA: It depends. So because we went about solving the problem rather than just building a specific vertical or a specific stack piece, we try to only play in this space of intelligent editing and intelligent understanding of the alignment between infrastructure and code. And so you could, as a developer, effectively with Klotho, write a plain application and have it be running in the cloud without knowing anything in the underlying cloud systems. It will set up storage, and persistence, and security, and secrets. All those elements are easily accessible within the code itself. It can also work in the context of a company where the infrastructure or platform team have set those rules and guidance within the tools. And then, developers can continue working the way they expect to work, either in code or in the infrastructure-as-code layer. And it would still allow them to do the same intents that they want only within that sandbox. Or if they can't be satisfied because they're trying to do something that isn't allowed, they have a mechanism of, one, knowing that but also asking, in our case, InfraCopilot to help it reshape what it's doing, what they're doing into the sandbox and the trade-offs that that brings in. VICTORIA: Got it. So you can both start from scratch and start a brand new application using it, or you can integrate it with your existing rules and systems and everything that already exists. ALA: Exactly. VICTORIA: Gotcha. Yeah, I think one interesting thing we've found with very new founders who are building their application for the first time is that there are some essential things, like, they don't even have an identity store like a Google [laughs] or Microsoft Azure Directory. So starting to work in the cloud, there are some basic elements you have to set up first that's a little bit of a barrier. So it sounds like what you're saying with Klotho is that you wouldn't necessarily have those same issues. Or how would you get that initial, like, cloud accounts set up? AARON: Yeah. So, for the situation where you're bootstrapping everything from scratch, you've done nothing; we haven't invested much in setting up the initial accounts. But assuming you get to the point where you have AWS credentials, and you're able to hit the AWS API using the CLI, that's sort of where we can take over. So, yeah, like, I would say right now, as a business, it's definitely where the value is coming is going to be these mid-sized companies. But for that scenario specifically, bootstrapping and starting something from scratch, if you have that initial setup in place, it's one of the fastest ways to go from a concept to something running in the cloud. ALA: And if you think about the two tools that we're building, there's Klotho, which InfraCopilot...or the Klotho engine, which Klotho the CLI tool uses and InfraCopilot uses. The Klotho engine is responsible for the intelligence. It knows how to translate things like I want a web API that talks to DynamoDB. And it will literally create everything or modify everything that is needed to give you that and plug in your code. You can also say things in a much higher level degree, which things like I want a lambda which handles 10,000 users. And I want it to be lowest latency talking to an RDS Instance or to a Postgres database. And what that would do is, in our side, in the Klotho engine, we understand that there needs to be a VPC and subnets, and spin up RDS, and connect an RDS proxy. Because for connection pooling with lambda specifically, you need one to scale to that degree of scale. And so that is the intelligence that is built into the Klotho engine if you want to start from the infrastructure. If you want to start from code, all you have to do is bring in the Redis instance, the Redis SDK, and, let's say, your favorite web framework, and just add the annotations or the metadata that says, I want this web framework to be exposed to the internet, and I want this Redis to be persisted in the cloud. And you run Klotho. And what comes out the other end is the cloud version that does that for you. And it's one command away from getting it to run. VICTORIA: So that's interesting how the two tools work together and how a developer might be able to get things spun up quickly on the cloud without having to know the details of each particular AWS service. And reading through your docs, it sounds like once you have something working in the cloud, then you'll also get automated recommendations on how to improve it for cost and reliability. Is that right? ALA: That's where we're headed. VICTORIA: Gotcha. I'm curious; for Aaron, it sounds like there is more in that organizational challenges that you alluded to earlier. So you want to be able to deliver this capability to developers. But what barriers have you found organizationally to getting this done? AARON: So I'm going to speak specifically on infrastructure here because I think this is one of the biggest ones we've seen. But typically, when you get to a larger-sized company, we'll call it a mid-sized company with, you know, a couple hundred engineers or more, you get to the point where it doesn't make sense for every team to own their entire vertical. And so you want to really put the cloud knowledge into a central team. And so you tend to build either a platform team, or an infrastructure team, or a cloud team who sort of owns how cloud resources are provisioned, which ones they support, et cetera. And so, really, some of the friction I'm talking about is the friction between that team and developer teams who really just want to write their application and get going quickly. But you don't have to fall within the boundaries set by that central team. To give, like, a real concrete example of that, if you wanted to prototype a new technology, like, let's say that some new database technology came out and you wanted to use it, it's a very coordinated effort between both teams in terms of the roadmap. Like, the infrastructure team needs to get on that roadmap, that they need to make a sandbox and how that's going to work. The code team needs to make an application to test it. And the whole thing requires a lot more communication than just tech. VICTORIA: Yeah, no, I've been part of kind of one of those classic DevOps problems. It's where now you've built the ops team and the dev team, [laughs] and now you're back to those coordination issues that you had before. So, if I were a dev using Klotho or the infrastructure-as-code copilot, I would theoretically have access to any AWS sandbox account. And I could just spin up whatever I wanted [laughs] within the limits that could be defined by your security team or by your, you know, I'm sure there's someone who's setting a limit on the size of databases you could spin up for fun. Does that sound right? AARON: Yeah, that's totally right. And in addition to just limits, it's also policies. So a good example is maybe in production for databases, you have a data retention policy. And you have something like we need to keep three months of backups for this amount of time. We want to make sure that if someone spins up a production database from any of those app teams, that they will follow their company policy there and not accidentally, like, lose data where it has to be maintained for some reason. ALA: That's an important distinction where we have our own set of, you know, best practice or rules that are followed roughly in the industry. But also, the key here is that the infrastructure central teams in every company can describe the different rulesets and guidelines, guardrails within the company on what developers can do, not only in low-level descriptions like instance sizes or how much something is, whether it's Spot Instances always or not in production versus dev. But also be able to teach the system when a developer says, "I want a database," spin up a Postgres database with this configuration that is wired to the larger application that they have. Or, if I want to run a service, then it spins up the correct elements and configures them to work, let's say, Kubernetes pods, or lambdas, or a combination based on what the company has described as the right way for that company to do things. And so it gives flexibility to not know the specific details but still get the company's specific way of doing them. And the key here is that we're trying to codify the communication patterns that do happen, and they need to happen if there's no tools to facilitate it between the infrastructure platform team and the feature teams. Only in this case, we try and capture that in a way that the central teams can define it. And the developers on feature teams can consume it without having as much friction. VICTORIA: So that will be different than, like, an infrastructure team that's putting out everything in Terraform and doing pull requests based off GitHub repository to that. It makes it a little more easier to read, and understand, and share the updates and changes. AARON: Right. And also, I mean, so, like, the thing you're describing of, like, the central team, having Terraform tends to be, like, these golden templates. Like they say, "If you want to make a database, here's your database template." And then you get a lot of interesting issues like drift, where maybe some teams are using the old versions of the templates, and they're not picking up the new changes. And how do you kind of reconcile all that? So it is meant to help with all of those things. VICTORIA: That makes a lot of sense. And I'm curious, what questions came up in the customer discovery process for this product that surprised you? ALA: I think there's one...I don't know that it was a question, but I think there was...So, when we started with Klotho, Klotho has the ability to enable a code-first approach, which means that you give the tool to developers as the infrastructure or platform team, or if you're a smaller shop, then you can just use Klotho directly. You set the rules on what's allowed or what's not allowed, and then developers can work very freely. They can describe very succinctly how to turn a plain object, SDK, et cetera, how to build larger architectures very quickly with a few annotations that we describe and that give cloud powers. We had always thought that some teams will feel that this encroaches on their jobs. We've heard from people on infra, you know, platform teams, "This is amazing. But this is my job." And so, one of our hypotheses was that we are encroaching into what they see as their responsibility. And we built more and more mechanisms that would clean up that interface and give them the ability to control more so they can free themselves up, just like most automations that happen in the world, to do more things. What happened later surprised us. And by having a few or several more discoveries, we found out that the feeling isn't a fear of the tool replacing their job. The fear or worry is that the tool will make their jobs boring, what is left of the job be boring, and nobody wants to go to work and not have cool and fun things to do. And because I think we all, on a certain degree, believe that, you know, if we take away some of the work that we're doing, we'll find something higher level and harder to solve, but until that exists in people's minds, there's nothing there. And therefore, they're left with whatever they don't want to do or didn't want to do. And so that's where we tried to take a step back from all the intelligence the Klotho engine provides through that code-first Klotho. And we built out focusing on one of the pillars in the tech to create InfraCopilot, which helps with keeping or making the things that we already do much simpler but also in a way that maintains and does it in a fun way. VICTORIA: That makes sense because my understanding of where to use AI and where to use machine learning for best purposes is to automate those, like, repetitive, boring tasks and allow people to focus on the creative and more interesting work, right? ALA: Yes and no. The interesting bit about our approach to ML is that we don't actually use machine learning or ChatGPT for any of the intelligence layers, meaning we don't ask ChatGPT to generate Terraform or any kind of GPT model to analyze a certain aspect of the infrastructure. That is all deterministic and happens in the Klotho engine. That is the uniqueness of why this always works rather than if GPT happened to get it right. What we use ML for is the ability to parse the intent. So we actually use it as a language model to parse the intent from what the user is trying to convey, meaning I want a lambda with an API gateway. What we get back from our use of ML is the user has asked for a lambda, an AWS lambda, and API gateway and that they be connected. That is the only thing we get back. And that is fed into the Klotho engine. And then, we do the intelligence to translate that to an actual architecture. VICTORIA: That's a really cool way to use natural language processing to build cloud infrastructure. MID-ROLL AD: Are you an entrepreneur or start-up founder looking to gain confidence in the way forward for your idea? At thoughtbot, we know you're tight on time and investment, which is why we've created targeted 1-hour remote workshops to help you develop a concrete plan for your product's next steps. Over four interactive sessions, we work with you on research, product design sprint, critical path, and presentation prep so that you and your team are better equipped with the skills and knowledge for success. Find out how we can help you move the needle at: tbot.io/entrepreneurs. VICTORIA: I'm curious; you said you're already working on some issues about being able to suggest improvements for cost reduction and efficiencies. What else is on your roadmap for what's coming up next? AARON: So there's a bunch of things in the long-term roadmap. And I'll say that, like, in the short term, it's much more about just expanding the breadth of what we support. If you think about just generating all the different permutations and types of infrastructure, it's, like, a huge matrix problem. Like, there's many, many dimensions that you could go in. And if you add an extra cloud or you add an extra capability, it expands everything. So you can imagine, like, testing it to make sure things work, and everything becomes very complicated. So, really, a lot of what we're doing is still foundational and trying to just increase the breadth, make the intent processing more intelligent, make the other bits work. And then one of the areas right now is for our initial release of the product; we chose to use Discord as our interface for the chatbot. And the reason for that is because it gives us a lot of benefits of having sort of the community built in and the engagement built in where we can actually talk with users and try and understand what they're doing. However, we really have a lot of UI changes and expansions that we'd like to do. And even from some of our early demo material, we have things like being able to right-click and being able to configure your lambda directly from the UI. So there's a lot of areas there that we can expand into an intent, too, once we get sort of the foundational stuff done, as an example. The intelligence bit is a much bigger process, like, there's a lot of things to unpack there. So I won't talk about it too much. But if we were to just talk about the most simple things, it'd be setting up alerts somehow and then feeding into our system that, like, we're hitting those alerts, and we have to make modifications. A good example of that would be, like, configuring auto scaling on an instance for [inaudible 22:17]. So we can get some of those benefits now. The bigger vision of what we want to do with optimization requires a lot more exploration and also the ability to look at what's happening to your application while it's running in the cloud. ALA: Let me maybe shed a bit more light on the problems we're trying to solve and where we're headed. When it comes to optimization, to truly optimize a cloud application, you have to reason about it on the application level rather than on the one service level. To do that, we have to be able to look at the application as an application. And today, there's a multi-repo approach to building cloud applications. So one of the future work that we're going to do is be able to reason about existing infrastructure-as-code from different portions of the teams or organization or even multiple services that the same team works and link them together. So, when we look at reasoning about an architecture, it is within the entire context of the application rather than just the smaller bits and pieces. That's one layer. Another layer is being able to ingest the real runtime application metrics and infrastructure metrics, let's say, from AWS or Azure into the optimizer system to be able to not only say, oh well, I want low latency. Then this is hard-coded to use a Fargate instance instead of a lambda. But more realistically, being able to see what that means in lambda world and maybe increase the concurrency count. Because we know that within the confines of cost limitations or constraints that the company wants to have, it is more feasible and cost-effective to raise the minimum concurrency rate of that lambda instead of using Fargate. You can only do that by having real-time data, or aggregated data come from the performance characteristics of the applications. And so that's another layer that we're going to be focusing on. The third one is, just like Aaron said, being able to approach that editing experience and operational experience, not just through one system like InfraCopilot but also through a web UI, or an app, or even as an extension to other systems that want to integrate with Klotho's engine. The last thing that I think is key is that we're still holding on to the vision that infrastructure should be invisible to most developers. Infrastructure definition is similar to how we approach assembly code. It's the bits and pieces. It's the underlying components, the CPUs, the storage. And as long as we're building microservices in that level of fidelity, of like, thinking about the wiring and how things interconnect, then we're not going to get the gains of 10x productivity building cloud applications. We have to enable developers and operators to work on a higher abstraction. And so our end game, where we're headed, is still what we want to build with Klotho, which is the ability to write code and have it be translated into what's allowed in the infrastructure within the constraints of the underlying platforms that infrastructure or platform teams set for the rest of the organization. It can be one set or multiple sets, but it's still that type of developers develop, and the infrastructure teams set them up to be able to develop, and there's separation. VICTORIA: Those are all really interesting problems to be solving. I also saw on your roadmap that you have published on Klotho that you're thinking of open-sourcing Klotho on GitHub. AARON: So, at this point, we already have the core engine of Klotho open-sourced, so the same engine that's powering InfraCopilot and Klotho, the tool itself is open source today. So, if anyone wants to take a look, it is on github.com/klothoplatform/klotho. VICTORIA: Super interesting. And it sounds like you mentioned you have a Discord. So that's where you're also getting feedback from developers on how to do this. And I think that challenge you mentioned about creating abstractions so that developers don't have to worry as much about the infrastructure and platform teams can just enable them to get their work done; I'm curious what you think is the biggest challenge with that. It seems like a problem that a lot of companies are trying to solve. So, what's the biggest challenge? And I think what do you think is unique about Klotho and solving that challenge? AARON: I guess what I would say the biggest challenge today is that every company is different enough that they all saw this in a slightly different way. So it's like, right now, the tools that are available are the building blocks to make the solution but not the solution itself. So, like, every cloud team approaches it on, let's build our own platform. We're building our own platform that every one of our developers is going to use. In some cases, we're building, like, frameworks and SDKs that everyone's going to use. But then the problem is that you're effectively saying my company is entering the platform management business. And there's no way the economies of scale will make sense forever in that world. So I think that's the biggest issue. And I think the reason it hasn't been solved is it's just a very hard problem. There's many approaches, but there's not a clear solution that kind of brings it all together. And I think our product is positioned better than most to solve some of the higher-level abstractions. It still doesn't solve the whole problem. There's still some things that are going to be tricky. But the idea is, if you can get to the point where you're using some of our abstractions, then you've guaranteed yourself portability into the future, like, your architecture will be able to evolve, even in technologies that don't exist yet once they become available. ALA: To tack on to what Aaron said, a key difference, and to our knowledge, this doesn't exist in any other tool or technology, is a fundamentally new architecture we call adaptive architecture. It is not microservices. It is not monoliths. It's a superset that combines all the benefits from monoliths, microservices, and serverless if you consider it a different platform or paradigm. What that means is that you get the benefits without the drawbacks. And the reason we can do it is because of the compiler approach that we're taking, where everything in the architecture that we produce is interchangeable. The team has decided to use Kubernetes, a specific version of Kubernetes with Istio. That works great. And, a year later, it turns out that that choice no longer scales well for the use. And we need to use Linkerd. The problem in today's world and what companies have to do is retrofit everything and not only the technology itself, but it's the ripple effects of changing it into everything else that all the other choices that were made that depended on it. In the Klotho world, because of the compilation step or the compilation approach and its extensibility, you could say, I want to take out Istio and replace it with Linkerd. And it would percolate all the changes that need to happen everywhere for that to maintain its semantic behavior. To our knowledge, that doesn't exist anywhere today. VICTORIA: So it would do, maybe not, like, would do migrations for you as well? ALA: I think migrations are a special case. When it comes to stateless things, yes. When it comes to data, we are much more conservative. Again, bringing what we've learned in different companies in, a lot of solutions try to solve all the things versus we're trying to play in a very specific niche, which is the adaptive architecture of it all. But if you want to move data, there's fantastic tools for it, and we will guide you through getting the access to the actual underlying services and, say, great, write a migration system, or we can generate for you. But you will run it to move the data from, let's say, Postgres to MySQL or from being able to drain a unit on Kubernetes to a lambda. Some of those things are much more automatic. And the transition happened through the underlying technologies like Terraform or Pulumi. Others will require you to take a step, not because we can't do it for you but we want to be conservative with the choices. AARON: I would also add that another aspect of this is that we don't position ourselves as being the center of the universe for these teams. Like a lot of products, you kind of have to adopt the platform, and everything has to plug into it, and if you don't adhere, it doesn't work. We're trying very, very hard with our design to make it so that existing apps will continue to function like they've always functioned. If app developers want to continue using direct SDKs and managing config themselves, they can absolutely do that. And then they'll interact well with Klotho apps that are also in that same company. So we're trying to make it so that you can adopt incrementally without having to go all in. VICTORIA: So that makes a lot of sense. So it's really helpful if you're trying to swap out those stateless parts of your infrastructure and you want to make some changes there. And then, if you were going to do a data migration, it would help you and guide you to where additional tools might be needed to do that. And at your market segment, you're really focusing on having it be an additional tool, as opposed to, like, an all-encompassing platform. Did I get it all right? [crosstalk 31:07] ALA: Exactly. VICTORIA: [laughs] Cool. All right. Well, that's exciting. That's a lot of cool things that you all are working on. I'm curious how overall the workload is for you two. How big of a team do you have so far? How are you balancing out this work of creating something new and exciting that has such a broad potential scope? AARON: Yeah. So, right now, the team is currently six people. So it's Ala and I, plus four additional engineers is the current team. And in terms of, like, where we're focusing, the real answer is that it's somewhat reactive, and it's very fast. So, like, it could be, like...in fact, Copilot went from ideation to us acting on it extremely quickly. And it wasn't even in the pipeline before that. So I'd actually say the biggest challenge has been where do we sort of focus our energy to get the best results? And a lot of where we spend our time is sort of meta-process of, like, making sure we're investing in the right things. ALA: And I think that comes from both Aaron and I have been in the industry for over 15 years. We don't, you know, drop everything and now switch to something new. We're very both tactical and strategic with the pace and when we pivot. But the idea is when we decide to change and focus on something that we think will be higher value, and it's almost always rooted in the signals and hypotheses that we set out to kind of learn from, from every iteration that we go after. We are not the type that would say, "Oh, we saw this. Let's drop everything, and let's go do it." I think we've seen enough in the industry that there's a measure of knowing when to switch, and when to refocus, and what to do when these higher tidbits come, and then being able to execute aggressively when that choice or decision happens. VICTORIA: Are there any trends that you're watching right now that the outcome would influence a change in direction for you? ALA: Not technically. I think what we're seeing in the industry is there's no real approaches to solving the problem. I would say most of the solutions and trends that we're seeing are...I call them streamlined complexity. We choose a set of technologies, and we make that easy. We make the SaaS version, and it can do these workloads, and it makes that easy. But the minute you step out of the comfort zone of those tools, you're back into the nightmare that building distributed systems brings with it, and then you're back to, you know, square one. What we're trying to do is fundamentally solve the problem. And we haven't seen many at least make a lot of headway there. We are seeing a few of the startups that are starting to think in the same vein, which is the zeitgeist. And that's fantastic. We actually work with them closely to try and broaden the category. VICTORIA: Right. Do you feel that other companies who are working in a similar problem space that there is...is it competitive between each other? Or do you think it's actually more collaborative? ALA: It depends on the companies and what they're trying to achieve. Every set of companies have different incentives. So Google, Amazon, and Microsoft have, you know, are incentivized to keep you on their clouds. They may care less about what they have in there as long as you are happy to stay. So you'll see more open source being adopted. You will see Amazon trying to copy or operationalize a lot of open-source tools. Microsoft will give their...because they are working with larger companies to have more vertical solutions. Google is trying to catch up. If you look at startups, you will see some focus more on developers. You'll see others focus on infra team. So it really depends on the intersection of the companies, and then they either collaborate or they compete, depending on how it affects their strategy. In our case, we recognize that our competition is the incumbents and the current way of doing things. And so we are happy to collaborate with all the startups that are doing something in the vicinity of what we're doing, startups like Ampt, and Encore, and Winglang. And there's several others. We have our own Slack channel where we talk about, like, where we're headed or at least what we can do to support one another. VICTORIA: Great. And I wonder if that's part of your business decision to open source your product as well or if there are other factors involved. ALA: I think the biggest factor that we've seen, realistically, is the expectation in the developer community to have a core that is open source, not even the source available model but to have an open-source core that they can rely on always existing and referencing when, you know if the company disappears or Oracle buys them. And so I would say that that was the biggest determining factor in the end to open-sourcing the Klotho engine. It's a very pragmatic view. VICTORIA: That makes sense. Well, I wanted to make sure we had time to ask one of my favorite questions that I ask on the podcast, and you can both answer. But if you could go back in time to when you first started this project, what advice would you give yourself? ALA: I guess the advice that I would give is keep selling and start selling as early as you can, even before the vision is realized. Or let's say you're making kind of headway towards what you'll wind up sharing and giving companies, the lead time to creating the opportunities and the belief and the faith that you can solve problems for companies, and the entire machinery of doing that is a lot more complex than most founders, I think, or at least first-time founders or, honestly, myself have found it to be. AARON: Yeah. If I try and answer that same question, it's very challenging. I guess my perspective now is there's nothing I could tell myself that would make me go any faster because a lot of it really is the journey. Like, the amount of stuff that we've learned in the last year of working on this and exploring and talking with people and everything else has been so vast that there's nothing I can communicate to past me that would prepare me any better. So [laughs] I think I would try just my best to be encouraging to just stick with it. VICTORIA: Well, that's good. And who knows what you're going to learn in the next year that [laughs] probably might not help you in the past either? That's wonderful. Do you have any final takeaways for our listeners today or anything you'd like to promote? ALA: So, from my lens, I've always wanted to do a startup but felt that the life setting wasn't quite ready. And a lot of the startup culture is talking about younger, earlier founders. I think having had the industry experience and understanding both the organizational and technical challenges, knowing more people, and engineers, and founders, potential founders, has been vastly more helpful than what I would have been able to pull off ten years ago. So, if you are thinking maybe it's too late, it is not. It's probably easier in some regards now. And yeah, check out InfraCopilot. It's on infracopilot.io. We would love to have you try it out and go on this journey with us. AARON: Yeah, I would definitely echo that. I mean, sort of the same thing on the journey. Like, it's never too late to start. And yeah, like, I would say being in the industry and actually seeing these problems first-hand makes it so much more fulfilling to actually try and solve them. VICTORIA: That's [inaudible 38:15]. I'm excited to see what you all accomplish. And I appreciate you coming on the show. You can subscribe to the show and find notes along with a complete transcript for this episode at giantrobots.fm. If you have questions or comments, you can email us at hosts@giantrobots.fm. And you could find me on Twitter @victori_ousg or on Mastodon @vguido@thoughtbot.social. This podcast is brought to you by thoughtbot and produced and edited by Mandy Moore. Thanks for listening. See you next time. ANNOUNCER: This podcast is brought to you by thoughtbot, your expert strategy, design, development, and product management partner. We bring digital products from idea to success and teach you how because we care. Learn more at thoughtbot.com. Special Guests: Aaron Torres and Ala Shiban.

Fouad Bazzi is the Founder and CEO of Crescent Retreats, a Luxury Short Term Rental Brand. Prior to that, he was a Corporate Banker for 21 years, with the last nine years of that leading a consulting group for Community Banks. Having been a Short Term Rental guest dozens of times, Fouad combined his passion of Entrepreneurship, Real Estate Investing and Travel to launch his company in 2021. Fouad's Mission is to promote Health, Peace, Love and Travel, and he ensures that all Crescent Retreats have an element of each of these four values. Fouad is passionate about helping others take control of their own time by way of investing in real estate. He does so through Real Estate partnerships and a mentorship program. He is also a member of three high level Masterminds and is launching his own Mastermind in Detroit to help his community leverage his learnings. In this episode, we sit down with Fouad as he shares his insights on what luxury means in the short term rental space and how he incorporated the core values of Health, Peace, Love, and Travel to make his luxury brand more personal and cater to the many different ways luxury means to different people. He also discusses the changes he underwent to get into luxury short term rentals and what to expect when entering this space. Lastly he treats us to what's next for Crescent Retreats and the exciting developments on the horizon. Tune in now and Let's Build! Get Your Airbnb Millionaire Blueprint: https://experimentrealestate.com/#blueprint HIGHLIGHTS OF THE EPISODE: 13:12 Fouad talks about taking advantage of the research and development done by other thriving businesses 17:12 Fouad talks about how everybody wins in the real estate industry KEEPING IT REAL: 05:27 Health Peace Love and Travel as core values 08:46 The shift needed to build a luxury brand 11:10 What luxury means 14:54 Designing health peace love and travel 16:38 Core competency that was carried over to the short term rental industry 20:41 Strategy when raising capital and assets in focus 27:29 Key indicators of market appreciation 37:03 What to expect when getting in the luxury space 39:53 Let's talk about revenue 41:12 Exit or refinance 43:03 What to look for in partners 48:14 The best way to structure LLCs 49:58 What is next for Crescent Retreats? 53:21 How to connect with Fouad NOTABLE QUOTE (KEY LESSONS): “When you're in the luxury space, we don't have to figure anything out. There's a Four Seasons or a Ritz Carlton close by. There has to be. So Google it. Figure out how they furnish the local one.” - Fouad Bazzi CONNECTING WITH THE GUEST Instagram: https://www.instagram.com/crescentretreats/ Facebook: https://www.facebook.com/crescentretreats/ Linkedin: https://www.linkedin.com/in/fouad-bazzi-4415973/ Get Your Airbnb Millionaire Blueprint: https://experimentrealestate.com/#blueprint Get coached on how you can repurpose your existing home or investment property into an Airbnb with me: https://experimentrealestate.com/hospitable-hosts Guesty Unique Link: https://hosts.guesty.com/?=therealestateexperiment Hospitable Hosts book: https://amzn.to/3e4LEhE Apply for an American Express Card with this link: http://refer.amex.us/RUBENKr8et?xl=cp01&mpt=v0 #shorttermrentals #luxury #FiveStarExperience #corevalues

Al and Bev talk about Coral Island Timings 00:00:00: Theme Tune 00:00:30: Intro 00:01:46: What Have We Been Up To 00:08:02: News 00:17:22: Coral Island 01:40:08: Outro Links Research Story Update Sun Down Update APICO Console Update Grave Seasons Coral Island Contact Al on Twitter: https://twitter.com/TheScotBot Al on Mastodon: https://mastodon.scot/@TheScotBot Email Us: https://harvestseason.club/contact/ Transcript [00:00.000 –> 00:23.960] Theme tune [00:23.960 –> 00:38.000] Hello farmers and welcome to another episode of The Harvest Season. My name is Al. And [00:38.000 –> 00:45.240] my name is Bev. And we are here today to talk about cottagecore games. Woo! I still called [00:45.240 –> 00:51.800] them farmers. I mean, they’re always going to be farmers to us, I feel like. And I also [00:51.800 –> 01:00.920] discovered that goblincore is actually a real thing. And that is my vibe. I thought [01:00.920 –> 01:10.600] it was a joke. I didn’t realize it was real. And now I’ve accepted. Yeah, absolutely. Awesome. [01:10.600 –> 01:16.060] Um, cool. Well, let’s just get straight into it. And today we’re going to talk about Coral [01:16.060 –> 01:21.720] Island. We finally got here. We finally doing this episode. I feel like we’re right at the [01:21.720 –> 01:25.920] wrong time to talk about this game, right? Because it’s not just in early access. But [01:25.920 –> 01:30.400] we’re also like, not just about to have the fight the 1.0 release. So we’re just kind [01:30.400 –> 01:37.120] of like in that awkward middle. But the summer update is coming in like two weeks, I think [01:37.120 –> 01:42.280] maybe one week, something like that. So maybe maybe that’s good enough. And so we’re going [01:42.280 –> 01:47.200] to talk about that. Lots of thoughts. Before that, we’ll have some news. And but first [01:47.200 –> 01:56.720] of all, Bev, what have you been up to? I am starting my best farming life IRL. I recently [01:56.720 –> 02:04.800] bought a coop and built it. And we just got two little chicks yesterday. So we are chicken [02:04.800 –> 02:11.960] owners and they are adorable and they’re going to be our inside pets. And they’re so cute [02:11.960 –> 02:17.880] and so cuddly. And one purrs and one is just they like they’re only three days old, I think [02:17.880 –> 02:23.200] and they already have personalities. And I’m really excited about this. And I really hope [02:23.200 –> 02:29.800] my cats don’t eat them. That feels like it would be like the kind of bare minimum you’d [02:29.800 –> 02:37.880] want. Right, right. Like can we just not cats? Can we can we not? So the younger one who [02:37.880 –> 02:43.760] always has always been inside. He is just very curious and confused by them. He thinks [02:43.760 –> 02:48.280] they are toys and like, why can’t I play with the toys? I think it’s the older one that [02:48.280 –> 02:52.560] we’re a little bit more worried about because she used to be an outside cat. Right? Yeah. [02:52.560 –> 03:00.840] So she’s she could shoot. Yeah. So we’ll just we’ll just take it slow and see how it goes. [03:00.840 –> 03:07.880] Not eat the chicks. That’s the that’s that would be a good plan. [03:07.880 –> 03:13.800] And I’ve been playing roots of Pacha and have been obsessed lately, I guess with just random [03:13.800 –> 03:20.400] mobile games. I don’t know why. But like solitaire and the ads are getting to me. So I’m just [03:20.400 –> 03:24.280] constantly downloading the new ones and be like, Oh, let me try this and see if I actually [03:24.280 –> 03:33.840] like this. Cool. How about you? I have been playing a ridiculous amount of Coral Island, [03:33.840 –> 03:38.920] right? Like, we, we decided to do this episode. Well, we’ve been planning to do this episode [03:38.920 –> 03:44.040] for quite a while. Like, I think we must have decided this like, I don’t know, like, six [03:44.040 –> 03:47.760] months ago, we decided we were going to do this episode. But we never actually put in [03:47.760 –> 03:51.880] the time, mainly because they didn’t have like the controller support to start with. [03:51.880 –> 03:58.380] And I was not playing this on the PC. No, you can’t make me. So we were waiting on on [03:58.380 –> 04:04.400] controller support first. And then eventually, we got there. I think just like life got in [04:04.400 –> 04:08.440] the way. So eventually, you were like, Oh, yeah, we can I can do this next week. And [04:08.440 –> 04:14.160] I was like, Okay, all right, let’s do this then. And I have put 25 hours into it this [04:14.160 –> 04:21.840] week, which is a lot of time like I don’t normally play 2425 hours of a game in a week. [04:21.840 –> 04:27.480] But I was like, I need to do this one properly. So that is most of my gaming time. I’ve been [04:27.480 –> 04:34.000] playing a bit of Pokemon, not Pokemon Snap, Marvel Snap. And Pokemon Go. You know, the [04:34.000 –> 04:41.520] usuals. Nothing too special. And yeah, just bits and pieces here and there. Coral Island. [04:41.520 –> 04:42.520] That’s about it. [04:42.520 –> 04:48.480] Beautiful. Yeah, I’ve, I think I put up in about the same amount of time in a short while [04:48.480 –> 04:52.800] I’ve taken a little break from it since since rooster pacha came out. But yeah, I was the [04:52.800 –> 04:59.240] same. You want to go for that prehistoric farming sim instead? Uh huh. I was excited [04:59.240 –> 05:04.720] about it. And it’s not early access. So do you have any first thoughts on it? Obviously, [05:04.720 –> 05:09.920] we’re not in a proper episode on that yet. Yeah. Any first thoughts? I am enjoying it [05:09.920 –> 05:15.680] a lot. It’s a like I, you know, like you look at these farming sends and like often you’ll [05:15.680 –> 05:19.800] have to compare it to Stardew because of course, that’s where they’re based off of. [05:19.800 –> 05:27.480] Yeah. And I feel like this is like, like new enough that it it’s exciting and doesn’t quite [05:27.480 –> 05:32.560] feel like it’s just a Stardew in a different world. Yeah. Because it’s a whole different [05:32.560 –> 05:39.240] mechanics. I, I thought we would almost get like a completely capitalistic free like game. [05:39.240 –> 05:44.640] But there’s still a currency. But where I think they just can’t do it. It’s like an [05:44.640 –> 05:51.100] exchange system. But yeah, okay. Okay. But like the there’s like this, I don’t know, [05:51.100 –> 05:56.200] I don’t want to get too into it. But I’ve been I mean, just to be fair currency doesn’t [05:56.200 –> 06:02.000] equal capitalism, right? Like, you can have currency without being capitalistic, I guess [06:02.000 –> 06:07.880] like there’s, there’s almost two different types of currency. There’s like, like overall [06:07.880 –> 06:13.760] contribution. So there’s a contribution. And that’s like what you use for purchasing, like, [06:13.760 –> 06:18.600] upgrades and, and furniture and stuff. And then there’s another one that I forget what it’s called, [06:18.600 –> 06:24.560] that just continues to add up. And that’s just like the value of your society. Yes. [06:24.560 –> 06:31.240] Okay. GDP. Yeah, yeah, yeah. They’ve implemented GDP in a prehistoric game. [06:31.240 –> 06:38.160] It’s fun. It’s cute. The animals are cute. [06:38.160 –> 06:46.360] All right, fair enough. Well, we’ll see how that continues. I think we have a we have a planned, [06:46.360 –> 06:52.480] hopefully planned episode for that upcoming. So we’ll see what people think of that. I haven’t [06:52.480 –> 06:58.560] played at all. I might at some point, but like, there’s enough, there’s enough people playing it [06:58.600 –> 07:02.800] that I can, I’m going to at least wait until the episode comes out. To decide, [07:02.800 –> 07:05.920] I can actually use my podcast. So the way that I want my podcast to be used, [07:05.920 –> 07:10.400] I just don’t get to use it like that very often, right? Because I’m normally the one playing things [07:10.400 –> 07:15.040] and telling people what it’s like. But this way, I can actually listen to it and decide whether I [07:15.040 –> 07:20.360] want to play or not. Because weirdly, I never backed that one. Like, it’s I don’t know why [07:20.360 –> 07:28.440] I backed almost every single Kickstarter farming game for the past five years. And didn’t with that [07:28.480 –> 07:37.480] one. And I’m not sure why. I don’t know. Maybe just too many games. That’s why. Yeah, but I feel [07:37.480 –> 07:43.080] like if not, I didn’t really feel that, that there’s too many games until the last few months [07:43.080 –> 07:48.240] released, or maybe last year. And this, this was like, quite a few years ago, this, this game, [07:48.240 –> 07:52.960] when it started, I feel like it was like three years ago, it was a 2021 maybe a Kickstarter [07:52.960 –> 07:57.080] or something like that. I don’t know. It’s been a while. Well, yeah, there’s too much to remember. [07:57.080 –> 08:09.560] Anyway, shall we talk about some news? News. So first of all, research story. We’ve got a 0.2 [08:09.560 –> 08:14.920] update coming out. And Bev, this has something that I think you might be excited about. I am [08:14.920 –> 08:23.120] very much so. Because that means I can not have to play handheld on my Steam Deck. Controller [08:23.160 –> 08:30.200] support. Whoo. Realized I hadn’t actually said the words. We’re pretty much a fan of controller [08:30.200 –> 08:34.680] support on this podcast. Yeah, I think this was the one of the big things that you had on the on [08:34.680 –> 08:39.040] the podcast episode about this was the lack of controller support, causing issues for for both [08:39.040 –> 08:43.920] of you, I think. Even though Cody doesn’t have a Steam Deck, it was still causing some issues. So [08:43.920 –> 08:50.720] good that that is that is coming out. It’s out. Yes, it’s out now. So you got that there. And [08:51.600 –> 08:56.160] we’ll see how that goes. It looks like they have proper remapping as well. So if you have any [08:56.160 –> 09:02.480] issues with with the the automatic mapping, you can change change what control does what, [09:02.480 –> 09:08.400] which is the good way to do it, not just for you and me, but for you know, accessibility and stuff [09:08.400 –> 09:12.800] like that, as well as always, always very important. It has some other things as well, [09:12.800 –> 09:22.320] like cloud saves and support for 5120 by 1440 resolution. You got to love that 32 by nine [09:22.320 –> 09:30.560] resolution. So if you’ve been wanting to play an eight bit graphics game on a 32 by nine screen, [09:30.560 –> 09:33.600] you can you can now do it without the start menu being caught off. [09:34.480 –> 09:36.480] Beautiful. [09:36.480 –> 09:43.680] Absolutely bizarre. I think there was some issues if you have played the game on multiple devices, [09:43.680 –> 09:48.000] and you hadn’t had cloud save set up. If you’ve only played on one device, and it’s not an issue, [09:48.000 –> 09:54.960] but yeah, so if you if you’ve played research story, before this update, and you have played [09:54.960 –> 10:01.120] it on multiple devices with the same steam account, you’ll need you might need to do some [10:01.200 –> 10:07.680] jiggery pokery with with the cloud. Is that not a term you would know? [10:09.360 –> 10:16.000] Um, I feel like that is so actually googling it is gives up with a different definition than what [10:16.000 –> 10:24.560] I would assume. So Google says deceitful or dishonest behavior. I would just mean like, [10:24.560 –> 10:30.240] kind of fiddling around with things. Anyway, so you might need to fiddle around with things [10:30.240 –> 10:35.360] in your data structure to get it to work. But they’ve got on steam, they have instructions [10:35.360 –> 10:42.560] on how to do it if if you come up with an issue. Beautiful. And that will be linked in the show [10:42.560 –> 10:49.360] notes. It has like UI improvements as well, which I greatly appreciate as well that they took the [10:49.360 –> 10:55.520] time to like put in very small quality of life improvements. So there you go. I’m done. Yeah, [10:55.520 –> 11:03.360] there’s a bunch of what we’ll have to fill the full update notes in the show notes. Which, [11:03.360 –> 11:08.560] yeah, a lot of these small changes will probably not make it not be important to anybody who other [11:08.560 –> 11:14.080] than people who’ve played it. But yeah, this this, that’s a good point. They are they are improving [11:14.640 –> 11:19.040] things, not just adding what they wanted to, but also changing things based on feedback, [11:19.040 –> 11:23.840] which is always good. I mean, that’s why you do early access. You don’t just do early access [11:23.840 –> 11:27.440] to get your game out earlier. That’s a bad way to do early access, you do early access [11:27.440 –> 11:35.040] to get feedback from the community. Our next update is Sun Down version 1.2. So this is the [11:35.040 –> 11:44.640] one that is from the developers of Sun Haven. And it’s, I think it’s meant to be a rogue light, [11:44.640 –> 11:52.480] or is it just your favorite? I’ve just to come. I lose track of these things. Look, [11:52.480 –> 11:56.560] there’s lots of, it looks like it’s heavily focused on combat. Yeah, that’s what it looks [11:56.560 –> 12:01.200] like. I’m just confused. I’m just confused as to the difference between it and Sun Haven. [12:02.080 –> 12:07.680] But I’ve not played either of the games yet. Yeah, same. Although Sun Haven, Sun Haven wouldn’t [12:07.680 –> 12:13.760] change my switch code to a steam code when I got steam deck. So I haven’t been able to play it yet. [12:13.760 –> 12:19.760] Oh, sad. Yeah. Anyway, so the update has a bunch of a bunch of stuff. I mean, nothing that’s like, [12:19.760 –> 12:23.360] this is really important. If you’ve played the game, the link will be in the show notes. [12:23.360 –> 12:25.440] If you haven’t played the game, you don’t care about what there is. [12:27.520 –> 12:32.320] But it’s just a generic update with characters and content and stuff like that. So [12:33.280 –> 12:39.280] and the third update news, apparently almost all the news is update news this week [12:39.280 –> 12:44.160] is the Apico 2.0 update. So that was the butterflies update, which is already on PC [12:45.040 –> 12:51.760] is coming to PlayStation and switch on the 11th of May. Very exciting. Yeah, yeah. Uh huh. [12:52.320 –> 12:55.120] I still haven’t played this update. So I need to do it at some point. But [12:55.840 –> 12:59.040] I haven’t gotten to the point where I can actually take advantage of it. [13:00.800 –> 13:06.400] Yeah, but I just want to go in to meet Cody. Right. I haven’t met Cody yet either. Yeah, [13:06.400 –> 13:09.440] I want to get to that point. Because yeah, I said this is the butterflies update, [13:09.440 –> 13:14.640] but really is the Cody update. Mm hmm. Yes. So if you want the Cody update, [13:14.640 –> 13:18.080] and you’re only you’re playing the game on PlayStation or switch, you can get it [13:18.720 –> 13:25.360] in just over a week. Yes. Excited. You can get it two days before the Eurovision Eurovision final. [13:25.360 –> 13:29.200] Oh, goodness. Yes. It’s not time. It’s not time. [13:31.840 –> 13:37.360] And the final piece of news that we’ve got is about a new game being announced. We don’t have [13:37.360 –> 13:42.560] a huge amount of information about this game. Because it’s like literally just been talked [13:42.560 –> 13:47.360] about, like we don’t have any kind of information on what is coming out on blah, blah, blah. But [13:47.360 –> 13:55.120] this is called grave seasons. It’s an upcoming project. And it is a farming sim, where each time [13:55.120 –> 14:02.480] you load a new game, randomly one of the villagers is selected to be a serial killer. So it is a [14:02.480 –> 14:13.200] murder mystery farming game. Uh huh. And I love this so much. I so much. Is this sarcasm? I can’t [14:13.200 –> 14:18.240] tell. No, no, no. I think it’s amazing. Well, this is the thing like we were talking about [14:18.240 –> 14:24.640] so many games are just like, Oh, it started you. But it’s this, but it’s that right? Whereas this [14:24.640 –> 14:29.520] is like, completely. Well, it’s not completely different. But like actually has a hook that’s [14:29.520 –> 14:36.640] interesting, not just like words in space. It’s like no one of the one of the people is out to [14:36.640 –> 14:48.400] kill you. So you get to get you can keep track of who’s been murdered. And I just, I love it. [14:48.400 –> 14:53.760] It’s great. I want this game. I want this, but I’m also a little confused by it. Like, [14:53.760 –> 14:58.640] is this a rogue blight? Where like, like, what does it mean when you start the game? Because I [14:58.640 –> 15:03.280] like every time you open up the game? No, no. So it’s like each each new save file, right? So I [15:03.280 –> 15:06.560] don’t think it’s meant to be like, rogue lite. So it’s not meant to be like you play for half [15:06.560 –> 15:12.880] an hour and figure it out. But I think it’s like, okay, I think the idea is that you’ll probably [15:12.880 –> 15:18.080] play this more than once. So it’s kind of, I guess I see what you can think in terms of like, [15:18.080 –> 15:22.160] is it like a rogue like? And I guess the kind of the answer is like, well, you could probably play [15:22.160 –> 15:26.960] it like that and just play until you either. I don’t know whether you can die. You either play [15:26.960 –> 15:30.960] till you die or you figured out who the murderer is. And then you can start a new one and start [15:30.960 –> 15:36.960] again. Like that’s that would probably be like a unexpected way to play this. But you can probably [15:36.960 –> 15:40.240] also just play it like a normal farming sim. And if you figure out who the murderer is, [15:40.240 –> 15:47.280] then you just keep going without them. So, uh huh. Okay, yeah, this is this is good. This is [15:47.280 –> 15:53.120] this is I’m excited for it. Yeah. Now that I know maybe hopefully a little more understanding of what [15:53.120 –> 15:57.120] it might actually look like. Yeah, I mean, well, this is the thing like, oh, there’s so much still [15:57.120 –> 16:03.040] unknown. Like we don’t have any idea like, can you be murdered? Can you keep playing after you’ve [16:03.040 –> 16:09.040] figured out the murderer? Can you accuse somebody and it turns out they’re not the murderer? What [16:09.040 –> 16:13.920] happens then? You know, like, there’s just so many unknowns. What happens if everybody’s murdered [16:13.920 –> 16:19.920] except you? Like, what do you do about these things? I’m just I’m really fascinated to see [16:19.920 –> 16:26.320] what they do with it. I love the idea. Yes, more like this. Yeah. And now we just need somebody to [16:26.320 –> 16:30.720] make the one where you can break up the couples because that’s the one. Oh, my goodness. Uh huh. [16:31.920 –> 16:37.040] I won’t shut up about it till somebody does it. Someone will eventually. I hope so. Yeah, [16:37.040 –> 16:44.160] I’m sure they will. It’s gonna happen. I just need to get like properly famous and then I can [16:44.160 –> 16:48.560] like pay someone to do it. Right. That’s I don’t want to make it myself, right? I don’t want to [16:48.880 –> 16:55.600] figure all that out. I just want it to exist. I want to play it. Mm hmm. You know? Yeah. [16:56.560 –> 17:02.000] We clearly have a lot of ideas on this podcast. So we need to just partner up with a dev, I guess, [17:02.000 –> 17:06.480] and figure out how I don’t even want to partner up. Right. I just want someone to do. I will. [17:08.480 –> 17:13.440] I will. I am a visionary, right? Like, I come up with the overall ideas. I’m like, I want [17:14.080 –> 17:19.040] this. And then I just like, just make it. I don’t, I don’t really care about the details, [17:19.040 –> 17:23.360] right? I’ll care about the details when I’m playing the game. Uh huh. Anyway, [17:25.600 –> 17:28.320] speaking of caring about the details when we’re playing a game, [17:28.320 –> 17:33.600] should we talk about Coral Island? Yes, let’s do it. Alright, so just some quick [17:34.240 –> 17:38.880] overall information before we get down into the ridiculously deep depths of mechanics, [17:38.880 –> 17:47.280] which we always end up doing in this game. We are going to talk about Coral Island. It is [17:47.280 –> 17:55.120] currently in early access on Steam and Game Pass. It’s also going to come out on Xbox, PlayStation, [17:55.120 –> 18:00.240] and Switch at some point. But obviously, early access don’t tend to come to them. [18:01.200 –> 18:08.720] It came in early access on October, in October last year. And there has been two updates so far, [18:08.720 –> 18:14.000] and there’s a third one just about to come out. And I think it’s literally less than two weeks. [18:14.000 –> 18:21.200] I’m so excited. In nine days. I checked. And then I think is it two more updates that are slated [18:21.200 –> 18:26.480] before the final version? I think so. I can’t really scroll on like the screen where it has [18:26.720 –> 18:38.880] the roadmap. So I’m not entirely sure. As far as I can see, they’ve said that the full release is [18:38.880 –> 18:45.280] probably coming out this year. But who knows? Who knows? I’m excited. I hope so. [18:45.280 –> 18:50.400] Let me just double check the roadmap. Oh, there’s more than just two updates. So there’s the summer [18:50.480 –> 18:56.080] update is just just about to come out. And then there’s the merfolk update, the fall update, [18:56.080 –> 19:01.200] the winter update, the end game update, the marriage and children update, and then the 1.0. [19:02.160 –> 19:07.040] Wow. Okay. Yeah, this game isn’t this game isn’t releasing finally this year. Like, [19:07.040 –> 19:13.120] we’ll be lucky if it’s next year. I’m fine with that, honestly. [19:13.120 –> 19:17.440] Oh, sure. Sure. I’m not complaining about it. I’m just saying like, they it’s it’s just one of [19:17.440 –> 19:20.640] these classic things, right? Where they’re like, Oh, yeah, it’ll take us about a year to get all [19:20.640 –> 19:29.120] of the updates. You’re like, nah, no, no chance. No chance, mate. Like every time I see us on early [19:29.120 –> 19:33.680] access, and one of the things you have to answer when you’re doing early access is how long do you [19:33.680 –> 19:39.840] expect this to be an early access? And they’re always like six months or a year. And you’re [19:39.840 –> 19:43.680] like, come on, at least give us something realistic, right? Like, nobody is ever going [19:43.680 –> 19:47.360] to complain that you do it too early. Right? Say three years, it will take me three years, [19:47.360 –> 19:51.600] because like, it could take you three years. And if it doesn’t, everyone’s going to be like, [19:51.600 –> 19:54.480] wow, that’s amazing. They did it in last time than they said they would. [19:54.480 –> 20:00.880] True. Yeah, more people should take that route. Like, over overshoot instead of undershoot. [20:01.840 –> 20:04.960] Absolutely. More people should listen to what I see. Oh, my God. [20:04.960 –> 20:13.200] I agree. Cool. So I guess just overall, this is a very much a Stardew like, [20:14.240 –> 20:18.880] and we can, we’ll have debates, I’m sure later on about whether it does the thing that we want, [20:18.880 –> 20:24.640] which is being different, or whether it’s just the same, but in a different skin on a different [20:25.200 –> 20:32.560] place, etc, etc. But it’s very definitely Stardew. Like, that is absolutely what it’s taken a lot of [20:32.640 –> 20:36.960] stuff from. There’s a lot of stuff in this that are like, is like, that was that was Stardew, [20:36.960 –> 20:41.520] right? Like, yeah, that’s what Stardew did. And therefore, that’s what they do. They have done a [20:41.520 –> 20:47.920] bunch of things on top of that. And I guess the biggest thing is it is a, it is set on an island. [20:48.640 –> 20:54.400] I think it’s meant to be a tropical island. Yeah. Oh, it’s definitely a tropical. I think [20:54.400 –> 21:01.440] that’s the only place that coral actually grows right in the tropics. And they kind of will get [21:01.440 –> 21:06.240] to the specifics of the mechanics. But there’s this, you can go underwater is basically the big, [21:06.240 –> 21:12.080] the big functional change, obviously, like, story wise and stuff. It’s kind of built more around. [21:13.360 –> 21:19.840] It feels like it’s meant to be Pacific Island type thing. But it’s hard to like, it’s obviously not [21:19.840 –> 21:23.760] like setting itself in a specific place. But that’s kind of the vibes I get from [21:24.800 –> 21:31.120] I think it definitely is set in a specific place. Because they reference pokio all the time. [21:31.440 –> 21:41.760] So I think it’s like, maybe like a an island off of Japan is where I’m thinking it would [21:41.760 –> 21:47.280] like geographically land. I mean, I think that would make sense. Because I need to double check. [21:47.280 –> 21:55.200] But I feel like they’re the company’s based in something like, as Indonesia, I was thinking the [21:55.200 –> 22:01.760] Philippines, but Indonesia. Yeah. So that would kind of make, it’s obviously not this exactly [22:01.760 –> 22:06.320] there. But like, that’s, that’s the kind of vague, vague area where we’re talking. [22:08.720 –> 22:16.800] So yeah, kind of West Pacific. And I feel like I would also describe this game as like Sturdy [22:16.800 –> 22:23.120] Valley meets Animal Crossing, more so in the environment. Yeah, that’s, that’s a very good [22:23.120 –> 22:27.760] point. There is a lot that it takes from Animal Crossing as well. I, I think I guess it’s, [22:27.760 –> 22:31.920] it’s really easy to merge those two together. So like, if we talk, for example, about like, [22:31.920 –> 22:38.080] the museum, the museum is very much Animal Crossing. Whereas, you know, Sturdy’s, it’s, [22:38.080 –> 22:44.240] if you actually think about it, Sturdy’s museum is very simple, right? Like, it’s, it’s, it’s just [22:44.240 –> 22:49.440] the artifacts, it doesn’t have anything else. Whereas this is very much, very much Animal [22:49.440 –> 22:54.000] Crossing inspired museum. Which I appreciate. That’s a good, that’s a good point. And there’s, [22:54.000 –> 22:56.960] yeah, there’s some other things and stuff, obviously, like, it’s island based. So you [22:56.960 –> 23:04.480] could argue that blah, blah, blah, blah. Yeah, no, that’s fair. Is that a good introduction? [23:04.480 –> 23:08.000] Have I missed anything introduction wise that we want to talk about? No, I think it’s good. [23:08.000 –> 23:12.640] It’s good. I mean, other than like, the art style being 100% different than Sturdy. [23:12.640 –> 23:17.040] Yeah, yeah. Yeah, that’s, that’s fair. So let’s, let’s get, let’s go into that first [23:17.040 –> 23:22.160] then. Should we talk about the look and feel, the art style, the music, that sort of stuff? [23:22.720 –> 23:29.520] What are your thoughts? I absolutely love it. I think it’s so pretty. Everything is just, [23:29.520 –> 23:34.880] I don’t know, like, like the, the menu screen is just so very pleasing to look at, like the font [23:35.440 –> 23:45.040] choices, the NPC art, the animals, like, I don’t have any complaints, like, about the art style. [23:46.000 –> 23:49.840] I absolutely love it. I think it’s gorgeous and beautiful. [23:51.520 –> 23:56.720] Yeah, I think I’d agree with that is it’s very good looking art. They’ve clearly put a lot of [23:56.720 –> 24:02.720] time and effort into, as you talk about the portraits, the character portraits are really [24:02.720 –> 24:10.720] good. I will say the 3d models of the characters a little bit weird. The kind of the way that they [24:10.800 –> 24:18.160] stand and walk is a little bit awkward. But yeah, like the the character portraits in particular [24:18.160 –> 24:26.240] lovely. The most of the other non people models are great. I don’t know, it might be deliberate, [24:26.240 –> 24:30.960] the kind of weirdness of how they stand and walk. I think so, like a little bit of a swagger, [24:30.960 –> 24:36.880] or like an attempt at swagger. It just, it just feels weird to me. And maybe that’s just a [24:36.880 –> 24:41.200] personal preference thing. I’m not sure but and it doesn’t, it doesn’t kill me. Like it [24:41.200 –> 24:45.840] doesn’t cause me great issues. Right? Like most of the time, I’m not really seeing them anyway. [24:47.280 –> 24:53.600] Because I’m going to be honest, I just basically completely ignored the people in this game. [24:54.640 –> 24:59.520] Other than like, story wise, and the giants, like everyone else, I just completely ignored. [25:00.240 –> 25:02.400] Because there’s only so much you can do. [25:03.360 –> 25:06.720] Yes, you have to focus your time on things. [25:06.720 –> 25:11.680] And so I ignored relationships. Like I have no relationships with anybody other than the giants. [25:11.680 –> 25:13.520] I’ve also taken the same route. [25:15.280 –> 25:23.120] Okay, so we’ll probably talk about the people in another episode. I suspect we’ll end up doing [25:23.120 –> 25:28.560] like a blitz where we do a one panel episode as well. But we’ll see. Maybe we can talk about [25:28.560 –> 25:33.920] people in that episode. I think a lot of it is like, there’s obviously a lot of the relationship [25:33.920 –> 25:39.120] based stuff that isn’t fully implemented as well. So it kind of feels like a sensible one to kind [25:39.120 –> 25:42.960] of skip out at this point. But I know some people really love the relationship stuff. [25:44.240 –> 25:51.200] I don’t hate it in all games, but like, I’m fine with just ignoring that for for something. So [25:53.040 –> 25:55.360] yeah, okay, cool. So music, what about music? [25:55.680 –> 26:02.080] Oh, I feel like I don’t hear the music as much. Even with like, having it like all the way up. [26:02.640 –> 26:08.400] I mostly maybe I just like go into the settings and, and fix that. So I can’t really comment on [26:08.400 –> 26:12.560] the music because I haven’t really like noticed anything. How about you? [26:12.560 –> 26:17.280] I’ve liked the music, I will say it’s the one place I’ve noticed some performance issues. [26:17.280 –> 26:22.320] So I think if we talk about this game in general, like a lot of early access games go early access [26:22.320 –> 26:27.760] to like bug fix. This is done the thing that I like, when they do early access that ooblets did [26:27.760 –> 26:34.800] as well, which was this is like a fully functional part of a game. And the bits that we don’t have [26:34.800 –> 26:40.080] are bits that you can’t do. And you know, it’s not we’ve implemented everything, but it’s all [26:40.080 –> 26:45.200] slightly buggy, and we’ll fix it at some point. It’s no the core loop is there and so solid, [26:45.760 –> 26:50.400] the and the bits that we’ve implemented in terms of like story and all these things, [26:50.400 –> 26:56.640] they are there and they are good and you will not complain about them. You know, there are going to [26:56.640 –> 27:03.040] be missing bits, but that’s mainly content and some features. It’s not, it’s not really bugs. [27:03.040 –> 27:08.800] So I find it to be pretty stable. But yeah, the one issue I found is that sometimes the music can [27:08.800 –> 27:16.400] be a bit clippy, like it will stutter a bit. And so hopefully that will be fixed at some point. [27:16.960 –> 27:21.920] But that’s, that’s the only thing I’ve really noticed performance wise, or kind of buggy, [27:21.920 –> 27:26.560] bugginess wise. There’s been a couple of bits here and there. But like, if this, [27:27.360 –> 27:32.800] if this was the quality of a released game, I think it was perfectly acceptable. Like the [27:32.800 –> 27:39.680] bugs are not huge bugs. It’s like, occasionally, like, I mean, you know, I’ve played a year in this [27:39.680 –> 27:47.520] game. And three times I have tried to go out my house, and it’s just put me back in the house [27:48.560 –> 27:53.040] three times in a year. Right. So like that, that’s the sort of level of bug we’re talking about. [27:53.040 –> 27:57.840] It’s not a huge amount. It’s something that hopefully will be fixed. But it’s not like, [27:57.840 –> 28:00.560] oh, I can’t believe they put this out sort of thing, you know. [28:01.280 –> 28:07.760] I feel like the only bugs I’ve experienced thus far have been like starting the day. So I like [28:07.760 –> 28:13.200] every time I get to the TV and might like restart, or it will crash and then I’ll have [28:13.200 –> 28:17.840] to restart the day. But I had that twice as well when checking the weather, it crashes when checking [28:17.840 –> 28:23.760] the weather. Yeah. I forgot about that. That’s literally happened twice in my year as well. And [28:23.760 –> 28:31.360] yeah, you check the weather every single morning. So that’s not a high number of times. It was [28:31.360 –> 28:36.800] happening pretty consistently at some point for me, like every day. Oh, interesting. Again, like, [28:36.800 –> 28:41.440] it’s such a low complaint because it’s the start of the day. It’s literally the first [28:41.440 –> 28:46.800] thing you do. Yeah. It’s like the annoyances. Oh, I have to load the game up again. Yeah, [28:46.800 –> 28:51.840] I don’t care as much. So I’m just glad that I haven’t lost like a whole day’s worth of like, [28:51.840 –> 28:57.360] yeah, process. That’s true. I haven’t haven’t had a single crash other than that. So I’ve not had [28:57.360 –> 29:02.480] any time where I’ve had to redo something in a day. I thought I had one today, but I didn’t. [29:02.480 –> 29:09.200] It was just being a bit slow waking up. So that’s good, because I’ve had them in stardew. I’ve had [29:09.200 –> 29:15.760] stardew crashes. You know, so like, it’s pretty impressive that they had not a single one where [29:15.760 –> 29:21.280] I’ve had to redo a thing in a day, other than check the weather, which is not difficult. [29:22.320 –> 29:30.960] So that’s pretty impressive. Okay, cool. So that’s kind of just generic look and feel and sound and [29:30.960 –> 29:37.200] how we feel about it as a whole in terms of quality. Let’s talk about some mechanics. So [29:37.200 –> 29:41.440] I guess there’s a bunch of things that are kind of standard mechanics that we can quickly chat [29:41.440 –> 29:44.800] through, but I don’t think we’ll have a huge amount to talk about. So like the farming, [29:45.360 –> 29:51.440] it is standard farming, some farming, right? There is nothing special here. There’s nothing special, [29:51.440 –> 29:57.440] and there’s nothing that really drives me insane. And there’s a few kind of like, whatever, where [29:58.240 –> 30:02.720] as far as I can see, there’s no seed maker in the game yet. Yeah. Or it is in the game, [30:02.720 –> 30:08.160] you just can’t get it. It looks like they had it available, and then they put a level cap on, [30:08.160 –> 30:14.240] so now you can’t get it. Stuff like that, that’s just a bit annoying. And presumably, at some [30:14.240 –> 30:21.200] point, there will be a seed maker that you can get. And, you know, there’s just a bunch of kind [30:21.200 –> 30:29.120] of small things like that, that are just like the scythe, you can’t upgrade for some reason. [30:29.120 –> 30:33.280] Yeah, which is interesting. Very odd, especially as it’s such, we’ll come back to that when we talk [30:33.280 –> 30:38.320] about diving, but it’s like such a fundamental tool that you can upgrade, which is bizarre to me. [30:38.320 –> 30:45.360] Or equipped stuff to it, I guess, like the whatever magic upgrades. You can do that. Oh, [30:45.360 –> 30:49.520] it’s just because you can’t upgrade it, you can only do one. Okay, maybe I did do that. I just [30:49.520 –> 30:54.640] forgot. Okay. Yeah, because I checked that yesterday. I did one. Okay, nevermind. I [30:54.640 –> 31:02.240] will check that. You can add the enchantments, I think it’s called. Okay. But yeah, you can upgrade [31:02.240 –> 31:08.560] it. You can only do one enchantment. It’s a bit annoying. Yeah, I mean, I don’t really have [31:08.560 –> 31:13.520] anything really interesting to say about the farming specifically. I mean, I think that the [31:13.520 –> 31:19.040] farming is interesting in that there’s a difference, the different materials needed for crafting the [31:19.120 –> 31:23.600] sprinklers, which I thought was an interesting choice. Yeah, I’m annoyed about that. [31:25.120 –> 31:28.080] I’m not because I let me tell you, I’m annoyed about that. [31:29.840 –> 31:34.800] So I don’t like I’m not against them doing different things, right? And that’s fine. [31:35.840 –> 31:42.400] But my annoyance with it is, it’s basically as far as I can tell, it’s actually impossible without [31:42.400 –> 31:46.880] spending an insane amount of money to do that. Because you can’t actually get the gold kelp in [31:46.880 –> 31:55.520] the game. Yeah. So you have you can buy it, but it’s like 1500 pair gold kelp, and you need five [31:55.520 –> 32:02.480] gold kelp and five silver kelp to build a build a decent sprinkler. There’s a there’s a three by [32:02.480 –> 32:09.120] three sprinkler sprinkler, but that’s basically useless, right? So that is just a bit annoying [32:09.120 –> 32:15.360] that you have that something so kind of fundamental to me, a sprinkler is so difficult [32:15.440 –> 32:21.040] to do. And even the three by three, you have to progress the diving so far to get to that point, [32:21.920 –> 32:27.120] where you can actually do that, you know, it seems like it’s so hard to do, it feels like it’s almost [32:27.120 –> 32:32.480] like a level higher than it should be. Right? Like the the five by five should be the one that has [32:32.480 –> 32:37.600] silver kelp, the three by three should have bronze kelp. And then there should be another one on top [32:37.600 –> 32:43.200] that does the the gold kelp that it kind of feels like they’re missing a sprinkler there. [32:43.200 –> 32:48.160] I don’t know, I kind of disagree, because I almost feel like, like, I got it in a normal [32:48.160 –> 32:54.000] progression, I feel like, as I would have started, had I just been upgrading them via, [32:54.000 –> 33:00.000] you know, bars, iron bars, or gold bars, or what have you. So I feel like it, the progression [33:00.000 –> 33:06.000] made sense in terms of like, the different obstacles they put in place to slow you down [33:06.000 –> 33:11.280] with that development. Like, I think I got to summer, and I was able to, to upgrade them to [33:11.280 –> 33:17.840] the eight by eight. And that worked fine for me. So I would kind of, I think, disagree there. [33:17.840 –> 33:19.440] You mean the five by five ones? [33:19.440 –> 33:21.120] Yes. Wait. [33:21.120 –> 33:22.880] So how did you get the gold kelp? [33:22.880 –> 33:26.880] Oh, no, no, I like the eight tiles were ill. The three by three. [33:26.880 –> 33:30.080] Yeah, the three by three. Yeah, yeah, the three by three is the basic one. [33:31.280 –> 33:35.200] And they’re reasonably easy to get. It doesn’t take a ridiculous amount of time to get to them. [33:35.200 –> 33:38.800] But like, I’m in the second year, and I have not even had a single thing of gold kelp. [33:39.760 –> 33:44.480] I feel like my current complaint is not really about like, the being able to access the the [33:44.480 –> 33:49.360] highest gold kelp one. It’s the fact that sprinkler one and sprinkler two are essentially the same [33:49.360 –> 33:54.800] thing. Like they both do three by three tiles, like eight, eight around and there’s no [33:55.920 –> 33:56.560] Check this out. [33:57.360 –> 34:02.880] Because I thought I was just aware of two sprinklers. And I thought it was three by [34:02.880 –> 34:08.080] three and five by five. Yeah, it is. Sprinkle two is sprinkle two is five by five, not three by [34:08.080 –> 34:14.480] three. I don’t know. I feel like that’s okay. Well, that’s what it is. That’s what it says. [34:14.480 –> 34:20.800] That’s what it says in game. That’s weird. Okay. Great. I’ve seen spring. No, it doesn’t. [34:20.800 –> 34:24.800] No, it doesn’t. It says it says in game eight. Yeah. In game it says eight. [34:27.920 –> 34:34.560] The wiki says 24. Oh, is it just labeled mislabeled? And I never tried building a higher [34:34.560 –> 34:40.640] one for that reason. I think it might be. I suspect it’s an incorrect label. It’s also [34:40.640 –> 34:44.560] it’s also incorrect when it says because it says a radius of eight tiles. It doesn’t mean a radius [34:44.560 –> 34:50.160] of eight tiles. That would that would be that would be what? So eight and eight. That’d be 16 [34:50.160 –> 34:58.480] by 16, which would be incredible. Yeah, it’s unclear exactly what it does then because I [34:58.480 –> 35:03.600] haven’t been able to craft one. Because you need five gold kelp for it and haven’t even had one. [35:04.560 –> 35:11.520] Um, you can buy the gold kelp. But if you were to buy that five gold kelp, it would be 7500, [35:11.520 –> 35:17.200] which I mean, I have the money for now. But like, I would just be buying if I like, [35:17.200 –> 35:27.840] I’m currently at the point where I have, I’ve got 11 22 by 22 farming plots, right? So to try and [35:27.840 –> 35:35.280] cover that with, I would need 20 sprinklers to cover everything with sprinklers, 20 of those [35:35.280 –> 35:46.960] those five by five ones. And so that would give us 100 gold kelp, which would cost 150,000, [35:47.520 –> 35:50.640] which obviously I don’t have them. Oh, my goodness. Yeah, that’s gonna take that’s like [35:50.640 –> 35:55.920] endgame stuff. I think that’s my point. I think that’s my point is the sprinkler two feels [35:56.800 –> 36:02.240] so difficult to get to that. And I don’t even know if it’s possible because I have not seen [36:02.240 –> 36:07.520] Have you seen any gold kelp in the game? I haven’t. But like, part of the so I think [36:07.520 –> 36:12.400] the issue with the gold kelp is that the the area that it’s located is not accessible right now. [36:13.120 –> 36:16.480] So well, that’s, that’s, that’s the question. I don’t know whether that’s the case or not. [36:16.480 –> 36:20.400] So part of the problem with this game is that it tells you about things that don’t exist in [36:20.400 –> 36:24.800] the game yet. And so it’s hard to know whether something is possible in the game. And it’s just [36:24.800 –> 36:31.440] telling you about it or not possible in the game. I promise you that it will be I think possible [36:31.440 –> 36:38.080] because I have like looked at the map and I have like mapped it out in that I’ve tried to access [36:38.080 –> 36:43.600] every single part of the map. And I physically cannot access like the north eastern corner. [36:43.600 –> 36:50.480] Like there’s just barriers there that the underwater Yeah, right. Yeah. So I think it [36:50.480 –> 36:56.240] will be there. It’s just gonna I think it’s a content that they need a update. It’s a content [36:56.240 –> 37:00.720] update that they haven’t gotten around to yet. I think this is probably my only thing that I [37:00.720 –> 37:06.080] think they needed to do better as an early access is do better with telling us what is available and [37:06.080 –> 37:11.840] not. Because it’s a bit frustrating to get to the point where like, okay, well, I’m aiming towards [37:11.840 –> 37:15.360] this. Okay, I’ve got to the area where I’ve got the silver kelp. That’s great. I shouldn’t be [37:15.360 –> 37:20.080] too much longer till I get to the gold kelp. And then you just never get it. Yeah. And you don’t [37:20.080 –> 37:25.200] know whether you’ve done something wrong, or whether it’s not available. And it’s not a problem [37:25.200 –> 37:29.920] these things aren’t available. But I think they need to make it clear. Like, first of all, [37:29.920 –> 37:39.200] why give you the why have they given the crafting blueprint to us? Like, why is it possible to get [37:39.200 –> 37:45.360] crafting blueprint that you can’t do? That’s the weird thing to me. Like I have the crafting thing [37:45.360 –> 37:51.360] for the second sprinkler. But I can’t get the kelp to do that. That that feels like a kind of [37:51.360 –> 37:59.120] mismatch there and, and expectations of the game. I feel like it could it could be a little bit of [37:59.120 –> 38:07.120] wanting to appease people who want a more efficient way to farm, but are willing to grind [38:07.120 –> 38:12.960] for that method. And but you can’t grind Oh, you mean because you can get the you can buy the gold [38:12.960 –> 38:20.400] cap. Yeah. And I just I feel like there needs to be a way in game that they would tell you that [38:20.400 –> 38:27.680] that’s the case. Like, I mean, I suspect with other things, like they’ve told you with other [38:27.680 –> 38:33.200] things like this is not accessible yet. Like, there’s a couple events like scenes or cut scenes. [38:33.200 –> 38:40.160] So they, in a way they have, they just fail to tell you that there’s the gold is not the gold [38:40.720 –> 38:47.120] kelp is not accessible other than buying it. Yeah, it’s not just that. And so there’s also the [38:47.120 –> 38:52.240] the osmium ore is not possible to get in game either. And you can still get things [38:52.240 –> 38:58.720] crafting resources and blueprints for that that would I’d like you can, and you can assume that [38:58.720 –> 39:02.960] it’s not available because you can’t get to the final bit of the mine. But like, that’s an [39:02.960 –> 39:07.760] assumption. That’s not a definite, right? So there’s a there’s a few other things as well [39:07.760 –> 39:15.280] that’s like that what is it gives you something that you could craft or or buy that requires [39:15.280 –> 39:18.640] something that you can’t get in the game. And there’s no indication that you can get that in [39:18.640 –> 39:24.800] the game. And with the osmium ore, you can’t even buy that one. Maybe, maybe in an update, [39:24.800 –> 39:32.000] they’ll update what you can again. Because I think they tried it, they made the attempt at telling [39:32.000 –> 39:36.400] you like, this is physically not available in the game right now, or, you know, virtually whatever [39:36.400 –> 39:41.520] you want to call it. But then they just, I think, miss these portions and then forgot to [39:42.880 –> 39:50.240] the implemented it for events. Yeah, and areas. They didn’t implement it for items. Okay. That’s [39:50.240 –> 39:56.880] like, I don’t I’ve not been aware of any item where it is shown on screen, you can’t get it [39:56.880 –> 40:00.880] and they tell you that you can’t get it. Right? You always just have to assume that for that. [40:01.680 –> 40:07.520] And I like, you know, for example, like, so we have the, there’s the temple, the sea, the [40:08.240 –> 40:12.960] Lake Temple, which is your, here we go, we’re always comparing back to Stardew. [40:12.960 –> 40:17.520] It’s your community center, right? Like you have, you go there and you give offerings of [40:17.520 –> 40:24.800] certain things. There’s no, so first of all, there’s no way to go on to the things actually [40:24.800 –> 40:30.080] know what they’re called. Right? So like, if I’m looking at a list of things in a in a bundle, [40:30.080 –> 40:35.600] there’s no way to know what’s in that bundle, except by just guessing based on the the image, [40:35.600 –> 40:39.920] which in some is obvious, right? Like, oh, that’s clearly a daisy, right? You know, that’s a daisy [40:39.920 –> 40:44.720] because that’s a daisy you can you can buy daisy seeds. That’s easy. That one’s clearly wheat. [40:44.720 –> 40:49.440] But like, Brussels sprouts. Did you know what the Brussels sprouts was before you find a [40:49.440 –> 40:54.960] Brussels sprout? No, it is not obvious. Yeah, yeah, that’s

The most recent YCombinator W23 batch graduated 59 companies building with Generative AI for everything from sales, support, engineering, data, and more:Many of these B2B startups will be seeking to establish an AI foothold in the enterprise. As they look to recent success, they will find Glean, started in 2019 by a group of ex-Googlers to finally solve AI-enabled enterprise search. In 2022 Sequoia led their Series C at a $1b valuation and Glean have just refreshed their website touting new logos across Databricks, Canva, Confluent, Duolingo, Samsara, and more in the Fortune 50 and announcing Enterprise-ready AI features including AI answers, Expert detection, and In-context recommendations.We talked to Deedy Das, Founding Engineer at Glean and a former Tech Lead on Google Search, on why he thinks many of these startups are solutions looking for problems, and how Glean's holistic approach to enterprise probllem solving has brought so much success. Deedy is also just a fascinating commentator on AI current events, being both extremely qualified and great at distilling insights, so we also went over his many viral tweets diving into Google's competitive threats, AI Startup investing, and his exposure of Indian University Exam Fraud!Show Notes* Deedy on LinkedIn and Twitter and Personal Site* Glean* Glean and Google Moma* Golinks.io* Deedy on Google vs ChatGPT* Deedy on Google Ad Revenue* Deedy on How much does it cost to train a state-of-the-art foundational LLM?* Deedy on Google LaMDA cost* Deedy's Indian Exam Fraud Story* Lightning Round* Favorite Products: (covered in segment)* Favorite AI People: AI Pub* Predictions: Models will get faster for the same quality* Request for Products: Hybrid Email Autoresponder* Parting Takeaway: Read the research!Timestamps* [00:00:21] Introducing Deedy* [00:02:27] Introducing Glean* [00:05:41] From Syntactic to Semantic Search* [00:09:39] Why Employee Portals* [00:12:01] The Requirements of Good Enterprise Search* [00:15:26] Glean Chat?* [00:15:53] Google vs ChatGPT* [00:19:47] Search Issues: Freshness* [00:20:49] Search Issues: Ad Revenue* [00:23:17] Search Issues: Latency* [00:24:42] Search Issues: Accuracy* [00:26:24] Search Issues: Tool Use* [00:28:52] Other AI Search takes: Perplexity and Neeva* [00:30:05] Why Document QA will Struggle* [00:33:18] Investing in AI Startups* [00:35:21] Actually Interesting Ideas in AI* [00:38:13] Harry Potter IRL* [00:39:23] AI Infra Cost Math* [00:43:04] Open Source LLMs* [00:46:45] Other Modalities* [00:48:09] Exam Fraud and Generated Text Detection* [00:58:01] Lightning RoundTranscript[00:00:00] Hey everyone. Welcome to the Latent Space Podcast. This is Alessio, partner and CTO and residence at Decibel Partners. I'm joined by my, cohost swyx, writer and editor of[00:00:19] Latent Space. Yeah. Awesome.[00:00:21] Introducing Deedy[00:00:21] And today we have a special guest. It's Deedy Das from Glean. Uh, do you go by Deedy or Debarghya? I go by Deedy. Okay.[00:00:30] Uh, it's, it's a little bit easier for the rest of us to, uh, to, to spell out. And so what we typically do is I'll introduce you based on your LinkedIn profile, and then you can fill in what's not on your LinkedIn. So, uh, you graduated your bachelor's and masters in CS from Cornell. Then you worked at Facebook and then Google on search, specifically search, uh, and also leading a sports team focusing on cricket.[00:00:50] That's something that we, we can dive into. Um, and then you moved over to Glean, which is now a search unicorn in building intelligent search for the workplace. What's not on your LinkedIn that people should know about you? Firstly,[00:01:01] guys, it's a pleasure. Pleasure to be here. Thank you so much for having me.[00:01:04] What's not on my LinkedIn is probably everything that's non-professional. I think the biggest ones are I'm a huge movie buff and I love reading, so I think I get through, usually I like to get through 10 books ish a year, but I hate people who count books, so I should say the number. And increasingly, I don't like reading non-fiction books.[00:01:26] I actually do prefer reading fiction books purely for pleasure and entertainment. I think that's the biggest omission from my LinkedIn.[00:01:34] What, what's, what's something that, uh, caught your eye for fiction stuff that you would recommend people?[00:01:38] Oh, I recently, we started reading the Three Body Problem and I finished it and it's a three part series.[00:01:45] And, uh, well, my controversial take is I did not really enjoy the second part, and so I just stopped. But the first book was phenomenal. Great concept. I didn't know you could write alien fiction with physics so Well, and Chinese literature in particular has a very different cadence to it than Western literature.[00:02:03] It's very less about the, um, let's describe people and what they're all about and their likes and dislikes. And it's like, here's a person, he's a professor of physics. That's all you need to know about him. Let's continue with the story. Um, and, and I, I, I, I enjoy it. It's a very different style from, from what I'm used.[00:02:21] Yeah, I, I heard it's, uh, very highly recommended. I think it's being adapted to a TV show, so looking forward[00:02:26] to that.[00:02:27] Introducing Glean[00:02:27] Uh, so you spend now almost four years at gle. The company's not unicorn, but you were on the founding team and LMS and tech interfaces are all the reach now. But you were building this before.[00:02:38] It was cool, so to speak. Maybe tell us more about the story, how it became, and some of the technological advances you've seen. Because I think you started, the company started really close to some of the early GPT models. Uh, so you've seen a lot of it from, from day one.[00:02:53] Yeah. Well, the first thing I'll say is Glean was never started to be a.[00:02:58] Technical product looking for a solution. We were always wanted to solve a very critical problem first that we saw, not only in the companies that we'd worked in before, but in all of the companies that a lot of our, uh, a lot of the founding team had been in past their time at Google. So Google has a really neat tool that already kind of does this internally.[00:03:18] It's called MoMA, and MoMA sort of indexes everything that you'd use inside Google because they have first party API accessed who has permissions to what document and what documents exist, and they rank them with their internal search tool. It's one of those things where when you're at Google, you sort of take it for granted, but when you leave and go anywhere else, you're like, oh my God, how do I function without being able to find things that I've worked on?[00:03:42] Like, oh, I remember this guy had a presentation that he made three meetings ago and I don't remember anything about it. I don't know where he shared it. I don't know if he shared it, but I do know the, it was a, something about X and I kind of wanna find that now. So that's the core. Information retrieval problem that we had set out to tackle, and we realized when we started looking at this problem that enterprise search is actually, it's not new.[00:04:08] People have been trying to tackle enterprise search for decades. Again, pre two thousands people have been trying to build these on-prem enterprise search systems. But one thing that has really allowed us to build it well, A, you now have, well, you have distributed elastic, so that really helps you do a lot of the heavy lifting on core infra.[00:04:28] But B, you also now have API support that's really nuanced on all of the SaaS apps that you use. So back in the day, it was really difficult to integrate with a messaging app. They didn't have an api. It didn't have any way to sort of get the permissions information and get the messaging information. But now a lot of SaaS apps have really robust APIs that really let.[00:04:50] Index everything that you'd want though though. That's two. And the third sort of big macro reason why it's happening now and why we're able to do it well is the fact that the SaaS apps have just exploded. Like every company uses, you know, 10 to a hundred apps. And so just the urgent need for information, especially with, you know, remote work and work from home, it's just so critical that people expect this almost as a default that you should have in your company.[00:05:17] And a lot of our customers just say, Hey, I don't, I can't go back to a life without internal search. And I think we think that's just how it should be. So that's kind of the story about how Glean was founded and a lot of the LLM stuff. It's neat that all, a lot of that's happening at the same time that we are trying to solve this problem because it's definitely applicable to the problem we're trying to solve.[00:05:37] And I'm really excited by some of the stuff that we are able to do with it.[00:05:41] From Syntactic to Semantic Search[00:05:41] I was talking with somebody last weekend, they were saying the last couple years we're going from the web used to be syntex driven. You know, you siegal for information retrieval, going into a symantics driven where the syntax is not as important.[00:05:55] It's like the, how you actually explain the question. And uh, we just asked Sarah from Seek.ai on the previous episode and instead of doing natural language and things like that for enterprise knowledge, it's more for business use cases. So I'm curious to see, you know, The enterprise of the future, what that looks like, you know, is there gonna be way less dropdowns and kind of like, uh, SQL queries and stuff like that.[00:06:19] And it's more this virtual, almost like person that embodies the company that is like a, an LLM in a way. But how do you do that without being able to surface all the knowledge that people have in the organization? So something like Lean is, uh, super useful for[00:06:35] that. Yeah, I mean, already today we see these natural language queries as well.[00:06:39] I, I will say at, at this point, it's still a small fraction of the queries. You see a lot of, a lot of the queries are, hey, what is, you know, just a name of a project or an acronym or a name of a person or some someone you're looking for. Yeah, I[00:06:51] think actually the Glean website explains gleans features very well.[00:06:54] When I, can I follow the video? Actually, video wasn't that, that informative video was more like a marketing video, but the, the actual website was showing screenshots of what you see there in my language is an employee portal. That happens to have search because you also surface like collections, which proactively show me things without me searching anything.[00:07:12] Right. Like, uh, you even have Go links, you should copy it, I think from Google, right? Which like, it's basically, uh, you know, in my mind it's like this is ex Googlers missing Google internal stuff. So they just built it for everyone else. So,[00:07:25] well, I can, I can comment on that. So a, I should just plug that we have a new website as of today.[00:07:30] I don't know how, how it's received. So I saw it yesterday, so let, let me know. I think today we just launch, I don't know when we launched a new one, I think today or yesterday. Yeah,[00:07:38] it's[00:07:38] new. I opened it right now it's different than yesterday.[00:07:41] Okay. It's, it's today and yeah. So one thing that we find is that, Search in itself.[00:07:48] This is actually, I think, quite a big insight. Search in itself is not a compelling enough use case to keep people drawn to your product. It's easy to say Google search is like that, but Google Search was also in an era where that was the only website people knew, and now it's not like that. When you are a new tool that's coming into a company, you can't sit on your high horse and say, yeah, of course you're gonna use my tool to search.[00:08:13] No, they're not gonna remember who you are. They're gonna use it once and completely forget to really get that retention. You need to sort of go from being just a search engine to exactly what you said, Sean, to being sort of an employee portal that does much more than that. And yeah, the Go Links thing, I, I mean, yes, it is copied from Google.[00:08:33] I will say there's a complete other startup called Go links.io that has also copied it from Google and, and everyone, everyone misses Go Links. It's very useful to be able to write a document and just be like, go to go slash this. And. That's where the document is. And, and so we have built a big feature set around it.[00:08:50] I think one of the critical ones that I will call out is the feed. Just being able to see, not just, so documents that are trending in your sub-organization documents that you, we think you should see are a limited set of them, as well as now we've launched something called Mentions, which is super useful, which is all of your tags across all of your apps in one place in the last whatever, you know, time.[00:09:14] So it's like all of the hundred Slack pings that you have, plus the Jira pings, plus the, the, the email, all of that in one place is super useful to have. So you did GitHub. Yeah, we do get up to, we do get up to all the mentions.[00:09:28] Oh my God, that's amazing. I didn't know you had it, but, uh, um, this is something I wish for myself.[00:09:33] It's amazing.[00:09:34] It's still a little buggy right now, but I think it's pretty good. And, and we're gonna make it a lot better as as we go.[00:09:39] Why Employee Portals[00:09:39] This[00:09:39] is not in our preset list of questions, but I have one follow up, which is, you know, I've worked in quite a few startups now that don't have employee portals, and I've worked at Amazon, which had an employee portal, but it wasn't as beautiful or as smart as as glean.[00:09:53] Why isn't this a bigger norm in all[00:09:56] companies? Well, there's several reasons. I would say one reason is just the dynamics of how enterprise sales happens is. I wouldn't say broken. It is, it is what it is, but it doesn't always cater to employees being happy with the best tools. What it does cater to is there's different incentive structures, right?[00:10:16] So if I'm an IT buyer, I have a budget and I need to understand that for a hundred of these tools that are pitched to me all the time, which ones really help the company And the way usually those things are evaluated is does it increase revenue and does it cut cost? Those are the two biggest ones. And for a software like Glean or a search portal or employee portal, it's actually quite difficult when you're in, generally bucketed in the space of productivity to say, Hey, here's a compelling use use case for why we will cut your cost or increase your revenue.[00:10:52] It's just a softer argument that you have to make there. It's just a fundamental nature of the problem versus if you say, Hey, we're a customer support tool. Everyone in SaaS knows that customer support tools is just sort of the. The last thing that you go to when you're looking for ideas, because it's easy to sell.[00:11:08] It's like, here's a metric. How many tickets can your customer support agent resolve? We've built a thing that makes it 20% better. That means it's 1,000 thousand dollars cost savings. Pay us 50 k. Call it a deal. That's a good argument. That's a very simple, easy to understand argument. It's very difficult to make that argument with search, which you're like, okay, you're gonna get see about 10 to 20 searches that's gonna save about this much time, uh, a day.[00:11:33] And that results in this much employee productivity. People just don't buy it as easily. So the first reaction is, oh, we work fine without it. Why do we need this now? It's not like the company didn't work without this tool, and uh, and only when they have it do they realize what they were missing out on.[00:11:50] So it's a difficult thing to sell in, in some ways. So even though the product is, in my opinion, fantastic, sometimes the buyer isn't easily convinced because it doesn't increase revenue or cut cost.[00:12:01] The Requirements of Good Enterprise Search[00:12:01] In terms of technology, can you maybe talk about some of the stack and you see a lot of companies coming up now saying, oh, we help you do enterprise search.[00:12:10] And it's usually, you know, embedding to then do context for like a LLM query mostly. I'm guessing you started as like closer to like the vector side of thing maybe. Yeah. Talk a bit about that and some learning siva and as founders try to, to build products like this internally, what should they think[00:12:27] about?[00:12:28] Yeah, so actually leading back from the last answer, one of the ways a lot of companies who are in the enterprise search space are trying to tackle the problem of sales is to lean into how advance the technology is, which is useful. It's useful to say we are AI powered, LLM powered vector search, cutting edge, state-of-the-art, yada, yada, yada.[00:12:47] Put it all your buzzwords. That's nice, but. The question is how often does that translate to better user experience is sort of, a fuzzy area where it, it's really hard for even users to tell, to be honest. Like you can have one or two great queries and one really bad query and be like, I don't know if this thing is smart.[00:13:06] And it takes time to evaluate and understand how a certain engine is doing. So to that, I think one of the things that we learned from Google, a lot of us come from an ex Google search background, and one of the key learnings is often with search, it's not about how advanced or how complex the technology is, it's about the rigor and intellectual honesty that you put into tuning the ranking algorithm.[00:13:30] That's a painstaking long-term and slow process at Google until I would say maybe 20 17, 20 18. Everything was run off of almost no real ai, so to speak. It was just information retrieval at its core, very basic from the seventies, eighties, and a bunch of these ranking components that are put stacked on top of it that do various tasks really, really well.[00:13:57] So one task in search is query understanding what does the query mean? One task is synonymous. What are other synonyms for this thing that we can also match on? One task is document understanding. Is this document itself a high quality document or not? Or is it some sort of SEO spam? And admittedly, Google doesn't do so well on that anymore, but there's so many tough sub problems that it breaks search down into and then just gets each of those problems, right, to create a nice experience.[00:14:24] So to answer your question, also, vector search we do, but it is not the only way we get results. We do a hybrid approach both using, you know, core IR signal synonymy. Query accentuation with things like acronym expansion, as well as stuff like vector search, which is also useful. And then we apply our level of ranking understanding on top of that, which includes personalization, understanding.[00:14:50] If you're an engineer, you're probably not looking for Salesforce documents. You know, you're probably looking for documents that are published or co-authored by people in your team, in your immediate team, and our understanding of all of your interactions with people around you. Our personalization layer, our good work on ranking is what makes us.[00:15:09] Good. It's not sort of, Hey, drop in LLM and embeddings and we become amazing at search. That's not how we think it[00:15:16] works. Yeah. I think there's a lot of polish that mix into quality products, and that's the difference that you see between Hacker News, demos and, uh, glean, which is, uh, actual, you know, search and chat unicorn.[00:15:26] Glean Chat?[00:15:26] But also is there a glean chat coming? Is is, what do you think about the[00:15:30] chat form factor? I can't say anything about it, but I think that we are experi, my, my politically correct answer is we're experimenting with many technologies that use modern AI and LLMs, and we will launch what we think users like best.[00:15:49] Nice. You got some media training[00:15:51] again? Yeah. Very well handed.[00:15:53] Google vs ChatGPT[00:15:53] We can, uh, move off of Glean and just go into Google search. Uh, so you worked on search for four years. I've always wanted to ask what happens when I type something into Google? I feel like you know more than others and you obviously there's the things you cannot say, but I'm sure Google does a lot of the things that Glean does as well.[00:16:08] How do you think about this Google versus ChatGPT debate? Let's, let's maybe start at a high level based on what you see out there, and I think you, you see a lot of[00:16:15] misconceptions. Yeah. So, okay, let me, let me start with Google versus ChatGPT first. I think it's disingenuous, uh, if I don't say my own usage pattern, which is I almost don't go back to Google for a large section of my queries anymore.[00:16:29] I just use ChatGPT I am a paying plus subscriber and it's sort of my go-to for a lot of things. That I ask, and I also have to train my mind to realize that, oh, there's a whole set of questions in your head that you never realize the internet could answer for you, and that now you're like, oh, wait, I could actually ask this, and then you ask it.[00:16:48] So that's my current usage pattern. That being said, I don't think that ChatGPT is the best interface or technology for all sets of queries. I think humans are obviously very easily excited by new technology, but new technology does not always mean the previous technology was worse. The previous technology is actually really good for a lot of things, and for search in particular, if you think about all the queries that come into Google search, they fall into various kinds of query classes, depending on whatever taxonomy you want to use.[00:17:24] But one sort of way of, of of understanding broad, generally, the query classes is something that is information seeking or exploratory. And for information for exploratory queries. I think there are uses where Google does really well. Like for example, let's say you want to just know a list of songs of this artist in this year.[00:17:49] Google will probably be able to add a hundred percent, tell you that pretty accurately all the time. Or if you want to say understand like what showtimes of movies came out today. So fresh queries, another query class, Google will be really good at that chat, not so good at that. But if you look at information seeking queries, you could even argue that if I ask for information about Donald Trump, Maybe ChatGPT will spit out a reasonable sounding paragraph and it makes sense, but it doesn't give me enough stuff to like click on and go to and navigate to in a news article here.[00:18:25] And I just kind wanna see a lot of stuff happening. So if you really break down the problem, I think it's not as easy as saying ChatGPT is a silver bullet for every kind of information need. There's a lot of information needs, especially for tail queries. So for long. Un before seen queries like, Hey, tell me the cheat code in Doom three.[00:18:43] This level, this boss ChatGPTs gonna blow it out the water on those kind of queries cuz it's gonna figure out all of these from these random sparse documents and random Reddit threads and assemble one consistent answer for you where it takes forever to find this kind of stuff on Google. For me personally, coding is the biggest use case for anything technical.[00:19:02] I just go to ChatGPT cuz parsing through Stack Overflow is just too mentally taxing and I don't care about, even if ChatGPT hallucinates a wrong answer, I can verify that. But I like seeing a coherent, nice answer that I can just kind of good starting point for my research on whatever I'm trying to understand.[00:19:20] Did you see the, the statistic that, uh, the Allin guys have been saying, which is, uh, stack overflow traffic is down 15%? Yeah, I did, I did.[00:19:27] See that[00:19:28] makes sense. But I, I, I don't know if it's like only because of ChatGPT, but yeah, sure. I believe[00:19:33] it. No, the second part was just about if some of the enterprise product search moves out of Google, like cannot, that's obviously a big AdWords revenue driver.[00:19:43] What are like some of the implications in terms of the, the business[00:19:46] there?[00:19:47] Search Issues: Freshness[00:19:47] Okay,[00:19:47] so I would split this answer into two parts. My first part is just talking about freshness, cuz the query that you mentioned is, is specifically the, the issue there is being able to access fresh information. Google just blanket calls his freshness.[00:20:01] Today's understanding of large language models is that it cannot do anything that's highly fresh. You just can't train these things fast enough and cost efficiently enough to constantly index new, new. Sources of data and then serve it at the same time in any way that's feasible. That might change in the future, but today it's not possible.[00:20:20] The best thing that you can get that's close to it is what, you know, the fancy term is retrieval, augmented generation, but it's a fancy way of saying just do the search in the background and then use the results to create the actual response. That's what Bing does today. So to answer the question about freshness, I would say it is possible to do with these methods, but those methods all in all involve using search in the backend to, to sort of get the context to generate the answer.[00:20:49] Search Issues: Ad Revenue[00:20:49] The second part of the answer is, okay, talk about ad revenue. A lot of Google's ad revenue just comes from the fact that over the last two decades, it's figured out how to put ad links on top of a search result page that sometimes users click. Now the user behavior on a chat product is not to click on anything.[00:21:10] You don't click on stuff you just read and you move on. And that actually, in my opinion, has severe impacts on the web ecosystem, on all of Google and all of technology and how we use the internet in the future. And, and the reason is one thing we also take for granted is that this ad revenue where everyone likes to say Google is bad, Google makes money off ads, yada, yada, yada, but this ad revenue kind of sponsored the entire internet.[00:21:37] So you have Google Maps and Google search and photos and drive and all of this great free stuff basically because of ads. Now, when you have this new interface, sure it, it comes with some benefits, but if users aren't gonna click on ads and you replace the search interface with just chat, that can actually be pretty dangerous in terms of what it even means.[00:21:59] To have to create a website, like why would I create a website if no one's gonna come to my. If it's just gonna be used to train a model and then someone's gonna spit out whatever my website says, then there's no incentive. And that kind of dwindles the web ecosystem. In the end, it means less ad revenue.[00:22:15] And then the other existential question is, okay, I'm okay with saying the incumbent. Google gets defeated and there's this new hero, which is, I don't know, open AI and Microsoft. Now reinvent the wheel. All of that stuff is great, but how are they gonna make money? They can make money off, I guess, subscriptions.[00:22:31] But subscriptions is not nearly gonna make you enough. To replace what you can make on ad revenue. Even for Bing today. Bing makes it 11 billion off ad revenue. It's not a society product like it's a huge product, and they're not gonna make 11 billion off subscriptions, I'll tell you that. So even they can't really replace search with this with chat.[00:22:51] And then there are some arguments around, okay, what if you start to inject ads in textual form? But you know, in my view, if the natural user inclination is not to click on something or chat, they're clearly not gonna click on something. No matter how much you try to inject, click targets into your result.[00:23:10] So, That's, that's my long answer to the ads question. I don't really know. I just smell danger in the horizon.[00:23:17] Search Issues: Latency[00:23:17] You mentioned the information augmented generation as well. Uh, I presumably that is literally Bing is probably just using the long context of GPT4 and taking the full text of all the links that they find, dumping it in, and then generating some answer.[00:23:34] Do you think like speed is a concern or people are just people willing to wait for smarter?[00:23:40] I think it's a concern. We noticed that every, every single product I've worked on, there's almost a linear, at least for some section of it, a very linear curve. A linear line that says the more the latency, the less the engagement, so there's always gonna be some drop off.[00:23:55] So it is a concern, but with things like latency, I just kind of presume that time solves these things. You optimize stuff, you make things a little better, and the latency will get down with time. And it's a good time to even mention that. Bard, we just came out today. Google's LLM. For Google's equivalent, I haven't tried it, but I've been reading about it, and that's based off a model called LamDA.[00:24:18] And LamDA intrinsically actually does that. So it does query what they call a tool set and they query search or a calculator or a compiler or a translator. Things that are good at factual, deterministic information. And then it keeps changing its response depending on the feedback from the tool set, effectively doing something very similar to what Bing does.[00:24:42] Search Issues: Accuracy[00:24:42] But I like their framing of the problem where it's just not just search, it's any given set of tools. Which is similar to what a Facebook paper called Tool Former, where you can think of language as one aspect of the problem and language interfaces with computation, which is another aspect of the problem.[00:24:58] And if you can separate those two, this one just talks to these things and figures out what to, how to phrase it. Yeah, so it's not really coming up with the answer. Their claim is like GPT4, for example. The reason it's able to do factual accuracy without search is just by memorizing facts. And that doesn't scale.[00:25:18] It's literally somewhere in the whole model. It knows that the CEO of Tesla is Elon Musk. It just knows that. But it doesn't know that this is a competition. It just knows that. Usually I see CEO, Tesla, Elon, that's all it knows. So the abstraction of language model to computational unit or tool set is an interesting one that I think is gonna be more explored by all of these engines.[00:25:40] Um, and the latency, you know, it'll.[00:25:42] I think you're focusing on the right things there. I actually saw another article this morning about the memorization capability. You know how GPT4 is a lot of, uh, marketed on its ability to answer SAT questions and GRE questions and bar exams and, you know, we covered this in our benchmarks podcast Alessio, but like I forgot to mention that all these answers are out there and were probably memorized.[00:26:05] And if you change them just, just a little bit, the model performance will probably drop a lot.[00:26:10] It's true. I think the most compelling, uh, proof of that, of what you just said is the, the code forces one where somebody I think tweeted, tweeted, tweeted about the, yeah, the 2021. Everything before 2021. It solves everything after.[00:26:22] It doesn't, and I thought that was interesting.[00:26:24] Search Issues: Tool Use[00:26:24] It's just, it's just dumb. I'm interested in two former, and I'm interested in react type, uh, patterns. Zapier just launched a natural language integration with LangChain. Are you able to compare contrast, like what approaches you like when it comes to LMS using[00:26:36] tools?[00:26:37] I think it's not boiled down to a science enough for me to say anything that's uh, useful. Like I think everyone is at a point of time where they're just playing with it. There's no way to reason about what LLMs can and can't do. And most people are just throwing things at a wall and seeing what sticks.[00:26:57] And if anyone claims to be doing better, they're probably lying because no one knows how these things behaves. You can't predict what the output is gonna be. You just think, okay, let's see if this works. This is my prompt. And then you measure and you're like, oh, that worked. Versus the stint and things like react and tool, form are really cool.[00:27:16] But those are just examples of things that people have thrown at a wall that stuck. Well, I mean, it's provably, it works. It works pretty, pretty well. I will say that one of the. It's not really of the framing of what kind of ways can you use LLMs to make it do cool things, but people forget when they're looking at cutting edge stuff is a lot of these LLMs can be used to generate synthetic data to bootstrap smaller models, and it's a less sexy space of it all.[00:27:44] But I think that stuff is really, really cool. Where, for example, I want to tag entities in a sentence that's a very simple classical natural language problem of NER. And what I do is I just, before I had to gather training data, train model, tune model, all of this other stuff. Now what I can do is I can throw GPT4 at it to generate a ton of synthetic data, which looks actually really good.[00:28:11] And then I can either just train whatever model I wanted to train before on this data, or I can use something called like low rank adaptation, which is distilling this large model into a much smaller, cost effective, fast model that does that task really well. And in terms of productionable natural language systems, that is amazing that this is stuff you couldn't do before.[00:28:35] You would have teams working for years to solve NER and that's just what that team does. And there's a great red and viral thread about our, all the NLP teams at Big Tech, doomed and yeah, I mean, to an extent now you can do this stuff in weeks, which is[00:28:51] huge.[00:28:52] Other AI Search takes: Perplexity and Neeva[00:28:52] What about some of the other kind of like, uh, AI native search, things like perplexity, elicit, have you played with, with any of them?[00:29:00] Any thoughts on[00:29:01] it? Yeah. I have played with perplexity and, and niva. Everyone. I think both of those products sort of try to do, again, search results, synthesis. Personally, I think Perplexity might be doing something else now, but I don't see the, any of those. Companies or products are disrupting either open AI or ChatGPT or Google being whatever prominent search engines with what they do, because they're all built off basically the Bing API or their own version of an index and their search itself is not good enough and there's not a compelling use case enough, I think, to use those products.[00:29:40] I don't know how they would make money, a lot of Neeva's way of making money as subscriptions. Perplexity I don't think has ever turned on the revenue dial. I just have more existential concerns about those products actually functioning in the long run. So, um, I think I see them as they're, they're nice, they're nice to play with.[00:29:56] It's cool to see the cutting edge innovation, but I don't really understand if they will be long lasting widely used products.[00:30:05] Why Document QA will Struggle[00:30:05] Do you have any idea of what it might take to actually do like a new kind of like, type of company in this space? Like Google's big thing was like page rank, right? That was like one thing that kind of set them apart.[00:30:17] Like people tried doing search before, like. Do you have an intuition for what, like the LM native page rank thing is gonna be to make something like this exist? Or have we kinda, you know, hit the plateau when it comes to search innovation?[00:30:31] So I, I talk to so many of my friends who are obviously excited about this technology as well, and many of them who are starting LLM companies.[00:30:38] You know, how many companies in the YC batch of, you know, winter 23 are LM companies? Crazy half of them. Right? Right. It's, it's ridiculous. But what I always, I think everyone's struggling with this problem is what is your advantage? What is your moat? I don't see it for a lot of these companies, and, uh, it's unclear.[00:30:58] I, I don't have a strong intuition. My sense is that the people who focus on problem first usually get much further than the people who focus solution first. And there's way too many companies that are solutions first. Which makes sense. It's always been the, a big achilles heel of the Silicon Valley.[00:31:16] We're a bunch of nerds that live in a whole different dimension, which nobody else can relate to, but nobody else. The problem is nobody else can relate to them and we can't relate to their problems either. So we look at tech first, not problem first a lot. And I see a lot of companies just, just do that.[00:31:32] Where I'll tell you one, this is quite entertaining to me. A very common theme is, Hey, LMS are cool, that, that's awesome. We should build something. Well, what should we build? And it's like, okay, consumer, consumer is cool, we should build consumer. Then it's like, ah, nah man. Consumers, consumer's pretty hard.[00:31:49] Uh, it's gonna be a clubhouse gonna blow up. I don't wanna blow up, I just wanna build something that's like, you know, pretty easy to be consistent with. We should go enter. Cool. Let's go enterprise. So you go enterprise. It's like, okay, we brought LMS to the enterprise. Now what problem do we tackle? And it's like, okay, well we can do q and A on documents.[00:32:06] People know how to do that, right? We've seen a couple of demos on that. So they build it, they build q and a on documents, and then they struggle with selling, or they're like, or people just ask, Hey, but I don't ask questions to my documents. Like, you realize this is just not a flow that I do, like I, oh no.[00:32:22] I ask questions in general, but I don't ask them to my documents. And also like what documents can you ask questions to? And they'll be like, well, any of them is, they'll say, can I ask them to all of my documents? And they'll be like, well, sure, if you give them, give us all your documents, you can ask anything.[00:32:39] And then they'll say, okay, how will you take all my document? Oh, it seems like we have to build some sort of indexing mechanism and then from one thing to the other, you get to a point where it's like we're building enterprise search and we're building an LM on top of it, and that is our product. Or you go to like ML ops and I'm gonna help you host models, I'm gonna help you train models.[00:33:00] And I don't know, it's, it seems very solution first and not problem first. So the only thing I would recommend is if you think about the actual problems and talk to users and understand what this can be useful for. It doesn't have to be that sexy of how it's used, but if it works and solves the problem, you've done your job.[00:33:18] Investing in AI Startups[00:33:18] I love that whole evolution because I think quite a few companies ha are, independently finding this path and, going down this route to build a glorified, you know, search spot. We actually interviewed a very problem focused builder, Mickey Friedman, who's very, very focused on products placement, image generation.[00:33:34] , and, you know, she's not focused on anything else in terms of image generation, like just focused on product placement and branding. And I think that's probably the right approach, you know, and, and if you think about like Jasper, right? Like they, they're out of all the other GPT3 companies when, when GPT3 first came out, they built focusing on, you know, writers on Facebook, you know, didn't even market on Twitter.[00:33:56] So like most people haven't heard of them. Uh, I think it's a timeless startup lesson, but it's something to remind people when they're building with, uh, language models. I mean, as a, as an investor like you, you know, you are an investor, you're your scout with me. Doesn't that make it hard to invest in anything like, cuz.[00:34:10] Mostly it's just like the incumbents will get to the innovation faster than startups will find traction.[00:34:16] Really. Like, oh, this is gonna be a hot take too. But, okay. My, my in, in investing, uh, with people, especially early, is often for me governed by my intuition of how they approach the problem and their experience with the technology, and pretty much solely that I don.[00:34:37] Really pretend to be an expert in the industry or the space that's their problem. If I think they're smart and they understand the space better than me, then I mostly convinced as if they've thought through enough of the business stuff, if they've thought through the, the market and everything else. I'm convinced I typically stray away from, you know, just what I just said.[00:34:57] Founders who are like LMS are cool and we should build something with them. That's not like usually very convincing to me. That's not a thesis. But I don't concern myself too much with pretending to understand what this space means. I trust them to do that. If I'm convinced that they're smart and they've thought about it, well then I'm pretty convinced that that they're a good person to, to, to[00:35:20] back.[00:35:21] Cool.[00:35:21] Actually Interesting Ideas in AI[00:35:21] Kinda like super novel idea that you wanna shout.[00:35:25] There's a lot of interesting explorations, uh, going on. Um, I, I, okay, I'll, I'll preface this with I, anything in enterprise I just don't think is cool. It's like including, like, it's just, it's, you can't call it cool, man. You're building products for businesses.[00:35:37] Glean is pretty cool. I'm impressed by Glean. This is what I'm saying. It's, it's cool for the Silicon Valley. It's not cool. Like, you're not gonna go to a dinner party with your parents and be like, Hey mom, I work on enterprise search. Isn't that awesome? And they're not all my, all my[00:35:51] notifications in one place.[00:35:52] Whoa.[00:35:55] So I will, I'll, I'll start by saying, for in my head, cool means like, the world finds this amazing and, and it has to be somewhat consumer. And I do think that. The ideas that are being played with, like Quora is playing with Poe. It's kind of strange to think about, and may not stick as is, but I like that they're approaching it with a very different framing, which is, Hey, how about you talk to this, this chat bot, but let's move out of this, this world where everyone's like, it's not WhatsApp or Telegram, it's not a messaging app.[00:36:30] You are actually generating some piece of content that now everybody can make you use of. And is there something there Not clear yet, but it's an interesting idea. I can see that being something where, you know, people just learn. Or see cool things that GPT4 has said or chatbots have said that's interesting in the image space.[00:36:49] Very contrasted to the language space. There's so much like I don't even begin to understand the image space. Everything I see is just like blows my mind. I don't know how mid journey gets from six fingers to five fingers. I don't understand this. It's amazing. I love it. I don't understand what the value is in terms of revenue.[00:37:08] I don't know where the markets are in, in image, but I do think that's way, way cooler because that's a demo where, and I, and I tried this, I showed GPT4 to, to my mom and my mom's like, yeah, this is pretty cool. It does some pretty interesting stuff. And then I showed the image one and she is just like, this is unbelievable.[00:37:28] There's no way a computer could write do this, and she just could not digest it. And I love when you see those interactions. So I do think image world is a whole different beast. Um, and, and in terms of coolness, lot more cool stuff happening in image video multimodal I think is really, really cool. So I haven't seen too many startups that are doing something where I'm like, wow, that's, that's amazing.[00:37:51] Oh, 11 labs. I'll, I'll mention 11 labs is pretty cool. They're the only ones that I know that are doing Oh, the voice synthesis. Have you tried it? I've only played with it. I haven't really tried generating my own voice, but I've seen some examples and it looks really, really awesome. I've heard[00:38:06] that Descript is coming up with some stuff as well to compete, cuz yeah, this is definitely the next frontier in terms of, podcasting.[00:38:13] Harry Potter IRL[00:38:13] One last thing I I will say on the cool front is I think there is something to be said about. A product that brings together all these disparate advancements in ai. And I have a view on what that looks like. I don't know if everyone shares that view, but if you bring together image generation, voice recognition, language modeling, tts, and like all of the other image stuff they can do with like clip and Dream booth and putting someone's actual face in it.[00:38:41] What you can actually make, this is my view of it, is the Harry Potter picture come to life where you actually have just a digital stand where there's a person who's just capable of talking to you in their voice, in, you know, understandable dialogue. That is how they speak. And you could just sort of walk by, they'll look at you, you can say hi, they'll be, they'll say hi back.[00:39:03] They'll start talking to you. You start talking back to it. That's sort of my, that's my my wild science fiction dream. And I think the technology exists to put all of those pieces together and. The implications for people who are older or saving people over time are huge. This could be a really cool thing to productionize.[00:39:23] AI Infra Cost Math[00:39:23] There's one more part of you that also tweets about numbers and math, uh, AI math essentially is how I'm thinking about it. What gets you into talking about costs and math and, and you know, just like first principles of how to think about language models.[00:39:39] One of my biggest beefs with big companies is how they abstract the cost away from all the engineers.[00:39:46] So when you're working on a Google search, I can't tell you a single number that is cost related at all. Like I just don't know the cost numbers. It's so far down the chain that I have no clue how much it actually costs to run search, and how much these various things cost aside from what the public knows.[00:40:03] And I found that very annoying because when you are building a startup, particularly maybe an enterprise startup, you have to be extremely cognizant about the cost because that's your unit economics. Like your primary cost is the money you spend on infrastructure, not your actual labor costs. The whole thesis is the labor doesn't scale, but the inf.[00:40:21] Does scale. So you need to understand how your infra costs scale. So when it comes to language models, given that these things are so compute heavy, but none of the papers talk about cost either. And it's just bothers me. I'm like, why can't you just tell me how much it costs you to, to build this thing?[00:40:39] It's not that hard to say. And it's also not that hard to figure out. They give you everything else, which is, you know, how many TPUs it took and how long they trained it for and all of that other stuff, but they don't tell you the cost. So I've always been curious because ev all everybody ever says is it's expensive and a startup can't do it, and an individual can't do it.[00:41:01] So then the natural question is, okay, how expensive is it? And that's sort of the, the, the background behind. Why I started doing some more AI math and, and one of the tweets that probably the one that you're talking about is where I compare the cost of LlaMA, which is Facebook's LLM, to PaLM with, uh, my best estimates.[00:41:23] And, uh, the only thing I'll add to that is it is quite tricky to even talk about these things publicly because you get rammed in the comments because by people who are like, oh, don't you know that this assumption that you made is completely BS because you should have taken this cost per hour? Because obviously people do bulk deals.[00:41:42] And yeah, I have two 80 characters. This is what I could have said. But I think ballpark, I think I got close. I, I'd like to imagine, I think I was off maybe by, by by two x on the lower side. I think I took an upper bound and I might have been off by, by two x. So my quote was 4 million for LlaMA and 27 for PaLM.[00:42:01] In fact, later today I'm going to do, uh, one on Bard. So. Oh oh one bar. Oh, the exclusive is that It's four, it's 4 million for Bard two.[00:42:10] Nice. Nice. Which is like, do you think that's like, don't you think that's actually not a lot, like it's a drop in the bucket for these[00:42:17] guys. One, and one of the, the valuable things to note when you're talking about this cost is this is the cost of the final training step.[00:42:24] It's not the cost of the entire process. And a common rebuttal is, well, yeah, this is your cost of the final training process, but in total it's about 10 x this amount cost. Because you have to experiment. You have to tune hyper parameters, you have to understand different architectures, you have to experiment with different kinds of training data.[00:42:43] And sometimes you just screw it up and you don't know why. And you have, you're just spend a lot of time figuring out why you screwed it up. And that's where the actual cost buildup happens, not in the one final last step where you actually train the final model. So even assuming like a 10 x on top of this, I think is, is, is fair for how much it would actually cost a startup to build this from scratch?[00:43:03] I would say.[00:43:04] Open Source LLMs[00:43:04] How do you think about open source in this then? I think a lot of people's big 2023 predictions are an LLM, you know, open source LLM, that is comparable performance to the GPT3 model. Who foots the bill for the mistakes? You know, like when when somebody opens support request that it's not good.[00:43:25] It doesn't really cost people much outside of like a GitHub actions run as people try entering these things separately. Like do you think open source is actually bad because you're wasting so much compute by so many people trying to like do their own things and like, do you think it's better to have a centralized team that organizes these experiments or Yeah.[00:43:43] Any thoughts there? I have some thoughts. I. The most easy comparison to make is to image generation world where, you know, you had Mid Journey and Dolly come out first, and then you had Imad come out with stability, which was completely open source. But the difference there is I think stability. You can pretty much run on your machine and it's okay.[00:44:06] It works pretty fast. So it, so the entire concept of, of open sourcing, it worked and people made forks that fine tuned it on a bunch of different random things and it made variance of stability that could. A bunch of things. So I thought the stability thing, agnostic of the general ethical concerns of training on everyone's art.[00:44:25] I thought it was a cool, cool addition to the sort of trade-offs in different models that you can have in image generation for text generation. We're seeing an equivalent effect with LlaMA and alpaca, which LlaMA being, being Facebook's model, which they didn't really open source, but then the weights got leaked and then people clone them and then they tuned them using GPT4 generated synthetic data and made alpaca.[00:44:50] So the version I think that's out there is only the 7,000,000,001 and then this crazy European c plus plus God. Came and said, you know what, I'm gonna write this entire thing in c plus plus so you can actually run it locally and and not have to buy GPUs. And a combination of those. And of course a lot of people have done work in optimizing these things to make it actually function quickly.[00:45:13] And we can get into details there, but a function of all of these things has enabled people to actually. Semi-good models on their computer. I don't have that much, I don't have any comments on, you know, energy usage and all of that. I don't really have an opinion on that. I think the fact that you can run a local version of this is just really, really cool, but also supremely dangerous because with images, conceivably, people can tell what's fake and what's real, even though there, there's some concerns there as well. But for text it's, you know, like you can do a lot of really bad things with your own, you know, text generation algorithm. You know, if I wanted to make somebody's life hell, I could spam them in the most insidious ways with all sorts of different kinds of text generation indefinitely, which I, I can't really do with images.[00:46:02] I don't know. I find it somewhat ethically problematic in terms of the power is too much for an individual to wield. But there are some libertarians who are like, yeah, why should only open AI have this power? I want this power too. So there's merits to both sides of the argument. I think it's generally good for the ecosystem.[00:46:20] Generally, it will get faster and the latency will get better and the models may not ever reach the size of the cutting edge that's possible, but it could be good enough to do. 80% of the things that bigger model could do. And I think that's a really good start for innovation. I mean, you could just have people come up with stuff instead of companies, and that always unlocks a whole vector of innovation that didn't previously exist.[00:46:45] Other Modalities[00:46:45] That was a really good, conclusion. I, I, I want to ask follow up questions, but also, that was a really good place to end it. Was there any other AI topics that you wanted to[00:46:52] touch on? I think Runway ML is the one company I didn't mention and that, that one's, uh, one to look out for.[00:46:58] I think doing really cool stuff in terms of video editing with generative techniques. So people often talk about the open AI and the Googles of the world and philanthropic and clo and cohere and big journey, all the image stuff. But I think the places that people aren't paying enough attention to that will get a lot more love in the next couple of years.[00:47:19] Better whisper, so better streaming voice recognition, better t t s. So some open source version of 11 labs that people can start using. And then the frontier is sort of multi-modality and videos. Can you do anything with videos? Can you edit videos? Can you stitch things together into videos from images, all sorts of different cool stuff.[00:47:40] And then there's sort of the long tail of companies like Luma that are working on like 3D modeling with generative use cases and taking an image and creating a 3D model from nothing. And uh, that's pretty cool too, although the practical use cases to me are a little less clear. Uh, so that's kind of covers the entire space in my head at least.[00:48:00] I[00:48:00] like using the Harry Potter image, like the moving and speaking images as a end goal. I think that's something that consumers can really get behind as well. That's super cool.[00:48:09] Exam Fraud and Generated Text Detection[00:48:09] To double back a little bit before we go into the lining round, I have one more thing, which is, relevant to your personal story, but then also relevant to our debate, which is a nice blend.[00:48:18] You're concerned about the safety of everyone having access to language models and you know, the potential harm that you can do there. My guess is that you're also not that positive on watermarking. Techniques from internal languages, right? Like maybe randomly sprinkling weird characters so that people can see like that this is generated by an AI model, but also like you have some personal experience with this because you found manipulation in the Indian Exam Board, which, uh, maybe you might be a similar story.[00:48:48] I, I don't know if you like, have any thoughts about just watermarking manipulation, like, you know, ethical deployments of, of, uh,[00:48:55] generated data.[00:48:57] Well, I think those two things are a little separate. Okay. One I would say is for watermarking text data. There is a couple of different approaches. I think there is actual value to that because from a pure technical perspective, you don't want models to train on stuff they've generated.[00:49:13] That's kind of bad for models. Yes. And two is obviously you don't want people to keep using Chatt p t for i, I don't know if you want this to use it for all their assignments and never be caught. Maybe you don't. Maybe you don't. But it, it seems like it's valuable to at least understand that this is a machine generated text versus not just ethically that seems, seems like something that should exist.[00:49:33] So I do think watermarking is, is. A good direction of research and it's, and I'm fairly positive on it. I actually do think people should standardize how that water marketing works across language models so that everyone can detect and understand language models and not just, OpenAI does its own models, but not the other ones and, and so on.[00:49:51] So that's my view on that. And then, and sort of transitioning into the exam data, this is really old one, but it's one of my favorite things to talk about is I. In America, as you know. Usually the way it works is you give your, you, you take your s a t exam, uh, you take a couple of aps, you do your school grades, you apply to colleges, you do a bunch of fluff.[00:50:10] You try to prove how you're good at everything. And then you, you apply to colleges and then it's a, a weird decision based on a hundred other factors. And then they decide whether you get in or not. But if you're rich, you're basically gonna get in anyway. And if you're a legacy, you're probably gonna get in and there's a whole bunch of stuff going on.[00:50:23] And I don't think the system is necessarily bad, but it's just really complicated. And some of the things are weird in India and in a lot of the non developed world, people are like, yeah, okay, we can't scale that. There's no way we can have enough people like. Non rigorously evaluate this cuz there's gonna be too much corruption and it's gonna be terrible at the end cuz people are just gonna pay their way in.[00:50:45] So usually it works in a very simple way where you take an exam that is standardized and sometimes you have many exams, sometimes you have an exam for a different subject. Sometimes it's just one for everything. And you get ranked on that exam and depending on your rank you get to choose the quality and the kind of thing you want to study.[00:51:03] Which this, the kind of thing always surprises people in America where it's not like, oh it's glory land, where you walk in and you're like, I think this is interesting and I wanna study this. Like, no, in the most of the world it's like you're not smart enough to study this, so you're probably not gonna study it.[00:51:18] And there's like a rank order of things that you need to be smart enough to do. So it's, it's different. And therefore these exams. Much more critical for the functioning of the system. So when there's fraud, it's not like a small part of your application going wrong, it's your entire application going wrong.[00:51:36] And that's why, that's just me explaining why this is severe. Now, one such exam is the one that you take in school. There's a, it's called a board exam. You take one in the 10th grade, which doesn't really matter for much, but, and then you take one in the 12th grade when you're about to graduate and that.[00:51:53] How you, where you go to college for a large set of colleges, not all, but a large set of colleges, and based on how much you get on your top five average, you're sort of slotted into a different stream in a d in a, in a different college. And over time, because of the competition between two of the boards that are a duopoly, there's no standardization.[00:52:13] So everyone's trying to like, give more marks than the, the, the other person to attract more students into their board because oh, that means that you can then claim, oh, you're gonna get into a better college if you take our exam and don't go to a school that administers the other exam. What? So it's, and that's, that's the, everyone knew that was happening ish, but there was no data to back it.[00:52:34] But when you actually take this exam as I did, you start realizing that the numbers, the marks make no sense because you're looking at. Kid who's also in your class and you're like, dude, this guy's not smart. How did he get a 90 in English? He's not good at English. Like, you can't speak it. You cannot give him a 90.[00:52:54] You gave me a 90. How did this guy get a 90? So everyone has like their anecdotal, this doesn't make any sense me, uh, moments with, with this exam, but no one has access to the data. So way back when, what I did was I realized they have very little security surrounding the data where the only thing that you need to put in to get access is your role number.[00:53:15] And so as long as you predict the right set of role numbers, you can get everybody's results. So unlike America, also exam results aren't treated with a level of privacy. In India, it's very common to sort of the entire class's results on a bulletin board. And you just see how everyone did and you shamed the people who are stupid.[00:53:32] That's just how it works. It's changed over time, but that's fundamentally a cultural difference. And so when I scraped all these results and I published it, and I, and I did some analysis, what I found was, A couple of very insidious things. One is that in, if you plot the distribution of marks, you generally tend to see some sort of skewed, but pseudo normal distribution where it's a big peak and a, and it falls off on both ends, but you see two interesting patterns.[00:54:01] One that is just the most obvious one, which is Grace Marks, which is the pass grade is 33. You don't see nobody got between 29 and 32 because what they did for every single exam is they just made you pass. They just rounded up to 33, which is okay. I'm not that concerned about whether you give Grace Marks.[00:54:21] It's kind of messed up that you do that, but okay, fine. You want to pass a bunch of people who deserve to fail, do it. Then the other more concerning thing was between 33 and 93, right? That's about 60 numbers, 61 numbers, 30 of those numbers were just missing, as in nobody got 91 on this exam. In any subject in any year.[00:54:44] How, how does that happen? You, you don't get a 91, you don't get a 93, 89, 87, 85, 84. Some numbers were just missing. And at first when I saw this, I'm like, this is definitely some bug in my code. There's no way that, like, there's 91 never happened. And so I started, I remember I asked a bunch of my friends, I'm like, dude, did you ever get a 9 81 in anything?[00:55:06] And they're like, no. And it just unraveled that this is obviously problematic cuz that means that they're screwing with your final marks in some way or the other. Yeah. And, and they're not transparent about how they do it. Then I did, I did the same thing for the other board. We found something similar there, but not, not, not the same.[00:55:24] The problem there was, there was a huge spike at 95 and then I realized what they were doing is they'd offer various exams and to standardize, they would blanket add like a, a, a, a raw number. So if you took the harder math exam, everyone would get plus 10. Arbitrarily, no one. This is not revealed or publicized.[00:55:41] It's randomly, that was the harder exam you guys all get plus 10, but it's capped at 95. That's just this stupid way to standardize. It doesn't make any sense. Ah, um, they're not transparent about it. And it affects your entire life because yeah, this is what gets you into college. And yeah, if you add the two exams up, this is 1.1 million kids taking it every year.[00:56:02] So that's a lot of people's lives that you're screwing with by not understanding numbers and, and not being transparent about how you're manipulating them. So that was the thesis in my view, looking back on it, 10 years later, it's been 10 years at this point. I think the media never did justice to it because to be honest, nobody understands statistics.[00:56:23] So over time it became a big issue then. And then there was a big Supreme court or high court ruling, which said, Hey, you guys can't do this, but there's no transparency. So there's no way of actually ensuring that they're not doing it. They just added a, a level of password protection, so now I can't scrape it anymore.[00:56:40] And, uh, they probably do the same thing and it's probably still as bad, but people aren't. Raising an issue about it. It's really hard to make this people understand the significance of it because people are so compelled to just go lean into the narrative of exams are b******t and we should never trust ex

Get up to speed with the Digital Marketing News and Updates from the week of Feb 27-Mar 3, 2023.1. PSA: US TikTok Ban Moves a Step Closer - More bad news for TikTok, with the US House Foreign Affairs Committee voted to give President Joe Biden the power to ban the Chinese-owned app, if he deems such a move necessary, amid ongoing security discussions around its potential connection to the Chinese Communist Part (CCP).TikTok responded to the vote by tweeting that “A U.S ban on TikTok is a ban on the export of American culture and values to the billion-plus people who use our  service worldwide…”While Today's announcement doesn't give Biden the full green light to ban the app, with the US Senate still required to give sign-off before a ban could be implemented. But it's another step towards that next stage, which increasingly feels like it will lead to a TikTok ban, or at the least, a significant change in direction for the app.Remember that TikTok, along with 58 other Chinese-created apps, was banned completely in India by the Ministry of Electronics and Information Technology on 29 June 2020. So if you are relying on traffic from TikTok then it is high time you diversify your traffic sources.2. Google Shares How Its Keyword-Matching System For Search Ads Work - Google has released a 28 page comprehensive guide during Google Search Ads Week 2023, providing a unique behind-the-scenes glimpse into its keyword-matching system for search ads.To achieve better results, advertisers can optimise their campaigns by gaining an understanding of Google Ads keyword-matching process.Google's guide provides a comprehensive breakdown of the system, which includes how the company utilises machine learning and natural language understanding technologies to determine keyword eligibility, and how the responsive search ads creative system selects the best-performing creative for users.It is essential to note that grouping keywords is critical to campaign optimisation. By eliminating the need to add the same keyword in multiple match types, advertisers can avoid segmenting and reducing the available data that Smart Bidding can use for optimisation, which can result in fewer conversions and higher costs.The guide is an invaluable resource for anyone seeking to enhance their Google Ads campaigns. Incorporating the insights and best practices outlined in the guide can boost the chances of success and drive more conversions.  This is why I always tell my listeners to work with a reputable learning and growing agency who is in the know. Afterall, you can not make moves or leverage opportunities if you are not in the know.3. Google Ads Is Changing Location Targeting Settings In March 2023 - Starting March 2023, “Search Interest” targeting will no longer be available in Google Ads. Campaigns that use “Search Interest” targeting will be migrated to “Presence or Interest” targeting. These changes will be consistent in Search, Display, Performance Max, and Shopping campaigns. The Presence option lets you show your ads to people who are likely to be located, or regularly located in the locations you've targeted.The Search Interest option lets you show your ads to anyone searching on Google for your targeted location. If a person doesn't specify a location in their search, then the system uses the location where a user is likely to be located for targeting. This option is only available for Search campaigns.So after this change is in effect, a person who lives in Northern VA but often travels to Maryland for shopping or work. While home in VA, the person searches for "plumber near me." Now Google is going to show some Maryland plumbers who are not licensed in VA.  Am I the only one who thinks that the real winner of this change is Google!!4. Google Ads Introduces AI-Powered Search Ads - During the Google's Search Ads Week, a new customer acquisition goal for Search campaigns has been launched globally. This goal utilizes Smart Bidding and first-party data to optimize campaigns and attract new customers during peak periods. According to Google, by combining the new customer acquisition goal with bidding strategies like Maximize conversion value with a target ROAS, advertisers can prioritize and target high-value customers. The new customer acquisition goal has two modes that help you to reach your campaign goals: Value New Customer: Bid higher for new customers than for existing customers New Customers Only: Bid for new customers only. 5. Microsoft Bing's Fabrice Canel : SEO Will Never Be "dead" - Fabrice Canel, the Principal Product Manager for Microsoft Bing, gave a keynote presentation at the Pubcon convention in Austin, Texas. His presentation offered valuable information on optimizing websites for the new Bing search experience as well as shared the benefits of using Bing Webmaster Tools to monitor traffic data and make necessary adjustments to improve visibility in search results.First, Canel suggested to stay with the same SEO playbooks for optimizing content for Bing's AI experience because it's still the early days for AI search. Throughout his keynote at Pubcon, Canel stressed the importance of SEO professionals in guiding Bing's search crawlers to high-quality content.Then Canel emphasized the importance of setting the lastmod tag to the date a page was last modified, not when the sitemap was generated. Remember lastmod was covered in previous episodes in details. ICYMI, the lastmod tag is an HTML attribute indicating when a particular webpage or URL received significant changes. This tag is used in sitemaps to help search engines like Bing understand when a page was last updated. Lastmod also helps searchers identify and access the most up-to-date content available. When a lastmod tag is present, Bing will display the updated date in search results. This signals to searchers that the webpage may have new or updated information they haven't seen yet. According to Canel, 18% of sitemaps have lastmod values not correctly set, typically set to the date and time the sitemap is generated.Thirdly, Canel recommended website to  adopting IndexNow to inform search engines of recent modifications to website content instantly. FYI: IndexNow was covered in episode# 90 (Jan 10-15, 2022). According to Canel, 20 million websites have already adopted IndexNow, and he expects more top websites, search engines, and content management systems to follow suit. Canel adds that manually crawling a webpage to see if its content has changed wastes resources and energy and creates CO2. He also suggests having sitemaps to provide search engines with all relevant URLs and corresponding modification dates.Most importantly, he wanted website owner focus on writing quality content and use semantic markup to convey information about the pages.Lastly, we learned Bing Webmaster Tools will soon include traffic data from Bing's AI chat.6. Google On ‘lastmod' Tag In XML Sitemap - I covered “lastmod” in episode#146. It is back again. Google's John Mueller said on Twitter if you are "providing something new for search engines that you'd like reflected in search," then update the date, if not, then don't. John added, "The issue is more that some CMS's / servers set the lastmod to the current date/time for all pages. This makes that data useless. Good CMS's setting it thoughtfully, even if not always perfect, is much more useful."The current Google documentation says, "Google uses the lastmod value if it's consistently and verifiably (for example by comparing to the last modification of the page) accurate." And according to a recent study at Bing (also covered in episode#146) revealed that among websites with at least one URL indexed by Bing: 58% of hosts have at least one XML sitemap (sitemap known by Bing).84% of these sitemaps have a lastmod attribute set 79% have lastmod values correct.  18% have lastmod values not correctly set.  3% has lastmod values for only some of the URLs. 42% of hosts don't have one XML sitemap (Bing does not know it) P.S: Don't be the business that is skipping the basics and easy to do stuff and looking to do advanced stuff. #DoTheBasics first.7. Google: Don't Combine Site Moves With Other Big Changes - Sometimes businesses make changes to their top-level domain as well as update their website. So Google Search Advocate John Mueller during a recent Search Of The Record Podcast with Gary Illyes, and Senior Technical Writer Lizzi Sassman asked “What happens if I do a domain change, and move from a “.ch”, which is a Swiss top level domain, to “.com”? Is that a problem? Like if I combine a domain change with other stuff?”In response, Illyes, shared that these changes should be done in smaller pieces over months. Making too many changes at once could result in lower rankings and lost traffic. For example, if a website is moving from “example.ch” and “example.fr” to “example.com,” Illyes recommended moving “example.fr” first and waiting before moving “example.ch.”Mueller and Sassman questioned Illyes on why he's so concerned about spreading out site moves. Illyes admitted that many site moves he's been involved with have resulted in lost traffic. Illyes also mentioned that misconfigurations, such as incorrect redirects, are common mistakes that can cause traffic loss. However, traffic shouldn't be lost during a domain change if everything is done correctly.If all you're doing is redirecting URLs from one site to another, there's a low risk for adverse effects. On the other hand, if you do lose rankings and traffic, there's no specific timeframe for a full recovery.8. Google's Gary Illyes: Google Does Not Care Who Authors or Links To The Content - Gary Illyes from Google gave a keynote and a Q&A session at PubCon and while the keynote was pretty vanilla stuff, the Q&A did reconfirm a lot of what has been said in the past around authorship, links and disavowing links. In short, Google does not give too much weight to who writes your content. So if you get a Walt Mossberg to write a piece of content on your site, just because it is Walt, doesn't make it rank well. If the content is written well, it will rank well, but by default, just because Walt wrote it, doesn't make it rank well. Gary also said that links are not as important as SEOs think they are.  And disavowing links is just a waste of time.P.S: All these topics have been covered in the past shows. 9. Google: PageRank Sculpting Is A Myth - Every website is assigned a unique value by the Google PageRank algorithm. This value, also called PageRank, has long been an important factor in link building and link exchange. PageRank sculpting is a technique in which an attempt is made to distribute the PageRank of a website to other subpages. Assuming that the home page receives the highest PageRank because it is the most important within the sites hierarchy, the PageRank will decrease as you go further down into the structure. Before 2009, it was common practice to control the PageRank through sculpting so that only certain pages would benefit. For example, function pages such as the imprint or contact page were linked internally with the attribute “nofollow.” Thus, the link power increased (as measured by PageRank) for the remaining internal links. Unfortunately, some SEO Experts still feel that they can control how Google passes your link equity throughout your site by using the nofollow link attribute. So Google's John Muller said on Twitter that it is an SEO myth to say you can use the nofollow attribute on links sculpt PageRank. Remember, back in 2019 he tweeted that Internal PageRank Sculpting Is A Waste Of Time. Another #SEOMythBusted. I'll file this under #AvoidBadSEOAdvice.10. Check Domain Reputation Before You Buy A Domain - Google's John Mueller was asked about a domain name purchased several months ago but still does not rank well in Google Search. John explained that if a domain has a "long and complicated history." "It's going to be hard to convince search engines that it's something very different & unrelated to what was done in the past decades," John added.In short, he is saying that not only was this domain abusing search engines for a long, long time, but also that the new content on this old domain is not different enough or unrelated enough from what the topic was previously where the search engine would consider it a brand new site and wipe the site clean.Basically the issue here is “domain legacy penalty” - It's a penalty that's associated with a domain from when it was registered by someone else in the past. Apparently the penalty remains after the domain is registered by someone else years later. Which makes sense or else bad actors will keep on transferring domain ownership to bypass the penalty. The way to prevent is to check the past history of a domain name is to visit Archive.org. Archive.org downloads and creates an archive of websites throughout the Internet.A similar issue happened a few years ago to ZDNet. One of their domains was hyphenated (CXO-Talk.com). So they purchased the non-hyphenated variant (CXOTalk.com) from a third party domain auction. ZDNet was unaware that the domain had been used by spammers.  Soon after ZDNet migrated all their content from CXO-Talk.com to CXOTalk.com, their website was banned from Google. ZDNet wrote an article about what happened to them and had the following advice: Before purchasing any domain at auction, be sure to check its history using backlink tools If the domain has a bad history, use Google Webmaster Tools to do a clean-up before putting the domain into service Google's system of problem remediation lacks transparency and responsiveness. They can and should do better. I still don't really know what caused the problem or how to fix it. 11. Should You Rewrite Your Content With ChatGPT? - Google's John Mueller went back and forth on Twitter with some SEO practitioners on the topic of using ChatGPT to (re)write existing content. Basically Ujesh was wondering if he can rewrite his own content with the help of tools like #ChatGPT without losing its helpfulness and relevancy. He was curious to see if it will  reduce the quality of the article due to AI involvement or does it boost the article considering the quality revamp ?To that question, John asked “Why do you need to rewrite your own content? Is it bad?” IMO, this is a fair question.To John's question Paulo replied, “let's say that English is not my main language. Then, I write something in my mother tongue, translate it in my own limited vocabulary, and ask AI to enhance the vocabulary. The content is not bad, but limited by my knowledge of a language, not the topic I'm trying to cover.”And John responded by saying “Why do you want to just publish something for the sake of publishing something, rather than publishing something you know to be useful & good? (This is not unique to LLM/AI NLG, it's the same with unknown-quality human-written content.) What do you want your site known for?”John is saying that, if your content is bad, why are you writing it in the first place? If you know your content is bad, then it is not helpful, will ChatGPT make it helpful for you? How do you know if the ChatGPT version is helpful and quality if your content you originally wrote is not quality? Maybe instead of using ChatGPT to improve the quality of your content, maybe you should focus on topics that you can write quality content about?

The playbook for artists to go viral on TikTok has changed a lot since 2019. Sean Taylor aka “BrandMan Sean” has written and executed that playbook for his clients since the early days of TikTok. He's the co-founder of the ContraBrand Agency, which specializes in TikTok marketing for music talent. The agency has helped artists like Macy Gray, 24kGoldn, and Trap Beckham, among others.Sean and his team just released a global report on How Artists are Going Viral on TikTok. The report is packed with insights on artist virality on the platform. According to the report, artist-generated content (AGC) is the key to going viral today. It's more impactful than not user-generated content (UGC) from fans and other users. AGC not only works, but it's also a cost-effective way for independent artists to break through.However, Sean points out that virality isn't as easy as before. TikTok has matured, and overnight success is harder to achieve. Still, with the right strategy, Sean believes TikTok is still a second-to-none top-of-funnel marketing play. We broke down this tested TikTok system in our discussion. Here's everything we covered about the platform:[1:51] TikTok entering its maturation stage[5:39] Second wave TikTok music artists vs. first wave[9:10] Biggest shift on TikTok for artists[17:13] No, artists don't have to post dance content[24:00] YouTube shorts lack of culture[26:29] YouTube's advantage over TikTok[31:31] The problem with IG Reels[33:32] TikTok pushing Google for search dominance[38:55] TikTok as a marketing funnel[42:21] The rise of TikTok live[46:10] Predicting where TikTok will be in three yearsHow Artists are Going Viral on TikTok in 2022 report:https://www.contrabrand.agency/tiktokglobalreport2022Listen: Apple Podcasts | Spotify | SoundCloud | Stitcher | Overcast | Amazon | Google Podcasts | Pocket Casts | RSSHost: Dan Runcie, @RuncieDan, trapital.coGuests: Sean Taylor, @brandmanseanEnjoy this podcast? Rate and review the podcast here! ratethispodcast.com/trapitalTrapital is home for the business of music, media and culture. Learn more by reading Trapital's free memo.TRANSCRIPTIONTrapital #Sean Taylor[00:00:00] Sean Taylor: One of the problems that people were having were them blowing up right? Without being able to connect to an actual face, right? So it solves so many of the problems that come with that, and even helps the problem of TikTok's algorithm where people just hop on and start running things up with ads and you haven't really even understood what your content looks like, that creates some algorithmic problems, which probably aren't worth getting into, here, or maybe they are, but yeah. Man, artists generate content. It's gonna be a love hate relationship for sure with artists, the labels, all of us, right? But, if anything, it'll force collaboration and synergy between teams, in ways that it hasn't before.[00:00:42] Dan Runcie Intro: Hey, welcome to the Trapital Podcast. I'm your host and the founder of Trapital, Dan Runcie. This podcast is your place to gain insights from the executives in music, media, entertainment, and more. Who are taking hip hop culture to the next level.[00:00:42] Dan Runcie: All right, today we are joined by my guy, Brandman Sean, Sean Taylor, who is back on the podcast for a second time now, and I wanted to have him on because there's so much that's happening with TikTok, with short form video and how artists are using it. And his company, the contraband agency just put out a report that dives deep into this, and he talks about this often on his platform, the Brandman Network. So Sean, let's level for a little bit, and I feel like TikTok is in such an interesting place right now, 2023. It's not some of that same rapid growth that it may have had a couple years ago, but it's still so essential for artists. How do you feel about where the platform is right now?[00:01:51] Sean Taylor: I think it's in a really good space actually. It's in a maturation space. The problem with that is people aren't seeing hits come as easy on the platform. and they're actually using that to downplay the platform and say, TikTok isn't that impactful, or it's not that big of a deal. It's hard to get a hit on TikTok. The difference is it's now a normal marketing infrastructure within your whole overall marketing stack. So yeah, there was this hot period where you were getting like gains that you probably didn't even deserve. Right. Every shock, swish, nothing but net. Now you have to do what you're supposed to do in every other space. So I think a lot of the pain that people are feeling isn't necessarily TikTok not being effective. It's TikTok not being unreasonably effective, unbelievably effective. The thing that made me get on TikTok, back in 2019. It's in an interesting space, but I think it's in a good space actually. And I can go deeper into that specific argument and why I see it that way. Cuz there's some numbers and milestones that I kind of think of it and approach it from, but yeah, that's where I think TikTok is right now. It's new, it's a viable marketing channel, but it's not the marketing channel that everybody is going to be as excited about as they were.[00:03:24] Dan Runcie: I'm glad you said this because there's been a bunch of reports about how TikTok has slowed down about how artists are starting to complain, and I've heard many A-list artists, even privately and publicly complain that things are popping the way they used to. But this isn't 2019 anymore. It may take some actual marketing expertise since some clever thinking about how to find things in. I remember one of the reports I said was talking about how you can't just give some post or some link to Addison Rae and then hope that someone like that goes and blows the whole thing up for you and makes you a superstar. You have to find your niches and build from there. And in reading that, it's like, well that sounds like what it's like to grow any type of career, and that's probably how it should be, right?[00:04:11] Sean Taylor: Exactly. Should it be that you pay one person and everything just blows up. Not really. I would love it to be that way for me, you know? But look, that's just the reality of how marketing works. So you can still get that number to grow and get millions of streams, but that millions might come a little bit slower. And now when it hits that 2 million mark, 3 million mark, probably even before that, it's gonna take a lot more heavy lifting to get it over the hump where, That thing could just keep going like a rocket ship straight to 2030 and not stop, right? So it's a great space to get things off the ground and create the spark, but going beyond that spark is more difficult.[00:04:59] Dan Runcie: In past years, we saw record labels signing a bunch of artists that came from TikTok, and I would assume that because of this rocket ship success, people didn't have the infrastructure behind them. A lot of those stories probably didn't end up panning out the way that they thought they would, maybe even at a lower rate than the average hit rate for. Otherwise artists at a record label are assigned. But I would think now that things have matured a bit, the artists that are actually coming to the forefront are likely gonna have more behind them. And because of that, B, the potential to actually maybe have a more sustainable career than that first wave of artists who just benefited from a very aggressive area.[00:05:39] Sean Taylor: Yeah. I mean, I think the thing is people hadn't really seen anything like that before, right? Like yeah, there had been one hit wonder. That has happened and someone who's seasoning the game probably understands what needs to take place. But to constantly have day after day someone popping out of nowhere like a breakneck speed level and trying to figure out how to bring infrastructure up, up under all these artists at the same time is a completely different story. Cuz it's also a different story when you have these artists housed under you, and then things take off really fast. You're taking them, you're trying to create a deal and figure out how to sign them, and then create infrastructure. By the time some of these deals take place, a lot of that moment is already missed, right? So, it was a really weird space, and I'm sure there's labels that have more of an infrastructure that's prepared for that situation. It's like, oh, if we bring somebody in from that particular climate, then there's a specific path that we can take 'em. Whether we expedite some things or we start here versus there, I'm sure that's there. But TikTok was really weird watching in the beginning because you had all these people blowing up and many didn't even wanna blow up, right? Like you had kids just using the platform and blowing up, they were an artist or just a regular influence or whatever you call 'em. They were just doing what kids normally do on apps and became stars overnight, which is very different from the artist who wants to be an artist. And then they take off. These are kids who are in their experimentational experimentation phase, kind of just having fun playing with things. And then it might be a hit song, right in a bed without even them trying to pursue it. So it created this really interesting space on TikTok and unfortunately, where I saw early on there were so many artists I don't wanna say artists, actually, less artists, more general content, creators falling prey to opportunist managers and companies because artists fortunately, have had a lot of education in these pages. I'm not saying artists don't ever have bad deals and situations, but there's a very common knowledge almost at this point that's been put out for artists getting in bad deals, avoiding bad deals, what you should do, in the culture, that education is out there as a regular content creator. That information isn't out there. Right. But it's very similar. So I actually saw like a lot of kids being signed by managers who had nothing to do with the industry at all. They're just like, "Hey, I'm just about to sign 51 situation I'm literally thinking about and he's telling me, yeah man, I just signed 50 content creators right to a management deal." And then thinking of it only from the standpoint of if I leverage these 50, then I'm gonna be able to get me a bigger deal, hopefully. But he doesn't have any relationships in place. There's no individual incentive to make any of the individual influencers blow up. It's more just, Hey, let me get stable so I can leverage the stable and, most of those deals fell apart, down the road. Or hopefully the parents kind of figured it out. But I know some who got burned really bad, but things were moving so fast. Like it was crazy. So a lot of parents were. Okay, this guy knows two or three people in the industry and you know, but everybody in the industry knows two or three people. So, but for people who don't have a child in the entertainment industry, and they never had any plans and they have no idea what to do, that sounds good. So TikTok was very crazy at the beginning. It was the wild, wild west. Now we're in this period where I think everybody has figured it out. Not everybody, but many people have figured out how to create more infrastructure. The problem is now the game is harder and that's how life works, right? It's like, dang, the moment I figured this shit out. Right? Things change a little bit. but you referenced my report earlier. I think the thing that was the biggest shift was the artist has to do more work. And that's what people feel more than anything. We could do everything and the artists were doing nothing and we were blowing songs. and now it's like, dang, I gotta get my artist to participate. And we all know how hard it can be to get the artist to participate in some things, especially content, right? but you know, that's created a space for those artists who truly do have a knack for content and that drive and honestly stamina to play that content. They've been able to make a lot happen, get a lot of organic streams, which makes it so much easier on the team cuz you still gotta do your job and make it, blow from there. But I know several artists that we work with who are getting their songs to 500,000 streams, 1 million streams, 10 million streams. Right. Any other form of marketing, just their content. So that's a huge benefit, and that's what I think the silver lining needs to be. The fact that we have that is still something we did not have in 2018 for music specifically, so that we need to appreciate and have gratitude for our blessings.[00:11:00] Dan Runcie: Let's dive into this a little bit because I think this point about artist generated content versus user-generated content is key. And I know it is a big part of your report as well, because I think for years now, we've heard so many people, even TikTok Head of Music just said this at the Nylon conference a couple days ago, was talking about how it's so key to be able to get the fans, to make the videos and get involved and things like that. And while that's still important, you're saying what actually can move the needle even more is getting the artist, even if they're reluctant to do it, getting the artist to do it themselves and having the two of them together and even more so the artist piece of it can really help push things forward.[00:11:42] Sean Taylor: Right. 100%. See, we realized this in 2020, in the trenches, you see this guy post a video, right? And we construct this concept. and you get a hundred thousand streams and just off of your video. Right? And that was amazing at that time to really see that he got a hundred thousand streams. And oh, by the way, there weren't really any replications to his video or sound. It had nothing to do with the dance. It hadn't had anything to do with influencers at all. He had the right creative concept, right? Hundred thousand streams. And for the artist that he was, you know, you're talking about pretty much no listeners, that's a massive number, especially just from one post and even better a post from him. Right. With not much of a following at all. He probably only had like 20,000 followers on Instagram at the time. Right? So we saw that and then I devised this campaign with the artist. Ironically, I just got off a call with this artist. We did like a little Google chat named Fash and Kid in Australia, right? He has some followers, probably a hundred thousand, 200,000 at this time. literally never dropped the song a day in his life at this moment, right? And he's like, “Yo, Sean, I've watched some of your videos and stuff on YouTube, and like, I wanna figure out how to release this song. I'm releasing it next week. What should I do? First of all, "Hey, don't release it next week", you know what I mean? Like, let's talk. Right? So, we made it a month from there, we created this entire narrative driven campaign. And just from him posting it was all based on his post, right? I actually took the marketing method that I blew up my music festival with, before I was doing, like working with artists, and it was all organic posts, right? So I had a structure that I used, and that was literally just him posting on his page. He got 1 million streams on his very first song, right? So, It wasn't one single post that made everything take off, but it was a system of post, and those were all pre, well, primarily pre-release. And then there were some things that were done, but this is 2020, so we're like, man, just posting all right on your page can take you far. The problem, I won't even say the problem was, but the thing is, paying influencers were still working like crazy at that time, right? So we didn't have an incentive to like to lean in it as heavy for artists that, you know, we would, were a little bit harder to get onto the platform and make work. Now, it's one of those things where, okay, look, we really want you to start here because the way things are set up today. If you don't do this and create this foundation, a lot of that other stuff won't bring anywhere near as big of a gain as it did. But yeah, back then we saw success with, or artist generated content, influencers. There were things that we called TikTok creators. We saw all these different types of games, but literally paying influencers was working so great at that time it was like, ah, why do anything else But yeah, artist generated content. Man, it's the way, man, it is the foundation of how I believe. things should be ran today. But of course, the caveat with every artist still has a different path, right? So your artist generated content might look different or artist generated content. There are outliers where that just won't be prevalent for you, but as a general way in a business approach. I love the fact that, one, you're creating fans in visibility for no money, right? You know, however much it costs to create your content, but generally speaking, no money. Two, you're testing. Songs before you actually put money behind them. Right? Three, if something blows, you already have a presence on the platform to connect people to. Because one of the problems that people were having was, were them blowing up right? Without being able to connect to an actual face, right? So it solves so many of the problems that come with that, and even helps the problem of TikTok's algorithm where people just hop on and start running things up with ads and you haven't really even understood what your content looks like, that creates some algorithmic problems, which probably aren't worth getting into, here, or maybe they are, but yeah. Man, artists generate content. It's gonna be a love hate relationship for sure with artists, the labels, all of us, right? But, if anything, it'll force collaboration and synergy between teams, in ways that it hasn't before.[00:16:16] Dan Runcie: Yeah, I got the impression that from the TikTok head of music, making the comments about user-generated content, of course there's plenty to back that up, but I also saw it as a bit of a positioning to not take the stance that I think some of the labels have taken, where I think that the labels have come a bit of the public enemy of the artists who don't wanna be active on TikTok. We all saw the viral post that happened last towards the end of 2022. It was Florence, Florence of the machine and Halsey and others saying like, Hey, the label's making me do this. But I feel like there's so many ways to go about making short form videos and making content. How involved do you get with that piece of it? Cuz I think some of that is because people still think that artists need to be doing one of these like, you know, vertical TikTok dances that fit in something like they're Jason Derulo or something like that, but you don't necessarily have to do.[00:17:13] Sean Taylor: No, you do not. So again, this is one of those things that I was telling people back in 2020, but the problem was, again, dancers were working so hard and so well, no one's gonna believe you. Right? But we were only seeing the commercial level, right? And everything has levels to it, just like the industry, right? You have pop music and there's some genres that. For less far reaching than pop, but they're successful. Right. So that's what I attribute seeing Dances in 2020 work on TikTok. However, there were other things that were working right? like people thought you had to be a super upbeat hip hop song, cash pages song that blew, was nothing of the sort. Right? But so I think that for one, people have to understand that it just goes back to being creative. At the end of the day, and unfortunately many artists stop thinking creatively once they leave the studio, right? And I don't think it's all the artists now. I used to just blame it on the artist where it's like, bro, you're supposed to be an artist. You wanna be creative, right? Artist means more than a musician. Musician is just music, but artist creativity, that's what we're looking at you for. You have to show creativity and how you present yourself in this content. But I think what happened was there was so much working in terms of these trends, and they saw so many. Finding success so fast, it kind of demoralized them into thinking I have to follow these specific formats to find success myself. Right? So when I hear TikTok, I hear TikTok in a specific way, not just another platform that I can distribute my video on. You know what I mean? It would be like, oh yeah, you could create a movie, but it has to be a romcom. That's kind of what they're hearing, right?. That's not the truth though, right?[00:19:08] Dan Runcie: It reminds me too, of what you used to hear of MTV back in the day as well. Right. A lot of artists, especially late eighties, early nineties, a lot of artists that went on to be huge music video artists resisted it and they would always have a bit of a you know, high brow about it. Like, oh, I'm not trying to be like the Sir Mix-a-lot baby who got back a music video, like dancing on, you know, butts and booty shaking and stuff like that. But they found their own way to make the platform unique years down the road when it became the main thing.[00:19:38] Sean Taylor: That's it, because it's you. At the end of the day, you can create the content and the platform is just how you distribute it. Now, I think there's something to be said for using the unique qualities of a platform, right? Just like albums. what people created, just like CDs impacted what people created. Just like the internet and internet culture has impacted, oh, shorter songs cause shorter intention, span longer songs. Cuz now we have more space to create. Like all those things were like, music has always been impacted by the mediums and the culture around it. Right. And I think for some reason we constantly fall into this trap of, you know, oh, SoundCloud music, TikTok music. You know, at one point in time there was I mean, well, people complain about, I've seen people complain about tape cassettes. You know, like when you look up enough, you're gonna find everybody complaining about everything, right? And then the Grammy's, what's the Grammy's formula? Everything has its success, but truthfully, I'm in their own formula for success. But truthfully, you know, especially in this independent business, you know, you don't have to play every single game. I think sometimes we find ourselves wanting things that cause us to play a game we don't want to, which is like that weird love hate thing. It's like, oh, you know, black people shouldn't pay attention and value the Grammys yet. We still want Grammys. Right. You know what I mean? It is that love and hate relationship. I think everybody's doing that with different platforms in, in, in some form of fashion. and you asked earlier, Deep and involved that we get in people's content creation. It's varying, right? we don't do it with every project, every person. It depends on the vision and also their willingness and the need that's there. But, you know, we've gone as deep as recording things ourselves. I remember one campaign. This wasn't artist generated content, it was an influencer, but we bought something off of Amazon to send her for her to wear in it because it connected with the idea. And she had like 5,000 followers at the time. And the video ended up doing like 2 million. Right. So we were like really A and R ing, cuz sometimes it's, you know, TikTok is about narrative and with the presentation, so just hold, let me go to how many followers. Cuz the beauty of TikTok is you can not have a lot of followers and still get a lot of views.But if so, if you find the right person and can contrast it in the right way. Right. You can make it move. Right. I don't want to get into that campaign cause it might be semi uncontroversial in a way. I gotta explain[00:22:23] Dan Runcie: We'll save that one for offline then,[00:22:25] Sean Taylor: yeah. We'll say that one for, offline. For sure. For sure. But yeah, man, I mean, I think what I've seen is, if people can just open their mind and not start what's moving on a platform and just think literally in Word out, "Hey, what do I want to communicate now? How do I communicate that on this platform?" It'll save a lot of stress, particularly for the artists, because artists wanna do music videos. This is nothing but another video, right? So why can't I in 60 seconds? Be creative. Use that box. That box is a framework that will inspire creativity. How can I communicate and make something really dope in 60 seconds? We've had an artist last August blow up, his profile from like 20K to 400K and did 2 million streams in about a month with very, very high quality videos. And everybody thinks you gotta be really low. To find success on TikTok and record it from your phone and have the bubbles. These were very, very high quality shots and editing, and it's darker and it worked. Right. So it's really just about dope content at the end of the day.[00:23:35] Dan Runcie: Yeah, for sure. Let's switch gears a bit. I want to talk. Talk's, competitors that are also in this space wanna talk YouTube and Instagram. But let's start with YouTube first, because you had recently put out a video where you were talking about YouTube shorts, their efforts there, and you said you're not concerned about YouTube shorts' impact because it just doesn't have the culture that exists on TikTok. Can you talk more about that?[00:24:00] Sean Taylor: yeah. So the thing that made TikTok so unique, early on was it developed a culture, like once it hit that network effect, I knew it wasn't gonna go away overnight cuz there's too much money involved outside of the government stuff. But that's a different story. Right? And then culture, like people, have a different presentation and expectation on how you act on TikTok. It's looser than Instagram. That was the beauty of it, right? So that created a culture. YouTube has an established culture and relationship that they have. their audience. YouTube isn't as interactive. It's a little closer to tv. You know what I mean? And Instagram's a little bit more of a resume. Most people are putting on their best, their Sunday best, if you will. TikTok, we're involved in this together. People feel like they have the power to blow a song up on TikTok. the users feel like they're giving heavy feedback. You should drop this song. When is this gonna come out, right? It's a completely different culture that you can't just copy overnight. That's where competitive advantages get created, right? Culture. Cause it's very, very hard to mimic that. I think it's gonna be successful, but it's just not going to be a threat to TikTok in that specific way where, you know, it's like a TikTok killer or something. It's like the Jordan Stoppers. Oh yeah. You know, Jordan only scored 39 instead of 35. Cool. You might see.[00:25:26] Dan Runcie: So if it's thinking about the Kobe stopper thing, so if TikTok is Kobe Bryant then is YouTube shorts, Ruben Patterson.[00:25:37] Sean Taylor: You too might wanna take offense to that, but in this analogy, yes.[00:25:42] Dan Runcie: Yeah, but I've been thinking a lot about the YouTube piece, and I will give them credit. I think the trajectory of YouTube is greater than Ruben Patterson. No disrespect, but I do think that Lyor Cohen had said something interesting. Of course, he's the head of YouTube and one of his big things is that the fact that YouTube shorts has the connection directly to the platform [00:26:04] Sean Taylor: Yeah.[00:26:04] Dan Runcie: on-demand listening happens, he feels like that conversion rate and that connection is stronger. And he didn't name TikTok specifically, but he was essentially talking about the fact that TikTok doesn't have that same type of win. I know they have Rezo, but it's just not the same. What do you think about that? Cuz I think the underlying aspect of that is conversion and just being able to transport an audience from one to the other. What have you seen from that perspective.[00:26:29] Sean Taylor: I think Lyor Cohen is extremely smart and savvy, and reading that statement. It was hilarious cuz you know the elephants in the room that he's addressing and it was like this competitive moment happening. It was like, come on man. Say their name. Say their name. But you're like, I'm not gonna give them any clout that was really, funny to read. But I think that they do have an advantage that TikTok doesn't. Right in that way, the long form content and that mentality, I think it's gonna be a lot harder for TikTok to get people to consume shorts on their platform than it is for No. It's gonna be a lot harder for TikTok to consume long form content on their flat platform than it is gonna be for YouTube people to consume short form. Does that make sense?[00:27:19] Dan Runcie: Yeah, that makes sense. And I feel like part of it too is the conversion rate is one thing. I don't doubt that there's likely could be a higher conversion rate, but I think that absolute number is still what makes the difference at the end of the day. And I just don't know if the overall absolute number of people that are converting from. Hearing a song on TikTok and then going to stream that artist and then becoming a follower and an avid fan of that artist is necessarily going to be a number that's ever smaller than what we may see otherwise from YouTube. I think YouTube has still had great strides in that area, but I just don't see it coming to that level.[00:27:56] Sean Taylor: no. Cause again, it's about that culture, right? So two things, I'll get to the YouTube TikTok second. But when we first got on TikTok and started working with some people on TikTok, it was ridiculous to see the conversion of people who left TikTok and went to Spotify. Instagram, these other places, but specifically, let's talk about Spotify. Why was it so ridiculous? Because at that time, TikTok did not acknowledge the music on the platform at all, right? You had people hearing the song and then googling the lyrics to find the song name and then going to stream it, and it was happening in droves. That much friction told us, holy shit, when they get rid of this friction, it's gonna go. And of course TikTok got rid of that friction. But pa the fact that people were doing that, like I remember telling some artists, yo man, you got like a dollar sign and this version of the song, and then it's like no dollar sign. Like, and so people were having difficulty finding it. It's like, bro, you're ruining people's ability to stream your song. Like that was a thing. And then people started to rename their songs or added lyrics in the parentheses, right? Because of the culture. And that was happening and they wanted to make sure people could find the song right. Now we see less of that. It goes back to like the mediums and how things are influencing. Now we see less of that cuz you can figure out what it is within TikTok and people know how to name it. So that transferability from TikTok to other platforms has just been there for so long and people almost expected it. It's almost like TikTok is the megaphone, the amplifier. But then you don't even really expect to go super deep on TikTok, right? YouTube, you kind of do expect to go deep, but when we. Look at the platforms that TikTok converted to, and this is where I say the competitive advantage of YouTube goes. YouTube was one of the greatest conversions from TikTok that we would see. Like so many people left TikTok to go to Spotify and YouTube. Instagram was last particularly for artists. Right? So now, yes, being on YouTube already, Is a great competitive advantage. I think there's some fluidity issues that they need to solve. Like right now, from the phone, from your phone, I could take this video that we're creating right now and say, yeah, I want 10 seconds to 20 seconds, and take that into a short and it'll automatically be connected. But I can't make a cool edited short that's specific to the short format, and then say it's from this video. Right. And that'll make things even smoother because it's hard to take a snippet from the long form content just from timestamps and that would be a good piece of short form content. So people have to be able to edit and then connect it back for that to really come into place. And also they would have to make it a little bit more obvious that you can do that because the culture is not yet to go from long. I'm in short form to long form within YouTube. I know it happens, but people aren't naturally having that expectation. Oh shoot. This is probably from a larger video that's on this platform. What people are seeing more on YouTube is actually, mm, it's the opposite of what allowed TikTok to become what it became, and I don't know how it's gonna play out, so I'll tell you what I'm saying.[00:31:23] Dan Runcie: What are your thoughts on Instagram reels? I think you talked a little bit about it, how it's a bit of a resume but where do they stand?[00:31:31] Sean Taylor: I don't like reels, in terms of the value add yet, it's very inaccurate. So you'll get higher numbers, less engagement, where it's pretty clear it's not going out to the right people. The best people you know, they're, padding the numbers, so to speak. Right. It's cool that it got more reach, but if it wasn't accurate and I didn't get that much following, what does it really mean? That's where reels are at large. Now, can reels work and has it still helped some songs? Yes. It's just not it's not at the proportion that TikTok has been, and I think YouTube shorts are going. definitely, beat reels.[00:32:11] Dan Runcie: Yeah. I think the clear desire from Instagram to try to turn your entire feed into a for you page is forcing this content to not necessarily hit the right people, which is why. Yes, it could be good from a viral discovery thing where, okay, if you have a post that's doing better than 80 or 90% of your other posts, then yeah, it may reach a larger than initially intended audience, but I don't know if it could be necessarily relied on in the same type of way.[00:32:41] Sean Taylor: Yep. I agree with you there. We'll see if they figure it out maybe they should focus less on the Metaverse.[00:32:52] Dan Runcie: Another thing that you had brought up a little bit earlier was about search in general and just how powerful that's become on TikTok. I think it's clear that they want to, well, I think TikTok is trying to do anything and everything. I know this is something for folks that follow Trapital been writing about this recently, all the things they're getting involved with. But I do think search is one of those interesting things because they are trying to take on a Google Head on, and people have seen how, especially Gen Z, they may be more likely to look up something through TikTok than looking it up through Google. What do you see as the potential of that moving forward, and do you think that would be a credible threat to Google at some point?[00:33:32] Sean Taylor: yes. You know why tutorial culture, that's what TikTok cut into. And Instagram never did that. It was never really a place that you went to look up tutorials, right? So it's less about music and entertainment and that side of things. It's the fact that people are looking at recipes, right? How to fix things. And then once you have that, that's what creates the. For looking things up in the, at the seo, right? it's not that it can't be created in other ways, but that's like a hack. If I know I can go find this and, oh man, I can find it done in 60 seconds versus three minutes or 10 minutes because on YouTube the video's not as valuable and might not go as far if I don't do an intro and all this leads in, right? Oh, these videos won't go straight to it. So they have a lot of ground to make on YouTube. But I think they're going to succeed, I'm not into the speculation of which one's gonna be number one. There might be days where it beats YouTube or whatever, but it's going to be, a legitimate search engine. You know, Yahoo, Google at least.[00:34:43] Dan Runcie: I do think that the tutorial piece is key, and I'm even thinking about times I've used it in the past. We bought a mattress recently, and of course you could Google search, what does this mattress like? But sometimes it's easier to just put it in TikTok and have someone show me some unboxing video to show me what that's like and compare a few. I liked it for that. I think more broadly in terms of all of the search pieces of it. I think that what Google has done in this space and even thinking about, you know, decades back about how they beat Legos and Alta Vistas, some of those others, I think it will be hard to ever replace that for everything that's possible, that people would wanna search. But I do think that the video tutorial piece of it, which is a subsection of it, but I do think that that's a unique place where they can, if you get core to the market there, you can figure things out. I also think that either misinformation or wasted just credibly have or understanding for users to know, okay, "What is legitimate?, What is not legitimate? Is a concern, and I think it's maybe harder to do in a way where I think text, you can have some of those clear things come up where I think the nature of a viral platform wants to show things that you know, may be sensationalized to some extent through video. That may take time, but I think that's one thing that will need to develop, especially on the TikTok side of things for sure.[00:36:02] Sean Taylor: See what you're explaining is the different personalities of seo. Just like we talk about the different personalities of TikTok, Instagram and YouTube. So Google dominates damn near Monopoly. You couldn't just be a Yahoo or actually, truly become a threat, but what you could. Was be Amazon and eventually have so much shit on there that people just think, "Oh, if I want a table, I'm gonna go Amazon, look for a table, I'm not gonna go to Google", right? Or you could be a YouTube, right? And have so many videos. If I want a video tutorial, I'm just gonna go to YouTube, right? So you enter the search engine market from the side versus, you know, the [inaudible] so I think TikTok has successfully gotten there. and even that personality being short form is something that attributes to that and the personality that Google will probably hold on to,in the long term for sure is probably gonna be the more scholarly approach, right? The more credible approach because of those other platforms. that's almost in conflict with what makes things move and the way people use them, and incentives that are in place.[00:37:15] Dan Runcie: Right. That makes sense. That makes sense. The other thing too, that it'd be good to get your thoughts on with TikTok is I think a lot of this conversation and a lot of how people have been measuring the success for TikTok is that conversion from TikTok and broader social engagement to streams, what are you seeing though from the next level down? From how looking at TikTok itself can eventually translate to concert ticket sales, or whether it's V I P clubs or other high end opportunities that fans are engaging it with an artist.[00:37:50] Sean Taylor: It's there. We've already had artists do that. The difficulty is the geography of it all. So you can do that, but you aren't completely sure that the video is going to go viral enough within your own audience, right? Because still, especially like most of last year, you still can drop a TikTok and it's mostly gonna be seen by new people, especially for people earlier on, right? So if all of your followers aren't necessarily going to see your posts, it's gonna be new. then that creates this issue with going deep with your audience, right? It's great for going viral and gaining and blowing up fast. That's why it happened, right? That built TikTok in a sort of way to show it to more new people than people who are following you. But then at some point it becomes, well, what are my followers really worth? right on TikTok. And I think some people are starting to figure that out. Like, man, I don't know if this really matters all that much. It really only matters what the individual video itself does, right. So the problem with that, if I'm doing a show and I don't know if my followers will see it, or I have no idea if it's good, or enough of video because it's outside of my normal format to get enough people to see it in general. Then, man, that's not predictable enough. It could be my strategy, but it's not predictable enough. Now, the advertising might come into play, which is a different conversation, but there's that, and then again, also, who's gonna see it geographically in the world? We have no, you know, way of controlling that. Again, outside of ads so far. It's definitely something that's useful for selling things like merch, for creating awareness for your shows, but the best way we've seen it with shows for the most part, is almost to talk about your tour as a whole, right?. So you bring awareness that multiple are going to happen, and then if your artist is down for it and it kind of works within your format, you can also vlog in a way, or like let people know, oh yeah, I'm gonna be in Atlanta tonight. Right? So they see and get reminded that you are on tour, right? So it's trying to create this awareness of all the spaces and places that they will be going. And then also reminding them that you're in process of this to remind them, oh yeah, when he's gonna, when is he gonna beat in my city? That's kind of like the best middle ground we've found. But it's hard to be like, Hey, I'm gonna be in Atlanta next week and all my Atlanta people and expect all the Atlanta people to actually see that and convert.[00:40:27] Dan Runcie: Right. it's a funnel at the end of the day, right? And TikTok sits at the top of it, even higher than some other social media platforms, right? And then from there, it's always going to be hardened, honestly foolish to an extent. If your main message on the top of the funnel awareness platform is, "Hey, Join my V I P club or join my Patreon or buy tickets to my concert, right?" You need to introduce people, let them know who you are, and maybe at that next level of engagement, then you can start to push more of those things. Then you can start to have more of these things come through, because there's just gonna be less friction there and you're doing the job that should be done at each level of the.[00:41:08] Sean Taylor: I agree man. I think, like you said, at the end of the day, it always goes back to the fundamentals of it and there might be aberrations. Give us more for moments of time, but things are always gonna default back to that basic infrastructure and use the thing with the right expectations versus expecting everything from it.[00:41:30] Dan Runcie: right. The other thing that I've thought a lot about for this conversation is, and even for reading the report and rolling into the show notes, so others can take a look at it as well, but thinking about how artists generate content and having artists push is what the wave is. At least at this particular moment. And I think there's a lot of reasons to think that yes, this is what makes sense now moving forward, but we also know how quick these things change and how things have evolved. Do you see another element, like I know eventually gaining steam eventually, I know that we talked about ads, we talked about influencer campaigns and just UTC and how a lot of these things were stronger and now relatively weaker to artists generated campaigns. Is there another thing that you think is going to play a role or that we may see another shift in this.[00:42:21] Sean Taylor: The dark horse is TikTok lives. Everybody's actually. Investing more in a live culture in general when you look at YouTube, as well. but TikTok lives the way they use that for you. Page is ridiculous, man. On Instagram, you're gonna see the lives of people that you're following. Again, on TikTok, you can go live and people will discover, right? It'll pop up on the people's For You page, and that's a different paradigm, right? I've seen it live when my partner was live where all of a sudden, like thousands of people came in, right? Because TikTok was feeding him to so many people. and then he would see it also trickle off. Whereas like experimenting, they wanna find a live that's engaging in a way that content isn't, moves up the algorithmic letter, right? So they're, look, that's how they display lives. So the fact that you can blow up doing lives, it's a completely different paradigm because it's like having a show but the people who do it well, they're getting money in these lives. A lot of money. I've seen people make a lot of money in their lives, but it's also a great format to build a relationship far deeper than you can through individual content and lead people over to buy tickets. We've used lives to like getting emails and, I mean, I did one campaign even back in 2020. , that artist Fash we probably got 10,000 emails more so from him going live, not more so, only from him going live. Actually, he even use a little bit of IG live. So the fact that you can do that is going to create this other performance skill that artists will have.[00:44:06] Sean Taylor: Almost like being a salesman, right? But doing it in a way where you give the presentation and the ask isn't so blatant. it's, we're going into this climate that's going to breed so many different types of artists like that have these, you know how you could be an artist that plays an instrument or you could be an artist that sings, maybe you could do multiple, where there's now these soft skills that we'll see artists, oh man, this dude is, he's just a salesman and he knows how to entertain people on live. And that's how he plays his game. This person still is just a pure musician, or this person creates a really dope content in the box of the regular feed. There's gonna be things like that. And the thing is, these other platforms are, you know, homogenous in many ways. They keep copying each other. So that culture that starts on TikTok, it's not just a TikTok thing. I always communicated it as a new language to learn because the new generation would be used to hearing and seeing things and consuming things in this format. So they'll wanna see it on other platforms. And inevitably, right, like we have an hour podcast, two hour podcasts, and people were like, Hey man, can you make this in 60 seconds? Like they expect to be able to learn something really valuable that's gonna change their. You know, in literally 60 seconds. I mean, we literally have had those conversations and seen those comments, but everything's not, you know, how you bake a cake? People like you can't rip at everything and change your artist's career in 60 seconds. But that's what we're seeing, like live is truly a dark horse, and I think it is gonna become more prevalent in TikTok and YouTube as well, to be honest.[00:45:45] Dan Runcie: No, I can see that happening definitely just with the way things are going. But, last question before we let you go here though. So let's fast forward three years. 2026. Is TikTok still in the dominant position that it is right now? And if it isn't, is it because of geopolitical concerns or is it because of another competitor that now has the next big thing?[00:46:10] Sean Taylor: If it isn't geopolitical, I think it's gonna be pretty dominant from what I've seen. in terms of their vertical integration and investment, particularly in music. It's just nothing like any other platform like you've seen, I mean Sound On, right. Rezo. Right. It's just different in what they're trying to do. They have deals that they've offered artists. Right. Which is really nice. Right. And unique because, oh, you're on, sound on and you. You have a song that blows up using their distribution platform. They have all the data. So now we can offer you a deal and you don't have to pay it back cuz it's gonna be paid back through the royalties. We're probably using the algorithm to calculate how much we should give you anyway. Right? This is already happening. Right. So the way they invest in it, I think it's just gonna be hard to get a pool away from it in three years. again, it's going to. More of a norm, less hot in its way, but I think they're gonna be pretty dominant in three, three years down the road. Yeah. I'll, I'll leave that at that.[00:47:10] Dan Runcie: Yeah, I'm with you on that. I think three years, because even though I do think that TikTok has been the fastest to grow to a billion users, at least what we've seen from a social app, I do think that the next app will probably be even shorter just thinking about how much faster adoption is, but. It still took TikTok several years to get to this point, so I think maybe five, six years would be a different conversation. But no, I agree with you. Three years. If it ends up being shut down, it'll be for some geopolitical concerns, but we'll see between now and then. We'll have to check in again at some point if any of that ends up going in that direction. But Sean, it's been a pleasure man. Thank you for coming on and for people that wanna learn more about your insights on TikTok and the stuff you're doing at Contraband Agency, where should we reach out?[00:47:56] Sean Taylor: Brandman Network on YouTube is a nice place to start. You know, you watch the podcast or you just go to no labels necessary on, Spotify, but no labels necessary is our podcast. So type in no labels necessary on YouTube or, or Spotify. At the moment, I think the podcast is probably the best place to go. But if you're immediately interested in services and want to speak with our team, that would be contraband.agency. There's no.com, www.contrabrand.agency[00:48:28] Dan Runcie: Good stuff. All right. Thanks again, Sean. Appreciate you.[00:48:31] Sean Taylor: Always good speaking with you man.[00:48:33] Dan Runcie Outro: If you enjoyed this podcast, go ahead and share it with a friend. Copy the link, text it to a friend, post it in your group chat. Post it in your Slack groups. Wherever you and your people talk, spread the word. That's how Trapital continues to grow and continues to reach the right people. And while you're at it, if you use Apple Podcast, go ahead. Rate the podcast, give it a high rating, and leave a review. Tell people why you like the podcast. That helps more people. Discover the show. Thank you in advance. Talk to you next week.

Subscribe to Changelog++: https://changelog.com/podcast/519/discussFeaturing Shawn Wang – Twitter, GitHub, Website Adam Stacoviak – Mastodon, Twitter, GitHub, LinkedIn, Website Jerod Santo – Mastodon, Twitter, GitHub, LinkedIn Notes and Links AI Notes Why “Prompt Engineering” and “Generative AI” are overhyped Multiverse, not Metaverse The Particle/Wave Duality Theory of Knowledge OpenRAIL: Towards open and responsible AI licensing frameworks Open-ish from Luis Villa ChatGPT for Google The Myth of The Infrastructure Phase ChatGPT examples in the wild Debugging code TypeScript answer is wrong Fix code and explain fix dynamic programming Translating/refactoring Wasplang DSL AWS IAM policies Code that combines multiple cloud services Solving a code problem Explain computer networks homework Rewriting code from elixir to PHP Turning ChatGPT into an interpreter for a custom language, and then generating code and executing it, and solving Advent of Code correctly Including getting #1 place “I haven't done a single google search or consulted any external documentation to do it and I was able to progress faster than I have ever did before when learning a new thing.” Build holy grail website and followup with framework, copy, repsonsiveness For ++ subscribers Getting Senpai To Notice You Moving to Obsidian as a Public Second Brain Transcript**Jerod Santo:** Alright, well we have Sean Wang here again. Swyx, welcome back to the show.**Shawn Wang:** Thanks for having me back on. I have lost count of how many times, but I need to track my annual appearance on the Changelog.**Adam Stacoviak:** Is that twice this year on this show, and then once on JS Party at least, right?**Shawn Wang:** Something like that, yeah. I don't know, it's a dream come true, because, I changed careers into tech listening to the Changelog, so every time I'm asked on, I'm always super-grateful. So yeah, here to chat about all the hottest, latest things, right?**Adam Stacoviak:** Yeah.**Jerod Santo:** That's right, there's so much going on right now. It seems like things just exploded this fall. So we had Stable Diffusion back in late August; it really blew up at the end of August. And then in September is when we had Simon Willison on the show to talk about Stable Diffusion breaking the internet. You've been tracking this stuff really closely. You even have a Substack, and you've got Obsidian notes out there in the wild, and then of course, you're learning in public, so whenever Swyx is learning something, we're all kind of learning along with you... Which is why we brought you back on. I actually included your Stable Diffusion 2.0 summary stuff in our Changelog News episode a couple of weeks back, and a really interesting part of that post that you have, that I didn't talk about much, but I touched on and I want you to expand upon here is this idea of prompt engineering, not as a cool thing, but really as a product smell. And when I first saw it, I was like, "No, man, it's cool." And then I read your explainer and I'm like, "No, he's right. This is kind of a smell."**Adam Stacoviak:** "Dang it, he's right again."**Jerod Santo:** Yeah. We just learned about prompt engineering back in September, with Simon, and talking about casting spells and all this, and now it's like, well, you think it's overhyped. I'll stop prompting you, and I'll just let you engineer an answer.**Jerod Santo:** Well, so I don't know if you know, but the Substack itself got its start because I listened to the Simon episode, and I was like, "No, no, no. Spellcasting is not the way to view this thing. It's not something we glorify." And that's why I wrote "Multiverse, not Metaverse", because the argument was that prompting is -- you can view prompting as a window into a different universe, with a different seed, and every seed is a different universe. And funny enough, there's a finite number of seeds, because basically, Stable Diffusion has a 512x512 space that determines the total number of seeds.So yeah, prompt engineering [unintelligible 00:04:23.23] is not my opinion. I'm just reporting on what the AI thought leaders are already saying, and I just happen to agree with it, which is that it's very, very brittle. The most interesting finding in the academic arena about prompt engineering is that default GPT-3, they ran it against some benchmarks and it came up with like a score of 17 out of 100. So that's a pretty low benchmark of like just some logical, deductive reasoning type intelligence tests. But then you add the prompt "Let's think step by step" to it, and that increases the score from 17 to 83... Which is extremely -- like, that sounds great. Like I said, it's a magic spell that I can just kind of throw onto any problems and make it think better... But if you think about it a little bit more, like, would you actually use this in a real work environment, if you said the wrong thing and it suddenly deteriorates in quality - that's not good, and that's not something that you want to have in any stable, robust product; you want robustness, you want natural language understanding, to understand what you want, not to react to random artifacts and keywords that you give.Since then, we actually now know why "Let's think step by step" is a magic keyword, by the way, because -- and this is part of transformer architecture, which is that the neural network has a very limited working memory, and if you ask a question that requires too many steps to calculate the end result, it doesn't have the working memory to store the result, therefore it makes one up. But if you give it the working memory, which is to ask for a longer answer, the longer answer stores the intermediate steps, therefore giving you the correct result.**Jerod Santo:** [06:00] Talk about implementation detail, right?**Shawn Wang:** It's yeah, it's leaking implementation detail, it's not great, and that's why a lot of the thought leaders - I think I quoted Andrej Karpathy, who was head of AI at Tesla, and now he's a YouTuber... [laughter] And Sam Altman, who is the CEO of -- yeah, he quit Tesla to essentially pursue an independent creator lifestyle, and now he's a YouTuber.**Jerod Santo:** I did not know that.**Adam Stacoviak:** All roads lead to creator land, you know what I'm saying? You'll be an expert in something for a while, and eventually you'll just eject and be like "I want to own my own thing, and create content, and educate people around X."**Shawn Wang:** So at my day job I'm a head of department now, and I work with creators, and some of them have very valuable side hustles... And I just had this discussion yesterday, of like "Why do you still have a job if you're an independent creator? Like, isn't total independence great." And I had to remind them, "No. Like, career progression is good. You're exposed to new things etc." but that's just me trying to talk him out of quitting. [laughter] No, I have a serious answer, but we're not here to talk about that.**Jerod Santo:** Right.**Shawn Wang:** So I'll read out this quote... So Sam Altman, CEO of OpenAI, says "I don't think we'll still be doing prompt engineering in five years. It's not about figuring out how to hack the prompt by adding one magic word to the end that changes everything else. What will matter is the quality of ideas and the understanding that you want." I think that is the prevailing view, and I think as people change models, they are understanding the importance of this.So when Stable Diffusion 1 came out, everyone was like, "Alright, we know how to do this. I'm going to build an entire business on this" etc. And then Stable Diffusion 2 came out and everything broke. All the [unintelligible 00:07:40.21] stopped working, because they just expected a different model, and you have to increase your negative prompting, and people are like "What is negative prompting?" etc. These are all new techniques that arise out of the model, and this is going to happen again and again and again, because you're relying on a very, very brittle foundation.Ultimately, what we want to get people to is computers should understand what we want. And if we haven't specified it well enough, they should be able to ask us what we want, and we should be able to tell them in some capacity, and eventually, they should produce something that we like. That is the ultimate alignment problem.We talk about AI a lot, and you hear about this alignment problem, which is basically some amount of getting it to do what we want it to do, which is a harder problem than it sounds until you work with a programmer, and try to give them product specs and see how many different ways they can get it wrong. But yeah, this is an interesting form of the alignment problem, and it interestingly has a very strong tie with Neuralink as well, because the problem, ultimately, is the amount of bandwidth that we can transfer from our brain to an artificial brain. And right now it's prompts. But why does it have to be prompts? It could be images. That's why you have image-to-image in Stable Diffusion. And it could also be brain neural connections. So there's a lot in there; I'll give you time to pick on whatever you respond to...**Jerod Santo:** Well, I went from -- so I was super-excited about prompting after talking with Simon a few months back, and I was super-excited about Stable Diffusion. And I went from like giddy schoolboy who's just like "Gonna learn all the spells" very quickly to like aggravated end user who's like "Nah, I don't want to go to this other website and copy and paste this paragraph of esoterica in order to get a result that I like." And so I wonder what's so exciting about the whole prompt engineering thing to us nerds, and I think maybe there's like a remnant of "Well, I still get to have esoteric knowledge" or "I still get to be special somehow if I can learn this skill..."[09:46] But in reality, what we're learning, I think, by all the people using ChatGPT - the ease of use of it, as opposed to the difficulty of getting an image out of Stable Diffusion 1.0 at least, is quite a bit different. And it goes from aggravating and insider baseball kind of terms, keywords, spells, to plain English, explain what you want, and maybe modify that with a follow-up, which we'll get into ChatGPT, but we don't necessarily have to go into the depths of that right now... But I changed very quickly, even though I still thought prompt engineering was pretty rad... And then when you explain to me how Stable Diffusion 2 completely broke all the prompts, I'm like, "Oh yeah, this is a smell. This doesn't work. You can't just completely change the way it works on people..." That doesn't scale.**Shawn Wang:** Yeah. And then think about all the businesses that have been built already. There haven't been any huge businesses built on Stable Diffusion, but GPT-3 has internal models as well. So Jasper recently raised like a 1.5 billion valuation, and then ChatGPT came out, basically validating Jasper... So all the people who bought stock are probably not feeling so great right now. [laughs]That's it. So I don't want to overstate my position. There are real moats to be built around AI, and I think that the best entrepreneurs are finding that regardless of all these flaws. The fact that there are flaws right now is the opportunity, because so many people are scared off by it. They're like, "AI has no moats. You're just a thin wrapper around OpenAI." But the people who are real entrepreneurs figure it out. So I think it's just a really fascinating case study in technology and entrepreneurship, because here's a new piece of technology nobody knows how to use and productize, and the people who figure out the playbook are the ones who win.**Adam Stacoviak:** Yeah. Are we back to this -- I mean, it was like this years ago, when big data became a thing... But are we back to this whole world where -- or maybe we never left, where "Data is the new oil", is the quote... Because to train these models, you have to have data. So you could be an entrepreneur, you could be a technologist, you could be a developer, you could be in ML, you could be whatever it might take to build these things, but at some point you have to have a dataset, right? Like, how do you get access to these datasets? It's the oil; you've got to have money to get these things, you've got to have money to run the hardware to enable... Jerod, you were saying before the call, there was speculation of how much it costs to run ChatGPT daily, and it's just expensive. But the data is the new oil thing - how does that play into training these models and being able to build the moat?**Shawn Wang:** Yeah. So one distinction we must make there is there is a difference between running the models, which is just inferences, which is probably a few orders of magnitude cheaper than training the models, which are essentially a one-time task. Not that many people continuously train, which is nice to have, but I don't think people actually care about that in reality.So the training of the models ranges between -- and let's just put some bounds for people. I love dropping numbers in podcasts, by the way, because it helps people contextualize. You made an oblique reference to how much ChatGPT costs, but let's give real numbers. I think the guy who did an estimate said it was running at $3 million a month. I don't know if you heard any different, but that's...**Jerod Santo:** I heard a different estimate, that would have been more expensive, but I think yours is probably more reliable than mine... So let's just go with that.**Shawn Wang:** I went through his stuff, and I was like, "Yeah, okay, this is on the high end." I came in between like one to three as well. It's fine. And then for training the thing - so it's widely known or widely reported that Stable Diffusion cost 600k for a single run. People think the full thing, including R&D and stuff, was on the order of 10 million. And GPT-3 also costs something on the order of tens of millions. So I think that is the cost, but then also that is training; that is mostly like GPU compute. We're not talking about data collection, which is a whole other thing, right?[13:46] And I think, basically, there's a towering stack of open source contributions to this data collective pool that we have made over time. I think the official numbers are like 100,000 gigabytes of data that was trained for Stable Diffusion... And it's basically pooled from like Flickr, from Wikipedia, from like all the publicly-available commons of photos. And that is obviously extremely valuable, because -- and another result that came out recently that has revolutionized AI thinking is the concept of Chinchilla Laws. Have you guys covered that on the show, or do I need to explain that?**Adam Stacoviak:** Chinchilla Laws misses the mark for me. Please tell. I like the idea though; it sounds cool, so please...**Shawn Wang:** Yeah, they just had a bunch of models, and the one that won happened to be named Chinchilla, so they kind of went with it. It's got a cute name. But the main idea is that we have discovered scaling laws for machine learning, which is amazing.So in the sort of classical understanding of machine learning, you would have a point at which there's no further point to train. You're sort of optimizing for a curve, and you get sort of like diminishing returns up to a certain point, and then that's about it. You would typically conclude that you have converged on a global optimum, and you kind of just stop there. And mostly, in the last 5 to 10 years, the very depressing discovery is that this is a mirage. This is not a global optimum, this is a local optimum... And this is called the Double Dissent Problem. If you google it, on Wikipedia you'll find it... Which is you just throw more data at it, it levels off for a bit, and then it continues improving. And that's amazing for machine learning, because that basically precipitated the launch of all these large models. Because essentially, what it concludes is that there's essentially no limit to how good these models are, as long as you can throw enough data at it... Which means that, like you said, data is the new oil again, but not for the old reason, which is like "We're gonna analyze it." No, we're just gonna throw it into all these neural nets, and let them figure it out.**Adam Stacoviak:** Yeah. Well, I think there's a competitive advantage though if you have all the data. So if you're the Facebooks, or if you're the Google, or X, Y, or Z... Instagram, even. Like, Instagram ads are so freakin relevant that --**Jerod Santo:** Apple...**Adam Stacoviak:** Yeah, Apple for sure.**Jerod Santo:** TikTok...**Adam Stacoviak:** Yeah. Gosh... Yeah, TikTok. Yeah, the point is, these have a competitive advantage, because they essentially have been collecting this data, would-be to analyze, potentially to advertise to us more, but what about in other ways that these modes can be built? I just think like, when you mentioned the entrepreneurial mind, being able to take this idea, this opportunity as this new AI landscape, to say, "Let me build a moat around this, and not just build a thin layer on top of GPT, but build my own thing on all together", I've gotta imagine there's a data problem at some point, right? Obviously, there's a data problem at some point.**Shawn Wang:** So obviously, the big tech companies have a huge headstart. But how do you get started collecting this data as a founder? I think the story of Midjourney is actually super-interesting. So between Midjourney, Stability AI and OpenAI, as of August, who do you think was making the most money? I'll give you the answer, it was Midjourney.**Jerod Santo:** Oh, I was gonna guess that. You can't just give us the answer...**Shawn Wang:** Oh... [laughs]**Jerod Santo:** I had it.**Shawn Wang:** But it's not obvious, right? Like, the closed source one, that is not the big name, that doesn't have all the industry partnerships, doesn't have the celebrity CEO, that's the one that made the most money.**Jerod Santo:** Yeah. But they launched with a business model immediately, didn't they? They had a subscription out of the box.**Shawn Wang:** Yeah, they did. But also, something that they've been doing from the get-go is that you can only access Midjourney through Discord. Why is that?**Jerod Santo:** Right. Because it's social, or... I don't know. What do you think? That's my guess, because they're right in front of everybody else.**Shawn Wang:** Data.**Adam Stacoviak:** Data.**Jerod Santo:** Oh...**Adam Stacoviak:** Please tell us more, Shawn.**Shawn Wang:** Because the way that you experience Midjourney is you put in a prompt, it gives you four images, and you pick the ones that you like for enhancing. So the process of using Midjourney generates proprietary data for Midjourney to improve Midjourney. So from v3 to v4 of Midjourney they improved so much that they have carved out a permanent space for their kind of visual AI-driven art, that is so much better than everyone else because they have data that no one else has.**Jerod Santo:** [17:55] That's really cool.**Adam Stacoviak:** And that's relevance, or is it like quality takes? What is the data they actually get?**Shawn Wang:** Preference, right?**Jerod Santo:** What's good.**Shawn Wang:** Yeah. Literally, you type in a prompt, unstructuredly it tells you -- they give you four low-res images, and you have to pick one of the four to upscale it. By picking that four, they now have the data that says "Okay, out of these four, here's what a human picks." And it's and it's proprietary to them, and they paid nothing for it, because it's on Discord. It's amazing.**Jerod Santo:** That is awesome.**Shawn Wang:** They didn't build a UI, they just used Discord. I don't know if Discord knows this, or cares... But it's pretty freakin' phenomenal...**Jerod Santo:** That's pretty smart.**Shawn Wang:** ...because now they have this--**Adam Stacoviak:** It's the ultimate in scrappy, right? It's like, by any means necessary. That's the ultimate binding that's necessary, right? You'll make a beat however you can to put up the track and become the star.**Jerod Santo:** Right.**Adam Stacoviak:** That's amazing.**Jerod Santo:** That's really cool.**Shawn Wang:** So just to close this out, the thing I was saying about Chinchilla was "More data is good, we've found the double descent problem. Now let's go get all the data that's possible." I should make a mention about the open source data attempts... So people understand the importance of data, and basically Luther.AI is kind of the only organization out there that is collecting data that anyone can use to train anything. So they have two large collections of data called The Stack and The Pile, I think is what it's called. Basically, the largest collection of open source permissively-licensed text for you to train whatever language models you want, and then a similar thing for code. And then they are training their open source equivalents of GPT-3 and Copilot and what have you. But I think those are very, very important steps to have. Basically, researchers have maxed out the available data, and part of why Open AI Whisper is so important for OpenAI is that it's unlocking sources of text that are not presently available in the available training data. We've basically exhausted, we're data-constrained in terms of our ability to improve our models. So the largest source of untranscribed text is essentially on YouTube, and there's a prevailing theory that the primary purpose of Whisper is to transcribe all video, to get text, to train the models... [laughs] Because we are so limited on data.**Adam Stacoviak:** Yeah. We've helped them already with our podcasts. Not that it mattered, but we've been transcribing our podcasts for a while, so we just gave them a leg up.**Shawn Wang:** You did.**Adam Stacoviak:** And that's open source on GitHub, too. They probably -- I mean, ChatGPT knows about Changelog. They know that -- Jerod, I don't know if I told you this yet, but I prompted that; I said "Complete the sentence "Who's the hosts of the Changelog podcast?" "Well, that's the dynamic duo, Jerod Santo and Adam Stacoviak." It knows who we are. I mean, maybe it's our transcripts, I don't know, but it knows...**Jerod Santo:** Please tell me it called us "the dynamic duo"... [laughs]**Adam Stacoviak:** I promise you!**Jerod Santo:** It said that?**Adam Stacoviak:** I promise you it said that. "The dynamic duo..."**Jerod Santo:** Oh, [unintelligible 00:20:34.05]**Adam Stacoviak:** It actually reversed the order. It said Adams Stacoviak first and then Jerod Santo... Because usually, my name is, I guess, first, because - I have no clue why it's ever been that way, but... It said "The dynamic duo, Adam Stacoviak and Jerod Santo..."**Jerod Santo:** That's hilarious.**Adam Stacoviak:** ...hosts of the Changelog Podcast.**Jerod Santo:** It already understands flattery.**Adam Stacoviak:** Yeah, it does. Well, actually, the first prompt didn't include us, and I said "Make it better, and include the hosts." And that's all I said, was "Make it better and include the hosts." So in terms of re-prompting, or refining the response that you get from the prompts - that to me is like the ultimate human way to conjure the next available thing, which is try again, or do it better by giving me the hosts, too. And the next one was flattery, and actually our names in the thing. So... It's just crazy. Anyways...**Shawn Wang:** Yeah, so that is the big unlock that ChatGPT enabled.**Jerod Santo:** Totally.**Shawn Wang:** Which is why usually I take a few weeks for my takes to marinate, for me to do research, and then for me to write something... But I had to write something immediately after ChatGPT to tell people how important this thing is. It is the first real chat AI, which means that you get to give human feedback. And this theme of reinforcement learning through human feedback is - the low-res version of it was Midjourney. Actually, the lowest-res version of it was TikTok, because every swipe is human feedback. And being able to incorporate that into your -- and same for Google; every link click is a is human feedback. But the ability to incorporate that and to improve the recommendations and the generations is essentially your competitive advantage, and being able to build that as part of your UI... Which is why, by the way, I have been making the case that frontend engineers should take this extremely seriously, because guess who's very good at making a UI?**Adam Stacoviak:** Yeah, for sure.**Shawn Wang:** But yeah, ChatGPT turns it from a one-off zero-shot experience where you prompt the thing, and then you get the result, and it's good or bad, that's about the end of the story - now it's an interactive conversation between you and the bot, and you can shape it to whatever you want... Which is a whole different experience.**Break:** [22:31]**Adam Stacoviak:** "Complete the sentence" has been a hack for me to use, particularly with ChatGPT. "Complete the sentence" is a great way to easily say "Just give me somebody long, given these certain constraints."**Jerod Santo:** Well, that's effectively what these models are, right? They're auto-complete on steroids. Like, they are basically auto-completing with a corpus of knowledge that's massive, and guessing what words semantically should come next, kind of a thing... In layman's terms; it's more complicated than that, of course, but they are basically auto-completers.**Adam Stacoviak:** Yeah. On that note though, we have a show coming out... So we're recording this on a Friday, the same day we release the same podcast, but it's the week before. So we had Christina Warren on, and so I was like "You know what? I'm gonna use ChatGPT to give me a leg up. Let me make my intro maybe a little easier, and just spice it up a little bit." So I said "Complete the sentence "This week on the Changelog we're talking to Christina Warren about..." and then I ended the quote, and I said "and mention her time at Mashable, film and pop culture, and now being a developer advocate at GitHub." And I've gotta say, most of, 50% of the intro for the episode with Christina is thanks to ChatGPT. I don't know if I break the terms of service by doing that or not, but like -- do I? I don't know. If I do, sue me. I'm sorry. But... Don't sue me. Don't sue us. We'll take it down. We'll axe it out.**Jerod Santo:** We'll rewrite it.**Adam Stacoviak:** Yeah, we'll rewrite it. But, I mean, it's basically what I would have said. So...**Shawn Wang:** There's a nice poetry -- there's a YouTuber who's been on this forever, Two Minute Papers, and what he often says is, "What a time to be alive." And this is very much what a time to be alive. But not just because we're seeing this evolve live, but because we get to be part of the training data. And there was a very interesting conversation between Lex Fridman and Andrej Andrej Karpathy; he was inviting him on to the show... He said, "Our conversation will be immortalized in the training data. This is a form of immortality, because we get to be the first humans essentially baked in." [laughter]**Jerod Santo:** Essentially baked in... Hello, world.**Shawn Wang:** Like, 100-200 years from now, if someone has the Changelog podcast, they will keep having Jerod and Adam pop up, because they're in the goddamn training data. [laughs]**Jerod Santo:** They're like "Come on, these guys have been dead for a long time."**Adam Stacoviak:** [26:05] Let them go. Give them their RIP. [laughter]**Shawn Wang:** Which is poetic and nice. Yeah.**Adam Stacoviak:** Yeah, it is a good time to be alive... I think it is interesting, too... I just wonder -- I mean, this might be jumping the shark a little bit, but I often wonder, at what point does humanity stop creating? And at some point, 100 years from now, or maybe more, I don't know, we're gonna be -- maybe sooner, given how fast this is advancing, that we'll create only through what was already created. "At what point is the snake eating the snake?" kind of thing. Like, is there an end to human creativity at some point, because we are just so reliant, at some point, shape, or form, on [unintelligible 00:26:45.20] because of training data, and this just kind of like morphing to something much, much bigger in the future?**Shawn Wang:** So I have an optimistic attitude to that... This question basically is asking, "Can we exhaust infinity?" And so my obvious answer is no. There is a more concrete stat I can give you, which is I think - this is floating around out there. Don't quote me on the exact number, but apparently, 10% of all Google searches every single year have never been asked before. And Google's been around for like 20 years.**Adam Stacoviak:** That's a big percentage.**Shawn Wang:** It's still true. So it's on that order; it might be like 7%, it might be 13%.**Adam Stacoviak:** Well, is it trending down though? Is it trending down? Is it 10% per year, but is it like trending down to like 8%?**Jerod Santo:** Is it because we put the year in our searches? [laughter]**Adam Stacoviak:** Yeah, it's true, Jerod. Good one.**Shawn Wang:** Yeah. But anyway, so that's what the SEO people talk about when they talk about long tail... The amount of infinity is always bigger than our capability of creating to fill it.**Jerod Santo:** I mean, I feel like if you look at us in an abstract way, humans, we are basically taking in inputs and then generating outputs. But that's creativity, right? So I think what we're just doing is adding more to the inputs. Now we have computers that also take in inputs and generate outputs, but like, everything's already a remix, isn't it? Our life experience and everything that goes into us, and then something else produces a brand new thing, which isn't really new, but it's a remix of something else that we experienced... So I feel like we're just going to keep doing that, and we'll have computer aid at doing that, and the computer eventually maybe will just do the actual outputting part, but we somehow instruct it. I'm with Swyx on this one; I don't think there's going to be an end to human creativity, as the AI gets more and more output... What's the word? When you're just -- not notorious. What's it called when you just can't stop outputting stuff?**Adam Stacoviak:** I don't know.**Jerod Santo:** Prolific!**Adam Stacoviak:** Prolific.**Jerod Santo:** As the AI gets more and more output-prolific, and overwhelms us with output, I think we're still going to be doing our thing.**Adam Stacoviak:** Yeah. It's the ultimate reduction in latency to new input, right? Think of 100 years ago - creative folks were few and far between. They had miles between them, depending on your system; maybe it's kilometers. No offense. But there's distance of some sort of magnitude, and the lack of connection and shared ideas. So that's the latency, right? And now, the latency to the next input is just so small in comparison, and will get reduced to basically nothing. So we'll just constantly be inputting and outputting creativity, we'll just become like a creative [unintelligible 00:29:31.17] system with zero latency, nonstop creativity... Go, go, go...**Shawn Wang:** Well, I think this is where you start -- I don't know about you, but I feel a little bit uncomfortable with that, right? Entropy is always increasing in the universe; we're contributing to increasing noise and not signal. And that is a primary flaw of all these language models, is just they are very confidently incorrect. They have no sense of physics, no sense of logic; they will confidently assert things that are not true, and they're trained on sounding plausible, rather than being true.**Jerod Santo:** Right. They're kind of like me when I was in college, you know?**Shawn Wang:** Exactly. [laughter]**Jerod Santo:** [30:10] Just so much confidence, but wrong most of the time. [laughs]**Shawn Wang:** Exactly. Which happens to Galactica, which is this sort of science LLM from Meta, where Yann LeCun, who is one of the big names in tech, was like "This thing will generate papers for you." And within three days, the internet tore it apart, and they had to take it down. It was a very, very dramatic failure, this kind of tech... Because you're talking about biology, and science, and medicine, and you can't just make stuff up like that. [laughs]**Jerod Santo:** Right. So like in the world where chat GPT operates today, which is really in the world of fiction, and kind of BS-ing, for lack of a better term, like writing intros to a podcast - you know, like, it doesn't have to be correct necessarily; it can be like close enough to correct, and then you can massage it, of course, you can cherry pick to get the one that you like... But when the rubber hits the road, like on serious things, like science, or "How many of these pills do I need to take?" I guess that is also -- that's health science. So science, and other things... It's like, it can't be correct 60% of the time, or 80%, or even like 95%. It's gotta reach that point where you actually can trust it. And because we're feeding it all kinds of information that's not correct, de facto... Like, how much of the internet's wrong? Most of it, right?**Adam Stacoviak:** I mean, medicine though has evolved too, and it hasn't always been correct, though it's also very serious... You'd get advice from a doctor 10-15 years ago, they'd say it with full confidence and full accuracy, but it's only based on that current dataset.**Jerod Santo:** But you can sue them for malpractice and stuff, right? Like, how do we take recourse against--**Adam Stacoviak:** You can if they actually have malpractice; they can be wrong, because it's as much science as possible to make the most educated guess. It's malpractice when there's negligence; it's not malpractice when they're wrong.**Jerod Santo:** A good doctor will actually go up to the fringe and say, "You know what - I'm not 100% sure about this. It's beyond my knowledge."**Adam Stacoviak:** Sure. For sure.**Jerod Santo:** "Here's what you can do. Here's the risks of doing that." Whereas the chat bots, the ChatGPT thing is like, "The answer is 7", and you're like, "It actually was 12." And it's like, "Ah, shoot..." [laughter]**Adam Stacoviak:** Well, I think when there's mortality involved, maybe there's going to be a timeframe when we actually begin to trust the future MedGPT, for example; I don't know if that's a thing in the future, but something that gives you medical results or responses based upon data, real data, potentially, that you get there, but it's not today.**Jerod Santo:** Well, I think this goes back to the data point that you made, and I think where we go from like the 95 -- I'm making up numbers here, but like 95% accuracy, to get it to like 98.5%, or 99%. Like, that's gonna require niche, high-value, high-signal data that maybe this medical facility has, because they've been collecting it for all these years. And they're the only ones who have it. And so maybe that's where you like carve out proprietary datasets that take these models from a baseline of accuracy, to like, in this particular context of health it's this much accuracy. And then maybe eventually you combine all those and have a super model. I don't know... Swyx, what do you think?**Shawn Wang:** I love the term super-model. I think the term [unintelligible 00:33:23.10] in the industry is ensemble. But that just multiplies the costs, right? Like if you want to run a bank of five models, and pick the best one, that obviously 6x-es your cost. So not super-interesting; good for academic papers, but not super-interesting in practice, because it's so expensive.There's so many places to go with this stuff... Okay, there's one law that I love, which is Brandolini's Law. I have this tracking list of eponymous laws... Brandolini's law is people's ability to create bulls**t far exceeds the ability of people to refute it. Basically, if all of these results of this AI stuff is that we create better bulls***t engines, it's not great. And what you're talking about, the stuff with like the 90% correct, 95% correct - that is actually a topic of discussion. It's pretty interesting to have the SRE type conversation of "How many nines do you need for your use case, and where are we at right now?" Because the number of nines will actually improve. We are working on -- sorry, "we" as in the collective human we, not me personally...**Adam Stacoviak:** [34:32] The royal we, yes.**Shawn Wang:** The role royal we... Like, humanity is working on ways to improve, to get that up. It's not that great right now, so that's why it's good for creativity and not so much for precision, but it will get better. One of the most viral posts on Hacker News is something that you featured, which is the ability to simulate virtual machines instead of ChatGPT-3, where people literally opened -- I mean, I don't know how crazy you have to be, but open ChatGPT-3, type in LS, and it gives you a file system. [laughter]**Jerod Santo:** But that only exists -- it's not a real file system, it's just one that's [unintelligible 00:35:00.05]**Shawn Wang:** It's not a real file system, for now. It's not a real set file system for now, because they hallucinate some things... Like, if you ask it for a Git hash, it's gonna make up a Git hash that's not real, because you can verify [unintelligible 00:35:10.25] MD5. But like, how long before it learns MD5? And how long before it really has a virtual machine inside of the language model? And if you go that far, what makes you so confident that we're not in one right now? [laughs]**Jerod Santo:** Now I'm uncomfortable... That actually is a very short hop into the simulation hypothesis, because we are effectively simulating a brain... And if you get good enough at simulating brains, what else can you simulate?**Adam Stacoviak:** What else WOULD you want to simulate? I mean, that's the Holy Grail, a brain.**Shawn Wang:** Yeah. So Emad Mostaque is the CEO of Stability AI. He's like, "We're completely unconcerned with the AGI. We don't know when it'll get here. We're not working on it. But what we're concerned about is the ability to augment human capability. People who can't draw now can draw; people who can't write marketing texts or whatever, now can do that." And I think that's a really good way to approach this, which is we don't know what the distant future is gonna hold, but in the near future, this can help a lot of people.**Adam Stacoviak:** It's the ultimate tool in equality, right? I mean, if you can do --**Shawn Wang:** Yeah, that's a super-interesting use case. So there was a guy who was like sort of high school-educated, not very professional, applying for a job. And what he used ChatGPT to do was like "Here's what I want to say, and please reward this in a professional email." And it basically helped to pass the professional class status check. Do you know about the status checks? All the other sort of informal checks that people have, like "Oh, we'll fly you in for your job interview... Just put the hotel on your credit card." Some people don't have credit cards. And likewise, when people email you, you judge them by their email, even though some haven't been trained to write professionally, right? And so yeah, GPT is helping people like that, and it's a huge enabler for those people.**Adam Stacoviak:** Hmm... That is -- I mean, I like that idea, honestly, because it does enable more people who are less able... It's a net positive.**Shawn Wang:** Yeah. I mean, I seem generally capable, but also, I have RSI on my fingers, and sometimes I can't type. And so what Whisper is enabling me to do, and Copilot... So GitHub, at their recent GitHub Universe, recently announced voice-enabled Copilot... And it is good enough for me to navigate VS Code, and type code with Copilot and voice transcription. Those are the two things that you need; and they're now actually good enough that I don't have to learn a DSL for voice coding, like you would with Talon, or the prior solutions.**Adam Stacoviak:** You know, it's the ultimate -- if you're creative enough, it's almost back to the quote that Sam had said, that you liked... Well, I'm gonna try and go back to it; he says "At the end, because they were just able to articulate it with a creative eye that I don't have." So that to me is like insight, creativity; it's not skill, right? It's the ability to dream, which is the ultimate human skill, which is - since the beginning of time, we've been dreamers.**Shawn Wang:** [38:01] This is a new brush. Some artists are learning to draw with it. There'll be new kinds of artists created.**Adam Stacoviak:** Provided that people keep making the brush, though. It's a new brush...**Shawn Wang:** Well, the secret's out; the secret's out that you can make these brushes.**Jerod Santo:** Right.**Adam Stacoviak:** Yeah, but you still have to have the motivation to maintain the brush, though.**Jerod Santo:** What about access, too? I mean, right now you're talking about somebody who's made able, that isn't otherwise, with let's just say ChatGPT, which is free for now. But OpenAI is a for profit entity, and they can't continue to burn money forever; they're gonna have to turn on some sort of a money-making machine... And that's going to inevitably lock some people out of it. So now all of a sudden, access becomes part of the class, doesn't it? Like, you can afford an AI and this person cannot. And so that's gonna suck. Like, it seems like open source could be for the win there, but like you said, Swyx, there's not much moving and shaking in that world.**Adam Stacoviak:** Well, I haven't stopped thinking about what Swyx said last time we talked, which was above or below the API, which is almost the same side of the coin that we talked about last time, which is like, this the same thing.**Jerod Santo:** Yeah. Well, ChatGPT is an API, isn't it?**Shawn Wang:** Nice little callback. Nice. [laughter]**Adam Stacoviak:** I really haven't been able to stop thinking about it. Every time I use any sort of online service to get somebody to do something for me that I don't want to do, because I don't have the time for it, or I'd rather trade dollars for my time, I keep thinking about that above or below the API, which is what we talked about. And that's what Jerod has just brought up; it's the same exact thing.**Shawn Wang:** Yep, it is. One more thing I wanted to offer, which is the logical conclusion to generative. So that post where we talked about why prompt engineering is overrated - the second part of it is why you shouldn't think about this as generative... Because right now, the discussion we just had was only thinking about it as a generative type of use case. But really, what people want to focus on going forward is -- well, two things. One is the ability for it to summarize and understand and reason, and two, for it to perform actions. So the emerging focuses on agentic AI; AI agents that can perform actions on your behalf. Essentially, hooking it up to -- giving it legs and legs and arms and asking it to do stuff autonomously.So I think that's super-interesting to me, because then you get to have it both ways. You get AI to expand bullet points into prose, and then to take prose into bullet points. And there's a very funny tweet from Josh Browder, who is the CEO DoNotPay, which is kind of like a --**Adam Stacoviak:** Yeah, I'm a fan of him.**Shawn Wang:** Yeah. Fantastic, right? So what DoNotPay does is they get rid of annoying payment UX, right? Like, sometimes it was parking tickets, but now they are trying to sort of broaden out into different things. So he recently tweeted that DoNotPay is working on a way to talk to Comcast to negotiate your cable bill down. And since Comcast themselves are going to have a chat bot as well, it's going to be chat bots talking to each other to resolve this... [laughter]**Adam Stacoviak:** Wow, man...**Jerod Santo:** It's like a scene out of Futurama, or something...**Shawn Wang:** Yeah. So I'm very excited about the summarization aspects, right? One of the more interesting projects that came out of this recent wave was Explained Paper, which is - you can throw any academic paper at it and it explains the paper to you in approachable language, and you can sort of query it back and forth. I think those are super-interesting, because that starts to reverse Brandolini is law. Instead of generating bulls**t, you're taking bulls**it in, getting into some kind of order. And that's very exciting.**Adam Stacoviak:** Yeah. 17 steps back, it makes me think about when I talk to my watch, and I say "Text my wife", and I think about like who is using this to their betterment? And I'm thinking like, we're only talking about adults, for the most part. My kid, my son, Eli - he talks to Siri as if like she knows everything, right? But here's me using my watch to say "Text my wife." I say it, it puts it into the phone... And the last thing it does for me, which I think is super-interesting for the future, as like this AI assistant, is "Send it" is the final prompt back to me as the human; should I send this? And if I say no, Siri doesn't send it. But if I say "Send it", guess what she does? She sends it. But I love this idea of the future, like maybe some sort of smarter AI assistant like that. I mean, to me, that's a dream. I'd love that.**Shawn Wang:** [42:21] Yeah, I was watching this clip of the first Iron Man, when Robert Downey Jr. is kind of working with his bot to work on his first suit... And he's just talking to the bot, like "Here's what I want you to do." Sometimes it gets it wrong and he slaps it on the ahead... But more often than not, he gets it right. And this is why I've been -- you know, Wes Boss recently tweeted -- this is actually really scary. "Should we be afraid as engineers, like this is going to come for our jobs?" And I'm like, "No. All of us just got a personal junior developer." That should excite you.**Jerod Santo:** Yeah. And it seems like it's particularly good at software development answers. You'd think it's because there's lots of available text... I mean, think about like things that it's good at; it seems like it knows a lot about programming.**Shawn Wang:** I have a list. Do you want a list?**Jerod Santo:** Yeah.**Shawn Wang:** So writing tutorials - it's very good. Literally, tables of contents, section by section, explaining "First you should npm install. Then you should do X. Then you should do Y." Debugging code - just paste in your error, and paste in the source code, and it tells you what's wrong with it. Dynamic programming, it does really well. Translating DSLs. I think there'll be a thousand DSLs blooming, because the barrier to adoption of a DSL has just disappeared. [laughs] So why would you not write a DSL? No one needs to learn your DSL.**Adam Stacoviak:** What is this, Copilot you're using, or ChatGPT, that you're--**Shawn Wang:** ChatGPT-3. I have a bunch of examples here I can drop in the show notes. AWS IAM policies. "Hey, I want to do X and Y in AWS." Guess what? There's tons of documentation. ChatGPT knows AWS IAM policies. Code that combines multiple cloud services. This one comes from Corey Quinn. 90% of our jobs is hooking up one service to another service. You could just tell it what to do, and it just does it, right? There a guy who was like, "I fed my college computer network's homework to it, and they gave the right result", which is pretty interesting.Refactoring code from Elixir to PHP is another one that has been has been done... And obviously, Advent of Code, which - we're recording this in December now. The person who won -- so Advent of Code for the first 100 people is a race; whoever submits the correct answer first, wins it. And the number one place in the first Advent of Code this year was a ChatGPT guy. So it broke homework. Like, this thing has broken homework and take-home interviews, basically. [laughs]**Jerod Santo:** Completely. It's so nice though; like, I've only used it a little bit while coding, but it's two for two, of just like drilling my exact questions. And just stuff like "How do you match any character that is not an [unintelligible 00:44:43.28] regular expression?"**Shawn Wang:** Oh, yeah. Explaining regexes.**Jerod Santo:** Yeah. That was my question. Like, I know exactly what I want, but I can't remember which is the character, and so I just asked it, and it gave me the exact correct answer, and an example, and explained it in more detail, if I wanted to go ahead and read it. And it warned me, "Hey, this is not the best way to test against email addresses... But here it is." So I was like, "Alright..." This is a good thing for developers, for sure.**Shawn Wang:** Yeah. But you can't trust it -- so you have a responsibility as well. You can't write bad code, have something bad happen, and go, "Oh, it wasn't my fault. It was ChatGPT."**Jerod Santo:** Well, you can't paste Stack Overflow answers into your code either.**Shawn Wang:** You have the responsibility. Exactly.**Jerod Santo:** Yeah. I mean, you can, but you're gonna get fired, right? Like, if the buck stops at you, not at the Stack Overflow answer person, you can't go find them and be like, "Why were you wrong?" Right? It stops at you.**Shawn Wang:** Yeah. So I think the way I phrased it was -- do you know about this trade offer meme that is going around? So it's "Trade offer - you receive better debugging, code explanation, install instructions, better documentation, elimination of your breaking of flow from copy and pasting in Stack Overflow - you receive all these benefits, in exchange for more code review." There is a cost, which is code review. You have to review the code that your junior programmer just gave you. But hey, that's better and easier than writing code yourself.**Jerod Santo:** [46:04] Yeah, because you've got a free junior programmer working for you now. [laughter]**Shawn Wang:** There's a guy that says, "I haven't done a single Google search or consulted any external documentation for the past few days, and I was able to progress faster than I ever had when delivering a new thing." I mean, it's just... It's amazing, and Google should be worried.**Jerod Santo:** Yeah, that's what I was gonna say - is this an immediate threat to Google? Now, I did see a commenter on Hacker News - Swyx, I'm not sure if you saw this one - from inside of Google, talking about the cost of integration?**Shawn Wang:** Yes. Yeah, I've read basically every thread... [laughter] Which is a full-time job, but... This is so important. Like, I don't do this for most things, right? Like, I think this is big enough that I had to drop everything and go read up on it... And not be an overnight expert, but at least try to be informed... And that's all I'm doing here, really. But yeah, do you want to read it up?**Jerod Santo:** Yeah. So in summary, they were responding... This is on a thread about ChatGPT, and they say -- this is a Googler, and they say "It's one thing to put up a demo that interested nerds can play with, but it's quite another thing to try to integrate it deeply in a system that serves billions of requests a day, when you take into account serving costs, added latency, and the fact that average revenue on something like a Google search is close to infinitesimal (which is the word I can't say out loud) already. I think I remember the presenter saying something like they'd want to reduce the cost by at least 10 times before it could be feasible to integrate models like this in products like Google search. A 10x or even 100x improvement is obviously an attainable target in the next few years, so I think technology like this is coming in the next few years."So that's one insider's take on where Google stands. Obviously, Google has tons of resources dedicated to these areas of expertise, right? It's not like Google's asleep at the wheel, and is going to completely have their lunch eaten by OpenAI. But right now, there's a lot of people who are training new habits, right? They're like, "I'm not gonna use Google anymore. I'm gonna start using OpenAI." I think it's something on the order of one million users in their first few days have signed up... How long can Google potentially bleed people before it becomes an actual problem? I don't know. I don't know the answer to these things.**Shawn Wang:** So there's one way in which you can evaluate for yourself right now, and I think that's the most helpful, constructive piece of advice that we can give on this podcast, which is -- we're covering something that is moving very live, very fast. Everything that we say could be invalidated tomorrow by something new. But you could just run ChatGPT-3 alongside of all your Google searches. That's a very, very simple way to evaluate if this would replace Google for you; just run it twice, every single time. And so there's a Google extension - and I'll link it - [unintelligible 00:48:47.04] ChatGPT Google extension; I'll put it in the show notes. And yeah, I have it running; it's not that great. [laughs] Surprisingly. So ChatGPT is optimized for answering questions. Sometimes I don't put questions in there. I just put the thing I'm looking for, and Google's pretty good at that, it turns out... [laughs]**Jerod Santo:** Right. See, because you are an expert-level Google prompt engineer, right? Like, you know how to talk to Google.**Shawn Wang:** We have optimized to Google prompting, yes.**Jerod Santo:** Exactly.**Shawn Wang:** If I need to search within a certain date range, I know how to do that in Google. I can't do that in ChatGPT-3. If I need to look for PDFs, I know how to do that. If I want to look for Reddit, and constrain the site to Reddit, I know how to do that. ChatGPT-3 has no concept of attribution, no concept of date ranges, and stuff like that.**Jerod Santo:** Right.**Shawn Wang:** But yeah, it is just like better at some things, and worse at other things, and that is the nature of all new technology. It just has to be better at one thing, that you cannot get anywhere else, and it has a permanent hold in your mind. Whenever you need that thing done, you will turn to ChatGPT-3, or any other new technology.[49:53] I love this sort of meta philosophy about technology adoption, because all new toys just generally are worse than the things that they replace, except in one area, and that's the area needs to matter. And if it does matter, it will win, because they will fix the bugs.**Jerod Santo:** Yeah, oftentimes with disruption, that area is cost; like acquisition cost. Sometimes it's convenience, and maybe I guess sometimes it's accuracy. There's different metrics, but it's got to be the one that matters. If it's marginally better at things that don't matter, you're not going to disrupt. But if it's a lot better at one thing that matters a lot, even if everything else sucks, you'll use that thing.**Shawn Wang:** Yeah, exactly. So it's interesting, because -- you know, Google has a few things going for it. By the way, it has one of the largest training repositories of text that no one else has, which is Gmail. But the most impressive thing it's being able to ship with Gmail is the little autocomplete, like, "Looks good", Okay", the little buttons that you see in the smart replies.**Jerod Santo:** Do you guys ever use those? Do you ever click on those?**Shawn Wang:** I use that. I use that. Save some typing.**Adam Stacoviak:** Yeah, well, I used to actually use Gmail directly to compose my emails, or respond. I would tap to complete all the time, if the response was like, "Yeah, I was gonna say that."**Shawn Wang:** There's a billion little ways that AI is built into Google right now, that we just take for granted, because we don't feel it, because there's no prompts. [laughter]**Jerod Santo:** We need a prompt!**Adam Stacoviak:** Even if OpenAI did eat Google's lunch, Google would just acquire it, or something...**Shawn Wang:** You would think so...**Jerod Santo:** Maybe...**Shawn Wang:** But I would say that probably OpenAI is not for sale. Like, they have this world-conquering ambition that would just not let them settle for anything less than global domination... Which is a little bit scary, right?**Jerod Santo:** Yeah, I think they're probably going the distance, is their plan, it seems like...**Shawn Wang:** Well, if anything, Microsoft should have bought them when they had the chance, because that was Bing's opportunity, and I don't think that ever came to pass... Probably because Sam Altman was smart enough not to do that deal. But yeah, so let's take that line of thinking to its logical conclusion. What would you feel if Google started autocompleting your entire email for you, and not just like individual, like two or three words? You would feel different, you would feel creeped out. So Google doesn't have the permission to innovate.**Adam Stacoviak:** I wouldn't freak out if I opted in, though. If I was like, "This technology exists, and it's helpful. I'll use that." Now, if it just suddenly started doing it, yeah, creeped out. But if I'm like, "Yeah, this is kind of cool. I opt into this enhanced AI, or this enhanced autocompletion", or whatever, simplifies the usage of it, or whatever.**Shawn Wang:** Yeah, so there's actually some people working on the email client that does that for you. So Evan Conrad is working on EveryPrompt email, which is essentially you type a bunch of things that you want to say, and you sort of batch answer all your emails with custom generated responses from GPT-3. It's a really smart application of this tech to email that I've seen. But I just think, like, you would opt in; the vast majority of people never opt into anything.**Jerod Santo:** Yeah, most people don't opt in.**Shawn Wang:** Like, that's just not the default experience. So I'm just saying, one reason that Google doesn't do it is "Yeah, we're just too big." Right? That is essentially the response that you read out from that engineer; like, "This doesn't work at Google scale. We can't afford it. It would be too slow", whatever. That's kind of a cop out, I feel like... Because Google should be capable. These are the best engineers in the world, they should they should be able to do it.**Jerod Santo:** Well, he does say he thinks it's coming in the next few years. So he's not saying it's impossible, he's saying they're not there yet. And I will say, I'm giving ChatGPT the benefit of my wait time that I do not afford to Google. I do not wait for Google to respond. I will give ChatGPT three to five seconds, because I know it's like a new thing that everyone's hitting hard... But like, if they just plugged that in, it would be too slow. I wouldn't wait three to five seconds for a Google search.**Shawn Wang:** Yeah. By the way, that's a fascinating cloud story that you guys have got to have on - find the engineer at OpenAI that scaled ChatGPT-3 in one week from zero to one million users?**Jerod Santo:** Yeah, totally.**Adam Stacoviak:** [53:58] Well, if you're listening, or you know the person, this is an open invite; we'd love to have that conversation.**Shawn Wang:** Yeah. I've seen the profile of the guy that claimed to [unintelligible 00:54:04.00] so that he would know... But I don't know who would be responsible for that. That is one of the most interesting cloud stories probably of the year. And Azure should be all over this. Azure should be going like, "Look, they handled it no problem. This is most successful consumer product of all time come at us", right?**Jerod Santo:** That's true. They should.**Shawn Wang:** They're the number three cloud right now. This is like their one thing, this is their time to shine. They've got to do it.**Jerod Santo:** And does anybody even know that Azure is behind OpenAI? I'm sure you can find out, but like, is that well known? I didn't know that.**Shawn Wang:** Oh, it's very public. Microsoft invested a billion dollars in OpenAI.**Jerod Santo:** Okay. Did you know that, Adam?**Adam Stacoviak:** No.**Jerod Santo:** So I'm trying to gauge the public knowledge...**Shawn Wang:** What we didn't know was that it was at a valuation of $20 billion, which... So OpenAI went from like this kind of weird research lab type thing into one of the most highly valued startups in the world. [laughs]**Jerod Santo:** Do you think Microsoft got their money's worth?**Shawn Wang:** I think so... It's awash right now, because --**Jerod Santo:** Too early.**Shawn Wang:** ...they probably cut them a lot of favorable deals for training, and stuff... So it's more about like being associated with one of the top AI names. Like, this is the play that Microsoft has been doing for a long time, so it's finally paying off... So I'm actually pretty happy for that. But then they have to convert into like getting people who are not [unintelligible 00:55:21.00] onto this thing.**Break:** [55:26]**Adam Stacoviak:** What's the long-term play here though? I mean, if Microsoft invested that kind of money, and we're using ChatGPT right now, we're willing to give it extra seconds, potentially even a minute if the answer is that important to you, that you wouldn't afford to Google... Like, what's the play for them? Will they turn this into a product? How do you make billions from this? Do you eventually just get absorbed by the FAANGs of the world, and next thing you know now this incredible future asset to humanity is now owned by essentially folks we try to like host our own services for? Like, we're hosting Nextcloud locally, so we can get off the Google Drives and whatnot... And all this sort of anti-whatever. I mean, what's the endgame here?**Shawn Wang:** Am I supposed to answer that? [laughs]**Adam Stacoviak:** Do you have an answer? I mean, that's what I think about...**Jerod Santo:** Let's ask ChatGPT what the endgame is... No, I mean, short-term it doesn't seem like OpenAI becomes the API layer for every AI startup that's gonna start in the next 5 or 10 years, right? Like, aren't they just charging their fees to everybody who wants to integrate AI into their products, pretty much? That's not an end game, but that's a short-term business model, right?**Shawn Wang:** That is a short-term business model, yeah. I bet they have much more up their sleeves... I don't actually know. But they did just hire their first developer advocate, which is interesting, because I think you'll start to hear a lot more from them.[58:12] Well, there's two things I will offer for you. One, it's a very common view or perception that AI is a centralizing force, right? Which is, Adam, what you're talking about, which is, "Does this just mean that the big always get bigger?" Because the big have the scale and size and data advantage. And one of the more interesting blog posts - sorry, I can't remember who I read this from - was that actually one of the lessons from this year is that it's not necessarily true, because AI might be a more decentralized force, because it's more amenable to open source... And crypto, instead of being decentralized, turned out to be more centralized than people thought.So the two directions of centralized versus decentralized - the common perception is that AI is very centralized, and crypto very decentralized. The reality was that it's actually the opposite, which is fascinating to me as a thesis. Like, is that the end game, that AI eventually gets more decentralized, because people want this so badly that there are enough researchers who go to NeurIPS to present their research papers and tweet out all this stuff, that diffuses these techniques all over the place? And we're seeing that happen, helped in large probably by Stability AI. The proof that Stability as an independent, outsider company, like not a ton of connections in the AI field, did this humongous achievement I think is just a remarkable encouragement that anyone could do it... And that's a really encouraging thing for those people who are not FAANG and trying to make some extra headroom in this world. So that's one way to think about the future.The second way to think about who monetizes and who makes the billion dollars on this... There's a very influential post that I was introduced to recently from Union Square Ventures, called "The myth of the infrastructure phase", which is directly tackling this concept that everyone says "When you have a gold rush, sell picks and shovels", right? And it's a very common thing, and presumably AI being the gold rush right now, you should sell picks and shovels, which is you should build AI infrastructure companies. But really, there are tons of AI infrastructure companies right now, they're a dime a dozen; really, they're all looking for use cases, and basically, the argument, the myth of the infrastructure phase is that technology swings back and forth between app constraint and infra constraint. And right now, we're not infrastructure-constrained, we're app-constrained. And really, it's the builders of AI-enabled products like TikTok that know what to do with the AI infrastructure tha