Podcasts about Elasticsearch

Turbopuffer came out of a reading app.In 2022, Simon was helping his friends at Readwise scale their infra for a highly requested feature: article recommendations and semantic search. Readwise was paying ~$5k/month for their relational database and vector search would cost ~$20k/month making the feature too expensive to ship. In 2023 after mulling over the problem from Readwise, Simon decided he wanted to “build a search engine” which became Turbopuffer.We discuss:• Simon's path: Denmark → Shopify infra for nearly a decade → “angel engineering” across startups like Readwise, Replicate, and Causal → turbopuffer almost accidentally becoming a company • The Readwise origin story: building an early recommendation engine right after the ChatGPT moment, seeing it work, then realizing it would cost ~$30k/month for a company spending ~$5k/month total on infra and getting obsessed with fixing that cost structure • Why turbopuffer is “a search engine for unstructured data”: Simon's belief that models can learn to reason, but can't compress the world's knowledge into a few terabytes of weights, so they need to connect to systems that hold truth in full fidelity • The three ingredients for building a great database company: a new workload, a new storage architecture, and the ability to eventually support every query plan customers will want on their data • The architecture bet behind turbopuffer: going all in on object storage and NVMe, avoiding a traditional consensus layer, and building around the cloud primitives that only became possible in the last few years • Why Simon hated operating Elasticsearch at Shopify: years of painful on-call experience shaped his obsession with simplicity, performance, and eliminating state spread across multiple systems • The Cursor story: launching turbopuffer as a scrappy side project, getting an email from Cursor the next day, flying out after a 4am call, and helping cut Cursor's costs by 95% while fixing their per-user economics • The Notion story: buying dark fiber, tuning TCP windows, and eating cross-cloud costs because Simon refused to compromise on architecture just to close a deal faster • Why AI changes the build-vs-buy equation: it's less about whether a company can build search infra internally, and more about whether they have time especially if an external team can feel like an extension of their own • Why RAG isn't dead: coding companies still rely heavily on search, and Simon sees hybrid retrieval semantic, text, regex, SQL-style patterns becoming more important, not less • How agentic workloads are changing search: the old pattern was one retrieval call up front; the new pattern is one agent firing many parallel queries at once, turning search into a highly concurrent tool call • Why turbopuffer is reducing query pricing: agentic systems are dramatically increasing query volume, and Simon expects retrieval infra to adapt to huge bursts of concurrent search rather than a small number of carefully chosen calls • The philosophy of “playing with open cards”: Simon's habit of being radically honest with investors, including telling Lachy Groom he'd return the money if turbopuffer didn't hit PMF by year-end • The “P99 engineer”: Simon's framework for building a talent-dense company, rejecting by default unless someone on the team feels strongly enough to fight for the candidate —Simon Hørup Eskildsen• LinkedIn: https://www.linkedin.com/in/sirupsen• X: https://x.com/Sirupsen• https://sirupsen.com/aboutturbopuffer• https://turbopuffer.com/Full Video PodTimestamps00:00:00 The PMF promise to Lachy Groom00:00:25 Intro and Simon's background00:02:19 What turbopuffer actually is00:06:26 Shopify, Elasticsearch, and the pain behind the company00:10:07 The Readwise experiment that sparked turbopuffer00:12:00 The insight Simon couldn't stop thinking about00:17:00 S3 consistency, NVMe, and the architecture bet00:20:12 The Notion story: latency, dark fiber, and conviction00:25:03 Build vs. buy in the age of AI00:26:00 The Cursor story: early launch to breakout customer00:29:00 Why code search still matters00:32:00 Search in the age of agents00:34:22 Pricing turbopuffer in the AI era00:38:17 Why Simon chose Lachy Groom00:41:28 Becoming a founder on purpose00:44:00 The “P99 engineer” philosophy00:49:30 Bending software to your will00:51:13 The future of turbopuffer00:57:05 Simon's tea obsession00:59:03 Tea kits, X Live, and P99 LiveTranscriptSimon Hørup Eskildsen: I don't think I've said this publicly before, but I just called Lockey and was like, local Lockie. Like if this doesn't have PMF by the end of the year, like we'll just like return all the money to you. But it's just like, I don't really, we, Justine and I don't wanna work on this unless it's really working.So we want to give it the best shot this year and like we're really gonna go for it. We're gonna hire a bunch of people. We're just gonna be honest with everyone. Like when I don't know how to play a game, I just play with open cards. Lockey was the only person that didn't, that didn't freak out. He was like, I've never heard anyone say that before.Alessio: Hey everyone, welcome to the Leading Space podcast. This is Celesio Pando, Colonel Laz, and I'm joined by Swix, editor of Leading Space.swyx: Hello. Hello, uh, we're still, uh, recording in the Ker studio for the first time. Very excited. And today we are joined by Simon Eski. Of Turbo Farer welcome.Simon Hørup Eskildsen: Thank you so much for having me.swyx: Turbo Farer has like really gone on a huge tear, and I, I do have to mention that like you're one of, you're not my newest member of the Danish AHU Mafia, where like there's a lot of legendary programmers that have come out of it, like, uh, beyond Trotro, Rasmus, lado Berg and the V eight team and, and Google Maps team.Uh, you're mostly a Canadian now, but isn't that interesting? There's so many, so much like strong Danish presence.Simon Hørup Eskildsen: Yeah, I was writing a post, um, not that long ago about sort of the influences. So I grew up in Denmark, right? I left, I left when, when I was 18 to go to Canada to, to work at Shopify. Um, and so I, like, I've, I would still say that I feel more Danish than, than Canadian.This is also the weird accent. I can't say th because it, this is like, I don't, you know, my wife is also Canadian, um, and I think. I think like one of the things in, in Denmark is just like, there's just such a ruthless pragmatism and there's also a big focus on just aesthetics. Like, they're like very, people really care about like where, what things look like.Um, and like Canada has a lot of attributes, US has, has a lot of attributes, but I think there's been lots of the great things to carry. I don't know what's in the water in Ahu though. Um, and I don't know that I could be considered part of the Mafi mafia quite yet, uh, compared to the phenomenal individuals we just mentioned.Barra OV is also, uh, Danish Canadian. Okay. Yeah. I don't know where he lives now, but, and he's the PHP.swyx: Yeah. And obviously Toby German, but moved to Canada as well. Yes. Like this is like import that, uh, that, that is an interesting, um, talent move.Alessio: I think. I would love to get from you. Definition of Turbo puffer, because I think you could be a Vector db, which is maybe a bad word now in some circles, you could be a search engine.It's like, let, let's just start there and then we'll maybe run through the history of how you got to this point.Simon Hørup Eskildsen: For sure. Yeah. So Turbo Puffer is at this point in time, a search engine, right? We do full text search and we do vector search, and that's really what we're specialized in. If you're trying to do much more than that, like then this might not be the right place yet, but Turbo Buffer is all about search.The other way that I think about it is that we can take all of the world's knowledge, all of the exabytes and exabytes of data that there is, and we can use those tokens to train a model, but we can't compress all of that into a few terabytes of weights, right? Compress into a few terabytes of weights, how to reason with the world, how to make sense of the knowledge.But we have to somehow connect it to something externally that actually holds that like in full fidelity and truth. Um, and that's the thing that we intend to become. Right? That's like a very holier than now kind of phrasing, right? But being the search engine for unstructured, unstructured data is the focus of turbo puffer at this point in time.Alessio: And let's break down. So people might say, well, didn't Elasticsearch already do this? And then some other people might say, is this search on my data, is this like closer to rag than to like a xr, like a public search thing? Like how, how do you segment like the different types of search?Simon Hørup Eskildsen: The way that I generally think about this is like, there's a lot of database companies and I think if you wanna build a really big database company, sort of, you need a couple of ingredients to be in the air.We don't, which only happens roughly every 15 years. You need a new workload. You basically need the ambition that every single company on earth is gonna have data in your database. Multiple times you look at a company like Oracle, right? You will, like, I don't think you can find a company on earth with a digital presence that it not, doesn't somehow have some data in an Oracle database.Right? And I think at this point, that's also true for Snowflake and Databricks, right? 15 years later it's, or even more than that, there's not a company on earth that doesn't, in. Or directly is consuming Snowflake or, or Databricks or any of the big analytics databases. Um, and I think we're in that kind of moment now, right?I don't think you're gonna find a company over the next few years that doesn't directly or indirectly, um, have all their data available for, for search and connect it to ai. So you need that new workload, like you need something to be happening where there's a new workload that causes that to happen, and that new workload is connecting very large amounts of data to ai.The second thing you need. The second condition to build a big database company is that you need some new underlying change in the storage architecture that is not possible from the databases that have come before you. If you look at Snowflake and Databricks, right, commoditized, like massive fleet of HDDs, like that was not possible in it.It just wasn't in the air in the nineties, right? So you just didn't, we just didn't build these systems. S3 and and and so on was not around. And I think the architecture that is now possible that wasn't possible 15 years ago is to go all in on NVME SSDs. It requires a particular type of architecture for the database that.It's difficult to retrofit onto the databases that are already there, including the ones you just mentioned. The second thing is to go all in on OIC storage, more so than we could have done 15 years ago. Like we don't have a consensus layer, we don't really have anything. In fact, you could turn off all the servers that Turbo Buffer has, and we would not lose any data because we have all completely all in on OIC storage.And this means that our architecture is just so simple. So that's the second condition, right? First being a new workload. That means that every company on earth, either indirectly or directly, is using your database. Second being, there's some new storage architecture. That means that the, the companies that have come before you can do what you're doing.I think the third thing you need to do to build a big database company is that over time you have to implement more or less every Cory plan on the data. What that means is that you. You can't just get stuck in, like, this is the one thing that a database does. It has to be ever evolving because when someone has data in the database, they over time expect to be able to ask it more or less every question.So you have to do that to get the storage architecture to the limit of what, what it's capable of. Those are the three conditions.swyx: I just wanted to get a little bit of like the motivation, right? Like, so you left Shopify, you're like principal, engineer, infra guy. Um, you also head of kernel labs, uh, inside of Shopify, right?And then you consulted for read wise and that it kind of gave you that, that idea. I just wanted you to tell that story. Um, maybe I, you've told it before, but, uh, just introduce the, the. People to like the, the new workload, the sort of aha moment for turbo PufferSimon Hørup Eskildsen: For sure. So yeah, I spent almost a decade at Shopify.I was on the infrastructure team, um, from the fairly, fairly early days around 2013. Um, at the time it felt like it was growing so quickly and everything, all the metrics were, you know, doubling year on year compared to the, what companies are contending with today. It's very cute in growth. I feel like lot some companies are seeing that month over month.Um, of course. Shopify compound has been compounding for a very long time now, but I spent a decade doing that and the majority of that was just make sure the site is up today and make sure it's up a year from now. And a lot of that was really just the, um, you know, uh, the Kardashians would drive very, very large amounts of, of data to, to uh, to Shopify as they were rotating through all the merch and building out their businesses.And we just needed to make sure we could handle that. Right. And sometimes these were events, a million requests per second. And so, you know, we, we had our own data centers back in the day and we were moving to the cloud and there was so much sharding work and all of that that we were doing. So I spent a decade just scaling databases ‘cause that's fundamentally what's the most difficult thing to scale about these sites.The database that was the most difficult for me to scale during that time, and that was the most aggravating to be on call for, was elastic search. It was very, very difficult to deal with. And I saw a lot of projects that were just being held back in their ambition by using it.swyx: And I mean, self-hosted.Self-hosted. ‘causeSimon Hørup Eskildsen: it's, yeah, and it commercial, this is like 2015, right? So it's like a very particular vintage. Right. It's probably better at a lot of these things now. Um, it was difficult to contend with and I'm just like, I just think about it. It's an inverted index. It should be good at these kinds of queries and do all of this.And it was, we, we often couldn't get it to do exactly what we needed to do or basically get lucine to do, like expose lucine raw to, to, to what we needed to do. Um, so that was like. Just something that we did on the side and just panic scaled when we needed to, but not a particular focus of mine. So I left, and when I left, I, um, wasn't sure exactly what I wanted to do.I mean, it spent like a decade inside of the same company. I'd like grown up there. I started working there when I was 18.swyx: You only do Rails?Simon Hørup Eskildsen: Yeah. I mean, yeah. Rails. And he's a Rails guy. Uh, love Rails. So good. Um,Alessio: we all wish we could still work in Rails.swyx: I know know. I know, but some, I tried learning Ruby.It's just too much, like too many options to do the same thing. It's, that's my, I I know there's a, there's a way to do it.Simon Hørup Eskildsen: I love it. I don't know that I would use it now, like given cloud code and, and, and cursor and everything, but, um, um, but still it, like if I'm just sitting down and writing a teal code, that's how I think.But anyway, I left and I wasn't, I talked to a couple companies and I was like, I don't. I need to see a little bit more of the world here to know what I'm gonna like focus on next. Um, and so what I decided is like I was gonna, I called it like angel engineering, where I just hopped around in my friend's companies in three months increments and just helped them out with something.Right. And, and just vested a bit of equity and solved some interesting infrastructure problem. So I worked with a bunch of companies at the time, um, read Wise was one of them. Replicate was one of them. Um, causal, I dunno if you've tried this, it's like a, it's a spreadsheet engine Yeah. Where you can do distribution.They sold recently. Yeah. Um, we've been, we used that in fp and a at, um, at Turbo Puffer. Um, so a bunch of companies like this and it was super fun. And so we're the Chachi bt moment happened, I was with. With read Wise for a stint, we were preparing for the reader launch, right? Which is where you, you cue articles and read them later.And I was just getting their Postgres up to snuff, like, which basically boils down to tuning, auto vacuum. So I was doing that and then this happened and we were like, oh, maybe we should build a little recommendation engine and some features to try to hook in the lms. They were not that good yet, but it was clear there was something there.And so I built a small recommendation engine just, okay, let's take the articles that you've recently read, right? Like embed all the articles and then do recommendations. It was good enough that when I ran it on one of the co-founders of Rey's, like I found out that I got articles about, about having a child.I'm like, oh my God, I didn't, I, I didn't know that, that they were having a child. I wasn't sure what to do with that information, but the recommendation engine was good enough that it was suggesting articles, um, about that. And so there was, there was recommendations and uh, it actually worked really well.But this was a company that was spending maybe five grand a month in total on all their infrastructure and. When I did the napkin math on running the embeddings of all the articles, putting them into a vector index, putting it in prod, it's gonna be like 30 grand a month. That just wasn't tenable. Right?Like Read Wise is a proudly bootstrapped company and it's paying 30 grand for infrastructure for one feature versus five. It just wasn't tenable. So sort of in the bucket of this is useful, it's pretty good, but let us, let's return to it when the costs come down.swyx: Did you say it grows by feature? So for five to 30 is by the number of, like, what's the, what's the Scaling factor scale?It scales by the number of articles that you embed.Simon Hørup Eskildsen: It does, but what I meant by that is like five grand for like all of the other, like the Heroku, dinos, Postgres, like all the other, and this then storage is 30. Yeah. And then like 30 grand for one feature. Right. Which is like, what other articles are related to this one.Um, so it was just too much right to, to power everything. Their budget would've been maybe a few thousand dollars, which still would've been a lot. And so we put it in a bucket of, okay, we're gonna do that later. We'll wait, we will wait for the cost to come down. And that haunted me. I couldn't stop thinking about it.I was like, okay, there's clearly some latent demand here. If the cost had been a 10th, we would've shipped it and. This was really the only data point that I had. Right. I didn't, I, I didn't, I didn't go out and talk to anyone else. It was just so I started reading Right. I couldn't, I couldn't help myself.Like I didn't know what like a vector index is. I, I generally barely do about how to generate the vectors. There was a lot of hype about, this is a early 2023. There was a lot of hype about vector databases. There were raising a lot of money and it's like, I really didn't know anything about it. It's like, you know, trying these little models, fine tuning them.Like I was just trying to get sort of a lay of the land. So I just sat down. I have this. A GitHub repository called Napkin Math. And on napkin math, there's just, um, rows of like, oh, this is how much bandwidth. Like this is how many, you know, you can do 25 gigabytes per second on average to dram. You can do, you know, five gigabytes per second of rights to an SSD, blah blah.All of these numbers, right? And S3, how many you could do per, how much bandwidth can you drive per connection? I was just sitting down, I was like, why hasn't anyone build a database where you just put everything on O storage and then you puff it into NVME when you use the data and you puff it into dram if you're, if you're querying it alive, it's just like, this seems fairly obvious and you, the only real downside to that is that if you go all in on o storage, every right will take a couple hundred milliseconds of latency, but from there it's really all upside, right?You do the first go, it takes half a second. And it sort of occurred to me as like, well. The architecture is really good for that. It's really good for AB storage, it's really good for nvm ESSD. It's, well, you just couldn't have done that 10 years ago. Back to what we were talking about before. You really have to build a database where you have as few round trips as possible, right?This is how CPUs work today. It's how NVM E SSDs work. It's how as, um, as three works that you want to have a very large amount of outstanding requests, right? Like basically go to S3, do like that thousand requests to ask for data in one round trip. Wait for that. Get that, like, make a new decision. Do it again, and try to do that maybe a maximum of three times.But no databases were designed that way within NVME as is ds. You can drive like within, you know, within a very low multiple of DRAM bandwidth if you use it that way. And same with S3, right? You can fully max out the network card, which generally is not maxed out. You get very, like, very, very good bandwidth.And, but no one had built a database like that. So I was like, okay, well can't you just, you know, take all the vectors right? And plot them in the proverbial coordinate system. Get the clusters, put a file on S3 called clusters, do json, and then put another file for every cluster, you know, cluster one, do js O cluster two, do js ON you know that like it's two round trips, right?So you get the clusters, you find the closest clusters, and then you download the cluster files like the, the closest end. And you could do this in two round trips.swyx: You were nearest neighbors locally.Simon Hørup Eskildsen: Yes. Yes. And then, and you would build this, this file, right? It's just like ultra simplistic, but it's not a far shot from what the first version of Turbo Buffer was.Why hasn't anyone done thatAlessio: in that moment? From a workload perspective, you're thinking this is gonna be like a read heavy thing because they're doing recommend. Like is the fact that like writes are so expensive now? Oh, with ai you're actually not writing that much.Simon Hørup Eskildsen: At that point I hadn't really thought too much about, well no actually it was always clear to me that there was gonna be a lot of rights because at Shopify, the search clusters were doing, you know, I don't know, tens or hundreds of crew QPS, right?‘cause you just have to have a human sit and type in. But we did, you know, I don't know how many updates there were per second. I'm sure it was in the millions, right into the cluster. So I always knew there was like a 10 to 100 ratio on the read write. In the read wise use case. It's, um, even, even in the read wise use case, there'd probably be a lot fewer reads than writes, right?There's just a lot of churn on the amount of stuff that was going through versus the amount of queries. Um, I wasn't thinking too much about that. I was mostly just thinking about what's the fundamentally cheapest way to build a database in the cloud today using the primitives that you have available.And this is it, right? You just, now you have one machine and you know, let's say you have a terabyte of data in S3, you paid the $200 a month for that, and then maybe five to 10% of that data and needs to be an NV ME SSDs and less than that in dram. Well. You're paying very, very little to inflate the data.swyx: By the way, when you say no one else has done that, uh, would you consider Neon, uh, to be on a similar path in terms of being sort of S3 first and, uh, separating the compute and storage?Simon Hørup Eskildsen: Yeah, I think what I meant with that is, uh, just build a completely new database. I don't know if we were the first, like it was very much, it was, I mean, I, I hadn't, I just looked at the napkin math and was like, this seems really obvious.So I'm sure like a hundred people came up with it at the same time. Like the light bulb and every invention ever. Right. It was just in the air. I think Neon Neon was, was first to it. And they're trying, they're retrofitted onto Postgres, right? And then they built this whole architecture where you have, you have it in memory and then you sort of.You know, m map back to S3. And I think that was very novel at the time to do it for, for all LTP, but I hadn't seen a database that was truly all in, right. Not retrofitting it. The database felt built purely for this no consensus layer. Even using compare and swap on optic storage to do consensus. I hadn't seen anyone go that all in.And I, I mean, there, there, I'm sure there was someone that did that before us. I don't know. I was just looking at the napkin mathswyx: and, and when you say consensus layer, uh, are you strongly relying on S3 Strong consistency? You are. Okay.SoSimon Hørup Eskildsen: that is your consensus layer. It, it is the consistency layer. And I think also, like, this is something that most people don't realize, but S3 only became consistent in December of 2020.swyx: I remember this coming out during COVID and like people were like, oh, like, it was like, uh, it was just like a free upgrade.Simon Hørup Eskildsen: Yeah.swyx: They were just, they just announced it. We saw consistency guys and like, okay, cool.Simon Hørup Eskildsen: And I'm sure that they just, they probably had it in prod for a while and they're just like, it's done right.And people were like, okay, cool. But. That's a big moment, right? Like nv, ME SSDs, were also not in the cloud until around 2017, right? So you just sort of had like 2017 nv, ME SSDs, and people were like, okay, cool. There's like one skew that does this, whatever, right? Takes a few years. And then the second thing is like S3 becomes consistent in 2020.So now it means you don't have to have this like big foundation DB or like zookeeper or whatever sitting there contending with the keys, which is how. You know, that's what Snowflake and others have do so muchswyx: for goneSimon Hørup Eskildsen: Exactly. Just gone. Right? And so just push to the, you know, whatever, how many hundreds of people they have working on S3 solved and then compare and swap was not in S3 at this point in time,swyx: by the way.Uh, I don't know what that is, so maybe you wanna explain. Yes. Yeah.Simon Hørup Eskildsen: Yes. So, um, what Compare and swap is, is basically, you can imagine that if you have a database, it might be really nice to have a file called metadata json. And metadata JSON could say things like, Hey, these keys are here and this file means that, and there's lots of metadata that you have to operate in the database, right?But that's the simplest way to do it. So now you have might, you might have a lot of servers that wanna change the metadata. They might have written a file and want the metadata to contain that file. But you have a hundred nodes that are trying to contend with this metadata that JSON well, what compare and Swap allows you to do is basically just you download the file, you make the modifications, and then you write it only if it hasn't changed.While you did the modification and if not you retry. Right? Should just have this retry loops. Now you can imagine if you have a hundred nodes doing that, it's gonna be really slow, but it will converge over time. That primitive was not available in S3. It wasn't available in S3 until late 2024, but it was available in GCP.The real story of this is certainly not that I sat down and like bake brained it. I was like, okay, we're gonna start on GCS S3 is gonna get it later. Like it was really not that we started, we got really lucky, like we started on GCP and we started on GCP because tur um, Shopify ran on GCP. And so that was the platform I was most available with.Right. Um, and I knew the Canadian team there ‘cause I'd worked with them at Shopify and so it was natural for us to start there. And so when we started building the database, we're like, oh yeah, we have to build a, we really thought we had to build a consensus layer, like have a zookeeper or something to do this.But then we discovered the compare and swap. It's like, oh, we can kick the can. Like we'll just do metadata r json and just, it's fine. It's probably fine. Um, and we just kept kicking the can until we had very, very strong conviction in the idea. Um, and then we kind of just hinged the company on the fact that S3 probably was gonna get this, it started getting really painful in like mid 2024.‘cause we were closing deals with, um, um, notion actually that was running in AWS and we're like, trust us. You, you really want us to run this in GCP? And they're like, no, I don't know about that. Like, we're running everything in AWS and the latency across the cloud were so big and we had so much conviction that we bought like, you know, dark fiber between the AWS regions in, in Oregon, like in the InterExchange and GCP is like, we've never seen a startup like do like, what's going on here?And we're just like, no, we don't wanna do this. We were tuning like TCP windows, like everything to get the latency down ‘cause we had so high conviction in not doing like a, a metadata layer on S3. So those were the three conditions, right? Compare and swap. To do metadata, which wasn't in S3 until late 2024 S3 being consistent, which didn't happen until December, 2020.Uh, 2020. And then NVMe ssd, which didn't end in the cloud until 2017.swyx: I mean, in some ways, like a very big like cloud success story that like you were able to like, uh, put this all together, but also doing things like doing, uh, bind our favor. That that actually is something I've never heard.Simon Hørup Eskildsen: I mean, it's very common when you're a big company, right?You're like connecting your own like data center or whatever. But it's like, it was uniquely just a pain with notion because the, um, the org, like most of the, like if you're buying in Ashburn, Virginia, right? Like US East, the Google, like the GCP and, and AWS data centers are like within a millisecond on, on each other, on the public exchanges.But in Oregon uniquely, the GCP data center sits like a couple hundred kilometers, like east of Portland and the AWS region sits in Portland, but the network exchange they go through is through Seattle. So it's like a full, like 14 milliseconds or something like that. And so anyway, yeah. It's, it's, so we were like, okay, we can't, we have to go through an exchange in Portland.Yeah. Andswyx: you'd rather do this than like run your zookeeper and likeSimon Hørup Eskildsen: Yes. Way rather. It doesn't have state, I don't want state and two systems. Um, and I think all that is just informed by Justine, my co-founder and I had just been on call for so long. And the worst outages are the ones where you have state in multiple places that's not syncing up.So it really came from, from a a, like just a, a very pure source of pain, of just imagining what we would be Okay. Being woken up at 3:00 AM about and having something in zookeeper was not one of them.swyx: You, you're talking to like a notion or something. Do they care or do they just, theySimon Hørup Eskildsen: just, they care about latency.swyx: They latency cost. That's it.Simon Hørup Eskildsen: They just cared about latency. Right. And we just absorbed the cost. We're just like, we have high conviction in this. At some point we can move them to AWS. Right. And so we just, we, we'll buy the fiber, it doesn't matter. Right. Um, and it's like $5,000. Usually when you buy fiber, you buy like multiple lines.And we're like, we can only afford one, but we will just test it that when it goes over the public internet, it's like super smooth. And so we did a lot of, anyway, it's, yeah, it was, that's cool.Alessio: You can imagine talking to the GCP rep and it's like, no, we're gonna buy, because we know we're gonna turn, we're gonna turn from you guys and go to AWS in like six months.But in the meantime we'll do this. It'sSimon Hørup Eskildsen: a, I mean, like they, you know, this workload still runs on GCP for what it's worth. Right? ‘cause it's so, it was just, it was so reliable. So it was never about moving off GCP, it was just about honesty. It was just about giving notion the latency that they deserved.Right. Um, and we didn't want ‘em to have to care about any of this. We also, they were like, oh, egress is gonna be bad. It was like, okay, screw it. Like we're just gonna like vvc, VPC peer with you and AWS we'll eat the cost. Yeah. Whatever needs to be done.Alessio: And what were the actual workloads? Because I think when you think about ai, it's like 14 milliseconds.It's like really doesn't really matter in the scheme of like a model generation.Simon Hørup Eskildsen: Yeah. We were told the latency, right. That we had to beat. Oh, right. So, so we're just looking at the traces. Right. And then sort of like hand draw, like, you know, kind of like looking at the trace and then thinking what are the other extensions of the trace?Right. And there's a lot more to it because it's also when you have, if you have 14 versus seven milliseconds, right. You can fit in another round trip. So we had to tune TCP to try to send as much data in every round trip, prewarm all the connections. And there was, there's a lot of things that compound from having these kinds of round trips, but in the grand scheme it was just like, well, we have to beat the latency of whatever we're up against.swyx: Which is like they, I mean, notion is a database company. They could have done this themselves. They, they do lots of database engineering themselves. How do you even get in the door? Like Yeah, just like talk through that kind of.Simon Hørup Eskildsen: Last time I was in San Francisco, I was talking to one of the engineers actually, who, who was one of our champions, um, at, AT Notion.And they were, they were just trying to make sure that the, you know, per user cost matched the economics that they needed. You know, Uhhuh like, it's like the way I think about, it's like I have to earn a return on whatever the clouds charge me and then my customers have to earn a return on that. And it's like very simple, right?And so there has to be gross margin all the way up and that's how you build the product. And so then our customers have to make the right set of trade off the turbo Puffer makes, and if they're happy with that, that's great.swyx: Do you feel like you're competing with build internally versus buy or buy versus buy?Simon Hørup Eskildsen: Yeah, so, sorry, this was all to build up to your question. So one of the notion engineers told me that they'd sat and probably on a napkin, like drawn out like, why hasn't anyone built this? And then they saw terrible. It was like, well, it literally that. So, and I think AI has also changed the buy versus build equation in terms of, it's not really about can we build it, it's about do we have time to build it?I think they like, I think they felt like, okay, if this is a team that can do that and they, they feel enough like an extension of our team, well then we can go a lot faster, which would be very, very good for them. And I mean, they put us through the, through the test, right? Like we had some very, very long nights to to, to do that POC.And they were really our biggest, our second big customer off the cursor, which also was a lot of late nights. Right.swyx: Yeah. That, I mean, should we go into that story? The, the, the sort of Chris's story, like a lot, um, they credit you a lot for. Working very closely with them. So I just wanna hear, I've heard this, uh, story from Sole's point of view, but like, I'm curious what, what it looks like from your side.Simon Hørup Eskildsen: I actually haven't heard it from Sole's point of view, so maybe you can now cross reference it. The way that I remember it was that, um, the day after we launched, which was just, you know, I'd worked the whole summer on, on the first version. Justine wasn't part of it yet. ‘cause I just, I didn't tell anyone that summer that I was working on this.I was just locked in on building it because it's very easy otherwise to confuse talking about something to actually doing it. And so I was just like, I'm not gonna do that. I'm just gonna do the thing. I launched it and at this point turbo puffer is like a rust binary running on a single eight core machine in a T Marks instance.And me deploying it was like looking at the request log and then like command seeing it or like control seeing it to just like, okay, there's no request. Let's upgrade the binary. Like it was like literally the, the, the, the scrappiest thing. You could imagine it was on purpose because just like at Shopify, we did that all the time.Like, we like move, like we ran things in tux all the time to begin with. Before something had like, at least the inkling of PMF, it was like, okay, is anyone gonna hear about this? Um, and one of the cursor co-founders Arvid reached out and he just, you know, the, the cursor team are like all I-O-I-I-M-O like, um, contenders, right?So they just speak in bullet points and, and facts. It was like this amazing email exchange just of, this is how many QPS we have, this is what we're paying, this is where we're going, blah, blah, blah. And so we're just conversing in bullet points. And I tried to get a call with them a few times, but they were, so, they were like really writing the PMF bowl here, just like late 2023.And one time Swally emails me at like five. What was it like 4:00 AM Pacific time saying like, Hey, are you open for a call now? And I'm on the East coast and I, it was like 7:00 AM I was like, yeah, great, sure, whatever. Um, and we just started talking and something. Then I didn't know anything about sales.It was something that just comp compelled me. I have to go see this team. Like, there's something here. So I, I went to San Francisco and I went to their office and the way that I remember it is that Postgres was down when I showed up at the office. Did SW tell you this? No. Okay. So Postgres was down and so it's like they were distracting with that.And I was trying my best to see if I could, if I could help in any way. Like I knew a little bit about databases back to tuning, auto vacuum. It was like, I think you have to tune out a vacuum. Um, and so we, we talked about that and then, um, that evening just talked about like what would it look like, what would it look like to work with us?And I just said. Look like we're all in, like we will just do what we'll do whatever, whatever you tell us, right? They migrated everything over the next like week or two, and we reduced their cost by 95%, which I think like kind of fixed their per user economics. Um, and it solved a lot of other things. And we were just, Justine, this is also when I asked Justine to come on as my co-founder, she was the best engineer, um, that I ever worked with at Shopify.She lived two blocks away and we were just, okay, we're just gonna get this done. Um, and we did, and so we helped them migrate and we just worked like hell over the next like month or two to make sure that we were never an issue. And that was, that was the cursor story. Yeah.swyx: And, and is code a different workload than normal text?I, I don't know. Is is it just text? Is it the same thing?Simon Hørup Eskildsen: Yeah, so cursor's workload is basically, they, um, they will embed the entire code base, right? So they, they will like chunk it up in whatever they would, they do. They have their own embedding model, um, which they've been public about. Um, and they find that on, on, on their evals.It. There's one of their evals where it's like a 25% improvement on a very particular workload. They have a bunch of blog posts about it. Um, I think it works best on larger code basis, but they've trained their own embedding model to do this. Um, and so you'll see it if you use the cursor agent, it will do searches.And they've also been public around, um, how they've, I think they post trained their model to be very good at semantic search as well. Um, and that's, that's how they use it. And so it's very good at, like, can you find me on the code that's similar to this, or code that does this? And just in, in this queries, they also use GR to supplement it.swyx: Yeah.Simon Hørup Eskildsen: Um, of courseswyx: it's been a big topic of discussion like, is rag dead because gr you know,Simon Hørup Eskildsen: and I mean like, I just, we, we see lots of demand from the coding company to ethicsswyx: search in every part. Yes.Simon Hørup Eskildsen: Uh, we, we, we see demand. And so, I mean, I'm. I like case studies. I don't like, like just doing like thought pieces on this is where it's going.And like trying to be all macroeconomic about ai, that's has turned out to be a giant waste of time because no one can really predict any of this. So I just collect case studies and I mean, cursor has done a great job talking about what they're doing and I hope some of the other coding labs that use Turbo Puffer will do the same.Um, but it does seem to make a difference for particular queries. Um, I mean we can also do text, we can also do RegX, but I should also say that cursors like security posture into Tur Puffer is exceptional, right? They have their own embedding model, which makes it very difficult to reverse engineer. They obfuscate the file paths.They like you. It's very difficult to learn anything about a code base by looking at it. And the other thing they do too is that for their customers, they encrypt it with their encryption keys in turbo puffer's bucket. Um, so it's, it's, it's really, really well designed.swyx: And so this is like extra stuff they did to work with you because you are not part of Cursor.Exactly like, and this is just best practice when working in any database, not just you guys. Okay. Yeah, that makes sense. Yeah. I think for me, like the, the, the learning is kind of like you, like all workloads are hybrid. Like, you know, uh, like you, you want the semantic, you want the text, you want the RegX, you want sql.I dunno. Um, but like, it's silly to like be all in on like one particularly query pattern.Simon Hørup Eskildsen: I think, like I really like the way that, um, um, that swally at cursor talks about it, which is, um, I'm gonna butcher it here. Um, and you know, I'm a, I'm a database scalability person. I'm not a, I, I dunno anything about training models other than, um, what the internet tells me and what.The way he describes is that this is just like cash compute, right? It's like you have a point in time where you're looking at some particular context and focused on some chunk and you say, this is the layer of the neural net at this point in time. That seems fundamentally really useful to do cash compute like that.And, um, how the value of that will change over time. I'm, I'm not sure, but there seems to be a lot of value in that.Alessio: Maybe talk a bit about the evolution of the workload, because even like search, like maybe two years ago it was like one search at the start of like an LLM query to build the context. Now you have a gentech search, however you wanna call it, where like the model is both writing and changing the code and it's searching it again later.Yeah. What are maybe some of the new types of workloads or like changes you've had to make to your architecture for it?Simon Hørup Eskildsen: I think you're right. When I think of rag, I think of, Hey, there's an 8,000 token, uh, context window and you better make it count. Um, and search was a way to do that now. Everything is moving towards the, just let the agent do its thing.Right? And so back to the thing before, right? The LLM is very good at reasoning with the data, and so we're just the tool call, right? And that's increasingly what we see our customers doing. Um, what we're seeing more demand from, from our customers now is to do a lot of concurrency, right? Like Notion does a ridiculous amount of queries in every round trip just because they can't.And I'm also now, when I use the cursor agent, I also see them doing more concurrency than I've ever seen before. So a bit similar to how we designed a database to drive as much concurrency in every round trip as possible. That's also what the agents are doing. So that's new. It means just an enormous amount of queries all at once to the dataset while it's warm in as few turns as possible.swyx: Can I clarify one thing on that?Simon Hørup Eskildsen: Yes.swyx: Is it, are they batching multiple users or one user is driving multiple,Simon Hørup Eskildsen: one user driving multiple, one agent driving.swyx: It's parallel searching a bunch of things.Simon Hørup Eskildsen: Exactly.swyx: Yeah. Yeah, exactly. So yeah, the clinician also did, did this for the fast context thing, like eight parallel at once.Simon Hørup Eskildsen: Yes.swyx: And, and like an interesting problem is, well, how do you make sure you have enough diversity so you're not making the the same request eight times?Simon Hørup Eskildsen: And I think like that's probably also where the hybrid comes in, where. That's another way to diversify. It's a completely different way to, to do the search.That's a big change, right? So before it was really just like one call and then, you know, the LLM took however many seconds to return, but now we just see an enormous amount of queries. So the, um, we just see more queries. So we've like tried to reduce query, we've reduced query pricing. Um, this is probably the first time actually I'm saying that, but the query pricing is being reduced, like five x.Um, and we'll probably try to reduce it even more to accommodate some of these workloads of just doing very large amounts of queries. Um, that's one thing that's changed. I think the right, the right ratio is still very high, right? Like there's still a, an enormous amount of rights per read, but we're starting probably to see that change if people really lean into this pattern.Alessio: Can we talk a little bit about the pricing? I'm curious, uh, because traditionally a database would charge on storage, but now you have the token generation that is so expensive, where like the actual. Value of like a good search query is like much higher because they're like saving inference time down the line.How do you structure that as like, what are people receptive to on the other side too?Simon Hørup Eskildsen: Yeah. I, the, the turbo puffer pricing in the beginning was just very simple. The pricing on these on for search engines before Turbo Puffer was very server full, right? It was like, here's the vm, here's the per hour cost, right?Great. And I just sat down with like a piece of paper and said like, if Turbo Puffer was like really good, this is probably what it would cost with a little bit of margin. And that was the first pricing of Turbo Puffer. And I just like sat down and I was like, okay, like this is like probably the storage amp, but whenever on a piece of paper I, it was vibe pricing.It was very vibe price, and I got it wrong. Oh. Um, well I didn't get it wrong, but like Turbo Puffer wasn't at the first principle pricing, right? So when Cursor came on Turbo Puffer, it was like. Like, I didn't know any VCs. I didn't know, like I was just like, I don't know, I didn't know anything about raising money or anything like that.I just saw that my GCP bill was, was high, was a lot higher than the cursor bill. So Justine and I was just like, well, we have to optimize it. Um, and I mean, to the chagrin now of, of it, of, of the VCs, it now means that we're profitable because we've had so much pricing pressure in the beginning. Because it was running on my credit card and Justine and I had spent like, like tens of thousands of dollars on like compute bills and like spinning off the company and like very like, like bad Canadian lawyers and like things like to like get all of this done because we just like, we didn't know.Right. If you're like steeped in San Francisco, you're just like, you just know. Okay. Like you go out, raise a pre-seed round. I, I never heard a word pre-seed at this point in time.swyx: When you had Cursor, you had Notion you, you had no funding.Simon Hørup Eskildsen: Um, with Cursor we had no funding. Yeah. Um, by the time we had Notion Locke was, Locke was here.Yeah. So it was really just, we vibe priced it 100% from first Principles, but it wasn't, it, it was not performing at first principles, so we just did everything we could to optimize it in the beginning for that, so that at least we could have like a 5% margin or something. So I wasn't freaking out because Cursor's bill was also going like this as they were growing.And so my liability and my credit limit was like actively like calling my bank. It was like, I need a bigger credit. Like it was, yeah. Anyway, that was the beginning. Yeah. But the pricing was, yeah, like storage rights and query. Right. And the, the pricing we have today is basically just that pricing with duct tape and spit to try to approach like, you know, like a, as a margin on the physical underlying hardware.And we're doing this year, you're gonna see more and more pricing changes from us. Yeah.swyx: And like is how much does stuff like VVC peering matter because you're working in AWS land where egress is charged and all that, you know.Simon Hørup Eskildsen: We probably don't like, we have like an enterprise plan that just has like a base fee because we haven't had time to figure out SKU pricing for all of this.Um, but I mean, yeah, you can run turbo puffer either in SaaS, right? That's what Cursor does. You can run it in a single tenant cluster. So it's just you. That's what Notion does. And then you can run it in, in, in BYOC where everything is inside the customer's VPC, that's what an for example, philanthropic does.swyx: What I'm hearing is that this is probably the best CRO job for somebody who can come in and,Simon Hørup Eskildsen: I mean,swyx: help you with this.Simon Hørup Eskildsen: Um, like Turbo Puffer hired, like, I don't know what, what number this was, but we had a full-time CFO as like the 12th hire or something at Turbo Puffer, um, I think I hear are a lot of comp.I don't know how they do it. Like they have a hundred employees and not a CFO. It's like having a CFO is like a runningswyx: business man. Like, you know,Simon Hørup Eskildsen: it's so good. Yeah, like money Mike, like he just, you know, just handles the money and a lot of the business stuff and so he came in and just hopped with a lot of the operational side of the business.So like C-O-O-C-F-O, like somewhere in between.swyx: Just as quick mention of Lucky, just ‘cause I'm curious, I've met Lock and like, he's obviously a very good investor and now on physical intelligence, um, I call it generalist super angel, right? He invests in everything. Um, and I always wonder like, you know, is there something appealing about focusing on developer tooling, focusing on databases, going like, I've invested for 10 years in databases versus being like a lock where he can maybe like connect you to all the customers that you need.Simon Hørup Eskildsen: This is an excellent question. No, no one's asked me this. Um, why lockey? Because. There was a couple of people that we were talking to at the time and when we were raising, we were almost a little, we were like a bit distressed because one of our, one of our peers had just launched something that was very similar to Turbo Puffer.And someone just gave me the advice at the time of just choose the person where you just feel like you can just pick up the phone and not prepare anything. And just be completely honest, and I don't think I've said this publicly before, but I just called Lockey and was like local Lockie. Like if this doesn't have PMF by the end of the year, like we'll just like return all the money to you.But it's just like, I don't really, we, Justine and I don't wanna work on this unless it's really working. So we want to give it the best shot this year and like we're really gonna go for it. We're gonna hire a bunch of people and we're just gonna be honest with everyone. Like when I don't know how to play a game, I just play with open cards and.Lockey was the only person that didn't, that didn't freak out. He was like, I've never heard anyone say that before. As I said, I didn't even know what a seed or pre-seed round was like before, probably even at this time. So I was just like very honest with him. And I asked him like, Lockie, have you ever have, have you ever invested in database company?He was just like, no. And at the time I was like, am I dumb? Like, but I think there was something that just like really drew me to Lockie. He is so authentic, so honest, like, and there was something just like, I just felt like I could just play like, just say everything openly. And that was, that was, I think that that was like a perfect match at the time, and, and, and honestly still is.He was just like, okay, that's great. This is like the most honest, ridiculous thing I've ever heard anyone say to me. But like that, like that, whyswyx: is this ridiculous? Say competitor launch, this may not work out. It wasSimon Hørup Eskildsen: more just like. If this doesn't work out, I'm gonna close up shop by the end of the mo the year, right?Like it was, I don't know, maybe it's common. I, I don't know. He told me it was uncommon. I don't know. Um, that's why we chose him and he'd been phenomenal. The other people were talking at the, at the time were database experts. Like they, you know, knew a lot about databases and Locke didn't, this turned out to be a phenomenal asset.Right. I like Justine and I know a lot about databases. The people that we hire know a lot about databases. What we needed was just someone who didn't know a lot about databases, didn't pretend to know a lot about databases, and just wanted to help us with candidates and customers. And he did. Yeah. And I have a list, right, of the investors that I have a relationship with, and Lockey has just performed excellent in the number of sub bullets of what we can attribute back to him.Just absolutely incredible. And when people talk about like no ego and just the best thing for the founder, I like, I don't think that anyone, like even my lawyer is like, yeah, Lockey is like the most friendly person you will find.swyx: Okay. This is my most glow recommendation I've ever heard.Alessio: He deserves it.He's very special.swyx: Yeah. Yeah. Yeah. Okay. Amazing.Alessio: Since you mentioned candidates, maybe we can talk about team building, you know, like, especially in sf, it feels like it's just easier to start a company than to join a company. Uh, I'm curious your experience, especially not being n SF full-time and doing something that is maybe, you know, a very low level of detail and technical detail.Simon Hørup Eskildsen: Yeah. So joining versus starting, I never thought that I would be a founder. I would start with it, like Turbo Puffer started as a blog post, and then it became a project and then sort of almost accidentally became a company. And now it feels like it's, it's like becoming a bigger company. That was never the intention.The intentions were very pure. It's just like, why hasn't anyone done this? And it's like, I wanna be the, like, I wanna be the first person to do it. I think some founders have this, like, I could never work for anyone else. I, I really don't feel that way. Like, it's just like, I wanna see this happen. And I wanna see it happen with some people that I really enjoy working with and I wanna have fun doing it and this, this, this has all felt very natural on that, on that sense.So it was never a like join versus versus versus found. It was just dis found me at the right moment.Alessio: Well I think there's an argument for, you should have joined Cursor, right? So I'm curious like how you evaluate it. Okay, I should actually go raise money and make this a company versus like, this is like a company that is like growing like crazy.It's like an interesting technical problem. I should just build it within Cursor and then they don't have to encrypt all this stuff. They don't have to obfuscate things. Like was that on your mind at all orSimon Hørup Eskildsen: before taking the, the small check from Lockie, I did have like a hard like look at myself in the mirror of like, okay, do I really want to do this?And because if I take the money, I really have to do it right. And so the way I almost think about it's like you kind of need to ha like you kind of need to be like fucked up enough to want to go all the way. And that was the conversation where I was like, okay, this is gonna be part of my life's journey to build this company and do it in the best way that I possibly can't.Because if I ask people to join me, ask people to get on the cap table, then I have an ultimate responsibility to give it everything. And I don't, I think some people, it doesn't occur to me that everyone takes it that seriously. And maybe I take it too seriously, I don't know. But that was like a very intentional moment.And so then it was very clear like, okay, I'm gonna do this and I'm gonna give it everything.Alessio: A lot of people don't take it this seriously. But,swyx: uh, let's talk about, you have this concept of the P 99 engineer. Uh, people are 10 x saying, everyone's saying, you know, uh, maybe engineers are out of a job. I don't know.But you definitely see a P 99 engineer, and I just want you to talk about it.Simon Hørup Eskildsen: Yeah, so the P 99 engineer was just a term that we started using internally to talk about candidates and talk about how we wanted to build the company. And you know, like everyone else is, like we want a talent dense company.And I think that's almost become trite at this point. What I credit the cursor founders a lot with is that they just arrived there from first principles of like, we just need a talent dense, um, talent dense team. And I think I've seen some teams that weren't talent dense and like seemed a counterfactual run, which if you've run in been in a large company, you will just see that like it's just logically will happen at a large company.Um, and so that was super important to me and Justine and it's very difficult to maintain. And so we just needed, we needed wording for it. And so I have a document called Traits of the P 99 Engineer, and it's a bullet point list. And I look at that list after every single interview that I do, and in every single recap that we do and every recap we end with.End with, um, some version of I'm gonna reject this candidate completely regardless of what the discourse was, because I wanna see people fight for this person because the default should not be, we're gonna hire this person. The default should be, we're definitely not hiring this person. And you know, if everyone was like, ah, maybe throw a punch, then this is not the right.swyx: Do, do you operate, like if there's one cha there must have at least one champion who's like, yes, I will put my career on, on, on the line for this. You know,Simon Hørup Eskildsen: I think career on the line,swyx: maybe a chair, butSimon Hørup Eskildsen: yeah. You know, like, um, I would say so someone needs to like, have both fists up and be like, I'd fight.Right? Yeah. Yeah. And if one person said, then, okay, let's do it. Right?swyx: Yeah.Simon Hørup Eskildsen: Um. It doesn't have to be absolutely everyone. Right? And like the interviews are always the sign that you're checking for different attributes. And if someone is like knocking it outta the park in every single attribute, that's, that's fairly rare.Um, but that's really important. And so the traits of the P 99 engineer, there's lots of them. There's also the traits of the p like triple nine engineer and the quadruple nine engineer. This is like, it's a long list.swyx: Okay.Simon Hørup Eskildsen: Um, I'll give you some samples, right. Of what we, what we look for. I think that the P 99 engineer has some history of having bent, like their trajectory or something to their will.Right? Some moment where it was just, they just, you know, made the computer do what it needed to do. There's something like that, and it will, it will occur to have them at some point in their career. And, uh. Hopefully multiple times. Right.swyx: Gimme an example of one of your engineers that like,Simon Hørup Eskildsen: I'll give an eng.Uh, so we, we, we launched this thing called A and NV three. Um, we could, we're also, we're working on V four and V five right now, but a and NV three can search a hundred billion vectors with a P 50 of around 40 milliseconds and a p 99 of 200 milliseconds. Um, maybe other people have done this, I'm sure Google and others have done this, but, uh, we haven't seen anyone, um, at least not in like a public consumable SaaS that can do this.And that was an engineer, the chief architect of Turbo Puffer, Nathan, um, who more or less just bent this, the software was not capable of this and he just made it capable for a very particular workload in like a, you know, six to eight week period with the help of a lot of the team. Right. It's been, been, there's numerous of examples of that, like at, at turbo puff, but that's like really bending the software and X 86 to your will.It was incredible to watch. Um. You wanna see some moments like that?swyx: Isn't that triple nine?Simon Hørup Eskildsen: Um, I think Nathan, what's calledAlessio: group nine, that was only nine. I feel like this is too high forSimon Hørup Eskildsen: Nathan. Nathan is, uh, Nathan is like, yeah, there's a lot of nines. Okay. After that p So I think that's one trait. I think another trait is that, uh, the P 99 spends a lot of time looking at maps.Generally it's their preferred ux. They just love looking at maps. You ever seen someone who just like, sits on their phone and just like, scrolls around on a map? Or did you not look at maps A lot? You guys don't look atswyx: maps? I guess I'm not feeling there. I don't know, butSimon Hørup Eskildsen: you just dis What about trains?Do you like trains?swyx: Uh, I mean they, not enough. Okay. This is just like weapon nice. Autism is what I call it. Like, like,Simon Hørup Eskildsen: um, I love looking at maps, like, it's like my preferred UX and just like I, you know, I likeswyx: lotsAlessio: of, of like random places, soswyx: like,youswyx: know.Alessio: Yes. Okay. There you go. So instead of like random places, like how do you explore the maps?Simon Hørup Eskildsen: No, it's, it's just a joke.swyx: It's autism laugh. It's like you are just obsessed by something and you like studying a thing.Simon Hørup Eskildsen: The origin of this was that at some point I read an interview with some IOI gold medalistswyx: Uhhuh,Simon Hørup Eskildsen: and it's like, what do you do in your spare time? I was just like, I like looking at maps.I was like, I feel so seen. Like, I just like love, like swirling out. I was like, oh, Canada is so big. Where's Baffin Island? I don't know. I love it. Yeah. Um, anyway, so the traits of P 99, P 99 is obsessive, right? Like, there's just like, you'll, you'll find traits of that we do an interview at, at, at, at turbo puffer or like multiple interviews that just try to screen for some of these things.Um, so. There's lots of others, but these are the kinds of traits that we look for.swyx: I'll tell you, uh, some people listen for like some of my dere stuff. Uh, I do think about derel as maps. Um, you draw a map for people, uh, maps show you the, uh, what is commonly agreed to be the geographical features of what a boundary is.And it shows also shows you what is not doing. And I, I think a lot of like developer tools, companies try to tell you they can do everything, but like, let's, let's be real. Like you, your, your three landmarks are here, everyone comes here, then here, then here, and you draw a map and, and then you draw a journey through the map.And like that. To me, that's what developer relations looks like. So I do think about things that way.Simon Hørup Eskildsen: I think the P 99 thinks in offs, right? The P 99 is very clear about, you know, hey, turbo puffer, you can't run a high transaction workload on turbo puffer, right? It's like the right latency is a hundred milliseconds.That's a clear trade off. I think the P 99 is very good at articulating the trade offs in every decision. Um. Which is exactly what the map is in your case, right?swyx: Uh, yeah, yeah. My, my, my world. My world.Alessio: How, how do you reconcile some of these things when you're saying you bend the will the computer versus like the trade

Leah Sutton, Chief Talent Portfolio Officer at Balderton Capital, has had an incredible career journey — from her early work in impact projects in South Africa to moving into tech, and eventually being headhunted for her current role at one of Europe's leading venture firms.At Balderton, Leah partners with some of Europe's most successful teams — think Revolut, Cleo, and Lyst — helping shape the talent strategies behind these powerhouse startups.I was fascinated to hear Leah's story, especially her time scaling at Elastic Search and the lessons she's carried forward. Her energy completely fills a room, and this conversation is packed with insights on growth, leadership, and building exceptional teams.This episode is sponsored by ⁠Treatwell⁠, Europe's No.1 hair and beauty booking app. Treatwell makes it easy to discover trusted salons, compare reviews, and book everything from a last‑minute blow dry to a full self‑care reset in just a few taps, so you can feel your best without the faff.We're also partnered with ⁠RISER⁠, it's like a dating app - but for YOUR career. Get ready to make a move by uploading your 60 second elevator pitch (Your Video CV) and connect with real hirers who have uploaded their video too! You'll get recommendations based on compatibility not just simply keywords. Join 1000's of RISERs on the app landing opportunties with companies like Beauty Pie and Tiktok.

Radar processes billions of location events daily, powering geofencing and location APIs for companies like Uber, Lyft, and thousands of other apps. When their existing infrastructure started hitting performance and cost limits, they built HorizonDB, a specialized database which replaced both Elasticsearch and MongoDB with a custom single binary written in Rust and backed by RocksDB.In this episode, we dive deep into the technical journey from prototype to production. We talk about RocksDB internals, finite-state transducers, the intricacies of geospatial indexing with Hilbert curves, and why Rust's type system and performance characteristics made it the perfect choice for rewriting critical infrastructure that processes location data at massive scale.

Welcome back to the Ultimate Guide to Partnering® Podcast. AI agents are your next customers. Subscribe to our Newsletter: https://theultimatepartner.com/ebook-subscribe/ Check Out UPX:https://theultimatepartner.com/experience/ In this exclusive interview, Vince Menzione sits down with Darryl Peek, Vice President for Partner Sales (Public Sector) at Elastic, to decode how Elastic achieved the rare “triple crown”—winning Partner of the Year across Microsoft, Amazon, and Google Cloud simultaneously. Darryl breaks down the engineering-first approach that makes Elastic sticky with hyperscalers, reveals the rigorous metrics behind their partner health scorecard, and shares his personal “one-page strategy” for aligning mission, vision, and execution. From leveraging generative AI for cleaner sales hygiene to the timeless lesson of the “Acre of Diamonds,” this conversation offers a masterclass in building high-performance partner ecosystems in the public sector and beyond. https://youtu.be/__GE0r2fPuk Key Takeaways Elastic achieved “Pinnacle” status by aligning engineering roadmaps directly with hyperscaler innovations to become essential infrastructure. Successful public sector sales require a dual approach: leveraging resellers for contract access while driving domain-specific co-sell motions. Partner relationships outperform contracts; consistency in communication is more valuable than only showing up for renewals. Effective partner organizations track “influence” revenue just as rigorously as direct bookings to capture the full value of SI relationships. Generative AI can automate sales hygiene, turning scattered meeting notes into actionable CRM data and reducing friction for sales teams. The “Acre of Diamonds” philosophy reminds leaders that the greatest opportunities often lie within their current ecosystem, not in distant new markets. If you're ready to lead through change, elevate your business, and achieve extraordinary outcomes through the power of partnership—this is your community. At Ultimate Partner® we want leaders like you to join us in the Ultimate Partner Experience – where transformation begins. Keywords: Elastic, Darryl Peek, public sector sales, hyperscaler partnership, Microsoft Partner of the Year, AWS Partner of the Year, Google Cloud Partner, partner ecosystem strategy, co-sell motion, partner metrics, channel sales, government contracting, Carahsoft, generative AI in sales, sales hygiene, Russell Conwell, Acre of Diamonds, open source search, observability, security SIM, vector search, retrieval augmented generation, LLM agnostic, partner enablement, influence revenue, channel booking, SI relationships, strategic alliances. Transcript: Darryl Peek Audio Episode [00:00:00] Darryl Peek: I say, I tell my team from time to time, the difference between contacts and contracts is the R and that’s the relationship. So if you’re not building the relationship, then how do you expect that partner to want to lean in? Don’t just show up when you have a contract. Don’t just show up when you have a renewal. [00:00:13] Darryl Peek: Make sure that you are reaching out and letting them know what is happening. Don’t just talk to me when you need a renewal, right? When you’re at end of quarter and you want me to bring a deal forward, [00:00:23] Vince Menzione: welcome to the Ultimate Guide to Partnering. I’m Vince Menzi. Own your host, and my mission is to help leaders like you achieve your greatest results through successful partnering. [00:00:34] Vince Menzione: We just came off Ultimate Partner live at Caresoft Training Center in Reston, Virginia. Over two days, we gathered top leaders to tackle the real shifts shaping our industry. If you weren’t in the room, this episode brings you right to the edge of what’s next. Let’s dive in. So we have another privilege, an incredible partner, another like we call these, if you’ve heard our term, pinnacle. [00:01:00] Vince Menzione: I think it’s a term that’s not widely used, but we refer to Pinnacle as the partners that have achieved the top rung. They’ve become partners of the year. And our next presenter, our next interview is going to be with an organization. And a person that represents an organization that has been a pinnacle partner actually for all three Hyperscalers, which is really unusual. [00:01:24] Vince Menzione: Elastic has been partner of the Year award winner across Microsoft, Amazon, and Google Cloud, so very interesting. And Darrell Peak, who is the leader for the public sector organization, he’s here in the Washington DC area, was kind enough. Elastic is a sponsor event, and Darryl’s been kind enough to join me for a discussion about what it takes to be a Pinnacle partner. [00:01:47] Vince Menzione: So incredibly well. Excited to welcome you, Darryl. Thank you, sir. Good to have you. I love you. I love your smile, man. You got an incredible smile. Thank you. Thank you, Vince. Thank you. So Darryl, I probably didn’t do it any justice, but I was hoping you could take us through your role and responsibilities at Elastic, which is an incredible organization. [00:02:08] Vince Menzione: Alright. Yeah, [00:02:09] Darryl Peek: absolutely. So Darrell Peak vice President for partner sales for the US public sector at Elastic. I’ve been there about two and a half years. Responsible for our partner relationships across all partner types, whether that’s the system integrators, resellers, MSPs, OEMs, distribution Hyperscalers, and our Technology Alliance partners. [00:02:26] Darryl Peek: And those are partners that aren’t built on the Elastic platform. In regards to how my partner team interacts with our team. Our ecosystem. We are essentially looking to further and lean in with our partners in order for them to, one, understand what Elastic does since we’re such a diverse tool, but also work with our field to understand what are their priorities and how do they identify the right partners for the right requirements. [00:02:50] Darryl Peek: In regards to what Elastic is and what it does elastic is a solution that is actually founded on search and we’re an open source company. And one of the things that I actually did when I left the government, so I worked for the government for a number of years. I left, went and worked for Salesforce, then worked for Google ran their federal partner team and then came over to Elastic because I wanted to. [00:03:11] Darryl Peek: Understand what it meant to be at an open source company. Being at an open source company is quite interesting ’cause you’re competing against yourself. [00:03:17] Vince Menzione: Yeah, that’s true. [00:03:18] Darryl Peek: So it’s pretty interesting. But elastic was founded in 2012 as a search company. So when you talk about search, we are the second most used platform behind Google. [00:03:28] Darryl Peek: So many of you have already used Elastic. Maybe on your way here, if you use Uber and Lyft, that is elastic. That is helping you get here. Oh, that is interesting. If you use Netflix, if you use wikipedia.com, booking.com, eBay, home Depot, all of those are search capabilities. That Elastic is happening to power in regards to what else we do. [00:03:47] Darryl Peek: We also do observability, which is really around application monitoring, logging, tracing, and metrics. So we are helping your operations team. Pepsi is a customer as well as Cisco. Wow. And then the last thing that we do is security when we’re a SIM solution. So when we talk about sim, we are really looking to protect networks. [00:04:03] Darryl Peek: So we all, we think that it’s a data problem. So with that data problem, what we’re trying to do is not only understand what is happening in the network, but also we are helping with threat intelligence, endpoint and cloud security. So all those elements together is what Elastic does. And we only do it two ways. [00:04:18] Darryl Peek: We’re one platform and we can be deployed OnPrem and in the cloud. So that’s a little bit about me and the company. Hopefully it was clear, [00:04:24] Vince Menzione: I’ve had elastic people on stage. You’ve done, that’s the best answer I’ve had. What does Elastic do? I used to hear all this hyperbole and what? [00:04:32] Vince Menzione: What? Now I really understand what you do is an organiz. And the name of the company was Elasticsearch. [00:04:36] Darryl Peek: It was [00:04:37] Vince Menzione: elastic at one time when I first. Worked with you. It was Elasticsearch. [00:04:40] Darryl Peek: Absolutely. Yeah. So many moons ago used to be called the Elk Stack and it stood for three things. E was the Elasticsearch which is a search capability. [00:04:48] Darryl Peek: L is Logstash, which is our logging capability. And Cabana is essentially our visualization capability. So it was called Elk. But since we’ve acquired so many companies and built so much capability into the platform, we can now call it the elastic. Platform. [00:05:00] Vince Menzione: So talk to me about your engagement with the hyperscalers. [00:05:02] Vince Menzione: You’ve been partner of the Year award winner with all three, right? I mentioned that, and you were, you worked for Google for a period of time. Yes. So tell us about, like, how does that work? What does that engagement look like? And why do you get chosen as partner of the year? What are the things that stand out when you’re working with these hyperscalers [00:05:19] Darryl Peek: and with that we are very fortunate to be recognized. [00:05:23] Darryl Peek: So many of the organizations that are out there are doing some of the same capabilities that we do, but they can’t claim that they won a part of the year for all three hyperscalers in the same year. We are able to do that because we believe in the power of partnership, not only from a technology perspective, but also from a sales perspective. [00:05:39] Darryl Peek: So we definitely lean in with our partnerships, so having our engineers talk, having our product teams talk, and making sure that we’re building capabilities that actually integrate within the cloud service providers. And also consistently building a roadmap that aligns with the innovation that the cloud service providers are also building towards. [00:05:56] Darryl Peek: And then making sure that we’re a topic of discussion. So elastic. From a search capability, we do semantic search, vector search, but also retrieval augmented generation, which actually is LLM Agnostic. So when you say LLM Agnostic, whether you want to use Gemini, Claude or even Chad, GBT, those things are something that Elastic can integrate in, but it actually helps reduce the likelihood of hallucination. [00:06:18] Darryl Peek: So when we’re building that kind of solution, the cloud service provider’s you’re making it easy for us, and when you make it easy, you become very attractive and therefore you’re. Likely gonna come. So it becomes [00:06:28] Vince Menzione: sticky in that regard. Very sticky. So it sounds like very much an engineer, a lot of emphasis on the engineering aspects of the business. [00:06:35] Vince Menzione: I know you’re an engineer by background too, right? So the engineering aspects of the business means that you’re having alignment with the engineering organizations of those companies at a very deep level. [00:06:44] Darryl Peek: Absolutely. So I’m [00:06:45] Vince Menzione: here. [00:06:45] Darryl Peek: Yeah. And being at Elastic has been pretty amazing. So coming from Google, we had so many different solutions, so many different SKUs, but Elastic releases every eight weeks. [00:06:54] Darryl Peek: So right before you start to understand the last release, the next release is coming out and we’re already at 9.2 and we just released 9.0 in May. So it’s really blazing fast on the capability that we’re really pushing the market, but it’s really hard to make sure that we get it in front of our partners. [00:07:10] Darryl Peek: So when we talk about our partner enablement strategy, we’re just trying to make sure that we get the right information in front of the right partners at the right time, so this way they can best service their customers. [00:07:19] Vince Menzione: So let’s talk about partner strategy. Alyssa Fitzpatrick was on stage with me at our last event, and she Alyssa’s fantastic. [00:07:25] Vince Menzione: She is incredible. Yes, she is. She was a former colleague at Microsoft Days. Yes. And then she, we had a really interesting conversation. About what it takes, like being in, in a company and then working with the partners in general. And you have, I’m sure you have a lot of the similarities in how you have to engage with these organizations. [00:07:42] Vince Menzione: You’re working across the hyperscalers, you’re also working with the ecosystem too. Yes. ’cause the delivery, you have delivery partners as well. Absolutely. So tell us more about that. [00:07:50] Darryl Peek: So we kinda look at it from a two, two ways from the pre-sales motion and then the post-sales. From the pre-sales side. [00:07:56] Darryl Peek: What we’re trying to do is really maximize our, not only working with partners, because within public sector, you need to get access to customers through contract vehicles. So if you want to get access to some, for instance, the VA or through GSA or others, you have to make sure you’re aligned with the right partners who have access to. [00:08:12] Darryl Peek: That particular agency, but also you want domain expertise. So as you’re working with those system integrators, you wanna make sure that they have capability that aligns. So whether it is a security requirement, you wanna work with someone who specializes in security, observability and search. So that’s the way that we really look at our partner ecosystem, but those who are interested in working with us. [00:08:30] Darryl Peek: Because everybody doesn’t necessarily have a emphasis on working with a new technology partner, [00:08:36] Vince Menzione: right? [00:08:36] Darryl Peek: So what we’re trying to do is saying how do we build programs, incentives and sales plays that really does align and strike the interest of that particular partner? So when we talk about it I tell my team, you have to, my grandfather to say, plan your work and work your plan. And if you fail a plan, you plan to fail. So being able to not only have a strong plan in place, but then execute against that plan, check against that plan as you go through the fiscal year, and then see how you come out at the end of the fiscal year to see are we making that progress? [00:09:01] Darryl Peek: But on the other side of it, and what I get stressed about with my sales team and saying what does partners bring to us? So where are those partner deal registrations? What is the partner source numbers? How are we creating more pipeline? And that is where we’re now saying, okay, how can we navigate and how can we make it easier? [00:09:17] Darryl Peek: And how can we reduce friction in order for the partner to say, okay, elastic’s easy to work with. I can see value in, oh, by the way, I can make some money with. [00:09:25] Vince Menzione: So take us through, have there been examples of areas where you’ve had to like, break through to this other side in terms of growing the partner ecosystem? [00:09:33] Vince Menzione: What’s worked, what hasn’t worked? Yes, I’d love to learn more about that. [00:09:36] Darryl Peek: I’ll say that and I tell my team one, you partner program is essential, right? If you don’t have an attractive partner program in regards to how they come on board, how they’re incentivized the right amount of margin, they won’t even look at you. [00:09:49] Darryl Peek: The second thing is really how do you engage? So a lot of things start with relationships. I think partnerships are really about relationships. I say I tell my team from time to time, the difference between contacts and contracts is the R and that’s the relationship. So if you’re not building the relationship, then how do you expect that partner to want to lean in? [00:10:07] Darryl Peek: Don’t just show up when you have a contract. Don’t just show up when you have a renewal. Make sure that you are reaching out and letting them know what is happening. I like the what Matt brought up in saying, okay, talk to me when you have a win. Talk to me when you have something to talk about. [00:10:22] Darryl Peek: Don’t just talk to me when you need a renewal. When you’re at end the quarter and you want me to bring a deal forward, that doesn’t help ab absolutely. [00:10:28] Vince Menzione: So engineering organizations, sales organizations, what are, what does a healthy partnership look like for you? [00:10:35] Darryl Peek: So I look at metrics a lot and we use a number of tools and I know folks are using tools out there. [00:10:41] Darryl Peek: I won’t name any tools for branding purposes, but in regards to how we look at tools. So some things that we measure closely. Of course it’s our partner source numbers, so partner source, bookings, and pipeline. We look at our partner attached numbers and pipeline as well as the amount or percentage of partner attached business that we have in regards to our overall a CV number. [00:11:00] Darryl Peek: We also look at co-sell numbers, so therefore we are looking at not only how. A partner is coming to us, but how is a partner helping us in closing the deal even though they didn’t bring us the deal? We’re also looking at our cloud numbers and saying what amount of deals and how much business are we doing with our cloud service providers? [00:11:15] Darryl Peek: Because of course we wanna see that number go up year over year. We wanna actually help with that consumption number because not only are we looking at it from a SaaS perspective, but also if the customer has to commit we can help burn that down as well. We also look at influence numbers. [00:11:27] Darryl Peek: Now, one of the harder things to do within a technology business is. Capturing all that si goodness. And saying how do I reflect the SI if they’re not bringing me the deal? And I can’t attribute that amount of deal to that particular partner, right? And the way that we do that is we just tag them to the influence. [00:11:44] Darryl Peek: So we’re able to now track influence. And also the M-S-P-O-E-M work that we are also tracking and also we’re tracking the royalties. And lastly is the professional service work that we do with those partners. So we’re looking to go up into the right where we start them out at our select level, we go to our premier level and then our elite level. [00:12:00] Darryl Peek: But left and to the right, I say you gotta go from zero to one, one to five, five to 10, and then 10 to 25. So if we can actually see that progression. That is where we’re really starting to see health in the partnership, but also the executive alignment is really important. So when our CEO is able to meet with the fellow CEO of the co partner company that is really showing how we are progressing, but also our VPs and others that are engaged. [00:12:20] Darryl Peek: So those are things that we really do measure. We do have a health score card and also, we track accreditations, we track certifications as well as training outcomes based on our sales place. [00:12:30] Vince Menzione: Wow. There’s a lot of metrics there. Yeah. So you didn’t bring, you didn’t bring any slides with that out? [00:12:35] Darryl Peek: Oh, no. I’m not looking at slides, by the way. [00:12:40] Vince Menzione: Let’s talk about marketplace. [00:12:42] Darryl Peek: All right? [00:12:42] Vince Menzione: Because we’ve had a lot of conversations about marketplace. We’ve got both vendors up here talking about marketplace and the importance of marketplace, right? You’ve been a Marketplace Award winner. We haven’t really talked about that, like that motion per se. [00:12:55] Vince Menzione: I’d love to s I’d love to hear from you like how you, a, what you had to overcome to get to marketplace, what the marketplace motion looks like for your organization, what a marketplace first motion looks like. ’cause a lot of your cut a. Are all your customers requiring a lot of direct selling effort or is it some of it through Marketplace? [00:13:14] Vince Menzione: Like how does it, how does that work for you? [00:13:15] Darryl Peek: So Elastic is a global organization. Yeah. So we’re, 40 different countries. So it depends on where we’re talking. So if we talk about our international business, which is our A PJ and EMEA business we are seeing a lot more marketplace and we’re seeing that those direct deals with customers. [00:13:28] Darryl Peek: Okay. And we’re talking about our mirror business. A significant amount goes through marketplace and where our customers are transacting with the marketplace and are listing. On the marketplace within public sector, it’s more of a resell motion. Okay. So we are working with our resellers. [00:13:39] Darryl Peek: So we work our primary distribution partner is Carahsoft. So you heard from Craig earlier. Yes. We have a strong relationship with Carahsoft and definitely a big fan of this organization. But in regards to how we do that and how we track it we are looking at better ways to, track that orchestration and consumption numbers in order to see not only what customers we’re working with, but how can we really accelerate that motion and really get those leads and transactions going. [00:14:03] Vince Menzione: Very cool. Very cool. And I think part of the reason why in, in the government or public sector space it has a lot to do with the commitments are different. Absolutely. So it’s not government agencies aren’t able to make the same level of commitments that, private sector organizations were able to make, so they were able to the Mac or Microsoft parlance and also a AWS’s parlance. [00:14:23] Vince Menzione: Yeah, [00:14:24] Darryl Peek: definitely a different dynamic. Yeah. And especially within the public sector. ’cause we have Gov Cloud to work with, right? That’s right. So we’re working with Microsoft or we’re working with AWS, they have their Gov cloud and then we Google, they don’t have a Gov cloud, but we still have to work with them differently. [00:14:35] Darryl Peek: Yeah. Within that space. That’s [00:14:36] Vince Menzione: right. That’s right. So it makes the motion a little bit differently there. So I think we talked through some of this. I just wanna make sure we cover our points [00:14:43] Darryl Peek: here. One thing I’ll do an aside, you talked about the acre of diamonds. I’m a big fan of that story. [00:14:47] Vince Menzione: Yeah, let’s talk about Russ Con. Yeah, [00:14:49] Darryl Peek: let’s talk about it. Do you all know about the Acre Diamonds? Have you all heard that story before? No. You have some those in the audience. [00:14:55] Vince Menzione: I, you know what, let’s talk about it. All [00:14:56] Darryl Peek: See, I’m from Philadelphia. [00:14:57] Vince Menzione: I didn’t know you were a family. My daughter went to Temple University. [00:14:59] Vince Menzione: Ah, [00:15:00] Darryl Peek: okay. That’s all I know. So Russell Conwell. So he was, a gentleman out of the Philadelphia area and he went around town to raise money and he wanted to raise money because he believed that there was a promise within a specific area. And as he continued to raise this money, he would tell a story. [00:15:14] Darryl Peek: And basically it was a story about a farmer in Africa. And the farmer in Africa, to make it really short was essentially looking to be become very wealthy. And because he wanted to become very wealthy, he believed that selling his farm and going off to a long distant land was the primary way for him to find diamonds. [00:15:28] Darryl Peek: And this farmer didn’t sold us. Sold his place, then went off to to this foreign land, and he ended up dying. And people thought that was the end of the story, but there was another farmer who bought that land and one time this big, and they called him the ot, came to the door and said you mind if I have some tea with you? [00:15:43] Darryl Peek: He said, all right, come on in. Have a drink. And as he had the drink, he looked upon the mantle and his mouth dropped. And then the farmer said what’s wrong? What do you say? He says, do you know what that is? No. He said no. Do you know what that is? He says, no. He said, that’s the biggest diamond I’ve ever seen, and the farmer goes. [00:16:01] Darryl Peek: That’s weird because there’s a bunch right in the back where I go grab my fruits and crops every day. So the idea of the acre diamonds and sometimes that you don’t need to go off to a far off land. It is actually sometimes right under your feet, and that is a story that helped fund the starting of Temple University. [00:16:16] Vince Menzione: I’m gonna need to take you at every single event so you can tell this story again. That’s an awesome job. Oh, I love it. And yeah, they founded a Temple University. Yeah. Which has become an incredible university. My daughter, like I said, my daughter’s a graduate, so we’re Temple fan. That’s great story. [00:16:31] Vince Menzione: That is a very cool, I didn’t realize you were a Philadelphia guy too, so that is awesome. Go birds. Go birds. All right, good. So let’s talk, I think we talked a little bit about your ecosystem approach, but maybe just a little bit more on this, like you said, like a lot of data, a lot of metrics but also a lot of these organizations also have to under understand the engineering side of things. [00:16:53] Vince Menzione: Oh, yeah. There’s a tremendous amount to become. Not everybody could just show up one day and become an elastic partner [00:16:58] Darryl Peek: absolutely. Absolutely. So take us [00:16:59] Vince Menzione: through that process. [00:17:00] Darryl Peek: Yeah. So one of the things that we are trying to mature and we have matured is our partner go to market. [00:17:06] Darryl Peek: So in order to join our partner ecosystem, you have to sign ’em through our partner portal. You have to sign our indirect reseller agreement. ’cause we do sell primarily within the public sector through distribution. And we only go direct if it is by exception. So you have to get justification through myself as well as our VP for public sector. [00:17:21] Darryl Peek: But we really do try to make sure that we can aggregate this because one thing that we have to monitor is terms and conditions. ’cause of course, working with the government, there’s a lot of terms and conditions. So we try to alleviate that by having it go through caresoft, they’re able to absorb some, so this way we can actually transact with the government. [00:17:36] Darryl Peek: In regards to the team though we try to really work closely with our solution architecture team. So this way we can develop clear enablement strategies with our partners so this way they know what it is we do, but also how to properly bring us up in a conversation. Also handle objections and also what are we doing to implement our solutions within other markets. [00:17:55] Darryl Peek: So those are things that we are doing as well as partner marketing. Top of funnel activity is really important, so we’re trying to differentiate what we’re doing with the field and field marketing. So you’re doing the leads and m qls and things of that nature also with partner marketing. So our partner marketing actually is driven by leads, but also we’re trying to transact. [00:18:10] Darryl Peek: And get Ps of which our partner deal registration. So that is how we align our partner go to market. And that is actually translating into our partner source outcomes. [00:18:18] Vince Menzione: And I think we have a slide that talks a little bit about your public sector partner strategy. [00:18:23] Darryl Peek: Oh yeah. Oh, I share that. So I thought maybe we could spin it. [00:18:25] Darryl Peek: Absolutely. [00:18:25] Vince Menzione: I know you we can’t see it, but they can. Oh, they can. Okay. Great. [00:18:29] Darryl Peek: There it’s there. [00:18:30] Vince Menzione: It’s career. [00:18:31] Darryl Peek: One thing, I think this was Einstein has said, if you can’t explain it simply, you don’t understand it well enough. So that was the one thing. So I always was a big fan of creating a one page strategy. [00:18:39] Darryl Peek: And based on this one page strategy one of the things when I worked at Salesforce it was really about a couple things and the saying, okay, what are your bookings? And if you don’t have bookings, what does your pipeline look like? If you don’t have pipeline, what does your prospecting look like? [00:18:51] Darryl Peek: Yeah. If you don’t have prospecting what does your account plan look like? And if you don’t have an account plan, why are you here? Why are you here? Exactly. So those are the things that I really talk to my team about is just really a, it’s about bookings. It’s about pipeline. It’s about planning, enablement and execution. [00:19:05] Darryl Peek: It’s about marketing, branding and evangelism, and also about operational excellence and how to execute. Very cool. So being able to do that and also I, since I came from Salesforce, I talk to my team a lot about Salesforce hygiene. So we really talk about that a lot. So make, making sure we’re making proper use of chatter, but also as we talk about utilizing ai, we just try to. [00:19:21] Darryl Peek: How do we simplify that, right? So if we’re using Zoom or we’re using Google, how do we make sure that we’re capturing those meeting minutes, translating that, putting that into the system, so therefore we have a record of that engagement with that partner. So this is a continuous threat. So this way I don’t have to call my partner manager the entire time. [00:19:36] Darryl Peek: I can look back, see what actions, see what was discussed, and say, okay, how can we keep this conversation going? Because we shouldn’t have to have those conversations every time. I shouldn’t have to text you to say, give me the download on every partner. Every time. How do we automate that? And that’s really where you’re creating this context window with your Genive ai. [00:19:53] Darryl Peek: I think they said what 75% of organizations are using one AI tool. And I think 1% are mature in that. But also a number of organizations, it’s 90% of organizations are using generative AI tools to some degree. So we are using gen to bi. We do use a number of them. We have elastic GPT. Nice little brand there. [00:20:11] Darryl Peek: But yeah, we use that for not only understanding what’s in our our repositories and data lakes and data warehouses, but also what are some answers that we can have in regards to proposal responses, RP responses, RFI, responses and the like. [00:20:23] Vince Menzione: And you’re reaching out to the other LLMs through your tool? [00:20:26] Darryl Peek: We can actually interact with any LLM. So we are a LLM Agnostic. [00:20:29] Vince Menzione: Got it. Yep. That’s fantastic. And this slide is we’ll make this available if you don’t have a, yeah, have a chance. We’ll share it. I [00:20:36] Darryl Peek: am happy to share, yeah. And obviously happy to talk, reach out about it. Of, of course. I simplified it in order to account for you, but one of the things that I talk about is mission, vision of values. [00:20:45] Darryl Peek: And as we start with that is what is your mission now? How is anybody from Pittsburgh, anybody steal a fan? Oh wow. No, there’s a steel fan over [00:20:54] Vince Menzione: here. There’s one here. There’s a couple of ’em are out here. So I feel bad. [00:20:57] Darryl Peek: The reason why I put immaculate in there is for the immaculate reception, actually. [00:21:00] Darryl Peek: Yes. And basically saying that if you ever seen that play, it was not pretty at all. It was a very discombobulated play. Yeah. And I usually say that’s the way that you work with partners too, because when that deal doesn’t come in, when you gotta make a call, when you’re texting somebody at 11 o’clock at night, when you’re trying to get that at, right before quarter end. [00:21:17] Darryl Peek: Yeah. Before the end of it. It really is difficult, but it’s really creating that immaculate experience. You want that partner to come back. I know it’s challenging, but I appreciate how you leaned in with us. Yes, absolutely. I appreciate how you work with us. I appreciate how you held our hand through the process, and that’s what I tell my team, that we have to create that partner experience. [00:21:32] Darryl Peek: And maybe that’s a carryover from Salesforce, Dave. I don’t know. But also when we talk about enhancing or accelerating our partner. Our public sector outcomes that is really working with the customer, right? So customer experience has to be part of it. Like all of us have to be focused on that North star, and that is really how do we service the customer, and that’s what we choose to do. [00:21:48] Darryl Peek: But also the internal part. So I used to survey my team many moves ago, and I said, if we don’t get 80% satisfaction rate from our employees how do we get 60% satisfaction rate from our customers? Yeah. So really focus on that employee success and employee satisfaction. It’s so important, is very important. [00:22:03] Darryl Peek: So being able to understand what are the needs of your employees? Are you really addressing their concerns and are you really driving them forward? Are you challenging them? Are you creating pathways for progression? So those are things that I definitely try to do with my team. As well as just really encouraging, inspiring, yeah. [00:22:19] Darryl Peek: And just making sure that they’re having fun at the same time. [00:22:21] Vince Menzione: It shows up in such, I, there’s an airline I don’t fly any longer, and it was a million mile member of and I know it’s because of the way they treat their employees. [00:22:29] Vince Menzione: Because it cascades Right? [00:22:30] Darryl Peek: It does. Culture is important. [00:22:32] Vince Menzione: Yeah. Absolutely. [00:22:32] Darryl Peek: What is it? What Anderson Howard they say what col. Mark Andresen culture eat strategy for [00:22:37] Vince Menzione: breakfast. He strategy for breakfast? Yes. Very much this has been insightful. I really enjoyed having you here today. Really a great, you’re a lot of fun. You’re a lot of fun. [00:22:43] Vince Menzione: Darry, isn’t you? Amazing. So thank you for joining us. Thank you all. Thank And you’re gonna be, you’re gonna be sticking around for a little while today. I’m sticking around for a little while. I’ll be back in little later. I think people are gonna just en enjoy having a conversation with you, a little sidebar. [00:22:55] Darryl Peek: Absolutely. I’m looking forward to it. Thank you all for having me. Glad to be here. And thank you for giving the time today. [00:23:01] Vince Menzione: Thank you Darryl, so much. So appreciate it. And you’re gonna have to come join me on this Story Diamond tool. Yeah, absolutely. Thanks for tuning into this episode of Ultimate Guide to Partnering. [00:23:12] Vince Menzione: We’re bringing these episodes to you to help you level up your strategy. If you haven’t yet, now’s the time to take action and think about joining our community. We created a unique place, UPX or Ultimate partner experience. It’s more than a community. It’s your competitive edge with insider insights, real-time education, and direct access to people who are driving the ecosystem forward. [00:23:38] Vince Menzione: UPX helps you get results, and we’re just getting started as we’re taking this studio. And we’ll be hosting live stream and digital events here, including our January live stream, the Boca Winter Retreat, and more to come. So visit our website, the ultimate partner.com to learn more and join us. Now’s the time to take your partnerships to the next level.

Jack Ellis recently shared that storing page views and custom events in separate database tables was his biggest mistake at Fathom Analytics. That got me thinking about my own data modeling decisions at Podscan—choices I made on day one that now, two years and 45 million episodes later, either enable or constrain everything I build. Today, I'm exploring how your data model doesn't just store information, it fundamentally shapes how you think about your product. From the simple decision of whether to include teams in your authentication system to the complex realities of running full-text search across terabytes of transcript data, I'll share the migrations, the blue-green deployments, and the hard lessons about building flexibility into both your infrastructure and your founder mindset.I'm running a time-limited Black Friday sale of The Bootstrapper's Bundle: all my books, all my courses, all formats, for $25 instead of $100+. Grab it here: https://tbf.link/bffThis episode of The Bootstraped Founder is sponsored by Paddle.comYou'll find the Black Friday Guide here: https://www.paddle.com/learn/grow-beyond-black-fridayThe blog post: https://thebootstrappedfounder.com/how-your-data-model-shapes-your-product/ The podcast episode: https://tbf.fm/episodes/426-how-your-data-model-shapes-your-product Check out Podscan, the Podcast database that transcribes every podcast episode out there minutes after it gets released: https://podscan.fmSend me a voicemail on Podline: https://podline.fm/arvidYou'll find my weekly article on my blog: https://thebootstrappedfounder.comPodcast: https://thebootstrappedfounder.com/podcastNewsletter: https://thebootstrappedfounder.com/newsletterMy book Zero to Sold: https://zerotosold.com/My book The Embedded Entrepreneur: https://embeddedentrepreneur.com/My course Find Your Following: https://findyourfollowing.comHere are a few tools I use. Using my affiliate links will support my work at no additional cost to you.- Notion (which I use to organize, write, coordinate, and archive my podcast + newsletter): https://affiliate.notion.so/465mv1536drx- Riverside.fm (that's what I recorded this episode with): https://riverside.fm/?via=arvid- TweetHunter (for speedy scheduling and writing Tweets): http://tweethunter.io/?via=arvid- HypeFury (for massive Twitter analytics and scheduling): https://hypefury.com/?via=arvid60- AudioPen (for taking voice notes and getting amazing summaries): https://audiopen.ai/?aff=PXErZ- Descript (for word-based video editing, subtitles, and clips): https://www.descript.com/?lmref=3cf39Q- ConvertKit (for email lists, newsletters, even finding sponsors): https://convertkit.com?lmref=bN9CZw

Today, we are dropping our final episode in the series "The Railsware Way", sponsored by our good friends at Railsware. Railsware is a leading product studio with two main focuses - services and products. They have created amazing products like Mailtrap, Coupler and TitanApps, while also partnering with teams like Calendly and Bright Bytes. They deliver amazing products, and have happy customers to prove it.In this series, we are digging into the company's methods around product engineering and development. In particular, we will cover relevant topics to not only highlight their expertise, but to educate you on industry trends alongside their experience.In today's episode, we are speaking with Oleksii Ianchuk, Product Lead at Railsware, specifically for Mailtrap. Thought he doesn't like to limit his activities to product development, Oleksii has spent six years in product and project management, and is keen on searching for insights and putting them to work, as well as gauging the effects of his input.Questions:The story of Mailtrap starts with accidentally sending test emails to real users in 2011. How did Mailtrap evolve from an internal "fail" to a platform serving hundreds of thousands of users? How did that mistake spark the creation of Mailtrap, and what lessons did you learn about turning problems into opportunities?What made you decide to expand from email testing into Email API/SMTP delivery - and why was it harder than expected? What specific challenges around deliverability, spam fighting, and infrastructure caught you off guard?Can you walk us through the "splitting the product" mistake and its long-term consequences? Your team decided to separate testing and sending into different repositories and isolated VPC projects. What seemed like a good engineering decision at the time - how did this create problems as you scaled, and what would you do differently?You spent a year struggling with Redshift before switching to Elasticsearch - what did that teach you about technology decisions? You ran tests, evaluated alternatives, and still picked the wrong database for your use case. How do you balance thorough research with the reality that you can't always predict what will work until you're in production?When do you buy external expertise versus rely on your internal team? How do you decide when to hire outside knowledge, and how do you find the right consultants for niche problems?Why didn't existing Mailtrap users immediately adopt the Email API/SMTP feature, and what did that teach you?You expected current users to quickly transition to the new sending functionality. What did you learn about switching costs, user perception, and the challenge of changing how people think about your product?What business insights around deliverability, spam prevention, and compliance surprised you most?Email delivery isn't just about infrastructure - there's a whole ecosystem of postmasters, anti-spam systems, and compliance requirements. What aspects of this business were most unexpected, and how did they shape your product strategy?Looking at Mailtrap's 13-year journey, what's your philosophy on "failing fast" versus "building solid foundations"?Linkshttps://railsware.com/https://www.linkedin.com/in/yanch/Our Sponsors:* Check out Incogni: https://incogni.com/codestory* Check out NordProtect: https://nordprotect.com/codestorySupport this podcast at — https://redcircle.com/code-story-insights-from-startup-tech-leaders/donationsAdvertising Inquiries: https://redcircle.com/brandsPrivacy & Opt-Out: https://redcircle.com/privacy

В 85 выпуске подкаста Javaswag в гостях Роман Гребенников, инженер с огромным опытом в разработке поисковых движков (Findify, Delivery Hero) и создатель open-source проектов Metarank и Nixie Search. Мы обсудили эволюцию поиска от “просто возьми Elastic” до хайпа по векторным базам данных и обратно. Поговорили о том, почему Scala всё еще жива, зачем нужен GraalVM в 2024 году, и как построить свой поисковый движок поверх S3 и AWS Lambda. 00:00 — Начало 04:44 — Findify: скраперы на C++, переход на Scala 13:25 — Эволюция поиска - ElasticSearch 19:37 — Elasticsearch vs OpenSearch 22:50 — Apache Lucene Deep Dive 28:53 — Как выбрать поиск для своего проекта? 38:40 — Spark vs Apache Flink 48:30 — MetaRank 53:48 — Почему Scala 01:05:25 — Python в ML 01:13:41 — Стартапы vs Корпорации 01:21:17 — Nixie Search 01:36:58 — Рынок векторных БД: Qdrant, Meilisearch, TurboPuffer 01:47:15 — Опыт с GraalVM: Как засунуть Scala и Lucene в AWS Lambda с холодным стартом в 20 мс 01:57:24 — Непопулярное мнение Гость: https://twitter.com/public_void_grv Ссылки: Nixie Search https://github.com/nixiesearch/nixiesearch MetaRank https://github.com/metarank/metarank Apache Lucene https://lucene.apache.org/ Apache Flink https://flink.apache.org/ GraalVM https://www.graalvm.org Qdrant https://qdrant.tech/ Ссылки на подкаст: Сайт - https://javaswag.github.io/ Телеграм - https://t.me/javaswag Youtube - https://www.youtube.com/@javaswag Linkedin - https://www.linkedin.com/in/volyihin/ X - https://x.com/javaswagpodcast

** AWS re:Invent 2025 Dec 1-5, Las Vegas - Register Here! **Uri Cohen reveals how Elastic transformed from managing 50,000 complex clusters to building a seamless serverless platform that eliminates operational overhead while scaling globallyTopics Include:Johan Broman of AWS hosts Uri Cohen who leads Elastic's platform products teamUri shares his nine-year journey at Elastic from small company to global scaleElasticsearch started 15 years ago, becoming popular for search, logs, and security eventsElastic Cloud launched 2015, but users struggled with shards, nodes, and infrastructure complexityServerless eliminates operational concerns, letting users just ingest and analyze their dataDesign goal: maintain familiar Elasticsearch experience while removing all infrastructure management burdenChose complete architectural redesign over retrofitting auto-scaling to existing infrastructureNew architecture uses S3 persistence with lightweight routing layer serving 50,000+ clustersCell-based design limits blast radius and improves multi-tenancy across 40+ global regionsLearned S3 API costs can explode unexpectedly without careful request pattern optimizationAI transforms security workflows: 10,000 alerts become 3 actionable attack summaries automaticallyWeekly continuous deployment enables faster innovation delivery without waiting for version releasesParticipants:Uri Cohen – Vice President of Product Management, Platform, ElasticJohan Broman – EMEA ISV Head of Solutions Architecture, Amazon Web ServicesSee how Amazon Web Services gives you the freedom to migrate, innovate, and scale your software company at https://aws.amazon.com/isv/

Si tienes un servidor Linux expuesto a Internet, ya sea un VPS o una Raspberry Pi alojando tus servicios Docker, este es un episodio que no te puedes saltar. Detrás de ese proxy inverso (Traefik es mi elección), se esconde un tráfico que rara vez revisamos, y te aseguro que no todo el mundo tiene buenas intenciones.Tras un incidente reciente que me obligó a abrir mi servidor al mundo (y no solo a España, como lo tenía restringido inicialmente), la cantidad de visitantes desconocidos y peticiones curiosas que encontré me hizo poner manos a la obra. No es solo un tema de seguridad; es de recursos.Cada visita cuesta. Sí, has oído bien. Cada interacción con tu servidor requiere un gasto de CPU y memoria RAM. Los bots y scanners que buscan vulnerabilidades o hacen peticiones inútiles están consumiendo silenciosamente la capacidad de tu sistema, dejando menos para tus visitas de calidad (las que realmente quieres). Es esencial saber quién te visita, dónde va, y con qué intenciones, para poder actuar y liberar esos recursos.Mi objetivo, como siempre en atareao con Linux, era encontrar una solución de código abierto que fuera sencilla de implementar y, crucialmente, que no se llevara por delante todos los recursos de mi propio servidor.El punto de partida de la investigación es siempre el access.log de Traefik, que es el registro fundamental de todas las peticiones.Estuve probando distintas combinaciones, incluyendo algunas pesadas y complejas, como:Vector, Prometheus, Grafana y Loki.Vector, Victorialogs, Grafana y Loki.Si bien estas son soluciones potentes, su complejidad y el alto consumo de recursos me hicieron descartarlas. La solución no debe ser un problema de rendimiento en sí misma.Finalmente, di con la combinación que es simple, eficiente y con la que estoy realmente enamorado por su facilidad de uso e implementación.Vector es la herramienta clave para recopilar, transformar y enrutar todos tus logs, métricas y trazas. Es de código abierto, hasta 10 veces más rápido que cualquier alternativa y es lo que me permite un enriquecimiento de datos sin precedentes.En este episodio aprenderás cómo:Configurar el compose.yml de Vector en tu entorno Docker.Utilizar las Transforms de Vector para parsear los logs de Traefik.Integrar la base de datos GeoIP (GeoLite2-City.mmdb) para geolocalizar la IP de procedencia de cada petición.Enrutar los logs enriquecidos a la base de datos de destino.OpenObserve (O2) es la plataforma de observabilidad nativa de la nube que unifica logs, métricas y trazas en una única interfaz. Es la alternativa que he adoptado a soluciones como ElasticSearch y se ha convertido en una herramienta imprescindible en mi día a día.Es increíblemente sencillo de instalar y configurar (lo tienes funcionando en minutos).Es el lugar donde guardo y analizo toda la información de tráfico y rendimiento de mi infraestructura Docker y Traefik.Te proporciono el código compose.yml para que puedas desplegar esta base de datos en cuestión de minutos y empezar a interactuar con los datos geolocalizados que envía Vector.Además de la solución Vector/OpenObserve, te presento un interesante descubrimiento: el Traefik Log Dashboard. Este proyecto de código abierto (backend en Go, frontend en React) te permite tener información en tiempo real de los logs de Traefik con geolocalización incluida.Monitorización en tiempo real vía WebSocket.Soporte para trazas en tiempo real (OpenTelemetry OTLP).Analíticas completas de tiempos de respuesta, códigos de estado y tasas de solicitud.Más información y enlaces en las notas del episodio

Modernizing Search Infrastructure: How Instacart Transitioned from Elasticsearch to PostgreSQL for Enhanced Performance and Simplicity. In this episode of The Data Engineering Show, host Benjamin Wagner speaks with Ankit Mittal, former senior engineer at Instacart, about the company's innovative approach to modernizing their search infrastructure by transitioning from Elasticsearch to PostgreSQL for single-retailer search functionality.

At Berlin Buzzwords, industry voices highlighted how search is evolving with AI and LLMs.- Kacper Łukawski (Qdrant) stressed hybrid search (semantic + keyword) as core for RAG systems and promoted efficient embedding models for smaller-scale use.- Manish Gill (ClickHouse) discussed auto-scaling OLAP databases on Kubernetes, combining infrastructure and database knowledge.- André Charton (Kleinanzeigen) reflected on scaling search for millions of classifieds, moving from Solr/Elasticsearch toward vector search, while returning to a hands-on technical role.- Filip Makraduli (Superlinked) introduced a vector-first framework that fuses multiple encoders into one representation for nuanced e-commerce and recommendation search.- Brian Goldin (Voyager Search) emphasized spatial context in retrieval, combining geospatial data with AI enrichment to add the “where” to search.- Atita Arora (Voyager Search) highlighted geospatial AI models, the renewed importance of retrieval in RAG, and the cautious but promising rise of AI agents.Together, their perspectives show a common thread: search is regaining center stage in AI—scaling, hybridization, multimodality, and domain-specific enrichment are shaping the next generation of retrieval systems.Kacper Łukawski Senior Developer Advocate at Qdrant, he educates users on vector and hybrid search. He highlighted Qdrant's support for dense and sparse vectors, the role of search with LLMs, and his interest in cost-effective models like static embeddings for smaller companies and edge apps. Connect: https://www.linkedin.com/in/kacperlukawski/Manish Gill Engineering Manager at ClickHouse, he spoke about running ClickHouse on Kubernetes, tackling auto-scaling and stateful sets. His team focuses on making ClickHouse scale automatically in the cloud. He credited its speed to careful engineering and reflected on the shift from IC to manager. Connect: https://www.linkedin.com/in/manishgill/André Charton Head of Search at Kleinanzeigen, he discussed shaping the company's search tech—moving from Solr to Elasticsearch and now vector search with Vespa. Kleinanzeigen handles 60M items, 1M new listings daily, and 50k requests/sec. André explained his career shift back to hands-on engineering. Connect: https://www.linkedin.com/in/andrecharton/Filip Makraduli Founding ML DevRel engineer at Superlinked, an open-source framework for AI search and recommendations. Its vector-first approach fuses multiple encoders (text, images, structured fields) into composite vectors for single-shot retrieval. His Berlin Buzzwords demo showed e-commerce search with natural-language queries and filters. Connect: https://www.linkedin.com/in/filipmakraduli/Brian Goldin Founder and CEO of Voyager Search, which began with geospatial search and expanded into documents and metadata enrichment. Voyager indexes spatial data and enriches pipelines with NLP, OCR, and AI models to detect entities like oil spills or windmills. He stressed adding spatial context (“the where”) as critical for search and highlighted Voyager's 12 years of enterprise experience. Connect: https://www.linkedin.com/in/brian-goldin-04170a1/Atita Arora Director of AI at Voyager Search, with nearly 20 years in retrieval systems, now focused on geospatial AI for Earth observation data. At Berlin Buzzwords she hosted sessions, attended talks on Lucene, GPUs, and Solr, and emphasized retrieval quality in RAG systems. She is cautiously optimistic about AI agents and values the event as both learning hub and professional reunion. Connect: https://www.linkedin.com/in/atitaarora/

docs.search("indie, founder, rails, successful") => Jason Bosco / TypesenseIn this episode, Jeremy and Jess sit down with Jason Bosco, co-founder of Typesense, an open source, typo-tolerant search engine. Jason shares how he and his co-founder committed to simply showing up every day, putting in consistent effort, no matter how small, and how that patience eventually compounded into success.We dive into Jason's journey from VP of Engineering at Dollar Shave Club to building his own company, why Typesense has chosen to stay customer-funded instead of VC-funded, and how open source has been central to their mission of democratizing search. Along the way, Jason offers insights on perseverance, product focus, and the long game of building an indie software company and how it can all good and difficult can take a toll on health. Jason BoscoCEO & Co-Founder at TypesensePreviously VP of Engineering at Dollar Shave Club Previously VP of Technology at Verishophttps://x.com/jasonboscohttps://www.linkedin.com/in/jasonbosco/https://www.crunchbase.com/person/jason-boscohttps://github.com/jasonboscoA good way to describe Typesense is that it's an open source alternative to Algolia and an easier-to-use alternative to ElasticSearch. https://github.com/jasonboscoFeatured Videoshttps://www.youtube.com/watch?v=1cdH1F6zbIghttps://www.youtube.com/watch?v=ER8FDiCMPCYhttps://www.youtube.com/watch?v=QymF4NUmALMhttps://www.youtube.com/watch?v=QNLA8RCrYwkhttps://www.youtube.com/watch?v=u4Z3BYwizzwFeatured Linkshttps://typesense.org/abouthttps://typesense.org/blog/To Raise VC, or Not. Choosing The Road Less Travelled.The unreasonable effectiveness of just showing up everydayWe bought HUNDREDS of billboards in San Francisco, for our open source producthttps://gorelay.co/t/pursuing-an-open-source-bootstrapped-long-run-path-towards-serving-the-fortune-1-million-with-typesense-s-co-founder-jason-bosco/915https://livecycle.io/blogs/dev-x-project-jason-bosco/

The ClickHouse open source project has gained interest in the observability community, thanks to its outstanding performance benchmarks. Now ClickHouse is doubling down on observability with the release of ClickStack, a new open source observability stack that bundles in ClickHouse, OpenTelemetry and  HyperDX frontend. I invited Mike Shi, the co-founder of HyperDX and co-creator of ClickStack, to tell us all about this new project. Mike is Head of Observability at ClickHouse, and brings prior observability experience with Elasticsearch and more.You can read the recap post: https://medium.com/p/73f129a179a3/Show Notes:00:00 episode and guest intro04:38 taking the open source path as an entrepreneur10:51 the HyperDX observability user experience 16:08 challenges in implementing observability directly on ClickHouse20:03 intro to ClickStack and incorporating OpenTelemetry32:35 balancing simplicity and flexibility36:15 SQL vs. Lucene query languages 39:06 performance, cardinality and the new JSON type52:14 use cases in production by OpenAI, Anthropic, Tesla and more55:38 episode outroResources:HyperDX https://github.com/hyperdxio/hyperdx ClickStack https://clickhouse.com/docs/use-cases/observability/clickstack Shopify's Journey to Planet-Scale Observability: https://medium.com/p/9c0b299a04ddClickHouse: Breaking the Speed Limit for Observability and Analytics https://medium.com/p/2004160b2f5e New JSON data type for ClickHouse: https://clickhouse.com/blog/a-new-powerful-json-data-type-for-clickhouseSocials:BlueSky: https://bsky.app/profile/openobservability.bsky.socialTwitter: ⁠https://twitter.com/OpenObserv⁠LinkedIn: https://www.linkedin.com/company/openobservability/YouTube: ⁠https://www.youtube.com/@openobservabilitytalks⁠Dotan Horovits============Twitter: @horovitsLinkedIn: www.linkedin.com/in/horovitsMastodon: @horovits@fosstodonBlueSky: @horovits.bsky.socialMike Shi=======Twitter: https://x.com/MikeShi42LinkedIn: https://www.linkedin.com/in/mikeshi42BlueSky: https://bsky.app/profile/mikeshi42.bsky.socialOpenObservability Talks episodes are released monthly, on the last Thursday of each month and are available for listening on your favorite podcast app and on YouTube.

Gagan Singh of Elastic discuses how agentic AI systems reduce analyst burnout by automatically triaging security alerts, resulting in measurable ROI for organizationsTopics Include:AI breaks security silos between teams, data, and tools in SOCsAttackers gain system access; SOC teams have only 40 minutes to detect/containAlert overload causes analyst burnout; thousands of low-value alerts overwhelm teams dailyAI inevitable for SOCs to process data, separate false positives from real threatsAgentic systems understand environment, reason through problems, take action without hand-holdingAttack discovery capability reduces hundreds of alerts to 3-4 prioritized threat discoveriesAI provides ROI metrics: processed alerts, filtered noise, hours saved for organizationsRAG (Retrieval Augmented Generation) prevents hallucination by adding enterprise context to LLMsAWS integration uses SageMaker, Bedrock, Anthropic models with Elasticsearch vector database capabilitiesEnd-to-end LLM observability tracks costs, tokens, invocations, errors, and performance bottlenecksJunior analysts detect nation-state attacks; teams shift from reactive to proactive securityFuture requires balancing costs, data richness, sovereignty, model choice, human-machine collaborationParticipants:Gagan Singh – Vice President Product Marketing, ElasticAdditional Links:Elastic – LinkedIn - Website – AWS Marketplace See how Amazon Web Services gives you the freedom to migrate, innovate, and scale your software company at https://aws.amazon.com/isv/

Increased Elasticsearch Recognizance Scans Our honeypots noted an increase in reconnaissance scans for Elasticsearch. In particular, the endpoint /_cluster/settings is hit hard. https://isc.sans.edu/diary/Increased%20Elasticsearch%20Recognizance%20Scans/32212 Microsoft Patch Tuesday Issues Microsoft noted some issues deploying the most recent patches with WSUS. There are also issues with certain SSDs if larger files are transferred. https://learn.microsoft.com/en-us/windows/release-health/status-windows-11-24h2#3635msgdesc https://www.tomshardware.com/pc-components/ssds/latest-windows-11-security-patch-might-be-breaking-ssds-under-heavy-workloads-users-report-disappearing-drives-following-file-transfers-including-some-that-cannot-be-recovered-after-a-reboot SAP Vulnerabilities Exploited CVE-2025-31324, CVE-2025-42999 Details explaining how to take advantage of two SAP vulnerabilities were made public https://onapsis.com/blog/new-exploit-for-cve-2025-31324/

Alex & Chris get into a fairly recent technological change at CodePen where we ditched our Elasticsearch implementation for just using our own Postgres database for search. Sometimes choices like this are more about team expertise, dev environment practicalities, and complexity tradeoffs. We found this change to be much better for us, which matters! For the most part search is better and faster. Postgres is not nearly as fancy and capable as Elasticsearch, but we werent taking advantage of what Elasticsearch had to offer anyway. For the power users out there: it's true that we've lost the ability to do in-code search for now. But it's temporary and will be coming back in time. Time Jumps

In this episode of the Don't Panic, It's Just Data podcast, Kevin Petrie, VP of Research at BARC and the podcast host, is joined by Dainius Jocas, Search Engineer at Vinted, and Radu Gheorghe, Software Engineer at Vespa.ai. They discuss how Vinted, an online marketplace for secondhand products, modernised its data architecture to address new AI search use cases and the challenges faced with Elasticsearch. From the switch to Vespa and the advantages of supporting multiple languages and complex queries, the podcast offers insights on the trade-offs organisations must think about when updating their search systems, especially regarding AI and machine learning applications.Vinted Elasticsearch ChallengesVinted's search architecture was built on Elasticsearch before they switched to Vespa. Elasticsearch is a functional system that presents a few major challenges. With over 20 supported languages, the company's "index per language" approach created significant sharding problems, leading to infrastructure imbalances and constant adjustments."The index for the French language, the biggest language that we support, was more than three times bigger than the second biggest language, which created imbalances in the Elasticsearch data nodes' load," Jocas explained.In addition to these technical obstacles, organisational issues arose as teams responsible for different parts of the search process found themselves "pointing fingers at each other at an increasing rate." The need for a more integrated, effective solution became clear.The Solution: A New Platform for a New EraThe search for a better solution led Vinted to Vespa. The initial adoption was a "one success story" when a machine learning engineer, working on recommendations, discovered that Vespa was ten times faster than Elasticsearch for their use case. This initial benchmark, run on a single decommissioned server, was a "true testament to how efficient Vespa is when it comes to serving requests,” Jocas told Petrie.Vespa helped Vinted solve their language problem by allowing it to set a language per document. Thus, it eliminates the need for separate indexes and the associated sharding headaches. As Jocas put it, "We got out of the sharding problem once and for all."TakeawaysVinted faced challenges with its initial Elasticsearch architecture.The need for better integration between matching and ranking was identified.Vespa outperformed Elasticsearch in handling image search and recommendations.Transitioning to Vespa involved significant learning and support from developers.Vespa allows for language-specific document handling, simplifying architecture.Organisations must evaluate the complexity and volume of their data before transitioning.Vespa is optimised for query performance, while Elasticsearch excels in data writing.The learning curve for Vespa can be steep, but support is available.It's important to focus on optimising new systems rather than emulating old ones.Partial updates in Vespa are more efficient than in Elasticsearch.Chapters00:00 Introduction to Vinted and...

ParadeDB built a Postgres extension that facilitates full-text search and analytics on Postgres without the need to transfer data. Learn more about your ad choices. Visit podcastchoices.com/adchoices

In this special in-person episode, Sanne Grinovero shares the story of Java's evolution from his unique perspective as a long-time open-source contributor. He shares his 16-year career journey at Red Hat, highlighting his amazing work on key projects like Hibernate, Infinispan, and especially the creation of Quarkus. His career trajectory, from a student who initially disliked Java's complexity to a leading figure in its modernization, shows the transformative power of open source.A key part of the conversation focuses on how technical challenges spark innovation. Sanne explains how the task of making the popular Hibernate framework compatible with GraalVM's limitations led directly to the birth of Quarkus. This journey tells the bigger story of how Java adapted for cloud-native development, ensuring it continues to be a top choice for developers seeking high performance and a great developer experience.  Timestamps:(00:00:00) Trailer & Intro(00:02:16) Career Turning Points(00:04:52) Winning an Innovation Award(00:06:35) Java Heroes(00:08:04) Working as a Consultant(00:09:56) Taking a Massive Pay Cut to Work on Open Source(00:10:59) Contributing to Big Open Source as a Youngster(00:12:53) State of Hibernate Project(00:15:15) Spring Boot(00:16:54) Making Hibernate Work on GraalVM(00:21:05) GraalVM Limitations for Running Hibernate(00:26:09) Java for Cloud Native Application(00:28:04) Quarkus vs Spring Boot(00:33:21) JRebel & Quarkus(00:34:35) Java vs New Programming Languages(00:39:22) The ORM Dilemma(00:42:38) Some Hibernate Design Pattern Tips(00:46:40) Getting Paid Working on Open Source(00:48:41) Hibernate License Change(00:51:05) Intellectual Property & Meaningful Contributions(00:52:52) AI Usage & Copyright in Open Source(00:55:21) Biggest Challenge Working in a Big Open Source(00:56:08) Politics in Open Source(00:58:32) Security Risks in Open Source(01:02:25) Donating Hibernate to Commonhaus Foundation(01:04:49) The Future of Red Hat(01:06:39) 3 Tech Lead Wisdom_____Sanne Grinovero's BioSanne Grinovero has been a member of the Hibernate team for 10 years; today he leads this project in his role of Sr. Principal Software Engineer at Red Hat, while also working on Quarkus as a founding R&D engineer.Deeply interested in solving performance and concurrency challenges around data access, scalability, and exploring integration with new storage technologies, distributed systems and search engines.Working on Hibernate features led him to contribute to related open source technologies; most notably to Apache Lucene and Elasticsearch, Infinispan and JGroups, ANTLR, WildFly, various JDBC drivers, the OpenJDK and more recently getting interested in GraalVM.After being challenged to reduce memory consumption and improve bootstrap times of Hibernate, Sanne worked as part of a small R&D team at Red Hat on some ideas which have evolved into what is known today as Quarkus.Follow Sanne:LinkedIn – linkedin.com/in/sannegrinoveroTwitter – twitter.com/SanneGrinoveroGitHub – github.com/sanneLike this episode?Show notes & transcript: techleadjournal.dev/episodes/220.Follow @techleadjournal on LinkedIn, Twitter, and Instagram.Buy me a coffee or become a patron.

Wer darf eigentlich was? Und sollten wir alle wirklich alles dürfen?Jedes Tech-Projekt beginnt mit einer simplen Frage: Wer darf eigentlich was? Doch spätestens wenn das Startup wächst, Kunden Compliance fordern oder der erste Praktikant an die Produktionsdatenbank rührt, wird Role Based Access Control (RBAC) plötzlich zur Überlebensfrage – und wer das Thema unterschätzt, hat schnell die Rechtehölle am Hals.In dieser Folge nehmen wir das altbekannte Konzept der rollenbasierten Zugriffskontrolle auseinander. wir klären, welches Problem RBAC eigentlich ganz konkret löst, warum sich hinter den harmlosen Checkboxen viel technische Tiefe und organisatorisches Drama verbirgt und weshalb RBAC nicht gleich RBAC ist.Dabei liefern wir dir Praxis-Insights: Wie setzen Grafana, Sentry, Elasticsearch, OpenSearch oder Tracing-Tools wie Jäger dieses Rechtekonzept um? Wo liegen die Fallstricke in komplexen, mehrmandantenfähigen Systemen?Ob du endlich verstehen willst, warum RBAC, ABAC (Attribute-Based), ReBAC (Relationship-Based) und Policy Engines mehr als nur Buzzwords sind oder wissen möchtest, wie du Policies, Edge Cases und Constraints in den Griff bekommst, darum geht es in diesem Deep Dives.Auch mit dabei: Open Source-Highlights wie Casbin, SpiceDB, OpenFGA und OPA und echte Projekt- und Startup-Tipps für pragmatischen Start und spätere Skalierung.Bonus: Ein Märchen mit Kevin und Max, wo auch manchmal der Praktikant trotzdem gegen den Admin gewinnt

What if you could use ElasticSearch serverless? While at Build, Carl and Richard chatted with Ken Exner about the new announcements around Elastic providing serverless storage and search! Ken talks about paying for only the data you move and store with serverless, rather than needing to operate any infrastructure for Elastic. The conversation digs into the potential of Elastic in Azure AI Foundry to provide ultra-fast access to current company data for your LLM implementations. Elastic did vector databases before LLMs made them essential for RAG - and you can take advantage of it!

This episode was sponsored by Elastic! Elastic is the company behind Elasticsearch, they help teams find, analyze, and act on their data in real-time through their Search, Observability, and Security solutions. Thanks Elastic! This episode was recorded at Elastic's offices in San Francisco during a meetup.Find info about the show, past episodes including transcripts, our swag store, Patreon link, and more at https://cupogo.dev/.

What if you could use ElasticSearch serverless? While at Build, Carl and Richard chatted with Ken Exner about the new announcements around Elastic providing serverless storage and search! Ken talks about paying for only the data you move and store with serverless, rather than needing to operate any infrastructure for Elastic. The conversation digs into the potential of Elastic in Azure AI Foundry to provide ultra-fast access to current company data for your LLM implementations. Elastic did vector databases before LLMs made them essential for RAG - and you can take advantage of it!

Join hosts Daniel Garcia and Grant Copley as they dive into the latest news and updates in the BoxLang and CFML world. Don't miss out on insights, discussions, and what's coming next for modern software development!

In a new season of the Oracle University Podcast, Lois Houston and Nikita Abraham dive into the world of Oracle GoldenGate 23ai, a cutting-edge software solution for data management. They are joined by Nick Wagner, a seasoned expert in database replication, who provides a comprehensive overview of this powerful tool.   Nick highlights GoldenGate's ability to ensure continuous operations by efficiently moving data between databases and platforms with minimal overhead. He emphasizes its role in enabling real-time analytics, enhancing data security, and reducing costs by offloading data to low-cost hardware. The discussion also covers GoldenGate's role in facilitating data sharing, improving operational efficiency, and reducing downtime during outages.   Oracle GoldenGate 23ai: Fundamentals: https://mylearn.oracle.com/ou/course/oracle-goldengate-23ai-fundamentals/145884/237273 Oracle University Learning Community: https://education.oracle.com/ou-community LinkedIn: https://www.linkedin.com/showcase/oracle-university/ X: https://x.com/Oracle_Edu   Special thanks to Arijit Ghosh, David Wright, Kris-Ann Nansen, Radhika Banka, and the OU Studio Team for helping us create this episode. ---------------------------------------------------------------   Episode Transcript: 00:00 Welcome to the Oracle University Podcast, the first stop on your cloud journey. During this series of informative podcasts, we'll bring you foundational training on the most popular Oracle technologies. Let's get started! 00:25 Nikita: Welcome to the Oracle University Podcast! I'm Nikita Abraham, Team Lead: Editorial Services with Oracle University, and with me is Lois Houston: Director of Innovation Programs. Lois: Hi everyone! Welcome to a new season of the podcast. This time, we're focusing on the fundamentals of Oracle GoldenGate. Oracle GoldenGate helps organizations manage and synchronize their data across diverse systems and databases in real time.  And with the new Oracle GoldenGate 23ai release, we'll uncover the latest innovations and features that empower businesses to make the most of their data. Nikita: Taking us through this is Nick Wagner, Senior Director of Product Management for Oracle GoldenGate. He's been doing database replication for about 25 years and has been focused on GoldenGate on and off for about 20 of those years.  01:18 Lois: In today's episode, we'll ask Nick to give us a general overview of the product, along with some use cases and benefits. Hi Nick! To start with, why do customers need GoldenGate? Nick: Well, it delivers continuous operations, being able to continuously move data from one database to another database or data platform in efficiently and a high-speed manner, and it does this with very low overhead. Almost all the GoldenGate environments use transaction logs to pull the data out of the system, so we're not creating any additional triggers or very little overhead on that source system. GoldenGate can also enable real-time analytics, being able to pull data from all these different databases and move them into your analytics system in real time can improve the value that those analytics systems provide. Being able to do real-time statistics and analysis of that data within those high-performance custom environments is really important. 02:13 Nikita: Does it offer any benefits in terms of cost?  Nick: GoldenGate can also lower IT costs. A lot of times people run these massive OLTP databases, and they are running reporting in those same systems. With GoldenGate, you can offload some of the data or all the data to a low-cost commodity hardware where you can then run the reports on that other system. So, this way, you can get back that performance on the OLTP system, while at the same time optimizing your reporting environment for those long running reports. You can improve efficiencies and reduce risks. Being able to reduce the amount of downtime during planned and unplanned outages can really make a big benefit to the overall operational efficiencies of your company.  02:54 Nikita: What about when it comes to data sharing and data security? Nick: You can also reduce barriers to data sharing. Being able to pull subsets of data, or just specific pieces of data out of a production database and move it to the team or to the group that needs that information in real time is very important. And it also protects the security of your data by only moving in the information that they need and not the entire database. It also provides extensibility and flexibility, being able to support multiple different replication topologies and architectures. 03:24 Lois: Can you tell us about some of the use cases of GoldenGate? Where does GoldenGate truly shine?  Nick: Some of the more traditional use cases of GoldenGate include use within the multicloud fabric. Within a multicloud fabric, this essentially means that GoldenGate can replicate data between on-premise environments, within cloud environments, or hybrid, cloud to on-premise, on-premise to cloud, or even within multiple clouds. So, you can move data from AWS to Azure to OCI. You can also move between the systems themselves, so you don't have to use the same database in all the different clouds. For example, if you wanted to move data from AWS Postgres into Oracle running in OCI, you can do that using Oracle GoldenGate. We also support maximum availability architectures. And so, there's a lot of different use cases here, but primarily geared around reducing your recovery point objective and recovery time objective. 04:20 Lois: Ah, reducing RPO and RTO. That must have a significant advantage for the customer, right? Nick: So, reducing your RPO and RTO allows you to take advantage of some of the benefits of GoldenGate, being able to do active-active replication, being able to set up GoldenGate for high availability, real-time failover, and it can augment your active Data Guard and Data Guard configuration. So, a lot of times GoldenGate is used within Oracle's maximum availability architecture platinum tier level of replication, which means that at that point you've got lots of different capabilities within the Oracle Database itself. But to help eke out that last little bit of high availability, you want to set up an active-active environment with GoldenGate to really get true zero RPO and RTO. GoldenGate can also be used for data offloading and data hubs. Being able to pull data from one or more source systems and move it into a data hub, or into a data warehouse for your operational reporting. This could also be your analytics environment too. 05:22 Nikita: Does GoldenGate support online migrations? Nick: In fact, a lot of companies actually get started in GoldenGate by doing a migration from one platform to another. Now, these don't even have to be something as complex as going from one database like a DB2 on-premise into an Oracle on OCI, it could even be simple migrations. A lot of times doing something like a major application or a major database version upgrade is going to take downtime on that production system. You can use GoldenGate to eliminate that downtime. So this could be going from Oracle 19c to Oracle 23ai, or going from application version 1.0 to application version 2.0, because GoldenGate can do the transformation between the different application schemas. You can use GoldenGate to migrate your database from on premise into the cloud with no downtime as well. We also support real-time analytic feeds, being able to go from multiple databases, not only those on premise, but being able to pull information from different SaaS applications inside of OCI and move it to your different analytic systems. And then, of course, we also have the ability to stream events and analytics within GoldenGate itself.  06:34 Lois: Let's move on to the various topologies supported by GoldenGate. I know GoldenGate supports many different platforms and can be used with just about any database. Nick: This first layer of topologies is what we usually consider relational database topologies. And so this would be moving data from Oracle to Oracle, Postgres to Oracle, Sybase to SQL Server, a lot of different types of databases. So the first architecture would be unidirectional. This is replicating from one source to one target. You can do this for reporting. If I wanted to offload some reports into another server, I can go ahead and do that using GoldenGate. I can replicate the entire database or just a subset of tables. I can also set up GoldenGate for bidirectional, and this is what I want to set up GoldenGate for something like high availability. So in the event that one of the servers crashes, I can almost immediately reconnect my users to the other system. And that almost immediately depends on the amount of latency that GoldenGate has at that time. So a typical latency is anywhere from 3 to 6 seconds. So after that primary system fails, I can reconnect my users to the other system in 3 to 6 seconds. And I can do that because as GoldenGate's applying data into that target database, that target system is already open for read and write activity. GoldenGate is just another user connecting in issuing DML operations, and so it makes that failover time very low. 07:59 Nikita: Ok…If you can get it down to 3 to 6 seconds, can you bring it down to zero? Like zero failover time?   Nick: That's the next topology, which is active-active. And in this scenario, all servers are read/write all at the same time and all available for user activity. And you can do multiple topologies with this as well. You can do a mesh architecture, which is where every server talks to every other server. This works really well for 2, 3, 4, maybe even 5 environments, but when you get beyond that, having every server communicate with every other server can get a little complex. And so at that point we start looking at doing what we call a hub and spoke architecture, where we have lots of different spokes. At the end of each spoke is a read/write database, and then those communicate with a hub. So any change that happens on one spoke gets sent into the hub, and then from the hub it gets sent out to all the other spokes. And through that architecture, it allows you to really scale up your environments. We have customers that are doing up to 150 spokes within that hub architecture. Within active-active replication as well, we can do conflict detection and resolution, which means that if two users modify the same row on two different systems, GoldenGate can actually determine that there was an issue with that and determine what user wins or which row change wins, which is extremely important when doing active-active replication. And this means that if one of those systems fails, there is no downtime when you switch your users to another active system because it's already available for activity and ready to go. 09:35 Lois: Wow, that's fantastic. Ok, tell us more about the topologies. Nick: GoldenGate can do other things like broadcast, sending data from one system to multiple systems, or many to one as far as consolidation. We can also do cascading replication, so when data moves from one environment that GoldenGate is replicating into another environment that GoldenGate is replicating. By default, we ignore all of our own transactions. But there's actually a toggle switch that you can flip that says, hey, GoldenGate, even though you wrote that data into that database, still push it on to the next system. And then of course, we can also do distribution of data, and this is more like moving data from a relational database into something like a Kafka topic or a JMS queue or into some messaging service. 10:24 Raise your game with the Oracle Cloud Applications skills challenge. Get free training on Oracle Fusion Cloud Applications, Oracle Modern Best Practice, and Oracle Cloud Success Navigator. Pass the free Oracle Fusion Cloud Foundations Associate exam to earn a Foundations Associate certification. Plus, there's a chance to win awards and prizes throughout the challenge! What are you waiting for? Join the challenge today by visiting visit oracle.com/education. 10:58 Nikita: Welcome back! Nick, does GoldenGate also have nonrelational capabilities?  Nick: We have a number of nonrelational replication events in topologies as well. This includes things like data lake ingestion and streaming ingestion, being able to move data and data objects from these different relational database platforms into data lakes and into these streaming systems where you can run analytics on them and run reports. We can also do cloud ingestion, being able to move data from these databases into different cloud environments. And this is not only just moving it into relational databases with those clouds, but also their data lakes and data fabrics. 11:38 Lois: You mentioned a messaging service earlier. Can you tell us more about that? Nick: Messaging replication is also possible. So we can actually capture from things like messaging systems like Kafka Connect and JMS, replicate that into a relational data, or simply stream it into another environment. We also support NoSQL replication, being able to capture from MongoDB and replicate it onto another MongoDB for high availability or disaster recovery, or simply into any other system. 12:06 Nikita: I see. And is there any integration with a customer's SaaS applications? Nick: GoldenGate also supports a number of different OCI SaaS applications. And so a lot of these different applications like Oracle Financials Fusion, Oracle Transportation Management, they all have GoldenGate built under the covers and can be enabled with a flag that you can actually have that data sent out to your other GoldenGate environment. So you can actually subscribe to changes that are happening in these other systems with very little overhead. And then of course, we have event processing and analytics, and this is the final topology or flexibility within GoldenGate itself. And this is being able to push data through data pipelines, doing data transformations. GoldenGate is not an ETL tool, but it can do row-level transformation and row-level filtering.  12:55 Lois: Are there integrations offered by Oracle GoldenGate in automation and artificial intelligence? Nick: We can do time series analysis and geofencing using the GoldenGate Stream Analytics product. It allows you to actually do real time analysis and time series analysis on data as it flows through the GoldenGate trails. And then that same product, the GoldenGate Stream Analytics, can then take the data and move it to predictive analytics, where you can run MML on it, or ONNX or other Spark-type technologies and do real-time analysis and AI on that information as it's flowing through.  13:29 Nikita: So, GoldenGate is extremely flexible. And given Oracle's focus on integrating AI into its product portfolio, what about GoldenGate? Does it offer any AI-related features, especially since the product name has “23ai” in it? Nick: With the advent of Oracle GoldenGate 23ai, it's one of the two products at this point that has the AI moniker at Oracle. Oracle Database 23ai also has it, and that means that we actually do stuff with AI. So the Oracle GoldenGate product can actually capture vectors from databases like MySQL HeatWave, Postgres using pgvector, which includes things like AlloyDB, Amazon RDS Postgres, Aurora Postgres. We can also replicate data into Elasticsearch and OpenSearch, or if the data is using vectors within OCI or the Oracle Database itself. So GoldenGate can be used for a number of things here. The first one is being able to migrate vectors into the Oracle Database. So if you're using something like Postgres, MySQL, and you want to migrate the vector information into the Oracle Database, you can. Now one thing to keep in mind here is a vector is oftentimes like a GPS coordinate. So if I need to know the GPS coordinates of Austin, Texas, I can put in a latitude and longitude and it will give me the GPS coordinates of a building within that city. But if I also need to know the altitude of that same building, well, that's going to be a different algorithm. And GoldenGate and replicating vectors is the same way. When you create a vector, it's essentially just creating a bunch of numbers under the screen, kind of like those same GPS coordinates. The dimension and the algorithm that you use to generate that vector can be different across different databases, but the actual meaning of that data will change. And so GoldenGate can replicate the vector data as long as the algorithm and the dimensions are the same. If the algorithm and the dimensions are not the same between the source and the target, then you'll actually want GoldenGate to replicate the base data that created that vector. And then once GoldenGate replicates the base data, it'll actually call the vector embedding technology to re-embed that data and produce that numerical formatting for you.  15:42 Lois: So, there are some nuances there… Nick: GoldenGate can also replicate and consolidate vector changes or even do the embedding API calls itself. This is really nice because it means that we can take changes from multiple systems and consolidate them into a single one. We can also do the reverse of that too. A lot of customers are still trying to find out which algorithms work best for them. How many dimensions? What's the optimal use? Well, you can now run those in different servers without impacting your actual AI system. Once you've identified which algorithm and dimension is going to be best for your data, you can then have GoldenGate replicate that into your production system and we'll start using that instead. So it's a nice way to switch algorithms without taking extensive downtime. 16:29 Nikita: What about in multicloud environments?  Nick: GoldenGate can also do multicloud and N-way active-active Oracle replication between vectors. So if there's vectors in Oracle databases, in multiple clouds, or multiple on-premise databases, GoldenGate can synchronize them all up. And of course we can also stream changes from vector information, including text as well into different search engines. And that's where the integration with Elasticsearch and OpenSearch comes in. And then we can use things like NVIDIA and Cohere to actually do the AI on that data.  17:01 Lois: Using GoldenGate with AI in the database unlocks so many possibilities. Thanks for that detailed introduction to Oracle GoldenGate 23ai and its capabilities, Nick.  Nikita: We've run out of time for today, but Nick will be back next week to talk about how GoldenGate has evolved over time and its latest features. And if you liked what you heard today, head over to mylearn.oracle.com and take a look at the Oracle GoldenGate 23ai Fundamentals course to learn more. Until next time, this is Nikita Abraham… Lois: And Lois Houston, signing off! 17:33 That's all for this episode of the Oracle University Podcast. If you enjoyed listening, please click Subscribe to get all the latest episodes. We'd also love it if you would take a moment to rate and review us on your podcast app. See you again on the next episode of the Oracle University Podcast.

OpenSearch has evolved significantly since its 2021 launch, recently reaching a major milestone with its move to the Linux Foundation. This shift from company-led to foundation-based governance has accelerated community contributions and enterprise adoption, as discussed by NetApp's Amanda Katona in a New Stack Makers episode recorded at KubeCon + CloudNativeCon Europe. NetApp, an early adopter of OpenSearch following Elasticsearch's licensing change, now offers managed services on the platform and contributes actively to its development.Katona emphasized how neutral governance under the Linux Foundation has lowered barriers to enterprise contribution, noting a 56% increase in downloads since the transition and growing interest from developers. OpenSearch 3.0, featuring a Lucene 10 upgrade, promises faster search capabilities—especially relevant as data volumes surge. NetApp's ongoing investments include work on machine learning plugins and developer training resources.Katona sees the Linux Foundation's involvement as key to OpenSearch's long-term success, offering vendor-neutral governance and reassuring users seeking openness, performance, and scalability in data search and analytics.Learn more from The New Stack about OpenSearch: Report: OpenSearch Bests ElasticSearch at Vector ModelingAWS Transfers OpenSearch to the Linux Foundation OpenSearch: How the Project Went From Fork to FoundationJoin our community of newsletter subscribers to stay on top of the news and at the top of your game. 

Brandon Liu is an open source developer and creator of the Protomaps basemap project. We talk about how static maps help developers build sites that last, the PMTiles file format, the role of OpenStreetMap, and his experience funding and running an open source project full time. Protomaps Protomaps PMTiles (File format used by Protomaps) Self-hosted slippy maps, for novices (like me) Why Deploy Protomaps on a CDN User examples Flickr Pinball Map Toilet Map Related projects OpenStreetMap (Dataset protomaps is based on) Mapzen (Former company that released details on what to display based on zoom levels) Mapbox GL JS (Mapbox developed source available map rendering library) MapLibre GL JS (Open source fork of Mapbox GL JS) Other links HTTP range requests (MDN) Hilbert curve Transcript You can help correct transcripts on GitHub. Intro [00:00:00] Jeremy: I'm talking to Brandon Liu. He's the creator of Protomaps, which is a way to easily create and host your own maps. Let's get into it. [00:00:09] Brandon: Hey, so thanks for having me on the podcast. So I'm Brandon. I work on an open source project called Protomaps. What it really is, is if you're a front end developer and you ever wanted to put maps on a website or on a mobile app, then Protomaps is sort of an open source solution for doing that that I hope is something that's way easier to use than, um, a lot of other open source projects. Why not just use Google Maps? [00:00:36] Jeremy: A lot of people are gonna be familiar with Google Maps. Why should they worry about whether something's open source? Why shouldn't they just go and use the Google maps API? [00:00:47] Brandon: So Google Maps is like an awesome thing it's an awesome product. Probably one of the best tech products ever right? And just to have a map that tells you what restaurants are open and something that I use like all the time especially like when you're traveling it has all that data. And the most amazing part is that it's free for consumers but it's not necessarily free for developers. Like if you wanted to embed that map onto your website or app, that usually has an API cost which still has a free tier and is affordable. But one motivation, one basic reason to use open source is if you have some project that doesn't really fit into that pricing model. You know like where you have to pay the cost of Google Maps, you have a side project, a nonprofit, that's one reason. But there's lots of other reasons related to flexibility or customization where you might want to use open source instead. Protomaps examples [00:01:49] Jeremy: Can you give some examples where people have used Protomaps and where that made sense for them? [00:01:56] Brandon: I follow a lot of the use cases and I also don't know about a lot of them because I don't have an API where I can track a hundred percent of the users. Some of them use the hosted version, but I would say most of them probably use it on their own infrastructure. One of the cool projects I've been seeing is called Toilet Map. And what toilet map is if you're in the UK and you want find a public restroom then it maps out, sort of crowdsourced all of the public restrooms. And that's important for like a lot of people if they have health issues, they need to find that information. And just a lot of different projects in the same vein. There's another one called Pinball Map which is sort of a hobby project to find all the pinball machines in the world. And they wanted to have a customized map that fit in with their theme of pinball. So these sorts of really cool indie projects are the ones I'm most excited about. Basemaps vs Overlays [00:02:57] Jeremy: And if we talk about, like the pinball map as an example, there's this concept of a basemap and then there's the things that you lay on top of it. What is a basemap and then is the pinball locations is that part of it or is that something separate? [00:03:12] Brandon: It's usually something separate. The example I usually use is if you go to a real estate site, like Zillow, you'll open up the map of Seattle and it has a bunch of pins showing all the houses, and then it has some information beneath it. That information beneath it is like labels telling, this neighborhood is Capitol Hill, or there is a park here. But all that information is common to a lot of use cases and it's not specific to real estate. So I think usually that's the distinction people use in the industry between like a base map versus your overlay. The overlay is like the data for your product or your company while the base map is something you could get from Google or from Protomaps or from Apple or from Mapbox that kind of thing. PMTiles for hosting the basemap and overlays [00:03:58] Jeremy: And so Protomaps in particular is responsible for the base map, and that information includes things like the streets and the locations of landmarks and things like that. Where is all that information coming from? [00:04:12] Brandon: So the base map information comes from a project called OpenStreetMap. And I would also, point out that for Protomaps as sort of an ecosystem. You can also put your overlay data into a format called PMTiles, which is sort of the core of what Protomaps is. So it can really do both. It can transform your data into the PMTiles format which you can host and you can also host the base map. So you kind of have both of those sides of the product in one solution. [00:04:43] Jeremy: And so when you say you have both are you saying that the PMTiles file can have, the base map in one file and then you would have the data you're laying on top in another file? Or what are you describing there? [00:04:57] Brandon: That's usually how I recommend to do it. Oftentimes there'll be sort of like, a really big basemap 'cause it has all of that data about like where the rivers are. Or while, if you want to put your map of toilets or park benches or pickleball courts on top, that's another file. But those are all just like assets you can move around like JSON or CSV files. Statically Hosted [00:05:19] Jeremy: And I think one of the things you mentioned was that your goal was to make Protomaps or the, the use of these PMTiles files easy to use. What does that look like for, for a developer? I wanna host a map. What do I actually need to, to put on my servers? [00:05:38] Brandon: So my usual pitch is that basically if you know how to use S3 or cloud storage, that you know how to deploy a map. And that, I think is the main sort of differentiation from most open source projects. Like a lot of them, they call themselves like, like some sort of self-hosted solution. But I've actually avoided using the term self-hosted because I think in most cases that implies a lot of complexity. Like you have to log into a Linux server or you have to use Kubernetes or some sort of Docker thing. What I really want to emphasize is the idea that, for Protomaps, it's self-hosted in the same way like CSS is self-hosted. So you don't really need a service from Amazon to host the JSON files or CSV files. It's really just a static file. [00:06:32] Jeremy: When you say static file that means you could use any static web host to host your HTML file, your JavaScript that actually renders the map. And then you have your PMTiles files, and you're not running a process or anything, you're just putting your files on a static file host. [00:06:50] Brandon: Right. So I think if you're a developer, you can also argue like a static file server is a server. It's you know, it's the cloud, it's just someone else's computer. It's really just nginx under the hood. But I think static storage is sort of special. If you look at things like static site generators, like Jekyll or Hugo, they're really popular because they're a commodity or like the storage is a commodity. And you can take your blog, make it a Jekyll blog, hosted on S3. One day, Amazon's like, we're charging three times as much so you can move it to a different cloud provider. And that's all vendor neutral. So I think that's really the special thing about static storage as a primitive on the web. Why running servers is a problem for resilience [00:07:36] Jeremy: Was there a prior experience you had? Like you've worked with maps for a very long time. Were there particular difficulties you had where you said I just gotta have something that can be statically hosted? [00:07:50] Brandon: That's sort of exactly why I got into this. I've been working sort of in and around the map space for over a decade, and Protomaps is really like me trying to solve the same problem I've had over and over again in the past, just like once and forever right? Because like once this problem is solved, like I don't need to deal with it again in the future. So I've worked at a couple of different companies before, mostly as a contractor, for like a humanitarian nonprofit for a design company doing things like, web applications to visualize climate change. Or for even like museums, like digital signage for museums. And oftentimes they had some sort of data visualization component, but always sort of the challenge of how to like, store and also distribute like that data was something that there wasn't really great open source solutions. So just for map data, that's really what motivated that design for Protomaps. [00:08:55] Jeremy: And in those, those projects in the past, were those things where you had to run your own server, run your own database, things like that? [00:09:04] Brandon: Yeah. And oftentimes we did, we would spin up an EC2 instance, for maybe one client and then we would have to host this server serving map data forever. Maybe the client goes away, or I guess it's good for business if you can sign some sort of like long-term support for that client saying, Hey, you know, like we're done with a project, but you can pay us to maintain the EC2 server for the next 10 years. And that's attractive. but it's also sort of a pain, because usually what happens is if people are given the choice, like a developer between like either I can manage the server on EC2 or on Rackspace or Hetzner or whatever, or I can go pay a SaaS to do it. In most cases, businesses will choose to pay the SaaS. So that's really like what creates a sort of lock-in is this preference for like, so I have this choice between like running the server or paying the SaaS. Like businesses will almost always go and pay the SaaS. [00:10:05] Jeremy: Yeah. And in this case, you either find some kind of free hosting or low-cost hosting just to host your files and you upload the files and then you're good from there. You don't need to maintain anything. [00:10:18] Brandon: Exactly, and that's really the ideal use case. so I have some users these, climate science consulting agencies, and then they might have like a one-off project where they have to generate the data once, but instead of having to maintain this server for the lifetime of that project, they just have a file on S3 and like, who cares? If that costs a couple dollars a month to run, that's fine, but it's not like S3 is gonna be deprecated, like it's gonna be on an insecure version of Ubuntu or something. So that's really the ideal, set of constraints for using Protomaps. [00:10:58] Jeremy: Yeah. Something this also makes me think about is, is like the resilience of sites like remaining online, because I, interviewed, Kyle Drake, he runs Neocities, which is like a modern version of GeoCities. And if I remember correctly, he was mentioning how a lot of old websites from that time, if they were running a server backend, like they were running PHP or something like that, if you were to try to go to those sites, now they're like pretty much all dead because there needed to be someone dedicated to running a Linux server, making sure things were patched and so on and so forth. But for static sites, like the ones that used to be hosted on GeoCities, you can go to the internet archive or other websites and they were just files, right? You can bring 'em right back up, and if anybody just puts 'em on a web server, then you're good. They're still alive. Case study of news room preferring static hosting [00:11:53] Brandon: Yeah, exactly. One place that's kind of surprising but makes sense where this comes up, is for newspapers actually. Some of the users using Protomaps are the Washington Post. And the reason they use it, is not necessarily because they don't want to pay for a SaaS like Google, but because if they make an interactive story, they have to guarantee that it still works in a couple of years. And that's like a policy decision from like the editorial board, which is like, so you can't write an article if people can't view it in five years. But if your like interactive data story is reliant on a third party, API and that third party API becomes deprecated, or it changes the pricing or it, you know, it gets acquired, then your journalism story is not gonna work anymore. So I have seen really good uptake among local news rooms and even big ones to use things like Protomaps just because it makes sense for the requirements. Working on Protomaps as an open source project for five years [00:12:49] Jeremy: How long have you been working on Protomaps and the parts that it's made up of such as PMTiles? [00:12:58] Brandon: I've been working on it for about five years, maybe a little more than that. It's sort of my pandemic era project. But the PMTiles part, which is really the heart of it only came in about halfway. Why not make a SaaS? [00:13:13] Brandon: So honestly, like when I first started it, I thought it was gonna be another SaaS and then I looked at it and looked at what the environment was around it. And I'm like, uh, so I don't really think I wanna do that. [00:13:24] Jeremy: When, when you say you looked at the environment around it what do you mean? Why did you decide not to make it a SaaS? [00:13:31] Brandon: Because there already is a lot of SaaS out there. And I think the opportunity of making something that is unique in terms of those use cases, like I mentioned like newsrooms, was clear. Like it was clear that there was some other solution, that could be built that would fit these needs better while if it was a SaaS, there are plenty of those out there. And I don't necessarily think that they're well differentiated. A lot of them all use OpenStreetMap data. And it seems like they mainly compete on price. It's like who can build the best three column pricing model. And then once you do that, you need to build like billing and metrics and authentication and like those problems don't really interest me. So I think, although I acknowledge sort of the indie hacker ethos now is to build a SaaS product with a monthly subscription, that's something I very much chose not to do, even though it is for sure like the best way to build a business. [00:14:29] Jeremy: Yeah, I mean, I think a lot of people can appreciate that perspective because it's, it's almost like we have SaaS overload, right? Where you have so many little bills for your project where you're like, another $5 a month, another $10 a month, or if you're a business, right? Those, you add a bunch of zeros and at some point it's just how many of these are we gonna stack on here? [00:14:53] Brandon: Yeah. And honestly. So I really think like as programmers, we're not really like great at choosing how to spend money like a $10 SaaS. That's like nothing. You know? So I can go to Starbucks and I can buy a pumpkin spice latte, and that's like $10 basically now, right? And it's like I'm able to make that consumer choice in like an instant just to spend money on that. But then if you're like, oh, like spend $10 on a SaaS that somebody put a lot of work into, then you're like, oh, that's too expensive. I could just do it myself. So I'm someone that also subscribes to a lot of SaaS products. and I think for a lot of things it's a great fit. Many open source SaaS projects are not easy to self host [00:15:37] Brandon: But there's always this tension between an open source project that you might be able to run yourself and a SaaS. And I think a lot of projects are at different parts of the spectrum. But for Protomaps, it's very much like I'm trying to move maps to being it is something that is so easy to run yourself that anyone can do it. [00:16:00] Jeremy: Yeah, and I think you can really see it with, there's a few SaaS projects that are successful and they're open source, but then you go to look at the self-hosting instructions and it's either really difficult to find and you find it, and then the instructions maybe don't work, or it's really complicated. So I think doing the opposite with Protomaps. As a user, I'm sure we're all appreciative, but I wonder in terms of trying to make money, if that's difficult. [00:16:30] Brandon: No, for sure. It is not like a good way to make money because I think like the ideal situation for an open source project that is open that wants to make money is the product itself is fundamentally complicated to where people are scared to run it themselves. Like a good example I can think of is like Supabase. Supabase is sort of like a platform as a service based on Postgres. And if you wanted to run it yourself, well you need to run Postgres and you need to handle backups and authentication and logging, and that stuff all needs to work and be production ready. So I think a lot of people, like they don't trust themselves to run database backups correctly. 'cause if you get it wrong once, then you're kind of screwed. So I think that fundamental aspect of the product, like a database is something that is very, very ripe for being a SaaS while still being open source because it's fundamentally hard to run. Another one I can think of is like tailscale, which is, like a VPN that works end to end. That's something where, you know, it has this networking complexity where a lot of developers don't wanna deal with that. So they'd happily pay, for tailscale as a service. There is a lot of products or open source projects that eventually end up just changing to becoming like a hosted service. Businesses going from open source to closed or restricted licenses [00:17:58] Brandon: But then in that situation why would they keep it open source, right? Like, if it's easy to run yourself well, doesn't that sort of cannibalize their business model? And I think that's really the tension overall in these open source companies. So you saw it happen to things like Elasticsearch to things like Terraform where they eventually change the license to one that makes it difficult for other companies to compete with them. [00:18:23] Jeremy: Yeah, I mean there's been a number of cases like that. I mean, specifically within the mapping community, one I can think of was Mapbox's. They have Mapbox gl. Which was a JavaScript client to visualize maps and they moved from, I forget which license they picked, but they moved to a much more restrictive license. I wonder what your thoughts are on something that releases as open source, but then becomes something maybe a little more muddy. [00:18:55] Brandon: Yeah, I think it totally makes sense because if you look at their business and their funding, it seems like for Mapbox, I haven't used it in a while, but my understanding is like a lot of their business now is car companies and doing in dash navigation. And that is probably way better of a business than trying to serve like people making maps of toilets. And I think sort of the beauty of it is that, so Mapbox, the story is they had a JavaScript renderer called Mapbox GL JS. And they changed that to a source available license a couple years ago. And there's a fork of it that I'm sort of involved in called MapLibre GL. But I think the cool part is Mapbox paid employees for years, probably millions of dollars in total to work on this thing and just gave it away for free. Right? So everyone can benefit from that work they did. It's not like that code went away, like once they changed the license. Well, the old version has been forked. It's going its own way now. It's quite different than the new version of Mapbox, but I think it's extremely generous that they're able to pay people for years, you know, like a competitive salary and just give that away. [00:20:10] Jeremy: Yeah, so we should maybe look at it as, it was a gift while it was open source, and they've given it to the community and they're on continuing on their own path, but at least the community running Map Libre, they can run with it, right? It's not like it just disappeared. [00:20:29] Brandon: Yeah, exactly. And that is something that I use for Protomaps quite extensively. Like it's the primary way of showing maps on the web and I've been trying to like work on some enhancements to it to have like better internationalization for if you are in like South Asia like not show languages correctly. So I think it is being taken in a new direction. And I think like sort of the combination of Protomaps and MapLibre, it addresses a lot of use cases, like I mentioned earlier with like these like hobby projects, indie projects that are almost certainly not interesting to someone like Mapbox or Google as a business. But I'm happy to support as a small business myself. Financially supporting open source work (GitHub sponsors, closed source, contracts) [00:21:12] Jeremy: In my previous interview with Tom, one of the main things he mentioned was that creating a mapping business is incredibly difficult, and he said he probably wouldn't do it again. So in your case, you're building Protomaps, which you've admitted is easy to self-host. So there's not a whole lot of incentive for people to pay you. How is that working out for you? How are you supporting yourself? [00:21:40] Brandon: There's a couple of strategies that I've tried and oftentimes failed at. Just to go down the list, so I do have GitHub sponsors so I do have a hosted version of Protomaps you can use if you don't want to bother copying a big file around. But the way I do the billing for that is through GitHub sponsors. If you wanted to use this thing I provide, then just be a sponsor. And that definitely pays for itself, like the cost of running it. And that's great. GitHub sponsors is so easy to set up. It just removes you having to deal with Stripe or something. 'cause a lot of people, their credit card information is already in GitHub. GitHub sponsors I think is awesome if you want to like cover costs for a project. But I think very few people are able to make that work. A thing that's like a salary job level. It's sort of like Twitch streaming, you know, there's a handful of people that are full-time streamers and then you look down the list on Twitch and it's like a lot of people that have like 10 viewers. But some of the other things I've tried, I actually started out, publishing the base map as a closed source thing, where I would sell sort of like a data package instead of being a SaaS, I'd be like, here's a one-time download, of the premium data and you can buy it. And quite a few people bought it I just priced it at like $500 for this thing. And I thought that was an interesting experiment. The main reason it's interesting is because the people that it attracts to you in terms of like, they're curious about your products, are all people willing to pay money. While if you start out everything being open source, then the people that are gonna be try to do it are only the people that want to get something for free. So what I discovered is actually like once you transition that thing from closed source to open source, a lot of the people that used to pay you money will still keep paying you money because like, it wasn't necessarily that that closed source thing was why they wanted to pay. They just valued that thought you've put into it your expertise, for example. So I think that is one thing, that I tried at the beginning was just start out, closed source proprietary, then make it open source. That's interesting to people. Like if you release something as open source, if you go the other way, like people are really mad if you start out with something open source and then later on you're like, oh, it's some other license. Then people are like that's so rotten. But I think doing it the other way, I think is quite valuable in terms of being able to find an audience. [00:24:29] Jeremy: And when you said it was closed source and paid to open source, do you still sell those map exports? [00:24:39] Brandon: I don't right now. It's something that I might do in the future, you know, like have small customizations of the data that are available, uh, for a fee. still like the core OpenStreetMap based map that's like a hundred gigs you can just download. And that'll always just be like a free download just because that's already out there. All the source code to build it is open source. So even if I said, oh, you have to pay for it, then someone else can just do it right? So there's no real reason like to make that like some sort of like paywall thing. But I think like overall if the project is gonna survive in the long term it's important that I'd ideally like to be able to like grow like a team like have a small group of people that can dedicate the time to growing the project in the long term. But I'm still like trying to figure that out right now. [00:25:34] Jeremy: And when you mentioned that when you went from closed to open and people were still paying you, you don't sell a product anymore. What were they paying for? [00:25:45] Brandon: So I have some contracts with companies basically, like if they need a feature or they need a customization in this way then I am very open to those. And I sort of set it up to make it clear from the beginning that this is not just a free thing on GitHub, this is something that you could pay for if you need help with it, if you need support, if you wanted it. I'm also a little cagey about the word support because I think like it sounds a little bit too wishy-washy. Pretty much like if you need access to the developers of an open source project, I think that's something that businesses are willing to pay for. And I think like making that clear to potential users is a challenge. But I think that is one way that you might be able to make like a living out of open source. [00:26:35] Jeremy: And I think you said you'd been working on it for about five years. Has that mostly been full time? [00:26:42] Brandon: It's been on and off. it's sort of my pandemic era project. But I've spent a lot of time, most of my time working on the open source project at this point. So I have done some things that were more just like I'm doing a customization or like a private deployment for some client. But that's been a minority of the time. Yeah. [00:27:03] Jeremy: It's still impressive to have an open source project that is easy to self-host and yet is still able to support you working on it full time. I think a lot of people might make the assumption that there's nothing to sell if something is, is easy to use. But this sort of sounds like a counterpoint to that. [00:27:25] Brandon: I think I'd like it to be. So when you come back to the point of like, it being easy to self-host. Well, so again, like I think about it as like a primitive of the web. Like for example, if you wanted to start a business today as like hosted CSS files, you know, like where you upload your CSS and then you get developers to pay you a monthly subscription for how many times they fetched a CSS file. Well, I think most developers would be like, that's stupid because it's just an open specification, you just upload a static file. And really my goal is to make Protomaps the same way where it's obvious that there's not really some sort of lock-in or some sort of secret sauce in the server that does this thing. How PMTiles works and building a primitive of the web [00:28:16] Brandon: If you look at video for example, like a lot of the tech for how Protomaps and PMTiles works is based on parts of the HTTP spec that were made for video. And 20 years ago, if you wanted to host a video on the web, you had to have like a real player license or flash. So you had to go license some server software from real media or from macromedia so you could stream video to a browser plugin. But now in HTML you can just embed a video file. And no one's like, oh well I need to go pay for my video serving license. I mean, there is such a thing, like YouTube doesn't really use that for DRM reasons, but people just have the assumption that video is like a primitive on the web. So if we're able to make maps sort of that same way like a primitive on the web then there isn't really some obvious business or licensing model behind how that works. Just because it's a thing and it helps a lot of people do their jobs and people are happy using it. So why bother? [00:29:26] Jeremy: You mentioned that it a tech that was used for streaming video. What tech specifically is it? [00:29:34] Brandon: So it is byte range serving. So when you open a video file on the web, So let's say it's like a 100 megabyte video. You don't have to download the entire video before it starts playing. It streams parts out of the file based on like what frames... I mean, it's based on the frames in the video. So it can start streaming immediately because it's organized in a way to where the first few frames are at the beginning. And what PMTiles really is, is it's just like a video but in space instead of time. So it's organized in a way where these zoomed out views are at the beginning and the most zoomed in views are at the end. So when you're like panning or zooming in the map all you're really doing is fetching byte ranges out of that file the same way as a video. But it's organized in, this tiled way on a space filling curve. IIt's a little bit complicated how it works internally and I think it's kind of cool but that's sort of an like an implementation detail. [00:30:35] Jeremy: And to the person deploying it, it just looks like a single file. [00:30:40] Brandon: Exactly in the same way like an mp3 audio file is or like a JSON file is. [00:30:47] Jeremy: So with a video, I can sort of see how as someone seeks through the video, they start at the beginning and then they go to the middle if they wanna see the middle. For a map, as somebody scrolls around the map, are you seeking all over the file or is the way it's structured have a little less chaos? [00:31:09] Brandon: It's structured. And that's kind of the main technical challenge behind building PMTiles is you have to be sort of clever so you're not spraying the reads everywhere. So it uses something called a hilbert curve, which is a mathematical concept of a space filling curve. Where it's one continuous curve that essentially lets you break 2D space into 1D space. So if you've seen some maps of IP space, it uses this crazy looking curve that hits all the points in one continuous line. And that's the same concept behind PMTiles is if you're looking at one part of the world, you're sort of guaranteed that all of those parts you're looking at are quite close to each other and the data you have to transfer is quite minimal, compared to if you just had it at random. [00:32:02] Jeremy: How big do the files get? If I have a PMTiles of the entire world, what kind of size am I looking at? [00:32:10] Brandon: Right now, the default one I distribute is 128 gigabytes, so it's quite sizable, although you can slice parts out of it remotely. So if you just wanted. if you just wanted California or just wanted LA or just wanted only a couple of zoom levels, like from zero to 10 instead of zero to 15, there is a command line tool that's also called PMTiles that lets you do that. Issues with CDNs and range queries [00:32:35] Jeremy: And when you're working with files of this size, I mean, let's say I am working with a CDN in front of my application. I'm not typically accustomed to hosting something that's that large and something that's where you're seeking all over the file. is that, ever an issue or is that something that's just taken care of by the browser and, and taken care of by, by the hosts? [00:32:58] Brandon: That is an issue actually, so a lot of CDNs don't deal with it correctly. And my recommendation is there is a kind of proxy server or like a serverless proxy thing that I wrote. That runs on like cloudflare workers or on Docker that lets you proxy those range requests into a normal URL and then that is like a hundred percent CDN compatible. So I would say like a lot of the big commercial installations of this thing, they use that because it makes more practical sense. It's also faster. But the idea is that this solution sort of scales up and scales down. If you wanted to host just your city in like a 10 megabyte file, well you can just put that into GitHub pages and you don't have to worry about it. If you want to have a global map for your website that serves a ton of traffic then you probably want a little bit more sophisticated of a solution. It still does not require you to run a Linux server, but it might require (you) to use like Lambda or Lambda in conjunction with like a CDN. [00:34:09] Jeremy: Yeah. And that sort of ties into what you were saying at the beginning where if you can host on something like CloudFlare Workers or Lambda, there's less time you have to spend keeping these things running. [00:34:26] Brandon: Yeah, exactly. and I think also the Lambda or CloudFlare workers solution is not perfect. It's not as perfect as S3 or as just static files, but in my experience, it still is better at building something that lasts on the time span of years than being like I have a server that is on this Ubuntu version and in four years there's all these like security patches that are not being applied. So it's still sort of serverless, although not totally vendor neutral like S3. Customizing the map [00:35:03] Jeremy: We've mostly been talking about how you host the map itself, but for someone who's not familiar with these kind of tools, how would they be customizing the map? [00:35:15] Brandon: For customizing the map there is front end style customization and there's also data customization. So for the front end if you wanted to change the water from the shade of blue to another shade of blue there is a TypeScript API where you can customize it almost like a text editor color scheme. So if you're able to name a bunch of colors, well you can customize the map in that way you can change the fonts. And that's all done using MapLibre GL using a TypeScript API on top of that for customizing the data. So all the pipeline to generate this data from OpenStreetMap is open source. There is a Java program using a library called PlanetTiler which is awesome, which is this super fast multi-core way of building map tiles. And right now there isn't really great hooks to customize what data goes into that. But that's something that I do wanna work on. And finally, because the data comes from OpenStreetMap if you notice data that's missing or you wanted to correct data in OSM then you can go into osm.org. You can get involved in contributing the data to OSM and the Protomaps build is daily. So if you make a change, then within 24 hours you should see the new base map. Have that change. And of course for OSM your improvements would go into every OSM based project that is ingesting that data. So it's not a protomap specific thing. It's like this big shared data source, almost like Wikipedia. OpenStreetMap is a dataset and not a map [00:37:01] Jeremy: I think you were involved with OpenStreetMap to some extent. Can you speak a little bit to that for people who aren't familiar, what OpenStreetMap is? [00:37:11] Brandon: Right. So I've been using OSM as sort of like a tools developer for over a decade now. And one of the number one questions I get from developers about what is Protomaps is why wouldn't I just use OpenStreetMap? What's the distinction between Protomaps and OpenStreetMap? And it's sort of like this funny thing because even though OSM has map in the name it's not really a map in that you can't... In that it's mostly a data set and not a map. It does have a map that you can see that you can pan around to when you go to the website but the way that thing they show you on the website is built is not really that easily reproducible. It involves a lot of c++ software you have to run. But OpenStreetMap itself, the heart of it is almost like a big XML file that has all the data in the map and global. And it has tagged features for example. So you can go in and edit that. It has a web front end to change the data. It does not directly translate into making a map actually. Protomaps decides what shows at each zoom level [00:38:24] Brandon: So a lot of the pipeline, that Java program I mentioned for building this basemap for protomaps is doing things like you have to choose what data you show when you zoom out. You can't show all the data. For example when you're zoomed out and you're looking at all of a state like Colorado you don't see all the Chipotle when you're zoomed all the way out. That'd be weird, right? So you have to make some sort of decision in logic that says this data only shows up at this zoom level. And that's really what is the challenge in optimizing the size of that for the Protomaps map project. [00:39:03] Jeremy: Oh, so those decisions of what to show at different Zoom levels those are decisions made by you when you're creating the PMTiles file with Protomaps. [00:39:14] Brandon: Exactly. It's part of the base maps build pipeline. and those are honestly very subjective decisions. Who really decides when you're zoomed out should this hospital show up or should this museum show up nowadays in Google, I think it shows you ads. Like if someone pays for their car repair shop to show up when you're zoomed out like that that gets surfaced. But because there is no advertising auction in Protomaps that doesn't happen obviously. So we have to sort of make some reasonable choice. A lot of that right now in Protomaps actually comes from another open source project called Mapzen. So Mapzen was a company that went outta business a couple years ago. They did a lot of this work in designing which data shows up at which Zoom level and open sourced it. And then when they shut down, they transferred that code into the Linux Foundation. So it's this totally open source project, that like, again, sort of like Mapbox gl has this awesome legacy in that this company funded it for years for smart people to work on it and now it's just like a free thing you can use. So the logic in Protomaps is really based on mapzen. [00:40:33] Jeremy: And so the visualization of all this... I think I understand what you mean when people say oh, why not use OpenStreetMaps because it's not really clear it's hard to tell is this the tool that's visualizing the data? Is it the data itself? So in the case of using Protomaps, it sounds like Protomaps itself has all of the data from OpenStreetMap and then it has made all the decisions for you in terms of what to show at different Zoom levels and what things to have on the map at all. And then finally, you have to have a separate, UI layer and in this case, it sounds like the one that you recommend is the Map Libre library. [00:41:18] Brandon: Yeah, that's exactly right. For Protomaps, it has a portion or a subset of OSM data. It doesn't have all of it just because there's too much, like there's data in there. people have mapped out different bushes and I don't include that in Protomaps if you wanted to go in and edit like the Java code to add that you can. But really what Protomaps is positioned at is sort of a solution for developers that want to use OSM data to make a map on their app or their website. because OpenStreetMap itself is mostly a data set, it does not really go all the way to having an end-to-end solution. Financials and the idea of a project being complete [00:41:59] Jeremy: So I think it's great that somebody who wants to make a map, they have these tools available, whether it's from what was originally built by Mapbox, what's built by Open StreetMap now, the work you're doing with Protomaps. But I wonder one of the things that I talked about with Tom was he was saying he was trying to build this mapping business and based on the financials of what was coming in he was stressed, right? He was struggling a bit. And I wonder for you, you've been working on this open source project for five years. Do you have similar stressors or do you feel like I could keep going how things are now and I feel comfortable? [00:42:46] Brandon: So I wouldn't say I'm a hundred percent in one bucket or the other. I'm still seeing it play out. One thing, that I really respect in a lot of open source projects, which I'm not saying I'm gonna do for Protomaps is the idea that a project is like finished. I think that is amazing. If a software project can just be done it's sort of like a painting or a novel once you write, finish the last page, have it seen by the editor. I send it off to the press is you're done with a book. And I think one of the pains of software is so few of us can actually do that. And I don't know obviously people will say oh the map is never finished. That's more true of OSM, but I think like for Protomaps. One thing I'm thinking about is how to limit the scope to something that's quite narrow to where we could be feature complete on the core things in the near term timeframe. That means that it does not address a lot of things that people want. Like search, like if you go to Google Maps and you search for a restaurant, you will get some hits. that's like a geocoding issue. And I've already decided that's totally outta scope for Protomaps. So, in terms of trying to think about the future of this, I'm mostly looking for ways to cut scope if possible. There are some things like better tooling around being able to work with PMTiles that are on the roadmap. but for me, I am still enjoying working on the project. It's definitely growing. So I can see on NPM downloads I can see the growth curve of people using it and that's really cool. So I like hearing about when people are using it for cool projects. So it seems to still be going okay for now. [00:44:44] Jeremy: Yeah, that's an interesting perspective about how you were talking about projects being done. Because I think when people look at GitHub projects and they go like, oh, the last commit was X months ago. They go oh well this is dead right? But maybe that's the wrong framing. Maybe you can get a project to a point where it's like, oh, it's because it doesn't need to be updated. [00:45:07] Brandon: Exactly, yeah. Like I used to do a lot of c++ programming and the best part is when you see some LAPACK matrix math library from like 1995 that still works perfectly in c++ and you're like, this is awesome. This is the one I have to use. But if you're like trying to use some like React component library and it hasn't been updated in like a year, you're like, oh, that's a problem. So again, I think there's some middle ground between those that I'm trying to find. I do like for Protomaps, it's quite dependency light in terms of the number of hard dependencies I have in software. but I do still feel like there is a lot of work to be done in terms of project scope that needs to have stuff added. You mostly only hear about problems instead of people's wins [00:45:54] Jeremy: Having run it for this long. Do you have any thoughts on running an open source project in general? On dealing with issues or managing what to work on things like that? [00:46:07] Brandon: Yeah. So I have a lot. I think one thing people point out a lot is that especially because I don't have a direct relationship with a lot of the people using it a lot of times I don't even know that they're using it. Someone sent me a message saying hey, have you seen flickr.com, like the photo site? And I'm like, no. And I went to flickr.com/map and it has Protomaps for it. And I'm like, I had no idea. But that's cool, if they're able to use Protomaps for this giant photo sharing site that's awesome. But that also means I don't really hear about when people use it successfully because you just don't know, I guess they, NPM installed it and it works perfectly and you never hear about it. You only hear about people's negative experiences. You only hear about people that come and open GitHub issues saying this is totally broken, and why doesn't this thing exist? And I'm like, well, it's because there's an infinite amount of things that I want to do, but I have a finite amount of time and I just haven't gone into that yet. And that's honestly a lot of the things and people are like when is this thing gonna be done? So that's, that's honestly part of why I don't have a public roadmap because I want to avoid that sort of bickering about it. I would say that's one of my biggest frustrations with running an open source project is how it's self-selected to only hear the negative experiences with it. Be careful what PRs you accept [00:47:32] Brandon: 'cause you don't hear about those times where it works. I'd say another thing is it's changed my perspective on contributing to open source because I think when I was younger or before I had become a maintainer I would open a pull request on a project unprompted that has a hundred lines and I'd be like, Hey, just merge this thing. But I didn't realize when I was younger well if I just merge it and I disappear, then the maintainer is stuck with what I did forever. You know if I add some feature then that person that maintains the project has to do that indefinitely. And I think that's very asymmetrical and it's changed my perspective a lot on accepting open source contributions. I wanna have it be open to anyone to contribute. But there is some amount of back and forth where it's almost like the default answer for should I accept a PR is no by default because you're the one maintaining it. And do you understand the shape of that solution completely to where you're going to support it for years because the person that's contributing it is not bound to those same obligations that you are. And I think that's also one of the things where I have a lot of trepidation around open source is I used to think of it as a lot more bazaar-like in terms of anyone can just throw their thing in. But then that creates a lot of problems for the people who are expected out of social obligation to continue this thing indefinitely. [00:49:23] Jeremy: Yeah, I can totally see why that causes burnout with a lot of open source maintainers, because you probably to some extent maybe even feel some guilt right? You're like, well, somebody took the time to make this. But then like you said you have to spend a lot of time trying to figure out is this something I wanna maintain long term? And one wrong move and it's like, well, it's in here now. [00:49:53] Brandon: Exactly. To me, I think that is a very common failure mode for open source projects is they're too liberal in the things they accept. And that's a lot of why I was talking about how that choice of what features show up on the map was inherited from the MapZen projects. If I didn't have that then somebody could come in and say hey, you know, I want to show power lines on the map. And they open a PR for power lines and now everybody who's using Protomaps when they're like zoomed out they see power lines are like I didn't want that. So I think that's part of why a lot of open source projects eventually evolve into a plugin system is because there is this demand as the project grows for more and more features. But there is a limit in the maintainers. It's like the demand for features is exponential while the maintainer amount of time and effort is linear. Plugin systems might reduce need for PRs [00:50:56] Brandon: So maybe the solution to smash that exponential down to quadratic maybe is to add a plugin system. But I think that is one of the biggest tensions that only became obvious to me after working on this for a couple of years. [00:51:14] Jeremy: Is that something you're considering doing now? [00:51:18] Brandon: Is the plugin system? Yeah. I think for the data customization, I eventually wanted to have some sort of programmatic API to where you could declare a config file that says I want ski routes. It totally makes sense. The power lines example is maybe a little bit obscure but for example like a skiing app and you want to be able to show ski slopes when you're zoomed out well you're not gonna be able to get that from Mapbox or from Google because they have a one size fits all map that's not specialized to skiing or to golfing or to outdoors. But if you like, in theory, you could do this with Protomaps if you changed the Java code to show data at different zoom levels. And that is to me what makes the most sense for a plugin system and also makes the most product sense because it enables a lot of things you cannot do with the one size fits all map. [00:52:20] Jeremy: It might also increase the complexity of the implementation though, right? [00:52:25] Brandon: Yeah, exactly. So that's like. That's really where a lot of the terrifying thoughts come in, which is like once you create this like config file surface area, well what does that look like? Is that JSON? Is that TOML, is that some weird like everything eventually evolves into some scripting language right? Where you have logic inside of your templates and I honestly do not really know what that looks like right now. That feels like something in the medium term roadmap. [00:52:58] Jeremy: Yeah and then in terms of bug reports or issues, now it's not just your code it's this exponential combination of whatever people put into these config files. [00:53:09] Brandon: Exactly. Yeah. so again, like I really respect the projects that have done this well or that have done plugins well. I'm trying to think of some, I think obsidian has plugins, for example. And that seems to be one of the few solutions to try and satisfy the infinite desire for features with the limited amount of maintainer time. Time split between code vs triage vs talking to users [00:53:36] Jeremy: How would you say your time is split between working on the code versus issue and PR triage? [00:53:43] Brandon: Oh, it varies really. I think working on the code is like a minority of it. I think something that I actually enjoy is talking to people, talking to users, getting feedback on it. I go to quite a few conferences to talk to developers or people that are interested and figure out how to refine the message, how to make it clearer to people, like what this is for. And I would say maybe a plurality of my time is spent dealing with non-technical things that are neither code or GitHub issues. One thing I've been trying to do recently is talk to people that are not really in the mapping space. For example, people that work for newspapers like a lot of them are front end developers and if you ask them to run a Linux server they're like I have no idea. But that really is like one of the best target audiences for Protomaps. So I'd say a lot of the reality of running an open source project is a lot like a business is it has all the same challenges as a business in terms of you have to figure out what is the thing you're offering. You have to deal with people using it. You have to deal with feedback, you have to deal with managing emails and stuff. I don't think the payoff is anywhere near running a business or a startup that's backed by VC money is but it's definitely not the case that if you just want to code, you should start an open source project because I think a lot of the work for an opensource project has nothing to do with just writing the code. It is in my opinion as someone having done a VC backed business before, it is a lot more similar to running, a tech company than just putting some code on GitHub. Running a startup vs open source project [00:55:43] Jeremy: Well, since you've done both at a high level what did you like about running the company versus maintaining the open source project? [00:55:52] Brandon: So I have done some venture capital accelerator programs before and I think there is an element of hype and energy that you get from that that is self perpetuating. Your co-founder is gungho on like, yeah, we're gonna do this thing. And your investors are like, you guys are geniuses. You guys are gonna make a killing doing this thing. And the way it's framed is sort of obvious to everyone that it's like there's a much more traditional set of motivations behind that, that people understand while it's definitely not the case for running an open source project. Sometimes you just wake up and you're like what the hell is this thing for, it is this thing you spend a lot of time on. You don't even know who's using it. The people that use it and make a bunch of money off of it they know nothing about it. And you know, it's just like cool. And then you only hear from people that are complaining about it. And I think like that's honestly discouraging compared to the more clear energy and clearer motivation and vision behind how most people think about a company. But what I like about the open source project is just the lack of those constraints you know? Where you have a mandate that you need to have this many customers that are paying by this amount of time. There's that sort of pressure on delivering a business result instead of just making something that you're proud of that's simple to use and has like an elegant design. I think that's really a difference in motivation as well. Having control [00:57:50] Jeremy: Do you feel like you have more control? Like you mentioned how you've decided I'm not gonna make a public roadmap. I'm the sole developer. I get to decide what goes in. What doesn't. Do you feel like you have more control in your current position than you did running the startup? [00:58:10] Brandon: Definitely for sure. Like that agency is what I value the most. It is possible to go too far. Like, so I'm very wary of the BDFL title, which I think is how a lot of open source projects succeed. But I think there is some element of for a project to succeed there has to be somebody that makes those decisions. Sometimes those decisions will be wrong and then hopefully they can be rectified. But I think going back to what I was talking about with scope, I think the overall vision and the scope of the project is something that I am very opinionated about in that it should do these things. It shouldn't do these things. It should be easy to use for this audience. Is it gonna be appealing to this other audience? I don't know. And I think that is really one of the most important parts of that leadership role, is having the power to decide we're doing this, we're not doing this. I would hope other developers would be able to get on board if they're able to make good use of the project, if they use it for their company, if they use it for their business, if they just think the project is cool. So there are other contributors at this point and I want to get more involved. But I think being able to make those decisions to what I believe is going to be the best project is something that is very special about open source, that isn't necessarily true about running like a SaaS business. [00:59:50] Jeremy: I think that's a good spot to end it on, so if people want to learn more about Protomaps or they wanna see what you're up to, where should they head? [01:00:00] Brandon: So you can go to Protomaps.com, GitHub, or you can find me or Protomaps on bluesky or Mastodon. [01:00:09] Jeremy: All right, Brandon, thank you so much for chatting today. [01:00:12] Brandon: Great. Thank you very much.

In this episode of Code with Jason, host Jason Swett interviews Prarthana Shiva, a senior software engineer at NexHealth, who shares how her team is handling massive database scaling challenges. Prarthana explains their PostgreSQL database's growth to 24 terabytes (with projections to triple within a year) and details their innovative solutions including read replicas, Elasticsearch implementation, Redis caching, external write-ahead logs, and optimized vacuuming processes. The conversation also touches on Jason's own database challenges with his CI platform and concludes with Prarthana's upcoming presentation at Sin City Ruby 2025, where she'll discuss their transition from schema-based to row-based multi-tenancy for better scalability.Prarthana Shiva on LinkedInSin City Ruby

How to speed up GenAI? Find out how on this episode of Six Five On the Road at AWS re:Invent with host Keith Townsend and Elastic's Ken Exner, CPO, for a conversation on how Elastic is at the forefront of accelerating generative AI (GenAI) innovation. Fast track this ⤵️ Insights into the adoption of generative AI applications among Elastic's customer base and how Elastic facilitates the acceleration of Gen AI initiatives. Future directions for Elastic's product portfolio with the integration of AI and machine learning. Developer feedback on Elasticsearch's usage in GenAI projects and its prominence as the top vector database. The launch of Elastic Cloud Serverless and Elastic's commitment to balancing usability with flexibility for both developers and end-users. A reflection on Elastic's product developments in the past year and anticipations for innovations in 2025.  

At All Things Open in October, Anandhi Bumstead, AWS's director of software engineering, highlighted OpenSearch's journey and the advantages of the Linux Foundation's stewardship. OpenSearch, an open source data ingestion and analytics engine, was transferred by Amazon Web Services (AWS) to the Linux Foundation in September 2024, seeking neutral governance and broader community collaboration. Originally forked from Elasticsearch after a licensing change in 2021, OpenSearch has evolved into a versatile platform likened to a “Swiss Army knife” for its broad use cases, including observability, log and security analytics, alert detection, and semantic and hybrid search, particularly in generative AI applications.Despite criticism over slower indexing speeds compared to Elasticsearch, significant performance improvements have been made. The latest release, OpenSearch 2.17, delivers 6.5x faster query performance and a 25% indexing improvement due to segment replication. Future efforts aim to enhance indexing, search, storage, and vector capabilities while optimizing costs and efficiency. Contributions are welcomed via opensearch.org.Learn more from The New Stack about deploying applications on OpenSearchAWS Transfers OpenSearch to the Linux FoundationFrom Flashpoint to Foundation: OpenSearch's Path ClearsSemantic Search with Amazon OpenSearch Serverless and TitanJoin our community of newsletter subscribers to stay on top of the news and at the top of your game. 

Philippe Noël is Co-Founder & CEO of ParadeDB, the modern Elasticsearch alternative built on Postgres. They're purpose-built for heavy, real-time workloads and their open source project, also called paradedb, has over 6K stars on GitHub. ParadeDB has raised $2M from investors including General Catalyst & YC. In this episode, we dig into the benefits of connecting search directly to the database (ie. no ETL), the types of users / use cases that really benefit from ParadeDB (e-commerce, FinTech, etc.), the decision to focus on Postgres, making adoption super easy, Philippe's learnings as a second-time founder & more!

Nicholas Knize discusses optimizing geospatial indexing and hybrid search using advanced data structures within the Lucene framework at FOSS4G NA 2024. He emphasizes reducing cloud infrastructure waste and improving geospatial data processing efficiency. Highlights

Shay Banon, the creator of Elasticsearch, joins us to discuss pulling off a reverse rug pull. Yes, Elasticsearch is open source, again! We discuss the complexities surrounding open source licensing and what made Elastic change their license, the implications of trademark law, the personal and business impact of moving away from open source, and ultimately what made them hit rewind and return to open source.

Shay Banon, the creator of Elasticsearch, joins us to discuss pulling off a reverse rug pull. Yes, Elasticsearch is open source, again! We discuss the complexities surrounding open source licensing and what made Elastic change their license, the implications of trademark law, the personal and business impact of moving away from open source, and ultimately what made them hit rewind and return to open source.

Jerod & Adam share our Zulip first impressions, react to Elasticsearch going open source (again), discuss Christian Hollinger's blog post on why he still self-hosts & answer a listener question: how do we produce podcasts?

Jerod & Adam share our Zulip first impressions, react to Elasticsearch going open source (again), discuss Christian Hollinger's blog post on why he still self-hosts & answer a listener question: how do we produce podcasts?

Welcome to episode 274 of The Cloud Pod, where the forecast is always cloudy! Justin, Ryan and Matthew are your hosts this week as we explore the world of SnapShots, Maia, Open Source, and VMware – just to name a few of the topics. And stay tuned for an installment of our continuing Cloud Journey Series to explore ways to decrease tech debt, all this week on The Cloud Pod.   Titles we almost went with this week: The Cloud Pod in Parallel Cluster The Cloud Pod cringes at managing 1000 aws accounts The Cloud Pod welcomes Imagen 3 with less Wokeness The Cloud Pod wants to be instantly snapshotted The Cloud pod hates tech debt A big thanks to this week's sponsor: We're sponsorless! Want to get your brand, company, or service in front of a very enthusiastic group of cloud news seekers? You've come to the right place! Send us an email or hit us up on our slack channel for more info.  General News 00:32 Elasticsearch is Open Source, Again Shay Banon is pleased to call ElasticSearch and Kibana “open source” again.  He says everyone at Elastic is ecstatic to be open source again, it’s part of his and “Elastics DNA.”  They’re doing this by adding AGPL as another license option next to ELv2 and SSPL in the coming weeks.  They never stopped believing or behaving like an OSS company after they changed the license, but by being able to use the term open source and by using AGPL – an OSI approved license – removes any questions or fud people might have.  Shay says the change 3 years ago was because they had issues with AWS and the market confusion their offering was causing.  So, after trying all the other options, changing the license – all while knowing it would result in a fork with a different name – was the path they took.  While it was painful, they said it worked.  3 years later, Amazon is fully invested in their OpenSearch fork, the market confusion has mostly gone, and their partnership with AWS is stronger than ever. They are even being named partner of the year with AWS.  They want to “make life of our users as simple as possible,” so if you’re ok with the ELv2 or the SSPL, then you can keep using that license. They aren't removing anything, just giving you another option with AGPL. He calls out trolls and people who will pick at this announcement, so they are attempting to address the trolls in advance.  “Changing the license was a mistake, and Elastic now backtracks from it”. We removed a lot of market confusion when we changed our license 3 years ago. And because of our actions, a lot has changed. It's an entirely different landscape now. We aren't living in the past. We want to build a better future for our users. It's because we took action then, that we are in a position to take action now. “AGPL i

This week we dig back into home automation, we talk a bit about choosing cameras for a large camera system, and of course we answer your questions! -- During The Show -- 00:52 Intro Home automation Weekend of learning 03:48 Monitoring Remote Location (Cameras) - Rob Powerline adapters might work Ubiquiti Nano Beam Synology Surveillance Station (https://www.synology.com/en-global/surveillance) Frigate Do not put the NVR on the internet Privacy File server upload Home Assistant events 17:18 Camera Systems for Tribal Lands - William NDAA compliant cameras and NVRs ReoLink NVR banned ReoLink Cameras depends - bad idea NDAA compliant brands 360 Vision Technology (360 VTL) Avigilon Axis Communications BCD International Commend FLIR Geutebrück iryx JCI/Tyco Security Mobotix Pelco Rhombus Systems Seek Thermal Solink Vaion/Ava WatchGuard Main 3 NVR in use Exac Vision Avigilon Milestone NDAA conversation Noah's favorites Axis FLIR #### 25:09 Charlie Finds e-ink android - Charlie Boox Palma (https://shop.boox.com/products/palma) Why a camera? Nice for reading Lineage or Graphene will NOT work 27:57 ESPDevices for Light Switches - Avri Shelly's are ESP32 devices Devices can talk to each other 30:00 Beaming podcasts to Volumio and Roku - Tiny Pulse Audio Write in! 31:40 News Wire 4M Linux 46 - opensourcefeed.org (https://www.opensourcefeed.org/4mlinux-46-release/) Debain Bookwork 12.7 - debian.org (https://www.debian.org/News/2024/20240831) Porteus 1.6 - porteus.org (https://forum.porteus.org/viewtopic.php?t=11426) Rhino Linux 2nd Release - itsfoss.com (https://news.itsfoss.com/rhino-linux-2024-2-release/) GNU Screen 5 - theregister.com (https://www.theregister.com/2024/09/03/gnu_screen_5/) Wireshark 4.4 - wireshark.org (https://www.wireshark.org/docs/relnotes/wireshark-4.4.0) Bugzilla releases - bugzilla.org (https://www.bugzilla.org/blog/2024/09/03/release-of-bugzilla-5.2-5.0.4.1-and-4.4.14/) Armbian 24.8 - armbian.com (https://www.armbian.com/newsflash/armbian-24-8-yelt/) Elasticsearch and Kibana licensing - businesswire.com (https://www.businesswire.com/news/home/20240829537786/en/Elastic-Announces-Open-Source-License-for-Elasticsearch-and-Kibana-Source-Code) Xe2 Linux Support - wccftech.com (https://wccftech.com/intel-push-out-xe2-graphics-enablement-linux-6-12-kernel/) Cicada3301 - thehackernews.com (https://thehackernews.com/2024/09/new-rust-based-ransomware-cicada3301.html) New Phi-3.5 AI Models - infoq.com (https://www.infoq.com/news/2024/08/microsoft-phi-3-5/) Open-Source, EU AI Act Compliant LLMs - techzine.eu (https://www.techzine.eu/blogs/privacy-compliance/123863/aleph-alphas-open-source-llms-fully-comply-with-the-ai-act/) View on Why AI Models Should be Open and Free for All - businessinsider.com (https://www.businessinsider.com/anima-anandkumar-ai-climate-change-open-source-caltech-nvidia-2024-8) 33:53 Hoptodesk Comparison to Team Viewer Hoptodesk (https://www.hoptodesk.com/) Free & Open Source Cross platform E2E Encryption Can self host the server Wayland is not officially supported 38:05 EmuDeck ArsTechnica (https://arstechnica.com/gaming/2024/08/emudeck-machines-pack-popular-emulation-suite-in-linux-powered-plug-and-play-pc/) Seeking funding Already been doing this on the steamdeck For retro games Drawing unwanted attention Powered by Bazzite 41:05 Home Automation Zwave Great for nerds/tinkering Not for professional installs RadioRA 2 Licensed dedicated frequency Central planning Never had a failure Designed to be integrated Orbit Panels and Shelly Pro Line Game changer 100% reliable People don't want a wall of dimmers Seeed Studio mmWave Sensor (https://wiki.seeedstudio.com/mmwave_human_detection_kit/) I don't like WiFi for automation Steve's experience -- The Extra Credit Section -- For links to the articles and material referenced in this week's episode check out this week's page from our podcast dashboard! This Episode's Podcast Dashboard (http://podcast.asknoahshow.com/406) Phone Systems for Ask Noah provided by Voxtelesys (http://www.voxtelesys.com/asknoah) Join us in our dedicated chatroom #GeekLab:linuxdelta.com on Matrix (https://element.linuxdelta.com/#/room/#geeklab:linuxdelta.com) -- Stay In Touch -- Find all the resources for this show on the Ask Noah Dashboard Ask Noah Dashboard (http://www.asknoahshow.com) Need more help than a radio show can offer? Altispeed provides commercial IT services and they're excited to offer you a great deal for listening to the Ask Noah Show. Call today and ask about the discount for listeners of the Ask Noah Show! Altispeed Technologies (http://www.altispeed.com/) Contact Noah live [at] asknoahshow.com -- Twitter -- Noah - Kernellinux (https://twitter.com/kernellinux) Ask Noah Show (https://twitter.com/asknoahshow) Altispeed Technologies (https://twitter.com/altispeed)

The Cursor AI code editor raises $60 million, RedMonk's Rachel Stephens tries to determine if rug pulls are worth it, Caleb Porzio details how he made $1 million on GitHub Sponsors, Elastic founder Shay Banon announces that Elasticsearch is open source (again) & Tomas Stropus writes about the art of finishing.

The Cursor AI code editor raises $60 million, RedMonk's Rachel Stephens tries to determine if rug pulls are worth it, Caleb Porzio details how he made $1 million on GitHub Sponsors, Elastic founder Shay Banon announces that Elasticsearch is open source (again) & Tomas Stropus writes about the art of finishing.

The Cursor AI code editor raises $60 million, RedMonk's Rachel Stephens tries to determine if rug pulls are worth it, Caleb Porzio details how he made $1 million on GitHub Sponsors, Elastic founder Shay Banon announces that Elasticsearch is open source (again) & Tomas Stropus writes about the art of finishing.

Welcome to another episode of Category Visionaries — the show that explores GTM stories from tech's most innovative B2B founders. In today's episode, we're speaking with Robert Cowart, CEO & Co-Founder of ElastiFlow, a network performance and security analytics platform that's raised $8 Million in funding. Here are the most interesting points from our conversation: Network Dependency: Robert emphasizes the critical role of network infrastructure in today's world, impacting commerce, healthcare, entertainment, and social interactions. Genesis of ElastiFlow: The company started as an experiment to see how new data platforms like Elasticsearch could improve network observability, leading to a successful GitHub project. Community's Role: The initial success and growth of ElastiFlow were significantly boosted by a loyal community built around the GitHub project, highlighting the importance of community-led growth. Market Entry and Growth: ElastiFlow quickly transitioned from community support to paying customers, even before launching their beta product, showcasing the power of having a dedicated user base. Building a Marketing Strategy: Initially relying on inbound marketing, ElastiFlow has now invested in outbound sales and marketing, including paid ads and content creation, to increase brand awareness and drive growth. Future Vision: The company aims to continue enhancing network observability, adding more context to network traffic records, and ensuring comprehensive support for hybrid IT environments. //   Sponsors: Front Lines — We help B2B tech companies launch, manage, and grow podcasts that drive demand, awareness, and thought leadership. www.FrontLines.io The Global Talent Co. — We help tech startups find, vet, hire, pay, and retain amazing marketing talent that costs 50-70% less than the US & Europe.  www.GlobalTalent.co    

Welcome to episode 265 of the Cloud Pod Podcast – where the forecast is always cloudy! It's a full house this week – Matthew, Jonathan, Ryan and Justin are all here to bring you the latest in cloud news – including FOCUS features in AWS Billing, Magic Quadrants, and AWS Metis. Plus, we have an Andoid vs. Apple showdown in the Aftershow, so be sure to stay tuned for that!  Titles we almost went with this week: Tech reports show Gartner leads in the BS quadrant  Oracle adds cloud and legal expenses to their FinOps hub AWS Metis: Great chatbot, or Greek tragedy waiting to happen?  The cloud pod rocks Cargo Pants  A sonnet is written for FOCUSing on spend A big thanks to this week's sponsor: We're sponsorless! Want to reach a dedicated audience of cloud engineers? Send us an email, or hit us up on our Slack Channel and let's chat!  General News 01:40 Finops X Recently Justin attended FinOps in beautiful and sunny San Diego – and if you weren't there, you really should plan on attending next year. This year's topics included: Focus 1.0 State of Vendors Conference size – they will most likely outgrow this particular conference center, seeing as how they're either selling out or pretty close to it.  Coolest thing about the conference – on stage all the biggies – TOGETHER.  It's great to see them all together talking about how they're making Finops better, and introducing new things for Finops and not just saving them for their own conferences.  Next Year  – Is Oracle going to be on stage next year?  08:22 Justin – “The shift left of FinOps was a big topic. You know, how do we get visibility? How do we show people what things are going to cost? How do we make sure that, you know, people are aware of what they’re doing? And so I think, you know, it’s just a recognition that is important and just as important as security is your cost. And in some ways security is part of your cost story. Because if you bankrupt your company, that’s a pretty bad security situation.” 10:17 Introducing Managed OpenSearch: Gain Control of Your Cloud with Powerful Log Analysis  Listen. We don't really *care* about OpenSearch – but the reality is it's taking over the world. Nobody is doing ElasticSearch anymore.  Digital Ocean is launching Managed OpenSearch offering, a comprehensive solution designed for in depth log analysis, simplifying troubleshooting, and optimizing application performance.  With Digital ocean you can Pinpoint and analyze log data with ease, customize log retention, enhance security and can scale with your business and receive forwarded logs from multiple sources including Digital Ocean droplets, managed databases, etc.  Interested in pricing? You can find that here. Or, if you'd like to take a product tour, you can do that

У дванадцятому випуску подкасту 1-2-3 Techno поговорили з Дмитром Чаплинським про культуру передачі знань в стартапах та великих компаніях, пошук мотивації, правильне делегування та вихід за межі стандартних рішень.

У восьмий випуск подкасту 1-2-3 Techno до нас завітав Всеволод Соловйов, CTO та co-founder Prophy Science. Він розповів про «надійність» Elasticsearch, роботу над проєктом для Збройних Сил України та співпрацю маленької компанії з бюрократичною Єврокомісією.

In Elixir Wizards Office Hours Episode 2, "Discovery Discoveries," SmartLogic's Project Manager Alicia Brindisi and VP of Delivery Bri LaVorgna join Elixir Wizards Sundi Myint and Owen Bickford on an exploratory journey through the discovery phase of the software development lifecycle. This episode highlights how collaboration and communication transform the client-project team dynamic into a customized expedition. The goal of discovery is to reveal clear business goals, understand the end user, pinpoint key project objectives, and meticulously document the path forward in a Product Requirements Document (PRD). The discussion emphasizes the importance of fostering transparency, trust, and open communication. Through a mutual exchange of ideas, we are able to create the most tailored, efficient solutions that meet the client's current goals and their vision for the future. Key topics discussed in this episode: Mastering the art of tailored, collaborative discovery Navigating business landscapes and user experiences with empathy Sculpting project objectives and architectural blueprints Continuously capturing discoveries and refining documentation Striking the perfect balance between flexibility and structured processes Steering clear of scope creep while managing expectations Tapping into collective wisdom for ongoing discovery Building and sustaining a foundation of trust and transparency Links mentioned in this episode: https://smartlogic.io/ Follow SmartLogic on social media: https://twitter.com/smartlogic Contact Bri: bri@smartlogic.io What is a PRD? https://en.wikipedia.org/wiki/Productrequirementsdocument Special Guests: Alicia Brindisi and Bri LaVorgna.

In this episode of the Laravel Podcast, we talk about the recent announcement of hiring a new head of engineering at Laravel and the impact it will have on the future of Laravel. We also dive into the upcoming conferences and events, including Laracon EU, Laracon US, and Laracon India. Additionally, we talk about Typesense, a potential alternative to Meilisearch and Algolia for self-hosted search functionality.Taylor Otwell's Twitter - https://twitter.com/taylorotwellMatt Stauffer's Twitter - https://twitter.com/stauffermattLaravel Twitter - https://twitter.com/laravelphpLaravel Website - https://laravel.com/Tighten.co - https://tighten.com/VP/Head of Engineering at Laravel - https://frequent-pick-a8d.notion.site/VP-Head-of-Engineering-at-Laravel-149b566a670841f7a74b3e904e261693Laracon EU - https://laracon.eu/Laracon US - https://laracon.us/Laravel Herd - https://herd.laravel.com/Laravel 11 - https://laravel.com/docs/master/releasesLaravel Live Denmark -https://laravellive.dk/Laravel Live UK - https://laravellive.uk/Laracon India - https://laracon.in/Caleb Porzio Twitter - https://twitter.com/calebporzioLivewire: https://laravel-livewire.com/ThePrimeagen Twitter - https://twitter.com/ThePrimeagenThe Factory - https://www.thefactoryindeepellum.com/Eric Barnes Twitter - https://twitter.com/ericlbarnesJoe Dixon Twitter - https://twitter.com/_joedixonJames Brooks - https://twitter.com/jbrooksukFreek VAn der Herten Twitter - https://twitter.com/freekmurze?lang=enPeter Suhm Twitter - https://twitter.com/petersuhmMichele Hansen Twitter - https://twitter.com/mjwhansenLaracon AU Twitter - https://twitter.com/LaraconAULaravel Scout - https://laravel.com/docs/10.x/scoutTypesense - https://typesense.org/Algolia -https://algolia.com/Meilisearch - https://www.meilisearch.com/Elasticsearch - https://www.elastic.co/elasticsearchLaravel Sail - https://laravel.com/docs/10.x/sailLaravel Vapor - https://vapor.laravel.com/Early Vapor Tweet - https://x.com/taylorotwell/status/1748782542663131442?s=20Tailwind CSS - https://tailwindcss.com/-----Editing and transcription sponsored by Tighten.

DShield Sensor Log Collection with Elasticsearch https://isc.sans.edu/forums/diary/DShield%20Sensor%20Log%20Collection%20with%20Elasticsearch/30616/ Anydesk Breach https://anydesk.com/en/public-statement Leaky Vessels https://snyk.io/blog/leaky-vessels-docker-runc-container-breakout-vulnerabilities/

DShield Sensor Log Collection with Elasticsearch https://isc.sans.edu/forums/diary/DShield%20Sensor%20Log%20Collection%20with%20Elasticsearch/30616/ Anydesk Breach https://anydesk.com/en/public-statement Leaky Vessels https://snyk.io/blog/leaky-vessels-docker-runc-container-breakout-vulnerabilities/

Jake and Michael discuss all the latest Laravel releases, tutorials, and happenings in the community.Show linksCompare Algolia vs ElasticSearch vs Meilisearch vs TypesenseAspen - The ultimate free API testing tool for macOS with AI integration(02:02) - Laravel 10.41 - Conditional Job Chains, a Number::spell() Threshold, Configurable model:prune Path, and More (06:46) - Laravel 10.42 - Global Defaults for the HTTP Client, a Max Validation Rule for Passwords, and more (09:43) - Laravel Scout Adds Typesense, A Lightening-fast Open-source Search (12:59) - Laravel 11 Introduces the Dumpable Trait (14:46) - Eager Load Limit is Coming to Laravel 11 (18:12) - Dive into the Streamlined Directory Structure in Laravel 11 (23:14) - Meet Aspen: Speedier & Smarter API Testing, Outshining Postman and Insomnia (26:53) - Laravel Live UK (28:25) - Write Tabular Assertions with Pest and PHPUnit (30:39) - Create Beautiful Charts in Filament With the Apex Charts Plugin (31:50) - Generate Tailwind Utility Stylesheets on Demand with Curlwind (34:16) - Download Over 1,500 Google Fonts in Your Laravel Project (35:17) - Create Dynamic Discounts with Custom Conditions on Laravel With the Discountify Package (37:46) - Handling Bulk Imports in Filament