Podcasts about modern data stack

Marion Azoulai est Staff Data Scientist chez Astronomer, l'éditeur d'Astro, la plateforme managée d'Airflow qui est aussi l'un des principaux contributeurs de la solution open source.Créé en 2018, Astronomer a levé plus de 370 millions de dollars et accompagne des centaines d'organisations dans le monde, parmi lesquelles la Société Générale, Booking.com, Autodesk ou encore WeWork.Marion nous raconte comment ils ont lancé l'équipe Data & IA et comment ils ont construit leur stack autour d'Airflow.On aborde :

We're excited to have Databricks join us at AIEWF, among hundreds of the top companies in the AI Engineer ecosystem. LS subscribers can use their discount to get past the late bird pricing and access over $50k in sponsor offers! Everyone is still talking about Satya's Frontier Ecosystems post, but few have actually built a (now $175 billion) frontier ecosystem and cloud like our guests today.From open-sourcing the layer above coding agents to rethinking databases for the agent era, Databricks cofounders Matei Zaharia and Reynold Xin are pushing the company beyond the lakehouse into a full data-and-AI operating system. In this episode, Matei and Reynold join swyx at the 2026 Data + AI Summit to unpack Omnigent, LTAP, Lakebase, agent security, open formats, Mosaic, and why databases may matter more than ever once AI agents start doing real work.We go deep on Omnigent: Databricks' open-source meta-harness for combining, controlling, and sharing agents across Claude Code, Codex, Cursor, Pi, custom agents, and internal tools. Matei explains why coding agents and enterprise agents run into the same problems: portability, collaboration, session history, security, spend controls, and the need for a common API above every harness.Then Reynold walks through Databricks' database dream: why CDC is brittle enough to joke that it means “continuous data corruption,” why HTAP has been the holy grail of database engineering, and why Databricks thinks LTAP gets most of the benefits by unifying the storage layer instead of collapsing every query engine. We also cover Databricks' infrastructure scale, the culture behind rapid prototyping, the difference between tech and enterprise customers, Databricks vs Snowflake, whether vector databases should have ever existed, the Mosaic model strategy, Genie, AI Runtime, RL fine-tuning, and the thesis that traditional software gets rewritten once the data is in the right place and agents sit on top.Databricks began as a company for the big data era. The origination of Spark from the Berkeley AMPLab which eventually turned into the product Lakehouse convinced enterprises that they didn't need a separate data lake, warehouse, ML platform, and governance layer. They just needed one open foundation where all of their data could live and be reasoned over.Since then a lot has changed, but data has only become more important. Data is no longer something you keep track of and analyze ad hoc, it's the necessary context agents need in order to act. So the framing has shifted from “where do we put all of our data?” to “how do we expose the right slice of state, history, permissions, and business logic to an AI system at the exact moment it's doing work?”If frontier model performance becomes commoditized, the durable advantage then becomes the company-specific context around them: proprietary data, governed access, operational state, transaction logs, workflows, and feedback loops. Which makes Databricks positioned perfectly.Now coming fresh off the Data + AI Summit 2026, the company is moving just as fast to keep up, announcing Genie One, Omnigent, LTAP, and many more, indicating a central mission in its newer work: Databricks is trying to become the operating system for enterprise agents.Models are getting good enough, but agents are only useful if they have the right context, permissions, memory, state, cost controls, and access to live business data. Fundamentally it appears that significantly better model performance in production is a systems problem, one that data guys like us are remarkably well prepared to solve!We discuss:* Why Databricks built Omnigent as a meta-harness above existing AI agents* Why coding agents and custom enterprise agents need the same infrastructure* The common API for agent sessions, files, streams, tool calls, and cancellation* Why persistent sessions, cloud sandboxes, sharing, search, and collaboration matter* Why Databricks open-sourced Omnigent instead of keeping it proprietary* Databricks' internal agent usage, cloud sandboxes, and coding workflows* The scale of Databricks: 50–60 million virtual machines a day and exabytes before breakfast* Why agent security needs contextual and stateful policies* How an agent could read confidential docs, install a compromised npm package, and leak data* Why spend control matters when an agent can burn $500 reading logs* Startup opportunities around coding-agent analytics, quality, skills, and spend* LTAP, Lakebase, and why Databricks wants to rethink the database stack* OLTP vs OLAP, CDC, and why data pipelines break at 3 a.m.* Why HTAP has historically been the holy grail of database engineering* Why Databricks thinks LTAP is “HTAP done right”* How writing transactional data into column-oriented formats changes analytics* Why agents need live operational context from databases, not just telemetry* How Databricks prototypes strategic systems without endless process* Enterprise vs tech customers, governance, procurement, and DIY culture* The “second system syndrome” risk of rewriting a database engine* Building a database engine from a decade of traces and quadrillions of data points* Why vector databases should never have been a separate category* Why open formats and AI changed the race with Snowflake* The Mosaic story, DBRX, Genie, document parsing models, and specialized model training* Why model customization and RL fine-tuning may become mainstream* Why “get the data there, slap some agent on top” may rewrite traditional softwareMatei Zaharia* LinkedIn: https://www.linkedin.com/in/mateizaharia* X: https://x.com/matei_zahariaReynold Xin* LinkedIn: https://www.linkedin.com/in/rxin* X: https://x.com/rxinDatabricks* Website: https://www.databricks.com* X: https://x.com/databricksTimestamps00:00:00 Introduction00:02:22 Omnigent and the Agent Infrastructure Layer00:08:39 Agent Clouds, Common APIs, and Open Source00:16:52 Databricks Scale and Internal AI Workflows00:18:03 Agent Security, Governance, and Spend Controls00:27:34 LTAP and the Database Dream00:30:30 CDC, HTAP, and Why Data Pipelines Break00:34:05 Lakebase, Parquet, and Live Data for Agents00:36:47 Databricks' Culture of Fast Prototyping00:43:40 The Dream Engine and Rewriting the Database Stack00:51:02 Vector Databases, Query Engines, and LTAP00:52:36 Databricks vs Snowflake00:57:48 Mosaic, DBRX, Genie, and Specialized Models01:03:11 Context, AI Runtime, and RL Fine-Tuning01:06:15 Why Data + Agents May Rewrite Software01:07:09 Closing ThoughtsTranscriptIntroduction: Databricks, Data + AI Summit, and Founder DynamicsSwyx [00:00:00]: Matei and Reynold from Databricks, welcome to Latent Space.Reynold Xin [00:00:06]: Hey, thanks for having us.Swyx [00:00:07]: Yeah.Matei Zaharia [00:00:08]: Yeah, thanks so much.Swyx [00:00:09]: thanks for taking time out. You have your Databricks, Data AI Summit going on. You were just telling me how the first summit that you guys ran was just 50 peopleReynold Xin [00:00:17]: Yeah, it wasSwyx [00:00:17]: in BerkeleyReynold Xin [00:00:18]: little meetup at Berkeley, I thinkMatei Zaharia [00:00:19]: YeahReynold Xin [00:00:19]: put togetherMatei Zaharia [00:00:20]: We were doing these tutorials and, yeah, just teach people Spark.Swyx [00:00:23]: Yeah. obviously now it's like, I think like the headline number's like 100,000 people around the world, 30,000 in person.Swyx [00:00:30]: it's a crazyMatei Zaharia [00:00:31]: AmazingSwyx [00:00:31]: community. Well, I just saw the keynote.Swyx [00:00:35]: Ali's just. Did was it obvious or that back when that Ali would be, like, such a great, like, CEO? LikeReynold Xin [00:00:42]: OhSwyx [00:00:42]: such a great presenter?Reynold Xin [00:00:43]: What do you think?Matei Zaharia [00:00:44]: I think among our group of founders it was clear that, I think he'd be the best at this.Swyx [00:00:50]: Yeah.Matei Zaharia [00:00:50]: And yeah, it turned out great. And he's, he's ramped up on so many topics growing a company. He would just go in and, like, study it and, be talk to all the experts. Like, even if he can't hire the person, learn enough about, like, finance and sales and whatever it was, and, and go from there. Yeah.Swyx [00:01:09]: Yeah.Reynold Xin [00:01:10]: he's obviously very high IQ and a very high EQ, but it wasn't. Like, Ali today is quite different from Ali from, like 10 years ago. I think there's a lot of work that he put in to, get to this point.Swyx [00:01:20]: Yeah. no, to me the most appealing thing about him is that he's funny. And like, it, it's, it'Matei Zaharia [00:01:26]: It's true, yeahSwyx [00:01:26]: it's hard to make jokes about, data warehousesReynold Xin [00:01:30]: About serious topicsSwyx [00:01:31]: securityMatei Zaharia [00:01:32]: YeahSwyx [00:01:32]: what have you.Matei Zaharia [00:01:33]: Oh, yeah. That's for sure.Swyx [00:01:34]: Yeah. So you guys launched a whole bunch of things. I'll, I'll just name check briefly, the stuff because we're not gonna cover everything. Omnigentt, your baby. LTAP, your baby, your dream engine.Swyx [00:01:47]: we're also gonna cover Genie, cover CustomerLake, you acquired PantherMatei Zaharia [00:01:52]: YeahSwyx [00:01:52]: Open Sharing, and there's Unity AI Gateway. A lot of these, I think, like, are things that you would expect a Databricks to do. It's, it's like part of the roadmap. Everyone in your category has similar things. But I think, probably the two of you are leading the two most unique and differentiated initiativesOmnigent and the Agent Infrastructure LayerSwyx [00:02:09]: on, in the landscape. Maybe we'll start with, Omnigentt we'll, we'll, we'll, we'll go into it. I do think that a lot of people are exploring this meta harness concept.Matei Zaharia [00:02:21]: Yeah, totally.Swyx [00:02:21]: What led you to it?Matei Zaharia [00:02:22]: Yeah. There were a couple of, like, converging lines, which I think is a good sign that you need something new. So on the one hand, there's all the coding agent info internally. We have really great, dev infra team. they built something called Isaac, that's like a wrapper on Claude Code and Codex, and, lets you use them either on the web in, like, sandboxes or, just on your dev machine or on your laptop or whatever. And then, they were adding all kinds of stuff there. And we saw all the more advanced engineers like, were building their own workflows with tons of agents, and they were building their own UIs and stuff on top or even on top of that. And then the other one was, like, us building agents. We ship this, like, data science agent called Genie on the research team, which I lead. We also build a lot of internal ones for various things, and then we have all the customer ones. And all of them running into this thing of like, “Oh, I need to switch model and harness and so on,” every few months. Plus the agent is, like, completely useless if you can't share sessions with someone and have history and have search and all this, like, layer on top of it for collaboration. I thought a bit about it from both contexts and, at first people thought it was weird. They're like, “Why are you doing coding agents and custom agents in the same thing?” But I said it's, it's the same problems and, you just wanna build the stuff that lets you deliver the agent, maybe control it if you care about security, and, make it portable across things. And then we prototyped some things as experiments. We saw, yeah, we can make it work, and then we built that for real.Swyx [00:04:06]: I'm wondering if this let's call it architectureMatei Zaharia [00:04:11]: YeahSwyx [00:04:11]: maps to anything in your careers in the past. like I always think about how a lot of things just tie back to operating systems.Swyx [00:04:18]: A lot of operatingMatei Zaharia [00:04:19]: YeahSwyx [00:04:20]: systems tie back to databases,Matei Zaharia [00:04:21]: SoSwyx [00:04:21]: or the other way aroundMatei Zaharia [00:04:22]: so the thing, I do think it ties a lot to, like, network protocols, internet protocol. we alsoSwyx [00:04:29]: Communication between entities.Matei Zaharia [00:04:30]: Yeah. We did stuff with, like, data sharing also, which is probably, most viewers probably won't know unless they'Swyx [00:04:36]: Yeah, open protocol is the term.Matei Zaharia [00:04:37]: Yeah.Swyx [00:04:38]: Open sharing. Open sharing.Matei Zaharia [00:04:38]: Open sharing.Swyx [00:04:39]: Yes.Matei Zaharia [00:04:39]: Yeah. So it's like you have a company, you maintain some table, like let's say like a Walmart or something. They have like the, inventory and what's been sold in each store. And then you also have suppliers, and they would love to produce more things and ship them, like, exactly the moment you need them. So they would love, like, real-time access to your table. So instead of like sending emails around or Excel sheets or phone calls, why can't you share like a view of that table in real time with them? Then they query, they, join it with their data, and they decide what to send. So it's one of these things where you, like you might ask like today since we can vibe code anything so fast, why do we even need to design like protocols or APIs or software? Why can't you just vibe code things on demand? But for this type of interoperability where multiple parties that are moving at different speeds are building stuff and you still want some layer on top to coordinate, you do wanna design it and build it. So it reminds me of that, like agents talking to each other and, users talking to agents and tools.Agent Clouds, Cloud Sandboxes, and Keeping Sessions AliveSwyx [00:05:42]: Reynold, any other comments alternative viewpoints?Reynold Xin [00:05:46]: I think, by the way, we had a debate on exactly which set of benefits would, matter a lot, and I think around the time we decided to do this thing I was telling Matei, “Hey,” it just happened to be there's a particular week that I was coding nonstopSwyx [00:06:00]: from the moment I woke up to, like, the moment I went to bed, I was, like, looking at my Claude sessions, my Codex sessions. And one of the things that was particularly annoying was having to keep my laptop open.Swyx [00:06:12]: I was driving to a doctor's appointment, and I remember because I wanted to make sure the whole thing continues working.Matei Zaharia [00:06:18]: But by the way, it's so comforting to hear you say that because I'm like, “I don't know if I'm a clown and I'm doing this or like.”Swyx [00:06:25]: Yeah. Like honestly, I was driving and I was tethering my laptop to my phone.Matei Zaharia [00:06:29]: huh.Swyx [00:06:29]: Keeping it on the side. Whenever I hit a red light, I started looking at what's going on my laptop.Matei Zaharia [00:06:35]: Yeah.Swyx [00:06:35]: And I just felt that was ridiculous.Matei Zaharia [00:06:37]: Yeah.Swyx [00:06:37]: It felt like we went back to the dark agesMatei Zaharia [00:06:39]: YeahSwyx [00:06:40]: programming. the productivity you gain from all this coding age is amazing, but, yeah.Matei Zaharia [00:06:45]: Have you heard of cloud?Swyx [00:06:47]: Yeah.Swyx [00:06:48]: It was crazy to me.Matei Zaharia [00:06:49]: Oh, the thing you were working on was the sandboxes or was this before that?Swyx [00:06:52]: It was a sandbox.Matei Zaharia [00:06:53]: Okay.Swyx [00:06:54]: I was workMatei Zaharia [00:06:54]: So you were inSwyx [00:06:55]: So I was approaching from a very different angle. I wanted to, “Hey, we're gonna have cloud sandboxes that doesn't shut down. You can get one very quickly,” but not just for running agentic sessions.Matei Zaharia [00:07:06]: Yeah.Swyx [00:07:06]: It's also for running development. So I was personally building that week, and through building that, I ran into all these issues, and then I wroteMatei Zaharia [00:07:15]: YeahSwyx [00:07:15]: a document for Matei, it's like, “Here's my wish list of what the actual environment should do.” And I think he ended up almost implementingMatei Zaharia [00:07:22]: YeahSwyx [00:07:22]: every single one of them.Matei Zaharia [00:07:23]: Yeah, I remember Reynolds saying, ‘cause my first prototype of this had just chats with your agent and he said, “I have to be able to open a shell, like my own shell and like list files and like tail them and stuff.” SoSwyx [00:07:36]: So SSH into a mainframe.Matei Zaharia [00:07:37]: Yeah. it has that now.Swyx [00:07:39]: Tailing my log.Matei Zaharia [00:07:40]: Yeah.Matei Zaharia [00:07:41]: Yeah.Swyx [00:07:41]: And also another thing I think I asked was, I had. I still use cursor for the sole purpose of rendering markdown files.Matei Zaharia [00:07:48]: huh. Yes.Swyx [00:07:49]: So I said, “If you just give me a way to see my markdown files and renderMatei Zaharia [00:07:53]: YeahSwyx [00:07:53]: them properly, I don't need a separate tool anymore.”Matei Zaharia [00:07:55]: Yeah.Swyx [00:07:56]: And I think you also built that in.Matei Zaharia [00:07:57]: Yeah, we, yeah, we did that, yeah. Yeah, we had a lot of engineers building, their own vibe coding setup. But then the other thing they all said is like, “Hey, I built something that's amazing for me, but, like, no one else on the team can use it ‘cause I don't have a server to collaborate.” And this is why we tried to set up, Omnigent, so you can have a server and have the security, set up in there. So, like log in with Google or whatever and, like securely share stuff. which. And that's where we've seen a lot of other agents like hit things. Like people think they prototyped an awesome agent, but it's not allowed to connect to like some really important data or whatever because of the security team.Omnigent Architecture, Open Source, and Common APIsSwyx [00:08:38]: Yeah.Matei Zaharia [00:08:38]: So yeah.Swyx [00:08:39]: Yeah. At this point, so for those watching along on YouTube, we're gonna putting up a image of the structure here, and we can talk a little bit of the architecture. I think I just want to have people understand, ‘cause like when we're talking about software, it can be very abstract and like here is what we're talking about. You've worked out in open source this entire platform and there's a runner component and server component with a uniform API that you've, you've figured out. any other element and obviously you can plug in all this, persistence layers and compute layers. This is a whole cloud. It's an agent cloud.Matei Zaharia [00:09:12]: Yeah. It's, it's got these components to work with it. The, a lot of the action happens like on the machine where you deploy your agent too. So whatever you've got on there, you can run. But yeah, it's, I think it's the minimal thing you want to have hosted, like collaborative agents and to have that server. And one of the reasons we open sourced it is, anyone building agents, this gives them an app they can start with and customize, which we were seeing in Databricks too. Like someone would make a nice, agent app and then other teams would ask, “Oh, can I just use yours for my agent?”Swyx [00:09:45]: Yeah, I think we had like five or six different agentic frameworksMatei Zaharia [00:09:48]: YeahSwyx [00:09:48]: built by every different team. They do all do more or less the same thing. Yeah, you need to. people wanna take something that works in Forkit, and you might as well have something open source. Yeah, which also was another question, which is interesting for Databricks. Like what do you choose to open source? What do you choose to make it proprietary? It's in. this goes back to Spark, right?Matei Zaharia [00:10:05]: Yeah.Matei Zaharia [00:10:06]: One, so one of the reasons to open source something is if you think it's a layer that will there'll be some network effect, it'll benefit from many, people collaborating, on it. So, for example, with Spark, I don't know if when Spark came out, we also focused a lot on letting you have libraries on top. So like there used to be differentSwyx [00:10:28]: EcosystemMatei Zaharia [00:10:28]: distributed computing engines for like machine learning and graph computation. We said they should all be libraries that you can compose. And we made it super easy to add connectors to data sources too. And then we benefit because, we don't have the time to write like connectors to like, 1,000 like different databases and file formats, but we can just use the ones people make, and of course they benefit from joining, this thing. So that's like one of these as it. Another way to think about it is like imagine, we our thing wasn't open. We had some agent hosting thing, but it's not open and then there is an open one. if you're. Which one's gonna win in the long run? So like here, because there is this benefit from like people writing integrations, it'll be, it'll be that. And then there are other things that like you just can't, even deliver as open source that are things the company does. Like for example, how do you make sure you're like streaming, jobs or your Lakebase database doesn't like, lose all your data at night? Well, that requires an operational team that's gonna sit there. There's no way it has to be a service. So like we wanna make sure as a company we're really good at those infra services and then we're as open as we can in terms of like what you build on top.Swyx [00:11:42]: speaking from a benefits, I think we are already seeing pull requestsMatei Zaharia [00:11:45]: YeahSwyx [00:11:45]: of all kinds of ecosystem integration, even though it was only released on Saturday.Matei Zaharia [00:11:50]: Yeah, Saturday. Yeah. So someoneSwyx [00:11:51]: Let's see, let's see what's going on. Yeah, you can look at the merge ones. I asked Sam Nigon this morning aboutMatei Zaharia [00:11:59]: 400 merge already?Matei Zaharia [00:12:00]: Yeah. I think Recent quite, I would guess around half are not from our team. but for example, someone added support for running it on Kubernetesrnetes. people added, many cloud sandboxes, so this can launch a cloud sandbox and run your agent in there, which is great for sharing too, ‘cause it's not, like, on your laptop and someone's, like, running scary code on there. so yeah, many startups have put those in, and, we expect to see more of them. We also have more agent harnesses already. Cursor, CLI, and Antigravity also.The Modern Data Stack and the Emerging AI StackMatei Zaharia [00:12:34]: Yeah. That's all, beautiful. And I, I feel like the last time this happens, there was the rise of the modern data stack.Matei Zaharia [00:12:42]: I don't know if it's that useful. I'm, I'm curious in your postmortem.Matei Zaharia [00:12:46]: I think most peopleSwyx [00:12:47]: AgreeMatei Zaharia [00:12:47]: will agree that it is finally dead. but maybe this arises to a new modern AI stack that, like, does the same thing.Matei Zaharia [00:12:52]: I don't know.Reynold Xin [00:12:54]: I think the modern data stack was a pretty useful thing, probably even up until this day. I think what, maybe for the audience who don't understand the history, I think the modern data stack is effectively decomposed into you need a layer to ingest the data in, you need a layer to transform your data, and then all of this are run, and then you need a layer to maybe visualize your data. And all of this runs on some data warehouse, or later on, as we're doing data warehouse or lakehouse.Reynold Xin [00:13:21]: I think that concepts are all very powerful and very useful. They enable a lot of workloads. What people eventually run into is a question of unification and consolidation is, hey, do you really need to chop all this into different pieces and work with so many different vendors and platforms in order to get, like, a very simple visualization done, right? So I think, like, over time, everybody started realizing that customers are pushing us. We started, we can realize that, so we started building more and more capabilities and trying to consolidate. And at the end of the day now, customers don't have to worry about having me hook up five different systems in orderMatei Zaharia [00:13:55]: YeahReynold Xin [00:13:55]: produce a chart. But the. I think, honestly, something like this is probably happening, in how many different frameworks do you want to hook up together in order to produce, like do a very simple agent.Matei Zaharia [00:14:06]: Just to be clear, I would say the core of this is this common API on top of all the harnesses. So the API is like, you've got an agent session, and you can send in a message or, like, a file. That's what you can send in, and then you get out, these streams as it's streaming text or as it's doing tool calls. And, or the other thing you can send in is you can, like, tell it to cancel a turn. So that's the API. Now, the thing we did is we could get you that on top of, like, cloud code running in a terminal, Codex, Py, OpenAI SDK, all that stuff. We map them all to that same interface. So that is something that you'd have to maintain yourself if you built your own, like, agent orchestrator, and then whenever cloud changes its API, you gotta, tweak your thing or it's gonna lose some messages. So that's the thing that's valuable to maintain. Then on top of that, like, we built a few apps. I think we built a pretty cool UI and stuff, but that's, And we built a security and control piece, which I'm excited about. But it's that common interface, so we don't. We. That doesn't try to be a stack. And in fact, you could plug in your own UI on top of this, server. That, and that's one of the use cases we care a lot about, ‘cause we want to use this in our own products.Compute, Sandboxes, and Databricks ScaleSwyx [00:15:20]: Yeah. It should be everywhere.Matei Zaharia [00:15:22]: Yeah.Swyx [00:15:22]: I think one of those things that is really interesting to me is, like, well, first of all, I'll, I'll endeavor to do everything and not call it the modern AI stack because like it needs a different name.Matei Zaharia [00:15:32]: Yeah.Swyx [00:15:32]: But like, yes, like, so one of the first people that told me about compute, sandboxing was Nikita from Neon.Swyx [00:15:39]: Because a lot of people think about Neon as like, well, it's serverless Postgres with, like, the separation of compute and storage and, instant branching and all those things. But every database company is also a compute company.Matei Zaharia [00:15:51]: Yeah. Yeah.Swyx [00:15:52]: And so he was showing to me his whole, his sandboxing solution. I don't think he have ever launched it.Matei Zaharia [00:15:57]: So our sandbox solution, the reason we could build it so quickly was because we realized if you just take the actual Lakebase architectureSwyx [00:16:05]: YeahMatei Zaharia [00:16:05]: and remove the database from it, by the coming from NeonSwyx [00:16:08]: Exactly, rightMatei Zaharia [00:16:09]: you have this sandboxSwyx [00:16:09]: Every database company has it already, yeah.Matei Zaharia [00:16:11]: Now, there are some differences. For example, in the one to support this particular workflow, it's important to have local persistence,Swyx [00:16:19]: YeahMatei Zaharia [00:16:19]: because you want your state to persist. Your libraries, you don't have to install your library every time, right?Matei Zaharia [00:16:24]: whereas the Neon architecture, because of the separation of storage from compute, you don't need persistent local disk.Swyx [00:16:30]: Yeah.Matei Zaharia [00:16:30]: So there's some differences.Swyx [00:16:32]: Yeah.Matei Zaharia [00:16:32]: But the, at the end of the day, yeah, it's, Yeah, so this is when you run, like, a coding sandbox. Like, if I use it, yeah, we have the dev env internally at Databricks. There's, like, many, like, tens of gigabytes of data just for, like, all the source code and, like, artifacts and stuff that I built, and I want that to come back next time, so.Matei Zaharia [00:16:51]: Yeah.Matei Zaharia [00:16:51]: But yeah.Matei Zaharia [00:16:52]: Before the show, we was talking about some statistics that might be surprising at the adoption.Matei Zaharia [00:16:56]: It could be internal, it could be external, whatever comes to mind, just to impress people the scale this is happening.Swyx [00:17:02]: So we, on the analytics side, I think we launchedReynold Xin [00:17:06]: Maybe 50 or 60 million virtual machines a day across all three clouds, so we're one of the biggest compute orchestrators out there.Reynold Xin [00:17:13]: Stuff for sure for CPU compute.Swyx [00:17:14]: Yeah.Matei Zaharia [00:17:14]: Yeah.Reynold Xin [00:17:15]: the. And all of this process, I think exabytes of data, I joked about depending on which time zone you are, typically before you have breakfast, Databricks would have processed exabytes of data already on that day. and on Neon, it's pretty interesting, too. It's launching, I think, 13 million databasesSwyx [00:17:34]: YeahReynold Xin [00:17:34]: a day now.Swyx [00:17:35]: Yeah, to me that was, like, aReynold Xin [00:17:36]: And that's just likeSwyx [00:17:37]: Like, what do you mean?Matei Zaharia [00:17:38]: Yeah. And that's the point.Reynold Xin [00:17:40]: And a lot of those were thanks to agent- agents and branching experimentationSwyx [00:17:44]: YeahReynold Xin [00:17:44]: because we made it so easy and so quickly, and thanks a lot to Nikita's team, to launch databases. It's, the. So it's changing the way people use databases.Swyx [00:17:54]: Yeah. Okay, we're gonna go into more database talk in a bit, but I wanna make sure we close up anything on Omnigentt. you mentioned, you were excited about the securityOmnigent Security, Contextual Policies, and Spend ControlsSwyx [00:18:03]: control side.Matei Zaharia [00:18:04]: Yeah.Swyx [00:18:04]: a lot of companies are figuring that out right now, as well as the spend side.Matei Zaharia [00:18:08]: Yep.Swyx [00:18:09]: what have you found there?Matei Zaharia [00:18:11]: Yeah, so I spent quite a bit of time talking to internal users, developers, security team, managers, and also lots of customers, and there's a few things. Like, first of all, one thing, that immediately was. became obvious is for security, there's this tension between, like, usability and security. And, the way people do. Like, a lot of coding agents today have very basic things like you can tell me which tool patterns I'll allow or disallow or whatever. It's like yes or no. But that puts you in a very tough spot. So just as an example, like, should my agent be able to read, some confidential documents, or let's say, should it be able to install new packages from npm, which, maybe it's compromised. Yes or no? Like, maybe I wanna allow it. Should my agent be able to publish stuff to the company website? Well, if I'm using it to code on the website, yes. But should it be able to do both, so it can, like grab a confidential document and be prompt injected and leak it? Probably not. So the thing we decided we need is stateful or what we call contextual policies where you keep track of the state of that session. It's not like is it allowed to push to the marketing site or not, but, like, hey, if it did a risky thing, like it installed, a old package from npm, or it read, like, 1,000 confidential docs, then no. Then don't, don't do it. Otherwise, maybe it's okay. That's one example of, like, moving that trade-off so it's both more secure and more useful by having a more powerful engine, essentially. This requires tracking sessions. The other piece that was interesting there is, like, there are these very level events it's doing, and you want some libraries on top that parse them. Like, for example, we have a, MCP server on Google Drive internally. It's got 60 API calls. like, how do I know which of those, like, will share a document with stuff on the internet and which ones won't? It's, it's annoying. So we designed in Omnigentt the policy layer so that it's functions and you can have libraries. Like, someone can make something that maps the level events to high-level ones, and then you write a policy about the high-level things that came out. so and thatSwyx [00:20:25]: This is related to the Panther,Matei Zaharia [00:20:27]: Yeah, Panther is. will help with that. PantherSwyx [00:20:30]: YeahMatei Zaharia [00:20:30]: a similar idea on the event processing side, and it's Python-based versus a weird custom language. this is more, as in realSwyx [00:20:39]: I didn't even know we were good yeah.Matei Zaharia [00:20:41]: Those things are happening, yeah.Swyx [00:20:42]: Yeah.Matei Zaharia [00:20:42]: So yeah, but these are the cool things. I think the contextual or stateful part, and then the way it can be libraries, and that was another reason to make it open source because others will write libraries and, like, we and our customers can use them. And the final thing, because it's stateful, one of the states we track is how much you spent in that session. So I can. I've had, like, I ask an agent to debug something, and it spent $500 because it decided to read a lot of log files and burn a lot of tokens. but I can literally say, “Okay, launch a agent to do this and cap it to spending $5.” Like, ask me for permission if it needs more. And because we're counting that within that session, it'll pop up and tell me, “Okay, you spent five, $5. Do you wanna go on?”Reynold Xin [00:21:27]: So important context here. Matei spent the last five years, a lot of his time was architecting Unity Catalog at DatabricksMatei Zaharia [00:21:34]: YeahReynold Xin [00:21:34]: which is the governance layer for data.Matei Zaharia [00:21:35]: That's right, yeah.Reynold Xin [00:21:36]: And he's combining expertise at that layer together with all the AI governance he knows.Matei Zaharia [00:21:41]: Yeah.Swyx [00:21:41]: DoMatei Zaharia [00:21:41]: But I also spent a lot of time being annoyed by coding agents and getting prompts.Matei Zaharia [00:21:46]: And also as theReynold Xin [00:21:48]: All the aboveMatei Zaharia [00:21:48]: I don't want to end up on the front page as, like, I installed some weird npm package and leakedSwyx [00:21:53]: YeahMatei Zaharia [00:21:53]: all the code, so I'm especially paranoid. But also I have very little time, so I don't want to sit there approving, like, do you want to run a 20-line, bash script, yes or no? so that's why I spend a lot of time figuring out, like, how can I make it as safe as possible and not annoying?Swyx [00:22:10]: Yeah. Is safety and mmm, let's call it security a bigger concern than token maxing or token budgets? which one is, likeMatei Zaharia [00:22:19]: Oh, yeah, they're both there. I don't know. I guess it depends on the type of company you are. So I think, some companies, like, the budget is, limited and, they really care about thatSwyx [00:22:34]: you can be Uber and still be concerned?Matei Zaharia [00:22:36]: Yeah. Oh, yeah, totally. Yeah. If you haveReynold Xin [00:22:38]: for us, securityMatei Zaharia [00:22:39]: YeahReynold Xin [00:22:40]: super paramount.Matei Zaharia [00:22:40]: For us, security is absolutely critical as a, cloud provider. It's, it's the most important thing, and, token maxing, we're not so worried about it yet, but I've seen the Like, for example, I talked to some consulting companies. They have, like, 100,000 employees who are all coding for customers. If those each spend, like, an extra $1,000 a month, that's, that's not fun.Swyx [00:23:04]: YeahMatei Zaharia [00:23:04]: we have, like, only a few thousand engineers.Swyx [00:23:06]: What's the policy in Databricks? Is it just unlimited or what'Matei Zaharia [00:23:08]: It's, it's unlimited, but we do. we use our own product to, like, analyze the traces and stuff, and we have a team that'looking to optimize and to see if anyone's doing something weird. And, we had some really cool insights just from analyzing current traces, like whichSwyx [00:23:24]: YeahMatei Zaharia [00:23:25]: models are better at, say, Rust versus like TypeScript or whatever. So yeah, at least in our code base.Swyx [00:23:31]: Yeah. Amazing. Obviously, I have to ask the token question, obviously.Matei Zaharia [00:23:34]: Yeah.Swyx [00:23:34]: I think it'sReynold Xin [00:23:34]: YeahSwyx [00:23:34]: it's a key thing. But yes, security and control above that, and figuring out a sane layer there you can have some autonomy, but, not too much.Matei Zaharia [00:23:43]: Yeah. Yeah, and we wanna make it super easy. As a engineer, you should set a thing. So in Omnigentt, you can ask your agent, “Set a policy on yourself to do this.” So it can likeSwyx [00:23:52]: But if there's something I should be showingMatei Zaharia [00:23:53]: YeahSwyx [00:23:53]: I don't, I don't see it on the GitHub, but,Matei Zaharia [00:23:55]: Oh, yeahSwyx [00:23:56]: there's justMatei Zaharia [00:23:56]: Well, in the docs there's something.Swyx [00:23:57]: Yeah, this is it.Matei Zaharia [00:23:58]: You can look at it later.Swyx [00:23:59]: Okay. Yeah.Matei Zaharia [00:23:59]: Just look in the docsSwyx [00:24:00]: YeahMatei Zaharia [00:24:00]: contextual policies if you wanna see.Swyx [00:24:04]: I just like to point peopleMatei Zaharia [00:24:05]: look at the built-in policies.Swyx [00:24:06]: Yeah.Reynold Xin [00:24:06]: Yeah.Swyx [00:24:06]: If you want to, follow up on this is exactly where to look, right?Reynold Xin [00:24:10]: Yeah.Matei Zaharia [00:24:10]: Yeah. yeah, and the story of these is, like, I just wrote, like, I wrote a doc with like 10 ideas for things before as you were working on them. Well, that was, like, my wish list of things people asked, and I told the team, like, “Hey, can you do like at least five of these for the launch?” And then they just got back with all of them, so.Swyx [00:24:29]: Oh, wow.Matei Zaharia [00:24:29]: so you can come up with more, but them- some of them are just meant to be examples. really you can intercept, like, any event the agent is making, and you can then either block or force it to ask the user or, like, allow, and you can update state to keepSwyx [00:24:45]: YeahMatei Zaharia [00:24:45]: track stuff.Swyx [00:24:46]: Yeah, ‘cause ultimately you're, I think of you as, like, a systems designer.Swyx [00:24:50]: You let people plug in, right? That's the wholeMatei Zaharia [00:24:51]: YeahSwyx [00:24:52]: modus operandi of what you do.Matei Zaharia [00:24:53]: Yeah.Swyx [00:24:54]: It's likeMatei Zaharia [00:24:54]: And we care a lot about also composab- like, can someone else write a library that others use, whichSwyx [00:24:59]: YeahMatei Zaharia [00:24:59]: this is meant to.Reynold Xin [00:25:00]: There's also a batteries included philosophy hereMatei Zaharia [00:25:03]: YesReynold Xin [00:25:03]: probably very similar to how you did Spark, which is you could just start using.Swyx [00:25:06]: Yeah.Matei Zaharia [00:25:06]: Yeah, that's right. It has to be good out of the box at certain things, and then you can build your own things on top that, like, we don't wanna do. But in Spark, if you just wanna like, I don't know, like read a table or do, like, a aggregation, it should be awesome at that out of the box.Building on Omnigent: Contributions, Startups, and AnalyticsSwyx [00:25:23]: Yeah. People wanna catch up on Omnigentt, they should watch your keynote.Swyx [00:25:26]: they should go through the GitHub and the docs. If they wanted to contribute, or they want to build on this ecosystem what would you call out as the most high-leverage places get involved?Matei Zaharia [00:25:36]: Yeah, do get involved in the Discord and in GitHub. Our team is there, is monitoring, and, some of the things people ask for we just built ourselves. Some of them, we're, we're collaborating with them to build it. and also tell us, likeSwyx [00:25:49]: Yeah, they're gonna be veryMatei Zaharia [00:25:49]: how you would like to use it because I think especially for developers, like, everyone wants it to work their own way, and a really good developer tool, like you have to hear the feedback on all the ways and figure out the abstractions and how to let people customize. So we'd love to hear, like, if you think, “Hey, I, I don't want it to work this way,” tell us. We really just wanna get that compatibility layer across agents and then let you do stuff on top.Swyx [00:26:14]: Yeah. is there any, in terms of like the startup side, I'm, I'm a founder.Swyx [00:26:18]: I wantMatei Zaharia [00:26:18]: YeahSwyx [00:26:18]: I see an opportunity, I wanna get in front of you. What's your request for, like, a startup that, like, I wish someoneMatei Zaharia [00:26:23]: Oh, like you wanna integrate with us?Swyx [00:26:24]: someone was working on this.Matei Zaharia [00:26:26]: Oh, for a startup?Swyx [00:26:27]: Yeah.Swyx [00:26:28]: Like, your, you got your own startup. It's doing well.Matei Zaharia [00:26:30]: Yeah.Swyx [00:26:30]: But like, if you weren't working on your own startup, what is, like, obvious that you should You advise many startups too, obviously.Matei Zaharia [00:26:37]: I do think, just as a company with a lot of engineers, like anything that helps me make sense of how people are usingSwyx [00:26:46]: SpendMatei Zaharia [00:26:46]: coding agents and,Swyx [00:26:48]: Yeah. AnalyticsMatei Zaharia [00:26:48]: spend, but also quality or like you should write, you should add this skill, or you should write this thing, or your agents are really horrible at tasks involving this service, so I go spend time. That would be nice. yeah.Swyx [00:27:00]: Yeah. The closest I've found is, this team, GitAI.Matei Zaharia [00:27:03]: Oh, cool. Yeah.Swyx [00:27:04]: They started with, like, we will just do, code and human attribution, but they're building the analytics layer on top of that.Matei Zaharia [00:27:12]: Yeah.Swyx [00:27:12]: I do think, like, there are a bunch of, like, artificial analysis is obviously,Matei Zaharia [00:27:18]: Yeah, they have their benchmarksSwyx [00:27:18]: doing super wellMatei Zaharia [00:27:19]: YeahSwyx [00:27:19]: with their stuff. so there's, there will be people. I think this is like the domain of consultants first, but then peopleMatei Zaharia [00:27:26]: YeahSwyx [00:27:26]: will build software that, let's say, it's kinda like the management planeMatei Zaharia [00:27:29]: YeahSwyx [00:27:30]: for coding agents.Matei Zaharia [00:27:30]: Yeah, I think there'll be a lot of insights there. You have it in other areas.Swyx [00:27:34]: Okay. Well, and then the other, big thing is your dream engine.LTAP: Lake Transactional/Analytical ProcessingSwyx [00:27:39]: maybe you wanna tell the story of, LTAP.Reynold Xin [00:27:45]: So, and background with. I'm, I'm gonna make people listen to our Ankur Goyal episode where we talked about SingleStore, HTAPMatei Zaharia [00:27:52]: YeahReynold Xin [00:27:52]: and all that history.Matei Zaharia [00:27:52]: Yeah. The LTAP idea is pretty simple. so if people have heard of the, Ankur's, talk about HTAP, it's effectively the world of databases. Sorry, there's like maybe a lot of context needs to be injected here. The world of databasesSwyx [00:28:06]: I am happy to be the database podcast that I'm forcing people to, like, learn your databases, guys.Swyx [00:28:11]: You cannot vibe code with just markdown files.Reynold Xin [00:28:13]: Yeah.Swyx [00:28:13]: Like,Reynold Xin [00:28:14]: It's one of the most important fundamental systems technologies out there. But the world of database effectively split into roughly two halves. There's what we call OLTP databases, which are transactional, and think of your Postgres, your MySQL, your Oracle databases, and the other side is what we call analytics, and sometime might refer to term OLAP. And the difference is on OLTP, you typically have maybe run some transaction on some event that looks up at one specific row. We update that row, right? It's a very oriented data structure. And on analytics, you're trying to reason on the data. You're trying to compute, “Hey, what's my revenue per store? What's my. How's my website doing every day?” And then you, eventually want to probably end up running anal- machine learning on it to predict, “Hey, how will my maybe sales be going in the future?” they are so very different architecture, and everybody start with OLTP databases. Every app, when you become serious enough, that needs more than markdown files, you need to have a database. You want to lose your data, you want to have some transactional consistency. But once you want to reason on the data, if you only have like- A hundred rows, it's probably okay to run it on your Postgres or your own, your MySQL database. But once you have more data and want to run more complicated analysis, the very analysis might crush your Postgres database. So you start doing, getting data out of the OLTP databaseSwyx [00:29:35]: Replication.Reynold Xin [00:29:36]: Replicate them into the analytic systems and just startSwyx [00:29:39]: Yeah, which for people, Elasticsearch is, like, aReynold Xin [00:29:42]: Yeah. So some of them get into Elasticsearch for, like, blocked analysis. A lot of our customers obviously get into Databricks to run more sophisticated things.Swyx [00:29:51]: Yeah.Reynold Xin [00:29:51]: And there's this term called CDC, whichMatei Zaharia [00:29:54]: Change data captureReynold Xin [00:29:55]: change data capture. and what it does, it reads the binlog of the database, and if you don't understand what binlog is, it's fine. The, but it's a little delta of the data, and it reconstructs based on the delta, the state of the database, on the analytics side. But CDC is, like, a very painful thing. It's how standard in the industry, everybody uses it, but, it ends up being. I think many data engineers ends up being waken up at, like, 3:00 a.m, because there's some pipeline thing.Swyx [00:30:22]: my explanation is, like, Airbyte is like a, became a $5 billion company just doing CDC.Reynold Xin [00:30:27]: Yeah, exactly.Reynold Xin [00:30:28]: CDC is, like, a veryMatei Zaharia [00:30:30]: It's hard.Reynold Xin [00:30:30]: It's one of the most boring but one of the most fundamental operations, like, powering modern society.Matei Zaharia [00:30:37]: huh.Reynold Xin [00:30:37]: But it's so brittle that, we joke that it's, should be called continuous data corruption, because you might change your schema on your OLTP database, and then the CDC pipeline fails to handleSwyx [00:30:48]: YeahReynold Xin [00:30:48]: the schema change.Swyx [00:30:49]: Yeah.Reynold Xin [00:30:49]: And then everything goes out.Swyx [00:30:51]: And there's all sorts of tricks that you can do, like, you add in, like, some versioning or whatever, but yeah.Reynold Xin [00:30:55]: Yeah, but it's a very, in general, very complicated. Like, I think at my keynote, I asked the audience put up their hand if they love their CDC pipeline. Only, like, maybe two people put it up. So if single store, like, about maybe a decade ago, I think the industry had this idea, hey, what if I built a single database that can handle both workloads? Now I don't.Swyx [00:31:12]: Which, like, by the way, every database person ever has ever always dreamed about this.Reynold Xin [00:31:15]: Yes. Yes.Reynold Xin [00:31:16]: This is the holy grail of database engineering is why not build a single system that can do both of this? But it ends up just being a lot of compromises. one, I think one of the first issue is that, hey, each. they say Postgres has a massive ecosystem, right? You want to be using the tools that's built for Postgres. And Spark, for example, had a massive ecosystem. There's a lot of libraries you want to use. If you were to create now a new thing, you don't have a ecosystem. You tend to create a new, smaller proprietary API, and you're lacking both, and it's also very difficult to make it performance-wise to be, comparable on either side. So it ends up being sucking on both. And our whole idea of LTAP, it's obviously a wordplay on the term HTAP, is that we think this is HTAP done right. HTAP wants to build a single engine for both. We think you can get 99% of what you need by unifying the storage, and just have a single storage layer. And once you have the single storage layer, if your Postgres databases are writing data in a column-oriented format, everything analytics can just go read that data directly without any delay, right? There's no pipeline in between, so all the data will immediately be available for reasoning analytics. I think I was telling some customers earlier, hey, when we talked about this is gonna be super useful for agents, I at first didn't really believe in it myself, even though we wrote that positioning.Lakebase, Agents, and Live Operational DataMatei Zaharia [00:32:39]: Yeah.Reynold Xin [00:32:40]: But then last night I was having dinner with a Australian customer, and they told me, “Oh, hey, one of the big issue we have is we have all these logs from our services, and we see SLA dips and want to investigate. But then there's no way for those agents to even understand what's going on in the actual databases themselves. All we see is just, like, product telemetry of the database and the services.” It would make those agents 10 times more powerful if understand, for example, who's placing those orders, what is happening, what exactly are they doing. So now I'm sold on our own message.Swyx [00:33:13]: Yeah.Reynold Xin [00:33:14]: I think it's really. It gets you the almost all of the benefits of the HTAP holy grail, which is, hey, make the data available immediately for reasoning analyticsSwyx [00:33:26]: Yeah, I think,Reynold Xin [00:33:27]: without compromiseSwyx [00:33:28]: in the way that humans are generally intelligent and want to have the ability and access to query anythingReynold Xin [00:33:34]: YeahSwyx [00:33:35]: while they do the work, they also need history and need context.Swyx [00:33:38]: And, like, where else does they get context? That's it's an analytical workload.Reynold Xin [00:33:41]: Exactly.Matei Zaharia [00:33:42]: Yeah. Yeah. And I remember when we had incidents with our databases and engineers said, “Well, I can't just run a giant query on it to see what's going on because that's gonna bring down the database and hoard it even more.” Like, that's the stuff that this gets rid of, because you spin up a whole separate fleet of machines that's doing the analytics. You're not overloading, like, the main databaseReynold Xin [00:34:02]: RightMatei Zaharia [00:34:02]: that's still trying to serve stuff.Reynold Xin [00:34:04]: Yeah.Matei Zaharia [00:34:04]: Yeah.Why LTAP Works Now: Parquet, Postgres, and LakebaseSwyx [00:34:05]: So this has been a dream for a while. what had to get done in order to get to today? Like,Reynold Xin [00:34:11]: Yeah.Swyx [00:34:11]: I feel like, you have announced variants of this several times, but it wasn't as clear as LTAP.Reynold Xin [00:34:18]: Yeah.Swyx [00:34:18]: I think LTAP is like Like, okay, we've got it, guys.Matei Zaharia [00:34:21]: This thing, yeah.Reynold Xin [00:34:21]: I was talking to somebody at Meta, and then he was asking me, “Hey, what's the catch? Why is it possible now?” And I think the reality is we took a lot of time to work on the Lakebase architecture. obviously a lot of it came from the Neon team, which is a separation of storage from compute. And it turned out it was just a tiny little step away going from that to this LTAP idea, which is, hey, we just. in the Neon architecture and in Lakebase architecture, we're writing data in oriented format to the open data lake, but in there we're writing in Postgres pages. Ali and I were spending a lot of time debating, hey, can we just change that to write in column-oriented format? And we're just debating, and one day, one of our engineers who's, like, super smart came in, he's like, “Hey, I just prototyped it. It works.”Swyx [00:35:07]: Wait, it's, prototype what?Reynold Xin [00:35:09]: Prototype, instead of storing the data in the data lake in the oriented formatSwyx [00:35:15]: ColumnReynold Xin [00:35:15]: like Postgres pagesSwyx [00:35:15]: YeahReynold Xin [00:35:16]: write them in Parquet.Swyx [00:35:17]: Yeah.Reynold Xin [00:35:18]: and he just made the observation that, hey, our storage fleet has a lot of extra idle CPUs And we could use those CPUs to do the transcoding from row to column, where row is good for OLTP, but column is good for analytics. so let's do that transcoding at that time. And as a matter of fact, once you transcode the data compresses better. So from those services writing to, for example, S3 or other data lake, like object stores, you can write them faster ‘cause now they are now smaller.Matei Zaharia [00:35:49]: Yeah.Reynold Xin [00:35:49]: So there's no overhead, it's no compromise in performanceMatei Zaharia [00:35:52]: Some CPU overhead.Swyx [00:35:54]: Yeah, because,Matei Zaharia [00:35:55]: YeahSwyx [00:35:55]: we had extra CPUs anyway.Matei Zaharia [00:35:56]: We had that fleet anyway, yeah.Swyx [00:35:57]: so the debate ended. it's one of the classics of, tech, issue of a lot of debate, but then somebody went ahead and just tried to prototype it and it worked.Matei Zaharia [00:36:06]: But, like, something this strategicSwyx [00:36:07]: That's rightMatei Zaharia [00:36:07]: and important to the company, I expect there to be, like, a kickoff thing, like a design doc. Nothing like that.Swyx [00:36:13]: Nothing like that.Swyx [00:36:14]: He just. We were debating in many meetingsMatei Zaharia [00:36:17]: Yeah.Swyx [00:36:17]: and then we're just debating whether it's possible or not from first principle.Matei Zaharia [00:36:20]: YeahSwyx [00:36:20]: and then, somebody just did it.Matei Zaharia [00:36:23]: Yeah, if you set yourself up so people do that'll be great. And that happened a bit with Omnigentt too. I think if I just had a doc on, like, we can make these together, everyone would, would think, “Oh, what about this? What about this?” But then you. if you try it out, it helps. And then if you have real users and they bash it and, like, it's still working, or in this case, if you have the workload, what the workload looks like, you can just test the same pattern then.Databricks' Culture of Fast PrototypingSwyx [00:36:47]: Yeah.Matei Zaharia [00:36:47]: Yeah.Swyx [00:36:47]: Tech aside, which is very cool, this is, like, the most important thing, the culture of innovation, and you don't have to ask my permission, you don't have like, do a whole form- formal process, just do it?Matei Zaharia [00:36:59]: Well, especially these days, I think withSwyx [00:37:01]: YeahMatei Zaharia [00:37:01]: AI, it's easier to buildSwyx [00:37:02]: But so, likeMatei Zaharia [00:37:03]: a prototypeSwyx [00:37:03]: I think you are very I made a lot of suite of, like, large companies and, like, I think that at scale, things slow down, and I'm sure you felt it already, but somehow you have this core of people that, like, are exempt. How? I think we hire and we work with really good people, and that's a very important part of it, and empowering them, but also spending a lot of time, maybe us in the trenches matter a lot also.Matei Zaharia [00:37:28]: Yeah, I think, I think first, people can adapt to being in the larger company, so that helps. And we wanna make sure they know that they can try stuff and settle debates and have a lot of examples of how it was done before, or launch a thing in beta or whatever. and then the other thing I do think as a company, like despite the size, we don't launch that many, like, products. We try to keep it pretty coherent. That's, that was the whole, like, theory of the company, was like instead of having, like, 20 Amazon services you need to set up, like a analytics and machine learning stack, you just have one, and it's, like, the same API, the same semantics across all of them, the same copy of the data. So that requires, like, unification. And then we added one more thing at a time. Like, we added storage with Delta Lake. We didn't used to do any storage. Then we added SQL, we added, machine learning platform stuff. So, but yeah, don't, don't do too many, but do those things well and, that also helps, it helps keep it manageable.Reynold Xin [00:38:33]: Yeah. The other thing we encourage a lot is instead of building, boil the ocean for everything, let's figure out how do we do it incrementally, how do we do it very quickly. Like, many of our productsMatei Zaharia [00:38:43]: YeahReynold Xin [00:38:43]: they're built in the span of weeks, and then we go to, hey. Like, usually my first question to whoever team is building is who's the target customer? Who are you working with? Are you on a first-name basis with them? Are you texting with them? I think having that very tight loop,Matei Zaharia [00:38:59]: Can you bring up another launch that comes to mind when, in this thing? I just want to give examples.Reynold Xin [00:39:04]: Omnigentt itself happened that way.Reynold Xin [00:39:05]: Yeah.Matei Zaharia [00:39:06]: Who's the customer? That's a good oneReynold Xin [00:39:34]: storage layer we did. we had, our largest customer at the time said like, “Okay, I need some. I want something in the cloud ‘cause, I. if the rest of our network is compromised, like this thing needs to be separate to store and query the events.” And then, talked to us, he said, “Okay, this is the rate of events per second. This is, like, the freshness I want. Can you do it?” So that was, like, way larger than any workload we had, and we had our, engineer, working on that, Michael Armbrust, and he worked just to make this work. And once it worked for them, it worked for everyone else. Yeah. This was early in the company, probably like four years in or something.Matei Zaharia [00:40:24]: 20- 2018?Swyx [00:40:26]: Yeah, ‘17, ‘18.Matei Zaharia [00:40:28]: Few companiesSwyx [00:40:28]: Do you have other examples?Matei Zaharia [00:40:30]: there'Swyx [00:40:31]: Maybe you have othersMatei Zaharia [00:40:31]: yeah, Clean Room, which is how you share data in a way without sharingSwyx [00:40:35]: YeahMatei Zaharia [00:40:35]: underlying data, but you allow specific operations. Those were done effectively initially just for two customers. I think the industry has a sense of, hey, maybe if you overfit to, like, one or two customers, it's gonna be really bad for you. But I think the, downside of overfitting is much smaller than the upside itself. And if you try to be too ambitious and boil the ocean, it's a much bigger problem.Swyx [00:40:58]: Yeah. Yeah.Matei Zaharia [00:40:58]: ‘Cause you might end up having no customer.Swyx [00:41:00]: Yeah, that's more, that's the more likely outcome.Matei Zaharia [00:41:02]: Yeah.Tech Companies vs. EnterprisesSwyx [00:41:03]: than you can pivot from there. I do think there is such a thing as a bad customer that sometimes you should fire. Yeah.Matei Zaharia [00:41:08]: They could exist sometimes if you drive. well, one of the challenge I think we probably see, and maybe many AI, so newer generation companies are seeing is, so tech companies are very different from tech companies or traditional enterprises.Swyx [00:41:22]: Yeah.Matei Zaharia [00:41:22]: And, if you optimize everything just for tech companies, you might have various challengesSwyx [00:41:27]: OhMatei Zaharia [00:41:27]: scaling them outside of tech companies.Swyx [00:41:28]: Okay, what likeMatei Zaharia [00:41:30]: YeahSwyx [00:41:30]: what like top three differences that you always think about?Reynold Xin [00:41:33]: Governance is a big oneMatei Zaharia [00:41:34]: I think, yeah, a big one is like, yeah, security, data privacy, governance, all that stuff. So usually if you're building some kinda like B2B or developer tool, like your biggest market is gonna be enterprises, but it's just very different. A company that's existed for like, it's had some form of IT for like 30 years, they have so many legacy systems or they operate in a regulated space. whereas a startup or, even like a, like sorta more recent tech company, all the. everything is new and pristine. So yeah, it's just different, and if you've never worked with enterprises or been in one, you just won't know about it.Reynold Xin [00:42:13]: Yeah.Matei Zaharia [00:42:13]: Yeah.Reynold Xin [00:42:13]: And the procurement process is probably quite different. There's far more stakeholders.Matei Zaharia [00:42:17]: Yeah, that is one. Yeah.Matei Zaharia [00:42:18]: Another piece that's interesting is I think some tech companies, people, will say, “Oh, I can build that myself,” right? I'll just build that myself.Matei Zaharia [00:42:27]: So then you go,Reynold Xin [00:42:28]: I don't think people say that about Databricks, butMatei Zaharia [00:42:31]: yeah, it dependsReynold Xin [00:42:32]: They do.Matei Zaharia [00:42:32]: They do?Matei Zaharia [00:42:32]: Yeah, the. Yeah, and it depends on the teams and things. So, but, on the other hand, like many of the enterprises say, “I don't, I never wanna be in the business of building that.” Like, I don't want my, whatever, I'm a retailer or something, I never wannaReynold Xin [00:42:45]: Yeah, sell clothes,Matei Zaharia [00:42:46]: be down because like some weird like nerd like couldn't get streaming pipelines working.Matei Zaharia [00:42:51]: That is not what I'm doing.Reynold Xin [00:42:53]: Yeah.Reynold Xin [00:42:53]: Yeah. This makes them great customers, to be honest, right?Matei Zaharia [00:42:55]: Yeah. But you have to understand that it's hard without having worked there and stuff, like you may not appreciate.Reynold Xin [00:43:01]: Look, I think they're all great. don't get me wrong, they have different challenges. But the, many of the tech companies, for sure there's a lot, far more DIY.Matei Zaharia [00:43:10]: On the flip side, you have people who are. they're very much experts in their domain, like they're building airplanes, they're, designing medicines, whatever, and they just want to bridge the technology, where like they don't wanna learn, databases or whatever. As cool as we think it is, even as interesting as the average software engineer might think it is to read a little bit, like they just never wanna know. They just say, “I have a, giant like, matrix or whatever with my, clinical data, like how do I, how do I like cluster it or whatever?” So yeah.The Dream Engine and Rewriting the Database StackReynold Xin [00:43:40]: Yeah. That's true. Okay, so and then I wanted to build out the dream engine, vision. where does this all lead? So one of the thing we, realized maybe a couple years back is that every single database engine out there, especially on the analytics side, are a decade old. pretty much everything that have reasonable traction are about a decade old. And they all started targeting some very specific narrow use cases, and then over time it's become more and more successful. They have grown in their ambition, and then they try to support more and more use cases. But the fastest way to support those use cases tend to be hacked around the abstractions that were initially created, that were not for those use cases.Matei Zaharia [00:44:23]: Yeah.Reynold Xin [00:44:23]: And then, but you can support them more or less okay. And before it, after 10 years of organic evolution that way, it becomes a gigantic pile of s**t.Reynold Xin [00:44:31]: the. And, but that includes Databricks. And very few company or very few systems, I think, have the gut to say, let's go start from scratch. Let's go back to the drawing board and design, knowing everything we know today after a decade of workloads and probably billions in revenue, let's attempt to rewrite it from scratch and make sure it will work and it can support all of these use cases. So we started doing that, but it's a very ambitious project. by the way, you can search on Wikipedia, there's this thing called second system syndrome.Matei Zaharia [00:45:08]: Yeah, I know that. Yes.Reynold Xin [00:45:09]: Or second system effect.Matei Zaharia [00:45:11]: Every developer must know what a second syndrome is.Reynold Xin [00:45:12]: It's you built your first thing and it works out great, and the second one's bound to fail because you become too ambitious.Reynold Xin [00:45:19]: And then you ask so many requirements.Matei Zaharia [00:45:20]: Or like you think everythingReynold Xin [00:45:21]: YeahMatei Zaharia [00:45:21]: and then you're likeReynold Xin [00:45:22]: You justMatei Zaharia [00:45:22]: you're, “I'm gonna design the perfect system this time.”Reynold Xin [00:45:24]: Yeah. And it turned out it's not perfect, and then it start failing and you're too ambitious, never launch, and you get killed. The, and the engineering team that started this, they were brilliant. I think we hired some of the best database engineers, on the planet into Databricks, and they were brilliant. Thank God it's not their second system. Many of them have built more than two in the past.Matei Zaharia [00:45:44]: Ah, nice.Reynold Xin [00:45:45]: But they were still worried about this, hey, building a database engine from scratch, I think the conventional wisdom is gonna take like five years to mature. This would be a very long-term project. It could fail. I think one of the engineers jokingly said, “Hey, maybe we just call it Reynolds Stream Engine.” If we name after a founder, maybe we then may get canceled or killed. But I think they built something pretty remarkable. they went back to. They changed the way the database engines were built from a paradigm point of view. Usually when y

We have been building dashboards for 20 years. Now everyone is adding AI on top of them. But what if the real issue is the stack itself? That is where my conversation with Soham Mazumdar, Co-Founder and CEO, WisdomAI went at Gartner D&A on The Ravit Show!!!!WisdomAI calls what they are building “Agentic Analytics.” Not a chatbot on top of BI. Not a copilot that still depends on humans to interpret everything. We talked about what is fundamentally broken in today's analytics world:- Dashboards answer questions you already thought of- Executives need answers to questions they did not know to askSoham shared how enterprises are moving from static reporting to agents that reason across metrics, detect issues, and explain why something happened. The trust problem came up quickly. Most AI analytics tools look impressive in a demo. Very few hold up under real enterprise scrutiny.We also discussed a real customer story with Cisco and what changed after deploying WisdomAI. The shift was not just faster answers. It was decision confidence.Looking ahead, Soham believes analytics teams will not disappear. They will evolve into designers and supervisors of intelligent systems that operate continuously across the business.For enterprise leaders rethinking the future of BI, this was a forward-looking and very practical discussion.#data #ai #gartnerda #wisdomai #theravitshow

Jessica Matheron est CDO Europe chez Nestlé. Avant cela, elle était CDO chez Agicap, une scale-up française.On aborde :

Pour ce troisième épisode de la saison 4 de Morning Data Chat, Magalie Cordier-Jubet, Chief Data Officer de Oney, revient sur la transformation majeure engagée par l'entreprise : la sortie progressive d'un legacy SAS historique au profit d'une modern data stack construite autour de Snowflake.Elle explique les bénéfices attendus de cette migration ainsi que l'usage de l'IA pour accélérer la conversion des scripts, documenter le code et réaliser les tests de non-régression. Magalie détaille enfin l'organisation mise en place pour réussir ce changement à grande échelle : stratégie hybride, mobilisation de plusieurs centaines de data analysts, création de communautés, montée en compétences, et pilotage FinOps dans un modèle de facturation à l'usage.Hébergé par Ausha. Visitez ausha.co/politique-de-confidentialite pour plus d'informations.

Matthieu Rousseau est un expert en Data Engineering, sur le sujet de la Modern Data Stack et notamment sur dbt. Il a monté Modeo, l'agence spécialisée sur l'IA et le Data Engineering (dbt, Snowflake, Airflow, DLT, Databricks...).On aborde :

Christophe Blefari est le créateur de la newsletter data Blef.fr la plus connue en France. Il a été Head of Data, Head of Data Engineering et Staff Data Engineer dans des startups et des grands groupes et est selon moi l'un des plus grands experts data en France. Récemment, il a cofondé Nao, un éditeur de code à destination des équipes Data qui utilisent l'IA.On décrypte 4 tendances Data & IA de 2026.On aborde :

Louise est Data Team Lead au Ministère de la Transition Écologique. Après un parcours en startups, elle a rejoint le Ministère pour piloter un projet qui permet d'accélérer la transition écologique : avec son équipe, elle a construit des indicateurs pour aider les territoires à mesurer leurs progrès climatiques.On aborde :

This episode is a re-air of one of our most popular conversations from this year, featuring insights worth revisiting. Thank you for being part of the Data Stack community. Stay up to date with the latest episodes at datastackshow.com. This week on The Data Stack Show, John and Matt welcome Pedram Navid, Chief Dashboard Officer at Dagster Labs. During the conversation, Pedram shares his career evolution from consulting to his current role, where he oversees data, developer relations (DevRel), and marketing. The discussion delves into the synergies between DevRel and marketing, emphasizing the importance of understanding developers' learning preferences. Pedram explains data orchestration, highlighting its role in managing and automating data workflows. He also discusses Daxter's unique asset-based approach, which enhances visibility and control over data processes, catering to users from novices to experts, and so much more. Highlights from this week's conversation include:Pedram's Background and Journey in Data (0:47)Joining Dagster Labs (1:41)Synergies Between Teams (2:56)Developer Marketing Preferences (6:06)Bridging Technical Gaps (9:54)Understanding Data Orchestration (11:05)Dagster's Unique Features (16:07)The Future of Orchestration (18:09)Freeing Up Team Resources (20:30)Market Readiness of the Modern Data Stack (22:20)Career Journey into DevRel and Marketing (26:09)Understanding Technical Audiences (29:33)Building Trust Through Open Source (31:36)Understanding Vendor Lock-In (34:40)AI and Data Orchestration (36:11)Modern Data Stack Evolution (39:09)The Cost of AI Services (41:58)Differentiation Through Integration (44:13)Language and Frameworks in Orchestration (49:45)Future of Orchestration and Closing Thoughts (51:54)The Data Stack Show is a weekly podcast powered by RudderStack, the CDP for developers. Each week we'll talk to data engineers, analysts, and data scientists about their experience around building and maintaining data infrastructure, delivering data and data products, and driving better outcomes across their businesses with data.RudderStack helps businesses make the most out of their customer data while ensuring data privacy and security. To learn more about RudderStack visit rudderstack.com. Hosted by Simplecast, an AdsWizz company. See pcm.adswizz.com for information about our collection and use of personal data for advertising.

In this episode, Ciro Greco (Co-founder & CEO, Bauplan) joins me to discuss why the future of data infrastructure must be "Code-First" and how this philosophy accidentally created the perfect environment for AI Agents.We explore why the "Modern Data Stack" isn't ready for autonomous agents and why a programmable lakehouse is the solution. Ciro explains that while we trust agents to write code (because we can roll it back), allowing them to write data requires strict safety rails. He breaks down how Bauplan uses "Git for Data" semantics - branching, isolation, and transactionality - to provide an air-gapped sandbox where agents can safely operate without corrupting production data. Welcome to the future of the lakehouse.Bauplan: https://www.bauplanlabs.com/

“It's one thing to govern the data, but you also have to govern the usage of the data.”

It's all about acquisitions, acquisitions, acquisitions! Matt Housley joins me to tackle the biggest rumor in the data world this week: the potential acquisition of dbt Labs by Fivetran. This news sparks a wide-ranging discussion on the inevitable consolidation of the Modern Data Stack, a trend we predicted as the era of zero-interest-rate policy ended.We also talk about financial pressures, vendor exposure to the rise of AI, the future of data tooling, and more.

Grégoire Hornung est Head of Data chez Qover, une pépite belge de l'InsurTech qui a levé 70 millions d'euros et des beaux clients tels que Revolut, Qonto ou Mastercard.On aborde :

Alexis Ego est Solution Engineer chez Retool, l'outil low code massivement adopté par les équipes Data & IA aux US et en Europe. Ils ont déjà convaincu des géants comme OpenAI et Nvidia aux Etats-Unis ou Pernod Ricard et Decathlon en France.On aborde :

In this episode of AI + a16z, dbt Labs founder and CEO Tristan Handy sits down with a16z's Jennifer Li and Matt Bornstein to explore the next chapter of data engineering — from the rise (and plateau) of the modern data stack to the growing role of AI in analytics and data engineering. As they sum up the impact of AI on data workflows: The interesting question here is human-in-the-loop versus human-not-in-the-loop. AI isn't about replacing analysts — it's about enabling self-service across the company. But without a human to verify the result, that's a very scary thing.Among other specific topics, they also discuss how automation and tooling like SQL compilers are reshaping how engineers work with data; dbt's new Fusion Engine and what it means for developer workflows; and what to make of the spate of recent data-industry acquisitions and ambitious product launches.Follow everyone on X:Tristan HandyJennifer LiMatt Bornstein Check out everything a16z is doing with artificial intelligence here, including articles, projects, and more podcasts.

Dive into the future of data engineering with Sumit Gupta, Lead BI Engineer at Notion, as he shares insights with the bros on navigating the AI revolution in modern data stacks. From leveraging tools like Snowflake and dbt to automating content creation with AI, discover how traditional technical skills are evolving alongside the rise of AI. Whether you're a seasoned data professional or just starting your journey, learn why embracing AI isn't optional and how to balance technical expertise with crucial soft skills in this rapidly changing landscape. Get an insider's perspective on working at tech giants like Notion, Snowflake, and Dropbox, while exploring practical applications of AI in both professional and personal contexts.

Highlights from this week's conversation include:Background of ClickHouse (1:14)PostgreSQL Data Replication Tool (3:19)Emerging Technologies Observations (7:25)Observability and Market Dynamics (11:26)Product Development Challenges (12:39)Challenges with PostgreSQL Performance (15:30)Philosophy of Open Source (18:01)Open Source Advantages (22:56)Simplified Stack Vision (24:48)End-to-End Use Cases (28:13)Migration Strategies (30:21)Final Thoughts and Takeaways (33:29)The Data Stack Show is a weekly podcast powered by RudderStack, the CDP for developers. Each week we'll talk to data engineers, analysts, and data scientists about their experience around building and maintaining data infrastructure, delivering data and data products, and driving better outcomes across their businesses with data.RudderStack helps businesses make the most out of their customer data while ensuring data privacy and security. To learn more about RudderStack visit rudderstack.com.

Colin Zima is the cofounder and CEO of Omni, a data platform that combines the consistency of a shared data model with the speed and freedom of SQL. They recently raised their $69M Series B led by ICONIQ Growth. He was previously the Chief Analytics Officer at Looker.Colin's favorite book: Blink (Author: Malcolm Gladwell)(00:01) Introduction(01:10) What Is a Data Model and Why It Matters(03:27) Gaps in the Modern Data Stack(05:38) The Staying Power of SQL(07:29) Origin Story: Why Omni Was Created(10:13) Lessons from Building the MVP(12:48) Go-to-Market Insights: Zero to Ten Customers(16:02) Founder-Led Sales and Marketing Tactics(18:58) Company Building: Recruiting and Product Challenges(21:34) Product Positioning in a Crowded Market(23:26) Design Philosophy in Enterprise Software(28:21) Omni's Tech Stack and Development Strategy(28:57) Real-World Use of AI Inside the Company(31:01) Future of Data Tooling and Role of AI(33:49) Rapid Fire Round--------Where to find Colin Zima: LinkedIn: https://www.linkedin.com/in/colinzima/--------Where to find Prateek Joshi: Newsletter: https://prateekjoshi.substack.com Website: https://prateekj.com LinkedIn: https://www.linkedin.com/in/prateek-joshi-infiniteX: https://x.com/prateekvjoshi 

In this episode, we sit down with Sridhar Ramaswamy, CEO of Snowflake, for an in-depth conversation about the company's transformation from a cloud analytics platform into a comprehensive AI data cloud. Sridhar shares insights on Snowflake's shift toward open formats like Apache Iceberg and why monetizing storage was, in his view, a strategic misstep.We also dive into Snowflake's growing AI capabilities, including tools like Cortex Analyst and Cortex Search, and discuss how the company scaled AI deployments at an impressive pace. Sridhar reflects on lessons from his previous startup, Neeva, and offers candid thoughts on the search landscape, the future of BI tools, real-time analytics, and why partnering with OpenAI and Anthropic made more sense than building Snowflake's own foundation models.SnowflakeWebsite - https://www.snowflake.comX/Twitter - https://x.com/snowflakedbSridhar RamaswamyLinkedIn - https://www.linkedin.com/in/sridhar-ramaswamyX/Twitter - https://x.com/RamaswmySridharFIRSTMARKWebsite - https://firstmark.comX/Twitter - https://twitter.com/FirstMarkCapMatt Turck (Managing Director)LinkedIn - https://www.linkedin.com/in/turck/X/Twitter - https://twitter.com/mattturck(00:00) Intro and current market tumult(02:48) The evolution of Snowflake from IPO to Today(07:22) Why Snowflake's earliest adopters came from financial services(15:33) Resistance to change and the philosophical gap between structured data and AI(17:12) What is the AI Data Cloud?(23:15) Snowflake's AI agents: Cortex Search and Cortex Analyst(25:03) How did Sridhar's experience at Google and Neeva shape his product vision?(29:43) Was Neeva simply ahead of its time?(38:37) The Epiphany mafia(40:08) The current state of search and Google's conundrum(46:45) “There's no AI strategy without a data strategy”(56:49) Embracing Open Data Formats with Iceberg(01:01:45) The Modern Data Stack and the future of BI(01:08:22) The role of real-time data(01:11:44) Current state of enterprise AI: from PoCs to production(01:17:54) Building your own models vs. using foundation models(01:19:47) Deepseek and open source AI(01:21:17) Snowflake's 1M Minds program(01:21:51) Snowflake AI Hub

In this podcast episode, we talked with Adrian Brudaru about ​the past, present and future of data engineering.About the speaker:Adrian Brudaru studied economics in Romania but soon got bored with how creative the industry was, and chose to go instead for the more factual side. He ended up in Berlin at the age of 25 and started a role as a business analyst. At the age of 30, he had enough of startups and decided to join a corporation, but quickly found out that it did not provide the challenge he wanted.As going back to startups was not a desirable option either, he decided to postpone his decision by taking freelance work and has never looked back since. Five years later, he co-founded a company in the data space to try new things. This company is also looking to release open source tools to help democratize data engineering.0:00 Introduction to DataTalks.Club1:05 Discussing trends in data engineering with Adrian2:03 Adrian's background and journey into data engineering5:04 Growth and updates on Adrian's company, DLT Hub9:05 Challenges and specialization in data engineering today13:00 Opportunities for data engineers entering the field15:00 The "Modern Data Stack" and its evolution17:25 Emerging trends: AI integration and Iceberg technology27:40 DuckDB and the emergence of portable, cost-effective data stacks32:14 The rise and impact of dbt in data engineering34:08 Alternatives to dbt: SQLMesh and others35:25 Workflow orchestration tools: Airflow, Dagster, Prefect, and GitHub Actions37:20 Audience questions: Career focus in data roles and AI engineering overlaps39:00 The role of semantics in data and AI workflows41:11 Focusing on learning concepts over tools when entering the field 45:15 Transitioning from backend to data engineering: challenges and opportunities 47:48 Current state of the data engineering job market in Europe and beyond 49:05 Introduction to Apache Iceberg, Delta, and Hudi file formats 50:40 Suitability of these formats for batch and streaming workloads 52:29 Tools for streaming: Kafka, SQS, and related trends 58:07 Building AI agents and enabling intelligent data applications 59:09Closing discussion on the place of tools like DBT in the ecosystem

In this episode, Sean Zinsmeister from Google joins Mark Rittman to discuss the latest developments for Looker including the integration of Looker Studio, new modeling capabilities and the exciting potential of generative AI for BI.We discuss how Looker is evolving to be a more open, composable platform that can power advanced analytics and data storytelling, with Sean sharing insights on Google's purpose-built Gemini models for natural language to SQL translation and how Looker customers can leverage these AI capabilities. We also explore Looker's agentic API strategy, the long-term vision of using Looker Studio as the primary Looker front-end and the opening up of LookML to tools beyond just Looker. Driving Looker customer innovations in the generative AI eraPreviewing Studio in Looker, the (Eventual) Future of Self-Service Reporting for LookerLooker now available from Google Cloud consoleDelivering the third wave of BI in the AI era with LookerDrill to Detail Ep.100 Special ‘Past, Present and Future of the Modern Data Stack' with Special Guests Keenan Rice, Stewart Bryson and Jake SteinDrill to Detail Ep. 73 'Luck, Thinking Different and Designing Looker Data Platform' with Special Guest Colin Zima

Highlights from this week's conversation include:Pedram's Background and Journey in Data (0:47)Joining Dagster Labs (1:41)Synergies Between Teams (2:56)Developer Marketing Preferences (6:06)Bridging Technical Gaps (9:54)Understanding Data Orchestration (11:05)Dagster's Unique Features (16:07)The Future of Orchestration (18:09)Freeing Up Team Resources (20:30)Market Readiness of the Modern Data Stack (22:20)Career Journey into DevRel and Marketing (26:09)Understanding Technical Audiences (29:33)Building Trust Through Open Source (31:36)Understanding Vendor Lock-In (34:40)AI and Data Orchestration (36:11)Modern Data Stack Evolution (39:09)The Cost of AI Services (41:58)Differentiation Through Integration (44:13)Language and Frameworks in Orchestration (49:45)Future of Orchestration and Closing Thoughts (51:54)The Data Stack Show is a weekly podcast powered by RudderStack, the CDP for developers. Each week we'll talk to data engineers, analysts, and data scientists about their experience around building and maintaining data infrastructure, delivering data and data products, and driving better outcomes across their businesses with data.RudderStack helps businesses make the most out of their customer data while ensuring data privacy and security. To learn more about RudderStack visit rudderstack.com.

Join Mark Rittman in this special end-of-year episode as he speaks with Noel Gomez, co-founder of DataCoves about the challenges and opportunities of orchestrating dbt and other tools within the open-source Modern Data Stack, navigating the evolving semantic layer landscape and the future of modular, vendor-agnostic data solutions.Datacoves Platform OverviewBuild vs Buy Analytics Platform: Hosting Open-Source ToolsScale the benefits of Core with dbt CloudDagster vs. Airflow

This morning, a great article came across my feed that gave me PTSD, asking if Iceberg is the Hadoop of the Modern Data Stack? In this rant, I bring the discussion back to a central question you should ask with any hot technology - do you need it at all? Do you need a tool built for the top 1% of companies at a sufficient data scale? Or is a spreadsheet good enough? Link: https://blog.det.life/apache-iceberg-the-hadoop-of-the-modern-data-stack-c83f63a4ebb9

My guest today is Moham Aref, CEO of Relational AI, a company that recently closed a $75 million Series B funding round. Moham shares his incredible journey spanning over 30 years in the AI and machine learning space, offering invaluable insights for aspiring entrepreneurs and tech enthusiasts alike. Moham Aref, CEO of Relational AI, Moham brings a wealth of experience, having previously led LogicBlox and Predictix, and now spearheading Relational AI's mission to simplify intelligent application development. → Website: ⁠https://relational.ai/ → Linkedin: https://www.linkedin.com/in/molham/ Nataraj is the host & creator of Startup Project podcast, he is a full time product manager at Microsoft, early stage investor & advisor. → Linkedin: https://www.linkedin.com/in/natarajsindam/ → Twitter: https://x.com/natarajsindam → Email updates: ⁠https://startupproject.substack.com/⁠ → Website: ⁠⁠⁠https://thestartupproject.io⁠⁠⁠ Podcast Highlights: This episode covers a wide range of fascinating topics, from Moham's extensive career journey to the intricacies of the modern data stack and the transformative potential of Relational AI's technology. We unravel the complexities of descriptive, predictive, and prescriptive analytics, demystifying these crucial concepts for a broader audience. We also discuss the challenges of finding those first five customers in the B2B world, the strategic decision to build on Snowflake, and the potential for future competition and cannibalization by larger platforms. Moham thoughtfully shares his perspective on the current hype surrounding Generative AI and its practical applications in the enterprise space. We finish with advice on leadership, mentorship, and the overall challenges and rewards of a career in tech. Timestamps: 00:00 - Introduction and Guest Introduction 01:55 - Moham Aref's Career Journey and Transition to Relational AI 08:30 - Understanding Descriptive, Predictive, and Prescriptive Analytics 12:00 - Early Use Cases and Target Customers for Relational AI 17:30 - The Decision to Build on Snowflake: Strategy and Competition 22:15 - Securing the First Five Customers in the B2B World 27:40 - The Modern Data Stack and Relational AI's Place Within It 34:30 - Generative AI: Hype, Reality, and Enterprise Applications 40:00 - Leveraging Generative AI Internally and for Customer Value 45:00 - B2B Sales Strategies: Content, Relationships, and Customer Focus 51:30 - Relational AI's Future Plans and Growth Strategy 54:00 - Moham's Consumption Habits: Historical Insights and Mentorship 58:30 - Lessons Learned as a Founder and CEO Don't forget to like and subscribe for more insightful conversations about the world of AI! → YouTube: ⁠https://youtu.be/9-J4eV8qvZg⁠ → Spotify: ⁠https://open.spotify.com/episode/3Og8mbra1cokQ5cRJdjZn1?si=iqEOqKLLSqSbk8ehkniFqg⁠ → Apple podcasts: ⁠https://podcasts.apple.com/us/podcast/85-ai-should-not-be-regulated-author-ml-researcher/id1551300319?i=1000673806783⁠ → Email updates: ⁠https://startupproject.substack.com/⁠ → Others: ⁠https://spotifyanchor-web.app.link/e/qYaG6vhTRNb⁠#ModernDataStack #RelationalAI #AI #MachineLearning #DataAnalytics #PredictiveAnalytics #PrescriptiveAnalytics #GenerativeAI #Snowflake #B2B #Entrepreneurship #TechPodcast #DataManagement #BusinessIntelligence #CloudComputing #TechLeadership #CareerAdvice #Innovation #DataStrategy

A founding engineer on Google BigQuery and now at the helm of MotherDuck, Jordan Tigani challenges the decade-long dominance of Big Data and introduces a compelling alternative that could change how companies handle data. Jordan discusses why Big Data technologies are an overkill for most companies, how MotherDuck and DuckDB offer fast analytical queries, and lessons learned as a technical founder building his first startup. Watch the episode with Tomasz Tunguz: https://youtu.be/gU6dGmZzmvI Website - https://motherduck.com Twitter - https://x.com/motherduck Jordan Tigani LinkedIn - https://www.linkedin.com/in/jordantigani Twitter - https://x.com/jrdntgn FIRSTMARK Website - https://firstmark.com Twitter - https://twitter.com/FirstMarkCap Matt Turck (Managing Director) LinkedIn - https://www.linkedin.com/in/turck/ Twitter - https://twitter.com/mattturck (00:00) Intro (00:56) What is the Small Data? (06:56) Marketing strategy of MotherDuck (08:39) Processing Small Data with Big Data stack (15:30) DuckDB (17:21) Creation of DuckDB (18:48) Founding story of MotherDuck (24:08) MotherDuck's community (25:25) MotherDuck of today ($100M raised) (33:15) Why MotherDuck and DuckDB are so fast? (39:08) The limitations and the future of MotherDuck's platform (39:49) Small Models (42:37) Small Data and the Modern Data Stack (46:47) Making things simpler with a shift from Big Data to Small Data (50:04) Jordan Tigani's entrepreneurial journey (58:31) Outro

Lukas Schulte is Co-Founder and CEO of SDF Labs (Semantic Data Fabric), the data transformation layer and query engine platform. They're an open core company powered by the Apache Data Fusion query engine. SDF Labs has raised $9M from investors including RTP Global and Two Sigma Ventures. In this episode, we dig into the complications and pain points with the Modern Data Stack, shifting left with data (ie. moving more over to the client), competing with DBT by adding a query engine, why building in Rust was important, why their CLI is closed source, the importance of a strong partner strategy as a data company & more!

Mark Rittman is joined by returning guest David Jayatillake, VP of AI at Cube.dev, to talk about Delphi Labs' journey from a standalone data analytics chatbot to now becoming the basis of Cube's new AI features within its composable semantic model product.Drill to Detail Ep.102 'LLMs, Semantic Models and Bringing AI to the Modern Data Stack' with Special Guest David JayatillakeDrill to Detail Ep.107 'Cube, Headless BI and the AI Semantic Layer' with Special Guest Artyom KeydunovIntroducing the AI API and Chart Prototyping in Cube CloudA Practical Guide to Getting Started with Cube's AI APICube Rollup London : Bringing Cube Users Together

Benn Stancil's weekly Substack on data and technology provides a fascinating perspective on the modern data stack & the industry building it. On this episode, Benn joins Jerod to dissect a few of his essays, discuss opportunities he sees during this slowdown & discuss why he thinks maybe we should disband the analytics team.

Benn Stancil's weekly Substack on data and technology provides a fascinating perspective on the modern data stack & the industry building it. On this episode, Benn joins Jerod to dissect a few of his essays, discuss opportunities he sees during this slowdown & explain why he thinks maybe we should disband the analytics team.

Jake Yormak of Story Ventures joins Nate to discuss Why Hardware is Attractive, The Most Interesting Areas in AI Outside of GenAI, and the Modern Data Stack. In this episode we cover: Concentrating on Early-Stage Companies with Potential for Growth Investing in Hardware Companies, Challenges and Opportunities Focusing on Power Law Outliers AI Commoditization, Impact on Profit Pools, with a Focus on Computer Vision and Proprietary Data AI in Workflows, Incentivizing Users to Contribute Context Guest Links: LinkedIn X Story Ventures The hosts of The Full Ratchet are Nick Moran and Nate Pierotti of New Stack Ventures, a venture capital firm committed to investing in founders outside of the Bay Area. Want to keep up to date with The Full Ratchet? Follow us on social. You can learn more about New Stack Ventures by visiting our LinkedIn and Twitter. Are you a founder looking for your next investor? Visit our free tool VC-Rank and we'll send a list of potential investors right to your inbox!

In this episode, we sat down with Tomasz Tunguz (https://twitter.com/ttunguz), the founder of Theory Ventures and a leading voice in the tech investment space. We discussed the transformative potential of Ethereum as a database company, the importance of data security in a decentralized world, and the evolving landscape of AI technologies from foundational models to AI-native applications.

Safiyy Momen and I chat about the good and bad of the Modern Data Stack, controlling cloud costs, boring engineering, and much more. LinkedIn: https://www.linkedin.com/in/safiyy-momen/

Highlights from this week's conversation include:The Evolution of Data Systems (0:47)The Role of Open Source Software (2:39)Challenges of Time Series Data (6:38)Architecting InfluxDB (9:34)High Cardinality Concepts (11:36)Trade-Offs in Time Series Databases (15:35)High Cardinality Data (18:24)Evolution to InfluxDB 3.0 (21:06)Modern Data Stack (23:04)Evolution of Database Systems (29:48)InfluxDB Re-Architecture (33:14)Building an Analytic System with Data Fusion (37:33)Challenges of Mapping Time Series Data into Relational Model (44:55)Adoption and Future of Data Fusion (46:51)Externalized Joins and Technical Challenges (51:11)Exciting Opportunities in Data Tooling (55:20)Emergence of New Architectures (56:35)Final thoughts and takeaways (57:47)The Data Stack Show is a weekly podcast powered by RudderStack, the CDP for developers. Each week we'll talk to data engineers, analysts, and data scientists about their experience around building and maintaining data infrastructure, delivering data and data products, and driving better outcomes across their businesses with data.RudderStack helps businesses make the most out of their customer data while ensuring data privacy and security. To learn more about RudderStack visit rudderstack.com.

Modern Data Stack Hello, this is Hall T. Martin with the Startup Funding Espresso -- your daily shot of startup funding and investing. The modern data stack is the term for the tools used by tech companies to analyze and integrate data. It's cloud-based which alleviates many of the challenges in analyzing data with legacy systems. Here are the components of the modern data stack: Data sources -- this includes databases, company products that produce a stream of data, and event streams which log each action a user takes. Data warehouse -- these are the tools used to store the voluminous amounts of data that come from data analysis work. This includes data lakes and other large-scale formats for storing the data. Data analytics -- this includes the ability to query into the data sets and apply analytics to the data. Data transformation -- this moves the data into a format that end users can use for their own queries and analysis.  Data monitoring -- this captures metrics about the data such as how often the data is being used and for what applications. Data governance -- this monitors the use of the data to comply with government regulations. Data applications -- the set of applications which use the data output from the system for applications such as business intelligence.  In setting up a data analytics program at your company consider the modern data stack and its components.   Thank you for joining us for the Startup Funding Espresso where we help startups and investors connect for funding. Let's go startup something today. _______________________________________________________ For more episodes from Investor Connect, please visit the site at:   Check out our other podcasts here:   For Investors check out:   For Startups check out:   For eGuides check out:   For upcoming Events, check out    For Feedback please contact info@tencapital.group    Please , share, and leave a review. Music courtesy of .

Highlights from this week's conversation include:Chad's background and journey in data (0:46)Importance of Data Supply Chain (2:19)Challenges with Modern Data Stack (3:28)Comparing Data Supply Chain to Real-world Supply Chains (4:49)Overview of Gable.ai (8:05)Rethinking Data Catalogs (11:42)New Ideas for Managing Data (15:16)Data Discovery and Governance Challenges (18:51)Static Code Analysis and AI Impact on Data (24:55)Creating Contracts and Defining Data Lineage (27:31)Data Quality Issues and Upstream Problems (32:32)Challenges with Third-Party Vendors and External Data (34:29)Incentivizing Engineers for Data Quality (40:28)Feedback Loops and Actionability in Data Catalogs (45:30)Missing metadata (48:57)Role of AI in data semantics (50:27)Data as a product (54:26)Slowing down to go faster (57:38)Quantifying the cost of data changes (1:01:24)The Data Stack Show is a weekly podcast powered by RudderStack, the CDP for developers. Each week we'll talk to data engineers, analysts, and data scientists about their experience around building and maintaining data infrastructure, delivering data and data products, and driving better outcomes across their businesses with data.RudderStack helps businesses make the most out of their customer data while ensuring data privacy and security. To learn more about RudderStack visit rudderstack.com.

Investor Sabrina Wu hosts Bobsled Co-founder and CEO Jake Graham for the latest episode of Founded & Funded. Jake is revolutionizing data sharing across platforms, enabling customers to get to analysis faster directly in the platforms where they work. Madrona co-led Bobsled's $17 million series A last year, which put the company at an $87 million valuation. In this episode, Jake — who had stints at Neo4j, Intel, and Microsoft — provides his perspective on why enabling cross-cloud data sharing is often cumbersome yet so important in the age of AI. He also shares why you can't PLG the enterprise, how to convince customers to adopt new technologies in a post-zero interest rate environment, and what it takes to land and partner with the hyperscalers. Transcript here: https://www.madrona.com/bobsled-cross-cloud-data-sharing/ (00:00) Introduction (01:36) Why found a startup? (03:00) The Genesis of Bobsled: From Inspiration to Reality (05:26) Understanding Bobsled's Functionality: Cross-Cloud Data Sharing (09:48) The Role of Cross-Cloud Data Sharing in the Age of AI (13:05) Redefining the Modern Data Stack and Its Future (18:04) Navigating Enterprise Sales and Partnerships in Tech (23:22) Strategic Partnerships and Navigating the Hyperscaler Landscape (29:40) Leadership Lessons and Vision for Bobsled

Benn Stancil, cofounder and CTO at Mode, returns to The Analytics Engineering Podcast to discuss the evolution of the term "modern data stack" and its value today. Tristan wrote on this idea for The Analytics Engineering Roundup in Is the Modern Data Stack Still a Useful Idea? For full show notes and to read 6+ years of back issues of the podcast's companion newsletter, head to https://roundup.getdbt.com. The Analytics Engineering Podcast is sponsored by dbt Labs.

In this episode, we explore the dynamic world of modern analytics with Tristan Handy, CEO of dbt Labs (https://twitter.com/jthandy). DBT, which helps more than 30,000 enterprises ship trusted data products faster, has raised more than $400 million dollars, most recently at a $4B valuation.We discuss how dbt has revolutionized analytics engineering, enabling seamless data transformation and orchestration in the cloud. This innovation fosters greater collaboration among data teams and integrates software engineering principles into data analytics workflows.We also talk about dbt's Semantic Layer, a game-changer that streamlines data operations by standardizing key business metrics for consistent use across various analytical tools.In this conversation, we tackle pressing questions about the current state and future of data management and analytics. Is the "modern data stack" becoming obsolete? What's next for data engineering? And how is AI reshaping the analytics landscape?Tune in to discover our insights.

Joe Reis and Matt Housley are back for another listener Q&A. They chat about the demise of the Modern Data Stack, architecture, data modeling, AI, and much more.

My voice is sort of working, and I chat about Tristan Handy's article that raised quite a ruckus this week, "Is the "Modern Data Stack" Still a Useful Idea?" In the end, the Modern Data Stack won - people use the cloud for analytics. And everything ends, so I'm excited for what's next. Article: https://roundup.getdbt.com/p/is-the-modern-data-stack-still-a?r=oc02

In this bonus episode, Eric and Kostas preview their upcoming conversation with Artyom Keydunov of Cube Dev.

Summary The "modern data stack" promised a scalable, composable data platform that gave everyone the flexibility to use the best tools for every job. The reality was that it left data teams in the position of spending all of their engineering effort on integrating systems that weren't designed with compatible user experiences. The team at 5X understand the pain involved and the barriers to productivity and set out to solve it by pre-integrating the best tools from each layer of the stack. In this episode founder Tarush Aggarwal explains how the realities of the modern data stack are impacting data teams and the work that they are doing to accelerate time to value. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. You specify the customer traits, then Profiles runs the joins and computations for you to create complete customer profiles. Get all of the details and try the new product today at dataengineeringpodcast.com/rudderstack (https://www.dataengineeringpodcast.com/rudderstack) You shouldn't have to throw away the database to build with fast-changing data. You should be able to keep the familiarity of SQL and the proven architecture of cloud warehouses, but swap the decades-old batch computation model for an efficient incremental engine to get complex queries that are always up-to-date. With Materialize, you can! It's the only true SQL streaming database built from the ground up to meet the needs of modern data products. Whether it's real-time dashboarding and analytics, personalization and segmentation or automation and alerting, Materialize gives you the ability to work with fresh, correct, and scalable results — all in a familiar SQL interface. Go to dataengineeringpodcast.com/materialize (https://www.dataengineeringpodcast.com/materialize) today to get 2 weeks free! Data lakes are notoriously complex. For data engineers who battle to build and scale high quality data workflows on the data lake, Starburst powers petabyte-scale SQL analytics fast, at a fraction of the cost of traditional methods, so that you can meet all your data needs ranging from AI to data applications to complete analytics. Trusted by teams of all sizes, including Comcast and Doordash, Starburst is a data lake analytics platform that delivers the adaptability and flexibility a lakehouse ecosystem promises. And Starburst does all of this on an open architecture with first-class support for Apache Iceberg, Delta Lake and Hudi, so you always maintain ownership of your data. Want to see Starburst in action? Go to dataengineeringpodcast.com/starburst (https://www.dataengineeringpodcast.com/starburst) and get $500 in credits to try Starburst Galaxy today, the easiest and fastest way to get started using Trino. Your host is Tobias Macey and today I'm welcoming back Tarush Aggarwal to talk about what he and his team at 5x data are building to improve the user experience of the modern data stack. Interview Introduction How did you get involved in the area of data management? Can you describe what 5x is and the story behind it? We last spoke in March of 2022. What are the notable changes in the 5x business and product? What are the notable shifts in the data ecosystem that have influenced your adoption and product direction? What trends are you most focused on tracking as you plan the continued evolution of your offerings? What are the points of friction that teams run into when trying to build their data platform? Can you describe design of the system that you have built? What are the strategies that you rely on to support adaptability and speed of onboarding for new integrations? What are some of the types of edge cases that you have to deal with while integrating and operating the platform implementations that you design for your customers? What is your process for selection of vendors to support? How would you characterize your relationships with the vendors that you rely on? For customers who have pre-existing investment in a portion of the data stack, what is your process for engaging with them to understand how best to support their goals? What are the most interesting, innovative, or unexpected ways that you have seen 5XData used? What are the most interesting, unexpected, or challenging lessons that you have learned while working on 5XData? When is 5X the wrong choice? What do you have planned for the future of 5X? Contact Info LinkedIn (https://www.linkedin.com/in/tarushaggarwal/) @tarush (https://twitter.com/tarush) on Twitter Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today? Closing Announcements Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ (https://www.pythonpodcast.com) covers the Python language, its community, and the innovative ways it is being used. The Machine Learning Podcast (https://www.themachinelearningpodcast.com) helps you go from idea to production with machine learning. Visit the site (https://www.dataengineeringpodcast.com) to subscribe to the show, sign up for the mailing list, and read the show notes. If you've learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com (mailto:hosts@dataengineeringpodcast.com)) with your story. To help other people find the show please leave a review on Apple Podcasts (https://podcasts.apple.com/us/podcast/data-engineering-podcast/id1193040557) and tell your friends and co-workers Links 5X (https://5x.co) Informatica (https://www.informatica.com/) Snowflake (https://www.snowflake.com/en/) Podcast Episode (https://www.dataengineeringpodcast.com/snowflakedb-cloud-data-warehouse-episode-110/) Looker (https://cloud.google.com/looker/) Podcast Episode (https://www.dataengineeringpodcast.com/looker-with-daniel-mintz-episode-55/) DuckDB (https://duckdb.org/) Podcast Episode (https://www.dataengineeringpodcast.com/duckdb-in-process-olap-database-episode-270/) Redshift (https://aws.amazon.com/redshift/) Reverse ETL (https://medium.com/memory-leak/reverse-etl-a-primer-4e6694dcc7fb) Fivetran (https://www.fivetran.com/) Podcast Episode (https://www.dataengineeringpodcast.com/fivetran-data-replication-episode-93/) Rudderstack (https://www.rudderstack.com/) Podcast Episode (https://www.dataengineeringpodcast.com/rudderstack-open-source-customer-data-platform-episode-263/) Peak.ai (https://peak.ai/) The intro and outro music is from The Hug (http://freemusicarchive.org/music/The_Freak_Fandango_Orchestra/Love_death_and_a_drunken_monkey/04_-_The_Hug) by The Freak Fandango Orchestra (http://freemusicarchive.org/music/The_Freak_Fandango_Orchestra/) / CC BY-SA (http://creativecommons.org/licenses/by-sa/3.0/)

Model deployment, data warehouse options for running models, and how to best leverage BI tools: Harry Glaser and Jon Krohn discuss Modelbit's capabilities to automate ML models from notebooks into production-ready models, reducing the time and effort in ‘translating' information from one mode to another. Harry's conversation with host Jon Krohn expanded on the importance of automating this task, and how developments in ML modeling have widened access to entire teams to analyze data, whatever their level of expertise. This episode is brought to you by the AWS Insiders Podcast (https://pod.link/1608453414). Interested in sponsoring a SuperDataScience Podcast episode? Visit JonKrohn.com/podcast for sponsorship information. In this episode you will learn: • What the modern data stack is [03:28] • Version control for data scientists [13:30] • CI/CD, load balancing and logging [20:38] • Snowflake vs. Redshift [30:10] • How tools like Looker and Tableau help monitor models [35:26] Additional materials: www.superdatascience.com/699

No episódio de hoje, Luan Moreno e Mateus Oliveira conversam com Matheus Willian, atualmente Head de Engenharia de Dados na One Way Solution.dbt é uma das tecnologias mais faladas e utilizadas fora do país, possibilitando aos times de todos os portes trabalhar com o conceito de Modern Data Stack, tornando o desenvolvimento de transformações dos dados de forma simples e com SQL.Com dbt, você tem os seguintes benefícios:Desenvolvimento de pipeline de dados usando SQL;Reutilização dos códigos usando estruturas de git;Simplificação da Stack de dados;Processamento em Modern Data Warehouses dentro outros adapters.Falamos também nesse bate-papo sobre os seguintes temas:Dados como pilar central;Dbt;Times de BI Moderno.Aprenda mais sobre dbt, como utilizar uma tecnologia para Modern Data Stack, junto com o time da One Way Solution, que mais impulsiona a comunidade, tanto com conteúdo, como com treinamentos e eventos para ajudar os profissionais de dados brasileiros em vagas de trabalho dentro e fora do país.Matheus Willian = https://www.linkedin.com/in/matheuswillian/https://www.getdbt.com/ Luan Moreno = https://www.linkedin.com/in/luanmoreno/

Modern Data Stack.

With the increasing rate at which new data tools and platforms are being created, the modern data stack risks becoming just another buzzword data leaders use when talking about how they solve problems. Alongside the arrival of new data tools is the need for leaders to see beyond just the modern data stack and think deeply about how their data work can align with business outcomes, otherwise, they risk falling behind trying to create value from innovative, but irrelevant technology. In this episode, Yali Sassoon joins the show to explore what the modern data stack really means, how to rethink the modern data stack in terms of value creation, data collection versus data creation, and the right way businesses should approach data ingestion, and much more. Yali is the Co-Founder and Chief Strategy Officer at Snowplow Analytics, a behavioral data platform that empowers data teams to solve complex data challenges. Yali is an expert in data with a background in both strategy and operations consulting teaching companies how to use data properly to evolve their operations and improve their results.