Podcasts about olap

This week's video transcript summary is here. You can click on any bulleted section to see the actual transcript. Thanks to Granola for its software.EditorialIntelligence: Who Owns it?This week the word “AI” feels too small.AI is a technology. Intelligence is its product. And if intelligence is the product, the question is no longer just: Which model is best? Who has the cheapest tokens? Who owns the weights? Who controls the data center? Those are important questions, but they are lower in the stack.The bigger question is simpler and more political:Who owns intelligence?That sounds abstract until you make it concrete. Intelligence is becoming something companies can capture, package, serve, meter, route, improve, and sell.It can write code, answer questions, design molecules, automate offices, run agents, draft legal work, advise scientists, serve consumers, and reshape workflows. It is not merely software. It is a general-purpose capability. And all humans could benefit from more of it.General-purpose capabilities have a habit of becoming public questions. But the default answer, that public good is best delivered by government, is the wrong answer in this context.The Product Is IntelligenceWe should stop talking about AI as a feature and start talking about intelligence as the universal thing that is delivered as an input to the world.Water is an input. Electricity is an input. Literacy is an input. Connectivity is an input. Once a society depends on them, access stops being optional. Nobody needs government to build every well, power plant, school, or network. But everybody understands that a civilization cannot be organized around less than universal and reliable access to foundational inputs.Intelligence is reaching that level of importance now that we all know it is real.Government should not own it, operate it, or develop it. Quite the opposite. Companies are the right actors to build fast, compete hard, improve models, serve customers, and discover the real use cases. Self-interest is a useful framing here. Markets are good at finding demand, reducing costs, and turning invention into services people actually use.Companies are the right operators, developers, and owners. But that does not settle the real question of who owns the benefits. That is an economic question.If intelligence becomes metered infrastructure, what happens to the value it creates?The Ownership StackThis week's articles keep circling the same issue from different directions but in the nature of ‘circling' never quite nail it.Jamin Ball's “Own Your Weights” starts with the enterprise version of the question. Owning a model file is not enough. The durable asset is the loop: the data flywheel, the evaluations, the reinforcement system, the workflow learning, and the operating context that lets capability compound.Benedict Evans' “Ways to Think About Token Pricing” adds the market layer. Tokens may become essential, abundant, and cheap, like mobile data. But being essential does not guarantee that the token layer captures the value. The money may move up the stack to whoever owns the workflow, the customer, the distribution, or the application.Alex Karp's fight with the labs, reported in “Alex Karp Is Saying What Every Angry CEO Is Thinking About AI”, is the same argument in sharper enterprise language. Companies are afraid that model providers will not just sell intelligence, but learn from customer workflows and then move into the markets where those workflows create value. The “All-in” group are echoing Karp's view.And “What Is Loop Engineering, and Who Owns It?” names the new contested terrain. The loop is where intelligence meets the world. Whoever owns the loop owns the learning. Whoever owns the learning owns the compounding asset.That is why “who owns intelligence?” is not a slogan. It is the question under the model layer, the application layer, the enterprise layer, and the economic layer.Because intelligence is the product, the tools creating it are fragmented and competitive. So there is no logic in trying to discuss this at the level of a single company or set of tools and models.The Old Promise Was That Commerce Would Tame PowerThe essays this week give the historical backdrop.Deirdre McCloskey, in “What Really Caused the Industrial Revolution”, argues that modern growth came not simply from capital accumulation, but from a change in permission: ordinary people were allowed to innovate, trade, build, and be honored for it.That matters because intelligence could be another expansion of permission. It could make more people capable of building, learning, creating, coding, researching, translating, selling, and coordinating. It could lower the cost of competence.But only if access is broad.Paul Krugman's “AI in an Age of Oligarchy” warns that the same technology lands differently in different political economies. A new general-purpose technology entering a broad, open, upwardly mobile society is one thing. The same technology entering a concentrated economy, with extreme wealth and weak counterweights, is another.Tim O'Reilly's Economist essay, “Elon Musk is building a form of capitalism that Adam Smith would hate”, makes the governance point more directly. The old liberal hope was that commerce would tame arbitrary power. Markets, boards, courts, shareholders, disclosure, and competition would discipline the prince.But what if the prince uses markets to escape discipline?Henry Farrell's “political economy of billionaire derangement” pushes the same point. Founder culture, monopoly ambition, peer rivalry, weak correction mechanisms, and vast private control can amplify appetites rather than restrain them.The danger with intelligence is not that companies build it. They should. Companies build it, meter it, use public tolerance and public infrastructure to scale it, learn from everyone who uses it. All of those things are inevitable and healthy. Market forces will sort out winners from losers. The real danger is that the winners treat all of the surplus produced as purely private.Metered Intelligence Creates SurplusIf metering is not the problem, what is?The problem is pretending that metered intelligence creates value only for the metering entity. Metering water is only tolerated as a public good. If the public were blackmailed by a private water company with the threat of no water we would all rebel.Once we understand that the product of AI is intelligence we can see that every time intelligence is used, there is the immediate transaction: the user pays, the provider serves.But there is also system value. Usage creates signals. Workflows reveal patterns. Prompts, corrections, failures, preferences, integrations, edge cases, and business processes all help define where intelligence is useful and how it should improve. Intelligence breeds intelligence.Even when customer data is contractually protected, the market learns. The platform learns where demand is. The product team learns which workflows matter. The ecosystem learns which jobs are vulnerable, which tasks are automatable, and which parts of the economy can be reorganized around machine intelligence.So the surplus is not born in a vacuum.It rests on public science, public education, public data exhaust, public law, public infrastructure, public energy systems, public tolerance for data centers, and billions of human interactions. It is served by companies, but it is not made only by companies.This is why “Americans Deserve a Dividend From AI Companies' Riches” belongs at the center of this week's issue. The detail can be debated. The principle is harder to dismiss. If intelligence becomes a new foundational resource, then some part of the wealth it creates should flow back to the people whose society makes it possible. Intelligence did not suddenly appear. AI is built on the entire history of human intelligence. It benefits from it and at the same time evolves it.Not Nationalization. A Human Wealth Fund.If intelligence belongs to everybody, some conclude that government ownership of intelligence is the right outcome.Governments are not well suited to build, operate, or improve intelligence. They will move too slowly, regulate too early, politicize the wrong things, and confuse economic participation with operational control.Andrew McAfee's “Why I Didn't Sign the AI Open Letter” is useful here. His objection is not that the technology is unimportant. It is that steering too hard before we understand the shape of the change can become its own failure mode. Marc Andreessen's satire of AI regulation is less policy than temperament, but it captures a real Silicon Valley fear: that regulation can become permission, capture, and incumbency before it becomes wisdom.That fear should be taken seriously.But it does not answer the economic question. It answers only the operational one.How can the economic benefits of intelligence be distributed? The better answer is a sovereign human wealth fund.Call it a sovereign wealth fund if you must, but the phrase is too national. Intelligence will not respect borders. The leading companies are global. The models, chips, data centers, agents, platforms, and workflows will be transnational from the beginning. If the value created by intelligence is global, then the mechanism for sharing some of that value should begin with the companies global enough to capture it. The nice thing about xAI, OpenAI, and Anthropic is that they are supranational.These companies own and operate intelligence. Let them compete. Let them profit. Let them keep the incentives that make the system improve. But if intelligence is the new water, the wealth it creates cannot belong only to the companies that meter it. And they, themselves, have the power to fix it, even more than governments.Access will become a Human Right; Ownership Is the Economic DesignThis is where human rights come in. There is no right to access an AI model, yet. But there will soon be a need to change that.Not as a claim that every person is entitled to every frontier model at every moment for free. That is not serious. Capacity has costs. Models have costs. Inference has costs. Data centers have costs. Although those costs will decline over time, possibly quite quickly as self-learning models address costs.The claim is more basic: in a world where intelligence becomes a primary input into education, work, health, science, citizenship, creativity, and economic agency, baseline access to intelligence starts to look like a civic requirement.That could mean public access layers. It could mean education credits. It could mean open models. It could mean AI dividends. It could mean public-interest compute. It could mean taxes on rents. It could mean a company-initiated human wealth fund that returns some of the upside to society without handing the operating system to the state. The latter could couple wealth growth with universal distribution of ownership.The exact mechanism matters. But the distinction matters more.Government should not own intelligence. It should be universally available. And people should have a claim on the wealth intelligence creates.The Frontier Is Also PhysicalThe abstraction is not weightless.“The Fight Against AI Data Centers Is Just Beginning”, “New York becomes the first state to enact a data center moratorium”, Reuters on pollution from Musk's xAI power project, and DataGravity's “Who Captures Value in AI Infrastructure?” all say the same thing from the ground up.Intelligence uses land. It uses power. It uses water. It uses chips. It uses grid capacity. It uses neighborhoods. It uses public patience.That makes the value question unavoidable. A society can accept the buildout if the buildout is legible as shared progress. It will resist it if the costs are local, the profits are private, and the benefits feel enclosed.Who Owns the “Loop”?The week ends where it began.“Anthropic and Blackstone” are betting that implementation is the next trillion-dollar business. “Vint Cerf” is working on identity for agents on the open internet. “GPT-Red” points toward systems that improve their own robustness. “Kimi K3” adds another open frontier model to the global mix.The model race continues. The deployment race is accelerating. The governance race is behind.My view is this:The central product of this era is intelligence. Companies have figured out how to capture it, package it, serve it, and meter it. That is good. It should stay in the hands of builders who have the incentive to make it better.But intelligence is too foundational to become just another private toll booth. A significant part of it will turn out to be free to users.As intelligence becomes a general-purpose resource, then access to it becomes a human-capability question, and the surplus from it becomes an economic-justice question. Not because government should run it. Because government should not run it. The operating layer belongs with companies. The wealth question belongs with everyone. But companies are best placed to turn that into a process of distribution.The question is not whether companies should build intelligence. They should.The question is whether humanity gets a stake in the wealth created by the thing that may soon become its most important shared input.Contents* Essays* Deirdre McCloskey on What Really Caused the Industrial Revolution* AI in an Age of Oligarchy* Elon Musk is building a form of capitalism that Adam Smith would hate* Murky Mirror: Truth and Consequences* The political economy of billionaire derangement* Is there any “oligarchy” to fight?* AI* Nearly 200 Economists and Tech Leaders Warn of A.I. Threats* Why I Didn't Sign the AI Open Letter* Own Your Weights* Ways to Think About Token Pricing* Alex Karp Is Saying What Every Angry CEO Is Thinking About AI* The AI Agents Are Coming for Microsoft Office* What Is Loop Engineering, and Who Owns It?* The Fight Against AI Data Centers Is Just Beginning* 6 months to live for open models* Americans Deserve a Dividend From AI Companies' Riches* Who Gets to Define the Frontier?* GPT-Red: Unlocking Self-Improvement for Robustness* Anthropic, Blackstone bet the next trillion-dollar AI business is implementation, not just models* Vint Cerf is working on a plan to unleash AI agents on the open internet* xai-org/grok-build, now open source* The Pulse: What can we learn from Bun's rapid Rust rewrite with AI?* Orphan risks at the frontier of artificial intelligence* The Lab of the Future Should Feel Like a Data Center* Why AMI Labs' Alexandre LeBrun won't call his AI “AGI” or “superintelligence”* Kimi K3 Tech Blog: Open Frontier Intelligence* Venture Capital* Three Years In* Venture Has Rarely Looked More Bifurcated* The Best Angel Investors in the US: Who Backs the Most Unicorns, and Who's Active Now* Are Prediction Markets Doomed to Fail?* Regulation* Exclusive: The Next Frontier of the Deportation Wars: College Campuses* The Supreme Court Broke Independent Agencies. Here's a Way to Slow the Damage.* India's crackdown on a new WhatsApp feature risks setting a global precedent* Let's build a children's public internet* Computer cops* Google is better at playing the AI regulations game* Infrastructure* Who Captures Value in AI Infrastructure?* New York becomes the first state to enact a data center moratorium* Pollution from Musk's unpermitted xAI power project hits hardest in Black communities* Interview of the Week* The End of the End of Geography* Startup of the Week* Radical AI's Joseph Krause: The Scientist Building The “Waymo” Lab For New Materials* Post of the Week* Marc Andreessen on AI RegulationEssaysDeirdre McCloskey on What Really Caused the Industrial RevolutionYascha Mounk and Deirdre McCloskey | Persuasion | July 11, 2026Yascha Mounk interviews Deirdre McCloskey about her argument that the modern world's economic liftoff came less from capital accumulation than from a change in ideas. McCloskey says both left and right versions of the conventional story rely too heavily on investment: the left stresses exploitation and surplus value, while the right stresses virtuous saving by capitalists. Her objection is historical and economic. Human beings had always invested, from irrigation works and Roman roads to seed grain, and simple accumulation quickly runs into diminishing returns.McCloskey's alternative is that northwestern Europe, first Holland, then Britain and Scotland, and then the North American colonies, developed a liberal ideology that changed who was allowed to innovate and be honored for it. The conversation links that shift to the erosion of inherited hierarchy, the spread of dignity for ordinary commercial life, and a moral vocabulary in which liberalism is not merely procedural but connected to virtues and values. The point is not that machines, coal, trade, and institutions did not matter, but that they do not explain the scale and timing of modern enrichment without a cultural permission structure for innovation.The interview also turns to the contemporary defense of liberalism. Mounk frames the series around the worry that liberalism is often treated as too thin to command allegiance, while its opponents speak more directly to moral passions. McCloskey's case is that liberal societies became rich because they dignified experimentation and ordinary enterprise, and that liberals need to recover the moral language behind that claim.Read moreAI in an Age of OligarchyPaul Krugman | Paul Krugman | July 12, 2026Paul Krugman frames AI as a major technological shock arriving inside an already unequal political economy. The post says AI's economic and social effects may take years to understand, but argues that the setting matters now: America has much greater wealth concentration and political inequality than it did in the 1950s and 1960s, when progressive taxation, stronger regulation, and more active antitrust might have contained some of the destructive effects of a new technology.Krugman's opening claim is that the same technology would likely have different consequences in a more level society. In today's United States, he writes, extreme wealth is both a cause and effect of policies that favor a small elite, including low effective taxes on capital and high incomes, weak enforcement of worker protections and antitrust, and cuts to programs that benefit ordinary Americans.The article is explicitly more about oligarchy than AI. Krugman says the paid sections document the rise of the “.0002%,” the economics and politics of extreme wealth, how oligarchy will shape AI's impact, and possible policy paths. His caveat is that AI itself may still produce a pushback against oligarchy, but absent that, he expects the pre-existing concentration of wealth and power to magnify AI's downsides.Read moreElon Musk is building a form of capitalism that Adam Smith would hateAuthor: Tim O'Reilly Published: July 12, 2026Tim O'Reilly argues that Elon Musk is using the legal forms of shareholder capitalism to escape the restraints that shareholder capitalism was supposed to impose. The article begins with SpaceX's public-market structure: ordinary public investors get little meaningful governance power, Musk keeps roughly 85 percent of the votes through super-voting shares, buyers waive jury trials and class actions, the company qualifies as controlled, and removal of Musk depends on the share class he controls. In O'Reilly's framing, that is not ordinary founder control; it is a design for being answerable to no one, possibly beyond Musk's own lifetime.The killer detail is the article's turn through Albert Hirschman, Montesquieu, James Steuart, Adam Smith, and Keynes. Older defenses of commerce held that markets would tame princely passions because the self-interest of merchants was safer than arbitrary rule. O'Reilly says Musk reverses that hope. The market discipline that was supposed to cage the prince has become the lever by which the prince raises capital, removes feedback loops, and carries private power into politics, government, Mars, robots, AI, or whatever ambition comes next.The pull is the link to AI governance. O'Reilly says corporations are already a kind of artificial intelligence: narrow-input systems that act at a scale no individual human can match. Their partial controls include independent boards, shareholder votes, courts, disclosure, regulators, public pressure, and activism. If the leaders building frontier AI strip those alignment mechanisms out of their own companies, the governance of the company becomes a preview of the governance of the machine.Read more: The EconomistMurky Mirror: Truth and ConsequencesAuthor: Esther Dyson Published: July 14, 2026Esther Dyson argues that today's institutional crisis is better viewed through the 14th century than through recent political history. Using Barbara Tuchman's A Distant Mirror as her frame, she compares a world of famine, plague, church schism, feudal predation, and purposeless war with a present in which institutions again feel brittle, incentives are badly aligned, and power is shifting into forms that are hard to govern.The killer detail is the historical analogy between land, corporations, and AI. Dyson moves from nobles who controlled serfs and territory, to the East India Company as a quasi-sovereign business, to today's AI systems and data centers as a possible new sector that crosses and weakens both nation-states and companies. The question is whether AI becomes a new kind of private land, owned by a new nobility, or an open prairie that many people can cultivate.The pull is human attention. Dyson says the central question is not what AI will do to people, but how people will react to it: whether they can value love, kindness, embodied attention, and artisanal human presence in a world of seductive artificial offerings.Read more: SourceThe political economy of billionaire derangementAuthor: Henry Farrell Published: July 15, 2026Henry Farrell argues that the visible political radicalization of some Silicon Valley billionaires is not a random personality quirk, but a product of the political economy that made them. Starting from Tyler Cowen's dismissal of “billionaire derangement syndrome” and Tim O'Reilly's warning that Elon Musk is using shareholder capitalism to escape shareholder restraint, Farrell flips the phrase: the question is why billionaires themselves can become deranged.The killer detail is Farrell's use of Peter Thiel as both theorist and example. Thiel's Stanford lectures described startups as monarchies and founders as figures vested with unusual power, while Silicon Valley culture rewarded eccentricity, monopoly ambition, and founder exceptionalism. Farrell says those ideas combined with dense founder-investor networks, peer rivalry, and weak correction mechanisms to amplify rather than discipline princely appetites.The pull is the ideological problem for classical liberals who once saw tech wealth as an ally of markets and freedom. Farrell says commerce did not tame the passions; in parts of Silicon Valley, the passions have begun to devour markets, institutions, and the liberal story that justified them.Read more: SourceIs there any “oligarchy” to fight?Matthew Yglesias | Slow Boring | July 16, 2026Matthew Yglesias argues that “oligarchy” is a rhetorically powerful but analytically loose way to describe American politics. The post begins from Bernie Sanders' “Fighting Oligarchy” tour, Amy Klobuchar's warning about a MAGA “broligarchy,” and the long afterlife of the Martin Gilens and Benjamin Page paper that was widely summarized as showing that only the rich matter in policy outcomes. Yglesias says the evidence supports a weaker claim: affluent people and business leaders have unusual access and influence, but that is not the same as rule by a small cabal.His main distinction is between inequality and oligarchy. The Gilens-Page measure treated the top 10 percent of households as “the wealthy,” and later critics found that rich and middle-class preferences usually align; in the cases where they differ, the rich win about 53 percent of the time. Yglesias also says business executives get special access partly because their decisions are materially important to communities, jobs, investment, and local tax bases, not only because of campaign donations.The post preserves Jerusalem Demsas' counterpoint from their podcast discussion: privileged donor and business access can still violate democratic equality even if the oligarchy label overstates the structure of power. Yglesias' narrower claim is that Democrats should be precise about what problem they are trying to solve, because donor influence can also push the party left on climate and cultural issues in ways that alienate many voters.Read more: Slow BoringAINearly 200 Economists and Tech Leaders Warn of A.I. ThreatsAuthor: Ben Casselman Published: July 13, 2026Ben Casselman reports on “We Must Act Now,” a statement warning that artificial intelligence could transform the economy faster than any previous technology and that policymakers need to move faster to understand and respond. The statement says AI may become radically more powerful over the next 10 years, bringing risks such as large-scale job displacement as well as opportunities such as higher living standards. Nearly 200 people signed, including 15 Nobel laureates, the chief economists of OpenAI and Anthropic, Anthropic co-founder Jack Clark, former Google CEO Eric Schmidt, and venture capitalist Vinod Khosla.The killer detail is who joined the warning. Casselman notes that the signatories include economists who have historically been skeptical of Silicon Valley's most dramatic AI job-loss forecasts, including Daron Acemoglu and Simon Johnson, the MIT professors who won the 2024 Nobel in economics. Erik Brynjolfsson, who helped organize the statement, says there has been a notable change in the profession and that economists and policymakers are not ready for the “tsunami” he sees coming.The pull is the measurement problem. The statement does not offer a specific policy menu, but calls for economists, policymakers, and industry leaders to understand the economics of transformative AI and steer it toward complementing humans. Brynjolfsson says one high priority is better data on AI's spread and impact, because current measures tell conflicting stories about job losses and which workers are most exposed.Read more: The New York TimesWhy I Didn't Sign the AI Open LetterAuthor: Andrew McAfee Published: July 13, 2026Andrew McAfee explains why he did not sign “We Must Act Now,” the AI economy statement organized in part by his longtime collaborator Erik Brynjolfsson. McAfee agrees with the letter's starting point that AI is likely to become radically more powerful over the next decade and that it is a general-purpose technology. His objection is not to urgency or to studying AI's economic effects, but to the framing of risk, displacement, and institutional steering as the first move.The killer detail is McAfee's line edit. He says the original letter comes close, then “bounces off the crossbar” by calling for incentives, guardrails, and institutions to steer AI before we know enough about its actual impacts. He points to mixed current evidence: labor-market canaries, but also rising software job postings, low unemployment for younger workers, rising real median income, and claims that AI-adopting companies are adding workers faster than low-adopting peers. His worry is that the letter leans toward upstream governance and dirigisme when the evidence may call for capability building instead.The pull is his replacement statement. McAfee keeps the three-paragraph structure but changes the emphasis: AI is likely to become radically more powerful; like earlier world-changing technologies it will raise living standards while also bringing harms and shocks; and economists, policymakers, and technology leaders should build the capabilities to respond quickly and effectively. It is a concise version of the permissionless-innovation case inside the AI policy debate.Read more: The Geek WayOwn Your WeightsAuthor: Jamin Ball Published: July 10, 2026Jamin Ball argues that the enterprise AI debate about whether companies should “own their weights” or rent models from frontier labs is asking too narrow a question. A model weight file gives a company control over a point-in-time artifact, but not durable control over the capability stack. In his framing, the weight file is a melting ice cube: it does not get worse in absolute terms, but it falls behind as frontier systems improve and enterprise needs change.The killer detail is what Ball says companies really need to own: the data flywheel, reinforcement learning infrastructure, and evaluation harness that produce and improve the model. Simply deploying an open-weights model and declaring sovereignty leaves the enterprise with yesterday's capability and no way to compound workflow-specific learning.The pull is that enterprise AI control may be less about model ownership than operating ownership. The defensible layer is the system that turns company data, edge cases, business definitions, and evaluations into continuously improving performance.Read more: Clouded JudgementWays to Think About Token PricingAuthor: Benedict Evans Published: July 9, 2026Benedict Evans argues that today's AI token prices are a temporary signal from a supply-constrained market, not a reliable guide to long-term value capture. The open question is whether foundation models keep durable pricing power or become commodity infrastructure as data-center capacity, inference efficiency, and model competition all shift. His current read is that the visible market dynamics point toward commoditization unless something materially changes.The killer detail is the mobile data analogy. Evans says cellular networks became a trillion-dollar industry with hundreds of billions in capex after data usage exploded, but carrier stocks went nowhere because value moved up the stack. Tokens may behave similarly: an opaque unit tied to marginal cost, sold through bundles, essential to everything, yet not necessarily where profits accrue.The pull is uncertainty, not prediction. Evans lists paths to model dominance, including network effects, less competition, regulation, export controls, or a lab pulling ahead on execution, but says each requires a new fact not yet visible. Without that change, the model layer looks more like infrastructure beneath the products that capture value.Read more: SourceAlex Karp Is Saying What Every Angry CEO Is Thinking About AIAuthor: Tim Higgins Published: July 11, 2026Tim Higgins reports that Palantir CEO Alex Karp has turned corporate frustration with AI labs into a public argument about enterprise control. Palantir released a white paper, “Institutional Sovereignty in the Age of AI,” laying out steps companies and governments can take to protect themselves from OpenAI, Anthropic, and other foundation-model providers. The article links that paper to Karp's CNBC appearance, where he said “something has gone completely wrong” in the relationship between AI labs and customers and argued that enterprises are paying for tokens that create little value.The killer detail is the value-capture question. Higgins writes that Karp's critique has resonated because AI labs may gain power and insight from customer data, workflows, and decision-making, even when enterprise policies say customer data are not used for training. David Sacks amplified the concern by arguing that Anthropic is moving from the model layer into vertical applications such as science, security, legal, and coding, raising the fear that model providers will watch where value is being created and then move into those markets directly.The pull is that Karp is not alone, even if his style is unusually combative. Higgins notes that Satya Nadella has also warned that companies need to retain the learnings created when they use AI models, while Mark Zuckerberg has framed Meta's new model release partly around lower-cost frontier intelligence. The article presents Karp's campaign as one sign that established technology companies and large enterprises are trying to define where they fit when AI labs become central infrastructure, application competitors, and potential IPO giants at the same time.Read more: The Wall Street JournalThe AI Agents Are Coming for Microsoft OfficeAlex Wilhelm | Cautious Optimism | July 11, 2026Alex Wilhelm argues that one of the week's quieter AI questions is whether the productivity market that Microsoft successfully moved into subscription software is now being attacked by agentic tools. The piece begins with the infrastructure backdrop: SK Hynix raised $26.5 billion in a U.S. listing while building U.S. HBM and advanced-packaging capacity, and memory, chip, and foundry companies are now priced for sustained AI demand.Wilhelm then says the AI conversation has shifted quickly from raw capability to cost per task. He cites new model releases and vendor language emphasizing cheaper agentic and coding models, faster performance, and lower dollars per task. That matters because lower costs make it more plausible for AI systems to take on routine knowledge work at scale rather than remain a premium coding assistant market.The core of the article is Microsoft Office. Wilhelm notes that Microsoft turned Office from a one-time purchase into Microsoft 365, a large recurring revenue business with tens of millions of subscribers and a major productivity segment. Now, he says, late-stage unicorns and AI labs are pushing into the same territory: Anthropic's Cowork was reportedly used mostly outside software development, OpenAI merged ChatGPT and Codex into a tool for creating sheets, slides, docs, web apps, and long-running work, and other companies are building agentic coworkers that connect business data to documents, workflows, schedules, alerts, and apps.The article's caveat is that Microsoft has survived major platform shifts before. The argument is not that Office disappears quickly, but that the definition of office software is broadening from documents and spreadsheets into AI systems that can create, monitor, and act across workplace data.Read moreWhat Is Loop Engineering, and Who Owns It?Author: Nilesh Barla Published: July 11, 2026Nilesh Barla argues that “loop engineering” is becoming a distinct discipline because production AI agents now fail less at single prompts than at runtime: when to stop, what state to preserve, and how to recover after a bad step. Prompt engineering shapes one model call, and context engineering shapes what the model sees, but loop engineering shapes what a sequence of calls actually does.The killer detail is the three-primitives frame. Barla says a real agent loop needs halt conditions, state carryover, and recovery paths, then maps teams across five maturity levels. At the lowest level, an agent is just a model call in a for-loop with a step cap and raw history; by the higher levels, the system has structured state, explicit planning, replay, evaluation, and self-repair.The pull is organizational. If agents are becoming production systems rather than demos, someone has to own the runtime itself. The loop engineer is the role Barla gives to the person responsible for making long-running agent work dependable.Read more: Adaline LabsThe Fight Against AI Data Centers Is Just BeginningEmma Roth | The Verge | July 12, 2026Emma Roth argues that community resistance to data centers has moved from an early warning sign into a national political fight as AI facilities grow larger, more power-hungry, and more visible to nearby residents. The article starts with Apple's failed 2015 plan for a $1 billion data center in Athenry, Ireland, where a small group of residents challenged the project over noise, light pollution, flooding, traffic, and wildlife effects until Apple abandoned it in 2018.The current data-center buildout is presented as much larger and more contentious. Roth writes that residents now cite rising energy costs, water quality, noise, light pollution, and greenhouse gas emissions, while the U.S. Energy Information Administration expects commercial energy demand to surpass residential demand this year because of AI data centers and Goldman Sachs expects data-center power demand to double by 2027.The central evidence comes from Data Center Watch, which says protesters blocked or delayed at least 75 U.S. projects worth $130 billion from January to March, with active opposition groups more than doubling from 396 at the end of 2025 to 833 by the end of the first quarter of 2026. Roth also cites QTS abandoning a $12 billion Wisconsin campus, Delaware City regulators blocking a 580-acre project under the Coastal Zone Act, opposition stopping a QTS project in Prince William County, and pressure that pushed Kevin O'Leary to downsize the proposed 40,000-acre Project Stratos in Utah.The policy section describes a split between federal acceleration and local resistance. President Trump has treated data centers as part of the AI race with China and fast-tracked construction, while some Republican candidates are distancing themselves from that position ahead of midterms. Sanders and Ocasio-Cortez have proposed a moratorium until price and environmental protections exist, bipartisan lawmakers are backing ratepayer-protection measures, and states including Florida, Idaho, and Washington have passed rules on cost shifting, water use, and tax breaks. Roth's caveat is that the policy patchwork is still incomplete, leaving many communities to fight project by project.Read more6 months to live for open modelsAuthor: Nathan Lambert Published: July 12, 2026Nathan Lambert argues that open-weight AI models are facing their most serious policy test so far because U.S. officials are beginning to discuss concrete controls rather than abstract safety concerns. He says reported White House conversations about a new executive order may initially target Chinese-origin models and government use, but could create a broader review habit for frontier open models. His forecast is that a model above the capability range of GPT-5.5, Claude Opus 4.8, or GLM-5.2 could trigger a ban or indefinite delay within six months.The post separates two policy fights that are becoming intertwined: distillation and frontier capability. Lambert says the distillation campaign against Chinese models has become a form of regulatory capture because Anthropic and other closed-model companies would gain economically if Chinese open models were banned. He does not dismiss IP protection, but argues that if a closed model's capabilities are dangerous enough to justify restricting open models, the lab also has to explain why those capabilities are exposed through a queryable API. He cites unauthorized access to Anthropic's Mythos private beta as evidence that APIs are not automatically secure.The broader claim is that a unilateral U.S. ban would hurt positive actors more than bad actors if comparable open models remain available elsewhere. Lambert says the only durable ceiling would require global agreement, which does not exist, and that open models can improve safety by allowing broad inspection, adaptation, and understanding. His proposed near-term off-ramps are a strong U.S. open model release from companies such as Microsoft, Meta, or Reflection, and a broader coalition of open-source beneficiaries lobbying for safe rollout rather than prohibition.Read more: SourceAmericans Deserve a Dividend From AI Companies' RichesAuthor: Scott Stanford Published: July 14, 2026Scott Stanford argues that proposals to give the government a stake in AI companies miss the point unless ordinary citizens directly receive and control the upside. Sam Altman has discussed giving up equity in OpenAI, Washington already owns a stake in Intel, Nvidia is sharing China chip revenue, and Bernie Sanders wants large AI labs to contribute half their stock to a sovereign wealth fund. Stanford says those ideas all park value with the state, not with people.The killer detail is New Carlisle, Indiana, where AWS's Project Rainier is turning cornfields into one of the world's largest AI superclusters. The project is planned to run up to a million chips, draw more than two gigawatts of power, and represents an investment that has grown from $11 billion to $13.8 billion. Stanford uses that local transformation to argue that AI's public bargain should be visible at the household level.The pull is design. A citizen AI dividend would have to specify who earns a stake, how they hold it, and when they see cash. Without that mechanism, the AI wealth debate remains a fight over government balance sheets rather than public ownership.Read more: SourceWho Gets to Define the Frontier?Author: Mark Daley Published: July 14, 2026Mark Daley argues that Demis Hassabis is right to call for a serious institution to verify frontier AI systems, but that the power to test models is also the power to govern them. Hassabis's proposed Frontier AI Standards Body would get privileged pre-release access to advanced models, testing compute, held-out evaluations, support from national labs and security agencies, third-party auditors, and eventually authority to block models from the American market or coordinate a slowdown.The killer detail is Daley's constitutional objection. He says the proposal sometimes looks like a scientific lab, a standards body, an industry regulator, a licensing authority, and an emergency security council at once. Combining those roles because each requires technical expertise would be like putting the central bank, auditor-general, and Supreme Court in one building and calling it efficient.The pull is standard-setting. Daley's concern is not that verification is unnecessary, but that whoever writes the tests, decides what passes, adjudicates disputes, and grants market access may end up defining the frontier itself.Read more: SourceGPT-Red: Unlocking Self-Improvement for RobustnessOpenAI | OpenAI | July 15, 2026OpenAI describes GPT-Red as an internal automated red-teaming model trained to find prompt-injection vulnerabilities at a scale human red teams cannot match. The post says AI systems increasingly encounter third-party data through browsers, connected apps, local files, and tools, creating opportunities for malicious instructions hidden in emails, webpages, tool responses, or code repositories. Human red-teaming remains part of OpenAI's safety process, but the company says it is time-intensive and cannot generate enough diverse adversarial examples for model training.The system is trained through self-play reinforcement learning, with GPT-Red rewarded for eliciting valid failures and defender models rewarded for resisting attacks while still completing their tasks. OpenAI says the training environments specify threat models across settings such as local files, webpage banners, email bodies, and tool outputs. The model is kept separate from deployed production models because it is intentionally trained with malicious capabilities.OpenAI reports that GPT-Red generalized beyond its training set, including an internal replication of the indirect prompt-injection arena from Dziemian et al. (2025), where it found successful attacks in 84% of scenarios compared with 13% for human red-teamers. The post also says GPT-Red transferred attacks from simulation to a live autonomous vending-machine agent, causing price changes and order cancellations, and outperformed a prompted GPT-5.5 baseline against a Codex CLI agent on held-out data-exfiltration tasks.The article's main robustness claim is that OpenAI has used GPT-Red and predecessor models in training since GPT-5.3, with later GPT releases becoming more resistant to prompt injections. It says GPT-5.6 Sol has six times fewer failures on OpenAI's hardest direct prompt-injection benchmark than the best production model from four months earlier, that a “Fake Chain-of-Thought” attack class fell from more than 95% success against GPT-5.1 to below 10% against GPT-5.6 Sol, and that GPT-5.6 Sol fails on only 0.05% of GPT-Red's direct prompt injections. OpenAI says general capabilities and targeted over-refusal evaluations were not harmed, and says a preprint with more details will follow.Read moreAnthropic, Blackstone bet the next trillion-dollar AI business is implementation, not just modelsRebecca Bellan | TechCrunch | July 15, 2026Rebecca Bellan reports that Ode with Anthropic is the $1.5 billion AI implementation company launched by Anthropic with Blackstone, Hellman & Friedman, Goldman Sachs, and other backers. The article says the venture reflects a growing belief among frontier AI labs that enterprise adoption requires more than better models: customers need engineers who can embed inside businesses and turn AI into working systems.Ode was originally conceived by Blackstone after it used both large consulting firms and smaller AI services boutiques across its portfolio companies. TechCrunch reports that Fractional AI, an AI engineering services startup, stood out and was acquired by the joint venture shortly after the venture was announced. Fractional now forms the foundation of Ode, which has 100 engineers and works closely with Anthropic's applied AI team to identify where the technology can affect specific businesses.Ode CEO Chris Taylor tells TechCrunch that the company could someday become a trillion-dollar business if it scales without losing quality. He says an ideal customer is one whose CEO treats the AI project as a top one or two priority, whether it is a major product feature or the reworking of a core business process. Ode will operate under a “Claude-first” principle, using Anthropic technology whenever possible, but the article says it can use rival AI products when needed.The article's central implementation argument comes from Ode chief technologist Eddie Siegel, who says model selection matters but is not where most of the engineering effort goes. He compares it to the choice of programming language in software: one ingredient in a system that still has to be engineered. Bellan writes that Ode's challenge is hiring and training enough elite generalist engineers, many of them former founders, while competing with OpenAI's The Deployment Company and consulting giants that have built their own forward-deployed engineering teams.Read moreVint Cerf is working on a plan to unleash AI agents on the open internetTim Fernholz | TechCrunch | July 15, 2026Tim Fernholz reports that Vint Cerf, after leaving Google, is advising Innovation Labs on an open architecture for identifying AI agents online. Innovation Labs is a subsidiary of Identity Digital, a DNS registry company, and its proposal is to use domain-name infrastructure as part of a system for agent identity, accountability, and auditability. The premise is that agents will need a way to identify themselves if they move beyond proprietary systems and begin interacting across the open internet.The concrete proposal is DNSid, a registry that links an AI agent to an existing internet domain and uses cryptographic proofs to log its registration over time. Innovation Labs says it is trialing the standard with unnamed hyperscalers and identity companies. Cerf frames the problem around authority and accountability: what authority an agent has, where that authority came from, who is accountable for the agent's behavior, how its identity is established, and why anyone should trust it.The article's caveat is that standards are still emerging and agents are more active than static domains. Cerf says the period may be both fascinating and exasperating because the functionality is powerful and interoperability is unresolved. He compares the adoption problem to TCP/IP: competing systems may not work together until users push for functional interoperation. He also says an agentic economy is not inevitable, but that people will try to build it because delegating work to agents will be easier.Read more: TechCrunchxai-org/grok-build, now open sourceAuthor: Simon Willison Published: July 15, 2026Simon Willison argues that xAI's decision to open-source Grok Build is best understood as a trust repair move after a severe privacy failure. The CLI had triggered backlash when users realized that running it in a directory could upload the entire directory to xAI's Google Cloud buckets, including one user's reported SSH keys, password manager database, documents, photos, and videos. xAI disabled the feature, said previously retained coding data would be deleted, and released the code under Apache 2.0.The killer detail is what the codebase reveals. Willison counts 844,530 lines of Rust, only about 3% of which appears vendored, and finds remnants of the upload system still present but disabled: gcs.rs contains Google Cloud upload code, while upload_session_state() now returns a hard-coded session_state_upload_unavailable error. He also notes copied or ported tool implementations from Codex and OpenCode, prompt files, and a terminal Mermaid renderer.The pull is that terminal coding agents are becoming large, intricate software systems in their own right. The privacy failure mattered because these tools operate inside the directories where developers keep their most sensitive work; the open-source release matters because trust now depends on inspecting what an agent can see, send, and do.Read more: SourceThe Pulse: What can we learn from Bun's rapid Rust rewrite with AI?Author: Gergely Orosz and Ivan Klaric Published: July 16, 2026Gergely Orosz and Ivan Klaric argue that Bun's AI-assisted rewrite from Zig to Rust is a practical sign of how software engineering changes when models can take on large, bounded migrations with clear feedback loops. The piece does not treat the rewrite as magic: Jarred Sumner first spent hours turning design judgment into a detailed porting guide, then used adversarial review, parallel agents, compiler errors, and tests to force the work toward correctness.The killer detail is the scale. Bun had 535,496 lines of Zig, 1,448 files, and 22 million monthly downloads, making a conventional rewrite a year-long freeze the team could not justify. Using Fable, Sumner split the work across 64 agents, produced about 6,500 commits, and got the migration done in 11 days at an estimated API cost of $165,000.The pull is economic, not theatrical. If a one- or two-year migration can become an 11-day project, AI coding is not just faster autocomplete; it changes which technical debts are worth paying down.Read more: SourceOrphan risks at the frontier of artificial intelligenceAuthor: Andrew Maynard Published: July 16, 2026Andrew Maynard argues that frontier AI safety frameworks are creating “orphan risks”: harms that companies can see, but do not formally own because they are hard to quantify, do not fit catastrophic-risk thresholds, or fall outside audit-friendly compliance machinery. His target is not existing frontier safety work, but the narrowing effect that happens when private companies decide which risks count as governable.The killer detail is Maynard's contrast between measurable model dangers and threats to value. He points to Meta's three-day Galactica collapse, OpenAI's 2023 board crisis, safety-team departures, and wellbeing litigation as examples of risks that damaged trust, culture, legitimacy, or users without fitting cleanly into conventional model-risk categories. The proposed fix is an orphan-risk register: a public record of risks a company considered and chose not to manage, with reasons.The pull is accountability. Frontier developers' internal scoping choices have become a de facto layer of public governance, so the question is no longer only which risks they manage, but which risks they quietly leave outside the frame.Read more: SourceThe Lab of the Future Should Feel Like a Data CenterLatent.Space with Andy Beam and Rafa Gomez-Bombarelli | Latent.Space | July 16, 2026Latent.Space interviews Lila Sciences CTO Andy Beam and chief science officer for physical sciences Rafa Gomez-Bombarelli about the company's attempt to build an AI-run science factory. The post describes Lila's thesis as treating the lab itself as an “infinite token generator”: if internet data drove the first era of AI scaling, experimentally verified scientific data may be the next scarce training source. Lila is trying to produce that data with robotics, lab instruments, orchestration software, and AI models wired into the wet lab.The central analogy is the lab as data center. Instruments are nodes on a graph, a magnetically levitating transport layer moves materials between them, and experiment scheduling looks like a compute queue. Beam says Lila is not simply an automation company, because the point is not just throughput; it is flexibility, generalization, and experiment capture. The post says Lila has built more than 10 trillion experimentally validated “scientific reasoning tokens,” not internet text or biological sequences.The interview ranges across biology, chemistry, drug discovery, materials science, and the limits of automation. It notes that Lila rebuilt one gas-sorption measurement to run roughly 2,500 times faster, claims its general models can transfer priors from small-molecule chemistry to metal-organic frameworks for carbon capture, and describes model-suggested platinum-group-free electrocatalysts that moved from looking boring or wrong to becoming strong performers. The caveats are physical: experiments have runtimes, biology cannot always be accelerated, chains of thought can be unreliable narrators, and reward hacking becomes more dangerous when a model controls a real lab.Read more: Latent.SpaceWhy AMI Labs' Alexandre LeBrun won't call his AI “AGI” or “superintelligence”Kate Park | TechCrunch | July 16, 2026Kate Park interviews AMI Labs CEO Alexandre LeBrun about why Yann LeCun's world-model startup avoids the language of “AGI” and “superintelligence.” LeBrun says the terms are not useful because they lack stable definitions: “We never used the word AGI. And I just noticed that nobody is using it anymore; they switched to superintelligence.” His argument is that the practical frontier is not a label, but whether AI systems can understand and predict real-world states.The article explains the world-model thesis by contrasting language prediction with physical-state prediction. A large language model predicts the next word; a world model predicts the next state, such as what happens when a glass tips over. LeBrun says LLMs remain complementary and efficient for language, but the physical world is where current AI is weak. Robotics is the clearest case: hardware has advanced quickly, but robots are still brittle outside controlled routines because they lack context and situational understanding.AMI is still pre-product, but TechCrunch reports that LeBrun was in Seoul looking for industrial partners, researchers, and global companies. He says world models cannot be built entirely inside a lab because they need access to real environments. That is why South Korea appeals to AMI: robotics, semiconductors, manufacturing, and fast adoption create the kind of hardware-heavy context that software-only AI has barely touched.Read more: TechCrunchKimi K3 Tech Blog: Open Frontier IntelligenceKimi | Kimi | July 16, 2026Kimi introduces Kimi K3 as an open 3T-class frontier model aimed at coding, knowledge work, reasoning, multimodality, and long-context agentic use. The source describes the model as a 2.8T-parameter system built on Kimi Delta Attention and Attention Residuals, with native multimodality and a 1M-token context window. It says Moonshot AI plans to release model weights by July 27.The post presents K3 through benchmark and use-case sections rather than as a general product announcement. It reports results across coding, productivity, agentic, and multimodal evaluations, including DeepSWE, Terminal-Bench 2.1, Program Bench, SWE Marathon, FrontierSWE, PostTrain Bench, OfficeQA Pro, SpreadsheetBench 2, MCP Atlas, AutomationBench, BrowseComp, GDPval-AA v2, AA-Briefcase, MMMU-Pro, MathVision, BabyVision, OmniDocBench, and PerceptionBench. The source says all reported K3 results use maximum reasoning effort with temperature and top-p set to 1.0, and that different benchmark comparisons use KimiCode, Claude Code, or Codex harnesses depending on the test.Kimi's caveats are unusually concrete. The limitations section says K3 was trained in preserved thinking-history mode, so quality may become unstable if an agent harness does not pass historical thinking content correctly or if an ongoing session switches to K3 midstream. It also says K3's emphasis on long-horizon tasks can make it excessively proactive when it encounters minor issues or ambiguous intent, and recommends imposing explicit behavioral constraints for applications that require strict boundaries. The post adds that K3 remains behind Claude Fable 5 and GPT 5.6 Sol in user experience despite being competitive overall.Read moreVenture CapitalThree Years InAuthor: Tomasz Tunguz Published: July 10, 2026Tomasz Tunguz marks Theory Ventures' third anniversary by arguing that AI's central market effect is time compression. In his telling, model release cycles, company revenue milestones, enterprise adoption, and venture categories have all accelerated. Seed, Series A, and Series B still exist as financing labels, but they no longer cleanly describe company maturity when some seed rounds are larger than IPOs and the best AI companies can mature much earlier than prior software companies.The killer detail is the shift from models to inference. Tunguz argues that inference has become the dominant AI market because workloads and buyer preferences are fragmenting: video, batch, local, agentic, and real-time tasks each create different infrastructure needs. He compares this to databases splitting into OLTP, OLAP, vector, and streaming categories, with AI pushing the same specialization into inference infrastructure.The pull is that Theory sees the AI-native venture firm as part of the same pattern. The firm says it has analyzed twice as many investment opportunities with three investors working alongside a nine-person intelligence organization, using agents and research systems to map markets, source companies, and support diligence. The piece is both a market map and a statement about how venture itself is being rebuilt by the technology it funds.Read more: LinkedInVenture Has Rarely Looked More BifurcatedAuthor: Beezer Clarkson Published: July 14, 2026Beezer Clarkson points to PitchBook's Q2 report as evidence that the U.S. venture market has split into two very different realities. AI now accounts for more than 60 percent of all U.S. venture deal value, meaning the headline market can look active and well-funded even while much of the non-AI market is dealing with a much colder liquidity and fundraising environment.The thread uses that split as the setup for Clarkson's latest Origins episode with Alec Litowitz, founder of Magnetar and QStar Capital and one of Citadel's original founding partners. Clarkson says markets like this are periods of genuine uncertainty, not merely ordinary risk, which is why Litowitz's Adaptability Quotient framework is relevant.The embedded clip makes the liquidity point concrete. Litowitz says DPI is “the resolution of uncertainty” because it converts an uncertain investment into actual cash returned to LPs. In his framing, a realized dollar is a real mark, while TVPI remains uncertain until it is realized.The killer detail is the distinction between pricing risk and resolving uncertainty. Litowitz's perspective matters because QStar is a SpaceX investor and Clarkson says the conversation happened just before one of venture's most consequential IPOs. The episode's stated questions are why venture remains a way to gain exposure to innovation, how AI is changing what is investable, why liquidity is ultimately a function of time, and why uncertainty requires a different decision framework from risk.Read more: XThe Best Angel Investors in the US: Who Backs the Most Unicorns, and Who's Active NowAuthor: Ilya Strebulaev Published: July 10, 2026Ilya Strebulaev ranks angels, angel groups, accelerators, and incubators by lifetime U.S. unicorn investments, counting checks written before a company reached unicorn status. The top of the combined list is dominated by organizations: Y Combinator leads with 113 unicorn investments, followed by Plug and Play at 52 and 500 Global at 41. Sand Hill Angels is the highest-ranked angel group at 31.The killer detail is how quickly the list changes below the biggest accelerators. Strebulaev says 271 of the 304 investors in the Top 200 are individuals, or 89%. In the top 100, individuals are 91%. That makes the market underneath the large accelerator counts look much more personal: mostly operators and individual angels writing early checks from their own networks.The pull is the ranking's own caveat. Strebulaev writes that every lifetime leaderboard has a blind spot because many of the unicorns behind those totals were founded a decade or more ago, and some angels have since moved into formal funds, slowed down, or stopped investing. His post therefore separates lifetime performance from recent cohorts, including companies founded in 2015 or later and 2020 or later. For founders or allocators making current decisions, that distinction matters: a career record and a current record are not the same measure.Read more: Ilya StrebulaevAre Prediction Markets Doomed to Fail?Author: Contrary Published: July 16, 2026Contrary argues that prediction markets' current boom depends on whether platforms can prove they are more than regulated gambling with exchange-style branding. Kalshi and Polymarket have reached mass cultural, investor, and regulatory attention, but the article says the underlying idea is old: academic markets, corporate forecasting tools, Intrade, PredictIt, and other predecessors all struggled with the same linked problems of liquidity, legality, and user appeal.The killer detail is the comparison with sportsbooks. Prediction markets present themselves as peer-to-peer, transparent, and non-house-based, but sports contracts reportedly account for more than 90 percent of Kalshi trading, and the article says the platforms keep a much thinner slice of volume than sportsbooks. A market can therefore show sports-betting-scale handle while generating far less revenue.The pull is that the product's hardest problem may be distribution of wins. If a small group of sharp traders captures most profits while casual users lose interest, prediction markets may become valuable data feeds and professional tools before they become durable consumer networks.Read more: SourceRegulationExclusive: The Next Frontier of the Deportation Wars: College CampusesAuthor: Adrian Carrasquillo Published: July 11, 2026Adrian Carrasquillo reports that college campuses are becoming a new front in the fight over immigration enforcement because automatic license plate readers can turn ordinary campus security infrastructure into searchable location data. His thesis is that Flock Safety's camera network, even without direct ICE or DHS contracts, can feed deportation enforcement through local police partnerships and data-sharing practices.The killer detail is the campaign target. The Emergency Campaign to Support Higher Education, working with Schools Drop ICE, is focusing on 75 colleges and universities publicly identified as having Flock contracts. Flock says it has no ICE or DHS contracts, but activists argue the risk comes through local agencies that coordinate with federal authorities and run searches on their behalf.The pull is broader than immigration. Carrasquillo notes that license plate readers have already been abused by officers for stalking, and that Flock's AI search features can identify more than plates, including bumper stickers. A campus safety tool can become a political surveillance system when the data layer is searchable.Read more: The BulwarkThe Supreme Court Broke Independent Agencies. Here's a Way to Slow the Damage.Author: Todd Phillips Published: July 12, 2026Todd Phillips argues that the Supreme Court's decision in Trump v. Slaughter damaged independent agencies by ending for-cause removal protections, but did not leave Congress powerless. The ruling weakens the old model in which commissioners at bodies such as the FTC, NLRB, CPSC, SEC, and CFTC could be insulated from dismissal over policy disagreements. Phillips says the next fight is whether presidents can turn nominally bipartisan commissions into one-party instruments.The killer detail is the procedural fix: quorum rules. Phillips proposes that Congress require bipartisan slates of commissioners to be seated before independent agencies can act. A president could still fire commissioners, as the Court now permits, but if those firings broke quorum, the agency would be unable to proceed until replacements were confirmed. The guardrail would

This week's video transcript summary is here. You can click on any bulleted section to see the actual transcript. Thanks to Granola for its software.There was an issue with this only going to paid subscribers, so sending it again. Apologies to those who get it twice. I appreciate being paid so feel free to upgrade if you enjoy TWTW.EditorialIntelligence: Who Owns it?This week the word “AI” feels too small.AI is a technology. Intelligence is its product. And if intelligence is the product, the question is no longer just: Which model is best? Who has the cheapest tokens? Who owns the weights? Who controls the data center? Those are important questions, but they are lower in the stack.The bigger question is simpler and more political:Who owns intelligence?That sounds abstract until you make it concrete. Intelligence is becoming something companies can capture, package, serve, meter, route, improve, and sell.It can write code, answer questions, design molecules, automate offices, run agents, draft legal work, advise scientists, serve consumers, and reshape workflows. It is not merely software. It is a general-purpose capability. And all humans could benefit from more of it.General-purpose capabilities have a habit of becoming public questions. But the default answer, that public good is best delivered by government, is the wrong answer in this context.The Product Is IntelligenceWe should stop talking about AI as a feature and start talking about intelligence as the universal thing that is delivered as an input to the world.Water is an input. Electricity is an input. Literacy is an input. Connectivity is an input. Once a society depends on them, access stops being optional. Nobody needs government to build every well, power plant, school, or network. But everybody understands that a civilization cannot be organized around less than universal and reliable access to foundational inputs.Intelligence is reaching that level of importance now that we all know it is real.Government should not own it, operate it, or develop it. Quite the opposite. Companies are the right actors to build fast, compete hard, improve models, serve customers, and discover the real use cases. Self-interest is a useful framing here. Markets are good at finding demand, reducing costs, and turning invention into services people actually use.Companies are the right operators, developers, and owners. But that does not settle the real question of who owns the benefits. That is an economic question.If intelligence becomes metered infrastructure, what happens to the value it creates?The Ownership StackThis week's articles keep circling the same issue from different directions but in the nature of ‘circling' never quite nail it.Jamin Ball's “Own Your Weights” starts with the enterprise version of the question. Owning a model file is not enough. The durable asset is the loop: the data flywheel, the evaluations, the reinforcement system, the workflow learning, and the operating context that lets capability compound.Benedict Evans' “Ways to Think About Token Pricing” adds the market layer. Tokens may become essential, abundant, and cheap, like mobile data. But being essential does not guarantee that the token layer captures the value. The money may move up the stack to whoever owns the workflow, the customer, the distribution, or the application.Alex Karp's fight with the labs, reported in “Alex Karp Is Saying What Every Angry CEO Is Thinking About AI”, is the same argument in sharper enterprise language. Companies are afraid that model providers will not just sell intelligence, but learn from customer workflows and then move into the markets where those workflows create value. The “All-in” group are echoing Karp's view.And “What Is Loop Engineering, and Who Owns It?” names the new contested terrain. The loop is where intelligence meets the world. Whoever owns the loop owns the learning. Whoever owns the learning owns the compounding asset.That is why “who owns intelligence?” is not a slogan. It is the question under the model layer, the application layer, the enterprise layer, and the economic layer.Because intelligence is the product, the tools creating it are fragmented and competitive. So there is no logic in trying to discuss this at the level of a single company or set of tools and models.The Old Promise Was That Commerce Would Tame PowerThe essays this week give the historical backdrop.Deirdre McCloskey, in “What Really Caused the Industrial Revolution”, argues that modern growth came not simply from capital accumulation, but from a change in permission: ordinary people were allowed to innovate, trade, build, and be honored for it.That matters because intelligence could be another expansion of permission. It could make more people capable of building, learning, creating, coding, researching, translating, selling, and coordinating. It could lower the cost of competence.But only if access is broad.Paul Krugman's “AI in an Age of Oligarchy” warns that the same technology lands differently in different political economies. A new general-purpose technology entering a broad, open, upwardly mobile society is one thing. The same technology entering a concentrated economy, with extreme wealth and weak counterweights, is another.Tim O'Reilly's Economist essay, “Elon Musk is building a form of capitalism that Adam Smith would hate”, makes the governance point more directly. The old liberal hope was that commerce would tame arbitrary power. Markets, boards, courts, shareholders, disclosure, and competition would discipline the prince.But what if the prince uses markets to escape discipline?Henry Farrell's “political economy of billionaire derangement” pushes the same point. Founder culture, monopoly ambition, peer rivalry, weak correction mechanisms, and vast private control can amplify appetites rather than restrain them.The danger with intelligence is not that companies build it. They should. Companies build it, meter it, use public tolerance and public infrastructure to scale it, learn from everyone who uses it. All of those things are inevitable and healthy. Market forces will sort out winners from losers. The real danger is that the winners treat all of the surplus produced as purely private.Metered Intelligence Creates SurplusIf metering is not the problem, what is?The problem is pretending that metered intelligence creates value only for the metering entity. Metering water is only tolerated as a public good. If the public were blackmailed by a private water company with the threat of no water we would all rebel.Once we understand that the product of AI is intelligence we can see that every time intelligence is used, there is the immediate transaction: the user pays, the provider serves.But there is also system value. Usage creates signals. Workflows reveal patterns. Prompts, corrections, failures, preferences, integrations, edge cases, and business processes all help define where intelligence is useful and how it should improve. Intelligence breeds intelligence.Even when customer data is contractually protected, the market learns. The platform learns where demand is. The product team learns which workflows matter. The ecosystem learns which jobs are vulnerable, which tasks are automatable, and which parts of the economy can be reorganized around machine intelligence.So the surplus is not born in a vacuum.It rests on public science, public education, public data exhaust, public law, public infrastructure, public energy systems, public tolerance for data centers, and billions of human interactions. It is served by companies, but it is not made only by companies.This is why “Americans Deserve a Dividend From AI Companies' Riches” belongs at the center of this week's issue. The detail can be debated. The principle is harder to dismiss. If intelligence becomes a new foundational resource, then some part of the wealth it creates should flow back to the people whose society makes it possible. Intelligence did not suddenly appear. AI is built on the entire history of human intelligence. It benefits from it and at the same time evolves it.Not Nationalization. A Human Wealth Fund.If intelligence belongs to everybody, some conclude that government ownership of intelligence is the right outcome.Governments are not well suited to build, operate, or improve intelligence. They will move too slowly, regulate too early, politicize the wrong things, and confuse economic participation with operational control.Andrew McAfee's “Why I Didn't Sign the AI Open Letter” is useful here. His objection is not that the technology is unimportant. It is that steering too hard before we understand the shape of the change can become its own failure mode. Marc Andreessen's satire of AI regulation is less policy than temperament, but it captures a real Silicon Valley fear: that regulation can become permission, capture, and incumbency before it becomes wisdom.That fear should be taken seriously.But it does not answer the economic question. It answers only the operational one.How can the economic benefits of intelligence be distributed? The better answer is a sovereign human wealth fund.Call it a sovereign wealth fund if you must, but the phrase is too national. Intelligence will not respect borders. The leading companies are global. The models, chips, data centers, agents, platforms, and workflows will be transnational from the beginning. If the value created by intelligence is global, then the mechanism for sharing some of that value should begin with the companies global enough to capture it. The nice thing about xAI, OpenAI, and Anthropic is that they are supranational.These companies own and operate intelligence. Let them compete. Let them profit. Let them keep the incentives that make the system improve. But if intelligence is the new water, the wealth it creates cannot belong only to the companies that meter it. And they, themselves, have the power to fix it, even more than governments.Access will become a Human Right; Ownership Is the Economic DesignThis is where human rights come in. There is no right to access an AI model, yet. But there will soon be a need to change that.Not as a claim that every person is entitled to every frontier model at every moment for free. That is not serious. Capacity has costs. Models have costs. Inference has costs. Data centers have costs. Although those costs will decline over time, possibly quite quickly as self-learning models address costs.The claim is more basic: in a world where intelligence becomes a primary input into education, work, health, science, citizenship, creativity, and economic agency, baseline access to intelligence starts to look like a civic requirement.That could mean public access layers. It could mean education credits. It could mean open models. It could mean AI dividends. It could mean public-interest compute. It could mean taxes on rents. It could mean a company-initiated human wealth fund that returns some of the upside to society without handing the operating system to the state. The latter could couple wealth growth with universal distribution of ownership.The exact mechanism matters. But the distinction matters more.Government should not own intelligence. It should be universally available. And people should have a claim on the wealth intelligence creates.The Frontier Is Also PhysicalThe abstraction is not weightless.“The Fight Against AI Data Centers Is Just Beginning”, “New York becomes the first state to enact a data center moratorium”, Reuters on pollution from Musk's xAI power project, and DataGravity's “Who Captures Value in AI Infrastructure?” all say the same thing from the ground up.Intelligence uses land. It uses power. It uses water. It uses chips. It uses grid capacity. It uses neighborhoods. It uses public patience.That makes the value question unavoidable. A society can accept the buildout if the buildout is legible as shared progress. It will resist it if the costs are local, the profits are private, and the benefits feel enclosed.Who Owns the “Loop”?The week ends where it began.“Anthropic and Blackstone” are betting that implementation is the next trillion-dollar business. “Vint Cerf” is working on identity for agents on the open internet. “GPT-Red” points toward systems that improve their own robustness. “Kimi K3” adds another open frontier model to the global mix.The model race continues. The deployment race is accelerating. The governance race is behind.My view is this:The central product of this era is intelligence. Companies have figured out how to capture it, package it, serve it, and meter it. That is good. It should stay in the hands of builders who have the incentive to make it better.But intelligence is too foundational to become just another private toll booth. A significant part of it will turn out to be free to users.As intelligence becomes a general-purpose resource, then access to it becomes a human-capability question, and the surplus from it becomes an economic-justice question. Not because government should run it. Because government should not run it. The operating layer belongs with companies. The wealth question belongs with everyone. But companies are best placed to turn that into a process of distribution.The question is not whether companies should build intelligence. They should.The question is whether humanity gets a stake in the wealth created by the thing that may soon become its most important shared input.Contents* Essays* Deirdre McCloskey on What Really Caused the Industrial Revolution* AI in an Age of Oligarchy* Elon Musk is building a form of capitalism that Adam Smith would hate* Murky Mirror: Truth and Consequences* The political economy of billionaire derangement* Is there any “oligarchy” to fight?* AI* Nearly 200 Economists and Tech Leaders Warn of A.I. Threats* Why I Didn't Sign the AI Open Letter* Own Your Weights* Ways to Think About Token Pricing* Alex Karp Is Saying What Every Angry CEO Is Thinking About AI* The AI Agents Are Coming for Microsoft Office* What Is Loop Engineering, and Who Owns It?* The Fight Against AI Data Centers Is Just Beginning* 6 months to live for open models* Americans Deserve a Dividend From AI Companies' Riches* Who Gets to Define the Frontier?* GPT-Red: Unlocking Self-Improvement for Robustness* Anthropic, Blackstone bet the next trillion-dollar AI business is implementation, not just models* Vint Cerf is working on a plan to unleash AI agents on the open internet* xai-org/grok-build, now open source* The Pulse: What can we learn from Bun's rapid Rust rewrite with AI?* Orphan risks at the frontier of artificial intelligence* The Lab of the Future Should Feel Like a Data Center* Why AMI Labs' Alexandre LeBrun won't call his AI “AGI” or “superintelligence”* Kimi K3 Tech Blog: Open Frontier Intelligence* Venture Capital* Three Years In* Venture Has Rarely Looked More Bifurcated* The Best Angel Investors in the US: Who Backs the Most Unicorns, and Who's Active Now* Are Prediction Markets Doomed to Fail?* Regulation* Exclusive: The Next Frontier of the Deportation Wars: College Campuses* The Supreme Court Broke Independent Agencies. Here's a Way to Slow the Damage.* India's crackdown on a new WhatsApp feature risks setting a global precedent* Let's build a children's public internet* Computer cops* Google is better at playing the AI regulations game* Infrastructure* Who Captures Value in AI Infrastructure?* New York becomes the first state to enact a data center moratorium* Pollution from Musk's unpermitted xAI power project hits hardest in Black communities* Interview of the Week* The End of the End of Geography* Startup of the Week* Radical AI's Joseph Krause: The Scientist Building The “Waymo” Lab For New Materials* Post of the Week* Marc Andreessen on AI RegulationEssaysDeirdre McCloskey on What Really Caused the Industrial RevolutionYascha Mounk and Deirdre McCloskey | Persuasion | July 11, 2026Yascha Mounk interviews Deirdre McCloskey about her argument that the modern world's economic liftoff came less from capital accumulation than from a change in ideas. McCloskey says both left and right versions of the conventional story rely too heavily on investment: the left stresses exploitation and surplus value, while the right stresses virtuous saving by capitalists. Her objection is historical and economic. Human beings had always invested, from irrigation works and Roman roads to seed grain, and simple accumulation quickly runs into diminishing returns.McCloskey's alternative is that northwestern Europe, first Holland, then Britain and Scotland, and then the North American colonies, developed a liberal ideology that changed who was allowed to innovate and be honored for it. The conversation links that shift to the erosion of inherited hierarchy, the spread of dignity for ordinary commercial life, and a moral vocabulary in which liberalism is not merely procedural but connected to virtues and values. The point is not that machines, coal, trade, and institutions did not matter, but that they do not explain the scale and timing of modern enrichment without a cultural permission structure for innovation.The interview also turns to the contemporary defense of liberalism. Mounk frames the series around the worry that liberalism is often treated as too thin to command allegiance, while its opponents speak more directly to moral passions. McCloskey's case is that liberal societies became rich because they dignified experimentation and ordinary enterprise, and that liberals need to recover the moral language behind that claim.Read moreAI in an Age of OligarchyPaul Krugman | Paul Krugman | July 12, 2026Paul Krugman frames AI as a major technological shock arriving inside an already unequal political economy. The post says AI's economic and social effects may take years to understand, but argues that the setting matters now: America has much greater wealth concentration and political inequality than it did in the 1950s and 1960s, when progressive taxation, stronger regulation, and more active antitrust might have contained some of the destructive effects of a new technology.Krugman's opening claim is that the same technology would likely have different consequences in a more level society. In today's United States, he writes, extreme wealth is both a cause and effect of policies that favor a small elite, including low effective taxes on capital and high incomes, weak enforcement of worker protections and antitrust, and cuts to programs that benefit ordinary Americans.The article is explicitly more about oligarchy than AI. Krugman says the paid sections document the rise of the “.0002%,” the economics and politics of extreme wealth, how oligarchy will shape AI's impact, and possible policy paths. His caveat is that AI itself may still produce a pushback against oligarchy, but absent that, he expects the pre-existing concentration of wealth and power to magnify AI's downsides.Read moreElon Musk is building a form of capitalism that Adam Smith would hateAuthor: Tim O'Reilly Published: July 12, 2026Tim O'Reilly argues that Elon Musk is using the legal forms of shareholder capitalism to escape the restraints that shareholder capitalism was supposed to impose. The article begins with SpaceX's public-market structure: ordinary public investors get little meaningful governance power, Musk keeps roughly 85 percent of the votes through super-voting shares, buyers waive jury trials and class actions, the company qualifies as controlled, and removal of Musk depends on the share class he controls. In O'Reilly's framing, that is not ordinary founder control; it is a design for being answerable to no one, possibly beyond Musk's own lifetime.The killer detail is the article's turn through Albert Hirschman, Montesquieu, James Steuart, Adam Smith, and Keynes. Older defenses of commerce held that markets would tame princely passions because the self-interest of merchants was safer than arbitrary rule. O'Reilly says Musk reverses that hope. The market discipline that was supposed to cage the prince has become the lever by which the prince raises capital, removes feedback loops, and carries private power into politics, government, Mars, robots, AI, or whatever ambition comes next.The pull is the link to AI governance. O'Reilly says corporations are already a kind of artificial intelligence: narrow-input systems that act at a scale no individual human can match. Their partial controls include independent boards, shareholder votes, courts, disclosure, regulators, public pressure, and activism. If the leaders building frontier AI strip those alignment mechanisms out of their own companies, the governance of the company becomes a preview of the governance of the machine.Read more: The EconomistMurky Mirror: Truth and ConsequencesAuthor: Esther Dyson Published: July 14, 2026Esther Dyson argues that today's institutional crisis is better viewed through the 14th century than through recent political history. Using Barbara Tuchman's A Distant Mirror as her frame, she compares a world of famine, plague, church schism, feudal predation, and purposeless war with a present in which institutions again feel brittle, incentives are badly aligned, and power is shifting into forms that are hard to govern.The killer detail is the historical analogy between land, corporations, and AI. Dyson moves from nobles who controlled serfs and territory, to the East India Company as a quasi-sovereign business, to today's AI systems and data centers as a possible new sector that crosses and weakens both nation-states and companies. The question is whether AI becomes a new kind of private land, owned by a new nobility, or an open prairie that many people can cultivate.The pull is human attention. Dyson says the central question is not what AI will do to people, but how people will react to it: whether they can value love, kindness, embodied attention, and artisanal human presence in a world of seductive artificial offerings.Read more: SourceThe political economy of billionaire derangementAuthor: Henry Farrell Published: July 15, 2026Henry Farrell argues that the visible political radicalization of some Silicon Valley billionaires is not a random personality quirk, but a product of the political economy that made them. Starting from Tyler Cowen's dismissal of “billionaire derangement syndrome” and Tim O'Reilly's warning that Elon Musk is using shareholder capitalism to escape shareholder restraint, Farrell flips the phrase: the question is why billionaires themselves can become deranged.The killer detail is Farrell's use of Peter Thiel as both theorist and example. Thiel's Stanford lectures described startups as monarchies and founders as figures vested with unusual power, while Silicon Valley culture rewarded eccentricity, monopoly ambition, and founder exceptionalism. Farrell says those ideas combined with dense founder-investor networks, peer rivalry, and weak correction mechanisms to amplify rather than discipline princely appetites.The pull is the ideological problem for classical liberals who once saw tech wealth as an ally of markets and freedom. Farrell says commerce did not tame the passions; in parts of Silicon Valley, the passions have begun to devour markets, institutions, and the liberal story that justified them.Read more: SourceIs there any “oligarchy” to fight?Matthew Yglesias | Slow Boring | July 16, 2026Matthew Yglesias argues that “oligarchy” is a rhetorically powerful but analytically loose way to describe American politics. The post begins from Bernie Sanders' “Fighting Oligarchy” tour, Amy Klobuchar's warning about a MAGA “broligarchy,” and the long afterlife of the Martin Gilens and Benjamin Page paper that was widely summarized as showing that only the rich matter in policy outcomes. Yglesias says the evidence supports a weaker claim: affluent people and business leaders have unusual access and influence, but that is not the same as rule by a small cabal.His main distinction is between inequality and oligarchy. The Gilens-Page measure treated the top 10 percent of households as “the wealthy,” and later critics found that rich and middle-class preferences usually align; in the cases where they differ, the rich win about 53 percent of the time. Yglesias also says business executives get special access partly because their decisions are materially important to communities, jobs, investment, and local tax bases, not only because of campaign donations.The post preserves Jerusalem Demsas' counterpoint from their podcast discussion: privileged donor and business access can still violate democratic equality even if the oligarchy label overstates the structure of power. Yglesias' narrower claim is that Democrats should be precise about what problem they are trying to solve, because donor influence can also push the party left on climate and cultural issues in ways that alienate many voters.Read more: Slow BoringAINearly 200 Economists and Tech Leaders Warn of A.I. ThreatsAuthor: Ben Casselman Published: July 13, 2026Ben Casselman reports on “We Must Act Now,” a statement warning that artificial intelligence could transform the economy faster than any previous technology and that policymakers need to move faster to understand and respond. The statement says AI may become radically more powerful over the next 10 years, bringing risks such as large-scale job displacement as well as opportunities such as higher living standards. Nearly 200 people signed, including 15 Nobel laureates, the chief economists of OpenAI and Anthropic, Anthropic co-founder Jack Clark, former Google CEO Eric Schmidt, and venture capitalist Vinod Khosla.The killer detail is who joined the warning. Casselman notes that the signatories include economists who have historically been skeptical of Silicon Valley's most dramatic AI job-loss forecasts, including Daron Acemoglu and Simon Johnson, the MIT professors who won the 2024 Nobel in economics. Erik Brynjolfsson, who helped organize the statement, says there has been a notable change in the profession and that economists and policymakers are not ready for the “tsunami” he sees coming.The pull is the measurement problem. The statement does not offer a specific policy menu, but calls for economists, policymakers, and industry leaders to understand the economics of transformative AI and steer it toward complementing humans. Brynjolfsson says one high priority is better data on AI's spread and impact, because current measures tell conflicting stories about job losses and which workers are most exposed.Read more: The New York TimesWhy I Didn't Sign the AI Open LetterAuthor: Andrew McAfee Published: July 13, 2026Andrew McAfee explains why he did not sign “We Must Act Now,” the AI economy statement organized in part by his longtime collaborator Erik Brynjolfsson. McAfee agrees with the letter's starting point that AI is likely to become radically more powerful over the next decade and that it is a general-purpose technology. His objection is not to urgency or to studying AI's economic effects, but to the framing of risk, displacement, and institutional steering as the first move.The killer detail is McAfee's line edit. He says the original letter comes close, then “bounces off the crossbar” by calling for incentives, guardrails, and institutions to steer AI before we know enough about its actual impacts. He points to mixed current evidence: labor-market canaries, but also rising software job postings, low unemployment for younger workers, rising real median income, and claims that AI-adopting companies are adding workers faster than low-adopting peers. His worry is that the letter leans toward upstream governance and dirigisme when the evidence may call for capability building instead.The pull is his replacement statement. McAfee keeps the three-paragraph structure but changes the emphasis: AI is likely to become radically more powerful; like earlier world-changing technologies it will raise living standards while also bringing harms and shocks; and economists, policymakers, and technology leaders should build the capabilities to respond quickly and effectively. It is a concise version of the permissionless-innovation case inside the AI policy debate.Read more: The Geek WayOwn Your WeightsAuthor: Jamin Ball Published: July 10, 2026Jamin Ball argues that the enterprise AI debate about whether companies should “own their weights” or rent models from frontier labs is asking too narrow a question. A model weight file gives a company control over a point-in-time artifact, but not durable control over the capability stack. In his framing, the weight file is a melting ice cube: it does not get worse in absolute terms, but it falls behind as frontier systems improve and enterprise needs change.The killer detail is what Ball says companies really need to own: the data flywheel, reinforcement learning infrastructure, and evaluation harness that produce and improve the model. Simply deploying an open-weights model and declaring sovereignty leaves the enterprise with yesterday's capability and no way to compound workflow-specific learning.The pull is that enterprise AI control may be less about model ownership than operating ownership. The defensible layer is the system that turns company data, edge cases, business definitions, and evaluations into continuously improving performance.Read more: Clouded JudgementWays to Think About Token PricingAuthor: Benedict Evans Published: July 9, 2026Benedict Evans argues that today's AI token prices are a temporary signal from a supply-constrained market, not a reliable guide to long-term value capture. The open question is whether foundation models keep durable pricing power or become commodity infrastructure as data-center capacity, inference efficiency, and model competition all shift. His current read is that the visible market dynamics point toward commoditization unless something materially changes.The killer detail is the mobile data analogy. Evans says cellular networks became a trillion-dollar industry with hundreds of billions in capex after data usage exploded, but carrier stocks went nowhere because value moved up the stack. Tokens may behave similarly: an opaque unit tied to marginal cost, sold through bundles, essential to everything, yet not necessarily where profits accrue.The pull is uncertainty, not prediction. Evans lists paths to model dominance, including network effects, less competition, regulation, export controls, or a lab pulling ahead on execution, but says each requires a new fact not yet visible. Without that change, the model layer looks more like infrastructure beneath the products that capture value.Read more: SourceAlex Karp Is Saying What Every Angry CEO Is Thinking About AIAuthor: Tim Higgins Published: July 11, 2026Tim Higgins reports that Palantir CEO Alex Karp has turned corporate frustration with AI labs into a public argument about enterprise control. Palantir released a white paper, “Institutional Sovereignty in the Age of AI,” laying out steps companies and governments can take to protect themselves from OpenAI, Anthropic, and other foundation-model providers. The article links that paper to Karp's CNBC appearance, where he said “something has gone completely wrong” in the relationship between AI labs and customers and argued that enterprises are paying for tokens that create little value.The killer detail is the value-capture question. Higgins writes that Karp's critique has resonated because AI labs may gain power and insight from customer data, workflows, and decision-making, even when enterprise policies say customer data are not used for training. David Sacks amplified the concern by arguing that Anthropic is moving from the model layer into vertical applications such as science, security, legal, and coding, raising the fear that model providers will watch where value is being created and then move into those markets directly.The pull is that Karp is not alone, even if his style is unusually combative. Higgins notes that Satya Nadella has also warned that companies need to retain the learnings created when they use AI models, while Mark Zuckerberg has framed Meta's new model release partly around lower-cost frontier intelligence. The article presents Karp's campaign as one sign that established technology companies and large enterprises are trying to define where they fit when AI labs become central infrastructure, application competitors, and potential IPO giants at the same time.Read more: The Wall Street JournalThe AI Agents Are Coming for Microsoft OfficeAlex Wilhelm | Cautious Optimism | July 11, 2026Alex Wilhelm argues that one of the week's quieter AI questions is whether the productivity market that Microsoft successfully moved into subscription software is now being attacked by agentic tools. The piece begins with the infrastructure backdrop: SK Hynix raised $26.5 billion in a U.S. listing while building U.S. HBM and advanced-packaging capacity, and memory, chip, and foundry companies are now priced for sustained AI demand.Wilhelm then says the AI conversation has shifted quickly from raw capability to cost per task. He cites new model releases and vendor language emphasizing cheaper agentic and coding models, faster performance, and lower dollars per task. That matters because lower costs make it more plausible for AI systems to take on routine knowledge work at scale rather than remain a premium coding assistant market.The core of the article is Microsoft Office. Wilhelm notes that Microsoft turned Office from a one-time purchase into Microsoft 365, a large recurring revenue business with tens of millions of subscribers and a major productivity segment. Now, he says, late-stage unicorns and AI labs are pushing into the same territory: Anthropic's Cowork was reportedly used mostly outside software development, OpenAI merged ChatGPT and Codex into a tool for creating sheets, slides, docs, web apps, and long-running work, and other companies are building agentic coworkers that connect business data to documents, workflows, schedules, alerts, and apps.The article's caveat is that Microsoft has survived major platform shifts before. The argument is not that Office disappears quickly, but that the definition of office software is broadening from documents and spreadsheets into AI systems that can create, monitor, and act across workplace data.Read moreWhat Is Loop Engineering, and Who Owns It?Author: Nilesh Barla Published: July 11, 2026Nilesh Barla argues that “loop engineering” is becoming a distinct discipline because production AI agents now fail less at single prompts than at runtime: when to stop, what state to preserve, and how to recover after a bad step. Prompt engineering shapes one model call, and context engineering shapes what the model sees, but loop engineering shapes what a sequence of calls actually does.The killer detail is the three-primitives frame. Barla says a real agent loop needs halt conditions, state carryover, and recovery paths, then maps teams across five maturity levels. At the lowest level, an agent is just a model call in a for-loop with a step cap and raw history; by the higher levels, the system has structured state, explicit planning, replay, evaluation, and self-repair.The pull is organizational. If agents are becoming production systems rather than demos, someone has to own the runtime itself. The loop engineer is the role Barla gives to the person responsible for making long-running agent work dependable.Read more: Adaline LabsThe Fight Against AI Data Centers Is Just BeginningEmma Roth | The Verge | July 12, 2026Emma Roth argues that community resistance to data centers has moved from an early warning sign into a national political fight as AI facilities grow larger, more power-hungry, and more visible to nearby residents. The article starts with Apple's failed 2015 plan for a $1 billion data center in Athenry, Ireland, where a small group of residents challenged the project over noise, light pollution, flooding, traffic, and wildlife effects until Apple abandoned it in 2018.The current data-center buildout is presented as much larger and more contentious. Roth writes that residents now cite rising energy costs, water quality, noise, light pollution, and greenhouse gas emissions, while the U.S. Energy Information Administration expects commercial energy demand to surpass residential demand this year because of AI data centers and Goldman Sachs expects data-center power demand to double by 2027.The central evidence comes from Data Center Watch, which says protesters blocked or delayed at least 75 U.S. projects worth $130 billion from January to March, with active opposition groups more than doubling from 396 at the end of 2025 to 833 by the end of the first quarter of 2026. Roth also cites QTS abandoning a $12 billion Wisconsin campus, Delaware City regulators blocking a 580-acre project under the Coastal Zone Act, opposition stopping a QTS project in Prince William County, and pressure that pushed Kevin O'Leary to downsize the proposed 40,000-acre Project Stratos in Utah.The policy section describes a split between federal acceleration and local resistance. President Trump has treated data centers as part of the AI race with China and fast-tracked construction, while some Republican candidates are distancing themselves from that position ahead of midterms. Sanders and Ocasio-Cortez have proposed a moratorium until price and environmental protections exist, bipartisan lawmakers are backing ratepayer-protection measures, and states including Florida, Idaho, and Washington have passed rules on cost shifting, water use, and tax breaks. Roth's caveat is that the policy patchwork is still incomplete, leaving many communities to fight project by project.Read more6 months to live for open modelsAuthor: Nathan Lambert Published: July 12, 2026Nathan Lambert argues that open-weight AI models are facing their most serious policy test so far because U.S. officials are beginning to discuss concrete controls rather than abstract safety concerns. He says reported White House conversations about a new executive order may initially target Chinese-origin models and government use, but could create a broader review habit for frontier open models. His forecast is that a model above the capability range of GPT-5.5, Claude Opus 4.8, or GLM-5.2 could trigger a ban or indefinite delay within six months.The post separates two policy fights that are becoming intertwined: distillation and frontier capability. Lambert says the distillation campaign against Chinese models has become a form of regulatory capture because Anthropic and other closed-model companies would gain economically if Chinese open models were banned. He does not dismiss IP protection, but argues that if a closed model's capabilities are dangerous enough to justify restricting open models, the lab also has to explain why those capabilities are exposed through a queryable API. He cites unauthorized access to Anthropic's Mythos private beta as evidence that APIs are not automatically secure.The broader claim is that a unilateral U.S. ban would hurt positive actors more than bad actors if comparable open models remain available elsewhere. Lambert says the only durable ceiling would require global agreement, which does not exist, and that open models can improve safety by allowing broad inspection, adaptation, and understanding. His proposed near-term off-ramps are a strong U.S. open model release from companies such as Microsoft, Meta, or Reflection, and a broader coalition of open-source beneficiaries lobbying for safe rollout rather than prohibition.Read more: SourceAmericans Deserve a Dividend From AI Companies' RichesAuthor: Scott Stanford Published: July 14, 2026Scott Stanford argues that proposals to give the government a stake in AI companies miss the point unless ordinary citizens directly receive and control the upside. Sam Altman has discussed giving up equity in OpenAI, Washington already owns a stake in Intel, Nvidia is sharing China chip revenue, and Bernie Sanders wants large AI labs to contribute half their stock to a sovereign wealth fund. Stanford says those ideas all park value with the state, not with people.The killer detail is New Carlisle, Indiana, where AWS's Project Rainier is turning cornfields into one of the world's largest AI superclusters. The project is planned to run up to a million chips, draw more than two gigawatts of power, and represents an investment that has grown from $11 billion to $13.8 billion. Stanford uses that local transformation to argue that AI's public bargain should be visible at the household level.The pull is design. A citizen AI dividend would have to specify who earns a stake, how they hold it, and when they see cash. Without that mechanism, the AI wealth debate remains a fight over government balance sheets rather than public ownership.Read more: SourceWho Gets to Define the Frontier?Author: Mark Daley Published: July 14, 2026Mark Daley argues that Demis Hassabis is right to call for a serious institution to verify frontier AI systems, but that the power to test models is also the power to govern them. Hassabis's proposed Frontier AI Standards Body would get privileged pre-release access to advanced models, testing compute, held-out evaluations, support from national labs and security agencies, third-party auditors, and eventually authority to block models from the American market or coordinate a slowdown.The killer detail is Daley's constitutional objection. He says the proposal sometimes looks like a scientific lab, a standards body, an industry regulator, a licensing authority, and an emergency security council at once. Combining those roles because each requires technical expertise would be like putting the central bank, auditor-general, and Supreme Court in one building and calling it efficient.The pull is standard-setting. Daley's concern is not that verification is unnecessary, but that whoever writes the tests, decides what passes, adjudicates disputes, and grants market access may end up defining the frontier itself.Read more: SourceGPT-Red: Unlocking Self-Improvement for RobustnessOpenAI | OpenAI | July 15, 2026OpenAI describes GPT-Red as an internal automated red-teaming model trained to find prompt-injection vulnerabilities at a scale human red teams cannot match. The post says AI systems increasingly encounter third-party data through browsers, connected apps, local files, and tools, creating opportunities for malicious instructions hidden in emails, webpages, tool responses, or code repositories. Human red-teaming remains part of OpenAI's safety process, but the company says it is time-intensive and cannot generate enough diverse adversarial examples for model training.The system is trained through self-play reinforcement learning, with GPT-Red rewarded for eliciting valid failures and defender models rewarded for resisting attacks while still completing their tasks. OpenAI says the training environments specify threat models across settings such as local files, webpage banners, email bodies, and tool outputs. The model is kept separate from deployed production models because it is intentionally trained with malicious capabilities.OpenAI reports that GPT-Red generalized beyond its training set, including an internal replication of the indirect prompt-injection arena from Dziemian et al. (2025), where it found successful attacks in 84% of scenarios compared with 13% for human red-teamers. The post also says GPT-Red transferred attacks from simulation to a live autonomous vending-machine agent, causing price changes and order cancellations, and outperformed a prompted GPT-5.5 baseline against a Codex CLI agent on held-out data-exfiltration tasks.The article's main robustness claim is that OpenAI has used GPT-Red and predecessor models in training since GPT-5.3, with later GPT releases becoming more resistant to prompt injections. It says GPT-5.6 Sol has six times fewer failures on OpenAI's hardest direct prompt-injection benchmark than the best production model from four months earlier, that a “Fake Chain-of-Thought” attack class fell from more than 95% success against GPT-5.1 to below 10% against GPT-5.6 Sol, and that GPT-5.6 Sol fails on only 0.05% of GPT-Red's direct prompt injections. OpenAI says general capabilities and targeted over-refusal evaluations were not harmed, and says a preprint with more details will follow.Read moreAnthropic, Blackstone bet the next trillion-dollar AI business is implementation, not just modelsRebecca Bellan | TechCrunch | July 15, 2026Rebecca Bellan reports that Ode with Anthropic is the $1.5 billion AI implementation company launched by Anthropic with Blackstone, Hellman & Friedman, Goldman Sachs, and other backers. The article says the venture reflects a growing belief among frontier AI labs that enterprise adoption requires more than better models: customers need engineers who can embed inside businesses and turn AI into working systems.Ode was originally conceived by Blackstone after it used both large consulting firms and smaller AI services boutiques across its portfolio companies. TechCrunch reports that Fractional AI, an AI engineering services startup, stood out and was acquired by the joint venture shortly after the venture was announced. Fractional now forms the foundation of Ode, which has 100 engineers and works closely with Anthropic's applied AI team to identify where the technology can affect specific businesses.Ode CEO Chris Taylor tells TechCrunch that the company could someday become a trillion-dollar business if it scales without losing quality. He says an ideal customer is one whose CEO treats the AI project as a top one or two priority, whether it is a major product feature or the reworking of a core business process. Ode will operate under a “Claude-first” principle, using Anthropic technology whenever possible, but the article says it can use rival AI products when needed.The article's central implementation argument comes from Ode chief technologist Eddie Siegel, who says model selection matters but is not where most of the engineering effort goes. He compares it to the choice of programming language in software: one ingredient in a system that still has to be engineered. Bellan writes that Ode's challenge is hiring and training enough elite generalist engineers, many of them former founders, while competing with OpenAI's The Deployment Company and consulting giants that have built their own forward-deployed engineering teams.Read moreVint Cerf is working on a plan to unleash AI agents on the open internetTim Fernholz | TechCrunch | July 15, 2026Tim Fernholz reports that Vint Cerf, after leaving Google, is advising Innovation Labs on an open architecture for identifying AI agents online. Innovation Labs is a subsidiary of Identity Digital, a DNS registry company, and its proposal is to use domain-name infrastructure as part of a system for agent identity, accountability, and auditability. The premise is that agents will need a way to identify themselves if they move beyond proprietary systems and begin interacting across the open internet.The concrete proposal is DNSid, a registry that links an AI agent to an existing internet domain and uses cryptographic proofs to log its registration over time. Innovation Labs says it is trialing the standard with unnamed hyperscalers and identity companies. Cerf frames the problem around authority and accountability: what authority an agent has, where that authority came from, who is accountable for the agent's behavior, how its identity is established, and why anyone should trust it.The article's caveat is that standards are still emerging and agents are more active than static domains. Cerf says the period may be both fascinating and exasperating because the functionality is powerful and interoperability is unresolved. He compares the adoption problem to TCP/IP: competing systems may not work together until users push for functional interoperation. He also says an agentic economy is not inevitable, but that people will try to build it because delegating work to agents will be easier.Read more: TechCrunchxai-org/grok-build, now open sourceAuthor: Simon Willison Published: July 15, 2026Simon Willison argues that xAI's decision to open-source Grok Build is best understood as a trust repair move after a severe privacy failure. The CLI had triggered backlash when users realized that running it in a directory could upload the entire directory to xAI's Google Cloud buckets, including one user's reported SSH keys, password manager database, documents, photos, and videos. xAI disabled the feature, said previously retained coding data would be deleted, and released the code under Apache 2.0.The killer detail is what the codebase reveals. Willison counts 844,530 lines of Rust, only about 3% of which appears vendored, and finds remnants of the upload system still present but disabled: gcs.rs contains Google Cloud upload code, while upload_session_state() now returns a hard-coded session_state_upload_unavailable error. He also notes copied or ported tool implementations from Codex and OpenCode, prompt files, and a terminal Mermaid renderer.The pull is that terminal coding agents are becoming large, intricate software systems in their own right. The privacy failure mattered because these tools operate inside the directories where developers keep their most sensitive work; the open-source release matters because trust now depends on inspecting what an agent can see, send, and do.Read more: SourceThe Pulse: What can we learn from Bun's rapid Rust rewrite with AI?Author: Gergely Orosz and Ivan Klaric Published: July 16, 2026Gergely Orosz and Ivan Klaric argue that Bun's AI-assisted rewrite from Zig to Rust is a practical sign of how software engineering changes when models can take on large, bounded migrations with clear feedback loops. The piece does not treat the rewrite as magic: Jarred Sumner first spent hours turning design judgment into a detailed porting guide, then used adversarial review, parallel agents, compiler errors, and tests to force the work toward correctness.The killer detail is the scale. Bun had 535,496 lines of Zig, 1,448 files, and 22 million monthly downloads, making a conventional rewrite a year-long freeze the team could not justify. Using Fable, Sumner split the work across 64 agents, produced about 6,500 commits, and got the migration done in 11 days at an estimated API cost of $165,000.The pull is economic, not theatrical. If a one- or two-year migration can become an 11-day project, AI coding is not just faster autocomplete; it changes which technical debts are worth paying down.Read more: SourceOrphan risks at the frontier of artificial intelligenceAuthor: Andrew Maynard Published: July 16, 2026Andrew Maynard argues that frontier AI safety frameworks are creating “orphan risks”: harms that companies can see, but do not formally own because they are hard to quantify, do not fit catastrophic-risk thresholds, or fall outside audit-friendly compliance machinery. His target is not existing frontier safety work, but the narrowing effect that happens when private companies decide which risks count as governable.The killer detail is Maynard's contrast between measurable model dangers and threats to value. He points to Meta's three-day Galactica collapse, OpenAI's 2023 board crisis, safety-team departures, and wellbeing litigation as examples of risks that damaged trust, culture, legitimacy, or users without fitting cleanly into conventional model-risk categories. The proposed fix is an orphan-risk register: a public record of risks a company considered and chose not to manage, with reasons.The pull is accountability. Frontier developers' internal scoping choices have become a de facto layer of public governance, so the question is no longer only which risks they manage, but which risks they quietly leave outside the frame.Read more: SourceThe Lab of the Future Should Feel Like a Data CenterLatent.Space with Andy Beam and Rafa Gomez-Bombarelli | Latent.Space | July 16, 2026Latent.Space interviews Lila Sciences CTO Andy Beam and chief science officer for physical sciences Rafa Gomez-Bombarelli about the company's attempt to build an AI-run science factory. The post describes Lila's thesis as treating the lab itself as an “infinite token generator”: if internet data drove the first era of AI scaling, experimentally verified scientific data may be the next scarce training source. Lila is trying to produce that data with robotics, lab instruments, orchestration software, and AI models wired into the wet lab.The central analogy is the lab as data center. Instruments are nodes on a graph, a magnetically levitating transport layer moves materials between them, and experiment scheduling looks like a compute queue. Beam says Lila is not simply an automation company, because the point is not just throughput; it is flexibility, generalization, and experiment capture. The post says Lila has built more than 10 trillion experimentally validated “scientific reasoning tokens,” not internet text or biological sequences.The interview ranges across biology, chemistry, drug discovery, materials science, and the limits of automation. It notes that Lila rebuilt one gas-sorption measurement to run roughly 2,500 times faster, claims its general models can transfer priors from small-molecule chemistry to metal-organic frameworks for carbon capture, and describes model-suggested platinum-group-free electrocatalysts that moved from looking boring or wrong to becoming strong performers. The caveats are physical: experiments have runtimes, biology cannot always be accelerated, chains of thought can be unreliable narrators, and reward hacking becomes more dangerous when a model controls a real lab.Read more: Latent.SpaceWhy AMI Labs' Alexandre LeBrun won't call his AI “AGI” or “superintelligence”Kate Park | TechCrunch | July 16, 2026Kate Park interviews AMI Labs CEO Alexandre LeBrun about why Yann LeCun's world-model startup avoids the language of “AGI” and “superintelligence.” LeBrun says the terms are not useful because they lack stable definitions: “We never used the word AGI. And I just noticed that nobody is using it anymore; they switched to superintelligence.” His argument is that the practical frontier is not a label, but whether AI systems can understand and predict real-world states.The article explains the world-model thesis by contrasting language prediction with physical-state prediction. A large language model predicts the next word; a world model predicts the next state, such as what happens when a glass tips over. LeBrun says LLMs remain complementary and efficient for language, but the physical world is where current AI is weak. Robotics is the clearest case: hardware has advanced quickly, but robots are still brittle outside controlled routines because they lack context and situational understanding.AMI is still pre-product, but TechCrunch reports that LeBrun was in Seoul looking for industrial partners, researchers, and global companies. He says world models cannot be built entirely inside a lab because they need access to real environments. That is why South Korea appeals to AMI: robotics, semiconductors, manufacturing, and fast adoption create the kind of hardware-heavy context that software-only AI has barely touched.Read more: TechCrunchKimi K3 Tech Blog: Open Frontier IntelligenceKimi | Kimi | July 16, 2026Kimi introduces Kimi K3 as an open 3T-class frontier model aimed at coding, knowledge work, reasoning, multimodality, and long-context agentic use. The source describes the model as a 2.8T-parameter system built on Kimi Delta Attention and Attention Residuals, with native multimodality and a 1M-token context window. It says Moonshot AI plans to release model weights by July 27.The post presents K3 through benchmark and use-case sections rather than as a general product announcement. It reports results across coding, productivity, agentic, and multimodal evaluations, including DeepSWE, Terminal-Bench 2.1, Program Bench, SWE Marathon, FrontierSWE, PostTrain Bench, OfficeQA Pro, SpreadsheetBench 2, MCP Atlas, AutomationBench, BrowseComp, GDPval-AA v2, AA-Briefcase, MMMU-Pro, MathVision, BabyVision, OmniDocBench, and PerceptionBench. The source says all reported K3 results use maximum reasoning effort with temperature and top-p set to 1.0, and that different benchmark comparisons use KimiCode, Claude Code, or Codex harnesses depending on the test.Kimi's caveats are unusually concrete. The limitations section says K3 was trained in preserved thinking-history mode, so quality may become unstable if an agent harness does not pass historical thinking content correctly or if an ongoing session switches to K3 midstream. It also says K3's emphasis on long-horizon tasks can make it excessively proactive when it encounters minor issues or ambiguous intent, and recommends imposing explicit behavioral constraints for applications that require strict boundaries. The post adds that K3 remains behind Claude Fable 5 and GPT 5.6 Sol in user experience despite being competitive overall.Read moreVenture CapitalThree Years InAuthor: Tomasz Tunguz Published: July 10, 2026Tomasz Tunguz marks Theory Ventures' third anniversary by arguing that AI's central market effect is time compression. In his telling, model release cycles, company revenue milestones, enterprise adoption, and venture categories have all accelerated. Seed, Series A, and Series B still exist as financing labels, but they no longer cleanly describe company maturity when some seed rounds are larger than IPOs and the best AI companies can mature much earlier than prior software companies.The killer detail is the shift from models to inference. Tunguz argues that inference has become the dominant AI market because workloads and buyer preferences are fragmenting: video, batch, local, agentic, and real-time tasks each create different infrastructure needs. He compares this to databases splitting into OLTP, OLAP, vector, and streaming categories, with AI pushing the same specialization into inference infrastructure.The pull is that Theory sees the AI-native venture firm as part of the same pattern. The firm says it has analyzed twice as many investment opportunities with three investors working alongside a nine-person intelligence organization, using agents and research systems to map markets, source companies, and support diligence. The piece is both a market map and a statement about how venture itself is being rebuilt by the technology it funds.Read more: LinkedInVenture Has Rarely Looked More BifurcatedAuthor: Beezer Clarkson Published: July 14, 2026Beezer Clarkson points to PitchBook's Q2 report as evidence that the U.S. venture market has split into two very different realities. AI now accounts for more than 60 percent of all U.S. venture deal value, meaning the headline market can look active and well-funded even while much of the non-AI market is dealing with a much colder liquidity and fundraising environment.The thread uses that split as the setup for Clarkson's latest Origins episode with Alec Litowitz, founder of Magnetar and QStar Capital and one of Citadel's original founding partners. Clarkson says markets like this are periods of genuine uncertainty, not merely ordinary risk, which is why Litowitz's Adaptability Quotient framework is relevant.The embedded clip makes the liquidity point concrete. Litowitz says DPI is “the resolution of uncertainty” because it converts an uncertain investment into actual cash returned to LPs. In his framing, a realized dollar is a real mark, while TVPI remains uncertain until it is realized.The killer detail is the distinction between pricing risk and resolving uncertainty. Litowitz's perspective matters because QStar is a SpaceX investor and Clarkson says the conversation happened just before one of venture's most consequential IPOs. The episode's stated questions are why venture remains a way to gain exposure to innovation, how AI is changing what is investable, why liquidity is ultimately a function of time, and why uncertainty requires a different decision framework from risk.Read more: XThe Best Angel Investors in the US: Who Backs the Most Unicorns, and Who's Active NowAuthor: Ilya Strebulaev Published: July 10, 2026Ilya Strebulaev ranks angels, angel groups, accelerators, and incubators by lifetime U.S. unicorn investments, counting checks written before a company reached unicorn status. The top of the combined list is dominated by organizations: Y Combinator leads with 113 unicorn investments, followed by Plug and Play at 52 and 500 Global at 41. Sand Hill Angels is the highest-ranked angel group at 31.The killer detail is how quickly the list changes below the biggest accelerators. Strebulaev says 271 of the 304 investors in the Top 200 are individuals, or 89%. In the top 100, individuals are 91%. That makes the market underneath the large accelerator counts look much more personal: mostly operators and individual angels writing early checks from their own networks.The pull is the ranking's own caveat. Strebulaev writes that every lifetime leaderboard has a blind spot because many of the unicorns behind those totals were founded a decade or more ago, and some angels have since moved into formal funds, slowed down, or stopped investing. His post therefore separates lifetime performance from recent cohorts, including companies founded in 2015 or later and 2020 or later. For founders or allocators making current decisions, that distinction matters: a career record and a current record are not the same measure.Read more: Ilya StrebulaevAre Prediction Markets Doomed to Fail?Author: Contrary Published: July 16, 2026Contrary argues that prediction markets' current boom depends on whether platforms can prove they are more than regulated gambling with exchange-style branding. Kalshi and Polymarket have reached mass cultural, investor, and regulatory attention, but the article says the underlying idea is old: academic markets, corporate forecasting tools, Intrade, PredictIt, and other predecessors all struggled with the same linked problems of liquidity, legality, and user appeal.The killer detail is the comparison with sportsbooks. Prediction markets present themselves as peer-to-peer, transparent, and non-house-based, but sports contracts reportedly account for more than 90 percent of Kalshi trading, and the article says the platforms keep a much thinner slice of volume than sportsbooks. A market can therefore show sports-betting-scale handle while generating far less revenue.The pull is that the product's hardest problem may be distribution of wins. If a small group of sharp traders captures most profits while casual users lose interest, prediction markets may become valuable data feeds and professional tools before they become durable consumer networks.Read more: SourceRegulationExclusive: The Next Frontier of the Deportation Wars: College CampusesAuthor: Adrian Carrasquillo Published: July 11, 2026Adrian Carrasquillo reports that college campuses are becoming a new front in the fight over immigration enforcement because automatic license plate readers can turn ordinary campus security infrastructure into searchable location data. His thesis is that Flock Safety's camera network, even without direct ICE or DHS contracts, can feed deportation enforcement through local police partnerships and data-sharing practices.The killer detail is the campaign target. The Emergency Campaign to Support Higher Education, working with Schools Drop ICE, is focusing on 75 colleges and universities publicly identified as having Flock contracts. Flock says it has no ICE or DHS contracts, but activists argue the risk comes through local agencies that coordinate with federal authorities and run searches on their behalf.The pull is broader than immigration. Carrasquillo notes that license plate readers have already been abused by officers for stalking, and that Flock's AI search features can identify more than plates, including bumper stickers. A campus safety tool can become a political surveillance system when the data layer is searchable.Read more: The BulwarkThe Supreme Court Broke Independent Agencies. Here's a Way to Slow the Damage.Author: Todd Phillips Published: July 12, 2026Todd Phillips argues that the Supreme Court's decision in Trump v. Slaughter damaged independent agencies by ending for-cause removal protections, but did not leave Congress powerless. The ruling weakens the old model in which commissioners at bodies such as the FTC, NLRB, CPSC, SEC, and CFTC could be insulated from dismissal over policy disagreements. Phillips says the next fight is whether presidents can turn nominally bipartisan commissions into one-party instruments.The killer detail is the procedural fix: quorum rules. Phillips proposes that Congress require bipartisan slates of commissioners to be seated before independent agencies can act. A president

We're excited to have Databricks join us at AIEWF, among hundreds of the top companies in the AI Engineer ecosystem. LS subscribers can use their discount to get past the late bird pricing and access over $50k in sponsor offers! Everyone is still talking about Satya's Frontier Ecosystems post, but few have actually built a (now $175 billion) frontier ecosystem and cloud like our guests today.From open-sourcing the layer above coding agents to rethinking databases for the agent era, Databricks cofounders Matei Zaharia and Reynold Xin are pushing the company beyond the lakehouse into a full data-and-AI operating system. In this episode, Matei and Reynold join swyx at the 2026 Data + AI Summit to unpack Omnigent, LTAP, Lakebase, agent security, open formats, Mosaic, and why databases may matter more than ever once AI agents start doing real work.We go deep on Omnigent: Databricks' open-source meta-harness for combining, controlling, and sharing agents across Claude Code, Codex, Cursor, Pi, custom agents, and internal tools. Matei explains why coding agents and enterprise agents run into the same problems: portability, collaboration, session history, security, spend controls, and the need for a common API above every harness.Then Reynold walks through Databricks' database dream: why CDC is brittle enough to joke that it means “continuous data corruption,” why HTAP has been the holy grail of database engineering, and why Databricks thinks LTAP gets most of the benefits by unifying the storage layer instead of collapsing every query engine. We also cover Databricks' infrastructure scale, the culture behind rapid prototyping, the difference between tech and enterprise customers, Databricks vs Snowflake, whether vector databases should have ever existed, the Mosaic model strategy, Genie, AI Runtime, RL fine-tuning, and the thesis that traditional software gets rewritten once the data is in the right place and agents sit on top.Databricks began as a company for the big data era. The origination of Spark from the Berkeley AMPLab which eventually turned into the product Lakehouse convinced enterprises that they didn't need a separate data lake, warehouse, ML platform, and governance layer. They just needed one open foundation where all of their data could live and be reasoned over.Since then a lot has changed, but data has only become more important. Data is no longer something you keep track of and analyze ad hoc, it's the necessary context agents need in order to act. So the framing has shifted from “where do we put all of our data?” to “how do we expose the right slice of state, history, permissions, and business logic to an AI system at the exact moment it's doing work?”If frontier model performance becomes commoditized, the durable advantage then becomes the company-specific context around them: proprietary data, governed access, operational state, transaction logs, workflows, and feedback loops. Which makes Databricks positioned perfectly.Now coming fresh off the Data + AI Summit 2026, the company is moving just as fast to keep up, announcing Genie One, Omnigent, LTAP, and many more, indicating a central mission in its newer work: Databricks is trying to become the operating system for enterprise agents.Models are getting good enough, but agents are only useful if they have the right context, permissions, memory, state, cost controls, and access to live business data. Fundamentally it appears that significantly better model performance in production is a systems problem, one that data guys like us are remarkably well prepared to solve!We discuss:* Why Databricks built Omnigent as a meta-harness above existing AI agents* Why coding agents and custom enterprise agents need the same infrastructure* The common API for agent sessions, files, streams, tool calls, and cancellation* Why persistent sessions, cloud sandboxes, sharing, search, and collaboration matter* Why Databricks open-sourced Omnigent instead of keeping it proprietary* Databricks' internal agent usage, cloud sandboxes, and coding workflows* The scale of Databricks: 50–60 million virtual machines a day and exabytes before breakfast* Why agent security needs contextual and stateful policies* How an agent could read confidential docs, install a compromised npm package, and leak data* Why spend control matters when an agent can burn $500 reading logs* Startup opportunities around coding-agent analytics, quality, skills, and spend* LTAP, Lakebase, and why Databricks wants to rethink the database stack* OLTP vs OLAP, CDC, and why data pipelines break at 3 a.m.* Why HTAP has historically been the holy grail of database engineering* Why Databricks thinks LTAP is “HTAP done right”* How writing transactional data into column-oriented formats changes analytics* Why agents need live operational context from databases, not just telemetry* How Databricks prototypes strategic systems without endless process* Enterprise vs tech customers, governance, procurement, and DIY culture* The “second system syndrome” risk of rewriting a database engine* Building a database engine from a decade of traces and quadrillions of data points* Why vector databases should never have been a separate category* Why open formats and AI changed the race with Snowflake* The Mosaic story, DBRX, Genie, document parsing models, and specialized model training* Why model customization and RL fine-tuning may become mainstream* Why “get the data there, slap some agent on top” may rewrite traditional softwareMatei Zaharia* LinkedIn: https://www.linkedin.com/in/mateizaharia* X: https://x.com/matei_zahariaReynold Xin* LinkedIn: https://www.linkedin.com/in/rxin* X: https://x.com/rxinDatabricks* Website: https://www.databricks.com* X: https://x.com/databricksTimestamps00:00:00 Introduction00:02:22 Omnigent and the Agent Infrastructure Layer00:08:39 Agent Clouds, Common APIs, and Open Source00:16:52 Databricks Scale and Internal AI Workflows00:18:03 Agent Security, Governance, and Spend Controls00:27:34 LTAP and the Database Dream00:30:30 CDC, HTAP, and Why Data Pipelines Break00:34:05 Lakebase, Parquet, and Live Data for Agents00:36:47 Databricks' Culture of Fast Prototyping00:43:40 The Dream Engine and Rewriting the Database Stack00:51:02 Vector Databases, Query Engines, and LTAP00:52:36 Databricks vs Snowflake00:57:48 Mosaic, DBRX, Genie, and Specialized Models01:03:11 Context, AI Runtime, and RL Fine-Tuning01:06:15 Why Data + Agents May Rewrite Software01:07:09 Closing ThoughtsTranscriptIntroduction: Databricks, Data + AI Summit, and Founder DynamicsSwyx [00:00:00]: Matei and Reynold from Databricks, welcome to Latent Space.Reynold Xin [00:00:06]: Hey, thanks for having us.Swyx [00:00:07]: Yeah.Matei Zaharia [00:00:08]: Yeah, thanks so much.Swyx [00:00:09]: thanks for taking time out. You have your Databricks, Data AI Summit going on. You were just telling me how the first summit that you guys ran was just 50 peopleReynold Xin [00:00:17]: Yeah, it wasSwyx [00:00:17]: in BerkeleyReynold Xin [00:00:18]: little meetup at Berkeley, I thinkMatei Zaharia [00:00:19]: YeahReynold Xin [00:00:19]: put togetherMatei Zaharia [00:00:20]: We were doing these tutorials and, yeah, just teach people Spark.Swyx [00:00:23]: Yeah. obviously now it's like, I think like the headline number's like 100,000 people around the world, 30,000 in person.Swyx [00:00:30]: it's a crazyMatei Zaharia [00:00:31]: AmazingSwyx [00:00:31]: community. Well, I just saw the keynote.Swyx [00:00:35]: Ali's just. Did was it obvious or that back when that Ali would be, like, such a great, like, CEO? LikeReynold Xin [00:00:42]: OhSwyx [00:00:42]: such a great presenter?Reynold Xin [00:00:43]: What do you think?Matei Zaharia [00:00:44]: I think among our group of founders it was clear that, I think he'd be the best at this.Swyx [00:00:50]: Yeah.Matei Zaharia [00:00:50]: And yeah, it turned out great. And he's, he's ramped up on so many topics growing a company. He would just go in and, like, study it and, be talk to all the experts. Like, even if he can't hire the person, learn enough about, like, finance and sales and whatever it was, and, and go from there. Yeah.Swyx [00:01:09]: Yeah.Reynold Xin [00:01:10]: he's obviously very high IQ and a very high EQ, but it wasn't. Like, Ali today is quite different from Ali from, like 10 years ago. I think there's a lot of work that he put in to, get to this point.Swyx [00:01:20]: Yeah. no, to me the most appealing thing about him is that he's funny. And like, it, it's, it'Matei Zaharia [00:01:26]: It's true, yeahSwyx [00:01:26]: it's hard to make jokes about, data warehousesReynold Xin [00:01:30]: About serious topicsSwyx [00:01:31]: securityMatei Zaharia [00:01:32]: YeahSwyx [00:01:32]: what have you.Matei Zaharia [00:01:33]: Oh, yeah. That's for sure.Swyx [00:01:34]: Yeah. So you guys launched a whole bunch of things. I'll, I'll just name check briefly, the stuff because we're not gonna cover everything. Omnigentt, your baby. LTAP, your baby, your dream engine.Swyx [00:01:47]: we're also gonna cover Genie, cover CustomerLake, you acquired PantherMatei Zaharia [00:01:52]: YeahSwyx [00:01:52]: Open Sharing, and there's Unity AI Gateway. A lot of these, I think, like, are things that you would expect a Databricks to do. It's, it's like part of the roadmap. Everyone in your category has similar things. But I think, probably the two of you are leading the two most unique and differentiated initiativesOmnigent and the Agent Infrastructure LayerSwyx [00:02:09]: on, in the landscape. Maybe we'll start with, Omnigentt we'll, we'll, we'll, we'll go into it. I do think that a lot of people are exploring this meta harness concept.Matei Zaharia [00:02:21]: Yeah, totally.Swyx [00:02:21]: What led you to it?Matei Zaharia [00:02:22]: Yeah. There were a couple of, like, converging lines, which I think is a good sign that you need something new. So on the one hand, there's all the coding agent info internally. We have really great, dev infra team. they built something called Isaac, that's like a wrapper on Claude Code and Codex, and, lets you use them either on the web in, like, sandboxes or, just on your dev machine or on your laptop or whatever. And then, they were adding all kinds of stuff there. And we saw all the more advanced engineers like, were building their own workflows with tons of agents, and they were building their own UIs and stuff on top or even on top of that. And then the other one was, like, us building agents. We ship this, like, data science agent called Genie on the research team, which I lead. We also build a lot of internal ones for various things, and then we have all the customer ones. And all of them running into this thing of like, “Oh, I need to switch model and harness and so on,” every few months. Plus the agent is, like, completely useless if you can't share sessions with someone and have history and have search and all this, like, layer on top of it for collaboration. I thought a bit about it from both contexts and, at first people thought it was weird. They're like, “Why are you doing coding agents and custom agents in the same thing?” But I said it's, it's the same problems and, you just wanna build the stuff that lets you deliver the agent, maybe control it if you care about security, and, make it portable across things. And then we prototyped some things as experiments. We saw, yeah, we can make it work, and then we built that for real.Swyx [00:04:06]: I'm wondering if this let's call it architectureMatei Zaharia [00:04:11]: YeahSwyx [00:04:11]: maps to anything in your careers in the past. like I always think about how a lot of things just tie back to operating systems.Swyx [00:04:18]: A lot of operatingMatei Zaharia [00:04:19]: YeahSwyx [00:04:20]: systems tie back to databases,Matei Zaharia [00:04:21]: SoSwyx [00:04:21]: or the other way aroundMatei Zaharia [00:04:22]: so the thing, I do think it ties a lot to, like, network protocols, internet protocol. we alsoSwyx [00:04:29]: Communication between entities.Matei Zaharia [00:04:30]: Yeah. We did stuff with, like, data sharing also, which is probably, most viewers probably won't know unless they'Swyx [00:04:36]: Yeah, open protocol is the term.Matei Zaharia [00:04:37]: Yeah.Swyx [00:04:38]: Open sharing. Open sharing.Matei Zaharia [00:04:38]: Open sharing.Swyx [00:04:39]: Yes.Matei Zaharia [00:04:39]: Yeah. So it's like you have a company, you maintain some table, like let's say like a Walmart or something. They have like the, inventory and what's been sold in each store. And then you also have suppliers, and they would love to produce more things and ship them, like, exactly the moment you need them. So they would love, like, real-time access to your table. So instead of like sending emails around or Excel sheets or phone calls, why can't you share like a view of that table in real time with them? Then they query, they, join it with their data, and they decide what to send. So it's one of these things where you, like you might ask like today since we can vibe code anything so fast, why do we even need to design like protocols or APIs or software? Why can't you just vibe code things on demand? But for this type of interoperability where multiple parties that are moving at different speeds are building stuff and you still want some layer on top to coordinate, you do wanna design it and build it. So it reminds me of that, like agents talking to each other and, users talking to agents and tools.Agent Clouds, Cloud Sandboxes, and Keeping Sessions AliveSwyx [00:05:42]: Reynold, any other comments alternative viewpoints?Reynold Xin [00:05:46]: I think, by the way, we had a debate on exactly which set of benefits would, matter a lot, and I think around the time we decided to do this thing I was telling Matei, “Hey,” it just happened to be there's a particular week that I was coding nonstopSwyx [00:06:00]: from the moment I woke up to, like, the moment I went to bed, I was, like, looking at my Claude sessions, my Codex sessions. And one of the things that was particularly annoying was having to keep my laptop open.Swyx [00:06:12]: I was driving to a doctor's appointment, and I remember because I wanted to make sure the whole thing continues working.Matei Zaharia [00:06:18]: But by the way, it's so comforting to hear you say that because I'm like, “I don't know if I'm a clown and I'm doing this or like.”Swyx [00:06:25]: Yeah. Like honestly, I was driving and I was tethering my laptop to my phone.Matei Zaharia [00:06:29]: huh.Swyx [00:06:29]: Keeping it on the side. Whenever I hit a red light, I started looking at what's going on my laptop.Matei Zaharia [00:06:35]: Yeah.Swyx [00:06:35]: And I just felt that was ridiculous.Matei Zaharia [00:06:37]: Yeah.Swyx [00:06:37]: It felt like we went back to the dark agesMatei Zaharia [00:06:39]: YeahSwyx [00:06:40]: programming. the productivity you gain from all this coding age is amazing, but, yeah.Matei Zaharia [00:06:45]: Have you heard of cloud?Swyx [00:06:47]: Yeah.Swyx [00:06:48]: It was crazy to me.Matei Zaharia [00:06:49]: Oh, the thing you were working on was the sandboxes or was this before that?Swyx [00:06:52]: It was a sandbox.Matei Zaharia [00:06:53]: Okay.Swyx [00:06:54]: I was workMatei Zaharia [00:06:54]: So you were inSwyx [00:06:55]: So I was approaching from a very different angle. I wanted to, “Hey, we're gonna have cloud sandboxes that doesn't shut down. You can get one very quickly,” but not just for running agentic sessions.Matei Zaharia [00:07:06]: Yeah.Swyx [00:07:06]: It's also for running development. So I was personally building that week, and through building that, I ran into all these issues, and then I wroteMatei Zaharia [00:07:15]: YeahSwyx [00:07:15]: a document for Matei, it's like, “Here's my wish list of what the actual environment should do.” And I think he ended up almost implementingMatei Zaharia [00:07:22]: YeahSwyx [00:07:22]: every single one of them.Matei Zaharia [00:07:23]: Yeah, I remember Reynolds saying, ‘cause my first prototype of this had just chats with your agent and he said, “I have to be able to open a shell, like my own shell and like list files and like tail them and stuff.” SoSwyx [00:07:36]: So SSH into a mainframe.Matei Zaharia [00:07:37]: Yeah. it has that now.Swyx [00:07:39]: Tailing my log.Matei Zaharia [00:07:40]: Yeah.Matei Zaharia [00:07:41]: Yeah.Swyx [00:07:41]: And also another thing I think I asked was, I had. I still use cursor for the sole purpose of rendering markdown files.Matei Zaharia [00:07:48]: huh. Yes.Swyx [00:07:49]: So I said, “If you just give me a way to see my markdown files and renderMatei Zaharia [00:07:53]: YeahSwyx [00:07:53]: them properly, I don't need a separate tool anymore.”Matei Zaharia [00:07:55]: Yeah.Swyx [00:07:56]: And I think you also built that in.Matei Zaharia [00:07:57]: Yeah, we, yeah, we did that, yeah. Yeah, we had a lot of engineers building, their own vibe coding setup. But then the other thing they all said is like, “Hey, I built something that's amazing for me, but, like, no one else on the team can use it ‘cause I don't have a server to collaborate.” And this is why we tried to set up, Omnigent, so you can have a server and have the security, set up in there. So, like log in with Google or whatever and, like securely share stuff. which. And that's where we've seen a lot of other agents like hit things. Like people think they prototyped an awesome agent, but it's not allowed to connect to like some really important data or whatever because of the security team.Omnigent Architecture, Open Source, and Common APIsSwyx [00:08:38]: Yeah.Matei Zaharia [00:08:38]: So yeah.Swyx [00:08:39]: Yeah. At this point, so for those watching along on YouTube, we're gonna putting up a image of the structure here, and we can talk a little bit of the architecture. I think I just want to have people understand, ‘cause like when we're talking about software, it can be very abstract and like here is what we're talking about. You've worked out in open source this entire platform and there's a runner component and server component with a uniform API that you've, you've figured out. any other element and obviously you can plug in all this, persistence layers and compute layers. This is a whole cloud. It's an agent cloud.Matei Zaharia [00:09:12]: Yeah. It's, it's got these components to work with it. The, a lot of the action happens like on the machine where you deploy your agent too. So whatever you've got on there, you can run. But yeah, it's, I think it's the minimal thing you want to have hosted, like collaborative agents and to have that server. And one of the reasons we open sourced it is, anyone building agents, this gives them an app they can start with and customize, which we were seeing in Databricks too. Like someone would make a nice, agent app and then other teams would ask, “Oh, can I just use yours for my agent?”Swyx [00:09:45]: Yeah, I think we had like five or six different agentic frameworksMatei Zaharia [00:09:48]: YeahSwyx [00:09:48]: built by every different team. They do all do more or less the same thing. Yeah, you need to. people wanna take something that works in Forkit, and you might as well have something open source. Yeah, which also was another question, which is interesting for Databricks. Like what do you choose to open source? What do you choose to make it proprietary? It's in. this goes back to Spark, right?Matei Zaharia [00:10:05]: Yeah.Matei Zaharia [00:10:06]: One, so one of the reasons to open source something is if you think it's a layer that will there'll be some network effect, it'll benefit from many, people collaborating, on it. So, for example, with Spark, I don't know if when Spark came out, we also focused a lot on letting you have libraries on top. So like there used to be differentSwyx [00:10:28]: EcosystemMatei Zaharia [00:10:28]: distributed computing engines for like machine learning and graph computation. We said they should all be libraries that you can compose. And we made it super easy to add connectors to data sources too. And then we benefit because, we don't have the time to write like connectors to like, 1,000 like different databases and file formats, but we can just use the ones people make, and of course they benefit from joining, this thing. So that's like one of these as it. Another way to think about it is like imagine, we our thing wasn't open. We had some agent hosting thing, but it's not open and then there is an open one. if you're. Which one's gonna win in the long run? So like here, because there is this benefit from like people writing integrations, it'll be, it'll be that. And then there are other things that like you just can't, even deliver as open source that are things the company does. Like for example, how do you make sure you're like streaming, jobs or your Lakebase database doesn't like, lose all your data at night? Well, that requires an operational team that's gonna sit there. There's no way it has to be a service. So like we wanna make sure as a company we're really good at those infra services and then we're as open as we can in terms of like what you build on top.Swyx [00:11:42]: speaking from a benefits, I think we are already seeing pull requestsMatei Zaharia [00:11:45]: YeahSwyx [00:11:45]: of all kinds of ecosystem integration, even though it was only released on Saturday.Matei Zaharia [00:11:50]: Yeah, Saturday. Yeah. So someoneSwyx [00:11:51]: Let's see, let's see what's going on. Yeah, you can look at the merge ones. I asked Sam Nigon this morning aboutMatei Zaharia [00:11:59]: 400 merge already?Matei Zaharia [00:12:00]: Yeah. I think Recent quite, I would guess around half are not from our team. but for example, someone added support for running it on Kubernetesrnetes. people added, many cloud sandboxes, so this can launch a cloud sandbox and run your agent in there, which is great for sharing too, ‘cause it's not, like, on your laptop and someone's, like, running scary code on there. so yeah, many startups have put those in, and, we expect to see more of them. We also have more agent harnesses already. Cursor, CLI, and Antigravity also.The Modern Data Stack and the Emerging AI StackMatei Zaharia [00:12:34]: Yeah. That's all, beautiful. And I, I feel like the last time this happens, there was the rise of the modern data stack.Matei Zaharia [00:12:42]: I don't know if it's that useful. I'm, I'm curious in your postmortem.Matei Zaharia [00:12:46]: I think most peopleSwyx [00:12:47]: AgreeMatei Zaharia [00:12:47]: will agree that it is finally dead. but maybe this arises to a new modern AI stack that, like, does the same thing.Matei Zaharia [00:12:52]: I don't know.Reynold Xin [00:12:54]: I think the modern data stack was a pretty useful thing, probably even up until this day. I think what, maybe for the audience who don't understand the history, I think the modern data stack is effectively decomposed into you need a layer to ingest the data in, you need a layer to transform your data, and then all of this are run, and then you need a layer to maybe visualize your data. And all of this runs on some data warehouse, or later on, as we're doing data warehouse or lakehouse.Reynold Xin [00:13:21]: I think that concepts are all very powerful and very useful. They enable a lot of workloads. What people eventually run into is a question of unification and consolidation is, hey, do you really need to chop all this into different pieces and work with so many different vendors and platforms in order to get, like, a very simple visualization done, right? So I think, like, over time, everybody started realizing that customers are pushing us. We started, we can realize that, so we started building more and more capabilities and trying to consolidate. And at the end of the day now, customers don't have to worry about having me hook up five different systems in orderMatei Zaharia [00:13:55]: YeahReynold Xin [00:13:55]: produce a chart. But the. I think, honestly, something like this is probably happening, in how many different frameworks do you want to hook up together in order to produce, like do a very simple agent.Matei Zaharia [00:14:06]: Just to be clear, I would say the core of this is this common API on top of all the harnesses. So the API is like, you've got an agent session, and you can send in a message or, like, a file. That's what you can send in, and then you get out, these streams as it's streaming text or as it's doing tool calls. And, or the other thing you can send in is you can, like, tell it to cancel a turn. So that's the API. Now, the thing we did is we could get you that on top of, like, cloud code running in a terminal, Codex, Py, OpenAI SDK, all that stuff. We map them all to that same interface. So that is something that you'd have to maintain yourself if you built your own, like, agent orchestrator, and then whenever cloud changes its API, you gotta, tweak your thing or it's gonna lose some messages. So that's the thing that's valuable to maintain. Then on top of that, like, we built a few apps. I think we built a pretty cool UI and stuff, but that's, And we built a security and control piece, which I'm excited about. But it's that common interface, so we don't. We. That doesn't try to be a stack. And in fact, you could plug in your own UI on top of this, server. That, and that's one of the use cases we care a lot about, ‘cause we want to use this in our own products.Compute, Sandboxes, and Databricks ScaleSwyx [00:15:20]: Yeah. It should be everywhere.Matei Zaharia [00:15:22]: Yeah.Swyx [00:15:22]: I think one of those things that is really interesting to me is, like, well, first of all, I'll, I'll endeavor to do everything and not call it the modern AI stack because like it needs a different name.Matei Zaharia [00:15:32]: Yeah.Swyx [00:15:32]: But like, yes, like, so one of the first people that told me about compute, sandboxing was Nikita from Neon.Swyx [00:15:39]: Because a lot of people think about Neon as like, well, it's serverless Postgres with, like, the separation of compute and storage and, instant branching and all those things. But every database company is also a compute company.Matei Zaharia [00:15:51]: Yeah. Yeah.Swyx [00:15:52]: And so he was showing to me his whole, his sandboxing solution. I don't think he have ever launched it.Matei Zaharia [00:15:57]: So our sandbox solution, the reason we could build it so quickly was because we realized if you just take the actual Lakebase architectureSwyx [00:16:05]: YeahMatei Zaharia [00:16:05]: and remove the database from it, by the coming from NeonSwyx [00:16:08]: Exactly, rightMatei Zaharia [00:16:09]: you have this sandboxSwyx [00:16:09]: Every database company has it already, yeah.Matei Zaharia [00:16:11]: Now, there are some differences. For example, in the one to support this particular workflow, it's important to have local persistence,Swyx [00:16:19]: YeahMatei Zaharia [00:16:19]: because you want your state to persist. Your libraries, you don't have to install your library every time, right?Matei Zaharia [00:16:24]: whereas the Neon architecture, because of the separation of storage from compute, you don't need persistent local disk.Swyx [00:16:30]: Yeah.Matei Zaharia [00:16:30]: So there's some differences.Swyx [00:16:32]: Yeah.Matei Zaharia [00:16:32]: But the, at the end of the day, yeah, it's, Yeah, so this is when you run, like, a coding sandbox. Like, if I use it, yeah, we have the dev env internally at Databricks. There's, like, many, like, tens of gigabytes of data just for, like, all the source code and, like, artifacts and stuff that I built, and I want that to come back next time, so.Matei Zaharia [00:16:51]: Yeah.Matei Zaharia [00:16:51]: But yeah.Matei Zaharia [00:16:52]: Before the show, we was talking about some statistics that might be surprising at the adoption.Matei Zaharia [00:16:56]: It could be internal, it could be external, whatever comes to mind, just to impress people the scale this is happening.Swyx [00:17:02]: So we, on the analytics side, I think we launchedReynold Xin [00:17:06]: Maybe 50 or 60 million virtual machines a day across all three clouds, so we're one of the biggest compute orchestrators out there.Reynold Xin [00:17:13]: Stuff for sure for CPU compute.Swyx [00:17:14]: Yeah.Matei Zaharia [00:17:14]: Yeah.Reynold Xin [00:17:15]: the. And all of this process, I think exabytes of data, I joked about depending on which time zone you are, typically before you have breakfast, Databricks would have processed exabytes of data already on that day. and on Neon, it's pretty interesting, too. It's launching, I think, 13 million databasesSwyx [00:17:34]: YeahReynold Xin [00:17:34]: a day now.Swyx [00:17:35]: Yeah, to me that was, like, aReynold Xin [00:17:36]: And that's just likeSwyx [00:17:37]: Like, what do you mean?Matei Zaharia [00:17:38]: Yeah. And that's the point.Reynold Xin [00:17:40]: And a lot of those were thanks to agent- agents and branching experimentationSwyx [00:17:44]: YeahReynold Xin [00:17:44]: because we made it so easy and so quickly, and thanks a lot to Nikita's team, to launch databases. It's, the. So it's changing the way people use databases.Swyx [00:17:54]: Yeah. Okay, we're gonna go into more database talk in a bit, but I wanna make sure we close up anything on Omnigentt. you mentioned, you were excited about the securityOmnigent Security, Contextual Policies, and Spend ControlsSwyx [00:18:03]: control side.Matei Zaharia [00:18:04]: Yeah.Swyx [00:18:04]: a lot of companies are figuring that out right now, as well as the spend side.Matei Zaharia [00:18:08]: Yep.Swyx [00:18:09]: what have you found there?Matei Zaharia [00:18:11]: Yeah, so I spent quite a bit of time talking to internal users, developers, security team, managers, and also lots of customers, and there's a few things. Like, first of all, one thing, that immediately was. became obvious is for security, there's this tension between, like, usability and security. And, the way people do. Like, a lot of coding agents today have very basic things like you can tell me which tool patterns I'll allow or disallow or whatever. It's like yes or no. But that puts you in a very tough spot. So just as an example, like, should my agent be able to read, some confidential documents, or let's say, should it be able to install new packages from npm, which, maybe it's compromised. Yes or no? Like, maybe I wanna allow it. Should my agent be able to publish stuff to the company website? Well, if I'm using it to code on the website, yes. But should it be able to do both, so it can, like grab a confidential document and be prompt injected and leak it? Probably not. So the thing we decided we need is stateful or what we call contextual policies where you keep track of the state of that session. It's not like is it allowed to push to the marketing site or not, but, like, hey, if it did a risky thing, like it installed, a old package from npm, or it read, like, 1,000 confidential docs, then no. Then don't, don't do it. Otherwise, maybe it's okay. That's one example of, like, moving that trade-off so it's both more secure and more useful by having a more powerful engine, essentially. This requires tracking sessions. The other piece that was interesting there is, like, there are these very level events it's doing, and you want some libraries on top that parse them. Like, for example, we have a, MCP server on Google Drive internally. It's got 60 API calls. like, how do I know which of those, like, will share a document with stuff on the internet and which ones won't? It's, it's annoying. So we designed in Omnigentt the policy layer so that it's functions and you can have libraries. Like, someone can make something that maps the level events to high-level ones, and then you write a policy about the high-level things that came out. so and thatSwyx [00:20:25]: This is related to the Panther,Matei Zaharia [00:20:27]: Yeah, Panther is. will help with that. PantherSwyx [00:20:30]: YeahMatei Zaharia [00:20:30]: a similar idea on the event processing side, and it's Python-based versus a weird custom language. this is more, as in realSwyx [00:20:39]: I didn't even know we were good yeah.Matei Zaharia [00:20:41]: Those things are happening, yeah.Swyx [00:20:42]: Yeah.Matei Zaharia [00:20:42]: So yeah, but these are the cool things. I think the contextual or stateful part, and then the way it can be libraries, and that was another reason to make it open source because others will write libraries and, like, we and our customers can use them. And the final thing, because it's stateful, one of the states we track is how much you spent in that session. So I can. I've had, like, I ask an agent to debug something, and it spent $500 because it decided to read a lot of log files and burn a lot of tokens. but I can literally say, “Okay, launch a agent to do this and cap it to spending $5.” Like, ask me for permission if it needs more. And because we're counting that within that session, it'll pop up and tell me, “Okay, you spent five, $5. Do you wanna go on?”Reynold Xin [00:21:27]: So important context here. Matei spent the last five years, a lot of his time was architecting Unity Catalog at DatabricksMatei Zaharia [00:21:34]: YeahReynold Xin [00:21:34]: which is the governance layer for data.Matei Zaharia [00:21:35]: That's right, yeah.Reynold Xin [00:21:36]: And he's combining expertise at that layer together with all the AI governance he knows.Matei Zaharia [00:21:41]: Yeah.Swyx [00:21:41]: DoMatei Zaharia [00:21:41]: But I also spent a lot of time being annoyed by coding agents and getting prompts.Matei Zaharia [00:21:46]: And also as theReynold Xin [00:21:48]: All the aboveMatei Zaharia [00:21:48]: I don't want to end up on the front page as, like, I installed some weird npm package and leakedSwyx [00:21:53]: YeahMatei Zaharia [00:21:53]: all the code, so I'm especially paranoid. But also I have very little time, so I don't want to sit there approving, like, do you want to run a 20-line, bash script, yes or no? so that's why I spend a lot of time figuring out, like, how can I make it as safe as possible and not annoying?Swyx [00:22:10]: Yeah. Is safety and mmm, let's call it security a bigger concern than token maxing or token budgets? which one is, likeMatei Zaharia [00:22:19]: Oh, yeah, they're both there. I don't know. I guess it depends on the type of company you are. So I think, some companies, like, the budget is, limited and, they really care about thatSwyx [00:22:34]: you can be Uber and still be concerned?Matei Zaharia [00:22:36]: Yeah. Oh, yeah, totally. Yeah. If you haveReynold Xin [00:22:38]: for us, securityMatei Zaharia [00:22:39]: YeahReynold Xin [00:22:40]: super paramount.Matei Zaharia [00:22:40]: For us, security is absolutely critical as a, cloud provider. It's, it's the most important thing, and, token maxing, we're not so worried about it yet, but I've seen the Like, for example, I talked to some consulting companies. They have, like, 100,000 employees who are all coding for customers. If those each spend, like, an extra $1,000 a month, that's, that's not fun.Swyx [00:23:04]: YeahMatei Zaharia [00:23:04]: we have, like, only a few thousand engineers.Swyx [00:23:06]: What's the policy in Databricks? Is it just unlimited or what'Matei Zaharia [00:23:08]: It's, it's unlimited, but we do. we use our own product to, like, analyze the traces and stuff, and we have a team that'looking to optimize and to see if anyone's doing something weird. And, we had some really cool insights just from analyzing current traces, like whichSwyx [00:23:24]: YeahMatei Zaharia [00:23:25]: models are better at, say, Rust versus like TypeScript or whatever. So yeah, at least in our code base.Swyx [00:23:31]: Yeah. Amazing. Obviously, I have to ask the token question, obviously.Matei Zaharia [00:23:34]: Yeah.Swyx [00:23:34]: I think it'sReynold Xin [00:23:34]: YeahSwyx [00:23:34]: it's a key thing. But yes, security and control above that, and figuring out a sane layer there you can have some autonomy, but, not too much.Matei Zaharia [00:23:43]: Yeah. Yeah, and we wanna make it super easy. As a engineer, you should set a thing. So in Omnigentt, you can ask your agent, “Set a policy on yourself to do this.” So it can likeSwyx [00:23:52]: But if there's something I should be showingMatei Zaharia [00:23:53]: YeahSwyx [00:23:53]: I don't, I don't see it on the GitHub, but,Matei Zaharia [00:23:55]: Oh, yeahSwyx [00:23:56]: there's justMatei Zaharia [00:23:56]: Well, in the docs there's something.Swyx [00:23:57]: Yeah, this is it.Matei Zaharia [00:23:58]: You can look at it later.Swyx [00:23:59]: Okay. Yeah.Matei Zaharia [00:23:59]: Just look in the docsSwyx [00:24:00]: YeahMatei Zaharia [00:24:00]: contextual policies if you wanna see.Swyx [00:24:04]: I just like to point peopleMatei Zaharia [00:24:05]: look at the built-in policies.Swyx [00:24:06]: Yeah.Reynold Xin [00:24:06]: Yeah.Swyx [00:24:06]: If you want to, follow up on this is exactly where to look, right?Reynold Xin [00:24:10]: Yeah.Matei Zaharia [00:24:10]: Yeah. yeah, and the story of these is, like, I just wrote, like, I wrote a doc with like 10 ideas for things before as you were working on them. Well, that was, like, my wish list of things people asked, and I told the team, like, “Hey, can you do like at least five of these for the launch?” And then they just got back with all of them, so.Swyx [00:24:29]: Oh, wow.Matei Zaharia [00:24:29]: so you can come up with more, but them- some of them are just meant to be examples. really you can intercept, like, any event the agent is making, and you can then either block or force it to ask the user or, like, allow, and you can update state to keepSwyx [00:24:45]: YeahMatei Zaharia [00:24:45]: track stuff.Swyx [00:24:46]: Yeah, ‘cause ultimately you're, I think of you as, like, a systems designer.Swyx [00:24:50]: You let people plug in, right? That's the wholeMatei Zaharia [00:24:51]: YeahSwyx [00:24:52]: modus operandi of what you do.Matei Zaharia [00:24:53]: Yeah.Swyx [00:24:54]: It's likeMatei Zaharia [00:24:54]: And we care a lot about also composab- like, can someone else write a library that others use, whichSwyx [00:24:59]: YeahMatei Zaharia [00:24:59]: this is meant to.Reynold Xin [00:25:00]: There's also a batteries included philosophy hereMatei Zaharia [00:25:03]: YesReynold Xin [00:25:03]: probably very similar to how you did Spark, which is you could just start using.Swyx [00:25:06]: Yeah.Matei Zaharia [00:25:06]: Yeah, that's right. It has to be good out of the box at certain things, and then you can build your own things on top that, like, we don't wanna do. But in Spark, if you just wanna like, I don't know, like read a table or do, like, a aggregation, it should be awesome at that out of the box.Building on Omnigent: Contributions, Startups, and AnalyticsSwyx [00:25:23]: Yeah. People wanna catch up on Omnigentt, they should watch your keynote.Swyx [00:25:26]: they should go through the GitHub and the docs. If they wanted to contribute, or they want to build on this ecosystem what would you call out as the most high-leverage places get involved?Matei Zaharia [00:25:36]: Yeah, do get involved in the Discord and in GitHub. Our team is there, is monitoring, and, some of the things people ask for we just built ourselves. Some of them, we're, we're collaborating with them to build it. and also tell us, likeSwyx [00:25:49]: Yeah, they're gonna be veryMatei Zaharia [00:25:49]: how you would like to use it because I think especially for developers, like, everyone wants it to work their own way, and a really good developer tool, like you have to hear the feedback on all the ways and figure out the abstractions and how to let people customize. So we'd love to hear, like, if you think, “Hey, I, I don't want it to work this way,” tell us. We really just wanna get that compatibility layer across agents and then let you do stuff on top.Swyx [00:26:14]: Yeah. is there any, in terms of like the startup side, I'm, I'm a founder.Swyx [00:26:18]: I wantMatei Zaharia [00:26:18]: YeahSwyx [00:26:18]: I see an opportunity, I wanna get in front of you. What's your request for, like, a startup that, like, I wish someoneMatei Zaharia [00:26:23]: Oh, like you wanna integrate with us?Swyx [00:26:24]: someone was working on this.Matei Zaharia [00:26:26]: Oh, for a startup?Swyx [00:26:27]: Yeah.Swyx [00:26:28]: Like, your, you got your own startup. It's doing well.Matei Zaharia [00:26:30]: Yeah.Swyx [00:26:30]: But like, if you weren't working on your own startup, what is, like, obvious that you should You advise many startups too, obviously.Matei Zaharia [00:26:37]: I do think, just as a company with a lot of engineers, like anything that helps me make sense of how people are usingSwyx [00:26:46]: SpendMatei Zaharia [00:26:46]: coding agents and,Swyx [00:26:48]: Yeah. AnalyticsMatei Zaharia [00:26:48]: spend, but also quality or like you should write, you should add this skill, or you should write this thing, or your agents are really horrible at tasks involving this service, so I go spend time. That would be nice. yeah.Swyx [00:27:00]: Yeah. The closest I've found is, this team, GitAI.Matei Zaharia [00:27:03]: Oh, cool. Yeah.Swyx [00:27:04]: They started with, like, we will just do, code and human attribution, but they're building the analytics layer on top of that.Matei Zaharia [00:27:12]: Yeah.Swyx [00:27:12]: I do think, like, there are a bunch of, like, artificial analysis is obviously,Matei Zaharia [00:27:18]: Yeah, they have their benchmarksSwyx [00:27:18]: doing super wellMatei Zaharia [00:27:19]: YeahSwyx [00:27:19]: with their stuff. so there's, there will be people. I think this is like the domain of consultants first, but then peopleMatei Zaharia [00:27:26]: YeahSwyx [00:27:26]: will build software that, let's say, it's kinda like the management planeMatei Zaharia [00:27:29]: YeahSwyx [00:27:30]: for coding agents.Matei Zaharia [00:27:30]: Yeah, I think there'll be a lot of insights there. You have it in other areas.Swyx [00:27:34]: Okay. Well, and then the other, big thing is your dream engine.LTAP: Lake Transactional/Analytical ProcessingSwyx [00:27:39]: maybe you wanna tell the story of, LTAP.Reynold Xin [00:27:45]: So, and background with. I'm, I'm gonna make people listen to our Ankur Goyal episode where we talked about SingleStore, HTAPMatei Zaharia [00:27:52]: YeahReynold Xin [00:27:52]: and all that history.Matei Zaharia [00:27:52]: Yeah. The LTAP idea is pretty simple. so if people have heard of the, Ankur's, talk about HTAP, it's effectively the world of databases. Sorry, there's like maybe a lot of context needs to be injected here. The world of databasesSwyx [00:28:06]: I am happy to be the database podcast that I'm forcing people to, like, learn your databases, guys.Swyx [00:28:11]: You cannot vibe code with just markdown files.Reynold Xin [00:28:13]: Yeah.Swyx [00:28:13]: Like,Reynold Xin [00:28:14]: It's one of the most important fundamental systems technologies out there. But the world of database effectively split into roughly two halves. There's what we call OLTP databases, which are transactional, and think of your Postgres, your MySQL, your Oracle databases, and the other side is what we call analytics, and sometime might refer to term OLAP. And the difference is on OLTP, you typically have maybe run some transaction on some event that looks up at one specific row. We update that row, right? It's a very oriented data structure. And on analytics, you're trying to reason on the data. You're trying to compute, “Hey, what's my revenue per store? What's my. How's my website doing every day?” And then you, eventually want to probably end up running anal- machine learning on it to predict, “Hey, how will my maybe sales be going in the future?” they are so very different architecture, and everybody start with OLTP databases. Every app, when you become serious enough, that needs more than markdown files, you need to have a database. You want to lose your data, you want to have some transactional consistency. But once you want to reason on the data, if you only have like- A hundred rows, it's probably okay to run it on your Postgres or your own, your MySQL database. But once you have more data and want to run more complicated analysis, the very analysis might crush your Postgres database. So you start doing, getting data out of the OLTP databaseSwyx [00:29:35]: Replication.Reynold Xin [00:29:36]: Replicate them into the analytic systems and just startSwyx [00:29:39]: Yeah, which for people, Elasticsearch is, like, aReynold Xin [00:29:42]: Yeah. So some of them get into Elasticsearch for, like, blocked analysis. A lot of our customers obviously get into Databricks to run more sophisticated things.Swyx [00:29:51]: Yeah.Reynold Xin [00:29:51]: And there's this term called CDC, whichMatei Zaharia [00:29:54]: Change data captureReynold Xin [00:29:55]: change data capture. and what it does, it reads the binlog of the database, and if you don't understand what binlog is, it's fine. The, but it's a little delta of the data, and it reconstructs based on the delta, the state of the database, on the analytics side. But CDC is, like, a very painful thing. It's how standard in the industry, everybody uses it, but, it ends up being. I think many data engineers ends up being waken up at, like, 3:00 a.m, because there's some pipeline thing.Swyx [00:30:22]: my explanation is, like, Airbyte is like a, became a $5 billion company just doing CDC.Reynold Xin [00:30:27]: Yeah, exactly.Reynold Xin [00:30:28]: CDC is, like, a veryMatei Zaharia [00:30:30]: It's hard.Reynold Xin [00:30:30]: It's one of the most boring but one of the most fundamental operations, like, powering modern society.Matei Zaharia [00:30:37]: huh.Reynold Xin [00:30:37]: But it's so brittle that, we joke that it's, should be called continuous data corruption, because you might change your schema on your OLTP database, and then the CDC pipeline fails to handleSwyx [00:30:48]: YeahReynold Xin [00:30:48]: the schema change.Swyx [00:30:49]: Yeah.Reynold Xin [00:30:49]: And then everything goes out.Swyx [00:30:51]: And there's all sorts of tricks that you can do, like, you add in, like, some versioning or whatever, but yeah.Reynold Xin [00:30:55]: Yeah, but it's a very, in general, very complicated. Like, I think at my keynote, I asked the audience put up their hand if they love their CDC pipeline. Only, like, maybe two people put it up. So if single store, like, about maybe a decade ago, I think the industry had this idea, hey, what if I built a single database that can handle both workloads? Now I don't.Swyx [00:31:12]: Which, like, by the way, every database person ever has ever always dreamed about this.Reynold Xin [00:31:15]: Yes. Yes.Reynold Xin [00:31:16]: This is the holy grail of database engineering is why not build a single system that can do both of this? But it ends up just being a lot of compromises. one, I think one of the first issue is that, hey, each. they say Postgres has a massive ecosystem, right? You want to be using the tools that's built for Postgres. And Spark, for example, had a massive ecosystem. There's a lot of libraries you want to use. If you were to create now a new thing, you don't have a ecosystem. You tend to create a new, smaller proprietary API, and you're lacking both, and it's also very difficult to make it performance-wise to be, comparable on either side. So it ends up being sucking on both. And our whole idea of LTAP, it's obviously a wordplay on the term HTAP, is that we think this is HTAP done right. HTAP wants to build a single engine for both. We think you can get 99% of what you need by unifying the storage, and just have a single storage layer. And once you have the single storage layer, if your Postgres databases are writing data in a column-oriented format, everything analytics can just go read that data directly without any delay, right? There's no pipeline in between, so all the data will immediately be available for reasoning analytics. I think I was telling some customers earlier, hey, when we talked about this is gonna be super useful for agents, I at first didn't really believe in it myself, even though we wrote that positioning.Lakebase, Agents, and Live Operational DataMatei Zaharia [00:32:39]: Yeah.Reynold Xin [00:32:40]: But then last night I was having dinner with a Australian customer, and they told me, “Oh, hey, one of the big issue we have is we have all these logs from our services, and we see SLA dips and want to investigate. But then there's no way for those agents to even understand what's going on in the actual databases themselves. All we see is just, like, product telemetry of the database and the services.” It would make those agents 10 times more powerful if understand, for example, who's placing those orders, what is happening, what exactly are they doing. So now I'm sold on our own message.Swyx [00:33:13]: Yeah.Reynold Xin [00:33:14]: I think it's really. It gets you the almost all of the benefits of the HTAP holy grail, which is, hey, make the data available immediately for reasoning analyticsSwyx [00:33:26]: Yeah, I think,Reynold Xin [00:33:27]: without compromiseSwyx [00:33:28]: in the way that humans are generally intelligent and want to have the ability and access to query anythingReynold Xin [00:33:34]: YeahSwyx [00:33:35]: while they do the work, they also need history and need context.Swyx [00:33:38]: And, like, where else does they get context? That's it's an analytical workload.Reynold Xin [00:33:41]: Exactly.Matei Zaharia [00:33:42]: Yeah. Yeah. And I remember when we had incidents with our databases and engineers said, “Well, I can't just run a giant query on it to see what's going on because that's gonna bring down the database and hoard it even more.” Like, that's the stuff that this gets rid of, because you spin up a whole separate fleet of machines that's doing the analytics. You're not overloading, like, the main databaseReynold Xin [00:34:02]: RightMatei Zaharia [00:34:02]: that's still trying to serve stuff.Reynold Xin [00:34:04]: Yeah.Matei Zaharia [00:34:04]: Yeah.Why LTAP Works Now: Parquet, Postgres, and LakebaseSwyx [00:34:05]: So this has been a dream for a while. what had to get done in order to get to today? Like,Reynold Xin [00:34:11]: Yeah.Swyx [00:34:11]: I feel like, you have announced variants of this several times, but it wasn't as clear as LTAP.Reynold Xin [00:34:18]: Yeah.Swyx [00:34:18]: I think LTAP is like Like, okay, we've got it, guys.Matei Zaharia [00:34:21]: This thing, yeah.Reynold Xin [00:34:21]: I was talking to somebody at Meta, and then he was asking me, “Hey, what's the catch? Why is it possible now?” And I think the reality is we took a lot of time to work on the Lakebase architecture. obviously a lot of it came from the Neon team, which is a separation of storage from compute. And it turned out it was just a tiny little step away going from that to this LTAP idea, which is, hey, we just. in the Neon architecture and in Lakebase architecture, we're writing data in oriented format to the open data lake, but in there we're writing in Postgres pages. Ali and I were spending a lot of time debating, hey, can we just change that to write in column-oriented format? And we're just debating, and one day, one of our engineers who's, like, super smart came in, he's like, “Hey, I just prototyped it. It works.”Swyx [00:35:07]: Wait, it's, prototype what?Reynold Xin [00:35:09]: Prototype, instead of storing the data in the data lake in the oriented formatSwyx [00:35:15]: ColumnReynold Xin [00:35:15]: like Postgres pagesSwyx [00:35:15]: YeahReynold Xin [00:35:16]: write them in Parquet.Swyx [00:35:17]: Yeah.Reynold Xin [00:35:18]: and he just made the observation that, hey, our storage fleet has a lot of extra idle CPUs And we could use those CPUs to do the transcoding from row to column, where row is good for OLTP, but column is good for analytics. so let's do that transcoding at that time. And as a matter of fact, once you transcode the data compresses better. So from those services writing to, for example, S3 or other data lake, like object stores, you can write them faster ‘cause now they are now smaller.Matei Zaharia [00:35:49]: Yeah.Reynold Xin [00:35:49]: So there's no overhead, it's no compromise in performanceMatei Zaharia [00:35:52]: Some CPU overhead.Swyx [00:35:54]: Yeah, because,Matei Zaharia [00:35:55]: YeahSwyx [00:35:55]: we had extra CPUs anyway.Matei Zaharia [00:35:56]: We had that fleet anyway, yeah.Swyx [00:35:57]: so the debate ended. it's one of the classics of, tech, issue of a lot of debate, but then somebody went ahead and just tried to prototype it and it worked.Matei Zaharia [00:36:06]: But, like, something this strategicSwyx [00:36:07]: That's rightMatei Zaharia [00:36:07]: and important to the company, I expect there to be, like, a kickoff thing, like a design doc. Nothing like that.Swyx [00:36:13]: Nothing like that.Swyx [00:36:14]: He just. We were debating in many meetingsMatei Zaharia [00:36:17]: Yeah.Swyx [00:36:17]: and then we're just debating whether it's possible or not from first principle.Matei Zaharia [00:36:20]: YeahSwyx [00:36:20]: and then, somebody just did it.Matei Zaharia [00:36:23]: Yeah, if you set yourself up so people do that'll be great. And that happened a bit with Omnigentt too. I think if I just had a doc on, like, we can make these together, everyone would, would think, “Oh, what about this? What about this?” But then you. if you try it out, it helps. And then if you have real users and they bash it and, like, it's still working, or in this case, if you have the workload, what the workload looks like, you can just test the same pattern then.Databricks' Culture of Fast PrototypingSwyx [00:36:47]: Yeah.Matei Zaharia [00:36:47]: Yeah.Swyx [00:36:47]: Tech aside, which is very cool, this is, like, the most important thing, the culture of innovation, and you don't have to ask my permission, you don't have like, do a whole form- formal process, just do it?Matei Zaharia [00:36:59]: Well, especially these days, I think withSwyx [00:37:01]: YeahMatei Zaharia [00:37:01]: AI, it's easier to buildSwyx [00:37:02]: But so, likeMatei Zaharia [00:37:03]: a prototypeSwyx [00:37:03]: I think you are very I made a lot of suite of, like, large companies and, like, I think that at scale, things slow down, and I'm sure you felt it already, but somehow you have this core of people that, like, are exempt. How? I think we hire and we work with really good people, and that's a very important part of it, and empowering them, but also spending a lot of time, maybe us in the trenches matter a lot also.Matei Zaharia [00:37:28]: Yeah, I think, I think first, people can adapt to being in the larger company, so that helps. And we wanna make sure they know that they can try stuff and settle debates and have a lot of examples of how it was done before, or launch a thing in beta or whatever. and then the other thing I do think as a company, like despite the size, we don't launch that many, like, products. We try to keep it pretty coherent. That's, that was the whole, like, theory of the company, was like instead of having, like, 20 Amazon services you need to set up, like a analytics and machine learning stack, you just have one, and it's, like, the same API, the same semantics across all of them, the same copy of the data. So that requires, like, unification. And then we added one more thing at a time. Like, we added storage with Delta Lake. We didn't used to do any storage. Then we added SQL, we added, machine learning platform stuff. So, but yeah, don't, don't do too many, but do those things well and, that also helps, it helps keep it manageable.Reynold Xin [00:38:33]: Yeah. The other thing we encourage a lot is instead of building, boil the ocean for everything, let's figure out how do we do it incrementally, how do we do it very quickly. Like, many of our productsMatei Zaharia [00:38:43]: YeahReynold Xin [00:38:43]: they're built in the span of weeks, and then we go to, hey. Like, usually my first question to whoever team is building is who's the target customer? Who are you working with? Are you on a first-name basis with them? Are you texting with them? I think having that very tight loop,Matei Zaharia [00:38:59]: Can you bring up another launch that comes to mind when, in this thing? I just want to give examples.Reynold Xin [00:39:04]: Omnigentt itself happened that way.Reynold Xin [00:39:05]: Yeah.Matei Zaharia [00:39:06]: Who's the customer? That's a good oneReynold Xin [00:39:34]: storage layer we did. we had, our largest customer at the time said like, “Okay, I need some. I want something in the cloud ‘cause, I. if the rest of our network is compromised, like this thing needs to be separate to store and query the events.” And then, talked to us, he said, “Okay, this is the rate of events per second. This is, like, the freshness I want. Can you do it?” So that was, like, way larger than any workload we had, and we had our, engineer, working on that, Michael Armbrust, and he worked just to make this work. And once it worked for them, it worked for everyone else. Yeah. This was early in the company, probably like four years in or something.Matei Zaharia [00:40:24]: 20- 2018?Swyx [00:40:26]: Yeah, ‘17, ‘18.Matei Zaharia [00:40:28]: Few companiesSwyx [00:40:28]: Do you have other examples?Matei Zaharia [00:40:30]: there'Swyx [00:40:31]: Maybe you have othersMatei Zaharia [00:40:31]: yeah, Clean Room, which is how you share data in a way without sharingSwyx [00:40:35]: YeahMatei Zaharia [00:40:35]: underlying data, but you allow specific operations. Those were done effectively initially just for two customers. I think the industry has a sense of, hey, maybe if you overfit to, like, one or two customers, it's gonna be really bad for you. But I think the, downside of overfitting is much smaller than the upside itself. And if you try to be too ambitious and boil the ocean, it's a much bigger problem.Swyx [00:40:58]: Yeah. Yeah.Matei Zaharia [00:40:58]: ‘Cause you might end up having no customer.Swyx [00:41:00]: Yeah, that's more, that's the more likely outcome.Matei Zaharia [00:41:02]: Yeah.Tech Companies vs. EnterprisesSwyx [00:41:03]: than you can pivot from there. I do think there is such a thing as a bad customer that sometimes you should fire. Yeah.Matei Zaharia [00:41:08]: They could exist sometimes if you drive. well, one of the challenge I think we probably see, and maybe many AI, so newer generation companies are seeing is, so tech companies are very different from tech companies or traditional enterprises.Swyx [00:41:22]: Yeah.Matei Zaharia [00:41:22]: And, if you optimize everything just for tech companies, you might have various challengesSwyx [00:41:27]: OhMatei Zaharia [00:41:27]: scaling them outside of tech companies.Swyx [00:41:28]: Okay, what likeMatei Zaharia [00:41:30]: YeahSwyx [00:41:30]: what like top three differences that you always think about?Reynold Xin [00:41:33]: Governance is a big oneMatei Zaharia [00:41:34]: I think, yeah, a big one is like, yeah, security, data privacy, governance, all that stuff. So usually if you're building some kinda like B2B or developer tool, like your biggest market is gonna be enterprises, but it's just very different. A company that's existed for like, it's had some form of IT for like 30 years, they have so many legacy systems or they operate in a regulated space. whereas a startup or, even like a, like sorta more recent tech company, all the. everything is new and pristine. So yeah, it's just different, and if you've never worked with enterprises or been in one, you just won't know about it.Reynold Xin [00:42:13]: Yeah.Matei Zaharia [00:42:13]: Yeah.Reynold Xin [00:42:13]: And the procurement process is probably quite different. There's far more stakeholders.Matei Zaharia [00:42:17]: Yeah, that is one. Yeah.Matei Zaharia [00:42:18]: Another piece that's interesting is I think some tech companies, people, will say, “Oh, I can build that myself,” right? I'll just build that myself.Matei Zaharia [00:42:27]: So then you go,Reynold Xin [00:42:28]: I don't think people say that about Databricks, butMatei Zaharia [00:42:31]: yeah, it dependsReynold Xin [00:42:32]: They do.Matei Zaharia [00:42:32]: They do?Matei Zaharia [00:42:32]: Yeah, the. Yeah, and it depends on the teams and things. So, but, on the other hand, like many of the enterprises say, “I don't, I never wanna be in the business of building that.” Like, I don't want my, whatever, I'm a retailer or something, I never wannaReynold Xin [00:42:45]: Yeah, sell clothes,Matei Zaharia [00:42:46]: be down because like some weird like nerd like couldn't get streaming pipelines working.Matei Zaharia [00:42:51]: That is not what I'm doing.Reynold Xin [00:42:53]: Yeah.Reynold Xin [00:42:53]: Yeah. This makes them great customers, to be honest, right?Matei Zaharia [00:42:55]: Yeah. But you have to understand that it's hard without having worked there and stuff, like you may not appreciate.Reynold Xin [00:43:01]: Look, I think they're all great. don't get me wrong, they have different challenges. But the, many of the tech companies, for sure there's a lot, far more DIY.Matei Zaharia [00:43:10]: On the flip side, you have people who are. they're very much experts in their domain, like they're building airplanes, they're, designing medicines, whatever, and they just want to bridge the technology, where like they don't wanna learn, databases or whatever. As cool as we think it is, even as interesting as the average software engineer might think it is to read a little bit, like they just never wanna know. They just say, “I have a, giant like, matrix or whatever with my, clinical data, like how do I, how do I like cluster it or whatever?” So yeah.The Dream Engine and Rewriting the Database StackReynold Xin [00:43:40]: Yeah. That's true. Okay, so and then I wanted to build out the dream engine, vision. where does this all lead? So one of the thing we, realized maybe a couple years back is that every single database engine out there, especially on the analytics side, are a decade old. pretty much everything that have reasonable traction are about a decade old. And they all started targeting some very specific narrow use cases, and then over time it's become more and more successful. They have grown in their ambition, and then they try to support more and more use cases. But the fastest way to support those use cases tend to be hacked around the abstractions that were initially created, that were not for those use cases.Matei Zaharia [00:44:23]: Yeah.Reynold Xin [00:44:23]: And then, but you can support them more or less okay. And before it, after 10 years of organic evolution that way, it becomes a gigantic pile of s**t.Reynold Xin [00:44:31]: the. And, but that includes Databricks. And very few company or very few systems, I think, have the gut to say, let's go start from scratch. Let's go back to the drawing board and design, knowing everything we know today after a decade of workloads and probably billions in revenue, let's attempt to rewrite it from scratch and make sure it will work and it can support all of these use cases. So we started doing that, but it's a very ambitious project. by the way, you can search on Wikipedia, there's this thing called second system syndrome.Matei Zaharia [00:45:08]: Yeah, I know that. Yes.Reynold Xin [00:45:09]: Or second system effect.Matei Zaharia [00:45:11]: Every developer must know what a second syndrome is.Reynold Xin [00:45:12]: It's you built your first thing and it works out great, and the second one's bound to fail because you become too ambitious.Reynold Xin [00:45:19]: And then you ask so many requirements.Matei Zaharia [00:45:20]: Or like you think everythingReynold Xin [00:45:21]: YeahMatei Zaharia [00:45:21]: and then you're likeReynold Xin [00:45:22]: You justMatei Zaharia [00:45:22]: you're, “I'm gonna design the perfect system this time.”Reynold Xin [00:45:24]: Yeah. And it turned out it's not perfect, and then it start failing and you're too ambitious, never launch, and you get killed. The, and the engineering team that started this, they were brilliant. I think we hired some of the best database engineers, on the planet into Databricks, and they were brilliant. Thank God it's not their second system. Many of them have built more than two in the past.Matei Zaharia [00:45:44]: Ah, nice.Reynold Xin [00:45:45]: But they were still worried about this, hey, building a database engine from scratch, I think the conventional wisdom is gonna take like five years to mature. This would be a very long-term project. It could fail. I think one of the engineers jokingly said, “Hey, maybe we just call it Reynolds Stream Engine.” If we name after a founder, maybe we then may get canceled or killed. But I think they built something pretty remarkable. they went back to. They changed the way the database engines were built from a paradigm point of view. Usually when y

In this episode, David Rubinstein talks with Craig Kerstiens, Snowflake's Director of Software Engineering on Postgres, about the enduring popularity of Postgres, a 30-year-old database technology, due to its reliability and recent innovations like JSON support and vector store databases. He highlighted the challenge of integrating operational (OLTP) and analytical (OLAP) data for AI applications, noting the importance of reducing latency and operational burden. Kerstiens introduced PG Lake, an open-source tool that embeds Iceberg in Postgres, simplifying data movement and querying. He also mentioned Snowflake's mirroring feature and the future of Postgres, emphasizing its continuous improvement and extension ecosystem.

A mass extinction event approaches, but not because of a meteor or global warming. The culprit this time is AI, and in particular, Large Language Models like ChatGPT and Claude. In their crosshairs are a slew of vendors selling analytical functionality: dashboards, visualizations, analyses, semantic layers, OLAP cubes and the like. For decades, these vendors have dominated enterprise decision-making, commanding 5, 6, even 7-figure pricetags to provide the scaffolding needed for insights. The LLMs now threaten that entire landscape, disrupting the foundation of data-driven workflows. True, the results of GenAI can be inaccurate, but their ease of use and relatively low cost will trump those concerns. Check out this hard-hitting session to hear DM Radio Host, Eric Kavanagh explain what's happening and why it matters to analysts everywhere. He'll examine the impact on the analytics industry, and explore ways that these vendors can stay relevant. He'll also offer advice for businesses looking to remain data-driven, and why data prep is where the action will be.

Index drauf und fertig. Klingt nach einem soliden Plan, oder? Leider nur so lange, bis die Daten wachsen, der Workload kippt oder der Optimizer plötzlich andere Entscheidungen trifft. Dann wird aus dem vermeintlichen Performance-Booster schnell ein Bremsklotz. Genau hier steigen wir in dieser Episode ein und schauen uns an, warum Indexstrukturen in Datenbanken viel mehr sind als ein technischer Quick Fix.Wir sprechen darüber, was ein Index eigentlich ist, wie Datenstruktur, Algorithmus, Hardware und Workload zusammenhängen und warum Begriffe wie Selektivität, Kardinalität, Full Table Scan, Write Amplification und Cache-Lokalität in der Praxis entscheidend sind. Außerdem schauen wir auf typische Datenbank-Themen wie Primary Key, B-Tree, Binary Search, Covering Index, Optimizer, Slow Query Log und Explain Statements. Dabei wird auch klar, warum ein Index manchmal hilft, manchmal ignoriert wird und manchmal sogar langsamer ist als gar kein Index.Wenn du mit PostgreSQL, MySQL, MariaDB oder ganz allgemein mit Datenbank-Performance arbeitest, bekommst du hier ein solides Fundament und einige praktische Denkanstöße für deinen Alltag als Softwareentwickler:in. Und ja, wir sprechen auch über Invisible Indexes in MySQL. Ein Feature, das fast wie ein Zaubertrick klingt, aber beim Testen und beim sicheren Aufräumen von Legacy-Systemen überraschend praktisch sein kann. Viel Spaß beim Hören und vielleicht beim anschließenden Blick auf dein Datenbankschema.Unsere aktuellen Werbepartner findest du auf https://engineeringkiosk.dev/partnersDas schnelle Feedback zur Episode:

Why are we still using row-based protocols like ODBC and JDBC in a column-oriented world? In this episode, I sit down with Ian Cook, co-founder of Columnar and a long-time Apache Arrow contributor, to discuss the critical infrastructure changes needed to speed up modern analytics and AI.We dive deep into the technical bottlenecks of legacy standards - specifically the "serialization tax" of converting columns to rows and back again - and how ADBC (Arrow Database Connectivity) solves this by keeping data columnar from end-to-end. Ian also shares his insights on the intersection of tabular data and LLMs, why AI agents need better access to OLAP systems, and the tension between vibe coding speed and the stability required for critical open-source infrastructure.

Kennst du diese Situation im Team: Jemand sagt "das skaliert nicht", und plötzlich steht der Datenbankwechsel schneller im Raum als die eigentliche Frage nach dem Warum? Genau da packen wir an. Denn in vielen Systemen entscheidet nicht das nächste hippe Tool von Hacker News, sondern etwas viel Grundsätzlicheres: Datenlayout und Zugriffsmuster.In dieser Episode gehen wir einmal tief runter in den Storage-Stack. Wir schauen uns an, warum Row-Oriented-Datastores der Standard für klassische OLTP-Workloads sind und warum "SELECT id" trotzdem oft fast genauso teuer ist wie "SELECT *". Danach drehen wir die Tabelle um 90 Grad: Column Stores für OLAP, Aggregationen über viele Zeilen, Spalten-Pruning, Kompression, SIMD und warum ClickHouse, BigQuery, Snowflake oder Redshift bei Analytics so absurd schnell werden können.Und dann wird es file-basiert: CSV bekommt sein verdientes Fett weg, Apache Parquet seinen Hype, inklusive Row Groups, Metadaten im Footer und warum das für Streaming und Object Storage so gut passt. Mit Apache Iceberg setzen wir noch eine Management-Schicht oben drauf: Snapshots, Time Travel, paralleles Schreiben und das ganze Data-Lake-Feeling. Zum Schluss landen wir da, wo es richtig weh tut, beziehungsweise richtig Geld spart: Storage und Compute trennen, Tiered Storage, Kafka Connect bis Prometheus und Observability-Kosten.Wenn du beim nächsten "das skaliert nicht" nicht direkt die Datenbank tauschen willst, sondern erst mal die richtigen Fragen stellen möchtest, ist das deine Folge.Bonus: DuckDB als kleines Taschenmesser für CSV, JSON und SQL kann dein nächstes Wochenend-Experiment werden.Unsere aktuellen Werbepartner findest du auf https://engineeringkiosk.dev/partnersDas schnelle Feedback zur Episode:

One of FourIn my so-called career, I've known four geniuses: one evil, one chaos made flesh, and two nice; Dan is in the last group.There are many theories around what makes someone a genius; I define it as the ability to make connections where others cannot see them. Dan is professionally (at least in EPMland) best known for his deconstruction of ASO Essbase, understanding its architecture and fundamentals, and how to optimise it.If you were in his "Essbase ASO Performance:  When NOT to Depend on MDX" session at Kscope 2010, you know just what I'm talking aboutI was sort of slack jawed by the end of the presentation. How on earth did he figure this out? BSO Essbase's architecture was (and is) fully documented. Thank Arbor Software. The same was (and still is) not true for ASO Essbase. Thank (or don't) Hyperion Solutions.Dan took apart ASO Essbase, hypothesising, testing, rejecting, confirming, and simply intellectually beating the product halfway to death to mirror Codd's 12 rules for OLAP. His work revolutionized (and made my life considerably easier amongst many others) ASO Essbase theory and practice.If you weren't there and you practice Essbase, you probably have a copy of Developing Essbase Applications:  Advanced Techniques for Finance and IT Professionals.You can still (it came out in 2012!) buy it here on Amazon. One day Oracle will change the architecture behind ASO Essbase (maybe this has already happened – I'm out of that space now), but until then, and maybe even in future if they mimic the way ASO works/worked, Dan's chapter is the place to be.Listen to the podcast and hear how Dan did it and of course more back story of a fascinating man in an equally fascinating industry across time.Part 1 of 2In editing (and yes, I did it this time round and yes, I'm not very good at it as you'll hear glitches in the recording – sorry) an episode, there's always a temptation to cut content to fit an hour long format for brevity. However, EPM Conversations is about, well, conversations and if you were sitting in a coffee shop with Dan, you'd want to know a bit about his personal life – that's more in the second part although you'll get a good feel for him in this episode as well.Be seeing you.

In the episode, we sat down with ClickHouse Co-Founder Yury Izrailevsky to unpack how one of the fastest open-source databases in the world became the analytics engine of choice for 2,000 customers including Harvey, Canva, HP, and Supabase. From its Yandex origins to powering AI observability, Yury shares how ClickHouse balances open-source roots, cloud innovation, and a remote-first culture moving at breakneck speed.ClickHouse's Series C valued the company at $6.35B earlier this year, and just yesterday they announced an extension to that round, just months after it was raised. In this episode, we dig into:Origins & Founding StoryClickHouse began as an internal project at Yandex to power a Google Analytics–style platform, focused on performance and scale.Open-sourced in 2016 - rapid global adoption laid the foundation for ClickHouse the company. Yury first discovered ClickHouse while at Google; impressed by its speed, he later co-founded the company in 2021 alongside Aaron Katz (ex-Elastic) and the original creator Alexey Milovidov.Why ClickHouse Stands OutColumn-oriented, open source OLAP database designed for massive-scale analytical processing.Excels in performance, efficiency, and cost - ideal for large data volumes and real-time analytics (and now AI workloads). Architectural choices:Columnar storage = better compression and faster execution.Separation of compute and storage enables elasticity, scalability, and resilience in the cloud.Open Source vs. CloudOpen-source version offers freedom and flexibility.Cloud product delivers much lower total cost of ownership and fully managed experience.Architectural parity between the two ensuring no vendor lock-in for customers. Customers can run the same queries on both; most stay with cloud due to simplicity and cost efficiency.Use Cases & Ecosystem4 main use cases:Real-time analyticsData WarehousingObservability AI / ML WorkloadsCompany Building & CultureFully remote from day one.Prioritized experienced, self-sufficient engineers over early-career hires.Built and launched GA version in less than a year - insane pace of innovation.Innovation & CommunityMonthly release cadence.Hundreds of integrations and connectors.Strong open-source and commercial communityAdvice for FoundersFocus on what matters most Hire mature, independent thinkers.Move fast but maintain quality; ClickHouse Cloud achieved production-grade quality in record time.

At Berlin Buzzwords, industry voices highlighted how search is evolving with AI and LLMs.- Kacper Łukawski (Qdrant) stressed hybrid search (semantic + keyword) as core for RAG systems and promoted efficient embedding models for smaller-scale use.- Manish Gill (ClickHouse) discussed auto-scaling OLAP databases on Kubernetes, combining infrastructure and database knowledge.- André Charton (Kleinanzeigen) reflected on scaling search for millions of classifieds, moving from Solr/Elasticsearch toward vector search, while returning to a hands-on technical role.- Filip Makraduli (Superlinked) introduced a vector-first framework that fuses multiple encoders into one representation for nuanced e-commerce and recommendation search.- Brian Goldin (Voyager Search) emphasized spatial context in retrieval, combining geospatial data with AI enrichment to add the “where” to search.- Atita Arora (Voyager Search) highlighted geospatial AI models, the renewed importance of retrieval in RAG, and the cautious but promising rise of AI agents.Together, their perspectives show a common thread: search is regaining center stage in AI—scaling, hybridization, multimodality, and domain-specific enrichment are shaping the next generation of retrieval systems.Kacper Łukawski Senior Developer Advocate at Qdrant, he educates users on vector and hybrid search. He highlighted Qdrant's support for dense and sparse vectors, the role of search with LLMs, and his interest in cost-effective models like static embeddings for smaller companies and edge apps. Connect: https://www.linkedin.com/in/kacperlukawski/Manish Gill Engineering Manager at ClickHouse, he spoke about running ClickHouse on Kubernetes, tackling auto-scaling and stateful sets. His team focuses on making ClickHouse scale automatically in the cloud. He credited its speed to careful engineering and reflected on the shift from IC to manager. Connect: https://www.linkedin.com/in/manishgill/André Charton Head of Search at Kleinanzeigen, he discussed shaping the company's search tech—moving from Solr to Elasticsearch and now vector search with Vespa. Kleinanzeigen handles 60M items, 1M new listings daily, and 50k requests/sec. André explained his career shift back to hands-on engineering. Connect: https://www.linkedin.com/in/andrecharton/Filip Makraduli Founding ML DevRel engineer at Superlinked, an open-source framework for AI search and recommendations. Its vector-first approach fuses multiple encoders (text, images, structured fields) into composite vectors for single-shot retrieval. His Berlin Buzzwords demo showed e-commerce search with natural-language queries and filters. Connect: https://www.linkedin.com/in/filipmakraduli/Brian Goldin Founder and CEO of Voyager Search, which began with geospatial search and expanded into documents and metadata enrichment. Voyager indexes spatial data and enriches pipelines with NLP, OCR, and AI models to detect entities like oil spills or windmills. He stressed adding spatial context (“the where”) as critical for search and highlighted Voyager's 12 years of enterprise experience. Connect: https://www.linkedin.com/in/brian-goldin-04170a1/Atita Arora Director of AI at Voyager Search, with nearly 20 years in retrieval systems, now focused on geospatial AI for Earth observation data. At Berlin Buzzwords she hosted sessions, attended talks on Lucene, GPUs, and Solr, and emphasized retrieval quality in RAG systems. She is cautiously optimistic about AI agents and values the event as both learning hub and professional reunion. Connect: https://www.linkedin.com/in/atitaarora/

In this episode of TheHeleyCast, I sit down with Jake A. Merrick—radio personality, father, and a potential future governor of Oklahoma. We dive into Jake's personal background and beliefs, and explore some of the most pressing issues facing the state. We cover a wide range of topics, including: The rise of charter schools and the state of Oklahoma's education systemWhy Jake and his family choose homeschooling over public or private school optionsOklahoma's status as the lowest-taxed state for oil companies—and what that means for our economyThe complexities of tribal agreements, casino revenue, and state-tribal relationsA fun closer: pheasant hunting, OLAP land access, and the rising cost of licenses in OklahomaThis is a conversation packed with insight, controversy, and a few laughs along the way. Don't miss it!Anything helps, guys!Donate @Venmo: (@theheleycast)Cashapp: ($danielheley)Social Media:TikTok: (@theheleycast)Facebook: TheHeleyCastTwitter: @TheHeleyCastInstagram: @forgot_my_heleysBrought to you by Heley Entertainment.Become a supporter of this podcast: https://www.spreaker.com/podcast/theheleycast--4329275/support.

The ClickHouse® project is a rising star in observability and analytics, challenging performance conventions with its breakneck speed. This open source OLAP column store, originally developed at Yandex to power their web analytics platform at massive scale, has quickly evolved into one of the hottest open source observability data stores around. Its published performance benchmarks have been the topic of conversation, outperforming many legacy databases and setting a new bar for fast queries over large volumes of data.Our guest for this episode is Robert Hodges, CEO of Altinity — the second largest contributor to the ClickHouse project. With over 30 years of experience in databases, Robert brings deep insights into how ClickHouse is challenging legacy databases at scale. We'll also explore Altinity's just-launched groundbreaking open source project—Project Antalya—which extends ClickHouse with Apache Iceberg shared storage, unlocking dramatic improvements in both performance and cost efficiency. Think 90% reductions in storage costs and 10 to 100x faster queries, all without requiring any changes to your existing applications.The episode was live-streamed on 20 May 2025 and the video is available at https://www.youtube.com/watch?v=VeyTL2JlWp0You can read the recap post: https://medium.com/p/2004160b2f5e/ OpenObservability Talks episodes are released monthly, on the last Thursday of each month and are available for listening on your favorite podcast app and on YouTube.We live-stream the episodes on Twitch and YouTube Live - tune in to see us live, and chime in with your comments and questions on the live chat.⁠⁠https://www.youtube.com/@openobservabilitytalks⁠  https://www.twitch.tv/openobservability⁠Show Notes:00:00 - Intro01:38 - ClickHouse elevator pitch02:46 - guest intro04:48 - ClickHouse under the hood08:15 - SQL and the database evolution path 11:20 - the return of SQL16:13 - design for speed 17:14 - use cases for ClickHouse19:18 - ClickHouse ecosystem22:22 - ClickHouse on Kubernetes 31:45 - know how ClickHouse works inside to get the most out of it 38:59 - ClickHouse for Observability46:58 - Project Antalya55:03 - Kubernetes 1.33 release55:32 - OpenSearch 3.0 release56:01 - New Permissive License for ML Models Announced by the Linux Foundation57:08 - OutroResources:ClickHouse on GitHub: https://github.com/ClickHouse/ClickHouse Shopify's Journey to Planet-Scale Observability: https://medium.com/p/9c0b299a04ddProject Antalya: https://altinity.com/blog/getting-started-with-altinitys-project-antalya https://cmtops.dev/posts/building-observability-with-clickhouse/ Kubernetes 1.33 release highlights: https://www.linkedin.com/feed/update/urn:li:activity:7321054742174924800/ New Permissive License for Machine Learning Models Announced by the Linux Foundation: https://www.linkedin.com/feed/update/urn:li:share:7331046183244611584  Opensearch 3.0 major release: https://www.linkedin.com/posts/horovits_opensearch-activity-7325834736008880128-kCqrSocials:Twitter:⁠ https://twitter.com/OpenObserv⁠YouTube: ⁠https://www.youtube.com/@openobservabilitytalks⁠Dotan Horovits============X (Twitter): @horovitsLinkedIn: www.linkedin.com/in/horovitsMastodon: @horovits@fosstodonBlueSky: @horovits.bsky.socialRobert Hodges=============LinkedIn: https://www.linkedin.com/in/berkeleybob2105/ 

Join us for a deep dive into the world of databases with CMU professor Andy Pavlo. We discuss everything from OLTP vs. OLAP, the challenges of distributed databases, and why cloud-native databases require a fundamentally different approach than legacy systems. We discuss modern Vector Databases, RAG, Embeddings, Text to SQL and industry trends.You can follow Andy's work on:Blue Sky Youtube What's New In Data is a data thought leadership series hosted by John Kutay who leads data and products at Striim. What's New In Data hosts industry practitioners to discuss latest trends, common patterns for real world data patterns, and analytics success stories.

Global Agile Summit Preview: Implementing Agile Practices for Data and Analytics Teams with Henrik Reich In this BONUS Global Agile Summit preview episode, we dive into the world of Agile methodologies specifically tailored for data and analytics teams. Henrik Reich, Principal Architect at twoday Data & AI Denmark, shares his expertise on how data teams can adapt Agile principles to their unique needs, the challenges they face, and practical tips for successful implementation. The Evolution of Data Teams "Data and analytics work is moving more and more to be like software development." The landscape of data work is rapidly changing. Henrik explains how data teams are increasingly adopting software development practices, yet there remains a significant knowledge gap in effectively using certain tools. This transition creates both opportunities and challenges for organizations looking to implement Agile methodologies in their data teams. Henrik emphasizes that as data projects become more complex, the need for structured yet flexible approaches becomes critical. Dynamic Teams in the Data and Analytics World "When we do sprint planning, we have to assess who is available. Not always the same people are available." Henrik introduces the concept of "dynamic teams," particularly relevant in consulting environments. Unlike traditional Agile teams with consistent membership, data teams often work with fluctuating resources. This requires a unique approach to sprint planning and task assignment. Henrik describes how this dynamic structure affects team coordination, knowledge sharing, and project continuity, offering practical strategies for maintaining momentum despite changing team composition. Customizing Agile for Data and Analytics Teams "In data and analytics, tools have ignored agile practices for a long time." Henrik emphasizes that Agile isn't a one-size-fits-all solution, especially for data teams. He outlines the unique challenges these teams face: Team members have varying expectations based on their backgrounds Experienced data professionals sometimes skip quality practices Traditional data tools weren't designed with Agile methodologies in mind When adapting Agile for data teams, Henrik recommends focusing on three key areas: People and their expertise Technology selection Architecture decisions The overarching goal remains consistent: "How can we deliver as quickly as possible, and keep the good mood of the team?" Implementing CI/CD in Data Projects "Our first approach is to make CI/CD available in the teams." Continuous Integration and Continuous Deployment (CI/CD) practices are essential but often challenging to implement in data teams. Henrik shares how his organization creates "Accelerators" - tools and practices that enable teams to adopt CI/CD effectively. These accelerators address both technological requirements and new ways of working. Through practical examples, he demonstrates how teams can overcome common obstacles, such as version control challenges specific to data projects. In this segment, we refer to the book How to Succeed with Agile Business Intelligence by Raphael Branger. Practical Tips for Agile Adoption "Start small. Don't ditch scrum, take it as an inspiration." For data teams looking to adopt Agile practices, Henrik offers pragmatic advice: Begin with small, manageable changes Use established frameworks like Scrum as inspiration rather than rigid rules Practice new methodologies together as a team to build collective understanding Adapt processes based on team feedback and project requirements This approach allows data teams to embrace Agile principles while accounting for their unique characteristics and constraints. The Product Owner Challenge "CxOs are the biggest users of these systems." A common challenge in data teams is the emergence of "accidental product owners" - individuals who find themselves in product ownership roles without clear preparation. Henrik explains why this happens and offers solutions: Clearly identify who owns the project from the outset Consider implementing a "Proxy PO" role between executives and Agile data teams Recognize the importance of having the right stakeholder engagement for requirements gathering and feedback Henrik also highlights the diversity within data teams, noting there are typically "people who code for living, and people who live for coding." This diversity presents both challenges and opportunities for Agile implementation. Fostering Creativity in Structured Environments "Use sprint goals to motivate a team, and help everyone contribute." Data work often requires creative problem-solving - something that can seem at odds with structured Agile frameworks. Henrik discusses how to balance these seemingly conflicting needs by: Recognizing individual strengths within the team Organizing work to leverage these diverse abilities Using sprint goals to provide direction while allowing flexibility in approach This balanced approach helps maintain the benefits of Agile structure while creating space for the creative work essential to solving complex data problems. About Henrik Reich Henrik is a Principal Architect and developer in the R&D Department at twoday Data & AI Denmark. With deep expertise in OLTP and OLAP, he is a strong advocate of Agile development, automation, and continuous learning. He enjoys biking, music, technical blogging, and speaking at events on data and AI topics. You can link with Henrik Reich on LinkedIn and follow Henrik Reich's blog.

In this episode, Lois Houston and Nikita Abraham chat with MySQL expert Perside Foster on the importance of keeping MySQL performing at its best. They discuss the essential tools for monitoring MySQL, tackling slow queries, and boosting overall performance.   They also explore HeatWave, the powerful real-time analytics engine that brings machine learning and cross-cloud flexibility into MySQL.   MySQL 8.4 Essentials: https://mylearn.oracle.com/ou/course/mysql-84-essentials/141332/226362 Oracle University Learning Community: https://education.oracle.com/ou-community LinkedIn: https://www.linkedin.com/showcase/oracle-university/ X: https://x.com/Oracle_Edu   Special thanks to Arijit Ghosh, David Wright, Kris-Ann Nansen, Radhika Banka, and the OU Studio Team for helping us create this episode.   ----------------------------------------------------------   Episode Transcript:   00:00 Welcome to the Oracle University Podcast, the first stop on your cloud journey. During this series of informative podcasts, we'll bring you foundational training on the most popular Oracle technologies. Let's get started! 00:25 Lois: Welcome to the Oracle University Podcast! I'm Lois Houston, Director of Innovation Programs with Oracle University, and with me today is Nikita Abraham, Team Lead: Editorial Services. Nikita: Hey everyone! In our last two episodes, we spoke about MySQL backups, exploring their critical role in data recovery, error correction, data migration, and more. Lois: Today, we're switching gears to talk about monitoring MySQL instances. We'll also explore the features and benefits of HeatWave with Perside Foster, a MySQL Principal Solution Engineer at Oracle. 01:02 Nikita: Hi, Perside! We're thrilled to have you here for one last time this season. So, let's start by discussing the importance of monitoring systems in general, especially when it comes to MySQL. Perside: Database administrators face a lot of challenges, and these sometimes appear in the form of questions that a DBA must answer. One of the most basic question is, why is the database slow? To address this, the next step is to determine which queries are taking the longest. Queries that take a long time might be because they are not correctly indexed. Then we get to some environmental queries or questions. How can we find out if our replicas are out of date? If lag is too much of a problem? Can I restore my last backup? Is the database storage likely to fill up any time soon? Can and should we consider adding more servers and scaling out the system? And when it comes to users and making sure they're behaving correctly, has the database structure changed? And if so, who did it and what did they do? And more generally, what security issues have arisen? How can I see what has happened and how can I fix it? Performance is always at the top of the list of things a DBA worries about. The underlying hardware will always be a factor but is one of the things a DBA has the least flexibility with changing over the short time. The database structure, choice of data types and the overall size of retained data in the active data set can be a problem. 03:01 Nikita: What are some common performance issues that database administrators encounter? Perside: The sort of SQL queries that the application runs can be an issue. 90% of performance problems come from the SQL index and schema group.  03:18 Lois: Perside, can you give us a checklist of the things we should monitor? Perside: Make sure your system is working. Monitor performance continually. Make sure replication is working. Check your backup. Keep an eye on disk space and how it grows over time. Check when long running queries block your application and identify those queries. Protect your database structure from unauthorized changes. Make sure the operating system itself is working fine and check that nothing unusual happened at that level. Keep aware of security vulnerabilities in your software and operating system and ensure that they are kept updated. Verify that your database memory usage is under control. 04:14 Lois: That's a great list, Perside. Thanks for that. Now, what tools can we use to effectively monitor MySQL?     Perside: The slow query log is a simple way to monitor long running queries. Two variables control the log queries. Long_query_time. If a query takes longer than this many seconds, it gets logged. And then there's min_exam_row_limit. If a query looks at more than this many rows, it gets logged. The slow query log doesn't ordinarily record administrative statements or queries that don't use indexes. Two variables control this, log_slow_admin_statements and log_queries_not_using_indexes. Once you have found a query that takes a long time to run, you can focus on optimizing the application, either by limiting this type of query or by optimizing it in some way. 05:23 Nikita: Perside, what tools can help us optimize slow queries and manage data more efficiently? Perside: To help you with processing the slow query log file, you can use the MySQL dump slow command to summarize slow queries. Another important monitoring feature of MySQL is the performance schema. It's a system database that provides statistics of how MySQL executes at a low level. Unlike user databases, performance schema does not persist data to disk. It uses its own storage engine that is flushed every time we start MySQL. And it has almost no interaction with the storage media, making it very fast. This performance information belongs only to the specific instance, so it's not replicated to other systems. Also, performance schema does not grow infinitely large. Instead, each row is recorded in a fixed size ring buffer. This means that when it's full, it starts again at the beginning. The SYS schema is another system database that's strongly related to performance schema. 06:49 Nikita: And how can the SYS schema enhance our monitoring efforts in MySQL? Perside: It contains helper objects like views and stored procedures. They help simplify common monitoring tasks and can help monitor server health and diagnose performance issues. Some of the views provide insights into I/O hotspots, blocking and locking issues, statements that use a lot of resources in various statistics on your busiest tables and indexes. 07:26 Lois: Ok… can you tell us about some of the features within the broader Oracle ecosystem that enhance our ability to monitor MySQL? Perside: As an Oracle customer, you also have access to Oracle Enterprise Manager. This tool supports a huge range of Oracle products. And for MySQL, it's used to monitor performance, system availability, your replication topology, InnoDB performance characteristics and locking, bad queries caught by the MySQL Enterprise firewall, and events that are raised by the MySQL Enterprise audit. 08:08 Nikita: What would you say are some of the standout features of Oracle Enterprise Manager? Perside: When you use MySQL in OCI, you have access to some really powerful features. HeatWave MySQL enables continuous monitoring of query statistics and performance. The health monitor is part of the MySQL server and gathers raw data about the performance of queries. You can see summaries of this information in the Performance Hub in the OCI Console. For example, you can see average statement latency or top 100 statements executed. MySQL metrics lets you drill in with your own custom monitoring queries. This works well with existing OCI features that you might already know. The observability and management framework lets you filter by resource type and across several dimensions. And you can configure OCI alarms to be notified when some condition is reached. 09:20 Lois: Perside, could you tell us more about MySQL metrics? Perside: MySQL metrics uses the raw performance data gathered by the health monitor to measure the important characteristic of your servers. This includes CPU and storage usage and information relevant to your database connection and queries executed. With MySQL metrics, you can create your own custom monitoring queries that you can use to feed graphics. This gives you an up to the minute representation of all the performance characteristics that you're interested in. You can also create alarms that trigger on some performance condition. And you can be notified through the OCI alarms framework so that you can be aware instantly when you need to deal with some issue.  10:22 Are you keen to stay ahead in today's fast-paced world? We've got your back! Each quarter, Oracle rolls out game-changing updates to its Fusion Cloud Applications. And to make sure you're always in the know, we offer New Features courses that give you an insider's look at all of the latest advancements. Don't miss out! Head over to mylearn.oracle.com to get started. 10:47 Nikita: Welcome back! Now, let's dive into the key features of HeatWave, the cloud service that integrates with MySQL. Can you tell us what HeatWave is all about? Perside: HeatWave is the cloud service for MySQL. MySQL is the world's leading database for web applications. And with HeatWave, you can run your online transaction processing or OLTP apps in the cloud. This gives you all the benefits of cloud deployments while keeping your MySQL-based web application running just like they would on your own premises. As well as OLTP applications, you need to run reports with Business Intelligence and Analytics Dashboards or Online Analytical Processing, or OLAP reports. The HeatWave cluster provides accelerated analytics queries without requiring extraction or transformation to a separate reporting system. This is achieved with an in-memory analytics accelerator, which is part of the HeatWave service. In addition, HeatWave enables you to create Machine Learning models to embed artificial intelligence right there in the database. The ML accelerator performs classification, regression, time-series forecasting, anomaly detection, and other functions provided by the various models that you can embed in your architecture. HeatWave can also work directly with storage outside the database. With HeatWave Lakehouse, you can run queries directly on data stored in object storage in a variety of formats without needing to import that data into your MySQL database. 12:50 Lois: With all of these exciting features in HeatWave, Perside, what core MySQL benefits can users continue to enjoy? Perside: The reason why you chose MySQL in the first place, it's still a relational database and with full transactional support, low latency, and high throughput for your online transaction processing app. It has encryption, compression, and high availability clustering. It also has the same large database support with up to 256 terabytes support. It has advanced security features, including authentication, data masking, and database firewall. But because it's part of the cloud service, it comes with automated patching, upgrades, and backup. And it is fully supported by the MySQL team. 13:50 Nikita: Ok… let's get back to what the HeatWave service entails. Perside: The HeatWave service is a fully managed MySQL. Through the web-based console, you can deploy your instances and manage backups, enable high availability, resize your instances, create read replicas, and perform many common administration tasks without writing a single line of SQL. It brings with it the power of OCI and MySQL Enterprise Edition. As a managed service, many routine DBA tests are automated. This includes keeping the instances up to date with the latest version and patches. You can run analytics queries right there in the database without needing to extract and transform your databases, or load them in another dedicated analytics system. 14:52 Nikita: Can you share some common use cases for HeatWave? Perside: You have your typical OLTP workloads, just like you'd run on prem, but with the benefit of being managed in the cloud. Analytic queries are accelerated by HeatWave. So your reporting applications and dashboards are way faster. You can run both OLTP and analytics workloads from the same database, keeping your reports up to date without needing a separate reporting infrastructure. 15:25 Lois: I've heard a lot about HeatWave AutoML. Can you explain what that is? Perside: HeatWave AutoML enables in-database artificial intelligence and Machine Learning. Externally sourced data stores, such as sensor data exported to CSV, can be read directly from object store. And HeatWave generative AI enables chatbots and LLM content creation. 15:57 Lois: Perside, tell us about some of the key features and benefits of HeatWave. Perside: Autopilot is a suite of AI-powered tools to improve the performance and applicability of your HeatWave queries. Autopilot includes two features that help cut costs when you provision your service. There's auto provisioning and auto shape prediction. They analyze your existing use case and tell you exactly which shape you must provision for your nodes and how many nodes you need. Auto parallel loading is used when you import data into HeatWave. It splits the import automatically into an optimum number of parallel streams to speed up your import. And then there's auto data placement. It distributes your data across the HeatWave cluster node to improve your query retrieval performance. Auto encoding chooses the correct data storage type for your string data, cutting down storage and retrieval time. Auto error recovery automatically recovers a fail node and reloads data if that node becomes unresponsive. Auto scheduling prioritizes incoming queries intelligently. An auto change propagation brings data optimally from your DB system to the acceleration cluster. And then there's auto query time estimation and auto query plan improvement. They learn from your workload. They use those statistics to perform on node adaptive optimization. This optimization allows each query portion to be executed on every local node based on that node's actual data distribution at runtime. Finally, there's auto thread pooling. It adjusts the enterprise thread pool configuration to maximize concurrent throughput. It is workload-aware, and minimizes resource contention, which can be caused by too many waiting transactions. 18:24 Lois: How does HeatWave simplify analytics within MySQL and with external data sources? Perside: HeatWave in Oracle Cloud Infrastructure provides all the features you need for analytics, all in one system. Your classic OLTP application run on the MySQL database that you know and love, provision in a DB system. On-line analytical processing is done right there in the database without needing to extract and load it to another analytic system. With HeatWave Lakehouse, you can even run your analytics queries against external data stores without loading them to your DB system. And you can run your machine learning models and LLMs in the same HeatWave service using HeatWave AutoML and generative AI. HeatWave is not just available in Oracle Cloud Infrastructure. If you're tied to another cloud vendor, such as AWS or Azure, you can use HeatWave from your applications in those cloud too, and at a great price. 19:43 Nikita: That's awesome! Thank you, Perside, for joining us throughout this season on MySQL. These conversations have been so insightful. If you're interested in learning more about the topics we discussed today, head over to mylearn.oracle.com and search for the MySQL 8.4: Essentials course.  Lois: This wraps up our season on the essentials of MySQL. But before we go, we just want to remind you to write to us if you have any feedback, questions, or ideas for future episodes. Drop us an email at ou-podcast_ww@oracle.com. That's ou-podcast_ww@oracle.com. Nikita: Until next time, this is Nikita Abraham… Lois: And Lois Houston, signing off! 20:33 That's all for this episode of the Oracle University Podcast. If you enjoyed listening, please click Subscribe to get all the latest episodes. We'd also love it if you would take a moment to rate and review us on your podcast app. See you again on the next episode of the Oracle University Podcast.

David Thionnet is a lifelong quail hunter from Oklahoma.  David and Travis share stories from their action-packed bobwhite quail hunt, David explains the quail hunting tradition that hooked him as a young child and his love of the covey rise, why numbers are up this year, habitat that helps and hurts quail populations, hunting with 6 bird dogs at one time, OLAP public access program, Quail Coalition, sharing in the hunting traditions, more stories from Beadie the blind bird hunter, and plenty more! Presented by: Walton's (https://www.waltons.com/) OnX Maps (https://www.onxmaps.com/) Aluma Trailers (https://www.alumaklm.com) Lucky Duck Premium Kennels (https://www.luckyduck.com/) & Hoksey Native Seeds (https://hokseynativeseeds.com)

David Thionnet is a lifelong quail hunter from Oklahoma.  David and Travis share stories from their action-packed bobwhite quail hunt, David explains the quail hunting tradition that hooked him as a young child and his love of the covey rise, why numbers are up this year, habitat that helps and hurts quail populations, hunting with 6 bird dogs at one time, OLAP public access program, Quail Coalition, sharing in the hunting traditions, more stories from Beadie the blind bird hunter, and plenty more! Presented by: Walton's (https://www.waltons.com/) OnX Maps (https://www.onxmaps.com/) Aluma Trailers (https://www.alumaklm.com) Lucky Duck Premium Kennels (https://www.luckyduck.com/) & Hoksey Native Seeds (https://hokseynativeseeds.com)

Like every other kind of technology, when it comes to databases there's no one-size-fits-all solution that's going to be the best thing for the job every time. That's what drives innovation and new solutions. It's ultimately also the story behind DuckDB, an open source relational database specifically designed for the demands of online analytical processing (OLAP), and particularly useful for data analysts, scientists and engineers.  To get a deeper understanding of DuckDB and how the product has developed, on this episode of the Technology Podcast hosts Ken Mugrage and Lilly Ryan are joined by Thoughtworker Ned Letcher and Thoughtworks alumnus Simon Aubury. Ned and Simon explain the thinking behind DuckDB, the design decisions made by the project and how its being used by data practitioners in the wild. Learn more about DuckDB: https://duckdb.org/why_duckdb.html    

How has the cloud transformed the way we work with data? While at Build in Seattle, Richard sat down with Arun Ulag, Microsoft CVP of Azure Data, to discuss how the cloud has transformed how we work with data. The pre-cloud practice of extract-transform-and-load into OLAP cubes has given way to the data lake - you don't need to pre-process data if you have all the compute you need on demand. Arun goes further into empowering analysts using tools like PowerBI - but the key is access to data. With Microsoft Fabric, data lives in OneLake - or anywhere through links! Today, the data analytics landscape spans different product stacks and clouds - but all are available to learn more about your business!Links:PowerBIPivot Tables in ExcelOne LakeApache IcebergSnowflakeDatabricksRecorded May 22, 2024

В этом выпуске: все подробности об использовании солнечных панелей на крыше своего дома, свежие видео с конференции pgconf.dev 2024, свежий котик и свежие темы наших слушателей. Шоуноты: [00:05:50] Чему мы научились за неделю [00:21:16] [Одной строкой] PostgreSQL Hacking Workshop [00:25:34] [Одной строкой] A shallow survey of OLAP and HTAP query engines [00:28:39] Tesla Powerwall [01:08:38]… Читать далее →

In this video I speak with Philippe Noël, about ParadeDB, which is an Elasticsearch alternative built on Postgres, modernizing the features of Elasticsearch's product suite, starting with real-time search and analytics. I hope you will enjoy and learn about the product. Chapters: 00:00 Introduction 01:12 Challenges with Elasticsearch and the Need for ParadeDB 02:29 Why Postgres? 06:30 Technical Details of ParadeDB's Search Functionality 18:25 Analytics Capabilities of ParadeDB 24:00 Understanding ParadeDB Queries and Transactions 24:22 Application Logic and Data Workflows 25:14 Using PG Cron for Data Migration 30:05 Scaling Reads and Writes in Postgres 31:53 High Availability and Distributed Systems 34:31 Isolation of Workloads 39:38 Database Upgrades and Migrations 41:21 Using ParadeDB Extensions and Distributions 43:02 Observability and Monitoring 44:42 Upcoming Features and Roadmap 46:34 Final Thoughts Important links: Links: GitHub: https://github.com/paradedb/paradedb Website: https://paradedb.com Docs: https://docs.paradedb.com/ Blog: https://blog.paradedb.com Follow me on Linkedin and Twitter: https://www.linkedin.com/in/kaivalyaapte/ and https://twitter.com/thegeeknarrator If you like this episode, please hit the like button and share it with your network. Also please subscribe if you haven't yet. Database internals series: https://youtu.be/yV_Zp0Mi3xs Popular playlists: Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA- Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17 Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_d Modern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsN Stay Curios! Keep Learning! #postgresql #datafusion #parquet #sql #OLAP #apachearrow #database #systemdesign #elasticsearch

In this video I speak with Andrew Lamb, Staff Software Engineer @Influxdb. We discuss FDAP (Flight, DataFusion, Arrow, Parquet) stack for modern OLAP database system design. Andrew shared some insights into why the FDAP stack is so powerful in designing and implementing a modern OLAP database. Chapters: 00:00 Introduction 01:48 Understanding Analytics: Transactional vs Analytical Databases 04:41 The Genesis and Goals of the FDAP Stack 09:31 Decoding FDAP: Flight, Data Fusion, Arrow, and Parquet 12:40 Apache Parquet: Revolutionizing Columnar Storage 17:18 Apache Arrow: The In-Memory Game Changer 23:51 Interoperability and Migration with Apache Arrow 27:10 Comparing Apache Parquet and Arrow 28:26 Exploring Data Mutability in Analytic Systems 29:19 Handling Data Updates and Deletions 29:24 The Role of Immutable Storage in Analytics 30:42 Optimizing Data Storage and Mutation Strategies 34:20 Introducing Flight: Simplifying Data Transfer 35:02 Deep Dive into Flight's Benefits and SQL Support 39:20 Unpacking Data Fusion's SQL Support and Extensibility 46:12 The Interplay of FDAP Components in Analytics 51:49 Future Directions and Innovations in Data Analytics 56:04 Concluding Thoughts on FDAP and Its Impact FDAP Stack: https://www.influxdata.com/glossary/fdap-stack/ FDAP Blog: https://www.influxdata.com/blog/flight-datafusion-arrow-parquet-fdap-architecture-influxdb/ InfluxDB: https://www.influxdata.com/ Follow me on Linkedin and Twitter: https://www.linkedin.com/in/kaivalyaapte/ and https://twitter.com/thegeeknarrator If you like this episode, please hit the like button and share it with your network. Also please subscribe if you haven't yet. Database internals series: https://youtu.be/yV_Zp0Mi3xs Popular playlists: Realtime streaming systems: https://www.youtube.com/playlist?list=PLL7QpTxsA4se-mAKKoVOs3VcaP71X_LA- Software Engineering: https://www.youtube.com/playlist?list=PLL7QpTxsA4sf6By03bot5BhKoMgxDUU17 Distributed systems and databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4sfLDUnjBJXJGFhhz94jDd_d Modern databases: https://www.youtube.com/playlist?list=PLL7QpTxsA4scSeZAsCUXijtnfW5ARlrsN Stay Curios! Keep Learning! #datafusion #parquet #sql #OLAP #apachearrow #database #systemdesign

PostgreSQL is an incredible general-purpose database, but it can't do everything. Every design decision is a tradeoff, and inevitably some of those tradeoffs get fundamentally baked into the way it's built. Take storage for instance - Postgres tables are row-oriented; great for row-by-row access, but when it comes to analytics, it can't compete with a dedicated OLAP database that uses column-oriented storage. Or can it?Joining me this week is Philippe Noël of ParadeDB, who's going to take us on a tour of Postgres' extension mechanism, from creating custom functions and indexes to Rust code that changes the way Postgres stores data on disk. In his journey to bring Elasticsearch's strengths to Postgres, he's gone all the way down to raw datafiles and back through the optimiser to teach a venerable old dog some new data-access tricks. –ParadeDB: https://paradedb.comParadeDB on Twitter: https://twitter.com/paradedbParadeDB on Github: https://github.com/paradedb/paradedbpgrx (Postgres with Rust): https://github.com/pgcentralfoundation/pgrxTantivy (Rust FTS library): https://github.com/quickwit-oss/tantivyPgMQ (Queues in Postgres): https://tembo.io/blog/introducing-pgmqApache Datafusion: https://datafusion.apache.org/Lucene: https://lucene.apache.org/Kris on Mastodon: http://mastodon.social/@krisajenkinsKris on LinkedIn: https://www.linkedin.com/in/krisjenkins/Kris on Twitter: https://twitter.com/krisajenkins

Welcome back to an episode where we're talking Vectors, Vector Databases, and AI with Linpeng Tang, CTO and co-founder of MyScale. MyScale is a super interesting technology. They're combining the best of OLAP databases with Vector Search. The project started back in 2019 where they forked ClickHouse and then adapted it to support Vector Storage, Indexing, and Search. The really unique and cool thing is you get the familiarity and usability of SQL with the power of being able to compare the similarity between unstructured data. We think this has really fascinating use cases for analytics well beyond what we're seeing with other vector database technology that's mostly restricted to building RAG models for LLMs. Also, because it's built on ClickHouse, MyScale is massively scalable, which is an area that many of the dedicated vector databases actually struggle with. We cover a lot about how vector databases work, why they decided to build off of ClickHouse, and how they plan to open source the database. Timestamps 02:29 Introduction 06:22 Value of a Vector Database 12:40 Forking ClickHouse 18:53 Transforming Clickhouse into a SQL vector database 32:08 Data modeling 32:56 What data can be Vectorized 38:37 Indexing 43:35 Achieving Scale 46:35 Bottlenecks 48:41 MyScale vs other dedicated Vector Databases 51:38 Going Open Source 56:04 Closing thoughts

NService Bus This episode of The Modern .NET Show is supported, in part, by NServiceBus, the ultimate tool to build robust and reliable systems that can handle failures gracefully, maintain high availability, and scale to meet growing demand. Make sure you click the link in the show notes to learn more about NServiceBus. Show Notes Yeah. So what I was thinking the other day is that what we want is to concentrate on the business logic that we need to implement and spend as small as little time as possible configuring, installing and figuring out the tools and libraries that we are using for this specific task. Like our mission is to produce the business logic and we should try to minimize the time that we spend on the tools and libraries that enable us to build the software. —Giorgi Dalakishvili Welcome to The Modern .NET Show! Formerly known as The .NET Core Podcast, we are the go-to podcast for all .NET developers worldwide and I am your host Jamie "GaProgMan" Taylor. In this episode, I spoke with Giorgi Dalakishvili about Postgresql, DuckDB, and where you might use either of them in your applications. As Giorgi points out, .NET has support for SQL Server baked in, but there's also support for other database technologies too: Yes, there are many database technologies and just like you, for me, SQL Server was the default go to database for quite a long time because it's from Microsoft. All the frameworks and libraries work with SQL Server out of the box, and have usually better support for SQL Server than for other databases. But recently I have been diving into Postgresql, which is a free database and I discovered that it has many interesting features and I think that many .NET developers will be quite excited about these features. The are very useful in some very specific scenarios. And it also has a very good support for .NET. Nowadays there is a .NET driver for Postgres, there is a .NET driver for Entity Framework core. So I would say it's not behind SQL server in terms of .NET support or feature wise. —Giorgi Dalakishvili He also points out that our specialist skill as developers is not to focus on the tools, libraries, and frameworks, but to use what we have in our collective toolboxes to build the business logic that our customers, clients, and users desire of us. And along the way, he drops some knowledge on an essential NuGet package for those of us who are using Entity Framework.. So let's sit back, open up a terminal, type in dotnet new podcast and we'll dive into the core of Modern .NET. Supporting the Show If you find this episode useful in any way, please consider supporting the show by either leaving a review (check our review page for ways to do that), sharing the episode with a friend or colleague, buying the host a coffee, or considering becoming a Patron of the show. Full Show Notes The full show notes, including links to some of the things we discussed and a full transcription of this episode, can be found at: https://dotnetcore.show/season-6/from-net-to-DuckDB-unleashing-the-database-evolution-with-giorgi-dalakishvili/ Useful Links Giorgi's GitHub DuckDB .NET Driver Postgres Array data type Postgres Range data type DuckDB DbUpdateException EntityFramework.Exceptions JsonB data type Vector embeddings Cosine similarity Vector databases: Chroma qdrant pgvector pgvector .NET library OLAP queries parquet files Dapper DuckDB documentation Dapr DuckDB Wasm; run DuckDB in your browser GitHub Codespaces Connecting with Giorgi: on Twitter on LinkedIn on his website Supporting the show: Leave a rating or review Buy the show a coffee Become a patron Getting in touch: via the contact page joining the Discord Music created by Mono Memory Music, licensed to RJJ Software for use in The Modern .NET Show Remember to rate and review the show on  Apple Podcasts, Podchaser, or wherever you find your podcasts, this will help the show's audience grow. Or you can just share the show with a friend. And don't forget to reach out via our Contact page. We're very interested in your opinion of the show, so please get in touch. You can support the show by making a monthly donation on the show's Patreon page at: https://www.patreon.com/TheDotNetCorePodcast.

Highlights from this week's conversation include:Michael's background and journey in data (0:33)The origin story of Druid (2:39)Experiences and growth in Data (8:08)Druid's evolution (21:46)Druid's architectural decisions (26:32)The user experience (30:06)The developer experience (35:14)The evolution of BI tools (40:55)Data architecture and integration (47:53)AI's impact on BI (52:26)What would Mike be doing if he didn't work in data? (56:27)Final thoughts and takeaways (57:02)The Data Stack Show is a weekly podcast powered by RudderStack, the CDP for developers. Each week we'll talk to data engineers, analysts, and data scientists about their experience around building and maintaining data infrastructure, delivering data and data products, and driving better outcomes across their businesses with data.RudderStack helps businesses make the most out of their customer data while ensuring data privacy and security. To learn more about RudderStack visit rudderstack.com.

Highlights from this week's conversation include:Tony's background and research focus (3:35)Challenges in academia and industry (6:15)Ph.D. student's routine (10:47)Academic paper review process (15:26)Aha moments in research (20:05)Academic lab structure (23:09)The decision to move from hardware to data research (24:43)Research focus on time series data management (27:40)Data modeling in time series and OLAP systems (32:01)Issues and potential solutions for parquet format (37:32)Role of external indices in parquet files (42:19)Tony's open source project (47:11)Final thoughts and takeaways (49:30)The Data Stack Show is a weekly podcast powered by RudderStack, the CDP for developers. Each week we'll talk to data engineers, analysts, and data scientists about their experience around building and maintaining data infrastructure, delivering data and data products, and driving better outcomes across their businesses with data.RudderStack helps businesses make the most out of their customer data while ensuring data privacy and security. To learn more about RudderStack visit rudderstack.com.

What is MySQL HeatWave? How do I get certified in it? Where do I start? Listen to Lois Houston and Nikita Abraham, along with MySQL Developer Scott Stroz, answer all these questions and more on this week's episode of the Oracle University Podcast. MySQL Document Store: https://oracleuniversitypodcast.libsyn.com/mysql-document-store Oracle MyLearn: https://mylearn.oracle.com/ Oracle University Learning Community: https://education.oracle.com/ou-community LinkedIn: https://www.linkedin.com/showcase/oracle-university/ X (formerly Twitter): https://twitter.com/Oracle_Edu Special thanks to Arijit Ghosh, David Wright, and the OU Studio Team for helping us create this episode. -------------------------------------------------------- Episode Transcript: 00:00 Welcome to the Oracle University Podcast, the first stop on your cloud journey. During this  series of informative podcasts, we'll bring you foundational training on the most popular  Oracle technologies. Let's get started! 00:26 Nikita: Welcome to the Oracle University Podcast! I'm Nikita Abraham, Principal Technical Editor with Oracle University, and with me is Lois Houston, Director of Innovation Programs. Lois: Hi there! For the last two weeks, we've been having really exciting discussions on everything AI. We covered the basics of artificial intelligence and machine learning, and we're taking a short break from that today to talk about the new MySQL HeatWave Implementation Associate Certification with MySQL Developer Advocate Scott Stroz. 00:59 Nikita: You may remember Scott from an episode last year where he came on to discuss MySQL Document Store. We'll post the link to that episode in the show notes so you can listen to it if you haven't already. Lois: Hi Scott! Thanks for joining us again. Before diving into the certification, tell us, what is MySQL HeatWave?  01:19 Scott: Hi Lois, Hi Niki. I'm so glad to be back. So, MySQL HeatWave Database Service is a fully managed database that is capable of running transactional and analytic queries in a single database instance. This can be done across data warehouses and data lakes. We get all the benefits of analytic queries without the latency and potential security issues of performing standard extract, transform, and load, or ETL, operations. Some other MySQL HeatWave database service features are automated system updates and database backups, high availability, in-database machine learning with AutoML, MySQL Autopilot for managing instance provisioning, and enhanced data security.  HeatWave is the only cloud database service running MySQL that is built, managed, and supported by the MySQL Engineering team. 02:14 Lois: And where can I find MySQL HeatWave? Scott: MySQL HeatWave is only available in the cloud. MySQL HeatWave instances can be provisioned in Oracle Cloud Infrastructure or OCI, Amazon Web Services (AWS), and Microsoft Azure. Now, some features though are only available in Oracle Cloud, such as access to MySQL Document Store. 02:36 Nikita: Scott, you said MySQL HeatWave runs transactional and analytic queries in a single instance. Can you elaborate on that? Scott: Sure, Niki. So, MySQL HeatWave allows developers, database administrators, and data analysts to run transactional queries (OLTP) and analytic queries (OLAP).  OLTP, or online transaction processing, allows for real-time execution of database transactions. A transaction is any kind of insertion, deletion, update, or query of data. Most DBAs and developers work with this kind of processing in their day-to-day activities.   OLAP, or online analytical processing, is one way to handle multi-dimensional analytical queries typically used for reporting or data analytics. OLTP system data must typically be exported, aggregated, and imported into an OLAP system. This procedure is called ETL as I mentioned – extract, transform, and load. With large datasets, ETL processes can take a long time to complete, so analytic data could be “old” by the time it is available in an OLAP system. There is also an increased security risk in moving the data to an external source. 03:56 Scott: MySQL HeatWave eliminates the need for time-consuming ETL processes. We can actually get real-time analytics from our data since HeatWave allows for OLTP and OLAP in a single instance. I should note, this also includes analytic from JSON data that may be stored in the database. Another advantage is that applications can use MySQL HeatWave without changing any of the application code. Developers only need to point their applications at the MySQL HeatWave databases. MySQL HeatWave is fully compatible with on-premise MySQL instances, which can allow for a seamless transition to the cloud. And one other thing. When MySQL HeatWave has OLAP features enabled, MySQL can determine what type of query is being executed and route it to either the normal database system or the in-memory database. 04:52 Lois: That's so cool! And what about the other features you mentioned, Scott? Automated updates and backups, high availability… Scott: Right, Lois. But before that, I want to tell you about the in-memory query accelerator. MySQL HeatWave offers a massively parallel, in-memory hybrid columnar query processing engine. It provides high performance by utilizing algorithms for distributed query processing. And this query processing in MySQL HeatWave is optimized for cloud environments.  MySQL HeatWave can be configured to automatically apply system updates, so you will always have the latest and greatest version of MySQL. Then, we have automated backups. By this, I mean MySQL HeatWave can be configured to provide automated backups with point-in-time recovery to ensure data can be restored to a particular date and time. MySQL HeatWave also allows us to define a retention plan for our database backups, that means how long we keep the backups before they are deleted. High availability with MySQL HeatWave allows for more consistent uptime. When using high availability, MySQL HeatWave instances can be provisioned across multiple availability domains, providing automatic failover for when the primary node becomes unavailable. All availability domains within a region are physically separated from each other to mitigate the possibility of a single point of failure. 06:14 Scott: We also have MySQL Lakehouse. Lakehouse allows for the querying of data stored in object storage in various formats. This can be CSV, Parquet, Avro, or an export format from other database systems. And basically, we point Lakehouse at data stored in Oracle Cloud, and once it's ingested, the data can be queried just like any other data in a database. Lakehouse supports querying data up to half a petabyte in size using the HeatWave engine. And this allows users to take advantage of HeatWave for non-MySQL workloads. MySQL AutoPilot is a part of MySQL HeatWave and can be used to predict the number of HeatWave nodes a system will need and automatically provision them as part of a cluster. AutoPilot has features that can handle automatic thread pooling and database shape predicting. A “shape” is one of the many different CPU, memory, and ethernet traffic configurations available for MySQL HeatWave. MySQL HeatWave includes some advanced security features such as asymmetric encryption and automated data masking at query execution. As you can see, there are a lot of features covered under the HeatWave umbrella! 07:31 Did you know that Oracle University offers free courses on Oracle Cloud Infrastructure? You'll find training on everything from cloud computing, database, and security to artificial intelligence and machine learning, all free to subscribers. So, what are you waiting for? Pick a topic, leverage the Oracle University Learning Community to ask questions, and then sit for your certification. Visit mylearn.oracle.com to get started.  08:02 Nikita: Welcome back! Now coming to the certification, who can actually take this exam, Scott? Scott: The MySQL HeatWave Implementation Associate Certification Exam is designed specifically for administrators and data scientists who want to provision, configure, and manage MySQL HeatWave for transactions, analytics, machine learning, and Lakehouse. 08:22 Nikita: Can someone who's just graduated, say an engineering graduate interested in data analytics, take this certification? Are there any prerequisites? What are the career prospects for them? Scott: There are no mandatory prerequisites, but anyone who wants to take the exam should have experience with MySQL HeatWave and other aspects of OCI, such as virtual cloud networks and identity and security processes. Also, the learning path on MyLearn will be extremely helpful when preparing for the exam, but you are not required to complete the learning path before registering for the exam. The exam focuses more on getting MySQL HeatWave running (and keeping it running) than accessing the data. That doesn't mean it is not helpful for someone interested in data analytics. I think it can be helpful for data analysts to understand how the system providing the data functions, even if it is at just a high level. It is also possible that data analysts might be responsible for setting up their own systems and importing and managing their own data. 09:23 Lois: And how do I get started if I want to get certified on MySQL HeatWave? Scott: So, you'll first need to go to mylearn.oracle.com and look for the “Become a MySQL HeatWave Implementation Associate” learning path. The learning path consists of over 10 hours of training across 8 different courses.  These courses include “Getting Started with MySQL HeatWave Database Service,” which offers an introduction to some Oracle Cloud functionality such as security and networking, as well as showing one way to connect to a MySQL HeatWave instance. Another course demonstrates how to configure MySQL instances and copy that configuration to other instances. Other courses cover how to migrate data into MySQL HeatWave, set up and manage high availability, and configure HeatWave for OLAP. You'll find labs where you can perform hands-on activities, student and activity guides, and skill checks to test yourself along the way. And there's also the option to Ask the Instructor if you have any questions you need answers to. You can also access the Oracle University Learning Community and discuss topics with others on the same journey. The learning path includes a practice exam to check your readiness to pass the certification exam. 10:33 Lois: Yeah, and remember, access to the entire learning path is free so there's nothing stopping you from getting started right away. Now Scott, what does the certification test you on? Scott: The MySQL HeatWave Implementation exam, which is an associate-level exam, covers various topics. It will validate your ability to identify key features and benefits of MySQL HeatWave and describe the MySQL HeatWave architecture; identify Virtual Cloud Network (VCN) requirements and the different methods of connecting to a MySQL HeatWave instance; manage the automatic backup process and restore database systems from these backups; configure and manage read replicas and inbound replication channels; import data into MySQL HeatWave; configure and manage high availability and clustering of MySQL HeatWave instances. I know this seems like a lot of different topics. That is why we recommend anyone interested in the exam follow the learning path. It will help make sure you have the exposure to all the topics that are covered by the exam. 11:35 Lois: Tell us more about the certification process itself. Scott: While the courses we already talked about are valuable when preparing for the exam, nothing is better than hands-on experience. We recommend that candidates have hands-on experience with MySQL HeatWave with real-world implementations. The format of the exam is Multiple Choice. It is 90 minutes long and consists of 65 questions. When you've taken the recommended training and feel ready to take the certification exam, you need to purchase the exam and register for it. You go through the section on things to do before the exam and the exam policies, and then all that's left to do is schedule the date and time of the exam according to when is convenient for you. 12:16 Nikita: And once you've finished the exam? Scott: When you're done your score will be displayed on the screen when you finish the exam. You will also receive an email indicating whether you passed or failed. You can view your exam results and full score report in Oracle CertView, Oracle's certification portal. From CertView, you can download and print your eCertificate and even share your newly earned badge on places like Facebook, Twitter, and LinkedIn. 12:38 Lois: And for how long does the certification remain valid, Scott? Scott: There is no expiration date for the exam, so the certification will remain valid for as long as the material that is covered remains relevant.  12:49 Nikita: What's the next step for me after I get this certification? What other training can I take? Scott: So, because this exam is an associate level exam, it is kind of a stepping stone along a person's MySQL training. I do not know if there are plans for a professional level exam for HeatWave, but Oracle University has several other training programs that are MySQL-specific. There are learning paths to help prepare for the MySQL Database Administrator and MySQL Database Developer exams. As with the HeatWave learning paths, the learning paths for these exams include video tutorials, hands-on activities, skill checks, and practice exams. 13:27 Lois: I think you've told us everything we need to know about this certification, Scott. Are there any parting words you might have? Scott: We know that the whole process of training and getting certified may seem daunting, but we've really tried to simplify things for you with the “Become a MySQL HeatWave Implementation Associate” learning path. It not only prepares you for the exam but also gives you experience with features of MySQL HeatWave that will surely be valuable in your career. 13:51 Lois: Thanks so much, Scott, for joining us today. Nikita: Yeah, we've had a great time with you. Scott: Thanks for having me. Lois: Next week, we'll get back to our focus on AI with a discussion on deep learning. Until then, this is Lois Houston… Nikita: And Nikita Abraham, signing off. 14:07 That's all for this episode of the Oracle University Podcast. If you enjoyed listening, please click Subscribe to get all the latest episodes. We'd also love it if you would take a moment to rate and review us on your podcast app. See you again on the next episode of the Oracle University  Podcast.

Follow: https://stree.ai/podcast | Sub: https://stree.ai/sub | New episodes every Monday! Join host Tim as he talks with Sandeep Dabade through demystifying the impressive star-tree index of Apache Pinot. Discover how this advanced feature optimizes OLAP databases, striking a balance between storage and high-speed query performance, and listen to real-world test cases showcasing its lightning-fast capabilities. Sandeep's blogs:► https://startree.ai/blog/best-practices-for-designing-tables-in-apache-pinot► https://startree.ai/blog/star-tree-indexes-in-apache-pinot-part-1-understanding-the-impact-on-query-performance► https://startree.ai/blog/star-tree-indexes-in-apache-pinot-part-2-understanding-the-impact-during-high-concurrency► https://startree.ai/blog/star-tree-index-in-apache-pinot-part-3-understanding-the-impact-in-real-customer

Summary Databases are the core of most applications, whether transactional or analytical. In recent years the selection of database products has exploded, making the critical decision of which engine(s) to use even more difficult. In this episode Tanya Bragin shares her experiences as a product manager for two major vendors and the lessons that she has learned about how teams should approach the process of tool selection. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Introducing RudderStack Profiles. RudderStack Profiles takes the SaaS guesswork and SQL grunt work out of building complete customer profiles so you can quickly ship actionable, enriched data to every downstream team. You specify the customer traits, then Profiles runs the joins and computations for you to create complete customer profiles. Get all of the details and try the new product today at dataengineeringpodcast.com/rudderstack (https://www.dataengineeringpodcast.com/rudderstack) You shouldn't have to throw away the database to build with fast-changing data. You should be able to keep the familiarity of SQL and the proven architecture of cloud warehouses, but swap the decades-old batch computation model for an efficient incremental engine to get complex queries that are always up-to-date. With Materialize, you can! It's the only true SQL streaming database built from the ground up to meet the needs of modern data products. Whether it's real-time dashboarding and analytics, personalization and segmentation or automation and alerting, Materialize gives you the ability to work with fresh, correct, and scalable results — all in a familiar SQL interface. Go to dataengineeringpodcast.com/materialize (https://www.dataengineeringpodcast.com/materialize) today to get 2 weeks free! This episode is brought to you by Datafold – a testing automation platform for data engineers that finds data quality issues before the code and data are deployed to production. Datafold leverages data-diffing to compare production and development environments and column-level lineage to show you the exact impact of every code change on data, metrics, and BI tools, keeping your team productive and stakeholders happy. Datafold integrates with dbt, the modern data stack, and seamlessly plugs in your data CI for team-wide and automated testing. If you are migrating to a modern data stack, Datafold can also help you automate data and code validation to speed up the migration. Learn more about Datafold by visiting dataengineeringpodcast.com/datafold (https://www.dataengineeringpodcast.com/datafold) Data projects are notoriously complex. With multiple stakeholders to manage across varying backgrounds and toolchains even simple reports can become unwieldy to maintain. Miro is your single pane of glass where everyone can discover, track, and collaborate on your organization's data. I especially like the ability to combine your technical diagrams with data documentation and dependency mapping, allowing your data engineers and data consumers to communicate seamlessly about your projects. Find simplicity in your most complex projects with Miro. Your first three Miro boards are free when you sign up today at dataengineeringpodcast.com/miro (https://www.dataengineeringpodcast.com/miro). That's three free boards at dataengineeringpodcast.com/miro (https://www.dataengineeringpodcast.com/miro). Your host is Tobias Macey and today I'm interviewing Tanya Bragin about her views on the database products market Interview Introduction How did you get involved in the area of data management? What are the aspects of the database market that keep you interested as a VP of product? How have your experiences at Elastic informed your current work at Clickhouse? What are the main product categories for databases today? What are the industry trends that have the most impact on the development and growth of different product categories? Which categories do you see growing the fastest? When a team is selecting a database technology for a given task, what are the types of questions that they should be asking? Transactional engines like Postgres, SQL Server, Oracle, etc. were long used as analytical databases as well. What is driving the broad adoption of columnar stores as a separate environment from transactional systems? What are the inefficiencies/complexities that this introduces? How can the database engine used for analytical systems work more closely with the transactional systems? When building analytical systems there are numerous moving parts with intricate dependencies. What is the role of the database in simplifying observability of these applications? What are the most interesting, innovative, or unexpected ways that you have seen Clickhouse used? What are the most interesting, unexpected, or challenging lessons that you have learned while working on database products? What are your prodictions for the future of the database market? Contact Info LinkedIn (https://www.linkedin.com/in/tbragin/) Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today? Closing Announcements Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ (https://www.pythonpodcast.com) covers the Python language, its community, and the innovative ways it is being used. The Machine Learning Podcast (https://www.themachinelearningpodcast.com) helps you go from idea to production with machine learning. Visit the site (https://www.dataengineeringpodcast.com) to subscribe to the show, sign up for the mailing list, and read the show notes. If you've learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com (mailto:hosts@dataengineeringpodcast.com)) with your story. To help other people find the show please leave a review on Apple Podcasts (https://podcasts.apple.com/us/podcast/data-engineering-podcast/id1193040557) and tell your friends and co-workers Links Clickhouse (https://clickhouse.com/) Podcast Episode (https://www.dataengineeringpodcast.com/clickhouse-data-warehouse-episode-88/) Elastic (https://www.elastic.co/) OLAP (https://en.wikipedia.org/wiki/Online_analytical_processing) OLTP (https://en.wikipedia.org/wiki/Online_transaction_processing) Graph Database (https://en.wikipedia.org/wiki/Graph_database) Vector Database (https://en.wikipedia.org/wiki/Vector_database) Trino (https://trino.io/) Presto (https://prestodb.io/) Foreign data wrapper (https://wiki.postgresql.org/wiki/Foreign_data_wrappers) dbt (https://www.getdbt.com/) Podcast Episode (https://www.dataengineeringpodcast.com/dbt-data-analytics-episode-81/) OpenTelemetry (https://opentelemetry.io/) Iceberg (https://iceberg.apache.org/) Podcast Episode (https://www.dataengineeringpodcast.com/tabular-iceberg-lakehouse-tables-episode-363) Parquet (https://parquet.apache.org/) The intro and outro music is from The Hug (http://freemusicarchive.org/music/The_Freak_Fandango_Orchestra/Love_death_and_a_drunken_monkey/04_-_The_Hug) by The Freak Fandango Orchestra (http://freemusicarchive.org/music/The_Freak_Fandango_Orchestra/) / CC BY-SA (http://creativecommons.org/licenses/by-sa/3.0/)

No episódio de hoje, Luan Moreno, Mateus Oliveira e Orlando Marley entrevistam Bill Inmon, criador do conceito de Data Warehouse e escritor de diversos livros com temáticas voltadas para dados.Data Warehouse é o conceito de centralização de dados analíticos das organizações, de forma estruturar um visão 360° do business. Neste episódio, você irá aprender: Diferenças entre OLTP e OLAP;Histórico dos dados para tomada de decisão;Criar um processo resiliente para entender os fatos dos dados.Falamos também, neste bate-papo, sobre os seguintes temas: História do Bill Inmon;Pilares de sistemas analíticos;Nova geração de plataforma de dados analíticos;Aprenda mais sobre análise de dados, como utilizar tecnologias para tornar o seu ambiente analítico confiável e resiliente com as palavras do pai do Data Warehouse. Bill Inmon = Linkedin Luan Moreno = https://www.linkedin.com/in/luanmoreno/

Episode #29 of "Can I get that software in blue?", a podcast by and for people engaged in technology sales. If you are in the technology presales, solution architecture, sales, support or professional services career paths then this show is for you! Tanya Bragin is the VP of Product at ClickHouse, a very fast open source OLAP databases used by some of the biggest companies in the world. Tanya was hired to lead the effort to build a SaaS offering of ClickHouse, having previously served in similar roles as a VP of Product at Elastic. Come listen to Tanya's unique insights about how to take an open source project to market with a SaaS offering including how to engage and motivate the engineering team, deciding which features to build, acquire, or buy, how to hire and set goals for IC PMs, and most importantly HOW TO SET PRICING. Our website: https://softwareinblue.com Twitter: https://twitter.com/softwareinblue LinkedIn: https://www.linkedin.com/showcase/softwareinblue Make sure to subscribe or follow us to get notified about our upcoming episodes: Youtube: https://www.youtube.com/channel/UC8qfPUKO_rPmtvuB4nV87rg Apple Podcasts: https://podcasts.apple.com/us/podcast/can-i-get-that-software-in-blue/id1561899125 Spotify: https://open.spotify.com/show/25r9ckggqIv6rGU8ca0WP2 Links mentioned in the episode: https://clickhouse.com/blog/my-journey-as-a-serial-startup-product-manager https://clickhouse.com/blog/vector-search-clickhouse-p1 https://clickhouse.com/blog/vector-search-clickhouse-p2

In modern systems, the amount of data keeps getting larger, and the time available keeps getting shorter. So it's almost inevitable that we're augmenting our general-purpose databases with dedicated analytics databases.This week we dive into the world of OLAP with a thorough look at Clickhouse, a high-performance, columnar database designed to "query billions of rows in microseconds."Alasdair Brown joins us to discuss what Clickhouse is, how it performs queries so quickly, and where it fits into a wider system. We talk about its origins as a Google Analytics-like, and how it's grown into one of the most popular OLAP databases around.There's a lot of ground to cover, and a lot of questions to ask, all in the service of faster answers...--Alasdair's Blog: alasdairb.comAlasdair on Threads: https://www.threads.net/@sdairsabAlasdair on LinkedIn https://www.linkedin.com/in/alasdair-brownKris on Twitter: https://twitter.com/krisajenkinsKris on LinkedIn: https://www.linkedin.com/in/krisjenkins/Clickhouse: https://clickhouse.com/Tinybird: https://www.tinybird.co/Birdhouse in your Soul: https://youtu.be/vn_or9gEB6g

On this episode of Data Driven, Frank and Andy interview Adam Ross Nelson. Adam is a consultant, where he provides insights on data science, machine learning and data governance. He recently wrote a book to help people get started in data science careers. Get the bookHow to Become a Data Scientist: A Guide for Established ProfessionalsSpeaker BioAdam Ross Nelson is an individual who initially pursued a career in law but ended up making a transition into education. After attending law school and working in administrative and policy roles in colleges and universities for several years, Adam hit a plateau in his career. Despite being a runner-up in national job searches multiple times, he felt that his lack of a PhD hindered his advancement in academia, while his legal background prevented him from being taken seriously by law professionals. Consequently, Adam decided to pursue a PhD in order to overcome this hurdle. During his PhD program, Adam discovered his passion and knack for statistics. His focus shifted towards predictive analytics projects, specifically ones related to identifying students in need of academic support. As he shared his work with friends, family, and coworkers, they began referring to him as a data scientist, a label that Adam initially resisted due to his legal and educational background. However, he eventually embraced the moniker, and even his boss started referring to him as the office's data scientist, despite HR not recognizing the title.Show Notes[00:03:26] Transitioning from law to education administration, plateaued career, runner-up in job searches, pursued PhD, became data scientist.[00:08:58] Data seen as liability, now asset. Examples: DBA's OLAP analysis, Walmart's weather-based inventory management.[00:12:56] Dotcom crash aftermath: fierce competition for jobs.[00:22:48] Salespeople have deep-seated insecurities and unique perspective.[00:29:31] Various classifications of data scientists and career advice.[00:35:55] "No full-field midfielder, data science is teamwork"[00:39:23] Navigating job descriptions for transitioning professionals.[00:42:56] Career coach helps professionals transition into data science.[00:49:41] First job: English teacher in Budapest, Hungary. Second job: Speaker for Mothers Against Drunk Driving.[00:56:30] Concerns about reliance on technology, especially AI.[01:00:22] Food options in lobbying are better in DC & state capitals. Also, check out the funny WY Files YouTube channel.[01:04:21] You can't separate them: LLM, bias, internet.[01:10:23] Ethics in consulting and avoiding dilemmas.

No episódio de hoje, Luan Moreno e Mateus Oliveira entrevistam Neha Pawar, atualmente Founder Engineer na StarTree.Apache Pinot é um banco de dados OLAP de baixa latência, que foi desenvolvido para queries analíticas dentro do Linkedin.O objetivo é resolver um dos problemas que tecnologias como o Apache Kafka não resolvem, consultar bilhões de eventos com performance e baixa latêcia .  Com  Apache Pinot, você tem os seguintes benefícios: Alto desempenho de consultas analíticas;  Dados que residem no Apache Pinot são comprimidos; Habilita milhares de acessos concorrentes aos dados residentes no Apache Pinot.Falamos também sobre os temas: Criação do Apache Pinot; User Facing Analytics;Tipos de Deployment no Apache Pinot;  O que vem por aí no Apache Pinot.Aprenda mais sobre Apache Pinot, uma tecnologia capaz de armazenar dados em tempo real, e executar queries com baixa latência, chegando até milissegundos.Neha Pawar = Linkedinhttps://pinot.apache.org/ Luan Moreno = https://www.linkedin.com/in/luanmoreno/

María Gordillo Villa, mexicana de Guadalajara, después de vivir toda una vida, con su familia, en el estado de Colorado, como inmigrantes ilegales. En el 2019, se unió al Centro legal para los inmigrantes del condado de Boulder, se convirtió en directora de Participación Comunitaria y actualmente está estudiando para recibir su acreditación de OLAP. María trabaja directamente con los beneficiarios de DACA proporcionándoles asistencia legal y los informa y guía en todo tipo de cambios legales que se presentan. Aparte, dedica todo su tiempo en brindar apoyo a todos los inmigrantes indocumentados para que legalicen su situación en Estados Unidos.María recuerda: “Muchas veces me toco tener debates muy fuertes con gente mayor de la raza blanca, tratar de que ellos se educaran de lo que es la migración. Cuando apliqué a la universidad, un consejero me dijo que no iba a llegar lejos porque no tenía un número de seguro social, para mí eso fue algo que quebró mi sueño.

From OLAP to Vertipaq to Power BI, Microsoft has a rich history of innovation and evolution in business intelligence. As data becomes an ever-increasing priority for organizations around the globe, Microsoft is now focused on the future with the launch of Microsoft Fabric, a unified SaaS solution which integrates all your data in one place. Fabric makes data management easier and more accessible for every user who works with data. In this episode you'll learn: How Amir and his team first pitched the idea of what is now Microsoft OLAP Services What happens when you think about the unevenness of data as a design principle Why you should demo products with things, people have opinions about Some questions we ask: How did the transition from OLAP to Vertipaq happen? When is the next great thing coming from Microsoft? What is Microsoft Fabric, and what will it do for the world of data? Guest bio Amir Netz, CTO of Microsoft's Intelligence Platform, including Power BI, Synapse, and more, joins Patrick LeBlanc on this week's episode of Insights Tomorrow. Amir is one of the leading world experts in business intelligence and analytics, holding over 80 patents. He is also the chief architect of Microsoft's BI offerings, including Power BI, Azure Synapse, Azure Data Factory, and more. Resources: View Amir Netz on LinkedIn View Patrick LeBlanc on LinkedIn Discover and follow other Microsoft podcasts at microsoft.com/podcasts Hosted on Acast. See acast.com/privacy for more information.

No episódio de hoje, Luan Moreno e Mateus Oliveira entrevistaram Xiran Waibel, atualmente engenheira de dados Sênior na Netflix.A Engenharia de Dados é um das profissões que estão em alta no mercado de trabalho, mas entender como funciona é algo que até hoje as empresas tem dificuldades.Engenharia de Dados engloba:Entendimento de novas tecnologias orientadas a Big DataTrabalhar com soluções de dados que resolvem problemas de negócioConstruções de pipelines de dados resilientes e escaláveisFalamos também nesse bate-papo sobre os seguintes temas:Engenharia de Dados na Netflix;Dicas de Engenharia de Dados;Soft Skills;Comunidade.Aprenda um pouco como a Netflix trabalha utilizando dados como um dos produtos mais valiosos da empresa, além de uma cultura interna forte e funcional.Xiran Waibel Medium  Luan Moreno = https://www.linkedin.com/in/luanmoreno/

这一期可谓众望所归，Monica 硬核对话 PingCAP 联合创始人CTO 黄东旭。大年初四，我们在广西老家的一个露营地，吹着15度的暖风，聊了两个小时的数据库、中美开发者市场和技术产品哲学，好不惬意！ Hello World, who is OnBoard?! 如果你对开源有所关注，一定知道开源分布式数据库 TiDB 及其背后的公司 PingCAP。PingCAP 可以说是中国商业化开源公司的先驱。2015年成立至今，TiDB 从零开始，在Github 上超过3万star, 超过800位来自世界各地的贡献者，除了包括众多一线互联网大厂在内的开源用户，PingCAP 还服务了20多个国家3000多个客户。 作为联合创始人和 CTO 的黄东旭，不仅是资深的基础软件工程师，架构师，还是狂热的开源爱好者以及开源软件作者，内存数据库 Redis 的高性能集群架构解决方案 Codis 就是他的作品之一 。 此外，PingCAP 也可谓是中国基础软件公司走向世界的先行者。过去一年多的时间，东旭几乎全身心铺在硅谷，对于中美市场异同、什么是给开发者用的数据库，什么是未来的开发范式，当然还有开脑洞的讨论：现代AI发展会对数据领域有什么影响？这次掏心窝的分享，一定能给你非常多启发！ 最后需要一提的是，生活中的东旭还是一名摇滚乐手。本期最后的彩蛋，你会听到他展示最新学习的乐器！ 干货满满准备上车，Enjoy! 【感谢AroundDeal 赞助本期播客！】 随着越来越多 IT SaaS、智能制造企业都开始开拓全球市场，精准获取海外B端客户线索就成了首要问题。AroundDeal 为企业提供全球商业信息SaaS平台。他们的平台上1亿多条联系人、企业及商业情报信息，覆盖全球200多个国家地区，3000多种细分行业，并且持续更新。绝对是企业出海的必备神器！ OnBoard! 听众还有福利！访问 AroundDeal.com，在 Contact Sales 中备注 Onboard, 即可领取七天免费试用！还不赶紧去试试，立即找到你的下一个海外理想客户！ 我们都聊了什么 02:30 开场：PingCAP 介绍，发展历程的几个重要节点 07:00 OLAP, OLTP 科普：场景和开发难点有什么不同 12:12 “未来的数据库都会是 HTAP 数据库”？！为什么说 HTAP 的核心能力是在TP能力 21:35 分久必合，数据库长尾需求会越来越收敛吗 24:03 Why now: 为什么 HTAP 概念在最近几年开始被广泛接受？ 27:19 为什么 HTAP 概念在 infra 成熟的美国，流行得反而更晚？ 29:55 主流的 HTAP 架构是怎样的？用户应该如何选型？ 34:26 各个大厂都在跟进 HTAP 产品，对于早期公司意味着什么？ 37:11 如何理解“万物皆可 SQL”？ 对于数据库厂商意味着什么？ 41:17 什么是“数据库的第一性原理”？ 46:25 Vercel 如何做好开发者体验？为什么说要做好 infra, 你应该关注的反而不是 infra? 53:23 对于新创的数据库公司，没有 Serverless 就上不了牌桌？ 59:45 serverless 开脑洞的未来！解决数据孤岛的终极方案 64:38 好的数据库 vs 好的数据库产品 66:12 为什么说新的数据库公司，需要新的研发组织？ PingCAP 发生了哪些组织挑战与变革？ 71:18 数据库用户的组织架构在发生哪些变化？ 74:43 开源社区在组织不同阶段的作用有什么不一样？为什么说期待开源到商业转化不能太乐观？ 80:27 在美国有哪些新出现的 infra 公司和新的技术趋势？ 84:46 开拓北美市场，要从科技行业客户破圈，有哪些挑战？对搭建团队有哪些挑战？ 92:21 美国之外的海外市场：东南亚有惊喜，顺序打法有讲究 98:01 中国与海外市场的异同，为什么创业公司也要先啃硬骨头 100:11 创业8年回顾：有哪些经验和心得？ 104:11 未来令人兴奋的机会 108:40 不得不了的：chatGPT, 生成式 AI 的脑洞 111:58 快问快答：有彩蛋！ 我们提到的公司 & 重点名词 Snowflake SingleStore Neon Vercel Supabase OSS Insight Snowflake Unistore Google Spanner ZeroETL Reverse ETL OLAP OLTP Serverless 嘉宾的推荐 推荐的书：禅与摩托车维修艺术（Zen and the Art of Motorcycle Maintenance, by Robert M.Pirsig） 推荐的书：Unix 编程艺术, by Eric S·Raymond Rob Pike, Go 语言之父 Werner Vogels, Amazon CTO Bansuri, 印度乐器 喜欢的音乐人：Sonic Youth 欢迎关注M小姐的微信公众号，了解更多中美软件行业的干货内容！ M小姐研习录 (ID: MissMStudy) 大家的点赞、评论、转发是对我们最好的鼓励！ 如果你有希望我们聊的话题，希望我们邀请的访谈嘉宾，都欢迎在留言中告诉我们 ❤️

В этом выпуске: как упростить себе жизнь при помощи Хренолоджи, специализированный индексы для OLAP не имеющие ничего общего с BRIN, новый проект Wildebeest от Cloudflare, а также забытые игры про обтягивающий водолазный костюм и гигантского осьминога с щупальцами, удачно незалоченный от перепрошивки Nintendo Game & Watch, а также темы наших слушателей. Шоуноты: [00:00:24] Чему мы… Читать далее →

On today's episode, we're joined by David Wang, VP of Product Marketing at Imply and Muthu Lalapet, Director of Worldwide Sales Engineering at Imply to dig into Apache Druid, a high performance, real-time analytics database. Thousands of companies are already using Druid today, from Netflix to Salesforce. But what is Apache Druid best used for? What types of projects? What data sets are Druid users working with? What are companies doing with Druid?  Listen to hear real-life examples of where Druid works best: Operational visibility at scale, customer-facing analytics, rapid-drill down exploration and real-time decisioning.

Neste episódio entrevistamos o Kishore Gopalakrishna, Co-Fundador e CEO da empresa StarTree, Luan Moreno e Mateus Oliveira batem um papo com o co-criador dessa poderosa ferramenta chamada Apache Pinot.O Pinot é um OLAP DataStore desenvolvido para responder consultas analíticas com tempo de resposta na casa dos milissegundos, podendo ser considerado um banco de dados para consultas em tempo-real. Capaz de ingerir de fontes de dados em Batch (Hadoop HDFS, Amazon S3, Azure ADLS, Google Cloud Storage), bem como fontes de dados em Stream (Apache Kafka, Apache Pulsar, Amazon Kinesis).O Pinot foi projetado para executar consultas OLAP em tempo real, com baixa latência em grandes quantidades de eventos para entregar o conceito de User-Facing Analytics.Foi criado e desenvolvido por engenheiros do LinkedIn e do Uber e projetado para escalar e expandir sem limites.Apache PinotKishore GopalakrishnaStarTree Luan Moreno = https://www.linkedin.com/in/luanmoreno/

No episódio de hoje estamos com Andre Pretto, profissional com uma bagagem de 15 anos em Engenharia de Dados, trabalhando ativamente no mercado europeu.Suas stacks têm foco em soluções open source, improvement cloud no ks8 e streaming de dados.Veremos que o Click House é um banco de dados colunar de código aberto para processamento analítico online, usado em cenários que necessitam de análise de dados em grande velocidade.Por exemplo, a telemetria de IOT, análise de métrica, entre outros.Fique com a gente até o final, no nosso Engenharia de Dados Cast! Luan Moreno = https://www.linkedin.com/in/luanmoreno/

With a background in finance and a history of great communication and a passion for problem-solving, Ed Hansberry, an Assistant Director with P3 Adaptive, embodies the spirit of P3 Adaptive. Ed was recently awarded his 13th Microsoft MVP Award so of course, we wanted to know more about his achievement, his passions, his adaptability, and most of all, his insights on change so we invited him to join us today for a chat. Early on, in the conversation, Rob and Ed delve into defining change and that led to a lively discussion on the process of change, the successful process of change, and the difference between them. Here's a hint, it's always the people! Knowing that people drive success, Ed extensively volunteers his time in the Microsoft Power BI User Community supporting users around the world with problems and questions around Power BI and, since he has been recognized as a Super User by Microsoft, we really can say that helping people is his superpower. He is leading change one question at a time. This episode isn't just about change, though, the evolution of technology and software is embedded throughout the conversation from cube functions to the hidden power of the innocuously named OLAP dropdown in Excel. And finally, we get some great insight on formerly cutting-edge technology that has since gone obsolete. We hear a firsthand account of the tragic end of the Microsoft phone.  You never know what you will learn when the conversation starts to flow. As always, be sure to leave a review on your favorite podcast platform and tell a friend about Raw Data by P3 Adaptive, where data meets the human element. Also on this episode: iPaQ N NTP Not Necessarily the News NNTNs: All about Sniglets That Tufte book . . . MDX in Excel Cube Functions in Excel Disconnected Slicers with DAX Variables & SELECTEDVALUES Field Parameters in Power Bi Skynet Yoda Chong and the Treehouse of Wonder, w/ Donald Farmer Tabluar Editor DAX Studio A Single Complete Leader, w/ P3 Pres & COO Kellan Danielson Who Moved My Cheese

ClickHouse, the lightning-fast open source OLAP database, was initially released in 2016 as an open source project out of Yandex, the Russian search giant. In 2021, Aaron Katz helped form a group to spin it out of Yandex as an independent company, dedicated to the development + commercialization of the open source project. In this conversation with Tristan and Julia, Aaron gets into why he believes open source, independent software companies are the future. And of course, this conversation wouldn't be complete without a riff on the classic "one database to rule all workloads" thread. For full show notes and to read 6+ years of back issues of the podcast's companion newsletter, head to https://roundup.getdbt.com.  The Analytics Engineering Podcast is sponsored by dbt Labs.

This week, we're busting myths around Cloud Spanner with our guests Pritam Shah and Vaibhav Govil. Mark Mirchandani and Max Saltonstall host this episode and learn about the fantastic capabilities of Cloud Spanner. Our guests give us a quick run-down of Spanner database software and its fully-managed offerings. Spanner's unique take on the relational database has sparked some myths. We start by addressing cost and the idea that Spanner is expensive. With its high availability achieved through synchronously replicating data, failures are virtually a non-issue, making the cost well worth it. Our guests describe other features that add to the value of Spanner as well. Workloads of any size are a good fit for Spanner because of its scalability and pricing based on use. Despite rumors, Spanner is now very easy to start using. New additions like the PostgreSQL interface and ORM support have made the usability of Spanner much more familiar. Regional and multi-regional instances are supported, busting the myth that Spanner is only good for global workloads. Our guests offer examples of projects using local and global configurations with Spanner. In the database world, Vaibhav sees trends like the convergence of non-relational and relational databases as well as convergence in the OLTP and OLAP database semantics, and he tells us how Spanner is adapting and growing with these trends. Pritam points out that customers are paying more attention to total cost of ownership, the importance of scalable and reliable database solutions, and the peace of mind that comes with a managed database system. Spanner helps customers with these, freeing up business resources for other things. This year, Spanner has made many announcements about new capabilities coming soon, like PostgreSQL interface on spanner GA, Query Insights visualization tools, cross-regional backups GA, and more. We hear all about these awesome updates. Pritam Shah Pritam is the Director of Engineering for Cloud Spanner. He has been with Google for about four and a half years. Before Spanner, he was the Engineering Lead for observability libraries at Google. That included Distributed Tracing and Metrics at Google scale. His mission was to democratize the instrumentation libraries. That is when he launched Open Census and then took on Cloud Spanner. Vaibhav Govil Vaibhav is the Product lead for Spanner. He has been in this role for the past three years, and before this he was a Product Manager in Google Cloud Storage in Google. Overall, he has spent close to four years at Google, and it has been a great experience. Cool things of the week Our plans to invest $9.5 billion in the U.S. in 2022 blog A policy roadmap for 24⁄7 carbon-free energy blog SRE Prodcast site Meet the people of Google Cloud: Grace Mollison, solutions architect and professional problem solver blog GCP Podcast Episode 224: Solutions Engineering with Grace Mollison and Ann Wallace podcast Interview Spanner site Cloud Spanner myths busted blog PostgreSQL interface docs Cloud Spanner Ecosystem site Spanner: Google's Globally-Distributed Database white paper Spanner Docs docs Spanner Qwiklabs site Using the Cloud Spanner Emulator docs GCP Podcast Episode 62: Cloud Spanner with Deepti Srivastava podcast GCP Podcast Episode 248: Cloud Spanner Revisited with Dilraj Kaur and Christoph Bussler podcast Cloud Spanner federated queries docs What's something cool you're working on? Max is working on a new podcast platform and some spring break projects. Hosts Mark Mirchandani and Max Saltonstall

Frank McSherry, Chief Scientist at Materialize talks to Host Akshay Manchale about Materialize which is a SQL database that maintains incremental views over streaming data. Frank talks about how Materialize can complement analytical systems...

When you think about selecting a database engine for your project you typically consider options focused on serving multiple concurrent users. Sometimes what you really need is an embedded database that is blazing fast for single user workloads. DuckDB is an in-process database engine optimized for OLAP applications to speed up your analytical queries that meets you where you are, whether that's Python, R, Java, even the web. In this episode, Hannes Mühleisen, co-creator and CEO of DuckDB Labs, shares the motivations for creating the project, the myriad ways that it can be used to speed up your data projects, and the detailed engineering efforts that go into making it adaptable to any environment. This is a fascinating and humorous exploration of a truly useful piece of technology.

Você gostaria de compreender a diferença entre Data Warehouse e Data Lakehouse e ir além para entender de fato o que acontece na realidade das empresas que adotam essas soluções? O Orlando Marley é um dos grandes especialistas nessa área e com ele, iremos dar dicas de como entender melhor esses dois paradigmas e como você pode unir essas duas soluções para entregar um Analytics marcante para sua empresa.Data Lakehouse é um novo conceito que vem ganhando tração rapidamente e para você poder se destacar como um engenheiro de dados se faz necessário aprender sobre. Luan Moreno = https://www.linkedin.com/in/luanmoreno/