Podcasts about data engineering

In this episode of The Tech Trek, Amir speaks with Patrick Leung, CTO of Faro Health, about what it takes to lead an engineering organization through a transformation to become an AI-first company. From redefining the product roadmap to managing cultural and technical shifts, Patrick shares practical insights on team structure, skill development, and delivering AI-enabled features in a regulated domain like clinical trials. This is a must-listen for tech leaders navigating similar transitions.

Highlights from this week's conversation include:Pedram's Background and Journey in Data (1:13)Marketing vs. Data Engineering (2:30)Understanding Marketing Pressures (4:16)Attribution Models and Accountability (8:13)Balancing Marketing and Team Management (12:25)Introduction to Dagster Components (15:00)AI Integration with Data Engineering (19:05)Challenges in Data Support (22:05)Self-Service Data Access (26:07)AI in Data Management (28:25)Organizing Data in Technical Teams (31:25)Challenges in Real-Time Data (33:28)Final Thoughts and Takeaways (37:01)The Data Stack Show is a weekly podcast powered by RudderStack, the CDP for developers. Each week we'll talk to data engineers, analysts, and data scientists about their experience around building and maintaining data infrastructure, delivering data and data products, and driving better outcomes across their businesses with data.RudderStack helps businesses make the most out of their customer data while ensuring data privacy and security. To learn more about RudderStack visit rudderstack.com.

Host Charlotte Roberts engages with Adarsh Mahesan Shyna, Lead Project Leader for Data Engineering & Architecture Strategy at Nordea; Dhany Saputra, Head of Data Science, AI, and Cloud Architecture at Micropower Group; and Charanya Thangaraj, Service Area Manager for Data Access Services at Volvo Buses. The discussion explores how data is driving innovation across finance, manufacturing, and mobility. Learn how these leaders implement scalable architectures and use advanced analytics to fuel smarter, faster business decisions in today's digital economy.

Do you know what truly separates exceptional leaders from the rest? Zoe Kelly, Head of Data Engineering at Reward, journey reveals a powerful truth: leadership isn't about technical prowess alone, but about creating human-centered environments where data professionals can thrive. In a world obsessed with complex technologies and buzzwords, Kelly demonstrates that vulnerability, communication, and genuine connection are the real superpowers of data leadership. Her remarkable story of scaling a data team from three to seventeen members while navigating personal mental health challenges offers a blueprint for modern leadership. By prioritising psychological safety, continuous learning, and empathetic communication, she transforms data engineering from a purely technical discipline into a deeply human experience. Kelly's approach challenges traditional narratives, showing that the most innovative data strategies emerge not from tools, but from understanding and supporting the people behind the data. Her insights are a masterclass in authentic, transformative leadership.0:00 Intro2:56 Zoe's Journey into Data 4:26 Evolution of Data and Leadership 7:41 Balancing Leadership and Team Dynamics 9:28 Challenges and Strategies in Data Leadership 15:05 Job Descriptions and Industry Expectations 18:39 Diversity and Inclusion in Data 23:11 Mental Health and Remote Work 31:36 Data in Golf and Personal Insights 47:11 Lightning Round and Final ThoughtsFor more information head to https://www.miraitalent.com/Welcome to “Let's Talk Data”, a deep dive into the transformative realm of data with the trailblazers of this disruptive industry.Your host Emma Crabtree explores the latest trends and developments in the data sector, while also delving into the personal stories of these data pioneers.They reveal how they harness data, not just in business, but in shaping their everyday lives, from optimising daily routines to making data-informed decisions. You'll hear about their motivations, role models, and how they plan to use their position to innovate, inspire and influence change.Many companies today are still missing the golden opportunity to unlock true potential with their data. Our conversations will shed light on this untapped potential as well as the pivotal role that data professionals play in driving progress.“Let's Talk Data” is your go-to source for inspiration and knowledge, providing a front-row seat to the future of data-driven insights and innovations.

AI Data Engineers - Data Engineering after AI // MLOps Podcast #309 with Vikram Chennai, Founder/CEO of Ardent AI.Join the Community: https://go.mlops.community/YTJoinIn Get the newsletter: https://go.mlops.community/YTNewsletter // AbstractA discussion of Agentic approaches to Data Engineering. Exploring the benefits and pitfalls of AI solutions and how to design product-grade AI agents, especially in data.// BioSecond Time Founder. 5 years building Deep learning models. Currently, AI Data Engineers// Related LinksWebsite: tryardent.com~~~~~~~~ ✌️Connect With Us ✌️ ~~~~~~~Catch all episodes, blogs, newsletters, and more: https://go.mlops.community/TYExploreJoin our slack community [https://go.mlops.community/slack]Follow us on X/Twitter [@mlopscommunity](https://x.com/mlopscommunity) or [LinkedIn](https://go.mlops.community/linkedin)] Sign up for the next meetup: [https://go.mlops.community/register]MLOps Swag/Merch: [https://shop.mlops.community/]Connect with Demetrios on LinkedIn: /dpbrinkmConnect with Vikram on LinkedIn: /vikram-chennai/

Comment l'IA générative transforme-t-elle la gestion du risque dans l'assurance ?Aurélien Couloumy, actuaire de formation et CEO de Dylogy, partage son retour d'expérience sur l'automatisation de l'analyse documentaire dans un secteur exigeant et réglementé.Hébergé par Ausha. Visitez ausha.co/politique-de-confidentialite pour plus d'informations.

This is episode 292 recorded on April 22nd, 2025 where John & Jason talk the Microsoft Fabric February 2025 Feature Summary including Fabric Platform, OneLake, and Data Engineering.

Hey humans! Welcome back to part two of our data literacy series! In this episode, I continue my round table discussion with Jennifer Wright, Karen Brieger, and Kelly Oliver as we dive deeper into the practical aspects of HR data. We explore the critical importance of "clean data" - what it really means and why it matters so much. Kelly breaks down the three-phase framework of data engineering, data analysis, and data insights, explaining why you can't just jump straight to insights without proper preparation. Jennifer shares a powerful example about how software conversion challenges can create data issues, while Karen emphasizes the importance of basic data accuracy - from employee addresses to departmental coding. Our conversation also addresses the unique challenges faced by fractional HR leaders who need to quickly adapt to different organizational cultures and data preferences. As Karen explains, some clients want workshops and training while others just want quick data points without the deeper understanding. We discuss how to navigate these different environments while still delivering value. We also touch on the future of HR data and analytics with cautious optimism about AI's potential to help identify anomalies and patterns. There are still so many organizations still mastering the basics of data literacy. Enjoy part two of our data conversation! Stacie More episodes at StacieBaird.com. LinkedIn Pages for each of my guests: Jennifer Wright Karen Brieger Kelly Oliver

In this episode, we look back on invaluable leadership insights shared by renowned executives across the technology industry. Featuring wisdom from Christine Sandman Stone, Arne Saupe, Jorie Sax, Thulasi Kethini, and Tony Newcome, the discussion delves into the distinct skill sets required for leadership, the importance of fostering a culture of innovation, and the critical need for supportive and open communication within teams. These leaders unpack practical advice for setting meaningful goals, building complementary partnerships, and maintaining resilience. We learn about the ways leadership requires courage, decisiveness, and the ability to connect deeply with team members. Their words provide a rich tapestry of experiences and strategies to inspire aspiring and current leaders.(00:00) Introduction to Leadership Insights(02:10) Christine Sandman Stone on Leadership vs. Development(03:18) Arne Saupe on Leadership Culture and Talent(04:54) Jorie Sax on Empowering Creativity and Innovation(06:19) Thulasi Kethini on Fostering Curiosity(08:26) Tony Newcome on Connecting and Supporting Your Team(10:43) Christine Sandman Stone on Setting Meaningful Goals(11:16) Arne Saupe on Vision(11:55) Tony Newcome on Mentorship(14:03) Christine Sandman Stone on Being a First-Time Leader(14:45) Conclusion: Reflecting on Effective LeadershipChristine Sandman Stone is the former Global Head of Product & Engineering Operations & Strategy at Groupon. Arne Saupe is the former CTO at Farmer's Fridge and newly appointed CTO at Wellfound Foods. Jorie Sax heads United Airlines' Innovation Lab. Thulasi Kethini is the Executive Director and Head of Data Engineering at JPMorgan, Tony Newcome is CTO at ActiveCampaign.If you'd like to receive new episodes as they're published, please subscribe to Innovation and the Digital Enterprise in Apple Podcasts, Spotify, or wherever you get your podcasts. If you enjoyed this episode, please consider leaving a review in Apple Podcasts. It really helps others find the show.Podcast episode production by Dante32.

Data engineering is a critical field in data science that involves preparing the "big data" infrastructure to be analyzed by data scientists. In this episode we are discussing the differences and how important each is with our guests.

What if the cost of writing code dropped to zero — but the cost of understanding it skyrocketed? In this episode, Hugo sits down with Joe Reis to unpack how AI tooling is reshaping the software development lifecycle — from experimentation and prototyping to deployment, maintainability, and everything in between. Joe is the co-author of Fundamentals of Data Engineering and a longtime voice on the systems side of modern software. He's also one of the sharpest critics of “vibe coding” — the emerging pattern of writing software by feel, with heavy reliance on LLMs and little regard for structure or quality. We dive into: • Why “vibe coding” is more than a meme — and what it says about how we build today • How AI tools expand the surface area of software creation — for better and worse • What happens to technical debt, testing, and security when generation outpaces understanding • The changing definition of “production” in a world of ephemeral, internal, or just-good-enough tools • How AI is flattening the learning curve — and threatening the talent pipeline • Joe's view on what real craftsmanship means in an age of disposable code This conversation isn't about doom, and it's not about hype. It's about mapping the real, messy terrain of what it means to build software today — and how to do it with care. LINKS * Joe's Practical Data Modeling Newsletter on Substack (https://practicaldatamodeling.substack.com/) * Joe's Practical Data Modeling Server on Discord (https://discord.gg/HhSZVvWDBb) * Vanishing Gradients YouTube Channel (https://www.youtube.com/channel/UC_NafIo-Ku2loOLrzm45ABA) * Upcoming Events on Luma (https://lu.ma/calendar/cal-8ImWFDQ3IEIxNWk)

The GeekNarrator memberships can be joined here: https://www.youtube.com/channel/UC_mGuY4g0mggeUGM6V1osdA/joinMembership will get you access to member only videos, exclusive notes and monthly 1:1 with me. Here you can see all the member only videos: https://www.youtube.com/playlist?list=UUMO_mGuY4g0mggeUGM6V1osdA------------------------------------------------------------------------------------------------------------------------------------------------------------------About this episode: ------------------------------------------------------------------------------------------------------------------------------------------------------------------In this conversation, Jacopo and Ciro discuss their journey in building Bauplan, a platform designed to simplify data management and enhance developer experience. They explore the challenges faced in data bottlenecks, the integration of development and production environments, and the unique approach of Bauplan using serverless functions and Git-like versioning for data. The discussion also touches on scalability, handling large data workloads, and the critical aspects of reproducibility and compliance in data management. Chapters:00:00 Introduction03:00 The Data Bottleneck: Challenges in Data Management06:14 Bridging Development and Production: The Need for Integration09:06 Serverless Functions and Git for Data17:03 Developer Experience: Reducing Complexity in Data Management19:45 The Role of Functions in Data Pipelines: A New Paradigm23:40 Building Robust Data Solutions: Versioning and Parameters30:13 Optimizing Data Processing: Bauplan Runtime46:46 Understanding Control Planes and Data Management48:51 Ensuring Robustness in Data Pipelines52:38 Data Quality and Testing Mechanisms54:43 Branching and Collaboration in Data Development57:09 Scalability and Resource Management in Data Functions01:01:13 Handling Large Data Workloads and Use Cases01:09:05 Reproducibility and Compliance in Data Management01:16:46 Future Directions in Data Engineering and Use CasesLinks and References:Bauplan website:https://www.bauplanlabs.com

Highlights from this week's conversation include:Pete's Career Overview (1:00)AI and Data Engineering Discussion (2:05)Themes of Data Council (4:19)High-Frequency Trading Insights (8:04)San Francisco's Unique Advantages (10:27)Data Council Conference Preview (13:23)The Magic of In-Person Events (15:45)Collapsing Batch and Streaming Systems (19:47)Leveraging Local Hardware for Data Processing (22:07)Future of Blockchain in Computing (23:57)Intersection of AI and Data Management (26:47)Advice for AI Startup Founders (28:44)Blurring Lines Between Data Roles (32:46)The Evolving Role of Engineers (36:56)Discount Code for Data Council and Parting Thoughts (38:23)The Data Stack Show is a weekly podcast powered by RudderStack, the CDP for developers. Each week we'll talk to data engineers, analysts, and data scientists about their experience around building and maintaining data infrastructure, delivering data and data products, and driving better outcomes across their businesses with data.RudderStack helps businesses make the most out of their customer data while ensuring data privacy and security. To learn more about RudderStack visit rudderstack.com.

People often ask me what I'd change in Fundamentals of Data Engineering. Usually, I reply "not much", as the the Data Engineering Lifecycle still remains intact. However, I see the role of data engineers shifting both left and right. What does this mean? Have a listen.

Willis Nana and I chat about the challenges of data engineering leadership, foundational skills, and his journey to a content creator on YouTube.#dataengineering #data #ai #datateam #leadership

✅ Book 1-on-1 Roadmap Call with ME, Related to Data Analytics, Data Science, AI, Generative AI and Data Engineering: https://bepec.in/courses/artificial-intelligence-course-bangalore/

Simon Späti and I discuss various aspects of writing, data engineering, and the impact of AI on the writing process. Simon shares his journey from business intelligence to data engineering and his current focus on writing. We also discuss the future of writing in the age of AI. Enjoy!

In this podcast episode, we talked with Adrian Brudaru about ​the past, present and future of data engineering.About the speaker:Adrian Brudaru studied economics in Romania but soon got bored with how creative the industry was, and chose to go instead for the more factual side. He ended up in Berlin at the age of 25 and started a role as a business analyst. At the age of 30, he had enough of startups and decided to join a corporation, but quickly found out that it did not provide the challenge he wanted.As going back to startups was not a desirable option either, he decided to postpone his decision by taking freelance work and has never looked back since. Five years later, he co-founded a company in the data space to try new things. This company is also looking to release open source tools to help democratize data engineering.0:00 Introduction to DataTalks.Club1:05 Discussing trends in data engineering with Adrian2:03 Adrian's background and journey into data engineering5:04 Growth and updates on Adrian's company, DLT Hub9:05 Challenges and specialization in data engineering today13:00 Opportunities for data engineers entering the field15:00 The "Modern Data Stack" and its evolution17:25 Emerging trends: AI integration and Iceberg technology27:40 DuckDB and the emergence of portable, cost-effective data stacks32:14 The rise and impact of dbt in data engineering34:08 Alternatives to dbt: SQLMesh and others35:25 Workflow orchestration tools: Airflow, Dagster, Prefect, and GitHub Actions37:20 Audience questions: Career focus in data roles and AI engineering overlaps39:00 The role of semantics in data and AI workflows41:11 Focusing on learning concepts over tools when entering the field 45:15 Transitioning from backend to data engineering: challenges and opportunities 47:48 Current state of the data engineering job market in Europe and beyond 49:05 Introduction to Apache Iceberg, Delta, and Hudi file formats 50:40 Suitability of these formats for batch and streaming workloads 52:29 Tools for streaming: Kafka, SQS, and related trends 58:07 Building AI agents and enabling intelligent data applications 59:09Closing discussion on the place of tools like DBT in the ecosystem

What if you could redefine enterprise data management with AI—while boosting productivity by 30%? In this episode of Bringing Data and AI to Life, host Amy Horowitz sits down with Gaurav Pathak, VP of Product Management for AI and Metadata at Informatica, to explore how AI is turning data from a manual headache into an enterprise-wide superpower. As a DataIQ 2025 Data and AI Leader of the Year nominee, Gaurav delivers actionable strategies to help CDOs, CIOs, and data architects scale operations, enhance governance, and harness AI-driven automation. Tune in now to embrace innovation—without sacrificing security or trust.

In this episode of Product by Design, Kyle chats with Sandy Ryza, lead engineer on the Dagster project, author, and thought leader in data engineering. Sandy shares his journey through the world of data—from building big data tools at Cloudera to working as a data scientist, product manager, and engineer—and how those experiences led him to help create Dagster, an open-source data orchestration platform.We discuss:The evolution of data engineering and the growing complexity of modern data pipelines.The role of AI and unstructured data in shaping the future of data platforms.How organizations should think about data platforms to avoid costly rework.Best practices for managing data complexity using software engineering principles.The future of open-source tools in data infrastructure and the push toward interoperability.Sandy RyzaSandy is a lead engineer, author, and thought leader in the domain of data engineering. Sandy co-wrote “Advanced Analytics with PySpark” and “Advanced Analytics with Spark”. He led ML and data science teams at Cloudera, Remix, Clover Health, and KeepTruckin.Sandy is currently the lead engineer on the Dagster project, an open-source data orchestration platform used in MLOps, data science, IOT and analytics. Sandy is a regular speaker at data engineering and ML conferences.Links from the Show:Twitter: @s_RYZDagster: dagster.ioBook: Advanced Analytics with Spark – O'ReillyPodcast Recommendation: Empire (British Empire & Ottoman Empire history)Books Sandy is Reading: The Shortest History of India, The Sun Also Rises, Werner Herzog's AutobiographyMore by Kyle:Follow Prodity on Twitter and TikTokFollow Kyle on Twitter and TikTokSign up for the Prodity Newsletter for more updates.Kyle's writing on MediumProdity on MediumLike our podcast, consider Buying Us a Coffee or supporting us on Patreon

S1 Ep#30: From Engineering to Data Strategy: Driving AI and Decision-Making The Data Product Management In Action podcast, season 1, is brought to you by Soda and executive producer Scott Hirleman, is a platform for data product management practitioners to share insights and experiences. Our guest this week is Theo Bell, a data product manager. She chats with host Nick Zerviudis and shares her transition from mechanical engineering to roles at Goldman Sachs and Palantir, emphasizing the importance of data integration in strategic decision-making. She discusses how Palantir helped a manufacturer prioritize client orders during raw material shortages and explores the challenges of convincing stakeholders to adopt new data models, advocating for production-ready pilots over proof-of-concepts. Theo also offers insights on fostering AI adoption within organizations, using a news summarization tool for a CEO as an example. She recommends the GTD framework and Surrounded by Idiots for enhancing productivity and communication. About our Host Nick Zervoudis: Nick is Head of Product at CKDelta, an AI software business within the CK Hutchison Holdings group. Nick oversees a portfolio of data products and works with sister companies to uncover new opportunities to innovate using data, analytics, and machine learning. Nick's career has revolved around data and advanced analytics from day one, having worked as an analyst, consultant, product manager, and instructor for startups, SMEs, and enterprises including PepsiCo, Sainsbury's, Lloyds Banking Group, IKEA, Capgemini Invent, BrainStation, QuantSpark, and Hg Capital. Nick is also the co-host of London's Data Product Management meetup, and speaks & writes regularly about data & AI product management. Connect with Nick on LinkedIn and through his newsletter, Value from Data & AI. About our Guest Theo Bell: Theo is the Head of AI Product at Rimes, where she leads the company's efforts to leverage AI technology in order to provide cutting-edge data management solutions to clients. Previously, Theo held key roles at Palantir Technologies and Goldman Sachs, where she enabled various industries to leverage data through AI/ML-driven software, notably Airbus' Skywise platform, the NHS, and the UK Ministry of Defense. Theo is dedicated to using AI and technology for global challenges, particularly in improving health, enhancing society, and fostering sustainable businesses. She holds a PhD in Engineering from the University of Cambridge.  Connect with Theo in LinkedIn. All views and opinions expressed are those of the individuals and do not necessarily reflect their employers or anyone else.  Join the conversation on LinkedIn.  Apply to be a guest or nominate someone that you know.  Do you love what you're listening to? Please rate and review the podcast, and share it with fellow practitioners you know. Your support helps us reach more listeners and continue providing valuable insights!    .

Season 1 Episode 29: Navigating Trade-Offs and Balancing Priorities The Data Product Management In Action podcast, brought to you by executive producer Scott Hirleman, is a platform for data product management practitioners to share insights and experiences. In this episode of Data Product Management in Action, host Alexa Westlake talks with Anita Chen, diving into the complexities of managing data products. Anita, a product manager at PagerDuty, shares her approach to defining data products, prioritizing work, and balancing project work with interrupt-driven tasks. They discuss the critical roles of governance, security, and user enablement while emphasizing the importance of transparency and communication. The conversation also explores the transformative potential of generative AI in data product interactions and the build-vs-buy decision-making process. Gain insights into how data product management uniquely differs from traditional software product management and learn actionable strategies for success. Meet our Host Alexa Westlake: Alexa is a Data Analytics Leader in the Identity and Access Management space with a proven track record scaling high-growth SaaS companies. As a Staff Data Analyst at Okta, she brings a wealth of expertise in enterprise data, business intelligence, and strategic decision-making from the various industries she's worked in including telecommunications, strategy execution, and cloud computing. With a passion for harnessing the power of data for actionable insights, Alexa plays a crucial role in driving Okta's security, growth, and scale, helping the organization leverage data to execute on their market opportunity. Connect with Alexa on LinkedIn. Meet our guest Anita Chen:  Anita is a Data Product Manager at PagerDuty, a digital operations company helping teams resolve issues faster, eliminate alert fatigue, and build more reliable services! Her background is mainly in the People Analytics space which has now expanded to data at scale with our Enterprise Data Team. She currently helps build data products that enable our teams to deliver the best possible customer experience. Anita is most passionate about how data can impact someone's lived experience and endeavor to democratize data in everything she builds. Connect with Anita on LinkedIn. All views and opinions expressed are those of the individuals and do not necessarily reflect their employers or anyone else.  Join the conversation on LinkedIn.  Apply to be a guest or nominate someone that you know.  Do you love what you're listening to? Please rate and review the podcast, and share it with fellow practitioners you know. Your support helps us reach more listeners and continue providing valuable insights! 

As AI takes over the world, data is more than ever “the new oil”, and data engineering is the discipline that makes data usable behind the scenes. In this episode, we dive deep into the present and future of data engineering with Ben Rogojan, also known as the Seattle Data Guy. A seasoned data engineering consultant, Ben has built a big brand and reputation in the field with over 100k followers on platforms like YouTube and Substack. We started the conversation with a deep dive into data engineering as a profession: what do data engineers actually do? What is the career path, and what should aspiring data engineers learn? We then explored some of the biggest stories of 2024 (including the rise of Iceberg) and went into some predictions for 2025, as a way to discuss some key topics everyone should be familiar with in data engineering, including the integration of AI in data workflows, the potential for automation, and why SQL isn't going anywhere. Discover how companies are navigating the complexities of data infrastructure, the rise of open table formats like Iceberg, and the ongoing battle between data giants like Snowflake and Databricks. Ben Rogojan Website - https://www.theseattledataguy.com Newsletter - https://seattledataguy.substack.com LinkedIn - https://www.linkedin.com/company/seattle-data-guy X/Twitter - https://x.com/seattledataguy FIRSTMARK Website - https://firstmark.com X/Twitter - https://twitter.com/FirstMarkCap Matt Turck (Managing Director) LinkedIn - https://www.linkedin.com/in/turck/ X/Twitter - https://twitter.com/mattturck (00:00) Intro (01:20) Why 2025 will be huge for data engineering (02:55) The story of the Seattle Data Guy (06:51) What exactly is data engineering? (07:41) Data, AI, and ML: where do they overlap? (09:23) Data analyst vs. data engineer vs. data scientist: what's the difference? (11:20) A day in the life of a data engineer (12:58) Data engineering: Silicon Valley vs. everywhere else (15:27) How to become an AI engineer (28:46) Will AI replace AI engineers? (33:42) Why is the data world so complex? (36:53) The functional consolidation of the data world (38:34) Big data stories from 2024 (39:28) Why Iceberg is a game-changer (46:02) How startups manage data in their early days (48:44) Seattle Data Guy's favorite tools (50:09) Bold predictions for 2025

In this season premiere of The Data Chief podcast, your host Cindi Howson sits down with three industry visionaries to explore the trends, predictions, and must-take actions for data leaders in 2025. Get ready for a deep dive into: The generative AI revolution with Matt Turck, Partner at FirstMark CapitalThe future of data science and genAI with Steve Nouri, Founder of GenAI Works and AI for DiversityData Engineering in the Age of AI with Joe Reis, author of "Fundamentals of Data Engineering" and the upcoming "Mixed Model Arts."Plus: Hear their fun predictions for everything from sports to space travel!Key Moments:The generative AI revolution: Matt Turck, Partner at FirstMark Capital shares his insights on the evolving AI landscape, the rise of unstructured data, and why now is the time for enterprises to embrace AI. (1:40) The Future of Data Science: Steve Nouri, Founder of GenAI Works (an 8-million-strong community!) and AI for Diversity, discusses the impact of GenAI on data science roles, the ethical considerations of AI, and exciting trends like embodied AI and agentic AI. (29:36) Data Engineering in the Age of AI: Joe Reis, author of "Fundamentals of Data Engineering" and the upcoming "Mixed Model Arts," provides his expert perspective on the importance of data modeling, the need for upskilling in data teams, and the potential for a universal semantic layer. (1:00:00) Key Quotes:“I would predict that there's going to be a number of big acquisitions in our general space in 2025. This whole tension between the public markets doing very well, especially in tech, but the private markets still recovering - I think lends itself well to a wave of consolidation.”  - Matt Turck“Anything that requires democratization, I'm a big fan of. And certainly, the ability to query natural language databases and all things, making that available to everyone is a very powerful idea. You guys at ThoughtSpot know this better than anyone.” - Matt Turck“We are seeing people doing less coding, more relying on their co-pilots. It's going to evolve to become more and more robust. So we will be relying more on AI to do the coding.” - Steve Nouri“Well, that's what, you know, the tagline is, AI will do everything for you. It'll even do your laundry, the jobs that we don't like. And so you're actually saying you see a future where that actually is not too far off.” - Steve Nouri“I think that there's definitely a FOMO and a bit of a prisoner's dilemma problem with adopting AI in the organization because they're getting a lot of pressure from the top down, especially to do AI. Understanding what that means to your organization should be table stakes.” - Joe Reis“Learning never stops, investment never stops. And the best investment you can make is always improving yourself, no matter what that looks like.” Joe ReisMentions:FirstMark MAD Landscape 2024The MAD Podcast with Matt TurckAI4DiversityGenAI.WorksFundamentals of Data EngineeringJoe Reis Substack Guest Bios:Matt Turck is a Partner at FirstMark, where he focuses primarily on early-stage enterprise investing in the US and Europe. Matt is particularly active in the data, machine learning and AI space. For the last 10+ years, he has been organizing Data Driven NYC, the largest data/AI community in the US, and publishing the MAD Landscape, an annual analysis of the data/AI industry. He also hosts the weekly MAD (ML, AI, Data) Podcast. He can be followed on X/Twitter at @mattturck.Steve Nouri is the CEO and Co-founder of GenAI Works, the largest AI community. He is a renowned AI leader and Australia's ICT Professional of the Year, has revolutionized AI perspectives while championing Responsible and inclusive AI, founding a global non-profit initiative.Joe Reis, a "recovering data scientist" with 20 years in the data industry, is the co-author of the best-selling O'Reilly book, "Fundamentals of Data Engineering." He's also the instructor for the wildly popular Data Engineering Professional Certificate on Coursera, in partnership with DeepLearning.ai and AWS.Joe's extensive experience encompasses data engineering, data architecture, machine learning, and more. He regularly keynotes major data conferences globally, advises and invests in innovative data product companies, writes at Practical Data Modeling and his personal blog, and hosts the popular data podcasts "The Monday Morning Data Chat" and "The Joe Reis Show." In his free time, Joe is dedicated to writing new books and articles, and thinking of ways to advance the data industry. Hear more from Cindi Howson here. Sponsored by ThoughtSpot.

The Future of AI and Data in 2025:  There's a lot of excitement and prototyping around AI. "People are excited that they can ask questions and get real answers instead of getting the proverbial 10 blue links. I think people see the promise of things like that. How do you make that work at scale? How do you make that work in a reliable way, and how do you make it work in a way that's not expensive?" Ramaswamy shares a recent experience in India, describing the cost-consciousness of customers and the focus on making data accessible and valuable.Shifting Priorities for Snowflake Customers:  What do customers want? Ramaswamy identifies goals including empowering business users with data faster and unlocking the value of unstructured information. In response, Snowflake is helping customers access and utilize historical information and build hybrid systems. He notes that SQL enables easier data transformations and extractions.Cultural Change and AI Adoption: Cultural change is required for companies to handle AI. Even within Snowflake, adopting AI has been a challenging process. Snowflake's engineering and sales teams are beginning to use AI tools like Copilot and Cortex, but there's a need for ongoing practice to fully integrate AI into daily operations. The Promise of AI and Data Platforms: AI has the potential to free up significant time for employees, allowing them to focus on higher-value tasks. Snowflake aspires to make data platforms simple and accessible, allowing users to perform complex tasks with ease. Its consumption model has several benefits and aligns it with business goals of making more money or spending less. Companies like Siemens are using chatbots to improve efficiency and access to information.Competition and Innovation in the Data Market: Competition, Ramaswamy says, drives innovation and improves products. It only strengthens Snowflake, which stands apart due to its unified platform, tight integration, and ease of use. "If you create a chatbot with Snowflake, it will automatically obey all of the access control rules that you have set up. It will automatically obey all of the data masking that your administrator has set up." Ramaswamy discusses the challenges of balancing innovation with maintaining simplicity and efficiency in Snowflake's product offerings.Mega Trends and Challenges for Snowflake: Ramaswamy sees two mega trends for 2025: the rise of interoperable data formats and the acceleration of AI. Snowflake is well-positioned to take advantage of these trends through its end-to-end offerings and data engineering capabilities. However, there are challenges to maintaining data quality and speed while innovating rapidly. AI tools like agents must drive real utility and value for customers.Data Literacy Initiative: One Million Minds Plus One: Snowflake has a new initiative aimed at improving data literacy. Data can be difficult to interpret, and the right skills are necessary to do so. Snowflake is spearheading an ambitious program to educate over a million people in data literacy, free of charge. The initiative includes teaching concepts like data engineering, data analytics, and using BI tools and notebooks for interactive analysis.Excitement for 2025: Sridhar expresses excitement about the potential for AI and data to drive real business transformations in 2025. Snowflake will continue to play a vital role in helping customers become truly data-driven organizations. Navigating growth, competition, and innovation while maintaining simplicity and value for customers is key.

A look inside at the data work happening at a company making some of the most advanced technologies in the industry. Rahul Jain, data engineering manager at Snowflake, joins Tristan to discuss Iceberg, streaming, and all things Snowflake.  For full show notes and to read 6+ years of back issues of the podcast's companion newsletter, head to https://roundup.getdbt.com. The Analytics Engineering Podcast is sponsored by dbt Labs.

In this episode of The Data Engineering Show, host Benjamin and co-host Eldad are joined by Chad Sanderson, CEO and co-founder of Gable AI to discuss the revolution of data quality and governance, the importance of understanding data flow and the processes that help organizations manage their data more effectively.

It's 2025! We made it! ;) In this podcast, I rant about why data modeling matters more than ever, AI, and why humans will seek out "human" things in 2025 and beyond. ❤️ Your support means a lot. Please like and rate this podcast on your favorite podcast platform.

Hugo Bowne-Anderson hosts a panel discussion from the MLOps World and Generative AI Summit in Austin, exploring the long-term growth of AI by distinguishing real problem-solving from trend-based solutions. If you're navigating the evolving landscape of generative AI, productionizing models, or questioning the hype, this episode dives into the tough questions shaping the field. The panel features: - Ben Taylor (Jepson) (https://www.linkedin.com/in/jepsontaylor/) – CEO and Founder at VEOX Inc., with experience in AI exploration, genetic programming, and deep learning. - Joe Reis (https://www.linkedin.com/in/josephreis/) – Co-founder of Ternary Data and author of Fundamentals of Data Engineering. - Juan Sequeda (https://www.linkedin.com/in/juansequeda/) – Principal Scientist and Head of AI Lab at Data.World, known for his expertise in knowledge graphs and the semantic web. The discussion unpacks essential topics such as: - The shift from prompt engineering to goal engineering—letting AI iterate toward well-defined objectives. - Whether generative AI is having an electricity moment or more of a blockchain trajectory. - The combinatorial power of AI to explore new solutions, drawing parallels to AlphaZero redefining strategy games. - The POC-to-production gap and why AI projects stall. - Failure modes, hallucinations, and governance risks—and how to mitigate them. - The disconnect between executive optimism and employee workload. Hugo also mentions his upcoming workshop on escaping Proof-of-Concept Purgatory, which has evolved into a Maven course "Building LLM Applications for Data Scientists and Software Engineers" launching in January (https://maven.com/hugo-stefan/building-llm-apps-ds-and-swe-from-first-principles?utm_campaign=8123d0&utm_medium=partner&utm_source=instructor). Vanishing Gradient listeners can get 25% off the course (use the code VG25), with $1,000 in Modal compute credits included. A huge thanks to Dave Scharbach and the Toronto Machine Learning Society for organizing the conference and to the audience for their thoughtful questions. As we head into the new year, this conversation offers a reality check amidst the growing AI agent hype. LINKS Hugo on twitter (https://x.com/hugobowne) Hugo on LinkedIn (https://www.linkedin.com/in/hugo-bowne-anderson-045939a5/) Vanishing Gradients on twitter (https://x.com/vanishingdata) "Building LLM Applications for Data Scientists and Software Engineers" course (https://maven.com/hugo-stefan/building-llm-apps-ds-and-swe-from-first-principles?utm_campaign=8123d0&utm_medium=partner&utm_source=instructor).

We are back with another installment of our Exploring Tech Job Series where we dig into different technical roles to provide insight into what that role is all about. This week we are chatting about Data Engineering and we are joined by a real-life Data Engineer, Rosie! Join us as we chat about what a Data Engineer is, the day-to-day of the job, how you might get into Data Engineering, and so much more. New episodes come out fortnightly on Wednesday morning (NZT). Check out Datacamp to learn more about data engineering Where to Find Us: ⁠ ⁠Instagram⁠ ⁠Tik Tok⁠ ⁠The Hot Girls Code Website⁠⁠ Sponsored by: ⁠Trade Me Jobs⁠

Former Moser intern turned Moser Consultant Alex Payne joins us this week to discuss the updates included in the latest release of Snowflake, a cloud-based data warehousing service that allows organizations to store, manage, and analyze large amounts of data.Alex  works in Data Engineering, specializing in Data Migrations, Power Apps, Streamlit Development, and Snowflake Data Management. He is also a leader of the AI-Tig, where he presents monthly on cutting-edge AI technology. Outside of work, he enjoys playing and watching sports, as well as working out.

Wouter Trappers is the founder of Xudo and shares his slightly unconventional path from philosopher to data consultant with the Bros in this latest episode of The Data Engineering Show. Wouter's grounding in philosophy has proved to be a shaping influence on his approach to business intelligence. Much more than just a software solution, for Wouter, BI is all about change management and aligning leadership with data projects.

Discover how Alex Noonan transitioned from the flight deck of a Marine aircraft to the intricate world of data engineering. His unique journey, enriched by a stint in finance, gives us a firsthand view of the diverse backgrounds shaping the data industry. As Alex recounts his experiences, we explore the vibrant community he found on data Twitter, a realm buzzing with shared insights and collaborative spirit. However, the landscape shifted following Elon Musk's takeover of Twitter, leading to content fragmentation and a migration towards emerging platforms like Blue Sky. Join us as Alex discusses how these changes have impacted the cohesion and knowledge-sharing dynamics within the data community.Navigate the complex world of professional networking with tips from Alex, as he breaks down the strategic use of platforms like LinkedIn, Reddit, and Hacker News for data professionals. Learn how to creatively tailor your content to fit the quirks of each platform's algorithm, and prepare to engage with varied audiences. The conversation also highlights the transformative potential of AI tools in elevating data processes, reducing mundane tasks, and fostering high-value work. Discover innovations like Dagster and its role as an orchestrator, integrating key business intelligence tools to streamline the data engineer's experience. This episode is a must-listen for anyone intrigued by the evolving interplay of technology, social media, and the power of community.Follow Alex on:Linkedin Twitter BlueskyDagsterWhat's New In Data is a data thought leadership series hosted by John Kutay who leads data and products at Striim. What's New In Data hosts industry practitioners to discuss latest trends, common patterns for real world data patterns, and analytics success stories.

Data drives the world we live in, but have you ever wondered how data flows around the tech world? Or what the difference is between a data engineer, a data scientist, and a data analyst? Well this is the episode for you! Come learn all about the basics of data engineering and key concepts like data pipelines, data storage, and the infamous ETL process. And as you'll hear in the episode, closets and Cluedo are essential to explain all of these topics! New episodes come out fortnightly on Wednesday morning (NZT). Where to Find Us: ⁠ Instagram Tik Tok The Hot Girls Code Website⁠ Sponsored by: Trade Me Jobs

This episode of Tartlecast emphasizes the critical importance of remembering your TARTLE pin. We explain why your pin is essential for protecting your data and accessing your earnings, and we offer tips for creating a secure and memorable pin. We also address the common issue of forgotten pins and why TARTLE cannot retrieve or reset them for users. Key Takeaways: Your TARTLE pin is the key to accessing your account and protecting your data.     TARTLE cannot access or reset your pin due to security and privacy protocols.     Choose a pin that is both secure and memorable to avoid losing access to your account.      If you forget your pin, there is currently no way to recover it. You will need to create a new account.     TCAST is a tech and data podcast co-hosted by Alexander McCaig and Jason Rigby. The podcast delves into the latest trends in Big Data, Artificial Intelligence, and Humanity, examining the evolving landscape of digital transformation and innovation.   The show features interviews with data scientists, thought leaders, and industry experts at the forefront of technological advancement for human progress. Listeners can explore a wide range of TCAST episodes on their preferred podcast platforms.   Connect with TCAST: Website: https://tartle.co/ Facebook: https://go.tartle.co/fb Instagram: https://go.tartle.co/ig Twitter: https://go.tartle.co/tweet   What's your data worth? Find out at https://tartle.co/  

We've noticed some confusion about how earnings are displayed on TARTLE.  Let's clear things up!  Our latest Tartlecast episode explains the correct way to read your wallet balance and why some data elements have a value of less than a penny. TCAST is a tech and data podcast co-hosted by Alexander McCaig and Jason Rigby. The podcast delves into the latest trends in Big Data, Artificial Intelligence, and Humanity, examining the evolving landscape of digital transformation and innovation.   The show features interviews with data scientists, thought leaders, and industry experts at the forefront of technological advancement for human progress. Listeners can explore a wide range of TCAST episodes on their preferred podcast platforms.   Connect with TCAST: Website: https://tartle.co/ Facebook: https://go.tartle.co/fb Instagram: https://go.tartle.co/ig Twitter: https://go.tartle.co/tweet   What's your data worth? Find out at https://tartle.co/

This Tartlecast episode clarifies the difference between the small rewards given for completing actions on TARTLE (like creating a wallet or publishing a data packet) and the larger earnings that come from bids placed on your data packets. We explain why TARTLE offers these small rewards and how they contribute to the overall goal of creating a universal basic income. TCAST is a tech and data podcast co-hosted by Alexander McCaig and Jason Rigby. The podcast delves into the latest trends in Big Data, Artificial Intelligence, and Humanity, examining the evolving landscape of digital transformation and innovation.   The show features interviews with data scientists, thought leaders, and industry experts at the forefront of technological advancement for human progress. Listeners can explore a wide range of TCAST episodes on their preferred podcast platforms.   Connect with TCAST: Website: https://tartle.co/ Facebook: https://go.tartle.co/fb Instagram: https://go.tartle.co/ig Twitter: https://go.tartle.co/tweet   What's your data worth? Find out at https://tartle.co/

In this episode of Tartlecast, we address a common question among TARTLE users: "What happens to my data packet after I publish it?" We clarify the process, explaining that published data packets are securely stored in the user's data vault and are not sold, moved, or transferred until a bid is placed on them. We also discuss the importance of completing data packets and building a robust data vault to attract potential buyers.     Key Takeaways: Published data packets are NOT immediately sold. They are stored in your secure data vault.     TARTLE does not have access to your data vault. Only YOU have the keys.     Once a bid is placed on your data packet, you will be notified and can choose to accept or decline the offer.     Completing more data packets increases your visibility to buyers and your earning potential.     TCAST is a tech and data podcast co-hosted by Alexander McCaig and Jason Rigby. The podcast delves into the latest trends in Big Data, Artificial Intelligence, and Humanity, examining the evolving landscape of digital transformation and innovation.   The show features interviews with data scientists, thought leaders, and industry experts at the forefront of technological advancement for human progress. Listeners can explore a wide range of TCAST episodes on their preferred podcast platforms.   Connect with TCAST: Website: https://tartle.co/ Facebook: https://go.tartle.co/fb Instagram: https://go.tartle.co/ig Twitter: https://go.tartle.co/tweet   What's your data worth? Find out at https://tartle.co/

People often ask me for career advice. In a tough job market where people are sending out thousands of resumes and hearing nothing back, I notice a lot of people have weak networks and are unknown to the companies they're applying to. This results in lots of frustration and disappointment for job seekers. Is there a better way? Yes. People need to know who you are. Obscurity is your enemy. Also, the name of the Friday show changed because I can't seem to keep things to five minutes ;) My works:

We've heard your questions, and we're here to help!  Our latest Tartlecast episode tackles the common question of why some data packets don't receive bids.  Learn how to make your data more appealing to buyers and increase your earning potential on TARTLE. TCAST is a tech and data podcast co-hosted by Alexander McCaig and Jason Rigby. The podcast delves into the latest trends in Big Data, Artificial Intelligence, and Humanity, examining the evolving landscape of digital transformation and innovation.   The show features interviews with data scientists, thought leaders, and industry experts at the forefront of technological advancement for human progress. Listeners can explore a wide range of TCAST episodes on their preferred podcast platforms.   Connect with TCAST: Website: https://tartle.co/ Facebook: https://go.tartle.co/fb Instagram: https://go.tartle.co/ig Twitter: https://go.tartle.co/tweet   What's your data worth? Find out at https://tartle.co/

TARTLE's executives discuss how their Real Intelligence API serves different market segments in AI development, from startups to government agencies. The episode reveals how the API provides 50% of the total AI/ML lifecycle needs through a single integration point, offering ethical data sourcing and real-time human feedback across diverse global demographics. Key Segments Covered: Enterprise AI Departments: • S3 bucket integration capabilities • Streamlined legal compliance • Scalable data acquisition solutions • 20-30 minute integration timeline AI Startups: • Initial 1,000-5,000 data sample requirements • Cost-effective scaling options • Market differentiation through unique datasets • Real-time feedback for rapid development Government Initiatives: • Fourth, Fifth, and Sixth Amendment compliance • Auditable data trails • Representative population sampling • Direct public engagement capabilities Integration Benefits: • Flexible volume scaling • Documentation support • Quick implementation • Custom data collection options TCAST is a tech and data podcast co-hosted by Alexander McCaig and Jason Rigby. The podcast delves into the latest trends in Big Data, Artificial Intelligence, and Humanity, examining the evolving landscape of digital transformation and innovation.   The show features interviews with data scientists, thought leaders, and industry experts at the forefront of technological advancement for human progress. Listeners can explore a wide range of TCAST episodes on their preferred podcast platforms.   Connect with TCAST: Website: https://tartle.co/ Facebook: https://go.tartle.co/fb Instagram: https://go.tartle.co/ig Twitter: https://go.tartle.co/tweet   What's your data worth? Find out at https://tartle.co/

In this episode of Tartlecast, we address a common question from our global community: "Why can't I withdraw my earnings to PayPal in my country?" We discuss the challenges of international payments and the steps TARTLE is taking to provide more withdrawal options for our members worldwide. We also emphasize the importance of continuing to build your data vault on TARTLE, even while we work on expanding payment solutions. Key Takeaways: TARTLE is aware that PayPal is not available in all countries.   We are actively working to add more withdrawal options, including bank transfers, digital payment methods, and gift cards.     Your earnings are safe and secure on TARTLE, even if you cannot withdraw them immediately.     We encourage you to continue completing data packets and building your data vault, as this helps us attract more buyers for your data.     TCAST is a tech and data podcast co-hosted by Alexander McCaig and Jason Rigby. The podcast delves into the latest trends in Big Data, Artificial Intelligence, and Humanity, examining the evolving landscape of digital transformation and innovation.   The show features interviews with data scientists, thought leaders, and industry experts at the forefront of technological advancement for human progress. Listeners can explore a wide range of TCAST episodes on their preferred podcast platforms.   Connect with TCAST: Website: https://tartle.co/ Facebook: https://go.tartle.co/fb Instagram: https://go.tartle.co/ig Twitter: https://go.tartle.co/tweet   What's your data worth? Find out at https://tartle.co/

TARTLE's CEO Alexander McCaig and Chief Conscious Marketing Officer Jason Rigby discuss the limitations of pure IoT/sensor data collection (like Tesla's robot training suits) and why Real-Time Human Feedback (RTHF) is crucial for authentic AI development. The episode reveals how TARTLE's Real Intelligence API provides ethically-sourced, direct human data that includes critical qualitative insights missing from synthetic datasets. Key Points Covered: Why Tesla's robot training suits only capture 30-40% of necessary human behavior data How authenticity tracking through blockchain ensures data provenance Cost comparison: $85-120K for synthetic data vs. significant savings with direct human feedback Case study of sourcing millions of data points in India and rural Nigeria Why model training needs human intent data, not just mechanical movements How TARTLE's API enables automated trust-building and feedback loops Compliance advantages: Getting ahead of GDPR, CCPA through zero-party data Real examples of demographic targeting for global brands (energy drink case study) TCAST: A Tech and Data Podcast Hosted by: Alexander McCaig and Jason Rigby About: TCAST is a tech and data podcast exploring the most exciting trends in Big Data, Artificial Intelligence, and Humanity. Join Alexander and Jason as they fearlessly examine the latest developments in digital transformation and innovation. Each episode features insightful interviews with data scientists, thought leaders, and industry pioneers who are shaping the skills and technologies we need for human progress. Explore TCAST: Tune in on your favorite podcast platform and explore our extensive selection of episodes at your own pace. Connect with TCAST: Website: https://tartle.co/ Facebook: https://go.tartle.co/fb Instagram: https://go.tartle.co/ig Twitter: https://go.tartle.co/tweet What's your data worth? Find out at https://tartle.co/

TARTLE's Alexander McCaig and Jason Rigby discuss why AI companies are struggling with model accuracy and how real-time human feedback (RTHF) provides the solution to synthetic data limitations. Key insights include why AI models collapse at 50% synthetic data usage, the importance of demographic diversity in training sets, and how TARTLE's zero-party data approach ensures both compliance and quality across global markets. The episode explores: Why public and synthetic data lead to increasing error rates The competitive advantage of real human behavioral data How TARTLE's API enables 45-minute integration for accelerated AI development Global compliance across GDPR, CCPA, and HIPAA through zero-party data The methodology for reducing algorithmic bias through diverse demographic sampling TCAST: A Tech and Data Podcast Hosted by: Alexander McCaig and Jason Rigby About: TCAST is a tech and data podcast exploring the most exciting trends in Big Data, Artificial Intelligence, and Humanity. Join Alexander and Jason as they fearlessly examine the latest developments in digital transformation and innovation. Each episode features insightful interviews with data scientists, thought leaders, and industry pioneers who are shaping the skills and technologies we need for human progress. Explore TCAST: Tune in on your favorite podcast platform and explore our extensive selection of episodes at your own pace. Connect with TCAST: Website: https://tartle.co/ Facebook: https://go.tartle.co/fb Instagram: https://go.tartle.co/ig Twitter: https://go.tartle.co/tweet What's your data worth? Find out at https://tartle.co/

In a world driven by data, it takes a visionary to see beyond the noise and build something truly scalable. Viraj Parekh, co-founder of Astronomer, didn't have a clear roadmap. His journey from childhood curiosity in technology to becoming a key figure in data orchestration is fascinating. Astronomer has attracted funding from top-tier investors like Venrock, Bain Capital Ventures, Insight Partners, and Sutter Hill Ventures.

Let's do things the right way, not just the fast way. My works:

Stephen Bailey, a seasoned data leader and experienced data engineer, joins us to discuss his journey in data and tech. From his academic background in neuroscience to his current role at Whatnot, Bailey has consistently pushed the boundaries of data-driven innovation. We delve into his insights on the challenges and opportunities in the modern data landscape, including details into the problem solving skills developed in his PhD program, how he got his start in technology, a fascination with low tech problems in non-profits and the impact of AI on data engineering roles. Bailey shares his thoughts on how to navigate the complexities of AI in the workforce and how to remain relevant in the ever-evolving data space. Throughout the conversation, Bailey emphasizes the significance of human-centered approaches to data, emphasizing the need for strong collaboration between data teams and business stakeholders. He even discusses why the CEO is like “the best analyst in the business” which is an enlightening part of the discussion, and a refreshing perspective. As a prolific writer, Bailey shares his motivations for blogging and how he uses it to explore the intersection of data, technology, and society. His Substack, "Data, People, Etc.," offers insightful commentary on a range of topics, from data ethics to the future of work. Ultimately, this episode provides valuable insights for data professionals seeking to navigate the complexities of modern data landscapes, embrace emerging technologies, and build strong, data-driven organizations. Find Stephen on his Substack: https://stkbailey.substack.com/

I speak at a lot of conferences, and I've lost track of how many questions I've answered. Since conferences are top of mind for me right now, here are some tips for asking good (and bad) questions of speakers. My works:

In this episode of Crazy Wisdom, Stewart Alsop chats with Ian Mason, who works on architecture and delivery of AI and ML solutions, including LLMs and retrieval-augmented generation (RAG). They explore topics like the evolution of knowledge graphs, how AI models like BERT and newer foundational models function, and the challenges of integrating deterministic systems with language models. Ian explains his process of creating solutions for clients, particularly using RAG and LLMs to support automated tasks, and discusses the future potential of AI, contrasting the hype with practical use cases. You can find more about Ian on his LinkedIn profile.Check out this GPT we trained on the conversation!Timestamps00:00 Introduction and Guest Welcome00:32 Understanding Knowledge Graphs02:03 Hybrid Systems and AI Models03:39 Philosophical Insights on AI05:01 RAG and Knowledge Graph Integration07:11 Challenges in AI and Knowledge Graphs11:40 Multimodal AI and Future Prospects13:44 Artificial Intelligence vs. Artificial Linear Algebra17:50 Silicon Valley and AI Hype30:44 Defining AGI and Embodied Intelligence32:29 Potential Risks and Mistakes of AI Agents35:04 The Role of Human Oversight in AI38:00 Understanding Vector Databases43:28 Building Solutions with Modern Tools46:52 The Future of Solution Development47:43 Personal Journey into Coding57:25 The Importance of Practical Learning59:44 Conclusion and Contact InformationKey InsightsThe evolution of AI models: Ian Mason discusses how foundational models like BERT have been overtaken by newer, more capable language models, which can perform tasks that once required multiple models. He highlights that while earlier models like BERT still have their uses, foundational models have simplified and expanded AI's capabilities.The role of knowledge graphs: Knowledge graphs provide structured, deterministic ways of handling data, which can complement language models. Ian explains that while LLMs are great for articulating responses based on large datasets, they lack the ability to handle logical and architectural connections between pieces of information, which knowledge graphs can provide.RAG (Retrieval-Augmented Generation) systems: Ian delves into how RAG systems help refine AI output by feeding language models relevant data from a pre-searched database, reducing hallucinations. By narrowing down the possible answers and focusing the LLM on high-quality data, RAG ensures more accurate and contextually appropriate responses.Limitations of language models: While LLMs can generate plausible-sounding responses, they lack deep architectural understanding and can easily hallucinate or provide inaccurate results without carefully curated input. Ian points out the importance of combining LLMs with structured data systems like knowledge graphs or vector databases to ground the output.Vector databases and embeddings: Ian explains how vector databases, which use embeddings and cosine similarity, are crucial for narrowing down the most relevant data in a RAG system. This modern approach outperforms traditional keyword searches by considering semantic meaning rather than just text similarity.AI's impact on business solutions: The conversation highlights how AI, particularly through tools like RAG and LLMs, can streamline business processes. For instance, Ian uses AI to automate customer service email drafting, breaking down complex customer queries and retrieving the most relevant answers, significantly improving operational efficiency.The future of AI in business: Ian believes AI's real-world impact will come from its integration into larger systems rather than revolutionary standalone changes. While there is significant hype around AGI and other speculative technologies, the focus for the near future should be on practical applications like automating business workflows, where AI can create measurable value without over-promising its capabilities.