Podcasts about data engineering

  • 365PODCASTS
  • 1,019EPISODES
  • 38mAVG DURATION
  • 5WEEKLY NEW EPISODES
  • Jun 20, 2025LATEST

POPULARITY

20172018201920202021202220232024


Best podcasts about data engineering

Show all podcasts related to data engineering

Latest podcast episodes about data engineering

The Evolution Exchange Podcast Nordics
Evo Nordics #617 - Evolution Of Data And The Impact Of AI

The Evolution Exchange Podcast Nordics

Play Episode Listen Later Jun 20, 2025 41:30


Host Luke Vickers sits down with Eevi-Inkeri Forsman (Tech Lead, Data Engineering, SOK), Erno Savela (Data & Analytics Lead, Operations, Posti), Jacopo Himberg (Director of Data, Wolt), and Peter Bamrud (Consultant Manager, Data, AI & Analytics, Innofactor) to explore the shifting landscape of data and the transformative role of artificial intelligence. Together, they examine how AI is evolving data strategies, driving innovation, and impacting analytics in Nordic enterprises, highlighting emerging trends in machine learning and data-driven business growth.

The Evolution Exchange Podcast Nordics
Evo Nordics #619 - Journeys & Challenges Of Becoming A Leader

The Evolution Exchange Podcast Nordics

Play Episode Listen Later Jun 20, 2025 43:03


Luke Vickers speaks with Mikolas Hamalainen (Director Consulting Expert, CGI), Elif Kocaman (Head of Analytics, Wolt), Mikko Alutoin (Chief Data Engineer, Kone), and Arnob Khan (Chapter Lead, Data Engineering and Platform, Posti Group) about the challenges and growth involved in stepping into leadership roles. They share personal insights into building high-performing teams, navigating the evolving data landscape, and leading through uncertainty. This episode dives into the real-world experiences and career transitions of emerging leaders across the Nordic tech industry.

AgileBI
Data Engineering Patterns with Chris Gambill - Episode #65

AgileBI

Play Episode Listen Later Jun 13, 2025 64:28


Join Shane Gibson as he chats with Chris Gambill about a number of Data Engineering patterns. You can get in touch with Chris via LinkedIn or view his YouTube channel https://www.youtube.com/@GambillDataEngineering If you want to download the transcript for the podcast, head over to: https://agiledata.io/podcast/agiledata-podcast/data-engineering-patterns-with-chris-gambill/#read   If you want want to read a summary generated with GenAI, head over to: https://agiledata.substack.com/p/data-engineering-patterns-with-chris   Listen to more podcasts on applying AgileData patterns over at https://agiledata.io/podcasts/ Read more on the Agile Data Way of Working over at https://AgileDataGuides.com/   If you want to join us on the next podcast, get in touch over at https://agiledata.io/podcasts/#contact   Or if you just want to talk about making magic happen with agile and data you can connect with Shane @shagility on LinkedIn.   Subscribe: Apple Podcast | Spotify | Google Podcast  | Amazon Audible | TuneIn | iHeartRadio | PlayerFM | Listen Notes | Podchaser |  Deezer | Podcast Addict |  Buy the Green Book now! Simply Magical Data

alphalist.CTO Podcast - For CTOs and Technical Leaders
#123 - From Nokia to AI-IoT: Engineering the Physical World with Bernd Groß // CEO @ Cumulocity

alphalist.CTO Podcast - For CTOs and Technical Leaders

Play Episode Listen Later Jun 12, 2025 63:07 Transcription Available


The physical world is becoming digital—and it requires fundamentally different technical architecture than traditional IT systems. Bernd Groß leads technical leaders through the evolution from enterprise software to industrial IoT, where real-time data from 30,000 wind turbines and millisecond-level decision-making define system requirements. As co-founder and CEO of Cumulocity, Bernd has navigated one of tech's most complex domains: connecting industrial hardware through standardized platforms. His journey from Nokia's early cloud computing initiatives to building Germany's leading IoT platform offers unique insights on technical leadership in physical-digital convergence. Technical leaders will gain valuable perspectives on: •

The Joe Reis Show
Matthew Scullion - The Agentic Data Engineering Team

The Joe Reis Show

Play Episode Listen Later Jun 10, 2025 45:13


Matthew Scullion (CEO, Co-Founder of Matillion) joins me to chat about the future of data engineering, namely agentic data engineering teams. What does this new world look like? Matthew shares some ideas of what he's building at Matillion, and the broader context of what agentic AI means for the data ecosystem, teams, and workflows.

The Measure Pod
#122 Digital transformation in the age of AI (with Antony Mayfield at Brilliant Noise)

The Measure Pod

Play Episode Listen Later Jun 6, 2025 72:02


Full show notes, transcript and AI chatbot - https://bit.ly/45GkOF6 Watch on YouTube - https://youtu.be/EOCkEORVToo 00:00:00 - Introduction 00:00:47 - News Overview 00:02:25 - Speculations on AI and Hardware Integration 00:03:25 - AI Devices and Market Speculations 00:04:03 - Subscription Models and Data Privacy Concerns 00:05:21 - AI and Consumer Technology 00:05:48 - Anthropic's New Claude Models 00:06:55 - AI Model Benchmarks and Comparisons 00:08:34 - Practical Uses of New Claude Versions 00:09:45 - AI in Data Engineering and Analysis 00:10:30 - Google Ads Data Manager API 00:11:02 - Google Tag Gateway 00:13:10 - First-Party Data and Privacy 00:16:11 - Transition to AI-Themed Episode 00:17:00 - Introduction to Guest Anthony Mayfield 00:18:00 - Background of Brilliant Noise 00:19:28 - Digital Transformation and AI 00:22:13 - Technological Evolution and Complexity 00:25:35 - Human Adaptation to Rapid Technological Change 00:30:20 - Understanding and Interfacing with Complex Technology 00:32:44 - Brilliant Noise's Pivot to AI 00:37:13 - Practical AI Adoption in Companies 00:39:13 - Importance of Data Management for AI 00:40:06 - Bottom-Up Approach to AI Adoption 00:43:47 - Encouraging AI Adoption in Companies 00:45:07 - Effective Training and Learning for AI Adoption ----- Episode Summary: In this episode of The Measure Pod, Dara and Matt are joined by Antony Mayfield to talk about what 15+ years of digital transformation work actually looks like and how it's changed. From the rise of influencer marketing to the sudden urgency of AI, Anthony shares what he's seeing on the ground with big brands and why adapting to new tech is no longer optional. Expect a wide-ranging chat on systems, shifts, and why ChatGPT isn't just a tool, it's a turning point. ----- About The Measure Pod: The Measure Pod is your go-to fortnightly podcast hosted by seasoned analytics pros. Join Dara Fitzgerald (Co-Founder at Measurelab) & Matthew Hooson (Head of Engineering at Measurelab) as they dive into the world of data, analytics and measurement—with a side of fun. ----- If you liked this episode, don't forget to subscribe to The Measure Pod on your favourite podcast platform and leave us a review. Let's make sense of the analytics industry together! The post #122 Digital transformation in the age of AI (with Antony Mayfield at Brilliant Noise) appeared first on Measurelab.

SQL Data Partners Podcast
Episode 285: Who Is Using Microsoft Fabric

SQL Data Partners Podcast

Play Episode Listen Later May 29, 2025 34:05


Fabric personas were originally designed to break down the various functional roles within Microsoft Fabric—such as Power BI, Data Factory, Data Activator, Data Engineering, Data Science, Data Warehouse, and Real-time Analytics—into more manageable, bite-sized sections. The goal was to prevent users from feeling overwhelmed by the platform's breadth. However, this feature has since been discontinued, as it did not effectively communicate the seamless integration between these roles. Still, the underlying concepts can be useful when thinking about how you might approach Fabric from a functional standpoint. Do you like the change on one large white canvas, or did personas have a use for you? Let us know in the comments below. We hope you enjoyed this conversation on personas in Microsoft Fabric. If you have questions or comments, please send them our way. We would love to answer your questions on a future episode. Leave us a comment and some love ❤️on LinkedIn, X, Facebook, or Instagram. The show notes for today's episode can be found at Episode 285: Who is Using Microsoft Fabric. Have fun on the SQL Trail!

The Data Stack Show
246: AI, Abstractions, and the Future of Data Engineering with Pete Hunt of Dagster

The Data Stack Show

Play Episode Listen Later May 28, 2025 48:59


Highlights from this week's conversation include:Pete's Background and Journey in Data (1:36)Evolution of Data Practices (3:02)Integration Challenges with Acquired Companies (5:13)Trust and Safety as a Service (8:12)Transition to Dagster (11:26)Value Creation in Networking (14:42)Observability in Data Pipelines (18:44)The Era of Big Complexity (21:38)Abstraction as a Tool for Complexity (24:41)Composability and Workflow Engines (28:08)The Need for Guardrails (33:13)AI in Development Tools (36:24)Internal Components Marketplace (40:14)Reimagining Data Integration (43:03)Importance of Abstraction in Data Tools (46:17)Parting Advice for Listeners and Closing Thoughts (48:01)The Data Stack Show is a weekly podcast powered by RudderStack, customer data infrastructure that enables you to deliver real-time customer event data everywhere it's needed to power smarter decisions and better customer experiences. Each week, we'll talk to data engineers, analysts, and data scientists about their experience around building and maintaining data infrastructure, delivering data and data products, and driving better outcomes across their businesses with data.RudderStack helps businesses make the most out of their customer data while ensuring data privacy and security. To learn more about RudderStack visit rudderstack.com.

The Data Stack Show
The PRQL: Breaking Down Silos: Collaborative Data Engineering in the AI Era with Pete Hunt of Dagster

The Data Stack Show

Play Episode Listen Later May 26, 2025 3:20


The Data Stack Show is a weekly podcast powered by RudderStack, customer data infrastructure that enables you to deliver real-time customer event data everywhere it's needed to power smarter decisions and better customer experiences. Each week, we'll talk to data engineers, analysts, and data scientists about their experience around building and maintaining data infrastructure, delivering data and data products, and driving better outcomes across their businesses with data.RudderStack helps businesses make the most out of their customer data while ensuring data privacy and security. To learn more about RudderStack visit rudderstack.com.

What's new in Cloud FinOps?
WNiCF - Interview with Udam - do FinOps like a Fortune 50

What's new in Cloud FinOps?

Play Episode Listen Later May 19, 2025 52:00


Send us a textIn this episode, Frank and Stephen talk with Udam about the changing world of FinOps. They focus on the challenges organizations face in managing cloud costs and data accuracy. Udam shares insights from his vast experience in the field. He highlights the need for data standardization, the role of data engineering, and ongoing investment in FinOps practices. The conversation also explores innovative solutions like Stitcher AI, which aim to simplify data management and improve decision-making in financial operations. The speakers dive into the complexities of FinOps, stressing the need to understand fully loaded costs. They discuss engaging teams in cost allocation and the importance of contextualized data for informed decisions. They address issues in traditional cost management and how new tools can democratize access to financial insights. These tools help organizations of all sizes optimize cloud spending and boost operational efficiency.Sound Bites"We don't keep any of the data.""We have to be really accurate with our costs.""You cost double your salary to the company""Cost allocation is hugely important for FinOps""FinOps has to be a bi-directional interaction"

Data Culture Podcast
Trends in Data Engineering – from AI to Automation – with Joe Reis

Data Culture Podcast

Play Episode Listen Later May 19, 2025 37:00


Joe Reis and Carsten discuss the evolving landscape of data engineering.

The Tech Trek
Building AI Products? Start Here

The Tech Trek

Play Episode Listen Later May 16, 2025 26:11


In this episode of The Tech Trek, Amir speaks with Patrick Leung, CTO of Faro Health, about what it takes to lead an engineering organization through a transformation to become an AI-first company. From redefining the product roadmap to managing cultural and technical shifts, Patrick shares practical insights on team structure, skill development, and delivering AI-enabled features in a regulated domain like clinical trials. This is a must-listen for tech leaders navigating similar transitions.

Data Gen
#203 - Masterclass | Mettre en place une Modern Data Stack avec Matthieu Rousseau

Data Gen

Play Episode Listen Later May 14, 2025 24:48


Matthieu Rousseau, expert en Data Engineering et DataOps, a fondé Modeo, un cabinet de conseil spécialisé sur la Modern Data Stack et le DataOps qui travaille avec des Grands Groupes et des Startups.

The Data Stack Show
241: Marketing Meets Data: Measuring Impact and Driving Results with Pedram Navid of Dagster Labs

The Data Stack Show

Play Episode Listen Later May 12, 2025 38:42


Highlights from this week's conversation include:Pedram's Background and Journey in Data (1:13)Marketing vs. Data Engineering (2:30)Understanding Marketing Pressures (4:16)Attribution Models and Accountability (8:13)Balancing Marketing and Team Management (12:25)Introduction to Dagster Components (15:00)AI Integration with Data Engineering (19:05)Challenges in Data Support (22:05)Self-Service Data Access (26:07)AI in Data Management (28:25)Organizing Data in Technical Teams (31:25)Challenges in Real-Time Data (33:28)Final Thoughts and Takeaways (37:01)The Data Stack Show is a weekly podcast powered by RudderStack, the CDP for developers. Each week we'll talk to data engineers, analysts, and data scientists about their experience around building and maintaining data infrastructure, delivering data and data products, and driving better outcomes across their businesses with data.RudderStack helps businesses make the most out of their customer data while ensuring data privacy and security. To learn more about RudderStack visit rudderstack.com.

Data Gen
#202 - On compare le Staff, le Senior et l'Engineering Manager avec Christophe Blefari (aka Blef)

Data Gen

Play Episode Listen Later May 12, 2025 30:35


Christophe Blefari est Staff Data Engineer, auteur de la newsletter data la plus connue au sein de l'écosystème français (Blef.fr), co-fondateur de nao et surtout selon moi l'un des plus gros experts data en France.On aborde :

The Evolution Exchange Podcast Nordics
Evo Nordics #576 - Data-Driven Business Strategy

The Evolution Exchange Podcast Nordics

Play Episode Listen Later May 2, 2025 45:45


Host Charlotte Roberts engages with Adarsh Mahesan Shyna, Lead Project Leader for Data Engineering & Architecture Strategy at Nordea; Dhany Saputra, Head of Data Science, AI, and Cloud Architecture at Micropower Group; and Charanya Thangaraj, Service Area Manager for Data Access Services at Volvo Buses. The discussion explores how data is driving innovation across finance, manufacturing, and mobility. Learn how these leaders implement scalable architectures and use advanced analytics to fuel smarter, faster business decisions in today's digital economy.

Let’s Talk Data
Beyond Data: The Human Algorithm of Leadership

Let’s Talk Data

Play Episode Listen Later Apr 30, 2025 47:17


Do you know what truly separates exceptional leaders from the rest? Zoe Kelly, Head of Data Engineering at Reward, journey reveals a powerful truth: leadership isn't about technical prowess alone, but about creating human-centered environments where data professionals can thrive. In a world obsessed with complex technologies and buzzwords, Kelly demonstrates that vulnerability, communication, and genuine connection are the real superpowers of data leadership. Her remarkable story of scaling a data team from three to seventeen members while navigating personal mental health challenges offers a blueprint for modern leadership. By prioritising psychological safety, continuous learning, and empathetic communication, she transforms data engineering from a purely technical discipline into a deeply human experience. Kelly's approach challenges traditional narratives, showing that the most innovative data strategies emerge not from tools, but from understanding and supporting the people behind the data. Her insights are a masterclass in authentic, transformative leadership.0:00 Intro2:56 Zoe's Journey into Data 4:26 Evolution of Data and Leadership 7:41 Balancing Leadership and Team Dynamics 9:28 Challenges and Strategies in Data Leadership 15:05 Job Descriptions and Industry Expectations 18:39 Diversity and Inclusion in Data 23:11 Mental Health and Remote Work 31:36 Data in Golf and Personal Insights 47:11 Lightning Round and Final ThoughtsFor more information head to https://www.miraitalent.com/Welcome to “Let's Talk Data”, a deep dive into the transformative realm of data with the trailblazers of this disruptive industry.Your host Emma Crabtree explores the latest trends and developments in the data sector, while also delving into the personal stories of these data pioneers.They reveal how they harness data, not just in business, but in shaping their everyday lives, from optimising daily routines to making data-informed decisions. You'll hear about their motivations, role models, and how they plan to use their position to innovate, inspire and influence change.Many companies today are still missing the golden opportunity to unlock true potential with their data. Our conversations will shed light on this untapped potential as well as the pivotal role that data professionals play in driving progress.“Let's Talk Data” is your go-to source for inspiration and knowledge, providing a front-row seat to the future of data-driven insights and innovations.

MLOps.community
AI Data Engineers - Data Engineering After AI // Vikram Chennai // #309

MLOps.community

Play Episode Listen Later Apr 25, 2025 49:40


AI Data Engineers - Data Engineering after AI // MLOps Podcast #309 with Vikram Chennai, Founder/CEO of Ardent AI.Join the Community: https://go.mlops.community/YTJoinIn Get the newsletter: https://go.mlops.community/YTNewsletter // AbstractA discussion of Agentic approaches to Data Engineering. Exploring the benefits and pitfalls of AI solutions and how to design product-grade AI agents, especially in data.// BioSecond Time Founder. 5 years building Deep learning models. Currently, AI Data Engineers// Related LinksWebsite: tryardent.com~~~~~~~~ ✌️Connect With Us ✌️ ~~~~~~~Catch all episodes, blogs, newsletters, and more: https://go.mlops.community/TYExploreJoin our slack community [https://go.mlops.community/slack]Follow us on X/Twitter [@mlopscommunity](https://x.com/mlopscommunity) or [LinkedIn](https://go.mlops.community/linkedin)] Sign up for the next meetup: [https://go.mlops.community/register]MLOps Swag/Merch: [https://shop.mlops.community/]Connect with Demetrios on LinkedIn: /dpbrinkmConnect with Vikram on LinkedIn: /vikram-chennai/

Morning Data Chat
S4 - #1 Dylogy : Automatiser l'analyse documentaire dans l'assurance grâce à l'IA générative

Morning Data Chat

Play Episode Listen Later Apr 25, 2025 27:29


Comment l'IA générative transforme-t-elle la gestion du risque dans l'assurance ?Aurélien Couloumy, actuaire de formation et CEO de Dylogy, partage son retour d'expérience sur l'automatisation de l'analyse documentaire dans un secteur exigeant et réglementé.Hébergé par Ausha. Visitez ausha.co/politique-de-confidentialite pour plus d'informations.

BIFocal - Clarifying Business Intelligence
Episode 292 - Microsoft Fabric March 2025 Feature Summary

BIFocal - Clarifying Business Intelligence

Play Episode Listen Later Apr 24, 2025 37:59


This is episode 292 recorded on April 22nd, 2025 where John & Jason talk the Microsoft Fabric February 2025 Feature Summary including Fabric Platform, OneLake, and Data Engineering.

HR to HX: From Human Resources to the Human Experience
How Data Literacy Transforms HR Leadership (Part Two)

HR to HX: From Human Resources to the Human Experience

Play Episode Listen Later Apr 15, 2025 25:57


Hey humans! Welcome back to part two of our data literacy series! In this episode, I continue my round table discussion with Jennifer Wright, Karen Brieger, and Kelly Oliver as we dive deeper into the practical aspects of HR data. We explore the critical importance of "clean data" - what it really means and why it matters so much. Kelly breaks down the three-phase framework of data engineering, data analysis, and data insights, explaining why you can't just jump straight to insights without proper preparation. Jennifer shares a powerful example about how software conversion challenges can create data issues, while Karen emphasizes the importance of basic data accuracy - from employee addresses to departmental coding. Our conversation also addresses the unique challenges faced by fractional HR leaders who need to quickly adapt to different organizational cultures and data preferences. As Karen explains, some clients want workshops and training while others just want quick data points without the deeper understanding. We discuss how to navigate these different environments while still delivering value. We also touch on the future of HR data and analytics with cautious optimism about AI's potential to help identify anomalies and patterns. There are still so many organizations still mastering the basics of data literacy. Enjoy part two of our data conversation! Stacie More episodes at StacieBaird.com. LinkedIn Pages for each of my guests: Jennifer Wright Karen Brieger Kelly Oliver

Innovation and the Digital Enterprise
Resonant Insights On Leadership: A Compilation

Innovation and the Digital Enterprise

Play Episode Listen Later Apr 10, 2025 15:12 Transcription Available


In this episode, we look back on invaluable leadership insights shared by renowned executives across the technology industry. Featuring wisdom from Christine Sandman Stone, Arne Saupe, Jorie Sax, Thulasi Kethini, and Tony Newcome, the discussion delves into the distinct skill sets required for leadership, the importance of fostering a culture of innovation, and the critical need for supportive and open communication within teams. These leaders unpack practical advice for setting meaningful goals, building complementary partnerships, and maintaining resilience. We learn about the ways leadership requires courage, decisiveness, and the ability to connect deeply with team members. Their words provide a rich tapestry of experiences and strategies to inspire aspiring and current leaders.(00:00) Introduction to Leadership Insights(02:10) Christine Sandman Stone on Leadership vs. Development(03:18) Arne Saupe on Leadership Culture and Talent(04:54) Jorie Sax on Empowering Creativity and Innovation(06:19) Thulasi Kethini on Fostering Curiosity(08:26) Tony Newcome on Connecting and Supporting Your Team(10:43) Christine Sandman Stone on Setting Meaningful Goals(11:16) Arne Saupe on Vision(11:55) Tony Newcome on Mentorship(14:03) Christine Sandman Stone on Being a First-Time Leader(14:45) Conclusion: Reflecting on Effective LeadershipChristine Sandman Stone is the former Global Head of Product & Engineering Operations & Strategy at Groupon. Arne Saupe is the former CTO at Farmer's Fridge and newly appointed CTO at Wellfound Foods. Jorie Sax heads United Airlines' Innovation Lab. Thulasi Kethini is the Executive Director and Head of Data Engineering at JPMorgan, Tony Newcome is CTO at ActiveCampaign.If you'd like to receive new episodes as they're published, please subscribe to Innovation and the Digital Enterprise in Apple Podcasts, Spotify, or wherever you get your podcasts. If you enjoyed this episode, please consider leaving a review in Apple Podcasts. It really helps others find the show.Podcast episode production by Dante32.

Geeksblabla
#180 - Data Engineering 101

Geeksblabla

Play Episode Listen Later Apr 8, 2025 105:18


Data engineering is a critical field in data science that involves preparing the "big data" infrastructure to be analyzed by data scientists. In this episode we are discussing the differences and how important each is with our guests.

Vanishing Gradients
Episode 47: The Great Pacific Garbage Patch of Code Slop with Joe Reis

Vanishing Gradients

Play Episode Listen Later Apr 7, 2025 79:12


What if the cost of writing code dropped to zero — but the cost of understanding it skyrocketed? In this episode, Hugo sits down with Joe Reis to unpack how AI tooling is reshaping the software development lifecycle — from experimentation and prototyping to deployment, maintainability, and everything in between. Joe is the co-author of Fundamentals of Data Engineering and a longtime voice on the systems side of modern software. He's also one of the sharpest critics of “vibe coding” — the emerging pattern of writing software by feel, with heavy reliance on LLMs and little regard for structure or quality. We dive into: • Why “vibe coding” is more than a meme — and what it says about how we build today • How AI tools expand the surface area of software creation — for better and worse • What happens to technical debt, testing, and security when generation outpaces understanding • The changing definition of “production” in a world of ephemeral, internal, or just-good-enough tools • How AI is flattening the learning curve — and threatening the talent pipeline • Joe's view on what real craftsmanship means in an age of disposable code This conversation isn't about doom, and it's not about hype. It's about mapping the real, messy terrain of what it means to build software today — and how to do it with care. LINKS * Joe's Practical Data Modeling Newsletter on Substack (https://practicaldatamodeling.substack.com/) * Joe's Practical Data Modeling Server on Discord (https://discord.gg/HhSZVvWDBb) * Vanishing Gradients YouTube Channel (https://www.youtube.com/channel/UC_NafIo-Ku2loOLrzm45ABA) * Upcoming Events on Luma (https://lu.ma/calendar/cal-8ImWFDQ3IEIxNWk)

The GeekNarrator
Are your Data Pipelines Complex?

The GeekNarrator

Play Episode Listen Later Apr 7, 2025 83:28


The GeekNarrator memberships can be joined here: https://www.youtube.com/channel/UC_mGuY4g0mggeUGM6V1osdA/joinMembership will get you access to member only videos, exclusive notes and monthly 1:1 with me. Here you can see all the member only videos: https://www.youtube.com/playlist?list=UUMO_mGuY4g0mggeUGM6V1osdA------------------------------------------------------------------------------------------------------------------------------------------------------------------About this episode: ------------------------------------------------------------------------------------------------------------------------------------------------------------------In this conversation, Jacopo and Ciro discuss their journey in building Bauplan, a platform designed to simplify data management and enhance developer experience. They explore the challenges faced in data bottlenecks, the integration of development and production environments, and the unique approach of Bauplan using serverless functions and Git-like versioning for data. The discussion also touches on scalability, handling large data workloads, and the critical aspects of reproducibility and compliance in data management. Chapters:00:00 Introduction03:00 The Data Bottleneck: Challenges in Data Management06:14 Bridging Development and Production: The Need for Integration09:06 Serverless Functions and Git for Data17:03 Developer Experience: Reducing Complexity in Data Management19:45 The Role of Functions in Data Pipelines: A New Paradigm23:40 Building Robust Data Solutions: Versioning and Parameters30:13 Optimizing Data Processing: Bauplan Runtime46:46 Understanding Control Planes and Data Management48:51 Ensuring Robustness in Data Pipelines52:38 Data Quality and Testing Mechanisms54:43 Branching and Collaboration in Data Development57:09 Scalability and Resource Management in Data Functions01:01:13 Handling Large Data Workloads and Use Cases01:09:05 Reproducibility and Compliance in Data Management01:16:46 Future Directions in Data Engineering and Use CasesLinks and References:Bauplan website:https://www.bauplanlabs.com

The Data Stack Show
235: Pete Soderling on the Evolution of Data Engineering

The Data Stack Show

Play Episode Listen Later Apr 2, 2025 43:23


Highlights from this week's conversation include:Pete's Career Overview (1:00)AI and Data Engineering Discussion (2:05)Themes of Data Council (4:19)High-Frequency Trading Insights (8:04)San Francisco's Unique Advantages (10:27)Data Council Conference Preview (13:23)The Magic of In-Person Events (15:45)Collapsing Batch and Streaming Systems (19:47)Leveraging Local Hardware for Data Processing (22:07)Future of Blockchain in Computing (23:57)Intersection of AI and Data Management (26:47)Advice for AI Startup Founders (28:44)Blurring Lines Between Data Roles (32:46)The Evolving Role of Engineers (36:56)Discount Code for Data Council and Parting Thoughts (38:23)The Data Stack Show is a weekly podcast powered by RudderStack, the CDP for developers. Each week we'll talk to data engineers, analysts, and data scientists about their experience around building and maintaining data infrastructure, delivering data and data products, and driving better outcomes across their businesses with data.RudderStack helps businesses make the most out of their customer data while ensuring data privacy and security. To learn more about RudderStack visit rudderstack.com.

The Joe Reis Show
Freestyle Fridays - Shifting Left AND Right. The Data Engineering Lifecycle in 2025. What's Changed?

The Joe Reis Show

Play Episode Listen Later Mar 28, 2025 9:52


People often ask me what I'd change in Fundamentals of Data Engineering. Usually, I reply "not much", as the the Data Engineering Lifecycle still remains intact. However, I see the role of data engineers shifting both left and right. What does this mean? Have a listen.

The Joe Reis Show
Willis Nana - Navigating Data Engineering Leadership, YouTube, and more

The Joe Reis Show

Play Episode Listen Later Mar 20, 2025 43:37


Willis Nana and I chat about the challenges of data engineering leadership, foundational skills, and his journey to a content creator on YouTube.#dataengineering #data #ai #datateam #leadership

The Joe Reis Show
Simon Späti - The Art of Writing about Data Engineering

The Joe Reis Show

Play Episode Listen Later Mar 13, 2025 50:46


Simon Späti and I discuss various aspects of writing, data engineering, and the impact of AI on the writing process. Simon shares his journey from business intelligence to data engineering and his current focus on writing. We also discuss the future of writing in the age of AI. Enjoy!

DataTalks.Club
Trends in Data Engineering – Adrian Brudaru

DataTalks.Club

Play Episode Listen Later Mar 7, 2025 56:59


In this podcast episode, we talked with Adrian Brudaru about ​the past, present and future of data engineering.About the speaker:Adrian Brudaru studied economics in Romania but soon got bored with how creative the industry was, and chose to go instead for the more factual side. He ended up in Berlin at the age of 25 and started a role as a business analyst. At the age of 30, he had enough of startups and decided to join a corporation, but quickly found out that it did not provide the challenge he wanted.As going back to startups was not a desirable option either, he decided to postpone his decision by taking freelance work and has never looked back since. Five years later, he co-founded a company in the data space to try new things. This company is also looking to release open source tools to help democratize data engineering.0:00 Introduction to DataTalks.Club1:05 Discussing trends in data engineering with Adrian2:03 Adrian's background and journey into data engineering5:04 Growth and updates on Adrian's company, DLT Hub9:05 Challenges and specialization in data engineering today13:00 Opportunities for data engineers entering the field15:00 The "Modern Data Stack" and its evolution17:25 Emerging trends: AI integration and Iceberg technology27:40 DuckDB and the emergence of portable, cost-effective data stacks32:14 The rise and impact of dbt in data engineering34:08 Alternatives to dbt: SQLMesh and others35:25 Workflow orchestration tools: Airflow, Dagster, Prefect, and GitHub Actions37:20 Audience questions: Career focus in data roles and AI engineering overlaps39:00 The role of semantics in data and AI workflows41:11 Focusing on learning concepts over tools when entering the field 45:15 Transitioning from backend to data engineering: challenges and opportunities 47:48 Current state of the data engineering job market in Europe and beyond 49:05 Introduction to Apache Iceberg, Delta, and Hudi file formats 50:40 Suitability of these formats for batch and streaming workloads 52:29 Tools for streaming: Kafka, SQS, and related trends 58:07 Building AI agents and enabling intelligent data applications 59:09Closing discussion on the place of tools like DBT in the ecosystem

Bringing Data and AI to Life
Inside Informatica: How AI is Automating & Scaling Enterprise Data Management with Gaurav Pathak

Bringing Data and AI to Life

Play Episode Listen Later Mar 6, 2025 25:10


What if you could redefine enterprise data management with AI—while boosting productivity by 30%? In this episode of Bringing Data and AI to Life, host Amy Horowitz sits down with Gaurav Pathak, VP of Product Management for AI and Metadata at Informatica, to explore how AI is turning data from a manual headache into an enterprise-wide superpower. As a DataIQ 2025 Data and AI Leader of the Year nominee, Gaurav delivers actionable strategies to help CDOs, CIOs, and data architects scale operations, enhance governance, and harness AI-driven automation. Tune in now to embrace innovation—without sacrificing security or trust.

DOU Podcast
Найращі ІТ-компаній України | Вихід Netcracker з рф | Новий iPhone 16e — DOU News #186

DOU Podcast

Play Episode Listen Later Feb 24, 2025 37:52


Product by Design
Lessons in Data Engineering: Scaling, AI, and Open Source with Sandy Ryza

Product by Design

Play Episode Listen Later Feb 7, 2025 46:28


In this episode of Product by Design, Kyle chats with Sandy Ryza, lead engineer on the Dagster project, author, and thought leader in data engineering. Sandy shares his journey through the world of data—from building big data tools at Cloudera to working as a data scientist, product manager, and engineer—and how those experiences led him to help create Dagster, an open-source data orchestration platform.We discuss:The evolution of data engineering and the growing complexity of modern data pipelines.The role of AI and unstructured data in shaping the future of data platforms.How organizations should think about data platforms to avoid costly rework.Best practices for managing data complexity using software engineering principles.The future of open-source tools in data infrastructure and the push toward interoperability.Sandy RyzaSandy is a lead engineer, author, and thought leader in the domain of data engineering. Sandy co-wrote “Advanced Analytics with PySpark” and “Advanced Analytics with Spark”. He led ML and data science teams at Cloudera, Remix, Clover Health, and KeepTruckin.Sandy is currently the lead engineer on the Dagster project, an open-source data orchestration platform used in MLOps, data science, IOT and analytics. Sandy is a regular speaker at data engineering and ML conferences.Links from the Show:Twitter: @s_RYZDagster: dagster.ioBook: Advanced Analytics with Spark – O'ReillyPodcast Recommendation: Empire (British Empire & Ottoman Empire history)Books Sandy is Reading: The Shortest History of India, The Sun Also Rises, Werner Herzog's AutobiographyMore by Kyle:Follow Prodity on Twitter and TikTokFollow Kyle on Twitter and TikTokSign up for the Prodity Newsletter for more updates.Kyle's writing on MediumProdity on MediumLike our podcast, consider Buying Us a Coffee or supporting us on Patreon

Data Product Management in Action: The Practitioner's Podcast
S1 Ep#30: From Engineering to Data Strategy: Driving AI and Decision-Making

Data Product Management in Action: The Practitioner's Podcast

Play Episode Listen Later Feb 5, 2025 48:17 Transcription Available


S1 Ep#30: From Engineering to Data Strategy: Driving AI and Decision-Making The Data Product Management In Action podcast, season 1, is brought to you by Soda and executive producer Scott Hirleman, is a platform for data product management practitioners to share insights and experiences. Our guest this week is Theo Bell, a data product manager. She chats with host Nick Zerviudis and shares her transition from mechanical engineering to roles at Goldman Sachs and Palantir, emphasizing the importance of data integration in strategic decision-making. She discusses how Palantir helped a manufacturer prioritize client orders during raw material shortages and explores the challenges of convincing stakeholders to adopt new data models, advocating for production-ready pilots over proof-of-concepts. Theo also offers insights on fostering AI adoption within organizations, using a news summarization tool for a CEO as an example. She recommends the GTD framework and Surrounded by Idiots for enhancing productivity and communication. About our Host Nick Zervoudis: Nick is Head of Product at CKDelta, an AI software business within the CK Hutchison Holdings group. Nick oversees a portfolio of data products and works with sister companies to uncover new opportunities to innovate using data, analytics, and machine learning. Nick's career has revolved around data and advanced analytics from day one, having worked as an analyst, consultant, product manager, and instructor for startups, SMEs, and enterprises including PepsiCo, Sainsbury's, Lloyds Banking Group, IKEA, Capgemini Invent, BrainStation, QuantSpark, and Hg Capital. Nick is also the co-host of London's Data Product Management meetup, and speaks & writes regularly about data & AI product management. Connect with Nick on LinkedIn and through his newsletter, Value from Data & AI. About our Guest Theo Bell: Theo is the Head of AI Product at Rimes, where she leads the company's efforts to leverage AI technology in order to provide cutting-edge data management solutions to clients. Previously, Theo held key roles at Palantir Technologies and Goldman Sachs, where she enabled various industries to leverage data through AI/ML-driven software, notably Airbus' Skywise platform, the NHS, and the UK Ministry of Defense. Theo is dedicated to using AI and technology for global challenges, particularly in improving health, enhancing society, and fostering sustainable businesses. She holds a PhD in Engineering from the University of Cambridge.  Connect with Theo in LinkedIn. All views and opinions expressed are those of the individuals and do not necessarily reflect their employers or anyone else.  Join the conversation on LinkedIn.  Apply to be a guest or nominate someone that you know.  Do you love what you're listening to? Please rate and review the podcast, and share it with fellow practitioners you know. Your support helps us reach more listeners and continue providing valuable insights!    .

Data Product Management in Action: The Practitioner's Podcast
S1 Ep#29 : Navigating Trade-Offs and Balancing Priorities

Data Product Management in Action: The Practitioner's Podcast

Play Episode Listen Later Jan 29, 2025 49:47 Transcription Available


Season 1 Episode 29: Navigating Trade-Offs and Balancing Priorities The Data Product Management In Action podcast, brought to you by executive producer Scott Hirleman, is a platform for data product management practitioners to share insights and experiences. In this episode of Data Product Management in Action, host Alexa Westlake talks with Anita Chen, diving into the complexities of managing data products. Anita, a product manager at PagerDuty, shares her approach to defining data products, prioritizing work, and balancing project work with interrupt-driven tasks. They discuss the critical roles of governance, security, and user enablement while emphasizing the importance of transparency and communication. The conversation also explores the transformative potential of generative AI in data product interactions and the build-vs-buy decision-making process. Gain insights into how data product management uniquely differs from traditional software product management and learn actionable strategies for success. Meet our Host Alexa Westlake: Alexa is a Data Analytics Leader in the Identity and Access Management space with a proven track record scaling high-growth SaaS companies. As a Staff Data Analyst at Okta, she brings a wealth of expertise in enterprise data, business intelligence, and strategic decision-making from the various industries she's worked in including telecommunications, strategy execution, and cloud computing. With a passion for harnessing the power of data for actionable insights, Alexa plays a crucial role in driving Okta's security, growth, and scale, helping the organization leverage data to execute on their market opportunity. Connect with Alexa on LinkedIn. Meet our guest Anita Chen:  Anita is a Data Product Manager at PagerDuty, a digital operations company helping teams resolve issues faster, eliminate alert fatigue, and build more reliable services! Her background is mainly in the People Analytics space which has now expanded to data at scale with our Enterprise Data Team. She currently helps build data products that enable our teams to deliver the best possible customer experience. Anita is most passionate about how data can impact someone's lived experience and endeavor to democratize data in everything she builds. Connect with Anita on LinkedIn. All views and opinions expressed are those of the individuals and do not necessarily reflect their employers or anyone else.  Join the conversation on LinkedIn.  Apply to be a guest or nominate someone that you know.  Do you love what you're listening to? Please rate and review the podcast, and share it with fellow practitioners you know. Your support helps us reach more listeners and continue providing valuable insights! 

The MAD Podcast with Matt Turck
Understanding Data Engineering in 2025 | Ben Rogojan, Seattle Data Guy

The MAD Podcast with Matt Turck

Play Episode Listen Later Jan 23, 2025 62:24


As AI takes over the world, data is more than ever “the new oil”, and data engineering is the discipline that makes data usable behind the scenes. In this episode, we dive deep into the present and future of data engineering with Ben Rogojan, also known as the Seattle Data Guy. A seasoned data engineering consultant, Ben has built a big brand and reputation in the field with over 100k followers on platforms like YouTube and Substack. We started the conversation with a deep dive into data engineering as a profession: what do data engineers actually do? What is the career path, and what should aspiring data engineers learn? We then explored some of the biggest stories of 2024 (including the rise of Iceberg) and went into some predictions for 2025, as a way to discuss some key topics everyone should be familiar with in data engineering, including the integration of AI in data workflows, the potential for automation, and why SQL isn't going anywhere. Discover how companies are navigating the complexities of data infrastructure, the rise of open table formats like Iceberg, and the ongoing battle between data giants like Snowflake and Databricks. Ben Rogojan Website - https://www.theseattledataguy.com Newsletter - https://seattledataguy.substack.com LinkedIn - https://www.linkedin.com/company/seattle-data-guy X/Twitter - https://x.com/seattledataguy FIRSTMARK Website - https://firstmark.com X/Twitter - https://twitter.com/FirstMarkCap Matt Turck (Managing Director) LinkedIn - https://www.linkedin.com/in/turck/ X/Twitter - https://twitter.com/mattturck (00:00) Intro (01:20) Why 2025 will be huge for data engineering (02:55) The story of the Seattle Data Guy (06:51) What exactly is data engineering? (07:41) Data, AI, and ML: where do they overlap? (09:23) Data analyst vs. data engineer vs. data scientist: what's the difference? (11:20) A day in the life of a data engineer (12:58) Data engineering: Silicon Valley vs. everywhere else (15:27) How to become an AI engineer (28:46) Will AI replace AI engineers? (33:42) Why is the data world so complex? (36:53) The functional consolidation of the data world (38:34) Big data stories from 2024 (39:28) Why Iceberg is a game-changer (46:02) How startups manage data in their early days (48:44) Seattle Data Guy's favorite tools (50:09) Bold predictions for 2025

The Data Chief
Data and AI Predictions for 2025 with Matt Turck, Steve Nouri, and Joe Reis

The Data Chief

Play Episode Listen Later Jan 22, 2025 80:45


In this season premiere of The Data Chief podcast, your host Cindi Howson sits down with three industry visionaries to explore the trends, predictions, and must-take actions for data leaders in 2025. Get ready for a deep dive into: The generative AI revolution with Matt Turck, Partner at FirstMark CapitalThe future of data science and genAI with Steve Nouri, Founder of GenAI Works and AI for DiversityData Engineering in the Age of AI with Joe Reis, author of "Fundamentals of Data Engineering" and the upcoming "Mixed Model Arts."Plus: Hear their fun predictions for everything from sports to space travel!Key Moments:The generative AI revolution: Matt Turck, Partner at FirstMark Capital shares his insights on the evolving AI landscape, the rise of unstructured data, and why now is the time for enterprises to embrace AI. (1:40) The Future of Data Science: Steve Nouri, Founder of GenAI Works (an 8-million-strong community!) and AI for Diversity, discusses the impact of GenAI on data science roles, the ethical considerations of AI, and exciting trends like embodied AI and agentic AI. (29:36) Data Engineering in the Age of AI: Joe Reis, author of "Fundamentals of Data Engineering" and the upcoming "Mixed Model Arts," provides his expert perspective on the importance of data modeling, the need for upskilling in data teams, and the potential for a universal semantic layer. (1:00:00) Key Quotes:“I would predict that there's going to be a number of big acquisitions in our general space in 2025. This whole tension between the public markets doing very well, especially in tech, but the private markets still recovering - I think lends itself well to a wave of consolidation.”  - Matt Turck“Anything that requires democratization, I'm a big fan of. And certainly, the ability to query natural language databases and all things, making that available to everyone is a very powerful idea. You guys at ThoughtSpot know this better than anyone.” - Matt Turck“We are seeing people doing less coding, more relying on their co-pilots. It's going to evolve to become more and more robust. So we will be relying more on AI to do the coding.” - Steve Nouri“Well, that's what, you know, the tagline is, AI will do everything for you. It'll even do your laundry, the jobs that we don't like. And so you're actually saying you see a future where that actually is not too far off.” - Steve Nouri“I think that there's definitely a FOMO and a bit of a prisoner's dilemma problem with adopting AI in the organization because they're getting a lot of pressure from the top down, especially to do AI. Understanding what that means to your organization should be table stakes.” - Joe Reis“Learning never stops, investment never stops. And the best investment you can make is always improving yourself, no matter what that looks like.” Joe ReisMentions:FirstMark MAD Landscape 2024The MAD Podcast with Matt TurckAI4DiversityGenAI.WorksFundamentals of Data EngineeringJoe Reis Substack Guest Bios:Matt Turck is a Partner at FirstMark, where he focuses primarily on early-stage enterprise investing in the US and Europe. Matt is particularly active in the data, machine learning and AI space. For the last 10+ years, he has been organizing Data Driven NYC, the largest data/AI community in the US, and publishing the MAD Landscape, an annual analysis of the data/AI industry. He also hosts the weekly MAD (ML, AI, Data) Podcast. He can be followed on X/Twitter at @mattturck.Steve Nouri is the CEO and Co-founder of GenAI Works, the largest AI community. He is a renowned AI leader and Australia's ICT Professional of the Year, has revolutionized AI perspectives while championing Responsible and inclusive AI, founding a global non-profit initiative.Joe Reis, a "recovering data scientist" with 20 years in the data industry, is the co-author of the best-selling O'Reilly book, "Fundamentals of Data Engineering." He's also the instructor for the wildly popular Data Engineering Professional Certificate on Coursera, in partnership with DeepLearning.ai and AWS.Joe's extensive experience encompasses data engineering, data architecture, machine learning, and more. He regularly keynotes major data conferences globally, advises and invests in innovative data product companies, writes at Practical Data Modeling and his personal blog, and hosts the popular data podcasts "The Monday Morning Data Chat" and "The Joe Reis Show." In his free time, Joe is dedicated to writing new books and articles, and thinking of ways to advance the data industry. Hear more from Cindi Howson here. Sponsored by ThoughtSpot.

Cloud Wars Live with Bob Evans
Cloud Wars CEO Outlook 2025 with Snowflake's Sridhar Ramaswamy

Cloud Wars Live with Bob Evans

Play Episode Listen Later Jan 16, 2025 27:56


The Future of AI and Data in 2025:  There's a lot of excitement and prototyping around AI. "People are excited that they can ask questions and get real answers instead of getting the proverbial 10 blue links. I think people see the promise of things like that. How do you make that work at scale? How do you make that work in a reliable way, and how do you make it work in a way that's not expensive?" Ramaswamy shares a recent experience in India, describing the cost-consciousness of customers and the focus on making data accessible and valuable.Shifting Priorities for Snowflake Customers:  What do customers want? Ramaswamy identifies goals including empowering business users with data faster and unlocking the value of unstructured information. In response, Snowflake is helping customers access and utilize historical information and build hybrid systems. He notes that SQL enables easier data transformations and extractions.Cultural Change and AI Adoption: Cultural change is required for companies to handle AI. Even within Snowflake, adopting AI has been a challenging process. Snowflake's engineering and sales teams are beginning to use AI tools like Copilot and Cortex, but there's a need for ongoing practice to fully integrate AI into daily operations. The Promise of AI and Data Platforms: AI has the potential to free up significant time for employees, allowing them to focus on higher-value tasks. Snowflake aspires to make data platforms simple and accessible, allowing users to perform complex tasks with ease. Its consumption model has several benefits and aligns it with business goals of making more money or spending less. Companies like Siemens are using chatbots to improve efficiency and access to information.Competition and Innovation in the Data Market: Competition, Ramaswamy says, drives innovation and improves products. It only strengthens Snowflake, which stands apart due to its unified platform, tight integration, and ease of use. "If you create a chatbot with Snowflake, it will automatically obey all of the access control rules that you have set up. It will automatically obey all of the data masking that your administrator has set up." Ramaswamy discusses the challenges of balancing innovation with maintaining simplicity and efficiency in Snowflake's product offerings.Mega Trends and Challenges for Snowflake: Ramaswamy sees two mega trends for 2025: the rise of interoperable data formats and the acceleration of AI. Snowflake is well-positioned to take advantage of these trends through its end-to-end offerings and data engineering capabilities. However, there are challenges to maintaining data quality and speed while innovating rapidly. AI tools like agents must drive real utility and value for customers.Data Literacy Initiative: One Million Minds Plus One: Snowflake has a new initiative aimed at improving data literacy. Data can be difficult to interpret, and the right skills are necessary to do so. Snowflake is spearheading an ambitious program to educate over a million people in data literacy, free of charge. The initiative includes teaching concepts like data engineering, data analytics, and using BI tools and notebooks for interactive analysis.Excitement for 2025: Sridhar expresses excitement about the potential for AI and data to drive real business transformations in 2025. Snowflake will continue to play a vital role in helping customers become truly data-driven organizations. Navigating growth, competition, and innovation while maintaining simplicity and value for customers is key.

The Analytics Engineering Podcast
Data engineering at Snowflake (w/ Rahul Jain)

The Analytics Engineering Podcast

Play Episode Listen Later Jan 12, 2025 44:04


A look inside at the data work happening at a company making some of the most advanced technologies in the industry. Rahul Jain, data engineering manager at Snowflake, joins Tristan to discuss Iceberg, streaming, and all things Snowflake.  For full show notes and to read 6+ years of back issues of the podcast's companion newsletter, head to https://roundup.getdbt.com. The Analytics Engineering Podcast is sponsored by dbt Labs.

The Data Engineering Show
AI and Data Change Management with Chad Sanderson, CEO Gable AI

The Data Engineering Show

Play Episode Listen Later Jan 7, 2025 36:43


In this episode of The Data Engineering Show, host Benjamin and co-host Eldad are joined by Chad Sanderson, CEO and co-founder of Gable AI to discuss the revolution of data quality and governance, the importance of understanding data flow and the processes that help organizations manage their data more effectively.

The Joe Reis Show
Freestyle Fridays - The Year Ahead (Why Data Modeling Matters, AI, Being Human, etc)

The Joe Reis Show

Play Episode Listen Later Jan 3, 2025 21:39


It's 2025! We made it! ;) In this podcast, I rant about why data modeling matters more than ever, AI, and why humans will seek out "human" things in 2025 and beyond. ❤️ Your support means a lot. Please like and rate this podcast on your favorite podcast platform.

Vanishing Gradients
Episode 41: Beyond Prompt Engineering: Can AI Learn to Set Its Own Goals?

Vanishing Gradients

Play Episode Listen Later Dec 30, 2024 43:51


Hugo Bowne-Anderson hosts a panel discussion from the MLOps World and Generative AI Summit in Austin, exploring the long-term growth of AI by distinguishing real problem-solving from trend-based solutions. If you're navigating the evolving landscape of generative AI, productionizing models, or questioning the hype, this episode dives into the tough questions shaping the field. The panel features: - Ben Taylor (Jepson) (https://www.linkedin.com/in/jepsontaylor/) – CEO and Founder at VEOX Inc., with experience in AI exploration, genetic programming, and deep learning. - Joe Reis (https://www.linkedin.com/in/josephreis/) – Co-founder of Ternary Data and author of Fundamentals of Data Engineering. - Juan Sequeda (https://www.linkedin.com/in/juansequeda/) – Principal Scientist and Head of AI Lab at Data.World, known for his expertise in knowledge graphs and the semantic web. The discussion unpacks essential topics such as: - The shift from prompt engineering to goal engineering—letting AI iterate toward well-defined objectives. - Whether generative AI is having an electricity moment or more of a blockchain trajectory. - The combinatorial power of AI to explore new solutions, drawing parallels to AlphaZero redefining strategy games. - The POC-to-production gap and why AI projects stall. - Failure modes, hallucinations, and governance risks—and how to mitigate them. - The disconnect between executive optimism and employee workload. Hugo also mentions his upcoming workshop on escaping Proof-of-Concept Purgatory, which has evolved into a Maven course "Building LLM Applications for Data Scientists and Software Engineers" launching in January (https://maven.com/hugo-stefan/building-llm-apps-ds-and-swe-from-first-principles?utm_campaign=8123d0&utm_medium=partner&utm_source=instructor). Vanishing Gradient listeners can get 25% off the course (use the code VG25), with $1,000 in Modal compute credits included. A huge thanks to Dave Scharbach and the Toronto Machine Learning Society for organizing the conference and to the audience for their thoughtful questions. As we head into the new year, this conversation offers a reality check amidst the growing AI agent hype. LINKS Hugo on twitter (https://x.com/hugobowne) Hugo on LinkedIn (https://www.linkedin.com/in/hugo-bowne-anderson-045939a5/) Vanishing Gradients on twitter (https://x.com/vanishingdata) "Building LLM Applications for Data Scientists and Software Engineers" course (https://maven.com/hugo-stefan/building-llm-apps-ds-and-swe-from-first-principles?utm_campaign=8123d0&utm_medium=partner&utm_source=instructor).

Hot Girls Code
59. Exploring Tech Jobs: Data Engineer

Hot Girls Code

Play Episode Listen Later Dec 3, 2024 28:00


We are back with another installment of our Exploring Tech Job Series where we dig into different technical roles to provide insight into what that role is all about. This week we are chatting about Data Engineering and we are joined by a real-life Data Engineer, Rosie! Join us as we chat about what a Data Engineer is, the day-to-day of the job, how you might get into Data Engineering, and so much more. New episodes come out fortnightly on Wednesday morning (NZT). Check out Datacamp to learn more about data engineering Where to Find Us: ⁠ ⁠Instagram⁠ ⁠Tik Tok⁠ ⁠The Hot Girls Code Website⁠⁠ Sponsored by: ⁠Trade Me Jobs⁠

What's New In Data
From Marine Aircraft to Data Engineering: Navigating Social Media Shifts and AI Innovation with Alex Noonan

What's New In Data

Play Episode Listen Later Nov 21, 2024 39:58 Transcription Available


Discover how Alex Noonan transitioned from the flight deck of a Marine aircraft to the intricate world of data engineering. His unique journey, enriched by a stint in finance, gives us a firsthand view of the diverse backgrounds shaping the data industry. As Alex recounts his experiences, we explore the vibrant community he found on data Twitter, a realm buzzing with shared insights and collaborative spirit. However, the landscape shifted following Elon Musk's takeover of Twitter, leading to content fragmentation and a migration towards emerging platforms like Blue Sky. Join us as Alex discusses how these changes have impacted the cohesion and knowledge-sharing dynamics within the data community.Navigate the complex world of professional networking with tips from Alex, as he breaks down the strategic use of platforms like LinkedIn, Reddit, and Hacker News for data professionals. Learn how to creatively tailor your content to fit the quirks of each platform's algorithm, and prepare to engage with varied audiences. The conversation also highlights the transformative potential of AI tools in elevating data processes, reducing mundane tasks, and fostering high-value work. Discover innovations like Dagster and its role as an orchestrator, integrating key business intelligence tools to streamline the data engineer's experience. This episode is a must-listen for anyone intrigued by the evolving interplay of technology, social media, and the power of community.Follow Alex on:Linkedin Twitter BlueskyDagsterWhat's New In Data is a data thought leadership series hosted by John Kutay who leads data and products at Striim. What's New In Data hosts industry practitioners to discuss latest trends, common patterns for real world data patterns, and analytics success stories.

Tcast
TARTLE PIN: Why It's Crucial & How to Protect It

Tcast

Play Episode Listen Later Nov 18, 2024 1:23


This episode of Tartlecast emphasizes the critical importance of remembering your TARTLE pin. We explain why your pin is essential for protecting your data and accessing your earnings, and we offer tips for creating a secure and memorable pin. We also address the common issue of forgotten pins and why TARTLE cannot retrieve or reset them for users. Key Takeaways: Your TARTLE pin is the key to accessing your account and protecting your data.     TARTLE cannot access or reset your pin due to security and privacy protocols.     Choose a pin that is both secure and memorable to avoid losing access to your account.      If you forget your pin, there is currently no way to recover it. You will need to create a new account.     TCAST is a tech and data podcast co-hosted by Alexander McCaig and Jason Rigby. The podcast delves into the latest trends in Big Data, Artificial Intelligence, and Humanity, examining the evolving landscape of digital transformation and innovation.   The show features interviews with data scientists, thought leaders, and industry experts at the forefront of technological advancement for human progress. Listeners can explore a wide range of TCAST episodes on their preferred podcast platforms.   Connect with TCAST: Website: https://tartle.co/ Facebook: https://go.tartle.co/fb Instagram: https://go.tartle.co/ig Twitter: https://go.tartle.co/tweet   What's your data worth? Find out at https://tartle.co/  

Tcast
How to Read Your TARTLE Wallet Balance

Tcast

Play Episode Listen Later Nov 17, 2024 2:34


We've noticed some confusion about how earnings are displayed on TARTLE.  Let's clear things up!  Our latest Tartlecast episode explains the correct way to read your wallet balance and why some data elements have a value of less than a penny. TCAST is a tech and data podcast co-hosted by Alexander McCaig and Jason Rigby. The podcast delves into the latest trends in Big Data, Artificial Intelligence, and Humanity, examining the evolving landscape of digital transformation and innovation.   The show features interviews with data scientists, thought leaders, and industry experts at the forefront of technological advancement for human progress. Listeners can explore a wide range of TCAST episodes on their preferred podcast platforms.   Connect with TCAST: Website: https://tartle.co/ Facebook: https://go.tartle.co/fb Instagram: https://go.tartle.co/ig Twitter: https://go.tartle.co/tweet   What's your data worth? Find out at https://tartle.co/

Tcast
Understanding TARTLE Rewards: Data Packets vs. Profile Actions

Tcast

Play Episode Listen Later Nov 16, 2024 2:11


This Tartlecast episode clarifies the difference between the small rewards given for completing actions on TARTLE (like creating a wallet or publishing a data packet) and the larger earnings that come from bids placed on your data packets. We explain why TARTLE offers these small rewards and how they contribute to the overall goal of creating a universal basic income. TCAST is a tech and data podcast co-hosted by Alexander McCaig and Jason Rigby. The podcast delves into the latest trends in Big Data, Artificial Intelligence, and Humanity, examining the evolving landscape of digital transformation and innovation.   The show features interviews with data scientists, thought leaders, and industry experts at the forefront of technological advancement for human progress. Listeners can explore a wide range of TCAST episodes on their preferred podcast platforms.   Connect with TCAST: Website: https://tartle.co/ Facebook: https://go.tartle.co/fb Instagram: https://go.tartle.co/ig Twitter: https://go.tartle.co/tweet   What's your data worth? Find out at https://tartle.co/

Tcast
What Happens to Your TARTLE Data Packets After you hit Publish?

Tcast

Play Episode Listen Later Nov 15, 2024 2:32


In this episode of Tartlecast, we address a common question among TARTLE users: "What happens to my data packet after I publish it?" We clarify the process, explaining that published data packets are securely stored in the user's data vault and are not sold, moved, or transferred until a bid is placed on them. We also discuss the importance of completing data packets and building a robust data vault to attract potential buyers.     Key Takeaways: Published data packets are NOT immediately sold. They are stored in your secure data vault.     TARTLE does not have access to your data vault. Only YOU have the keys.     Once a bid is placed on your data packet, you will be notified and can choose to accept or decline the offer.     Completing more data packets increases your visibility to buyers and your earning potential.     TCAST is a tech and data podcast co-hosted by Alexander McCaig and Jason Rigby. The podcast delves into the latest trends in Big Data, Artificial Intelligence, and Humanity, examining the evolving landscape of digital transformation and innovation.   The show features interviews with data scientists, thought leaders, and industry experts at the forefront of technological advancement for human progress. Listeners can explore a wide range of TCAST episodes on their preferred podcast platforms.   Connect with TCAST: Website: https://tartle.co/ Facebook: https://go.tartle.co/fb Instagram: https://go.tartle.co/ig Twitter: https://go.tartle.co/tweet   What's your data worth? Find out at https://tartle.co/

Tcast
No Bids on Your TARTLE Data Packets? We've Got Answers!

Tcast

Play Episode Listen Later Nov 14, 2024 2:21


We've heard your questions, and we're here to help!  Our latest Tartlecast episode tackles the common question of why some data packets don't receive bids.  Learn how to make your data more appealing to buyers and increase your earning potential on TARTLE. TCAST is a tech and data podcast co-hosted by Alexander McCaig and Jason Rigby. The podcast delves into the latest trends in Big Data, Artificial Intelligence, and Humanity, examining the evolving landscape of digital transformation and innovation.   The show features interviews with data scientists, thought leaders, and industry experts at the forefront of technological advancement for human progress. Listeners can explore a wide range of TCAST episodes on their preferred podcast platforms.   Connect with TCAST: Website: https://tartle.co/ Facebook: https://go.tartle.co/fb Instagram: https://go.tartle.co/ig Twitter: https://go.tartle.co/tweet   What's your data worth? Find out at https://tartle.co/

DealMakers
Viraj Parekh On Raising $290 Million To Allow Companies To Build Powerful Workflow Orchestrations For Data Engineering And Data Science

DealMakers

Play Episode Listen Later Nov 9, 2024 23:32


In a world driven by data, it takes a visionary to see beyond the noise and build something truly scalable. Viraj Parekh, co-founder of Astronomer, didn't have a clear roadmap. His journey from childhood curiosity in technology to becoming a key figure in data orchestration is fascinating. Astronomer has attracted funding from top-tier investors like Venrock, Bain Capital Ventures, Insight Partners, and Sutter Hill Ventures.