Pipeline Conversations

Learn how leading companies like Dropbox, NVIDIA, and Slack tackle LLM security in production. This comprehensive guide covers practical strategies for preventing prompt injection, securing RAG systems, and implementing multi-layered defenses, based on real-world case studies from the LLMOps database. Discover battle-tested approaches to input validation, data privacy, and monitoring for building secure AI applications. Please read the full blog post here (https://www.zenml.io/blog/production-llm-security-real-world-strategies-from-industry-leaders) and the associated LLMOps database entries here (https://zenml.io/llmops-database).

In this episode, we dive deep into the world of LLM optimization and cost management - a critical challenge facing AI teams today. Join us as we explore real-world strategies from companies like Dropbox, Meta, and Replit who are pushing the boundaries of what's possible with large language models. From clever model selection techniques and knowledge distillation to advanced inference optimization and cost-saving strategies, we'll unpack the tools and approaches that are helping organizations squeeze maximum value from their LLM deployments. Whether you're dealing with runaway API costs, struggling with inference latency, or looking to optimize your model infrastructure, this episode provides practical insights that you can apply to your own AI initiatives. Perfect for ML engineers, technical leads, and anyone responsible for maintaining LLM systems in production. Please read the full blog post here (https://www.zenml.io/blog/optimizing-llm-performance-and-cost-squeezing-every-drop-of-value) and the associated LLMOps database entries here (https://zenml.io/llmops-database).

A comprehensive exploration of real-world lessons in LLM evaluation and quality assurance, examining how industry leaders tackle the challenges of assessing language models in production. Through diverse case studies, we cover the transition from traditional ML evaluation, establishing clear metrics, combining automated and human evaluation strategies, and implementing continuous improvement cycles to ensure reliable LLM applications at scale. Please read the full blog post here (https://www.zenml.io/blog/the-evaluation-playbook-making-llms-production-ready) and the associated LLMOps database entries here (https://zenml.io/llmops-database).

Prompt engineering is the art and science of crafting instructions that unlock the potential of large language models (LLMs). It's a critical skill for anyone working with LLMs, whether you're building cutting-edge applications or conducting fundamental research. But what does effective prompt engineering look like in practice, and how can we systematically improve our prompts over time? To answer these questions, we've distilled key insights and techniques from a collection of LLMOps case studies spanning diverse industries and applications. From designing robust prompts to iterative refinement, optimization strategies to management infrastructure, these battle-tested lessons provide a roadmap for prompt engineering mastery. Please read the full blog post here (https://www.zenml.io/blog/prompt-engineering-management-in-production-practical-lessons-from-the-llmops-database) and the associated LLMOps database entries here (https://zenml.io/llmops-database).

An in-depth exploration of LLM agents in production environments, covering key architectures, practical challenges, and best practices. Drawing from real-world case studies, this article examines the current state of AI agent deployment, infrastructure requirements, and critical considerations for organizations looking to implement these systems safely and effectively. Please read the full blog post here (https://www.zenml.io/blog/llm-agents-in-production-architectures-challenges-and-best-practices) and the associated LLMOps database entries here (https://zenml.io/llmops-database).

Discover how embeddings power modern search and recommendation systems with LLMs, using case studies from the LLMOps Database. From RAG systems to personalized recommendations, learn key strategies and best practices for building intelligent applications that truly understand user intent and deliver relevant results. Please read the full blog post here (https://www.zenml.io/blog/building-advanced-search-retrieval-and-recommendation-systems-with-llms) and the associated LLMOps database entries here (https://zenml.io/llmops-database).

Explore real-world applications of Retrieval Augmented Generation (RAG) through case studies from leading companies. Learn how RAG enhances LLM applications with external knowledge sources, examining implementation strategies, challenges, and best practices for building more accurate and informed AI systems. Please read the full blog post here (www.zenml.io/blog/building-llm-applications-that-know-what-theyre-talking-about) and the associated LLMOps database entries here (https://zenml.io/llmops-database).

The LLMOps Database offers a curated collection of 300+ real-world generative AI implementations, providing technical teams with practical insights into successful LLM deployments. This searchable resource includes detailed case studies, architectural decisions, and AI-generated summaries of technical presentations to help bridge the gap between demos and production systems. Please read the full blog post here (https://www.zenml.io/blog/demystifying-llmops-a-practical-database-of-real-world-generative-ai-implementations) and the associated database entries here (https://zenml.io/llmops-database).

This week I spoke with Daniel van Strien, a digital curator working at the British Library. Daniel has worked on a number of projects at the intersection of archives, libraries and machine learning and I was really happy to have the chance to get to unpack some of the ways he's finding to apply these techniques and tools. In particular, I found it interesting how important the annotation process is as part of many overall workflows, as well as how simple out-of-the-box techniques like image classification using a fine-tuned model could satisfy many low-hanging fruit-type use cases. Special Guest: Daniel van Strien.

This week I spoke with Lak Lakhshmanan, who worked for years at Google on ML and AI projects and products at a senior level and he also brings years of experience working on meteorology and other scientific projects previously. Lak brings a ton of experience to the table and it was interesting to hear his suggestions around when it is and isn't appropriate to bring the full set of MLOps tools to the table, for example. We also discussed the fundamentals of doing ML-backed projects as well as the teams needed to make those projects succeed. Special Guest: Lak Lakshmanan.

This week I spoke with Charles Frye. Not only has Charles volunteered to be a judge on our Month of MLOps competition happening right now, he's part of the core team working on the Full Stack Deep Learning course. Naturally, we get into education for practitioners as well as the things that Charles has seen in his own prior background working on production use cases. We also discuss the ways that tooling to support education as well as productive machine learning can and is being improved. Special Guest: Charles Frye.

In today's conversation, I'm speaking with Goku Mohandas, founder and creator of the amazing online resource MadeWithML (https://madewithml.com/). Goku has a bunch of practical experience, from working with Apple to a startup in the oncology space and much more. In this conversation we continued to unpack the theme of education in ML, the challenges when it comes to working across the full stack of ML applications, and what he's seen work in his experience working on MadeWithML (https://madewithml.com/). We also discuss some of the patterns he's seen in the production stacks he's seen in his experience consulting with various ML teams as well as where he sees room for improvement in the abstractions that we all rely on to do our work. Goku has generously agreed to be an external judge for our Month of MLOps competition that starts on October 10. If you haven't signed up yet, or want to learn more, please visit zenml.io/competition (https://zenml.io/competition). Special Guest: Goku Mohandas.

So excited to be able to announce our

This week I spoke with Eric Landau, co-founder of Encord, a platform for data-centric computer vision. This podcast contains a lot of geekery about annotation, and even though Encord aren't an annotation tool per se, Eric and his team have tackled a bunch of quite complicated problems relating to that domain. We also discuss the much-used term 'data-centric AI' and consider where it's useful and where perhaps there's a little bit of hype. We also get into some of the technical tradeoffs and decisions that come when building a platform. I'm really excited to get to present this episode to you today as I really enjoyed the discussion. Special Guest: Eric Landau.

This week we dive into the abstractions that we're all trying to layer on top of the core ML processes and workflows. I spoke with Phil Howes, co-founder and chief scientist at BaseTen. BaseTen is a platform that allows data scientists to go from an initial model to an MVP web app quickly. We got into some of the big challenges he had working to build out the platform, as well as the core issue of iteration speed that motivates why they're building BaseTen. Phil has experienced quite a few of the industry's end-to-end patterns in the years that he's been working on machine learning and it was great to have that context inform the conversation, too. Special Guest: Phil Howes.

This week I spoke with Savin Goyal and Hugo Bowne-Anderson from Outerbounds. They both work on leading, building and helping people put models into production through Metaflow, and I'm sure current users of ZenML will find this conversation interesting to hear how they think through the broader questions and engineering problems involved with MLOps. Above all, we spoke about the challenges involved in building a tool that handles the whole machine learning story, from collecting data to training models, to deployment and back again. In many ways it's great that there are lots of smart people thinking about this really hard problem, and even though it is by no means 'solved' conversations like this make me feel cautiously optimistic about the space. Special Guests: Hugo Bowne-Anderson and Savin Goyal.

This week I spoke with Mateo Rojas-Carulla, the CTO and a co-founder of Lakera (https://www.lakera.ai/) and Matthias Kraft, also a co-founder and the CPO there. Lakera (https://www.lakera.ai/) is an AI safety company that does a lot of work in the computer vision domain, building a platform and tools for users to gain more confidence in the output and functionality of their models. We discuss how they think about the testing of machine learning models, and about how having this safety element upfront has implications for how you go about the testing and ensuring robustness. We specifically dive into how to go about testing computer vision models and the various pitfalls that are to be found in that domain. Special Guests: Mateo Rojas-Carulla and Matthias Kraft.

This week I spoke with Robin Cole, a senior data scientist at Satellite Vu (https://www.satellitevu.com), a company that's about to launch a thermal imaging satellite into space in order to provide new ways of seeing the earth from above. Robin generously took the time to discuss his day to day work involving satellite data, the stack they work with at Satellite Vu as well as some of the difficulties that come up in the domain. We also discuss the extremely popular satellite-image-deep-learning GitHub repo (https://github.com/robmarkcole/satellite-image-deep-learning) that presents resources for those working with or seeking to learn about this kind of data. Special Guest: Robin Cole.

This week on the podcast I spoke with Gerard Kruisheer, the CTO and co-founder of Captain AI (https://www.captainai.com/), a company based in the Netherlands working on autonomous shipping out of the busy Rotterdam port. We discussed the unique problems that come with building autonomous vehicles, the extent to which the latest and greatest research informs their work, their production stack and how they handle deployment for their particular setup. As always please let us know if you have guests you'd like me to speak to by sending a message to us on slack or by emailing podcast@zenml.io (podcast@zenml.io). Special Guest: Gerard Kruisheer.

I'll be having some conversations with the people behind the tools that ZenML offers as integrations. We spoke with Ben Wilson a few weeks back, and today I'm pleased to publish this conversation with Emeli Dral, co-founder and CTO of Evidently, an open-source tool tackling the problem of monitoring of models and data for machine learning. We discussed the challenges around building a tool that is both straightforward to use while also customisable and powerful. We also got into the thinking behind how they grew their community and blog along the way. Special Guest: Emeli Dral.

This week I spoke with Karthik Kannan, cofounder and CTO of Envision (https://www.letsenvision.com/), a company that builds on top of the Google Glass and using Augmented Reality features of phones to allow visually impaired people to better sense the environment or objects around them. Their software and devices are pretty popular and as you'll hear in this conversation, they've been on a real journey to get to where they are now. In particular, I really enjoyed the parts where Karthik explained their development and deployment process in detail. It's not too often that you get a deep dive into the workflows and stacks of an embedded computer vision company and tool and so I think you're going to really enjoy this one. Special Guest: Karthik Kannan.

In this episode, I'm really happy to be able to continue the dialogue we've been having with our users and community around the role of data annotation and labeling in MLOps. We were lucky to get to talk to Iva Gumnishka (https://www.linkedin.com/in/ivagumnishka/), the founder of Humans in the Loop (https://humansintheloop.org/). They are an organisation that provides data annotation and collection services. Their teams are primarily made up of those who have been affected by conflict and now are asylum seekers or refugees. Iva has a ton of experience working with annotation and has seen how different companies build this into their production machine learning lifecycles. We're continuing to work on a feature that will allow you to do this as part of your MLOps workflow when using ZenML, and I welcome any feedback you might have on the back of this podcast or the articles we've been publishing on the ZenML blog. Special Guest: Iva Gumnishka.

We took a few weeks break to reach out to some new guests and so I think we can go so far as declaring this next series of episodes as season 2 of Pipeline Conversations. Today, I'm extremely excited to present this conversation I had with Ben Wilson who works over at Databricks and who has also just released a new book called 'Machine Learning Engineering in Action (https://www.manning.com/books/machine-learning-engineering-in-action)'. It's a jam-backed guide to all the lessons that Ben has learned over his years working to help companies get models out into the world and run them in production. I was really lucky to get to talk to Ben about his new book and also about the mental models he thinks are useful to bring to bear on this complicated problem many of us are working on. Special Guest: Ben Wilson.

Adam and Hamza return for a short discussion of what we've been busy working on during the previous few months, where we're going with ZenML and why it's so amazing to be building an open-source tool.

I enthusiastically read Kush Varshney's book when it was released for free to the world several months back. Trustworthy Machine Learning (http://www.trustworthymachinelearning.com/) is a concise and clear overview of many of the ways that machine learning can go wrong, and so I was especially keen to get Kush (http://krvarshney.github.io/) on to talk more about his work and research. I also got a stronger sense of appreciation for how good MLOps practices and workflows offered a clear path to ensuring that your machine learning models and behaviours could become more trustworthy. Kush has done a lot of interesting work, particularly with the AI Fairness 360 (https://ai-fairness-360.org/) and AI Explainability 360 (https://ai-explainability-360.org/) toolkits that I'm sure listeners of this podcast would find worth checking out. Special Guest: Kush Varshney.

This week I spoke with Matt Squire, the CTO and co-founder of Fuzzy Labs (https://www.fuzzylabs.ai), where they help partner organisations think through how best to productionise their machine learning workflows. Matt and FuzzyLabs are also behind the Awesome Open Source MLOps (https://github.com/fuzzylabs/awesome-open-mlops) GitHub repo where you can find all the options for an open-source MLOps stack of your dreams. Matt has been an enthusiastic early supporter of the work we do at ZenML so it was really amazing to get to talk to him and get his take based on the many experiences he's had seeing how ML is done out in the field. Special Guest: Matt Squire.

This week I spoke with Emmanuel Ameisen, a data scientist and ML engineer currently based at Stripe. Emmanuel also wrote an excellent O'Reilly book called "Building Machine Learning Powered Applications", a book I find myself often returning to for inspiration and that I was pleased to get the chance to reread in preparation for our discussion. Emmanuel has previously worked at Insight Data Science where he was involved in mentoring and guiding dozens of data scientists who were working on building their ML portfolio projects. He brings a wealth of experience to the table and I'm really excited to present our conversation to you. Special Guest: Emmanuel Ameisen.

This week I spoke with Johnny Greco (https://johnnygreco.space), a data scientist working at Radiology Partners. Johnny transitioned into his current work from a career as an academic — working in astronomy — where also worked in the open-source space to build a really interesting synthetic image data project. We get into that project in our conversation but we also discuss his experience of crossing over into industry, the skills that have served him in his new job, and his experience of working in a world where the stakes around models in production are much higher. Special Guest: Johnny Greco.

This week I spoke with Tristan Zajonc (https://www.linkedin.com/in/tristanzajonc/), the CEO and cofounder of Continual (https://continual.ai/), a company that provides an AI layer for enterprise companies or, as we'll get into in the podcast, the so-called 'modern data stack'. He previously worked at Cloudera as a CTO for machine learning and as the head of the data science platform there, and he holds a PhD in public policy from Harvard University. In our conversation we discussed the different levels of abstraction one can take when dealing with the MLOps problem. We spoke about all the different ways that machine learning can fail in production settings and of course we discussed the concept of the 'modern data stack' and what that means. Special Guest: Tristan Zajonc.

Our guest this week was Mohan Mahadevan, a senior VP at Onfido, a machine-learning powered identity verification platform. He has previously worked at Amazon heading up a computer vision team working on robotics applications as well as for many years at KLA, a leading semiconductor hardware company. He holds a doctorate in theoretical physics from Colorado State University. Mohan had mentioned that he thought it might be interesting to discuss neurosymbolic AI, and the implications of a shift towards that as a core paradigm for production AI systems. In particular, we discuss the practical consequences of such a shift, both in terms of team composition as well as infrastructure requirements. Special Guest: Mohan Mahadevan.

Our guest this week is Ines Montani, co-founder and CEO of Explosion, a company based out of Berlin that produce tools that you probably know and love like Spacy, a Python Natural Language Processing library and Prodigy, a data annotation tool. I've always found Ines to be personally inspiring in the work that she and her team produce as well as how they present themselves to the world, so it was a real pleasure to get to dive into the weeds as to exactly how that happens. We also discuss how NLP works in production, what reproducibility means for ML projects and much more. Special Guest: Ines Montani.

This week, we spoke with Danny Leybzon, currently working with WhyLabs to help data scientists monitor their models in production and prevent model performance from degrading. He previously worked as a kind of roving data scientist and engineer, helping companies put their models into production. As such, we had a really interesting discussion of some of the ways that tooling and the general context for data science sometimes lets practitioners down, And of course we also discussed why monitoring and logging is actually a kind of baseline practice that should be part of any and every data scientist's toolkit. Luckily for us, Danny added in a bunch of examples from his wide experience doing all this in the real world. Special Guest: Danny Leybzon.

Noah Gift is the founder of Pragmatic A.I. Labs and author of 'Practical MLOps'. We discuss the role of MLOps in an organisation, some deployment war stories from his career as well as what he considers to be 'best practices' in production machine learning. Read the summary blogpost (https://blog.zenml.io/practical-mlops-noah-gift/) on the ZenML blog. Special Guest: Noah Gift.

Adam and Hamza introduce themselves for the first episode of Pipeline Conversations. They discuss the world of MLOps, where ZenML sits within this space, and why it's such a complicated problem to solve.