Podcasts about istio

In this episode of The Tech Trek, Amir sits down with Matt Moore, CTO and co-founder of Chainguard, to explore the escalating importance of software supply chain security. From Chainguard's origin story at Google to the systemic risks enterprises face when consuming open source, Matt shares the lessons, best practices, and technical innovations that help make open source software safer and more reliable. The conversation also touches on AI's impact on the attack surface, mitigating threats with engineering rigor, and why avoiding long-lived credentials could be your best defense.

In deze aflevering spreken Ronald Kers (CNCF Ambassador) en Jan Stomphorst (Solutions Architect bij ACC ICT) met Jens Kat, Dev Engineer & Product Owner bij ING. Al acht jaar werkt Jens aan de interne service mesh van ING – een essentieel platform dat inmiddels zo'n 10.000 workloads ondersteunt.We duiken in de evolutie van deze mesh: van eigen tooling op virtuele machines tot aan een sidecar-gebaseerde mesh die naadloos integreert met Kubernetes. Jens legt uit waarom ING vanaf het begin TLS en mutual TLS heeft toegepast, en waarom ze kozen voor een eigen oplossing in plaats van Istio of Linkerd. Ook gaat het gesprek over developer onboarding, community building binnen een grote organisatie, en hoe data uit productie wordt gebruikt om features te verbeteren of juist af te bouwen.Een aflevering vol praktische lessen over schaalbaarheid, security, en het behouden van controle in een complexe microservices-wereld.Stuur ons een bericht.Support the showLike and subscribe! It helps out a lot.You can also find us on:De Nederlandse Kubernetes Podcast - YouTubeNederlandse Kubernetes Podcast (@k8spodcast.nl) | TikTokDe Nederlandse Kubernetes PodcastWhere can you meet us:EventsThis Podcast is powered by:ACC ICT - IT-Continuïteit voor Bedrijfskritische Applicaties | ACC ICT

Without this, developers waste time managing infrastructure instead of focusing on code. VMware addresses this with VCF, a pre-integrated Kubernetes solution that includes components like Harbor, Valero, and Istio, all managed by VMware. While some worry about added complexity from abstraction, Turner dismissed concerns about virtualization overhead, pointing to benchmarks showing 98.3% of bare metal performance for virtualized AI workloads. He emphasized that AI is driving nearly half of Kubernetes deployments, prompting VMware's partnership with Nvidia to support GPU virtualization. Turner also highlighted VMware's open source leadership, contributing to major projects and ensuring Kubernetes remains cloud-independent and standards-based. VMware aims to simplify Kubernetes and AI workload management while staying committed to the open ecosystem.Learn more from The New Stack about the latest insights with VMware Has VMware Finally Caught Up With Kubernetes?VMware's Golden PathJoin our community of newsletter subscribers to stay on top of the news and at the top of your game.  

Nina Polshakova is a software engineer at Solo.io, where she's worked on Istio and API Gateway projects. She's been part of the Kubernetes release team since v1.27 and is currently serving as the Release Lead for v1.33.   Do you have something cool to share? Some questions? Let us know: - web: kubernetespodcast.com - mail: kubernetespodcast@google.com - twitter: @kubernetespod - bluesky: @kubernetespodcast.com   News of the week 229 new things Google announced at Next 25 MCO: Multi-Cluster Orchestrator Golden Kubestronaut Cloud Native Platform Engineering Associate The kube-scheduler-simulator K0s and k0smotron are now CNCF Sandbox projects   Links from the interview Nina Polshakova Kubernetes Deprecation Policy Kubernetes Dev Google Group solo.io Istio API Gateway (General concept, linking to K8s Gateway API) Kubernetes Release Team GitHub Istio revisions Working in Public by Nadia Eghbal (Link to publisher's site about the book) Kubernetes Maintainers Read Mean Comments (KubeCon EU 2024) Kubernetes 1.33 release blog (Link to release announcement blog) Kubernetes Enhancement Proposals (KEPs) Sidecar Containers Multiple Service CIDR support (KEP link) Dynamic Resource Allocation (DRA) DRA support for partitioned devices (KEP link) DRA device taints and tolerations (KEP link) DRA: Prioritized Alternatives in Device Requests (KEP link) Kubernetes 1.33 sneak peak (Link to pre-release highlights) EndpointSlices API Kubernetes Gateway API node.status.nodeInfo.kubeProxyVersion is a lie (issue) KEP-4004: Deprecate the kubeProxyVersion field of v1.Node #4005 (KEP link) Kubelet Removal: Host network support for Windows pods (KEP link) Containerd SIG Windows HostProcess Containers (Windows) Removal: KEP-5040: Disable git_repo volume driver (KEP link) User Namespaces (Beta, Enabled by Default) CRI-O Runc In-place Resource Resize for Pods (Link to the alpha announcement, but now beta) Vertical Pod Autoscaler (VPA) KEP-5080: Ordered Namespace Deletion PyTorch Linkerd Terry Pratchett's Discworld series Tiffany Aching series Guards! Guards! Going Postal Kubernetes Slack New Contributor Orientation  

With Kubecon coming up next week, we speak to Lukas Gentele, co-founder and CEO at Loft Labs, about virtualizing K8sSHOW: 909SHOW TRANSCRIPT: The Cloudcast #909 TranscriptSHOW VIDEO: https://youtube.com/@TheCloudcastNET CLOUD NEWS OF THE WEEK - http://bit.ly/cloudcast-cnotwNEW TO CLOUD? CHECK OUT OUR OTHER PODCAST - "CLOUDCAST BASICS" SPONSORS:Try Postman AI Agent Builder Todaypostman.com/podcast/cloudcast/SHOW NOTES:Loft Labs websiteLoft Labs on TechCrunchLoft Labs vCluster CloudTopic 1 - Welcome to the show, Lukas. Give everyone a quick introduction.Topic 2 - Our topic today is virtualizing Kubernetes. Let's get the most obvious question out of the way… Why virtualize k8s? Isn't this another abstraction layer to manage and more complexity in the stack?Topic 3 - What are the most common use cases? Combining test/dev and production? Topic 4 - How does this impact other parts of the stack? I think about Istio, Rancher, etc. Does the complexity increase or decrease?Topic 4a - How is the control plane handled vs. the data plane?Topic 5 - With vm virtualization, a trend developed as the technology matured. In the beginning, consolidation was good, and as the technology supported greater and greater density, a tipping point was reached where fault domains were needed. Where is the virtualization of K8s on this scale?Topic 6 - A few months ago at KubeCon in Salt Lake City, you announced vCluster Cloud. Are there any hints for our listeners for KubeCon EU?Topic 7 - If anyone is interested, what's the best way to get started?FEEDBACK?Email: show at the cloudcast dot netBluesky: @cloudcastpod.bsky.socialTwitter/X: @cloudcastpodInstagram: @cloudcastpodTikTok: @cloudcastpodDrunk AgileDan Vacanti and Prateek Singh drink whisk(e)y and discuss various facets of agile...Listen on: Apple Podcasts Spotify

We cover how Istio Ambient Mesh eliminates sidecar proxies to significantly reduce Kubernetes resource consumption. This episode covers the architectural differences between traditional service mesh and ambient mesh, practical migration strategies for different workload types, key metrics for measuring performance improvements, and real-world operational benefits like simplified troubleshooting and easier version upgrades.

In this episode, Henrik Blixt, a product manager at Intuit and Argo maintainer, shares his experiences and insights into managing platform engineering teams that handle Kubernetes, service mesh, API gateways, and more. He emphasizes the importance of product management within platform engineering and discusses his involvement with the CNCF's end user technical advisory board. Henrik also highlights the significance of open source in his professional journey and details the ongoing initiatives and advancements within the Argo project.   00:00 Introduction and Guest Welcome 00:53 Discussion on Argo and Developer Tools 01:41 Open Source Community Involvement 02:06 CNCF End User Technical Advisory Board 03:11 Reference Architectures and Initiatives 08:18 Challenges and Solutions for End Users 13:20 Argo Project Insights 16:03 The Importance of Product Management 17:16 Conclusion and Final Thoughts   Guest: Henrik Blixt leads a Product Management team responsible for the Intuit core platform, where he defines the strategy and direction that has shaped Intuit's cloud native platform based on CNCF projects like Kubernetes, Envoy, Istio, Prometheus, Argo (and many more!) that's used by 7000 developers and serving over 100M users. Being a passionate member of the open source community for almost 30 years, from Linux through OpenStack and Kubernetes, Henrik is currently focused on the Argo project as a core maintainer. He also represents Intuit across other committees, like the CNOE project and the broader Linux Foundation, where he shares experiences and best practices from Intuit's use of open source, making sure end users are heard and their pain points understood. He loves engaging with the community and has been a prolific speaker and event program committee member across ArgoCon, GitOpsCon, Kubecon over the years. A native of Sweden, earning his B.Sc in information systems from the University of Gothenburg, he now resides in California with his family.    

Jetstack's cert-manager, a leading open-source project in Kubernetes certificate management, began as a job interview challenge. Co-founder Matt Barker recalls asking a prospective engineer to automate Let's Encrypt within Kubernetes. By Monday, the candidate had created kube-lego, which evolved into cert-manager, now downloaded over 500 million times monthly.Cert-manager's journey to CNCF graduation, achieved in September, began with its donation to the foundation four years ago. Relaunched as cert-manager, the project grew under engineer James Munnelly, becoming the de facto standard for certificate lifecycle management. The thriving community and ecosystem around cert-manager highlighted its suitability for CNCF stewardship. However, maintainers, including Ashley Davis, noted challenges in navigating differing opinions within its vast user base.With graduation achieved, cert-manager's roadmap includes sub-projects like trust-manager, addressing TLS trust bundle management and Istio integration. Barker aims to streamline enterprise-scale deployments and educate security teams on cert-manager's impact. Cert-manager has become integral to cloud-native workflows, promising to simplify hybrid, multicloud, and edge deployments.Learn more from The New Stack about cert-manager:Jetstack's cert-manager Joins the CNCF Sandbox of Cloud Native TechnologiesJetstack Secure Promises to Ease Kubernetes TLS SecurityJoin our community of newsletter subscribers to stay on top of the news and at the top of your game. 

In deze aflevering van De Nederlandse Kubernetes Podcast spreken hosts Ronald Kers en Jan Stomphorst met René Schoonrok (IT Engineering Manager) en Jeroen van Bijnen (Product Owner) van Nationale Nederlanden. Het gesprek duikt diep in de strategie en toekomst van het containerplatform binnen Nationale Nederlanden, waarbij onderwerpen als multicloud-omgevingen, DevSecOps, en nieuwe technologieën als GitOps aan bod komen.Hoofdpunten van de aflevering:Het Container Platform en Multicloud-aanpak: René en Jeroen leggen de uit hoe Nationale Nederlanden werkt met een containerplatform dat flexibel inzetbaar is in een multicloud-omgeving, en welke voordelen en uitdagingen dit met zich meebrengt vergeleken met cloud-provider-specifieke oplossingen.Extra Diensten en DevSecOps: Naast het containerplatform biedt hun team cruciale aanvullende services, zoals DevSecOps, algemene IT-taken en integratie met Azure DevOps. Deze services ondersteunen de ontwikkelteams en zorgen voor beveiliging en efficiëntie in de processen.Innovaties en de Toekomst van het Platform: Ze bespreken spannende plannen, waaronder de lancering van een Machine Learning-platform op Kubernetes met GitOps en ArgoCD, ondersteund door tools zoals Knative en Istio. Ook wordt er gekeken naar serverless opties op Kubernetes om flexibiliteit verder te vergroten.Toekomstvisie van Nationale Nederlanden: Het Future Ready-programma van Nationale Nederlanden krijgt aandacht.Aflevering 52: Autoscaling Magic with KEDA | De Nederlandse Kubernetes PodcastStuur ons een bericht.Like and subscribe! It helps out a lot.You can also find us on:De Nederlandse Kubernetes Podcast - YouTubeNederlandse Kubernetes Podcast (@k8spodcast.nl) | TikTokDe Nederlandse Kubernetes PodcastWhere can you meet us:EventsThis Podcast is powered by:ACC ICT - IT-Continuïteit voor Bedrijfskritische Applicaties | ACC ICT

Próximo episódio do LowOpsCast contará com a presença de Pedro Ignácio (https://www.linkedin.com/in/pedroirufo/), engenheiro de soluções no Itaú Unibanco e Microsoft MVP em Cloud Security. Com uma sólida experiência em cibersegurança, design de sistemas e computação em nuvem, Pedro tem se destacado na construção de soluções de alto desempenho utilizando Kubernetes, Istio, Terraform,. Neste episódio, vamos nos aprofundar na preparação de Pedro para a KubeCon, o maior evento de Kubernetes e tecnologias cloud native do mundo. Durante nossa conversa, Pedro compartilha: * Sua trajetória profissional: Como suas experiências anteriores o levaram a este ponto da carreira e como a participação na KubeCon pode impactar seu futuro profissional. •⁠ ⁠Motivações para participar da KubeCon: O que o inspirou a participar do evento e como isso se alinha com seus objetivos profissionais. •⁠ ⁠Planejamento da viagem: Os desafios e estratégias envolvidos na organização da viagem, desde a logística até a preparação técnica para aproveitar ao máximo o evento. •⁠ ⁠Expectativas para a KubeCon: Quais são os tópicos e sessões que mais chamam sua atenção, e o que ele espera aprender e trazer de volta para sua atuação no Itaú Unibanco. •⁠ ⁠Dicas para profissionais que desejam participar de eventos internacionais: Conselhos sobre como se preparar, tanto profissionalmente quanto pessoalmente, para tirar o máximo proveito dessas oportunidades. Se você está interessado em saber mais sobre a importância de eventos como a KubeCon, obter insights sobre planejamento e preparação para participar de conferências internacionais, ou simplesmente busca inspiração para impulsionar sua carreira em tecnologia, não perca este episódio!

Próximo episódio do LowOpsCast contará com a presença de Pedro Ignácio (https://www.linkedin.com/in/pedroirufo/), engenheiro de soluções no Itaú Unibanco e Microsoft MVP em Cloud Security. Com uma sólida experiência em cibersegurança, design de sistemas e computação em nuvem, Pedro tem se destacado na construção de soluções de alto desempenho utilizando Kubernetes, Istio, Terraform,. Neste episódio, vamos nos aprofundar na preparação de Pedro para a KubeCon, o maior evento de Kubernetes e tecnologias cloud native do mundo. Durante nossa conversa, Pedro compartilha: * Sua trajetória profissional: Como suas experiências anteriores o levaram a este ponto da carreira e como a participação na KubeCon pode impactar seu futuro profissional. •⁠ ⁠Motivações para participar da KubeCon: O que o inspirou a participar do evento e como isso se alinha com seus objetivos profissionais. •⁠ ⁠Planejamento da viagem: Os desafios e estratégias envolvidos na organização da viagem, desde a logística até a preparação técnica para aproveitar ao máximo o evento. •⁠ ⁠Expectativas para a KubeCon: Quais são os tópicos e sessões que mais chamam sua atenção, e o que ele espera aprender e trazer de volta para sua atuação no Itaú Unibanco. •⁠ ⁠Dicas para profissionais que desejam participar de eventos internacionais: Conselhos sobre como se preparar, tanto profissionalmente quanto pessoalmente, para tirar o máximo proveito dessas oportunidades. Se você está interessado em saber mais sobre a importância de eventos como a KubeCon, obter insights sobre planejamento e preparação para participar de conferências internacionais, ou simplesmente busca inspiração para impulsionar sua carreira em tecnologia, não perca este episódio!

Estamos super animados em compartilhar com vocês o lançamento do episódio 154 do Kubicast! Neste episódio, tivemos o prazer de receber o especialista em Service Mesh Paulo Romão, que nos contou tudo sobre sua jornada no Itaú, desde seus dias como desenvolvedor até sua transição para o universo DevOps.Batemos um papo sobre a implementação de service mesh e os desafios que vêm com essa tarefa. O Paulo compartilhou insights valiosos sobre as principais ferramentas do mercado, como o Istio e o Linkerd, e como escolher a que melhor se adapta ao seu projeto. Ele enfatizou a importância de entender profundamente o comportamento das aplicações para aproveitar ao máximo essas tecnologias.Também conversamos sobre a importância da cultura DevOps nas organizações e como ela impacta diretamente na eficiência e qualidade das entregas. O Paulo nos contou como foi promover essa cultura dentro do Itaú e os obstáculos que enfrentou no caminho.Outro ponto alto foi a discussão sobre observabilidade em ambientes Kubernetes. Discutimos como a observabilidade é fundamental para diagnosticar problemas e melhorar a performance das aplicações. O Paulo destacou ferramentas e práticas que podem ajudar nessa jornada.Falamos ainda sobre certificações em DevOps e Kubernetes. O Paulo compartilhou quais certificações ele buscou e como elas contribuíram para o seu desenvolvimento profissional. Ele ressaltou que as certificações não só enriquecem o currículo, mas também ajudam a consolidar o conhecimento e a ganhar confiança para enfrentar os desafios do dia a dia.Uma última dica valiosa que ele nos deixou foi sobre a importância do networking. Participar de comunidades, eventos e trocar experiências com outros profissionais pode abrir portas e trazer aprendizados que vão além dos livros e cursos.Não percam este episódio recheado de informações e dicas práticas para quem deseja mergulhar mais fundo em DevOps e Kubernetes. Está imperdível!Esperamos que gostem tanto quanto nós. Até a próxima!O Kubicast é uma produção da Getup, empresa especialista em Kubernetes e projetos open source para Kubernetes. Os episódios do podcast estão nas principais plataformas de áudio digital e no YouTube.com/@getupcloud.

In this episode of the Kubernetes Bytes podcast, Ryan and Bhavin talk to Christian Posta - VP and Global Field CTO at Solo.io about all things Service Mesh. They discuss how things have evolved from the early Linkerd days to sidecar less istio service mesh implementations. They also talk about how service mesh can help you connect to application components running outside Kubernetes, and how developers and platform engineers have a shared responsibility model when it comes to implementing service mesh using internal developer platforms. Check out our website at https://kubernetesbytes.com/ Episode Sponsor: Nethopper Learn more about KAOPS: @nethopper.io For a supported-demo: info@nethopper.io Try the free version of KAOPS now! https://mynethopper.com/auth Cloud Native News: https://loft.sh/blog/our-24m-series-a-led-by-khosla-ventures/ https://www.harness.io/blog/celebrating-150m-in-new-financing-to-accelerate-innovation https://www.akamai.com/newsroom/press-release/akamai-announces-intent-to-acquire-api-security-company-noname https://www.linkedin.com/posts/rouvenbesters_its-official-the-otomi-platform-has-activity-7194604616901120000-48g7?utm_source=share&utm_medium=member_desktop https://www.wiz.io/blog/celebrating-our-1-billion-funding-round-and-12-billion-valuation Show Links: https://devsummit.infoq.com/conference/boston2024 https://www.solo.io/topics/cakes-stack/ https://www.solo.io/ Timestamps: 00:06:10 Cloud Native News 00:15:37 Interview with Torsten 01:01:58 Key takeaways

Guests Thomas Poignant and Todd Baert are Software engineers with long experience working on IAM systems and feature flagging software. Today they are both maintainers and members of the Technical Committee of OpenFeature which is a CNCF incubated project.   Do you have something cool to share? Some questions? Let us know: - web: kubernetespodcast.com - mail: kubernetespodcast@google.com - twitter: @kubernetespod News of the week Istio service Mesh add-on on Azure Kubernetes Services The CNCF released their 2023 annual survey Women Who code closed its doors Vulnerability in OpenMetadata version 1.31 or lower Links from the interview Thomas Poignant LinkedIn Twitter/X Todd Baert LinkedIn Twitter/X OpenFeature Feature Flagging Pete Hodgson article on feature flags Go feature flag Flagd FlagSmith

This interview was recorded for the GOTO Book Club.http://gotopia.tech/bookclubRead the full transcription of the interview hereThomas Vitale - Software Architect at Systematic & Author of "Cloud Native Spring in Action"  Josh Long - Spring Developer Advocate & Host of "A Bootiful Podcast"RESOURCESThomashttps://www.thomasvitale.comhttps://twitter.com/vitalethomashttps://linkedin.com/in/vitalethomashttps://github.com/ThomasVitaleJoshhttps://twitter.com/Starbuxmanhttps://www.linkedin.com/in/joshlonghttps://mastodon.online/@starbuxmanhttps://bootifulpodcast.fmhttps://joshlong.comhttps://www.biodrop.io/joshlongDESCRIPTIONIn Cloud Native Spring in Action, you'll learn how to containerize your Spring Boot applications with Cloud Native Buildpacks and deploy them on Kubernetes. This practical guide delivers unique insights into hosting microservices, serverless applications, and other modern architectures on cloud platforms. You'll learn how to use Spring-based methodologies, practices, and patterns that you won't find anywhere else.In Cloud Native Spring in Action you'll learn:• Cloud native best practices and design patterns• Build and test cloud native apps with Spring Boot and Spring Cloud• Handle security, resilience, and scalability in imperative and reactive applications• Configure, deploy, and observe applications on Kubernetes• Continuous delivery and GitOps to streamline your software lifecycle* Book description: © ManningRECOMMENDED BOOKSThomas Vitale • Cloud Native Spring in ActionJosh Long • Reactive SpringJosh Long, Marten Deinum & Daniel Rubio • Spring 6 RecipesMauricio Salatino • Platform Engineering on KubernetesMark Heckler • Spring Boot: Up & RunningLaurentiu Spilca • Spring, Start HereCornelia Davis • Cloud Native PatternsJez Humble & Dave Farley • Continuous DeliveryKevin Hoffman • Beyond the Twelve-Factor AppTwitterInstagramLinkedInFacebookLooking for a unique learning experience?Attend the next GOTO conference near you! Get your ticket: gotopia.techSUBSCRIBE TO OUR YOUTUBE CHANNEL - new videos posted daily!

KubeCon Europe 2024 in Paris was the biggest event of the Cloud Native Computing Foundation (CNCF) to date, with over 12k participants. Have you missed it? We've got you covered! Join not one but two CNCF Ambassadors as they explore the latest and greatest highlights from the event that every tech enthusiast is talking about.   But that's not all! We'll also zoom in on K8sgpt, a new entrant to the CNCF's sandbox that uses generative AI to give Kubernetes superpowers to everyone. Does this open source project go beyond the GenAI hype and get us closer to diagnosing and triaging issues in plain English? Let's ask the maintainers behind the project. Our guest is Thomas Schuetz, a Principal Cloud Architect with a keen interest in cloud-native application delivery. Thomas teaches at an Austrian University of Applied Sciences, focusing on cloud-native technologies. Thomas is enthusiastic about open source projects, contributing as a Keptn GC Member and K8sGPT Maintainer, alongside his role as Co-Chair of the CNCF TAG App Delivery. He also brings a deep industry background from his past roles at Dynatrace and more. The episode was live-streamed on 14 April 2024 and the video is available at https://www.youtube.com/watch?v=xr3viuhssdg OpenObservability Talks episodes are released monthly, on the last Thursday of each month and are available for listening on your favorite podcast app and on YouTube. We live-stream the episodes on Twitch and YouTube Live - tune in to see us live, and chime in with your comments and questions on the live chat. https://www.youtube.com/@openobservabilitytalks   ⁠https://www.twitch.tv/openobservability⁠ Show Notes: 00:00 - Show intro 00:59 - Episode and guest intro 03:15 - Redis moves off open source 05:21 - AI white paper 07:47 - TAG App Delivery updates 12:15 - Istio beta release of ambient mode 12:57 - Fluent Bit v3 major release 13:57 - Keptn project updates 17:20 - OpenCost adds environment sustainability  18:43 - OpenFeature adds client-side support with web SDK v1 20:08 - Perses 0.44 release 21:40 - K8sGPT founding team 24:07 - K8sGPT intro 27:36 - how K8sGPT works 31:28 - no vendor behind K8sGPT 36:10 - integration with multiple Gen AI services and local models 40:16 - K8sGPT current state and maturity 45:11 - K8sGPT traction 48:40 - K8sGPT acceptance into the sandbox and adopter companies 54:11 - how to reach out to Thomas Schuetz 56:09 - who's behind K8sGPT 59:07 - where to follow K8sGPT   1:00:17 - Outro Resources: https://github.com/k8sgpt-ai/k8sgpt https://k8sgpt.ai/ Cloud Native Artificial Intelligence whitepaper TAG App Delivery update at KubeCon Paris Fluent Bit v3.0 release OpenFeature Web SDK v1 k8sgpt slack Socials: Twitter:⁠ https://twitter.com/OpenObserv⁠ YouTube: ⁠https://www.youtube.com/@openobservabilitytalks⁠ Dotan Horovits ============ Twitter: https://twitter.com/horovits LinkedIn: https://www.linkedin.com/in/horovits/ Mastodon: https://fosstodon.org/@horovits Thomas Schuetz =============== Twitter: https://twitter.com/thschue LinkedIn: https://linkedin.com/in/thschue Mastodon: https://hachyderm.io/@thschue

This week on The Business of Open Source, I have an episode recorded on site at KubeCon EU in Paris with William Morgan, CEO of Buoyant. We had a fabulous conversation, which touched on some touchy subjects, including Buoyant's slightly changing relationship with Linkerd. But we talked about:Being an open source mercenary, but also being dedicated to making Linkerd a ‘proper' open source projectFeeling like open source was table stakes for a company in the space Buoyant plays in. This is an under-appreciated reason for being an open source company — you feel like it's just expected in the market you play in, so you do. Waiting too long (or is it too long?) to commercializeStarting out by selling support, but the problem with that because Linkerd worked well and people kept saying that they didn't need support because they never had problemsCompeting against Istio, which was backed by the Google engine and how that made Linkerd / Buoyant an underdog (or cockroach). For those of you who haven't been following Linkerd / Buoyant… Buoyant recently announced that they would be doing edge releases for Linkerd, but not stable releases. We talked about why they made this change and how the ecosystem responded. Check out the full episode! 

Lin Sun, Director of Open Source at Solo.io, is an influential figure in the cloud-native world. We spoke at All Things Open and she shared insights into her experiences and contributions in the open source community. Discussing her prominent role in the Istio project, she shares how Istio fits into the landscape of cloud-native service mesh, offering connectivity, security, and observability. She also highlights the launch of Istio Ambient Service Mesh, which reduces the complexity of Sidecar. Venturing into the world of AI, Lin envisions a future where AI assists in coding and improves software security while predicting a transition to a more conversational interaction with technology. She emphasizes the importance of human supervision in AI's development and its usefulness in making developers more efficient. 00:00 Introduction and Guest Welcome 00:29 Discussing Open Source Contributions and Community 01:53 Deep Dive into Istio and Service Mesh 02:49 Roles and Responsibilities in the Istio Community 04:24 Journey into Open Source Contributions 06:52 Advice for New Open Source Contributors 09:36 Exciting Updates in Istio 14:14 Exploring the Potential of AI in Open Source 19:33 Closing Remarks and Future Expectations Resources: Istio Ambient Service Mesh Made Easy Guest: Lin Sun is the Director of Open Source at Solo.io and an ex-CNCF ambassador. She has worked on Istio service mesh since 2017 and serves on the Istio Technical Oversight Committee. Previously, she was a Senior Technical Staff Member and Master Inventor at IBM for 15+ years. She is the author of the book “Istio Ambient Explained” and has more than 200 patents to her name.

In this KubeCon interview, Keith Mattix, engineering lead at Microsoft, talks about his experience and involvement in the open source community and the Istio project. He discusses his work in upstream service mesh and related technologies in the Kubernetes networking space. The conversation delves deeply into the evolution of open source offerings, highlighting the development and application of the Gateway API and the GAMMA Initiative. Maddox shares his vision for the future of these projects and talks about the value that developments in the open source ecosystem bring to end users.   00:00 Introduction and Guest Background 01:26 Role and Involvement in Open Source 01:55 Understanding Service Mesh 05:18 Istio's Journey in CNCF 07:19 Benefits of Open Source Collaboration 09:58 Gateway API Version 1.0 and Its Importance 13:55 Collaboration with the Gateway API Community 14:23 Future Goals for Gateway API 14:49 The Need for Standardization in Mesh Implementations 16:07 The Importance of User Adoption 16:43 Excitement for Open Source 18:09 The Impact of Infrastructure Improvements 20:10 The Importance of Kubernetes 21:51 Closing Remarks and Future Aspirations Guest:    Keith Mattix is a Senior Engineer on the Open Service Mesh team at Microsoft Azure. As a maintainer of the Service Mesh Interface CNCF project, Keith is a founding lead of the GAMMA initiative under Kubernetes SIG Network. Keith has written a lot of code that's led to some interesting side quests, and he's passionate about sharing those lessons and experiences with others. His love of distributed systems is eventually consistent.

Have you missed KubeCon North America in Chicago? This one's for you! In this episode, we explored the latest and greatest highlights from the event that every tech enthusiast is talking about. From cutting-edge innovations to industry insights, we've got the broad spectrum covered.  But that's not all! We'll also zoomed in on Istio, the popular service mesh open source project that has just recently reached CNCF graduation. Join us as we map out the service mesh universe, and then dive into Istio's galaxy, unraveling its architecture, features, and the roadmap direction with Ambient. And you'll get to hear it from the Istio authority, Lin Sun. Lin is the Director of Open Source at Solo.io and a CNCF ambassador. She has worked on the Istio service mesh since the beginning of the project in 2017 and serves on the Istio Steering Committee and Technical Oversight Committee. Previously, she was a Senior Technical Staff Member and Master Inventor at IBM for 15+ years. She is the author of the book "Istio Ambient Explained" and co-author of “Istio Explained”, and has more than 200 patents to her name. The episode was live-streamed on 15 November 2023 and the video is available at https://www.youtube.com/watch?v=bxnDH6LH-cA You can read the recap post: https://logz.io/blog/kubecon-na-2023-recap/?utm_source=devrel&utm_medium=devrel OpenObservability Talks episodes are released monthly, on the last Thursday of each month and are available for listening on your favorite podcast app and on YouTube. We live-stream the episodes on Twitch and YouTube Live - tune in to see us live, and chime in with your comments and questions on the live chat.https://www.twitch.tv/openobservabilityhttps://www.youtube.com/@openobservabilitytalks   Show Notes: 01:27 - Episode and guest intro 06:34 - KubeCon Highlights: Fluent Bit 09:16 - OpenTelemetry Logging, OTLP is GA 12:53 - OpenTelemetry project journey report 13:43 - WASM Day and Istio Day updates 16:18 - Keynote: the future of Kubernetes 18:51 -Crossplane latest release v1.14  19:24 - Kyverno supports non-Kubernetes workloads 20:12 - Vitess 18 is now GA 20:43 - AI is nascent in CNCF 22:56 - CNCF's GitOps microsurvey  23:56 - eBPF documentary released 27:08 - Service Mesh architecture and landscape 31:36 - Envoy proxy  33:48 - maturity of the projects 39:36 - Istio unique value proposition and adoption 43:55 - Kubernetes released native sidecar support 47:02 - The GAMMA initiative in Kubernetes Gateway API  50:04 - Istio updates: Ambient, multi-claster, Gateway API GA impl. For N-S  53:40 - CNCF Training & Certification Launch Istio Certification 54:56 - Istio roadmap 56:50 - how to follow Istio and Lin Sun and episode wrapup Resources: KubeCon Updates: https://www.cncf.io/blog/2023/11/07/opentelemetry-at-kubecon-cloudnativecon-north-america-2023-update/ https://opentelemetry.io/blog/2023/http-conventions-declared-stable/ https://www.cncf.io/reports/opentelemetry-project-journey-report/  https://blog.crossplane.io/crossplane-v1-14/   https://www.cncf.io/blog/2023/11/06/kyverno-expands-beyond-kubernetes/  https://planetscale.com/blog/announcing-vitess-18  https://www.cncf.io/blog/2023/11/07/cncf-gitops-microsurvey-learning-on-the-job-as-gitops-goes-mainstream/  Istio Spotlight: https://istio.io/latest/blog/2023/native-sidecars/ https://istio.io/latest/blog/2022/introducing-ambient-mesh/ https://gateway-api.sigs.k8s.io/concepts/gamma/ https://www.cncf.io/announcements/2023/07/12/cloud-native-computing-foundation-reaffirms-istio-maturity-with-project-graduation/  https://istio.io/latest/get-involved/ https://training.linuxfoundation.org/blog/istio-certification/  https://www.oreilly.com/library/view/istio-ambient-explained/9781098142698/ Socials: Twitter:⁠ https://twitter.com/OpenObserv⁠ YouTube: ⁠https://www.youtube.com/@openobservabilitytalks⁠

O Kubicast é uma produção da Getup, empresa especialista em Kubernetes e projetos open source para Kubernetes. Os episódios do podcast estão em getup.io/kubicast, nas principais plataformas de áudio digital e no YouTube.com/@getupcloud.

Laurent Doguin, Director of Developer Relations & Strategy at Couchbase, joins Corey on Screaming in the Cloud to talk about the work that Couchbase is doing in the world of databases and developer relations, as well as the role of AI in their industry and beyond. Together, Corey and Laurent discuss Laurent's many different roles throughout his career including what made him want to come back to a role at Couchbase after stepping away for 5 years. Corey and Laurent dig deep on how Couchbase has grown in recent years and how it's using artificial intelligence to offer an even better experience to the end user.About LaurentLaurent Doguin is Director of Developer Relations & Strategy at Couchbase (NASDAQ: BASE), a cloud database platform company that 30% of the Fortune 100 depend on.Links Referenced: Couchbase: https://couchbase.com XKCD #927: https://xkcd.com/927/ dbdb.io: https://dbdb.io DB-Engines: https://db-engines.com/en/ Twitter: https://twitter.com/ldoguin LinkedIn: https://www.linkedin.com/in/ldoguin/ TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Are you navigating the complex web of API management, microservices, and Kubernetes in your organization? Solo.io is here to be your guide to connectivity in the cloud-native universe!Solo.io, the powerhouse behind Istio, is revolutionizing cloud-native application networking. They brought you Gloo Gateway, the lightweight and ultra-fast gateway built for modern API management, and Gloo Mesh Core, a necessary step to secure, support, and operate your Istio environment.Why struggle with the nuts and bolts of infrastructure when you can focus on what truly matters - your application. Solo.io's got your back with networking for applications, not infrastructure. Embrace zero trust security, GitOps automation, and seamless multi-cloud networking, all with Solo.io.And here's the real game-changer: a common interface for every connection, in every direction, all with one API. It's the future of connectivity, and it's called Gloo by Solo.io.DevOps and Platform Engineers, your journey to a seamless cloud-native experience starts here. Visit solo.io/screaminginthecloud today and level up your networking game.Corey: Welcome to Screaming in the Cloud, I'm Corey Quinn. This promoted guest episode is brought to us by our friends at Couchbase. And before we start talking about Couchbase, I would rather talk about not being at Couchbase. Laurent Doguin is the Director of Developer Relations and Strategy at Couchbase. First, Laurent, thank you for joining me.Laurent: Thanks for having me. It's a pleasure to be here.Corey: So, what I find interesting is that this is your second time at Couchbase, where you were a developer advocate there for a couple of years, then you had five years of, we'll call it wilderness I suppose, and then you return to be the Director of Developer Relations. Which also ties into my personal working thesis of, the best way to get promoted at a lot of companies is to leave and then come back. But what caused you to decide, all right, I'm going to go work somewhere else? And what made you come back?Laurent: So, I've joined Couchbase in 2014. Spent about two or three years as a DA. And during those three years as a developer advocate, I've been advocating SQL database and I—at the time, it was mostly DBAs and ops I was talking to. And DBA and ops are, well, recent, modern ops are writing code, but they were not the people I wanted to talk to you when I was a developer advocate. I came from a background of developer, I've been a platform engineer for an enterprise content management company. I was writing code all day.And when I came to Couchbase, I realized I was mostly talking about Docker and Kubernetes, which is still cool, but not what I wanted to do. I wanted to talk about developers, how they use database to be better app, how they use key-value, and those weird thing like MapReduce. At the time, MapReduce was still, like, a weird thing for a lot of people, and probably still is because now everybody's doing SQL. So, that's what I wanted to talk about. I wanted to… engage with people identify with, really. And so, didn't happen. Left. Built a Platform as a Service company called Clever Cloud. They started about four or five years before I joined. We went from seven people to thirty-one LFs, fully bootstrapped, no VC. That's an interesting way to build a company in this age.Corey: Very hard to do because it takes a lot of upfront investment to build software, but you can sort of subsidize that via services, which is what we've done here in some respects. But yeah, that's a hard road to walk.Laurent: That's the model we had—and especially when your competition is AWS or Azure or GCP, so that was interesting. So entrepreneurship, it's not for everyone. I did my four years there and then I realized, maybe I'm going to do something else. I met my former colleagues of Couchbase at a software conference called Devoxx, in France, and they told me, “Well, there's a new sheriff in town. You should come back and talk to us. It's all about developers, we are repositioning, rehandling the way we do marketing at Couchbase. Why not have a conversation with our new CMO, John Kreisa?”And I said, “Well, I mean, I don't have anything to do. I actually built a brewery during that past year with some friends. That was great, but that's not going to feed me or anything. So yeah, let's have a conversation about work.” And so, I talked to John, I talked to a bunch of other people, and I realized [unintelligible 00:03:51], he actually changed, like, there was a—they were purposely going [against 00:03:55] developer, talking to developer. And that was not the case, necessarily, five, six years before that.So, that's why I came back. The product is still amazing, the people are still amazing. It was interesting to find a lot of people that still work there after, what, five years. And it's a company based in… California, headquartered in California, so you would expect people to, you know, jump around a bit. And I was pleasantly surprised to find the same folks there. So, that was also one of the reasons why I came back.Corey: It's always a strong endorsement when former employees rejoin a company. Because, I don't know about you, but I've always been aware of those companies you work for, you leave. Like, “Aw, I'm never doing that again for love or money,” just because it was such an unpleasant experience. So, it speaks well when you see companies that do have a culture of boomerangs, for lack of a better term.Laurent: That's the one we use internally, and there's a couple. More than a couple.Corey: So, one thing that seems to have been a thread through most of your career has been an emphasis on developer experience. And I don't know if we come at it from the same perspective, but to me, what drives nuts is honestly, with my work in cloud, bad developer experience manifests as the developer in question feeling like they're somehow not very good at their job. Like, they're somehow not understanding how all this stuff is supposed to work, and honestly, it leads to feeling like a giant fraud. And I find that it's pernicious because even when I intellectually know for a fact that I'm not the dumbest person ever to use this tool when I don't understand how something works, the bad developer experience manifests to me as, “You're not good enough.” At least, that's where I come at it from.Laurent: And also, I [unintelligible 00:05:34] to people that build these products because if we build the products, the user might be in the same position that we are right now. And so, we might be responsible for that experience [unintelligible 00:05:43] a developer, and that's not a great feeling. So, I completely agree with you. I've tried to… always on software-focused companies, whether it was Nuxeo, Couchbase, Clever Cloud, and then Couchbase. And I guess one of the good thing about coming back to a developer-focused era is all the product alignments.Like, a lot of people talk about product that [grows 00:06:08] and what it means. To me what it means was, what it meant—what it still means—building a product that developer wants to use, and not just want to, sometimes it's imposed to you, but actually are happy to use, and as you said, don't feel completely stupid about it in front of the product. It goes through different things. We've recently revamped our Couchbase UI, Couchbase Capella UI—Couchbase Capella is a managed cloud product—and so we've added a lot of in-product getting started guidelines, snippets of code, to help developers getting started better and not have that feeling of, “What am I doing? Why is it not working and what's going on?”Corey: That's an interesting decision to make, just because historically, working with a bunch of tools, the folks who are building the documentation working with that tool, tend to generally be experts at it, so they tend to optimize for improving things for the experience of someone has been using it for five years as opposed to the newcomer. So, I find that the longer a product is in existence, in many cases, the worse the new user experience becomes because companies tend to grow and sprawl in different ways, the product does likewise. And if you don't know the history behind it, “Oh, your company, what does it do?” And you look at the website and there's 50 different offerings that you have—like, the AWS landing page—it becomes overwhelming very quickly. So, it's neat to see that emphasis throughout the user interface on the new developer experience.On the other side of it, though, how are the folks who've been using it for a while respond to those changes? Because it's frustrating for me at least, when I log into a new account, which happens periodically within AWS land, and I have this giant series of onboarding pop-ups that I have to click to make go away every single time. How are they responding to it?Laurent: Yeah, it's interesting. One of the first things that struck me when I joined Couchbase the first time was the size of the technical documentation team. Because the whole… well, not the whole point, but part of the reason why they exist is to do that, to make sure that you understand all the differences and that it doesn't feel like the [unintelligible 00:08:18] what the documentation or the product pitch or everything. Like, they really, really, really emphasize on this from the very beginning. So, that was interesting.So, when you get that culture built into the products, well, the good thing is… when people try Couchbase, they usually stick with Couchbase. My main issue as a Director of the Developer Relations is not to make people stick with Couchbase because that works fairly well with the product that we have; it's to make them aware that we exist. That's the biggest issue I have. So, my goal as DevRel is to make sure that people get the trial, get through the trial, get all that in-app context, all that helps, get that first sample going, get that first… I'm not going to say product built because that's even a bit further down the line, but you know, get that sample going. We have a code playground, so when you're in the application, you get to actually execute different pieces of code, different languages. And so, we get those numbers and we're happy to see that people actually try that. And that's a, well, that's a good feeling.Corey: I think that there's a definite lack of awareness almost industry-wide around the fact that as the diversity of your customers increases, you have to have different approaches that meet them at various points along the journey. Because things that I've seen are okay, it's easy to ass—even just assuming a binary of, “Okay, I've done this before a thousand times; this is the thousand and first, I don't need the Hello World tutorial,” versus, “Oh, I have no idea what I'm doing. Give me the Hello World tutorial,” there are other points along that continuum, such as, “Oh, I used to do something like this, but it's been three years. Can you give me a refresher,” and so on. I think that there's a desire to try and fit every new user into a predefined persona and that just doesn't work very well as products become more sophisticated.Laurent: It's interesting, we actually have—we went through that work of defining those personas because there are many. And that was the origin of my departure. I had one person, ops slash DBA slash the person that maintain this thing, and I wanted to talk to all the other people that built the application space in Couchbase. So, we broadly segment things into back-end, full-stack, and mobile because Couchbase is also a mobile database. Well, we haven't talked too much about this, so I can explain you quickly what Couchbase is.It's basically a distributed JSON database with an integrated caching layer, so it's reasonably fast. So it does cache, and when the key-value is JSON, then you can create with SQL, you can do full-text search, you can do analytics, you can run user-defined function, you get triggers, you get all that actual SQL going on, it's transactional, you get joins, ANSI joins, you get all those… windowing function. It's modern SQL on the JSON database. So, it's a general-purpose database, and it's a general-purpose database that syncs.I think that's the important part of Couchbase. We are very good at syncing cluster of databases together. So, great for multi-cloud, hybrid cloud, on-prem, whatever suits you. And we also sync on the device, there's a thing called Couchbase Mobile, which is a local database that runs in your phone, and it will sync automatically to the server. So, a general-purpose database that syncs and that's quite modern.We try to fit as much way of growing data as possible in our database. It's kind of a several-in-one database. We call that a data platform. It took me a while to warm up to the word platform because I used to work for an enterprise content management platform and then I've been working for a Platform as a Service and then a data platform. So, it took me a bit of time to warm up to that term, but it explained fairly well, the fact that it's a several-in-one product and we empower people to do the trade-offs that they want.Not everybody needs… SQL. Some people just need key-value, some people need search, some people need to do SQL and search in the same query, which we also want people to do. So, it's about choices, it's about empowering people. And that's why the word platform—which can feel intimidating because it can seem complex, you know, [for 00:12:34] a lot of choices. And choices is maybe the enemy of a good developer experience.And, you know, we can try to talk—we can talk for hours about this. The more services you offer, the more complicated it becomes. What's the sweet spots? We did—our own trade-off was to have good documentation and good in-app help to fix that complexity problem. That's the trade-off that we did.Corey: Well, we should probably divert here just to make sure that we cover the basic groundwork for those who might not be aware: what exactly is Couchbase? I know that it's a database, which honestly, anything is a database if you hold it incorrectly enough; that's my entire shtick. But what is it exactly? Where does it start? Where does it stop?Laurent: Oh, where does it start? That's an interesting question. It's a… a merge—some people would say a fork—of Apache CouchDB, and membase. Membase was a distributed key-value store and CouchDB was this weird Erlang and C JSON REST API database that was built by Damian Katz from Lotus Notes, and that was in 2006 or seven. That was before Node.js.Let's not care about the exact date. The point is, a JSON and REST API-enabled database before Node.js was, like, a strong [laugh] power move. And so, those two merged and created the first version of Couchbase. And then we've added all those things that people want to do, so SQL, full-text search, analytics, user-defined function, mobile sync, you know, all those things. So basically, a general-purpose database.Corey: For what things is it not a great fit? This is always my favorite question to ask database folks because the zealot is going to say, “It's good for every use case under the sun. Use it for everything, start to finish”—Laurent: Yes.Corey: —and very few databases can actually check that box.Laurent: It's a very interesting question because when I pitch like, “We do all the things,” because we are a platform, people say, “Well, you must be doing lots of trade-offs. Where is the trade-off?” The trade-off is basically the way you store something is going to determine the efficiency of your [growing 00:14:45]—or the way you [grow 00:14:47] it. And that's one of the first thing you learn in computer science. You learn about data structure and you know that it's easier to get something in a hashmap when you have the key than passing your whole list of elements and checking your data, is it right one? It's the same for databases.So, our different services are different ways to store the data and to query it. So, where is it not good, it's where we don't have an index or a service that answer to the way you want to query data. We don't have a graph service right now. You can still do recursive common table expression for the SQL nerds out there, that will allow you to do somewhat of a graph way of querying your data, but that's not, like, actual—that's not a great experience for people were expecting a graph, like a Neo4j or whatever was a graph database experience.So, that's the trade-off that we made. We have a lot of things at the same place and it can be a little hard, intimidating to operate, and the developer experience can be a little, “Oh, my God, what is this thing that can do all of those features?” At the same time, that's just, like, one SDK to learn for all of the features we've just talked about. So, that's what we did. That's a trade-off that we did.It sucks to operate—well, [unintelligible 00:16:05] Couchbase Capella, which is a lot like a vendor-ish thing to say, but that's the value props of our managed cloud. It's hard to operate, we'll operate this for you. We have a Kubernetes operator. If you are one of the few people that wants to do Kubernetes at home, that's also something you can do. So yeah, I guess what we cannot do is the thing that Route 53 and [Unbound 00:16:26] and [unintelligible 00:16:27] DNS do, which is this weird DNS database thing that you like so much.Corey: One thing that's, I guess, is a sign of the times, but I have to confess that I'm relatively skeptical around, when I pull up couchbase.com—as one does; you're publicly traded; I don't feel that your company has much of a choice in this—but the first thing it greets me with is Couchbase Capella—which, yes, that is your hosted flagship product; that should be the first thing I see on the website—then it says, “Announcing Capella iQ, AI-powered coding assistance for developers.” Which oh, great, not another one of these.So, all right, give me the pitch. What is the story around, “Ooh, everything that has been a problem before, AI is going to make it way better.” Because I've already talked to you about developer experience. I know where you stand on these things. I have a suspicion you would not be here to endorse something you don't believe in. How does the AI magic work in this context?Laurent: So, that's the thing, like, who's going to be the one that get their products out before the other? And so, we're announcing it on the website. It's available on the private preview only right now. I've tried it. It works.How does it works? The way most chatbot AI code generation work is there's a big model, large language model that people use and that people fine-tune into in order to specialize it to the tasks that they want to do. The way we've built Couchbase iQ is we picked a very famous large language model, and when you ask a question to a bot, there's a context, there's a… the size of the window basically, that allows you to fit as much contextual information as possible. The way it works and the reason why it's integrated into Couchbase Capella is we make sure that we preload that context as much as possible and fine-tune that model, that [foundation 00:18:19] model, as much as possible to do whatever you want to do with Couchbase, which usually falls into several—a couple of categories, really—well maybe three—you want to write SQL, you want to generate data—actually, that's four—you want to generate data, you want to generate code, and if you paste some SQL code or some application code, you want to ask that model, what does do? It's especially true for SQL queries.And one of the questions that many people ask and are scared of with chatbot is how does it work in terms of learning? If you give a chatbot to someone that's very new to something, and they're just going to basically use a chatbot like Stack Overflow and not really think about what they're doing, well it's not [great 00:19:03] right, but because that's the example that people think most developer will do is generate code. Writing code is, like, a small part of our job. Like, a substantial part of our job is understanding what the code does.Corey: We spend a lot more time reading code than writing it, if we're, you know—Laurent: Yes.Corey: Not completely foolish.Laurent: Absolutely. And sometimes reading big SQL query can be a bit daunting, especially if you're new to that. And one of the good things that you get—Corey: Oh, even if you're not, it can still be quite daunting, let me assure you.Laurent: [laugh]. I think it's an acquired taste, let's be honest. Some people like to write assembly code and some people like to write SQL. I'm sort of in the middle right now. You pass your SQL query, and it's going to tell you more or less what it does, and that's a very nice superpower of AI. I think that's [unintelligible 00:19:48] that's the one that interests me the most right now is using AI to understand and to work better with existing pieces of code.Because a lot of people think that the cost of software is writing the software. It's maintaining the codebase you've written. That's the cost of the software. That's our job as developers should be to write legacy code because it means you've provided value long enough. And so, if in a company that works pretty well and there's a lot of legacy code and there's a lot of new people coming in and they'll have to learn all those things, and to be honest, sometimes we don't document stuff as much as we should—Corey: “The code is self-documenting,” is one of the biggest lies I hear in tech.Laurent: Yes, of course, which is why people are asking retired people to go back to COBOL again because nobody can read it and it's not documented. Actually, if someone's looking for a company to build, I guess, explaining COBOL code with AI would be a pretty good fit to do in many places.Corey: Yeah, it feels like that's one of those things that would be of benefit to the larger world. The counterpoint to that is you got that many business processes wrapped around something running COBOL—and I assure you, if you don't, you would have migrated off of COBOL long before now—it's making sure that okay well, computers, when they're in the form of AI, are very, very good at being confident-sounding when they talk about things, but they can also do that when they're completely wrong. It's basically a BS generator. And that is a scary thing when you're taking a look at something that broad. I mean, I'll use the AI coding assistance for things all the time, but those things look a lot more like, “Okay, I haven't written CloudFormation from scratch in a while. Build out the template, just because I forget the exact sequence.” And it's mostly right on things like that. But then you start getting into some of the real nuanced areas like race conditions and the rest, and often it can make things worse instead of better. That's the scary part, for me, at least.Laurent: Most coding assistants are… and actually, each time you ask its opinion to an AI, they say, “Well, you should take this with a grain of salt and we are not a hundred percent sure that this is the case.” And this is, make sure you proofread that, which again, from a learning perspective, can be a bit hard to give to new students. Like, you're giving something to someone and might—that assumes is probably as right as Wikipedia but actually, it's not. And it's part of why it works so well. Like, the anthropomorphism that you get with chatbots, like, this, it feels so human. That's why it get people so excited about it because if you think about it, it's not that new. It's just the moment it took off was the moment it looked like an assertive human being.Corey: As you take a look through, I guess, the larger ecosystem now, as well as the database space, given that is where you specialize, what do you think people are getting right and what do you think people are getting wrong?Laurent: There's a couple of ways of seeing this. Right now, when I look at from the outside, every databases is going back to SQL, I think there's a good reason for that. And it's interesting to put into perspective with AI because when you generate something, there's probably less chance to generate something wrong with SQL than generating something with code directly. And I think five generation—was it four or five generation language—there some language generation, so basically, the first innovation is assembly [into 00:23:03] in one and then you get more evolved languages, and at some point you get SQL. And SQL is a way to very shortly express a whole lot of business logic.And I think what people are doing right now is going back to SQL. And it's been impressive to me how even new developers that were all about [ORMs 00:23:25] and [no-DMs 00:23:26], and you know, avoiding writing SQL as much as possible, are actually back to it. And that's, for an old guy like me—well I mean, not that old—it feels good. I think SQL is coming back with a vengeance and that makes me very happy. I think what people don't realize is that it also involves doing data modeling, right, and stuff because database like Couchbase that are schemaless exist. You should store your data without thinking about it, you should still do data modeling. It's important. So, I think that's the interesting bits. What are people doing wrong in that space? I'm… I don't want to say bad thing about other databases, so I cannot even process that thought right now.Corey: That's okay. I'm thrilled to say negative things about any database under the sun. They all haunt me. I mean, someone wants to describe SQL to me is the chess of the programming world and I feel like that's very accurate. I have found that it is far easier in working with databases to make mistakes that don't wash off after a new deployment than it is in most other realms of technology. And when you're lucky and have a particular aura, you tend to avoid that stuff, at least that was always my approach.Laurent: I think if I had something to say, so just like the XKCD about standards: like, “there's 14 standards. I'm going to do one that's going to unify them all.” And it's the same with database. There's a lot… a [laugh] lot of databases. Have you ever been on a website called dbdb.io?Corey: Which one is it? I'm sorry.Laurent: Dbdb.io is the database of databases, and it's very [laugh] interesting website for database nerds. And so, if you're into database, dbdb.io. And you will find Couchbase and you will find a whole bunch of other databases, and you'll get to know which database is derived from which other database, you get the history, you get all those things. It's actually pretty interesting.Corey: I'm familiar with DB-Engines, which is sort of like the ranking databases by popularity, and companies will bend over backwards to wind up hitting all of the various things that they want in that space. The counterpoint with all of it is that it's… it feels historically like there haven't exactly been an awful lot of, shall we say, huge innovations in databases for the past few years. I mean, sure, we hear about vectors all the time now because of the joy that's AI, but smarter people than I are talking about how, well that's more of a feature than it is a core database. And the continual battle that we all hear about constantly is—and deal with ourselves—of should we use a general-purpose database, or a task-specific database for this thing that I'm doing remains largely unsolved.Laurent: Yeah, what's new? And when you look at it, it's like, we are going back to our roots and bringing SQL again. So, is there anything new? I guess most of the new stuff, all the interesting stuff in the 2010s—well, basically with the cloud—were all about the distribution side of things and were all about distributed consensus, Zookeeper, etcd, all that stuff. Couchbase is using an RAFT-like algorithm to keep every node happy and under the same cluster.I think that's one of the most interesting things we've had for the past… well, not for the past ten years, but between, basically, 20 or… between the start of AWS and well, let's say seven years ago. I think the end of the distribution game was brought to us by the people that have atomic clock in every data center because that's what you use to synchronize things. So, that was interesting things. And then suddenly, there wasn't that much innovation in the distributed world, maybe because Aphyr disappeared from Twitter. That might be one of the reason. He's not here to scare people enough to be better at that.Aphyr was the person behind the test called the Jepsen Test [shoot 00:27:12]. I think his blog engine was called Call Me Maybe, and he was going through every distributed system and trying to break them. And that was super interesting. And it feels like we're not talking that much about this anymore. It really feels like database have gone back to the status of infrastructure.In 2010, it was not about infrastructure. It was about developer empowerment. It was about serving JSON and developer experience and making sure that you can code faster without some constraint in a distributed world. And like, we fixed this for the most part. And the way we fixed this—and as you said, lack of innovation, maybe—has brought databases back to an infrastructure layer.Again, it wasn't the case 15 years a—well, 2023—13 years ago. And that's interesting. When you look at the new generation of databases, sometimes it's just a gateway on top of a well-known database and they call that a database, but it provides higher-level services, provides higher-level bricks, better developer experience to developer to build stuff faster. We've been trying to do this with Couchbase App Service and our sync gateway, which is basically a gateway on top of a Couchbase cluster that allow you to manage authentication, authorization, that allows you to manage synchronization with your mobile device or with websites. And yeah, I think that's the most interesting thing to me in this industry is how it's been relegated back to infrastructure, and all the cool stuff, new stuff happens on the layer above that.Corey: I really want to thank you for taking the time to speak with me. If people want to learn more, where's the best place for them to find you?Laurent: Thanks for having me and for entertaining this conversation. I can be found anywhere on the internet with these six letters: L-D-O-G-U-I-N. That's actually 7 letters. Ldoguin. That's my handle on pretty much any social network. Ldoguin. So X, [BlueSky 00:29:21], LinkedIn. I don't know where to be anymore.Corey: I hear you. We'll put links to all of it in the [show notes 00:29:27] and let people figure out where they want to go on that. Thank you so much for taking the time to speak with me today. I really do appreciate it.Laurent: Thanks for having me.Corey: Laurent Doguin, Director of Developer Relations and Strategy at Couchbase. I'm Cloud Economist Corey Quinn and this episode has been brought to us by our friends at Couchbase. If you enjoyed this episode, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, along with an angry comment that you're not going to be able to submit properly because that platform of choice did not pay enough attention to the experience of typing in a comment.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.

Chris Hill, owner of HumblePod and host of the We Built This Brand podcast, joins Corey on Screaming in the Cloud to discuss the future of podcasting and the role emerging technologies will play in the podcasting space. Chris describes why AI is struggling to make a big impact in the world of podcasting, and also emphasizes the importance of authenticity and finding a niche when producing a show. Corey and Chris discuss where video podcasting works and where it doesn't, and why it's more important to focus on the content of your podcast than the technical specs of your gear. Chris also shares insight on how to gauge the health of your podcast audience with his Podcast Listener Lifecycle evaluation tool.About ChrisChris Hill is a Knoxville, TN native and owner of the podcast production company, HumblePod. He helps his customers create, develop, and produce podcasts and is working with clients in Knoxville as well as startups and entrepreneurs across the United States, Silicon Valley, and the world.In addition to producing podcasts for nationally-recognized thought leaders, Chris is the co-host and producer of the award-winning Our Humble Beer Podcast and the host of the newly-launched We Built This Brand podcast. He also lectures at the University of Tennessee, where he leads non-credit courses on podcasts and marketing.  He received his undergraduate degree in business at the University of Tennessee at Chattanooga where he majored in Marketing & Entrepreneurship, and he later received his MBA from King University.Chris currently serves his community as the President of the American Marketing Association in Knoxville. In his spare time, he enjoys hanging out with the local craft beer community, international travel, exploring the great outdoors, and his many creative pursuits.Links Referenced: HumblePod: https://www.humblepod.com/ HumblePod Quick Edit: https://humblepod.com/services/quick-edit Podcast Listener Lifecycle: https://www.humblepod.com/podcast/grow-your-podcast-with-the-listener-lifecycle/ Twitter: https://twitter.com/christopholies Transcript:Announcer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Are you navigating the complex web of API management, microservices, and Kubernetes in your organization? Solo.io is here to be your guide to connectivity in the cloud-native universe!Solo.io, the powerhouse behind Istio, is revolutionizing cloud-native application networking. They brought you Gloo Gateway, the lightweight and ultra-fast gateway built for modern API management, and Gloo Mesh Core, a necessary step to secure, support, and operate your Istio environment.Why struggle with the nuts and bolts of infrastructure when you can focus on what truly matters - your application. Solo.io's got your back with networking for applications, not infrastructure. Embrace zero trust security, GitOps automation, and seamless multi-cloud networking, all with Solo.io.And here's the real game-changer: a common interface for every connection, in every direction, all with one API. It's the future of connectivity, and it's called Gloo by Solo.io.DevOps and Platform Engineers, your journey to a seamless cloud-native experience starts here. Visit solo.io/screaminginthecloud today and level up your networking game.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. My returning guest probably knows more about this podcast than I do. Chris Hill is not only the CEO of HumblePod, but he's also the producer of a lot of my various media endeavors, ranging from the psychotic music videos that I wind up putting out to mock executives on their birthdays to more normal videos that I wind up recording when I'm forced into the studio and can't escape because they bar the back exits, to this show. Chris, thank you for joining me, it's nice to see you step into the light.Chris: It's a pleasure to be here, Corey.Corey: So, you have been, effectively, producing this entire podcast after I migrated off of a previous vendor, what four years ago? Five?Chris: About four or five years ago now, yeah. It's been a while.Corey: Time is a flat circle. It's hard to keep track of all of that. But it's weird that you and I don't get to talk nearly as much as we used to, just because, frankly, the process is working and therefore, you disappear into the background.Chris: Yeah.Corey: One of the dangerous parts of that is that the only time I ever wind up talking to you is when something has gone wrong somewhere and frankly, that does not happen anymore. Which means we don't talk.Chris: Yeah. And I'm okay with that. I'm just kidding. I love talking to you, Corey.Corey: Oh, I tolerate you. And every once in a while, you irritate me massively, which is why I'm punishing you this year by—Chris: [laugh].Corey: Making you tag along for re:Invent.Chris: I'm really excited about that one. It's going to be fun to be there with you and Jeremy and Mike and everybody. Looking forward to it.Corey: You know how I can tell that you've never been to re:Invent before?Chris: “I'm looking forward to it.”Corey: Exactly. You still have life in your eyes and a spark in your step. And yeah… that'll change. That'll change. So, a lot of this show is indirectly your fault because this is a weird thing for a podcaster to admit, but I genuinely don't listen to podcasts. I did when I was younger, back when I had what the kids today call ‘commute' or ‘RTO' as they start slipping into the office, but I started working from home almost a decade ago, and there aren't too many podcasts that fit into the walk from the kitchen to my home office. Like great, give me everything you want me to know in about three-and-a-half seconds. Go… and we're done. It doesn't work. So, I'm a producer, but I don't consume my own content, which I think generally is something you only otherwise see in, you know, drug dealers.Chris: Yeah. Well, and I mean, I think a lot of professional media, like, you get to a point where you're so busy and you're creating so much content that it's hard to sit down and review your own stuff. I mean, even at HumblePod, I'm in a place where we're producing our own show now called We Built This Brand, and I end up in a place where some weeks I'm like, “I can't review this. I approve it. You send it out, I trust you.” So, Corey, I'm starting to echo you in a lot of ways and it's just—it makes me laugh from time to time.Corey: Somewhat recently, I wound up yet again, having to do a check on, “Hey, you use HumblePod for your podcasting work. Do you like them?” And it's fun. It's almost like when someone reaches out about someone you used to work with. Like, “We're debating hiring this person. Should we?” And I love being able to give the default response for the people I've worked with for this long, which is, “Shut up and hire them. Why are you talking to me and not hiring them faster? Get on with it.”Because I'm a difficult customer. I know that. The expectations I have are at times unreasonably high. And the fact that I don't talk to you nearly as much as I used to shows that this all has been working. Because there was a time we talked multiple times a day back—Chris: Mm-hm.Corey: When I had no idea what I was doing. Now, 500-some-odd episodes in, I still have no idea what I'm doing, but by God, I've gotten it down to a science.Chris: Absolutely you have. And you know, technically we're over 1000 episodes together, I think, at this point because if you combine what you're doing with Screaming in the Cloud, with Last Week in AWS slash AWS Morning Brief, yeah, we've done a lot with you. But yes, you've come a long way.Corey: Yes, I have become the very whitest of guys. It works out well. It's like, one podcast isn't enough. We're going to have two of them. But it's easy to talk about the past. Let's talk instead about the future a little bit. What does the future of podcasting look like? I mean, one easy direction to go in with this, as you just mentioned, there's over 1000 episodes of me flapping my gums in the breeze. That feels like it's more than enough data to train an AI model to basically be me without all the hard work, but somehow I kind of don't see it happening anytime soon.Chris: Yeah, I think listeners still value authenticity a lot and I think that's one of the hard things you're seeing in podcasting as a whole is that these organizations come in and they're like, “We're going to be the new podcast killer,” or, “We're going to be the next thing for podcasting,” and if it's too overproduced, too polished, like, I think people can detect that and see that inauthenticity, which is why, like, AI coming in and taking over people's voices is so crazy. One of the things that's happening right now at Spotify is that they are beta testing translation software so that Screaming in the Cloud could automatically be in Spanish or Last Week in AWS could automatically be in French or what have you. It's just so surreal to me that they're doing this, but they're doing exactly what you said. It's language learning models that understand what the host is saying and then they're translating it into another language.The problem is, what if that automation gets that word wrong? You know how bad one wrong word could be, translating from Spanish or French or any other language from English. So, there's a lot of challenges to be met there. And then, of course, you know, once they've got your voice, what do they do with it? There's a lot of risk there.Corey: The puns don't translate very well, most of the time, either.Chris: Oh, yes.Corey: Especially when I mis-intentionally mispronounce words like Ku-BER-netees.Chris: Exactly. I mean, it's going to be auto-translated into text at some point before it's then put out as, you know, an audio source, and so if you say something wrong, it's going to be an issue. And Ku-BER-netees or Chat-Gippity or any of those great terms that you have, they're going to also be translated wrong as well, and that creates its own can of worms so to speak.Corey: Well, let me ask you something because you have always been one to embrace emerging technologies. It's one of the things I appreciate about you; you generally don't recommend solutions from the Dark Ages when it comes to what equipment should I have and how should I maintain it and the rest. But there are a lot of services out there that will now do automatic transcription and the service that you use at the moment remains a woman named Cecilia, who's remarkably good at what she does. But why have you not replaced her with a robot?Chris: [laugh]. Very simply put, I mean, it kind of goes back to what I was just saying about language translation. AI does not understand context for human words as well as humans do, and so words are wrong a lot of times in auto transcription. I mean, I can remember a time when, you know, we first started working with you all were, if there was one thing wrong in a transcript, an executive at AWS would potentially make fun of you on Twitter for it. And so, we knew we had to be on our A-game when it came to that, so finding someone who had that niche expertise of being able to translate not just words and understand words, but also understand tech terminology, you know, I think that that's, that's its own animal and its own challenge. So yeah, I mean, you could easily get away with something—Corey: Especially with my attentional mispronunciation where she's, “I don't quite know what you're saying here, and neither does the entire rest of the industry.” Like, “Postgres-squ—do you mean Postgres? Who the hell calls it Postgres-squeal?” I do. I call it that. Two warring pronunciations, I will unify them by coming up with a third that is far worse. It's kind of my shtick. The problem is, at some point, it becomes too inside-jokey when I have 15 words that I'm doing that too, and suddenly no one knows what the hell I'm talking about and the joke gets old quickly.Chris: Yep.Corey: So, I've tried to scale that back. But there are still a few that I… I can't help but play with.Chris: Yeah. And it's always fun bringing someone new in to work on—work with you all because they're always like, “What is he saying? Does he mean this?” And [laugh] it's always an adventure.Corey: It keeps life fun though.Chris: Absolutely.Corey: So, one thing that you did for a while, back when I was starting out, it almost felt like you were in cahoots with Big Microphone because once I would wind up getting a setup all working and ready for the recording, like, “Great. Everything working terrifically? Cool, throw it away. It's time for generation three of this.” I think I'm on, like, gen six, or gen seven now, but it's been relatively static for the past few years. Are the checks not as big as they used to be? I mean, if we hit a point of equilibrium? What's going on?Chris: Yeah, unfortunately, Big Microphone isn't paying what they used to. The economy and interest rates and all that, it's just making it hard. But once you get to a certain level of gear, it's going to be more important that you have good content than better and better gear. Could we keep going? Sure. If you wanted to buy a studio and you wanted to get Neumann microphones or something like that, we could keep going. But again, Big Microphone is not paying what they used to.Corey: When people reach out because they're debating starting a podcast and they ask me for advice, other than hire HumblePod, the next question they usually get around to is gear. And I don't think that they are expecting my answer, which is, it does not matter. Because if the content is good, the listeners will forgive an awful lot. You could record it into your iPhone in a quiet room and they will put up with that. Whereas if the content isn't good, it doesn't matter what the production value is because people are constantly being offered better things to do with their time. You've got to grab them, you have to be compelling to your target audience or the rest of it does not matter.Chris: Yeah. And I think that's the big challenge with audio is a lot of people get excited, especially I find this true of people in the tech industry of like, “Okay, I want to learn all the tech stuff, I love all the cool tech stuff, and so I'm going to go out and buy all this equipment first.” And then they spend $5,000 on equipment and they never record a single episode because they put all their time and energy into researching and buying gear and never thought about the content of the show. The truth is, you could start with your iPhone and that's it. And while I don't necessarily advise that, you'd be surprised at the quality of audio on an iPhone.I've had a client have to re-record something while they were traveling remotely and I said, “You just need to get your iPhone out.” They took their AirPods, plugged them in and, I said, “No. Take them out, use the microphone on the iPhone.” And you can start with something as simple as that. Now, once you want to start making it better, sure, that's a great way to grow and that does influence people staying with your podcast over time, but I think in the long run, content trumps all.Corey: One of the problems I keep seeing is that people also want to record a podcast because they have a great idea for a few episodes. My rule of thumb—because I've gotten this wrong before—is, okay, if you want to do a whole new podcast, come up with the first 12 episodes. Because two, three, four, of course, you've got your ideas. And then by the—you'll find in many cases, you're going to have a problem by the end of it. Years ago, I did a mini-series inside of AMB called “Networking in the Cloud” where it was sponsored by, at the time, ThousandEyes, before Cisco bought them and froze them in amber for all eternity.But it was fun for the first six episodes and then I realized I'd said all I needed to say about networking, and I was on the hook for six more. And Ivan Pepeinjak, who's his own influencer type in the BGP IP space was like, “This is why you should stay in your lane. He's terrible. He got it all wrong.” Like, “Great. Come on and tell me exactly how I got it wrong,” because I was trying to approach it from a very surface topical area, but BGP is one of those areas where I get very wrapped around my own axle just because I've never used it in anger. Being able to pivot the show format is what saved me on that. But if I had started doing this as its own individual podcast and launched, it would have died on the vine, just because it would not have had enough staying power and I didn't have the interest to continue working on it. Could someone else come up with a networking-in-the-cloud podcast that had hundreds of episodes? Absolutely, but those people are what we call competent and good at things in a way that I very much am not.Chris: Yep. And I completely agree. I mean, 12 is my default number, so—I'm not going to take credit for your saying 12, but I know we've talked about that before. And—Corey: It was a 12-episode miniseries is why. And I remember by ten, I had completely scraped the bottom of the barrel. Then Ivan saved me on one of them, and then I did, I think, a mini-series-in-review, which is cheating but worked.Chris: Yeah. I remember that, the trials and travails of giving that out. It was fun, though. But with that, yeah, like, 12 is a good number because, like, to your point, if you have 12 and you want to do a monthly show, you've got a year's worth of content, if you do bi-weekly, that's six months, and if it's a weekly show, it's at least a quarter's worth of content. So, it does help you think through and at least come up with any potential roadblocks you might have by at least listing out, here's what episodes one, two, three, four, five and so on would be. And so, I do think that's a great approach.Corey: And don't be an idiot like I was and launch a newsletter and then podcast that focus on last week's news because you can't work ahead on that. If you can, why are you not a multi-billionaire for playing the markets? If you can predict the future, there's a more lucrative career for you than podcasting, I promise. But that means that I have to be on the treadmill on some level. I've gotten it down to a point where I can stretch it to ten days. I can take ten days off if I preload, do it as early as I possibly can beforehand and then as late as I possibly can when I return. Anything more than that, I'm either skipping a week or delaying the show or have to get a guest author or artist in.Chris: Yeah. And you definitely need that time off, and so that's the one big challenge, I think with podcasting, too, is like you create this treadmill for yourself that you constantly have to fill content after content after content. I think that's one of the big challenges in podcasting and one of the reasons we see so many podcasts fade out. I don't know if you're familiar, but there is a term called podfade, which is just that: people burning out, fading out in their excitement for a podcast. And most podcasters fade out by episode seven or eight, somewhere in that range, so to see someone go for say, like, you have 500 episodes plus, we're talking about a ton of good content. You've found your rhythm, you've found your groove. That can do it. But yeah, it's always, always a challenge staying motivated.Corey: One thing that consistently surprises me is that the things I care about as the creator and the things the audience cares about are not the same. And you have to be respectful of your audience's time. I've done the numbers on the shows that I put out and it's something on the order of over a year of human time for every episode that I put out. If I'm going to take a year from humanity's collective lifetimes in order to say my inane thoughts, then I have to be respectful of the audience's time. Which means, “Oh, I'm going to have a robot do it so I don't have to put the work in.” It doesn't work that way. That's not how you sustain.Chris: Right. In and again, it takes out that humanity that makes podcasting so special and makes that connection with even the listener so special. And I'm sure you've experienced this too. When you go to re:Invent, like, we're going to have here in just a few short months, people know you, and they probably say things and bring up things that you haven't even thought about. And you're like, “Where did you even learn that I did that?” And then you realize, “Oh, I said that on a podcast episode.”Corey: Yeah. What's weird is I don't get much feedback online for it, but people will talk to me in depth about the show. They'll come up to me near constantly and talk about it. They don't reach out the same way, which I guess makes sense. There are a couple of podcasts that I've really admired and listened to on and off in the car for years, but I've never reached out to the creators because I feel like I would sound ridiculous. It's not true. I know intellectually it's not true, but it feels weird to do it.Chris: One of the ways I got into podcasting was a podcast that just invited me to—you know, invited their listeners to sign up and engage with them. And I think that's something in the medium that does make it interesting is once you do engage, you find out that these creators respond. And where else do you get that, you know? If you're watching a big TV show and you tweet at somebody online that you admire in the show, the chance of them even liking what you said about them online is very slim to none. But with podcasting, there's just a different level of accessibility I find with most productions and most shows that makes it really something special.Corey: One thing that still surprises me—and I don't think I've ever been this explicit about it on the show, but why the hell not I have nothing to hide—Thursday evening, 5 p.m. Pacific time. That's when the automation fires and rotates everything for the newsletter and the AWS Morning Brief. Anything that comes in after that, unless I manually do an override, will not be in the next week's issue; it'll be the week after.That applies to Security as well, which means 5 p.m. on Thursday, it seals it, I write and record it and it goes ou—that particular one goes out Thursday morning the following week. And no one has ever said anything about this seems awfully late. Occasionally, there's been news the day before and someone said, “Oh, why didn't you include this?”And it's because, believe it or not, I don't just type this in and hit the send button. There's a bit more to it than that these days. But people don't need the sense of immediacy. This idea of striving to be first is not sustainable and it leads to terrible outcomes. My entire philosophy has not been to have the first take but rather the best take.Chris: Mm-hm.Corey: Sometimes I even get it right.Chris: And I mean in podcasting, too. Like, it's about, you serve a certain niche, right? Like, the people who are interested in AWS services and in this world of cloud computing listen to what you say, listen to the people you interview, and really enjoy those conversations. But that's not everybody in the world. That's not a very broad audience. And so, I think that those niches really serve a purpose.And the way I've always thought about it is, like, if you go to the grocery store, you know how you always have that rack of magazines with the most random interests? That's essentially what podcasting is. It's like each podcast is a different magazine that serves someone's random—and hyper-specific sometimes—niche interest in things. I mean, the number of things you can find podcasts on is just ridiculous. And I think the same is true for this. But the people who do follow, they're very serious, they're very dedicated, they do listen, and yeah, I think it's just a fascinating, fascinating thing.Corey: The way that I see it has been that I've been learning more from the audience and the things that people say that most people would believe, but… I make a lot of mistakes doing this, but talking to people does tend to shine a light on a lot of this. But enough about the past. Most of my episodes are about things that have previously happened. What does the future of podcasting look like? Where's it going from here?Chris: Oh, man. Well, I think the big question on everybody's mind is, do I need a video podcast? And I think that for most people, that's where the big question lies right now. I get a lot of questions about it, I get people reaching out, and I think the short answer to that is… not really. Or to answer a question I know you love, Corey, it depends.And the reason for that is, there's a lot with the tech of podcasting that just isn't going to distribute to everywhere, all at once anymore. The beauty of podcasting is that it's all based on an RSS feed. If you build an RSS feed and you put it in Apple Podcasts and Spotify, that RSS feed will distribute everywhere and it will distribute your audio everywhere. And what we see happening right now, and really one of the bigger challenges in podcasting, is that the RSS feed only provides audio. Technically, that's not accurate, but it does for most services.So, YouTube has recently come out and said that they are going to start integrating RSS feeds, so you'll be able to do those audiogram-esque things that a lot of people have done through apps like Headliner and stuff for a long time, or even their podcast host may automatically translate a version of their audio podcast into a video and just do, like, a waveform. They're going to have that in YouTube. TikTok is taking a similar approach. And they're both importing just the audio. And the reason I said earlier, that's technically not accurate is because RSS feeds can also support MP4s, but neither service is going to accept that or ingest it directly into their service from what you provide outbound.So, it's a very interesting time because it feels like we're getting there with video, but we're still not there, and we're still probably several years off from it. So, there's a lot of interest in video and I think the future is going to be video, but I think it's going to be a combination, too, with audio because who wants to sit and watch something for an hour-and-a-half when you're used to listening to it your commute or while you do the dishes or any number of other things that don't involve having your eyeballs directly on the content.Corey: We've tried it with this show. I found that it made the recording process a bit more onerous because everyone is suddenly freaking out about how they look and I can't indulge my constant nose-picking habit. Kidding. So, it was more work, I had to gussy myself up a bit more than dressing like a slob like I do some mornings because I do have young children and a deadline to get them to school by. But I never saw the audience to materialize there and be worth it.Because watching a video of two people talking with each other, it feels too much like a Zoom call that you can't participate in, so what's the point?Chris: Right.Corey: So, there's that. There's the fact that I also have very intentionally built most of what I do around newsletters and podcasts because at least so far, those are not dependent upon algorithmic discovery in the same way. I don't have to bias for things that YouTube likes this month. Instead, I can focus on the content that people originally signed up to hear me put out and I don't have to worry about it in the same way. Email predates me, it'll be here long after I'm gone, and that seems to make sense.I also look at how I have consumed podcasts, and times when I do, it's almost always while I'm doing something else. And if I have to watch a screen, that becomes significantly more distracting, and harder for me to find the time to do it with.Chris: I think what you're seeing is that, like, there's some avenues to where video podcasting is really good and really interesting, and I think the real place where that works best right now is in-person interviews. So, Corey, if you went out and interviewed Andy Jassy in person in Seattle, that to me would be something that would warrant bringing the cameras out for and putting online because people would want to see you in the office interacting with him. That would be interesting. To your point, during the Zoom calls and things like that, you end up in a place where people just aren't as interested in sitting and watching the Zoom call. And I think that's something that is a clear distinction to make.Entertainment, comedy, doing things in person, I think that's where the real interest in video is and that's why I don't think video will be for everybody all the time. The thing that is starting to come up as well is discoverability, and that has always been a challenge, but as we get into—and we probably don't want to go down this rabbit hole, but you know, what's happened to Twitter and X, like, discoverability is becoming more of a challenge because they're limiting access to that platform. They're limiting discoverability if you're not willing to pay for a blue checkmark. They're doing all these things to make it harder for small independent podcasts to grow.And the places that are opening up for it to grow are places like YouTube, places like TikTok, that have the ability to not only just put your full podcasts online now, but you can actually do, like, YouTube shorts or highlighted clips, and directly link those back to the long-form content that you're producing. So, there is some value in it, there is a technology and a future there for it, but it's just a very complicated time to be in podcasting and figuring out where to go to grow. That's probably the biggest challenge that we face and I think ultimately, that just comes down to developing an audience outside of these social media channels.Corey: One thing that you were talking about a while back in a conversation that I don't think I've ever followed up with you on—and there's no time like in front of a bunch of public people to do that—Chris: [laugh].Corey: You were talking to me about something that you were calling the Podcast Listener Lifecycle.Chris: Yes.Corey: What's your point on that?Chris: So, the Listener Lifecycle is something I developed, just to be frank, working with you guys, learning from you all, and also my background in marketing, and in building audiences and things, from my own podcasts and other things that I did prior to building HumblePod, led me to a place of going, how can we best explain to a client where their podcast is? How does it exist? Where does it exist? All that good stuff. And basically, the Listener Lifecycle is just that.It's a design—and we'll have links to it because I actually did a whole podcast season on the Listener Lifecycle from beginning to end, so that's probably the easiest way to talk about it. But essentially, it's the idea of, you're curious about a show, and how do you go from being curious about a show to exploring a podcast, to then becoming a follower of the podcast, literally clicking the Follow button. What does it take to get through each one of those stages? How can you identify where your audience is? And basically, it's a tool you can use to say, “Well, this is where my listener is in the stages.” And then once they get to be a follower, how do I build them into something more?Well, get them to be a subscriber, subscribe to a newsletter, subscribe to a Patreon or Substack or whatever that subscription service is that you prefer to use, and get them off of just being on social media and following you there and following you in a podcast audio form. Because things can happen: your podcast host could break and you'd lose your audience, right? We've seen Twitter, which we may have thought years ago that it would never go away, and now we don't know how long it's going to be there. It could be gone by the time we're done with this conversation for all we know. I've got all my notifications turned off, so we're basically in a liminal space at this point.But with that said, there's a lot of risk in audiences changing and things like that, so audience portability is really important. So, the more you can collect email addresses, collect contact information, and communicate with that group of people, the better your audience is going to be. And so, that's what it's about is helping people get to that stage where they can do that so that they don't lose audiences and so that they can even build and grow audiences beyond that to the point where they get to the last phase, which is the ‘true fan' phase. And that's where you get people who love your show, retweet everything you do, repost everything you do, and share it with all their friends every time you're creating new content. And that's ultimately what you want: those die-hard people that come up to you and know everything about you at re:Invent, those are the people that you want to create more of because they're going to help you grow your show and your audience, ultimately. So, that's what it's about. I know that's a lot. But again, like, we'll have a link in the show notes to where you can learn more about it.Corey: Indeed, we will. Normally I'm the one that says, “And we'll include a link to that in the show notes.” But you're the one that has to actually make all that happen. Here's another glimpse behind the curtain. I have a Calendly link that I pass out to people to book time on the show. They fill out the form, which is relatively straightforward and low effort by design, and the next time I think about it is ten minutes beforehand when it pops up with, “Hey, you have a recording to go to.” Great. I book an hour for a half-hour recording. I wind up going through this entire conversation. When we're done, we close out the episode, we chat a bit, I close the tab, and I don't think about it again, it's passed off to you folks entirely. It is the very whitest of white glove treatments. Because I, once again, am the very whitest of white guys.Chris: We aim to please [laugh].Corey: Exactly. Because I remember before this, I used to have things delayed by months because I would forget to copy the freaking file into Dropbox, of all things. And that was just wild to me.Chris: And we stay on you about that because we want to make sure that your show gets out and—Corey: And now it automatically transfers and I—when the automation works—I don't have to think about it again. What is fun to me is despite all the time that I spend in enterprise cloud services, we still use things that are prosumer, like Dropbox and other things that are human-centric because for some reason, most of your team are not also highly competent cloud developers. And I still think it is such a miss that something like S3, which would be perfect for this, requires that level of engineering. And I have more self-respect than that. I'd have to build some stuff in order to make that work effectively on my end, let alone folks who have actual jobs that don't involve messing around with cloud services all day.But it blows my mind that there's still such this gulf between things that sound like you would have one of your aging parents deal with versus something that is extraordinarily capable and state-of-the-art. I know they're launching a bunch of things like Amazon's IVS, which is a streaming offering, a lot of their elemental offerings for media packaging, but I look at it, it's like wow, not only is this expensive, it doesn't solve any problems that we actually have and would add significant extra steps to every part of it. Thanks, but no thanks. And sure, maybe we're not the target market, but I can't shake the feeling that there are an awful lot of people like us that fit that profile.Chris: Yeah. And I mean, you bring up a good point about not using S3, things like that. It has occurred to me as well that, hey, maybe we should find somebody to help us develop a technology like this to make it easier on us on the back end to do all the recording and the production in one place, one database, and be able to move on. So, at some point I would love to get there. That's probably a conversation for after the podcast, Corey, but definitely is something that we've been thinking about at HumblePod is, how do we reach that next step of making it even easier on our clients?Corey: Well, it is certainly appreciated. But again, remember, your task is to continue to perform the service excellently, not be the poster child for cloud services with dumb names.Chris: [laugh]. Yes, yes. And I'm sure we could come up with a bunch.Corey: One last question before we wind up calling in an episode. I know that I've been emphasizing the white glove treatment that I get—and let's be clear, you are not inexpensive, but you're also well worth it; you deliver value extraordinarily for our needs—do you offer things that are not quite as, we'll call it, high-touch and comprehensive?Chris: Yes, we do actually. We just recently launched a new service called Quick Edit and it's just that. It's still humans touching the service, so it's not a bunch of automated, hey, we're just running this through an AI program and it's going to spit it out on the other end. We actually have a human that touches your audio, cleans it up, and sends it back. And yeah, we're there to make sure that we can clean things up quickly and easily and affordably for those folks that are just in a pinch.Maybe you edit most weeks and you're just tired of doing the editing, maybe you're close to podfading and you just want an extra boost to see if you can keep the show going. That's what we have the Quick Edit service for. And that starts at $150 an episode and we'll edit up to 45 minutes of audio for you within that. And yeah, there's some other options available as well if you start to add more stuff, but just come check us out. You can go to humblepod.com/services/quick-edit and find that there.Corey: And we will, of course, put links to that in the show notes. Or at least you will. I certainly won't.Chris: [laugh].Corey: Chris, thank you so much for taking the time to speak with me. If people want to learn more, other than hunting you down at re:Invent, which they absolutely should do, where's the best place for them to find you?Chris: I mean@HumblePod anywhere is the quickest, easiest way to find me anywhere—or at least find the business—and you can find me at @christopholies. And we'll have a link to that in the show notes for sure because it's not worth spelling out on the podcast.Corey: I would have pronounced it chris-to-files, but that's all right. That's how it works.Chris: [laugh].Corey: Thank you so much, Chris for everything that you do, as well as suffering my nonsensical slings and arrows for the last half hour. We'll talk soon.Chris: You're welcome, Corey.Corey: Chris Hill, CEO at HumblePod. I'm Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you hated this episode, please leave a five-star review on your podcast platform of choice along with an angry, insulting comment that I'm sure Chris or one of his colleagues will spend time hunting down from all corners of the internet to put into a delightful report, which I will then never read.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.

Adnan Khan, Lead Security Engineer at Praetorian, joins Corey on Screaming in the Cloud to discuss software bill of materials and supply chain attacks. Adnan describes how simple pull requests can lead to major security breaches, and how to best avoid those vulnerabilities. Adnan and Corey also discuss the rapid innovation at Github Actions, and the pros and cons of having new features added so quickly when it comes to security. Adnan also discusses his view on the state of AI and its impact on cloud security. About AdnanAdnan is a Lead Security Engineer at Praetorian. He is responsible for executing on Red-Team Engagements as well as developing novel attack tooling in order to meet and exceed engagement objectives and provide maximum value for clients.His past experience as a software engineer gives him a deep understanding of where developers are likely to make mistakes, and has applied this knowledge to become an expert in attacks on organization's CI/CD systems.Links Referenced: Praetorian: https://www.praetorian.com/ Twitter: https://twitter.com/adnanthekhan Praetorian blog posts: https://www.praetorian.com/author/adnan-khan/ TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Are you navigating the complex web of API management, microservices, and Kubernetes in your organization? Solo.io is here to be your guide to connectivity in the cloud-native universe!Solo.io, the powerhouse behind Istio, is revolutionizing cloud-native application networking. They brought you Gloo Gateway, the lightweight and ultra-fast gateway built for modern API management, and Gloo Mesh Core, a necessary step to secure, support, and operate your Istio environment.Why struggle with the nuts and bolts of infrastructure when you can focus on what truly matters - your application. Solo.io's got your back with networking for applications, not infrastructure. Embrace zero trust security, GitOps automation, and seamless multi-cloud networking, all with Solo.io.And here's the real game-changer: a common interface for every connection, in every direction, all with one API. It's the future of connectivity, and it's called Gloo by Solo.io.DevOps and Platform Engineers, your journey to a seamless cloud-native experience starts here. Visit solo.io/screaminginthecloud today and level up your networking game.Corey: As hybrid cloud computing becomes more pervasive, IT organizations need an automation platform that spans networks, clouds, and services—while helping deliver on key business objectives. Red Hat Ansible Automation Platform provides smart, scalable, sharable automation that can take you from zero to automation in minutes. Find it in the AWS Marketplace.Corey: Welcome to Screaming in the Cloud, I'm Corey Quinn. I've been studiously ignoring a number of buzzword, hype-y topics, and it's probably time that I addressed some of them. One that I've been largely ignoring, mostly because of its prevalence at Expo Hall booths at RSA and other places, has been software bill of materials and supply chain attacks. Finally, I figured I would indulge the topic. Today I'm speaking with Adnan Khan, lead security engineer at Praetorian. Adnan, thank you for joining me.Adnan: Thank you so much for having me.Corey: So, I'm trying to understand, on some level, where the idea of these SBOM or bill-of-material attacks have—where they start and where they stop. I've seen it as far as upstream dependencies have a vulnerability. Great. I've seen misconfigurations in how companies wind up configuring their open-source presences. There have been a bunch of different, it feels almost like orthogonal concepts to my mind, lumped together as this is a big scary thing because if we have a big single scary thing we can point at, that unlocks budget. Am I being overly cynical on this or is there more to it?Adnan: I'd say there's a lot more to it. And there's a couple of components here. So first, you have the SBOM-type approach to security where organizations are looking at which packages are incorporated into their builds. And vulnerabilities can come out in a number of ways. So, you could have software actually have bugs or you could have malicious actors actually insert backdoors into software.I want to talk more about that second point. How do malicious actors actually insert backdoors? Sometimes it's compromising a developer. Sometimes it's compromising credentials to push packages to a repository, but other times, it could be as simple as just making a pull request on GitHub. And that's somewhere where I've spent a bit of time doing research, building off of techniques that other people have documented, and also trying out some attacks for myself against two Microsoft repositories and several others that have reported over the last few months that would have been able to allow an attacker to slip a backdoor into code and expand the number of projects that they are able to attack beyond that.Corey: I think one of the areas that we've seen a lot of this coming from has been the GitHub Action space. And I'll confess that I wasn't aware of a few edge-case behaviors around this. Most of my experience with client-side Git configuration in the .git repository—pre-commit hooks being a great example—intentionally and by design from a security perspective, do not convey when you check that code in and push it somewhere, or grab someone else's, which is probably for the best because otherwise, it's, “Oh yeah, just go ahead and copy your password hash file and email that to something else via a series of arcane shell script stuff.” The vector is there. I was unpleasantly surprised somewhat recently to discover that when I cloned a public project and started running it locally and then adding it to my own fork, that it would attempt to invoke a whole bunch of GitHub Actions flows that I'd never, you know, allowed it to do. That was… let's say, eye-opening.Adnan: [laugh]. Yeah. So, on the particular topic of GitHub Actions, the pull request as an attack vector, like, there's a lot of different forms that an attack can take. So, one of the more common ones—and this is something that's been around for just about as long as GitHub Actions has been around—and this is a certain trigger called ‘pull request target.' What this means is that when someone makes a pull request against the base repository, maybe a branch within the base repository such as main, that will be the workflow trigger.And from a security's perspective, when it runs on that trigger, it does not require approval at all. And that's something that a lot of people don't really realize when they're configuring their workflows. Because normally, when you have a pull request trigger, the maintainer can check a box that says, “Oh, require approval for all external pull requests.” And they think, “Great, everything needs to be approved.” If someone tries to add malicious code to run that's on the pull request target trigger, then they can look at the code before it runs and they're fine.But in a pull request target trigger, there is no approval and there's no way to require an approval, except for configuring the workflow securely. So, in this case, what happens is, and in one particular case against the Microsoft repository, this was a Microsoft reusable GitHub Action called GPT Review. It was vulnerable because it checked out code from my branch—so if I made a pull request, it checked out code from my branch, and you could find this by looking at the workflow—and then it ran tests on my branch, so it's running my code. So, by modifying the entry points, I could run code that runs in the context of that base branch and steal secrets from it, and use those to perform malicious Actions.Corey: Got you. It feels like historically, one of the big threat models around things like this is al—[and when 00:06:02] you have any sort of CI/CD exploit—is either falls down one of two branches: it's either the getting secret access so you can leverage those credentials to pivot into other things—I've seen a lot of that in the AWS space—or more boringly, and more commonly in many cases, it seems to be oh, how do I get it to run this crypto miner nonsense thing, with the somewhat large-scale collapse of crypto across the board, it's been convenient to see that be less prevalent, but still there. Just because you're not making as much money means that you'll still just have to do more of it when it's all in someone else's account. So, I guess it's easier to see and detect a lot of the exploits that require a whole bunch of compute power. The, oh by the way, we stole your secrets and now we're going to use that to lateral into an organization seem like it's something far more… I guess, dangerous and also sneaky.Adnan: Yeah, absolutely. And you hit the nail on the head there with sneaky because when I first demonstrated this, I made a test account, I created a PR, I made a couple of Actions such as I modified the name of the release for the repository, I just put a little tag on it, and didn't do any other changes. And then I also created a feature branch in one of Microsoft's repositories. I don't have permission to do that. That just sat there for about almost two weeks and then someone else exploited it and then they responded to it.So, sneaky is exactly the word you could describe something like this. And another reason why it's concerning is, beyond the secret disclosure for—and in this case, the repository only had an OpenAI API key, so… okay, you can talk to ChatGPT for free. But this was itself a Github Action and it was used by another Microsoft machine-learning project that had a lot more users, called SynapseML, I believe was the name of the other project. So, what someone could do is backdoor this Action by creating a commit in a feature branch, which they can do by stealing the built-in GitHub token—and this is something that all Github Action runs have; the permissions for it vary, but in this case, it had the right permissions—attacker could create a new branch, modify code in that branch, and then modify the tag, which in Git, tags are mutable, so you can just change the commit the tag points to, and now, every time that other Microsoft repository runs GPT Review to review a pull request, it's running attacker-controlled code, and then that could potentially backdoor that other repository, steal secrets from that repository.So that's, you know, one of the scary parts of, in particular backdooring a Github Action. And I believe there was a very informative Blackhat talk this year, that someone from—I'm forgetting the name of the author, but it was a very good watch about how Actions vulnerabilities can be vulnerable, and this is kind of an example of—it just happened to be that this was an Action as well.Corey: That feels like this is an area of exploit that is becoming increasingly common. I tie it almost directly to the rise of GitHub Actions as the default CI/CD system that a lot of folks have been using. For the longest time, it seemed like a poorly configured Jenkins box hanging out somewhere in your environment that was the exception to the Infrastructure as Code rule because everyone has access to it, configures it by hand, and invariably it has access to production was the way that people would exploit things. For a while, you had CircleCI and Travis-CI, before Travis imploded and Circle did a bunch of layoffs. Who knows where they're at these days?But it does seem that the common point now has been GitHub Actions, and a .github folder within that Git repo with a workflows YAML file effectively means that a whole bunch of stuff can happen that you might not be fully aware of when you're cloning or following along with someone's tutorial somewhere. That has caught me out in a couple of strange ways, but nothing disastrous because I do believe in realistic security boundaries. I just worry how much of this is the emerging factor of having a de facto standard around this versus something that Microsoft has actively gotten wrong. What's your take on it?Adnan: Yeah. So, my take here is that Github could absolutely be doing a lot more to help prevent users from shooting themselves in the foot. Because their documentation is very clear and quite frankly, very good, but people aren't warned when they make certain configuration settings in their workflows. I mean, GitHub will happily take the settings and, you know, they hit commit, and now the workflow could be vulnerable. There's no automatic linting of workflows, or a little suggestion box popping up like, “Hey, are you sure you want to configure it this way?”The technology to detect that is there. There's a lot of third-party utilities that will lint Actions workflows. Heck, for looking for a lot of these pull request target-type vulnerabilities, I use a Github code search query. It's just a regular expression. So, having something that at least nudges users to not make that mistake would go really far in helping people not make these mista—you know, adding vulnerabilities to their projects.Corey: It seems like there's also been issues around the GitHub Actions integration approach where OICD has not been scoped correctly a bunch of times. I've seen a number of articles come across my desk in that context and fortunately, when I wound up passing out the ability for one of my workflows to deploy to my AWS account, I got it right because I had no idea what I was doing and carefully followed the instructions. But I can totally see overlooking that one additional parameter that leaves things just wide open for disaster.Adnan: Yeah, absolutely. That's one where I haven't spent too much time actually looking for that myself, but I've definitely read those articles that you mentioned, and yeah, it's very easy for someone to make that mistake, just like, it's easy for someone to just misconfigure their Action in general. Because in some of the cases where I found vulnerabilities, there would actually be a commit saying, “Hey, I'm making this change because the Action needs access to these certain secrets. And oh, by the way, I need to update the checkout steps so it actually checks out the PR head so that it's [testing 00:12:14] that PR code.” Like, people are actively making a decision to make it vulnerable because they don't realize the implication of what they've just done.And in the second Microsoft repository that I found the bug in, was called Microsoft Confidential Sidecar Containers. That repository, the developer a week prior to me identifying the bug made a commit saying that we're making a change and it's okay because it requires approval. Well, it doesn't because it's a pull request target.Corey: Part of me wonders how much of this is endemic to open-source as envisioned through enterprises versus my world of open-source, which is just eh, I've got this weird side project in my spare time, and it seemed like it might be useful to someone else, so I'll go ahead and throw it up there. I understand that there's been an awful lot of commercialization of open-source in recent years; I'm not blind to that fact, but it also seems like there's a lot of companies playing very fast and loose with things that they probably shouldn't be since they, you know, have more of a security apparatus than any random contributors standing up a clone of something somewhere will.Adnan: Yeah, we're definitely seeing this a lot in the machine-learning space because of companies that are trying to move so quickly with trying to build things because OpenAI AI has blown up quite a bit recently, everyone's trying to get a piece of that machine learning pie, so to speak. And another thing of what you're seeing is, people are deploying self-hosted runners with Nvidia, what is it, the A100, or—it's some graphics card that's, like, $40,000 apiece attached to runners for running integration tests on machine-learning workflows. And someone could, via a pull request, also just run code on those and mine crypto.Corey: I kind of miss the days when exploiting computers is basically just a way for people to prove how clever they were or once in a blue moon come up with something innovative. Now, it's like, well, we've gone all around the mulberry bush just so we can basically make computers solve a sudoku form, and in return, turn that into money down the road. It's frustrating, to put it gently.Adnan: [laugh].Corey: When you take a look across the board at what companies are doing and how they're embracing the emerging capabilities inherent to these technologies, how do you avoid becoming a cautionary tale in the space?Adnan: So, on the flip side of companies having vulnerable workflows, I've also seen a lot of very elegant ways of writing secure workflows. And some of the repositories are using deployment environments—which is the GitHub Actions feature—to enforce approval checks. So, workflows that do need to run on pull request target because of the need to access secrets for pull requests will have a step that requires a deployment environment to complete, and that deployment environment is just an approval and it doesn't do anything. So essentially, someone who has permissions to the repository will go in, approve that environment check, and only then will the workflow continue. So, that adds mandatory approvals to pull requests where otherwise they would just run without approval.And this is on, particularly, the pull request target trigger. Another approach is making it so the trigger is only running on the label event and then having a maintainer add a label so the tests can run and then remove the label. So, that's another approach where companies are figuring out ways to write secure workflows and not leave their repositories vulnerable.Corey: It feels like every time I turn around, Github Actions has gotten more capable. And I'm not trying to disparage the product; it's kind of the idea of what we want. But it also means that there's certainly not an awareness in the larger community of how these things can go awry that has kept up with the pace of feature innovation. How do you balance this without becoming the Department of No?Adnan: [laugh]. Yeah, so it's a complex issue. I think GitHub has evolved a lot over the years. Actions, it's—despite some of the security issues that happen because people don't configure them properly—is a very powerful product. For a CI/CD system to work at the scale it does and allow so many repositories to work and integrate with everything else, it's really easy to use. So, it's definitely something you don't want to take away or have an organization move away from something like that because they are worried about the security risks.When you have features coming in so quickly, I think it's important to have a base, kind of like, a mandatory reading. Like, if you're a developer that writes and maintains an open-source software, go read through this document so you can understand the do's and don'ts instead of it being a patchwork where some people, they take a good security approach and write secure workflows and some people just kind of stumble through Stack Overflow, find what works, messes around with it until their deployment is working and their CI/CD is working and they get the green checkmark, and then they move on to their never-ending list of tasks that—because they're always working on a deadline.Corey: Reminds me of a project I saw a few years ago when it came out that Volkswagen had been lying to regulators. It was a framework someone built called ‘Volkswagen' that would detect if it was running inside of a CI/CD environment, and if so, it would automatically make all the tests pass. I have a certain affinity for projects like that. Another one was a tool that would intentionally degrade the performance of a network connection so you could simulate having a latent or stuttering connection with packet loss, and they call that ‘Comcast.' Same story. I just thought that it's fun seeing people get clever on things like that.Adnan: Yeah, absolutely.Corey: When you take a look now at the larger stories that are emerging in the space right now, I see an awful lot of discussion coming up that ties to SBOMs and understanding where all of the components of your software come from. But I chased some stuff down for fun once, and I gave up after 12 dependency leaps from just random open-source frameworks. I mean, I see the Dependabot problem that this causes as well, where whenever I put something on GitHub and then don't touch it for a couple of months—because that's how I roll—I come back and there's a whole bunch of terrifyingly critical updates that it's warning me about, but given the nature of how these things get used, it's never going to impact anything that I'm currently running. So, I've learned to tune it out and just ignore it when it comes in, which is probably the worst of all possible approaches. Now, if I worked at a bank, I should probably take a different perspective on this, but I don't.Adnan: Mm-hm. Yeah. And that's kind of a problem you see, not just with SBOMs. It's just security alerting in general, where anytime you have some sort of signal and people who are supposed to respond to it are getting too much of it, you just start to tune all of it out. It's like that human element that applies to so much in cybersecurity.And I think for the particular SBOM problem, where, yeah, you're correct, like, a lot of it… you don't have reachability because you're using a library for one particular function and that's it. And this is somewhere where I'm not that much of an expert in where doing more static source analysis and reachability testing, but I'm certain there are products and tools that offer that feature to actually prioritize SBOM-based alerts based on actual reachability versus just having an as a dependency or not.[midroll 00:20:00]Corey: I feel like, on some level, wanting people to be more cautious about what they're doing is almost shouting into the void because I'm one of the only folks I found that has made the assertion that oh yeah, companies don't actually care about security. Yes, they email you all the time after they failed to protect your security, telling you how much they care about security, but when you look at where they invest, feature velocity always seems to outpace investment in security approaches. And take a look right now at the hype we're seeing across the board when it comes to generative AI. People are excited about the capabilities and security is a distant afterthought around an awful lot of these things. I don't know how you drive a broader awareness of this in a way that sticks, but clearly, we haven't collectively found it yet.Adnan: Yeah, it's definitely a concern. When you see things on—like for example, you can look at Github's roadmap, and there's, like, a feature there that's, oh, automatic AI-based pull request handling. Okay, so does that mean one day, you'll have a GitHub-powered LLM just approve PRs based on whether it determines that it's a good improvement or not? Like, obviously, that's not something that's the case now, but looking forward to maybe five, six years in the future, in the pursuit of that ever-increasing velocity, could you ever have a situation where actual code contributions are reviewed fully by AI and then approved and merged? Like yeah, that's scary because now you have a threat actor that could potentially specifically tailor contributions to trick the AI into thinking they're great, but then it could turn around and be a backdoor that's being added to the code.Obviously, that's very far in the future and I'm sure a lot of things will happen before that, but it starts to make you wonder, like, if things are heading that way. Or will people realize that you need to look at security at every step of the way instead of just thinking that these newer AI systems can just handle everything?Corey: Let's pivot a little bit and talk about your day job. You're a lead security engineer at what I believe to be a security-focused consultancy. Or—Adnan: Yeah.Corey: If you're not a SaaS product. Everything seems to become a SaaS product in the fullness of time. What's your day job look like?Adnan: Yeah, so I'm a security engineer on Praetorian's red team. And my day-to-day, I'll kind of switch between application security and red-teaming. And that kind of gives me the opportunity to, kind of, test out newer things out in the field, but then also go and do more traditional application security assessments and code reviews, and reverse engineering to kind of break up the pace of work. Because red-teaming can be very fast and fast-paced and exciting, but sometimes, you know, that can lead to some pretty late nights. But that's just the nature of being on a red team [laugh].Corey: It feels like as soon as I get into the security space and start talking to cloud companies, they get a lot more defensive than when I'm making fun of, you know, bad service naming or APIs that don't make a whole lot of sense. It feels like companies have a certain sensitivity around the security space that applies to almost nothing else. Do you find, as a result, that a lot of the times when you're having conversations with companies and they figure out that, oh, you're a red team for a security researcher, oh, suddenly, we're not going to talk to you the way we otherwise might. We thought you were a customer, but nope, you can just go away now.Adnan: [laugh]. I personally haven't had that experience with cloud companies. I don't know if I've really tried to buy a lot. You know, I'm… if I ever buy some infrastructure from cloud companies as an individual, I just kind of sign up and put in my credit card. And, you know, they just, like, oh—you know, they just take my money. So, I don't really think I haven't really, personally run into anything like that yet [laugh].Corey: Yeah, I'm curious to know how that winds up playing out in some of these, I guess, more strategic, larger company environments. I don't get to see that because I'm basically a tiny company that dabbles in security whenever I stumble across something, but it's not my primary function. I just worry on some level one of these days, I'm going to wind up accidentally dropping a zero-day on Twitter or something like that, and suddenly, everyone's going to come after me with the knives. I feel like [laugh] at some point, it's just going to be a matter of time.Adnan: Yeah. I think when it comes to disclosing things and talking about techniques, the key thing here is that a lot of the things that I'm talking about, a lot of the things that I'll be talking about in some blog posts that have coming out, this is stuff that these companies are seeing themselves. Like, they recognize that these are security issues that people are introducing into code. They encourage people to not make these mistakes, but when it's buried in four links deep of documentation and developers are tight on time and aren't digging through their security documentation, they're just looking at what works, getting it to work and moving on, that's where the issue is. So, you know, from a perspective of raising awareness, I don't feel bad if I'm talking about something that the company itself agrees is a problem. It's just a lot of the times, their own engineers don't follow their own recommendations.Corey: Yeah, I have opinions on these things and unfortunately, it feels like I tend to learn them in some of the more unfortunate ways of, oh, yeah, I really shouldn't care about this thing, but I only learned what the norm is after I've already done something. This is, I think, the problem inherent to being small and independent the way that I tend to be. We don't have enough people here for there to be a dedicated red team and research environment, for example. Like, I tend to bleed over a little bit into a whole bunch of different things. We'll find out. So far, I've managed to avoid getting it too terribly wrong, but I'm sure it's just a matter of time.So, one area that I think seems to be a way that people try to avoid cloud issues is oh, I read about that in the last in-flight magazine that I had in front of me, and the cloud is super insecure, so we're going to get around all that by running our own infrastructure ourselves, from either a CI/CD perspective or something else. Does that work when it comes to this sort of problem?Adnan: Yeah, glad you asked about that. So, we've also seen open-s—companies that have large open-source presence on GitHub just opt to have self-hosted Github Actions runners, and that opens up a whole different Pandora's box of attacks that an attacker could take advantage of, and it's only there because they're using that kind of runner. So, the default GitHub Actions runner, it's just an agent that runs on a machine, it checks in with GitHub Actions, it pulls down builds, runs them, and then it waits for another build. So, these are—the default state is a non-ephemeral runner with the ability to fork off tasks that can run in the background. So, when you have a public repository that has a self-hosted runner attached to it, it could be at the organization level or it could be at the repository level.What an attacker can just do is create a pull request, modify the pull request to run on a self-hosted runner, write whatever they want in the pull request workflow, create a pull request, and now as long as they were a previous contributor, meaning you fixed a typo, you… that could be a such a, you know, a single character typo change could even cause that, or made a small contribution, now they create the pull request. The arbitrary job that they wrote is now picked up by that self-hosted runner. They can fork off it, process it to run in the background, and then that just continues to run, the job finishes, their pull request, they'll just—they close it. Business as usual, but now they've got an implant on the self-hosted runner. And if the runners are non-ephemeral, it's very hard to completely lock that down.And that's something that I've seen, there's quite a bit of that on GitHub where—and you can identify it just by looking at the run logs. And that's kind of comes from people saying, “Oh, let's just self-host our runners,” but they also don't configure that properly. And that opens them up to not only tampering with their repositories, stealing secrets, but now depending on where your runner is, now you potentially could be giving an attacker a foothold in your cloud environment.Corey: Yeah, that seems like it's generally a bad thing. I found that cloud tends to be more secure than running it yourself in almost every case, with the exception that once someone finds a way to break into it, there's suddenly a lot more eggs in a very large, albeit more secure, basket. So, it feels like it's a consistent trade-off. But as time goes on, it feels like it is less and less defensible, I think, to wind up picking out an on-prem strategy from a pure security point of view. I mean, there are reasons to do it. I'm just not sure.Adnan: Yeah. And I think that distinction to be made there, in particular with CI/CD runners is there's cloud, meaning you let your—there's, like, full cloud meaning you let your CI/CD provider host your infrastructure as well; there's kind of that hybrid approach you mentioned, where you're using a CI/CD provider, but then you're bringing your own cloud infrastructure that you think you could secure better; or you have your runners sitting in vCenter in your own data center. And all of those could end up being—both having a runner in your cloud and in your data center could be equally vulnerable if you're not segmenting builds properly. And that's the core issue that happens when you have a self-hosted runner is if they're not ephemeral, it's very hard to cut off all attack paths. There's always something an attacker can do to tamper with another build that'll have some kind of security impact. You need to just completely isolate your builds and that's essentially what you see in a lot of these newer guidances like the [unintelligible 00:30:04] framework, that's kind of the core recommendation of it is, like, one build, one clean runner.Corey: Yeah, that seems to be the common wisdom. I've been doing a lot of work with my own self-hosted runners that run inside of Lambda. Definitionally those are, of course, ephemeral. And there's a state machine that winds up handling that and screams bloody murder if there's a problem with it. So far, crossing fingers hoping it works out well.And I have a bounded to a very limited series of role permissions, and of course, its own account of constraint blast radius. But there's still—there are no guarantees in this. The reason I build it the way I do is that, all right, worst case someone can get access to this. The only thing they're going to have the ability to do is, frankly, run up my AWS bill, which is an area I have some small amount of experience with.Adnan: [laugh]. Yeah, yeah, that's always kind of the core thing where if you get into someone's cloud, like, well, just sit there and use their compute resources [laugh].Corey: Exactly. I kind of miss when that was the worst failure mode you had for these things.Adnan: [laugh].Corey: I really want to thank you for taking the time to speak with me today. If people want to learn more, where's the best place for them to find you?Adnan: I do have a Twitter account. Well, I guess you can call it Twitter anymore, but, uh—Corey: Watch me. Sure I can.Adnan: [laugh]. Yeah, so I'm on Twitter, and it's @adnanthekhan. So, it's like my first name with ‘the' and then K-H-A-N because, you know, my full name probably got taken up, like, years before I ever made a Twitter account. So, occasionally I tweet about GitHub Actions there.And on Praetorian's website, I've got a couple of blog posts. I have one—the one that really goes in-depth talking about the two Microsoft repository pull request attacks, and a couple other ones that are disclosed, will hopefully drop on the twenty—what is that, Tuesday? That's going to be the… that's the 26th. So, it should be airing on the Praetorian blog then. So, if you—Corey: Excellent. It should be out by the time this is published, so we will, of course, put a link to that in the [show notes 00:32:01]. Thank you so much for taking the time to speak with me today. I appreciate it.Adnan: Likewise. Thank you so much, Corey.Corey: Adnan Khan, lead security engineer at Praetorian. I'm Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, along with an insulting comment that's probably going to be because your podcast platform of choice is somehow GitHub Actions.Adnan: [laugh].Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.

Joe Karlsson, Data Engineer at Tinybird, joins Corey on Screaming in the Cloud to discuss what it's like working in the world of data right now and how he manages the overlap between his social media presence and career. Corey and Joe chat about the rise of AI and whether or not we're truly seeing advancements in that realm or just trendy marketing plays, and Joe shares why he feels data is getting a lot more attention these days and what it's like to work in data at this time. Joe also shares insights into how his mental health has been impacted by having a career and social media presence that overlaps, and what steps he's taken to mitigate the negative impact. About JoeJoe Karlsson (He/They) is a Software Engineer turned Developer Advocate at Tinybird. He empowers developers to think creatively when building data intensive applications through demos, blogs, videos, or whatever else developers need.Joe's career has taken him from building out database best practices and demos for MongoDB, architecting and building one of the largest eCommerce websites in North America at Best Buy, and teaching at one of the most highly-rated software development boot camps on Earth. Joe is also a TEDx Speaker, film buff, and avid TikToker and Tweeter.Links Referenced: Tinybird: https://www.tinybird.co/ Personal website: https://joekarlsson.com TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Are you navigating the complex web of API management, microservices, and Kubernetes in your organization? Solo.io is here to be your guide to connectivity in the cloud-native universe!Solo.io, the powerhouse behind Istio, is revolutionizing cloud-native application networking. They brought you Gloo Gateway, the lightweight and ultra-fast gateway built for modern API management, and Gloo Mesh Core, a necessary step to secure, support, and operate your Istio environment.Why struggle with the nuts and bolts of infrastructure when you can focus on what truly matters - your application. Solo.io's got your back with networking for applications, not infrastructure. Embrace zero trust security, GitOps automation, and seamless multi-cloud networking, all with Solo.io.And here's the real game-changer: a common interface for every connection, in every direction, all with one API. It's the future of connectivity, and it's called Gloo by Solo.io.DevOps and Platform Engineers, your journey to a seamless cloud-native experience starts here. Visit solo.io/screaminginthecloud today and level up your networking game.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn and I am joined today by someone from well, we'll call it the other side of the tracks, if I can—Joe: [laugh].Corey: —be blunt and disrespectful. Joe Karlsson is a data engineer at Tinybird, but I really got to know who he is by consistently seeing his content injected almost against my will over on the TikToks. Joe, how are you?Joe: I'm doing so well and I'm so sorry for anything I've forced down your throat online. Thanks for having me, though.Corey: Oh, it's always a pleasure to talk to you. No, the problem I've got with it is that when I'm in TikTok mode, I don't want to think about computers anymore. I want to find inane content that I can just swipe six hours away without realizing it because that's how I roll.Joe: TikTok is too smart, though. I think it knows that you are doing a lot of stuff with computers and even if you keep swiping away, it's going to keep serving it up to you.Corey: For a long time, it had me pinned as a lesbian, which was interesting. Which I suppose—Joe: [laugh]. It happened to me, too.Corey: Makes sense because I follow a lot of women who are creators in comics and the rest, but I'm not interested in the thirst trap approach. So, it's like, “Mmm, this codes as lesbian.” Then they started showing me ads for ADHD, which I thought was really weird until I'm—oh right. I'm on TikTok. And then they started recommending people that I'm surprised was able to disambiguate until I realized these people have been at my house and using TikTok from my IP address, which probably is going to get someone murdered someday, but it's probably easy to wind up doing an IP address match.Joe: I feel like I have to, like, separate what is me and what is TikTok, like, trying to serve it up because I've been on lesbian TikTok, too, ADHD, autism, like TikTok. And, like, is this who I am? I don't know. [unintelligible 00:02:08] bring it to my therapist.Corey: You're learning so much about yourself based upon an algorithm. Kind of wild, isn't it?Joe: [laugh]. Yeah, I think we may be a little, like, neuro-spicy, but I think it might be a little overblown with what TikTok is trying to diagnose us with. So, it's always good to just keep it in check, you know?Corey: Oh, yes. So, let's see, what's been going on lately? We had Google Next, which I think the industry largely is taking not seriously enough. For years, it felt like a try-hard, me too version of re:Invent. And this year, it really feels like it's coming to its own. It is defining itself as something other than oh, us too.Joe: I totally agree. And that's where you and I ran into recently, too. I feel like post-Covid I'm still, like, running into people I met on the internet in real life, and yeah, I feel like, yeah, re:Invent and Google Next are, like, the big ones.I totally agree. It feels like—I mean, it's definitely, like, heavily inspired by it. And it still feels like it's a little sibling in some ways, but I do feel like it's one of the best conferences I've been to since, like, a pre-Covid 2019 AWS re:Invent, just in terms of, like… who was there. The energy, the vibes, I feel like people were, like, having fun. Yeah, I don't know, it was a great conference this year.Corey: Usually, I would go to Next in previous years because it was a great place to go to hang out with AWS customers. These days, it feels like it's significantly more than that. It's, everyone is using everything at large scale. I think that is something that is not fully understood. You talk to companies that are, like, Netflix, famously all in on AWS. Yeah, they have Google stuff, too.Everyone does. I have Google stuff. I have a few things in Azure, for God's sake. It's one of those areas where everything starts to diffuse throughout a company as soon as you hire employee number two. And that is, I think, the natural order of things. The challenge, of course, is the narrative people try and build around it.Joe: Yep. Oh, totally. Multi-cloud's been huge for you know, like, starting to move up. And it's impossible not to. It was interesting seeing, like, Google trying to differentiate itself from Azure and AWS. And, Corey, I feel like you'd probably agree with this, too, AI was like, definitely the big buzzword that kept trying to, like—Corey: Oh, God. Spare me. And I say that, as someone who likes AI, I think that there's a lot of neat stuff lurking around and value hiding within generative AI, but the sheer amount of hype around it—and frankly—some of the crypto bros have gone crashing into the space, make me want to distance myself from it as far as humanly possible, just because otherwise, I feel like I get lumped in with that set. And I don't want that.Joe: Yeah, I totally agree. I know it feels like it's hard right now to, like, remain ungrifty, but, like, still, like—trying—I mean, everyone's trying to just, like, hammer in an AI perspective into every product they have. And I feel like a lot of companies, like, still don't really have a good use case for it. You're still trying to, like, figure that out. We're seeing some cool stuff.Honestly, the hard part for me was trying to differentiate between people just, like, bragging about OpenAI API addition they added to the core product or, like, an actual thing that's, like, AI is at the center of what it actually does, you know what I mean? Everything felt like it's kind of like tacked on some sort of AI perspective to it.Corey: One of the things that really is getting to me is that you have these big companies—Google and Amazon most notably—talk about how oh, well, we've actually been working with AI for decades. At this point, they keep trying to push out how long it's been. It's like, “Okay, then not for nothing, then why does”—in Amazon's case—“why does Alexa suck? If you've been working on it for this long, why is it so bad at all the rest?” It feels like they're trying to sprint out with a bunch of services that very clearly were not conceptualized until Chat-Gippity's breakthrough.And now it's oh, yeah, we're there, too. Us, too. And they're pivoting all the marketing around something that, frankly, they haven't demonstrated excellence with. And I feel like they're leaving a lot of their existing value proposition completely in the dust. It's, your customers are not using you because of the speculative future, forward-looking AI things; it's because you are able to solve business problems today in ways that are not highly speculative and are well understood. That's not nothing and there needs to be more attention paid to that. And I feel like there's this collective marketing tripping over itself to wrap itself in hype that does them no services.Joe: I totally agree. I feel like honestly, just, like, a marketing perspective, I feel like it's distracting in a lot of ways. And I know it's hot and it's cool, but it's like, I think it's harder right now to, like, stay focused to what you're actually doing well, as opposed to, like, trying to tack on some AI thing. And maybe that's great. I don't know.Maybe that's—honestly, maybe you're seeing some traction there. I don't know. But I totally agree. I feel like everyone right now is, like, selling a future that we don't quite have yet. I don't know. I'm worried that what's going to happen again, is what happened back in the IBM Watson days where everyone starts making bold—over-promising too much with AI until we see another AI winter again.Corey: Oh, the subtext is always, we can't wait to fire our entire customer service department. That one—Joe: Yeah.Corey: Just thrills me.Joe: [laugh].Corey: It's like, no, we're just going to get rid of junior engineers and just have senior engineers. Yeah, where do you think those people come from, by the way? We aren't—they aren't just emerging fully formed from the forehead of some god somewhere. And we're also seeing this wild divergence from reality. Remember, I fix AWS bills for a living. I see very large companies, very large AWS spend.The majority of spend remains on EC2 across the board. So, we don't see a lot of attention paid to that at re:Invent, even though it's the lion's share of everything. When we do contract negotiations, we talk about generative AI plan and strategy, but no one's saying, oh, yeah, we're spending 100 million a year right now on AWS but we should commit 250 because of all this generative AI stuff we're getting into. It's all small-scale experimentation and seeing if there's value there. But that's a far cry from being the clear winner what everyone is doing.I'd further like to point out that I can tell that there's a hype cycle in place and I'm trying to be—and someone's trying to scam me. As soon as there's a sense of you have to get on this new emerging technology now, now, now, now, now. I didn't get heavily into cloud till 2016 or so and I seem to have done all right with that. Whenever someone is pushing you to get into an emerging thing where it hasn't settled down enough to build a curriculum yet, I feel like there's time to be cautious and see what the actual truth is. Someone's selling something; if you can't spot the sucker, chances are, it's you.Joe: [laugh]. Corey, have you thought about making an AI large language model that will help people with their cloud bills? Maybe just feed it, like, your invoices [laugh].Corey: That has been an example, I've used a number of times with a variety of different folks where if AI really is all it's cracked up to be, then the AWS billing system is very much a bounded problem space. There's a lot of nuance and intricacy to it, but it is a finite set of things. Sure, [unintelligible 00:08:56] space is big. So, training something within those constraints and within those confines feels like it would be a terrific proof-of-concept for a lot of these things. Except that when I've experimented a little bit and companies have raised rounds to throw into this, it never quite works out because there's always human context involved. The, oh yeah, we're going to wind up turning off all those idle instances, except they're in idle—by whatever metric you're using—for a reason. And the first time you take production down, you're not allowed to save money anymore.Joe: Nope. That's such a good point. I agree. I don't know about you, Corey. I've been fretting about my job and, like, what I'm doing. I write a lot, I do a lot of videos, I'm programming a lot, and I think… obviously, we've been hearing a lot about, you know, if it's going to replace us or not. I honestly have been feeling a lot better recently about my job stability here. I don't know. I totally agree with you. There's always that, like, human component that needs to get added to it. But who knows, maybe it's going to get better. Maybe there'll be an AI-automated billing management tool, but it'll never be as good as you, Corey. Maybe it will. I don't know. [laugh].Corey: It knows who I am. When I tell it to write in the style of me and give it a blog post topic and some points I want to make, almost everything it says is wrong. But what I'll do is I'll copy that into a text editor, mansplain-correct the robot for ten minutes, and suddenly I've got the bones of a decent rough draft because. And yeah, I'll wind up plagiarizing three or four words in a row at most, but that's okay. I'm plagiarizing the thing that's plagiarizing from me and there's a beautiful symmetry to that. What I don't understand is some of the outreach emails and other nonsensical stuff I'll see where people are letting unsupervised AI just write things under their name and sending it out to people. That is anathema to me.Joe: I totally agree. And it might work today, it might work tomorrow, but, like, it's just a matter of time before something blows up. Corey, I'm curious. Like, personally, how do you feel about being in the ChatGPT, like, brain? I don't know, is that flattering? Does that make you nervous at all?Corey: Not really because it doesn't get it in a bunch of ways. And that's okay. I found the same problem with people. In my time on Twitter, when I started live-tweet shitposting about things—as I tend to do as my first love language—people will often try and do exactly that. The problem that I run into is that, “The failure mode of ‘clever' is ‘asshole,'” as John Scalzi famously said, and as a direct result of that, people wind up being mean and getting it wrong in that direction.It's not that I'm better than they are. It's, I had a small enough following, and no one knew who I was in my mean years, and I realized I didn't feel great making people sad. So okay, you've got to continue to correct the nosedive. But it is perilous and it is difficult to understand the nuance. I think occasionally when I prompt it correctly, it comes up with some amazing connections between things that I wouldn't have seen, but that's not the same thing as letting it write something completely unfettered.Joe: Yeah, I totally agree. The nuance definitely gets lost. It may be able to get, like, the tone, but I think it misses a lot of details. That's interesting.Corey: And other people are defending it when that hallucinates. Like, yeah, I understand there are people that do the same thing, too. Yeah, the difference is, in many cases, lying to me and passing it off otherwise is a firing offense in a lot of places. Because if you're going to be 19 out of 20 times, you're correct, but 5% wrong, you're going to bluff, I can't trust anything you tell me.Joe: Yeah. It definitely, like, brings your, like—the whole model into question.Corey: Also, remember that my medium for artistic creation is often writing. And I think that, on some level, these AI models are doing the same things that we do. There are still turns of phrase that I use that I picked up floating around Usenet in the mid-90s. And I don't remember who said it or the exact context, but these words and phrases have entered my lexicon and I'll use them and I don't necessarily give credit to where the first person who said that joke 30 years ago. But it's a—that is how humans operate. We are influenced by different styles of writing and learn from the rest.Joe: True.Corey: That's a bit different than training something on someone's artistic back catalog from a painting perspective and then emulating it, including their signature in the corner. Okay, that's a bit much.Joe: [laugh]. I totally agree.Corey: So, we wind up looking right now at the rush that is going on for companies trying to internalize their use of enterprise AI, which is kind of terrifying, and it all seems to come back to data.Joe: Yes.Corey: You work in the data space. How are you seeing that unfold?Joe: Yeah, I do. I've been, like, making speculations about the future of AI and data forever. I've had dreams of tools I've wanted forever, and I… don't have them yet. I don't think they're quite ready yet. I don't know, we're seeing things like—tha—I think people are working on a lot of problems.For example, like, I want AI to auto-optimize my database. I want it to, like, make indexes for me. I want it to help me with queries or optimizing queries. We're seeing some of that. I'm not seeing anyone doing particularly well yet. I think it's up in the air.I feel like it could be coming though soon, but that's the thing, though, too, like, I mean, if you mess up a query, or, like, a… large language model hallucinates a really shitty query for you, that could break your whole system really quickly. I feel like there still needs to be, like, a human being in the middle of it to, like, kind of help.Corey: I saw a blog post recently that AWS put out gave an example that just hard-coded a credential into it. And they said, “Don't do this, but for demonstration purposes, this is how it works.” Well, that nuance gets lost when you use that for AI training and that's, I think, in part, where you start seeing a whole bunch of the insecure crap these things spit out.Joe: Yeah, I totally agree. Well, I thought the big thing I've seen, too, is, like, large language models typically don't have a secure option and you're—the answer is, like, help train the model itself later on. I don't know, I'm sure, like, a lot of teams don't want to have their most secret data end up public on a large language model at some point in the future. Which is, like, a huge issue right now.Corey: I think that what we're seeing is that you still need someone with expertise in a given area to review what this thing spits out. It's great at solving a lot of the busy work stuff, but you still need someone who's conversant with the concepts to look at it. And that is, I think, something that turns into a large-scale code review, where everyone else just tends to go, “Oh, okay. We're—do this with code review.” “Oh, how big is the diff?” “50,000 lines.” “Looks good to me.” Whereas, “Three lines.” “I'm going to criticize that thing with four pages of text.” People don't want to do the deep-dive stuff, and—when there's a huge giant project that hits. So, they won't. And it'll be fine, right up until it isn't.Joe: Corey, you and I know people and developers, do you think it's irresponsible to put out there an example of how to do something like that, even with, like, an asterisk? I feel like someone's going to still go out and try to do that and probably push that to production.Corey: Of course they are.Joe: [laugh].Corey: I've seen this with some of my own code. I had something on Docker Hub years ago with a container that was called ‘Terrible Ideas.' And I'm sure being used in, like—it was basically the environment I use for a talk I gave around Git, which makes sense. And because I don't want to reset all the repositories back to the way they came from with a bunch of old commands, I just want a constrained environment that will be the same every time I give the talk. Awesome.I'm sure it's probably being run in production at a bank somewhere because why wouldn't it be? That's people. That's life. You're not supposed to just copy and paste from Chat-Gippity. You're supposed to do that from Stack Overflow like the rest of us. Where do you think your existing code's coming from in a lot of these shops?Joe: Yep. No, I totally agree. Yeah, I don't know. It'll be interesting to see how this shakes out with, like, people going to doing this stuff, or how honest they're going to be about it, too. I'm sure it's happening. I'm sure people are tripping over themselves right now, [adding 00:16:12].Corey: Oh, yeah. But I think, on some level, you're going to see a lot more grift coming out of this stuff. When you start having things that look a little more personalized, you can use it for spam purposes, you can use it for, I'm just going to basically copy and paste what this says and wind up getting a job on Upwork or something that is way more than I could handle myself, but using this thing, I'm going to wind up coasting through. Caveat emptor is always the case on that.Joe: Yeah, I totally agree.Corey: I mean, it's easy for me to sit here and talk about ethics. I believe strongly in doing the right thing. But I'm also not worried about whether I'm able to make rent this month or put food on the table. That's a luxury. At some point, like, a lot of that strips away and you do what you have to do to survive. I don't necessarily begrudge people doing these things until it gets to a certain point of okay, now you're not doing this to stay alive anymore. You're doing this to basically seek rent.Joe: Yeah, I agree. Or just, like, capitalize on it. I do think this is less—like, the space is less grifty than the crypto space, but as we've seen over and over and over and over again, in tech, there's a such a fine line between, like, a genuinely great idea, and somebody taking advantage of it—and other people—with that idea.Corey: I think that's one of those sad areas where you're not going to be able to fix human nature, regardless of the technology stack you bring to bear.Joe: Yeah, I totally agree.Corey: So, what else are you seeing these days that interesting? What excites you? What do you see that isn't getting enough attention in the space?Joe: I don't know, I guess I'm in the data space, I'm… the thing I think I do see a lot of is huge interest in data. Data right now is the thing that's come up. Like, I don't—that's the thing that's training these models and everyone trying to figure out what to do with these data, all these massive databases, data lakes, whatever. I feel like everyone's, kind of like, taking a second look at all of this data they've been collecting for years and haven't really known what to do with it and trying to figure out either, like, if you can make a model out of that, if you try to, like… level it up, whatever. Corey, you and I were joking around recently—you've had a lot of data people on here recently, too—I feel like us data folks are just getting extra loud right now. Or maybe there's just the data spaces, that's where the action's at right now.I don't know, the markets are really weird. Who knows? But um, I feel like data right now is super valuable and more so than ever. And even still, like, I mean, we're seeing, like, companies freaking out, like, Twitter and Reddit freaking out about accessing their data and who's using it and how. I don't know, I feel like there's a lot of action going on there right now.Corey: I think that there's a significant push from the data folks where, for a long time data folks were DBAs—Joe: Yeah.Corey: —let's be direct. And that role has continued to evolve in a whole bunch of different ways. It's never been an area I've been particularly strong in. I am not great at algorithmic complexity, it turns out, you can saturate some beefy instances with just a little bit of data if your queries are all terrible. And if you're unlucky—as I tend to be—and have an aura of destroying things, great, you probably don't want to go and make that what you do.Joe: [laugh]. It's a really good point. I mean, I don't know about, like, if you blow up data at a company, you're probably going to be in big trouble. And especially the scale we're talking about with most companies these days, it's super easy to either take down a server or generate an insane bill off of some shitty query.Corey: Oh, when I was at Reach Local years and years ago—my first Linux admin job—when I broke the web server farm, it was amusing; when I broke part of the data warehouse, nobody was laughing.Joe: [laugh]. I wonder why.Corey: It was a good faith mistake and that's fair. It was a convoluted series of things that set up and honestly, the way the company and my boss responded to me at the time set the course of the rest of my career. But it was definitely something that got my attention. It scares me. I'm a big believer in backups as a direct result.Joe: Yeah. Here's the other thing, too. Actually, our company, Tinybird, is working on versioning with your data sources right now and treating your data sources like Git, but I feel like even still today, most companies are just run by some DBA. There's, like, Mike down the hall is the one responsible keeping their SQL servers online, keeping them rebooted, and like, they're manually updating any changes on there.And I feel like, generally speaking across the industry, we're not taking data seriously. Which is funny because I'm with you on there. Like, I get terrified touching production databases because I don't want anything bad to happen to them. But if we could, like, make it easier to rollback or, like, handle that stuff, that would be so much easier for me and make it, like, less scary to deal with it. I feel like databases and, like, treating it as, like, a serious DevOps practice is not really—I'm not seeing enough of it. It's definitely, people are definitely doing it. Just, I want more.Corey: It seems like with data, there's a lack of iterative approaches to it. A line that someone came up with when I was working with them a decade and change ago was that you can talk about agile all you want, but when it comes to payments, everyone's doing waterfall. And it feels like, on some level, data's kind of the same.Joe: Yeah. And I don't know, like, how to fix it. I think everyone's just too scared of it to really touch it. Migrating over to a different version control, trying to make it not as manual, trying to iterate on it better, I think it's just—I don't blame them. It's hard, it really takes a long time, making sure everything, like, doesn't blow up while you're doing a migration is a pain in the ass. But I feel like that would make everyone's lives so much easier if, like, you could, like, treat it—understand your data and be able to rollback easier with it.Corey: When you take a look across the ecosystem now, are you finding that things have improved since the last time I was in the space, where the state of the art was, “Oh, we need some developer data. We either have this sanitized data somewhere or it's a copy of production that we move around, but only a small bit.” Because otherwise, we always found that oh, that's an extra petabyte of storage was going on someone's developer environment they messed up on three years ago, they haven't been here for two, and oops.Joe: I don't. I have not seen it. Again, that's so tricky, too. I think… yeah, the last time I, like, worked doing that was—usually you just have a really crappy version of production data on staging or development environments and it's hard to copy those over. I think databases are getting better for that.I've been working on, like, the real-time data space for a long time now, so copying data over and kind of streaming that over is a lot easier. I do think seeing, like, separating storage and compute can make it easier, too. But it depends on your data stack. Everyone's using everything all the time and it's super complicated to do that. I don't know about you, Corey, too. I'm sure you've seen, like, services people running, but I feel like we've made a switch as an industry from, like, monoliths to microservices.Now, we're kind of back in the monolith era, but I'm not seeing that happen in the database space. We're seeing, like, data meshing and lots of different databases. I see people who, like, see the value of data monoliths, but I don't see any actual progress in moving back to a single source of [truth of the data 00:23:02]. And I feel like the cat's kind of out of the bag on all the data existing everywhere, all the time, and trying to wrangle that up.Corey: This stuff is hard and there's no easy solution here. There just isn't.Joe: Yeah, there's no way. And embracing that chaos, I think, is going to be huge. I think you have to do it right now. Or trying to find some tool that can, like, wrangle up a bunch of things together and help work with them all at once. Products need to meet people where they're at, too. And, like, data is all over the place and I feel like we kind of have to, like, find tooling that can kind of help work with what you have.Corey: It's a constant challenge, but also a joy, so we'll give it that.Joe: [laugh].Corey: So, I have to ask. Your day job has you doing developer advocacy at Tinybird—Joe: Yes.Corey: But I had to dig in to find that out. It wasn't obvious based upon the TikToks and the Twitter nonsense and the rest. How do you draw the line between day job and you as a person shitposting on the internet about technology?Joe: Corey, I'd be curious to hear your thoughts on this, too. I don't know. I feel like I've been in different places where, like, my job is my life. You know what I mean? There's a very thin line there. Personally, I've been trying to take a step back from that, just from a mental health perspective. Having my professional life be so closely tied to, like, my personal value and who I am has been really bad for my brain.And trying to make that clear at my company is, like, what is mine and what I can help with has been really huge. I feel like the boundaries between myself and my job has gotten too thin. And for a while, I thought that was a great idea; it turns out that was not a great idea for my brain. It's so hard. So, I've been a software engineer and I've done full-time developer advocacy, and I felt like I had a lot more freedom to say what I wanted as, like, a full-time software engineer as opposed to being a developer advocate and kind of representing the company.Because the thing is, I'm always representing the company [online 00:24:56], but I'm not always working, which is kind of like—that—it's kind of a hard line. I feel like there's been, like, ways to get around it though with, like, less private shitposting about things that could piss off a CEO or infringe on an NDA or, you know, whatever, you know what I mean? Yeah, trying to, like, find that balance or trying to, like, use tools to try to separate that has been big. But I don't know, I've been—personally, I've been trying to step—like, start trying to make more of a boundary for that.Corey: Yeah. I don't have much of one, but I also own the company, so my approach doesn't necessarily work for other people. I don't advertise in public that I fix AWS bills very often. That's not the undercurrent to most of my jokes and the rest. Because the people who have that painful problem aren't generally in the audience directly and they certainly don't talk about it extensively.It's word of mouth. It's being fun and engaging so people stick around. And when I periodically do mention it that sort of sticks with them. And in the fullness of time, it works as a way of, “Oh, yeah, everyone knows what you're into. And yeah, when we have this problem, reaching out to you is our first thought.” But I don't know that it's possible to measure its effectiveness. I just know that works.Joe: Yeah. For me, it's like, don't be an asshole and teach don't sell are like, the two biggest things that I'm trying to do all the time. And the goal is not to, like, trick people into, like, thinking I'm not working for a company. I think I try to be transparent, or if, like, I happen to be talking about a product that I'm working for, I try to disclose that. But yeah, I don't know. For me, it's just, like, trying to build up a community of people who, like, understand what I'm trying to put out there. You know what I mean?Corey: Yeah, it's about what you want to be known for, on some level. Part of the problem that I've had for a long time is that I've been pulled in so many directions. [They're 00:26:34] like, “Oh, you're great. Where do I go to learn more?” It's like, “Well, I have this podcast, I have the newsletter, I have the other podcast that I do in the AWS Morning Brief. I have the duckbillgroup.com. I have lastweekinaws.com. I have a Twitter account. I had a YouTube thing for a while.”It's like, there's so many different ways to send people. It's like, what is the top-of-funnel? And for me, my answer has been, sign up for the newsletter at lastweekinaws.com. That keeps you apprised of everything else and you can dial it into taste. It's also, frankly, one of those things that doesn't require algorithmic blessing to continue to show up in people's inboxes. So far at least, we haven't seen algorithms have a significant impact on that, except when they spam-bin something. And it turns out when you write content people like, the providers get yelled at by their customers of, “Hey, I'm trying to read this. What's going on?” I had a couple of reach out to me asking what the hell happened. It's kind of fun.Joe: I love that. And, Corey, I think that's so smart, too. It's definitely been a lesson, I think, for me and a lot of people on—that are terminally online that, like, we don't own our social following on other platforms. With, like, the downfall of Twitter, like, I'm still posting on there, but we still have a bunch of stuff on there, but my… that following is locked in. I can't take that home. But, like, you still have your email newsletter. And I even feel it for tech companies who might be listening to this, too. I feel like owning your email list is, like, not the coolest thing, but I feel like it's criminally underrated, as, like, a way of talking to people.Corey: It doesn't matter what platforms change, what my personal situation changes, I am—like, whatever it is that I wind up doing next, whenever next happens, I'll need a platform to tell people about, and that's what I've been building. I value newsletter subscribers in a metric sense far more highly and weight them more heavily than I do Twitter followers. Anyone can click a follow and then never check Twitter again. Easy enough. Newsletters? Well, that winds up requiring a little bit extra work because we do confirmed opt-ins, for obvious reasons.And we never sell the list. We never—you can't transfer permission for, like that, and we obviously respect it when people say I don't want to hear from your nonsense anymore. Great. Cool. I don't want to send this to people that don't care. Get out of here.Joe: [laugh]. No, I think that's so smart.Corey: Podcasts are impossible on the other end, but I also—you know, I control the domain and that's important to me.Joe: Yeah.Corey: Why don't you build this on top of Substack? Because as soon as Substack pivots, I'm screwed.Joe: Yeah, yeah. Which we've—I think we've seen that they've tried to do, even with the Twitter clone that tried to build last couple years. I've been burned by so many other publishing platforms over and over and over again through the years. Like, Medium, yeah, I criminally don't trust any sort of tech publishing platform anymore that I don't own. [laugh]. But I also don't want to maintain it. It's such a fine line. I just want to, like, maintain something without having to, like, maintain all the infrastructure all the time, and I don't think that exists and I don't really trust anything to help me with that.Corey: You can on some level, I mean, I wind up parking in the newsletter stuff over at ConvertKit. But I can—I have moved it twice already. I could move it again if I needed to. It's about controlling the domain. I have something that fires off once or twice a day that backs up the entire subscriber list somewhere.I don't want to build my own system, but I can also get that in an export form wherever I need it to go. Frankly, I view it as the most valuable asset that I have here because I can always find a way to turn relationships and an audience into money. I can't necessarily find a way to go the opposite direction of, well have money. Time to buy an audience. Doesn't work that way.Joe: [laugh]. No, I totally agree. You know what I do like, though, is Threads, which has kind of fallen off, but I do love the idea of their federated following [and be almost 00:30:02] like, unlock that a little bit. I do think that that's probably going to be the future. And I have to say, I just care as someone who, like, makes shit online. I don't think 98% of people don't really care about that future, but I do. Just getting burned so often on social media platforms, it helps to then have a little bit of flexibility there.Corey: Oh, yeah. And I wish it were different. I feel like, at some level, Elon being Elon has definitely caused a bit of a diaspora of social media and I think that's a good thing.Joe: Yeah. Yeah. I hope it settles down a little bit, but it definitely got things moving again.Corey: Oh, yes. I really want to thank you for taking the time to go through how you view these things. Where's the best place for people to go to follow you learn more, et cetera? Just sign up for TikTok and you'll be all over them, apparently.Joe: Go to the website that I own joekarlsson.com. It's got the links to everything on there. Opt in or out of whatever you find you want. Otherwise, I'm just going to quick plug for the company I work for: tinybird.co. If you're trying to make APIs on top of data, definitely want to check out Tinybird. We work with Kafka, BigQuery, S3, all the data sources could pull it in. [unintelligible 00:31:10] on it and publishes it as an API. It's super easy. Or you could just ignore me. That's fine, too. You could—that's highly encouraged as well.Corey: Always a good decision.Joe: [laugh]. Yeah, I agree. I'm biased, but I agree.Corey: Thanks, Joe. I appreciate your taking the time to speak with me and we'll, of course, put links to all that in the [show notes 00:31:26]. And please come back soon and regale us with more stories.Joe: I will. Thanks, Corey.Corey: Joe Karlsson, data engineer at Tinybird. I'm Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, along with an insulting comment that I'll never read because they're going to have a disk problem and they haven't learned the lesson of backups yet.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started. Tinybird: https://www.tinybird.co/ Personal website: https://joekarlsson.com TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn and I am joined today by someone from well, we'll call it the other side of the tracks, if I can—Joe: [laugh].Corey: —be blunt and disrespectful. Joe Karlsson is a data engineer at Tinybird, but I really got to know who he is by consistently seeing his content injected almost against my will over on the TikToks. Joe, how are you?Joe: I'm doing so well and I'm so sorry for anything I've forced down your throat online. Thanks for having me, though.Corey: Oh, it's always a pleasure to talk to you. No, the problem I've got with it is that when I'm in TikTok mode, I don't want to think about computers anymore. I want to find inane content that I can just swipe six hours away without realizing it because that's how I roll.Joe: TikTok is too smart, though. I think it knows that you are doing a lot of stuff with computers and even if you keep swiping away, it's going to keep serving it up to you.Corey: For a long time, it had me pinned as a lesbian, which was interesting. Which I suppose—Joe: [laugh]. It happened to me, too.Corey: Makes sense because I follow a lot of women who are creators in comics and the rest, but I'm not interested in the thirst trap approach. So, it's like, “Mmm, this codes as lesbian.” Then they started showing me ads for ADHD, which I thought was really weird until I'm—oh right. I'm on TikTok. And then they started recommending people that I'm surprised was able to disambiguate until I realized these people have been at my house and using TikTok from my IP address, which probably is going to get someone murdered someday, but it's probably easy to wind up doing an IP address match.Joe: I feel like I have to, like, separate what is me and what is TikTok, like, trying to serve it up because I've been on lesbian TikTok, too, ADHD, autism, like TikTok. And, like, is this who I am? I don't know. [unintelligible 00:02:08] bring it to my therapist.Corey: You're learning so much about yourself based upon an algorithm. Kind of wild, isn't it?Joe: [laugh]. Yeah, I think we may be a little, like, neuro-spicy, but I think it might be a little overblown with what TikTok is trying to diagnose us with. So, it's always good to just keep it in check, you know?Corey: Oh, yes. So, let's see, what's been going on lately? We had Google Next, which I think the industry largely is taking not seriously enough. For years, it felt like a try-hard, me too version of re:Invent. And this year, it really feels like it's coming to its own. It is defining itself as something other than oh, us too.Joe: I totally agree. And that's where you and I ran into recently, too. I feel like post-Covid I'm still, like, running into people I met on the internet in real life, and yeah, I feel like, yeah, re:Invent and Google Next are, like, the big ones.I totally agree. It feels like—I mean, it's definitely, like, heavily inspired by it. And it still feels like it's a little sibling in some ways, but I do feel like it's one of the best conferences I've been to since, like, a pre-Covid 2019 AWS re:Invent, just in terms of, like… who was there. The energy, the vibes, I feel like people were, like, having fun. Yeah, I don't know, it was a great conference this year.Corey: Usually, I would go to Next in previous years because it was a great place to go to hang out with AWS customers. These days, it feels like it's significantly more than that. It's, everyone is using everything at large scale. I think that is something that is not fully understood. You talk to companies that are, like, Netflix, famously all in on AWS. Yeah, they have Google stuff, too.Everyone does. I have Google stuff. I have a few things in Azure, for God's sake. It's one of those areas where everything starts to diffuse throughout a company as soon as you hire employee number two. And that is, I think, the natural order of things. The challenge, of course, is the narrative people try and build around it.Joe: Yep. Oh, totally. Multi-cloud's been huge for you know, like, starting to move up. And it's impossible not to. It was interesting seeing, like, Google trying to differentiate itself from Azure and AWS. And, Corey, I feel like you'd probably agree with this, too, AI was like, definitely the big buzzword that kept trying to, like—Corey: Oh, God. Spare me. And I say that, as someone who likes AI, I think that there's a lot of neat stuff lurking around and value hiding within generative AI, but the sheer amount of hype around it—and frankly—some of the crypto bros have gone crashing into the space, make me want to distance myself from it as far as humanly possible, just because otherwise, I feel like I get lumped in with that set. And I don't want that.Joe: Yeah, I totally agree. I know it feels like it's hard right now to, like, remain ungrifty, but, like, still, like—trying—I mean, everyone's trying to just, like, hammer in an AI perspective into every product they have. And I feel like a lot of companies, like, still don't really have a good use case for it. You're still trying to, like, figure that out. We're seeing some cool stuff.Honestly, the hard part for me was trying to differentiate between people just, like, bragging about OpenAI API addition they added to the core product or, like, an actual thing that's, like, AI is at the center of what it actually does, you know what I mean? Everything felt like it's kind of like tacked on some sort of AI perspective to it.Corey: One of the things that really is getting to me is that you have these big companies—Google and Amazon most notably—talk about how oh, well, we've actually been working with AI for decades. At this point, they keep trying to push out how long it's been. It's like, “Okay, then not for nothing, then why does”—in Amazon's case—“why does Alexa suck? If you've been working on it for this long, why is it so bad at all the rest?” It feels like they're trying to sprint out with a bunch of services that very clearly were not conceptualized until Chat-Gippity's breakthrough.And now it's oh, yeah, we're there, too. Us, too. And they're pivoting all the marketing around something that, frankly, they haven't demonstrated excellence with. And I feel like they're leaving a lot of their existing value proposition completely in the dust. It's, your customers are not using you because of the speculative future, forward-looking AI things; it's because you are able to solve business problems today in ways that are not highly speculative and are well understood. That's not nothing and there needs to be more attention paid to that. And I feel like there's this collective marketing tripping over itself to wrap itself in hype that does them no services.Joe: I totally agree. I feel like honestly, just, like, a marketing perspective, I feel like it's distracting in a lot of ways. And I know it's hot and it's cool, but it's like, I think it's harder right now to, like, stay focused to what you're actually doing well, as opposed to, like, trying to tack on some AI thing. And maybe that's great. I don't know.Maybe that's—honestly, maybe you're seeing some traction there. I don't know. But I totally agree. I feel like everyone right now is, like, selling a future that we don't quite have yet. I don't know. I'm worried that what's going to happen again, is what happened back in the IBM Watson days where everyone starts making bold—over-promising too much with AI until we see another AI winter again.Corey: Oh, the subtext is always, we can't wait to fire our entire customer service department. That one—Joe: Yeah.Corey: Just thrills me.Joe: [laugh].Corey: It's like, no, we're just going to get rid of junior engineers and just have senior engineers. Yeah, where do you think those people come from, by the way? We aren't—they aren't just emerging fully formed from the forehead of some god somewhere. And we're also seeing this wild divergence from reality. Remember, I fix AWS bills for a living. I see very large companies, very large AWS spend.The majority of spend remains on EC2 across the board. So, we don't see a lot of attention paid to that at re:Invent, even though it's the lion's share of everything. When we do contract negotiations, we talk about generative AI plan and strategy, but no one's saying, oh, yeah, we're spending 100 million a year right now on AWS but we should commit 250 because of all this generative AI stuff we're getting into. It's all small-scale experimentation and seeing if there's value there. But that's a far cry from being the clear winner what everyone is doing.I'd further like to point out that I can tell that there's a hype cycle in place and I'm trying to be—and someone's trying to scam me. As soon as there's a sense of you have to get on this new emerging technology now, now, now, now, now. I didn't get heavily into cloud till 2016 or so and I seem to have done all right with that. Whenever someone is pushing you to get into an emerging thing where it hasn't settled down enough to build a curriculum yet, I feel like there's time to be cautious and see what the actual truth is. Someone's selling something; if you can't spot the sucker, chances are, it's you.Joe: [laugh]. Corey, have you thought about making an AI large language model that will help people with their cloud bills? Maybe just feed it, like, your invoices [laugh].Corey: That has been an example, I've used a number of times with a variety of different folks where if AI really is all it's cracked up to be, then the AWS billing system is very much a bounded problem space. There's a lot of nuance and intricacy to it, but it is a finite set of things. Sure, [unintelligible 00:08:56] space is big. So, training something within those constraints and within those confines feels like it would be a terrific proof-of-concept for a lot of these things. Except that when I've experimented a little bit and companies have raised rounds to throw into this, it never quite works out because there's always human context involved. The, oh yeah, we're going to wind up turning off all those idle instances, except they're in idle—by whatever metric you're using—for a reason. And the first time you take production down, you're not allowed to save money anymore.Joe: Nope. That's such a good point. I agree. I don't know about you, Corey. I've been fretting about my job and, like, what I'm doing. I write a lot, I do a lot of videos, I'm programming a lot, and I think… obviously, we've been hearing a lot about, you know, if it's going to replace us or not. I honestly have been feeling a lot better recently about my job stability here. I don't know. I totally agree with you. There's always that, like, human component that needs to get added to it. But who knows, maybe it's going to get better. Maybe there'll be an AI-automated billing management tool, but it'll never be as good as you, Corey. Maybe it will. I don't know. [laugh].Corey: It knows who I am. When I tell it to write in the style of me and give it a blog post topic and some points I want to make, almost everything it says is wrong. But what I'll do is I'll copy that into a text editor, mansplain-correct the robot for ten minutes, and suddenly I've got the bones of a decent rough draft because. And yeah, I'll wind up plagiarizing three or four words in a row at most, but that's okay. I'm plagiarizing the thing that's plagiarizing from me and there's a beautiful symmetry to that. What I don't understand is some of the outreach emails and other nonsensical stuff I'll see where people are letting unsupervised AI just write things under their name and sending it out to people. That is anathema to me.Joe: I totally agree. And it might work today, it might work tomorrow, but, like, it's just a matter of time before something blows up. Corey, I'm curious. Like, personally, how do you feel about being in the ChatGPT, like, brain? I don't know, is that flattering? Does that make you nervous at all?Corey: Not really because it doesn't get it in a bunch of ways. And that's okay. I found the same problem with people. In my time on Twitter, when I started live-tweet shitposting about things—as I tend to do as my first love language—people will often try and do exactly that. The problem that I run into is that, “The failure mode of ‘clever' is ‘asshole,'” as John Scalzi famously said, and as a direct result of that, people wind up being mean and getting it wrong in that direction.It's not that I'm better than they are. It's, I had a small enough following, and no one knew who I was in my mean years, and I realized I didn't feel great making people sad. So okay, you've got to continue to correct the nosedive. But it is perilous and it is difficult to understand the nuance. I think occasionally when I prompt it correctly, it comes up with some amazing connections between things that I wouldn't have seen, but that's not the same thing as letting it write something completely unfettered.Joe: Yeah, I totally agree. The nuance definitely gets lost. It may be able to get, like, the tone, but I think it misses a lot of details. That's interesting.Corey: And other people are defending it when that hallucinates. Like, yeah, I understand there are people that do the same thing, too. Yeah, the difference is, in many cases, lying to me and passing it off otherwise is a firing offense in a lot of places. Because if you're going to be 19 out of 20 times, you're correct, but 5% wrong, you're going to bluff, I can't trust anything you tell me.Joe: Yeah. It definitely, like, brings your, like—the whole model into question.Corey: Also, remember that my medium for artistic creation is often writing. And I think that, on some level, these AI models are doing the same things that we do. There are still turns of phrase that I use that I picked up floating around Usenet in the mid-90s. And I don't remember who said it or the exact context, but these words and phrases have entered my lexicon and I'll use them and I don't necessarily give credit to where the first person who said that joke 30 years ago. But it's a—that is how humans operate. We are influenced by different styles of writing and learn from the rest.Joe: True.Corey: That's a bit different than training something on someone's artistic back catalog from a painting perspective and then emulating it, including their signature in the corner. Okay, that's a bit much.Joe: [laugh]. I totally agree.Corey: So, we wind up looking right now at the rush that is going on for companies trying to internalize their use of enterprise AI, which is kind of terrifying, and it all seems to come back to data.Joe: Yes.Corey: You work in the data space. How are you seeing that unfold?Joe: Yeah, I do. I've been, like, making speculations about the future of AI and data forever. I've had dreams of tools I've wanted forever, and I… don't have them yet. I don't think they're quite ready yet. I don't know, we're seeing things like—tha—I think people are working on a lot of problems.For example, like, I want AI to auto-optimize my database. I want it to, like, make indexes for me. I want it to help me with queries or optimizing queries. We're seeing some of that. I'm not seeing anyone doing particularly well yet. I think it's up in the air.I feel like it could be coming though soon, but that's the thing, though, too, like, I mean, if you mess up a query, or, like, a… large language model hallucinates a really shitty query for you, that could break your whole system really quickly. I feel like there still needs to be, like, a human being in the middle of it to, like, kind of help.Corey: I saw a blog post recently that AWS put out gave an example that just hard-coded a credential into it. And they said, “Don't do this, but for demonstration purposes, this is how it works.” Well, that nuance gets lost when you use that for AI training and that's, I think, in part, where you start seeing a whole bunch of the insecure crap these things spit out.Joe: Yeah, I totally agree. Well, I thought the big thing I've seen, too, is, like, large language models typically don't have a secure option and you're—the answer is, like, help train the model itself later on. I don't know, I'm sure, like, a lot of teams don't want to have their most secret data end up public on a large language model at some point in the future. Which is, like, a huge issue right now.Corey: I think that what we're seeing is that you still need someone with expertise in a given area to review what this thing spits out. It's great at solving a lot of the busy work stuff, but you still need someone who's conversant with the concepts to look at it. And that is, I think, something that turns into a large-scale code review, where everyone else just tends to go, “Oh, okay. We're—do this with code review.” “Oh, how big is the diff?” “50,000 lines.” “Looks good to me.” Whereas, “Three lines.” “I'm going to criticize that thing with four pages of text.” People don't want to do the deep-dive stuff, and—when there's a huge giant project that hits. So, they won't. And it'll be fine, right up until it isn't.Joe: Corey, you and I know people and developers, do you think it's irresponsible to put out there an example of how to do something like that, even with, like, an asterisk? I feel like someone's going to still go out and try to do that and probably push that to production.Corey: Of course they are.Joe: [laugh].Corey: I've seen this with some of my own code. I had something on Docker Hub years ago with a container that was called ‘Terrible Ideas.' And I'm sure being used in, like—it was basically the environment I use for a talk I gave around Git, which makes sense. And because I don't want to reset all the repositories back to the way they came from with a bunch of old commands, I just want a constrained environment that will be the same every time I give the talk. Awesome.I'm sure it's probably being run in production at a bank somewhere because why wouldn't it be? That's people. That's life. You're not supposed to just copy and paste from Chat-Gippity. You're supposed to do that from Stack Overflow like the rest of us. Where do you think your existing code's coming from in a lot of these shops?Joe: Yep. No, I totally agree. Yeah, I don't know. It'll be interesting to see how this shakes out with, like, people going to doing this stuff, or how honest they're going to be about it, too. I'm sure it's happening. I'm sure people are tripping over themselves right now, [adding 00:16:12].Corey: Oh, yeah. But I think, on some level, you're going to see a lot more grift coming out of this stuff. When you start having things that look a little more personalized, you can use it for spam purposes, you can use it for, I'm just going to basically copy and paste what this says and wind up getting a job on Upwork or something that is way more than I could handle myself, but using this thing, I'm going to wind up coasting through. Caveat emptor is always the case on that.Joe: Yeah, I totally agree.Corey: I mean, it's easy for me to sit here and talk about ethics. I believe strongly in doing the right thing. But I'm also not worried about whether I'm able to make rent this month or put food on the table. That's a luxury. At some point, like, a lot of that strips away and you do what you have to do to survive. I don't necessarily begrudge people doing these things until it gets to a certain point of okay, now you're not doing this to stay alive anymore. You're doing this to basically seek rent.Joe: Yeah, I agree. Or just, like, capitalize on it. I do think this is less—like, the space is less grifty than the crypto space, but as we've seen over and over and over and over again, in tech, there's a such a fine line between, like, a genuinely great idea, and somebody taking advantage of it—and other people—with that idea.Corey: I think that's one of those sad areas where you're not going to be able to fix human nature, regardless of the technology stack you bring to bear.Joe: Yeah, I totally agree.[midroll 00:17:30]Corey: So, what else are you seeing these days that interesting? What excites you? What do you see that isn't getting enough attention in the space?Joe: I don't know, I guess I'm in the data space, I'm… the thing I think I do see a lot of is huge interest in data. Data right now is the thing that's come up. Like, I don't—that's the thing that's training these models and everyone trying to figure out what to do with these data, all these massive databases, data lakes, whatever. I feel like everyone's, kind of like, taking a second look at all of this data they've been collecting for years and haven't really known what to do with it and trying to figure out either, like, if you can make a model out of that, if you try to, like… level it up, whatever. Corey, you and I were joking around recently—you've had a lot of data people on here recently, too—I feel like us data folks are just getting extra loud right now. Or maybe there's just the data spaces, that's where the action's at right now.I don't know, the markets are really weird. Who knows? But um, I feel like data right now is super valuable and more so than ever. And even still, like, I mean, we're seeing, like, companies freaking out, like, Twitter and Reddit freaking out about accessing their data and who's using it and how. I don't know, I feel like there's a lot of action going on there right now.Corey: I think that there's a significant push from the data folks where, for a long time data folks were DBAs—Joe: Yeah.Corey: —let's be direct. And that role has continued to evolve in a whole bunch of different ways. It's never been an area I've been particularly strong in. I am not great at algorithmic complexity, it turns out, you can saturate some beefy instances with just a little bit of data if your queries are all terrible. And if you're unlucky—as I tend to be—and have an aura of destroying things, great, you probably don't want to go and make that what you do.Joe: [laugh]. It's a really good point. I mean, I don't know about, like, if you blow up data at a company, you're probably going to be in big trouble. And especially the scale we're talking about with most companies these days, it's super easy to either take down a server or generate an insane bill off of some shitty query.Corey: Oh, when I was at Reach Local years and years ago—my first Linux admin job—when I broke the web server farm, it was amusing; when I broke part of the data warehouse, nobody was laughing.Joe: [laugh]. I wonder why.Corey: It was a good faith mistake and that's fair. It was a convoluted series of things that set up and honestly, the way the company and my boss responded to me at the time set the course of the rest of my career. But it was definitely something that got my attention. It scares me. I'm a big believer in backups as a direct result.Joe: Yeah. Here's the other thing, too. Actually, our company, Tinybird, is working on versioning with your data sources right now and treating your data sources like Git, but I feel like even still today, most companies are just run by some DBA. There's, like, Mike down the hall is the one responsible keeping their SQL servers online, keeping them rebooted, and like, they're manually updating any changes on there.And I feel like, generally speaking across the industry, we're not taking data seriously. Which is funny because I'm with you on there. Like, I get terrified touching production databases because I don't want anything bad to happen to them. But if we could, like, make it easier to rollback or, like, handle that stuff, that would be so much easier for me and make it, like, less scary to deal with it. I feel like databases and, like, treating it as, like, a serious DevOps practice is not really—I'm not seeing enough of it. It's definitely, people are definitely doing it. Just, I want more.Corey: It seems like with data, there's a lack of iterative approaches to it. A line that someone came up with when I was working with them a decade and change ago was that you can talk about agile all you want, but when it comes to payments, everyone's doing waterfall. And it feels like, on some level, data's kind of the same.Joe: Yeah. And I don't know, like, how to fix it. I think everyone's just too scared of it to really touch it. Migrating over to a different version control, tr

  This week we explore what's new in Istio with core maintainers John Howard and Keith Mattix   Do you have something cool to share? Some questions? Let us know: - web: kubernetespodcast.com - mail: kubernetespodcast@google.com - twitter: @kubernetespod   News of the week Announcing Linkerd 2.14: Improved enterprise multi-cluster, Gateway API conformance, and more! Amazon to invest up to $4 billion in AI startup Anthropic KubeCon EU 2024 CFP is open until November 26th CNCF Security Slam NEW Certification: Istio Certified Associate (ICA) npm packages caught exfiltrating Kubernetes config, SSH keys Links from the interview Kubernetes Native Sidecars in Istio (Blog from Istio) Kubernetes v1.28: Introducing native sidecar containers Argo Workflows Apache Airflow Envoy Proxy Istio Ambient Mesh Introducing Rust-Based Ztunnel for Istio Ambient Service Mesh eBPF Kernel TLS HTTP Based Overlay Network Environment (HBONE) KubeCon EU 2023: “Future of Service Mesh - Sidecar or Sidecarless or Proxyless?” - Idit Levine & Yuval Kohavi, Solo.io; Keith Mattix II, Microsoft; Eric Van Norman, IBM; John Howard, Google Istio Ambient Waypoint Proxy Made Simple kiali.io Kubernetes Gateway API (Istio) Getting Started with Istio and Kubernetes Gateway API Istio Desitination Rule Announcing Istio's graduation within the CNCF Istio sails into the Cloud Native Computing Foundation (CNCF Blog)

Today, Woody is joined by Senior Principal Software Engineer Mitch Connors to discuss the world of containers and the frequent challenges posed.Mitch Connors is an experienced software engineer with a remarkable career spanning 18 years. Having worked for various companies, including renowned tech giants such as F5, Amazon, and Google, he brings a wealth of knowledge and expertise to his new role at Aviatrix.They discuss the transition from a large corporate environment to a startup, highlighting the expanded scope and opportunities at Aviatrix. They dive into the world of containers, Kubernetes, and Istio, exploring the reasons behind the success of containers in the industry and the challenges they pose for enterprise adoption. About Altitude and Host Woody: https://aviatrix.com/altitude/ Mitch's LinkedIn: https://www.linkedin.com/in/mitchconnors/Timestamps:[00:01:54] Change in scope moving from Google to startup.[00:04:58] Containers like cloud adoption for enterprises. Developers pioneer, policy systems lag, Aviatrix helps network admins with multicloud presence.[00:06:22] Container networks have frequent IP address changes, causing identification issues.[00:10:48] A Network engineer's job is more complex.[00:13:31] Scalability: VMs vs. containers in e-commerce.[00:18:49] Advancing technology with Google's involvement in Istio. Aviatrix offers a customer journey towards containerization.[00:22:03] Legacy systems endure, consider them in platform design.[00:26:35] Usability lessons: upgrading should be easy.

Guest is Grace Nguyen. Kubernetes 1.28 release lead and student at the University of Waterloo. Grace had to juggle exams and community work to bring Kubernetes 1.28 to life. We will get to know grace and learn what work went into release, where the theme come from and what's special about it Do you have something cool to share? Some questions? Let us know: - web: kubernetespodcast.com - mail: kubernetespodcast@google.com - twitter: @kubernetespod News of the week Docker Desktop 4.22 is live The CNCF announced the End User Technical Advisory Board The Go community released v1.21 Configu raised a $3M pre-seed round Links from the interview Grace Nguyen LinkedIn X Kubernetes SIG-Security Kubernetes 1.28 Planternetes API Awareness of SideCars Native SideCar containers in Istio pkgs.k8s.io: Kubernetes Community-Owned Package Repositories Expanding support skew between control plane and node components Non-Graceful node shutdown Pod replacement policy for Jobs (alpha) Match conditions for admission webhooks Feature graduations and deprecations in Kubernetes v1.28 Kubernetes 1.28 webinar. Sept 6th 2023 9am PDT Kubernetes 1.29 PR to assemble team Kubernetes 1.29 shadow program is open Kubernetes 1.27 release lead Xander Grzywinski Links from the post-interview chat Beta support for enabling swap space on Linux SideCars handling is the most popular issue on kubernetes tracker Reddit conversation about native SideCars Native SideCars explained

Bret and Nirmal welcome Idit Levine, Founder/CEO Solo.io. Idit focuses on Service Mesh, API-GW and Multi-Cloud networking, and security.Idit has been involved in the Containers/DevOps community for 10+ years, building products from Docker to Envoy to Kubernetes, and now Istio and Cilium. We talk about Istio, Ambient Mesh, Envoy, Zero-Trust Security, Cilium, eBPF, Multi-Cloud and more.This is not the first time we've talked about Solo or Service Mesh. Ambient Mesh is Solo's new product that simplifies the install and infrastructure costs of essentially running Istio. I'm really hopeful that this is going to help a lot more people implement Istio because traditionally, it does have a lot of parts and a lot of costs with the sidecar approach, but this new approach reduces the number of essentially proxies and parts that you're running on each node of your Kubernetes cluster. Live recording of the complete show from June 29, 2023 is on YouTube (Ep. #223).★Topics★Solo.ioIstio Ambient MeshSolo Academy (free courses)Istio Ambient Mesh ebookGloo FabricSupport this show and get exclusive benefits on Patreon, YouTube, or bretfisher.com!★Join my Community★Get on the waitlist for my next live course on CI automation and gitops deploymentsBest coupons for my Docker and Kubernetes coursesChat with us and fellow students on our Discord Server DevOps FansGrab some merch at Bret's Loot BoxHomepage bretfisher.comCreators & Guests Bret Fisher - Host Cristi Cotovan - Editor Beth Fisher - Producer Nirmal Mehta - Host Idit Levine - Guest (00:00) - Intro (03:59) - How did Solo.io start? (21:03) - The difference between service mesh and API gateway (30:55) - Where is service mesh going? (41:53) - Is Ambient Mesh as secure as the sidecar model? (48:11) - Opportunities after adopting Ambient Mesh (53:41) - Phipps compliance (55:46) - Unikernel vs WebAssembly

Kendall Miller is the Co-Founder and COO of CTO Lunches, a network of engineering leaders to get trusted advice and connections. The first half of the conversation with host Victoria Guido and special guest host, Joe Ferris, CTO of thoughtbot revolves around the use, adoption, and growth of Kubernetes within the technology industry. The discussion explores Kubernetes' history, influence, and its comparison with other platforms like Heroku and WordPress, emphasizing its adaptability and potential. The second half focuses on more practical aspects of Kubernetes, including its adoption and scalability. It centers on the appropriateness of adopting Kubernetes for different projects and how it can future-proof infrastructure. The importance of translating technical language into business speak is emphasized to influence executives and others in the decision-making process and Kendall also discuss communication and empathy in tech, particularly the skill of framing questions and understanding others' emotional states. __ CTO Lunches (http://ctolunches.com/) Follow CTO Lunches on LinkedIn (https://www.linkedin.com/company/ctolunches/) or Twitter (https://twitter.com/cto_lunches). Follow Kendall Miller on LinkedIn (https://www.linkedin.com/in/kendallamiller/). Follow thoughtbot on Twitter (https://twitter.com/thoughtbot) or LinkedIn (https://www.linkedin.com/company/150727/). Become a Sponsor (https://thoughtbot.com/sponsorship) of Giant Robots! Transcript: VICTORIA: This is the Giant Robots Smashing Into Other Giant Robots Podcast, where we explore the design, development, and business of great products. I'm your host, Victoria Guido. And with me today is Kendall Miller, Co-Founder and COO of CTO Lunches, a network of engineering leaders to get trusted advice and connections. Kendall, thank you for joining me. KENDALL: Thanks for having me. I'm excited. VICTORIA: And today, we have a special guest host, Joe Ferris, CTO of thoughtbot. Joe, thank you for joining us. JOE: Hello there. Thank you for having me. KENDALL: Hi, Joe. Thanks for being here. It's exciting. VICTORIA: Yes. It's so exciting. I think this is going to be a great episode. So, Kendall, I met you at a San Diego CTO lunch recently, and I know that's not the only thing that you do. So, you're also an advisor, a board member, and CXO. So, maybe tell us a little bit more about your background. KENDALL: Gosh, my background is complicated. I've been involved in tech for a very long time. In college, I worked for a company that started Twitter about five years too soon, and then worked in the nonprofit space in China for ten years, then came back, got back involved in tech. Today, I'm usually the business guy. So, when technical founders start technical products and want help turning them into successful technical businesses, that's when they call me. So, I have the technical background. I have never been paid to write code, which is probably a good thing. But I can hang in the technical conversations for the most part, but I'm much more interested in the business side and the people leadership side of business. So that tends to be where I play. Every organization hires me to do something different. VICTORIA: Thank you for that. And I'm just curious about the CTO Lunches. Just tell me a little bit more about that. And what's the idea behind it that led you to co-found it? KENDALL: CTO Lunches has actually been around for about eight years. And I didn't start the initial incarnation of it. It was two people that got us started, and I was trying to hire one of them; one thing led to another. Actually, originally, they did not want me to join. I think, at the time, my title was COO at a company that I was working with. About six months later, I took over engineering as VP of engineering, and then they're like, you can join the group now. We're less strict about that [laughs] now. Although it is highly focused on senior engineering leaders, it's not exclusively CTOs. But the group's been in place for a very long time, just intended as a place to network, have conversation with people who are in that senior-most technical position at technical organization. So, the CTO role is a lonely role. CTOs get fired all the time. There's not a technical person at the company that doesn't think they can do the job better than them. So, the CTO is always getting feedback. You're doing this wrong. The trade-offs you're making are wrong. This isn't going where it should be going. We should automate that. Why haven't we automated that? We should switch to this other tool. I've used it before; it's 100 times better. Joe, let me know if I'm getting any of this wrong. But that's the experience that I've had. Having a place where people can get together and, you know, half the time just complain to each other, hey, this is hard, is really why the networking group exists. So, it's a listserv. And there are local lunches that started in Boulder, Colorado. It's gotten pretty global. About a year ago, a little over a year ago, I was talking with one of the people who'd gotten it started. I've been involved in the Denver chapter for most of those eight years. And I was suggesting to him that he change a few things about it, to monetize it so that he could invest in it further. And he came back a few months later and said, "I want to take your advice and do this, but I want you to come do it with me." So, we founded the company officially...I think in December is when paperwork went into place. And we started investing in it a little bit more heavily. I was living in Europe last year, so we went and put on lunches in Paris, and Lisbon, and London, and, gosh, all over the place. I'm sure I'm missing some, Amsterdam. But there's been chapters all over the U.S. and a couple of other parts of the world for a long time. VICTORIA: That reflects my experience attending a CTO lunch. It's just very casual, like, just get together and eat food and talk about what you've worked on recently, issues you're having, just get ideas and make some friends. So, I really appreciated the group, and I'm going to personally plug the San Diego Chapter has picked up again. And we're meeting next Friday down in Del Mar. And we're going to be meeting on the last Friday of every month through October. So, I'm super excited to be a part of the group. And Joe, yeah, I'm curious about your perspective. As a CTO with thoughtbot, just what are your thoughts about that kind of thing? KENDALL: Yeah. How right am I about how lonely you are, Joe? JOE: [laughs] You know, I've been lonelier since we went remote. I used to work in the office, and I was a CTO, but also, I had lunch with people, which was nice. So, I'm lonelier. But yeah, I think everybody needs a group like that, like, senior developer therapy just to talk about your woes together, drown your sorrows. KENDALL: Well, I think years ago, I heard that CTOs are the most fired C-level executive. JOE: You're making me nervous now. KENDALL: [laughs] You've been there a long time, Joe. I know you've been there a long time. If you haven't been fired yet, you probably got a little while longer in you. This will be really awkward if it's published and you've already been fired. VICTORIA: We can always edit that out afterwards. [laughter] KENDALL: Yeah, no, I think it is a particularly lonely position. And, again, I think a lot of it is the average engineer in a technical company doesn't look at the COO or the CFO or even the CEO and think I could do that. But they're all looking at the CTO and thinking, what does that person do that I can't do? It's ridiculous because most of them would make terrible CTOs because it does require some of the business sense. Or, you know, right out of the gate, they might make terrible CTOs. It actually is quite a skill to be the most technical person and speak the business language. I mean, am I right about that, Joe? Like, was that hard for you to learn? JOE: Yeah, I definitely think...so, my background is also technical. I have a background in consulting. So, I always did a lot of metaprogramming, if you will. But making that transition to thinking about organizations that way, thinking about how all the other pieces play into it, was a pretty big step for me, even before I became a CTO as a consultant. KENDALL: Well, because you can't just chase the newest, hottest technology. You have to make business trade-offs. And not everything can be resume-driven development, right? Even if that technology over there is newer or hotter, it doesn't mean you have a business model that supports it. And it doesn't mean that migrating to it can be done, right? JOE: Yeah. I mean, even beyond choosing technologies, just choosing where to invest in your software stack, like, what needs to be reworked, what doesn't, and trying to explain those trade-offs, I think, is a rare skill. Being able to explain why something would be harder than something else when you're working with the leadership to prioritize a backlog it's a puzzle. KENDALL: Well, and I think when I'm in an executive conversation, and the CTO says, "Here's the thing that I think is the best decision technically, and I think it's the wrong decision for the business because of X, Y, or Z," I'm always super impressed, right? Like, this is the right technical solution for what we want. However, we shouldn't pursue that for business reasons right now. Maybe we can in six months, but right now, we need to prioritize this other thing. I don't know, that's always when I feel like, oh, this person knows what they're doing. JOE: There's nothing more dangerous to software than a bored developer. [laughs] One nice thing about being a consultant is that I don't have to invent problems to solve with technology at my company because sooner or later, I'll run across a company that has those problems, and I'll get to use that technology. But I think a lot of people are mostly happy...they might be happy in their role. They might be happy with our team. But they're very interested in whatever is hot right now, like machine learning, AI. And so, suddenly, that surreptitiously makes its way into the tech stack. And then, years later, it's somebody's problem to maintain. KENDALL: [laughs] Well, I have a specific memory of a firm in New York City that was, you know, this is relevant to y'all as thoughtbot is that, you know, at least historically, it was, to me, the premier Ruby on Rails consulting shop. I think that's still largely y'alls focus. Am I right about that? JOE: We still do a ton of Rails, yeah. KENDALL: Okay. Well, so this organization was all Ruby on Rails. It was a big organization. They had a very large customer base. And they hired a new CTO who came in, told everybody in the company they were stupid, laid off 70% of the engineering organization, and told the CEO he was going to completely rewrite the product from scratch in .NET, and he could do it in three weeks. And I'm pretty sure the business went under about three months later [laughs] because that was just so outrageously nuts to me. JOE: It's too bad he laid everybody off beforehand. I've been in that situation where somebody tells me, "I'm going to rewrite this. It'll be ready in three weeks." And I could fight with them and try and convince them they're wrong. But I feel like somebody who's approaching that with that attitude they're missing all of the nuance and context that would make it possible to explain to them why it's not going to work. And so, it's easier to just say, "You know, take the three weeks. I'll talk to you in three weeks." But if you've already laid off your development team, that's hard [laughs] to recover from. KENDALL: That's exactly right. VICTORIA: There's got to be a name for that kind of CTO who just wants to come in and blow everything up [laughs]. Yeah, so you spend a lot of time talking to different CTOs and doing this social networking aspect. I'm wondering if there's, like, patterns that you see. You've mentioned already one about just, like, the most often getting fired. [laughs] But what are the patterns you see, like, in challenges, and then what makes someone successful in that CTO role? KENDALL: Well, oh gosh, I have so many thoughts about this. First of all, I run into a couple of different categories of CTOs. There's a lot of people who come to CTO Lunches who are small company CTOs. I mean, it makes sense that there's a lot more small company CTOs than there are big company CTOs. But the small company CTO who maybe it's their first gig in the role or they're a serial CTO. There's the fractional CTOs that come that are doing it across several different organizations at the same time, and then there's the big company CTO who shows up. And honestly, all of their problems are very different. The thing that they have in common is even at a very large organization, in that position, they can make a decision that causes the company to go under. So, there is a significant amount of volatility in the amount of power that they wield. So, what's interesting about that is not everybody understands that. And so, first of all, there's the kind of CTO that just doesn't get that, and that doesn't matter if they're fractional, or a small company CTO, or a big company CTO. If they don't understand that, they're going to cause significant problems, right? Like the person I just mentioned who said, "I can just re-platform this in three weeks in .NET." There's that. I mean, I think, as with any senior leadership position, the comfort with volatility, the ability to know what to communicate down versus across and versus up, and then the ability to speak the business language. For everybody, the CFO's job is to communicate the financial needs alongside of the business leads, right? If the CFO's sole goal is to cut costs or make sure we're running as lean as possible, they're a bad CFO. But they're not as good of a CFO as the CFO who can say, "Hey, we're underspending right here. And I can look at the numbers and know we should invest more there. How can we invest more there and invest it well?" And it's the same thing for a technology executive to be able to look at the business context and communicate it back. And there are so many CTOs that I've worked with who they're the most technical person in the room, and they know it. And as a result, they're just a jerk to everyone around them, like, everything you did here was wrong. You know, that's where they fail. And so, if they can communicate the business needs, navigate the volatility, and support a team that's going to make decisions that aren't always the same decision they're going to make, they're going to be successful. Honestly, there's very, very few CTOs that I've met like that. People who are excited to meet you at work, excited to see you succeed, excited to see that you went and built a thing is great. I mean, the reason I was VP of engineering is the CTO that I was working with at the time...it's a terrible story. There was an engineer who had seen something that we were doing on repeat all the time and, in his spare time, spent about 40 hours outside of work, not during work hours, automating this task that we were doing regularly. And it was related to standing up a whole bunch of things in our standard infrastructure. He brings it to the CTO and says, "Look what I built." And the CTO, instead of saying, "Hey, this is incredible. Thank you. This is going to save us a bunch of time. Let's iterate on it. Here's some things I'd like to tweak. Can we bring it in this direction? Can we..." you know, whatever, said, "Why is this in Python? It should be in Ansible," something like that. I can't remember. And the engineer literally burst into tears. [laughs] JOE: Oh my God. KENDALL: [laughs] Well, I mean, yeah, it was like; literally, that's why the CTO stopped managing people that day. There's a lot of examples that I have like that. Joe, I appreciate that your response is, "Oh my God." Because I think there's a lot of people who'd be like, wait, what was wrong with that? Shouldn't it have been in Ansible? JOE: [laughs] Yeah, I've seen CTOs come into primarily two groups. One is the CTO who just tells, you know, like, they make the decisions, and they tell everybody what to do. They obviously don't have all of the information because you can't be in every room all the time. And the other is the CTO, who just wants to be one of the team members and doesn't make any decisions and tries to get people to make decisions collectively on their own without any particular guidance or structure. And finding that middle spot of, like, not just saying, "Hey, everything's in Ansible," allowing for the creativity and initiative, but also coalescing the group into a single direction, I think, is what makes a good CTO. KENDALL: Well, yeah, because the CTO does have to say no, sometimes, right? Like, the best product, people say, "No." Good CTOs say, "No." There is some amount of, hey, I need you to come to me with trade-offs about this. Why are you going to make that decision? And I'm sorry, you still didn't convince me, right? Like, I mean, those are appropriate things to say. But yeah, I'm with you on that. You said they fall into two categories. But you really mean the third and that middle ground. Is it easy for you to walk that middle ground, Joe? JOE: I wouldn't say it's easy. [laughs] KENDALL: Yeah. Well, I'm always nervous to say something. I'm doing well because I know there's a report out there that can point at every time I failed at it, right? So... MID-ROLL AD: Are you an entrepreneur or start-up founder looking to gain confidence in the way forward for your idea? At thoughtbot, we know you're tight on time and investment, which is why we've created targeted 1-hour remote workshops to help you develop a concrete plan for your product's next steps. Over four interactive sessions, we work with you on research, product design sprint, critical path, and presentation prep so that you and your team are better equipped with the skills and knowledge for success. Find out how we can help you move the needle at: tbot.io/entrepreneurs. VICTORIA: Yeah, what I'm getting from what you're saying, too, is this communication ability and not just, like, to communicate clearly but with a high level of empathy. So, if you say like, "Well, why is it in Python and not Ansible?" is different than being like, "What makes Python the best solution here?" Like, it's a different way to frame the question that could put someone on the defensive that just really requires, like, a high level of emotional intelligence. And also, if they've just worked, like, an 80-hour week, [laughs] I probably would maybe choose a different time to bring those questions up and notice that they have been burning the candle at both ends and prioritize getting them some rest. So, speaking of, like, communication and getting prioritization for [inaudible 15:34], especially on, like, infrastructure teams, maybe we could talk a little bit about Kubernetes, like, when that comes up as an appropriate solution, and how you talk about it with the business. KENDALL: My background with Kubernetes is long because a company that I still work with, Fairwinds, used to be called ReactiveOps, has been in the Kubernetes space for a very long time. I think we were one of the very first companies working with Kubernetes. It was coming up that people were running into the limits of something like Heroku, right? And I think it's Kelsey Hightower who said every company wants a PaaS. They just want the Paas that they built themselves. And that's really accurate. And I think Kubernetes isn't quite a framework for building your own PaaS or isn't quite a foundation where I think of a foundation for a house. Instead, it's more like rebar and cement and somebody saying, "Good luck, buddy." You know, you still have to know how to put the rebar and cement together to even make the foundation, but it is the building blocks that help get you to a custom-built PaaS. And it's become something that a lot of people have landed on as, you know, the broadly accepted way to build cloud-native infrastructure. The reason I've been in the Kubernetes space and the space that I see Kubernetes still filling is we need to standardize on something. We can choose a cloud provider's PaaS. We can choose a third-party PaaS, or we can standardize on something like Kubernetes. And even though we're not going to migrate from AWS to Azure, the flexibility that Kubernetes gives us as a broadly adopted pattern is going to give us some ability to be future-proofed in our infrastructure in a way that previous stacks were not, you know, it was Puppet, and it was Ansible. And it was SaltStack. And it was all Terraform all the time. I'm not saying those things don't exist anymore. I'm saying Kubernetes kind of has won that battle. Joe, since you're here and I know y'all are doing some Kubernetes work now at thoughtbot, I'm curious if you agree with that characterization. JOE: Yeah, I think that's true. I think it's the center for people to coalesce around. Like, for an effort in the industry to move forward, there needs to be some common language, some common ground. And I think Kubernetes struck the right balance of being abstract. So, you can use it in different environments but still making some decisions, so you don't have to make them all. And so, like, all of the things you had to do with containers like figuring out what your data solution is going to be, what your networking solution is going to be, Kubernetes didn't even really make those decisions. [laughs] They just made a platform where those decisions can be made in a common way. And that allowed the community and the ecosystem to grow. KENDALL: I mean, I think of it a lot like WordPress; you know, WordPress is hated by many. When WordPress came out, it was hot, right? And it was PHP, which everybody was super excited about at the time. Kubernetes is going to reach a point where it's as long in the tooth and terrible as people think WordPress is, but it has become the standard. And the advantage of the standard is you can use the not standard. You can go build a website in Jekyll instead of WordPress, and there's going to be some things that are nicer about Jekyll. But because WordPress is so broadly adopted, there's a plugin for everything. And I think that's where Kubernetes sits is because it's become so widely adopted everybody's building for it. Everybody's adapting for it. If you run into a problem, you're going to find somebody else out there who has that problem. In fact, I think of one organization that I know that was on HashiCorp's Nomad. And they said, "Actually, we think Nomad has better technology through and through. But we think we're the only company at this size and scale using Nomad. And so, when we run into a problem, we can't Google for it. There's no such thing as a plugin that exists to solve this. Nobody has ever run into this before on Nomad. But there's 100 companies dealing with the same problem in Kubernetes, and there's ten solutions." And I think that's the power that it brings. VICTORIA: So, it's not just a trend that CTOs are moving towards, you think. KENDALL: I mean, I think it's already won the battle and the hockey stick of adoption. We're still right at the very bottom of that tick-up because it takes people a long time to adapt new technology like this, especially in their infrastructure. It's a big migration, to move. So, I don't think it's the widely adopted infrastructure technology even yet. I think a lot of the biggest organizations are still running on things that predate Kubernetes. But I think it has won the battle, and it is winning the battle and is going to be the thing going forward, so yeah. JOE: I think it also has a lot of room to grow still. Like, there are other technologies that I used previously, like Docker, and they were a big step up from some of the things I was doing at the time. But you quickly hit the ceiling, or it was, like, I don't know where to go with this next. I don't know what else is going to happen. Whereas with Kubernetes, there are so many directions it can go in. Like, the serverless Kubernetes offerings that are starting to pop up are extremely interesting, where, you know, you don't actually maintain a cluster or anything. You just deploy things to this ethereal cluster that always exists. And so, that sort of combination of platform as a service, function as a service, Kubernetes, as that evolves, I think there are a lot of exciting things that have yet to come in the Kubernetes space. KENDALL: Well, so say more about that, Joe, because I've been going to KubeCon for a very long time, maybe...I don't know if it's 2016 or so when I first went. And it felt for a number of years...maybe those first four-ish years it was always the people at KubeCon were the, like, big dreamers and thinkers and, like, we're here to change the future of cloud infrastructure. And this is going places, and we're excited to be here and be a part of it. And here's what I'm going to do that changes the next thing. And I feel like now if I go to KubeCon, it's a lot of people from, you know, IBM and some big bank that are, like, deep sigh, well, I have to adopt Kubernetes. I need to know what the vendors are. What do you guys do, and how does this work? Can you please teach it to me? Because I'm being told by my boss, I have to do it. I don't see that excitement around Kubernetes anymore. The excitement I see is all around further up the stack, you know, things like Wasm, WebAssembly, or eBPF, the networking things and tracing things that are possible. Maybe that's further down the stack. I guess it depends on how you think about it, but different part of the stack. So, I'm curious, touching on the serverless components of Kubernetes; sure, I get that. And I do think, increasingly, the PaaSs of the future are all going to be Kubernetes-based, whether that's exposed or not. But where are the places that you think it's still going to go? Because I feel like it's already gotten boring, maybe in a positive way. But I don't see the excitement around it like I saw a few years back. And I'm curious what else you think is going to happen. JOE: Yeah, I mean, I don't think I disagree. I think Kubernetes itself, the core concept, is, like, it's still changing. But you're right that the excitement about Kubernetes existing has gone down because it's been there for a while. But I feel like the ecosystem is still growing pretty rapidly. Like, the things you mentioned, like Wasm and Istio, and all the tools in that ecosystem that continue to grow, is where I think the interesting things will happen. Like, it's created this new lower-level layer of abstraction that makes it possible to build concepts and technology that could not have existed before. KENDALL: Yeah, well, and I'm, you know, talking to people who are working really hard at making short-run ephemeral workloads work better on things like GPUs for the sake of AI, right? Like, I mean, there is some really interesting things happening, and people are doing this in Kubernetes. So, I get that. I agree with that. It is interesting that Kubernetes has become sort of the stable thing, and now it's about who can build the interesting add-ons. It's almost like, okay, we've built Half-Life. What is Counter-Strike going to look like? You know. That's a terrible (I'm aging myself.) example. But still. VICTORIA: I think it's interesting, I mean, to look at the size of the market for platform engineering right now. In 2022, was 4.8 billion, and it's estimated to be in 10 years $41 billion. So, there is this emerging trend of different platform engineering products, different abstractions on top of Kubernetes. And I wonder what advice you would have for a technical founder who's looking to build and solve some of these interesting issues in Kubernetes and create a business around it. KENDALL: Well, okay, let me clarify that question. Are you thinking, I'm a startup, and I need to build my infrastructure, and I'm going to choose Kubernetes. What advice do I need? Or are you thinking, I am founder, and I want to go build on the Kubernetes ecosystem. What advice do you have? VICTORIA: Now I want to know the answer to both. But my question was the second one to start. KENDALL: One of the things that is hard about the Kubernetes ecosystem is there's not a ton of companies that have made a whole bunch of money in Kubernetes because, as I said, I still think we're actually really early in the adoption curve. The kinds of companies that have adopted Kubernetes are the kinds of companies that don't spend lots and lots of money on an infrastructure. [laughs] They're the kinds of companies that are fast-moving, early adopters, or, you know, those first followers, and so they're under $100 million companies for the most part. Where the JP Morgans and Chase are running Kubernetes somewhere in their stack, but they haven't adopted it across the stack to need the biggest, best tools about it. So, the first piece of advice that I'd give is, be a little wary. It's still very early to the market. Maybe now is the time to build the thing. When ReactiveOps pivoted to Kubernetes, I think it was six months of having conversations with companies who were just, like, so excited about it, and this is definitely what we want to do. But nobody was doing it yet. You know, it was, we have, like, six solid months of just excitement and nobody actually pulling the trigger. And, you know, we were a little too early to that market. And that was just the people adopting it. So, I think there is some nervousness that cloud-native solutions the only people who are really making money in Kubernetes are named Amazon, Google, and Microsoft because it's the cloud providers that are making a ton off of it. Now, there's Rancher. There is StackPointCloud. There's a few others that have had big exits in this space. But I don't think it's actually as big of a booming economy as a lot of people think, in part because EKS is an incredibly amazing product. Like, eight years ago, the thing people paid us the most to do at ReactiveOps was just stand up Kubernetes because it was so stinking hard to just get it up and working. And now you click some buttons. Anybody can go do that. So, it's changed a lot, right? And I think be wary when you're entering that ecosystem. And then, my advice to the founder that's not building on the ecosystem but just looking to adopt a technology that's going to be a future-proofed infrastructure is just adopt one of the cloud-native platforms. And there are a whole bunch of sort of default best-in-class add-ons out there that you need to throw in. Don't adopt too many because then you have to maintain them forever. That's the easiest way to get started. You can figure out all the rest of it later. But if you go use EKS, or GKE, AKS, you can get started pretty easily and build something that is going to be future-proofed. I don't know, Joe; I'm curious if you disagree with any of that. JOE: Well, I think it's interesting to think about who's making money in Kubernetes. Like, I think there might not be as many companies who are doing only Kubernetes and Kubernetes-focused products that are massively successful. But I think because it has had a good amount of adoption and because it's easier to work with something that's standardized, it has helped companies sell things that they wanted to sell anyway. Like, all the Datadog, all the Scalas, the logging companies, they all have Kubernetes add-ons. And now everybody is paying Datadog [laughs] to have a dashboard for their Kubernetes cluster. I think they're making more money than they would have been without targeting the market. And so, I think that's really...if you want to get into the market, it's not, like, I'm going to build a Kubernetes product. It's if I'm building operations and an infrastructure product, I should definitely have it work with Kubernetes, and people will want to click and install it. KENDALL: So, to be clear, you know, one of the companies that I work with is called Axiom, and they play in the same, you know, monitoring, observability space as Datadog does. And part of what makes Kubernetes interesting in that space is in a microservice environment; there's so much happening. Where are problems being caused? We don't live in a day where I can just run my code, and it tells me that there's an unexpected semicolon on line 23, right? Like, that still happens. You're still doing those things. But this microservice talking to that microservice is where things tend to break down. Did I communicate this correctly? What was sent? What was received? Where did it break down? What was the latency? And if you were doing things in the old way back when you were standing it up with, say, Ansible, or Puppet, or something like that, and you were orchestrating all of these cloud virtual machines, you had to really work hard to instrument the tracing and logging and everything involved in order to track what was going on. Whereas that's one of the magic things about Kubernetes is with a few of the add-ons or some of the things out of the box with Kubernetes, it's a couple of clicks to get so, so much of the data and have insight into where things are going and what's going wrong. And so, I 100% agree with that. Kubernetes is generating a tremendous amount of data. And if you're a data company, it's really nice to have all that come in, and it helps them make money, helps the user of Kubernetes in that situation understand where problems are happening and breaking down. Yeah, there's definitely some network effects of what Kubernetes is doing in that. I completely agree. JOE: I think there are also some interesting companies, like, where they make...Emissary, Ambassador, and they have that sort of dual -- KENDALL: Komodor, is that -- JOE: Yeah, maybe. They have open source, but then they have a product. KENDALL: You're thinking of Ambassador Labs. JOE: Yeah. Ambassador Labs, yeah. I guess I don't really know how much money they're making. But I think that's a really interesting concept as people who make open-source things then make a well-supported product built around it. KENDALL: Sure. What's interesting is, I think in the VC world, at least right now, and it may pick up again, but post-Silicon Valley Bank nearly caving in, I think that the VC tolerance for, yeah, just go get a billion open-source adopters, and we'll figure out how to monetize later I think that the tolerance for that is a lot lower than it was even six months ago. JOE: Yeah, I think you have to have a dual model right from the beginning now. KENDALL: Yeah. Agreed. VICTORIA: You got to figure out how to make money on Kubernetes before you can. [laughs] KENDALL: You know, minor detail. That's why I think services companies in this space still have a lot going for it. Because in order to even be able to sell software to a company using Kubernetes, you half the time have to go stand up Kubernetes for them because it is still that hard for so many people to really adopt it. VICTORIA: Yeah. And maybe, like, talking more about, like, when it is the right decision to start on Kubernetes because I think the question I get sometimes is just, is it overkill? Is it too much for what we're building? Especially, like, if you're building a brand-new product, you're not even sure if it's going to get adopted that widely. KENDALL: I mean, and I'm [laughs] curious your thought on this, Joe, but there's a good argument to be made that Heroku was enough for the vast majority of founders early on. But the thing is, Kubernetes isn't as hard as it used to be. Going and clicking a couple of buttons on GKE and deploying something into Kubernetes with GKE Autopilot running it's not as easy as Heroku, but it's not wildly far off. And it does substantially future-proof you. So, when is it too early? I'm not sure it's ever too early if you have an intention of scaling if you're planning on running some kind of legacy workload, like, things that are going to be stateful. Or maybe WordPress, for example, you don't probably need to deploy your WordPress blog onto Kubernetes. You can do that in your cPanel on Bluehost. I don't actually know if Bluehost even exists anymore, but I assume it's still a thing. I don't know, what would you say, Joe? JOE: I agree with that. I think it's a hard first pill to swallow. But I think the reality is that it's very easy to underestimate the infrastructure needs of even an early product. Like, it doesn't really matter what you're building. You're still going to have things like secrets management. You're still going to have to worry about networking. They just don't go away. There's no way you have a product without them. And so, rather than slowly solving all those problems from scratch on a platform that isn't designed for it, I think it's easier to just bite the bullet and use one of the managed solutions, especially, as you said, I think it's getting easier and easier. The activation energy from going from credit card to Kubernetes cluster is just getting lower. KENDALL: And so, the role of the CTO is just getting easier and easier because they can just adopt the one technology, and it's obviously Kubernetes. And it's obviously Rust, right? [laughter] Yeah, no, I'm with you. And I think if you find somebody who knows Kubernetes inside and out, it's really not going to take them long to get started. VICTORIA: Yeah, once again, change management is the biggest challenge for any new innovation coming into adoption. So, I'm curious to talk more about the influence that you need and how you influence others to come around to these types of ideas, like, in the executive suite and with the leadership of a company, especially on these types of topics, which can feel maybe a little abstract for people. KENDALL: How you influence them specifically to use Kubernetes, or just how you talk with them about technology adoption in general? Or what are you asking? VICTORIA: Yeah, like, how do I get people to not just turn their ears off when I say the word Kubernetes? [laughs] KENDALL: Yeah, I mean, I think...so I think that's where it's the technologist's job and the role of the CTO to translate these things into business speak. And that's why I'm using words like future-proofing your infrastructure is because there are companies that...I know one company that made a conscious decision that they were going to try to re-platform every single year, and that is not a good idea or sustainable for the vast majority [laughs] of companies. In fact, I can't think of a single situation where that makes sense. But if you can say to the CFO, "Hey, it's going to cost us a little bit more right now. It's going to save us substantially in the long term because this is the thing that's winning. And if we go standardize on Heroku right now, every company does eventually have to migrate off of Heroku. They either go out of business, or they get too big for it." That's the kind of thing that needs to be communicated in order to get people to adopt it. They don't care what the word is. They don't care if you're saying Kubernetes; you know, most CFOs understand it about as well as my mom does. My mom tries to bring it up in conversation because she's heard me use it. And she thinks it makes her sound smart, which maybe it does in the right climate. VICTORIA: My partner does the same thing. He says DevOps and Kubernetes all the time. I'm like; you don't know what you're talking about. [laughter] JOE: Those words do not come up in my house. KENDALL: One of my kids asked me to explain Kubernetes. And I do a whole talk, particularly at organizations where understanding Kubernetes is essential to the salespeople's role. And I give a whole talk about the background of how we got here from deploying on some servers in our back room. And, you know, what's different about the cloud, what containerization did, et cetera. And I have this long explanation. And I remember taking a deep breath and saying to my kids, "Do you really want to hear this?" And I had one son say, "Yes, absolutely." And my wife and three of the other kids all stood up and said, "No way," and left the room. So, when somebody asks me, "What do you do?" Actually, one of the key relationships I built with some of the early people at GCP when we were partnering closely with them was a person that I met, and I asked, "What do you do for a living?" And he said, "I can tell you, but it's not going to mean anything to you." And I was like, "That's what I say to people." And it turned out he was in charge of, you know, Kubernetes partnerships for Google. I can explain to you what it means and why it's important. But you're not going to be happy that I spent that time explaining it to you. VICTORIA: [laughs] That sounds awesome, though. It sounds like you built a server rack just to demo to your children what it was. KENDALL: No, no. I just talked back through the history of...that company that I mentioned that built Twitter about five years too early; we had a, you know, we had a server rack in the...literally physically in our closet that was serving up our product at the time. VICTORIA: Probably the best demo I ever saw was at Google headquarters in Herndon, and someone had built...They had 3D-printed a little mini server rack that they had put Raspberry Pis onto, and then they had Kubernetes deployed on it. And they did an automatic failover of a node to just demo how it works and had little lights that went with it. It was pretty fun. So maybe you should get one for yourself. [laughter] It's a fun project. KENDALL: They remember the things that it enables. They don't remember what it does. And so, when I say so, and so is a client that's using this technology, then they get real excited because they're like, "My dad makes that work." And I'm like, well, okay, that's kind of a stretch, but you get the idea. VICTORIA: Yeah, you got to lean into that kind of reputation in your house. KENDALL: That's right. VICTORIA: And you're like, yes, that's correct. KENDALL: That's right. [laughs] VICTORIA: I do make Kubernetes. I make all the clouds work, yeah. KENDALL: Actually, my most common explanation is Kubernetes is the plumbing of the internet. Unless you're a plumber, you don't care about the pipes. You just want your shit to flush when you use the toilet. You want the things to load when you click your buttons. You don't actually care what's going on behind the scenes, but this is what's orchestrating it increasingly across the internet. VICTORIA: So far, we've called Kubernetes WordPress or the toilet. [laughs] KENDALL: The plumbing. [laughter] VICTORIA: You are really good at selling it. [laughter] KENDALL: Hey, if you want to build a nice, clean city, you need good plumbing. You might not care what the pipes are made of, but you need good plumbing. [laughs] VICTORIA: Works for me. On that note -- [laughs] KENDALL: Yeah. Right? Right? VICTORIA: That's [inaudible 36:41] on a high note. Is there anything else that you'd like to promote? KENDALL: With regards to CTO Lunches, we have a free listserv. There are local lunches. If there isn't a local lunch where you are, it's very lightweight to start up a chapter. We often have folks who are willing to sponsor that first lunch to get you going. We do have a paid tier of CTO Lunches. If you want a small back room Slack channel of people to discuss, I think it's $99 a month. Yeah, if you're a CTO and/or a senior engineering leader and you want a community of people to process with, be it our free tier or our paid tier, we've got something for you. We're trying to invest in this to build community around it. And it's something we enjoy doing more than almost anything. Come take part. VICTORIA: You can subscribe to the show and find notes along with a complete transcript for this episode at giantrobots.fm. If you have questions or comments, email us at hosts@giantrobots.fm. And you can find me on Twitter @victori_ousg. This podcast is brought to you by thoughtbot and produced and edited by Mandy Moore. Thanks for listening. See you next time. ANNOUNCER: This podcast is brought to you by thoughtbot, your expert strategy, design, development, and product management partner. We bring digital products from idea to success and teach you how because we care. Learn more at thoughtbot.com. Special Guest: Kendall Miller.

“The State of Kubernetes Cost Optimization,” is a recent report based on research into best practices for running Kubernetes clusters. If you're running your workloads as efficiently as possible, your costs will be optimal too. The report reviews the data and offers recommendations on tools and techniques you can use to optimize your Kubernetes clusters. We talk with two of the report's creators, Fernando Rubbo and Kent Hua, to learn more.   Do you have something cool to share? Some questions? Let us know: - web: kubernetespodcast.com - mail: kubernetespodcast@google.com - twitter: @kubernetespod   News of the week - CNCF Istio Graduation blog - Istio's blog about CNCF Graduation - CNCF Blog on Flux v2 GA release - Redhat Blog on Kubevirt 1.0 - Pulumi blog on v4.0 of their Kubernetes Provider - VMware Wasm Labs blog on serverless with wasm - CNCF announcement of over 30 new members  - VMware docs on self-hosted Tanzu Links from the interview - The State of Kubernetes Cost Optimization report - “Sharing the inaugural State of Kubernetes Cost Optimization report” blog - Resource Management for Pods and Containers (Kubernetes Documentation) Links from the post-interview chat - Google Site Reliability Engineering (SRE) books - Google Cloud Managed Service for Prometheus

Aaron Torres and Ala Shiban are from Klotho, which powers Infrastructure Copilot, the most advanced infrastructure design tool that understands how to define, connect, and scale your infrastructure-as-code. Victoria talks to Aaron and Ala about the Klotho engine, Klotho the CLI tool, and InfraCopilot and how they work together to help enable developer teams to iterate on applications and features quickly. Klotho (https://klo.dev/) Infrastructure Copilot (https://infracopilot.io/) Follow Klotho on Github (https://github.com/klothoplatform/klotho), Discord (https://discord.com/invite/4wwBRqqysY), Twitter (https://twitter.com/GetKlotho), or LinkedIn (https://www.linkedin.com/company/klothoplatform/). Follow Aaron Torres on LinkedIn (https://www.linkedin.com/in/torresaaron/), or Twitter (https://twitter.com/aarontorres). Follow Ala Shiban on LinkedIn (https://www.linkedin.com/in/alashiban/) or Twitter (https://twitter.com/AlaShiban). Follow thoughtbot on Twitter (https://twitter.com/thoughtbot) or LinkedIn (https://www.linkedin.com/company/150727/). Become a Sponsor (https://thoughtbot.com/sponsorship) of Giant Robots! Transcript: VICTORIA: This is the Giant Robots Smashing Into Other Giant Robots Podcast, where we explore the design, development, and business of great products. I'm your host, Victoria Guido. And with me today is Aaron Torres and Ala Shiban from Klotho, which powers Infrastructure Copilot, the most advanced infrastructure design tool that understands how to define, connect, and scale your infrastructure-as-code. Aaron and Ala, thank you for joining me. ALA: Thank you for having us. AARON: Yeah, thank you very much. VICTORIA: Well, great. I wanted to just start with a little bit of a icebreaker; maybe tell me a little bit more about what the weather is like where you're currently at. AARON: So I'm in St. Louis, Missouri. Right now, it is definitely...it feels like summer finally. So we're getting some nice, warm days and clear skies. ALA: And I'm in LA. And it's gloomier than I would like compared to what it's been in the last few years. But I'll take it if this means we're getting closer to summer. VICTORIA: Right. And I'm not too far from you, Ala, in San Diego, and it's a little chillier than I would prefer as well. But that's what we get for living close to the beach. So there's always trade-offs. Well, wonderful. I'm so excited to talk to you about your product here today. Let me start with a question about, let's say, I'm a non-technical founder, and I've just heard about your product. What's your pitch to someone in that position on the value of your tool? ALA: For somebody who isn't technical, I would say you can enable your team, your developer team, to quickly iterate on their applications or features and let InfraCopilot and Klotho take care of taking that application or features and deploy them and getting them running on the cloud. VICTORIA: Okay. So maybe I've been thinking about having to hire an AWS engineer or someone who's an infrastructure engineer. I could consider getting a tool like Klotho and Infrastructure Copilot to allow my developers to take on more of that responsibility themselves. ALA: Absolutely. VICTORIA: Gotcha. Okay, well, great. So let me ask about how did it all get started? What was the impetus that set you on this journey ALA: Both Aaron and I used to work at Riot Games, and I used to lead the cloud services org at Riot. I had about 50 people, 40 engineers, as part of a larger 120-person org, infrastructure platform org, which was tasked with building the platform that runs League of Legends, VALORANT, for 200 million people all around the world, in China. Full DevOps mode for Riot developers and full ops mode for running in China. It took us three years, a lot of effort. And by the time we were done, it was already legacy, and that seemed broken to me. We were already getting started to do another round of upgrades and iterations. At that point, I decided to leave. But I couldn't let go of this feeling that we shouldn't have had to spend so many years solving a problem only for it not to be solved. And based on research and conversations, it was clear that this was an industry-wide phenomena. And so I went about trying to figure out why that happens and then how we can solve it, and that's how Klotho came about. VICTORIA: That's so interesting. And I've certainly been a part of similar situations where you spend so much time solving a big problem and infrastructure only to get to the end of it and realize now you have a whole nother set of problems. [laughs] And you get upgrade. And they've also invented new ways of doing things in the cloud that you want to be able to take advantage of. So you had that time with Riot Games and League of Legends and building this globally responsive infrastructure. What lessons learned did you take from that into building Klotho and building your product, Infrastructure Copilot? AARON: We learned a bunch of things. One of the more difficult problems to solve isn't technical at all; it's organizational and understanding how the organization flows and how the different teams interact with each other. So we really endeavor to solve that problem. I mean, our product is a technical product, but it is meant to help bridge that gulf and make that problem a little bit easier as well. Otherwise, yeah, exactly to your point, part of the problem with these migrations is that new technology comes along. And there's definitely a feeling of when you hire new developers, they are excited about the new thing, and there's other reasons as well. But you get this kind of natural, eternal migration going to the newer technology. VICTORIA: That makes sense. And you bring up a great point on some of the issues, not being technical but organizational. And when I look at a lot of infrastructure-as-code tools, when we get to security, I wonder how it fits in with the organizational requirements for security, right? Like, you have to have defined groups who have defined access to different levels and have the tools in place to be able to manage identities in your organization. So I'm curious how that fits into what you built with Klotho and the Infrastructure Copilot. ALA: The way we think about infrastructure is as a set of intents or things that developers, and operators, cloud engineers, infrastructure engineers are trying to satisfy or to do. So you have tasks. You're trying to build a solution. You're trying to build an architecture or add something to it. And organizations have constraints, whether it's their own Terraform, or their own ruleset, or security expectations, or compliance expectations. And the way we look at this dynamic is those rules are encoded in a way that Klotho, which is a cloud compiler, it has the ability to reason about both the application and the infrastructure-as-code and enforce or at least warn about mismatches between the constraints that the organization sets, and what the developer or operator are trying to do, or the intent that is being described high level or low level within the tools. And then that is reflected both visually and in code and in the infrastructure-as-code, one or more. And so it's very much rooted in how the entire set of technologies and product and tools are designed. VICTORIA: Got it. So do you see the tool will be more fit for the market of larger development shops who maybe have existing infrastructure but want to experiment with a different way of managing it for their developers? ALA: It depends. So because we went about solving the problem rather than just building a specific vertical or a specific stack piece, we try to only play in this space of intelligent editing and intelligent understanding of the alignment between infrastructure and code. And so you could, as a developer, effectively with Klotho, write a plain application and have it be running in the cloud without knowing anything in the underlying cloud systems. It will set up storage, and persistence, and security, and secrets. All those elements are easily accessible within the code itself. It can also work in the context of a company where the infrastructure or platform team have set those rules and guidance within the tools. And then, developers can continue working the way they expect to work, either in code or in the infrastructure-as-code layer. And it would still allow them to do the same intents that they want only within that sandbox. Or if they can't be satisfied because they're trying to do something that isn't allowed, they have a mechanism of, one, knowing that but also asking, in our case, InfraCopilot to help it reshape what it's doing, what they're doing into the sandbox and the trade-offs that that brings in. VICTORIA: Got it. So you can both start from scratch and start a brand new application using it, or you can integrate it with your existing rules and systems and everything that already exists. ALA: Exactly. VICTORIA: Gotcha. Yeah, I think one interesting thing we've found with very new founders who are building their application for the first time is that there are some essential things, like, they don't even have an identity store like a Google [laughs] or Microsoft Azure Directory. So starting to work in the cloud, there are some basic elements you have to set up first that's a little bit of a barrier. So it sounds like what you're saying with Klotho is that you wouldn't necessarily have those same issues. Or how would you get that initial, like, cloud accounts set up? AARON: Yeah. So, for the situation where you're bootstrapping everything from scratch, you've done nothing; we haven't invested much in setting up the initial accounts. But assuming you get to the point where you have AWS credentials, and you're able to hit the AWS API using the CLI, that's sort of where we can take over. So, yeah, like, I would say right now, as a business, it's definitely where the value is coming is going to be these mid-sized companies. But for that scenario specifically, bootstrapping and starting something from scratch, if you have that initial setup in place, it's one of the fastest ways to go from a concept to something running in the cloud. ALA: And if you think about the two tools that we're building, there's Klotho, which InfraCopilot...or the Klotho engine, which Klotho the CLI tool uses and InfraCopilot uses. The Klotho engine is responsible for the intelligence. It knows how to translate things like I want a web API that talks to DynamoDB. And it will literally create everything or modify everything that is needed to give you that and plug in your code. You can also say things in a much higher level degree, which things like I want a lambda which handles 10,000 users. And I want it to be lowest latency talking to an RDS Instance or to a Postgres database. And what that would do is, in our side, in the Klotho engine, we understand that there needs to be a VPC and subnets, and spin up RDS, and connect an RDS proxy. Because for connection pooling with lambda specifically, you need one to scale to that degree of scale. And so that is the intelligence that is built into the Klotho engine if you want to start from the infrastructure. If you want to start from code, all you have to do is bring in the Redis instance, the Redis SDK, and, let's say, your favorite web framework, and just add the annotations or the metadata that says, I want this web framework to be exposed to the internet, and I want this Redis to be persisted in the cloud. And you run Klotho. And what comes out the other end is the cloud version that does that for you. And it's one command away from getting it to run. VICTORIA: So that's interesting how the two tools work together and how a developer might be able to get things spun up quickly on the cloud without having to know the details of each particular AWS service. And reading through your docs, it sounds like once you have something working in the cloud, then you'll also get automated recommendations on how to improve it for cost and reliability. Is that right? ALA: That's where we're headed. VICTORIA: Gotcha. I'm curious; for Aaron, it sounds like there is more in that organizational challenges that you alluded to earlier. So you want to be able to deliver this capability to developers. But what barriers have you found organizationally to getting this done? AARON: So I'm going to speak specifically on infrastructure here because I think this is one of the biggest ones we've seen. But typically, when you get to a larger-sized company, we'll call it a mid-sized company with, you know, a couple hundred engineers or more, you get to the point where it doesn't make sense for every team to own their entire vertical. And so you want to really put the cloud knowledge into a central team. And so you tend to build either a platform team, or an infrastructure team, or a cloud team who sort of owns how cloud resources are provisioned, which ones they support, et cetera. And so, really, some of the friction I'm talking about is the friction between that team and developer teams who really just want to write their application and get going quickly. But you don't have to fall within the boundaries set by that central team. To give, like, a real concrete example of that, if you wanted to prototype a new technology, like, let's say that some new database technology came out and you wanted to use it, it's a very coordinated effort between both teams in terms of the roadmap. Like, the infrastructure team needs to get on that roadmap, that they need to make a sandbox and how that's going to work. The code team needs to make an application to test it. And the whole thing requires a lot more communication than just tech. VICTORIA: Yeah, no, I've been part of kind of one of those classic DevOps problems. It's where now you've built the ops team and the dev team, [laughs] and now you're back to those coordination issues that you had before. So, if I were a dev using Klotho or the infrastructure-as-code copilot, I would theoretically have access to any AWS sandbox account. And I could just spin up whatever I wanted [laughs] within the limits that could be defined by your security team or by your, you know, I'm sure there's someone who's setting a limit on the size of databases you could spin up for fun. Does that sound right? AARON: Yeah, that's totally right. And in addition to just limits, it's also policies. So a good example is maybe in production for databases, you have a data retention policy. And you have something like we need to keep three months of backups for this amount of time. We want to make sure that if someone spins up a production database from any of those app teams, that they will follow their company policy there and not accidentally, like, lose data where it has to be maintained for some reason. ALA: That's an important distinction where we have our own set of, you know, best practice or rules that are followed roughly in the industry. But also, the key here is that the infrastructure central teams in every company can describe the different rulesets and guidelines, guardrails within the company on what developers can do, not only in low-level descriptions like instance sizes or how much something is, whether it's Spot Instances always or not in production versus dev. But also be able to teach the system when a developer says, "I want a database," spin up a Postgres database with this configuration that is wired to the larger application that they have. Or, if I want to run a service, then it spins up the correct elements and configures them to work, let's say, Kubernetes pods, or lambdas, or a combination based on what the company has described as the right way for that company to do things. And so it gives flexibility to not know the specific details but still get the company's specific way of doing them. And the key here is that we're trying to codify the communication patterns that do happen, and they need to happen if there's no tools to facilitate it between the infrastructure platform team and the feature teams. Only in this case, we try and capture that in a way that the central teams can define it. And the developers on feature teams can consume it without having as much friction. VICTORIA: So that will be different than, like, an infrastructure team that's putting out everything in Terraform and doing pull requests based off GitHub repository to that. It makes it a little more easier to read, and understand, and share the updates and changes. AARON: Right. And also, I mean, so, like, the thing you're describing of, like, the central team, having Terraform tends to be, like, these golden templates. Like they say, "If you want to make a database, here's your database template." And then you get a lot of interesting issues like drift, where maybe some teams are using the old versions of the templates, and they're not picking up the new changes. And how do you kind of reconcile all that? So it is meant to help with all of those things. VICTORIA: That makes a lot of sense. And I'm curious, what questions came up in the customer discovery process for this product that surprised you? ALA: I think there's one...I don't know that it was a question, but I think there was...So, when we started with Klotho, Klotho has the ability to enable a code-first approach, which means that you give the tool to developers as the infrastructure or platform team, or if you're a smaller shop, then you can just use Klotho directly. You set the rules on what's allowed or what's not allowed, and then developers can work very freely. They can describe very succinctly how to turn a plain object, SDK, et cetera, how to build larger architectures very quickly with a few annotations that we describe and that give cloud powers. We had always thought that some teams will feel that this encroaches on their jobs. We've heard from people on infra, you know, platform teams, "This is amazing. But this is my job." And so, one of our hypotheses was that we are encroaching into what they see as their responsibility. And we built more and more mechanisms that would clean up that interface and give them the ability to control more so they can free themselves up, just like most automations that happen in the world, to do more things. What happened later surprised us. And by having a few or several more discoveries, we found out that the feeling isn't a fear of the tool replacing their job. The fear or worry is that the tool will make their jobs boring, what is left of the job be boring, and nobody wants to go to work and not have cool and fun things to do. And because I think we all, on a certain degree, believe that, you know, if we take away some of the work that we're doing, we'll find something higher level and harder to solve, but until that exists in people's minds, there's nothing there. And therefore, they're left with whatever they don't want to do or didn't want to do. And so that's where we tried to take a step back from all the intelligence the Klotho engine provides through that code-first Klotho. And we built out focusing on one of the pillars in the tech to create InfraCopilot, which helps with keeping or making the things that we already do much simpler but also in a way that maintains and does it in a fun way. VICTORIA: That makes sense because my understanding of where to use AI and where to use machine learning for best purposes is to automate those, like, repetitive, boring tasks and allow people to focus on the creative and more interesting work, right? ALA: Yes and no. The interesting bit about our approach to ML is that we don't actually use machine learning or ChatGPT for any of the intelligence layers, meaning we don't ask ChatGPT to generate Terraform or any kind of GPT model to analyze a certain aspect of the infrastructure. That is all deterministic and happens in the Klotho engine. That is the uniqueness of why this always works rather than if GPT happened to get it right. What we use ML for is the ability to parse the intent. So we actually use it as a language model to parse the intent from what the user is trying to convey, meaning I want a lambda with an API gateway. What we get back from our use of ML is the user has asked for a lambda, an AWS lambda, and API gateway and that they be connected. That is the only thing we get back. And that is fed into the Klotho engine. And then, we do the intelligence to translate that to an actual architecture. VICTORIA: That's a really cool way to use natural language processing to build cloud infrastructure. MID-ROLL AD: Are you an entrepreneur or start-up founder looking to gain confidence in the way forward for your idea? At thoughtbot, we know you're tight on time and investment, which is why we've created targeted 1-hour remote workshops to help you develop a concrete plan for your product's next steps. Over four interactive sessions, we work with you on research, product design sprint, critical path, and presentation prep so that you and your team are better equipped with the skills and knowledge for success. Find out how we can help you move the needle at: tbot.io/entrepreneurs. VICTORIA: I'm curious; you said you're already working on some issues about being able to suggest improvements for cost reduction and efficiencies. What else is on your roadmap for what's coming up next? AARON: So there's a bunch of things in the long-term roadmap. And I'll say that, like, in the short term, it's much more about just expanding the breadth of what we support. If you think about just generating all the different permutations and types of infrastructure, it's, like, a huge matrix problem. Like, there's many, many dimensions that you could go in. And if you add an extra cloud or you add an extra capability, it expands everything. So you can imagine, like, testing it to make sure things work, and everything becomes very complicated. So, really, a lot of what we're doing is still foundational and trying to just increase the breadth, make the intent processing more intelligent, make the other bits work. And then one of the areas right now is for our initial release of the product; we chose to use Discord as our interface for the chatbot. And the reason for that is because it gives us a lot of benefits of having sort of the community built in and the engagement built in where we can actually talk with users and try and understand what they're doing. However, we really have a lot of UI changes and expansions that we'd like to do. And even from some of our early demo material, we have things like being able to right-click and being able to configure your lambda directly from the UI. So there's a lot of areas there that we can expand into an intent, too, once we get sort of the foundational stuff done, as an example. The intelligence bit is a much bigger process, like, there's a lot of things to unpack there. So I won't talk about it too much. But if we were to just talk about the most simple things, it'd be setting up alerts somehow and then feeding into our system that, like, we're hitting those alerts, and we have to make modifications. A good example of that would be, like, configuring auto scaling on an instance for [inaudible 22:17]. So we can get some of those benefits now. The bigger vision of what we want to do with optimization requires a lot more exploration and also the ability to look at what's happening to your application while it's running in the cloud. ALA: Let me maybe shed a bit more light on the problems we're trying to solve and where we're headed. When it comes to optimization, to truly optimize a cloud application, you have to reason about it on the application level rather than on the one service level. To do that, we have to be able to look at the application as an application. And today, there's a multi-repo approach to building cloud applications. So one of the future work that we're going to do is be able to reason about existing infrastructure-as-code from different portions of the teams or organization or even multiple services that the same team works and link them together. So, when we look at reasoning about an architecture, it is within the entire context of the application rather than just the smaller bits and pieces. That's one layer. Another layer is being able to ingest the real runtime application metrics and infrastructure metrics, let's say, from AWS or Azure into the optimizer system to be able to not only say, oh well, I want low latency. Then this is hard-coded to use a Fargate instance instead of a lambda. But more realistically, being able to see what that means in lambda world and maybe increase the concurrency count. Because we know that within the confines of cost limitations or constraints that the company wants to have, it is more feasible and cost-effective to raise the minimum concurrency rate of that lambda instead of using Fargate. You can only do that by having real-time data, or aggregated data come from the performance characteristics of the applications. And so that's another layer that we're going to be focusing on. The third one is, just like Aaron said, being able to approach that editing experience and operational experience, not just through one system like InfraCopilot but also through a web UI, or an app, or even as an extension to other systems that want to integrate with Klotho's engine. The last thing that I think is key is that we're still holding on to the vision that infrastructure should be invisible to most developers. Infrastructure definition is similar to how we approach assembly code. It's the bits and pieces. It's the underlying components, the CPUs, the storage. And as long as we're building microservices in that level of fidelity, of like, thinking about the wiring and how things interconnect, then we're not going to get the gains of 10x productivity building cloud applications. We have to enable developers and operators to work on a higher abstraction. And so our end game, where we're headed, is still what we want to build with Klotho, which is the ability to write code and have it be translated into what's allowed in the infrastructure within the constraints of the underlying platforms that infrastructure or platform teams set for the rest of the organization. It can be one set or multiple sets, but it's still that type of developers develop, and the infrastructure teams set them up to be able to develop, and there's separation. VICTORIA: Those are all really interesting problems to be solving. I also saw on your roadmap that you have published on Klotho that you're thinking of open-sourcing Klotho on GitHub. AARON: So, at this point, we already have the core engine of Klotho open-sourced, so the same engine that's powering InfraCopilot and Klotho, the tool itself is open source today. So, if anyone wants to take a look, it is on github.com/klothoplatform/klotho. VICTORIA: Super interesting. And it sounds like you mentioned you have a Discord. So that's where you're also getting feedback from developers on how to do this. And I think that challenge you mentioned about creating abstractions so that developers don't have to worry as much about the infrastructure and platform teams can just enable them to get their work done; I'm curious what you think is the biggest challenge with that. It seems like a problem that a lot of companies are trying to solve. So, what's the biggest challenge? And I think what do you think is unique about Klotho and solving that challenge? AARON: I guess what I would say the biggest challenge today is that every company is different enough that they all saw this in a slightly different way. So it's like, right now, the tools that are available are the building blocks to make the solution but not the solution itself. So, like, every cloud team approaches it on, let's build our own platform. We're building our own platform that every one of our developers is going to use. In some cases, we're building, like, frameworks and SDKs that everyone's going to use. But then the problem is that you're effectively saying my company is entering the platform management business. And there's no way the economies of scale will make sense forever in that world. So I think that's the biggest issue. And I think the reason it hasn't been solved is it's just a very hard problem. There's many approaches, but there's not a clear solution that kind of brings it all together. And I think our product is positioned better than most to solve some of the higher-level abstractions. It still doesn't solve the whole problem. There's still some things that are going to be tricky. But the idea is, if you can get to the point where you're using some of our abstractions, then you've guaranteed yourself portability into the future, like, your architecture will be able to evolve, even in technologies that don't exist yet once they become available. ALA: To tack on to what Aaron said, a key difference, and to our knowledge, this doesn't exist in any other tool or technology, is a fundamentally new architecture we call adaptive architecture. It is not microservices. It is not monoliths. It's a superset that combines all the benefits from monoliths, microservices, and serverless if you consider it a different platform or paradigm. What that means is that you get the benefits without the drawbacks. And the reason we can do it is because of the compiler approach that we're taking, where everything in the architecture that we produce is interchangeable. The team has decided to use Kubernetes, a specific version of Kubernetes with Istio. That works great. And, a year later, it turns out that that choice no longer scales well for the use. And we need to use Linkerd. The problem in today's world and what companies have to do is retrofit everything and not only the technology itself, but it's the ripple effects of changing it into everything else that all the other choices that were made that depended on it. In the Klotho world, because of the compilation step or the compilation approach and its extensibility, you could say, I want to take out Istio and replace it with Linkerd. And it would percolate all the changes that need to happen everywhere for that to maintain its semantic behavior. To our knowledge, that doesn't exist anywhere today. VICTORIA: So it would do, maybe not, like, would do migrations for you as well? ALA: I think migrations are a special case. When it comes to stateless things, yes. When it comes to data, we are much more conservative. Again, bringing what we've learned in different companies in, a lot of solutions try to solve all the things versus we're trying to play in a very specific niche, which is the adaptive architecture of it all. But if you want to move data, there's fantastic tools for it, and we will guide you through getting the access to the actual underlying services and, say, great, write a migration system, or we can generate for you. But you will run it to move the data from, let's say, Postgres to MySQL or from being able to drain a unit on Kubernetes to a lambda. Some of those things are much more automatic. And the transition happened through the underlying technologies like Terraform or Pulumi. Others will require you to take a step, not because we can't do it for you but we want to be conservative with the choices. AARON: I would also add that another aspect of this is that we don't position ourselves as being the center of the universe for these teams. Like a lot of products, you kind of have to adopt the platform, and everything has to plug into it, and if you don't adhere, it doesn't work. We're trying very, very hard with our design to make it so that existing apps will continue to function like they've always functioned. If app developers want to continue using direct SDKs and managing config themselves, they can absolutely do that. And then they'll interact well with Klotho apps that are also in that same company. So we're trying to make it so that you can adopt incrementally without having to go all in. VICTORIA: So that makes a lot of sense. So it's really helpful if you're trying to swap out those stateless parts of your infrastructure and you want to make some changes there. And then, if you were going to do a data migration, it would help you and guide you to where additional tools might be needed to do that. And at your market segment, you're really focusing on having it be an additional tool, as opposed to, like, an all-encompassing platform. Did I get it all right? [crosstalk 31:07] ALA: Exactly. VICTORIA: [laughs] Cool. All right. Well, that's exciting. That's a lot of cool things that you all are working on. I'm curious how overall the workload is for you two. How big of a team do you have so far? How are you balancing out this work of creating something new and exciting that has such a broad potential scope? AARON: Yeah. So, right now, the team is currently six people. So it's Ala and I, plus four additional engineers is the current team. And in terms of, like, where we're focusing, the real answer is that it's somewhat reactive, and it's very fast. So, like, it could be, like...in fact, Copilot went from ideation to us acting on it extremely quickly. And it wasn't even in the pipeline before that. So I'd actually say the biggest challenge has been where do we sort of focus our energy to get the best results? And a lot of where we spend our time is sort of meta-process of, like, making sure we're investing in the right things. ALA: And I think that comes from both Aaron and I have been in the industry for over 15 years. We don't, you know, drop everything and now switch to something new. We're very both tactical and strategic with the pace and when we pivot. But the idea is when we decide to change and focus on something that we think will be higher value, and it's almost always rooted in the signals and hypotheses that we set out to kind of learn from, from every iteration that we go after. We are not the type that would say, "Oh, we saw this. Let's drop everything, and let's go do it." I think we've seen enough in the industry that there's a measure of knowing when to switch, and when to refocus, and what to do when these higher tidbits come, and then being able to execute aggressively when that choice or decision happens. VICTORIA: Are there any trends that you're watching right now that the outcome would influence a change in direction for you? ALA: Not technically. I think what we're seeing in the industry is there's no real approaches to solving the problem. I would say most of the solutions and trends that we're seeing are...I call them streamlined complexity. We choose a set of technologies, and we make that easy. We make the SaaS version, and it can do these workloads, and it makes that easy. But the minute you step out of the comfort zone of those tools, you're back into the nightmare that building distributed systems brings with it, and then you're back to, you know, square one. What we're trying to do is fundamentally solve the problem. And we haven't seen many at least make a lot of headway there. We are seeing a few of the startups that are starting to think in the same vein, which is the zeitgeist. And that's fantastic. We actually work with them closely to try and broaden the category. VICTORIA: Right. Do you feel that other companies who are working in a similar problem space that there is...is it competitive between each other? Or do you think it's actually more collaborative? ALA: It depends on the companies and what they're trying to achieve. Every set of companies have different incentives. So Google, Amazon, and Microsoft have, you know, are incentivized to keep you on their clouds. They may care less about what they have in there as long as you are happy to stay. So you'll see more open source being adopted. You will see Amazon trying to copy or operationalize a lot of open-source tools. Microsoft will give their...because they are working with larger companies to have more vertical solutions. Google is trying to catch up. If you look at startups, you will see some focus more on developers. You'll see others focus on infra team. So it really depends on the intersection of the companies, and then they either collaborate or they compete, depending on how it affects their strategy. In our case, we recognize that our competition is the incumbents and the current way of doing things. And so we are happy to collaborate with all the startups that are doing something in the vicinity of what we're doing, startups like Ampt, and Encore, and Winglang. And there's several others. We have our own Slack channel where we talk about, like, where we're headed or at least what we can do to support one another. VICTORIA: Great. And I wonder if that's part of your business decision to open source your product as well or if there are other factors involved. ALA: I think the biggest factor that we've seen, realistically, is the expectation in the developer community to have a core that is open source, not even the source available model but to have an open-source core that they can rely on always existing and referencing when, you know if the company disappears or Oracle buys them. And so I would say that that was the biggest determining factor in the end to open-sourcing the Klotho engine. It's a very pragmatic view. VICTORIA: That makes sense. Well, I wanted to make sure we had time to ask one of my favorite questions that I ask on the podcast, and you can both answer. But if you could go back in time to when you first started this project, what advice would you give yourself? ALA: I guess the advice that I would give is keep selling and start selling as early as you can, even before the vision is realized. Or let's say you're making kind of headway towards what you'll wind up sharing and giving companies, the lead time to creating the opportunities and the belief and the faith that you can solve problems for companies, and the entire machinery of doing that is a lot more complex than most founders, I think, or at least first-time founders or, honestly, myself have found it to be. AARON: Yeah. If I try and answer that same question, it's very challenging. I guess my perspective now is there's nothing I could tell myself that would make me go any faster because a lot of it really is the journey. Like, the amount of stuff that we've learned in the last year of working on this and exploring and talking with people and everything else has been so vast that there's nothing I can communicate to past me that would prepare me any better. So [laughs] I think I would try just my best to be encouraging to just stick with it. VICTORIA: Well, that's good. And who knows what you're going to learn in the next year that [laughs] probably might not help you in the past either? That's wonderful. Do you have any final takeaways for our listeners today or anything you'd like to promote? ALA: So, from my lens, I've always wanted to do a startup but felt that the life setting wasn't quite ready. And a lot of the startup culture is talking about younger, earlier founders. I think having had the industry experience and understanding both the organizational and technical challenges, knowing more people, and engineers, and founders, potential founders, has been vastly more helpful than what I would have been able to pull off ten years ago. So, if you are thinking maybe it's too late, it is not. It's probably easier in some regards now. And yeah, check out InfraCopilot. It's on infracopilot.io. We would love to have you try it out and go on this journey with us. AARON: Yeah, I would definitely echo that. I mean, sort of the same thing on the journey. Like, it's never too late to start. And yeah, like, I would say being in the industry and actually seeing these problems first-hand makes it so much more fulfilling to actually try and solve them. VICTORIA: That's [inaudible 38:15]. I'm excited to see what you all accomplish. And I appreciate you coming on the show. You can subscribe to the show and find notes along with a complete transcript for this episode at giantrobots.fm. If you have questions or comments, you can email us at hosts@giantrobots.fm. And you could find me on Twitter @victori_ousg or on Mastodon @vguido@thoughtbot.social. This podcast is brought to you by thoughtbot and produced and edited by Mandy Moore. Thanks for listening. See you next time. ANNOUNCER: This podcast is brought to you by thoughtbot, your expert strategy, design, development, and product management partner. We bring digital products from idea to success and teach you how because we care. Learn more at thoughtbot.com. Special Guests: Aaron Torres and Ala Shiban.

Liz Rice, Chief Open Source Officer at Isovalent, joins Corey on Screaming in the Cloud to discuss the release of her newest book, Learning eBPF, and the exciting possibilities that come with eBPF technology. Liz explains what got her so excited about eBPF technology, and what it was like to write a book while also holding a full-time job. Corey and Liz also explore the learning curve that comes with kernel programming, and Liz illustrates why it's so important to be able to explain complex technologies in simple terminology. About LizLiz Rice is Chief Open Source Officer with eBPF specialists Isovalent, creators of the Cilium cloud native networking, security and observability project. She sits on the CNCF Governing Board, and on the Board of OpenUK. She was Chair of the CNCF's Technical Oversight Committee in 2019-2022, and Co-Chair of KubeCon + CloudNativeCon in 2018. She is also the author of Container Security, and Learning eBPF, both published by O'Reilly.She has a wealth of software development, team, and product management experience from working on network protocols and distributed systems, and in digital technology sectors such as VOD, music, and VoIP. When not writing code, or talking about it, Liz loves riding bikes in places with better weather than her native London, competing in virtual races on Zwift, and making music under the pseudonym Insider Nine.Links Referenced: Isovalent: https://isovalent.com/ Learning eBPF: https://www.amazon.com/Learning-eBPF-Programming-Observability-Networking/dp/1098135121 Container Security: https://www.amazon.com/Container-Security-Fundamental-Containerized-Applications/dp/1492056707/ GitHub for Learning eBPF: https://github.com/lizRice/learning-eBPF TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. Our returning guest today is Liz Rice, who remains the Chief Open Source Officer with Isovalent. But Liz, thank you for returning, suspiciously closely timed to when you have a book coming out. Welcome back.Liz: [laugh]. Thanks so much for having me. Yeah, I've just—I've only had the physical copy of the book in my hands for less than a week. It's called Learning eBPF. I mean, obviously, I'm very excited.Corey: It's an O'Reilly book; it has some form of honeybee on the front of it as best I can tell.Liz: Yeah, I was really pleased about that. Because eBPF has a bee as its logo, so getting a [early 00:01:17] honeybee as the O'Reilly animal on the front cover of the book was pretty pleasing, yeah.Corey: Now, this is your second O'Reilly book, is it not?Liz: It's my second full book. So, I'd previously written a book on Container Security. And I've done a few short reports for them as well. But this is the second, you know, full-on, you can buy it on Amazon kind of book, yeah.Corey: My business partner wrote Practical Monitoring for O'Reilly and that was such an experience that he got entirely out of observability as a field and ran running to AWS bills as a result. So, my question for you is, why would anyone do that more than once?Liz: [laugh]. I really like explaining things. And I had a really good reaction to the Container Security book. I think already, by the time I was writing that book, I was kind of interested in eBPF. And we should probably talk about what that is, but I'll come to that in a moment.Yeah, so I've been really interested in eBPF, for quite a while and I wanted to be able to do the same thing in terms of explaining it to people. A book gives you a lot more opportunity to go into more detail and show people examples and get them kind of hands-on than you can do in their, you know, 40-minute conference talk. So, I wanted to do that. I will say I have written myself a note to never do a full-size book while I have a full-time job because it's a lot [laugh].Corey: You do have a full-time job and then some. As we mentioned, you're the Chief Open Source Officer over at Isovalent, you are on the CNCF governing board, you're on the board of OpenUK, and you've done a lot of other stuff in the open-source community as well. So, I have to ask, taking all of that together, are you just allergic to things that make money? I mean, writing the book as well on top of that. I'm told you never do it for the money piece; it's always about the love of it. But it seems like, on some level, you're taking it to an almost ludicrous level.Liz: Yeah, I mean, I do get paid for my day job. So, there is that [laugh]. But so, yeah—Corey: I feel like that's the only way to really write a book is, in turn, to wind up only to just do it for—what someone else is paying you to for doing it, viewing it as a marketing exercise. It pays dividends, but those dividends don't, in my experience from what I've heard from everyone say, pay off as of royalties on book payments.Liz: Yeah, I mean, it's certainly, you know, not a bad thing to have that income stream, but it certainly wouldn't make you—you know, I'm not going to retire tomorrow on the royalty stream unless this podcast has loads and loads of people to buy the book [laugh].Corey: Exactly. And I'm always a fan of having such [unintelligible 00:03:58]. I will order it while we're on the call right now having this conversation because I believe in supporting the things that we want to see more of in the world. So, explain to me a little bit about what it is. Whatever you talking about learning X in a title, I find that that's often going to be much more approachable than arcane nonsense deep-dive things.One of the O'Reilly books that changed my understanding was Linux Kernel Internals, or Understanding the Linux Kernel. Understanding was kind of a heavy lift at that point because it got very deep very quickly, but I absolutely came away understanding what was going on a lot more effectively, even though I was so slow I needed a tow rope on some of it. When you have a book that started with learning, though, I imagined it assumes starting at zero with, “What's eBPF?” Is that directionally correct, or does it assume that you know a lot of things you don't?Liz: Yeah, that's absolutely right. I mean, I think eBPF is one of these technologies that is starting to be, particularly in the cloud-native world, you know, it comes up; it's quite a hot technology. What it actually is, so it's an acronym, right? EBPF. That acronym is almost meaningless now.So, it stands for extended Berkeley Packet Filter. But I feel like it does so much more than filtering, we might as well forget that altogether. And it's just become a term, a name in its own right if you like. And what it really does is it lets you run custom programs in the kernel so you can change the way that the kernel behaves, dynamically. And that is… it's a superpower. It's enabled all sorts of really cool things that we can do with that superpower.Corey: I just pre-ordered it as a paperback on Amazon and it shows me that it is now number one new release in Linux Networking and Systems Administration, so you're welcome. I'm sure it was me that put it over the top.Liz: Wonderful. Thank you very much. Yeah [laugh].Corey: Of course, of course. Writing a book is one of those things that I've always wanted to do, but never had the patience to sit there and do it or I thought I wasn't prolific enough, but over the holidays, this past year, my wife and business partner and a few friends all chipped in to have all of the tweets that I'd sent bound into a series of leather volumes. Apparently, I've tweeted over a million words. And… yeah, oh, so I have to write a book 280 characters at a time, mostly from my phone. I should tweet less was really the takeaway that I took from a lot of that.But that wasn't edited, that wasn't with an overall theme or a narrative flow the way that an actual book is. It just feels like a term paper on steroids. And I hated term papers. Love reading; not one to write it.Liz: I don't know whether this should make it into the podcast, but it reminded me of something that happened to my brother-in-law, who's an artist. And he put a piece of video on YouTube. And for unknowable reasons if you mistyped YouTube, and you spelt it, U-T-U-B-E, the page that you would end up at from Google search was a YouTube video and it was in fact, my brother-in-law's video. And people weren't expecting to see this kind of art movie about matches burning. And he just had the worst comment—like, people were so mean in the comments. And he had millions of views because people were hitting this page by accident, and he ended up—Corey: And he made the cardinal sin of never read the comments. Never break that rule. As soon as you do that, it doesn't go well. I do read the comments on various podcast platforms on this show because I always tell people to insulted all they want, just make sure you leave a five-star review.Liz: Well, he ended up publishing a book with these comments, like, one comment per page, and most of them are not safe for public consumption comments, and he just called it Feedback. It was quite something [laugh].Corey: On some level, it feels like O'Reilly books are a little insulated from the general population when it comes to terrible nonsense comments, just because they tend to be a little bit more expensive than the typical novel you'll see in an airport bookstore, and again, even though it is approachable, Learning eBPF isn't exactly the sort of title that gets people to think that, “Ooh, this is going to be a heck of a thriller slash page-turner with a plot.” “Well, I found the protagonist unrelatable,” is not sort of the thing you're going to wind up seeing in the comments because people thought it was going to be something different.Liz: I know. One day, I'm going to have to write a technical book that is also a murder mystery. I think that would be, you know, quite an achievement. But yeah, I mean, it's definitely aimed at people who have already come across the term, want to know more, and particularly if you're the kind of person who doesn't want to just have a hand-wavy explanation that involves boxes and diagrams, but if, like me, you kind of want to feel the code, and you want to see how things work and you want to work through examples, then that's the kind of person who might—I hope—enjoy working through the book and end up with a possible mental model of how eBPF works, even though it's essentially kernel programming.Corey: So, I keep seeing eBPF in an increasing number of areas, a bunch of observability tools, a bunch of security tools all tend to tie into it. And I've seen people do interesting things as far as cost analysis with it. The problem that I run into is that I'm not able to wind up deploying it universally, just because when I'm going into a client engagement, I am there in a purely advisory sense, given that I'm biasing these days for both SaaS companies and large banks, that latter category is likely going to have some problems if I say, “Oh, just take this thing and go ahead and deploy it to your entire fleet.” If they don't have a problem with that, I have a problem with their entire business security posture. So, I don't get to be particularly prescriptive as far as what to do with it.But if I were running my own environment, it is pretty clear by now that I would have explored this in some significant depth. Do you find that it tends to be something that is used primarily in microservices environments? Does it effectively require Kubernetes to become useful on day one? What is the onboard path where people would sit back and say, “Ah, this problem I'm having, eBPF sounds like the solution.”Liz: So, when we write tools that are typically going to be some sort of infrastructure, observability, security, networking tools, if we're writing them using eBPF, we're instrumenting the kernel. And the kernel gets involved every time our application wants to do anything interesting because whenever it wants to read or write to a file, or send receive network messages, or write something to the screen, or allocate memory, or all of these things, the kernel has to be involved. And we can use eBPF to instrument those events and do interesting things. And the kernel doesn't care whether those processes are running in containers, under Kubernetes, just running directly on the host; all of those things are visible to eBPF.So, in one sense, doesn't matter. But one of the reasons why I think we're seeing eBPF-based tools really take off in cloud-native is that you can, by applying some programming, you can link events that happened in the kernel to specific containers in specific pods in whatever namespace and, you know, get the relationship between an event and the Kubernetes objects that are involved in that event. And then that enables a whole lot of really interesting observability or security tools and it enables us to understand how network packets are flowing between different Kubernetes objects and so on. So, it's really having this vantage point in the kernel where we can see everything and we didn't have to change those applications in any way to be able to use eBPF to instrument them.Corey: When I see the stories about eBPF, it seems like it's focused primarily on networking and flow control. That's where I'm seeing it from a security standpoint, that's where I'm seeing it from cost allocation aspect. Because, frankly, out of the box, from a cloud provider's perspective, Kubernetes looks like a single-tenant application with a really weird behavioral pattern, and some of that crosstalk gets very expensive. Is there a better way than either using eBPF and/or VPC flow logs to figure out what's talking to what in the Kubernetes ecosystem, or is BPF really your first port of call?Liz: So, I'm coming from a position of perspective of working for the company that created the Cilium networking project. And one of the reasons why I think Cilium is really powerful is because it has this visibility—it's got a component called Hubble—that allows you to see exactly how packets are flowing between these different Kubernetes identities. So, in a Kubernetes environment, there's not a lot of point having network flows that talk about IP addresses and ports when what you really want to know is, what's the Kubernetes namespace, what's the application? Defining things in terms of IP addresses makes no sense when they're just being refreshed and renewed every time you change pods. So yeah, Kubernetes changes the requirements on networking visibility and on firewalling as well, on network policy, and that, I think, is you don't have to use eBPF to create those tools, but eBPF is a really powerful and efficient platform for implementing those tools, as we see in Cilium.Corey: The only competitor I found to it that gives a reasonable explanation of why random things are transferring multiple petabytes between each other in the middle of the night has been oral tradition, where I'm talking to people who've been around there for a while. It's, “So, I'm seeing this weird traffic pattern at these times a day. Any idea what that might be?” And someone will usually perk up and say, “Oh, is it—” whatever job that they're doing. Great. That gives me a direction to go in.But especially in this era of layoffs and as environments exist for longer and longer, you have to turn into a bit of a data center archaeologist. That remains insufficient, on some level. And some level, I'm annoyed with trying to understand or needing to use tooling like this that is honestly this powerful and this customizable, and yes, on some level, this complex in order to get access to that information in a meaningful sense. But on the other, I'm glad that that option is at least there for a lot of workloads.Liz: Yeah. I think, you know, that speaks to the power of this new generation of tooling. And the same kind of applies to security forensics, as well, where you might have an enormous stream of events, but unless you can tie those events back to specific Kubernetes identities, which you can use eBPF-based tooling to do, then how do you—the forensics job of tying back where did that event come from, what was the container that was compromised, it becomes really, really difficult. And eBPF tools—like Cilium has a sub-project called Tetragon that is really good at this kind of tying events back to the Kubernetes pod or whether we want to know what node it was running on what namespace or whatever. That's really useful forensic information.Corey: Talk to me a little bit about how broadly applicable it is. Because from my understanding from our last conversation, when you were on the show a year or so ago, if memory serves, one of the powerful aspects of it was very similar to what I've seen some of Brendan Gregg's nonsense doing in his kind of various talks where you can effectively write custom programming on the fly and it'll tell you exactly what it is that you need. Is this something that can be instrument once and then effectively use it for basically anything, [OTEL 00:16:11]-style, or instead, does it need to be effectively custom configured every time you want to get a different aspect of information out of it?Liz: It can be both of those things.Corey: “It depends.” My least favorite but probably the most accurate answer to hear.Liz: [laugh]. But I think Brendan did a really great—he's done many talks talking about how powerful BPF is and built lots of specific tools, but then he's also been involved with Bpftrace, which is kind of like a language for—a high-level language for saying what it is that you want BPF to trace out for you. So, a little bit like, I don't know, awk but for events, you know? It's a scripting language. So, you can have this flexibility.And with something like Bpftrace, you don't have to get into the weeds yourself and do kernel programming, you know, in eBPF programs. But also there's gainful employment to be had for people who are interested in that eBPF kernel programming because, you know, I think there's just going to be a whole range of more tools to come, you know>? I think we're, you know, we're seeing some really powerful tools with Cilium and Pixie and [Parker 00:17:27] and Kepler and many other tools and projects that are using eBPF. But I think there's also a whole load of more to come as people think about different ways they can apply eBPF and instrument different parts of an overall system.Corey: We're doing this over audio only, but behind me on my wall is one of my least favorite gifts ever to have been received by anyone. Mike, my business partner, got me a thousand-piece puzzle of the Kubernetes container landscape where—Liz: [laugh].Corey: This diagram is psychotic and awful and it looks like a joke, except it's not. And building that puzzle was maddening—obviously—but beyond that, it was a real primer in just how vast the entire container slash Kubernetes slash CNCF landscape really is. So, looking at this, I found that the only reaction that was appropriate was a sense of overwhelmed awe slash frustration, I guess. It's one of those areas where I spend a lot of time focusing on drinking from the AWS firehose because they have a lot of products and services because their product strategy is apparently, “Yes,” and they're updating these things in a pretty consistent cadence. Mostly. And even that feels like it's multiple full-time jobs shoved into one.There are hundreds of companies behind these things and all of them are in areas that are incredibly complex and difficult to go diving into. EBPF is incredibly powerful, I would say ridiculously so, but it's also fiendishly complex, at least shoulder-surfing behind people who know what they're doing with it has been breathtaking, on some level. How do people find themselves in a situation where doing a BPF deep dive make sense for them?Liz: Oh, that's a great question. So, first of all, I'm thinking is there an AWS Jigsaw as well, like the CNCF landscape Jigsaw? There should be. And how many pieces would it have? [It would be very cool 00:19:28].Corey: No, because I think the CNCF at one point hired a graphic designer and it's unclear that AWS has done such a thing because their icons for services are, to be generous here, not great. People have flashcards that they've built for is what services does logo represent? Haven't a clue, in almost every case because I don't care in almost every case. But yeah, I've toyed with the idea of doing it. It's just not something that I'd ever want to have my name attached to it, unfortunately. But yeah, I want someone to do it and someone else to build it.Liz: Yes. Yeah, it would need to refresh every, like, five minutes, though, as they roll out a new service.Corey: Right. Because given that it appears from the outside to be impenetrable, it's similar to learning VI in some cases, where oh, yeah, it's easy to get started with to do this trivial thing. Now, step two, draw the rest of the freaking owl. Same problem there. It feels off-putting just from a perspective of you must be at least this smart to proceed. How do you find people coming to it?Liz: Yeah, there is some truth in that, in that beyond kind of Hello World, you quite quickly start having to do things with kernel data structures. And as soon as you're looking at kernel data structures, you have to sort of understand, you know, more about the kernel. And if you change things, you need to understand the implications of those changes. So, yeah, you can rapidly say that eBPF programming is kernel programming, so why would anybody want to do it? The reason why I do it myself is not because I'm a kernel programmer; it's because I wanted to really understand how this is working and build up a mental model of what's happening when I attach a program to an event. And what kinds of things can I do with that program?And that's the sort of exploration that I think I'm trying to encourage people to do with the book. But yes, there is going to be at some point, a pretty steep learning curve that's kernel-related but you don't necessarily need to know everything in order to really have a decent understanding of what eBPF is, and how you might, for example—you might be interested to see what BPF programs are running on your existing system and learn why and what they might be doing and where they're attached and what use could that be.Corey: Falling down that, looking at the process table once upon a time was a heck of an education, one week when I didn't have a lot to do and I didn't like my job in those days, where, “Oh, what is this Avahi daemon that constantly running? MDNS forwarding? Who would need that?” And sure enough, that tickled something in the back of my mind when I wound up building out my networking box here on top of BSD, and oh, yeah, I want to make sure that I can still have discovery work from the IoT subnet over to whatever it is that my normal devices live. Ah, that's what that thing always running for. Great for that one use case. Almost never needed in other cases, but awesome. Like, you fire up a Raspberry Pi. It's, “Why are all these things running when I'm just want to have an embedded device that does exactly one thing well?” Ugh. Computers have gotten complicated.Liz: I know. It's like when you get those pop-ups on—well certainly on Mac, and you get pop-ups occasionally, let's say there's such and such a daemon wants extra permissions, and you think I'm not hitting that yes button until I understand what that daemon is. And it turns out, it's related, something completely innocuous that you've actually paid for, but just under a different name. Very annoying. So, if you have some kind of instrumentation like tracing or logging or security tooling that you want to apply to all of your containers, one of the things you can use is a sidecar container approach. And in Kubernetes, that means you inject the sidecar into every single pod. And—Corey: Yes. Of course, the answer to any Kubernetes problem appears to be have you tried running additional containers?Liz: Well, right. And there are challenges that can come from that. And one of the reasons why you have to do that is because if you want a tool that has visibility over that container that's inside the pod, well, your instrumentation has to also be inside the pod so that it has visibility because your pod is, by design, isolated from the host it's running on. But with eBPF, well eBPF is in the kernel and there's only one kernel, however many containers were running. So, there is no kind of isolation between the host and the containers at the kernel level.So, that means if we can instrument the kernel, we don't have to have a separate instance in every single pod. And that's really great for all sorts of resource usage, it means you don't have to worry about how you get those sidecars into those pods in the first place, you know that every pod is going to be instrumented if it's instrumented in the kernel. And then for service mesh, service mesh usually uses a sidecar as a Layer 7 Proxy injected into every pod. And that actually makes for a pretty convoluted networking path for a packet to sort of go from the application, through the proxy, out to the host, back into another pod, through another proxy, into the application.What we can do with eBPF, we still need a proxy running in userspace, but we don't need to have one in every single pod because we can connect the networking namespaces much more efficiently. So, that was essentially the basis for sidecarless service mesh, which we did in Cilium, Istio, and now we're using a similar sort of approach with Ambient Mesh. So that, again, you know, avoiding having the overhead of a sidecar in every pod. So that, you know, seems to be the way forward for service mesh as well as other types of instrumentation: avoiding sidecars.Corey: On some level, avoiding things that are Kubernetes staples seems to be a best practice in a bunch of different directions. It feels like it's an area where you start to get aligned with the idea of service meesh—yes, that's how I pluralize the term service mesh and if people have a problem with that, please, it's imperative you've not send me letters about it—but this idea of discovering where things are in a variety of ways within a cluster, where things can talk to each other, when nothing is deterministically placed, it feels like it is screaming out for something like this.Liz: And when you think about it, Kubernetes does sort of already have that at the level of a service, you know? Services are discoverable through native Kubernetes. There's a bunch of other capabilities that we tend to associate with service mesh like observability or encrypted traffic or retries, that kind of thing. But one of the things that we're doing with Cilium, in general, is to say, but a lot of this is just a feature of the networking, the underlying networking capability. So, for example, we've got next generation mutual authentication approach, which is using SPIFFE IDs between an application pod and another application pod. So, it's like the equivalent of mTLS.But the certificates are actually being passed into the kernel and the encryption is happening at the kernel level. And it's a really neat way of saying we don't need… we don't need to have a sidecar proxy in every pod in order to terminate those TLS connections on behalf of the application. We can have the kernel do it for us and that's really cool.Corey: Yeah, at some level, I find that it still feels weird—because I'm old—to have this idea of one shared kernel running a bunch of different containers. I got past that just by not requiring that [unintelligible 00:27:32] workloads need to run isolated having containers run on the same physical host. I found that, for example, running some stuff, even in my home environment for IoT stuff, things that I don't particularly trust run inside of KVM on top of something as opposed to just running it as a container on a cluster. Almost certainly stupendous overkill for what I'm dealing with, but it's a good practice to be in to start thinking about this. To my understanding, this is part of what AWS's Firecracker project starts to address a bit more effectively: fast provisioning, but still being able to use different primitives as far as isolation boundaries go. But, on some level, it's nice to not have to think about this stuff, but that's dangerous.Liz: [laugh]. Yeah, exactly. Firecracker is really nice way of saying, “Actually, we're going to spin up a whole VM,” but we don't ne—when I say ‘whole VM,' we don't need all of the things that you normally get in a VM. We can get rid of a ton of things and just have the essentials for running that Lambda or container service, and it becomes a really nice lightweight solution. But yes, that will have its own kernel, so unlike, you know, running multiple kernels on the same VM where—sorry, running multiple containers on the same virtual machine where they would all be sharing one kernel, with Firecracker you'll get a kernel per instance of Firecracker.Corey: The last question I have for you before we wind up wrapping up this episode harkens back to something you said a little bit earlier. This stuff is incredibly technically nuanced and deep. You clearly have a thorough understanding of it, but you also have what I think many people do not realize is an orthogonal skill of being able to articulate and explain those complex concepts simply an approachably, in ways that make people understand what it is you're talking about, but also don't feel like they're being spoken to in a way that's highly condescending, which is another failure mode. I think it is not particularly well understood, particularly in the engineering community, that there are—these are different skill sets that do not necessarily align congruently. Is this something you've always known or is this something you've figured out as you've evolved your career that, oh I have a certain flair for this?Liz: Yeah, I definitely didn't always know it. And I started to realize it based on feedback that people have given me about talks and articles I'd written. I think I've always felt that when people use jargon or they use complicated language or they, kind of, make assumptions about how things are, it quite often speaks to them not having a full understanding of what's happening. If I want to explain something to myself, I'm going to use straightforward language to explain it to myself [laugh] so I can hold it in my head. And I think people appreciate that.And you can get really—you know, you can get quite in-depth into something if you just start, step by step, build it up, explain everything as you go along the way. And yeah, I think people do appreciate that. And I think people, if they get lost in jargon, it doesn't help anybody. And yeah, I very much appreciate it when people say that, you know, they saw a talk or they read something I wrote and it meant that they finally grokked whatever that concept was that that I was trying to explain. I will say at the weekend, I asked ChatGPT to explain DNS in the style of Liz Rice, and it started off, it was basically, “Hello there. I'm Liz Rice and I'm here to explain DNS in very simple terms.” I thought, “Okay.” [laugh].Corey: Every time I think I've understood DNS, there's another level to it.Liz: I'm pretty sure there is a lot about DNS that I don't understand, yeah. So, you know, there's always more to learn out there.Corey: There's certainly is. I really want to thank you for taking time to speak with me today about what you're up to. Where's the best place for people to find you to learn more? And of course, to buy the book.Liz: Yeah, so I am Liz Rice pretty much everywhere, all over the internet. There is a GitHub repo that accompanies the books that you can find that on GitHub: lizRice/learning-eBPF. So, that's a good place to find some of the example code, and it will obviously link to where you can download the book or buy it because you can pay for it; you can also download it from Isovalent for the price of your contact details. So, there are lots of options.Corey: Excellent. And we will, of course, put links to that in the [show notes 00:32:08]. Thank you so much for your time. It's always great to talk to you.Liz: It's always a pleasure, so thanks very much for having me, Corey.Corey: Liz Rice, Chief Open Source Officer at Isovalent. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, along with an angry comment that you have somehow discovered this episode by googling for knitting projects.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.

Welcome to the newest episode of The Cloud Pod podcast! Justin, Ryan and Jonathan are your hosts this week as we discuss all the latest news and announcements in the world of the cloud and AI - including Amazon's new AI, Bedrock, as well as new AI tools from other developers. We also address the new updates to AWS's CodeWhisperer, and return to our Cloud Journey Series where we discuss *insert dramatic music* - Kubernetes!  Titles we almost went with this week: ⭐I'm always Whispering to My Code as an Individual

Michael Isbitski, Director of Cybersecurity Strategy at Sysdig, joins Corey on Screaming in the Cloud to discuss the nuances of an effective cybersecurity strategy. Michael explains that many companies are caught between creating a strategy that's truly secure and one that's merely compliant and within the bounds of cost-effectiveness, and what can be done to help balance the two aims more effectively. Corey and Michael also explore what it means to hire for transferrable skills in the realm of cybersecurity and tech, and Michael reveals that while there's no such thing as a silver-bullet solution for cybersecurity, Sysdig can help bridge many gaps in a company's strategy. About MichaelMike has researched and advised on cybersecurity for over 5 years. He's versed in cloud security, container security, Kubernetes security,  API security, security testing, mobile security, application protection, and secure continuous delivery. He's guided countless organizations globally in their security initiatives and supporting their business.Prior to his research and advisory experience, Mike learned many hard lessons on the front lines of IT with over twenty years of practitioner and leadership experience focused on application security, vulnerability management, enterprise architecture, and systems engineering.Links Referenced: Sysdig: https://sysdig.com/ LinkedIn: https://www.linkedin.com/in/michael-isbitski/ TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Tailscale SSH is a new, and arguably better way to SSH. Once you've enabled Tailscale SSH on your server and user devices, Tailscale takes care of the rest. So you don't need to manage, rotate, or distribute new SSH keys every time someone on your team leaves. Pretty cool, right? Tailscale gives each device in your network a node key to connect to your VPN, and uses that same key for SSH authorization and encryption. So basically you're SSHing the same way that you're already managing your network.So what's the benefit? Well, built-in key rotation, the ability to manage permissions as code, connectivity between any two devices, and reduced latency. You can even ask users to re-authenticate SSH connections for that extra bit of security to keep the compliance folks happy. Try Tailscale now - it's free forever for personal use.Corey: Do you wish your developers had less permanent access to AWS? Has the complexity of Amazon's reference architecture for temporary elevated access caused you to sob uncontrollably? With Sym, you can protect your cloud infrastructure with customizable, just-in-time access workflows that can be setup in minutes. By automating the access request lifecycle, Sym helps you reduce the scope of default access while keeping your developers moving quickly. Say goodbye to your cloud access woes with Sym. Go to symops.com/corey to learn more. That's S-Y-M-O-P-S.com/coreyCorey: Welcome to Screaming in the Cloud, I'm Corey Quinn. I periodically find myself in something of a weird spot when it comes to talking about security. I spent a lot of my time in previous lives having to care about it, but the word security was never in my job title. That's who my weekly podcast on the AWS Morning Brief and the accompanying newsletter goes out to: it's people who have to care about security but don't have it as part of their job title. They just want to know what's going on without all of the buzzwords.This promoted guest episode is brought to us by our friends at Sysdig and my guest is Mike Isbitski, Director of Cybersecurity Strategy at Sysdig. Mike, thanks for joining me.Michael: Thanks, Corey. Yeah, it's great to be here.Corey: So, you've been at Sysdig for a little bit, but your history is fascinating to me. You were at Gartner, which on the one hand would lead someone to think, “Oh okay, you talk about this stuff a lot, but might not have been particularly hands-on,” but that's not true. Either. You have a strong background as a practitioner, but not directly security-focused. Is that right?Michael: Yeah. Yeah, that is correct. I can certainly give the short version of the history lesson [laugh]. It is true, yes. As a Gartner analyst, you don't always get as hands-on, certainly talking to practitioners and leaders from all walks of life, different industries, different company sizes, and organization sizes.But yeah, as a Gartner analyst, I was in a different division that was much more technical. So, for me personally, I did actually try to tinker a lot: set up Docker, deploy Kubernetes clusters, all that fun stuff. But yeah, prior to my life, as an analyst, I was a practitioner, a security leader for close to 20 years at Verizon so, saw quite a bit. And actually started as enterprise architect building, kind of, systems and infrastructure to support all of those business needs, then I kind of transitioned over to application security towards the tail end of that career at Verizon.Corey: And one of the things that I find that I enjoy doing is talking with folks in positions like yours, the folks who did not come to the cybersecurity side of the world from a pure strategy advisory sense, but have been hands-on with these things at varying points in our careers, just because otherwise I feel like I'm sort of coming at this from a very different world. When I walk around the RSA show floor, I am consistently confronted by people trying to sell me the same dozen products over and over again with different words and different branding, but it seems like it's all buzzwords aimed from security people who are deep in the weeds to other security people who are deep in the weeds and it's just presumed that everyone knows what they're talking about already. And obviously worse. I'm not here to tell them that they're going about their business wrong, but for smaller companies, SMBs, folks who have to care about security but don't know the vernacular in the same way and don't have sophisticated security apparatus at their companies, it feels like a dense thicket of impenetrable buzzwords.Michael: Yes. Very, very fair assessment, [laugh] I would say. So, I'd say my life as an analyst was a lot of lengthy conversations. I guess a little bit of the secret behind analyst inquiry, I mean, a lot of times, they are hour-long conversations, sometimes multiple sets of them. But yeah, it's very true, right?There's a lot of nuance to how you work with technology and how you build things, but then also how you secure it, it's very hard to, kind of, condense that, you know, hours of conversation and many pages of documentation down into some bite-size nuggets that marketers might run with. So, I try to kind of live in that in-between world where I can kind of explain deep technology problems and business realities, and kind of explain that in more common language to people. Sometimes it's easier said than done when you're speaking it as opposed to writing it. But yeah, that's kind of where I tried to bring my skills and experience.Corey: It's a little counterintuitive to folks coming out from the other side, I suspect. For me, at least the hardest part of getting into the business of cloud cost optimization the way that I do with the Duckbill Group was learning to talk. Where I come from a background of heavy on the engineering and operations side, but being able to talk to business stakeholders who do not particularly care what a Kubernetes might be, is critical. You have to effectively be able to speak to different constituencies, sometimes in the same conversation, without alienating the rest of them. That was the hard part for me.Michael: Yeah, that's absolutely true and I certainly ran into that quite a bit as an enterprise architect at Verizon. There's kind of really need to work to identify, like, what is the business need. And typically, that is talking to the stakeholders, you know, what are they trying to achieve? They might not even know that, right, [laugh] because not everybody is very structured in how they think about the problem you're trying to solve. And then what is their daily workflow?And then you kind of arrive at the technology. I'd say, a common pitfall for anybody, right, Whether you're an engineer or a security practitioner is to kind of start with the technology or the solution and then try to force that on people, right? “Here's your solution to the problem that maybe you didn't know you had.” [laugh]. It kind of should work in reverse, right? What's the actual business need? What's your workflow? And what's the appropriate technology for that, right?Whether it's right-sizing the infrastructure or a particular type of functionality or protection, all those things, right? So, very similar kind of way of approaching the problem. It's just what you're trying to solve but [laugh] I've definitely seen that, kind of, Kubernetes is all the rage, right, or service mesh. Like, everybody needs to start deploying Istio, and you really should be asking the question—Corey: Oh, it's all resume-driven development.Michael: Yep, exactly. Yeah. It's kind of the new kid on the block, right? Let's push out this cool new technology and then problems be damned, right?Corey: I'm only half-kidding on that. I've talked to folks who are not running those types of things and they said that it is a bit of a drag on their being able to attract talent.Michael: Yeah, it's—you know, I mean, it's newer technologies, right, so it can be hard to find them, right, kind of unicorn status. I used to talk quite a bit in advisory calls to find DevOps practitioners that were kind of full-stack. That's tricky.Corey: I always wonder if it's possible to find them, on some level.Michael: Yeah. And it's like, well, can you find them and then when you do find them, can you afford them?Corey: Oh, yeah. What I'm seeing in these other direction, though, is people who are making, you know, sensible technology choices where you actually understand what lives were without turning it into a murder mystery where you need to hire a private investigator to track it down. Those are the companies that are having trouble hiring because it seems that an awful lot of the talent, or at least a significant subset of it, want to have the latest and greatest technologies on their resume on their next stop. Which, I'm not saying they're wrong for doing that, but it is a strange outcome that I wasn't quite predicting.Michael: Yeah. No, it is very true, I definitely see that quite a bit in tech sector. I've run into it myself, even with the amount of experience I have and skills. Yeah, companies sometimes get in a mode where they're looking for very specific skills, potentially even products or technologies, right? And that's not always the best way to go about it.If you understand concepts, right, with technology and systems engineering, that should translate, right? So, it's kind of learning the new syntax, or semantics, working with a framework or a platform or a piece of technology.Corey: One of the reasons that I started the security side of what I do on the newsletter piece, and it caught some people by surprise, but the reason I did it was because I have always found that, more or less, security and cost are closely aligned spiritually, if nothing else. They're reactive problems and they don't, in the general sense, get companies one iota closer to the business outcome they're chasing, but it's something you have to do, like buying fire insurance for the building. You can spend infinite money on those things, but it doesn't advance. It's all on the defensive, reactive side. And you tend to care about these things a lot right after you failed to care about them sufficiently. Does that track at all from your experience?Michael: Yeah. Yeah, absolutely. I'm just kind of flashing back to some war stories at Verizon, right? It was… I'd say very common that, once you've kind of addressed, well, these are the business problems we want to solve for and we're off to the races, right, we're going to build this cool thing. And then you deploy it, right [laugh], and then you forgot to account for backup, right? What's your disaster recovery plan? Do you have logging in place? Are you monitoring the thing effectively? Are your access controls accounted for?All those, kind of, tangential processes, but super-critical, right, when you think about, kind of, production systems, like, they have to be in place. So, it's absolutely true, right, and it's kind of definitely for just general availability, you need to be thinking about these things. And yeah, they almost always translate to that security piece of it as well, right, particularly with all the regulations that organizations are impacted with today. You really need to be thinking about, kind of, all these pieces of the puzzle, not just hey, let's build this thing and get it on running infrastructure and we're done with our work.Corey: A question that I've got for you—because I'm seeing a very definite pattern emerging tied to the overall macro environment, now, where after a ten-year bull run, suddenly a bunch of companies are discovering, holy crap, money means something again, where instead of being able to go out and gets infinite money, more or less, to throw at an AWS bill, suddenly, oh, that's a big number, and we have no idea what's in it. We should care about that. So, almost overnight, we've seen people suddenly caring about their bill. How are you seeing security over the past year or so? Has there been a similar awareness around that or has that not really been tied to the overall macro-cycle?Michael: Very good question, yeah. So unfortunately, security's often an afterthought, right, just like, kind of those things that support availability—probably going to get a little bit better ranking because it's going to support your customers and employees, so you're going to get budget and headcount to support that. Security, usually in the pecking order, is below that, right, which is unfortunate because [laugh] there can be severe repercussions with that, such as privacy impacts, or data breach, right, lost revenue, all kinds of things. But yeah, typically, security has been undercut, right? You're always seeking headcount, you need more budget.So, security teams tend to look to delegate security process out, right? So, you kind of see a lot of DevOps programs, like, can we empower engineers to run some of these processes and tooling, and then security, kind of, becomes the overseer. So, we see a lot of that where can we kind of have people satisfy some of these pieces. But then with respect to, like, security budgets, it is often security tools consolidation because a lot organizations tend to have a lot of things, right? So, security leaders are looking to scale that back, right, so they can work more effectively, but then also cut costs, which is definitely true these days in the current macroeconomic environment.Corey: I'm curious as well, to see what your take is on the interplay between cost and security. And what I mean by that is, I did the numbers once, and if you were to go into an AWS native environment, ignore third-party vendors for a second, just configure all of the AWS security services in your account, so the way that best practices dictate that you should, you're pretty quickly going to end up in a scenario where the cost of that outweighs that of the data breach that you're ostensibly trying to prevent. So—Michael: Yes.Corey: It's an infinite money pit that you can just throw everything into. So, people care about security, but they also care about cost. Plus, let's be very direct here, you can spend all the money on security and still lose. How do companies think about that now?Michael: A lot of leaders will struggle with, are we trying to be compliant or are we trying to be secure? Because those can be very different conversations and solutions to the problem. I mean, ideally, everybody would pursue that perfect model of security, right, enable all the things, but that's not necessarily cost-effective to do that. And so, most organizations and most security teams are going to prioritize their risks, right? So, they'll start to carve out, maybe these are all our internet-facing applications, these are the business-critical ones, so we're going to allocate more security focus to them and security spend, so [maybe we will be turn up 00:13:20] more security services to protect those things and monitor them.Then [laugh], unfortunately, you can end up with a glut of maybe internal applications or non-critical things that just don't get that TLC from security, unfortunately, for security teams, but fortunate for attackers, those things become attack targets, right? So, they don't necessarily care how you've prioritized your controls or your risk. They're going to go for the low-hanging fruit. So, security teams have always struggled with that, but it's very true. Like, in a cloud environment like AWS, yeah, if you start turning everything up, be prepared for a very, very costly cloud expense bill.Corey: Yeah, in my spare time, I'm working on a project that I was originally going to open-source, but I realized if I did it, it would cause nothing but pain and drama for everyone, of enabling a whole bunch of AWS misconfiguration options, given a set of arbitrary credentials, that just effectively try to get the high score on the bill. And it turned out that my early tests were way more successful than anticipated, and instead, I'm just basically treating it as a security vulnerability reporting exercise, just because people don't think about this in quite the same way. And again, it's not that these tools are necessarily overpriced; it's not that they aren't delivering value. It's that in many cases, it is unexpectedly expensive when they bill across dimensions that people are not aware of. And it's one of those everyone's aware of that trap the second time type of situations.It's a hard problem. And I don't know that there's a great way to answer it. I don't think that AWS is doing anything untoward here; I don't think that they're being intentionally malicious around these things, but it's very vast, very complex, and nobody sees all of it.Michael: Very good point, yes. Kind of, cloud complexity and ephemeral nature of cloud resources, but also the cost, right? Like, AWS isn't in the business of providing free service, right? Really, no cloud provider is. They are a business, right, so they want to make money on Cloud consumption.And it's interesting, I remember, like, the first time I started exploring Kubernetes, I did deploy clusters in cloud providers, so you can kind of tinker and see how these things work, right, and they give you some free credits, [a month of credit 00:15:30], to kind of work with this stuff. And, you know, if you spin up a [laugh] Kubernetes cluster with very bare bones, you're going to chew through that probably within a day, right? There's a lot of services in it. And that's even with defaults, which includes things like minimal, if anything, with respect to logging. Which is a problem, right, because then you're going to miss general troubleshooting events, but also actual security events.So, it's not necessarily something that AWS could solve for by turning everything up, right, because they are going to start giving away services. Although I'm starting to see some tide shifts with respect to cybersecurity. The Biden administration just released their cybersecurity strategy that talks about some of this, right? Like, should cloud providers start assuming more of the responsibility and accountability, potentially just turning up logging services? Like, why should those be additional cost to customers, right, because that's really critical to even support basic monitoring and security monitoring so you can report incidents and breaches.Corey: When you look across what customers are doing, you have a different problem than I do. I go in and I say, “Oh, I fixed the horrifying AWS bill.” And then I stop talking and I wait. Because if people [unintelligible 00:16:44] to that, “Ooh, that's a problem for us,” great. We're having a conversation.If they don't, then there's no opportunity for my consulting over in that part of the world. I don't have to sit down and explain to people why their bill is too high or why they wouldn't want it to be they intrinsically know and understand it or they're honestly not fit to be in business if they can't make a strategic evaluation of whether or not their bill is too high for what they're doing. Security is very different, especially given how vast it is and how unbounded the problem space is, relatively speaking. You have to first educate customers in some ways before attempting to sell them something. How do you do that without, I guess, drifting into the world of FUD where, “Here are all the terrible things that could happen. The solution is to pay me.” Which in many cases is honest, but people have an aversion to it.Michael: Yeah. So, that's how I feel [laugh] a lot of my days here at Sysdig. So, I do try to explain, kind of, these problems in general terms as opposed to just how Sysdig can help you solve for it. But you know, in reality, it is larger strategic challenges, right, there's not necessarily going to be one tool that's going to solve all your problems, the silver bullet, right, it's always true. Yes, Sysdig has a platform that can address a lot of cloud security-type issues, like over-permissioning or telling you what are the actual exploitable workloads in your environment, but that's not necessarily going to help you with, you know, if you have a regulator breathing down your neck and wants to know about an incident, how do you actually relay that information to them, right?It's really just going to help surface event data, stitch things together, that now you have to carry that over to that person or figure out within your organization who's handling that. So, there is kind of this larger piece of, you know, governance risk and compliance, and security tooling helps inform a lot of that, but yeah, every organization is, kind of, have to answer to [laugh] those authorities, often within their own organization, but it could also be government authorities.Corey: Part of the challenge as well is that there's—part of it is tooling, absolutely, but an awful lot of it is a people problem where you have these companies in the security space talking about a variety of advanced threats, of deeply sophisticated attackers that are doing incredibly arcane stuff, and then you have the CEO yelling about what they're doing on a phone call in the airport lounge and their password—which is ‘kitty' by the way—is on a Post-It note on their laptop for everyone to see. It feels like it's one of those, get the basic stuff taken care of first, before going down the path to increasingly arcane attacks. There's an awful lot of vectors to wind up attacking an infrastructure, but so much of what we see from data breaches is simply people not securing S3 buckets, as a common example. It's one of those crawl, walk, run types of stories. For what you do, is there a certain level of sophistication that companies need to get to before what you offer starts to bear fruit?Michael: Very good question, right, and I'd start with… right, there's certainly an element of truth that we're lagging behind on some of the security basics, right, or good security hygiene. But it's not as simple as, like, well, you picked a bad password or you left the port exposed, you know? I think certainly security practitioners know this, I'd even put forth that a lot of engineers know it, particularly if they're been trained more recently. There's been a lot of work to promote security awareness, so we know that we should provide IDs and passwords of sufficient strength, don't expose things you shouldn't be doing. But what tends to happen is, like, as you build monitoring systems, they're just extremely complex and distributed.Not to go down the weeds with app designs, with microservices architecture patterns, and containerized architectures, but that is what happens, right, because the days of building some heavyweight system in the confines of a data center in your organization, those things still do happen, but that's not typically how new systems are being architected. So, a lot of the old problems still linger, there's just many more instances of it and it's highly distributed. So it, kind of, the—the problem becomes very amplified very quickly.Corey: That's, I think, on some level, part of the challenge. It's worse in some ways that even the monitoring and observability space where, “All right, we have 15 tools that we're using right now. Why should we talk to yours?” And the answer is often, “Because we want to be number 16.” It's one of those stories where it winds up just adding incremental cost. And by cost, I don't just mean money; I mean complexity on top of these things. So, you folks are, of course, sponsoring this episode, so the least I can do is ask you, where do you folks start and stop? Sysdig: you do a lot of stuff. What's the sweet spot?Michael: Yeah, I mean, there's a few, right, because it is a larger platform. So, I often talk in terms of full lifecycle security, right? And a lot of organizations will split their approaches. We'll talk about shift left, which is really, let's focus very heavily on secure design, let's test all the code and all the artifacts prior to delivering that thing, try to knock out all quality issues, right, for kind of that general IT, but also security problems, which really should be tracked as quality issues, but including those things like vulnerabilities and misconfigs. So, Sysdig absolutely provides that capability that to satisfy that shift left approach.And Sysdig also focuses very heavily on runtime security or the shield right side of the equation. And that's, you know, give me those capabilities that allow me to monitor all types of workloads, whether they're virtual machines, or containers, serverless abstractions like Fargate because I need to know what's going on everywhere. In the event that there is a potential security incident or breach, I need all that information so I can actually know what happened or report that to a regulatory authority.And that's easier said than done, right? Because when you think about containerized environments, they are very ephemeral. A container might spin up a tear down within minutes, right, and if you're not thinking about your forensics and incident response processes, that data is going to be lost [unintelligible 00:23:10] [laugh]. You're kind of shooting yourself in the foot that way. So yeah, Sysdig kind of provides that platform to give you that full range of capabilities throughout the lifecycle.Corey: I think that that is something that is not fully understood in a lot of cases. I remember a very early Sysdig, I don't know if it was a demo or what exactly it was, I remember was the old Heavybit space in San Francisco, where they came out, it was, I believe, based on an open-source project and it was still taking the perspective, isn't this neat? It gives you really in-depth insight into almost a system-call level of what it is the system is doing. “Cool. So, what's the value proposition for this?”It's like, “Well, step one, be an incredibly gifted engineer when it comes to systems internals.” It's like, “Okay, I'll be back in five years. What's step two?” It's like, “We'll figure it out then.” Now, the story has gone up the stack. It originally felt a little bit like it was a solution in search of a problem. Now, I think you have found that problem, you have clearly hit product-market fit. I see you folks in the wild in many of my customer engagements. You are doing something very right. But it was neat watching, like, it's almost for me, I turned around, took my eye off the ball for a few seconds and it went from, “We have no idea of what we're doing” to, “We know exactly what we're doing.” Nice work.Michael: Yeah. Yeah. Thanks, Corey. Yeah, and there's quite a history with Sysdig in the open-source community. So, one of our co-founders, Loris Degioanni, was one of the creators of Wireshark, which some of your listeners may be familiar with.So, Wireshark was a great network traffic inspection and observability tool. It certainly could be used by, you know, just engineers, but also security practitioners. So, I actually used it quite a bit in my days when I would do pen tests. So, a lot of that design philosophy carried over to the Sysdig open source. So, you're absolutely correct.Sysdig open source is all about gathering that sys-call data on what is happening at that low level. But it's just one piece of the puzzle, exactly as you described. The other big piece of open-source that Sysdig does provide is Falco, which is kind of a threat detection and response engine that can act on all of those signals to tell you, well, what is actually happening is this potentially a malicious event? Is somebody trying to compromise the container runtime? Are they trying to launch a suspicious process? So that those pieces are there under the hood, right, and then Sysdig Secure is, kind of, the larger platform of capabilities that provide a lot of the workflow, nice visualizations, all those things you kind of need to operate at scale when you're supporting your systems and security.Corey: One thing that I do find somewhat interesting is there's always an evolution as companies wind up stumbling through the product lifecycle, where originally it starts off as we have an idea around one specific thing. And that's great. And for you folks, it feels like it was security. Then it started changing a little bit, where okay, now we're going to start doing different things. And I am very happy with the fact right now that when I look at your site, you have two offerings and not two dozen, like a number of other companies tend to. You do Sysdig Secure, which is around the security side of the world, and Sysdig Monitor, which is around the observability side of the world. How did that come to be?Michael: Yeah, it's a really good point, right, and it's kind of in the vendor space [laugh], there's also, like, chasing the acronyms. And [audio break 00:26:41] full disclosure, we are guilty of that at times, right, because sometimes practitioners and buyers seek those things. So, you have to kind of say, yeah, we checked that box for CSPM or CWPP. But yeah, it's kind of talking more generally to organizations and how they operate their businesses, like, that's more well-known constructs, right? I need to monitor this thing or I need to get some security. So, lumping into those buckets helps that way, right, and then you turn on those capabilities you need to support your environment, right?Because you might not be going full-bore into a containerized environment, and maybe you're focusing specifically on the runtime pieces and you're going to, kind of, circle back on security testing in your build pipeline. So, you're only going to use some of those features at the moment. So, it is kind of that platform approach to addressing that problem.Corey: Oh, I would agree. I think that one of the challenges I still have around the observability space—which let's remind people, is hipster monitoring; I don't care what other people say. That's what it is—is that it is depressingly tied to a bunch of other things. To this day, the only place to get a holistic view of everything in your AWS account in every region is the bill. That somehow has become an observability tool. And that's ridiculous.On the other side of it, I have had several engagements that inadvertently went from, “We're going to help optimize your cost,” to, “Yay. We found security incidents.” I don't love a lot of these crossover episodes we wind up seeing, but it is the nature of reality where security, observability, and yes, costs all seem to tie together to some sort of unholy triumvirate. So, I guess the big question is when does Sysdig launch a cost product?Michael: Well, we do have one [laugh], specifically for—Corey: [laugh]. Oh, events once again outpace me.Michael: [laugh]. But yeah, I mean, you touched on this a few times in our discussion today, right? There's heavy intersections, right, and the telemetry you need to gather, right, or the log data you need to gather to inform monitoring use cases or security use cases, a lot of the times that telemetry is the same set of data, it's just you're using it for different purposes. So, we actually see this quite commonly where Sysdig customers might pursue, Monitor or Secure, and then they actually find that there's a lot of value-add to look at the other pieces.And it goes both ways, right? They might start with the security use cases and then they find, well, we've over-allocated on our container environments and we're over-provisioning in Kubernetes resources, so all right, that's cool. We can actually reduce costs that could help create more funding to secure more hosts or more workloads in an environment, right? So it's, kind of, show me the things I'm doing wrong on this side of the equation, whether that's general IT security problems and then benefit the other. And yeah, typically we find that because things are so complex, yeah, you're over-permissioning you're over-allocating, it's just very common, rights? Kubernetes, as amazing as it can be or is, it's really difficult to operate that in practice, right? Things can go off the rails very, very quickly.Corey: I really want to thank you for taking time to speak about how you see the industry and the world. If people want to learn more, where's the best place for them to find you?Michael: Yes, thanks, Corey. It's really been great to be here and talk with you about these topics. So, for me personally, you know, I try to visit LinkedIn pretty regularly. Probably not daily but, you know, at least once a week, so please, by all means, if you ever have questions, do contact me. I love talking about this stuff.But then also on Sysdig, sysdig.com, I do author content on there. I speak regularly in all types of event formats. So yeah, you'll find me out there. I have a pretty unique last name. And yeah, that's kind of it. That's the, I'd say the main sources for me at the moment. Don't fall for the other Isbitski; that's actually my brother, who does work for AWS.Corey: [laugh]. That's okay. There's no accounting for family, sometimes.Michael: [laugh].Corey: I kid, I kid. Okay, great company. Great work. Thank you so much for your time. I appreciate it.Michael: Thank you, Corey.Corey: Mike Isbitski, Director of Cybersecurity Strategy at Sysdig. I'm Cloud Economist Corey Quinn and this has been a promoted guest episode brought to us by our friends at Sysdig. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with an angry, insulting comment from your place, which is no doubt expensive, opaque, and insecure, hitting all three points of that triumvirate.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.

In this episode, we talk with Steve McNutt, a cybersecurity professional at Cisco. Steve has seen the industry change several times, and together we cover the traditional security design as well as how to design for cloud security in the new age. We bounced around between a lot of security topics in this show, so there are a lot of links to share!How to connect with our guest:Twitter: [https://twitter.com/densem0de]Blog: [https://densemode.com/]Topics:NIST: [https://www.nist.gov]AWS GWLB: [https://aws.amazon.com/elasticloadbalancing/gateway-load-balancer/]Azure GWLB: [https://learn.microsoft.com/en-us/azure/load-balancer/gateway-overview]AWS GWLB Design/Packet Walk: [https://aws.amazon.com/blogs/networking-and-content-delivery/vpc-routing-enhancements-and-gwlb-deployment-patterns/]Layered VPC Security and Inspection (re:Invent 2022 presentation): [https://youtu.be/Ya4WFO9P0i8]John Savill Azure GWLB Deep Dive: [https://youtu.be/JLx7ZFzjdSs]TechTalk Reduce NGFW Deployments for AWS Public Facing Workloads: [https://www.youtube.com/watch?v=6cEnF7MdB0o]VPC Lattice: [https://youtu.be/fRjD1JI0H5w]What is a service mesh? [https://linkerd.io/what-is-a-service-mesh/]Istio: [https://istio.io]Check out the Fortnightly Cloud Networking NewsVisit our website and subscribe: https://www.cables2clouds.com/Follow us on Twitter: https://twitter.com/cables2cloudsFollow us on YouTube: https://www.youtube.com/@cables2clouds/Follow us on TikTok: https://www.tiktok.com/@cables2cloudsMerch Store: https://store.cables2clouds.com/Join the Discord Study group: https://artofneteng.com/iaatjArt of Network Engineering (AONE): https://artofnetworkengineering.com

Let's say you have a set of microservices running on a Kubernetes cluster. In the past, developers used to program features like service discovery, observability, who's allowed to talk to whom and other security related features directly into the application code. This slowed down the dev cycle and it made these microservices bigger and just The post Istio Ambient Mesh with Brian Gracely appeared first on Software Engineering Daily.

At an SaaS company like Intuit that has hundreds of services spread out across multiple products, maintaining development velocity at scale means baking some of the features that every service needs into the architecture of their systems. That's where a service mesh comes in. It automatically adds features like observability, traffic management, and security to every service in the network without adding any code. In this sponsored episode of the podcast, we talk with Anil Attuluri, principal software engineer, and Yasen Simeonov, senior product manager, both of Intuit, about how their engineering organization uses a service mesh to solve problems, letting their engineers stay focused on writing business logic. Along the way, we discuss how the service mesh keeps all the financial data secure, how it moves network traffic to where it needs to go, and the open source software they've written on top of the mesh. Episode notes:For those looking to get the same service mesh capabilities as Intuit, check out Istio, a Cloud Native Computing Foundation project. In order to provide a better security posture for their products, each business case operates on a discrete network. But much of the Istio service mesh needs to discover services across all products. Enter Admiral, their open-sourced solution. When Intuit deploys a new service version, they can progressively scale the amount of traffic that hits it instead of the old version using Argo Rollouts. It's better to find a bug in production on 1% of requests than 100%.If you want to learn more about what Intuit engineering is doing, check out their blog. Congrats to Great Question badge winner, HelpMeStackOverflowMyOnlyHope, for asking Detect whether input element is focused within ReactJS

Louis Bailleul is a Chief Enterprise Architect at PGS. After years of running highly-ranked super computers to process PGS' seismic data, Louis's team at PGS has lead a transition to Google Cloud. Listen in to learn about HPC in Google Cloud with GKE, and to explore using Kubernetes to do processing on vessels at sea! Do you have something cool to share? Some questions? Let us know: web: kubernetespodcast.com mail: kubernetespodcast@google.com twitter: @kubernetespod Chatter of the week Listen to the KubeCon NA 2022 recap episode News of the week Docker + Wasm Istio control plane vulnerability CVE-2022-39278 KubeFlow joins CNCF as an Incubating Project CNCF Backstage course CNCF Istio intro course Links from the interview PGS A picture of a PGS vessel PGS post from 2021 about their supercomputing rankings and transition to Google Cloud Top500 List Kubernetes Custom Resources (CRDs) Scaling Kubernetes to Thousands of CRDs Google Cloud Spot Instances Google Cloud Preemptible VM Instances Google Cloud - Manage capacity and quota KubeCon NA 2019: How the Department of Defense Moved to Kubernetes and Istio - Nicolas Chaillan Bare Metal K8s Clustering at Chick-fil-A Scale by Brian Chambers, Caleb Hurd, and Alex Crane

Today on Day Two Cloud we examine Istio Ambient Mesh, a new option for building service meshes in a microservices environment. Istio Ambient Mesh essentially brings the concept of a load balancer to a cluster of containers. Rather than run a sidecar proxy for each pod or container, you can run Ambient Mesh per node. Our guest and guide to this open source project is Christian Posta, Global Field CTO at Solo.io.

Today on Day Two Cloud we examine Istio Ambient Mesh, a new option for building service meshes in a microservices environment. Istio Ambient Mesh essentially brings the concept of a load balancer to a cluster of containers. Rather than run a sidecar proxy for each pod or container, you can run Ambient Mesh per node. Our guest and guide to this open source project is Christian Posta, Global Field CTO at Solo.io. The post Day Two Cloud 173: Istio Ambient Mesh Minimizes Sidecar Proxies appeared first on Packet Pushers.

Today on Day Two Cloud we examine Istio Ambient Mesh, a new option for building service meshes in a microservices environment. Istio Ambient Mesh essentially brings the concept of a load balancer to a cluster of containers. Rather than run a sidecar proxy for each pod or container, you can run Ambient Mesh per node. Our guest and guide to this open source project is Christian Posta, Global Field CTO at Solo.io.

Today on Day Two Cloud we examine Istio Ambient Mesh, a new option for building service meshes in a microservices environment. Istio Ambient Mesh essentially brings the concept of a load balancer to a cluster of containers. Rather than run a sidecar proxy for each pod or container, you can run Ambient Mesh per node. Our guest and guide to this open source project is Christian Posta, Global Field CTO at Solo.io. The post Day Two Cloud 173: Istio Ambient Mesh Minimizes Sidecar Proxies appeared first on Packet Pushers.

About ChenChen Goldberg is GM and Vice President of Engineering at Google Cloud, where she leads the Cloud Runtimes (CR) product area, helping customers deliver greater value, effortlessly. The CR  portfolio includes both Serverless and Kubernetes based platforms on Google Cloud, private cloud and other public clouds. Chen is a strong advocate for customer empathy, building products and solutions that matter. Chen has been core to Google Cloud's open core vision since she joined the company six years ago. During that time, she has led her team to focus on helping development teams increase their agility and modernize workloads. Prior to joining Google, Chen wore different hats in the tech industry including leadership positions in IT organizations, SI teams and SW product development, contributing to Chen's broad enterprise perspective. She enjoys mentoring IT talent both in and outside of Google. Chen lives in Mountain View, California, with her husband and three kids. Outside of work she enjoys hiking and baking.Links Referenced: Twitter: https://twitter.com/GoldbergChen LinkedIn: https://www.linkedin.com/in/goldbergchen/ TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Forget everything you know about SSH and try Tailscale. Imagine if you didn't need to manage PKI or rotate SSH keys every time someone leaves. That'd be pretty sweet, wouldn't it? With Tailscale SSH, you can do exactly that. Tailscale gives each server and user device a node key to connect to its VPN, and it uses the same node key to authorize and authenticate SSH.Basically you're SSHing the same way you manage access to your app. What's the benefit here? Built-in key rotation, permissions as code, connectivity between any two devices, reduce latency, and there's a lot more, but there's a time limit here. You can also ask users to reauthenticate for that extra bit of security. Sounds expensive?Nope, I wish it were. Tailscale is completely free for personal use on up to 20 devices. To learn more, visit snark.cloud/tailscale. Again, that's snark.cloud/tailscaleCorey: Welcome to Screaming in the Cloud, I'm Corey Quinn. When I get bored and the power goes out, I find myself staring at the ceiling, figuring out how best to pick fights with people on the internet about Kubernetes. Because, well, I'm basically sad and have a growing collection of personality issues. My guest today is probably one of the best people to have those arguments with. Chen Goldberg is the General Manager of Cloud Runtimes and VP of Engineering at Google Cloud. Chen, Thank you for joining me today.Chen: Thank you so much, Corey, for having me.Corey: So, Google has been doing a lot of very interesting things in the cloud, and the more astute listener will realize that interesting is not always necessarily a compliment. But from where I sit, I am deeply vested in the idea of a future where we do not have a cloud monoculture. As I've often said, I want, “What cloud should I build something on in five to ten years?” To be a hard question to answer, and not just because everything is terrible. I think that Google Cloud is absolutely a bright light in the cloud ecosystem and has been for a while, particularly with this emphasis around developer experience. All of that said, Google Cloud is sort of a big, unknowable place, at least from the outside. What is your area of responsibility? Where do you start? Where do you stop? In other words, what can I blame you for?Chen: Oh, you can blame me for a lot of things if you want to. I [laugh] might not agree with that, but that's—Corey: We strive for accuracy in these things, though.Chen: But that's fine. Well, first of all, I've joined Google about seven years ago to lead the Kubernetes and GKE team, and ever since, continued at the same area. So evolved, of course, Kubernetes, and Google Kubernetes Engine, and leading our hybrid and multi-cloud strategy as well with technologies like Anthos. And now I'm responsible for the entire container runtime, which includes Kubernetes and the serverless solutions.Corey: A while back, I, in fairly typical sarcastic form, wound up doing a whole inadvertent start of a meme where I joked about there being 17 ways to run containers on AWS. And then as that caught on, I wound up listing out 17 services you could use to do that. A few months went past and then I published a sequel of 17 more services you can use to run Kubernetes. And while that was admittedly tongue-in-cheek, it does lead to an interesting question that's ecosystem-wide. If I look at Google Cloud, I have Cloud Run, I have GKE, I have GCE if I want to do some work myself.It feels like more and more services are supporting Docker in a variety of different ways. How should customers and/or people like me—though, I am sort of a customer as well since I do pay you folks every month—how should we think about containers and services in which to run them?Chen: First of all, I think there's a lot of credit that needs to go to Docker that made containers approachable. And so, Google has been running containers forever. Everything within Google is running on containers, even our VMs, even our cloud is running on containers, but what Docker did was creating a packaging mechanism to improve developer velocity. So, that's on its own, it's great. And one of the things, by the way, that I love about Google Cloud approach to containers and Docker that yes, you can take your Docker container and run it anywhere.And it's actually really important to ensure what we call interoperability, or low barrier to entry to a new technology. So, I can take my Docker container, I can move it from one platform to another, and so on. So, that's just to start with on a containers. Between the different solutions, so first of all, I'm all about managed services. You are right, there are many ways to run a Kubernetes. I'm taking a lot of pride—Corey: The best way is always to have someone else run it for you. Problem solved. Great, the best kind of problems are always someone else's.Chen: Yes. And I'm taking a lot of pride of what our team is doing with Kubernetes. I mean, we've been working on that for so long. And it's something that you know, we've coined that term, I think back in 2016, so there is a success disaster, but there's also what we call sustainable success. So, thinking about how to set ourselves up for success and scale. Very proud of that service.Saying that, not everybody and not all your workloads you need the flexibility that Kubernetes gives you in all the ecosystem. So, if you start with containers your first time, you should start with Cloud Run. It's the easiest way to run your containers. That's one. If you are already in love with Kubernetes, we won't take it away from you. Start with GKE. Okay [laugh]? Go all-in. Okay, we are all in loving Kubernetes as well. But what my team and I are working on is to make sure that those will work really well together. And we actually see a lot of customers do that.Corey: I'd like to go back a little bit in history to the rise of Docker. I agree with you it was transformative, but containers had been around in various forms—depending upon how you want to define it—dating back to the '70s with logical partitions on mainframes. Well, is that a container? Is it not? Well, sort of. We'll assume yes for the sake of argument.The revelation that I found from Docker was the developer experience, start to finish. Suddenly, it was a couple commands and you were just working, where previously it had taken tremendous amounts of time and energy to get containers working in that same context. And I don't even know today whether or not the right way to contextualize containers is as sort of a lite version of a VM, as a packaging format, as a number of other things that you could reasonably call it. How do you think about containers?Chen: So, I'm going to do, first of all, a small [unintelligible 00:06:31]. I actually started my career as a system mainframe engineer—Corey: Hmm.Chen: And I will share that when you know, I've learned Kubernetes, I'm like, “Huh, we already have done all of that, in orchestration, in workload management on mainframe,” just to the side. The way I think about containers is as a—two things: one, it is a packaging of an application, but the other thing which is also critical is the decoupling between your application and the OS. So, having that kind of abstraction and allowing you to portable and move it between environments. So, those are the two things that are when I think about containers. And what technologies like Kubernetes and serverless gives on top of that is that manageability and making sure that we take care of everything else that is needed for you to run your application.Corey: I've been, how do I put this, getting some grief over the past few years, in the best ways possible, around a almost off-the-cuff prediction that I made, which was that in five years, which is now a lot closer to two, basically, nobody is going to care about Kubernetes. And I could have phrased that slightly more directly because people think I was trying to say, “Oh, Kubernetes is just hype. It's going to go away. Nobody's going to worry about it anymore.” And I think that is a wildly inaccurate prediction.My argument is that people are not going to have to think about it in the same way that they are today. Today, if I go out and want to go back to my days of running production services in anger—and by ‘anger,' I of course mean in production—then it would be difficult for me to find a role that did not at least touch upon Kubernetes. But people who can work with that technology effectively are in high demand and they tend to be expensive, not to mention then thinking about all of the intricacies and complexities that Kubernetes brings to the foreground, that is what doesn't feel sustainable to me. The idea that it's going to have to collapse down into something else is, by necessity, going to have to emerge. How are you seeing that play out? And also, feel free to disagree with the prediction. I am thrilled to wind up being told that I'm wrong it's how I learn the most.Chen: I don't know if I agree with the time horizon of when that will happen, but I will actually think it's a failure on us if that won't be the truth, that the majority of people will not need to know about Kubernetes and its internals. And you know, we keep saying that, like, hey, we need to make it more, like, boring, and easy, and I've just said like, “Hey, you should use managed.” And we have lots of customers that says that they're just using GKE and it scales on their behalf and they don't need to do anything for that and it's just like magic. But from a technology perspective, there is still a way to go until we can make that disappear.And there will be two things that will push us into that direction. One is—you mentioned that is as well—the talent shortage is real. All the customers that I speak with, even if they can find those great people that are experts, they're actually more interesting things for them to work on, okay? You don't need to take, like, all the people in your organization and put them on building the infrastructure. You don't care about that. You want to build innovation and promote your business.So, that's one. The second thing is that I do expect that the technology will continue to evolve and are managed solutions will be better and better. So hopefully, with these two things happening together, people will not care that what's under the hood is Kubernetes. Or maybe not even, right? I don't know exactly how things will evolve.Corey: From where I sit, what are the early criticisms I had about Docker, which I guess translates pretty well to Kubernetes, are that they solve a few extraordinarily painful problems. In the case of Docker, it was, “Well, it works on my machine,” as a grumpy sysadmin, the way I used to be, the only real response we had to that was, “Well. Time to backup your email, Skippy, because your laptop is going into production, then.” Now, you can effectively have a high-fidelity copy of production, basically anywhere, and we've solved the problem of making your Mac laptop look like a Linux server. Great, okay, awesome.With Kubernetes, it also feels, on some level, like it solves for very large-scale Google-type of problems where you want to run things across at least a certain point of scale. It feels like even today, it suffers from having an easy Hello World-style application to deploy on top of it. Using it for WordPress, or some other form of blogging software, for example, is stupendous overkill as far as the Hello World story tends to go. Increasingly as a result, it feels like it's great for the large-scale enterprise-y applications, but the getting started story of how do I have a service I could reasonably run in production? How do I contextualize that, in the world of Kubernetes? How do you respond to that type of perspective?Chen: We'll start with maybe a short story. I started my career in the Israeli army. I was head of the department and one of the lead technology units and I was responsible for building a PAS. In essence, it was 20-plus years ago, so we didn't really call it a PAS but that's what it was. And then at some point, it was amazing, developers were very productive, we got innovation again, again. And then there was some new innovation just at the beginning of web [laugh] at some point.And it was actually—so two things I've noticed back then. One, it was really hard to evolve the platform to allow new technologies and innovation, and second thing, from a developer perspective, it was like a black box. So, the developers team that people were—the other development teams couldn't really troubleshoot environment; they were not empowered to make decisions or [unintelligible 00:12:29] in the platform. And you know, when it was just started with Kubernetes—by the way, beginning, it only supported 100 nodes, and then 1000 nodes. Okay, it was actually not for scale; it actually solved those two problems, which I'm—this is where I spend most of my time.So, the first one, we don't want magic, okay? To be clear on, like, what's happening, I want to make sure that things are consistent and I can get the right observability. So, that's one. The second thing is that we invested so much in the extensibility an environment that it's, I wouldn't say it's easy, but it's doable to evolve Kubernetes. You can change the models, you can extend it you can—there is an ecosystem.And you know, when we were building it, I remember I used to tell my team, there won't be a Kubernetes 2.0. Which is for a developer, it's [laugh] frightening. But if you think about it and you prepare for that, you're like, “Huh. Okay, what does that mean with how I build my APIs? What does that mean of how we build a system?” So, that was one. The second thing I keep telling my team, “Please don't get too attached to your code because if it will still be there in 5, 10 years, we did something wrong.”And you can see areas within Kubernetes, again, all the extensions. I'm very proud of all the interfaces that we've built, but let's take networking. This keeps to evolve all the time on the API and the surface area that allows us to introduce new technologies. I love it. So, those are the two things that have nothing to do with scale, are unique to Kubernetes, and I think are very empowering, and are critical for the success.Corey: One thing that you said that resonates most deeply with me is the idea that you don't want there to be magic, where I just hand it to this thing and it runs it as if by magic. Because, again, we've all run things in anger in production, and what happens when the magic breaks? When you're sitting around scratching your head with no idea how it starts or how it stops, that is scary. I mean, I recently wound up re-implementing Google Cloud Distinguished Engineer Kelsey Hightower's “Kubernetes the Hard Way” because he gave a terrific tutorial that I ran through in about 45 minutes on top of Google Cloud. It's like, “All right, how do I make this harder?”And the answer is to do it on AWS, re-implement it there. And my experiment there can be found at kubernetesthemuchharderway.com because I have a vanity domain problem. And it taught me he an awful lot, but one of the challenges I had as I went through that process was, at one point, the nodes were not registering with the controller.And I ran out of time that day and turned everything off—because surprise bills are kind of what I spend my time worrying about—turn it on the next morning to continue and then it just worked. And that was sort of the spidey sense tingling moment of, “Okay, something wasn't working and now it is, and I don't understand why. But I just rebooted it and it started working.” Which is terrifying in the context of a production service. It was understandable—kind of—and I think that's the sort of thing that you understand a lot better, the more you work with it in production, but a counterargument to that is—and I've talked about it on this show before—for this podcast, I wind up having sponsors from time to time, who want to give me fairly complicated links to go check them out, so I have the snark.cloud URL redirector.That's running as a production service on top of Google Cloud Run. It took me half an hour to get that thing up and running; I haven't had to think about it since, aside from a three-second latency that was driving me nuts and turned out to be a sleep hidden in the code, which I can't really fault Google Cloud Run for so much as my crappy nonsense. But it just works. It's clearly running atop Kubernetes, but I don't have to think about it. That feels like the future. It feels like it's a glimpse of a world to come, we're just starting to dip our toes into. That, at least to me, feels like a lot more of the abstractions being collapsed into something easily understandable.Chen: [unintelligible 00:16:30], I'm happy you say that. When talking with customers and we're showing, like, you know, yes, they're all in Kubernetes and talking about Cloud Run and serverless, I feel there is that confidence level that they need to overcome. And that's why it's really important for us in Google Cloud is to make sure that you can mix and match. Because sometimes, you know, a big retail customer of ours, some of their teams, it's really important for them to use a Kubernetes-based platform because they have their workloads also running on-prem and they want to serve the same playbooks, for example, right? How do I address issues, how do I troubleshoot, and so on?So, that's one set of things. But some cloud only as simple as possible. So, can I use both of them and still have a similar developer experience, and so on? So, I do think that we'll see more of that in the coming years. And as the technology evolves, then we'll have more and more, of course, serverless solutions.By the way, it doesn't end there. Like, we see also, you know, databases and machine learning, and like, there are so many more managed services that are making things easy. And that's what excites me. I mean, that's what's awesome about what we're doing in cloud. We are building platforms that enable innovation.Corey: I think that there's an awful lot of power behind unlocking innovation from a customer perspective. The idea that I can use a cloud provider to wind up doing an experiment to build something in the course of an evening, and if it works, great, I can continue to scale up without having to replace, you know, the crappy Raspberry Pi-level hardware in my spare room with serious enterprise servers in a data center somewhere. The on-ramp and the capability and the lack of long-term commitments is absolutely magical. What I'm also seeing that is contributing to that is the de facto standard that's emerged of most things these days support Docker, for better or worse. There are many open-source tools that I see where, “Oh, how do I get this up and running?”“Well, you can go over the river and through the woods and way past grandmother's house to build this from source or run this Docker file.” I feel like that is the direction the rest of the world is going. And as much fun as it is to sit on the sidelines and snark, I'm finding a lot more capability stories emerging across the board. Does that resonate with what you're seeing, given that you are inherently working at very large scale, given the [laugh] nature of where you work?Chen: I do see that. And I actually want to double down on the open standards, which I think this is also something that is happening. At the beginning, we talked about I want it to be very hard when I choose the cloud provider. But innovation doesn't only come from cloud providers; there's a lot of companies and a lot of innovation happening that are building new technologies on top of those cloud providers, and I don't think this is going to stop. Innovation is going to come from many places, and it's going to be very exciting.And by the way, things are moving super fast in our space. So, the investment in open standard is critical for our industry. So, Docker is one example. Google is in [unintelligible 00:19:46] speaking, it's investing a lot in building those open standards. So, we have Docker, we have things like of course Kubernetes, but we are also investing in open standards of security, so we are working with other partners around [unintelligible 00:19:58], defining how you can secure the software supply chain, which is also critical for innovation. So, all of those things that reduce the barrier to entry is something that I'm personally passionate about.Corey: Scaling containers and scaling Kubernetes is hard, but a whole ‘nother level of difficulty is scaling humans. You've been at Google for, as you said, seven years and you did not start as a VP there. Getting promoted from Senior Director to VP at Google is a, shall we say, heavy lift. You also mentioned that you previously started with, I believe, it was a seven-person team at one point. How have you been able to do that? Because I can see a world in which, “Oh, we just write some code and we can scale the computers pretty easily,” I've never found a way to do that for people.Chen: So yes, I started actually—well not 7, but the team was 30 people [laugh]. And you can imagine how surprised I was when I joining Google Cloud with Kubernetes and GKE and it was a pretty small team, to the beginning of those days. But the team was already actually on the edge of burning out. You know, pings on Slack, the GitHub issues, there was so many things happening 24/7.And the thing was just doing everything. Everybody were doing everything. And one of the things I've done on my second month on the team—I did an off-site, right, all managers; that's what we do; we do off-sites—and I brought the team in to talk about—the leadership team—to talk about our team values. And in the beginning, they were a little bit pissed, I would say, “Okay, Chen. What's going on? You're wasting two days of our lives to talk about those things. Why we are not doing other things?”And I was like, “You know guys, this is really important. Let's talk about what's important for us.” It was an amazing it worked. By the way, that work is still the foundation of the culture in the team. We talked about the three values that we care about and how that will look like.And the reason it's important is that when you scale teams, the key thing is actually to scale decision-making. So, how do you scale decision-making? I think there are two things there. One is what you're trying to achieve. So, people should know and understand the vision and know where we want to get to.But the second thing is, how do we work? What's important for us? How do we prioritize? How do we make trade-offs? And when you have both the what we're trying to do and the how, you build that team culture. And when you have that, I find that you're set up more for success for scaling the team.Because then the storyteller is not just the leader or the manager. The entire team is a storyteller of how things are working in this team, how do we work, what you're trying to achieve, and so on. So, that's something that had been a critical. So, that's just, you know, from methodology of how I think it's the right thing to scale teams. Specifically, with a Kubernetes, there were more issues that we needed to work on.For example, building or [recoding 00:23:05] different functions. It cannot be just engineering doing everything. So, hiring the first product managers and information engineers and marketing people, oh my God. Yes, you have to have marketing people because there are so many events. And so, that was one thing, just you know, from people and skills.And the second thing is that it was an open-source project and a product, but what I was personally doing, I was—with the team—is bringing some product engineering practices into the open-source. So, can we say, for example, that we are going to focus on user experience this next release? And we're not going to do all the rest. And I remember, my team was like worried about, like, “Hey, what about that, and what about this, and we have—” you know, they were juggling everything together. And I remember telling them, “Imagine that everything is on the floor. All the balls are on the floor. I know they're on the floor, you know they're on the floor. It's okay. Let's just make sure that every time we pick something up, it never falls again.” And that idea is a principle that then evolved to ‘No Heroics,' and it evolved to ‘Sustainable Success.' But building things towards sustainable success is a principle which has been very helpful for us.Corey: This episode is sponsored in part by our friend at Uptycs. Attackers don't think in silos, so why would you have siloed solutions protecting cloud, containers, and laptops distinctly? Meet Uptycs - the first unified solution that prioritizes risk across your modern attack surface—all from a single platform, UI, and data model. Stop by booth 3352 at AWS re:Invent in Las Vegas to see for yourself and visit uptycs.com. That's U-P-T-Y-C-S.com. My thanks to them for sponsoring my ridiculous nonsense.Corey: When I take a look back, it's very odd to me to see the current reality that is Google, where you're talking about empathy, and the No Heroics, and the rest of that is not the reputation that Google enjoyed back when a lot of this stuff got started. It was always oh, engineers should be extraordinarily bright and gifted, and therefore it felt at the time like our customers should be as well. There was almost an arrogance built into, well, if you wrote your code more like Google will, then maybe your code wouldn't be so terrible in the cloud. And somewhat cynically I thought for a while that oh Kubernetes is Google's attempt to wind up making the rest of the world write software in a way that's more Google-y. I don't think that observation has aged very well. I think it's solved a tremendous number of problems for folks.But the complexity has absolutely been high throughout most of Kubernetes life. I would argue, on some level, that it feels like it's become successful almost in spite of that, rather than because of it. But I'm curious to get your take. Why do you believe that Kubernetes has been as successful as it clearly has?Chen: [unintelligible 00:25:34] two things. One about empathy. So yes, Google engineers are brilliant and are amazing and all great. And our customers are amazing, and brilliant, as well. And going back to the point before is, everyone has their job and where they need to be successful and we, as you say, we need to make things simpler and enable innovation. And our customers are driving innovation on top of our platform.So, that's the way I think about it. And yes, it's not as simple as it can be—probably—yet, but in studying the early days of Kubernetes, we have been investing a lot in what we call empathy, and the customer empathy workshop, for example. So, I partnered with Kelsey Hightower—and you mentioned yourself trying to start a cluster. The first time we did a workshop with my entire team, so then it was like 50 people [laugh], their task was to spin off a cluster without using any scripts that we had internally.And unfortunately, not many folks succeeded in this task. And out of that came the—what you you call it—a OKR, which was our goal for that quarter, is that you are able to spin off a cluster in three commands and troubleshoot if something goes wrong. Okay, that came out of that workshop. So, I do think that there is a lot of foundation on that empathetic engineering and the open-source of the community helped our Google teams to be more empathetic and understand what are the different use cases that they are trying to solve.And that actually bring me to why I think Kubernetes is so successful. People might be surprised, but the amount of investment we're making on orchestration or placement of containers within Kubernetes is actually pretty small. And it's been very small for the last seven years. Where do we invest time? One is, as I mentioned before, is on the what we call the API machinery.So, Kubernetes has introduced a way that is really suitable for a cloud-native technologies, the idea of reconciliation loop, meaning that the way Kubernetes is—Kubernetes is, like, a powerful automation machine, which can automate, of course, workload placement, but can automate other things. Think about it as a way of the Kubernetes API machinery is observing what is the current state, comparing it to the desired state, and working towards it. Think about, like, a thermostat, which is a different automation versus the ‘if this, then that,' where you need to anticipate different events. So, this idea about the API machinery and the way that you can extend it made it possible for different teams to use that mechanism to automate other things in that space.So, that has been one very powerful mechanism of Kubernetes. And that enabled all of innovation, even if you think about things like Istio, as an example, that's how it started, by leveraging that kind of mechanism to separate storage and so on. So, there are a lot of operators, the way people are managing their databases, or stateful workloads on top of Kubernetes, they're extending this mechanism. So, that's one thing that I think is key and built that ecosystem. The second thing, I am very proud of the community of Kubernetes.Corey: Oh, it's a phenomenal community success story.Chen: It's not easy to build a community, definitely not in open-source. I feel that the idea of values, you know, that I was talking about within my team was actually a big deal for us as we were building the community: how we treat each other, how do we help people start? You know, and we were talking before, like, am I going to talk about DEI and inclusivity, and so on. One of the things that I love about Kubernetes is that it's a new technology. There is actually—[unintelligible 00:29:39] no, even today, there is no one with ten years experience in Kubernetes. And if anyone says they have that, then they are lying.Corey: Time machine. Yes.Chen: That creates an opportunity for a lot of people to become experts in this technology. And by having it in open-source and making everything available, you can actually do it from your living room sofa. That excites me, you know, the idea that you can become an expert in this new technology and you can get involved, and you'll get people that will mentor you and help you through your first PR. And there are some roles within the community that you can start, you know, dipping your toes in the water. It's exciting. So, that makes me really happy, and I know that this community has changed the trajectory of many people's careers, which I love.Corey: I think that's probably one of the most impressive things that it's done. One last question I have for you is that we've talked a fair bit about the history and how we see it progressing through the view toward the somewhat recent past. What do you see coming in the future? What does the future of Kubernetes look like to you?Chen: Continue to be more and more boring. There is the promise of hybrid and multi-cloud, for example, is only possible by technologies like Kubernetes. So, I do think that, as a technology, it will continue to be important by ensuring portability and interoperability of workloads. I see a lot of edge use cases. If you think about it, it's like just lagging a bit around, like, innovation that we've seen in the cloud, can we bring that innovation to the edge, this will require more development within Kubernetes community as well.And that's really actually excites me. I think there's a lot of things that we're going to see there. And by the way, you've seen it also in KubeCon. I mean, there were some announcements in that space. In Google Cloud, we just announced before, like, with customers like Wendy's and Rite Aid as well. So, taking advantage of this technology to allow innovation everywhere.But beyond that, my hope is that we'll continue and hide the complexity. And our challenge will be to not make it a black box. Because that will be, in my opinion, a failure pattern, doesn't help those kinds of platforms. So, that will be the challenge. Can we scope the project, ensure that we have the right observability, and from a use case perspective, I do think edge is super interesting.Corey: I would agree. There are a lot of workloads out there that are simply never going to be hosted in the cloud provider region, for a variety of reasons of varying validity, but it is the truth. I think that the focus on addressing customers where they are has been an emerging best practice for cloud providers and I'm thrilled to see Google leading the charge on that.Chen: Yeah. And you just reminded me, the other thing that we see also more and more is definitely AI and ML workloads running on Kubernetes, which is part of that, right? So, Google Cloud is investing a lot in making an AI/ML easy. And I don't know if many people know, but, like, even Vertex AI, our own platform, is running on GKE. So, that's part of seeing how do we make sure that platform is suitable for these kinds of workloads and really help customers do the heavy lifting.So, that's another set of workloads that are very relevant at the edge. And one of our customers—MLB, for example—two things are interesting there. The first one, I think a lot of people sometimes say, “Okay, I'm going to move to the cloud and I want to know everything right now, how that will evolve.” And one of the things that's been really exciting with working with MLB for the last four years is the journey and the iterations. So, they started somewhat, like, at one phase and then they saw what's possible, and then moved to the next one, and so on. So, that's one. The other thing is that, really, they have so much ML running at the stadium with Google Cloud technology, which is very exciting.Corey: I'm looking forward to seeing how this continues to evolve and progress, particularly in light of the recent correction we're seeing in the market where a lot of hype-driven ideas are being stress test, maybe not in the way we might have hoped that they would, but it'll be really interesting to see what shakes out as far as things that deliver business value and are clear wins for customers versus a lot of the speculative stories that we've been hearing for a while now. Maybe I'm totally wrong on this. And this is going to be a temporary bump in the road, and we'll see no abatement in the ongoing excitement around so many of these emerging technologies, but I'm curious to see how it plays out. But that's the beautiful part about getting to be a pundit—or whatever it is people call me these days that's at least polite enough to say on a podcast—is that when I'm right, people think I'm a visionary, and when I'm wrong, people don't generally hold that against you. It seems like futurist is the easiest job in the world because if you predict and get it wrong, no one remembers. Predict and get it right, you look like a genius.Chen: So, first of all, I'm optimistic. So usually, my predictions are positive. I will say that, you know, what we are seeing, also what I'm hearing from our customers, technology is not for the sake of technology. Actually, nobody cares [laugh]. Even today.Okay, so nothing needs to change for, like, nobody would c—even today, nobody cares about Kubernetes. They need to care, unfortunately, but what I'm hearing from our customers is, “How do we create new experiences? How we make things easy?” Talent shortage is not just with tech people. It's also with people working in the warehouse or working in the store.Can we use technology to help inventory management? There's so many amazing things. So, when there is a real business opportunity, things are so much simpler. People have the right incentives to make it work. Because one thing we didn't talk about—right, we talked about all these new technologies and we talked about scaling team and so on—a lot of time, the challenge is not the technology.A lot of time, the challenge is the process. A lot of time, the challenge is the skills, is the culture, there's so many things. But when you have something—going back to what I said before—how you unite teams, when there's something a clear goal, a clear vision that everybody's excited about, they will make it work. So, I think this is where having a purpose for the innovation is critical for any successful project.Corey: I think and I hope that you're right. I really want to thank you for spending as much time with me as you have. If people want to learn more, where's the best place for them to find you?Chen: So, first of all, on Twitter. I'm there or on LinkedIn. I will say that I'm happy to connect with folks. Generally speaking, at some point in my career, I recognized that I have a voice that can help people, and I've experienced that can also help people build their careers. I'm happy to share that and [unintelligible 00:36:54] folks both in the company and outside of it.Corey: I think that's one of the obligations on a lot of us, once we wanted to get into a certain position or careers to send the ladder back down, for lack of a better term. It's I've never appreciated the perspective, “Well, screw everyone else. I got mine.” The whole point the next generation should have it easier than we did.Chen: Yeah, definitely.Corey: Chen Goldberg, General Manager of Cloud Runtimes and VP of Engineering at Google. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, along with an angry rant of a comment talking about how LPARs on mainframes are absolutely not containers, making sure it's at least far too big to fit in a reasonably-sized Docker container.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

In this episode we bring you with us to KubeCon NA 2022 in Detroit, Michigan. We interviewed 15 attendees from various backgrounds and learned some cool insights. Featuring: Mo Khan, Software Engineer, Microsoft. Katrina Verey, Senior Staff Production Engineer, Shopify. Aishwarya Harpalem, Student, Rutgers University. Jeffery Sica, Principal Developer Experience Engineer, CNCF. Kirsten Schumy, Software Engineer, AWS. Jean-Paul Robinson, HPC Architect, University of Alabama at Birmingham. Madhav Jivrajani, Software Engineer, Vmware. Leigh Capili, Developer Advocate, Vmware Tanzu. Nim Jayawardena, Developer Programs Engineer, Google. Charlie Yu, Developer Programs Engineer, Google. Ahrar Monsur, Developer Programs Engineer, Google. Mickey Boxell, Product Manager, Oracle. Eddie Zaneski, Software Engineer, Chainuard. Andy Piggott, Chief Product Officer, Section. Logan Smith, Director of Business Development, GrafanaLabs. Brian Dorsey, Developer Advocate, Google - Shoutout for recommending the microphones for interviews. Do you have something cool to share? Some questions? Let us know: web: kubernetespodcast.com mail: kubernetespodcast@google.com twitter: @kubernetespod News of the week CrowdStrike cryptojacking finding Skaffold v2 Generally Available GKE Security Posture Dashboard Blog Video Cdk8s+ from AWS Blog Project page CNCF Sandbox project application information Istio becomes a CNCF Incubating project Cert-manager becomes a CNCF Incubating project Cisco OpenClarity Kube-router bug Google Cloud Next Wrap-Up Microsoft Ignite highlights blog Cloud Native SecurityCon Linux Foundation partnership with Razom for Ukraine Links from the interview Kubernetes SIG Auth Kubernetes SIG API Machinery FluxCD Online Boutique Sample App Kubernetes SIG-CLI Cloud Native 101: Motor City Edition by Bob Killen and Jeffrey Sica Consumers to Contributors by Brendan O'Leary Kubernet-Bees: How Bees Solve the Problems of Distributed Systems SchedMD Slurm Kube-bind Contribute to etcd! Cloud Native WASM Day Cloud Native SecurityCon Backstage (Incubating CNCF Project) eBPF Cilium (Incubating CNCF Project) Acorn Labs Vulcan Mind-Meld (Star Trek) Kids' Day at KubeCon NA 2022

After four and a half years hosting this podcast (and almost 9 years at Google) Craig Box is moving on from the latter, which unfortunately means leaving the former. But the show must go on. In this episode Craig introduces new hosts Abdel Sghiouar and Kaslin Fields. We take a small look forward, and then a big look back. Do you have something cool to share? Some questions? Let us know: web: kubernetespodcast.com mail: kubernetespodcast@google.com twitter: @kubernetespod Links from the show Adam’s last episode Abdelfettah Sghiouar Devoxx MA Cloud Careers Podcast You probably DON’T need a Service Mesh Kaslin Fields Containers as cookies Biscuits and gravy Contributor comms First-gen stickers Second-gen stickers Episode 60, with Mark Shuttleworth Episode 15, with Dan Ciruli and Jasmine Jaksic Dan on sticker duty Episode 30, with Joe Zou A rare team photo Music and musicians Kaossilator Episode 191, with DJ Fresh Episode 127, with David Pait Episode 83, with Guinevere Saenger Episode 120, with Melanie Cebula Episode 121, with Ed Huang Double guest trivia: Episodes 1 and 100 with Paris Pittman Episodes 62 and 180 with Ricardo Rocha (on a technicality) The Adam face Corey Quinn: separated at birth? One of many booth meetups Follow Craig Box on Twitter Follow Adam Glick on LinkedIn

Dan Stein is an engineering manager at General Bioinformatics. Dan Stein is also DJ Fresh, a multi-million selling artist with two UK number one records. Learn about the surprising overlap between these two careers. Do you have something cool to share? Some questions? Let us know: web: kubernetespodcast.com mail: kubernetespodcast@google.com twitter: @kubernetespod and @craigbox Chatter of the week Trevor Noah stepping down as host of Daily Show Follow @craigbox to learn what’s next News of the week Google Cloud adds GPU support to Autopilot Pricing CVE-2021-36782 in Rancher State of DevOps Report for 2022 Congratulations to the 27 Summer LFX Program CNCF interns Reviewing the 2019 Kubernetes security audit Links from the interview DJ Fresh Atari 800 and Atari ST Pong Atari BASIC Commodore Amiga OctaMED Fatboy Slim and the Atari ST Dogs on Acid music forum Taylor Hawkins Tribute Concerts Abolishing the high tax rate in the UK, or not Breakbeat Kaos Hold Your Colour by Pendulum Kryptonite by DJ Fresh Gold Dust Subsequent hits: Louder Hot Right Now Kyma (sound design language) and Max/MSP We Got Coders General Bioinformatics NGS gene sequencing Ensembl Hasura GraphQL Playground NCBI - National Center for Biotechnology Information Max Martin How Music Works by John Powell Learning: Treehouse Udemy 3Blue1Brown Codeacademy DJ Fresh’s new single, Higher DJ Fresh on Facebook Dan Stein on Twitter

Betty Junod, VP of Product Marketing at VMware Tanzu, kindly took up Craig’s challenge to explain the various parts of the Tanzu ecosystem, and how the traditional IT buyer and the modern cloud native really aren’t that different. Do you have something cool to share? Some questions? Let us know: web: kubernetespodcast.com mail: kubernetespodcast@google.com twitter: @kubernetespod and @craigbox Chatter of the week NASA DART mission Deep Impact Armageddon Apparent retrograde motion Planets beyond Neptune News of the week Istio sails into the CNCF SPIFFE and SPIRE graduate Episode 45, with Andrew Jessup Brigade archived Sysdig 2022 Cloud Native threat report The nice TeamTNT Episode 188, with Kateryna Ivashchenko Episode 169, with Anna Belak Chainguard introduces Wolfi workerd, from Cloudflare Introducing Palaemon Custom org policy for GKE in preview Leveraging Kubernetes for an elastic platform at Blablacar by Sebastien Doido Links from the interview VMware History Docker Solo.io VMware Tanzu introduction blog VMware acquires Heptio VMware acquires Pivotal Tanzu Mission Control Tanzu for Kubernetes Operations Tanzu Application Platform Tanzu Kubernetes Grid Bring your own host to TKG Project Pacific introduction TKG 2.0 VMware Aria Operations for Applications Tanzu Application Service Cloud Foundry Open source projects: Velero Antrea Carvel Cartographer Michigan cider Detroit-style pizza Betty Junod on Twitter

On today's Day Two Cloud we dive into Istio with Kevin Davin, a senior back end engineer who works deeply with Istio. We discuss Istio's promises, balancing its complexity with the capabilities it enables, understanding when and when not to use it, and more.

On today's Day Two Cloud we dive into Istio with Kevin Davin, a senior back end engineer who works deeply with Istio. We discuss Istio's promises, balancing its complexity with the capabilities it enables, understanding when and when not to use it, and more. The post Day Two Cloud 165: Does Your Infrastructure Need Istio? appeared first on Packet Pushers.