Ship It! DevOps, Infra, Cloud Native

This is our 9th Kaizen with Adam & Jerod. We start today's conversation with the most important thing: embracing change. For Gerhard, this means putting Ship It on hold after this episode. It also means making more time to experiment, maybe try a few of those small bets that we recently talked about with Daniel. Kaizen will continue, we are thinking on the Changelog. Stick around to hear the rest.

Tim McNamara is known as New Zealand's Rust guy. He is the author of Rust in Action, and also a Senior Software Engineer at AWS, where he helps other builders with all things Rust. The main reason why Gerhard is intrigued by Rust is the incredible resource frugality. Fewer CPUs means less energy used, which is good for the planet, and good for the monthly bill. This becomes most noticeable at Amazon's scale, when S3, Lambda, CloudFront and other services start adding Rust components.

In our ops & infra world, we learn to optimise for redundancy, for mean time to recovery and for graceful degradation. We instinctively recognise single points of failure, and try to mitigate the risks associated with them. For some years now, Daniel Vassallo has been doing the same, but in the context of life & work. Daniel talks about the role of randomness, about learning from small wins & about optimising for a lifestyle that matches your true preferences,. Apparently, ideas too should be treated like cattle, not pets.

Last September, at the

Lars is big on Elixir. Think apps that scale really well, tend to be monolithic, and have one of the most mature deployment models: self-contained releases & built-in hot code reloading. In episode 7, Gerhard talked to Lars about “Why Kubernetes”. There is a follow-up YouTube stream that showed how to automate deploys for an Elixir app using K3s & ArgoCD. More than a year later, how does Lars think about running applications in production? What does simple & straightforward mean to him? Gerhard's favourite: what is “human scale deployments”?

Marcos Nils has been into platform engineering for the best part of the last decade. He helped architect & build developer platforms using VMs & OpenStack, containers with Docker, and even Kubernetes. He did this at startups with 10 people, as well as large, publicly traded companies with 1000+ software engineers. Today we talk with Marcos about the hard parts of platform engineering.

Welcome to 2023! A new year is the perfect time to start with a fresh perspective. Given a few bare metal hosts with fast, local storage, how would you run your workloads on them? Would you cluster them for redundancy? What operating system would you choose? Steve Francis, CEO at Sidero Labs and Andrew Rynhard, CTO at Sidero Labs join us today to talk about running Talos Linux on bare metal.

Eight months ago, in

Narayanan Raghavan leads the global SRE organization that runs Red Hat managed cloud services including OpenShift Dedicated, Azure Red Hat Openshift, Red Hat OpenShift Service on AWS, and Red Hat OpenShift Data Science among others across the three major cloud providers: AWS, GCP & Azure. We start with a high-level discussion about DevOps, SRE & platform engineering, and then we dig into SRE specifics, including what it takes to safely roll out updates across many tens of thousands of OpenShift clusters.

In today's episode, we have the pleasure of two guests: Whitney Lee, Staff Technical Advocate at VMware, the one behind the ⚡️ Enlightning episodes, and Mauricio Salatino, which you already know from

For our last 2022 Kaizen episode, we went all out:

In your company, who designs the end-to-end developer experience? From design to implementation, what is the developer experience that you actually ship? Even though the average developer wastes almost half of their working hours because of bad DX, many of us don't even know what that means, or how to improve it. Kenneth Auchenberg is working at Stripe, building economic infrastructure for the internet. Gerhard found his perspective on Developer Experience Infrastructure (DXI) refreshingly simple, as well as very useful.

In today's episode we have the pleasure of Audun Fauchald Strand, Principal Software Engineer at NAV.no, Norway's Labour & Welfare Administration. We will be talking about NAIS.io, the application platform that runs on-prem, as well as on the public cloud. Imagine hundreds of developers shipping on an average day 300 changes into a system which processes $100,000,000 worth of transactions on a quiet week. If you think this is hard, consider the context: a government institution which must comply with all laws & regulations.

15 years ago, Gerhard discovered magic in the form of Ruby on Rails. It was intuitive and it just worked. That is the context in which Gerhard fell in love with infrastructure and operations. Today, for special episode 77, we start at Seven Shipping Principles, and, in the true spirit of Ship It, we'll see what happens next. Our guest is David Heinemeier Hansson, creator of Ruby on Rails, co-founder of Basecamp & HEY, and a lot more - check out dhh.dk.

In today's episode, we talk about distroless, ko, apko, melange, musl and glibc. The context is Wolfi OS, a community Linux OS designed for the container and cloud-native era. If you are looking for the lightest possible container base image with 0 CVEs and both glibc and musl support, Wolfi OS & the related chainguard-images are worth checking out. Ariadne Conill is an Alpine Linux TSC member & Software Engineer at Chainguard.

Few genuinely need a multi-cloud setup. There is plenty of advice out there which mostly boils down to don't do it, you will be worse off. Vex.dev is a startup that provides APIs for video and audio streaming. The hard part is real-time combined with massive scale - think hundreds of thousands of concurrent connections. They achieve this by using a combination of Fly.io, AWS and GCP. Jason Carter, founder of Vex Communications, is joining us today to talk about the multi-cloud setup that vex.dev runs.

I don't think that you can imagine just how excited Gerhard was to find out that Audi, his favourite car company, has a Kubernetes competence centre. We have Sebastian Kister joining us today to tell us why people, followed by tech make the process. The right thing to focus on is the genuine smiles that people give in response to something we do or say. That is an important SLI & SLO for reducing friction between silos. How does this impact the flow of artefacts into production systems that design & build cars?

Matias Pan is a Staff Software Engineer at Lemon Cash, a crypto startup based in Argentina. Lemon infrastructure runs digital wallets & physical cards, which technically makes them a bank. How does Matias & his team think about enabling developers get code from their workstations into production? Remember, we are talking about a bank - a bad deploy is a big deal. And when a bad database migration goes out, what happens then?

One of our listeners, Andrew Welker, suggested that we talk about Klustered, so a few hours before David Flanagan was about to do his workshop at Container Days, we recorded this episode. We talked about all the weird and wonderful Kubernetes debugging sessions on Klustered, a YouTube playlist with 43 videos and counting. We then talked about Rawkode Academy, and we finished with conferences. Good thing we did, because David almost forgot about KubeHuddle, the conference that he is co-organising next week. Gerhard is looking forward to talking at it! No, seriously, check it out at kubehuddle.com.

Dave Farley, co-author of Continuous Delivery, is back to talk about his latest book, Modern Software Engineering, a Top 3 Software Engineering best seller on Amazon UK this September. Shipping good software starts with you giving yourself permission to do a good job. It continues with a healthy curiosity, admitting that you don't know, and running many experiments, safely, without blowing everything up. And then there is scope creep…

In today's Kaizen episode, we talk about shipping Adam's Christmas present: chapter support for all Changelog episodes that we now publish. This feature was hard because there are many subtle differences in how the ID3 spec is implemented. Of course, once the PR shipped, there were other issues to solve, including an upgrade the world kind of scenario. Since Lars Wikman did all the heavy ID3 lifting, he is here with us too.

Maybe it's the Californian sun. Or perhaps it's the time spent at Disney Studios, the home of the best stories. One thing is for sure: Taylor Dolezal is one of the happiest cloud native people that Gerhard knows. As a former Lead SRE for Disney Studios, Taylor has significant hands-on experience running cloud native technologies in a large company. After a few years as a HashiCorp Developer Advocate, Taylor is now Head of End User Ecosystem at CNCF. In his current role, he is helping enable cloud native success for end-users like Boeing, Mercedes Benz & many others.

Most of you already know what it's like to work in a startup or a small company. A few of you have been asking us for conversations with engineers that work for big companies, the kind that run everything from big title games to banking, and even critical national infrastructure. In today's episode, we talk to Ganeshkumar, a Software Engineer in the Azure Kubernetes Service team, who works on Node Lifecycle and Kubernetes Versioning, and Brendan, Kubernetes project co-founder and engineering Corporate Vice President of Microsoft Azure OSS and Cloud-native Compute. We talk about what it's like to work for Microsoft, how mentoring works in practice, and what Kubernetes, Omega, & Borg have to do with it all.

A few weeks ago, Jerod spoke with Liz Rice about the power of eBPF on The Changelog. Today, we have the pleasure of both Liz Rice, Chief Open Source Office at Isovalent & Thomas Graf, CTO & co-founder at Isovalent, the creators of Cilium. Around 2014, Facebook achieved a 10x performance improvement by replacing their traditional load balancers with eBPF. In 2017, every single packet that went to Facebook was processed by eBPF. Nowadays, every Android phone is using it. Truth be told, if it's network-related and it matters, eBPF is most likely a part of it.

Why are the right values important for a company that changed the way the world builds software? How does pair programming help scale & maintain the company culture? What is it like to grow a company to 3000 employees over 30 years? Today we have the privilege of Rob Mee, former CEO of Pivotal, the real home of Cloud Foundry and Concourse CI. Rob is now the CEO of Geometer.io, an incubator where Elixir is behind many great ideas executed well, including the US COVID response programme.

Tammer Saleh, founder of Super Orbital, a tiny team of exceptional Kubernetes engineers and teachers, is joining us today to talk about what is cool in the Cloud Native world. Yes, it's the same Tammer that we had the pleasure of on shipit.show/31 - Is Kubernetes a platform? In today's episode, we also cover two great blog posts: Zero to GitOps: Terraform and the AWS EKS Blueprints project by Sean Kane Hunting Down an Intermittent Failure in Cilium by James McShane We wrap up with ✨ The Cool Wall of Cloud Native ✨

Our today's guest spent 4 days building a feature for his side project so that we could ship it together on Ship It!, while recording. The feature is called rave mode, and the context is Bass, an interpreted functional scripting language written in Go, riffing on the ideas of Kernel & Clojure. When the local build runs, you can now press r to synchronise the beats of your currently playing Spotify track with the build output. For a demo, see bass v0.9.0 release. Please welcome Alex Suraci, a.k.a. vito, the creator of Concourse CI and Bass. This episode is dedicated to the late John Shutt, the creator of Kernel. Your ideas continue in Bass. Thank you for getting them out into the world.

We know that many of you listen to this podcast while running

Gerhard's transition to a senior engineer started 10 years ago, when he embraced the vim mindset, functional core & imperative shell, and was inspired to seek simplicity in his code & infrastructure. Most of it can be traced back to one person: Gary Bernhardt, the creator of Execute Program, Destroy all Software and the now famous Wat idea. Few stick around long enough to understand the long-term impact of their decisions on production systems. Even fewer are able to talk about them as well as Gary does.

Today we talk with two lovely folks from Transistor.fm: Jason Pearl, Senior Software Developer & Jon Buda, co-founder. Gerhard was curious to find out about their setup & how did it change with the launch of the new podcast website builder. After all, you have been hearing us talk about our setup for years, so it was high-time to challenge some assumptions and learn how another team is solving similar problems. TL;DL: keeping it simple is at the root of smooth operations & stable systems.

In our 6th Kaizen, we talk with Jerod about all the things that we cleaned up after migrating changelog.com from a managed Kubernetes to Fly.io. We deleted the K8s cluster and moved wildcard cert management to Fastly & all our vanity domain certs to Fly.io. We migrated the Docker Engine that our GitHub Actions is using - PR #416 has all the details. We did a few other things in preparation for our secrets plan. Thank you Maikel Vlasman, James Harr, Adrian Mester, Omri Gabay & Owen Valentine for kicking it off in our Slack #shipit channel. Gerhard's favourite improvement: the new shipit.show domain.

Ben Johnson, the creator of Litestream, joined Fly.io a few weeks after we migrated changelog.com - episode 50 has all the details. That was pure coincidence. What was not a coincidence, is Gerhard jumping at the opportunity to talk to Ben about Postgres vs SQLite with Litestream. The prospect of running a cluster of our app instances spread across all regions, with local SQLite & Litestream replication, is mind boggling. Let's find out from Ben what will it take to get there. Thanks Kürt for kicking off this dream.

Rob Barnes (a.k.a. Devops Rob) and Rosemary Wang (author of Infrastructure as Code - Patterns & Practices) are joining us today to talk about infrastructure secrets. What do Rosemary and Rob think about committing encrypted secrets into a repository? How do they suggest that we improve on storing secrets in LastPass? And if we were to choose HashiCorp Vault, what do we need to know? Thank you Thomas Eckert for the intro. Thank you Nabeel Sulieman (ep. 46) & Kelsey Hightower (ep. 44) for your gentle nudges towards improving our infra secrets management.

Today we are talking with Frederic Branczyk, founder of Polar Signals & Prometheus maintainer. You may remember Frederic from episode 33 when we introduced Parca.dev. This time, we talk about a database built for observability: FrostDB, formerly known as ArcticDB. eBPF generates a lot of high cardinality data, which requires a new approach to writing, persisting & then reading back this state. TL;DR FrostDB is sub zero cool & well worthy of its name.

Today we are talking with Maikel Vlasman, technical lead for a large Dutch machine construction company, and a cloud engineer by heart. We cover self-updating GitLab & ArgoCD, Maikel's thinking behind dev environment setup and a Kubernetes workshop that he is preparing for his team. The goal is to function as a true DevOps team with shared responsibilities. This conversation started as a thread in our community Slack - link in the show notes. Thank you Maikel for being a long-time Changelog listener and for reaching out to us - we enjoyed telling this story.

Today we are talking how to optimise sociotechnical systems with Ben Ford, founder & CEO of Mission Control. The correct order is: people, process & technology. The tools are important, and we talk about specific ones in the second half of this episode, but there are rules and principles that govern how people interact, and we need to start there.

This is the post-KubeCon CloudNativeCon EU 2022 week. Gerhard is talking to Matt Moore, founder & CTO of Chainguard about all things Knative and Sigstore. The most important topic is swag, because none has better stickers than Chainguard. The other topic is the equivalent of Let's Encrypt for securing software.

Today we are at KubeCon CloudNativeCon EU 2022, talking to Adolfo García Veytia about securing Kubernetes releases. Adolfo is a Staff Software Engineer at Chainguard, and one of the technical leads for SIG release, meaning that he helps ship Kubernetes. You most likely know him as Puerco, and have seen first-hand his passion for securing software via SBOMs, cosign and SLSA. Puerco's love for bikes and Chainguard are a great match

Today we talk to Priyanka Sharma (E.D. at the Cloud Native Computing Foundation) about all things KubeCon Europe 2022. We start with Gerhard's favourite subject - Priyanka's Happy Hour - and then we switch focus to the conference. For many, this will be the first in-person KubeCon since 2019. As for Gerhard, he is not sure that he remember how airports work. If he succeeds, he looks forward to meeting some of you in Valencia. If not, send help.

Today we talk to Mark Ericksen about all the things that we could be doing on the new platform - this is a follow-up to episode 50. Mark specialises in Elixir, he hosts the Thinking Elixir podcast, and he also helps make Fly.io the best place to run Phoenix apps, such as changelog.com. In the interest of holding our new platform right, we thought that it would be a great idea to talk to someone that does this all day, every day, for many years now. We touch up on how to run database migrations safely, and how to upgrade our application config to the latest Phoenix version. We also talked about some of the more advanced platform features that we may want to start leveraging, like the multi-region PostgreSQL.

This is our 5th Kaizen where we talk about the next improvement to changelog.com: we are now running on fly.io and our PostgreSQL is managed. This is a migration that many were curious about, including Simmy de Klerk, the person that requested this episode. After migrating all our media files to AWS S3 (check episode 40), we thought that this part was going to be easy. Plan met reality. Pull request 407 has all the details. We want to emphasise the type of partner relationships that we seek at Changelog & why they are important to us, as well as to our listeners. Honeycomb & Fly embody the principles that we care about, and Gerhard thinks that we are currently missing a Kubernetes partner.

Alex Sims, a Senior Software Engineer at James & James, an eCommerce fulfilment company, reached out to us about the Kaizen story of the third-party logistics (3PL) platform that he has been involved with for several years now. The system delivered 16 millions of orders in 10 years, and 4.5 million in the last year alone. All the numbers are going up, and there is only so much that a single PHP monolith deployed as VM images can handle. So how do you even start thinking about the architectural improvements, and inspire everyone involved to move towards better? We encourage you to look at the architectural diagrams in the show notes, especially the 10 year roadmap, and ask Alex for a blog post follow-up. While today's episode was a good conversation starter, there is a lot that we did not have time to cover.