Podcasts about saltstack

Alle Jahre wieder: das CfgMgmtCamp in Gent drehte sich wieder um Infrastruktur-Automatisierung und Cloud-Themen. In großer Runde besprechen wir mit Niklas Werker, Mar Sydymanov, Leon Krass und Jasper Wiegratz unsere Eindrücke. Neben spannenden Keynotes gab es auch spannende Entwicklungen von Pkl-, Puppet- und OpenTofu-Projekten.

Thomas Hatch has spent years building and scaling infrastructure, from the early days of SaltStack to watching Kubernetes and microservices reshape the industry. In this episode, he shares how automation has changed operations at scale and why some companies are rethinking their cloud strategy.About the GuestThomas is currently the CEO of a stealth startup developing the next generation infrastructure management platform. He is the creator of the Salt open source software project and founded the company SaltStack which he sold to VMware in 2020. He has spent his career writing software to orchestrate and automate the work of securing and maintaining enterprise IT infrastructure from core data center systems to the very edge of the network and IoT. Thomas built fully automated, secure IT environments for the U.S. intelligence community in addition to decades of experience implementing global infrastructures for the largest businesses in the world. He has shared his knowledge of IT security and management automation with tens of thousands of practitioners at more than 100 industry events. Thomas and SaltStack have been recognized with numerous awards ranging from open source community growth to innovation in automation and cybersecurity. For his work on Salt, in 2012 Tom received the Black Duck “Rookie of the Year” award and was named to the GitHub Octoverse list in both 2012 and 2013 for leading a project with the highest number of unique contributors, rubbing shoulders with projects from Android, Mozilla, and OpenStack. More recently, SaltStack SecOps was chosen by CSO Magazine as one of the hottest new products at RSA Conference 2019.

Amir Szekely, Owner at CloudSnorkel, joins Corey on Screaming in the Cloud to discuss how he got his start in the early days of cloud and his solo project, CloudSnorkel. Throughout this conversation, Corey and Amir discuss the importance of being pragmatic when moving to the cloud, and the different approaches they see in developers from the early days of cloud to now. Amir shares what motivates him to develop open-source projects, and why he finds fulfillment in fixing bugs and operating CloudSnorkel as a one-man show. About AmirAmir Szekely is a cloud consultant specializing in deployment automation, AWS CDK, CloudFormation, and CI/CD. His background includes security, virtualization, and Windows development. Amir enjoys creating open-source projects like cdk-github-runners, cdk-turbo-layers, and NSIS.Links Referenced: CloudSnorkel: https://cloudsnorkel.com/ lasttootinaws.com: https://lasttootinaws.com camelcamelcamel.com: https://camelcamelcamel.com github.com/cloudsnorkel: https://github.com/cloudsnorkel Personal website: https://kichik.com TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn, and this is an episode that I have been angling for for longer than you might imagine. My guest today is Amir Szekely, who's the owner at CloudSnorkel. Amir, thank you for joining me.Amir: Thanks for having me, Corey. I love being here.Corey: So, I've been using one of your open-source projects for an embarrassingly long amount of time, and for the longest time, I make the critical mistake of referring to the project itself as CloudSnorkel because that's the word that shows up in the GitHub project that I can actually see that jumps out at me. The actual name of the project within your org is cdk-github-runners if I'm not mistaken.Amir: That's real original, right?Corey: Exactly. It's like, “Oh, good, I'll just mention that, and suddenly everyone will know what I'm talking about.” But ignoring the problems of naming things well, which is a pain that everyone at AWS or who uses it knows far too well, the product is basically magic. Before I wind up basically embarrassing myself by doing a poor job of explaining what it is, how do you think about it?Amir: Well, I mean, it's a pretty simple project, which I think what makes it great as well. It creates GitHub runners with CDK. That's about it. It's in the name, and it just does that. And I really tried to make it as simple as possible and kind of learn from other projects that I've seen that are similar, and basically learn from my pain points in them.I think the reason I started is because I actually deployed CDK runners—sorry, GitHub runners—for one company, and I ended up using the Kubernetes one, right? So, GitHub in themselves, they have two projects they recommend—and not to nudge GitHub, please recommend my project one day as well—they have the Kubernetes controller and they have the Terraform deployer. And the specific client that I worked for, they wanted to use Kubernetes. And I tried to deploy it, and, Corey, I swear, I worked three days; three days to deploy the thing, which was crazy to me. And every single step of the way, I had to go and read some documentation, figure out what I did wrong, and apparently the order the documentation was was incorrect.And I had to—I even opened tickets, and they—you know, they were rightfully like, “It's open-source project. Please contribute and fix the documentation for us.” At that point, I said, “Nah.” [laugh]. Let me create something better with CDK and I decided just to have the simplest setup possible.So usually, right, what you end up doing in these projects, you have to set up either secrets or SSM parameters, and you have to prepare the ground and you have to get your GitHub token and all those things. And that's just annoying. So, I decided to create a—Corey: So much busy work.Amir: Yes, yeah, so much busy work and so much boilerplate and so much figuring out the right way and the right order, and just annoying. So, I decided to create a setup page. I thought, “What if you can actually install it just like you install any app on GitHub,” which is the way it's supposed to be right? So, when you install cdk-github-runners—CloudSnorkel—you get an HTML page and you just click a few buttons and you tell it where to install it and it just installs it for you. And it sets the secrets and everything. And if you want to change the secret, you don't have to redeploy. You can just change the secret, right? You have to roll the token over or whatever. So, it's much, much easier to install.Corey: And I feel like I discovered this project through one of the more surreal approaches—and I had cause to revisit it a few weeks ago when I was redoing my talk for the CDK Community Day, which has since happened and people liked the talk—and I mentioned what CloudSnorkel had been doing and how I was using the runners accordingly. So, that was what I accidentally caused me to pop back up with, “Hey, I've got some issues here.” But we'll get to that. Because once upon a time, I built a Twitter client for creating threads because shitposting is my love language, I would sit and create Twitter threads in the middle of live keynote talks. Threading in the native client was always terrible, and I wanted to build something that would help me do that. So, I did.And it was up for a while. It's not anymore because I'm not paying $42,000 a month in API costs to some jackass, but it still exists in the form of lasttootinaws.com if you want to create threads on Mastodon. But after I put this out, some people complained that it was slow.To which my response was, “What do you mean? It's super fast for me in San Francisco talking to it hosted in Oregon.” But on every round trip from halfway around the world, it became a problem. So, I got it into my head that since this thing was fully stateless, other than a Lambda function being fronted via an API Gateway, that I should deploy it to every region. It didn't quite fit into a Cloudflare Worker or into one of the Edge Lambda functions that AWS has given up on, but okay, how do I deploy something to every region?And the answer is, with great difficulty because it's clear that no one was ever imagining with all those regions that anyone would use all of them. It's imagined that most customers use two or three, but customers are different, so which two or three is going to be widely varied. So, anything halfway sensible about doing deployments like this didn't work out. Again, because this thing was also a Lambda function and an API Gateway, it was dirt cheap, so I didn't really want to start spending stupid amounts of money doing deployment infrastructure and the rest.So okay, how do I do this? Well, GitHub Actions is awesome. It is basically what all of AWS's code offerings wish that they were. CodeBuild is sad and this was kind of great. The problem is, once you're out of the free tier, and if you're a bad developer where you do a deploy on every iteration, suddenly it starts costing for what I was doing in every region, something like a quarter of per deploy, which adds up when you're really, really bad at programming.Amir: [laugh].Corey: So, their matrix jobs are awesome, but I wanted to do some self-hosted runners. How do I do that? And I want to keep it cheap, so how do I do a self-hosted runner inside of a Lambda function? Which led me directly to you. And it was nothing short of astonishing. This was a few years ago. I seem to recall that it used to be a bit less well-architected in terms of its elegance. Did it always use step functions, for example, to wind up orchestrating these things?Amir: Yeah, so I do remember that day. We met pretty much… basically as a joke because the Lambda Runner was a joke that I did, and I posted on Twitter, and I was half-proud of my joke that starts in ten seconds, right? But yeah, no, the—I think it always used functions. I've been kind of in love with the functions for the past two years. They just—they're nice.Corey: Oh, they're magic, and AWS is so bad at telling their story. Both of those things are true.Amir: Yeah. And the API is not amazing. But like, when you get it working—and you know, you have to spend some time to get it working—it's really nice because then you have nothing to manage, ever. And they can call APIs directly now, so you don't have to even create Lambdas. It's pretty cool.Corey: And what I loved is you wind up deploying this thing to whatever account you want it to live within. What is it, the OIDC? I always get those letters in the wrong direction. OIDC, I think, is correct.Amir: I think it's OIDC, yeah.Corey: Yeah, and it winds up doing this through a secure method as opposed to just okay, now anyone with access to the project can deploy into your account, which is not ideal. And it just works. It spins up a whole bunch of these Lambda functions that are using a Docker image as the deployment environment. And yeah, all right, if effectively my CDK deploy—which is what it's doing inside of this thing—doesn't complete within 15 minutes, then it's not going to and the thing is going to break out. We've solved the halting problem. After 15 minutes, the loop will terminate. The end.But that's never been a problem, even with getting ACM certificates spun up. It completes well within that time limit. And its cost to me is effectively nothing. With one key exception: that you made the choice to use Secrets Manager to wind up storing a lot of the things it cares about instead of Parameter Store, so I think you wind up costing me—I think there's two of those different secrets, so that's 80 cents a month. Which I will be demanding in blood one of these days if I ever catch you at re:Invent.Amir: I'll buy you beer [laugh].Corey: There we go. That'll count. That'll buy, like, several months of that. That works—at re:Invent, no. The beers there are, like, $18, so that'll cover me for years. We're set.Amir: We'll split it [laugh].Corey: Exactly. Problem solved. But I like the elegance of it, I like how clever it is, and I want to be very clear, though, it's not just for shitposting. Because it's very configurable where, yes, you can use Lambda functions, you can use Spot Instances, you can use CodeBuild containers, you can use Fargate containers, you can use EC2 instances, and it just automatically orchestrates and adds these self-hosted runners to your account, and every build gets a pristine environment as a result. That is no small thing.Amir: Oh, and I love making things configurable. People really appreciate it I feel, you know, and gives people kind of a sense of power. But as long as you make that configuration simple enough, right, or at least the defaults good defaults, right, then, even with that power, people still don't shoot themselves in the foot and it still works really well. By the way, we just added ECS recently, which people really were asking for because it gives you the, kind of, easy option to have the runner—well, not the runner but at least the runner infrastructure staying up, right? So, you can have auto-scaling group backing ECS and then the runner can start up a lot faster. It was actually very important to other people because Lambda, as fast that it is, it's limited, and Fargate, for whatever reason, still to this day, takes a minute to start up.Corey: Yeah. What's wild to me about this is, start to finish, I hit a deploy to the main branch and it sparks the thing up, runs the deploy. Deploy itself takes a little over two minutes. And every time I do this, within three minutes of me pushing to commit, the deploy is done globally. It is lightning fast.And I know it's easy to lose yourself in the idea of this being a giant shitpost, where, oh, who's going to do deployment jobs in Lambda functions? Well, kind of a lot of us for a variety of reasons, some of which might be better than others. In my case, it was just because I was cheap, but the massive parallelization ability to do 20 simultaneous deploys in a matrix configuration that doesn't wind up smacking into rate limits everywhere, that was kind of great.Amir: Yeah, we have seen people use Lambda a lot. It's mostly for, yeah, like you said, small jobs. And the environment that they give you, it's kind of limited, so you can't actually install packages, right? There is no sudo, and you can't actually install anything unless it's in your temp directory. But still, like, just being able to run a lot of little jobs, it's really great. Yeah.Corey: And you can also make sure that there's a Docker image ready to go with the stuff that you need, just by configuring how the build works in the CDK. I will admit, I did have a couple of bug reports for you. One was kind of useful, where it was not at all clear how to do this on top of a Graviton-based Lambda function—because yeah, that was back when not everything really supported ARM architectures super well—and a couple of other times when the documentation was fairly ambiguous from my perspective, where it wasn't at all clear, what was I doing? I spent four hours trying to beat my way through it, I give up, filed an issue, went to get a cup of coffee, came back, and the answer was sitting there waiting for me because I'm not convinced you sleep.Amir: Well, I am a vampire. My last name is from the Transylvania area [laugh]. So—Corey: Excellent. Excellent.Amir: By the way, not the first time people tell me that. But anyway [laugh].Corey: There's something to be said for getting immediate responsiveness because one of the reasons I'm always so loath to go and do a support ticket anywhere is this is going to take weeks. And then someone's going to come back with a, “I don't get it.” And try and, like, read the support portfolio to you. No, you went right into yeah, it's this. Fix it and your problem goes away. And sure enough, it did.Amir: The escalation process that some companies put you through is very frustrating. I mean, lucky for you, CloudSnorkel is a one-man show and this man loves solving bugs. So [laugh].Corey: Yeah. Do you know of anyone using it for anything that isn't ridiculous and trivial like what I'm using it for?Amir: Yeah, I have to think whether or not I can… I mean, so—okay. We have a bunch of dedicated users, right, the GitHub repo, that keep posting bugs and keep posting even patches, right, so you can tell that they're using it. I even have one sponsor, one recurring sponsor on GitHub that uses it.Corey: It's always nice when people thank you via money.Amir: Yeah. Yeah, it is very validating. I think [BLEEP] is using it, but I also don't think I can actually say it because I got it from the GitHub.Corey: It's always fun. That's the beautiful part about open-source. You don't know who's using this. You see what other things people are working on, and you never know, is one of their—is this someone's side project, is it a skunkworks thing, or God forbid, is this inside of every car going forward and no one bothered to tell me about that. That is the magic and mystery of open-source. And you've been doing open-source for longer than I have and I thought I was old. You were originally named in some of the WinAMP credits, for God's sake, that media player that really whipped the llama's ass.Amir: Oh, yeah, I started real early. I started about when I was 15, I think. I started off with Pascal or something or even Perl, and then I decided I have to learn C and I have to learn Windows API. I don't know what possessed me to do that. Win32 API is… unique [laugh].But once I created those applications for myself, right, I think there was—oh my God, do you know the—what is it called, Sherlock in macOS, right? And these days, for PowerToys, there is the equivalent of it called, I don't know, whatever that—PowerBar? That's exactly—that was that. That's a project I created as a kid. I wanted something where I can go to the Run menu of Windows when you hit Winkey R, and you can just type something and it will start it up, right?I didn't want to go to the Start menu and browse and click things. I wanted to do everything with the keyboard. So, I created something called Blazerun [laugh], which [laugh] helped you really easily create shortcuts that went into your path, right, the Windows path, so you can really easily start them from Winkey R. I don't think that anyone besides me used it, but anyway, that thing needed an installer, right? Because Windows, you got to install things. So, I ended up—Corey: Yeah, these days on Mac OS, I use Alfred for that which is kind of long in the tooth, but there's a launch bar and a bunch of other stuff for it. What I love is that if I—I can double-tap the command key and that just pops up whatever I need it to and tell the computer what to do. It feels like there's an AI play in there somewhere if people can figure out how to spend ten minutes on building AI that does something other than lets them fire their customer service staff.Amir: Oh, my God. Please don't fire customer service staff. AI is so bad.Corey: Yeah, when I reach out to talk to a human, I really needed a human.Amir: Yes. Like, I'm not calling you because I want to talk to a robot. I know there's a website. Leave me alone, just give me a person.Corey: Yeah. Like, you already failed to solve my problem on your website. It's person time.Amir: Exactly. Oh, my God. Anyway [laugh]. So, I had to create an installer, right, and I found it was called NSIS. So, it was a Nullsoft “SuperPiMP” installation system. Or in the future, when Justin, the guy who created Winamp and NSIS, tried to tone down a little bit, Nullsoft Scriptable Installation System. And SuperPiMP is—this is such useless history for you, right, but SuperPiMP is the next generation of PiMP which is Plug-in Mini Packager [laugh].Corey: I remember so many of the—like, these days, no one would ever name any project like that, just because it's so off-putting to people with sensibilities, but back then that was half the stuff that came out. “Oh, you don't like how this thing I built for free in the wee hours when I wasn't working at my fast food job wound up—you know, like, how I chose to name it, well, that's okay. Don't use it. Go build your own. Oh, what you're using it anyway. That's what I thought.”Amir: Yeah. The source code was filled with profanity, too. And like, I didn't care, I really did not care, but some people would complain and open bug reports and patches. And my policy was kind of like, okay if you're complaining, I'm just going to ignore you. If you're opening a patch, fine, I'm going to accept that you're—you guys want to create something that's sensible for everybody, sure.I mean, it's just source code, you know? Whatever. So yeah, I started working on that NSIS. I used it for myself and I joined the forums—and this kind of answers to your question of why I respond to things so fast, just because of the fun—I did the same when I was 15, right? I started going on the forums, you remember forums? You remember that [laugh]?Corey: Oh, yeah, back before they all became terrible and monetized.Amir: Oh, yeah. So, you know, people were using NSIS, too, and they had requests, right? They wanted. Back in the day—what was it—there was only support for 16-bit colors for the icon, so they want 32-bit colors and big colors—32—big icon, sorry, 32 pixels by 32 pixels. Remember, 32 pixels?Corey: Oh, yes. Not well, and not happily, but I remember it.Amir: Yeah. So, I started just, you know, giving people—working on that open-source and creating up a fork. It wasn't even called ‘fork' back then, but yeah, I created, like, a little fork of myself and I started adding all these features. And people were really happy, and kind of created, like, this happy cycle for myself: when people were happy, I was happy coding. And then people were happy by what I was coding. And then they were asking for more and they were getting happier, the more I responded.So, it was kind of like a serotonin cycle that made me happy and made everybody happy. So, it's like a win, win, win, win, win. And that's how I started with open-source. And eventually… NSIS—again, that installation system—got so big, like, my fork got so big, and Justin, the guy who works on WinAMP and NSIS, he had other things to deal with. You know, there's a whole history there with AOL. I'm sure you've heard all the funny stories.Corey: Oh, yes. In fact, one thing that—you want to talk about weird collisions of things crossing, one of the things I picked up from your bio when you finally got tired of telling me no and agreed to be on the show was that you're also one of the team who works on camelcamelcamel.com. And I keep forgetting that's one of those things that most people have no idea exists. But it's very simple: all it does is it tracks Amazon products that you tell it to and alerts you when there's a price drop on the thing that you're looking at.It's something that is useful. I try and use it for things of substance or hobbies because I feel really pathetic when I'm like, get excited emails about a price drop in toilet paper. But you know, it's very handy just to keep an idea for price history, where okay, am I actually being ripped off? Oh, they claim it's their big Amazon Deals day and this is 40% off. Let's see what camelcamelcamel has to say.Oh, surprise. They just jacked the price right beforehand and now knocked 40% off. Genius. I love that. It always felt like something that was going to be blown off the radar by Amazon being displeased, but I discovered you folks in 2010 and here you are now, 13 years later, still here. I will say the website looks a lot better now.Amir: [laugh]. That's a recent change. I actually joined camel, maybe two or three years ago. I wasn't there from the beginning. But I knew the guy who created it—again, as you were saying—from the Winamp days, right? So, we were both working in the free—well, it wasn't freenode. It was not freenode. It was a separate IRC server that, again, Justin created for himself. It was called landoleet.Corey: Mmm. I never encountered that one.Amir: Yeah, no, it was pretty private. The only people that cared about WinAMP and NSIS ended up joining there. But it was a lot of fun. I met a lot of friends there. And yeah, I met Daniel Green there as well, and he's the guy that created, along with some other people in there that I think want to remain anonymous so I'm not going to mention, but they also were on the camel project.And yeah, I was kind of doing my poor version of shitposting on Twitter about AWS, kind of starting to get some traction and maybe some clients and talk about AWS so people can approach me, and Daniel approached me out of the blue and he was like, “Do you just post about AWS on Twitter or do you also do some AWS work?” I was like, “I do some AWS work.”Corey: Yes, as do all of us. It's one of those, well crap, we're getting called out now. “Do you actually know how any of this stuff works?” Like, “Much to my everlasting shame, yes. Why are you asking?”Amir: Oh, my God, no, I cannot fix your printer. Leave me alone.Corey: Mm-hm.Amir: I don't want to fix your Lambdas. No, but I do actually want to fix your Lambdas. And so, [laugh] he approached me and he asked if I can help them move camelcamelcamel from their data center to AWS. So, that was a nice big project. So, we moved, actually, all of camelcamelcamel into AWS. And this is how I found myself not only in the Winamp credits, but also in the camelcamelcamel credits page, which has a great picture of me riding a camel.Corey: Excellent. But one of the things I've always found has been that when you take an application that has been pre-existing for a while in a data center and then move it into the cloud, you suddenly have to care about things that no one sensible pays any attention to in the land of the data center. Because it's like, “What do I care about how much data passes between my application server and the database? Wait, what do you mean that in this configuration, that's a chargeable data transfer? Oh, dear Lord.” And things that you've never had to think about optimizing are suddenly things are very much optimizing.Because let's face it, when it comes to putting things in racks and then running servers, you aren't auto-scaling those things, so everything tends to be running over-provisioned, for very good reasons. It's an interesting education. Anything you picked out from that process that you think it'd be useful for folks to bear in mind if they're staring down the barrel of the same thing?Amir: Yeah, for sure. I think… in general, right, not just here. But in general, you always want to be pragmatic, right? You don't want to take steps are huge, right? So, the thing we did was not necessarily rewrite everything and change everything to AWS and move everything to Lambda and move everything to Docker.Basically, we did a mini lift-and-shift, but not exactly lift-and-shift, right? We didn't take it as is. We moved to RDS, we moved to ElastiCache, right, we obviously made use of security groups and session connect and we dropped SSH Sage and we improved the security a lot and we locked everything down, all the permissions and all that kind of stuff, right? But like you said, there's stuff that you start having to pay attention to. In our case, it was less the data transfer because we have a pretty good CDN. There was more of IOPS. So—and IOPS, specifically for a database.We had a huge database with about one terabyte of data and a lot of it is that price history that you see, right? So, all those nice little graphs that we create in—what do you call them, charts—that we create in camelcamelcamel off the price history. There's a lot of data behind that. And what we always want to do is actually remove that from MySQL, which has been kind of struggling with it even before the move to AWS, but after the move to AWS, where everything was no longer over-provisioned and we couldn't just buy a few more NVMes on Amazon for 100 bucks when they were on sale—back when we had to pay Amazon—Corey: And you know, when they're on sale. That's the best part.Amir: And we know [laugh]. We get good prices on NVMe. But yeah, on Amazon—on AWS, sorry—you have to pay for io1 or something, and that adds up real quick, as you were saying. So, part of that move was also to move to something that was a little better for that data structure. And we actually removed just that data, the price history, the price points from MySQL to DynamoDB, which was a pretty nice little project.Actually, I wrote about it in my blog. There is, kind of, lessons learned from moving one terabyte from MySQL to DynamoDB, and I think the biggest lesson was about hidden price of storage in DynamoDB. But before that, I want to talk about what you asked, which was the way that other people should make that move, right? So again, be pragmatic, right? If you Google, “How do I move stuff from DynamoDB to MySQL,” everybody's always talking about their cool project using Lambda and how you throttle Lambda and how you get throttled from DynamoDB and how you set it up with an SQS, and this and that. You don't need all that.Just fire up an EC2 instance, write some quick code to do it. I used, I think it was Go with some limiter code from Uber, and that was it. And you don't need all those Lambdas and SQS and the complication. That thing was a one-time thing anyway, so it doesn't need to be super… super-duper serverless, you know?Corey: That is almost always the way that it tends to play out. You encounter these weird little things along the way. And you see so many things that are tied to this is how architecture absolutely must be done. And oh you're not a real serverless person if you don't have everything running in Lambda and the rest. There are times where yeah, spin up an EC2 box, write some relatively inefficient code in ten minutes and just do the thing, and then turn it off when you're done. Problem solved. But there's such an aversion to that. It's nice to encounter people who are pragmatists more than they are zealots.Amir: I mostly learned that lesson. And both Daniel Green and me learned that lesson from the Winamp days. Because we both have written plugins for Winamp and we've been around that area and you can… if you took one of those non-pragmatist people, right, and you had them review the Winamp code right now—or even before—they would have a million things to say. That code was—and NSIS, too, by the way—and it was so optimized. It was so not necessarily readable, right? But it worked and it worked amazing. And Justin would—if you think I respond quickly, right, Justin Frankel, the guy who wrote Winamp, he would release versions of NSIS and of Winamp, like, four versions a day, right? That was before [laugh] you had CI/CD systems and GitHub and stuff. That was just CVS. You remember CVS [laugh]?Corey: Oh, I've done multiple CVS migrations. One to Git and a couple to Subversion.Amir: Oh yeah, Subversion. Yep. Done ‘em all. CVS to Subversion to Git. Yep. Yep. That was fun.Corey: And these days, everyone's using Git because it—we're beginning to have a monoculture.Amir: Yeah, yeah. I mean, but Git is nicer than Subversion, for me, at least. I've had more fun with it.Corey: Talk about damning with faint praise.Amir: Faint?Corey: Yeah, anything's better than Subversion, let's be honest here.Amir: Oh [laugh].Corey: I mean, realistically, copying a bunch of files and directories to a.bak folder is better than Subversion.Amir: Well—Corey: At least these days. But back then it was great.Amir: Yeah, I mean, the only thing you had, right [laugh]?Corey: [laugh].Amir: Anyway, achieving great things with not necessarily the right tools, but just sheer power of will, that's what I took from the Winamp days. Just the entire world used Winamp. And by the way, the NSIS project that I was working on, right, I always used to joke that every computer in the world ran my code, every Windows computer in the world when my code, just because—Corey: Yes.Amir: So, many different companies use NSIS. And none of them cared that the code was not very readable, to put it mildly.Corey: So, many companies founder on those shores where they lose sight of the fact that I can point to basically no companies that died because their code was terrible, yeah, had an awful lot that died with great-looking code, but they didn't nail the business problem.Amir: Yeah. I would be lying if I said that I nailed exactly the business problem at NSIS because the most of the time I would spend there and actually shrinking the stub, right, there was appended to your installer data, right? So, there's a little stub that came—the executable, basically, that came before your data that was extracted. I spent, I want to say, years of my life [laugh] just shrinking it down by bytes—by literal bytes—just so it stays under 34, 35 kilobytes. It was kind of a—it was a challenge and something that people appreciated, but not necessarily the thing that people appreciate the most. I think the features—Corey: Well, no I have to do the same thing to make sure something fits into a Lambda deployment package. The scale changes, the problem changes, but somehow everything sort of rhymes with history.Amir: Oh, yeah. I hope you don't have to disassemble code to do that, though because that's uh… I mean, it was fun. It was just a lot.Corey: I have to ask, how much work went into building your cdk-github-runners as far as getting it to a point of just working out the door? Because I look at that and it feels like there's—like, the early versions, yeah, there wasn't a whole bunch of code tied to it, but geez, the iterative, “How exactly does this ridiculous step functions API work or whatnot,” feels like I'm looking at weeks of frustration. At least it would have been for me.Amir: Yeah, yeah. I mean, it wasn't, like, a day or two. It was definitely not—but it was not years, either. I've been working on it I think about a year now. Don't quote me on that. But I've put a lot of time into it. So, you know, like you said, the skeleton code is pretty simple: it's a step function, which as we said, takes a long time to get right. The functions, they are really nice, but their definition language is not very straightforward. But beyond that, right, once that part worked, it worked. Then came all the bug reports and all the little corner cases, right? We—Corey: Hell is other people's use cases. Always is. But that's honestly better than a lot of folks wind up experiencing where they'll put an open-source project up and no one ever knows. So, getting users is often one of the biggest barriers to a lot of this stuff. I've found countless hidden gems lurking around on GitHub with a very particular search for something that no one had ever looked at before, as best I can tell.Amir: Yeah.Corey: Open-source is a tricky thing. There needs to be marketing brought into it, there needs to be storytelling around it, and has to actually—dare I say—solve a problem someone has.Amir: I mean, I have many open-source projects like that, that I find super useful, I created for myself, but no one knows. I think cdk-github-runners, I'm pretty sure people know about it only because you talked about it on Screaming in the Cloud or your newsletter. And by the way, thank you for telling me that you talked about it last week in the conference because now we know why there was a spike [laugh] all of a sudden. People Googled it.Corey: Yeah. I put links to it as well, but it's the, yeah, I use this a lot and it's great. I gave a crappy explanation on how it works, but that's the trick I've found between conference talks and, dare I say, podcast episodes, you gives people a glimpse and a hook and tell them where to go to learn more. Otherwise, you're trying to explain every nuance and every intricacy in 45 minutes. And you can't do that effectively in almost every case. All you're going to do is drive people away. Make it sound exciting, get them to see the value in it, and then let them go.Amir: You have to explain the market for it, right? That's it.Corey: Precisely.Amir: And I got to say, I somewhat disagree with your—or I have a different view when you say that, you know, open-source projects needs marketing and all those things. It depends on what open-source is for you, right? I don't create open-source projects so they are successful, right? It's obviously always nicer when they're successful, but—and I do get that cycle of happiness that, like I was saying, people create bugs and I have to fix them and stuff, right? But not every open-source project needs to be a success. Sometimes it's just fun.Corey: No. When I talk about marketing, I'm talking about exactly what we're doing here. I'm not talking take out an AdWords campaign or something horrifying like that. It's you build something that solved the problem for someone. The big problem that worries me about these things is how do you not lose sleep at night about the fact that solve someone's problem and they don't know that it exists?Because that drives me nuts. I've lost count of the number of times I've been beating my head against a wall and asked someone like, “How would you handle this?” Like, “Oh, well, what's wrong with this project?” “What do you mean?” “Well, this project seems to do exactly what you want it to do.” And no one has it all stuffed in their head. But yeah, then it seems like open-source becomes a little more corporatized and it becomes a lead gen tool for people to wind up selling their SaaS services or managed offerings or the rest.Amir: Yeah.Corey: And that feels like the increasing corporatization of open-source that I'm not a huge fan of.Amir: Yeah. I mean, I'm not going to lie, right? Like, part of why I created this—or I don't know if it was part of it, but like, I had a dream that, you know, I'm going to get, oh, tons of GitHub sponsors, and everybody's going to use it and I can retire on an island and just make money out of this, right? Like, that's always a dream, right? But it's a dream, you know?And I think bottom line open-source is… just a tool, and some people use it for, like you were saying, driving sales into their SaaS, some people, like, may use it just for fun, and some people use it for other things. Or some people use it for politics, even, right? There's a lot of politics around open-source.I got to tell you a story. Back in the NSIS days, right—talking about politics—so this is not even about politics of open-source. People made NSIS a battleground for their politics. We would have translations, right? People could upload their translations. And I, you know, or other people that worked on NSIS, right, we don't speak every language of the world, so there's only so much we can do about figuring out if it's a real translation, if it's good or not.Back in the day, Google Translate didn't exist. Like, these days, we check Google Translate, we kind of ask a few questions to make sure they make sense. But back in the day, we did the best that we could. At some point, we got a patch for Catalan language, I'm probably mispronouncing it—but the separatist people in Spain, I think, and I didn't know anything about that. I was a young kid and… I just didn't know.And I just included it, you know? Someone submitted a patch, they worked hard, they wanted to be part of the open-source project. Why not? Sure I included it. And then a few weeks later, someone from Spain wanted to change Catalan into Spanish to make sure that doesn't exist for whatever reason.And then they just started fighting with each other and started making demands of me. Like, you have to do this, you have to do that, you have to delete that, you have to change the name. And I was just so baffled by why would someone fight so much over a translation of an open-source project. Like, these days, I kind of get what they were getting at, right?Corey: But they were so bad at telling that story that it was just like, so basically, screw, “You for helping,” is how it comes across.Amir: Yeah, screw you for helping. You're a pawn now. Just—you're a pawn unwittingly. Just do what I say and help me in my political cause. I ended up just telling both of them if you guys can agree on anything, I'm just going to remove both translations. And that's what I ended up doing. I just removed both translations. And then a few months later—because we had a release every month basically, I just added both of them back and I've never heard from them again. So sort of problem solved. Peace the Middle East? I don't know.Corey: It's kind of wild just to see how often that sort of thing tends to happen. It's a, I don't necessarily understand why folks are so opposed to other people trying to help. I think they feel like there's this loss of control as things are slipping through their fingers, but it's a really unwelcoming approach. One of the things that got me deep into the open-source ecosystem surprisingly late in my development was when I started pitching in on the SaltStack project right after it was founded, where suddenly everything I threw their way was merged, and then Tom Hatch, the guy who founded the project, would immediately fix all the bugs and stuff I put in and then push something else immediately thereafter. But it was such a welcoming thing.Instead of nitpicking me to death in the pull request, it just got merged in and then silently fixed. And I thought that was a classy way to do it. Of course, it doesn't scale and of course, it causes other problems, but I envy the simplicity of those days and just the ethos behind that.Amir: That's something I've learned the last few years, I would say. Back in the NSIS day, I was not like that. I nitpicked. I nitpicked a lot. And I can guess why, but it just—you create a patch—in my mind, right, like you create a patch, you fix it, right?But these days I get, I've been on the other side as well, right? Like I created patches for open-source projects and I've seen them just wither away and die, and then five years later, someone's like, “Oh, can you fix this line to have one instead of two, and then I'll merge it.” I'm like, “I don't care anymore. It was five years ago. I don't work there anymore. I don't need it. If you want it, do it.”So, I get it these days. And these days, if someone creates a patch—just yesterday, someone created a patch to format cdk-github-runners in VS Code. And they did it just, like, a little bit wrong. So, I just fixed it for them and I approved it and pushed it. You know, it's much better. You don't need to bug people for most of it.Corey: You didn't yell at them for having the temerity to contribute?Amir: My voice is so raw because I've been yelling for five days at them, yeah.Corey: Exactly, exactly. I really want to thank you for taking the time to chat with me about how all this stuff came to be and your own path. If people want to learn more, where's the best place for them to find you?Amir: So, I really appreciate you having me and driving all this traffic to my projects. If people want to learn more, they can always go to cloudsnorkel.com; it has all the projects. github.com/cloudsnorkel has a few more. And then my private blog is kichik.com. So, K-I-C-H-I-K dot com. I don't post there as much as I should, but it has some interesting AWS projects from the past few years that I've done.Corey: And we will, of course, put links to all of that in the show notes. Thank you so much for taking the time. I really appreciate it.Amir: Thank you, Corey. It was really nice meeting you.Corey: Amir Szekely, owner of CloudSnorkel. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, along with an insulting comment. Heck, put it on all of the podcast platforms with a step function state machine that you somehow can't quite figure out how the API works.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.

Jeff Geerling, Owner of Midwestern Mac, joins Corey on Screaming in the Cloud to discuss the importance of storytelling, problem-solving, and community in the world of cloud. Jeff shares how and why he creates content that can appeal to anybody, rather than focusing solely on the technical qualifications of his audience, and how that strategy has paid off for him. Corey and Jeff also discuss the impact of leading with storytelling as opposed to features in product launches, and what's been going on in the Raspberry Pi space recently. Jeff also expresses the impact that community has on open-source companies, and reveals his take on the latest moves from Red Hat and Hashicorp. About JeffJeff is a father, author, developer, and maker. He is sometimes called "an inflammatory enigma".Links Referenced:Personal webpage: https://jeffgeerling.com/ TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. A bit off the beaten path of the usual cloud-focused content on this show, today I'm speaking with Jeff Geerling, YouTuber, author, content creator, enigma, and oh, so much more. Jeff, thanks for joining me.Jeff: Thanks for having me, Corey.Corey: So, it's hard to figure out where you start versus where you stop, but I do know that as I've been exploring a lot of building up my own home lab stuff, suddenly you are right at the top of every Google search that I wind up conducting. I was building my own Kubernete on top of a Turing Pi 2, and sure enough, your teardown was the first thing that I found that, to be direct, was well-documented, and made it understandable. And that's not the first time this year that that's happened to me. What do you do exactly?Jeff: I mean, I do everything. And I started off doing web design and then I figured that design is very, I don't know, once it started transitioning to everything being JavaScript, that was not my cup of tea. So, I got into back-end work, databases, and then I realized to make that stuff work well, you got to know the infrastructure. So, I got into that stuff. And then I realized, like, my home lab is a great place to experiment on this, so I got into Raspberry Pis, low-power computing efficiency, building your own home lab, all that kind of stuff.So, all along the way, with everything I do, I always, like, document everything like crazy. That's something my dad taught me. He's an engineer in radio. And he actually hired me for my first job, he had me write an IT operations manual for the Radio Group in St. Louis. And from that point forward, that's—I always start with documentation. So, I think that was probably what really triggered that whole series. It happens to me too; I search for something, I find my old articles or my own old projects on GitHub or blog posts because I just put everything out there.Corey: I was about to ask, years ago, I was advised by Scott Hanselman to—the third time I find myself explaining something, write a blog post about it because it's easier to refer people back to that thing than it is for me to try and reconstruct it on the fly, and I'll drop things here and there. And the trick is, of course, making sure it doesn't sound dismissive and like, “Oh, I wrote a thing. Go read.” Instead of having a conversation with people. But as a result, I'll be Googling how to do things from time to time and come up with my own content as a result.It's at least a half-step up from looking at forums and the rest, where I realized halfway through that I was the one asking the question. Like, “Oh, well, at least this is useful for someone.” And I, for better or worse, at least have a pattern of going back and answering how I solved a thing after I get there, just because otherwise, it's someone asked the question ten years ago and never returns, like, how did you solve it? What did you do? It's good to close that loop.Jeff: Yeah, and I think over 50% of what I do, I've done before. When you're setting up a Kubernetes cluster, there's certain parts of it that you're going to do every time. So, whatever's not automated or the tricky bits, I always document those things. Anything that is not in the readme, is not in the first few steps, because that will help me and will help others. I think that sometimes that's the best success I've found on YouTube is also just sharing an experience.And I think that's what separates some of the content that really drives growth on a YouTube channel or whatever, or for an organization doing it because you bring the experience, like, I'm a new person to this Home Assistant, for instance, which I use to automate things at my house. I had problems with it and I just shared those problems in my video, and that video has, you know, hundreds of thousands of views. Whereas these other people who know way more than I could ever know about Home Assistant, they're pulling in fewer views because they just get into a tutorial and don't have that perspective of a beginner or somebody that runs into an issue and how do you solve that issue.So, like I said, I mean, I just always share that stuff. Every time that I have an issue with anything technological, I put it on GitHub somewhere. And then eventually, if it's something that I can really formulate into an outline of what I did, I put a blog post up on my blog. I still, even though I write I don't know how many words per week that goes into my YouTube videos or into my books or anything, I still write two or three blog posts a week that are often pretty heavy into technical detail.Corey: One of the challenges I've always had is figuring out who exactly I'm storytelling for when I'm putting something out there. Because there's a plethora, at least in cloud, of beginner content of, here's how to think about cloud, here's what the service does, here's why you should use it et cetera, et cetera. And that's all well and good, but often the things that I'm focusing on presuppose a certain baseline level of knowledge that you should have going into this. If you're trying to figure out the best way to get some service configured, I probably shouldn't have to spend the first half of the article talking about what AWS is, as a for instance. And I think that inherently limits the size of the potential audience that would be interested in the content, but it's also the kind of stuff that I wish was out there.Jeff: Yeah. There's two sides to that, too. One is, you can make content that appeals to anybody, even if they have no clue what you're talking about, or you can make content that appeals to the narrow audience that knows the base level of understanding you need. So, a lot of times with—especially on my YouTube channel, I'll put things in that is just irrelevant to 99% of the population, but I get so many comments, like, “I have no clue what you said or what you're doing, but this looks really cool.” Like, “This is fun or interesting.” Just because, again, it's bringing that story into it.Because really, I think on a base level, a lot of programmers especially don't understand—and infrastructure engineers are off the deep end on this—they don't understand the interpersonal nature of what makes something good or not, what makes something relatable. And trying to bring that into technical documentation a lot of times is what differentiates a project. So, one of the products I love and use and recommend everywhere and have a book on—a best-selling book—is Ansible. And one of the things that brought me into it and has brought so many people is the documentation started—it's gotten a little bit more complex over the years—but it started out as, “Here's some problems. Here's how you solve them.”Here's, you know, things that we all run into, like how do you connect to 12 servers at the same time? How do you have groups of servers? Like, it showed you all these little examples. And then if you wanted to go deeper, there was more documentation linked out of that. But it was giving you real-world scenarios and doing it in a simple way. And it used some little easter eggs and fun things that made it more interesting, but I think that that's missing from a lot of technical discussion and a lot of technical documentation out there is that playfulness, that human side, the get from Point A to Point B and here's why and here's how, but here's a little interesting way to do it instead of just here's how it's done.Corey: In that same era, I was one of the very early developers behind SaltStack, and I think one of the reasons that Ansible won in the market was that when you started looking into SaltStack, it got wrapped around its own axle talking about how it uses ZeroMQ for a full mesh between all of the systems there, as long—sorry [unintelligible 00:07:39] mesh network that all routes—not really a mesh network at all—it talks through a single controller that then talks to all of its subordinate nodes. Great. That's awesome. How do I use this to install a web server, is the question that people had. And it was so in love with its own cleverness in some ways. Ansible was always much more approachable in that respect and I can't understate just how valuable that was for someone who just wants to get the problem solved.Jeff: Yeah. I also looked at something like NixOS. It's kind of like the arch of distributions of—Corey: You must be at least this smart to use it in some respects—Jeff: Yeah, it's—Corey: —has been the every documentation I've had with that.Jeff: [laugh]. There's, like, this level of pride in what it does, that doesn't get to ‘and it solves this problem.' You can get there, but you have to work through the barrier of, like, we're so much better, or—I don't know what—it's not that. Like, it's just it doesn't feel like, “You're new to this and here's how you can solve a problem today, right now.” It's more like, “We have this golden architecture and we want you to come up to it.” And it's like, well, but I'm not ready for that. I'm just this random developer trying to solve the problem.Corey: Right. Like, they should have someone hanging out in their IRC channel and just watch for a week of who comes in and what questions do they have when they're just getting started and address those. Oh, you want to wind up just building a Nix box EC2 for development? Great, here's how you do that, and here's how to think about your workflow as you go. Instead, I found that I had to piece it together from a bunch of different blog posts and the rest and each one supposed that I had different knowledge coming into it than the others. And I felt like I was getting tangled up very easily.Jeff: Yeah, and I think it's telling that a lot of people pick up new technology through blog posts and Substack and Medium and whatever [Tedium 00:09:19], all these different platforms because it's somebody that's solving a problem and relating that problem, and then you have the same problem. A lot of times in the documentation, they don't take that approach. They're more like, here's all our features and here's how to use each feature, but they don't take a problem-based approach. And again, I'm harping on Ansible here with how good the documentation was, but it took that approach is you have a bunch of servers, you want to manage them, you want to install stuff on them, and all the examples flowed from that. And then you could get deeper into the direct documentation of how things worked.As a polar opposite of that, in a community that I'm very much involved in still—well, not as much as I used to be—is Drupal. Their documentation was great for developers but not so great for beginners and that was always—it still is a difficulty in that community. And I think it's a difficulty in many, especially open-source communities where you're trying to build the community, get more people interested because that's where the great stuff comes from. It doesn't come from one corporation that controls it, it comes from the community of users who are passionate about it. And it's also tough because for something like Drupal, it gets more complex over time and the complexity kind of kills off the initial ability to think, like, wow, this is a great little thing and I can get into it and start using it.And a similar thing is happening with Ansible, I think. We were at when I got started, there were a couple hundred modules. Now there's, like, 4000 modules, or I don't know how many modules, and there's all these collections, and there's namespaces now, all these things that feel like Java overhead type things leaking into it. And that diminishes that ability for me to see, like, oh, this is my simple tool that solving these problems.Corey: I think that that is a lost art in the storytelling side of even cloud marketing, where they're so wrapped around how they do what they do that they forget, customers don't care. Customers care very much about their problem that they're trying to solve. If you have an answer for solving that problem, they're very interested. Otherwise, they do not care. That seems to be a missing gap.Jeff: I think, like, especially for AWS, Google, Azure cloud platforms, when they build their new services, sometimes you're, like, “And that's for who?” For some things, it's so specialized, like, Snowmobile from Amazon, like, there's only a couple customers on the planet in a given year that needs something like that. But it's a cool story, so it's great to put that into your presentation. But some other things, like, especially nowadays with AI, seems like everybody's throwing tons of AI stuff—spaghetti—at the wall, seeing what will stick and then that's how they're doing it. But that really muddies up everything.If you have a clear vision, like with Apple, they just had their presentation on the new iPhone and the new neural engine and stuff, they talk about, “We see your heart patterns and we tell you when your heart is having problems.” They don't talk about their AI features or anything. I think that leading with that story and saying, like, here's how we use this, here's how customers can build off of it, those stories are the ones that are impactful and make people remember, like, oh Apple is the company that saves people's lives by making watches that track their heart. People don't think that about Google, even though they might have the same feature. Google says we have all these 75 sensors in our thing and we have this great platform and Android and all that. But they don't lead with the story.And that's something where I think corporate Apple is better than some of the other organizations, no matter what the technology is. But I get that feeling a lot when I'm watching launches from Amazon and Google and all their big presentations. It seems like they're tech-heavy and they're driven by, like, “What could we do with this? What could you do with this new platform that we're building,” but not, “And this is what we did with this other platform,” kind of building up through that route.Corey: Something I've been meaning to ask someone who knows for a while, and you are very clearly one of those people, I spend a lot of time focusing on controlling cloud costs and I used to think that Managed NAT Gateways were very expensive. And then I saw the current going rates for Raspberries Pi. And that has been a whole new level of wild. I mean, you mentioned a few minutes ago that you use Home Assistant. I do too.But I was contrasting the price between a late model, Raspberry Pi 4—late model; it's three years old if this point of memory serves, maybe four—versus a used small form factor PC from HP, and the second was less expensive and far more capable. Yeah it drags a bit more power and it's a little bit larger on the shelf, but it was basically no contest. What has been going on in that space?Jeff: I think one of the big things is we're at a generational improvement with those small form-factor little, like, tiny-size almost [nook-sized 00:13:59] PCs that were used all over the place in corporate environments. I still—like every doctor's office you go to, every hospital, they have, like, a thousand of these things. So, every two or three or four years, however long it is on their contract, they just pop all those out the door and then you get an E-waste company that picks up a thousand of these boxes and they got to offload them. So, the nice thing is that it seems like a year or two ago, that really started accelerating to the point where the price was driven down below 100 bucks for a fully built-out little x86 Mini PC. Sure, it's, you know, like you said, a few generations old and it pulls a little bit more power, usually six to eight watts at least, versus a Raspberry Pi at two to three watts, but especially for those of us in the US, electricity is not that expensive so adding two or three watts to your budget for a home lab computer is not that bad.The other part of that is, for the past two-and-a-half years because of the global chip shortages and because of the decisions that Raspberry Pi made, there were so few Raspberry Pis available that their prices shot up through the roof if you wanted to get one in any timely fashion. So, that finally is clearing up, although I went to the Micro Center near me yesterday, and they said that they have not had stock of Raspberry Pi 4s for, like, two months now. So, they're coming, but they're not distributed evenly everywhere. And still, the best answer, especially if you're going to run a lot of things on it, is probably to buy one of those little mini PCs if you're starting out a home lab.Or there's some other content creators who build little Kubernetes clusters with multiple mini PCs. Three of those stack up pretty nicely and they're still super quiet. I think they're great for home labs. I have two of them over on my shelf that I'm using for testing and one of them is actually in my rack. And I have another one on my desk here that I'm trying to set up for a five gigabit home router since I finally got fiber internet after years with cable and I'm still stuck on my old gigabit router.Corey: Yeah, I wound up switching to a Protectli, I think is what it's called for—it's one of those things I've installed pfSense on. Which, I'm an old FreeBSD hand and I haven't kept up with it, but that's okay. It feels like going back in time ten years, in some respects—Jeff: [laugh].Corey: —so all right. And I have a few others here and there for various things that I want locally. But invariably, I've had the WiFi controller; I've migrated that off. That lives on an EC2 box in Ohio now. And I do wind up embracing cloud services when I don't want it to go down and be consistently available, but for small stuff locally, I mean, I have an antenna on the roof doing an ADS-B receiver dance that's plugged into a Pi Zero.I have some backlogged stuff on this, but they've gotten expensive as alternatives have dropped in price significantly. But what I'm finding as I'm getting more into 3D printing and a lot of hobbyist maker tools out there, everything is built with the Raspberry Pi in mind; it has the mindshare. And yeah, I can get something with similar specs that are equivalent, but then I've got to do a whole bunch of other stuff as soon as it gets into controlling hardware via GPIO pins or whatnot. And I have to think about it very differently.Jeff: Yeah, and that's the tough thing. And that's the reason why Raspberry Pis, even though they're three years old, even though they're hard to get, they still are fetching—on the used market—way more than the original MSRP. It's just crazy. But the reason for that is the Raspberry Pi organization. And there's two: there's the Raspberry Pi Foundation that's goals are to increase educational computing and accessibility for computers for kids and learning and all that, then there's the Raspberry Pi trading company that makes the Raspberry Pis.The Trading Company has engineers who sit there 24/7 working on the software, working on the kernel drivers, working on hardware bugs, listening to people on the forums and in GitHub and everywhere, and they're all English-speaking people there—they're over in the UK—and they manufacture their own boards. So, there's a lot of things on top of that, even though they're using some silicons of Broadcom chips that are a little bit locked down and not completely open-source like some other chips might be, they're a phone number you could call if you need the support or there's a forum that has activity that you can get help in and their software that's supported. And there's a newer Linux kernel and the kernel is updated all the time. So, all those advantages mean you get a little package that will work, it'll sip two watts of power, sitting 24/7. It's reliable hardware.There's so many people that use it that it's so well tested that almost any problem you could ever run into, someone else has and there's a blog post or a forum post talking about it. And even though the hardware is not super powerful—it's three years old—you can add on a Coral TPU and do face recognition and object recognition. And throw in Frigate for Home Assistant to get notifications on your phone when your mom walks up to the door. There's so many things you can do with them and they're so flexible that they're still so valuable. I think that they really knocked it out of the park with that model, the Raspberry Pi 4, and the compute module 4, which is still impossible to get. I have not been able to buy one for two years now. Luckily, I bought 12 two-and-a-half years ago [laugh] otherwise I would be running out for all my projects that I do.Corey: Yeah. I got two at the moment and two empty slots in the Turing Pi 2, which I'll care more about if I can actually get the thing up and booted. But it presupposes you have a Windows computer or otherwise, ehh, watch this space; more coming. Great. Like, do I build a virtual machine on top of something else? It leads down the path super quickly of places I thought I'd escaped from.Jeff: Yeah, you know, outside of the Pi realm, that's the state of the communities. It's a lot of, like, figuring out your own things. I did a project—I don't know if you've heard of Mr. Beast—but we did a project for him that involves a hundred single-board computers. We couldn't find Raspberry Pi's so we had to use a different single-board computer that was available.And so, I bought an older one thinking, oh, this is, like, three or four years old—it's older than the Pi 4—and there must be enough support now. But still, there's, like, little rough edges everywhere I went and we ended up making them work, but it took us probably an extra 30 to 40 hours of development work to get those things running the same way as a Raspberry Pi. And that's just the way of things. There's so much opportunity.If one of these Chinese manufacturers that makes most of these things, if one of them decided, you know what? We're going to throw tons of money into building support for these things, get some English-speaking members of these forums to build up the community, all that stuff, I think that they could have a shot at Raspberry Pi's giant portion of the market. But so far, I haven't really seen that happen. So far, they're spamming hardware. And it's like, the hardware is awesome. These chips are great if you know how to deal with them and how to get the software running and how to deal with Linux issues, but if you don't, then they're not great because you might not even get the thing to boot.Corey: I want to harken back to something you said a minute ago, where there's value in having a community around something, where you can see everyone else has already encountered a problem like this. I think that folks who weren't around for the rise of cloud have no real insight into how difficult it used to be just getting servers into racks and everything up, and okay, they're identical, and seven of them are working, but that eighth one isn't for some strange reason. And you spend four hours troubleshooting what turns out to be a bad cable or something not seated properly and it's awful. Cloud got away from a lot of that nonsense. But it's important—at least to me—to not be Captain Edgecase, where if you pick some new cloud provider and Google for how to set up a load balancer and no one's done it before you, that's not great. Whereas if I'm googling now in the AWS realm and no one has done, the thing I'm trying to do, that should be something of a cautionary flag of maybe this isn't how most people go about approaching production. Really think twice about this.Jeff: Yep. Yeah, we ran into that on a project I was working on was using Magento—which I don't know if anybody listening uses Magento, but it's not fun—and we ran into some things where it's like, “We're doing this, and it says that they do this on their official supported platform, but I don't know how they are because the code just doesn't exist here.” So, we ran into some weird edge cases on AWS with some massive infrastructure for the databases, and I ran into scaling issues. But even there, there were forum posts in AWS here and there that had little nuggets that helped us to figure out a way to get around it. And like you say, that is a massive advantage for AWS.And we ran into an issue with, we were one of the first customers trying out the new Lambda functions for RDS—or I don't remember exactly what it was called initially—but we ended up not using that. But we ran into some of these issues and figured out we were the first customer running into this weird scaling thing when we had a certain size of database trying to use it with these Lambda calls. And eventually, they got those things solved, but with AWS, they've seen so many things and some other cloud providers haven't seen these things. So, when you have certain types of applications that need to scale in certain ways, that is so valuable and the community of users, the ability to pull from that community when you need to hire somebody in an emergency, like, we need somebody to help us get this project done and we're having this issue, you can find somebody that is, like, okay, I know how to get you from Point A to Point B and get this project out the door. You can't do that on certain platforms.And open-source projects, too. We've always had that problem in Drupal. The amount of developers who are deep into Drupal to help with the hard problems is not vast, so the ones who can do that stuff, they're all hired off and paid a handsome sum. And if you have those kinds of problems you realize, I either going to need to pay a ton of money or we're just going to have to not do that thing that we wanted to do. And that's tough.Corey: What I've found, sort of across the board, has been that there's a lot of, I guess, open-source community ethos that has bled into a lot of this space and I wanted to make sure that we have time to talk about this because I was incensed a while back when Red Hat decided, “Oh, you know that whole ten-year commitment on CentOS? That project that we acquired and are now basically stabbing in the face?”—disclosure. I used to be part of the CentOS project years ago when I was on network staff for the Freenode IRC network—then it was, “Oh yeah, we're just going to basically undermine our commitments to you and now you can pay us if you want to get that support there.” And that really set me off. Was nice to see you were right there as well in almost lockstep with me, pointing out that this is terrible, just as far as breaking promises you've made to customers. Has your anger cooled any? Because mine hasn't.Jeff: It has not. My temper has cooled. My anger has not. I don't think that they get it. After all the backlash that they got after that, I don't think that the VP-level folks at Red Hat understand that this is already impacting them and will impact them much more in the future because people like me and you, people who help other people build infrastructure and people who recommend operating systems and people who recommend patterns and things, we're just going to drop off using CentOS because it doesn't exist. It does exist and some other people are saying, “Oh, it's actually better to use this new CentOS, you know, Stream. Stream is amazing.” It's not. It's not the same thing. It's different. And—Corey: I used to work at a bank. That was not an option. I mean, granted at the bank for the production systems it was always [REL 00:25:18], but being able to spin up a pre-production environment without having to pay license fees on every VM. Yeah.Jeff: Yeah. And not only that, they did this announcement and framed it a certain way, and the community immediately saw. You know, I think that they're just angry about something, and whether it was a NASA contract with Rocky Linux, or whether it was something Oracle did, who knows, but it seems petty in retrospect, especially in comparison to the amount of backlash that came out of it. And I really don't think that they understand the thing that they had with that Red Hat Enterprise Linux is not a massive growth opportunity for Red Hat. It's, in some ways, a dying product in terms of compared to using cloud stuff, it doesn't matter.You could use CoreOS, you could use NixOS, and you could use anything, it doesn't really matter. For people like you and me, we just want to deploy our software. And if it's containers, it really doesn't matter. It's just the people in government or in certain organizations that have these roles that you have to use whatever FIPS and all that kind of stuff. So, it's not like it's a hyper-growth opportunity for them.CentOS was, like, the only reason why all the software, especially on the open-source side, was compatible with Red Hat because we could use CentOS and it was easy and simple. They took that—well, they tried to take that away and everybody's like, “That's—what are you doing?” Like, I posted my blog post and I think that sparked off quite a bit of consternation, to the point where there was a lot of personal stuff going on. I basically said, “I'm not supporting Red Hat Enterprise Linux for any of my work anymore.” Like, “From this point forward, it's not supported.”I'll support OpenELA, I'll support Rocky Linux or Oracle Linux or whatever because I can get free versions that I don't have to sign into a portal and get a license and download the license and integrate it with my CI work. I'm an open-source developer. I'm not going to pay for stuff or use 16 free licenses. Or I was reached out to and they said, “We'll give you more licenses. We'll give you extra.” And it's like, that's not how this works. Like, I don't have to call Debian and Ubuntu and [laugh] I don't even have to call Oracle to get licenses. I can just download their software and run it.So, you know, I don't think they understood the fact that they had that. And the bigger problem for me was the two-layer approach to destroying all the trust that the community had. First was in, I think it was 2019 when they said—we're in the middle of CentOS 8's release cycle—they said, “We're dropping CentOS 8. It's going to be Stream now.” And everybody was up in arms.And then Rocky Linux and [unintelligible 00:27:52] climbed in and gave us what we wanted: basically, CentOS. So, we're all happy and we had a status quo, and Rocky Linux 9 and [unintelligible 00:28:00] Linux nine came out after Red Hat 9, and the world was a happy place. And then they just dumped this thing on us and it's like, two major release cycles in a row, they did it again. Like, I don't know what this guy's thinking, but in one of the interviews, one of the Red Hat representatives said, “Well, we wanted to do this early in Red Hat 9's release cycle because people haven't started migrating.” It's like, well, I already did all my automation upgrades for CI to get all my stuff working in Rocky Linux 9 which was compatible with Red Hat Enterprise Linux 9. Am I not one of the people that's important to you?Like, who's important to you? Is it only the people who pay you money or is it also the people that empower your operating system to be a premier Enterprise Linux operating system? So, I don't know. You can tell. My anger has not died down. The amount of temper that I have about it has definitely diminished because I realize I'm talking at a wall a lot of times, when I'm having conversations on Twitter, private conversations and email, things like that.Corey: People come to argue; they don't come to actually have a discussion.Jeff: Yeah. I think that they just, they don't see the community aspect of it. They just see the business aspect. And the business aspect, if they want to figure out ways that they can get more people to pay them for their software, then maybe they should provide more value and not just cut off value streams. It doesn't make sense to me from a long-term business perspective.From a short term, maybe there were some clients who said, “Oh, shoot. We need this thing stable. We're going to pay for some more licenses.” But the engineers that those places are going to start making plans of, like, how do we make this not happen again. And the way to not make that happen, again is to use, maybe Ubuntu or maybe [unintelligible 00:29:38] or something. Who knows? But it's not going to be increasing our spend with Red Hat.Corey: That's what I think a lot of companies are missing when it comes to community as well, where it's not just a place to go to get support for whatever it is you're doing and it's not a place [where 00:29:57] these companies view prospective customers. There's more to it than that. There has to be a social undercurrent on this. I look at the communities I spend time in and in some of them dating back long enough, I've made lifelong significant friendships out of those places, just through talking about our lives, in addition to whatever the community is built around. You have to make space for that, and companies don't seem to fully understand that.Jeff: Yeah, I think that there's this thing that a community has to provide value and monetizable value, but I don't think that you get open-source if you think that that's what it is. I think some people in corporate open-source think that corporate open-source is a value stream opportunity. It's a funnel, it's something that is going to bring you more customers—like you say—but they don't realize that it's a community. It's like a group of people. It's friends, it's people who want to make the world a better place, it's people who want to support your company by wearing your t-shirt to conferences, people want to put on your red fedora because it's cool. Like, it's all of that. And when you lose some of that, you lose what makes your product differentiated from all the other ones on the market.Corey: That's what gets missed. I think that there's a goodwill aspect of it. People who have used the technology and understand its pitfalls are likelier to adopt it. I mean, if you tell me to get a website up and running, I am going to build an architecture that resembles what I've run before on providers that I've run on before because I know what the failure modes look like; I know how to get things up and running. If I'm in a hurry, trying to get something out the door, I'm going to choose the devil that I know, on some level.Don't piss me off as a community member and incentivize me to change that estimation the next time I've got something to build. Well, that doesn't show up on this quarter's numbers. Well, we have so little visibility into how decisions get made many companies that you'll never know that you have a detractor who's still salty about something you did five years ago and that's the reason the bank decided not to because that person called in their political favors to torpedo that deal and have a sweetheart offer from your competitor, et cetera and so on and so forth. It's hard to calculate the actual cost of alienating goodwill. But—Jeff: Yeah.Corey: I wish companies had a longer memory for these things.Jeff: Yeah. I mean, and thinking about that, like, there was also the HashiCorp incident where they kind of torpedoed all developer goodwill with their Terraform and other—Terraform especially, but also other products. Like, I probably, through my book and through my blog posts and my GitHub examples have brought in a lot of people into the HashiCorp ecosystem through Vagrant use, and through Packer and things like that. At this point, because of the way that they treated the open-source community with the license change, a guy like me is not going to be enthusiastic about it anymore and I'm going to—I already had started looking at alternatives for Vagrant because it doesn't mesh with modern infrastructure practices for local development as much, but now it's like that enthusiasm is completely gone. Like I had that goodwill, like you said earlier, and now I don't have that goodwill and I'm not going to spread that, I'm not going to advocate for them, I'm not going to wear their t-shirt [laugh], you know when I go out and about because it just doesn't feel as clean and cool and awesome to me as it did a month ago.And I don't know what the deal is. It's partly the economy, money's drying up, things like that, but I don't understand how the people at the top can't see these things. Maybe it's just their organization isn't set up to show the benefits from the engineers underneath, who I know some of these engineers are, like, “Yeah, I'm sorry. This was dumb. I still work here because I get a paycheck, but you know, I can't say anything on social media, but thank you for saying what you did on Twitter.” Or X.Corey: Yeah. It's nice being independent where you don't really have to fear the, well if I say this thing online, people might get mad at me and stop doing business with me or fire me. It's well, yeah, I mean, I would have to say something pretty controversial to drive away every client and every sponsor I've got at this point. And I don't generally have that type of failure mode when I get it wrong. I really want to thank you for taking the time to talk with me. If people want to learn more, where's the best place for them to find you?Jeff: Old school, my personal website, jeffgeerling.com. I link to everything from there, I have an About page with a link to every profile I've ever had, so check that out. It links to my books, my YouTube, all that kind of stuff.Corey: There's something to be said for picking a place to contact you that will last the rest of your career as opposed to, back in the olden days, my first email address was the one that my ISP gave me 25 years ago. I don't use that one anymore.Jeff: Yep.Corey: And having to tell everyone I corresponded with that it was changing was a pain in the butt. We'll definitely put a link to that one in the [show notes 00:34:44]. Thank you so much for taking the time to speak with me. I appreciate it.Jeff: Yeah, thanks. Thanks so much for having me.Corey: Jeff Geerling, YouTuber, author, content creator, and oh so very much more. I'm Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with an angry comment that we will, of course, read [in action 00:35:13], just as soon as your payment of compute modules for Raspberries Pi show up in a small unmarked bag.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.

Kelsey Hightower joins Corey on Screaming in the Cloud to discuss his reflections on how the tech industry is progressing. Kelsey describes what he's been getting out of retirement so far, and reflects on what he learned throughout his high-profile career - including why feature sprawl is such a driving force behind the complexity of the cloud environment and the tactics he used to create demos that are engaging for the audience. Corey and Kelsey also discuss the importance of remaining authentic throughout your career, and what it means to truly have an authentic voice in tech. About KelseyKelsey Hightower is a former Distinguished Engineer at Google Cloud, the co-chair of KubeCon, the world's premier Kubernetes conference, and an open source enthusiast. He's also the co-author of Kubernetes Up & Running: Dive into the Future of Infrastructure. Recently, Kelsey announced his retirement after a 25-year career in tech.Links Referenced:Twitter: https://twitter.com/kelseyhightower TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Do you wish there were cheat codes for database optimization? Well, there are – no seriously. If you're using Postgres or MySQL on Amazon Aurora or RDS, OtterTune uses AI to automatically optimize your knobs and indexes and queries and other bits and bobs in databases. OtterTune applies optimal settings and recommendations in the background or surfaces them to you and allows you to do it. The best part is that there's no cost to try it. Get a free, thirty-day trial to take it for a test drive. Go to ottertune dot com to learn more. That's O-T-T-E-R-T-U-N-E dot com.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. You know, there's a great story from the Bible or Torah—Old Testament, regardless—that I was always a big fan of where you wind up with the Israelites walking the desert for 40 years in order to figure out what comes next. And Moses led them but could never enter into what came next. Honestly, I feel like my entire life is sort of going to be that direction. Not the biblical aspects, but rather always wondering what's on the other side of a door that I can never cross, and that door is retirement. Today I'm having returning guest Kelsey Hightower, who is no longer at Google. In fact, is no longer working and has joined the ranks of the gloriously retired. Welcome back, and what's it like?Kelsey: I'm happy to be here. I think retirement is just like work in some ways: you have to learn how to do it. A lot of people have no practice in their adult life what to do with all of their time. We have small dabs in it, like, you get the weekend off, depending on what your work, but you never have enough time to kind of unwind and get into something else. So, I'm being honest with myself. It's going to be a learning curve, what to do with that much time.You're probably still going to do work, but it's going to be a different type of work than you're used to. And so, that's where I am. 30 days into this, I'm in that learning mode, I'm on-the-job training.Corey: What's harder than you expected?Kelsey: It's not the hard part because I think mentally I've been preparing for, like, the last ten years, being a minimalist, learning how to kind of live within my means, learn to appreciate things that are just not work-related or status symbols. And so, to me, it felt like a smooth transition because I started to value my time more than anything else, right? Just waking up the next day became valuable to me. Spending time in the moment, right, you go to these conferences, there's, like, 10,000 people, but you learn to value those one-on-one encounters, those one-off, kind of, let's just go grab lunch situations. So, to me, retirement just makes more room for that, right? I no longer have this calendar that is super full, so I think for me, it was a nice transition in terms of getting more of that valuable time back.Corey: It seems to me that you're in a similar position to the one that I find myself in where the job that you were doing and I still am is tied, more or less, to a sense of identity as opposed to a particular task or particular role that you fill. You were Kelsey Hightower. That was a complete sentence. People didn't necessarily need to hear the rest of what you were working on or what you were going to be talking about at a given conference or whatnot. So, it seemed, at least from the outside, that an awful lot of what you did was quite simply who you were. Do you feel that your sense of identity has changed?Kelsey: So, I think when you have that much influence, when you have that much reputation, the words you say travel further, they tend to come with a little bit more respect, and so when you're working with a team on new product, and you say, “Hey, I think we should change some things.” And when they hear those words coming from someone that they trust or has a name that is attached to reputation, you tend to be able to make a lot of impact with very few words. But what you also find is that no matter what you get involved in—configuration management, distributed systems, serverless, working with customers—it all is helped and aided by the reputation that you bring into that line of work. And so yes, who you are matters, but one thing that I think helped me, kind of greatly, people are paying attention maybe to the last eight years of my career: containers, Kubernetes, but my career stretches back to the converting COBOL into Python days; the dawn of DevOps, Puppet, Chef, and Ansible; the Golang appearance and every tool being rewritten from Ruby to Golang; the Docker era.And so, my identity has stayed with me throughout those transitions. And so, it was very easy for me to walk away from that thing because I've done it three or four times before in the past, so I know who I am. I've never had, like, a Twitter bio that said, “Company X. X person from company X.” I've learned long ago to just decouple who I am from my current employer because that is always subject to change.Corey: I was fortunate enough to not find myself in the public eye until I owned my own company. But I definitely remember times in my previous incarnations where I was, “Oh, today I'm working at this company,” and I believed—usually inaccurately—that this was it. This was where I really found my niche. And then surprise I'm not there anymore six months later for, either their decision, my decision, or mutual agreement. And I was always hesitant about hanging a shingle out that was tied too tightly to any one employer.Even now, I was little worried about doing it when I went independent, just because well, what if it doesn't work? Well, what if, on some level? I think that there's an authenticity that you can bring with you—and you certainly have—where, for a long time now, whenever you say something, I take it seriously, and a lot of people do. It's not that you're unassailably correct, but I've never known you to say something you did not authentically believe in. And that is an opinion that is very broadly shared in this industry. So, if nothing else, you definitely were a terrific object lesson in speaking the truth, as you saw it.Kelsey: I think what you describe is one way that, whether you're an engineer doing QA, working in the sales department, when you can be honest with the team you're working with, when you can be honest with the customers you're selling into when you can be honest with the community you're part of, that's where the authenticity gets built, right? Companies, sometimes on the surface, you believe that they just want you to walk the party line, you know, they give you the lines and you just read them verbatim and you're doing your part. To be honest, you can do that with the website. You can do that with a well-placed ad in the search queries.What people are actually looking for are real people with real experiences, sharing not just fact, but I think when you mix kind of fact and opinion, you get this level of authenticity that you can't get just by pure strategic marketing. And so, having that leverage, I remember back in the day, people used to say, “I'm going to do the right thing and if it gets me fired, then that's just the way it's going to be. I don't want to go around doing the wrong thing because I'm scared I'm going to lose my job.” You want to find yourself in that situation where doing the right thing, is also the best thing for the company, and that's very rare, so when I've either had that opportunity or I've tried to create that opportunity and move from there.Corey: It resonates and it shows. I have never had a lot of respect for people who effectively are saying one thing today and another thing the next week based upon which way they think that the winds are blowing. But there's also something to be said for being able and willing to publicly recant things you have said previously as technology evolves, as your perspective evolves and, in light of new information, I'm now going to change my perspective on something. I've done that already with multi-cloud, for example. I thought it was ridiculous when I heard about it. But there are also expressions of it that basically every company is using, including my own. And it's a nuanced area. Where I find it challenging is when you see a lot of these perspectives that people are espousing that just so happen to deeply align with where their paycheck comes from any given week. That doesn't ring quite as true to me.Kelsey: Yeah, most companies actually don't know how to deal with it either. And now there has been times at any number of companies where my authentic opinion that I put out there is against party line. And you get those emails from directors and VPs. Like, “Hey, I thought we all agree to think this way or to at least say this.” And that's where you have to kind of have that moment of clarity and say, “Listen, that is undeniably wrong. It's so wrong in fact that if you say this in public, whether a small setting or large setting, you are going to instantly lose credibility going forward for yourself. Forget the company for a moment. There's going to be a situation where you will no longer be effective in your job because all of your authenticity is now gone. And so, what I'm trying to do and tell you is don't do that. You're better off saying nothing.”But if you go out there, and you're telling what is obviously misinformation or isn't accurate, people are not dumb. They're going to see through it and you will be classified as a person not to listen to. And so, I think a lot of people struggle with that because they believe that enterprise's consensus should also be theirs.Corey: An argument that I made—we'll call it a prediction—four-and-a-half years ago, was that in five years, nobody would really care about Kubernetes. And people misunderstood that initially, and I've clarified since repeatedly that I'm not suggesting it's going away: “Oh, turns out that was just a ridiculous fever dream and we're all going back to running bare metal with our hands again,” but rather that it would slip below the surface-level of awareness. And I don't know that I got the timing quite right on that, I think it's going to depend on the company and the culture that you find yourself in. But increasingly, when there's an application to run, it's easy to ask someone just, “Oh, great. Where's the Kubernetes cluster live so we can throw this on there and just add it to the rest of the pile?”That is sort of what I was seeing. My intention with that was not purely just to be controversial, as much fun as that might be, but also to act as a bit of a warning, where I've known too many people who let their identities become inextricably tangled with the technology. But technologies rise and fall, and at some point—like, you talk about configuration management days; I learned to speak publicly as a traveling trainer for Puppet. I wrote part of SaltStack once upon a time. But it was clear that that was not the direction the industry was going, so it was time to find something else to focus on. And I fear for people who don't keep an awareness or their feet underneath them and pay attention to broader market trends.Kelsey: Yeah, I think whenever I was personally caught up in linking my identity to technology, like, “I'm a Rubyist,” right?“, I'm a Puppeteer,” and you wear those names proudly. But I remember just thinking to myself, like, “You have to take a step back. What's more important, you or the technology?” And at some point, I realized, like, it's me, that is more important, right? Like, my independent thinking on this, my independent experience with this is far more important than the success of this thing.But also, I think there's a component there. Like when you talked about Kubernetes, you know, maybe being less relevant in five years, there's two things there. One is the success of all infrastructure things equals irrelevancy. When flights don't crash, when bridges just work, you do not think about them. You just use them because they're so stable and they become very boring. That is the success criteria.Corey: Utilities. No one's wondering if the faucet's going to work when they turn it on in the morning.Kelsey: Yeah. So, you know, there's a couple of ways to look at your statement. One is, you believe Kubernetes is on the trajectory that it's going to stabilize itself and hit that success criteria, and then it will be irrelevant. Or there's another part of the irrelevancy where something else comes along and replaces that thing, right? I think Cloud Foundry and Mesos are two good examples of Kubernetes coming along and stealing all of the attention from that because those particular products never gained that mass adoption. Maybe they got to the stable part, but they never got to the mass adoption part. So, I think when it comes to infrastructure, it's going to be irrelevant. It's just what side of that [laugh] coin do you land on?Corey: It's similar to folks who used to have to work at a variety of different companies on very specific Linux kernel subsystems because everyone had to care because there were significant performance impacts. Time went on and now there's still a few of those people that very much need to care, but for the rest of us, it is below the level of things that we have to care about. For me, the signs of the unsustainability were, oh, you can run Kubernetes effectively in production? That's a minimum of a quarter-million dollars a year in comp or up in some cases. Not every company is going to be able to field a team of those people and still remain a going concern in business. Nor frankly, should they have to.Kelsey: I'm going to pull on that thread a little bit because it's about—we're hitting that ten-year mark of Kubernetes. So, when Kubernetes comes out, why were people drawn to it, right? Why did it even get the time of day to begin with? And I think Docker kind of opened Pandora's box there. This idea of Chef, Puppet, Ansible, ten thousand package managers, and honestly, that trajectory was going to continue forever and it was helping no one. It was literally people doing duplicate work depending on the operating system you're dealing with and we were wasting time copying bits to servers—literally—in a very glorified way.So, Docker comes along and gives us this nicer, better abstraction, but it has gaps. It has no orchestration. It's literally this thing where now we've unified the packaging situation, we've learned a lot from Red Hat, YUM, Debian, and the various package repo combinations out there and so we made this universal thing. Great. We also learned a little bit about orchestration through brute force, bash scripts, config management, you name it, and so we serialized that all into this thing we call Kubernetes.It's pretty simple on the surface, but it was probably never worthy of such fanfare, right? But I think a lot of people were relieved that now we finally commoditized this expertise that the Googles, the Facebooks of the world had, right, building these systems that can copy bits to other systems very fast. There you go. We've gotten that piece. But I think what the market actually wants is in the mobile space, if you want to ship software to 300 million people that you don't even know, you can do it with the app store.There's this appetite that the boring stuff should be easy. Let's Encrypt has made SSL certificates beyond easy. It's just so easy to do the right thing. And I think for this problem we call deployments—you know, shipping apps around—at some point we have to get to a point where that is just crazy easy. And it still isn't.So, I think some of the frustration people express ten years later, they're realizing that they're trying to recreate a Rube Goldberg machine with Kubernetes is the base element and we still haven't understood that this whole thing needs to simplify, not ten thousand new pieces so you can build your own adventure.Corey: It's the idea almost of what I'm seeing AWS go through, and to some extent, its large competitors. But building anything on top of AWS from scratch these days is still reminiscent of going to Home Depot—or any hardware store—and walking up and down the aisles and getting all the different components to piece together what you want. Sometimes just want to buy something from Target that's already assembled and you have to do all of that work. I'm not saying there isn't value to having a Home Depot down the street, but it's also not the panacea that solves for all use cases. An awful lot of customers just want to get the job done and I feel that if we cling too tightly to how things used to be, we lose it.Kelsey: I'm going to tell you, being in the cloud business for almost eight years, it's the customers that create this. Now, I'm not blaming the customer, but when you start dealing with thousands of customers with tons of money, you end up in a very different situation. You can have one customer willing to pay you a billion dollars a year and they will dictate things that apply to no one else. “We want this particular set of features that only we will use.” And for a billion bucks a year times ten years, it's probably worth from a business standpoint to add that feature.Now, do this times 500 customers, each major provider. What you end up with is a cloud console that is unbearable, right? Because they also want these things to be first-class citizens. There's always smaller companies trying to mimic larger peers in their segment that you just end up in that chaos machine of unbound features forever. I don't know how to stop it. Unless you really come out maybe more Apple style and you tell people, “This is the one and only true way to do things and if you don't like it, you have to go find an alternative.” The cloud business, I think, still deals with the, “If you have a large payment, we will build it.”Corey: I think that that is a perspective that is not appreciated until you've been in the position of watching how large enterprises really interact with each other. Because it's, “Well, what customer the world is asking for yet another way to run containers?” “Uh, this specific one and their constraints are valid.” Every time I think I've seen everything there is to see in the world of cloud, I just have to go talk to one more customer and I'm learning something new. It's inevitable.I just wish that there was a better way to explain some of this to newcomers, when they're looking at, “Oh, I'm going to learn how this cloud thing works. Oh, my stars, look at how many services there are.” And then they wind up getting lost with analysis paralysis, and every time they get started and ask someone for help, they're pushed in a completely different direction and you keep spinning your wheels getting told to start over time and time again when any of these things can be made to work. But getting there is often harder than it really should be.Kelsey: Yeah. I mean, I think a lot of people don't realize how far you can get with, like, three VMs, a load balancer, and Postgres. My guess is you can probably build pretty much any clone of any service we use today with at least 1 million customers. Most people never reached that level—I don't even want to say the word scale—but that blueprint is there and most people will probably be better served by that level of simplicity than trying to mimic the behaviors of large customers—or large companies—with these elaborate use cases. I don't think they understand the context there. A lot of that stuff is baggage. It's not [laugh] even, like, best-of-breed or great design. It's like happenstance from 20 years of trying to buy everything that's been sold to you.Corey: I agree with that idea wholeheartedly. I was surprising someone the other day when I said that if you were to give me a task of getting some random application up and running by tomorrow, I do a traditional three-tier architecture, some virtual machines, a load balancer, and a database service. And is that the way that all the cool kids are doing it today? Well, they're not talking about it, but mostly. But the point is, is that it's what I know, it's where my background is, and the thing you already know when you're trying to solve a new problem is incredibly helpful, rather than trying to learn everything along that new path that you're forging down. Is that architecture the best approach? No, but it's perfectly sufficient for an awful lot of stuff.Kelsey: Yeah. And so, I mean, look, I've benefited my whole career from people fantasizing about [laugh] infrastructure—Corey: [laugh].Kelsey: And the truth is that in 2023, this stuff is so powerful that you can do almost anything you want to do with the simplest architecture that's available to us. The three-tier architecture has actually gotten better over the years. I think people are forgotten: CPUs are faster, RAM is much bigger quantities, the networks are faster, right, these databases can store more data than ever. It's so good to learn the fundamentals, start there, and worst case, you have a sound architecture people can reason about, and then you can go jump into the deep end, once you learn how to swim.Corey: I think that people would be depressed to understand just how much the common case for the value that Kubernetes brings is, “Oh yeah, now we can lose a drive or a server and the application stays up.” It feels like it's a bit overkill for that one somewhat paltry use case, but that problem has been hounding companies for decades.Kelsey: Yeah, I think at some point, the whole ‘SSH is my only interface into these kinds of systems,' that's a little low level, that's a little bare bones, and there will probably be a feature now where we start to have this not Infrastructure as Code, not cloud where we put infrastructure behind APIs and you pay per use, but I think what Kubernetes hints at is a future where you have APIs that do something. Right now the APIs give you pieces so you can assemble things. In the future, the APIs will just do something, “Run this app. I need it to be available and here's my money budget, my security budget, and reliability budget.” And then that thing will say, “Okay, we know how to do that, and here's roughly what is going to cost.”And I think that's what people actually want because that's how requests actually come down from humans, right? We say, “We want this app or this game to be played by millions of people from Australia to New York.” And then for a person with experience, that means something. You kind of know what architecture you need for that, you know what pieces that need to go there. So, we're just moving into a realm where we're going to have APIs that do things all of a sudden.And so, Kubernetes is the warm-up to that era. And that's why I think that transition is a little rough because it leaks the pieces part, so where you can kind of build all the pieces that you want. But we know what's coming. Serverless also hints at this. But that's what people should be looking for: APIs that actually do something.Corey: This episode is sponsored in part by Panoptica.  Panoptica simplifies container deployment, monitoring, and security, protecting the entire application stack from build to runtime. Scalable across clusters and multi-cloud environments, Panoptica secures containers, serverless APIs, and Kubernetes with a unified view, reducing operational complexity and promoting collaboration by integrating with commonly used developer, SRE, and SecOps tools. Panoptica ensures compliance with regulatory mandates and CIS benchmarks for best practice conformity. Privacy teams can monitor API traffic and identify sensitive data, while identifying open-source components vulnerable to attacks that require patching. Proactively addressing security issues with Panoptica allows businesses to focus on mitigating critical risks and protecting their interests. Learn more about Panoptica today at panoptica.app.Corey: You started the show by talking about how your career began with translating COBOL into Python. I firmly believe someone starting their career today listening to this could absolutely find that by the time their career starts drawing to their own close, that Kubernetes is right in there as far as sounding like the deprecated thing that no one really talks about or thinks about anymore. And I hope so. I want the future to be brighter than the past. I want getting a business or getting software together in a way that helps people to not require the amount of, “First, spend six weeks at a boot camp,” or, “Learn how to write just enough code that you can wind up getting funding and then have it torn apart.”What's the drag-and-drop story? What's the describe the application to a robot and it builds it for you? I'm optimistic about the future of infrastructure, just because based upon its power to potentially make reliability and scale available to folks who have no idea of what's involved with that. That's kind of the point. That's the end game of having won this space.Kelsey: Well, you know what? Kubernetes is providing the metadata to make that possible, right? Like in the early days, people were writing one-off scripts or, you know, writing little for loops to get things in the right place. And then we get config management that kind of formalizes that, but it still had no metadata, right? You'd have things like Puppet report information.But in the world of, like, Kubernetes, or any cloud provider, now you get semantic meaning. “This app needs this volume with this much space with this much memory, I need three of these behind this load balancer with these protocols enabled.” There is now so much metadata about applications, their life cycles, and how they work that if you were to design a new system, you can actually use that data to craft a much better API that made a lot of this boilerplate the defaults. Oh, that's a web application. You do not need to specify all of this boilerplate. Now, we can give you much better nouns and verbs to describe what needs to happen.So, I think this is that transition as all the new people coming up, they're going to be dealing with semantic meaning to infrastructure, where we were dealing with, like, tribal knowledge and intuition, right? “Run this script, pipe it to this thing, and then this should happen. And if it doesn't, run the script again with this flag.” Versus, “Oh, here's the semantic meaning to a working system.” That's a game-changer.Corey: One other topic I wanted to ask you about—I've it's been on my list of things to bring up the next time I ran into you and then you went ahead and retired, making it harder to run into you. But a little while back, I was at a tech conference and someone gave a demo, and it didn't go as well as they had hoped. And a few of us were talking about it afterwards. We've all been speakers, we've all lived that life. Zero shade.But someone brought you up in particular—unprompted; your legend does precede you—and the phrase that they used was that Kelsey's demos were always picture-perfect. He was so lucky with how the demos worked out. And I just have to ask—because you don't strike me as someone who is not careful, particularly when all eyes are upon you—and real experts make things look easy, did you have demos periodically go wrong that the audience just didn't see going wrong along the way? Or did you just actually YOLO all of your demos and got super lucky every single time for the last eight years?Kelsey: There was a musician who said, “Hey, your demos are like jazz. You improvise the whole thing.” There's no script, there's no video. The way I look at the demo is, like, you got this instrument, the command prompt, and the web browser. You can do whatever you want with them.Now, I have working code. I wrote the code, I wrote the deployment scenarios, I delete it all and I put it all back. And so, I know how it's supposed to work from the ground up. And so, what that means is if anything goes wrong, I can improvise. I could go into fixing the code. I can go into doing a redeploy.And I'll give you one good example. The first time Kubernetes came out, there was this small meetup in San Francisco with just the core contributors, right? So, there is no community yet, there's no conference yet, just people hacking on Kubernetes. And so, we decided, we're going to have the first Kubernetes meetup. And everyone got, like, six, seven minutes, max. That's it. You got to move.And so, I was like, “Hey, I noticed that in the lineup, there is no ‘What is Kubernetes?' talk. We're just getting into these nuts and bolts and I don't think that's fair to the people that will be watching this for the first time.” And I said, “All right, Kelsey, you should give maybe an intro to what it is.” I was like, “You know what I'll do? I'm going to build a Kubernetes cluster from the ground up, starting with VMs on my laptop.”And I'm in it and I'm feeling confident. So, confidence is the part that makes it look good, right? Where you're confident in the commands you type. One thing I learned to do is just use your history, just hit the up arrow instead of trying to copy all these things out. So, you hit the up arrow, you find the right command and you talk through it and no one looks at what's happening. You're cycling through the history.Or you have multiple tabs where you know the next up arrow is the right history. So, you give yourself shortcuts. And so, I'm halfway through this demo. We got three minutes left, and it doesn't work. Like, VMware is doing something weird on my laptop and there's a guy calling me off stage, like, “Hey, that's it. Cut it now. You're done.”I'm like, “Oh, nope. Thou shalt not go out like this.” It's time to improvise. And so, I said, “Hey, who wants to see me finish this?” And now everyone is locked in. It's dead silent. And I blow the whole thing away. I bring up the VMs, I [pixie 00:28:20] boot, I installed the kubelet, I install Docker. And everyone's clapping. And it's up, it's going, and I say, “Now, if all of this works, we run this command and it should start running the app.” And I do kubectl apply-f and it comes up and the place goes crazy.And I had more to the demo. But you stop. You've gotten the point across, right? This is what Kubernetes is, here's how it works, and look how you do it from scratch. And I remember saying, “And that's the end of my presentation.” You need to know when to stop, you need to know when to pivot, and you need to have confidence that it's supposed to work, and if you've seen it work a couple of times, your confidence is unshaken.And when I walked off that stage, I remember someone from Red Hat was like—Clayton Coleman; that's his name—Clayton Coleman walked up to me and said, “You planned that. You planned it to fail just like that, so you can show people how to go from scratch all the way up. That was brilliant.” And I was like, “Sure. That's exactly what I did.”Corey: “Yeah, I meant to do that.” I like that approach. I found there's always things I have to plan for in demos. For example, I can never count on having solid WiFi from a conference hall. The show has to go on. It's, okay, the WiFi doesn't work. I've at one point had to give a talk where the projector just wasn't working to a bunch of students. So okay, close the laptop. We're turning this into a bunch of question-and-answer sessions, and it was one of the better talks I've ever given.But the alternative is getting stuck in how you think a talk absolutely needs to go. Now, keynotes are a little harder where everything has been scripted and choreographed and at that point, I've had multiple fallbacks for demos that I've had to switch between. And people never noticed I was doing it for that exact reason. But it takes work to look polished.Kelsey: I will tell you that the last Next keynote I gave was completely irresponsible. No dry runs, no rehearsals, no table reads, no speaker notes. And I think there were 30,000 people at that particular Next. And Diane Greene was still CEO, and I remember when marketing was like, “Yo, at least a backup recording.” I was like, “Nah, I don't have anything.”And that demo was extensive. I mean, I was building an app from scratch, starting with Postgres, adding the schema, building an app, deploying the app. And something went wrong halfway. And there's this joke that I came up with just to pass over the time, they gave me a new Chromebook to do the demo. And so, it's not mine, so none of the default settings were there, I was getting pop-ups all over the place.And I came up with this joke on the way to the conference. I was like, “You know what'd be cool? When I show off the serverless stuff, I would just copy the code from Stack Overflow. That'd be like a really cool joke to say this is what senior engineers do.” And I go to Stack Overflow and it's getting all of these pop-ups and my mouse couldn't highlight the text.So, I'm sitting there like a deer in headlights in front of all of these people and I'm looking down, and marketing is, like, “This is what… this is what we're talking about.” And so, I'm like, “Man do I have to end this thing here?” And I remember I kept trying, I kept trying, and came to me. Once the mouse finally got in there and I cleared up all the popups, I just came up with this joke. I said, “Good developers copy.” And I switched over to my terminal and I took the text from Stack Overflow and I said, “Great developers paste,” and the whole room start laughing.And I had them back. And we kept going and continued. And at the end, there was like this Google Assistant, and when it was finished, I said, “Thank you,” to the Google Assistant and it was talking back through the live system. And it said, “I got to admit, that was kind of dope.” So, I go to the back and Diane Greene walks back there—the CEO of Google Cloud—and she pats me on the shoulder. “Kelsey, that was dope.”But it was the thrill because I had as much thrill as the people watching it. So, in real-time, I was going through all these emotions. But I think people forget, the demo is supposed to convey something. The demo is supposed to tell some story. And I've seen people overdo their demos with way too much code, way too many commands, almost if they're trying to show off their expertise versus telling a story. And so, when I think about the demo, it has to complement the entire narrative. And so, sometimes you don't need as many commands, you don't need as much code. You can keep things simple and that gives you a lot more ins and outs in case something does go crazy.Corey: And I think the key takeaway here that so many people lose sight of is you have to know the material well enough that whatever happens, well, things don't always go the way I planned during the day, either, and talking through that is something that I think serves as a good example. It feels like a bit more of a challenge when you're trying to demo something that a company is trying to sell someone, “Oh, yeah, it didn't work. But that's okay.” But I'm still reminded by probably one of the best conference demo fails I've ever seen on video. One day, someone was attempting to do a talk that hit Amazon S3 and it didn't work.And the audience started shouting at him that yeah, S3 is down right now. Because that was the big day that S3 took a nap for four hours. It was one of those foundational things you'd should never stop to consider. Like, well, what if the internet doesn't work tomorrow when I'm doing my demo? That's a tough one to work around. But rough timing.Kelsey: [breathy sound]Corey: He nailed the rest of the talk, though. You keep going. That's the thing that people miss. They get stuck in the demo that isn't working, they expect the audience knows as much as they do about what's supposed to happen next. You're the one up there telling a story. People forget it's storytelling.Kelsey: Now, I will be remiss to say, I know that the demo gods have been on my side for, like, ten, maybe fifteen years solid. So, I retired from doing live demos. This is why I just don't do them anymore. I know I'm overdue as an understatement. But the thing I've learned though, is that what I found more impressive than the live demo is to be able to convey the same narratives through story alone. No slides. No demo. Nothing. But you can still make people feel where you would try to go with that live demo.And it's insanely hard, especially for technologies people have never seen before. But that's that new challenge that I kind of set up for myself. So, if you see me at a keynote and you've noticed why I've been choosing these fireside chats, it's mainly because I'm also trying to increase my ability to share narrative, technical concepts, but now in a new form. So, this new storytelling format through the fireside chat has been my substitute for the live demo, normally because I think sometimes, unless there's something really to show that people haven't seen before, the live demo isn't as powerful to me. Once the thing is kind of known… the live demo is kind of more of the same. So, I think they really work well when people literally have never seen the thing before, but outside of that, I think you can kind of move on to, like, real-life scenarios and narratives that help people understand the fundamentals and the philosophy behind the tech.Corey: An awful lot of tools and tech that we use on a day-to-day basis as well are thankfully optimized for the people using them and the ergonomics of going about your day. That is orthogonal, in my experience, to looking very impressive on stage. It's the rare company that can have a product that not only works well but also presents well. And that is something I don't tend to index on when I'm selecting a tool to do something with. So, it's always a question of how can I make this more visually entertaining? For while I got out of doing demos entirely, just because talking about things that have more staying power than a screenshot that is going to wind up being irrelevant the next week when they decide to redo the console for some service yet again.Kelsey: But you know what? That was my secret to doing software products and projects. When I was at CoreOS, we used to have these meetups we would used to do every two weeks or so. So, when we were building things like etcd, Fleet was a container management platform that came before Kubernetes, we would always run through them as a user, start install them, use them, and ask how does it feel? These command line flags, they don't feel right. This isn't a narrative you can present with the software alone.But once we could, then the meetups were that much more engaging. Like hey, have you ever tried to distribute configuration to, like, a thousand servers? It's insanely hard. Here's how you do with Puppet. But now I'm going to show you how you do with etcd. And then the narrative will kind of take care of itself because the tool was positioned behind what people would actually do with it versus what the tool could do by itself.Corey: I think that's the missing piece that most marketing doesn't seem to quite grasp is, they talk about the tool and how awesome it is, but that's why I love customer demos so much. They're showing us how they use a tool to solve a real-world problem. And honestly, from my snarky side of the world and the attendant perspective there, I can make an awful lot of fun about basically anything a company decides to show me, but put a customer on stage talking about how whatever they've built is solving a real-world problem for them, that's the point where I generally shut up and listen because I'm going to learn something about a real-world story. Because you don't generally get to tell customers to go on stage and just make up a story that makes us sound good, and have it come off with any sense of reality whatsoever. I haven't seen that one happen yet, but I'm sure it's out there somewhere.Kelsey: I don't know how many founders or people building companies listen in to your podcast, but this is right now, I think the number one problem that especially venture-backed startups have. They tend to have great technology—maybe it's based off some open-source project—with tons of users who just know how that tool works, it's just an ingredient into what they're already trying to do. But that isn't going to ever be your entire customer base. Soon, you'll deal with customers who don't understand the thing you have and they need more than technology, right? They need a product.And most of these companies struggle painting that picture. Here's what you can do with it. Or here's what you can't do now, but you will be able to do if you were to use this. And since they are missing that, a lot of these companies, they produce a lot of code, they ship a lot of open-source stuff, they raise a lot of capital, and then it just goes away, it fades out over time because they can bring on no newcomers. The people who need help the most, they don't have a narrative for them, and so therefore, they're just hoping that the people who have all the skills in the world, the early adopters, but unfortunately, those people are tend to be the ones that don't actually pay. They just kind of do it themselves. It's the people who need the most help.Corey: How do we monetize the bleeding edge of adoption? In many cases you don't. They become your community if you don't hug them to death first.Kelsey: Exactly.Corey: Ugh. None of this is easy. I really want to thank you for taking the time to catch up and talk about how you seen the remains of a career well spent, and now you're going off into that glorious sunset. But I have a sneaking suspicion you'll still be around. Where should people go if they want to follow up on what you're up to these days?Kelsey: Right now I still use… I'm going to keep calling it Twitter.Corey: I agree.Kelsey: I kind of use that for my real-time interactions. And I'm still attending conferences, doing fireside chats, and just meeting people on those conference floors. But that's what where I'll be for now. So yeah, I'll still be around, but maybe not as deep. And I'll be spending more time just doing normal life stuff, maybe less building software.Corey: And we will, of course, put a link to that in the show notes. Thank you so much for taking the time to catch up and share your reflections on how the industry is progressing.Kelsey: Awesome. Thanks for having me, Corey.Corey: Kelsey Hightower, now gloriously retired. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with an angry comment that you're going to type on stage as part of a conference talk, and then accidentally typo all over yourself while you're doing it.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.

Kendall Miller is the Co-Founder and COO of CTO Lunches, a network of engineering leaders to get trusted advice and connections. The first half of the conversation with host Victoria Guido and special guest host, Joe Ferris, CTO of thoughtbot revolves around the use, adoption, and growth of Kubernetes within the technology industry. The discussion explores Kubernetes' history, influence, and its comparison with other platforms like Heroku and WordPress, emphasizing its adaptability and potential. The second half focuses on more practical aspects of Kubernetes, including its adoption and scalability. It centers on the appropriateness of adopting Kubernetes for different projects and how it can future-proof infrastructure. The importance of translating technical language into business speak is emphasized to influence executives and others in the decision-making process and Kendall also discuss communication and empathy in tech, particularly the skill of framing questions and understanding others' emotional states. __ CTO Lunches (http://ctolunches.com/) Follow CTO Lunches on LinkedIn (https://www.linkedin.com/company/ctolunches/) or Twitter (https://twitter.com/cto_lunches). Follow Kendall Miller on LinkedIn (https://www.linkedin.com/in/kendallamiller/). Follow thoughtbot on Twitter (https://twitter.com/thoughtbot) or LinkedIn (https://www.linkedin.com/company/150727/). Become a Sponsor (https://thoughtbot.com/sponsorship) of Giant Robots! Transcript: VICTORIA: This is the Giant Robots Smashing Into Other Giant Robots Podcast, where we explore the design, development, and business of great products. I'm your host, Victoria Guido. And with me today is Kendall Miller, Co-Founder and COO of CTO Lunches, a network of engineering leaders to get trusted advice and connections. Kendall, thank you for joining me. KENDALL: Thanks for having me. I'm excited. VICTORIA: And today, we have a special guest host, Joe Ferris, CTO of thoughtbot. Joe, thank you for joining us. JOE: Hello there. Thank you for having me. KENDALL: Hi, Joe. Thanks for being here. It's exciting. VICTORIA: Yes. It's so exciting. I think this is going to be a great episode. So, Kendall, I met you at a San Diego CTO lunch recently, and I know that's not the only thing that you do. So, you're also an advisor, a board member, and CXO. So, maybe tell us a little bit more about your background. KENDALL: Gosh, my background is complicated. I've been involved in tech for a very long time. In college, I worked for a company that started Twitter about five years too soon, and then worked in the nonprofit space in China for ten years, then came back, got back involved in tech. Today, I'm usually the business guy. So, when technical founders start technical products and want help turning them into successful technical businesses, that's when they call me. So, I have the technical background. I have never been paid to write code, which is probably a good thing. But I can hang in the technical conversations for the most part, but I'm much more interested in the business side and the people leadership side of business. So that tends to be where I play. Every organization hires me to do something different. VICTORIA: Thank you for that. And I'm just curious about the CTO Lunches. Just tell me a little bit more about that. And what's the idea behind it that led you to co-found it? KENDALL: CTO Lunches has actually been around for about eight years. And I didn't start the initial incarnation of it. It was two people that got us started, and I was trying to hire one of them; one thing led to another. Actually, originally, they did not want me to join. I think, at the time, my title was COO at a company that I was working with. About six months later, I took over engineering as VP of engineering, and then they're like, you can join the group now. We're less strict about that [laughs] now. Although it is highly focused on senior engineering leaders, it's not exclusively CTOs. But the group's been in place for a very long time, just intended as a place to network, have conversation with people who are in that senior-most technical position at technical organization. So, the CTO role is a lonely role. CTOs get fired all the time. There's not a technical person at the company that doesn't think they can do the job better than them. So, the CTO is always getting feedback. You're doing this wrong. The trade-offs you're making are wrong. This isn't going where it should be going. We should automate that. Why haven't we automated that? We should switch to this other tool. I've used it before; it's 100 times better. Joe, let me know if I'm getting any of this wrong. But that's the experience that I've had. Having a place where people can get together and, you know, half the time just complain to each other, hey, this is hard, is really why the networking group exists. So, it's a listserv. And there are local lunches that started in Boulder, Colorado. It's gotten pretty global. About a year ago, a little over a year ago, I was talking with one of the people who'd gotten it started. I've been involved in the Denver chapter for most of those eight years. And I was suggesting to him that he change a few things about it, to monetize it so that he could invest in it further. And he came back a few months later and said, "I want to take your advice and do this, but I want you to come do it with me." So, we founded the company officially...I think in December is when paperwork went into place. And we started investing in it a little bit more heavily. I was living in Europe last year, so we went and put on lunches in Paris, and Lisbon, and London, and, gosh, all over the place. I'm sure I'm missing some, Amsterdam. But there's been chapters all over the U.S. and a couple of other parts of the world for a long time. VICTORIA: That reflects my experience attending a CTO lunch. It's just very casual, like, just get together and eat food and talk about what you've worked on recently, issues you're having, just get ideas and make some friends. So, I really appreciated the group, and I'm going to personally plug the San Diego Chapter has picked up again. And we're meeting next Friday down in Del Mar. And we're going to be meeting on the last Friday of every month through October. So, I'm super excited to be a part of the group. And Joe, yeah, I'm curious about your perspective. As a CTO with thoughtbot, just what are your thoughts about that kind of thing? KENDALL: Yeah. How right am I about how lonely you are, Joe? JOE: [laughs] You know, I've been lonelier since we went remote. I used to work in the office, and I was a CTO, but also, I had lunch with people, which was nice. So, I'm lonelier. But yeah, I think everybody needs a group like that, like, senior developer therapy just to talk about your woes together, drown your sorrows. KENDALL: Well, I think years ago, I heard that CTOs are the most fired C-level executive. JOE: You're making me nervous now. KENDALL: [laughs] You've been there a long time, Joe. I know you've been there a long time. If you haven't been fired yet, you probably got a little while longer in you. This will be really awkward if it's published and you've already been fired. VICTORIA: We can always edit that out afterwards. [laughter] KENDALL: Yeah, no, I think it is a particularly lonely position. And, again, I think a lot of it is the average engineer in a technical company doesn't look at the COO or the CFO or even the CEO and think I could do that. But they're all looking at the CTO and thinking, what does that person do that I can't do? It's ridiculous because most of them would make terrible CTOs because it does require some of the business sense. Or, you know, right out of the gate, they might make terrible CTOs. It actually is quite a skill to be the most technical person and speak the business language. I mean, am I right about that, Joe? Like, was that hard for you to learn? JOE: Yeah, I definitely think...so, my background is also technical. I have a background in consulting. So, I always did a lot of metaprogramming, if you will. But making that transition to thinking about organizations that way, thinking about how all the other pieces play into it, was a pretty big step for me, even before I became a CTO as a consultant. KENDALL: Well, because you can't just chase the newest, hottest technology. You have to make business trade-offs. And not everything can be resume-driven development, right? Even if that technology over there is newer or hotter, it doesn't mean you have a business model that supports it. And it doesn't mean that migrating to it can be done, right? JOE: Yeah. I mean, even beyond choosing technologies, just choosing where to invest in your software stack, like, what needs to be reworked, what doesn't, and trying to explain those trade-offs, I think, is a rare skill. Being able to explain why something would be harder than something else when you're working with the leadership to prioritize a backlog it's a puzzle. KENDALL: Well, and I think when I'm in an executive conversation, and the CTO says, "Here's the thing that I think is the best decision technically, and I think it's the wrong decision for the business because of X, Y, or Z," I'm always super impressed, right? Like, this is the right technical solution for what we want. However, we shouldn't pursue that for business reasons right now. Maybe we can in six months, but right now, we need to prioritize this other thing. I don't know, that's always when I feel like, oh, this person knows what they're doing. JOE: There's nothing more dangerous to software than a bored developer. [laughs] One nice thing about being a consultant is that I don't have to invent problems to solve with technology at my company because sooner or later, I'll run across a company that has those problems, and I'll get to use that technology. But I think a lot of people are mostly happy...they might be happy in their role. They might be happy with our team. But they're very interested in whatever is hot right now, like machine learning, AI. And so, suddenly, that surreptitiously makes its way into the tech stack. And then, years later, it's somebody's problem to maintain. KENDALL: [laughs] Well, I have a specific memory of a firm in New York City that was, you know, this is relevant to y'all as thoughtbot is that, you know, at least historically, it was, to me, the premier Ruby on Rails consulting shop. I think that's still largely y'alls focus. Am I right about that? JOE: We still do a ton of Rails, yeah. KENDALL: Okay. Well, so this organization was all Ruby on Rails. It was a big organization. They had a very large customer base. And they hired a new CTO who came in, told everybody in the company they were stupid, laid off 70% of the engineering organization, and told the CEO he was going to completely rewrite the product from scratch in .NET, and he could do it in three weeks. And I'm pretty sure the business went under about three months later [laughs] because that was just so outrageously nuts to me. JOE: It's too bad he laid everybody off beforehand. I've been in that situation where somebody tells me, "I'm going to rewrite this. It'll be ready in three weeks." And I could fight with them and try and convince them they're wrong. But I feel like somebody who's approaching that with that attitude they're missing all of the nuance and context that would make it possible to explain to them why it's not going to work. And so, it's easier to just say, "You know, take the three weeks. I'll talk to you in three weeks." But if you've already laid off your development team, that's hard [laughs] to recover from. KENDALL: That's exactly right. VICTORIA: There's got to be a name for that kind of CTO who just wants to come in and blow everything up [laughs]. Yeah, so you spend a lot of time talking to different CTOs and doing this social networking aspect. I'm wondering if there's, like, patterns that you see. You've mentioned already one about just, like, the most often getting fired. [laughs] But what are the patterns you see, like, in challenges, and then what makes someone successful in that CTO role? KENDALL: Well, oh gosh, I have so many thoughts about this. First of all, I run into a couple of different categories of CTOs. There's a lot of people who come to CTO Lunches who are small company CTOs. I mean, it makes sense that there's a lot more small company CTOs than there are big company CTOs. But the small company CTO who maybe it's their first gig in the role or they're a serial CTO. There's the fractional CTOs that come that are doing it across several different organizations at the same time, and then there's the big company CTO who shows up. And honestly, all of their problems are very different. The thing that they have in common is even at a very large organization, in that position, they can make a decision that causes the company to go under. So, there is a significant amount of volatility in the amount of power that they wield. So, what's interesting about that is not everybody understands that. And so, first of all, there's the kind of CTO that just doesn't get that, and that doesn't matter if they're fractional, or a small company CTO, or a big company CTO. If they don't understand that, they're going to cause significant problems, right? Like the person I just mentioned who said, "I can just re-platform this in three weeks in .NET." There's that. I mean, I think, as with any senior leadership position, the comfort with volatility, the ability to know what to communicate down versus across and versus up, and then the ability to speak the business language. For everybody, the CFO's job is to communicate the financial needs alongside of the business leads, right? If the CFO's sole goal is to cut costs or make sure we're running as lean as possible, they're a bad CFO. But they're not as good of a CFO as the CFO who can say, "Hey, we're underspending right here. And I can look at the numbers and know we should invest more there. How can we invest more there and invest it well?" And it's the same thing for a technology executive to be able to look at the business context and communicate it back. And there are so many CTOs that I've worked with who they're the most technical person in the room, and they know it. And as a result, they're just a jerk to everyone around them, like, everything you did here was wrong. You know, that's where they fail. And so, if they can communicate the business needs, navigate the volatility, and support a team that's going to make decisions that aren't always the same decision they're going to make, they're going to be successful. Honestly, there's very, very few CTOs that I've met like that. People who are excited to meet you at work, excited to see you succeed, excited to see that you went and built a thing is great. I mean, the reason I was VP of engineering is the CTO that I was working with at the time...it's a terrible story. There was an engineer who had seen something that we were doing on repeat all the time and, in his spare time, spent about 40 hours outside of work, not during work hours, automating this task that we were doing regularly. And it was related to standing up a whole bunch of things in our standard infrastructure. He brings it to the CTO and says, "Look what I built." And the CTO, instead of saying, "Hey, this is incredible. Thank you. This is going to save us a bunch of time. Let's iterate on it. Here's some things I'd like to tweak. Can we bring it in this direction? Can we..." you know, whatever, said, "Why is this in Python? It should be in Ansible," something like that. I can't remember. And the engineer literally burst into tears. [laughs] JOE: Oh my God. KENDALL: [laughs] Well, I mean, yeah, it was like; literally, that's why the CTO stopped managing people that day. There's a lot of examples that I have like that. Joe, I appreciate that your response is, "Oh my God." Because I think there's a lot of people who'd be like, wait, what was wrong with that? Shouldn't it have been in Ansible? JOE: [laughs] Yeah, I've seen CTOs come into primarily two groups. One is the CTO who just tells, you know, like, they make the decisions, and they tell everybody what to do. They obviously don't have all of the information because you can't be in every room all the time. And the other is the CTO, who just wants to be one of the team members and doesn't make any decisions and tries to get people to make decisions collectively on their own without any particular guidance or structure. And finding that middle spot of, like, not just saying, "Hey, everything's in Ansible," allowing for the creativity and initiative, but also coalescing the group into a single direction, I think, is what makes a good CTO. KENDALL: Well, yeah, because the CTO does have to say no, sometimes, right? Like, the best product, people say, "No." Good CTOs say, "No." There is some amount of, hey, I need you to come to me with trade-offs about this. Why are you going to make that decision? And I'm sorry, you still didn't convince me, right? Like, I mean, those are appropriate things to say. But yeah, I'm with you on that. You said they fall into two categories. But you really mean the third and that middle ground. Is it easy for you to walk that middle ground, Joe? JOE: I wouldn't say it's easy. [laughs] KENDALL: Yeah. Well, I'm always nervous to say something. I'm doing well because I know there's a report out there that can point at every time I failed at it, right? So... MID-ROLL AD: Are you an entrepreneur or start-up founder looking to gain confidence in the way forward for your idea? At thoughtbot, we know you're tight on time and investment, which is why we've created targeted 1-hour remote workshops to help you develop a concrete plan for your product's next steps. Over four interactive sessions, we work with you on research, product design sprint, critical path, and presentation prep so that you and your team are better equipped with the skills and knowledge for success. Find out how we can help you move the needle at: tbot.io/entrepreneurs. VICTORIA: Yeah, what I'm getting from what you're saying, too, is this communication ability and not just, like, to communicate clearly but with a high level of empathy. So, if you say like, "Well, why is it in Python and not Ansible?" is different than being like, "What makes Python the best solution here?" Like, it's a different way to frame the question that could put someone on the defensive that just really requires, like, a high level of emotional intelligence. And also, if they've just worked, like, an 80-hour week, [laughs] I probably would maybe choose a different time to bring those questions up and notice that they have been burning the candle at both ends and prioritize getting them some rest. So, speaking of, like, communication and getting prioritization for [inaudible 15:34], especially on, like, infrastructure teams, maybe we could talk a little bit about Kubernetes, like, when that comes up as an appropriate solution, and how you talk about it with the business. KENDALL: My background with Kubernetes is long because a company that I still work with, Fairwinds, used to be called ReactiveOps, has been in the Kubernetes space for a very long time. I think we were one of the very first companies working with Kubernetes. It was coming up that people were running into the limits of something like Heroku, right? And I think it's Kelsey Hightower who said every company wants a PaaS. They just want the Paas that they built themselves. And that's really accurate. And I think Kubernetes isn't quite a framework for building your own PaaS or isn't quite a foundation where I think of a foundation for a house. Instead, it's more like rebar and cement and somebody saying, "Good luck, buddy." You know, you still have to know how to put the rebar and cement together to even make the foundation, but it is the building blocks that help get you to a custom-built PaaS. And it's become something that a lot of people have landed on as, you know, the broadly accepted way to build cloud-native infrastructure. The reason I've been in the Kubernetes space and the space that I see Kubernetes still filling is we need to standardize on something. We can choose a cloud provider's PaaS. We can choose a third-party PaaS, or we can standardize on something like Kubernetes. And even though we're not going to migrate from AWS to Azure, the flexibility that Kubernetes gives us as a broadly adopted pattern is going to give us some ability to be future-proofed in our infrastructure in a way that previous stacks were not, you know, it was Puppet, and it was Ansible. And it was SaltStack. And it was all Terraform all the time. I'm not saying those things don't exist anymore. I'm saying Kubernetes kind of has won that battle. Joe, since you're here and I know y'all are doing some Kubernetes work now at thoughtbot, I'm curious if you agree with that characterization. JOE: Yeah, I think that's true. I think it's the center for people to coalesce around. Like, for an effort in the industry to move forward, there needs to be some common language, some common ground. And I think Kubernetes struck the right balance of being abstract. So, you can use it in different environments but still making some decisions, so you don't have to make them all. And so, like, all of the things you had to do with containers like figuring out what your data solution is going to be, what your networking solution is going to be, Kubernetes didn't even really make those decisions. [laughs] They just made a platform where those decisions can be made in a common way. And that allowed the community and the ecosystem to grow. KENDALL: I mean, I think of it a lot like WordPress; you know, WordPress is hated by many. When WordPress came out, it was hot, right? And it was PHP, which everybody was super excited about at the time. Kubernetes is going to reach a point where it's as long in the tooth and terrible as people think WordPress is, but it has become the standard. And the advantage of the standard is you can use the not standard. You can go build a website in Jekyll instead of WordPress, and there's going to be some things that are nicer about Jekyll. But because WordPress is so broadly adopted, there's a plugin for everything. And I think that's where Kubernetes sits is because it's become so widely adopted everybody's building for it. Everybody's adapting for it. If you run into a problem, you're going to find somebody else out there who has that problem. In fact, I think of one organization that I know that was on HashiCorp's Nomad. And they said, "Actually, we think Nomad has better technology through and through. But we think we're the only company at this size and scale using Nomad. And so, when we run into a problem, we can't Google for it. There's no such thing as a plugin that exists to solve this. Nobody has ever run into this before on Nomad. But there's 100 companies dealing with the same problem in Kubernetes, and there's ten solutions." And I think that's the power that it brings. VICTORIA: So, it's not just a trend that CTOs are moving towards, you think. KENDALL: I mean, I think it's already won the battle and the hockey stick of adoption. We're still right at the very bottom of that tick-up because it takes people a long time to adapt new technology like this, especially in their infrastructure. It's a big migration, to move. So, I don't think it's the widely adopted infrastructure technology even yet. I think a lot of the biggest organizations are still running on things that predate Kubernetes. But I think it has won the battle, and it is winning the battle and is going to be the thing going forward, so yeah. JOE: I think it also has a lot of room to grow still. Like, there are other technologies that I used previously, like Docker, and they were a big step up from some of the things I was doing at the time. But you quickly hit the ceiling, or it was, like, I don't know where to go with this next. I don't know what else is going to happen. Whereas with Kubernetes, there are so many directions it can go in. Like, the serverless Kubernetes offerings that are starting to pop up are extremely interesting, where, you know, you don't actually maintain a cluster or anything. You just deploy things to this ethereal cluster that always exists. And so, that sort of combination of platform as a service, function as a service, Kubernetes, as that evolves, I think there are a lot of exciting things that have yet to come in the Kubernetes space. KENDALL: Well, so say more about that, Joe, because I've been going to KubeCon for a very long time, maybe...I don't know if it's 2016 or so when I first went. And it felt for a number of years...maybe those first four-ish years it was always the people at KubeCon were the, like, big dreamers and thinkers and, like, we're here to change the future of cloud infrastructure. And this is going places, and we're excited to be here and be a part of it. And here's what I'm going to do that changes the next thing. And I feel like now if I go to KubeCon, it's a lot of people from, you know, IBM and some big bank that are, like, deep sigh, well, I have to adopt Kubernetes. I need to know what the vendors are. What do you guys do, and how does this work? Can you please teach it to me? Because I'm being told by my boss, I have to do it. I don't see that excitement around Kubernetes anymore. The excitement I see is all around further up the stack, you know, things like Wasm, WebAssembly, or eBPF, the networking things and tracing things that are possible. Maybe that's further down the stack. I guess it depends on how you think about it, but different part of the stack. So, I'm curious, touching on the serverless components of Kubernetes; sure, I get that. And I do think, increasingly, the PaaSs of the future are all going to be Kubernetes-based, whether that's exposed or not. But where are the places that you think it's still going to go? Because I feel like it's already gotten boring, maybe in a positive way. But I don't see the excitement around it like I saw a few years back. And I'm curious what else you think is going to happen. JOE: Yeah, I mean, I don't think I disagree. I think Kubernetes itself, the core concept, is, like, it's still changing. But you're right that the excitement about Kubernetes existing has gone down because it's been there for a while. But I feel like the ecosystem is still growing pretty rapidly. Like, the things you mentioned, like Wasm and Istio, and all the tools in that ecosystem that continue to grow, is where I think the interesting things will happen. Like, it's created this new lower-level layer of abstraction that makes it possible to build concepts and technology that could not have existed before. KENDALL: Yeah, well, and I'm, you know, talking to people who are working really hard at making short-run ephemeral workloads work better on things like GPUs for the sake of AI, right? Like, I mean, there is some really interesting things happening, and people are doing this in Kubernetes. So, I get that. I agree with that. It is interesting that Kubernetes has become sort of the stable thing, and now it's about who can build the interesting add-ons. It's almost like, okay, we've built Half-Life. What is Counter-Strike going to look like? You know. That's a terrible (I'm aging myself.) example. But still. VICTORIA: I think it's interesting, I mean, to look at the size of the market for platform engineering right now. In 2022, was 4.8 billion, and it's estimated to be in 10 years $41 billion. So, there is this emerging trend of different platform engineering products, different abstractions on top of Kubernetes. And I wonder what advice you would have for a technical founder who's looking to build and solve some of these interesting issues in Kubernetes and create a business around it. KENDALL: Well, okay, let me clarify that question. Are you thinking, I'm a startup, and I need to build my infrastructure, and I'm going to choose Kubernetes. What advice do I need? Or are you thinking, I am founder, and I want to go build on the Kubernetes ecosystem. What advice do you have? VICTORIA: Now I want to know the answer to both. But my question was the second one to start. KENDALL: One of the things that is hard about the Kubernetes ecosystem is there's not a ton of companies that have made a whole bunch of money in Kubernetes because, as I said, I still think we're actually really early in the adoption curve. The kinds of companies that have adopted Kubernetes are the kinds of companies that don't spend lots and lots of money on an infrastructure. [laughs] They're the kinds of companies that are fast-moving, early adopters, or, you know, those first followers, and so they're under $100 million companies for the most part. Where the JP Morgans and Chase are running Kubernetes somewhere in their stack, but they haven't adopted it across the stack to need the biggest, best tools about it. So, the first piece of advice that I'd give is, be a little wary. It's still very early to the market. Maybe now is the time to build the thing. When ReactiveOps pivoted to Kubernetes, I think it was six months of having conversations with companies who were just, like, so excited about it, and this is definitely what we want to do. But nobody was doing it yet. You know, it was, we have, like, six solid months of just excitement and nobody actually pulling the trigger. And, you know, we were a little too early to that market. And that was just the people adopting it. So, I think there is some nervousness that cloud-native solutions the only people who are really making money in Kubernetes are named Amazon, Google, and Microsoft because it's the cloud providers that are making a ton off of it. Now, there's Rancher. There is StackPointCloud. There's a few others that have had big exits in this space. But I don't think it's actually as big of a booming economy as a lot of people think, in part because EKS is an incredibly amazing product. Like, eight years ago, the thing people paid us the most to do at ReactiveOps was just stand up Kubernetes because it was so stinking hard to just get it up and working. And now you click some buttons. Anybody can go do that. So, it's changed a lot, right? And I think be wary when you're entering that ecosystem. And then, my advice to the founder that's not building on the ecosystem but just looking to adopt a technology that's going to be a future-proofed infrastructure is just adopt one of the cloud-native platforms. And there are a whole bunch of sort of default best-in-class add-ons out there that you need to throw in. Don't adopt too many because then you have to maintain them forever. That's the easiest way to get started. You can figure out all the rest of it later. But if you go use EKS, or GKE, AKS, you can get started pretty easily and build something that is going to be future-proofed. I don't know, Joe; I'm curious if you disagree with any of that. JOE: Well, I think it's interesting to think about who's making money in Kubernetes. Like, I think there might not be as many companies who are doing only Kubernetes and Kubernetes-focused products that are massively successful. But I think because it has had a good amount of adoption and because it's easier to work with something that's standardized, it has helped companies sell things that they wanted to sell anyway. Like, all the Datadog, all the Scalas, the logging companies, they all have Kubernetes add-ons. And now everybody is paying Datadog [laughs] to have a dashboard for their Kubernetes cluster. I think they're making more money than they would have been without targeting the market. And so, I think that's really...if you want to get into the market, it's not, like, I'm going to build a Kubernetes product. It's if I'm building operations and an infrastructure product, I should definitely have it work with Kubernetes, and people will want to click and install it. KENDALL: So, to be clear, you know, one of the companies that I work with is called Axiom, and they play in the same, you know, monitoring, observability space as Datadog does. And part of what makes Kubernetes interesting in that space is in a microservice environment; there's so much happening. Where are problems being caused? We don't live in a day where I can just run my code, and it tells me that there's an unexpected semicolon on line 23, right? Like, that still happens. You're still doing those things. But this microservice talking to that microservice is where things tend to break down. Did I communicate this correctly? What was sent? What was received? Where did it break down? What was the latency? And if you were doing things in the old way back when you were standing it up with, say, Ansible, or Puppet, or something like that, and you were orchestrating all of these cloud virtual machines, you had to really work hard to instrument the tracing and logging and everything involved in order to track what was going on. Whereas that's one of the magic things about Kubernetes is with a few of the add-ons or some of the things out of the box with Kubernetes, it's a couple of clicks to get so, so much of the data and have insight into where things are going and what's going wrong. And so, I 100% agree with that. Kubernetes is generating a tremendous amount of data. And if you're a data company, it's really nice to have all that come in, and it helps them make money, helps the user of Kubernetes in that situation understand where problems are happening and breaking down. Yeah, there's definitely some network effects of what Kubernetes is doing in that. I completely agree. JOE: I think there are also some interesting companies, like, where they make...Emissary, Ambassador, and they have that sort of dual -- KENDALL: Komodor, is that -- JOE: Yeah, maybe. They have open source, but then they have a product. KENDALL: You're thinking of Ambassador Labs. JOE: Yeah. Ambassador Labs, yeah. I guess I don't really know how much money they're making. But I think that's a really interesting concept as people who make open-source things then make a well-supported product built around it. KENDALL: Sure. What's interesting is, I think in the VC world, at least right now, and it may pick up again, but post-Silicon Valley Bank nearly caving in, I think that the VC tolerance for, yeah, just go get a billion open-source adopters, and we'll figure out how to monetize later I think that the tolerance for that is a lot lower than it was even six months ago. JOE: Yeah, I think you have to have a dual model right from the beginning now. KENDALL: Yeah. Agreed. VICTORIA: You got to figure out how to make money on Kubernetes before you can. [laughs] KENDALL: You know, minor detail. That's why I think services companies in this space still have a lot going for it. Because in order to even be able to sell software to a company using Kubernetes, you half the time have to go stand up Kubernetes for them because it is still that hard for so many people to really adopt it. VICTORIA: Yeah. And maybe, like, talking more about, like, when it is the right decision to start on Kubernetes because I think the question I get sometimes is just, is it overkill? Is it too much for what we're building? Especially, like, if you're building a brand-new product, you're not even sure if it's going to get adopted that widely. KENDALL: I mean, and I'm [laughs] curious your thought on this, Joe, but there's a good argument to be made that Heroku was enough for the vast majority of founders early on. But the thing is, Kubernetes isn't as hard as it used to be. Going and clicking a couple of buttons on GKE and deploying something into Kubernetes with GKE Autopilot running it's not as easy as Heroku, but it's not wildly far off. And it does substantially future-proof you. So, when is it too early? I'm not sure it's ever too early if you have an intention of scaling if you're planning on running some kind of legacy workload, like, things that are going to be stateful. Or maybe WordPress, for example, you don't probably need to deploy your WordPress blog onto Kubernetes. You can do that in your cPanel on Bluehost. I don't actually know if Bluehost even exists anymore, but I assume it's still a thing. I don't know, what would you say, Joe? JOE: I agree with that. I think it's a hard first pill to swallow. But I think the reality is that it's very easy to underestimate the infrastructure needs of even an early product. Like, it doesn't really matter what you're building. You're still going to have things like secrets management. You're still going to have to worry about networking. They just don't go away. There's no way you have a product without them. And so, rather than slowly solving all those problems from scratch on a platform that isn't designed for it, I think it's easier to just bite the bullet and use one of the managed solutions, especially, as you said, I think it's getting easier and easier. The activation energy from going from credit card to Kubernetes cluster is just getting lower. KENDALL: And so, the role of the CTO is just getting easier and easier because they can just adopt the one technology, and it's obviously Kubernetes. And it's obviously Rust, right? [laughter] Yeah, no, I'm with you. And I think if you find somebody who knows Kubernetes inside and out, it's really not going to take them long to get started. VICTORIA: Yeah, once again, change management is the biggest challenge for any new innovation coming into adoption. So, I'm curious to talk more about the influence that you need and how you influence others to come around to these types of ideas, like, in the executive suite and with the leadership of a company, especially on these types of topics, which can feel maybe a little abstract for people. KENDALL: How you influence them specifically to use Kubernetes, or just how you talk with them about technology adoption in general? Or what are you asking? VICTORIA: Yeah, like, how do I get people to not just turn their ears off when I say the word Kubernetes? [laughs] KENDALL: Yeah, I mean, I think...so I think that's where it's the technologist's job and the role of the CTO to translate these things into business speak. And that's why I'm using words like future-proofing your infrastructure is because there are companies that...I know one company that made a conscious decision that they were going to try to re-platform every single year, and that is not a good idea or sustainable for the vast majority [laughs] of companies. In fact, I can't think of a single situation where that makes sense. But if you can say to the CFO, "Hey, it's going to cost us a little bit more right now. It's going to save us substantially in the long term because this is the thing that's winning. And if we go standardize on Heroku right now, every company does eventually have to migrate off of Heroku. They either go out of business, or they get too big for it." That's the kind of thing that needs to be communicated in order to get people to adopt it. They don't care what the word is. They don't care if you're saying Kubernetes; you know, most CFOs understand it about as well as my mom does. My mom tries to bring it up in conversation because she's heard me use it. And she thinks it makes her sound smart, which maybe it does in the right climate. VICTORIA: My partner does the same thing. He says DevOps and Kubernetes all the time. I'm like; you don't know what you're talking about. [laughter] JOE: Those words do not come up in my house. KENDALL: One of my kids asked me to explain Kubernetes. And I do a whole talk, particularly at organizations where understanding Kubernetes is essential to the salespeople's role. And I give a whole talk about the background of how we got here from deploying on some servers in our back room. And, you know, what's different about the cloud, what containerization did, et cetera. And I have this long explanation. And I remember taking a deep breath and saying to my kids, "Do you really want to hear this?" And I had one son say, "Yes, absolutely." And my wife and three of the other kids all stood up and said, "No way," and left the room. So, when somebody asks me, "What do you do?" Actually, one of the key relationships I built with some of the early people at GCP when we were partnering closely with them was a person that I met, and I asked, "What do you do for a living?" And he said, "I can tell you, but it's not going to mean anything to you." And I was like, "That's what I say to people." And it turned out he was in charge of, you know, Kubernetes partnerships for Google. I can explain to you what it means and why it's important. But you're not going to be happy that I spent that time explaining it to you. VICTORIA: [laughs] That sounds awesome, though. It sounds like you built a server rack just to demo to your children what it was. KENDALL: No, no. I just talked back through the history of...that company that I mentioned that built Twitter about five years too early; we had a, you know, we had a server rack in the...literally physically in our closet that was serving up our product at the time. VICTORIA: Probably the best demo I ever saw was at Google headquarters in Herndon, and someone had built...They had 3D-printed a little mini server rack that they had put Raspberry Pis onto, and then they had Kubernetes deployed on it. And they did an automatic failover of a node to just demo how it works and had little lights that went with it. It was pretty fun. So maybe you should get one for yourself. [laughter] It's a fun project. KENDALL: They remember the things that it enables. They don't remember what it does. And so, when I say so, and so is a client that's using this technology, then they get real excited because they're like, "My dad makes that work." And I'm like, well, okay, that's kind of a stretch, but you get the idea. VICTORIA: Yeah, you got to lean into that kind of reputation in your house. KENDALL: That's right. VICTORIA: And you're like, yes, that's correct. KENDALL: That's right. [laughs] VICTORIA: I do make Kubernetes. I make all the clouds work, yeah. KENDALL: Actually, my most common explanation is Kubernetes is the plumbing of the internet. Unless you're a plumber, you don't care about the pipes. You just want your shit to flush when you use the toilet. You want the things to load when you click your buttons. You don't actually care what's going on behind the scenes, but this is what's orchestrating it increasingly across the internet. VICTORIA: So far, we've called Kubernetes WordPress or the toilet. [laughs] KENDALL: The plumbing. [laughter] VICTORIA: You are really good at selling it. [laughter] KENDALL: Hey, if you want to build a nice, clean city, you need good plumbing. You might not care what the pipes are made of, but you need good plumbing. [laughs] VICTORIA: Works for me. On that note -- [laughs] KENDALL: Yeah. Right? Right? VICTORIA: That's [inaudible 36:41] on a high note. Is there anything else that you'd like to promote? KENDALL: With regards to CTO Lunches, we have a free listserv. There are local lunches. If there isn't a local lunch where you are, it's very lightweight to start up a chapter. We often have folks who are willing to sponsor that first lunch to get you going. We do have a paid tier of CTO Lunches. If you want a small back room Slack channel of people to discuss, I think it's $99 a month. Yeah, if you're a CTO and/or a senior engineering leader and you want a community of people to process with, be it our free tier or our paid tier, we've got something for you. We're trying to invest in this to build community around it. And it's something we enjoy doing more than almost anything. Come take part. VICTORIA: You can subscribe to the show and find notes along with a complete transcript for this episode at giantrobots.fm. If you have questions or comments, email us at hosts@giantrobots.fm. And you can find me on Twitter @victori_ousg. This podcast is brought to you by thoughtbot and produced and edited by Mandy Moore. Thanks for listening. See you next time. ANNOUNCER: This podcast is brought to you by thoughtbot, your expert strategy, design, development, and product management partner. We bring digital products from idea to success and teach you how because we care. Learn more at thoughtbot.com. Special Guest: Kendall Miller.

In our special we're thrilled talking to Don Vosburg and Miguel Pérez Colino about SUSE Manager and the Uyuni project. Don as an long-time user and contributor had a lot of great stories to share. Together with Miguel we had a look at the history and future of the project. Coming from the Spacewalk codebase that was published for the first time in 2008 a lot of innovation took place in the meantime. Beside various code optimizations and a new web interface, SaltStack also became central part of the product. For the future, the project will shift to container-based deployments - proxy servers are already support since October 2022. Note: Please excuse the occasional crackles and sound dropouts. We tried to get the best out of the recordings, but apparently our travel equipment is dying.

Nickolas Means, VP Engineering at Sym, joins Corey on Screaming in the Cloud to discuss how Sym is looking to solve the most common and most frustrating elements of compliance. Nick reveals why he finds it valuable to focus on making it easy for people to do the right thing over preventing them from doing the wrong thing, and why he feels the true spirit of compliance involves helping teams collaboratively come up with mutually beneficial solutions. Corey and Nick also dive into the common problems that engineers experience as a result of traditional compliance methods, and why historically the compliance industry has gotten a bad rap. About NickolasNickolas Means loves nothing more than a story of engineering triumph (except maybe a story of engineering disaster). When he's not stuck in a Wikipedia loop reading about plane crashes, he leads the engineering team at Sym, helping create the building blocks engineering teams need to build delightful developer access and approval workflows.Nick has been leading software engineering teams for more than a decade in the healthtech and devtools spaces. His focus is on building distributed organizations defined by their cultures of high trust and autonomy. He's also an international keynote speaker, having shared his unique brand of storytelling with audiences around the world. He works remotely from Austin, TX, and spends his spare time going on adventures with his wife and kids, running very slowly, and trying to brew the perfect cup of coffee.Links Referenced: symops.com: https://symops.com Twitter: https://twitter.com/nmeans TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Developers are responsible for more than ever these days. Not just the code they write, but also the containers and cloud infrastructure their apps run on. And probably the billing on top of that - which is neither here nor there. And a big part of that responsibility is app security — from code to cloud.That's where Snyk comes in. Snyk is a frictionless security platform that meets teams where they are, automating application security controls across their existing tools, workflows, and the AWS application stack — including seamless integrations with AWS CodePipeline, Amazon EKS, Amazon Inspector and several others.Deploy on AWS. Secure with Snyk. Learn more at snyk.co/scream. That's S-N-Y-K-dot-C-O/scream. And my thanks to them for sponsoring this ridiculous nonsense!Corey:  LANs of the late 90's and early 2000's were a magical place to learn about computers, hang out with your friends, and do cool stuff like share files, run websites & game servers, and occasionally bring the whole thing down with some ill-conceived software or network configuration. That's not how things are done anymore, but what if we could have a 90's style LAN experience along with the best parts of the 21st century internet? (Most of which are very hard to find these days.) Tailscale thinks we can, and I'm inclined to agree. With Tailscale I can use trusted identity providers like Google, or Okta, or GitHub to authenticate users, and automatically generate & rotate keys to authenticate devices I've added to my network. I can also share access to those devices with friends and teammates, or tag devices to give my team broader access. And that's the magic of it, your data is protected by the simple yet powerful social dynamics of small groups that you trust.Try now - it's free forever for personal use. I've been using it for almost two years personally, and am moderately annoyed that they haven't attempted to charge me for what's become an essential-to-my-workflow service.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. This promoted guest episode is brought to us by our friends over at Sym, and into my verbal grist mill, they have thrown their VP of Engineering, Nickolas Means. Nickolas, thank you for joining me.Nickolas: Thank you so much for having me, Corey. And feel free to call me Nick.Corey: I certainly shall. So, let's begin at a high level. When you're starting a company and trying to, sort of, bootstrap and raise initial rounds of funding and the rest, you're trying to save money in a bunch of places. And one of the most expensive things you can buy when starting a company is, of course, a vowel. You wound up not naming the company—or the vowel, really—the y is sometimes a vowel, sometimes not. It's S-Y-M. What is it you folks do exactly? What do you folks start? Where do you stop?Nickolas: So, the name of the company comes from the idea of helping humans and machines work together more effectively. And that's really nice and high level; it doesn't tell you any information about what we do.Corey: It feels like we're—we'd assume that most startups pivot at some point; we're just going to set—Nickolas: [laugh].Corey: —[crosstalk 00:01:33] seeds for that nice and early on, and dive on in.Nickolas: So, what we actually do, the two co-founders and myself all have a background in highly compliant industries. I've done VPN stints at a couple of health tech startups; they've done similarly. And all three of us ended up building sort of a certain set of things every time we were at one of these companies. Because you have to be compliant with things, and in order to be compliant with things, you have to have a set of controls, you have to restrict certain things: how people get to production, how people access customer data. And those controls, by and large, all suck. They're all painful and every company ends up building something from scratch at some point to make them not suck quite so bad. And it seemed like there was a product opportunity there.Corey: I would argue there absolutely is. One of the big problems that I've found throughout the time that I've been fixing AWS bills on a consultancy basis has been, we're really talking about cloud governance. But even now, by using the phrase cloud governance, three-quarters of the audience immediately wound up skipping to the next podcast over on their playlist because it sounds like it is one of those incredibly boring things. And to be fair, usually, when it comes to compliance, you want some of the most boring, least creative people in the world overseeing that. Like, when you wind up talking to someone at a company and they have a great sense of humor and they are constantly cracking jokes constantly, it's like, “What do you do?” Like, “Oh, I'm the CFO.” All you hear from that is, “Oh, I'm about to go to prison. Awesome.”Like, you want the wild, cutting-loose CEO to have three drinks and then confide, “I really like typing the number six.” You want them [laugh] to be predictable in a whole bunch of ways. And it always feels like compliance takes that entire mindset of, it's always about risk management, it's about wanting to make sure that people don't go off script in a bunch of weird ways, but as an engineer, what I always heard from that is slow down, don't be creative, go ahead and do things in very predictable ways. Only release things once a quarter, et cetera, et cetera. And yes, that's one way to meet compliance goals, but it's a crappy way, in my experience. I'm going to guess, though, that you have a lot more experience with the compliance world than I do because having worked a few times now, for big regulated finance companies, I wanted to get the hell out of the compliance universe.Nickolas: Yeah, I mean, you used an interesting turn of phrase there. You used the phrase, “Avoid going off script,” and I think there's a subtle turn there that actually makes all of this work a lot better. Instead of focusing on keeping people from going off script, you focus on keeping them on script. You focus on making it easier to do the right thing than to do the wrong thing. And that takes away a significant amount of the pain involved in compliance stuff.You look at implementing controls—and everybody has the exact same reaction you just brought up about governance—because there's so much FUD around this stuff. Everybody has been slowed down by one of these silly rules that makes no sense, that's checking a box and not actually meeting the spirit of any kind of meaningful improvement.Corey: Oh, cloud has absolutely doubled our speed of iteration because it used to take six weeks to get a server racked in the data center and we moved our processes to cloud and now to spin up an EC2 instance, it only takes three weeks of approvals. And at that point, it's what are you really doing? You wind up with people building on shadow IT. It's part of what contributed to the rise of cloud in the first place. Well, I can go through the annoying thing that this company wants me to do, or I have a corporate credit card and by the time it raises the level of spend to a point where it gets scrutiny, it's in production serving customers and what are they going to do?Some of the very early AWS sales conversations with customers started off as, “Well, why should we build on top of your cloud?” asks the exec, and they say, “Oh, sorry, you have 87 different accounts throughout your organization currently with us. We're just trying to give you some unified view into it and possibly some discounting if you want.” Yeah, these days, that's a fast track to getting yourself fired in some companies, if you wind up deviating from that story. But also, people are not doing this out of malfeasance; they're trying to get their job done.And as soon as guardrails start increasing friction, making it harder to do things the right way than to go around it, people will not comply. I strongly believe that, whether it's cost—which is my universe, and frankly, only a business hours problem—or actual governance issues with some compliance regimes, which get those wrong and hope you enjoy some time in prison.Nickolas: Yeah, exactly. I mean, you know, if you look at SOC 2, for example, there's a lot of companies out there that are willing to sell you a program that will help you become SOC 2 compliant. They show you all the steps you need to take, all the programs you need to put in place. The thing they don't do is help you establish the controls that are required. They'll tell you that you have to have somebody formally approving before software goes out to production. They won't give you any guidance whatsoever on how to put that control in place. And so, it's really easy for a compliance person that's not looking to collaborate with engineering just to go, “Okay, I need you to put a button in the deploy process and I need the CTO to click that button.”Corey: Yes. We've always seen that as reactions to different things. I was at a company once where there were some outages caused by bad deploys, so they decided that a VP had to sign off on every deploy. Now, I come from the sysadmin ops world, which explains so much about my cynical perspective on life, so the way we got that overturned within two days is we did the malicious compliance thing, where oh, we need to deploy this. Great, we are walking into the middle of a senior leadership team meeting to get them to—with a tablet or comput—laptop—“I need you to click the button right now.”And doing that out of hours and all kinds of other things, it's oh. Yeah. How about we wind up only doing that for significant large changes? How about that? Maybe you don't need to wake someone up at home in the middle of the night when there's a deploy going out that fixes a typo on the marketing page; little things like that.And at some point, you're always felt like the goal of governance was either ossified scar tissue around all the ways that things have failed before, or through a, frankly, misguided belief that if we wind up distilling everything down to processes and procedures, eventually, someday, we can have a bunch of trained monkeys doing this job instead of people who are expensive and, you know, cynical, and difficult to please. I feel like that is not the right way to think about these things.Nickolas: Well, I mean, the thing about those controls, you know, it's exactly what you just said. Nowhere in SOC 2 does it say that your VPN [unintelligible 00:07:56] or CTO has to approve all code deploys, that's not in there. But that's the reality of life at a bunch of companies. In reality, if you just follow a software development life cycle that has multiple people looking at code before it gets deployed, multiple people signing off on that code being okay to deploy and you have a staging environment before you hit production, you've met the control. And SOC 2 gives you so much flexibility in how you write the control.So, I think the thing that I've seen that makes compliance so much less painful, is when you have somebody that is 95% the boring persona like you're talking about, but 5% creative. 5% willing to kind of get their hands dirty, empathize with the engineering team, collaborate with the engineering team, and find a way to put some of these controls in place that doesn't just bring things to a grinding halt.Corey: I have to assume that, given that you've built an entire product slash company around this idea, that you have some opinions other than doing what I do, which is sitting in my lofty ivory tower and oh, you should, in this idealized case, do things a little bit differently. But it's going to be bespoke and the answer to any complex question, the more senior you get is, “It depends.” You, of course, have built something that scales out in a bunch of different ways. How do you view that in a way that makes it not either completely useless or overly prescriptive?Nickolas: We focus on giving the power to engineering teams and giving the security complexity [unintelligible 00:09:23] the power to oversee those things. You know, it would be easy to give somebody, like, a clickbox UI, let them design controls for SOC 2 or whatever, end-user interface, but that's not how engineers think; engineers think and express ideas and code. So, we've made the rather controversial decision in the face of a bunch of no-code tools to go low-code instead. So, to build a compliance workflow in Sym, you're going to write some Terraform, you're going to write a little bit of Python—a lot less than if you were building it from scratch—but you're going to end up with something that perfectly fits the way that you already work versus having to shift your work practices around to fit the tool.Corey: If you have inadvertently stumbled upon one of my hot buttons. There's a lot of people that take a perspective around low code. And I just want to say that that perspective is often garbage. Like, oh, that's not a real program—great. Hypothetically, if you have an idea for a business or a product or something, and involve software as most things seem to these days, maybe having to go to a boot camp for six months first as a prerequisite is not the best path forward.“Well, you're never going to build something hyperscale in a low-code environment.” Great, how many things that we built that actually need to be hyperscale that don't go through 16 different architectural iterations between ridiculous idea one day and thing that is actually hyperscale? It's an early optimization. I have an entire production pipeline in Retool that I built using low code. I think that that is a very powerful thing. And this idea that, “Oh, that's not real code.” Cool. What's your point?Nickolas: Well, and for us, one of the things that we're trying to enable is for software engineering teams, ops teams, whoever is building these controls, to interact with a security person or a compliance person, for them to be able to read the code, understand what it does, understand the way that the control has been implemented. And so, we provide a bunch of frameworks around that and a bunch of things. Like, you don't have to go and build a Slack workflow from scratch and nobody has to understand that code because it's buried in the platform. The only thing that the security or compliance person has to understand is the business logic that's been put into place. Who can approve it? Who can't approve it? How does that change after hours? How does that change if there's an incident? All of that is in very simple Python that you don't have to be an experienced programmer to be able to read.Corey: One of the big powerful things behind that is it really reduces the interrupt volume of someone coming by to an engineer who is deep in the middle of something else, and, “Hey, guess what I have? A surprise context switch for something that's going to take you probably 30 seconds, but then you're going to be distracted by all of this.” If you give people the ability to self-serve, everything tends to work a lot more smoothly.Nickolas: Yeah, absolutely. And, you know, that's one of the ways we use Sym at Sym: we've got it in front of our AWS production environment, so if you need to go and do anything in production, you just have to get approval from any other engineer that happens to be in the approval channel, sort of a two-keys-to-launch-a-missile model. And that works fine for our compliance needs and it avoids there being a single point of failure that every time you need to go and get into production, you have to go and say, “Mother, may I?”Corey: Exactly. It's one of those things where every time you wind up with something that injects friction, people are going to find ways around it. And in some cases, this leads to positive outcomes where, when you're subject to PCI, which is a lot more prescriptive than a number of other compliance regimes, it's, great; this is a lot of things that don't necessarily reflect how we work, how we want to work, et cetera. We can ignore it, which is not a great plan, we can wind up having to slow everything down, which is the common case, or the right answer is, we're going to build the PCI environment that is very self-contained, just the critical stuff that needs to be in there is going to be in there, and then we can build everything that touches it around it in ways that are a lot more aligned with how we believe software should be built.Nickolas: Yeah, absolutely. I mean, you silo off those high-control places, but there are controls that have to extend into the rest of the business. And one of the things that I'm a very firm believer in is, if you're going to impose a control upon somebody, they need to have the agency to shape and to change that control so that it lets them work the way that they want to work.Corey: I just want to call out how wonderful that is because I had a belief that looked borderline heretical, 12 years ago, when I said that, “Okay, simple rule. If you want me on call, I am empowered to change the thing that wakes me up.” Whether that is the code itself, the system itself, the paging threshold and frequency, or ultimately, I'm turning the physical pager off. It's one of those things where I decide what's an emergency outside of hours on that point. If it's going to wake me up, I need the power to make sure it never does. Otherwise, you have no agency. It just feels like you're being victimized by the stuff.Nickolas: Yeah, absolutely. I mean, there was a wave of on-call regimes that ran through large companies for a while where there would be a centralized on-call team that would be responsible for responding to hundreds of services. And thankfully, we are maturing past that; we're distributing on-call rotations so that teams that actually build services are responsible for them. And it's the same mindset, right? If you're going to be participating, if you're going to be working with a system or working with a control, then you need to be able to change it, you need to be able to make it work the way that you think that it ought to work.And in the context of compliance, you need to bring somebody along with you. You need to bring the person that's responsible for the controls that actually has to sign on the dotted line at the end of the audit period, saying that we do all of these things. So, you have to be able to explain what you're doing to them. But you have to be able to iterate.Corey: I have to ask, given that what you are building is going to have heavy involvement from engineering, how do you respond to the probably most common engineering objection I imagine you get, which is, “Well, this doesn't look hard. I could build this in a weekend.”Nickolas: You know, it's funny. We joke that our biggest competitor is build in-house, right? It's pretty easy to start looking at what it takes to build a from scratch workflow in Slack to build a Slack app, to understand the cost of building it in-house. Because nothing about building an elegant user interface in Slack is easy or cheap. That API is difficult to work with and hard to get good user experience out of.And we've spent a lot of time polishing a lot of places in the platform: we've got good documentation, we've got a good SDK, we've got good integration with third-party services that make all of this stuff easy to do. And it does look easy on the surface, it does look like ‘I can build it,' but we've had customers that have had that objection gone and tried to build it and come back. Because it's not as easy as anybody thinks.Corey: My biggest competitor for fixing AWS bills has always been Microsoft Excel. It's the, we're going to do it ourselves—badly—internally. Okay, great. If that works for you, terrific—Nickolas: Yeah.Corey: —but very often it doesn't. I mean, I think a classic case study of this is, in the terms of something that is well designed but is almost mind-bogglingly complex—and we're getting a case study in it this year—is Twitter because it looks from the outside, very simple. I wind up writing a thing and I hit the post button and it shows up in a timeline. And then other people can subscribe to it or not, and they see it themselves. That sounds like something you can build on a weekend. And we look at all the ways it's now exploding and collapsing and having weird bugs that no one anticipated, to realize, oh, this is a very challenging, very sophisticated application. But because it was well designed at one point, it looks easy.Nickolas: Yeah. Yeah, it continues to run despite the fact that it's having less than a quarter of the staff that originally maintained it, maintaining it because the services were well designed in the first place. They're resilient on their own and they're self-healing in a lot of cases. It's the same thing with Sym. You can build these tools in-house, you can build them yourself, but then you've got more software to maintain. Because once you build something, you own it, forever. And the cheapest code is no code; the cheapest code is code that you don't have to write.It's easy to look at a simple use case and understand a little bit of the cost of this. If you want a Slack workflow that gives you access to production in AWS, you can wire that up fairly quickly. Those APIs are not all that difficult. Now, let's say you want to add an integration where if you're on-call in PagerDuty, you can get to production without having to get an approval. Okay, well, now you've got a new API that you need to wire in.And let's say that every time that happens, you want to open a Jira ticket so that you can record that that's happened. Well, there's another API that you've got to wire in.j, whereas with Sym, it's just, it's right there. It's a few lines of code to wire it all together. And it deploys in Terraform alongside the rest of your infrastructure, so you manage it the same way you're used to managing things.Corey: It reminds me of my earlier career when I was deep in the configuration management weeds with Puppet and SaltStack, where the biggest competitor we had any of those projects was always someone writing a bash script to do it themselves. And yes, you can do that, but then the requirements change, or you're going to hit a point of scale that was surprising. And one of the valuable parts of it is that when the future is uncertain, as it always is—Nickolas: Always.Corey: Having folks who work in environments that aren't just yours who encounter a lot of those edge cases you're going to stumble into and can build things in is incredibly valuable. I don't think I've ever met anyone who ran an infrastructure that said, “I would build it the same way if I had to start over again.” They always want to, “I would fix these annoying things.” Well, by having a product focused on a space like this, it's yeah, today, you can have that VP click the approve button inside the GitHub Actions workflow. Good for you.But when you get just a little bit further down the path, you aren't going to want to do that anymore. There needs to be some decision-making it builds into it, and for certain high-risk changes, maybe a second person and so on. How do you build that logic engine? How do you build that workflow approach? How do you have a break glass thing for middle of the night when the site is down? Et cetera, et cetera, et cetera.And that's exactly the sort of thing that I would expect something like Sym to get very right, just because there's always a bigger fish. You've seen this [unintelligible 00:19:17] before in other shops. And more to the point, if there's something I want to do as a part of this that Sym doesn't support and you are looking at me strangely if I asked how to do it, that's usually a good early warning sign that maybe there's something I'm not thinking about here. Because whatever the problem space is, I'm probably not the only person that has to do this. How are other companies solving for this? And it turns out that all my copy of our SOC 2 report has a typo on it. That would explain a lot. That's a ‘can' instead of ‘can't.' Nevermind. Or something like that.Nickolas: Well, and the flip side of that is also true. I mean, the interesting thing about working on something that is sort of wide open with what you can wire up and build with it is we're always learning from our customers. We're always learning from the things that they're doing. And so, you know, when somebody approaches us of, “Hey, we need to solve this particular problem,” if we don't have a ready answer, we brainstorm and help figure that out. And to your point, that always extrapolates to other customers finding the same sort of thing useful.The other bit of this that's really interesting beyond the durability and the ability to kind of rapidly evolve these workflows is the audibility. It's helpful in a lot of these compliance regimes to have a third-party tracking this data for you. So, when somebody accesses AWS production, who approved that access? When somebody deploys code, who approved those deploys? Well, we sit there as kind of a third party on the side, observing all of this, taking all these notes for you, and piping them into whatever audit tool that you want.So, you've got that data long-term and when it comes time to audit, you've got all the evidence you need; it's already there, already collected. You don't have to go through and write a regex to parse a bunch of logs to get the information you need.Corey: And invariably, that regex is always going to be different, depending upon the log stuff. It's great having a unified central approach that is the trusted repository for this stuff. As you've been going to market and talking to your earlier customers and seeing the problems that you folks solve, what have you learned about the market space since you've gone into this direction? Because I feel like this is one of those products where you start designing and thinking you know a lot about the space, and you learn so much more just from the customer conversations and seeing that you can build the most finely crafted torque wrench in the world and the customer complains because it turns out, you built a crappy hammer.Nickolas: So, I think what's been really interesting to me is how much use our Lambda integration gets. We have a lot of first-party integrations with things like IAM and IAM identity center and Aptible and a bunch of tools that you can interact with, but a lot of our customers have wanted to do very specific things inside their infrastructure and put those things behind an approval. And the Lambda integration turns out to be a great Swiss army knife to do that because you can wire it up—it runs inside your firewall—to take essentially whatever action that you need it to. And that gets a ton of use. Probably more than half of our customers have at least one Lambda workflow in production, and I would not have expected that going in.Corey: It's wild to me just how pervasive Lambda has become. And even from a compliance perspective, it's great because unlike, “Well, it's a script that runs on a server somewhere,” yeah, it's immutable. It's versioned. There's a way to conclusively prove that at invocation, this is the code that ran, the end, with the following parameters. Done.There's no, “Well, looking at the timestamp on the file”—like, no. None of that nonsense. It's arguable that something that I have seen has been that Lambda is one of those rare technologies where you're seeing faster adoption in the enterprise and you are in startup land.Nickolas: Yeah, I would say that's true. I mean, it's so great for running undifferentiated workloads. I just need this one thing to happen really quickly and I don't want to mess with standing up a server to run this thing that runs once a week. Okay, well, here's a computer that will run just long enough for you to run this thing and then go away. It tracks exactly what ran, exactly when it ran, exactly how it got kicked off.And in our case, it has access to all of the internal AWS APIs that we wall off in our platform because we obviously don't want you using those things in the Sym runtime. But you can do anything that you want to your AWS environment from your own Lambda and we will gladly provide the approval step ahead of kicking that job off.Corey: Are you seeing people use Lambda-based workflows to manage on-premises things or is it more heavily in environments that are already within the AWS boundary?Nickolas: The Lambda stuff that we see is almost entirely—I think it is entirely for things that are within the AWS boundary. I can't think of an instance when somebody is managing something on-prem with it.Corey: I am increasingly discovering, through the magic of Tailscale—among a few other things—that I can use that for things on-premises that talk directly and interface with my Raspberry Pi in the spare room, et cetera. Which is—I think some people call it hybrid, which is the business enterprise term for ‘horrifying—Nickolas: Yep.Corey: —because it's a terrible pattern in some ways. But it's so convenient and it's so nice not to have to worry about some of these things, just an infrastructure point of view. One thing that I think that AWS has done very well at, as they've evolved, has been with AWS Artifact, which ties directly to their own compliance reports, where in the early days when I was responsible for SOC 2 controls at a company, I found myself answering security questionnaires from vendors as if I was running in a data center. And sure enough, they wanted to tour us-east-1. And it turns out, you can't really do that.So now, just pointing them to the stuff that comes out of Artifact, it's written by auditors for auditors and they go away and leave you alone without having to explain your bespoke artisanal nonsense to them. There's something very pleasant about being able to throw the lion's share of the work over to someone who already knows how to do it.Nickolas: Our audit period is ending here shortly and I have recently been and spending time in Artifact. So yes, a hundred percent.Corey: It used to be that you would only be able to get those things under explicit NDAs, you'd have to talk to your account manager for every one, it was a back-and-forth process, and you didn't really know if what you were going to get was going to answer the questions that they had. Now it's, you show up, you click things three times, and you're done. The hardest part is sorting out which ones you need from the hundreds of things available within Artifact.It's like, okay, that's great, but this one is in Spanish for some reason. And that's awesome, but on some level, it feels like that should be an easy filter option. But yeah, no one ever accused AWS of building a good user interface. But once you get the thing you need and can pass it off, great. Job over. It's one of my favorite services that most people who are what we know as ‘happy' don't know exist.Nickolas: Yeah well, and that, it points to a larger industry trend, right, that companies are getting SOC 2 specifically earlier and earlier because it is becoming table stakes to be able to sell into other companies. They want to see your SOC 2 report before they're willing to work with you before they're willing to let your software touch their infrastructure. And there is a lot of value in these compliance programs as essentially a stamp of approval that you're taking these things seriously, even with as much flexibility as SOC 2 has, just the stamp that we've thought about these things and we have serious answers to them is a pretty important signal to be able to send to somebody that's wanting to buy your software.Corey: We've toyed with the idea of going through the process ourselves because we get asked about it all the time, but it feels like the procurement processes that ask us for it expect us to come in with a whole software suite and the rest. And yeah, if that's the world we're operating in, it makes a lot of sense. We're a services-based consultancy; we come in as individuals, we have conversations with people, and we talk about this and we have no write access to anything in your environment and give you scoped-down permissions for what we talk to because we don't want the responsibility of that stuff.And a lot of companies get that intrinsically, but there's occasionally a few you have to go round and round and round with. It just it feels like it's one of those, okay, you're not quite there yet. You're trying to view everything through this very specific worldview. Maybe it works for your constraints and requirements, but I've never understood it. And I've learned the older I get, the more time I spend around this, I used to have such a negative perspective on compliance.And now it's, you know, everything's nuanced. There's a reason that these things are there. It's not just a make-work project for an industry that wants to slow everyone else down. It's, there are risks here; these things exist for a reason. There's a reason that you can go start Twitter for Pets tonight and not be regulated, but the same is not true of First Bank of Twitter Pets.It's okay, yeah, one of those things is going to require a fair bit of regulatory scrutiny, and as a society, we want that. Now, the counterargument that I don't necessarily want to get too far into is, should Twitter for Pets be regulated?Nickolas: [laugh].Corey: And that's a can of worms that I think we'll leave for another episode.Nickolas: Yeah, I mean, that's—you know, the people that hate compliance the most are the people that are on the sharp end of compliance, people that are having to actually deal with the controls that are imposed upon them by these compliance regimes and by somebody who's taking a very literal view in interpreting the things that some of these compliance programs say that you'd have to put in place. And I think, you know, that's—kind of bring the conversation full circle—that's the thing that we want to change more than anything. If we can wave a magic wand and change the compliance universe, the thing that I most want is to help compliance and security people collaborate with their engineering teams and come up with mutually beneficial solutions. Things that actually—the spirit of compliance.Corey: Oh, yeah. My first PCI audit was a little bit of a challenge, just because the auditor wasn't really conversant with anything that wasn't a large company. So, they show up at our twelve-person start off, and, “Okay, where's the Active Directory?” It's like, “We don't have one of those.” “Okay, well how do you authenticate to the WiFi?” It's like, “Oh, the password's on the wall.”It's, “Well, what happens if I get on that WiFi?” It's, “What can I do that I couldn't do from anywhere else?” Like, “Use that printer over there. That's it.” Because everything else was the idea of the security boundary was built on identity, not on what blessed network you happened to be on; there was no special permissioning that didn't apply to the Starbucks WiFi next town over.But that was one of those things where at first they thought this was a horrifying problem and they were not going to be able to certify us, and it turned into no, we had significantly advanced culture of security compliance, oversight, separation of duties, all the things you really care about. We just didn't have the trappings that usually came across with when you're thinking about this or starting—or having the temerity to start a company, you know, longer than 18 months ago at a place that wasn't San Francisco on the latest version of a MacBook Pro running the bleeding edge version of Chrome. It turns out that there's a big universe out there. And not that there's anything wrong with either side of it, until they start forgetting that not everyone operates the way that they do.Nickolas: Yeah. I mean, you know, we talked about checkbox compliance a lot and I think that's probably the biggest problem is there is a lot of checkbox compliance out there. And people have seen it not actually solve anything and just make everything harder. And so, compliance gets a bad rap.Corey: Oh, for me, the one that I've been picking fights on social media about for a few years now is encryption-at-rest in the cloud. Like, yes, you want full-disk encryption turned on your laptops, your phones, your tablets, et cetera. Someone steals it from the coffee shop, you want to be out the cost the hardware. The end. But if you can get a hard drive intact out of an AWS facility and then reassemble it with the right number of drives in the right places, without… and hasn't been encrypted. Congratulations, you earned it. As far as I'm concerned, that's yours. You can keep it.Because AWS employees aren't able to do that, let alone third parties. But it is easier by far to click the box to enable encryption-at-rest and not spend half an hour arguing with the auditor… and just get on with your day. And recently in S3, for example, they wound up making that a default. Good for them. It's just, can we please focus on the part of the story that's relevant and germane to our business? Because that is not the threat model of modern attacks.Nickolas: Yeah, I mean, for a long time, how much of the internet ran on unencrypted HTTP, but it was being served off of an encrypted disk? Great. What have we solved?Corey: Oh, absolutely. It's wild to me. Even now, I still we feel like there should be a reasonable way to handle—to [unintelligible 00:31:17] basically encryption between two points that doesn't depend on the third-party CA's with expiring certs and the rest. Drives me up a wall every time because it's always the worst possible time. It causes the strangest issues and there is something deeply and profoundly wrong with the fact that the failure mode from the user perspective between, “Your connection is being intercepted by a third party,” and, “Holy shit. This certificate expired two hours ago.” Like, those are very different use cases, but the scary warnings have trained people to treat them the same way.Nickolas: Yep. Yep, exactly the same. Ugh.Corey: I really want to thank you for being so generous with your time. If people want to learn more, where's the best place for them to find you?Nickolas: Yeah, so the best place to find out more about Sym is our website, symops.com, SYMOPS dot com. And I should mention that Sym is completely free for teams of up to ten people. If any of you out there listening check it out, please reach out. We'd love to hear about your experiences, help any way we can. And if you want to get in touch with me directly, the best place to do that for now, while it lasts is still Twitter. I'm on there as @nmeans.Corey: And we will, of course, include a link to that in the [show notes 00:32:27]. Thank you so much for agreeing to talk to me about all this stuff. I really appreciate it.Nickolas: Yeah. Thanks so much for having me on, Corey. It's been a lot of fun.Corey: Nick Means, VP of Engineering at Sym. I'm Cloud Economist Corey Quinn, and this has been a promoted guest episode, brought to us by our friends at Sym. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, along with an angry bitter comment that will get posted in six weeks, after you track down your elusive VP to click the approve button.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.

Tom and Chunga had just entered their respective studios to record this weeks episode of The Hacks, when BOOM! Breaking news reports started coming in saying that Silicon Valley Bank had just collapsed, which immediately changed the topic of the show! Tom has used SVB many times throughout his career.  He even used it to start SaltStack! He's certainly not alone, most tech startups use this bank to launch their businesses. The failure of Silicon Valley Bank has sent shockwaves through the tech industry and the U.S. economy as a whole! Experts and officials will no doubt say that this failure isn't a big deal and every thing is fine... It's just fine. Do you agree with them, or do you think this is a sign of things to come? Tom and Chunga have an opinion of what they think is going to happen.  Here's a sneak peak, they both say this problem goes all they way back to the fall of Rome!

When working with orchestration in automated systems, how do you find the right balance between things that are event driven and things that are workflow driven, or more linear? We go through some of the history of where we went from linear orchestration (Ansible) to timed orchestration (Chef or Puppet). We also discussed SaltStack, which had an event driven system into it, but didn't gain the traction that we might have expected as we look at the amount of orchestration systems that are now coming to light. In this conversation, we address the balance between when you orchestrate and when you want to do workflow and linear transactions, and how to find that sweet spot. One of the things that we've determined is, there aren't a lot of tools that hit that sweet spot. And I think if you listen carefully, you'll see why. Transcript: https://otter.ai/u/ynb8KndfUhKsTvQG6965ktepv-Y Image: https://www.pexels.com/photo/a-choir-singing-while-holding-a-flower-7569413/

About AnaïsAnaïs is a Developer Advocate at Aqua Security, where she contributes to Aqua's cloud native open source projects. When she is not advocating DevOps best practices, she runs her own YouTube Channel centered around cloud native technologies. Before joining Aqua, Anais worked as SRE at Civo, a cloud native service provider, where she helped enhance the infrastructure for hundreds of tenant clusters. As CNCF ambassador of the year 2021, her passion lies in making tools and platforms more accessible to developers and community members.Links Referenced: Aqua Security: https://www.aquasec.com/ Aqua Open Source YouTube channel: https://www.youtube.com/c/AquaSecurityOpenSource Personal blog: https://anaisurl.com TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by our friends at AWS AppConfig. Engineers love to solve, and occasionally create, problems. But not when it's an on-call fire-drill at 4 in the morning. Software problems should drive innovation and collaboration, NOT stress, and sleeplessness, and threats of violence. That's why so many developers are realizing the value of AWS AppConfig Feature Flags. Feature Flags let developers push code to production, but hide that that feature from customers so that the developers can release their feature when it's ready. This practice allows for safe, fast, and convenient software development. You can seamlessly incorporate AppConfig Feature Flags into your AWS or cloud environment and ship your Features with excitement, not trepidation and fear. To get started, go to snark.cloud/appconfig That's snark.cloud/appconfig.Corey: This episode is sponsored in part by Honeycomb. When production is running slow, it's hard to know where problems originate. Is it your application code, users, or the underlying systems? I've got five bucks on DNS, personally. Why scroll through endless dashboards while dealing with alert floods, going from tool to tool to tool that you employ, guessing at which puzzle pieces matter? Context switching and tool sprawl are slowly killing both your team and your business. You should care more about one of those than the other; which one is up to you. Drop the separate pillars and enter a world of getting one unified understanding of the one thing driving your business: production. With Honeycomb, you guess less and know more. Try it for free at honeycomb.io/screaminginthecloud. Observability: it's more than just hipster monitoring.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. Every once in a while, when I start trying to find guests to chat with me and basically suffer my various slings and arrows on this show, I encounter something that I've never really had the opportunity to explore further. And today's guest leads me in just such a direction. Anaïs is an open-source developer advocate at Aqua Security, and when I was asking her whether or not she wanted to talk about various topics, one of the first thing she said was, “Don't ask me much about AWS because I've never used it,” which, oh my God. Anaïs, thank you for joining me. You must be so very happy never to have dealt with the morass of AWS.Anaïs: [laugh]. Yes, I'm trying my best to stay away from it. [laugh].Corey: Back when I got into the cloud space, for lack of a better term, AWS was sort of really the only game in town unless you wanted to start really squinting hard at what you define cloud as. I mean yes, I could have gone into Salesforce or something, but I was already sad and angry all the time. These days, you can very much go all in-on cloud. In fact, you were a CNCF ambassador, if I'm not mistaken. So, you absolutely are in the infrastructure cloud space, but you haven't dealt with AWS. That is just an interesting path. Have you found others who have gone down that same road, or are you sort of the first of a new breed?Anaïs: I think to find others who are in a similar position or have a similar experience, as you do, you first have to talk about your experience, and this is the first time, or maybe the second, that I'm openly [laugh] saying it on something that will be posted live, like, to the internet. Before I, like, I tried to stay away from mentioning it at all, do the best that I can because I'm at this point where I'm so far into my cloud-native Kubernetes journey that I feel like I should have had to deal with AWS by now, and I just didn't. And I'm doing my best and I'm very successful in avoiding it. [laugh]. So, that's where I am. Yeah.Corey: We're sort of on opposite sides of a particular fence because I spend entirely too much time being angry at AWS, but I've never really touched Kubernetes and anger. I mean, I see it in a lot of my customer accounts and I get annoyed at its data transfer bills and other things that it causes in an economic sense, but as far as the care and feeding of a production cluster, back in my SRE days, I had very old-school architectures. It's, “Oh, this is an ancient system, just like grandma used to make,” where we had the entire web tier, then a job applic—or application server tier, and then a database at the end, and everyone knew where everything was. And then containers came out of nowhere, and it seemed like okay, this solves a bunch of problems and introduces a whole bunch more. How do I orchestrate them? How do I ensure that they're healthy?And then ah, Kubernetes was the answer. And for a while, it seemed like no matter what the problem was, Kubernetes was going to be the answer because people were evangelizing it pretty hard. And now I see it almost everywhere that I turn. What's your journey been like? How did you get into the weeds of, “You know what I want to do when I grow up? That's right. I want to work on container orchestration systems.” I have a five-year-old. She has never once said that because I don't abuse my children by making them learn how clouds work. How did you wind up doing what you do?Anaïs: It's funny that you mention that. So, I'm actually of the generation of engineers who doesn't know anything else but Kubernetes. So, when you mentioned that you used to use something before, I don't really know what that looks like. I know that you can still deploy systems without Kubernetes, but I have no idea how. My journey into the cloud-native space started out of frustration from the previous industry that I was working at.So, I was working for several years as developer advocate in the open-source blockchain cryptocurrency space and it's highly similar to all of the cliches that you hear online and across the news. And out of this frustration, [laugh] I was looking at alternatives. One of them was either going into game development, into the gaming industry, or the cloud-native space and infrastructure development and deployment. And yeah, that's where I ended up. So, at the end of 2020, I joined a startup in the cloud-native space and started my social media journey.Corey: One of the things that I found that Kubernetes solved for—and to be clear, Kubernetes really came into its own after I was doing a lot more advisory work and a lot more consulting style activity rather than running my own environments, but there's an entire universe of problems that the modern day engineer never has to think about due to, partially cloud and also Kubernetes as well, which is the idea of hardware or node failure. I've had middle of the night driving across Los Angeles in a panic getting to the data center because the disk array on the primary database had degraded because the drive failed. That doesn't happen anymore. And clouds have mostly solved that. It's okay, drives fail, but yeah, that's the problem for some people who live in Virginia or Oregon. I don't have to think about it myself.But you do have to worry about instances failing; what if the primary database instance dies? Well, when everything lives in a container then that container gets moved around in the stateless way between things, well great, you really only have to care instead about okay, what if all of my instances die? Or, what if my code is really crappy? To which my question is generally, what do you mean, ‘if?' All of us write crappy code.That's the nature of the universe. We open-source only the small subset that we are not actively humiliated by, which is, in a lot of ways, what you're focusing on now, over at Aqua Sec, you are an advocate for open-source. One of the most notable projects that come out of that is Trivy, if I'm pronouncing that correctly.Anaïs: Yeah, that's correct. Yeah. So, Trivy is our main open-source project. It's an all-in-one cloud-native security scanner. And it's actually—it's focused on misconfiguration issues, so it can help you to build more robust infrastructure definitions and configurations.So ideally, a lot of the things that you just mentioned won't happen, but it obviously, highly depends on so many different factors in the cloud-native space. But definitely misconfigurations of one of those areas that can easily go wrong. And also, not just that you have data might cease to exist, but the worst thing or, like, as bad might be that it's completely exposed online. And they are databases of different exposures where you can see all the kinds of data of information from just health data to dating apps, just being online available because the IP address is not protected, right? Things like that. [laugh].Corey: We all get those emails that start with, “Your security is very important to us,” and I know just based on that opening to an email, that the rest of that email is going to explain how security was not very important to you folks. And it's the apology, “Oops, we have messed up,” email. Now, the whole world of automated security scanners is… well, it's crowded. There are a number of different services out there that the cloud providers themselves offer a bunch of these, a whole bunch of scareware vendors at the security conferences do as well. Taking a quick glance at Trivy, one of the problems I see with it, from a cloud provider perspective, is that I see nothing that it does that winds up costing extra money on your cloud bill that you then have to pay to the cloud provider, so maybe they'll put a pull request in for that one of these days. But my sarcasm aside, what is it that differentiates Trivy from a bunch of other offerings in various spaces?Anaïs: So, there are multiple factors. If we're looking from an enterprise perspective, you could be using one of the in-house scanners from any of the cloud providers available, depending which you're using. The thing is, they are not generally going to be the ones who have a dedicated research team that provides the updates based on the vulnerabilities they find across the space. So, with an open-source security scanner or from a dedicated company, you will likely have more up-to-date information in your scans. Also, lots of different companies, they're using Trivy under the hood ultimately, or for their own scans.I can link a few where you can also find them in a Trivy repository. But ultimately, a lot of companies rely on Trivy and other open-source security scanners under the hood because they are from dedicated companies. Now, the other part to Trivy and why you might want to consider using Trivy is that in larger teams, you will have different people dealing with different components of your infrastructure, of your deployments, and you could end up having to use multiple different security scanners for all your different components from your container images that you're using, whether or not they are secure, whether or not they're following best practices that you defined to your infrastructure-as-code configurations, to you're running deployments inside of your cluster, for instance. So, each of those different stages across your lifecycle, from development to runtime, will maybe either need different security scanners, or you could use one security scanner that does it all. So, you could have in a team more knowledge sharing, you could have dedicated people who know how to use the tool and who can help out across a team across the lifecycle, and similar. So, that's one of the components that you might want to consider.Another thing is how mature is a tool, right? A lot of cloud providers, what they end up doing is they provide you with a solution, but it's nice to decoupled from anything else that you're using. And especially in the cloud-native space, you're heavily reliant on open-source tools, such as for your observability stack, right? Coming from Site Reliability Engineering also myself, I love using metrics and Grafana. And for me, if anything open-source from Loki to accessing my logs, to Grafana to dashboards, and all their integrations.I love that and I want to use the same tools that I'm using for everything else, also for my security tools. I don't want to have the metrics for my security tools visualized in a different solution to my reliability metrics for my application, right? Because that ultimately makes it more difficult to correlate metrics. So, those are, like, some of the factors that you might want to consider when you're choosing a security scanner.Corey: When you talk about thinking about this, from the perspective of an SRE is—I mean, this is definitely an artifact of where you come from and how you approach this space. Because in my world, when you have ten web servers, five application servers, and two database servers and you wind up with a problem in production, how do you fix this? Oh, it's easy. You log into one of those nodes and poke around and start doing diagnostics in production. In a containerized world, you generally can't do that, or there's a problem on a container, and by the time you're aware of that, that container hasn't existed for 20 minutes.So, how do you wind up figuring out what happens? And instrumenting for telemetry and metrics and observability, particularly at scale becomes way more important than it ever was, for me. I mean, my version of monitoring was always Nagios, which was the original Call of Duty that wakes you up at two in the morning when the hard drive fails. The world has thankfully moved beyond that and a bunch of ways. But it's not first nature for me. It's always, “Oh, yeah, that's right. We have a whole telemetry solution where I can go digging into.” My first attempt is always, oh, how do I get into this thing and poke it with a stick? Sometimes that's helpful, but for modern applications, it really feels like it's not.Anaïs: Totally. When we're moving to an infrastructure to an environment where we can deploy multiple times a day, right, and update our application multiple times a day, multiple times a day, we can introduce new security issues or other things can go wrong, right? So, I want to see—as much as I want to see all of the other failures, I want to see any security-related issues that might be deployed alongside those updates at the same frequency, right?Corey: The problem that I see across all this stuff, though, is there are a bunch of tools out there that people install, but then don't configure because, “Oh, well, I bought the tool. The end.” I mean, I think it was reported almost ten years ago or so on the big Target breach that they wound up installing some tool—I want to say FireEye, but please don't quote me on that—and it wound up firing off a whole bunch of alerts, and they figured was just noise, so they turned it all off. And it turned out no, no, this was an actual breach in progress. But people are so used to all the alarms screaming at them, that they don't dig into this.I mean, one of the original security scanners was Nessus. And I seen a lot of Nessus reports because for a long time, what a lot of crappy consultancies would do is they would white-label the output of whatever it was that Nessus said and deliver that in as the report. So, you'd wind up with 700 pages of quote-unquote, “Security issues.” And you'd have to flip through to figure out that, ah, this supports a somewhat old SSL negotiation protocol, and you're focusing on that instead of the oh, and by the way, the primary database doesn't have a password set. Like, it winds up just obscuring it because there is so much. How does Trivy approach avoiding the information overload problem?Anaïs: That's a great question because everybody's complaining about vulnerability fatigue, of them, for the first time, scanning their container images and workloads and seeing maybe even hundreds of vulnerabilities. And one of the things that can be done to counteract that right from the beginning is investing your time into looking at the different flags and configurations that you can do before actually deploying Trivy to, for example, your cluster. That's one part of it. The other part is I mentioned earlier, you would use a security scan at different parts of your deployment. So, it's really about integrating scanning not just once you—like, in your production environment, once you've deployed everything, but using it already before and empowering engineers to actually use it on their machines.Now, they can either decide to do it or not; it's not part of most people's job to do security scanning, but as you move along, the more you do, the more you can reduce the noise and then ultimately, when you deploy Trivy, for example, inside of your cluster, you can do a lot of configuration such as scanning just for critical vulnerabilities, only scanning for vulnerabilities that already have a fix available, and everything else should be ignored. Those are all factors and flags that you can place into Trivy, for instance, and make it easier. Now, with Trivy, you won't have automated PRs and everything out of the box; you would have to set up the actions or, like, the ways to mitigate those vulnerabilities manually by yourself with tools, as well as integrating Trivy with your existing stack, and similar. But then obviously, if you want to have something more automated, if you want to have something that does more for you in the background, that's when you want to use to an enterprise solution and shift to something like Aqua Security Enterprise Platform that actually provides you with the automated way of mitigating vulnerabilities where you don't have to know much about it and it just gives you the solution and provides you with a PR with the updates that you need in your infrastructure-as-code configurations to mitigate the vulnerability [unintelligible 00:15:52]?Corey: I think that's probably a very fair answer because let's be serious when you're running a bank or someone for whom security matters—and yes, yes, I know, security should matter for everyone, but let's be serious, I care a little bit less about the security impact of, for example, I don't know, my Twitter for Pets nonsense, than I do a dating site where people are not out about their orientation or whatnot. Like, there is a world of difference between the security concerns there. “Oh, no, you might be able to shitpost as me if you compromise my lasttweetinaws.com Twitter client that I put out there for folks to use.” Okay, great. That is not the end of the world compared to other stuff.By the time you're talking about things that are critically important, yeah, you want to spend money on this, and you want to have an actual full-on security team. But open-source tools like this are terrific for folks who are just getting started or they're building something for fun themselves and as it turns out, don't have a full security budget for their weird late-night project. I think that there's a beautiful, I guess, spectrum, as far as what level of investment you can make into security. And it's nice to see the innovation continued happening in the space.Anaïs: And you just mentioned that dedicated security companies, they likely have a research team that's deploying honeypots and seeing what happens to them, right? Like, how are attackers using different vulnerabilities and misconfigurations and what can be done to mitigate them. And that ultimately translates into the configurations of the open-source tool as well. So, if you're using, for instance, a security scanner that doesn't have an enterprise company with a research team behind it, then you might have different input into the data of that security scanner than if you do, right? So, these are, like, additional considerations that you might want to take when choosing a scanner. And also that obviously depends on what scanning you want to do, on the size of your company, and similar, right?Corey: This episode is sponsored in part by our friend EnterpriseDB. EnterpriseDB has been powering enterprise applications with PostgreSQL for 15 years. And now EnterpriseDB has you covered wherever you deploy PostgreSQL on-premises, private cloud, and they just announced a fully-managed service on AWS and Azure called BigAnimal, all one word. Don't leave managing your database to your cloud vendor because they're too busy launching another half-dozen managed databases to focus on any one of them that they didn't build themselves. Instead, work with the experts over at EnterpriseDB. They can save you time and money, they can even help you migrate legacy applications—including Oracle—to the cloud. To learn more, try BigAnimal for free. Go to biganimal.com/snark, and tell them Corey sent you.Corey: Something that I do find fairly interesting is that you started off, as you say, doing DevRel in the open-source blockchain world, then you went to work as an SRE, and then went back to doing DevRel-style work. What got you into SRE and what got you out of SRE, other than the obvious having worked in SRE myself and being unhappy all the time? I kid, but what was it that got you into that space and then out of it?Anaïs: Yeah. Yeah, but no, it's a great question. And it's, I guess, also was shaped my perspective on different tools and, like, the user experience of different tools. But ultimately, I first worked in the cloud-native space for an enterprise tool as developer advocate. And I did not like the experience of working for a paid solution. Doing developer advocacy for it, it felt wrong in a lot of ways. A lot of times you were required to do marketing work in those situations.And that kind of got me out of developer advocacy into SRE work. And now I was working partially or mainly as SRE, and then on the side, I was doing some presentations in developer advocacy. However, that split didn't quite work, either. And I realized that the value that I add to a project is really the way I convey information, which I can't do if I'm busy fixing the infrastructure, right? I can't convey the information of as much of how the infrastructure has been fixed as I can if I'm working with an engineering team and then doing developer advocacy, solely developer advocacy within the engineering team.So, how I ultimately got back into developer advocacy was just simply by being reached out to by my manager at Aqua Security, and Itay telling me, him telling me that he has a role available and if I want to join his team. And it was open-source-focused. Given that I started my career for several years working in the open-source space and working with engineers, contributing to open-source tools, it was kind of what I wanted to go back to, what I really enjoy doing. And yeah, that's how that came about [laugh].Corey: For me, I found that I enjoy aspects of the technology part, but I find I enjoy talking to people way more. And for me, the gratifying moment that keeps me going, believe it or not, is not necessarily helping giant companies spend slightly less money on another giant company. It's watching people suddenly understand something they didn't before, it's watching the light go on in their eyes. And that's been addictive to me for a long time. I've also found that the best way for me to learn something is to teach someone else.I mean, the way I learned Git was that I foolishly wound up proposing a talk, “Terrible Ideas in Git”—we'll teach it by counterexample—four months before the talk. And they accepted it, and crap, I'd better learn enough get to give this talk effectively. I don't recommend this because if you miss the deadline, I checked, they will not move the conference for you. But there really is something to be said for watching someone learn something by way of teaching it to them.Anaïs: It's actually a common strategy for a lot of developer advocates of making up a talk and then waiting whether or not it will get accepted. [laugh] and once it gets accepted, that's when you start learning the tool and trying to figure it out. Now, it's not a good strategy, obviously, to do that because people can easily tell that you just did that for a conference. And—Corey: Sounds to me, like, you need to get better at bluffing. I kid.Anaïs: [laugh].Corey: I kid. Don't bluff your way through conference talks as a general rule. It tends not to go well. [laugh].Anaïs: No. It's a bad idea. It's a really bad idea. And so, I ultimately started learning the technologies or, like, the different tools and projects in the cloud-native space. And there are lots, if you look at the CNCF landscape, right? But just trying to talk myself through them on my YouTube channel. So, my early videos on my channel, it's just very much on the go of me looking for the first time at somebody's documentation and not making any sense out of them.Corey: It's surprising to me how far that gets you. I mean, I guess I'm always reminded of that Tom Hanks movie from my childhood Big where he wakes up—the kid wakes up as an adult one day, goes to work, and bluffs his way into working at a toy company. He's in a management meeting and just they're showing their new toy they're going to put out there and he's, “I don't get it.” Everyone looks at him like how dare you say it? And, “I don't get it. What's fun about this?” Because he's a kid.And he wants to getting promoted to vice president because wow, someone pointed out the obvious thing. And so often, it feels like using a tool or a product, be it open-source or enterprise, it is clearly something different in my experience of it when I try to use this thing than the person who developed it. And very often it's that I don't see the same things or think of the problem space the same way that the developers did, but also very often—and I don't mean to call anyone in particular out here—it's a symptom of a terrible user interface or user experience.Anaïs: What you've just said, a lot of times, it's just about saying the thing that nobody that dares to say or nobody has thought of before, and that gets you obviously, easier, further [laugh] then repeating what other people have already mentioned, right? And a lot of what you see a lot of times in these—also an open-source projects, but I think more even in closed-source enterprise organizations is that people just repeat whatever everybody else is saying in the room, right? You don't have that as much in the open-source world because you have more input or easier input in public than you do otherwise, but it still happens that I mean, people are highly similar to each other. If you're contributing to the same project, you probably have a similar background, similar expertise, similar interests, and that will get you to think in a similar way. So, if there's somebody like, like a high school student maybe, somebody just graduated, somebody from a completely different industry who's looking at those tools for the first time, it's like, “Okay, I know what I'm supposed to do, but I don't understand why I should use this tool for that.” And just pointing that out, gets you a response, most of the time. [laugh].Corey: I use Twitter and use YouTube. And obviously, I bias more for short, pithy comments that are dripping in sarcasm, whereas in a long-form video, you can talk a lot more about what you're seeing. But the problem I have with bad user experience, particularly bad developer experience, is that when it happens to me—and I know at a baseline level, that I am reasonably competent in technical spaces, but when I encounter a bad interface, my immediate instinctive reaction is, “Oh, I'm dumb. And this thing is for smart people.” And that is never, ever true, except maybe with quantum computing. Great, awesome. The Hello World tutorial for that stuff is a PhD from Berkeley. Good luck if you can get into that. But here in the real world where the rest of us play, it's just a bad developer experience, but my instinctive reaction is that there's stuff I don't know, and I'm not good enough to use this thing. And I get very upset about that.Anaïs: That's one of the things that you want to do with any technical documentation is that the first experience that anybody has, no matter the background, with your tool should be a success experience, right? Like people should look at it, use maybe one command, do one thing, one simple thing, and be like, “Yeah, this makes sense,” or, like, this was fun to do, right? Like, this first positive interaction. And it doesn't have to be complex. And that's what many people I think get wrong, that they try to show off how powerful a tool is, of like, oh, “My God, you can do all those things. It's so exciting, right?” But [laugh] ultimately, if nobody can use it or if most of the people, 99% of the people who try it for the first time have a bad experience, it makes them feel uncomfortable or any negative emotion, then it's really you're approaching it from the wrong perspective, right?Corey: That's very apt. I think it's so much of whether people stick with something long enough to learn it and find the sharp edges has to do with what their experience looks like. I mean, back when I was more or less useless when it comes to anything that looked like programming—because I was a sysadmin type—I started contributing to SaltStack. And what was amazing about that was Tom Hatch, the creator of the project had this pattern that he kept up for way too long, where whenever anyone submitted an issue, he said, “Great, well, how about you fix it?” And because we had a patch, like, “Well, I'm not good at programming.” He's like, “That's okay. No one is. Try it and we'll see.”And he accepted every patch and then immediately, you'd see another patch come in ten minutes later that fixed the problems in your patch. But it was the most welcoming and encouraging experience, and I'm not saying that's a good workflow for an open-source maintainer, but he still remains one of the best humans I know, just from that perspective alone.Anaïs: That's amazing. I think it's really about pointing out that there are different ways of doing open-source [laugh] and there is no one way to go about it. So, it's really about—I mean, it's about the community, ultimately. That's what it boils down to, of you are dependent, as an open-source project, on the community, so what is the best experience that you can give them? If that's something that you want to and can invest in, then yeah [laugh] that's probably the best outcome for everybody.Corey: I do have one more question, specifically around things that are more timely. Now, taking a quick look at Trivy and recent features, it seems like you've just now—now-ish—started supporting cloud scanning as well. Previously, it was effectively, “Oh, this scans configuration and containers. Okay, great.” Now, you're targeting actually scanning cloud providers themselves. What does this change and what brought you to this place, as someone who very happily does not deal with AWS?Anaïs: Yeah, totally. So, I just started using AWS, specifically to showcase this feature. So, if you look at the Aqua Open Source YouTube channel, you will find several tutorials that show you how to use that feature, among others.Now, what I mentioned earlier in the podcast already is that Trivy is really versatile, it allows you to scan different aspects of your stack at different stages of your development lifecycle. And that's made possible because Trivy is ultimately using different open-source projects under the hood. For example, if you want to scan your infrastructure-as-code misconfigurations, it's using a tool called tfsec, specifically for Terraform. And then other tools for other scanning, for other security scanning. Now, we have—or had; it's going to be probably deprecated—a tool called CloudSploit in the Aqua open-source project suite.Now, that's going to, kind of like, the functionality that CloudSploit was providing is going to get converted to become part of Trivy, so everything scanning-related is going to become part of Trivy that really, like, once you understand how Trivy works and all of the CLI commands in Trivy have exactly the same structure, it's really easy to scan from container images to infrastructure-as-code, to generating s-bombs to scanning also now, your cloud infrastructure and Trivy can scan any of your AWS services for misconfigurations, and it's using basically the AWS client under the hood to connect with the services of everything you have set up there, and then give you the list of misconfigurations. And once it has done the scan, you can then drill down further into the different aspects of your misconfigurations without performing the entire scan again, since you likely have lots and lots of resources, so you wouldn't want to scan them every time again, right, when you perform the scan. So, once something has been scanned, Trivy will know whether the resource changed or not, it won't scan it again. That's the same way that in-classes scanning works right now. Once a container image has been scanned for vulnerabilities, it won't scan the same container image again because that would just waste time. [laugh]. So yeah, do check it out. It's our most recent feature, and it's going to come out also to the other cloud providers out there. But we're starting with AWS and this kind of forced me to finally [laugh] look at it for the sake of it. But I'm not going to be happy. [laugh].Corey: No, I don't think anyone is. It's every time I see on a resume that someone says, “Oh, I'm an expert in AWS,” it's, “No you're not.” They have 400-some-odd services now. We have crossed the point long ago, where I can very convincingly talk about AWS services that do not exist to Amazonians and not get called out for it because who in the world knows what they run? And half of their services sound like something I made up to be funny, but they're very real. It's wild to me that it is a sprawling as it is and apparently continues to work as a viable business.But no one knows all of it and everyone feels confused, lost, and overwhelmed every time they look at the AWS console. This has been my entire career in life for the last six years, and I still feel that way. So, I'm sure everyone else does, too.Anaïs: And this is how misconfigurations happen, right? You're confused about what you're actually supposed to do and how you're supposed to do it. And that's, for example, with all the access rights in Google Cloud, something that I'm very familiar with, that completely overwhelms you and you get super frustrated by, and you don't even know what you give access to. It's like, if you've ever had to configure Discord user roles, it's a similar disaster. You will not know which user has access to which. They kind of changed it and try to improve it over the past year, but it's a similar issue that you face in cloud providers, just on a much larger-scale, not just on one chat channel. [laugh]. So.Corey: I think that is probably a fair place to leave it. I really want to thank you for spending as much time with me as you have talking about the trials and travails of, well, this industry, for lack of a better term. If people want to learn more, where's the best place to find you?Anaïs: So, I have a weekly DevOps newsletter on my blog, which is anaisurl—like, how you spell U-R-L—and then dot com. anaisurl.com. That's where I have all the links to my different channels, to all of the resources that are published where you can find out more as well. So, that's probably the best place. Yeah.Corey: And we will, of course, put a link to that in the show notes. I really want to thank you for being as generous with your time as you have been. Thank you.Anaïs: Thank you for having me. It was great.Corey: Anaïs, open-source developer advocate at Aqua Security. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with an angry, insulting comment that I will never see because it's buried under a whole bunch of minor or false-positive vulnerability reports.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

About MattMatt is a Sr. Architect in Belfast, an AWS DevTools Hero, Serverless Architect, Author and conference speaker. He is focused on creating the right environment for empowered teams to rapidly deliver business value in a well-architected, sustainable and serverless-first way.You can usually find him sharing reusable, well architected, serverless patterns over at cdkpatterns.com or behind the scenes bringing CDK Day to life.Links Referenced: Previous guest appearance: https://www.lastweekinaws.com/podcast/screaming-in-the-cloud/slinging-cdk-knowledge-with-matt-coulter/ The CDK Book: https://thecdkbook.com/ Twitter: https://twitter.com/NIDeveloper TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. One of the best parts about, well I guess being me, is that I can hold opinions that are… well, I'm going to be polite and call them incendiary, and that's great because I usually like to back them in data. But what happens when things change? What happens when I learn new things?Well, do I hold on to that original opinion with two hands at a death grip or do I admit that I was wrong in my initial opinion about something? Let's find out. My guest today returns from earlier this year. Matt Coulter is a senior architect since he has been promoted at Liberty Mutual. Welcome back, and thanks for joining me.Matt: Yeah, thanks for inviting me back, especially to talk about this topic.Corey: Well, we spoke about it a fair bit at the beginning of the year. And if you're listening to this, and you haven't heard that show, it's not that necessary to go into; mostly it was me spouting uninformed opinions about the CDK—the Cloud Development Kit, for those who are unfamiliar—I think of it more or less as what if you could just structure your cloud resources using a programming language you claim to already know, but in practice, copy and paste from Stack Overflow like the rest of us? Matt, you probably have a better description of what the CDK is in practice.Matt: Yeah, so we like to say it's imperative code written in a declarative way, or declarative code written in an imperative way. Either way, it lets you write code that produces CloudFormation. So, it doesn't really matter what you write in your script; the point is, at the end of the day, you still have the CloudFormation template that comes out of it. So, the whole piece of it is that it's a developer experience, developer speed play, that if you're from a background that you're more used to writing a programming language than a YAML, you might actually enjoy using the CDK over writing straight CloudFormation or SAM.Corey: When I first kicked the tires on the CDK, my first initial obstacle—which I've struggled with in this industry for a bit—is that I'm just good enough of a programmer to get myself in trouble. Whenever I wind up having a problem that StackOverflow doesn't immediately shine a light on, my default solution is to resort to my weapon of choice, which is brute force. That sometimes works out, sometimes doesn't. And as I went through the CDK, a couple of times in service to a project that I'll explain shortly, I made a bunch of missteps with it. The first and most obvious one is that AWS claims publicly that it has support in a bunch of languages: .NET, Python, there's obviously TypeScript, there's Go support for it—I believe that went generally available—and I'm sure I'm missing one or two, I think? Aren't I?Matt: Yeah, it's: TypeScript, JavaScript, Python Java.Net, and Go. I think those are the currently supported languages.Corey: Java. That's the one that I keep forgetting. It's the block printing to the script that is basically Java cursive. The problem I run into, and this is true of most things in my experience, when a company says that we have deployed an SDK for all of the following languages, there is very clearly a first-class citizen language and then the rest that more or less drift along behind with varying degrees of fidelity. In my experience, when I tried it for the first time in Python, it was not a great experience for me.When I learned just enough JavaScript, and by extension TypeScript, to be dangerous, it worked a lot better. Or at least I could blame all the problems I ran into on my complete novice status when it comes to JavaScript and TypeScript at the time. Is that directionally aligned with what you've experienced, given that you work in a large company that uses this, and presumably, once you have more than, I don't know, two developers, you start to take on aspects of a polyglot shop no matter where you are, on some level?Matt: Yeah. So personally, I jump between Java, Python, and TypeScript whenever I'm writing projects. So, when it comes to the CDK, you'd assume I'd be using all three. I typically stick to TypeScript and that's just because personally, I've had the best experience using it. For anybody who doesn't know the way CDK works for all the languages, it's not that they have written a custom, like, SDK for each of these languages; it's a case of it uses a Node process underneath them and the language actually interacts with—it's like the compiled JavaScript version is basically what they all interact with.So, it means there are some limitations on what you can do in that language. I can't remember the full list, but it just means that it is native in all those languages, but there are certain features that you might be like, “Ah,” whereas, in TypeScript, you can just use all of TypeScript. And my first inclination was actually, I was using the Python one and I was having issues with some compiler errors and things that are just caused by that process. And it's something that talking in the cdk.dev Slack community—there is actually a very active—Corey: Which is wonderful, I will point out.Matt: [laugh]. Thank you. There is actually, like, an awesome Python community in there, but if you ask them, they would all ask for improvements to the language. So, personally if someone's new, I always recommend they start with TypeScript and then branch out as they learn the CDK so they can understand is this a me problem, or is this a problem caused by the implementation?Corey: From my perspective, I didn't do anything approaching that level of deep dive. I took a shortcut that I find has served me reasonably well in the course of my career, when I'm trying to do something in Python, and you pull up a tutorial—which I'm a big fan of reading experience reports, and blog posts, and here's how to get started—and they all have the same problem, which is step one, “Run npm install.” And that's “Hmm, you know, I don't recall that being a standard part of the Python tooling.” It's clearly designed and interpreted and contextualized through a lens of JavaScript. Let's remove that translation layer, let's remove any weird issues I'm going to have in that transpilation process, and just talk in the language it written in. Will this solve my problems? Oh, absolutely not, but it will remove a subset of them that I am certain to go blundering into like a small lost child trying to cross an eight-lane freeway.Matt: Yeah. I've heard a lot of people say the same thing. Because the CDK CLI is a Node process, you need it no matter what language you use. So, if they were distributing some kind of universal binary that just integrated with the languages, it would definitely solve a lot of people's issues with trying to combine languages at deploy time.Corey: One of the challenges that I've had as I go through the process of iterating on the project—but I guess I should probably describe it for those who have not been following along with my misadventures; I write blog posts about it from time to time because I need a toy problem to kick around sometimes because my consulting work is all advisory and I don't want to be a talking head-I have a Twitter client called lasttweetinaws.com. It's free; go and use it. It does all kinds of interesting things for authoring Twitter threads.And I wanted to deploy that to a bunch of different AWS regions, as it turns out, 20 or so at the moment. And that led to a lot of interesting projects and having to learn how to think about these things differently because no one sensible deploys an application simultaneously to what amounts to every AWS region, without canary testing, and having a phased rollout in the rest. But I'm reckless, and honestly, as said earlier, a bad programmer. So, that works out. And trying to find ways to make this all work and fit together led iteratively towards me discovering that the CDK was really kind of awesome for a lot of this.That said, there were definitely some fairly gnarly things I learned as I went through it, due in no small part to help I received from generous randos in the cdk.dev Slack team. And it's gotten to a point where it's working, and as an added bonus, I even mostly understand what he's doing, which is just kind of wild to me.Matt: It's one of those interesting things where because it's a programming language, you can use it out of the box the way it's designed to be used where you can just write your simple logic which generates your CloudFormation, or you can do whatever crazy logic you want to do on top of that to make your app work the way you want it to work. And providing you're not in a company like Liberty, where I'm going to do a code review, if no one's stopping you, you can do your crazy experiments. And if you understand that, it's good. But I do think something like the multi-region deploy, I mean, with CDK, if you'd have a construct, it takes in a variable that you can just say what the region is, so you can actually just write a for loop and pass it in, which does make things a lot easier than, I don't know, try to do it with a YAML, which you can pass in parameters, but you're going to get a lot more complicated a lot quicker.Corey: The approach that I took philosophically was I wrote everything in a region-agnostic way. And it would be instantiated and be told what region to run it in as an environment variable that CDK deploy was called. And then I just deploy 20 simultaneous stacks through GitHub Actions, which invoke custom runners that runs inside of a Lambda function. And that's just a relatively basic YAML file, thanks to the magic of GitHub Actions matrix jobs. So, it fires off 20 simultaneous processes and on every commit to the main branch, and then after about two-and-a-half minutes, it has been deployed globally everywhere and I get notified on anything that fails, which is always fun and exciting to learn those things.That has been, overall, just a really useful experiment and an experience because you're right, you could theoretically run this as a single CDK deploy and then wind up having an iterate through a list of regions. The challenge I have there is that unless I start getting into really convoluted asynchronous concurrency stuff, it feels like it'll just take forever. At two-and-a-half minutes a region times 20 regions, that's the better part of an hour on every deploy and no one's got that kind of patience. So, I wound up just parallelizing it a bit further up the stack. That said, I bet they are relatively straightforward ways, given the async is a big part of JavaScript, to do this simultaneously.Matt: One of the pieces of feedback I've seen about CDK is if you have multiple stacks in the same project, it'll deploy them one at a time. And that's just because it tries to understand the dependencies between the stacks and then it works out which one should go first. But a lot of people have said, “Well, I don't want that. If I have 20 stacks, I want all 20 to go at once the way you're saying.” And I have seen that people have been writing plugins to enable concurrent deploys with CDK out of the box. So, it may be something that it's not an out-of-the-box feature, but it might be something that you can pull in a community plug-in to actually make work.Corey: Most of my problems with it at this point are really problems with CloudFormation. CloudFormation does not support well, if at all, secure string parameters from the AWS Systems Manager parameter store, which is my default go-to for secret storage, and Secrets Manager is supported, but that also cost 40 cents a month per secret. And not for nothing, I don't really want to have all five secrets deployed to Secrets Manager in every region this thing is in. I don't really want to pay $20 a month for this basically free application, just to hold some secrets. So, I wound up talking to some folks in the Slack channel and what we came up with was, I have a centralized S3 bucket that has a JSON object that lives in there.It's only accessible from the deployment role, and it grabs that at deploy time and stuffs it into environment variables when it pushes these things out. That's the only stateful part of all of this. And it felt like that is, on some level, a pattern that a lot of people would benefit from if it had better native support. But the counterargument that if you're only deploying to one or two regions, then Secrets Manager is the right answer for a lot of this and it's not that big of a deal.Matt: Yeah. And it's another one of those things, if you're deploying in Liberty, we'll say, “Well, your secret is unencrypted at runtime, so you probably need a KMS key involved in that,” which as you know, the costs of KMS, it depends on if it's a personal solution or if it's something for, like, a Fortune 100 company. And if it's personal solution, I mean, what you're saying sounds great that it's IAM restricted in S3, and then that way only at deploy time can be read; it actually could be a custom construct that someone can build and publish out there to the construct library—or the construct hub, I should say.Corey: To be clear, the reason I'm okay with this, from a security perspective is one, this is in a dedicated AWS account. This is the only thing that lives in that account. And two, the only API credentials we're talking about are the application-specific credentials for this Twitter client when it winds up talking to the Twitter API. Basically, if you get access to these and are able to steal them and deploy somewhere else, you get no access to customer data, you get—or user data because this is not charge for anything—you get no access to things that have been sent out; all you get to do is submit tweets to Twitter and it'll have the string ‘Last Tweet in AWS' as your client, rather than whatever normal client you would use. It's not exactly what we'd call a high-value target because all the sensitive to a user data lives in local storage in their browser. It is fully stateless.Matt: Yeah, so this is what I mean. Like, it's the difference in what you're using your app for. Perfect case of, you can just go into the Twitter app and just withdraw those credentials and do it again if something happens, whereas as I say, if you're building it for Liberty, that it will not pass a lot of our Well-Architected reviews, just for that reason.Corey: If I were going to go and deploy this at a more, I guess, locked down environment, I would be tempted to find alternative approaches such as having it stored encrypted at rest via KMS in S3 is one option. So, is having global DynamoDB tables that wind up grabbing those things, even grabbing it at runtime if necessary. There are ways to make that credential more secure at rest. It's just, I look at this from a real-world perspective of what is the actual attack surface on this, and I have a really hard time just identifying anything that is going to be meaningful with regard to an exploit. If you're listening to this and have a lot of thoughts on that matter, please reach out I'm willing to learn and change my opinion on things.Matt: One thing I will say about the Dynamo approach you mentioned, I'm not sure everybody knows this, but inside the same Dynamo table, you can scope down a row. You can be, like, “This row and this field in this row can only be accessed from this one Lambda function.” So, there's a lot of really awesome security features inside DynamoDB that I don't think most people take advantage of, but they open up a lot of options for simplicity.Corey: Is that tied to the very recent announcement about Lambda getting SourceArn as a condition key? In other words, you can say, “This specific Lambda function,” as opposed to, “A Lambda in this account?” Like that was a relatively recent Advent that I haven't fully explored the nuances of.Matt: Yeah, like, that has opened a lot of doors. I mean, the Dynamo being able to be locked out in your row has been around for a while, but the new Lambda from SourceArn is awesome because, yeah, as you say, you can literally say this thing, as opposed to, you have to start going into tags, or you have to start going into something else to find it.Corey: So, I want to talk about something you just alluded to, which is the Well-Architected Framework. And initially, when it launched, it was a whole framework, and AWS made a lot of noise about it on keynote stages, as they are want to do. And then later, they created a quote-unquote, “Well-Architected Tool,” which let's be very direct, it's the checkbox survey form, at least the last time I looked at it. And they now have the six pillars of the Well-Architected Framework where they talk about things like security, cost, sustainability is the new pillar, I don't know, absorbency, or whatever the remainders are. I can't think of them off the top of my head. How does that map to your experience with the CDK?Matt: Yeah, so out of the box, the CDK from day one was designed to have sensible defaults. And that's why a lot of the things you deploy have opinions. I talked to a couple of the Heroes and they were like, “I wish it had less opinions.” But that's why whenever you deploy something, it's got a bunch of configuration already in there. For me, in the CDK, whenever I use constructs, or stacks, or deploying anything in the CDK, I always build it in a well-architected way.And that's such a loaded sentence whenever you say the word ‘well-architected,' that people go, “What do you mean?” And that's where I go through the six pillars. And in Liberty, we have a process, it used to be called SCORP because it was five pillars, but not SCORPS [laugh] because they added sustainability. But that's where for every stack, we'll go through it and we'll be like, “Okay, let's have the discussion.” And we will use the tool that you mentioned, I mean, the tool, as you say, it's a bunch of tick boxes with a text box, but the idea is we'll get in a room and as we build the starter patterns or these pieces of infrastructure that people are going to reuse, we'll run the well-architected review against the framework before anybody gets to generate it.And then we can say, out of the box, if you generate this thing, these are the pros and cons against the Well-Architected Framework of what you're getting. Because we can't make it a hundred percent bulletproof for your use case because we don't know it, but we can tell you out of the box, what it does. And then that way, you can keep building so they start off with something that is well documented how well architected it is, and then you can start having—it makes it a lot easier to have those conversations as they go forward. Because you just have to talk about the delta as they start adding their own code. Then you can and you go, “Okay, you've added these 20 lines. Let's talk about what they do.” And that's why I always think you can do a strong connection between infrastructure-as-code and well architected.Corey: As I look through the actual six pillars of the Well-Architected Framework: sustainability, cost optimization, performance, efficiency, reliability, security, and operational excellence, as I think through the nature of what this shitpost thread Twitter client is, I am reasonably confident across all of those pillars. I mean, first off, when it comes to the cost optimization pillar, please, don't come to my house and tell me how that works. Yeah, obnoxiously the security pillar is sort of the thing that winds up causing a problem for this because this is an account deployed by Control Tower. And when I was getting this all set up, my monthly cost for this thing was something like a dollar in charges and then another sixteen dollars for the AWS config rule evaluations on all of the deploys, which is… it just feels like a tax on going about your business, but fine, whatever. Cost and sustainability, from my perspective, also tend to be hand-in-glove when it comes to this stuff.When no one is using the client, it is not taking up any compute resources, it has no carbon footprint of which to speak, by my understanding, it's very hard to optimize this down further from a sustainability perspective without barging my way into the middle of an AWS negotiation with one of its power companies.Matt: So, for everyone listening, watch as we do a live well-architected review because—Corey: Oh yeah, I expect—Matt: —this is what they are. [laugh].Corey: You joke; we should do this on Twitter one of these days. I think would be a fantastic conversation. Or Twitch, or whatever the kids are using these days. Yeah.Matt: Yeah.Corey: And again, if so much of it, too, is thinking about the context. Security, you work for one of the world's largest insurance companies. I shitpost for a living. The relative access and consequences of screwing up the security on this are nowhere near equivalent. And I think that's something that often gets lost, per the perfect be the enemy of the good.Matt: Yeah that's why, unfortunately, the Well-Architected Tool is quite loose. So, that's why they have the Well-Architected Framework, which is, there's a white paper that just covers anything which is quite big, and then they wrote specific lenses for, like, serverless or other use cases that are shorter. And then when you do a well-architected review, it's like loose on, sort of like, how are you applying the principles of well-architected. And the conversation that we just had about security, so you would write that down in the box and be, like, “Okay, so I understand if anybody gets this credential, it means they can post this Last Tweet in AWS, and that's okay.”Corey: The client, not the Twitter account, to be clear.Matt: Yeah. So, that's okay. That's what you just mark down in the well-architected review. And then if we go to day one on the future, you can compare it and we can go, “Oh. Okay, so last time, you said this,” and you can go, “Well, actually, I decided to—” or you just keep it as a note.Corey: “We pivoted. We're a bank now.” Yeah.Matt: [laugh]. So, that's where—we do more than tweets now. We decided to do microtransactions through cryptocurrency over Twitter. I don't know but if you—Corey: And that ends this conversation. No no. [laugh].Matt: [laugh]. But yeah, so if something changes, that's what the well-architected reviews for. It's about facilitating the conversation between the architect and the engineer. That's all it is.Corey: This episode is sponsored in part by our friend EnterpriseDB. EnterpriseDB has been powering enterprise applications with PostgreSQL for 15 years. And now EnterpriseDB has you covered wherever you deploy PostgreSQL on-premises, private cloud, and they just announced a fully-managed service on AWS and Azure called BigAnimal, all one word. Don't leave managing your database to your cloud vendor because they're too busy launching another half-dozen managed databases to focus on any one of them that they didn't build themselves. Instead, work with the experts over at EnterpriseDB. They can save you time and money, they can even help you migrate legacy applications—including Oracle—to the cloud. To learn more, try BigAnimal for free. Go to biganimal.com/snark, and tell them Corey sent you.Corey: And the lens is also helpful in that this is a serverless application. So, we're going to view it through that lens, which is great because the original version of the Well-Architected Tool is, “Oh, you built this thing entirely in Lambda? Have you bought some reserved instances for it?” And it's, yeah, why do I feel like I have to explain to AWS how their own systems work? This makes it a lot more streamlined and talks about this, though, it still does struggle with the concept of—in my case—a stateless app. That is still something that I think is not the common path. Imagine that: my code is also non-traditional. Who knew?Matt: Who knew? The one thing that's good about it, if anybody doesn't know, they just updated the serverless lens about, I don't know, a week or two ago. So, they added in a bunch of more use cases. So, if you've read it six months ago, or even three months ago, go back and reread it because they spent a good year updating it.Corey: Thank you for telling me that. That will of course wind up in next week's issue of Last Week in AWS. You can go back and look at the archives and figure out what week record of this then. Good work. One thing that I have learned as well as of yesterday, as it turns out, before we wound up having this recording—obviously because yesterday generally tends to come before today, that is a universal truism—is it I had to do a bit of refactoring.Because what I learned when I was in New York live-tweeting the AWS Summit, is that the Route 53 latency record works based upon where your DNS server is. Yeah, that makes sense. I use Tailscale and wind up using my Pi-hole, which lives back in my house in San Francisco. Yeah, I was always getting us-west-1 from across the country. Cool.For those weird edge cases like me—because this is not the common case—how do I force a local region? Ah, I'll give it its own individual region prepend as a subdomain. Getting that to work with both the global lasttweetinaws.com domain as well as the subdomain on API Gateway through the CDK was not obvious on how to do it.Randall Hunt over at Caylent was awfully generous and came up with a proof-of-concept in about three minutes because he's Randall, and that was extraordinarily helpful. But a challenge I ran into was that the CDK deploy would fail because the way that CloudFormation was rendered in the way it was trying to do stuff, “Oh, that already has that domain affiliated in a different way.” I had to do a CDK destroy then a CDK deploy for each one. Now, not the end of the world, but it got me thinking, everything that I see around the CDK more or less distills down to either greenfield or a day one experience. That's great, but throw it all away and start over is often not what you get to do.And even though Amazon says it's always day one, those of us in, you know, real companies don't get to just treat everything as brand new and throw away everything older than 18 months. What is the day two experience looking like for you? Because you clearly have a legacy business. By legacy, I of course, use it in the condescending engineering term that means it makes actual money, rather than just telling really good stories to venture capitalists for 20 years.Matt: Yeah. We still have mainframes running that make a lot of money. So, I don't mock legacy at all.Corey: “What's that piece of crap do?” “Well, about $4 billion a year in revenue. Perhaps show some respect.” It's a common refrain.Matt: Yeah, exactly. So yeah, anyone listening, don't mock legacy because as Corey says, it is running the business. But for us when it comes to day two, it's something that I'm actually really passionate about this in general because it is really easy. Like I did it with CDK patterns, it's really easy to come out and be like, “Okay, we're going to create a bunch of starter patterns, or quickstarts”—or whatever flavor that you came up with—“And then you're going to deploy this thing, and we're going to have you in production and 30 seconds.” But even day one later that day—not even necessarily day two—it depends on who it was that deployed it and how long they've been using AWS.So, you hear these stories of people who deployed something to experiment, and they either forget to delete, it cost them a lot of money or they tried to change it and it breaks because they didn't understand what was in it. And this is where the community starts to diverge in their opinions on what AWS CDK should be. There's a lot of people who think that at the minute CDK, even if you create an abstraction in a construct, even if I create a construct and put it in the construct library that you get to use, it still unravels and deploys as part of your deploy. So, everything that's associated with it, you don't own and you technically need to understand that at some point because it might, in theory, break. Whereas there's a lot of people who think, “Okay, the CDK needs to go server side and an abstraction needs to stay an abstraction in the cloud. And then that way, if somebody is looking at a 20-line CDK construct or stack, then it stays 20 lines. It never unravels to something crazy underneath.”I mean, that's one pro tip thing. It'd be awesome if that could work. I'm not sure how the support for that would work from a—if you've got something running on the cloud, I'm pretty sure AWS [laugh] aren't going to jump on a call to support some construct that I deployed, so I'm not sure how that will work in the open-source sense. But what we're doing at Liberty is the other way. So, I mean, we famously have things like the software accelerator that lets you pick a pattern or create your pipelines and you're deployed, but now what we're doing is we're building a lot of telemetry and automated information around what you deployed so that way—and it's all based on Well-Architected, common theme. So, that way, what you can do is you can go into [crosstalk 00:26:07]—Corey: It's partially [unintelligible 00:26:07], and partially at a glance, figure out okay, are there some things that can be easily remediated as we basically shift that whole thing left?Matt: Yeah, so if you deploy something, and it should be good the second you deploy it, but then you start making changes. Because you're Corey, you just start adding some stuff and you deploy it. And if it's really bad, it won't deploy. Like, that's the Liberty setup. There's a bunch of rules that all go, “Okay, that's really bad. That'll cause damage to customers.”But there's a large gap between bad and good that people don't really understand the difference that can cost a lot of money or can cause a lot of grief for developers because they go down the wrong path. So, that's why what we're now building is, after you deploy, there's a dashboard that'll just come up and be like, “Hey, we've noticed that your Lambda function has too little memory. It's going to be slow. You're going to have bad cold starts.” Or you know, things like that.The knowledge that I have had the gain through hard fighting over the past couple of years putting it into automation, and that way, combined with the well-architected reviews, you actually get me sitting in a call going, “Okay, let's talk about what you're building,” that hopefully guides people the right way. But I still think there's so much more we can do for day two because even if you deploy the best solution today, six months from now, AWS are releasing ten new services that make it easier to do what you just did. So, someone also needs to build something that shows you the delta to get to the best. And that would involve AWS or somebody thinking cohesively, like, these are how we use our products. And I don't think there's a market for it as a third-party company, unfortunately, but I do think that's where we need to get to, that at day two somebody can give—the way we're trying to do for Liberty—advice, automated that says, “I see what you're doing, but it would be better if you did this instead.”Corey: Yeah, I definitely want to spend more time thinking about these things and analyzing how we wind up addressing them and how we think about them going forward. I learned a lot of these lessons over a decade ago. I was fairly deep into using Puppet, and came to the fair and balanced conclusion that Puppet was a steaming piece of crap. So, the solution was that I was one of the very early developers behind SaltStack, which was going to do everything right. And it was and it was awesome and it was glorious, right up until I saw an environment deployed by someone else who was not as familiar with the tool as I was, at which point I realized hell is other people's use cases.And the way that they contextualize these things, you craft a finely balanced torque wrench, it's a thing of beauty, and people complain about the crappy hammer. “You're holding it wrong. No, don't do it that way.” So, I have an awful lot of sympathy for people building platform-level tooling like this, where it works super well for the use case that they're in, but not necessarily… they're not necessarily aligned in other ways. It's a very hard nut to crack.Matt: Yeah. And like, even as you mentioned earlier, if you take one piece of AWS, for example, API Gateway—and I love the API Gateway team; if you're listening, don't hate on me—but there's, like, 47,000 different ways you can deploy an API Gateway. And the CDK has to cover all of those, it would be a lot easier if there was less ways that you could deploy the thing and then you can start crafting user experiences on a platform. But whenever you start thinking that every AWS component is kind of the same, like think of the amount of ways you're can deploy a Lambda function now, or think of the, like, containers. I'll not even go into [laugh] the different ways to run containers.If you're building a platform, either you support it all and then it sort of gets quite generic-y, or you're going to do, like, what serverless cloud are doing though, like Jeremy Daly is building this unique experience that's like, “Okay, the code is going to build the infrastructure, so just build a website, and we'll do it all behind it.” And I think they're really interesting because they're sort of opposites, in that one doesn't want to support everything, but should theoretically, for their slice of customers, be awesome, and then the other ones, like, “Well, let's see what you're going to do. Let's have a go at it and I should hopefully support it.”Corey: I think that there's so much that can be done on this. But before we wind up calling it an episode, I had one further question that I wanted to explore around the recent results of the community CDK survey that I believe is a quarterly event. And I read the analysis on this, and I talked about it briefly in the newsletter, but it talks about adoption and a few other aspects of it. And one of the big things it looks at is the number of people who are contributing to the CDK in an open-source context. Am I just thinking about this the wrong way when I think that, well, this is a tool that helps me build out cloud infrastructure; me having to contribute code to this thing at all is something of a bug, whereas yeah, I want this thing to work out super well—Docker is open-source, but you'll never see me contributing things to Docker ever, as a pull request, because it does, as it says on the tin; I don't have any problems that I'm aware of that, ooh, it should do this instead. I mean, I have opinions on that, but those aren't pull requests; those are complete, you know, shifts in product strategy, which it turns out is not quite done on GitHub.Matt: So, it's funny I, a while ago, was talking to a lad who was the person who came up with the idea for the CDK. And CDK is pretty much the open-source project for AWS if you look at what they have. And the thought behind it, it's meant to evolve into what people want and need. So yes, there is a product manager in AWS, and there's a team fully dedicated to building it, but the ultimate aspiration was always it should be bigger than AWS and it should be community-driven. Now personally, I'm not sure—like you just said it—what the incentive is, given that right now CDK only works with CloudFormation, which means that you are directly helping with an AWS tool, but it does give me hope for, like, their CDK for Terraform, and their CDK for Kubernetes, and there's other flavors based on the same technology as AWS CDK that potentially could have a thriving open-source community because they work across all the clouds. So, it might make more sense for people to jump in there.Corey: Yeah, I don't necessarily think that there's a strong value proposition as it stands today for the idea of the CDK becoming something that works across other cloud providers. I know it technically has the capability, but if I think that Python isn't quite a first-class experience, I don't even want to imagine what other providers are going to look like from that particular context.Matt: Yeah, and that's from what I understand, I haven't personally jumped into the CDK for Terraform and we didn't talk about it here, but in CDK, you get your different levels of construct. And is, like, a CloudFormation-level construct, so everything that's in there directly maps to a property in CloudFormation, and then L2 is AWS's opinion on safe defaults, and then L3 is when someone like me comes along and turns it into something that you may find useful. So, it's a pattern. As far as I know, CDK for Terraform is still on L1. They haven't got the rich collection—Corey: And L4 is just hiring you as a consultant—Matt: [laugh].Corey: —to come in fix my nonsense for me?Matt: [laugh]. That's it. L4 could be Pulumi recently announced that you can use AWS CDK constructs inside it. But I think it's one of those things where the constructs, if they can move across these different tools the way AWS CDK constructs now work inside Pulumi, and there's a beta version that works inside CDK for Terraform, then it may or may not make sense for people to contribute to this stuff because we're not building at a higher level. It's just the vision is hard for most people to get clear in their head because it needs articulated and told as a clear strategy.And then, you know, as you said, it is an AWS product strategy, so I'm not sure what you get back by contributing to the project, other than, like, Thorsten—I should say, so Thorsten who wrote the book with me, he is the number three contributor, I think, to the CDK. And that's just because he is such a big user of it that if he sees something that annoys him, he just comes in and tries to fix it. So, the benefit is, he gets to use the tool. But he is a super user, so I'm not sure, outside of super users, what the use case is.Corey: I really want to thank you for, I want to say spending as much time talking to me about this stuff as you have, but that doesn't really go far enough. Because so much of how I think about this invariably winds up linking back to things that you have done and have been advocating for in that community for such a long time. If it's not you personally, just, like, your fingerprints are all over this thing. So, it's one of those areas where the entire software developer ecosystem is really built on the shoulders of others who have done a lot of work that came before. Often you don't get any visibility of who those people are, so it's interesting whenever I get to talk to someone whose work I have directly built upon that I get to say thank you. So, thank you for this. I really do appreciate how much more straightforward a lot of this is than my previous approach of clicking in the console and then lying about it to provision infrastructure.Matt: Oh, no worries. Thank you for the thank you. I mean, at the end of the day, all of this stuff is just—it helps me as much as it helps everybody else, and we're all trying to do make everything quicker for ourselves, at the end of the day.Corey: If people want to learn more about what you're up to, where's the best place to find you these days? They can always take a job at Liberty; I hear good things about it.Matt: Yeah, we're always looking for people at Liberty, so come look up our careers. But Twitter is always the best place. So, I'm @NIDeveloper on Twitter. You should find me pretty quickly, or just type Matt Coulter into Google, you'll get me.Corey: I like it. It's always good when it's like, “Oh, I'm the top Google result for my own name.” On some level, that becomes an interesting thing. Some folks into it super well, John Smith has some challenges, but you know, most people are somewhere in the middle of that.Matt: I didn't used to be number one, but there's a guy called the Kangaroo Kid in Australia, who is, like, a stunt driver, who was number one, and [laugh] I always thought it was funny if people googled and got him and thought it was me. So, it's not anymore.Corey: Thank you again for, I guess, all that you do. And of course, taking the time to suffer my slings and arrows as I continue to revise my opinion of the CDK upward.Matt: No worries. Thank you for having me.Corey: Matt Coulter, senior architect at Liberty Mutual. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice and leave an angry comment as well that will not actually work because it has to be transpiled through a JavaScript engine first.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

About ChrisChris Short has been a proponent of open source solutions throughout his over two decades in various IT disciplines, including systems, security, networks, DevOps management, and cloud native advocacy across the public and private sectors. He currently works on the Kubernetes team at Amazon Web Services and is an active Kubernetes contributor and Co-chair of OpenGitOps. Chris is a disabled US Air Force veteran living with his wife and son in Greater Metro Detroit. Chris writes about Cloud Native, DevOps, and other topics at ChrisShort.net. He also runs the Cloud Native, DevOps, GitOps, Open Source, industry news, and culture focused newsletter DevOps'ish.Links Referenced: DevOps'ish: https://devopsish.com/ EKS News: https://eks.news/ Containers from the Couch: https://containersfromthecouch.com opengitops.dev: https://opengitops.dev ChrisShort.net: https://chrisshort.net Twitter: https://twitter.com/ChrisShort TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. Coming back to us since episode two—it's always nice to go back and see the where are they now type of approach—I am joined by Senior Developer Advocate at AWS Chris Short. Chris, been a few years. How has it been?Chris: Ha. Corey, we have talked outside of the podcast. But it's been good. For those that have been listening, I think when we recorded I wasn't even—like, when was season two, what year was that? [laugh].Corey: Episode two was first pre-pandemic and the rest. I believe—Chris: Oh. So, yeah. I was at Red Hat, maybe, when I—yeah.Corey: Yeah. You were doing Red Hat stuff, back when you got to work on open-source stuff, as opposed to now, where you're not within 1000 miles of that stuff, right?Chris: Actually well, no. So, to be clear, I'm on the EKS team, the Kubernetes team here at AWS. So, when I joined AWS in October, they were like, “Hey, you do open-source stuff. We like that. Do more.” And I was like, “Oh, wait, do more?” And they were like, “Yes, do more.” “Okay.”So, since joining AWS, I've probably done more open-source work than the three years at Red Hat that I did. So, that's kind of—you know, like, it's an interesting point when I talk to people about it because the first couple months are, like—you know, my friends are like, “So, are you liking it? Are you enjoying it? What's going on?” And—Corey: Do they beat you with reeds? Like, all the questions people have about companies? Because—Chris: Right. Like, I get a lot of random questions about Amazon and AWS that I don't know the answer to.Corey: Oh, when I started telling people, I fixed Amazon bills, I had to quickly pivot that to AWS bills because people started asking me, “Well, can you save me money on underpants?” It's I—Chris: Yeah.Corey: How do you—fine. Get the prime credit card. It docks 5% off the bill, so there you go. But other than that, no, I can't.Chris: No.Corey: It's—Chris: Like, I had to call my bank this morning about a transaction that I didn't recognize, and it was from Amazon. And I was like, that's weird. Why would that—Corey: Money just flows one direction, and that's the wrong direction from my employer.Chris: Yeah. Like, what is going on here? It shouldn't have been on that card kind of thing. And I had to explain to the person on the phone that I do work at Amazon but under the Web Services team. And he was like, “Oh, so you're in IT?”And I'm like, “No.” [laugh]. “It's actually this big company. That—it's a cloud company.” And they're like, “Oh, okay, okay. Yeah. The cloud. Got it.” [laugh]. So, it's interesting talking to people about, “I work at Amazon.” “Oh, my son works at Amazon distribution center,” blah, blah, blah. It's like, cool. “I know about that, but very little. I do this.”Corey: Your son works in Amazon distribution center. Is he a robot? Is normally my next question on that? Yeah. That's neither here nor there.So, you and I started talking a while back. We both write newsletters that go to a somewhat similar audience. You write DevOps'ish. I write Last Week in AWS. And recently, you also have started EKS News because, yeah, the one thing I look at when I'm doing these newsletters every week is, you know what I want to do? That's right. Write more newsletters.Chris: [laugh].Corey: So, you are just a glutton for punishment? And, yeah, welcome to the addiction, I suppose. How's it been going for you?Chris: It's actually been pretty interesting, right? Like, we haven't pushed it very hard. We're now starting to include it in things. Like we did Container Day; we made sure that EKS news was on the landing page for Container Day at KubeCon EU. And you know, it's kind of just grown organically since then.But it was one of those things where it's like, internally—this happened at Red Hat, right—when I started live streaming at Red Hat, the ultimate goal was to do our product management—like, here's what's new in the next version thing—do those live so anybody can see that at any point in time anywhere on Earth, the second it's available. Similar situation to here. This newsletter actually is generated as part of a report my boss puts together to brief our other DAs—or developer advocates—you know, our solutions architects, the whole nine yards about new EKS features. So, I was like, why can't we just flip that into a weekly newsletter, you know? Like, I can pull from the same sources you can.And what's interesting is, he only does the meeting bi-weekly. So, there's some weeks where it's just all me doing it and he ends up just kind of copying and pasting the newsletter into his document, [laugh] and then adds on for the week. But that report meeting for that team is now getting disseminated to essentially anyone that subscribes to eks.news. Just go to the site, there's a subscribe thing right there. And we've gotten 20 issues in and it's gotten rave reviews, right?Corey: I have been a subscriber for a while. I will say that it has less Chris Short personality—Chris: Mm-hm.Corey: —to it than DevOps'ish does, which I have to assume is by design. A lot of The Duckbill Group's marketing these days is no longer in my voice, rather intentionally, because it turns out that being a sarcastic jackass and doing half-billion dollar AWS contracts can not to be the most congruent thing in the world. So okay, we're slowly ameliorating that. It's professional voice versus snarky voice.Chris: Well, and here's the thing, right? Like, I realized this year with DevOps'ish that, like, if I want to take a week off, I have to do, like, what you did when your child was born. You hired folks to like, do the newsletter for you, or I actually don't do the newsletter, right? It's binary: hire someone else to do it, or don't do it. So, the way I structured this newsletter was that any developer advocate on my team could jump in and take over the newsletter so that, you know, if I'm off that week, or whatever may be happening, I, Chris Short, am not the voice. It is now the entire developer advocate team.Corey: I will challenge you on that a bit. Because it's not Chris Short voice, that's for sure, but it's also not official AWS brand voice either.Chris: No.Corey: It is clearly written by a human being who is used to communicating with the audience for whom it is written. And that is no small thing. Normally, when oh, there's a corporate newsletter; that's just a lot of words to say it's bad. This one is good. I want to be very clear on that.Chris: Yeah, I mean, we have just, like, DevOps'ish, we have sections, just like your newsletter, there's certain sections, so any new, what's new announcements, those go in automatically. So, like, that can get delivered to your inbox every Friday. Same thing with new blog posts about anything containers related to EKS, those will be in there, then Containers from the Couch, our streaming platform, essentially, for all things Kubernetes. Those videos go in.And then there's some ecosystem news as well that I collect and put in the newsletter to give people a broader sense of what's going on out there in Kubernetes-land because let's face it, there's upstream and then there's downstream, and sometimes those aren't in sync, and that's normal. That's how Kubernetes kind of works sometimes. If you're running upstream Kubernetes, you are awesome. I appreciate you, but I feel like that would cause more problems and it's worse sometimes.Corey: Thank you for being the trailblazers. The rest of us can learn from your misfortune.Chris: [laugh]. Yeah, exactly. Right? Like, please file your bugs accordingly. [laugh].Corey: EKS is interesting to me because I don't see a lot of it, which is, probably, going to get a whole lot of, “Wait, what?” Moments because wait, don't you deal with very large AWS bills? And I do. But what I mean by that is that EKS, until you're using its Fargate expression, charges for the control plane, which rounds to no money, and the rest is running on EC2 instances running in a company's account. From the billing perspective, there is no difference between, “We're running massive fleets of EKS nodes.” And, “We're managing a whole bunch of EC2 instances by hand.”And that feels like an interesting allegory for how Kubernetes winds up expressing itself to cloud providers. Because from a billing perspective, it just looks like one big single-tenant application that has some really strange behaviors internally. It gets very chatty across AZs when there's no reason to, and whatnot. And it becomes a very interesting study in how to expose aspects of what's going on inside of those containers and inside of the Kubernetes environment to the cloud provider in a way that becomes actionable. There are no good answers for this yet, but it's something I've been seeing a lot of. Like, “Oh, I thought you'd be running Kubernetes. Oh, wait, you are and I just keep forgetting what I'm looking at sometimes.”Chris: So, that's an interesting point. The billing is kind of like, yeah, it's just compute, right? So—Corey: And my insight into AWS and the way I start thinking about it is always from a billing perspective. That's great. It's because that means the more expensive the services, the more I know about it. It's like, “IAM. What is that?” Like, “Oh, I have no idea. It's free. How important could it be?” Professional advice: do not take that philosophy, ever.Chris: [laugh]. No. Ever. No.Corey: Security: it matters. Oh, my God. It's like you're all stars. Your IAM policy should not be. I digress.Chris: Right. Yeah. Anyways, so two points I want to make real quick on that is, one, we've recently released an open-source project called Carpenter, which is really cool in my purview because it looks at your Kubernetes file and says, “Oh, you want this to run on ARM instance.” And you can even go so far as to say, right, here's my limits, and it'll find an instance that fits those limits and add that to your cluster automatically. Run your pod on that compute as long as it needs to run and then if it's done, it'll downsize—eventually, kind of thing—your cluster.So, you can basically just throw a bunch of workloads at it, and it'll auto-detect what kind of compute you will need and then provision it for you, run it, and then be done. So, that is one-way folks are probably starting to save money running EKS is to adopt Carpenter as your autoscaler as opposed to the inbuilt Kubernetes autoscaler. Because this is instance-aware, essentially, so it can say, like, “Oh, your massive ARM application can run here,” because you know, thank you, Graviton. We have those processors in-house. And you know, you can run your ARM64 instances, you can run all the Intel workloads you want, and it'll right size the compute for your workloads.And I'll look at one container or all your containers, however you want to configure it. Secondly, the good folks over at Kubecost have opencost, which is the open-source version of Kubecost, basically. So, they have a service that you can run in your clusters that will help you say, “Hey, maybe this one notes too heavy; maybe this one notes too light,” and you know, give you some insights into Kubernetes spend that are a little bit more granular as far as usage and things like that go. So, those two projects right there, I feel like, will give folks an optimal savings experience when it comes to Kubernetes. But to your point, it's just compute, right? And that's really how we treat it, kind of, here internally is that it's a way to run… compute, Kubernetes, or ECS, or any of those tools.Corey: A fairly expensive one because ignoring entirely for a second the actual raw cost of compute, you also have the other side of it, which is in every environment, unless you are doing something very strange or pre-funding as a one-person startup in your spare time, your payroll costs will it—should—exceed your AWS bill by a fairly healthy amount. And engineering time is always more expensive than services time. So, for example, looking at EKS, I would absolutely recommend people use that rather than rolling their own because—Chris: Rolling their own? Yeah.Corey: —get out of that engineering space where your time is free. I assure you from a business context, it is not. So, there's always that question of what you can do to make things easier for people and do more of the heavy lifting.Chris: Yeah, and to your rather cheeky point that there's 17 ways to run a container on AWS, it is answering that question, right? Like those 17 ways, like, how much of this do you want to run yourself, you could run EKS distro on EC2 instances if you want full control over your environment.Corey: And then run IoT Greengrass core on top within that cluster—Chris: Right.Corey: So, I can run my own Lambda function runtime, so I'm not locked in. Also, DynamoDB local so I'm not locked into AWS. At which point I have gone so far around the bend, no one can help me.Chris: Well—Corey: Pro tip, don't do that. Just don't do that.Chris: But to your point, we have all these options for compute, and specifically containers because there's a lot of people that want to granularly say, “This is where my engineering team gets involved. Everything else you handle.” If I want EKS on Spot Instances only, you can do that. If you want EKS to use Carpenter and say only run ARM workloads, you can do that. If you want to say Fargate and not have anything to manage other than the container file, you can do that.It's how much does your team want to manage? That's the customer obsession part of AWS coming through when it comes to containers is because there's so many different ways to run those workloads, but there's so many different ways to make sure that your team is right-sized, based off the services you're using.Corey: I do want to change gears a bit here because you are mostly known for a couple of things: the DevOps'ish newsletter because that is the oldest and longest thing you've been doing the time that I've known you; EKS, obviously. But when prepping for this show, I discovered you are now co-chair of the OpenGitOps project.Chris: Yes.Corey: So, I have heard of GitOps in the context of, “Oh, it's just basically your CI/CD stuff is triggered by Git events and whatnot.” And I'm sitting here going, “Okay, so from where you're sitting, the two best user interfaces in the world that you have discovered are YAML and Git.” And I just have to start with the question, “Who hurt you?”Chris: [laugh]. Yeah, I share your sentiment when it comes to Git. Not so much with YAML, but I think it's because I'm so used to it. Maybe it's Stockholm Syndrome, maybe the whole YAML thing. I don't know.Corey: Well, it's no XML. We'll put it that way.Chris: Thankfully, yes because if it was, I would have way more, like, just template files laying around to build things. But the—Corey: And rage. Don't forget rage.Chris: And rage, yeah. So, GitOps is a little bit more than just Git in IaC—infrastructure as Code. It's more like Justin Garrison, who's also on my team, he calls it infrastructure software because there's four main principles to GitOps, and if you go to opengitops.dev, you can see them. It's version one.So, we put them on the website, right there on the page. You have to have a declared state and that state has to live somewhere. Now, it's called GitOps because Git is probably the most full-featured thing to put your state in, but you could use an S3 bucket and just version it, for example. And make it private so no one else can get to it.Corey: Or you could use local files: copy-of-copy-of-this-thing-restored-parentheses-use-this-one-dot-final-dot-doc-dot-zip. You know, my preferred naming convention.Chris: Ah, yeah. Wow. Okay. [laugh]. Yeah.Corey: Everything I touch is terrifying.Chris: Yes. Geez, I'm sorry. So first, it's declarative. You declare your state. You store it somewhere. It's versioned and immutable, like I said. And then pulled automatically—don't focus so much on pull—but basically, software agents are applying the desired state from source. So, what does that mean? When it's—you know, the fourth principle is implemented, continuously reconciled. That means those software agents that are checking your desired state are actually putting it back into the desired state if it's out of whack, right? So—Corey: You're talking about agents running it persistently on instances, validating—Chris: Yes.Corey: —a checkpoint on a cron. How is this meaningfully different than a Puppet agent running in years past? Having spent I learned to speak publicly by being a traveling trainer for Puppet; same type of model, and in fact, when I was at Pinterest, we wound up having a fair bit—like, that was their entire model, where they would have—the Puppet's code would live in an S3 bucket that was then copied down, I believe, via Git, and then applied to the instance on a schedule. Like, that sounds like this was sort of a early days GitOps.Chris: Yeah, exactly. Right? Like so it's, I like to think of that as a component of GitOps, right? DevOps, when you talk about DevOps in general, there's a lot of stuff out there. There's a lot of things labeled DevOps that maybe are, or maybe aren't sticking to some of those DevOps core things that make you great.Like the stuff that Nicole Forsgren writes about in books, you know? Accelerate is on my desk for a reason because there's things that good, well-managed DevOps practices do. I see GitOps as an actual implementation of DevOps in an open-source manner because all the tooling for GitOps these days is open-source and it all started as open-source. Now, you can get, like, Flux or Argo—Argo, specifically—there's managed services out there for it, you can have Flux and not maintain it, through an add-on, on EKS for example, and it will reconcile that state for you automatically. And the other thing I like to say about GitOps, specifically, is that it moves at the speed of the Kubernetes Audit Log.If you've ever looked at a Kubernetes audit log, you know it's rather noisy with all these groups and versions and kinds getting thrown out there. So, GitOps will say, “Oh, there's an event for said thing that I'm supposed to be watching. Do I need to change anything? Yes or no? Yes? Okay, go.”And the change gets applied, or, “Hey, there's a new Git thing. Pull it in. A change has happened inGit I need to update it.” You can set it to reconcile on events on time. It's like a cron or it's like an event-driven architecture, but it's combined.Corey: How does it survive the stake through the heart of configuration management? Because before I was doing all this, I wasn't even a T-shaped engineer: you're broad across a bunch of things, but deep in one or two areas, and one of mine was configuration management. I wrote part of SaltStack, once upon a time—Chris: Oh.Corey: —due to a bunch of very strange coincidences all hitting it once, like, I taught people how to use Puppet. But containers ultimately arose and the idea of immutable infrastructure became a thing. And these days when we were doing full-on serverless, well, great, I just wind up deploying a new code bundle to the Lambdas function that I wind up caring about, and that is a immutable version replacement. There is no drift because there is no way to log in and change those things other than through a clear deployment of this as the new version that goes out there. Where does GitOps fit into that imagined pattern?Chris: So, configuration management becomes part of your approval process, right? So, you now are generating an audit log, essentially, of all changes to your system through the approval process that you set up as part of your, how you get things into source and then promote that out to production. That's kind of the beauty of it, right? Like, that's why we suggest using Git because it has functions, like, requests and issues and things like that you can say, “Hey, yes, I approve this,” or, “Hey, no, I don't approve that. We need changes.” So, that's kind of natively happening with Git and, you know, GitLab, GitHub, whatever implementation of Git. There's always, kind of—Corey: Uh, JIF-ub is, I believe, the pronunciation.Chris: JIF-ub? Oh.Corey: Yeah. That's what I'm—Chris: Today, I learned. Okay.Corey: Exactly. And that's one of the things that I do for my lasttweetinaws.com Twitter client that I build—because I needed it, and if other people want to use it, that's great—that is now deployed to 20 different AWS commercial regions, simultaneously. And that is done via—because it turns out that that's a very long to execute for loop if you start down that path—Chris: Well, yeah.Corey: I wound up building out a GitHub Actions matrix—sorry a JIF-ub—actions matrix job that winds up instantiating 20 parallel builds of the CDK deploy that goes out to each region as expected. And because that gets really expensive with native GitHub Actions runners for, like, 36 cents per deploy, and I don't know how to test my own code, so every time I have a typo, that's another quarter in the jar. Cool, but that was annoying for me so I built my own custom runner system that uses Lambda functions as runners running containers pulled from ECR that, oh, it just runs in parallel, less than three minutes. Every time I commit something between I press the push button and it is out and running in the wild across all regions. Which is awesome and also terrifying because, as previously mentioned, I don't know how to test my code.Chris: Yeah. So, you don't know what you're deploying to 20 regions sometime, right?Corey: But it also means I have a pristine, re-composable build environment because I can—Chris: Right.Corey: Just automatically have that go out and the fact that I am making a—either merging a pull request or doing a direct push because I consider main to be my feature branch as whenever something hits that, all the automation kicks off. That was something that I found to be transformative as far as a way of thinking about this because I was very tired of having to tweak my local laptop environment to, “Oh, you didn't assume the proper role and everything failed again and you broke it. Good job.” It wound up being something where I could start developing on more and more disparate platforms. And it finally is what got me away from my old development model of everything I build is on an EC2 instance, and that means that my editor of choice was Vim. I use the VS Code now for these things, and I'm pretty happy with it.Chris: Yeah. So, you know, I'm glad you brought up CDK. CDK gives you a lot of the capabilities to implement GitOps in a way that you could say, like, “Hey, use CDK to declare I need four Amazon EKS clusters with this size, shape, and configuration. Go.” Or even further, connect to these EKS clusters to RDS instances and load balancers and everything else.But you put that state into Git and then you have something that deploys that automatically upon changes. That is infrastructure as code. Now, when you say, “Okay, main is your feature branch,” you know, things happen on main, if this were running in Kubernetes across a fleet of clusters or the globe-wide in 20 regions, something like Flux or Argo would kick in and say, “There's been a change to source, main, and we need to roll this out.” And it'll start applying those changes. Now, what do you get with GitOps that you don't get with your configuration?I mean, can you rollback if you ever have, like, a bad commit that's just awful? I mean, that's really part of the process with GitOps is to make sure that you can, A, roll back to the previous good state, B, roll forward to a known good state, or C, promote that state up through various environments. And then having that all done declaratively, automatically, and immutably, and versioned with an audit log, that I think is the real power of GitOps in the sense that, like, oh, so-and-so approve this change to security policy XYZ on this date at this time. And that to an auditor, you just hand them a log file on, like, “Here's everything we've ever done to our system. Done.” Right?Like, you could get to that state, if you want to, which I think is kind of the idea of DevOps, which says, “Take all these disparate tools and processes and procedures and culture changes”—culture being the hardest part to adopt in DevOps; GitOps kind of forces a culture change where, like, you can't do a CAB with GitOps. Like, those two things don't fly. You don't have a configuration management database unless you absolutely—Corey: Oh, you CAB now but they're all the comments of the pull request.Chris: Right. Exactly. Like, don't push this change out until Thursday after this other thing has happened, kind of thing. Yeah, like, that all happens in GitHub. But it's very democratizing in the sense that people don't have to waste time in an hour-long meeting to get their five minutes in, right?Corey: DoorDash had a problem. As their cloud-native environment scaled and developers delivered new features, their monitoring system kept breaking down. In an organization where data is used to make better decisions about technology and about the business, losing observability means the entire company loses their competitive edge. With Chronosphere, DoorDash is no longer losing visibility into their applications suite. The key? Chronosphere is an open-source compatible, scalable, and reliable observability solution that gives the observability lead at DoorDash business, confidence, and peace of mind. Read the full success story at snark.cloud/chronosphere. That's snark.cloud slash C-H-R-O-N-O-S-P-H-E-R-E.Corey: So, would it be overwhelmingly cynical to suggest that GitOps is the means to implement what we've all been pretending to have implemented for the last decade when giving talks at conferences?Chris: Ehh, I wouldn't go that far. I would say that GitOps is an excellent way to implement the things you've been talking about at all these conferences for all these years. But keep in mind, the technology has changed a lot in the, what 11, 12 years of the existence of DevOps, now. I mean, we've gone from, let's try to manage whole servers immutably to, “Oh, now we just need to maintain an orchestration platform and run containers.” That whole compute interface, you go from SSH to a Docker file, that's a big leap, right?Like, you don't have bespoke sysadmins; you have, like, a platform team. You don't have DevOps engineers; they're part of that platform team, or DevOps teams, right? Like, which was kind of antithetical to the whole idea of DevOps to have a DevOps team. You know, everybody's kind of in the same boat now, where we see skill sets kind of changing. And GitOps and Kubernetes-land is, like, a platform team that manages the cluster, and its state, and health and, you know, production essentially.And then you have your developers deploying what they want to deploy in when whatever namespace they've been given access to and whatever rights they have. So, now you have the potential for one set of people—the platform team—to use one set of GitOps tooling, and your applications teams might not like that, and that's fine. They can have their own namespaces with their own tooling in it. Like, Argo, for example, is preferred by a lot of developers because it has a nice UI with green and red dots and they can show people and it looks nice, Flux, it's command line based. And there are some projects out there that kind of take the UI of Argo and try to run Flux underneath that, and those are cool kind of projects, I think, in my mind, but in general, right, I think GitOps gives you the choice that we missed somewhat in DevOps implementations of the past because it was, “Oh, we need to go get cloud.” “Well, you can only use this cloud.” “Oh, we need to go get this thing.” “Well, you can only use this thing in-house.”And you know, there's a lot of restrictions sometimes placed on what you can use in your environment. Well, if your environment is Kubernetes, how do you restrict what you can run, right? Like you can't have an easily configured say, no open-source policy if you're running Kubernetes. [laugh] so it becomes, you know—Corey: Well, that doesn't stop some companies from trying.Chris: Yeah, that's true. But the idea of, like, enabling your developers to deploy at will and then promote their changes as they see fit is really the dream of DevOps, right? Like, same with production and platform teams, right? I want to push my changes out to a larger system that is across the globe. How do I do that? How do I manage that? How do I make sure everything's consistent?GitOps gives you those ways, with Kubernetes native things like customizations, to make consistent environments that are robust and actually going to be reconciled automatically if someone breaks the glass and says, “Oh, I need to run this container immediately.” Well, that's going to create problems because it's deviated from state and it's just that one region, so we'll put it back into state.Corey: It'll be dueling banjos, at some point. You'll try and doing something manually, it gets reverted automatically. I love that pattern. You'll get bored before the computer does, always.Chris: Yeah. And GitOps is very new, right? When you think about the lifetime of GitOps, I think it was coined in, like, 2018. So, it's only four years old, right? When—Corey: I prefer it to ChatOps, at least, as far as—Chris: Well, I mean—Corey: —implementation and expression of the thing.Chris: —ChatOps was a way to do DevOps. I think GitOps—Corey: Well, ChatOps is also a way to wind up giving whoever gets access to your Slack workspace root in production.Chris: Mmm.Corey: But that's neither here nor there.Chris: Mm-hm.Corey: It's yeah, we all like to pretend that's not a giant security issue in our industry, but that's a topic for another time.Chris: Yeah. And that's why, like, GitOps also depends upon you having good security, you know, and good authorization and approval processes. It enforces that upon—Corey: Yeah, who doesn't have one of those?Chris: Yeah. If it's a sole operation kind of deal, like in your setup, your case, I think you kind of got it doing right, right? Like, as far as GitOps goes—Corey: Oh, to be clear, we are 11 people and we do have dueling pull requests and all the rest.Chris: Right, right, right.Corey: But most of the stuff I talk about publicly is not our production stuff, so it really is just me. Just as a point of clarity there. I've n—the 11 people here do not all—the rest of you don't just sit there and clap as I do all the work.Chris: Right.Corey: Most days.Chris: No, I'm sure they don't. I'm almost certain they don't clap… for you. I mean, they would—Corey: No. No, they try and talk me out of it in almost every case.Chris: Yeah, exactly. So, the setup that you, Corey Quinn, have implemented to deploy these 20 regions is kind of very GitOps-y, in the sense that when main changes, it gets updated. Where it's not GitOps-y is what if the endpoint changes? Does it get reconciled? That's the piece you're probably missing is that continuous reconciliation component, where it's constantly checking and saying, “This thing out there is deployed in the way I want it. You know, the way I declared it to be in my source of truth.”Corey: Yeah, when you start having other people getting involved, there can—yeah, that's where regressions enter. And it's like, “Well, I know where things are so why would I change the endpoint?” Yeah, it turns out, not everyone has the state of the entire application in their head. Ideally it should live in—Chris: Yeah. Right. And, you know—Corey: —you know, Git or S3.Chris: —when I—yeah, exactly. When I think about interactions of the past coming out as a new DevOps engineer to work with developers, it's always been, will developers have access to prod or they don't? And if you're in that environment with—you're trying to run a multi-billion dollar operation, and your devs have direct—or one Dev has direct access to prod because prod is in his brain, that's where it's like, well, now wait a minute. Prod doesn't have to be only in your brain. You can put that in the codebase and now we know what is in your brain, right?Like, you can almost do—if you document your code, well, you can have your full lifecycle right there in one place, including documentation, which I think is the best part, too. So, you know, it encourages approval processes and automation over this one person has an entire state of the system in their head; they have to go in and fix it. And what if they're not on call, or in Jamaica, or on a cruise ship somewhere kind of thing? Things get difficult. Like, for example, I just got back from vacation. We were so far off the grid, we had satellite internet. And let me tell you, it was hard to write an email newsletter where I usually open 50 to 100 tabs.Corey: There's a little bit of internet out Californ-ie way.Chris: [laugh].Corey: Yeah it's… it's always weird going from, like, especially after pandemic; I have gigabit symmetric here and going even to re:Invent where I'm trying to upload a bunch of video and whatnot.Chris: Yeah. Oh wow.Corey: And the conference WiFi was doing its thing, and well, Verizon 5G was there but spotty. And well, yeah. Usual stuff.Chris: Yeah. It's amazing to me how connectivity has become so ubiquitous.Corey: To the point where when it's not there anymore, it's what do I do with myself? Same story about people pushing back against remote development of, “Oh, I'm just going to do it all on my laptop because what happens if I'm on a plane?” It's, yeah, the year before the pandemic, I flew 140,000 miles domestically and I was almost never hamstrung by my ability to do work. And my only local computer is an iPad for those things. So, it turns out that is less of a real world concern for most folks.Chris: Yeah I actually ordered the components to upgrade an old Nook that I have here and turn it into my, like, this is my remote code server, that's going to be all attached to GitHub and everything else. That's where I want to be: have Tailscale and just VPN into this box.Corey: Tailscale is transformative.Chris: Yes. Tailscale will change your life. That's just my personal opinion.Corey: Yep.Chris: That's not an AWS opinion or anything. But yeah, when you start thinking about your network as it could be anywhere, that's where Tailscale, like, really shines. So—Corey: Tailscale makes the internet work like we all wanted to believe that it worked.Chris: Yeah. And Wireguard is an excellent open-source project. And Tailscale consumes that and puts an amazingly easy-to-use UI, and troubleshooting tools, and routing, and all kinds of forwarding capabilities, and makes it kind of easy, which is really, really, really kind of awesome. And Tailscale and Kubernetes—Corey: Yeah, ‘network' and ‘easy' don't belong in the same sentence, but in this case, they do.Chris: Yeah. And trust me, the Kubernetes story in Tailscale, there is a lot of there. I understand you might want to not open ports in your VPC, maybe, but if you use Tailscale, that node is just another thing on your network. You can connect to that and see what's going on. Your management cluster is just another thing on the network where you can watch the state.But it's all—you're connected to it continuously through Tailscale. Or, you know, it's a much lighter weight, kind of meshy VPN, I would say, if I had to sum it up in one sentence. That was not on our agenda to talk about at all. Anyways. [laugh]Corey: No, no. I love how many different topics we talk about on these things. We'll have to have you back soon to talk again. I really want to thank you for being so generous with your time. If people want to learn more about what you're up to and how you view these things, where can they find you?Chris: Go to ChrisShort.net. So, Chris Short—I'm six-four so remember, it's Short—dot net, and you will find all the places that I write, you can go to devopsish.com to subscribe to my newsletter, which goes out every week. This year. Next year, there'll be breaks. And then finally, if you want to follow me on Twitter, Chris Short: at @ChrisShort on Twitter. All one word so you see two s's. Like, it's okay, there's two s's there.Corey: Links to all of that will of course be in the show notes. It's easier for people to do the clicky-clicky thing as a general rule.Chris: Clicky things are easier than the wordy things, yes.Corey: Says the Kubernetes guy.Chris: Yeah. Says the Kubernetes guy. Yeah, you like that, huh? Like I said, Argo gives you a UI. [laugh].Corey: Thank you [laugh] so much for your time. I really do appreciate it.Chris: Thank you. This has been fun. If folks have questions, feel free to reach out. Like, I am not one of those people that hides behind a screen all day and doesn't respond. I will respond to you eventually.Corey: I'm right here, Chris. Come on, come on. You're calling me out in front of myself. My God.Chris: Egh. It might take a day or two, but I will respond. I promise.Corey: Thanks again for your time. This has been Chris Short, senior developer advocate at AWS. I'm Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice and if it's YouTube, click the thumbs-up button. Whereas if you've hated this podcast, same thing, smash the buttons five-star review and leave an insulting comment that is written in syntactically correct YAML because it's just so easy to do.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

Old man yells at cloud - Oder: Wie managed man seine Infrastruktur mit Stil (und Software)Anders als gewohnt nimmt in dieser Episode Andy die Dozenten-Rolle ein und beantwortet Wolfgang all seine Fragen zum Thema Infrastructure as Code. Wir klären wozu man das ganze eigentlich braucht, was Terraform und Pulumi ist, klären über einen weit verbreiteten Mythos auf, wo der Unterschied zwischen Infrastructure Orchestration und Configuration Management ist, was das beste Configuration Management Tool ist und wo es Herausforderungen bei der Verwendung von Infrastructure as Code gibt.Bonus: Was ein Data Engineer ist, ob Wolfgang Holz-Clogs trägt und wie Deutschland mit dem 9€ Ticket umgeht.Feedback an stehtisch@engineeringkiosk.dev oder via Twitter an https://twitter.com/EngKioskLinks“Old man yells at cloud” Herkunft: https://knowyourmeme.com/memes/old-man-yells-at-cloud Zeit "Servus Grüezi Hallo" Podcast "Die passen doch niemals alle rein": https://www.zeit.de/gesellschaft/2022-06/9-euro-ticket-klimaticket-oesterreich-politikpodcast Terraform: https://www.terraform.io/Pulumi: https://www.pulumi.com/AWS Cloud Formation: https://aws.amazon.com/de/cloudformation/Saltstack: https://github.com/saltstack/saltAnsible: https://www.ansible.com/Puppet: https://puppet.com/Chef: https://www.chef.io/products/chef-infraCFEngine: https://cfengine.com/Terraform "depends_on for providers" bug: https://github.com/hashicorp/terraform/issues/2430Hetzner Terraform Provider: https://registry.terraform.io/providers/hetznercloud/hcloud Sprungmarken(00:00:00) Intro(00:00:46) Was ist ein Data Engineer? Und wie spielt ein Data Analyst und Data Scientist da rein?(00:05:00) Das 9€ Ticket in Deutschland und das Klima-Ticket in Österreich(00:08:23) Heutiges Thema: Infrastructure as Code(00:09:27) Was ist DevOps und warum es keine Stellenbeschreibung oder Person ist(00:10:52) Warum niemand sich selbst als Spezialist ansieht(00:11:44) Was ist eigentlich Infrastructure as Code?(00:17:41) Was bringt mir Infrastructure as Code eigentlich?(00:20:55) Mythos: Code ist einmal definiert, und auf alle Cloud Provider anwendbar(00:22:25) Warum nutzen wir dann nicht die Cloud spezifische Sprache, wie zum Beispiel Cloud Formation von Amazon Web Services? (00:24:08) Bin ich schneller, wenn ich die Cloud migrieren möchte und die Infrastruktur bereits als Code definiert habe?(00:27:05) Zurück zu: Warum braucht man Infrastructure as Code eigentlich?(00:29:14) Wo ist der Unterschied zwischen Ansible und Terraform?(00:34:12) Wird Configuration Management wie Ansible heutzutage noch gebraucht?(00:36:25) Was sind Terraform Provider?(00:39:53) Was ist denn das beste Configuration Management-Tool: Ansible, Salt, Chef, Puppet, CFEngine oder Bash-Scripte?(00:45:34) Was sind Nachteile von Infrastructure as Code?(00:57:55) Zusammenfassung der Episode und OutroHostsWolfgang Gassler (https://twitter.com/schafele)Andy Grunwald (https://twitter.com/andygrunwald)Engineering Kiosk Podcast: Anfragen an stehtisch@engineeringkiosk.dev oder via Twitter an https://twitter.com/EngKiosk

About TomaszTomasz is a Frontend Engineer at Stedi, Co-Founder/Head of React at Cloudash, egghead.io instructor with over 200 lessons published, a tech speaker, an AWS Community Hero and a lifelong learner.Links Referenced: Cloudash: https://cloudash.dev/ Twitter: https://twitter.com/tlakomy TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by Honeycomb. When production is running slow, it's hard to know where problems originate. Is it your application code, users, or the underlying systems? I've got five bucks on DNS, personally. Why scroll through endless dashboards while dealing with alert floods, going from tool to tool to tool that you employ, guessing at which puzzle pieces matter? Context switching and tool sprawl are slowly killing both your team and your business. You should care more about one of those than the other; which one is up to you. Drop the separate pillars and enter a world of getting one unified understanding of the one thing driving your business: production. With Honeycomb, you guess less and know more. Try it for free at honeycomb.io/screaminginthecloud. Observability: it's more than just hipster monitoring.Corey: This episode is sponsored in part by our friends at ChaosSearch. You could run Elasticsearch or Elastic Cloud—or OpenSearch as they're calling it now—or a self-hosted ELK stack. But why? ChaosSearch gives you the same API you've come to know and tolerate, along with unlimited data retention and no data movement. Just throw your data into S3 and proceed from there as you would expect. This is great for IT operations folks, for app performance monitoring, cybersecurity. If you're using Elasticsearch, consider not running Elasticsearch. They're also available now in the AWS marketplace if you'd prefer not to go direct and have half of whatever you pay them count towards your EDB commitment. Discover what companies like Equifax, Armor Security, and Blackboard already have. To learn more, visit chaossearch.io and tell them I sent you just so you can see them facepalm, yet again.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. It's always a pleasure to talk to people who ask the bold questions. One of those great bold questions is, what if CloudWatch's web page didn't suck? It's a good question. It's one I ask myself all the time.And then I stumbled across a product that wound up solving this for me, and I'm a happy customer. To be clear, they're not sponsoring anything that I do, nor should they. It's one of those bootstrapped, exciting software projects called Cloudash. Today, I'm joined by the Head of React at Cloudash, Tomasz Łakomy. Tomasz, thank you for joining me.Tomasz: It's a pleasure to be here.Corey: So, where did this entire idea come from? Because I sit and I get upset every time I have to go into the CloudWatch dashboard because first, something's broken. In an ideal scenario, I don't have to care about monitoring or observability or anything like that. But then it's quickly overshadowed by the fact that this interface is terrible. And the reason I know it's terrible is that every time I'm in there, I feel dumb.My belief is—for the longest time, I thought that was a problem with me. But no, invariably, when you wind up working with something and consistently finding it a bad—you don't know enough to solve for it, it's not you. It is, in fact, the signs of a poorly designed experience, start to finish. “You should be smarter to use this tool,” is very rarely correct. And there are a bunch of observability tools and monitoring tools for serverless things that have made sense over the years and made this easier, but one of the most—and please don't take this the wrong way—stripped down, bare essentials of just the facts, style of presentation is Cloudash. It's why I continue to pay for it every month with a smile on my face. How did you get here from there?Tomasz: Yeah that's a good question. I would say that. Cloudash was born out of desire for simple things to be simple. So, as you mentioned, Cloudash is basically the monitoring and troubleshooting tool for serverless applications, made for serverless developers because I am very much into serverless space, as is Maciej Winnicki, who is the another half of Cloudash team. And, you know, the whole premise of serverless was things are going to be simpler, right?So, you know, you have a bunch of code, you're going to dump it into a Lambda function, and that's it. You don't have to care about servers, you don't have to care about, you know, provisioning stuff, you don't have to care about maintenance, and so on. And that is not exactly true because why PagerDuty still continues to be [unintelligible 00:02:56] business even in serverless spaces. So, you will get paged every now and then. The problem is—what we kind of found is once you have an incident—you know, PagerDuty always tends to call it in the middle of the night; it's never, like, 11 a.m. during the workday; it's always the middle of the night.Corey: And no one's ever happy when it calls them either. It's, “Ah, hell.” Whatever it rings, it's yeah, the original Call of Duty. PagerDuty hooked up to Nagios. I am old enough to remember those days.Tomasz: [unintelligible 00:03:24] then business, like, imagine paying for something that's going to wake you up in the middle of the night. It doesn't make sense. In any case—Corey: “So, why do you pay for that product? Because it's really going to piss me off.” “Okay, well… does that sound like a good business to you? Well, AWS seems to think so. No one's happy working with that stuff.” “Fair. Fair enough.”Tomasz: So, in any case, like we've established an [unintelligible 00:03:43]. So you wake up, you go to AWS console because you saw a notification that this-and-this API has, you know, this threshold was above it, something was above the threshold. And then you go to the CloudWatch console. And then you see, okay, those are the logs, those are the metrics. I'm going to copy this request ID. I'm going to go over here. I'm going to go to X-Ray.And again, it's 3 a.m. so you don't exactly remember what do you investigate; you have, like, ten minutes. And this is a problem. Like, we've kind of identified that it's not simple to do these kinds of things, too—it's not simple to open something and have an understanding, okay, what exactly is happening in my serverless app at this very moment? Like, what's going on?So, we've built that. So, Cloudash is a desktop app; it lives on your machine, which is a single pane of glass. It's a single pane of glass view into your serverless system. So, if you are using CloudFormation in order to provision something, when you open Cloudash, you're going to see, you know, all of the metrics, all the Lambda functions, all of the API Gateways that you have provisioned. As of yesterday, API Gateway is no longer cool because they did launch the direct integration, so you have—you can call Lambda functions with [crosstalk 00:04:57]—Corey: Yeah, it's the one they released, and then rolled back and somehow never said a word—because that's an AWS messaging story, and then some—right around re:Invent last year. And another quarter goes by and out it goes.Tomasz: It's out yesterday.Corey: Yeah, it's terrific. I love that thing. The only downside to it is, ah, you have to use one of their—you have to use their domain; no custom domain support. Really? Well, you can hook up CloudFront to it, but the pricing model that way makes it more expensive than API Gateway.Okay, so I could use Cloudflare in front of it, and then it becomes free, so I bought a domain just for that purpose. That's right, my serverl—my direct Lambda URLs now live behind the glorious domain of cheapass.cloud because of course. They are. It's a day-one product from AWS, so of course, it's not feature-complete.But one of the things I like about the serverless model, and it's also a challenge when it comes to troubleshooting stuff is that it's very much set it and forget it style because serverless in many cases, at least the way that I tend to use it, is back-office stuff, its back-end things, it's processing on things that are not necessarily always direct front and center. So, these things can run on their own for years until finally, you find a strange bug in a new use case, or you want to go and change something. And then it's how the hell did this ever work? And it's still working, kind of, but what fool built this? Of course, it was me; it's always me.But what happened here? You're basically excavating your own legacy code, trying to understand what's going on. And so, you're already upset then. Cloudash makes this easier to find the things, to navigate through a whole bunch of different accounts. And there are a bunch of decisions that you made while building the app that are so clearly correct, that I get actively annoyed when others don't because oh, it looks at your AWS configuration file in your user home directory. Great, awesome. It's a desktop app, but it still consults that file. Yay, integration between ClickOps and the terminal. Wonderful.But ah, use SSO for a lot of stuff, so that's going to fix your little red wagon. I click on that app, and suddenly, bam, a browser opens asking me to log in and authenticate, allow the request. It works, and then suddenly, it goes back to doing exactly what you'd expect it to. It's really nice. The affordances behind this are glorious.Tomasz: Like I said, one of our kind of design goals when building Cloudash was to make simple things simple again. The whole purpose is to make sure that you can get into the root cause of an issue within, like, five minutes, if not less. And this is kind of the app that you're going to tend to open whenever that—as I said, because some of the systems can be around for, like, ages, literally without any incident whatsoever, then the data is going to change because somebody [unintelligible 00:07:30] got that the year is 2020 and off you go, we have an incident.But what's important about Cloudash is that we don't send logs anywhere. And that's kind of important because you don't pay for [PUT 00:07:42] metric API because we are not sending those logs anywhere. If you install Cloudash on your machine, we are not going to get your logs from the last ten years, put them in into a system, charge you for that, just so you are able to, you know, find out what happened in this particular hour, like, two weeks ago. We genuinely don't care about your logs; we have enough of our own logs at work to, you know, to analyze, to investigate, and so on; we are not storing them anywhere.In fact, you know, whatever happens on your machine stays on the machine. And that is partially why this is a desktop app. Because we don't want to handle your credentials. We don't—absolutely, we don't want you to give us any of your credentials or access keys, you know, whatever. We don't want that.So, that is why you install Cloudash, it's going to run on your machine, it's going to use your local credentials. So, it's… effectively, you could say that this is a much more streamlined and much more laser-focused browser or like, an eye into AWS systems, which live on the serverless side of things.Corey: I got to deal with it in a bit of an interesting way, recently. I have a detector in my company's production AWS org, to detect when ClickOps is afoot. Now, I'm a big proponent of ClickOps, but I also want to know what's going on, so I have a whole thing that [runs detects 00:09:04] when people are doing things in the console versus via API. And it alerts on certain subsets of them. I had to build a special case for the user agent string coming out of Cloudash because no, no, this is an app, this is not technically ClickOps—it is also read-only, which is neither here nor there, to my understanding.But it was, “Oh yeah, this is effectively an Electron app.” It just wraps, effectively, a browser and presents that as an application. And cool. From my perspective, that's an implementation detail. It feels like a native app—because it is—and I can suddenly see the things I care about in a way that is much more straightforward without having to have four different browser tabs open where, okay, here's the CloudTrail log for this thing, here's the metrics next to it. Oh, those are two separate windows already, and so on and so forth. It just makes hunting down to the obnoxious problems so much nicer.It's also, you're one of those rare products where if I don't use it for a month, I don't get the bill at the end of the month and think, “Ooh, that's going to—did I waste the money?” It's no, nice. I had a whole month where I didn't have to mess with this. It's great.Tomasz: Exactly. I feel like, you know, it's one of those systems where, as you said, we send you an email at the end of every month that we're going to charge you X dollars for the month—by the way, we have fixed pricing and then you can cancel anytime—and it's like one of those things that, you know, I didn't have to open this up for a month. This is awesome because I didn't have any incidents. But I know whenever again, PagerDuty is going to decide, “Hey, dude, wake up. You know, if slept for three hours. That is definitely long enough,” then you know that; you know, this app is there and you can use that.We very much care about, you know, building this stuff, not only for our customers, but we also use that on a daily basis. In fact, I… every single time that I have to—I want to investigate something in, like, our serverless systems at Stedi because everything that we do at work, at Stedi, since this incident serverless paradigm. So, I tend to open Cloudash, like, 95% of the time whenever I want to investigate something. And whenever I am not able to do something in Cloudash, this goes, like, straight to the top of our, you know, issue lists or backlog or whatever you want to call it. Because we want to make this product, not only awesome, you know, for customers to buy a [unintelligible 00:11:22] or whatever, but we also want to be able to use that on a daily basis.And so far, I think we've kind of succeeded. But then again, we have quite a long way to go because we have more ideas, than we have the time, definitely, so we have to kind of prioritize what exactly we're going to build. So, [unintelligible 00:11:39] integrations with alarms. So, for instance, we want to be able to see the alarms directly in the Cloudash UI. Secondly, integration with logs insights, and many other ideas. I could probably talk for hours about what we want to build.Corey: I also want to point out that this is still your side gig. You are by day a front-end engineer over at Stedi, which has a borderline disturbing number of engineers with side gigs, generally in the serverless space, doing interesting things like this. Dynobase is another example, a DynamoDB desktop client; very similar in some respects. I pay for that too. Honestly, for a company in Stedi's space, which is designed as basically a giant API for deep, large enterprise business stuff, there's an awful lot of stuff for small-scale coming out of that.Like, I wind up throwing a disturbing amount of money in the general direction of Stedi for not being their customer. But there's something about the culture that you folks have built over there that's just phenomenal.Tomasz: Yeah. For the record, you know, having a side gig is another part of interview process at Stedi. You don't have to have [laugh] a side project, but yeah, you're absolutely right, you know, the amount of kind of side projects, and you know, some of those are monetized, as you mentioned, you know, Cloudash and Dynobase and others. Some of those—because for instance, you talked to Aidan, I think a couple of weeks ago about his shenanigans, whenever you know, AWS is going to announce something he gets in and try to [unintelligible 00:13:06] this in the most amusing ways possible. Yeah, I mean, I could probably talk for ages about why Stedi is by far the best company I've ever worked at, but I'm going to say this: that this is the most talented group of people I've ever met, and myself, honestly.And, you know, the fact that I think we are the second largest, kind of, group of AWS experts outside of AWS because the density of AWS Heroes, or ex-AWS employees, or people who have been doing cloud stuff for years, is frankly, massive, I tend to learn something new about cloud every single day. And not only because of the Last Week in AWS but also from our Slack.Corey: This episode is sponsored by our friends at Oracle Cloud. Counting the pennies, but still dreaming of deploying apps instead of “Hello, World” demos? Allow me to introduce you to Oracle's Always Free tier. It provides over 20 free services and infrastructure, networking, databases, observability, management, and security. And—let me be clear here—it's actually free. There's no surprise billing until you intentionally and proactively upgrade your account. This means you can provision a virtual machine instance or spin up an autonomous database that manages itself, all while gaining the networking, load balancing, and storage resources that somehow never quite make it into most free tiers needed to support the application that you want to build. With Always Free, you can do things like run small-scale applications or do proof-of-concept testing without spending a dime. You know that I always like to put asterisks next to the word free? This is actually free, no asterisk. Start now. Visit snark.cloud/oci-free that's snark.cloud/oci-free.Corey: There's something to be said for having colleagues that you learn from. I have never enjoyed environments where I did not actively feel like the dumbest person in the room. That's why I love what I do now. I inherently am. I have to talk about so many different things, that whenever I talk to a subject matter expert, it is a certainty that they know more about the thing than I do, with the admitted and depressing exception of course of the AWS bill because it turns out the reason I had to start becoming the expert in that was because there weren't any. And here we are now.I want to talk as well about some of—your interaction outside of work with AWS. For example, you've been an Egghead instructor for a while with over 200 lessons that you published. You're an AWS Community Hero, which means you have the notable distinction of volunteering for a for-profit company—good work—no, the community is very important. It's helping each other make sense of the nonsense coming out of there. You've been involved within the ecosystem for a very long time. What is it about, I guess—the thing I'm wondering about myself sometimes—what is it about the AWS universe that drew you in, and what keeps you here?Tomasz: So, give you some context, I've started, you know, learning about the cloud and AWS back in early-2019. So, fun fact: Maciej Winnicki—again, the co-founder of Cloudash—was my manager at the time. So, we were—I mean, the company I used to work for at the time, OLX Group, we are in the middle of cloud transformation, so to speak. So, going from, you know, on-premises to AWS. And I was, you know, hired as a senior front-end engineer doing, you know, all kinds of front-end stuff, but I wanted to grow, I wanted to learn more.So, the idea was, okay, maybe you can get AWS Certified because, you know, it's one of those corporate goals that you have to have something to put that checkbox next to it. So, you know, getting certified, there you go, you have a checkbox. And off you go. So, I started, you know, diving in, and I saw this whole ocean of things that, you know, I was not entirely aware of. To be fair, at the time I knew about this S3, I knew that you can put a file in an S3 bucket and then you can access it from the internet. This is, like, the [unintelligible 00:16:02] idea of my AWS experiences.Corey: Ideally, intentionally, but one wonders sometimes.Tomasz: Yeah, exactly. That is why you always put stuff as public, right? Because you didn't have to worry about who [unintelligible 00:16:12] [laugh] public [unintelligible 00:16:15]. No, I'm kidding, of course. But still, I think what's [unintelligible 00:16:20] to AWS is what—because it is this endless ocean of things to learn and things to play with, and, you know, things to teach.I do enjoy teaching. As you said, I have quite a lot of, you know, content, videos, blog posts, conference talks, and a bunch of other stuff, and I do that for two reasons. You know, first of all, I tend to learn the best by teaching, so it helps me very much, kind of like, solidify my own knowledge. Whenever I record—like, I have two courses about CDK, you know, when I was recording those, I definitely—that kind of solidify my, you know, ideas about CDK, I get to play with all those technologies.And secondly, you know, it's helpful for others. And, you know, people have opinions about certificates, and so on and so forth, but I think that for somebody who's trying to get into either the tech industry or, you know, cloud stuff in general, being certified helps massively. And I've heard stories about people who are basically managed to double or triple their salaries by going into tech, you know, with some of those certificates. That is why I strongly believe, by the way, that those certificates should be free. Like, if you can pass the exam, you shouldn't have to worry about this $150 of the fee.Corey: I wrote a blog post a while back, “The Dumbest Dollars a Cloud Provider Can Make,” and it's charging for training and certification because if someone's going to invest that kind of time in learning your platform, you're going to try and make $150 bucks off them? Which in some cases, is going to put people off from even beginning that process. “What cloud provider I'm not going to build a project on?” Obviously, the one I know how to work with and have a familiarity with, in almost every case. And the things you learn in your spare time as an independent learner when you get a job, you tend to think about your work the same way. It matters. It's an early on-ramp that pays off down the road and the term of years.I used to be very anti-cert personally because it felt like I was jumping through hoops, and paying, in some cases, for the privilege. I had a CCNA for a while from Cisco. There were a couple of smaller companies, SaltStack, for example, that I got various certifications from at different times. And that was sort of cheating because I helped write the software, but that's neither here nor there. It's the—and I do have a standing AWS cert that I get a different one every time—mine is about to expire—because it gets me access to lounges at physical events, which is the dumbest of all reasons to get certs, but here you go. I view it as the $150 lounge pass with a really weird entrance questionnaire.But in my case it certs don't add anything to what I do. I am not the common case. I am not early in my career. Because as you progress through your career, things—there needs to be a piece of paper that says you know things, and early on degree or certifications are great at that. In the time it becomes your own list of experience on your resume or CV or LinkedIn or God knows what. Polywork if you're doing it the right way these days.And it shows a history of projects that are similar in scope and scale and impact to the kinds of problems that your prospective employer is going to have to solve themselves. Because the best answer to hear—especially in the ops world—when there's a problem is, “Oh, I've seen this before. Here's how you fix it.” As opposed to, “Well, I don't know. Let me do some research.”There's value to that. And I don't begrudge anyone getting certs… to a point. At least that's where I sit on it. At some point when you have 25 certs, it's when you actually do any work? Because it's taking the tests and learning all of these things, which in many ways does boil down to trivia, it stands in counterbalance to a lot of these things.Tomasz: Yeah. I mean, I definitely, totally agree. I remember, you know, going from zero to—maybe not Hero; I'm not talking about AWS Hero—but going from zero to be certified, there was the Solutions Architect Associate. I think it took me, like, 200 hours. I am not the, you know, the brightest, you know, the sharpest tool in the shed, so it probably took me, kind of, somewhat more.I think it's doable in, like, 100 hours, but I tend to over-prepare for stuff, so I didn't actually take the actual exam until I was able to pass the sample exams with, like, 90% pass, just to be extra sure that I'm actually going to pass it. But still, I think that, you know, at some point, you probably should focus on, you know, getting into the actual stuff because I hold two certificates, you know, one of those is going to expire, and I'm not entirely sure if I want to go through the process again. But still, if AWS were to introduce, like, a serverless specialty exam, I would be more than happy to have that. I genuinely enjoy, kind of, serverless, and you know, the fact that I would be able to solidify my knowledge, I have this kind of established path of the things that I should learn about in order to get this particular certificate, I think this could be interesting. But I am not probably going to chase all the 12 certificates.Maybe if AWS IQ was available in Poland, maybe that would change because I do know that with IQ, those certs do matter. But as of [unintelligible 00:21:26] now, I'm quite happy with my certs that I have right now.Corey: Part of the problem, too, is the more you work with these things, the harder it becomes to pass the exams, which sounds weird and counterintuitive, but let me use myself as an example. When I got the cloud practitioner cert, which I believe has lapsed since then, and I got one of the new associate-level betas—I'll keep moving up the stack until I start failing exams. But I got a question wrong on the cloud practitioner because it was, “How long does it take to restore an RDS database from a snapshot backup?” And I gave the honest answer of what I've seen rather than what it says in the book, and that honest answer can be measured in days or hours. Yeah.And no, that's not the correct answer. Yeah, but it is the real one. Similarly, a lot of the questions get around trivia, syntax of which of these is the correct argument, and which ones did we make up? It's, I can explain in some level of detail, virtually every one of AWS has 300 some-odd services to you. Ask me about any of them, I could tell you what it is, how it works, how it's supposed to work and make a dumb joke about it. Fine, whatever.You'll forgive me if I went down that path, instead of memorizing what is the actual syntax of this YAML construct inside of a CloudFormation template? Yeah, I can get the answer to that question in the real world, with about ten seconds of Googling and we move on. That's the way most of us learn. It's not cramming trivia into our heads. There's something broken about the way that we do certifications, and tech interviews in many cases as well.I look back at some of the questions I used to ask people for Linux sysadmin-style jobs, and I don't remember the answer to a lot of these things. I could definitely get back into it, but if I went through one of these interviews now, I wouldn't get the job. One would argue I shouldn't because of my personality, but that's neither here nor there.Tomasz: [laugh]. I mean, that's why you use CDK, so you'd have to remember random YAML comments. And if you [unintelligible 00:23:26] you don't have YAML anymore. [unintelligible 00:23:27].Corey: Yes, you're quite the CDK fanboy, apparently.Tomasz: I do like CDK, yes. I don't like, you know, mental overhead, I don't like context switching, and the way we kind of work at Stedi is everything is written in TypeScript. So, I am a front-end engineer, so I do stuff in the front-end line in TypeScript, all of our Lambda functions are written in TypeScript, and our [unintelligible 00:23:48] is written in TypeScript. So, I can, you know, open up my Visual Studio Code and jump between all of those files, and the language stays the same, the syntax stays the same, the tools stay the same. And I think this is one of the benefits of CDK that is kind of hard to replicate otherwise.And, you know, people have many opinions about the best to deploy infrastructure in the cloud, you know? The best infrastructure-as-code tool is the one that you use at work or in your private projects, right? Because some people enjoy ClickOps like you do; people—Corey: Oh yeah.Tomasz: Enjoy CloudFormation by hand, which I don't; people are very much into Terraform or Serverless Framework. I'm very much into CDK.Corey: Or the SAM CLI, like, three or four more, and I use—Tomasz: Oh, yeah. [unintelligible 00:24:33]—Corey: —all of these things in various ways in some of my [monstrous 00:24:35] projects to keep up on all these things. I did an exploration with the CDK. Incidentally, I think you just answered why I don't like it.Tomasz: Because?Corey: Because it is very clear that TypeScript is a first-class citizen with the CDK. My language of choice is shitty bash because, grumpy old sysadmin; it happens. And increasingly, that is switching over to terrible Python because I'm very bad at that. And the problem that I run into as I was experimenting with this is, it feels like the Python support is not fully baked, most people who are using the CDK are using a flavor of JavaScript and, let's be very clear here, the every time I have tried to explore front-end, I have come away more confused than I was when I started, part of me really thinks I should be learning some JavaScript just because of its versatility and utility to a whole bunch of different problems. But it does not work the way I think, on some level, that it should because of my own biases and experiences. So, if you're not a JavaScript person, I think that you have a much rockier road with the CDK.Tomasz: I agree. Like I said, I tend to talk about my own experiences and my kind of thoughts about stuff. I'm not going to say that, you know, this tool or that tool is the best tool ever because nothing like that exists. Apart from jQuery, which is the best thing that ever happened to the web since, you know, baked bread, honestly. But you are right about CDK, to the best of my knowledge, kind of, all the other languages that are supported by CDK are effectively transpiled down from TypeScript. So it's, like, first of all, it is written in TypeScript, and then kind of the Python, all of the other languages… kind of come second.You know, and afterwards, I tend to enjoy CDK because as I said, I use TypeScript on a daily basis. And you know, with regards to front-end, you mentioned that you are, every single time you is that you end up being more confused. It never goes away. I've been doing front-end stuff for years, and it's, you know, kind of exactly the same. Fun story, I actually joined Cloudash because, well, Maciej started working on Cloudash alone, and after quite some time, he was so frustrated with the modern front-end landscape that he asked me, “Dude, you need to help me. Like, I genuinely need some help. I am tired of React. I am tired of React hooks. This is way too complex. I want to go back to doing back-end stuff. I want to go back, you know, thinking about how we're going to integrate with all those APIs. I don't want to do UI stuff anymore.”Which was kind of like an interesting shift because I remember at the very beginning of my career, where people were talking about front-end—you know, “Front-end is not real programming. Front-end is, you know, it's easy, it's simple. I can learn CSS in an hour.” And the amount of people who say that CSS is easy, and are good at CSS is exactly zero. Literally, nobody who's actually good at CSS says that, you know, CSS, or front-end, or anything like that is easy because it's not. It's incredibly complex. It's getting probably more and more complex because the expectations of our front-end UIs [unintelligible 00:27:44].Corey: It's challenging, it is difficult, and one of the things I find most admirable about you is not even your technical achievements, it's the fact that you're teaching other people to do this. In fact, this gets to the last point I want to cover on our conversation today. When I was bouncing topic ideas off of you, one of the points you brought up that I'm like, “Oh, we're keeping that and saving that for the end,” is why—to your words—why speaking at tech events gets easier, but never easy. Let's dive into that. Tell me more about it.Tomasz: Basically, I've accidentally kickstarted my career by speaking at meetups which later turned into conferences, which later turned into me publishing courses online, which later turned into me becoming an AWS Hero, and here we are, you know, talking to each other. I do enjoy, you know, going out in public and speaking and being on stage. I think, you know, if somebody has, kind of, the heart, the ability to do that, I do strongly recommend, you know, giving it a shot, not only to give, like, an honestly life-changing experience because the first time you go in front of hundreds of people, this is definitely, you know, something that's going to shake you, while at the same time acknowledging that this is absolutely, definitely not for everyone. But if you are able to do that, I think this is definitely worth your time. But as you said—by quoting me—that it gets easier, so every single time you go on stage, talk at a meetup or at a conference or online conferences—which I'm not exactly a fan of, for the record—it's—Corey: It's too much like work, too much like meetings. There's nothing different about it.Tomasz: Yeah, exactly. Like, there's no journey. There's no adventure in online conferences. I know that, of course, you know, given all of that, you know, we had to kind of switch to online conferences for quite some time where I think we are pretending that Covid is not a thing anymore, so we, you know, we're effectively going back, but kind of the point I wanted to make is that I am a somewhat experienced public speaker—I'd like to say that because I've been doing that for years—but I've been, you know, talking to people who actually get paid to speak at the conferences, to actually kind of do that for a living, and they all say the same thing. It gets simpler, it gets easier, but it's never freaking easy, you know, to go out there, and you know, to share whatever you've learned.Corey: I'm one of those people. I am a paid public speaker fairly often, even ignoring the podcast side, and I've spoken on conference stages a couple hundred times at least. And it does get easier but never easy. That's a great way of framing it. You… I get nervous before every talk I give.There are I think two talks I've given that I did not have an adrenaline hit and nervous energy before I went onstage, and both of those were duds. Because I think that it's part of the process, at least for me. And it's like, “Oh, how do you wind up not being scared for before you go on stage?” You don't. You really don't.But if that appeals to you and you enjoy the adrenaline rush of the rest, do it. If you're one of those people who've used public speaking as, “I would prefer death over that,” people are more scared of public speaking their death, in some cases, great. There are so many ways to build audiences and to reach people that fine, if you don't like doing it on stage, don't force yourself to. I'd say try it once; see how it feels meetups are great for this.Tomasz: Yeah. Meetups are basically the best way to get started. I'm yet to meet a meetup, either, you know, offline or online, who is not looking for speakers. It's always quite the opposite, you know? I was, you know, co-organizing a meetup in my city here in Poznań, Poland, and the story always goes like this: “Okay, we have a date. We have a venue. Where are the speakers?” And then you know, the tumbleweed is going to roll across the road and, “Oh, crap, we don't have any speakers.” So, we're going to try to find some, reach out to people. “Hey, I know that you did this fantastic project at your workplace. Come to us, talk about this.” “No, I don't want to. You know, I'm not an expert. I am, you know, I have on the 50 years of experience as an engineer. This is not enough.” Like I said, I do strongly recommend it, but as you said, if you're more scared of public speaking than, like, literally dying, maybe this is not for you.Corey: Yeah. It comes down to stretching your limits, finding yourself interesting. I find that there are lots of great engineers out there. The ones that I find myself drawn to are the ones who aren't just great at building something, but at storytelling around the thing that they are built of, yes, you build something awesome, but you have to convince me to care about it. You have to show me the thing that got you excited about this.And if you can't inspire that excitement in other people, okay. Are you really excited about it? Or what is the story here? And again, it's a different skill set. It is not for everyone, but it is absolutely a significant career accelerator if it's leveraged right.Tomasz: [crosstalk 00:32:45].Corey: [crosstalk 00:32:46] on it.Tomasz: Yeah, absolutely. I think that we don't talk enough about, kind of, the overlap between engineering and marketing. In the good sense of marketing, not the shady kind of marketing. The kind of marketing that you do for yourself in order to elevate yourself, your projects, your successes to others. Because, you know, try as you might, but if you are kind of like sitting in the corner of an office, you know, just jamming on your keyboard 40 hours per week, you're not exactly likely to be promoted because nobody's going to actively reach out to you to find out about your, you know, recent successes and so on.Which at the same time, I'm not saying that you should go @channel in Slack every single time you push a commit to the main branch, but there's definitely, you know, a way of being, kind of, kind to yourself by letting others know that, “Okay, I'm here. I do exist, I have, you know, those particular skills that you may be interested about. And I'm able to tell a story which is, you know, convincing.” So it's, you know, you can tell a story on stage, but you can also tell your story to your customers by building a future that they're going to use. [unintelligible 00:33:50].Corey: I really want to thank you for taking the time to speak with me today. If people want to learn more, where's the best place to find you?Tomasz: So, the best place to find me is on Twitter. So, my Twitter handle is @tlakomy. So, it's T-L-A-K-O-M-Y. I'm assuming this is going to be in the [show notes 00:34:06] as well.Corey: Oh, it absolutely is. You beat me to it.Tomasz: [laugh]. So, you can find Cloudash at cloudash.dev. You can probably also find my email, but don't email me because I'm terrible, absolutely terrible at email, so the best way to kind of reach out to me is via my Twitter DMs. I'm slightly less bad at those.Corey: Excellent. And we will, of course, put links to that in the [show notes 00:34:29]. Thank you so much for being so generous with your time. I appreciate it.Tomasz: Thank you. Thank you for having me.Corey: Tomasz Łakomy, Head of React at Cloudash. I'm Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, and if you're on the YouTubes, smash the like and subscribe button, as the kids say. Whereas if you've hated this episode, please do the exact same thing—five-star reviews smash the buttons—but this time also leave an insulting and angry comment written in the form of a CloudWatch log entry that no one is ever able to find in the native interface.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

About MichaelMichael is the creator of IT automation platforms Cobbler and Ansible, the latter allegedly used by ~60% of the Fortune 500, and at one time one of the top 10 contributed to projects on GitHub.Links Referenced: Speaking Tech: https://michaeldehaan.substack.com/ michaeldehaan.net: https://michaeldehaan.net Twitter: https://twitter.com/laserllama TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored by our friends at Revelo. Revelo is the Spanish word of the day, and its spelled R-E-V-E-L-O. It means “I reveal.” Now, have you tried to hire an engineer lately? I assure you it is significantly harder than it sounds. One of the things that Revelo has recognized is something I've been talking about for a while, specifically that while talent is evenly distributed, opportunity is absolutely not. They're exposing a new talent pool to, basically, those of us without a presence in Latin America via their platform. It's the largest tech talent marketplace in Latin America with over a million engineers in their network, which includes—but isn't limited to—talent in Mexico, Costa Rica, Brazil, and Argentina. Now, not only do they wind up spreading all of their talent on English ability, as well as you know, their engineering skills, but they go significantly beyond that. Some of the folks on their platform are hands down the most talented engineers that I've ever spoken to. Let's also not forget that Latin America has high time zone overlap with what we have here in the United States, so you can hire full-time remote engineers who share most of the workday as your team. It's an end-to-end talent service, so you can find and hire engineers in Central and South America without having to worry about, frankly, the colossal pain of cross-border payroll and benefits and compliance because Revelo handles all of it. If you're hiring engineers, check out revelo.io/screaming to get 20% off your first three months. That's R-E-V-E-L-O dot I-O slash screaming.Corey: This episode is sponsored in part by LaunchDarkly. Take a look at what it takes to get your code into production. I'm going to just guess that it's awful because it's always awful. No one loves their deployment process. What if launching new features didn't require you to do a full-on code and possibly infrastructure deploy? What if you could test on a small subset of users and then roll it back immediately if results aren't what you expect? LaunchDarkly does exactly this. To learn more, visit launchdarkly.com and tell them Corey sent you, and watch for the wince.Corey: Once upon a time, Docker came out and change an entire industry forever. But believe it or not, for many of you, this predates your involvement in the space. There was a time where we had to manage computer systems ourselves with our hands—kind of—like in the prehistoric days, chiseling bits onto disk and whatnot. It was an area crying out for automation, as we started using more and more computers to run various websites. “Oh, that's a big website. It needs three servers now.” Et cetera.The times have changed rather significantly. One of the formative voices in that era was Michael DeHaan, who's joining me today, originally one of the—or if not the creator of Cobbler, and later—for which you became better known—Ansible. First, thanks for joining me.Michael: Thank you for having me. You're also making me feel very, very old there. So, uh, yes.Corey: I hear you. I keep telling people, I'm in my mid-30s, and my wife gets incensed because I'm turning 40 in July. But still. I go for the idea of yeah, the middle is expanding all the time, but it's always disturbing talking to people who are in our sector, who are younger than some of the code that we're using, which is just bizarre to me. We're all standing on the backs of giants. Like it or not, one of them's you.Michael: Oh, well, thank you. Thank you very much. Yeah, I was, like, talking to some undergrads, I was doing a little bit of stuff helping out my alma mater for a little bit, and teaching somebody the REST lecture. I was like, “In another year, REST is going to be older than everybody in the room.” And then I was just kind of… scared.Corey: Yeah. It's been a wild ride for basically everyone who's been around long enough if you don't fall off the teeter-totter and wind up breaking a limb somewhere. So, back in the bad old days, before cloud, when everything was no longer things back then were constrained by how much room you had on your credit card like they are today with cloud, but instead by things like how much space you had in the data center, what kind of purchase order you could ram through your various accounting departments. And one of the big problems you have is, great. So, finally—never on time—Dell has shipped out a whole bunch of servers—or HP or Supermicro or whoever—and the remote hands—which is always distinct from smart hands, which says something very insulting, but they seem to be good about it—would put them into racks for you.And great, so you'd walk in and see all of these brand new servers with nothing on them. How do we go ahead and configure these things? And by hand was how most of us started, and that means, oh, great, we're going to screw things up and not do them all quite the same, and it's just a treasure and a joy. Cobbler was something that you came up with that revolutionized how provisioning of bare-metal systems worked. Tell me about it.Michael: Yeah, um, so it's basically just glue. So, the story of how I came up with that is I was working for the Emerging Technologies Group at Red Hat, and I just joined. And they were like, “We have to have a solution to install Xen and KVM virtual machines.” So obviously, everybody's familiar with, like, EC2 and things now, but this was about people running non-VMware virtualization themselves. So, that was part of the problem, but in order to make that interesting, we really needed to have some automation around bare-metal installs.And that's PXE boot. So, it's TFTP and DHCP protocol and all that kind of boring stuff. And there was glue that existed, but it was usually humans would have to click on buttons to—like Red Hat had system-config-netboot, but what really happened was sysadmins all wrote their own automation at, like, every single company. And the idea that I had, and it was sort of cemented by the fact that, like, my boss, a really good guy left for another company and I didn't have a boss for, like, a couple years, was like, I'm just going to make IRC my boss, and let's get all these admins together and build a tool we can share, right?So, that was a really good experience, and it's just basically gluing all that stuff together to fully automate an install over a network so that when a system comes on, you can either pick it out from a menu; or maybe you've already got the MAC address and you can just say, “When you see this MAC address, go install this operating system.” And there's a kickstart file, or a preseed in the case of Debian, that says, “When you're booting up through the installer, basically, here's just the answers and go do these things.” And that install processes a lot slower than what we're used to, but for a bare-metal machine, that's a pretty good way to do it.Corey: Yeah, it got to a point where you could walk through and just turn on all the servers in a rack and go out to lunch, come back, they would all be configured and ready to go. And it sounds relatively basic the way we're talking about it now, but there were some gnarly cases. Like, “When I've rebooted the database server, why did it wipe itself and reprovision?” And it's, “Oh, dear.” And you have to make sure that things are—that there's a safety built into these things.And you also don't want to have to wind up plugging in a keyboard and monitor to all of these individual machines one-by-one to hit yes and acknowledge the thing. And it was a colossal pain in the ass. That's one of the things that cloud has freed us from.Michael: Yeah, definitely. And one of the nice things about the whole cloud environment is like, if you want to experiment with those ideas, like, I want to set up some DHCP or DNS, I don't have to have this massive lab and all the electricity and costs. But like, if I want to play with a load balancer, I can just get one. That kind of gives the experience of playing with all these data center technologies to everybody, which is pretty cool.Corey: On some level, you can almost view the history of all these things as speeding things up. With a well-tuned Cobbler install, it still took multiple minutes, in some cases, tens of minutes to go from machine you're powering on to getting it provisioned and ready to go. Virtual machines dropped that down to minutes. And cloud, of course, accelerated that a bit. But then you wind up with things like Docker and it gets down to less than a second. It's the meantime to dopamine.But in between the world of containers and bare-metal, there was another project—again, the one you're best known for—Ansible. Tell me about that because I have opinions on this whole space.Michael: [laugh]. Yeah. So, how Ansible got started—well, I guess configuration management is pretty old, so the people writing their own scripts, CFEngine came out, Puppet was a much better CFEngine. I was working at a company and I kind of wanted another open-source project because I enjoyed the Cobbler experience. So, I started Ansible on the side, kind of based on some frustrations around Puppet but also the desire to unify Capistrano kind of logic, which was like, “How do I push out my apps onto these servers that are already running,” with Puppet-style logic was like, “Is this computer's firewall configured directly? And is the time set correctly?”And you can obviously use that to install apps, but there's some places where that blurred together where a lot of people are using two different tools. And there's some prior art that I worked on called Funk, which I wrote with Seth Vidal and Adrian Likins at Red Hat, which was, like, 50% of the Ansible idea, and we just never built the config management layer on top. So, the idea was make something really, really simple that just uses SSH, which was controversial at the time because people thought it, like, wouldn't scale, because I was having trouble with setting up Puppet security because, like, it had DNS or timing issues or whatever.Corey: Yeah. Let's dive in a bit to what config management is first because it turns out that not everyone was living in the trenches in quite the same way that we were. I was a traveling trainer for Puppet for a summer once, and the best descriptor I found to explain it to people who are not in this space was, “All right, let's say that you go and you buy a new computer. What do you do? Well, you're going to install the applications you'd like to use, you're going to set up your own user account, you're going to set your password correctly, you're going to set up preferences, copy some files over so you have the stuff you care about. Great. Now, imagine you need to do that to a thousand computers and they all need to be the same. How do you do that?” Well, that is the world of configuration management.And there was sort of a bifurcation there, where there was the idea of, first, we're going to have configuration management that just describes what the system should look like, and that's going to run on a schedule or whatnot, and then you're going to have the other side of it, which is the idea of remote execution, of I want to run an arbitrary command on this server, or this set of servers, or all the servers, depending upon what it is. And depending on where you started on the side of that world, you wound up wanting things from the other side of that space. With Puppet, for example, is very oriented configuration management and the question became, well, can you use this for remote execution with arbitrary commands? And they wound up doing some work with Mcollective, which was a very complicated and expensive way to say, “No, not really.” There was a need for things that needed to hang out in that space.The two that really stuck out from that era were Ansible, which had its wild runaway success, and the one that I was smacking around for a bit, SaltStack, which never saw anywhere approaching that level of popularity.Michael: Yeah, sure. I mean, I think that you hit it pretty much exactly right. And it's hard to say what makes certain things take off, but I think, like, the just SSH approach was interesting because, well for one, everybody's running it. But there was this belief that this would not scale. And I tried to optimize the heck out of that because I liked performance, but it turns out that wasn't really a business problem because if you can imagine you just wrote this little bit of automation, and you're going to run it against your entire infrastructure and you've got 30,000 machines, do you want that to—if you were to, like, run an update command on 30,000 machines at once, you're going to DDoS something. Definitely, right?Corey: Yeah. Suddenly you have 30,000 machines all talk to the same things at the same times. And you want to do them in batches or smear it across.Michael: Right, so because that was there, like, you just add batch support in Ansible and things are fine, right? People want to target little small groups of things. So, like, that whole story wasn't true, and I think it was just a matter of testing this belief that everybody thought that we needed to have this whole network of things. And honestly, Salt's idea of using a message bus is great, but we took a little bit different approach with YAML because we have YAML variables in it, but they had something that compiled down to YAML. And I think those are some differences in the dialect and some things other people preferred, but—Corey: And they use Jinja, at one point to wind up making it effectively Turing complete; you could wind up having this ridicu—like, loop flow control and loops and the rest. And it was an interesting exposure to things, but yikes, at some l—at the same time.Michael: If you use all the language features in anything you can make something complicated, and too complicated. And I was like, I wanted automation to look like grocery lists. And when I started out, I said, “Hey, if anybody is doing this all day, for a day job, I will have failed.” And it clearly shows you that I have because there are people that are doing that all day. And the goal was, let me concentrate on dev and ops and my other things and keep this really, really simple.And some people just, like, get really, really into that automation technology, which is—in my opinion—why some of the earlier stuff was really popular because sysadmin were bored, so they see something new and it's kind of like a Java developer finding Perl for the first time. They're like, “I'm going to use all these things.” And they have all their little widgets, and it gets, like, really complicated.Corey: The thing that I always found interesting and terrifying at the same time about Ansible was the fact that you did ride on top of SSH, which is great because every company already had a way of controlling access by SSH to IT systems; everyone uses it, so it has an awful lot of eyes on the security protocol on the rest. The thing that I found terrifying in the early days was that more or less every ops person would wind up checking this out onto their laptop or whatnot, so whenever they wanted to run something, they would just run it from their laptop over a VPN or whatnot from wherever they happen to be, and you wind up with a dueling banjos type of circumstance where people were often not doing it from a centralized place. And in time, best practices emerged where, okay, that is going to be the command and control server where that runs at, and you log into it. And then you start guarding that with CI/CD flows and the rest. And like anything else, it wound up building some operational approaches to it.Michael: Yeah. Like, I kind of think that created a problem that allowed us to sell a product, right, which was good. If you knew what you were doing, you could use Jenkins completely and you'd be fine, right, if you had some level of discipline and access control, and you wanted to wire that up. And if you think about cloud, this whole, like, shadow IT idea of, “I just want to do this thing, therefore I'm going to get an Amazon account,” it's kind of the same thing. It's like, “I want to use this config management, but it's not approved. Who can stop me?” Right?And that kind of probably got us in the door in few accounts that way. But yeah, it did definitely create the problem where multiple people could be running things at the same time. So yeah, I mean, that's true.Corey: And the idea of, “Hey, maybe I should be controlling these things in Git,” or some other form of version control was sort of one of those evolutionary ideas that, oh, we could treat this like code. And the early days of DevOps, that was a controversial thing. These days, you say you're not doing it and people look at you very strangely. And things were going reasonably well in that direction for a while. Then this whole Docker thing showed up, where, well, what if instead of having these long-lived servers where you have to install updates and run patches and maintain a whole user list on them, instead you had this immutable infrastructure that every time there was a change, you would just go ahead and deploy a brand new set of servers?And you could do this in the olden days with virtual machines and whatnot; it just took a long time to push things out, so do I really want to roll the entire fleet for a two-line config change? Probably not, so we're going to batch it up, or maybe do this hybrid model. With Docker, it takes less than a second to wind up provisioning the—switching over to the new container series and you're done; you can keep going with that. That really solved a lot of these problems.But there were companies that, like, the entire configuration management space, who suddenly found themselves in a really weird position. Some of them tried to fight the tide forever and say, “Oh, this is terrible because it means we don't have a business model anymore.” But you can only fight the future for so long. And I think today, we'd be hard-pressed to say that Docker hasn't won, on some level.Michael: I mean, I think it has, like, the technology has won. But I guess the interesting thing is, config management now seems to be trying to pivot towards networking where I think the tool hasn't ever been designed for networking, so it's kind of a round peg, square hole. But it's all people have that unless they're buying something. Or, like, deploying the undercloud because, like, people are still running essentially clouds on top of clouds to get their Kubernetes deployments going and those are monstrous. Or maybe to deploy a data layer; like, I know Kafka has gotten off of ZooKeeper, but the Kafka-ZooKeeper thing—and I don't remember ZooKeeper [unintelligible 00:14:37] require [unintelligible 00:14:38] or not, but managing those sort of long, persistent implications, it still has a little bit of a place where it exists.But I mean, I think the whole immutable systems idea is theoretically completely great. I never was really happy with the whole Docker development workflow, and I think it does create a problem where people don't know what they're deploying and you kind of encourage that to where they could be deploying different versions of libraries, or—and that's kind of just a problem of the whole microservices thing in general where, “Did somebody change this?” And then I was working very briefly at one company where we essentially built a whole dashboard to detect service versions and what version of the base image everybody was on, and all these other things, and it can get out of hand, too. So, it's kind of like trading some problems for other problems, I think to me. But in general, containerization is good. I just wished the management glue around it was easy, right?Corey: I wound up giving a talk at a conference a while back, 2015 or so, called, “Heresy in the Church of Docker,” and it was a throwaway five-minute lightning talk, and someone approached me afterwards with, “Hey, can you give the full version of that at ContainerCon?” “There's a full version? Yes. Yes, I can.” And it talked about a number of problems with the management layer and the rest.Now, Kubernetes absolutely solves virtually every problem that I identified with it, but when you look at the other side of it, getting Kubernetes rolled out is effectively you get to cosplay being a cloud provider yourself. It is incredibly complicated, and of course, we're right back to managing it all with YAML.Michael: Right. And I think that's an interesting point, too, is I don't know who's exactly responsible for, like, the YAML explosion. And I like it as a data format; it's really good for humans. Cobbler originally used it more of an internal storage, which I think was a mistake because, like, even—I was trying to avoid setting up a database at the time, so—because I knew if I had to require setting up a database in 2007 or 2008, I'd get way less users, so it used flat files.A lot of the YAML dialects people are developing now are very, very nested and they requires, like, loading a webpage, for the Docks, like, all the time and reading what's valid here, what's valid there. I think people learn the wrong lesson from Ansible's YAML usage, right? It was supposed to be, like, YAML's good for things that are grocery lists. And there's a lot of places where I didn't do a good job. But when you see methods taking 15 parameters and you have to constantly have the reference up, maybe that's a sign that you should do something else.Corey: At least you saved us, on some level, from having to do this all in XML. But still, there are wrong ways and more wrong ways to do it. I don't think anyone could ever agree on the right way to approach these things.Michael: Yeah. I mean, and YAML, at the time was a good answer because I knew I didn't want to write and maintain a parser as, like, a guy that was running a project. We had a lot of awesome contributors, but if I had to also maintain a DSL, not only does that mean that I have to write the code for this thing—which I, you know, observed slowing down some other projects—but also that I'd have to explain it to people. Looking kind of like Bash was not a bad thing. Not having to know and learn something, so you can kind of feel really effective in about 15 minutes or something like that.Corey: One of the things that I find really interesting about you personally is that you were starting off in a bare-metal world; Ansible was sort of wherever you wanted to run it. Great, as long as there are systems that can receive these things, we're great. And now the world has changed, and for better or worse, configuration management slash remote execution is not the problem it once was and it is not a best practice way of solving a lot of those problems either. But you aren't spending your time basically remembering the glory years. You're actively moving forward doing some fairly interesting stuff. How would you describe what you're into these days?Michael: I tried to create a few projects to, like, kind of build other systems management things for the same audience for a while. I was building a build server and a new—trying to do some next-gen config stuff. And I saw people weren't interested. But I like having conversations with people, right, and I think one of the lessons from Ansible was how to explain highly technical things to technical audiences and cut out a lot of the marketing goo and all that; how to get people excited about an idea and make a community be really authentic. So, I've been writing about that for really, it's—rebooted blog is only a couple of weeks old. But also kind of trying to do some—helping out companies with some, like, basic marketing kind of stuff, right?There's just this pattern that everybody has where every website starts with this little basic slogan and two buttons and then there's a bunch of adjectives, but it doesn't say anything. So, how can you have really good documentation, and how can you explain an idea? Because, like, really, the reason you're in it is not just to sell stuff, but it's to help people and to see them get excited about your ideas. And there's just, like, we're not doing a good job in this, like, world where there's thousands upon thousands of applications, all competing at once to, like—how do you rise above that?Corey: And that's always the hard part is at some point, this does become your identity and you become known for a thing. And when you start branching out from that thing, you bring the expertise from that area that you were in, but you start applying it to new things. I feel like so many companies get focused—and people get focused—on assuming that their audience is just like them, where they're coming in with the exact same biases, the exact same experiences. And given that basically no one was as deep in the weeds as you were when it came to configuration management, that meant that you were spending time in that side of the world, not in other pursuits which aligned in some ways more directly with people developing other things. So, I suspect this might be one of the weird things we have in common when we show up and see something new.And a company is really excited. It's like, it's basically a few people talking [unintelligible 00:20:12] that both founders are technical. And they're super excited about something they can't quite articulate. And it's this, “Slow down. Tell me exactly what it is your product does.” And that's a hard thing to do because my default response was always the if I don't understand that is clearly the way in which I am deficient somehow. But marketing is really about clear communication and there's not that much of it in our space, at least not for early-stage companies.Michael: Yeah, I don't know why that is. I mean, I think there's this belief in that there's, like, this buyer audience where there's some vice president that's going to buy your stuff if you drop the right buzzwords. And 15 years ago, like, you had to say ‘synergy,' and now you say ‘time to value' or ‘total cost of ownership' or something. And I don't think that's true. I mean, I think people use products that they like and that they need to be shown them to try them out.So like, why can't your webpage have a diagram and a screenshot instead of this, like, picture of a couple of people drinking coffee around a computer, right? It's basic stuff. But I agree with you, I kind of feel dumb when I'm looking at all these tech products that I should be excited about, and, like, the way that we get there, as we ask questions. And the way that I've actually figured out what some of these things do is usually having to ask questions from someone who uses them that I randomly find on my diminishing circle of friends, right? And that's kind of busted.So, Ansible definitely had a lot of privilege in the way that it was launched in the sense that I launched it off Cobbler list and Cobbler list started off of [ET Management Tools 00:21:34] which was a company list. But people can do things like meetup groups really easily, they can give talks, they can get their blogs reblogged, and, you know, hope for some Hacker News or Reddit juice or whatever. But in order to get that to happen, you have to be able to talk to engineers that really want to know what you're doing, and they should be excited about it. So, learn to talk to them.Corey: You have to speak their language but without going so deep in the weeds that the only people that understand it are the folks who are never going to use your product because they want to build it themselves. It's a delicate balance to strike.Michael: And it's a difficult thing to do, too, when you know about it. So, when I was, like, developing all the Ansible docs, I've told people many times—and I hope it's true—that I, like, spent, like, 40% of my time just on the website and the docs, and whenever I heard somebody complain, I tried to fix it. But the idea was like, you can lose somebody really fast, but you kind of have to forget what you know about the product. So, the worst person to sometimes look at that as the person that built it. So, you have to forget what you know, and try to see, like, what questions they're asking, what do they need to find out? How do they want to learn something?And for me, I want to see a lot of pictures. A lot of people write a bunch of giant walls of text, or worse for me is when there's just these little pithy expressions and I don't know what they mean, right? And everybody's, like, kind of doing that these days.Corey: This episode is sponsored in part by our friends at ChaosSearch. You could run Elasticsearch or Elastic Cloud—or OpenSearch as they're calling it now—or a self-hosted ELK stack. But why? ChaosSearch gives you the same API you've come to know and tolerate, along with unlimited data retention and no data movement. Just throw your data into S3 and proceed from there as you would expect. This is great for IT operations folks, for app performance monitoring, cybersecurity. If you're using Elasticsearch, consider not running Elasticsearch. They're also available now in the AWS marketplace if you'd prefer not to go direct and have half of whatever you pay them count towards your EDB commitment. Discover what companies like Equifax, Armor Security, and Blackboard already have. To learn more, visit chaossearch.io and tell them I sent you just so you can see them facepalm, yet again.48]Corey: One thing that I've really found myself enjoying recently has been your substack-based newsletter, Speaking Techis what you call it. And I didn't quite know what to expect when I signed up for it, but it's been a few weeks now, and you are more or less hitting across the board on a bunch of different things, ranging from engineering design patterns, to a teardown of random company's entire website from a marketing and messaging perspective—which I just adore personally; like that is very aligned with how I see the world—Michael: There's more of that coming.Corey: Yeah, [unintelligible 00:23:17] a bunch of other stuff. Let's talk about, for example, the idea of those teardowns. I always found that I have to be somewhat careful in how I talk about it when I'm doing a tweet thread or something like that because you are talking about people's work, let's be clear here, and I tend to be a lot kinder to small, early-stage companies than I am to, you know, $1.6 trillion companies who really should have solved for this by now, on some level. But so much of it misses the mark of great, here's the way that I think about these things. Here's the way that I don't understand what the hell you're telling me.An easy example of this for me, at least I'm curious to get your thoughts on it, I tend to almost always just skim what they're saying, great. Let's look at the pricing page because I find that speaks to people in a way that very often companies forget that they're speaking to customers.Michael: Yeah, for sure. I always tried to find the product page lately, and then, like, the product page now is, like, a regurgitation of the homepage. But it's what you said earlier. I think I try to stay nice to everybody, but it's good to show people how to understand things by counterexample, to some extent, right? Like, oh, I've got some stuff coming out—I don't know when this is actually going to get published—but next week, where I was like just taking random snippets of home pages, and like, “What's everybody doing with the header these days?”And there's just, like, ridiculous amounts of copying going on. But it's not just for, like, people's companies because everybody listening here isn't going to have a company. If you have a project and you wanted to get it noticed, right, I think, like, in the early days, the projects that I paid attention to and got excited about were often the ones that spend time on their website and their messaging and their experience. So, everybody kind of understands you have to write a good readme now but some of, like, the early Ruby crowd, for instance, did awesome, awesome web pages. They know how to pick out fonts, and I still don't know how to pick out fonts. But—Corey: I ask someone good at those things. That's how I pick ‘em.Michael: Yeah, yeah. That's not my job; get somebody that's good at that. But all that matters, right? So, if you do invest a little bit in not promoting yourself, not promoting your company, but trying to help people and communicate to them, you can build that audience around your thing and it makes it a lot more interesting.Corey: There's so many great tools out there that I find on GitHub that other people have to either point me to or I find it when I'm looking at it from a code-first perspective, just trying to find a particular example of the library being used, where they do such a terrible job of describing the problem that they solve, and it doesn't take much; it takes a paragraph or two at most. Or the idea that, “Oh, yeah, here's a way to do this thing. Just go ahead and get your credential file somewhere else.” Great. Could you maybe link to an example of how to do that?It's the basic stuff; assume that someone who isn't you might possibly want to use this. And I'm not even slightly suggesting that you wind up talking your way through how to do all of that. Just link to somewhere that has a good write-up of it and call it good. Just don't get in the way of people's first-time user experiences.Michael: Yeah, for sure. And—Corey: For some reason, that's a radical thought.Michael: Yeah, I think one of the things the industry has—well, not the industry; it's not their problem to solve, but, like, we don't really have a way for people to find what's cool and interesting anymore. So, various people have their own little lists on GitHub or whatever, but there's just so many people posting on the one or two forums people read and it goes by in a day. So, it's really, really hard to get attention. Even your own circle of followers isn't really logging into Twitter or anything, or LinkedIn. Or there's all the congratulations for your five years of Acme Corp kind of posts, and it's really, really hard to get attention.And I feel for everybody, so like, if somebody like GitHub or Microsoft is listening, and you wanted to build, like, a dashboard of here's the cool 15 projects for the week, kind of thing where everybody would see it, and start spotlight some of these really cool new things, that would be awesome, right?Corey: Whenever you see those roundups, that was things like Kubernetes and Docker. And great, I don't think those projects need the help in the same way.Michael: No, no, they don't. It's like maybe somebody's cool data thing, or a cool visualization, or the other thing that's—it's completely random, but I used to write fun graphics programs for fun or games and libraries. And I don't see that anymore, right? Maybe if you find it, you can look for it, but the things that get people excited about programming. Maybe they have no commercial value at all, but the way that people discover stuff is getting so consolidated is about Docker and Kubernetes. And everyone's talking about these three things, and if you're not Google or you're not Facebook, it's really—or Amazon, obviously—it's hard to get attention.Corey: Open-source on some level has changed from a community perspective. And part of it is because once upon a time, you could start with the very low-level stuff and build something, get it up and working. And that's where things like [Cobbler 00:27:44] and Ansible came out of. Now it's, “Click the button and use the thing everyone else is using. And if you're not doing that, what are you doing over there?”So, the idea of getting started tinkering with computers are built on top of so many frameworks and other things. And that's always been the case, but now it's much more apparent in some ways. “Okay, I'm going to go ahead and build out my first HTML file and serve it out using something in Node.” “Great, what is those NPM stuff that's scrolling past?” It's like, “The devil. That is the devil's own language you are seeing scroll past. And you don't need to worry about that; just pretend it's not there.”But back when I was learning all this stuff, we're paying attention to things scrolling past, like, you know, compilation messages and the Linux boot story as it wound up scrolling past. Terrible story; the protagonist was unreliable, but all right. And you start learning how these things work when you start scratching at the things that you're just sort of hand-waving and glossing over. These days, it feels like every time I use a modern project, that's everything.Michael: I mean, it is. And like what, React has, like, 2000 dependencies, right? So, how do you ever feel like you understand it? Or when recruiters are asking for ten years at Amazon. And then—or we find a library that it can only explain itself by being like this other library and requiring these other five.And you read one of those, and it becomes, like, this… tree of knowledge that you have no way of possibly understanding. So, we've just built these stacks upon stacks upon stacks of things. And I tend to think I kind of believe in minimalism. And like, wouldn't it be cool if we just burned this all and start—you know, we burn the forest and let something new regrow. But we tend to not do that. We just—now running a cloud on top of a cloud, and our JavaScript is thousands of miles high.Corey: I really wish that there were better paths for getting started. Like, I used to think that the right way to wind up learning how all this stuff work is to do what I did: Start off as, you know, the grumpy sysadmin type, and then—or help desk—and then work your way up and the rest. Those jobs aren't there anymore, and it doesn't leave people in a productive environment. “Oh, you want to build a computer game. Great. For an iPhone? Terrific.” Where do you go to get started on that? It's a hard thing to do.And people don't care at that scale, nor should they necessarily, on how to run your own servers. Back in the day when you wanted to have a blog on the internet, you were either reduced to using LiveJournal or MySpace, or you were running your own web server and had to learn how to make sure that it didn't become an attack platform. There was a learning curve that was fairly steep. Now, there are so many different paths to go down, you don't really need to know how any of these things even work.Michael: Yeah, I think, like, one of the—I don't know whether DevOps means anything as a topic or not, but one of the original pieces around that movement was systems administrators learning to code things and really starting to enjoy it, whether that was Python or Ruby, and so on. And now it feels like we're gluing all the things together, but that's happening in App Dev as well, right? The number of people that can build a really, really good library from the ground up, like, something that has C bindings, that's a really, really small crowd. And most of it, what we're doing is gluing together other people's libraries and compensating for the flaws and bugs in them, and duct tape and error handling or whatever. And it feels like programming has changed a lot because of this—and it's good if you want to get an idea up quickly, no doubt. But it's a different experience.Corey: The problem I always ran into was the similar problems I had with doing Debian packaging. It was always the, oh, great, there's going to be four or five different guides on how to do it—same story with RPM—and they're all going to be assuming different things, and you can crossover between them without realizing it. And then you just do something monstrous that kind of works until an actual Debian developer shoves you aside like you were a hazard to everyone around you. Let me do it for you. And there we go.It's basically, get people to do work for you by being really bad at it. And I don't love that pattern, but I'm still reminded of that because there are so many different ways to achieve any outcome that, okay, I want to run a ridiculous Hotdog or Not Hotdog style website out there. Great. I can upload things. Well, Docker or serverless? What provider do I want to put it on? And oh, by the way, a lot of those decisions very early on are one-way doors that you don't realize you're crossing through, as well as not knowing what the nuances of all of those things are. And that's dangerous.Michael: I think people are also learning the vendor as well, right? Some people get really engrossed in whether it's Amazon, or Google, or HashiCorp, or somebody's API, and you spend so much of your brain cells just learning how these people's systems work versus, like, general programming practices or whatever.Corey: I make it a point to build something on other cloud providers that aren't Amazon every now and then, just because I don't want to wind up effectively embracing a monoculture.Michael: Yeah, for sure. I mean, I think that's kind of the trend I see with people looking just at the Kubernetes stuff, or whatever, it's that I don't think it necessarily existed in web dev; there seems to be a lot of—still a lot of creativity and different frameworks there, but people are kind of… what's popular? What gets me my next job, and that kind of thing. Whereas before it was… I wasn't necessarily a sysadmin; I kind of stumbled into building admin tools. I kind of made hammers not houses or whatever, but basically, everybody was kind of building their own tools and deciding what they wanted. Now, like, people that are wanting to make money or deciding what people want for them. And it's kind of not always the simplest, easiest thing.Corey: So, many open-source projects now are—for example, one that I was dealing with recently was the AWS CLI. Great, like, I'm thrilled to throw in issues and challenges here, but I'm not going to spend significant time writing code against it because, one, it's basically impossible to get these things accepted when all the maintainers work at Amazon, and two, is it really an open-source project in the way that you and I think about community and the rest, but it's basically sole purpose is to funnel money to Amazon faster. Like, that isn't really a community ethos I feel comfortable getting behind to be perfectly honest. They're a big company; they can afford to pay people to build these things out, full time.Michael: Yeah. And GitHub, I mean, we all mostly, I think, appreciate the fact that we can host the Git repo and it's performant and everything, and we don't have blazing unicorns quite as often or whatever they used to have, but it kind of changed the whole open-source culture because we used to talk about things on mailing lists, like, what should this be, and there was like an upfront conversation, or it might happen on IRC. And now people are used to just saying, “I've got a problem. Fix it for me.” Or they're throwing code over the wall and it might not be the code or feature that you wanted because they're not really part of your thing.So before, people would get really engrossed with, like, just a couple of projects, and if they were working on them as kind of like a collective of people working against different organizations, we'd talk about things, and they kind of know what was going on. And now it's very easy to get a patch that you don't want and you're, like, “Oh, can you change all of these things?” And then somebody's, like, now they're offended because now they have to do all this extra work, whereas that conversation didn't happen. And GitHub could absolutely remodel themselves to encourage those kinds of conversations and communities, but part of the death of open-source and the fact that now it's, “Give me free code,” is because of that kind of absence of the—because we're looking at that is, like, the front of a community versus, like, a conversation.Corey: I really want to appreciate your taking so much time out of your day to basically reminisce about some of these things. But on a forward-looking basis, if people want to learn more about how you see things, where's the best place to find you?Michael: Yeah. So, if you're interested in my blog, it's pretty random, but it's michaeldehaan.substack.com. I run a small emerging consultancy thing off of michaeldehaan.net. And that's basically it. My Twitter is laserllama if you want to follow that. Yeah, thank you very much for having me. Great conversation. Definitely making all this technology feel old and busted, but maybe there's still some merit in going back—Corey: Old and busted because it wasn't built this year? Great—Michael: Yes.Corey: —yes, its legacy, which is a condescending engineering term for ‘it makes money.' Yeah, there's an entire universe of stuff out there. There are companies that are still toying with virtualization: “Is this something we get on board with?” There's nothing inherently wrong with that. I find that judging what a bunch of startups are doing or ‘company started today' is a poor frame of reference to look at what you should do with your 200-year-old insurance company.Michael: Yeah, like, [unintelligible 00:35:53] software engineering is just ridiculously new. Like, if you compare it to, like, bridge-building, or even electrical engineering, right? The industry doesn't know what it's doing and it's kind of stumbling around trying to escape local maxima and things like that.Corey: I will, of course, put links to where to find you into the [show notes 00:36:09]. Thanks again for being so generous with your time. It's appreciated.Michael: Yeah, thank you very much.Corey: Michael DeHaan, founder of Cobbler, Ansible, and oh, so much more than that. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice—and/or smash the like and subscribe buttons on the YouTubes—whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, smash the buttons as mentioned, and leave a loud, angry comment explaining what you hated about it that I will then summarily reject because it wasn't properly formatted YAML.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

In this episode we are going to look at Configuration Management Tools.We will be discussing Traditional Network Configuration, Network Automation, Configuration Management Tools, and finally Compare Ansible, Chef, Puppet, and SaltStack.Thank you so much for listening to this episode of my series on Enterprise Networking, Security, and Automation for the Cisco Certified Network Associate (CCNA).Once again, I'm Kevin and this is KevTechify. Let's get this adventure started.All my details and contact information can be found on my website, https://KevTechify.com-------------------------------------------------------Cisco Certified Network Associate (CCNA)Enterprise Networking, Security, and Automation v3 (ENSA)Episode 14 - Network AutomationPart E - Configuration Management ToolsPodcast Number: 73-------------------------------------------------------Equipment I like.Home Lab ►► https://kit.co/KevTechify/home-labNetworking Tools ►► https://kit.co/KevTechify/networking-toolsStudio Equipment ►► https://kit.co/KevTechify/studio-equipment 

Our discoveries including a better diff, a way to replace Snaps with Flatpacks, a command line cheat sheet, help with YAML, and signing PDFs. Plus your feeback about supporting us with crypto nonsense, running Linux on work machines, an esoteric browser, and more.   Discoveries  Saltstack linter difftastic Xournalppnavi qddcswitch unsnap asciinema   Feedback Qutebrowser... Read More

Everybody makes New Year's resolutions. In this episode we discuss our plans and initiatives for 2022 that focus more on evolving ourselves rather than reinventing. Jim NSX-T- NSX-V going End of Support 1/16/2022 https://kb.vmware.com/s/article/2149616 Kubernetes and Containers Automation- Ansible, SaltStack, Terraform Reading things longer than 240 characters at a time Focus on being more impactful with my time Joe  Structured Plans Deep Work Focusing on life goals and living/enjoying the journey Mental health & unplugging from tech - needing hobbies/distractions Writing (both journals & a book)    Matt Get back to content creation Blogging Presenting Podcasting (hopefully!) Cloud focus Azure AWS Traditional on-prem Level up existing certs Brian Business and industry focused Security industry More non-IT related projects Training Links: https://www.udemy.com/course/vmware-nsxt-fundamentals/ https://www.udemy.com/course/vmware-nsx-t-30-fundamentals-part-two-security-2020-nsx Learning.kasten.com - https://learning.kasten.com CKA - https://training.linuxfoundation.org/training/cka-cks-exam-bundle/ KubeAcademy - https://kube.academy/ Geerling Ansible Series - https://www.youtube.com/playlist?list=PL2_OBreMn7FplshFCWYlaN2uS8et9RjNG https://www.udemy.com/course/terraform-hands-on-labs/ https://www.udemy.com/course/hashicorp-packer/ https://app.pluralsight.com/library/courses/terraform-getting-started-2021 https://app.pluralsight.com/library/courses/terraform-deep-dive https://app.pluralsight.com/paths/skill/implementing-it-automation-with-salt-open

About Micheal BenedictMicheal Benedict leads Engineering Productivity at Pinterest. He and his team focus on developer experience, building tools and platforms for over a thousand engineers to effectively code, build, deploy and operate workloads on the cloud. Mr. Benedict has also built Infrastructure and Cloud Governance programs at Pinterest and previously, at Twitter -- focussed on managing cloud vendor relationships, infrastructure budget management, cloud migration, capacity forecasting and planning and cloud cost attribution (chargeback). Links: Pinterest: https://www.pinterest.com Twitter: https://twitter.com/micheal LinkedIn: https://www.linkedin.com/in/michealb/ TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: You know how git works right?Announcer: Sorta, kinda, not really Please ask someone else!Corey: Thats all of us. Git is how we build things, and Netlify is one of the best way I've found to build those things quickly for the web. Netlify's git based workflows mean you don't have to play slap and tickle with integrating arcane non-sense and web hooks, which are themselves about as well understood as git. Give them a try and see what folks ranging from my fake Twitter for pets startup, to global fortune 2000 companies are raving about. If you end up talking to them, because you don't have to, they get why self service is important—but if you do, be sure to tell them that I sent you and watch all of the blood drain from their faces instantly. You can find them in the AWS marketplace or at www.netlify.com. N-E-T-L-I-F-Y.comCorey: This episode is sponsored in part by our friends at Vultr. Spelled V-U-L-T-R because they're all about helping save money, including on things like, you know, vowels. So, what they do is they are a cloud provider that provides surprisingly high performance cloud compute at a price that—while sure they claim its better than AWS pricing—and when they say that they mean it is less money. Sure, I don't dispute that but what I find interesting is that it's predictable. They tell you in advance on a monthly basis what it's going to going to cost. They have a bunch of advanced networking features. They have nineteen global locations and scale things elastically. Not to be confused with openly, because apparently elastic and open can mean the same thing sometimes. They have had over a million users. Deployments take less that sixty seconds across twelve pre-selected operating systems. Or, if you're one of those nutters like me, you can bring your own ISO and install basically any operating system you want. Starting with pricing as low as $2.50 a month for Vultr cloud compute they have plans for developers and businesses of all sizes, except maybe Amazon, who stubbornly insists on having something to scale all on their own. Try Vultr today for free by visiting: vultr.com/screaming, and you'll receive a $100 in credit. Thats v-u-l-t-r.com slash screaming.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. Sometimes when I have conversations with guests here, we run long. Really long. And then we wind up deciding it was such a good conversation, and there's still so much more to say that we schedule a follow-up, and that's what happened today. Please welcome back Micheal Benedict, who is, as of the last time we spoke and presumably still now, the head of engineering productivity at Pinterest. Micheal, how are you?Micheal: I'm doing great, and thanks for that introduction, Corey. Thankfully, yes, I am still the head of engineering productivity; I'm really glad to speak more about it today.Corey: The last time that we spoke, we went up one side and down the other of large-scale environments running on AWS and billing aspects thereof, et cetera, et cetera. I want to stay away from that this time and instead focus on the rest of engineering productivity, which is always an interesting and possibly loaded term. So, what is productivity engineering? It sounds almost like it's an internal dev tools team, or is it something more?Micheal: Well, thanks for asking because I get this question asked a lot of times. So, for one, our primary job is to enable every developer, at least at our company, to do their best work. And we want to do this by providing them a fast, safe, and a reliable path to take any idea into production without ever worrying about the infrastructure. As you clearly know, learning anything about how AWS works—or any public cloud provider works—is a ton of investment, and we do want our product engineers, our mobile engineers, and all the other folks to be focused on delivering amazing experiences to our Pinners. So, we could be doing some of the hard work in providing those abstractions for them in such way, and taking away the pain of managing infrastructure.Corey: The challenge, of course, that I've seen is that a lot of companies take the approach of, “Ah. We're going to make AWS available to all of our engineers in it's raw, unfiltered form.” And that lasts until the first bill shows up. And then it's, “Okay. We're going to start building some guardrails around that.” Which makes a lot of sense. There then tends to be a move towards internal platforms that effectively wrap cloud services.And for a while now, I've been generally down on the concept and publicly so in the general sense. That said, what I say that applies as a best practice or something that most people should consider does tend to fall apart when we talk about specific use cases. You folks are an extremely large environment; how do you view it? First off, do you do internal platforms like that? And secondly, would you recommend that other companies do the same thing?Micheal: I think that's such a great question because every company evolves with its own pace of development. And I wouldn't say Pinterest by itself had a developer productivity or an engineering productivity organization from the get-go. I think this happens when you start realizing that your core engineers who are working on product are now spending a certain fraction of time—which starts ballooning pretty fast—in managing the underlying systems and the infrastructure. And at that point in time, it's probably a good question to ask, how can I reduce the friction in those people's lives such that they could be focused more on the product. And, kind of, centralize or provide some sort of common abstractions through a central team which can take away all that pain.So, that is generally a good guiding principle to think about when your engineers are spending at least 30% of their time on operating the systems rather than building capabilities, that's probably a good time to revisit and see whether a central team would make sense to take away some of that. And just simple examples, right? This includes upgrading OS on your EC2 machines, or just trying to make sure you're patching all the right versions on your next big Kubernetes cluster you're running for serving x number of users. The moment you start seeing that, you want to start thinking about, if there is a central team who could take away that pain, what are the things they could be investing on to help up-level every other engineer within your organization. And I think that's one of the best ways to be thinking about it.And it was also a guiding principle for us within Pinterest to view what investments we could make in these central teams which can up-level each and every different type of engineer in the company as well. And just an example on that could be your mobile engineer would have very different expectations from your backend engineer who was working on certain aspects of code in your product. And it is truly important to understand where you want to centralize capabilities, which both these types of engineers could use, or you want to divest and have unique capabilities where it's going to make them productive. There's no one-size-fits-all solution for this, but I'm happy to talk about what we have at Pinterest, which has been reasonably working well. But I do think there's a lot more improvements we could be doing.Corey: Yeah, but let's also be clear that, as you've mentioned, you are heavily biased towards EC2 instances for a lot of what you do. If we look at the AWS console and we see hundreds of different services now, and it's easy to sit here and say, “Oh, internal platforms are terrible because all of those services are going to be enhanced in various ways and you're never going to be able to keep up with feature parity.” Yeah, but if you can wrap something like EC2 in an internal platform wrapper, that begins to be a different story because sure, someone's going to go and try something new with a different AWS service, they're going to need direct access. But the EC2 product across the board generally does not evolve in leaps and bounds with transformative changes overnight. Let's also not forget that at a company with the scale that Pinterest operates at, “Hey, AWS just dusted off a new feature and docs are still rolling out, and it's not in CloudFormation yet, but we're going to roll it out to production,” probably seems like the wrong direction to go in, I would assume.Micheal: And yes, I think that brings one of the key guardrails, I think, which these groups provide. So, when we start thinking about what teams, centralized teams like engineering productivity, developer tools, developer platforms actually do is they help with a couple of things. The top three are: they can help pave a path for the most common use cases. Like to your point, provisioning EC2 does take a set of steps, all the time. If you're going to have a thousand people doing that every time they're building a new service or trying to expand capacity playing with their launch templates, those are things you can start streamlining and making it simple by some wrapper because you want to address those 80% use cases which are usually common, and you can have a wrapper or could just automate that. And that's one of the key things: can you provide a paved path for those use cases?The second thing is, can you do that by having the right guardrails in place? How often have you heard the story that, “I just clicked a button and that now spun up, like, a thousand-plus instances.” And now you have to juggle between trying to stop them or do something about it.Corey: Back in 2013, you folks were still focusing on this fair bit. I remember because Jeremy Carroll, who I believe was your first SRE there once upon a time, wound up doing a whole series of talks around how Pinterest approached doing an AMI Factory. And back in those days, the challenges were, “Okay. We have the baseline AMI, and that's great, but we also want to do deployments of things and we don't really want to do a new deploy of an entire fleet of EC2 instances for a single line of config change, so how do we wind up weighing off of when you bake a new AMI versus when you just change something that has—in what is deployed to them?” And it was really a complicated problem back then.I'm not convinced it's not still a complicated problem, but the answers are a lot more cohesive. And making sure that every team—when you're talking about a company as large as Pinterest with that many teams—is doing things in the same way, seems like it's critically important otherwise you wind up with a whole bunch of unique-looking instances that each have to be managed by hand as opposed to something that can be reasoned around collectively.Micheal: Yep. And that last part you mentioned is extremely crucial as well because like I said, our audience or our customers are just not the engineers; we do work with our product managers and business partners as well because at times, we have to tie or change our architecture based on certain cost optimizations which would make sense, like you just articulated. We don't want to have all the instance types. It does not add much value to a developer unless they're explicitly seeking a high-memory instance or a [GP-based instance in a 00:10:25] certain way. So, we can then work with our business partners to make sure that we're committing to only a certain type of instances, and how we can abstract our tools to only give you that. For example, our deployment system, Teletraan which is an open-source system, actually condenses down all these instance types to a couple of categories like high-compute, high-memory—and you've probably seen that in many of the new cloud providers as well—so people don't have to learn or know the underlying instance type.When we moved from c3 to c5, it was just called as a high-compute system, so the next time someone provisioned a new service or deployed it using our system, they would just select high-compute as the de facto instance type and we would just automatically provision a C5 for them. So, that just reduces the extra complexity or the cognitive overhead individuals would have to go through in learning each instance type, what is the base AMI that comes on it, what are the different configurations that need to go in terms of setting up your AZ-scaling properties. We give them a good reasonable set of defaults to get started with, and then they can then work on optimizing or making changes to it.Corey: Ignoring entirely your mispronunciation of AMI, which is, of course, three syllables—and that is a petty hill upon which I will die—it occurs to me the more I work with AWS in various ways, the easier it gets. And I used to think in some respects, it was because the platform was so—it was improving so dramatically around me. But no, in many cases, it's because the first time you write some CloudFormation by hand, it's a nightmare and you keep smacking into weird issues. But the second or third time, it's super easy because you just copy the thing you've already built and change the relevant bits around. And that was the learning curve that I went through playing around with a lot of these things.When you start looking at this from a large-scale environment where it's not just about upskilling the people that you have to understand how these things integrate in AWS land, but also the consistent onboarding of engineers at a fairly progressive clip is, great, you effectively have to start doing trainings on all these things, and there's a lot of knobs and dials that can blow up and hurt people. At some point, building the guardrails or building the environment in which you are getting all the stuff abstracted away from where the application engineers have to think about this at all, it eventually reaches a tipping point where it starts to feel like it's no longer optional if you want to continue growing as a company because you don't have the luxury of spending six months of onboarding before you let someone touch the thing they were hired to build.Micheal: And you will see that many companies very often have very similar programming practices like you just described. Even I learned that the same way: you have a base template, you just copy-paste it and start from there on. And no one goes through the bootstrapping process manually anymore; you want to—I think we call it cargo-culting, but in general, just get something to bootstrap and start from there. But one of the things we learned in sort of the hard way is that can also lead to, kind of, you pushing, you know, not great practices because people don't know what is a blessed version of a good template or what actually would make sense. So, some of those things, we have been working on.And this is where centralized teams like engineering productivity are really helpful is we provide you with the blessed or the canonical way to do certain things. Case in point example is a CI/CD pipeline or delivery of software services. We have invested enough in experimenting on what works with some of the more nuanced use cases at Pinterest, in helping generate, sort of, a canonical version which would cover 80% of the use cases. Someone could just go and try to build a service and they could just use the same canonical pipeline without learning much or making changes to it. This also reduces that cargo-culting nature which I called, rather than copying it from unknown sources and trying to like—again, it may cause havoc to our systems, so we can avoid a lot of that because of these practices.Corey: So, let's step a little bit beyond AWS—I know I hate doing it, too—but I'm going to assume that your remit is broader than, oh, AWS whisperer-slash-Wrangler. So, tell me a little bit more about what it is that your day-to-day looks like if there is anything that could be said not to focus purely around AWS whispering.Micheal: So, one of the challenges—and I want to talk about this a bit more—is our environments have become extremely complex over time. And it's the nature of, like, rising entropy. Like, we've just noticed that there's two things: we have a diverse set of customer base, and these include everyone trying to do different workloads or work service types. What that essentially translates into is that we realized that our solution may not fit all of them. For example, what works for a machine-learning engineer in terms of iterating on building a model and delivering a model is not the same as someone working on a long-running service and trying to deploy that. The same would apply for someone trying to operate a Kafka system.And that has made, I think, definitely our job a bit challenging in trying to assess where do you actually draw the line on the abstraction? What is the right layer of abstraction across your local development experience, across when you move over to staging your code in a PR model and getting feedback and subsequently actually releasing it to production? Because this changes dramatically based on what is the workload type you're working on. And we feel like that has been one of the biggest challenges where I know I spent my day-to-day and my team does too, in trying to help provide some of the right solutions for these individuals. There's—very often we'll also get asked from individuals trying to do a very nuanced thing.Of late, we have been talking about thinking about how you operate functions, like provide Functions as a Service within the company? It just put us in a difficult spot at times because we have to ask the hard question, “Is this required?” I know the industry is doing it; it's definitely there. I personally believe, yes, it could be a future, but is that absolutely important? Is that going to benefit Pinterest in any formal way if we invest on some core abstractions?And those are difficult conversations to have because we have exciting engineers coming in trying to do amazing things; it puts us in a hard spot, as well, as to sometimes saying graciously, no. I know many companies deal with it when they have these centralized teams, but I think it's part of that job. Like when you say it's day-to-day, I would say I'm probably saying no a couple of times in that day.Corey: Let's pretend for the sake of argument that I am, tomorrow morning, starting another company—Twitter for Pets—and over the next ten years, it grows to be larger than Pinterest in terms of infrastructure, probably not revenue because it turns out pets are not the lucrative source of ad revenue that I was hoping it would be but, you know, directionally the same thing. It seems to me that building out this sort of function with this sort of approach to things is dramatically early as far as optimizations go when it's just me puttering around on something. I'm always cognizant of the wrong people taking the wrong message when we're talking about things that happen like this at scale. When does having an engineering productivity group begin to make sense?Micheal: I mentioned this earlier; like, yeah, there is definitely not a right answer, but we can start small. For example, this group actually started more as a delivery team. You know, when we started, we realized that we had different ways of deploying services or software at Pinterest, so we first gathered together to figure out, okay, what are the different ways and can we start simplifying that part? And that's where it started expanding. Okay, we are doing button-based deployments right now we have thousand-plus microservices, and we are seeing more incidents than we wanted to because anything where there's a human involved means there's a potential gap for error. I myself was involved in a SEV 0 incident, and I will be honest; we ended up deploying a Hello World application in one of our production fleet. Not the thing I wanted to be associated with my name, but, you know—Corey: And you were suddenly saying hello to the world, in fact—Micheal: [laugh].Corey: —and oops-a-doozy.Micheal: Yeah. So—and that really prompted us to rethink how we need to enable guardrails to do safe production rollouts. And that's how those conversations start ballooning out.Corey: And the healthy correct way. We've all broken production in various ways, and it's—you correctly are identifying, I believe, the direction you're heading in where this is a process problem and a tooling problem; it is not that you are secretly crap and should never have been allowed near anything in production. I mean, that's my excuse for me, but in your case, this is a common thing where it's, if someone can unintentionally cause issues like that, there needs to be better processes and procedures as the organization matures.Micheal: Yep. And that's kind of like always the route or the starting point for these discussions. And it starts growing from there on because, okay, you've helped improve the deploy process but now we're seeing insane amount of slowness, say on the build processes, or even post-deploy, there's, like, issues on how we monitor and look into data.And that I think forces these conversations, okay, where do we have these bespoke tools available? What are people doing today? And you have to ask those hard questions, like what can we actually remove from here? The goal is not to introduce yet another new system. Many a times, to be honest bash just gets the job done. [laugh].Personally, I'm okay with that as long as it's consistent and people, you know, are able to contribute to it and you have good practices in validating it, if it works, we should go for it rather than introducing yet another YAML [laugh] and some of that other aspects of doing that work. And that's what we encourage as well. That's how I think a lot of this starts connecting together in terms of, okay, now this is becoming a productivity group; they're focused on certain challenges where investing probably one person here may up-level a few other engineers who don't have to do that on a day-to-day basis. And I think that's one of the key items for, especially, folks who are running mid-sized companies to realize and start investing in these type of teams to really up-level, sort of, the rest of the engineering.Corey: You've been doing this for a fair while. If you were to go back and start over again on day one—which is always a terrifying question, on some level—what would you have done differently about building out this function as Pinterest continued to scale out?Micheal: Well, first, I must acknowledge that this was just not me, and there's, like, ton of people involved in helping make this happen.Corey: No, that's fair. We'll blame them for the missteps; that is—Micheal: [laugh].Corey: —just fine with me. I kid. I kid.Micheal: I think, definitely the nuances. If I look back, all the decisions that were made then at that point in time, there was a decision made to move to Phabricator, which was back then a great open-source code management system where with the current information at that point in time. And I'm not—I think it's very hard to always look back and say, “Oh, we could have chosen x at one point in time.” And I think in reality, that's how engineering organizations always evolve, that you have to make do with the information you have right now to make a decision that works for you over a couple of years.And I'll give you a small example of this. There was a time when Pinterest was actually on GitHub Enterprise—this was like circa 2013, I would say—and it really served as well for, like, five-plus years. Only then at certain point, we realized that it's hard to hire PHP engineers to support a tool like that, and we had to rethink what is the ROI and the investments we've made here? Can we ever map up or match back to one of the offerings in the industry today? And that's when you make decisions that, okay, at this point in time, it's clear that business continuity talks, you know, and it's hard to operate a system, which is, at this moment not supported, and then you make a call about making a shift or moving.And I think that's the key item. I don't think there's anything dramatically I would have changed since the start. Perhaps definitely investing a bit more individuals into the group and going from there. But that said, I'm really, sort of, at least proud of the fact that usually these teams are extremely lean and small, and they always have an outsized impact, especially when they're working with other engineers, other [opinionated 00:22:13] engineers for what it's worth.This episode is sponsored by our friends at Oracle Cloud. Counting the pennies, but still dreaming of deploying apps instead of "Hello, World" demos? Allow me to introduce you to Oracle's Always Free tier. It provides over 20 free services and infrastructure, networking databases, observability, management, and security.And - let me be clear here - it's actually free. There's no surprise billing until you intentionally and proactively upgrade your account. This means you can provision a virtual machine instance or spin up an autonomous database that manages itself all while gaining the networking load, balancing and storage resources that somehow never quite make it into most free tiers needed to support the application that you want to build.With Always Free you can do things like run small scale applications, or do proof of concept testing without spending a dime. You know that I always like to put asterisks next to the word free. This is actually free. No asterisk. Start now. Visit https://snark.cloud/oci-free that's https://snark.cloud/oci-free.Corey: Most folks show up intending to do good today, and you make the best decision at the time with the context and constraints that you have, but my question I think is less around, “Well, what were the biggest mistakes you made?” But more to do with the idea of, based upon what you've learned and as you have shown—as you've shined light on these dark areas, as you have been exploring it, has anything jumped out at you that is, “Oh, yeah. Now, that I know—if I had known then what I know now, I would definitely have made this other decision.” Ideally, something that applies a little more globally than specific within Pinterest, just because the whole idea, aspirationally, is that people might learn something from our conversation. At least I will, if nothing else.Micheal: No, I think that's a great question. And I think the three things that jump to me, top of mind. I think technology is means to an end unless it gives you a competitive edge. And it's really hard to figure out at what point in time what technology and why we adopted it, it's going to make the biggest difference. Humans always tend to have a bias towards aligning towards where we want to go. So, that's the first one in my mind.The second one is, and we spoke about this last time, embrace your cloud provider as much as possible. You'd want to avoid taking on operational burden which is not going to add value to the business. If there is something you see your operating which can be offloaded—because your provider can, trust me, do a way better job than you or your team of few can ever do—embrace that as soon as possible. It's better that way because then it frees up your time to focus on the most important thing, which I've realized over time is—I really think teams like ours are actually—we're probably the most value as a glue to all the different experiences a software engineer would go through as part of their SDLC lifecycle.If we can simplify someone's life by giving them a clear view as to where their commit or the work is in this grand scheme of rolling out and giving them the right amount of data to take action when something goes wrong, trust me, they will love you for what you're doing because you're saving them ton of time. Many times, we don't realize that when we publish 11 different ways for you to go and check to just get your basic validation of work done. We tend to so much focus on the technological aspect of what the tool does, rather than the experience of it, and I've realized, if you can bridge the experience, especially for teams like ours, people really don't even need to know whether you're running Kubernetes or any of those solutions behind the scenes. And I think that's one of the biggest takeaways I have.Corey: I want to double down on something you said about the fact that you are not going to be able to run these services as effectively as your provider can. And relatively recently—in fact, since the first time we spoke—AWS has released a investment report in Virginia. And from 2011 through 2020, they have invested in building AWS data centers there, $35 billion. I promise almost no company that employs people listening to this that are not themselves a cloud provider is going to make that kind of investment in running these things themselves.Now, do cloud providers have sharp edges? Yes, absolutely. That is what my entire career is about, unfortunately. But you're not going to do a better job of running things more sustainably, more reliably, et cetera, et cetera. But there are other problems with this—and that's what I want to start exploring here—where in the olden days, when I ran things in data centers and they went down a lot more as a result, sometimes when there were outages, I would have the CEO of the company just standing there nervous worrying over my shoulder as I frantically typed to fix things.Spoiler: my typing accuracy did not improve by having someone looming over me. Now, when there's an outage that your cloud provider takes, in many cases the thing that you are doing to fix it is reloading the status page and waiting for an update because it is completely out of your hands. Is that something that you've had to encounter? Because you can push buttons and turn dials when things are broken and you control it, but in an AWS—or other cloud provider—outage, all you can really do is wait unless you have a DR plan that is large-scale and effective enough that you won't feel foolish or have wasted a huge amount of time and energy migrating off and then—because then it gets repaired in ten minutes. How do you approach that, from your perspective? I guess, the expectation management piece?Micheal: It's definitely I know something which keeps a lot of folks within infrastructure up at night because, like you just said, at times we can feel extremely powerless when we obviously don't have direct control—or visibility at times, as well—on what's happening. One of the things we have realized over time as part of running on our cloud provider for over a decade now, it forces us to rethink a bit on our priority workflows, what we want our Pinners to always have access to, what they need to see, what is not important or critical. Because it puts into perspective, even for the infrastructure teams, is to what is the most important thing we should always have it available and running, what is okay to be in a degraded state, until what time, right? So, it actually forces us to define SLOs and availability criteria within the team where we can broadcast that to the larger audience including the executives. So, none of this comes as a surprise at that point.I mean, it's not the answer, probably, you're looking for because is there's nothing we can do except set expectations clearly on what we can do and how when you think about the business when these things do happen. So, I know people may have I have a different view on this; I'm definitely curious to hear as well, but I know at Pinterest at least we have converged on our priority workflows. When something goes out, how do we jump in to provide a degraded experience? We have very clear run books to do that, and especially when it's a SEV 0, we do have clear processes in place on how often we need to update our entire company on where things are. And especially this is where your partnership with the cloud provider is going to be a big, big boon because you really want to know or have visibility, at the minimum some predictability on when things can get resolved, and how you want to work with them on some creative solutions. This is outside the DR strategy, obviously; you should still be focused on a DR strategy, but these are just simple things we've learned over time on how to just make it predictable for individuals within the company, so not everyone is freaking out.Corey: Yeah, from my perspective, I think the big things that I found that have worked, in my experience—mostly by getting them wrong the first time—is explain that someone else running the infrastructure when they take an outage; there's not much we can do. And no, it's not the sort of thing where picking up the phone and screaming at someone is going to help us, is the sort of thing that is best to communicate to executive stakeholders when things are running well, not in the middle of that incident.Then when things break, it's one of those, “Great, you're an exec. You know what your job is? Literally anything other than standing in the middle of the engineering floor, making everyone freak out even more. We'll have a discussion later about what the contributing factors were when you demand that we fire someone because of an outage. Then we're going to have a long and hard talk about what kind of culture you're trying to build here again?” But there are no perfect answers here.It's easy to sit here in the silver light of day with things working correctly and say, “Oh, yeah. This is how outages should be handled.” But then when it goes down, we're all basically an inch away at best from running around with our hair on fire, screaming, “Fix it, fix it, fix it, fix it, now.” And I am empathetic to that. There's a reason but I fix AWS bills for a living, and one of those big reasons is that it's a strictly business-hours problem and I don't have to run production infrastructure that faces anything that people care about, which is kind of amazing and freeing for someone who spent too many years on call.Micheal: Absolutely. And one of the things is that this is not only with the cloud provider, I think in today's nature of how our businesses are set up, there's probably tons of other APIs you are using or you're working with you may not be aware of. And we ended up finding that the hard way as well. There were a certain set of APIs or services we were using in the critical path which we were not aware of. When these outages happen, that's when you find that out.So, you're not only beholden to your provider at that point in time; you have to have those SLO expectations set with your other SaaS providers as well, other folks you're working with. Because I don't think that's going to change; it's probably only going to get complicated with all the different types of tools you're using. And then that's a trade-off you need to really think about. An example here is just like—you know, like I said, we moved in the past from GitHub to Phabricator—I didn't close the loop on that because we're moving back to GitHub right now [laugh] and that's one of the key projects I'm working with. Yeah, it's circle of life.But the thing is, we did a very strong evaluation here because we felt like, “Okay, there's a probability that GitHub can go down and that means people will be not productive for that couple of hours. What do we do then?” And we had to put a plan together to how we can mitigate that part and really build that confidence with the engineering teams, internally. And it's not the best solution out there; the other solution was just run our own, but how is that going to make any other difference because we do have libraries being pulled out of GitHub and so many other aspects of our systems which are unknowingly dependent on it anyways. So, you have to still mitigate those issues at some point in your entire SDLC process.So, that was just one example I shared, but it's not always on the cloud provider; I think there are just many aspects of—at least today how businesses are run, you're dependent; you have critical dependencies, probably, on some SaaS provider you haven't really vetted or evaluated. You will find out when they go down.Corey: So, I don't think I've told this story before, but before I started this place, I was doing a fair bit of consulting work for other companies. And I was doing a project at Pinterest years ago. And this was one of the best things I've ever experienced at a company site, let alone a client site, where I was there early in the morning, eight o'clock or so, so you know, engineers love to show up at the crack of 11:30. But so I was working a little early; it was great. And suddenly my SSH session that I was using to remote into something or other hung.And it's tap up, tap enter a couple of times, tap it a couple more. It was hung hard. “What's the—” and then someone gently taps me on the shoulder. So, I take the headphones off. It was someone from corporate IT was coming around saying, “Hey, there's a slight problem with our corporate firewall that we're fixing. Here's a MiFi device just for you that you can tether to get back online and get worked on until the firewall gets back.”And it was incredible, just the level of just being on top of things, and the focus on keeping the people who were building things and doing expensive engineering work that was awesome—and also me—productive during that time frame was just something I hadn't really seen before. It really made me think about the value of where do you remove bottlenecks from people getting their jobs done? It was—it remains one of the most impressive things I've seen.Micheal: That is great. And as you were telling me that I did look up our [laugh] internal system to see whether a user called Corey Quinn existed, and I should confirm this with you. I do see entries over here, a couple of commits, but this was 2015. Was that the time you were around, or is this before that even?Corey: That would have been around then, yes. I didn't start this place until late 2016.Micheal: I do see your commits, like, from 2015, and I—Corey: And they're probably terrible, I have no doubt. There's a reason I don't read code for a living anymore.Micheal: Okay, I do see a lot of GIFs—and I hope it's pronounced as GIF—okay, this is cool. We should definitely have a chat about this separately, Corey?Corey: Oh, yeah. “Would you explain this code?” “Absolutely not. I wrote it. Of course, I have no idea what it does. That's the rule. That's the way code always works.”Micheal: Oh, you are an honorary Pinterest engineer at this point, and you have—yes—contributed to our API service and a couple of Puppet profiles I see over here.Corey: Oh, yes—Micheal: [Amazing 00:36:11]. [laugh].Corey: You don't wind up thinking that's a risk factor that should be disclosed. I kid. I kid. It's, I made a joke about this when VMware acquired SaltStack and I did some analytics and found that 60 some odd lines of code I had written, way back when that were still in the current version of what was being shipped. And they thought, “Wait, is this actually a risk?”And no, I am making a joke. The joke is, is my code is bad. Fortunately, there are smart people around me who review these things. This is why code review is so important. But there was a lot to admire when I was there doing various things at Pinterest. It was a fun environment to work in, the level of professionalism was phenomenal, and I was just a big fan of a lot of the automation stuff.Phabricator was great. I love working with it, and, “Great, I'm going to use this to the next place I go.” And I did and then it was—I looked at what it took to get it up and running, and oh, yeah, I can see why GitHub is so popular these days. But it was neat. It was interesting seeing that type of environment up close.Micheal: That is great to hear. You know, this is what I enjoy, like, hearing some of these war stories. I am surprised; you seem to have committed way more than I've ever done in my [laugh] duration here at Pinterest. I do managing for a living, but then again—Corey, the good news is your code is still running on production. And we—Corey: Oh dear.Micheal: —haven't—[laugh]. We haven't removed or made any changes to it, so that's pretty amazing. And thank you for all your contributions.Corey: Oh, please, you don't have to thank me. I was paid, it was fine. That's the value of—Micheal: [laugh].Corey: —[work 00:37:38] for hire. It's kind of amazing. And the best part about consultants is, is when we're done with a project, we get the hell out everyone's happy about it.More happy when it's me that's leaving because of obvious personality-related reasons. But it was just an interesting company from start to finish. I remember one other time, I wound up opening a ticket about having a slight challenge with a flickering on my then Apple-branded display that everyone was using before they discontinued those. And I expected there to be, “Oh, okay. You're a consultant. Great. How did we not put you in the closet with a printer next to that thing, breathing the toner?” Like most consulting clients tend to do, and sure enough, three minutes later, I'm getting that tap on the shoulder again; they have a whole replacement monitor. “Can you go grab a cup of coffee? We'll run the cable for it. It'll just be about five minutes.” I started to feel actively bad about requesting things because I did a lot of consulting work for a lot of different companies, and not to be unkind, but treating consultants and contractors super well is not something that a lot of companies optimize for. I can't necessarily blame them for that. It just really stood out.Micheal: Yep, I do hope we are keeping up with that right now because I know our team definitely has a lot of consultants working with us as well. And it's always amazing to see; we do want to treat them as FTs. It doesn't even matter at that point because we're all individuals and we're trying to work towards common goals. Like you just said, I think I personally have learned a few items as well from some of these folks. Which is again, I think speaks to how we want to work and create a culture of, like, we're all engineers; we want to be solving problems together, and as you were doing it, we want to do it in such a way that it's still fun, and we're not having the restrictions of titles or roles and other pieces. But I think I digressed. It was really fun to see your commits though, I do want to track this at some point before we move completely over to GitHub, at least keep this as a record, for what it's worth.Corey: Yeah basically look at this graffiti in the codebase of, “A shit-poster was here,” and here I am. And that tends to be, on some level, the mark we live on the universe. What's always terrifying is looking at things I did 15 years ago in my first Linux admin job. Can I still ping the thing that I built there? Yes, I can. And how is that even possible? That should not have outlived me; honestly, it should never have seen the light of day in production, but here we are. And you never know how long that temporary kluge you put together is going to last.Micheal: You know, one of the things I was recalling, I was talking to someone in my team about this topic as well. We always talk about 10x engineers. I don't know what your thoughts are on that, but the fact that you just mentioned you built something; it still pings. And there's a bunch of things, in my mind, when you are writing code or you're working on some projects, the fact that it can outlast you and live on, I think that's a big, big contribution. And secondly, if your code can actually help up-level, like, ten other people, I think you've really made the mark of 10x engineer at that point.Corey: Yeah, the idea of the superhuman engineer is always been a strange and dangerous one. If for nothing else, from where I sit, excellence is inherently situational. Like we just talked about someone at Pinterest: is potentially going to be able to have that kind of impact specifically because—to my worldview—that there's enough process and things around there that empower them to succeed. Then if you were to take that engineer and drop them into a five-person startup where none of those things exist, they might very well flounder. It's why I'm always a little suspicious of this is a startup founded by engineers from Google or Facebook, or wherever it is.It's, yeah, and what aspects of that culture do you think are one-to-one matches with the small scrappy startup in the garage? Right, I predicting some challenges here. Excellence is always situational. An amazing employee at one company can get fired at a second one for lack of performance, and that does not mean that there's anything wrong with them and it does not mean that they are a fraud. It means that what they needed to be successful was present in one of those shops, but not the other.Micheal: This is so true. And I really appreciate you bringing this up because whenever we discuss any form of performance management, that is a—in my view personally—I think that's an incorrect term to be using. It is really at that point in time, either you have outlived the environment you are in, or the environment is going in a different direction where I think your current skill set probably could be best used in the environment where it's going to work. And I know it's very fuzzy at that point, but like you said, yes, excellence really means you don't want to tie it to the number of commits you have pushed out, or any specific aspect of your deliverables or how you work.Corey: There are no easy answers to any of these things, and it's always situational. It's why I think people are sometimes surprised when I will make comments about the general case of how things should be, then I talk to a specific environment where they do the exact opposite, and I don't yell at them for it. It's there—in a general sense, I have some guidance, but they are usually reasons things are the way they are, and I'm interested in hearing them out. Everything's situational, the worst consultant in the world is the one that shows up, has no idea what's going on, and then asked, “What moron set this up?” Invariably, two said, quote-unquote, “Moron.” And the engagement doesn't go super well from there. It's, “Okay, why is this the way that it is? What constraints shaped it? What was the context behind the problem you were trying to solve?” And, “Well, why didn't you use this AWS service?” “Because it didn't exist for another three years when we were building that thing,” is a—Micheal: Yes.Corey: —common answer.Micheal: Yes, you should definitely appreciate that of all the decisions that have been made in past. People tend to always forget why they were made. You're absolutely right; what worked back then will probably not work now, or vice versa, and it's always situational. So, I think I can go on about this for hours, but I think you hit that to the point, Corey.Corey: Yeah, I do my best. I want to thank you for taking another block of time out of your day to wind up talking with me about various aspects of what it takes to effectively achieve better levels of engineering productivity at large companies, with many teams, working on shared codebases. If people want to learn more about what you're up to, where can they find you?Micheal: I'm definitely on Twitter. So, please note that I'm spelled M-I-C-H-E-A-L on Twitter. So, you can definitely read on to my tweets there. But otherwise, you can always reach out to me on LinkedIn, too.Corey: Fantastic and we will, of course, include a link to that in the [show notes 00:44:02]. Thanks once again for your time. I appreciate it.Micheal: Thanks a lot, Corey.Corey: Micheal Benedict, head of engineering productivity at Pinterest. I'm Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with a comment telling me that you work at Pinterest, have looked at the codebase, and would very much like a refund and an apology.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

About Betty Betty Junod is the Senior Director of Multi-Cloud Solutions at VMware helping organizations along their journey to cloud. This is her second time at VMware, having previously led product marketing for end user computing products.  Prior to VMware she held marketing leadership roles at Docker and solo.io in following the evolution of technology abstractions from virtualization, containers, to service mesh. She likes to hang out at the intersection of open source, distributed systems, and enterprise infrastructure software. @bettyjunod  Links: Twitter: https://twitter.com/BettyJunod Vmware.com/cloud: https://vmware.com/cloud TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: You know how git works right?Announcer: Sorta, kinda, not really Please ask someone else!Corey: Thats all of us. Git is how we build things, and Netlify is one of the best way I've found to build those things quickly for the web. Netlify's git based workflows mean you don't have to play slap and tickle with integrating arcane non-sense and web hooks, which are themselves about as well understood as git. Give them a try and see what folks ranging from my fake Twitter for pets startup, to global fortune 2000 companies are raving about. If you end up talking to them, because you don't have to, they get why self service is important—but if you do, be sure to tell them that I sent you and watch all of the blood drain from their faces instantly. You can find them in the AWS marketplace or at www.netlify.com. N-E-T-L-I-F-Y.comCorey: This episode is sponsored in part by our friends at Vultr. Spelled V-U-L-T-R because they're all about helping save money, including on things like, you know, vowels. So, what they do is they are a cloud provider that provides surprisingly high performance cloud compute at a price that—while sure they claim its better than AWS pricing—and when they say that they mean it is less money. Sure, I don't dispute that but what I find interesting is that it's predictable. They tell you in advance on a monthly basis what it's going to going to cost. They have a bunch of advanced networking features. They have nineteen global locations and scale things elastically. Not to be confused with openly, because apparently elastic and open can mean the same thing sometimes. They have had over a million users. Deployments take less that sixty seconds across twelve pre-selected operating systems. Or, if you're one of those nutters like me, you can bring your own ISO and install basically any operating system you want. Starting with pricing as low as $2.50 a month for Vultr cloud compute they have plans for developers and businesses of all sizes, except maybe Amazon, who stubbornly insists on having something to scale all on their own. Try Vultr today for free by visiting: vultr.com/screaming, and you'll receive a $100 in credit. Thats v-u-l-t-r.com slash screaming.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. Periodically, I like to poke fun at a variety of different things, and that can range from technologies or approaches like multi-cloud, and that includes business functions like marketing, and sometimes it extends even to companies like VMware. My guest today is the Senior Director of Multi-Cloud Solutions at VMware, so I'm basically spoilt for choice. Betty Junod, thank you so much for taking the time to speak with me today and tolerate what is no doubt going to be an interesting episode, one way or the other.Betty: Hey, Corey, thanks for having me. I've been a longtime follower, and I'm so happy to be here. And good to know that I'm kind of like the ultimate cross-section of all the things [laugh] that you can get snarky about.Corey: The only thing that's going to make that even better is if you tell me, “Oh, yeah, and I moonlight on a contract gig by naming AWS services.” And then I just won't even know where to go. But I'll assume they have to generate those custom names in-house.Betty: Yes. Yes, I think they do those there. I may comment on it after the fact.Corey: So, periodically I am, let's call it miscategorized, in my position on multi-cloud, which is that it's a worst practice that when you're designing something from scratch, you should almost certainly not be embracing unless you're targeting a very specific corner case. And I stand by that, but what that has been interpreted as by the industry, in many cases because people lack nuance when you express your opinions in tweet-sized format—who knew—as me saying, “Multi-cloud bad.” Maybe, maybe not. I'm not interested in assigning value judgment to it, but the reality is that there are an awful lot of multi-cloud deployments out there. And yes, some of them started off as, “We're going to migrate from one to the other,” and then people gave up and called it multi-cloud, but it is nuanced. VMware is a company that's been around for a long time. It has reinvented itself in a few different ways at different periods of its evolution, and it's still highly relevant. What is the Multi-Cloud Solutions group over at VMware? What do you folks do exactly?Betty: Yeah. And so I will start by multi-cloud; we're really taking it from a position of meeting the customer where they are. So, we know that if anything, the only thing that's a given in our industry is that there will be something new in the next six months, next year, and the whole idea of multi-cloud, from our perspective, is giving customers the optionality, so don't make it so that it's a closed thing for them. But if they decide—it's not that they're going to start, “Hey, I'm going to go to cloud, so day one, I'm going to go all-in on every cloud out there.” That doesn't make sense, right, as—Corey: But they all gave me such generous free credit offers when I founded my startup; I feel obligated to at this point.Betty: I mean, you can definitely create your account, log in, play around, get familiar with the console, but going from zero to being fully operationalized team to run production workloads with the same kind of SLAs you had before, across all three clouds—what—within a week is not feasible for people getting trained up and actually doing that. Our position is that meeting customers where they are and knowing that they may change their mind, or something new will come up—a new service—and they really want to use a new service from let's say GCP or AWS, they want to bring that with an application they already have or build a new app somewhere, we want to help enable that choice. And whether that choice applies to taking an existing app that's been running in their data center—probably on vSphere—to a new place, or building new stuff with containers, Kubernetes, serverless, whatever. So, it's all just about helping them actually take advantage of those technologies.Corey: So, it's interesting to me about your multi-cloud group, for lack of a better term, is there a bunch of things fall under its umbrella? I believe Bitnami does—or as I insist on calling it, ‘bitten-A-M-I'—I believe that SaltStack—which I wrote a little bit of once upon a time, which tells me you folks did no due diligence whatsoever because everything I've ever written is molten garbage—Betty: Not [unintelligible 00:04:33].Corey: And—so to be clear, SaltStack is good; just the parts that I wrote are almost certainly terrible because have you met me?Betty: I'll make a note. [laugh].Corey: You have Wavefront, you have CloudHealth, you have a bunch of other things in the portfolio, and yeah, all those things do work across multiple clouds, but there's nothing that makes using any of those things a particularly bad idea even if you're all-in on one cloud provider, too. So, it's a portfolio that applies to a whole bunch have different places from your perspective, but it can be used regardless of where folks stand ideologically.Betty: Yes. So, this goes back to the whole idea that we meet the customers where they are and help them do what they want to do. So, with that, making sure these technologies that we have work on all the clouds, whether that be in the data center or the different vendors, so that if a customer wants to just use one, or two, or three, it's fine. That part's up to them.Corey: The challenge I've run into is that—and maybe this is a ‘Twitter Bubble' problem, but unfortunately, having talked to a whole bunch of folks in different contexts, I know it isn't—there's almost this idea that you have to be incredibly dogmatic about a particular technology that you're into. I joke periodically about the Rust Evangelism Strikeforce where their entire job is talking about using Rust; their primary IDE is PowerPoint because they're giving talks all the time about it rather than writing code. And great, that's a bit of an exaggeration, but there are the idea of a technology purist who is taking, “Things must be this way,” well past a point of being reasonable, and disregarding the reality that, yeah, the world is messy in a way that architectural diagrams never are.Betty: Yeah. The architectural diagrams are always 2D, right? Back to that PowerPoint slide: how can I make pretty boxes? And then I just redraw a line because something new came out. But you and I have been in this industry for a long time, there's always something new.And I think that's where the dogmatism gets problematic because if you say we're only going to do containers this way—you know, I could see Swarm and Kubernetes, or all-in on AWS and we're going to use all the things from AWS and there's only this way. Things are generational and so the idea that you want to face the reality and say that there is a little bit of everything. And then it's kind of like, how do you help them with a part of that? As a vendor, it could be like, “I'm going to help us with a part of it, or I'm going to help address certain eras of it.” That's where I think it gets really bad to be super dogmatic because it closes you off to possibly something new and amazing, new thinking, different ways to solve the same problem.Corey: That's the problem is left to our own devices, most of us who are building things, especially for random ideas, yeah, there's a whole modern paradigm of how I can build these things, but I'm going to shortcut to the thing I know best, which may very well the architectures that I was using 15 years ago, maybe tools that I was using 15 years ago. There's a reason that Vim is still as popular as it is. Would I recommend it to someone who's a new user? Absolutely not; it's user-hostile, but back in my days of being a grumpy sysadmin, you learned vi because it was on everything you could get into, and you never knew in what environment you were going to be encountering stuff. These days, you aren't logging in to remote systems to manage them, in most cases, and when it happens, it's a rarity and a bug.The world changes; different approaches change, but you have to almost reinvent your entire philosophy on how things work and what your career trajectory looks like. And you have to give up aspects of what you've considered to be part of your identity and embrace something new. It was hard for me to accept that, for example, Docker and the wave of containerization that was rolling out was effectively displacing the world that I was deep in of configuration management with Puppet and with Salt. And the world changes; I said, “Okay, now I'll work on cloud.” And if something else happens, and mainframes are coming back again, instead, well, I'm probably not going to sit here railing against the tide. It would be ridiculous to do that from my perspective. But I definitely understand the temptation to fight against it.Betty: Mm-hm. You know, we spend so much time learning parts of our craft, so it's hard to say, “I'm now not going to be an expert in my thing,” and I have to admit that something else might be better and I have to be a newbie again. That can be scary for someone who's spent a lot of time to be really well-versed in a specific technology. It's funny that you bring up the whole Docker and Puppet config management; I just had a healthy discussion over Slack with some friends. Some people that we know and comment about some of the newer areas of config management, and the whole idea is like, is it a new category or an evolution of? And I went back to the point that I made earlier is like, it's generations. We continually find new ways to solve a problem, and one thing now is it [sigh] it just all goes so much faster, now. There's a new thing every week. [laugh] it seems sometimes.Corey: It is, and this is the joy of having been in this industry for a while—toxic and broken in many ways though it is—is that you go through enough cycles of seeing today's shiny, new, amazing thing become tomorrow's legacy garbage that we're stuck supporting, which means that—at least from my perspective—I tend to be fairly conservative with adopting new technologies with respect to things that matter. That means that I'm unlikely to wind up looking at the front page of Hacker News to pick a framework to build a banking system in, and I'm unlikely to be the first kid on my block to update to a new file system or database, just because, yeah, if I break a web server, we all laugh, we make fun of the fact that it throws an error for ten minutes, and then things are back up and running. If I break the database, there's a terrific chance that we don't have a company anymore. So, it's the ‘mistakes will show' area and understanding when to be aggressive and when to hold back as far as jumping into new technologies is always a nuanced decision. And let's be clear as well, an awful lot of VMware's customers are large companies that were founded, somehow—this is possible—before 2010. Imagine that. Did people—Betty: [laugh]. I know, right?Corey: —even have businesses or lives back then? I thought we all used horse-driven carriages and whatnot. And they did not build on cloud—not because of any perception of distrust; because it functionally did not exist at the time that they were building these things. And, “Oh, come out into the cloud. It's fine now.” It… yeah, that application is generating hundreds of millions in revenue every quarter. Maybe we treat that with a little bit of respect, rather than YOLO-ing it into some Lambda-driven monster that's constructed—Betty: One hundred—Corey: —out of popsicle sticks and glue.Betty: —percent. Yes. I think people forget that. And it's not that these companies don't want to go to cloud. It's like, “I can't break this thing. That could be, like, millions of dollars lost, a second.”Corey: I write my weekly newsletters in a custom monstrosity of a system that has something like 30-some-odd Lambda functions, a bunch of API gateways that are tied together with things, and periodically there are challenges with it that break as the system continues to evolve. And that's fine. And I'm okay with using something like that as a part of my workflow because absolute worst case, I can go back to the way that my newsletter was originally written: in Google Docs, and it doesn't look anywhere near the same way, and it goes back to just a text email that starts off with, “I have messed up.” And that would be a better story than most of the stuff I put out as a common basis. Similarly, yeah, durability is important.If this were a serious life-critical app, it would not just be hanging out in a single region of a single provider; it would probably be on one provider, as I've talked about, but going multi-region and having backups to a different cloud provider. But if AWS takes a significant enough outage to us-west-2 in Oregon, to the point where my ridiculous system cannot function to write the newsletter, that too, is a different handwritten email that goes out that week because there's no announcement they've made that anyone's going to give the slightest toss about, given the fact that it's basically Cloud Armageddon. So, we'll see. It's about understanding the blast radius and understanding your use case.Betty: Yep. A hundred percent.Corey: So, you've spent a fair bit of time doing interesting things in your career. This is your second outing at VMware, and in the interim, you were at solo.io for a bit, and before that you were in a marketing leadership role at Docker. Let's dive in, if you will. Given that you are no longer working at Docker, they recently made an announcement about a pricing model change, whereas it is free to use Docker Desktop for anyone's personal projects, and for small companies.But if you're a large company, which they define is ten million in revenue a year or 250 employees—those two things don't go alike, but okay—then you have to wind up having a paid plan. And I will say it's a novel approach, but I'm curious to hear what you have to say about it.Betty: Well, I'd say that I saw that there was a lot of flutter about that news, and it's kind of a, it doesn't matter where you draw the line in the sand for the tier, there's always going to be some pushback on it. So, you have to draw a line somewhere. I haven't kept up with the details around the pricing models that they've implemented since I left Docker a few years ago, but monetization is a really important part for a startup. You do have to make money because there are people that you have to pay, and eventually, you want to get off of raising money from VCs all the time. Docker Desktop has been something that has been a real gem from a local developer experience, right, giving the—so that has been well-received by the community.I think there was an enterprise application for it, but when I saw that, I was like, yeah, okay, cool. They need to do something with that. And then it's always hard to see the blowback. I think sometimes with the years that we've had with Docker, it's kind of like no matter what they do, the Twitterverse and Hacker News is going to just give them a hard time. I mean, that is my honest opinion on that. If they didn't do it, and then, say, they didn't make the kind of revenue they needed, people would—that would become another Twitter thread and Hacker News blow up, and if they do it, you'll still have that same reaction.Corey: This episode is sponsored by our friends at Oracle Cloud. Counting the pennies, but still dreaming of deploying apps instead of "Hello, World" demos? Allow me to introduce you to Oracle's Always Free tier. It provides over 20 free services and infrastructure, networking databases, observability, management, and security.And - let me be clear here - it's actually free. There's no surprise billing until you intentionally and proactively upgrade your account. This means you can provision a virtual machine instance or spin up an autonomous database that manages itself all while gaining the networking load, balancing and storage resources that somehow never quite make it into most free tiers needed to support the application that you want to build.With Always Free you can do things like run small scale applications, or do proof of concept testing without spending a dime. You know that I always like to put asterisks next to the word free. This is actually free. No asterisk. Start now. Visit https://snark.cloud/oci-free that's https://snark.cloud/oci-free.Corey: It seems to be that Docker has been trying to figure out how to monetize for a very long time because let's be clear here; I think it is difficult to overstate just how impactful and transformative Docker was to the industry. I gave a talk “Heresy in the Church of Docker” that listed a bunch of things that didn't get solved with Docker, and I expected to be torn to pieces for it, and instead I was invited to give it at ContainerCon one year. And in time, a lot of those things stopped being issues because the industry found answers to it. Now, unfortunately, some of those answers look like Kubernetes, but that's neither here nor there. But now it's, okay, so giving everything that you do that is core and central away for free is absolutely part of what drove the adoption that it saw, but goodwill from developers is not the sort of thing that generally tends to lead to interesting revenue streams.So, they had to do something. And they've tried a few different things that haven't seemed to really pan out. Then they spun off that pesky part of their business that made money selling support contracts, over to Mirantis, which was apparently looking for something now that OpenStack was no longer going to be a thing, and Kubernetes is okay, “Well, we'll take Docker enterprise stuff.” Great. What do they do, as far as turning this into a revenue model?There's a lot of the, I guess, noise that I tend to ignore when it comes to things like this because angry people on Twitter, or on Hacker News, or other terrible cesspools on the internet, are not where this is going to be decided. What I'm interested in is what the actual large companies are going to say about it. My problem with looking at it from the outside is that it feels as if there's significant ambiguity across the board. And if there's one thing that I know about large company procurement departments, it's that they do not like ambiguity. This change takes effect in three or four months, which is underwear-outside-the-pants-superhero-style speed for a lot of those companies, and suddenly, for a lot of developers, they're so far removed from the procurement side of the house that they are never going to have a hope of getting that approved on a career-wide timespan.And suddenly, for a lot of those companies, installing and running Docker Desktop just became a fireable offense because from the company's perspective, the sheer liability side of it, if they were getting subject to audit, is going to be a problem. I don't believe that Docker is going to start pulling Oracle-like audit tactics, but no procurement or risk management group in the world is going to take that on faith. So, the problem is not that it's expensive because that can be worked around; it's not that there's anything inherently wrong with their costing model. The problem is the ambiguity of people who just don't know, “Does this apply to me or doesn't this apply to me?” And that is the thing that is the difficult, painful part.And now, as a result, the [unintelligible 00:17:28] groups and their champions of Docker Desktop are having to spend a lot more time, energy, and thought on this than it would simply be for cutting a check because now it's a risk org-wide, and how do we audit to figure out who's installed this previously free open-source thing? Now what?Betty: Yeah, I'll agree with you on that because once you start making it into corporate-issued software that you have to install on the desktop, that gets a lot harder. And how do you know who's downloaded it? Like my own experience, right? I have a locked-down laptop; I can't just install whatever I want. We have a software portal, which lets me download the approved things.So, it's that same kind of model. I'd be curious because once you start looking at from a large enterprise perspective, your developers are working on IP, so you don't want that on something that they've downloaded using their personal account because now it sits—that code is sitting with their personal account that's using this tool that's super productive for them, and that transition to then go to an enterprise, large enterprise and going through a procurement cycle, getting a master services agreement, that's no small feat. That's a whole motion that is different than someone swiping a credit card or just downloading something and logging in. It's similar to what you see sometimes with the—how many people have signed up for and paid 99 bucks for Dropbox, and then now all of a sudden, it's like, “Wow, we have all of megacorp [laugh] signed up, and then now someone has to sell them a plan to actually manage it and make sure it's not just sitting on all these personal drives.”Corey: Well, that's what AWS's original sales motion looked a lot like they would come in and talk to the CTO or whatnot at giant companies. And the CTO would say, “Great, why should we pick AWS for our cloud needs?” And the answer is, “Oh, I'm sorry. You have 87 distinct accounts within your organization that we've [unintelligible 00:19:12] up for you. We're just trying to offer you some management answers and unify the billing and this, and probably give you a discount as well because there is price breaks available at certain sizing.” It was a different conversation. It's like, “I'm not here to sell you anything. We're already there. We're just trying to formalize the relationship.” And that is a challenge.Again, I'm not trying to cast aspersions on procurement groups. I mean, I do sell enterprise consulting here at The Duckbill Group; we deal with an awful lot of procurement groups who have processes and procedures that don't often align to the way that we do things as a ten-person, fully remote company. We do not have commercial vehicle insurance, for example, because we do not have a commercial vehicle and that is a prerequisite to getting the insurance, for one. We're unlikely to buy one to wind up satisfying some contractual requirements, so we have to go back and forth and get things like that removed. And that is the nature of the beast.And we can say yes, we can say no on a lot of those questionnaires, but, “It depends,” or, “I don't know,” is the sort of thing that's going to cause giant red flags and derail everything. But that is exactly what Docker is doing. Now, it's the well, we have a sort of sloppy, weird set of habits with some of our engineers around the bring your own device to work thing. So, that's the enterprise thing. Let me be very clear, here at The Duckbill Group, we have a policy of issuing people company machines, we manage them very lightly just to make sure the drives are encrypted, so they—and that the screensaver comes out with a password, so if someone loses a laptop, it's just, “Replace the hardware,” not, “We have a data breach.”Let's be clear here; we are responsible about these things. But beyond that, it's oh, you want to have some personal thing installed on your machine or do some work on that stuff? Fine. By all means. It's a situation of we have no policy against it; we understand this is how work happens, and we trust people to effectively be grownups.There are some things I would strongly suggest that any employee—ours or anyone else—not cross the streams on for obvious IP ownership rights and the rest, we have those conversations with our team for a reason. It's, understand the nuances of what you're doing, and we're always willing to throw hardware at people to solve these problems. Not every company is like that. And ten million in revenue is not necessarily a very large company. I was doing the math out for ten million in revenue or 250 employees; assuming that there's no outside investment—which with VC is always a weird thing—it's possible—barely—to have a $10 million in revenue company that has 250 employees, but if they're full time they are damn close to a $15 an hour minimum wage. So, who does it apply to? More people than you might believe.Betty: Yeah, I'm really curious to how they're going to like—like you say, if it takes place in three or four months, roll that out, and how would you actually track it and true that up for people? So.Corey: Yeah. And there are tools and processes to do this, but it's also not in anyone's roadmap because people are not sitting here on their annual planning periods—which is always aspirational—but no one's planning for, “Oh, yeah, Q3, one of our software suppliers is going to throw a real procurement wrench at us that we have to devote time, energy, resources, and budget to figure out.” And then you have a problem. And by resources, I do mean resources of basically assigning work and tooling and whatnot and energy, not people. People are humans, they are not resources; I will die on that hill.Betty: Well, you know, actually resource-wise, the thing that's interesting is when you say supplier, if it's something that people have been able to download for free so far, it's not considered a supplier. So, it's—now they're going to go from just a thing I can use and maybe you've let your developers use to now it has to be something that goes through the official internal vetting as being a supplier. So, that's just—it's a whole different ball game entirely.Corey: My last job before I started this place, was a highly regulated financial institution, and even grabbing things were available for free, “Well, hang on a minute because what license is it using and how is it going to potentially be incorporated?” And this stuff makes sense, and it's important. Now, admittedly, I have the advantage of a number of my engineering peers in that I've been married to a corporate attorney for 11 years and have insight into that side of the world, which to be clear, is all about risk mitigation which is helpful. It is a nuanced and difficult field to—as are most things once you get into them—and it's just the uncertainty that befuddles me a bit. I wish them well with it, truly I do. I think the world is better with an independent Docker in it, but I question whether this is going to find success. That said, it doesn't matter what I think; what matters is what customers say and do, and I'm really looking forward to seeing how it plays out.Betty: A hundred percent; same here. As someone who spent a good chunk of my life there, their mark on the industry is not to be ignored, like you said, with what happened with containers. But I do wish them well. There's lot of good people over there, it's some really cool tech, and I want to see a future for them.Corey: One last topic I want to get into before we wind up wrapping this episode is that you are someone who was nominated to come on the show by a couple of folks, which is always great. I'm always looking for recommendations on this. But what's odd is that you are—if we look at it and dig a little bit beneath the titles and whatnot, you even self-describe as your history is marketing leadership positions. It is uncommon for engineering-types to recommend that I talk to marketing folks.s personally I think that is a mistake; I consider myself more of a marketer than not in some respects, but it is uncommon, which means I have to ask you, what is your philosophy of marketing because it very clearly is differentiated in the public eye.Betty: I'm flattered. I will say that—and this goes to how I hire people and how I coach teams—it's you have to be super curious because there's a ton of bad marketing out there, where it's just kind of like, “Hey, we do these five things and we always do these five things: blah, blah, blah, blah, blah.” But I think it's really being curious about what is the thing that you're marketing? There are people who are just focused on the function of marketing and not the thing. Because you're doing your marketing job in the service of a thing, this new widget, this new whatever, and you got to be super curious about it.And I'll tell you that, for me, it's really hard for me to market something if I'm not excited about it. I have to personally be super excited about the tech or something happening in the industry, and it's, kind of like, an all-in thing for me. And so in that sense, I do spend a ton of time with engineers and end-users, and I really try to understand what's going on. I want to understand how the thing works, and I always ask them, “Well”—so I'll ask the engineers, like, “So… okay, this sounds really cool. You just described this new feature and you're super excited about it because you wrote it, but how is your end-user, the person you're building this for, how did they do this before? Help me understand. How did they do this before and why is this better?”Just really dig into it because for me, I want to understand it deeply before I talk about it. I think the thing is, it shows a tremendous amount of respect for the builder, and then to try to really be empathetic, to understand what they're doing and then partner with them—I mean, this sounds so business-y the way I'm talking about this—but really be a partner with them and just help them make their thing really successful. I'm like the other end; you're going to build this great thing and now I'm going to make it sound like it's the best thing that's ever happened. But to do that, I really need to deeply understand what it is, and I have to care about it, too. I have to care about it in the way that you care about it.Corey: I cannot effectively market or sell something that I don't believe in, personally. I also, to be clear because you are a marketing professional—or at least far more of one than I ever was—I do not view what I do is marketing; I view it as spectacle. And it's about telling stories to people, it's about learning what the market thinks about it, and that informs product design in many respects. It's about understanding the product itself. It's about being able to use the product.And if people are listening to this and think, “Wait a minute, that sounds more like DevRel.” I have news for you. DevRel is marketing, they're just scared to tell you that. And I know people are going to disagree with me on that. You're wrong. But that's okay; reasonable people can disagree.And that's how I see it is that, okay, I'll talk to people building the service, I'll talk to people using the service, but then I'm going to build something with the service myself because until then, it's all a game of who sounds the most convincing in the stories that they tell. But okay, you can tell an amazing story about something, but if it falls over when I tried to use it, well, I'm sorry, you're not being accurate in your descriptions of it.Betty: A hundred percent. I hate to say, like, you're storytellers, but that's a big part of it, but it's kind of like you want to tell the story, so you do something to that people believe a certain thing. But that's part of a curated experience because you want them to try this thing in a certain way. Because you've designed it for something. “I built a spoon. I want you to use that to eat your soup because you can't eat soup with a fork.”So, then you'll have this amazing soup-eating experience, but if I build you a spoon and then not give you any directions and you start throwing it at cars, you're going to be like, “This thing sucks.” So, I kind of think of it in that way. To your point of it has to actually work, it's like, but they also need to know, “What am I supposed to use it for?”Corey: The problem I've always had on some visceral level with formal marketing departments for companies is that they can say that a product that they sell is good, they can say that the product is great, or they can choose to say nothing at all about that product, but when there's a product in the market that is clearly a turd, a marketing department is never going to be able to say that, which I think erodes its authenticity in many respects. I understand the constraints behind, that truly I do, but it's the one superpower I think that I bring to the table where even when I do sponsorship stuff it's, you can buy my attention but not my opinion. Because the authenticity of me being trusted to call them like I see them, for lack of a better term, to my mind at least outweighs any short-term benefit from saying good things about a product that doesn't deserve them. Now, I've been wrong about things, sure. I have also been misinformed in both directions, thinking something is great when it's not, or terrible when it isn't or not understanding the use case, and I am thrilled to engage in those debates. “But this is really expensive when you run for this use case,” and the answer can be, “Well, it's not designed for that use case.” But the answer should not be, “No it's not.” I promise you, expensive is in the eye of the customer not the person building the thing.Betty: Yes. This goes back to I have to believe in the thing. And I do agree it's, like not [sigh]—it's not a panacea. You're not going to make Product A and it's going to solve everything. But being super clear and focused on what it is good for, and then please just try it in this way because that's what we built it for.Corey: I want to thank you for taking the time to have a what for some people is no doubt going to be perceived as a surprisingly civil conversation about things that I have loud, heated opinions about. If people want to learn more, where can they find you?Betty: Well, they can follow me on Twitter. But um, I'd say go to vmware.com/cloud for our work thing.Corey: Exactly. VM where? That's right. VM there. And we will, of course, put links to that in the [show notes 00:30:07].Betty: [laugh].Corey: Thank you so much for taking the time to speak with me. I appreciate it.Betty: Thanks, Corey.Corey: Betty Junod, Senior Director of Multi-Cloud Solutions at VMware. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with a loud, ranting comment at the end. Then, if you work for a company that is larger than 250 people or $10 million in revenue, please also Venmo me $5.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

About AbbyWith over twenty years in the tech world, Abby Kearns is a true veteran of the technology industry. Her lengthy career has spanned product marketing, product management and consulting across Fortune 500 companies and startups alike. At Puppet, she leads the vision and direction of the current and future enterprise product portfolio. Prior to joining Puppet, Abby was the CEO of the Cloud Foundry Foundation where she focused on driving the vision for the Foundation as well as  growing the open source project and ecosystem. Her background also includes product management at companies such as Pivotal and Verizon, as well as infrastructure operations spanning companies such as Totality, EDS, and Sabre.Links: Cloud Foundry Foundation: https://www.cloudfoundry.org Puppet: https://puppet.com Twitter: https://twitter.com/ab415 TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by Liquibase. If you're anything like me, you've screwed up the database part of a deployment so severely that you've been banned from touching every anything that remotely sounds like SQL, at at least three different companies. We've mostly got code deployments solved for, but when it comes to databases we basically rely on desperate hope, with a roll back plan of keeping our resumes up to date. It doesn't have to be that way. Meet Liquibase. It is both an open source project and a commercial offering. Liquibase lets you track, modify, and automate database schema changes across almost any database, with guardrails to ensure you'll still have a company left after you deploy the change. No matter where your database lives, Liquibase can help you solve your database deployment issues. Check them out today at liquibase.com. Offer does not apply to Route 53.Corey: This episode is sponsored in part by Honeycomb. When production is running slow, it's hard to know where problems originate: is it your application code, users, or the underlying systems? I've got five bucks on DNS, personally. Why scroll through endless dashboards, while dealing with alert floods, going from tool to tool to tool that you employ, guessing at which puzzle pieces matter? Context switching and tool sprawl are slowly killing both your team and your business. You should care more about one of those than the other, which one is up to you. Drop the separate pillars and enter a world of getting one unified understanding of the one thing driving your business: production. With Honeycomb, you guess less and know more. Try it for free at Honeycomb.io/screaminginthecloud. Observability, it's more than just hipster monitoring.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. Once upon a time, I was deep into the weeds of configuration management, which explains a lot, such as why it seems I don't know happiness in any meaningful sense. Then I wound up progressing into other areas of exploration, like the cloud, and now we know for a fact why happiness isn't a thing for me. My guest today is the former CEO of the Cloud Foundry Foundation and today is the CTO over at a company called Puppet, which we've talked about here from time to time. Abby Kearns, thank you for joining me. I appreciate your taking the time out of your day to suffer my slings and arrows.Abby: Thank you for having me. I have been looking forward to this for weeks.Corey: My stars, it seems like things are slow over there, and I kind of envy you for that. So, help me understand something; you went from this world of cloud-native everything, which is the joy of working with Cloud Foundry, to now working with configuration management. How is that not effectively Benjamin Button-ing your career. It feels like the opposite direction that most quote-unquote, “Digital transformations” like to play with. But I have a sneaking suspicion, there's more to it than I might guess from just looking at the label on the tin.Abby: Beyond I just love enterprise infrastructure? I mean, come on, who doesn't?Corey: Oh, yeah. Everyone loves to talk about digital transformation, reading about books like a Head in the Cloud to my children used to be a fun nightly activity before it was formally classified as child abuse. So yeah, I hear you, but it turns out the rest of the world doesn't necessarily agree with us.Abby: I do not understand it. I have been in enterprise infrastructure my entire career, which has been a really, really long time, back when Unix and Sun machines were still a thing. And I'll be a little biased here; I think that enterprise infrastructure is actually the most fascinating part of technology right now. And why is that? Well, we're in the process of actively rewritten everything that got us here.And we talk about infrastructure and everyone's like, “Yeah, sure, whatever,” but at the end of the day, it's the foundation that everything that you think is cool about technology is built on. And for those of us that really enjoy this space, having a front-row seat at that evolution and the innovation that's happening is really, really exciting and it creates a lot of interesting conversation, debate, evolution of technologies, and innovation. And are they all going to be on the money five, ten years from now? Maybe not, but they're creating an interesting space and discussion and just the work ahead for all of us across the board. And I'm kind of bucketing this pretty broadly, intentionally so because I think at the end of the day, all of us play a role in a bigger piece of pie, and it's so interesting to see how these things start to fit together.Corey: One of the things that I've noticed is that the things that get attention on the keynote stage of, “This is this far future, serverless, machine-learning Kubernetes, dingus nonsense,” great is—Abby: You forgot blockchain. [laugh].Corey: Oh, yeah. Oh, yeah blockchain as well. Like, what other things can we wind up putting into the buzzword thing to wind up guaranteeing that your seed round is at least $200 million? Great. There's that.But when you look at the actual AWS bill—my specialty, of course—and seeing where the money is actually going, it doesn't really look that different, as far as percentages go—even though the numbers are higher—than it did ten years ago, at least in the enterprise world. You're still buying a bunch of EC2 instances, you're still potentially modernizing to some of the managed services like RDS—which is Amazon's reimagining of what a database could be if you still had to manage the finicky bits, but had no control over when and how they worked—and of course, data transfer and disk. These are the basic building blocks of everything in cloud. And despite how much we talk about the super neat stuff, what we're doing is not reflected on the conference stage. So, I tend to view the idea of aspirational architecture as its own little world.There are still seasoned companies out there that are migrating from where they are today into this idea of, well, virtualization, we've just finally got our heads around that. Now, let's talk about this cloud thing; seems like a fad—in 2021. And people take longer to get to where they think they're going or where they intend to go than they plan for, and they get stuck somewhere and instead of a cloud migration, they're now hybrid because they can redefine things and declare victory when they plant that flag, and here we are. I'm not here to make fun of these companies because they're doing important work and these are super hard problems. But increasingly, it seems that the technology is not the thing that's holding them back or even responsible for their outcome so much as it is people.The more I work with tech, the more I realized that everything that's hard becomes people issues. Curious to get your take on that, given your somewhat privileged perspective as having a foot standing very deeply in each world.Abby: Yeah, and that's a super great point. And I also realized I didn't fully answer the first question either. So, I'll tie those two things together.Corey: That's okay, we're going to keep circling around until you get there. It's fine.Abby: It's been a long week, and it's only Wednesday.Corey: All day long, as it turns out.Abby: I have a whole soapbox that I drag around behind me about people and process, and how that's your biggest problem, not technology, and if you don't solve for the people in the process, I don't care what technology you choose to use, isn't going to fix your problem. On the other hand, if you get your people and process right, you can borderline use crayons and paper and get [laugh] really close to what you need to solve for.Corey: I have it on good authority that's known as IBM Cloud. Please continue.Abby: [laugh]. And so I think people and process are at the heart of everything. They're our biggest accelerators with technology and they're our biggest limitation. And you can cloud-native serverless your way into it, but if you do not actually do continuous delivery, if you did not actually automate your responses, if you do not actually set up the cross-functional teams—or sometimes fondly referred to as two-pizza teams—if you don't have those things set up, there isn't any technology that's going to make you deliver software better, faster, cheaper. And so I think I care a lot about the focus on that because I do think it is so important, but it's also—the reason a lot of people don't like to talk about it and deal with it because it's also the hardest.People, culture change, digital transformation, whatever you want to call it, is hard work. There's a reason so many books are written around DevOps. And you mentioned Gene Kim earlier, there's a reason he wrote The Phoenix Project; it's the people-process part is the hardest. And I do think technology should be an enabler and an accelerator, but it really has to pair up nicely with the people part. And you asked your earlier question about my move to Puppet.One of the things that I've learned a lot in running the Cloud Foundry Foundation, running an open-source software foundation, is you could a real good crash course in how teams can collaborate effectively, how teams work together, how decisions get made, the need for that process and that practice. And there was a lot of great context because I had access to so much interesting information. I got to see what all of these large enterprises were doing across the board. And I got to have a literal seat at the table for how a lot of the decisions are getting made around not only the open-source technologies that are going into building the future of our enterprise infrastructure but how a lot of these companies are using and leveraging those technologies. And having that visibility was amazing and transformational for myself.It gave me so much richness and context, which is why I have firmly believed that the people and process part were so crucial for many years. And I decided to go to a company that sold products. [laugh]. You're like, “What? What is she talking about now? Where is this going?”And I say that because running an open-source software foundation is great and it gives you so much information and so much context, but you have no access to customers and no access to products. You have no influence over that. And so when I thought about what I wanted to do next, it's like, I really want to be close to customers, I really want to be close to product, and I really want to be part of something that's solving what I look at over the next five to ten years, our biggest problem area, which is that tweener phase that we're going to be in for many years, which we were just talking about, which is, “I have some stuff on-prem and I have some stuff in a cloud—usually more than one cloud—and I got to figure out how to manage all of that.” And that is a really, really, really hard problem. And so when I looked at what Puppet was trying to do, and the opportunity that existed with a lot of the fantastic work that Puppet has done over the last 12 years around Desired State Configuration management, I'm like, “Okay, there's something here.”Because clearly, that problem doesn't go away because I'm running some stuff in the cloud. So, how do we start to think about this more broadly and expansively across the hybrid estate that is all of these different environments? And who is the most well-positioned to actually drive an innovative product that addresses that? So, that's my long way of addressing both of those things.Corey: No, it's a fair question. Friend of the show, Matt Stratton, is famous for saying that, “You cannot buy DevOps, but I sure would like to sell it to you,” and if you're looking at it from that perspective, Puppet is not far from what that product store look like in some ways. My first encounter with Puppet was back around 2009, 2010 or so, and I was using it in an environment I was working within and thought, “Okay, this is terrible, and it's crap, and obviously, I know what I'm doing far better than this, and the problem is the Puppet's a bad product.” So, I was one of the early developers behind SaltStack, which was a terrific, great way of approaching the problem from a novel perspective, and it wasn't crap; it was awesome. Right up until I saw the first time a customer deployed it and looked at their environment, and it wasn't crap, it was worse because it turns out that you can build a super finely crafted precision instrument that makes a fairly bad hammer, but that's how customers are going to use it anyway.Abby: Well, I mean, [sigh] look, you actually hit something that I think we don't actually talk about, which is how hard all of this shit really is. Automation is hard. Automation for distributed systems at scale is super duper hard. There isn't an easy way to solve that problem. And I feel like I learned a lot working with Cloud Foundry.Cloud Foundry is a Platform as a Service and it sits a layer up, but it had the same challenges in that solving the ability to run cloud-native applications and cloud-native workloads at scale and have that ephemerality to it and that resilience to it, and the things everyone wants but don't recognize how difficult it is, actually, to do that well. And I think the same—you know, that really set me up for the way that I think about the problem, even the layer down which is, running and managing desired state, which at the end of the day is a really fancy way of saying, “Does your environment look like the way you think it should? And if it doesn't, what are you going to do about it?” And it seems like, in this year of—what year are we again? 2021, maybe? I don't know. It feels like the last two years of, sort of, munged together?Corey: Yeah, the passing of time is something it's very hard for me to wrap my head around.Abby: But it feels like, I know some people, particularly those of us that have been in tech a long time are probably like, “Why are we still talking about that? Why is that a thing?” But that is still an incredibly hard problem for most organizations, large and small. So, I tend to spend a lot of time thinking about large enterprises, but in the day, you've got more than 20 servers, you're probably sitting around thinking, “Does my environment actually look the way I think it does? There's a new CVE that just came out. Am I able to address that?”And I think at the end of the day, figuring out how you can solve for that on-prem has been one of the things that Puppet has worked for, and done really, really well the last 12 years. Now, I think the next challenge is okay, how do you extend that out across your now bananas complex estate that is—I got a huge data estate, maybe one or two data centers, I got some stuff in AWS, I got some stuff in GCP, oh yeah, got a little thing over here and Azure, and oh, some guy spun up something on OCI. So, we got a little bit of everything. And oh, my God, the SolarWinds breach happened. Are we impacted? I don't know. What does that mean? [laugh].And I think you start to unravel the little pieces of that and it gets more and more complex. And so I think the problems that I was solving in the early aughts with servers seems trite now because you're like, I can see all of my servers; there's eight of them. Things seem fine. To now, you've got hundreds of thousands of applications and workloads, and some of them are serverless, and they're all over the place. And who has what, and where does it sit?And does it look like the way that I think it needs to so that I can run my business effectively? And I think that's really the power of it, but it's also one of those things that I don't feel like a lot of people like to acknowledge the complexity and the hardness of that because it's not just the technology problem—going back to your other question, how do we work? How do we communicate? What are our processes around dealing with this? And I think there's so much wrapped up in that it becomes almost like, how do you eat an elephant story, right? Yes, one bite at a time, but when you first look at the elephant, you're like, “Holy shit. This is big. What do I need to do?” And that I think is not something we all collectively spend enough time talking about is how hard this stuff is.Corey: One of the biggest challenges I see across the board is this idea of conference-ware style architecture; the greatest lie you ever see is someone talking about their infrastructure in public because peel it back a little bit and everything's messy, everything's disastrous, and everything's a tire fire. And we have this cult in tech—Abby: [laugh].Corey: —it's almost a cult where we have this idea that anything that isn't rewritten completely within the last six months based upon whatever is the hot framework now that is designed to run only in Google Chrome running on the latest generation MacBook Pro on a gigabit internet connection is somehow less than. It's like, “So, what does that piece of crap do?” And the answer is, “Well, a few $100 million a quarter in revenue, so how about you watch your mouth?” Moving those things is delicate; moving those things is fraught, and there are a lot of different stakeholders to the point where one of the lessons I keep learning is, people love to ask me, “What is Amazon's opinion of you?” Turns out that there's no Ted Amazon who works over there who forms a single entity's opinion. It's a bunch of small teams. Some of them like me, some of them can't stand me, far and away the majority don't know who I am. And that is okay. In theory; in practice, I find it completely unforgivable because how dare you? But I understand it's—Abby: You write a memo, right now. [laugh].Corey: Exactly. Companies are people and people are messy, and for better or worse, it is impossible to patch them. So, you have to almost route around them. And that was something that I found that Puppet did very well, coming from the olden days of sysadmin work where we spend time doing management [bump 00:15:53] the systems by hand. Like, oh, I'm going to do a for loop. Once I learned how to script. Before that, I use Cluster SSH and inadvertently blew away a University's entire config file what starts up on boot across their entire FreeBSD server fleet.Abby: You only did it once, so it's fine.Corey: Oh, yeah. I'm never going to screw up again. Well, not like that. In other ways. Absolutely, but at least my errors will be novel.Abby: Yeah. It's learning. We all learn. If you haven't taken something down in production in real-time, you have not lived. And also you [laugh] haven't done tech. [laugh].Corey: Oh, yeah, you either haven't been allowed close enough to anything that's important enough to be able to take down, you're lying to me, or thirdly—and this is possible, too—you're not yet at a point in your career where you're allowed to have access to the breaky parts. And that's fine. I mean, my argument has always been about why I'd be a terrible employee at Google, for example, is if I went in maliciously on day one, I would be hard-pressed to take down google.com for one hour. If I can't have that much impact intentionally going in as a bad actor, it feels like there'd be how much possible upside, positive impact can I have what everyone's ostensibly aligned around the same thing?It's the challenge of big companies. It's gaining buy-in, it's gaining investment in the idea and the direction you're going in. Things always take longer, you have to wind up getting multiple stakeholders on board. My consulting practice is entirely around helping save money on the AWS bill. You'd think it would be the easiest thing in the world to sell, but talking to big companies means a series of different sales conversations with different folks, getting them all on the same page. What we do functionally isn't so much look at the computer parts as it is marriage counseling between engineering and finance. Different languages, different ways of thinking about things, ostensibly the same goals.Abby: I mean, I don't think that's a big company problem. I think that's an every company problem if you have more than, like, five people in your company.Corey: The first few years here, it was just me and I had none of those problems. I had very different problems, but you know—and then we started bringing other people in, it's like, “Oh, yeah, things were great until we hired people. Ugh, mistake. Never do that.” And yeah, it turns out that's not particularly sustainable.Abby: Stakeholder management is hard. And you mentioned something about routing around. Well, you can't actually route around people, unfortunately. You have to get people to buy in, you have to bring people along on the journey. And not everybody is at the same place in the way they think about the work you're doing.And that's true at any company, big or small. I think it just gets harder and more complex as the company gets bigger because it's harder to make the changes you need to make fast enough, but I'd say even at a company the size of Puppet, we have the exact same challenges. You know, are the teams aligned? Are we aligned on the right things? Are we focusing on the right things?Or, do we have the right priorities in our backlog? How are we doing the work that we do? And if you're trying to drive innovation, how fast are we innovating? Are we innovating fast enough? How tight are our feedback loops?It's one of those things where the conversations that you and I have had externally with customers are the same conversations I have internally all the time, too. Let's talk about innovators' dilemma. [laugh]. Let's talk about feedback loop. Let's talk about what does it mean to get tighter feedback loops from customers and the field?And how do you align those things to the priorities in your backlog? And it's one of those never-ending challenges that's messy and complicated. And technology can enable it, but the technology is also messy and hard. And I do love going to conferences and seeing how pretty and easy things could look, and it's definitely a great aspiration for us to all shoot for, but at the end of the day, I think we all have to recognize there's a ton of messiness that goes on behind to make that a reality and to make that really a product and a technology that we can sell and get behind, but also one that we buy in, too, and are able to use. So, I think we as a technology industry, and particularly those of us in the Bay Area, we do a disservice by talking about how easy things are and why—you know, I remember a conversation I had in 2014 where someone asked me if Docker was already passe because everybody was doing containerized applications, and I was like, “Are they? Really? Is that an everyone thing? Or is that just an ‘us' thing?” [laugh].Corey: Well, they talk about it on the conference stages an awful lot, but yeah. New problems that continue to arise. I mean, I look back at my early formative years as someone who could theoretically be brought out in public and it was through a consulting project, where I was a traveling trainer for Puppet back in 2014, 2015, and teaching people who hadn't had exposure before what Puppet was about. And there was a definite experience in some of the people attending class where they were very opposed to the idea. And dig down a little bit, it's not that they had a problem with the software, it's not that they had a problem with any of the technical bits.It's that they made the mistake that so many technologists made—I know I have, repeatedly—of identifying themselves with the technology that they work on. And well, in some cases, yeah, the answer was that they ran a particular script a bunch of times and if you can automate that through something like Puppet or something else, well, what does that mean for them? We see it much larger-scale now with people who are, okay, I'm in the data center working on the storage arrays. When that becomes just an API call or—let's be serious, despite what we see in conference stages—when it becomes clicking buttons in the AWS console, then what does that mean for the future of their career? The tide is rising.And I can't blame them too much for this; you've been doing this for 25 years, you don't necessarily want to throw all that away and start over with a whole new set of concepts and the rest because unlike what Twitter believes, there are a bunch of legitimate paths in this industry that do treat it as a job rather than an all-consuming passion. And I have no negative judgment toward folks who walk down that direction.Abby: Most people do. And I think we have to be realistic. It's not just some. A lot of people do. A lot of people, “This is my nine-to-five job, Monday through Friday, and I'm going to go home and I'm going to spend time with my family.”Or I'm going to dare I say—quietly—have a life outside of technology. You know, but this is my job. And I think we have done a disservice to a lot of those individuals who for better or for worse, they just want to go in and do a job. They want to get their job done to the best of their abilities, and don't necessarily have the time—or if you're a single parent, have the flexibility in your day to go home and spend another five, six hours learning the latest technology, the latest programming language, set up your own demo environment at home, play around with AWS, all of these things that you may not have the opportunity to do. And I think we as an industry have done a disservice to both those individuals, as well in putting up really imaginary gates on who can actually be a technologist, too.Corey: This episode is sponsored by our friends at Oracle Cloud. Counting the pennies, but still dreaming of deploying apps instead of "Hello, World" demos? Allow me to introduce you to Oracle's Always Free tier. It provides over 20 free services and infrastructure, networking databases, observability, management, and security.And - let me be clear here - it's actually free. There's no surprise billing until you intentionally and proactively upgrade your account. This means you can provision a virtual machine instance or spin up an autonomous database that manages itself all while gaining the networking load, balancing and storage resources that somehow never quite make it into most free tiers needed to support the application that you want to build.With Always Free you can do things like run small scale applications, or do proof of concept testing without spending a dime. You know that I always like to put asterisks next to the word free. This is actually free. No asterisk. Start now. Visit https://snark.cloud/oci-free that's https://snark.cloud/oci-free.Corey: Gatekeeping, on some level, is just—it's a horrible thing. Something I found relatively early on is that I didn't enjoy communities where that was a thing in a big way. In minor ways, sure, absolutely. I wound up gravitating toward Ubuntu rather than Debian because it turned out that being actively insulted when I asked how to do something wasn't exactly the most welcoming, constructive experience, where they, “Read the manual.” “Yeah, I did that and it was incomplete and contradictory, and that's why I'm here asking you that question, but please continue to be a condescending jackwagon. I appreciate that. It really just reminds me that I'm making good choices with my life.”Abby: Hashtag-RTFM. [laugh].Corey: Exactly. In my case, fine, its water off a duck's back. I can certainly take it given the way that I dish it out, but by the same token, not everyone has a quote-unquote, thick skin, and I further posit that not everyone should have to have one. You should not get used to personal attacks as a prerequisite for working in this space. And I'm very sensitive to the idea that people who are just now exploring the cloud somehow feel that they've missed out on their career, and that so there's somehow not appropriate for this field, or that it's not for them.And no, are you kidding me? You know that overwhelming sense of confusion you get when you look at the AWS console and try and understand what all those services do? Yeah, I had the same impression the first time I saw it and there were 12 services; there's over 200 now. Guess what? I've still got it.And if I am overwhelmed by it, I promise there's no shame in anyone else being overwhelmed by it, too. We're long since past the point where I can talk incredibly convincingly about AWS services that don't exist to AWS employees and not get called out on it because who in the world has that entire Rolodex of services shoved into their heads who isn't me?Abby: I'd say you should put out… a call for anyone that does because I certainly do not memorize the services that are available. I don't know that anyone does. And I think even more broadly, is, remember when the landscape diagram came out from the CNCF a couple of years ago, which it's now, like… it's like a NASCAR logo of every logo known to man—Corey: Oh today, there's over 400 icons on it the last time I saw—I saw that thing come out and I realized, “Wow, I thought I was going to shit-posting,” but no, this thing is incredible. It's, “This is great.” My personal favorite was zooming all the way in finding a couple of logos on in the same box three times, which is just… spot on. I was told later, it's like, “Oh, those represent different projects.” I'm like, “Oh, yeah, must have missed that in the legend somewhere.” [laugh]. It's this monstrous, overdone thing.Abby: But the whole point of it was just, if I am running an IT department, and I'm like, “Here you go. Here's a menu of things to choose,” you're just like, “What do I do with this information? Do I choose one of each? All the above? Where do I go? And then, frankly, how do I make them all work together in my environment?” Because they all serve very different problems and they're tackling different aspects of that problem.And I think I get really annoyed with myself as an industry—like, ourselves as an industry because it's like, “What are we doing here?” We're trying to make it harder for people, not only to use the technology, to be part of it. And I think any efforts we can make to make it easier and more simple or clear, we owe it to ourselves to be able to tell that story. Which now the flip side of that is describing cloud-native in the cloud, and infrastructure and automation is really, really hard to do [laugh] in a way that doesn't use any of those words. And I'm just as guilty of this, of describing things we do and using the same language, and all of a sudden you're looking at it this says the same thing is 7500 other websites. [laugh]. So.Corey: Yep. I joke at RSA's Expo Hall is basically about twelve companies selling different things. Sure, each one has a whole bunch of booths with different logos and different marketing copy, but it's the same fundamental product. Same challenge here. And this is, to me, the future of cloud, this is where it's going, where I want something that will—in my case, I built a custom URL shortener out of DynamoDB, API Gateway, Lambda, et cetera, and I built this thing largely as a proof of concept because I wanted to have experience playing with these tools.And that was great, not but if I'm doing something like that in production, I'm going with Bitly or one of the other services that provide this where someone is going to maintain it full time. Unless it is the core of what I'm doing, I don't want to build it myself from popsicle sticks. And moving up the stack to a world of folks who are trying to solve a business problem and they don't want to deal with the ten prerequisite services to understand the cloud, and then a whole bunch of other things tied together, and the billing, and the flow becomes incredibly problematic to understand—not to mention insecure: because we don't understand it, you don't know what your risk exposure is—people don't want that. They—Abby: Or to manage it.Corey: Yeah.Abby: Just the day-to-day management. Care and feeding, beyond security. [laugh].Corey: People's time is free. So, yeah. For example, do I write my own payroll system? Absolutely not. I have the good sense to pay a turnkey company to handle that for me because mistakes will show.I started my career running email systems. I pay for Google workspaces—or GSuite, or Gmail, or whatever the hell they're calling it this week—because it's not core and central to my business. I want a thing that winds up solving a business problem, and I will pay commensurately to the value that thing delivers, not the individual constituent costs of the components that build it together. Because until you're significantly scaled out and it is the core of what you do, you're spending more on people to run the monstrous thing than you are for the thing itself. That's always the way it works.So, put your innovation where it matters for your business. I posit the for an awful lot of the things we're building, in order to achieve those outcomes, this isn't it.Abby: Agreed. And I am a big believer in if I can use off-the-shelf software, I will because I don't believe in reinventing everything. Now, having said that, and coming off my soapbox for just a hot minute, I will say that a lot of what's happening, and going back to where I started around the enterprise infrastructure, we're reinventing so many things that there is a lot of new things coming up. We've talked about containers, we've talked about Kubernetes, around container scheduling, container orchestration, we haven't even mentioned service mesh, and sidecars, and all of the new ways we're approaching solving some of these older problems. So, there is the need for a broad proliferation of technology until the contraction phase, where it all starts to fundamentally clicks together.And that's really where the interesting parts happen, but it's also where the confusion happens because, “Okay, what do I use? How do I use it? How do these pieces fit together? What happens when this changes? What does this mean?”And by the way, if I'm an enterprise company, I'm a payroll company, what's the one thing I care about? My payroll software. [laugh]. And that's the problem I'm solving for. So, I take a little umbrage sometimes with the frame that every company is a software company because every company is not a software company.Every company can use technology in ways to further their business and more and more frequently, that is delivering their business value through software, but if I'm a payroll company, I care about delivering that payroll capabilities to my customer, and I want to do it as quickly as possible, and I want to leverage technology to help me do that. But my endgame is not that technology; my endgame is delivering value to my customers in real and meaningful ways. And I worry, sometimes, that those two things get conflated together. And one is an enabler of the other; the technology is not the outcome.Corey: And that is borderline heresy for an awful lot of folks out there in the space, I wish that people would wake up a little bit more and realize that you have to build a thing that solves customer pain, ideally, an expensive customer pain, and then they will basically rush to hurl money at you. Now, there are challenges and inflections as you go, and there's a whole bunch of nuances that can span entire fields of endeavor that I am hand-waving over here, and that's fine, but this is the direction I think we're going and this is the dawning awareness that I hope and trust we'll see start to take root in this industry.Abby: I mean, I hope so. I do take comfort in the fact that a lot of the industry leaders I'm starting to see, kind of, equate those two things more closely in the top [track 00:31:20]. Because it's a good forcing function for those of us that are technologists. At the end of the day, what am I doing? I am a product company, I am selling software to someone.So clearly, obviously, I have a vested interest in building the best software out there, but at the end of the day, for me, it's, “Okay, how do I make that truly impactful for customers, and how do I help them solve a problem?” And for me, I'm hyper-focused on automation because I honestly feel like that is the biggest challenge for most companies; it's the hardest thing to solve. It's like getting into your auto-driving car for the first time and letting go the steering wheel and praying to the software gods that that software is actually going to work. But it's the same thing with automation; it's like, “Okay, I have to trust that this is going to manage my environment and manage my infrastructure in a factual way and not put me on CNN because I just shut down entire customer environment,” or if I'm an airline and I've just had a really bad week because I've had technology problems. [laugh]. And so I think we have to really take into consideration that there are real customer problems on the other end of that we have to help solve for.Corey: My biggest problem is the failure mode of this is not when people watch the conference-ware presentations is that they're not going to sit there and think, “Oh, yeah, they're just talking about a nuanced thing that doesn't apply to our constraints, and they're hand-waving over a lot of stuff,” it's that, “Wow, we suck.” And that's not the takeaway anyone should ever have. Even Netflix doesn't operate the way that Netflix says that they do in their conference talks. It's always fun sitting next to someone from the company that's currently presenting and saying something to them, like, “Wow, I wish we did things that way.” And they said, “Yeah, I wish we did, too.”And it's always the case because it's very hard to get on stage and talk for 45 minutes about here's what we completely screwed up on, especially at the large publicly traded companies where it's, “Wait, why did our stock price just dive five perce—oh, my God, what did you say on stage?” People care [laugh] about those things, and I get it; there's a risk factor that I don't have to deal with here.Abby: I wish people would though. It would be so refreshing to hear someone like, “You know what? Ohh, we really messed this up, and let me walk you through what we did.” [laugh]. I think that would be nice.Corey: On some level, giving that talk in enough detail becomes indistinguishable from rage-quitting in public.Abby: [laugh].Corey: I mean, I'm there for it. Don't get me wrong. But I would love to see it.Abby: I don't think it has to be rage-quitting. One of the things that I talk to my team a lot about is the safety to fail. You can't take risk if you're too afraid to fail, right? And I think you can frame failure in a way of, “Hey, this didn't work, but let me walk you through all the amazing things we learned from this. And here's how we used that to take this and make this thing better.”And I think there's a positive way to frame it that's not rage-quitting, but I do think we as an industry gloss over those learnings that you absolutely have to do. You fail; everything does not work the first time perfectly. It is not brilliant out the gate. If you've done an MVP and it's perfect and every customer loves it, well then, you sat on that for way too long. [laugh]. And I think it's just really getting comfortable with this didn't work the first time or the fourth, but look, at time seven, this is where we got and this is what we've learned.Corey: I want to thank you for taking so much time out of your day to wind up speaking to me about things that in many cases are challenging to talk about because it's the things people don't talk about in the real world. If people want to learn more about what you're up to, who you are, et cetera, where can they find you?Abby: They can find me on the Twitters at @ab415. I think that's the best way to start, although I will say that I am not as prolific as you are on Twitter.Corey: That's a good thing.Abby: I'm a half-assed Tweeter. [laugh]. I will own it.Corey: Oh, I put my full ass into it every time, in every way.Abby: [laugh]. I do skim it a lot. I get a lot of my tech news from there. Like, “What are people mad about today?” And—Corey: The daily outrage. Oh, yeah.Abby: The daily outrage. “What's Corey ranting about today? Let's see.” [laugh].Corey: We will, of course, put a link to your Twitter profile in the [show notes 00:35:39]. Thank you so much for taking the time to speak with me. I appreciate it.Abby: Hey, it was my pleasure.Corey: Abby Kearns, CTO at Puppet. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, along with a comment telling me about the amazing podcast content you create, start to finish, at Netflix.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

About JordanJordan is a self proclaimed “hacker.” Links:Twitter: https://twitter.com/jordansissel TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by “you”—gabyte. Distributed technologies like Kubernetes are great, citation very much needed, because they make it easier to have resilient, scalable, systems. SQL databases haven't kept pace though, certainly not like no SQL databases have like Route 53, the world's greatest database. We're still, other than that, using legacy monolithic databases that require ever growing instances of compute. Sometimes we'll try and bolt them together to make them more resilient and scalable, but let's be honest it never works out well. Consider Yugabyte DB, its a distributed SQL database that solves basically all of this. It is 100% open source, and there's not asterisk next to the “open” on that one. And its designed to be resilient and scalable out of the box so you don't have to charge yourself to death. It's compatible with PostgreSQL, or “postgresqueal” as I insist on pronouncing it, so you can use it right away without having to learn a new language and refactor everything. And you can distribute it wherever your applications take you, from across availability zones to other regions or even other cloud providers should one of those happen to exist. Go to yugabyte.com, thats Y-U-G-A-B-Y-T-E dot com and try their free beta of Yugabyte Cloud, where they host and manage it for you. Or see what the open source project looks like—its effortless distributed SQL for global apps. My thanks to Yu—gabyte for sponsoring this episode.Corey: This episode is sponsored in part by our friends at VMware. Let's be honest—the past year has been far from easy. Due to, well, everything. It caused us to rush cloud migrations and digital transformation, which of course means long hours refactoring your apps, surprises on your cloud bill, misconfigurations and headache for everyone trying manage disparate and fractured cloud environments. VMware has an answer for this. With VMware multi-cloud solutions, organizations have the choice, speed, and control to migrate and optimize applications seamlessly without recoding, take the fastest path to modern infrastructure, and operate consistently across the data center, the edge, and any cloud. I urge to take a look at vmware.com/go/multicloud. You know my opinions on multi cloud by now, but there's a lot of stuff in here that works on any cloud. But don't take it from me thats: VMware.com/go/multicloud and my thanks to them again for sponsoring my ridiculous nonsense.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. I've been to a lot of conference talks in my life. I've seen good ones, I've seen terrible ones, and then I've seen the ones that are way worse than that. But we don't tend to think in terms of impact very often, about how conference talks can move the audience.In fact, that's the only purpose of giving a talk ever—to my mind—is you're trying to spark some form of alchemy or shift in the audience and convince them to do something. Maybe in the banal sense, it's to sign up for something that you're selling, or to go look at your website, or to contribute to a project, or maybe it's to change the way they view things. One of the more transformative talks I've ever seen that shifted my outlook on a lot of things was at [SCALE 00:01:11] in 2012. Person who gave that talk is my guest today, Jordan Sissel, who, among many other things in his career, was the original creator behind logstash, which is the L in ELK Stack. Jordan, thank you for joining me.Jordan: Thanks for having me, Corey.Corey: I don't know how well you remember those days in 2012. It was the dark times; we thought oh, the world is going to end; that wouldn't happen until 2020. But it was an interesting conference full of a bunch of open-source folks, it was my local conference because I lived in Los Angeles. And it was the thing I looked forward to every year because I would always go and learn something new. I was in the trenches in those days, and I had a bunch of problems that looked an awful lot like other people's problems, and having a hallway track where, “Hey, how are you solving this problem?” Was a big deal. I missed those days in some ways.Jordan: Yeah, SCALE was a particularly good conference. I think I made it twice. Traveling down to LA was infrequent for me, but I always enjoyed how it was a very communal setting. They had dedicated hallway tracks. They had kids tracks, which I thought was great because folks couldn't usually come to conferences if they couldn't bring their kids or they had to take care of that stuff. But having a kids track was great, they had kids presenting. It felt more organic than a lot of other conferences did, and that's kind of what drew me to it initially.Corey: Yeah, it was my local network. It turns out that the Southern California tech community is relatively small, and we all go different lives. And it's LA, let's face it, I lived there for over a decade. Flaking as a way of life. So yeah, well, “Oh, we'll go out and catch dinner. Ooh, have to flake at the last minute.” If you're one of the good people, you tell people you're flaking instead of just no-showing, but it happens.But this was the thing that we would gather and catch up every year. And, “Oh, what have you been doing?” “Wow, you work in that company now? Congratulations, slash, what's wrong with you?” It was fun, just sort of a central sync point. It started off as hanging out with friends.And in those days, I was approaching the idea of, “You know what? I should learn to give a conference talk someday. But let's be clear. People don't give conference talks; legends give conference talks. And one day, I'll be good enough to get on stage and give a talk to my peers at a conference.”Now, the easy, cynical interpretation would be, “Well, but I saw your talk and I figured, hey, any jackhole can get up there. If he can do it, anyone can.” But that's not at all how it wound up impacting me. You were talking about logstash, which let's start there because that's a good entry point. Logstash was transformative for me.Before that, I'd spent a lot of time playing around with syslog, usually rsyslog, but there are other stories here of when a system does something and it spits out logs—ideally—how do you make sure you capture those logs in a reliable way so if you restart a computer, you don't wind up with a gap in your logs? If it's the right computer, it could be a gap in everything's logs while that thing is coming back up. And let's avoid single points of failure and the rest. And I had done all kinds of horrible monstrosities, and someone asked me at one point—Jordan: [laugh]. Guilty.Corey: Yeah. Someone said, “Well, there are a couple of options. Why don't you use Splunk?” And the answer is that I don't have a spare princess lying around that I can ransom back to her kingdom, so I can't afford it. “Okay, what about logstash?” And my answer was, “What's a logstash?” And thus that sound was Pandora's Box creaking open.So, I started playing with it and realized, “Okay, this is interesting.” And I lost track of it because we have demands on our time. Then I was dragged into a session that you gave and you explained what logstash was. I'm not going to do nearly as good of a job as you can on this. What the hell was logstash, for folks who are not screaming at syslog while they first hear of it.Jordan: All right. So, you mentioned rsyslog, and there's—old is often a pejorative of more established projects because I don't think these projects are bad. But rsyslog, syslog-ng, things like that were common to see for me as a sysadmin. But to talk about logstash, we need to go back a little further than 2012. So, the logstash project started—Corey: I disagree because I wasn't aware of it until 2012. Until I become aware of something it doesn't really exist. That's right, I have the object permanence of an infant.Jordan: [laugh].That's fair. And I've always felt like perception is reality, so if someone—this gets into something I like to say, but if someone is having a bad time or someone doesn't know about something, then it might as well not exist. So, logstash as a project started in 2008, 2009. I don't remember when the first commits landed, but it was, gosh, it's more than ten years ago now.But even before that in college, I was fortunate to, through a network of friends, get a job as a sysadmin. And as a sysadmin, you stare at logs a lot to figure out what's going on. And I wanted a more interesting way to process the logs. I had taught myself regular expressions and it wasn't finding joy in it… at all, like pretty much most people, probably. Either they look at regular expressions and just… evacuate with disgust, which is absolutely an appropriate response, or they dive into it and they have to use it for their job.But it wasn't enjoyable, and I found myself repeating stuff a lot. Matching IP addresses, matching strings, URLs, just trying to pull out useful information about what is going on?Corey: Oh, and the timestamp problem, too. One of the things that I think people don't understand who have not played in this space, is that all systems do have logs unless you've really pooched something somewhere—Jordan: Yeah.Corey: —and it shows that at this point in time, this thing happened. As we start talking about multiple computers and distributed systems—but even on the same computer—great, so at this time there was something that showed up in the system log because there was a disk event or something, and at the same time you have application logs that are talking about what the application running is talking about. And that is ideally using a somewhat similar system to do this, but often not. And the way that timestamps are expressed in these are radically different and the way that the log files themselves are structured. One might be timestamp followed by hostname followed by error code.The other one might be hostname followed by a timestamp—in a different format—followed by a copyright notice because a big company got to it followed by the actual event notice, and trying to disambiguate all of these into a standardized form was first obnoxious, and secondly, very important because you want to see the exact chain of events. This also leads to a separate sidebar on making sure that all the clocks are synchronized, but that's a separate story for another time. And that's where you enter the story in many respects.Jordan: Right. So, my thought around what led to logstash is you can take a sysadmin or software IT developer—whatever—expert, and you can sit them in front of a bunch of logs and they can read them and say, “That's the time it happened. That's the user who caused this action. This is the action.” But if you try and abstract and step away, and so you ask how many times did this action happen? When did this user appear? What time did this happen?You start losing the ability to ask those questions without being an expert yourself, or sitting next to an expert and having them be your keyboard. Kind of a phenomenon I call the human keyboard problem where you're speaking to a computer, but someone has to translate for you. And so in around 2004, I was super into Perl. No shocker that I enjoyed—ish. I sort of enjoyed regular expressions, but I was super into Perl, and there was a Perl module called Regexp::Common which is a library of regular expressions to match known things: IP addresses, certain kinds of timestamps, quoted strings, and whatnot.Corey: And this stuff is always challenging because it sounds like oh, an IP address. One of the interview questions I hated the most someone asked me was write a regular expression to detect an IP address. It turns out that to do this correctly, even if you bound it to ipv4 only, the answer takes up multiple lines on a screen.Jordan: Oh, for sure.Corey: It's enormous.Jordan: It's like a full page of—Corey: It is.Jordan: —of code you can't read. And that's one of the things that, it was sort of like standing on the shoulders of the person who came before; it was kind of an epiphany to me.Corey: Yeah. So, I can copy and paste that into my code, but someone who has to maintain that thing after I get fired is going to be, “What the hell is this and what does it do?” It's like it's the blessed artifact that the ancients built it and left it there like it's a Stargate sitting in your code. And it's, “We don't know how it works; we're scared to break it, so we don't even look at that thing directly. We just know that we put nonsense in, an IP address comes out, and let's not touch it, ever again.”Jordan: Exactly. And even to your example, even before you get fired and someone replaces you and looks at your regular expression, the problem I was having was, I would have this library of copy and pasteable things, and then I would find a bug, and edge case. And I would fix that edge case but the other 15 scripts that were using the same way regular expression, I can't even read them anymore because I don't carry that kind of context in my head for all of that syntax. So, you either have to go back and copy and paste and fix all those old regular expressions. Or you just say, “You know what? We're not going to fix the old code. We have a new version of it that works here, but everywhere else this edge case fails.”So, that's one of the things that drew me to the Regexp::Common library in Perl was that it was reusable and things had names. It was, “I want to match an IP address.” You didn't have to memorize that long piece of text to precisely and accurately accept only regular expressions and rejects things that are not. You just said, “Give me the regular expression that matches an IP.” And from that library gave me the idea to write grok.Well, if we could name things, then maybe we could turn that into some kind of data structure, sort of the combination of, “I have a piece of log data, and I as an expert, I know that's an IP address, that's the username, and that's the timestamp.” Well, now I can apply this library of regular expressions that I didn't have to write and hopefully has a unit test suite, and say, now we can pull out instead of that plain piece of text that is hard to read as a non-expert, now I can have a data structure we can format however we want, that non-experts can see. And even experts can just relax and not have to be full experts all the time, using that part of your brain. So, now you can start getting towards answering search-oriented questions. “How many login attempts happened yesterday from this IP address?”Corey: Right. And back then, the way that people would do these things was Elasticsearch. So, that's the thing you shove all your data into in a bunch of different ways and you can run full-text queries on it. And that's great, but now we want to have that stuff actually structured, and that is sort of the magic of logstash—which was used in conjunction with Elasticsearch a lot—and it turns out that typing random SQL queries in the command line is not generally how most business users like to interact with this stuff, seems to be something dashboard-y-like, and the project that folks use for that was Kibana. And ELK Stack became a thing because Elasticsearch in isolation can do a lot but it doesn't get you all the way there for what people were using to look at logs.Jordan: You're right.Corey: And Kibana is also one of the projects that Elastic owned, and at some point, someone looks around, like, “Oh, logstash. People are using that with us an awful lot. How big is the company that built that? Oh, it's an open-source project run by some guy? Can we hire that guy?” And the answer is, “Apparently,” because you wound up working as an Elastic employee for a while.Jordan: Yeah. It was kind of an interesting journey. So, in the beginning of logstash in 2009, I kind of had this picture of how I wanted to solve log processing search challenges. And I broke it down into a couple of parts of visualization—to be clear, I broke it down in my head, not into code, but visualization, kind of exploration, there's the processing and transmission, and then there's storage and search. And I only felt confident really attending to a solution for one of those parts. And I picked log processing partly because I already had a jumpstart from a couple of years prior, working on grok and feeling really comfortable with regular expressions. I don't want to say good because that's—Corey: You heard it here first—Jordan: [laugh].Corey: —we found the person that knows regular expressions. [laugh].Jordan: [laugh]. And logstash was being worked on to solve this problem of taking your data, processing it, and getting it somewhere. That's why logstash has so many outputs, has so many inputs, and lots of filters. And about I think a year into building logstash, I had experimented with storage and search backends, and I never found something that really clicked with me. And I was experimenting with Leucine, and knowing that I could not complete this journey because that the problem space is so large, it would be foolish of me to try to do distributed log stores or anything like that, plus visualization.I just didn't have the skills or the time in the day. I ended up writing a frontend for logstash called logstash-web—naming things is hard—and I wasn't particularly skilled or attentive to that project, and it was more of a very lightweight frontend to solve the visualization, the exploration aspect. And about a year into logstash being alive, I found Elasticsearch. And what clicked with me from being a sysadmin and having worked at large data center companies in the past is I know the logs on a single system are going to quickly outgrow it. So, whatever storage system will accept these logs, it's got to be easy to add new storage.And Elasticsearch first-day promise was it's distributed; you can add more nodes and go about your day. And it fulfilled that promise and I think it still fulfills that promise that if you're going to be processing terabytes of data, yeah, just keep dumping it in there. That's one of the reasons I didn't try and even use MySQL, or Postgres, or other data systems because it didn't seem obvious how to have multiple storage servers collecting this data with those solutions, for me at the time.Corey: It turns out that solving problems like this that are global and universal lead to massive adoption very quickly. I want to get this back a bit before you wound up joining Elastic because you get up on stage and you talked through what this is. And I mentioned at the start of this recording, that it was one of those transformative talks. But let's be clear here, I don't remember 95% of how logstash works. Like, the technology you talked about ten years ago is largely outmoded slash replaced slash outdated today. I assure you, I did not take anything of note whatsoever from your talk regarding regular expressions, I promise. And—Jordan: [laugh]. Good.Corey: But that's not the stuff that was transformative to me. What was, was the way that you talked about these things. And there was the first time I'd ever heard the phrase that if a new user has a bad time, it's a bug. This was 2012. The idea of empathy hadn't really penetrated into the ops and engineering spaces in any meaningful way yet. It was about gatekeeping, it was about, “Read the manual fool”—Jordan: Yes.Corey: —if people had questions. And it was actively user-hostile. And it was something that I found transformative of, forget the technology piece for a second; this is a story about how it could be different. Because logstash was the vehicle to deliver a message that transcended far beyond the boundaries of how to structure your logs, or maybe the other boundaries of regular expressions, I'm never quite sure where those things start and stop. But it was something that was actively transformative where you're on stage as someone who is a recognized authority in the space, and you're getting up there and you're sending an implicit message—both explicitly and by example—of be nice to people; demonstrate empathy. And that left a hell of an impact. And—Jordan: Thank you.Corey: I wound up doing a spot check just now, and I wound up looking at this and sure enough, early in 2013, I wound up committing—it's still in the history of the changelog for logstash because it's open-source—I committed two pull requests and minutes apart, two submissions—I don't know if pull requests were even a thing back then—but it wound up in the log. Because another project you were renowned for was fpm: Effing Package Manager if I'm—is that what the acronym stands for, or am I misremembering?Jordan: [laugh]. We'll go with that. I'm sure, vulgar viewers will know what the F stands for, but you don't have to say it. It's just Effing Package Management.Corey: Yeah.Jordan: But yeah, I think I really do believe that if a user, especially if a new user has a bad time, it's a bug, and that came from many years of participating at various levels in open-source, where if you came at it with a tinkerer's or a hacker's mindset and you think, “This project is great. I would like it to do one additional thing, and I would like to talk to someone about how to make it do that one additional thing.” And you go find the owners or the maintainers of that project, and you come in with gusto and energy, and you describe what you want to do and, first, they say, “What you want to do is not possible.” They don't even say they don't want to do it; they frame the whole universe against you. “It's not possible. Why would you want to do that? If you want to make that, do it yourself.”You know, none of these things are an extended hand, a lowered ladder, an open door, none of those. It's always, “You're bothering me. Go away. Please read the documentation and see where we clearly”—which they don't—“Document that this is not a thing we're interested in.” And I came to the conclusion that any future open-source or collaborative work that I worked on, it's got to be from a place where, “You're welcome, and whatever contributions or participation levels you choose, are okay. And if you have an idea, let's talk about it. If you're having a bad time, let's figure out how to solve it.”Maybe the solution is we point you in the right direction to the documentation, if documentation exists; maybe we find a bug that we need to fix. The idea that the way to build communities is through kindness and collaboration, not through walls or gatekeeping or just being rude. And I really do think that's one of the reasons logstash became so successful. I mean, any particular technology could have succeeded in the space that logstash did, but I believe that it did so because of that one piece of framework where if a new user has a bad time, it's a bug. Because to me, that opens the door to say, “Yeah, you know what? Some of the code I write is not going to be good. Or, the thing you want to do is undocumented. Or the documentation is out of date. It told you a lie and you followed the documentation and it misled you because it's incorrect.”We can fix that. Maybe we don't have time to fix it right now. Maybe there's no one around to fix it, but we can at least say, “You know what? That information is incorrect, and I'm sorry you were misled. Come on into the community and we'll figure it out.” And one of the patterns I know is, on the IRC channel, which is where the logstash real-time community chat… I don't know how to describe that.Corey: No, it was on freenode. That's part of the reason I felt okay, talking to you. At that point. I was volunteer network staff. This is before freenode turned into basically a haven for Nazis this past year.Jordan: Yeah. It was still called lilo… lilonet [crosstalk 00:20:20]—Corey: No, the open freenode network, that predates me. This was—yeah, lilo—Jordan: Okay.Corey: —died about six years prior. But—Jordan: Oh, all right.Corey: Freenode's been around a long time. What make this thing work was that I was network staff, and that means that I had a bit of perceived authority—it's a chat room; not really—but it was one of those things where it was at least, “Okay, this is not just some sketchy drive-by rando,” which I very much was, but I didn't present that way, so I could strike up conversations. But with you talking about this stuff, I never needed to be that person. It was just if someone wants to pitch in on this, great; more hands make lighter work. Sure.Jordan: Yeah, for sure.Corey: And for me, the interesting part is not even around the logstash aspects so much; it's your other project, fbm. Well, one of your other projects. Back in 2012, that was an interesting year for me. Another area that got very near and dear to my heart in open-source world was the SaltStack project; I was contributor number 15. And I didn't know how Python worked. Not that I do now, but I can fake it better now.And Tom Hatch, the guy that ran the project before it was a company was famous for this where I could send in horrifying levels of code, and every time he would merge it in and then ten minutes later, there would be another patch that comes in that fixes all bugs I just introduced and it was just such a warm onboarding. I'm not suggesting that approach and I'm not saying it's scalable, but I started contributing. And I became the first Debian and Ubuntu packager for SaltStack, which was great. And I did a terrible job at it because—let me explain. I don't know if it's any better now, but back in those days, there were multiple documentation sources on the proper way to package software.They were all contradictory with each other, there was no guidance as to when to follow each one, there was never a, “You know nothing about packaging; here's what you need to know, step-by-step,” and when you get it wrong, they yell at you. And it turns out that the best practice then to get it formally accepted upstream—which is what I did—is do a crap-ass job, and then you'll wind up with a grownup coming in, like, “This is awful. Move.” And then they'll fix it and yell at you, and gatekeep like hell, and then you have a package that works and gets accepted upstream because the magic incantation has been said somewhere. And what I loved about fpm was that I could take any random repo or any source tarball or anything I wanted, run it through with a single command, and it would wind up building out a RPM and a Deb file—and I don't know what else it's supported; those are the ones I cared about—that I could then install on a system. I put in a repo and add that to a sources list on systems, and get to automatically install so I could use configuration management—like SaltStack—to wind up installing custom local packages. And oh, my God, did the packaging communities for multiple different distros hate you—Jordan: Yep.Corey: —and specifically what you had built because this was not the proper way to package. How dare you solve an actual business problem someone has instead of forcing them to go to packaging school where the address is secret, and you have to learn that. It was awful. It was the clearest example that I can come up with of gatekeeping, and then you're coming up with fbm which gets rid of user pain, and I realized that in that fight between the church of orthodoxy of, “This is how it should be done,” and the, “You're having a problem; here's a tool that makes it simple,” I know exactly what side of that line I wanted to be on. And I hadn't always been previously, and that is what clarified it for me.Jordan: Yeah, fbm was a really delightful enjoyment for me to build. The origins of that was I worked at a company and they were all… I think, at that time, we were RPM-based, and then as folks tend to do, I bounced around between jobs almost every year, so I went from one place that—Corey: Hey, it's me.Jordan: [laugh]. Right? And there's absolutely nothing wrong with leaving every year or staying longer. It's just whatever progresses your career in the way that you want and keeps you safe and your family safe. But we were using RPM and we were building packages already not following the orthodoxy.A lot of times if you ask someone how to build a package for Fedora, they'll point you at the Maximum RPM book, and that's… a lot of pages, and honestly, I'm not going to sit down and read it. I just want to take a bunch of files, name it, and install it on 30 machines with Puppet. And that's what we were doing. Cue one year later, I moved to a new company, and we were using Debian packages. And they're the same thing.What struck me is they are identical. It's a bunch of files—and don't pedant me about this—it's a bunch of files with a name, with some other sometimes useful metadata, like other names that you might depend on. And I really didn't find it enjoyable to transfer my knowledge of how to build RPMs, and the tooling and the structures and the syntaxes, to building Debian packages. And this was not for greater publication; this was I have a bunch of internal applications I needed to package and deploy with, at the time it was Puppet. And it wasn't fun.So, I did what we did with grok which was codify that knowledge to reduce the burden. And after a few, probably a year or so of that, it really dawned on me that a generality is all packaging formats are largely solving the same problem and I wanted to build something that was solving problems for folks like you and me: sysadmins, who were handed a pile of code and they needed to get it into production. And I wasn't interested in formalities or appeasing any priesthoods or orthodoxies about what really—you know, “You should really shine your package with this special wax,” kind of thing. Because all of the documentation for Debian packages, Fedora packages are often dedicated to those projects. You're going to submit a package to Fedora so that the rest of the world can use it on Fedora. That wasn't my use case.Corey: Right. I built a thing and a thing that I built is awesome and I want the world to use it, so now I have to go to packaging school? Not just once but twice—Jordan: Right.Corey: —and possibly more. That's awful.Jordan: Or more. Yeah. And it's tough.Corey: This episode is sponsored in part by our friends at Jellyfish. So, you're sitting in front of your office chair, bleary eyed, parked in front of a powerpoint and—oh my sweet feathery Jesus its the night before the board meeting, because of course it is! As you slot that crappy screenshot of traffic light colored excel tables into your deck, or sift through endless spreadsheets looking for just the right data set, have you ever wondered, why is it that sales and marketing get all this shiny, awesome analytics and inside tools? Whereas, engineering basically gets left with the dregs. Well, the founders of Jellyfish certainly did. That's why they created the Jellyfish Engineering Management Platform, but don't you dare call it JEMP! Designed to make it simple to analyze your engineering organization, Jellyfish ingests signals from your tech stack. Including JIRA, Git, and collaborative tools. Yes, depressing to think of those things as your tech stack but this is 2021. They use that to create a model that accurately reflects just how the breakdown of engineering work aligns with your wider business objectives. In other words, it translates from code into spreadsheet. When you have to explain what you're doing from an engineering perspective to people whose primary IDE is Microsoft Powerpoint, consider Jellyfish. Thats Jellyfish.co and tell them Corey sent you! Watch for the wince, thats my favorite part.Corey: And this gets back to what I found of—it was rare that I could find a way to contribute to something meaningfully, and I was using logstash after your talk, I'd started using it and rolling it out somewhere, and I discovered that there wasn't a Debian package for it—the environment I was in at that time—or Ubuntu package, and, “Hey Jordan, are you the guy that wrote fpm and there isn't a package here?” And the thing is is that you would never frame it this way, but the answer was, of course, “Pull requests welcome,” which is often an invitation to do free volunteer work for companies, but this was an open-source project that was not backed by a publicly-traded company; it was some guy. And of course, I'll pitch in on that. And I checked the commit log on this for what it is that I see, and sure enough, I have two commits. The first one was on Sunday night in February of 2013, and my commit message was, “Initial packaging work for Deb building.” And sure enough, there's a bunch of files I put up there and that's great. And my second and last commit was 12 minutes later saying, “Remove large binary because I'm foolish.” Yeah.Jordan: Was that you? [laugh].Corey: Yeah. Oh, yeah, I'm sure—yeah, it was great. I didn't know how Git worked back then. I'm sure it's still in the history there. I wonder how big that binary is, and exactly how much I have screwed people over in the last decade since.Jordan: I've noticed this over time. And every now and then you'd be—I would be or someone would be on a slow internet connection—which again, is something that we need to optimize for, or at least be aware of and help where we can—someone would be cloning logstash on an airplane or something like that, or rural setting, and they would say, “It gets stuck at 76% for, like, ten minutes.” And you would go back and dust off your tome of how to use Git because it's very difficult piece of software to use, and you would find this one blob and I never even looked at it who committed it or whatever, but it was like I think it was 80 Megs of a JAR file or a Debian package that was [unintelligible 00:28:31] logstash release. And… [laugh] it's such a small world that you're like, yep, that was me.Corey: Oh, yeah. Oh, yeah. Let's check this just for fun here. To be clear, the entire repository right now is 167 Megs, so that file that I had up there for all of 13 minutes lives indelibly in Git history, and it is fully half of the size—Jordan: Yep.Corey: —of the entirety of the logstash project. All right, then. I didn't realize this was one of those confess your sins episodes, but here we are.Jordan: Look, sometimes we put flags on the moon, sometimes we put big files in git. You could just for posterity, we could go back and edit the history and remove that, but it never became important to do it, it wasn't loud, people weren't upset enough by it, or it didn't come up enough to say, “You know what? This is a big file.” So, it's there. You left your mark.Corey: You know, we take what we can get. It's an odd time. I'll have to do some digging around; I'm sure I'll tweet about this as soon as I get a bit more data on it, but I wonder how often people have had frustration caused by that. There's no ill intent here, to be very clear, but it was instead, I didn't know how Git worked very well. I didn't know what I was doing in a lot of respects, and sure enough in the fullness of time, some condescending package people came in and actually made this right.And there is a reasonable, responsible package now because, surprise, of course there is. But I wonder how much inadvertent pain I caused people by that ridiculous commit. And it's the idea of impact and how this stuff works. I'm not happy that people are on a plane with a slow connection had a wait an extra minute or two to download that nonsense. It's one of those things that is, oops. I feel like a bit of a heel for that, not for not knowing something, but for causing harm to folks. Intent doesn't outweigh impact. There is a lesson in there for it.Jordan: Agreed. On that example, I think one of the things… code is not the most important thing I can contribute to a project, even though I feel very confident in my skills in programming in a variety of environments. I think the number one thing I can do is listen and look for sources of pain. And people would come in and say, “I can't get this to work.” And we would work together and figure out how to make it work for their use case, and that could result in a new feature, a bug fix, or some documentation improvements, or a blog post, or something like that.And I think in this case, I don't really recall any amount of noise for someone saying, “Cloning the Git repository is just a pain in the butt.” And I think a lot of that is because either the people who would be negatively impacted by that weren't doing that use case, they were downloading the releases, which were as small as we can possibly get them, or they were editing files using the GitHub online edit the file thing, which is a totally acceptable, it's perfectly fine way to do things in Git. So, I don't remember anyone complaining about that particular file size issue. The Elasticsearch repository is massive and I don't think it even has binaries. It just has so much more—Corey: Someone accidentally committed their entire production test data set at one point and oops-a-doozy. Yeah, it's not the most egregious harm I've ever caused—Jordan: Yeah.Corey: —but it's there. The thing that, I guess, resonates with me and still does is the lessons I learned from you, I could sum them up as being not just empathy-driven—because that's the easy answer—but the other layers were that you didn't need to be the world's greatest expert in a thing in order to credibly give a conference talk. To be clear, you were miles ahead of me and still are in a lot of different areas—Jordan: Thanks.Corey: —and that's fine. But you don't need to be the—like, you are not the world's greatest expert on empathy, but that's what I took from the talk and that's what it was about. It also taught me that things you can pick up from talks—and other means—there are things you can talk about in terms of technology and there are things you can talk about in terms of people, and the things about people do not have expiration dates in the same way that technology does. And if I'm going to be remembered for impact on people versus impact on technology, for me, there's no contest. And you forced me to really think about a lot of those things that it started my path to, I guess, becoming a public speaker and then later all the rest that followed, like this podcast, the nonsense on Twitter, and all the rest. So, it is, I guess, we can lay the responsibility for all that at your feet. Enjoy the hate mail.Jordan: Uhh, my email address is now closed. I'm sorry.Corey: Exactly.Jordan: Well, I appreciate the kind words.Corey: We'll get letters on this one.Jordan: [laugh].Corey: It's the impact that people have, and someti—I don't think you knew at the time that that's the impact you were having. It matters.Jordan: I agree. I think a lot of it came from how do I want to experience this? And it was much later that it became something that was really outside of me, in the sense that it was building communities. One of the things I learned shortly after—or even just before—joining Elastic was how many folks were looking to solve a problem, found logstash, became a participant in the community, and that participation could just be anything, just hanging out on IRC, on the mailing list, whatever, and the next step for them was to get a better paying job in an environment they enjoyed that helped them take the next step in their career. Some of those people came to work with me at Elastic; some of them started to work on the logstash team at some point they decided because a lot of logstash users were sysadmins.And on the logstash team, we were all developers; we weren't sysadmins, there was nothing to operate. And a lot of folks would come on board and they were like, “You know what? I'm not enjoying writing Ruby for my job.” And they could take the next step to transition to the support team or the sales engineer team, or cloud operations team at Elastic. So, it was really, like you mentioned, it has nothing to do with the technology of—to me—why these projects are important.They became an amplifier and a hand to pull people up to go the next step they need to go. And on the way maybe they can make a positive impact in the communities they participate in. If those happen to be fpm or logstash, that's great, but I think I want folks to see that technology doesn't have to be a grind of getting through gatekeepers, meeting artificial barriers, and things like that.Corey: The thing that I took, too, is that I gave a talk in 2015 or'16, which is strangely appropriate now: “Terrible ideas in Git.” And yes, checking large binaries in is one of the terrible ideas I talk about. It's Git through counter-example. And around that time, I also gave a talk for a while on how to handle a job interview and advance your career. Only one of those talks has resulted in people approaching me even years later saying that what I did had changed aspects of their life. It wasn't the Git one. And that's the impact it comes down to. That is the change that I wanted to start having because I saw someone else do it and realized, you know, maybe I could possibly be that good someday. Well, I'd like to think I made it, on some level.Jordan: [laugh]. I'm proud of the impact you've made. And I agree with you, it is about people. Even with fpm where I was very selfishly tickling my own itch, I don't want to remember all of this stuff and I also enjoy operating outside of the boundaries of a church or whatever the priesthoods that say, “This is how you must do a thing,” I knew there was a lot of folks who worked at jobs and they didn't have authority, and they had to deploy something, and they knew if they could just package it into a Debian format, or an RPM format, or whatever they needed to do, they could get it deployed and it would make their lives easier. Well, they didn't have the time or the energy or the support in order to learn how to do that and fpm brought them that success where you can say, “Here's a bunch of files; here's a name, poof, you have a package for whatever format you want.”Where I found fpm really take off is when Gem and Python and Node.js support were added. The sysadmins were kind of sandwiched in between—in two impossible worlds where they are only authorized to deploy a certain package format, but all of their internal application developer teams were using Node.js and newer technologies, and all of those package formats were not permitted by whoever had the authority to permit those things at their job. But now they had a tool that said, “You know what? We can just take that thing, we'll take Django and Python, and we'll make it an RPM and we won't have to think a lot about it.”And that really, I think—to me, my hope was that it de-stresses that sort of work environment where you're not having to do three weeks of brand new work every time someone releases something internally in your company; you can just run a script that you wrote a month ago and maintain it as you go.Corey: Wouldn't that be something?Jordan: [laugh]. Ideally, ideally.Corey: Jordan, I want to thank you for not only the stuff you did ten years ago, but also the stuff you just said now. If people want to learn more about you, how you view the world, see what you're up to these days, where can they find you?Jordan: I'm mostly active on Twitter, at @jordansissel, all one word. Mostly these days, I post repair stuff I do on the house. I'm a stay-at-home full0 time dad these days, and… I'm still doing maintenance on the projects that need maintenance, like fpm or xdotool, so if you're one of those users, I hope you're happy. If you're not happy, please reach out and we'll figure out what the next steps can be. But yeah. If you like bugs, especially spiders—or if you don't like spiders and you want to like spiders, check me out on Twitter. I'm often posting macro photos, close-up photos of butterflies, bees, spiders, and the like.Corey: And we will, of course, throw links to that in the [show notes 00:38:10]. Jordan, thank you so much for your time today. It's appreciated.Jordan: Thank you, Corey. It's good talking to you.Corey: Jordan Sissel, founder of logstash and currently, blissfully, not working on a particular corporate job. I envy him, some days. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, along with an angry comment in which you have also embedded a large binary.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

About CraigCraig McLuckie is a VP of R&D at VMware in the Modern Applications Business Unit.  He joined VMware through the Heptio acquisition where he was CEO and co-founder. Heptio was a startup that supported the enterprise adoption of open source technologies like Kubernetes.  He previously worked at Google where he co-founded the Kubernetes project, was responsible for the formation of CNCF, and was the original product lead for Google Compute Engine.Links: VMware: https://www.vmware.com Twitter: https://twitter.com/cmcluck LinkedIn: https://www.linkedin.com/in/craigmcluckie/ TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at the Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part my Cribl Logstream. Cirbl Logstream is an observability pipeline that lets you collect, reduce, transform, and route machine data from anywhere, to anywhere. Simple right? As a nice bonus it not only helps you improve visibility into what the hell is going on, but also helps you save money almost by accident. Kind of like not putting a whole bunch of vowels and other letters that would be easier to spell in a company name. To learn more visit: cribl.ioCorey: This episode is sponsored in part by Thinkst. This is going to take a minute to explain, so bear with me. I linked against an early version of their tool, canarytokens.org in the very early days of my newsletter, and what it does is relatively simple and straightforward. It winds up embedding credentials, files, that sort of thing in various parts of your environment, wherever you want to; it gives you fake AWS API credentials, for example. And the only thing that these things do is alert you whenever someone attempts to use those things. It's an awesome approach. I've used something similar for years. Check them out. But wait, there's more. They also have an enterprise option that you should be very much aware of canary.tools. You can take a look at this, but what it does is it provides an enterprise approach to drive these things throughout your entire environment. You can get a physical device that hangs out on your network and impersonates whatever you want to. When it gets Nmap scanned, or someone attempts to log into it, or access files on it, you get instant alerts. It's awesome. If you don't do something like this, you're likely to find out that you've gotten breached, the hard way. Take a look at this. It's one of those few things that I look at and say, “Wow, that is an amazing idea. I love it.” That's canarytokens.org and canary.tools. The first one is free. The second one is enterprise-y. Take a look. I'm a big fan of this. More from them in the coming weeks.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. My guest today is Craig McLuckie, who's a VP of R&D at VMware, specifically in their modern applications business unit. Craig, thanks for joining me. VP of R&D sounds almost like it's what's sponsoring a Sesame Street episode. What do you do exactly?Craig: Hey, Corey, it's great to be on with you. So, I'm obviously working within the VMware company, and my charter is really looking at modern applications. So, the modern application platform business unit is really grounded in the work that we're doing to make technologies like Kubernetes and containers, and a lot of developer-centric technologies like Spring, more accessible to developers to make sure that as developers are using those technologies, they shine through on the VMware infrastructure technologies that we are working on.Corey: Before we get into, I guess, the depths of what you're focusing on these days, let's look a little bit backwards into the past. Once upon a time, in the dawn of the modern cloud era—I guess we'll call it—you were the original product lead for Google Compute Engine or GCE. How did you get there? That seems like a very strange thing to be—something that, “Well, what am I going to build? Well, that's right; basically a VM service for a giant company that is just starting down the cloud path,” back when that was not an obvious thing for a company to do.Craig: Yeah, I mean, it was as much luck and serendipity as anything else, if I'm going to be completely honest. I spent a lot of time working at Microsoft, building enterprise technology, and one of the things I was extremely excited about was, obviously, the emergence of cloud. I saw this as being a fascinating disrupter. And I was also highly motivated at a personal level to just make IT simpler and more accessible. I spent a fair amount of time building systems within Microsoft, and then even a very small amount of time running systems within a hedge fund.So, I got, kind of, both of those perspectives. And I just saw this cloud thing as being an extraordinarily exciting way to drive out the cost of operations, to enable organizations to just focus on what really mattered to them which was getting those production systems deployed, getting them updated and maintained, and just having to worry a little bit less about infrastructure. And so when that opportunity arose, I jumped with both feet. Google obviously had a reputation as a company that was born in the cloud, it had a reputation of being extraordinarily strong from a technical perspective, so having a chance to bridge the gap between enterprise technology and that cloud was very exciting to me.Corey: This was back in an era when, in my own technical evolution, I was basically tired of working with Puppet as much as I had been, and I was one of the very early developers behind SaltStack, once upon a time—which since then you folks have purchased, which shows that someone didn't do their due diligence because something like 41 lines of code in the current release version is still assigned to me as per git-blame. So, you know, nothing is perfect. And right around then, then I started hearing about this thing that was at one point leveraging SaltStack, kind of, called Kubernetes, which, “I can't even pronounce that, so I'm just going to ignore it. Surely, this is never going to be something that I'm going to have to hear about once this fad passes.” It turns out that the world moved on a little bit differently.And you were also one of the co-founders of the Kubernetes project, which means that it seems like we have been passing each other in weird ways for the past decade or so. So, you're working on GCE, and then one day you want to, what, sitting up and deciding, “I know, we're going to build a container orchestration system because I want to have something that's going to take me 20 minutes to explain to someone who's never heard of these concepts before.” How did this come to be?Craig: It's really interesting, and a lot of it was driven by necessity, driven by a view that to make a technology like Google Compute Engine successful, we needed to go a little bit further. When you look at a technology like Google Compute Engine, we'd built something that was fabulous and Google's infrastructure is world-class, but there's so much more to building a successful cloud business than just having a great infrastructure technology. There's obviously everything that goes with that in terms of being able to meet enterprises where they are and all the—Corey: Oh, yeah. And everything at Google is designed for Google scale. It's, “We built this thing and we can use it to stand up something that is world-scale and get 10 million customers on the first day that it launches.” And, “That's great. I'm trying to get a Hello World page up and maybe, if I shoot for the moon, it can also run WordPress.” There's a very different scale of problem.Craig: It's just a very different thing. When you look at what an organization needs to use a technology, it's nice that you can take that, sort of, science-fiction data center and carve it up into smaller pieces and offer it as a virtual machine to someone. But you also need to look at the ISV ecosystem, the people that are building the software, making sure that it's qualified. You need to make sure that you have the ability to engage with the enterprise customer and support them through a variety of different functions. And so, as we were looking at what it would take to really succeed, it became clear that we needed a little more; we needed to, kind of, go a little bit further.And around that time, Docker was really coming into its full. You know, Docker solved some of the problems that organizations had always struggled with. Virtual machine is great, but it's difficult to think about. And inside Google, containers we're a thing.Corey: Oh, containers have a long and storied history in different areas. From my perspective, Docker solves the problem of, “Well, it works on my machine,” because before something like Docker, the only answer was, “Well, backup your email because your laptop's about to be in production.”Craig: [laugh]. Yeah, that's exactly right. You know, I think when I look at what Docker did, and it was this moment of clarity because a lot of us had been talking about this and thinking about it. I remember turning to Joe while we were building Compute Engine and basically said, “Whoever solves the packaging the way that Google did internally, and makes that accessible to the world is ultimately going to walk away with a game.” And I think Docker put lightning in a bottle.They really just focused on making some of these technologies that underpinned the hyperscalers, that underpinned the way that, like, a Google, or a Facebook, or a Twitter tended to operate, just accessible to developers. And they solved one very specific thing which was that packaging problem. You could take a piece of software and you could now package it up and deploy it as an immutable thing. So, in some ways, back to your own origins with SaltStack and some of the technologies you've worked on, it really was an epoch of DevOps; let's give developers tools so that they can code something up that renders a production system. And now with Docker, you're able to shift that all left. So, what you produced was the actual deployable artifact, but that obviously wasn't enough by itself.Corey: No, there needed to be something else. And according to your biography, not only it says here that, I quote, “You were responsible for the formation of the CNCF, or Cloud Native Computing Foundation,” and I'm trying to understand is that something that you're taking credit for or being blamed for? It really seems like it could go either way, given the very careful wording there.Craig: [laugh]. Yeah, it could go either way. It certainly got away from us a little bit in terms of just the scope and scale of what was going on. But the whole thesis behind Kubernetes, if you just step back a little bit, was we didn't need to own it; Google didn't need to own it. We just needed to move the innovation boundary forwards into an area that we had some very strong advantages.And if you look at the way that Google runs, it kind of felt like when people were working with Docker, and you had technologies like Mesos and all these other things, they were trying to put together a puzzle, and we already had the puzzle box in front of us because we saw how that technology worked. So, we didn't need to control it, we just needed people to embrace it, and we were confident that we could run it better. But for people to embrace it, it couldn't be seen as just a Google thing. It had to be a Google thing, and a Red Hat thing, and an Amazon thing, and a Microsoft thing, and something that was really owned by the community. So, the inspiration behind CNCF was to really put the technology forwards to build a collaborative community around it and to enable and foster this disruption.Corey: At some point after Kubernetes was established, and it was no longer an internal Google project but something that was handed over to a foundation, something new started to become fairly clear in the larger ecosystem. And it's sort of a microcosm of my observation that the things that startups are doing today are what enterprises are going to be doing five years from now. Every enterprise likes to imagine itself a startup; the inverse is not particularly commonly heard. You left Google to go found Heptio, where you were focusing on enterprise adoption of open-source technologies, specifically Kubernetes, but it also felt like it was more of a cultural shift in many respects, which is odd because there aren't that many startups, at least in that era, that were focused on bringing startup technologies to the enterprise, and sneaking in—or at least that's how it felt—the idea of culture change as well.Craig: You know, it's really interesting. Every enterprise has to innovate, and people tend to look at startups as being a source of innovation or a source of incubation. What we were trying to do with Heptio was to go the other way a little bit, which was, when you look at what West Coast tech companies were doing, and you look at a technology like Kubernetes—or any new technology: Kubernetes, or KNative, or there's some of these new observability capabilities that are starting to emerge in this ecosystem—there's this sort of trickle-across effect, where it's starts with the West Coast tech companies that build something, and then it trickles across to a lot of the progressive forward-leaning enterprise organizations that have the scale to consume those technologies. And then over time, it becomes mainstream. And when I looked at a technology like Kubernetes, and certainly through the lens of a company like Google, there was an opportunity to step back a little bit and think about, well, Google's really this West Coast tech company, and it's producing this technology, and it's working to make that more enterprise-centric, but how about going the other way?How about meeting enterprise organizations where they are—enterprise organizations that aspire to adopt some of these practices—and build a startup that's really about just walking the journey with customers, advocating for their needs, through the lens of these open-source communities, making these open-source technologies more accessible. And that was really the thesis around what we were doing with Heptio. And we worked very hard to do exactly as you said which is, it's not just about the tech, it's about how you use it, it's about how you operate it, how you set yourself up to manage it. And that was really the core thesis around what we were pursuing there. And it worked out quite well.Corey: Sitting here in 2021, if I were going to build something from scratch, I would almost certainly not use Kubernetes to do it. I'd probably pick a bunch of serverless primitives and go from there, but what I respect and admire about the Kubernetes approach is companies can't generally do that with existing workloads; you have to meet them where they are, as you said. ‘Legacy' is a condescending engineering phrase for ‘it makes money.' It's, “Oh, what does that piece of crap do?” “Oh, about $4 billion a year.” So yeah, we're going to be a little delicate with what it does.Craig: I love that observation. I always prefer the word ‘heritage' over the word legacy. You got to—Corey: Yeah.Craig: —have a little respect. This is the stuff that's running the world. This is the stuff that every transaction is flowing through.And it's funny, when you start looking at it, often you follow the train along and eventually you'll find a mainframe somewhere, right? It is definitely something that we need to be a little bit more thoughtful about.Corey: Right. And as cloud continues to eat the world well, as of the time of this recording, there is no AWS/400, so there is no direct mainframe option in most cloud providers, so there has to be a migration path; there has to be a path forward, that doesn't include, “Oh, and by the way, take 18 months to rewrite everything that you've built.” And containers, particularly with an orchestration model, solve that problem in a way that serverless primitives, frankly, don't.Craig: I agree with you. And it's really interesting to me as I work with enterprise organizations. I look at that modernization path as a journey. Cloud isn't just a destination: there's a lot of different permutations and steps that need to be taken. And every one of those has a return on investment.If you're an enterprise organization, you don't modernize for modernization's sake, you don't embrace cloud for cloud's sake. You have a specific outcome in mind, “Hey, I want to drive down this cost,” or, “Hey, I want to accelerate my innovation here,” “Hey, I want to be able to set my teams up to scale better this way.” And so a lot of these technologies, whether it's Kubernetes, or even serverless is becoming increasingly important, is a capability that enables a business outcome at the end of the day. And when I think about something like Kubernetes, it really has, in a way, emerged as a Goldilocks abstraction. It's low enough level that you can run pretty much anything, it's high enough level that it hides away the specifics of the environment that you want to deploy it into. And ultimately, it renders up what I think is economies of scope for an organization. I don't know if that makes sense. Like, you have these economies of scale and economies of scope.Corey: Given how down I am on Kubernetes across the board and—at least, as it's presented—and don't take that personally; I'm down on most modern technologies. I'm the person that said the cloud was a passing fad, that virtualization was only going to see limited uptake, that containers were never going to eat the world. And I finally decided to skip ahead of the Kubernetes thing for a minute and now I'm actually going to be positive about serverless. Given how wrong I am on these things, that almost certainly dooms it. But great, I was down on Kubernetes for a long time because I kept seeing these enterprises and other companies talking about their Kubernetes strategy.It always felt like Kubernetes was a means to an end, not an end in and of itself. And I want to be clear, I'm not talking about vendors here because if you are a software provider to a bunch of companies and providing Kubernetes is part and parcel of what you do, yeah, you need a Kubernetes strategy. But the blue-chip manufacturing company that is modernizing its entire IT estate, doesn't need a Kubernetes strategy as such. Am I completely off base with that assessment?Craig: No, I think you're pointing at something which I feel as well. I mean, I'll be honest, I've been talking about [laugh] Kubernetes since day one, and I'm kind of tired of talking about Kubernetes. It should just be something that's there; you shouldn't have to worry about it, you shouldn't have to worry about operationalizing it. It's just an infrastructure abstraction. It's not in and of itself an end, it's simply a means to an end, which is being able to start looking at the destination you're deploying your software into as being more favorable for building distributed systems, not having to worry about the mechanics of what happens if a single node fails? What happens if I have to scale this thing? What happens if I have to update this thing?So, it's really not intended—and it never was intended—to be an end unto itself. It was really just intended to raise the waterline and provide an environment into which distributed applications can be deployed that felt entirely consistent, whether you're building those on-premises, in the public cloud, and increasingly out to the edge.Corey: I wound up making a tweet, couple years back, specifically in 2019, that the nuclear hot take: “Nobody will care about Kubernetes in five years.” And I stand by it, but I also think that's been wildly misinterpreted because I am not suggesting in any way that it's going to go away and no one is going to use it anymore. But I think it's going to matter in the same way as the operating system is starting to, the way that the Linux virtual memory management subsystem does now. Yes, a few people in specific places absolutely care a lot about those things, but most companies don't because they don't have to. It's just the way things are. It's almost an operating system for the data center, or the cloud environment, for lack of a better term. But is that assessment accurate? And if you don't wildly disagree with it, what do you think of the timeline?Craig: I think the assessment is accurate. The way I always think about this is you want to present your engineers, your developers, the people that are actually taking a business problem and solving it with code, you want to deliver to them the highest possible abstraction. The less they have to worry about the infrastructure, the less they have to worry about setting up their environment, the less they have to worry about the DevOps or DevSecOps pipeline, the better off they're going to be. And so if we as an industry do our job right, Kubernetes is just the water in which IT swims. You know, like the fish doesn't see the water; it's just there.We shouldn't be pushing the complexity of the system—because it is a fancy and complex system—directly to developers. They shouldn't necessarily have to think like, “Oh, I need to understand all of the XYZ is about how this thing works to be able to build a system.” There will be some engineers that benefit from it, but there are going to be other engineers that don't. The one thing that I think is going to—you know, is a potential change on what you said is, we're going to see people starting to program Kubernetes more directly, whether they know it or not. I don't know if that makes sense, but things like the ability for Kubernetes to offer up a way for organizations to describe the desired state of something and then using some of the patterns of Kubernetes to make the world into that shape is going to be quite pervasive, and I'm really seeing signs that we're seeing it.So yes, most developers are going to be working with higher abstractions. Yes, technologies like Knative and all of the work that we at VMware are doing within the ecosystem will render those higher abstractions to developers. But there's going to be some really interesting opportunities to take what made Kubernetes great beyond just, “Hey, I can put a Docker container down on a virtual machine,” and start to think about reconciler-driven IT: being able to describe what you want to have happen in the world, and then having a really smart system that just makes the world into that shape.Corey: This episode is sponsored by our friends at Oracle HeatWave is a new high-performance accelerator for the Oracle MySQL Database Service. Although I insist on calling it “my squirrel.” While MySQL has long been the worlds most popular open source database, shifting from transacting to analytics required way too much overhead and, ya know, work. With HeatWave you can run your OLTP and OLAP, don't ask me to ever say those acronyms again, workloads directly from your MySQL database and eliminate the time consuming data movement and integration work, while also performing 1100X faster than Amazon Aurora, and 2.5X faster than Amazon Redshift, at a third of the cost. My thanks again to Oracle Cloud for sponsoring this ridiculous nonsense. Corey: So, you went from driving Kubernetes adoption into the enterprise as the founder and CEO of Heptio, to effectively, acquired by one of the most enterprise-y of enterprise companies, in some respects, VMware, and your world changed. So, I understand what Heptio does because, to my mind, a big company is one that is 200 people. VMware has slightly more than that at last count, and I sort of lose track of all the threads of the different things that VMware does and how it operates. I could understand what Heptio does. What I don't understand is what, I guess, your corner of VMware does. Modern applications means an awful lot of things to an awful lot of people. I prefer to speak it with a condescending accent when making fun of those legacy things that make money—not a popular take, but it's there—how do you define what you do now?Craig: So, for me, when you talk about modern application platform, you can look at it one of two ways. You can say it's a platform for modern applications, and when people have modern applications, they have a whole variety of different ideas in the head: okay, well, it's microservices-based, or it's API-fronted, it's event-driven, it's supporting stream-based processing, blah, blah, blah, blah, blah. There's all kinds of fun, cool, hip new patterns that are happening in the segment. The other way you could look at it is it's a modern platform for applications of any kind. So, it's really about how do we make sense of going from where you are today to where you need to be in the future?How do we position the set of tools that you can use, as they make sense, as your organization evolves, as your organization changes? And so I tend to look at my role as being bringing these capabilities to our existing product line, which is, obviously, the vSphere product line, and it's almost a hyperscale unto itself, but it's really about that private cloud experience historically, and making those capabilities accessible in that environment. But there's another part to this as well, which is, it's not just about running technologies on vSphere. It's also about how can we make a lot of different public clouds look and feel consistent without hiding the things that they are particularly great at. So, every public cloud has its own set of capabilities, its own price-performance profile, its own service ecosystem, and richness around that.So, what can we do to make it so that as you're thinking about your journey from taking an existing system, one of those heritage systems, and thinking through the evolution of that system to meet your business requirements, to be able to evolve quickly, to be able to go through that digital transformation journey, and package it up and deliver the right tools at the right time in the right environment, so that we can walk the journey with our customers?Corey: Does this tie into Tanzu, or is that a different VMware initiative slash division? And my apologies on that one, just because it's difficult for me to wrap my head around where Tanzu starts and stops. If I'm being frank.Craig: So, [unintelligible 00:21:49] is the heart of Tanzu. So Tanzu, in a way, is a new branch, a new direction for VMware. It's about bringing this richness of capabilities to developers running in any cloud environment. It's an amalgamation of a lot of great technologies that people aren't even aware of that VMware has been building, or that VMware has gained through acquisition, certainly Heptio and the ability to bring Kubernetes to an enterprise organization is part of that. But we're also responsible for things like Spring.Spring is a critical anchor for Java developers. If you look at the Spring community, we participate in one and a half million new application starts a month. And you wouldn't necessarily associate VMware with that, but we're absolutely driving critical innovation in that space. Things, like full-stack observability, being able to not only deploy these container-packaged applications, but being able to actually deal with the day two operations, and how to deal with the APM considerations, et cetera. So, Tanzu is an all-in push from VMware to bring the technologies like Kubernetes and everything that exists above Kubernetes to our customers, but also to new customers in the public cloud that are really looking for consistency across those environments.Corey: When I look at what you've been doing for the past decade or so, it really tells a story of transitions, where you went from product lead on GCE, to working on Kubernetes. You took Kubernetes from an internal Google reimagining of Borg into an open-source project that has been given over to the CNCF. You went from running Heptio, which was a startup, to working at one of the least startup-y-like companies, by some measures, in the world.s you seem to have gone from transiting from one thing to almost its exact opposite, repeatedly, throughout your career. What's up with that theme?Craig: I think if you look back on the transitions and those key steps, the one thing that I've consistently held in my head, and I think my personal motivation was really grounded in this view that IT is too hard, right? IT is just too challenging. So, the transition from Microsoft, where I was responsible building package software, to Google, which was about cloud, was really marking that transition of, “Hey, we just need to do better for the enterprise organization.” The transition from focusing on a virtual machine-based system, which was the state of the art at the time to unlocking these modern orchestrated container-based system was in service of that need, which was, “Hey, you know, if you can start to just treat a number of virtual machines as a destination that has a distributed operating system on top of it, we're going to be better off.” The need to transition to a community-centric outcome because while Google is amazing in so many ways, being able to benefit from the perspective that traditional enterprise organizations brought to the table was significant to transitioning into a startup where we were really serving enterprise organizations and providing that interface back into the community to ultimately joining VMware because at the end of the day, there's a lot of work to be done here.And when you're selling a startup, it's—you're either selling out or you're buying in, and I'm not big on the idea of selling out. In this case, having access to the breadth of VMware, having access to the place where most of the customers are really cared about were living, and all of those heritage systems that are just running the world's business. So, for me, it's really been about walking that journey on behalf of that individual that's just trying to make ends meet; just trying to make sure that their IT systems stay lit; that are trying to make sure that the debt that they're creating today in the IT environment isn't payday loan debt, it's more like a mortgage. I can get into an environment that's going to serve me and my family well. And so, each of those transitions has really just been marked by need.And I tend to look at the needs of that enterprise organization that's walking this journey as being an anchor for me. And I'm pleased with every transition I've made. Like, at every point we've—sort of, Joe and myself, who's been on this journey for a while, have been able to better serve that individual.Corey: Now, I know that it's always challenging to talk about the future, but do you think you're done with those radical transitions, as you continue to look forward to what's coming? I mean, it's impossible to predict the future, but you're clearly where you are for a reason, and I'm assuming part of that reason is because you see an opportunity; you see a transformation that is currently unfolding. What does that look like from where you sit?Craig: Well, I mean, my work in VMware [laugh] is very far from done. There's just an amazing amount of continued opportunity to deliver value not only to those existing customers where they're running on-prem but to make the public cloud more intrinsically accessible and to increasingly solve the problems as more computational resources fanning back out to the edge. So, I'm extremely excited about the opportunity ahead of us from the VMware perspective. I think we have some incredible advantages because, at the end of the day, we're both a neutral party—you know, we're not a hyperscaler. We're not here to compete with the hyperscalers on the economies of scale that they render.But we're also working to make sure that as the hyperscalers are offering up these new services and everything else, that we can help the enterprise organization make best use of that. We can help them make best use of that infrastructure environment, we can help them navigate the complexities of things like concentration risk, or being able to manage through the luck and potential that some of these things represent. So, I don't want to see the world collapse back into the mainframe era. I think that's the thing that really motivates me, I think, the transition from mainframe to client-server, the work that Wintel did—the Windows-Intel consortium—to unlock that ecosystem just created massive efficiencies and massive benefits from everyone. And I do feel like with the combination of technologies like Kubernetes and everything that's happening on top of that, and the opportunity that an organization like VMware has to be a neutral party, to really bridge the gap between enterprises and those technologies, we're in a situation where we can create just tremendous value in the world: making it so that modernization is a journey rather than a destination, helping customers modernize at a pace that's reasonable to them, and ultimately serving both the cloud providers in terms of bringing some critical workloads to the cloud, but also serving customers so that as they live with the harsh realities of a multi-cloud universe where I don't know one enterprise organization that's just all-in on one cloud, we can provide some really useful capabilities and technologies to make them feel more consistent, more familiar, without hiding what's great about each of them.Corey: Craig, thank you so much for taking the time to speak with me today about where you sit, how you see the world, where you've been, and little bits of where we're going. If people want to learn more, where can they find you?Craig: Well, I'm on Twitter, @cmcluck, and obviously, on LinkedIn. And we'll continue to invite folks to attend a lot of our events, whether that's the Spring conferences that VMware sponsors, or VMWorld. And I'm really excited to have an opportunity to talk more about what we're doing and some of the great things we're up to.Corey: I will certainly be following up as the year continues to unfold. Thanks so much for your time. I really appreciate it.Craig: Thank you so much for your time as well.Corey: Craig McLuckie, Vice President of R&D at VMware in their modern applications business unit. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with a comment that I won't bother to read before designating it legacy or heritage.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need the Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

About NathenNathen Harvey, Cloud Developer Advocate at Google, helps the community understand and apply DevOps and SRE practices in the cloud.  Nathen formerly led the Chef community, co-hosted the Food Fight Show, and managed operations and infrastructure for a diverse range of web applications. Links: cloud.google.com/devops: https://cloud.google.com/devops 97 Things every Cloud Engineer Should Know: https://shop.aer.io/oreilly/p/97-things-every/9781492076735-9149 Twitter: https://twitter.com/nathenharvey TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by Thinkst. This is going to take a minute to explain, so bear with me. I linked against an early version of their tool, canarytokens.org in the very early days of my newsletter, and what it does is relatively simple and straightforward. It winds up embedding credentials, files, that sort of thing in various parts of your environment, wherever you want to; it gives you fake AWS API credentials, for example. And the only thing that these things do is alert you whenever someone attempts to use those things. It’s an awesome approach. I’ve used something similar for years. Check them out. But wait, there’s more. They also have an enterprise option that you should be very much aware of canary.tools. You can take a look at this, but what it does is it provides an enterprise approach to drive these things throughout your entire environment. You can get a physical device that hangs out on your network and impersonates whatever you want to. When it gets Nmap scanned, or someone attempts to log into it, or access files on it, you get instant alerts. It’s awesome. If you don’t do something like this, you’re likely to find out that you’ve gotten breached, the hard way. Take a look at this. It’s one of those few things that I look at and say, “Wow, that is an amazing idea. I love it.” That’s canarytokens.org and canary.tools. The first one is free. The second one is enterprise-y. Take a look. I’m a big fan of this. More from them in the coming weeks.Corey: This episode is sponsored in part by our friends at Lumigo. If you've built anything from serverless, you know that if there's one thing that can be said universally about these applications, it's that it turns every outage into a murder mystery. Lumigo helps make sense of all of the various functions that wind up tying together to build applications.It offers one-click distributed tracing so you can effortlessly find and fix issues in your serverless and microservices environment. You've created more problems for yourself. Make one of them go away. To learn more, visit lumigo.io.Corey: Welcome to Screaming in the Cloud. I’m Corey Quinn. I’m joined this week by Nathen Harvey, a cloud developer advocate at a small startup called Google. Nathen, thank you for joining me.Nathen: Hey, Corey. It’s really great to be here.Corey: We’ll get to the Google bits in a little, but first, I want to start back in the beginning with your origin story. It turns out, for example, that you were at a lot of places, and the first thing going through your history that I really recognized was way back at the end of 2009, where you were the web operations manager at Custom Ink. They’re a t-shirt company—and other apparel—that I’ve been using for three years now for the charity t-shirt drive here, as well as other sundry things. Longtime listeners of the show might remember we had Ken Collins on to talk about Ruby in Lambda and other horrifying things, before it was cool.Nathen: Yes, indeed, I was at Custom Ink. And, you know, you talk about them being a t-shirt company, and I don’t know… maybe I’m still a shill for Custom Ink, but I really look at them as an experience company. And you’ve recognized that yourself. They produce and help people, really encourage that group and experiences, and really drive what does it mean to connect with other humans, and how can you do that through custom apparel? To me, that’s what Custom Ink has always been about. They’re not selling t-shirts; they are selling an experience.Corey: In my case, I view them as a t-shirt company because, let’s be fair here, I wind up doing charity t-shirt drives, and they’ve always been extremely supportive of—well, there’s really no other way to put this—my ridiculous nonsense. The year I had linked campaigns of the ‘AMI has three syllables’ shirt that was on sale, and then for the Amazonians, ‘ah-mi’ is how it’s pronounced instead and that one was $10 more because there’s a price to being wrong. And all proceeds, of course, went to benefit the charity of the year. And that was a fun thing. And I talked to a number of other folks on this, and they look at me very strangely, and Custom Ink didn’t even blink.Nathen: Right, right. Absolutely. Absolutely.Corey: And yes, they said lots of other apparel, but for whatever reason, it seems that sending out complicated multiple options of things that need each hit minimum order quantities to print during a fundraiser, and the fact that I don’t have to deal with the money because they just wind up sending it over directly. It’s just easier. It’s one of those things where back when I was a single person who was doing this stuff, I didn’t have to worry about it. Now that I’ve grown and my needs have multiplied, I still like doing business with them. Great folks.Nathen: Absolutely. And that’s exactly what I mean by—like, they’ve sold you on that experience. That’s why you continue to do business with them. It’s not just because of the t-shirts. It’s the whole package that goes along with it.Corey: And then in 2012, the world didn’t end. But yours kind of did because you stopped working at Custom Ink and went to another company called Chef. You were there for a little over six years. You started off as a community director and then became the VP of Community Development. And I think you did an amazing job, but first tell me about that, then I will give my hot take.Nathen: All right, great. I’m always up for the hot takes. So listen, Chef was an amazing community of people. Oh, it was also a company. And so I really fell in love with—while I was at Custom Ink, actually, we were using Chef, and I fell in love with the community.And I was doing a lot of community support, running my own podcast, or participating with some co-hosts on a podcast called the Food Fight Show back in the day—it was all about Chef—running meetups and so forth. And at one point I decided, you know, what I should do maybe is stop being on call and start supporting this community full time. And that’s exactly what I did. I went to Chef and yes, as you mentioned, spent just over six years there, or just about six years there, and it was really, really an incredible time. Lots of hugs to be given, and just a great community in the DevOps space.Corey: I took a somewhat, I guess, agreeing or disagreeing position. I was on the Puppet side of the configuration management debate, and it was challenging. And then, ah, I was one of the very early developers behind SaltStack because clearly, the problem with all of these things was that no one had written it correctly, and we were going to fix that. And it turns out no, no, the problem was customers the whole time. But that’s a separate debate.So, I was never in the Chef ecosystem. That was the one system I never really touched in anger. And it’s easy to turn this into a, “Oh, you folks were the competition,” despite the fact I’ve never actually worked directly for either of those companies. But it was never like that because our real enemies were people configuring things by hand, for one because that’s unnecessary toil; don’t do it, and it was also just such an uplifting sense of community. Some of the best people I knew were in the Chef ecosystem, in the Chef orbit.For a while, they’re, on some level—and this is something I’d love to get your thoughts on—it seems that a failure mode that Chef exhibited was hiring directly from its community, where if someone was a big fan of Chef, start a stopwatch, they’re going to be working there before the month is out.Nathen: I think that Chef, the company definitely pulled a lot of community members into the organization. And frankly, when the company started, that was really, really great because it was an early startup. And as the company grew, it was still wonderful, of course, to pull in people from the community to really help drive the future direction, how our customers are using it. But like you said, there is a little bit of a challenge or concern when you start pulling too many of your most vocal supporters out of the community and putting them into the company, sometimes in places or roles where they didn’t have the opportunity to be as vocal, as big a champions for the product, for the services.Corey: I think at some level, it was—again, it helps to have people who are passionate about the product working there, but on the other, it felt like over time, it wound up weakening the community in some respects, just because everyone who worked there eventually found themselves in a scenario of well, I work here, it’s what we do, and now I have to say nice things. It winds up weakening the grassroot story.Nathen: Mmm. There’s definitely some truth to that, but I think there’s also some truths to just the evolution of community as you went from a community in the early days where there were a lot of contributors to over time—gratefully so—the community that—or sort of the proportion of the community that were consumers of Chef versus contributors to Chef, that balance changed. And so you had a lot more customers using the product. So, I don’t disagree with you, but I do think that it’s part of the natural evolution of community as well.Corey: And all things must end. And of course, Chef got acquired, I believe, after you left. So, I mean, at that point, you left, they were rudderless and what else were they to do? And you went to Google. And that is always an interesting story because Google’s community interaction before the time you wound up there, and after—I don’t know that you were necessarily the proximate cause, but I’m going to hang that around your neck because it’s all been a positive change since then—look radically different.Nathen: Yeah. Well, thank you. It is definitely not something that I should wear or carry alone, but going to Google was an interesting choice for me and I recognize that. And, you know, honestly, Corey, one of the things that drove me to Google was a good friend of mine, Seth Vargo. And just to kind of tie the complete throughline here, Seth and I worked together at Custom Ink, we’ve worked together at Chef, he left Chef and went to Hashi, and then went to Google. And the day after I knew that he was going to Google, I called him up and I said, “Seth, come on. Google’s so big. Why? Why? And how? I don’t understand. I don’t understand the move.”Corey: I asked him many of the same questions back in episode three of this show. He was a very early guest when I was scared speechless having conversations. It’s improved since then, a couple hundred in. But yeah, very friendly; very open; very warm.Nathen: Yeah. And, you know—Corey: “Why are you at Google?” was sort of the next follow-on question there in that era.Nathen: [laugh]. Yes, indeed. And I do think that Google, and specifically Google Cloud, has really taken to heart this idea that there’s a lot that we can learn from each other. And I don’t mean from each other within Google. Although, of course, we can learn a lot from each other.But we can also learn a lot from our community, from our customer base. How are they using Google Cloud? How are they using technology to drive their business forward? These are all things that we can learn. It turns out, not every company has Google, and that’s a good thing.Not every company should be Google or Google-sized, and certainly don’t have Google customers. And I think that it’s really important that we recognize that when we work with a customer, they’re the experts in their customers, and in their systems, and so forth.Corey: A lot has changed with Google’s approach to, well, basically everything. It turns out that when you’re a company that is, what, 26 years old now—27, something like that—starting with humble beginnings and then becoming a trillion-dollar entity, things change. Culture change, your community changes, what you do changes, and that becomes something that I think is not necessarily fully appreciated or fully understood in some corners. But then 2018 hit. You went to Google; what did you do then?Because it is such a large company that it is very difficult to know what any individual is up to there, and the primary means that I engage in the DevRel community space—specifically via aggressively shitposting on Twitter—isn’t really your means of interacting with the community. So, from that particular point of view, it’s, “Oh, yeah, he went to Google, and no one ever heard from him again.” What is it you say it is you do there?Nathen: Yeah. So, for sure. What I do here as a cloud advocate, is I really focus in on kind of two areas, I would say: DevOps—and I recognize that is a terrible, terrible word because when I say it, we all think of different things, but I definitely focus on the DevOps—and then SRE practices as well, or Site Reliability Engineering. And specifically what I work on is how do we bring the principles and practices of DevOps, of SRE, into our customer base and into the community at large? How do we drive what is the state of the art?How do we approach these particular topics? And so that’s really what I’ve been focused on since joining Google. Well, frankly, I was focused on that while at Chef, as well, maybe without the SRE bend so much, but certainly at Google SRE comes in, but it’s always—for the past decade for me—been about DevOps and how do we use technology to align the humans and work towards the business outcomes that we’re driving for?Corey: And business outcomes become an interesting story in the world of cloud because it distills down, for a cloud service provider is, we would like people to use our cloud, more of it, in perpetuity. It is not a complicated business model—if I can be direct—because business models inherently are not. “Whatever it is your company does, we would like you to do it here.” And that turns into a bunch of differentiated services across the spectrum, in some cases hilariously so, when it turns into basically pick an English word, and there’s a 50/50 shot that’s part of a service name somewhere. But a lot of it distills down to baseline distinct primitives.You’re talking about the DevOps aspect of it, which is—we talk about, is it culture? Is it tools? No, it’s a means to sell conferences, and books, and things like that. But what is it in the context of a cloud service provider? Specifically, Google because let’s be clear here, DevOps apparently for other providers is Azure DevOps. That’s right. It’s a service name, and DevOps Guru on the AWS side because everything is terrible.Nathen: Absolutely. Look, I think that I used to snark that the only DevOps tool was the manager of DevOps. But the truth is that DevOps is… it is tooling, and it is culture, and to separate the two is really a fool’s errand. I think that your tooling amplifies your culture, your culture amplifies your tooling. Together, this is how we make progress.Now, when it comes to Google, what do we mean when we say DevOps? Well, one of the good things is, shortly after I joined Google Cloud, Google Cloud acquired DORA, the DevOps Research and Assessment Organization.Corey: Jez Humble, and Dr. Nicole Forsgren. And then, for all intents and purposes, they googled it. Relatively shortly thereafter, by which I mean, we never really heard from DORA again. In 2020, the “State of DevOps Report” didn’t exist, which was what they were famous for doing. And it was, “Oh, yep. That’s a Google acquisition all right.” Is that what happened? Did I miss some nuance there?Nathen: Yeah. Let’s talk about that. So first, you’re right, it was Dr. Nicole Forsgren, who founded DORA. So, when the acquisition happened, she came along to Google Cloud, Jez Humble came along through that acquisition as well. And frankly, what happened in 2020? Well, Corey, I don’t know if you noticed, but there was a lot happening in 2020, much of it not very good. I think when we look at the global scale, like, 2020 was not a great year for us—Corey: It was a rebuilding year.Nathen: Oh, all right, fair enough. Fair enough. [laugh]. A rebuilding year. But so here’s what happened with DORA, quite frankly. We—Google Cloud—continue to invest in that research program. And really, in a sense, 2020 was a rebuilding year, in that our focus was really about how do we help our customers and our community apply the lessons of DORA?And so one of the things that we’ve done is we’ve released much of the research under cloud.google.com/devops, including right there, a DevOps quick check where, as a team you can go in and, using the metrics and the research program from DORA, you can assess, are you a low, medium, high, or elite performer?And then beyond that assessment, actually use the research to help you identify which capabilities should my team invest in improving. So, those capabilities might be technical capabilities, things like continuous integration; it might be process or measurement capabilities, or in fact, cultural capabilities. So, all of these capabilities come together to help you improve your overall software delivery and operations performance. And so in 2020, the big thing that we did was release and continue to update this Quick Check, release the research, make it fully available. We’ve also spent some time internally on the program that, you know, is not super interesting to talk about on the podcast.But the other thing that we did in 2020 with the DORA research program was update the ROI research, the return on investment research. This is something that maybe your listeners don’t care about, but their managers might care about, their CIOs, CTOs, CFOs might care about. How do we get money back on this transformation thing? And the research paper really digs into exactly that. How do we measure that? What returns can we expect? And so forth. So, that was released in 2020.Corey: I have a whole bunch of angry thoughts about a lot of takes in that space, but this is neither the time nor the place for me to begin ranting incoherently for an hour and a half. But yeah, I get that it was a year that was off, and now you’re doing it again, apparently, in 2021. And the one thing I never really saw historically because I don’t know if I’m playing in the wrong environments, or I’m certainly not the target [laugh] audience now, if I ever really was, but most years, I missed the release of the survey of where people can go to fill in these questions. I would be interested to know where that is now. And then I would be interested to know, how have you been socializing in that in the past? In other words, where are you finding these people?Nathen: Yeah, for sure. So, the place you go to find the survey right now is cloud.google.com/devops, you’ll find a button on the page that says something like, “Take the survey,” or, “Take the 2021 survey.”And what we’ve done in the past, and really what DORA has done in the past is use a number of different ways to get out information about the survey, when the survey is open, and so forth. Primarily Twitter, but also we have partners, and DORA historically has used partners as well to help share that the survey itself is open. So, I would absolutely recommend that you go and check out the survey because I’ll tell you what, one of the things that’s really interesting, Corey, over the years, I’ve talked to a bunch of people that have taken the survey, and that have read the State of DevOps Report that comes out each year, and some of the consistent feedback I’ve heard from folks is that simply taking the survey and considering the questions that are asked as part of the survey gives great insight immediately into how their team can improve. What things, what capabilities are they lacking? Or what capabilities are they doing really well with and they don’t need to make investments on? They can immediately see that just by answering and carefully considering the questions that are part of the survey.Corey: This episode is sponsored by ExtraHop. ExtraHop provides threat detection and response for the Enterprise (not the starship). On-prem security doesn’t translate well to cloud or multi-cloud environments, and that’s not even counting IoT. ExtraHop automatically discovers everything inside the perimeter, including your cloud workloads and IoT devices, detects these threats up to 35 percent faster, and helps you act immediately. Ask for a free trial of detection and response for AWS today at extrahop.com/trial.Corey: Very often, in some cases looking at things like maturity models and the like, the actual report is less valuable than the exercise of filling it out and going through the process. I mean, compliance reports, audit framework, et cetera, often lead to the same outcomes. The question is, are you taking it seriously, or are you one of those folks who is filling out a survey because do this and you’ll be entered to win a $25 gift card somewhere? Probably Blockbuster because it no longer exists. I get those in my email constantly of, “Yeah, give half an hour of your time in return for some paltry chance to win something.”No, I have a job to do. And I worry if at that level of that approach, who are you actually getting that’s going to sit down and fill this thing out? That said, the State of DevOps Reports have been for a long time, sort of the gold standard in this space and I would encourage people listening to this to absolutely take the time to fill that out. cloud.google.com/devops.I’m looking forward to seeing what comes out of it. And I love it because of the casual shade you can use to throw at other companies, too. Like, “Are you an elite team?” With the implicit baked-in sentiment being, no, you’re not, but I want to hear you say it.Nathen: Yeah, one of the things that really sets DORA apart, also, I think, is just the—well, two of the things I guess I would say. One is the length of time that the research program has been running. It’s going on seven years now that this research program has been running, and so given that, you have tens of thousands of IT professionals that have taken the survey and provided insights into sort of what’s the state of our industry today, and where are we heading, but it’s also an academically rigorous survey. The survey and the research itself has always and continues to be completely platform and program agnostic. This is not a survey about Google Cloud.This is not a survey where we’re trying to help understand exactly what products on Google Cloud should you use in order to be an elite performer. No. That’s not what this is about. It is about, truly, capabilities that your team needs in order to improve their software delivery and operations performance. And I think that’s really, really important.Dr. Nicole Forsgren who founded DORA, she didn’t come up with all of these ideas: “Hey, I think that you get better by doing this.” No. Instead, she researched all of these ideas. She got this input from across organizations of all sizes, organizations in every industry, and that, I think, really sets it apart.And our ability to really stay committed to that academic rigor, and the platform-agnostic approach to capturing and investigating these capabilities, I think is so important to this research. And again, this is why you should participate in the survey because you truly are going to help us move the state of the art of our industry.Corey: No, historically, there’s been a challenge where the mantle of thought leadership in conjunction with Google have intersected because there’s a common trope—historical—and I think that it is no longer accurately true. It’s an easy cheap shot, but I don’t think it holds water like it once did. Where, “Oh, Googler. It’s another word for condescending.” And there is an element of “Oh, this is how DevOps should be; this is how we’re moving things forward.” How do you distance it from being Google says you should do it like this?Nathen: Yeah. This comes up a lot. And frankly, I get in conversations with customers asking, “How does Google do this? How does Google do that?” And my answer always is, “You know, I can tell you how Google does something, and that might be interesting, but the fact is, it’s not much more than that, much more than interesting. Because what really matters is how are you going to do this? How are you going to improve your outcomes, whether that’s you’re delivering faster, you’re delivering more reliable, you’re running more reliable services? You’re the experts. As I mentioned earlier, you’re the experts in your teams, in your technology, and your customers. So, I’m here to learn right along with you. How are you going to do this? How are you going to improve?” Knowing how Google does it, eh, it’s interesting, but it’s not the path that you will follow.Corey: I think that’s one of those statements that can’t ever be outright stated on a marketing website, somewhere; it’s one of those shifts that you have to live. And I think that Google’s done a pretty decent job of that. The condescending Googler jokes are dated at this point, and it’s not because there was ever an ad campaign about, we’re not condescending anymore. It was a very subtle shift in the way that Google spoke to its customers, spoke about themselves. I no longer feel the need to stand up in a blinding white rage in the Q&A portion of conference talks given by Google employees.A lot has changed, and it’s not one thing that I can point to, it’s a bunch of different things that all add up to dramatically shifted credibility models. Realistically, I feel like that is a microcosm of a DevOps transformation. It’s not a tool; it’s not a single person being hired; it’s not, we’re taking an agile class for three days for all of engineering, and now things will be better. It’s a whole bunch of sustained work with a whole bunch of thought, and effort put into making it an actual transformation, which is such a toxically overloaded term, I dare not use it.Nathen: Indeed. And there’s no maturity model that shows, are you there yet? And it is something that you don’t flip on or flip off like a switch, right? It doesn’t happen overnight. It takes iteration and iterative change across the entire organization.And just like every change that you have across an organization, there are places where it’s going better than other places. And how do you learn from that? I think that’s really, really important. And to recognize and to bring some of that humility to the table is so important.Corey: So, what’s interesting about folks that I talk to on this show—well, there are many interesting things, but one of the interesting things is, is that they have a higher rate than the general population of having at one point in their careers, written a book of some form, and you are, of course, no exception. You and Emily Freeman co-authored recently, a book entitled 97 Things every Cloud Engineer Should Know. And it’s interesting because it only has one nine in the title. Okay, that is at least an attempt at being available. I know it’s available wherever most books are sold. Tell me more.Nathen: Yeah, so first, let’s start with the 97. Why 97? Corey, I don’t know if this or not, but 100% is the wrong reliability target for just about everything. So, 97. That feels achievable.Corey: It also feels like three people said they would do it and then backed out at the last minute, but that’s my cynicism speaking.Nathen: Well, for better or worse, O’Reilly. Has a whole 97 Things series and this is part of it. So, it is, in fact, 97 things. The other thing that I think is really important about the book: you mentioned that Emily and I wrote it, and the beauty is, for a long time, I’ve wanted to have written a book, and I have never wanted to be writing a book.Corey: That is what every author has ever said. It’s, no one wants to write a book; they want to have written one. And then you get a couple of beers into people and ask them, “So, I’m debating writing a book. Should I?” The response is, “No. Absolutely not. No.” And at some point, when you calmed them down again, and they stop screaming, they tell you the horrifying stories, and you realize, “Oh, wow, I really never want to write a book.”Nathen: [laugh]. Yes. Well, the beauty of 97 Things and this book in particular, or the whole series, really, is its subtitle is Collective Wisdom from the Experts. So, in fact, we had over 80 different contributors sharing things that other cloud engineers should know. And I think this is also really, really important because having 80-plus contributors to this book gave us, not 97 things that Emily and Nathen think every cloud engineer should know, but instead, a wide variety of experience levels, a wide variety of perspectives, and so I think that is the thing that makes the book really powerful.It also means that those 80-some folks that contributed to the book, had to write a very short article. So, of course, with 80 authors and 97 Things, the book is not—it doesn’t weigh 27 pounds, right? It’s less than 300 pages long, where you get these 97 tidbits. But really, the hope and the intent behind the book is to give you an idea about what should you explore deeper and, just as importantly, who are some people that you can, maybe, reach out to and talk to about a particular topic, a particular thing that a cloud engineer should know. Here are 80 people that are here, helping you and really cheering you on as you take this journey into cloud engineering.Corey: I think there’s something to be said from having the stories for this is what we do, this is how we do it. But the lessons learned stories, those are the great ones, and it’s harder to get people on stage to talk about that without turning into, “And that’s how we snagged victory from the jaws of defeat.” No one ever gets on stage and says and that’s why the entire project was a boondoggle and four years later, we’re still struggling to recover. Especially, you know, publicly traded companies tend not to say those things. But it’s true.You wind up with people getting on stage and talking instead about these high-level amazing things that they’ve done in the project went flawlessly, and you turn to the person next to you and say, “Yeah, I wish I could work in a place like that.” And they say, “Yeah, me too.” And you check, and they work at the same place as the presenter. Because it’s conference-ware; it’s never a real story. I’m hoping that these stories go a bit more in-depth into the nitty-gritty of what worked, what didn’t work, and it’s not always ‘author as hero protagonist.’Nathen: Oh, you will definitely find that in this book. These are true stories. These are stories of pain, of heartache, of victory and success, and learnings along the way. Absolutely. And frankly, in the DevOps space, we do an okay job of talking openly about our failures.We often talk about things that we tried that went wrong, or epic failures in our systems, and then how we recovered from them. And yes, oftentimes, those stories have a great sort of storybook ending to them, but there’s a lot of truth in a lot of those stories as well because we all know that no organization is uniformly good at everything. That may be the stories that they want to share most, but, you know, there’s some truth in those stories that hopefully we can find. And certainly, in this book, you will find the good, the bad, the ugly, the learnings, and all of the lessons there.Corey: Where can people find it if they want to buy it?Nathen: Oh, you know, you can find it wherever you buy books. There are of course, ebooks, O’Reilly’s website, you know with the—Corey: Wherever fine books are pirated. Yes, yes.Nathen: That’s a good place to go for books, yeah. For sure.Corey: And we will, of course, throw a link to the book in the [show notes 00:29:12]. Thank you so much for taking the time to speak with me. If people want to learn more about the rest of what you’re up to, how you’re thinking about it, what wise wisdom you have for the rest of us, okay can they find you, other than the book?Nathen: Yeah, a great place to reach out to me is on Twitter. I am at @nathenharvey. But I should warn you, my father misspelled my name. So, it’s N-A-T-H-E-N-H-A-R-V-E-Y. So, you can find me on Twitter; reach out to me there.Corey: And we will of course include links to all of that in the [show notes 00:29:43] as well. Thank you so much for speaking to me today. I really appreciate it.Nathen: Thank you, Corey. It’s been a pleasure.Corey: Nathen Harvey, cloud developer advocate at Google. I’m Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you’ve enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you’ve hated this podcast, please leave a five-star review on your podcast platform of choice, along with an angry comment telling me that I’m completely wrong. You can instantly get DevOps in your environment if I only purchase whatever crap it is your company sells.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.

About SanjaySanjay Poonen is the former COO of VMware, where he was responsible for worldwide sales, services, support, marketing and alliances. He was also responsible for the Security strategy and business at VMware. Prior to SAP, Poonen held executive roles at SAP, Symantec, VERITAS and Informatica, and he began his career as a software engineer at Microsoft, followed by Apple. Poonen holds two patents as well as an MBA from Harvard Business School, where he graduated a Baker Scholar; a master's degree in management science and engineering from Stanford University; and a bachelor's degree in computer science, math and engineering from Dartmouth College, where he graduated summa cum laude and Phi Beta Kappa.Links: VMware: https://www.vmware.com/ leadership values: https://www.youtube.com/watch?v=lxkysDMBM0Q Twitter: https://twitter.com/spoonen LinkedIn: https://www.linkedin.com/in/sanjaypoonen/ spoonen@vmware.com: mailto:spoonen@vmware.com TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by Thinkst. This is going to take a minute to explain, so bear with me. I linked against an early version of their tool, canarytokens.org in the very early days of my newsletter, and what it does is relatively simple and straightforward. It winds up embedding credentials, files, that sort of thing in various parts of your environment, wherever you want to; it gives you fake AWS API credentials, for example. And the only thing that these things do is alert you whenever someone attempts to use those things. It’s an awesome approach. I’ve used something similar for years. Check them out. But wait, there’s more. They also have an enterprise option that you should be very much aware of canary.tools. You can take a look at this, but what it does is it provides an enterprise approach to drive these things throughout your entire environment. You can get a physical device that hangs out on your network and impersonates whatever you want to. When it gets Nmap scanned, or someone attempts to log into it, or access files on it, you get instant alerts. It’s awesome. If you don’t do something like this, you’re likely to find out that you’ve gotten breached, the hard way. Take a look at this. It’s one of those few things that I look at and say, “Wow, that is an amazing idea. I love it.” That’s canarytokens.org and canary.tools. The first one is free. The second one is enterprise-y. Take a look. I’m a big fan of this. More from them in the coming weeks.Corey: Let’s be honest—the past year has been a nightmare for cloud financial management. The pandemic forced us to move workloads to the cloud sooner than anticipated, and we all know what that means—surprises on the cloud bill and headaches for anyone trying to figure out what caused them. The CloudLIVE 2021 virtual conference is your chance to connect with FinOps and cloud financial management practitioners and get a behind-the-scenes look into proven strategies that have helped organizations like yours adapt to the realities of the past year. Hosted by CloudHealth by VMware on May 20th, the CloudLIVE 2021 conference will be 100% virtual and 100% free to attend, so you have no excuses for missing out on this opportunity to connect with the cloud management community. Visit cloudlive.com/coreyto learn more and save your virtual seat today. That’s cloud-l-i-v-e.com/corey to register.Corey: Welcome to Screaming in the Cloud. I’m Corey Quinn. I talk a lot about cloud in a variety of different contexts; this show is about the business of cloud. But, fundamentally, where cloud comes from was this novel concept, once upon a time, of virtualization. And that gave rise to a whole bunch of other things that later became, then containers, now it becomes Kubernetes, and if you want to go down the serverless path, you can.But it’s hard to think of a company that has had more impact on virtualization and that narrative than VMware. My guest today is Sanjay Poonen, Chief Operating Officer of VMware. Thank you for joining me.Sanjay: Thanks, Corey Quinn, it’s great to be with you and with your audience on this show.Corey: So, let’s start with the fun slash difficult questions. It’s easy to look at VMware as a way of virtualizing existing bare-metal workloads and moving those VMs around, but in many respects, that is perceived by some—ehem, ehem—to be something of a legacy model of cloud interaction where it solves the problem of on-premises, which is I’m really bad at running data centers so I’m just going to treat the cloud like a data center. And for some companies and some workloads, where, great, that’s fine. But isn’t that, I guess, a V1 vision of cloud, and if it is, why is VMware relevant to that?Sanjay: Great question, Corey. And I think it’s great to be straight up on a topic [unintelligible 00:02:01]. Yeah, I think you’re right. Listen, the ‘V’ in VMware is virtualization. The ‘VM’ is virtual machines.A lot of what is the underpinning of what made the private cloud, as we call it today, but the data center of the past successful was this virtualization technology. In the old days, people would send us electricity bills, before and after VMware, and how much they’re saving. So, this energy-saving concept of virtualization has been profound in the modernization of the data center and the advent of what’s called the private cloud. But as you looked at the public cloud innovate, whether it was AWS or even the SaaS applications—I mean, listen, the most popular capability initially on AWS was EC2 and S3, and the core of EC2 is virtualization. I think what we had to do, as this happened, was the foundation was certainly those services like EC2 and S3, but very quickly, the building phenomenon that attracted hundreds of thousands and I think now probably a few million customers to AWS was the large number of services, probably now 150, 200-odd services, that were built on top of that for everything from data, to AI, to a variety of other things that every year Andy Jassy and the team would build up.So, we had to make sure that over the course of the last, I’d say, certainly the last five to maybe eight years, we were becoming relevant to our customers that were a mix. There were customers who were large—I mean, we have about half a million customers—and in many cases, they have about 80, 90% of their workloads running on-prem and they want to move those workloads to the cloud, but they can’t just refactor and re-platform all of those apps that are running in the on-premise world. When they will try to do it by the end of the year—they may have 1000 applications—they got 10 done.Corey: Oh, and it’s not realistic and it’s unfair. I mean, there’s the idea of, “Oh, that’s legacy,” which is condescending engineering speak for it actually makes money because it’s been around for longer than six months. And sure you can have Twitter For Pets roll stuff out every day that you want; when you’re a bank, you have different constraints forced upon you. And I’m very sympathetic to folks who are in scenarios where they aren’t, for whatever reason, able to technically, culturally, or for regulatory reasons, be able to do continuous deployment of everything. I want to be very clear that I’ve in no way passing judgment on an entire sector of enterprise.Sanjay: But while that sector is important, there was also another sector starting to emerge: the Airbnbs, the Pinterests, the modern companies who may not need VMware at all as they’re building native, but may need some of our container in a new open-source capabilities. SaltStack was one of them; we will talk about that, I’m sure. So, we needed to be relevant to both customer communities because the Airbnbs of today, will be the Marriotts of tomorrow. So, we had to really rethink what is the future of VMware, what’s our existence in a public cloud phenomenon? That’s really what led to a complete watershed moment.I called publicly in the past sort of a Berlin Wall moment where Amazon and VMware were positioned pretty much as competitors for a long period of time when AWS was first started. Not that Andy was going around talking negatively about VMware, but I think people view these as two separate doors, and never the twain would meet. But when we decided to partner with them—I then quite frankly, the precursor to that was us divesting our public cloud strategy. We’d tried to build a competitive public cloud called vCloud Air between the period of 2012 and 2015, 2016—we had to reach an end of that movement, and catharsis of that, divest that asset, and it opened the door for a strategic partnership. But now we can go back to those customers and help them move their applications in a way that’s highly efficient, almost like a house on wheels, and then once it’s in that location in AWS—or one of the other public clouds—you can modernize it, too.So, then you get to both get the best of both worlds: get it into the public cloud, maybe retire some of your data centers if that’s what you want to do, and then modernize it with all the beautiful services. And that’s the best of both worlds. Now, if you have 1000 applications, you’re moving hundreds of them into the public cloud, and then using all of the powerful developer services on that VMware stack that’s built on the bare metal of AWS. So, we started out with AWS, but very quickly then, all the other public clouds, maybe the five or six that are named in the Gartner Magic Quadrant, came to us and said, “Well, if you’re doing that with AWS, would you consider doing that with us, too?”Corey: There’s definitely been an evolution of VMware. I mean, it’s in the name; you have the term VM sitting there. It’s easy to, at least from where I sit, think of, “Oh, VMware, back when running virtual machines was novel.” And there was a lot of skepticism around the idea. I’m going to level with you; I was a skeptic around virtualization. Then around cloud. Then around containers.And now I’m trying—all right I’m going to be in favor of serverless, which is almost certain to doom it because everything else that I’ve been skeptical of in this sense beyond any reasonable measure. So, there is this idea that VMs are this sort of old-school thinking. And that’s great if you have an existing workload that needs to be migrated, but there are a finite number of those in the world. As we turn towards net-new and greenfield build-outs, a lot of things are a lot more cloud-native than just hosting a bunch of—if you take the AWS example—EC2 instances hanging out in the network talking to other EC2 instances. Taking advantage of native offerings definitely seems to be on the rise. And there have been acquisitions that VMware has made. You talk about SaltStack, which was a great example, given that I wrote part of that very early on, and I don’t think the internet’s ever forgiven me for it. But also Bitnami—or BittenAMI, as I insist on pronouncing it—and you also acquired Wavefront. There’s a lot of interesting stuff that feels almost like a setting up a dichotomy of new VMware versus old VMware. What are the points of commonality there? What is the vision for the next 15 years of the company?Sanjay: Yeah, I think when we think about it, it’s very important that, first off, we acknowledge that our roots are what gives us sustenance because we have a large customer base that uses us. We have 80 million workloads running on that VMware infrastructure, formerly ESX, now vSphere. And that’s our heritage, and those customers are happy. In fact, they’re not, like, fleeing like birds into there, so we want to care for those customers.But we have to have a north star, like a magnet that pulls us into the modern world. And that’s been—you know, I talked about phase one was this really charting of the future of VMware for the cloud. Just as important has been focused on cloud-native and containers the last three, four years. So, we acquired Heptio. As you know, Heptio was founded by some of the inventors of Kubernetes who left Google, Joe Beda, and Craig McLuckie.And with that came a strong I would say relevancy, and trust to the Kubernetes, we’ve become one of the leading contributors to open-source Kubernetes. And that brain trust now, some of whom are at VMWare and many are in the community think of us very differently. And then we’ve supplemented that with many other moves that are much more cloud-native. You mentioned two or three of them: Bitnami, for that sort of marketplace; and then SaltStack for what we have been able to do in configuration management and infrastructure automation; Wavefront for container-based workloads. And we’re not done, and we think, listen, there will be many, many more things that the first 10, 15 years of VMware was very much about optimizing the private cloud, the next 10, 15 years could be optimizing for that app modernization cloud-native world.And we think that customers will want something that can work in a multi-cloud fashion. Now, multi-cloud for us is certainly private cloud and edge cloud, which may have very little to do with hardware that’s in the public cloud, but also AWS, Azure, and two or three other clouds. And if you think of each of these public clouds as mini skyscrapers—so AWS has 50 billion in revenue; I’m going to guess Azure is, like, 30, and then Google is I don’t know 12, 13; and then everyone else, and they’re all skyscrapers are different—it’s like, if we can be that company that fills the crevices between them with cement that’s valuable so that people can then build their houses on top of that, you’re probably not going to be best served with a container Stack that’s trapped to just one cloud. And then over time, you don’t have reasonable amount of flexibility if you choose to change that direction. Now, some people might say, “Listen, multi-cloud is—who cares about that?”But I think increasingly, we’re hearing from customers a desire to have more than just one cloud for a variety of reasons. They want to have options, portability, flexibility, negotiating price, in addition to their private cloud. So, it’s a two plus one, sometimes it might be a two plus two, meaning it’s a private cloud and the edge cloud. And I think VMware is a tremendous proposition to be that Switzerland-type company that’s relevant in a private cloud, one or two public clouds, and an edge cloud environment, Corey.Corey: Are you seeing folks having individual workloads that they want to flow from one cloud to another in a seamless way, or is it more aligned along an approach of having workload A lives in this cloud and workload B lives in this cloud? And you’re in a terrific position to opine on that more than most, given who you are.Sanjay: Yeah. We’re not yet as yet seeing these floating workloads that start here and move around, that’s—usually you build an application with purpose. Like, it sits here in this cloud and of course. But we’re seeing, increasingly, interest at customers’ not tethering it to proprietary services only. I mean, certainly, if you’re going to optimize it for AWS, you’re going to take advantage of EC2, S3, and then many of the, kind of, very capable [unintelligible 00:11:24], Aurora, there are others that might be there.But over time, especially the open-source movement that brings out open-source data services, open-source tooling, containers, all of that stuff, give ultimately customers the hope that certainly they should add economic value and developer productivity value, but they should also create some potential portability so that if in the future you wanted to make a change, you’re not bound to that cloud platform. And a particular cloud may not like us saying this, but that’s just the fact of how CIOs today are starting to think much more so as they build these up and as many of the other public clouds start to climb in functionality. Now, there are other use cases where particular SaaS applications of SaaS services are optimized for a particular [unintelligible 00:12:07], for example, Office 365, someone’s using a collaboration app, typically, there’s choices of one or two, you’re either using a G Suite and then it’s tied to Google, or it’s Office 365. But even there, we’re starting to see some nibbling around the edges. Just the phenomenon of Zoom; that wasn’t a capability that Microsoft brought very—and the services from Google, or Amazon, or Microsoft was just not as good as Zoom.And Zoom just took off and has become the leading video collaboration platform because they’re just simple, easy to use, and delightful. It doesn’t matter what infrastructure they run on, whether it’s AWS, I mean, now they’re running some of their workloads on Oracle. Who cares? It’s a SaaS service. So, I think increasingly, I think there will be a propensity towards SaaS applications over custom building. If I can buy it why would I want to build a video collaboration app myself internally, if I can buy it as a SaaS service from Zoom, or whoever have you?Corey: Oh, building it yourself would be ludicrous unless that was one of your core competencies.Sanjay: Exactly.Corey: And Zoom seems to have that on lock.Sanjay: Right. And so similarly, to the extent that I think IT folks can buy applications that are more SaaS than custom-built, or even on-prem, I mean, Salesforce—the success of Salesforce, and Workday, and Adobe, and then, of course, the smaller ones like Zoom, and Slack, and so on. So, it’s clear evidence that the world is going to move towards SaaS applications. But where you have to custom build an application because it’s very unique to your business or to something you need to very snap quickly together, I think there’s going to be increasingly a propensity towards using open-source types of tooling, or open-source platforms—Kubernetes being the best example of that—that then have some multi-cloud characteristics.Corey: In a similar note, I know that the term is apparently, at least this week on Twitter, being argued against, but what about cloud repatriation? A lot of noise has been made about people moving workloads from public cloud back to private cloud. And the example they always give is Dropbox moving its centralized storage service into an on-prem environment, and the second example is basically a pile of tumbleweeds because people don’t really have anything concrete to point at. Does that align with your experience? Is there a, I guess, a hidden wave of people doing a reverse cloud migration that just doesn’t get discussed?Sanjay: I think there’s a couple of phenomenons, Corey, that we watch here. Now, clearly a company of the scale of Dropbox has economics on data and storage, and I’ve talked to Drew and a variety of the folks there, as well as Box, on how they think about this because at that scale, they probably could get some advantages that I’m sure they’ve thought through in both the engineering and the cost. I mean, there’s both engineering optimization and costs that I’m sure Drew and the folks there are thinking through. But there’s a couple of phenomena that we do—I mean, if you go back to, I think, maybe three or four quarters ago, Brian Moynihan, the CEO of Bank of America, I think in 2019, mid to late 2019 made a statement in his earnings call, he was asked, “How do you think about cloud?” And he said, “Listen, I can run a private cloud cheaper and better than any of the public clouds, and I save 240%,” if I remember the data right.Now, his private cloud and Bank of America is a key customer [unintelligible 00:15:04] of us, we find that some of the bigger companies at scale are able to either get hardware at really good pricing, are able to engineer—because they have hundreds of thousands—they’re almost mini VMware, right, [unintelligible 00:15:18] themselves because they’ve got so many engineers. They can do certain things that a company that doesn’t want to hire those many—companies, Pinterest, Airbnb may not do. So, there are customers who are going to basically say, even prior to repatriation, that the best opportunity is a private cloud. And in that place, we have to work with our private cloud partners, whether it’s Dell or others, to make sure that stack of hardware from them plus the software VMware in the containers on top of that is as competitive and is best cost of ownership, best ROI. Now, when you get to your second—your question around repatriation, what we have found in certain regions outside the US because of sovereign data, sovereign clouds, sometimes some distrust of some of those countries of the US public cloud, are they worried about them getting too big, fear by monopoly, all those types of things, lead certain countries outside the US to think about something that they would need that’s sovereign to their country.And the idea of sovereign data and sovereign clouds does lead those to then investing in local cloud providers. I mean, for example in France, there is a provider called OVH that’s kind of trying to do some of that. In China, there’s a whole bunch of them, obviously, Alibaba being the biggest. And I think that’s going to continue to be a phenomenon where there’s a [federated said 00:16:32], we have a cloud provider program with this 4000 cloud providers, Corey, who built their stack on VMware; we’ve got to feed them. Now, while they are an individual revenue way smaller than the public clouds were, but collectively, they represent a significant mass of where those countries want to run in a local cloud provider.And from our perspective, we spent years and years enabling that group to be successful. We don’t see any decline. In fact, that business for us has been growing. I would have thought that business would just completely decline with the hyperscalers. If anything, they’ve grown.So, there’s a little bit of the rising tide is helping all boats rise, so to speak. And the hyperscaler’s growth has also relied on many of these, sort of, sovereign clouds. So, there’s repatriation happening; I think those sovereign clouds will benefit some, and it could also be in some cases where customers will invest appropriately in private cloud. But I don’t see that—I think if anything, it’s going to be the public cloud growing, the private cloud, and edge cloud growing. And then some of these, sort of, country-specific sovereign clouds also growing. I don’t see this being in a huge threat to the public cloud phenomena that we’re in.Corey: This episode is sponsored in part by our friends at Lumigo. If you've built anything from serverless, you know that if there's one thing that can be said universally about these applications, it's that it turns every outage into a murder mystery. Lumigo helps make sense of all of the various functions that wind up tying together to build applications. It offers one-click distributed tracing so you can effortlessly find and fix issues in your serverless and microservices environment. You've created more problems for yourself. Make one of them go away. To learn more, visit lumigo.io. Corey: I want to very clear, I think that there’s a common misconception that there’s this, somehow, ongoing fight between all the cloud providers, and all this cloud growth, and all this revenue is coming at the expense of other cloud providers. I think that it is simultaneously workloads that are being migrated from on-premises environments—yes—but a lot of it also feels like it’s net-new. It’s not just about increasingly capturing ever larger portions of the market but rather about the market itself expanding geometrically. For a long time, it felt like that was what tech was doing. Looking at the global IT spend numbers coming out of Gartner and other places, it seems like it’s certainly not slowing down. Does that align with your perception of it? Or are there clear winners and losers that are I guess, differentiating out?Sanjay: I think, Corey, you’re right. I think if you just use some of the data, the entire IT market, let’s just say it’s about $1 trillion, some estimates have it higher than that. Let’s break it down a little bit. Inside that 1 trillion market it is growing—I mean, obviously COVID, and GDP declined last year in calendar 2020 did affect overall IT, but I think let’s assume that we have some kind of U-shape or other kind of recovery, going into the second half of certainly into next year; technology should lead GDP in terms of its incline. But inside that trillion-dollar market, if you add up the SaaS market, it’s about $115 billion market.And these are companies like Salesforce, and Adobe, and Workday, and ServiceNow. You add them all up, and those are growing, I think the numbers were in the order of 15 or 20% in aggregate. But that SaaS market is [unintelligible 00:19:08]. And that’s growing, certainly faster than the on-prem applications market, just evidenced by the growth of those companies relative to on-premise investments in SAP or Oracle. And then if you look at the infrastructure market, it’s slightly bigger, it’s about $125 billion, growing slightly faster—20, 25%—and there you have the companies like AWS, Azure, and Google, and Alibaba, and whoever have you. And certainly, that growth is faster than some of the on-premise growth, but it’s not like the on-premise folks are declining. They’re growing at slower paces.Corey: It is harder to leave an on-premise environment running and rack up charges and blow out the bill that way, but it—not impossible, I suppose, but it’s harder to do than it is in public cloud. But I definitely agree that the growth rate surpasses what you would see if it were just people turning things on and forgetting to turn them off all the time.Sanjay: Yeah, and I think that phenomenon is a shift in spending where certainly last year we saw more spending in the cloud than on-premise. I think the on-premise vendors have a tremendous opportunity in front of them, which is to optimize every last dollar that is going to be spent in the data centers, private cloud. And between us and our partners like Dell and others, we’ve got to make sure we do that for our customer base that we’ve accumulated over last 10, 15 years. But there’s also a significant investment now moving to the edge. When I look at retailers, CPG companies—consumer packaged good companies—manufacturers, the conversation that I’m having with their C-level tech or business executives is all about putting compute in the stores.I mean, listen, what is the retailer concerned about? Fraud, and some of those other things, and empowering a quick self-service experience for a consumer who comes in and wants to check out of a Safeway or Walmart really quickly. These are just simple applications with local compute in the store, and the more that we can make that possible on top of almost like a nano data center or micro data center, running in the store with those applications resident there, talking—you know, you can’t just take all of that data, go back and forth to the cloud, but with resident services and capability right there, that’s a beautiful opportunity for the VMware and the Dells of the world. And that’s going to be a significant place where I think you’re going to see expansion of their focus. The Edge market today is I think, projected to be about $6 or $8 billion this year, and growing to $25 billion the next four or five years.So, much smaller than the previous numbers I shared—you know, $125, $115 billion for SaaS and IaaS—but I think the opportunity there, especially these industries that are federated: CPG, consumer packaged goods, manufacturing, retail, and logistics, too—you know, FedEx made a big announcement with VMware and Dell a few months ago about how they’re thinking about putting compute and local infrastructure at their distribution sites. I think this phenomenon, Corey, is going to happen in a number of different [unintelligible 00:21:48], and is a tremendous opportunity. Certainly, the public cloud vendors are trying to do that with Outposts and Azure Stack, but I think it does favor the on-premise vendors also having a very strong proposition for the edge cloud.Corey: I assumed that the whole discussion with FedEx started by someone dramatically misunderstanding what it meant to ship code to production.Sanjay: [laugh]. I mean, listen, at the end of the day, all of these folks who are in traditional industries are trying to hire world-class developers—like software companies—because all of them are becoming software companies. And I think the open-source movement, and all of these ways in which you have a software supply chain that’s more modernized, it’s affecting every company. So, I think if you went into the engineering product teams of Rob Carter, who runs technology for FedEx, you’ll find them and they may not have all of the sophistication as a world-class software company, but they’re getting increasingly very much digital in their focus of next generation. And same thing with UPS.I was talking to the CEO of UPS, we had her come and speak at our kickoff. It’s amazing how much her lingo—she was the former CFO of Home Depot—I felt like I was talking to a software executive, and this is the CEO of UPS, a logistics company. So, I think increasingly, every company is becoming a software company at their core. And you don’t need to necessarily know all the details of containers and virtualization, but you need to understand how software and digital transformation, how technology can power your digital transformation.Corey: One thing that I’ve noticed the more I get to talk to people doing different things in different roles was, at first I was excited because I get to talk to the people where they’re really doing it right and everything’s awesome. And I’ve increasingly of the opinion that those sites don’t actually exist. Everyone talks about the great thing is that they’re doing and aspirationally in certain areas in the terms of conference-ware, but you get down into the weeds, and everyone views their environment as being a burning tire fire of sadness and regret. Everyone thinks other people are doing it way better than they are. And in some cases they’re embarrassed about it, in some cases they’re open about it, but I feel like we’re still in the early days where no one is doing things in the quote-unquote, “Right ways,” but everyone thinks everyone else is.Sanjay: Yeah, I think, Corey, that’s absolutely right. We are very much early days in all of this phenomenon. I mean, listen, even the public cloud, Andy himself would say it’s [laugh]—he wouldn’t say it’s quite day one, but he would say it’s very early [unintelligible 00:24:03], even though they’ve had 15 years of incredible success and a $50 billion business. I would agree. And when you look at the customers and their persona—when I ask a CIO what percentage of—of an established company, not one of the modern ones who are built all cloud-native—but what percentage of your workloads are in a public cloud versus private cloud, the vast majority is still in a data center or private cloud.But with the intent—if it’s 90/10, let’s say 90 private 10—for that to become 70/30, 50/50. But very rarely do I hear a one of these large companies say it’s going to be 10/90 the opposite way in three, five years. Now, listen, I think every company as it grows that is more modern. I mean the Zooms of the world, the Modernas, the Airbnbs, as they get bigger and bigger, they represent a completely new phenomenon of how they are building applications that are all cloud-native. And the beautiful thing for me is just as a former engineering and developer, I mean, I grew up writing code in C, and C++ and then came BEA WebLogic, and IBM WebSphere, and [JGUI 00:25:04].And I was so excited for these frameworks. I’m not writing code, thankfully, anymore because it would create lots of problems if I did. But when I watched the phenomena, I think to myself, “Man, if I was a 22 year old entering the workforce now, it’s one of the most exciting times to write code and be a developer because what’s available to you, both in the combination of these cloud frameworks and open-source frameworks, is immense.” To be able to innovate much, much faster than we did 25, 30 years ago when I was a developer.Corey: It’s amazing there’s the pace of innovation, if cloud has changed nothing else, from my perspective, it’s been the idea that you can provision things without these hefty waiting periods. But I want to shift gears slightly because we’ve been talking about cloud for a bit in the context of infrastructure, and containers, and the rest, but if we start moving up the stack a little bit, that’s also considered cloud, which just seems to have that naming problem of namespace collision, just to confuse folks. But VMware is also active in this space, too. You’ve got things like Workspace ONE, you’ve got a bunch of other endpoint options as well that are focused on the security space. Is that aligned?Is that just sort of a different business unit? How does that, I guess, resonate between the various things that you folks do? Because it turns out, you’re kind of a big company, and it’s difficult to keep it all straight from an external perspective.Sanjay: Well, I think—listen, we’re roughly a little less than $12 billion in revenue last year. You can think of us in two buckets: everything in the first bucket is all that we talked about. Think of that as modernization of applications and cloud infrastructure, or what people might think about PaaS and IaaS without the underlying hardware; we’re not trying to build servers and storage and networking at the hardware level, you know, and so and so. But the software layer is about, that’s the first conversation we had for the last 15, 20 minutes. The second part of our business is where we’re touching end-users and infrastructure, and securing it.And we think that’s an important part because that also is something through software, and the cloud could be optimized. And we’ve had a long-standing digital workspace. In fact, when I came to VMware, it was the first business I was running in terms of all the products and end-user computing. And our thesis was many of the current tools, whether it’s the virtual desktop technology that people have from existing vendors, or even today, the security tools that they use is just too cumbersome. It’s too heavy.In many cases, people complain about the number of agents they have on their laptops, or the way in which they secure firewalls is too expensive and too many. We felt we could radically—VMware gets involved in problems where we can radically simplify thing with some disruptive innovation. And the idea was, first in the digital workspace was to radically reduce cost with software that was built for the cloud. And Workspace ONE and all of those things radically reduce the need for disparate technologies for virtual desktops, identity management, and endpoint management. We’ve done very well in that.We’re a leader in that segment, if you look at any of the analysts ratings, whether it’s Gardner or others. But security has been a more recent phenomenon where we felt like it leads us very quickly into securing those laptops because on those same laptops, you have antivirus, you have a variety of tools, and on the average, the CSOs, the Chief Security Officers tell me they have way too many agents, way too many consoles, way too many alerts, and if we could reduce that and have a single agent on a laptop, or maybe even agentless technology that secure this, that’s the Nirvana. And if you look at some of the recent things that have happened with SolarWinds, or Petya, WannaCry in the past, security’s of top concern, Corey, to boards. And the more that we could do to clean that up, I think we can emerge—which we’re already starting to—as a cybersecurity layer. So, that’s a smaller part of our business, but, I mean, it’s multi-billion now, and we think it’s a tremendous opportunity for us to take what we’re doing in workspace and security and make that a growth vector.So, I think both of these core areas, the cloud infrastructure, and modern applications—topic number one—workspace and security—topic number two—I’m both tremendous opportunities for VMware in our journey to grow from a $12 billion company to one day, hopefully, a $20 billion company.Corey: Would that we all had such problems, on some level. It’s really interesting seeing the evolution of companies going from relatively small companies and humble beginnings to these giant—I guess, I want to use the term Colossus, but I’m not sure if that’s insulting or [laugh] not—it’s phenomenal just to see the different areas of business that VMware has expanded into. I mean, I’ve had other folks from your org talking about what a Tanzu is or might be, so we aren’t even going to go down that rabbit hole due to time constraints at this point, but one thing that I do want to get into, slightly, has been a recurring theme in the show, which is where does the next generation of leaders come from? Where do the next generation engineers come from? And you’ve been devoting a bit of time to this. I think I saw one of your YouTube videos somewhat recently about your leadership values. Talk to me a little bit about that.Sanjay: Yeah. Corey, listen, I’m glad that we’re closing out this on some of the soft topics because I love talking to you, or other talented analysts and thought leaders around technology. It’s my roots; I’m a technical person at heart. I love technology. But I think the soft stuff is often the hard stuff.And the hard stuff is often the soft stuff. And what I mean by that is, when all this peels away, what your lasting legacy to the company are the people you invest in, the character you build. And, I mean, as an immigrant who came to this country, when I was 18 years old, $50 in my pocket, I was very fortunate to have a scholarship to go to a really nice University, Dartmouth College, to study computer science. I mean, I grew up in India and if it wasn’t for the opportunity to come here on a scholarship, I wouldn’t have [been here 00:30:32]. So, everything I consider a blessing and a learning opportunity where I’m looking at the advent of life as a growth mindset: what can I learn? And we all need to cultivate more and more aspects of that growth mindset where we move from being know-it-alls to learn-it-alls.And one of the key things that I talk about—and all of your listeners on this, listening to this, I welcome to go to YouTube and search Sanjay Poonen and leadership, it’s a 10-minute video—I’ll pick one of them. Most often as we get higher and higher in an organization, leaders tend to view things as a pyramid, and they’re kind of like this chief bird sitting at the top of the pyramid, and all these birds that are looking—below them on branches are looking up and all they see is crap falling down. Literally. That’s what happens when you look at the bird up. And our job as leaders is to invert that pyramid.And to actually think about the person who is on the front lines. In a software company, it’s an engineer and a sales rep. They are the folks on the frontline: they’re writing code or selling code. They are the true people who are making things happen. And when we as leaders look at ourselves as the bottom of the pyramid—some people call that, “Servant leadership.”Whatever way you call it, the phrase isn’t the point—the point is, invert that pyramid and to take obstacles out of people from the frontline. You really become not interested as much around what your own personal wellbeing, it’s about ensuring that those people in the middle layers and certainly at the leaf levels of the organization are enormously successful. Their success becomes your joy, and it becomes almost like a parent, right? I mean, Corey, you have kids; I’ve got kids. Imagine if you were a parent and you were jealous of your kid’s success.I mean, I want my three children, my daughter, my two children to do better than me, running races or whatever it is that they do. And I think as a leader, the more that we celebrate the successes of our teams and people, and our lasting legacy is not our own success; it’s what we have left behind, other people. I’ve say often there’s no success without successors. So, that mindset takes a lot of work because the natural tendency of the human mind and the human behavior is to be selfish and think about ourselves. But yeah, it’s a natural phenomenon.We’re born that way, we live in act that way, but the more that we start to create that, then taking that not just to our team, but also to the community allows us to build a better society. And that’s something I’m deeply passionate about, try to do my small piece for it, and in fact, I’m sometimes more excited about these topics of leadership than even technology.Corey: It feels like it’s the stuff that lasts; it has staying power. I could record a video now about technology choices and how to work with those technologies and unless it’s about Git, it’s probably not going to be too relevant in 10 years. But leadership is one of those eternal things where it’s, once you’ve experienced a certain level of success, you can really see what people do with that the people that I like to surround myself with, generally make it a point to send the elevator back down, so to speak.Sanjay: I agree, Corey, it’s—glad that you do it. I’m always looking for people that I can learn from, and it doesn’t matter where they are in society. I mean, I think you often—I mean, this is classic Dale Carnegie; one of the books that my dad gave to me at a young age that I encourage everyone to read, How to Win Friends and Influence People, talked about how you can detect a person’s character based on the way they treat the receptionist, or their assistants, the people who might be lower down the totem pole from them. And most often you have people who kiss up and kick down. And I think when you build an organization that’s that typical.A lot of companies are built that way where they kiss up and kick down, you actually have an inverted sense of values. And I think you have to go back to some of those old-school ways that Dale Carnegie or Steven Covey talked about because you don’t have to build a culture that’s obnoxious; you can build a company that’s both nice and competitive. It doesn’t mean that anything we’ve talked about for the last few minutes means that I’m any less competitive and I don’t want to beat the competition and win a deal. What you can do it nicely. And even that’s something that I’ve had to grow in.So, I think when we all look at ourselves as sculptures, work in progress, and we’re perfecting our craft, so to speak, both on the technical front, and the product front and customer relationship, but then also on the leadership and the personal growth front, we actually become both better people and then we also build better companies.Corey: And sometimes that’s really all that we can ask for. If people want to learn more about what you have to say and get your opinion on these things, okay can they find you?Sanjay: Listen, I’m very approachable. You can follow me on Twitter, I’m on LinkedIn [unintelligible 00:34:54], or my email spoonen@vmware.com. I’m out there.I read voraciously, and probably not as responsive, sometimes, but I try—certainly, customers will hear from me within 24 hours because I try to be very responsive to our customers. But you can connect with me on social media. And I’m honored to be on your show, Corey. I’ve been reading your stuff since it first came out, and then, obviously, a fan of the way you’re thinking about things. Sometimes I feel I need to correct your opinion, and some of that we did today. [laugh]. But you’ve been very—Corey: Oh, I would agree. I come out of this conversation with a different view of VMware than I went into it with. I’m being fully transparent on that.Sanjay: And you’ve helped us. I mean, quite frankly, your blogs and your focus on this and, like, is the V in VMware, like, a bad word? Is it legacy? It’s forced us to think, so I think it’s iron sharpens iron. I’m very delighted that we connected, I don’t know if it was a year or two years ago.And I’ve been a fan; I watch the stuff that you do at re:Invent, so keep going with what you’re doing. I think all of what you write and what you talk about is hopefully making an impact on people who read and listen. And look forward to continuing this dialogue, not just with me, but I think you’re talking to other people in VMware in the future. I’m not the smartest person at VMware, but I’m very fortunate to be [laugh] surrounded by many of them. So hopefully, you get to talk to them, also, in the near future.Corey: [laugh]. I will, of course, will put links to all that in the [show notes 00:36:11]. Thank you so much for taking the time to speak with me today. I really appreciate it.Sanjay: Thanks, Corey, and all the best of you and your organization.Corey: Sanjay Poonen, Chief Operating Officer of VMware, I’m Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you’ve enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you’ve hated this podcast, please leave a five-star review on your podcast platform of choice, along with a condescending comment telling me that in fact, it is a best practice to ship your code to production via FedEx.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.This has been a HumblePod production. Stay humble.

In this episode of the Virtual Coffee with Ashish edition, we spoke with Barak Schoster Goihman (@barakschoster) is the Co-Founder and CTO of Bridgecrew (@Bridgecrewio). Host: Ashish Rajan - Twitter @hashishrajan Guest: Barak Schoster Goihman @barakschoster In this episode, Barak & Ashish spoke about What is Infrastructure as Code Security Application Security vs Infrastructure as Code Security - are they same? What is DevSecOps? Where should one start? Ansible? Terraform? Kubernetes? Saltstack? Configuration and Policy as Code - What are these? How to get started on Infrastructure Security? Open source vs Paid product, what should one consider before going down either path? The future of Infrastructure as Code Security? Difference between a DSL and a general purpose programming language? Becoming a successful startup founder as a developer, what are some tips you can share for future startup founders? And much more… ShowNotes and Episode Transcript on www.cloudsecuritypodcast.tv Twitter - @kaizenteq @hashishrajan If you want to watch videos of this and previous episodes: - Youtube Channel: https://lnkd.in/gUHqSai

In this episode of Cloud Management Monthly, Dan and Matt talk with Vincent Riccio, a Senior Technical Marketing Manager at VMware in the Cloud Management Business Unit about leveraging the power of SaltStack to provide automated consistent configuration management and SecOps across private, public and hybrid clouds. Check out Vincent's blog https://blogs.vmware.com/management/a...​ Give Vincent a follow on Twitterhttps://twitter.com/vincent_riccio

Panelists Eric Berry | Justin Dorfman | Alyssa Wright | Richard Littauer Guest Travis Oliphant | Russell Pekrul Show Notes Hello and welcome to Sustain! Today, we have two guests from OpenTeams in Austin, Travis Oliphant and Russell Pekrul. Travis is the CEO and Russell is the Program Manager and the Founder and Director of FairOSS. We learn all about what OpenTeams and FairOSS are and how they work. Also, Travis tells us about the non-profit he started called NumFOCUS. Other topics discussed are dependencies and how their values are assigned, NumPy and SciPy, and building relationships with companies, which Russell mentions there is a bit of a “chicken and egg” problem here. There is some incredible advice and fascinating stories shared today so go ahead and download this episode now! [00:01:10] We find out what OpenTeams is and how it works. Travis also tells us when he wrote NumPy and SciPy and when he started OpenTeams. [00:07:18] Travis tells us about a non-profit he started with a bunch of people called NumFOCUS so there could be a home for the fiscal sponsor for open source projects. [00:09:24] Russell tells us what FairOSS is and how it works. [00:11:32] Alyssa asks Russell how does he first see the dependencies and then how does he assign that value? He mentions BackYourStack as a starting point. [00:13:00] Eric brings up one of the problems he’s found with trying to fund up open source is that it’s very difficult to solve the problem on more a grand scale. He wonders how Travis and Russell make the impact they want with the magnitude of problems they see. A key piece Travis brings up that they recognize is there’s a data gap and projects have to be participating. Alyssa wonders if projects are aware of their dependencies. [00:17:22] Richard asks about the dependency graph that they are making. He wonders how do you go down the stack and look all the way at the base and how do you judge the usefulness of what dependencies really matter for what code matters for the business proposition? Richard also wonders if anyone has done equity stuff for open source maintainers. [00:23:06] Alyssa is interested in learning more about how Travis and Russell are building the relationships with these companies and what we can do to help. [00:26:35] Alyssa asks Travis and Russell to talk about why this, why now, with this being a time of economic contraction, why is this important? Also, why have they been seeing traction during what can be difficult times for a lot of companies? [00:27:40] Eric asks if Travis can give an example of a project that he feels does that well, that doesn’t have to go through and do it twice, essentially. [00:29:48] Alyssa brings up investments around open source start-ups and how they start with a commitment towards open source and once the investment happens there’s a pivot. She wonders if Travis could talk about how this type of sustainability is shifting that model of these investments. Travis tells a story about speaking to the Founder of SaltStack and how their views matched. [00:34:03] We find out where you can learn more about FairOSS and follow them on this journey, invest, and join in. Spotlight [00:34:52] Justin’s spotlight is Curiefense, which extends Envoy proxy to protect all forms of web traffic. [00:35:15] Alyssa’s spotlight is Pixel8.earth. [00:36:06] Eric’s spotlight is OctoPrint. [00:36:53] Richard’s spotlight is Michael Oliphant’s work. [00:37:36] Russell’s spotlight is Conda. [00:38:20] Travis’s spotlight is Matplotlib. Quotes [00:03:25] “We were connecting and creating a social network long before the social networks started. That was the early days of social networks and it was addicting.” [00:04:14] “New libraries are starting to be written on numarray and we had SciPy written on numeric and there was this fork in this flegging scientific community in Python.” [00:21:18] “So that was a very exciting day. Actually, I remember I told my wife you know the problem I’ve been searching on for twenty years, I finally figured it out. I’ve been trying to figure out twenty years how to make this work, and I finally figured it out. I had to go start several companies and start a venture fund and get involved in finance and cap tables to really pull it off, but that got me excited. Now I also said, but we’re at the base of Mount Everest, like all we’ve got to do is climb to the top of this mountain and we’re there.” [00:22:44] “So you basically have a company and its value is spread to all the values of the projects. You have a bunch of those, have a thousand of those, that each add incrementally the value of a project. Invert the matrix and every project now has a linear dependency on companies that effectively you created an index fund out of every project.” [00:24:52] “The idea is if you can get open source contributors to recognize that they want to work only for companies that are participating people want to hire open source contributors. They’re some of the best people to bring into your company.” [00:25:21] “We found that companies would absolutely sponsor PyData and the reason they would is because they’re trying to hire people. They wanted to hire the best developers and they would. So, they really didn’t care so much about the projects they started, but they wanted the people.” [00:27:10] “Go make an open source project, then get somebody or connect with somebody who’s going to help you build a company that they’ll vest in and build something else. So, you basically have to do it twice.” [00:28:34] “I’ve had the chance to work at companies large and small, go in and see that’s used to do x, and realized it’s added billions of dollars of value to a lot of work for the world. And yet, the same time NumPy struggled, not enough funding to maintain itself.” [00:30:15] “I spoke to the founder of SaltStack that just got acquired by VMware. I spoke to him about his view and it was amazing how much it matched mine, in a sense that he recognized that open source is you build some of the value and you use it. The way you need to make money is to build something that uses it but isn’t the open source.” [00:32:41] “It’s not you’re monetizing open source, you’re empowering, you’re sustaining open source, by selling and connecting the economic value to the functional value that’s there.” [00:33:04] “There will still be challenges. I’m not naïve. Every new thing comes with a whole set of new challenges.” Links OpenTeams (https://openteams.com/about) FairOSS (https://faiross.org/) FairOSS, PBC Twitter (https://twitter.com/faiross_pbc) FairOSS Community (https://community.faiross.org/login) Travis Oliphant Twitter (https://twitter.com/teoliphant?lang=en) Anaconda Dividend Program (https://www.anaconda.com/blog/sustaining-the-open-source-ds-ml-ecosystem-with-the-anaconda-dividend-program) Quansight (https://www.quansight.com/) NumFOCUS (https://numfocus.org/) BackYourStack (https://backyourstack.com/) Dask (https://dask.org/) SaltStack (https://www.saltstack.com/) SciPy (https://www.scipy.org/) NumPy (https://numpy.org/) Curiefense (https://www.curiefense.io/) Pixel8.earth Ambassador Program (https://pixel8earth.medium.com/kicking-off-the-pixel8-earth-ambassador-program-80a87a70fb3a) OctoPrint (https://octoprint.org/) Michael Oliphant’s work (https://langev.com/index.php/author/moliphant/Michael+Oliphant) Conda (https://github.com/conda/conda) Matplotlib.com (https://matplotlib.org/) Credits Produced by Richard Littauer (https://www.burntfen.com/) Edited by Paul M. Bahr at Peachtree Sound (https://www.peachtreesound.com/) Show notes by DeAnn Bahr at Peachtree Sound (https://www.peachtreesound.com/) Special Guests: Russell Pekrul and Travis Oliphant.

Alongside the rise of public clouds, managing the infrastructure of private clouds has never been easier. Tools like Terraform are available, but increasing dependence on them means it's necessary to understand the security implications they present. After all, your entire infrastructure is dependent on, and accessible through, such a configuration—it's essentially infrastructure as code, or "IAC", passed through a tool like Terraform. IAC has effectively become the go-to for most modern infrastructure, given the scale and complexities involved in managing everything manually. For users already using IAC-based setups, we'll look into how one can assess and manage the various security risks present in IAC-powered setups and examine the various tools available for reducing attack surface. For users looking into setting up or switching to IAC-based setups, we'll look at the various advantages of IACs as well. What is infrastructure as code (IAC)? Infrastructure as code is, as its name suggests, a modern way of managing infrastructure in the form of formatted code or simply defined templates which are system readable. These allow for rapid scaling up and down of infrastructure. IAC applies to commonly used cloud infrastructure as well as any other infrastructure that goes along with it, for example, bare-metal systems. IACs allow for easier management, understanding and monitoring of the infrastructure in place, as IACs combine everything into one infrastructure template. IAC definitions or templates are fed into tools like Terraform, Ansible or Saltstack. These tools parse the templates written to manage the infrastructure as defined in the templates. Benefits of IAC IAC provides multiple benefits to DevOps teams, with advantages seen from development to deployment that include the following: Automation IAC allows Devops teams to skip all of the manual work involved in the setting up or scaling upf of infrastructure in use. Automating setup of all the parts of the infrastructure in use can save hours—and in many cases, days—when considering the major deployments found in numerous large organizations. Financial savings Cost savings are possible when using IACs. With infrastructure as code, infrastructure deployments are known well in advance and tested over time, allowing for streamlining and cleaning up. With manual deployments, bits of infrastructure are often found to be in excess or even missing, leading to last-minute delays and unnecessary expense. IAC templates show that every single element of the infrastructure to be deployed is present and accounted for, allowing for better understanding. Replication of infrastructure Tried and tested IAC templates allow for replication of infrastructure from development and testing environments into production environments. This saves time, as templates can be re-used when setting up new production environments as applications are shipped to customers. With the entire infrastructure defined in code, it's possible to amend or remove bits of infrastructure as needed, depending on customer requirement. This also allows for easier customer infrastructure documentation between support and DevOps teams as well. Easier documentation and understanding IAC allows for easier understanding of the infrastructure in place. Having all of the infrastructure defined in a file, or set of files, makes it possible to keep track of everything in use. This prevents teams from forgetting any part of their infrastructure when dealing with system updates, patches, and the like, better ensuring overall security. Infrastructure as code security risks With all the benefits that IAC carries, there are risks as well. As the whole of your infrastructure defined in a file or set of files, let's examine the following considerations: Credential management risks The storing of passwords and other access information (such as SSH keys) is a major security point to keep in mind. T...

VMware sponsored this podcast. SaltStack's Salt is a leading automation and security platform for configuration management for on premises and cloud native environments. Created with Python, Salt is in use among Juniper, Cisco, Cloudflare, Nutanix, SUSE and Tieto, as well as a number of other Fortune 500 technology companies, as well as banks. Saltstack also offers a suite of tools, including SaltStack Enterprise for Salt, Plugin Oriented Programming (POP) and Tiamat, SaltStack's portfolio has also been merged with VMware's suite of offerings, following VMware's purchase of SaltStack earlier this year. In this The New Stack Makers podcast, SaltStack's Thomas Hatch, founder and CTO and Salt's creator, and Janae Andrus, community manager for Salt, discuss SaltStack's roots, evolution and integration with VMware' platforms and technologies. The future of SaltStack's open source projects were also discussed. Alex Williams, founder and publisher of The New Stack, hosted this podcast.

Our guest at today's Breakfast event is Marc Chenn, CEO & Co-founder of SaltStack. SaltStack develops award-winning software used by IT and security operations teams to help modern business more efficiently secure and maintain all aspects of their digital infrastructure. Listen to hear Marc's story and his experience with SaltStack, including the recent announcement of VMware's intent to acquire SaltStack.

Our guest at today's Breakfast event is Marc Chenn, CEO & Co-founder of SaltStack. SaltStack develops award-winning software used by IT and security operations teams to help modern business more efficiently secure and maintain all aspects of their digital infrastructure. Listen to hear Marc's story and his experience with SaltStack, including the recent announcement of VMware's intent to acquire SaltStack.

Today's podcast reports on the alleged end of the Maze ransomware gang, security updates for Oracle, Chrome, Acrobat and SaltStack, and a GrowDiaries' database exposed

Your hosts have an action-packed episode in store for you on The Cloud Pod this week, and Ryan is back after surviving the wild Oregon forest.   A big thanks to this week's sponsors:   Foghorn Consulting, which provides full-stack cloud solutions with a focus on strategy, planning and execution for enterprises seeking to take advantage of the transformative capabilities of AWS, Google Cloud and Azure. Cloud Academy, which provides an intuitive and scalable training platform to meet teams wherever they are along the cloud maturity curve. Use the code THECLOUDPOD for 50% off its training platform.   This week's highlights Amazon is helping you figure out where your money is going.  Google isn't wowing anyone with its AI Platform Prediction improved reliability. Azure has some underwhelming improvements you should read about. General News: This Is What Happens When You Go On Vacation   VMware, Inc. is acquiring SaltStack, Inc. to enhance its vRealize cloud management software suite. It's interesting that this comes only a few weeks after Chef was acquired.     Amazon Web Services: Always Comes Through For Us AWS launches Glue Studio, which provides

Keith Townsend stops by Network Break to lend analysis and commentary on our review of the biggest announcements to come out of VMworld, including Project Monterey and the SaltStack acquisition. We also discuss new products from Arista, acquisitions by Arista and Juniper, Google joining the Linux Foundation's LF Networking, and more.

Keith Townsend stops by Network Break to lend analysis and commentary on our review of the biggest announcements to come out of VMworld, including Project Monterey and the SaltStack acquisition. We also discuss new products from Arista, acquisitions by Arista and Juniper, Google joining the Linux Foundation's LF Networking, and more.

Keith Townsend stops by Network Break to lend analysis and commentary on our review of the biggest announcements to come out of VMworld, including Project Monterey and the SaltStack acquisition. We also discuss new products from Arista, acquisitions by Arista and Juniper, Google joining the Linux Foundation's LF Networking, and more.

Keith Townsend stops by Network Break to lend analysis and commentary on our review of the biggest announcements to come out of VMworld, including Project Monterey and the SaltStack acquisition. We also discuss new products from Arista, acquisitions by Arista and Juniper, Google joining the Linux Foundation's LF Networking, and more. The post Network Break 304: The VMworld 2020 Roundup; Arista Acquires Awake Security appeared first on Packet Pushers.

Keith Townsend stops by Network Break to lend analysis and commentary on our review of the biggest announcements to come out of VMworld, including Project Monterey and the SaltStack acquisition. We also discuss new products from Arista, acquisitions by Arista and Juniper, Google joining the Linux Foundation's LF Networking, and more. The post Network Break 304: The VMworld 2020 Roundup; Arista Acquires Awake Security appeared first on Packet Pushers.

Keith Townsend stops by Network Break to lend analysis and commentary on our review of the biggest announcements to come out of VMworld, including Project Monterey and the SaltStack acquisition. We also discuss new products from Arista, acquisitions by Arista and Juniper, Google joining the Linux Foundation's LF Networking, and more. The post Network Break 304: The VMworld 2020 Roundup; Arista Acquires Awake Security appeared first on Packet Pushers.

During this year's VMworld question and answer, VMware CEO Pat Gelsinger announced the company's intention to acquire SaltStack! In this unique episode of The Hacks, Tom and Chunga take a personal look back to the very beginning. Back to a time when Tom sat by himself, in his basement, developing a computer program that would end up changing his life and ultimately be used by millions of people around the world. How did Salt and eventually Saltstack shape the person that Tom is today? Did it change the way he looks at himself? Is he satisfied with the work he's done? They also take a look at what the future holds for both SaltStack and VMware and discuss how they plan to grow and champion the Salt open source community that has been so vital to SaltStack's success story! To learn more about this episode: Tom's VMware Acquisition Statement  

Tom Hatch is many things, to many people. He's the creator of Salt, He co-founded SaltStack and he's also the company's CTO. Once, while teaching his new students at some college somewhere, he even pretended to be Canadian for a full week, just because he loves the movie "Strange Brew"! Although he's been given many titles, nobody (much to his dismay) has ever given him Royal status--until now. Chunga has proudly crowned Tom Hatch as "The King of SaltConf"! Every year, SaltStack hosts a special gathering of IT Operations and SecOps professionals that are focused on building and maintaining a more secure and reliable digital infrastructure.  SaltConf20 will be the 7th annual SaltStack user conference and this year, for obvious reasons, it will be virtual. FREE registration is open now and the first 100 people to sign up get a free SaltConf20 T-Shirt!  Every year, one of the most common things people say is "We want more Tom"! This year, that's exactly what we're going to give you. Janae Andrus is our special guest for this episode of The Hacks. She'll give you a rundown of what to expect at SaltConf20, how to reserve your space and how you can get your hands on some brand new SaltConf20 SWAG!  Be sure to listen for the secret code as well.  Also, in this episode of The Hacks, Chunga reads some listener and feedback. Listen now! Register for SaltConf20: www.saltconf.com Follow us on Twitter: @saltstack @thehackscast  

Tom and Chunga are talking about IOT (Internet of Things) in this episode of The Hacks! Recently, Chunga read an article which featured industry experts claiming that regardless of size, virtually all IOT development projects will take at least twice as long as originally planned. Tom isn't surprised by this assessment at all. He says people are notoriously bad at understanding the scope and conditions of IOT projects. They're also really bad at understanding how their product will actually be used in everyday life. A big reason for this is simply because IOT designers have to imagine what the world will look like when the project is completed rather than what it looks like in present day. Oftentimes, the technology required to finish a specific product doesn't even exist at the time the project is started. Why would anyone in their right mind choose to get into the business of creating IOT devices?  Tom says "it's because of fame and fortune man!!" For many people, IOT is the modern-day gold rush.  While this pursuit may be irresistible for hordes of businesses, Tom says the issue and complexities surrounding IOT are only going to get worse. Chunga on the other had is skeptical and feels that most of the IOT devices that are currently being pushed to market are gimmicks, nothing more.  Is Chunga right?  Listen now! To learn more about SaltStack and how it can help your business: Visit SaltStack Follow us on Twitter: @thehackscast @saltstack

On April 29, 2020, the Salt management framework, authored by the IT automation company SaltStack, received a patch concerning two CVEs; CVE-2020-11651, an authentication bypass vulnerability, and CVE-2020-11652, a directory-traversal vulnerability. On April 30, 2020, researchers at F-Secure disclosed their vulnerability findings to the public, with an urgent warning for Salt users - patch now. Before the weekend was out, criminals were deploying malware and targeting vulnerable Salt installations, successfully affecting operations at Ghost, DigiCert, and LineageOS. The malware is a cryptominer, but there is an additional component, a Remote Access Tool written in Go called nspps. Researchers at Akamai have also observed in-the-wild attacks on Salt vulnerabilities.  Joining us on this week's Research Saturday is Larry Cashdollar, Senior Security Response Engineer at Akamai, to discuss this issue.  The research can be found here:  SaltStack Vulnerabilities Actively Exploited in the Wild

On April 29, 2020, the Salt management framework, authored by the IT automation company SaltStack, received a patch concerning two CVEs; CVE-2020-11651, an authentication bypass vulnerability, and CVE-2020-11652, a directory-traversal vulnerability. On April 30, 2020, researchers at F-Secure disclosed their vulnerability findings to the public, with an urgent warning for Salt users - patch now. Before the weekend was out, criminals were deploying malware and targeting vulnerable Salt installations, successfully affecting operations at Ghost, DigiCert, and LineageOS. The malware is a cryptominer, but there is an additional component, a Remote Access Tool written in Go called nspps. Researchers at Akamai have also observed in-the-wild attacks on Salt vulnerabilities.  Joining us on this week's Research Saturday is Larry Cashdollar, Senior Security Response Engineer at Akamai, to discuss this issue.  The research can be found here:  SaltStack Vulnerabilities Actively Exploited in the Wild The CyberWire's Research Saturday is presented by Juniper Networks. Thanks to our sponsor Enveil, closing the last gap in data security.

Now that Eric has finished his big move. He and Brandon sit down to discuss one of the most misunderstood concepts in all of IT, DevOps. What does it mean? How do you use it? And can you actually achieve a state of DevOps? Destination Linux Network (https://destinationlinux.network) Sponsor: Bitwarden (https://bitwarden.com/dln) Sudo Show Website (https://sudo.show) Contact Us: * DLN Discourse (https://sudo.show/discuss) * Matrix: +sudoshow:matrix.org What have we been working on? Language Tool (https://languagetool.org) The IT Guy thought moving his family was a good enough excuse not to play with something new this week... Audience Feedback Blog - Ansible and HashiCorp: Better Together (https://www.hashicorp.com/resources/ansible-terraform-better-together/) What is DevOps? Rackspace Video - What is DevOps? In Simple English (https://www.youtube.com/watch?v=_I94-tJlovg) New Stack Blog - Understanding the Difference Between CI and CD (https://thenewstack.io/understanding-the-difference-between-ci-and-cd/) HashiCorp Blog - Infrastructure as Code: What is it? Why is it important? (https://www.hashicorp.com/resources/what-is-infrastructure-as-code/) Call to Action DevOps works! Get involved with DevOps Days (https://devopsdays.org/) Suggested Reading The DevOps Handbook (https://www.amazon.com/gp/offer-listing/1942788002/ref=as_li_tl?ie=UTF8&camp=1789&creative=9325&creativeASIN=1942788002&linkCode=am2&tag=itguyeric-20&linkId=b228c402de7d56748335e3afdd7a45a0) The Phoenix Project (https://www.amazon.com/gp/offer-listing/1942788290/ref=as_li_tl?ie=UTF8&camp=1789&creative=9325&creativeASIN=1942788290&linkCode=as2&tag=itguyeric-20&linkId=d5d1d61df4e7e1d53e366e20a18c2722) DevOps Tools OKD (okd.io) Gitlab (https://gitlab.org) Ansible (https://www.ansible.com/) SaltStack (https://www.saltstack.com/) Jenkins (https://www.jenkins.io/) Terraform (https://www.terraform.io/) Rancher (https://rancher.io) Kubic (https://kubic.opensuse.org/)

Tom and Chunga welcome a special guest to this episode of The Hacks! SaltStack has just announced the release of Enterprise 6.3 and Mehul Revankar is here to talk about all of the updates and the many new features that are packed into the biggest and best Enterprise release yet! Tom, Chunga and Mehul will also tell you how you can get early access.  Listen now! For more about this episode: Reach New Heights With Enterprise 6.3 Follow us on Twitter: @thehackscast Send us an email! thehacks@saltstack.com  

There was much flailing of arms recently as an international examination body decided to rank a CISSP at the same level as a Master's Degree. Kev flexes his honeypots and talks Saltstack. And Paul takes a closer look at a newly discovered Evil Maid with a distinctly Bond-esque moniker.

For the second time, Alex Peay, Senior VP of Product and Marketing at SaltStack joins Tom & Chunga. The Hacks have asked Alex to return to talk about a blog he wrote called "3-Lies SecOps Teams Tell Themselves". Alex wrote this blog after attending this year's RSA Conference in San Francisco. After attending several meetings and speeches, he started noticing a common narrative that was threaded throughout all of the presentations; regardless of who was talking. This narrative had a problem--it wasn't accurate. Actually, it wasn't even true! Over the course of this conference, Alex determined that he was hearing 3-lies, over and over.  What were they? Listen now to find out. To learn more about this episode: Alex Peay's blog "3-Lies SecOps Teams Tell Themselves So They Can Sleep At Night" Follow us on Twitter: @thehackscast Questions about this episode? Send us an email: thehacks@saltstack.com