POPULARITY
40 episodes with cloudformation
12 episodes with cloudformation
7 episodes with cloudformation
13 episodes with cloudformation
7 episodes with cloudformation
4 episodes with cloudformation
4 episodes with cloudformation
3 episodes with cloudformation
2 episodes with cloudformation
Unlock the power of generative AI for cloud architecture! In this vBrownBag episode, Alex Kearns demonstrates how to build a well-architected review crew using agentic AI, AWS Bedrock, and open-source tools. Learn to automate AWS Well-Architected Framework reviews, leverage knowledge bases, and see a live demo analyzing CloudFormation templates. Whether you're a cloud consultant or developer, discover practical ways to scale best practices and save time with GenAI. #cloud #AWS #AI #GenerativeAI #WellArchitected #vBrownBag #CloudComputing #DevOps Chapters: 00:00 – Introduction & Guest Welcom 04:00 – Alex's Cloud & AI Journey 17:00 – Building the GenAI Review Crew 34:00 – Live Demo: Automated Well-Architected Review 53:00 – Q&A & Future of AI in Cloud Architecture Resources:
Une série d'annonce pour cet épisode, c'est aujourd'hui que sont présentés les nouveaux ASA A20/30/50, jusqu'à 171% plus performant que les anciens modèles.SANtricity n'a pas été oublié avec les nouveaux EF-Series EF400C et EF600CCoté Data Management, Data Infrastructure Insights (DII, ex-CloudInsights) évolue pour une analyse encore plus poussée de vos environnements SAN, et la prise en charge de VMware VSAN.La sécurité n'est pas en reste avec une assistance gratuite au rétablissement de service suite à une attaque ransomware, la prise en charge de vos workloads SAN avec Autonomous Ransomware Protection for Block, et la simulation d'attaques ransomware dans BlueXP.Et le cloud ? ANF propose maintenant des volumes de 50GB seulement, et le catalogue CloudFormation de AWS prend en charge les fonctionnalités clé de ONTAP avec FSxN.Les annonces sur le site web de NetAppYann Bizeul (Linked-In)Guillaume Sowinski (Linked-In)Yves Weisser (Linked-In)
AWS Morning Brief for the week of February 10, with Corey Quinn. Links:Amazon EBS now supports additional resource-level permissions for creating EBS volumes from snapshotsAmazon Managed Service for Prometheus collector adds support for cross-account ingestionAmazon Redshift Concurrency Scaling is now available in 1 additional regionAmazon Q Developer now troubleshoots AWS Console errors in all AWS Commercial regionsAmazon Q Developer introduces a new, simplified setup experience for Pro tier subscriptionsAWS IAM Identity Center now offers improved error messages and AWS CloudTrail logging for provisioning issuesAWS Step Functions now supports 100,000 state machines and activities per AWS accountCost Optimization Hub supports more EC2 Auto Scaling group recommendationsAnnouncing the general availability of AWS Database Migration Service Serverless support for files with an S3 source endpointAI-Powered Football Match Analysis: SAP Sports One on AWSIntroducing AWS CloudFormation Stack RefactoringAWS Tightens the Reins: New AWS SaaS Marketplace Rules Will Impact Your Commitments
Smart Segmenting (feat. Toby Höbel) --- Mautic 5.2 LTS, ELTS, Integration Framework Tutorial, AWS Fargate and CloudFormation, M5 plugins, NHIs, GDPR proposal, Year in Review, Certification Provider RFP, Mauticon Lisbon, mautic.org relaunch, MautiCon 9-10 July 2025 / Speakers --- Mautic & ELTS Mautic 5.2: https://github.com/mautic/mautic/releases/tag/5.2.0 ELTS: https://www.mautic.org/blog/community/announcing-launch-mautics-extended-long-term-support-elts-program 3rd-Party & Know-How Tutorials by Matic: https://mzagmajster.si/leverage-the-integration-framework-in-mautic-5/ and https://mzagmajster.si/how-to-add-a-custom-email-token-to-mautic/ DevOps-Tutorials by Avinash: Mautic in AWS Fargate (“Serverless”, i.e. Docker) https://x.com/AvinashDalvi_/status/1856977257404981428 plus CloudFormation (https://x.com/AvinashDalvi_/status/1861286049315725484) All Kuzmani plugins now Mautic 5 https://mtcextendee.com/ – ELTS support coming up! Initiatives Community (DACH): NHIs & countermeasures (some background: https://www.linkedin.com/posts/alisongootee_deliverability-emailgeeks-botornot-activity-7097719979738890240-FMJT/) Preparing GDPR proposal – please contribute ideas: https://forum.mautic.org/t/rfc-improving-mautics-gdpr-compliance-cookie-management/27801/15 Interview: Smart Segmenting (feat. Toby Höbel) https://audienture.com Community Year in Review: https://forum.mautic.org/t/2024-the-year-in-review/34577 Vote for new Council member (through 17.Jan – be a member to vote!): https://community.mautic.org/processes/council-election-24 Certification Provider wanted: https://www.mautic.org/blog/rfp-certification-provider Mauticon Europe Lisbon Aftermath: https://www.mautic.org/blog/whats-new-mautic-learnings-mautic-conference-europe and https://johnlinhart.com/blog/mauticon-lisbon-2024#mauticon-lisbon-2024 MautiCon / Save the date 9-10 July 2025 / Speakers!
Smart Segmenting (feat. Toby Höbel) --- Mautic 5.2 LTS, ELTS, Integration Framework Tutorial, AWS Fargate und CloudFormation, M5-Plugins, NHIs, DSGVO-Initiative, Jahresrückblick, Ausschreibung "Certification Provider", Mauticon Lissabon, mautic.org Relaunch, MautiCon 9-10 July 2025 / Sprecher --- Mautic & ELTS Mautic 5.2: https://github.com/mautic/mautic/releases/tag/5.2.0 ELTS: https://www.mautic.org/blog/community/announcing-launch-mautics-extended-long-term-support-elts-program 3rd-Party Features & Knowhow Tutorials von Matic: https://mzagmajster.si/leverage-the-integration-framework-in-mautic-5/ und https://mzagmajster.si/how-to-add-a-custom-email-token-to-mautic/ DevOps-Tutorials von Avinash: Mautic in AWS Fargate („Serverless“, also Docker) https://x.com/AvinashDalvi_/status/1856977257404981428 plus CloudFormation (https://x.com/AvinashDalvi_/status/1861286049315725484) Alle Kuzmani-Plugins jetzt auf Mautic 5 https://mtcextendee.com/ – mit ELTS Support! Initiativen Community (DACH): NHIs – und Gegenmaßnahmen (etwas Hintergrund: https://www.linkedin.com/posts/alisongootee_deliverability-emailgeeks-botornot-activity-7097719979738890240-FMJT) Vorbereitung DSGVO-Proposal – bitte Ideen beisteuern: https://forum.mautic.org/t/rfc-improving-mautics-gdpr-compliance-cookie-management/27801/15 Interview: Smartes Segmentieren (feat. Toby Höbel) https://audienture.com Community Jahresrückblick: https://forum.mautic.org/t/2024-the-year-in-review/34577 Neues Council-Mitglied (Abstimmung bis 17.1.): https://community.mautic.org/processes/council-election-24 Ausschreibung: Certification Provider gesucht: https://www.mautic.org/blog/rfp-certification-provider Mauticon Europe Lissabon Rückblick: https://www.mautic.org/blog/whats-new-mautic-learnings-mautic-conference-europe und https://johnlinhart.com/blog/mauticon-lisbon-2024#mauticon-lisbon-2024 MautiCon / Save the Date: 9-10 Juli 2025 / Speakers gesucht!
J'ai compté 86 nouveautés ces deux dernières semaines, en ligne avec le rythme d'octobre. C'est une semaine d'anniversaire puisque nous fetons les 20 ans du blogs AWS News et les 10 ans de AWS Lambda. J'ai aussi accroché pour vous des nouveautés sympas pour EBS et CloudFormation, de nouvelles APIs pour Amazon Location Service, des brokers express pour Kafka et un nouveau type de policy IAM - si vous travaillez dans des équipes sécurité, vous allez adorer : les resource control policy ou RCP.
In this episode, we discuss the concept of CloudFormation drift, what causes it, how to detect it, and strategies for resolving it. We explain that drift happens when the actual state of resources diverges from what is defined in the CloudFormation templates. Common causes include manual changes, third party tools, mixing IaC solutions, and automation. We then cover built-in drift detection in CloudFormation and integrating it with alarms. Finally, we suggest approaches for reconciling drift like change sets, deletion protection, and bringing up parallel stacks.
In this episode, Brian LeRoux, co-founder of Begin.com, discusses the evolution and rise of serverless full stack development. Brian shares insights on the history and future of JavaScript, the benefits of serverless architecture, and how front-end developers can leverage these technologies to build scalable and maintainable applications. Links https://brian.io https://webdev.rip https://github.com/brianleroux https://www.npmjs.com/~brianleroux https://twitter.com/brianleroux https://indieweb.social/@brianleroux https://www.linkedin.com/in/brianleroux https://begin.com https://arc.codes https://enhance.dev We want to hear from you! How did you find us? Did you see us on Twitter? In a newsletter? Or maybe we were recommended by a friend? Let us know by sending an email to our producer, Emily, at emily.kochanekketner@logrocket.com (mailto:emily.kochanekketner@logrocket.com), or tweet at us at PodRocketPod (https://twitter.com/PodRocketpod). Follow us. Get free stickers. Follow us on Apple Podcasts, fill out this form (https://podrocket.logrocket.com/get-podrocket-stickers), and we'll send you free PodRocket stickers! What does LogRocket do? LogRocket provides AI-first session replay and analytics that surfaces the UX and technical issues impacting user experiences. Start understand where your users are struggling by trying it for free at [LogRocket.com]. Try LogRocket for free today.(https://logrocket.com/signup/?pdr) Special Guest: Brian LeRoux.
Andreas and Michael Wittig were pretty jazzed about writing unit tests using mocks for the AWS SDK v3 in JavaScript. They broke down Amazon's new GuardDuty malware protection for S3 and how it compares to their own product bucketAV. The duo also covered testing Terraform modules and using aws-nuke to clean up leftover resources from failed tests. They gave their two cents on some recent AWS service announcements too - CloudWatch, Fargate, CloudFormation and more!
Spacelift is a platform that adds workflows on top of infrastructure as code tooling. One of its USPs is that it is agnostic to the underlying infrastructure as code tool, so it can be used with Terraform, CloudFormation, Pulomi, to name but a few. This concept becomes super powerful in that you can combine stacks written for different infrastructure as code tools into dependency workflows, for example the output of a cloud formation stack could feed into a stack written in Terraform. Jean-Marc Fontaine, head of solutions & support at Spacelift, takes us through some of the possibilities that Spacelift enables and the different angle of attack they have in terms of solving infrastructure as code pipelines.
In this episode, we discuss 5 different ways to extend CloudFormation capabilities beyond what it natively supports. We started with a quick recap of what CloudFormation is and why we might need to extend it. We then covered using custom scripts and templating engines, which can be effective but require extra maintenance. We recommended relying instead on tools like Serverless Framework, SAM, and CDK which generate CloudFormation templates but provide abstractions and syntax improvements. When you need custom resources, CloudFormation macros allow pre-processing templates, while custom resources and the CloudFormation registry allow defining new resource types. We summarized recommendations for when to use each approach based on our experience. Overall, we covered multiple options for extending CloudFormation to support more complex infrastructure needs.
Join Dave in this enlightening conversation with Dan Blanco, Senior Developer Advocate for the AWS Infrastructure as Code (IaC) team, as they dive deep into the transformative world of IaC. In an era where manual processes no longer suffice, IaC emerges as a pivotal technology, enabling developers to provision and manage computing infrastructure through code, streamlining the setup, update, and maintenance of essential infrastructure components for application development, testing, and deployment. In this episode, Dan unveils the latest innovations from his team, including the groundbreaking AWS CloudFormation Git sync, CDK Migrate, and the IaC Generator. Discover how the IaC Generator revolutionizes template creation, allowing developers to automatically generate CloudFormation templates for resources in their accounts, enhancing efficiency and control over cloud environments. Beyond the technical, Dave and Dan explore the unique journey of being a developer in the gaming industry, the transition to cloud careers, and share a feast of food analogies along with insider knowledge from culinary school. This episode is a must-listen for anyone looking to understand the future of cloud infrastructure management, control costs, reduce risks, and seize new business opportunities with agility and speed. Dan on Twitter: https://twitter.com/thedanblanco Dan on LinkedIn: https://www.linkedin.com/in/thedanblanco/ Dave on Twitter: https://twitter.com/thedavedev Dave on LinkedIn: https://www.linkedin.com/in/davidisbitski [BLOG] AWS DevOps Blog: https://aws.amazon.com/blogs/devops [BLOG] Dan's Blog Post on IaC Generator- Import entire applications into AWS CloudFormation: https://aws.amazon.com/blogs/devops/import-entire-applications-into-aws-cloudformation/ [BLOG] Announcing CDK Migrate: A Single Command to Migrate to the AWS CDK - https://aws.amazon.com/blogs/devops/announcing-cdk-migrate-a-single-command-to-migrate-to-the-aws-cdk/ [DOCS] Generating templates for existing resources - https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/generate-IaC.html [DOCS] Working with AWS CloudFormation Git Sync - https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/git-sync.html [TRAINING] AWS IaC Workshops: https://cdkworkshop.com and https://catalog.workshops.aws/cfn101/en-US Subscribe: Spotify: https://open.spotify.com/show/7rQjgnBvuyr18K03tnEHBI Apple Podcasts: https://podcasts.apple.com/us/podcast/aws-developers-podcast/id1574162669 Stitcher: https://www.stitcher.com/show/1065378 Pandora: https://www.pandora.com/podcast/aws-developers-podcast/PC:1001065378 TuneIn: https://tunein.com/podcasts/Technology-Podcasts/AWS-Developers-Podcast-p1461814/ Amazon Music: https://music.amazon.com/podcasts/f8bf7630-2521-4b40-be90-c46a9222c159/aws-developers-podcast Google Podcasts: https://podcasts.google.com/feed/aHR0cHM6Ly9mZWVkcy5zb3VuZGNsb3VkLmNvbS91c2Vycy9zb3VuZGNsb3VkOnVzZXJzOjk5NDM2MzU0OS9zb3VuZHMucnNz RSS Feed: https://feeds.soundcloud.com/users/soundcloud:users:994363549/sounds.rss
Matthew Bonig, Chief Cloud Architect at Defiance Digital, joins Corey on Screaming in the Cloud to discuss his experiences in CDK, why developers can't be solely reliant on AI or coding tools to fill in the blanks, and his biggest grievances with AWS. Matthew gives an in-depth look at how and why CDK has been so influential for him, as well as the positive work that Defiance Digital is doing as a managed service provider. Corey and Matthew debate the need for AWS to focus on innovating instead of simply surviving off its existing customer base.About MatthewChief Cloud Architect at Defiance Digital. AWS DevTools Hero, co-author of The CDK Book, author of the Advanced CDK Course. All things CDK and Star Trek.Links Referenced:CDK Book: https://www.thecdkbook.com/cdk.dev: https://cdk.devTwitter: https://twitter.com/mattbonigLinkedIn: https://www.linkedin.com/in/matthewbonig/Personal website: https://matthewbonig.comduckbillgroup.com: https://duckbillgroup.comTranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. And I'm back with my first recording that was conducted post-re:Invent and all of its attendant glory and nonsense; we might talk a little bit about what happened at the show. But my guest today is the Chief Cloud Architect at Defiance Digital, Matthew Bonig. Matthew, thank you for joining me.Matthew: Thank you, Corey. Thanks for having me today.Corey: So, you are deep into the CDK. You're one of the AWS Dev Tools Heros, and you're the co-author of the CDK Book, you've done a lot, really. You have a course now for Advanced CDK work. Honestly, at this point, it starts to feel like when I say the CDK is a cult, you're one of the cult leaders, or at least very high up in the cult.Matthew: [laugh] Yes, it was something that I discovered—Corey: Your robe has a fringe on it.Matthew: Yeah, yeah. I discovered this at re:Invent, and it kind of hit me a little surprised that I got called out by a couple people by being the CDK guy. And I didn't realize that I'd hit that status yet, so I got to get myself a hat, and a cloak, and maybe some fun stuff to wear.Corey: For me, what I saw on the—it was in the run-up to re:Invent, but the big CDK sized announcement was the fact that the new version of Amplify now is much closer tied to the CDK than it was in previous incarnations, which is great. It sort of solves the problem, how do I build a thing through a variety of different tools? Great, and how do I manage that thing programmatically? It seems if, according to what it says on the tin, that it narrows that gap. Of course, here in reality, I haven't had time to pick anything like that up, and I won't for months, just because so much comes out all at the same time. What happened in the CDK world? What did I miss? What's exciting?Matthew: Well, you know, the CDK world has been, I've said, fairly mature for a while now. You know, fundamentally the way the CDK works and the functionality within it hasn't changed drastically. Even when 2.0 came out a couple of years ago, there wasn't a drastic fundamental change in the way that the API worked. Really, the efforts that we've been seeing for the last year or so, and especially the last few months, is trying to button up some functionality, hit some of those edge cases have been rough for some users, and ultimately just continue to fill out things like L2 constructs and maybe try to build out some L3s.I think what they're doing with Amplify is a good sign that they are trying to, sort of, reach across the aisle and work with other frameworks and work with other systems within AWS to make the experience better, shows their commitment to the CDK of making it really the first class citizen for doing IaC work in AWS.Corey: I think that that is a—that's a long road, and it's also a lot of work under the hood that's not easily appreciated. You've remarked at one point that my talk at the CDK Community Day was illuminating, if nothing else, if for no other reason than I dressed up as a legitimate actual cultist and a robe to give the talk—Matthew: Yeah. Loved it.Corey: Because I have deep-seated emotional problems. But it was fun. It talked a bit about my journey with it, where originally I viewed it as, more or less, this thing that was not for me. And a large part of that because I come from a world of sysadmin ops types, where, “I don't really know how to code,” was sort of my approach to this. Because I was reaff—I had that reaffirmed every time I talked to a developer. Like, “You call this a bash script? It's terrible.” And sure, but it worked, and it tied into a different knowledge set.Then, when I encountered the CDK for the first time, I tried to use it in Python, which at the time was not really well-supported and led to unfortunate outcomes—I do not know if that's still the case—what got me into it, in seriousness, was when I tried it a few months later with TypeScript and that started to work a little bit more clearly, with the caveat that I did not know JavaScript, I did not know TypeScript, I had to learn it as I went in service to the CDK. And it works really well insofar as it scratched an itch that I had. There's a whole class of problems that I don't have to deal with, which include getting someone who isn't me involved in some of that codebase, or working in environments where you have either a monorepo or a crap ton of tiny repos scattered everywhere and collaborating with other people. I cannot speak authoritatively to any of that. I will say it's incredibly annoying when I'm trying to update something written in the CDK, and then I have touched it in a year-and-a-half, and the first thing I have to do is upgrade a whole a bunch of dependencies, clear half a day just to get the warnings to clear before I can go ahead and deploy the things, let alone implement the tiny change I'm logging into the thing to fix.Matthew: Oh, yeah, yes. Yeah, the dependency updates are probably one of the most infuriating things about any Node.js system, and I don't think that I've ever run across any application project framework, anything in which doing dependency upgrades wasn't a nightmare. And I think it's because the Node.js community, more so than I've seen any other, doesn't care about semantic versioning. And unfortunately, the CDK doesn't technically care about semantic versioning, either, which makes it very tricky to do upgrades properly.Corey: There also seems to be the additional problem layered on top, which is all of the various documentation sources that I stumble upon, the official documentation, not terrific at giving real-world use case. It feels like it's trying to read the dictionary to learn how English works, not really its purpose. So, I find a bunch of blog posts, and all of them tend to approach this ecosystem slightly differently. One talks about using NPM. Another talks about Yarn.If you're doing anything that involves a web app, as seems to be increasingly common, some will say, “Oh, use WEBrick,” others will recommend using Vite. There's the whole JavaScript framework wars, and the only unifying best practice seems to be, “Oh, there's another way to do it that you should be using instead of the way you currently are on.” And if you listen to that, you wind up in hell.Matthew: Oh, horribly so. Yeah, the split in the ecosystem between NPM and Yarn, I think, has been incredibly detrimental to the overall comfort level in Node.js development. You know, I was an NPM guy for many, many years, and then actually, the CDK got me more using Yarn, simply because Yarn handles cross-library dependency resolution a bit different from NPM. And I just ran into fewer errors and fewer problems if I use Yarn along the way.But NPM then came a long way since then. Now, there's also a PNPM, which is good if you're using monorepos. But then if you're going to be using monorepos, there's another 15 tools out there that you can use for those sorts of things. And ultimately, I think it's going to be what is the thing that causes you the least amount of problems when dealing with them. And every single dependency issue that I've ever run into when upgrading any project, whether it be a web application, a back-end API, or the CDK, it's always unique enough that there isn't a one-size-fits-all answer to solving those problems.Corey: The most recent experience I had with the CDK—since you know, you're basically Mr. CDK at this point, whether you want to be or not, and this is what I do, instead of filing issues anywhere or asking for help, I drag people onto this show, and then basically assault them with my weird use cases—I'm in the process of building something out in the service of shitposting, because that is my nature, and I decided, oh, there's a new thing called the Dynamo table v2—Matthew: Yes.Corey: Which is great. I looked into it. The big difference is that it addresses it from the beginning as a global table, so you have optionality. Cool. Trying to migrate something that is existing from a Dynamo table to a Dynamo v2 table started throwing CloudFormation issues, so my answer was—this was pre-production—just tear down the stack and rebuild it. That feels like that would be a problem if this had been something that was actually full of data at this point.Matthew: There's a couple of ways that you could maybe go about it. Now, this is a very special case that you mentioned because you're talking about fundamentally changing the CloudFormation resource that you are creating, so of course, the CDK being an abstraction layer over top of CloudFormation and the Dynamo table v2 using the global table resource rather than just the table resource. If you had a case where you have to do that migration—and I've actually got a client right now who's very much looking to do that—the process would probably be to orphan the existing table so that you can retain the data and then using an import routine with CloudFormation to bring that in under the new resource. I haven't tried it yet—Corey: In this case, the table was empty, so it was easy enough to just destroy and then recreate, but it meant that I also had to tear down and recreate everything else in the stack as well, including CloudFront distributions, ACM certificates, so it took 20 minutes.Matthew: Yes. And that is one of the reasons why I often will stick any sort of stateful resource into their own stack so that if I have to go through an operation like this, I'm know that I'm not going to be modifying things that are very painful to drop and recreate, like, CloudFront distributions, which can take a half an hour or more to re-initialize.Corey: Yeah. So, that was fun. The problem got sorted out, but it was still a bit challenging. I feel like at some level, the CDK is hobbled by the fact that under the hood, it really just is just CloudFormation once all is said and done, and CloudFormation has never been the speediest thing. I didn't understand that until I started playing with Terraform and I saw how much more quickly it could provision things just by calling the service APIs directly. It sort of raises the question of what the hell the CloudFormation service is doing when it takes five times longer to do effectively the same thing.Matthew: Yeah, and the big thing that I appreciate about Terraform versus CloudFormation—speed being kind of the big win—is the fact that Terraform doesn't obfuscate or hide state from you. If you absolutely need to, you can go in and change that state that relates your Terraform definitions to the back-end resources. You can't do that with CloudFormation. So CloudFormation, did release few years ago, that import routine, and that was pretty good—not great, but pretty good; it's getting better all the time—whereas this was a complete and unneeded feature with Terraform because if it came down to the point where you already had a resource, and you just want to tie it to your IaC, you just edit a state file. And they've got their import routines and tie-in routines as well, but having that underlying state exposed was a big advantage, in my mind, to Terraform that I missed going to CloudFormation, and still to this day frustrates me that I can't do that underlying state change.Corey: It becomes painful and challenging, for better or worse.Matthew: Yep.Corey: But yeah, that was what I ran into. Things have improved, though. When I google various topics, I find that the v2 documentation comes up instead of the v1. That was maddening for a little while. I find that there are still things that annoy me, but they become less all the time, partially because I feel like I'm getting better at knowing how to search for them, and also because I think I'm becoming broken in the right ways that the CDK tends to expect.Matthew: Oh, like how?Corey: Oh, easy example here: I was recently trying to get something set up and running, and I don't know why this is the case, I don't know if it holds true and other programming languages, but I'm getting more used to the fact that there are two files in TypeScript-land that run a project. One is generally small and in a side directory that no one cares about, I think it's in a lib or the bin subdirectory. I don't remember which because I don't care. And then there are things you have to do within the other equivalent that basically reference each other. And I've gotten better at understanding that those aren't one file, for example. Though they seem to sure be a lot in all the demos, but it's not how the init process, when you're starting something new, spins up.Matthew: Yeah, this is the hell of TypeScript, the fact that Node.js, as a runtime, cannot process TypeScript files, so you always have to pass them through a compiler. This is actually one of the things that I like about using Projen for all of my projects instead of using CDK init to start them is that those baseline configurations handle the TypeScript nature of the runtime—or I should say, the anti-TypeScript nature of the runtime a little bit better, and you run into fewer problems. You never have to worry about necessarily doing build routines or other things because they actually use the ts-node runtime to handle your CDK files instead of the node runtime. And I think that's a big benefit in terms of the developer experience. It just makes it so I generally never have to care about those JavaScript files that get compiled from TypeScript. In the, you know, two years or so I've been using Projen, I never have to worry about a build routine to turn that into JavaScript. And that makes the developer experience significantly better.Corey: Yeah, I still miss an awful lot of things that I feel like I should be understanding. I've never touched Projen, for example. It's on my backlog of things to look into.Matthew: Highly recommend it.Corey: Yeah, I also am still in that area of… my TypeScript knowledge has not yet gotten to a point where I see the value of it. It feels like I've spent far more time fighting with the arbitrary restrictions that are TypeScript than it has saved me from typing errors in anything that I've built. I believe it has to come back around at some point of familiarity with the language, but I'm not there yet.Matthew: Got you. So, Python developer before this?Corey: Ish. Mostly brute force and enthusiasm, but yeah, Python.Matthew: Python, and I think you said bash scripting and other things that have no inherent typing built into it.Corey: Right.Matthew: Yeah, that is a problem, I think… that I thankfully avoided. I was an application developer for many years. My background and my experience has always been around strongly typed languages, so when it came to adopting the CDK, everything felt very natural to me. But as I've worked with people over the years, both internally at Defiance as well as people in the community that don't have a background in that, I've been exposed to how problematic TypeScript as a language truly can be for someone who has never had this experience of, I've got this thing and it has a well-defined shape to it, and if I don't respect that, then I'm going to bang my head against to these weird errors that are hard to comprehend and hard to grok way more than it feels like I'm getting value from it.Corey: There's also a lack of understanding around how to structure projects, in my case, where all right, I have a front-end and I have a back-end. Is this all within the context of the CDK project? And this, of course, also presupposes that everything I'm doing is effectively greenfield, in which case, great, do I use the front-end wizard tutorial thing that I'm following, and how does that integrate when I'm using the CDK to deploy it somewhere, and so on and so forth. It's stuff that makes sense once you have angry and loud enough opinions, but I don't yet.Matthew: Yeah, so the key thing that I tell people about project structure—because it does often come up a lot—is that ultimately, the CDK itself doesn't really care how you structure things. So, how you structure, where you put certain files, how you organize them, is your personal preference. Now, there are some exceptions to that. When it comes to things like Lambda functions that you're building or Docker files, there are probably some better practices you can go through, but it's actually more dependent on those systems rather than the CDK directly itself. So I go through, in the Advanced CDK course, you know, my basic starting directory structure for everything, which is stacks, constructs, apps, and stages all go into their own specific directories.But then once those directories start growing—because I've added more stacks, more constructs, and things—once I get to around five to maybe seven files in a directory, then I look at them and go, “Okay, how can I group these together?” I create subdirectories, I move those files around. My development tool of choice, which is WebStorm—JetBrains's long-running tool—handles the moving of those files for me, so all of my imports, all of my references automatically get updated accordingly, which is really nice, and I can refactor things as much as I want to without too much of a problem. So, as a project grows over time, my directory structure can change to make sure that it is readable, well organized, and understandable, and it's never been too much of a problem.Corey: Yeah, it's one of those things that does take some getting used to. It helps, I think, having a mentor of sorts to take you under their wing and explain these things to you, but that's a hard thing to scale as well. So, in the absence of that we wind up defaulting to oh, whatever the most recent blog post we read is.Matthew: Yeah. Yeah, and I think one of the truest, I think, and truthful complaints I've heard about the CDK and why it can be fundamentally very difficult is that it has no guardrails. It is a general-purpose languages, and general purpose languages don't have guardrails. They don't want to be in the way of you building whatever you need to build.But when it comes to an Infrastructure as Code project, which is inherently very different from an API or a website or other, sort of, more typical programming projects, having guardrail—or not having guardrails is a bad thing, and it can really lead you down some bad paths. I remember working with a client this last year who had leveraged context instead of properties on classes to hand configuration value down through code, down through stacks and constructs and things like that. And it worked. It functionally got them what they needed, up until a point, and then all of sudden, they were like, “Well, now we want to do X with the CDK, and we simply cannot because we've now painted ourselves into a corner.” And that's the downside of not having these good guard rails.And I think that early, they needed to do this early on. When the CDK was initially released, and it got popular back around the 0.4, 0.5 timeframe—I think I picked it up right around 0.4, too—when it officially hit a 1.0 release, there should have been a better set of guidelines and best practices published. You can go to the documents and see them, and they have been published, but it really didn't go far enough to really explain how and why you had to take the steps to make sure you didn't screw yourself six months later.Corey: It's sort of those one-way doors you don't realize you're passing through when you first start building something. And I find, especially when you follow my development approach of more or less used to be copying and pasting for various places, now it's copying and pasting from one place which is Chat-Gippity-4, then—although I've seen increasingly GitHub's Copilot has been great at this and Code Whisperer, in my experience, has not yet been worth the energy it takes to really go diving into it. Your mileage may of course vary on that. But I found it was not making materially better or suggestions on CDK stuff then Copilot was.Matthew: Yeah, I haven't tried Code Whisperer outside of the shell. I've been using Copilot for the last year and absolutely adore it. I think it has completely changed the way that I felt about coding. I saw writing code for the last couple of years as being very tedious and very boring in terms of there weren't interesting problems to solve, and Copilot, as I've seen it, is autocomplete on steroids. So, it doesn't keep me from having to solve the interesting problems; it just keeps me from having to type out the boring solutions, and it's the thing that I love about it.Now, hopefully, Code Whisperer continues to get better over time. I'm hoping all of Amazon's GenAI products continue to get better over time and I can maybe ditch a subscription to Copilot, but for now, Copilot is still my thing. And it's producing good enough results for me. Thankfully because I've been working with it for four years now, I don't rely on it to answer my questions about how to use constructs. I go back to the docs for those. If I need to.Corey: It occurs to me that I can talk about this now because this episode will not air until after this has become generally available, but what's really spanked it from my perspective has been Google's Duet. And the key defining difference is, as I'm in one of these files—in many cases, I'm doing something with React these days due to an escalating series of weird choices—and—Matthew: My apologies, by the way. My condolences, I should say.Corey: Well, yeah. Well, things like Copilot Chat are great when they say, “Oh yeah, assuming that you're handling the state this way in your component, now…” What I love about Duet is it goes, and it actually checks, which is awesome. And it has contextual awareness of the entire project, not just the three lines that I'm talking about, or the file that I'm looking at this moment. It goes ahead and does the intelligent thing of looking at some of these things. It still has some problems where it's confidently wrong about things that really shouldn't be, but okay, early days.Matthew: Sure. Yeah, I'll need to check that out a little bit more because I still, to this day, despise working with React. It is still my framework of choice because the ecosystem is so good around it. And so, established that I know that whatever problem I have, I'll find 14 blogs, and maybe one of them is the answer that I want, versus any other framework where it still feels so very new and so very immature that I will probably beat my head more than I want to. Web development now is a hobby, not a job, so I don't want to bang my head against a hobby project.Corey: I tend to view, on some level, that these AIs coding assistants are good enough to get me almost anywhere I need to go, to the point where a beginner or enthusiastic amateur will be able to get sorted out. And for a lot of what I'm building, that's all I really need. I don't need this to be something that will withstand the rigors of production at a bank, for example. One challenge I have seen with all these things is there's a delay in something being released and their training data growing to understand those things. Very often it'll wind up giving me recommendations for—I forget the name of it, but there was a state manager in React that the first thing you saw when you installed it was, “This has been deprecated. This is the new replacement.” And if you explicitly ask about the replacement, it does the right thing, but it just cheerfully goes ahead and tells you to use ancient stuff or apply poor security practices or the rest.Matthew: Yeah, that's very scary to me, to be honest because I think these AI development tools—for me, it's revitalized my interest in doing development, but where I get really, really scared is where they become a dependency in writing the right code. And every time I ever use Copilot to fill out stuff, I'm always double-checking, and I'm always making sure that this is right or that is right. And what I worry about is those developers who are maybe still learning some things, or are having to write in-line SQL on to their back-end and let Copilot, or Code Whisperer, or whatever tool they pick fill this stuff out, and that answer is based on a solution that works for a 10,000 record database, but fails horribly on a 100 million record database. And now all of a sudden, and you've got this problem that is just festering in through a dev environment, in through a QA environment, and even maybe into a prod environment, and you don't find out that failure until six months later, when some database table runs past its magical limit and now all of sudden, you've got these queries that are failing, they're crashing databases, they're running into problems, and this developer that didn't really know what they built in the first place is now being asked, “Why doesn't your code work,” and they just sort of have to go, “Maybe ChatGPT can tell me why my code doesn't work.” And that's the scariest part of me to these things is that they're a little bit too good at answering difficult questions with a simple answer. There is no, “It depends,” with these answers, and there needs to be for a lot of what we do in complex systems that, for example, in the AWS world, we're expected to build complex systems, and ChatGPT and these other tools are bad at that.Corey: We're required to build complex systems, and, on some level, I would put that onus on Amazon in many respects. I mean, the challenge I keep smacking into is that they're building—they're giving you a bunch of components and expecting you to assemble them all yourself to achieve even relatively simple things. It increasingly feels like this is the direction that they want customers to go in because they're bad at moving up the stack and develop—delivering integrated solutions themselves.Matthew: Well, so I would wonder, would you consider a relatively simple system, then?Corey: Okay, one of the things I like to do is go out in the evenings, and sometimes with a friend, I'll have a few too many beers. And then I'll come up with an idea for I want to redirect this random domain that I want to buy to someone else's website. The end. Now, if you go with Namecheap, or GoDaddy, or one of these various things, you can set that up in their mobile app with a couple of clicks and a payment, and you're done. With AWS, you have a minimum of six different services you need to work with, many of which do not support anything on a mobile basis and don't talk to one another relatively well. I built a state machine out of step functions that will do a lot of it for me, but it's an example of having to touch so many different things just for a relatively straightforward solution space that is a common problem. And that's a small example, but you see it across the board.Matthew: Yeah, yeah. I was expecting you to come up with a little bit of a different answer for what a simple system is, for example, a website. Everyone likes to say, “Oh, a static website with just raw HTML. That's a simple”—Corey: No, that's hard as hell because the devil is in the details, and it slices you to ribbons whenever you go down that path.Matthew: Exactly.Corey: No, I'm talking things that a human being would do without needing to be an expert in getting that many different AWS services to talk to one another.Matthew: Yeah, and I agree that AWS traditionally is very bad at moving up that stack and getting those things to work. You had mentioned at the very top of this about Amplify. Amplify is a system that I have tried once or twice, and I generally think that, for the right use case, is an excellent system and I really like a lot of what it does.Corey: It is. I agree. Having gone down that, building up my scavenger hunt app that I'll be open-sourcing at some point next year.Matthew: Yeah. And it's fantastic, but it has a very steep cliff where you hit that point where all of a sudden, you go, “Okay, I added this, and I added this, and I added this, and now I want to add this one other thing, but to do it, now all of a sudden, I have to go through a tremendous amount of work.” It wasn't just the simple push button that the previous four steps were. Now, I have this one other thing that I need to do, and now it's a very difficult thing to incorporate into my system. And I'm having to learn all new stuff that I never had to care about before because Amplify made it way too easy.And I don't think this is necessarily an AWS problem. I think this is just a fundamentally difficult software problem to solve. Microsoft, I spent years and years in the Microsoft world, and this was my biggest complaint about Microsoft was that they made extremely difficult things, far too simple to solve. And then once those systems became either buggy, problematic, misconfigured, whatever you want to call it, once they stopped working for some reason, the people who were responsible for figuring those answers out didn't have the preceding knowledge because they didn't need it. And then all of a sudden, they go, “Well, I don't know how to solve this problem because I was told it was just this push-button thing.”So, Amplify is great, and I think it's fantastic, but it is a very, very difficult problem to solve. Amazon has proven to be very, very good at building the fundamentals, and I think that they function very well as a platform service, as a building blocks. But they give you the Lego pieces, and they expect you to build the very complex Batmobile. And they can maybe give you some custom pieces here and there, like the fenders, and the tires, and stuff like that, but that's not their bread and butter.Corey: Well, even starting with the CDK is a perfect example. Like, you can use the CDK init to create a new project from scratch, which is awesome. I love the fact that that exists, but it doesn't go far enough. It doesn't automatically create a repo you store the thing in that in turn hooks up to a CI/CD process that will wind up doing the build and deploy. Instead, it expects to do that all locally, which is a counter pattern. That's an anti-pattern. It'll lead you down the wrong path. And you always have to build these things from scratch yourself as you keep going. At least that's what it feels like.Matthew: Yeah, it is. And I think that here at Defiance Digital, our job as an MSP is to talk to the customer and figure out, but what are those very specific things you need? So, we do build new CDK repos all the time for our customers. But some of our customers want a trunk base system. Some of them want a branching or a development branch base system. Some of them have a very complex SDLC process within a PR stage of code changes versus a slightly less complex one after things have been merged into trunk.So, we fundamentally look at it like we're that bridge between the two, and in that case, AWS works great. In fact, all SaaS solutions are really nice because they give us those building blocks and then we provide value by figuring out which one of those we need to incorporate in for our clients. But every single one of our clients is very different. And we've only got, you know, less than a dozen right now. But you know, I've got project managers and directors always coming back to me and saying, “Well, how do we cookie-cutter this process?” And you can't do it. It's just very, very difficult.Not in a small-scale. Maybe when you're really big, and you're a company like AWS who has thousands, if not potentially millions of customers, you can find those patterns, but it is a very fundamentally difficult problem to solve, and we've seen multiple companies over the last two decades try to do these things and ultimately fail. So, I don't necessarily blame AWS for not having these things or not doing them well.Corey: Yes and no. I mean, GitHub delivers excellent experience for the user, start to finish. There's—Vercel does something very similar over in the front-end universe, too, where it is clearly possible, but it seems that designing user interfaces and integrating disparate things together is not an Amazon's DNA, which makes sense when you view the two-pizza teams assembling to build larger things. But man, is that a frustration.Matthew: Yeah. I really wonder if this two-pizza team mentality can ever work well for products that are bigger than just the fundamental concepts. I think Amplify is pretty good, but if you really want something that is this service that works for 80% of customers, you can't do it with five people. You can't do it with six. You need to have teams like what GitHub and what Vercel and other things, where teams are potentially dozens of people that really coordinate things and have a good project manager and product owner and understand the problem very well. And it's just very difficult with these very, very small teams to get that going.I don't know what the future of AWS looks like. It feels like a very Microsoft in the mid-2000s, which is, they're running off of their existing customers, they don't really have a need to innovate significantly because they have a lot of people locked in, they would be just fine for years on years on end with the products they have. So, there isn't a huge driver for doing it, not like, maybe, GCP or Azure really need to start to continue to innovate stronger in this space to pick up more customers. AWS doesn't have a problem getting customers.And if there isn't a significant change in the mentality, like what Microsoft saw at the end of the 2000s with getting rid of Ballmer, bringing in Satya and really changing the mentality inside the company, I don't see AWS breaking out from this anytime soon. But I think that's actually a good thing. I think AWS should stick to just building the fundamentals, and I think that they should rely on their partners and their third parties to bridge that gap. I think Jeremy Daly at Ampt and what they're building over there is a fantastic product.Corey: Yeah. The problem is that Amazon seems to be in denial about a lot of this, at least with what they're saying publicly.Matthew: Yeah, but what they say publicly and how they feel internally could be very, very different. I would say that, you know, we don't know what they're thinking internally. And that's fine. I don't necessarily need to. I think more specifically, we need to understand what their roadmap looks like and we need to understand, you know, what, are they going to change in the future to maybe fill in some of these gaps.I would say that the problem you said earlier about being able to do a simple website redirect, I don't think that's Amazon's desire to build those things. I think there should be a third-party that's built on top of AWS, and maybe even works directly within your AWS account as a marketplace product for doing that, but I don't think that's necessarily in the benefit of AWS to build that directly.Corey: We'll see. I'm very curious to see how this unfolds because a lot of customers want answers that require things that have to be assembled for them. I mean, honestly, a lot of the GenAI stuff is squarely in that category.Matthew: Agreed, but is this something where AWS needs to build it internally, and then we've got a product like App Composer, or Copilot, or things where they try, and then because they don't get enough traction, it just feels like they stall out and get stagnant? I mean, App Composer was a keynote product announcement during last year's re:Invent, and this year, we saw them introduce the ability to step function editing within it, and introduce the functionality into your IDE, VS Code directly. Both good things, but a year's worth of development effort to release those two features feels slow to me. The integration to VS Code should have been simple.Corey: Yeah. They are not the innovative company that would turn around and deliver something incredible three months after something had launched, “And here's a great new series of features around it.” It feels like the pace of innovation and face of delivery has massively slowed.Matthew: Yeah. And that's the scariest thing for me. And, you know, we saw this a little bit with a discussion recently in the cdk.dev server because if you take a look at what's been happening with the CDK application for the last six months and even almost a year now, it feels like the pace of changes within the codebase has slowed.There have been multiple releases over the course of the last year where the release at the end of the week—and they hit a pretty regular cadence of a release every week—that release at the end of the week fixes one bug or adds one small feature change to one construct in some library that maybe 10% of users are going to use. And that's troublesome. One of the main reasons why I ditched the Terraform and went hard on the CDK was that I looked at how many issues were open on the Terraform AWS provider, and how many missing features were, and how slow they were to incorporate those in, and said, “I can't invest another two years into this product if there isn't going to be that innovation.” And I wasn't in a place to do the development work myself—despite the fact that you can because it's open-source and providers are forkable—and the CDK is getting real close to that same spot right now. So, this weekend—and I know this is going to come out, you know, weeks later—but you know, the weekend of December 10th, they announced a change to the way that they were going to take contributions from the CDK community.And the long and short of it right now—and there's still some debate over exactly what they said—is, we're not going to accept brand-new L2 constructs from the community. Those have to be built internally by AWS only. That's a dr—step in the wrong direction. I understand why they're taking that approach. Contributions in the CDK have been very rough for the last four or five months because of the previous policies they put into place, but this is an open-source product. It's supposed to be an open-source product. It's also a very complex set of code because of all of the various AWS services that are being hit by it. This isn't just Amplify, which is hitting a couple of things here and there. This is potentially—Corey: It touches everything.Matthew: It touches everything.Corey: Yeah, I can see their perspective, but they've got to get way better at supporting things rapidly if they want to play that game.Matthew: And they can't do that internally with AWS, not with a two-pizza team.Corey: No. And there's an increasing philosophy I'm hearing from teams of, “Well, my service supports it. Other stuff, that's not my area of responsibility.” The wisdom that I've seen that really encapsulates this is written on Colm MacCárthaigh's old laptop in 2019: “AWS is the product.” That's the truth. It's not about the individual components; it's about the whole, collectively.Matthew: Right. And so, if we're not getting these L2 constructs and these things being built out for all of the services that CloudFormation hits, then the product feels stalled, there isn't a good initiative for users to continue trying to adopt it because over time, users are just going to hit more and more services in AWS, not fewer as they use the products. That's what AWS wants. They want people to be using VPC Lattice and all the GenAI stuff, and Glue, and SageMaker, and all these things, but if you don't have those L2 constructs, then there's no advantage of the CDK over top of just raw CloudFormation. So, the step in the right direction, in my opinion, would have been to make it easier and better for outside contributions to get into CDK, and they went the opposite way, and that's scary.Now, they basically said, go build these on your own, go publish them on the Construct Hub, and if they're good, we'll incorporate them in. But they also didn't define what good was, and what makes a good API. API development is very difficult. How do you build a construct that's going to hit 80% of use cases and still give you an out for those other 20 you missed? That's fundamentally hard.Corey: It is. And I don't know if there are good answers, yet. Maybe they're going in the right direction, maybe they're not.Matthew: Time will tell. My hope is that I can try to do some videos here after the new year to try to maybe make this a better experience for people. What does good API design look like? What is it like to implement these things well so they can be incorporated in? There has been a lot of pushback already, just after the first couple of days, from some very vocal users within the CDK community saying, “This is bad. This is fundamentally bad stuff.”Even from big fanboys like myself, who have supported the CDK, who co-authored the CDK Book, and they said, “This is not good.” So, we'll see what happens. Maybe they change direction after a couple of days. Maybe this is— turns out to be a great way to do it. Only time will really tell at this point.Corey: Awesome. And where can people go to find out more as you continue your exploration in this space and find out what you're up to in general?Matthew: So, I do have a Twitter account at@mattbonig on Twitter, however, I am probably going to be doing less and less over there. Engagement and the community as a whole over there has been problematic for a while, and I'll probably be doing more on LinkedIn, so you can find me there. Just search for Matthew Bonig. It's a very unique name.I've also got a website, matthewbonig.com, and from there, you can see blog articles, and a link to my Advanced CDK course, which I'm going to continue adding sessions to over the course of the next few months. I've got one coming out shortly about the deadly embrace and how you can work through that problem with the deadly embrace and hopefully not be so scared about multi-stack applications.Corey: I look forward to that because Lord knows, I'm running into that one myself increasingly frequently.Matthew: Well, good. I will hopefully be able to get this video out and solve all of your problems very easily.Corey: Awesome. Thank you so much for taking the time to speak with me. I appreciate it.Matthew: Thank you for having me. I really appreciate it.Corey: Matthew Bonig, Chief Cloud Architect at Defiance Digital, AWS Dev Tools Hero, and oh so much more. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with an angry comment that you will then have to wind up building the implementation for that constructs that power that comment yourself because apparently we're not allowed to build them globally anymore.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business, and we get to the point. Visit duckbillgroup.com to get started.
Ned Bellavance worked in the world of tech for more than a decade before joining the family profession as an educator. He joins Corey on Screaming in the Cloud to discuss his shift from engineer to educator and content creator, the intricacies of Terraform, and how changes in licensing affect the ecosystem.About NedNed is an IT professional with more than 20 years of experience in the field. He has been a helpdesk operator, systems administrator, cloud architect, and product manager. In 2019, Ned founded Ned in the Cloud LLC to work as an independent educator, creator, and consultant. In this new role, he develops courses for Pluralsight, runs multiple podcasts, writes books, and creates original content for technology vendors.Ned is a Microsoft MVP since 2017 and a HashiCorp Ambassador since 2020.Ned has three guiding principles: embrace discomfort, fail often, and be kind.Links Referenced: Ned in the Cloud: https://nedinthecloud.com/ LinkedIn: https://www.linkedin.com/in/ned-bellavance/ TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. My guest today is Ned Bellavance, who's the founder and curious human over at Ned in the Cloud. Ned, thank you for joining me.Ned: Yeah, it's a pleasure to be here, Corey.Corey: So, what is Ned in the Cloud? There are a bunch of easy answers that I feel don't give the complete story like, “Oh, it's a YouTube channel,” or, “Oh no, it's the name that you wound up using because of, I don't know, easier to spell the URL or something.” Where do you start? Where do you stop? What are you exactly?Ned: What am I? Wow, I didn't know we were going to get this deep into philosophical territory this early. I mean, you got to ease me in with something. But so, Ned in the Cloud is the name of my blog from back in the days when we all started up a blog and hosted on WordPress and had fun. And then I was also at the same time working for a value-added reseller as a consultant, so a lot of what went on my blog was stuff that happened to me in the world of consulting.And you're always dealing with different levels of brokenness when you go to clients, so you see some interesting things, and I blogged about them. At a certain point, I decided I want to go out and do my own thing, mostly focused on training and education and content creation and I was looking for a company name. And I went through—I had a list of about 40 different names. And I showed them to my wife, and she's like, “Why don't you go Ned in the Cloud? Why are you making this more complicated than it needs to be?”And I said, “Well, I'm an engineer. That is my job, by definition, but you're probably right. I should just go with Ned in the Cloud.” So, Ned in the Cloud now is a company, just me, focused on creating educational content for technical learners on a variety of different platforms. And if I'm delivering educational content, I am a happy human, and if I'm not doing that, I'm probably out running somewhere.Corey: I like that, and I'd like to focus on education first. There are a number of reasons that people will go in that particular direction, but what was it for you?Ned: I think it's kind of in the heritage of my family. It's in my blood to a certain degree because my dad is a teacher, my mom is a teacher-turned-librarian, my sister is a teacher, my wife is a teacher, her mother is a teacher. So, there was definitely something in the air, and I think at a certain point, I was the black sheep in the sense that I was the engineer. Look, this guy over here. And then I ended up deciding that I really liked training people and learning and teaching, and became a teacher of sorts, and then they all went, “Welcome to the fold.”Corey: It's fun when you get to talk to people about the things that they're learning because when someone's learning something I find that it's the time when their mind is the most open. I don't think that that's something that you don't get to see nearly as much once someone already, quote-unquote, “Knows a thing,” because once that happens, why would you go back and learn something new? I have always learned the most—even about things that I've built myself—by putting it in the hands of users and seeing how they honestly sometimes hold it wrong and make mistakes that don't make sense to me, but absolutely make sense to them. Learning something—or rather, teaching something—versus building that thing is very much an orthogonal skill set, and I don't think that there's enough respect given to that understanding.Ned: It's an interesting sphere of people who can both build the thing and then teach somebody else to build the thing because you're right, it's very different skill sets. Being able to teach means that you have to empathize with the human being that you're teaching and understand that their perspective is not yours necessarily. And one of the skills that you build up as an instructor is realizing when you're making a whole bunch of assumptions because you know something really well, and that the person that you're teaching is not going to have that context, they're not going to have all those assumptions baked in, so you have to actually explain that stuff out. Some of my instruction has been purely online video courses through, like, Pluralsight; less of a feedback loop there. I have to publish the entire course, and then I started getting feedback, so I really enjoy doing live trainings as well because then I get the questions right away.And I always insist, like, if I'm delivering a lecture, and you have a question, please don't wait for the end. Please interrupt me immediately because you're going to forget what that question is, you're going to lose your train of thought, and then you're not going to ask it. And the whole class benefits when someone asks a question, and I benefit too. I learn how to explain that concept better. So, I really enjoy the live setting, but making the video courses is kind of nice, too.Corey: I learned to speak publicly and give conference talks as a traveling contract trainer for Puppet years ago, and that was an eye-opening experience, just because you don't really understand something until you're teaching other people how it works. It's how I learned Git. I gave a conference talk that explained Git to people, and that was called a forcing function because I had four months to go to learn this thing I did not fully understand and welp, they're not going to move the conference for me, so I guess I'd better hustle. I wouldn't necessarily recommend that approach. These days, it seems like you have a, let's say, disproportionate level of focus on the area of Infrastructure as Code, specifically you seem to be aiming at Terraform. Is that an accurate way of describing it?Ned: That is a very accurate way of describing it. I discovered Terraform while I was doing my consulting back in 2016 era, so this was pretty early on in the product's lifecycle. But I had been using CloudFormation, and at that time, CloudFormation only supported JSON, which meant it was extra punishing. And being able to describe something more succinctly and also have access to all these functions and loops and variables, I was like, “This is amazing. Where were you a year ago?” And so, I really just jumped in with both feet into Terraform.And at a certain point, I was at a conference, and I went past the Pluralsight booth, and they mentioned that they were looking for instructors. And I thought to myself, well, I like talking about things, and I'm pretty excited about this Terraform thing. Why don't I see if they're looking for someone to do a Terraform course? And so, I went through their audition process and sure enough, that is exactly what they were looking for. They had no getting started course for Terraform at the time. I published the course in 2017, and it has been in the top 50 courses ever since on Pluralsight. So, that told me that there's definitely an appetite and maybe this is an area I should focus on a little bit more.Corey: It's a difficult area to learn. About two months ago, I started using Terraform for the first time in anger in ages. I mean, I first discovered it when I was on my way back from one of those Puppet trainings, and the person next to me was really excited about this thing that we're about to launch. Turns out that was Mitchell Hashimoto and Armon was sitting next to him on the other side. Why he had a middle seat, I'll never know.But it was a really fun conversation, just talking about how he saw the world and what he was planning on doing. And a lot of that vision was realized. What I figured out a couple months ago is both that first, I'm sort of sad that Terraform is as bad as it is, but it's the best option we've got because everything else is so much worse. It is omnipresent, though. Effectively, every client I've ever dealt with on AWS billing who has a substantial estate is managing it via Terraform.It is the lingua franca of cloud across the board. I just wish it didn't require as much care and feeding, especially for the getting-started-with-a-boilerplate type of scenario. So, much of what you type feels like it's useless stuff that should be implicit. I understand why it's not, but it feels that way. It's hard to learn.Ned: It certainly can be. And you're right, there's a certain amount of boilerplate and [sigh] code that you have to write that seems pointless. Like, do I have to actually spell this all out? And sometimes the answer is yes, and sometimes the answer is you should use a module for that. Why are you writing this entire VPC configuration out yourself? And that's the sort of thing that you learn over time is that there are shortcuts, there are ways to make the code simpler and require less care and feeding.But I think ultimately, your infrastructure, just like your software, evolves, changes has new requirements, and you need to manage it in the same way that you want to manage your software. And I wouldn't tell a software developer, “Oh, you know, you could just write it once and never go back to it. I'm sure it's fine.” And by the same token, I wouldn't tell an infrastructure developer the same thing. Now, of course, people do that and never go back and touch it, and then somebody else inherits that infrastructure and goes, “Oh, God. Where's the state data?” And no one knows, and then you're starting from scratch. But hopefully, if you have someone who's doing it responsibly, they'll be setting up Terraform in such a way that it is maintainable by somebody else.Corey: I'd sure like to hope so. I have encountered so many horrible examples of code and wondering what malicious person wrote this. And of course, it was me, 6 or 12 months ago.Ned: Always [laugh].Corey: I get to play architect around a lot of these things. In fact, that's one of the problems that I've had historically with an awful lot of different things that I've basically built, called it feature complete, let it sit for a while using the CDK or whatnot, and then oh, I want to make a small change to it. Well, first, I got to spend half a day during the entire line dependency updates and seeing what's broken and how all of that works. It feels like for better or worse, Terraform is a lot more stable than that, as in, old versions of Terraform code from blog posts from 2016 will still effectively work. Is that accurate? I haven't done enough exploring in that direction to be certain.Ned: The good thing about Terraform is you can pin the version of various things that you're using. So, if you're using a particular version of the AWS provider, you can pin it to that specific version, and it won't automatically upgrade you to the latest and greatest. If you didn't do that, then you'll get bit by the update bug, which certainly happens to some folks when they changed the provider from version 3 to version 4 and completely changed how the S3 bucket object was created. A lot of people's scripts broke that day, so I think that was the time for everyone to learn what the version argument is and how it works. But yeah, as long as you follow that general convention of pinning versions of your modules and of your resource provider, you should be in a pretty stable place when you want to update it.Corey: Well, here's the $64,000 question for you, then. Does Dependabot on your GitHub repo begin screaming at you as soon as you've done that because in one of its dependencies in some particular weird edge cases when they're dealing with unsanitized, internet-based input could wind up taking up too many system resources, for example? Which is, I guess, in an ideal world, it wouldn't be an issue, but in practice, my infrastructure team is probably not trying to attack the company from the inside. They have better paths to get there, to be very blunt.Ned: [laugh].Corey: Turns out giving someone access to a thing just directly is way easier than making them find it. But that's been one of the frustrating parts where, especially when it encounters things like, I don't know, corporate security policies of, “Oh, you must clear all of these warnings,” which well-intentioned, poorly executed seems to be the takeaway there.Ned: Yeah, I've certainly seen some implementations of tools that do static scanning of Terraform code and will come up with vulnerabilities or violations of best practice, then you have to put exceptions in there. And sometimes it'll be something like, “You shouldn't have your S3 bucket public,” which in most cases, you shouldn't, but then there's the one team that's actually publishing a front-facing static website in the S3 bucket, and then they have to get, you know, special permission from on high to ignore that warning. So, a lot of those best practices that are in the scanning tools are there for very good reasons and when you onboard them, you should be ready to see a sea of red in your scan the first time and then look through that and kind of pick through what's actually real, and we should improve in our code, and what's something that we can safely ignore because we are intentionally doing it that way.Corey: I feel like there's an awful lot of… how to put this politely… implicit dependencies that are built into things. I'll wind up figuring out how to do something by implementing it and that means I will stitch together an awful lot of blog posts, things I found on Stack Overflow, et cetera, just like a senior engineer and also Chat-Gippity will go ahead and do those things. And then the reason—like, someone asks me four years later, “Why is that thing there?” And… “Well, I don't know, but if I remove it, it might stop working, so…” there was almost a cargo-culting style of, well, it's always been there. So, is that necessary? Is it not?I'm ashamed by how often I learned something very fundamental in a system that I've been using for 20 years—namely, the command line—just by reading the man page for a command that I already, quote-unquote, “Already know how to use perfectly well.” Yeah, there's a lot of hidden gems buried in those things.Ned: Oh, my goodness, I learned something about the Terraform CLI last week that I wish I'd known two years ago. And it's been there for a long time. It's like, when you want to validate your code with the terraform validate, you can initialize without initializing the back-end, and for those who are steeped in Terraform, that means something and for everybody else, I'm sorry [laugh]. But I discovered that was an option, and I was like, “Ahhh, this is amazing.” But to get back to the sort of dependency problems and understanding your infrastructure better—because I think that's ultimately what's happening when you have to describe something using Infrastructure as Code—is you discover how the infrastructure actually works versus how you thought it worked.If you look at how—and I'm going to go into Azure-land here, so try to follow along with me—if you go into Azure-land and you look at how they construct a load balancer, the load balancer is not a single resource. It's about eight different resources that are all tied together. And AWS has something similar with how you have target groups, and you have the load balancer component and the listener and the health check and all that. Azure has the same thing. There's no actual load balancer object, per se.There's a bunch of different components that get slammed together to form that load balancer. When you look in the portal, you don't see any of that. You just see a load balancer, and you might think this is a very simple resource to configure. When it actually comes time to break it out into code, you realize, oh, this is eight different components, each of which has its own options and arguments that I need to understand. So, one of the great things that I have seen a lot of tooling up here around is doing the import of existing infrastructure into Terraform by pointing the tool at a collection of resources—whatever they are—and saying, “Go create the Terraform code that matches that thing.” And it's not going to be the most elegant code out there, but it will give you a baseline for what all the settings actually are, and other resource types are, and then you can tweak it as needed to add in input variables or remove some arguments that you're not using.Corey: Yeah, I remember when they first announced the importing of existing state. It's wow, there's an awful lot of stuff that it can be aware of that I will absolutely need to control unless I want it to start blowing stuff away every time I run the—[unintelligible 00:15:51] supposedly [unintelligible 00:15:52] thing against it. And that wasn't a lot of fun. But yeah, this is the common experience of it. I only recently was reminded of the fact that I once knew, and I'd forgotten that a public versus private subnet in AWS is a human-based abstraction, not something that is implicit to the API or the way they envision subnets existing. Kind of nice, but also weird when you have to unlearn things that you've thought you'd learned.Ned: That's a really interesting example of we think of them as very different things, and when we draw nice architecture diagrams there—these are the private subnets and these are the public ones. And when you actually go to create one using Terraform—or really another tool—there's no box that says ‘private' or ‘make this public.' It's just what does your route table look like? Are you sending that traffic out the internet gateway or are you sending it to some sort of NAT device? And how does traffic come back into that subnet? That's it. That's what makes it private versus public versus a database subnet versus any other subnet type you want to logically assign within AWS.Corey: Yeah. It's kind of fun when that stuff hits.Ned: [laugh].Corey: I am curious, as you look across the ecosystem, do you still see that learning Terraform is a primary pain point for, I guess, the modern era of cloud engineer, or has that sunk below the surface level of awareness in some ways?Ned: I think it's taken as a given to a certain degree that if you're a cloud engineer or an aspiring cloud engineer today, one of the things you're going to learn is Infrastructure as Code, and that Infrastructure as Code is probably going to be Terraform. You can still learn—there's a bunch of other tools out there; I'm not going to pretend like Terraform is the end-all be-all, right? We've got—if you want to use a general purpose programming language, you have something like Pulumi out there that will allow you to do that. If you want to use one of the cloud-native tools, you've got something like CloudFormation or Azure has Bicep. Please don't use ARM templates because they hurt. They're still JSON only, so at least CloudFormation added YAML support in there. And while I don't really like YAML, at least it's not 10,000 lines of code to spin up, like, two domain controllers in a subnet.Corey: I personally wind up resolving the dichotomy between oh, should we go with JSON or should we go with YAML by picking the third option everyone hates more. That's why I'm a staunch advocate for XML.Ned: [laugh]. I was going to say XML. Yeah oh, as someone who dealt with SOAP stuff for a while, yeah, XML was particularly painful, so I'm not sad that went away. JSON for me, I work with it better, but YAML is more readable. So, it's like it's, pick your poison on that. But yeah, there's a ton of infrastructure tools out there.They all have basically the same concepts behind them, the same core concepts because they're all deploying the same thing at the end of the day and there's only so many ways you can express that concept. So, once you learn one—say you learned CloudFormation first—then Terraform is not as big of a leap. You're still declaring stuff within a file and then having it go and make those things exist. It's just nuances between the implementation of Terraform versus CloudFormation versus Bicep.Corey: I wish that there were more straightforward abstractions, but I think that as soon as you get those, that inherently limits what you're able to do, so I don't know how you square that circle.Ned: That's been a real difficult thing is, people want some sort of universal cloud or infrastructure language and abstraction. I just want a virtual machine. I don't care what kind of platform I'm on. Just give me a VM. But then you end up very much caring [laugh] what kind of VM, what operating system, what the underlying hardware is when you get to a certain level.So, there are some workloads where you're like, I just needed to run somewhere in a container and I really don't care about any of the underlying stuff. And that's great. That's what Platform as a Service is for. If that's your end goal, go use that. But if you're actually standing up infrastructure for any sort of enterprise company, then you need an abstraction that gives you access to all the underlying bits when you want them.So, if I want to specify different placement groups about my VM, I need access to that setting to create a placement group. And if I have this high-level of abstraction of a virtual machine, it doesn't know what a placement group is, and now I'm stuck at that level of abstraction instead of getting down to the guts, or I'm going into the portal or the CLI and modifying it outside of the tool that I'm supposed to be using.Corey: I want to change gears slightly here. One thing that has really been roiling some very particular people with very specific perspectives has been the BSL license change that Terraform has wound up rolling out. So far, the people that I've heard who have the strongest opinions on it tend to fall into one of three categories: either they work at HashiCorp—fair enough, they work at one of HashiCorp's direct competitors—which yeah, okay, sure, or they tend to be—how to put this delicately—open-source evangelists, of which I freely admit I used to be one and then had other challenges I needed to chase down in other ways. So, I'm curious as to where you, who are not really on the vendor side of this at all, how do you see it shaking out?Ned: Well, I mean, just for some context, essentially what HashiCorp decided to do was to change the licensing from Mozilla Public licensing to BSL for, I think eight of their products and Terraform was amongst those. And really, this sort of tells you where people are. The only one that anybody really made any noise about was Terraform. There's plenty of people that use Vault, but I didn't see a big brouhaha over the fact that Vault changed its licensing. It's really just about Terraform. Which tells you how important it is to the ecosystem.And if I look at the folks that are making the most noise about it, it's like you said, they basically fall into one of two camps: it's the open-source code purists who believe everything should be licensed in completely open-source ways, or at least if you start out with an open-source license, you can't convert to something else later. And then there is a smaller subset of folks who work for HashiCorp competitors, and they really don't like the idea of having to pay HashiCorp a regular fee for what used to be ostensibly free to them to use. And so, what they ended up doing was creating a fork of Terraform, just before the licensing change happened and that fork of Terraform was originally called OpenTF, and they had an OpenTF manifesto. And I don't know about you, when I see the word ‘manifesto,' I back away slowly and try not to make any sudden moves.Corey: You really get the sense there's going to be a body count tied to this. And people are like, “What about the Agile Manifesto?” “Yeah, what about it?”Ned: [laugh]. Yeah, I'm just—when I see ‘manifesto,' I get a little bit nervous because either someone is so incredibly passionate about something that they've kind of gone off the deep end a little bit, or they're being somewhat duplicitous, and they have ulterior motives, let's say. Now, I'm not trying to cast aspersions on anybody. I can't read anybody's mind and tell you exactly what their intention was behind it. I just know that the manifesto reads a little bit like an open-source purist and a little bit like someone having a temper tantrum, and vacillating between the two.But cooler heads prevailed a little bit, and now they have changed the name to OpenTofu, and it has been accepted by the Linux Foundation as a project. So, it's now a member of the Linux Foundation, with all the gravitas that that comes with. And some people at HashiCorp aren't necessarily happy about the Linux Foundation choosing to pull that in.Corey: Yeah, I saw a whole screed, effectively, that their CEO wound up brain-dumping on that frankly, from a messaging perspective, he would have been better served as not to say anything at all, to be very honest with you.Ned: Yeah, that was a bit of a yikes moment for me.Corey: It's very rare that you will listen yourself into trouble as opposed to opening your mouth and getting yourself into trouble.Ned: Exactly.Corey: You wouldn't think I would be one of those—of all people who would have made that observation, you wouldn't think I would be on that list, yet here I am.Ned: Yeah. And I don't think either side is entirely blameless. I understand the motivations behind HashiCorp wanting to make the change. I mean, they're a publicly traded company now and ostensibly that means that they should be making some amount of money for their investors, so they do have to bear that in mind. I don't necessarily think that changing the licensing of Terraform is the way to make that money.I think in the long-term, it's not going—it may not hurt them a lot, but I don't think it's going to help them out a lot, and it's tainted the goodwill of the community to a certain degree. On the other hand, I don't entirely trust what the other businesses are saying as well in their stead. So, there's nobody in this that comes out a hundred percent clean [laugh] on the whole process.Corey: Yeah, I feel like, to be direct, the direct competitors to HashiCorp along its various axes are not the best actors necessarily to complain about what is their largest competitor no longer giving them access to continue to compete against them with their own product. I understand the nuances there, but it also doesn't feel like they are the best ambassadors for that. I also definitely understand where HashiCorp is coming from where, why are we investing all this time, energy, and effort for people to basically take revenue away from us? But there's also the bigger problem, which is, by and large, compared to how many sites are running Terraform and the revenues that HashiCorp puts up for it, they're clearly failing to capture the value they have delivered in a massive way. But counterpoint, if they hadn't been open-source for their life until this point, would they have ever captured that market share? Probably not.Ned: Yeah, I think ultimately, the biggest competitor to their paid offering of Terraform is their free version of Terraform. It literally has enough bells and whistles already included and plenty of options for automating those things and solving the problems that their enterprise product solves that their biggest problem is not other competitors in the Terraform landscape; it's the, “Well, we already have something, and it's good enough.” And I'm not sure how you sell to that person, that's why I'm not in marketing, but I think that is their biggest competitor is the people who already have a solution and are like, “Why do I need to pay for your thing when my thing works well enough?”Corey: That's part of the strange thing that I'm seeing as I look across this entire landscape is it feels like this is not something that is directly going to impact almost anyone out there who's just using this stuff, either the open-source version as a paying customer of any of these things, but it is going to kick up a bunch of dust. And speaking of poor messaging, HashiCorp is not really killing it this quarter, where the initial announcement led to so many questions that were unclear, such as—like, they fixed this later in the frequently asked questions list, but okay, “I'm using Terraform right now and that's fine. I'm building something else completely different. Am I going to lose my access to Terraform if you decide to launch a feature that does what my company does?” And after a couple of days, they put up an indemnity against that. Okay, fine.Like, when Mongo did this, there was a similar type of dynamic that was emerging, but a lot fewer people are writing their own database engine to then sell onward to customers that are provisioning infrastructure on behalf of their customers. And where the boundaries lay for who was considered a direct Terraform competitor was unclear. I'm still not convinced that it is clear enough to bet the business on for a lot of these folks. It comes down to say what you mean, not—instead of hedging, you're not helping your cause any.Ned: Yeah, I think out of the different products that they have, some are very clear-cut. Like, Vault is a server that runs as a service, and so that's very clear what that product is and where the lines of delineation are around Vault. If I go stand up a bunch of Vault servers and offer them as a service, then that is clearly a competitor. But if I have an automation pipeline service and people can technically automate Terraform deployments with my service, even if that's not the core thing that I'm looking to do, am I now a competitor? Like, it's such a fuzzy line because Terraform isn't an application, it's not a server that runs somewhere, it's a CLI tool and a programming language. So yeah, those lines are very, very fuzzy. And I… like I said, it would be better if they say what they meant, as opposed to sort of the mealy-mouthed language that they ended up using and the need to publish multiple revisions of that FAQ to clarify their position on very specific niche use cases.Corey: Yeah, I'm not trying to be difficult or insulting or anything like that. These are hard problems that everyone involved is wrestling with. It just felt a little off, and I think the messaging did them no favors when that wound up hitting. And now, everyone is sort of trying to read the tea leaves and figure out what does this mean because in isolation, it doesn't mean anything. It is a forward-looking thing.Whatever it is you're doing today, no changes are needed for you, until the next version comes out, in which case, okay, now do we incorporate the new thing or don't we? Today, to my understanding, whether I'm running Terraform or OpenTofu entirely comes down to which binary am I invoking to do the apply? There is no difference of which I am aware. That will, of course, change, but today, I don't have to think about that.Ned: Right. OpenTofu is a literal fork of Terraform, and they haven't really added much in the way of features, so it should be completely compatible with Terraform. The two will diverge in the future as feature as new features get added to each one. But yeah, for folks who are using it today, they might just decide to stay on the version pre-fork and stay on that for years. I think HashiCorp has pledged 18 months of support for any minor version of Terraform, so you've got at least a year-and-a-half to decide. And we were kind of talking before the recording, 99% of people using Terraform do not care about this. It does not impact their daily workflow.Corey: No. I don't see customers caring at all. And also, “Oh, we're only going to use the pre-fork version of Terraform,” they're like, “Thanks for the air cover because we haven't updated any of that stuff in five years, so tha”—Ned: [laugh].Corey: “Oh yeah, we're doing it out of license concern. That's it. That's the reason we haven't done anything recent with it.” Because once it's working, changes are scary.Ned: Yeah.Corey: Terraform is one of those scary things, right next to databases, that if I make a change that I don't fully understand—and no one understands everything, as we've covered—then this could really ruin my week. So, I'm going to be very cautious around that.Ned: Yeah, if metrics are to be believed across the automation platforms, once an infrastructure rollout happens with a particular version of Terraform, that version does not get updated. For years. So, I have it on good authority that there's still Terraform version 0.10 and 0.11 running on these automation platforms for really old builds where people are too scared to upgrade to, like, post 0.12 where everything changed in the language.I believe that. People don't want to change it, especially if it's working. And so, for most people, this licensing chain doesn't matter. And all the constant back and forth and bickering just makes people feel a little nervous, and it might end up pushing people away from Terraform as a platform entirely, as opposed to picking a side.Corey: Yeah, and I think that that is probably the fair way to view it at this point where right now—please, friends at HashiCorp and HashiCorp competitors don't yell at me for this—it's basically a nerd slap-fight at the moment.Ned: [laugh].Corey: And of one of the big reasons that I also stay out of these debates almost entirely is that I married a corporate attorney who used to be a litigator and I get frustrated whenever it comes down to license arguments because you see suddenly a bunch of engineers who get to cosplay as lawyers, and reading the comments is infuriating once you realize how a little bit of this stuff works, which I've had 15 years of osmotic learning on this stuff. Whenever I want to upset my wife, I just read some of these comments aloud and then our dinner conversation becomes screaming. It's wonderful.Ned: Bad legal takes? Yeah, before—Corey: Exactly.Ned: Before my father became a social studies teacher, he was a lawyer for 20 years, and so I got to absorb some of the thought process of the lawyer. And yeah, I read some of these takes, and I'm like, “That doesn't sound right. I don't think that would hold up in any court of law.” Though a lot of the open-source licensing I don't think has been tested in any sort of court of law. It's just kind of like, “Well, we hope this stands up,” but nobody really has the money to check.Corey: Yeah. This is the problem with these open-source licenses as well. Very few have never been tested in any meaningful way because I don't know about you, but I don't have a few million dollars in legal fees lying around to prove the point.Ned: Yeah.Corey: So, it's one of those we think this is sustainable, and Lord knows the number of companies that have taken reliances on these licenses, they're probably right. I'm certainly not going to disprove the fact—please don't sue me—but yeah, this is one of those things that we're sort of assuming is the case, even if it's potentially not. I really want to thank you for taking the time to discuss how it is you view these things and talk about what it is you're up to. If people want to learn more, where's the best place for them to find you?Ned: Honestly, just go to my website. It's nedinthecloud.com. And you can also find me on LinkedIn. I don't really go for Twitter anymore.Corey: I envy you. I wish I could wean myself off of it. But we will, of course, include a link to that in the show notes. Thank you so much for being so generous with your time. It's appreciated.Ned: It's been a pleasure. Thanks, Corey.Corey: Net Bellavance, founder and curious human at Ned in the Cloud. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, along with an angry comment that I will then fork under a different license and claim as my own.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business, and we get to the point. Visit duckbillgroup.com to get started.
Laurent Doguin, Director of Developer Relations & Strategy at Couchbase, joins Corey on Screaming in the Cloud to talk about the work that Couchbase is doing in the world of databases and developer relations, as well as the role of AI in their industry and beyond. Together, Corey and Laurent discuss Laurent's many different roles throughout his career including what made him want to come back to a role at Couchbase after stepping away for 5 years. Corey and Laurent dig deep on how Couchbase has grown in recent years and how it's using artificial intelligence to offer an even better experience to the end user.About LaurentLaurent Doguin is Director of Developer Relations & Strategy at Couchbase (NASDAQ: BASE), a cloud database platform company that 30% of the Fortune 100 depend on.Links Referenced: Couchbase: https://couchbase.com XKCD #927: https://xkcd.com/927/ dbdb.io: https://dbdb.io DB-Engines: https://db-engines.com/en/ Twitter: https://twitter.com/ldoguin LinkedIn: https://www.linkedin.com/in/ldoguin/ TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Are you navigating the complex web of API management, microservices, and Kubernetes in your organization? Solo.io is here to be your guide to connectivity in the cloud-native universe!Solo.io, the powerhouse behind Istio, is revolutionizing cloud-native application networking. They brought you Gloo Gateway, the lightweight and ultra-fast gateway built for modern API management, and Gloo Mesh Core, a necessary step to secure, support, and operate your Istio environment.Why struggle with the nuts and bolts of infrastructure when you can focus on what truly matters - your application. Solo.io's got your back with networking for applications, not infrastructure. Embrace zero trust security, GitOps automation, and seamless multi-cloud networking, all with Solo.io.And here's the real game-changer: a common interface for every connection, in every direction, all with one API. It's the future of connectivity, and it's called Gloo by Solo.io.DevOps and Platform Engineers, your journey to a seamless cloud-native experience starts here. Visit solo.io/screaminginthecloud today and level up your networking game.Corey: Welcome to Screaming in the Cloud, I'm Corey Quinn. This promoted guest episode is brought to us by our friends at Couchbase. And before we start talking about Couchbase, I would rather talk about not being at Couchbase. Laurent Doguin is the Director of Developer Relations and Strategy at Couchbase. First, Laurent, thank you for joining me.Laurent: Thanks for having me. It's a pleasure to be here.Corey: So, what I find interesting is that this is your second time at Couchbase, where you were a developer advocate there for a couple of years, then you had five years of, we'll call it wilderness I suppose, and then you return to be the Director of Developer Relations. Which also ties into my personal working thesis of, the best way to get promoted at a lot of companies is to leave and then come back. But what caused you to decide, all right, I'm going to go work somewhere else? And what made you come back?Laurent: So, I've joined Couchbase in 2014. Spent about two or three years as a DA. And during those three years as a developer advocate, I've been advocating SQL database and I—at the time, it was mostly DBAs and ops I was talking to. And DBA and ops are, well, recent, modern ops are writing code, but they were not the people I wanted to talk to you when I was a developer advocate. I came from a background of developer, I've been a platform engineer for an enterprise content management company. I was writing code all day.And when I came to Couchbase, I realized I was mostly talking about Docker and Kubernetes, which is still cool, but not what I wanted to do. I wanted to talk about developers, how they use database to be better app, how they use key-value, and those weird thing like MapReduce. At the time, MapReduce was still, like, a weird thing for a lot of people, and probably still is because now everybody's doing SQL. So, that's what I wanted to talk about. I wanted to… engage with people identify with, really. And so, didn't happen. Left. Built a Platform as a Service company called Clever Cloud. They started about four or five years before I joined. We went from seven people to thirty-one LFs, fully bootstrapped, no VC. That's an interesting way to build a company in this age.Corey: Very hard to do because it takes a lot of upfront investment to build software, but you can sort of subsidize that via services, which is what we've done here in some respects. But yeah, that's a hard road to walk.Laurent: That's the model we had—and especially when your competition is AWS or Azure or GCP, so that was interesting. So entrepreneurship, it's not for everyone. I did my four years there and then I realized, maybe I'm going to do something else. I met my former colleagues of Couchbase at a software conference called Devoxx, in France, and they told me, “Well, there's a new sheriff in town. You should come back and talk to us. It's all about developers, we are repositioning, rehandling the way we do marketing at Couchbase. Why not have a conversation with our new CMO, John Kreisa?”And I said, “Well, I mean, I don't have anything to do. I actually built a brewery during that past year with some friends. That was great, but that's not going to feed me or anything. So yeah, let's have a conversation about work.” And so, I talked to John, I talked to a bunch of other people, and I realized [unintelligible 00:03:51], he actually changed, like, there was a—they were purposely going [against 00:03:55] developer, talking to developer. And that was not the case, necessarily, five, six years before that.So, that's why I came back. The product is still amazing, the people are still amazing. It was interesting to find a lot of people that still work there after, what, five years. And it's a company based in… California, headquartered in California, so you would expect people to, you know, jump around a bit. And I was pleasantly surprised to find the same folks there. So, that was also one of the reasons why I came back.Corey: It's always a strong endorsement when former employees rejoin a company. Because, I don't know about you, but I've always been aware of those companies you work for, you leave. Like, “Aw, I'm never doing that again for love or money,” just because it was such an unpleasant experience. So, it speaks well when you see companies that do have a culture of boomerangs, for lack of a better term.Laurent: That's the one we use internally, and there's a couple. More than a couple.Corey: So, one thing that seems to have been a thread through most of your career has been an emphasis on developer experience. And I don't know if we come at it from the same perspective, but to me, what drives nuts is honestly, with my work in cloud, bad developer experience manifests as the developer in question feeling like they're somehow not very good at their job. Like, they're somehow not understanding how all this stuff is supposed to work, and honestly, it leads to feeling like a giant fraud. And I find that it's pernicious because even when I intellectually know for a fact that I'm not the dumbest person ever to use this tool when I don't understand how something works, the bad developer experience manifests to me as, “You're not good enough.” At least, that's where I come at it from.Laurent: And also, I [unintelligible 00:05:34] to people that build these products because if we build the products, the user might be in the same position that we are right now. And so, we might be responsible for that experience [unintelligible 00:05:43] a developer, and that's not a great feeling. So, I completely agree with you. I've tried to… always on software-focused companies, whether it was Nuxeo, Couchbase, Clever Cloud, and then Couchbase. And I guess one of the good thing about coming back to a developer-focused era is all the product alignments.Like, a lot of people talk about product that [grows 00:06:08] and what it means. To me what it means was, what it meant—what it still means—building a product that developer wants to use, and not just want to, sometimes it's imposed to you, but actually are happy to use, and as you said, don't feel completely stupid about it in front of the product. It goes through different things. We've recently revamped our Couchbase UI, Couchbase Capella UI—Couchbase Capella is a managed cloud product—and so we've added a lot of in-product getting started guidelines, snippets of code, to help developers getting started better and not have that feeling of, “What am I doing? Why is it not working and what's going on?”Corey: That's an interesting decision to make, just because historically, working with a bunch of tools, the folks who are building the documentation working with that tool, tend to generally be experts at it, so they tend to optimize for improving things for the experience of someone has been using it for five years as opposed to the newcomer. So, I find that the longer a product is in existence, in many cases, the worse the new user experience becomes because companies tend to grow and sprawl in different ways, the product does likewise. And if you don't know the history behind it, “Oh, your company, what does it do?” And you look at the website and there's 50 different offerings that you have—like, the AWS landing page—it becomes overwhelming very quickly. So, it's neat to see that emphasis throughout the user interface on the new developer experience.On the other side of it, though, how are the folks who've been using it for a while respond to those changes? Because it's frustrating for me at least, when I log into a new account, which happens periodically within AWS land, and I have this giant series of onboarding pop-ups that I have to click to make go away every single time. How are they responding to it?Laurent: Yeah, it's interesting. One of the first things that struck me when I joined Couchbase the first time was the size of the technical documentation team. Because the whole… well, not the whole point, but part of the reason why they exist is to do that, to make sure that you understand all the differences and that it doesn't feel like the [unintelligible 00:08:18] what the documentation or the product pitch or everything. Like, they really, really, really emphasize on this from the very beginning. So, that was interesting.So, when you get that culture built into the products, well, the good thing is… when people try Couchbase, they usually stick with Couchbase. My main issue as a Director of the Developer Relations is not to make people stick with Couchbase because that works fairly well with the product that we have; it's to make them aware that we exist. That's the biggest issue I have. So, my goal as DevRel is to make sure that people get the trial, get through the trial, get all that in-app context, all that helps, get that first sample going, get that first… I'm not going to say product built because that's even a bit further down the line, but you know, get that sample going. We have a code playground, so when you're in the application, you get to actually execute different pieces of code, different languages. And so, we get those numbers and we're happy to see that people actually try that. And that's a, well, that's a good feeling.Corey: I think that there's a definite lack of awareness almost industry-wide around the fact that as the diversity of your customers increases, you have to have different approaches that meet them at various points along the journey. Because things that I've seen are okay, it's easy to ass—even just assuming a binary of, “Okay, I've done this before a thousand times; this is the thousand and first, I don't need the Hello World tutorial,” versus, “Oh, I have no idea what I'm doing. Give me the Hello World tutorial,” there are other points along that continuum, such as, “Oh, I used to do something like this, but it's been three years. Can you give me a refresher,” and so on. I think that there's a desire to try and fit every new user into a predefined persona and that just doesn't work very well as products become more sophisticated.Laurent: It's interesting, we actually have—we went through that work of defining those personas because there are many. And that was the origin of my departure. I had one person, ops slash DBA slash the person that maintain this thing, and I wanted to talk to all the other people that built the application space in Couchbase. So, we broadly segment things into back-end, full-stack, and mobile because Couchbase is also a mobile database. Well, we haven't talked too much about this, so I can explain you quickly what Couchbase is.It's basically a distributed JSON database with an integrated caching layer, so it's reasonably fast. So it does cache, and when the key-value is JSON, then you can create with SQL, you can do full-text search, you can do analytics, you can run user-defined function, you get triggers, you get all that actual SQL going on, it's transactional, you get joins, ANSI joins, you get all those… windowing function. It's modern SQL on the JSON database. So, it's a general-purpose database, and it's a general-purpose database that syncs.I think that's the important part of Couchbase. We are very good at syncing cluster of databases together. So, great for multi-cloud, hybrid cloud, on-prem, whatever suits you. And we also sync on the device, there's a thing called Couchbase Mobile, which is a local database that runs in your phone, and it will sync automatically to the server. So, a general-purpose database that syncs and that's quite modern.We try to fit as much way of growing data as possible in our database. It's kind of a several-in-one database. We call that a data platform. It took me a while to warm up to the word platform because I used to work for an enterprise content management platform and then I've been working for a Platform as a Service and then a data platform. So, it took me a bit of time to warm up to that term, but it explained fairly well, the fact that it's a several-in-one product and we empower people to do the trade-offs that they want.Not everybody needs… SQL. Some people just need key-value, some people need search, some people need to do SQL and search in the same query, which we also want people to do. So, it's about choices, it's about empowering people. And that's why the word platform—which can feel intimidating because it can seem complex, you know, [for 00:12:34] a lot of choices. And choices is maybe the enemy of a good developer experience.And, you know, we can try to talk—we can talk for hours about this. The more services you offer, the more complicated it becomes. What's the sweet spots? We did—our own trade-off was to have good documentation and good in-app help to fix that complexity problem. That's the trade-off that we did.Corey: Well, we should probably divert here just to make sure that we cover the basic groundwork for those who might not be aware: what exactly is Couchbase? I know that it's a database, which honestly, anything is a database if you hold it incorrectly enough; that's my entire shtick. But what is it exactly? Where does it start? Where does it stop?Laurent: Oh, where does it start? That's an interesting question. It's a… a merge—some people would say a fork—of Apache CouchDB, and membase. Membase was a distributed key-value store and CouchDB was this weird Erlang and C JSON REST API database that was built by Damian Katz from Lotus Notes, and that was in 2006 or seven. That was before Node.js.Let's not care about the exact date. The point is, a JSON and REST API-enabled database before Node.js was, like, a strong [laugh] power move. And so, those two merged and created the first version of Couchbase. And then we've added all those things that people want to do, so SQL, full-text search, analytics, user-defined function, mobile sync, you know, all those things. So basically, a general-purpose database.Corey: For what things is it not a great fit? This is always my favorite question to ask database folks because the zealot is going to say, “It's good for every use case under the sun. Use it for everything, start to finish”—Laurent: Yes.Corey: —and very few databases can actually check that box.Laurent: It's a very interesting question because when I pitch like, “We do all the things,” because we are a platform, people say, “Well, you must be doing lots of trade-offs. Where is the trade-off?” The trade-off is basically the way you store something is going to determine the efficiency of your [growing 00:14:45]—or the way you [grow 00:14:47] it. And that's one of the first thing you learn in computer science. You learn about data structure and you know that it's easier to get something in a hashmap when you have the key than passing your whole list of elements and checking your data, is it right one? It's the same for databases.So, our different services are different ways to store the data and to query it. So, where is it not good, it's where we don't have an index or a service that answer to the way you want to query data. We don't have a graph service right now. You can still do recursive common table expression for the SQL nerds out there, that will allow you to do somewhat of a graph way of querying your data, but that's not, like, actual—that's not a great experience for people were expecting a graph, like a Neo4j or whatever was a graph database experience.So, that's the trade-off that we made. We have a lot of things at the same place and it can be a little hard, intimidating to operate, and the developer experience can be a little, “Oh, my God, what is this thing that can do all of those features?” At the same time, that's just, like, one SDK to learn for all of the features we've just talked about. So, that's what we did. That's a trade-off that we did.It sucks to operate—well, [unintelligible 00:16:05] Couchbase Capella, which is a lot like a vendor-ish thing to say, but that's the value props of our managed cloud. It's hard to operate, we'll operate this for you. We have a Kubernetes operator. If you are one of the few people that wants to do Kubernetes at home, that's also something you can do. So yeah, I guess what we cannot do is the thing that Route 53 and [Unbound 00:16:26] and [unintelligible 00:16:27] DNS do, which is this weird DNS database thing that you like so much.Corey: One thing that's, I guess, is a sign of the times, but I have to confess that I'm relatively skeptical around, when I pull up couchbase.com—as one does; you're publicly traded; I don't feel that your company has much of a choice in this—but the first thing it greets me with is Couchbase Capella—which, yes, that is your hosted flagship product; that should be the first thing I see on the website—then it says, “Announcing Capella iQ, AI-powered coding assistance for developers.” Which oh, great, not another one of these.So, all right, give me the pitch. What is the story around, “Ooh, everything that has been a problem before, AI is going to make it way better.” Because I've already talked to you about developer experience. I know where you stand on these things. I have a suspicion you would not be here to endorse something you don't believe in. How does the AI magic work in this context?Laurent: So, that's the thing, like, who's going to be the one that get their products out before the other? And so, we're announcing it on the website. It's available on the private preview only right now. I've tried it. It works.How does it works? The way most chatbot AI code generation work is there's a big model, large language model that people use and that people fine-tune into in order to specialize it to the tasks that they want to do. The way we've built Couchbase iQ is we picked a very famous large language model, and when you ask a question to a bot, there's a context, there's a… the size of the window basically, that allows you to fit as much contextual information as possible. The way it works and the reason why it's integrated into Couchbase Capella is we make sure that we preload that context as much as possible and fine-tune that model, that [foundation 00:18:19] model, as much as possible to do whatever you want to do with Couchbase, which usually falls into several—a couple of categories, really—well maybe three—you want to write SQL, you want to generate data—actually, that's four—you want to generate data, you want to generate code, and if you paste some SQL code or some application code, you want to ask that model, what does do? It's especially true for SQL queries.And one of the questions that many people ask and are scared of with chatbot is how does it work in terms of learning? If you give a chatbot to someone that's very new to something, and they're just going to basically use a chatbot like Stack Overflow and not really think about what they're doing, well it's not [great 00:19:03] right, but because that's the example that people think most developer will do is generate code. Writing code is, like, a small part of our job. Like, a substantial part of our job is understanding what the code does.Corey: We spend a lot more time reading code than writing it, if we're, you know—Laurent: Yes.Corey: Not completely foolish.Laurent: Absolutely. And sometimes reading big SQL query can be a bit daunting, especially if you're new to that. And one of the good things that you get—Corey: Oh, even if you're not, it can still be quite daunting, let me assure you.Laurent: [laugh]. I think it's an acquired taste, let's be honest. Some people like to write assembly code and some people like to write SQL. I'm sort of in the middle right now. You pass your SQL query, and it's going to tell you more or less what it does, and that's a very nice superpower of AI. I think that's [unintelligible 00:19:48] that's the one that interests me the most right now is using AI to understand and to work better with existing pieces of code.Because a lot of people think that the cost of software is writing the software. It's maintaining the codebase you've written. That's the cost of the software. That's our job as developers should be to write legacy code because it means you've provided value long enough. And so, if in a company that works pretty well and there's a lot of legacy code and there's a lot of new people coming in and they'll have to learn all those things, and to be honest, sometimes we don't document stuff as much as we should—Corey: “The code is self-documenting,” is one of the biggest lies I hear in tech.Laurent: Yes, of course, which is why people are asking retired people to go back to COBOL again because nobody can read it and it's not documented. Actually, if someone's looking for a company to build, I guess, explaining COBOL code with AI would be a pretty good fit to do in many places.Corey: Yeah, it feels like that's one of those things that would be of benefit to the larger world. The counterpoint to that is you got that many business processes wrapped around something running COBOL—and I assure you, if you don't, you would have migrated off of COBOL long before now—it's making sure that okay well, computers, when they're in the form of AI, are very, very good at being confident-sounding when they talk about things, but they can also do that when they're completely wrong. It's basically a BS generator. And that is a scary thing when you're taking a look at something that broad. I mean, I'll use the AI coding assistance for things all the time, but those things look a lot more like, “Okay, I haven't written CloudFormation from scratch in a while. Build out the template, just because I forget the exact sequence.” And it's mostly right on things like that. But then you start getting into some of the real nuanced areas like race conditions and the rest, and often it can make things worse instead of better. That's the scary part, for me, at least.Laurent: Most coding assistants are… and actually, each time you ask its opinion to an AI, they say, “Well, you should take this with a grain of salt and we are not a hundred percent sure that this is the case.” And this is, make sure you proofread that, which again, from a learning perspective, can be a bit hard to give to new students. Like, you're giving something to someone and might—that assumes is probably as right as Wikipedia but actually, it's not. And it's part of why it works so well. Like, the anthropomorphism that you get with chatbots, like, this, it feels so human. That's why it get people so excited about it because if you think about it, it's not that new. It's just the moment it took off was the moment it looked like an assertive human being.Corey: As you take a look through, I guess, the larger ecosystem now, as well as the database space, given that is where you specialize, what do you think people are getting right and what do you think people are getting wrong?Laurent: There's a couple of ways of seeing this. Right now, when I look at from the outside, every databases is going back to SQL, I think there's a good reason for that. And it's interesting to put into perspective with AI because when you generate something, there's probably less chance to generate something wrong with SQL than generating something with code directly. And I think five generation—was it four or five generation language—there some language generation, so basically, the first innovation is assembly [into 00:23:03] in one and then you get more evolved languages, and at some point you get SQL. And SQL is a way to very shortly express a whole lot of business logic.And I think what people are doing right now is going back to SQL. And it's been impressive to me how even new developers that were all about [ORMs 00:23:25] and [no-DMs 00:23:26], and you know, avoiding writing SQL as much as possible, are actually back to it. And that's, for an old guy like me—well I mean, not that old—it feels good. I think SQL is coming back with a vengeance and that makes me very happy. I think what people don't realize is that it also involves doing data modeling, right, and stuff because database like Couchbase that are schemaless exist. You should store your data without thinking about it, you should still do data modeling. It's important. So, I think that's the interesting bits. What are people doing wrong in that space? I'm… I don't want to say bad thing about other databases, so I cannot even process that thought right now.Corey: That's okay. I'm thrilled to say negative things about any database under the sun. They all haunt me. I mean, someone wants to describe SQL to me is the chess of the programming world and I feel like that's very accurate. I have found that it is far easier in working with databases to make mistakes that don't wash off after a new deployment than it is in most other realms of technology. And when you're lucky and have a particular aura, you tend to avoid that stuff, at least that was always my approach.Laurent: I think if I had something to say, so just like the XKCD about standards: like, “there's 14 standards. I'm going to do one that's going to unify them all.” And it's the same with database. There's a lot… a [laugh] lot of databases. Have you ever been on a website called dbdb.io?Corey: Which one is it? I'm sorry.Laurent: Dbdb.io is the database of databases, and it's very [laugh] interesting website for database nerds. And so, if you're into database, dbdb.io. And you will find Couchbase and you will find a whole bunch of other databases, and you'll get to know which database is derived from which other database, you get the history, you get all those things. It's actually pretty interesting.Corey: I'm familiar with DB-Engines, which is sort of like the ranking databases by popularity, and companies will bend over backwards to wind up hitting all of the various things that they want in that space. The counterpoint with all of it is that it's… it feels historically like there haven't exactly been an awful lot of, shall we say, huge innovations in databases for the past few years. I mean, sure, we hear about vectors all the time now because of the joy that's AI, but smarter people than I are talking about how, well that's more of a feature than it is a core database. And the continual battle that we all hear about constantly is—and deal with ourselves—of should we use a general-purpose database, or a task-specific database for this thing that I'm doing remains largely unsolved.Laurent: Yeah, what's new? And when you look at it, it's like, we are going back to our roots and bringing SQL again. So, is there anything new? I guess most of the new stuff, all the interesting stuff in the 2010s—well, basically with the cloud—were all about the distribution side of things and were all about distributed consensus, Zookeeper, etcd, all that stuff. Couchbase is using an RAFT-like algorithm to keep every node happy and under the same cluster.I think that's one of the most interesting things we've had for the past… well, not for the past ten years, but between, basically, 20 or… between the start of AWS and well, let's say seven years ago. I think the end of the distribution game was brought to us by the people that have atomic clock in every data center because that's what you use to synchronize things. So, that was interesting things. And then suddenly, there wasn't that much innovation in the distributed world, maybe because Aphyr disappeared from Twitter. That might be one of the reason. He's not here to scare people enough to be better at that.Aphyr was the person behind the test called the Jepsen Test [shoot 00:27:12]. I think his blog engine was called Call Me Maybe, and he was going through every distributed system and trying to break them. And that was super interesting. And it feels like we're not talking that much about this anymore. It really feels like database have gone back to the status of infrastructure.In 2010, it was not about infrastructure. It was about developer empowerment. It was about serving JSON and developer experience and making sure that you can code faster without some constraint in a distributed world. And like, we fixed this for the most part. And the way we fixed this—and as you said, lack of innovation, maybe—has brought databases back to an infrastructure layer.Again, it wasn't the case 15 years a—well, 2023—13 years ago. And that's interesting. When you look at the new generation of databases, sometimes it's just a gateway on top of a well-known database and they call that a database, but it provides higher-level services, provides higher-level bricks, better developer experience to developer to build stuff faster. We've been trying to do this with Couchbase App Service and our sync gateway, which is basically a gateway on top of a Couchbase cluster that allow you to manage authentication, authorization, that allows you to manage synchronization with your mobile device or with websites. And yeah, I think that's the most interesting thing to me in this industry is how it's been relegated back to infrastructure, and all the cool stuff, new stuff happens on the layer above that.Corey: I really want to thank you for taking the time to speak with me. If people want to learn more, where's the best place for them to find you?Laurent: Thanks for having me and for entertaining this conversation. I can be found anywhere on the internet with these six letters: L-D-O-G-U-I-N. That's actually 7 letters. Ldoguin. That's my handle on pretty much any social network. Ldoguin. So X, [BlueSky 00:29:21], LinkedIn. I don't know where to be anymore.Corey: I hear you. We'll put links to all of it in the [show notes 00:29:27] and let people figure out where they want to go on that. Thank you so much for taking the time to speak with me today. I really do appreciate it.Laurent: Thanks for having me.Corey: Laurent Doguin, Director of Developer Relations and Strategy at Couchbase. I'm Cloud Economist Corey Quinn and this episode has been brought to us by our friends at Couchbase. If you enjoyed this episode, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, along with an angry comment that you're not going to be able to submit properly because that platform of choice did not pay enough attention to the experience of typing in a comment.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.
Amir Szekely, Owner at CloudSnorkel, joins Corey on Screaming in the Cloud to discuss how he got his start in the early days of cloud and his solo project, CloudSnorkel. Throughout this conversation, Corey and Amir discuss the importance of being pragmatic when moving to the cloud, and the different approaches they see in developers from the early days of cloud to now. Amir shares what motivates him to develop open-source projects, and why he finds fulfillment in fixing bugs and operating CloudSnorkel as a one-man show. About AmirAmir Szekely is a cloud consultant specializing in deployment automation, AWS CDK, CloudFormation, and CI/CD. His background includes security, virtualization, and Windows development. Amir enjoys creating open-source projects like cdk-github-runners, cdk-turbo-layers, and NSIS.Links Referenced: CloudSnorkel: https://cloudsnorkel.com/ lasttootinaws.com: https://lasttootinaws.com camelcamelcamel.com: https://camelcamelcamel.com github.com/cloudsnorkel: https://github.com/cloudsnorkel Personal website: https://kichik.com TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn, and this is an episode that I have been angling for for longer than you might imagine. My guest today is Amir Szekely, who's the owner at CloudSnorkel. Amir, thank you for joining me.Amir: Thanks for having me, Corey. I love being here.Corey: So, I've been using one of your open-source projects for an embarrassingly long amount of time, and for the longest time, I make the critical mistake of referring to the project itself as CloudSnorkel because that's the word that shows up in the GitHub project that I can actually see that jumps out at me. The actual name of the project within your org is cdk-github-runners if I'm not mistaken.Amir: That's real original, right?Corey: Exactly. It's like, “Oh, good, I'll just mention that, and suddenly everyone will know what I'm talking about.” But ignoring the problems of naming things well, which is a pain that everyone at AWS or who uses it knows far too well, the product is basically magic. Before I wind up basically embarrassing myself by doing a poor job of explaining what it is, how do you think about it?Amir: Well, I mean, it's a pretty simple project, which I think what makes it great as well. It creates GitHub runners with CDK. That's about it. It's in the name, and it just does that. And I really tried to make it as simple as possible and kind of learn from other projects that I've seen that are similar, and basically learn from my pain points in them.I think the reason I started is because I actually deployed CDK runners—sorry, GitHub runners—for one company, and I ended up using the Kubernetes one, right? So, GitHub in themselves, they have two projects they recommend—and not to nudge GitHub, please recommend my project one day as well—they have the Kubernetes controller and they have the Terraform deployer. And the specific client that I worked for, they wanted to use Kubernetes. And I tried to deploy it, and, Corey, I swear, I worked three days; three days to deploy the thing, which was crazy to me. And every single step of the way, I had to go and read some documentation, figure out what I did wrong, and apparently the order the documentation was was incorrect.And I had to—I even opened tickets, and they—you know, they were rightfully like, “It's open-source project. Please contribute and fix the documentation for us.” At that point, I said, “Nah.” [laugh]. Let me create something better with CDK and I decided just to have the simplest setup possible.So usually, right, what you end up doing in these projects, you have to set up either secrets or SSM parameters, and you have to prepare the ground and you have to get your GitHub token and all those things. And that's just annoying. So, I decided to create a—Corey: So much busy work.Amir: Yes, yeah, so much busy work and so much boilerplate and so much figuring out the right way and the right order, and just annoying. So, I decided to create a setup page. I thought, “What if you can actually install it just like you install any app on GitHub,” which is the way it's supposed to be right? So, when you install cdk-github-runners—CloudSnorkel—you get an HTML page and you just click a few buttons and you tell it where to install it and it just installs it for you. And it sets the secrets and everything. And if you want to change the secret, you don't have to redeploy. You can just change the secret, right? You have to roll the token over or whatever. So, it's much, much easier to install.Corey: And I feel like I discovered this project through one of the more surreal approaches—and I had cause to revisit it a few weeks ago when I was redoing my talk for the CDK Community Day, which has since happened and people liked the talk—and I mentioned what CloudSnorkel had been doing and how I was using the runners accordingly. So, that was what I accidentally caused me to pop back up with, “Hey, I've got some issues here.” But we'll get to that. Because once upon a time, I built a Twitter client for creating threads because shitposting is my love language, I would sit and create Twitter threads in the middle of live keynote talks. Threading in the native client was always terrible, and I wanted to build something that would help me do that. So, I did.And it was up for a while. It's not anymore because I'm not paying $42,000 a month in API costs to some jackass, but it still exists in the form of lasttootinaws.com if you want to create threads on Mastodon. But after I put this out, some people complained that it was slow.To which my response was, “What do you mean? It's super fast for me in San Francisco talking to it hosted in Oregon.” But on every round trip from halfway around the world, it became a problem. So, I got it into my head that since this thing was fully stateless, other than a Lambda function being fronted via an API Gateway, that I should deploy it to every region. It didn't quite fit into a Cloudflare Worker or into one of the Edge Lambda functions that AWS has given up on, but okay, how do I deploy something to every region?And the answer is, with great difficulty because it's clear that no one was ever imagining with all those regions that anyone would use all of them. It's imagined that most customers use two or three, but customers are different, so which two or three is going to be widely varied. So, anything halfway sensible about doing deployments like this didn't work out. Again, because this thing was also a Lambda function and an API Gateway, it was dirt cheap, so I didn't really want to start spending stupid amounts of money doing deployment infrastructure and the rest.So okay, how do I do this? Well, GitHub Actions is awesome. It is basically what all of AWS's code offerings wish that they were. CodeBuild is sad and this was kind of great. The problem is, once you're out of the free tier, and if you're a bad developer where you do a deploy on every iteration, suddenly it starts costing for what I was doing in every region, something like a quarter of per deploy, which adds up when you're really, really bad at programming.Amir: [laugh].Corey: So, their matrix jobs are awesome, but I wanted to do some self-hosted runners. How do I do that? And I want to keep it cheap, so how do I do a self-hosted runner inside of a Lambda function? Which led me directly to you. And it was nothing short of astonishing. This was a few years ago. I seem to recall that it used to be a bit less well-architected in terms of its elegance. Did it always use step functions, for example, to wind up orchestrating these things?Amir: Yeah, so I do remember that day. We met pretty much… basically as a joke because the Lambda Runner was a joke that I did, and I posted on Twitter, and I was half-proud of my joke that starts in ten seconds, right? But yeah, no, the—I think it always used functions. I've been kind of in love with the functions for the past two years. They just—they're nice.Corey: Oh, they're magic, and AWS is so bad at telling their story. Both of those things are true.Amir: Yeah. And the API is not amazing. But like, when you get it working—and you know, you have to spend some time to get it working—it's really nice because then you have nothing to manage, ever. And they can call APIs directly now, so you don't have to even create Lambdas. It's pretty cool.Corey: And what I loved is you wind up deploying this thing to whatever account you want it to live within. What is it, the OIDC? I always get those letters in the wrong direction. OIDC, I think, is correct.Amir: I think it's OIDC, yeah.Corey: Yeah, and it winds up doing this through a secure method as opposed to just okay, now anyone with access to the project can deploy into your account, which is not ideal. And it just works. It spins up a whole bunch of these Lambda functions that are using a Docker image as the deployment environment. And yeah, all right, if effectively my CDK deploy—which is what it's doing inside of this thing—doesn't complete within 15 minutes, then it's not going to and the thing is going to break out. We've solved the halting problem. After 15 minutes, the loop will terminate. The end.But that's never been a problem, even with getting ACM certificates spun up. It completes well within that time limit. And its cost to me is effectively nothing. With one key exception: that you made the choice to use Secrets Manager to wind up storing a lot of the things it cares about instead of Parameter Store, so I think you wind up costing me—I think there's two of those different secrets, so that's 80 cents a month. Which I will be demanding in blood one of these days if I ever catch you at re:Invent.Amir: I'll buy you beer [laugh].Corey: There we go. That'll count. That'll buy, like, several months of that. That works—at re:Invent, no. The beers there are, like, $18, so that'll cover me for years. We're set.Amir: We'll split it [laugh].Corey: Exactly. Problem solved. But I like the elegance of it, I like how clever it is, and I want to be very clear, though, it's not just for shitposting. Because it's very configurable where, yes, you can use Lambda functions, you can use Spot Instances, you can use CodeBuild containers, you can use Fargate containers, you can use EC2 instances, and it just automatically orchestrates and adds these self-hosted runners to your account, and every build gets a pristine environment as a result. That is no small thing.Amir: Oh, and I love making things configurable. People really appreciate it I feel, you know, and gives people kind of a sense of power. But as long as you make that configuration simple enough, right, or at least the defaults good defaults, right, then, even with that power, people still don't shoot themselves in the foot and it still works really well. By the way, we just added ECS recently, which people really were asking for because it gives you the, kind of, easy option to have the runner—well, not the runner but at least the runner infrastructure staying up, right? So, you can have auto-scaling group backing ECS and then the runner can start up a lot faster. It was actually very important to other people because Lambda, as fast that it is, it's limited, and Fargate, for whatever reason, still to this day, takes a minute to start up.Corey: Yeah. What's wild to me about this is, start to finish, I hit a deploy to the main branch and it sparks the thing up, runs the deploy. Deploy itself takes a little over two minutes. And every time I do this, within three minutes of me pushing to commit, the deploy is done globally. It is lightning fast.And I know it's easy to lose yourself in the idea of this being a giant shitpost, where, oh, who's going to do deployment jobs in Lambda functions? Well, kind of a lot of us for a variety of reasons, some of which might be better than others. In my case, it was just because I was cheap, but the massive parallelization ability to do 20 simultaneous deploys in a matrix configuration that doesn't wind up smacking into rate limits everywhere, that was kind of great.Amir: Yeah, we have seen people use Lambda a lot. It's mostly for, yeah, like you said, small jobs. And the environment that they give you, it's kind of limited, so you can't actually install packages, right? There is no sudo, and you can't actually install anything unless it's in your temp directory. But still, like, just being able to run a lot of little jobs, it's really great. Yeah.Corey: And you can also make sure that there's a Docker image ready to go with the stuff that you need, just by configuring how the build works in the CDK. I will admit, I did have a couple of bug reports for you. One was kind of useful, where it was not at all clear how to do this on top of a Graviton-based Lambda function—because yeah, that was back when not everything really supported ARM architectures super well—and a couple of other times when the documentation was fairly ambiguous from my perspective, where it wasn't at all clear, what was I doing? I spent four hours trying to beat my way through it, I give up, filed an issue, went to get a cup of coffee, came back, and the answer was sitting there waiting for me because I'm not convinced you sleep.Amir: Well, I am a vampire. My last name is from the Transylvania area [laugh]. So—Corey: Excellent. Excellent.Amir: By the way, not the first time people tell me that. But anyway [laugh].Corey: There's something to be said for getting immediate responsiveness because one of the reasons I'm always so loath to go and do a support ticket anywhere is this is going to take weeks. And then someone's going to come back with a, “I don't get it.” And try and, like, read the support portfolio to you. No, you went right into yeah, it's this. Fix it and your problem goes away. And sure enough, it did.Amir: The escalation process that some companies put you through is very frustrating. I mean, lucky for you, CloudSnorkel is a one-man show and this man loves solving bugs. So [laugh].Corey: Yeah. Do you know of anyone using it for anything that isn't ridiculous and trivial like what I'm using it for?Amir: Yeah, I have to think whether or not I can… I mean, so—okay. We have a bunch of dedicated users, right, the GitHub repo, that keep posting bugs and keep posting even patches, right, so you can tell that they're using it. I even have one sponsor, one recurring sponsor on GitHub that uses it.Corey: It's always nice when people thank you via money.Amir: Yeah. Yeah, it is very validating. I think [BLEEP] is using it, but I also don't think I can actually say it because I got it from the GitHub.Corey: It's always fun. That's the beautiful part about open-source. You don't know who's using this. You see what other things people are working on, and you never know, is one of their—is this someone's side project, is it a skunkworks thing, or God forbid, is this inside of every car going forward and no one bothered to tell me about that. That is the magic and mystery of open-source. And you've been doing open-source for longer than I have and I thought I was old. You were originally named in some of the WinAMP credits, for God's sake, that media player that really whipped the llama's ass.Amir: Oh, yeah, I started real early. I started about when I was 15, I think. I started off with Pascal or something or even Perl, and then I decided I have to learn C and I have to learn Windows API. I don't know what possessed me to do that. Win32 API is… unique [laugh].But once I created those applications for myself, right, I think there was—oh my God, do you know the—what is it called, Sherlock in macOS, right? And these days, for PowerToys, there is the equivalent of it called, I don't know, whatever that—PowerBar? That's exactly—that was that. That's a project I created as a kid. I wanted something where I can go to the Run menu of Windows when you hit Winkey R, and you can just type something and it will start it up, right?I didn't want to go to the Start menu and browse and click things. I wanted to do everything with the keyboard. So, I created something called Blazerun [laugh], which [laugh] helped you really easily create shortcuts that went into your path, right, the Windows path, so you can really easily start them from Winkey R. I don't think that anyone besides me used it, but anyway, that thing needed an installer, right? Because Windows, you got to install things. So, I ended up—Corey: Yeah, these days on Mac OS, I use Alfred for that which is kind of long in the tooth, but there's a launch bar and a bunch of other stuff for it. What I love is that if I—I can double-tap the command key and that just pops up whatever I need it to and tell the computer what to do. It feels like there's an AI play in there somewhere if people can figure out how to spend ten minutes on building AI that does something other than lets them fire their customer service staff.Amir: Oh, my God. Please don't fire customer service staff. AI is so bad.Corey: Yeah, when I reach out to talk to a human, I really needed a human.Amir: Yes. Like, I'm not calling you because I want to talk to a robot. I know there's a website. Leave me alone, just give me a person.Corey: Yeah. Like, you already failed to solve my problem on your website. It's person time.Amir: Exactly. Oh, my God. Anyway [laugh]. So, I had to create an installer, right, and I found it was called NSIS. So, it was a Nullsoft “SuperPiMP” installation system. Or in the future, when Justin, the guy who created Winamp and NSIS, tried to tone down a little bit, Nullsoft Scriptable Installation System. And SuperPiMP is—this is such useless history for you, right, but SuperPiMP is the next generation of PiMP which is Plug-in Mini Packager [laugh].Corey: I remember so many of the—like, these days, no one would ever name any project like that, just because it's so off-putting to people with sensibilities, but back then that was half the stuff that came out. “Oh, you don't like how this thing I built for free in the wee hours when I wasn't working at my fast food job wound up—you know, like, how I chose to name it, well, that's okay. Don't use it. Go build your own. Oh, what you're using it anyway. That's what I thought.”Amir: Yeah. The source code was filled with profanity, too. And like, I didn't care, I really did not care, but some people would complain and open bug reports and patches. And my policy was kind of like, okay if you're complaining, I'm just going to ignore you. If you're opening a patch, fine, I'm going to accept that you're—you guys want to create something that's sensible for everybody, sure.I mean, it's just source code, you know? Whatever. So yeah, I started working on that NSIS. I used it for myself and I joined the forums—and this kind of answers to your question of why I respond to things so fast, just because of the fun—I did the same when I was 15, right? I started going on the forums, you remember forums? You remember that [laugh]?Corey: Oh, yeah, back before they all became terrible and monetized.Amir: Oh, yeah. So, you know, people were using NSIS, too, and they had requests, right? They wanted. Back in the day—what was it—there was only support for 16-bit colors for the icon, so they want 32-bit colors and big colors—32—big icon, sorry, 32 pixels by 32 pixels. Remember, 32 pixels?Corey: Oh, yes. Not well, and not happily, but I remember it.Amir: Yeah. So, I started just, you know, giving people—working on that open-source and creating up a fork. It wasn't even called ‘fork' back then, but yeah, I created, like, a little fork of myself and I started adding all these features. And people were really happy, and kind of created, like, this happy cycle for myself: when people were happy, I was happy coding. And then people were happy by what I was coding. And then they were asking for more and they were getting happier, the more I responded.So, it was kind of like a serotonin cycle that made me happy and made everybody happy. So, it's like a win, win, win, win, win. And that's how I started with open-source. And eventually… NSIS—again, that installation system—got so big, like, my fork got so big, and Justin, the guy who works on WinAMP and NSIS, he had other things to deal with. You know, there's a whole history there with AOL. I'm sure you've heard all the funny stories.Corey: Oh, yes. In fact, one thing that—you want to talk about weird collisions of things crossing, one of the things I picked up from your bio when you finally got tired of telling me no and agreed to be on the show was that you're also one of the team who works on camelcamelcamel.com. And I keep forgetting that's one of those things that most people have no idea exists. But it's very simple: all it does is it tracks Amazon products that you tell it to and alerts you when there's a price drop on the thing that you're looking at.It's something that is useful. I try and use it for things of substance or hobbies because I feel really pathetic when I'm like, get excited emails about a price drop in toilet paper. But you know, it's very handy just to keep an idea for price history, where okay, am I actually being ripped off? Oh, they claim it's their big Amazon Deals day and this is 40% off. Let's see what camelcamelcamel has to say.Oh, surprise. They just jacked the price right beforehand and now knocked 40% off. Genius. I love that. It always felt like something that was going to be blown off the radar by Amazon being displeased, but I discovered you folks in 2010 and here you are now, 13 years later, still here. I will say the website looks a lot better now.Amir: [laugh]. That's a recent change. I actually joined camel, maybe two or three years ago. I wasn't there from the beginning. But I knew the guy who created it—again, as you were saying—from the Winamp days, right? So, we were both working in the free—well, it wasn't freenode. It was not freenode. It was a separate IRC server that, again, Justin created for himself. It was called landoleet.Corey: Mmm. I never encountered that one.Amir: Yeah, no, it was pretty private. The only people that cared about WinAMP and NSIS ended up joining there. But it was a lot of fun. I met a lot of friends there. And yeah, I met Daniel Green there as well, and he's the guy that created, along with some other people in there that I think want to remain anonymous so I'm not going to mention, but they also were on the camel project.And yeah, I was kind of doing my poor version of shitposting on Twitter about AWS, kind of starting to get some traction and maybe some clients and talk about AWS so people can approach me, and Daniel approached me out of the blue and he was like, “Do you just post about AWS on Twitter or do you also do some AWS work?” I was like, “I do some AWS work.”Corey: Yes, as do all of us. It's one of those, well crap, we're getting called out now. “Do you actually know how any of this stuff works?” Like, “Much to my everlasting shame, yes. Why are you asking?”Amir: Oh, my God, no, I cannot fix your printer. Leave me alone.Corey: Mm-hm.Amir: I don't want to fix your Lambdas. No, but I do actually want to fix your Lambdas. And so, [laugh] he approached me and he asked if I can help them move camelcamelcamel from their data center to AWS. So, that was a nice big project. So, we moved, actually, all of camelcamelcamel into AWS. And this is how I found myself not only in the Winamp credits, but also in the camelcamelcamel credits page, which has a great picture of me riding a camel.Corey: Excellent. But one of the things I've always found has been that when you take an application that has been pre-existing for a while in a data center and then move it into the cloud, you suddenly have to care about things that no one sensible pays any attention to in the land of the data center. Because it's like, “What do I care about how much data passes between my application server and the database? Wait, what do you mean that in this configuration, that's a chargeable data transfer? Oh, dear Lord.” And things that you've never had to think about optimizing are suddenly things are very much optimizing.Because let's face it, when it comes to putting things in racks and then running servers, you aren't auto-scaling those things, so everything tends to be running over-provisioned, for very good reasons. It's an interesting education. Anything you picked out from that process that you think it'd be useful for folks to bear in mind if they're staring down the barrel of the same thing?Amir: Yeah, for sure. I think… in general, right, not just here. But in general, you always want to be pragmatic, right? You don't want to take steps are huge, right? So, the thing we did was not necessarily rewrite everything and change everything to AWS and move everything to Lambda and move everything to Docker.Basically, we did a mini lift-and-shift, but not exactly lift-and-shift, right? We didn't take it as is. We moved to RDS, we moved to ElastiCache, right, we obviously made use of security groups and session connect and we dropped SSH Sage and we improved the security a lot and we locked everything down, all the permissions and all that kind of stuff, right? But like you said, there's stuff that you start having to pay attention to. In our case, it was less the data transfer because we have a pretty good CDN. There was more of IOPS. So—and IOPS, specifically for a database.We had a huge database with about one terabyte of data and a lot of it is that price history that you see, right? So, all those nice little graphs that we create in—what do you call them, charts—that we create in camelcamelcamel off the price history. There's a lot of data behind that. And what we always want to do is actually remove that from MySQL, which has been kind of struggling with it even before the move to AWS, but after the move to AWS, where everything was no longer over-provisioned and we couldn't just buy a few more NVMes on Amazon for 100 bucks when they were on sale—back when we had to pay Amazon—Corey: And you know, when they're on sale. That's the best part.Amir: And we know [laugh]. We get good prices on NVMe. But yeah, on Amazon—on AWS, sorry—you have to pay for io1 or something, and that adds up real quick, as you were saying. So, part of that move was also to move to something that was a little better for that data structure. And we actually removed just that data, the price history, the price points from MySQL to DynamoDB, which was a pretty nice little project.Actually, I wrote about it in my blog. There is, kind of, lessons learned from moving one terabyte from MySQL to DynamoDB, and I think the biggest lesson was about hidden price of storage in DynamoDB. But before that, I want to talk about what you asked, which was the way that other people should make that move, right? So again, be pragmatic, right? If you Google, “How do I move stuff from DynamoDB to MySQL,” everybody's always talking about their cool project using Lambda and how you throttle Lambda and how you get throttled from DynamoDB and how you set it up with an SQS, and this and that. You don't need all that.Just fire up an EC2 instance, write some quick code to do it. I used, I think it was Go with some limiter code from Uber, and that was it. And you don't need all those Lambdas and SQS and the complication. That thing was a one-time thing anyway, so it doesn't need to be super… super-duper serverless, you know?Corey: That is almost always the way that it tends to play out. You encounter these weird little things along the way. And you see so many things that are tied to this is how architecture absolutely must be done. And oh you're not a real serverless person if you don't have everything running in Lambda and the rest. There are times where yeah, spin up an EC2 box, write some relatively inefficient code in ten minutes and just do the thing, and then turn it off when you're done. Problem solved. But there's such an aversion to that. It's nice to encounter people who are pragmatists more than they are zealots.Amir: I mostly learned that lesson. And both Daniel Green and me learned that lesson from the Winamp days. Because we both have written plugins for Winamp and we've been around that area and you can… if you took one of those non-pragmatist people, right, and you had them review the Winamp code right now—or even before—they would have a million things to say. That code was—and NSIS, too, by the way—and it was so optimized. It was so not necessarily readable, right? But it worked and it worked amazing. And Justin would—if you think I respond quickly, right, Justin Frankel, the guy who wrote Winamp, he would release versions of NSIS and of Winamp, like, four versions a day, right? That was before [laugh] you had CI/CD systems and GitHub and stuff. That was just CVS. You remember CVS [laugh]?Corey: Oh, I've done multiple CVS migrations. One to Git and a couple to Subversion.Amir: Oh yeah, Subversion. Yep. Done ‘em all. CVS to Subversion to Git. Yep. Yep. That was fun.Corey: And these days, everyone's using Git because it—we're beginning to have a monoculture.Amir: Yeah, yeah. I mean, but Git is nicer than Subversion, for me, at least. I've had more fun with it.Corey: Talk about damning with faint praise.Amir: Faint?Corey: Yeah, anything's better than Subversion, let's be honest here.Amir: Oh [laugh].Corey: I mean, realistically, copying a bunch of files and directories to a.bak folder is better than Subversion.Amir: Well—Corey: At least these days. But back then it was great.Amir: Yeah, I mean, the only thing you had, right [laugh]?Corey: [laugh].Amir: Anyway, achieving great things with not necessarily the right tools, but just sheer power of will, that's what I took from the Winamp days. Just the entire world used Winamp. And by the way, the NSIS project that I was working on, right, I always used to joke that every computer in the world ran my code, every Windows computer in the world when my code, just because—Corey: Yes.Amir: So, many different companies use NSIS. And none of them cared that the code was not very readable, to put it mildly.Corey: So, many companies founder on those shores where they lose sight of the fact that I can point to basically no companies that died because their code was terrible, yeah, had an awful lot that died with great-looking code, but they didn't nail the business problem.Amir: Yeah. I would be lying if I said that I nailed exactly the business problem at NSIS because the most of the time I would spend there and actually shrinking the stub, right, there was appended to your installer data, right? So, there's a little stub that came—the executable, basically, that came before your data that was extracted. I spent, I want to say, years of my life [laugh] just shrinking it down by bytes—by literal bytes—just so it stays under 34, 35 kilobytes. It was kind of a—it was a challenge and something that people appreciated, but not necessarily the thing that people appreciate the most. I think the features—Corey: Well, no I have to do the same thing to make sure something fits into a Lambda deployment package. The scale changes, the problem changes, but somehow everything sort of rhymes with history.Amir: Oh, yeah. I hope you don't have to disassemble code to do that, though because that's uh… I mean, it was fun. It was just a lot.Corey: I have to ask, how much work went into building your cdk-github-runners as far as getting it to a point of just working out the door? Because I look at that and it feels like there's—like, the early versions, yeah, there wasn't a whole bunch of code tied to it, but geez, the iterative, “How exactly does this ridiculous step functions API work or whatnot,” feels like I'm looking at weeks of frustration. At least it would have been for me.Amir: Yeah, yeah. I mean, it wasn't, like, a day or two. It was definitely not—but it was not years, either. I've been working on it I think about a year now. Don't quote me on that. But I've put a lot of time into it. So, you know, like you said, the skeleton code is pretty simple: it's a step function, which as we said, takes a long time to get right. The functions, they are really nice, but their definition language is not very straightforward. But beyond that, right, once that part worked, it worked. Then came all the bug reports and all the little corner cases, right? We—Corey: Hell is other people's use cases. Always is. But that's honestly better than a lot of folks wind up experiencing where they'll put an open-source project up and no one ever knows. So, getting users is often one of the biggest barriers to a lot of this stuff. I've found countless hidden gems lurking around on GitHub with a very particular search for something that no one had ever looked at before, as best I can tell.Amir: Yeah.Corey: Open-source is a tricky thing. There needs to be marketing brought into it, there needs to be storytelling around it, and has to actually—dare I say—solve a problem someone has.Amir: I mean, I have many open-source projects like that, that I find super useful, I created for myself, but no one knows. I think cdk-github-runners, I'm pretty sure people know about it only because you talked about it on Screaming in the Cloud or your newsletter. And by the way, thank you for telling me that you talked about it last week in the conference because now we know why there was a spike [laugh] all of a sudden. People Googled it.Corey: Yeah. I put links to it as well, but it's the, yeah, I use this a lot and it's great. I gave a crappy explanation on how it works, but that's the trick I've found between conference talks and, dare I say, podcast episodes, you gives people a glimpse and a hook and tell them where to go to learn more. Otherwise, you're trying to explain every nuance and every intricacy in 45 minutes. And you can't do that effectively in almost every case. All you're going to do is drive people away. Make it sound exciting, get them to see the value in it, and then let them go.Amir: You have to explain the market for it, right? That's it.Corey: Precisely.Amir: And I got to say, I somewhat disagree with your—or I have a different view when you say that, you know, open-source projects needs marketing and all those things. It depends on what open-source is for you, right? I don't create open-source projects so they are successful, right? It's obviously always nicer when they're successful, but—and I do get that cycle of happiness that, like I was saying, people create bugs and I have to fix them and stuff, right? But not every open-source project needs to be a success. Sometimes it's just fun.Corey: No. When I talk about marketing, I'm talking about exactly what we're doing here. I'm not talking take out an AdWords campaign or something horrifying like that. It's you build something that solved the problem for someone. The big problem that worries me about these things is how do you not lose sleep at night about the fact that solve someone's problem and they don't know that it exists?Because that drives me nuts. I've lost count of the number of times I've been beating my head against a wall and asked someone like, “How would you handle this?” Like, “Oh, well, what's wrong with this project?” “What do you mean?” “Well, this project seems to do exactly what you want it to do.” And no one has it all stuffed in their head. But yeah, then it seems like open-source becomes a little more corporatized and it becomes a lead gen tool for people to wind up selling their SaaS services or managed offerings or the rest.Amir: Yeah.Corey: And that feels like the increasing corporatization of open-source that I'm not a huge fan of.Amir: Yeah. I mean, I'm not going to lie, right? Like, part of why I created this—or I don't know if it was part of it, but like, I had a dream that, you know, I'm going to get, oh, tons of GitHub sponsors, and everybody's going to use it and I can retire on an island and just make money out of this, right? Like, that's always a dream, right? But it's a dream, you know?And I think bottom line open-source is… just a tool, and some people use it for, like you were saying, driving sales into their SaaS, some people, like, may use it just for fun, and some people use it for other things. Or some people use it for politics, even, right? There's a lot of politics around open-source.I got to tell you a story. Back in the NSIS days, right—talking about politics—so this is not even about politics of open-source. People made NSIS a battleground for their politics. We would have translations, right? People could upload their translations. And I, you know, or other people that worked on NSIS, right, we don't speak every language of the world, so there's only so much we can do about figuring out if it's a real translation, if it's good or not.Back in the day, Google Translate didn't exist. Like, these days, we check Google Translate, we kind of ask a few questions to make sure they make sense. But back in the day, we did the best that we could. At some point, we got a patch for Catalan language, I'm probably mispronouncing it—but the separatist people in Spain, I think, and I didn't know anything about that. I was a young kid and… I just didn't know.And I just included it, you know? Someone submitted a patch, they worked hard, they wanted to be part of the open-source project. Why not? Sure I included it. And then a few weeks later, someone from Spain wanted to change Catalan into Spanish to make sure that doesn't exist for whatever reason.And then they just started fighting with each other and started making demands of me. Like, you have to do this, you have to do that, you have to delete that, you have to change the name. And I was just so baffled by why would someone fight so much over a translation of an open-source project. Like, these days, I kind of get what they were getting at, right?Corey: But they were so bad at telling that story that it was just like, so basically, screw, “You for helping,” is how it comes across.Amir: Yeah, screw you for helping. You're a pawn now. Just—you're a pawn unwittingly. Just do what I say and help me in my political cause. I ended up just telling both of them if you guys can agree on anything, I'm just going to remove both translations. And that's what I ended up doing. I just removed both translations. And then a few months later—because we had a release every month basically, I just added both of them back and I've never heard from them again. So sort of problem solved. Peace the Middle East? I don't know.Corey: It's kind of wild just to see how often that sort of thing tends to happen. It's a, I don't necessarily understand why folks are so opposed to other people trying to help. I think they feel like there's this loss of control as things are slipping through their fingers, but it's a really unwelcoming approach. One of the things that got me deep into the open-source ecosystem surprisingly late in my development was when I started pitching in on the SaltStack project right after it was founded, where suddenly everything I threw their way was merged, and then Tom Hatch, the guy who founded the project, would immediately fix all the bugs and stuff I put in and then push something else immediately thereafter. But it was such a welcoming thing.Instead of nitpicking me to death in the pull request, it just got merged in and then silently fixed. And I thought that was a classy way to do it. Of course, it doesn't scale and of course, it causes other problems, but I envy the simplicity of those days and just the ethos behind that.Amir: That's something I've learned the last few years, I would say. Back in the NSIS day, I was not like that. I nitpicked. I nitpicked a lot. And I can guess why, but it just—you create a patch—in my mind, right, like you create a patch, you fix it, right?But these days I get, I've been on the other side as well, right? Like I created patches for open-source projects and I've seen them just wither away and die, and then five years later, someone's like, “Oh, can you fix this line to have one instead of two, and then I'll merge it.” I'm like, “I don't care anymore. It was five years ago. I don't work there anymore. I don't need it. If you want it, do it.”So, I get it these days. And these days, if someone creates a patch—just yesterday, someone created a patch to format cdk-github-runners in VS Code. And they did it just, like, a little bit wrong. So, I just fixed it for them and I approved it and pushed it. You know, it's much better. You don't need to bug people for most of it.Corey: You didn't yell at them for having the temerity to contribute?Amir: My voice is so raw because I've been yelling for five days at them, yeah.Corey: Exactly, exactly. I really want to thank you for taking the time to chat with me about how all this stuff came to be and your own path. If people want to learn more, where's the best place for them to find you?Amir: So, I really appreciate you having me and driving all this traffic to my projects. If people want to learn more, they can always go to cloudsnorkel.com; it has all the projects. github.com/cloudsnorkel has a few more. And then my private blog is kichik.com. So, K-I-C-H-I-K dot com. I don't post there as much as I should, but it has some interesting AWS projects from the past few years that I've done.Corey: And we will, of course, put links to all of that in the show notes. Thank you so much for taking the time. I really appreciate it.Amir: Thank you, Corey. It was really nice meeting you.Corey: Amir Szekely, owner of CloudSnorkel. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, along with an insulting comment. Heck, put it on all of the podcast platforms with a step function state machine that you somehow can't quite figure out how the API works.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.
In the vast universe of enterprise data, governance is the north star guiding organizations to harness the full potential of their analytics and machine learning endeavors. In this pivotal episode, join Brooke and Dave as they sit down with Amazon DataZone's Principal Engineers, Fei Yuan and Aleatha Parker-Wood, to celebrate their recent GA launch and navigate the complexities of data governance. Amazon DataZone emerges as a beacon of innovation, offering a comprehensive suite of enterprise data governance tools designed to ensure quality, compliance, security, and optimal use of data. From data mesh catalogs to project-based access, and policy-driven data management, DataZone is the architect behind a robust data infrastructure seamlessly integrated with AWS's powerhouse services like Analytics, AI, IAM, S3, and CloudFormation. Tune in to this deep dive into Amazon DataZone and discover how it can revolutionize your enterprise data mesh, propelling your organization into a new era of data enlightenment.
Today we embark on a fascinating journey into the world of AWS Lambda functions and how to make them accessible to the public. In a recent use case, involving the creation of a public Lambda function for AWS users, we asked ourselves some interesting questions. How can you securely, cost-effectively, and conveniently publish AWS resources, especially Lambda functions, for others to use? And... can we possibly make some money out of this? Join us as we explore various options and share our findings for making your AWS resources available to the world. We dive into the Serverless Application Repository (SAR), an AWS treasure trove for publishing resources. And SAR isn't the only way! We also discuss alternatives like CloudFormation templates, GitHub publishing, Terraform modules, and container images. We explore the pros and cons of these methods and debate the implications in terms of cost, security, and ease of use. Finally, we touch on the AWS Marketplace as a platform to monetize your AWS resources.
Should you upgrade to Terraform 1.6 already? How to avoid blind spots when monitoring Lambda functions? An unusual way to utilize the CDK.
Levi McCormick, Cloud Architect at Jamf, joins Corey on Screaming in the Cloud to discuss his work modernizing baseline cloud infrastructure and his experience being on the compliance side of cloud engineering. Levi explains how he works to ensure the different departments he collaborates with are all on the same page so that different definitions don't end up in miscommunications, and why he feels a sandbox environment is an important tool that leads to a successful production environment. Levi and Corey also explore the ethics behind the latest generative AI craze. About LeviLevi is an automation engineer, with a focus on scalable infrastructure and rapid development. He leverages deep understanding of DevOps culture and cloud technologies to build platforms that scale to millions of users. His passion lies in helping others learn to cloud better.Links Referenced: Jamf: https://www.jamf.com/ Twitter: https://twitter.com/levi_mccormick LinkedIn: https://www.linkedin.com/in/levimccormick/ TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. A longtime friend and person has been a while since he's been on the show, Levi McCormick has been promoted or punished for his sins, depending upon how you want to slice that, and he is now the Director of Cloud Engineering at Jamf. Levi, welcome back.Levi: Thanks for having me, Corey.Corey: I have to imagine internally, you put that very pronounced F everywhere, and sometimes where it doesn't belong, like your IAMf policies and whatnot.Levi: It is fun to see how people like to interpret how to pronounce our name.Corey: So, it's been a while. What were you doing before? And how did you wind up stumbling your way into your current role?Levi: [laugh]. When we last spoke, I was a cloud architect here, diving into just our general practices and trying to shore up some of them. In between, I did a short stint as director of FedRAMP. We are pursuing some certifications in that area and I led, kind of, the engineering side of the compliance journey.Corey: That sounds fairly close to hell on earth from my particular point of view, just because I've dealt in the compliance side of cloud engineering before, and it sounds super interesting from a technical level until you realize just how much of it revolves around checking the boxes, and—at least in the era I did it—explaining things to auditors that I kind of didn't feel I should have to explain to an auditor, but there you have it. Has the state of that world improved since roughly 2015?Levi: I wouldn't say it has improved. While doing this, I did feel like I drove a time machine to work, you know, we're certifying VMs, rather than container-based architectures. There was a lot of education that had to happen from us to auditors, but once they understood what we were trying to do, I think they were kind of on board. But yeah, it was a [laugh] it was a journey.Corey: So, one of the things you do—in fact, the first line in your bio talking about it—is you modernize baseline cloud infrastructure provisioning. That means an awful lot of things depending upon who it is that's answering the question. What does that look like for you?Levi: For what we're doing right now, we're trying to take what was a cobbled-together part-time project for one engineer, we're trying to modernize that, turn it into as much self-service as we can. There's a lot of steps that happen along the way, like a new workload needs to be spun up, they decide if they need a new AWS account or not, we pivot around, like, what does the access profile look like, who needs to have access to it, which things does it need to connect to, and then you look at the billing side, compliance side, and you just say, you know, “Who needs to be informed about these things?” We apply tags to the accounts, we start looking at lower-level tagging, depending on if it's a shared workload account or if it's a completely dedicated account, and we're trying to wrap all of that in automation so that it can be as click-button as possible.Corey: Historically, I found that when companies try to do this, the first few attempts at it don't often go super well. We'll be polite and say their first attempts resemble something artisanal and handcrafted, which might not be ideal for this. And then in many cases, the overreaction becomes something that is very top-down, dictatorial almost, is the way I would frame that. And the problem people learn then is that, “Oh, everyone is going to route around us because they don't want to deal with us at all.” That doesn't quite seem like your jam from what I know of you and your approach to things. How do you wind up keeping the guardrails up without driving people to shadow IT their way around you?Levi: I always want to keep it in mind that even if it's not an option, I want to at least pretend like a given team could not use our service, right? I try to bring a service mentality to it, so we're talking Accounts as a Service. And then I just think about all of the things that they would have to solve if they didn't go through us, right? Like, are they managing their finances w—imagine they had to go in and negotiate some kind of pricing deal on their own, right, all of these things that come with being part of our organization, being part of our service offering. And then just making sure, like, those things are always easier than doing it on their own.Corey: How diverse would you say that the workloads are that are in your organization? I found that in many cases, you'll have a SaaS-style company where there's one primary workload that is usually bearing the name of the company, and that's the thing that they provide to everyone. And then you have the enterprise side of the world where they have 1500 or 2000 distinct application teams working on different things, and the only thing they really have in common is, well, that all gets billed to the same company, eventually.Levi: They are fairly diverse in how… they're currently created. We've gone through a few acquisitions, we've pulled a bunch of those into our ecosystem, if you will. So, not everything has been completely modernized or brought over to, you know, standards, if you will, if such a thing even exists in companies. You know [laugh], you may pretend that they do, but you're probably lying to yourself, right? But you know, there are varying platforms, we've got a whole laundry list of languages that are being used, we've got some containerized, some VM-based, some serverless workloads, so it's all over the place. But you nailed it. Like, you know, the majority of our footprint lives in maybe a handful of, you know, SaaS offerings.Corey: Right. It's sort of a fun challenge when you start taking a looser approach to these things because someone gets back from re:Invent, like, “Well, I went to the keynote and now I have my new shopping list of things I'm going to wind up deploying,” and ehh, that never goes well, having been that person in a previous life.Levi: Yeah. And you don't want to apply too strict of governance over these things, right? You want people to be able to play, you want them to be inspired and start looking at, like, what would be—what's something that's going to move the needle in terms of our cloud architecture or product offerings or whatever we have. So, we have sandbox accounts that are pretty much wide open, we've got some light governance over those, [laugh] moreso for billing than anything. And all of our internal tooling is available, you know, like if you're using containers or whatever, like, all of that stuff is in those sandbox accounts.And that's where our kind of service offering comes into play, right? Sandbox is still an account that we tried to vend, if you will, out of our service. So, people should be building in your sandbox environments just like they are in your production as much as possible. You know, it's a place where tools can get the tires kicked and smooth out bugs before you actually get into, you know, roadmap-impacting problems.Corey: One of the fun challenges you have is, as you said, the financial aspect of this. When you've got a couple of workloads that drive most things, you can reason about them fairly intelligently, but trying to predict the future—especially when you're dealing with multi-year contract agreements with large cloud providers—becomes a little bit of a guessing game, like, “Okay. Well, how much are we going to spend on generative AI over the next three years?” The problem with that is that if you listen to an awful lot of talking heads or executive types, like, “Oh, yeah, if we're spending $100 million a year, we're going to add another 50 on top of that, just in terms of generative AI.” And it's like, press X to doubt, just because it's… I appreciate that you're excited about these things and want to play with them, but let's make sure that there's some ‘there' there before signing contracts that are painful to alter.Levi: Yeah, it's a real struggle. And we have all of these new initiatives, things people are excited for. Meanwhile, we're bringing old architecture into a new platform, if you will, or a new footprint, so we have to constantly measure those against each other. We have a very active conversation with finance and with leadership every month, or even weekly, depending on the type of project and where that spend is coming from.Corey: One of the hard parts has always been, I think, trying to get people on the finance side of the world, the engineering side of the world, and the folks who are trying to predict what the business was going to do next, all speaking the same language. It just feels like it's too easy to wind up talking past each other if you're not careful.Levi: Yeah, it's really hard. Recently taken over the FinOps practice. It's been really important for me, for us to align on what our words mean, right? What are these definitions mean? How do we come to common consensus so that eventually the communication gets faster? But we can't talk past each other. We have to know what our words mean, we have to know what each person cares about in this conversation, or what does their end goal look like? What do they want out of the conversation? So, that's been—that's taken a significant amount of time.Corey: One of the problems I have is with the term FinOps as a whole, ignoring the fact entirely that it was an existing term of art within finance for decades; great, we're just going to sidestep past that whole mess—the problem you'll see is that it just seems like that it means something different to almost everyone who hears it. And it's sort of become a marketing term more so that it has an actual description of what people are doing. Just because some companies will have a quote-unquote, “FinOps team,” that is primarily going to be run by financial analysts. And others, “Well, we have one of those lying around, but it's mostly an engineering effort on our part.”And I've seen three or four different expressions as far as team composition goes and I'm not convinced any of them are right. But again, it's easy for me to sit here and say, “Oh, that's wrong,” without having an environment of my own to run. I just tend to look at what my clients do. And, “Well, I've seen a lot of things, and they all work poorly in different ways,” is not uplifting and helpful.Levi: Yeah. I try not to get too hung up on what it's called. This is the name that a lot of people inside the company have rallied around and as long as people are interested in saving money, cool, we'll call it FinOps, you know? I mean, DevOps is the same thing, right? In some companies, you're just a sysadmin with a higher pay, and in some companies, you're building extensive cloud architecture and pipelines.Corey: Honestly, for the whole DevOps side of the world, I maintain we're all systems administrators. The tools have changed, the methodologies have changed, the processes have changed, but the responsibility of ‘keep the site up' generally has not. But if you call yourself a sysadmin, you're just asking him to, “Please pay me less money in my next job.” No, thanks.Levi: Yeah. “Where's the Exchange Server for me to click on?” Right? That's the [laugh]—if you call yourself a sysadmin [crosstalk 00:11:34]—Corey: God. You're sending me back into twitching catatonia from my early days.Levi: Exactly [laugh].Corey: So, you've been paying attention to this whole generative AI hype monster. And I want to be clear, I say this as someone who finds the technology super neat and I'm optimistic about it, but holy God, it feels like people have just lost all sense. If that's you, my apologies in advance, but I'm still going to maintain the point.Levi: I've played with all the various toys out there. I'm very curious, you know? I think it's really fun to play with them, but to, like, make your entire business pivot on a dime and pursue it just seems ridiculous to me. I hate that the cryptocurrency space has pivoted so hard into it, you know? All the people that used to be shilling coins are now out there trying to cobble together a couple API calls and turn it into an AI, right?Corey: It feels like it's just a hype cycle that people are more okay with being a part of. Like, Andy Jassy, in the earnings call a couple of weeks ago saying that every Amazon team is working with generative AI. That's not great. That's terrifying. I've been playing with the toys as well and I've asked it things like, “Oh, spit out an IAM policy for me,” or, “Oh, great, what can I do to optimize my AWS bill?” And it winds up spitting out things that sound highly plausible, but they're also just flat-out wrong. And that, it feels like a lot of these spaces, it's not coming up with a plausible answer—that's the hard part—is coming up with the one that is correct. And that's what our jobs are built around.Levi: I've been trying to explain to a lot of people how, if you only have surface knowledge of the thing that it's telling you, it probably seems really accurate, but when you have deep knowledge on the topic that you're interacting with this thing, you're going to see all of the errors. I've been using GitHub's Copilot since the launch. You know, I was in one of the previews. And I love it. Like, it speeds up my development significantly.But there have been moments where I—you know, IAM policies are a great example. You know, I had it crank out a Lambda functions policy, and it was just frankly, wrong in a lot of places [laugh]. It didn't quite imagine new AWS services, but it was really [laugh] close. The API actions were—didn't exist. It just flat-out didn't exist.Corey: I love that. I've had some magic happen early on where it could intelligently query things against the AWS pricing API, but then I asked it the same thing a month later and it gave me something completely ridiculous. It's not deterministic, which is part of the entire problem with it, too. But it's also… it can help incredibly in some weird ways I didn't see coming. But it can also cause you to spend more time chasing that thing than just doing it yourself the first time.I found a great way to help it—you know, it helped me write blog posts with it. I tell it to write a blog post about a topic and give it some bullet points and say, “Write in my voice,” and everything it says I take issue with, so then I just copy that into a text editor and then mansplain-correct the robot for 20 minutes and, oh, now I've got a serviceable first draft.Levi: And how much time did you save [laugh] right? It is fun, you know?Corey: It does help because that's better for me at least and staring at an empty page of what am I going to write? It gets me past the writer's block problem.Levi: Oh, that's a great point, yeah. Just to get the ball rolling, right, once you—it's easier to correct something that's wrong, and you're almost are spite-driven at that point, right? Like, “Let me show this AI how wrong it was and I'll write the perfect blog post.” [laugh].Corey: It feels like the companies jumping on this, if you really dig into what we're talking about, it seems like they're all very excited about the possibility of we don't have to talk to customers anymore because the robots will all do that. And I don't think that's going to go the way you want to. We just have this minor hallucination problem. Yeah, that means that lies and tries to book customers to hotel destinations that don't exist. Think about this a little more. The failure mode here is just massive.Levi: It's scary, yeah. Like, without some kind of review process, I wouldn't ship that straight to my customers, right? I wouldn't put that in front of my customer and say, like, “This is”—I'm going to take this generative output and put it right in front of them. That scares me. I think as we get deeper into it, you know, maybe we'll see… I don't know, maybe we'll put some filters or review process, or maybe it'll get better. I mean, who was it that said, you know, “This is the worst it's ever going to be?” Right, it will only get better.Corey: Well, the counterargument to that is, it will get far worse when we start putting this in charge [unintelligible 00:16:08] safety-critical systems, which I'm sure it's just a matter of time because some of these boosters are just very, very convincing. It's just thinking, how could this possibly go the worst? Ehhh. It's not good.Levi: Yeah, well, I mean, we're talking impact versus quality, right? The quality will only ever get better. But you know, if we run before we walk, the impact can definitely get wider.Corey: From where I sit, I want to see this really excel within bounded problem spaces. The one I keep waiting for is the AWS bill because it's a vast space, yes, and it's complicated as all hell, but it is bounded. There are a finite—though large—number of things you can see in an AWS bill, and there are recommendations you can make based on top of that. But everything I've seen that plays in this space gets way overconfident far too quickly, misses a bunch of very obvious lines of inquiry. Ah, I'm skeptical.Then you pass that off to unbounded problem spaces like human creativity and that just turns into an absolute disaster. So, much of what I've been doing lately has been hamstrung by people rushing to put in safeguards to make sure it doesn't accidentally say something horrible that it's stripped out a lot of the fun and the whimsy and the sarcasm in the approach, of I—at one point, I could bully a number of these things into ranking US presidents by absorbency. That's getting harder to do now because, “Nope, that's not respectful and I'm not going to do it,” is basically where it draws the line.Levi: The one thing that I always struggle with is, like, how much of the models are trained on intellectual property or, when you distill it down, pure like human suffering, right? Like, this is somebody's art, they've worked hard, they've suffered for it, they put it out there in the world, and now it's just been pulled in and adopted by this tool that—you know, how many of the examples of, “Give me art in the style of,” right, and you just see hundreds and hundreds of pieces that I mean, frankly, are eerily identical to the style.Corey: Even down to the signature, in some cases. Yeah.Levi: Yeah, exactly. You know, and I think that we can't lose sight of that, right? Like, these tools are fun and you know, they're fun to play with, it's really interesting to explore what's possible, but we can't lose sight of the fact that there are ultimately people behind these things.Corey: This episode is sponsored in part by Panoptica. Panoptica simplifies container deployment, monitoring, and security, protecting the entire application stack from build to runtime. Scalable across clusters and multi-cloud environments, Panoptica secures containers, serverless APIs, and Kubernetes with a unified view, reducing operational complexity and promoting collaboration by integrating with commonly used developer, SRE, and SecOps tools. Panoptica ensures compliance with regulatory mandates and CIS benchmarks for best practice conformity. Privacy teams can monitor API traffic and identify sensitive data, while identifying open-source components vulnerable to attacks that require patching. Proactively addressing security issues with Panoptica allows businesses to focus on mitigating critical risks and protecting their interests. Learn more about Panoptica today at panoptica.app.Corey: I think it matters, on some level, what the medium is. When I'm writing, I will still use turns of phrase from time to time that I first encountered when I was reading things in the 1990s. And that phrase stuck with me and became part of my lexicon. And I don't remember where I originally encountered some of these things; I just know I use those raises an awful lot. And that has become part and parcel of who and what I am.Which is also, I have no problem telling it to write a blog post in the style of Corey Quinn and then ripping a part of that out, but anything that's left in there, cool. I'm plagiarizing the thing that plagiarized from me and I find that to be one of those ethically just moments there. But written word is one thing depending on what exactly it's taking from you, but visual style for art, that's something else entirely.Levi: There's a real ethical issue here. These things can absorb far much more information than you ever could in your entire lifetime, right, so that you can only quote-unquote, you know, “Copy, borrow, steal,” from a handful of other people in your entire life, right? Whereas this thing could do hundreds or thousands of people per minute. I think that's where the calculus needs to be, right? How many people can we impact with this thing?Corey: This is also nothing new, where originally in the olden times, great, copyright wasn't really a thing because writing a book was a massive, massive undertaking. That was something that you'd have to do by hand, and then oh, you want a copy of the book? You'd have to have a scribe go and copy the thing. Well then, suddenly the printing press came along, and okay, that changes things a bit.And then we continue to evolve there to digital distribution where suddenly it's just bits on a disk that I can wind up throwing halfway around the internet. And when the marginal cost of copying something becomes effectively zero, what does that change? And now we're seeing, I think, another iteration in that ongoing question. It's a weird world and I don't know that we have the framework in place even now to think about that properly. Because every time we start to get a handle on it, off we go again. It feels like if they were doing be invented today, libraries would absolutely not be considered legal. And yet, here we are.Levi: Yeah, it's a great point. Humans just do not have the ethical framework in place for a lot of these things. You know, we saw it even with the days of Napster, right? It's just—like you said, it's another iteration on the same core problem. I [laugh] don't know how to solve it. I'm not a philosopher, right?Corey: Oh, yeah. Back in the Napster days, I was on that a fair bit in high school and college because I was broke, and oh, I wanted to listen to this song. Well, it came on an album with no other good songs on it because one-hit wonders were kind of my jam, and that album cost 15, 20 bucks, or I could grab the thing for free. There was no reasonable way to consume. Then they started selling individual tracks for 99 cents and I gorged myself for years on that stuff.And now it feels like streaming has taken over the world to the point where the only people who really lose on this are the artists themselves, and I don't love that outcome. How do we have a better tomorrow for all of this? I know we're a bit off-topic from you know, cloud management, but still, this is the sort of thing I think about when everything's running smoothly in a cloud environment.Levi: It's hard to get people to make good decisions when they're so close to the edge. And I think about when I was, you know, college-age scraping by on minimum wage or barely above minimum wage, you know, it was hard to convince me that, oh yeah, you shouldn't download an MP3 of that song; you should go buy the disc, or whatever. It was really hard to make that argument when my decision was buy an album or figure out where I'm going to, you know, get my lunch. So, I think, now that I'm in a much different place in my life, you know, these decisions are a lot easier to make in an ethical way because that doesn't impact my livelihood nearly as much. And I think that is where solutions will probably come out of. The more people doing better, the easier it is for them to make good decisions.Corey: I sure hope you're right, but something I found is that okay we made it easy for people to make good decisions. Like, “Nope, you've just made it easier for me to scale a bunch of terrible ones. I can make 300,000 more terrible decisions before breakfast time now. Thanks.” And, “No, that's not what I did that for.” Yet here we are. Have you been tracking lately what's been going on with the HashiCorp license change?Levi: Um, a little bit, we use—obviously use Terraform in the company and a couple other Hashi products, and it was kind of a wildfire of, you know, how does this impact us? We dove in and we realized that it doesn't, but it is concerning.Corey: You're not effectively wrapping Terraform and then using that as the basis for how you do MDM across your customer fleets.Levi: Yeah. You know, we're not deploying customers' written Terraform into their environments or something kind of wild like that. Yeah, it doesn't impact us. But it is… it is concerning to watch a company pivot from an open-source, community-based project to, “Oh, you can't do that anymore.” It doesn't impact a lot of people who use it day-to-day, but I'm really worried about just the goodwill that they've lit on fire.Corey: One of the problems, too, is that their entire write-up on this was so vague that it was—there is no way to get an actual… piece of is it aimed at us or is it not without very deep analysis, and hope that when it comes to court, you're going to have the same analysis as—that is sympathetic. It's, what is considered to be a competitor? At least historically, it was pretty obvious. Some of these databases, “Okay great. Am I wrapping their database technology and then selling it as a service? No? I'm pretty good.”But with HashiCorp, what they do is so vast in a few key areas that no one has the level of certainty. I was pretty freaking certain that I'm not shipping MongoDB with my own wrapper around it, but am I shipping something that looks like Terraform if I'm managing someone's environment for them? I don't know. Everything's thrown into question. And you're right. It's the goodwill that currently is being set on fire.Levi: Yeah, I think people had an impression of Hashi that they were one of the good guys. You know, the quote-unquote, “Good guys,” in the space, right? Mitchell Hashimoto is out there as a very prominent coder, he's an engineer at heart, he's in the community, pretty influential on Twitter, and I think people saw them as not one of the big, faceless corporations, so to see moves like this happen, it… I think it shook a lot of people's opinions of them and scared them.Corey: Oh, yeah. They've always been the good guys in this context. Mitch and Armon were fantastic folks. I'm sure they still are. I don't know if this is necessarily even coming from them. It's market forces, what are investors demanding? They see everyone is using Terraform. How does that compare to HashiCorp's market value?This is one of the inherent problems if I'm being direct, of the end-stages of capitalism, where it's, “Okay, we're delivering on a lot of value. How do we capture ever more of it and growing massively?” And I don't know. I don't know what the answer is, but I don't think anyone's thrilled with this outcome. Because, let's be clear, it is not going to meaningfully juice their numbers at all. They're going to be setting up a lot of ill will against them in the industry, but I don't see the upside for them. I really don't.Levi: I haven't really done any of the analysis or looked for it, I should say. Have you seen anything about what this might actually impact any providers or anything? Because you're right, like, what kind of numbers are we actually talking about here?Corey: Right. Well, there are a few folks that have done things around this that people have named for me: Spacelift being one example, Pulumi being another, and both of them are saying, “Nope, this doesn't impact us because of X, Y, and Z.” Yeah, whether it does or doesn't, they're not going to sit there and say, “Well, I guess we don't have a company anymore. Oh, well.” And shut the whole thing down and just give their customers over to HashiCorp.Their own customers would be incensed if that happened and would not go to HashiCorp if that were to be the outcome. I think, on some level, they're setting the stage for the next evolution in what it takes to manage large-scale cloud environments effectively. I think basically, every customer I've ever dealt with on my side has been a Terraform shop. I finally decided to start learning the ins and outs of it myself a few weeks ago, and well, it feels like I should have just waited a couple more weeks and then it would have become irrelevant. Awesome. Which is a bit histrionic, but still, this is going to plant seeds for people to start meaningfully competing. I hope.Levi: Yeah, I hope so too. I have always awaited releases of Terraform Cloud with great anticipation. I generally don't like managing my Terraform back-ends, you know, I don't like managing the state files, so every time Terraform Cloud has some kind of release or something, I'm looking at it because I'm excited, oh finally, maybe this is the time I get to hand it off, right? Maybe I start to get to use their product. And it has never been a really compelling answer to the problems that I have.And I've always said, like, the [laugh] cloud journey would be Google's if they just released a managed Terraform [laugh] service. And this would be one way for them to prevent that from happening. Because Google doesn't even have an Infrastructure as Code competitor. Not really. I mean, I know they have their, what, Plans or their Projects or whatever they… their Infrastructure as Code language was, but—Corey: Isn't that what Stackdriver was supposed to be? What happened with that? It's been so long.Levi: No, that's a logging solution [laugh].Corey: That's the thing. It all runs together. Not it was their operations suite that was—Levi: There we go.Corey: —formerly Stackdriver. Yeah. Now, that does include some aspects—yeah. You're right, it's still hanging out in the observability space. This is the problem is all this stuff conflates and companies are terrible at naming and Google likes to deprecate things constantly. And yeah, but there is no real competitor. CloudFormation? Please. Get serious.Levi: Hey, you're talking to a member of the CloudFormation support group here. So, I'm still a huge fan [laugh].Corey: Emotional support group, more like it, it seems these days.Levi: It is.Corey: Oh, good. It got for loops recently. We've been asking for basically that to make them a lot less wordy only for, what, ten years?Levi: Yeah. I mean, my argument is that I'm operating at the account level, right? I need to deploy to 250, 300, 500 accounts. Show me how to do that with Terraform that isn't, you know, stab your eyes out with a fork.Corey: It can be done, but it requires an awful lot of setting things up first.Levi: Exactly.Corey: That's sort of a problem. Like yeah, once you have the first 500 going, the rest are just like butter. But that's a big step one is massive, and then step two becomes easy. Yeah… no, thank you.Levi: [laugh]. I'm going to stick with my StacksSets, thank you.Corey: [laugh]. I really want to thank you for taking the time to come back on and honestly kibitz about the state of the industry with me. If people want to learn more, where's the best place for them to find you?Levi: Well, I'm still active on the space normally known as—formerly known as Twitter. You can reach out to me there. DMs are open. I'm always willing to help people learn how to cloud better. Hopefully trying to make my presence known a little bit more on LinkedIn. If you happen to be over there, reach out.Corey: And we will, of course, put links to that in the [show notes 00:30:16]. Thank you so much for taking the time to speak with me again. It's always a pleasure.Levi: Thanks, Corey. I always appreciate it.Corey: Levi McCormick, Director of Cloud Engineering at Jamf. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, and along with an insulting comment that tells us that we completely missed the forest for the trees and that your programmfing is going to be far superior based upon generative AI.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.
Anna Belak, Director of The Office of Cybersecurity Strategy at Sysdig, joins Corey on Screaming in the Cloud to discuss the findings in this year's newly-released Sysdig Global Cloud Threat Report. Anna explains the challenges that teams face in ensuring their cloud is truly secure, including quantity of data versus quality, automation, and more. Corey and Anna also discuss how much faster attacks are able to occur, and Anna gives practical insights into what can be done to make your cloud environment more secure. About AnnaAnna has nearly ten years of experience researching and advising organizations on cloud adoption with a focus on security best practices. As a Gartner Analyst, Anna spent six years helping more than 500 enterprises with vulnerability management, security monitoring, and DevSecOps initiatives. Anna's research and talks have been used to transform organizations' IT strategies and her research agenda helped to shape markets. Anna is the Director of The Office of Cybersecurity Strategy at Sysdig, using her deep understanding of the security industry to help IT professionals succeed in their cloud-native journey.Anna holds a PhD in Materials Engineering from the University of Michigan, where she developed computational methods to study solar cells and rechargeable batteries.Links Referenced: Sysdig: https://sysdig.com/ Sysdig Global Cloud Threat Report: https://www.sysdig.com/2023threatreport duckbillgroup.com: https://duckbillgroup.com TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. This promoted guest episode is brought to us by our friends over at Sysdig. And once again, I am pleased to welcome Anna Belak, whose title has changed since last we spoke to Director of the Office of Cybersecurity Strategy at Sysdig. Anna, welcome back, and congratulations on all the adjectives.Anna: [laugh]. Thank you so much. It's always a pleasure to hang out with you.Corey: So, we are here today to talk about a thing that has been written. And we're in that weird time thing where while we're discussing it at the moment, it's not yet public but will be when this releases. The Sysdig Global Cloud Threat Report, which I am a fan of. I like quite a bit the things it talks about and the ways it gets me thinking. There are things that I wind up agreeing with, there are things I wind up disagreeing with, and honestly, that makes it an awful lot of fun.But let's start with the whole, I guess, executive summary version of this. What is a Global Cloud Threat Report? Because to me, it seems like there's an argument to be made for just putting all three of the big hyperscale clouds on it and calling it a day because they're all threats to somebody.Anna: To be fair, we didn't think of the cloud providers themselves as the threats, but that's a hot take.Corey: Well, an even hotter one is what I've seen out of Azure lately with their complete lack of security issues, and the attackers somehow got a Microsoft signing key and the rest. I mean, at this point, I feel like Charlie Bell was brought in from Amazon to head cybersecurity and spent the last two years trapped in the executive washroom or something. But I can't prove it, of course. No, you target the idea of threats in a different direction, towards what people more commonly think of as threats.Anna: Yeah, the bad guys [laugh]. I mean, I would say that this is the reason you need a third-party security solution, buy my thing, blah, blah, blah, but [laugh], you know? Yeah, so we are—we have a threat research team like I think most self-respecting security vendors these days do. Ours, of course, is the best of them all, and they do all kinds of proactive and reactive research of what the bad guys are up to so that we can help our customers detect the bad guys, should they become their victims.Corey: So, there was a previous version of this report, and then you've, in long-standing tradition, decided to go ahead and update it. Unlike many of the terrible professors I've had in years past, it's not just slap a new version number, change the answers to some things, and force all the students to buy a new copy of the book every year because that's your retirement plan, you actually have updated data. What are the big changes you've seen since the previous incarnation of this?Anna: That is true. In fact, we start from scratch, more or less, every year, so all the data in this report is brand new. Obviously, it builds on our prior research. I'll say one clearly connected piece of data is, last year, we did a supply chain story that talked about the bad stuff you can find in Docker Hub. This time we upleveled that and we actually looked deeper into the nature of said bad stuff and how one might identify that an image is bad.And we found that 10% of the malware scary things inside images actually can't be detected by most of your static tools. So, if you're thinking, like, static analysis of any kind, SCA, vulnerability scanning, just, like, looking at the artifact itself before it's deployed, you actually wouldn't know it was bad. So, that's a pretty cool change, I would say [laugh].Corey: It is. And I'll also say what's going to probably sound like a throwaway joke, but I assure you it's not, where you're right, there is a lot of bad stuff on Docker Hub and part of the challenge is disambiguating malicious-bad and shitty-bad. But there are serious security concerns to code that is not intended to be awful, but it is anyway, and as a result, it leads to something that this report gets into a fair bit, which is the ideas of, effectively, lateralling from one vulnerability to another vulnerability to another vulnerability to the actual story. I mean, Capital One was a great example of this. They didn't do anything that was outright negligent like leaving an S3 bucket open; it was a determined sophisticated attacker who went from one mistake to one mistake to one mistake to, boom, keys to the kingdom. And that at least is a little bit more understandable even if it's not great when it's your bank.Anna: Yeah. I will point out that in the 10% that these things are really bad department, it was 10% of all things that were actually really bad. So, there were many things that were just shitty, but we had pared it down to the things that were definitely malicious, and then 10% of those things you could only identify if you had some sort of runtime analysis. Now, runtime analysis can be a lot of different things. It's just that if you're relying on preventive controls, you might have a bad time, like, one times out of ten, at least.But to your point about, kind of, chaining things together, I think that's actually the key, right? Like, that's the most interesting moment is, like, which things can they grab onto, and then where can they pivot? Because it's not like you barge in, open the door, like, you've won. Like, there's multiple steps to this process that are sometimes actually quite nuanced. And I'll call out that, like, one of the other findings we got this year that was pretty cool is that the time it takes to get through those steps is very short. There's a data point from Mandiant that says that the average dwell time for an attacker is 16 days. So like, two weeks, maybe. And in our data, the average dwell time for the attacks we saw was more like ten minutes.Corey: And that is going to be notable for folks. Like, there are times where I have—in years past; not recently, mind you—I have—oh, I'm trying to set something up, but I'm just going to open this port to the internet so I can access it from where I am right now and I'll go back and shut it in a couple hours. There was a time that that was generally okay. These days, everything happens so rapidly. I mean, I've sat there with a stopwatch after intentionally committing AWS credentials to Gif-ub—yes, that's how it's pronounced—and 22 seconds until the first probing attempt started hitting, which was basically impressively fast. Like, the last thing in the entire sequence was, and then I got an alert from Amazon that something might have been up, at which point it is too late. But it's a hard problem and I get it. People don't really appreciate just how quickly some of these things can evolve.Anna: Yeah. And I think the main reason, from at least what we see, is that the bad guys are into the cloud saying, right, like, we good guys love the automation, we love the programmability, we love the immutable infrastructure, like, all this stuff is awesome and it's enabling us to deliver cool products faster to our customers and make more money, but the bad guys are using all the same benefits to perpetrate their evil crimes. So, they're building automation, they're stringing cool things together. Like, they have scripts that they run that basically just scan whatever's out there to see what new things have shown up, and they also have scripts for reconnaissance that will just send a message back to them through Telegram or WhatsApp, letting them know like, “Hey, I've been running, you know, for however long and I see a cool thing you may be able to use.” Then the human being shows up and they're like, “All right. Let's see what I can do with this credential,” or with this misconfiguration or what have you. So, a lot of their initial, kind of, discovery into what they can get at is heavily automated, which is why it's so fast.Corey: I feel like, on some level, this is an unpleasant sharp shock for an awful lot of executives because, “Wait, what do you mean attackers can move that quickly? Our crap-ass engineering teams can't get anything released in less than three sprints. What gives?” And I don't think people have a real conception of just how fast bad actors are capable of moving.Anna: I think we said—actually [unintelligible 00:07:57] last year, but this is a business for them, right? They're trying to make money. And it's a little bleak to think about it, but these guys have a day job and this is it. Like, our guys have a day job, that's shipping code, and then they're supposed to also do security. The bad guys just have a day job of breaking your code and stealing your stuff.Corey: And on some level, it feels like you have a choice to make in which side you go at. And it's, like, which one of those do I spend more time in meetings with? And maybe that's not the most legitimate way to pick a job; ethics do come into play. But yeah, there's it takes a certain similar mindset, on some level, to be able to understand just how the security landscape looks from an attacker's point of view.Anna: I'll bet the bad guys have meetings too, actually.Corey: You know, you're probably right. Can you imagine the actual corporate life of a criminal syndicate? That's a sitcom in there that just needs to happen. But again, I'm sorry, I shouldn't talk about that. We're on a writer's strike this week, so there's that.One thing that came out of the report that makes perfect sense—and I've heard about it, but I haven't seen it myself and I wanted to dive into on this—specifically that automation has been weaponized in the cloud. Now, it's easy to misinterpret that the first time you read it—like I did—as, “Oh, you mean the bad guys have discovered the magic of shell scripts? No kidding.” It's more than that. You have reports of people using things like CloudFormation to stand up resources that are then used to attack the rest of the infrastructure.And it's, yeah, it makes perfect sense. Like, back in the data center days, it was a very determined attacker that went through the process of getting an evil server stuffed into a rack somewhere. But it's an API call away in cloud. I'm surprised we haven't seen this before.Anna: Yeah. We probably have; I don't know if we've documented before. And sometimes it's hard to know that that's what's happening, right? I will say that both of those things are true, right? Like the shell scripts are definitely there, and to your point about how long it takes, you know, to stopwatch, these things, on the short end of our dwell time data set, it's zero seconds. It's zero seconds from, like, A to B because it's just a script.And that's not surprising. But the comment about CloudFormation specifically, right, is we're talking about people, kind of, figuring out how to create policy in the cloud to prevent bad stuff from happening because they're reading all the best practices ebooks and whatever, watching the YouTube videos. And so, you understand that you can, say, write policy to prevent users from doing certain things, but sometimes we forget that, like, if you don't want a user to be able to attach user policy to something. If you didn't write the rule that says you also can't do that in CloudFormation, then suddenly, you can't do it in command line, but you can do it in CloudFormation. So there's, kind of, things like this, where for every kind of tool that allows this beautiful, programmable, immutable infrastructure, kind of, paradigm, you now have to make sure that you have security policies that prevent those same tools from being used against you and deploying evil things because you didn't explicitly say that you can't deploy evil things with this tool and that tool and that other tool in this other way. Because there's so many ways to do things, right?Corey: That's part of the weird thing, too, is that back when I was doing the sysadmin dance, it was a matter of taking a bunch of tools that did one thing well—or, you know, aspirationally well—and then chaining them together to achieve things. Increasingly, it feels like that's what cloud providers have become, where they have all these different services with different capabilities. One of the reasons that I now have a three-part article series, each one titled, “17 Ways to Run Containers on AWS,” adding up for a grand total of 51 different AWS services you can use to run containers with, it's not just there to make fun of the duplication of efforts because they're not all like that. But rather, each container can have bad acting behaviors inside of it. And are you monitoring what's going on across that entire threatened landscape?People were caught flat-footed to discover that, “Wait, Lambda functions can run malware? Wow.” Yes, effectively, anything that can bang two bits together and return a result is capable of running a lot of these malware packages. It's something that I'm not sure a number of, shall we say, non-forward-looking security teams have really wrapped their heads around yet.Anna: Yeah, I think that's fair. And I mean, I always want to be a little sympathetic to the folks, like, in the trenches because it's really hard to know all the 51 ways to run containers in the cloud and then to be like, oh, 51 ways to run malicious containers in the cloud. How do I prevent all of them, when you have a day job?Corey: One point that it makes in the report here is that about who the attacks seem to be targeting. And this is my own level of confusion that I imagine we can probably wind up eviscerating neatly. Back when I was running, like, random servers for me for various projects I was working on—or working at small companies—there was a school of thought in some quarters that, well, security is not that important to us. We don't have any interesting secrets. Nobody actually cares.This was untrue because a lot of these things are running on autopilot. They don't have enough insight to know that you're boring and you have to defend just like everyone else does. But then you see what can only be described as dumb attacks. Like there was the attack on Twitter a few years ago where a bunch of influential accounts tweeted about some bitcoin scam. It's like, you realize with the access you had, you had so many other opportunities to make orders of magnitude more money if you want to go down that path or to start geopolitical conflict or all kinds of other stuff. I have to wonder how much these days are attacks targeted versus well, we found an endpoint that doesn't seem to be very well secured; we're going to just exploit it.Anna: Yeah. So, that's correct intuition, I think. We see tons of opportunistic attacks, like, non-stop. But it's just, like, hitting everything, honeypots, real accounts, our accounts, your accounts, like, everything. Many of them are pretty easy to prevent, honestly, because it's like just mundane stuff, whatever, so if you have decent security hygiene, it's not a big deal.So, I wouldn't say that you're safe if you're not special because none of us are safe and none of us are that special. But what we've done here is we actually deliberately wanted to see what would be attacked as a fraction, right? So, we deployed a honey net that was indicative of what a financial org would look like or what a healthcare org would look like to see who would bite, right? And what we expected to see is that we probably—we thought the finance would be higher because obviously, that's always top tier. But for example, we thought that people would go for defense more or for health care.And we didn't see that. We only saw, like, 5% I think for health—very small numbers for healthcare and defense and very high numbers for financial services and telcos, like, around 30% apiece, right? And so, it's a little curious, right, because you—I can theorize as to why this is. Like, telcos and finance, obviously, it's where the money is, like, great [unintelligible 00:14:35] for fraud and all this other stuff, right?Defense, again, maybe people don't think defense and cloud. Healthcare arguably isn't that much in cloud, right? Like a lot of health healthcare stuff is on-premise, so if you see healthcare in cloud, maybe, you, like, think it's a honeypot or you don't [laugh] think it's worth your time? You know, whatever. Attacker logic is also weird. But yeah, we were deliberately trying to see which verticals were the most attractive for these folks. So, these attacks are infected targeted because the victim looked like the kind of thing they should be looking for if they were into that.Corey: And how does it look in that context? I mean, part of me secretly suspects that an awful lot of terrible startup names where they're so frugal they don't buy vowels, is a defense mechanism. Because you wind up with something that looks like a cat falling on a keyboard as a company name, no attacker is going to know what the hell your company does, so therefore, they're not going to target you specifically. Clearly, that's not quite how it works. But what are those signals that someone gets into an environment and says, “Ah, this is clearly healthcare,” versus telco versus something else?Anna: Right. I think you would be right. If you had, like… hhhijk as your company name, you probably wouldn't see a lot of targeted attacks. But where we're saying either the company and the name looks like a provider of that kind, and-slash-or they actually contain some sort of credential or data inside the honeypot that appears to be, like, a credential for a certain kind of thing. So, it really just creatively naming things so they look delicious.Corey: For a long time, it felt like—at least from a cloud perspective because this is how it manifested—the primary purpose of exploiting a company's cloud environment was to attempt to mine cryptocurrency within it. And I'm not sure if that was ever the actual primary approach, or rather, that was just the approach that people noticed because suddenly, their AWS bill looks a lot more like a telephone number than it did yesterday, so they can as a result, see that it's happening. Are these attacks these days, effectively, just to mine Bitcoin, if you'll pardon the oversimplification, or are they focused more on doing more damage in different ways?Anna: The analyst answer: it depends. So, again, to your point about how no one's safe, I think most attacks by volume are going to be opportunistic attacks, where people just want money. So, the easiest way right now to get money is to mine coins and then sell those coins, right? Obviously, if you have the infrastructure as a bad guy to get money in other ways, like, you could do extortion through ransomware, you might pursue that. But the overhead on ransomware is, like, really high, so most people would rather not if they can get money other ways.Now, because by volume APTs, or Advanced Persistent Threats, are much smaller than all the opportunistic guys, they may seem like they're not there or we don't see them. They're also usually better at attacking people than the opportunistic guys who will just spam everybody and see what they get, right? But even folks who are not necessarily nation states, right, like, we see a lot of attacks that probably aren't nation states, but they're quite sophisticated because we see them moving through the environment and pivoting and creating things and leveraging things that are quite interesting, right? So, one example is that they might go for a vulnerable EC2 instance—right, because maybe you have Log4J or whatever you have exposed—and then once they're there, they'll look around to see what else they can get. So, they'll pivot to the Cloud Control Plane, if it's possible, or they'll try to.And then in a real scenario we actually saw in an attack, they found a Terraform state file. So, somebody was using Terraform for provisioning whatever. And it requires an access key and this access key was just sitting in an S3 bucket somewhere. And I guess the victim didn't know or didn't think it was an issue. And so, this state file was extracted by the attacker and they found some [unintelligible 00:18:04], and they logged into whatever, and they were basically able to access a bunch of information they shouldn't have been able to see, and this turned into a data [extraction 00:18:11] scenario and some of that data was intellectual property.So, maybe that wasn't useful and maybe that wasn't their target. I don't know. Maybe they sold it. It's hard to say, but we increasingly see these patterns that are indicative of very sophisticated individuals who understand cloud deeply and who are trying to do intentionally malicious things other than just like, I popped [unintelligible 00:18:30]. I'm happy.Corey: This episode is sponsored in part by our friends at Calisti.Introducing Calisti. With Integrated Observability, Calisti provides a single pane of glass for accelerated root cause analysis and remediation. It can set, track, and ensure compliance with Service Level Objectives.Calisti provides secure application connectivity and management from datacenter to cloud, making it the perfect solution for businesses adopting cloud native microservice-based architectures. If you're running Apache Kafka, Calisti offers a turnkey solution with automated operations, seamless integrated security, high-availability, disaster recovery, and observability. So you can easily standardize and simplify microservice security, observability, and traffic management. Simplify your cloud-native operations with Calisti. Learn more about Calisti at calisti.app.Corey: I keep thinking of ransomware as being a corporate IT side of problem. It's a sort of thing you'll have on your Windows computers in your office, et cetera, et cetera, despite the fact that intellectually I know better. There were a number of vendors talking about ransomware attacks and encrypting data within S3, and initially, I thought, “Okay, this sounds like exactly a story people would talk about some that isn't really happening in order to sell their services to guard against it.” And then AWS did a blog post saying, “We have seen this, and here's what we have learned.” It's, “Oh, okay. So, it is in fact real.”But it's still taking me a bit of time to adapt to the new reality. I think part of this is also because back when I was hands-on-keyboard, I was unlucky, and as a result, I was kept from taking my aura near anything expensive or long-term like a database, and instead, it's like, get the stateless web servers. I can destroy those and we'll laugh and laugh about it. It'll be fine. But it's not going to destroy the company in the same way. But yeah, there are a lot of important assets in cloud that if you don't have those assets, you will no longer have a company.Anna: It's funny you say that because I became a theoretical physicist instead of experimental physicist because when I walked into the room, all the equipment would stop functioning.Corey: Oh, I like that quite a bit. It's one of those ideas of, yeah, your aura just winds up causing problems. Like, “You are under no circumstances to be within 200 feet of the SAN. Is that clear?” Yeah, same type of approach.One thing that I particularly like that showed up in the report that has honestly been near and dear to my heart is when you talk about mitigations around compromised credentials at one point when GitHub winds up having an AWS credential, AWS has scanners and a service that will catch that and apply a quarantine policy to those IAM credentials. The problem is, is that policy goes nowhere near far enough at all. I wound up having fun thought experiment a while back, not necessarily focusing on attacking the cloud so much as it was a denial of wallet attack. With a quarantined key, how much money can I cost? And I had to give up around the $26 billion dollar mark.And okay, that project can't ever see the light of day because it'll just cause grief for people. The problem is that the mitigations around trying to list the bad things and enumerate them mean that you're forever trying to enumerate something that is innumerable in and of itself. It feels like having a hard policy of once this is compromised, it's not good for anything would be the right answer. But people argue with me on that.Anna: I don't think I would argue with you on that. I do think there are moments here—again, I have to have sympathy for the folks who are actually trying to be administrators in the cloud, and—Corey: Oh God, it's hard.Anna: [sigh]. I mean, a lot of the things we choose to do as cloud users and cloud admins are things that are very hard to check for security goodness, if you will, right, like, the security quality of the naming convention of your user accounts or something like that, right? One of the things we actually saw in this report it—and it almost made me cry, like, how visceral my reaction was to this thing—is, there were basically admin accounts in this cloud environment, and they were named according to a specific convention, right? So, if you were, like, admincorey and adminanna, like, that, if you were an admin, you've got an adminanna account, right? And then there was a bunch of rules that were written, like, policies that would prevent you from doing things to those accounts so that they couldn't be compromised.Corey: Root is my user account. What are you talking about?Anna: Yeah, totally. Yeah [laugh]. They didn't. They did the thing. They did the good accounts. They didn't just use root everybody. So, everyone had their own account, it was very neat. And all that happened is, like, one person barely screwed up the naming of their account, right? Instead of a lowercase admin, they use an uppercase Admin, and so all of the policy written for lowercase admin didn't apply to them, and so the bad guy was able to attach all kinds of policies and basically create a key for themselves to then go have a field day with this admin account that they just found laying around.Now, they did nothing wrong. It's just, like, a very small mistake, but the attacker knew what to do, right? The attacker went and enumerated all these accounts or whatever, like, they see what's in the environment, they see the different one, and they go, “Oh, these suckers created a convention, and like, this joker didn't follow it. And I've won.” Right? So, they know to check with that stuff.But our guys have so much going on that they might forget, or they might just you know, typo, like, whatever. Who cares. Is it case-sensitive? I don't know. Is it not case-sensitive? Like, some policies are, some policies aren't. Do you remember which ones are and which ones aren't? And so, it's a little hopeless and painful as, like, a cloud defender to be faced with that, but that's sort of the reality.And right now we're in kind of like, ah, preventive security is the way to save yourself in cloud mode, and these things just, like, they don't come up on, like, the benchmarks and, like the configuration checks and all this other stuff that's just going, you know, canned, did you, you know, put MFA on your user account? Like, yeah, they did, but [laugh] like, they gave it a wrong name and now it's a bad na—so it's a little bleak.Corey: There's too much data. Filtering it becomes nightmarish. I mean, I have what I think of as the Dependabot problem, where every week, I get this giant list of Dependabot freaking out about every repository I have on Gif-ub and every dependency thereof. And some of the stuff hasn't been deployed in years and I don't care. Other stuff is, okay, I can see how that markdown parser could have malicious input passed to it, but it's for an internal project that only ever has very defined things allowed to talk to it so it doesn't actually matter to me.And then at some point, it's like, you expect to read, like, three-quarters of the way down the list of a thousand things, like, “Oh, and by the way, the basement's on fire.” And then have it keep going on where it's… filtering the signal from noise is such a problem that it feels like people only discover the warning signs after they're doing forensics when something has already happened rather than when it's early enough to be able to fix things. How do you get around that problem?Anna: It's brutal. I mean, I'm going to give you, like, my [unintelligible 00:24:28] vendor answer: “It's just easy. Just do what we said.” But I think [laugh] in all honesty, you do need to have some sort of risk prioritization. I'm not going to say I know the answer to what your algorithm has to be, but our approach of, like, oh, let's just look up the CVSS score on the vulnerabilities. Oh, look, 600,000 criticals. [laugh]. You know, you have to be able to filter past that, too. Like, is this being used by the application? Like, has this thing recently been accessed? Like, does this user have permissions? Have they used those permissions?Like, these kinds of questions that we know to ask, but you really have to kind of like force the security team, if you will, or the DevOps team or whatever team you have to actually, instead of looking at the list and crying, being like, how can we pare this list down? Like anything at all, just anything at all. And do that iteratively, right? And then on the other side, I mean, it's so… defense-in-depth, like, right? I know it's—I'm not supposed to say that because it's like, not cool anymore, but it's so true in cloud, like, you have to assume that all these controls will fail and so you have to come up with some—Corey: People will fail, processes will fail, controls will fail, and great—Anna: Yeah.Corey: How do you make sure that one of those things failing isn't winner-take-all?Anna: Yeah. And so, you need some detection mechanism to see when something's failed, and then you, like, have a resilience plan because you know, if you can detect that it's failed, but you can't do anything about it, I mean, big deal, [laugh] right? So detection—Corey: Good job. That's helpful.Anna: And response [laugh]. And response. Actually, mostly response yeah.Corey: Otherwise, it's, “Hey, guess what? You're not going to believe this, but…” it goes downhill from there rapidly.Anna: Just like, how shall we write the news headline for you?Corey: I have to ask, given that you have just completed this report and are absolutely in a place now where you have a sort of bird's eye view on the industry at just the right time, over the past year, we've seen significant macro changes affect an awful lot of different areas, the hiring markets, the VC funding markets, the stock markets. How has, I guess, the threat space evolved—if at all—during that same timeframe?Anna: I'm guessing the bad guys are paying more than the good guys.Corey: Well, there is part of that and I have to imagine also, crypto miners are less popular since sanity seems to have returned to an awful lot of people's perspective on money.Anna: I don't know if they are because, like, even fractions of cents are still cents once you add up enough of them. So, I don't think [they have stopped 00:26:49] mining.Corey: It remains perfectly economical to mine Bitcoin in the cloud, as long as you use someone else's account to do it.Anna: Exactly. Someone else's money is the best kind of money.Corey: That's the VC motto and then some.Anna: [laugh]. Right? I think it's tough, right? I don't want to be cliche and say, “Look, oh automate more stuff.” I do think that if you're in the security space on the blue team and you are, like, afraid of losing your job—you probably shouldn't be afraid if you do your job at all because there's a huge lack of talent, and that pool is not growing quick enough.Corey: You might be out of work for dozens of minutes.Anna: Yeah, maybe even an hour if you spend that hour, like, not emailing people, asking for work. So yeah, I mean, blah, blah, skill up in cloud, like, automate, et cetera. I think what I said earlier is actually the more important piece, right? We have all these really talented people sitting behind these dashboards, just trying to do the right thing, and we're not giving them good data, right? We're giving them too much data and it's not good quality data.So, whatever team you're on or whatever your business is, like, you will have to try to pare down that list of impossible tasks for all of your cloud-adjacent IT teams to a list of things that are actually going to reduce risk to your business. And I know that's really hard to do because you're asking now, folks who are very technical to communicate with folks who are very non-technical, to figure out how to, like, save the business money and keep the business running, and we've never been good at this, but there's no time like the present to actually get good at it.Corey: Let's see, what is it, the best time to plant a tree was 20 years ago. The second best time is now. Same sort of approach. I think that I'm seeing less of the obnoxious whining that I saw for years about how there's a complete shortage of security professionals out there. It's, “Okay, have you considered taking promising people and training them to do cybersecurity?” “No, that will take six months to get them productive.” Then they sit there for two years with the job rec open. It's hmm. Now, I'm not a professor here, but I also sort of feel like there might be a solution that benefits everyone. At least that rhetoric seems to have tamped down.Anna: I think you're probably right. There's a lot of awesome training out there too. So there's, like, folks giving stuff away for free that's super resources, so I think we are doing a good job of training up security folks. And everybody wants to be in security because it's so cool. But yeah, I think the data problem is this decade's struggle, more so than any other decades.Corey: I really want to thank you for taking the time to speak with me. If people want to learn more, where can they go to get their own copy of the report?Anna: It's been an absolute pleasure, Corey, and thanks, as always for having us. If you would like to check out the report—which you absolutely should—you can find it ungated at www.sysdig.com/2023threatreport.Corey: You had me at ungated. Thank you so much for taking the time today. It's appreciated. Anna Belak, Director of the Office of Cybersecurity Strategy at Sysdig. This promoted guest episode has been brought to us by our friends at Sysdig and I'm Cloud Economist Corey Quinn.If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with an insulting comment that no doubt will compile into a malicious binary that I can grab off of Docker Hub.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.
Andreas Wittig, Co-Author of Amazon Web Services in Action and Co-Founder of marbot, joins Corey on Screaming in the Cloud to discuss ways to keep a book up to date in an ever-changing world, the advantages of working with a publisher, and how he began the journey of writing a book in the first place. Andreas also recalls how much he learned working on the third edition of Amazon Web Services in Action and how teaching can be an excellent tool for learning. Since writing the first edition, Adreas's business has shifted from a consulting business to a B2B product business, so he and Corey also discuss how that change came about and the pros and cons of each business model. About AndreasAndreas is the Co-Author of Amazon Web Services in Action and Co-Founder of marbot - AWS Monitoring made simple! He is also known on the internet as cloudonaut through the popular blog, podcast, and youtube channel he created with his brother Michael. Links Referenced: Amazon Web Services in Action: https://www.amazon.com/Amazon-Services-Action-Andreas-Wittig/dp/1617295116 Rapid Docker on AWS: https://cloudonaut.io/rapid-docker-on-aws/ bucket/av: https://bucketav.com/ marbot: https://marbot.io/ cloudonaut.io: https://cloudonaut.io TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. It's been a few years since I caught up with Andreas Wittig, who is also known in the internet as cloudonaut, and much has happened since then. Andreas, how are you?Andreas: Hey, absolutely. Thank you very much. I'm happy to be here in the show. I'm doing fine.Corey: So, one thing that I have always held you in some high regard for is that you have done what I have never had the attention span to do: you wrote a book. And you published it a while back through Manning, it was called Amazon Web Services in Action. That is ‘in action' two words, not Amazon Web Services Inaction of doing absolutely nothing about it, which is what a lot of companies in the space seem to do instead.Andreas: [laugh]. Yeah, absolutely. So. And it was not only me. I've written the book together with my brother because back in 2015, Manning, for some reason, wrote in and asked us if we would be interested in writing the book.And we had just founded our own consulting company back then and we had—we didn't have too many clients at the very beginning, so we had a little extra time free. And then we decided, okay, let's do the book. And let's write a book about Amazon Web Services, basically, a deep introduction into all things AWS. So, this was 2015, and it was indeed a lot of work, much more [laugh] than we expected. So, first of all, the hard part is, what do you want to have in the book? So, what's the TOC? What is important and must be in?And then you start writing and have examples and everything. So, it was really an interesting journey. And doing it together with a publisher like Manning was also really interesting because we learned a lot about writing. You have kind of a coach, an editor that helps you through that process. So, this was really a hard and fun experience.Corey: There's a lot of people that have said very good things about writing the book through a traditional publisher. And they also say that one of the challenges is it's a blessing and a curse, where you basically have someone standing over your shoulder saying, “Is it done yet? Is it done yet? Is it done yet?” The consensus that seems to have emerged from people who have written books is, “That was great, please don't ever ask me to do it again.”And my operating theory is that no one wants to write a book. They want to have written a book. Which feels like two very different things most of the time. But the reason you're back on now is that you have gone the way of the terrible college professor, where you're going to update the book, and therefore you get to do a whole new run of textbooks and make everyone buy it and kill the used market, et cetera. And you've done that twice now because you have just recently released the third edition. So, I have to ask, how different is version one from version two and from version three? Although my apologies; we call them ‘editions' in the publishing world.Andreas: [laugh]. Yeah, yeah. So, of course, as you can imagine, things change a lot in AWS world. So, of course, you have to constantly update things. So, I remember from first to second edition, we switched from CloudFormation in JSON to YAML. And now to the third edition, we added two new chapters. This was also important to us, so to keep also the scope of the book in shape.So, we have in the third edition, two new chapters. One is about automating deployments, recovering code deploy, [unintelligible 00:03:59], CloudFormation rolling updates in there. And then there was one important topic missing at all in the book, which was containers. And we finally decided to add that in, and we have now container chapter, starting with App Runner, which I find quite an interesting service to observe right now, and then our bread and butter service: ECS and Fargate. So, that's basically the two new chapters. And of course, then reworking all the other chapters is also a lot of work. And so, many things change over time. Cannot imagine [laugh].Corey: When was the first edition released? Because I believe the second one was released in 2018, which means you've been at this for a while.Andreas: Yeah. So, the first was 2015, the second 2018, three years later, and then we had five years, so now this third edition was released at the beginning of this year, 2023.Corey: Eh, I think you're right on schedule. Just March of 2020 lasted three years. That's fine.Andreas: Yeah [laugh].Corey: So, I have to ask, one thing that I've always appreciated about AWS is, it feels like with remarkably few exceptions, I can take a blog post written on how to do something with AWS from 2008 and now in 2023, I can go through every step along with that blog post. And yeah, I might have trouble getting some of the versions and services and APIs up and running, but the same steps will absolutely work. There are very few times where a previously working API gets deprecated and stops working. Is this the best way to proceed? Absolutely not.But you can still spin up the m1.medium instance sizes, or whatever it was, or [unintelligible 00:05:39] on small or whatever the original only size that you could get was. It's just there are orders of magnitude and efficiency gains you can do by—you can go through by using more modern approaches. So, I have to ask, was there anything in the book as you revised it—two times now—that needed to come out because it was now no longer working?Andreas: So, related to the APIs that's—they are really very stable, you're right about that. So, the problem is, our first few chapters where we have screenshots of how you go through the management—Corey: Oh no.Andreas: —console [laugh]. And you can probably, you can redo them every three months, probably, because the button moves or a step is included or something like that. So, the later chapters in the book, where we focus a lot on the CLI or CloudFormation and stuff like—or SDKs, they are pretty stable. But the first few [ones 00:06:29] are a nightmare to update all those screenshots. And then sometimes, so I was going through the book, and then I noticed, oh, there's a part of this chapter that I can completely remove nowadays.So, I can give you an example. So, I was going through the chapter about Simple Storage Service S3, and I—there was a whole section in the chapter about read-after-write consistency. Because back then, it was important that you knew that after updating an object or reading an object before it was created the first time, you could get outdated versions for a little while, so this was eventually consistent. But nowadays, AWS has changed that and basically now, S3 has this strong read-after-write consistency. So, I basically could remove that whole part in the chapter which was quite complicated to explain to the reader, right, so I [laugh] put a lot of effort into that.Corey: You think that was confusing? I look at the sea of systems I had to oversee at one company, specifically to get around that problem. It's like, well, we can now take this entire application and yeet it into the ocean because it was effectively a borderline service to that just want to ens—making consistency guarantees. It's not a common use case, but it is one that occurs often enough to be a problem. And of course, when you need it, you really need it. That was a nice under-the-hood change that was just one day, surprise, it works that way. But I'm sure it was years of people are working behind the scenes, solving for impossible problems to get there, and cetera, et cetera.Andreas: Yeah, yeah. But that's really cool is to remove parts of the book that are now less complicated. This is really cool. So, a few other examples. So, things change a lot. So, for example, EFS, so we have EFS, Elastic File System, in the book as well. So, now we have new throughput modes, different limits. So, there's really a lot going on and you have to carefully go through all the—Corey: Oh, when EFS launched, it was terrible. Now, it's great just because it's gotten so much more effective and efficient as a service. It's… AWS releases things before they're kind of ready, it feels like sometimes, and then they improve with time. I know there have been feature deprecations. For example, for some reason, they are no longer allowing us to share out a bucket via BitTorrent, which, you know, in 2006 when it came out, seemed like a decent idea to save on bandwidth. But here in 2023, no one cares about it.But I'm also keeping a running list of full-on AWS services that have been deprecated or have the deprecations announced. Are any of those in the book in any of its editions? And if and when there's a fourth edition, will some of those services have to come out?Andreas: [laugh]. Let's see. So, right after the book was published—because the problem with books is they get printed, right; that's the issue—but the target of the book, AWS, changes. So, a few weeks after the printed book was out, we found out that we have an issue in our one of our examples because now S3 buckets, when you create them, they have locked public access enabled by default. And this was not the case before. And one of our example relies on that it can create object access control lists, and this is not working now anymore. [laugh].So yeah, there are things changing. And we have, the cool thing about Manning is they have that what they call a live book, so you can read it online and you can have notes from other readers and us as the authors along the text, and there we can basically point you in the right direction and explain what happened here. So, this is how we try to keep the book updated. Of course, the printed one stays the same, but the ebook can change over time a little bit.Corey: Yes, ebooks are… at least keeping them updated is a lot easier, I would imagine. It feels like that—speaking of continuous builds and automatic CI/CD approaches—yeah, well, we could build a book just by updating some text in a Git repo or its equivalent, and pressing go, but it turns out that doing a whole new print run takes a little bit more work.Andreas: Yeah. Because you mentioned the experience of writing a book with a publisher and doing it on your own with self-publishing, so we did both in the past. We have Amazon Web Services in Action with Manning and we did another book, Rapid Docker on AWS in self-publishing. And what we found out is, there's really a lot of effort that goes into typesetting and layouting a book, making sure it looks consistent.And of course, you can just transform some markdown into a epub and PDF versions, but if a publisher is doing that, the results are definitely different. So, that was, besides the other help that we got from the publisher, very helpful. So, we enjoyed that as well.Corey: What is the current state of the art—since I don't know the answer to this one—around updating ebook versions? If I wind up buying an ebook on Kindle, for example, will they automatically push errata down automatically through their system, or do they reserve that for just, you know, unpublishing books that they realized shouldn't be on the Marketplace after people have purchased them?Andreas: [laugh]. So—Corey: To be fair, that only happened once, but I'm still giving them grief for it a decade and change later. But it was 1984. Of all the books to do that, too. I digress.Andreas: So, I'm not a hundred percent sure how it works with the Kindle. I know that Manning pushes out new versions by just emailing all the customers who bought the book and sending them a new version. Yeah.Corey: Yeah. It does feel, on some level, like there needs to be at least a certain degree of substantive change before they're going to start doing that. It's like well, good news. There was a typo on page 47 that we're going to go ahead and fix now. Two letters were transposed in a word. Now, that might theoretically be incredibly important if it's part of a code example, which yes, send that out, but generally, A, their editing is on point, so I didn't imagine that would sneak through, and 2, no one cares about a typo release and wants to get spammed over it?Andreas: Definitely, yeah. Every time there's a reprint of the book, you have the chance to make small modifications, to add something or remove something. That's also a way to keep it in shape a little bit.Corey: I have to ask, since most people talk about AWS services to a certain point of view, what is your take on databases? Are you sticking to the actual database services or are you engaged in my personal hobby of misusing everything as a database by holding it wrong?Andreas: [laugh]. So, my favorite database for starting out is DynamoDB. So, I really like working with DynamoDB and I like the limitations and the thing that you have to put some thoughts into how to structure your data set in before. But we also use a lot of Aurora, which really find an interesting technology. Unfortunately, Aurora Serverless, it's not becoming a product that I want to use. So, version one is now outdated, version two is much too expensive and restricted. So—Corey: I don't even know that it's outdated because I'm seeing version one still get feature updates to it. It feels like a divergent service. That is not what I would expect a version one versus version two to be. I'm with you on Dynamo, by the way. I started off using that and it is cheap is free for most workloads I throw at it. It's just a great service start to finish. The only downside is that if I need to move it somewhere else, then I have a problem.Andreas: That's true. Yeah, absolutely.Corey: I am curious, as far as you look across the sea of change—because you've been doing this for a while and when you write a book, there's nothing that I can imagine that would be better at teaching you the intricacies of something like AWS than writing a book on it. I got a small taste of this years ago when I shot my mouth off and committed to give a talk about Git. Well, time to learn Git. And teaching it to other people really solidifies a lot of the concepts yourself. Do you think that going through the process of writing this book has shaped how you perform as an engineer?Andreas: Absolutely. So, it's really interesting. So,I added the third edition and I worked on it mostly last year. And I didn't expect to learn a lot during that process actually, because I just—okay, I have to update all the examples, make sure everything work, go through the text, make sure everything is up to date. But I learned things, not only new things, but I relearned a lot of things that I wasn't aware of anymore. Or maybe I've never been; I don't know exactly [laugh].But it's always, if you go into the details and try to explain something to others, you learn a lot about that. So, teaching is a very good way to, first of all gather structure and a deep understanding of a topic and also dive into the details. Because when you write a book, every time you write a sentence, ask the question, is that really correct? Do I really know that or do I just assume that? So, I check the documentation, try to find out, is that really the case or is that something that came up myself?So, you'll learn a lot by doing that. And always come to the limits of the AWS documentation because sometimes stuff is just not documented and you need to figure out, what is really happening here? What's the real deal? And then this is basically the research part. So, I always find that interesting. And I learned a lot in during the third edition, while was only adding two new chapters and rewriting a lot of them. So, I didn't expect that.Corey: Do you find that there has been an interesting downstream effect from having written the book, that for better or worse, I've always no—I always notice myself responding to people who have written a book with more deference, more acknowledgment for the time and effort that it takes. And some books, let's be clear, are terrible, but I still find myself having that instinctive reaction because they saw something through to be published. Have you noticed it changing other aspects of your career over the past, oh, dear Lord, it would have been almost ten years now.Andreas: So, I think it helped us a lot with our consulting business, definitely. Because at the very beginning, so back in 2015, at least here in Europe and Germany, AWS was really new in the game. And being the one that has written a book about AWS was really helping… stuff. So, it really helped us a lot for our consulting work. I think now we are into that game of having to update the book [laugh] every few years, to make sure it stays up to date, but I think it really helped us for starting our consulting business.Corey: And you've had a consulting business for a while. And now you have effectively progressed to the next stage of consulting business lifecycle development, which is, it feels like you're becoming much more of a product company than you were in years past. Is that an accurate perception from the outside or am I misunderstanding something fundamental?Andreas: You know, absolutely, that's the case. So, from the very beginning, basically, when we founded our company, so eight years ago now, so we always had to go to do consulting work, but also do product work. And we had a rule of thumb that 20% of our time goes into product development. And we tried a lot of different things. So, we had just a few examples that failed completely.So, we had a Time [Series 00:17:49] as a Service offering at the very beginning of our journey, which failed completely. And now we have Amazon Timestream, which makes that totally—so now the market is maybe there for that. We tried a lot of things, tried content products, but also as we are coming from the software development world, we always try to build products. And over the years, we took what we learned from consulting, so we learned a lot about, of course, AWS, but also about the market, about the ecosystem. And we always try to bring that into the market and build products out of that.So nowadays, we really transitioned completely from consulting to a product company, as you said. So, we do not do any consulting anymore with one few exception with one of our [laugh] best or most important clients. But we are now a product company. And we only a two-person company. So, the idea was always how to scale a company without growing the team or hiring a lot of people, and a consulting business is definitely not a good way to do that, so yeah, this was why always invested into products.And now we have two products in the AWS Marketplace which works very well for us because it allows us to sell worldwide and really easily get a relationship up and running with our customers, and that pay through their AWS bill. So, that's really helping us a lot. Yeah.Corey: A few questions on that. At first it always seems to me that writing software or building a product is a lot like real estate in that you're doing a real estate development—to my understanding since I live in San Francisco and this is a [two exit 00:19:28] town; I still rent here—I found though, that you have to spend a lot of money and effort upfront and you don't get to start seeing revenue on that for years, which is why the VC model is so popular where you'll take $20 million, but then in return they want to see massive, outsized returns on that, which—it feels—push an awful lot of perfectly sustainable products into things that are just monstrous.Andreas: Hmm, yeah. Definitely.Corey: And to my understanding, you bootstrapped. You didn't take a bunch of outside money to do this, right?Andreas: No, no, we have completely bootstrapping and basically paying the bills with our consulting work. So yeah, I can give you one example. So, bucketAV is our solution to scan S3 buckets for malware, and basically, this started as an open-source project. So, this was just a side project we are working on. And we saw that there is some demand for that.So, people need ways to scan their objects—for example, user uploads—for malware, and we just tried to publish that in the AWS Marketplace to sell it through the Marketplace. And we don't really expect that this is a huge deal, and so we just did, I don't know, Michael spent a few days to make sure it's possible to publish that and get in shape. And over time, this really grew into an important, really substantial part of our business. And this doesn't happen overnight. So, this adds up, month by month. And you get feedback from customers, you improve the product based on that. And now this is one of the two main products that we sell in the Marketplace.Corey: I wanted to ask you about the Marketplace as well. Are you finding that that has been useful for you—obviously, as a procurement vehicle, it means no matter what country a customer is in, they can purchase it, it shows up on the AWS bill, and life goes on—but are you finding that it has been an effective way to find new customers?Andreas: Yes. So, I definitely would think so. It's always funny. So, we have completely inbound sales funnel. So, all customers find us through was searching the Marketplace or Google, probably. And so, what I didn't expect that it's possible to sell a B2B product that way. So, we don't know most of our customers. So, we know their name, we know the company name, but we don't know anyone there. We don't know the person who buys the product.This is, on the one side, a very interesting thing as a two-person company. You cannot build a huge sales process and I cannot invest too much time into the sales process or procurement process, so this really helps us a lot. The downside of it is a little bit that we don't have a close relationship with our customers and sometimes it's a little tricky for us to find important person to talk to, to get feedback and stuff. But on the other hand, yeah, it really helps us to sell to businesses all over the world. And we sell to very small business of course, but also to large enterprise customers. And they are fine with that process as well. And I think, even the large enterprises, they enjoy that it's so easy [laugh] to get a solution up and running and don't have to talk to any salespersons. So, enjoy it and I think our customers do as well.Corey: This is honestly the first time I've ever heard a verifiable account a vendor saying, “Yeah, we put this thing on the Marketplace, and people we've never talked to find us on the Marketplace and go ahead and buy.” That is not the common experience, let's put it that way. Now true, an awful lot of folks are selling enterprise software on this and someone—I forget who—many years ago had a great blog post on why no enterprise software costs $5,000. It either is going to cost $500 or it's going to cost 100 grand and up because the difference is, is at some point, you'd have a full-court press enterprise sales motion to go and sell the thing. And below a certain point, great, people are just going to be able to put it on their credit card and that's fine. But that's why you have this giant valley of there is very little stuff priced in that sweet spot.Andreas: Yeah. So, I think maybe it's important to mention that our products are relatively simple. So, they are just for a very small niche, a solution for a small problem. So, I think that helps a lot. So, we're not selling a full-blown cloud security solution; we only focus on that very small part: scanning S3 objects for malware.For example, on marbot,f the other product that we sell, which is monitoring of AWS accounts. Again, we focus on a very simple way to monitor AWS workloads. And so, I think that is probably why this is a successful way for us to find new customers because it's not a very complicated product where you have to explain a lot. So, that's probably the differentiator here.Corey: Having spent a fair bit of time doing battle with compliance goblins—which is, to be clear, I'm not describing people; I'm describing processes—in many cases, we had to do bucket scanning for antivirus, just to check a compliance box. From our position, there was remarkably little risk of a user-generated picture of a receipt that is input sanitized to make sure it is in fact a picture, landing in an S3 bucket and then somehow infecting one of the Linux servers through which it passed. So, we needed something that just checked the compliance box or we would not be getting the gold seal on our website today. And it was, more or less, a box-check as opposed to something that solved a legitimate problem. This was also a decade and change ago. Has that changed to a point now where there are legitimate threats and concerns around this, or is it still primarily just around make the auditor stop yelling at me, please?Andreas: Mmm. I think it's definitely to tick the checkbox, to be compliant with this, some regulation. On the other side, I think there are definitely use cases where it makes a lot of sense, especially when it comes to user-generated content of all kinds, especially if you're not only consuming it internally, but maybe also others can immediately start downloading that. So, that is where we see many of our customers are coming with that scenario that they want to make sure that the files that people upload and others can download are not infected. So, that is probably the most important use case.Corey: There's also, on some level, an increasing threat of ransomware. And for a long time, I was very down on the ideas of all these products that hit the market to defend S3 buckets against ransomware. Until one day, there was an AWS security blog post talking about how they found it. And yeah, we've we have seen this in the wild; it is causing problems for companies; here's what to do about it. Because it's one of those areas where I can't trust a vendor who's trying to sell me something to tell me that this problem exists.I mean, not to cast aspersions, but they're very interested, they're very incentivized to tell that story, whereas AWS is not necessarily incentivized to tell a story like that. So, that really brought it home for me that no, this is a real thing. So, I just want to be clear that my opinion on these things does in fact, evolve. It's not, “Well, I thought it was dumb back in 2012, so clearly it's still dumb now.” That is not my position, I want to be very clear on that.I do want to revisit for a moment, the idea of going from a consultancy that is a services business over to a product business because we've toyed with aspects of that here at The Duckbill Group a fair bit. We've not really found long-term retainer services engagements that add value that we are comfortable selling. And that means as a result that when you sell fixed duration engagements, it's always a sell, sell, sell, where's the next project coming from? Whereas with product businesses, it's oh, the grass is always greener on the other side. It's recurring revenue. Someone clicks, the revenue sticks around and never really goes away. That's the dream from where I sit on the services side of the fence, wistfully looking across and wondering what if. Now that you've made that transition, what sucks about product businesses that you might not have seen going into it?Andreas: [laugh]. Yeah, that a good question. So, on the one side, it was really also our dream to have a product business because it really changes the way we work. We can block large parts of our calendar to do deep-focus work, focus on things, find new solutions, and really try to make a solution that really fits to problem and uses all the AWS capabilities to do so. And on the other side, a product business involves, of course, selling the product, which is hard.And we are two software engineers, [laugh] and really making sure that we optimize our sales and there's search engine optimization, all that stuff, this is really hard for us because we don't know anything about that and we always have to find an expert, or we need to build a knowledge ourself, try things out, and so on. So, that whole part of selling the product, this is really a challenge for us. And then of course, product business evolves a lot of support work. So, we get support emails multiple times per hour, and we have to answer them and be as fast as possible with that. So, that is, of course, something that you do not have to do with consulting work.And not always that, the questions are many times really simple questions that pointed people in the right direction, find part of the documentation that answers the question. So, that is a constant stream of questions coming in that you have to answer. So, the inbox is always full [laugh]. So, that is maybe a small downside of a product business. But other than that, yeah, compared to a consulting business, it really gives us many flexibilities with planning our work day around the rest of our lives. That's really what we enjoy about a product company.Corey: I was very careful to pick an expensive problem that was only a business-hours problem. So, I don't wind up with a surprise, middle-of-the-night panic phone call. It's yeah, it turns out that AWS billing operate during business hours in the US Pacific Time. The end. And there are no emergencies here; there are simply curiosities that will, in the fullness of time take weeks to get resolved.Andreas: Mmm. Yeah.Corey: I spent too many years on call, in that sense. Everyone who's built a product company the first time always says the second time, the engineering? Meh, there are ways to solve that. Solving the distribution problem. That's the thing I want to focus on next.And I feel like I sort of went into this backwards in that I don't really have a product to sell people but I somehow built an audience. And to be honest, it's partly why. It's because I didn't know what I was going to be doing after 18 months and I knew that whatever it was going to be, I needed an audience to tell about it, so may as well start the work of building the audience now. So, I have to imagine if nothing else, your book has been a tremendous source of building a community. When I mentioned the word cloudonaut to people who have been learning AWS, more often than not, they know who you are.Andreas: Yeah.Corey: Although I admit they sometimes get you confused with your brother.Andreas: [laugh]. Yes, that's not too hard. Yeah, yeah, cloudonaut is definitely—this was always our, also a side project of we was just writing about things that we learned about AWS. Whenever we, I don't know, for example, looked into a new series, we wrote a blog post about that. Later, we did start a podcast and YouTube videos during the pandemic, of course, as everyone did. And so, I think this was always fun stuff to do. And we like sharing what we learn and getting into discussion with the community, so this is what we still do and enjoy as well, of course. Yeah.Corey: I really want to thank you for taking the time to catch up and see what you've been up to these last few years with a labor of love and the pivot to a product company. If people want to learn more, where's the best place for them to find you?Andreas: So definitely, the best place to find me is cloudonaut.io. So, this basically points you to all [laugh] what I do. Yeah, that's basically the one domain and URL that you need to know.Corey: Excellent. And we will put that in the show notes, of course. Thank you so much for taking the time to speak with me today. I really appreciate it.Andreas: Yeah, it was a pleasure to be back here. I'm big fan of podcasts and also of Screaming in the Cloud, of course, so it was a pleasure to be here again.Corey: [laugh]. You are always welcome. Andreas Wittig, co-author of Amazon Web Services in Action, now up to its third edition. And of course, the voice behind cloudonaut. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with an angry, insulting comment that I will at one point be able to get to just as soon as I find something to deal with your sarcasm on the AWS Marketplace.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.
Welcome to the epic tale of AWS Bites! In this chapter, we embark on a perilous journey through the challenges of developing distributed applications on AWS. We encounter fierce foes in the form of deployment times and limited access to real AWS services during local development. But fear not, for we have powerful tools at our disposal, including the legendary LocalStack and Serverless offline. And if that's not enough, we have tips and tricks for optimizing our development flow without local simulations, using well-structured code and unit tests. We even share CloudFormation tricks to speed up deployment times and reveal the secret of speeding up the development of IAM policies with Session Policies. So grab your swords and join us on this epic adventure to overcome the challenges of local development on AWS!
Welcome to the newest episode of The Cloud Pod podcast - where the forecast is always cloudy! Today your hosts Justin, Jonathan, and Matt discuss all things cloud and AI, as well as some really interesting forays into quantum computing, changes to Google domains, Google accusing Microsoft of cloud monopoly shenanigans, and the fact that Azure wants all your industry secrets. Also, Finops and all the logs you could hope for. Are your secrets safe? Better tune in and find out! Titles we almost went with this week: The Cloud Pod Adds Domains to the Killed by Google list The Cloud Pod Whispers it's Secrets to Azure OpenAI The Cloud Pod Accuses the Cloud of Being a Monopoly The Cloud Pod Does Not Pass Go and Does Not collect $200 A big thanks to this week's sponsor: Foghorn Consulting, provides top-notch cloud and DevOps engineers to the world's most innovative companies. Initiatives stalled because you have trouble hiring? Foghorn can be burning down your DevOps and Cloud backlogs as soon as next week.
Kenneth Rose, CTO at OpsLevel, joins Corey on Screaming in the Cloud to discuss how OpsLevel is helping developer teams to scale effectively. Kenneth reveals what a developer portal is, how he thinks about the functionality of a developer portal, and the problems a developer portal solves for large developer teams. Corey and Kenneth discuss how to drive adoption of a developer portal, and Kenneth explains why it's so necessary to have executive buy-in throughout that process. Kenneth also discusses how using their own portal internally along with seeking out customer feedback has allowed OpsLevel to make impactful innovations. About KenKenneth (Ken) Rose is the CTO and Co-Founder of OpsLevel. Ken has spent over 15 years scaling engineering teams as an early engineer at PagerDuty and Shopify. Having in-the-trenches experience has allowed Ken a unique perspective on how some of the best teams are built and scaled and lends this viewpoint to building products for OpsLevel, a service ownership platform built to turn chaos into consistency for engineering leaders.Links Referenced: OpsLevel: https://www.opslevel.com/ LinkedIn: https://www.linkedin.com/company/opslevel/ Twitter: https://twitter.com/OpsLevelHQ TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Welcome to Screaming in the Cloud, I'm Corey Quinn, about, oh I don't know, two years ago and change, I wound up writing a blog post titled, “Developer Portals are An Anti Pattern,” and I haven't really spent a lot of time thinking about them since. This promoted guest episode is brought to us by our friends at OpsLevel, and they have sent their CTO and co-founder Ken Rose, presumably in an attempt to change my perspective on these things. Let's find out. Ken, thank you for agreeing to, well, run the gauntlet, for lack of a better term.Ken: Hey, Corey. Thanks again for having me. And I've heard, you know, heard and listened to your show a bunch, and really excited to be here today.Corey: Let's begin with defining our terms. I'm curious to know what a developer portal is. ‘What would you say a developer portal means to you?' Like it's a college entrance essay.Ken: Right? Definitely. You know, so really, a developer portal is this consolidated place for developers to come to, especially in large organizations to be able to get their jobs done more easily, right? A large challenge that developers have in large organizations, there's just a lot to do and a lot to take care of. So, a developer portal is a place for developers to be able to better own, manage, and run the services, they're responsible for that run in production, and they can do that through access, easy access to self-service tooling.Corey: I guess, on some level, this turns into one of those alignment charts of, like, what is a database and, like, how prescriptive you want to be. It's like, well is a senior engineer a database because you can query them and they have information? Would you consider, for example, Kubernetes be a developer platform, and/or would the AWS console?Ken: Yeah, that's actually an interesting question, right? So, I think there's actually two—we're going to get really niggly here—there's developer platform and developer portal, right? And the word portal for me is something that sits above a developer platform. I don't know if you remember, like, the late-90s, early-2000s, like, portals were all the rage.Like, Yahoo and AltaVistas were like search portals, they were trying to, at the time, consolidate all this information on a much smaller internet to make it easy to access. A developer portal is sort of the same thing, but custom-built for developers and trying to consolidate a lot of the tooling that exists. Now, in terms of the AWS console? Yeah, maybe. Like, it has a suite of tools and suite of offerings. It doesn't do a lot on the well, how do I quickly find out what's running in production and who is responsible for it? I don't know, unless AWS shipped, like, their, you know, three-hundredth new offering in the last week that I haven't, you know, kept on top of.But you know, there's definitely some spectrum in terms of what goes into a developer portal. For me, there's kind of three main things you need. You do need some kind of a catalog, like, what's out there who owns it; you need some kind of a way to measure, like, how good are those services, like, how well built are they; and then you need some access to self-service tooling. And that last part is where, like, the Kubernetes or AWS could be, you know, sort of a dev portal as well.Corey: My experience with developer portals—there was a time when I loved it. RightScale was what I used—at some depth—back in I want to say 2010, 2011 because the EC2 console was clearly not built or designed by anyone who had not built EC2 themselves with their bare hands and sweat of their brow. And in time, the EC2 console got better where it wasn't written in hieroglyphics, as best we could tell, and it became ‘click button to launch instance.' And RightScale really didn't have a second act and they wound up getting acquired by our friends over at Flexera years later. And I haven't seen their developer portal in at least eight years as a direct result of this.So, the problem, at least when I was viewing it purely in the context of AWS services, it feels like you are competing against AWS iterating forward on developer experience, which they iterate slowly, sometimes, and unevenly across their breadth of services, but it does feel like at some level by building an internal portal, you are, first, trying to out-innovate AWS, in some ways, and two, you are inherently making the trade-off of not using recent features and enhancements that have not themselves been incorporated into the portal. That's where the, I guess the start, the genesis of my opposition to the developer portal approach comes from. Is that philosophy valid these days? Not as much. Because I can see an argument for it shifting.Ken: Yeah, I think it's slightly different. I think of a developer portal as again, it's something that sort of sits on top of AWS or Google Cloud or whatever cloud provider use, right? You give an example for example with RightScale and EC2. So, provisioning instances is one part of the activity you have to do as a developer. Now, in most modern organizations, you have, like, your product developers that ship features. They don't actually care about provisioning instance themselves. There are another group called the platform engineers or platform group that are responsible for building automation and tooling to help spin up instances and create CI/CD pipelines and get everything you need set up.And they might use AWS under the covers to do that, but the automation built on top and making that accessible to developers, that's really what a developer portal can provide. In addition, it also provides links to operational tooling that you need, technical documentation, it's everything you need as a developer to do your job, in one place. And though AWs bills itself is that, I think of them as more, they have a lot of platform offerings, right, they have a lot of infra-offerings, but they still haven't been able to, I think, customize that, unless you're an organization that builds—that has kind of gone in-all on AWS and doesn't build any of your own tooling, that's where a developer portal helps. It really helps by consolidating all that information in one place, by making that information discoverable for every developer so they have less… less cognitive load, right? We've asked developers to kind of do too much that we don't… we've asked to shift left and well, how do we make that information more accessible?Regarding the point of, you know, AWS adds new features or new capabilities all the time and, like, well you have this dev portal, that's sort of your interface for how to get things done. Like, how will you use those? Dev portal doesn't stop you from doing that, right? So, my mental model is, if I'm a developer, and I want to spin up a new service, I can just press a button inside of my dev portal in my company and do that. And I have a service that is built according to the latest standards, it has a CI/CD pipeline, it already has a—you know, it's registered in PagerDuty, it's registered in Datadog, it has all the various bits.And then there's something else that I want to do that isn't really on the golden path because maybe this is some new service or some experiment, nothing stops us from doing that. Like, you still can use all those tools from AWS, you know, kind of raw. And if those prove to be valuable for the rest of the organization, great. They can make their way into the dev portal; they can actually become a source of leverage. But if they're not, then they can also just sit there on the vine. Like, not everything that eight of us ever produces will be used by every company.Corey: Many years ago, I got a Cisco pair of certifications because recession was hitting and I needed to be better at networking. And taking those certifications, in those days before Cisco became the sad corporate dragon with no friends we all know today, they were highly germane and relevant. But I distinctly remember, even now, 15 years later, that there was this entire philosophy of pretend that the entire world is Cisco only, which in networking is absolutely never true. It feels like a lot of the AWS designs and patterns tend to assume, oh yeah, you're going to use AWS services for everything. I have never yet found that to be true, other than when I'm just trying to be obstinate.And hell is interoperability between a bunch of different things. Yes, I may want to spin up an EC2 instance and an AWS load balancer and some S3 storage or whatnot, but I'm also going to want to monitor it with PagerDuty, I'm going to want to have a CDN that isn't CloudFront because most CDN these days don't hate you in quite the same economic ways and are simpler to work with, et cetera, et cetera, et cetera. So, there's definitely a story wherein I've found that there's an—the interoperability of tying these things together is helpful. How do you avoid falling down the trap of oh, everyone should be multi-cloud, single pane of glass at cetera, et cetera? In practice that always seems to turn to custard.Ken: Yeah, I think multi-cloud and single pane of glass are actually two different things. So multi-cloud, like, I agree with you to some sense. Like, pick a cloud and go with it, like, unless you have really good business reasons to go for multi-cloud. And sometimes you do, like, years ago, I worked at PagerDuty, they were multi-cloud for a reliability reason, that hey, if one cloud provider goes down, you don't want [crosstalk 00:08:40]—Corey: They were an example I used all the time for that story—Ken: Right.Corey: —specifically the thing woke you up was homed in a bunch of different places, whereas the marketing site, the onboarding flow, the periphery stuff around it was not because it didn't need to be.Ken: Exactly.Corey: Like, the core business need of wake you up was very much multi-cloud because once upon a time, it wasn't and it went down with the rest of us-east-1 and people weren't woken up to be told their site was on fire.Ken: A hundred percent. And on the kind of like application side where, even then, pick a cloud and go with it, unless there's a really compelling business reason for your business to go multi-cloud. Maybe there's something credits or compliance or availability, right? There might be reasons, but you have to be articulate about whether they're right for you.Now, single pane of glass, I think that's different, right? I do think that's something that, ultimately, is a net boon for developers. In any large organization, there is a myriad of internal tools that have been built. And it's like, well, how do I provision a new topic in the Kafka cluster? How do I actually get access to the AWS console? How do I spin up a new service, right? How do I kind of do these things?And if I'm a developer, I just want to ship features. Like, that's what I'm incented to do, that's what I'm optimizing for. And all this other stuff I have to do as part of my job, but I don't want to have to become, like, a Kubernetes guru to be able to do it, right? So, what a developer portal is trying to do is be that single pane of glass, bringing all these common set of tools and responsibilities that you have as a developer in one place. They're easy to search for, they're easy to find, they're easy to query, they're easy to use.Corey: I should probably have asked this earlier on, but let's disambiguate for a little bit here. Because when I'm setting up to use a new service or product and kick the tires on it, no two explorations really look the same. Whereas at most responsible mature companies that are building products that are—services that are going to production use, they've standardized around a number of different approaches. What does your target customer look like? Is there a certain point of scale, a certain level of complexity, a certain maturity of process?Ken: Absolutely. So, a tool like OpsLevel or a developer portal really only makes sense when you hit some critical mass in terms of the number of services you have running in production, or the number of developers that you have. So, when you hit 20, 30, 50 developers or 20, 30, 50 services, an important part of a developer portal is this catalog of what's out there. Once you kind of hit the Dunbar number of services, like, when you have more than you keep in your head, that's when you start to need tooling like this. If you look at our customer base, they're all you know, kind of medium to large-sized companies. If you're a startup with, like, ten people, OpsLevel is probably not right for you. We use all playable internally at OpsLevel, and you know, like, we're still a small company. It's like, we make it work for us because we know how to get the most out of it, but like, it's not the perfect fit because it's not really meant for, you know, smaller companies.Corey: Oh, I hear you. I think I'm probably… I have a better AWS bill analytic system running internally here at The Duckbill Group than some banks do. So, I hear you on that front.Ken: I believe it.Corey: But also implies to me that there's no OpsLevel prospect or customer deployment that has ever been greenfield. It's always you're building existing things, there's already infrastructure in place, vendors have been selected across the board. You aren't—don't to want to starting a company day one, they're going to all right, time to spin up our AWS account and we're also going to wind up signing up for OpsLevel, from the sound of it.Ken: Correct—Corey: Accurate? Inaccurate?Ken: I think that's actually accurate. Like, a lot of the problems, we solve other problems that come as you start to scale both your product and your engineering team. And it's the problems of complexity.Corey: What do those painful problems look like? In other words, what is someone sitting at home right now listening to this, or driving to work debating whether want to ram a bridge abutment or go into the office depending on their mental state today, what painful problem did they have that OpsLevel is designed to fix?Ken: Yeah, for sure. So, let's help people self-select. So, here's my mental model for any [unintelligible 00:12:25]. There are product developers, platform developers, and engineering leaders. Product developers, if you're asking questions like, “I just got paged for the service. I don't know what this does.” Or, “It's upstream from here. Where do I find the technical documentation?” Or, “I think I have to do something with the payment service. Where do I find the API for that?”You know, when you get to that scale, a developer portal can help you. If you're a platform engineer and you have questions like, “Okay, we got to migrate. We're migrating, I don't know, from Datadog to Honeycomb, right? We got to get these fifty or a hundred or thousands of services and all these different owners to, like, switch to some new tool.” Or, “Hey, we've done all this work to ship the golden path. Like, how to actually measure the adoption of all this work that we're doing and if it's actually valuable?” Right?Like, we want everybody to be on a certain set of CI tooling or a certain minimum version of some library or framework. How do we do that? How do we measure that? OpsLevel is for you, right? We have a whole bunch of stuff around maturity.And if you're engineering leader, ultimately, questions you care about, like, “How fast are my developers working? I have this massive team, we've made this massive investment in hiring all these humans to write software and bring value for our customers. How can we be more efficient as a business in terms of that value delivery?” And that's where OpsLevel can help as well.Corey: Guardrails, whether they be economic, regulatory, or otherwise, have to make it easier than doing things incorrectly because one of the miracle aspects of cloud also turns into a bit of a problem, which is shadow IT is only ever a corporate credit card away. Make it too difficult to comply with corporate policies and people won't. And they're good actors; they're trying to get work done. They're not trying to make people's lives harder, but they don't want to spend six weeks provisioning an EC2 cluster. So, there's always that weird trade-off.Now, it feels—and please correct me if I'm wrong—once someone has rolled out OpsLevel at their organization, where it really shines is spinning up a new service where okay, great, you're going to spin up the automatic observability portion of it, you're going to spin up the underlying infrastructure in certain ways that comply with our policies, it's going to build the CI/CD pipelines around it, you're going to wind up having the various cost instrumentation rolled out to it. But for services that are already excellent within the environment, is there an OpsLevel story for them?Ken: Oh, absolutely. So, I look at it as, like, the first problem OpsLevel helps solve is the catalog and what's out there and who owns it. So, not even getting developers to spin up new services that are kind of on the golden path, but just understanding the taxonomy of what are the services we have? How do those services compose into higher-level things like systems or domains? What's the whole set of infrastructure we have?Like, I have 50 AWS accounts, maybe a handful of GCP ones, also, some Azure. I have all this infrastructure that, like, how do I start to get a handle on, like, what's out there in prod and who's responsible for it. And that helps you get in front of compliance risks, security risks. That's really the starting point for OpsLevel building that catalog. And we have a bunch of integrations that kind of slurp all this data to automatically assemble that catalog, or YAML as well if that's your thing. But that's the starting point is building that catalog and figuring out this assignment of, like, okay, this service and this human, or this—sorry—team, like, they're paired together.Corey: A number of offerings in this space, which honestly, my exposure to it is bounded simultaneously to things that are ten years old and no one uses anymore, or a bunch of things I found on GitHub. And the challenge that both of those products tend to have is that they assume certain things to be true about a given environment: that they're using Terraform to manage everything, or they're always going to be using CloudFormation, or everyone there knows Python or something else like that. What are the prerequisites to get started with OpsLevel?Ken: Yeah, so we worked pretty hard to build just a ton of integrations. I would say integrations is are just continuing thing we have going on in the background. Like, when we started, like, we only supported a GitHub. Now, we support all the gits, you know, like GitHub, GitLab, Bitbucket, Azure DevOps, like, we're building [unintelligible 00:16:19]. There's just a whole, like, long tail of integrations.The same with APM tooling. The same with vulnerability management tooling, right? And the reason we do that is because there's just this huge vendor footprint, and people, you know, want OpsLevel to work for them. Now, the other thing we try to do is we also build APIs. So, anything we have as, like, a core integration, we also have kind of like an underlying API for, so that there's, no matter what you have an escape hatch. If like, you're using some tool that we don't support or you have some homegrown thing, there's always a way to try to be able to integrate that into OpsLevel.Corey: When people think about developer portals, the most common one that pops to mind is Backstage, which Spotify wound up building, internally, championing, open-sourcing, and I believe, on some level, turned into a product because if there's one thing people want, it's to have their podcast music company become a SaaS vendor, which is weird to me. But the criticisms that I've seen about and across the board have rung relatively true, including from people internal at Spotify who have used the thing, which is, well first is underestimating the amount of effort that is necessary to maintain Backstage itself, that the build versus buy discussion is always harder to bu—engineers love to build, but they shouldn't be building things outside of their core competency half the time, and the other is driving adoption within the org where you can have the most amazing developer portal in the known universe, but if people don't use it, it may as well not exist and doing the carrot and stick approach often doesn't work. I think you have a pretty good answer that I need not even ask you to elaborate on, “Well, how do we avoid having to maintain this ourselves,” since you have a company that does this, but how do you find companies are driving adoption successfully once they have deployed OpsLevel?Ken: Yeah, that's a great question. So, absolutely. Like, I think the biggest thing you need first, is kind of cultural buy-in and that this is a tool that we want to invest in, right? I think one of the reasons Spotify was successful with Backstage and I think it was System Z before that was that they had this kind of flywheel of, like, they saw that their developers were getting, you know better faster, working happier, by using this type of tooling, by reducing the cognitive load. The way that we approach it is sort of similar, right?We want to make sure that there is executive buy-in that, like, everybody agrees this is, like, a problem that's worth solving. The first step we do is trying to build out that catalog again and helping assign ownership. And that helps people understand, like, hey, these are the services I'm responsible for. Oh, look, and now here's this other context that I didn't have before. And then helping organizations, you know, what—it depends on the problem we're trying to solve, but whether it's rolling out self-serve automation to help developers, like, reduce what was before a ton of cognitive load or if it's helping platform teams define what good looks like so they can start to level up the overall health of what's running in production, we kind of work on different problems, but it's picking one problem and then you know, kind of working with the customers and driving it forward.Corey: On some level, I think that this is going to be looked down upon inherently just by automatic reflex of folks with infrastructure engineering backgrounds. It's taken me some time to learn to overcome my own negative reaction to it. Because it's, I'm here to build things and I want to build things out in such a way that it's portable and reusable without having to be tied to a particular vendor and move on. And it took me a long time to realize that what that instinct was whispering in my ear was in fact, no, you should be your own cloud provider. If that's really what I want to do, I probably should just brush up on you know, computer science trivia from 20 years ago and then go see if I can pass Google's SRE interview.I'm not here to build the things that just provision infrastructure from scratch every company I wind up landing at. It feels like there's more important, impactful work that I can do. And let's be clear, people are never going to follow guardrails themselves when they have to do a bunch of manual steps. It has to be something that is done for them. And I don't know how you necessarily get there without having some form of blueprint or something like that, provided for them with something that is self-service because otherwise, it's not going to work.Ken: I a hundred percent agree, by the way, Corey. Like, the take that, like, automation is the only way to drive a lot of this forward is true, right? If for every single thing you're trying—like, we have a concept called a rubric and it's basically how you measure the service health. And you can—it's very customizable, you have different dimensions. But if, for any check that's on your rubric, it requires manual effort from all your developers, that is going to be harder than something you can just automate away.So, vulnerability management is a great example. If you tell developers, “Hey, you have to go upgrade this library,” okay, some percentage [unintelligible 00:20:47], if you give developers, “Here's a pull request that's already been done and has a test passing and now you just need to merge it,” you're going to have a much better adoption rate with that. Similarly with, like, applying templates being able to [up-level 00:20:57], you know, kind of apply the latest version of a template to an existing service, those types of capabilities, anything where you can automate what the fixes are, absolutely you're going to get better adoption.Corey: As you take a look at your existing reference customers—which is something I always look for on vendor websites because, like, oh, we have many customers who will absolutely not admit to being customers, it's like, that sounds like something that's easy to say—you have actual names tied to these things. Not just companies, but also individuals. If you were to sit down and ask your existing customer base, “So, why did you wind up implementing OpsLevel and what has the value that's delivered to you been since that implementation?” What do they say?Ken: Definitely. I actually had to check our website because we, you know, land new customers and put new logos on it. I was like, “Oh, I wonder what the current set is out right now?”Corey: I have the exact same challenge. Like oh, we have some mutual customers. And it's okay. I don't know if I can mention them by name because I haven't checked our own list of testimonials [unintelligible 00:21:51] lately because say the wrong thing and that's how you wind up being sued and not having a company anymore.Ken: Yeah. So, I don't—I definitely, you know, want to stay [on side 00:22:00] on that part, but in terms of, like, kind of sample reference customer, a lot of the folks that we initially worked with are the platform teams, right? They're the teams that care about what's out there, and they need to know who's responsible for it because they're trying to drive some kind of cross-cutting change across the entire, you know, production footprint. And so, the first thing that generally people will say is—and I love this quote. This came—I won't name them, but like, it's in one of our case studies.It was like, “I had, like, 50 different attempts at making a spreadsheet and they're all, like, in the graveyard, like, to be able to capture what's out there and who's responsible for it.” And just OpsLevel helping automate that has been one of the biggest values that they've gotten. The second point, then is now be able to drive maturity and be able to measure how well those services are being built. And again, it's sort of this interesting thing where we start with the platform teams. And then sometime later security teams find out about OpsLevel, and they're like, “Oh, this is a tool I can use to, like, get developers to do stuff? Like, I've been trying to get developers to do stuff for the longest time.”And they—I file Jira tickets and they just sit there and nothing gets done. But when it becomes part of this, like, overall health score that you're trying to increase a part of the across the board, yeah, it's just a way to kind of drive action.Corey: I think that there's a dichotomy of companies that emerge. And I tend to see the world through a lens of AWS bills, so let's go down that path. I feel like there are some companies presumably like OpsLevel, whereas if I—assuming you're running on top of AWS—if I were to pull your AWS bill, I would see upwards of 80% of your spend is going to be on this application called OpsLevel, the service that you provide to people. As opposed to the other side of the world, which is large enterprises, where they're spending hundreds of millions of dollars a year, but the largest application they have is a million-and-a-half a year in spend because just, they have thousands of these things scattered everywhere. That latter case is where I tend to see more platform teams, where I start to see a lot of managing a whole bunch of relatively small workloads. And developer platforms really seem to be where a lot of solutions lead, whereas 80% of our workload is one application, we don't feel the need for that as much. Is that accurate? Am I misunderstanding some aspect of it?Ken: No, a hundred percent you'd hit the nail on the head. Like, okay, think about the typical, like, microservices adoption journey. Like, you started with, you know, some small company—like us—you started with a monolith. Ah, maybe you built out a second app—Corey: Then you read on Hacker News and realize, “Oh, if we want to hire people, we've got to be doing what all the cool kids are up to.”Ken: Right. We got a microservice all the thing—but that's actually you know, microservices should come later, right, as a response to you needing scale your org and scale your—Corey: As someone who started building some application with microservices, I could not agree more.Ken: A hundred percent. So, it's as you're starting to take steps to having just more moving parts in your production infrastructure, right? If you have one moving part, unless it's like a really large moving part that you can internally break down, like, kind of this majestic monolith where you do have kind of like individual domains that are owned by different teams, but really the problem we're trying to solve, it's more about, like, who owns what. Now, if that's a single atomic unit, great, but can you decompose that? But if you just have, like, one small application, kind of like the whole team is owning everything, again, a developer portal is probably not the right tool for you. It really is a tool that you need as you start to scale your engineer work and as you start to scale the number of moving parts in your production infrastructure.Corey: I tended to use to think of that in terms of boring companies versus innovative ones and I don't think that's accurate. I think it is the question of maturity and where companies lead to. On some level, of OpsLevel starts growing and becomes larger and larger in different ways and starts doing acquisitions and launching into other areas, at some point, you don't have just one product offering, you have a multitude of them. At which point having something like that is going to be critical. But I have to ask, given that you are sort of not exactly your target customer profile, what are the sharp edges been on using it for your use case?Ken: Yeah. So, we actually have an internal Slack channel, we call OpsLevel on OpsLevel. And finding those sharp edges actually has been really useful for us. You know, all the good stuff, dogfooding and it makes your own product better. Okay, so we have our main app, we also do have a bunch of smaller things and it's like, oh yeah, you know, we have, like, I don't know, various Hackaday things that go on, it's important we kind of wind those down for, you know, compliance, we have our marketing site, we have, like, our Terraform.Like, so there's, like, stuff. It's not, like, hundreds or thousands of things, but there's more than just the main app. The second though, is it's really on the maturity piece that we really try to get a lot of value out of our own product, right? Helping—we have our own platform team. They're also trying to drive certain initiatives with our product developers.There is that usual tension of our, like, our own product developers are like, “I want to ship features.” What's this security thing I have to go take care of right now? But OpsLevel itself, like, helps reflect that. We had an operational review today and it was like, “Oh, this one service is actually now”—we have platinum as a level. It's in gold instead of platinum. It's like, “Why?” “Oh, there's this thing that came up. We got to go fix that.” “Great. Let's go actually go fix that so we're back into platinum.”Corey: Do you find that there's often a choice you have to make internally, where you could make the product more effective for your specific use case, but that also diverges from where your typical customer needs or wants the product to go?Ken: No, I think a lot of the things we find for our use case are, like, they're more small paper cuts, right? They're just as we're using it, it's like, “Hey, like, as I'm using this, I want to see the report for this particular check. Why do I have to click six times to get?” You know, like, “Wouldn't it be great if we had a button?” Right?And so, it's those type of, like, small innovations that kind of come up. And those ultimately lead to, you know, a better product for our customers. We also work really closely with our customers and developers are not shy about telling you what they don't like about your product. And I say this with love, like, a lot of our customers give us phenomenal feedback just on how our product can be better and we try to internalize that and you know, roll that feedback into the product.Corey: You have a number of integrations of different SaaS providers, infrastructure providers, et cetera, that you wind up working with. I imagine that given your scale and scope and whatnot, those offerings are dictated by what customers say, “Hey, we're using this thing. Are you going to support that or are you not going to maintain our business?” Which is a great way to wind up financing a lot of product development and figuring out what matters to people. My question for you is, if you look across the totality of your user base, what are the most popularly used integrations, if you can say?Ken: Yeah, for sure. I think right now—I could actually dive in to pull the numbers—GitHub and GitLab—or… I think GitHub, like, has slightly more adoption across our customer base. At least with our customers, almost nobody uses Bitbucket. I mean, we have, like, a small number, but, like, it's… I think, single-digit percentage. A lot of people use PagerDuty, which you know, hey, I'm an ex-PagerDuty person [crosstalk 00:28:24] and I'm glad to see that.Corey: I have a free tier PagerDuty account that will automatically page me for my home automation stuff. Specifically, if you know, the fire alarm goes off. Like, yeah, okay, there are certain things I want to be woken up for, but it's a very short list.Ken: Yeah, it's funny, the running default message when we use a test PagerDuty was, “The server is on fire.” [unintelligible 00:28:44] be like, “The house is on fire.” Like you know, go get that taken care of. There's one other tool so that's used a lot. Datadog actually is used a ton by just across our entire customer base, despite its… we're also Data—we're a Datadog partner, we're a Datadog customer, you know? It's not cheap, but it's a good product for, you know, monitoring and logs and there are [crosstalk 00:29:01]—Corey: No other than cloud infrastructure providers, I get the number one most common source of inquiries is Datadog optimization. It has now risen to a board-level concern in many cases because observability is expensive. That's a sign of success, on some level. Meanwhile, I'm sitting here, like, Date-a-dog? Oh, my God, that's disgusting. It's like Tinder for Pets. Which it turns out is not at all what they do.Ken: Nice.Corey: Yeah.[audio break 00:29:23]—optimizing their Slack integrations, their GitHub integration, et cetera. Or are they starting with the spinning up the servers piece of it?Ken: A lot of the time—and again, that first problem they're trying to solve is just get me a handle on everything we have running in production. You know, if you have multiple AWS accounts, multiple Kubernetes clusters, dozens or even hundreds of teams, God help you if you're going to try to, like, build a list manually to consolidate all that information. That's really the first part is, like, integrate Kubernetes, integrate your CI/CD pipelines, integrate Git, integrate your Cloud account, like, will integrate with everything and will try to build that map of, like, here's everything that's out there, and start to try to assign it to, like, and here's people that we think might be responsible in terms of owning the software. That's generally the starting point.Corey: Which makes an awesome amount of sense. I think going at it from the infrastructure first perspective is where I've seen most developer platforms founder. And to be fair, the job is easier now than it was years ago because it used to be that you were being out-innovated by AWS constantly. Innovation has slow down there. And you know that because of how much they say the pace of innovation has only sped up.And whenever AWS says something in a marketing context, they're insecure about it. I've learned this through the fullness of time observing that company. And these days, most customers do not use the majority of features available for any given service. They have solidified to a point where you can responsibly build on top of these things. Now, it seems that the problem is all the ‘yes, and' stuff that gets built on top of it.Ken: Yeah. Do you have an example, actually, like, one of the kinds of, like, ‘yes, and' tools that you're thinking about?Corey: Oh, absolutely. We have a bunch of AWS environment stuff so we should configure CloudWatch to look at all these things from an observability perspective. No, you should not. You should set up Datadog. And the first time someone does that by hand, they enable all have the observability and the rest and suddenly get charged approximately the GDP of Guam.And okay, maybe we shouldn't do that because then you have the downstream impact of that on your CloudWatch bill. So okay, how do we optimize this for the observability piece directly tied to that? How do we make sure that we get woken up when the site is down or preferably before that, but not every time basically, a EBS volume starts to get a little bit toasty? You have to start dialing this stuff in. And once you've found a lot of those aspects, being able to templatize that and roll that out on an ongoing basis and having the integrations all work together feels like it's the right problem to be solving.Ken: Yeah, absolutely. And the group that I think is responsible for that kind of—because it's a set of problems you described—is really, like, platform teams. Sometimes service owners for like, how should we get paged, but really, what you're describing are these kind of cross-cutting engineering concerns that platform teams are uniquely poised to help solve in an [unintelligible 00:32:03] organization, right? I was thinking what you said earlier. Like, nobody just wants to rebuild the same info over and over, but it's sort of like, it's not just building an [unintelligible 00:32:09]; it's kind of like solving this, like, how do we ship? Can we actually run stuff in prod? And not just run it but get observability and ensure that we're woken up for it and, like, what's that total end-to-end look like from, like, developers writing code to running software in production that's serving traffic? And solving all the problems [unintelligible 00:32:24], that's what I think of was platform engineering.Corey: So, my last question before we wind up wrapping this episode comes down to, I am very adept at two different programming languages, and those are brute force and enthusiasm. What implementation language is most of what you find yourself working with? And why is it in invariably going to be YAML?Ken: Yeah, that's a great question. So, I think there's, in terms of implementing OpsLevel and implementing a service catalog, we support YAML. Like, you know, there's this very common workflow, you just drop a YAML spec, basically, in your repo, if you're a service owner. And that, we can support that. I don't think that's a great take, though.Like, we have other integrations. Again, if the problem you're trying to solve is I want to build a catalog of everything that's out there, asking each of your developers hey, can you please all write YAML files that, like, describe the services you own and drop them into this repo? You've inverted this, like, database that essentially you're trying to build, like, what's out there and stored it in Git, potentially across several hundreds or thousands of repos. You put a lot of toil now on individual product developers to go write and maintain these files. And if you ever had to, like, make a blanket update to these files, there's no atomic way to kind of do that, right?So, I look at YAML as, like, I get it, you know? Like, we use the YAML for all the things in DevOps, so why not their service catalog as well, but I think it's toil. Like, there are easier ways to build a catalog. By, kind of, just integrate. Like, hook up AWS, hook up GitHub, hook up Kubernetes, hook up your CI/CD pipeline, hook up all these different sources that have information about what's running in prod, and let the software, let the tool, automatically infer what's actually running as opposed to requiring humans to manually enter data.Corey: I find that there are remarkably few technical holy wars that I cannot unify both sides on by nominating something far worse. Like, the VI versus Emacs stuff, the tabs versus spaces, and of course, the JSON versus YAML folks. My JSON versus YAML answer is XML: God's language. I find that as soon as you suggest that, people care a hell of a lot less about the differences between JSON and YAML because their job is to now kill the apostate, which is me.Ken: Right. Yeah. I remember XML, like, oh, man, 2002. SOAP. I remember SOAP as a protocol. That was a thing.Corey: Some of the earliest S3 API calls were done in SOAP, and I think they finally just used it to wash their mouths out when all was said and done.Ken: Nice. Yeah.Corey: I really want to thank you for taking the time to do your level best to attempt to convert me, and I would argue in many respects, you have succeeded. I'm thinking about this differently than I did half an hour ago. If people want to learn more, where's the best place for them to find you?Ken: Absolutely. So, you can always check out our website, opslevel.com. We're also fairly active on LinkedIn. If Twitter hasn't imploded by the time this episode becomes launched, then they can also check us out at twitter.com/OpsLevelHQ. We're always posting, just different content on, like, how to be successful with service maturity, DevOps, developer productivity, so that you know, ultimately, that you can ship out to customers faster.Corey: And we will, of course, put links to that in the [show notes 00:35:23]. Thank you so much for taking the time, not just to speak with me, but also for sponsoring this episode. It is appreciated.Ken: Cheers.Corey: Ken Rose, CTO and co-founder at OpsLevel. I'm Cloud Economist Corey Quinn and this has been a promoted guest episode of Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, along with an angry comment which, upon further reflection, you could have posted to all of the podcast platforms if only you had the right developer platform to pull it off.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.
In this episode, I caught up with Ben Kehoe, who is an AWS Serverless Hero and one of the earliest adopters of serverless technologies.In a wide-ranging conversation, we discussed many topics around serverless and AI, including:The natural evolution of marketing terms and the need to focus on specific functional characteristics rather than defending the term. For example, including of arguing about what "serverless" means, we should instead talk about "pay-per-use".AWS should focus DX around the core service (e.g. CloudFormation) rather than trying to find client-side solutions by adding workarounds in SAM, CDK, etc.These client-side answers have a higher Total Cost of Ownership (TCO). Developers often don't see the increased TCO they are taking on, but when things break, it's a problem.Developers put too much emphasis on author time benefits and not enough on runtime and operational time costs. They should be more thoughtful about the operational time cost.The “infrastructure from code” movement is taking burdens off the developer but leaving them with the developer's business, which is a bad thing.Developers often have a hard time separating delivering business value vs. coding.As an industry, a flawed narrative has emerged that developers are somehow special within an organisation and that it's OK for them to ignore their responsibilities to security if there is friction in the process.Ai has the potential to impede human growth as the current AI systems are not designed to generate new ideas and challenge the status quo. “an AI generator that is trained on modernist art would never invent post-modernism”.Links from the episode:The meaning(lessness) of serverlessServerless is a state of mindThe serverless spectrumEp16 - Serverless at iRobot with Ben KehoeFor more stories about real-world use of serverless technologies, please follow me on Twitter as @theburningmonk and subscribe to this podcast.Want to step up your AWS game and learn how to build production-ready serverless applications? Check out my upcoming workshops and I will teach you everything I know.Opening theme song:Cheery Monday by Kevin MacLeodLink: https://incompetech.filmmusic.io/song/3495-cheery-mondayLicense: http://creativecommons.org/licenses/by/4.0
Welcome to the newest episode of The Cloud Pod podcast! Justin, Ryan, Jonathan, Matthew are your hosts this week. Join us as we discuss all things cloud, AI, the upcoming Google AI Conference, AWS Console, and Duet AI for Google cloud. Titles we almost went with this week:
An airhacks.fm conversation with Maximilian Schellhorn (@maschnetwork) about: playing Halo with Fujitsu Siemens Scaleo, amazing graphics with crytec and crysis, learning HTML and the great marquee tag, semi-professional Call of Duty 4 gaming, learning Delphi and GUI programming, button oriented programming in Delphi, building ski school software in Delphi, from Delphi to Java and Spring, learning patterns with Java, starting at cloudflight.io, from Zalando to AWS, starting at AWS as Solution Architect, deploying quarkus as AWS Lambda, full stack Infrastructure as Code with Java, creating Java content on AWS, SAM, CloudFormation and CDK, CDK with SAM CLI, declarative development with SAM, state management and IaC, AWS Serverless Java Container, Lambda SnapStart optimises the startup time, SnapStart snapshots, using CRaC hooks for SnapShot, caching with ShapStart vs. PostConstruct, from Kotlin to Java 17, data classes in Kotlin vs. Java Records, sealed classes as error handlers, serverless on-premise and in the clouds, using S3 and DynamoDB, DynamoDB and IaM security, lift-and-shift, lambda SQS integration, Java on AWS Lambda Workshop, Caching Data in the Snapshot, announcement: AWS Lambda adds support for Java 17 Maximilian Schellhorn on twitter: @maschnetwork
John Mille, Principal Cloud Engineer at Sainsbury's UK joins Corey on Screaming in the Cloud to discuss how retail companies are using cloud services. John describes the lessons he's learned since joining the Sainsbury's UK team, including why it's important to share knowledge across your team if you don't want to be on call 24/7, as well as why he doesn't subscribe to the idea that every developer needs access to production. Corey and John also discuss an open-source project John created called ECS Compose-X.About JohnJohn is an AWS Community Builder (devtools), Open Source enthusiast, SysAdmin born in the cloud, and has worked with AWS since his very first job. He enjoys writing code and creating projects. John likes to focus on automation & architecture that delivers business value, and has been dabbling with data & the wonderful world of Kafka for the past 3 years.Links Referenced: AWS Open-Source Roundup newsletter blog post about ECS Compose-X: https://aws.amazon.com/blogs/opensource/automating-your-ecs-container-architecture-deployments-with-ecs-composex/ ECS Compose-X: https://docs.compose-x.io/ LinkedIn: https://www.linkedin.com/in/john-mille/ Twitter: https://twitter.com/JohnPre32286850 TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: It's easy to **BEEP** up on AWS. Especially when you're managing your cloud environment on your own!Mission Cloud un **BEEP**s your apps and servers. Whatever you need in AWS, we can do it. Head to missioncloud.com for the AWS expertise you need. Corey: Do you wish your developers had less permanent access to AWS? Has the complexity of Amazon's reference architecture for temporary elevated access caused you to sob uncontrollably? With Sym, you can protect your cloud infrastructure with customizable, just-in-time access workflows that can be setup in minutes. By automating the access request lifecycle, Sym helps you reduce the scope of default access while keeping your developers moving quickly. Say goodbye to your cloud access woes with Sym. Go to symops.com/corey to learn more. That's S-Y-M-O-P-S.com/coreyCorey: Welcome to Screaming in the Cloud. I'm Corey Quinn. Today my guest is a long-time listener, first-time caller. John Mille is a Principal Cloud Engineer at Sainsbury's, which is UK-speak for ‘grocery store.' John, thank you for joining me.John: Hi, Corey. Thanks for having me.Corey: So, I have to begin with, I guess, the big question that I used to run into people in San Francisco with all the time. They would work at Walmart Labs and they would mention in conversation that they work at Walmart, and people who weren't aware that there was a labs out here figured they were a greeter at the grocery store. Do you ever wind up with people making that sort of fundamental assumption around the fact, oh, you work at Sainsbury's as a checker or whatnot?John: No. But it actually is one of the—if you look at one of the job descriptions from Sainsbury's, the first thing is, why would you join a retail company to do tech? And as it turns out, tech—I mean, I think retail companies, as any other companies in the world, rely on Cloud more and more and more. And I think that one of the things that is interesting today is, if you look at the landscape of retailers, I've heard many times people saying, “We don't want to go for AWS because we're giving money to the competition.” And actually, I think AWS does a fantastic job overall giving you all the tools to actually beat them as your competition. And as it turns out, we've had really, really great success running a lot of our workloads on AWS for many, many years now.Corey: On some level, if you can't come to terms with the idea of Amazon as competition, you shouldn't be using AWS, regardless of what industry you're in, because their entire company strategy is yes. It's very hard to start to even come up with industries that they don't have some form of presence within. On some level, that's a problem. In fact a lot of levels, that's something of a problem.Everyone tends to wind up viewing the world in a bunch of different ways. I like to divide companies into two groups. More or less it's, is the AWS bill one of the top three line items at the company? And if the answer's no, on some level, you know, that usually is an indicator that there's a sustainable business there that, you know, both our grandparents and our grandchildren will be able to recognize, in the fullness of time. You absolutely have a business that winds up falling into that category, whereas, “Oh yeah, I fix the AWS bill,” yeah, my parents would have no idea what I do and my kids don't have much of a better one. It feels like it's very point-in-time type of problem. At least I hope.Technology is not the core of what grocery stores tend to do, but I also don't get the sense that what you're doing is sitting there doing the back office corporate IT style of work, either. How do you use technology in the overall context of the business?John: Well, so we use it in a very wide variety of sense. So, you obviously have everything that has to do with online shopping, orders and all of those sort of things, which obviously, especially with the drive of Covid and being everybody from home, has been a huge driver to improve our ability to deliver to customers. But certainly, I think that Sainsbury's sees AWS as a key partner to be able to go and say we want to deliver more value. And so, there's been a number of transformation over the years to—and one of the reasons I was hired is actually to be part of one of those transformation, where we're going to take existing infrastructure servers that literally—I usually say to people, “Oh, are we doing an upgrade this month? Has somebody gotten their little brush to go and brush onto the hard drives to make sure that nothing is going to die?” And actually do that transformation and move over to the cloud in order to never have to really worry about whether or not they have to manage hardware and infrastructure.Corey: It's strange in that I never got very deep into containers until I was no longer hands-on hardware, managing things. I was more or less doing advisory work and then messing around with them. And you'd think given my proclivities historically, of being very unlucky when it comes to data, you would think that this would be great because, oh yeah, you blow away an ephemeral container? Well, that's kind of the point. We'll all laugh and it'll re-instantiate itself and life goes on.But no. Making fun of them was more or less how I tended to do approach them for the longest time until I started to see them a little bit… well I guess less as a culture, less as a religion, and more as an incredibly versatile packaging format, which is probably going to annoy the people I know who are the packaging [unintelligible 00:04:58] for Linux distributions. How do you tend to view them? And how did you start using them?John: Right. So, that's a great question. So historically, I was a student at, I think the school were one of the original creators of Docker were. And one of the things that you learn when you do development at the school is that, you know, containers [unintelligible 00:05:18] new invention. Docker, I think, came on the platform as the way to, you know, give everybody a great framework, a great API, to drive the deployment of containers in the world and bundle them and ship them around the world, on your laptop and somebody else's, and help a little bit with, you know, solving the problem of it works on my laptop, but not just on the laptop properly. Maybe.It's obviously gone viral over the years and I really enjoy containers; I quite like containers. What I find interesting is what people are going to do with. And I think that over the last few years, we've seen a number of technologies such as Kubernetes and others come into the scene and say—and trying to solve people's problem, but everybody seems to be doing, sort of, things on their own way. And historically, I started off using ECS, when it was terrible and you didn't have security groups per containers and all of this. But over the years, you know, you learn, and AWS has improved the service quite significantly with more and more features.And I think we are today in the place where there's this landscape, I think, where a lot of workloads are going to be extremely ephemeral and you can go [unintelligible 00:06:28], you know, wherever you want and you have a bit—if you have a platform or workflow that you need to have working in different places, maybe Kubernetes could be an easy way to have a different sort of sets of features that allows you to move around in maybe an easier way. But that also comes with a set of drawbacks. Again, I look at using EKS, for example, and I see okay, I have to manage IAM in our back now, whereas if I used something like ECS, for the whatever the [unintelligible 00:06:56] cloud vendor of choice, I don't have to deal with any of this. So, I think it's finding the fine balance between how you do orchestration of containers now and what works for you and is any sustainable over the time, more than about are you going to use containers? Because the chances are, somebody is using containers.Corey: My experiences and workflows and constraints are radically different than that of other folks because for a lot of the things I'm building, these are accounts that are I'm the only person that has access to them. It is me. So, the idea of fine-grained permissions for users from an ARBAC perspective doesn't really factor into it. Yes, yes, in theory, I should have a lot of the systems themselves with incidents roles being managed in safe and secure ways, but in many cases, the AWS account boundary is sufficient for that, depending on what it is we're talking about. But that changes when you start having a small team of people working with you and having to collaborate on these things.And we do a little bit of that with some of our consulting stuff that isn't just the shitpost stuff I build for fun. But there's multiple levels beyond that. You are clearly in a full-blown enterprise at this point where there are a bunch of different teams working on different things, all ideally going in the same direction. And it's easy to get stuck in the weeds of having to either go through central IT for these things, which gives rise to shadow IT every time you find a corporate credit card in the wild, or it winds up being everyone can do what they want, but then there's no consensus, there's no control, there's no architectural similarity. And I'm not sure which path is worse in some respects. How do you land on it?John: Right. So, what I've seen done in companies that works very well—and again, to the credit of my current company—is one of the things they've done really well is build a hub of people who are going to manage solely everything that has to do with accounts access, right? So, the control, IAM, Security Hub, all of those sorts of things, for you. There's things that are mandatory that you can't deal without, you have permissions boundary, that's it, you have to use those things, end of story. But beyond that point, once you have access to your accounts, you've been given all of the access that is necessary for you to deliver application and deploy them all the way up to production without asking permission for anybody else apart from your delivery managers, potentially.And I think from there, because there is the room to do all of this, one of the things that we've done within my business unit is that we've put in place a framework that enables developers—and when I say that it really is a question of allowing them to do everything they have to do, focus on the code, and I know it's a little catchy [unintelligible 00:09:33] a phrase that you hear these days, but the developers really are the customers that we have. And all that we do is to try to make sure that they have a framework in place that allows them to do what they need and deploy the applications in a secure fashion. And the only way to do that for us was to build the tools for them that allows them to do all of that. And I honestly haven't checked a single service IAM policies in a very are longtime because I know that by providing the tools to developers, they don't have this [will 00:10:05] to go and mess with the permissions because their application suddenly doesn't have the permissions. They just know that with the automation we've providing them, the application gets the access it needs and no more.Corey: On some level, it feels like there's a story around graduated development approach where in a dev environment you can do basically whatever you want with a big asterisk next to it. That's the same asterisk, by the way, next to the AWS free tier. But as you start elevating things into higher environments, you start to see gating around things like who has access to what, security reviews, et cetera, et cetera, and ideally, by the time you wind up getting into production, almost no one should have access and that access that people do have winds up being heavily gated. That is, of course, the vision that folks have. In practice, reality is what happens instead of what we plan on. The idea of it works in theory, but not in production is of course, why I call my staging environment ‘theory.' Does that tend to resonate as far as what you've seen in the wild?John: Yeah. Very much so. And when I joined the company, and we put together our [standard 00:11:11] pipelines for developers to be able to do everything, the rule that I would give to my team—so I manage a small team of cloud engineers—the one rule I would say is, “We have access to prod because we need to provision resources, but when we're going to build the pipelines for the developers, you have to build everything in such a way that the developers will only have read-only access to the production environment, and that is only to go and see their logs.” And at least try to foster this notion that developers do not need access to production, as much as possible because that avoids people going and do something they shouldn't be doing in those production environments.Now, as the pipeline progresses and applications get deployed to production, there are some operational capabilities that people need to have, and so in that case, what we do is we try to fine-tune what do people need to do and grant those people access to the accounts so that they can perform the jobs and I don't have to be woken up at two in the morning. The developers are.Corey: One thing that I think is going to be a cause of some consternation for folks—because I didn't really think about this in any meaningful sense until I started acting as a consultant, which means you're getting three years of experience for every year that you're in the wild, just by virtue of the variety of environments you encounter—on some level, there's a reasonable expectation you can have when you're at a small, scrappy startup, that everyone involved knows where all the moving parts live. That tends to break down with scale. So, the idea of a Cloud Center of Excellence has been bandied around a lot. And personally, I hate the term because it implies the ‘Data Center of Mediocrity,' which is a little on the nose for some people at times. So, the idea of having a sort of as a centralized tiger team that has the expertise and has the ability to go on deep dives and sort of loan themselves out to different teams seems to be a compromise between nobody knows what they're doing and, every person involved should have an in-depth knowledge of the following list of disciplines.For example, most folks do not need an in-depth primer on AWS billing constructs. They need about as much information fits on an index card. Do you find that having the centralized concentration of cloud knowledge on a particular team works out or do you find that effectively doing a rotating embedding story is the better answer?John: It varies a lot, I think, because it depends on the level of curiosity of the developers quite a lot. So, I have a huge developer background. People in my team are probably more coming from ex-IT environments or this sort of operation and then it just naturally went into the cloud. And in my opinion, is fairly rare to find somebody that is actually good at doing both AWS and coding. I am by no means really, really great at coding. I code pretty much every day but I wouldn't call myself a professional developer.However, it does bring to my knowledge the fact that there are some good patterns and good practices that you can bring into building your applications in the cloud and some really bad ones. However, I think it's really down to making sure that the knowledge is here within the team. If there's a specialized team, those really need to be specialists. And I think the important thing then is to make sure that the developers and the people around you that are curious and want to ask questions know that you're available to them to share that knowledge. Because at the end of the day, if I'm the only one with the knowledge, I'm going to be the one who is always going to be on call for this or doing that and this is no responsibility that I want. I am happy with a number of responsibilities, but not to be the only person to ever do this. I want to go on holidays from time to time.So, at the end of the day, I suppose it really is up to what people want or expect out of their careers. I do a job that it was a passion for me since I was about 14 years old. And I've always been extremely curious to understand how things work, but I do draw the line that I don't write anything else than Python these days. And if you ask me to write Java, I'll probably change job in the flip of a second. But that's the end of it. But I enjoy understanding how Java things work so that I can help my developers make better choices with what services in AWS to use.Corey: On some level, it feels like there's a, I guess, lack of the same kind of socialization that startups have sort of been somewhat guided by as far as core ethos goes, where, oh whatever I'm working on, I want to reach out to other people, and, “Hey, I'm trying to solve this problem. What is it that you have been working on that's germane to this and how can we collaborate together?” It has nothing to do, incidentally, with the idea that, oh, big company people aren't friendly or are dedicated or aren't good or aren't well-connected; none of that. But there are so many people internally that you're spending your time focusing on and there's so much more internal context that doesn't necessarily map to anything outside of the company that the idea of someone off the street who just solved a particular problem in a weird way could apply to what a larger company with, you know, regulatory burdens, starts to have in mind, it becomes a little bit further afield. Do you think that that's accurate? Do you think that there's still a strong sense of enterprise community that I'm just potentially not seeing in various ways because I don't work at big companies?John: It's a very fine line to walk. So, when I joined the company, I was made aware that there's a lot of Terraform and Kubernetes, which I went [unintelligible 00:16:28] all the way with CloudFormation is yes. So, that was one of the changes I knew I would have. But I can move an open mind and when I looked around at, okay, what are the Terraform modules—because I used Terraform with anger for an entire year of suffering—and I thought, “Okay, well, maybe people have actually got to a point where they've built great modules that I can just pick up off the shelf and reuse or customize only a tiny little bit, add maybe a couple of features and that's, it move on; it's good enough for me.” But as it turns out, there is I think, a lot of the time a case where the need for standardization goes against the need for business to move on.So, I think this is where you start to see silos start to being built within the company and people do their own thing and the other ones do their own. And I think it's always a really big challenge for a large company with extremely opinionated individuals to say, “All right, we're going to standardize on this way.” And it definitely was one of the biggest challenge that I had when I joined the company because again, big communities and Terraform place, we're going to need to do something else. So, then it was the case of saying, “Hey, I don't think we need Kubernetes and I definitely don't think we need Terraform for any the things—for any of those reasons, so how about we do something a little different?”Corey: Speaking of doing things a little bit different, you were recently featured in an AWS Open-Source Roundup newsletter that was just where you, I think, came across my desk one of the first times, has specifically around an open-source project that you built: ECS Compose-X.So, I assume it's like, oh, it's like Docker Compose for ECS and also the ‘X' implies that it is extreme, just, like, you know, snack foods at the convenience store. What does it do and where'd it come from?John: Right. So, you said most of it, right? It literally is a question where you take a Docker Compose file and you want to deploy your services that you worked on and all of that together, and you want to deploy it to AWS. So, ECS Compose-X is a CLI tool very much like the Copilot. I think it was released about four months just before Copilots came out—so, sorry, I beat you to the ball there—but with the Docker Compose specification supported.And again, it was really out of I needed to find a neat way to take my services and deploy them in AWS. So, Compose-X is just a CLI tool that is going to parse your Docker Compose file and create CloudFormation templates out of it. Now, the X is not very extreme or anything like that, but it's actually coming from the [finite 00:18:59] extension fields, which is something supported in Docker Compose. And so, you can do things like x-RDS, or x-DynamoDB, which Docker Compose on your laptop will totally ignore, but ECS Compose-X however will take that into account.And what it will do is if you need a database or a DynamoDB table, for example, in your Docker Compose file, you do [x-RDS, my database, some properties, 00:19:22]—exactly the same properties as CloudFormation, actually—and then you say, “I want this service to have access to it in read-only fashion.” And what ECS Compose-X is going to do is just understand what it has to do when—meaning creating IAM policies, opening security groups, all of that stuff, and make all of that available to the containers in one way or another.Corey: It feels like it's a bit of a miss for Copilot not to do this. It feels like they wanted to go off in their own direction with the way that they viewed the world—which I get; I'm not saying there's anything inherently wrong with that. There's a reason that I point kubernetestheeasyway.com to the ECS marketing site—but there's so much stuff out there that is shipped or made available in other ways with a Docker Compose file, and the question of okay, how do I take this and run it in Fargate or something because I don't want to run it locally for whatever reason, and the answer is, “That's the neat part. You don't.”And it just becomes such a clear miss. There have been questions about this Since Copilot launched. There's a GitHub issue tracking getting support for this that was last updated in September—we are currently recording this at the end of March—it just doesn't seem to be something that's a priority. I mean, I will say the couple of times that I've used Copilot myself, it was always for greenfield experiments, never for adopting something else that already existed. And that was… it just felt like a bit of a heavy lift to me of oh, you need to know from the beginning that this is the tool you're going to use for the thing. Docker Compose is what the ecosystem has settled on a long time ago and I really am disheartened by the fact that there's no direct ECS support for it today.John: Yeah, and it was definitely a motivation for me because I knew that ECS CLI version 1 was going into the sunset, and there wasn't going to be anything supporting it. And so, I just wanted to have Docker Compose because it's familiar to developers and again, if you want to have adoption and have people use your thing, it has to be easy. And when I looked at Copilot the first time around, I was extremely excited because I thought, “Yes, thank you, Amazon for making my life easy. I don't have to maintain this project anymore and I'm going to be able to just lift and shift, move over, and be happy about it.” But when the specification for Copilot was out and I could go for the documentation, I was equally disheartened because I was like, “Okay, not for me.”And something very similar happened when they announced Proton. I was extremely excited by Proton. I opened a GitHub issue on the roadmap immediately to say, “Hey, are you going to support to have some of those things together or not?” And the fact that the Proton templates—I mean, again, it was, what, two, three years ago now—and I haven't looked at Proton since, so it was a very long time now.Corey: The beta splasher was announced in 2020 and I really haven't seen much from it since.John: Well, and I haven't done anything [unintelligible 00:22:07] with it. And literally, one of the first thing did when the project came out. Because obviously, this is an open-source project that we use in Sainsbury's, right because we deploy everything in [ECS 00:22:17] so why would I reinvent the wheel the third time? It's been done, I might as well leverage it. But every time something on it came out, I was seeing it as the way out of nobody's going to need me anymore—which is great—and that doesn't create a huge potential dependency on the company for me, oh, well, we need this to, you know, keep working.Now, it's open-source, it's on the license you can fork it and do whatever you want with it, so from that point of view, nobody's going to ask me anything in the future, but from the point of view where I need to, as much as possible, use AWS native tools, or AWS-built tools, I differently wanted every time to move over to something different. But every time I tried and tiptoed with those alternative offerings, I just went back and said, “No, this [laugh] either is too new and not mature enough yet, or my tool is just better.” Right? And one of the things I've been doing for the past three years is look at the Docker ECS plugin, all of the issues, and I see all of the feature requests that people are asking for and just do that in my project. And some with Copilots. The only thing that Copilot does that I don't do is tell people how to do CI/CD pipelines.Corey: One thing you said a second ago just sort of, I guess, sent me spiraling for a second because I distinctly remember this particular painful part. You're right, there was an ECS CLI for a long time that has since been deprecated. But we had internal tooling built around that. When there was an issue with a particular task that failed, getting logs out of it was non-trivial, so great. Here's the magic incantation that does it.I still haven't found a great way to do that with the AWS v2 CLI and that feels like it's a gap where yes, I understand, old tools go away and new ones show up, but, “Hey, I [unintelligible 00:24:05] task. Can you tell me what the logs are?” “No. Well, Copilot's the new answer.” “Okay. Can I use this to get logs from something that isn't Copilot?” “Oh, absolutely not.” And the future is inherently terrible as a direct result.John: Yeah. Well, I mean, again, the [unintelligible 00:24:20]—the only thing that ECS Compose-X does is create all the templates for you so you can, you know, then just query it and know where everything has been created. And one of the things it definitely does create is all of the log groups. Because again, least-privileged permissions being something that is very dear to me, I create the log groups and just allow the services to only write in those log groups and that's it.Now, typically this is not a thing that I've thought Compose-X was going to do because that's not its purpose. It's not going to be an operational tool to troubleshoot all the things and this is where I think that other projects are much better suited and I would rather use them as an extension or library of the project as opposed to reinvent them. So, if you're trying to find a tool for yourself to look at logs, I highly recommend something called ‘AWS logs,' which is fantastic. You just say, “Hey, can you list the groups?” “Okay.” “Can you get me the groups and can I tell them on a terminal?”And that's it. Job done. So, as much as I enjoy building new features into the project, for example, I think that there's a clear definition between what the project is for and what it's not. And what it's for is giving people CloudFormation templates they can reuse in any region and deploy their services and not necessarily deal with their operations; that's up to them. At the end of the day, it's really up to the user to know what they want to do with it. I'm not trying to force anybody into doing something specific.Corey: I would agree. I think that there's value to there's more than one way to do it. The problem is, at some point, there's a tipping point where you have this proliferation of different options to the point where you end up in this analysis paralysis model where you're too busy trying to figure out what is the next clear step. And yes, that flexibility is incredibly valuable, especially when you get into, you know, large, sophisticated enterprises—ahem, ahem—but when you're just trying to kick the tires on something new, I feel like there's a certain lack of golden path where in the event of not having an opinion on any of these things, this is what you should do just to keep things moving forward, as opposed to here are two equal options that you can check with radio boxes and it's not at all clear what you which does what or what the longer-term implications are. We've all gotten caught with the one-way doors we didn't realize we were passing through at the time and then had to do significant technical debt repayment efforts to wind up making it right again.I just wish that those questions would be called out, but everything else just, it doesn't matter. If you don't like the name of the service that you're creating, you can change it later. Or if you can't, maybe you should know now, so you don't have—in my case—a DynamoDB table that is named ‘test' running in production forever.John: Yeah. You're absolutely right. And again, I think it goes back to one of the biggest challenges that I had when I joined the company, which was when I said, “I think we should be using CloudFormation, I think we should be using ECS and Terraforming Kubernetes for those reasons.” And one of the reasons was, the people. Meaning we were a very small team, only five cloud engineers at the time.And as I joined the company, they were already was three different teams using four different CI/CD tools. And they all wanted to use Kubernetes, for example, and they were all using different CI/CD—like I said, just now—different CI/CD tools. And so, the real big challenge for me was how do I pitch that simplicity is what's going to allow us to deliver value for the business? Because at the end of the day, like you said many, many times before, the AWS bill is a question of architecture, right? And there's a link and intricacy between the two things.So, the only thing that really mattered for me and the team was to find a way, find the service that was going to allow to do a number of things, A, delivering value quickly, being supported over time. Because one of the things that I think people forget these days—well, one of the things I'm allergic to and one of the things that makes me spiral is what I call CV-driven tech choices where people say, “Hey, I love this great thing I read about and I think that we should use that in production. How great idea.” But really, I don't know anything about it and is then up to somebody else to maintain it long-term.And that goes to the other point, which is, turnover-proof is what I call it. So, making tech choices that are going to be something that people will be able to use for many, many years, there is going to be a company behind the scenes that he's going to be able to support you as well as you go and use the service for the many, many years to go.Corey: I really want to thank you for taking the time to speak with me today. If people want to learn more, where's the best place for them to find you?John: So, people can find me on LinkedIn. I'm also around on Twitter these days, although I probably about have nine followers. Well, probably shouldn't say that [laugh] and that doesn't matter.Corey: It's fine. We'll put a link into it—we'll put a link to that in the [show notes 00:29:02] and maybe we'll come up with number ten. You never know. Thanks again for your time. I really appreciate it.John: Thanks so much, Corey, for having me.Corey: John Mille, Principal Cloud Engineer at Sainsbury's. I'm Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with an angry comment that you go to great pains to type out but then fails to post because the version of the tool you use to submit it has been deprecated without a viable replacement.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.
Raj Bala, Founder of Perspect, joins Corey on Screaming in the Cloud to discuss all things generative AI. Perspect is a new generative AI company that is democratizing the e-commerce space, by making it possible to place images of products in places that would previously require expensive photoshoots and editing. Throughout the conversation, Raj shares insights into the legal questions surrounding the rise of generative AI and potential ramifications of its widespread adoption. Raj and Corey also dig into the question, “Why were the big cloud providers beaten to the market by OpenAI?” Raj also shares his thoughts on why company culture has to be organic, and how he's hoping generative AI will move the needle for mom-and-pop businesses. About RajRaj Bala, formerly a VP, Analyst at Gartner, led the Magic Quadrant for Cloud Infrastructure and Platform Services since its inception and led the Magic Quadrant for IaaS before that. He is deeply in-tune with market dynamics both in the US and Europe, but also extending to China, Africa and Latin America. Raj is also a software developer and is capable of building and deploying scalable services on the cloud providers to which he wrote about as a Gartner analyst. As such, Raj is now building Perspect, which is a SaaS offering at the intersection of AI and E-commerce.Raj's favorite language is Python and he is obsessed with making pizza and ice cream. Links Referenced:Perspect: https://perspect.com TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by our friends at Thinkst Canary. Most folks find out way too late that they've been breached. Thinkst Canary changes this. Deploy Canaries and Canarytokens in minutes and then forget about them. Attackers tip their hand by touching 'em giving you one alert, when it matters. With 0 admin overhead and almost no false-positives, Canaries are deployed (and loved) on all 7 continents. Check out what people are saying at canary.love today!Corey: This episode is sponsored in part by our friends at Chronosphere. When it costs more money and time to observe your environment than it does to build it, there's a problem. With Chronosphere, you can shape and transform observability data based on need, context and utility. Learn how to only store the useful data you need to see in order to reduce costs and improve performance at chronosphere.io/corey-quinn. That's chronosphere.io/corey-quinn. And my thanks to them for sponsor ing my ridiculous nonsense. Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. Back again, after a relatively brief point in time since the last time he was on, is Raj Bala. Formerly a VP analyst at Gartner, but now instead of talking about the past, we are talking, instead, about the future. Raj, welcome back. You're now the Founder at Perspect. What are you doing over there?Raj: I am indeed. I'm building a SaaS service around the generative AI space at the intersection of e-commerce. So, those two things are things that I'm interested in. And so, I'm building a SaaS offering in that space.Corey: This is the first episode in which we're having an in-depth discussion about generative AI. It's mostly been a topic that I've avoided because until now, relatively recently, it's all been very visual. And it turns into sort of the next generation's crappy version of Instagram, where, “Okay. Well, Instagram's down, so can you just describe your lunch to me?” It's not compelling to describe a generated image on an audio-based podcast. But with the advent of things like ChatGPT, where suddenly it's muscling into my realm, which is the written word, suddenly it feels like there's a lot more attention and effort being paid to it in a bunch of places where it wasn't getting a lot of coverage before, including this one. So, when you talk about generative AI, are you talking in the sense of visual, in terms of the written word, in terms of all of the above, and more? Where's your interest lie?Raj: I think it's all of the above and more. My interest is in all of it, but my focus right now is on the image aspect of things. I've been pretty knee-deep in stable diffusion and all the things that it entails, and it is largely about images at this point.Corey: So, talk to me more about how you're building something that stands between the intersection of e-commerce and generative AI. Because when I go to perspect.com, I'm not staring at a web store in the traditional sense. I'm looking at something that—again, early days, I'm not judging you based upon the content of your landing page—but it does present as a bit more of a developer tool and a little bit less of a “look how pretty it is.”Raj: Yeah. It's very much a developer-focused e-commerce offering. So, as a software developer, if you want programmatic access to all things e-commerce and generative AI that are related to e-commerce, you can do that on perspect.com. So, yeah. It is about taking images of products and being able to put them in AI-generated places essentially.Corey: Got it. So, effectively you're trying to sell, I don't know, titanium jewelry for the sake of argument. And you're talking about now you can place it on a generated model's hand to display this rather than having to either fake it or alternately have a whole bunch of very expensive product shoots and modeling sessions.Raj: Exactly. Exactly. If you want to put this piece of jewelry in front of the Eiffel Tower or the Pyramids of Giza, you can do that in a few seconds as opposed to the expensive photo shoots that were once required.Corey: On some level, given that I spend most of my time kicking around various SaaS products, I kind of love the idea of stock photography modeling, I don't know, Datadog or whatnot. I don't know how that would even begin to pan out, but I'm totally here for it.Raj: That's funny.Corey: Now, the hard part that I'm seeing right now is—I mean, you used to work at Gartner for years.Raj: I did.Corey: You folks are the origin of the Gartner-hype cycle. And given all the breathless enthusiasm, massive amounts of attention, and frankly, hilarious, more than a little horrifying, missteps that we start seeing in public, it feels like we are very much in the heady early days of hype around generative AI.Raj: No doubt about it. No doubt about it. But just thinking about what's possible and what is being accomplished even week to week in this space is just mind-blowing. I mean, this stuff didn't exist really six months ago. And now, the open-source frameworks are out there. The ecosystems are developing around it. A lot of people have compared generative AI to the iPhone. And I think it's actually maybe bigger than that. It's more internet-scale disruption, not just a single device like the iPhone.Corey: It's one of those things that I have the sneaking suspicion is going to start showing up in a whole bunch of different places, manifesting in a whole host of different ways. I've been watching academia, largely, freak out about the idea that, “Well, kids are using it to cheat on their homework.” Okay. I understand the position that they're coming from. But it seems like whenever a new technology is unleashed on the world, that is the immediate, instantaneous, reflexive blowback—not necessarily picking on academics, in particular—but rather, the way that we've always done something is now potentially very easy to use thanks to this advance in technology. “Oh, crap. What do we do?” And there's usually a bunch of scurrying around in futile attempts to put the genie back in the bottle, which frankly, never works. And you also see folks trying to sprint, to sort of keep up with this. And it never really pans out. Society adapts, adjusts, and evolves. And I don't think that that's an inherently terrible thing.Raj: I don't think so either. I mean, the same thing existed with the calculator, right? Do you remember early days in school, they said you can't use a calculator, right? And—Corey: Because remember you will not have a calculator in your pocket as you go through life. Well, that was a lie.Raj: But during the test—during the test that you have to take, you will not have a calculator. And when the rubber meets the road in person during that test, you're going to have to show your skills. And the same thing will happen here. We'll just have to have ground rules and ways to check and balance whether people are cheating or not and adapt, just like you said.Corey: On some level, you're only really cheating yourself past a certain point.Raj: Exactly.Corey: There's value in being able to tell a story in a cohesive, understandable way.Raj: Absolutely.Corey: Oh, the computer will do it for me. And I don't know that you can necessarily depend on that.Raj: Absolutely. Absolutely. You have to understand more than just the inputs and outputs. You have to understand the black box in between enough to show that you understand the subject.Corey: One thing that I find interesting is the position of the cloud providers in this entire—Raj: Mm-hmm.Corey: —space. We have Google, who has had a bunch of execs talking about how they've been working on this internally for years. Like you get how that makes you look worse instead of better, right? Like they're effectively tripping over one another on LinkedIn to talk about how they've been working on this for such a long time, and they have something just like it. Well, yeah. Okay. You got beaten to market by a company less than a decade old.Azure has partnered with OpenAI and brought a lot of this to Bing so rapidly they didn't have time to update their more of a Bing app away from the “Use Bing and earn Microsoft coins” nonsense. It's just wow. Talk about a—being caught flat-footed on this. And Amazon, of course, has said effectively nothing. The one even slightly generative AI service that they have come out with that I can think of that anyone could be forgiven for having missed is—they unleashed this one year at re:Invent's Midnight Madness where they had Dr. Matt Wood get on stage with the DeepComposer and play a little bit of a song. And it would, in turn, iterate on that. And that was the last time any of us ever really heard anything about the DeepComposer. I've got one on my shelf. And I do not hear about it mentioned even in passing other than in trivia contests.Raj: Yeah. It's pretty fascinating. Amazon with all their might, and AWS in particular—I mean, AWS has Alexa, and so they've—the thing you give to Alexa is a prompt, right? I mean, it is generative AI in a large way. You're feeding it a prompt and saying do something. And it spits out something tokenized to you. But the fact that OpenAI has upended all of these companies I think is massive. And it tells us something else about Microsoft too is that they didn't have the wherewithal integrally to really compete themselves. They had to do it with someone else, right? They couldn't muster up the effort to really build this themselves. They had to use OpenAI.Corey: On some level, it's a great time to be a cloud provider because all of these experiments are taking place on top of a whole bunch of very expensive, very specific compute.Raj: Sure.Corey: But that is necessary but not sufficient as we look at it across the board. Because even AWS's own machine-learning powered services, it's only relatively recently that they seemed to have gotten above the “Step one, get a PhD in this stuff. Step two, here's all the nuts and bolts you have to understand about how to build machine-learning models.” Whereas the thing that's really caused OpenAI's stuff to proliferate in the public awareness is, “Okay. You got to a webpage, and you tell it what to draw, and it draws the thing.” Or “go ahead and rename an AWS service if the naming manager had a sense of creativity and a slight bit of whimsy.” And it comes out with names that are six times better than anything AWS has ever come out with.Raj: So, funny. I saw your tweet on that actually. Yeah. If you want to do generative AI on AWS today, it is hard. Oh, my gosh. That's if you can get the capacity. That's if you can get the GPU capacity. That's if you can put together the ML ops necessary to make it all happen. It is extremely hard. Yeah, so putting stuff into a chat interface is 1,000 times easier. I mean, doing something like containers on GPUs is just massively difficult in the cloud today.Corey: It's hard to get them in many cases as well. I had customers that asked, “Okay. What special preferential treatment can we get to get access to more GPUs?” It's like can you break the laws of physics or change global supply chain because if so, great. You've got this unlocked. Otherwise, good luck.Raj: I think us-east-2 a couple weeks ago for like the entire week was out of the GPU capacity necessary the entire week.Corey: I haven't been really tracking a lot of the GPU-specific stuff. Do you happen to know what a lot of OpenAI's stuff is built on top of from a vendoring perspective?Raj: I mean, it's got to be Nvidia, right? Is that what you're asking me?Corey: Yeah. I'm—I don't know a lot of the—again, this is not my area.Raj: Yeah, yeah.Corey: I am not particularly deep in the differences between the various silicon manufacturers. I know that AWS has their Inferentia chipset that's named after something that sounds like what my grandfather had. You've got a bunch of AMD stuff coming out. You've have—Intel's been in this space for a while. But Nvidia has sort of been the gold standard based upon GPU stories. So, I would assume it's Nvidia.Raj: At this point, they're the only game in town. No one else matters. The frameworks simply don't support anything other than Nvidia. So, in fact, OpenAI—them and Facebook—they are kind of leading some—a bunch of the open-source right now. So, it's—Stability AI, Hugging Face, OpenAI, Facebook, and all their stuff is dependent on Nvidia. None of it—if you look through the source code, none of it really relies on Inferentia or Trainium or AMD or Intel. It's all Nvidia.Corey: As you look across the current landscape—at least—let me rephrase that. As I look across the current landscape, I am very hard-pressed to identify any company except probably OpenAI itself as doing anything other than falling all over itself having been caught—what feels like—completely flat-footed. We've seen announcements rushed. We've seen people talking breathlessly about things that are not yet actively available. When does that stop? When do we start to see a little bit of thought and consideration put into a lot of the things that are being rolled out, as opposed to “We're going to roll this out as an assistant now to our search engine” and then having to immediately turn it off because it becomes deeply and disturbingly problematic in how it responds to a lot of things?Raj: You mean Sam Altman saying he's got a lodge in Montana with a cache of firearms in case AI gets out of control? You mean that doesn't alarm you in any way?Corey: A little bit. Just a little bit. And like even now you're trying to do things that, to be clear, I am not trying to push the boundaries of these things. But all right. Write a limerick about Elon Musk hurling money at things that are ridiculous. Like, I am not going to make fun of individual people. It's like I get that. But there is a punching-up story around these things. Like, you also want to make sure that's it not “Write a limerick about the disgusting habit of my sixth-grade classmate.” Like, you don't want to, basically, automate the process of cyber-bullying. Let's be clear here. But finding that nuance, it's a societal thing to wrestle with, on some level. But I think that we're anywhere near having cohesive ideas around it.Raj: Yeah. I mean, this stuff is going to be used for nefarious ways. And it's beyond just cyberbullying, too. I think nation-states are going to use this stuff to—as a way to create disinformation. I mean, if we saw a huge flux of disinformation in 2020, just imagine what's going to happen in 2024 with AI-generated disinformation. That's going to be off the charts.Corey: It'll be at a point where you fundamentally have to go back to explicitly trusted sources as opposed to, “Well, I saw a photo of it or a video of it” or someone getting onstage and dancing about it. Well, all those things can be generated now for, effectively, pennies.Raj: I mean, think about evidence in a courtroom now. If I can generate an image of myself holding a gun to someone's head, you have to essentially dismiss all sorts of photographic evidence or video evidence soon enough in court because you can't trust the authenticity of it.Corey: It makes providence and chain-of-custody issues far more important than they were before. And it was still a big deal. Photoshop has been around for a while. And I remember thinking when I was younger, “I wonder how long it'll be until videos become the next evolution of this.” Because there was—we got to a point fairly early on in my life where you couldn't necessarily take a photograph at face value anymore because—I mean, look at some of the special effects we see in movies. Yeah, okay. Theoretically, someone could come up with an incredibly convincing fake of whatever it is that they're trying to show. But back then, it required massive render farms and significant investment to really want to screw someone over. Now, it requires drinking a couple glasses of wine, going on the internet late at night, navigating to the OpenAI webpage, and typing in the right prompt. Maybe you iterate on it a little bit, and it spits it out for basically free.Raj: That's one of the sectors, actually, that's going to adopt this stuff the soonest. It's happening now, the film and movie industry. Stability AI actually has a film director on staff. And his job is to be sort of the liaison to Hollywood. And they're going to help build AI solutions into films and so forth. So, yeah. But that's happening now.Corey: One of the more amazing things that I've seen has been the idea of generative voice where it used to be that in order to get an even halfway acceptable model of someone's voice, they had to read a script for the better part of an hour. That—and they had to make sure that they did it with certain inflection points and certain tones. Now, you can train these things on, “All right. Great. Here's this person just talking for ten minutes. Here you go.” And the reason I know this—maybe I shouldn't be disclosing this as publicly as I am, but the heck with it. We've had one of me on backup that we've used intermittently on those rare occasions when I'm traveling, don't have my recording setup with me, and this needs to go out in a short time period. And we've used it probably a dozen times over the course of the 400 and some odd episodes we've done. One person has ever noticed.Raj: Wow.Corey: Now, having a conversation going back and forth, start pairing some of those better models with something like ChatGPT, and basically, you're building your own best friend.Raj: Yeah. I mean, soon enough you'll be able to do video AI, completely AI-generated of your podcast perhaps.Corey: That would be kind of wild, on some level. Like now we're going to animate the whole thing.Raj: Yeah.Corey: Like I still feel like we need more action sequences. Don't know about you, but I don't have quite the flexibility I did when I was younger. I can't very well even do a pratfall without wondering if I just broke a hip.Raj: You can have an action sequence where you kick off a CloudFormation task. How about that?Corey: One area where I have found that generative text AI, at least, has been lackluster, has been right a parody of the following song around these particular dimensions. Their meter is off. Their—the cleverness is missing.Raj: Hmm.Corey: They at least understand what a parody is and they understand the lyrics of the song, but they're still a few iterative generations away. That said, I don't want to besmirch the work of people who put into these things. They are basically—Raj: Mm-hmm.Corey: —magic.Raj: For sure. Absolutely. I mean, I'm in wonderment of some of the artwork that I'm able to generate with generative AI. I mean, it is absolutely awe-inspiring. No doubt about it.Corey: So, what's gotten you excited about pairing this specifically with e-commerce? That seems like an odd couple to wind up smashing together. But you have had one of the best perspectives on this industry for a long time now. So, my question is not, “What's wrong with you?” But rather, “What are you seeing that I'm missing?”Raj: I think it's the biggest opportunity from an impact perspective. Generating AI avatars of yourself is pretty cool. But ultimately, I think that's a pretty small market. I think the biggest market you can go after right now is e-commerce in the generative AI space. I think that's the one that's going to move the needle for a lot of people. So, it's a big opportunity for one. I think there are interesting things you can do in it. The technical aspects are equally interesting. So, you know, there are a handful of compelling things that draw me to it.Corey: I think you're right. There's a lot of interest and a lot of energy and a lot of focus built around a lot of the neat, flashy stuff. But it's “Okay. How does this windup serving problems that people will pay money for?” Like right now to get early access to ChatGPT and not get rate-limited out, I'm paying them 20 bucks a month which, fine, whatever. I am also in a somewhat privileged position. If you're generating profile photos that same way, people are going to be very hard-pressed to wind up paying more than a couple bucks for it. That's not where the money is. But solving business problems—and I think you're onto something with the idea of generative photography of products that are for sale—that has the potential to be incredibly lucrative. It tackles what to most folks is relatively boring, if I can say that, as far as business problems go. And that's often where a lot of value is locked away.Raj: I mean, in a way, you can think of generative AI in this space as similar to what cloud providers themselves do. So, the cloud providers themselves afforded much smaller entities the ability to provision large-scale infrastructure without high fixed costs. And in some ways, I know the same applies to this space too. So, now mom-and-pop shop-type people will be able to generate interesting product photos without high fixed costs of photoshoots and Photoshop and so forth. And so, I think in some ways it helps to democratize some of the bigger tools that people have had access to.Corey: That's really what it comes down to is these technologies have existed in labs, at least, for a little while. But now, they're really coming out as interesting, I guess, technical demos, for lack of a better term. But the entire general public is having access to these things. There's not the requirement that we wind up investing an enormous pile of money in custom hardware and the rest. It feels evocative of the revolution that was cloud computing in its early days. Where suddenly, if I have an idea, I don't need either build it on a crappy computer under my desk or go buy a bunch of servers and wait eight weeks for them to show up in a rack somewhere. I can just start kicking the tires on it immediately. It's about democratizing access. That, I think, is the magic pill here.Raj: Exactly. And the entry point for people who want to do this as a business, so like me, it is a huge hurdle still to get this stuff running, lots of jagged edges, lots of difficulty. And I think that ultimately is going to dissuade huge segments of the population from doing it themselves. They're going to want completed services. They're going to want finish product, at least in some consumable form, for their persona.Corey: What do you think the shaking out of this is going to look like from a cultural perspective? I know that right now everyone's excited, both in terms of excited about the possibility and shrieking that the sky is falling, that is fairly normal for technical cycles. What does the next phase look like?Raj: The next phase, unfortunately, is probably going to be a lot of litigation. I think there's a lot of that on the horizon already. Right? Stability AI's being sued. I think the courts are going to have to decide, is this stuff above board? You know, the fact that these models have been trained on otherwise copywritten data—copywritten images and music and so forth, that amounts to billions of parameters. How does that translate—how does that affect ages of intellectual property law? I think that's a question that—it's an open question. And I don't think we know.Corey: Yeah. I wish, on some level, that we could avoid a lot of the unpleasantness. But you're right. It's going to come down to a lot of litigation, some of which clearly has a point, on some level.Raj: For sure.Corey: But it's a—but that is, frankly, a matter for the courts. I'm privileged that I don't have to sit here and worry about this in quite the same way because I am not someone who makes the majority of my income through my creative works. And I am also not on the other side of it where I've taken a bunch of other people's creative output and use that to train a bunch of models. So, I'm very curious to know how that is going to shake out as a society.Raj: Yeah.Corey: I think that regulation is also almost certainly on the horizon, on some level. I think that tech has basically burned through 25 years of goodwill at this point. And nobody trusts them to self-regulate. And based upon their track record, I don't think they should.Raj: And interestingly, I think that's actually why Google was caught so flat-footed. Google was so afraid of the ramifications of being first and the downside optics of that, that they got a little complacent. And so, they weren't sure how the market would react to saying, “Here's this company that's known for doing lots of, you know, kind of crazy things with people's data. And suddenly they come out with this AI thing that has these huge superpowers.” And how does that reflect poorly on them? But it ended up reflecting poorly on them anyway because they ended up being viewed as being very, very late to market. So, yeah. They got pie on their face one way or the other.Corey: For better or worse, that's going to be one of those things that haunts them. This is the classic example of the innovator's dilemma. By becoming so focused on avoiding downside risk and revenue protection, they effectively let their lunch get eaten. I don't know that there was another choice that they could've made. I'm not sitting here saying, “That's why they're foolish.” But it's painful. If I'm—I'm in the same position right now. If I decide I want to launch something new and exciting, my downside risk is fairly capped. The world is my theoretical oyster. Same with most small companies. I don't know about you, what do you right now as a founder, but over here at The Duckbill Group, at no point in the entire history of this company, going back six years now, have we ever sat down for a discussion around, “Okay. If we succeed at this, what are the antitrust implications?” It has never been on our roadmap. It's—that's very firmly in the category of great problems to have.Raj: Really confident companies will eat their own lunch. So, you in fact see AWS do this all the time.Corey: Yes.Raj: They will have no problem disrupting themselves. And they're lots of data points we can talk about to show this. They will disrupt themselves first because they're afraid of someone else doing it before them.Corey: And it makes perfect sense. Amazon has always had a—I'd call it a strange culture, but that doesn't do it enough of a service just because it feels like compared to virtually any other company on the planet, they may as well be an alien organism that has been dropped into the world here. And we see a fair number of times where folks have left Amazon, and they wind up being so immersed in that culture, that they go somewhere else, and “Ah, I'm just going to bring the leadership principles with me.” And it doesn't work that way. A lot of them do not pan out outside of the very specific culture that Amazon has fostered. Now, I'm not saying that they're good or that they're bad. But it is a uniquely Amazonian culture that they have going on there. And those leadership principles are a key part of it. You can transplant that in the same way to other companies.Raj: Can I tell you one of the funniest things one of these cloud providers has said to me? I'm not going to mention the cloud provider. You may be able to figure out which one anyway, though.Corey: No. I suspect I have a laundry list to go out of these various, ridiculous things I have heard from companies. Please, I am always willing to add to the list. Hit me with it.Raj: So, a cloud provider—a big cloud provider, mine you—told me that they wanted Amazon's culture so bad that they began a thing where during a meeting—before each meeting, everyone would sit quietly and read a paper that was written by someone in the room so they all got on the same page. And that is distinctly an Amazon thing, right? And this is not Amazon that is doing this. This is some other cloud provider. So, people are so desperate for that bit of weirdness that you mentioned inside of Amazon, that they're willing to replicate some of the movements and the behaviors whole cloth hoping that they can get that same level of culture. But it has to be—it has to be organic. And it has to be at the root. You can't just take an arm and stick it onto a body and expect it to work, right?Corey: My last real job before I started this place was at a small, scrappy startup for three months. And then we were bought by an enormous financial company. And one of their stated reasons for doing it was, “Well, we really admire a lot of your startup culture, and we want to, basically, socialize that and adopt that where we are.” Spoiler. This did not happen. It was more or less coming from a perspective, “Well, we visited your offices, and we saw that you had bikes in the entryway and dogs in the office. And well, we went back to our office, and we threw in some bikes and added some dogs, but we didn't get any different result. What's the story here?” It's—you cannot cargo cult bits and pieces of a culture. It has to be something holistic. And let's be clear, you're only ever going to be second best at being another company. They're going to be first place. We saw this a lot in the early-2000s of “We're going to be the next Yahoo.” It's—why would I care about that? We already have original Yahoo. The fortune's faded, but here we are.Raj: Yeah. Agreed.Corey: On our last recording, you mentioned that you would be building this out on top of AWS. Is that how it's panned out? You still are?Raj: For the most part. For the most part. I've dipped my toes into trying to use GPU capacity elsewhere, using things like ECS Anywhere, which is an interesting route. There's some promise there, but there's still lots of jagged edges there too. But for the most part, there's not another cloud provider that really has everything I need from GPU capacity to serverless functions at the edge, to CDNs, to SQL databases. That's actually a pretty disparate set of things. And there's not another cloud provider that has literally all of that except AWS at this point.Corey: So far, positive experience or annoying? Let's put it that way.Raj: Some of it's really, really hard. So, like doing GPUs with generative AI, with containers for instance, is still really, really hard. And the documentation is almost nonexistent. The documentation is wrong. I've actually submitted pull requests to fix AWS documentation because a bunch of it is just wrong. So, yeah. It's hard. Some of it's really easy. Some it's really difficult.Corey: I really want to thank you for taking time to speak about what you're up to over at Perspect. Where can people go to learn more?Raj: www.perspect.com.Corey: And we will of course put a link to that in the [show notes 00:30:02]. Thank you so much for being so generous with your time. I appreciate it.Raj: Any time, Corey.Corey: Raj Bala, Founder at Perspect. I'm Cloud Economist Corey Quinn. And this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas, if you haven't hated this podcast, please, leave a five-star review on your podcast platform of choice along with an angry, insulting comment that you got an AI generative system to write for you.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.
Waldemar Hummer, Co-Founder & CTO of LocalStack, joins Corey on Screaming in the Cloud to discuss how LocalStack changed Corey's mind on the futility of mocking clouds locally. Waldemar reveals why LocalStack appeals to both enterprise companies and digital nomads, and explains how both see improvements in their cost predictability as a result. Waldemar also discusses how LocalStack is an open-source company first and foremost, and how they're working with their community to evolve their licensing model. Corey and Waldemar chat about the rising demand for esoteric services, and Waldemar explains how accommodating that has led to an increase of adoption from the big data space. About WaldemarWaldemar is Co-Founder and CTO of LocalStack, where he and his team are building the world-leading platform for local cloud development, based on the hugely popular open source framework with 45k+ stars on Github. Prior to founding LocalStack, Waldemar has held several engineering and management roles at startups as well as large international companies, including Atlassian (Sydney), IBM (New York), and Zurich Insurance. He holds a PhD in Computer Science from TU Vienna.Links Referenced: LocalStack website: https://localstack.cloud/ LocalStack Slack channel: https://slack.localstack.cloud LocalStack Discourse forum: https://discuss.localstack.cloud LocalStack GitHub repository: https://github.com/localstack/localstack TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. Until a bit over a year ago or so, I had a loud and some would say fairly obnoxious opinion around the futility of mocking cloud services locally. This is not to be confused with mocking cloud services on the internet, which is what I do in lieu of having a real personality. And then one day I stopped espousing that opinion, or frankly, any opinion at all. And I'm glad to be able to talk at long last about why that is. My guest today is Waldemar Hummer, CTO and co-founder at LocalStack. Waldemar, it is great to talk to you.Waldemar: Hey, Corey. It's so great to be on the show. Thank you so much for having me. We're big fans of what you do at The Duckbill Group and Last Week in AWS. So really, you know, glad to be here with you today and have this conversation.Corey: It is not uncommon for me to have strong opinions that I espouse—politely to be clear; I'll make fun of companies and not people as a general rule—but sometimes I find that I've not seen the full picture and I no longer stand by an opinion I once held. And you're one of my favorite examples of this because, over the course of a 45-minute call with you and one of your business partners, I went from, “What you're doing is a hilarious misstep and will never work,” to, “Okay, and do you have room for another investor?” And in the interest of full disclosure, the answer to that was yes, and I became one of your angel investors. It's not exactly common for me to do that kind of a hard pivot. And I kind of suspect I'm not the only person who currently holds the opinion that I used to hold, so let's talk a little bit about that. At the very beginning, what is LocalStack and what does it you would say that you folks do?Waldemar: So LocalStack, in a nutshell, is a cloud emulator that runs on your local machine. It's basically like a sandbox environment where you can develop your applications locally. We have currently a range of around 60, 70 services that we provide, things like Lambda Functions, DynamoDB, SQS, like, all the major AWS services. And to your point, it is indeed a pretty large undertaking to actually implement the cloud and run it locally, but with the right approach, it actually turns out that it is feasible and possible, and we've demonstrated this with LocalStack. And I'm glad that we've convinced you to think of it that way as well.Corey: A couple of points that you made during that early conversation really stuck with me. The first is, “Yeah, AWS has two, no three no four-hundred different service offerings. But look at your customer base. How many of those services are customers using in any real depth? And of those services, yeah, the APIs are vast, and very much a sprawling pile of nonsense, but how many of those esoteric features are those folks actually using?” That was half of the argument that won me over.The other half was, “Imagine that you're an enormous company that's an insurance company or a bank. And this year, you're hiring 5000 brand new developers, fresh out of school. Two to 3000 of those developers will still be working here in about a year as they wind up either progressing in other directions, not winding up completing internships, or going back to school after internships, or for a variety of reasons. So, you have that many people that you need to teach how to use cloud in the context that we use cloud, combined with the question of how do you make sure that one of them doesn't make a fun mistake that winds up bankrupting the entire company with a surprise AWS bill?” And those two things combined turned me from, “What you're doing is ridiculous,” to, “Oh, my God. You're absolutely right.”And since then, I've encountered you in a number of my client environments. You were absolutely right. This is something that resonates deeply and profoundly with larger enterprise customers in particular, but also folks who just don't want to wind up being beholden to every time they do a deploy to anything to test something out, yay, I get to spend more money on AWS services.Waldemar: Yeah, totally. That's spot on. So, to your first point, so definitely we have a core set of services that most people are using. So, things like Lambda, DynamoDB, SQS, like, the core serverless, kind of, APIs. And then there's kind of a long tail of more exotic services that we support these days, things like, even like QLDB, the quantum ledger database, or, you know, managed streaming for Kafka.But like, certainly, like, the core 15, 20 services are the ones that are really most used by the majority of people. And then we also, you know, pro offering have some very, sort of, advanced services for different use cases. So, that's to your first point.And second point is, yeah, totally spot on. So LocalStack, like, really enables you to experiment in the sandbox. So, we both see it as an experimentation, also development environment, where you don't need to think about cloud costs. And this, I guess, will be very close to your heart in the work that you're doing, the costs are becoming really predictable as well, right? Because in the cloud, you know, work to different companies before doing LocalStack where we were using AWS resources, and you can end up in a situation where overnight, you accumulate, you know, hundreds of thousands of dollars of AWS bill because you've turned on a certain feature, or some, you know, connectivity into some VPC or networking configuration that just turns out to be costly.Also, one more thing that is worth mentioning, like, we want to encourage, like, frequent testing, and a lot of the cloud's billing and cost structure is focused around, for example, hourly billing of resources, right? And if you have a test that just spins up resources that run for a couple of minutes, you still end up paying the entire hour. And we LocalStack, really, that brings down the cloud builds significantly because you can really test frequently, the cycles become much faster, and it's also again, more efficient, more cost-effective.Corey: There's something useful to be said for, “Well, how do I make sure that I turn off resources when I'm done?” In cloud, it's a bit of a game of guess-and-check. And you turn off things you think are there and you wait a few days and you check the bill again, and you go and turn more things off, and the cycle repeats. Or alternately, wait for the end of the month and wonder in perpetuity why you're being billed 48 cents a month, and not be clear on why. Restarting the laptop is a lot more straightforward.I also want to call out some of my own bias on this where I used to be a big believer in being able to build and deploy and iterate on things locally because well, what happens when I'm in a plane with terrible WiFi? Well, in the before times, I flew an awful lot and was writing a fair bit of, well, cloudy nonsense and I still never found that to be a particular blocker on most of what I was doing. So, it always felt a little bit precious to me when people were talking about, well, what if I can't access the internet to wind up building and deploying these things? It's now 2023. How often does that really happen? But is that a use case that you see a lot of?Waldemar: It's definitely a fair point. And probably, like, 95% of cloud development these days is done in a high internet bandwidth environment, maybe some corporate network where you have really fast internet access. But that's only a subset, I guess, of the world out there, right? So, there might be situations where, you know, you may have bad connectivity. Also, maybe you live in a region—or maybe you're traveling even, right? So, there's a lot more and more people who are just, “Digital nomads,” quote-unquote, right, who just like to work in remote places.Corey: You're absolutely right. My bias is that I live in San Francisco. I have symmetric gigabit internet at home. There's not a lot of scenarios in my day-to-day life—except when I'm, you know, on the train or the bus traveling through the city—because thank you, Verizon—where I have impeded connectivity.Waldemar: Right. Yeah, totally. And I think the other aspect of this is kind of the developers just like to have things locally, right, because it gives them the feeling of you know, better control over the code, like, being able to integrate into their IDEs, setting breakpoints, having these quick cycles of iterations. And again, this is something that there's more and more tooling coming up in the cloud ecosystem, but it's still inherently a remote execution that just, you know, takes the round trip of uploading your code, deploying, and so on, and that's just basically the pain point that we're addressing with LocalStack.Corey: One thing that did surprise me as well was discovering that there was a lot more appetite for this sort of thing in enterprise-scale environments. I mean, some of the reference customers that you have on your website include divisions of the UK Government and 3M—you know, the Post-It note people—as well as a number of other very large environments. And at first, that didn't make a whole lot of sense to me, but then it suddenly made an awful lot of sense because it seems—and please correct me if I'm wrong—that in order to use something like this at scale and use it in a way that isn't, more or less getting it into a point where the administration of it is more trouble than it's worth, you need to progress past a certain point of scale. An individual developer on their side project is likely just going to iterate against AWS itself, whereas a team of thousands of developers might not want to be doing that because they almost certainly have their own workflows that make that process high friction.Waldemar: Yeah, totally. So, what we see a lot is, especially in larger enterprises, dedicated teams, like, developer experience teams, whose main job is to really set up a workflow and environment where developers can be productive, most productive, and this can be, you know, on one side, like, setting up automated pipelines, provisioning maybe AWS sandbox and test accounts. And like some of these teams, when we introduce LocalStack, it's really a game-changer because it becomes much more decoupled and like, you know, distributed. You can basically configure your CI pipeline, just, you know, spin up the container, run your tests, tear down again afterwards. So, you know, it's less dependencies.And also, one aspect to consider is the aspect of cloud approvals. A lot of companies that we work with have, you know, very stringent processes around, even getting access to the clouds. Some SRE team needs to enable their IAM permissions and so on. With LocalStack, you can just get started from day one and just get productive and start testing from the local machine. So, I think those are patterns that we see a lot, in especially larger enterprise environments as well, where, you know, there might be some regulatory barriers and just, you know, process-wise steps as well.Corey: When I started playing with LocalStack myself, one of the things that I found disturbingly irritating is, there's a lot that AWS gets largely right with its AWS command-line utility. You can stuff a whole bunch of different options into the config for different profiles, and all the other tools that I use mostly wind up respecting that config. The few that extend it add custom lines to it, but everything else is mostly well-behaved and ignores the things it doesn't understand. But there is no facility that lets you say, “For this particular profile, use this endpoint for AWS service calls instead of the normal ones in public regions.” In fact, to do that, you effectively have to pass specific endpoint URLs to arguments, and I believe the syntax on that is not globally consistent between different services.It just feels like a living nightmare. At first, I was annoyed that you folks wound up having to ship your own command-line utility to wind up interfacing with this. Like, why don't you just add a profile? And then I tried it myself and, oh, I'm not the only person who knows how this stuff works that has ever looked at this and had that idea. No, it's because AWS is just unfortunate in that respect.Waldemar: That is a very good point. And you're touching upon one of the major pain points that we have, frankly, with the ecosystem. So, there are some pull requests against the AWS open-source repositories for the SDKs and various other tools, where folks—not only LocalStack, but other folks in the community have asked for introducing, for example, an AWS endpoint URL environment variable. These [protocols 00:12:32], unfortunately, were never merged. So, it would definitely make our lives a whole lot easier, but so far, we basically have to maintain these, you know, these wrapper scripts, basically, AWS local, CDK local, which basically just, you know, points the client to local endpoints. It's a good workaround for now, but I would assume and hope that the world's going to change in the upcoming years.Corey: I really hope so because everything else I can think of is just bad. The idea of building a custom wrapper around the AWS command-line utility that winds up checking the profile section, and oh, if this profile is that one, call out to this tool, otherwise it just becomes a pass-through. That has security implications that aren't necessarily terrific, you know, in large enterprise companies that care a lot about security. Yeah, pretend to be a binary you're not is usually the kind of thing that makes people sad when security politely kicks their door in.Waldemar: Yeah, we actually have pretty, like, big hopes for the v3 wave of the SDKs, AWS, because there is some restructuring happening with the endpoint resolution. And also, you can, in your profile, by now have, you know, special resolvers for endpoints. But still the case of just pointing all the SDKs and CLI to a custom endpoint is just not yet resolved. And this is, frankly, quite disappointing, actually.Corey: While we're complaining about the CLI, I'll throw one of my recurring issues with it in. I would love for it to adopt the Linux slash Unix paradigm of having a config.d directory that you can reference from within the primary config file, and then any file within that directory in the proper syntax winds up getting adopted into what becomes a giant composable config file, generated dynamically. The reason being is, I can have entire lists of profiles in separate files that I could then wind up dropping in and out on a client-by-client basis. So, I don't inadvertently expose who some of my clients are, in the event that winds up being part of the way that they have named their AWS accounts.That is one of those things I would love but it feels like it's not a common enough use case for there to be a whole lot of traction around it. And I guess some people would make a fair point if they were to say that the AWS CLI is the most widely deployed AWS open-source project, even though all it does is give money to AWS more efficiently.Waldemar: Yeah. Great point. Yeah, I think, like, how and some way to customize and, like, mingle or mangle your configurations in a more easy fashion would be super useful. I guess it might be a slippery slope to getting, you know, into something like I don't know, Helm for EKS and, like, really, you know, having to maintain a whole templating language for these configs. But certainly agree with you, to just you know, at least having [plug 00:15:18] points for being able to customize the behavior of the SDKs and CLIs would be extremely helpful and valuable.Corey: This is not—unfortunately—my first outing with the idea of trying to have AWS APIs done locally. In fact, almost a decade ago now, I did a build-out at a very large company of a… well, I would say that the build-out was not itself very large—it was about 300 nodes—that were all running Eucalyptus, which before it died on the vine, was imagined as a way of just emulating AWS APIs locally—done in Java, as I recall—and exposing local resources in ways that comported with how AWS did things. So, the idea being that you could write configuration to deploy any infrastructure you wanted in AWS, but also treat your local data center the same way. That idea unfortunately did not survive in the marketplace, which is kind of a shame, on some level. What was it that inspired you folks to wind up building this with an eye towards local development rather than run this as a private cloud in your data center instead?Waldemar: Yeah, very interesting. And I do also have some experience [unintelligible 00:16:29] from my past university days with Eucalyptus and OpenStack also, you know, running some workloads in an on-prem cluster. I think the main difference, first of all, these systems were extremely hard, notoriously hard to set up and maintain, right? So, lots of moving parts: you had your image server, your compute system, and then your messaging subsystems. Lots of moving parts, and wanting to have everything basically much more monolithic and in a single container.And Docker really sort of provides a great platform for us, which is create everything in a single container, spin up locally, make it very lightweight and easy to use. But I think really the first days of LocalStack, the idea was really, was actually with the use case of somebody from our team. Back then, I was working at Atlassian in the data engineering team and we had folks in the team were commuting to work on the train. And it was literally this use case that you mentioned before about being able to work basically offline on your commute. And this is kind of were the first lines of code were written and then kind of the idea evolves from there.We put it into the open-source, and then, kind of, it was growing over the years. But it really started as not having it as an on-prem, like, heavyweight server, but really as a lightweight system that you can easily—that is easily portable across different systems as well.Corey: That is a good question. Very often, when I'm using various tools that are aimed at development use cases, it is very clear that one particular operating system is invariably going to be the first-class citizen and everything else is a best effort. Ehh, it might work; it might not. Does LocalStack feel that way? And if so, what's the operating system that you want to be on?Waldemar: I would say we definitely work best on Mac OS and Linux. It also works really well on Windows, but I think given that some of our tooling in the ecosystem also pretty much geared towards Unix systems, I think those are the platforms it will work well with. Again, on the other hand, Docker is really a platform that helps us a lot being compatible across operating systems and also CPU architectures. We have a multi-arch build now for AMD and ARM64. So, I think in that sense, we're pretty broad in terms of the compatibility spectrum.Corey: I do not have any insight into how the experience goes on Windows, given that I don't use that operating system in anger for, wow, 15 years now, but I will say that it's been top-flight on Mac OS, which is what I spend most of my time. Depressed that I'm using, but for desktop experiences, it seems to work out fairly well. That said, having a focus on Windows seems like it would absolutely be a hard requirement, given that so many developer workstations in very large enterprises tend to skew very Windows-heavy. My hat is off to people who work with Linux and Linux-like systems in environments like that where even line endings becomes psychotically challenging. I don't envy them their problems. And I have nothing but respect for people who can power through it. I never had the patience.Waldemar: Yeah. Same here and definitely, I think everybody has their favorite operating system. For me, it's also been mostly Linux and Mac in the last couple of years. But certainly, we definitely want to be broad in terms of the adoption, and working with large enterprises often you have—you know, we want to fit into the existing landscape and environment that people work in. And we solve this by platform abstractions like Docker, for example, as I mentioned, and also, for example, Python, which is some more toolings within Python is also pretty nicely supported across platforms. But I do feel the same way as you, like, having been working with Windows for quite some time, especially for development purposes.Corey: What have you noticed that your customer usage patterns slash requests has been saying about AWS service adoption? I have to imagine that everyone cares whether you can mock S3 effectively. EC2, DynamoDB, probably. SQS, of course. But beyond the very small baseline level of offering, what have you seen surprising demand for, as I guess, customer implementation of more esoteric services continues to climb?Waldemar: Mm-hm. Yeah, so these days it's actually pretty [laugh] pretty insane the level of coverage we already have for different services, including some very exotic ones, like QLDB as I mentioned, Kafka. We even have Managed Airflow, for example. I mean, a lot of these services are essentially mostly, like, wrappers around the API. This is essentially also what AWS is doing, right? So, they're providing an API that basically provisions some underlying resources, some infrastructure.Some of the more interesting parts, I guess, we've seen is the data or big data ecosystem. So, things like Athena, Glue, we've invested quite a lot of time in, you know, making that available also in LocalStack so you can have your maybe CSV files or JSON files in an S3 bucket and you can query them from Athena with a SQL language, basically, right? And that makes it very—especially these big data-heavy jobs that are very heavyweight on AWS, you can iterate very quickly in LocalStack. So, this is where we're seeing a lot of adoption recently. And then also, obviously, things like, you know, Lambda and ECS, like, all the serverless and containerized applications, but I guess those are the more mainstream ones.Corey: I imagine you probably get your fair share of requests for things like CloudFormation or CloudFront, where, this is great, but can you go ahead and add a very lengthy sleep right here, just because it returns way too fast and we don't want people to get their hopes up when they use the real thing. On some level, it feels like exact replication of the AWS customer experience isn't quite in line with what makes sense from a developer productivity point of view.Waldemar: Yeah, that's a great point. And I'm sure that, like, a lot of code out there is probably littered with sleep statements that is just tailored to the specific timing in AWS. In fact, we recently opened an issue in the AWS Terraform provider repository to add a configuration option to configure the timings that Terraform is using for the resource deployment. So, just as an example, an S3 bucket creation takes 60 seconds, like, more than a minute against [unintelligible 00:22:37] AWS. I guess LocalStack, it's a second basically, right?And AWS Terraform provider has these, like, relatively slow cycles of checking whether the packet has already been created. And we want to get that configurable to actually reduce the time it takes for local development, right? So, we have an open, sort of, feature request, and we're probably going to contribute to a Terraform repository. But definitely, I share the sentiment that a lot of the tooling ecosystem is built and tailored and optimized towards the experience against the cloud, which often is just slow and, you know, that's what it is, right?Corey: One thing that I didn't expect, though, in hindsight, is blindingly obvious, is your support for a variety of different frameworks and deployment methodologies. I've found that it's relatively straightforward to get up and running with the CDK deploying to LocalStack, for instance. And in hindsight, of course; that's obvious. When you start out down that path, though it's well, you tend to think—at least I don't tend to think in that particular way. It's, “Well, yeah, it's just going to be a console-like experience, or I wind up doing CloudFormation or Terraform.” But yeah, that the world is advancing relatively quickly and it's nice to see that you are very comfortably keeping pace with that advancement.Waldemar: Yeah, true. And I guess for us, it's really, like, the level of abstraction is sort of increasing, so you know, once you have a solid foundation, with, you know, CloudFormation implementation, you can leverage a lot of tools that are sitting on top of it, CDK, serverless frameworks. So, CloudFormation is almost becoming, like, the assembly language of the AWS cloud, right, and if you have very solid support for that, a lot of, sort of, tools in the ecosystem will natively be supported on LocalStack. And then, you know, you have things like Terraform, and in the Terraform CDK, you know, some of these derived versions of Terraform which also are very straightforward because you just need to point, you know, the target endpoint to localhost and then the rest of the deployment loop just works out of the box, essentially.So, I guess for us, it's really mostly being able to focus on, like, the core emulation, making sure that we have very high parity with the real services. We spend a lot of time and effort into what we call parity testing and snapshot testing. We make sure that our API responses are identical and really the same as they are in AWS. And this really gives us, you know, a very strong confidence that a lot of tools in the ecosystem are working out-of-the-box against LocalStack as well.Corey: I would also like to point out that I'm also a proud LocalStack contributor at this point because at the start of this year, I noticed, ah, in one of the pages, the copyright year was still saying 2022 and not 2023. So, a single-character pull request? Oh, yes, I am on the board now because that is how you ingratiate yourself with an open-source project.Waldemar: Yeah. Eternal fame to you and kudos for your contribution. But, [laugh] you know, in all seriousness, we do have a quite an active community of contributors. We are an open-source first project; like, we were born in the open-source. We actually—maybe just touching upon this for a second, we use GitHub for our repository, we use a lot of automation around, you know, doing pull requests, and you know, service owners.We also participate in things like the Hacktoberfest, which we participated in last year to really encourage contributions from the community, and also host regular meetups with folks in the community to really make sure that there's an active ecosystem where people can contribute and make contributions like the one that you did with documentation and all that, but also, like, actual features, testing and you know, contributions of different levels. So really, kudos and shout out to the entire community out there.Corey: Do you feel that there's an inherent tension between being an open-source product as well as being a commercial product that is available for sale? I find that a lot of companies feel vaguely uncomfortable with the various trade-offs that they make going down that particular path, but I haven't seen anyone in the community upset with you folks, and it certainly hasn't seemed to act as a brake on your enterprise adoption, either.Waldemar: That is a very good point. So, we certainly are—so we're following an open-source-first model that we—you know, the core of the codebase is available in the community version. And then we have pro extensions, which are commercial and you basically, you know, setup—you sign up for a license. We are certainly having a lot of discussions on how to evolve this licensing model going forward, you know, which part to feed back into the community version of LocalStack. And it's certainly an ongoing evolving model as well, but certainly, so far, the support from the community has been great.And we definitely focus to, kind of, get a lot of the innovation that we're doing back into our open-source repo and make sure that it's, like, really not only open-source but also open contribution for folks to contribute their contributions. We also integrate with other third-party libraries. We're built on the shoulders of giants, if I may say so, other open-source projects that are doing great work with emulators. To name just a few, it's like, [unintelligible 00:27:33] which is a great project that we sort of use and depend upon. We have certain mocks and emulations, for Kinesis, for example, Kinesis mock and a bunch of other tools that we've been leveraging over the years, which are really great community efforts out there. And it's great to see such an active community that's really making this vision possible have a truly local emulated clouds that gives the best experience to developers out there.Corey: So, as of, well, now, when people are listening to this and the episode gets released, v2 of LocalStack is coming out. What are the big differences between LocalStack and now LocalStack 2: Electric Boogaloo, or whatever it is you're calling the release?Waldemar: Right. So, we're super excited to release our v2 version of LocalStack. Planned release date is end of March 2023, so hopefully, we will make that timeline. We did release our first version of OpenStack in July 2022, so it's been roughly seven months since then and we try to have a cadence of roughly six to nine months for the major releases. And what you can expect is we've invested a lot of time and effort in last couple of months and in last year to really make it a very rock-solid experience with enhancements in the current services, a lot of performance optimizations, we've invested a lot in parity testing.So, as I mentioned before, parity is really important for us to make sure that we have a high coverage of the different services and how they behave the same way as AWS. And we're also putting out an enhanced version and a completely polished version of our Cloud Pods experience. So, Cloud Pods is a state management mechanism in LocalStack. So, by default, the state in LocalStack is ephemeral, so when you restart the instance, you basically have a fresh state. But with Cloud Pods, we enable our users to take persistent snapshot of the states, save it to disk or to a server and easily share it with team members.And we have very polished experience with Community Cloud Pods that makes it very easy to share the state among team members and with the community. So, those are just some of the highlights of things that we're going to be putting out in the tool. And we're super excited to have it done by, you know, end of March. So, stay tuned for the v2 release.Corey: I am looking forward to seeing how the experience shifts and evolves. I really want to thank you for taking time out of your day to wind up basically humoring me and effectively re-covering ground that you and I covered about a year and a half ago now. If people want to learn more, where should they go?Waldemar: Yeah. So definitely, our Slack channel is a great way to get in touch with the community, also with the LocalStack team, if you have any technical questions. So, you can find it on our website, I think it's slack.localstack.cloud.We also host a Discourse forum. It's discuss.localstack.cloud, where you can just, you know, make feature requests and participate in the general conversation.And we do host monthly community meetups. Those are also available on our website. If you sign up, for example, for a newsletter, you will be notified where we have, you know, these webinars. Take about an hour or so where we often have guest speakers from different companies, people who are using, you know, cloud development, local cloud development, and just sharing the experiences of how the space is evolving. And we're always super happy to accept contributions from the community in these meetups as well. And last but not least, our GitHub repository is a great way to file any issues you may have, feature requests, and just getting involved with the project itself.Corey: And we will, of course, put links to that in the [show notes 00:31:09]. Thank you so much for taking the time to speak with me today. I appreciate it.Waldemar: Thank you so much, Corey. It's been a pleasure. Thanks for having me.Corey: Waldemar Hummer, CTO and co-founder at LocalStack. I'm Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with an angry comment, presumably because your compensation structure requires people to spend ever-increasing amounts of money on AWS services.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.
An airhacks.fm conversation with Dave Johnson (@snoopdave) about: PDP-8 with a paper tape reader, airhacks.tv questions and answers, TRS-80, playing asteroids, asteroids, Defender and Battlezone were based on vector graphics, learning Pascal and C, Data General Eclipse MV/8000, Geographic Resources Analysis Support System (GRASS GIS), working for University of Kingston, working on jfactory for Rouge Wave, HAHT Software, The Soul of a New Machine, distributed Visual Basic application server, using xdoclet to generate EJB, using castor for persistence, Apache Roller started as sample application, Sun hires dave, working on Lotus Notes social, starting at wayin, Roller supports Pingback, Lotus is using roller, using Rightscale to deploy Java software to AWS, using Jenkins and CloudFormation, episode with Scott McNealy "#19 SUN, JavaSoft, Java, Oracle", Roller uses Apache Velocity, working on RSS parser Rome, switching from MongoDB to Apache Cassandra, UserGrid data store, Oracle acquires apiary , starting at CloudBees, episode with Kohsuke Kawaguchi "#143 How Hudson and Jenkins happened", starting at Apollo, several thousand blogs on roller Dave Johnson on twitter: @snoopdave
On this episode of Remote Ruby, we have an awesome guest joining us. Today, we have Ken Collins, who's a Principal Engineer and Cloud Architect at Custom Ink, an active member in the Ruby community for over fifteen years, a Microsoft open source contributor, PC Gamer, and an AWS Serverless Hero. We have so much to discuss today, as Ken fills us in on Lamby, Custom Ink, how Lambda evolved, a gem called Lambdakiq, and if you're looking for cost optimization, why Lambda is the best compute service out there. We'll also learn how CloudFormation can help developers, how CloudWatch Events is used, and we'll hear about the different database options Amazon has such as Aurora Serverless, DynamoDB, and RDS. If you've never used Lambda, it's a good time to try it out. Andrew realized he's in the perfect place to try it since he recently built a proxy one. Download this episode to learn much more! [00:01:52] Ken tells us about himself and his background[00:04:47] Custom Ink makes some great products, and we'll learn how Lamby came to be, the stuff they build, the cool tech behind it, and the services, such as AWS Lambda.[00:08:16] How did Lambda evolve?[00:09:17] Ken details what the OCI format is, and how Lambda works compared to deploying to a traditional server. We hear about Lambda releasing Function URLs, a free API gateway, and what it does.[00:12:16] We hear the whole process from end-to-end, starting from a web request, what happens, how it gets to Rails, Dynos are running, the database gets affected, and how those containers can be used for other things like an event driven architectures.[00:16:03] Chris asks Ken how Kubernetes and Lambda compare. Also, we hear how background jobs and cron jobs fit in, and a gem that Ken wrote called, Lambdakiq.[00:20:30] How does Ken manage connections being made and the events being sent to the right place? Also, Chris wonders if CloudFormation is something you should learn as one of the starting points or you should later for it to be more useful, and Ken tells us about the AWS Cloud Development Kit and what it does.[00:24:10] Amazon has many different database options and Ken explains that you can use any database you want, wherever you want.[00:25:39] Ken explains the differences between Aurora Serverless, DynamoDB, and RDS. [00:30:23] We're going back to talking about Lambda now and Ken tells us about their website, a documentation website where they cover things, and a Quick Start Guide on how you can deploy a new Rails APP on Rails 3.2 to Lambda in 5 minutes.[00:33:02] Chris mentions how Taylor Otwell modified Laravel to run on Lambda, and Vapor is their tool for deploying to Lambda.[00:36:25] Are there any gotchas? Chris heard people were talking about Rails being slow to boot and issues with connecting to your Lambda to a VPC was slow. Ken tells us the VPC has been solved very well.[00:39:31] Ken and Chris chat about the hardest things are learning and change management, like setting up CI for the first time can be challenging, Heroku is amazing but has its limits, and using CloudWatch Logs which is a change for people. Also, Ken shares a hotspot with Lambda, and he tells us about Lambda Punch and New Relic. [00:42:47] Ken tells us to use CloudWatch Events for setting up Cronjobs that run on a schedule.[00:44:51] Chris wonders if there are concerns or ways you have to change things for assets, and Ken explains what they do with turning on the magic environment variable, but if you need something else, it goes into the CI/CD Pipeline creation.[00:48:30] Andrew is going to try Lambda now, and we hear Ken's thoughts on how different development is from production when you use Lambda. Find out why he loves Microsoft's Development Containers Specification, and Chris mentions DHH's MRSK project and what it's going to do.[00:56:06] Find out where to follow Ken, if you're interested in Custom Ink, check them out, and please try out Lambda because he could use some contributors to help write the guides.Panelists:Jason CharnesChris OliverAndrew MasonGuest:Ken CollinsSponsor:HoneybadgerLinks:Jason Charnes TwitterChris Oliver TwitterAndrew Mason TwitterKen Collins TwitterKen Collins GitHubKen Collins (Dev.to)Lamby-GitHubCustom InkCustom Ink ProductsLambdakiqAmazon Aurora ServerlessAmazon DynamoDBAmazon RDSLambyFull Stack Radio Podcast-Episode 120-Taylor Otwell-Serverless Laravel with VaporLambda PunchNew Relic-GitHubAmazon CloudWatch EventsDevelopment ContainersRemote Ruby Podcast-Episode 165: GitHub Codespaces & Docker with Benjamin WoodMRSK: Deploy Web apps anywhereRuby Radar TwitterRuby for All Podcast
Summary Data is a team sport, but it's often difficult for everyone on the team to participate. For a long time the mantra of data tools has been "by developers, for developers", which automatically excludes a large portion of the business members who play a crucial role in the success of any data project. Quilt Data was created as an answer to make it easier for everyone to contribute to the data being used by an organization and collaborate on its application. In this episode Aneesh Karve shares the journey that Quilt has taken to provide an approachable interface for working with versioned data in S3 that empowers everyone to collaborate. Announcements Hello and welcome to the Data Engineering Podcast, the show about modern data management Truly leveraging and benefiting from streaming data is hard - the data stack is costly, difficult to use and still has limitations. Materialize breaks down those barriers with a true cloud-native streaming database - not simply a database that connects to streaming systems. With a PostgreSQL-compatible interface, you can now work with real-time data using ANSI SQL including the ability to perform multi-way complex joins, which support stream-to-stream, stream-to-table, table-to-table, and more, all in standard SQL. Go to dataengineeringpodcast.com/materialize (https://www.dataengineeringpodcast.com/materialize) today and sign up for early access to get started. If you like what you see and want to help make it better, they're hiring (https://materialize.com/careers/) across all functions! Your host is Tobias Macey and today I'm interviewing Aneesh Karve about how Quilt Data helps you bring order to your chaotic data in S3 with transactional versioning and data discovery built in Interview Introduction How did you get involved in the area of data management? Can you describe what Quilt is and the story behind it? How have the goals and features of the Quilt platform changed since I spoke with Kevin in June of 2018? What are the main problems that users are trying to solve when they find Quilt? What are some of the alternative approaches/products that they are coming from? How does Quilt compare with options such as LakeFS, Unstruk, Pachyderm, etc.? Can you describe how Quilt is implemented? What are the types of tools and systems that Quilt gets integrated with? How do you manage the tension between supporting the lowest common denominator, while providing options for more advanced capabilities? What is a typical workflow for a team that is using Quilt to manage their data? What are the most interesting, innovative, or unexpected ways that you have seen Quilt used? What are the most interesting, unexpected, or challenging lessons that you have learned while working on Quilt? When is Quilt the wrong choice? What do you have planned for the future of Quilt? Contact Info LinkedIn (https://www.linkedin.com/in/aneeshkarve/) @akarve (https://twitter.com/akarve) on Twitter Parting Question From your perspective, what is the biggest gap in the tooling or technology for data management today? Closing Announcements Thank you for listening! Don't forget to check out our other shows. Podcast.__init__ (https://www.pythonpodcast.com) covers the Python language, its community, and the innovative ways it is being used. The Machine Learning Podcast (https://www.themachinelearningpodcast.com) helps you go from idea to production with machine learning. Visit the site (https://www.dataengineeringpodcast.com) to subscribe to the show, sign up for the mailing list, and read the show notes. If you've learned something or tried out a project from the show then tell us about it! Email hosts@dataengineeringpodcast.com (mailto:hosts@dataengineeringpodcast.com)) with your story. To help other people find the show please leave a review on Apple Podcasts (https://podcasts.apple.com/us/podcast/data-engineering-podcast/id1193040557) and tell your friends and co-workers Links Quilt Data (https://quiltdata.com/) Podcast Episode (https://www.dataengineeringpodcast.com/quilt-data-with-kevin-moore-episode-37/) UW Madison (https://www.wisc.edu/) Docker Swarm (https://docs.docker.com/engine/swarm/) Kaggle (https://www.kaggle.com/) open.quiltdata.com (https://open.quiltdata.com/) FinOS Perspective (https://perspective.finos.org/) LakeFS (https://lakefs.io/) Podcast Episode (https://www.dataengineeringpodcast.com/lakefs-data-lake-versioning-episode-157/) Pachyderm (https://www.pachyderm.com/) Podcast Episode (https://www.dataengineeringpodcast.com/pachyderm-data-lineage-episode-82) Unstruk (https://www.unstruk.com/) Podcast Episode (https://www.dataengineeringpodcast.com/unstruk-unstructured-data-warehouse-episode-196/) Parquet (https://parquet.apache.org/) Avro (https://avro.apache.org/) ORC (https://orc.apache.org/) Cloudformation (https://aws.amazon.com/cloudformation/) Troposphere (https://github.com/cloudtools/troposphere) CDK == Cloud Development Kit (https://aws.amazon.com/cdk/) Shadow IT (https://en.wikipedia.org/wiki/Shadow_IT) Podcast Episode (https://www.dataengineeringpodcast.com/shadow-it-data-analytics-episode-121) Delta Lake (https://delta.io/) Podcast Episode (https://www.dataengineeringpodcast.com/delta-lake-data-lake-episode-85/) Apache Iceberg (https://iceberg.apache.org/) Podcast Episode (https://www.dataengineeringpodcast.com/iceberg-with-ryan-blue-episode-52/) Datasette (https://datasette.io/) Frictionless (https://frictionlessdata.io/) DVC (https://dvc.org/) Podcast.__init__ Episode (https://www.pythonpodcast.com/data-version-control-episode-206/) The intro and outro music is from The Hug (http://freemusicarchive.org/music/The_Freak_Fandango_Orchestra/Love_death_and_a_drunken_monkey/04_-_The_Hug) by The Freak Fandango Orchestra (http://freemusicarchive.org/music/The_Freak_Fandango_Orchestra/) / CC BY-SA (http://creativecommons.org/licenses/by-sa/3.0/)
About AlyssAlyss Noland is the head of Developer Relations Relations and Product Marketing at Common Room, an intelligent community-led growth platform. She previously led product marketing for Developer Experience at GitHub where she focused on open source community investment and helping engineering teams find success through development metrics and developer-focused research. She's been working in tech since 2012 in various roles from Sales Engineering and Developer Advocacy to Product Marketing with companies such as GitHub, Box, Atlassian, and BigCommerce, as well as being an advisor at Heavybit. Links Referenced: Common Room: https://www.commonroom.io/ Heavybit: https://www.heavybit.com/ Twitter: https://twitter.com/PreciselyAlyss Twitch: https://www.twitch.tv/PreciselyAlyss TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Tailscale SSH is a new, and arguably better way to SSH. Once you've enabled Tailscale SSH on your server and user devices, Tailscale takes care of the rest. So you don't need to manage, rotate, or distribute new SSH keys every time someone on your team leaves. Pretty cool, right? Tailscale gives each device in your network a node key to connect to your VPN, and uses that same key for SSH authorization and encryption. So basically you're SSHing the same way that you're managing your network.So what's the benefit? You'll get built-in key rotation, the ability to manage permissions as code, connectivity between two devices, and reduced latency. You can even ask users to re-authenticate SSH connections for that extra bit of security. Try Tailscale now - it's free forever for personal use forever.Corey: This episode is sponsored by our friends at Logicworks. Getting to the cloud is challenging enough for many places, especially maintaining security, resiliency, cost control, agility, etc, etc, etc. Things break, configurations drift, technology advances, and organizations, frankly, need to evolve. How can you get to the cloud faster and ensure you have the right team in place to maintain success over time? Day 2 matters. Work with a partner who gets it - Logicworks combines the cloud expertise and platform automation to customize solutions to meet your unique requirements. Get started by chatting with a cloud specialist today at snark.cloud/logicworks. That's snark.cloud/logicworksCorey: Welcome to Screaming in the Cloud. I'm Corey Quinn. I often wonder how to start these conversations, but sometimes it's just handed to me and I don't even have to do a whole lot of work. My guest today is Alyss Noland, who's the Head of Developer Relations Relations and Product Marketing at Common Room. Alyss, thank you for joining me.Alyss: Thanks for having me, Corey. I'm really excited to be here.Corey: So, developer relations relations. It feels like an abstraction that has been forced to be built on top of another abstraction that has gotten too complicated, so as best I can tell, you are walking around as a human equivalent of Kubernetes.Alyss: Oh, gosh, I would really hope not to be a human equivalent of Kubernetes. I think that would make me an octopus. But—Corey: Yeah, “What did you say about me?” Yeah.Alyss: [laugh].Corey: “I didn't come here to be insulted, Quinn.” Yeah.Alyss: No, like listen, I love octopodes. Which [tattoo 00:01:24] is which? So, developer relations relations. Yes, it's an abstraction on an abstraction. A really critical level, it is how do I relate? Can I relate to people that are in the developer relations profession at large?We are at the point at which this is a somewhat poorly-defined area that is continuing to grow. And there's a lot of debates in that space and so I'm really excited to be at an organization that will give me a platform to try and move the industry forward.Corey: Your relatively recent career history is honestly fascinating to me. You spent about a year and a half as a senior developer advocate at Box. And as anyone who's ever tried it knows, it's very hard to beat Box [beatboxing noises]. But you tried and went to GitHub, in which case, you basically transitioned pretty quickly from a Senior Product Marketing Manager to Director of Product Marketing, where you were the go-to-market lead for GitHub Copilot.Alyss: Yeah, that was a really interesting project to be on. I started off at the technical preview back in 2021, launching that too—it ended up being with about a little over a million, two million folks in technical preview. And it's fairly new to the market. There was nothing else—or at the time, there had been nothing else that was using a descendant of GPT-3. There was nothing else using a descendant of GPT-3 to generate suggestions for code to—there were a couple that were using GPT-2, but the amount of language coverage they had was a little bit limited, what they were suggesting was a little bit limited.And it's hard to say, like, highlight of my career, but at that point in time, I would say probably, highlight of my career to be able to work on something with that opportunity for impact.Corey: As someone who was in the technical preview and now tried to be a paying customer of it, but I can't because of my open-source work, it wound up giving it to me for free. I found it to be absolutely transformative. And I know I'm going to get letters and I don't even slightly care because it's not, “I'm going to tab-complete my application.” If a tool can do that, your application is probably not that complex. No, for me, what I find incredibly valuable is the ability to tab-complete through obnoxious boilerplate. CloudFormation, I am not subtweeting you; I am calling you out directly. You are wordy and obnoxious. Fix yourself.And especially in languages that I don't deal with day-to-day—because I'm not a full-time developer—I forget certain parameters or argument order or things like that and being able to effectively tab-complete is awesome for that use case. It's not doing my job; it's automating the crappy part of my job. And I absolutely love it for that.Alyss: Yeah, and was really interesting working on a common portion of product marketing work is that we build messaging houses. We try to identify where's the value to the user, to the organization at large, depending on, like, who it is we're trying to sell to, how does that ladder up from, like, an IoT to a manager. And so, one of the things that I got really excited about as we started to see it—and there's some great work that Dr. Eirini Kallaimvakou has published that I would definitely refer to if you're interested in diving deeper into it—is the way in which Copilot and this, like, ability to improve the boilerplate experience, improve the boring shit—automate the boring shit, if you will—is about developer satisfaction. It's not about making you build your commits faster or about having more lines of code that you like get deployed out; it's about making your jobs suck less.Corey: Well, if you spent, what was it roughly two years, give or take, at GitHub between your various roles—and yes, I'm going to pronounce it ‘GIF-ub' because that's my brand of obnoxious, so I'm going to go for it—you went to Common Room. Let's begin there. What does Common Room do, exactly?Alyss: So, Common Room is an intelligent community-led growth platform. And there's a few things kind of packed into that really short description, but the idea is that we've seen all of these product-lead grows businesses. But at a critical point, and something we've seen at GitHub, which is a product-led growth company, it's something that we've seen at Atlassian, Asana, you name half a dozen different, like, SaaS companies, self-hosted software, open-source, community is at the heart of it. And so, how do you nurture that community? How do you measure that community? How do you prove that the work that you're doing is valuable?And that's what Common Room is setting out to do. And so, when I saw—like, they're not the only person or organization in the market that's doing this, but I think they're doing it exceptionally well, and with really great goals in mind. And so, I'm enthused to try and facilitate that investment in community for more organizations.Corey: One of the challenges that I have seen of products in the community space is it tended, historically, to go in really, I guess I'll call them uncomfortable directions. In the before times, I used to host dinner parties near constantly here, and someone confide into me once—after, you know, six beers or so, because that's when people get the excitingly honest—they mentioned that, “Yeah, I'm supposed to wind up putting these dinners into Salesforce”—or whatever the hell it was—“To track the contacts we have with influencers in this space.” And that made me feel so profoundly uncomfortable. It's, you're invited here to spend time with my friends and my family. You're meeting my kids, it's, yeah, this is just a go-to-market motion and you can [BLEEP] on out of here and never come back.And I did not get that sense to be clear and I'm told the company wound up canceling that horrifying program, but it does feel like it's very easy to turn an authentic relationship into something that feels remarkably sleazy. That said, Common Room has been around for a while and I have yet to hear a single accusation that you folks have come within a thousand miles of doing that. How do you avoid the trap?Alyss: It's a slippery slope, and I can't say that Common Room creates any kind of like enforcement or silos or prevents organizations from falling into this trap. Fundamentally, the way in which community can be abused, the way in which these relationships can be taken advantage of, at least from the perception of the parties that initially built the relationship, is to take the context out of them, to take the empathy out of them, take the people out of them. And so, that is fundamentally left to the organization's principles, it's left to how much authority does community have within the business relative to a sales team. And so first, being able to elevate community in such a way to show that they are having that impact already without having to turn the community into a prospect pool is, I think, one of the critical first steps, and it's something that we've been able to break through initially by connecting things like Slack, Discord, Twitter to show, here's all these people talking about you, here's all the things that they're saying, here's the sentiment analysis, and also, now we're going to push that into Salesforce. So, you can see that this started out in community and it was fostered there. Now, you can see the ROI, you don't need to go hitting up our community contacts to try and sell to them because we're doing it on your behalf in a very real way.Corey: Part of the challenge, I think, is that—and you've talked to me about this in previous conversations we've had—that so much of community is distilled down to a sales motion, which let's be direct, it kind of sucks at, in some levels, because it's okay, great, I'm here to talk to you about how community works. Well, in the AWS community, for example, the reason that formed and is as broad and fast as it is because AWS's documentation is Byzantine and there's a sort of shared suffering that we all get to commiserate over. And whenever AWS tries to take, “Ownership,” quote-unquote, of its community, right, that doesn't actually work that way. They have community watering holes, but to my understanding, the largest AWS-centric Slack team is the Open Guide to AWS's Slack team, which now has, at last count, 15,000 people in it. I'm lucky enough to be the community lead for that project.But it was pre-existing before I got there and it's great to be able to go and talk to people who are using these things. It doesn't feel like it is owned, run, or controlled—because it's not—by AWS themselves. It's clear from the way that your product has evolved, that you feel similarly around that where it's about being aware of the community rather than controlling the community. And that's important.Alyss: Absolutely. And one of the ways in which we, like, highlight this as soon as you're in the product, is being able to show community responsiveness and then what percentage of those responses are coming from my team members. And frankly, as someone who's previously set strategy for developer relations teams, for developer communities, what I want to see is community members responding to each other, community members knowing what's the right place to look, what's the right answer, how am I ensuring that they have the resources that they need, the answers that they need. Because at the end of the day, I can't scale one-to-one; no one can. And so, the community being able to support itself is at the heart of the definition of community.Corey: One of the other problems that I've seen historically, and I'll call it the Chef problem because Chef had an incredibly strong community, and as someone who is deep in the configuration management space myself, but never use Chef, it was the one that I avoided for a variety of reasons at the time, it was phenomenal. I wound up going to ChefConf, despite not being a Chef user, just to spend time with some of the great people that were involved. The blunder that they made before they were acquired into irrelevance by progress—and to be fair, the industry changed direction toward immutable infrastructure in ways that were hard to foresee—but the problem is, they made was hiring their entire community. And it doesn't sound like that would be a bad thing, but suddenly, everyone who was talking about the product had a Chef email address, and that hits very differently.Alyss: It does. And it goes back to that point of trying to maintain those authentic relationships. And if we're to step outside of tech, I have a background prior to tech in the video game industry, and that was a similar problem. Nearly every single community-made application, extension ends up getting acquired by some organization, like Curse, and then piped full of ads, or the person that you thought you could ask or to see build some other better experience of version control software, or a Git client ends up getting consumed into a large business and then the project never sees the light of day. And frankly, that's not how you run community in my estimation.My estimation is, if the community is doing things better than you are, take notes. Product management, pay attention. That's something that is another aspect of doing developer relations is about checking in with those teams, about showing them evidence. And like, it so often ends up being qualitative in a way that doesn't change people's minds or their feelings, where people want to see quantitative numbers in order to say, “Oh, this is the business justification. Like, this is the ROI. This proves that this is the thing we should invest in.” And frankly, no. Like, sometimes it is a little bit more about stepping back and letting the organic empathy and participation happen without having to own it.Corey: There's a sense, I think that a lot of companies feel the need to own every conversation that happens around them, their product, et cetera, and you can't. You just can't, unless—to be direct—your company is failing. Just because if no one's talking about you, then great, you're the only ones talking about you. And you can see this from time to time and it's depressing as hell when you have people who work for a company all tweeting the same cookie-cutter statement, and they get zero interaction except from a bot account. It's sad.Alyss: Yeah. And I've unfortunately seen this more times than I can count in community Slacks where people just, like, copy-paste whatever marketing handed to them, and I would be shocked if they got any engagement at all. Because that's… cool. What do I know about you? Why do I care about this event? Have you personalized it to me?And yeah, you don't want the organization to be the only one talking about you. If you are then you've already failed in this, you know, product-led growth motion. You've kind of—if we want to get into the murky water of NPS, like, nobody's going and telling their friends about your product [laugh]. And the thing that's so valuable is the authentic voice. It's the, “I'm excited to talk about this and I like it enough to tell you what I like about it.” I like it enough to tell you about this use case that might never seen the light of day, but because we're having a conversation between ourselves, it can all be personalized. It can all be about what's going on between us and about our shared experiences. And that is ten times more powerful than most Twitter-promoted ads you'll ever see.Corey: So, I want to unpack a little bit about not developer relations as such, but developer relations relations because I can mostly understand—badly—what product marketing is, but developer relations relations—or as you'd like to call it developer relations squared—that's something new. I've always called DevRel to be devrelopers, and people get annoyed enough at that. What is that newfound layer of abstraction on top of it?Alyss: Well, there's several things that I'm going to end up—and I say end up; I'm six weeks into the role, so I have a lot of high hopes for where I hope this goes. And one of those is things, like, we don't have a very shared understanding and shared definition of what developer advocacy even is, what is developer relations? Does developer marketing belong under that umbrella? How should organizations approach developer relations? How should they value it? Where should it, you know, belong in terms of business strategy?And there's an opportunity for a company whose business it is to elevate this industry, this career path, if you will, where we can spend the time, we can spend the money to say, here's what success looks like. We've interviewed all these groups, we've talked with the leaders in this space that are making it their jobs to think about this. Here's a set of group-developed recommendations for how the industry should mature. Or here's an open-source set of job descriptions and requirements. And like, let's get to some level of shared understanding.So, as an example of, kind of, where I'm leading to with all of this, and some of the challenges that developer relations faces is the State of Developer Relations report that just came out. There's a significant number of people that are coming into developer advocate, developer relations roles for the first time, they have one to two years of experience, they're coming into programs that have been around for one to two years, and so what does that tell you? That tells you you're bringing in people with no experience to try to establish brand new programs, that they're being asked to by their business, and they don't have the vocabulary, the tools, the frameworks in which to establish that for themselves. And so, they're going to be swayed by, you know, the tides of business, by the influences of their leadership without having their own pre-built notions. And so, how do we give them that equipment and how do we elevate the practice?Corey: Cloud native just means you've got more components or microservices than anyone (even a mythical 10x engineer) can keep track of. With OpsLevel, you can build a catalog in minutes and forget needing that mythical 10x engineer. Now, you'll have a 10x service catalog to accompany your 10x service count. Visit OpsLevel.com to learn how easy it is to build and manage your service catalog. Connect to your git provider and you're off to the races with service import, repo ownership, tech docs, and more. Corey: It feels like so much of the DevRel discourse has turned into, one, we define it by what is not, and two, it doesn't matter how you're measuring it, you're measuring it wrong. I feel like that is, I guess we'll call it counterproductive, for lack of a better descriptor. It feels like there's such a short-sighted perspective on all of this, but at the same time, you've absolutely got to find ways to articulate the value of DevRel slash community to the business otherwise, it turns into a really uncomfortable moment when, okay, time to cut costs. Why should we keep your function over a different function? If there's not a revenue or upside or time to market or some form of value story tied to that, that the business can understand that isn't just touchy-feely, it's a very difficult path forward from there. How do you see it?Alyss: I agree with you and I've, frankly, run into this problem several times in my career, and every time I've been a developer advocate. It's, you know—and where I've found the most success is not in saying, “Here's exactly the numbers that I'm going to be constantly looking at. I'm going to try to produce this many pieces of content, or I'm absolutely not speaking at events. And that's not my job. Or I'm not writing code. That's not my job.”It's about understanding what is driving the business forward. Who do I need participation and buy-in from and where am I hoping to go? Like, what does a year out from this look like? What does three years out from this look like? At Box, we do not want to be the API governance standard. That is not our job. That's not where we sit within engineering.That's frankly, if you really want to get into it, internal developer advocacy because it can influence the impact on the community. It is not the core focus and there are probably people better equipped and better educated on the core application. Big commerce, platform ecosystem, platform flywheel developers are fundamentally a part of continuing to grow the business and how do I go make that point to sales, how do I go make that point to partners, how do I go make that point to customer success, so that I can build a function that has more than one person. And so, I think to kind of bring it back to the larger question, that is where I see our greatest challenge is that we haven't given ourselves the vocabulary or the framework to understand the level of complexity that DevRel has become in being across so many industries, and being in B2B, and being in business to developer, and being in business to consumer. No one size fits all and we need to stop trying to treat it as though it can be.Corey: I think that there is a, how to put it, a problem in terms of how Twitter views a lot of these things. Someone wound up finally distilling it down for me in relatively recent times with a very resonant quote, which was simply put, that Twitter is not where you go for nuance. Twitter is where you go to be righteous. And I realized, oh, my God, that describes a good 80% of the things I've put up there. Like when I talk about how when companies do this thing to their staff and it's crappy, I am not necessarily for a nuanced debate, although of course there's always nuance and edge cases in the rest.As a counterpoint, whenever I wind up talking about things on Twitter and speak in generalities, I get a whole bunch of people pushing back with a, “Well, what about this edge case? That renders your entire point invalid.” And, ugh, not really. It feels like one of the casualties of the pandemic has been a sense of community in a sense of humans relating to other humans. I think we're all tired of the Zoom calls from hell I got to see you a couple of weeks before this recording at Monktoberfest in Portland, Maine, and oh, my God, dealing with people face to face, it was so much richer, at least from my perspective, compared to everything that we've been able to do during the pandemic. Am I alone on that? Are you seeing this across the board? Where companies are talking about this?Alyss: I will say with confidence, you're not alone in this. Whether or not companies are talking about it is also across the board. How rich are those understandings? How rich are those conversations? Because trying to step back as a brand is not really a way.Like, having nuance, being real, been community members, like that's not a way in which I think companies can participate in a way that feels truly authentic. That's why you need faces. That's why you need people. That's why you need folks whose job it is to do this. But in terms of things are lost, like, Twitter is not the right place to be having these conversations. It's not the right place in which to necessarily relate to people, absolutely.When you get distilled down all of your interactions into oh, I've got a notification. Oh, I have a checkmark, and so I have, like, better moderation tools. Oh, like, I made a statement and I don't want to hear a solution for it. We get all of these, uncurated experiences that are so dissatisfying that it does make us miss being around people who can read body language, that can understand my immediate relationship to them in spaces that we choose to be in, whereas Twitter is this big panopticon where we can just get yelled at and yell at each other. And it loves to amplify those conversations far more than any of the touchy-feely, good news success stories.Corey: When you take a look across the entire landscape of managing DevRel programs and ensuring that companies are receiving value for it, and—by which I mean, nurturing the long-term health of communities because yes, I am much more interested in that than I am in next quarter's numbers, how do you see that evolving, particularly with the recent economic recession or correction or drawback or everything's on fire, depending upon who it is you talk to? How do you see that evolving?Alyss: It goes back to what I said earlier about, I can speak in generalities, there will be specifics to various organizations, but at a fundamental part, like, I'll kind of take a step back and maybe make some very strong statements about what I think DevRel is, in a regard, which is, without documentation, without support, you don't have a product. And if you don't have folks going out and understanding what it is your customers need, and especially when those customers are maybe all the time or sometimes developers, and understanding what it is that they're saying and truly how having empathy for what's going on in their day-to-day, what task are they trying to complete, how relevant is this to them, if you don't invest in that, when that happens, you've lost the plot. And so, in those instances, unfortunately, that's a conversation with leadership team. Your leadership doesn't fundamentally understand the value and maybe it's worth it to make the argument in favor of to illustrate that without this feedback loop, without this investment in the educational journey of developers, without the investment in what is going on in our product, and where have we allowed ourselves to remain ignorant of what is happening in the day-to-day of our users. We need those folks.Product managers are in sprints, they're in standups. They're doing, like, strategic planning and their yearly planning. We need a group who is rewarded to care about this but also is innately driven to do so as well. And that's not something that you can make. And it's not something that we otherwise see. It's part of why we have such an absence in good developer marketing is because marketers aren't paid well enough to ever have learned the skills to be developers, and so there's no skills transfer.Corey: One last topic that I want to get into something you've only been doing for a short while, but you've become an advisor at Heavybit, which is a VC firm. How did that come about and what do you do?Alyss: So currently, I—I'll do the super-high level. What I do right now is I host office hours with seed startups and Series A that are in the dev tool space. And we generally talk about developer relations, a little bit in developer marketing go-to-market strategies. And it's super enriching for me because I love hearing about different experiences and problems and, like, areas of practice. But it was really interesting, and a little bit of a make-your-own-luck-and-opportunity type deal.Where I live in Austin, Texas; I do not live in the Bay Area, I don't have all those connections, I've been a bit distant from it. And I saw someone who had accepted a role that I had interviewed for, end up in some of their content. And I was like, “They're doing a great job. They definitely deserve to be there, but I also had similar qualifications, so why should I also be there?” And I found someone, his name's Tim, on LinkedIn, who runs their events. And I reached out and I said, “Hey, Tim, how would you like a new advisor?” And so, Tim responded back and we—Corey: Knock knock. Who's there? It's me.Alyss: Yeah, exactly. It's—and it was just, I want this thing to happen. How do I make it happen? I ask.Corey: And what does it day-to-day that look like? How much time does it take? What do you do exactly?Alyss: Yeah. I mean, right now, it's about five hours every quarter. So, I spend anywhere between 30 minutes to an hour with various organizations that are a part of Heavybit's portfolio, talking with them through their motion to go general availability, or they want to start participating in events, or they want to discover what are the right events for them to—or, like, DevOpsDays, should we participate in that? Should we hire a DevRel person? Should we hire a product marketing person? Just helping them sort wheat from chaff in terms of, like, how to proceed.And so, it's relatively, for me, lightweight. And Heavybit also gives us the opportunity to contribute back in blog posts, participate in podcasts and be able to have some of those richer conversations. So, I have a set of bookmarks, so there's over 100, bookmarks long, that is fully curated across several different categories. That was my first blog post was diving into a few of those where I think are critical areas of developer relations. What are some of the conversations on DevRel metrics? How do I think about setting a DevRel strategy for the first time? How do I do my first DevRel hire? And so, I wouldn't even call it a second job. It's more of a getting to, again, enrich my own experience, see a wider variety of different problems in this space and expand my own understanding.Corey: I really want to thank you for being so generous with your time. If people want to learn more about what you're up to, how you view the world, and basically just come along for the ride as you continue to demonstrate a side of tech that I don't think we get to see very often, where can they find you?Alyss: I am@PreciselyAlyss on Twitter, as well as Twitch. Aside from that, I would not recommend looking for me.Corey: Excellent. Always a good decision. I will put links to that in the [show notes 00:30:00]. Thank you so much for your time. I appreciate it.Alyss: Thanks, Corey.Corey: Alyss Noland, Head of Developer Relations Relations and Product Marketing at Common Room. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with an angry, insulting comment belittling community and letting the rest of us know by observation just why you've been thrown out of every community to which you've ever been a part.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.
The Cloud Pod recaps all of the positives and negatives of Amazon ReInvent 2022, the annual conference in Las Vegas, bringing together 50,000 cloud computing professionals. This year's keynote speakers include Adam Selpisky, CEO of Amazon Web Services, Swami Sivasubramanian, Vice President of Data and Machine Learning at AWS and Werner Vogels, Amazon's CTO. Attendees and web viewers were treated to new features and products, such as AWS Lambda Snapstart for Java Functions, New Quicksight capabilities and quality-of-life improvements to hundreds of services. Justin, Jonathan, Ryan, Peter and Special guest Joe Daly from the Finops foundation talk about the show and the announcements. Thank you to our sponsor, Foghorn Consulting, which provides top notch cloud and DevOps engineers to the world's most innovative companies. Initiatives stalled because you're having trouble hiring? Foghorn can be burning down your DevOps and Cloud backlogs as soon as next week. Episode Highlights ⏰ AWS Pricing Calculator now supports modernization cost estimates for Microsoft workloads. ⏰ AWS Re:Invent 2022 announcements and keynote updates. Top Quote
About ClintonClinton Herget is Field CTO at Snyk, the leader is Developer Security. He focuses on helping Snyk's strategic customers on their journey to DevSecOps maturity. A seasoned technnologist, Cliton spent his 20-year career prior to Snyk as a web software developer, DevOps consultant, cloud solutions architect, and engineering director. Cluinton is passionate about empowering software engineering to do their best work in the chaotic cloud-native world, and is a frequent conference speaker, developer advocate, and technical thought leader.Links Referenced: Snyk: https://snyk.io/ duckbillgroup.com: https://duckbillgroup.com TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is brought to us in part by our friends at Pinecone. They believe that all anyone really wants is to be understood, and that includes your users. AI models combined with the Pinecone vector database let your applications understand and act on what your users want… without making them spell it out.Make your search application find results by meaning instead of just keywords, your personalization system make picks based on relevance instead of just tags, and your security applications match threats by resemblance instead of just regular expressions. Pinecone provides the cloud infrastructure that makes this easy, fast, and scalable. Thanks to my friends at Pinecone for sponsoring this episode. Visit Pinecone.io to understand more.Corey: This episode is bought to you in part by our friends at Veeam. Do you care about backups? Of course you don't. Nobody cares about backups. Stop lying to yourselves! You care about restores, usually right after you didn't care enough about backups. If you're tired of the vulnerabilities, costs and slow recoveries when using snapshots to restore your data, assuming you even have them at all living in AWS-land, there is an alternative for you. Check out Veeam, thats V-E-E-A-M for secure, zero-fuss AWS backup that won't leave you high and dry when it's time to restore. Stop taking chances with your data. Talk to Veeam. My thanks to them for sponsoring this ridiculous podcast.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. One of the fun things about establishing traditions is that the first time you do it, you don't really know that that's what's happening. Almost exactly a year ago, I sat down for a previous promoted guest episode much like this one, With Clinton Herget at Snyk—or Synic; however you want to pronounce that. He is apparently a scarecrow of some sorts because when last we spoke, he was a principal solutions engineer, but like any good scarecrow, he was outstanding in his field, and now, as a result, is a Field CTO. Clinton, Thanks for coming back, and let me start by congratulating you on the promotion. Or consoling you depending upon how good or bad it is.Clinton: You know, Corey, a little bit of column A, a little bit of column B. But very glad to be here again, and frankly, I think it's because you insist on mispronouncing Snyk as Synic, and so you get me again.Corey: Yeah, you could add a couple of new letters to it and just call the company [Synack 00:01:27]. Now, it's a hard pivot to a networking company. So, there's always options.Clinton: I acknowledge what you did there, Corey.Corey: I like that quite a bit. I wasn't sure you'd get it.Clinton: I'm a nerd going way, way back, so we'll have to go pretty deep in the stack for you to stump me on some of this stuff.Corey: As we did with the, “I wasn't sure you'd get it.” See that one sailed right past you. And I win. Chalk another one up for me and the networking pun wars. Great, we'll loop back for that later.Clinton: I don't even know where I am right now.Corey: [laugh]. So, let's go back to a question that one would think that I'd already established a year ago, but I have the attention span of basically a goldfish, let's not kid ourselves. So, as I'm visiting the Snyk website, I find that it says different words than it did a year ago, which is generally a sign that is positive; when nothing's been updated including the copyright date, things are going really well or really badly. One wonders. But no, now you're talking about Snyk Cloud, you're talking about several other offerings as well, and my understanding of what it is you folks do no longer appears to be completely accurate. So, let me be direct. What the hell do you folks do over there?Clinton: It's a really great question. Glad you asked me on a year later to answer it. I would say at a very high level, what we do hasn't changed. However, I think the industry has certainly come a long way in the past couple years and our job is to adapt to that Snyk—again, pronounced like a pair of sneakers are sneaking around—it's a developer security platform. So, we focus on enabling the people who build applications—which as of today, means modern applications built in the cloud—to have better visibility, and ultimately a better chance of mitigating the risk that goes into those applications when it matters most, which is actually in their workflow.Now, you're exactly right. Things have certainly expanded in that remit because the job of a software engineer is very different, I think this year than it even was last year, and that's continually evolving over time. As a developer now, I'm doing a lot more than I was doing a few years ago. And one of the things I'm doing is building infrastructure in the cloud, I'm writing YAML files, I'm writing CloudFormation templates to deploy things out to AWS. And what happens in the cloud has a lot to do with the risk to my organization associated with those applications that I'm building.So, I'd love to talk a little bit more about why we decided to make that move, but I don't think that represents a watering down of what we're trying to do at Snyk. I think it recognizes that developer security vision fundamentally can't exist without some understanding of what's happening in the cloud.Corey: One of the things that always scares me is—and sets the spidey sense tingling—is when I see a company who has a product, and I'm familiar—ish—with what they do. And then they take their product name and slap the word cloud at the end, which is almost always codes to, “Okay, so we took the thing that we sold in boxes in data centers, and now we're making a shitty hosted version available because it turns out you rubes will absolutely pay a subscription for it.” Yeah, I don't get the sense that at all is what you're doing. In fact, I don't believe that you're offering a hosted managed service at the moment, are you?Clinton: No, the cloud part, that fundamentally refers to a new product, an offering that looks at the security or potentially the risks being introduced into cloud infrastructure, by now the engineers who were doing it who are writing infrastructure as code. We previously had an infrastructure-as-code security product, and that served alongside our static analysis tool which is Snyk Code, our open-source tool, our container scanner, recognizing that the kinds of vulnerabilities you can potentially introduce in writing cloud infrastructure are not only bad to the organization on their own—I mean, nobody wants to create an S3 bucket that's wide open to the world—but also, those misconfigurations can increase the blast radius of other kinds of vulnerabilities in the stack. So, I think what it does is it recognizes that, as you and I think your listeners well know, Corey, there's no such thing as the cloud, right? The cloud is just a bunch of fancy software designed to abstract away from the fact that you're running stuff on somebody else's computer, right?Corey: Unfortunately, in this case, the fact that you're calling it Snyk Cloud does not mean that you're doing what so many other companies in that same space do it would have led to a really short interview because I have no faith that it's the right path forward, especially for you folks, where it's, “Oh, you want to be secure? You've got to host your stuff on our stuff instead. That's why we called it cloud.” That's the direction that I've seen a lot of folks try and pivot in, and I always find it disastrous. It's, “Yeah, well, at Snyk if we run your code or your shitty applications here in our environment, it's going to be safer than if you run it yourself on something untested like AWS.” And yeah, those stories hold absolutely no water. And may I just say, I'm gratified that's not what you're doing?Clinton: Absolutely not. No, I would say we have no interest in running anyone's applications. We do want to scan them though, right? We do want to give the developers insight into the potential misconfigurations, the risks, the vulnerabilities that you're introducing. What sets Snyk apart, I think, from others in that application security testing space is we focus on the experience of the developer, rather than just being another tool that runs and generates a bunch of PDFs and then throws them back to say, “Here's everything you did wrong.”We want to say to developers, “Here's what you could do better. Here's how that default in a CloudFormation template that leads to your bucket being, you know, wide open on the internet could be changed. Here's the remediation that you could introduce.” And if we do that at the right moment, which is inside that developer workflow, inside the IDE, on their local machine, before that gets deployed, there's a much greater chance that remediation is going to be implemented and it's going to happen much more cheaply, right? Because you no longer have to do the round trip all the way out to the cloud and back.So, the cloud part of it fundamentally means completing that story, recognizing that once things do get deployed, there's a lot of valuable context that's happening out there that a developer can really take advantage of. They can say, “Wait a minute. Not only do I have a Log4Shell vulnerability, right, in one of my open-source dependencies, but that artifact, that application is actually getting deployed to a VPC that has ingress from the internet,” right? So, not only do I have remote code execution in my application, but it's being put in an enclave that actually allows it to be exploited. You can only know that if you're actually looking at what's really happening in the cloud, right?So, not only does Snyk cloud allows us to provide an additional layer of security by looking at what's misconfigured in that cloud environment and help your developers make remediations by saying, “Here's the actual IAC file that caused that infrastructure to come into existence,” but we can also say, here's how that affects the risk of other kinds of vulnerabilities at different layers in the stack, right? Because it's all software; it's all connected. Very rarely does a vulnerability translate one-to-one into risk, right? They're compound because modern software is compound. And I think what developers lack is the tooling that fits into their workflow that understands what it means to be a software engineer and actually helps them make better choices rather than punishing them after the fact for guessing and making bad ones.Corey: That sounds awesome at a very high level. It is very aligned with how executives and decision-makers think about a lot of these things. Let's get down to brass tacks for a second. Assume that I am the type of developer that I am in real life, by which I mean shitty. What am I going to wind up attempting to do that Snyk will flag and, in other words, protect me from myself and warn me that I'm about to commit a dumb?Clinton: First of all, I would say, look, there's no such thing as a non-shitty developer, right? And I built software for 20 years and I decided that's really hard. What's a lot easier is talking about building software for a living. So, that's what I do now. But fundamentally, the reason I'm at Snyk, is I want to help people who are in the kinds of jobs that I had for a very long time, which is to say, you have a tremendous amount of anxiety because you recognize that the success of the organization rests on your shoulders, and you're making hundreds, if not thousands of decisions every day without the right context to understand fully how the results of that decision is going to affect the organization that you work for.So, I think every developer in the world has to deal with this constant cognitive dissonance of saying, “I don't know that this is right, but I have to do it anyway because I need to clear that ticket because that release needs to get into production.” And it becomes really easy to short-sightedly do things like pull an open-source dependency without checking whether it has any CVEs associated with it because that's the version that's easiest to implement with your code that already exists. So, that's one piece. Snyk Open Source, designed to traverse that entire tree of dependencies in open-source all the way down, all the hundreds and thousands of packages that you're pulling in to say, not only, here's a vulnerability that you should really know is going to end up in your application when it's built, but also here's what you can do about it, right? Here's the upgrade you can make, here's the minimum viable change that actually gets you out of this problem, and to do so when it's in the right context, which is in you know, as you're making that decision for the first time, right, inside your developer environment.That also applies to things like container vulnerabilities, right? I have even less visibility into what's happening inside a container than I do inside my application. Because I know, say, I'm using an Ubuntu or a Red Hat base image. I have no idea, what are all the Linux packages that are on it, let alone what are the vulnerabilities associated with them, right? So, being able to detect, I've got a version of OpenSSL 3.0 that has a potentially serious vulnerability associated with it before I've actually deployed that container out into the cloud very much helps me as a developer.Because I'm limiting the rework or the refactoring I would have to do by otherwise assuming I'm making a safe choice or guessing at it, and then only finding out after I've written a bunch more code that relies on that decision, that I have to go back and change it, and then rewrite all of the things that I wrote on top of it, right? So, it's the identifying the layer in the stack where that risk could be introduced, and then also seeing how it's affected by all of those other layers because modern software is inherently complex. And that complexity is what drives both the risk associated with it, and also things like efficiency, which I know your audience is, for good reason, very concerned about.Corey: I'm going to challenge you on aspect of this because on the tin, the way you describe it, it sounds like, “Oh, I already have something that does that. It's the GitHub Dependabot story where it winds up sending me a litany of complaints every week.” And we are talking, if I did nothing other than read this email in that day, that would be a tremendously efficient processing of that entire thing because so much of it is stuff that is ancient and archived, and specific aspects of the vulnerabilities are just not relevant. And you talk about the OpenSSL 3.0 issues that just recently came out.I have no doubt that somewhere in the most recent email I've gotten from that thing, it's buried two-thirds of the way down, like all the complaints like the dishwasher isn't loaded, you forgot to take the trash out, that baby needs a change, the kitchen is on fire, and the vacuuming, and the r—wait, wait. What was that thing about the kitchen? Seems like one of those things is not like the others. And it just gets lost in the noise. Now, I will admit to putting my thumb a little bit on the scale here because I've used Snyk before myself and I know that you don't do that. How do you avoid that trap?Clinton: Great question. And I think really, the key to the story here is, developers need to be able to prioritize, and in order to prioritize effectively, you need to understand the context of what happens to that application after it gets deployed. And so, this is a key part of why getting the data out of the cloud and bringing it back into the code is so important. So, for example, take an OpenSSL vulnerability. Do you have it on a container image you're using, right? So, that's question number one.Question two is, is there actually a way that code can be accessed from the outside? Is it included or is it called? Is the method activated by some other package that you have running on that container? Is that container image actually used in a production deployment? Or does it just go sit in a registry and no one ever touches it?What are the conditions required to make that vulnerability exploitable? You look at something like Spring Shell, for example, yes, you need a certain version of spring-beans in a JAR file somewhere, but you also need to be running a certain version of Tomcat, and you need to be packaging those JARs inside a WAR in a certain way.Corey: Exactly. I have a whole bunch of Lambda functions that provide the pipeline system that I use to build my newsletter every week, and I get screaming concerns about issues in, for example, a version of the markdown parser that I've subverted. Yeah, sure. I get that, on some level, if I were just giving it random untrusted input from the internet and random ad hoc users, but I'm not. It's just me when I write things for that particular Lambda function.And I'm not going to be actively attempting to subvert the thing that I built myself and no one else should have access to. And looking through the details of some of these things, it doesn't even apply to the way that I'm calling the libraries, so it's just noise, for lack of a better term. It is not something that basically ever needs to be adjusted or fixed.Clinton: Exactly. And I think cutting through that noise is so key to creating developer trust in any kind of tool that scanning an asset and providing you what, in theory, are a list of actionable steps, right? I need to be able to understand what is the thing, first of all. There's a lot of tools that do that, right, and we tend to mock them by saying things like, “Oh, it's just another PDF generator. It's just another thousand pages that you're never going to read.”So, getting the information in the right place is a big part of it, but filtering out all of the noise by saying, we looked at not just one layer of the stack, but multiple layers, right? We know that you're using this open-source dependency and we also know that the method that contains the vulnerability is actively called by your application in your first-party code because we ran our static analysis tool against that. Furthermore, we know because we looked at your cloud context, we connected to your AWS API—we're big partners with AWS and very proud of that relationship—but we can tell that there's inbound internet access available to that service, right? So, you start to build a compound case that maybe this is something that should be prioritized, right? Because there's a way into the asset from the outside world, there's a way into the vulnerable functions through the labyrinthine, you know, spaghetti of my code to get there, and the conditions required to exploit it actually exist in the wild.But you can't just run a single tool; you can't just run Dependabot to get that prioritization. You actually have to look at the entire holistic application context, which includes not just your dependencies, but what's happening in the container, what's happening in your first-party, your proprietary code, what's happening in your IAC, and I think most importantly for modern applications, what's actually happening in the cloud once it gets deployed, right? And that's sort of the holy grail of completing that loop to bring the right context back from the cloud into code to understand what change needs to be made, and where, and most importantly why. Because it's a priority that actually translates into organizational risk to get a developer to pay attention, right? I mean, that is the key to I think any security concern is how do you get engineering mindshare and trust that this is actually what you should be paying attention to and not a bunch of rework that doesn't actually make your software more secure?Corey: One of the challenges that I see across the board is that—well, let's back up a bit here. I have in previous episodes talked in some depth about my position that when it comes to the security of various cloud providers, Google is number one, and AWS is number two. Azure is a distant third because it figures out what Crayons tastes the best; I don't know. But the reason is not because of any inherent attribute of their security models, but rather that Google massively simplifies an awful lot of what happens. It automatically assumes that resources in the same project should be able to talk to one another, so I don't have to painstakingly configure that.In AWS-land, all of this must be done explicitly; no one has time for that, so we over-scope permissions massively and never go back and rein them in. It's a configuration vulnerability more than an underlying inherent weakness of the platform. Because complexity is the enemy of security in many respects. If you can't fit it all in your head to reason about it, how can you understand the security ramifications of it? AWS offers a tremendous number of security services. Many of them, when taken in some totality of their pricing, cost more than any breach, they could be expected to prevent. Adding more stuff that adds more complexity in the form of Snyk sounds like it's the exact opposite of what I would want to do. Change my mind.Clinton: I would love to. I would say, fundamentally, I think you and I—and by ‘I,' I mean Snyk and you know, Corey Quinn Enterprises Limited—I think we fundamentally have the same enemy here, right, which is the cyclomatic complexity of software, right, which is how many different pathways do the bits have to travel down to reach the same endpoint, right, the same goal. The more pathways there are, the more risk is introduced into your software, and the more inefficiency is introduced, right? And then I know you'd love to talk about how many different ways is there to run a container on AWS, right? It's either 30 or 400 or eleventy-million.I think you're exactly right that that complexity, it is great for, first of all, selling cloud resources, but also, I think, for innovating, right, for building new kinds of technology on top of that platform. The cost that comes along with that is a lack of visibility. And I think we are just now, as we approach the end of 2022 here, coming to recognize that fundamentally, the complexity of modern software is beyond the ability of a single engineer to understand. And that is really important from a security perspective, from a cost control perspective, especially because software now creates its own infrastructure, right? You can't just now secure the artifact and secure the perimeter that it gets deployed into and say, “I've done my job. Nobody can breach the perimeter and there's no vulnerabilities in the thing because we scanned it and that thing is immutable forever because it's pets, not cattle.”Where I think the complexity story comes in is to recognize like, “Hey, I'm deploying this based on a quickstart or CloudFormation template that is making certain assumptions that make my job easier,” right, in a very similar way that choosing an open-source dependency makes my job easier as a developer because I don't have to write all of that code myself. But what it does mean is I lack the visibility into, well hold on. How many different pathways are there for getting things done inside this dependency? How many other dependencies are brought on board? In the same way that when I create an EKS cluster, for example, from a CloudFormation template, what is it creating in the background? How many VPCs are involved? What are the subnets, right? How are they connected to each other? Where are the potential ingress points?So, I think fundamentally, getting visibility into that complexity is step number one, but understanding those pathways and how they could potentially translate into risk is critically important. But that prioritization has to involve looking at the software holistically and not just individual layers, right? I think we lose when we say, “We ran a static analysis tool and an open-source dependency scanner and a container scanner and a cloud config checker, and they all came up green, therefore the software doesn't have any risks,” right? That ignores the fundamental complexity in that all of these layers are connected together. And from an adversaries perspective, if my job is to go in and exploit software that's hosted in the cloud, I absolutely do not see the application model that way.I see it as it is inherently complex and that's a good thing for me because it means I can rely on the fact that those engineers had tremendous anxiety, we're making a lot of guesses, and crossing their fingers and hoping something would work and not be exploitable by me, right? So, the only way I think we get around that is to recognize that our engineers are critical stakeholders in that security process and you fundamentally lack that visibility if you don't do your scanning until after the fact. If you take that traditional audit-based approach that assumes a very waterfall, legacy approach to building software, and recognize that, hey, we're all on this infinite loop race track now. We're deploying every three-and-a-half seconds, everything's automated, it's all built at scale, but the ability to do that inherently implies all of this additional complexity that ultimately will, you know, end up haunting me, right? If I don't do anything about it, to make my engineer stakeholders in, you know, what actually gets deployed and what risks it brings on board.Corey: This episode is sponsored in part by our friends at Uptycs. Attackers don't think in silos, so why would you have siloed solutions protecting cloud, containers, and laptops distinctly? Meet Uptycs - the first unified solution that prioritizes risk across your modern attack surface—all from a single platform, UI, and data model. Stop by booth 3352 at AWS re:Invent in Las Vegas to see for yourself and visit uptycs.com. That's U-P-T-Y-C-S.com. My thanks to them for sponsoring my ridiculous nonsense.Corey: When I wind up hearing you talk about this—I'm going to divert us a little bit because you're dancing around something that it took me a long time to learn. When I first started fixing AWS bills for a living, I thought that it would be mostly math, by which I mean arithmetic. That's the great secret of cloud economics. It's addition, subtraction, and occasionally multiplication and division. No, turns out it's much more psychology than it is math. You're talking in many aspects about, I guess, what I'd call the psychology of a modern cloud engineer and how they think about these things. It's not a technology problem. It's a people problem, isn't it?Clinton: Oh, absolutely. I think it's the people that create the technology. And I think the longer you persist in what we would call the legacy viewpoint, right, not recognizing what the cloud is—which is fundamentally just software all the way down, right? It is abstraction layers that allow you to ignore the fact that you're running stuff on somebody else's computer—once you recognize that, you realize, oh, if it's all software, then the problems that it introduces are software problems that need software solutions, which means that it must involve activity by the people who write software, right? So, now that you're in that developer world, it unlocks, I think, a lot of potential to say, well, why don't developers tend to trust the security tools they've been provided with, right?I think a lot of it comes down to the question you asked earlier in terms of the noise, the lack of understanding of how those pieces are connected together, or the lack of context, or not even frankly, caring about looking beyond the single-point solution of the problem that solution was designed to solve. But more importantly than that, not recognizing what it's like to build modern software, right, all of the decisions that have to be made on a daily basis with very limited information, right? I might not even understand where that container image I'm building is going in the universe, let alone what's being built on top of it and how much critical customer data is being touched by the database, that that container now has the credentials to access, right? So, I think in order to change anything, we have to back way up and say, problems in the cloud or software problems and we have to treat them that way.Because if we don't if we continue to represent the cloud as some evolution of the old environment where you just have this perimeter that's pre-existing infrastructure that you're deploying things onto, and there's a guy with a neckbeard in the basement who is unplugging cables from a switch and plugging them back in and that's how networking problems are solved, I think you missed the idea that all of these abstraction layers introduced the very complexity that needs to be solved back in the build space. But that requires visibility into what actually happens when it gets deployed. The way I tend to think of it is, there's this firewall in place. Everybody wants to say, you know, we're doing DevOps or we're doing DevSecOps, right? And that's a lie a hundred percent of the time, right? No one is actually, I think, adhering completely to those principles.Corey: That's why one of the core tenets of ClickOps is lying about doing anything in the console.Clinton: Absolutely, right? And that's why shadow IT becomes more and more prevalent the deeper you get into modern development, not less and less prevalent because it's fundamentally hard to recognize the entirety of the potential implications, right, of a decision that you're making. So, it's a lot easier to just go in the console and say, “Okay, I'm going to deploy one EC2 to do this. I'm going to get it right at some point.” And that's why every application that's ever been produced by human hands has a comment in it that says something like, “I don't know why this works but it does. Please don't change it.”And then three years later because that developer has moved on to another job, someone else comes along and looks at that comment and says, “That should really work. I'm going to change it.” And they do and everything fails, and they have to go back and fix it the original way and then add another comment saying, “Hey, this person above me, they were right. Please don't change this line.” I think every engineer listening right now knows exactly where that weak spot is in the applications that they've written and they're terrified of that.And I think any tool that's designed to help developers fundamentally has to get into the mindset, get into the psychology of what that is, like, of not fundamentally being able to understand what those applications are doing all of the time, but having to write code against them anyway, right? And that's what leads to, I think, the fear that you're going to get woken up because your pager is going to go off at 3 a.m. because the building is literally on fire and it's because of code that you wrote. We have to solve that problem and it has to be those people who's psychology we get into to understand, how are you working and how can we make your life better, right? And I really do think it comes with that the noise reduction, the understanding of complexity, and really just being humble and saying, like, “We get that this job is really hard and that the only way it gets better is to begin admitting that to each other.”Corey: I really wish that there were a better way to articulate a lot of these things. This the reason that I started doing a security newsletter; it's because cost and security are deeply aligned in a few ways. One of them is that you care about them a lot right after you failed to care about them sufficiently, but the other is that you've got to build guardrails in such a way that doing the right thing is easier than doing it the wrong way, or you're never going to gain any traction.Clinton: I think that's absolutely right. And you use the key term there, which is guardrails. And I think that's where in their heart of hearts, that's where every security professional wants to be, right? They want to be defining policy, they want to be understanding the risk posture of the organization and nudging it in a better direction, right? They want to be talking up to the board, to the executive team, and creating confidence in that risk posture, rather than talking down or off to the side—depending on how that org chart looks—to the engineers and saying, “Fix this, fix that, and then fix this other thing.” A, B, and C, right?I think the problem is that everyone in a security role or an organization of any size at this point, is doing 90% of the latter and only about 10% of the former, right? They're acting as gatekeepers, not as guardrails. They're not defining policy, they're spending all of their time creating Jira tickets and all of their time tracking down who owns the piece of code that got deployed to this pod on EKS that's throwing all these errors on my console, and how can I get the person to make a decision to actually take an action that stops these notifications from happening, right? So, all they're doing is throwing footballs down the field without knowing if there's a receiver there, right, and I think that takes away from the job that our security analysts really shouldn't be doing, which is creating those guardrails, which is having confidence that the policy they set is readily understood by the developers making decisions, and that's happening in an automated way without them having to create friction by bothering people all the time. I don't think security people want to be [laugh] hated by the development teams that they work with, but they are. And the reason they are is I think, fundamentally, we lack the tooling, we lack—Corey: They are the barrier method.Clinton: Exactly. And we lacked the processes to get the right intelligence in a way that's consumable by the engineers when they're doing their job, and not after the fact, which is typically when the security people have done their jobs.Corey: It's sad but true. I wish that there were a better way to address these things, and yet here we are.Clinton: If only there were better way to address these things.Corey: [laugh].Clinton: Look, I wouldn't be here at Snyk if I didn't think there were a better way, and I wouldn't be coming on shows like yours to talk to the engineering communities, right, people who have walked the walk, right, who have built those Terraform files that contain these misconfigurations, not because they're bad people or because they're lazy, or because they don't do their jobs well, but because they lacked the visibility, they didn't have the understanding that that default is actually insecure. Because how would I know that otherwise, right? I'm building software; I don't see myself as an expert on infrastructure, right, or on Linux packages or on cyclomatic complexity or on any of these other things. I'm just trying to stay in my lane and do my job. It's not my fault that the software has become too complex for me to understand, right?But my management doesn't understand that and so I constantly have white knuckles worrying that, you know, the next breach is going to be my fault. So, I think the way forward really has to be, how do we make our developers stakeholders in the risk being introduced by the software they write to the organization? And that means everything we've been talking about: it means prioritization; it means understanding how the different layers of the stack affect each other, especially the cloud pieces; it means an extensible platform that lets me write code against it to inject my own reasoning, right? The piece that we haven't talked about here is that risk calculation doesn't just involve technical aspects, there's also business intelligence that's involved, right? What are my critical applications, right, what actually causes me to lose significant amounts of money if those services go offline?We at Snyk can't tell that. We can't run a scanner to say these are your crown jewel services that can't ever go down, but you can know that as an organization. So, where we're going with the platform is opening up the extensible process, creating APIs for you to be able to affect that risk triage, right, so that as the creators have guardrails as the security team, you are saying, “Here's how we want our developers to prioritize. Here are all of the factors that go into that decision-making.” And then you can be confident that in their environment, back over in developer-land, when I'm looking at IntelliJ, or, you know, or on my local command line, I am seeing the guardrails that my security team has set for me and I am confident that I'm fixing the right thing, and frankly, I'm grateful because I'm fixing it at the right time and I'm doing it in such a way and with a toolset that actually is helping me fix it rather than just telling me I've done something wrong, right, because everything we do at Snyk focuses on identifying the solution, not necessarily identifying the problem.It's great to know that I've got an unencrypted S3 bucket, but it's a whole lot better if you give me the line of code and tell me exactly where I have to copy and paste it so I can go on to the next thing, rather than spending an hour trying to figure out, you know, where I put that line and what I actually have to change it to, right? I often say that the most valuable currency for a developer, for a software engineer, it's not money, it's not time, it's not compute power or anything like that, it's the right context, right? I actually have to understand what are the implications of the decision that I'm making, and I need that to be in my own environment, not after the fact because that's what creates friction within an organization is when I could have known earlier and I could have known better, but instead, I had to guess I had to write a bunch of code that relies on the thing that was wrong, and now I have to redo it all for no good reason other than the tooling just hadn't adapted to the way modern software is built.Corey: So, one last question before we wind up calling it a day here. We are now heavily into what I will term pre:Invent where we're starting to see a whole bunch of announcements come out of the AWS universe in preparation for what I'm calling Crappy Cloud Hanukkah this year because I'm spending eight nights in Las Vegas. What are you doing these days with AWS specifically? I know I keep seeing your name in conjunction with their announcements, so there's something going on over there.Clinton: Absolutely. No, we're extremely excited about the partnership between Snyk and AWS. Our vulnerability intelligence is utilized as one of the data sources for AWS Inspector, particularly around open-source packages. We're doing a lot of work around things like the code suite, building Snyk into code pipeline, for example, to give developers using that code suite earlier visibility into those vulnerabilities. And really, I think the story kind of expands from there, right?So, we're moving forward with Amazon, recognizing that it is, you know, sort of the de facto. When we say cloud, very often we mean AWS. So, we're going to have a tremendous presence at re:Invent this year, I'm going to be there as well. I think we're actually going to have a bunch of handouts with your face on them is my understanding. So, please stop by the booth; would love to talk to folks, especially because we've now released the Snyk Cloud product and really completed that story. So, anything we can do to talk about how that additional context of the cloud helps engineers because it's all software all the way down, those are absolutely conversations we want to be having.Corey: Excellent. And we will, of course, put links to all of these things in the [show notes 00:35:00] so people can simply click, and there they are. Thank you so much for taking all this time to speak with me. I appreciate it.Clinton: All right. Thank you so much, Corey. Hope to do it again next year.Corey: Clinton Herget, Field CTO at Snyk. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, along with an angry comment telling me that I'm being completely unfair to Azure, along with your favorite tasting color of Crayon.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.
About BenBen Whaley is a staff software engineer at Chime. Ben is co-author of the UNIX and Linux System Administration Handbook, the de facto standard text on Linux administration, and is the author of two educational videos: Linux Web Operations and Linux System Administration. He is an AWS Community Hero since 2014. Ben has held Red Hat Certified Engineer (RHCE) and Certified Information Systems Security Professional (CISSP) certifications. He earned a B.S. in Computer Science from Univ. of Colorado, Boulder.Links Referenced: Chime Financial: https://www.chime.com/ alternat.cloud: https://alternat.cloud Twitter: https://twitter.com/iamthewhaley LinkedIn: https://www.linkedin.com/in/benwhaley/ TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Forget everything you know about SSH and try Tailscale. Imagine if you didn't need to manage PKI or rotate SSH keys every time someone leaves. That'd be pretty sweet, wouldn't it? With Tailscale SSH, you can do exactly that. Tailscale gives each server and user device a node key to connect to its VPN, and it uses the same node key to authorize and authenticate SSH.Basically you're SSHing the same way you manage access to your app. What's the benefit here? Built-in key rotation, permissions as code, connectivity between any two devices, reduce latency, and there's a lot more, but there's a time limit here. You can also ask users to reauthenticate for that extra bit of security. Sounds expensive?Nope, I wish it were. Tailscale is completely free for personal use on up to 20 devices. To learn more, visit snark.cloud/tailscale. Again, that's snark.cloud/tailscaleCorey: Welcome to Screaming in the Cloud. I'm Corey Quinn and this is an episode unlike any other that has yet been released on this august podcast. Let's begin by introducing my first-time guest somehow because apparently an invitation got lost in the mail somewhere. Ben Whaley is a staff software engineer at Chime Financial and has been an AWS Community Hero since Andy Jassy was basically in diapers, to my level of understanding. Ben, welcome to the show.Ben: Corey, so good to be here. Thanks for having me on.Corey: I'm embarrassed that you haven't been on the show before. You're one of those people that slipped through the cracks and somehow I was very bad at following up slash hounding you into finally agreeing to be here. But you certainly waited until you had something auspicious to talk about.Ben: Well, you know, I'm the one that really should be embarrassed here. You did extend the invitation and I guess I just didn't feel like I had something to drop. But I think today we have something that will interest most of the listeners without a doubt.Corey: So, folks who have listened to this podcast before, or read my newsletter, or follow me on Twitter, or have shared an elevator with me, or at any point have passed me on the street, have heard me complain about the Managed NAT Gateway and it's egregious data processing fee of four-and-a-half cents per gigabyte. And I have complained about this for small customers because they're in the free tier; why is this thing charging them 32 bucks a month? And I have complained about this on behalf of large customers who are paying the GDP of the nation of Belize in data processing fees as they wind up shoving very large workloads to and fro, which is I think part of the prerequisite requirements for having a data warehouse. And you are no different than the rest of these people who have those challenges, with the singular exception that you have done something about it, and what you have done is so, in retrospect, blindingly obvious that I am embarrassed the rest of us never thought of it.Ben: It's interesting because when you are doing engineering, it's often the simplest solution that is the best. I've seen this repeatedly. And it's a little surprising that it didn't come up before, but I think it's in some way, just a matter of timing. But what we came up with—and is this the right time to get into it, do you want to just kind of name the solution, here?Corey: Oh, by all means. I'm not going to steal your thunder. Please, tell us what you have wrought.Ben: We're calling it AlterNAT and it's an alternative solution to a high-availability NAT solution. As everybody knows, NAT Gateway is sort of the default choice; it certainly is what AWS pushes everybody towards. But there is, in fact, a legacy solution: NAT instances. These were around long before NAT Gateway made an appearance. And like I said they're considered legacy, but with the help of lots of modern AWS innovations and technologies like Lambdas and auto-scaling groups with max instance lifetimes and the latest generation of networking improved or enhanced instances, it turns out that we can maybe not quite get as effective as a NAT Gateway, but we can save a lot of money and skip those data processing charges entirely by having a NAT instance solution with a failover NAT Gateway, which I think is kind of the key point behind the solution. So, are you interested in diving into the technical details?Corey: That is very much the missing piece right there. You're right. What we used to use was NAT instances. That was the thing that we used because we didn't really have another option. And they had an interface in the public subnet where they lived and an interface hanging out in the private subnet, and they had to be configured to wind up passing traffic to and fro.Well, okay, that's great and all but isn't that kind of brittle and dangerous? I basically have a single instance as a single point of failure and these are the days early on when individual instances did not have the level of availability and durability they do now. Yeah, it's kind of awful, but here you go. I mean, the most galling part of the Managed NAT Gateway service is not that it's expensive; it's that it's expensive, but also incredibly good at what it does. You don't have to think about this whole problem anymore, and as of recently, it also supports ipv4 to ipv6 translation as well.It's not that the service is bad. It's that the service is stonkingly expensive, particularly at scale. And everything that we've seen before is either oh, run your own NAT instances or bend your knee and pays your money. And a number of folks have come up with different options where this is ridiculous. Just go ahead and run your own NAT instances.Yeah, but what happens when I have to take it down for maintenance or replace it? It's like, well, I guess you're not going to the internet today. This has the, in hindsight, obvious solution, well, we just—we run the Managed NAT Gateway because the 32 bucks a year in instance-hour charges don't actually matter at any point of scale when you're doing this, but you wind up using that for day in, day out traffic, and the failover mode is simply you'll use the expensive Managed NAT Gateway until the instance is healthy again and then automatically change the route table back and forth.Ben: Yep. That's exactly it. So, the auto-scaling NAT instance solution has been around for a long time well, before even NAT Gateway was released. You could have NAT instances in an auto-scaling group where the size of the group was one, and if the NAT instance failed, it would just replace itself. But this left a period in which you'd have no internet connectivity during that, you know, when the NAT instance was swapped out.So, the solution here is that when auto-scaling terminates an instance, it fails over the route table to a standby NAT Gateway, rerouting the traffic. So, there's never a point at which there's no internet connectivity, right? The NAT instance is running, processing traffic, gets terminated after a certain period of time, configurable, 14 days, 30 days, whatever makes sense for your security strategy could be never, right? You could choose that you want to have your own maintenance window in which to do it.Corey: And let's face it, this thing is more or less sitting there as a network traffic router, for lack of a better term. There is no need to ever log into the thing and make changes to it until and unless there's a vulnerability that you can exploit via somehow just talking to the TCP stack when nothing's actually listening on the host.Ben: You know, you can run your own AMI that has been pared down to almost nothing, and that instance doesn't do much. It's using just a Linux kernel to sit on two networks and pass traffic back and forth. It has a translation table that kind of keeps track of the state of connections and so you don't need to have any service running. To manage the system, we have SSM so you can use Session Manager to log in, but frankly, you can just disable that. You almost never even need to get a shell. And that is, in fact, an option we have in the solution is to disable SSM entirely.Corey: One of the things I love about this approach is that it is turnkey. You throw this thing in there and it's good to go. And in the event that the instance becomes unhealthy, great, it fails traffic over to the Managed NAT Gateway while it terminates the old node and replaces it with a healthy one and then fails traffic back. Now, I do need to ask, what is the story of network connections during that failover and failback scenario?Ben: Right, that's the primary drawback, I would say, of the solution is that any established TCP connections that are on the NAT instance at the time of a route change will be lost. So, say you have—Corey: TCP now terminates on the floor.Ben: Pretty much. The connections are dropped. If you have an open SSH connection from a host in the private network to a host on the internet and the instance fails over to the NAT Gateway, the NAT Gateway doesn't have the translation table that the NAT instance had. And not to mention, the public IP address also changes because you have an Elastic IP assigned to the NAT instance, a different Elastic IP assigned to the NAT Gateway, and so because that upstream IP is different, the remote host is, like, tracking the wrong IP. So, those connections, they're going to be lost.So, there are some use cases where this may not be suitable. We do have some ideas on how you might mitigate that, for example, with the use of a maintenance window to schedule the replacement, replaced less often so it doesn't have to affect your workflow as much, but frankly, for many use cases, my belief is that it's actually fine. In our use case at Chime, we found that it's completely fine and we didn't actually experience any errors or failures. But there might be some use cases that are more sensitive or less resilient to failure in the first place.Corey: I would also point out that a lot of how software is going to behave is going to be a reflection of the era in which it was moved to cloud. Back in the early days of EC2, you had no real sense of reliability around any individual instance, so everything was written in a very defensive manner. These days, with instances automatically being able to flow among different hardware so we don't get instance interrupt notifications the way we once did on a semi-constant basis, it more or less has become what presents is bulletproof, so a lot of people are writing software that's a bit more brittle. But it's always been a best practice that when a connection fails okay, what happens at failure? Do you just give up and throw your hands in the air and shriek for help or do you attempt to retry a few times, ideally backing off exponentially?In this scenario, those retries will work. So, it's a question of how well have you built your software. Okay, let's say that you made the worst decisions imaginable, and okay, if that connection dies, the entire workload dies. Okay, you have the option to refactor it to be a little bit better behaved, or alternately, you can keep paying the Manage NAT Gateway tax of four-and-a-half cents per gigabyte in perpetuity forever. I'm not going to tell you what decision to make, but I know which one I'm making.Ben: Yeah, exactly. The cost savings potential of it far outweighs the potential maintenance troubles, I guess, that you could encounter. But the fact is, if you're relying on Managed NAT Gateway and paying the price for doing so, it's not as if there's no chance for connection failure. NAT Gateway could also fail. I will admit that I think it's an extremely robust and resilient solution. I've been really impressed with it, especially so after having worked on this project, but it doesn't mean it can't fail.And beyond that, upstream of the NAT Gateway, something could in fact go wrong. Like, internet connections are unreliable, kind of by design. So, if your system is not resilient to connection failures, like, there's a problem to solve there anyway; you're kind of relying on hope. So, it's a kind of a forcing function in some ways to build architectural best practices, in my view.Corey: I can't stress enough that I have zero problem with the capabilities and the stability of the Managed NAT Gateway solution. My complaints about it start and stop entirely with the price. Back when you first showed me the blog post that is releasing at the same time as this podcast—and you can visit that at alternat.cloud—you sent me an early draft of this and what I loved the most was that your math was off because of a not complete understanding of the gloriousness that is just how egregious the NAT Gateway charges are.Your initial analysis said, “All right, if you're throwing half a terabyte out to the internet, this has the potential of cutting the bill by”—I think it was $10,000 or something like that. It's, “Oh no, no. It has the potential to cut the bill by an entire twenty-two-and-a-half thousand dollars.” Because this processing fee does not replace any egress fees whatsoever. It's purely additive. If you forget to have a free S3 Gateway endpoint in a private subnet, every time you put something into or take something out of S3, you're paying four-and-a-half cents per gigabyte on that, despite the fact there's no internet transitory work, it's not crossing availability zones. It is simply a four-and-a-half cent fee to retrieve something that has only cost you—at most—2.3 cents per month to store in the first place. Flip that switch, that becomes completely free.Ben: Yeah. I'm not embarrassed at all to talk about the lack of education I had around this topic. The fact is I'm an engineer primarily and I came across the cost stuff because it kind of seemed like a problem that needed to be solved within my organization. And if you don't mind, I might just linger on this point and kind of think back a few months. I looked at the AWS bill and I saw this egregious ‘EC2 Other' category. It was taking up the majority of our bill. Like, the single biggest line item was EC2 Other. And I was like, “What could this be?”Corey: I want to wind up flagging that just because that bears repeating because I often get people pushing back of, “Well, how bad—it's one Managed NAT Gateway. How much could it possibly cost? $10?” No, it is the majority of your monthly bill. I cannot stress that enough.And that's not because the people who work there are doing anything that they should not be doing or didn't understand all the nuances of this. It's because for the security posture that is required for what you do—you are at Chime Financial, let's be clear here—putting everything in public subnets was not really a possibility for you folks.Ben: Yeah. And not only that but there are plenty of services that have to be on private subnets. For example, AWS Glue services must run in private VPC subnets if you want them to be able to talk to other systems in your VPC; like, they cannot live in public subnet. So essentially, if you want to talk to the internet from those jobs, you're forced into some kind of NAT solution. So, I dug into the EC2 Other category and I started trying to figure out what was going on there.There's no way—natively—to look at what traffic is transiting the NAT Gateway. There's not an interface that shows you what's going on, what's the biggest talkers over that network. Instead, you have to have flow logs enabled and have to parse those flow logs. So, I dug into that.Corey: Well, you're missing a step first because in a lot of environments, people have more than one of these things, so you get to first do the scavenger hunt of, okay, I have a whole bunch of Managed NAT Gateways and first I need to go diving into CloudWatch metrics and figure out which are the heavy talkers. Is usually one or two followed by a whole bunch of small stuff, but not always, so figuring out which VPC you're even talking about is a necessary prerequisite.Ben: Yeah, exactly. The data around it is almost missing entirely. Once you come to the conclusion that it is a particular NAT Gateway—like, that's a set of problems to solve on its own—but first, you have to go to the flow logs, you have to figure out what are the biggest upstream IPs that it's talking to. Once you have the IP, it still isn't apparent what that host is. In our case, we had all sorts of outside parties that we were talking to a lot and it's a matter of sorting by volume and figuring out well, this IP, what is the reverse IP? Who is potentially the host there?I actually had some wrong answers at first. I set up VPC endpoints to S3 and DynamoDB and SQS because those were some top talkers and that was a nice way to gain some security and some resilience and save some money. And then I found, well, Datadog; that's another top talker for us, so I ended up creating a nice private link to Datadog, which they offer for free, by the way, which is more than I can say for some other vendors. But then I found some outside parties, there wasn't a nice private link solution available to us, and yet, it was by far the largest volume. So, that's what kind of started me down this track is analyzing the NAT Gateway myself by looking at VPC flow logs. Like, it's shocking that there isn't a better way to find that traffic.Corey: It's worse than that because VPC flow logs tell you where the traffic is going and in what volumes, sure, on an IP address and port basis, but okay, now you have a Kubernetes cluster that spans two availability zones. Okay, great. What is actually passing through that? So, you have one big application that just seems awfully chatty, you have multiple workloads running on the thing. What's the expensive thing talking back and forth? The only way that you can reliably get the answer to that I found is to talk to people about what those workloads are actually doing, and failing that you're going code spelunking.Ben: Yep. You're exactly right about that. In our case, it ended up being apparent because we have a set of subnets where only one particular project runs. And when I saw the source IP, I could immediately figure that part out. But if it's a K8s cluster in the private subnets, yeah, how are you going to find it out? You're going to have to ask everybody that has workloads running there.Corey: And we're talking about in some cases, millions of dollars a month. Yeah, it starts to feel a little bit predatory as far as how it's priced and the amount of work you have to put in to track this stuff down. I've done this a handful of times myself, and it's always painful unless you discover something pretty early on, like, oh, it's talking to S3 because that's pretty obvious when you see that. It's, yeah, flip switch and this entire engagement just paid for itself a hundred times over. Now, let's see what else we can discover.That is always one of those fun moments because, first, customers are super grateful to learn that, oh, my God, I flipped that switch. And I'm saving a whole bunch of money. Because it starts with gratitude. “Thank you so much. This is great.” And it doesn't take a whole lot of time for that to alchemize into anger of, “Wait. You mean, I've been being ridden like a pony for this long and no one bothered to mention that if I click a button, this whole thing just goes away?”And when you mention this to your AWS account team, like, they're solicitous, but they either have to present as, “I didn't know that existed either,” which is not a good look, or, “Yeah, you caught us,” which is worse. There's no positive story on this. It just feels like a tax on not knowing trivia about AWS. I think that's what really winds me up about it so much.Ben: Yeah, I think you're right on about that as well. My misunderstanding about the NAT pricing was data processing is additive to data transfer. I expected when I replaced NAT Gateway with NAT instance, that I would be substituting data transfer costs for NAT Gateway costs, NAT Gateway data processing costs. But in fact, NAT Gateway incurs both data processing and data transfer. NAT instances only incur data transfer costs. And so, this is a big difference between the two solutions.Not only that, but if you're in the same region, if you're egressing out of your say us-east-1 region and talking to another hosted service also within us-east-1—never leaving the AWS network—you don't actually even incur data transfer costs. So, if you're using a NAT Gateway, you're paying data processing.Corey: To be clear you do, but it is cross-AZ in most cases billed at one penny egressing, and on the other side, that hosted service generally pays one penny ingressing as well. Don't feel bad about that one. That was extraordinarily unclear and the only reason I know the answer to that is that I got tired of getting stonewalled by people that later turned out didn't know the answer, so I ran a series of experiments designed explicitly to find this out.Ben: Right. As opposed to the five cents to nine cents that is data transfer to the internet. Which, add that to data processing on a NAT Gateway and you're paying between thirteen-and-a-half cents to nine-and-a-half cents for every gigabyte egressed. And this is a phenomenal cost. And at any kind of volume, if you're doing terabytes to petabytes, this becomes a significant portion of your bill. And this is why people hate the NAT Gateway so much.Corey: I am going to short-circuit an angry comment I can already see coming on this where people are going to say, “Well, yes. But it's a multi-petabyte scale. Nobody's paying on-demand retail price.” And they're right. Most people who are transmitting that kind of data, have a specific discount rate applied to what they're doing that varies depending upon usage and use case.Sure, great. But I'm more concerned with the people who are sitting around dreaming up ideas for a company where I want to wind up doing some sort of streaming service. I talked to one of those companies very early on in my tenure as a consultant around the billing piece and they wanted me to check their napkin math because they thought that at their numbers when they wound up scaling up, if their projections were right, that they were going to be spending $65,000 a minute, and what did they not understand? And the answer was, well, you didn't understand this other thing, so it's going to be more than that, but no, you're directionally correct. So, that idea that started off on a napkin, of course, they didn't build it on top of AWS; they went elsewhere.And last time I checked, they'd raised well over a quarter-billion dollars in funding. So, that's a business that AWS would love to have on a variety of different levels, but they're never going to even be considered because by the time someone is at scale, they either have built this somewhere else or they went broke trying.Ben: Yep, absolutely. And we might just make the point there that while you can get discounts on data transfer, you really can't—or it's very rare—to get discounts on data processing for the NAT Gateway. So, any kind of savings you can get on data transfer would apply to a NAT instance solution, you know, saving you four-and-a-half cents per gigabyte inbound and outbound over the NAT Gateway equivalent solution. So, you're paying a lot for the benefit of a fully-managed service there. Very robust, nicely engineered fully-managed service as we've already acknowledged, but an extremely expensive solution for what it is, which is really just a proxy in the end. It doesn't add any value to you.Corey: The only way to make that more expensive would be to route it through something like Splunk or whatnot. And Splunk does an awful lot for what they charge per gigabyte, but it just feels like it's rent-seeking in some of the worst ways possible. And what I love about this is that you've solved the problem in a way that is open-source, you have already released it in Terraform code. I think one of the first to-dos on this for someone is going to be, okay now also make it CloudFormation and also make it CDK so you can drop it in however you want.And anyone can use this. I think the biggest mistake people might make in glancing at this is well, I'm looking at the hourly charge for the NAT Gateways and that's 32-and-a-half bucks a month and the instances that you recommend are hundreds of dollars a month for the big network-optimized stuff. Yeah, if you care about the hourly rate of either of those two things, this is not for you. That is not the problem that it solves. If you're an independent learner annoyed about the $30 charge you got for a Managed NAT Gateway, don't do this. This will only add to your billing concerns.Where it really shines is once you're at, I would say probably about ten terabytes a month, give or take, in Managed NAT Gateway data processing is where it starts to consider this. The breakeven is around six or so but there is value to not having to think about things. Once you get to that level of spend, though it's worth devoting a little bit of infrastructure time to something like this.Ben: Yeah, that's effectively correct. The total cost of running the solution, like, all-in, there's eight Elastic IPs, four NAT Gateways, if you're—say you're four zones; could be less if you're in fewer zones—like, n NAT Gateways, n NAT instances, depending on how many zones you're in, and I think that's about it. And I said right in the documentation, if any of those baseline fees are a material number for your use case, then this is probably not the right solution. Because we're talking about saving thousands of dollars. Any of these small numbers for NAT Gateway hourly costs, NAT instance hourly costs, that shouldn't be a factor, basically.Corey: Yeah, it's like when I used to worry about costing my customers a few tens of dollars in Cost Explorer or CloudWatch or request fees against S3 for their Cost and Usage Reports. It's yeah, that does actually have a cost, there's no real way around it, but look at the savings they're realizing by going through that. Yeah, they're not going to come back and complaining about their five-figure consulting engagement costing an additional $25 in AWS charges and then lowering it by a third. So, there's definitely a difference as far as how those things tend to be perceived. But it's easy to miss the big stuff when chasing after the little stuff like that.This is part of the problem I have with an awful lot of cost tooling out there. They completely ignore cost components like this and focus only on the things that are easy to query via API, of, oh, we're going to cost-optimize your Kubernetes cluster when they think about compute and RAM. And, okay, that's great, but you're completely ignoring all the data transfer because there's still no great way to get at that programmatically. And it really is missing the forest for the trees.Ben: I think this is key to any cost reduction project or program that you're undertaking. When you look at a bill, look for the biggest spend items first and work your way down from there, just because of the impact you can have. And that's exactly what I did in this project. I saw that ‘EC2 Other' slash NAT Gateway was the big item and I started brainstorming ways that we could go about addressing that. And now I have my next targets in mind now that we've reduced this cost to effectively… nothing, extremely low compared to what it was, we have other new line items on our bill that we can start optimizing. But in any cost project, start with the big things.Corey: You have come a long way around to answer a question I get asked a lot, which is, “How do I become a cloud economist?” And my answer is, you don't. It's something that happens to you. And it appears to be happening to you, too. My favorite part about the solution that you built, incidentally, is that it is being released under the auspices of your employer, Chime Financial, which is immune to being acquired by Amazon just to kill this thing and shut it up.Because Amazon already has something shitty called Chime. They don't need to wind up launching something else or acquiring something else and ruining it because they have a Slack competitor of sorts called Amazon Chime. There's no way they could acquire you [unintelligible 00:27:45] going to get lost in the hallways.Ben: Well, I have confidence that Chime will be a good steward of the project. Chime's goal and mission as a company is to help everyone achieve financial peace of mind and we take that really seriously. We even apply it to ourselves and that was kind of the impetus behind developing this in the first place. You mentioned earlier we have Terraform support already and you're exactly right. I'd love to have CDK, CloudFormation, Pulumi supports, and other kinds of contributions are more than welcome from the community.So, if anybody feels like participating, if they see a feature that's missing, let's make this project the best that it can be. I suspect we can save many companies, hundreds of thousands or millions of dollars. And this really feels like the right direction to go in.Corey: This is easily a multi-billion dollar savings opportunity, globally.Ben: That's huge. I would be flabbergasted if that was the outcome of this.Corey: The hardest part is reaching these people and getting them on board with the idea of handling this. And again, I think there's a lot of opportunity for the project to evolve in the sense of different settings depending upon risk tolerance. I can easily see a scenario where in the event of a disruption to the NAT instance, it fails over to the Managed NAT Gateway, but fail back becomes manual so you don't have a flapping route table back and forth or a [hold 00:29:05] downtime or something like that. Because again, in that scenario, the failure mode is just well, you're paying four-and-a-half cents per gigabyte for a while until you wind up figuring out what's going on as opposed to the failure mode of you wind up disrupting connections on an ongoing basis, and for some workloads, that's not tenable. This is absolutely, for the common case, the right path forward.Ben: Absolutely. I think it's an enterprise-grade solution and the more knobs and dials that we add to tweak to make it more robust or adaptable to different kinds of use cases, the best outcome here would actually be that the entire solution becomes irrelevant because AWS fixes the NAT Gateway pricing. If that happens, I will consider the project a great success.Corey: I will be doing backflips like you wouldn't believe. I would sing their praises day in, day out. I'm not saying reduce it to nothing, even. I'm not saying it adds no value. I would change the way that it's priced because honestly, the fact that I can run an EC2 instance and be charged $0 on a per-gigabyte basis, yeah, I would pay a premium on an hourly charge based upon traffic volumes, but don't meter per gigabyte. That's where it breaks down.Ben: Absolutely. And why is it additive to data transfer, also? Like, I remember first starting to use VPC when it was launched and reading about the NAT instance requirement and thinking, “Wait a minute. I have to pay this extra management and hourly fee just so my private hosts could reach the internet? That seems kind of janky.”And Amazon established a norm here because Azure and GCP both have their own equivalent of this now. This is a business choice. This is not a technical choice. They could just run this under the hood and not charge anybody for it or build in the cost and it wouldn't be this thing we have to think about.Corey: I almost hate to say it, but Oracle Cloud does, for free.Ben: Do they?Corey: It can be done. This is a business decision. It is not a technical capability issue where well, it does incur cost to run these things. I understand that and I'm not asking for things for free. I very rarely say that this is overpriced when I'm talking about AWS billing issues. I'm talking about it being unpredictable, I'm talking about it being impossible to see in advance, but the fact that it costs too much money is rarely my complaint. In this case, it costs too much money. Make it cost less.Ben: If I'm not mistaken. GCPs equivalent solution is the exact same price. It's also four-and-a-half cents per gigabyte. So, that shows you that there's business games being played here. Like, Amazon could get ahead and do right by the customer by dropping this to a much more reasonable price.Corey: I really want to thank you both for taking the time to speak with me and building this glorious, glorious thing. Where can we find it? And where can we find you?Ben: alternat.cloud is going to be the place to visit. It's on Chime's GitHub, which will be released by the time this podcast comes out. As for me, if you want to connect, I'm on Twitter. @iamthewhaley is my handle. And of course, I'm on LinkedIn.Corey: Links to all of that will be in the podcast notes. Ben, thank you so much for your time and your hard work.Ben: This was fun. Thanks, Corey.Corey: Ben Whaley, staff software engineer at Chime Financial, and AWS Community Hero. I'm Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with an angry rant of a comment that I will charge you not only four-and-a-half cents per word to read, but four-and-a-half cents to reply because I am experimenting myself with being a rent-seeking schmuck.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.
About AllenAllen is a cloud architect at Tyler Technologies. He helps modernize government software by creating secure, highly scalable, and fault-tolerant serverless applications.Allen publishes content regularly about serverless concepts and design on his blog - Ready, Set Cloud!Links Referenced: Ready, Set, Cloud blog: https://readysetcloud.io Tyler Technologies: https://www.tylertech.com/ Twitter: https://twitter.com/allenheltondev Linked: https://www.linkedin.com/in/allenheltondev/ TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by our friends at AWS AppConfig. Engineers love to solve, and occasionally create, problems. But not when it's an on-call fire-drill at 4 in the morning. Software problems should drive innovation and collaboration, NOT stress, and sleeplessness, and threats of violence. That's why so many developers are realizing the value of AWS AppConfig Feature Flags. Feature Flags let developers push code to production, but hide that that feature from customers so that the developers can release their feature when it's ready. This practice allows for safe, fast, and convenient software development. You can seamlessly incorporate AppConfig Feature Flags into your AWS or cloud environment and ship your Features with excitement, not trepidation and fear. To get started, go to snark.cloud/appconfig. That's snark.cloud/appconfig.Corey: I come bearing ill tidings. Developers are responsible for more than ever these days. Not just the code that they write, but also the containers and the cloud infrastructure that their apps run on. Because serverless means it's still somebody's problem. And a big part of that responsibility is app security from code to cloud. And that's where our friend Snyk comes in. Snyk is a frictionless security platform that meets developers where they are - Finding and fixing vulnerabilities right from the CLI, IDEs, Repos, and Pipelines. Snyk integrates seamlessly with AWS offerings like code pipeline, EKS, ECR, and more! As well as things you're actually likely to be using. Deploy on AWS, secure with Snyk. Learn more at Snyk.co/scream That's S-N-Y-K.co/screamCorey: Welcome to Screaming in the Cloud. I'm Corey Quinn. Every once in a while I wind up stumbling into corners of the internet that I previously had not traveled. Somewhat recently, I wound up having that delightful experience again by discovering readysetcloud.io, which has a whole series of, I guess some people might call it thought leadership, I'm going to call it instead how I view it, which is just amazing opinion pieces on the context of serverless, mixed with APIs, mixed with some prognostications about the future.Allen Helton by day is a cloud architect at Tyler Technologies, but that's not how I encountered you. First off, Allen, thank you for joining me.Allen: Thank you, Corey. Happy to be here.Corey: I was originally pointed towards your work by folks in the AWS Community Builder program, of which we both participate from time to time, and it's one of those, “Oh, wow, this is amazing. I really wish I'd discovered some of this sooner.” And every time I look through your back catalog, and I click on a new post, I see things that are either I've really agree with this or I can't stand this opinion, I want to fight about it, but more often than not, it's one of those recurring moments that I love: “Damn, I wish I had written something like this.” So first, you're absolutely killing it on the content front.Allen: Thank you, Corey, I appreciate that. The content that I make is really about the stuff that I'm doing at work. It's stuff that I'm passionate about, stuff that I'd spend a decent amount of time on, and really the most important thing about it for me, is it's stuff that I'm learning and forming opinions on and wants to share with others.Corey: I have to say, when I saw that you were—oh, your Tyler Technologies, which sounds for all the world like, oh, it's a relatively small consultancy run by some guy presumably named Tyler, and you know, it's a petite team of maybe 20, 30 people on the outside. Yeah, then I realized, wait a minute, that's not entirely true. For example, for starters, you're publicly traded. And okay, that does change things a little bit. First off, who are you people? Secondly, what do you do? And third, why have I never heard of you folks, until now?Allen: Tyler is the largest company that focuses completely on the public sector. We have divisions and products for pretty much everything that you can imagine that's in the public sector. We have software for schools, software for tax and appraisal, we have software for police officers, for courts, everything you can think of that runs the government can and a lot of times is run on Tyler software. We've been around for decades building our expertise in the domain, and the reason you probably haven't heard about us is because you might not have ever been in trouble with the law before. If you [laugh] if you have been—Corey: No, no, I learned very early on in the course of my life—which will come as a surprise to absolutely no one who spent more than 30 seconds with me—that I have remarkably little filter and if ten kids were the ones doing something wrong, I'm the one that gets caught. So, I spent a lot of time in the principal's office, so this taught me to keep my nose clean. I'm one of those squeaky-clean types, just because I was always terrified of getting punished because I knew I would get caught. I'm not saying this is the right way to go through life necessarily, but it did have the side benefit of, no, I don't really engage with law enforcement going throughout the course of my life.Allen: That's good. That's good. But one exposure that a lot of people get to Tyler is if you look at the bottom of your next traffic ticket, it'll probably say Tyler Technologies on the bottom there.Corey: Oh, so you're really popular in certain circles, I'd imagine?Allen: Super popular. Yes, yes. And of course, you get all the benefits of writing that code that says ‘if defendant equals Allen Helton then return.'Corey: I like that. You get to have the exception cases built in that no one's ever going to wind up looking into.Allen: That's right. Yes.Corey: The idea of what you're doing makes an awful lot of sense. There's a tremendous need for a wide variety of technical assistance in the public sector. What surprises me, although I guess it probably shouldn't, is how much of your content is aimed at serverless technologies and API design, which to my way of thinking, isn't really something that public sector has done a lot with. Clearly I'm wrong.Allen: Historically, you're not wrong. There's an old saying that government tends to run about ten years behind on technology. Not just technology, but all over the board and runs about ten years behind. And until recently, that's really been true. There was a case last year, a situation last year where one of the state governments—I don't remember which one it was—but they were having a crisis because they couldn't find any COBOL developers to come in and maintain their software that runs the state.And it's COBOL; you're not going to find a whole lot of people that have that skill. A lot of those people are retiring out. And what's happening is that we're getting new people sitting in positions of power and government that want innovation. They know about the cloud and they want to be able to integrate with systems quickly and easily, have little to no onboarding time. You know, there are people in power that have grown up with technology and understand that, well, with everything else, I can be up and running in five or ten minutes. I cannot do this with the software I'm consuming now.Corey: My opinion on it is admittedly conflicted because on the one hand, yeah, I don't think that governments should be running on COBOL software that runs on mainframes that haven't been supported in 25 years. Conversely, I also don't necessarily want them being run like a seed series startup, where, “Well, I wrote this code last night, and it's awesome, so off I go to production with it.” Because I can decide not to do business anymore with Twitter for Pets, and I could go on to something else, like PetFlicks, or whatever it is I choose to use. I can't easily opt out of my government. The decisions that they make stick and that is going to have a meaningful impact on my life and everyone else's life who is subject to their jurisdiction. So, I guess I don't really know where I believe the proper, I guess, pace of technological adoption should be for governments. Curious to get your thoughts on this.Allen: Well, you certainly don't want anything that's bleeding edge. That's one of the things that we kind of draw fine lines around. Because when we're dealing with government software, we're dealing with, usually, critically sensitive information. It's not medical records, but it's your criminal record, and it's things like your social security number, it's things that you can't have leaking out under any circumstances. So, the things that we're building on are things that have proven out to be secure and have best practices around security, uptime, reliability, and in a lot of cases as well, and maintainability. You know, if there are issues, then let's try to get those turned around as quickly as we can because we don't want to have any sort of downtime from the software side versus the software vendor side.Corey: I want to pivot a little bit to some of the content you've put out because an awful lot of it seems to be, I think I'll call it variations on a theme. For example, I just read some recent titles, and to illustrate my point, “Going API First: Your First 30 Days,” “Solutions Architect Tips how to Design Applications for Growth,” “3 Things to Know Before Building A Multi-Tenant Serverless App.” And the common thread that I see running through all of these things are these are things that you tend to have extraordinarily strong and vocal opinions about only after dismissing all of them the first time and slapping something together, and then sort of being forced to live with the consequences of the choices that you've made, in some cases you didn't realize you were making at the time. Are you one of those folks that has the wisdom to see what's coming down the road, or did you do what the rest of us do and basically learn all this stuff by getting it hilariously wrong and having to careen into rebound situations as a result?Allen: [laugh]. I love that question. I would like to say now, I feel like I have the vision to see something like that coming. Historically, no, not at all. Let me talk a little bit about how I got to where I am because that will shed a lot of context on that question.A few years ago, I was put into a position at Tyler that said, “Hey, go figure out this cloud thing.” Let's figure out what we need to do to move into the cloud safely, securely, quickly, all that rigmarole. And so, I did. I got to hand-select team of engineers from people that I worked with at Tyler over the past few years, and we were basically given free rein to learn. We were an R&D team, a hundred percent R&D, for about a year's worth of time, where we were learning about cloud concepts and theory and building little proof of concepts.CI/CD, serverless, APIs, multi-tenancy, a whole bunch of different stuff. NoSQL was another one of the things that we had to learn. And after that year of R&D, we were told, “Okay, now go do something with that. Go build this application.” And we did, building on our theory our cursory theory knowledge. And we get pretty close to go live, and then the business says, “What do you do in this scenario? What do you do in that scenario? What do you do here?”Corey: “I update my resume and go work somewhere else. Where's the hard part here?”Allen: [laugh].Corey: Turns out, that's not a convincing answer.Allen: Right. So, we moved quickly. And then I wouldn't say we backpedaled, but we hardened for a long time before the—prior to the go-live, with the lessons that we've learned with the eyes of Tyler, the mature enterprise company, saying, “These are the things that you have to make sure that you take into consideration in an actual production application.” One of the things that I always pushed—I was a manager for a few years of all these cloud teams—I always push do it; do it right; do it better. Right?It's kind of like crawl, walk, run. And if you follow my writing from the beginning, just looking at the titles and reading them, kind of like what you were doing, Corey, you'll see that very much. You'll see how I talk about CI/CD, you'll see me how I talk about authorization, you'll see me how I talk about multi-tenancy. And I kind of go in waves where maybe a year passes and you see my content revisit some of the topics that I've done in the past. And they're like, “No, no, no, don't do what I said before. It's not right.”Corey: The problem when I'm writing all of these things that I do, for example, my entire newsletter publication pipeline is built on a giant morass of Lambda functions and API Gateways. It's microservices-driven—kind of—and each microservice is built, almost always, with a different framework. Lately, all the new stuff is CDK. I started off with the serverless framework. There are a few other things here and there.And it's like going architecting, back in time as I have to make updates to these things from time to time. And it's the problem with having done all that myself is that I already know the answer to, “What fool designed this?” It's, well, you're basically watching me learn what I was, doing bit by bit. I'm starting to believe that the right answer on some level, is to build an inherent shelf-life into some of these things. Great, in five years, you're going to come back and re-architect it now that you know how this stuff actually works rather than patching together 15 blog posts by different authors, not all of whom are talking about the same thing and hoping for the best.Allen: Yep. That's one of the things that I really like about serverless, I view that as a giant pro of doing Serverless is that when we revisit with the lessons learned, we don't have to refactor everything at once like if it was just a big, you know, MVC controller out there in the sky. We can refactor one Lambda function at a time if now we're using a new version of the AWS SDK, or we've learned about a new best practice that needs to go in place. It's a, “While you're in there, tidy up, please,” kind of deal.Corey: I know that the DynamoDB fanatics will absolutely murder me over this one, but one of the reasons that I have multiple Dynamo tables that contain, effectively, variations on the exact same data, is because I want to have the dependency between the two different microservices be the API, not, “Oh, and under the hood, it's expecting this exact same data structure all the time.” But it just felt like that was the wrong direction to go in. That is the justification I use for myself why I run multiple DynamoDB tables that [laugh] have the same content. Where do you fall on the idea of data store separation?Allen: I'm a big single table design person myself, I really like the idea of being able to store everything in the same table and being able to create queries that can return me multiple different types of entity with one lookup. Now, that being said, one of the issues that we ran into, or one of the ambiguous areas when we were getting started with serverless was, what does single table design mean when you're talking about microservices? We were wondering does single table mean one DynamoDB table for an entire application that's composed of 15 microservices? Or is it one table per microservice? And that was ultimately what we ended up going with is a table per microservice. Even if multiple microservices are pushed into the same AWS account, we're still building that logical construct of a microservice and one table that houses similar entities in the same domain.Corey: So, something I wish that every service team at AWS would do as a part of their design is draw the architecture of an application that you're planning to build. Great, now assume that every single resource on that architecture diagram lives in its own distinct AWS account because somewhere in some customer, there's going to be an account boundary at every interconnection point along the way. And so, many services don't do that where it's, “Oh, that thing and the other thing has to be in the same account.” So, people have to write their own integration shims, and it makes doing the right thing of putting different services into distinct bounded AWS accounts for security or compliance reasons way harder than I feel like it needs to be.Allen: [laugh]. Totally agree with you on that one. That's one of the things that I feel like I'm still learning about is the account-level isolation. I'm still kind of early on, personally, with my opinions in how we're structuring things right now, but I'm very much of a like opinion that deploying multiple things into the same account is going to make it too easy to do something that you shouldn't. And I just try not to inherently trust people, in the sense that, “Oh, this is easy. I'm just going to cross that boundary real quick.”Corey: For me, it's also come down to security risk exposure. Like my lasttweetinaws.com Twitter shitposting thread client lives in a distinct AWS account that is separate from the AWS account that has all of our client billing data that lives within it. The idea being that if you find a way to compromise my public-facing Twitter client, great, the blast radius should be constrained to, “Yay, now you can, I don't know, spin up some cryptocurrency mining in my AWS account and I get to look like a fool when I beg AWS for forgiveness.”But that should be the end of it. It shouldn't be a security incident because I should not have the credit card numbers living right next to the funny internet web thing. That sort of flies in the face of the original guidance that AWS gave at launch. And right around 2008-era, best practices were one customer, one AWS account. And then by 2012, they had changed their perspective, but once you've made a decision to build multiple services in a single account, unwinding and unpacking that becomes an incredibly burdensome thing. It's about the equivalent of doing a cloud migration, in some ways.Allen: We went through that. We started off building one application with the intent that it was going to be a siloed application, a one-off, essentially. And about a year into it, it's one of those moments of, “Oh, no. What we're building is not actually a one-off. It's a piece to a much larger puzzle.”And we had a whole bunch of—unfortunately—tightly coupled things that were in there that we're assuming that resources were going to be in the same AWS account. So, we ended up—how long—I think we took probably two months, which in the grand scheme of things isn't that long, but two months, kind of unwinding the pieces and decoupling what was possible at the time into multiple AWS accounts, kind of, segmented by domain, essentially. But that's hard. AWS puts it, you know, it's those one-way door decisions. I think this one was a two-way door, but it locked and you could kind of jimmy the lock on the way back out.Corey: And you could buzz someone from the lobby to let you back in. Yeah, the biggest problem is not necessarily the one-way door decisions. It's the one-way door decisions that you don't realize you're passing through at the time that you do them. Which, of course, brings us to a topic near and dear to your heart—and I only recently started have opinions on this myself—and that is the proper design of APIs, which I'm sure will incense absolutely no one who's listening to this. Like, my opinions on APIs start with well, probably REST is the right answer in this day and age. I had people, like, “Well, I don't know, GraphQL is pretty awesome.” Like, “Oh, I'm thinking SOAP,” and people look at me like I'm a monster from the Black Lagoon of centuries past in XML-land. So, my particular brand of strangeness side, what do you see that people are doing in the world of API design that is the, I guess, most common or easy to make mistakes that you really wish they would stop doing?Allen: If I could boil it down to one word, fundamentalism. Let me unpack that for you.Corey: Oh, please, absolutely want to get a definition on that one.Allen: [laugh]. I approach API design from a developer experience point of view: how easy is it for both internal and external integrators to consume and satisfy the business processes that they want to accomplish? And a lot of times, REST guidelines, you know, it's all about entity basis, you know, drill into the appropriate entities and name your endpoints with nouns, not verbs. I'm actually very much onto that one.But something that you could easily do, let's say you have a business process that given a fundamentally correct RESTful API design takes ten API calls to satisfy. You could, in theory, boil that down to maybe three well-designed endpoints that aren't, quote-unquote, “RESTful,” that make that developer experience significantly easier. And if you were a fundamentalist, that option is not even on the table, but thinking about it pragmatically from a developer experience point of view, that might be the better call. So, that's one of the things that, I know feels like a hot take. Every time I say it, I get a little bit of flack for it, but don't be a fundamentalist when it comes to your API designs. Do something that makes it easier while staying in the guidelines to do what you want.Corey: For me the problem that I've kept smacking into with API design, and it honestly—let me be very clear on this—my first real exposure to API design rather than API consumer—which of course, I complain about constantly, especially in the context of the AWS inconsistent APIs between services—was when I'm building something out, and I'm reading the documentation for API Gateway, and oh, this is how you wind up having this stage linked to this thing, and here's the endpoint. And okay, great, so I would just populate—build out a structure or a schema that has the positional parameters I want to use as variables in my function. And that's awesome. And then I realized, “Oh, I might want to call this a different way. Aw, crap.” And sometimes it's easy; you just add a different endpoint. Other times, I have to significantly rethink things. And I can't shake the feeling that this is an entire discipline that exists that I just haven't had a whole lot of exposure to previously.Allen: Yeah, I believe that. One of the things that you could tie a metaphor to for what I'm saying and kind of what you're saying, is AWS SAM, the Serverless Application Model, all it does is basically macros CloudFormation resources. It's just a transform from a template into CloudFormation. CDK does same thing. But what the developers of SAM have done is they've recognized these business processes that people do regularly, and they've made these incredibly easy ways to satisfy those business processes and tie them all together, right?If I want to have a Lambda function that is backed behind a endpoint, an API endpoint, I just have to add four or five lines of YAML or JSON that says, “This is the event trigger, here's the route, here's the API.” And then it goes and does four, five, six different things. Now, there's some engineers that don't like that because sometimes that feels like magic. Sometimes a little bit magic is okay.Corey: This episode is sponsored in part by our friends at Sysdig. Sysdig secures your cloud from source to run. They believe, as do I, that DevOps and security are inextricably linked. If you wanna learn more about how they view this, check out their blog, it's definitely worth the read. To learn more about how they are absolutely getting it right from where I sit, visit Sysdig.com and tell them that I sent you. That's S Y S D I G.com. And my thanks to them for their continued support of this ridiculous nonsense.Corey: I feel like one of the benefits I've had with the vast majority of APIs that I've built is that because this is all relatively small-scale stuff for what amounts to basically shitposting for the sake of entertainment, I'm really the only consumer of an awful lot of these things. So, I get frustrated when I have to backtrack and make changes and teach other microservices to talk to this thing that has now changed. And it's frustrating, but I have the capacity to do that. It's just work for a period of time. I feel like that equation completely shifts when you have published this and it is now out in the world, and it's not just users, but in many cases paying customers where you can't really make those changes without significant notice, and every time you do you're creating work for those customers, so you have to be a lot more judicious about it.Allen: Oh, yeah. There is a whole lot of governance and practice that goes into production-level APIs that people integrate with. You know, they say once you push something out the door into production that you're going to support it forever. I don't disagree with that. That seems like something that a lot of people don't understand.And that's one of the reasons why I push API-first development so hard in all the content that I write is because you need to be intentional about what you're letting out the door. You need to go in and work, not just with the developers, but your product people and your analysts to say, what does this absolutely need to do, and what does it need to do in the future? And you take those things, and you work with analysts who want specifics, you work with the engineers to actually build it out. And you're very intentional about what goes out the door that first time because once it goes out with a mistake, you're either going to version it immediately or you're going to make some people very unhappy when you make a breaking change to something that they immediately started consuming.Corey: It absolutely feels like that's one of those things that AWS gets astonishingly right. I mean, I had the privilege of interviewing, at the time, Jeff Barr and then Ariel Kelman, who was their head of marketing, to basically debunk a bunch of old myths. And one thing that they started talking about extensively was the idea that an API is fundamentally a promise to your customers. And when you make a promise, you'd better damn well intend on keeping it. It's why API deprecations from AWS are effectively unique whenever something happens.It's the, this is a singular moment in time when they turn off a service or degrade old functionality in favor of new. They can add to it, they can launch a V2 of something and then start to wean people off by calling the old one classic or whatnot, but if I built something on AWS in 2008 and I wound up sleeping until today, and go and try and do the exact same thing and deploy it now, it will almost certainly work exactly as it did back then. Sure, reliability is going to be a lot better and there's a crap ton of features and whatnot that I'm not taking advantage of, but that fundamental ability to do that is awesome. Conversely, it feels like Google Cloud likes to change around a lot of their API stories almost constantly. And it's unplanned work that frustrates the heck out of me when I'm trying to build something stable and lasting on top of it.Allen: I think it goes to show the maturity of these companies as API companies versus just vendors. It's one of the things that I think AWS does [laugh]—Corey: You see the similar dichotomy with Microsoft and Apple. Microsoft's new versions of Windows generally still have functionalities in them to support stuff that was written in the '90s for a few use cases, whereas Apple's like, “Oh, your computer's more than 18-months old? Have you tried throwing it away and buying a new one? And oh, it's a new version of Mac OS, so yeah, maybe the last one would get security updates for a year and then get with the times.” And I can't shake the feeling that the correct answer is in some way, both of those, depending upon who your customer is and what it is you're trying to achieve.If Microsoft adopted the Apple approach, their customers would mutiny, and rightfully so; the expectation has been set for decades that isn't what happens. Conversely, if Apple decided now we're going to support this version of Mac OS in perpetuity, I don't think a lot of their application developers wouldn't quite know what to make of that.Allen: Yeah. I think it also comes from a standpoint of you better make it worth their while if you're going to move their cheese. I'm not a Mac user myself, but from what I hear for Mac users—and this could be rose-colored glasses—but is that their stuff works phenomenally well. You know, when a new thing comes out—Corey: Until it doesn't, absolutely. It's—whenever I say things like that on this show, I get letters. And it's, “Oh, yeah, really? They'll come up with something that is a colossal pain in the ass on Mac.” Like, yeah, “Try building a system-wide mute key.”It's yeah, that's just a hotkey away on windows and here in Mac land. It's, “But it makes such beautiful sounds. Why would you want them to be quiet?” And it's, yeah, it becomes this back-and-forth dichotomy there. And you can even explain it to iPhones as well and the Android ecosystem where it's, oh, you're going to support the last couple of versions of iOS.Well, as a developer, I don't want to do that. And Apple's position is, “Okay, great.” Almost half of the mobile users on the planet will be upgrading because they're in the ecosystem. Do you want us to be able to sell things those people are not? And they're at a point of scale where they get to dictate those terms.On some level, there are benefits to it and others, it is intensely frustrating. I don't know what the right answer is on the level of permanence on that level of platform. I only have slightly better ideas around the position of APIs. I will say that when AWS deprecates something, they reach out individually to affected customers, on some level, and invariably, when they say, “This is going to be deprecated as of August 31,” or whenever it is, yeah, it is going to slip at least twice in almost every case, just because they're not going to turn off a service that is revenue-bearing or critical-load-bearing for customers without massive amounts of notice and outreach, and in some cases according to rumor, having engineers reach out to help restructure things so it's not as big of a burden on customers. That's a level of customer focus that I don't think most other companies are capable of matching.Allen: I think that comes with the size and the history of Amazon. And one of the things that they're doing right now, we've used Amazon Cloud Cams for years, in my house. We use them as baby monitors. And they—Corey: Yea, I saw this I did something very similar with Nest. They didn't have the Cloud Cam at the right time that I was looking at it. And they just announced that they're going to be deprecating. They're withdrawing them for sale. They're not going to support them anymore. Which, oh at Amazon—we're not offering this anymore. But you tell the story; what are they offering existing customers?Allen: Yeah, so slightly upset about it because I like my Cloud Cams and I don't want to have to take them off the wall or wherever they are to replace them with something else. But what they're doing is, you know, they gave me—or they gave all the customers about eight months head start. I think they're going to be taking them offline around Thanksgiving this year, just mid-November. And what they said is as compensation for you, we're going to send you a Blink Cam—a Blink Mini—for every Cloud Cam that you have in use, and then we are going to gift you a year subscription to the Pro for Blink.Corey: That's very reasonable for things that were bought years ago. Meanwhile, I feel like not to be unkind or uncharitable here, but I use Nest Cams. And that's a Google product. I half expected if they ever get deprecated, I'll find out because Google just turns it off in the middle of the night—Allen: [laugh].Corey: —and I wake up and have to read a blog post somewhere that they put an update on Nest Cams, the same way they killed Google Reader once upon a time. That's slightly unfair, but the fact that joke even lands does say a lot about Google's reputation in this space.Allen: For sure.Corey: One last topic I want to talk with you about before we call it a show is that at the time of this recording, you recently had a blog post titled, “What does the Future Hold for Serverless?” Summarize that for me. Where do you see this serverless movement—if you'll forgive the term—going?Allen: So, I'm going to start at the end. I'm going to work back a little bit on what needs to happen for us to get there. I have a feeling that in the future—I'm going to be vague about how far in the future this is—that we'll finally have a satisfied promise of all you're going to write in the future is business logic. And what does that mean? I think what can end up happening, given the right focus, the right companies, the right feedback, at the right time, is we can write code as developers and have that get pushed up into the cloud.And a phrase that I know Jeremy Daly likes to say ‘infrastructure from code,' where it provisions resources in the cloud for you based on your use case. I've developed an application and it gets pushed up in the cloud at the time of deploying it, optimized resource allocation. Over time, what will happen—with my future vision—is when you get production traffic going through, maybe it's spiky, maybe it's consistently at a scale that outperforms the resources that it originally provisioned. We can have monitoring tools that analyze that and pick that out, find the anomalies, find the standard patterns, and adjust that infrastructure that it deployed for you automatically, where it's based on your production traffic for what it created, optimizes it for you. Which is something that you can't do on an initial deployment right now. You can put what looks best on paper, but once you actually get traffic through your application, you realize that, you know, what was on paper might not be correct.Corey: You ever noticed that whiteboard diagrams never show the reality, and they're always aspirational, and they miss certain parts? And I used to think that this was the symptom I had from working at small, scrappy companies because you know what, those big tech companies, everything they build is amazing and awesome. I know it because I've seen their conference talks. But I've been a consultant long enough now, and for a number of those companies, to realize that nope, everyone's infrastructure is basically a trash fire at any given point in time. And it works almost in spite of itself, rather than because of it.There is no golden path where everything is shiny, new and beautiful. And that, honestly, I got to say, it was really [laugh] depressing when I first discovered it. Like, oh, God, even these really smart people who are so intelligent they have to have extra brain packs bolted to their chests don't have the magic answer to all of this. The rest of us are just screwed, then. But we find ways to make it work.Allen: Yep. There's a quote, I wish I remembered who said it, but it was a military quote where, “No battle plan survives impact with the enemy—first contact with the enemy.” It's kind of that way with infrastructure diagrams. We can draw it out however we want and then you turn it on in production. It's like, “Oh, no. That's not right.”Corey: I want to mix the metaphors there and say, yeah, no architecture survives your first fight with a customer. Like, “Great, I don't think that's quite what they're trying to say.” It's like, “What, you don't attack your customers? Pfft, what's your customer service line look like?” Yeah, it's… I think you're onto something.I think that inherently everything beyond the V1 design of almost anything is an emergent property where this is what we learned about it by running it and putting traffic through it and finding these problems, and here's how it wound up evolving to account for that.Allen: I agree. I don't have anything to add on that.Corey: [laugh]. Fair enough. I really want to thank you for taking so much time out of your day to talk about how you view these things. If people want to learn more, where is the best place to find you?Allen: Twitter is probably the best place to find me: @AllenHeltonDev. I have that username on all the major social platforms, so if you want to find me on LinkedIn, same thing: AllenHeltonDev. My blog is always open as well, if you have any feedback you'd like to give there: readysetcloud.io.Corey: And we will, of course, put links to that in the show notes. Thanks again for spending so much time talking to me. I really appreciate it.Allen: Yeah, this was fun. This was a lot of fun. I love talking shop.Corey: It shows. And it's nice to talk about things I don't spend enough time thinking about. Allen Helton, cloud architect at Tyler Technologies. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with an angry comment that I will reject because it was not written in valid XML.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.
About MattMatt is a Sr. Architect in Belfast, an AWS DevTools Hero, Serverless Architect, Author and conference speaker. He is focused on creating the right environment for empowered teams to rapidly deliver business value in a well-architected, sustainable and serverless-first way.You can usually find him sharing reusable, well architected, serverless patterns over at cdkpatterns.com or behind the scenes bringing CDK Day to life.Links Referenced: Previous guest appearance: https://www.lastweekinaws.com/podcast/screaming-in-the-cloud/slinging-cdk-knowledge-with-matt-coulter/ The CDK Book: https://thecdkbook.com/ Twitter: https://twitter.com/NIDeveloper TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. One of the best parts about, well I guess being me, is that I can hold opinions that are… well, I'm going to be polite and call them incendiary, and that's great because I usually like to back them in data. But what happens when things change? What happens when I learn new things?Well, do I hold on to that original opinion with two hands at a death grip or do I admit that I was wrong in my initial opinion about something? Let's find out. My guest today returns from earlier this year. Matt Coulter is a senior architect since he has been promoted at Liberty Mutual. Welcome back, and thanks for joining me.Matt: Yeah, thanks for inviting me back, especially to talk about this topic.Corey: Well, we spoke about it a fair bit at the beginning of the year. And if you're listening to this, and you haven't heard that show, it's not that necessary to go into; mostly it was me spouting uninformed opinions about the CDK—the Cloud Development Kit, for those who are unfamiliar—I think of it more or less as what if you could just structure your cloud resources using a programming language you claim to already know, but in practice, copy and paste from Stack Overflow like the rest of us? Matt, you probably have a better description of what the CDK is in practice.Matt: Yeah, so we like to say it's imperative code written in a declarative way, or declarative code written in an imperative way. Either way, it lets you write code that produces CloudFormation. So, it doesn't really matter what you write in your script; the point is, at the end of the day, you still have the CloudFormation template that comes out of it. So, the whole piece of it is that it's a developer experience, developer speed play, that if you're from a background that you're more used to writing a programming language than a YAML, you might actually enjoy using the CDK over writing straight CloudFormation or SAM.Corey: When I first kicked the tires on the CDK, my first initial obstacle—which I've struggled with in this industry for a bit—is that I'm just good enough of a programmer to get myself in trouble. Whenever I wind up having a problem that StackOverflow doesn't immediately shine a light on, my default solution is to resort to my weapon of choice, which is brute force. That sometimes works out, sometimes doesn't. And as I went through the CDK, a couple of times in service to a project that I'll explain shortly, I made a bunch of missteps with it. The first and most obvious one is that AWS claims publicly that it has support in a bunch of languages: .NET, Python, there's obviously TypeScript, there's Go support for it—I believe that went generally available—and I'm sure I'm missing one or two, I think? Aren't I?Matt: Yeah, it's: TypeScript, JavaScript, Python Java.Net, and Go. I think those are the currently supported languages.Corey: Java. That's the one that I keep forgetting. It's the block printing to the script that is basically Java cursive. The problem I run into, and this is true of most things in my experience, when a company says that we have deployed an SDK for all of the following languages, there is very clearly a first-class citizen language and then the rest that more or less drift along behind with varying degrees of fidelity. In my experience, when I tried it for the first time in Python, it was not a great experience for me.When I learned just enough JavaScript, and by extension TypeScript, to be dangerous, it worked a lot better. Or at least I could blame all the problems I ran into on my complete novice status when it comes to JavaScript and TypeScript at the time. Is that directionally aligned with what you've experienced, given that you work in a large company that uses this, and presumably, once you have more than, I don't know, two developers, you start to take on aspects of a polyglot shop no matter where you are, on some level?Matt: Yeah. So personally, I jump between Java, Python, and TypeScript whenever I'm writing projects. So, when it comes to the CDK, you'd assume I'd be using all three. I typically stick to TypeScript and that's just because personally, I've had the best experience using it. For anybody who doesn't know the way CDK works for all the languages, it's not that they have written a custom, like, SDK for each of these languages; it's a case of it uses a Node process underneath them and the language actually interacts with—it's like the compiled JavaScript version is basically what they all interact with.So, it means there are some limitations on what you can do in that language. I can't remember the full list, but it just means that it is native in all those languages, but there are certain features that you might be like, “Ah,” whereas, in TypeScript, you can just use all of TypeScript. And my first inclination was actually, I was using the Python one and I was having issues with some compiler errors and things that are just caused by that process. And it's something that talking in the cdk.dev Slack community—there is actually a very active—Corey: Which is wonderful, I will point out.Matt: [laugh]. Thank you. There is actually, like, an awesome Python community in there, but if you ask them, they would all ask for improvements to the language. So, personally if someone's new, I always recommend they start with TypeScript and then branch out as they learn the CDK so they can understand is this a me problem, or is this a problem caused by the implementation?Corey: From my perspective, I didn't do anything approaching that level of deep dive. I took a shortcut that I find has served me reasonably well in the course of my career, when I'm trying to do something in Python, and you pull up a tutorial—which I'm a big fan of reading experience reports, and blog posts, and here's how to get started—and they all have the same problem, which is step one, “Run npm install.” And that's “Hmm, you know, I don't recall that being a standard part of the Python tooling.” It's clearly designed and interpreted and contextualized through a lens of JavaScript. Let's remove that translation layer, let's remove any weird issues I'm going to have in that transpilation process, and just talk in the language it written in. Will this solve my problems? Oh, absolutely not, but it will remove a subset of them that I am certain to go blundering into like a small lost child trying to cross an eight-lane freeway.Matt: Yeah. I've heard a lot of people say the same thing. Because the CDK CLI is a Node process, you need it no matter what language you use. So, if they were distributing some kind of universal binary that just integrated with the languages, it would definitely solve a lot of people's issues with trying to combine languages at deploy time.Corey: One of the challenges that I've had as I go through the process of iterating on the project—but I guess I should probably describe it for those who have not been following along with my misadventures; I write blog posts about it from time to time because I need a toy problem to kick around sometimes because my consulting work is all advisory and I don't want to be a talking head-I have a Twitter client called lasttweetinaws.com. It's free; go and use it. It does all kinds of interesting things for authoring Twitter threads.And I wanted to deploy that to a bunch of different AWS regions, as it turns out, 20 or so at the moment. And that led to a lot of interesting projects and having to learn how to think about these things differently because no one sensible deploys an application simultaneously to what amounts to every AWS region, without canary testing, and having a phased rollout in the rest. But I'm reckless, and honestly, as said earlier, a bad programmer. So, that works out. And trying to find ways to make this all work and fit together led iteratively towards me discovering that the CDK was really kind of awesome for a lot of this.That said, there were definitely some fairly gnarly things I learned as I went through it, due in no small part to help I received from generous randos in the cdk.dev Slack team. And it's gotten to a point where it's working, and as an added bonus, I even mostly understand what he's doing, which is just kind of wild to me.Matt: It's one of those interesting things where because it's a programming language, you can use it out of the box the way it's designed to be used where you can just write your simple logic which generates your CloudFormation, or you can do whatever crazy logic you want to do on top of that to make your app work the way you want it to work. And providing you're not in a company like Liberty, where I'm going to do a code review, if no one's stopping you, you can do your crazy experiments. And if you understand that, it's good. But I do think something like the multi-region deploy, I mean, with CDK, if you'd have a construct, it takes in a variable that you can just say what the region is, so you can actually just write a for loop and pass it in, which does make things a lot easier than, I don't know, try to do it with a YAML, which you can pass in parameters, but you're going to get a lot more complicated a lot quicker.Corey: The approach that I took philosophically was I wrote everything in a region-agnostic way. And it would be instantiated and be told what region to run it in as an environment variable that CDK deploy was called. And then I just deploy 20 simultaneous stacks through GitHub Actions, which invoke custom runners that runs inside of a Lambda function. And that's just a relatively basic YAML file, thanks to the magic of GitHub Actions matrix jobs. So, it fires off 20 simultaneous processes and on every commit to the main branch, and then after about two-and-a-half minutes, it has been deployed globally everywhere and I get notified on anything that fails, which is always fun and exciting to learn those things.That has been, overall, just a really useful experiment and an experience because you're right, you could theoretically run this as a single CDK deploy and then wind up having an iterate through a list of regions. The challenge I have there is that unless I start getting into really convoluted asynchronous concurrency stuff, it feels like it'll just take forever. At two-and-a-half minutes a region times 20 regions, that's the better part of an hour on every deploy and no one's got that kind of patience. So, I wound up just parallelizing it a bit further up the stack. That said, I bet they are relatively straightforward ways, given the async is a big part of JavaScript, to do this simultaneously.Matt: One of the pieces of feedback I've seen about CDK is if you have multiple stacks in the same project, it'll deploy them one at a time. And that's just because it tries to understand the dependencies between the stacks and then it works out which one should go first. But a lot of people have said, “Well, I don't want that. If I have 20 stacks, I want all 20 to go at once the way you're saying.” And I have seen that people have been writing plugins to enable concurrent deploys with CDK out of the box. So, it may be something that it's not an out-of-the-box feature, but it might be something that you can pull in a community plug-in to actually make work.Corey: Most of my problems with it at this point are really problems with CloudFormation. CloudFormation does not support well, if at all, secure string parameters from the AWS Systems Manager parameter store, which is my default go-to for secret storage, and Secrets Manager is supported, but that also cost 40 cents a month per secret. And not for nothing, I don't really want to have all five secrets deployed to Secrets Manager in every region this thing is in. I don't really want to pay $20 a month for this basically free application, just to hold some secrets. So, I wound up talking to some folks in the Slack channel and what we came up with was, I have a centralized S3 bucket that has a JSON object that lives in there.It's only accessible from the deployment role, and it grabs that at deploy time and stuffs it into environment variables when it pushes these things out. That's the only stateful part of all of this. And it felt like that is, on some level, a pattern that a lot of people would benefit from if it had better native support. But the counterargument that if you're only deploying to one or two regions, then Secrets Manager is the right answer for a lot of this and it's not that big of a deal.Matt: Yeah. And it's another one of those things, if you're deploying in Liberty, we'll say, “Well, your secret is unencrypted at runtime, so you probably need a KMS key involved in that,” which as you know, the costs of KMS, it depends on if it's a personal solution or if it's something for, like, a Fortune 100 company. And if it's personal solution, I mean, what you're saying sounds great that it's IAM restricted in S3, and then that way only at deploy time can be read; it actually could be a custom construct that someone can build and publish out there to the construct library—or the construct hub, I should say.Corey: To be clear, the reason I'm okay with this, from a security perspective is one, this is in a dedicated AWS account. This is the only thing that lives in that account. And two, the only API credentials we're talking about are the application-specific credentials for this Twitter client when it winds up talking to the Twitter API. Basically, if you get access to these and are able to steal them and deploy somewhere else, you get no access to customer data, you get—or user data because this is not charge for anything—you get no access to things that have been sent out; all you get to do is submit tweets to Twitter and it'll have the string ‘Last Tweet in AWS' as your client, rather than whatever normal client you would use. It's not exactly what we'd call a high-value target because all the sensitive to a user data lives in local storage in their browser. It is fully stateless.Matt: Yeah, so this is what I mean. Like, it's the difference in what you're using your app for. Perfect case of, you can just go into the Twitter app and just withdraw those credentials and do it again if something happens, whereas as I say, if you're building it for Liberty, that it will not pass a lot of our Well-Architected reviews, just for that reason.Corey: If I were going to go and deploy this at a more, I guess, locked down environment, I would be tempted to find alternative approaches such as having it stored encrypted at rest via KMS in S3 is one option. So, is having global DynamoDB tables that wind up grabbing those things, even grabbing it at runtime if necessary. There are ways to make that credential more secure at rest. It's just, I look at this from a real-world perspective of what is the actual attack surface on this, and I have a really hard time just identifying anything that is going to be meaningful with regard to an exploit. If you're listening to this and have a lot of thoughts on that matter, please reach out I'm willing to learn and change my opinion on things.Matt: One thing I will say about the Dynamo approach you mentioned, I'm not sure everybody knows this, but inside the same Dynamo table, you can scope down a row. You can be, like, “This row and this field in this row can only be accessed from this one Lambda function.” So, there's a lot of really awesome security features inside DynamoDB that I don't think most people take advantage of, but they open up a lot of options for simplicity.Corey: Is that tied to the very recent announcement about Lambda getting SourceArn as a condition key? In other words, you can say, “This specific Lambda function,” as opposed to, “A Lambda in this account?” Like that was a relatively recent Advent that I haven't fully explored the nuances of.Matt: Yeah, like, that has opened a lot of doors. I mean, the Dynamo being able to be locked out in your row has been around for a while, but the new Lambda from SourceArn is awesome because, yeah, as you say, you can literally say this thing, as opposed to, you have to start going into tags, or you have to start going into something else to find it.Corey: So, I want to talk about something you just alluded to, which is the Well-Architected Framework. And initially, when it launched, it was a whole framework, and AWS made a lot of noise about it on keynote stages, as they are want to do. And then later, they created a quote-unquote, “Well-Architected Tool,” which let's be very direct, it's the checkbox survey form, at least the last time I looked at it. And they now have the six pillars of the Well-Architected Framework where they talk about things like security, cost, sustainability is the new pillar, I don't know, absorbency, or whatever the remainders are. I can't think of them off the top of my head. How does that map to your experience with the CDK?Matt: Yeah, so out of the box, the CDK from day one was designed to have sensible defaults. And that's why a lot of the things you deploy have opinions. I talked to a couple of the Heroes and they were like, “I wish it had less opinions.” But that's why whenever you deploy something, it's got a bunch of configuration already in there. For me, in the CDK, whenever I use constructs, or stacks, or deploying anything in the CDK, I always build it in a well-architected way.And that's such a loaded sentence whenever you say the word ‘well-architected,' that people go, “What do you mean?” And that's where I go through the six pillars. And in Liberty, we have a process, it used to be called SCORP because it was five pillars, but not SCORPS [laugh] because they added sustainability. But that's where for every stack, we'll go through it and we'll be like, “Okay, let's have the discussion.” And we will use the tool that you mentioned, I mean, the tool, as you say, it's a bunch of tick boxes with a text box, but the idea is we'll get in a room and as we build the starter patterns or these pieces of infrastructure that people are going to reuse, we'll run the well-architected review against the framework before anybody gets to generate it.And then we can say, out of the box, if you generate this thing, these are the pros and cons against the Well-Architected Framework of what you're getting. Because we can't make it a hundred percent bulletproof for your use case because we don't know it, but we can tell you out of the box, what it does. And then that way, you can keep building so they start off with something that is well documented how well architected it is, and then you can start having—it makes it a lot easier to have those conversations as they go forward. Because you just have to talk about the delta as they start adding their own code. Then you can and you go, “Okay, you've added these 20 lines. Let's talk about what they do.” And that's why I always think you can do a strong connection between infrastructure-as-code and well architected.Corey: As I look through the actual six pillars of the Well-Architected Framework: sustainability, cost optimization, performance, efficiency, reliability, security, and operational excellence, as I think through the nature of what this shitpost thread Twitter client is, I am reasonably confident across all of those pillars. I mean, first off, when it comes to the cost optimization pillar, please, don't come to my house and tell me how that works. Yeah, obnoxiously the security pillar is sort of the thing that winds up causing a problem for this because this is an account deployed by Control Tower. And when I was getting this all set up, my monthly cost for this thing was something like a dollar in charges and then another sixteen dollars for the AWS config rule evaluations on all of the deploys, which is… it just feels like a tax on going about your business, but fine, whatever. Cost and sustainability, from my perspective, also tend to be hand-in-glove when it comes to this stuff.When no one is using the client, it is not taking up any compute resources, it has no carbon footprint of which to speak, by my understanding, it's very hard to optimize this down further from a sustainability perspective without barging my way into the middle of an AWS negotiation with one of its power companies.Matt: So, for everyone listening, watch as we do a live well-architected review because—Corey: Oh yeah, I expect—Matt: —this is what they are. [laugh].Corey: You joke; we should do this on Twitter one of these days. I think would be a fantastic conversation. Or Twitch, or whatever the kids are using these days. Yeah.Matt: Yeah.Corey: And again, if so much of it, too, is thinking about the context. Security, you work for one of the world's largest insurance companies. I shitpost for a living. The relative access and consequences of screwing up the security on this are nowhere near equivalent. And I think that's something that often gets lost, per the perfect be the enemy of the good.Matt: Yeah that's why, unfortunately, the Well-Architected Tool is quite loose. So, that's why they have the Well-Architected Framework, which is, there's a white paper that just covers anything which is quite big, and then they wrote specific lenses for, like, serverless or other use cases that are shorter. And then when you do a well-architected review, it's like loose on, sort of like, how are you applying the principles of well-architected. And the conversation that we just had about security, so you would write that down in the box and be, like, “Okay, so I understand if anybody gets this credential, it means they can post this Last Tweet in AWS, and that's okay.”Corey: The client, not the Twitter account, to be clear.Matt: Yeah. So, that's okay. That's what you just mark down in the well-architected review. And then if we go to day one on the future, you can compare it and we can go, “Oh. Okay, so last time, you said this,” and you can go, “Well, actually, I decided to—” or you just keep it as a note.Corey: “We pivoted. We're a bank now.” Yeah.Matt: [laugh]. So, that's where—we do more than tweets now. We decided to do microtransactions through cryptocurrency over Twitter. I don't know but if you—Corey: And that ends this conversation. No no. [laugh].Matt: [laugh]. But yeah, so if something changes, that's what the well-architected reviews for. It's about facilitating the conversation between the architect and the engineer. That's all it is.Corey: This episode is sponsored in part by our friend EnterpriseDB. EnterpriseDB has been powering enterprise applications with PostgreSQL for 15 years. And now EnterpriseDB has you covered wherever you deploy PostgreSQL on-premises, private cloud, and they just announced a fully-managed service on AWS and Azure called BigAnimal, all one word. Don't leave managing your database to your cloud vendor because they're too busy launching another half-dozen managed databases to focus on any one of them that they didn't build themselves. Instead, work with the experts over at EnterpriseDB. They can save you time and money, they can even help you migrate legacy applications—including Oracle—to the cloud. To learn more, try BigAnimal for free. Go to biganimal.com/snark, and tell them Corey sent you.Corey: And the lens is also helpful in that this is a serverless application. So, we're going to view it through that lens, which is great because the original version of the Well-Architected Tool is, “Oh, you built this thing entirely in Lambda? Have you bought some reserved instances for it?” And it's, yeah, why do I feel like I have to explain to AWS how their own systems work? This makes it a lot more streamlined and talks about this, though, it still does struggle with the concept of—in my case—a stateless app. That is still something that I think is not the common path. Imagine that: my code is also non-traditional. Who knew?Matt: Who knew? The one thing that's good about it, if anybody doesn't know, they just updated the serverless lens about, I don't know, a week or two ago. So, they added in a bunch of more use cases. So, if you've read it six months ago, or even three months ago, go back and reread it because they spent a good year updating it.Corey: Thank you for telling me that. That will of course wind up in next week's issue of Last Week in AWS. You can go back and look at the archives and figure out what week record of this then. Good work. One thing that I have learned as well as of yesterday, as it turns out, before we wound up having this recording—obviously because yesterday generally tends to come before today, that is a universal truism—is it I had to do a bit of refactoring.Because what I learned when I was in New York live-tweeting the AWS Summit, is that the Route 53 latency record works based upon where your DNS server is. Yeah, that makes sense. I use Tailscale and wind up using my Pi-hole, which lives back in my house in San Francisco. Yeah, I was always getting us-west-1 from across the country. Cool.For those weird edge cases like me—because this is not the common case—how do I force a local region? Ah, I'll give it its own individual region prepend as a subdomain. Getting that to work with both the global lasttweetinaws.com domain as well as the subdomain on API Gateway through the CDK was not obvious on how to do it.Randall Hunt over at Caylent was awfully generous and came up with a proof-of-concept in about three minutes because he's Randall, and that was extraordinarily helpful. But a challenge I ran into was that the CDK deploy would fail because the way that CloudFormation was rendered in the way it was trying to do stuff, “Oh, that already has that domain affiliated in a different way.” I had to do a CDK destroy then a CDK deploy for each one. Now, not the end of the world, but it got me thinking, everything that I see around the CDK more or less distills down to either greenfield or a day one experience. That's great, but throw it all away and start over is often not what you get to do.And even though Amazon says it's always day one, those of us in, you know, real companies don't get to just treat everything as brand new and throw away everything older than 18 months. What is the day two experience looking like for you? Because you clearly have a legacy business. By legacy, I of course, use it in the condescending engineering term that means it makes actual money, rather than just telling really good stories to venture capitalists for 20 years.Matt: Yeah. We still have mainframes running that make a lot of money. So, I don't mock legacy at all.Corey: “What's that piece of crap do?” “Well, about $4 billion a year in revenue. Perhaps show some respect.” It's a common refrain.Matt: Yeah, exactly. So yeah, anyone listening, don't mock legacy because as Corey says, it is running the business. But for us when it comes to day two, it's something that I'm actually really passionate about this in general because it is really easy. Like I did it with CDK patterns, it's really easy to come out and be like, “Okay, we're going to create a bunch of starter patterns, or quickstarts”—or whatever flavor that you came up with—“And then you're going to deploy this thing, and we're going to have you in production and 30 seconds.” But even day one later that day—not even necessarily day two—it depends on who it was that deployed it and how long they've been using AWS.So, you hear these stories of people who deployed something to experiment, and they either forget to delete, it cost them a lot of money or they tried to change it and it breaks because they didn't understand what was in it. And this is where the community starts to diverge in their opinions on what AWS CDK should be. There's a lot of people who think that at the minute CDK, even if you create an abstraction in a construct, even if I create a construct and put it in the construct library that you get to use, it still unravels and deploys as part of your deploy. So, everything that's associated with it, you don't own and you technically need to understand that at some point because it might, in theory, break. Whereas there's a lot of people who think, “Okay, the CDK needs to go server side and an abstraction needs to stay an abstraction in the cloud. And then that way, if somebody is looking at a 20-line CDK construct or stack, then it stays 20 lines. It never unravels to something crazy underneath.”I mean, that's one pro tip thing. It'd be awesome if that could work. I'm not sure how the support for that would work from a—if you've got something running on the cloud, I'm pretty sure AWS [laugh] aren't going to jump on a call to support some construct that I deployed, so I'm not sure how that will work in the open-source sense. But what we're doing at Liberty is the other way. So, I mean, we famously have things like the software accelerator that lets you pick a pattern or create your pipelines and you're deployed, but now what we're doing is we're building a lot of telemetry and automated information around what you deployed so that way—and it's all based on Well-Architected, common theme. So, that way, what you can do is you can go into [crosstalk 00:26:07]—Corey: It's partially [unintelligible 00:26:07], and partially at a glance, figure out okay, are there some things that can be easily remediated as we basically shift that whole thing left?Matt: Yeah, so if you deploy something, and it should be good the second you deploy it, but then you start making changes. Because you're Corey, you just start adding some stuff and you deploy it. And if it's really bad, it won't deploy. Like, that's the Liberty setup. There's a bunch of rules that all go, “Okay, that's really bad. That'll cause damage to customers.”But there's a large gap between bad and good that people don't really understand the difference that can cost a lot of money or can cause a lot of grief for developers because they go down the wrong path. So, that's why what we're now building is, after you deploy, there's a dashboard that'll just come up and be like, “Hey, we've noticed that your Lambda function has too little memory. It's going to be slow. You're going to have bad cold starts.” Or you know, things like that.The knowledge that I have had the gain through hard fighting over the past couple of years putting it into automation, and that way, combined with the well-architected reviews, you actually get me sitting in a call going, “Okay, let's talk about what you're building,” that hopefully guides people the right way. But I still think there's so much more we can do for day two because even if you deploy the best solution today, six months from now, AWS are releasing ten new services that make it easier to do what you just did. So, someone also needs to build something that shows you the delta to get to the best. And that would involve AWS or somebody thinking cohesively, like, these are how we use our products. And I don't think there's a market for it as a third-party company, unfortunately, but I do think that's where we need to get to, that at day two somebody can give—the way we're trying to do for Liberty—advice, automated that says, “I see what you're doing, but it would be better if you did this instead.”Corey: Yeah, I definitely want to spend more time thinking about these things and analyzing how we wind up addressing them and how we think about them going forward. I learned a lot of these lessons over a decade ago. I was fairly deep into using Puppet, and came to the fair and balanced conclusion that Puppet was a steaming piece of crap. So, the solution was that I was one of the very early developers behind SaltStack, which was going to do everything right. And it was and it was awesome and it was glorious, right up until I saw an environment deployed by someone else who was not as familiar with the tool as I was, at which point I realized hell is other people's use cases.And the way that they contextualize these things, you craft a finely balanced torque wrench, it's a thing of beauty, and people complain about the crappy hammer. “You're holding it wrong. No, don't do it that way.” So, I have an awful lot of sympathy for people building platform-level tooling like this, where it works super well for the use case that they're in, but not necessarily… they're not necessarily aligned in other ways. It's a very hard nut to crack.Matt: Yeah. And like, even as you mentioned earlier, if you take one piece of AWS, for example, API Gateway—and I love the API Gateway team; if you're listening, don't hate on me—but there's, like, 47,000 different ways you can deploy an API Gateway. And the CDK has to cover all of those, it would be a lot easier if there was less ways that you could deploy the thing and then you can start crafting user experiences on a platform. But whenever you start thinking that every AWS component is kind of the same, like think of the amount of ways you're can deploy a Lambda function now, or think of the, like, containers. I'll not even go into [laugh] the different ways to run containers.If you're building a platform, either you support it all and then it sort of gets quite generic-y, or you're going to do, like, what serverless cloud are doing though, like Jeremy Daly is building this unique experience that's like, “Okay, the code is going to build the infrastructure, so just build a website, and we'll do it all behind it.” And I think they're really interesting because they're sort of opposites, in that one doesn't want to support everything, but should theoretically, for their slice of customers, be awesome, and then the other ones, like, “Well, let's see what you're going to do. Let's have a go at it and I should hopefully support it.”Corey: I think that there's so much that can be done on this. But before we wind up calling it an episode, I had one further question that I wanted to explore around the recent results of the community CDK survey that I believe is a quarterly event. And I read the analysis on this, and I talked about it briefly in the newsletter, but it talks about adoption and a few other aspects of it. And one of the big things it looks at is the number of people who are contributing to the CDK in an open-source context. Am I just thinking about this the wrong way when I think that, well, this is a tool that helps me build out cloud infrastructure; me having to contribute code to this thing at all is something of a bug, whereas yeah, I want this thing to work out super well—Docker is open-source, but you'll never see me contributing things to Docker ever, as a pull request, because it does, as it says on the tin; I don't have any problems that I'm aware of that, ooh, it should do this instead. I mean, I have opinions on that, but those aren't pull requests; those are complete, you know, shifts in product strategy, which it turns out is not quite done on GitHub.Matt: So, it's funny I, a while ago, was talking to a lad who was the person who came up with the idea for the CDK. And CDK is pretty much the open-source project for AWS if you look at what they have. And the thought behind it, it's meant to evolve into what people want and need. So yes, there is a product manager in AWS, and there's a team fully dedicated to building it, but the ultimate aspiration was always it should be bigger than AWS and it should be community-driven. Now personally, I'm not sure—like you just said it—what the incentive is, given that right now CDK only works with CloudFormation, which means that you are directly helping with an AWS tool, but it does give me hope for, like, their CDK for Terraform, and their CDK for Kubernetes, and there's other flavors based on the same technology as AWS CDK that potentially could have a thriving open-source community because they work across all the clouds. So, it might make more sense for people to jump in there.Corey: Yeah, I don't necessarily think that there's a strong value proposition as it stands today for the idea of the CDK becoming something that works across other cloud providers. I know it technically has the capability, but if I think that Python isn't quite a first-class experience, I don't even want to imagine what other providers are going to look like from that particular context.Matt: Yeah, and that's from what I understand, I haven't personally jumped into the CDK for Terraform and we didn't talk about it here, but in CDK, you get your different levels of construct. And is, like, a CloudFormation-level construct, so everything that's in there directly maps to a property in CloudFormation, and then L2 is AWS's opinion on safe defaults, and then L3 is when someone like me comes along and turns it into something that you may find useful. So, it's a pattern. As far as I know, CDK for Terraform is still on L1. They haven't got the rich collection—Corey: And L4 is just hiring you as a consultant—Matt: [laugh].Corey: —to come in fix my nonsense for me?Matt: [laugh]. That's it. L4 could be Pulumi recently announced that you can use AWS CDK constructs inside it. But I think it's one of those things where the constructs, if they can move across these different tools the way AWS CDK constructs now work inside Pulumi, and there's a beta version that works inside CDK for Terraform, then it may or may not make sense for people to contribute to this stuff because we're not building at a higher level. It's just the vision is hard for most people to get clear in their head because it needs articulated and told as a clear strategy.And then, you know, as you said, it is an AWS product strategy, so I'm not sure what you get back by contributing to the project, other than, like, Thorsten—I should say, so Thorsten who wrote the book with me, he is the number three contributor, I think, to the CDK. And that's just because he is such a big user of it that if he sees something that annoys him, he just comes in and tries to fix it. So, the benefit is, he gets to use the tool. But he is a super user, so I'm not sure, outside of super users, what the use case is.Corey: I really want to thank you for, I want to say spending as much time talking to me about this stuff as you have, but that doesn't really go far enough. Because so much of how I think about this invariably winds up linking back to things that you have done and have been advocating for in that community for such a long time. If it's not you personally, just, like, your fingerprints are all over this thing. So, it's one of those areas where the entire software developer ecosystem is really built on the shoulders of others who have done a lot of work that came before. Often you don't get any visibility of who those people are, so it's interesting whenever I get to talk to someone whose work I have directly built upon that I get to say thank you. So, thank you for this. I really do appreciate how much more straightforward a lot of this is than my previous approach of clicking in the console and then lying about it to provision infrastructure.Matt: Oh, no worries. Thank you for the thank you. I mean, at the end of the day, all of this stuff is just—it helps me as much as it helps everybody else, and we're all trying to do make everything quicker for ourselves, at the end of the day.Corey: If people want to learn more about what you're up to, where's the best place to find you these days? They can always take a job at Liberty; I hear good things about it.Matt: Yeah, we're always looking for people at Liberty, so come look up our careers. But Twitter is always the best place. So, I'm @NIDeveloper on Twitter. You should find me pretty quickly, or just type Matt Coulter into Google, you'll get me.Corey: I like it. It's always good when it's like, “Oh, I'm the top Google result for my own name.” On some level, that becomes an interesting thing. Some folks into it super well, John Smith has some challenges, but you know, most people are somewhere in the middle of that.Matt: I didn't used to be number one, but there's a guy called the Kangaroo Kid in Australia, who is, like, a stunt driver, who was number one, and [laugh] I always thought it was funny if people googled and got him and thought it was me. So, it's not anymore.Corey: Thank you again for, I guess, all that you do. And of course, taking the time to suffer my slings and arrows as I continue to revise my opinion of the CDK upward.Matt: No worries. Thank you for having me.Corey: Matt Coulter, senior architect at Liberty Mutual. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice and leave an angry comment as well that will not actually work because it has to be transpiled through a JavaScript engine first.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.
About AdamAdam is an independent cloud consultant that helps startups build products on AWS. He's also the host of AWS FM, a podcast with guests from around the AWS community, and an AWS DevTools Hero.Adam is passionate about open source and has made a handful of contributions to the AWS CDK over the years. In 2020 he created Ness, an open source CLI tool for deploying web sites and apps to AWS.Previously, Adam co-founded StatMuse—a Disney backed startup building technology that answers sports questions—and served as CTO for five years. He lives in Nixa, Missouri, with his wife and two children.Links Referenced: 17 Ways to Run Containers On AWS: https://www.lastweekinaws.com/blog/the-17-ways-to-run-containers-on-aws/ Twitter: https://twitter.com/aeduhm Twitch: https://www.twitch.tv/adamelmore TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. Every once in a while, I encounter someone in the wild that… well, I'll just be direct, makes me feel a little bit uneasy, almost like someone's walking over my grave. And I think I've finally figured out elements of what that is. It feels sometimes like I run into people—ideally not while driving—who are trying to occupy sort of the same space in the universe, and I never quite know how to react to that.Today's guest is just one such person. Adam Elmore is an independent AWS consultant, has been all over the Twitters for a while, recently started live streaming basically his every waking moment because he is just that interesting. Adam, thank you for suffering my slings and arrows—Adam: [laugh].Corey: —and agreeing to chat with me today.Adam: I would say first of all, you don't need to be worried about anyone walking over your grave. [laugh]. That was very flattering.Corey: No, honestly, I have big enterprise companies looking to put me in my grave, but that's a separate threat model. We're good on that, for now.Adam: [laugh]. I got to set myself up here to—I'm just going to laugh a lot, and your editor or somebody's going to have to deal with that. And maybe the audience will see—[laugh].Corey: Hey, I prefer that as opposed to talking to people who have absolutely no sense of humor of which they are aware. Awesome, I have a list of companies that they should apply for immediately. So, when I say that we're trying to occupy elements of the same space in the universe, let me talk a little bit about what I mean by that. You are independent as a consultant, which is how I started this whole nonsense, and then I started gathering a company around me almost accidentally. You are an AWS Dev Tools Hero, whereas I am an AWS community villain, which is kind of a polar opposite slash anti-hero approach, and it's self-granted in my case. How did you stumble into the universe of AWS? You just realized one day you were too happy and what can you do to make yourself miserable, and this was the answer, or what?Adam: Yeah, I guess. So. I mean, I've been a software developer for 15 years, like, my whole career, that's kind of what I've done. And at some point, I started a startup called StatMuse. And I was able, as sort of a co-founder there, with venture backing, like, I was able to just kind of play with the cloud.And we deployed everything on AWS, so that was—like, I was there five years; it was sort of five years of running this, I would call it like a Digital Media Studio. Like, we built technology, but we did lots of experiments, so it felt like playing on AWS. Because we built kind of weird one-offs, these digital experiences for various organizations. The Hall of Fame was one of them. We did, like, a, like, a 3-D Talking bust of John Madden, so it was like all kinds of weird technology involved.But that was sort of five years of, I guess, spending venture money [laugh] to play on AWS. And some of that was Google money; I guess I never thought about that, but Google was an investor in StatMuse. [laugh]. Yeah, so we sort of like—I ran that for five years and was able to learn just a lot of AWS stuff that really excited me. I guess, coming from normal web development stuff, it was exciting just how much leverage you have with AWS, so I sort of dove in pretty hard. And then yeah, when I left StatMuse in 2019 I've just been, I guess, going even harder into that direction. I just really enjoy it.Corey: My first real exposure to AWS was at a company where the CTO was a, I guess we'll call him an extraordinarily early cloud evangelist. I was there as a contractor, and he was super excited and would tweet nonsensical things like, “I'm never going to rack a server ever again.” And I was a grumpy sysadmin type; I came from the ops world where anything that is new shouldn't be treated with disdain and suspicion because once you've been a sysadmin for 20 minutes, you've been there long enough to see today's shiny new shit become tomorrow's legacy garbage that you're stuck supporting. So, “Oh, great. What now?”I was very down on Cloud in those days and I encountered it with increasing frequency as I stumbled my way through my career. And at the end of 2016, I wound up deciding to go out independent and fix… well, what problems am I good at fixing that I can articulate in a sentence, and well, I'd gotten surprised by AWS bills from time to time—fortunately with someone else's money; the best kind of mistake to make—and well I know a few things. Let's get really into it. In time, I came to learn that cost and architecture the same thing in cloud, and now I don't know how the hell to describe myself. Other people love to describe me, usually with varying forms of profanity, but here we are. It really turns into the idea of forging something of your own path. And you've absolutely been doing that for at least the last three years as you become someone who's increasingly well known and simultaneously harder to describe.Adam: Yeah, I would say if you figure it out, if you know how to describe me, I would love to know because just coming up with the title—for this episode you needed, like, my title, I don't know what my title is. I'm also—like, we talked about independent, so nobody sort of gives me a title. I would love to just receive one if you think of one, [laugh] if anyone listening thinks of one… it's increasingly hard to, sort of like, even decide what I care most about. I know I need to, like, probably niche down, I feel like you've kind of niched into the billing stuff. I can't just be like, “I'm an AWS guy,” because AWS is so big. But yeah, I have no idea.Corey: Anyone who claims, “Oh, I'm an expert in AWS,” is lying or trying to sell something.Adam: [laugh]. Exactly.Corey: I love that. It's, “Really? I have some questions to establish that for you.” As far as naming what it is, you do, first piece of advice, never ever, ever, ever listen to someone who works at AWS; those people are awful at naming things, as evidenced by basically every service they've ever launched. But you are actually fairly close to being an AWS expert. You did a six-week speed-run through every certification that they offer and that is nothing short of astonishing. How'd it come about?Adam: It's a unique intersection of skills that I think I have. And I'm not very self-aware, I don't know all my strengths and weaknesses and I struggle to sort of nail those down, but I think one of my strengths is just ability to, like, consume information, I guess at a high volume. So, I'm like an auditory learner; I can listen to content really fast and sort of retain enough. And then I think the other skill I have is just I'm good at tests. I've always said that, like, going back to school, like, high school, I always felt like I was really good at multiple-choice tests. I don't know if that's a skill or some kind of innate talent.But I think those two things combined, and then, like, eight years of building on AWS, and that sort of frames how I was able to take all that on. And I don't know that I really set out thinking I will do it in six weeks. I took the first few and then did them pretty fast and thought, “I wonder how quickly I could do all of them.” And I just kind of at that point, it became this sort of goal. I have to take on certain challenges occasionally that just sound fun for no reason other than they sound fun and that was kind of the thing for those six weeks. [laugh].Corey: I have two certifications: Cloud Practitioner and the SysOps Administrator Associate. Those were interesting.Adam: You took the new one, right? The new SysOps with the labs and stuff I'd love to hear about that.Corey: I did, back when it was in beta. That was a really interesting experience and I'll definitely get to that, but I wound up, for example, getting a question wrong in the Cloud Practitioner exam four years ago or so, when it was, “How long does it take to restore an RDS instance from backup?” And I gave the honest answer instead of the by-the-book, correct answer. That's part of the problem is that I've been doing this stuff too long and I know how these things break and what the real world looks like. Certifications are also very much a snapshot at a point in time.Because I write the Last Week in AWS newsletter, I'm generally up-to-the-minute on what has changed, and things that were not possible yesterday, suddenly are possible today, so I need to know when was this certification launched. Oh, it was in early 2021. Yeah, I needed to be a lot more specific; which week? And then people look at me very strangely and here we are.The Systems Administrator Certification was interesting because this is the first one, to my knowledge, where they started doing a live lab as a—Adam: Yeah.Corey: Component of this. And I don't think it's a breach of the NDA to point out that one of the exams was, “Great. Configure CloudWatch out of the box to do this thing that it's supposed to do out of the box.” And I've got to say that making the service do what it's supposed to do with no caveats is probably the sickest shade I've ever seen anyone throw at AWS, like, configuring the service is so bad that it is going to be our test to prove you know what you're doing. That is amazing.Adam: [laugh]. Yeah, I don't have any shade through I'm not as good with the, like, ability to come off, like, witty and kind while still criticizing things. So, I generally just try not to because I'm bad at it. [laugh].Corey: It's why I generally advise people don't try, in seriousness. It's not that people can't be clever; it's that the failure mode of clever is ‘asshole' and I'm not a big fan of making people feel worse based upon the things that I say and do. It's occasionally I wind up getting yelled at by Amazonians saying that the people who built a service didn't feel great about something I said, and my instinctive immediate reaction is, “Oh, shit, that wasn't my intention. How did I screw this up?” Given a bit of time, I realized that well hang on a minute because I'm not—they're not my target audience. I'm trying to explain this to other customers.And, on some level, if you're going to charge tens of millions of dollars a month for a service or more, maybe make a better one, not for nothing. So, I see both sides of it. I'm not intentionally trying to cause pain, but I'm also not out here insulting people individually. Like, sometimes people make bad decisions, sometimes individually, sometimes in a group. And then we have a service name we have to live with, and all right, I guess I'm going to make fun of that forever. It's fun that keeps it engaging for me because otherwise, it's boring.Adam: No, I hear you. No, and somebody's got to do it. I'm glad you do it and do it so well because, I mean, you got to keep them honest. Like, that's the thing. Keep AWS in check.Corey: Something that I went through somewhat recently was a bit of an awakening. I have no problem revisiting old opinions and discovering that huh, I no longer agree with it; it's time to evolve that opinion. The CDK specifically was one of those where I looked at it and thought this thing looks a little hokey. So, I started using it in Python and sure enough, the experience was garbage. So cool, the CDK is a piece of crap. There we go. My job is easy.I was convinced to take a second look at it via TypeScript, a language I do not know and did not have any previous real experience with. So, I spent a few days just powering through it, and now I'm a convert. I think it's amazing. It is my default go-to for building AWS infrastructure. And all it took was a little bit of poking and prodding to get me to change my mind on that. You've taken it to another level and you started actively contributing to the AWS CDK. What was your journey with that, honestly, remarkable piece of software?Adam: Yeah, so I started contributing to CDK when I was actually doing a lot of Python development. So, I worked with a company that was doing—there was a Python shop. So actually, the first thing I contributed was a Python function construct, which is sort of the equivalent of the Node.js function construct, which like, you can just basically point at a TypeScript file and it transpiles it, bundles it, and does all that, right? So, it makes it easy to deploy TypeScript as a Lambda function.Well, I mean, it ends up being a JavaScript Lambda function, but anyway, that was the Python function construct. And then I sort of got really into it. So, I got pretty hooked on using the CDK in every place that I could. I'm a huge fan, and I do primarily write in TypeScript these days. I love being able to write TypeScript front-end and back, so built a lot of, like, Next.JS front-ends, and then I'm building back-ends with CDK TypeScript.Yeah, I've had, like, a lot of conversations about CDK. I think there's definitely a group that's sort of, against the CDK, if you're thinking in terms of, like, beginners. And I do see where, for people who aren't as familiar with AWS, or maybe this is their entry point into cloud development, it does a lot of things that maybe you're not aware of that, you know, you're now kind of responsible for. So, it's deploying—like, it makes it really easy to write, like, three lines of TypeScript that stand up an entire VPC with all this configuration and Managed NAT Gateways and [laugh] everything else. And you may not be aware of all the things you just stood up.So, CloudFormation maybe is a little more—sort of gives you that better visibility into what you're creating. So, I've definitely seen that pushback. But I think for people who really, like, have built a lot of applications on AWS, I think the CDK is just such a time-saver. I mean, I spend so much less time building the same things in the CDK versus CloudFormation. I'm a big fan.Corey: For me, I've learned enough about JavaScript to be dangerous and it seems like TypeScript is more or less trying to automate a bunch of people's jobs away, which is basically, from I can tell, their job is to go on the internet and complain about someone's JavaScript. So great, that that's really all it does is it complains, “Oh, this ambiguous. You should be more specific about it.” And great. Awesome. I still haven't gotten into scenarios where I've been caught out by typing issues, and very often I find that it just feels like sheer bloodymindedness, but I smile, nod, bend the knee and life goes on.Adam: [laugh]. When you've got a project that's, like, I don't know, a few months old—or better, a few years old—and you need to do, like, major refactoring, that's when TypeScript really saves you just a ton of time. Like, when you can make a change in a type or in actual implementation stuff and then see the ripple effects and then sort of go around the codebase and fix those things, it's just a lot easier than doing it in JavaScript and discovering stuff at runtime. So, I'm a big TypeScript fan. I don't know where it's all headed. I know there's people that are not fans of, like, transpiling your Lambda functions, for instance. Like, why not just ship good JavaScript? And I get that case, too. Yeah, but I've definitely—I felt the productivity boost, I guess—if that's the thing—from TypeScript.Corey: For me, I'm still at a point where I'm learning the edges of where things start and where they stop. But one of the big changes I made was that I finally, after 15 years, gave up my beloved Vim as my editor for this and started using VS Code. Because the reasons that I originally went with Vi were understandable when you realize what I was. I'm always going to be remoting into network gear or random—on maintained Unix boxes. Vi is going to be everywhere on everything and that's fine.Yeah, I don't do that anymore, and increasingly, I find that everything I'm writing is local. It is not something that is tied to a remote thing that I need to login and edit by hand. At that point, we are in disaster area. And suddenly it's nice. I mean things like tab completion, where it just winds up completing the rest of the variable name or, once you enable Copilot and absolutely not CodeWhisperer yet, it winds up you tab complete your entire application. Why not? It's just outsourcing it to Stack Overflow without that pesky copy and paste step.Adam: Yeah, I don't know how in the weeds you want to get on your p—I don't know, in terms of technical stuff, but Copilot both blows me away—there are days where it autocompletes something that I just, I can't fathom how—it pulled in not just, like, the patterns that it found, obviously, in training, but, like, the context in the file I'm working and sort of figured out what I was trying to do. Sometimes it blows me away. A lot of times, though, it frustrates me because of TypeScript. Like, I'm used to Typescript and types saving me from typing a lot. Like, I can tab-complete stuff because I have good types defined, right, or it's just inferred from the libraries I'm using.It's tough though when GitHub is fighting with TypeScript and VS Code. But it's funny that you came from Vim and you now live in VS Code. I really am trying to move from VS Code to, like, the Vim world, mostly because of Twitch streamers that blow my mind with what they can do in Vim [laugh] and how fast they can move. I do—every time I move my hand, like, over to the arrow keys, I feel a little sad and I wish I just did Vim.Corey: This episode is sponsored in part by our friends at Lambda Cloud. They offer GPU instances with pricing that's not only scads better than other cloud providers, but is also accessible and transparent. Also, check this out, they get a lot more granular in terms of what's available. AWS offers NVIDIA A100 GPUs on instances that only come in one size and cost $32/hour. Lambda offers instances that offer those GPUs as single card instances for $1.10/hour. That's 73% less per GPU. That doesn't require any long term commitments or predicting what your usage is gonna look like years down the road. So if you need GPUs, check out Lambda. In beta, they're offering 10TB of free storage and, this is key, data ingress and egress are both free. Check them out at lambdalabs.com/cloud. That's l-a-m-b-d-a-l-a-b-s.com/cloud.Corey: There are people who have just made it into an entire lifestyle, on some level. And I'm fair to middling; I've known people who are dark wizards at it. In practice, I found that my productivity was never constrained by how quickly I can type. It's one of those things where it's, I actually want to stop and have my brain catch up sometimes, believe it or not, for those who follow me on Twitter. It's the idea of wanting to make sure that I am able to intelligently and rationally wrap my head around what it is I'm doing.And okay, just type out a whole bunch of boilerplate is, like, the least valuable use of anything and that is where I find things like Copilot working super well, where I, if I'm doing CloudFormation, for example, the fact that it tab-completes all the necessary attributes and can go back and change them or whatnot, that's an enormous time saver. Same story with the CDK, although with some constructs, it doesn't quite understand which ones get certain values to it. And I really liked the idea behind it. I think this is in some ways, the future of IDEs, to a point.Adam: Oh, for sure. I think, like, the case, you call that with CloudFormation, you don't have really typeahead in VS Code, at least I'm not using anything. Maybe there are extensions that give you that in VS Code. But to have Copilot fill in required prompts on a CloudFormation template, that's a lifesaver. Because I just, every time I write CloudFormation, I've just got the docs up and I'm copying stuff I've done before or whatever; like, to save that time it's huge. But CodeWhisperer, not so much? Is it not, I guess, up to snuff? I haven't seen it or played with it at all.Corey: It's still very early days and it hasn't had exposure outside of Amazonian codebases to my understanding, so it's, like, “Learn to code like an Amazonian.” And you can fill in your own joke here on that one. I imagine it's like—isn't that—aren't they primarily a Java shop, for one? And all right. It turns out most of my code doesn't need to operate the way that there's does.Adam: I didn't know that they were training it just internally. Like, I'm assuming Copilot is trained on, like, Stack Overflow or something, right? Or just all of GitHub, I guess.Corey: And GitHub and a bunch of other things, and people are yelling at them for it, and I haven't been tracking that. But honestly, the CodeWhisperer announcement taught me things about Copilot, which is weird, which tells me that none of these companies are great at explaining this. Like I can just write a comment in this of, “Add an S3 bucket,” and then Copilot will tab-complete the entirety of adding an S3 bucket, usually even secure, which is awesome. They also fix the early Copilot teething problems of tab-completing people's AWS API credentials. You know, the—yeah, they've fixed a lot of that, thankfully.Adam: Yeah.Corey: But it's still one of those neat things that you can just basically start—it gets a little bit closer to describe what you want the application to do and then it'll automatically write it for you on the back-end. Sure, sometimes it makes naive decisions that do not bear out, but again, it's still early days. I'm optimistic.Adam: Yeah, that reminds me of, like, the, I mean, the serverless cloud, so serverless framework folks, like, what they're doing where they're sort of inferring your infrastructure based on you just write an app and it sort of creates the infrastructure as code for you, or just sort of infers it all from your code. So, if you start using a bucket, it'll create a bucket for that. That definitely seems to be a movement as well, where just do less as a developer [laugh] seems to be the theme.Corey: Yeah, just move up the stack. We see this time and time again. I mean, look at the—I use this analogy from time to time from the sysadmin world, but in the late-90s, if you wanted to build a web server, you needed a spare week and an intimate knowledge of GCC compiler flags. In time, it became oh, great, now it's rpm install, then yum install, then ensure present with something like Puppet, and then Docker has it, and now it's just a checkbox on the S3 page, and you're running a static site. Things don't get harder with time, and I don't think that as a developer, your time is best spent writing by hand the proper syntax for a for loop or whatnot.It's not the differentiated value. Talk to me instead about what you want that thing to do. That was my big problem with Lambda when it first came out and I spent two weeks writing my first Lambda function—because I'm bad at programming—where I had to learn the exact format of expected for input and output, and now any Lambda function I write takes me a couple of minutes to write because I'm also bad at programming and don't know what tests are.Adam: [laugh]. Tests are overrated, I don't spend a lot of time writing t—I mean, I do a lot of stuff alone and I do a lot of stuff for myself, so in those contexts, I'm not writing tests if I'm being honest. I stream now and everyone on the stream is constantly asking, “Where are the tests?” Like, there are no tests. I'm sorry. [laugh]. Was someone else's stream.Corey: Oh yeah, it used to be though, that you had to be a little sneakier to have other people do work for you. Copilot makes it easier and presumably CodeWhisperer will, too. Used to be that if AWS launched new service and I didn't know how to configure it, all I would do is restrict a role down to only being able to work with that service, attach that to a user and then just drop the credentials on Twitter or GitHub. And I waited 20 minutes and I came back and sure enough, someone configured it and was already up and mining Bitcoin. So, turn that off, take what they built, and off the production with it. Problem solved. Oh, and rotate those credentials, unless you enjoy pain. Problem solved. The end. And I don't know if it's a best practice, but it sure was effective.Adam: Yeah, that would do it. Well, they're just like scanners now, right, like they're just scanning GitHub public repos for any credentials that are leaked like that, and they're available within seconds. You can literally, like, push a public repo with credentials and it is being [laugh] used within minutes. It's nuts.Corey: GitHub has some automatic back channel thing—I believe; I haven't done an experiment lately, but I believe that AWS will intentionally shoot down the credential as soon as it gets reported, which is kind of amazing. I really should do some more experiments with it just to see how disastrous this can get.Adam: Yeah. No, I'd be curious. Please let me know. I guess you'll tweet about it so I'll see it.Corey: Can I borrow your account for a few minutes?Adam: Yeah. [laugh].Corey: Yeah, it's fun. Now, the secret to my 17 Ways to Run Containers On AWS is in almost every case, those containers can be crypto miners, so it's not just about having too many services do the same thing; it's the attack surface continues to grow and expand in the fullness of time. I'm not saying this is right or wrong; it is what it is, but it's also something that I think people have an understated appreciation for.Let's change topic a little bit. Something you've been doing lately and talking about is the idea of building a course on AWS. You're clearly capable of doing the engineering work. That's not in question. You've been a successful consultant for years, which tells me you also know how to deliver software that meets customer requirements, as opposed to, “Well, the spec was shitty, but I wrote it anyway,” because you don't last long as a consultant if you enjoy being able to afford to eat if that's the direction you go in. Now, you're drifting toward becoming a teacher. Tell me about that. First, what makes you think that's something you're good at?Adam: So, I don't know. I don't know that I'm good at it and I guess I'll find out. I've been streaming, like, on Twitch just my work days, and that's been early signs that I think I'm okay at it, at least. I think it's very different, obviously, like, a self-paced course are going to be very different from streaming for hours, so there's a lot more editing and thoughtfulness involved, but I do think, like, I've always wanted to teach. So, even before I got into technology—I was pretty late into technology; it was after high school. Like back in high school, I always thought I wanted to be a professor.I just enjoyed, I guess the idea of presenting ideas in ways that people understood. And I live in an area—so I live in the Ozarks, it's not a very tech literate area. It became, like, this thing where I felt like I could really explain technology to people who are non-technical. And that's not necessarily what my course—what I'm aiming to do. I'm trying to teach web developers how to leverage AWS, and then sort of get out of the maybe front-end only or maybe traditional web frameworks—like, they've only worked with stuff that they deploy to Heroku or whatever—trying to teach that crowd, how to leverage AWS and all these wonderful primitives that we have.So, that's not exactly the same thing, but that's sort of like, I feel like I do have the ability to translate technology to non-technical folks. And then I guess, like, for me, at this stage of my career, you know, I've done a lot of work for a company, for startups, for individual clients, and it feels very, like—I just always feel like I'm going in a hole. Like, I feel like, I'm doing this little thing and I'm serving this one customer, but the idea of being able to, I guess, serve more people and sort of spread my reach, the idea of creating something that I can share with a lot of developers who would maybe benefit from it, it just feels better, I guess. [laugh]. I don't know exactly all the reasons why that feels better, but like, at the end of the day, my consulting kind of feels like this thing I do because I just need money.And now that I need money less and less, I just feel like I'd rather do stuff that I actually am excited about. I'm actually really excited about the outcomes for creating a course where, you know, I think I can maybe—my style of teaching or something could resonate with some group of people. Yeah, so that's it. It's AWS for web devs. The thought is that I'm going to create courses after this. Like, I hope to move into more education, less consulting. That's where I'm at.Corey: I would say you're probably selling yourself fairly short. I've seen a lot of the content you've put out over the years and I learned a lot from it every time. I think that there are some folks who put courses out where, one, they don't have the baseline knowledge around what it is that they're teaching, it just feels like a grift, and another failure mode is that people know how to do the thing, but they have no idea how to teach it to someone who isn't them. And there's nothing inherently wrong with not knowing how to teach; it is its own distinct skill. The problem is when you don't recognize that about yourself and in turn, wind up having some somewhat significant challenges.Adam: Yeah. No, I know that one of the struggles is, I work with pretty obscure technologies on AWS. Not obscure, but like, I have a very specific way I build APIs on AWS and I don't know that's generally, if you're taking a bunch of web developers and trying to move them into AWS is probably not the stack that I use. So, that is part of it, but that's also kind of to my benefit, I guess. It works for me a little bit in that I'm less familiar with maybe the more beginner-friendly way to enter into AWS.It's been years, so I think I can kind of come at it a little fresh and that'll help me produce a course that maybe meets them where they're at better. Yeah, the grifting thing, I'm definitely sensitive to just this idea of putting out a course. It was hard for me to really go out there and say I was making a course, even on Twitter, because I just feel like there's, like, some stereotype—I don't know, there's an association with that, for me at least, for my perception of course creation. But I know that there are people who've done it right and do it for the right reasons. And I think to the extent that I could hit that, you know, both those things, do it right and do it for the right reasons, then it's exciting to me. And if I can't, and it turned out not good at teaching, then I'll move on and do more consulting, I guess, [laugh] or streaming on Twitch.Corey: You are very clearly self-aware enough that if you put something out and it isn't effective, I have zero doubt that you won't just stop selling it, you'll take it down and reach out to people. Because you, more so than most, seem very cognizant of the fact that a poor experience learning something does not in most people's cases, translate to, “Oh, my teacher is shitty.” Instead, it's, “Oh, I'm bad at this and I'm not smart enough to figure it out.” That's still the problem I run into with bad developer experience on a bunch of things that get launched. If I have a bad time, I assume it's, “Oh, I'm stupid. I wish someone had told me.”And first, they did, secondly, it's the sense that no, it's just not being very clearly explained and the folks who wrote the documentation or talking about it are too close to what they've built to understand what it's like to look at this thing from fresh eyes. They're doing a poor job of setting the stage to explain the value it brings and in what scenario, you should be using this.Adam: It's a long process. I want to launch the course in the fall, but in the process of building out the course, I'm really going to be doing workshops and individual—like, I just have a lot of friends that are web developers and I'm going to be kind of getting on with them and teaching them this material and just trying to see what resonates. I'm going to a lot of trouble, I guess, to make sure I'm not just putting out a thing just to say I made a course. Like, I don't actually want to say I made a course, so if I'm going to do it, it's like most things I do I really kind of throw myself into. And I know if I spend enough energy and effort, I think I can make something that at least helps some people. I guess we'll see.Corey: I look forward to it. Any idea as far as rough timeline goes?Adam: Yeah, I hope to launch in the fall. But if it takes longer, I don't know. I've heard people say, to do a course right, you should spend a year on it. And maybe that's what I do.Corey: No, I love that answer. It's great. You're just saying I want to launch in the fall, which is sufficiently vague, and if that winds up not being vague enough, you could always qualify with, “Well, I didn't say what year.”Adam: [laugh].Corey: So, great you know, it's always going to be the fall somewhere.Adam: [laugh]. I just know, like, when someone says you should spend a year I just do things very hard. Like I really, like, throw a lot of time and obsess, like, I'm very obsessive. And when I do something, it's hard for me imagine doing any one thing for a year because I burn myself out. Like, I obsess very hard for usually, like, three months, it's usually, like, a quarter, and then I fall off the face of the earth for three months and I basically mope around the house and I'm just too tired to do anything else. So, I think right now I'm streaming and that's kind of been my obsession. I'm three weeks in so we got a few more months and then we'll see, [laugh] we'll see how I maintain it.Corey: Well, I look forward to seeing how it comes out. You'll have to come back and let us know when it's ready for launch.Adam: Yeah, that sounds great.Corey: I really want to thank you for being so generous with your time and taking me through what you're up to. If people want to learn more, what's the best place for them to find you?Adam: Yeah, I think Twitter. I mean, I mostly hang out on Twitter, and these days Twitch. So, Twitter my handle—I guess you'll put it, like, in the thing description or something. It's like the phonetic—Corey: Oh, we will absolutely toss it into the show notes, where useful content goes to linger.Adam: [laugh]. It's like A-E-D-U-H-M. It's like a—it's the phonetic way of saying Adam, I guess. And then on Twitch, I'm adamelmore. So, those are the two places I spend most my time.Corey: And off to the show notes it goes. Thank you so much for being so generous with your time. I really appreciate it, Adam.Adam: Thank you so much for having me, Corey. I really appreciate it.Corey: Adam Elmore, independent AWS consultant. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice along with an insulting comment that attempts to teach us exactly what we got wrong, but fails utterly because you're terrible at teaching things.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.
About TimTim's tech career spans over 20 years through various sectors. Tim's initial journey into tech started as a US Marine. Later, he left government contracting for the private sector, working both in large corporate environments and in small startups. While working in the private sector, he honed his skills in systems administration and operations for large Unix-based datastores.Today, Tim leverages his years in operations, DevOps, and Site Reliability Engineering to advise and consult with clients in his current role. Tim is also a father of five children, as well as a competitive Brazilian Jiu-Jitsu practitioner. Currently, he is the reigning American National and 3-time Pan American Brazilian Jiu-Jitsu champion in his division.Links Referenced:Twitter: https://twitter.com/elchefe TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by Honeycomb. When production is running slow, it's hard to know where problems originate. Is it your application code, users, or the underlying systems? I've got five bucks on DNS, personally. Why scroll through endless dashboards while dealing with alert floods, going from tool to tool to tool that you employ, guessing at which puzzle pieces matter? Context switching and tool sprawl are slowly killing both your team and your business. You should care more about one of those than the other; which one is up to you. Drop the separate pillars and enter a world of getting one unified understanding of the one thing driving your business: production. With Honeycomb, you guess less and know more. Try it for free at honeycomb.io/screaminginthecloud. Observability: it's more than just hipster monitoring.Corey: I come bearing ill tidings. Developers are responsible for more than ever these days. Not just the code that they write, but also the containers and the cloud infrastructure that their apps run on. Because serverless means it's still somebody's problem. And a big part of that responsibility is app security from code to cloud. And that's where our friend Snyk comes in. Snyk is a frictionless security platform that meets developers where they are - Finding and fixing vulnerabilities right from the CLI, IDEs, Repos, and Pipelines. Snyk integrates seamlessly with AWS offerings like code pipeline, EKS, ECR, and more! As well as things you're actually likely to be using. Deploy on AWS, secure with Snyk. Learn more at Snyk.co/scream That's S-N-Y-K.co/screamCorey: Welcome to Screaming in the Cloud. I'm Corey Quinn. A bit of a sad episode Today. I am joined by Duckbill Group principal cloud economist, Tim Banks, but by the time this publishes, he will have left the Duckbill nest, as it were. Tim, thank you for joining me, and can I just start by saying, this is sad?Tim: It is. I have really enjoyed being with Duckbill and I will never forget that message you sent me. It's like, “Hey, would you like to do this?” And I was like, “Boy would I.” It's been a fantastic ride and I have enjoyed working with a friend. And I'm glad that we remain friends to this day and always will be, so far as I can tell.Corey: Yes, yes. What you can't see while recording this, I'm actually sitting in the same room as Tim with a weapon pointed at him to make sure that he stays exactly on message. Yeah, I kid. There's been a lot that's happened over the last year. We only got to spend time together in person once at re:Invent. I think because re:Invent is such a blur for me, I don't remember who the hell I talk to.Someone can walk up and say, “Oh yeah, we met at re:Invent,” and I'll nod and say, “Oh yeah,” and I will have no recollection of that whatsoever. But you don't argue with people. But I do distinctly remember hanging out with you there. But since then, it's been a purely distributed company, purely distributed work.Tim: Yeah, that's the only time I've seen you since I've worked here. It's the only time I met Mike. But it's weird because it's like, someone you work with you see every day virtually and talk to, and then you actually get to, like, IRL them and like, “Oh, wow. I had all these, kind of, conceptions of, you know, what you are or who you are as a person, and then you get to, like, check yourself. Was I right? Was I wrong?” I was like, “Oh, you're taller than I thought; you're shorter than I thought,” you know, whatever it was.But I think the fun part about it was we all end up being so close by the nature of how we work that it was just like going back and seeing family after a while; you already know who they are and how they are and about them. So, it felt good, but it felt familiar. That's a great feeling to have. To me, that's a sign of a very successful distributed culture.Corey: Yeah, it's weird the kinds of friendships we've built during the pandemic. When I was in New York for the summit, I got to meet Linda Haviv at AWS for the first time, despite spending the past year or so talking to her repeatedly. As I referred to her the entire time I was in New York, this is Linda, my new old friend because that is exactly how it felt. It's the idea of meeting someone in person that you've had a long-term ongoing friendship with. It's just a really—it's a strange way Everything's new but it's not, all at the same time.It reminds me of the early days of the internet culture where I had more friends online than off, which in my case was not hard. And finally meeting them, some people were exactly like they were described and others were nothing at all like they presented. Now that we have Zoom and this constant level of Slack chatter and whatnot, it's become a lot easier to get a read on what someone is like, I think.Tim: I think so too, you know, we've gotten away—and I think largely because of the pandemic—of just talking about work at work, right? The idea of embracing, you know, almost a cliche of the whole person. But it's become a very necessary thing as people have dealt with pandemic, social upheaval, political climates, and whatever, while they're working from home. You can't compartmentalize that safely in perpetuity, right? So, you do end up getting to know people very well, especially in what their concerns are, what their anxieties are, what makes them happy, what makes them sad, things that go on in their lives.You bring all that to your distributed culture because it's not like you leave it at the door, when you walk out. You're not walking out anymore; you're walking to another room, and it's hard to walk away from those things in this day and age. And we shouldn't have to, right? I feel like for a successful and nurturing culture—whatever it is, whether it's tech culture, whether it's whatever kind of work culture—you can't say, “I only want your productivity and nothing else about you,” and expect people to sustain that. So, you see these companies are, like, you know, “We don't have political discussions. We don't have personal discussions. We're just about the work.” I'm like, “All right, well, that's not going to last.” A person cannot just be an automaton in perpetuity and expect them to grow and thrive.Corey: And this is why you're leaving. And I want to give that a little context because without, sounds absolutely freaking horrifying. You've been a strong advocate for an awful lot of bringing the human to work, on your philosophy around leadership, around management. And you've often been acting in that capacity throughout, I would say, the majority of your career. But here at The Duckbill Group, we don't have a scale of team where you being the director of the team or leader of the team is going to happen in anything approaching the near or mid-term.And so, much of your philosophy is great and all because it's easy to sit here at a small company and start talking about, “Oh, this is how you should be doing it.” You have the opportunity to wind up making a much deeper impact on a lot more people from a management perspective, but you do in fact, need a team to manage as opposed to sitting around there, “Oh, yeah. Who do you manage?” “This one person and I'm doing all of these things to make their life and job awesome.” It's like, “Yeah, how many hours a week are you spending in one-on-ones?” “20 to 25.”Okay, maybe you need a slightly larger team so you can diffuse that out a little bit. And we are definitely sad to be losing you; super excited to see where you wind up going next. This has been a long time coming where there are things that you have absolutely knocked out of the park here at The Duckbill Group, but you also have that growing—from what I picked up on anyway—need to set a good management example. And lord knows this industry needs more of those. So first, sad to lose you. Secondly, very excited for where you wind up next and what they're in for, even though it has a strong likelihood that they don't know the half of it yet.Tim: One of the things that I like about The Duckbill Group and how my time here has been is the first thing that I was asked in the interview was very sincere, like, “Well, what's your next job?” And I was very clear. It's like, “After this, I want to be a director or VP of engineering because I would like to be a force multiplier, right?” I would like to make engineering orgs better. I would like to make engineering practices better. I want to make the engineers better, right?And not by driving KPIs and not by management, right, not administrative functions. I want to do it via leadership. I want to do it by setting examples, making safe places for people, making people feel like they're important and invested in, nurturing them, right? I've said this before I—this analogy was getting me somewhere else and I love, it's like, if I plant a tree and I want it to grow apples, right, I'm not going to sit there and put a number down of apples it's expected to produce, and then put it on a performance plan if it doesn't get that number of apples, right? I need to nurture the tree, I need to fertilize it, I need to protect it, I need to keep it safe, I need to keep it safe from the elements, I need to make sure that it doesn't have parasites, I need to take care of that tree.And if that tree grows and it's healthy and it's thriving, it will produce, right? But I'm not—I can't just expect apples if I'm not taking care of the tree. Now, people are not trees, but you still have to take care of the people if you want them to do things. And if you can't take care of the people, if you can't manage the environment that they're in to make it safe, if you can't give them the things they need to be successful, then you're just going to be holding numbers over someone and expecting to hit them.And that doesn't work. That's not something that's sustainable. And it doesn't really—it's not even about how much you pay them. You must pay them well, right, but it has to be more than just that if you want people to succeed. And that doesn't necessarily mean—like, one thing is at the Duckbill Group I love, succeeding doesn't necessarily mean that I'm going to stay at—or your engineer is going to stay at one place in perpetuity. If you mentor and train and coach and give an engineer opportunity to grow and thrive and what they do is they go to another job for a title increase and a pay increase or something like that, you did your job.Corey: A lot of companies love to tell that lie and they almost convince themselves of it where I look at your resume, and great you have not generally crossed the two-year mark at companies for the last decade. I never did until I started at this place. But we magically always liked to pretend in job interviews that, “Oh yeah, this is my forever job—” like you're a rescue dog getting adopted or something, “—and I'm going to work here for 25 years and get a gold watch and a pension at the end of it.” It's lunacy. I have never seen the value in lying to ourselves like that, which is why we start our interviews with, “What's the job after this and how do we help you get there?”It's important that we ask those questions and acknowledge that reality. And the downside to it—if you can call it a downside—is you've got to live by it. It's not just words, you can slop onto an interview questionnaire; you actually have to mean it. People can see through insincerity.Tim: And it's one of the things, like, if you run an org and you grow your people and you don't have a place for them to grow into, you should expect and encourage them to find those opportunities elsewhere. It is not reasonable, I feel like, as a leader for you expect people to stay in a place where they have grown past or grown out of. You need to either need to give them a new pot to grow into or you need to let them move elsewhere and thrive and grow. And moving elsewhere—like, if you have a retention problem where you can't retain anybody, that's a problem, but if you have your junior engineers who become senior engineers at other places, right, and everyone leaves on good terms, and they got the role and you gave them a great recommendation and they give glowing recommendations to you, there's nothing wrong with that. That's not a failure; that's success.Corey: One bit of I would say pushback that I suspect you might get when talking to people about what's next is that, “Well, you are just a consultant, on some level, for a year.” You always know that someone is really arguing in good faith when they describe what you did with the word ‘just,' but we'll skip past that part. And it's, “You're just a consultant. What would you possibly know about team management and team dynamics?” And there is a little bit of truth to that insofar as the worst place in the world to get management advice is very clearly on Twitter.It turns out that most interpersonal scenarios are, one, far too personal to wind up tweeting about, and two, do not lend themselves to easy solutions that succinctly fit within 280 characters. Imagine that. The counter-argument though, is that you have—correctly from where I sit—identified a number of recurring dynamics on teams that you have encountered and worked with deeply as a large number of engagements. And these are recurring things, I want to be clear. So, I'm not talking about one particular client. If you're one of our clients and listen to this thinking that we're somehow subtweeting you with our voices—I don't know what that is; subwoofing, maybe?Tim: [crosstalk 00:12:05]—Corey: Is that what a subwoofer is? I'm not an audio person.Tim: Throwing shade, we'll just say—call it throwing shade.Corey: Yeah, we're not throwing shade at any one person, team, or group in particular; these are recurring things. Tim, what have you seen?Tim: And so, I think the biggest thing I see is folks that are on the precipice of a big technological change, right, and there is an extraordinary amount of anxiety, right? I've seen a number of customers through our engagements that, “We are moving away from this legacy platform,” or from this thing that we have been doing for X amount of time. And everyone has staked the other domains, staked out their areas of expertise and control and we're going to change that. And the solution to that is not a technical solution. You don't fix that by Helm charts, or Terraform, or CloudFormation. You fix that by conversations, and you fix that by listening. You fix that by finding ways to reassure folks and giving them confidence in their ability to adjust and thrive in a new environment.If you take somebody who's been, you know, an Oracle admin for 20 years, and you going to say, “Great. Now, you're going to learn, you're going to do this an RDS,” that's a whole new animal, and folks feel like, well, you know, I can't learn something new like that? Well, yeah you can. If you can learn Oracle, you can learn anything. I firmly believe that.But that's one of the conversations we have, it's never, almost never a technical problem folks have. We need to reassure people, right? And so, folks who reach out to us, it's typically folks who are trying to get their organizations in that direction. Another thing we see sometimes is that we find that there's a disconnect between leadership and the engineers. They have either different priorities or different understandings of what's going on. And we come in to solve a problem, which may be cost but that's not the problem we actually solve. The problem we actually solve is fixing this communication bridges between management and leadership.And that's almost an every time occurred. At some point or another, there's some disconnect there. And that's the best part of the job. Like, the reason I do this consulting gig is not because I want to bang away at code. If I've had to do that, that's an anomaly for sure because I want to have these conversations.And people want to have these conversations; they want to get these problems solved and sometimes they don't know how to. And that is the common thing, I think, through all of our customers. Like, we need some amount of expertise to help us find solutions to these things that aren't necessarily technical problems. And I think that's where we run into problems as an industry, right, where we think a lot of things are technical problems or have technical solutions, and they don't. There are people problems. They're—Corey: Here at The Duckbill Group, we're basically marriage counseling for engineering and finance in many cases.Tim: We really are.Corey: This is why were people not software.Tim: Yeah. And I will say this very firmly and you can quote me on this: like, you cannot replace us. You cannot replace the kind of engagements we do with software. You can't. Can't be done, right? Software is not empathetic.Corey: There are a whole series of questions we ask our clients at the start of an engagement and the answers to those questions change what we ask them going forward. In fact, even the level-setting in the conversation that we have at the start of that changes the nature of those. We're not reading from a list; we're trying to build an understanding. There is a process around what we do, but it's not process that can ever be scoped down to the point where it's just a list of questions or a questionnaire that isn't maddening for people to fill out because it's so deeply and clearly misses the mark around context of what they're actually doing.Tim: Mm-hm. Our engaged with their conversations. That's all they are. They're really in-depth conversations where we're going to start asking questions and we're going to ask questions about those answers. We're start pulling out strings and kicking over rocks and seeing what we find.And that's the kind of thing that, you know, you would expect anyone to do who's coming in and saying, “Okay, we have a problem. Now, let's figure it out.” Right? Well, you can't just look at something on the surface, and say, “Oh, I know what this is.” Right? You know, for someone to say, “Oh, I know how to fix this,” when they walk in is the surest way to know that someone doesn't know what they're talking about, right?Corey: Oh, easiest thing in the world is to walk in and say, “This is broken and wrong.” That can translate directly to, “Hi, I am very junior. Please feed my own ass to me.” Because no one shows up at work thinking they're going to do a crap job today on purpose. There's a reason things are the way that they are.Tim: Mm-hm. And that's the biggest piece of context we get from our customers is we can understand what the best practices are. You can go Google them right now and say, “This is the ten things you're supposed to do all the time,” right? And we would be really, really crappy consultants if we just read off that list, right? We need to have context: does this thing make sense? Is this the best practice? Maybe, but we want to know why you did it this way.And after you tell us that way, I'm like, “You know what? I would do it the exact same way for this use case.” And that's great. We can say like, “This is the best way to do that. Good job.” It's atypical; it's unusual, but it solves the problems that you need solving.And that's where I think a lot of people miss. Like, you know, you can go—and not to throw shade at AWS's Trusted Advisor, but we're going to throw shade at AWS's Trusted Advisor—and the fact that it will give you—Corey: It is Plausible Advisor at absolute best.Tim: [laugh]. It will give you suggestions that have no context. And a lot of the automated AI things that will recommend that you do this and this and this and this are pretty much all the same. And they have no context because they don't understand what you're trying to do. And that's what makes the difference between people. There's these people problems.And so, one of the things that I think is really interesting is that we have moved into doing a shorter engagement style that is very short. It's very quick, it's very kind of almost tactical, but we go in, we look at your bill, we ask you some questions, and we're going to give you a list of suggestions that are going to save you a significant amount of money right away, right? So, a lot of times, folks when they need quick wins, or they don't really need us to deep-dive into all their DynamoDB access patterns, right? They just want like, “Hey, what are the five things we can do to save us some money?” And we're like, “Well, here they are. And here's what we think they're going to save you.” And folks who really enjoyed that type of engagement. And it's one of my favorite ones to do.Corey: This episode is sponsored in part by LaunchDarkly. Take a look at what it takes to get your code into production. I'm going to just guess that it's awful because it's always awful. No one loves their deployment process. What if launching new features didn't require you to do a full-on code and possibly infrastructure deploy? What if you could test on a small subset of users and then roll it back immediately if results aren't what you expect? LaunchDarkly does exactly this. To learn more, visit launchdarkly.com and tell them Corey sent you, and watch for the wince.Corey: I can also predict that people are going to have questions for you—probably inane—of, well, you were a consultant, how are your actual technical chops? And I love answering these questions with data. So, I have here pulled up the last six months of The Duckbill Group's AWS bills. And for those who are unaware, every cloud economist has their own dedicated test account for testing out strange things that we come across. And again, can the correct answer in many consulting engagements is, “I don't know, but I'll find out.”Well, this is how we find out. We run tests and learn these things ourselves. I suppose we could extend this benefit, if you want to call it that, to people who aren't cloud economists but I'm not entirely sure what, I don't know, an audio engineer is going to do with an AWS account that isn't, you know, kind of horrifying. To the audio engineer that is editing this podcast, my condolences if you take that as a slight, and if there is something you would use an AWS account for, please let me know. We'll come talk about it here.But back to topic, looking at the last six months of your bill for your account—that's right, a ritualistic shaming of the AWS bill—in January you spent $16.06. In February, you spent 44 cents. And you realized that was too high, so back in March, you then spent 19 cents. And then $3.01 back in April. May wound up $10.02, and now you're $9.84 as of June. July has not yet finalized as of this recording.And what I want to highlight—and what that tells me when I look at these types of bills—and I assure you as the world's leading self-described expert in AWS billing, I'm right; listening to me is a best practice on these things—that shows the exact opposite of a steady-state workload. There's a lot of dynamism to those giant swings because we don't have cloud economists who are going to just run these things steady-state for the rest of our lives. Those are experiments of building and testing out new and exciting things in a whole bunch of very weird, very strange ways. Whenever I wind up talking to someone in one of the overarching AWS services at AWS and I pull up my account, a common refrain is, “Wow, you use an awful lot of services.” Right. I'm not just sitting here run and EC2 instances forever. Imagine that. And your account is a perfect microcosm of that entire philosophy.Tim: Well, I don't know all the answers, right? And I will never profess all the answers. And before I say, “You should do this—” or maybe I will say, “You might be able to do this. Let me go save as possible.” [laugh]. Right? And so, just let me just see, can you do this? Does this work? No, I guess it doesn't. Or AWS docu—especially, “The AWS documentation says this. Let me see if that's actually the case.”Corey: I don't believe that they intend to lie, but—Tim: No.Corey: —they also certainly don't get it correct all the time.Tim: And to be fair, they have, what, 728 services by this point, and that's a lot of documentation you're not going to get—Corey: Three more have launched since the start of this recording.Tim: I—yeah, actually—well, by the time this hits, they're probably going to have 22. But we'll [laugh] see. But yeah, no. And that's fine. And they're not going to have every use case, and every edge, kind of like, concern handled, and so that's why we need to kick the tires a little bit.And what I think more than anything else is, you know, sometimes we just do things out of convenience. Like, “Well, I don't want to run this on this; let me just fire it up because it's not my money.” [laugh]. But we also want to be fairly concerned about you know, how we do things. You don't want to run a fleet of z1ds, obviously.But there is a certain amount of tire-kicking and infrastructure spinning up that you have to do in order to maintain freshness, right? And it's not a thing where I'm going to say, “Oh, I know YAML off the top of my head, and I need to do—you know, I'm up to speed on every single possible API call that you can make.” No. My technical prowess has always been in architecture and operations. So, I think when we have these conversations, folks mostly tend to be impressed by not only business acumen and strategy, but also being able to get down to the weeds and talking with the developers and the engineers about the minutia. And you will have seen you know, the feedback that I've gotten about my technical prowess has always been good. You know, I can hang with anybody, I feel like.Corey: I would agree wholeheartedly. It's been really interesting watching you in conversations, internally and with our clients, where you will just idly bust out something fricking brilliant out of left field. And most of the time, I don't think you even realize it. It's just one of those things that makes intuitive and instinctive sense to you. And you basically just leave people stunned and their scribbling notes and trying to wrap their heads around what you just said.And it's adorable because sometimes you wind up almost, like, looking embarrassed, like, “Did I say something rude and not realize it? Like, I wasn't trying to be insulting.” It's like, “Nope, nope. You're just doing your thing, Tim. Just keep on doing it. That's fine.”Tim: Yeah, it's funny because, like you, one of the things that I've really enjoyed about it is, like, we'll just start bouncing ideas off of each other and come up with something brilliant. “Yeah, let's do that.” And then, “Okay, this is now a thing.” And it's like, you know, there's something to be said about being around smart people. So, it's not just me coming up with something brilliant; these are almost always fruits of a conversation and discussion being had, and then you formulate something great in your head.But again, this is why I love the aspect of talking and having conversations with people, so that way you can come up with something kind of brilliant. None of this is done in a silo. Like we're not really, really good at what we do because we don't rely or talk to or have conversations with other people.Corey: One thing that you did that I think is one of the most transformative things that has happened in company history in some respects has been when you started, and for the first half of your tenure here, we had two engagement types that we would wind up giving our consulting clients. There's contract negotiation, where we help companies negotiate their long-term commitment contracts with AWS—and we're effective at it and that's fun; that's basically what you would more or less expected to be—and the other is our cost optimization project engagements. And those tend to look six to eight weeks where we wind up going in deep-dives into the intricacies of an organization's AWS accounts, bills, strategy, growth plan, et cetera, et cetera, et cetera, to an exhaustive level of detail. And in an interest of being probably overly transparent here, I didn't like working on those engagements myself. I like coming in, finding the big things that will be transformative to reduce the bills—it's like solving a puzzle—and then the relatively in-depth analysis for things that are a relatively paltry portion of the AWS bill does not really lead me to enjoying the work very much.And I beat my head against that one for years. And you busted out one day with an idea that became our third type of engagement, which is the first pass, where we charge significantly less for the engagement and it essentially distills down into you get us to talk to your engineering teams for a day. Bring us any questions, give us access in advance to these things, and we will basically go on a whirlwind guided tour and lay waste to your AWS bill and highlight different opportunities that we see to optimize these things. And it has been an absolute smash success. People love the engagements.Very often, it leads to that second full-bore engagement that I was describing earlier, but it also aligns very well with the way that I like to think about these things. I'm a great consultant, specifically because once I've delivered the value, I like to leave. Whereas as an employee, I just sort of linger around, and then I go cause problems and other people's departments—ideally, not on purpose, but you know, I am me—and this really emphasizes that and keeps me moving quickly. I really, really like that engagement style and I have you to thank for coming up with the idea and finding a way to do it that didn't either not resonate with the market—in which case, we're not selling a damn thing—or wound up completely eviscerating the value of the longer-term deep-dive engagements, and you threaded that needle perfectly.Tim: I thank you; I appreciate that. There was this kind of vacuum that I saw where, both from a cost and from a resource point where six to eight weeks is a long time for an engineering org to dedicate to any one thing, especially if that one thing isn't directly making money. But engineering orgs are also very interested in saving money. But it's especially in smaller orgs where that velocity is very important, they don't have six to eight weeks for that. They can't dedicate the resources to those deep-dives all the time, and all the conversations we—and when we do a COP, it is exhaustive. We are exploring every avenue to almost an absurd level, right?And that's not the right engagement for a lot of orgs, right? So, coming in and saying, “Hey, you know, this is a quick one; these are the things that you can do. This is 90% of the savings you're going to realize. These things: bam, bam, bam, bam, bam.” Right?And then we give it to the folks and we let them work on it, and then they're like, “Hey, we need this because we want to negotiate EDP,” or, “We need this because, you know, we're just trying to make sure that our costs are in line so we can be more agile, so we can do this project, or whatever.” Right? And then there are a lot of other orgs that do need that exhaustive kind of thing, larger orgs especially, right? Larger, more complex orgs, orgs that are trying to maybe—like, if you're trying to make a play to get acquired, you want to get this very, very in-depth study so you know all your liabilities and all your assets, so that way you can fix those problems and make it very attractive for someone to buy you, right? Or orgs that just have, like, we are not having an impending EDP; we have a lot of time to be able to focus on these things, and we can build this into the roadmap, right?Then we can do a very exhaustive study of those things. But for a lot of times, people are just like, “Look, I just need to save X amount of money on my AWS bill and can you do that?” Well, sure. We can go in there and have those conversations and give you a lot of savings. And I'm very much in the camp of, you know, ‘perfect is the enemy of good.' I don't have to save down to the nth penny on your DynamoDB bill. But if I can, shave—cut it in half, that's great. Most people are very happy about those kinds of things. And that's a very routine finding for us.Corey: One other aspect that I really liked about it, too, is that it let us move down market a bit, away from companies that are spending millions of dollars a month. Because yeah, the ROI for those customers is a slam dunk on virtually any engagement that we could put together, but what about the smaller companies, the ones that are not spending that much money, yet? They've never felt great talk to them and say, “Oh, just go screw up your AWS bill some more. Then, then you will absolutely be able to generate some value. Maybe turn off MFA and post your credentials to GitHub or something. That'll speed up the process nicely.”That's terrible advice and we can't do it. But this enables us to move down to smaller companies that are earlier in their cloud estate build-out or are growing organically rather than trying to do a giant migration as sort of greenfield growth approach. I really, really like our ability to help companies that are a bit earlier in their cloud journey, as well as in smaller environments, just because I guess, on some level, for me, at least, when you see enormous multimillion-dollar levels of spend, the misconfigurations are generally less fun to find; they're less exciting. Because, yeah at a small scale, you can screw up and your Managed NAT Gateway bill is a third of your spend. When you're spending $80 million a year, you're not wasting that kind of money on Managed NAT Gateways because that misconfiguration becomes visible from frickin' orbit.So, someone has already found that stuff. And it's always then it's almost certainly EC2, RDS, and storage. Great. Then there's some weird data transfer stuff and it starts to look a lot more identical. Smaller accounts, at least from my perspective, tend to have a lot more of interesting things to learn hiding in the shadows.Tim: Oh, absolutely. And I think the impact that you make for the future for small companies much higher, right? You go in there and you have an engagement, you can say, “Okay, I understand the business reason why you did this here, but if you make these changes—bam, bam, bam—12 to 18 months and on, right, this is going to make a huge difference in your business. You're going to save a tremendous amount of money and you're going to be much more agile.”You did this thing because it worked for the POC, it worked for the MVP, right? That's great, but before it gets too big and becomes load-bearing technical debt, let's make some changes to put you in a better position, both for cost optimization and an architectural future that you don't have to then break a bone that's already set to try and fix it. So, getting in there before there's a tremendous load on their architecture—or rather on their infrastructure, it's super, super fun because you know that when you've done this, you have given that company more runway, or you've given them the things they need to actually be more successful, and so they can focus their time and efforts on growth and not on trying to stop the bleeding with their AWS bill.Corey: Tim, it's been an absolute pleasure to work with you. I'm going to miss working with you, but we are definitely going to remain in touch. Where can people find you to follow along with your continuing adventures?Tim: The best way to find me is on Twitter, I am @elchefe—E-L-C-H-E-F-E. And yeah, I will definitely keep in touch with you, Corey. Again, you have been a tremendous friend and I really appreciate you, your insights, and your honesty. Our partners are friends with each other and I do not think that they will let us ever drift too far apart. So.Corey: No, I think it is pretty clear that we are basically going to be both of their plus-ones forever.Tim: [laugh]. I think so.Corey: I'm just waiting for them when they pulled the prank of dressing us the exact same way because our styles are somewhat different, and I'm pretty sure that there's not a whole lot of convergence where we both wind up looking great. So, it's going to be hilarious regardless of what direction it goes in.Tim: Well, you do have velour tracksuits too, right?Corey: Not yet, but please don't tell that to Bethany.Tim: [laugh].Corey: Tim, it has been an absolute pleasure.Tim: The pleasure has been all mine, Corey. I really appreciate it.Corey: Tim Banks, for one last time, principal cloud economist at The Duckbill Group. I am Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice and an insulting comment that says that we are completely wrong in our approach to management and the real answer is as follows, making sure to keep that answer less than 280 characters.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.
About CarlaCarla Stickler is a professional multi-hyphenate advocating for the inclusion of artists in STEM. Currently, she works as a software engineer at G2 in Chicago. She loves chatting with folks interested in shifting gears from the arts to programming and especially hopes to get more women into the field. Carla spent over 10 years performing in Broadway musicals, most notably, “Wicked,” “Mamma Mia!” and “The Sound of Music.” She recently made headlines for stepping back into the role of Elphaba on Broadway for a limited time to help out during the covid surge after not having performed the role for 7 years. Carla is passionate about reframing the narrative of the “starving artist” and states, “When we choose to walk away from a full-time pursuit of the arts, it does not make us failed artists. The possibilities for what we can do and who we can be are unlimited.”Links Referenced: G2: https://www.g2.com/ Personal website: https://carlastickler.com Instagram: https://www.instagram.com/sticklercarla/ TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by Honeycomb. When production is running slow, it's hard to know where problems originate. Is it your application code, users, or the underlying systems? I've got five bucks on DNS, personally. Why scroll through endless dashboards while dealing with alert floods, going from tool to tool to tool that you employ, guessing at which puzzle pieces matter? Context switching and tool sprawl are slowly killing both your team and your business. You should care more about one of those than the other; which one is up to you. Drop the separate pillars and enter a world of getting one unified understanding of the one thing driving your business: production. With Honeycomb, you guess less and know more. Try it for free at honeycomb.io/screaminginthecloud. Observability: it's more than just hipster monitoring.Corey: What if there were a single place to get an inventory of what you're running in the cloud that wasn't "the monthly bill?" Further, what if there were a way to compare that inventory to what you were already managing via Terraform, Pulumi, or CloudFormation, but then automatically add the missing unmanaged or drifted parts to it? And what if there were a policy engine to immediately flag and remediate a wide variety of misconfigurations? Well, stop dreaming and start doing; visit snark.cloud/firefly to learn more.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn, there seems to be a trope in our industry that the real engineers all follow what more or less looks like the exact same pattern, where it's you wind up playing around with computers as a small child and then you wind up going to any college you want—as long as it's Stanford—and getting a degree in anything under the sun—as long as it's computer science—and then all of your next jobs are based upon how well you can re-implement algorithms on the whiteboard. A lot of us didn't go through that path. We wound up finding our own ways to tech. My guest today has one of the more remarkable stories that I've come across. Carla Stickler is a software engineer at G2. Carla, thank you for agreeing to suffer my slings and arrows today. It's appreciated.Carla: Thanks so much for having me, Corey.Corey: So, before you entered tech—I believe this is your first job as an engineer and as of the time we're recording this, it's been just shy of a year that you've done in the role. What were you doing before now?Carla: Oh, boy, Corey. What was I doing? I definitely was not doing software engineering. I was a Broadway actress. So, I spent about 15 years in New York doing musical theater, touring around the country and Asia in big Broadway shows. And that was pretty much all I did.I guess, I also was a teacher. I was a voice teacher and I taught voice lessons, and I had a studio and I taught it a couple of faculties in New York. But I was one hundred percent ride-or-die, like, all the way to the end musical theater or bust, from a very, very early age. So, it's been kind of a crazy time changing careers. [laugh].Corey: What inspired that? I mean, it doesn't seem like it's a common pattern of someone who had an established career as a Broadway actress to wake up one day and say, “You know what I don't like anymore. That's right being on stage, doing the thing that I spent 15 years doing. You know what I want to do instead? That's right, be mad at computers all the time and angry because some of the stuff is freaking maddening.” What was the catalyst that—Carla: Yeah, sounds crazy. [laugh].Corey: —inspired you to move?Carla: It sounds crazy. It was kind of a long time coming. I love performing; I do, and it's like, my heart and soul is with performing. Nothing else in my life really can kind of replace that feeling I get when I'm on stage. But the one thing they don't really talk about when you are growing up and dreaming of being a performer is how physically and emotionally taxing it is.I think there's, like, this narrative around, like, “Being an actor is really hard, and you should only do it if you can't see yourself doing anything else,” but they don't actually ever explain to you what hard means. You know, you expect that, oh, there's going to be a lot of other people doing it in, I'm going to be auditioning all the time, and I'm going to have a lot of competition, but you never quite grasp the physical and emotional toll that it takes on your body and your—you know, just ongoing in auditions and getting rejections all the time. And then when you're working in a show eight times a week and you're wearing four-inch heels on a stage that is on a giant angle, and you're wearing wigs that are, like, really, really massive, you don't really—no one ever tells you how hard that is on your body. So, for me, I just hit a point where I was performing nonstop and I was so tired. I was, like, living at my physical therapist's office, I was living at, like, my head therapist's office.I was just trying to, like, figure out why I was so miserable. And so, I actually left in 2015, performing full time. So, I went to get my Master's in Education at NYU thinking that teaching was my way out of performing full-time.Corey: It does seem that there's some congruities—there's some congruities there between your—instead of performing in front of a giant audience, you're performing in front of a bunch of students. And whether it's performing slash educating, well that comes down to almost stylistic differences. But I have a hard time imagining you just reading from your slides.Carla: Yeah, no, I loved it because it allowed me to create connections with my students, and I found I like to help inspire them on their journeys, and I really like to help influence them in a positive way. And so yeah, it came really natural to me. And my family—or I have a bunch of teachers in my family so, you know, teaching was kind of a thing I just assumed I would be good at, and I think I fell naturally into. But the thing that was really hard for me was while I was teaching, I was still… kind of—I had, like, one foot in performing. I was still, like, going in and out of the show that I've been working on, which I didn't mention.So, I was in Wicked for, like, ten years, that's kind of like my claim to fame. And I had been with that show for a really long time, and that was why—when I left to go teach, that was kind of my way out of that big show because it was hard for me to explain to people why it was leaving such a giant show. And teaching was just, like, a natural thing to go into. I felt like it was like a justifiable action, [laugh] you know, that I could explain to, like, my parents for why I was quitting Broadway.So, you know, I love teaching and—but I—and so I kept that one foot kind of in Broadway, and I was still going in and out of the show. It's like a vacation cover, filling in whenever they needed me, and I was still auditioning. But I was like, I was still so burned out, you know? Like, I still had those feelings of, like—and I wasn't booking work; I think my heart just wasn't really in it. Like, every time I'd go into audition, I would just feel awful about myself every time I left.And I was starting to really reject that feeling in my life because I was also starting to find there were other things in my life that made me really happy. Like, just having a life. Like, I had—for the first time in a very long time, I had friends that I could hang out with on the weekends because I wasn't working on the weekend. And I was able to, like, go to, you know, birthdays and weddings and I was having, like, this social life. And then every time I would go on an audition—Corey: And they did other things with their lives, and it wasn't—Carla: Yeah.Corey: All shop talk all the time—Carla: Right.Corey: Which speaking as someone who lives in San Francisco and worked in normal companies before starting this ridiculous one, it seems that your entire social circle can come out of your workplace. And congratulations, it's now all shop talk, all the time. And anyone you know or might be married to who's not deeply in tech just gets this long-suffering attitude on all of it. It's nice to be able to have varied conversations about different things.Carla: Yes. And so, I was like having all these, like—I was, like, having these life moments that felt really good, and then I would go to an audition and I would leave being, like, “Why do I do that to myself? Why do I need to feel like that?” Because I just feel awful every time I go. And so, then I was having trouble teaching my students because I was feeling really negative about it, and I was like, “I don't know how to encourage you to go into a business that's just going to, like, tear you down and make you feel awful about yourself all the time.”Corey: And then you got into tech?Carla: [laugh]. And then I was just, like, “Tech. That's great.” No, I—do you know what—Corey: Like, “I'm sad all the time and I feel like less than constantly. You know what I'm going to use to fix that? I'm going to learn JavaScript.” Oh, my God.Carla: Yeah. I'm going to just challenge myself and do the hardest thing I can think of because that's fun. But ki—I mean, sort of I [laugh] I, I was not ever—like, being an engineer was never, like, on my radar. My dad was an engineer for a long time, and he kind of always would be, like, “You're good at math. You should do engineering.”And I was like, “No, I'm an actor. [laugh]. I don't want to do that.” And so, I kind of always just, like, shooed it away. And when a friend of mine came to my birthday party in the summer of 2018, who had been a songwriter and I had done some readings of a musical of his, and he was like, “I'm an engineer now at Forbes. Isn't that great?”And I was like, “What? How does that happen? I need you to back up, explain to me what's going on.” And I just, like—but I went home and I could not stop thinking about it. I don't know if it was like my dad's voice in the back of my head, or there was like the stars aligned.My misery that I was feeling in my life, and, like, this new thing that just got thrown in my face was just such an exciting, interesting idea. I was like, “That sounds—I don't know what—I don't even know what that looks like or I don't even know what's involved in that, but I need to figure out how to do it.” And I went home when I first started teaching myself how to do it. And I would just sit on my couch and I would do, like, little coding challenges, and before I knew it, like, hours would have passed by, I forgot to eat, I forget to go to the bathroom. Like, I would just be, like, groove on the couch from where I was sitting for too long.And I was like, oh, I guess I really liked this. [laugh]. It's interesting, it's creative. Maybe I should do something with it.Corey: And then from there, did you decide at some point to pursue—like, a lot of paths into tech these days. There's a whole sea of boot camps, for example, that depending on how you look at them are either inspirational stories of how people can transform their lives, slash money-grabbing scams. And it really depends on the boot camp in particular, is that the path you took? Did you—Carla: Yes.Corey: Remain self-taught? How did you proceed from—there's a whole Couch-to-5k running program; what is about—I guess we'll call getting to tech—but what was your Couch-to-100k path?Carla: Yeah, I was just going to say, Couch-to-100k tech gig.Corey: Yeah.Carla: So, my friend to had gone to Flatiron School, which is a boot camp. I think they have a few locations around the country, and so I initially started looking at their program just because he had gone there, and it sounded great. And I was like, “Cool, great.” And they had a lot of free resources online. They have, like, this whole free, like, boot camp prep program that you can do that teaches Rails and JavaScript.And so, I started doing that online. And then I—at the time, they had, like, a part-time class. I like learning in person, which is funny because now I just work remote and I do everything on Google… it's like, Google and Stack Overflow. So—but I knew at the time—Corey: I have bad news about the people who are senior. It doesn't exactly change that much.Carla: Yeah, that's what I've heard, so I don't feel bad about telling people that I do it. [laugh].Corey: We're all Full Stack Overflow developers. It happens.Carla: Exactly. So yeah, I just. They had, like, a part-time front-end class that was, like, in person two nights a week for a couple months. And I was like, “Okay, that'll be a really good way to kind of get my feet wet with, like, a different kind of learning environment.”And I loved it. I fell in love with it. I loved being in a room of people trying to figure out how to do something hard. I liked talking about it with other people. I liked talking about it with my teachers.So, I was like, “Okay, I guess I'm going to invest in a boot camp.” And I did their, like, immersive, in-person boot camps. This was 2019 before everything shut down, so I was able to actually do it in person. And it was great. It was like, nine to six, five days a week, and it was really intense.Did I remember everything I learned when it was over? No. And did I have to, like, spend a lot of time relearning a lot of things just so I could have, like, a deeper understanding of it. Yes. But, like, I also knew that was part of it, you know? It's like, you throw a lot of information out you, hope some of it sticks, and then it's your job to make sure that you actually remember it and then know how to use it when you have to.Corey: One of the challenges that I've always found is that when I have a hobby that I'm into, similar to the way that you were doing this just for fun on your couch, and then it becomes your full-time focus, first as a boot camp and later as a job, that it has a tendency in some cases to turn a thing that you love into a thing that you view is this obligation or burden. Do you still love it? Is it still something that you find that's fun and challenging and exciting? Or is it more a means to an end for you? And there is no wrong answer there.Carla: Yeah, I think it's a little bit of both, right? Like, I found it was a creative thing I could do that I enjoy doing. Am I the most passionate software engineer that ever lived? No. Do I have aspirations to be, like, an architect one day? Absolutely not. I really, like, the small tickets that I do that are just, like, refactoring a button or, you know, like, I find that stuff creative and I think it's fun. Do I necessarily want to—Corey: You can see—Carla: —no.Corey: The results immediately as [crosstalk 00:15:15]—Carla: Yeah.Corey: More abstract stuff. It's like, “Well, when this 18 months migration finishes, and everything is 10% faster, oh, then I'll be vindicated.”Carla: Yeah. No.Corey: It's a little more attenuated from the immediate feedback.Carla: Yeah. I'm not that kind of developer, I'm learning. But I'm totally fine with that. I have no issue. Like, I am a very humble person about it. I don't have aspirations to be amazing.Don't ask me to do algorithm challenges. I'm terrible at them. I know that I'm terrible at them. But I also know that you can be a good developer and be terrible algorithm, like, challenges. So, I don't feel bad about it.Corey: The algorithm challenge is inherently biased for people who not only have a formal computer science education but have one relatively recently. I look back at some of the technical challenges I used to give candidates and take myself for jobs ten years ago, and I don't remember half of it because it's not my day-to-day anymore. It turns out that most of us don't have a job implementing quicksort. We just use the one built into the library and we move on with our lives to do something interesting and much more valuable, like, moving that button three pixels left, but because of CSS, that's now a two-week project.Carla: Yeah. Add a little border-radius, changes the su—you know. There are some database things I like. You know, I'm trying to get better at SQL. Rails is really nice because we use Active Record, and I don't really have to know SQL.But I find there are some things that you can do in Rails that are really cool, and I enjoyed working in their console. And that's exciting. You know when you write, like, a whole controller and then you make something but you can only see it in the console? That's cool. I think to me, that's fun. Being able to, like, generate things is fun. I don't have to always see them, like, on the page in a visual, pretty way, even though I tend to be more visual.Corey: This episode is sponsored in parts by our friend EnterpriseDB. EnterpriseDB has been powering enterprise applications with PostgreSQL for 15 years. And now EnterpriseDB has you covered wherever you deploy PostgreSQL on premises, private cloud, and they just announced a fully managed service on AWS and Azure called BigAnimal, all one word.Don't leave managing your database to your cloud vendor because they're too busy launching another half dozen manage databases to focus on any one of them that they didn't build themselves. Instead, work with the experts over at EnterpriseDB. They can save you time and money, they can even help you migrate legacy applications, including Oracle, to the cloud.To learn more, try BigAnimal for free. Go to biganimal.com/snark, and tell them Corey sent you.Corey: One of the big fictions that we tend to have as an industry is when people sit down and say, “Oh, so why did you get into tech?” And everyone expects it to be this aspirational story of the challenge, and I've been interested in this stuff since I was a kid. And we're all supposed to just completely ignore the very present reality of well, looking at all of my different opportunities, this is the one that pays three times what the others do. Like, we're supposed to pretend that money doesn't matter and we're all following our passion. That is actively ridiculous from where I sit.Carla: Mm-hm.Corey: Do you find that effectively going from the Broadway actress side of the world to—where, let's be clear, in the world of entertaining and arts—to my understanding—90% of people in that space are not able to do that as their only gig without side projects to basically afford to eat, whereas in tech, the median developer makes an extremely comfortable living that significantly outpaces the average median income for a family of four in the United States. Do you find that it has changed your philosophy on life in any meaningful way?Carla: Oh, my God, yeah. I love talking about on all of my social platforms the idea that you can learn tech skills and you can—like, there are so many different jobs that exist for an engineer, right? There are full-time jobs. There are full-time job that are flexible and they're remote, and nobody cares what time you're working as long as you get the work done. And because of that and because of the nature of how performing and being an artist works, where you also have a lot of downtime in between jobs or even when you are working, that I feel like the two go very, very well together, and that it allows—if an artist can spend a little bit of time learning the skill, they now have the ability to feel stable in their lives, also be creative how they want to, and decide what the art looks like for them without struggling and freaking out all the time about where's my next meal going to come from, or can I pay my rent?And, like, I sometimes think back to when I was on tour—I was on tour for three years with Wicked—and I had so much free time, Corey. Like, if I had known that I could have spent some time when I was just like hanging out in my hotel room watching TV all day, like, learning how to code. I would have been—I would have done this years ago. If I had known it was even, I don't even know actually if it was an option back then in, like, the early-2010s. I feel like boot camps kind of started around then, but they were mostly in person.But if I was—today, if I was right now starting my career as an artist, I would absolutely learn how to code as a side hustle. Because why wait tables? [laugh]. Why make, like, minimum wage in a terrible job that you hate when you can I have a skillset that you can do from home now because everything is remote for the most part? Why not?It doesn't make sense to me that anybody would go back to those kind of awful side gigs, side hustle jobs. Because at the end of the day, side hustle jobs end up actually being the things that you spend more time doing, just because theater jobs and art jobs and music jobs are so, you know, far apart when you have them. That might as well pick something that's lucrative and makes you feel less stressed out, you know, in the interim, between gigs. I see it as kind of a way to give artists a little more freedom in what they can choose to do with their art. Which I think is… it's kind of magical, right?Like, it takes away that narrative of if you can't see yourself—if you can see yourself doing anything else, you should do it, right? That's what we tell kids when they go into the arts. If you can see yourself doing any other thing, you know, you have to struggle to be an artist; that is part of the gig. That's what you sign up for. And I just call bullshit on it, Corey. I don't know if I can swear on this, but I call bullshit on [crosstalk 00:21:06]—[laugh].Corey: Oh, you absolutely can.Carla: I just think it's so unfair to young people, to how they get to view themselves and their creativity, right? Like, you literally stunt them when you tell them that. You say, “You can only do this one thing.” That's like the opposite of creative, right? That's like telling somebody that they can only do one thing without imagining that they can do all these other things. The most interesting artists that I know do, like, 400 things, they are creative people and they can't stop, right? They're like multi-hyphenates [crosstalk 00:21:39].Corey: It feels like it's setting people up for failure, on some level, in a big way where when you're building your entire life toward this make-or-break thing and then you don't get it, it's, well, what happens then?Carla: Yeah.Corey: I've always liked the idea of failure as a step forward. And well, that thing didn't work out; let's see if we can roll into it and see what comes out next. It's similar to the idea of a lot of folks who are career-changing, where they were working somewhere else in a white-collar environment, well time to go back to square one for an entry-level world. Hell with that. Pivot; take a half step toward what you want to be doing in your next role, and then a year or so later, take the other half step, and now you're doing it full time without having to start back at square one.I think that there are very few things in this world that are that binary as far as you either succeed or you're done and your whole life was a waste. It is easy get stuck in this idea that if your childhood dream doesn't come true, well give up and prepare for a life of misery. I just don't accept that.Carla: Yeah, I—Corey: But maybe it's because I have no choice because getting fired is my stock-in-trade. So, it wasn't until I built a company where I can't get fired from it that I really started to feel a little bit secure in that. But it does definitely leave its marks and its damages. I spent 12 years waiting for the surprise meeting with my boss and someone I didn't recognize from HR where they don't offer you coffee—that's always the tell when they don't offer coffee—and to realize it while I'm back on the job market again; time to find something new. It left me feeling more mercenary that I probably should have, which wasn't great for the career.What about you? Do you think that—did it take, on some level, a sense of letting go of old dreams? Was it—and did it feel like a creeping awareness that this was, like—that you felt almost cornered into it? Or how did you approach it?Carla: Yeah, I think I was the same way. I think I especially when you were younger because of that narrative, right, we tell people that if they decide to go into the arts, they have to be one hundred percent committed to it, and if they aren't one hundred percent and then they don't succeed, it is their fault, right? Like, if you give it everything that you have, and then it doesn't work out, you have clearly done something wrong, therefore you are a failure. You failed at your dream because you gave it everything that you have, so you kind of set yourself up for failure because you don't allow yourself to, you know, be more of who you are in other ways.For me, I just spent so many I had so many moments in my life where I thought that the world was over, right? Like, when I was—right out of college, I went to school to study opera. And I was studying at Cincinnati Conservatory of Music, it was, like, the great, great conservatory, and halfway through my freshman year, I got diagnosed with a cyst on my vocal cords. So, basically what this meant was that I had to have surgery to have it removed, and the doctor told me that I probably would never sing opera. And I was devastated.Like, I was—this was the thing I wanted to do with my life; I had committed myself one hundred percent, and now all of a sudden this thing happened, and I panicked. I thought it was my fault—because there was nobody to help me understand that it wasn't—and I was like, “I have failed this thing. I have failed my dream. What am I going to do with my life?” And I said, “Okay I'll be an actor because acting is a noble thing.” And that's sort of like act—that's sort of like performing; it's performing in a different way, it's just not singing.And I was terrified to sing again because I had this narrative in my head that I was a failed singer if I co,uldn't be an opera singer. And so, it took me, like, years, three years before I finally started singing again I got a voice teacher, and he—I would cry through all of my lessons. He was like, “Carly, you really have a—should be singing. Like, this is something that you're good at.” And I was like, no because if I can't sing, like, the way I want to sing, why would I sing?And he really kind of pushed me and helped me, like, figure out what my voice could do in a new way. And it was really magical for me. It made me realize that this narrative that I've been telling myself of what I thought that I was supposed to be didn't have to be true. It didn't have to be the only one that existed; there could be other possibilities for what I could do and they could look different. But I closed myself off to that idea because I had basically been told no, you can't do this thing that you want to do.So, I didn't even consider the possibilities of the other things that I could do. And when I relearned how to sing, it just blew my mind because I was like, “Oh, my God, I didn't know this was possible. I didn't know in my body it was possible of this. I didn't know if I could do this.” And, like, overcoming that and making me realize that I could do other things, that there were other versions of what I wanted, kind of blew my mind a little bit.And so, when I would hit road bumps and I'd hit these walls, I was like, “Okay, well, maybe I just need to pivot. Maybe the direction I'm going in isn't quite the right one, but maybe if I just, like, open my eyes a little bit, there's another—there's something else over here that is interesting and will be creative and will take me in a different way, an unexpected way that I wasn't expecting.” And so, I've kind of from that point on sort of living my life like that, in this way that, well, this might be a roadblock, and many people might view this thing as a failure, but for me, it allowed me to open up all these other new things that I didn't even know I could do, right? Like, what I'm doing now is something I never would have imagined I'd be doing five years ago. And now I'm also in a place where not only am I doing something completely different as a software engineer, but I have this incredible opportunity to also start incorporating art back into my life in a way that I can own and I can do for myself instead of having to do for other people.Which is also something I never thought because I thought it was all or nothing. I thought if I was an artist, I was an artist; I'm a software engineer, I'm a software engineer. And so, now I have the ability to kind of live in this weird gray area of getting to make those decisions for myself, and recognize that those little failures were, you know—like, I like to call them, like, the lowercase failures instead of the uppercase failure, right? Like, I am not a failure because I experienced failure. Those little failures are kind of what led me to grow my strength and my resilience and my ability to recognize it more free—like, more quickly when I see it so that I can bounce back faster, right?Like, when I hit a wall, instead of living in that feeling of, like, “Ugh, God, this is the worst thing that ever happened,” I allow myself to move faster through it and recognize that there will be light on the other side. I will get there. And I know that it's going to be okay, and I can trust that because it's always been okay. I always figure it out. And so, that's something—taken me a long time to, like, realize, you know? To, like, really learn, you have to fail a lot to learn that you're going to be okay every time it happens. [laugh].Corey: Yeah, what's the phrase? “Sucking at something is the first step to being kind of good at it?”Carla: Yeah. You got to let yourself suck at it. When I used to teach voice, I would make my students make just, like, the ugliest sounds because I was like, if we can just get past the fact that no matter what, when you sing you're going to sound awful at some point. We're going to try something, you're going to crack, it's not going to come out right, and if we can't own that it's going to suck a little bit on the journey to being good, like, you're going to have a really hard time getting there because you're just going to beat yourself up every time it sucks. Like, it's going to suck a lot [laugh] before you get good. And that's just part of it. That's, like, it is just a part of the process, and you have to kind of own it.Corey: I think that as people we are rarely as one-dimensional as we imagine we are when. And for example, I like working with cloud services, let's not kid ourselves on this. But I have a deep and abiding love affair with the sound of my own voice, so I'm always going to find ways to work that into it. I have a hard time seeing a future career for you that does not in some way, shape or form, tie back to your performing background because even now, talking about singing, you lit up when talking about that in a way that no one does—or at least should—light up when they're talking about React. So, do you think that there's a place between the performing side of the world and the technical side of the world, or those phases of your life, that's going to provide interesting paths for you down the road?Carla: That is a good question, Corey. And I don't know if I have the answer. You know, I think one thing—if there's anything I learned from all the crazy things that happened to me, is that I just kind of have to be open. You know, I like to say yes to things. And also learning to say no, which has been really a big deal for me.Corey: Oh, yes.“, no,” is a complete sentence and people know that sometimes at their own peril.Carla: Yes, I have said no to some things lately, and it's felt very good. But I like to be open, you know? I like to feel like if I'm putting out good things into the world, good things will come back to me, and so I'm just trying to keep that open. You know, I'm trying to be the best engineer that I can be. And I'm trying to also, you know—if I can use my voice and my platform to help inspire other people to see that there are other ways of being an artist, there are, you know, there are other paths in this world to take.I hope that, you know, I can, other things will come up to me, there'll be opportunities. And I don't know what those look like, but I'm open. So, if anybody out there hears this and you want to collaborate, hit me up. [laugh].Corey: Careful what you offer. People don't know—people have a disturbing tendency of saying, “Well, all right, I have an idea.” That's where a lot of my ridiculous parody music videos came from. It's like, “So, what's the business case for doing?” It's like, “Mmm, I think it'll be funny.”It's like, “Well, how are you going to justify the expense?” “Oh, there's a line item and the company budget labeled ‘Spite.' That's how.” And it's this weird combination of things that lead to a path that on some level makes perfect sense, but at the time you're building this stuff out, it feels like you're directionless and doing all these weird things. Like, one of the, I guess, strange parts of looking back at a path you take in the course of your career is, in retrospect, it feels like every step for the next was obvious and made intuitive sense, but going through it it's, “I have no idea what I'm doing. I'm like the dog that caught the car, and they need to desperately figure out how to drive the thing before it hits the wall.”It's just a—I don't pretend to understand how the tapestry of careers tie together, but I do know that I'm very glad to see people in this space, who do not all have the same ridiculous story for how they got in here. That's the thing that I find continually obnoxious, this belief that there's only one way to do it, or you're somehow less than because you didn't grow up programming in the '90s. Great. There's a lot of people like that. And yes, it is okay to just view computers as a job that pays the bills; there is nothing inherently wrong with that.Carla: Yeah. And I mean, and I—Corey: I just wish people were told that early on.Carla: Yeah, why not? Right? Why didn't anybody tell us that? Like, you don't—the thing that I did not—it took me a long time to realize is that you do not have to be passionate about your job. And that's like, that's okay, right? All you have to do is enjoy it enough to do it, but it does not have to be, like—Corey: You have to like it, on some level [crosstalk 00:33:10]—Carla: Yeah, you just do have to like it. [laugh].Corey: —dreading the 40 hours a week, that's a miserable life on some level.Carla: Like, I sit in front of a computer now all day, and I enjoy it. Like, I enjoy what I'm doing. But again, like, I don't need to be the greatest software engineer that ever lived; I have other things that I like to do, and it allows me to also do those things. And that is what I love about it. It allows me that ability to just enjoy my weekends and have a stable career and have a stable life and have health insurance. And then when I want—Corey: Oh, the luxuries of modern life.Carla: [laugh]. Yeah, the luxuries of modern life. Health insurance, who knew? Yeah, you know, so it's great. And then when creative projects come up, I can choose to say yes or no to them, and that's really exciting for me.Corey: I have a sneaking suspicion—I'll just place my bet now—that the world of performing is not quite done with you yet.Carla: Probably not. I would be lying if I said it was. I—so before all this stuff, I don't know if your listeners know this, but in January, the thing that kind of happened to me that went a little viral where I went back to Broadway after not being on Broadway for a little while, and the news media and everybody picked up on it, and there were like these headlines of, “Software engineer plays Elphaba on Broadway after seven years.” It surprised me, but it also didn't surprise me, you know? Like, when I left, I left thinking I was done.And I think it was easy to leave when I left because of the pandemic, right? There was nothing going on when I—like, I started my journey before the pandemic, but I fully shifted into software engineering during the pandemic. So, I never had feelings of, like, “I'm missing out on performing,” because performing didn't exist. There was no Broadway for a while. And so, once it kind of started to come back last year in the fall, I was like, “Oh, maybe I miss it a little bit.”And maybe I accidentally manifested it, but, you know, when Wicked called and I flew back to New York for those shows, and I was like, “Oh, this is really wonderful.” Also, I'm really glad I don't have to do this eight times a week. I'm so excited to go home. And I was like, having a little taste of it made me realize, “Oh, I can do this if I want to do this. I also don't have to do this if I don't want to do this.” And that was pretty—it was very empowering. I was like, “That feels nice.”Corey: I really appreciate your taking so much time to talk about how you've gone through what at the time has got to have felt like a very strange set of career steps, but it's starting to form into something that appears to have an arc to it. If people want to learn more and follow along as you continue to figure out what you're going to do next, where's the best place to find you?Carla: Oh, good question, Corey. I do a website, carlastickler.com. Because I've had a lot of people—artists, in particular—reaching out and asking how I did this, I'm starting to build some resources, and so you can sign up for my mailing list.I also am pretty big on Instagram if we're going to choose social media. So, my Instagram is stiglercarla. And there's links to all that stuff on my website. But—Corey: And they will soon be in the [show notes 00:36:26] as well.Carla: Ah yes, add them to the show notes. [laugh]. Yeah, and I want to make sure that I… I want—a lot of people who've seen my story and felt very inspired by it. A lot of artists who have felt that they, too, were failures because they chose not to go into art and get a regular nine to five. And so, I'm trying to, like, kind of put a little bit more of that out there so that people see that they're not alone.And so, on my social media, I do post a lot of stories that people send to me, just telling me their story about how they made the transition and how they keep art in their life in different ways. And so, that's something that also really inspires me. So, I tried to put their voices up, too. So, if anybody is interested in feeling not alone, feeling like there are other people out there, all of us, quote-unquote, “Failed artists,” and there's a lot of us. And so, I'm just trying to create a little space for all of us.Corey: I look forward to seeing it continue to evolve.Carla: Thank you.Corey: Thank you so much for your time. I appreciate it.Carla: Thanks, Corey.Corey: Carla Stickler, software engineer at G2 and also very much more. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, and if it's on the YouTubes, smash the like and subscribe buttons, as the kids of today are saying, whereas if you've hated this podcast, same thing: Five-star review, smash the buttons, but also leave an angry comment telling me exactly what you didn't like about this, and I will reply with the time and date for your audition.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.
About SimonFounder and CEO of SnapShooter a backup company Links Referenced: SnapShooter.com: https://SnapShooter.com MrSimonBennett: https://twitter.com/MrSimonBennett TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Finding skilled DevOps engineers is a pain in the neck! And if you need to deploy a secure and compliant application to AWS, forgettaboutit! But that's where DuploCloud can help. Their comprehensive no-code/low-code software platform guarantees a secure and compliant infrastructure in as little as two weeks, while automating the full DevSecOps lifestyle. Get started with DevOps-as-a-Service from DuploCloud so that your cloud configurations are done right the first time. Tell them I sent you and your first two months are free. To learn more visit: snark.cloud/duplo. Thats's snark.cloud/D-U-P-L-O-C-L-O-U-D.Corey: What if there were a single place to get an inventory of what you're running in the cloud that wasn't "the monthly bill?" Further, what if there were a way to compare that inventory to what you were already managing via Terraform, Pulumi, or CloudFormation, but then automatically add the missing unmanaged or drifted parts to it? And what if there were a policy engine to immediately flag and remediate a wide variety of misconfigurations? Well, stop dreaming and start doing; visit snark.cloud/firefly to learn more.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. One of the things that I learned early on in my career as a grumpy Unix systems administrator is that there are two kinds of people out there: those who care about backups an awful lot, and people who haven't lost data yet. I lost a bunch of data once upon a time and then I too fell on the side of backups are super important. Here to talk with me about them a bit today is Simon Bennett, founder and CEO of SnapShooter.com. Simon, thanks for joining me.Simon: Thanks for having me. Thank you very much.Corey: It's fun to be able to talk to people who are doing business in the cloud space—in this sense too—that is not venture-backed, that is not, “Well, we have 600 people here that are building this thing out.” And similar to the way that I handle things at The Duckbill Group, you are effectively one of those legacy things known as a profitable business that self-funds. What made you decide to pursue that model as opposed to, well, whatever the polite version of bilking venture capitalists out of enormous piles of money for [unintelligible 00:01:32]?Simon: I think I always liked the idea of being self-sufficient and running a business, so I always wanted to start a physical business when I was younger, but when I got into software, I realized that that's a really easy way, no capital needed, to get started. And I tried for years and years to build products, all of which failed until finally SnapShooter actually gained a customer. [laugh].Corey: “Oh, wait, someone finally is paying money for this, I guess I'm onto something.”Simon: Yeah.Corey: And it's sort of progressed from there. How long have you been in business?Simon: We started in 2017, as… it was an internal project for a company I was working at who had problems with DigitalOcean backups, or they had problems with their servers getting compromised. So, I looked at DigitalOcean API and realized I could build something. And it took less than a week to build a product [with billing 00:02:20]. And I put that online and people started using it. So, that was how it worked.Every other product I tried before, I'd spent months and months developing it and never getting a customer. And the one time I spent less than [laugh] less than a week's worth of evenings, someone started paying. I mean, admittedly, the first person was only paying a couple of dollars a month, but it was something.Corey: There's a huge turning point where you just validate the ability and willingness for someone to transfer one dollar from their bank account to yours. It speaks to validation in a way that social media nonsense generally doesn't. It's the oh, someone is actually willing to pay because I'm adding value to what they do. That's no small thing.Simon: Yeah. There's definitely a big difference between people saying they're going to and they'd love it, and actually doing it. So.Corey: I first heard about you when Patrick McKenzie—or @patio11, as he goes by on Twitter—wound up doing a mini-thread on you about, “I've now used SnapShooter.com for real, and it was such a joy, including making a server migration easier than it would otherwise have been. Now, I have automatically monitored backups to my own S3 account for a bunch of things, which already had a fairly remote risk of failure.” And he keeps talking about the awesome aspects of it. And okay, when Patrick says, “This is neat,” that usually means it's time for me to at least click the link and see what's going on.And the thing that jumped out at me was a few things about what it is that you offer. You talk about making sure that people can sleep well at night, that it's about why backups are important, about—you obviously check the boxes and talk about how you do things and why you do them the way that you do, but it resonates around the idea of helping people sleep well at night. Because no one wants to think about backups. Because no one cares about backups; they just care an awful lot about restores, usually right after they should have cared about the backups.Simon: Yeah. This is actually a big problem with getting customers because I don't think it's on a lot of people's minds, getting backups set up until, as you said in the intro, something's gone wrong. [laugh]. And then they're happy to be a customer for life.Corey: I started clicking around and looking at your testimonials, for example, on your website. And the first one I saw was from the CEO of Transistor.fm. For those who aren't familiar with what they do, they are the company that hosts this podcast. I pay them as a vendor for all the back issues and whatnot.Whenever you download the show. It's routing through their stuff. So yeah, I kind of want them to have backups of these things because I really don't want to have all these conversations [laugh] again with everyone. That's an important thing. But Transistor's business is not making sure that the data is safe and secure; it's making podcasts available, making it easy to publish to them.And in your case, you're handling the backup portion of it so they can pay their money and they set it up effectively once—set it and forget it—and then they can go back to doing the thing that they do, and not having to fuss with it constantly. I think a lot of companies get it wrong, where they seem to think that people are going to make sustained, engaged efforts in whatever platform or tool or service they build. People have bigger fish to fry; they just want the thing to work and not take up brain sweat.Simon: Yeah. Customers hardly ever log in. I think it's probably a good sign when they don't have to log in. So, they get their report emails, and that's that. And they obviously come back when they got new stuff to set up, but from a support point of view is pretty, pretty easy, really, people don't—[laugh] constantly on there.Corey: From where I sit, the large cloud providers—and some of the small ones, too—they all have backup functionality built into the offering that they've got. And some are great, some are terrible. I assume—perhaps naively—that all of them do what it says on the tin and actually back up the data. If that were sufficient, you wouldn't have any customers. You clearly have customers. What is it that makes those things not work super well?Simon: Some of them are inflexible. So, some of the providers have built-in server backups that only happen weekly, and six days of no backups can be a big problem when you've made a mistake. So, we offer a lot of flexibility around how often you backup your data. And then another key part is that we let you store your data where you want. A lot of the providers have either vendor lock-in, or they only store it in themselves. So… we let you take your data from one side of the globe to the other if you want.Corey: As anyone who has listened to the show is aware, I'm not a huge advocate for multi-cloud for a variety of excellent reasons. And I mean that on a per-workload basis, not, “Oh, we're going to go with one company called Amazon,” and you use everything that they do, including their WorkMail product. Yeah, even Amazon doesn't use WorkMail; they use Exchange like a real company would. And great, pick the thing that works best for you, but backups have always been one of those areas.I know that AWS has great region separation—most of the time. I know that it is unheard of for there to be a catastrophic data loss story that transcends multiple regions, so the story from their side is very often, oh, just back it up to a different region. Problem solved. Ignoring the data transfer aspect of that from a pricing perspective, okay. But there's also a risk element here where everyone talks about the single point of failure with the AWS account that it's there, people don't talk about as much: it's your payment instrument; if they suspend your account, you're not getting into any region.There's also the story of if someone gets access to your account, how do you back that up? If you're going to be doing backups, from my perspective, that is the perfect use case, to put it on a different provider. Because if I'm backing up from, I don't know, Amazon to Google Cloud or vice versa, I have a hard time envisioning a scenario in which both of those companies simultaneously have lost my data and I still care about computers. It is very hard for me to imagine that kind of failure mode, it's way out of scope for any disaster recovery or business continuity plan that I'm coming up with.Simon: Yeah, that's right. Yeah, I haven't—[laugh] I don't have that in my disaster recovery plan, to be honest about going to a different cloud, as in, we'll solve that problem when it happens. But the data is, as you say, in two different places, or more. But yeah, the security one is a key one because, you know, there's quite a lot of surface area on your AWS account for compromising, but if you're using either—even a separate AWS account or a different provider purely for storage, that can be very tightly controlled.Corey: I also appreciate the idea that when you're backing stuff up between different providers, the idea of owning both sides of it—I know you offer a solution where you wind up hosting the data as well, and that has its value, don't get me wrong, but there are also times, particularly for regulated industries, where yeah, I kind of don't want my backup data just hanging out with someone else's account with whatever they choose to do with it. There's also the verification question, which again, I'm not accusing you of in any way, shape, or form of being nefarious, but it's also one of those when I have to report to a board of directors of like, “Are you sure that they're doing what they say they're doing?” It's a, “Well, he seemed trustworthy,” is not the greatest answer. And the boards ask questions like that all the time. Netflix has talked about this where they backup a rehydrate-the-business level of data to Google Cloud from AWS, not because they think Amazon is going to disappear off the face of the earth, but because it's easier to do that and explain it than having to say, “Well, it's extremely unlikely and here's why,” and not get torn to pieces by auditors, shareholders, et cetera. It's the path of least resistance, and there is some validity to it.Simon: Yeah, when you see those big companies who've been with ransomware attacks and they've had to either pay the ransom or they've literally got to build the business from scratch, like, the cost associated with that is almost business-ending. So, just one backup for their data, off-site [laugh] they could have saved themselves millions and millions of pounds. So.Corey: It's one of those things where an ounce of prevention is worth a pound of cure. And we're still seeing that stuff continue to evolve and continue to exist out in the ecosystem. There's a whole host of things that I think about like, “Ooh, if I lost, that would be annoying but not disastrous.” When I was going through some contractual stuff when we were first setting up The Duckbill Group and talking to clients about this, they would periodically ask questions about, “Well, what's your DR policy for these things?” It's, “Well, we have a number of employees; no more than two are located in the same city anywhere, and we all work from laptops because it is the 21st century, so if someone's internet goes out, they'll go to a coffee shop. If everyone's internet goes out, do you really care about the AWS bill that month?”It's a very different use case and [unintelligible 00:11:02] with these things. Now, let's be clear, we are a consultancy that fixes AWS bills; we're not a hospital. There's a big difference in the use case and what is acceptable in different ways. But what I like is that you have really build something out that lets people choose their own adventure in how managed they want it to be, what the source is, what the target should be. And it gives people enough control but without having to worry about the finicky parts of aligning a bunch of scripts that wind up firing off in cron jobs.Simon: Yeah. I'd say a fair few people run into issues running scripts or, you know, they silently fail and then you realize you haven't actually been running backups for the last six months until you're trying to pull them, even if you were trying to—Corey: Bold of you to think that I would notice it that quickly.Simon: [laugh]. Yeah, right. True. Yeah, that's presuming you have a disaster recovery plan that you actually test. Lots of small businesses have never even heard of that as a thing. So, having as us, kind of, manage backups sort of enables us to very easily tell people that backups of, like—we couldn't take the backup. Like, you need to address this.Also, to your previous point about the control, you can decide completely where data flows between. So, when people ask us about what's GDPR policies around data and stuff, we can say, “Well, we don't actually handle your data in that sense. It goes directly from your source through almost a proxy that you control to your storage.” So.Corey: The best answer: GDPR is out of scope. Please come again. And [laugh] yeah, just pass that off to someone else.Simon: In a way, you've already approved those two: you've approved the person that you're managing servers with and you've already approved the people that are doing storage with. You kind of… you do need to approve us, but we're not handling the data. So, we're handling your data, like your actual customer; we're not handling your customer's customer's data.Corey: Oh, yeah. Now, it's a valuable thing. One of my famous personal backup issues was okay, “I'm going to back this up onto the shared drive,” and I sort of might have screwed up the backup script—in the better way, given the two possible directions this can go—but it was backing up all of its data and all the existing backup data, so you know, exponential growth of your backups. Now, my storage vendor was about to buy a boat and name it after me when I caught that. “Oh, yeah, let's go ahead and fix that.”But this stuff is finicky, it's annoying, and in most cases, it fails in silent ways that only show up as a giant bill in one form or another. And not having to think about that is valuable. I'm willing to spend a few hours setting up a backup strategy and the rest; I'm not willing to tend it on an ongoing basis, just because I have other things I care about and things I need to get done.Simon: Yeah. It's such a kind of simple and trivial thing that can quickly become a nightmare [laugh] when you've made a mistake. So, not doing it yourself is a good [laugh] solution.Corey: So, it wouldn't have been a @patio11 recommendation to look at what you do without having some insight into the rest of the nuts and bolts of the business and the rest. Your plans are interesting. You have a free tier of course, which is a single daily backup job and half a gig of storage—or bring your own to that it's unlimited storage—Simon: Yep. Yeah.Corey: Unlimited: the only limits are your budget. Yeah. Zombo.com got it slightly wrong. It's not your mind, it's your budget. And then it goes from Light to Startup to Business to Agency at the high end.A question I have for you is at the high end, what I've found has been sort of the SaaS approach. The top end is always been a ‘Contact Us' form where it's the enterprise scope of folks where they tend to have procurement departments looking at this, and they're going to have a whole bunch of custom contract stuff, but they're also not used to signing checks with fewer than two commas in them. So, it's the signaling and the messaging of, “Reach out and talk to us.” Have you experimented with that at all, yet? Is it something you haven't gotten to yet or do you not have interest in serving that particular market segment?Simon: I'd say we've been gearing the business from starting off very small with one solution to, you know, last—and two years ago, we added the ability to store data from one provider to a different provider. So, we're sort of stair-stepping our way up to enterprise. For example, at the end of last year, we went and got certificates for ISO 27001 and… one other one, I can't remember the name of them, and we're probably going to get SOC 2 at some point this year. And then yes, we will be pushing more towards enterprises. We add, like, APIs as well so people can set up backups on the fly, or so they can put it as part of their provisioning.That's hopefully where I'm seeing the business go, as in we'll become under-the-hood backup provider for, like, a managed hosting solution or something where their customers won't even realize it's us, but we're taking the backups away from—responsibility away from businesses.Corey: For those listeners who are fortunate enough to not have to have spent as long as I have in the woods of corporate governance, the correct answer to, “Well, how do we know that vendor is doing what they say that they're doing,” because the, “Well, he seemed like a nice guy,” is not going to carry water, well, here are the certifications that they have attested to. Here's copies under NDA, if their audit reports that call out what controls they claim to have and it validates that they are in fact doing what they say that they're doing. That is corporate-speak that attests that you're doing the right things. Now, you're going to, in most cases, find yourself spending all your time doing work for no real money if you start making those things available to every customer spending 50 cents a year with you. So generally, the, “Oh, we're going to go through the compliance, get you the reports,” is one of the higher, more expensive tiers where you must spend at least this much for us to start engaging down this rabbit hole of various nonsense.And I don't blame you in the least for not going down that path. One of these years, I'm going to wind up going through at least one of those certification approaches myself, but historically, we don't handle anything except your billing data, and here's how we do it has so far been sufficient for our contractual needs. But the world's evolving; sophistication of enterprise buyers is at varying places and at some point, it'll just be easier to go down that path.Simon: Yeah, to be honest, we haven't had many, many of those customers. Sometimes we have people who come in well over the plan limits, and that's where we do a custom plan for them, but we've not had too many requests for certification. But obviously, we have the certification now, so if anyone ever [laugh] did want to see it under NDA, we could add some commas to any price. [laugh].Corey: This episode is sponsored in parts by our friend EnterpriseDB. EnterpriseDB has been powering enterprise applications with PostgreSQL for 15 years. And now EnterpriseDB has you covered wherever you deploy PostgreSQL on premises, private cloud, and they just announced a fully managed service on AWS and Azure called BigAnimal, all one word.Don't leave managing your database to your cloud vendor because they're too busy launching another half dozen manage databases to focus on any one of them that they didn't build themselves. Instead, work with the experts over at EnterpriseDB. They can save you time and money, they can even help you migrate legacy applications, including Oracle, to the cloud.To learn more, try BigAnimal for free. Go to biganimal.com/snark, and tell them Corey sent you.Corey: What I like as well is that you offer backups for a bunch of different things. You can do snapshots from, effectively, every provider. I'm sorry, I'm just going to call out because I love this: AWS and Amazon LightSail are called out as two distinct things. And Amazonians will say, “Oh, well, under the hood, they're really the same thing, et cetera.” Yeah, the user experience is wildly different, so yeah, calling those things out as separate things make sense.But it goes beyond that because it's not just, “Well, I took a disk image. There we go. Come again.” You also offer backup recipes for specific things where you could, for example, back things up to a local file and external storage where someone is. Great, you also backup WordPress and MongoDB and MySQL and a whole bunch of other things.A unified cloud controller, which is something I have in my house, and I keep thinking I should find a way to back that up. Yeah, this is great. It's not just about the big server thing; it's about having data living in managed services. It's about making sure that the application data is backed up in a reasonable, responsible way. I really liked that approach. Was that an evolution or is that something you wound up focusing on almost from the beginning?Simon: It was an evolution. So, we started with the snapshots, which got the business quite far to be honest and it was very simple. It was just DigitalOcean to start with, actually, for the first two years. Pretty easy to market in a way because it's just focused on one thing. Then the other solutions came in, like the other providers and, you know, once you add one, it was easy to add many.And then came database backups and file backups. And I just had those two solutions because that was what people were asking for. Like, they wanted to make sure their whole server snapshot, if you have a whole server snapshot, the point in time data for MySQL could be corrupt. Like, there could be stuff in RAM that a MySQL dump would have pulled out, for example. Like… there's a possibility that the database could be corrupt from a snapshot, so people were asking for a bit of, more, peace of mind with doing proper backups of MySQL.So, that's what we added. And it soon became apparent when more customers were asking for more solutions that we really needed to, like, step back and think about what we're actually offering. So, we rebuilt this whole, kind of like, database engine, then that allowed us to consume data from anywhere. So, we can easily add more backup types. So, the reason you can see all the ones you've listed there is because that's kind of what people have been asking for. And every time someone comes up with a new, [laugh], like, a new open-source project or database or whatever, we'll add support, even ones I've never heard of before. When people ask for some weird file—Corey: All it takes is just waiting for someone to reach out and say, hey, can you back this thing up, please?Simon: Yeah, exactly, some weird file-based database system that I've never ever heard of. Yeah, sure. Just give us [laugh] a test server to mess around with and we'll build, essentially, like, we use bash in the background for doing the backups; if you can stream the data from a command, we can then deal with the whole management process. So, that's the reason why. And then, I was seeing in, like, the Laravel space, for example, people were doing MySQL backups and they'd have a script, and then for whatever reason, someone rotated the passwords on the database and the backup script… was forgotten about.So, there it is, not working for months. So, we thought we could build a backup where you could just point it at where the Laravel project is. We can get all the config we need at the runtime because it's all there with the project anyway, and then thus, you never need to tell us the password for your database and that problem goes away. And it's the same with WordPress.Corey: I'm looking at this now just as you go through this, and I'm a big believer in disclaiming my biases, conflicts of interest, et cetera. And until this point, neither of us have traded a penny in either direction between us that I'm ever aware of—maybe you bought a t-shirt or something once upon a time—but great, I'm about to become a customer of this because I already have backup solutions for a lot of the things that you currently support, but again, when you're a grumpy admin who's lost data in the past, it's, “Huh, you know what I would really like? That's right, another backup.” And if that costs me a few hundred bucks a year for the peace of mind is money well spent because the failure mode is I get to rewrite a whole lot of blog posts and re-record all podcasts and pay for a whole bunch of custom development again. And it's just not something that I particularly want to have to deal with. There's something to be said for a holistic backup solution. I wish that more people thought about these things.Simon: Can you imagine having to pull all the blog posts off [unintelligible 00:22:19]? [laugh]—Corey: Oh, my got—Simon: —to try and rebuild it.Corey: That is called the crappiest summer internship someone has ever had.Simon: Yeah.Corey: And that is just painful. I can't quite fathom having to do that as a strategy. Every once in a while some big site will have a data loss incident or go out of business or something, and there's a frantic archiving endeavor that happens where people are trying to copy the content out of the Google Search Engine's cache before it expires at whatever timeline that is. And that looks like the worst possible situation for any sort of giant backup.Simon: At least that's one you can fix. I mean, if you were to lose all the payment information, then you've got to restitch all that together, or anything else. Like, that's a fixable solution, but a lot of these other ones, if you lose the data, yeah, there's no two ways around it, you're screwed. So.Corey: Yeah, it's a challenging thing. And it's also—the question also becomes one of, “Well, hang on. I know about backups on this because I have this data, but it's used to working in an AWS environment. What possible good would it do me sitting somewhere else?” It's, yeah, the point is, it's sitting somewhere else, at least in my experience. You can copy it back to that sort of environment.I'm not suggesting this is a way that you can run your AWS serverless environment on DigitalOcean, but it's a matter of if everything turns against you, you can rebuild from those backups. That's the approach that I've usually taken. Do you find that your customers understand that going in or is there an education process?Simon: I'd say people come for all sorts of reasons for why they want backup. So, having your data in two places for that is one of the reasons but, you know, I think there's a lot of reasons why people want peace of mind: for either developer mistakes or migration mistakes or hacking, all these things. So, I guess the big one we come up with a lot is people talking about databases and they don't need backups because they've got replication. And trying to explain that replication between two databases isn't the same as a backup. Like, you make a mistake you drop—[laugh] you run your delete query wrong on the first database, it's gone, replicated or not.Corey: Right, the odds of me fat-fingering an S3 bucket command are incredibly likelier than the odds of AWS losing an entire region's S3 data irretrievably. I make mistakes a lot more than they tend to architecturally, but let's also be clear, they're one of the best. My impression has always been the big three mostly do a decent job of this. The jury's still out, in my opinion, on other third-party clouds that are not, I guess, tier one. What's your take?Simon: I have to be careful. I've got quite good relationships with some of these. [laugh].Corey: Oh, of course. Of course. Of course.Simon: But yes, I would say most customers do end up using S3 as their storage option, and I think that is because it is, I think, the best. Like, is in terms of reliability and performance, some storage can be a little slow at times for pulling data in, which could or could not be a problem depending on what your use case is. But there are some trade-offs. Obviously, S3, if you're trying to get your data back out, is expensive. If you were to look at Backblaze, for example, as well, that's considerably cheaper than S3, especially, like, when you're talking in the petabyte-scale, there can be huge savings there. So… they all sort of bring their own thing to the table. Personally, I store the backups in S3 and in Backblaze, and in one other provider. [laugh].Corey: Oh, yeah. Like—Simon: I like to have them spread.Corey: Like, every once in a while in the industry, there's something that happens that's sort of a watershed moment where it reminds everyone, “Oh, right. That's why we do backups.” I think the most recent one—and again, love to them; this stuff is never fun—was when that OVH data center burned down. And OVH is a somewhat more traditional hosting provider, in some respects. Like, their pricing is great, but they wind up giving you what amounts to here as a server in a rack. You get to build all this stuff yourself.And that backup story is one of those. Oh, okay. Well, I just got two of them and I'll copy backups to each other. Yeah, but they're in the same building and that building just burned down. Now, what? And a lot of people learned a very painful lesson. And oh, right, that's why we have to do that.Simon: Yeah. The other big lesson from that was that even if the people with data in a different region—like, they'd had cross-regional backups—because of the demand at the time for accessing backups, if you wanted to get your data quickly, you're in a queue because so many other people were in the same boat as you're trying to restore stored backups. So, being off-site with a different provider would have made that a little easier. [laugh].Corey: It's a herd of elephants problem. You test your DR strategy on a scheduled basis; great, you're the only person doing it—give or take—at that time, as opposed to a large provider has lost a region and everyone is hitting their backup service simultaneously. It generally isn't built for that type of scale and provisioning. One other question I have for you is when I make mistakes, for better or worse, they're usually relatively small-scale. I want to restore a certain file or I will want to, “Ooh, that one item I just dropped out of that database really should not have been dropped.” Do you currently offer things that go beyond the entire restore everything or nothing? Or right now are you still approaching this from the perspective of this is for the catastrophic case where you're in some pain already?Simon: Mostly the catastrophic stage. So, we have MySQL [bin logs 00:27:57] as an option. So, if you wanted to do, like, a point-in-time of store, which… may be more applicable to what you're saying, but generally, its whole, whole website recovery. For example, like, we have a WordPress backup that'll go through all the WordPress websites on the server and we'll back them up individually so you can restore just one. There are ways that we have helped customers in the past just pull one table, for example, from a backup.But yeah, we geared towards, kind of, the set and the forget. And people don't often restore backups, to be honest. They don't. But when they do, it's obviously [laugh] very crucial that they work, so I prefer to back up the whole thing and then help people, like, if you need to extract ten megabytes out of an entire gig backup, that's a bit wasteful, but at least, you know, you've got the data there. So.Corey: Yeah. I'm a big believer in having backups in a variety of different levels. Because I don't really want to do a whole server restore when I remove a file. And let's be clear, I still have that grumpy old Unix admin of before I start making changes to a file, yeah, my editor can undo things and remembers that persistently and all. But I have a disturbing number of files and directories whose names end in ‘.bac' with then, like, a date or something on it, just because it's—you know, like, “Oh, I have to fix something in Git. How do I do this?”Step one, I'm going to copy the entire directory so when I make a pig's breakfast out of this and I lose things that I care about, rather than having to play Git surgeon for two more days, I can just copy it back over and try again. Disk space is cheap for those things. But that's also not a holistic backup strategy because I have to remember to do it every time and the whole point of what you're building and the value you're adding, from my perspective, is people don't have to think about it.Simon: Yes. Yeah yeah yeah. Once it's there, it's there. It's running. It's as you say, it's not the most efficient thing if you wanted to restore one file—not to say you couldn't—but at least you didn't have to think about doing the backup first.Corey: I really want to thank you for taking the time out of your day to talk to me about all this. If people want to learn more for themselves, where can they find you?Simon: So, SnapShooter.com is a great place, or on Twitter, if you want to follow me. I am @MrSimonBennett.Corey: And we will, of course, put links to that in the [show notes 00:30:11]. Thank you once again. I really appreciate it.Simon: Thank you. Thank you very much for having me.Corey: Simon Bennett, founder and CEO of SnapShooter.com. I'm Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you've enjoyed this episode, please leave a five-star review on your podcast platform of choice, whereas if you've hated this episode, please leave a five-star review on your podcast platform of choice, along with an angry insulting comment that, just like your backup strategy, you haven't put enough thought into.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.
About WesleyWesley Faulkner is a first-generation American, public speaker, and podcaster. He is a founding member of the government transparency group Open Austin and a staunch supporter of racial justice, workplace equity, and neurodiversity. His professional experience spans technology from AMD, Atlassian, Dell, IBM, and MongoDB. Wesley currently works as a Developer Advocate, and in addition, co-hosts the developer relations focused podcast Community Pulse and serves on the board for SXSW.Links Referenced: Twitter: https://twitter.com/wesley83 Polywork: https://polywork.com/wesley83 Personal Website: https://www.wesleyfaulkner.com/ TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Finding skilled DevOps engineers is a pain in the neck! And if you need to deploy a secure and compliant application to AWS, forgettaboutit! But that's where DuploCloud can help. Their comprehensive no-code/low-code software platform guarantees a secure and compliant infrastructure in as little as two weeks, while automating the full DevSecOps lifestyle. Get started with DevOps-as-a-Service from DuploCloud so that your cloud configurations are done right the first time. Tell them I sent you and your first two months are free. To learn more visit: snark.cloud/duplo. Thats's snark.cloud/D-U-P-L-O-C-L-O-U-D.Corey: What if there were a single place to get an inventory of what you're running in the cloud that wasn't "the monthly bill?" Further, what if there were a way to compare that inventory to what you were already managing via Terraform, Pulumi, or CloudFormation, but then automatically add the missing unmanaged or drifted parts to it? And what if there were a policy engine to immediately flag and remediate a wide variety of misconfigurations? Well, stop dreaming and start doing; visit snark.cloud/firefly to learn more.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. I am joined again for a second time this year by Wesley Faulkner. Last time we spoke, he was a developer advocate. And since then, as so many have, he's changed companies. Wesley, thank you for joining me again. You're the Head of Community at SingleStore, now. Congrats on the promotion.Wesley: Thank you. It's been a very welcome change. I love developer advocates and developer advocacy. But I love people, too, so it's almost, I think, very analogous to the ebbs and flow that we all have gone through, through the pandemic, and leaning into my strong suits.Corey: It's a big deal having a ‘head of' in a role title, as opposed to Developer Advocate, Senior Developer Advocate. And it is a different role. It's easy to default into the world of thinking that it's a promotion. Management is in many ways orthogonal to what it takes to succeed in an actual role. And further, you're not the head of DevRel, or DevRelopers or whatever you want to call the term. You are instead the Head of Community. How tied is that to developer relations, developer advocacy, or other things that we are used to using as terms of art in this space?Wesley: If we're talking about other companies, I would say the Head of Community is something that's under the umbrella of developer relations, where it's just a peer to some of the other different elements or columns of developer relations. But in SingleStore specifically, I have to say that developer relations in terms of what you think about whole umbrella is very new to the company. And so, I consider myself the first person in the role of developer relations by being the Head of Community. So, a lot of the other parts are being bolted in, but under the focus of developer as a community. So, I'm liaisoning right now as helping with spearheading some of the design of the activities that the advocates do, as well as architecting the platform and the experiences of people coming in and experiencing SingleStore through the community's perspective.So, all that to say is, what I'm doing is extremely structured, and a lot of stuff that we're doing with the efficacy, I'm using some of my expertise to help guide that, but it's still something that's kind of like an offshoot and not well integrated at the moment.Corey: How has it changed the way that you view the function of someone who's advocating to developers, which is from my cynical perspective, “Oh, it's marketing, but we don't tell people it's marketing because they won't like it.” And yes, I know, I'll get emails about that. But how does it differ from doing that yourself versus being the head of the function of a company? Because leadership is a heck of a switch? I thought earlier in my career that oh, yeah, it's a natural evolution of being a mediocre engineer. Time to be a mediocre manager. And oh, no, no, I aspired to be a mediocre manager. It's a completely different skill set and I got things hilariously wrong. What's it like for you going through that shift?Wesley: First of all, it is kind of like advertising, and people may not think of it that way. Just to give an example, movie trailers is advertising. The free samples at the grocery store is advertising. But people love those because it gives an experience that they like in a package that they are accustomed to. And so, it's the same with developer relations; it's finding the thing that makes the experience worthwhile.On the community side, this is not new to me. I've done several different roles, maybe not in this combination. But when I was at MongoDB, I was a technical community manager, which is like a cog in the whole giant machine. But before that, in my other life, I managed social and community interactions for Walmart, and I had, at the slow period, around 65, but during the holidays, it would ramp up to 95 direct reports that I managed.It's almost—if you're a fan of The Princess Bride, it's different than fighting one person. Sometimes it's easier to fight, like, a squad or a gang of people. So, being Head of Community with such a young company is definitely a lot different than. In some ways, harder to deal with this type of community where we're just growing and emerging, rather than something more well-established.Corey: It probably gives you an interesting opportunity. Because back when I was doing engineering work as an SRE or whatever we call them in that era, it was, “Yeah, wow, my boss is terrible and has no idea what the hell they're doing.” So, then I found myself in the role, and it's, “Cool. Now, do all the things that you said you would do. Put up or shut up.”And it turns out that there's a lot you don't see that our strategic considerations. I completely avoided things like managing up or managing laterally or balancing trade-offs in different ways. Yeah, you're right. If you view the role of management as strictly being something that is between you and your direct reports, you can be an amazing manager from their perspective, but completely ineffective organizationally at accomplishing the goals that have been laid out for you.Wesley: Yeah. The good thing about being head of and the first head of is that you help establish those goals. And so, when you take a role with another company saying, “Hey, we have headcount for this,” and it's an established role, then you're kind of like streamlining into a process that's already underway. What's good about this role specifically, a ‘head of,' is that I help with not only designing what are the goals and the OKRs but deciding what the teams and what the team structure should look like. And so, I'm hiring for a specific position based on how it interacts with everything else.So, when I'm coming in, I don't say, “Well, what do you do?” Or, “How do you do it?” I said, “This is what needs to be done.” And that makes it so much easier just to say that if everything is working the way it should and to give marching orders based on the grand vision, instead of hitting the numbers this quarter or next quarter. Because what is core to my belief, and what's core, too, of how I approach things is at the heart of what I'm trying to do, which is really great, in terms of making something that didn't exist before.Corey: The challenge, too, is that everyone loves to say—and I love to see this at different ways—is the evolution and understanding of the DevRel folks who I work with and I have great relationships with realizing that you have to demonstrate business value. Because I struggle with this my entire career where I know intrinsically, that if I get on stage and tell a story about a thing that is germane to what my company does, that good things are going to happen. But it's very hard to do any form of attribution to it. In a different light, this podcast is a great example of this.We have sponsors. And people are listening. Ideally, they aren't fast-forwarding through sponsor messages; I do have interesting thoughts about the sponsors that I put into these ads. And that's great, but I also appreciate that people are driving while they're listening to this, and they are doing the dishes, they are mowing the lawn, and hopefully not turning up the volume too loudly so it damages their hearing. And the idea that they're going to suddenly stop any of those things and go punch in the link that I give is a little out to lunch there.Instead, it's partially brand awareness and it is occasionally the, “Wait. That resonates exactly with the problem that I have.” So, they get to work or they get back in front of a computer and the odds are terrific they're not going to punch in that URL of whatever I wound up giving; they're going to type in whatever phrases they remember and the company name into Google. Now—and doing attribution on something like that is very hard.It gets even more hard when we're talking about something that is higher up the stack that requires a bit more buy-in than individual developers. There's often a meeting or two about it. And then someone finally approaches the company to have a conversation. Now, does it work? Yes. There are companies that are sponsoring this stuff that spend a lot of time, effort, and money on that.I don't know how you do that sort of attribution; I don't pretend to know, but I know that it works. Because these people whose entire job is making sure that it does tell me it does. So, I smile, I nod, and that's great. But it's very hard to wind up building out a direct, “If you spend X dollars sponsoring this, you will see Y dollars in response.” But in the DevOps world, when your internal doing these things, well, okay because to the company, I look an awful lot like an expensive developer except I don't ever write production code.And then—at least in the before times—“So, what does your job do? Because looking at the achievements and accomplishments last quarter, it looks an awful lot like you traveled to exotic places on the company dime, give talks that are of only vague relevance to what we do, and then hang out at parties with your friends? Nice job, how can I get that?” But it's also first on the chopping block when okay, how do we trim expenses go? And I think it's a mistake to do that. I just don't think that story of the value of developer relations is articulated super-well. And I say that, but I don't know how to do a much better job of it myself.Wesley: Well, that's why corporate or executive buy-in is important because if they know from the get-go while you're there, it makes it a little bit easier to sell. But you do have to show that you are executing. So, there are always two parts to presenting a story, and that's one, the actual quantitative, like, I've done this many talks—so that output part—I've written this many blog posts, or I've stood up this many events that people can attend to. And then there's the results saying, people did read this post, people did show up to my event, people did listen to my talk that I gave. But you also need to give the subjective ones where people respond back and say, “I loved your talk,” or, “I heard you on Corey's podcast,” or, “I read your blog posts,” because even though you might not understand that it goes all the way down in a conversion funnel to a purchase, you can least use that stand-in to say there's probably, like, 20, 30 people behind this person to have that same sentiment, so you can see that your impact is reaching people and that it's having some sort of lasting effect.That said, you have to keep it up. You have to try to increase your output and increase your sphere of influence. Because when people go to solve their problem, they're going to look into their history and their own Rolodex of saying what was the last thing that I heard? What was the last thing that's relevant?There is a reason that Pepsi and Coke still do advertising. It's not because people don't know those brands, but being easily recalled, or a center of relevance based on how many touchpoints or how many times that you've seen them, either from being on American Idol and the logo facing the camera, or seeing a whole display when you go into the grocery store. Same with display advertising. All of this stuff works hand in hand so that you can be front-of-mind with the people and the decision-makers who will make that decision. And we went through this through the pandemic where… that same sentiment, it was like, “You just travel and now you can't travel, so we're just going to get rid of the whole department.”And then those same companies are hunting for those people to come back or to rebuild these departments that are now gone because maybe you don't see what we do, but when it's gone, you definitely notice a dip. And that trust is from the top-up. You have to do not just external advocacy, but you have to do internal advocacy about what impacts you're having so that at least the people who are making that decision can hopefully understand that you are working hard and the work is paying off.Corey: Since the last time that we spoke, you've given your first keynote, which—Wesley: Yes.Corey: Is always an interesting experience to go through. It was at a conference called THAT Conference. And I feel the need to specify that because otherwise, we're going to wind up with a ‘who's on first' situation. But THAT Conference is the name.Wesley: Specify THAT. Yes.Corey: Exactly. Better specify THAT. Yes. So, what was your keynote about? And for a bit of a behind-the-scenes look, what was that like for you?Wesley: Let me do the behind-the-scenes because it's going to lead up to actual the execution.Corey: Excellent.Wesley: So, I've been on several different podcasts. And one of the ones that I loved for years is one called This Week in Tech with Leo Laporte. Was a big fan of Leo Laporte back in the Screen Saver days back in TechTV days. Loved his opinion, follow his work. And I went to a South by Southwest… three, four years ago where I actually met him.And then from that conversation, he asked me to be on his show. And I've been on the show a handful of times, just talking about tech because I love tech. Tech is my passion, not just doing it, but just experiencing and just being on either side of creating or consuming. When I moved—I moved recently also since, I think, from the last time I was on your show—when I moved here to Wisconsin, the organizer of THAT Conference said that he's been following me for a while, since my first appearance on This Week in Tech, and loved my outlook and my take on things. And he approached me to do a keynote.Since I am now Wisconsin—THAT Conference is been in Wisconsin since inception and it's been going on for ten years—and he wanted me to just basically share my knowledge. Clean slate, have enough time to just say whatever I wanted. I said, “Yes, I can do that.” So, my experience on my end was like sheer excitement and then quickly sheer terror of not having a framework of what I was going to speak on or how I was going to deliver it. And knowing as a keynote, that it would be setting the tone for the whole conference.So, I decided to talk on the thing that I knew the most about, which was myself. Talked about my journey growing up and learning what my strengths, what my weaknesses are, how to navigate life, as well as the corporate jungle, and deciding where I wanted to go. Do I want to be the person that I feel like I need to be in order to be successful, which when we look at structures and examples and the things that we hold on a pedestal, we feel that we have to be perfect, or we have to be knowledgeable, and we have to do everything, well rounded in order to be accepted. Especially being a minority, there's a lot more caveats in terms of being socially acceptable to other people. And then the other path that I could have taken, that I chose to take, was to accept my things that are seen as false, but my own quirkiness, my own uniqueness and putting that front and center about, this is me, this is my person that over the years has formed into this version of myself.I'm going to make sure that is really transparent and so if I go anywhere, they know what they're getting, and they know what they're signing up for by bringing me on board. I have an opinion, I will share my opinion, I will bring my whole self, I won't just be the person that is technical or whimsical, or whatever you're looking for. You have to take the good with the bad, you have to take the I really understand technology, but I have ADHD and I might miss some deadlines. [laugh].Corey: This episode is sponsored in parts by our friend EnterpriseDB. EnterpriseDB has been powering enterprise applications with PostgreSQL for 15 years. And now EnterpriseDB has you covered wherever you deploy PostgreSQL on premises, private cloud, and they just announced a fully managed service on AWS and Azure called BigAnimal, all one word.Don't leave managing your database to your cloud vendor because they're too busy launching another half dozen manage databases to focus on any one of them that they didn't build themselves. Instead, work with the experts over at EnterpriseDB. They can save you time and money, they can even help you migrate legacy applications, including Oracle, to the cloud.To learn more, try BigAnimal for free. Go to biganimal.com/snark, and tell them Corey sent you.Corey: I have a very similar philosophy, and how I approach these things where it's there is no single speaking engagement that I can fathom even being presented to me, let alone me accepting that is going to be worth me losing the reputation I have developed for authenticity. It's you will not get me to turn into a shill for whatever it is that I am speaking in front of this week. Conversely, whether it's a paid speaking engagement or not, I have a standing policy of not using a platform that is being given to me by a company or organization to make them look foolish. In other words, I will not make someone regret inviting me to speak at their events. Full stop.And I have spoken at events for AWS; I have spoken at events for Oracle, et cetera, et cetera, and there's no company out there that I'm not going to be able to get on stage and tell an entertaining and engaging story, but it requires me to dunk on them. And that's fine. Frankly, if there is a company like that where I could not say nice things about them—such as Facebook—I would simply decline to pursue the speaking opportunity. And that is the way that I view it. And very few companies are on that list, to be very honest with you.Now, there are exceptions to this, if you're having a big public keynote, I will do my traditional live-tweet the keynote and make fun of people because that is, A, expected and, B, it's live-streamed anywhere on the planet I want to be sitting at that point in time, and yeah, if you're saying things in public, you can basically expect that to be the way that I approach these things. But it's a nuanced take, and that is something that is not fully understood by an awful lot of folks who run events. I'll be the first to admit that aspects of who and what I am mean that some speaking engagements are not open to me. And I'm okay with that, on some level, I truly am. It's a different philosophy.But I do know that I am done apologizing for who I am and what I'm about. And at some point that required a tremendous amount of privilege and a not insignificant willingness to take a risk that it was going to work out all right. I can't imagine going back anymore. Now, that road is certainly not what I would recommend to everyone, particularly folks earlier in their career, particularly for folks who don't look just like I do and have a failure mode of a board seat and a book deal somewhere, but figuring out where you will and will not compromise is always an important thing to get straight for yourself before you're presented with a situation where you have to make those decisions, but now there's a whole bunch of incentive to decide in one way or another.Wesley: And that's a journey. You can't just skip sections, right? You didn't get to where you are unless you went through the previous experience that you went through. And it's true for everyone. If you see those success books or how-to books written by people who are extremely rich, and, like, how to become successful and, like, okay, well, that journey is your own. It doesn't make it totally, like, inaccessible to everyone else, but you got to realize that not everyone can walk that path. And—Corey: You were in the right place at the right time, an early employee at a company that did phenomenally well and that catapulted you into reach beyond the wildest dreams of avarice territory. Good for you, but fundamentally, when you give talks like that as a result, what it often presents as is, “I won the lottery, and here's how you can too.” It doesn't work that way. The road you walked was unique to you and that opportunity is closed, not open anyone else, so people have to find their own paths.Wesley: Yeah, and lightning doesn't strike in the same place twice. But there are some things where you can understand some fundamentals. And depending on where you go, I think you do need to know yourself, you do need to know—like, be able to access yourself, but being able to share that, of course, you have to be at a point where you feel comfortable. And so, even if you're in a space where you don't feel that you can be your authentic self or be able to share all parts of you, you yourself should at least know yourself and then make that decision. I agree that it's a point of privilege to be able to say, “Take me how I am.”I'm lucky that I've gotten here, not everyone does, and just because you don't doesn't mean that you're a failure. It just means that the world hasn't caught up yet. People who are part of marginalized society, like, if you are, let's say trans, or if you are even gay, you take the same person, the same stance, the same yearning to be accepted, and then transport it to 50 years ago, you're not safe. You will not necessarily be accepted, or you may not even be successful. And if you have a lane where you can do that, all the power to you, but not everyone could be themselves, and you just need to make sure that at least you can know yourself, even if you don't share that with the world.Corey: It takes time to get there, and I think you're right that it's impossible to get there without walking through the various steps. It's one of the reasons I'm somewhat reluctant to talk overly publicly about my side project gig of paid speaking engagements, for instance, is that the way to get those is you start off by building a reputation as a speaker, and that takes an awful lot of time. And speaking at events where there's no budget even to pay you a speaking fee out of anyway. And part of what gets the keynote invitations to, “Hey, we want you to come and give a talk,” is the fact that people have seen you speak elsewhere and know what you're about and what to expect. Here's a keynote presented by someone who's never presented on stage before is a recipe for a terrifying experience, if not for the speaker or the audience, definitely [laugh] for the event organizers because what if they choke.?Easy example of this, even now hundreds of speaking engagements in, the adrenaline hit right before I go on stage means that sometimes my knees shake a bit before I walk out on stage. I make it a point to warn the people who are standing with me backstage, “Oh, this is a normal thing. Don't worry, it is absolutely expected. It happens every time. Don't sweat it.”And, like, “Thank you for letting us know. That is the sort of thing that's useful.” And then they see me shake, and they get a little skeptical. Like, I thought this guy was a professional. What's the story and I walk on stage and do my thing and I come back. Like, “That was incredible. I was worried at the beginning.” “I told you, we all have our rituals before going on stage. Mine is to shake like a leaf.”But the value there is that people know what to generally expect when I get on stage. It's going to have humor, there's going to be a point interwoven throughout what I tend to say, and in the case of paid speaking engagements, I always make sure I know where the boundaries are of things I can make fun of a big company for. Like, I can get on stage and make fun of service naming or I can make fun of their deprecation policy or something like that, but yeah, making fun of the way that they wind up handling worker relations is probably not going to be great and it could get the person who championed me fired or centered internally. So, that is off the table.Like, even on this podcast, for example, I sometimes get feedback from listeners of, “Well, you have someone from company X on and you didn't beat the crap out of them on this particular point.” It's yeah, you do understand that by having people on the show I'm making a tacit agreement not to attack them. I'm not a journalist. I don't pretend to be. But if I beat someone up with questions about their corporate policy, yeah, very rarely do I have someone who is in a position in those companies to change that policy, and they're certainly not authorized to speak on the record about those things.So, I can beat them up on it, they can say, “I can't answer that,” and we're not going to go anywhere. What is the value of that? It looks like it's not just gotcha journalism, but ineffective gotcha journalism. It doesn't work that way. And that's never been what this show is about.But there's that consistent effort behind the scenes of making sure that people will be entertained, will enjoy what they're seeing, but also are not going to deeply regret giving me a microphone, has always been the balancing act, at least for me. And I want to be clear, my style is humor. It is not for everyone. And my style of humor has a failure mode of being a jerk and making people feel bad, so don't think that my path is the only or even a recommended way for folks who want to get more into speaking to proceed.Wesley: You also mention, though, about, like, punching up versus punching down. And if you really tear down a company after you've been invited to speak, what you're doing is you're punching down at the person who booked you. They're not the CEO; they're not the owner of the company; they're the person who's in charge of running an event or booking speakers. And so, putting that person and throwing them under the bus is punching down because now you're threatening their livelihood, and it doesn't make any market difference in terms of changing the corporate's values or how they execute. So yeah, I totally agree with you in that one.And, like you were saying before, if there's a company you really thought was abhorrent, why speak there? Why give them or lend your reputation to this company if you absolutely feel that it's something you don't want to be associated with? You can just choose not to do that. For me, when I look at speaking, it is important for me to really think about why I'm speaking as well. So, not just the company who's hiring me, but the audience that I'll be serving.So, if I'm going to help with inspiring the next generation of developers, or helping along the thought of how to make the world a better place, or how people themselves can be better people so that we can just change the landscape and make it a lot friendlier, that is also its own… form of compensation and not just speaking for a speaker's fee. So, I do agree that you need to not just be super Negative Nancy, and try to fight all fights. You need to embrace some of the good things and try to make more of those experiences good for everyone, not just the people who are inviting you there, but the people who are attending. And when I started speaking, I was not a good speaker as well. I made a lot of mistakes, and still do, but I think speaking is easier than some people think and if someone truly wants to do it, they should go ahead and get started.What is the saying? If there's something is truly important, you'll be bad at it [laugh] and you'll be okay with it. I started speaking because of my role as a developer advocate. And if you just do a Google search for ‘CFPs,' you can start speaking, too. So, those who are not public speakers and want to get into it, just Google ‘CFP' and then start applying.And then you'll get better at your submissions, you'll get better at your slides, and then once you get accepted, then you'll get better at preparing, then you'll get better at actually speaking. There's a lot of steps between starting and stopping and it's okay to get started doing that route. The other thing I wanted to point out is I feel public speaking is the equivalent of lifting your own bodyweight. If you can do it, you're one of the small few of the population that is willing to do so or that can do it. If you start public speaking, that in itself is an accomplishment and an experience that is something that is somewhat enriching. And being bad at it doesn't take the passion away from you. If you just really want to do it, just keep doing it, even if you're a bad speaker.Corey: Yeah. The way to give a great talk because you have a bunch of terrible talks first.Wesley: Yeah. And it's okay to do that.Corey: And it's not the in entirety of community. It's not even a requirement to be involved with the community. If you're one of those people that absolutely dreads the prospect of speaking publicly, fine. I'm not suggesting that, oh, you need to get over that and get on stage. That doesn't help anyone. Don't do the things you dread doing because you know that it's not going to go well for you.That's the reason I don't touch actual databases. I mean, come on, let's be realistic. I will accidentally the data, and then we won't have a company anymore. So, I know what things I'm good at and things I'm not. I also don't do hostage negotiations, for obvious reasons.Wesley: And also, here's a little, like, secret tip. If you really want to do public speaking and you start doing public speaking and you're not so good at it from other peoples' perspective, but you still love doing it and you think you're getting better, doing public speaking is one of those things where you can say that you do it and no one will really question how good you are at it. [laugh]. If you're just in casual conversation, it's like, “Hey, I wrote a book.” People like, “Oh, wow. This person wrote the book on blah, blah, blah.”Corey: It's a self-published book that says the best way to run Kubernetes. It's a single page; it says, “Don't.” In 150-point type. “The end.” But I wrote a book.Wesley: Yeah.Corey: Yeah.Wesley: People won't probe too much and it'll help you with your development. So, go ahead and get started. Don't worry about doing that thing where, like, I have to be the best before I can present it. Call yourself a public speaker. Check, done.Corey: Always. We are the stories we tell, and nowhere is it more true than in the world of public speaking. I really want to thank you for taking the time out of your day to speak with me about this for a second time in a single year. Oh, my goodness. If people want to learn more about what you're up to, where can they find you?Wesley: I'm on Twitter, @wesley83 on Twitter. And you can find me also on PolyWork. So, polywork.com/wesley83. Or just go to wesleyfaulkner.com which redirects you there. I list pretty much everything that I am working on and any upcoming speaking opportunities, hopefully when they release that feature, will also be on that Polywork page.Corey: Excellent. And of course, I started Polywork recently, and I'm at thoughtleader.cloud because of course I am, which is neither here nor there. Thank you so much for taking the time to speak about this side of the industry that we never really get to talk about much, at least not publicly and not very often.Wesley: Well, thank you for having me on the show. And I wanted to take some time to say thank you for the work that you're doing. Not just elevating voices like myself, but talking truth to power, like we mentioned before, but being yourself and being a great representation of how people should be treating others: being honest without being mean, being snarky without being rude. And other companies and other people who've given me a chance, and given me a platform, I wanted to say thank you to you too, and I wouldn't be here unless it was people like you acknowledging the work that I've been doing.Corey: All it takes is just recognizing what you're doing and acknowledging it. People often want to thank me for this stuff, but it's just, what, for keeping my eyes open? I don't know, I feel like it's just the job; it's not something that is above and beyond any expected normal behavior. The only challenge is I look around the industry and I realize just how wrong that impression is, apparently. But here we are. It's about finding people doing interesting work and letting them tell their story. That's all this podcast has ever tried to be.Wesley: Yeah. And you do it. And doing the work is part of the reward, and I really appreciate you just going through the effort. Even having your ears open is something that I'm glad that you're able to at least know who the people are and who are making noises—or making noise to raise their profile up and then in turn, sharing that with the world. And so, that's a great service that you're providing, not just for me, but for everyone.Corey: Well, thank you. And as always, thank you for your time. Wesley Faulkner, Head of Community at SingleStore. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, along with a rambling comment telling me exactly why DevRel does not need success metrics of any kind.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.
About PriyankaPriyanka Vergadia is currently a Staff Developer Advocate at Google Cloud where she works with enterprises to build and architect their cloud platforms. She enjoys building engaging technical content and continuously experiments with new ways to tell stories and solve business problems using Google Cloud tools. You can check out some of the stories that she has created for the developer community on the Google Cloud Platform Youtube channel. These include "Deconstructing Chatbots", "Get Cooking in Cloud", "Pub/Sub Made Easy" and more. ..Links Referenced: LinkedIn: https://www.linkedin.com/in/pvergadia/ Twitter: https://twitter.com/pvergadia Priyanka's book: https://www.amazon.com/Visualizing-Google-Cloud-Illustrated-References/dp/1119816327 TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Finding skilled DevOps engineers is a pain in the neck! And if you need to deploy a secure and compliant application to AWS, forgettaboutit! But that's where DuploCloud can help. Their comprehensive no-code/low-code software platform guarantees a secure and compliant infrastructure in as little as two weeks, while automating the full DevSecOps lifestyle. Get started with DevOps-as-a-Service from DuploCloud so that your cloud configurations are done right the first time. Tell them I sent you and your first two months are free. To learn more visit: snark.cloud/duplo. Thats's snark.cloud/D-U-P-L-O-C-L-O-U-D. Corey: What if there were a single place to get an inventory of what you're running in the cloud that wasn't "the monthly bill?" Further, what if there were a way to compare that inventory to what you were already managing via Terraform, Pulumi, or CloudFormation, but then automatically add the missing unmanaged or drifted parts to it? And what if there were a policy engine to immediately flag and remediate a wide variety of misconfigurations? Well, stop dreaming and start doing; visit snark.cloud/firefly to learn more.Corey: Welcome to Screaming in the Cloud, I'm Corey Quinn. Periodically, I get the privilege of speaking to people who work in varying aspects of some would call it developer evangelism, some would call it developer advocacy, developer relations is a commonly accepted term, and I of course call it devrelopers because I enjoy annoying absolutely everyone by giving things terrible names. My guest today is Priyanka Vergadia, who is a staff developer advocate at Google Cloud. Priyanka, thank you for joining me.Priyanka: Thank you so much for having me. Corey. I'm so excited to be your developer—what did you call it again?Corey: Devreloper. Yes indeed.Priyanka: Devreloper. That is the term I'm going to be using from now on. I am a devreloper. Anyway.Corey: Excellent.Priyanka: Yeah.Corey: I'm starting to spread this out so that eventually we're going to form a giant, insufferable army of people who pronounce it that way, and it's going to be great.Priyanka: It's going to be awesome. [laugh].Corey: One of the challenges, even as I alluded to different titles within this space, everyone has a slightly different definition of where the role starts and stops, just in terms of its function, let alone the myriad ways that can be expressed. In the before times, I knew a number of folks in the developer advocacy space who were more or less worldwide experts in accumulating airline miles and racking up status and going from conference to conference to conference to more or less talk about things that had a tenuous at best connection to where they worked. Great. Other folks have done things in very different ways. Some people write extensively, blog posts and the rest, others build things a sample code, et cetera, et cetera.It seems like every time I talk to someone in the space, they have found some new and exciting way of carrying the message of what their company does to arguably a very cynical customer group. Where do you start and stop with your devrelopment?Priyanka: Yeah. So, that is such—like, all the devrelopers have their own style that they have either adopted or learned over time that works for them. When I started, I think about three years ago, I did go to conferences, did those events, give talks, all of that, but I was also—my actual introduction to DevRel [laugh] was with videos. I started creating my first series was deconstructing chatbots, and I was very interested in learning more about chatbots. So, I was like, you know what, I'm just going to teach everybody, and learn.So like, learn and teach at the same time was my motto, and that's kind of how I got started into, like, okay, I'm going to create a few videos to learn this and teach it. And during the process I was like, “I want to do this more.” And that's kind of transitioned, my move from being in front of customers, which I still end up doing, but I was doing more of just, you know, working with customers extensively to get their deployments done. This was a segue for me to, you know, think back, sit back and think about what's working and what I personally enjoy doing more, and that's what got me into creating videos. And it's like, okay, I'm going to become a devreloper now.And that's kind of how the whole, like, journey started. And for me, like you were pointing out earlier—should I just stop because I've been talking too long? [laugh].Corey: No, keep going. Please, [unintelligible 00:04:10] it's fine.Priyanka: [laugh]. For me, I started—I found my, I would say, in the last two years—it was all before the pandemic, we were all either writing blogs or doing videos or going to conferences, so it was, you know, the pandemic kind of brought us to a point where it's like, “Okay, let's think about—we can't meet each other; let's think about other ways to communicate and how can we make it creative and exciting?”Corey: And the old way started breaking down, too, where it's, “Yay, I'm going to watch an online conference.” “What is it?” “Oh, it's like a crappy Zoom only you don't have to pretend to pay attention in the same way.” And as a presenter, then you've got to modify what you're doing to understand that people's attention spans are shorter, distraction is always a browser tab away, and unlike a physical event, people don't feel the same sense of shame of getting up from the front row and weaving in front of 300 people, and not watching the rest of your talk. I mean, don't get me wrong, I'll still do it, but I'll feel bad about it.Now it's, “Oh, nope, I'm sitting here in my own little… hovel, I'm just going to do and watch whatever I want to do.” So, you've got to—it forces you to up your game, and it—Priyanka: Yep.Corey: Still doesn't quite have the same impact.Priyanka: Yeah. Or just switch off the camera, if you're like me, and just—uh, shut off the camera, go away or do something else. And, yeah, it's very easy to do that. So, it's not the same, which is why it prompted, I think all of us DevRel people to think about new ways to connect, which is for me that way to connect is art and visual aspects, to kind of bring that—because that—we are all whether we accept it or not or like it or not, we're all visual learners, so that's kind of how I think when it comes to creating content is visually appealing, and that's when people can dive in. [laugh].Corey: I am in the, I guess opposite side of the universe from you, where I acknowledge and agree with everything you're saying that people are visual creatures inherently, but I have effectively zero ability in that direction. My medium has always been playing games with words and language. And over time, I had the effectively significantly belated realization that wait a minute, just because I'm not good at a thing doesn't mean that other people might not be good at that thing, and I don't have to do every last part of it myself. Suddenly, I didn't have to do my own crappy graphic design because you can pay people who are worlds better than I'll ever be, and so on and so forth. I don't edit my own podcast audio because I'm bad at that, too.But talking about things is a different story, writing about things, building things is where I tend to see a lot of what I do tend to resonate. But I admit I bias for the things that I enjoy doing and the way that I enjoy consuming things. You do as well because relatively recently, as of time of this recording, you have done what I don't believe anyone actually wants to do. You wrote a book. Now, everyone wants to have written a book, but no one actually wants to write a book.Priyanka: So, true. [laugh].Corey: But it's not like most technical books. Tell me about it.Priyanka: Yeah, I actually never thought I would write a book. If you asked me two years ago—three years ago, I would say, I would have never thought that I would write a book because I am not a text person. So, I don't like to read a lot of texts because it zones out. So, for me, when I started creating some of these sketches, and sharing it on social media and in blogs and things like that, and gotten the attention that it has gotten from people, that's when I was like, okay, ding, ding, ding. I think I can do a visual book with these images.And this was like, halfway through, I'd already created, like, 30 sketches at this point. And I was like, “Okay, maybe I can turn this into a book,” which would be interesting for me because I like doing art-type things along with teaching, and it's not text because I wanted to do this in a very unique way. So yeah, that's kind of how it ended up happening.Corey: I have a keen appreciation for people who approach things with a different point of view. One of your colleagues, Forrest Brazeal, took a somewhat similar approach in the in his book, The Read Aloud Cloud, where it was illustrated, and everything he did was in rhyme, which is a constant source of envy for me, where it's, “Mmm, I've got to find a way to one-up him again.” And it's… he is inexorable, as far as just continuing to self-improve. So, all right, we're going to find a way to wind up defeating that. With you, it's way easier.I read a book, like, wow, this is gorgeous and well-written that it's attractive to look at, and I will never be able to do any of those things. That's all you. It doesn't feel like we're trying to stand at the same spot in the universe in quite the same way. Nothing but love for Forrest. Let's be clear. I am teasing. I consider him a friend.Priyanka: He is amazing. Well honestly, like, I actually got to know Forrest when I decided to do this book. Wiley, who's the publisher, sent me Forrest's book, and he said, “You should look at this book because the idea that you are presenting to me, we could lay it out in this format.” Like, in the, you know, physical format. So, he sent me that book. And that's how I know Forrest, honestly.So, I told him that—this is a little story that I told him after. But anyway, yeah. I—the—[sigh]—I was going to make a point about the vid—the aspect of creating images, like, honestly, like, I designed the aspects of, like, how you layout information in the sketches, I studied a bunch of stuff to come up with, how do I make it precise and things like that. But there's no way this book was possible without some design help. Like, I can't possibly do the entire thing unless I have, like, five years. [laugh]. So—Corey: Right on top of all of this, you do presumptively have a day job as well—and while—Priyanka: Exactly.Corey: This is definitely related. “I'm just going to go write a book.” “Oh, is it a dissertation?” “No, it's going to look more like a children's book than that,” is what they're going to hear. And it's yeah, I'm predicting some problems with the performance evaluation process at large companies when you start down those paths.Priyanka: Exactly. So, I ended up, like, showing all these numbers, like, of the blog views and reads and social media, the presence of some of these images that were going wider. And in the GCPSketchnote GitHub repo got a huge number of stars. And it was like, everybody could see that writing a book would be amazing. From that point on, I was just like, I don't think I can scale that.So, when I was drawing—this is an example—when I drew my first sketch, it took me an entire weekend to just draw one sketch, which is what—I was only doing that the entire weekend—like, assume, like, 16 hours of work, just drawing the one sketch. So, if I went with that pace, this book was not possible. So, you know, after I had the idea laid out, had the process in place, I got some design help, which made it—which expedited the process much, much faster. [laugh].Corey: There's a lot to be said, for doing something that you enjoy. Do you do live sketchnoting during conference talks as well, or do you tend to not do it while someone is talking at a reasonably fast clip, and well, in 45 minutes, this had better be done, so let's go. I've seen people who can do that, and I just marvel in awe at what they do.Priyanka: I don't do live. I don't do live sketching. For me, paper and pen is a better medium so that's just the medium that I like to work with. So, when the talk is happening, I'm actually taking notes on a pen and a paper. And then after, I can sketch it out, faster in a fast way.Like, I did one sketchnote for Next 2020, I think, and that was done, like, a day after Next was over so I could take all the bits and pieces that were important and put it into that sketch. But I can't do it live. That's just one of the things I haven't figured out yet. [laugh].Corey: For me, I was always writing my email newsletter, so it was relatively rapid turnaround, and Twitter was interesting for me. I finally cracked the nut on how to express myself in a way that worked. The challenge that I ran into then was okay, there are thoughts I occasionally have that don't lend themselves to then 140—now 280—characters, so I should probably start writing long-form. And then I want to start writing 1000 to 1500-word blog posts every week that goes out. And that forced me to become a better writer across the board. And then it became about one-upping myself, sort of, live-tweeting conference talks.And the personal secret of why I do that is I'm ADHD in a bottle. Someone gets on stage—you say you zone out when you read a giant quantity of data; you prefer something more visual, more interactive. For me, I'm the opposite, where when someone gets on stage and starts talking, it's, “Okay, get to—yes, you're doing the intro of what a cloud might be. I get that point. This is supposed to be a more advanced talk. Can we speed it up a bit?”And doing the live-tweeting about it, but not just relating what is said, but by making a joke about it, it's how I keep myself engaged and from zoning out. Because let's face it, this industry is extraordinarily boring, if you don't bring a little bit of light to it.Priyanka: Yeah, that is—Corey: And that how to continue and how to do that was hard, and it took me time to get there.Priyanka: Yeah. Yeah, no, I totally agree. Like, that's exactly why I got into, like, training videos and sketches. Like, and videos and also. Like, I come up with, like, fake examples of companies that may or may not exist.Like, I made up a dog shoe making company that ships out shoes when you need them and then return them and there's a size and stuff, like, you have to come up with interesting things to make the content interesting because otherwise, this can get boring pretty quickly, which is going back to your example of, “Speed it up; get to the point.” [laugh].Corey: This episode is sponsored in parts by our friend EnterpriseDB. EnterpriseDB has been powering enterprise applications with PostgreSQL for 15 years. And now EnterpriseDB has you covered wherever you deploy PostgreSQL on premises, private cloud, and they just announced a fully managed service on AWS and Azure called BigAnimal, all one word. Don't leave managing your database to your cloud vendor because they're too busy launching another half dozen manage databases to focus on any one of them that they didn't build themselves. Instead, work with the experts over at EnterpriseDB. They can save you time and money, they can even help you migrate legacy applications, including Oracle, to the cloud.To learn more, try BigAnimal for free. Go to biganimal.com/snark, and tell them Corey sent you.Corey: It's always just fun to start experimenting with it, too, because all right, once I was done learn learning how to live-tweet other people's talk and mostly get it correct because someone says something, I have three to five seconds to come up with what I want to talk about and maybe grab a picture and then move on to the next thing. And it's easy to get that wrong and say things you don't necessarily intend to and get taken the wrong way. I've mostly gotten past that. And—I'm not saying I'm always right, but I better than I used to be. And then it was okay, “How do I top this?”And I started live-tweeting conference talks that I was giving live, which is always fun, but being able to pre-write some tweets at certain times, have certain webhooks in your slide deck and whatnot that fire these things off. And again, I'm not saying that he this is recommended or even a good idea, but it definitely wasn't boring. And—Priyanka: Yeah.Corey: And continue to find ways to make the same type of material new and interesting is one of the challenges because the stuff is complex.Priyanka: Also bite-size, right? Like, it's—I think Twitter is, like, the [unintelligible 00:15:54] words are obviously limiting, but it also forces you to think about it in bite-size, right? Like, okay, if I have a blog post then I'm summarizing it, how would I do it in two sentences? It forces me to think about it that way, which makes it very applicable to the time span that we have now, right, which is maybe, like, 30 seconds, you can have somebody on [unintelligible 00:16:18]Corey: Attention is a rare and precious commodity.Priyanka: Yeah. Yeah.Corey: People who [unintelligible 00:16:21] engagement, I think that's the wrong metric to go after because that inspires a whole bunch of terrible incentives, whereas finding something that is interesting, and a way to bring light to it and have a perspective on it that makes people think about it differently. For me, it's been humor, but that's my own approach to things. Your direction, it seems to be telling a story through visual arts. And that is something we don't see nearly as much of.Priyanka: Yeah. I think it's also because it's something that you—you know, like, I grew up drawing and painting. I was drawing since I was three years old, so that's my way of thinking. Like, I don't—I was talking to another devreloper the other day, and we were talking about—Corey: It's catching on. I love it.Priyanka: —[laugh]. Two different ways of how we think. So, for me, when I design a piece of content, I have my visuals first, and then he was talking about when he designs his content, he has his bullet points and a blog post first. So, it's like, two very different ways of approaching this similar thing. And then from that, from the images or the deck that I'm building up, I would come up with the narrative and stuff like that.My thinking starts with images and narrative of tying, like, the images together. But it's, that is the whole, like, fun of being in DevRel, right? Like, you are your own personality, and bringing whatever your personality, like you mentioned, humor and your case, art in my case, in somebody else's case, it could be totally different thing, right? So, yeah.Corey: Now, please correct me if I'm wrong on this, but an area of emphasis for you has been data analytics as well as Kubernetes, more or less things that are traditionally considered to be much more back-end if you're looking at a spectrum of all things technology. Is that directionally accurate, or am I dramatically is understanding a lot of what you're saying?Priyanka: No, that's very much accurate. I like to—I tend to be on the infrastructure back-and creating pipeline, creating easier processes, sort of person, not much into front-end. I dabble into it, but don't enjoy it. [laugh].Corey: This makes you something of a unicorn, in the sense of there are a tremendous number of devreloper types in the front-end slash JavaScript world because their entire career is focused on making things look visually appealing. That is what front-end is. I know this because I am rubbish at it. My idea of a well-designed interface that everyone looks at and smiles at [unintelligible 00:19:12] of command-line arguments when you're writing a script for something. And it's on a green screen, and sometimes I'll have someone helped me coordinate to come up with a better color palette for the way that I'm looking at my terminal on my Mac. Real exciting times over here, I assure you.So, the folks who are working in that space and they have beautifully designed slides, yeah, you tend to expect that. I gave a talk years ago at the front-end conference in Zurich, and I was speaking in the afternoon. And I went there and every presentation, slides were beautiful. And this was before I was working here and had a graphic designer on retainer to make my slides look not horrible. It was black Helvetica text on a white background, and I'm looking at this and I'm feeling ashamed that it's—okay, I have two hours to fix this. What do I do?I did the only thing I could think of; I changed Helvetica text to Comic Sans because if it's going to look terrible and it's going to be a designer thing that puts them off, you may as well go all-in. And that was a recurring meme at the time. I've since learned that there is an argument—I don't know if it's true or not—that Comic Sans is easier to read for folks with dyslexia, for example. And that's fine. I don't know if that's accurate or not, but I stopped making jokes about it just because if people—even if it's not true, and people believe that it's, “Are you being unintentionally crappy to people?” It's, “Well, I sure hope not. I'm rarely intentionally crappy. But when I do, I don't want to be mistaken for not being.” It's, save it up and use it when it counts.Priyanka: Yeah, yeah. I've—yeah, I think, when it comes to these big events—and like front-end for me is—I would think, like, I actually thought that I would be great at front-end because I have interest in art and stuff. I do make things that [crosstalk 00:20:57]—Corey: That's my naive assumption, too. I'm learning as you speak here. Please continue[.Priyanka: Yeah. And I was just—I thought that I would be and I have tried it, and I only like it to an extent, to present my idea. But I don't like to go in deeper and, like, make my CSS pretty or make this—make it look pretty. I am very much intrigued by all the back-end stuff, and most of my experience, over the past ten years in Cloud has been in the back-end stuff, mainly just because I love APIs, I love—like, you know, as long as I can connect, or the idea of creating a demo or something that involves a bunch of APIs and a back-end, to present an idea in a front-end, I would work on that front-end. But otherwise, I'm not going to choose to do it. [laugh]. Which I found interesting for myself as well. It's a realization. [laugh].Corey: Every time I try and do something with front-end, it doesn't matter the framework, I find myself more confused at the end than I was when I started. There's something I don't get. And anytime I see someone on Twitter, for example, talking about how a front-end is easier or somehow less than, I read that and I can't help myself. It's, “You ridiculous clown. You have no idea what you're talking about.”I don't believe that I'm bad at all of the things under engineering—just most of them—and I think I pick things up reasonably quickly. It is a mystery that does not align with this, and if it's easy for you, you don't recognize—arguably—a skill that you have, but not everyone does, by a landslide. And that's a human nature thing, too. It's if it was easy for me, it's obviously easy for everyone. If something's hard for me, no one would understand how this works and the people that do are wizards from the future.Priyanka: Yep. So true.Corey: It never works that way.Priyanka: Yeah. It never works that way. At least we have this in common, that you don't like to work on front-ends. [laugh].Corey: There's that too. And I think that no matter where you fall on the spectrum of technology, I would argue that something that we all share in common is, it doesn't matter how far we are down in the course of our entire career, from the very beginning to the very end, it is always a consistent, constant process of being humbled and made to feel like a fool by things you are supposedly professionally good at. And oh my stars, I've just learned to finally give up and embrace it. It's like, “So, what's going to make me feel dumb today?”Priyanka: Exactly.Corey: It's the learn in public approach, which is important.Priyanka: It's so important. Especially, like, if you're thinking about it, like that's the part of DevRel that makes it so exciting, too, right? Like, just learning a new thing today and sharing it with you. Like, I'm not claiming that I'm an expert, but hey, let's talk about it. And sure, I might end up looking dumb one day, I might end up looking smart the other day, but that's not the point. The point is, I end up learning every day, right? And that's the most important part, which is why I love this particular job, which is—what did we call it—devreloper.Corey: Devreloping. And as a part of that, you're talking to people constantly, be it people in the community and ecosystem, people who—you say you've talk to customers, but you also talk to these other folks. I would challenge you on that, where when you're at a company like Google Cloud, increasingly everyone in the community in the ecosystem is in one way or another, indistinguishable from being your customer; it all starts to converge at some point. All major cloud providers have that luxury, to be perfectly honest. What do you see in the ecosystem that people are struggling with as you talk to them?And again, any one person is going to have a problem or bone to pick with some particular service or implementation, and okay, great. What I'm always interested in is what is the broad sweep of things? Because when I hear someone complaining that a given service from a given cloud provider is terrible. Okay, great. Everyone has an opinion. When I started to hear that four or five, six times, it's okay, there's something afoot here, and now I'm curious as to what it is. What patterns are you seeing emerge these days?Priyanka: Yeah. I think more and more patterns along the lines of how can you make it automated? How can you make anything automated, right? Like, from machine learning's perspective, how do I not need ML skills to build an ML model? Like, how can we get there faster, right?Same for, like, in the infrastructure side, the serverless… aspect? How can you make it easy for me so I can just build an application and just deploy it so it becomes your problem to run it and not mine?Corey: Oh, the—you are preaching to the choir on that. I feel like all of these services that talk about, “This is how you build and train a machine learning model,” yadda, yadda, it's for an awful lot of the use cases out there, it's exposing implementation details about which I could not possibly care less. It's the, I want an API that I throw something at—like, be it a picture—and then I want to get a response of, “Yes, it's a hot dog,” or, “That's disgusting,” or whatever it is that it decides that it wants to say, great because that's the business outcome I'm after, and I do not care what wizardry happens on the back-end, I don't care if it's people who are underpaid and working extremely quickly by hand to do it, as long as it's from a business perspective, it hits a certain level of performance, reliability, et cetera. And then price, of course, yeah.And that is not to say I'm in favor of exploiting people, let's be clear here because I'm pretty sure most of these are not actually humans on the back-end, but okay. I just want that as the outcome that I think people are after, and so much of the conversation around how to build and train models and all misses the point because there are companies out there that need that, absolutely, there are, but there are a lot more that need the outcome, not the focus on this. And let's face it, an awful lot of businesses that would benefit from this don't have the budget to hire the team of incredibly expensive people it takes to effectively leverage these things because I have an awful lot of observations about people in machine learning space, one of them is absolutely not that, “Wow, I bet those people are inexpensive for me to hire.” It doesn't work that way.Priyanka: It doesn't. Yeah. And so, yeah. I think the future of, like, the whole cloud space, like, when it started, we started with how can I run my server not in my basement, but somewhere else, right? Now, we are at a different stage where we have a different sets of problems and requirements for businesses, right?And that's where I see it growing. It's like, how can I make this automated fast, not my problem? How can I make it not my problem is, like, the biggest [laugh] biggest, I think, theme that we are seeing, whether it's infrastructure, data science, data analytics, in all of these spaces.Corey: I get a lot of interesting feedback for my comparative takes on the various cloud providers, and one thing that I've said for a while about Google Cloud has been that its developer experience is unparalleled compared to basically anything else on the market. It makes things just work, and that's important because a bad developer experience has the unfortunate expression—at least for me—of, “Oh, this isn't working the way I want it to. I must be dumb.” No, it's a bad user experience for you. What I am seeing emerge as well from Google Cloud is an incredible emphasis—and I do think they're aligned here—on storytelling, and doing so effectively.You're there communicating visually; Forrest is there, basically trying to be the me of Google Cloud—which is what I assume he's doing; he would argue everything about that and he'd be right to do it, but that's what I'm calling it because this is my show; he can come on and argue with me himself if he takes issue with it. But I love the emphasis on storytelling and unifying solutions and the rest, as opposed to throwing everything at the wall to see what sticks to it. I think there's more intention being put into an awful lot of not just what you're building, but how you're talking about it, now it's integrated with the other things that you're building. That's no small thing.Priyanka: Yeah. That is so hard, especially when you know the cloud space; like, hundreds of products, they all have their unique requirement to solve a problem, but nobody cares, right? Like, as a consumer, I shouldn't have to care that there are 127 products or whatever. It doesn't matter to me as a consumer or customer, all that matters is whether I can solve my business problem with a set of your tools, right? So, that's exactly why, like, we have this team that I work in that I'm a part of, which has an entire focus on storytelling.We do YouTube videos with storytelling, we do art like this, I've also dabbled into comics a little bit. And we continue to go back to the drawing board with how else we can tell these stories. I know—I mentioned this to Forrest—I'm working on a song as well, which I have never done before, and [laugh] I think I'm going to butcher it. I kind of have it ready for, like, six months but never released it, right, because I'm just too scared to do that. [laugh] but anyway.Corey: Ship and then turn the internet off for a week and it'll be gone regardless, by the time you come back. Problem solved until the reporters start calling, and then you have problems.Priyanka: I might have to just do that, and be, like, you know what world? Keep saying whatever you want to say, I'm not here. [laugh]. But anyway, going back to that point of storytelling, and it's so—I think we have weaved it into the process. And it's going really well, and now we are investing more in, like, R&D and doing more of how we can tell stories in different ways.Corey: I have to say, I'm a big fan of the way that you're approaching this. If people want to learn more about what you're up to—and arguably, as I argue they should get a copy of your book because it is glorious—where's the best place to find you?Priyanka: Thank you. Okay, so LinkedIn and Twitter are my platforms that I check every single day, so you can message me, connect with me, I am available as—my handle is pvergadia. I don't know if they have [crosstalk 00:31:11]—Corey: Oh, this is all going in the [show notes 00:31:13] you need not worry.Priyanka: Okay, perfect. So yeah, I don't have to spell it because my last name is hard. [laugh]. So, you'll find it in the show notes. But yeah, you can connect with me there. And you will find at the top of both of my profiles, the link to order the book, so you can do it there.Corey: Excellent. And I've already done so, and I'm just waiting for it to arrive. So, this is—it's going to be an exciting read if nothing else. One of these days, I'd have to actually live-tweet a reading thereof. We'll see how that plays out.Priyanka: That would be amazing.Corey: Be careful what you wish for. Some of the snark could be a little too cutting; we have to be cautious of that.Priyanka: [laugh]. I'm always scared of your tweets. Like, do I want to read this or not? [laugh].Corey: If nothing else, it at least tries to be funny. So, there is that.Priyanka: Yes. Yes, for sure.Corey: I really—Priyanka: No, I'm excited. I'm excited for when you get a chance to read it and just tweet whatever you feel like, from, you know, all the bits and pieces that I've brought together. So, I would love to get your take. [laugh].Corey: Oh, you will, one way or another. That's one of those non-optional things. It's one of the fun parts of dealing with me. It's, “Aw crap. That shitposter is back again.” Like the kid outside of your yard just from across the street, staring at your house and pointing and it's, “Oh, dear. Here we go.” Throwing stones.Priyanka: [laugh]. I'm excited either way. [laugh].Corey: He's got a platypus with him this time. What's going on? It happens. We deal with what we have to. Thank you so much for being so generous with your time. I appreciate it.Priyanka: Thank you so much for having me. It was amazing. You are a celebrity, and I wanted to be, you know, a part of your show for a long time, so I'm glad we're able to make it work.Corey: You are welcome back anytime.Priyanka: I will. [laugh].Corey: An absolute pleasure to talk with you. Thanks again.Priyanka: Thank you.Corey: Priyanka Vergadia staff developer—but you call it developer advocate—at Google Cloud. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on whatever platform you're using to listen to this thing, whereas if you've hated it, please do the exact same thing, making sure to hit the like and subscribe buttons on the YouTubes because that's where it is. But if you did hate it, also leave an insulting, angry comment but not using words. I want you to draw a picture telling me exactly what you didn't like about this episode.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.
About TomaszTomasz is a Frontend Engineer at Stedi, Co-Founder/Head of React at Cloudash, egghead.io instructor with over 200 lessons published, a tech speaker, an AWS Community Hero and a lifelong learner.Links Referenced: Cloudash: https://cloudash.dev/ Twitter: https://twitter.com/tlakomy TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by Honeycomb. When production is running slow, it's hard to know where problems originate. Is it your application code, users, or the underlying systems? I've got five bucks on DNS, personally. Why scroll through endless dashboards while dealing with alert floods, going from tool to tool to tool that you employ, guessing at which puzzle pieces matter? Context switching and tool sprawl are slowly killing both your team and your business. You should care more about one of those than the other; which one is up to you. Drop the separate pillars and enter a world of getting one unified understanding of the one thing driving your business: production. With Honeycomb, you guess less and know more. Try it for free at honeycomb.io/screaminginthecloud. Observability: it's more than just hipster monitoring.Corey: This episode is sponsored in part by our friends at ChaosSearch. You could run Elasticsearch or Elastic Cloud—or OpenSearch as they're calling it now—or a self-hosted ELK stack. But why? ChaosSearch gives you the same API you've come to know and tolerate, along with unlimited data retention and no data movement. Just throw your data into S3 and proceed from there as you would expect. This is great for IT operations folks, for app performance monitoring, cybersecurity. If you're using Elasticsearch, consider not running Elasticsearch. They're also available now in the AWS marketplace if you'd prefer not to go direct and have half of whatever you pay them count towards your EDB commitment. Discover what companies like Equifax, Armor Security, and Blackboard already have. To learn more, visit chaossearch.io and tell them I sent you just so you can see them facepalm, yet again.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. It's always a pleasure to talk to people who ask the bold questions. One of those great bold questions is, what if CloudWatch's web page didn't suck? It's a good question. It's one I ask myself all the time.And then I stumbled across a product that wound up solving this for me, and I'm a happy customer. To be clear, they're not sponsoring anything that I do, nor should they. It's one of those bootstrapped, exciting software projects called Cloudash. Today, I'm joined by the Head of React at Cloudash, Tomasz Łakomy. Tomasz, thank you for joining me.Tomasz: It's a pleasure to be here.Corey: So, where did this entire idea come from? Because I sit and I get upset every time I have to go into the CloudWatch dashboard because first, something's broken. In an ideal scenario, I don't have to care about monitoring or observability or anything like that. But then it's quickly overshadowed by the fact that this interface is terrible. And the reason I know it's terrible is that every time I'm in there, I feel dumb.My belief is—for the longest time, I thought that was a problem with me. But no, invariably, when you wind up working with something and consistently finding it a bad—you don't know enough to solve for it, it's not you. It is, in fact, the signs of a poorly designed experience, start to finish. “You should be smarter to use this tool,” is very rarely correct. And there are a bunch of observability tools and monitoring tools for serverless things that have made sense over the years and made this easier, but one of the most—and please don't take this the wrong way—stripped down, bare essentials of just the facts, style of presentation is Cloudash. It's why I continue to pay for it every month with a smile on my face. How did you get here from there?Tomasz: Yeah that's a good question. I would say that. Cloudash was born out of desire for simple things to be simple. So, as you mentioned, Cloudash is basically the monitoring and troubleshooting tool for serverless applications, made for serverless developers because I am very much into serverless space, as is Maciej Winnicki, who is the another half of Cloudash team. And, you know, the whole premise of serverless was things are going to be simpler, right?So, you know, you have a bunch of code, you're going to dump it into a Lambda function, and that's it. You don't have to care about servers, you don't have to care about, you know, provisioning stuff, you don't have to care about maintenance, and so on. And that is not exactly true because why PagerDuty still continues to be [unintelligible 00:02:56] business even in serverless spaces. So, you will get paged every now and then. The problem is—what we kind of found is once you have an incident—you know, PagerDuty always tends to call it in the middle of the night; it's never, like, 11 a.m. during the workday; it's always the middle of the night.Corey: And no one's ever happy when it calls them either. It's, “Ah, hell.” Whatever it rings, it's yeah, the original Call of Duty. PagerDuty hooked up to Nagios. I am old enough to remember those days.Tomasz: [unintelligible 00:03:24] then business, like, imagine paying for something that's going to wake you up in the middle of the night. It doesn't make sense. In any case—Corey: “So, why do you pay for that product? Because it's really going to piss me off.” “Okay, well… does that sound like a good business to you? Well, AWS seems to think so. No one's happy working with that stuff.” “Fair. Fair enough.”Tomasz: So, in any case, like we've established an [unintelligible 00:03:43]. So you wake up, you go to AWS console because you saw a notification that this-and-this API has, you know, this threshold was above it, something was above the threshold. And then you go to the CloudWatch console. And then you see, okay, those are the logs, those are the metrics. I'm going to copy this request ID. I'm going to go over here. I'm going to go to X-Ray.And again, it's 3 a.m. so you don't exactly remember what do you investigate; you have, like, ten minutes. And this is a problem. Like, we've kind of identified that it's not simple to do these kinds of things, too—it's not simple to open something and have an understanding, okay, what exactly is happening in my serverless app at this very moment? Like, what's going on?So, we've built that. So, Cloudash is a desktop app; it lives on your machine, which is a single pane of glass. It's a single pane of glass view into your serverless system. So, if you are using CloudFormation in order to provision something, when you open Cloudash, you're going to see, you know, all of the metrics, all the Lambda functions, all of the API Gateways that you have provisioned. As of yesterday, API Gateway is no longer cool because they did launch the direct integration, so you have—you can call Lambda functions with [crosstalk 00:04:57]—Corey: Yeah, it's the one they released, and then rolled back and somehow never said a word—because that's an AWS messaging story, and then some—right around re:Invent last year. And another quarter goes by and out it goes.Tomasz: It's out yesterday.Corey: Yeah, it's terrific. I love that thing. The only downside to it is, ah, you have to use one of their—you have to use their domain; no custom domain support. Really? Well, you can hook up CloudFront to it, but the pricing model that way makes it more expensive than API Gateway.Okay, so I could use Cloudflare in front of it, and then it becomes free, so I bought a domain just for that purpose. That's right, my serverl—my direct Lambda URLs now live behind the glorious domain of cheapass.cloud because of course. They are. It's a day-one product from AWS, so of course, it's not feature-complete.But one of the things I like about the serverless model, and it's also a challenge when it comes to troubleshooting stuff is that it's very much set it and forget it style because serverless in many cases, at least the way that I tend to use it, is back-office stuff, its back-end things, it's processing on things that are not necessarily always direct front and center. So, these things can run on their own for years until finally, you find a strange bug in a new use case, or you want to go and change something. And then it's how the hell did this ever work? And it's still working, kind of, but what fool built this? Of course, it was me; it's always me.But what happened here? You're basically excavating your own legacy code, trying to understand what's going on. And so, you're already upset then. Cloudash makes this easier to find the things, to navigate through a whole bunch of different accounts. And there are a bunch of decisions that you made while building the app that are so clearly correct, that I get actively annoyed when others don't because oh, it looks at your AWS configuration file in your user home directory. Great, awesome. It's a desktop app, but it still consults that file. Yay, integration between ClickOps and the terminal. Wonderful.But ah, use SSO for a lot of stuff, so that's going to fix your little red wagon. I click on that app, and suddenly, bam, a browser opens asking me to log in and authenticate, allow the request. It works, and then suddenly, it goes back to doing exactly what you'd expect it to. It's really nice. The affordances behind this are glorious.Tomasz: Like I said, one of our kind of design goals when building Cloudash was to make simple things simple again. The whole purpose is to make sure that you can get into the root cause of an issue within, like, five minutes, if not less. And this is kind of the app that you're going to tend to open whenever that—as I said, because some of the systems can be around for, like, ages, literally without any incident whatsoever, then the data is going to change because somebody [unintelligible 00:07:30] got that the year is 2020 and off you go, we have an incident.But what's important about Cloudash is that we don't send logs anywhere. And that's kind of important because you don't pay for [PUT 00:07:42] metric API because we are not sending those logs anywhere. If you install Cloudash on your machine, we are not going to get your logs from the last ten years, put them in into a system, charge you for that, just so you are able to, you know, find out what happened in this particular hour, like, two weeks ago. We genuinely don't care about your logs; we have enough of our own logs at work to, you know, to analyze, to investigate, and so on; we are not storing them anywhere.In fact, you know, whatever happens on your machine stays on the machine. And that is partially why this is a desktop app. Because we don't want to handle your credentials. We don't—absolutely, we don't want you to give us any of your credentials or access keys, you know, whatever. We don't want that.So, that is why you install Cloudash, it's going to run on your machine, it's going to use your local credentials. So, it's… effectively, you could say that this is a much more streamlined and much more laser-focused browser or like, an eye into AWS systems, which live on the serverless side of things.Corey: I got to deal with it in a bit of an interesting way, recently. I have a detector in my company's production AWS org, to detect when ClickOps is afoot. Now, I'm a big proponent of ClickOps, but I also want to know what's going on, so I have a whole thing that [runs detects 00:09:04] when people are doing things in the console versus via API. And it alerts on certain subsets of them. I had to build a special case for the user agent string coming out of Cloudash because no, no, this is an app, this is not technically ClickOps—it is also read-only, which is neither here nor there, to my understanding.But it was, “Oh yeah, this is effectively an Electron app.” It just wraps, effectively, a browser and presents that as an application. And cool. From my perspective, that's an implementation detail. It feels like a native app—because it is—and I can suddenly see the things I care about in a way that is much more straightforward without having to have four different browser tabs open where, okay, here's the CloudTrail log for this thing, here's the metrics next to it. Oh, those are two separate windows already, and so on and so forth. It just makes hunting down to the obnoxious problems so much nicer.It's also, you're one of those rare products where if I don't use it for a month, I don't get the bill at the end of the month and think, “Ooh, that's going to—did I waste the money?” It's no, nice. I had a whole month where I didn't have to mess with this. It's great.Tomasz: Exactly. I feel like, you know, it's one of those systems where, as you said, we send you an email at the end of every month that we're going to charge you X dollars for the month—by the way, we have fixed pricing and then you can cancel anytime—and it's like one of those things that, you know, I didn't have to open this up for a month. This is awesome because I didn't have any incidents. But I know whenever again, PagerDuty is going to decide, “Hey, dude, wake up. You know, if slept for three hours. That is definitely long enough,” then you know that; you know, this app is there and you can use that.We very much care about, you know, building this stuff, not only for our customers, but we also use that on a daily basis. In fact, I… every single time that I have to—I want to investigate something in, like, our serverless systems at Stedi because everything that we do at work, at Stedi, since this incident serverless paradigm. So, I tend to open Cloudash, like, 95% of the time whenever I want to investigate something. And whenever I am not able to do something in Cloudash, this goes, like, straight to the top of our, you know, issue lists or backlog or whatever you want to call it. Because we want to make this product, not only awesome, you know, for customers to buy a [unintelligible 00:11:22] or whatever, but we also want to be able to use that on a daily basis.And so far, I think we've kind of succeeded. But then again, we have quite a long way to go because we have more ideas, than we have the time, definitely, so we have to kind of prioritize what exactly we're going to build. So, [unintelligible 00:11:39] integrations with alarms. So, for instance, we want to be able to see the alarms directly in the Cloudash UI. Secondly, integration with logs insights, and many other ideas. I could probably talk for hours about what we want to build.Corey: I also want to point out that this is still your side gig. You are by day a front-end engineer over at Stedi, which has a borderline disturbing number of engineers with side gigs, generally in the serverless space, doing interesting things like this. Dynobase is another example, a DynamoDB desktop client; very similar in some respects. I pay for that too. Honestly, for a company in Stedi's space, which is designed as basically a giant API for deep, large enterprise business stuff, there's an awful lot of stuff for small-scale coming out of that.Like, I wind up throwing a disturbing amount of money in the general direction of Stedi for not being their customer. But there's something about the culture that you folks have built over there that's just phenomenal.Tomasz: Yeah. For the record, you know, having a side gig is another part of interview process at Stedi. You don't have to have [laugh] a side project, but yeah, you're absolutely right, you know, the amount of kind of side projects, and you know, some of those are monetized, as you mentioned, you know, Cloudash and Dynobase and others. Some of those—because for instance, you talked to Aidan, I think a couple of weeks ago about his shenanigans, whenever you know, AWS is going to announce something he gets in and try to [unintelligible 00:13:06] this in the most amusing ways possible. Yeah, I mean, I could probably talk for ages about why Stedi is by far the best company I've ever worked at, but I'm going to say this: that this is the most talented group of people I've ever met, and myself, honestly.And, you know, the fact that I think we are the second largest, kind of, group of AWS experts outside of AWS because the density of AWS Heroes, or ex-AWS employees, or people who have been doing cloud stuff for years, is frankly, massive, I tend to learn something new about cloud every single day. And not only because of the Last Week in AWS but also from our Slack.Corey: This episode is sponsored by our friends at Oracle Cloud. Counting the pennies, but still dreaming of deploying apps instead of “Hello, World” demos? Allow me to introduce you to Oracle's Always Free tier. It provides over 20 free services and infrastructure, networking, databases, observability, management, and security. And—let me be clear here—it's actually free. There's no surprise billing until you intentionally and proactively upgrade your account. This means you can provision a virtual machine instance or spin up an autonomous database that manages itself, all while gaining the networking, load balancing, and storage resources that somehow never quite make it into most free tiers needed to support the application that you want to build. With Always Free, you can do things like run small-scale applications or do proof-of-concept testing without spending a dime. You know that I always like to put asterisks next to the word free? This is actually free, no asterisk. Start now. Visit snark.cloud/oci-free that's snark.cloud/oci-free.Corey: There's something to be said for having colleagues that you learn from. I have never enjoyed environments where I did not actively feel like the dumbest person in the room. That's why I love what I do now. I inherently am. I have to talk about so many different things, that whenever I talk to a subject matter expert, it is a certainty that they know more about the thing than I do, with the admitted and depressing exception of course of the AWS bill because it turns out the reason I had to start becoming the expert in that was because there weren't any. And here we are now.I want to talk as well about some of—your interaction outside of work with AWS. For example, you've been an Egghead instructor for a while with over 200 lessons that you published. You're an AWS Community Hero, which means you have the notable distinction of volunteering for a for-profit company—good work—no, the community is very important. It's helping each other make sense of the nonsense coming out of there. You've been involved within the ecosystem for a very long time. What is it about, I guess—the thing I'm wondering about myself sometimes—what is it about the AWS universe that drew you in, and what keeps you here?Tomasz: So, give you some context, I've started, you know, learning about the cloud and AWS back in early-2019. So, fun fact: Maciej Winnicki—again, the co-founder of Cloudash—was my manager at the time. So, we were—I mean, the company I used to work for at the time, OLX Group, we are in the middle of cloud transformation, so to speak. So, going from, you know, on-premises to AWS. And I was, you know, hired as a senior front-end engineer doing, you know, all kinds of front-end stuff, but I wanted to grow, I wanted to learn more.So, the idea was, okay, maybe you can get AWS Certified because, you know, it's one of those corporate goals that you have to have something to put that checkbox next to it. So, you know, getting certified, there you go, you have a checkbox. And off you go. So, I started, you know, diving in, and I saw this whole ocean of things that, you know, I was not entirely aware of. To be fair, at the time I knew about this S3, I knew that you can put a file in an S3 bucket and then you can access it from the internet. This is, like, the [unintelligible 00:16:02] idea of my AWS experiences.Corey: Ideally, intentionally, but one wonders sometimes.Tomasz: Yeah, exactly. That is why you always put stuff as public, right? Because you didn't have to worry about who [unintelligible 00:16:12] [laugh] public [unintelligible 00:16:15]. No, I'm kidding, of course. But still, I think what's [unintelligible 00:16:20] to AWS is what—because it is this endless ocean of things to learn and things to play with, and, you know, things to teach.I do enjoy teaching. As you said, I have quite a lot of, you know, content, videos, blog posts, conference talks, and a bunch of other stuff, and I do that for two reasons. You know, first of all, I tend to learn the best by teaching, so it helps me very much, kind of like, solidify my own knowledge. Whenever I record—like, I have two courses about CDK, you know, when I was recording those, I definitely—that kind of solidify my, you know, ideas about CDK, I get to play with all those technologies.And secondly, you know, it's helpful for others. And, you know, people have opinions about certificates, and so on and so forth, but I think that for somebody who's trying to get into either the tech industry or, you know, cloud stuff in general, being certified helps massively. And I've heard stories about people who are basically managed to double or triple their salaries by going into tech, you know, with some of those certificates. That is why I strongly believe, by the way, that those certificates should be free. Like, if you can pass the exam, you shouldn't have to worry about this $150 of the fee.Corey: I wrote a blog post a while back, “The Dumbest Dollars a Cloud Provider Can Make,” and it's charging for training and certification because if someone's going to invest that kind of time in learning your platform, you're going to try and make $150 bucks off them? Which in some cases, is going to put people off from even beginning that process. “What cloud provider I'm not going to build a project on?” Obviously, the one I know how to work with and have a familiarity with, in almost every case. And the things you learn in your spare time as an independent learner when you get a job, you tend to think about your work the same way. It matters. It's an early on-ramp that pays off down the road and the term of years.I used to be very anti-cert personally because it felt like I was jumping through hoops, and paying, in some cases, for the privilege. I had a CCNA for a while from Cisco. There were a couple of smaller companies, SaltStack, for example, that I got various certifications from at different times. And that was sort of cheating because I helped write the software, but that's neither here nor there. It's the—and I do have a standing AWS cert that I get a different one every time—mine is about to expire—because it gets me access to lounges at physical events, which is the dumbest of all reasons to get certs, but here you go. I view it as the $150 lounge pass with a really weird entrance questionnaire.But in my case it certs don't add anything to what I do. I am not the common case. I am not early in my career. Because as you progress through your career, things—there needs to be a piece of paper that says you know things, and early on degree or certifications are great at that. In the time it becomes your own list of experience on your resume or CV or LinkedIn or God knows what. Polywork if you're doing it the right way these days.And it shows a history of projects that are similar in scope and scale and impact to the kinds of problems that your prospective employer is going to have to solve themselves. Because the best answer to hear—especially in the ops world—when there's a problem is, “Oh, I've seen this before. Here's how you fix it.” As opposed to, “Well, I don't know. Let me do some research.”There's value to that. And I don't begrudge anyone getting certs… to a point. At least that's where I sit on it. At some point when you have 25 certs, it's when you actually do any work? Because it's taking the tests and learning all of these things, which in many ways does boil down to trivia, it stands in counterbalance to a lot of these things.Tomasz: Yeah. I mean, I definitely, totally agree. I remember, you know, going from zero to—maybe not Hero; I'm not talking about AWS Hero—but going from zero to be certified, there was the Solutions Architect Associate. I think it took me, like, 200 hours. I am not the, you know, the brightest, you know, the sharpest tool in the shed, so it probably took me, kind of, somewhat more.I think it's doable in, like, 100 hours, but I tend to over-prepare for stuff, so I didn't actually take the actual exam until I was able to pass the sample exams with, like, 90% pass, just to be extra sure that I'm actually going to pass it. But still, I think that, you know, at some point, you probably should focus on, you know, getting into the actual stuff because I hold two certificates, you know, one of those is going to expire, and I'm not entirely sure if I want to go through the process again. But still, if AWS were to introduce, like, a serverless specialty exam, I would be more than happy to have that. I genuinely enjoy, kind of, serverless, and you know, the fact that I would be able to solidify my knowledge, I have this kind of established path of the things that I should learn about in order to get this particular certificate, I think this could be interesting. But I am not probably going to chase all the 12 certificates.Maybe if AWS IQ was available in Poland, maybe that would change because I do know that with IQ, those certs do matter. But as of [unintelligible 00:21:26] now, I'm quite happy with my certs that I have right now.Corey: Part of the problem, too, is the more you work with these things, the harder it becomes to pass the exams, which sounds weird and counterintuitive, but let me use myself as an example. When I got the cloud practitioner cert, which I believe has lapsed since then, and I got one of the new associate-level betas—I'll keep moving up the stack until I start failing exams. But I got a question wrong on the cloud practitioner because it was, “How long does it take to restore an RDS database from a snapshot backup?” And I gave the honest answer of what I've seen rather than what it says in the book, and that honest answer can be measured in days or hours. Yeah.And no, that's not the correct answer. Yeah, but it is the real one. Similarly, a lot of the questions get around trivia, syntax of which of these is the correct argument, and which ones did we make up? It's, I can explain in some level of detail, virtually every one of AWS has 300 some-odd services to you. Ask me about any of them, I could tell you what it is, how it works, how it's supposed to work and make a dumb joke about it. Fine, whatever.You'll forgive me if I went down that path, instead of memorizing what is the actual syntax of this YAML construct inside of a CloudFormation template? Yeah, I can get the answer to that question in the real world, with about ten seconds of Googling and we move on. That's the way most of us learn. It's not cramming trivia into our heads. There's something broken about the way that we do certifications, and tech interviews in many cases as well.I look back at some of the questions I used to ask people for Linux sysadmin-style jobs, and I don't remember the answer to a lot of these things. I could definitely get back into it, but if I went through one of these interviews now, I wouldn't get the job. One would argue I shouldn't because of my personality, but that's neither here nor there.Tomasz: [laugh]. I mean, that's why you use CDK, so you'd have to remember random YAML comments. And if you [unintelligible 00:23:26] you don't have YAML anymore. [unintelligible 00:23:27].Corey: Yes, you're quite the CDK fanboy, apparently.Tomasz: I do like CDK, yes. I don't like, you know, mental overhead, I don't like context switching, and the way we kind of work at Stedi is everything is written in TypeScript. So, I am a front-end engineer, so I do stuff in the front-end line in TypeScript, all of our Lambda functions are written in TypeScript, and our [unintelligible 00:23:48] is written in TypeScript. So, I can, you know, open up my Visual Studio Code and jump between all of those files, and the language stays the same, the syntax stays the same, the tools stay the same. And I think this is one of the benefits of CDK that is kind of hard to replicate otherwise.And, you know, people have many opinions about the best to deploy infrastructure in the cloud, you know? The best infrastructure-as-code tool is the one that you use at work or in your private projects, right? Because some people enjoy ClickOps like you do; people—Corey: Oh yeah.Tomasz: Enjoy CloudFormation by hand, which I don't; people are very much into Terraform or Serverless Framework. I'm very much into CDK.Corey: Or the SAM CLI, like, three or four more, and I use—Tomasz: Oh, yeah. [unintelligible 00:24:33]—Corey: —all of these things in various ways in some of my [monstrous 00:24:35] projects to keep up on all these things. I did an exploration with the CDK. Incidentally, I think you just answered why I don't like it.Tomasz: Because?Corey: Because it is very clear that TypeScript is a first-class citizen with the CDK. My language of choice is shitty bash because, grumpy old sysadmin; it happens. And increasingly, that is switching over to terrible Python because I'm very bad at that. And the problem that I run into as I was experimenting with this is, it feels like the Python support is not fully baked, most people who are using the CDK are using a flavor of JavaScript and, let's be very clear here, the every time I have tried to explore front-end, I have come away more confused than I was when I started, part of me really thinks I should be learning some JavaScript just because of its versatility and utility to a whole bunch of different problems. But it does not work the way I think, on some level, that it should because of my own biases and experiences. So, if you're not a JavaScript person, I think that you have a much rockier road with the CDK.Tomasz: I agree. Like I said, I tend to talk about my own experiences and my kind of thoughts about stuff. I'm not going to say that, you know, this tool or that tool is the best tool ever because nothing like that exists. Apart from jQuery, which is the best thing that ever happened to the web since, you know, baked bread, honestly. But you are right about CDK, to the best of my knowledge, kind of, all the other languages that are supported by CDK are effectively transpiled down from TypeScript. So it's, like, first of all, it is written in TypeScript, and then kind of the Python, all of the other languages… kind of come second.You know, and afterwards, I tend to enjoy CDK because as I said, I use TypeScript on a daily basis. And you know, with regards to front-end, you mentioned that you are, every single time you is that you end up being more confused. It never goes away. I've been doing front-end stuff for years, and it's, you know, kind of exactly the same. Fun story, I actually joined Cloudash because, well, Maciej started working on Cloudash alone, and after quite some time, he was so frustrated with the modern front-end landscape that he asked me, “Dude, you need to help me. Like, I genuinely need some help. I am tired of React. I am tired of React hooks. This is way too complex. I want to go back to doing back-end stuff. I want to go back, you know, thinking about how we're going to integrate with all those APIs. I don't want to do UI stuff anymore.”Which was kind of like an interesting shift because I remember at the very beginning of my career, where people were talking about front-end—you know, “Front-end is not real programming. Front-end is, you know, it's easy, it's simple. I can learn CSS in an hour.” And the amount of people who say that CSS is easy, and are good at CSS is exactly zero. Literally, nobody who's actually good at CSS says that, you know, CSS, or front-end, or anything like that is easy because it's not. It's incredibly complex. It's getting probably more and more complex because the expectations of our front-end UIs [unintelligible 00:27:44].Corey: It's challenging, it is difficult, and one of the things I find most admirable about you is not even your technical achievements, it's the fact that you're teaching other people to do this. In fact, this gets to the last point I want to cover on our conversation today. When I was bouncing topic ideas off of you, one of the points you brought up that I'm like, “Oh, we're keeping that and saving that for the end,” is why—to your words—why speaking at tech events gets easier, but never easy. Let's dive into that. Tell me more about it.Tomasz: Basically, I've accidentally kickstarted my career by speaking at meetups which later turned into conferences, which later turned into me publishing courses online, which later turned into me becoming an AWS Hero, and here we are, you know, talking to each other. I do enjoy, you know, going out in public and speaking and being on stage. I think, you know, if somebody has, kind of, the heart, the ability to do that, I do strongly recommend, you know, giving it a shot, not only to give, like, an honestly life-changing experience because the first time you go in front of hundreds of people, this is definitely, you know, something that's going to shake you, while at the same time acknowledging that this is absolutely, definitely not for everyone. But if you are able to do that, I think this is definitely worth your time. But as you said—by quoting me—that it gets easier, so every single time you go on stage, talk at a meetup or at a conference or online conferences—which I'm not exactly a fan of, for the record—it's—Corey: It's too much like work, too much like meetings. There's nothing different about it.Tomasz: Yeah, exactly. Like, there's no journey. There's no adventure in online conferences. I know that, of course, you know, given all of that, you know, we had to kind of switch to online conferences for quite some time where I think we are pretending that Covid is not a thing anymore, so we, you know, we're effectively going back, but kind of the point I wanted to make is that I am a somewhat experienced public speaker—I'd like to say that because I've been doing that for years—but I've been, you know, talking to people who actually get paid to speak at the conferences, to actually kind of do that for a living, and they all say the same thing. It gets simpler, it gets easier, but it's never freaking easy, you know, to go out there, and you know, to share whatever you've learned.Corey: I'm one of those people. I am a paid public speaker fairly often, even ignoring the podcast side, and I've spoken on conference stages a couple hundred times at least. And it does get easier but never easy. That's a great way of framing it. You… I get nervous before every talk I give.There are I think two talks I've given that I did not have an adrenaline hit and nervous energy before I went onstage, and both of those were duds. Because I think that it's part of the process, at least for me. And it's like, “Oh, how do you wind up not being scared for before you go on stage?” You don't. You really don't.But if that appeals to you and you enjoy the adrenaline rush of the rest, do it. If you're one of those people who've used public speaking as, “I would prefer death over that,” people are more scared of public speaking their death, in some cases, great. There are so many ways to build audiences and to reach people that fine, if you don't like doing it on stage, don't force yourself to. I'd say try it once; see how it feels meetups are great for this.Tomasz: Yeah. Meetups are basically the best way to get started. I'm yet to meet a meetup, either, you know, offline or online, who is not looking for speakers. It's always quite the opposite, you know? I was, you know, co-organizing a meetup in my city here in Poznań, Poland, and the story always goes like this: “Okay, we have a date. We have a venue. Where are the speakers?” And then you know, the tumbleweed is going to roll across the road and, “Oh, crap, we don't have any speakers.” So, we're going to try to find some, reach out to people. “Hey, I know that you did this fantastic project at your workplace. Come to us, talk about this.” “No, I don't want to. You know, I'm not an expert. I am, you know, I have on the 50 years of experience as an engineer. This is not enough.” Like I said, I do strongly recommend it, but as you said, if you're more scared of public speaking than, like, literally dying, maybe this is not for you.Corey: Yeah. It comes down to stretching your limits, finding yourself interesting. I find that there are lots of great engineers out there. The ones that I find myself drawn to are the ones who aren't just great at building something, but at storytelling around the thing that they are built of, yes, you build something awesome, but you have to convince me to care about it. You have to show me the thing that got you excited about this.And if you can't inspire that excitement in other people, okay. Are you really excited about it? Or what is the story here? And again, it's a different skill set. It is not for everyone, but it is absolutely a significant career accelerator if it's leveraged right.Tomasz: [crosstalk 00:32:45].Corey: [crosstalk 00:32:46] on it.Tomasz: Yeah, absolutely. I think that we don't talk enough about, kind of, the overlap between engineering and marketing. In the good sense of marketing, not the shady kind of marketing. The kind of marketing that you do for yourself in order to elevate yourself, your projects, your successes to others. Because, you know, try as you might, but if you are kind of like sitting in the corner of an office, you know, just jamming on your keyboard 40 hours per week, you're not exactly likely to be promoted because nobody's going to actively reach out to you to find out about your, you know, recent successes and so on.Which at the same time, I'm not saying that you should go @channel in Slack every single time you push a commit to the main branch, but there's definitely, you know, a way of being, kind of, kind to yourself by letting others know that, “Okay, I'm here. I do exist, I have, you know, those particular skills that you may be interested about. And I'm able to tell a story which is, you know, convincing.” So it's, you know, you can tell a story on stage, but you can also tell your story to your customers by building a future that they're going to use. [unintelligible 00:33:50].Corey: I really want to thank you for taking the time to speak with me today. If people want to learn more, where's the best place to find you?Tomasz: So, the best place to find me is on Twitter. So, my Twitter handle is @tlakomy. So, it's T-L-A-K-O-M-Y. I'm assuming this is going to be in the [show notes 00:34:06] as well.Corey: Oh, it absolutely is. You beat me to it.Tomasz: [laugh]. So, you can find Cloudash at cloudash.dev. You can probably also find my email, but don't email me because I'm terrible, absolutely terrible at email, so the best way to kind of reach out to me is via my Twitter DMs. I'm slightly less bad at those.Corey: Excellent. And we will, of course, put links to that in the [show notes 00:34:29]. Thank you so much for being so generous with your time. I appreciate it.Tomasz: Thank you. Thank you for having me.Corey: Tomasz Łakomy, Head of React at Cloudash. I'm Cloud Economist Corey Quinn and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, and if you're on the YouTubes, smash the like and subscribe button, as the kids say. Whereas if you've hated this episode, please do the exact same thing—five-star reviews smash the buttons—but this time also leave an insulting and angry comment written in the form of a CloudWatch log entry that no one is ever able to find in the native interface.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.
About YoavYoav is a security veteran recognized on Microsoft Security Response Center's Most Valuable Research List (BlackHat 2019). Prior to joining Orca Security, he was a Unit 8200 researcher and team leader, a chief architect at Hyperwise Security, and a security architect at Check Point Software Technologies. Yoav enjoys hunting for Linux and Windows vulnerabilities in his spare time.Links Referenced: Orca Security: https://orca.security Twitter: https://twitter.com/yoavalon TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by our friends at Vultr. Optimized cloud compute plans have landed at Vultr to deliver lightning fast processing power, courtesy of third gen AMD EPYC processors without the IO, or hardware limitations, of a traditional multi-tenant cloud server. Starting at just 28 bucks a month, users can deploy general purpose, CPU, memory, or storage optimized cloud instances in more than 20 locations across five continents. Without looking, I know that once again, Antarctica has gotten the short end of the stick. Launch your Vultr optimized compute instance in 60 seconds or less on your choice of included operating systems, or bring your own. It's time to ditch convoluted and unpredictable giant tech company billing practices, and say goodbye to noisy neighbors and egregious egress forever. Vultr delivers the power of the cloud with none of the bloat. "Screaming in the Cloud" listeners can try Vultr for free today with a $150 in credit when they visit getvultr.com/screaming. That's G E T V U L T R.com/screaming. My thanks to them for sponsoring this ridiculous podcast.Corey: Finding skilled DevOps engineers is a pain in the neck! And if you need to deploy a secure and compliant application to AWS, forgettaboutit! But that's where DuploCloud can help. Their comprehensive no-code/low-code software platform guarantees a secure and compliant infrastructure in as little as two weeks, while automating the full DevSecOps lifestyle. Get started with DevOps-as-a-Service from DuploCloud so that your cloud configurations are done right the first time. Tell them I sent you and your first two months are free. To learn more visit: snark.cloud/duplocloud. Thats's snark.cloud/D-U-P-L-O-C-L-O-U-D. Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn. Periodically, I would say that I enjoy dealing with cloud platform security issues, except I really don't. It's sort of forced upon me to deal with much like a dead dog is cast into their neighbor's yard for someone else to have to worry about. Well, invariably, it seems like it's my yard.And I'm only on the periphery of these things. Someone who's much more in the trenches in the wide world of cloud security is joining me today. Yoav Alon is the CTO at Orca Security. Yoav, thank you for taking the time to join me today and suffer the slings and arrows I'll no doubt be hurling your way.Yoav: Thank you, Corey, for having me. I've been a longtime listener, and it's an honor to be here.Corey: I still am periodically surprised that anyone listens to these things. Because it's unlike a newsletter where everyone will hit reply and give me a piece of their mind. People generally don't wind up sending me letters about things that they hear on the podcast, so whenever I talk to somebody listens to it as, “Oh. Oh, right, I did turn the microphone on. Awesome.” So, it's always just a little on the surreal side.But we're not here to talk necessarily about podcasting, or the modern version of an AM radio show. Let's start at the very beginning. What is Orca Security, and why would folks potentially care about what it is you do?Yoav: So, Orca Security is a cloud security company, and our vision is very simple. Given a customer's cloud environment, we want to detect all the risks in it and implement mechanisms to prevent it from occurring. And while it sounds trivial, before Orca, it wasn't really possible. You will have to install multiple tools and aggregate them and do a lot of manual work, and it was messy. And we wanted to change that, so we had, like, three guiding principles.We call it seamless, so I want to detect all the risks in your environment without friction, which is our speak for fighting with your peers. We also want to detect everything so you don't have to install, like, a tool for each issue: A tool for vulnerabilities, a tool for misconfigurations, and for sensitive data, IAM roles, and such. And we put a very high priority on context, which means telling you what's important, what's not. So, for example, S3 bucket open to the internet is important if it has sensitive data, not if it's a, I don't know, static website.Corey: Exactly. I have a few that I'd like to get screamed at in my AWS account, like, “This is an open S3 bucket and it's terrible.” I look at it the name is assets.lastweekinaws.com. Gee, I wonder if that's something that's designed to be a static hosted website.Increasingly, I've been slapping CloudFront in front of those things just to make the broken warning light go away. I feel like it's an underhanded way of driving CloudFront adoption some days, but not may not be the most charitable interpretation thereof. Orca has been top-of-mind for a lot of folks in the security community lately because let's be clear here, dealing with security problems in cloud providers from a vendor perspective is an increasingly crowded—and clouded—space. Just because there's so much—there's investment pouring into it, everyone has a slightly different take on the problem, and it becomes somewhat challenging to stand out from the pack. You didn't really stand out from the pack so much as leaped to the front of it and more or less have become the de facto name in a very short period of time, specifically—at least from my world—when you wound up having some very interesting announcements about vulnerabilities within AWS itself. You will almost certainly do a better job of relating the story, so please, what did you folks find?Yoav: So, back in September of 2021, two of my researchers, Yanir Tsarimi and Tzah Pahima, each one of them within a relatively short span of time from each other, found a vulnerability in AWS. Tzah found a vulnerability in CloudFormation which we named BreakingFormation and Yanir found a vulnerability in AWS Glue, which we named SuperGlue. We're not the best copywriters, but anyway—Corey: No naming things is hard. Ask any Amazonian.Yoav: Yes. [laugh]. So, I'll start with BreakingFormation which caught the eyes of many. It was an XXE SSRF, which is jargon to say that we were able to read files and execute HTTP requests and read potentially sensitive data from CloudFormation servers. This one was mitigated within 26 hours by AWS, so—Corey: That was mitigated globally.Yoav: Yes, globally, which I've never seen such quick turnaround anywhere. It was an amazing security feat to see.Corey: Particularly in light of the fact that AWS does a lot of things very right when it comes to, you know, designing cloud infrastructure. Imagine that, they've had 15 years of experience and basically built the idea of cloud, in some respects, at the scale that hyperscalers operate at. And one of their core tenets has always been that there's a hard separation between regions. There are remarkably few global services, and those are treated with the utmost of care and delicacy. To the point where when something like that breaks as an issue that spans more than one region, it is headline-making news in many cases.So it's, they almost never wind up deploying things to all regions at the same time. That can be irksome when we're talking about things like I want a feature that solves a problem that I have, and I have to wait months for it to hit a region that I have resources living within, but for security, stuff like this, I am surprised that going from, “This is the problem,” to, “It has been mitigated,” took place within 26 hours. I know it sounds like a long time to folks who are not deep in the space, but that is superhero speed.Yoav: A small correction, it's 26 hours for, like, the main regions. And it took three to four days to propagate to all regions. But still, it's speed of lighting in for security space.Corey: When this came out, I was speaking to a number of journalists on background about trying to wrap their head around this, and they said that, “Oh yeah, and security is always, like, the top priority for AWS, second only to uptime and reliability.” And… and I understand the perception, but I disagree with it in the sense of the nightmare scenario—that every time I mention to a security person watching the blood drain from their face is awesome—but the idea that take IAM, which as Werner said in his keynote, processes—was it 500 million or was it 500 billion requests a second, some ludicrous number—imagine fails open where everything suddenly becomes permitted. I have to imagine in that scenario, they would physically rip the power cables out of the data centers in order to stop things from going out. And that is the right move. Fortunately, I am extremely optimistic that will remain a hypothetical because that is nightmare fuel right there.But Amazon says that security is job zero. And my cynical interpretation is that well, it wasn't, but they forgot security, decided to bolt it on to the end, like everyone else does, and they just didn't want to renumber all their slides, so instead of making it point one, they just put another slide in front of it and called the job zero. I'm sure that isn't how it worked, but for those of us who procrastinate and building slide decks for talks, it has a certain resonance to it. That was one issue. The other seemed a little bit more pernicious focusing on Glue, which is their ETL-as-a-Service… service. One of them I suppose. Tell me more about it.Yoav: So, one of the things that we found when we found the BreakingFormation when we reported the vulnerability, it led us to do a quick Google search, which led us back to the Glue service. It had references to Glue, and we started looking around it. And what we were able to do with the vulnerability is given a specific feature in Glue, which we don't disclose at the moment, we were able to effectively take control over the account which hosts the Glue service in us-east-1. And having this control allowed us to essentially be able to impersonate the Glue service. So, every role in AWS that has a trust to the Glue service, we were able to effectively assume a role into it in any account in AWS. So, this was more critical a vulnerability in its effect.Corey: I think on some level, the game of security has changed because for a lot of us who basically don't have much in the way of sensitive data living in AWS—and let's be clear, I take confidentiality extremely seriously. Our clients on the consulting side view their AWS bills themselves as extremely confidential information that Amazon stuffs into a PDF and emails every month. But still. If there's going to be a leak, we absolutely do not want it to come from us, and that is something that we take extraordinarily seriously. But compared to other jobs I've had in the past, no one will die if that information gets out.It is not the sort of thing that is going to ruin people's lives, which is very often something that can happen in some data breaches. But in my world, one of the bad cases of a breach of someone getting access to my account is they could spin up a bunch of containers on the 17 different services that AWS offers that can run containers and mine cryptocurrency with it. And the damage to me then becomes a surprise bill. Okay, great. I can live with that.Something that's a lot scarier to a lot of companies with, you know, serious problems is, yep, fine, cost us money, whatever, but our access to our data is the one thing that is going to absolutely be the thing that cannot happen. So, from that perspective alone, something like Glue being able to do that is a lot more terrifying than subverting CloudFormation and being able to spin up additional resources or potentially take resources down. Is that how you folks see it too, or is—I'm sure there's nuance I'm missing.Yoav: So yeah, the access to data is top-of-mind for everyone. It's a bit scary to think about it. I have to mention, again, the quick turnaround time for AWS, which almost immediately issued a patch. It was a very fast one and they mitigated, again, the issue completely within days. About your comment about data.Data is king these days, there is nothing like data, and it has all the properties of everything that we care about. It's expensive to store, it's expensive to move, and it's very expensive if it leaks. So, I think a lot of people were more alarmed about the Glue vulnerability than the CloudFormation vulnerability. And they're right in doing so.Corey: I do want to call out that AWS did a lot of things right in this area. Their security posture is very clearly built around defense-in-depth. The fact that they were able to disclose—after some prodding—that they checked the CloudTrail logs for the service itself, dating back to the time the service launched, and verified that there had never been an exploit of this, that is phenomenal, as opposed to the usual milquetoast statements that companies have. We have no evidence of it, which can mean that we did the same thing and we looked through all the logs in it's great, but it can also mean that, “Oh, yeah, we probably should have logs, shouldn't we? But let's take a backlog item for that.” And that's just terrifying on some level.It becomes a clear example—a shining beacon for some of us in some cases—of doing things right from that perspective. There are other sides to it, though. As a customer, it was frustrating in the extreme to—and I mean, no offense by this—to learn about this from you rather than from the provider themselves. They wound up putting up a security notification many hours after your blog post went up, which I would also just like to point out—and we spoke about it at the time and it was a pure coincidence—but there was something that was just chef's-kiss perfect about you announcing this on Andy Jassy's birthday. That was just very well done.Yoav: So, we didn't know about Andy's birthday. And it was—Corey: Well, I see only one of us has a company calendar with notable executive birthdays splattered all over it.Yoav: Yes. And it was also published around the time that AWS CISO was announced, which was also a coincidence because the date was chosen a lot of time in advance. So, we genuinely didn't know.Corey: Communicating around these things is always challenging because on the one hand, I can absolutely understand the cloud providers' position on this. We had a vulnerability disclosed to us. We did our diligence and our research because we do an awful lot of things correctly and everyone is going to have vulnerabilities, let's be serious here. I'm not sitting here shaking my fist, angry at AWS's security model. It works, and I am very much a fan of what they do.And I can definitely understand then, going through all of that there was no customer impact, they've proven it. What value is there to them telling anyone about it, I get that. Conversely, you're a security company attempting to stand out in a very crowded market, and it is very clear that announcing things like this demonstrates a familiarity with cloud that goes beyond the common. I radically changed my position on how I thought about Orca based upon these discoveries. It went from, “Orca who,” other than the fact that you folks have sponsored various publications in the past—thanks for that—but okay, a security company. Great to, “Oh, that's Orca. We should absolutely talk to them about a thing that we're seeing.” It has been transformative for what I perceive to be your public reputation in the cloud security space.So, those two things are at odds: The cloud provider doesn't want to talk about anything and the security company absolutely wants to demonstrate a conversational fluency with what is going on in the world of cloud. And that feels like it's got to be a very delicate balancing act to wind up coming up with answers that satisfy all parties.Yoav: So, I just want to underline something. We don't do what we do in order to make a marketing stand. It's a byproduct of our work, but it's not the goal. For the Orca Security Research Pod, which it's the team at Orca which does this kind of research, our mission statement is to make cloud security better for everyone. Not just Orca customers; for everyone.And you get to hear about the more shiny things like big headline vulnerabilities, but we also have very sensible blog posts explaining how to do things, how to configure things and give you more in-depth understanding into security features that the cloud providers themselves provide, which are great, and advance the state of the cloud security. I would say that having a cloud vulnerability is sort of one of those things, which makes me happy to be a cloud customer. On the one side, we had a very big vulnerability with very big impact, and the ability to access a lot of customers' data is conceptually terrifying. The flip side is that everything was mitigated by the cloud providers in warp speed compared to everything else we've seen in all other elements of security. And you get to sleep better knowing that it happened—so no platform is infallible—but still the cloud provider do work for you, and you'll get a lot of added value from that.Corey: You've made a few points when this first came out, and I want to address them. The first is, when I reached out to you with a, “Wow, great work.” You effectively instantly came back with, “Oh, it wasn't me. It was members of my team.” So, let's start there. Who was it that found these things? I'm a huge believer giving people credit for the things that they do.The joy of being in a leadership position is if the company screws up, yeah, you take responsibility for that, whether the company does something great, yeah, you want to pass praise onto the people who actually—please don't take this the wrong way—did the work. And not that leadership is not work, it absolutely is, but it's a different kind of work.Yoav: So, I am a security researcher, and I am very mindful for the effort and skill it requires to find vulnerabilities and actually do a full circle on them. And the first thing I'll mention is Tzah Pahima, which found the BreakingFormation vulnerability and the vulnerability in CloudFormation, and Yanir Tsarimi, which found the AutoWarp vulnerability, which is the Azure vulnerability that we have not mentioned, and the Glue vulnerability, dubbed SuperGlue. Both of them are phenomenal researcher, world-class, and I'm very honored to work with them every day. It's one of my joys.Corey: Couchbase Capella Database-as-a-Service is flexible, full-featured and fully managed with built in access via key-value, SQL, and full-text search. Flexible JSON documents aligned to your applications and workloads. Build faster with blazing fast in-memory performance and automated replication and scaling while reducing cost. Capella has the best price performance of any fully managed document database. Visit couchbase.com/screaminginthecloud to try Capella today for free and be up and running in three minutes with no credit card required. Couchbase Capella: make your data sing.Corey: It's very clear that you have built an extraordinary team for people who are able to focus on vulnerability research. Which, on some level, is very interesting because you are not branded as it were as a vulnerability research company. This is not something that is your core competency; it's not a thing that you wind up selling directly that I'm aware of. You are selling a security platform offering. So, on the one hand, it makes perfect sense that you would have a division internally that works on this, but it's also very noteworthy, I think, that is not the core description of what it is that you do.It is a means by which you get to the outcome you deliver for customers, not the thing that you are selling directly to them. I just find that an interesting nuance.Yoav: Yes, it is. And I would elaborate and say that research informs the product, and the product informs research. And we get to have this fun dance where we learn new things by doing research. We [unintelligible 00:18:08] the product, and we use the customers to teach us things that we didn't know. So, it's one of those happy synergies.Corey: I want to also highlight a second thing that you have mentioned and been very, I guess, on message about since news of this stuff first broke. And because it's easy to look at this and sensationalize aspects of it, where, “See? The cloud providers security model is terrible. You shouldn't use them. Back to data centers we go.” Is basically the line taken by an awful lot of folks trying to sell data center things.That is not particularly helpful for the way that the world is going. And you've said, “Yeah, you should absolutely continue to be in cloud. Do not disrupt your cloud plan as a result.” And let's be clear, none of the rest of us are going to find and mitigate these things with anything near the rigor or rapidity that the cloud providers can and do demonstrate.Yoav: I totally agree. And I would say that the AWS security folks are doing a phenomenal job. I can name a few, but they're all great. And I think that the cloud is by far a much safer alternative than on-prem. I've never seen issues in my on-prem environment which were critical and fixed in such a high velocity and such a massive scale.And you always get the incremental improvements of someone really thinking about all the ins and outs of how to do security, how to do security in the cloud, how to make it faster, more reliable, without a business interruptions. It's just phenomenal to see and phenomenal to witness how far we've come in such a relatively short time as an industry.Corey: AWS in particular, has a reputation for being very good at security. I would argue that, from my perspective, Google is almost certainly slightly better at their security approach than AWS is, but to be clear, both of them are significantly further along the path than I am going to be. So great, fantastic. You also have found something interesting over in the world of Azure, and that honestly feels like a different class of vulnerability. To my understanding, the Azure vulnerability that you recently found was you could get credential material for other customers simply by asking for it on a random high port. Which is one of those—I'm almost positive I'm misunderstanding something here. I hope. Please?Yoav: I'm not sure you're misunderstanding. So, I would just emphasize that the vulnerability again, was found by Yanir Tsarimi. And what he found was, he used a service called Azure Automation which enables you essentially to run a Python script on various events and schedules. And he opened the python script and he tried different ports. And one of the high ports he found, essentially gave him his credentials. And he said, “Oh, wait. That's a really odd port for an HTTP server. Let's try, I don't know, a few ports on either way.” And he started getting credentials from other customers. Which was very surprising to us.Corey: That is understating it by a couple orders of magnitude. Yes, like, “Huh. That seems sub-optimal,” is sort of like the corporate messaging approved thing. At the time you discover that—I'm certain it was a three-minute-long blistering string of profanity in no fewer than four languages.Yoav: I said to him that this is, like, a dishonorable bug because he worked very little to find it. So it was, from start to finish, the entire research took less than two hours, which, in my mind, is not enough for this kind of vulnerability. You have to work a lot harder to get it. So.Corey: Yeah, exactly. My perception is that when there are security issues that I have stumbled over—for example, I gave a talk at re:Invent about it in the before times, one of them was an overly broad permission in a managed IAM policy for SageMaker. Okay, great. That was something that obviously was not good, but it also was more of a privilege escalation style of approach. It wasn't, “Oh, by the way, here's the keys to everything.”That is the type of vulnerability I have come to expect, by and large, from cloud providers. We're just going to give you access credentials for other customers is one of those areas that… it bugs me on a visceral level, not because I'm necessarily exposed personally, but because it more or less shores up so many of the arguments that I have spent the last eight years having with folks are like, “Oh, you can't go to cloud. Your data should live on your own stuff. It's more secure that way.” And we were finally it feels like starting to turn a cultural corner on these things.And then something like that happens, and it—almost have those naysayers become vindicated for it. And it's… it almost feels, on some level, and I don't mean to be overly unkind on this, but it's like, you are absolutely going to be in a better security position with the cloud providers. Except to Azure. And perhaps that is unfair, but it seems like Azure's level of security rigor is nowhere near that of the other two. Is that generally how you're seeing things?Yoav: I would say that they have seen more security issues than most other cloud providers. And they also have a very strong culture of report things to us, and we're very streamlined into patching those and giving credit where credit's due. And they give out bounties, which is an incentives for more research to happen on those platforms. So, I wouldn't say this categorically, but I would say that the optics are not very good. Generally, the cloud providers are much safer than on-prem because you only hear very seldom on security issues in the cloud.You hear literally every other day on issues happening to on-prem environments all over the place. And people just say they expect it to be this way. Most of the time, it's not even a headline. Like, “Company X affected with cryptocurrency or whatever.” It happens every single day, and multiple times a day, breaches which are massively bigger. And people who don't want to be in the cloud will find every reason not to be the cloud. Let us have fun.Corey: One of the interesting parts about this is that so many breaches that are on-prem are just never discovered because no one knows what the heck's running in an environment. And the breaches that we hear about are just the ones that someone had at least enough wherewithal to find out that, “Huh. That shouldn't be the way that it is. Let's dig deeper.” And that's a bad day for everyone. I mean, no one enjoys those conversations and those moments.And let's be clear, I am surprisingly optimistic about the future of Azure Security. It's like, “All right, you have a magic wand. What would you do to fix it?” It's, “Well, I'd probably, you know, hire Charlie Bell and get out of his way,” is not a bad answer as far as how these things go. But it takes time to reform a culture, to wind up building in security as a foundational principle. It's not something you can slap on after the fact.And perhaps this is unfair. But Microsoft has 30 years of history now of getting the world accustomed to oh, yeah, just periodically, terrible vulnerabilities are going to be discovered in your desktop software. And every once a month on Tuesdays, we're going to roll out a whole bunch of patches, and here you go. Make sure you turn on security updates, yadda, yadda, yadda. That doesn't fly in the cloud. It's like, “Oh, yeah, here's this month's list of security problems on your cloud provider.” That's one of those things that, like, the record-scratch, freeze-frame moment of wait, what are we doing here, exactly?Yoav: So, I would say that they also have a very long history of making those turnarounds. Bill Gates famously did his speech where security comes first, and they have done a very, very long journey and turn around the company from doing things a lot quicker and a lot safer. It doesn't mean they're perfect; everyone will have bugs, and Azure will have more people finding bugs into it in the near future, but security is a journey, and they've not started from zero. They're doing a lot of work. I would say it's going to take time.Corey: The last topic I want to explore a little bit is—and again, please don't take this as anyway being insulting or disparaging to your company, but I am actively annoyed that you exist. By which I mean that if I go into my AWS account, and I want to configure it to be secure. Great. It's not a matter of turning on the security service, it's turning on the dozen or so security services that then round up to something like GuardDuty that then, in turn, rounds up to something like Security Hub. And you look at not only the sheer number of these services and the level of complexity inherent to them, but then the bill comes in and you do some quick math and realize that getting breached would have been less expensive than what you're spending on all of these things.And somehow—the fact that it's complex, I understand; computers are like that. The fact that there is—[audio break 00:27:03] a great messaging story that's cohesive around this, I come to accept that because it's AWS; talking is not their strong suit. Basically declining to comment is. But the thing that galls me is that they are selling these services and not inexpensively either, so it almost feels, on some level like, shouldn't this on some of the built into the offerings that you folks are giving us?And don't get me wrong, I'm glad that you exist because bringing order to a lot of that chaos is incredibly important. But I can't shake the feeling that this should be a foundational part of any cloud offering. I'm guessing you might have a slightly different opinion than mine. I don't think you show up at the office every morning, “I hate that we exist.”Yoav: No. And I'll add a bit of context and nuance. So, for every other company than cloud providers, we expect them to be very good at most things, but not exceptional at everything. I'll give the Redshift example. Redshift is a pretty good offering, but Snowflake is a much better offering for a much wider range of—Corey: And there's a reason we're about to become Snowflake customers ourselves.Yoav: So, yeah. And there are a few other examples of that. A security company, a company that is focused solely on your security will be much better suited to help you, in a lot of cases more than the platform. And we work actively with AWS, Azure, and GCP requesting new features, helping us find places where we can shed more light and be more proactive. And we help to advance the conversation and make it a lot more actionable and improve from year to year. It's one of those collaborations. I think the cloud providers can do anything, but they can't do everything. And they do a very good job at security; it doesn't mean they're perfect.Corey: As you folks are doing an excellent job of demonstrating. Again, I'm glad you folks exist; I'm very glad that you are publishing the research that you are. It's doing a lot to bring a lot I guess a lot of the undue credit that I was giving AWS for years of, “No, no, it's not that they don't have vulnerabilities like everyone else does. It just that they don't ever talk about them.” And they're operationalizing of security response is phenomenal to watch.It's one of those things where I think you've succeeded and what you said earlier that you were looking to achieve, which is elevating the state of cloud security for everyone, not just Orca customers.Yoav: Thank you.Corey: Thank you. I really appreciate your taking the time out of your day to speak with me. If people want to learn more, where's the best place they can go to do that?Yoav: So, we have our website at orca.security. And you can reach me out on Twitter. My handle is at @yoavalon, which is @-Y-O-A-V-A-L-O-N.Corey: And we will of course put links to that in the [show notes 00:29:44]. Thanks so much for your time. I appreciate it.Yoav: Thank you, Corey.Corey: Yoav Alon, Chief Technology Officer at Orca Security. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, or of course on YouTube, smash the like and subscribe buttons because that's what they do on that platform. Whereas if you've hated this podcast, please do the exact same thing, five-star review, smash the like and subscribe buttons on YouTube, but also leave an angry comment that includes a link that is both suspicious and frightening, and when we click on it, suddenly our phones will all begin mining cryptocurrency.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.
About ScottCloud security historian.Developed flaws.cloud, CloudMapper, and Parliament.Founding team for fwd:cloudsecLinks: Block: https://block.xyz/ Twitter: https://twitter.com/0xdabbad00 TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: This episode is sponsored in part by our friends at Vultr. Optimized cloud compute plans have landed at Vultr to deliver lightning fast processing power, courtesy of third gen AMD EPYC processors without the IO, or hardware limitations, of a traditional multi-tenant cloud server. Starting at just 28 bucks a month, users can deploy general purpose, CPU, memory, or storage optimized cloud instances in more than 20 locations across five continents. Without looking, I know that once again, Antarctica has gotten the short end of the stick. Launch your Vultr optimized compute instance in 60 seconds or less on your choice of included operating systems, or bring your own. It's time to ditch convoluted and unpredictable giant tech company billing practices, and say goodbye to noisy neighbors and egregious egress forever. Vultr delivers the power of the cloud with none of the bloat. "Screaming in the Cloud" listeners can try Vultr for free today with a $150 in credit when they visit getvultr.com/screaming. That's G E T V U L T R.com/screaming. My thanks to them for sponsoring this ridiculous podcast.Corey: Couchbase Capella Database-as-a-Service is flexible, full-featured and fully managed with built in access via key-value, SQL, and full-text search. Flexible JSON documents aligned to your applications and workloads. Build faster with blazing fast in-memory performance and automated replication and scaling while reducing cost. Capella has the best price performance of any fully managed document database. Visit couchbase.com/screaminginthecloud to try Capella today for free and be up and running in three minutes with no credit card required. Couchbase Capella: make your data sing.Corey: Welcome to Screaming in the Cloud, I'm Corey Quinn. I am joined by a returning guest with a bit of a different job. Scott Piper was formerly an independent security researcher—basically the independent security researcher in the AWS space—but now he's a Principal Engineer over at Block. Scott, welcome back.Scott: Thanks for having me, again, Corey.Corey: So, you've taken a corporate job, and when that happened, I have to confess, I was slightly discouraged because oh, now it's going to be like one of those stories of when someone you know goes to work at Apple because no one knows anyone at Apple; we just used to know people who went there and then we kind of lost touch because it's a very insular thing. Not the Block slash Square slash whatever they're calling themselves this week has that reputation. But InfoSec is always a very nuanced space and companies that have large footprints and, you know, handle financial transaction processing generally don't encourage loud voices that attract attention around anything that isn't directly aligned with the core mission of the company. But you're still as public and prolific as ever. Was that a difficult balance for you to strike?Scott: So, when I was considering employment options, that was something that I made clear to any companies that I was talking to, that this is something that probably will and should continue because a lot of my value to these companies is because I'm able to have discussions, able to impact change because of that public persona. So yeah, so I think that it was something that they were aware of, and a risk that they took. [laugh]. But yeah, it's been useful.Corey: This is the sort of conversation I would have expected to have with, “Yeah, things seem to be continuing the same, and I haven't rocked any boats, yet and they haven't fired me, knock on wood.” Except that recently you've launched yet something else that I am personally a fan of. Now, before we get into the specifics of what it is you're up to these days, I should call out that since your last appearance on this show, I have really leaned into the Thursday newsletter podcast duo of Last Week in AWS: Security Edition. Rounding up what happened the previous week—yes, it was the previous week, and it comes out on Thursdays—because, you know, timing and publication, things are hard, computers, you know how it is—aimed at a target audience that is very much not you: People who have to care about security, but are not immersed in the space. It's a, “All right, what now? What do I have to pay attention to?”Because there's a lot of noise in this space, there's a lot of vendor-captured stuff out there. There's very little that is for people who work in security but don't have the word security anywhere near their job title. And I have to confess that one of my easy shortcuts is, “Oh, it's a pretty thin issue this week,” which is not inherently a bad thing, let's be clear, it's not yay, the three things you need to care about in security then eight more of filler; that's not what we're about. But I always want to make sure I didn't miss something meaningful, and one of my default publication steps is, “What's Scott been tweeting about this week?” Just to make sure that I didn't miss something that I really should be talking about.And every single time I pull up your Twitter feed, I find myself learning something, whether it's a new concept, or whether it is a nuance on an existing thing I was already aware of. So first, thank you for all the work that you do as a member of the community, despite having a, “Regular corporate job,” quote-unquote, you're still very present. It's appreciated.Scott: Thank you. Yeah. And I mean, that newsletter is great for people that don't want to be spending multiple hours per day trolling through Twitter and reading that. So, it provides, also, something great for the community to not have to spend all that time on Twitter like I do [laugh], unfortunately.Corey: It also strives—sort of—to be something approaching an upbeat position of not quite as cynical and sarcastic as the Monday issue. I try to be not just this is the thing that happened, but go a little bit into and this is why it matters. This is how to think about it. This thing that Amazon put out is nonsense, however, here's the kernel hidden within it that might lead to something, such as thinking about how you do sign-on, or how to think about protecting MFA devices, or stuff like that you normally care about a lot right after you really should have cared about it but didn't at all. So, it's just the idea of aiming in a slightly different audience.Scott: Yeah definitely. And it provides value that it does, it takes some delay so that you can read what everybody has written, how they've responded to the different news outtakes, you're not just including the hot takes. For example, as of this morning, there's a certain incident with an authentication provider, and it's not really clear if there was actually a breach or not. And so it's valuable to take a moment to understand what happened, get all the voices to have expressed their points, so you can summarize those issues.Corey: An internal term that we've used to describe the position here is that I am prolific but I also have things to do as a part of my job that do not involve sitting there hitting refresh on Twitter like mad all the time. The idea is to have the best take not the first take—Scott: Exactly.Corey: And if that means that I lose a bunch of eyeballs and early ad impressions in the middle of the night and whatnot, well, great. I don't sell ad impressions anyway, so what does it matter? It winds up lending itself to a more thoughtful analysis of figuring out, in the sober light of day, is this a nothing-burger or is this enormous? With that SSO issue that you're alluding to—[cough] Okta—sorry, something caught in my throat there—very clearly, something is going on, but if I had written next week's newsletter last night while it was still very unclear, it would have been a very different tone than the one that I would have written this morning after their public statement, and even still a certainly different tone that it would take a couple of days once more information is almost certain to come to light. And that is something that is, I think, underappreciated in certainly on Twitter, where an old tweet—there's nothing worse than an old tweet unless you're using it to drag someone for something—that, “Well, we have different perspectives on that nowadays. It's not 2018 anymore.” Right. Okay, cool.Scott: Yep. [laugh].Corey: But something that you've done has been a bit of a pivot lately. Historically, you have been right there in my sweet spot of needling cloud providers for their transgressions in various ways. Cool, right there with you. We could co-author a book on the subject. But lately, you've started a community list of [IMSDv2 00:07:04] abuses.Now, first, we should talk about what IMSDv2 is. It's the name that it clearly came from Amazon because that's a name only a cloud provider bad at naming things could possibly love. What is it?Scott: So, it's the Instance Metadata Service, Version Two. If there's a version two, you can imagine there was a version one at some point. And the version two—Corey: And there's a version two because Amazon prod—the first one was terrible, but they don't turn anything off, ever, so this is the way and the light and the future; we're going to leave that old thing around until your great-grandchild dies of old age.Scott: Exactly, yeah. So, when EC2s first came out, and IAM roles first came out, you wanted to give your EC2s the ability to use AWS privileges, so this is how those EC2s are getting access to their credentials that they can use. And the way in which this was originally done was there's this magic IP address, this 169.254.169.254 IP address, which is very important for security on AWS because if anything can access that magic IP address from an EC2 instance, you can steal their credentials of that EC2, and therefore basically become that EC2 instance, in terms of what it can do in the AWS environment.And so in 2019, there was a large breach of Capital One that was related to this. And so as a result of that—I think that AWS probably had this new version, probably, in the works for a while, but I think that motivated their faster release of this new version, and so IMDSv2 changed how you would obtain these credentials. So, you basically—instead of making a single GET request to this IP address, now you had to make multiple requests, they were now PUT request instead of a GET request, there was a challenge and response, there's the hop limit. So, there's all these various things that are going to make it harder and basically mitigate a lot of the different types of vulnerabilities that previously would be used in order to obtain these credentials. The problem, though, is that IMDSv1 still exists on EC2s, unless you as a customer are enforcing IMDSv2.And so, in order to do this in a large environment, it's difficult—theoretically, it's a simple thing; all you should have to do is update your SDK and now you're able to make use of the latest version. And if you're using any version of the SDK that was released in the past over two years, you already should be using IMDSv2 there, but you have to enforce it. And so that's where the problem is. And what was most problematic to me is now that I work for a company, we have run into the problem that there are some vendor solutions that we use that weren't allowing us to enforce IMDSv2 across all of our different accounts. And this is something I've heard from a number of other customers as well.And so I decided to create this list with vendors that I've had to deal with, vendors that other customers have had to deal with, in order to basically try and solve this problem once and for all. It's been multiple years now and a lot of these vendors, unfortunately, were also security vendors. And so that makes the conversation a little bit easier, to basically put them on this wall-of-shame and say, “You're a security vendor and you're not allowing your customers to enforce best practices of security.”Corey: I want to call on a couple of things around that. Originally the metadata service was used for a number of other things—still is—beyond credentials. It is not the credential service as envisioned by a lot of folks. The way that—also we'll find those credentials empty until there's an EC2 instance role, and those credentials will both be scoped what that instance does and automatically rotated in the fullness of time so they're not long-lived credentials that once you have them, they will last forever. This is, of course, a best practice and something you should be leveraging, but scope those credentials down, or you wind up with one of the ways that was chained together in the Capital One breach a few years ago.It's also worth noting that service would have been more useful earlier in time with a few functions. For example, you can use the metadata service to retrieve the instance tags about the EC2 instance. When I requested it in 2015, it was not possible. But they had released it in January of this year, 2022, long after we have all come up with workarounds for this, where we could have used that to set the hostname internally on the system, if you're looking for something basic and easy. It would have been something then you could have used to automatically self-register with DNS without having to jump through a whole bunch of hoops to do it manually.And you look at this, and it's wow, that's a whole lot of crappy tooling I can just throw into the trash heap of history you don't need anymore. But the IMSDv2, you're right, makes it a lot harder, there has to be a conversation, not just something you can sort of bankshot something off of to get access to it. And it's a terrific mitigation. What I've liked about your list of more or less shaming companies for doing this is, on the one hand, you have companies who take themselves off of the list as soon as it's up there. It's, “Oh, we love when people talk about us. Wait, what's that? They're saying something unkind? On the internet?” And they'll fix it, which honestly is better than I expected.And then every once in a while you'll see something that's horrifying of, “Oh, yeah, we're not vulnerable to that at all because we tell you to create permanent long-lived credentials, store them on disk and we'll use those instead.” And it's… that is, like, guaranteeing that no one is going to break down your door by making your walls out of tissue paper. Don't do that. Like, that has gone so far around the band that has come back around again. So, hopefully that got fixed.Scott: And I think you pointed out a couple of things I want to talk about with this is that, one, it has actually been very successful in terms of getting large vendors to make changes. Currently, of the seven vendors that have ever been listed there, are three of them have already made fixes and have been removed from the list. And the list has only been up for about a month. And so, in terms of getting enterprise solution vendors to make changes within, like, just a few weeks is very surprising to me. And these are things that people have been asking for for years now, and so it had motivated them a lot there.And the other thing that I want to point out is people have looked at the success that it's had and considered maybe we should make wall-of-shame lists, for all the things that we want. And I want to point out that there are some things about this problem, the IMDSv2 specifically, that make it work for having this wall-of-shame list like this. One of them is that not supporting or not allowing customers to enforce IMDSv2 is basically always bad. There is not a use case where you can make a claim—Corey: There is no nuance where that, in this case, is the thing to do, like having an open S3 bucket: There are use cases where that is very much something you want to do, but it's the uncommon case.Scott: Exactly. That I think is an important thing. Another thing is it's not just putting up a list, you know, like that is what people are seeing publicly, but behind the scenes, there's a lot of other things that are happening. One, I am communicating with various customers, customers that are reporting this issue to me, in order to try to better understand what's happening there, so that I can then relay that information to the company. So, I'm not just putting up the list; I'm also, behind the scenes, having conversations with these different companies to try to get timelines from them, to try to make sure that they are aware of the problem, they are aware that they're on this list, how to get off the list. So, there's that conversation happening.There's also the conversation that I'm happening with AWS in order to make various requests that AWS improve this for customers, to make this easier. And this is something that is public on that repo. I have my list of requests to AWS so that people can relay that to their own TAMs at AWS to basically say these are things we want as well. And so this includes things like, “I want an AWS account to have the ability to default to always be enforcing IMDSv2.” You know, so as an example, when you create an EC2 through the web console—which people can say, oh, you should always be using Infrastructure as Code; the reality is many folks are using the web console to create EC2s to do other changes.And when you create an EC2 in the web console, by default, it's going to allow IMDSv1 still. And so my request to AWS is, you should allow me to just default enforce IMDSv2. Also, the web console does not give you visibility into which EC2s are enforcing it and which ones are not. And also, you do not have the ability in the web console to enforce it. You cannot click on an EC2 and say, “Please enforce it now.”So, it's all these various, like, minor changes that I'm requesting AWS to do.Corey: It has to be done at instance creation time.Scott: Exactly. And so there is an API that you can make in order to change it afterwards, but that's only an API so you have to use the CLI or some other mechanism; you can't do it in the web console. But the other thing that I'm requesting AWS do is if security is a priority for AWS and they have all these other partners that are security companies, that they should be requiring their partners to also be enforcing this in their various products. So, if a partner is basically not allowing your AWS customers to enforce security best practices, then perhaps that partnership should be revoked in some way. And so that's a more aggressive thing that I'm asking AWS to do, but I think is reasonable.Corey: I'd also like them to get all of their own first-party services to support this, too.Scott: That's true as well. So, AWS is currently on the list. And so, they have one service, Data Pipelines, which if you are an AWS customer and you are using that service, you are not going to be able to enforce IMDSv2 in your environment. So, AWS themselves, unfortunately, is not allowing customers to enforce this. And then AWS themselves in their own production servers, we have seen indications that they do not enforce IMDSv2 on their own production servers.So, the best practice that they are telling customers to follow, they unfortunately are not following it themselves. And so the way in which we saw this was Orca is a security company that ended up finding this issue with AWS—and there's a lot of questions in terms of what all exactly they found—but they had this post that they called “Breaking Formation” in which they were somehow able to find—basically exploit to some degree—and again, it's unclear exactly what they were able to exploit here—but they were able to exploit AWS production servers that are responsible for the CloudFormation service. And in their blog post, they had a screenshot which showed that those production servers are not enforcing IMDSv2. And so AWS themselves is struggling with this as well, as are many customers. So, it's something that, you know, I put together this list of requests in hopes that AWS can make it easier for not only customers but also themselves to be able to enforce it.Corey: There are a lot of different things that we wish companies did differently, particularly if that company is AWS. Why is this the particular windmill that you've decided to tilt at given—let's say—it's not exactly slim pickins out there as far as changes that we wish companies would make? Obviously, you mentioned at one point, there is no drawback to enabling this, but a lot could be said for other aspects as well. Why is this one so important?Scott: So, in part, I personally have some, I guess, history with this [laugh], basically, IMDSv2, and so we can discuss this. This is back when Capital One had their breach in 2019, there was this Senator, Senator Ron Wyden, who sent this email over to AWS, to Steve Schmidt, who was the CISO at the time there and still is the CISO, and he basically—Corey: Now, he's head of security for all of Amazon.Scott: Yeah, yeah.Corey: CJ is now the AWS CISO. And he has the good sense to hide.Scott: Yeah. [laugh]. So, at the time, this Senator Ron Wyden had send over this email—and obviously it's not Senator Ron Wyden himself, you know, it's one of his, like, technical people on staff that is able to give him this information—and he sends this email to AWS saying, “Hey, this metadata service played a role in this very significant breach. Why hasn't this been fixed?” And Steve Schmidt responded, and because it's communications between a senator, I guess it has to become public.So, Steve Schmidt responds, saying that, “Hey, we never knew that this was an issue before,” is essentially what he responds with. And that irked me because I had reported this to AWS previously, as had many other people. So, there was a conference presentation by this guy Andrés Riancho at BlackHat, I believe in 2014, and he had presented previously in 2013, so it was a known issue; it had been around for a while. But I took the time to actually report it to AWS Security. So, I went through the correct channel of making sure that AWS was aware of a security concern, as a security researcher—so reporting it through that correct channel there—and provided Senator Ron Wyden with all this information.And so, then he then requested that the FTC begin a federal investigation into AWS, related to basically not following the best practices that security researchers have recommended. So, that was, kind of like, my early, I guess, involvement with this issue. So, it's something that I've been interested in for a while to make sure that this is resolved completely at some point.Corey: This episode is sponsored by our friends at Oracle Cloud. Counting the pennies, but still dreaming of deploying apps instead of “Hello, World” demos? Allow me to introduce you to Oracle's Always Free tier. It provides over 20 free services and infrastructure, networking, databases, observability, management, and security. And—let me be clear here—it's actually free. There's no surprise billing until you intentionally and proactively upgrade your account. This means you can provision a virtual machine instance or spin up an autonomous database that manages itself, all while gaining the networking, load balancing, and storage resources that somehow never quite make it into most free tiers needed to support the application that you want to build. With Always Free, you can do things like run small-scale applications or do proof-of-concept testing without spending a dime. You know that I always like to put asterisks next to the word free? This is actually free, no asterisk. Start now. Visit snark.cloud/oci-free that's snark.cloud/oci-free.Corey: It's always fun watching where people come from, as far as the security problems that they call out. There was, I believe in the cloud security forum Slack, a thread of recently about what security issues are top-of-mind and that should be fixed as a baseline expectation. In fact, let me dig it out because that is one of those things that I think is well worth having the conversation properly on this.Good examples of risky, insecure defaults in AWS. And people are talking about IMDSv1, and they're talking about all kinds of other in-depth things, and my contribution to it was, “If I go and I spin up an AWS account, until I go out of my way, I'm operating as root in that account. That seems bad.” And a few responses to that were oh, the basically facepalming, “Oh, of course.” I wish that there were an easy way to get AWS SSO as the default because it is the right answer for so many different things. It solves so many painful problems that otherwise you're going to wind up stuck with.And this stuff is hard and confusing; when people are starting out with this for the first time, they're not approaching this from, “All right, how do I be extremely secure?” They want to get some work done. For fun a year ago, I spun up a test account—unattached to any organization—and because account aliases are globally unique, I somehow came up with the account ‘shitposting' because that's pretty much what I use it for. The actual reason I wanted that was I wanted something completely unattached from any other account that I could easily take screenshots from at any point, and the worst case scenario is okay, I've exposed some credential of my own in an account that has no privileged access to anything; I just have to apologize for all the Bitcoin mining now. And honestly, I think AWS would love that marketing campaign; they'd see my face on a billboard looking horrified. It'll be great.But I turned on every security service as I went because, of course, security is the most important thing. And there were so many to turn on, and the bill was approaching 50 bucks a month for an empty account. And it's. It starts to feel a little weird and more than a little wrong.Scott: [laugh]. Yeah, my personal concern in terms of default security features is really that problem of the cost controls, I think that that still is a big issue that AWS does not have cost controls such that when a student wants to try and use AWS for the very first time and somehow they spin up large EC2 instance, or they just you know, end up creating an access key and that access key gets leaked and somehow their account gets compromised and used for Bitcoin mining, now they're stuck with that large AWS bill. For a student who has no budget, is in debt, and now is suddenly being, you know, hit with multiple thousands of dollars on their bill, that I think is very problematic, and that is something that I wish AWS would change as a default is basically, if you are creating AWS account for the very first time, have some type of—I don't know how this would look, but maybe just be able to say, like, I don't ever want this AWS account to spend more than $100 per month, and I'm okay if you end up destroying all my data in the account because I have no money and money is more important to me than whatever data I may store in here.Corey: Make an answer to that question mandatory, just as putting a credit card in is mandatory. Because there are two extremes here. It's more or less the same problem of AWS not knowing who its customers are beyond an AWS account, but there's a spectrum somewhere between I'm a student who wants to learn how the cloud works, and my approach to security is very much the same. Don't let randos spin up resources in my account, and I don't ever want to be charged. If that means you turn off my “Hello World” blog post, okay, great.On the other end, it's this is Netflix. And this is our, you know, eight-millionth account that we're spending up to do a thing and what do you mean you're applying service quotas to it? I thought we had an understanding?—everything is a service quota, let's be clear—Scott: Yep.Corey: —or a company that's about to run a Superbowl ad. Yeah, there's going to be a lot of traffic there. Don't touch it. Just make it work. We don't care what it costs.Understanding where you fall on the cost perspective—as well as a security point of view of, “We're a bank, which means forget security best practices, we have compliance obligations that cannot be altered in this account and here's what they are.” There has to be a way that is easy and approachable for people to wind up moving that slider to whatever position best represents them. Because there are accounts where I never want to be charged a thing. And that's an important thing because—and I've been talking about this for a while because I'm convinced it's a matter of time—that poor kid who wound up trading on margin at Robinhood, woke up saw that he was seven-hundred-and-some-odd grand in debt and killed himself. When it all settled out, I think he turned something like a $30,000 profit when all was said and done, which just serves to make it worse.I can see a scenario in which that happens, and part of the contributors to it are that we used to see that the surprise bill for compromised accounts was 10, 15, 20 grand. Now, they're 70 to 90 because there are more regions, more services to run containers—because of course there are—and the payoff is such that the people exploiting this have gotten very practiced and very operationalized at spinning up those resources quickly, and they cost a lot very quickly. I mean, the third use case that they're not aiming at yet is people like me, where it's, oh, you have a free account that sandboxed; I want to get the high score on the free tier because all their fraud is attuned to you making money. With me, it's nope, just going to run up the store to embarrass Amazon. That's not a common exploit vector, but I'm very much here.Scott: [laugh]. Yep. And that also is the thing though: The Denial of Wallet attack is also a concern on AWS, as well, where you've written a blog post about this, how if you are able to make use of data transfer in different ways, you can run up very high multi-million dollar bills in people's AWS accounts and even AWS's own protections and defenses against trying to look for cost spikes and things like that is delayed by multiple hours. And so you can still end up spending a lot of money in people's accounts, or one thing that's wild is an S3 object locking; that feature, the whole purpose behind it is to ensure data can never be deleted. It exists for various compliance reasons, so even AWS themselves cannot delete certain data.So, if an attacker is able to abuse that functionality in somebody's account, they can end up locking data such that for the next 100 years, it can never be deleted and you're going to have to pay for that for the next 100 years inside your account. The only way of not paying for that anymore is to move everything that you have in an AWS account to a new account, and then ask AWS to delete that account, which is not going to be reasonable under most circumstances.Corey: Yeah, alternatively, it's one of those scenarios where well, the only other option is to start physically ripping hard drives out of racks in a bunch of different data centers. It's wild to me. It's such an attack surface that honestly I believe for the longest time that AWS Security is otherworldly good. And as we start seeing from these breaches, no, what really is otherworldly good is their ability to apply pressure to people not to go public with things they discover that they then wind up keeping quiet because once this whole Orca stuff came out, we started digging, and Aidan Steele found some stuff where you could just get unfiltered, raw outputs of CloudTrail events by setting up a couple of rules in weird ways.And that was a giant problem, and it was never disclosed publicly. I don't know if any of my events were impacted; I can't trust that they would have told me if they were. And for the first time, I'm looking at things like confidential computing, which are designed around well, what if you don't trust your cloud provider? Historically, I guess I was naive because my approach was, “Well, then you shouldn't be using the cloud.” Now it's, “Well, that's actually kind of a good point.”Because it's not that I don't trust my cloud provider to necessarily do what they're telling me. I just don't trust them to tell me what they're doing. And that's part of it. The, “Well, we found an issue, but you can't prove we had an issue, so we're going to say nothing.” And when it comes to light—because it always does—it erodes trust in a big way. And trust is everything in cloud.Scott: Yeah. And so with some of the breaches that have come out, I created another GitHub repo to start tracking all the different security incidents that I could find for the three cloud providers, Azure, GCP, and AWS. And so on there, I started listing not only some of the blog posts from security companies that had been able to exploit vulnerabilities in the cloud providers, but also just anything else that I felt was a security mistake in some way. And so there's a number of things I tried to avoid on there. Like, I tried to avoid listing something that's kind of like a business decision, for example, services that get released that don't have CloudTrail support. That's a security concern to me, but that's kind of a business decision that they decided to release a service before it supported all that functionality.So, I tried to start listing off all those different things in order to also keep track of you know, is there a security provider that's worse than the others? Are there any type of common patterns that I can see? And so I tried to look through some of those different things. And that's been interesting because also I really only focus on AWS, and so I haven't really known what all has been happening with GCP and Azure. And that was interesting because there's been two issues that have happened on AWS where the exact same issue happened on the other cloud providers. And so that tells me, that's concerning to me because that tells me tht—Corey: Because those are not discovered at the same time let's be clear.Scott: Yeah. These were, like, over a year apart. And so basically, somebody had found something on GCP, and then a year-plus later, somebody else found the exact same issue on AWS. And then similarly, there was an issue with Azure and then a year-plus later, same issue on AWS. And that's concerning because that tells me that AWS may not be monitoring what are the security issues that are impacting other cloud providers, and therefore checking whether or not they happen to themselves?That's something that you would expect a mature security team to be doing is to be monitoring what are public incidents that are happening to my competitors, and am I impacted similarly? Or what can I do to try and identify those issues, fix them, make sure they never happen? All those types of steps in terms of security maturity. And that's something that then I'm a little concerned of that we've seen those issues happen before. There's also, on AWS specifically, they have had a number of issues related to their IAM-managed policies that keep cropping up.And so they have had a number of incidents where they were releasing policies that shouldn't have been released in some way. And that's concerning that showed that they don't really have a change management process that you would expect. Usually, you would expect a company to be having GitHub PRs and approval processes and things like that, in order to make sure that there's a second set of eyes on something before it gets released.Corey: Particularly things of this level of sensitivity. This is not—like, I was making fun of them a day or two ago for having broken the copyright footer and not updating them since 2020 because instead of the ‘copyright' symbol, they used an ‘at' symbol. Minor stuff, but like that's fun to needle people about, but it doesn't actually matter for anything.Scott: Yeah.Corey: Security matters and mistakes show.Scott: Yeah. And so there had been some examples where they released a policy that was called, like, ‘cheese puffs something' and it's like, okay, that's clearly, like, an internal service of some sort. But I'd called them out and, like, I'd sent an email to AWS Security being like, “Hey, you need to make sure that you have change management processes on your IAM policies because one day you're going to do something that is bad.” And one day they did. They made a change to the read-only access policy, and that basically—they removed every single privilege, somebody had ended up, you know, internally, removed every single privileges to the read-only access policy and replaced it with a whole bunch of write privileges for, I think, the Cassandra service.And so, that was like, clearly they've made a mistake that they should have made sure they were correcting because you know, they had these previous incidents. Another kind of similar one was in December, there was a support policy where they had added S3 GetObject to that policy, and that was concerning in terms of have they just given all of their support employees access to everybody's content in their S3 buckets? And so AWS made some statements saying that there were other controls in place there so it wouldn't have been possible. But it's those types of things that [crosstalk 00:33:17]—Corey: Originally, those statements were made on Twitter, let's be clear here.Scott: Yes. Yeah. [laugh].Corey: And I feel like there's a—while I deeply appreciate how accessible a lot of their senior people are, I cannot point the executive leadership team at a client to some tweets that someone made. That is not a public statement of record that works on this.Scott: Exactly.Corey: They're learning. We'll get there sooner or later, I presume. I want to thank you for taking the time to speak with me, as always, I'll throw links to these repos into the [show notes 00:33:46], but if they want to know more what you have to say, where's the best place to find you?Scott: So, my Twitter, which, unfortunately, is a handle written in hex, but it's—‘dabbadoo' is how you would pronounce it, but it's probably easiest to see a link for it. So, that's probably the main place to look for me.Corey: That's why my old Twitter handle was my amateur radio callsign. I don't use that one anymore. It's just easier. And I think that's the right answer. Besides, given what you do, it's easy enough if people want your attention. They screw up badly enough, you'll come to them.Scott: Yep. [laugh].Corey: Scott, I really appreciate your time. Thanks again.Scott: Thank you.Corey: Scott Piper, Principal Engineer at Block and, more or less, roving security troubadour for lack of a better term. I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice or a comment on the YouTubes saying that this episode is completely invalid because you wind up using the old version of the metadata service and you've never had a problem. That you know of.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.
About PeterPeter's spent more than a decade building scalable and robust systems at startups across adtech and edtech. At Remind, where he's VP of Technology, Peter pushes for building a sustainable tech company with mature software engineering. He lives in Southern California and enjoys spending time at the beach with his family.Links: Redis: https://redis.com/ Remind: https://www.remind.com/ Remind Engineering Blog: https://engineering.remind.com LinkedIn: https://www.linkedin.com/in/hamiltop Email: peterh@remind101.com TranscriptAnnouncer: Hello, and welcome to Screaming in the Cloud with your host, Chief Cloud Economist at The Duckbill Group, Corey Quinn. This weekly show features conversations with people doing interesting work in the world of cloud, thoughtful commentary on the state of the technical world, and ridiculous titles for which Corey refuses to apologize. This is Screaming in the Cloud.Corey: Today's episode is brought to you in part by our friends at MinIO the high-performance Kubernetes native object store that's built for the multi-cloud, creating a consistent data storage layer for your public cloud instances, your private cloud instances, and even your edge instances, depending upon what the heck you're defining those as, which depends probably on where you work. It's getting that unified is one of the greatest challenges facing developers and architects today. It requires S3 compatibility, enterprise-grade security and resiliency, the speed to run any workload, and the footprint to run anywhere, and that's exactly what MinIO offers. With superb read speeds in excess of 360 gigs and 100 megabyte binary that doesn't eat all the data you've gotten on the system, it's exactly what you've been looking for. Check it out today at min.io/download, and see for yourself. That's min.io/download, and be sure to tell them that I sent you.Corey: This episode is sponsored in part by our friends at Vultr. Spelled V-U-L-T-R because they're all about helping save money, including on things like, you know, vowels. So, what they do is they are a cloud provider that provides surprisingly high performance cloud compute at a price that—while sure they claim its better than AWS pricing—and when they say that they mean it is less money. Sure, I don't dispute that but what I find interesting is that it's predictable. They tell you in advance on a monthly basis what it's going to going to cost. They have a bunch of advanced networking features. They have nineteen global locations and scale things elastically. Not to be confused with openly, because apparently elastic and open can mean the same thing sometimes. They have had over a million users. Deployments take less that sixty seconds across twelve pre-selected operating systems. Or, if you're one of those nutters like me, you can bring your own ISO and install basically any operating system you want. Starting with pricing as low as $2.50 a month for Vultr cloud compute they have plans for developers and businesses of all sizes, except maybe Amazon, who stubbornly insists on having something to scale all on their own. Try Vultr today for free by visiting: vultr.com/screaming, and you'll receive a $100 in credit. Thats V-U-L-T-R.com slash screaming.Corey: Welcome to Screaming in the Cloud. I'm Corey Quinn and this is a fun episode. It is a promoted episode, which means that our friends at Redis have gone ahead and sponsored this entire episode. I asked them, “Great, who are you going to send me from, generally, your executive suite?” And they said, “Nah. You already know what we're going to say. We want you to talk to one of our customers.” And so here we are. My guest today is Peter Hamilton, VP of Technology at Remind. Peter, thank you for joining me.Peter: Thanks, Corey. Excited to be here.Corey: It's always interesting when I get to talk to people on promoted guest episodes when they're a customer of the sponsor because to be clear, you do not work for Redis. This is one of those stories you enjoy telling, but you don't personally have a stake in whether people love Redis, hate Redis, adopt that or not, which is exactly what I try and do on these shows. There's an authenticity to people who have in-the-trenches experience who aren't themselves trying to sell the thing because that is their entire job in this world.Peter: Yeah. You just presented three or four different opinions and I guarantee we felt all at the different times.Corey: [laugh]. So, let's start at the very beginning. What does Remind do?Peter: So, Remind is a messaging tool for education, largely K through 12. We support about 30 million active users across the country, over 2 million teachers, making sure that every student has, you know, equal opportunities to succeed and that we can facilitate as much learning as possible.Corey: When you say messaging that could mean a bunch of different things to a bunch of different people. Once on a lark, I wound up sitting down—this was years ago, so I'm sure the number is a woeful underestimate now—of how many AWS services I could use to send a message from me to you. And this is without going into the lunacy territory of, “Well, I can tag a thing and then mail it to you like a Snowball Edge or something.” No, this is using them as intended, I think I got 15 or 16 of them. When you say messaging, what does that mean to you?Peter: So, for us, it's about communication to the end-user. We will do everything we can to deliver whatever message a teacher or district administrator has to the user. We go through SMS, text messaging, we go through Apple and Google's push services, we go through email, we go through voice call, really pulling out all the stops we can to make sure that these important messages get out.Corey: And I can only imagine some of the regulatory pressure you almost certainly experience. It feels like it's not quite to HIPAA levels, where ohh, there's a private cause of action if any of this stuff gets out, but people are inherently sensitive about communications involving their children. I always sort of knew this in a general sense, and then I had kids myself, and oh, yeah, suddenly I really care about those sorts of things.Peter: Yeah. One of the big challenges, you can build great systems that do the correct thing, but at the end of the day, we're relying on a teacher choosing the right recipient when they send a message. And so we've had to build a lot of processes and controls in place, so that we can, kind of, satisfy two conflicting needs: One is to provide a clear audit log because that's an important thing for districts to know if something does happen, that we have clear communication; and the other is to also be able to jump in and intervene when something inappropriate or mistaken is sent out to the wrong people.Corey: Remind has always been one of those companies that has a somewhat exalted reputation in the AWS space. You folks have been early adopters of a bunch of different services—which let's be clear, in the responsible way, not the, “Well, they said it on stage; time to go ahead and put everything they just listed into production because we for some Godforsaken reason, view it as a todo list.”—but you've been thoughtful about how you approach things, and you have been around as a company for a while. But you've also been making a significant push toward being cloud-native by certain definitions of that term. So, I know this sounds like a college entrance essay, but what does cloud-native mean to you?Peter: So, one of the big gaps—if you take an application that was written to be deployed in a traditional data center environment and just drop it in the cloud, what you're going to get is a flaky data center.Corey: Well, that's unfair. It's also going to be extremely expensive.Peter: [laugh]. Sorry, an expensive, flaky data set.Corey: There we go. There we go.Peter: What we've really looked at–and a lot of this goes back to our history in the earlier days; we ran a top of Heroku and it was kind of the early days what they call the Twelve-Factor Application—but making aggressive decisions about how you structure your architecture and application so that you fit in with some of the cloud tools that are available and that you fit in, you know, with the operating models that are out there.Corey: When you say an aggressive decision, what sort of thing are you talking about? Because when I think of being aggressive with an approach to things like AWS, it usually involves Twitter, and I'm guessing that is not the direction you intend that to go.Peter: No, I think if you look at Twitter or Netflix or some of these players that, quite frankly, have defined what AWS is to us today through their usage patterns, not quite that.Corey: Oh, I mean using Twitter to yell at them explicitly about things—Peter: Oh.Corey: —because I don't do passive-aggressive; I just do aggressive.Peter: Got it. No, I think in our case, it's been plotting a very narrow path that allows us to avoid some of the bigger pitfalls. We have our sponsor here, Redis. Talk a little bit about our usage of Redis and how that's helped us in some of these cases. One of the pitfalls you'll find with pulling a non-cloud-native application and put it in the cloud is state is hard to manage.If you put state on all your machines and machines go down, networks fail, all those things, you now no longer have access to that state and we start to see a lot of problems. One of the decisions we've made is try to put as much data as we can into data stores like Redis or Postgres or something, in order to decouple our hardware from the state we're trying to manage and provide for users so that we're more resilient to those sorts of failures.Corey: I get the sense from the way that we're having this conversation, when you talk about Redis, you mean actual Redis itself, not ElastiCache for Redis, or as to I'm tending to increasingly think about AWS's services, Amazon Basics for Redis.Peter: Yeah. I mean, Amazon has launched a number of products. They have their ElastiCache, they have their new MemoryDB, there's a lot different ways to use this. We've relied pretty heavily on Redis, previously known as Redis Labs, and their enterprise product in their cloud, in order to take care of our most important data—which we just don't want to manage ourselves—trying to manage that on our own using something like ElastiCache, there's so many pitfalls, so many ways that we can lose that data. This data is important to us. By having it in a trusted place and managed by a great ops team, like they have at Redis, we're able to then lean in on the other aspects of cloud data to really get as much value as we can out of AWS.Corey: I am curious. As I said you've had a reputation as a company for a while in the AWS space of doing an awful lot of really interesting things. I mean, you have a robust GitHub presence, you have a whole bunch of tools that have come out Remind that are great, I've linked to a number of them over the years in the newsletter. You are clearly not afraid, culturally, to get your hands dirty and build things yourself, but you are using Redis Enterprise as opposed to open-source Redis. What drove that decision? I have to assume it's not, “Wait. You mean, I can get it for free as an open-source project? Why didn't someone tell me?” What brought you to that decision?Peter: Yeah, a big part of this is what we could call operating leverage. Building a great set of tools that allow you to get more value out of AWS is a little different story than babysitting servers all day and making sure they stay up. So, if you look through, most of our contributions in open-source space have really been around here's how to expand upon these foundational pieces from AWS; here's how to more efficiently launch a suite of servers into an auto-scaling group; here's, you know, our troposphere and other pieces there. This was all before Amazon CDK product, but really, it was, here's how we can more effectively use CloudFormation to capture our Infrastructure as Code. And so we are not afraid in any way to invest in our tooling and invest in some of those things, but when we look at the trade-off of directly managing stateful services and dealing with all the uncertainty that comes, we feel our time is better spent working on our product and delivering value to our users and relying on partners like Redis in order to provide that stability we need.Corey: You raise a good point. An awful lot of the tools that you've put out there are the best, from my perspective, approach to working with AWS services. And that is a relatively thin layer built on top of them with an eye toward making the user experience more polished, but not being so heavily opinionated that as soon as the service goes in a different direction, the tool becomes completely useless. You just decide to make it a bit easier to wind up working with specific environment variables or profiles, rather than what appears to be the AWS UX approach of, “Oh, now type in your access key, your secret key and your session token, and we've disabled copy and paste. Go, have fun.” You've really done a lot of quality of life improvements, more so than you have this is the entire system of how we do deploys, start to finish. It's opinionated and sort of a, like, a take on what Netflix, did once upon a time, with Asgard. It really feels like it's just the right level of abstraction.Peter: We did a pretty good job. I will say, you know, years later, we felt that we got it wrong a couple times. It's been really interesting to see that, that there are times when we say, “Oh, we could take these three or four services and wrap it up into this new concept of an application.” And over time, we just have to start poking holes in that new layer and we start to see we would have been better served by sticking with as thin a layer as possible that enables us, rather than trying to get these higher-level pieces.Corey: It's remarkably refreshing to hear you say that just because so many people love to tell the story on podcasts, or on conference stages, or whatever format they have of, “This is what we built.” And it is an aspirationally superficial story about this. They don't talk about that, “Well, firstly, without these three wrong paths first.” It's always a, “Oh, yes, obviously, we are smart people and we only make the correct decision.”And I remember in the before times sitting in conference talks, watching people talk about great things they'd done, and I'll turn next to the person next to me and say, “Wow, I wish I could be involved in a project like that.” And they'll say, “Yeah, so do I.” And it turns out they work at the company the speaker is from. Because all of these things tend to be the most positive story. Do you have an example of something that you have done in your production environment that going back, “Yeah, in hindsight, I would have done that completely differently.”Peter: Yeah. So, coming from Heroku moving into AWS, we had a great open-source project called Empire, which kind of bridge that gap between them, but used Amazon's ECS in order to launch applications. It was actually command-line compatible with the Heroku command when it first launched. So, a very big commitment there. And at the time—I mean, this comes back to the point I think you and I were talking about earlier, where architecture, costs, infrastructure, they're all interlinked.And I'm a big fan of Conway's Law, which says that an organization's structure needs to match its architecture. And so six, seven years ago, we're heavy growth-based company and we are interns running around, doing all the things, and we wanted to have really strict guardrails and a narrow set of things that our development team could do. And so we built a pretty constrained: You will launch, you will have one Docker image per ECS service, it can only do these specific things. And this allowed our development team to focus on pretty buttons on the screen and user engagement and experiments and whatnot, but as we've evolved as a company, as we built out a more robust business, we've started to track revenue and costs of goods sold more aggressively, we've seen, there's a lot of inefficient things that come out of it.One particular example was we used PgBouncer for our connection pooling to our Postgres application. In the traditional model, we had an auto-scaling group for a PgBouncer, and then our auto-scaling groups for the other applications would connect to it. And we saw additional latency, we saw additional cost, and we eventually kind of twirl that down and packaged that PgBouncer alongside the applications that needed it. And this was a configuration that wasn't available on our first pass; it was something we intentionally did not provide to our development team, and we had to unwind that. And when we did, we saw better performance, we saw better cost efficiency, all sorts of benefits that we care a lot about now that we didn't care about as much, many years ago.Corey: It sounds like you're describing some semblance of an internal platform, where instead of letting all your engineers effectively, “Well, here's the console. Ideally, you use some form of Infrastructure as Code. Good luck. Have fun.” You effectively gate access to that. Is that something that you're still doing or have you taken a different approach?Peter: So, our primary gate is our Infrastructure as Code repository. If you want to make a meaningful change, you open up a PR, got to go through code review, you need people to sign off on it. Anything that's not there may not exist tomorrow. There's no guarantees. And we've gone around, occasionally just shut random servers down that people spun up in our account.And sometimes people will be grumpy about it, but you really need to enforce that culture that we have to go through the correct channels and we have to have this cohesive platform, as you said, to support our development efforts.Corey: So, you're a messaging service in education. So, whenever I do a little bit of digging into backstories of companies and what has made, I guess, an impression, you look for certain things and explicit dates are one of them, where on March 13th of 2020, your business changed just a smidgen. What happened other than the obvious, we never went outside for two years?Peter: [laugh]. So, if we roll back a week—you know, that's March 13th, so if we roll back a week, we're looking at March 6th. On that day, we sent out about 60 million messages over all of our different mediums: Text, email, push notifications. On March 13th that was 100 million, and then, a few weeks later on March 30th, that was 177 million. And so our traffic effectively tripled over the course of those three weeks. And yeah, that's quite a ride, let me tell you.Corey: The opinion that a lot of folks have who have not gotten to play in sophisticated distributed systems is, “Well, what's the hard part there you have an auto-scaling group. Just spin up three times the number of servers in that fleet and problem solved. What's challenging?” A lot, but what did you find that the pressure points were?Peter: So, I love that example, that your auto-scaling group will just work. By default, Amazon's auto-scaling groups only support 1000 backends. So, when your auto-scaling group goes from 400 backends to 1200, things break, [laugh] and not in ways that you would have expected. You start to learn things about how database systems provided by Amazon have limits other than CPU and memory. And they're clearly laid out that there's network bandwidth limits and things you have to worry about.We had a pretty small team at that time and we'd gotten this cadence where every Monday morning, we would wake up at 4 a.m. Pacific because as part of the pandemic, our traffic shifted, so our East Coast users would be most active in the morning rather than the afternoon. And so at about 7 a.m. on the east coast is when everyone came online. And we had our Monday morning crew there and just looking to see where the next pain point was going to be.And we'd have Monday, walk through it all, Monday afternoon, we'd meet together, we come up with our three or four hypotheses on what will break, if our traffic doubles again, and we'd spend the rest of that next week addressing those the best we could and repeat for the next Monday. And we did this for three, four or five weeks in a row, and finally, it stabilized. But yeah, it's all the small little things, the things you don't know about, the limits in places you don't recognize that just catch up to you. And you need to have a team that can move fast and adapt quickly.Corey: You've been using Redis for six, seven years, something along those lines, as an enterprise offering. You've been working with the same vendor who provides this managed service for a while now. What are the fruits of that relationship? What is the value that you see by continuing to have a long-term relationship with vendors? Because let's be serious, most of us don't stay in jobs that long, let alone work with the same vendor.Peter: Yeah. So, coming back to the March 2020 story, many of our vendors started to see some issues here that various services weren't scaled properly. We made a lot of phone calls to a lot of vendors in working with them, and I… very impressed with how Redis Labs at the time was able to respond. We hopped on a call, they said, “Here's what we think we need to do, we'll go ahead and do this. We'll sort this out in a few weeks and figure out what this means for your contract. We're here to help and support in this pandemic because we recognize how this is affecting everyone around the world.”And so I think when you get in those deeper relationships, those long-term relationships, it is so helpful to have that trust, to have a little bit of that give when you need it in times of crisis, and that they're there and willing to jump in right away.Corey: There's a lot to be said for having those working relationships before you need them. So often, I think that a lot of engineering teams just don't talk to their vendors to a point where they may as well be strangers. But you'll see this most notably because—at least I feel it most acutely—with AWS service teams. They'll do a whole kickoff when the enterprise support deal is signed, three years go passed, and both the AWS team and the customer's team have completely rotated since then, and they may as well be strangers. Being able to have that relationship to fall back on in those really weird really, honestly, high-stress moments has been one of those things where I didn't see the value myself until the first time I went through a hairy situation where I found that that was useful.And now it's oh, I—I now bias instead for, “Oh, I can fit to the free tier of this service. No, no, I'm going to pay and become a paying customer.” I'd rather be a customer that can have that relationship and pick up the phone than someone whining at people in a forum somewhere of, “Hey, I'm a free user, and I'm having some problems with production.” Just never felt right to me.Peter: Yeah, there's nothing worse than calling your account rep and being told, “Oh, I'm not your account rep anymore.” Somehow you missed the email, you missed who it was. Prior to Covid, you know—and we saw this many, many years ago—one of the things about Remind is every back-to-school season, our traffic 10Xes in about three weeks. And so we're used to emergencies happening and unforeseen things happening. And we plan through our year and try to do capacity planning and everything, but we been around the block a couple of times.And so we have a pretty strong culture now leaning in hard with our support reps. We have them in our Slack channels. Our AWS team, we meet with often. Redis Labs, we have them on Slack as well. We're constantly talking about databases that may or may not be performing as we expect them, too. They're an extension of our team, we have an incident; we get paged. If it's related to one of the services, we hit them in Slack immediately and have them start checking on the back end while we're checking on our side. So.Corey: One of the biggest takeaways I wish more companies would have is that when you are dependent upon another company to effectively run your production infrastructure, they are no longer your vendor, they're your partner, whether you want them to be or not. And approaching it with that perspective really pays dividends down the road.Peter: Yeah. One of the cases you get when you've been at a company for a long time and been in relationship for a long time is growing together is always an interesting approach. And seeing, sometimes there's some painful points; sometimes you're on an old legacy version of their product that you were literally the last customer on, and you got to work with them to move off of. But you were there six years ago when they're just starting out, and they've seen how you grow, and you've seen how they've grown, and you've kind of been able to marry that experience together in a meaningful way.Corey: This episode is sponsored by our friends at Oracle Cloud. Counting the pennies, but still dreaming of deploying apps instead of “Hello, World” demos? Allow me to introduce you to Oracle's Always Free tier. It provides over 20 free services and infrastructure, networking, databases, observability, management, and security. And—let me be clear here—it's actually free. There's no surprise billing until you intentionally and proactively upgrade your account. This means you can provision a virtual machine instance or spin up an autonomous database that manages itself, all while gaining the networking, load balancing, and storage resources that somehow never quite make it into most free tiers needed to support the application that you want to build. With Always Free, you can do things like run small-scale applications or do proof-of-concept testing without spending a dime. You know that I always like to put asterisks next to the word free? This is actually free, no asterisk. Start now. Visit snark.cloud/oci-free that's snark.cloud/oci-free.Corey: Redis is, these days, of data platform back once upon a time, I viewed it as more of a caching layer. And I admit that the capabilities of the platform has significantly advanced since those days when I viewed it purely through lens of cache. But one of the interesting parts is that neither one of those use cases, in my mind, blends particularly well with heavy use of Spot Fleets, but you're doing exactly that. What are your folks doing over there?Peter: [laugh]. Yeah, so as I mentioned earlier, coming back to some of the Twelve-Factor App design, we heavily rely on Redis as sort of a distributed heap. One of our challenges of delivering all these messages is every single message has its in-flight state: Here's the content, here's who we sent it to, we wait for them to respond. On a traditional application, you might have one big server that stores it all in-memory, and you get the incoming requests, and you match things up. By moving all that state to Redis, all of our workers, all of our application servers, we know they can disappear at any point in time.We use Amazon's Spot Instances and their Spot Fleet for all of our production traffic. Every single web service, every single worker that we have runs on this infrastructure, and we would not be able to do that if we didn't have a reliable and robust place to store this data that is in-flight and currently being accessed. So, we'll have a couple hundred gigs of data at any point in time in a Redis Database, just representing in-flight work that's happening on various machines.Corey: It's really neat seeing Spot Fleets being used as something more than a theoretical possibility. It's something I've always been very interested in, obviously, given the potential cost savings; they approach cheap is free in some cases. But it turns out—we talked earlier about the idea of being cloud-native versus the rickety, expensive data center in the cloud, and an awful lot of applications are simply not built in a way that yeah, we're just going to randomly turn off a subset of your systems, ideally, with two minutes of notice, but all right, have fun with that. And a lot of times, it just becomes a complete non-starter, even for stateless workloads, just based upon how all of these things are configured. It is really interesting to watch a company that has an awful lot of responsibility that you've been entrusted with who embraces that mindset. It's a lot more rare than you'd think.Peter: Yeah. And again, you know, sometimes, we overbuild things, and sometimes we go down paths that may have been a little excessive, but it really comes down to your architecture. You know, it's not just having everything running on Spot. It's making effective use of SQS and other queueing products at Amazon to provide checkpointing abilities, and so you know that should you lose an instance, you're only going to lose a few seconds of productive work on that particular workload and be able to kick off where you left off.It's properly using auto-scaling groups. From the financial side, there's all sorts of weird quirks you'll see. You know, the Spot market has a wonderful set of dynamics where the big instances are much, much cheaper per CPU than the small ones are on the Spot market. And so structuring things in a way that you can colocate different workloads onto the same hosts and hedge against the host going down by spreading across multiple availability zones. I think there's definitely a point where having enough workload, having enough scale allows you to take advantage of these things, but it all comes down to the architecture and design that really enables it.Corey: So, you've been using Redis for longer than I think many of our listeners have been in tech.Peter: [laugh].Corey: And the key distinguishing points for me between someone who is an advocate for a technology and someone who's a zealot—or a pure critic—is they can identify use cases for which is great and use cases for which it is not likely to be a great experience. In your time with Redis, what have you found that it's been great at and what are some areas that you would encourage people to consider more carefully before diving into it?Peter: So, we like to joke that five, six years ago, most of our development process was, “I've hit a problem. Can I use Redis to solve that problem?” And so we've tried every solution possible with Redis. We've done all the things. We have number of very complicated Lua scripts that are managing different keys in an atomic way.Some of these have been more successful than others, for sure. Right now, our biggest philosophy is, if it is data we need quickly, and it is data that is important to us, we put it in Enterprise Redis, the cloud product from Redis. Other use cases, there's a dozen things that you can use for a cache, Redis is great for cache, memcache does a decent job as well; you're not going to see a meaningful difference between those sorts of products. Where we've struggled a little bit has been when we have essentially relational data that we need fast access to. And we're still trying to find a clear path forward here because you can do it and you can have atomic updates and you can kind of simulate some of the ACID characteristics you would have in a relational database, but it adds a lot of complexity.And that's a lot of overhead to our team as we're continuing to develop these products, to extend them, to fix any bugs you might have in there. And so we're kind of recalibrating a bit, and some of those workloads are moving to other data stores where they're more appropriate. But at the end of the day, it's data that we need fast, and it's data that's important, we're sticking with what we got here because it's been working pretty well.Corey: It sounds almost like you started off with the mindset of one database for a bunch of different use cases and you're starting to differentiate into purpose-built databases for certain things. Or is that not entirely accurate?Peter: There's a little bit of that. And I think coming back to some of our tooling, as we kind of jumped on a bit of the microservice bandwagon, we would see, here's a small service that only has a small amount of data that needs to be stored. It wouldn't make sense to bring up a RDS instance, or an Aurora instance, for that, you know, in Postgres. Let's just store it in an easystore like Redis. And some of those cases have been great, some of them have been a little problematic.And so as we've invested in our tooling to make all our databases accessible and make it less of a weird trade-off between what the product needs, what we can do right now, and what we want to do long-term, and reduce that friction, we've been able to be much more deliberate about the data source that we choose in each case.Corey: It's very clear that you're speaking with a voice of experience on this where this is not something that you just woke up and figured out. One last area I want to go into with you is when I asked you what is you care about primarily as an engineering leader and as you look at serving your customers well, you effectively had a dual answer, almost off the cuff, of stability and security. I find the two of those things are deeply intertwined in most of the conversations I have, but they're rarely called out explicitly in quite the way that you do. Talk to me about that.Peter: Yeah, so in our wild journey, stability has always been a challenge. And we've alway—you know, been an early startup mode, where you're constantly pushing what can we ship? How quickly can we ship it? And in our particular space, we feel that this communication that we foster between teachers and students and their parents is incredibly important, and is a thing that we take very, very seriously. And so, a couple years ago, we were trying to create this balance and create not just a language that we could talk about on a podcasts like this, but really recognizing that framing these concepts to our company internally: To our engineers to help them to think as they're building a feature, what are the things they should think about, what are the concerns beyond the product spec; to work with our marketing and sales team to help them to understand why we're making these investments that may not get particular feature out by X date but it's still a worthwhile investment.So, from the security side, we've really focused on building out robust practices and robust controls that don't necessarily lock us into a particular standard, like PCI compliance or things like that, but really focusing on the maturity of our company and, you know, our culture as we go forward. And so we're in a place now we are ISO 27001; we're heading into our third year. We leaned in hard our disaster recovery processes, we've leaned in hard on our bug bounties, pen tests, kind of, found this incremental approach that, you know, day one, I remember we turned on our bug bounty and it was a scary day as the reports kept coming in. But we take on one thing at a time and continue to build on it and make it an essential part of how we build systems.Corey: It really has to be built in. It feels like security is not something could be slapped on as an afterthought, however much companies try to do that. Especially, again, as we started this episode with, you're dealing with communication with people's kids. That is something that people have remarkably little sense of humor around. And rightfully so.Seeing that there is as much if not more care taken around security than there is stability is generally the sign of a well-run organization. If there's a security lapse, I expect certain vendors to rip the power out of their data centers rather than run in an insecure fashion. And your job done correctly—which clearly you have gotten to—means that you never have to make that decision because you've approached this the right way from the beginning. Nothing's perfect, but there's always the idea of actually caring about it being the first step.Peter: Yeah. And the other side of that was talking about stability, and again, it's avoiding the either/or situation. We can work in as well along those two—stability and security—we work in our cost of goods sold and our operating leverage in other aspects of our business. And every single one of them, it's our co-number one priorities are stability and security. And if it costs us a bit more money, if it takes our dev team a little longer, there's not a choice at that point. We're doing the correct thing.Corey: Saving money is almost never the primary objective of any company that you really want to be dealing with unless something bizarre is going on.Peter: Yeah. Our philosophy on, you know, any cost reduction has been this should have zero negative impact on our stability. If we do not feel we can safely do this, we won't. And coming back to the Spot Instance piece, that was a journey for us. And you know, we tested the waters a bit and we got to a point, we worked very closely with Amazon's team, and we came to that conclusion that we can safely do this. And we've been doing it for over a year and seen no adverse effects.Corey: Yeah. And a lot of shops I've talked to folks about well, when we go and do a consulting project, it's, “Okay. There's a lot of things that could have been done before we got here. Why hasn't any of that been addressed?” And the answer is, “Well. We tried to save money once and it caused an outage and then we weren't allowed to save money anymore. And here we are.” And I absolutely get that perspective. It's a hard balance to strike. It always is.Peter: Yeah. The other aspect where stability and security kind of intertwine is you can think about security as InfoSec in our systems and locking things down, but at the end of the day, why are we doing all that? It's for the benefit of our users. And Remind, as a communication platform, and safety and security of our users is as dependent on us being up and available so that teachers can reach out to parents with important communication. And things like attendance, things like natural disasters, or lockdowns, or any of the number of difficult situations schools find themselves in. This is part of why we take that stewardship that we have so seriously is that being up and protecting a user's data just has such a huge impact on education in this country.Corey: It's always interesting to talk to folks who insists they're making the world a better place. And it's, “What do you do?” “We're improving ad relevance.” I mean, “Okay, great, good for you.” You're serving a need that I would I would not shy away from classifying what you do, fundamentally, as critical infrastructure, and that is always a good conversation to have. It's nice being able to talk to folks who are doing things that you can unequivocally look at and say, “This is a good thing.”Peter: Yeah. And around 80% of public schools in the US are using Remind in some capacity. And so we're not a product that's used in a few civic regions. All across the board. One of my favorite things about working in Remind is meeting people and telling them where I work, and they recognize it.They say, “Oh, I have that app, I use that app. I love it.” And I spent years and ads before this, and you know, I've been there and no one ever told me they were glad to see an ad. That's never the case. And it's been quite a rewarding experience coming in every day, and as you said, being part of this critical infrastructure. That's a special thing.Corey: I look forward to installing the app myself as my eldest prepares to enter public school in the fall. So, now at least I'll have a hotline of exactly where to complain when I didn't get the attendance message because, you know, there's no customer quite like a whiny customer.Peter: They're still customers. [laugh]. Happy to have them.Corey: True. We tend to be. I want to thank you for taking so much time out of your day to speak with me. If people want to learn more about what you're up to, where's the best place to find you?Peter: So, from an engineering perspective at Remind, we have our blog, engineering.remind.com. If you want to reach out to me directly. I'm on LinkedIn; good place to find me or you can just reach out over email directly, peterh@remind101.com.Corey: And we will put all of that into the show notes. Thank you so much for your time. I appreciate it.Peter: Thanks, Corey.Corey: Peter Hamilton, VP of Technology at Remind. This has been a promoted episode brought to us by our friends at Redis, and I'm Cloud Economist Corey Quinn. If you've enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you've hated this podcast, please leave a five-star review on your podcast platform of choice, along with an angry and insulting comment that you will then hope that Remind sends out to 20 million students all at once.Corey: If your AWS bill keeps rising and your blood pressure is doing the same, then you need The Duckbill Group. We help companies fix their AWS bill by making it smaller and less horrifying. The Duckbill Group works for you, not AWS. We tailor recommendations to your business and we get to the point. Visit duckbillgroup.com to get started.Announcer: This has been a HumblePod production. Stay humble.
© 2020-2025 Ivy Podcast Discovery LLC
